# AI Proxy Multi Configuration

The `ai-proxy-multi` plugin distributes AI requests across OpenAI, DeepSeek, and other compatible APIs with load balancing, retries, fallbacks, and health checks.<!-- --> <!-- -->Configuration covers static settings and plugin parameters.

## Static Configurations[​](#static-configurations "Direct link to Static Configurations")

APISIX uses `ngx_http_ffi_client` by default for upstream requests from `ai-proxy`, `ai-proxy-multi`, and `ai-request-rewrite`. Set `http_client` to `lua-resty-http` to use the Lua client instead. This setting is available from APISIX 3.18.0. API7 Gateway 3.9 and 3.10 use the Lua client and do not expose this setting.

* Host or Docker
* Kubernetes (Helm)

To use the Lua client in an APISIX host or Docker deployment, configure the following setting:

config.yaml

```
plugin_attr:

  ai-proxy:

    http_client: lua-resty-http
```

Then reload APISIX for the change to take effect.

Export the full effective values for the installed APISIX release:

```
helm get values <release-name> -n <namespace> --all -o yaml > values.yaml
```

Add or update the following value:

values.yaml

```
apisix:

  pluginAttrs:

    ai-proxy:

      http_client: lua-resty-http
```

Then apply the values file with the chart used for this APISIX release:

```
helm upgrade <release-name> apisix/apisix -n <namespace> -f values.yaml
```

## Parameters[​](#parameters "Direct link to Parameters")

See plugin [common configurations](https://docs.api7.ai/apisix/reference/plugin-common-configurations.md) for configuration options available to all plugins.

* fallback\_strategy

  string or array

  valid value:

  string: `instance_health_and_rate_limiting`, `http_429`, or `http_5xx`

  <br />

  array: Any combination of `rate_limiting`, `http_429`, and `http_5xx`

  ***

  Fallback strategy. The option `instance_health_and_rate_limiting` is kept for backward compatibility and is functionally the same as `rate_limiting`.

  With `rate_limiting` or `instance_health_and_rate_limiting`, when the current instance's quota is exhausted, the request is forwarded to the next instance regardless of priority. With `http_429`, if an instance returns status code 429, the request is retried with other instances. With `http_5xx`, if an instance returns a 5xx status code, the request is retried with other instances. If all instances fail, the plugin returns the last upstream status, response body, and `Content-Type`.

  When not set, the plugin will not forward the request to low priority instances when tokens of the high priority instance are exhausted.

* max\_retries

  integer

  valid value:

  greater than or equal to 0

  ***

  Maximum number of fallback retries after the initial request fails. This bounds retries triggered by `fallback_strategy` or `fallback_http_statuses`, so a request does not exhaust every configured instance. When not set, there is no explicit cap and the plugin retries until an instance succeeds or all instances have been tried. Introduced in API7 Enterprise 3.9.14 and APISIX 3.17.0.

* retry\_on\_failure\_within\_ms

  integer

  valid value:

  greater than or equal to 1

  ***

  Only fall back to another instance when a failed attempt completes within this many milliseconds and its status qualifies for retry through `fallback_strategy` or `fallback_http_statuses`. A slower failure is returned to the client directly to avoid doubling the wait time. This bounds each failed attempt, not the total request duration. When not set, retries are not limited by elapsed time. Introduced in API7 Enterprise 3.9.14 and APISIX 3.17.0.

* fallback\_http\_statuses

  array\[integer]

  valid value:

  each between 400 and 599, no duplicates

  ***

  HTTP status codes from the upstream that trigger retry on another instance. This list works independently of `fallback_strategy` and can also supplement its `http_429` and `http_5xx` entries. The retry count and per-attempt time limits still apply.

  Select statuses that indicate an instance-specific problem, such as an expired provider credential. Do not retry every 4xx response; invalid client requests usually fail on every instance.

  Introduced in API7 Enterprise 3.9.19 and 3.10.6, and in APISIX 3.19.0.

* balancer

  object

  ***

  Load balancing configurations.

  * algorithm

    string

    default: `roundrobin`

    valid value:

    `roundrobin`, `chash`, or `semantic`

    ***

    Load balancing algorithm. When set to `roundrobin`, weighted round robin algorithm is used. When set to `chash`, consistent hashing algorithm is used. When set to `semantic`, the instance whose `examples` are semantically closest to the prompt is used, configured under `semantic_opts`.

    Configured health-check settings are not used while semantic selection is active, and selection does not retry after an upstream failure. Therefore, `fallback_strategy`, `fallback_http_statuses`, and `max_retries` do not affect semantic selection. An upstream failure on the selected instance is returned to the client. Semantic selection uses the fallback when no instance meets its effective threshold, no user prompt is available, embedding processing fails, or vector dimensions differ.

    The `semantic` algorithm is available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.

  * hash\_on

    string

    default: `vars`

    valid value:

    `vars`, `header`, `cookie`, `consumer`, or `vars_combinations`

    ***

    Used when `balancer.algorithm` is `chash`. Supports hashing on [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md), headers, cookies, consumers, or a combination of [built-in variables](https://docs.api7.ai/apisix/reference/built-in-variables.md).

  * key

    string

    ***

    Used when `balancer.algorithm` is `chash`. When `hash_on` is set to `header` or `cookie`, `key` is required. When `hash_on` is set to `consumer`, `key` is not required because the consumer name is used automatically.

* semantic\_opts

  object

  ***

  Configurations for the `semantic` balancer algorithm. Required when `balancer.algorithm` is `semantic`, and ignored otherwise.

  Available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.

  * embeddings

    object

    required

    ***

    Embedding service used to turn the prompt and each instance's `examples` into vectors. The prompt is embedded on every request, so this service is on the request path.

    * provider

      string

      required

      valid value:

      `openai` or `azure-openai`

      ***

      Embedding service provider.

    * model

      string

      required

      ***

      Name of the embedding model, such as `text-embedding-3-small`.

    * endpoint

      string

      ***

      Embedding API endpoint. Optional for `openai`, which defaults to the public API. Required for `azure-openai`, where it has to be the full URL, such as `https://{resource}.openai.azure.com/openai/deployments/{deployment}/embeddings?api-version={version}`.

    * auth

      object

      required

      ***

      Authentication for the embedding service, carried either as headers or as query parameters.

      * header

        object

        ***

        Key-value pairs sent as request headers to the embedding service.

      * query

        object

        ***

        Key-value pairs sent as query parameters to the embedding service.

    * timeout

      integer

      default: `3000`

      valid value:

      greater than or equal to 1

      ***

      Timeout in milliseconds for an embedding request. Because the prompt is embedded synchronously, this bounds the latency added to each request when the embedding service is slow. On a timeout the request is routed to the fallback instance rather than failed.

    * ssl\_verify

      boolean

      default: `true`

      ***

      If true, verify the embedding service's TLS certificate.

  * threshold

    number

    default: `0`

    valid value:

    between -1 and 1 inclusive

    ***

    Global minimum cosine similarity for selecting an instance. An instance's own `threshold` overrides this value. Scores must be greater than or equal to that effective threshold. The default `0` accepts nonnegative scores; `-1` accepts the full cosine-similarity range. If no instance meets its effective threshold, the fallback instance is used. Missing user prompts, embedding failures, and dimension mismatches use the fallback at any threshold.

  * fallback

    string

    ***

    Name of the instance to route to when no instance clears its threshold or the embedding request fails. It is otherwise a normally ranked instance and needs its own `examples`. Defaults to the first instance when unset.

  * debugging

    boolean

    default: `false`

    ***

    If true, return the per-instance similarity scores and the routing decision in the `X-AI-Semantic-Scores` and `X-AI-Semantic-Picked-Instance` response headers. Intended for tuning `examples` and thresholds, not for production traffic.

* instances

  array\[object]

  required

  ***

  LLM instance configurations.

  * name

    string

    required

    ***

    Name of the LLM service instance. Names must be unique across the entire `instances` array, including instances with different priorities. Configuration validation rejects duplicate names in API7 Enterprise 3.9.19 and 3.10.6 and in APISIX 3.19.0. When renaming an instance, update references in `ai-rate-limiting.instances` and `semantic_opts.fallback` as applicable.

  * examples

    array\[string]

    valid value:

    between 1 and 64 items

    ***

    Example utterances representing what this instance handles. Each one is embedded into its own reference vector, and the semantic balancer routes a request to the instance whose closest example is most similar to the prompt.

    Required for every instance when `balancer.algorithm` is `semantic`, including the instance named by `semantic_opts.fallback`. Ignored by the other algorithms.

    Available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.

  * threshold

    number

    valid value:

    between -1 and 1 inclusive

    ***

    Minimum cosine similarity a prompt has to reach for this instance to be selected by the semantic balancer. Overrides `semantic_opts.threshold` for this instance.

    Available in API7 Enterprise from version 3.9.18 on the 3.9 line and from version 3.10.5 on the 3.10 line, and in APISIX from version 3.18.0.

  * provider

    string

    required

    valid value:

    `openai`, `deepseek`, `azure-openai`, `aimlapi`, `gemini`, `vertex-ai`, `anthropic`, `openrouter`, `bedrock`, `openai-compatible`

    ***

    LLM service provider.

    When set to `openai`, the plugin sends detected Chat Completions, Responses API, and Embeddings requests to their corresponding OpenAI endpoints.

    When set to `deepseek`, the plugin will proxy requests to `https://api.deepseek.com/chat/completions`.

    When set to `gemini` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin will proxy requests to `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions`. If you are proxying requests to an embedding model, you should configure the embedding model endpoint in the `override`.

    When set to `vertex-ai` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin proxies requests to Google Cloud Vertex AI. For chat completions, the plugin will proxy requests to `https://{region}-aiplatform.googleapis.com/v1beta1/projects/{project_id}/locations/{region}/endpoints/openapi/chat/completions`. For embeddings, the plugin will proxy requests to `https://{region}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{region}/publishers/google/models/{model}:predict`. These require configuring `provider_conf` with `project_id` and `region`. Alternatively, you can configure `override` for a custom endpoint.

    When set to `anthropic` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin sends detected Chat Completions requests to `https://api.anthropic.com/v1/chat/completions` and native Anthropic Messages requests to `https://api.anthropic.com/v1/messages`.

    When set to `openrouter` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin will proxy requests to `https://openrouter.ai/api/v1/chat/completions`.

    When set to `bedrock` (available from API7 Enterprise 3.9.12 and APISIX 3.17.0), the plugin proxies requests to AWS Bedrock using the Converse API.

    When set to `aimlapi` (available from APISIX 3.14.0 and Enterprise 3.8.17), the plugin uses the OpenAI-compatible driver and proxies the request to `https://api.aimlapi.com/v1/chat/completions`.

    When set to `openai-compatible`, the plugin proxies requests to the custom endpoint configured in `override`.

    When set to `azure-openai`, the plugin also proxies requests to the custom endpoint configured in `override` and additionally removes the `model` parameter from user requests.

  * priority

    integer

    default: `0`

    ***

    Priority of the LLM instance in load balancing. `priority` takes precedence over `weight`.

  * weight

    integer

    required

    valid value:

    greater than or equal to 0

    ***

    Weight of the LLM instance in load balancing.

  * auth

    object

    required

    ***

    Authentication configurations.

    * header

      object

      ***

      Authentication headers. At least one of the `header` and `query` should be configured. You can configure additional custom headers that will be forwarded to the upstream LLM service.

    * query

      object

      ***

      Authentication query parameters. At least one of the `header` and `query` should be configured.

    * gcp

      object

      ***

      GCP service account authentication for Vertex AI. Available in API7 Enterprise from 3.9.2 and APISIX from version 3.17.0.

      * service\_account\_json

        string

        ***

        GCP service account JSON content used for authentication. This can be configured using this parameter or by setting the `GCP_SERVICE_ACCOUNT` environment variable.

      * max\_ttl

        integer

        ***

        Maximum TTL for GCP access token caching, in seconds.

      * expire\_early\_secs

        integer

        default: `60`

        ***

        Number of seconds to expire the access token before its actual expiration time. This prevents edge cases where tokens expire during active requests.

    * aws

      object

      ***

      AWS IAM credentials for SigV4 signing. Required when `provider` is `bedrock` (for Bedrock, `auth.aws` is sufficient and `auth.header`/`auth.query` are not required). Available in API7 Enterprise from version 3.9.12 and APISIX from version 3.17.0.

      * access\_key\_id

        string

        required

        ***

        AWS IAM access key ID.

      * secret\_access\_key

        string

        required

        ***

        AWS IAM secret access key.

      * session\_token

        string

        ***

        AWS session token for temporary credentials (e.g. from STS AssumeRole).

  * options

    object

    ***

    Model configurations.

    In addition to `model`, you can configure additional parameters and they will be forwarded to the upstream LLM service in the request body. For instance, if you are working with OpenAI or DeepSeek, you can configure additional parameters such as `max_tokens`, `temperature`, `top_p`, and `stream`. See your LLM provider's API documentation for more available options.

    * model

      string

      ***

      Name of the LLM model, such as `gpt-4` or `gpt-3.5`. See your LLM provider's API documentation for more available models.

  * provider\_conf

    object

    ***

    Provider-specific configuration. Required when `provider` is `bedrock`. When `provider` is `vertex-ai`, configure either `provider_conf` or `override.endpoint`.

    Available in API7 Enterprise from 3.9.2 and APISIX from version 3.17.0.

    * project\_id

      string

      ***

      Google Cloud Project ID for Vertex AI.

    * region

      string

      required

      ***

      Cloud region. For `vertex-ai`, this is the GCP region. For `bedrock`, this is the AWS region (e.g. `us-east-1`).

  * override

    object

    ***

    Override setting.

    * endpoint

      string

      ***

      Endpoint override. If it contains only a scheme and host, the plugin appends the path selected from this instance's provider and detected request protocol. If it contains a path or query, the plugin uses those values.

      Required when `provider` is `openai-compatible` or `azure-openai`.

      When omitted for another provider, the plugin derives the endpoint from this instance's `provider`, the detected request protocol, and `provider_conf`, where applicable. It does not use an upstream configured on the route or service.

    * llm\_options

      object

      ***

      Provider-aware LLM option overrides. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

      * max\_tokens

        integer

        ***

        Maximum number of output tokens. The gateway automatically maps this to the correct field name for the target provider, such as `max_completion_tokens` for OpenAI Chat or `max_output_tokens` for OpenAI Responses API, and overwrites the client value.

    * request\_body

      object

      ***

      Per target-protocol request body overrides. Keys are target protocol names, such as `openai-chat`, `openai-responses`, `openai-embeddings`, `anthropic-messages`, `bedrock-converse`, and `passthrough`. Values are partial request bodies that are deep-merged into the outgoing body. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

    * request\_body\_force\_override

      boolean

      default: `false`

      ***

      When `false` (default), client request body fields take priority and `request_body` override values only fill in missing fields. When `true`, `request_body` override values overwrite client fields. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

  * checks

    object

    ***

    Health check configurations for this instance. These checks are independent of health checks on an upstream configured on the route or service.

    The `roundrobin` and `chash` balancers use instance health checks only when at least two instances are configured. With one instance, the plugin sends requests to it without starting its active health check. The `semantic` balancer does not use instance health checks.

    Note that at the moment, OpenAI and DeepSeek do not provide an official health check endpoint. Other LLM services that you can configure under `openai-compatible` provider may have available health check endpoints.

    * active

      object

      required

      ***

      Active health check configurations.

      * type

        string

        default: `http`

        valid value:

        `http`, `https`, or `tcp`

        ***

        Type of health check connection.

      * timeout

        number

        default: `1`

        ***

        Health check timeout in seconds.

      * concurrency

        integer

        default: `10`

        ***

        Number of upstream nodes to be checked at the same time.

      * host

        string

        ***

        HTTP host.

      * port

        integer

        valid value:

        between 1 and 65535 inclusive

        ***

        HTTP port.

      * http\_path

        string

        default: `/`

        ***

        Path for HTTP probing requests.

      * http\_method

        string

        default: `GET`

        valid value:

        `CONNECT`, `DELETE`, `GET`, `HEAD`, `OPTIONS`, `PATCH`, `POST`, `PURGE`, `PUT`, or `TRACE`

        ***

        HTTP method for active health check probing requests. Available in API7 Enterprise and in APISIX from version 3.18.0.

      * http\_req\_body

        string

        ***

        Request body to send in active health check probing requests. This is useful when `http_method` is set to `POST`. Defaults to empty string. Available in API7 Enterprise and in APISIX from version 3.18.0.

      * https\_verify\_certificate

        boolean

        default: `true`

        ***

        If true, verify the node's TLS certificate.

      * healthy

        object

        ***

        Healthy check configurations.

        * interval

          integer

          default: `1`

          ***

          Time interval of checking healthy nodes, in seconds.

        * http\_statuses

          array\[integer]

          default: `[200,302]`

          valid value:

          status code between 200 and 599 inclusive

          ***

          An array of HTTP status codes that defines a healthy node.

        * successes

          integer

          default: `2`

          valid value:

          between 1 and 254 inclusive

          ***

          Number of successful probes to define a healthy node.

      * req\_headers

        array\[string]

        ***

        List of additional HTTP headers to send in health check probing requests, in `"Header: Value"` format.

      * unhealthy

        object

        ***

        Unhealthy check configurations.

        * interval

          integer

          default: `1`

          ***

          Time interval of checking unhealthy nodes, in seconds.

        * http\_statuses

          array\[integer]

          default: `[429,404,500,501,502,503,504,505]`

          valid value:

          status code between 200 and 599 inclusive

          ***

          An array of HTTP status codes that defines an unhealthy node.

        * http\_failures

          integer

          default: `5`

          valid value:

          between 1 and 254 inclusive

          ***

          Number of HTTP failures to define an unhealthy node.

        * tcp\_failures

          integer

          default: `2`

          valid value:

          between 1 and 254 inclusive

          ***

          Number of TCP failures to define an unhealthy node.

        * timeouts

          integer

          default: `3`

          valid value:

          between 1 and 254 inclusive

          ***

          Number of probe timeouts to define an unhealthy node.

* logging

  object

  ***

  Logging configurations. These configurations apply to access logs and logs sent to logging plugins, and do not affect the error log.

  * summaries

    boolean

    default: `false`

    ***

    If true, add an `llm_summary` object to logger entries with model, latency, and token usage. In API7 Enterprise 3.9.16 and 3.10.2, and in APISIX 3.18.0, the summary also includes stream status, tool count and usage, end-user ID, cache read and creation tokens, reasoning tokens, and content risk level when available.

  * payloads

    boolean

    default: `false`

    ***

    If true, log request and response payload.

* timeout

  integer

  default: `30000`

  valid value:

  between 1 and 600000 inclusive

  ***

  Timeout in milliseconds for each connect, send, or blocking read operation to the LLM service. It does not limit the total duration of a streaming response; use `max_stream_duration_ms` for that limit.

* max\_req\_body\_size

  integer

  default: `67108864`

  valid value:

  greater than or equal to 1

  ***

  Maximum request body size in bytes that the plugin reads into memory. Larger requests are rejected with HTTP 413. This prevents unbounded memory buffering of large request bodies. The default is 67108864 bytes (64 MiB). Available in API7 Enterprise from versions 3.9.14 and 3.10.1 in their respective release lines, and APISIX from version 3.17.0.

* max\_stream\_duration\_ms

  integer

  valid value:

  greater than or equal to 1

  ***

  Maximum wall-clock duration, in milliseconds, for a streaming AI response. The limit is optional. When reached, the gateway closes the connection; if output has already started, the stream ends without a protocol terminator such as `[DONE]`, `message_stop`, or `response.completed`. Enforcement occurs between upstream reads, so the final chunk can exceed the configured duration. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

* max\_response\_bytes

  integer

  valid value:

  greater than or equal to 1

  ***

  Maximum total bytes read from the upstream for one streaming or non-streaming AI response. The limit is optional and checked between upstream reads, so the final chunk can exceed it. If the limit is exceeded before output starts, the gateway returns `502 Bad Gateway`; after output starts, the gateway closes the stream without a protocol terminator. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

* streaming\_flush\_interval\_ms

  integer

  default: `10`

  valid value:

  greater than or equal to 0

  ***

  Background flush interval in milliseconds for streaming responses. A positive value periodically flushes buffered output to bound client latency when the upstream sends tokens in bursts. Set to 0 to flush each chunk synchronously. Available in API7 Enterprise from version 3.9.13 and APISIX from version 3.17.0.

* keepalive

  boolean

  default: `true`

  ***

  If true, keep the connection alive when requesting the LLM service.

* keepalive\_timeout

  integer

  default: `60000`

  valid value:

  greater than or equal to 1000

  ***

  Keepalive timeout in milliseconds when requesting the LLM service.

* keepalive\_pool

  integer

  default: `30`

  valid value:

  greater than or equal to 1

  ***

  Keepalive pool size for when connecting with the LLM service.

* ssl\_verify

  boolean

  default: `true`

  ***

  If true, verify the LLM service's certificate.
