# AI Proxy Configuration

The `ai-proxy` plugin connects LLM and embedding providers through a common configuration for OpenAI, DeepSeek, Anthropic, and other compatible APIs.<!-- --> <!-- -->Configuration covers static settings and plugin parameters.

## Static Configurations[​](#static-configurations "Direct link to Static Configurations")

APISIX uses `ngx_http_ffi_client` by default for upstream requests from `ai-proxy`, `ai-proxy-multi`, and `ai-request-rewrite`. Set `http_client` to `lua-resty-http` to use the Lua client instead. This setting is available from APISIX 3.18.0. API7 Gateway 3.9 and 3.10 use the Lua client and do not expose this setting.

* Host or Docker
* Kubernetes (Helm)

To use the Lua client in an APISIX host or Docker deployment, configure the following setting:

config.yaml

```
plugin_attr:

  ai-proxy:

    http_client: lua-resty-http
```

Then reload APISIX for the change to take effect.

Export the full effective values for the installed APISIX release:

```
helm get values <release-name> -n <namespace> --all -o yaml > values.yaml
```

Add or update the following value:

values.yaml

```
apisix:

  pluginAttrs:

    ai-proxy:

      http_client: lua-resty-http
```

Then apply the values file with the chart used for this APISIX release:

```
helm upgrade <release-name> apisix/apisix -n <namespace> -f values.yaml
```

## Parameters[​](#parameters "Direct link to Parameters")

See plugin [common configurations](https://docs.api7.ai/apisix/reference/plugin-common-configurations.md) for configuration options available to all plugins.

* provider

  string

  required

  valid value:

  `openai`, `deepseek`, `azure-openai`, `aimlapi`, `gemini`, `vertex-ai`, `anthropic`, `openrouter`, `bedrock`, `openai-compatible`

  ***

  LLM service provider.

  When set to `openai`, the plugin sends detected Chat Completions, Responses API, and Embeddings requests to their corresponding OpenAI endpoints.

  When set to `deepseek`, the plugin will proxy requests to `https://api.deepseek.com/chat/completions`.

  When set to `gemini` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin will proxy requests to `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions`. If you are proxying requests to an embedding model, you should configure the embedding model endpoint in the `override`.

  When set to `vertex-ai` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin proxies requests to Google Cloud Vertex AI. For chat completions, the plugin will proxy requests to `https://{region}-aiplatform.googleapis.com/v1beta1/projects/{project_id}/locations/{region}/endpoints/openapi/chat/completions`. For embeddings, the plugin will proxy requests to `https://{region}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{region}/publishers/google/models/{model}:predict`. These require configuring `provider_conf` with `project_id` and `region`. Alternatively, you can configure `override` for a custom endpoint.

  When set to `anthropic` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin sends detected Chat Completions requests to `https://api.anthropic.com/v1/chat/completions` and native Anthropic Messages requests to `https://api.anthropic.com/v1/messages`.

  When set to `openrouter` (available from APISIX 3.15.0 and Enterprise 3.9.2), the plugin will proxy requests to `https://openrouter.ai/api/v1/chat/completions`.

  When set to `bedrock` (available from API7 Enterprise 3.9.12 and APISIX 3.17.0), the plugin proxies requests to AWS Bedrock using the [Converse API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html). Requires configuring `auth.aws` with IAM credentials and `provider_conf.region` with the AWS region. Supports both non-streaming and streaming (ConverseStream) when `stream` is set to `true` in the request body.

  When set to `aimlapi` (available from APISIX 3.14.0 and Enterprise 3.8.17), the plugin uses the OpenAI-compatible driver and proxies the request to `https://api.aimlapi.com/v1/chat/completions`.

  When set to `openai-compatible`, the plugin proxies requests to the custom endpoint configured in `override`.

  When set to `azure-openai`, the plugin also proxies requests to the custom endpoint configured in `override` and additionally removes the `model` parameter from user requests.

* auth

  object

  required

  ***

  Authentication configurations.

  * header

    object

    ***

    Authentication headers.

  * query

    object

    ***

    Authentication query parameters.

  * gcp

    object

    ***

    GCP service account authentication for Vertex AI. Available in API7 Enterprise from 3.9.2 and APISIX from version 3.17.0.

    * service\_account\_json

      string

      ***

      GCP service account JSON content used for authentication. This can be configured using this parameter or by setting the `GCP_SERVICE_ACCOUNT` environment variable.

    * max\_ttl

      integer

      ***

      Maximum TTL for GCP access token caching, in seconds.

    * expire\_early\_secs

      integer

      default: `60`

      ***

      Number of seconds to expire the access token before its actual expiration time. This prevents edge cases where tokens expire during active requests.

  * aws

    object

    ***

    AWS IAM credentials for SigV4 signing. Required when `provider` is `bedrock` (for Bedrock, `auth.aws` is sufficient and `auth.header`/`auth.query` are not required). Available in API7 Enterprise from version 3.9.12 and APISIX from version 3.17.0.

    * access\_key\_id

      string

      required

      ***

      AWS IAM access key ID.

    * secret\_access\_key

      string

      required

      ***

      AWS IAM secret access key.

    * session\_token

      string

      ***

      AWS session token for temporary credentials (e.g. from STS AssumeRole).

* options

  object

  ***

  Model configurations.

  In addition to `model`, you can configure additional parameters and they will be forwarded to the upstream LLM service in the request body. For instance, if you are working with OpenAI, you can configure additional parameters such as `temperature`, `top_p`, and `stream`. See your LLM provider's API documentation for more available options.

  * model

    string

    ***

    Name of the LLM model, such as `gpt-4` or `gpt-3.5`. See your LLM provider's API documentation for more available models.

* provider\_conf

  object

  ***

  Provider-specific configuration. Required when `provider` is `bedrock`. When `provider` is `vertex-ai`, configure either `provider_conf` or `override.endpoint`.

  Available in API7 Enterprise from 3.9.2 and APISIX from version 3.17.0.

  * project\_id

    string

    ***

    Google Cloud Project ID. Required when `provider` is `vertex-ai`.

  * region

    string

    required

    ***

    Cloud region. For `vertex-ai`, this is the GCP region. For `bedrock`, this is the AWS region (e.g. `us-east-1`).

* override

  object

  ***

  Override setting.

  * endpoint

    string

    ***

    Endpoint override. If it contains only a scheme and host, the plugin appends the path selected from the provider and detected request protocol. If it contains a path or query, the plugin uses those values.

    Required when `provider` is `openai-compatible` or `azure-openai`.

    When omitted for another provider, the plugin derives the endpoint from `provider`, the detected request protocol, and `provider_conf`, where applicable. It does not use an upstream configured on the route or service.

  * llm\_options

    object

    ***

    Provider-aware LLM option overrides. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

    * max\_tokens

      integer

      ***

      Maximum number of output tokens. The gateway automatically maps this to the correct field name for the target provider (e.g. `max_completion_tokens` for OpenAI Chat, `max_output_tokens` for OpenAI Responses API). Always force-overwrites the client value.

  * request\_body

    object

    ***

    Per target-protocol request body overrides. Keys are target protocol names (`openai-chat`, `openai-responses`, `openai-embeddings`, `anthropic-messages`, `bedrock-converse`, `passthrough`); values are partial request bodies that are deep-merged into the outgoing body (objects merged recursively, arrays and scalars replaced wholesale). Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

  * request\_body\_force\_override

    boolean

    default: `false`

    ***

    When `false` (default), client request body fields take priority and `request_body` override values only fill in missing fields. When `true`, `request_body` override values forcefully overwrite client fields. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

* logging

  object

  ***

  Logging configurations. These configurations apply to access logs and logs sent to logging plugins, and do not affect the error log.

  * summaries

    boolean

    default: `false`

    ***

    If true, add an `llm_summary` object to logger entries with model, latency, and token usage. In API7 Enterprise 3.9.16 and 3.10.2, and in APISIX 3.18.0, the summary also includes stream status, tool count and usage, end-user ID, cache read and creation tokens, reasoning tokens, and content risk level when available.

  * payloads

    boolean

    default: `false`

    ***

    If true, log request and response payload.

* timeout

  integer

  default: `30000`

  valid value:

  between 1 and 600000 inclusive

  ***

  Timeout in milliseconds for each connect, send, or blocking read operation to the LLM service. It does not limit the total duration of a streaming response; use `max_stream_duration_ms` for that limit.

* max\_req\_body\_size

  integer

  default: `67108864`

  ***

  Maximum request body size in bytes that the plugin reads into memory (default 67108864 bytes, which is 64 MiB). Requests with a body larger than this limit are rejected with HTTP 413. This prevents unbounded memory buffering of large request bodies. Available in API7 Enterprise from versions 3.9.14 and 3.10.1 in their respective release lines, and APISIX from version 3.17.0.

* keepalive

  boolean

  default: `true`

  ***

  If true, keep the connection alive when requesting the LLM service.

* keepalive\_timeout

  integer

  default: `60000`

  valid value:

  greater than or equal to 1000

  ***

  Keepalive timeout in milliseconds when requesting the LLM service.

* keepalive\_pool

  integer

  default: `30`

  valid value:

  greater than or equal to 1

  ***

  Keepalive pool size for when connecting with the LLM service.

* ssl\_verify

  boolean

  default: `true`

  ***

  If true, verify the LLM service's certificate.

* max\_stream\_duration\_ms

  integer

  ***

  Maximum wall-clock duration, in milliseconds, for a streaming AI response. The limit is optional. When reached, the gateway closes the connection; if output has already started, the stream ends without a protocol terminator such as `[DONE]`, `message_stop`, or `response.completed`. Enforcement occurs between upstream reads, so the final chunk can exceed the configured duration. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

* max\_response\_bytes

  integer

  ***

  Maximum total bytes read from the upstream for one streaming or non-streaming AI response. The limit is optional and checked between upstream reads, so the final chunk can exceed it. If the limit is exceeded before output starts, the gateway returns `502 Bad Gateway`; after output starts, the gateway closes the stream without a protocol terminator. Available in API7 Enterprise from version 3.9.10 and APISIX from version 3.17.0.

* streaming\_flush\_interval\_ms

  integer

  default: `10`

  ***

  Background flush interval in milliseconds for streaming responses. A positive value starts a background thread that flushes output periodically to bound client latency when upstreams burst multiple tokens at once. Set to 0 to flush each chunk synchronously inline. Available in API7 Enterprise from version 3.9.13 and APISIX from version 3.17.0.
