# AI Proxy Multi

The `ai-proxy-multi` plugin distributes LLM and embedding requests across OpenAI, DeepSeek, Gemini, Gemini Enterprise Agent Platform, and other compatible provider instances. It adds load balancing, retries, fallbacks, and health checks.

The plugin sends requests directly to the selected instance endpoint. An upstream is not required. If the route or service has an upstream configured, the plugin bypasses it, so its health status does not affect these requests.

Before forwarding a request, the plugin identifies the client protocol from the request URI and body. Protocol conversion applies when the client uses Anthropic Messages but the selected instance's provider supports OpenAI Chat Completions instead of native Anthropic Messages. The plugin converts the client request to OpenAI format and the backend response to Anthropic format. This conversion supports only a subset of Anthropic Messages. See [Protocol Reference](https://docs.api7.ai/hub/ai-proxy-protocol-reference.md?plugin=ai-proxy-multi) for detection rules, field mappings, release boundaries, and limitations.

The plugin can also record LLM request information in the access log, not the error log, including token usage, model, and time to first response. Logging plugins can consume these entries.

<!-- -->

## Demo[​](#demo "Direct link to Demo")

The demo shows [instance priority and rate limiting](#configure-instance-priority-and-rate-limiting) in the API7 Enterprise Dashboard. It assigns different priorities to two models and rate limits the higher-priority instance. When that instance exhausts its quota and `fallback_strategy` is `["rate_limiting"]`, requests continue to the lower-priority instance.

## Examples[​](#examples "Direct link to Examples")

The following examples configure `ai-proxy-multi` for common routing and resilience scenarios.

Give every instance a unique name, even when priorities differ. If you rename an instance, update its [AI rate-limiting](https://docs.api7.ai/hub/ai-rate-limiting.md) entry and semantic fallback reference too.

### Load Balance between Instances[​](#load-balance-between-instances "Direct link to Load Balance between Instances")

The following example distributes traffic between two models at an 80:20 ratio. If the first attempt returns `429` or `5xx` within two seconds, the plugin retries one other instance. The `max_retries` and `retry_on_failure_within_ms` fields were introduced in API7 Enterprise 3.9.14 and APISIX 3.17.0.

The example uses OpenAI and DeepSeek instances so that the selected upstream is easy to distinguish.

Create a route and replace the provider-specific values as needed:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "fallback_strategy": ["http_429", "http_5xx"],

        "max_retries": 1,

        "retry_on_failure_within_ms": 2000,

        "instances": [

          {

            "name": "openai-instance",

            "provider": "openai",

            "weight": 8,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4.1-mini"

            }

          },

          {

            "name": "deepseek-instance",

            "provider": "deepseek",

            "weight": 2,

            "auth": {

              "header": {

                "Authorization": "Bearer $DEEPSEEK_API_KEY"

              }

            },

            "options": {

              "model": "deepseek-chat"

            }

          }

        ]

      }

    }

  }

EOF
```

❶ Configure the weight for `openai-instance` to be `8`.

❷ Configure the weight for `deepseek-instance` to be `2`.

adc.yaml

```
services:

  - name: ai-proxy-multi-load-balancing-service

    labels:

      docs-example: ai-proxy-multi-load-balancing

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            fallback_strategy:

              - http_429

              - http_5xx

            max_retries: 1

            retry_on_failure_within_ms: 2000

            instances:

              - name: openai-instance

                provider: openai

                weight: 8

                auth:

                  header:

                    Authorization: Bearer ${OPENAI_API_KEY}

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                weight: 2

                auth:

                  header:

                    Authorization: Bearer ${DEEPSEEK_API_KEY}

                options:

                  model: deepseek-chat

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-load-balancing
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-load-balancing
```

❶ Configure the weight for `openai-instance` to be `8`.

❷ Configure the weight for `deepseek-instance` to be `2`.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        fallback_strategy:

          - http_429

          - http_5xx

        max_retries: 1

        retry_on_failure_within_ms: 2000

        instances:

          - name: openai-instance

            provider: openai

            weight: 8

            auth:

              header:

                Authorization: "Bearer <OPENAI_API_KEY>"

            options:

              model: gpt-4.1-mini

          - name: deepseek-instance

            provider: deepseek

            weight: 2

            auth:

              header:

                Authorization: "Bearer <DEEPSEEK_API_KEY>"

            options:

              model: deepseek-chat

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Configure the weight for `openai-instance` to be `8`.

❷ Configure the weight for `deepseek-instance` to be `2`.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            fallback_strategy:

              - http_429

              - http_5xx

            max_retries: 1

            retry_on_failure_within_ms: 2000

            instances:

              - name: openai-instance

                provider: openai

                weight: 8

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                weight: 2

                auth:

                  header:

                    Authorization: "Bearer <DEEPSEEK_API_KEY>"

                options:

                  model: deepseek-chat
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Configure the weight for `openai-instance` to be `8`.

❷ Configure the weight for `deepseek-instance` to be `2`.

The ADC and Gateway API tabs include a conventional backend. When the plugin handles the request itself, the backend is not contacted. Verify the plugin-specific result below; a backend response alone does not confirm that the plugin ran.

Send 10 POST requests with a system prompt and sample user question. Count how many are forwarded to OpenAI and DeepSeek:

```
openai_count=0

deepseek_count=0



for i in {1..10}; do 

  model=$(curl -s "http://127.0.0.1:9080/anything" -X POST \

    -H "Content-Type: application/json" \

    -d '{

      "messages": [

        { "role": "system", "content": "You are a mathematician" },

        { "role": "user", "content": "What is 1+1?" }

      ]

    }' | jq -r '.model')



  if [[ "$model" == *"gpt-4"* ]]; then

    ((openai_count++))

  elif [[ "$model" == "deepseek-chat" ]]; then

    ((deepseek_count++))

  fi

done



echo "OpenAI responses: $openai_count"

echo "DeepSeek responses: $deepseek_count"
```

You should see a response similar to the following:

```
OpenAI responses: 8

DeepSeek responses: 2
```

### Load Balance Responses API Requests[​](#load-balance-responses-api-requests "Direct link to Load Balance Responses API Requests")

The following example configures `ai-proxy-multi` on the OpenAI Responses API route and distributes requests across two OpenAI models.

Responses and Embeddings requests both contain `input`. Keep the `/v1/responses` URI suffix so that the plugin identifies this request as Responses instead of Embeddings. See [Request Protocol Detection](https://docs.api7.ai/hub/ai-proxy-protocol-reference.md?plugin=ai-proxy-multi#request-protocol-detection).

Create a route with `uri` set to `/v1/responses`:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-responses-route",

    "uri": "/v1/responses",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "instances": [

          {

            "name": "openai-responses-primary",

            "provider": "openai",

            "weight": 1,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4.1"

            }

          },

          {

            "name": "openai-responses-secondary",

            "provider": "openai",

            "weight": 1,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4.1-mini"

            }

          }

        ]

      }

    }

  }

EOF
```

adc.yaml

```
services:

  - name: ai-proxy-multi-responses-service

    labels:

      docs-example: ai-proxy-multi-responses

    routes:

      - name: ai-proxy-multi-responses-route

        uris:

          - /v1/responses

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            instances:

              - name: openai-responses-primary

                provider: openai

                weight: 1

                auth:

                  header:

                    Authorization: Bearer ${OPENAI_API_KEY}

                options:

                  model: gpt-4.1

              - name: openai-responses-secondary

                provider: openai

                weight: 1

                auth:

                  header:

                    Authorization: Bearer ${OPENAI_API_KEY}

                options:

                  model: gpt-4.1-mini

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-responses
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-responses
```

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-responses-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        instances:

          - name: openai-responses-primary

            provider: openai

            weight: 1

            auth:

              header:

                Authorization: Bearer <OPENAI_API_KEY>

            options:

              model: gpt-4.1

          - name: openai-responses-secondary

            provider: openai

            weight: 1

            auth:

              header:

                Authorization: Bearer <OPENAI_API_KEY>

            options:

              model: gpt-4.1-mini

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-responses-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /v1/responses

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-responses-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-responses-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-responses-route

      match:

        paths:

          - /v1/responses

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            instances:

              - name: openai-responses-primary

                provider: openai

                weight: 1

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: gpt-4.1

              - name: openai-responses-secondary

                provider: openai

                weight: 1

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: gpt-4.1-mini
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

Send a request using the OpenAI Responses API format:

```
curl "http://127.0.0.1:9080/v1/responses" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "input": "Write one sentence about API gateways."

  }'
```

The request is forwarded to one of the configured OpenAI instances and the response is returned in the Responses API format.

### Route by Semantic Similarity[​](#route-by-semantic-similarity "Direct link to Route by Semantic Similarity")

The `semantic` balancer selects an instance by comparing the request prompt with example utterances assigned to each instance. It is available in API7 Enterprise 3.9.18 and 3.10.5, and in APISIX 3.18.0.

Export API keys for the embedding request and LLM requests:

```
export EMBEDDING_API_KEY="<YOUR_EMBEDDING_API_KEY>"

export LLM_API_KEY="<YOUR_LLM_API_KEY>"
```

Create a route with instances for programming, translation, and general prompts. The general instance is also the fallback when no score reaches the configured threshold or the embedding request fails:

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-semantic-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "balancer": {

          "algorithm": "semantic"

        },

        "semantic_opts": {

          "embeddings": {

            "provider": "openai",

            "model": "text-embedding-3-small",

            "auth": {

              "header": {

                "Authorization": "Bearer $EMBEDDING_API_KEY"

              }

            }

          },

          "threshold": 0.5,

          "fallback": "default",

          "debugging": true

        },

        "instances": [

          {

            "name": "code",

            "provider": "openai",

            "weight": 1,

            "auth": {

              "header": {

                "Authorization": "Bearer $LLM_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4o"

            },

            "examples": [

              "write a Python function",

              "debug this code"

            ]

          },

          {

            "name": "translate",

            "provider": "openai",

            "weight": 1,

            "auth": {

              "header": {

                "Authorization": "Bearer $LLM_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4o-mini"

            },

            "examples": [

              "translate this sentence into English"

            ]

          },

          {

            "name": "default",

            "provider": "openai",

            "weight": 1,

            "auth": {

              "header": {

                "Authorization": "Bearer $LLM_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4o-mini"

            },

            "examples": [

              "what is the weather today",

              "tell me a joke"

            ]

          }

        ]

      }

    }

  }

EOF
```

Send a programming request:

```
curl -i "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "messages": [

      {

        "role": "user",

        "content": "Help me debug this Python code"

      }

    ]

  }'
```

You should receive `HTTP/1.1 200 OK`. The response headers show the selected instance and the scores used to make the decision:

```
X-AI-Semantic-Picked-Instance: code

X-AI-Semantic-Scores: code:0.8213,default:0.2451,translate:0.1904
```

The scores depend on the embedding model and example utterances. Use them to choose a threshold that separates prompts that should be routed by similarity from prompts that should use the fallback. The fallback instance still requires `examples` because it participates in normal semantic ranking.

Configured health-check settings are not used while semantic selection is active, and selection does not retry after an upstream failure. Therefore, `fallback_strategy`, `fallback_http_statuses`, and `max_retries` do not affect semantic selection. An upstream failure from the selected instance is returned to the client. The semantic fallback is used only when no instance reaches its threshold or the embedding request fails.

The debugging headers expose instance names and scores. Disable `semantic_opts.debugging` after tuning and use the warning log emitted for fallback decisions in production.

### Load Balance between Gemini API and Agent Platform[​](#load-balance-between-gemini-api-and-agent-platform "Direct link to Load Balance between Gemini API and Agent Platform")

The following example load balances OpenAI-compatible Chat Completions requests across the Gemini API in Google AI Studio and Gemini Enterprise Agent Platform, previously documented as Vertex AI. The instance weights send approximately 70% of requests to Google AI Studio and 30% to Agent Platform. The released provider identifier for the second instance remains `vertex-ai`. This provider was introduced in API7 Enterprise 3.9.2 and APISIX 3.15.0.

Before proceeding:

1. For Google AI Studio Gemini, [obtain a Gemini API key](https://ai.google.dev/gemini-api/docs/api-key).
2. For Agent Platform, follow the [Agent Platform setup](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/start) instructions to enable billing and the Agent Platform API. Create a service account, grant it the Agent Platform User (`roles/aiplatform.user`) role, and [create a service-account key](https://docs.cloud.google.com/iam/docs/keys-create-delete) in JSON format. The plugin requires the downloaded key and does not discover Application Default Credentials or Workload Identity Federation credentials.

Save the API key, the raw service account JSON for the ADC example, and a JSON-encoded string for the Admin API example:

```
export GEMINI_API_KEY="<GEMINI_API_KEY>"

export GCP_SA_JSON="$(cat credentials.json)"

export GCP_SA_JSON_STRING="$(jq -Rs . < credentials.json)"
```

Create a route and replace the project-specific values:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-google-ai-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "fallback_strategy": ["rate_limiting"],

        "instances": [

          {

            "name": "gemini-instance",

            "provider": "gemini",

            "weight": 7,

            "auth": {

              "header": {

                "Authorization": "Bearer $GEMINI_API_KEY"

              }

            },

            "options": {

              "model": "gemini-3.6-flash"

            }

          },

          {

            "name": "vertex-ai-instance",

            "provider": "vertex-ai",

            "weight": 3,

            "auth": {

              "gcp": {

                "service_account_json": $GCP_SA_JSON_STRING

              }

            },

            "override": {

              "endpoint": "https://aiplatform.googleapis.com/v1beta1/projects/<GCP_PROJECT_ID>/locations/global/endpoints/openapi/chat/completions"

            },

            "options": {

              "model": "google/gemini-3.5-flash"

            }

          }

        ]

      }

    }

  }

EOF
```

❶ Configure the provider to be `gemini` for Google AI Studio Gemini access.

❷ Replace with your Gemini API key in the `Authorization` header.

❸ Specify the name of the Gemini model through Google AI Studio in the `<model>` format.

❹ Configure the provider as `vertex-ai` for Agent Platform access.

❺ Provide the service account credentials as a JSON string.

❻ Replace `<GCP_PROJECT_ID>` in the documented global Chat Completions endpoint with your project ID.

❼ Specify an Agent Platform model that supports Chat Completions on the global endpoint. The example uses `google/gemini-3.5-flash`, which supports the default Standard PayGo consumption mode on `global`.

adc.yaml

```
services:

  - name: ai-proxy-multi-google-service

    labels:

      docs-example: ai-proxy-multi-google

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            fallback_strategy:

              - rate_limiting

            instances:

              - name: gemini-instance

                provider: gemini

                weight: 7

                auth:

                  header:

                    Authorization: Bearer ${GEMINI_API_KEY}

                options:

                  model: gemini-3.6-flash

              - name: vertex-ai-instance

                provider: vertex-ai

                weight: 3

                auth:

                  gcp:

                    service_account_json: ${GCP_SA_JSON}

                override:

                  endpoint: https://aiplatform.googleapis.com/v1beta1/projects/<GCP_PROJECT_ID>/locations/global/endpoints/openapi/chat/completions

                options:

                  model: google/gemini-3.5-flash

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-google
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-google
```

❶ Configure the provider to be `gemini` for Google AI Studio Gemini access.

❷ Replace with your Gemini API key in the `Authorization` header.

❸ Specify the name of the Gemini model through Google AI Studio in the `<model>` format.

❹ Configure the provider as `vertex-ai` for Agent Platform access.

❺ Provide the service account credentials as a JSON string.

❻ Replace `<GCP_PROJECT_ID>` in the documented global Chat Completions endpoint with your project ID.

❼ Specify an Agent Platform model that supports Chat Completions on the global endpoint. The example uses `google/gemini-3.5-flash`, which supports the default Standard PayGo consumption mode on `global`.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        fallback_strategy:

          - rate_limiting

        instances:

          - name: gemini-instance

            provider: gemini

            weight: 7

            auth:

              header:

                Authorization: "Bearer <GEMINI_API_KEY>"

            options:

              model: gemini-3.6-flash

          - name: vertex-ai-instance

            provider: vertex-ai

            weight: 3

            auth:

              gcp:

                service_account_json: '{"type":"service_account","project_id":"<GCP_PROJECT_ID>","private_key_id":"...","private_key":"-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----","client_email":"<SERVICE_ACCOUNT_EMAIL>","client_id":"...","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/<URL_ENCODED_SERVICE_ACCOUNT_EMAIL>","universe_domain":"googleapis.com"}'

            override:

              endpoint: https://aiplatform.googleapis.com/v1beta1/projects/<GCP_PROJECT_ID>/locations/global/endpoints/openapi/chat/completions

            options:

              model: google/gemini-3.5-flash

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Configure the provider to be `gemini` for Google AI Studio Gemini access.

❷ Replace with your Gemini API key in the `Authorization` header.

❸ Specify the name of the Gemini model through Google AI Studio in the `<model>` format.

❹ Configure the provider as `vertex-ai` for Agent Platform access.

❺ Provide the service account credentials as a JSON string.

❻ Replace `<GCP_PROJECT_ID>` in the documented global Chat Completions endpoint with your project ID.

❼ Specify an Agent Platform model that supports Chat Completions on the global endpoint. The example uses `google/gemini-3.5-flash`, which supports the default Standard PayGo consumption mode on `global`.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            fallback_strategy:

              - rate_limiting

            instances:

              - name: gemini-instance

                provider: gemini

                weight: 7

                auth:

                  header:

                    Authorization: "Bearer <GEMINI_API_KEY>"

                options:

                  model: gemini-3.6-flash

              - name: vertex-ai-instance

                provider: vertex-ai

                weight: 3

                auth:

                  gcp:

                    service_account_json: '{"type":"service_account","project_id":"<GCP_PROJECT_ID>","private_key_id":"...","private_key":"-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----","client_email":"<SERVICE_ACCOUNT_EMAIL>","client_id":"...","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/<URL_ENCODED_SERVICE_ACCOUNT_EMAIL>","universe_domain":"googleapis.com"}'

                override:

                  endpoint: https://aiplatform.googleapis.com/v1beta1/projects/<GCP_PROJECT_ID>/locations/global/endpoints/openapi/chat/completions

                options:

                  model: google/gemini-3.5-flash
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Configure the provider to be `gemini` for Google AI Studio Gemini access.

❷ Replace with your Gemini API key in the `Authorization` header.

❸ Specify the name of the Gemini model through Google AI Studio in the `<model>` format.

❹ Configure the provider as `vertex-ai` for Agent Platform access.

❺ Provide the service account credentials as a JSON string.

❻ Replace `<GCP_PROJECT_ID>` in the documented global Chat Completions endpoint with your project ID.

❼ Specify an Agent Platform model that supports Chat Completions on the global endpoint. The example uses `google/gemini-3.5-flash`, which supports the default Standard PayGo consumption mode on `global`.

Send 10 POST requests to the route to see the load balancing distribution:

```
studio_count=0

vertex_count=0



for i in {1..10}; do 

  model=$(curl -s "http://127.0.0.1:9080/anything" -X POST \

    -H "Content-Type: application/json" \

    -d '{

      "messages": [

        { "role": "system", "content": "You are a mathematician" },

        { "role": "user", "content": "What is 1+1?" }

      ]

    }' | jq -r '.model')



  if [[ "$model" == "gemini-3.6-flash" ]]; then

    ((studio_count++))

  elif [[ "$model" == "google/gemini-3.5-flash" ]]; then

    ((vertex_count++))

  fi

done



echo "Google AI Studio Gemini responses: $studio_count"

echo "Agent Platform responses: $vertex_count"
```

You should see a response similar to the following:

```
Google AI Studio Gemini responses: 7

Agent Platform responses: 3
```

### Configure Instance Priority and Rate Limiting[​](#configure-instance-priority-and-rate-limiting "Direct link to Configure Instance Priority and Rate Limiting")

The following example assigns different priorities to two models and rate limits the higher-priority instance. With `fallback_strategy` set to `["rate_limiting"]`, requests move to the lower-priority instance after the higher-priority instance exhausts its quota.

Create a route and replace the provider-specific values as needed:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "fallback_strategy": ["rate_limiting"],

        "instances": [

          {

            "name": "openai-instance",

            "provider": "openai",

            "priority": 1,

            "weight": 0,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4.1-mini"

            }

          },

          {

            "name": "deepseek-instance",

            "provider": "deepseek",

            "priority": 0,

            "weight": 0,

            "auth": {

              "header": {

                "Authorization": "Bearer $DEEPSEEK_API_KEY"

              }

            },

            "options": {

              "model": "deepseek-chat"

            }

          }

        ]

      },

      "ai-rate-limiting": {

        "instances": [

          {

            "name": "openai-instance",

            "limit": 10,

            "time_window": 60

          }

        ],

        "limit_strategy": "total_tokens"

      }

    }

  }

EOF
```

❶ Set the `fallback_strategy` to `["rate_limiting"]`.

❷ Set a higher priority for `openai-instance`.

❸ Set a lower priority for `deepseek-instance`.

❹ Configure a quota of 10 tokens.

❺ Configure the time window to be 60 seconds.

❻ Configure the `limit_strategy` to `total_tokens`.

adc.yaml

```
services:

  - name: ai-proxy-multi-priority-service

    labels:

      docs-example: ai-proxy-multi-priority

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            fallback_strategy:

              - rate_limiting

            instances:

              - name: openai-instance

                provider: openai

                priority: 1

                weight: 0

                auth:

                  header:

                    Authorization: Bearer ${OPENAI_API_KEY}

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                priority: 0

                weight: 0

                auth:

                  header:

                    Authorization: Bearer ${DEEPSEEK_API_KEY}

                options:

                  model: deepseek-chat

          ai-rate-limiting:

            policy: local

            instances:

              - name: openai-instance

                limit: 10

                time_window: 60

            limit_strategy: total_tokens

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-priority
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-priority
```

❶ Set the `fallback_strategy` to `["rate_limiting"]`.

❷ Set a higher priority for `openai-instance`.

❸ Set a lower priority for `deepseek-instance`.

❹ Configure a quota of 10 tokens.

❺ Configure the time window to be 60 seconds.

❻ Configure the `limit_strategy` to `total_tokens`.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        fallback_strategy:

          - rate_limiting

        instances:

          - name: openai-instance

            provider: openai

            priority: 1

            weight: 0

            auth:

              header:

                Authorization: "Bearer <OPENAI_API_KEY>"

            options:

              model: gpt-4.1-mini

          - name: deepseek-instance

            provider: deepseek

            priority: 0

            weight: 0

            auth:

              header:

                Authorization: "Bearer <DEEPSEEK_API_KEY>"

            options:

              model: deepseek-chat

    - name: ai-rate-limiting

      config:

        policy: local

        instances:

          - name: openai-instance

            limit: 10

            time_window: 60

        limit_strategy: total_tokens

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Set the `fallback_strategy` to `["rate_limiting"]`.

❷ Set a higher priority for `openai-instance`.

❸ Set a lower priority for `deepseek-instance`.

❹ Configure a quota of 10 tokens.

❺ Configure the time window to be 60 seconds.

❻ Configure the `limit_strategy` to `total_tokens`.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            fallback_strategy:

              - rate_limiting

            instances:

              - name: openai-instance

                provider: openai

                priority: 1

                weight: 0

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                priority: 0

                weight: 0

                auth:

                  header:

                    Authorization: "Bearer <DEEPSEEK_API_KEY>"

                options:

                  model: deepseek-chat

        - name: ai-rate-limiting

          enable: true

          config:

            policy: local

            instances:

              - name: openai-instance

                limit: 10

                time_window: 60

            limit_strategy: total_tokens
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Set the `fallback_strategy` to `["rate_limiting"]`.

❷ Set a higher priority for `openai-instance`.

❸ Set a lower priority for `deepseek-instance`.

❹ Configure a quota of 10 tokens.

❺ Configure the time window to be 60 seconds.

❻ Configure the `limit_strategy` to `total_tokens`.

Send a POST request to the route with a system prompt and a sample user question in the request body:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "What is 1+1?" }

    ]

  }'
```

You should receive a response similar to the following:

```
{

  ...,

  "model": "gpt-4.1-mini-2025-04-14",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "1+1 equals 2.",

        "refusal": null

      },

      "logprobs": null,

      "finish_reason": "stop"

    }

  ],

  "usage": {

    "prompt_tokens": 23,

    "completion_tokens": 8,

    "total_tokens": 31,

    "prompt_tokens_details": {

      "cached_tokens": 0,

      "audio_tokens": 0

    },

    "completion_tokens_details": {

      "reasoning_tokens": 0,

      "audio_tokens": 0,

      "accepted_prediction_tokens": 0,

      "rejected_prediction_tokens": 0

    }

  },

  "service_tier": "default",

  "system_fingerprint": null

}
```

Since the `total_tokens` value exceeds the configured quota of `10`, the next request within the 60-second window is expected to be forwarded to the other instance.

Within the same 60-second window, send another POST request to the route:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "Explain Newton law" }

    ]

  }'
```

You should see a response similar to the following:

```
{

  ...,

  "model": "deepseek-chat",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"

      },

      ...

    }

  ],

  ...

}
```

### Load Balance and Rate Limit by Consumers[​](#load-balance-and-rate-limit-by-consumers "Direct link to Load Balance and Rate Limit by Consumers")

The following example load balances across two models and applies separate rate limits to each consumer.

Create the `ai-multi-johndoe` consumer with a 10-token quota per 60 seconds for `openai-instance`:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "username": "ai-multi-johndoe",

    "plugins": {

      "ai-rate-limiting": {

        "instances": [

          {

            "name": "openai-instance",

            "limit": 10,

            "time_window": 60

          }

        ],

        "rejected_code": 429,

        "policy": "local",

        "limit_strategy": "total_tokens"

      }

    }

  }'
```

Add a `key-auth` credential for `ai-multi-johndoe`:

```
curl "http://127.0.0.1:9180/apisix/admin/consumers/ai-multi-johndoe/credentials" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "id": "cred-john-key-auth",

    "plugins": {

      "key-auth": {

        "key": "john-key"

      }

    }

  }'
```

Create the `ai-multi-janedoe` consumer with a 10-token quota per 60 seconds for `deepseek-instance`:

```
curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "username": "ai-multi-janedoe",

    "plugins": {

      "ai-rate-limiting": {

        "instances": [

          {

            "name": "deepseek-instance",

            "limit": 10,

            "time_window": 60

          }

        ],

        "rejected_code": 429,

        "policy": "local",

        "limit_strategy": "total_tokens"

      }

    }

  }'
```

Add a `key-auth` credential for `ai-multi-janedoe`:

```
curl "http://127.0.0.1:9180/apisix/admin/consumers/ai-multi-janedoe/credentials" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d '{

    "id": "cred-jane-key-auth",

    "plugins": {

      "key-auth": {

        "key": "jane-key"

      }

    }

  }'
```

adc.yaml

```
consumers:

  - username: ai-multi-johndoe

    labels:

      docs-example: ai-proxy-multi-consumer-rate-limit

    plugins:

      ai-rate-limiting:

        instances:

          - name: openai-instance

            limit: 10

            time_window: 60

        rejected_code: 429

        policy: local

        limit_strategy: total_tokens

    credentials:

      - name: key-auth

        type: key-auth

        config:

          key: john-key

  - username: ai-multi-janedoe

    labels:

      docs-example: ai-proxy-multi-consumer-rate-limit

    plugins:

      ai-rate-limiting:

        instances:

          - name: deepseek-instance

            limit: 10

            time_window: 60

        rejected_code: 429

        policy: local

        limit_strategy: total_tokens

    credentials:

      - name: key-auth

        type: key-auth

        config:

          key: jane-key
```

Review the diff for unexpected changes or deletions before syncing these consumers:

```
adc diff -f adc.yaml \

  --include-resource-type consumer \

  --label-selector docs-example=ai-proxy-multi-consumer-rate-limit
```

Synchronize the reviewed consumers, including their nested credentials:

```
adc sync -f adc.yaml \

  --include-resource-type consumer \

  --label-selector docs-example=ai-proxy-multi-consumer-rate-limit
```

* Gateway API
* APISIX CRD

ai-proxy-multi-consumer-ic.yaml

```
apiVersion: apisix.apache.org/v1alpha1

kind: Consumer

metadata:

  namespace: aic

  name: ai-multi-johndoe

spec:

  gatewayRef:

    name: apisix

  plugins:

    - name: ai-rate-limiting

      config:

        instances:

          - name: openai-instance

            limit: 10

            time_window: 60

        rejected_code: 429

        policy: local

        limit_strategy: total_tokens

  credentials:

    - type: key-auth

      name: primary-key

      config:

        key: john-key

---

apiVersion: apisix.apache.org/v1alpha1

kind: Consumer

metadata:

  namespace: aic

  name: ai-multi-janedoe

spec:

  gatewayRef:

    name: apisix

  plugins:

    - name: ai-rate-limiting

      config:

        instances:

          - name: deepseek-instance

            limit: 10

            time_window: 60

        rejected_code: 429

        policy: local

        limit_strategy: total_tokens

  credentials:

    - type: key-auth

      name: primary-key

      config:

        key: jane-key
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-consumer-ic.yaml
```

ai-proxy-multi-consumer-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixConsumer

metadata:

  namespace: aic

  name: ai-multi-johndoe

spec:

  ingressClassName: apisix

  authParameter:

    keyAuth:

      value:

        key: john-key

  plugins:

    ai-rate-limiting:

      instances:

        - name: openai-instance

          limit: 10

          time_window: 60

      rejected_code: 429

      policy: local

      limit_strategy: total_tokens

---

apiVersion: apisix.apache.org/v2

kind: ApisixConsumer

metadata:

  namespace: aic

  name: ai-multi-janedoe

spec:

  ingressClassName: apisix

  authParameter:

    keyAuth:

      value:

        key: jane-key

  plugins:

    ai-rate-limiting:

      instances:

        - name: deepseek-instance

          limit: 10

          time_window: 60

      rejected_code: 429

      policy: local

      limit_strategy: total_tokens
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-consumer-ic.yaml
```

Create a route and replace the provider-specific values as needed:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "key-auth": {},

      "ai-proxy-multi": {

        "fallback_strategy": ["rate_limiting"],

        "instances": [

          {

            "name": "openai-instance",

            "provider": "openai",

            "weight": 0,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4.1-mini"

            }

          },

          {

            "name": "deepseek-instance",

            "provider": "deepseek",

            "weight": 0,

            "auth": {

              "header": {

                "Authorization": "Bearer $DEEPSEEK_API_KEY"

              }

            },

            "options": {

              "model": "deepseek-chat"

            }

          }

        ]

      }

    }

  }

EOF
```

❶ Enable `key-auth` on the route.

❷ Configure an `openai` instance.

❸ Configure a `deepseek` instance.

adc.yaml

```
services:

  - name: ai-proxy-multi-consumer-rate-limit-service

    labels:

      docs-example: ai-proxy-multi-consumer-rate-limit

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          key-auth: {}

          ai-proxy-multi:

            fallback_strategy:

              - rate_limiting

            instances:

              - name: openai-instance

                provider: openai

                weight: 0

                auth:

                  header:

                    Authorization: Bearer ${OPENAI_API_KEY}

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                weight: 0

                auth:

                  header:

                    Authorization: Bearer ${DEEPSEEK_API_KEY}

                options:

                  model: deepseek-chat

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-consumer-rate-limit
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-consumer-rate-limit
```

❶ Enable `key-auth` on the route.

❷ Configure an `openai` instance.

❸ Configure a `deepseek` instance.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: key-auth

      config:

        _meta:

          disable: false

    - name: ai-proxy-multi

      config:

        fallback_strategy:

          - rate_limiting

        instances:

          - name: openai-instance

            provider: openai

            weight: 0

            auth:

              header:

                Authorization: "Bearer <OPENAI_API_KEY>"

            options:

              model: gpt-4.1-mini

          - name: deepseek-instance

            provider: deepseek

            weight: 0

            auth:

              header:

                Authorization: "Bearer <DEEPSEEK_API_KEY>"

            options:

              model: deepseek-chat

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Enable `key-auth` on the route.

❷ Configure an `openai` instance.

❸ Configure a `deepseek` instance.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: key-auth

          enable: true

        - name: ai-proxy-multi

          enable: true

          config:

            fallback_strategy:

              - rate_limiting

            instances:

              - name: openai-instance

                provider: openai

                weight: 0

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                weight: 0

                auth:

                  header:

                    Authorization: "Bearer <DEEPSEEK_API_KEY>"

                options:

                  model: deepseek-chat
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Enable `key-auth` on the route.

❷ Configure an `openai` instance.

❸ Configure a `deepseek` instance.

Send a POST request to the route without any consumer key:

```
curl -i "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "What is 1+1?" }

    ]

  }'
```

You should receive an `HTTP/1.1 401 Unauthorized` response.

Send a POST request to the route with `ai-multi-johndoe`'s key:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -H 'apikey: john-key' \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "What is 1+1?" }

    ]

  }'
```

You should receive a response similar to the following:

```
{

  ...,

  "model": "gpt-4.1-mini-2025-04-14",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "1+1 equals 2.",

        "refusal": null

      },

      "logprobs": null,

      "finish_reason": "stop"

    }

  ],

  "usage": {

    "prompt_tokens": 23,

    "completion_tokens": 8,

    "total_tokens": 31,

    "prompt_tokens_details": {

      "cached_tokens": 0,

      "audio_tokens": 0

    },

    "completion_tokens_details": {

      "reasoning_tokens": 0,

      "audio_tokens": 0,

      "accepted_prediction_tokens": 0,

      "rejected_prediction_tokens": 0

    }

  },

  "service_tier": "default",

  "system_fingerprint": null

}
```

Since the `total_tokens` value exceeds the configured quota of the `openai` instance for `ai-multi-johndoe`, the next request within the 60-second window from `ai-multi-johndoe` is expected to be forwarded to the `deepseek` instance.

Within the same 60-second window, send another POST request to the route with `ai-multi-johndoe`'s key:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -H 'apikey: john-key' \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "Explain Newtons laws to me" }

    ]

  }'
```

You should see a response similar to the following:

```
{

  ...,

  "model": "deepseek-chat",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"

      },

      ...

    }

  ],

  ...

}
```

Send a POST request to the route with `ai-multi-janedoe`'s key:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -H 'apikey: jane-key' \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "What is 1+1?" }

    ]

  }'
```

You should receive a response similar to the following:

```
{

  ...,

  "model": "deepseek-chat",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "The sum of 1 and 1 is 2. This is a basic arithmetic operation where you combine two units to get a total of two units."

      },

      "logprobs": null,

      "finish_reason": "stop"

    }

  ],

  "usage": {

    "prompt_tokens": 14,

    "completion_tokens": 31,

    "total_tokens": 45,

    "prompt_tokens_details": {

      "cached_tokens": 0

    },

    "prompt_cache_hit_tokens": 0,

    "prompt_cache_miss_tokens": 14

  },

  "system_fingerprint": "fp_3a5770e1b4_prod0225"

}
```

Since the `total_tokens` value exceeds the configured quota of the `deepseek` instance for `ai-multi-janedoe`, the next request within the 60-second window from `ai-multi-janedoe` is expected to be forwarded to the `openai` instance.

Within the same 60-second window, send another POST request to the route with `ai-multi-janedoe`'s key:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -H 'apikey: jane-key' \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "Explain Newtons laws to me" }

    ]

  }'
```

You should see a response similar to the following:

```
{

  ...,

  "model": "gpt-4.1-mini-2025-04-14",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "Sure, here are Newton's three laws of motion:\n\n1) Newton's First Law, also known as the Law of Inertia, states that an object at rest will stay at rest, and an object in motion will stay in motion, unless acted on by an external force. In simple words, this law suggests that an object will keep doing whatever it is doing until something causes it to do otherwise. \n\n2) Newton's Second Law states that the force acting on an object is equal to the mass of that object times its acceleration (F=ma). This means that force is directly proportional to mass and acceleration. The heavier the object and the faster it accelerates, the greater the force.\n\n3) Newton's Third Law, also known as the law of action and reaction, states that for every action, there is an equal and opposite reaction. Essentially, any force exerted onto a body will create a force of equal magnitude but in the opposite direction on the object that exerted the first force.\n\nRemember, these laws become less accurate when considering speeds near the speed of light (where Einstein's theory of relativity becomes more appropriate) or objects very small or very large. However, for everyday situations, they provide a good model of how things move.",

        "refusal": null

      },

      "logprobs": null,

      "finish_reason": "stop"

    }

  ],

  ...

}
```

This shows `ai-proxy-multi` load balance the traffic with respect to the rate limiting rules in `ai-rate-limiting` by consumers.

### Restrict Maximum Number of Completion Tokens[​](#restrict-maximum-number-of-completion-tokens "Direct link to Restrict Maximum Number of Completion Tokens")

The following example limits the completion tokens that an instance can generate.

The example uses OpenAI and DeepSeek instances so that the selected upstream is easy to distinguish.

Create a route and replace the provider-specific values as needed:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "instances": [

          {

            "name": "openai-instance",

            "provider": "openai",

            "weight": 0,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4.1-mini",

              "max_tokens": 50

            }

          },

          {

            "name": "deepseek-instance",

            "provider": "deepseek",

            "weight": 0,

            "auth": {

              "header": {

                "Authorization": "Bearer $DEEPSEEK_API_KEY"

              }

            },

            "options": {

              "model": "deepseek-chat",

              "max_tokens": 100

            }

          }

        ]

      }

    }

  }

EOF
```

❶ Configure the `max_tokens` for the OpenAI instance to be `50`.

❷ Configure the `max_tokens` for the DeepSeek instance to be `100`.

adc.yaml

```
services:

  - name: ai-proxy-multi-max-tokens-service

    labels:

      docs-example: ai-proxy-multi-max-tokens

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            instances:

              - name: openai-instance

                provider: openai

                weight: 0

                auth:

                  header:

                    Authorization: Bearer ${OPENAI_API_KEY}

                options:

                  model: gpt-4.1-mini

                  max_tokens: 50

              - name: deepseek-instance

                provider: deepseek

                weight: 0

                auth:

                  header:

                    Authorization: Bearer ${DEEPSEEK_API_KEY}

                options:

                  model: deepseek-chat

                  max_tokens: 100

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-max-tokens
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-max-tokens
```

❶ Configure the `max_tokens` for the OpenAI instance to be `50`.

❷ Configure the `max_tokens` for the DeepSeek instance to be `100`.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        instances:

          - name: openai-instance

            provider: openai

            weight: 0

            auth:

              header:

                Authorization: "Bearer <OPENAI_API_KEY>"

            options:

              model: gpt-4.1-mini

              max_tokens: 50

          - name: deepseek-instance

            provider: deepseek

            weight: 0

            auth:

              header:

                Authorization: "Bearer <DEEPSEEK_API_KEY>"

            options:

              model: deepseek-chat

              max_tokens: 100

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Configure the `max_tokens` for the OpenAI instance to be `50`.

❷ Configure the `max_tokens` for the DeepSeek instance to be `100`.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            instances:

              - name: openai-instance

                provider: openai

                weight: 0

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: gpt-4.1-mini

                  max_tokens: 50

              - name: deepseek-instance

                provider: deepseek

                weight: 0

                auth:

                  header:

                    Authorization: "Bearer <DEEPSEEK_API_KEY>"

                options:

                  model: deepseek-chat

                  max_tokens: 100
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Configure the `max_tokens` for the OpenAI instance to be `50`.

❷ Configure the `max_tokens` for the DeepSeek instance to be `100`.

Send a POST request to the route with a system prompt and a sample user question in the request body:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "Explain Newtons law" }

    ]

  }'
```

If the request is proxied to OpenAI, you should see a response similar to the following, where the content is truncated per 50 `max_tokens` threshold:

```
{

  ...,

  "model": "gpt-4.1-mini-2025-04-14",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "Newton's Laws of Motion are three physical laws that form the bedrock for classical mechanics. They describe the relationship between a body and the forces acting upon it, and the body's motion in response to those forces. \n\n1. Newton's First Law",

        "refusal": null

      },

      "logprobs": null,

      "finish_reason": "length"

    }

  ],

  "usage": {

    "prompt_tokens": 20,

    "completion_tokens": 50,

    "total_tokens": 70,

    "prompt_tokens_details": {

      "cached_tokens": 0,

      "audio_tokens": 0

    },

    "completion_tokens_details": {

      "reasoning_tokens": 0,

      "audio_tokens": 0,

      "accepted_prediction_tokens": 0,

      "rejected_prediction_tokens": 0

    }

  },

  "service_tier": "default",

  "system_fingerprint": null

}
```

If the request is proxied to DeepSeek, you should see a response similar to the following, where the content is truncated per 100 `max_tokens` threshold:

```
{

  ...,

  "model": "deepseek-chat",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "Newton's Laws of Motion are three fundamental principles that form the foundation of classical mechanics. They describe the relationship between a body and the forces acting upon it, and the body's motion in response to those forces. Here's a brief explanation of each law:\n\n1. **Newton's First Law (Law of Inertia):**\n   - **Statement:** An object will remain at rest or in uniform motion in a straight line unless acted upon by an external force.\n   - **Explanation:** This law"

      },

      "logprobs": null,

      "finish_reason": "length"

    }

  ],

  "usage": {

    "prompt_tokens": 10,

    "completion_tokens": 100,

    "total_tokens": 110,

    "prompt_tokens_details": {

      "cached_tokens": 0

    },

    "prompt_cache_hit_tokens": 0,

    "prompt_cache_miss_tokens": 10

  },

  "system_fingerprint": "fp_3a5770e1b4_prod0225"

}
```

### Proxy to Embedding Models[​](#proxy-to-embedding-models "Direct link to Proxy to Embedding Models")

The following example distributes Embeddings requests between OpenAI and Azure OpenAI. The Azure instance uses the `openai-compatible` provider so the deployment name remains in the request body required by Azure OpenAI's v1 API. Both Responses and Embeddings requests contain `input`; because this route does not end in `/v1/responses`, the plugin identifies the request as Embeddings.

Create a route and replace the provider-specific values as needed:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "instances": [

          {

            "name": "openai-instance",

            "provider": "openai",

            "weight": 0,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "text-embedding-3-small"

            }

          },

          {

            "name": "az-openai-instance",

            "provider": "openai-compatible",

            "weight": 0,

            "auth": {

              "header": {

                "api-key": "$AZ_OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "<AZURE_OPENAI_EMBEDDING_DEPLOYMENT>"

            },

            "override": {

              "endpoint": "https://<AZURE_OPENAI_RESOURCE>.openai.azure.com/openai/v1/embeddings"

            }

          }

        ]

      }

    }

  }

EOF
```

❶ Specify the OpenAI embedding model. The `openai` provider sends the detected Embeddings request to `https://api.openai.com/v1/embeddings`.

❷ Specify the Azure OpenAI embedding deployment name.

❸ Specify the Azure OpenAI v1 Embeddings endpoint.

adc.yaml

```
services:

  - name: ai-proxy-multi-embeddings-service

    labels:

      docs-example: ai-proxy-multi-embeddings

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            instances:

              - name: openai-instance

                provider: openai

                weight: 0

                auth:

                  header:

                    Authorization: Bearer ${OPENAI_API_KEY}

                options:

                  model: text-embedding-3-small

              - name: az-openai-instance

                provider: openai-compatible

                weight: 0

                auth:

                  header:

                    api-key: ${AZ_OPENAI_API_KEY}

                options:

                  model: <AZURE_OPENAI_EMBEDDING_DEPLOYMENT>

                override:

                  endpoint: "https://<AZURE_OPENAI_RESOURCE>.openai.azure.com/openai/v1/embeddings"

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-embeddings
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-embeddings
```

❶ Specify the OpenAI embedding model. The `openai` provider sends the detected Embeddings request to `https://api.openai.com/v1/embeddings`.

❷ Specify the Azure OpenAI embedding deployment name.

❸ Specify the Azure OpenAI v1 Embeddings endpoint.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        instances:

          - name: openai-instance

            provider: openai

            weight: 0

            auth:

              header:

                Authorization: "Bearer <OPENAI_API_KEY>"

            options:

              model: text-embedding-3-small

          - name: az-openai-instance

            provider: openai-compatible

            weight: 0

            auth:

              header:

                api-key: "<AZURE_OPENAI_API_KEY>"

            options:

              model: <AZURE_OPENAI_EMBEDDING_DEPLOYMENT>

            override:

              endpoint: "https://<AZURE_OPENAI_RESOURCE>.openai.azure.com/openai/v1/embeddings"

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Specify the OpenAI embedding model. The `openai` provider sends the detected Embeddings request to `https://api.openai.com/v1/embeddings`.

❷ Specify the Azure OpenAI embedding deployment name.

❸ Specify the Azure OpenAI v1 Embeddings endpoint.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            instances:

              - name: openai-instance

                provider: openai

                weight: 0

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: text-embedding-3-small

              - name: az-openai-instance

                provider: openai-compatible

                weight: 0

                auth:

                  header:

                    api-key: "<AZURE_OPENAI_API_KEY>"

                options:

                  model: <AZURE_OPENAI_EMBEDDING_DEPLOYMENT>

                override:

                  endpoint: "https://<AZURE_OPENAI_RESOURCE>.openai.azure.com/openai/v1/embeddings"
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Specify the OpenAI embedding model. The `openai` provider sends the detected Embeddings request to `https://api.openai.com/v1/embeddings`.

❷ Specify the Azure OpenAI embedding deployment name.

❸ Specify the Azure OpenAI v1 Embeddings endpoint.

Send a POST request to the route with an input string:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "input": "hello world"

  }'
```

You should receive a response similar to the following:

```
{

  "object": "list",

  "data": [

    {

      "object": "embedding",

      "index": 0,

      "embedding": [

        -0.0067144386,

        -0.039197803,

        0.034177095,

        0.028763203,

        -0.024785956,

        -0.04201061,

        ...

      ],

    }

  ],

  "model": "text-embedding-3-small",

  "usage": {

    "prompt_tokens": 2,

    "total_tokens": 2

  }

}
```

### Enable Active Health Checks[​](#enable-active-health-checks "Direct link to Enable Active Health Checks")

The following example distributes requests equally across two OpenAI-compatible instances and checks the second. If all its targets become unhealthy, the gateway uses the first instance. Checks can be enabled on one or more instances.

Instance health checks are independent of route or service upstream checks. The `roundrobin` and `chash` balancers use them only when at least two instances are configured; single-instance and `semantic` selection do not use them.

For the Admin API and ADC examples, export the required values:

```
export YOUR_LLM_API_KEY="<API_KEY>"

export YOUR_LLM_MODEL="<MODEL>"

export YOUR_LLM_ENDPOINT_1="https://<LLM_HOST_1>/v1/chat/completions"

export YOUR_LLM_ENDPOINT_2="https://<LLM_HOST_2>/v1/chat/completions"

export YOUR_LLM_HEALTH_CHECK_PATH_2="/<HEALTH_CHECK_PATH_2>" # Example expects HTTP 200
```

Configure the route using your management surface:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "instances": [

          {

            "name": "llm-instance-1",

            "provider": "openai-compatible",

            "weight": 1,

            "auth": {

              "header": {

                "Authorization": "Bearer $YOUR_LLM_API_KEY"

              }

            },

            "options": {

              "model": "$YOUR_LLM_MODEL"

            },

            "override": {

              "endpoint": "$YOUR_LLM_ENDPOINT_1"

            }

          },

          {

            "name": "llm-instance-2",

            "provider": "openai-compatible",

            "weight": 1,

            "auth": {

              "header": {

                "Authorization": "Bearer $YOUR_LLM_API_KEY"

              }

            },

            "options": {

              "model": "$YOUR_LLM_MODEL"

            },

            "override": {

              "endpoint": "$YOUR_LLM_ENDPOINT_2"

            },

            "checks": {

              "active": {

                "type": "https",

                "http_path": "$YOUR_LLM_HEALTH_CHECK_PATH_2",

                "healthy": {

                  "interval": 2,

                  "successes": 1

                },

                "unhealthy": {

                  "interval": 1,

                  "http_failures": 3

                }

              }

            }

          }

        ]

      }

    }

  }

EOF
```

❶ `checks.active.type`: Match the scheme of the second instance endpoint.

❷ `checks.active.http_path`: Probe this path on the second endpoint's host and port. The path must begin with `/`; for this example, it should return HTTP `200`.

❸ `checks.active.healthy.interval`: Probe a healthy target every two seconds.

❹ `checks.active.healthy.successes`: Mark a target healthy after one successful probe.

❺ `checks.active.unhealthy.interval`: Probe an unhealthy target every second.

❻ `checks.active.unhealthy.http_failures`: Mark a target unhealthy after three probes return an unhealthy HTTP status.

adc.yaml

```
services:

  - name: ai-proxy-multi-health-checks-service

    labels:

      docs-example: ai-proxy-multi-health-checks

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            instances:

              - name: llm-instance-1

                provider: openai-compatible

                weight: 1

                auth:

                  header:

                    Authorization: Bearer ${YOUR_LLM_API_KEY}

                options:

                  model: ${YOUR_LLM_MODEL}

                override:

                  endpoint: ${YOUR_LLM_ENDPOINT_1}

              - name: llm-instance-2

                provider: openai-compatible

                weight: 1

                auth:

                  header:

                    Authorization: Bearer ${YOUR_LLM_API_KEY}

                options:

                  model: ${YOUR_LLM_MODEL}

                override:

                  endpoint: ${YOUR_LLM_ENDPOINT_2}

                checks:

                  active:

                    type: https

                    http_path: ${YOUR_LLM_HEALTH_CHECK_PATH_2}

                    healthy:

                      interval: 2

                      successes: 1

                    unhealthy:

                      interval: 1

                      http_failures: 3

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Review the diff for unexpected changes or deletions before syncing these services:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-health-checks
```

Synchronize the reviewed service configuration:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-proxy-multi-health-checks
```

❶ `checks.active.type`: Match the scheme of the second instance endpoint.

❷ `checks.active.http_path`: Probe this path on the second endpoint's host and port. The path must begin with `/`; for this example, it should return HTTP `200`.

❸ `checks.active.healthy.interval`: Probe a healthy target every two seconds.

❹ `checks.active.healthy.successes`: Mark a target healthy after one successful probe.

❺ `checks.active.unhealthy.interval`: Probe an unhealthy target every second.

❻ `checks.active.unhealthy.http_failures`: Mark a target unhealthy after three probes return an unhealthy HTTP status.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        instances:

          - name: llm-instance-1

            provider: openai-compatible

            weight: 1

            auth:

              header:

                Authorization: "Bearer <API_KEY>"

            options:

              model: your-model

            override:

              endpoint: https://<LLM_HOST_1>/v1/chat/completions

          - name: llm-instance-2

            provider: openai-compatible

            weight: 1

            auth:

              header:

                Authorization: "Bearer <API_KEY>"

            options:

              model: your-model

            override:

              endpoint: https://<LLM_HOST_2>/v1/chat/completions

            checks:

              active:

                type: https

                http_path: /<LLM_HEALTH_CHECK_PATH_2>

                healthy:

                  interval: 2

                  successes: 1

                unhealthy:

                  interval: 1

                  http_failures: 3

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ `checks.active.type`: Match the scheme of the second instance endpoint.

❷ `checks.active.http_path`: Probe this path on the second endpoint's host and port. The path must begin with `/`; for this example, it should return HTTP `200`.

❸ `checks.active.healthy.interval`: Probe a healthy target every two seconds.

❹ `checks.active.healthy.successes`: Mark a target healthy after one successful probe.

❺ `checks.active.unhealthy.interval`: Probe an unhealthy target every second.

❻ `checks.active.unhealthy.http_failures`: Mark a target unhealthy after three probes return an unhealthy HTTP status.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            instances:

              - name: llm-instance-1

                provider: openai-compatible

                weight: 1

                auth:

                  header:

                    Authorization: "Bearer <API_KEY>"

                options:

                  model: your-model

                override:

                  endpoint: https://<LLM_HOST_1>/v1/chat/completions

              - name: llm-instance-2

                provider: openai-compatible

                weight: 1

                auth:

                  header:

                    Authorization: "Bearer <API_KEY>"

                options:

                  model: your-model

                override:

                  endpoint: https://<LLM_HOST_2>/v1/chat/completions

                checks:

                  active:

                    type: https

                    http_path: /<LLM_HEALTH_CHECK_PATH_2>

                    healthy:

                      interval: 2

                      successes: 1

                    unhealthy:

                      interval: 1

                      http_failures: 3
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ `checks.active.type`: Match the scheme of the second instance endpoint.

❷ `checks.active.http_path`: Probe this path on the second endpoint's host and port. The path must begin with `/`; for this example, it should return HTTP `200`.

❸ `checks.active.healthy.interval`: Probe a healthy target every two seconds.

❹ `checks.active.healthy.successes`: Mark a target healthy after one successful probe.

❺ `checks.active.unhealthy.interval`: Probe an unhealthy target every second.

❻ `checks.active.unhealthy.http_failures`: Mark a target unhealthy after three probes return an unhealthy HTTP status.

#### Verify Active Health Checks[​](#verify-active-health-checks "Direct link to Verify Active Health Checks")

Send a request to initialize the instance checker. Replace the gateway address if it does not listen on `127.0.0.1:9080`:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "messages": [

      {

        "role": "user",

        "content": "What is 1+1?"

      }

    ]

  }'
```

Plugin-owned checker status was introduced in API7 Enterprise 3.9.20 and 3.10.7, and APISIX 3.19.0.

After at least one probe interval, inspect the checked instance through the local Control API. Run this command on the gateway host or inside the gateway container or pod so that `127.0.0.1:9090` reaches the gateway process:

```
curl -s "http://127.0.0.1:9090/v1/healthcheck" | \

  jq '.[]

    | select(.plugin == "ai-proxy-multi" and .meta.instance == "llm-instance-2")

    | {

        plugin,

        instance: .meta.instance,

        nodes: [.nodes[] | {ip, port, status}]

      }'
```

For an endpoint that resolves to one target, you should see a response similar to the following. If the endpoint resolves to multiple targets, each target appears in `nodes`.

```
{

  "plugin": "ai-proxy-multi",

  "instance": "llm-instance-2",

  "nodes": [

    {

      "ip": "<RESOLVED_IP>",

      "port": 443,

      "status": "healthy"

    }

  ]

}
```

On a test endpoint you control, make `YOUR_LLM_HEALTH_CHECK_PATH_2` return HTTP `500`. After every resolved target records three failed probes, run the Control API request again. For an endpoint that resolves to one target, you should see:

```
{

  "plugin": "ai-proxy-multi",

  "instance": "llm-instance-2",

  "nodes": [

    {

      "ip": "<RESOLVED_IP>",

      "port": 443,

      "status": "unhealthy"

    }

  ]

}
```

When all targets for the second instance are unhealthy, the gateway removes that instance from selection. Subsequent requests go to the first instance, which has no health check and remains eligible. If any target remains healthy, the second instance remains eligible. Restore a successful health response to mark a target healthy after its next successful probe.

The Control API reports local gateway state and has no authentication, so keep it on a loopback or trusted interface. See [Active Health Checks](https://docs.api7.ai/apisix/how-to-guide/traffic-management/health-check.md#configure-active-health-checks) for additional verification methods and the [Control API reference](https://docs.api7.ai/apisix/reference/control-api.md#tag/Health-Check) for response details.

### Include LLM Information in Access Log[​](#include-llm-information-in-access-log "Direct link to Include LLM Information in Access Log")

The following example adds LLM request information to the gateway access log for analytics and auditing. In addition to NGINX variables, the following variables are available:

* `apisix_upstream_response_time`: Time taken for the gateway to send the request to the upstream service and receive the full response, in milliseconds. Error responses returned by the LLM upstream are also recorded in milliseconds. Available from API7 Enterprise 3.8.8.
* `request_type`: Type of request, where the value could be `traditional_http`, `ai_chat`, or `ai_stream`.
* `llm_time_to_first_token`: Duration from request sending to the first token received from the LLM service, in milliseconds. For an upstream error response, this is the duration until the error response was received. It remains `0` if the gateway could not reach the upstream.
* `llm_model`: LLM model name forwarded to the upstream LLM service.
* `request_llm_model`: LLM model name specified in the request.
* `llm_prompt_tokens`: Number of tokens in the prompt.
* `llm_completion_tokens`: Number of completion tokens in the response.
* `llm_content_risk_level`: Content risk level reported by an AI content moderation plugin. Introduced in APISIX 3.14.0.

The gateway's AI HTTP transport also populates the standard upstream variables `upstream_addr`, `upstream_status`, `apisix_upstream_response_time`, `upstream_scheme`, `upstream_host`, `upstream_uri`, and `upstream_response_length`.

These variables apply to streaming and non-streaming AI responses in API7 Enterprise 3.9.11 and later, and APISIX 3.17.0 and later. They are also populated for APISIX requests sent through the FFI client.

The following additional variables were introduced in API7 Enterprise 3.9.14 and 3.10.1, and in APISIX 3.18.0:

* `llm_total_tokens`: Total number of tokens used, including prompt and completion tokens.
* `llm_stream`: Whether the request was a streaming request, either `true` or `false`.
* `llm_has_tool_calls`: Whether the LLM response contains tool calls, either `true` or `false`.
* `llm_tool_count`: Number of tools provided in the request.
* `llm_end_user_id`: End-user identifier extracted from the request body, such as `user`, `safety_identifier`, or `metadata.user_id`.
* `llm_cache_read_input_tokens`: Number of prompt tokens served from the provider's prompt cache.
* `llm_cache_creation_input_tokens`: Number of prompt tokens written to the provider's prompt cache.
* `llm_reasoning_tokens`: Number of reasoning tokens used by reasoning models.

Token accounting

* Anthropic: `llm_prompt_tokens` includes native input, cache-read, and cache-creation tokens. `llm_total_tokens` also includes output tokens. Cache-inclusive accounting was introduced in APISIX 3.20.0 and API7 Enterprise 3.9.22 and 3.10.9.
* Bedrock: `llm_prompt_tokens` includes `inputTokens` plus cache-read and cache-write tokens. This cache-inclusive accounting was introduced in APISIX 3.20.0 and API7 Enterprise 3.10.9. API7 Enterprise 3.9.23 uses only `inputTokens`.

These normalized prompt totals feed logs, Prometheus metrics, and built-in `ai-rate-limiting` token strategies.

Access logs and logging plugins can use these variables directly. In APISIX 3.18.0 and API7 Enterprise 3.9.18 and 3.10.5, `logging.summaries` adds produced values to `llm_summary` without the `llm_` prefix. Values not produced for the request are omitted.

To record these values in access logs, add the LLM variables to the gateway access log format:

* Host or Docker
* Kubernetes (Helm)

Add or update this section in the gateway configuration file:

config.yaml

```
nginx_config:

  http:

    access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\" \"$llm_total_tokens\" \"$llm_stream\" \"$llm_has_tool_calls\" \"$llm_tool_count\" \"$llm_end_user_id\" \"$llm_cache_read_input_tokens\" \"$llm_cache_creation_input_tokens\" \"$llm_reasoning_tokens\""
```

Reload the gateway for configuration changes to take effect.

For the APISIX Helm chart, set the following values:

values.yaml

```
apisix:

  nginx:

    logs:

      accessLogFormat: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\" \"$llm_total_tokens\" \"$llm_stream\" \"$llm_has_tool_calls\" \"$llm_tool_count\" \"$llm_end_user_id\" \"$llm_cache_read_input_tokens\" \"$llm_cache_creation_input_tokens\" \"$llm_reasoning_tokens\""
```

For the API7 Gateway Helm chart, set the following values:

values.yaml

```
logs:

  accessLogFormat: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\" \"$llm_total_tokens\" \"$llm_stream\" \"$llm_has_tool_calls\" \"$llm_tool_count\" \"$llm_end_user_id\" \"$llm_cache_read_input_tokens\" \"$llm_cache_creation_input_tokens\" \"$llm_reasoning_tokens\""
```

Then apply the values file with the chart used for this gateway release:

```
helm upgrade <release-name> <chart-name> -n <namespace> -f values.yaml
```

Create a route by following [Load Balance between Instances](#load-balance-between-instances), then send a request:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "model": "gpt-4o-mini",

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "What is 1+1?" }

    ]

  }'
```

When `ai-proxy-multi` selects the OpenAI instance, the plugin replaces the requested model with `gpt-4.1-mini`. You should receive a response similar to the following:

```
{

  ...,

  "model": "gpt-4.1-mini-2025-04-14",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "1+1 equals 2.",

        "refusal": null,

        "annotations": []

      },

      "logprobs": null,

      "finish_reason": "stop"

    }

  ],

  "usage": {

    "prompt_tokens": 23,

    "completion_tokens": 8,

    "total_tokens": 31,

    "prompt_tokens_details": {

      "cached_tokens": 0,

      "audio_tokens": 0

    },

    ...

  },

  "service_tier": "default",

  "system_fingerprint": null

}
```

In the gateway's access log, you should see a log entry similar to the following:

```
192.168.215.1 - - [<TIMESTAMP>] 127.0.0.1:9080 "POST /anything HTTP/1.1" 200 808 2.670 "-" "curl/8.6.0" - - 2670 "http://127.0.0.1:9080" "<REQUEST_ID>" "ai_chat" "2670" "gpt-4.1-mini" "gpt-4o-mini" "23" "8" "31" "false" "false" "0" "" "0" "0" "0"
```

The access log entry shows an upstream response time and time to first token of `2670` milliseconds. The request uses the `ai_chat` type, requests `gpt-4o-mini`, and is forwarded to `gpt-4.1-mini`. It uses 23 prompt tokens, 8 completion tokens, and 31 total tokens.

The remaining values show a non-streaming request with no tool calls, provided tools, end-user identifier, prompt-cache tokens, or reasoning tokens.

### Send AI Request Logs to Kafka[​](#send-ai-request-logs-to-kafka "Direct link to Send AI Request Logs to Kafka")

The following example sends AI request and response details, including model, token usage, and payloads, to Kafka. Complete the broker and `apisix-logs` topic setup in the [`kafka-logger`](https://docs.api7.ai/hub/kafka-logger.md) guide first.

Create a route to the LLM services and enable Kafka logging:

* Admin API
* ADC
* Ingress Controller

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \

  -H "X-API-KEY: ${ADMIN_API_KEY}" \

  -d @- <<EOF

  {

    "id": "ai-proxy-multi-route",

    "uri": "/anything",

    "methods": ["POST"],

    "plugins": {

      "ai-proxy-multi": {

        "instances": [

          {

            "name": "openai-instance",

            "provider": "openai",

            "weight": 8,

            "auth": {

              "header": {

                "Authorization": "Bearer $OPENAI_API_KEY"

              }

            },

            "options": {

              "model": "gpt-4.1-mini"

            }

          },

          {

            "name": "deepseek-instance",

            "provider": "deepseek",

            "weight": 2,

            "auth": {

              "header": {

                "Authorization": "Bearer $DEEPSEEK_API_KEY"

              }

            },

            "options": {

              "model": "deepseek-chat"

            }

          }

        ],

        "logging": {

          "summaries": true,

          "payloads": true

        }

      },

      "kafka-logger": {

        "brokers": [

          {

            "host": "kafka-server",

            "port": 9092

          }

        ],

        "kafka_topic": "apisix-logs",

        "key": "key1",

        "batch_max_size": 1

      }

    }

  }

EOF
```

❶ Log request LLM model, duration, request and response tokens.

❷ Log request and response payload.

❸ Update with your Kafka address.

❹ Update with your Kafka topic.

❺ Update with your Kafka key.

❻ Set to 1 to send the log entry immediately.

adc.yaml

```
services:

  - name: ai-proxy-multi-kafka-logging-service

    labels:

      docs-example: ai-kafka-logging

    routes:

      - name: ai-proxy-multi-route

        uris:

          - /anything

        methods:

          - POST

        plugins:

          ai-proxy-multi:

            instances:

              - name: openai-instance

                provider: openai

                weight: 8

                auth:

                  header:

                    Authorization: "Bearer ${OPENAI_API_KEY}"

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                weight: 2

                auth:

                  header:

                    Authorization: "Bearer ${DEEPSEEK_API_KEY}"

                options:

                  model: deepseek-chat

            logging:

              summaries: true

              payloads: true

          kafka-logger:

            brokers:

              - host: kafka-server

                port: 9092

            kafka_topic: apisix-logs

            key: key1

            batch_max_size: 1

    upstream:

      type: roundrobin

      nodes:

        - host: httpbin.org

          port: 80

          weight: 1
```

Preview the changes to services with the example label:

```
adc diff -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-kafka-logging
```

Synchronize the reviewed changes:

```
adc sync -f adc.yaml \

  --include-resource-type service \

  --label-selector docs-example=ai-kafka-logging
```

❶ Log request LLM model, duration, request and response tokens.

❷ Log request and response payload.

❸ Update with your Kafka address.

❹ Update with your Kafka topic.

❺ Update with your Kafka key.

❻ Set to 1 to send the log entry immediately.

* Gateway API
* APISIX CRD

ai-proxy-multi-ic.yaml

```
apiVersion: v1

kind: Service

metadata:

  name: httpbin-external-domain

  namespace: aic

spec:

  type: ExternalName

  externalName: httpbin.org

  ports:

    - name: http

      port: 80

      protocol: TCP

---

apiVersion: apisix.apache.org/v1alpha1

kind: PluginConfig

metadata:

  namespace: aic

  name: ai-proxy-multi-plugin-config

spec:

  plugins:

    - name: ai-proxy-multi

      config:

        instances:

          - name: openai-instance

            provider: openai

            weight: 8

            auth:

              header:

                Authorization: "Bearer <OPENAI_API_KEY>"

            options:

              model: gpt-4.1-mini

          - name: deepseek-instance

            provider: deepseek

            weight: 2

            auth:

              header:

                Authorization: "Bearer <DEEPSEEK_API_KEY>"

            options:

              model: deepseek-chat

        logging:

          summaries: true

          payloads: true

    - name: kafka-logger

      config:

        brokers:

          - host: kafka-server.aic.svc

            port: 9092

        kafka_topic: apisix-logs

        key: key1

        batch_max_size: 1

---

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  parentRefs:

    - name: apisix

  rules:

    - matches:

        - path:

            type: Exact

            value: /anything

          method: POST

      filters:

        - type: ExtensionRef

          extensionRef:

            group: apisix.apache.org

            kind: PluginConfig

            name: ai-proxy-multi-plugin-config

      backendRefs:

        - name: httpbin-external-domain

          port: 80
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Log request LLM model, duration, request and response tokens.

❷ Log request and response payload.

❸ Update with your Kafka address.

❹ Update with your Kafka topic.

❺ Update with your Kafka key.

❻ Set `batch_max_size` to 1.

ai-proxy-multi-ic.yaml

```
apiVersion: apisix.apache.org/v2

kind: ApisixRoute

metadata:

  namespace: aic

  name: ai-proxy-multi-route

spec:

  ingressClassName: apisix

  http:

    - name: ai-proxy-multi-route

      match:

        paths:

          - /anything

        methods:

          - POST

      plugins:

        - name: ai-proxy-multi

          enable: true

          config:

            instances:

              - name: openai-instance

                provider: openai

                weight: 8

                auth:

                  header:

                    Authorization: "Bearer <OPENAI_API_KEY>"

                options:

                  model: gpt-4.1-mini

              - name: deepseek-instance

                provider: deepseek

                weight: 2

                auth:

                  header:

                    Authorization: "Bearer <DEEPSEEK_API_KEY>"

                options:

                  model: deepseek-chat

            logging:

              summaries: true

              payloads: true

        - name: kafka-logger

          enable: true

          config:

            brokers:

              - host: kafka-server.aic.svc

                port: 9092

            kafka_topic: apisix-logs

            key: key1

            batch_max_size: 1
```

Apply the configuration to your cluster:

```
kubectl apply -f ai-proxy-multi-ic.yaml
```

❶ Log request LLM model, duration, request and response tokens.

❷ Log request and response payload.

❸ Update with your Kafka address.

❹ Update with your Kafka topic.

❺ Update with your Kafka key.

❻ Set to 1 to send the log entry immediately.

Send a POST request to the route:

```
curl "http://127.0.0.1:9080/anything" -X POST \

  -H "Content-Type: application/json" \

  -d '{

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "What is 1+1?" }

    ]

  }'
```

You should receive a response similar to the following if the request is forwarded to OpenAI:

```
{

  ...,

  "model": "gpt-4.1-mini-2025-04-14",

  "choices": [

    {

      "index": 0,

      "message": {

        "role": "assistant",

        "content": "1+1 equals 2.",

        "refusal": null

      },

      "logprobs": null,

      "finish_reason": "stop"

    }

  ],

  ...

}
```

In the Kafka topic, you should also see a log entry corresponding to the request with the LLM summary and request and response payloads. A request forwarded to OpenAI produces fields similar to the following:

```
{

  "llm_summary": {

    "model": "gpt-4.1-mini",

    "prompt_tokens": 23,

    "completion_tokens": 8,

    "total_tokens": 31

  },

  "llm_request": {

    "messages": [

      { "role": "system", "content": "You are a mathematician" },

      { "role": "user", "content": "What is 1+1?" }

    ],

    "stream": false

  },

  "llm_response_text": {

    "content": "1+1 equals 2."

  }

}
```
