Skip to main content
API7.ai Logo

ai-proxy-multi

The ai-proxy-multi plugin distributes LLM and embedding requests across OpenAI, DeepSeek, Gemini, Vertex AI, and other compatible provider instances. It adds load balancing, retries, fallbacks, and health checks.

Before forwarding a request, the plugin identifies the client protocol from the request URI and body. Protocol conversion applies when the client uses Anthropic Messages but the selected instance's provider supports OpenAI Chat Completions instead of native Anthropic Messages. The plugin converts the client request to OpenAI format and the backend response to Anthropic format. This conversion supports only a subset of Anthropic Messages. See Protocol Reference for detection rules, field mappings, release boundaries, and limitations.

The plugin can also record LLM request information in the access log, not the error log, including token usage, model, and time to first response. Logging plugins can consume these entries.

Demo​

The demo shows instance priority and rate limiting in the API7 Enterprise Dashboard. It assigns different priorities to two models and rate limits the higher-priority instance. When that instance exhausts its quota and fallback_strategy is ["rate_limiting"], requests continue to the lower-priority instance.

Examples​

The following examples configure ai-proxy-multi for common routing and resilience scenarios.

Give every instance a unique name, even when priorities differ. If you rename an instance, update its AI rate-limiting entry and semantic fallback reference too.

Load Balance between Instances​

The following example distributes traffic between two models at an 80:20 ratio. If the first attempt returns 429 or 5xx within two seconds, the plugin retries one other instance. The max_retries and retry_on_failure_within_ms fields were introduced in API7 Enterprise 3.9.14 and APISIX 3.17.0.

The example uses OpenAI and DeepSeek instances so that the selected upstream is easy to distinguish.

Create a route and replace the provider-specific values as needed:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["http_429", "http_5xx"],
        "max_retries": 1,
        "retry_on_failure_within_ms": 2000,
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ]
      }
    }
  }
EOF

❶ Configure the weight for openai-instance to be 8.

❷ Configure the weight for deepseek-instance to be 2.

The ADC and Gateway API tabs include a conventional backend. When the plugin handles the request itself, the backend is not contacted. Verify the plugin-specific result below; a backend response alone does not confirm that the plugin ran.

Send 10 POST requests with a system prompt and sample user question. Count how many are forwarded to OpenAI and DeepSeek:

openai_count=0
deepseek_count=0

for i in {1..10}; do 
  model=$(curl -s "http://127.0.0.1:9080/anything" -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "messages": [
        { "role": "system", "content": "You are a mathematician" },
        { "role": "user", "content": "What is 1+1?" }
      ]
    }' | jq -r '.model')

  if [[ "$model" == *"gpt-4"* ]]; then
    ((openai_count++))
  elif [[ "$model" == "deepseek-chat" ]]; then
    ((deepseek_count++))
  fi
done

echo "OpenAI responses: $openai_count"
echo "DeepSeek responses: $deepseek_count"

You should see a response similar to the following:

OpenAI responses: 8
DeepSeek responses: 2

Load Balance Responses API Requests​

The following example configures ai-proxy-multi on the OpenAI Responses API route and distributes requests across two OpenAI models.

Responses and Embeddings requests both contain input. Keep the /v1/responses URI suffix so that the plugin identifies this request as Responses instead of Embeddings. See Request Protocol Detection.

Create a route with uri set to /v1/responses:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-responses-route",
    "uri": "/v1/responses",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-responses-primary",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1"
            }
          },
          {
            "name": "openai-responses-secondary",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          }
        ]
      }
    }
  }
EOF

Send a request using the OpenAI Responses API format:

curl "http://127.0.0.1:9080/v1/responses" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Write one sentence about API gateways."
  }'

The request is forwarded to one of the configured OpenAI instances and the response is returned in the Responses API format.

Route by Semantic Similarity​

The semantic balancer selects an instance by comparing the request prompt with example utterances assigned to each instance. It is available in API7 Enterprise 3.9.18 and 3.10.5, and in APISIX 3.18.0.

Export API keys for the embedding request and LLM requests:

export EMBEDDING_API_KEY="<YOUR_EMBEDDING_API_KEY>"
export LLM_API_KEY="<YOUR_LLM_API_KEY>"

Create a route with instances for programming, translation, and general prompts. The general instance is also the fallback when no score reaches the configured threshold or the embedding request fails:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-semantic-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "balancer": {
          "algorithm": "semantic"
        },
        "semantic_opts": {
          "embeddings": {
            "provider": "openai",
            "model": "text-embedding-3-small",
            "auth": {
              "header": {
                "Authorization": "Bearer $EMBEDDING_API_KEY"
              }
            }
          },
          "threshold": 0.5,
          "fallback": "default",
          "debugging": true
        },
        "instances": [
          {
            "name": "code",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $LLM_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o"
            },
            "examples": [
              "write a Python function",
              "debug this code"
            ]
          },
          {
            "name": "translate",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $LLM_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            },
            "examples": [
              "translate this sentence into English"
            ]
          },
          {
            "name": "default",
            "provider": "openai",
            "weight": 1,
            "auth": {
              "header": {
                "Authorization": "Bearer $LLM_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4o-mini"
            },
            "examples": [
              "what is the weather today",
              "tell me a joke"
            ]
          }
        ]
      }
    }
  }
EOF

Send a programming request:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Help me debug this Python code"
      }
    ]
  }'

You should receive HTTP/1.1 200 OK. The response headers show the selected instance and the scores used to make the decision:

X-AI-Semantic-Picked-Instance: code
X-AI-Semantic-Scores: code:0.8213,default:0.2451,translate:0.1904

The scores depend on the embedding model and example utterances. Use them to choose a threshold that separates prompts that should be routed by similarity from prompts that should use the fallback. The fallback instance still requires examples because it participates in normal semantic ranking.

Configured health-check settings are not used while semantic selection is active, and selection does not retry after an upstream failure. Therefore, fallback_strategy, fallback_http_statuses, and max_retries do not affect semantic selection. An upstream failure from the selected instance is returned to the client. The semantic fallback is used only when no instance reaches its threshold or the embedding request fails.

The debugging headers expose instance names and scores. Disable semantic_opts.debugging after tuning and use the warning log emitted for fallback decisions in production.

Load Balance between Gemini and Vertex AI​

The following example load balances OpenAI-compatible Chat Completions requests across Gemini in Google AI Studio and Vertex AI. The instance weights send approximately 70% of requests to Google AI Studio and 30% to Vertex AI. This capability was introduced in API7 Enterprise 3.9.2 and APISIX 3.15.0.

Before proceeding:

  1. For Google AI Studio Gemini, obtain a Gemini API key.
  2. For Vertex AI Gemini, follow the Vertex AI project setup instructions to enable billing and the Vertex AI API. Create a service account, assign it the "Vertex AI User" role, and create a service account key in JSON format.

Save the API key, the raw service account JSON for the ADC example, and a JSON-encoded string for the Admin API example:

export GEMINI_API_KEY="<GEMINI_API_KEY>"
export GCP_SA_JSON="$(cat credentials.json)"
export GCP_SA_JSON_STRING="$(jq -Rs . < credentials.json)"

Create a route and replace the project-specific values:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-google-ai-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "gemini-instance",
            "provider": "gemini",
            "weight": 7,
            "auth": {
              "header": {
                "Authorization": "Bearer $GEMINI_API_KEY"
              }
            },
            "options": {
              "model": "gemini-3.5-flash"
            }
          },
          {
            "name": "vertex-ai-instance",
            "provider": "vertex-ai",
            "weight": 3,
            "auth": {
              "gcp": {
                "service_account_json": $GCP_SA_JSON_STRING
              }
            },
            "provider_conf": {
              "project_id": "<GCP_PROJECT_ID>",
              "region": "northamerica-northeast1"
            },
            "options": {
              "model": "google/gemini-3.5-flash"
            }
          }
        ]
      }
    }
  }
EOF

❶ Configure the provider to be gemini for Google AI Studio Gemini access.

❷ Replace with your Gemini API key in the Authorization header.

❸ Specify the name of the Gemini model through Google AI Studio in the <model> format.

❹ Configure the provider to be vertex-ai for Vertex AI Gemini access.

❺ Provide the service account credentials as a JSON string.

❻ Replace with your Vertex AI project ID and region.

❼ Specify the name of the Gemini model through Vertex AI in the <publisher>/<model> format.

Send 10 POST requests to the route to see the load balancing distribution:

studio_count=0
vertex_count=0

for i in {1..10}; do 
  model=$(curl -s "http://127.0.0.1:9080/anything" -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "messages": [
        { "role": "system", "content": "You are a mathematician" },
        { "role": "user", "content": "What is 1+1?" }
      ]
    }' | jq -r '.model')

  if [[ "$model" == "gemini-3.5-flash" ]]; then
    ((studio_count++))
  elif [[ "$model" == "google/gemini-3.5-flash" ]]; then
    ((vertex_count++))
  fi
done

echo "Google AI Studio Gemini responses: $studio_count"
echo "Vertex AI Gemini responses: $vertex_count"

You should see a response similar to the following:

Google AI Studio Gemini responses: 7
Vertex AI Gemini responses: 3

Configure Instance Priority and Rate Limiting​

The following example assigns different priorities to two models and rate limits the higher-priority instance. With fallback_strategy set to ["rate_limiting"], requests move to the lower-priority instance after the higher-priority instance exhausts its quota.

Create a route and replace the provider-specific values as needed:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "priority": 1,
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "priority": 0,
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ]
      },
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "openai-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "limit_strategy": "total_tokens"
      }
    }
  }
EOF

❶ Set the fallback_strategy to ["rate_limiting"].

❷ Set a higher priority for openai-instance.

❸ Set a lower priority for deepseek-instance.

❹ Configure a quota of 10 tokens.

❺ Configure the time window to be 60 seconds.

❻ Configure the limit_strategy to total_tokens.

Send a POST request to the route with a system prompt and a sample user question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

Since the total_tokens value exceeds the configured quota of 10, the next request within the 60-second window is expected to be forwarded to the other instance.

Within the same 60-second window, send another POST request to the route:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newton law" }
    ]
  }'

You should see a response similar to the following:

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"
      },
      ...
    }
  ],
  ...
}

Load Balance and Rate Limit by Consumers​

The following example load balances across two models and applies separate rate limits to each consumer.

Create the ai-multi-johndoe consumer with a 10-token quota per 60 seconds for openai-instance:

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "username": "ai-multi-johndoe",
    "plugins": {
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "openai-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "rejected_code": 429,
        "policy": "local",
        "limit_strategy": "total_tokens"
      }
    }
  }'

Add a key-auth credential for ai-multi-johndoe:

curl "http://127.0.0.1:9180/apisix/admin/consumers/ai-multi-johndoe/credentials" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "cred-john-key-auth",
    "plugins": {
      "key-auth": {
        "key": "john-key"
      }
    }
  }'

Create the ai-multi-janedoe consumer with a 10-token quota per 60 seconds for deepseek-instance:

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "username": "ai-multi-janedoe",
    "plugins": {
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "deepseek-instance",
            "limit": 10,
            "time_window": 60
          }
        ],
        "rejected_code": 429,
        "policy": "local",
        "limit_strategy": "total_tokens"
      }
    }
  }'

Add a key-auth credential for ai-multi-janedoe:

curl "http://127.0.0.1:9180/apisix/admin/consumers/ai-multi-janedoe/credentials" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "cred-jane-key-auth",
    "plugins": {
      "key-auth": {
        "key": "jane-key"
      }
    }
  }'

Create a route and replace the provider-specific values as needed:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "key-auth": {},
      "ai-proxy-multi": {
        "fallback_strategy": ["rate_limiting"],
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ]
      }
    }
  }
EOF

❶ Enable key-auth on the route.

❷ Configure an openai instance.

❸ Configure a deepseek instance.

Send a POST request to the route without any consumer key:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive an HTTP/1.1 401 Unauthorized response.

Send a POST request to the route with ai-multi-johndoe's key:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: john-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

Since the total_tokens value exceeds the configured quota of the openai instance for ai-multi-johndoe, the next request within the 60-second window from ai-multi-johndoe is expected to be forwarded to the deepseek instance.

Within the same 60-second window, send another POST request to the route with ai-multi-johndoe's key:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: john-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws to me" }
    ]
  }'

You should see a response similar to the following:

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Certainly! Newton's laws of motion are three fundamental principles that describe the relationship between the motion of an object and the forces acting on it. They were formulated by Sir Isaac Newton in the late 17th century and are foundational to classical mechanics.\n\n---\n\n### **1. Newton's First Law (Law of Inertia):**\n- **Statement:** An object at rest will remain at rest, and an object in motion will continue moving at a constant velocity (in a straight line at a constant speed), unless acted upon by an external force.\n- **Key Idea:** This law introduces the concept of **inertia**, which is the tendency of an object to resist changes in its state of motion.\n- **Example:** If you slide a book across a table, it eventually stops because of the force of friction acting on it. Without friction, the book would keep moving indefinitely.\n\n---\n\n### **2. Newton's Second Law (Law of Acceleration):**\n- **Statement:** The acceleration of an object is directly proportional to the net force acting on it and inversely proportional to its mass. Mathematically, this is expressed as:\n  \\[\n  F = ma\n  \\]\n  where:\n  - \\( F \\) = net force applied (in Newtons),\n  -"
      },
      ...
    }
  ],
  ...
}

Send a POST request to the route with ai-multi-janedoe's key:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: jane-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The sum of 1 and 1 is 2. This is a basic arithmetic operation where you combine two units to get a total of two units."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 31,
    "total_tokens": 45,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "prompt_cache_hit_tokens": 0,
    "prompt_cache_miss_tokens": 14
  },
  "system_fingerprint": "fp_3a5770e1b4_prod0225"
}

Since the total_tokens value exceeds the configured quota of the deepseek instance for ai-multi-janedoe, the next request within the 60-second window from ai-multi-janedoe is expected to be forwarded to the openai instance.

Within the same 60-second window, send another POST request to the route with ai-multi-janedoe's key:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H 'apikey: jane-key' \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons laws to me" }
    ]
  }'

You should see a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Sure, here are Newton's three laws of motion:\n\n1) Newton's First Law, also known as the Law of Inertia, states that an object at rest will stay at rest, and an object in motion will stay in motion, unless acted on by an external force. In simple words, this law suggests that an object will keep doing whatever it is doing until something causes it to do otherwise. \n\n2) Newton's Second Law states that the force acting on an object is equal to the mass of that object times its acceleration (F=ma). This means that force is directly proportional to mass and acceleration. The heavier the object and the faster it accelerates, the greater the force.\n\n3) Newton's Third Law, also known as the law of action and reaction, states that for every action, there is an equal and opposite reaction. Essentially, any force exerted onto a body will create a force of equal magnitude but in the opposite direction on the object that exerted the first force.\n\nRemember, these laws become less accurate when considering speeds near the speed of light (where Einstein's theory of relativity becomes more appropriate) or objects very small or very large. However, for everyday situations, they provide a good model of how things move.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

This shows ai-proxy-multi load balance the traffic with respect to the rate limiting rules in ai-rate-limiting by consumers.

Restrict Maximum Number of Completion Tokens​

The following example limits the completion tokens that an instance can generate.

The example uses OpenAI and DeepSeek instances so that the selected upstream is easy to distinguish.

Create a route and replace the provider-specific values as needed:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini",
              "max_tokens": 50
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat",
              "max_tokens": 100
            }
          }
        ]
      }
    }
  }
EOF

❶ Configure the max_tokens for the OpenAI instance to be 50.

❷ Configure the max_tokens for the DeepSeek instance to be 100.

Send a POST request to the route with a system prompt and a sample user question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Explain Newtons law" }
    ]
  }'

If the request is proxied to OpenAI, you should see a response similar to the following, where the content is truncated per 50 max_tokens threshold:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Newton's Laws of Motion are three physical laws that form the bedrock for classical mechanics. They describe the relationship between a body and the forces acting upon it, and the body's motion in response to those forces. \n\n1. Newton's First Law",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 50,
    "total_tokens": 70,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  },
  "service_tier": "default",
  "system_fingerprint": null
}

If the request is proxied to DeepSeek, you should see a response similar to the following, where the content is truncated per 100 max_tokens threshold:

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Newton's Laws of Motion are three fundamental principles that form the foundation of classical mechanics. They describe the relationship between a body and the forces acting upon it, and the body's motion in response to those forces. Here's a brief explanation of each law:\n\n1. **Newton's First Law (Law of Inertia):**\n   - **Statement:** An object will remain at rest or in uniform motion in a straight line unless acted upon by an external force.\n   - **Explanation:** This law"
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 100,
    "total_tokens": 110,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "prompt_cache_hit_tokens": 0,
    "prompt_cache_miss_tokens": 10
  },
  "system_fingerprint": "fp_3a5770e1b4_prod0225"
}

Proxy to Embedding Models​

The following example distributes Embeddings requests between OpenAI and Azure OpenAI. The Azure instance uses the openai-compatible provider so the deployment name remains in the request body required by Azure OpenAI's v1 API. Both Responses and Embeddings requests contain input; because this route does not end in /v1/responses, the plugin identifies the request as Embeddings.

Create a route and replace the provider-specific values as needed:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "text-embedding-3-small"
            }
          },
          {
            "name": "az-openai-instance",
            "provider": "openai-compatible",
            "weight": 0,
            "auth": {
              "header": {
                "api-key": "$AZ_OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "<AZURE_OPENAI_EMBEDDING_DEPLOYMENT>"
            },
            "override": {
              "endpoint": "https://<AZURE_OPENAI_RESOURCE>.openai.azure.com/openai/v1/embeddings"
            }
          }
        ]
      }
    }
  }
EOF

❶ Specify the OpenAI embedding model. The openai provider sends the detected Embeddings request to https://api.openai.com/v1/embeddings.

❷ Specify the Azure OpenAI embedding deployment name.

❸ Specify the Azure OpenAI v1 Embeddings endpoint.

Send a POST request to the route with an input string:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "input": "hello world"
  }'

You should receive a response similar to the following:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [
        -0.0067144386,
        -0.039197803,
        0.034177095,
        0.028763203,
        -0.024785956,
        -0.04201061,
        ...
      ],
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}

Enable Active Health Checks​

The following example load balances requests across models and enables active health checks. Health checks can be enabled on one or multiple instances.

Create a route and replace the provider-specific values and health-check settings:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "llm-instance-1",
            "provider": "openai-compatible",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $YOUR_LLM_API_KEY"
              }
            },
            "options": {
              "model": "$YOUR_LLM_MODEL"
            }
          },
          {
            "name": "llm-instance-2",
            "provider": "openai-compatible",
            "weight": 0,
            "auth": {
              "header": {
                "Authorization": "Bearer $YOUR_LLM_API_KEY"
              }
            },
            "options": {
              "model": "$YOUR_LLM_MODEL"
            },
            "checks": {
              "active": {
                "type": "https",
                "host": "yourhost.com",
                "http_path": "/your/probe/path",
                "healthy": {
                  "interval": 2,
                  "successes": 1
                },
                "unhealthy": {
                  "interval": 1,
                  "http_failures": 3
                }
              }
            }
          }
        ]
      }
    }
  }
EOF

❶ Update the type of active health checks.

❷ Update the host, if available.

❸ Update the probing path.

❹ Configure the time interval in seconds for periodically checking healthy nodes.

❺ Configure the success count threshold for ruling if an upstream node is healthy.

❻ Configure the time interval in seconds for periodically checking unhealthy nodes.

❼ Configure the timeout count threshold for ruling if an upstream node is unhealthy.

For verification, the behaviours should be consistent with the verification in Active Health Checks.

From API7 Enterprise 3.9.20 and 3.10.7, the health status of each instance is also reported by the data plane Control API. GET /v1/healthcheck lists one entry per instance that declares checks, where plugin is ai-proxy-multi and meta.instance names the instance. GET /v1/healthcheck/{src_type}/{src_id}/checkers returns every checker that route or service owns.

Include LLM Information in Access Log​

The following example adds LLM request information to the gateway access log for analytics and auditing. In addition to NGINX variables, the following variables are available:

  • apisix_upstream_response_time: Time taken for the gateway to send the request to the upstream service and receive the full response, in milliseconds. Error responses returned by the LLM upstream are also recorded in milliseconds. Available from API7 Enterprise 3.8.8.
  • request_type: Type of request, where the value could be traditional_http, ai_chat, or ai_stream.
  • llm_time_to_first_token: Duration from request sending to the first token received from the LLM service, in milliseconds. For an upstream error response, this is the duration until the error response was received. It remains 0 if the gateway could not reach the upstream.
  • llm_model: LLM model name forwarded to the upstream LLM service.
  • request_llm_model: LLM model name specified in the request.
  • llm_prompt_tokens: Number of tokens in the prompt.
  • llm_completion_tokens: Number of completion tokens in the response.
  • llm_content_risk_level: Content risk level reported by an AI content moderation plugin. Introduced in APISIX 3.14.0.

The gateway's AI HTTP transport also populates the standard upstream variables upstream_addr, upstream_status, apisix_upstream_response_time, upstream_scheme, upstream_host, upstream_uri, and upstream_response_length.

These variables apply to streaming and non-streaming AI responses in API7 Enterprise 3.9.11 and later, and APISIX 3.17.0 and later. They are also populated for APISIX requests sent through the FFI client.

The following additional variables were introduced in API7 Enterprise 3.9.14 and APISIX 3.18.0:

  • llm_total_tokens: Total number of tokens used, including prompt and completion tokens.
  • llm_stream: Whether the request was a streaming request, either true or false.
  • llm_has_tool_calls: Whether the LLM response contains tool calls, either true or false.
  • llm_tool_count: Number of tools provided in the request.
  • llm_end_user_id: End-user identifier extracted from the request body, such as user, safety_identifier, or metadata.user_id.
  • llm_cache_read_input_tokens: Number of prompt tokens served from the provider's prompt cache.
  • llm_cache_creation_input_tokens: Number of prompt tokens written to the provider's prompt cache.
  • llm_reasoning_tokens: Number of reasoning tokens used by reasoning models.
tip

The access log example uses these variables, and logging plugins can use them as well. In API7 Enterprise 3.9.18 and 3.10.5, and in APISIX 3.18.0, enabling logging.summaries adds them to the llm_summary object without the llm_ prefix. Values that were not produced for a request are omitted.

To record these values in access logs, add the LLM variables to the gateway access log format:

Add or update this section in the gateway configuration file:

config.yaml
nginx_config:
  http:
    access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\" \"$llm_total_tokens\" \"$llm_stream\" \"$llm_has_tool_calls\" \"$llm_tool_count\" \"$llm_end_user_id\" \"$llm_cache_read_input_tokens\" \"$llm_cache_creation_input_tokens\" \"$llm_reasoning_tokens\""

Reload the gateway for configuration changes to take effect.

Create a route by following Load Balance between Instances, then send a request:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

When ai-proxy-multi selects the OpenAI instance, the plugin replaces the requested model with gpt-4.1-mini. You should receive a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null,
        "annotations": []
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    ...
  },
  "service_tier": "default",
  "system_fingerprint": null
}

In the gateway's access log, you should see a log entry similar to the following:

192.168.215.1 - - [<TIMESTAMP>] 127.0.0.1:9080 "POST /anything HTTP/1.1" 200 808 2.670 "-" "curl/8.6.0" - - 2670 "http://127.0.0.1:9080" "<REQUEST_ID>" "ai_chat" "2670" "gpt-4.1-mini" "gpt-4o-mini" "23" "8" "31" "false" "false" "0" "" "0" "0" "0"

The access log entry shows an upstream response time and time to first token of 2670 milliseconds. The request uses the ai_chat type, requests gpt-4o-mini, and is forwarded to gpt-4.1-mini. It uses 23 prompt tokens, 8 completion tokens, and 31 total tokens.

The remaining values show a non-streaming request with no tool calls, provided tools, end-user identifier, prompt-cache tokens, or reasoning tokens.

Send AI Request Logs to Kafka​

The following example sends AI request and response details, including model, token usage, and payloads, to Kafka. Complete the broker and apisix-logs topic setup in the kafka-logger guide first.

Create a route to the LLM services and enable Kafka logging:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer $OPENAI_API_KEY"
              }
            },
            "options": {
              "model": "gpt-4.1-mini"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer $DEEPSEEK_API_KEY"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ],
        "logging": {
          "summaries": true,
          "payloads": true
        }
      },
      "kafka-logger": {
        "brokers": [
          {
            "host": "kafka-server",
            "port": 9092
          }
        ],
        "kafka_topic": "apisix-logs",
        "key": "key1",
        "batch_max_size": 1
      }
    }
  }
EOF

❶ Log request LLM model, duration, request and response tokens.

❷ Log request and response payload.

❸ Update with your Kafka address.

❹ Update with your Kafka topic.

❺ Update with your Kafka key.

❻ Set to 1 to send the log entry immediately.

Send a POST request to the route:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following if the request is forwarded to OpenAI:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

In the Kafka topic, you should also see a log entry corresponding to the request with the LLM summary and request and response payloads. A request forwarded to OpenAI produces fields similar to the following:

{
  "llm_summary": {
    "model": "gpt-4.1-mini",
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31
  },
  "llm_request": {
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ],
    "stream": false
  },
  "llm_response_text": {
    "content": "1+1 equals 2."
  }
}