Skip to main content
API7.ai Logo

ai-proxy

The ai-proxy plugin forwards LLM and embedding requests to OpenAI, DeepSeek, Anthropic, Gemini, Vertex AI, and other compatible providers. A common plugin configuration applies the provider-specific authentication, endpoint, and request format.

Before forwarding a request, the plugin identifies the client protocol from the request URI and body. Protocol conversion applies when the client uses Anthropic Messages but the selected provider supports OpenAI Chat Completions instead of native Anthropic Messages. The plugin converts the client request to OpenAI format and the backend response to Anthropic format. This conversion supports only a subset of Anthropic Messages. See Protocol Reference for detection rules, field mappings, release boundaries, and limitations.

The plugin can also record LLM request information in the access log, not the error log, including token usage, model, and time to first response. Logging plugins can consume these entries.

Examples​

The following examples configure ai-proxy for common provider and protocol scenarios.

Proxy to OpenAI​

The following example configures ai-proxy on a route that sends chat requests to OpenAI.

Obtain an OpenAI API key and optionally save it to an environment variable:

export OPENAI_API_KEY="<OPENAI_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options":{
          "model": "gpt-4.1-mini"
        }
      }
    }
  }
EOF

❶ Specify the provider to be openai.

❷ Attach OpenAI API key in the Authorization header.

❸ Specify the name of the model.

The ADC and Gateway API tabs include a conventional backend. When the plugin handles the request itself, the backend is not contacted. Verify the plugin-specific result below; a backend response alone does not confirm that the plugin ran.

Send a POST request to the route with a system prompt and a sample user question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

If the LLM upstream returns 429 or a 5xx response, the client receives the upstream status, response body, and Content-Type. This preserves any provider error details.

Proxy to DeepSeek​

The following example configures ai-proxy to send chat requests to DeepSeek.

Obtain the DeepSeek API key and optionally save it to an environment variable:

export DEEPSEEK_API_KEY="<DEEPSEEK_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "deepseek",
        "auth": {
          "header": {
            "Authorization": "Bearer $DEEPSEEK_API_KEY"
          }
        },
        "options": {
          "model": "deepseek-chat"
        }
      }
    }
  }
EOF

❶ Specify the provider to be deepseek, so that the plugin will proxy requests to https://api.deepseek.com/chat/completions.

❷ Attach DeepSeek API key in the Authorization header.

❸ Specify the name of the model.

Send a POST request to the route with a sample question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "system",
        "content": "You are an AI assistant that helps people find information."
      },
      {
        "role": "user",
        "content": "Write me a 50-word introduction for Apache APISIX."
      }
    ]
  }'

You should receive a response similar to the following:

{
  ...
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Apache APISIX is a dynamic, real-time, high-performance API gateway and cloud-native platform. It provides rich traffic management features like load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more. Designed for microservices and serverless architectures, APISIX ensures scalability, security, and seamless integration with modern DevOps workflows."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

Proxy to Azure OpenAI​

The following example sends Chat Completions requests to an Azure OpenAI deployment through its v1 API.

Obtain the Azure OpenAI API key and optionally save it to an environment variable:

export AZ_OPENAI_API_KEY="<AZURE_OPENAI_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai-compatible",
        "auth": {
          "header": {
            "api-key": "$AZ_OPENAI_API_KEY"
          }
        },
        "options":{
          "model": "<AZURE_OPENAI_DEPLOYMENT>"
        },
        "override": {
          "endpoint": "https://<AZURE_OPENAI_RESOURCE>.openai.azure.com/openai/v1/chat/completions"
        }
      }
    }
  }
EOF

❶ Set the provider to openai-compatible so the deployment name remains in the v1 request body.

❷ Attach Azure OpenAI API key in the api-key header.

❸ Specify the Azure OpenAI deployment name.

❹ Specify the Azure OpenAI v1 Chat Completions endpoint.

Send a POST request to the route with a sample question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "system",
        "content": "You are an AI assistant that helps people find information."
      },
      {
        "role": "user",
        "content": "Write me a 50-word introduction for Apache APISIX."
      }
    ],
    "max_tokens": 800,
    "temperature": 0.7,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "top_p": 0.95,
    "stop": null
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      ...,
      "message": {
        "content": "Apache APISIX is a modern, cloud-native API gateway built to handle high-performance and low-latency use cases. It offers a wide range of features, including load balancing, rate limiting, authentication, and dynamic routing, making it an ideal choice for microservices and cloud-native architectures.",
        "role": "assistant"
      }
    }
  ],
  ...
}

Proxy to Gemini​

The following example sends chat requests to the Gemini API. Gemini support was introduced in API7 Enterprise 3.9.2 and APISIX 3.15.0.

Obtain a Gemini API key and optionally save it to an environment variable:

export GEMINI_API_KEY="<GEMINI_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-gemini-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "gemini",
        "auth": {
          "header": {
            "Authorization": "Bearer $GEMINI_API_KEY"
          }
        },
        "options": {
          "model": "gemini-3.5-flash"
        }
      }
    }
  }
EOF

❶ Specify the provider to be gemini.

❷ Replace with your Gemini API key in the Authorization header.

❸ Specify the name of the Gemini model.

about model endpoint

The configuration above proxies requests to the chat completion endpoint at https://generativelanguage.googleapis.com/v1beta/openai/chat/completions. To proxy requests to an embeddings model, explicitly configure the embeddings model endpoint in the override field.

Send a POST request to the route with a system prompt and a sample user question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a helpful AI assistant" },
      { "role": "user", "content": "What is the capital of France?" }
    ]
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The capital of France is **Paris**.",
        "role": "assistant"
      }
    }
  ],
  "model": "gemini-3.5-flash",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 8,
    "prompt_tokens": 15,
    "total_tokens": 41
  },
  ...
}

Proxy to Vertex AI Chat Completion​

The following example sends chat requests to Vertex AI using a Google Cloud service account. Vertex AI support was introduced in API7 Enterprise 3.9.2 and APISIX 3.15.0.

Before proceeding:

Your credentials file should look similar to the following:

credentials.json
{
  "type": "service_account",
  "project_id": "<GCP_PROJECT_ID>",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "<SERVICE_ACCOUNT_EMAIL>",
  "client_id": "....",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<URL_ENCODED_SERVICE_ACCOUNT_EMAIL>",
  "universe_domain": "googleapis.com"
}

Save the raw JSON for the ADC example and a JSON-encoded string for the Admin API example:

export GCP_SA_JSON="$(cat credentials.json)"
export GCP_SA_JSON_STRING="$(jq -Rs . < credentials.json)"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-vertex-ai-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "vertex-ai",
        "auth": {
          "gcp": {
            "service_account_json": $GCP_SA_JSON_STRING
          }
        },
        "provider_conf": {
          "project_id": "<GCP_PROJECT_ID>",
          "region": "northamerica-northeast1"
        },
        "options": {
          "model": "google/gemini-3.5-flash"
        }
      }
    }
  }
EOF

❶ Specify the provider to be vertex-ai.

❷ Provide the service account credentials as a JSON string.

❸ Replace with your Vertex AI project ID and region.

❹ Specify the name of the Gemini model through Vertex AI in the <publisher>/<model> format.

Send a POST request to the route with a system prompt and a sample user question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "1 + 1 = 2\n"
      },
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "completion_tokens": 8,
    "extra_properties": {
      "google": {
        "traffic_type": "ON_DEMAND"
      }
    },
    "total_tokens": 19,
    "prompt_tokens": 11
  },
  "object": "chat.completion",
  "model": "google/gemini-3.5-flash",
  ...
}

Proxy to Vertex AI Embedding Models​

The following example sends embedding requests to Vertex AI using a Google Cloud service account. Vertex AI support was introduced in API7 Enterprise 3.9.2 and APISIX 3.15.0.

Before proceeding:

Your credentials file should look similar to the following:

credentials.json
{
  "type": "service_account",
  "project_id": "<GCP_PROJECT_ID>",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "<SERVICE_ACCOUNT_EMAIL>",
  "client_id": "....",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<URL_ENCODED_SERVICE_ACCOUNT_EMAIL>",
  "universe_domain": "googleapis.com"
}

Save the raw JSON for the ADC example and a JSON-encoded string for the Admin API example:

export GCP_SA_JSON="$(cat credentials.json)"
export GCP_SA_JSON_STRING="$(jq -Rs . < credentials.json)"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-vertex-ai-embeddings-route",
    "uri": "/embeddings",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "vertex-ai",
        "auth": {
          "gcp": {
            "service_account_json": $GCP_SA_JSON_STRING
          }
        },
        "provider_conf": {
          "project_id": "<GCP_PROJECT_ID>",
          "region": "us-central1"
        },
        "options": {
          "model": "gemini-embedding-001"
        }
      }
    }
  }
EOF

❶ Specify the provider to be vertex-ai.

❷ Provide the service account credentials as a JSON string.

❸ Replace with your Vertex AI project ID and region.

❹ Specify the name of the Vertex AI Gemini embedding model.

Send a POST request to the route with an input string:

curl "http://127.0.0.1:9080/embeddings" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "input": "hello world"
  }'

You should receive a response similar to the following:

{
  "model": "gemini-embedding-001",
  "usage": {
    "total_tokens": 2,
    "prompt_tokens": 2
  },
  "object": "list",
  "data": [
    {
      "index": 0,
      "object": "embedding",
      "embedding": [
        -0.0241838414222,
        0.0098769934847951,
        0.0074856607243419,
        -0.067302219569683,
        ...
      ]
    }
  ]
}

Proxy to OpenAI Embedding Models​

The following example configures ai-proxy to send embedding requests to OpenAI.

Obtain an OpenAI API key and optionally save it to an environment variable:

export OPENAI_API_KEY="<OPENAI_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-route",
    "uri": "/embeddings",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "text-embedding-3-small",
          "encoding_format": "float"
        }
      }
    }
  }
EOF

❶ Select the openai provider. When the request body contains input, the plugin detects an Embeddings request and sends it to https://api.openai.com/v1/embeddings.

❷ Attach OpenAI API key in the Authorization header.

❸ Specify the name of the embedding model.

❹ Add an additional parameter encoding_format to configure returned embedding vector to be a list of floating point numbers.

Send a POST request to the route with an input string:

curl "http://127.0.0.1:9080/embeddings" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "input": "hello world"
  }'

You should receive a response similar to the following:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [
        -0.0067144386,
        -0.039197803,
        0.034177095,
        0.028763203,
        -0.024785956,
        -0.04201061,
        ...
      ],
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {
    "prompt_tokens": 2,
    "total_tokens": 2
  }
}

Proxy to Anthropic​

The following example configures ai-proxy to send OpenAI-compatible Chat Completions requests to Claude through the Anthropic API.

Obtain an Anthropic API key and optionally save it to an environment variable:

export ANTHROPIC_API_KEY="<ANTHROPIC_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-anthropic-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "anthropic",
        "auth": {
          "header": {
            "x-api-key": "$ANTHROPIC_API_KEY"
          }
        },
        "options": {
          "model": "claude-sonnet-5-5"
        }
      }
    }
  }
EOF

❶ Specify the provider to be anthropic.

❷ Attach Anthropic API key in the x-api-key header.

❸ Specify the name of the Anthropic model.

Send a POST request to the route with a system prompt and a sample user question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "claude-sonnet-5-5",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 19,
    "completion_tokens": 11,
    "total_tokens": 30
  }
}

Convert Anthropic Requests to OpenAI-Compatible Backend​

The following example configures the ai-proxy plugin to accept Anthropic Messages requests, convert them to OpenAI Chat Completions, and forward them to an OpenAI-compatible backend. Use this configuration when client applications send Anthropic-formatted requests but the backend uses an OpenAI-compatible API.

The /v1/messages URI suffix identifies the request as Anthropic Messages. Because this example configures an OpenAI-compatible provider, the plugin converts requests to OpenAI Chat Completions and converts responses back to Anthropic format.

Obtain an API key for your chosen OpenAI-compatible backend service and save it to an environment variable. This example uses OpenAI:

export BACKEND_API_KEY="<BACKEND_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-anthropic-convert-route",
    "uri": "/v1/messages",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $BACKEND_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4.1-mini"
        }
      }
    }
  }
EOF

❶ Use a URI ending in /v1/messages to identify the request as Anthropic Messages.

❷ Configure a provider that supports OpenAI Chat Completions. The plugin converts the Anthropic request before forwarding it.

❸ Specify the model name of the backend provider.

Send a POST request to the route in Anthropic Messages API format:

curl "http://127.0.0.1:9080/v1/messages" -X POST \
  -H "Content-Type: application/json" \
  -H "x-api-key: ${BACKEND_API_KEY}" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "gpt-4.1-mini",
    "max_tokens": 1024,
    "messages": [
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

Although the request is sent in Anthropic format, it will be automatically converted to OpenAI format and forwarded to the backend. The response is converted back to Anthropic format:

{
  "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
  "type": "message",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "1+1 equals 2."
    }
  ],
  "model": "gpt-4.1-mini",
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 12,
    "output_tokens": 8
  }
}

This conversion covers common Anthropic Messages workflows: text messages, system prompts, streaming (SSE), and custom function tools. It converts the response back to Anthropic format.

The conversion supports a subset of the Anthropic Messages API, not an equivalent implementation. Some fields are approximated, and the plugin can drop unsupported fields without returning an error. See Protocol Reference for the field-level mapping, the release boundaries, and the known limitations.

If your client and backend both support the Anthropic Messages API natively, use Native Anthropic Messages API Pass-Through instead. That mode skips the conversion and preserves Anthropic-specific fields.

Native Anthropic Messages API Pass-Through​

Native Anthropic Messages pass-through was introduced in API7 Enterprise 3.9.8 and APISIX 3.17.0.

The following example passes Anthropic Messages requests directly to an Anthropic backend without protocol conversion. Use this approach when both the client and backend use the native Anthropic format.

The conversion example translates Anthropic client requests to OpenAI format and backend responses to Anthropic format. Native pass-through performs neither protocol transformation, so it preserves all Anthropic-specific fields including cache token usage (cache_creation_input_tokens, cache_read_input_tokens).

Obtain an Anthropic API key and save it to an environment variable:

export ANTHROPIC_API_KEY="<ANTHROPIC_API_KEY>"

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-anthropic-native",
    "uri": "/v1/messages",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "anthropic",
        "auth": {
          "header": {
            "x-api-key": "$ANTHROPIC_API_KEY"
          }
        },
        "options": {
          "model": "claude-sonnet-5-5"
        }
      }
    }
  }
EOF

Send a request using the Anthropic Messages API format:

curl -i "http://127.0.0.1:9080/v1/messages" -X POST \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-sonnet-5-5",
    "messages": [
      {"role": "user", "content": "Hello, Claude!"}
    ],
    "max_tokens": 1024
  }'

The request is forwarded directly to the Anthropic API without format conversion. The response is returned in native Anthropic format, preserving all provider-specific fields.

Proxy OpenAI Responses API​

OpenAI Responses API support was introduced in API7 Enterprise 3.9.8 and APISIX 3.17.0.

The following example configures ai-proxy for OpenAI Responses API requests (POST /v1/responses). The API supports built-in tools such as web search, file search, and computer use.

Obtain an OpenAI API key and save it to an environment variable:

export OPENAI_API_KEY="<OPENAI_API_KEY>"

Responses and Embeddings requests both contain input. Keep the /v1/responses URI suffix so that the plugin identifies this request as Responses instead of Embeddings. See Request Protocol Detection.

Create a route with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-responses-api",
    "uri": "/v1/responses",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4.1"
        }
      }
    }
  }
EOF

Send a request using the OpenAI Responses API format:

curl -i "http://127.0.0.1:9080/v1/responses" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1",
    "input": "Explain API gateways in one sentence."
  }'

Send a streaming request:

curl "http://127.0.0.1:9080/v1/responses" -X POST \
  --no-buffer \
  -H "Content-Type: application/json" \
  -d '{
    "stream": true,
    "input": "Count from one to five."
  }'

❶ Disable cURL output buffering so that it displays each server-sent event as it arrives.

❷ Request a streaming response from OpenAI.

The stream contains typed events such as response.output_text.delta and ends with a response.completed event.

See Failure Responses for empty and interrupted stream behavior.

Plugins that explicitly support the Responses request structure can be used on the same route. These include ai-prompt-decorator, ai-prompt-guard, ai-rag, and ai-aliyun-content-moderation.

Each plugin handles instructions and input differently. Check its plugin page before adding it to a Responses route; plugins designed for Chat Completions do not automatically support the Responses format.

Proxy to AWS Bedrock​

The bedrock provider signs and forwards requests to Amazon Bedrock's Converse API. It uses AWS Signature Version 4, so the plugin requires IAM credentials instead of an API key header. Bedrock Converse and ConverseStream are available in API7 Enterprise 3.9.12 and APISIX 3.17.0.

Obtain your AWS IAM credentials and save them to environment variables:

export AWS_ACCESS_KEY_ID="<AWS_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<AWS_SECRET_ACCESS_KEY>"

Create a route with the ai-proxy plugin configured for Bedrock:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-bedrock",
    "uri": "/bedrock/converse",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "bedrock",
        "auth": {
          "aws": {
            "access_key_id": "$AWS_ACCESS_KEY_ID",
            "secret_access_key": "$AWS_SECRET_ACCESS_KEY"
          }
        },
        "provider_conf": {
          "region": "us-east-1"
        },
        "options": {
          "model": "global.anthropic.claude-haiku-5-5"
        }
      }
    }
  }
EOF

The plugin uses the configured AWS credentials for SigV4 request signing.

Send a POST request to the route in Bedrock Converse format. The request URI must end with /converse:

curl "http://127.0.0.1:9080/bedrock/converse" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": [
          { "text": "Explain API gateways in one sentence." }
        ]
      }
    ],
    "inferenceConfig": {
      "maxTokens": 256
    }
  }'

The plugin forwards the request to the configured Bedrock model and applies SigV4 signing automatically.

To use streaming, add "stream": true to the request body:

curl "http://127.0.0.1:9080/bedrock/converse" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": [
          { "text": "Hello" }
        ]
      }
    ]
  }'

Streaming requests are routed to the /converse-stream endpoint and the binary AWS EventStream response is parsed and forwarded to the client.

Proxy to Selected Model using Request Body Parameter​

The following example routes requests on the same URI to OpenAI or DeepSeek based on the model field in the request body. The route uses a post_arg.* variable to read that field.

The example will use OpenAI and DeepSeek as the example LLM services. Obtain the OpenAI and DeepSeek API keys and save them to environment variables:

# replace with your API key
export OPENAI_API_KEY="<OPENAI_API_KEY>"
export DEEPSEEK_API_KEY="<DEEPSEEK_API_KEY>"

Create a route to the OpenAI API with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-openai-route",
    "uri": "/anything",
    "methods": ["POST"],
    "vars": [[ "post_arg.model", "==", "openai" ]],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4.1-mini"
        }
      }
    }
  }
EOF

❶ Set the route URI to be /anything.

❷ Match the route to requests where the body parameter model is set to openai.

Create another route /anything to the DeepSeek API with the ai-proxy plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-deepseek-route",
    "uri": "/anything",
    "methods": ["POST"],
    "vars": [[ "post_arg.model", "==", "deepseek" ]],
    "plugins": {
      "ai-proxy": {
        "provider": "deepseek",
        "auth": {
          "header": {
            "Authorization": "Bearer $DEEPSEEK_API_KEY"
          }
        },
        "options": {
          "model": "deepseek-chat"
        }
      }
    }
  }
EOF

❶ Set the route URI to be /anything, same as the previous route.

❷ Match the route to requests where the body parameter model is set to deepseek.

Send a POST request to the route with model set to openai:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai",
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

Send a POST request to the route with model set to deepseek:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek",
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  ...,
  "model": "deepseek-chat",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The sum of 1 and 1 is 2. This is a basic arithmetic operation where you combine two units to get a total of two units."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

You can also configure post_arg.* to fetch nested request body parameter. For instance, if the request format is:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": {
      "name": "openai"
    },
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You can configure the vars on the route to be [[ "post_arg.model.name", "==", "openai" ]].

For more information on the expressions, see APISIX Expressions.

Include LLM Information in Access Log​

The following example adds LLM request information to the gateway access log for analytics and auditing. In addition to NGINX variables, the following variables are available:

  • apisix_upstream_response_time: Time taken for the gateway to send the request to the upstream service and receive the full response, in milliseconds. Error responses returned by the LLM upstream are also recorded in milliseconds. Available from API7 Enterprise 3.8.8.
  • request_type: Type of request, where the value could be traditional_http, ai_chat, or ai_stream.
  • llm_time_to_first_token: Duration from request sending to the first token received from the LLM service, in milliseconds. For an upstream error response, this is the duration until the error response was received. It remains 0 if the gateway could not reach the upstream.
  • llm_model: LLM model name forwarded to the upstream LLM service.
  • request_llm_model: LLM model name specified in the request.
  • llm_prompt_tokens: Number of tokens in the prompt.
  • llm_completion_tokens: Number of completion tokens in the response.
  • llm_content_risk_level: Content risk level reported by an AI content moderation plugin. Introduced in APISIX 3.14.0.

The gateway's AI HTTP transport also populates the standard upstream variables upstream_addr, upstream_status, apisix_upstream_response_time, upstream_scheme, upstream_host, upstream_uri, and upstream_response_length.

These variables apply to streaming and non-streaming AI responses in API7 Enterprise 3.9.11 and later, and APISIX 3.17.0 and later. They are also populated for APISIX requests sent through the FFI client.

The following additional variables were introduced in API7 Enterprise 3.9.14 and APISIX 3.18.0:

  • llm_total_tokens: Total number of tokens used, including prompt and completion tokens.
  • llm_stream: Whether the request was a streaming request, either true or false.
  • llm_has_tool_calls: Whether the LLM response contains tool calls, either true or false.
  • llm_tool_count: Number of tools provided in the request.
  • llm_end_user_id: End-user identifier extracted from the request body, such as user, safety_identifier, or metadata.user_id.
  • llm_cache_read_input_tokens: Number of prompt tokens served from the provider's prompt cache.
  • llm_cache_creation_input_tokens: Number of prompt tokens written to the provider's prompt cache.
  • llm_reasoning_tokens: Number of reasoning tokens used by reasoning models.
tip

The access log example uses these variables, and logging plugins can use them as well. In API7 Enterprise 3.9.18 and 3.10.5, and in APISIX 3.18.0, enabling logging.summaries adds them to the llm_summary object without the llm_ prefix. Values that were not produced for a request are omitted.

To record these values in access logs, add the LLM variables to the gateway access log format:

Add or update this section in the gateway configuration file:

config.yaml
nginx_config:
  http:
    access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\" \"$llm_total_tokens\" \"$llm_stream\" \"$llm_has_tool_calls\" \"$llm_tool_count\" \"$llm_end_user_id\" \"$llm_cache_read_input_tokens\" \"$llm_cache_creation_input_tokens\" \"$llm_reasoning_tokens\""

Reload the gateway for configuration changes to take effect.

Create a route by following Proxy to OpenAI, then send a request:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

The plugin replaces the requested model with the configured gpt-4.1-mini model. You should receive a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null,
        "annotations": []
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "audio_tokens": 0
    },
    ...
  },
  "service_tier": "default",
  "system_fingerprint": null
}

In the gateway's access log, you should see a log entry similar to the following:

192.168.215.1 - - [<TIMESTAMP>] 127.0.0.1:9080 "POST /anything HTTP/1.1" 200 808 2.670 "-" "curl/8.6.0" - - 2670 "http://127.0.0.1:9080" "<REQUEST_ID>" "ai_chat" "2670" "gpt-4.1-mini" "gpt-4o-mini" "23" "8" "31" "false" "false" "0" "" "0" "0" "0"

The access log entry shows an upstream response time and time to first token of 2670 milliseconds. The request uses the ai_chat type, requests gpt-4o-mini, and is forwarded to gpt-4.1-mini. It uses 23 prompt tokens, 8 completion tokens, and 31 total tokens.

The remaining values show a non-streaming request with no tool calls, provided tools, end-user identifier, prompt-cache tokens, or reasoning tokens.

Send AI Request Logs to Kafka​

The following example sends AI request and response details, including model, token usage, and payloads, to Kafka. Complete the broker and apisix-logs topic setup in the kafka-logger guide first.

Create a route to the LLM service and enable Kafka logging:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-proxy-openai-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4.1-mini"
        },
        "logging": {
          "summaries": true,
          "payloads": true
        }
      },
      "kafka-logger": {
        "brokers": [
          {
            "host": "kafka-server",
            "port": 9092
          }
        ],
        "kafka_topic": "apisix-logs",
        "key": "key1",
        "batch_max_size": 1
      }
    }
  }
EOF

❶ Log request LLM model, duration, request and response tokens.

❷ Log request and response payload.

❸ Update with your Kafka address.

❹ Update with your Kafka topic.

❺ Update with your Kafka key.

❻ Set to 1 to send the log entry immediately.

Send a POST request to the route:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive a response similar to the following:

{
  ...,
  "model": "gpt-4.1-mini-2025-04-14",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

In the Kafka topic, you should also see a log entry corresponding to the request with the LLM summary and request and response payloads:

{
  "llm_summary": {
    "model": "gpt-4.1-mini",
    "prompt_tokens": 23,
    "completion_tokens": 8,
    "total_tokens": 31
  },
  "llm_request": {
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ],
    "stream": false
  },
  "llm_response_text": {
    "content": "1+1 equals 2."
  }
}