Skip to main content
API7.ai Logo

Proxy OpenAI Requests

OpenAI provides models for text generation, reasoning, multimodal input, and tool use.

Using APISIX centralizes provider authentication and model selection at the gateway, so client applications do not need OpenAI credentials.

Centralizing OpenAI traffic also gives operators one place to apply rate limits, logging, prompt controls, and other gateway policies across applications.

This guide shows how to use the ai-proxy plugin to proxy Chat Completions, Responses API, and Embeddings requests. Chat Completions remains supported, while OpenAI recommends the Responses API for new text-generation projects. APISIX attaches the configured API key and model to each request.

Prerequisite(s)​

  • Install Docker.
  • Install cURL to send requests for validation.
  • Follow the Getting Started Tutorial to start an APISIX instance in Docker or on Kubernetes.
  • Have an OpenAI account with API access to a model that supports both Chat Completions and the Responses API, and an embedding model.

Obtain an OpenAI API Key​

Create an OpenAI API key, then export the key and models:

export OPENAI_API_KEY="<your-api-key>"
export OPENAI_MODEL="<your-model-name>"
export OPENAI_EMBEDDING_MODEL="<your-embedding-model-name>"

Create Routes to OpenAI​

APISIX detects each OpenAI request protocol before selecting the corresponding upstream endpoint:

Client protocolDetectionOpenAI upstream path
Responses APIThe request body contains input and the request URI ends in /v1/responses./v1/responses
Chat CompletionsThe request body contains a messages array./v1/chat/completions
EmbeddingsThe request body contains input, and neither the Responses API nor Chat Completions rule matched./v1/embeddings

APISIX checks the rules in table order. Responses and Embeddings requests both use input, so a request containing input but not messages is identified as Embeddings unless its URI ends in /v1/responses. A Responses route URI can have a custom prefix, but it must keep that suffix.

Create one route for Chat Completions and Responses API requests, and another route with its own embedding model:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "id": "openai-apis",
    "uris": [
      "/openai/chat",
      "/openai/v1/responses"
    ],
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "$OPENAI_MODEL"
        }
      }
    }
  }
EOF
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "id": "openai-embeddings",
    "uri": "/openai/embeddings",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "$OPENAI_EMBEDDING_MODEL"
        }
      }
    }
  }
EOF

❶ Configure any client-facing path for Chat Completions. APISIX identifies the format from the messages field alone.

❷ Configure a client-facing path for Responses requests. APISIX identifies the format from the /v1/responses suffix together with the input field.

❸ Select the OpenAI provider. It sends each detected request format to its corresponding OpenAI upstream path.

❹ Add the configured text-generation model to each request on this route. A client-supplied model does not override this value.

❺ Configure any client-facing path for Embeddings. APISIX identifies the format from input after the Responses API and Chat Completions rules do not match.

❻ Add the configured embedding model to each request on the Embeddings route.

The route configuration contains the OpenAI API key. When data encryption with a keyring is enabled, APISIX encrypts the key before saving the route to etcd. Configure a custom keyring in production.

Verify​

Send requests in all three supported API formats to verify protocol detection and upstream routing.

Send a Chat Completions Request​

Send a request with a list of messages:

curl -i "http://127.0.0.1:9080/openai/chat" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "system",
        "content": "You explain technical concepts concisely."
      },
      {
        "role": "user",
        "content": "Explain what an API gateway does in one sentence."
      }
    ]
  }'

You should receive an HTTP 200 response. Chat Completions returns generated text in choices[].message.content:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "An API gateway routes, secures, and manages requests between clients and backend services."
      }
    }
  ]
}

Send a Responses API Request​

Send a request with separate instructions and input:

curl -i "http://127.0.0.1:9080/openai/v1/responses" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "instructions": "You explain technical concepts concisely.",
    "input": "Explain what an API gateway does in one sentence."
  }'

You should receive an HTTP 200 response. The Responses API returns typed items in output:

{
  "status": "completed",
  "output": [
    {
      "type": "message",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "An API gateway routes, secures, and manages requests between clients and backend services."
        }
      ]
    }
  ]
}

Stream a Responses API Request​

Send a streaming request:

curl "http://127.0.0.1:9080/openai/v1/responses" -X POST \
  --no-buffer \
  -H "Content-Type: application/json" \
  -d '{
    "stream": true,
    "input": "Count from one to five."
  }'

❶ Disable cURL output buffering so that it displays each event as it arrives.

❷ Request a streaming response from OpenAI.

The stream uses typed server-sent events, including response.output_text.delta events for generated text and a final response.completed event.

info

The proxy-buffering plugin is not required for ai-proxy streaming. It controls the standard NGINX upstream path, which ai-proxy does not use for provider requests. To change how often ai-proxy flushes streaming output, configure streaming_flush_interval_ms. Set it to 0 to flush each upstream chunk synchronously.

Send an Embeddings Request​

Send an input string to the Embeddings route:

curl -i "http://127.0.0.1:9080/openai/embeddings" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "input": "APISIX is an API gateway."
  }'

You should receive an HTTP 200 response containing a vector in data[].embedding:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [
        -0.0067,
        -0.0392
      ]
    }
  ]
}

Clean Up​

Delete the APISIX routes when you no longer need them:

curl "http://127.0.0.1:9180/apisix/admin/routes/openai-apis" -X DELETE \
  -H "X-API-KEY: ${ADMIN_API_KEY}"

curl "http://127.0.0.1:9180/apisix/admin/routes/openai-embeddings" -X DELETE \
  -H "X-API-KEY: ${ADMIN_API_KEY}"

Remove the OpenAI values from the shell:

unset OPENAI_API_KEY OPENAI_MODEL OPENAI_EMBEDDING_MODEL

If you used ADC, remove the API key from adc.yaml or protect the file according to your organization's secret-handling policy.

Next Steps​

You have now configured APISIX to proxy Chat Completions, Responses API, and Embeddings requests to OpenAI.

See OpenAI's migration guide to compare the two API formats and plan application migrations.