a6-plugin-ai-proxy

Overview

The ai-proxy plugin turns APISIX into an AI gateway. Clients can send requests in supported protocols to APISIX instead of handling provider authentication and endpoint selection themselves. The plugin detects the client protocol, selects a compatible provider endpoint, forwards the native format or converts it when an adapter is available, and handles response streaming.

When to Use

Proxy Chat Completions, Responses API, Embeddings, Anthropic Messages, or Bedrock Converse requests to a compatible provider
Centralize API keys at the gateway instead of distributing to clients
Add observability (token counts, latency) to LLM calls
Combine with ai-prompt-template, ai-prompt-decorator, or content moderation plugins for a full AI gateway pipeline

Supported Providers

Provider	Value	Endpoint Behavior
OpenAI	`openai`	Automatically selects `/v1/chat/completions`, `/v1/responses`, or `/v1/embeddings` on `https://api.openai.com`
DeepSeek	`deepseek`	`https://api.deepseek.com/chat/completions`
Azure OpenAI	`azure-openai`	Custom via `override.endpoint`
Anthropic	`anthropic`	Automatically selects `/v1/chat/completions` or `/v1/messages` on `https://api.anthropic.com`
AIMLAPI	`aimlapi`	`https://api.aimlapi.com/v1/chat/completions`
OpenRouter	`openrouter`	`https://openrouter.ai/api/v1/chat/completions`
Gemini	`gemini`	`https://generativelanguage.googleapis.com/v1beta/openai/chat/completions`
Vertex AI	`vertex-ai`	`https://aiplatform.googleapis.com`
Amazon Bedrock	`bedrock`	Region- and model-specific Bedrock Runtime endpoint
OpenAI-Compatible	`openai-compatible`	Custom via `override.endpoint`

Plugin Configuration Reference

Field	Type	Required	Default	Description
`provider`	string	Yes	—	One of the 10 supported providers
`auth`	object	Yes	—	Authentication config (see below)
`options`	object	No	—	Model and generation parameters
`options.model`	string	No	—	Model name (provider-specific)
`options.temperature`	number	No	—	Sampling temperature
`options.top_p`	number	No	—	Nucleus sampling
`options.max_tokens`	integer	No	—	Maximum tokens to generate
`options.stream`	boolean	No	—	Override the outgoing `stream` field; for Bedrock Converse, set `stream: true` in the client request instead
`override`	object	No	—	Provider endpoint and request-body override settings
`override.endpoint`	string	No	—	Provider scheme and host, or a full URL including the path and query
`provider_conf`	object	No	—	Provider-specific config for Vertex AI or Amazon Bedrock
`provider_conf.project_id`	string	No	—	GCP project ID for Vertex AI; required with `region` unless `override.endpoint` is configured
`provider_conf.region`	string	No	—	GCP region for Vertex AI; required AWS region for Amazon Bedrock
`logging`	object	No	—	Logging options
`logging.summaries`	boolean	No	`false`	Log model, duration, tokens
`logging.payloads`	boolean	No	`false`	Log request/response bodies
`timeout`	integer	No	`30000`	Request timeout (ms)
`keepalive`	boolean	No	`true`	Keep connection alive
`keepalive_timeout`	integer	No	`60000`	Connection reuse timeout (ms)
`keepalive_pool`	integer	No	`30`	Connection reuse pool size
`ssl_verify`	boolean	No	`true`	Verify SSL certificate

Authentication by Provider

OpenAI / DeepSeek / AIMLAPI / OpenRouter

{
  "auth": {
    "header": {
      "Authorization": "Bearer sk-your-api-key"
    }
  }
}

Anthropic

{
  "auth": {
    "header": {
      "x-api-key": "your-anthropic-api-key"
    }
  }
}

Native Anthropic Messages requests also require an anthropic-version header. Configure it in auth.header or require clients to send it.

Azure OpenAI

{
  "auth": {
    "header": {
      "api-key": "your-azure-key"
    }
  },
  "override": {
    "endpoint": "https://YOUR-RESOURCE.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview"
  }
}

Gemini

{
  "auth": {
    "header": {
      "Authorization": "Bearer your-gemini-key"
    }
  }
}

Vertex AI (GCP Service Account)

{
  "auth": {
    "gcp": {
      "service_account_json": "{ ... }",
      "max_ttl": 3600,
      "expire_early_secs": 60
    }
  },
  "provider_conf": {
    "project_id": "your-project-id",
    "region": "us-central1"
  }
}

The service_account_json can also be set via the GCP_SERVICE_ACCOUNT environment variable.

Amazon Bedrock

{
  "auth": {
    "aws": {
      "access_key_id": "your-access-key-id",
      "secret_access_key": "your-secret-access-key",
      "session_token": "your-session-token"
    }
  },
  "provider_conf": {
    "region": "us-east-1"
  },
  "options": {
    "model": "your-model-id"
  }
}

The session token is required when you use temporary AWS credentials.

Custom OpenAI-Compatible API

{
  "auth": {
    "header": {
      "Authorization": "Bearer your-token"
    }
  },
  "override": {
    "endpoint": "https://your-custom-llm.com/v1/chat/completions"
  }
}

Step-by-Step: Route to OpenAI

1. Create a route with ai-proxy

a6 route create -f - <<'EOF'
{
  "id": "openai-chat",
  "uri": "/v1/chat/completions",
  "methods": ["POST"],
  "plugins": {
    "ai-proxy": {
      "provider": "openai",
      "auth": {
        "header": {
          "Authorization": "Bearer sk-your-openai-key"
        }
      },
      "options": {
        "model": "gpt-4",
        "temperature": 0.7,
        "max_tokens": 1024
      }
    }
  }
}
EOF

2. Send a Request

curl "http://127.0.0.1:9080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is 1+1?"}
    ]
  }'

The gateway adds authentication and forwards to OpenAI. The client never sees the API key.

Common Patterns

Streaming Responses

{
  "plugins": {
    "ai-proxy": {
      "provider": "openai",
      "auth": {
        "header": {
          "Authorization": "Bearer sk-your-key"
        }
      },
      "options": {
        "model": "gpt-4",
        "stream": true,
        "stream_options": {
          "include_usage": true
        }
      }
    }
  }
}

The route forces every request to use streaming and asks OpenAI to include token usage in the final server-sent event. To let clients choose per request, omit both streaming options from the route and send stream: true only in requests that should stream. APISIX adds stream_options.include_usage: true automatically for OpenAI Chat Completions requests that enable streaming in the request body.

Azure OpenAI

{
  "plugins": {
    "ai-proxy": {
      "provider": "azure-openai",
      "auth": {
        "header": {
          "api-key": "your-azure-key"
        }
      },
      "options": {
        "model": "gpt-4"
      },
      "override": {
        "endpoint": "https://myresource.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview"
      },
      "timeout": 60000
    }
  }
}

Embeddings Endpoint

a6 route create -f - <<'EOF'
{
  "id": "embeddings",
  "uri": "/v1/embeddings",
  "methods": ["POST"],
  "plugins": {
    "ai-proxy": {
      "provider": "openai",
      "auth": {
        "header": {
          "Authorization": "Bearer sk-your-key"
        }
      },
      "options": {
        "model": "text-embedding-3-small"
      }
    }
  }
}
EOF

The openai provider detects the input field and selects https://api.openai.com/v1/embeddings automatically.

Enable Logging

{
  "plugins": {
    "ai-proxy": {
      "provider": "openai",
      "auth": {
        "header": {
          "Authorization": "Bearer sk-your-key"
        }
      },
      "options": {
        "model": "gpt-4"
      },
      "logging": {
        "summaries": true,
        "payloads": false
      }
    }
  }
}

Model Routing with Multiple Routes

The plugin does not natively route by model. Use separate routes with vars matching on request body fields:

# Route requests for gpt-4 to OpenAI
a6 route create -f - <<'EOF'
{
  "id": "openai-gpt4",
  "uri": "/v1/chat/completions",
  "methods": ["POST"],
  "vars": [["post_arg.model", "==", "gpt-4"]],
  "plugins": {
    "ai-proxy": {
      "provider": "openai",
      "auth": { "header": { "Authorization": "Bearer sk-openai-key" } },
      "options": { "model": "gpt-4" }
    }
  }
}
EOF

# Route requests for deepseek-chat to DeepSeek
a6 route create -f - <<'EOF'
{
  "id": "deepseek-chat",
  "uri": "/v1/chat/completions",
  "methods": ["POST"],
  "vars": [["post_arg.model", "==", "deepseek-chat"]],
  "plugins": {
    "ai-proxy": {
      "provider": "deepseek",
      "auth": { "header": { "Authorization": "Bearer sk-deepseek-key" } },
      "options": { "model": "deepseek-chat" }
    }
  }
}
EOF

Load Balancing with ai-proxy-multi

For load balancing, failover, and priority-based routing across providers, use ai-proxy-multi instead:

{
  "plugins": {
    "ai-proxy-multi": {
      "balancer": {
        "algorithm": "roundrobin"
      },
      "fallback_strategy": ["rate_limiting", "http_429", "http_5xx"],
      "instances": [
        {
          "name": "openai-primary",
          "provider": "openai",
          "priority": 1,
          "weight": 8,
          "auth": {
            "header": { "Authorization": "Bearer sk-openai-key" }
          },
          "options": { "model": "gpt-4" }
        },
        {
          "name": "deepseek-backup",
          "provider": "deepseek",
          "priority": 0,
          "weight": 2,
          "auth": {
            "header": { "Authorization": "Bearer sk-deepseek-key" }
          },
          "options": { "model": "deepseek-chat" }
        }
      ]
    }
  }
}

Access Log Variables

Configure APISIX to log LLM metrics:

Variable	Description
`$request_type`	`traditional_http`, `ai_chat`, or `ai_stream`
`$llm_time_to_first_token`	Time to first token (ms)
`$llm_model`	Actual model used by provider
`$request_llm_model`	Model requested by client
`$llm_prompt_tokens`	Prompt token count
`$llm_completion_tokens`	Completion token count

Configuration Sync Example

version: "1"
routes:
  - id: openai-chat
    uri: /v1/chat/completions
    methods:
      - POST
    plugins:
      ai-proxy:
        provider: openai
        auth:
          header:
            Authorization: Bearer sk-your-openai-key
        options:
          model: gpt-4
          max_tokens: 1024
          temperature: 0.7
        logging:
          summaries: true

Troubleshooting

Symptom	Cause	Fix
502 Bad Gateway	Wrong endpoint or provider value	Verify `provider` matches your API; check `override.endpoint` for Azure/custom
401 from upstream	Invalid API key	Check `auth.header` value; ensure key is active with the provider
Timeout errors	Slow LLM response	Increase `timeout` (default 30000ms); use streaming for long completions
No token counts in streaming	Upstream stream omits usage data	Confirm the provider and model return streaming usage; APISIX requests it automatically for OpenAI Chat Completions
Azure 404	Missing api-version in URL	Include `?api-version=YYYY-MM-DD-preview` in `override.endpoint`
Vertex AI auth failure	Bad service account JSON	Set via `auth.gcp.service_account_json` or `GCP_SERVICE_ACCOUNT` env var

This page is generated from a6-plugin-ai-proxy/SKILL.md in the api7/a6 repository. Browse all skills on the AI Agent Skills page.

Overview​

When to Use​

Supported Providers​

Plugin Configuration Reference​

Authentication by Provider​

OpenAI / DeepSeek / AIMLAPI / OpenRouter​

Anthropic​

Azure OpenAI​

Gemini​

Vertex AI (GCP Service Account)​

Amazon Bedrock​

Custom OpenAI-Compatible API​

Step-by-Step: Route to OpenAI​

1. Create a route with ai-proxy​

2. Send a Request​

Common Patterns​

Streaming Responses​

Azure OpenAI​

Embeddings Endpoint​

Enable Logging​

Model Routing with Multiple Routes​

Load Balancing with ai-proxy-multi​

Access Log Variables​

Configuration Sync Example​

Troubleshooting​

Overview

When to Use

Supported Providers

Plugin Configuration Reference

Authentication by Provider

OpenAI / DeepSeek / AIMLAPI / OpenRouter

Anthropic

Azure OpenAI

Gemini

Vertex AI (GCP Service Account)

Amazon Bedrock

Custom OpenAI-Compatible API

Step-by-Step: Route to OpenAI

1. Create a route with ai-proxy

2. Send a Request

Common Patterns

Streaming Responses

Azure OpenAI

Embeddings Endpoint

Enable Logging

Model Routing with Multiple Routes

Load Balancing with ai-proxy-multi

Access Log Variables

Configuration Sync Example

Troubleshooting