vLLM

Connect AISIX AI Gateway to a private vLLM OpenAI-compatible server. AISIX centralizes caller authentication, model aliases, usage reporting, and traffic controls while vLLM serves model inference.

vLLM is a private endpoint rather than an AISIX catalog provider. Configure it as byo and select the openai adapter.

Setting	Value used by AISIX
Provider ID	byo
Adapter	openai
Default vLLM API base	http://localhost:8000/v1
Example upstream model	NousResearch/Meta-Llama-3-8B-Instruct
Authentication	Bearer API key configured on vLLM

Prerequisites

Prepare:

• An AISIX Cloud environment with an attached gateway.
• A host with vLLM installed and enough compute and storage for the selected model.
• Network reachability from the AISIX data plane to vLLM.
• curl and jq.

Start the official example model with API-key checking enabled:

vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123

Export the AISIX connection details, the same upstream key, and an API base reachable from the AISIX data plane:

export AISIX_CP="http://localhost:8080/api"
export AISIX_TOKEN="aisix_pat_YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"
export AISIX_PROXY="http://127.0.0.1:3000"

export VLLM_API_KEY="token-abc123"
export VLLM_API_BASE="http://vllm.internal:8000/v1"

Replace vllm.internal with a resolvable host or service name. For Docker Desktop with vLLM on the host, http://host.docker.internal:8000/v1 is a typical value. If both services share a Docker or Kubernetes network, use the vLLM service DNS name.

caution

vLLM documents that --api-key protects its OpenAI-compatible API routes, not every endpoint the server may expose. Keep the vLLM service on a private network and apply network policy or an authenticated reverse proxy when administrative or diagnostic routes require protection.

Configure the Provider Upstream

Create a Provider Key

PROVIDER_KEY_ID=$(
  curl -sS -X POST "$AISIX_CP/provider_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "vllm-local",
      "provider": "byo",
      "adapter": "openai",
      "api_key": "'"${VLLM_API_KEY}"'",
      "api_base": "'"${VLLM_API_BASE}"'",
      "allowed_environments": ["'"${ENV_ID}"'"]
    }' | jq -er '.provider_key.id'
)

echo "$PROVIDER_KEY_ID"

A BYO key requires provider: "byo", a non-empty api_key, and api_base. This guide sets adapter: "openai" explicitly; when omitted, AISIX defaults a BYO key to the OpenAI-compatible adapter. AISIX sends VLLM_API_KEY as Authorization: Bearer <key>, matching the key passed to vllm serve.

If you intentionally run vLLM without --api-key, AISIX still requires a non-empty placeholder in the provider-key schema. An authenticated vLLM endpoint is preferred whenever traffic can originate outside a tightly controlled local network.

Create a Model

Use the model name served by vLLM:

MODEL_ID=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "vllm-llama-prod",
      "model_name": "NousResearch/Meta-Llama-3-8B-Instruct",
      "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
    }' | jq -er '.model.id'
)

echo "$MODEL_ID"

model_name must match the name vLLM exposes from /v1/models. If you start vLLM with a served-model-name override, use that override rather than the model repository path.

Create a Caller API Key

AISIX_API_KEY=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "vllm-caller",
      "allowed_models": ["'"${MODEL_ID}"'"]
    }' | jq -er '.plaintext'
)

echo "$AISIX_API_KEY"

Verify the Provider Connection

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm-llama-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from vLLM."
      }
    ]
  }'

AISIX sends the served model name to POST /v1/chat/completions with the configured vLLM bearer key.

Endpoint Coverage

vLLM documents OpenAI-compatible completions, chat completions, Responses, embeddings, and audio routes, with support depending on the served model type. Through AISIX, chat completions is the primary route; compatible Responses and Messages requests can use the chat bridges in AISIX. Use a separate alias for an embedding, transcription, or translation model.

The normalized AISIX image, video, and rerank routes do not accept the byo provider value. See Provider Compatibility.

Troubleshooting

Symptom	Check
Connection refused or timeout	Resolve and call VLLM_API_BASE from the AISIX data plane container.
Upstream 401	Use the same value for --api-key and the AISIX provider key's api_key.
Model not found	Compare model_name with GET $VLLM_API_BASE/models.
Chat-template error	Serve a chat model with a valid chat template or configure one in vLLM.
Provider key creation returns 400	Include provider: "byo", a non-empty api_key, and api_base. adapter is optional; if supplied, use a supported value such as openai.

For a generic private-server setup and custom pricing, see Bring Your Own Endpoint.