vLLM

vLLM is an open-source inference server that exposes an OpenAI-compatible API for models you run. AISIX adds caller authentication, stable model aliases, usage reporting, and traffic controls while vLLM continues to serve inference.

Prerequisites

Before starting, prepare the following:

• One AISIX setup:
  • For AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
  • For the open-source AISIX gateway, prepare either a local AISIX installation or the Docker setup from the Open-Source AISIX Gateway Quickstart. Configure the gateway to load a declarative resources file.
• A host with vLLM installed and enough compute and storage for the selected model.
• Network reachability from the AISIX gateway to vLLM.
• curl and jq.

Prepare the Inference Server

Start the official example model with API-key checking enabled:

vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123

Export the upstream key and an API base reachable from the AISIX gateway:

export VLLM_API_KEY="token-abc123"
export VLLM_API_BASE="http://vllm.internal:8000/v1"

Replace vllm.internal with a resolvable host or service name. For Docker Desktop with vLLM on the host, http://host.docker.internal:8000/v1 is a typical value. If both services share a Docker or Kubernetes network, use the vLLM service DNS name.

caution

vLLM documents that --api-key protects its OpenAI-compatible API routes, not every endpoint the server may expose. Keep the vLLM service on a private network and apply network policy or an authenticated reverse proxy when administrative or diagnostic routes require protection.

Configure with AISIX Cloud

Export the AISIX Cloud connection details:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

vLLM is a private endpoint rather than an AISIX catalog provider. Configure it with the byo provider value and select the openai adapter explicitly.

Create a Provider Key

PROVIDER_KEY_ID=$(
  curl -sS -X POST "$AISIX_CP/provider_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "vllm-local",
      "provider": "byo",
      "adapter": "openai",
      "api_key": "'"${VLLM_API_KEY}"'",
      "api_base": "'"${VLLM_API_BASE}"'",
      "allowed_environments": ["'"${ENV_ID}"'"]
    }' | jq -er '.provider_key.id'
)

echo "$PROVIDER_KEY_ID"

A BYO key requires provider: "byo", a non-empty api_key, and api_base. This guide sets adapter: "openai" explicitly; when omitted, AISIX defaults a BYO key to the OpenAI-compatible adapter. AISIX sends VLLM_API_KEY as Authorization: Bearer <key>, matching the key passed to vllm serve.

If you intentionally run vLLM without --api-key, AISIX still requires a non-empty placeholder in the provider-key schema. An authenticated vLLM endpoint is preferred whenever traffic can originate outside a tightly controlled local network.

Create a Model

Use the model name served by vLLM:

MODEL_ID=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "vllm-llama-prod",
      "model_name": "NousResearch/Meta-Llama-3-8B-Instruct",
      "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
    }' | jq -er '.model.id'
)

echo "$MODEL_ID"

model_name must match the name vLLM exposes from /v1/models. If you start vLLM with a served-model-name override, use that override rather than the model repository path.

Create a Caller API Key

AISIX_API_KEY=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "vllm-caller",
      "allowed_models": ["'"${MODEL_ID}"'"]
    }' | jq -er '.plaintext'
)

echo "$AISIX_API_KEY"

Configure with the Open-Source AISIX Gateway

Export the upstream credential and choose the caller API key that applications will send to the gateway:

export VLLM_API_KEY="token-abc123"
export VLLM_API_BASE="http://vllm.internal:8000/v1"
export CALLER_API_KEY="YOUR_CALLER_API_KEY"

Create a complete declarative resources file for this provider:

resources.yaml

_format_version: "1"

provider_keys:
  - display_name: "vllm-local"
    provider: "vllm"
    adapter: "openai"
    api_key: ${VLLM_API_KEY}
    api_base: "${VLLM_API_BASE}"

models:
  - display_name: "vllm-llama-prod"
    provider: "vllm"
    model_name: "NousResearch/Meta-Llama-3-8B-Instruct"
    provider_key: "vllm-local"

api_keys:
  - display_name: "vllm-caller"
    key_env: CALLER_API_KEY
    allowed_models:
      - "vllm-llama-prod"

If AISIX is installed locally, validate the file before loading it:

aisix validate --resources resources.yaml

After validation, start the gateway with the referenced environment variables in its process environment. Reload an existing gateway only if those variables are already available to the process; otherwise, restart it with the updated environment.

If you use Docker, adapt the validation and startup commands in the Open-Source AISIX Gateway Quickstart. Mount this resources.yaml file and pass every environment variable it references with -e in both commands. After the resources load, prepare the shared verification request below:

export AISIX_API_KEY="$CALLER_API_KEY"

Verify the Provider Connection

Export the AISIX gateway origin:

# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_ORIGIN"

Send a chat-completions request through the AISIX proxy:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm-llama-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from vLLM."
      }
    ]
  }'

AISIX sends the served model name to POST /v1/chat/completions with the configured vLLM bearer key.

Endpoint Coverage

The current vLLM online server exposes OpenAI-compatible generation, Responses, embeddings, and speech-to-text APIs. It also exposes Anthropic-compatible Messages and token-counting routes, plus pooling-model APIs such as rerank and score. Availability still depends on the model and task selected when the vLLM server starts.

Route	Behavior with the vLLM alias in this guide
/v1/chat/completions	Supported for a text-generation model with a chat template.
/v1/completions	Supported for a text-generation model. vLLM does not support the OpenAI suffix parameter.
/v1/responses	Supported through the AISIX Responses bridge, not the native vLLM Responses API. Fields without a chat-completions equivalent are ignored. Use /passthrough/byo/responses or /passthrough/vllm/responses, matching the prefix your passthrough route claims, when native Responses semantics are required.
/v1/messages	Supported through AISIX translation to chat completions, not the native Anthropic-compatible vLLM route. Use /passthrough/byo/messages or /passthrough/vllm/messages for the native vLLM contract.
/v1/messages/count_tokens	Rejected because the normalized route requires an Anthropic-backed model. Current vLLM releases implement this route natively; use your passthrough route's prefix followed by /messages/count_tokens.
/v1/embeddings	Supported when the alias points to a vLLM server running an embedding model.
/v1/audio/transcriptions and /v1/audio/translations	Supported when the alias points to a vLLM server running a compatible automatic speech recognition model with the vLLM audio dependencies installed. Translation support is model-dependent.
/v1/audio/speech	Not supported because vLLM does not publish an OpenAI-compatible text-to-speech route.
/v1/realtime	Supported when vLLM serves a Realtime-capable automatic speech recognition model and has its audio dependencies installed. AISIX relays the OpenAI-style vLLM WebSocket protocol through the normalized Realtime route. Current vLLM support is streaming speech-to-text, not the bidirectional speech-generation behavior available from some OpenAI Realtime models.
/v1/rerank	Rejected because neither byo nor vllm is in the normalized route's provider allowlist. A vLLM server running a scoring model exposes /v1/rerank; reach it at /passthrough/byo/rerank or /passthrough/vllm/rerank.
/v1/chat/completions/batch and /v1/score	AISIX has no normalized routes for these vLLM APIs. Reach them through your passthrough route's prefix followed by /chat/completions/batch or /score.
/v1/models	Returns caller-accessible AISIX aliases, not the models served by vLLM. Use /passthrough/byo/models or /passthrough/vllm/models for the native vLLM list.
/v1/images/generations and /v1/videos	Rejected because neither provider value is in the corresponding normalized route's allowlist. vLLM does not publish matching generation APIs.

Use a separate model alias, and normally a separate provider key and serving process, for each generation, embedding, or speech-to-text model. The same requirement applies to a reranking model reached through a passthrough route.

The /passthrough/byo and /passthrough/vllm prefixes above assume a passthrough route claiming the prefix with the vLLM server root as its target_url; grant the route name on the caller key's allowed_routes. A passthrough route relays the request body unchanged, so use the model identifier exposed by vLLM rather than the AISIX alias. It relays upstream responses, including SSE, incrementally. AISIX detects chat, completions, and Responses envelopes and records supported usage fields. Requests without a recognized carrier field remain opaque: buffered responses record zero tokens, while opaque SSE can still record top-level supported usage fields. A route binds one fixed target_url, so create one route per vLLM server instead of relying on model access to pick an endpoint.

vLLM also publishes root-level APIs such as /pooling and /classify. They are outside the /v1 API base configured in this guide and have no normalized AISIX route. Reaching them through a passthrough route requires a target_url at the vLLM server root; do not assume a route targeting the /v1 base above can reach them.

Troubleshooting

Symptom	Check
Connection refused or timeout	Resolve and call VLLM_API_BASE from the AISIX gateway container.
Upstream 401	Use the same value for --api-key and the AISIX provider key's api_key.
Model not found	Compare model_name with GET $VLLM_API_BASE/models.
Chat-template error	Serve a chat model with a valid chat template or configure one in vLLM.
Provider key creation returns 400	Include provider: "byo", adapter: "openai", a non-empty api_key, and api_base.

Next Steps

You have now connected AISIX to vLLM and verified the model alias. Continue with these guides:

• Bring Your Own Endpoint: review the reusable setup and custom pricing options for private OpenAI-compatible servers.
• Model Aliases: configure routing, retry behavior, or cost metadata for the alias.
• Provider Compatibility: review supported proxy endpoints and provider-specific boundaries.