Skip to main content
API7.ai Logo

Bring Your Own Endpoint

Private model servers such as vLLM, SGLang, and Ollama can expose an OpenAI-compatible API from infrastructure you control. AISIX can route model traffic to these servers or to a private proxy in front of your own models.

Use a BYO endpoint when applications should keep calling AISIX with the OpenAI-compatible API while AISIX forwards traffic to a private or air-gapped model service. The endpoint must accept OpenAI-compatible chat-completions requests.

For AISIX Cloud steps tested against each engine's documented API shape, use the dedicated Ollama or vLLM guide. For another private OpenAI-compatible server such as SGLang, use the generic AISIX Cloud resource shape on this page.

Prerequisites​

Before starting, prepare the following:

  • One AISIX setup:
    • For AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
    • For the open-source AISIX gateway, prepare either a local AISIX installation or the Docker setup from the Open-Source AISIX Gateway Quickstart. Configure the gateway to load a declarative resources file.
  • A reachable OpenAI-compatible endpoint and its served model name. The examples use vLLM serving meta-llama/Llama-3.1-8B-Instruct.
  • curl and jq.

Configure with AISIX Cloud​

Export the AISIX Cloud connection details:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Use byo as the provider identifier, select the openai adapter, and set the endpoint explicitly:

PROVIDER_KEY_ID=$(curl -sS -X POST "$AISIX_CP/provider_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "vllm-private",
    "provider": "byo",
    "adapter": "openai",
    "api_key": "not-used-by-vllm",
    "api_base": "http://10.0.0.5:8000/v1",
    "allowed_environments": ["'"$ENV_ID"'"]
  }' | jq -r '.provider_key.id')

The AISIX Cloud Admin API uses provider: "byo" so usage data can distinguish custom endpoints from catalog providers. For the open-source AISIX gateway, provider in resources.yaml can be a descriptive label such as vllm. Both paths require a non-empty api_key; use a placeholder only when the endpoint ignores authentication.

Create the model:

MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "llama-3-private",
    "model_name": "meta-llama/Llama-3.1-8B-Instruct",
    "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
  }' | jq -r '.model.id')

Create a caller API key that can access the model:

BYO_CALLER_KEY=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "byo-caller",
    "allowed_models": ["'"${MODEL_ID}"'"]
  }' | jq -r '.plaintext')

Configure BYO pricing under Model Pricing. The cost block used by the open-source gateway is not accepted in an AISIX Cloud model request.

Configure with the Open-Source AISIX Gateway​

Declare a provider key, model alias, and caller API key for the private endpoint in resources.yaml.

Create a Provider Key​

Many privately operated inference servers do not require an API key. For an unauthenticated endpoint, use a non-empty placeholder in the provider key; AISIX sends it as the bearer token, and your server can ignore it.

Use the endpoint root expected by the server, such as http://host:8000/v1 for vLLM, http://host:30000/v1 for SGLang, or http://host:11434/v1 for Ollama. The example below uses http://10.0.0.5:8000/v1.

Declare a provider key for the private OpenAI-compatible endpoint:

_format_version: "1"
provider_keys:
  - display_name: vllm-private
    provider: vllm
    adapter: openai
    api_key: not-used-by-vllm
    api_base: http://10.0.0.5:8000/v1
  • provider is any short label that makes sense for your environment.
  • adapter selects the OpenAI-compatible upstream format.
  • api_key is a non-empty placeholder for unauthenticated endpoints. For an authenticated endpoint, reference the real credential from an environment variable, such as api_key: ${VLLM_API_KEY}, instead of writing a literal secret into the file.
  • api_base is the endpoint root. Include /v1 when that is part of the server's route.

Provider key secrets follow the credential-handling behavior described in Provider Keys.

Create a Model​

Map a caller-facing alias to the upstream model ID your endpoint serves:

models:
  - display_name: llama-3-private
    provider: vllm
    model_name: meta-llama/Llama-3.1-8B-Instruct
    provider_key: vllm-private
    cost:
      input_per_1k: 0.0
      output_per_1k: 0.0
  • display_name is the alias callers send in model.
  • model_name is the upstream ID your endpoint expects. For vLLM and SGLang, use the served model name. For Ollama, use the local model tag, such as llama3.1:8b.
  • provider_key attaches the model alias to the provider key by its display_name.
  • cost is optional. It provides the pricing metadata described below.

Add Pricing Metadata​

Catalog providers carry pricing from the models.dev catalog. A BYO endpoint is not in that catalog, so set pricing metadata yourself if you need token-cost accounting.

Replace the placeholder zero cost values in the model entry with your real per-1K rates:

    cost:
      input_per_1k: 0.10
      output_per_1k: 0.30

Both values are in USD per 1,000 tokens. input_per_1k applies to prompt tokens and output_per_1k to completion tokens. Both fields are required when the cost block is present.

The open-source gateway uses this metadata for usage events and least_cost routing, but does not enforce budgets from it. AISIX Cloud does not consume the cost block from resources.yaml; configure BYO pricing separately through Model Pricing. See Model Aliases for the resources-file fields.

Create a Caller API Key​

Declare an API key resource with access to the private model alias. The plaintext key value comes from an environment variable and is hashed when the gateway loads the file:

api_keys:
  - display_name: byo-caller
    key_env: BYO_CALLER_KEY
    allowed_models: ["llama-3-private"]

The allowed_models value must match the model alias you created.

Export the caller key value before starting the gateway:

# Replace with your value
export BYO_CALLER_KEY="YOUR_CALLER_API_KEY"

Validate and Load the Configuration​

If AISIX is installed locally, validate the complete file before loading it:

aisix validate --resources resources.yaml

After validation, start the gateway with the referenced environment variables in its process environment. If those variables are already available to a locally installed gateway process, send it a SIGHUP to reload the file:

kill -HUP "$(pgrep -x aisix)"

If you introduced a variable or changed its value, restart the gateway with the updated process environment instead.

If you use Docker, adapt the validation and startup commands in the Open-Source AISIX Gateway Quickstart. Mount this resources.yaml file and pass every environment variable it references with -e in both commands.

Verify the Provider Connection​

Export the AISIX gateway origin:

# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_ORIGIN"

Send a request through the proxy with the caller API key and model alias you created:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${BYO_CALLER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-private",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from the private model."
      }
    ]
  }'

The response should be an OpenAI-compatible chat-completions response that echoes the caller-facing alias. Check the endpoint access log for a POST /v1/chat/completions entry from AISIX.

If AISIX returns an upstream route or connection error, check api_base, the served model name, and endpoint reachability.

Support Additional Endpoints​

The private endpoint must implement each OpenAI-compatible route that applications call through AISIX. AISIX can forward embedding requests when the endpoint provides a compatible embeddings route. Other routes have additional provider requirements; see Provider Compatibility.

For small differences in request or response shape, configure Provider-Specific Overrides on the provider key.

Next Steps​

You have now connected a private OpenAI-compatible endpoint to AISIX. Continue with these guides:

  • Model Aliases: configure routing, retry behavior, or cost metadata for the alias.
  • Budgets: use the pricing metadata for AISIX Cloud budget enforcement.
  • Provider Compatibility: review supported proxy endpoints and provider-specific boundaries.