Weights & Biases Inference

Weights & Biases Inference provides hosted inference for open models through the W&B platform. AISIX maps those models to stable aliases and controls caller access at the gateway.

Prerequisites

Before starting, prepare the following:

• One AISIX setup:
  • For AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
  • For the open-source AISIX gateway, prepare either a local AISIX installation or the Docker setup from the Open-Source AISIX Gateway Quickstart. Configure the gateway to load a declarative resources file.
• A W&B API key authorized for Inference.
• curl and jq.

Configure with AISIX Cloud

Export the AISIX Cloud connection details:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

W&B Inference is a community catalog provider with an OpenAI-compatible API. AISIX connects through the openai adapter and authenticates upstream requests with a bearer token.

Create a Provider Key

export WANDB_API_KEY="YOUR_WANDB_API_KEY"

PROVIDER_KEY_ID=$(
  curl -sS -X POST "$AISIX_CP/provider_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "wandb-prod",
      "provider": "wandb",
      "api_key": "'"${WANDB_API_KEY}"'",
      "api_base": "https://api.inference.wandb.ai/v1",
      "allowed_environments": ["'"${ENV_ID}"'"]
    }' | jq -er '.provider_key.id'
)

echo "$PROVIDER_KEY_ID"

The AISIX provider ID is wandb, not weights-and-biases. Do not add adapter; AISIX derives the openai adapter for catalog providers.

The explicit API base includes /v1. AISIX appends /chat/completions when it sends the upstream request.

W&B attributes requests to your default entity and the inference project when no project is specified. To use another W&B team and project, add an OpenAI-Project value in request.default_headers on the provider key. See Upstream Request Headers.

Create a Model

Create an alias with the complete W&B model ID:

MODEL_ID=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "wandb-llama-prod",
      "model_name": "meta-llama/Llama-3.1-8B-Instruct",
      "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
    }' | jq -er '.model.id'
)

echo "$MODEL_ID"

Preserve the publisher namespace and casing when substituting another model from the W&B Inference catalog.

Create a Caller API Key

AISIX_API_KEY=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "wandb-inference-caller",
      "allowed_models": ["'"${MODEL_ID}"'"]
    }' | jq -er '.plaintext'
)

echo "$AISIX_API_KEY"

Configure with the Open-Source AISIX Gateway

Export the upstream credential and choose the caller API key that applications will send to the gateway:

export WANDB_API_KEY="YOUR_PROVIDER_API_KEY"
export CALLER_API_KEY="YOUR_CALLER_API_KEY"

For a new gateway, use this complete resources file. For an existing gateway, merge the entries into its current file, preserving its other resources:

resources.yaml

_format_version: "1"

provider_keys:
  - display_name: "wandb-prod"
    provider: "wandb"
    adapter: "openai"
    api_key: ${WANDB_API_KEY}
    api_base: "https://api.inference.wandb.ai/v1"

models:
  - display_name: "wandb-llama-prod"
    provider: "wandb"
    model_name: "meta-llama/Llama-3.1-8B-Instruct"
    provider_key: "wandb-prod"

api_keys:
  - display_name: "wandb-inference-caller"
    key_env: CALLER_API_KEY
    allowed_models:
      - "wandb-llama-prod"

If AISIX is installed locally, validate the file before loading it:

aisix validate --resources resources.yaml

After validation, start the gateway with the referenced environment variables in its process environment. Reload an existing gateway only if those variables are already available to the process; otherwise, restart it with the updated environment.

If you use Docker, adapt the validation and startup commands in the Open-Source AISIX Gateway Quickstart. Mount this resources.yaml file and pass every environment variable it references with -e in both commands. After the resources load, prepare the shared verification request below:

export AISIX_API_KEY="$CALLER_API_KEY"

Verify the Provider Connection

Export the AISIX gateway origin:

# AISIX_PROXY has no trailing slash or endpoint path
# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"

Send a chat-completions request through the AISIX proxy:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "wandb-llama-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from W&B Inference."
      }
    ]
  }'

AISIX forwards the upstream model ID to POST /v1/chat/completions with the W&B key in the bearer header.

Endpoint Coverage

W&B Serverless Inference currently exposes OpenAI-compatible chat completions and model listing. AISIX adds its chat-based protocol bridges, but it cannot add an upstream capability W&B does not expose:

Route	Behavior with a wandb model alias
/v1/chat/completions	Supported, buffered and streaming.
/v1/responses	Supported through the Responses bridge. Fields without a chat-completions equivalent are ignored.
/v1/messages	Supported through AISIX translation to chat completions. /v1/messages/count_tokens is not supported because the model is not Anthropic-backed.
/v1/embeddings	Not supported. W&B Serverless Inference does not publish an embeddings endpoint on this API root.
/v1/models	Returns caller-accessible AISIX aliases, not the W&B catalog. Call GET /passthrough/wandb/models to reach W&B's native model-list endpoint with the configured provider credential.
/v1/images/generations, /v1/videos, and /v1/rerank	Not supported. W&B does not expose these APIs, and the wandb provider value is outside the normalized routes' allowlists.
/passthrough/wandb/*rest	Available through a configured passthrough route claiming this prefix with the W&B API root as its target_url; grant the route on the caller key's allowed_routes. Passthrough does not rewrite AISIX aliases and relays SSE responses incrementally. For the request shapes that record token usage, see Envelope Detection and Usage.

See Provider Compatibility for the full endpoint and provider matrix.

Troubleshooting

Symptom	Check
Upstream 401 or 403	Confirm the W&B key has Inference access.
Upstream 404	Keep /v1 in the API base and verify the model ID.
Model not found	Preserve the publisher namespace and casing.
Provider key creation returns 400	Use provider: "wandb" without adapter.

Next Steps

You have now connected AISIX to W&B Inference and verified the model alias. Continue with these guides:

• Provider Keys: manage provider-key rotation and environment access.
• Model Aliases: configure routing, retry behavior, or cost metadata for the alias.
• Routing and Failover: fail over between W&B Inference and another provider.
• Provider Compatibility: review supported proxy endpoints and provider-specific boundaries.