Weights & Biases Inference

Weights & Biases Inference provides hosted inference for open models through the W&B platform. AISIX maps those models to stable aliases and controls caller access at the gateway.

Prerequisites

Before starting, prepare the following:

• One AISIX setup:
  • For AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
  • For the open-source AISIX gateway, prepare either a local AISIX installation or the Docker setup from the Open-Source AISIX Gateway Quickstart. Configure the gateway to load a declarative resources file.
• A W&B API key authorized for Inference.
• curl and jq.

Configure with AISIX Cloud

Export the AISIX Cloud connection details:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

W&B Inference is a community catalog provider with an OpenAI-compatible API. AISIX connects through the openai adapter and authenticates upstream requests with a bearer token.

Create a Provider Key

export WANDB_API_KEY="YOUR_WANDB_API_KEY"

PROVIDER_KEY_ID=$(
  curl -sS -X POST "$AISIX_CP/provider_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "wandb-prod",
      "provider": "wandb",
      "api_key": "'"${WANDB_API_KEY}"'",
      "api_base": "https://api.inference.wandb.ai/v1",
      "allowed_environments": ["'"${ENV_ID}"'"]
    }' | jq -er '.provider_key.id'
)

echo "$PROVIDER_KEY_ID"

The AISIX provider ID is wandb, not weights-and-biases. Do not add adapter; AISIX derives the openai adapter for catalog providers.

The explicit API base includes /v1. AISIX appends /chat/completions when it sends the upstream request.

W&B attributes requests to your default entity and the inference project when no project is specified. To use another W&B team and project, add an OpenAI-Project value in request.default_headers on the provider key. See Upstream Request Headers.

Create a Model

Create an alias with the complete W&B model ID:

MODEL_ID=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "wandb-llama-prod",
      "model_name": "meta-llama/Llama-3.1-8B-Instruct",
      "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
    }' | jq -er '.model.id'
)

echo "$MODEL_ID"

Preserve the publisher namespace and casing when substituting another model from the W&B Inference catalog.

Create a Caller API Key

AISIX_API_KEY=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "wandb-inference-caller",
      "allowed_models": ["'"${MODEL_ID}"'"]
    }' | jq -er '.plaintext'
)

echo "$AISIX_API_KEY"

Configure with the Open-Source AISIX Gateway

Export the upstream credential and choose the caller API key that applications will send to the gateway:

export WANDB_API_KEY="YOUR_PROVIDER_API_KEY"
export CALLER_API_KEY="YOUR_CALLER_API_KEY"

Create a complete declarative resources file for this provider:

resources.yaml

_format_version: "1"

provider_keys:
  - display_name: "wandb-prod"
    provider: "wandb"
    adapter: "openai"
    api_key: ${WANDB_API_KEY}
    api_base: "https://api.inference.wandb.ai/v1"

models:
  - display_name: "wandb-llama-prod"
    provider: "wandb"
    model_name: "meta-llama/Llama-3.1-8B-Instruct"
    provider_key: "wandb-prod"

api_keys:
  - display_name: "wandb-inference-caller"
    key_env: CALLER_API_KEY
    allowed_models:
      - "wandb-llama-prod"

If AISIX is installed locally, validate the file before loading it:

aisix validate --resources resources.yaml

After validation, start the gateway with the referenced environment variables in its process environment. Reload an existing gateway only if those variables are already available to the process; otherwise, restart it with the updated environment.

If you use Docker, adapt the validation and startup commands in the Open-Source AISIX Gateway Quickstart. Mount this resources.yaml file and pass every environment variable it references with -e in both commands. After the resources load, prepare the shared verification request below:

export AISIX_API_KEY="$CALLER_API_KEY"

Verify the Provider Connection

Export the AISIX gateway origin:

# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_ORIGIN"

Send a chat-completions request through the AISIX proxy:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "wandb-llama-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from W&B Inference."
      }
    ]
  }'

AISIX forwards the upstream model ID to POST /v1/chat/completions with the W&B key in the bearer header.

Endpoint Coverage

W&B Serverless Inference currently exposes OpenAI-compatible chat completions and model listing. AISIX adds its chat-based protocol bridges, but it cannot add an upstream capability W&B does not expose:

Route	Behavior with a wandb model alias
/v1/chat/completions	Supported, buffered and streaming.
/v1/responses	Supported through the Responses bridge. Fields without a chat-completions equivalent are ignored.
/v1/messages	Supported through AISIX translation to chat completions. /v1/messages/count_tokens is not supported because the model is not Anthropic-backed.
/v1/embeddings	Not supported. W&B Serverless Inference does not publish an embeddings endpoint on this API root.
/v1/models	Returns caller-accessible AISIX aliases, not the W&B catalog. Call GET /passthrough/wandb/models to reach W&B's native model-list endpoint with the configured provider credential.
/v1/images/generations, /v1/videos, and /v1/rerank	Not supported. W&B does not expose these APIs, and the wandb provider value is outside the normalized routes' allowlists.
/passthrough/wandb/*rest	Available through a configured passthrough route claiming this prefix with the W&B API root as its target_url; grant the route on the caller key's allowed_routes. Passthrough does not rewrite AISIX aliases and relays SSE responses incrementally. Recognized chat, completions, and Responses envelopes record supported usage fields. Requests without a recognized carrier field remain opaque: buffered responses record zero tokens, while opaque SSE can still record top-level supported usage fields.

See Provider Compatibility for the full endpoint and provider matrix.

Troubleshooting

Symptom	Check
Upstream 401 or 403	Confirm the W&B key has Inference access.
Upstream 404	Keep /v1 in the API base and verify the model ID.
Model not found	Preserve the publisher namespace and casing.
Provider key creation returns 400	Use provider: "wandb" without adapter.

Next Steps

You have now connected AISIX to W&B Inference and verified the model alias. Continue with these guides:

• Provider Keys: manage provider-key rotation and environment access.
• Model Aliases: configure routing, retry behavior, or cost metadata for the alias.
• Routing and Failover: fail over between W&B Inference and another provider.
• Provider Compatibility: review supported proxy endpoints and provider-specific boundaries.