Skip to main content
API7.ai Logo
Version: Dev

CoreWeave Serverless Inference

CoreWeave Serverless Inference provides hosted access to open-source foundation models through an OpenAI-compatible API. AISIX maps those models to stable aliases and controls caller access at the gateway.

Prerequisites​

Before starting, prepare the following:

Configure with AISIX Cloud​

Export the AISIX Cloud connection details:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

CoreWeave Serverless Inference is a community catalog provider with an OpenAI-compatible API. AISIX connects through the openai adapter and authenticates upstream requests with a bearer token.

Create a Provider Key​

Create a provider key that stores the Forge API key and uses the legacy wandb provider ID:

export COREWEAVE_API_KEY="YOUR_FORGE_API_KEY"

PROVIDER_KEY_ID=$(
  curl -sS -X POST "$AISIX_CP/provider_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "coreweave-prod",
      "provider": "wandb",
      "api_key": "'"${COREWEAVE_API_KEY}"'",
      "api_base": "https://api.inference.wandb.ai/v1",
      "allowed_environments": ["'"${ENV_ID}"'"]
    }' | jq -er '.provider_key.id'
)

echo "$PROVIDER_KEY_ID"

The AISIX catalog keeps the legacy provider ID wandb for CoreWeave Serverless Inference. Do not add adapter; AISIX derives the openai adapter for catalog providers. CoreWeave also continues to serve this API from the legacy W&B hostname, api.inference.wandb.ai.

The explicit API base includes /v1. AISIX appends /chat/completions when it sends the upstream request.

Usage attribution still uses W&B entity and project identifiers. When no project is specified, the service uses your default entity and the inference project. To attribute usage elsewhere, set OpenAI-Project to {entity}/{project} in request.default_headers on the provider key. See Upstream Request Headers.

Create a Model​

Create an alias with the complete Serverless Inference model ID:

MODEL_ID=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "coreweave-llama-prod",
      "model_name": "meta-llama/Llama-3.1-8B-Instruct",
      "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
    }' | jq -er '.model.id'
)

echo "$MODEL_ID"

Preserve the publisher namespace and casing when substituting another model from the Serverless Inference catalog.

Create a Caller API Key​

Create a caller key that grants access to the model alias:

AISIX_API_KEY=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "coreweave-inference-caller",
      "allowed_models": ["'"${MODEL_ID}"'"]
    }' | jq -er '.plaintext'
)

echo "$AISIX_API_KEY"

Configure with the Open-Source AISIX Gateway​

Export the upstream credential and choose the caller API key that applications will send to the gateway:

export COREWEAVE_API_KEY="YOUR_FORGE_API_KEY"
export CALLER_API_KEY="YOUR_CALLER_API_KEY"

For a new gateway, use this complete resources file. For an existing gateway, merge the entries into its current file, preserving its other resources:

resources.yaml
_format_version: "1"

provider_keys:
  - display_name: "coreweave-prod"
    provider: "wandb"
    adapter: "openai"
    api_key: ${COREWEAVE_API_KEY}
    api_base: "https://api.inference.wandb.ai/v1"

models:
  - display_name: "coreweave-llama-prod"
    provider: "wandb"
    model_name: "meta-llama/Llama-3.1-8B-Instruct"
    provider_key: "coreweave-prod"

api_keys:
  - display_name: "coreweave-inference-caller"
    key_env: CALLER_API_KEY
    allowed_models:
      - "coreweave-llama-prod"

If AISIX is installed locally, validate the file before loading it:

aisix validate --resources resources.yaml

After validation, start the gateway with the referenced environment variables in its process environment. Reload an existing gateway only if those variables are already available to the process; otherwise, restart it with the updated environment.

If you use Docker, adapt the validation and startup commands in the Open-Source AISIX Gateway Quickstart. Mount this resources.yaml file and pass every environment variable it references with -e in both commands. After the resources load, prepare the shared verification request below:

export AISIX_API_KEY="$CALLER_API_KEY"

Verify the Provider Connection​

Export the AISIX gateway origin:

# AISIX_PROXY has no trailing slash or endpoint path
# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"

Send a chat-completions request through the AISIX proxy:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "coreweave-llama-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from CoreWeave Serverless Inference."
      }
    ]
  }'

AISIX forwards the upstream model ID to POST /v1/chat/completions with the Forge API key in the bearer header.

Endpoint Coverage​

CoreWeave Serverless Inference currently exposes OpenAI-compatible chat completions and model listing. AISIX adds its chat-based protocol bridges, but it cannot add an upstream capability that Serverless Inference does not expose:

RouteBehavior with a model using provider wandb
/v1/chat/completionsSupported, buffered and streaming.
/v1/responsesSupported through the Responses bridge. Fields without a chat-completions equivalent are ignored.
/v1/messagesSupported through AISIX translation to chat completions. /v1/messages/count_tokens is not supported because the model is not Anthropic-backed.
/v1/embeddingsNot supported. CoreWeave Serverless Inference does not publish an embeddings endpoint on this API root.
/v1/modelsReturns caller-accessible AISIX aliases, not the CoreWeave model catalog. Call GET /passthrough/wandb/models to reach CoreWeave's native model-list endpoint with the configured provider credential.
/v1/images/generations, /v1/videos, and /v1/rerankNot supported. CoreWeave does not expose these APIs on Serverless Inference, and the wandb provider value is outside the normalized routes' allowlists.
/passthrough/wandb/*restAvailable through a configured passthrough route claiming this prefix with the CoreWeave API root as its target_url; grant the route on the caller key's allowed_routes. Passthrough does not rewrite AISIX aliases and relays SSE responses incrementally. For the request shapes that record token usage, see Envelope Detection and Usage.

See Provider Compatibility for the full endpoint and provider matrix.

Troubleshooting​

SymptomCheck
Upstream 401 or 403Confirm the Forge API key is valid and belongs to an account with Serverless Inference access.
Upstream 404Keep /v1 in the API base and verify the model ID.
Model not foundPreserve the publisher namespace and casing.
Provider key creation returns 400Use provider: "wandb" without adapter.

Next Steps​

You have now connected AISIX to CoreWeave Serverless Inference and verified the model alias. Continue with these guides: