NVIDIA NIM

NVIDIA NIM provides inference microservices that can run in your infrastructure. NVIDIA also exposes selected NIMs through hosted API endpoints. Applications select those models through stable AISIX aliases while the gateway holds the NVIDIA API key.

This guide configures the shared hosted LLM chat API at https://integrate.api.nvidia.com/v1 and embedding models that publish the compatible /v1/embeddings route. The API Catalog also includes retrieval, visual-generation, speech, and other NIMs with model-specific paths, request bodies, or hosts. The hosted catalog and its available models change over time, so check the selected model's API reference before applying this shared configuration to another NIM family.

Prerequisites

Before starting, prepare the following:

One AISIX setup:
- For AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
- For the open-source AISIX gateway, prepare either a local AISIX installation or the Docker setup from the Open-Source AISIX Gateway Quickstart. Configure the gateway to load a declarative resources file.
An NVIDIA API key for the hosted NIM API, generated from a model page on build.nvidia.com.
curl and jq.

Configure with AISIX Cloud

Export the AISIX Cloud connection details:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Create a provider key, model alias, and caller API key for the NVIDIA-backed chat-completions route.

NVIDIA is a community catalog provider whose shared hosted LLM endpoint accepts OpenAI chat-completions requests. AISIX connects through the openai adapter, authenticates upstream requests with a bearer token, and uses the NVIDIA API root as api_base. AISIX does not register NVIDIA-specific request or response rewrites. Review Supply NVIDIA-Specific Behavior for fields that differ from the standard OpenAI shape.

The dashboard groups NVIDIA under All providers (community) and identifies the wire compatibility as assumed rather than verified.

Create a Provider Key

Create the provider key that stores the NVIDIA credential and API root:

# Replace with your value
export NVIDIA_API_KEY="YOUR_PROVIDER_API_KEY"

PROVIDER_KEY_ID=$(curl -sS -X POST "$AISIX_CP/provider_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "nvidia-prod",
    "provider": "nvidia",
    "api_key": "'"${NVIDIA_API_KEY}"'",
    "api_base": "https://integrate.api.nvidia.com/v1",
    "allowed_environments": ["'"${ENV_ID}"'"]
  }' | jq -r '.provider_key.id')

echo "$PROVIDER_KEY_ID"

❶ provider is nvidia. The AISIX Cloud Admin API accepts the value because nvidia is one of the models.dev catalog IDs it caches, and it assigns the openai adapter and bearer authentication from the catalog's community default rule. The adapter field is only accepted on BYO provider keys, so do not set it here.

❷ api_key stores the NVIDIA API key. NVIDIA authenticates the hosted NIM API with HTTP bearer authentication, which is what the openai adapter already sends. The value follows the credential-handling behavior in Provider Keys.

❸ api_base is https://integrate.api.nvidia.com/v1, the root NVIDIA documents for the shared hosted LLM API. The chat route hangs off that root as POST https://integrate.api.nvidia.com/v1/chat/completions, and AISIX appends /chat/completions to api_base, so the value must stop at /v1. AISIX strips a pasted endpoint suffix such as /chat/completions and any trailing slash, but treat the root as the contract rather than relying on that repair.

Do not reuse this root for every API Catalog entry. For example, hosted reranking uses a retrieval-specific endpoint under https://ai.api.nvidia.com, and visual-generation NIMs publish other paths. Create a separate provider key with the exact API root from the selected model's API reference when it does not use the shared LLM or embeddings route.

For nvidia the field is optional: models.dev publishes this same URL in its api field, and the AISIX Cloud Admin API fills it in when you omit it. The examples set it explicitly so the root each key targets stays visible in the configuration.

caution

Never leave an nvidia provider key without a resolved api_base. The OpenAI-family bridge refuses to fall back to the default OpenAI host for any non-OpenAI vendor, so an empty base URL produces an upstream configuration error at request time instead of a misdirected request that would carry your NVIDIA credential to another vendor's host.

The command captures the returned provider key ID in PROVIDER_KEY_ID.

Create a Model

On the shared hosted LLM and embeddings routes, NVIDIA generally namespaces model IDs by publisher organization, in the form <publisher>/<model>. The publisher segment is part of these IDs and is never optional. Current examples include the following:

Model ID	Publisher
`nvidia/nvidia-nemotron-nano-9b-v2`	NVIDIA
`meta/llama-3.3-70b-instruct`	Meta
`openai/gpt-oss-120b`	OpenAI

Two naming details cause most alias mistakes:

Some NVIDIA-published model names already begin with nvidia-, so the full ID repeats the segment, as in nvidia/nvidia-nemotron-nano-9b-v2. That is correct, not a typographical error.
A model page URL on build.nvidia.com is a page slug, not the model ID. The page for Llama 3.3 70B Instruct uses llama-3_3-70b-instruct in its path while the API model ID is meta/llama-3.3-70b-instruct. Copy the ID from the code sample on the model page rather than from the address bar.

Check the model's page under NVIDIA's NIM API reference for both the current endpoint and the request body's model ID before you create an alias. Some domain-specific APIs use a model value that differs from the publisher-qualified catalog card name.

Create the model alias callers will send in requests:

MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "nvidia-llama-prod",
    "model_name": "meta/llama-3.3-70b-instruct",
    "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
  }' | jq -r '.model.id')

echo "$MODEL_ID"

❶ display_name is the alias callers send in model.

❷ model_name is the NVIDIA model ID, including the publisher segment. Do not carry over a bare identifier such as llama-3.3-70b-instruct from a provider that serves the same weights without a publisher prefix.

❸ provider_key_id attaches the alias to the NVIDIA provider key.

Create a Caller API Key

Create the caller API key that can access the model alias. The plaintext key is server-generated and returned once in the create response, so capture it now:

AISIX_API_KEY=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "nvidia-caller",
    "allowed_models": ["'"${MODEL_ID}"'"]
  }' | jq -r '.plaintext')

echo "$AISIX_API_KEY"

The allowed_models value must reference the model ID captured in the previous step. After the write, the configuration projects to attached gateways automatically.

Configure with the Open-Source AISIX Gateway

Export the upstream credential and choose the caller API key that applications will send to the gateway:

export NVIDIA_API_KEY="YOUR_PROVIDER_API_KEY"
export CALLER_API_KEY="YOUR_CALLER_API_KEY"

Create a complete declarative resources file for this provider:

resources.yaml
_format_version: "1"

provider_keys:
  - display_name: "nvidia-prod"
    provider: "nvidia"
    adapter: "openai"
    api_key: ${NVIDIA_API_KEY}
    api_base: "https://integrate.api.nvidia.com/v1"

models:
  - display_name: "nvidia-llama-prod"
    provider: "nvidia"
    model_name: "meta/llama-3.3-70b-instruct"
    provider_key: "nvidia-prod"

api_keys:
  - display_name: "nvidia-caller"
    key_env: CALLER_API_KEY
    allowed_models:
      - "nvidia-llama-prod"

If AISIX is installed locally, validate the file before loading it:

aisix validate --resources resources.yaml

After validation, start the gateway with the referenced environment variables in its process environment. Reload an existing gateway only if those variables are already available to the process; otherwise, restart it with the updated environment.

If you use Docker, adapt the validation and startup commands in the Open-Source AISIX Gateway Quickstart. Mount this resources.yaml file and pass every environment variable it references with -e in both commands. After the resources load, prepare the shared verification request below:

export AISIX_API_KEY="$CALLER_API_KEY"

Verify the Provider Connection

Export the AISIX gateway origin:

# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_ORIGIN"

Send a chat-completions request through the AISIX proxy:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia-llama-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from NVIDIA NIM."
      }
    ]
  }'

The gateway returns an OpenAI-compatible response that echoes the caller-facing alias nvidia-llama-prod. If the request fails, check the provider key api_key, the api_base root, and the publisher-namespaced model ID in model_name. An upstream 404 often indicates an incorrect model ID, a model that is no longer available, or a valid catalog model sent to the wrong shared route.

Choose Between the Hosted API and a Self-Hosted NIM

NIM microservices can run in your infrastructure as containers or be accessed through NVIDIA-hosted API endpoints. Only the hosted API is represented by the nvidia catalog provider. An LLM NIM running in your own cluster serves an OpenAI-compatible API at its own address, such as http://10.0.0.5:8000/v1, and reaches AISIX through the private-endpoint path instead. Configure that with Bring Your Own Endpoint.

Aspect	Hosted NIM API	Self-hosted NIM microservice
Provider value	`nvidia`	`byo` on the AISIX Cloud Admin API, or your own label in a declarative `resources.yaml`
`adapter` field	Rejected. The adapter is derived from the catalog.	Accepted, and set to `openai`.
`api_base`	`https://integrate.api.nvidia.com/v1`, defaulted from the catalog when omitted	Required. Your container root, such as `http://10.0.0.5:8000/v1`.
Model ID	Publisher-namespaced, such as `meta/llama-3.3-70b-instruct`	The model name the container serves
Pricing metadata	Sourced from models.dev when available; verify or override `cost` on the model alias for billing	You supply `cost` on the model alias yourself

Running both is a normal setup: one provider key for burst capacity on the hosted API and one BYO key for a self-hosted NIM, with a routing model failing over between the two aliases. See Routing and Failover.

Current self-hosted LLM NIMs can expose native /v1/completions, /v1/responses, /v1/messages, and /v1/messages/count_tokens endpoints in addition to chat completions. Configuring the endpoint with the openai adapter does not make AISIX forward every route natively. The normalized /v1/responses and /v1/messages routes translate through the chat adapter. Normalized /v1/messages/count_tokens requires an Anthropic-protocol provider key.

Use a passthrough route when the application needs the NIM container's native request and response contract, with the NIM root as the route's target_url.

For native Messages and token counting, another option is a separate BYO provider key and model alias that use adapter: anthropic against the same NIM root. AISIX then sends normalized /v1/messages and /v1/messages/count_tokens calls to the Anthropic-compatible NIM endpoints. Keep this separate from the openai adapter key used for chat, Responses bridging, and embeddings.

Supply NVIDIA-Specific Behavior

Because nvidia has no AISIX-curated adapter mapping, AISIX registers no request or response rewrites for it. The gateway sends the OpenAI request shape as-is, which is correct for NVIDIA's chat route, and leaves every NVIDIA-specific difference to you. Configure the differences with provider-key overrides, which apply to every model that references the key.

Capability support remains model- and endpoint-specific. A catalog entry can support tools, structured output, reasoning, or multimodal input without every other NVIDIA model accepting the same fields or content-block shape. AISIX forwards OpenAI-shaped chat fields and unknown top-level parameters. Some multimodal NIMs instead require a model-specific route, HTML media tags, or NVCF asset references. Check the model's inference reference instead of inferring support from the NIM family name. AISIX also returns only the first choice from an OpenAI-compatible chat response. Do not request n greater than 1 when callers need every generated choice.

Reasoning Controls Are Per Model

NVIDIA does not define one provider-wide reasoning field. Each model NIM publishes its own request schema, so the control that enables or bounds reasoning differs from model to model. For example, nvidia/nvidia-nemotron-nano-9b-v2 toggles reasoning with /think and /no_think control tokens in the prompt, while other NIMs accept a top-level parameter such as reasoning_effort. Confirm the control in the model's own page under NVIDIA's NIM API reference before you rely on it, because a control one NIM accepts can be ignored or rejected by another.

No gateway configuration is needed to deliver these controls. Prompt-level tokens travel inside message content, and AISIX forwards unrecognized top-level chat parameters to the upstream verbatim, so a model-specific parameter reaches NVIDIA unchanged:

{
  "model": "nvidia-nemotron-prod",
  "messages": [
    {
      "role": "user",
      "content": "Plan a three-step migration."
    }
  ],
  "reasoning_effort": "low"
}

On the response side, AISIX preserves reasoning that an upstream already returns in the canonical reasoning_content field, for both streaming and non-streaming responses. If a NIM streams reasoning at a different delta path, set response.reasoning_field on the provider key.

Token Limit Parameter Names

The community default rule registers no param_renames for nvidia, so AISIX delivers max_tokens and max_completion_tokens under whichever name the caller sent. NVIDIA's LLM API documents max_tokens. A client that sends the newer max_completion_tokens therefore has its cap forwarded under a name the upstream may not act on. Add a rename on the provider key when that applies:

{
  "request": {
    "param_renames": {
      "max_completion_tokens": "max_tokens"
    }
  }
}

If a request carries both names, AISIX uses the value from the original caller-facing name.

Route Embeddings to NeMo Retriever Models

NVIDIA publishes NeMo Retriever embedding models in the same catalog, and AISIX dispatches /v1/embeddings through the same openai adapter, so an alias that names an embedding model works on that route.

Several NVIDIA embedding fields need special handling. NVIDIA's asymmetric retrieval families, such as the NV-EmbedQA and E5 models, require an input_type of query or passage, and using the wrong one degrades retrieval accuracy. Current Retriever NIM schemas can also define fields such as modality, embedding_type, and truncate. The AISIX embeddings route builds the upstream body from a closed set of fields: model, input, encoding_format, and dimensions. NVIDIA-specific fields placed in a caller request body therefore do not reach the upstream.

Set it on the provider key instead, with request.default_body_fields:

{
  "request": {
    "default_body_fields": {
      "input_type": "passage"
    }
  }
}

AISIX merges these fields into the outbound body on the embeddings path, not only on chat. Provider-key overrides apply to every model that references the key. Create one provider key with input_type: passage for indexing and another with input_type: query for search. Point the matching model aliases at those keys.

Use the same request.default_body_fields mechanism for another fixed NVIDIA field such as truncate. A value that must vary per request or per input, such as a mixed modality array, requires a passthrough route because default body fields are static provider-key configuration.

Symmetric embedding models that do not take input_type need no provider-key override. See request.default_body_fields for the field definition, and confirm whether your model requires input_type in NVIDIA's embedding API reference.

Endpoint Coverage

The nvidia provider value is accepted on the adapter-dispatched routes and rejected by the routes that carry their own provider allowlist.

Route	Behavior with an NVIDIA alias
`/v1/chat/completions`	Supported on the shared hosted LLM route, including `stream: true`. Model capabilities remain model-specific.
`/v1/completions`	AISIX forwards this route through the OpenAI adapter. Current self-hosted LLM NIMs publish it, but NVIDIA does not document it as a general route for the shared hosted LLM API; use it only when the selected upstream explicitly supports it.
`/v1/responses`	Supported through the Responses bridge over chat, even when a self-hosted NIM publishes a native Responses endpoint. The bridge preserves text and function-call turns but synthesizes a new response; state, hosted tools, reasoning controls, and other fields without a chat equivalent are dropped. Upstream reasoning text is not represented in the synthesized Responses output.
`/v1/messages`	Supported for Anthropic-shaped callers through translation over chat, not as native NIM Messages passthrough. NIM reasoning is not returned as Anthropic `thinking` blocks. Token counting at `/v1/messages/count_tokens` requires an Anthropic-protocol provider key.
`/v1/embeddings`	Supported when the alias names an NVIDIA embedding model. See Route Embeddings to NeMo Retriever Models.
`/v1/images/generations`	Rejected with `400` for an NVIDIA alias. NVIDIA Visual GenAI NIMs can publish a native OpenAI-compatible image route, but the normalized AISIX route accepts only models whose provider is `openai`.
`/v1/rerank`	Rejected with `400`. The route accepts only the `openai`, `cohere`, and `jina` provider values. See the note below.
`/v1/videos`	Rejected with `501 not_implemented`. Visual GenAI NIMs can publish native video generation at `/v1/videos/generations`, but the `nvidia` value is not in the normalized AISIX video route's provider allowlist.
`/v1/audio/*`	Forwarded to OpenAI-compatible audio paths, but the shared hosted LLM API and self-hosted LLM NIM do not publish those routes. Speech NIMs use separate APIs; reach them through a passthrough route targeting the documented root.
`/v1/files`, `/v1/batches`, `/v1/fine_tuning/jobs`	AISIX can dispatch these routes through an OpenAI adapter, but neither the shared hosted API nor current self-hosted LLM NIM publishes the corresponding OpenAI jobs APIs. The upstream therefore rejects the request. NIM model customization or LoRA management is a different API contract.
`/passthrough/nvidia/*`	Available through a configured passthrough route for provider-native routes, with limited gateway normalization.

NVIDIA does publish reranking models, but they are not reachable through /v1/rerank even apart from the provider allowlist. Current self-hosted Retriever NIM documents POST /v1/ranking with a query object and a passages array. Neither the path nor the body matches the normalized route, and hosted catalog reranking can use a different host and path. Reach a reranking model through a passthrough route using the endpoint and body from its NVIDIA reranking API reference.

The /passthrough/nvidia paths on this page assume a passthrough route claiming that prefix with https://integrate.api.nvidia.com/v1 as its target_url and the NVIDIA provider key for credential injection; grant the route name on the caller key's allowed_routes. A route binds one fixed target and credential — the request body's model does not choose credentials, and an AISIX alias is not rewritten to the upstream model ID — so create separate routes for NIMs served from other roots, such as retrieval endpoints under https://ai.api.nvidia.com. The route relays upstream responses, including SSE, incrementally. AISIX detects chat, completions, and Responses envelopes and records supported usage fields. Requests without a recognized carrier field remain opaque: buffered responses record zero tokens, while opaque SSE can still record top-level supported usage fields. Prefer normalized inference routes when you need alias rewriting, token accounting, or AISIX cost estimates. Catalog pricing is metadata for those AISIX estimates and does not represent NVIDIA invoicing; many NVIDIA catalog models currently have zero or missing published cost.

Next Steps

You have now connected AISIX to the NVIDIA NIM hosted API and verified the model alias. Continue with these guides:

Model Aliases: configure routing, retry behavior, or cost metadata for this alias.
Routing and Failover: fail over between the hosted NIM API and a self-hosted NIM.
Bring Your Own Endpoint: connect a NIM microservice you run yourself.
Provider Compatibility: review supported proxy endpoints and provider-specific boundaries.

Prerequisites​

Configure with AISIX Cloud​

Create a Provider Key​

Create a Model​

Create a Caller API Key​

Configure with the Open-Source AISIX Gateway​

Verify the Provider Connection​

Choose Between the Hosted API and a Self-Hosted NIM​

Supply NVIDIA-Specific Behavior​

Reasoning Controls Are Per Model​

Token Limit Parameter Names​

Route Embeddings to NeMo Retriever Models​

Endpoint Coverage​

Next Steps​