Skip to main content
API7.ai Logo

NVIDIA NIM

Connect AISIX AI Gateway to the NVIDIA NIM hosted API so applications can call NVIDIA-hosted models through the gateway's OpenAI-compatible API. AISIX keeps the NVIDIA credential on the gateway side. It authorizes model access with caller API keys and allowlists, and applies the same rate-limit and usage-accounting controls as other model aliases.

NVIDIA serves models from many publishers on one hosted endpoint, including NVIDIA Nemotron, Meta Llama, OpenAI gpt-oss, DeepSeek, Qwen, Google Gemma, Mistral AI, and Microsoft models. The endpoint accepts OpenAI chat-completions requests, so nvidia uses the openai adapter with bearer authentication and an NVIDIA api_base.

note

nvidia is a community catalog entry rather than a curated one. AISIX sources the entry from the models.dev catalog, and the dashboard groups it under All providers (community), described there as sourced from models.dev with wire compatibility assumed to be OpenAI-shaped unless the operator overrides it per key. The gateway registers no NVIDIA-specific request or response rewrites, so anything NVIDIA names differently from the OpenAI shape is yours to configure. See Supply NVIDIA-Specific Behavior.

Prerequisites​

The following examples configure the upstream through the AISIX Cloud Admin API. Before configuring the upstream, prepare the following:

  • Access to AISIX Cloud with an environment, an attached gateway, and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
  • An NVIDIA API key for the hosted NIM API, generated from a model page on build.nvidia.com.

Export the connection details used by every request below:

# Replace with your values
export AISIX_CP="http://localhost:8080/api"
export AISIX_TOKEN="aisix_pat_YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Configure the NVIDIA NIM Upstream​

Create a provider key, model alias, and caller API key for the NVIDIA-backed chat-completions route.

Create a Provider Key​

Create the provider key that stores the NVIDIA credential and API root:

# Replace with your value
export NVIDIA_API_KEY="YOUR_PROVIDER_API_KEY"

PROVIDER_KEY_ID=$(curl -sS -X POST "$AISIX_CP/provider_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "nvidia-prod",
    "provider": "nvidia",
    "api_key": "'"${NVIDIA_API_KEY}"'",
    "api_base": "https://integrate.api.nvidia.com/v1",
    "allowed_environments": ["'"${ENV_ID}"'"]
  }' | jq -r '.provider_key.id')

echo "$PROVIDER_KEY_ID"

❶ provider is nvidia. The Cloud Admin API accepts the value because nvidia is one of the models.dev catalog IDs it caches, and it assigns the openai adapter and bearer authentication from the catalog's community default rule. The adapter field is only accepted on BYO provider keys, so do not set it here.

❷ api_key stores the NVIDIA API key. NVIDIA authenticates the hosted NIM API with HTTP bearer authentication, which is what the openai adapter already sends. The value follows the credential-handling behavior in Provider Credentials.

❸ api_base is https://integrate.api.nvidia.com/v1, the root NVIDIA documents as the base_url for OpenAI client libraries against the hosted NIM API. The chat route hangs off that root as POST https://integrate.api.nvidia.com/v1/chat/completions, and AISIX appends /chat/completions to api_base, so the value must stop at /v1. AISIX strips a pasted endpoint suffix such as /chat/completions and any trailing slash, but treat the root as the contract rather than relying on that repair.

For nvidia the field is optional: models.dev publishes this same URL in its api field, and the Cloud Admin API fills it in when you omit it. The examples set it explicitly so the root each key targets stays visible in the configuration.

caution

Never leave an nvidia provider key without a resolved api_base. The OpenAI-family bridge refuses to fall back to the default OpenAI host for any non-OpenAI vendor, so an empty base URL produces an upstream configuration error at request time instead of a misdirected request that would carry your NVIDIA credential to another vendor's host.

The command captures the returned provider key ID in PROVIDER_KEY_ID.

Create a Model​

NVIDIA namespaces every hosted model ID by publisher organization, in the form <publisher>/<model>. The publisher segment is part of the ID and is never optional. Current examples include the following:

Model IDPublisher
nvidia/nvidia-nemotron-nano-9b-v2NVIDIA
meta/llama-3.3-70b-instructMeta
openai/gpt-oss-120bOpenAI

Two naming details cause most alias mistakes:

  • Some NVIDIA-published model names already begin with nvidia-, so the full ID repeats the segment, as in nvidia/nvidia-nemotron-nano-9b-v2. That is correct, not a typographical error.
  • A model page URL on build.nvidia.com is a page slug, not the model ID. The page for Llama 3.3 70B Instruct uses llama-3_3-70b-instruct in its path while the API model ID is meta/llama-3.3-70b-instruct. Copy the ID from the code sample on the model page rather than from the address bar.

Check the model's page under NVIDIA's NIM API reference for the current ID before you create an alias.

Create the model alias callers will send in requests:

MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "nvidia-llama-prod",
    "model_name": "meta/llama-3.3-70b-instruct",
    "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
  }' | jq -r '.model.id')

echo "$MODEL_ID"

❶ display_name is the alias callers send in model.

❷ model_name is the NVIDIA model ID, including the publisher segment. Do not carry over a bare identifier such as llama-3.3-70b-instruct from a provider that serves the same weights without a publisher prefix.

❸ provider_key_id attaches the alias to the NVIDIA provider key.

Create a Caller API Key​

Create the caller API key that can access the model alias. The plaintext key is server-generated and returned once in the create response, so capture it now:

AISIX_API_KEY=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "nvidia-caller",
    "allowed_models": ["'"${MODEL_ID}"'"]
  }' | jq -r '.plaintext')

echo "$AISIX_API_KEY"

The allowed_models value must reference the model ID captured in the previous step. After the write, the configuration projects to attached gateways automatically.

Verify the Provider Connection​

Send a chat-completions request through the AISIX proxy:

curl -sS -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia-llama-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from NVIDIA NIM."
      }
    ]
  }'

The gateway returns an OpenAI-compatible response that echoes the caller-facing alias nvidia-llama-prod. If the request fails, check the provider key api_key, the api_base root, and the publisher-namespaced model ID in model_name. An upstream 404 on a request that otherwise looks correct is almost always a missing or misspelled publisher segment.

Choose Between the Hosted API and a Self-Hosted NIM​

NIM is both a hosted API and a container you can run yourself, and only the hosted API is this catalog provider. A NIM microservice running in your own cluster serves an OpenAI-compatible route at its own address, such as http://10.0.0.5:8000/v1, and reaches AISIX through the private-endpoint path instead. Configure that with Bring Your Own Endpoint.

AspectHosted NIM APISelf-hosted NIM microservice
Provider valuenvidiabyo on the Cloud Admin API, or your own label in a declarative resources.yaml
adapter fieldRejected. The adapter is derived from the catalog.Accepted, and set to openai.
api_basehttps://integrate.api.nvidia.com/v1, defaulted from the catalog when omittedRequired. Your container root, such as http://10.0.0.5:8000/v1.
Model IDPublisher-namespaced, such as meta/llama-3.3-70b-instructThe model name the container serves
Pricing metadataCarried by the models.dev catalog entryYou supply cost on the model alias yourself

Running both is a normal setup: one provider key for burst capacity on the hosted API and one BYO key for a self-hosted NIM, with a routing model failing over between the two aliases. See Routing and Failover.

Supply NVIDIA-Specific Behavior​

Because nvidia has no curated catalog entry, AISIX registers no request or response rewrites for it. The gateway sends the OpenAI request shape as-is, which is correct for NVIDIA's chat route, and leaves every NVIDIA-specific difference to you. Configure the differences with provider-key overrides, which apply to every model that references the key.

Reasoning Controls Are Per Model​

NVIDIA does not define one provider-wide reasoning field. Each model NIM publishes its own request schema, so the control that enables or bounds reasoning differs from model to model. For example, nvidia/nvidia-nemotron-nano-9b-v2 toggles reasoning with /think and /no_think control tokens in the prompt, while other NIMs accept a top-level parameter such as reasoning_effort. Confirm the control in the model's own page under NVIDIA's NIM API reference before you rely on it, because a control one NIM accepts can be ignored or rejected by another.

No gateway configuration is needed to deliver these controls. Prompt-level tokens travel inside message content, and AISIX forwards unrecognized top-level chat parameters to the upstream verbatim, so a model-specific parameter reaches NVIDIA unchanged:

{
  "model": "nvidia-nemotron-prod",
  "messages": [
    {
      "role": "user",
      "content": "Plan a three-step migration."
    }
  ],
  "reasoning_effort": "low"
}

On the response side, AISIX preserves reasoning that an upstream already returns in the canonical reasoning_content field, for both streaming and non-streaming responses. If a NIM streams reasoning at a different delta path, set response.reasoning_field on the provider key.

Token Limit Parameter Names​

The community default rule registers no param_renames for nvidia, so AISIX delivers max_tokens and max_completion_tokens under whichever name the caller sent. NVIDIA's LLM API documents max_tokens. A client that sends the newer max_completion_tokens therefore has its cap forwarded under a name the upstream may not act on. Add a rename on the provider key when that applies:

{
  "request": {
    "param_renames": {
      "max_completion_tokens": "max_tokens"
    }
  }
}

If a request carries both names, AISIX uses the value from the original caller-facing name.

Route Embeddings to NeMo Retriever Models​

NVIDIA publishes NeMo Retriever embedding models in the same catalog, and AISIX dispatches /v1/embeddings through the same openai adapter, so an alias that names an embedding model works on that route.

One NVIDIA detail needs configuration. NVIDIA's asymmetric retrieval families, such as the NV-EmbedQA and E5 models, require an input_type of query or passage, and using the wrong one degrades retrieval accuracy. input_type is not an OpenAI embeddings parameter, and the AISIX embeddings route builds the upstream body from a closed set of fields — model, input, encoding_format, and dimensions — so an input_type placed in a caller request body never reaches NVIDIA.

Set it on the provider key instead, with request.default_body_fields:

{
  "request": {
    "default_body_fields": {
      "input_type": "passage"
    }
  }
}

AISIX merges these fields into the outbound body on the embeddings path, not only on chat. Because provider-key overrides apply to every model that references the key, create one provider key per retrieval mode — one carrying input_type: passage for indexing and one carrying input_type: query for search — and point the matching model aliases at them.

NVIDIA also publishes a second way to express the same thing, aimed exactly at OpenAI clients that cannot send input_type: append a -query or -passage suffix to the model name, as in nvidia/nv-embedqa-e5-v5-passage. That form needs no provider-key override at all, so both retrieval modes can share one provider key and differ only in the model_name on each alias. Prefer it when NVIDIA documents the suffix for your model, and fall back to default_body_fields when it does not. Symmetric embedding models that do not take input_type need neither. See request.default_body_fields for the field definition, and confirm which form your model accepts in NVIDIA's embedding API reference.

Endpoint Coverage​

The nvidia provider value is accepted on the adapter-dispatched routes and rejected by the routes that carry their own provider allowlist.

RouteBehavior with an NVIDIA alias
/v1/chat/completionsSupported, including stream: true.
/v1/responsesSupported through the Responses bridge over the chat adapter path.
/v1/messagesSupported for Anthropic-shaped callers through translation. Token counting at /v1/messages/count_tokens requires an Anthropic-backed model.
/v1/embeddingsSupported when the alias names an NVIDIA embedding model. See Route Embeddings to NeMo Retriever Models.
/v1/images/generationsRejected with 400. The route accepts only models whose provider is openai.
/v1/rerankRejected with 400. The route accepts only the openai, cohere, and jina provider values. See the note below.
/v1/videosRejected with 501 not_implemented. The nvidia value is not in the video route's provider allowlist.
/passthrough/nvidia/*Supported for provider-native routes, with limited gateway normalization. See Provider Passthrough.

NVIDIA does publish reranking models, but they are not reachable through /v1/rerank even apart from the provider allowlist: NVIDIA documents reranking as POST /v1/ranking with a query object and a passages array, which is neither the path nor the body that route normalizes. Reach a reranking model through provider passthrough using the path and body from NVIDIA's reranking API reference. Passthrough forwards to the selected model's provider key base URL, so if the retrieval route you need is served on a different host than the chat base, create a second provider key for that host and read Provider Passthrough on how AISIX selects among several keys of the same provider.

Next Steps​

You have now connected AISIX to the NVIDIA NIM hosted API and verified the model alias. Continue with these guides: