Skip to main content
API7.ai Logo

Ollama

Ollama runs language models locally or in private infrastructure and exposes an OpenAI-compatible API. AISIX adds caller keys, stable model aliases, traffic controls, and usage reporting while Ollama continues to serve inference in your environment.

Prerequisites​

Before starting, prepare the following:

  • One AISIX setup:
    • For AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
    • For the open-source AISIX gateway, prepare either a local AISIX installation or the Docker setup from the Open-Source AISIX Gateway Quickstart. Configure the gateway to load a declarative resources file.
  • Ollama installed on a host reachable from the AISIX gateway.
  • curl and jq.

Prepare Ollama​

Pull the model used in this guide:

ollama pull gpt-oss:20b

Ollama binds to 127.0.0.1:11434 by default. If AISIX runs in another container or host, configure a reachable bind address by following the Ollama server configuration. For example, a foreground server can listen on all interfaces:

OLLAMA_HOST="0.0.0.0:11434" ollama serve
caution

The local Ollama API does not require authentication. Binding it to 0.0.0.0 makes it reachable from other network peers. Restrict the listening network with firewall, container-network, or Kubernetes policy controls, and do not expose the port directly to the public Internet.

Export an API base reachable from the AISIX gateway:

# Docker Desktop gateway to Ollama on the host
export OLLAMA_API_BASE="http://host.docker.internal:11434/v1"

Choose the address for your topology:

AISIX gateway and Ollama topologyExample API base
Both processes on one hosthttp://127.0.0.1:11434/v1
AISIX in Docker Desktop, Ollama on the hosthttp://host.docker.internal:11434/v1
Both containers on one Docker networkhttp://ollama:11434/v1
Kuberneteshttp://ollama.<namespace>.svc.cluster.local:11434/v1

On Linux, host.docker.internal may require an explicit host-gateway mapping. A successful request from your laptop does not prove that the AISIX gateway container can reach the same address.

Configure with AISIX Cloud​

Export the AISIX Cloud connection details:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Ollama is a private endpoint rather than an AISIX catalog provider. Configure it with the byo provider value and select the openai adapter explicitly.

Create a Provider Key​

PROVIDER_KEY_ID=$(
  curl -sS -X POST "$AISIX_CP/provider_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "ollama-local",
      "provider": "byo",
      "adapter": "openai",
      "api_key": "ollama",
      "api_base": "'"${OLLAMA_API_BASE}"'",
      "allowed_environments": ["'"${ENV_ID}"'"]
    }' | jq -er '.provider_key.id'
)

echo "$PROVIDER_KEY_ID"

AISIX requires a non-empty api_key in the provider-key schema. Ollama's OpenAI client examples use ollama because the client requires a value, but the local Ollama server ignores it. AISIX sends the placeholder as a bearer token; it is not a security control.

A BYO key requires provider: "byo", a non-empty api_key, and api_base. This guide sets adapter: "openai" explicitly; when omitted, AISIX defaults a BYO key to the OpenAI-compatible adapter.

Create a Model​

Use the exact local Ollama model tag:

MODEL_ID=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "ollama-gpt-oss-prod",
      "model_name": "gpt-oss:20b",
      "provider_key_id": "'"${PROVIDER_KEY_ID}"'"
    }' | jq -er '.model.id'
)

echo "$MODEL_ID"

Run ollama ls to see installed model tags. The tag, including a suffix such as :20b, is sent upstream unchanged.

Create a Caller API Key​

AISIX_API_KEY=$(
  curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
    -H "Authorization: Bearer $AISIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "display_name": "ollama-caller",
      "allowed_models": ["'"${MODEL_ID}"'"]
    }' | jq -er '.plaintext'
)

echo "$AISIX_API_KEY"

Configure with the Open-Source AISIX Gateway​

Export the upstream credential and choose the caller API key that applications will send to the gateway:

export OLLAMA_API_BASE="http://host.docker.internal:11434/v1"
export CALLER_API_KEY="YOUR_CALLER_API_KEY"

Create a complete declarative resources file for this provider:

resources.yaml
_format_version: "1"

provider_keys:
  - display_name: "ollama-local"
    provider: "ollama"
    adapter: "openai"
    api_key: "ollama"
    api_base: "${OLLAMA_API_BASE}"

models:
  - display_name: "ollama-gpt-oss-prod"
    provider: "ollama"
    model_name: "gpt-oss:20b"
    provider_key: "ollama-local"

api_keys:
  - display_name: "ollama-caller"
    key_env: CALLER_API_KEY
    allowed_models:
      - "ollama-gpt-oss-prod"

If AISIX is installed locally, validate the file before loading it:

aisix validate --resources resources.yaml

After validation, start the gateway with the referenced environment variables in its process environment. Reload an existing gateway only if those variables are already available to the process; otherwise, restart it with the updated environment.

If you use Docker, adapt the validation and startup commands in the Open-Source AISIX Gateway Quickstart. Mount this resources.yaml file and pass every environment variable it references with -e in both commands. After the resources load, prepare the shared verification request below:

export AISIX_API_KEY="$CALLER_API_KEY"

Verify the Provider Connection​

Export the AISIX gateway origin:

# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_ORIGIN"

Send a chat-completions request through the AISIX proxy:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama-gpt-oss-prod",
    "messages": [
      {
        "role": "user",
        "content": "Say hello from Ollama."
      }
    ]
  }'

Ollama should log POST /v1/chat/completions, and AISIX should return an OpenAI-compatible response.

Endpoint Coverage​

Ollama documents OpenAI-compatible chat completions, completions, Responses, models, and embeddings routes. It also publishes an Anthropic-compatible Messages route. AISIX behavior depends on the adapter assigned to the BYO provider key; the openai adapter in this guide does not make every Ollama-native route a transparent passthrough.

RouteBehavior with the Ollama alias in this guide
/v1/chat/completionsSupported, including streaming, tools, structured output, vision, and reasoning controls when the installed model supports them.
/v1/completionsSupported through the OpenAI adapter. Ollama accepts only a string in prompt.
/v1/embeddingsSupported when the alias names an installed embedding model. Ollama and AISIX accept a string or array of strings.
/v1/responsesSupported through the Responses bridge over chat, not through Ollama's native Responses route. The bridge preserves text and function-call turns but drops state, reasoning controls, reasoning output, and fields without a chat equivalent.
/v1/messagesTranslated through chat because this provider key uses adapter: openai. See the native Messages option below.
/v1/messages/count_tokensRejected for the openai adapter. Ollama does not currently implement this endpoint even when a separate Anthropic adapter is used.
/v1/modelsReturns caller-accessible AISIX model aliases, not the models installed in Ollama. Use ollama ls or a passthrough route to query the Ollama inventory.
/v1/images/generations, /v1/rerankRejected with 400 because the normalized routes do not accept this guide's provider label (byo in AISIX Cloud or ollama in the resources file).
/v1/videosRejected with 501 not_implemented because neither provider label is in the video provider allowlist.
/v1/audio/*, /v1/files, /v1/batches, /v1/fine_tuning/jobsAISIX can forward these OpenAI-shaped routes, but Ollama does not publish the corresponding APIs. The upstream rejects the request.
/passthrough/byo/*Available through a configured passthrough route for Ollama-native routes. The conventional prefix is /passthrough/byo in AISIX Cloud and /passthrough/ollama with an open-source resources file, matching this page's provider label.

Ollama added native /v1/responses in version 0.13.3. It supports streaming, function tools, and reasoning summaries, but not state through previous_response_id or conversation. AISIX still bridges a normalized /v1/responses call for this BYO alias because the provider is not openai. Use a passthrough route when the application needs Ollama's native Responses contract.

For native Anthropic-shaped Messages, create a second BYO provider key and model alias that use adapter: anthropic against the same Ollama root. AISIX then sends normalized /v1/messages calls to Ollama's native Messages endpoint. Current Ollama limitations still apply: token counting, forced tool_choice, prompt caching, citations, and Messages batches are not supported, and extended-thinking budgets are accepted but not enforced.

Ollama's OpenAI-compatible chat route does not currently support tool_choice, logit_bias, user, or n. AISIX can forward these fields, but forwarding does not add upstream support. For vision models, send a base64 image in an image_url content part; Ollama does not support a remote image URL on this route.

The /passthrough paths above assume a passthrough route claiming the chosen prefix with the Ollama root as its target_url; grant the route name on the caller key's allowed_routes. Passthrough does not rewrite an AISIX alias in the request body, and a route relays to its fixed target with its bound provider key regardless of the requested model. It relays upstream responses incrementally, including SSE. AISIX detects chat, completions, and Responses envelopes and records supported usage fields. Requests without a recognized carrier field remain opaque: buffered responses record zero tokens, while opaque SSE can still record top-level supported usage fields. Prefer the normalized routes when you need alias rewriting, token accounting, or AISIX cost estimates. Because Ollama is a BYO endpoint without catalog pricing, configure Model Pricing in AISIX Cloud or cost metadata in an open-source model resource when you need cost estimates.

Troubleshooting​

SymptomCheck
Connection refused or timeoutTest OLLAMA_API_BASE from the AISIX gateway container, not only from the host.
Ollama listens only on 127.0.0.1Set OLLAMA_HOST through the supported service configuration and restart Ollama.
Model not foundRun ollama pull gpt-oss:20b and verify the tag with ollama ls.
Provider key creation returns 400Include provider: "byo", adapter: "openai", a non-empty api_key, and api_base.

Next Steps​

You have now connected AISIX to Ollama and verified the model alias. Continue with these guides: