Routing and Failover

In this guide, you will create a multi-target model that keeps one caller-facing model alias in front of multiple target models.

A multi-target model does not call a provider directly. It uses a routing block to point to target model aliases, and AISIX chooses one of those targets for each request.

Prerequisites

Before starting, prepare the following:

• A self-hosted gateway with the admin and proxy listeners available.
• The admin key from the gateway config.yaml.
• Two target models that can serve traffic. If you have not created them yet, configure Provider Credentials and Model Aliases first.
• A caller API key for verification. You can create one in Caller API Keys, or create one in this guide.

Choose a Strategy

Choose a strategy based on what the caller-facing alias should do:

• Use failover to keep one primary target with backups.
• Use round_robin to rotate requests across similar targets.
• Use weighted to send more traffic to some targets than others, and add sticky for stable A/B or canary assignment.
• Use least_cost, least_latency, or least_busy to rank targets by a runtime signal, such as price, observed latency, or in-flight load, and attempt the best one first.

You can also narrow the eligible targets per request with routing tags, and let one alias match many upstream models with a wildcard name.

AISIX retries and fails over on retryable upstream failures, such as 5xx responses, request timeouts, and transport errors. Most upstream 4xx responses are treated as caller-side problems and do not trigger failover, except 429 when retry_on_429 is enabled.

Create a Multi-Target Model

Set the admin key and the model aliases used in this guide:

export AISIX_ADMIN_KEY="YOUR_ADMIN_KEY"
export PRIMARY_MODEL="gpt-4o-primary"
export SECONDARY_MODEL="gpt-4o-secondary"
export ROUTING_MODEL="chat-prod"

Create a failover model that starts with the primary target and falls back to the secondary target:

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "'"${ROUTING_MODEL}"'",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model": "'"${PRIMARY_MODEL}"'"},
        {"model": "'"${SECONDARY_MODEL}"'"}
      ],
      "retries": 1,
      "max_fallbacks": 1,
      "retry_on_429": true
    }
  }'

You should see a response similar to the following:

{
  "id": "134c4b01-09e6-41b7-97c7-f4e9a608f4c2",
  "value": {
    "display_name": "chat-prod",
    "routing": {
      "strategy": "failover",
      "targets": [
        {
          "model": "gpt-4o-primary"
        },
        {
          "model": "gpt-4o-secondary"
        }
      ],
      "retries": 1,
      "max_fallbacks": 1,
      "retry_on_429": true
    }
  },
  "revision": 1
}

Copy the highlighted id if you plan to update, inspect, or delete this multi-target model later.

With this configuration, AISIX starts with gpt-4o-primary. If that target has a retryable failure, AISIX can retry it once and then fail over once to gpt-4o-secondary.

Allow Caller Access

The caller API key must be allowed to use the multi-target alias. If you already have a caller API key resource, update its allowlist to include chat-prod.

For a self-hosted check, create a caller API key that can call only chat-prod:

export AISIX_API_KEY="YOUR_CALLER_API_KEY"

AISIX_API_KEY_HASH=$(printf '%s' "${AISIX_API_KEY}" | shasum -a 256 | awk '{print $1}')

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/apikeys" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "key_hash": "'"${AISIX_API_KEY_HASH}"'",
    "allowed_models": ["'"${ROUTING_MODEL}"'"]
  }'

You should see a response similar to the following:

{
  "id": "9b7f01fd-5f26-4657-82ef-605cc2f0ce21",
  "value": {
    "key_hash": "dd08e1fdcc327a5f15dedfba33172b5412b887d9d12ffc1076f77683b1ddbe3e",
    "allowed_models": [
      "chat-prod"
    ]
  },
  "revision": 1
}

Copy the highlighted id if you plan to update, rotate, or delete this caller API key later.

Verify Routing

Send a request to the multi-target alias:

curl -sSi -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"${ROUTING_MODEL}"'",
    "messages": [
      {"role": "user", "content": "Hello from AISIX routing."}
    ]
  }'

A successful request starts with HTTP/1.1 200 OK and includes x-aisix-served-by: gpt-4o-primary. The response body is similar to the following:

{
  "id": "chatcmpl-***",
  "object": "chat.completion",
  "created": **********,
  "model": "chat-prod",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today with AISIX routing?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 13,
    "total_tokens": 26
  }
}

The response body keeps the caller-facing model name. The x-aisix-served-by header shows which target model served the request.

Use this header when you need to confirm which target handled a routed request. It is not present on cache hits, single-target model responses, error responses, or every endpoint family.

Streaming requests can resolve multi-target aliases, but they do not fail over after a stream has started.

Tune Retry and Runtime Behavior

The example uses one retry on the primary target and one fallback to the secondary target.

Adjust the routing fields only when the traffic plan needs different behavior:

• Set max_fallbacks: 0 when you want target selection without cross-target fallback.
• Enable retry_on_429 when upstream rate-limit responses should participate in retry and failover.
• Configure routing.when_all_unavailable: "try_anyway" when attempting a target is preferable to returning 503 all_candidates_unavailable.

For streaming requests, a timeout before the first upstream chunk can fail over to the next target. After streaming bytes have started, a timeout ends the stream instead of failing over mid-response.

Use GET /admin/v1/models/status to inspect runtime status for target models.

For complete model request fields and response shapes, see the Admin API Reference.

Route by Cost, Latency, or Load

The least_cost, least_latency, and least_busy strategies rank every target by a runtime signal. AISIX attempts the best target first, then falls forward through the rest. Order comes from the signal, not from target declaration order.

• least_cost ranks by each target model's combined input and output price per 1K tokens. Targets without a known price rank last.
• least_latency ranks by a moving average of recent upstream latency, using time to first token for streaming. Targets with no samples yet rank first, so each is probed before it is ranked.
• least_busy ranks by the number of in-flight requests currently dispatched to each target.

Set the strategy on the routing block. For example, to always prefer the cheapest healthy target:

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "chat-cheapest",
    "routing": {
      "strategy": "least_cost",
      "targets": [
        {"model": "'"${PRIMARY_MODEL}"'"},
        {"model": "'"${SECONDARY_MODEL}"'"}
      ]
    }
  }'

Where the price comes from depends on how the gateway is managed:

• Console-managed gateways resolve each target's price automatically: pricing overrides set on the console's pricing page take precedence, then the built-in model pricing catalog. Price changes re-rank existing groups without touching the models. Wildcard aliases have no catalog price, so they rank last under least_cost unless you add a pricing override for the alias name.
• Standalone gateways rank by the cost block set directly on each target model through the Admin API. See Model Aliases.

Split Traffic for A/B Tests and Canary Releases

Use weighted with sticky: true to send a fixed share of traffic to a canary target while keeping each caller on one variant. Weights set the split, and sticky makes the choice deterministic per caller, so a session does not flip between variants across requests.

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "chat-canary",
    "routing": {
      "strategy": "weighted",
      "sticky": true,
      "targets": [
        {"model": "gpt-4o-stable", "weight": 95},
        {"model": "gpt-4o-canary", "weight": 5}
      ]
    }
  }'

By default, sticky assignment is keyed by the caller's API key, so each key consistently lands on the same target. To key assignment by session or end user instead, send the x-aisix-routing-key header:

curl -sS -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-key: session-1a2b" \
  -H "Content-Type: application/json" \
  -d '{"model": "chat-canary", "messages": [{"role": "user", "content": "Hello."}]}'

Without sticky, weighted samples the split independently on every request.

Route by Request Tags

Tag targets to route by a per-request signal, such as a team, tier, or environment. Add tags to each target, then send the x-aisix-routing-tags header, comma-separated, on the request.

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "chat-tiered",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model": "gpt-4o-premium", "tags": ["premium"]},
        {"model": "gpt-4o-standard", "tags": ["default"]}
      ]
    }
  }'

A request carrying x-aisix-routing-tags: premium is served only by targets tagged premium. The configured strategy then orders whatever targets remain. When no target matches the request tags, or the request sends no tags, AISIX falls back to targets tagged default. If no target matches and none is tagged default, the request is rejected.

curl -sS -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-tags: premium" \
  -H "Content-Type: application/json" \
  -d '{"model": "chat-tiered", "messages": [{"role": "user", "content": "Hello."}]}'

The routing header is read from the request headers only. It is never forwarded to the upstream provider.

Match Model Names with a Wildcard

A model alias whose display_name contains a single * matches every request whose model name fits the pattern, so one alias can front many upstream models without a separate alias for each. This alias is a direct model, not a multi-target routing block. Set model_name to * to forward the matched portion upstream, or to a fixed value to pin every match to one upstream model.

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "openai/*",
    "provider": "openai",
    "model_name": "*",
    "provider_key_id": "YOUR_PROVIDER_KEY_ID"
  }'

With this alias, a request for openai/gpt-4o is served by the openai provider with upstream model gpt-4o, and openai/o3-mini with o3-mini. An exact alias always wins over a wildcard, and the most specific wildcard wins when several match.

A caller API key grants a wildcard with the same pattern in allowed_models, such as ["openai/*"]. Wildcard aliases do not appear in GET /v1/models, because they are patterns rather than concrete model names.

Next Steps

You have now configured a multi-target model and allowed a caller API key to use it. Continue with Multi-Target Model Failover to test failover end to end.