Routing and Failover

A multi-target model keeps one caller-facing alias in front of several target models. AISIX selects a target for each request, retries eligible failures, and can fail over without requiring the application to change model names.

The multi-target model contains a routing block rather than a provider configuration. Each routing target refers to an existing model alias with its own provider and credentials.

Prerequisites

Before starting, prepare the following:

• A self-hosted gateway with the admin and proxy listeners available.
• The admin key from the gateway config.yaml.
• Two target models that can serve traffic. If you have not created them yet, configure Provider Credentials and Model Aliases first.
• A caller API key for verification. You can create one in Caller API Keys, or create one in this guide.

Choose a Strategy

Choose a strategy based on how the caller-facing alias should distribute traffic:

Goal	Strategy	Selection Behavior
Keep a primary target with ordered backups.	failover	Attempts targets in declaration order.
Rotate traffic across similar targets.	round_robin	Advances through eligible targets in turn.
Control the share of traffic sent to each target.	weighted	Samples targets by weight. Add sticky for stable A/B or canary assignment.
Prefer the lowest estimated price.	least_cost	Ranks eligible targets by configured or managed pricing.
Prefer the fastest recently observed target.	least_latency	Ranks eligible targets by recent upstream latency.
Prefer the target with the fewest active requests.	least_busy	Ranks eligible targets by in-flight load.

You can also narrow the eligible targets per request with routing tags, and let one alias match many upstream models with a wildcard name.

AISIX retries and fails over on retryable upstream failures, such as 5xx responses, request timeouts, and transport errors. Most upstream 4xx responses are returned to the caller. Enable retry_on_429 for upstream rate limits, or configure fallback_on_statuses for other provider-specific transient statuses.

AISIX also skips a target that is over its own model rate limit. It records the skipped target as a failed 429 routing attempt and continues with the remaining targets in strategy order without retrying that target. When every target is over its limit, the request returns 429.

A target whose own allowed_cidrs excludes the caller is likewise not a candidate. That check runs before the strategy selects, so such a target is never attempted and does not consume the max_fallbacks budget. When every target excludes the caller, the request returns 403.

Create a Multi-Target Model

Set the admin key and the model aliases used in this guide:

# Replace with your values
export AISIX_ADMIN_KEY="YOUR_ADMIN_KEY"
export PRIMARY_MODEL="gpt-4o-primary"
export SECONDARY_MODEL="gpt-4o-secondary"
export ROUTING_MODEL="chat-prod"

Create a failover model that starts with the primary target and falls back to the secondary target:

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "display_name": "${ROUTING_MODEL}",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model": "${PRIMARY_MODEL}"},
        {"model": "${SECONDARY_MODEL}"}
      ],
      "retries": 1,
      "max_fallbacks": 1,
      "retry_on_429": true
    }
  }
EOF

You should see a response similar to the following:

{
  "id": "134c4b01-09e6-41b7-97c7-f4e9a608f4c2",
  "value": {
    "display_name": "chat-prod",
    "routing": {
      "strategy": "failover",
      "targets": [
        {
          "model": "gpt-4o-primary"
        },
        {
          "model": "gpt-4o-secondary"
        }
      ],
      "retries": 1,
      "max_fallbacks": 1,
      "retry_on_429": true
    }
  },
  "revision": 1
}

Save the highlighted id to update, inspect, or delete this multi-target model later.

With this configuration, AISIX starts with gpt-4o-primary. If that target has a retryable failure, AISIX can retry it once and then fail over once to gpt-4o-secondary.

Allow Caller Access

The caller API key must be allowed to use the multi-target alias. If you already have a caller API key resource, update its allowlist to include chat-prod.

For a self-hosted check, create a caller API key that can call only chat-prod:

# Replace with your values
export AISIX_API_KEY="YOUR_CALLER_API_KEY"

AISIX_API_KEY_HASH=$(printf '%s' "${AISIX_API_KEY}" | shasum -a 256 | awk '{print $1}')

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/apikeys" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "key_hash": "${AISIX_API_KEY_HASH}",
    "allowed_models": ["${ROUTING_MODEL}"]
  }
EOF

You should see a response similar to the following:

{
  "id": "9b7f01fd-5f26-4657-82ef-605cc2f0ce21",
  "value": {
    "key_hash": "dd08e1fdcc327a5f15dedfba33172b5412b887d9d12ffc1076f77683b1ddbe3e",
    "allowed_models": [
      "chat-prod"
    ]
  },
  "revision": 1
}

Save the highlighted id to update, rotate, or delete this caller API key later.

Verify Routing

Send a request to the multi-target alias:

curl -sSi -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "model": "${ROUTING_MODEL}",
    "messages": [
      {"role": "user", "content": "Hello from AISIX routing."}
    ]
  }
EOF

A successful request starts with HTTP/1.1 200 OK and includes x-aisix-served-by: gpt-4o-primary. The response body is similar to the following:

{
  "id": "chatcmpl-***",
  "object": "chat.completion",
  "created": **********,
  "model": "chat-prod",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today with AISIX routing?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 13,
    "total_tokens": 26
  }
}

The response body retains the caller-facing model name. The x-aisix-served-by header identifies the target model that served the request. This header is not present on cache hits, single-target model responses, error responses, or every endpoint family.

Streaming requests can resolve multi-target aliases, but they do not fail over after a stream has started.

Tune Retry and Runtime Behavior

The example retries the primary target once and can fall back once to the secondary target. Adjust these controls according to the expected failure mode:

Field	Use When
retries	A request should repeat the same target after a retryable failure, with increasingly longer delays between attempts.
max_fallbacks	The request can move to another eligible target. Set it to 0 for target selection without cross-target fallback.
retry_on_429	An upstream 429 response should be eligible for another attempt.
fallback_on_statuses	A provider uses another status for a transient condition that should be eligible for another attempt.
when_all_unavailable	AISIX should control what happens after runtime filtering removes every target. Use "try_anyway" when attempting a target is preferable to returning 503 all_candidates_unavailable.

Chat Completions, Messages, and Responses all honor retries for streaming and non-streaming requests. For a streaming request, both the same-target retries and any failover happen while AISIX is still establishing the upstream stream, so they are invisible to the caller; once response bytes are committed, a failure ends the response. Count Tokens can fail over across targets but does not perform same-target retries.

retries is read from the routing block, so it applies to multi-target models only. A model alias that points straight at a provider is attempted once.

Choose Additional Failover Statuses

Use fallback_on_statuses only for statuses that the provider uses for transient conditions such as model overload, queue saturation, or quota exhaustion. Configure it in the multi-target model's routing block:

{
  "routing": {
    "fallback_on_statuses": [408, 409]
  }
}

Entries must be between 400 and 599. 5xx responses already fail over, so listing them does not change behavior. Avoid authentication (401 and 403) and validation (400) statuses unless the provider is known to use them for a transient failure. Retrying a caller or credential error sends the same failing request to more targets.

fallback_on_statuses affects only the current request. The direct model's cooldown.trigger_statuses setting controls whether repeated failures remove a target from rotation for future requests. For example, listing 422 in fallback_on_statuses can move the current request to another target, but does not place the failed target in cooldown.

Handle Streaming Failures

For streaming Chat Completions, a retryable connection failure can move the request to the next target before AISIX commits response bytes. When a streaming deadline is set through stream_timeout or its timeout fallback, AISIX also waits for the first upstream chunk. An error, empty stream, or timeout at that point can fail over. After streaming bytes have started, a timeout ends the stream instead of failing over mid-response.

Use GET /admin/v1/models/status to inspect the runtime status of target models.

For complete model request fields and response shapes, see the Admin API Reference.

Route by Cost, Latency, or Load

The least_cost, least_latency, and least_busy strategies rank every target by a runtime signal. AISIX attempts the best-ranked target first, then moves through the remaining targets. The signal determines the order rather than the target declaration order.

• least_cost ranks by each target model's combined input and output price per 1K tokens. Targets without a known price rank last.
• least_latency ranks by a moving average of recent upstream latency, using time to first token for streaming. Targets with no samples yet rank first, so each is probed before it is ranked.
• least_busy ranks by the number of in-flight requests currently dispatched to each target.

Set the strategy on the routing block. For example, to always prefer the cheapest healthy target:

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "display_name": "chat-cheapest",
    "routing": {
      "strategy": "least_cost",
      "targets": [
        {"model": "${PRIMARY_MODEL}"},
        {"model": "${SECONDARY_MODEL}"}
      ]
    }
  }
EOF

The pricing source depends on how the gateway is managed:

• Gateways connected to the managed control plane project each target's routing price from its exact configured provider and upstream model_name. Organization overrides configured in Model Pricing take precedence over catalog prices, and price changes re-rank existing groups without requiring model configuration changes. The managed control plane rejects wildcard direct models as routing targets, so managed routing groups must target models with exact display and upstream model names.
• Self-hosted gateways rank by the cost block configured on each target model through the Admin API. See Model Aliases.

Split Traffic for A/B Tests and Canary Releases

Use weighted with sticky: true to send a fixed share of traffic to a canary target while keeping each caller on one variant. Weights set the split. Sticky assignment makes the choice deterministic per caller so a session does not move between variants across requests.

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "chat-canary",
    "routing": {
      "strategy": "weighted",
      "sticky": true,
      "targets": [
        {"model": "gpt-4o-stable", "weight": 95},
        {"model": "gpt-4o-canary", "weight": 5}
      ]
    }
  }'

By default, sticky assignment is keyed by the caller's API key, so each key consistently lands on the same target. To key assignment by session or end user instead, send the x-aisix-routing-key header:

curl -sS -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-key: session-1a2b" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-canary",
    "messages": [
      {"role": "user", "content": "Hello."}
    ]
  }'

Without sticky, weighted samples the split independently on every request.

Route by Request Tags

Tag targets to route by a request-specific signal such as a team, tier, or environment. Add tags to each target, then send the comma-separated x-aisix-routing-tags header on the request.

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "chat-tiered",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model": "gpt-4o-premium", "tags": ["premium"]},
        {"model": "gpt-4o-standard", "tags": ["default"]}
      ]
    }
  }'

A request carrying x-aisix-routing-tags: premium is served only by targets tagged premium. The configured strategy then orders whatever targets remain. When the request has no routing tags, AISIX uses targets tagged default when present; otherwise all targets remain eligible. When a request supplies tags but none match, AISIX tries default targets and rejects the request only if none exist.

curl -sS -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-tags: premium" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-tiered",
    "messages": [
      {"role": "user", "content": "Hello."}
    ]
  }'

The routing header is read from the request headers only. It is never forwarded to the upstream provider.

Match Model Names with a Wildcard

A model alias whose display_name contains one * matches every requested model name that fits the pattern. One alias can therefore front many upstream models without a separate resource for each name.

A wildcard alias is a direct model, not a multi-target routing block. Set model_name to * to forward the matched portion upstream, or use a fixed value to send every match to one upstream model.

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/models" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "openai/*",
    "provider": "openai",
    "model_name": "*",
    "provider_key_id": "YOUR_PROVIDER_KEY_ID"
  }'

With this alias, a request for openai/gpt-4o is served by the openai provider with upstream model gpt-4o, and openai/o3-mini with o3-mini. An exact alias always wins over a wildcard, and the most specific wildcard wins when several match.

A caller API key grants a wildcard with the same pattern in allowed_models, such as ["openai/*"]. Wildcard aliases do not appear in GET /v1/models, because they are patterns rather than concrete model names.

Next Steps

Continue with Multi-Target Model Failover to test failover end to end. Use Proxy Errors and Retries to design application retry behavior around gateway and upstream failures.