Multi-Target Routing and Failover

A multi-target model keeps one caller-facing alias in front of several target models. AISIX selects a target for each request, retries eligible failures, and can fail over without requiring the application to change model names.

The alias is a caller-facing model name, so it appears in GET /v1/models for every caller API key allowed to request it. Scope a key to the alias alone to publish it as the only entry point callers discover, and change its targets without touching the application.

The multi-target model contains a routing block rather than its own provider configuration. Each target is an existing direct model with its own provider key and upstream model name. AISIX Cloud references these targets by model ID, while the open-source AISIX gateway references them by display_name in the declarative resources file.

The routing model selects a direct target, and that target uses its provider key to call the upstream model:

Choose a Strategy

Choose a strategy based on how the caller-facing alias should distribute traffic:

Goal	Strategy	Selection Behavior
Keep a primary target with ordered backups.	`failover`	Attempts targets in declaration order.
Rotate traffic across similar targets.	`round_robin`	Advances through eligible targets in turn.
Control the share of traffic sent to each target.	`weighted`	Samples targets by weight. Add `sticky` for stable A/B or canary assignment.
Prefer the lowest estimated price.	`least_cost`	Ranks eligible targets by configured prices or AISIX Cloud catalog pricing.
Prefer the fastest recently observed target.	`least_latency`	Ranks eligible targets by recent upstream latency.
Prefer the target with the fewest active requests.	`least_busy`	Ranks eligible targets by in-flight load.

failover is the default when strategy is omitted. You can also narrow the eligible targets per request with routing tags.

AISIX retries and fails over on retryable upstream failures, such as 5xx responses, request timeouts, and transport errors. Most upstream 4xx responses are returned to the caller. Enable retry_on_429 for upstream rate limits, or configure fallback_on_statuses for other provider-specific transient statuses.

AISIX also skips a target that is over its own model rate limit. It records the skipped target as a failed 429 routing attempt and continues with the remaining targets in strategy order without retrying that target. When every target is over its limit, the request returns 429.

A target whose own allowed_cidrs excludes the caller is likewise not a candidate. That check runs before the strategy selects, so such a target is never attempted and does not consume the max_fallbacks budget. When every target excludes the caller, the request returns 403.

Configure and Test a Failover Model

The following example builds and tests a two-target model that keeps gpt-4o-primary as its preferred target and uses gpt-4o-secondary after an eligible failure. Instructions are provided for AISIX Cloud and the open-source AISIX gateway.

Prerequisites

Before starting, prepare the following:

Two or more direct models to use as routing targets. To run the failure test, the primary and secondary models must use separate provider-key resources, although both resources can contain the same upstream credential.
A caller API key that can call the multi-target model, or permission to create one.
cURL and jq to run the API and verification commands.
For AISIX Cloud, an environment with an attached gateway and permission to manage models and caller API keys.
For the open-source AISIX gateway, access to the declarative resources file and the gateway process.

Create the Model

Create the direct target models before the multi-target model. The examples below configure the same failover behavior through each management path: AISIX first uses gpt-4o-primary, then moves to gpt-4o-secondary after a retryable failure.

AISIX Cloud

Set the control-plane URL, admin token, and environment ID for your AISIX Cloud organization:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash.
# The local On-Premises quickstart uses http://localhost:8080/api.
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Export the IDs of the direct target models:

export PRIMARY_MODEL_ID="YOUR_PRIMARY_MODEL_ID"
export SECONDARY_MODEL_ID="YOUR_SECONDARY_MODEL_ID"

Create a failover model that starts with the primary target and falls back to the secondary target:

ROUTING_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-prod",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'"},
        {"model_id": "'"$SECONDARY_MODEL_ID"'"}
      ],
      "retries": 1,
      "max_fallbacks": 1,
      "retry_on_429": true
    }
  }' | jq -r '.model.id')

With this configuration, AISIX starts with gpt-4o-primary. If that target has a retryable failure, AISIX can retry it once and then fail over once to gpt-4o-secondary.

Use the captured ROUTING_MODEL_ID to allow caller access, and to update, inspect, or delete the model later. If an existing caller API key should use chat-prod, add this ID to its allowed_models list.

For a quick check, create a caller API key that can call only chat-prod:

KEY_RESPONSE=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "routing-guide-caller",
    "allowed_models": ["'"$ROUTING_MODEL_ID"'"]
  }')

export AISIX_API_KEY=$(echo "$KEY_RESPONSE" | jq -r '.plaintext')
API_KEY_ID=$(echo "$KEY_RESPONSE" | jq -r '.api_key.id')

The plaintext caller key is returned only once. The commands save it for the verification request and save the key ID for later allowlist updates. The model and caller key configurations are projected to attached gateways automatically.

Open-Source AISIX Gateway

Declare separate provider keys for the two direct targets, then add the multi-target model. The provider keys can use the same upstream credential; keeping them separate lets you test one target without changing the other. Targets reference direct models by display_name:

resources.yaml
_format_version: "1"

provider_keys:
  - display_name: openai-primary
    provider: openai
    adapter: openai
    api_key: ${OPENAI_API_KEY}
  - display_name: openai-secondary
    provider: openai
    adapter: openai
    api_key: ${OPENAI_API_KEY}

models:
  - display_name: gpt-4o-primary
    provider: openai
    model_name: gpt-4o
    provider_key: openai-primary
  - display_name: gpt-4o-secondary
    provider: openai
    model_name: gpt-4o-mini
    provider_key: openai-secondary
  - display_name: chat-prod
    routing:
      strategy: failover
      targets:
        - model: gpt-4o-primary
        - model: gpt-4o-secondary
      retries: 1
      max_fallbacks: 1
      retry_on_429: true

The direct models reference provider keys that must also exist in the complete resources file. Add chat-prod to the caller key's allowed_models, then validate and reload the file.

Verify Primary Routing

Export the gateway URL for the deployment you configured:

# Use the gateway origin without a trailing slash or endpoint path.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"

For AISIX Cloud, continue using the AISIX_API_KEY created earlier. For the open-source gateway, export the caller API key:

export AISIX_API_KEY="YOUR_CALLER_API_KEY"

Send a request to the multi-target alias:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-prod",
    "messages": [
      {"role": "user", "content": "Hello from AISIX routing."}
    ]
  }'

A successful request starts with HTTP/1.1 200 OK and includes x-aisix-served-by: gpt-4o-primary. The response body retains chat-prod as the caller-facing model name. The header identifies the direct model that served the request and is not present on cache hits, single-target model responses, error responses, or every endpoint family.

Streaming requests can resolve multi-target aliases, but they do not fail over after a stream has started.

Simulate a Primary Failure

After the normal request reaches gpt-4o-primary, make that target unreachable and repeat the request. For this test, the primary and secondary direct models must use different provider keys so changing the primary endpoint does not also affect the fallback.

caution

Use a dedicated non-production provider key for the primary target. In AISIX Cloud, a provider key is organization-scoped, and changing its endpoint affects every model that references it.

AISIX Cloud

Export the ID of the provider key used only by the primary target, then save its current endpoint:

export PRIMARY_PROVIDER_KEY_ID="YOUR_PRIMARY_PROVIDER_KEY_ID"

PRIMARY_API_BASE=$(curl -sS \
  "$AISIX_CP/provider_keys/$PRIMARY_PROVIDER_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  | jq -r '.provider_key.api_base // ""')

Point the provider key to a reserved, unreachable domain:

curl -sS -X PATCH \
  "$AISIX_CP/provider_keys/$PRIMARY_PROVIDER_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"api_base":"https://api.openai.invalid/v1"}'

The control plane projects the update to every environment allowed to use the provider key. Before sending the test request, confirm that the target gateway has applied the latest configuration. See Resource Projection for the control-plane revision and gateway-status checks.

Open-Source AISIX Gateway

In resources.yaml, change only the primary provider key's endpoint:

resources.yaml (primary provider key entry)
provider_keys:
  - display_name: openai-primary
    provider: openai
    adapter: openai
    api_key: ${OPENAI_API_KEY}
    api_base: https://api.openai.invalid/v1

If you followed the open-source quickstart, validate and reload the file with its existing container:

docker exec aisix-quickstart \
  /usr/local/bin/aisix validate --resources /etc/aisix/resources.yaml

docker kill --signal=HUP aisix-quickstart

Use your gateway container name and resources-file path if they differ from the quickstart. If the reload fails, the gateway keeps serving the last valid configuration. Confirm that GET /status/config reports a successful reload before sending the test request.

Verify Failover

After simulating the failure through either management path, repeat the request from Verify Primary Routing. A successful response still starts with HTTP/1.1 200 OK, but now includes:

x-aisix-served-by: gpt-4o-secondary

AISIX retries the unreachable primary target and then forwards the request to the secondary target. The response body continues to identify the caller-facing model as chat-prod.

If the gateway's private metrics and status listener is accessible, inspect the target states:

# The local quickstarts expose the private status listener on port 9090.
curl -sS "http://127.0.0.1:9090/status/models" \
  | jq '.[] | select(
      .display_name == "gpt-4o-primary" or
      .display_name == "gpt-4o-secondary" or
      .display_name == "chat-prod"
    )'

The primary target reports cooldown with status_reason set to transport_error, while the secondary target remains healthy. The multi-target model itself reports not_applicable because its availability derives from its direct targets. Keep the status listener private because its routes do not require authentication.

Restore Primary Routing

For AISIX Cloud, restore the saved endpoint:

jq -n --arg api_base "$PRIMARY_API_BASE" '{api_base: $api_base}' \
  | curl -sS -X PATCH \
      "$AISIX_CP/provider_keys/$PRIMARY_PROVIDER_KEY_ID" \
      -H "Authorization: Bearer $AISIX_TOKEN" \
      -H "Content-Type: application/json" \
      --data-binary @-

Confirm that the restored configuration reaches the gateway. For the open-source gateway, restore or remove the test api_base override, then validate and reload resources.yaml again.

The primary target remains in cooldown until its current cooldown period expires. After it returns to healthy, repeat the request and confirm that x-aisix-served-by identifies gpt-4o-primary again.

Customize Routing Behavior

Use the following options to tune how AISIX retries requests, ranks targets, divides traffic, or filters candidates for a particular request. Each option is independent, so apply only the behavior your routing policy requires.

Tune Retry and Runtime Behavior

The chat-prod failover model configured earlier retries gpt-4o-primary once and can fall back once to gpt-4o-secondary. The following fields control attempts and runtime fallback behavior:

Field	Use When
`retries`	A request should repeat the same target after a retryable failure, with increasingly longer delays between attempts. Sets the default for every target; a target that sets its own `retries` overrides it.
`max_fallbacks`	The request can move to another eligible target. When omitted, AISIX can attempt every later target; set it to `0` for target selection without cross-target fallback.
`retry_on_429`	An upstream `429` response should be eligible for another attempt. It is disabled by default.
`fallback_on_statuses`	A provider uses another status for a transient condition that should be eligible for another attempt.
`when_all_unavailable`	AISIX should control what happens after runtime filtering removes every target. Use `"try_anyway"` when attempting a target is preferable to returning `503 all_candidates_unavailable`.

retries applies only to the inference and passthrough routes listed in Retry Budget. It does not apply to Realtime or the Files, Batches, and Fine-tuning APIs. For a streaming request on a supported route, same-target retries and failover both happen while AISIX is still establishing the upstream stream, so they are invisible to the caller. Once response bytes are committed, a failure ends the response.

routing.retries is the default for every target of this model. A target that sets its own retries uses that instead, so a group can mix a target that tolerates repeated attempts with one that should be abandoned immediately. A model alias that points straight at a provider has its own budget as well — see Retry Budget for the full resolution order and the deployment-wide default.

When you leave retries unset, AISIX prefers moving to the next eligible target over repeating the current one, and falls back to the deployment default on the last target. Set retries explicitly when a target should be repeated before failover.

Choose Additional Failover Statuses

Use fallback_on_statuses only for statuses that the provider uses for transient conditions such as model overload, queue saturation, or quota exhaustion. Configure it in the multi-target model's routing block:

{
  "routing": {
    "fallback_on_statuses": [408, 409]
  }
}

Entries must be between 400 and 599. 5xx responses already fail over, so listing them does not change behavior. Avoid authentication (401 and 403) and validation (400) statuses unless the provider is known to use them for a transient failure. Retrying a caller or credential error sends the same failing request to more targets.

fallback_on_statuses affects only the current request. The direct model's cooldown.trigger_statuses setting controls whether repeated failures remove a target from rotation for future requests. For example, listing 422 in fallback_on_statuses can move the current request to another target, but does not place the failed target in cooldown.

Handle Streaming Failures

For streaming Chat Completions, AISIX can retry or fail over only before it sends response bytes to the caller. When a model or routing model sets stream_timeout, or falls back to its timeout, AISIX waits for the first upstream chunk. A connection error, empty stream, or timeout during that wait can move the request to another target. After streaming begins, a timeout ends the stream instead.

The deployment-wide upstream.timeout_ms and upstream.stream_timeout_ms defaults do not add this first-chunk wait. With only those defaults, AISIX commits the response when the upstream response begins. A later first-chunk stall therefore surfaces as an in-stream timeout rather than a failover.

Route by Cost, Latency, or Load

The least_cost, least_latency, and least_busy strategies rank every target by a runtime signal. AISIX attempts the best-ranked target first, then moves through the remaining targets. The signal determines the order rather than the target declaration order.

least_cost ranks by each target model's combined input and output price per 1,000 tokens. Targets without a known price rank last.
least_latency ranks by a moving average of recent upstream latency, using time to first token for streaming. Targets with no samples yet rank first, so each is probed before it is ranked.
least_busy ranks by the number of in-flight requests currently dispatched to each target.

Set the strategy in the routing block. The following AISIX Cloud example creates an alias that prefers the cheapest healthy target:

curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-cheapest",
    "routing": {
      "strategy": "least_cost",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'"},
        {"model_id": "'"$SECONDARY_MODEL_ID"'"}
      ]
    }
  }'

For the open-source gateway, configure least_cost with target model names:

resources.yaml (model entry)
- display_name: chat-cheapest
  routing:
    strategy: least_cost
    targets:
      - model: gpt-4o-primary
      - model: gpt-4o-secondary

The pricing source depends on whether the gateway uses the open-source resources file or connects to AISIX Cloud:

Gateways connected to the AISIX Cloud control plane project each target's routing price from its exact configured provider and upstream model_name. Organization overrides configured in Model Pricing take precedence over catalog prices, and price changes re-rank existing groups without requiring model configuration changes. The AISIX Cloud control plane rejects wildcard direct models as routing targets, so routing groups configured in AISIX Cloud must target models with exact display and upstream model names.
The open-source AISIX gateway ranks by the cost block configured on each target model in resources.yaml. See Model Aliases.

Split Traffic for A/B Tests and Canary Releases

Use weighted with sticky: true to send a fixed share of traffic to a canary target while keeping each caller on one variant. Weights set the split. Sticky assignment makes the choice deterministic per caller so a session does not move between variants across requests.

The canary example reuses the two target models from this guide as the stable and canary variants, then adds the new alias to the caller API key allowlist. allowed_models is a replacement list, so include every model the key should keep:

CANARY_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-canary",
    "routing": {
      "strategy": "weighted",
      "sticky": true,
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'", "weight": 95},
        {"model_id": "'"$SECONDARY_MODEL_ID"'", "weight": 5}
      ]
    }
  }' | jq -r '.model.id')

curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/api_keys/$API_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "allowed_models": ["'"$ROUTING_MODEL_ID"'", "'"$CANARY_MODEL_ID"'"]
  }'

By default, sticky assignment is keyed by the caller's API key, so each key consistently lands on the same target. To key assignment by session or end user instead, send the x-aisix-routing-key header:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-key: session-1a2b" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-canary",
    "messages": [
      {"role": "user", "content": "Hello."}
    ]
  }'

Without sticky, weighted samples the split independently on every request.

In the open-source resources file, configure a 95/5 split with model names:

resources.yaml (model entry)
- display_name: chat-canary
  routing:
    strategy: weighted
    sticky: true
    targets:
      - model: gpt-4o-primary
        weight: 95
      - model: gpt-4o-secondary
        weight: 5

Route by Request Tags

Tag targets to route by a request-specific signal such as a team, tier, or environment. Add tags to each target, then send the comma-separated x-aisix-routing-tags header on the request.

The following example tags the primary target as the premium tier and the secondary target as the default. The commands then add the new alias to the caller API key allowlist:

TIERED_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-tiered",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'", "tags": ["premium"]},
        {"model_id": "'"$SECONDARY_MODEL_ID"'", "tags": ["default"]}
      ]
    }
  }' | jq -r '.model.id')

curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/api_keys/$API_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "allowed_models": ["'"$ROUTING_MODEL_ID"'", "'"$TIERED_MODEL_ID"'"]
  }'

The PATCH request replaces the caller key's allowlist. If the key should also retain access to chat-canary, include CANARY_MODEL_ID in the list.

A request carrying x-aisix-routing-tags: premium is served only by targets tagged premium. The configured strategy then orders whatever targets remain. When the request has no routing tags, AISIX uses targets tagged default when present; otherwise all targets remain eligible. When a request supplies tags but none match, AISIX tries default targets and rejects the request only if none exist.

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-tags: premium" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-tiered",
    "messages": [
      {"role": "user", "content": "Hello."}
    ]
  }'

The routing header is read from the request headers only. It is never forwarded to the upstream provider.

In the open-source resources file, assign tags to targets by model name:

resources.yaml (model entry)
- display_name: chat-tiered
  routing:
    strategy: failover
    targets:
      - model: gpt-4o-primary
        tags:
          - premium
      - model: gpt-4o-secondary
        tags:
          - default

Next Steps

Continue with Semantic Routing when target selection should depend on request meaning. Use Proxy Errors and Retries to design application retry behavior around gateway and upstream failures.

Choose a Strategy​

Configure and Test a Failover Model​

Prerequisites​

Create the Model​

AISIX Cloud​

Open-Source AISIX Gateway​

Verify Primary Routing​

Simulate a Primary Failure​

AISIX Cloud​

Open-Source AISIX Gateway​

Verify Failover​

Restore Primary Routing​

Customize Routing Behavior​

Tune Retry and Runtime Behavior​

Choose Additional Failover Statuses​

Handle Streaming Failures​

Route by Cost, Latency, or Load​

Split Traffic for A/B Tests and Canary Releases​

Route by Request Tags​

Next Steps​

Choose a Strategy

Configure and Test a Failover Model

Prerequisites

Create the Model

AISIX Cloud

Open-Source AISIX Gateway

Verify Primary Routing

Simulate a Primary Failure

AISIX Cloud

Open-Source AISIX Gateway

Verify Failover

Restore Primary Routing

Customize Routing Behavior

Tune Retry and Runtime Behavior

Choose Additional Failover Statuses

Handle Streaming Failures

Route by Cost, Latency, or Load

Split Traffic for A/B Tests and Canary Releases

Route by Request Tags

Next Steps