Multi-Target Routing and Failover

A multi-target model keeps one caller-facing alias in front of several target models. AISIX selects a target for each request, retries eligible failures, and can fail over without requiring the application to change model names.

The alias is a caller-facing model name, so it appears in GET /v1/models for every caller API key allowed to request it. Scope a key to the alias alone to publish it as the only entry point callers discover, and change its targets without touching the application.

The multi-target model contains a routing block rather than its own provider configuration. Each target is an existing direct model with its own provider key and upstream model name. AISIX Cloud references these targets by model ID, while the open-source AISIX gateway references them by display_name in the declarative resources file.

The routing model selects a direct target, and that target uses its provider key to call the upstream model:

Choose a Strategy

Choose a strategy based on how the caller-facing alias should distribute traffic:

Goal	Strategy	Selection Behavior
Keep a primary target with ordered backups.	failover	Attempts targets in declaration order.
Rotate traffic across similar targets, optionally by share.	round_robin	Smooth weighted round-robin over target weights. Equal or absent weights rotate in declaration order.
Pin each session to one target.	consistent_hash	Hashes a per-request key over a weighted ring, so the same key keeps landing on the same healthy target. Weight changes the approximate share of hash keys, not a fixed percentage of requests. Ideal for A/B and canary splits and for session/cache affinity with pool failover.
Prefer the lowest estimated price.	least_cost	Ranks eligible targets by configured prices or AISIX Cloud catalog pricing.
Prefer the fastest recently observed target.	least_latency	Ranks eligible targets by recent upstream latency.
Prefer the target with the lowest weighted concurrency.	least_busy	Ranks eligible targets by (1 + current in-flight requests) / max(weight, 1), lowest first.

failover is the default when strategy is omitted. You can also narrow the eligible targets per request with routing tags.

Two per-target attributes are available under every strategy:

• weight (default 1) sets the rotation share under round_robin, the share of the hash ring under consistent_hash, and the denominator of the least_busy score. failover, least_cost, and least_latency accept the field but do not use it.
• priority (default 0) splits eligible targets into tiers. A higher value is preferred, so give backup targets -1. The strategy orders targets within each tier, and a lower tier receives traffic after no higher-tier target remains available. See Pin Sessions with Pool Failover.

AISIX applies routing-tag and allowed_cidrs filters before it partitions the surviving targets into priority tiers. A lower-priority target can therefore serve a request while a healthy higher-priority target is excluded by either filter for that request.

Changed in 0.10.0

The former weighted strategy is folded into round_robin (which now honors weights), and the sticky flag is replaced by the consistent_hash strategy. AISIX Cloud migrates stored routing models automatically; self-managed resource files must be updated (weighted → round_robin keeping weights; weighted + sticky → consistent_hash).

AISIX retries and fails over on retryable upstream failures, such as 5xx responses, request timeouts, and transport errors. Most upstream 4xx responses are returned to the caller. Enable retry_on_429 for upstream rate limits, or configure fallback_on_statuses for other provider-specific transient statuses.

AISIX also skips a target that is over its own model rate limit. It records the skipped target as a failed 429 routing attempt and continues with the remaining targets in strategy order without retrying that target. When every target is over its limit, the request returns 429.

A target whose own allowed_cidrs excludes the caller is likewise not a candidate. That check runs before the strategy selects, so such a target is never attempted and does not consume the max_fallbacks budget. When every target excludes the caller, the request returns 403.

Configure and Test a Failover Model

The following example builds and tests a two-target model that keeps gpt-4o-primary as its preferred target and uses gpt-4o-secondary after an eligible failure. Instructions are provided for AISIX Cloud and the open-source AISIX gateway.

Prerequisites

Before starting, prepare the following:

• Two or more direct models to use as routing targets. To run the failure test, the primary and secondary models must use separate provider-key resources, although both resources can contain the same upstream credential.
• A caller API key that can call the multi-target model, or permission to create one.
• cURL and jq to run the API and verification commands.
• For AISIX Cloud, an environment with an attached gateway and permission to manage models and caller API keys.
• For the open-source AISIX gateway, access to the declarative resources file and the gateway process.

Create the Model

Create the direct target models before the multi-target model. The examples below configure the same failover behavior through each management path: AISIX first uses gpt-4o-primary, then moves to gpt-4o-secondary after a retryable failure.

AISIX Cloud

Set the control-plane URL, admin token, and environment ID for your AISIX Cloud organization:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash
# The local On-Premises quickstart uses http://localhost:8080/api
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Export the IDs of the direct target models:

export PRIMARY_MODEL_ID="YOUR_PRIMARY_MODEL_ID"
export SECONDARY_MODEL_ID="YOUR_SECONDARY_MODEL_ID"

Create a failover model that starts with the primary target and falls back to the secondary target:

ROUTING_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-prod",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'"},
        {"model_id": "'"$SECONDARY_MODEL_ID"'"}
      ],
      "retries": 1,
      "max_fallbacks": 1,
      "retry_on_429": true
    }
  }' | jq -r '.model.id')

With this configuration, AISIX starts with gpt-4o-primary. If that target has a retryable failure, AISIX can retry it once and then fail over once to gpt-4o-secondary.

Use the captured ROUTING_MODEL_ID to allow caller access, and to update, inspect, or delete the model later. If an existing caller API key should use chat-prod, add this ID to its allowed_models list.

For a quick check, create a caller API key that can call only chat-prod:

KEY_RESPONSE=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/api_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "routing-guide-caller",
    "allowed_models": ["'"$ROUTING_MODEL_ID"'"]
  }')

export AISIX_API_KEY=$(echo "$KEY_RESPONSE" | jq -r '.plaintext')
API_KEY_ID=$(echo "$KEY_RESPONSE" | jq -r '.api_key.id')

The plaintext caller key is returned only once. The commands save it for the verification request and save the key ID for later allowlist updates. The model and caller key configurations are projected to attached gateways automatically.

Open-Source AISIX Gateway

Add two provider keys, their direct models, and the chat-prod routing model to the complete resources file. Keeping the provider keys separate lets you test one target without changing the other:

resources.yaml (routing provider keys and models)

provider_keys:
  - display_name: openai-primary
    provider: openai
    adapter: openai
    api_key: ${OPENAI_API_KEY}
  - display_name: openai-secondary
    provider: openai
    adapter: openai
    api_key: ${OPENAI_API_KEY}

models:
  - display_name: gpt-4o-primary
    provider: openai
    model_name: gpt-4o
    provider_key: openai-primary
  - display_name: gpt-4o-secondary
    provider: openai
    model_name: gpt-4o-mini
    provider_key: openai-secondary
  - display_name: chat-prod
    routing:
      strategy: failover
      targets:
        - model: gpt-4o-primary
        - model: gpt-4o-secondary
      retries: 1
      max_fallbacks: 1
      retry_on_429: true

Add chat-prod to the caller key's allowed_models, then validate and reload the assembled file.

Verify Primary Routing

Export the gateway URL for the deployment you configured:

# Use the gateway origin without a trailing slash or endpoint path.
# The local quickstarts use http://127.0.0.1:3000
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"

For AISIX Cloud, continue using the AISIX_API_KEY created earlier. For the open-source gateway, export the caller API key:

export AISIX_API_KEY="YOUR_CALLER_API_KEY"

Send a request to the multi-target alias:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-prod",
    "messages": [
      {"role": "user", "content": "Hello from AISIX routing."}
    ]
  }'

A successful request starts with HTTP/1.1 200 OK and includes x-aisix-served-by: gpt-4o-primary. The response body retains chat-prod as the caller-facing model name. The header identifies the direct model that served the request and is not present on cache hits, single-target model responses, error responses, or every endpoint family.

Streaming requests can resolve multi-target aliases, but they do not fail over after a stream has started.

Simulate a Primary Failure

After the normal request reaches gpt-4o-primary, make that target unreachable and repeat the request. For this test, the primary and secondary direct models must use different provider keys so changing the primary endpoint does not also affect the fallback.

caution

Use a dedicated non-production provider key for the primary target. In AISIX Cloud, a provider key is organization-scoped, and changing its endpoint affects every model that references it.

AISIX Cloud

Export the ID of the provider key used only by the primary target, then save its current endpoint:

export PRIMARY_PROVIDER_KEY_ID="YOUR_PRIMARY_PROVIDER_KEY_ID"

PRIMARY_API_BASE=$(curl -sS \
  "$AISIX_CP/provider_keys/$PRIMARY_PROVIDER_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  | jq -r '.provider_key.api_base // ""')

Point the provider key to a reserved, unreachable domain:

curl -sS -X PATCH \
  "$AISIX_CP/provider_keys/$PRIMARY_PROVIDER_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"api_base":"https://api.openai.invalid/v1"}'

The control plane projects the update to every environment allowed to use the provider key. Before sending the test request, confirm that the target gateway has applied the latest configuration. See Resource Projection for the control-plane revision and gateway-status checks.

Open-Source AISIX Gateway

In the complete resources.yaml, replace the existing openai-primary provider key entry with the following block to change only its endpoint. Preserve the other entries and collections:

resources.yaml (primary provider key entry)

provider_keys:
  - display_name: openai-primary
    provider: openai
    adapter: openai
    api_key: ${OPENAI_API_KEY}
    api_base: https://api.openai.invalid/v1

If you followed the open-source quickstart, validate and reload the file with its existing container:

docker exec aisix-quickstart \
  /usr/local/bin/aisix validate --resources /etc/aisix/resources.yaml

docker kill --signal=HUP aisix-quickstart

Use your gateway container name and resources-file path if they differ from the quickstart. If the reload fails, the gateway keeps serving the last valid configuration. Confirm that GET /status/config reports a successful reload before sending the test request.

Verify Failover

After simulating the failure through either management path, repeat the request from Verify Primary Routing. A successful response still starts with HTTP/1.1 200 OK, but now includes:

x-aisix-served-by: gpt-4o-secondary

AISIX retries the unreachable primary target and then forwards the request to the secondary target. The response body continues to identify the caller-facing model as chat-prod.

If the gateway's private metrics and status listener is accessible, inspect the target states:

# The open-source quickstart uses http://127.0.0.1:9090
curl -sS "http://127.0.0.1:9090/status/models" \
  | jq '.[] | select(
      .display_name == "gpt-4o-primary" or
      .display_name == "gpt-4o-secondary" or
      .display_name == "chat-prod"
    )'

Both direct targets report healthy, and the multi-target model itself reports not_applicable because its availability derives from its direct targets. Keep the status listener private because its routes do not require authentication.

The unreachable primary stays healthy because these target models configure no cooldown block, and cooldown is opt-in. Failover therefore runs per request: every request attempts gpt-4o-primary — twice, because this example sets retries: 1 — absorbs the same failure, and only then moves on to gpt-4o-secondary. Set cooldown.enabled to true on the primary target when you would rather have a repeatedly failing target dropped from the candidate list for a while, and spare every later request those attempts. The target then reports cooldown with a status_reason such as transport_error until the cooldown expires.

Restore Primary Routing

For AISIX Cloud, restore the saved endpoint:

jq -n --arg api_base "$PRIMARY_API_BASE" '{api_base: $api_base}' \
  | curl -sS -X PATCH \
      "$AISIX_CP/provider_keys/$PRIMARY_PROVIDER_KEY_ID" \
      -H "Authorization: Bearer $AISIX_TOKEN" \
      -H "Content-Type: application/json" \
      --data-binary @-

Confirm that the restored configuration reaches the gateway. For the open-source gateway, restore or remove the test api_base override, then validate and reload resources.yaml again.

Repeat the request and confirm that x-aisix-served-by identifies gpt-4o-primary again. If you enabled cooldown on the primary target, it stays out of rotation until its current cooldown period expires, so wait for it to report healthy again first.

Customize Routing Behavior

Use the following options to tune how AISIX retries requests, ranks targets, divides traffic, or filters candidates for a particular request. Each option is independent, so apply only the behavior your routing policy requires.

Tune Retry and Runtime Behavior

The chat-prod failover model configured earlier retries gpt-4o-primary once and can fall back once to gpt-4o-secondary. The following fields control attempts and runtime fallback behavior:

Field	Use When
retries	A request should repeat the same target after a retryable failure, with increasingly longer delays between attempts. Sets the default for every target; a target that sets its own retries overrides it.
max_fallbacks	The request can move to another eligible target. When omitted, AISIX can attempt every later target; set it to 0 for target selection without cross-target fallback.
retry_on_429	An upstream 429 response should be eligible for another attempt. It is disabled by default.
fallback_on_statuses	A provider uses another status for a transient condition that should be eligible for another attempt.
when_all_unavailable	AISIX should control what happens after runtime filtering removes every target. Use "try_anyway" when attempting a target is preferable to returning 503 all_candidates_unavailable.

retries applies only to the inference routes listed in Retry Budget. It does not apply to Realtime, passthrough routes, or the Files, Batches, and Fine-tuning APIs. For a streaming request on a supported route, same-target retries and failover both happen while AISIX is still establishing the upstream stream, so they are invisible to the caller. Once response bytes are committed, a failure ends the response.

routing.retries is the default for every target of this model. A target that sets its own retries uses that instead, so a group can mix a target that tolerates repeated attempts with one that should be abandoned immediately. A model alias that points straight at a provider has its own budget as well — see Retry Budget for the full resolution order and the deployment-wide default.

When you leave retries unset, AISIX prefers moving to the next eligible target over repeating the current one, and falls back to the deployment default on the last target. Set retries explicitly when a target should be repeated before failover.

Choose Additional Failover Statuses

Use fallback_on_statuses only for statuses that the provider uses for transient conditions such as model overload, queue saturation, or quota exhaustion. Configure it in the multi-target model's routing block:

{
  "routing": {
    "fallback_on_statuses": [408, 409]
  }
}

Entries must be between 400 and 599. 5xx responses already fail over, so listing them does not change behavior. Avoid authentication (401 and 403) and validation (400) statuses unless the provider is known to use them for a transient failure. Retrying a caller or credential error sends the same failing request to more targets.

fallback_on_statuses affects only the current request. On a target whose direct model enables cooldown, that model's cooldown.trigger_statuses setting separately controls whether failures remove the target from rotation for future requests. The two lists are independent: listing 422 in fallback_on_statuses can move the current request to another target, but does not place the failed target in cooldown.

Handle Streaming Failures

For streaming Chat Completions, AISIX can retry or fail over only before it sends response bytes to the caller. When a model or routing model sets stream_timeout, or falls back to its timeout, AISIX waits for the first upstream chunk. A connection error, empty stream, or timeout during that wait can move the request to another target. After streaming begins, a timeout ends the stream instead.

The deployment-wide upstream.timeout_ms and upstream.stream_timeout_ms defaults do not add this first-chunk wait. With only those defaults, AISIX commits the response when the upstream response begins. A later first-chunk stall therefore surfaces as an in-stream timeout rather than a failover.

Route by Cost, Latency, or Load

The least_cost, least_latency, and least_busy strategies rank every target by a runtime signal. AISIX attempts the best-ranked target first, then moves through the remaining targets. The signal determines the order rather than the target declaration order.

• least_cost ranks by each target model's combined input and output price per 1,000 tokens. Targets without a known price rank last.
• least_latency ranks by a moving average of recent upstream latency, using time to first token for streaming. Targets with no samples yet rank first, so each is probed before it is ranked.
• least_busy ranks by (1 + current in-flight requests) / max(weight, 1), lowest first. Adding 1 lets weight break the tie between idle targets and gives a higher-weight target proportionally more concurrency.

Set the strategy in the routing block. The following AISIX Cloud example creates an alias that prefers the cheapest healthy target:

curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-cheapest",
    "routing": {
      "strategy": "least_cost",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'"},
        {"model_id": "'"$SECONDARY_MODEL_ID"'"}
      ]
    }
  }'

For the open-source gateway, add the following chat-cheapest entry to the existing models collection. Preserve the other model entries and top-level collections:

resources.yaml (model entry)

- display_name: chat-cheapest
  routing:
    strategy: least_cost
    targets:
      - model: gpt-4o-primary
      - model: gpt-4o-secondary

The pricing source depends on whether the gateway uses the open-source resources file or connects to AISIX Cloud:

• Gateways connected to the AISIX Cloud control plane project each target's routing price from its exact configured provider and upstream model_name. Organization overrides configured in Model Pricing take precedence over catalog prices, and price changes re-rank existing groups without requiring model configuration changes. The AISIX Cloud control plane rejects wildcard direct models as routing targets, so routing groups configured in AISIX Cloud must target models with exact display and upstream model names.
• The open-source AISIX gateway ranks by the cost block configured on each target model in resources.yaml. See Model Aliases.

Split Traffic for A/B Tests and Canary Releases

Use consistent_hash to keep each session on one variant while distributing distinct hash keys approximately according to the target weights. While the hash key, eligible target set, and routing configuration remain unchanged, the session does not move between variants across requests. A target failure or a configuration or eligibility change can remap it. The actual request ratio can differ from the weights. Hash keys may not be uniformly distributed, and each session can send a different number of requests.

The canary example reuses the two target models from this guide as the stable and canary variants, then adds the new alias to the caller API key allowlist. allowed_models is a replacement list, so include every model the key should keep:

CANARY_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-canary",
    "routing": {
      "strategy": "consistent_hash",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'", "weight": 95},
        {"model_id": "'"$SECONDARY_MODEL_ID"'", "weight": 5}
      ]
    }
  }' | jq -r '.model.id')

curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/api_keys/$API_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "allowed_models": ["'"$ROUTING_MODEL_ID"'", "'"$CANARY_MODEL_ID"'"]
  }'

By default, the hash key is the x-aisix-routing-key request header, falling back to the caller's API key — so each API key consistently lands on the same target, and a caller can pin per session or end user by sending the header:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-key: session-1a2b" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-canary",
    "messages": [
      {"role": "user", "content": "Hello."}
    ]
  }'

To distribute individual requests by weight without session affinity, use round_robin with the same weights. For a stable eligible target set, smooth weighted round-robin produces the configured proportions across each complete rotation.

The hash_on field changes where the key comes from: an ordered chain of sources, the first non-empty one winning, from header and cookie (both named by name), the caller's api_key, or the resolved client_ip. For example, to key sessions by a cookie:

"routing": {
  "strategy": "consistent_hash",
  "hash_on": [
    {"type": "cookie", "name": "session_id"},
    {"type": "api_key"}
  ],
  "targets": [...]
}

In the open-source resources file, add the following chat-canary entry to the existing models collection. It configures a 95/5 split using the two existing model names:

resources.yaml (model entry)

- display_name: chat-canary
  routing:
    strategy: consistent_hash
    targets:
      - model: gpt-4o-primary
        weight: 95
      - model: gpt-4o-secondary
        weight: 5

Pin Sessions with Pool Failover

Combining consistent_hash with priority tiers covers the classic active/backup shape: several instances serving the same model in an active pool, a standby pool that takes over only when the whole active pool is down, and cache-friendly session affinity inside both. This suits self-hosted inference backends (vLLM, SGLang) where pinning a session to one instance keeps its KV cache warm.

For AISIX Cloud, create four direct models for the two pools and export their model IDs:

export VLLM_A1_ID="YOUR_ACTIVE_MODEL_1_ID"
export VLLM_A2_ID="YOUR_ACTIVE_MODEL_2_ID"
export VLLM_B1_ID="YOUR_BACKUP_MODEL_1_ID"
export VLLM_B2_ID="YOUR_BACKUP_MODEL_2_ID"

Create the multi-target model:

POOLED_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-pooled",
    "routing": {
      "strategy": "consistent_hash",
      "hash_on": [
        {"type": "header", "name": "x-session-id"},
        {"type": "api_key"}
      ],
      "targets": [
        {"model_id": "'"$VLLM_A1_ID"'"},
        {"model_id": "'"$VLLM_A2_ID"'"},
        {"model_id": "'"$VLLM_B1_ID"'", "priority": -1},
        {"model_id": "'"$VLLM_B2_ID"'", "priority": -1}
      ]
    }
  }' | jq -r '.model.id')

After defining vllm-a1, vllm-a2, vllm-b1, and vllm-b2, add chat-pooled to models. Keep the other resources unchanged:

resources.yaml (model entry)

- display_name: chat-pooled
  routing:
    strategy: consistent_hash
    hash_on:
      - type: header
        name: x-session-id
      - type: api_key
    targets:
      - model: vllm-a1          # priority 0 = active pool
      - model: vllm-a2
      - model: vllm-b1
        priority: -1            # backup pool
      - model: vllm-b2
        priority: -1

The behavior, in order of escalation:

• While the active pool is healthy, the backup pool receives no traffic. Each session key hashes to one active instance and stays there.
• When a single active instance fails, only the sessions hashed to it move — each to its ring successor inside the SAME pool. Other sessions keep their instance, and the backup pool still receives nothing.
• When every active instance is down, traffic shifts to the backup pool — hashed the same way inside it. The first request that discovers the outage still succeeds: the walk crosses into the backup tier within that request.
• When an active instance recovers, its sessions return to it. A failure takes an instance out of the ring only while it is in cooldown or marked unhealthy by a background model check, and both are opt-in on the target direct model. With neither configured, as in the example above, its sessions come back on the first request it serves successfully.

max_fallbacks caps how many targets one request may try across all tiers; by default every remaining target may be attempted.

Route by Request Tags

Tag targets to route by a request-specific signal such as a team, tier, or environment. Add tags to each target, then send the comma-separated x-aisix-routing-tags header on the request.

The following example tags the primary target as the premium tier and the secondary target as the default. The commands then add the new alias to the caller API key allowlist:

TIERED_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-tiered",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'", "tags": ["premium"]},
        {"model_id": "'"$SECONDARY_MODEL_ID"'", "tags": ["default"]}
      ]
    }
  }' | jq -r '.model.id')

curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/api_keys/$API_KEY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "allowed_models": ["'"$ROUTING_MODEL_ID"'", "'"$TIERED_MODEL_ID"'"]
  }'

The PATCH request replaces the caller key's allowlist. If the key should also retain access to chat-canary, include CANARY_MODEL_ID in the list.

A request carrying x-aisix-routing-tags: premium is served only by targets tagged premium. The configured strategy then orders whatever targets remain. When the request has no routing tags, AISIX uses targets tagged default when present; otherwise all targets remain eligible. When a request supplies tags but none match, AISIX tries default targets and rejects the request only if none exist.

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "x-aisix-routing-tags: premium" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-tiered",
    "messages": [
      {"role": "user", "content": "Hello."}
    ]
  }'

The routing header is read from the request headers only. It is never forwarded to the upstream provider.

In the open-source resources file, add the following chat-tiered entry to the existing models collection. It assigns tags to targets by model name:

resources.yaml (model entry)

- display_name: chat-tiered
  routing:
    strategy: failover
    targets:
      - model: gpt-4o-primary
        tags:
          - premium
      - model: gpt-4o-secondary
        tags:
          - default

Next Steps

Continue with Semantic Routing when target selection should depend on request meaning. Use Proxy Errors and Retries to design application retry behavior around gateway and upstream failures.