# Multi-Target Routing Patterns

Multi-target routing patterns let one alias optimize for cost, divide canary traffic, preserve session affinity, or filter targets by a request tag. Start with the direct models, caller API key, and environment variables from [Multi-Target Routing and Failover](https://docs.api7.ai/ai-gateway/routing/routing-and-failover.md).

## Route by Cost, Latency, or Load[​](#route-by-cost-latency-or-load "Direct link to Route by Cost, Latency, or Load")

Metric-based strategies rank eligible targets instead of following declaration order:

| Strategy        | Ranking signal                                                                                                              |
| --------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `least_cost`    | Combined input and output price per 1,000 tokens. Unpriced targets rank last.                                               |
| `least_latency` | Moving average of recent upstream latency, using time to first token for streams. Targets without samples are probed first. |
| `least_busy`    | `(1 + in-flight requests) / max(weight, 1)`, lowest first.                                                                  |

Create an alias that prefers the cheapest healthy target.

* AISIX Enterprise or Cloud
* AISIX OSS

- Dashboard
- Admin API

Create a model group as in [Create the Model](https://docs.api7.ai/ai-gateway/routing/routing-and-failover.md#create-the-model), choose **Least cost** as its strategy, and add the prepared targets. Create the group, then grant its alias to the caller API key.

![Model group configured with the Least cost strategy and two targets](https://static.api7.ai/uploads/2026/11/19/wpGwqKiY_routing-least-cost.webp)

Create a group that ranks the prepared targets by price:

```
curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "kind": "routing",

    "display_name": "chat-cheapest",

    "routing": {

      "strategy": "least_cost",

      "targets": [

        {"model_id": "'"$PRIMARY_MODEL_ID"'"},

        {"model_id": "'"$SECONDARY_MODEL_ID"'"}

      ]

    }

  }'
```

Add this entry to the existing `models` collection. Preserve the other model entries and top-level collections:

resources.yaml (model entry)

```
- display_name: chat-cheapest

  routing:

    strategy: least_cost

    targets:

      - model: gpt-4o-primary

      - model: gpt-4o-secondary
```

AISIX Enterprise and AISIX Cloud rank exact provider and upstream-model pairs, using organization prices before catalog prices. Price changes re-rank existing groups. Model-group prices and off-peak discounts do not affect ranking. AISIX OSS instead reads the target model's `cost` block. See [Model Pricing](https://docs.api7.ai/ai-gateway/cloud/model-pricing.md) and [Model Aliases](https://docs.api7.ai/ai-gateway/models/model-aliases.md).

## Split Traffic for A/B Tests and Canary Releases[​](#split-traffic-for-ab-tests-and-canary-releases "Direct link to Split Traffic for A/B Tests and Canary Releases")

Consistent hashing keeps a session on one variant while distributing distinct session keys by target weight. A session can move when a target fails or the eligible targets, weights, or routing configuration change. Weights approximate the share of keys, not an exact percentage of requests.

* AISIX Enterprise or Cloud
* AISIX OSS

- Dashboard
- Admin API

Create a model group named `chat-canary`. Choose **Consistent hashing**, add the two targets, and set their **weight** values to `95` and `5`. Create the group and grant the alias to the caller key as in the [initial setup](https://docs.api7.ai/ai-gateway/routing/routing-and-failover.md#create-the-model).

![Consistent-hash canary group with target weights of 95 and 5](https://static.api7.ai/uploads/2026/11/18/aGDmn7Qk_routing-canary.webp)

This example reuses the two prepared targets as the stable and canary variants. The allowlist update replaces the caller key's current list, so include every model the key should retain:

```
CANARY_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "kind": "routing",

    "display_name": "chat-canary",

    "routing": {

      "strategy": "consistent_hash",

      "targets": [

        {"model_id": "'"$PRIMARY_MODEL_ID"'", "weight": 95},

        {"model_id": "'"$SECONDARY_MODEL_ID"'", "weight": 5}

      ]

    }

  }' | jq -r '.model.id')



curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/api_keys/$API_KEY_ID" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "allowed_models": ["'"$ROUTING_MODEL_ID"'", "'"$CANARY_MODEL_ID"'"]

  }'
```

Add this entry to the existing `models` collection. It configures a 95/5 split across the two prepared targets:

resources.yaml (model entry)

```
- display_name: chat-canary

  routing:

    strategy: consistent_hash

    targets:

      - model: gpt-4o-primary

        weight: 95

      - model: gpt-4o-secondary

        weight: 5
```

The default hash key is `x-aisix-routing-key`, with the caller API key as fallback. Send the header to pin a specific session or user:

```
curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \

  -H "Authorization: Bearer ${AISIX_API_KEY}" \

  -H "x-aisix-routing-key: session-1a2b" \

  -H "Content-Type: application/json" \

  -d '{

    "model": "chat-canary",

    "messages": [

      {"role": "user", "content": "Hello."}

    ]

  }'
```

Use weighted round-robin when individual requests, rather than sessions, should follow the split.

Set `hash_on` to an ordered list of header, cookie, API-key, or client-IP sources. The first non-empty value wins. This example prefers a session cookie:

```
{

  "routing": {

    "strategy": "consistent_hash",

    "hash_on": [

      {"type": "cookie", "name": "session_id"},

      {"type": "api_key"}

    ],

    "targets": []

  }

}
```

## Pin Sessions with Pool Failover[​](#pin-sessions-with-pool-failover "Direct link to Pin Sessions with Pool Failover")

Combine consistent hashing with priority tiers when active instances should share sessions and a standby pool should take over only after the active pool fails. This pattern keeps a session's KV cache warm on self-hosted inference backends such as vLLM or SGLang.

* AISIX Enterprise or Cloud
* AISIX OSS

- Dashboard
- Admin API

After preparing the four direct models, create a `chat-pooled` model group. Choose **Consistent hashing**, set **Hash key** to **Custom header**, and enter `x-session-id`. Add the active models first and keep their **priority** at `0`. Set the two backup models' priority to `-1`, then create the group and grant caller access. Priority tiers control eligibility; weights distribute traffic within the selected tier.

![Session-affine routing group with two active models and two lower-priority backups](https://static.api7.ai/uploads/2026/11/18/4FadQWHw_routing-pools.webp)

Create four direct models for the two pools and export their model IDs:

```
export VLLM_A1_ID="YOUR_ACTIVE_MODEL_1_ID"

export VLLM_A2_ID="YOUR_ACTIVE_MODEL_2_ID"

export VLLM_B1_ID="YOUR_BACKUP_MODEL_1_ID"

export VLLM_B2_ID="YOUR_BACKUP_MODEL_2_ID"
```

Create the multi-target model:

```
POOLED_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "kind": "routing",

    "display_name": "chat-pooled",

    "routing": {

      "strategy": "consistent_hash",

      "hash_on": [

        {"type": "header", "name": "x-session-id"},

        {"type": "api_key"}

      ],

      "targets": [

        {"model_id": "'"$VLLM_A1_ID"'"},

        {"model_id": "'"$VLLM_A2_ID"'"},

        {"model_id": "'"$VLLM_B1_ID"'", "priority": -1},

        {"model_id": "'"$VLLM_B2_ID"'", "priority": -1}

      ]

    }

  }' | jq -r '.model.id')
```

After defining the four direct models for the active and backup pools, add this entry to `models`. Keep the other resources unchanged:

resources.yaml (model entry)

```
- display_name: chat-pooled

  routing:

    strategy: consistent_hash

    hash_on:

      - type: header

        name: x-session-id

      - type: api_key

    targets:

      - model: vllm-a1          # priority 0 = active pool

      - model: vllm-a2

      - model: vllm-b1

        priority: -1            # backup pool

      - model: vllm-b2

        priority: -1
```

The pool changes state as follows:

* A healthy active pool receives all traffic; each session stays on one active instance.
* If one active instance fails, only its sessions move to another active instance.
* If every active instance fails, sessions move to the backup pool within the same request.
* Cooldown or a [background health check](https://docs.api7.ai/ai-gateway/deployment/health-checks.md) can remove a failed instance temporarily. Without either feature, it remains eligible and later requests continue trying it before falling through to a successor.

`max_fallbacks` caps how many targets one request may try across all tiers; by default every remaining target may be attempted.

## Route by Request Tags[​](#route-by-request-tags "Direct link to Route by Request Tags")

Target tags route requests by a signal such as a team, tier, or environment. Send the selected tags in the `x-aisix-routing-tags` header.

* AISIX Enterprise or Cloud
* AISIX OSS

- Dashboard
- Admin API

Create or edit a model group and enter `premium` in the primary target's **tags** field and `default` in the secondary target's field. Create or save the group, then grant its alias to the caller key if it is new.

![Routing targets tagged premium and default](https://static.api7.ai/uploads/2026/11/18/5kuNKlQ2_routing-tags.webp)

The following example tags the primary target as the premium tier and the secondary target as the default. The commands then add the new alias to the caller API key allowlist:

```
TIERED_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "kind": "routing",

    "display_name": "chat-tiered",

    "routing": {

      "strategy": "failover",

      "targets": [

        {"model_id": "'"$PRIMARY_MODEL_ID"'", "tags": ["premium"]},

        {"model_id": "'"$SECONDARY_MODEL_ID"'", "tags": ["default"]}

      ]

    }

  }' | jq -r '.model.id')



curl -sS -X PATCH "$AISIX_CP/environments/$ENV_ID/api_keys/$API_KEY_ID" \

  -H "Authorization: Bearer $AISIX_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "allowed_models": ["'"$ROUTING_MODEL_ID"'", "'"$TIERED_MODEL_ID"'"]

  }'
```

Updating the caller key replaces its entire allowlist. Include the canary model ID if the key should retain access to that alias.

Add this entry to the existing `models` collection. It assigns the premium and default tags to the prepared targets:

resources.yaml (model entry)

```
- display_name: chat-tiered

  routing:

    strategy: failover

    targets:

      - model: gpt-4o-primary

        tags:

          - premium

      - model: gpt-4o-secondary

        tags:

          - default
```

The example request keeps only premium targets, then applies the configured strategy. Without a matching tag, AISIX prefers targets marked as the default. If no default exists, an untagged request can use every target, while a tagged request with no match is rejected.

```
curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \

  -H "Authorization: Bearer ${AISIX_API_KEY}" \

  -H "x-aisix-routing-tags: premium" \

  -H "Content-Type: application/json" \

  -d '{

    "model": "chat-tiered",

    "messages": [

      {"role": "user", "content": "Hello."}

    ]

  }'
```

The routing header is read from the request headers only. It is never forwarded to the upstream provider.

## Next Steps[​](#next-steps "Direct link to Next Steps")

See [Multi-Target Runtime Behavior](https://docs.api7.ai/ai-gateway/routing/routing-runtime-behavior.md) for retry, fallback, endpoint, and streaming rules. Use [Semantic Routing](https://docs.api7.ai/ai-gateway/routing/semantic-routing.md) when target selection should depend on request meaning.
