Skip to main content
Version: Dev

Multi-Target Routing Patterns

Multi-target routing patterns let one alias optimize for cost, divide canary traffic, preserve session affinity, or filter targets by a request tag. Start with the direct models, caller API key, and environment variables from Multi-Target Routing and Failover.

Route by Cost, Latency, or Load​

Metric-based strategies rank eligible targets instead of following declaration order:

StrategyRanking signal
least_costCombined input and output price per 1,000 tokens. Unpriced targets rank last.
least_latencyMoving average of recent upstream latency, using time to first token for streams. Targets without samples are probed first.
least_busy(1 + in-flight requests) / max(weight, 1), lowest first.

Create an alias that prefers the cheapest healthy target.

Create a model group as in Create the Model, choose Least cost as its strategy, and add the prepared targets. Create the group, then grant its alias to the caller API key.

Model group configured with the Least cost strategy and two targets

AISIX Enterprise and AISIX Cloud rank exact provider and upstream-model pairs, using organization prices before catalog prices. Price changes re-rank existing groups. Sale prices and off-peak discounts do not affect ranking. AISIX OSS instead reads the target model's cost block. See Model Pricing and Model Aliases.

Split Traffic for A/B Tests and Canary Releases​

Consistent hashing keeps a session on one variant while distributing distinct session keys by target weight. A session can move when a target fails or the eligible targets, weights, or routing configuration change. Weights approximate the share of keys, not an exact percentage of requests.

Create a model group named chat-canary. Choose Consistent hashing, add the two targets, and set their weight values to 95 and 5. Create the group and grant the alias to the caller key as in the initial setup.

Consistent-hash canary group with target weights of 95 and 5

The default hash key is x-aisix-routing-key, with the caller API key as fallback. Send the header to pin a specific session or user:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
-H "Authorization: Bearer ${AISIX_API_KEY}" \
-H "x-aisix-routing-key: session-1a2b" \
-H "Content-Type: application/json" \
-d '{
"model": "chat-canary",
"messages": [
{"role": "user", "content": "Hello."}
]
}'

Use weighted round-robin when individual requests, rather than sessions, should follow the split.

Set hash_on to an ordered list of header, cookie, API-key, or client-IP sources. The first non-empty value wins. This example prefers a session cookie:

{
"routing": {
"strategy": "consistent_hash",
"hash_on": [
{"type": "cookie", "name": "session_id"},
{"type": "api_key"}
],
"targets": []
}
}

Pin Sessions with Pool Failover​

Combine consistent hashing with priority tiers when active instances should share sessions and a standby pool should take over only after the active pool fails. This pattern keeps a session's KV cache warm on self-hosted inference backends such as vLLM or SGLang.

After preparing the four direct models, create a chat-pooled model group. Choose Consistent hashing, set Hash key to Custom header, and enter x-session-id. Add the active models first and keep their priority at 0. Set the two backup models' priority to -1, then create the group and grant caller access. Priority tiers control eligibility; weights distribute traffic within the selected tier.

Session-affine routing group with two active models and two lower-priority backups

The pool changes state as follows:

  • A healthy active pool receives all traffic; each session stays on one active instance.
  • If one active instance fails, only its sessions move to another active instance.
  • If every active instance fails, sessions move to the backup pool within the same request.
  • Cooldown or a background health check can remove a failed instance temporarily. Without either feature, it remains eligible and later requests continue trying it before falling through to a successor.

max_fallbacks caps how many targets one request may try across all tiers; by default every remaining target may be attempted.

Route by Request Tags​

Target tags route requests by a signal such as a team, tier, or environment. Send the selected tags in the x-aisix-routing-tags header.

Create or edit a model group and enter premium in the primary target's tags field and default in the secondary target's field. Create or save the group, then grant its alias to the caller key if it is new.

Routing targets tagged premium and default

The example request keeps only premium targets, then applies the configured strategy. Without a matching tag, AISIX prefers targets marked as the default. If no default exists, an untagged request can use every target, while a tagged request with no match is rejected.

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
-H "Authorization: Bearer ${AISIX_API_KEY}" \
-H "x-aisix-routing-tags: premium" \
-H "Content-Type: application/json" \
-d '{
"model": "chat-tiered",
"messages": [
{"role": "user", "content": "Hello."}
]
}'

The routing header is read from the request headers only. It is never forwarded to the upstream provider.

Next Steps​

See Multi-Target Runtime Behavior for retry, fallback, endpoint, and streaming rules. Use Semantic Routing when target selection should depend on request meaning.