Skip to main content
Version: Dev

Multi-Target Runtime Behavior

A multi-target model applies its strategy, retry budget, failover budget, and runtime filters whenever a supported request resolves to the alias. Configure the model group in Multi-Target Routing and Failover, then use these rules to predict which attempts AISIX makes and what the caller observes.

Supported Endpoints​

Multi-target models support these endpoint groups:

SupportEndpoints
Full routing behavior/v1/chat/completions, /v1/messages, /v1/messages/count_tokens, and /v1/responses
Requires every target to support the endpoint/v1/completions, /v1/embeddings, /v1/rerank, /v1/systemone, /v1/decisions, /v1/images/generations, /v1/images/edits, /v1/audio/transcriptions, /v1/audio/translations, /v1/audio/speech, and POST /v1/videos
Direct models only/v1/realtime and the Files, Batches, and Fine-tuning APIs

For the second group, an incompatible target ends the request; AISIX does not continue to another target. Treat these outcomes as group configuration errors:

ConditionEndpointsResult
The target uses a provider shape that cannot serve the route/v1/rerank, image generation and editing, /v1/systemone, and /v1/decisions400 naming the multi-target model and target, such as model `my-group` (target `claude-img`) is not an OpenAI provider
The provider is recognized for the route but lacks that capability/v1/completions, /v1/embeddings, image generation, and POST /v1/videos501
The target cannot count Messages tokens/v1/messages/count_tokensSkips the target and tries the next one

Usage and Stateful Requests​

Every upstream attempt emits its own usage event and appears in the routing metrics and access log. Failed attempts record zero tokens and no cost. Traffic on supported endpoints also updates latency and concurrency measurements. For synthesized speech and streamed transcription, latency is measured to the upstream response head, and the target remains busy until the relayed stream ends.

The video ID returned from a submission identifies the target that accepted it. Status and content requests therefore return to that provider, while the status response retains the caller-facing multi-target model name. AISIX never replays a submission after the upstream has answered because the paid task may already exist.

Responses conversations likewise stay with a target that can read their context. AISIX tags returned encrypted reasoning content with the serving target and its provider key, and response IDs with the serving target. A follow-up turn returns to that target first, then to the other targets that can read the context. A target that cannot read it receives the turn without its encrypted content. See Keep Conversation Context Across Models.

Guardrail Scope​

Attach guardrails to the multi-target model that callers address. A guardrail attached only to a target does not run when the caller requests the routing alias. See Guardrail Scope Follows the Addressed Model.

Configure Retry and Failover​

The following fields control attempts and fallback behavior:

FieldEffect
retriesRepeats the same target after a retryable failure. A target-level value overrides the routing-model default.
max_fallbacksLimits how many other targets one request can try. When omitted, AISIX can try every remaining target; 0 disables cross-target fallback.
retry_on_429Makes upstream 429 responses retryable. Disabled by default.
fallback_on_statusesAdds provider-specific transient statuses to the failover set.
when_all_unavailableControls what happens after runtime filters remove every target. "try_anyway" attempts a target instead of returning 503 all_candidates_unavailable.

When retries is unset, AISIX moves to the next eligible target before repeating the current one. The last target uses the deployment retry default. See Retry Budget for endpoint coverage and the full precedence order.

Retryable failures include 5xx responses, timeouts, and transport errors. Most upstream 4xx responses are returned directly. An unreadable 2xx response returns 502 without a retry or failover because the upstream has already served it.

AISIX removes targets excluded by allowed_cidrs before it applies the fallback budget. A target blocked by its own model rate limit still occupies a selected fallback slot, records a 429 attempt, and yields to the next selected target. See How Requests Are Counted for quota behavior across the caller, routing model, and targets.

Add Provider-Specific Failover Statuses​

Use fallback_on_statuses only for statuses that the provider uses for transient conditions, such as overload, queue saturation, or quota exhaustion:

{
"routing": {
"fallback_on_statuses": [408, 409]
}
}

Entries must be between 400 and 599. 5xx responses already fail over. Avoid authentication and validation statuses unless the provider documents them as transient; retrying a caller or credential error only sends the same failing request to more targets.

This setting affects the current request only. A direct model's cooldown.trigger_statuses separately controls whether a failure removes that target from later requests.

Handle Streaming Failures​

AISIX can retry or fail over a stream only before it sends response bytes to the caller. A model-level stream_timeout, or its fallback timeout, makes AISIX wait for the first upstream chunk. A connection error, empty stream, or timeout during that wait can move the request to another target. After streaming begins, a timeout ends the stream.

The deployment-wide upstream.timeout_ms and upstream.stream_timeout_ms defaults do not add this first-chunk wait. With only those defaults, a later first-chunk stall appears as an in-stream timeout rather than a failover.

Inspect Target State During Failover​

Without cooldown, an unreachable target remains healthy, so each request can try it again before failing over. The routing model reports not_applicable because its availability comes from its targets. Enable cooldown when repeated failures should remove a target temporarily.

If the private status listener is accessible, inspect the target states during a failure test:

# The open-source quickstart uses http://127.0.0.1:9090
curl -sS "http://127.0.0.1:9090/status/models" \
| jq '.[] | select(
.display_name == "gpt-4o-primary" or
.display_name == "gpt-4o-secondary" or
.display_name == "chat-prod"
)'

Keep this listener private because it does not require authentication.

Next Steps​

Apply these rules through the examples in Multi-Target Routing Patterns. Use Proxy Errors and Retries to design the application's retry policy around the gateway behavior.