# Multi-Target Runtime Behavior

A multi-target model applies its strategy, retry budget, failover budget, and runtime filters whenever a supported request resolves to the alias. Configure the model group in [Multi-Target Routing and Failover](https://docs.api7.ai/ai-gateway/routing/routing-and-failover.md), then use these rules to predict which attempts AISIX makes and what the caller observes.

## Supported Endpoints[​](#supported-endpoints "Direct link to Supported Endpoints")

Multi-target models support these endpoint groups:

| Support                                       | Endpoints                                                                                                                                                                                                                          |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Full routing behavior                         | `/v1/chat/completions`, `/v1/messages`, `/v1/messages/count_tokens`, and `/v1/responses`                                                                                                                                           |
| Requires every target to support the endpoint | `/v1/completions`, `/v1/embeddings`, `/v1/rerank`, `/v1/systemone`, `/v1/decisions`, `/v1/images/generations`, `/v1/images/edits`, `/v1/audio/transcriptions`, `/v1/audio/translations`, `/v1/audio/speech`, and `POST /v1/videos` |
| Direct models only                            | `/v1/realtime` and the Files, Batches, and Fine-tuning APIs                                                                                                                                                                        |

For the second group, an incompatible target ends the request; AISIX does not continue to another target. Treat these outcomes as group configuration errors:

| Condition                                                          | Endpoints                                                                        | Result                                                                                                                       |
| ------------------------------------------------------------------ | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| The target uses a provider shape that cannot serve the route       | `/v1/rerank`, image generation and editing, `/v1/systemone`, and `/v1/decisions` | `400` naming the multi-target model and target, such as ``model `my-group` (target `claude-img`) is not an OpenAI provider`` |
| The provider is recognized for the route but lacks that capability | `/v1/completions`, `/v1/embeddings`, image generation, and `POST /v1/videos`     | `501`                                                                                                                        |
| The target cannot count Messages tokens                            | `/v1/messages/count_tokens`                                                      | Skips the target and tries the next one                                                                                      |

## Usage and Stateful Requests[​](#usage-and-stateful-requests "Direct link to Usage and Stateful Requests")

Every upstream attempt emits its own [usage event](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md) and appears in the routing [metrics](https://docs.api7.ai/ai-gateway/reference/metrics.md) and [access log](https://docs.api7.ai/ai-gateway/observability/access-logs-and-request-correlation.md). Failed attempts record zero tokens and no cost. Traffic on supported endpoints also updates latency and concurrency measurements. For synthesized speech and streamed transcription, latency is measured to the upstream response head, and the target remains busy until the relayed stream ends.

The video ID returned from a submission identifies the target that accepted it. Status and content requests therefore return to that provider, while the status response retains the caller-facing multi-target model name. AISIX never replays a submission after the upstream has answered because the paid task may already exist.

Responses conversations likewise stay with the target that produced their context. AISIX tags returned response IDs and encrypted reasoning content with the serving target. A follow-up turn returns to that target, or to another target on the same provider key. See [Keep Conversation Context on Multi-Target Models](https://docs.api7.ai/ai-gateway/endpoints/responses-api.md#keep-conversation-context-on-multi-target-models).

## Guardrail Scope[​](#guardrail-scope "Direct link to Guardrail Scope")

Attach guardrails to the multi-target model that callers address. A guardrail attached only to a target does not run when the caller requests the routing alias. See [Guardrail Scope Follows the Addressed Model](https://docs.api7.ai/ai-gateway/traffic-controls/guardrails/behavior.md#guardrail-scope-follows-the-addressed-model).

## Configure Retry and Failover[​](#configure-retry-and-failover "Direct link to Configure Retry and Failover")

The following fields control attempts and fallback behavior:

| Field                  | Effect                                                                                                                                                   |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `retries`              | Repeats the same target after a retryable failure. A target-level value overrides the routing-model default.                                             |
| `max_fallbacks`        | Limits how many other targets one request can try. When omitted, AISIX can try every remaining target; `0` disables cross-target fallback.               |
| `retry_on_429`         | Makes upstream `429` responses retryable. Disabled by default.                                                                                           |
| `fallback_on_statuses` | Adds provider-specific transient statuses to the failover set.                                                                                           |
| `when_all_unavailable` | Controls what happens after runtime filters remove every target. `"try_anyway"` attempts a target instead of returning `503 all_candidates_unavailable`. |

When `retries` is unset, AISIX moves to the next eligible target before repeating the current one. The last target uses the deployment retry default. See [Retry Budget](https://docs.api7.ai/ai-gateway/models/model-aliases.md#retry-budget) for endpoint coverage and the full precedence order.

Retryable failures include `5xx` responses, timeouts, and transport errors. Most upstream `4xx` responses are returned directly. An unreadable `2xx` response returns `502` without a retry or failover because the upstream has already served it.

AISIX removes targets excluded by `allowed_cidrs` before it applies the fallback budget. A target blocked by its own model rate limit still occupies a selected fallback slot, records a `429` attempt, and yields to the next selected target. See [How Requests Are Counted](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limits.md#how-requests-are-counted) for quota behavior across the caller, routing model, and targets.

### Add Provider-Specific Failover Statuses[​](#add-provider-specific-failover-statuses "Direct link to Add Provider-Specific Failover Statuses")

Use `fallback_on_statuses` only for statuses that the provider uses for transient conditions, such as overload, queue saturation, or quota exhaustion:

```
{

  "routing": {

    "fallback_on_statuses": [408, 409]

  }

}
```

Entries must be between `400` and `599`. `5xx` responses already fail over. Avoid authentication and validation statuses unless the provider documents them as transient; retrying a caller or credential error only sends the same failing request to more targets.

This setting affects the current request only. A direct model's `cooldown.trigger_statuses` separately controls whether a failure removes that target from later requests.

### Handle Streaming Failures[​](#handle-streaming-failures "Direct link to Handle Streaming Failures")

AISIX can retry or fail over a stream only before it sends response bytes to the caller. A model-level `stream_timeout`, or its fallback `timeout`, makes AISIX wait for the first upstream chunk. A connection error, empty stream, or timeout during that wait can move the request to another target. After streaming begins, a timeout ends the stream.

The deployment-wide `upstream.timeout_ms` and `upstream.stream_timeout_ms` defaults do not add this first-chunk wait. With only those defaults, a later first-chunk stall appears as an in-stream timeout rather than a failover.

## Inspect Target State During Failover[​](#inspect-target-state-during-failover "Direct link to Inspect Target State During Failover")

Without cooldown, an unreachable target remains `healthy`, so each request can try it again before failing over. The routing model reports `not_applicable` because its availability comes from its targets. Enable [cooldown](https://docs.api7.ai/ai-gateway/reference/resources-file.md#direct-models) when repeated failures should remove a target temporarily.

If the private status listener is accessible, inspect the target states during a failure test:

```
# The open-source quickstart uses http://127.0.0.1:9090

curl -sS "http://127.0.0.1:9090/status/models" \

  | jq '.[] | select(

      .display_name == "gpt-4o-primary" or

      .display_name == "gpt-4o-secondary" or

      .display_name == "chat-prod"

    )'
```

Keep this listener private because it does not require authentication.

## Next Steps[​](#next-steps "Direct link to Next Steps")

Apply these rules through the examples in [Multi-Target Routing Patterns](https://docs.api7.ai/ai-gateway/routing/routing-patterns.md). Use [Proxy Errors and Retries](https://docs.api7.ai/ai-gateway/routing/proxy-errors-and-retries.md) to design the application's retry policy around the gateway behavior.
