Mid-Stream Fallback

A streaming response commits its HTTP status with its first chunk. After that point, AISIX cannot turn an upstream failure into a clean retry, because the caller has already received part of the answer. The failure can be a broken connection, a stalled stream, a malformed frame, or an error the provider reports inside the committed 200. By default, AISIX terminates the stream: it sends a terminal SSE event: error frame and closes the stream without the data: [DONE] sentinel, so SDKs can detect the truncation.

The optional stream_failure block on a multi-target model's routing configuration changes that outcome. With mode: continue, a qualifying mid-stream failure keeps the same client stream open while AISIX dispatches the remaining routing targets and asks a fallback model to continue the partial answer. On recovery, the caller sees one uninterrupted stream that ends with a single [DONE] instead of a truncated one.

Mid-stream fallback extends Multi-Target Routing and Failover, which covers retries and failover before the first chunk is sent. This page covers what happens after the response is committed.

Understand the Continuation Contract

Mid-stream continuation is best effort. AISIX builds the fallback request from the original messages, a continuation instruction, and the text already delivered to the caller as a trailing assistant message. The instruction asks the fallback model to continue without repeating itself. On targets that speak the Anthropic wire format, the trailing assistant message acts as native prefill. If the failure happened before any content was generated, AISIX retries with the untouched original messages instead.

The fallback model is a different model, or at least a different deployment, continuing text it did not generate:

Best-effort continuation

Switching models mid-answer can introduce slight repetition, a shift in style or tone, or semantic drift relative to what the original target would have produced. Mid-stream fallback cannot provide token-exact seamless resumption. Enable it when a completed, slightly imperfect answer is better than a truncated one — long generations and interactive chat are the typical cases. Keep the default terminate mode when consumers require exact output, or when the application prefers to detect truncation and replay the whole request itself.

AISIX never fabricates a clean completion. If every eligible fallback target also fails, the stream ends the same way it would without the feature: a terminal SSE error event and no [DONE].

Configure Mid-Stream Fallback

Add stream_failure inside the routing block of a multi-target model. All fields are optional; an omitted block keeps the terminate behavior exactly.

Field	Default	Description
stream_failure.mode	terminate	terminate ends a failed stream with a terminal SSE error event and no [DONE]. continue dispatches the remaining routing targets and resumes the same client stream with a best-effort continuation.
stream_failure.on	all four classes	Which mid-stream error classes trigger continuation. Omitted enables all four. Non-retryable errors never trigger continuation regardless of this list.
stream_failure.max_fallbacks	1	Maximum fallback targets dispatched for one stream. The default is deliberately tighter than the pre-stream routing.max_fallbacks, because each mid-stream attempt adds caller-visible latency in the middle of a response.

The on list accepts the following error classes:

Trigger	Fires When
transport_error	The upstream connection broke mid-stream, such as a reset or premature close.
read_timeout	The gap between two upstream chunks exceeded the effective stream_timeout.
upstream_decode_error	An upstream frame failed to parse as a chunk and was not a recognizable error envelope.
upstream_in_band_error	The provider reported an error inside the committed 200 stream.

Retryability follows the same rules as pre-stream failover. A provider error reported in-band with a 4xx status other than 429 never triggers continuation. retry_on_429 makes in-band 429 errors eligible, and fallback_on_statuses extends eligibility to configured provider-specific statuses. See Tune Retry and Runtime Behavior.

For read_timeout to be a practical trigger, set stream_timeout on the target models or on the multi-target model. Without one, the inter-chunk deadline falls back to the deployment-wide defaults, whose 6000-second backstop rarely fires. See How the Timeouts Relate.

AISIX Cloud

The following command creates a two-target failover model with mid-stream continuation enabled. It reuses the environment exports and direct target models from Multi-Target Routing and Failover:

curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "routing",
    "display_name": "chat-resilient",
    "routing": {
      "strategy": "failover",
      "targets": [
        {"model_id": "'"$PRIMARY_MODEL_ID"'"},
        {"model_id": "'"$SECONDARY_MODEL_ID"'"}
      ],
      "retry_on_429": true,
      "stream_failure": {
        "mode": "continue"
      }
    }
  }'

Add chat-resilient to a caller API key's allowed_models list before sending traffic, as shown in the failover guide.

Open-Source AISIX Gateway

In the declarative resources file, add the block to the multi-target model entry. This example spells out every field; the on list and max_fallbacks values shown are the defaults:

resources.yaml (model entry)

- display_name: chat-resilient
  stream_timeout: 30000
  routing:
    strategy: failover
    targets:
      - model: gpt-4o-primary
      - model: gpt-4o-secondary
    retry_on_429: true
    stream_failure:
      mode: continue
      on:
        - transport_error
        - read_timeout
        - upstream_decode_error
        - upstream_in_band_error
      max_fallbacks: 1

The stream_timeout on the model entry bounds the gap between upstream chunks, so a stalled stream surfaces as a read_timeout after 30 seconds instead of hanging.

Target Selection at Switch Time

When a qualifying failure occurs, AISIX dispatches the remaining targets in the configured strategy order. The failed target is not retried mid-stream; routing.retries applies only before the stream commits. Runtime state is re-checked at switch time: a target in cooldown or failing its health check is skipped without consuming the max_fallbacks budget. A dispatched target that fails to produce a stream consumes one. The failed target receives the same cooldown and health bookkeeping as a pre-stream failure, so repeated mid-stream failures remove it from rotation for future requests.

What the Client Sees

On a recovered stream, the client receives the primary target's chunks, then the fallback target's continuation chunks, then a single data: [DONE]. The bracketed line below is annotation, not stream content:

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Here are the main"}}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" considerations:"}}]}

[the upstream fails here; AISIX dispatches a fallback target into the same stream]

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" First, cost."}}]}

data: [DONE]

The HTTP status remains 200 throughout, and response headers do not change, because both were committed with the first chunk. The x-aisix-served-by header and the access-log target therefore name the target that started the stream; use usage events to identify the target that finished it.

If the caller set stream_options.include_usage: true, the final usage frame merges the estimated usage of the failed partial attempts with the usage of the attempt that completed the stream. The frame therefore reflects the full upstream cost of the response.

Continuation is impossible when the failure class is not eligible, when a safety boundary below applies, or when every fallback target also fails. The stream then ends with a terminal error event and no [DONE]:

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Here are the main"}}]}

event: error
data: {"error":{"message":"***","type":"upstream_error"}}

A caller that cancels the stream never triggers a fallback dispatch, so an abandoned request cannot generate hidden upstream traffic.

Safety Boundaries

Some streams terminate on a mid-stream failure regardless of configuration, because a continuation could not be spliced safely:

• Streams that have already emitted tool-call or reasoning (reasoning_content) deltas. A half-delivered tool call or a provider-signed reasoning stream cannot be continued by another model without producing an inconsistent response.
• Requests with structured output, response_format of type json_object or json_schema. A second model continuing a partial JSON document is unlikely to produce valid output.
• Streams whose delivered partial text exceeds 1 MiB. Past that bound, a faithful continuation is no longer realistic and the continuation request itself becomes oversized.

In the current release, mid-stream fallback applies to /v1/chat/completions streaming only. Streaming on /v1/messages and /v1/responses keeps the terminate behavior even when stream_failure is configured.

Observe Mid-Stream Fallbacks

A recovered stream still reports HTTP 200, so status codes alone cannot distinguish a clean stream from a truncated or recovered one. AISIX adds a stream-level outcome signal and per-attempt records for that purpose.

The terminal usage event of a streaming request carries a stream_outcome field:

Value	Meaning
success	The stream completed normally.
partial_failed	The stream terminated after the 200 with an in-band error event and no [DONE].
partial_recovered	A mid-stream failure was recovered on a fallback target and the stream then completed normally.

partial_failed is also recorded for terminate-mode failures, so the field separates truncated streams from clean ones even with mid-stream fallback disabled. The field is empty on non-streaming events, on failed-attempt events, and on client-abandoned streams, which keep status 499.

Each attempt keeps its own usage event, all sharing one request_id:

• The failed serving attempt emits an event at switch time with its error class and estimated partial spend: the prompt tokens it consumed plus the partial output it delivered. Those counts are marked as estimated.
• A fallback candidate that fails before producing a stream emits its own zero-token failure event.
• The terminal event is attributed to the target that actually finished the stream, with attempt_kind set to mid_stream_fallback and latency scoped to that attempt.

The aisix_mid_stream_fallbacks_total counter tracks how often streams enter mid-stream fallback and whether they recover. Each stream that dispatched at least one fallback target increments it once at stream end, labeled with the multi-target alias and the outcome:

sum by (model, outcome) (rate(aisix_mid_stream_fallbacks_total[5m]))

outcome="recovered" means a fallback target completed the stream; outcome="failed" means the stream still terminated. See the Metrics Reference.

Next Steps

Use Multi-Target Routing and Failover for target strategies and pre-stream retry behavior, and Proxy Errors and Retries to design client-side handling around gateway recovery. Configure Observability Exporters to deliver the per-attempt usage events to your analytics store.