# Ensemble Runtime Behavior

An ensemble turns one caller request into concurrent panel calls followed by a judge call. That execution model changes latency, accounting, limits, and failure handling even though the caller receives one response under the ensemble alias.

## Streaming and Caller Response[​](#streaming-and-caller-response "Direct link to Streaming and Caller Response")

Ensemble models accept `stream: true`, but the panel phase is not streamed. AISIX waits for the panel calls to finish or time out, checks `min_responses`, then streams only the judge output to the caller.

During the panel phase, the connection sends no bytes, including keep-alive frames. Configure client read timeouts to allow for the slowest panel member plus the time until the judge emits its first token. Once the judge starts streaming, AISIX sends SSE keep-alive frames to hold the connection open. For the general streaming behavior, see [Streaming](https://docs.api7.ai/ai-gateway/endpoints/streaming.md).

The caller-facing response keeps the ensemble abstraction intact. `response.model` echoes the ensemble alias the caller requested, and the response does not expose panel member aliases, the judge, or upstream provider model IDs. Ensemble models, like direct models, are listed on `GET /v1/models`.

## Usage Accounting[​](#usage-accounting "Direct link to Usage Accounting")

The response `usage` object is the aggregate of every panel call plus the judge call. As a result, `prompt_tokens` can be much larger than the number of tokens the caller sent. Each panel member processes the prompt, and the judge processes both the prompt context and the panel answers.

On a streaming request, this aggregate is delivered in the terminal `usage` chunk only when the caller sets `stream_options.include_usage: true`. Without it, a streaming response carries no `usage`, exactly as for a direct model.

## Sub-Call Telemetry[​](#sub-call-telemetry "Direct link to Sub-Call Telemetry")

Each sub-call emits its own usage event, so you can attribute cost and latency per panel member and judge. All events for one request share the same request ID.

| Field                                | Value                                                                                                                                                                                                       |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `attempt_kind`                       | `panel` for a panel member, `judge` for the judge call.                                                                                                                                                     |
| `attempt_index`                      | The member's slot in the panel. The judge's index is the panel size because it runs last. On a request the caller abandoned, the returned sub-calls are numbered consecutively instead, as described below. |
| `is_final_attempt`                   | `true` only on the request's last row: the judge event, or the terminal event when the request fails. Panel events are `false`.                                                                             |
| `attempt_model`                      | The sub-call's model `display_name`.                                                                                                                                                                        |
| `prompt_tokens`, `completion_tokens` | That sub-call's own token counts.                                                                                                                                                                           |

A three-member ensemble produces four usage events, three panel events plus one judge event, in addition to the aggregate returned to the client. Use these events to see which member is slow or expensive.

When the request fails, the panel and judge events are followed by one terminal event. This event uses the next `attempt_index` and an empty `attempt_kind`. Failures include too few panel answers, a failed judge, or an output guardrail block.

When the caller disconnects part-way, every panel member and judge call that had already returned still emits its event. These events are numbered consecutively from the request's next `attempt_index` and are followed by the `499` event. Their tokens are charged to the token rate limits at the caller-facing scopes, as a completed request's would be. For telemetry details, see [Metrics and Usage Events](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md) and [Observability Exporters](https://docs.api7.ai/ai-gateway/observability/exporters.md).

## Rate Limits[​](#rate-limits "Direct link to Rate Limits")

Rate limits apply at two scopes:

* Limits on the caller API key and ensemble alias apply once to the caller-facing request. In AISIX Enterprise and AISIX Cloud, applicable team and member limits also apply once. One ensemble request counts as one request at these scopes.
* Model-level limits apply to each referenced panel and judge model. A panel member that exceeds its own model limit becomes a failed sub-call and is dropped toward `min_responses`; the request still succeeds if enough other members answer. A judge that exceeds its own model limit fails the request with `429`.

Use a limit on the ensemble alias to control caller-facing requests, and limits on the underlying direct models to control individual sub-calls. See [API Key and Model Rate Limits](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limits.md) for resource-attached limits and [Rate Limit Policies](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limit-policies.md) for team, member, or conditional quotas.

## Guardrails and Request Constraints[​](#guardrails-and-request-constraints "Direct link to Guardrails and Request Constraints")

Applicable output guardrails run on the synthesized answer, which is the judge output, not on each panel response. The caller-facing result therefore reflects checks against the final answer the application receives. For guardrail configuration, see [Guardrails](https://docs.api7.ai/ai-gateway/traffic-controls/guardrails/keyword.md).

Guardrails attached to a panel member, or to the judge model, do not run on an ensemble request. Guardrail scope follows the model the caller addressed, which here is the ensemble model, so attach a guardrail to the ensemble itself to screen traffic that arrives through it. See [Guardrail Scope Follows the Addressed Model](https://docs.api7.ai/ai-gateway/traffic-controls/guardrails/behavior.md#guardrail-scope-follows-the-addressed-model).

Ensembles also have these request-shape constraints:

* Chat only. Ensembles are supported on `/v1/chat/completions`. Any other endpoint rejects an ensemble model with `400`, naming the model and the chat-only constraint.
* No tools. A request carrying a non-empty `tools` array, or a `tool_choice` that forces a call, is rejected with `400`. Broadcasting a forced tool call to multiple panel members would yield conflicting tool calls the caller cannot reconcile. An empty `tools: []`, or a `tool_choice` of `none` or `auto`, is treated as no tools and accepted.
* Direct models only. In AISIX Enterprise and AISIX Cloud, panel members and the judge must be direct models in the same environment. In the open-source resources file, they must be direct models in the same file. Ensembles do not nest and cannot use routing, semantic, or ensemble targets.
* Configuration-driven. There is no per-request panel override. Callers select an ensemble only by its model name.

## Failure Handling[​](#failure-handling "Direct link to Failure Handling")

| Condition                                                                | Behavior                                                                                                                 | Client status                             |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- |
| One or more panel members slow or failing, but `min_responses` still met | Synthesis proceeds with the answers that arrived; failed members are dropped.                                            | `200`                                     |
| Successful panel responses below `min_responses`                         | Request fails because AISIX refuses to synthesize from too little evidence.                                              | `502`                                     |
| A panel member exceeds its own model rate limit                          | Treated as a failed sub-call, dropped toward `min_responses`.                                                            | `200` if enough others answer, else `502` |
| Judge call fails after its configured retries                            | Request fails. A judge configuration or credential error (`4xx`) is preserved, and an upstream `5xx` collapses to `502`. | `4xx` / `502`                             |
| Judge exceeds its own model rate limit                                   | Request fails.                                                                                                           | `429`                                     |
| Request includes tools, or targets a non-chat endpoint                   | Rejected before fan-out.                                                                                                 | `400`                                     |
