Skip to main content
Version: 1.10.0

Ensemble Runtime Behavior

An ensemble turns one caller request into concurrent panel calls followed by a judge call. That execution model changes latency, accounting, limits, and failure handling even though the caller receives one response under the ensemble alias.

Streaming and Caller Response​

Ensemble models accept stream: true, but the panel phase is not streamed. AISIX waits for the panel calls to finish or time out, checks min_responses, then streams only the judge output to the caller.

During the panel phase, the connection sends no bytes, including keep-alive frames. Configure client read timeouts to allow for the slowest panel member plus the time until the judge emits its first token. Once the judge starts streaming, AISIX sends SSE keep-alive frames to hold the connection open. For the general streaming behavior, see Streaming.

The caller-facing response keeps the ensemble abstraction intact. response.model echoes the ensemble alias the caller requested, and the response does not expose panel member aliases, the judge, or upstream provider model IDs. Ensemble models, like direct models, are listed on GET /v1/models.

Usage Accounting​

The response usage object is the aggregate of every panel call plus the judge call. As a result, prompt_tokens can be much larger than the number of tokens the caller sent. Each panel member processes the prompt, and the judge processes both the prompt context and the panel answers.

On a streaming request, this aggregate is delivered in the terminal usage chunk only when the caller sets stream_options.include_usage: true. Without it, a streaming response carries no usage, exactly as for a direct model.

Sub-Call Telemetry​

Each sub-call emits its own usage event, so you can attribute cost and latency per panel member and judge. All events for one request share the same request ID.

FieldValue
attempt_kindpanel for a panel member, judge for the judge call.
attempt_indexThe member's slot in the panel. The judge's index is the panel size because it runs last. On a request the caller abandoned, the returned sub-calls are numbered consecutively instead, as described below.
is_final_attempttrue only on the request's last row: the judge event, or the terminal event when the request fails. Panel events are false.
attempt_modelThe sub-call's model display_name.
prompt_tokens, completion_tokensThat sub-call's own token counts.

A three-member ensemble produces four usage events, three panel events plus one judge event, in addition to the aggregate returned to the client. Use these events to see which member is slow or expensive.

When the request fails, the panel and judge events are followed by one terminal event. This event uses the next attempt_index and an empty attempt_kind. Failures include too few panel answers, a failed judge, or an output guardrail block.

When the caller disconnects part-way, every panel member and judge call that had already returned still emits its event. These events are numbered consecutively from the request's next attempt_index and are followed by the 499 event. Their tokens are charged to the token rate limits at the caller-facing scopes, as a completed request's would be. For telemetry details, see Metrics and Usage Events and Observability Exporters.

Rate Limits​

Rate limits apply at two scopes:

  • Limits on the caller API key and ensemble alias apply once to the caller-facing request. In AISIX Enterprise and AISIX Cloud, applicable team and member limits also apply once. One ensemble request counts as one request at these scopes.
  • Model-level limits apply to each referenced panel and judge model. A panel member that exceeds its own model limit becomes a failed sub-call and is dropped toward min_responses; the request still succeeds if enough other members answer. A judge that exceeds its own model limit fails the request with 429.

Use a limit on the ensemble alias to control caller-facing requests, and limits on the underlying direct models to control individual sub-calls. See API Key and Model Rate Limits for resource-attached limits and Rate Limit Policies for team, member, or conditional quotas.

Guardrails and Request Constraints​

Applicable output guardrails run on the synthesized answer, which is the judge output, not on each panel response. The caller-facing result therefore reflects checks against the final answer the application receives. For guardrail configuration, see Guardrails.

Guardrails attached to a panel member, or to the judge model, do not run on an ensemble request. Guardrail scope follows the model the caller addressed, which here is the ensemble model, so attach a guardrail to the ensemble itself to screen traffic that arrives through it. See Guardrail Scope Follows the Addressed Model.

Ensembles also have these request-shape constraints:

  • Chat only. Ensembles are supported on /v1/chat/completions. Any other endpoint rejects an ensemble model with 400, naming the model and the chat-only constraint.
  • No tools. A request carrying a non-empty tools array, or a tool_choice that forces a call, is rejected with 400. Broadcasting a forced tool call to multiple panel members would yield conflicting tool calls the caller cannot reconcile. An empty tools: [], or a tool_choice of none or auto, is treated as no tools and accepted.
  • Direct models only. In AISIX Enterprise and AISIX Cloud, panel members and the judge must be direct models in the same environment. In the open-source resources file, they must be direct models in the same file. Ensembles do not nest and cannot use routing, semantic, or ensemble targets.
  • Configuration-driven. There is no per-request panel override. Callers select an ensemble only by its model name.

Failure Handling​

ConditionBehaviorClient status
One or more panel members slow or failing, but min_responses still metSynthesis proceeds with the answers that arrived; failed members are dropped.200
Successful panel responses below min_responsesRequest fails because AISIX refuses to synthesize from too little evidence.502
A panel member exceeds its own model rate limitTreated as a failed sub-call, dropped toward min_responses.200 if enough others answer, else 502
Judge call fails after its configured retriesRequest fails. A judge configuration or credential error (4xx) is preserved, and an upstream 5xx collapses to 502.4xx / 502
Judge exceeds its own model rate limitRequest fails.429
Request includes tools, or targets a non-chat endpointRejected before fan-out.400