Ensemble Models

An ensemble model lets callers use one model alias while AISIX asks several panel models for candidate responses and then asks a judge model to synthesize the final answer. The caller sends one request and receives one answer under the requested alias. The fan-out and synthesis happen inside the gateway.

In AISIX, an ensemble is a model alias that AISIX resolves by calling multiple direct models, similar to routing groups and semantic routers. Instead of pointing to one upstream model directly, its ensemble configuration tells AISIX which direct models to call as panel members and which direct model to use as the judge.

An ensemble has two parts:

• Panel models produce independent candidate responses and are called concurrently.
• The judge model receives the successful panel responses and produces the single response returned to the caller.

Panel members and the judge must reference existing direct model aliases. Configure provider credentials, provider model names, health behavior, cooldown behavior, and model-level rate limits on those direct models.

Use Cases and Tradeoffs

Ensembles trade extra model calls for stronger answer synthesis. They are most useful when answer quality or consistency matters more than the lowest possible latency or cost.

Use an ensemble when a single model's answer is not reliable enough on its own:

• Reduce single-model variance and blind spots. Independent answers that agree are more likely correct. The judge resolves contradictions and discards unsupported claims.
• Cross-check hard reasoning or research prompts. Different models can reach an answer through different paths, and another candidate response can expose mistakes.
• Self-ensemble from one provider. A panel can be the same direct model repeated with different per-member temperature and seed values, so a team with a single provider key gets answer diversity without onboarding new vendors.

Because an ensemble sends one request to each panel member and then calls the judge, it is not a good fit for every workload. Use another model type for:

• Latency-critical or streaming-first paths. Time-to-first-token is high by construction. The Streaming and Caller Response section explains the delay.
• Tool-using or function-calling requests. These requests are not supported. See Guardrails and Request Constraints for the supported request shape.
• High-volume, cost-sensitive traffic, where the quality gain does not justify the extra spend.
• Endpoints other than chat completions. Ensembles are chat-only.

For those cases, use a direct model or a multi-target model instead. A multi-target model picks one target per request, while an ensemble calls all panel members and combines their output.

Request Flow

An ensemble request enters through one model alias. AISIX fans the request out to the configured panel, sends the successful panel responses to the judge, and returns only the judge's synthesized answer.

For each chat request, AISIX runs these phases:

1. Fan out to the panel. AISIX dispatches the prompt to every panel member concurrently, applying each member's own temperature and seed override.
2. Collect successful responses. AISIX keeps the successful answers. Once enough panel members succeed to satisfy min_responses, it continues. Otherwise, the request follows the behavior described in Failure Handling.
3. Synthesize the final answer. AISIX builds a synthesis prompt from the original request and the collected answers, then calls the judge at a fixed low temperature. The judge's output is returned to the caller under the ensemble alias.

A single slow or failing panel member does not fail the request, as long as min_responses is still met. The judge call is retried once on a transient failure (timeout, transport fault, or upstream 5xx).

Cost and Latency

An ensemble costs more and usually takes longer than a direct or multi-target model because each request can trigger several upstream calls.

Cost is the sum of every sub-call:

ensemble cost ≈ sum(panel member costs) + judge cost

A three-member panel plus a judge processes the prompt four times, and the judge additionally processes all the panel answers. The caller-facing usage reflects this real cost, as described in Usage Accounting.

Latency is dominated by the slowest panel member plus the judge:

ensemble latency ≈ max(panel member latency) + judge latency

Because the judge cannot start until the panel returns, time-to-first-token is inherently high. There is no token to stream until synthesis begins.

tip

Keep the panel small (two to four members), pick a fast judge, and set a timeout_ms so one stuck member cannot stall the whole request.

Configure an Ensemble Model

An ensemble model references existing direct models and defines how AISIX uses them as panel members and judge.

Create the Model

Panel members and the judge reference existing direct models by display_name, so create those direct models first. If you need the full model creation flow, see Model Aliases.

The ensemble model defines the panel, judge, minimum successful panel responses, and per-call timeout:

{
  "display_name": "research-ensemble",
  "ensemble": {
    "panel": [
      { "model": "gpt-4o-panel", "temperature": 0.7 },
      { "model": "claude-panel", "temperature": 0.7 },
      { "model": "gemini-panel", "temperature": 0.9 }
    ],
    "judge": { "model": "gpt-4o-judge" },
    "min_responses": 2,
    "timeout_ms": 30000
  }
}

After the ensemble model is stored and propagated, call research-ensemble like any other model on /v1/chat/completions:

curl -sS -X POST "http://127.0.0.1:3000/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_CALLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "research-ensemble",
    "messages": [
      {
        "role": "user",
        "content": "In one sentence, what is an API gateway?"
      }
    ]
  }' \
  | jq '{ model, choices: (.choices | length), total_tokens: .usage.total_tokens }'

The result shows the observable contract of an ensemble:

{
  "model": "research-ensemble",
  "choices": 1,
  "total_tokens": 508
}

The model value is the ensemble alias, choices is one synthesized answer, and total_tokens is the panel-plus-judge aggregate.

Ensemble Fields

Use these field descriptions to tune the panel, judge, minimum successful responses, and timeout shown in the example. For the full schema, see the Admin API reference.

Panel

A non-empty list of panel members. Each member references a direct model by display_name, with optional per-member sampling overrides:

Field	Type	Required	Description
model	string	Yes	Direct model display_name that receives one panel request.
temperature	number	No	Sampling temperature for this member's call. It overrides the request's temperature. Omit it to keep the request value.
seed	integer	No	Sampling seed for this member's call. Pair it with temperature when you need reproducible variation.
weight	integer	No	Reserved for a future voting strategy and ignored today. Safe to omit.

The same display_name may appear more than once. That is how a self-ensemble is expressed.

Judge

The model that synthesizes the panel answers into one response.

Field	Type	Required	Description
model	string	Yes	Direct model display_name that synthesizes the panel responses.
synthesis_prompt	string	No	Custom synthesis template. If set, it must contain {original_request} and {labeled_candidates}. Omit it to use the default. The Judge Synthesis section explains how the template is used.

The judge always runs at a fixed low temperature for stable synthesis. This is not configurable.

Minimum Successful Responses

min_responses sets the minimum number of successful panel responses required before the judge runs.

• When omitted, AISIX requires up to two successful panel responses, capped by the panel size. A value larger than the panel is clamped to the panel size, and the effective value is always at least 1, so a single-member self-ensemble panel needs one response.
• If fewer than min_responses panel members succeed, the request fails rather than synthesizing from too little evidence.

Per-Call Timeout

timeout_ms sets an optional per-call upstream deadline for each panel member and the judge call, in milliseconds. This is in addition to each referenced model's own timeout. 0 or absent means no ensemble-level per-call deadline.

Tuning

Tune an ensemble by adjusting panel size, per-member sampling, the minimum number of successful responses, and the per-call timeout.

Goal	Adjustment
Increase answer diversity	Use different panel models, or spread panel temperature values such as 0.5, 0.7, and 0.9.
Reproducible runs	Set a per-member seed with the chosen temperature.
Diversity with one provider	Repeat the same model two or three times with different temperature and seed values.
Tolerate panel failures	Add one or two panel members and keep min_responses below the panel size, so one failed member does not fail the request.
Bound tail latency	Set timeout_ms to the acceptable per-call ceiling. Slow members are dropped, and the run proceeds if min_responses is still met.
Lower cost	Shrink the panel and choose a cheaper judge. Synthesis quality depends more on the judge's reasoning than its size.

Judge Synthesis

The judge receives the original request and the successful panel answers as one message. Candidate answers are labeled neutrally (Answer 1, Answer 2, and so on) and do not include panel model names. This keeps the operator's provider and model choices out of the synthesized response.

Each candidate answer is capped before it reaches the judge, so a long panel cannot overflow the judge's context window. Oversized answers are truncated, not dropped.

The default synthesis instructions tell the judge to treat the candidates as evidence, favor consensus, resolve contradictions by reasoning, discard unsupported claims, and return only the final answer. The response uses the same language and format the user asked for and does not mention that multiple models were involved.

Customize the Judge Prompt

A custom synthesis prompt replaces the default instructions and can include the same {original_request} and {labeled_candidates} substitutions. Use it to give the judge a domain-specific rubric:

{
  "display_name": "code-council",
  "ensemble": {
    "panel": [
      { "model": "model-a" },
      { "model": "model-b" }
    ],
    "judge": {
      "model": "model-a",
      "synthesis_prompt": "You are a senior reviewer. From the candidate solutions, produce one correct, secure, idiomatic answer. Prefer code that compiles and handles edge cases. Output only the final code and a one-line rationale.\n\nRequest:\n{original_request}\n\nCandidates:\n{labeled_candidates}"
    }
  }
}

A custom synthesis prompt that omits {original_request} or {labeled_candidates} does not receive that content. Include both placeholders when the judge needs the original request and the candidate answers.

Ensemble Patterns

After creating a basic ensemble, adapt the panel and judge configuration for these common patterns.

Cross-Provider Panel

Maximize independence by drawing the panel from different providers, with a strong reasoning model as judge:

{
  "panel": [
    { "model": "model-a" },
    { "model": "model-b" },
    { "model": "model-c" }
  ],
  "judge": { "model": "model-b" },
  "min_responses": 2
}

Self-Ensemble with One Provider

Get answer diversity from a single model by repeating it at different temperatures without adding extra vendors:

{
  "panel": [
    { "model": "model-a", "temperature": 0.4, "seed": 1 },
    { "model": "model-a", "temperature": 0.7, "seed": 2 },
    { "model": "model-a", "temperature": 1.0, "seed": 3 }
  ],
  "judge": { "model": "model-a" }
}

Cost-Bounded Ensemble

Use two panel members, a cheaper judge, and a tight timeout when you want a modest quality improvement without a large cost increase:

{
  "panel": [
    { "model": "model-a" },
    { "model": "model-b" }
  ],
  "judge": { "model": "model-b" },
  "min_responses": 1,
  "timeout_ms": 20000
}

Runtime Behavior

After AISIX accepts an ensemble request, the caller still sees one model response. The differences from a direct model appear in streaming, usage accounting, telemetry, rate limits, and failure handling.

Streaming and Caller Response

Ensemble models accept stream: true, but the panel phase is not streamed. AISIX first waits for enough panel responses, then streams only the judge output to the caller.

During the panel phase, the connection sends no bytes, including keep-alive frames. Configure client read timeouts to allow for the slowest panel member plus the time until the judge emits its first token. Once the judge starts streaming, AISIX sends SSE keep-alive frames to hold the connection open. For the general streaming behavior, see Streaming.

The caller-facing response keeps the ensemble abstraction intact. response.model echoes the ensemble alias the caller requested, and the response does not expose panel member aliases, the judge, or upstream provider model IDs. Ensemble models, like direct models, are listed on GET /v1/models.

Usage Accounting

The response usage object is the aggregate of every panel call plus the judge call. This reflects the real cost of the request, so prompt_tokens can be much larger than the number of tokens the caller sent. Each panel member processes the prompt, and the judge processes both the prompt context and the panel answers.

On a streaming request, this aggregate is delivered in the terminal usage chunk only when the caller sets stream_options.include_usage: true. Without it, a streaming response carries no usage, exactly as for a direct model.

Sub-Call Telemetry

Each sub-call emits its own usage event, so you can attribute cost and latency per panel member and judge. All events for one request share the same request ID.

Field	Value
attempt_kind	panel for a panel member, judge for the judge call.
attempt_index	The member's slot in the panel. The judge's index is the panel size because it runs last.
attempt_model	The sub-call's model display_name.
prompt_tokens, completion_tokens	That sub-call's own token counts.

A three-member ensemble produces four usage events, three panel events plus one judge event, in addition to the aggregate returned to the client. Use these events to see which member is slow or expensive. For telemetry details, see Metrics and Logs and Observability Exporters.

Rate Limits

Rate limits apply at two scopes:

• Request-level limits, such as caller API key, team, and member limits, are reserved once on the caller-facing ensemble alias. One ensemble request counts as one request.
• Model-level limits apply to each referenced panel and judge model. A panel member that exceeds its own model limit becomes a failed sub-call and is dropped toward min_responses; the request still succeeds if enough other members answer. A judge that exceeds its own model limit fails the request with 429.

Set model-level limits on the underlying direct models, not on the ensemble alias. For limit configuration, see Rate Limits.

Guardrails and Request Constraints

The guardrail chain configured on the ensemble alias runs on the synthesized answer, which is the judge output, not on each panel member response. The caller-facing response reflects the final answer the caller receives. For guardrail configuration, see Guardrails.

Ensembles also have these request-shape constraints:

• Chat only. Ensembles are supported on /v1/chat/completions. Any other endpoint rejects an ensemble model with 400, naming the model and the chat-only constraint.
• No tools. A request carrying a non-empty tools array, or a tool_choice that forces a call, is rejected with 400. Broadcasting a forced tool call to multiple panel members would yield conflicting tool calls the caller cannot reconcile. An empty tools: [], or a tool_choice of none or auto, is treated as no tools and accepted.
• Direct models only. Panel members and the judge must reference direct models in the same environment. Ensembles do not nest, cannot use routing or ensemble targets, and are mutually exclusive with direct-upstream fields.
• Configuration-driven. There is no per-request panel override. Callers select an ensemble only by its model name.

Failure Handling

Condition	Behavior	Client status
One or more panel members slow or failing, but min_responses still met	Synthesis proceeds with the answers that arrived; failed members are dropped.	200
Successful panel responses below min_responses	Request fails because AISIX refuses to synthesize from too little evidence.	502
A panel member exceeds its own model rate limit	Treated as a failed sub-call, dropped toward min_responses.	200 if enough others answer, else 502
Judge call fails after its one retry	Request fails. A judge configuration or credential error (4xx) is preserved, and an upstream 5xx collapses to 502.	4xx / 502
Judge exceeds its own model rate limit	Request fails.	429
Request includes tools, or targets a non-chat endpoint	Rejected before fan-out.	400

AISIX Cloud

In AISIX Cloud, create and edit ensembles from the dashboard Models page instead of the Admin API. The form provides a panel picker, a judge selector, and controls for per-member sampling, minimum successful responses, and timeout. The panel picker allows the same model more than once for self-ensembles.

Cloud projects the configuration into the environment data plane, which runs the same logic described here. Per-sub-call usage appears in Usage Reporting.

Next Steps

Continue with Proxy Errors and Retries to understand caller-facing failures, or Observability Exporters when ensemble sub-call telemetry needs to leave AISIX.