Ensemble Models

An ensemble model lets callers use one model alias while AISIX asks several panel models for candidate responses and then asks a judge model to synthesize the final answer. The caller sends one request and receives one answer under the requested alias. The fan-out and synthesis happen inside the gateway.

In AISIX, an ensemble is a model alias that AISIX resolves by calling multiple direct models, similar to routing groups and semantic routers. Instead of pointing to one upstream model directly, its ensemble configuration tells AISIX which direct models to call as panel members and which direct model to use as the judge.

An ensemble has two parts:

• Panel models produce independent candidate responses and are called concurrently.
• The judge model receives the successful panel responses and produces the single response returned to the caller.

Panel members and the judge must reference existing direct models. Configure provider credentials, upstream model names, health behavior, cooldown behavior, and sub-call rate limits on those direct models.

Use Cases and Tradeoffs

Ensembles trade extra model calls for an additional synthesis step. They can improve answer quality or consistency for some workloads, at the cost of higher latency and spend.

Use an ensemble when candidate diversity and a judging step are likely to help:

• Reduce single-model variance and surface blind spots. Independent candidates give the judge more evidence to compare.
• Cross-check hard reasoning or research prompts. Different models can reach an answer through different paths, and another candidate response can expose mistakes.
• Self-ensemble from one provider. A panel can be the same direct model repeated with different per-member temperature and seed values, so a team with a single provider key gets answer diversity without onboarding new vendors.

An ensemble does not guarantee a more accurate answer. Panel members can repeat the same error, agreement can be wrong, and the judge can select or introduce mistakes. Before using an ensemble in production, compare it with a direct-model baseline on a task-specific evaluation set. Include representative failure cases, and measure answer quality, latency, token use, and cost.

Because an ensemble sends one request to each panel member and then calls the judge, it is not a good fit for:

• Latency-critical or streaming-first paths. Time-to-first-token is high by construction. The Streaming and Caller Response section explains the delay.
• Tool-using or function-calling requests. These requests are not supported. See Guardrails and Request Constraints for the supported request shape.
• High-volume, cost-sensitive traffic, where the evaluated benefit does not justify the extra spend.
• Endpoints other than chat completions. Ensembles are chat-only.

For those cases, use a direct model or a multi-target model instead. A multi-target model picks one target per request, while an ensemble calls all panel members and combines their output.

Request Flow

An ensemble request enters through one model alias. AISIX fans the request out to the configured panel, sends the successful panel responses to the judge, and returns only the judge's synthesized answer.

For each chat request, AISIX runs these phases:

1. Fan out to the panel. AISIX dispatches the prompt to every panel member concurrently, applying each member's own temperature and seed override.
2. Collect successful responses. AISIX waits for the panel calls to finish or time out, keeps the successful answers, and checks whether their count satisfies min_responses. Otherwise, the request follows the behavior described in Failure Handling.
3. Synthesize the final answer. AISIX builds a synthesis prompt from the original request and the collected answers, then calls the judge at a fixed low temperature. The judge's output is returned to the caller under the ensemble alias.

A single slow or failing panel member does not fail the request as long as min_responses is still met. Panel and judge calls use the retry budget configured on their referenced direct models.

Cost and Latency

An ensemble costs more and usually takes longer than a direct or multi-target model because each request can trigger several upstream calls.

Cost is the sum of every sub-call:

ensemble cost ≈ sum(panel member costs) + judge cost

A three-member panel plus a judge processes the prompt four times, and the judge additionally processes all the panel answers. The caller-facing usage reflects this real cost, as described in Usage Accounting.

Latency is dominated by the slowest panel member plus the judge:

ensemble latency ≈ max(panel member latency) + judge latency

Because the judge cannot start until the panel returns, time-to-first-token is inherently high. There is no token to stream until synthesis begins.

tip

Keep the panel small (two to four members), pick a fast judge, and set a timeout_ms so one stuck member cannot stall the whole request.

Prerequisites

Before starting, prepare the following:

• At least one direct model to use in the panel and one to use as the judge. The same direct model can fill both roles.
• A caller API key that can call the ensemble alias.
• For AISIX Cloud, an environment with an attached gateway and permission to manage models and caller API keys.
• For the open-source AISIX gateway, access to the declarative resources file and the gateway process.

Configure an Ensemble Model

Create the direct panel and judge models before the ensemble model. AISIX Cloud references them by model ID. The open-source AISIX gateway references them by display_name in the declarative resources file.

AISIX Cloud

Export the AISIX Cloud connection details and the direct-model IDs:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash.
# The local On-Premises quickstart uses http://localhost:8080/api.
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"
export PANEL_A_MODEL_ID="YOUR_FIRST_PANEL_MODEL_ID"
export PANEL_B_MODEL_ID="YOUR_SECOND_PANEL_MODEL_ID"
export JUDGE_MODEL_ID="YOUR_JUDGE_MODEL_ID"

Create the ensemble model and capture its ID:

ENSEMBLE_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "ensemble",
    "display_name": "research-ensemble",
    "ensemble": {
      "panel": [
        {
          "model_id": "'"$PANEL_A_MODEL_ID"'",
          "temperature": 0.7
        },
        {
          "model_id": "'"$PANEL_B_MODEL_ID"'",
          "temperature": 0.9
        }
      ],
      "judge": {
        "model_id": "'"$JUDGE_MODEL_ID"'"
      },
      "min_responses": 2,
      "timeout_ms": 30000
    }
  }' | jq -r '.model.id')

Add ENSEMBLE_MODEL_ID to the caller API key's allowed_models list.

You can also create and edit ensemble models on the dashboard Models page. The form provides a panel picker, a judge selector, and controls for sampling, minimum successful responses, and timeout.

Open-Source AISIX Gateway

Add the ensemble model to the models collection. Panel and judge references use direct model display_name values:

resources.yaml (ensemble model)

models:
  - display_name: research-ensemble
    ensemble:
      panel:
        - model: gpt-4o-panel
          temperature: 0.7
        - model: claude-panel
          temperature: 0.9
      judge:
        model: gpt-4o-judge
      min_responses: 2
      timeout_ms: 30000

The complete resources file must also contain the referenced direct models and their provider keys. Add research-ensemble to the caller key's allowed_models, then validate and reload the file.

Verify the Ensemble

Export the gateway URL and caller API key for the deployment you configured:

# Use the gateway origin without a trailing slash or endpoint path.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"

Call research-ensemble on the chat-completions endpoint:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "research-ensemble",
    "messages": [
      {
        "role": "user",
        "content": "In one sentence, what is an API gateway?"
      }
    ]
  }' \
  | jq '{model, choices: (.choices | length), total_tokens: .usage.total_tokens}'

A successful response contains one synthesized answer under the ensemble alias:

{
  "model": "research-ensemble",
  "choices": 1,
  "total_tokens": 508
}

The total_tokens value is the aggregate for the panel and judge calls.

Configure Ensemble Behavior

Each panel member requires a direct-model reference and can set temperature or seed for that call. The same direct model can appear more than once, which enables a self-ensemble with different sampling settings. The optional weight field is reserved for a future voting strategy and has no effect today.

The judge requires a direct-model reference and can set a custom synthesis_prompt. The judge always runs at a fixed low temperature for stable synthesis.

min_responses controls how many panel calls must succeed before the judge runs. When omitted, AISIX requires the smaller of two responses and the panel size. AISIX Cloud rejects a value larger than the panel. In the open-source resources file, the runtime caps a larger value at the panel size. If fewer than the effective minimum succeed, the request fails rather than synthesizing from too little evidence.

timeout_ms is an optional per-call upstream deadline for each panel member and the judge. It applies in addition to the referenced direct model's own timeout. Set it to 0 or omit it to disable the ensemble-level deadline.

Tuning

Tune an ensemble by adjusting panel size, per-member sampling, the minimum number of successful responses, and the per-call timeout.

Goal	Adjustment
Increase answer diversity	Use different panel models, or spread panel temperature values such as 0.5, 0.7, and 0.9.
Reproducible runs	Set a per-member seed with the chosen temperature.
Diversity with one provider	Repeat the same model two or three times with different temperature and seed values.
Tolerate panel failures	Add one or two panel members and keep min_responses below the panel size, so one failed member does not fail the request.
Bound tail latency	Set timeout_ms to the acceptable per-call ceiling. Slow members are dropped, and the run proceeds if min_responses is still met.
Lower cost	Shrink the panel and choose a cheaper judge. Re-evaluate answer quality after either change.

Judge Synthesis

The judge receives the original request and the successful panel answers as one message. Candidate answers are labeled neutrally (Answer 1, Answer 2, and so on) and do not include panel model names. This keeps the operator's provider and model choices out of the synthesized response.

Each candidate answer is capped before it reaches the judge, so a long panel cannot overflow the judge's context window. Oversized answers are truncated, not dropped.

The default synthesis instructions tell the judge to treat the candidates as evidence, favor consensus, resolve contradictions by reasoning, discard unsupported claims, and return only the final answer. These are generation instructions, not an independent check of whether a claim is correct. The response uses the same language and format the user asked for and does not mention that multiple models were involved.

Customize the Judge Prompt

A custom synthesis prompt replaces the default instructions and can include the same {original_request} and {labeled_candidates} substitutions. The following declarative resources example gives the judge a domain-specific rubric:

{
  "display_name": "code-council",
  "ensemble": {
    "panel": [
      { "model": "model-a" },
      { "model": "model-b" }
    ],
    "judge": {
      "model": "model-a",
      "synthesis_prompt": "You are a senior reviewer. From the candidate solutions, produce one correct, secure, idiomatic answer. Prefer code that compiles and handles edge cases. Output only the final code and a one-line rationale.\n\nRequest:\n{original_request}\n\nCandidates:\n{labeled_candidates}"
    }
  }
}

A custom synthesis prompt that omits {original_request} or {labeled_candidates} does not receive that content. Include both placeholders when the judge needs the original request and the candidate answers.

Ensemble Patterns

After creating a basic ensemble, adapt the panel and judge configuration for these common patterns. The examples below use direct-model names from the open-source resources file. In AISIX Cloud, use the corresponding model_id fields and model IDs.

Cross-Provider Panel

Increase candidate diversity by drawing the panel from different providers, with a strong reasoning model as judge:

{
  "panel": [
    { "model": "model-a" },
    { "model": "model-b" },
    { "model": "model-c" }
  ],
  "judge": { "model": "model-b" },
  "min_responses": 2
}

Self-Ensemble with One Provider

Get answer diversity from a single model by repeating it at different temperatures without adding extra vendors:

{
  "panel": [
    { "model": "model-a", "temperature": 0.4, "seed": 1 },
    { "model": "model-a", "temperature": 0.7, "seed": 2 },
    { "model": "model-a", "temperature": 1.0, "seed": 3 }
  ],
  "judge": { "model": "model-a" }
}

Cost-Bounded Ensemble

Use two panel members, a cheaper judge, and a tight timeout to evaluate an ensemble while limiting the added cost:

{
  "panel": [
    { "model": "model-a" },
    { "model": "model-b" }
  ],
  "judge": { "model": "model-b" },
  "min_responses": 1,
  "timeout_ms": 20000
}

Runtime Behavior

After AISIX accepts an ensemble request, the caller still sees one model response. The differences from a direct model appear in streaming, usage accounting, telemetry, rate limits, and failure handling.

Streaming and Caller Response

Ensemble models accept stream: true, but the panel phase is not streamed. AISIX waits for the panel calls to finish or time out, checks min_responses, then streams only the judge output to the caller.

During the panel phase, the connection sends no bytes, including keep-alive frames. Configure client read timeouts to allow for the slowest panel member plus the time until the judge emits its first token. Once the judge starts streaming, AISIX sends SSE keep-alive frames to hold the connection open. For the general streaming behavior, see Streaming.

The caller-facing response keeps the ensemble abstraction intact. response.model echoes the ensemble alias the caller requested, and the response does not expose panel member aliases, the judge, or upstream provider model IDs. Ensemble models, like direct models, are listed on GET /v1/models.

Usage Accounting

The response usage object is the aggregate of every panel call plus the judge call. This reflects the real cost of the request, so prompt_tokens can be much larger than the number of tokens the caller sent. Each panel member processes the prompt, and the judge processes both the prompt context and the panel answers.

On a streaming request, this aggregate is delivered in the terminal usage chunk only when the caller sets stream_options.include_usage: true. Without it, a streaming response carries no usage, exactly as for a direct model.

Sub-Call Telemetry

Each sub-call emits its own usage event, so you can attribute cost and latency per panel member and judge. All events for one request share the same request ID.

Field	Value
attempt_kind	panel for a panel member, judge for the judge call.
attempt_index	The member's slot in the panel. The judge's index is the panel size because it runs last.
attempt_model	The sub-call's model display_name.
prompt_tokens, completion_tokens	That sub-call's own token counts.

A three-member ensemble produces four usage events, three panel events plus one judge event, in addition to the aggregate returned to the client. Use these events to see which member is slow or expensive. For telemetry details, see Metrics and Logs and Observability Exporters.

Rate Limits

Rate limits apply at two scopes:

• Limits on the caller API key and ensemble alias apply once to the caller-facing request. In AISIX Cloud, applicable team and member limits also apply once. One ensemble request counts as one request at these scopes.
• Model-level limits apply to each referenced panel and judge model. A panel member that exceeds its own model limit becomes a failed sub-call and is dropped toward min_responses; the request still succeeds if enough other members answer. A judge that exceeds its own model limit fails the request with 429.

Use a limit on the ensemble alias to control caller-facing requests, and limits on the underlying direct models to control individual sub-calls. See API Key and Model Rate Limits for resource-attached limits and Rate Limit Policies for team, member, or conditional quotas.

Guardrails and Request Constraints

Applicable output guardrails run on the synthesized answer, which is the judge output, not on each panel response. The caller-facing result therefore reflects checks against the final answer the application receives. For guardrail configuration, see Guardrails.

Ensembles also have these request-shape constraints:

• Chat only. Ensembles are supported on /v1/chat/completions. Any other endpoint rejects an ensemble model with 400, naming the model and the chat-only constraint.
• No tools. A request carrying a non-empty tools array, or a tool_choice that forces a call, is rejected with 400. Broadcasting a forced tool call to multiple panel members would yield conflicting tool calls the caller cannot reconcile. An empty tools: [], or a tool_choice of none or auto, is treated as no tools and accepted.
• Direct models only. In AISIX Cloud, panel members and the judge must be direct models in the same environment. In the open-source resources file, they must be direct models in the same file. Ensembles do not nest and cannot use routing, semantic, or ensemble targets.
• Configuration-driven. There is no per-request panel override. Callers select an ensemble only by its model name.

Failure Handling

Condition	Behavior	Client status
One or more panel members slow or failing, but min_responses still met	Synthesis proceeds with the answers that arrived; failed members are dropped.	200
Successful panel responses below min_responses	Request fails because AISIX refuses to synthesize from too little evidence.	502
A panel member exceeds its own model rate limit	Treated as a failed sub-call, dropped toward min_responses.	200 if enough others answer, else 502
Judge call fails after its configured retries	Request fails. A judge configuration or credential error (4xx) is preserved, and an upstream 5xx collapses to 502.	4xx / 502
Judge exceeds its own model rate limit	Request fails.	429
Request includes tools, or targets a non-chat endpoint	Rejected before fan-out.	400

Next Steps

Continue with Proxy Errors and Retries to understand caller-facing failures. For AISIX Cloud, use Usage Reporting to inspect sub-call usage. Use Observability Exporters when ensemble telemetry needs to leave AISIX.