Semantic Routing

A semantic router lets callers use one model alias while AISIX forwards each request to a different upstream model based on the meaning of the request. Callers always send the same model. AISIX reads the latest user message, embeds it, and compares it against example utterances you attach to each route. It then forwards the request to the best-matching route's target, or to a default model when nothing matches.

In AISIX, a semantic router is a model alias that AISIX resolves through a routing decision, similar to routing groups and ensemble models. Instead of pointing to one upstream model directly, its semantic configuration tells AISIX how to score the request and which direct model to use for each route.

Use semantic routing when one entry point should fan out by topic. For example, send legal questions to a stronger reasoning model and translation requests to a cheaper multilingual model, with everything else going to a general-purpose default. The application never chooses a model per request.

Request Flow

For each request, AISIX uses the configured embedding model to compare the latest user message with route examples, then dispatches the request to one direct model.

For each chat request to a semantic router, AISIX:

1. Embeds the latest user message through the configured embedding model.
2. Scores it (cosine similarity) against every route's example embeddings.
3. Takes each route's best example score and keeps the highest route that clears its threshold.
4. Dispatches to that route's target model, or to the default model when no route clears its threshold.

Example utterances are embedded once and cached in the data plane, so the steady-state cost of a request is a single embedding call for the prompt plus local arithmetic.

Configure a Semantic Router

Semantic routing uses an embedding-capable direct model and a semantic router model. Create the embedding model and the route target models first, then create the semantic router that references them.

For the full model resource shape, see Model Aliases and the Admin API reference.

Embedding Model

An embedding model is a direct model that points at an OpenAI-compatible /v1/embeddings endpoint, plus an embedding block recording its output dimensionality. It can also be called directly through /v1/embeddings.

{
  "display_name": "bge-m3",
  "provider": "openai",
  "model_name": "bge-m3",
  "provider_key_id": "<provider key for the embeddings endpoint>",
  "embedding": {
    "dimensions": 1024,
    "normalize": true
  }
}

dimensions is required and must match the endpoint's output vector size. normalize defaults to true; set it to false only when the endpoint does not already return unit-length vectors.

The endpoint must use the OpenAI-compatible /v1/embeddings shape and return float vectors. Self-hosted runners such as Text Embeddings Inference, Ollama, and hosted providers can all be used.

For dashboard test and threshold helpers, the endpoint must be reachable from the control plane.

Semantic Router

A semantic router is a model with a semantic block that references the embedding model, lists the routes, and names a default. In self-hosted Admin API payloads, model references use the referenced model's display_name. A semantic router is mutually exclusive with direct-upstream fields, routing groups, and ensembles, and the Admin API rejects invalid combinations during schema validation.

{
  "display_name": "prod-chat",
  "semantic": {
    "embedding_model": "bge-m3",
    "default": "gpt-4o",
    "match": {
      "distance_metric": "cosine",
      "aggregation": "max",
      "threshold": 0.75
    },
    "routes": [
      {
        "name": "legal",
        "target": "claude-opus",
        "description": "Contract and legal-risk analysis",
        "examples": [
          "analyze this contract for legal risk",
          "review this NDA for liability exposure",
          "does this indemnity clause create unusual risk"
        ],
        "threshold": 0.8
      },
      {
        "name": "translate",
        "target": "gpt-4o-mini",
        "examples": ["translate this paragraph to French"]
      }
    ]
  }
}

Each route needs a name, a target pointing at a direct model, and at least one examples entry. description is optional. A route's own threshold overrides the router-level threshold for that route. The embedding model, default model, and every route target must already exist.

AISIX recomputes route example vectors automatically when you change an example's text, the embedding model, or the embedding dimensions.

In AISIX Cloud and on-premises control-plane deployments, the dashboard Models page exposes Embedding and Semantic Router sections that build these blocks for you. Pick the embedding model, the default model, and a target model per route from dropdowns, then add example utterances per route.

Matching Behavior

• Only the latest user message is embedded. When that message has several text parts, they are concatenated. Non-text content is ignored, and system, assistant, and tool turns do not affect routing.
• Each route's score is the maximum cosine similarity between the request and that route's example utterances.
• A route matches when its score is at least its effective threshold: its own threshold, otherwise the router-level threshold. Among matching routes, the highest score wins. When no route matches, the request goes to the default model.

Cross-lingual matching depends on the configured embedding model. With a multilingual embedding model, prompts and examples in different languages can still match. Cosine scores for related-but-not-identical text typically fall in the 0.4–0.65 range. Tune thresholds against your own examples rather than assuming high cutoffs.

Routing decisions can be influenced by adversarial prompts, so run input guardrails before routing and treat similarity scores as operator-only signals.

Tuning Thresholds

Two dashboard helpers, available on the semantic router form, make tuning concrete instead of guesswork. Both call the embedding endpoint from the control plane, so the endpoint must be publicly reachable.

• Test routing: Enter a prompt and see which route it resolves to, along with each route's similarity score and whether it cleared its threshold. Use it to confirm that representative prompts land where you expect before saving.
• Suggest thresholds: Compute a recommended threshold per route from the geometry of your example sets: how tightly each route's own examples cluster versus how far apart different routes sit. It needs no live traffic; it works from the configured examples alone. Apply the suggestions as a starting point, then refine with Test routing.

Caller-Facing Response

The response body reports the resolved upstream model. Two response headers expose the routing decision:

• x-aisix-route: The name of the route that matched. It is absent when the request fell through to the default model.
• x-aisix-served-by: The display name of the direct model that served the request.

Embedding Failure Behavior

If the embedding call errors or times out, the router applies its on_embedding_failure policy. Set an optional embedding_timeout_ms to bound how long the router waits for the embedding call.

{
  "semantic": {
    "embedding_timeout_ms": 500,
    "on_embedding_failure": "default"
  }
}

Configure on_embedding_failure with one of these shapes:

• default: Route to the router's default model. This is the behavior when on_embedding_failure is omitted.

• fail: Reject the request with 503.

• Explicit target object: Route to a specific safe model named by target.

  {
    "on_embedding_failure": {
      "target": "safe-model"
    }
  }

Next Steps

Continue with Ensemble Models when a request should use a full panel and judge instead of one semantic route. Use Guardrails when routing decisions need input policy checks before model dispatch.