Skip to main content
API7.ai Logo

Semantic Routing

A semantic router gives applications one stable model alias while AISIX selects a direct model based on the meaning of each chat request. AISIX embeds the latest user message, compares it with examples for each route, and dispatches to the best match. When no route reaches its threshold, AISIX uses a default model.

Use semantic routing when requests should reach different models by topic without requiring the application to choose a model. For example, one alias can send legal questions to a reasoning model, translation requests to a multilingual model, and all other requests to a general-purpose model.

Semantic routers are supported on the OpenAI-compatible /v1/chat/completions endpoint.

How Semantic Routing Works​

For each request, AISIX uses the configured embedding model to compare the latest user message with route examples, then dispatches the request to one direct model.

AISIX embeds route examples when it applies the configuration and caches their vectors in the gateway process. It recomputes the vectors when the examples, embedding model, or embedding dimensions change. Each request therefore requires one embedding call for the latest user message, followed by local similarity calculations.

Prerequisites​

Before starting, prepare the following:

  • A provider key for an OpenAI-compatible /v1/embeddings endpoint that returns float vectors.
  • A default direct model and at least one direct model to use as a route target.
  • A caller API key that can call the semantic router alias.
  • For AISIX Cloud, an environment with an attached gateway and permission to manage models and caller API keys.
  • For the open-source AISIX gateway, access to the declarative resources file and the gateway process.

Configure a Semantic Router​

Create the embedding model, default model, and route targets before creating the semantic router. AISIX Cloud references these resources by model ID. The open-source AISIX gateway references them by display_name in the declarative resources file.

The embedding model's dimensions must match the number of values returned by its upstream endpoint. Set normalize to false only when AISIX must normalize the vectors before comparing them.

Each route requires a name, a direct target model, and at least one example. The description is optional. A route-level threshold overrides the router-level threshold for that route.

AISIX Cloud​

Export the AISIX Cloud connection details and the provider key used by the embedding model:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash.
# The local On-Premises quickstart uses http://localhost:8080/api.
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"
export PROVIDER_KEY_ID="YOUR_EMBEDDING_PROVIDER_KEY_ID"

Create the embedding model and capture its ID:

EMBEDDING_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "embedding",
    "display_name": "embedding-prod",
    "model_name": "text-embedding-3-small",
    "provider_key_id": "'"$PROVIDER_KEY_ID"'",
    "embedding": {
      "dimensions": 1536,
      "normalize": true
    }
  }' | jq -r '.model.id')

Export the IDs of the direct target models:

export DEFAULT_MODEL_ID="YOUR_DEFAULT_MODEL_ID"
export LEGAL_MODEL_ID="YOUR_LEGAL_MODEL_ID"
export TRANSLATION_MODEL_ID="YOUR_TRANSLATION_MODEL_ID"

Create the semantic router. Its default threshold applies to every route that does not define its own:

SEMANTIC_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "semantic",
    "display_name": "topic-router",
    "semantic": {
      "embedding_model_id": "'"$EMBEDDING_MODEL_ID"'",
      "default_model_id": "'"$DEFAULT_MODEL_ID"'",
      "threshold": 0.75,
      "routes": [
        {
          "name": "legal",
          "target_model_id": "'"$LEGAL_MODEL_ID"'",
          "description": "Contract and legal-risk analysis",
          "examples": [
            "Analyze this contract for legal risk",
            "Review this NDA for liability exposure"
          ],
          "threshold": 0.8
        },
        {
          "name": "translation",
          "target_model_id": "'"$TRANSLATION_MODEL_ID"'",
          "examples": [
            "Translate this paragraph to French"
          ]
        }
      ]
    }
  }' | jq -r '.model.id')

Add SEMANTIC_MODEL_ID to the caller API key's allowed_models list. The saved configuration is projected to attached gateways automatically.

You can also create and edit the embedding model and semantic router on the dashboard Models page. The semantic router form provides two threshold-tuning helpers:

  • Test routing shows the route selected for a prompt, each route's similarity score, and whether the score reached its threshold.
  • Auto-detect thresholds recommends a starting threshold for each route based on the similarity within and between the configured example sets.

Both helpers call the embedding endpoint from the control plane, so that endpoint must be reachable from the control-plane services.

Open-Source AISIX Gateway​

Add the embedding model and semantic router to the models collection. References in the semantic block use model display_name values:

resources.yaml
models:
  - display_name: embedding-prod
    provider: openai
    model_name: text-embedding-3-small
    provider_key: embeddings-provider
    embedding:
      dimensions: 1536
      normalize: true

  - display_name: topic-router
    semantic:
      embedding_model: embedding-prod
      default: general-chat
      match:
        distance_metric: cosine
        aggregation: max
        threshold: 0.75
      routes:
        - name: legal
          target: legal-chat
          description: Contract and legal-risk analysis
          examples:
            - Analyze this contract for legal risk
            - Review this NDA for liability exposure
          threshold: 0.8
        - name: translation
          target: translation-chat
          examples:
            - Translate this paragraph to French

The complete resources file must also contain the referenced provider key and the general-chat, legal-chat, and translation-chat direct models. Add topic-router to the caller key's allowed_models, then validate and reload the file.

Verify Route Selection​

Export the gateway URL and caller API key for the deployment you configured:

# Use the gateway origin without a trailing slash or endpoint path.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"

Send a request that should match the legal route:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $AISIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "topic-router",
    "messages": [
      {"role": "user", "content": "Review this contract for liability risk."}
    ]
  }'

A matched request includes x-aisix-route: legal and x-aisix-served-by with the direct model that served the request. The response body retains topic-router as the caller-facing model name.

Send a prompt unrelated to every route to confirm that AISIX uses the default model and omits x-aisix-route.

Tune Matching Behavior​

AISIX applies these rules when selecting a route:

  • Only the latest user message is embedded. When it contains several text parts, AISIX concatenates them. System, assistant, tool, and non-text content do not affect the match.
  • Each route's score is the highest cosine similarity between the request and that route's examples.
  • A route matches when its score is at least its route-level threshold, or the router-level threshold when the route does not override it.
  • When several routes match, the route with the highest score wins. When none match, AISIX uses the default model.

Cross-lingual matching depends on the embedding model. Tune thresholds with representative prompts and examples from the intended workload rather than assuming one cutoff works for every embedding model. Routing decisions can also be influenced by adversarial prompts, so apply input guardrails before dispatch when the destination has security or compliance consequences.

Handle Embedding Failures​

Use embedding_timeout_ms to bound the embedding call and on_embedding_failure to choose what happens after an embedding error or timeout. The default behavior is to use the router's default model.

In AISIX Cloud, the fallback policy is an object. To reject the request with 503:

{
  "semantic": {
    "embedding_timeout_ms": 500,
    "on_embedding_failure": {
      "mode": "fail"
    }
  }
}

Set mode to default to use the default model. To use another direct model, set mode to target and provide its target_model_id.

In the open-source resources file, use default, fail, or an object that names a direct model:

semantic:
  embedding_timeout_ms: 500
  on_embedding_failure:
    target: safe-chat

Next Steps​

Continue with Ensemble Models when one request should call several panel models and synthesize their answers. Use Guardrails when requests need policy checks before model dispatch.