Skip to main content
Version: 1.11.0

Semantic Routing with TypeSafe Jev

A semantic router can decide in one of two ways. By default it compares an embedding of the request with each route's examples. With the TypeSafe jev decision backend, AISIX instead asks TypeSafe's hosted jev decision model which of the configured routes fits the request.

For each request, AISIX sends jev the latest user text and every route's name and description. See What the Router Classifies for how that text is chosen. Jev returns one route and a confidence score, so this mode needs no examples, embedding model, or similarity threshold.

After the decision, the selected route's target serves the request and the access log records the route. A request that no route serves uses the default model. Targets can be model groups, and session affinity avoids another jev call for a pinned session.

Before You Use It​

Review these constraints before choosing this backend:

ConsiderationBehavior
Request pathEach unpinned request makes one synchronous call to TypeSafe before dispatch. The gateway needs outbound access, and TypeSafe receives the latest user text.
LanguageEvaluate routing with representative prompts in every language the application serves.
AccountingAISIX Enterprise and AISIX Cloud record the jev call as an internal call and bill it at the decision model's price. AISIX OSS records the call without calculating cost. TypeSafe bills the call separately.
Dashboard helpersTest routing and Auto-detect thresholds support embedding-based routers only.

Prerequisites​

Before starting, prepare the following:

  • A TypeSafe API key.
  • In AISIX Enterprise or AISIX Cloud, a price for the decision model, if its decision calls should cost something.
  • A default target and at least one route target. Each is a direct model or a model group.
  • A caller API key that can call the semantic router alias.
  • For AISIX Enterprise or AISIX Cloud, an environment with an attached gateway. Dashboard users need permission to manage provider keys, models, and caller API keys. AISIX Cloud Admin API users need a write-scoped admin token.
  • For the open-source AISIX gateway, a running gateway and access to the declarative resources file.
  • If you use the Admin API to configure AISIX Enterprise or AISIX Cloud, install curl and jq.

Create a TypeSafe Provider Key and Decision Model​

The jev call authenticates with a provider key whose provider is typesafe. Its secret is your TypeSafe API key, which AISIX sends as a bearer token. api_base is optional and defaults to https://api.typesafe.ai.

The same key can back direct decision models on /v1/systemone and /v1/decisions. The managed API does not allow it on an embedding model or delete it while a model or router still references it. A TypeSafe model cannot serve as a route, default, or failure target because semantic-router targets receive Chat Completions requests.

Allow the provider key in the environment where the router will run.

  1. Open Provider keys and select New provider key. Enter typesafe as the Display name, choose the TypeSafe provider, and enter the secret in Upstream API key.
  2. Leave API base empty to use https://api.typesafe.ai, unless your deployment needs an override.
TypeSafe provider key name, provider, and upstream credential
  1. Under Allowed environments, select the router's environment. Select Create provider key.
TypeSafe provider key allowed in the router environment
  1. Open the environment's Models page. Select Create model and choose Direct model.
  2. Set Display name to jev, Upstream model id to jev-latest, and select the TypeSafe provider key.
Direct TypeSafe decision model with its provider key and upstream model name
  1. Create the model. Its credential and upstream model name are used for classifier calls. Configure its price if those internal calls should be billed.

Configure a Jev Router​

Every route needs a name, a target, and a description. Write each description as a statement of which requests the route handles, because it is the only thing jev reads about the route. Routes take no examples and no threshold in this mode.

Create the router using the prepared targets and TypeSafe configuration.

Open the environment's Models page, select Create model, and choose Semantic router.

  1. Set Display name to support-router and Decision backend to TypeSafe jev classifier.
  2. Select the jev model under Decision model. Keep Minimum confidence at 0.5 and Timeout (ms) at 3000.
TypeSafe classifier selection, minimum confidence, and decision timeout

If the selected decision model has no price, the form links to Model pricing. Configure pricing there if its decision calls should be billed.

  1. Under Routes, add billing, select the billing target, and enter its Description:
Questions about invoices, payments, refunds, and subscription plans
Billing route with its target and classifier description
  1. Add code, select the code target, and enter its description:
Writing, reviewing, or debugging source code
Code route with its target and classifier description
  1. Select the prepared default model. Keep On classifier failure set to fall back to that model for this example. This control is in the main form.
Default model and classifier-failure behavior for the TypeSafe router
  1. Select Create semantic router and check the saved classifier backend and route count. Reopen Edit to inspect the route targets and default model.
Saved TypeSafe router showing its classifier backend and route count

An all-model caller key already grants the new alias. Otherwise, open API keys and edit a caller whose plaintext you saved, or create a new key. Keep Selected models, add support-router under Allowed models, and retain every model the caller still needs.

Caller model access with support-router selected

For an existing key, select Save changes. For a new key, select Create API key, copy its plaintext from One-time view, then select I've saved it.

The provider-key and router rows support Edit and Delete. To remove the setup, delete or update the router before deleting its decision model. Delete the TypeSafe provider key only after all referencing models and routers have been removed. Confirm deletion when removing the router.

Verify Route Selection​

Export the gateway connection values:

# AISIX_PROXY is the gateway origin without a trailing slash or endpoint path.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"

Send a prompt that should match the code route:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
-H "Authorization: Bearer $AISIX_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "support-router",
"messages": [
{"role": "user", "content": "Help me debug a null pointer exception."}
]
}'

Use the x-aisix-request-id response header to find the request's access-log line. Confirm these signals:

SignalExpected result
Access logsemantic_route="code"; semantic_score contains the classifier's confidence
Debug response headerx-aisix-route: code when proxy.expose_debug_headers is enabled

Use semantic_score to tune min_confidence. If the classifier's confidence is too low, the access log records semantic_fallback="low_confidence" and preserves the score that fell short.

Reference the Decision Model​

In AISIX Enterprise and AISIX Cloud, the classifier references its decision model by model_id. Keeping the reference has these effects:

  • A decision model cannot be deleted while a router's classifier uses it. The deletion returns 409 MODEL_IN_USE naming the router. Point the router's classifier at another model, or delete the router, first.
  • A decision model cannot move to a provider key whose provider is not typesafe, or be renamed into a wildcard alias, while a router's classifier uses it. Changing its provider key or model_name takes effect on every router that uses it.

Older routers can reference the TypeSafe key and jev model directly. They read back with classifier.provider_key_id and classifier.model and keep routing, but their decision calls are not billed. The dashboard identifies this legacy configuration and requires a decision model before saving changes.

Legacy classifier notice and required decision-model selection in the router editor

Updates that retain a jev classifier return HTTP 400 until the same request sets classifier.model_id, even when changing only the name, routes, or thresholds. The error identifies the legacy provider-key and model reference.

To convert the router in the dashboard:

  1. Create a decision model on its TypeSafe provider key, using the same upstream model name.
  2. Edit the router and select that Decision model.
  3. Save the changes.

For API automation, send the complete semantic block with classifier.model_id set. Omit the legacy provider_key_id and model fields from the classifier.

The control plane still writes the decision model's provider key and model name to the gateway beside model_id. A gateway released before 1.11.0 reads only those, so it calls the same model with the same key but does not report the decision calls for billing.

Route Requests That Fit No Route​

Jev always chooses among the routes you configure. AISIX adds no "none of the above" option, so a request unrelated to every route still goes to whichever route jev finds closest, unless jev's confidence is below min_confidence.

To send unrelated requests somewhere specific, add a catch-all route of your own with a description such as "The request does not fit any of the other routes":

For AISIX Enterprise or AISIX Cloud, reference the target by model ID:

{
"name": "other",
"target_model_id": "YOUR_GENERAL_MODEL_ID",
"description": "The request does not fit any of the other routes"
}

For AISIX OSS, reference the target by its display_name:

{
"name": "other",
"target": "general-chat",
"description": "The request does not fit any of the other routes"
}

When writing descriptions:

  • Keep each description specific. A broadly written description, such as "General questions", absorbs requests that belong nowhere else.
  • Write descriptions in the same language as the traffic they route.

Handle Classifier Failures​

on_failure applies when the jev call fails, times out after timeout_ms, or names no configured route. It also applies when the gateway cannot resolve the classifier's model_id to a direct model on a typesafe provider key. It takes the same values as an embedding router's on_embedding_failure, and defaults to the router's default model.

Edit the router and choose the required On classifier failure behavior. To use a specific fallback, select the specific-model option and its target, then save the router.

Classifier failures routed to a specific fallback model

With fail, the 503 carries error.code semantic_routing_failed and a message naming the router and the reason, such as classifier_timeout or classifier_invalid_response. The access-log line records semantic_fallback="decision_failed" and error_class="semantic_decision_failed". See Semantic Routing Failures for the reasons.

A request with no user text to classify goes to the default model without a jev call, unless its session is pinned to a route.

Next Steps​