Skip to main content
API7.ai Logo
Version: Dev

Semantic Routing with TypeSafe Jev

A semantic router can decide in one of two ways. By default it compares an embedding of the request with each route's examples. With the TypeSafe jev decision backend, AISIX instead asks TypeSafe's hosted jev decision model which of the configured routes fits the request.

For each request, AISIX sends jev the latest user message together with every route's name and description. Jev picks one route and reports its confidence in that pick. The description is what jev chooses among, so this mode needs no example prompts, no embedding model, and no similarity threshold.

Everything after the decision is the same as for an embedding-based router: the picked route's direct model serves the request, x-aisix-route names the route, and a request that no route serves goes to the router's default model.

Before You Use It​

Consider the following before choosing this backend:

  • Gateway version. A jev router requires an AISIX gateway newer than 1.5.0. Gateways at 1.5.0 or earlier cannot load such a router. While one of them is registered in the environment, AISIX Cloud refuses to save a jev router with HTTP 422 and the error code DP_INCOMPATIBLE. See What the Control Plane Checks During the Window.
  • An extra call per request. Each routed request makes one synchronous call to TypeSafe's hosted API before the request is dispatched. The gateway needs outbound access to TypeSafe, and the latest user message is sent to TypeSafe.
  • Language. TypeSafe states that jev is currently less accurate on non-English input.
  • Accounting. The jev call is not recorded as usage or cost in AISIX. Your TypeSafe account bills it separately.
  • Playground helpers. The Test routing and Auto-detect thresholds helpers on the dashboard work only for embedding-based routers, and are not available for a jev router.

Prerequisites​

Before starting, prepare the following:

  • A TypeSafe API key.
  • A default direct model and at least one direct model to use as a route target.
  • A caller API key that can call the semantic router alias.
  • For AISIX Cloud, an environment with an attached gateway newer than 1.5.0 and permission to manage provider keys, models, and caller API keys.
  • For the open-source AISIX gateway, a gateway newer than 1.5.0 and access to the declarative resources file.

Create a TypeSafe Provider Key​

The jev call authenticates with a provider key whose provider is typesafe. Its secret is your TypeSafe API key, which AISIX sends as a bearer token. api_base is optional and defaults to https://api.typesafe.ai.

A typesafe provider key only authenticates a semantic router's decision calls. AISIX Cloud refuses it as the provider key of a direct or embedding model, and refuses to delete it while a semantic router uses it.

AISIX Cloud​

Export the AISIX Cloud connection details and your TypeSafe API key:

# AISIX_CP is the Admin API base URL; include /api and omit a trailing slash.
# The local On-Premises quickstart uses http://localhost:8080/api.
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"
export TYPESAFE_API_KEY="YOUR_TYPESAFE_API_KEY"

Create the provider key, allow it in the environment, and capture its ID:

TYPESAFE_KEY_ID=$(curl -sS -X POST "$AISIX_CP/provider_keys" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "typesafe",
    "provider": "typesafe",
    "api_key": "'"$TYPESAFE_API_KEY"'",
    "allowed_environments": ["'"$ENV_ID"'"]
  }' | jq -r '.provider_key.id')

On the dashboard, create the key on the Provider Keys page with the typesafe provider and allow it in the environment where the router will live.

Open-Source AISIX Gateway​

Add the key to the provider_keys collection and supply the secret through an environment variable:

resources.yaml (TypeSafe provider key)
provider_keys:
  - display_name: typesafe
    provider: typesafe
    api_key: ${TYPESAFE_API_KEY}

Configure a Jev Router​

Every route needs a name, a direct target model, and a description. Write each description as a statement of which requests the route handles, because it is the only thing jev reads about the route. Routes take no examples and no threshold in this mode.

AISIX Cloud​

Export the IDs of the direct models:

export DEFAULT_MODEL_ID="YOUR_DEFAULT_MODEL_ID"
export BILLING_MODEL_ID="YOUR_BILLING_MODEL_ID"
export CODE_MODEL_ID="YOUR_CODE_MODEL_ID"

Create the semantic router with a classifier block:

SEMANTIC_MODEL_ID=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "semantic",
    "display_name": "support-router",
    "semantic": {
      "classifier": {
        "type": "jev",
        "provider_key_id": "'"$TYPESAFE_KEY_ID"'",
        "model": "jev-latest",
        "min_confidence": 0.5,
        "timeout_ms": 3000,
        "on_failure": {"mode": "default"}
      },
      "default_model_id": "'"$DEFAULT_MODEL_ID"'",
      "routes": [
        {
          "name": "billing",
          "target_model_id": "'"$BILLING_MODEL_ID"'",
          "description": "Questions about invoices, payments, refunds, and subscription plans"
        },
        {
          "name": "code",
          "target_model_id": "'"$CODE_MODEL_ID"'",
          "description": "Writing, reviewing, or debugging source code"
        }
      ]
    }
  }' | jq -r '.model.id')

Add SEMANTIC_MODEL_ID to the caller API key's allowed_models list. The saved configuration is projected to attached gateways automatically.

The classifier block accepts these fields:

FieldRequiredDefaultDescription
typeYesThe decision model family. jev is the only value.
provider_key_idYesID of a typesafe provider key allowed in this environment.
modelNojev-latestThe jev model to call.
min_confidenceNo0.5Lowest confidence, from 0 to 1, at which the picked route is used. A pick below it sends the request to the default model.
timeout_msNo3000Deadline for the jev call, in milliseconds. At least 1.
on_failureNo{"mode": "default"}What to do when the call fails. See Handle Classifier Failures.

A router with a classifier rejects the embedding-only fields with HTTP 400: embedding_model_id, threshold, embedding_timeout_ms, and on_embedding_failure on the router, and examples and threshold on a route. It also requires a non-empty description on every route, accepts at most 254 routes, and requires route names to be unique.

AISIX Cloud Dashboard​

On the dashboard Models page, create a semantic router and set Decision backend to TypeSafe jev classifier instead of Embedding similarity. The form then shows the following fields:

  • TypeSafe provider key, which lists only typesafe provider keys allowed in the environment.
  • Decision model, which defaults to jev-latest.
  • Minimum confidence, which defaults to 0.5. A decision below it sends the request to the default model.
  • Timeout (ms), which defaults to 3000.
  • On classifier failure, which falls back to the default model, fails the request, or routes to a specific model.

Each route needs a Description. The router list shows a jev router with a jev: <model> badge, for example jev: jev-latest.

Open-Source AISIX Gateway​

Add the router to the models collection. Reference the TypeSafe provider key by name with classifier.provider_key, or by ID with classifier.provider_key_id. Model references use display_name values:

resources.yaml (jev semantic router)
models:
  - display_name: support-router
    semantic:
      classifier:
        type: jev
        provider_key: typesafe
        model: jev-latest
        min_confidence: 0.5
        timeout_ms: 3000
        on_failure: default
      default: general-chat
      routes:
        - name: billing
          target: billing-chat
          description: Questions about invoices, payments, refunds, and subscription plans
        - name: code
          target: code-chat
          description: Writing, reviewing, or debugging source code

The router must not set embedding_model, embedding_model_id, embedding_timeout_ms, on_embedding_failure, or match, and its routes must not set examples or threshold. The complete resources file must also contain the typesafe provider key and the general-chat, billing-chat, and code-chat direct models. Add support-router to the caller key's allowed_models, then validate and reload the file.

Route Requests That Fit No Route​

Jev always chooses among the routes you configure. AISIX adds no "none of the above" option, so a request unrelated to every route still goes to whichever route jev finds closest, unless jev's confidence is below min_confidence.

To send unrelated requests somewhere specific, add a catch-all route of your own with a description such as "The request does not fit any of the other routes":

{
  "name": "other",
  "target_model_id": "YOUR_GENERAL_MODEL_ID",
  "description": "The request does not fit any of the other routes"
}

When writing descriptions:

  • Keep each description specific. A broadly written description, such as "General questions", absorbs requests that belong nowhere else.
  • Write descriptions in the same language as the traffic they route.

Handle Classifier Failures​

on_failure applies when the jev call fails, times out after timeout_ms, or names no configured route. It takes the same values as an embedding router's on_embedding_failure, and defaults to the router's default model.

In AISIX Cloud, on_failure is an object. Set mode to default to use the default model, to fail to reject the request with 503, or to target with a target_model_id to use another direct model:

{
  "classifier": {
    "type": "jev",
    "provider_key_id": "YOUR_TYPESAFE_KEY_ID",
    "on_failure": {
      "mode": "target",
      "target_model_id": "YOUR_SAFE_MODEL_ID"
    }
  }
}

In the open-source resources file, use default, fail, or an object that names a direct model:

classifier:
  type: jev
  provider_key: typesafe
  on_failure:
    target: safe-chat

A request whose latest user message has no text goes to the default model without a jev call.

Verify Route Selection​

Send a request to the router as described in Verify Route Selection. A request served by a route carries x-aisix-route with the route's name.

The request's access-log line records the decision in semantic_route, semantic_score, and semantic_fallback. For a jev router, semantic_score is jev's confidence in its pick. Use it to tune min_confidence: a request that fell back with semantic_fallback="low_confidence" shows the confidence that fell short. See Read the Semantic Routing Decision.

Next Steps​