Semantic Caching

Semantic caching lets AISIX serve a cached response when a new request means the same thing as an earlier one, even when the wording differs. It extends Response Caching: the exact-match layer is always checked first, and a semantic layer answers requests that miss it.

In this guide, you will add a semantic block to a cache policy, verify exact and semantic hits, and purge cached entries. You will also learn the Redis requirements for sharing semantic entries across gateway instances.

How Semantic Caching Works

A cache policy with a semantic block runs two matching layers on each eligible request:

Exact layer. The request fingerprint — model alias, normalized messages, sampling settings, and other response-affecting options — is looked up first. An identical repeat request hits here without any embedding call.
Semantic layer. On an exact miss, AISIX embeds the request's message text with the policy's embedding model and compares it against stored entries. The nearest entry at or above the similarity threshold is served; otherwise the request goes upstream and the response is stored with its embedding.

Only the messages content is matched by similarity. The model alias and other fingerprint fields must still match. When present, temperature and top_p are compared at 0.001 precision, so explicit values in the same thousandth-wide bucket are treated as equal. An omitted field remains distinct from an explicit value. Requests whose messages contain images, audio, or tool results never match semantically; they keep using the exact layer only.

A semantic hit also refreshes nothing: entries expire on the policy TTL like exact entries.

Embedding Failures Fail Open

Semantic matching is an optimization, never a gate. If the embedding call fails or times out, the request proceeds to the upstream uncached, and the failure is counted on the cache metrics. If the embedding model is deleted or misconfigured, the policy logs a warning and keeps serving exact matches only.

Prerequisites

Before starting, prepare the following:

A working Response Caching setup: a cache policy path (AISIX Cloud or a declarative resources file), a model alias, and a caller API key.
An embedding model in the same environment. In AISIX Cloud this is a model of kind embedding; in a resources file it is a model entry carrying an embedding block. The embedding model embeds request text for similarity comparison; its dimensions value fixes the vector size for the policy's entries.
For Redis-backed policies: a Redis deployment with vector search — see Share Semantic Entries with Redis.

Export the shared connection values:

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="gpt-4o-mini"

Configure a Semantic Cache Policy

The semantic block enables the similarity layer on an existing cache policy. Add it to the policy that already covers the model instead of creating a second overlapping policy. When several enabled policies match one request, the first matching policy wins, so a later semantic policy might never run.

The threshold is required: it is the minimum cosine similarity, in [0, 1], for a stored entry to be served. Higher is stricter. Values below about 0.9 noticeably increase the risk of serving a wrong answer; 0.92 is a good starting point.

AISIX Cloud

Export the control-plane connection details:

export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_BASE_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Keep CACHE_POLICY_ID from the Response Caching workflow. If you are adding semantics to another existing policy, export that policy's ID:

export CACHE_POLICY_ID="YOUR_EXISTING_CACHE_POLICY_ID"

Look up the embedding model's ID, then patch the existing policy with a semantic block. The AISIX Cloud Admin API references the embedding model by resource ID:

export EMBED_MODEL_ID=$(curl -sS "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" |
  jq -r '.data[] | select(.kind == "embedding") | .id' | head -1)

curl -sS -X PATCH \
  "$AISIX_CP/environments/$ENV_ID/cache_policies/$CACHE_POLICY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF | jq
  {
    "semantic": {
      "embedding_model_id": "${EMBED_MODEL_ID}",
      "threshold": 0.92
    }
  }
EOF

The same block can be managed on the Cache Policies page of the AISIX Cloud dashboard. Edit the existing policy, enable Semantic matching, pick the embedding model, and adjust the threshold.

To turn the semantic layer off later without recreating the policy, send "semantic": null in a PATCH request. Omitting the field leaves the configuration unchanged.

Open-Source AISIX Gateway

Add text-embedder to models, then replace the default-chat-cache policy. Keep the other resources unchanged:

resources.yaml (semantic cache resources)
models:
  - display_name: text-embedder
    provider: openai
    model_name: text-embedding-3-small
    provider_key: openai-prod
    embedding:
      dimensions: 1536

cache_policies:
  - name: default-chat-cache
    enabled: true
    backend: memory
    applies_to: "model:gpt-4o-mini"
    ttl_seconds: 3600
    semantic:
      embedding_model: text-embedder
      threshold: 0.92

The semantic block references the embedding model by its display_name.

Validate the complete resources file, then reload the gateway.

The example continues the in-memory default-chat-cache policy from the response-caching workflow. If the existing policy uses another name or the Redis backend, keep those fields unchanged and add only the semantic block.

Optional Semantic Settings

Field	Default	Behavior
`max_entries`	`1000`	Upper bound on stored entries per policy on the `memory` backend (1–10,000); the oldest entry is evicted first. Shared backends bound growth by TTL and ignore this value.
`embedding_timeout_ms`	none	Per-call deadline for the embedding request. On timeout, the request proceeds upstream uncached.

Verify Semantic Hits

Send a request, then repeat the same meaning in different words. The response headers distinguish the layers:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  --data '{"model": "'"$AISIX_MODEL"'", "messages": [{"role": "user", "content": "How do I reset my password?"}]}'

The first request reports a miss and stores the response:

x-aisix-cache: miss

Repeat the identical request. It hits the exact layer:

x-aisix-cache: hit
x-aisix-cache-layer: exact

Now paraphrase the prompt:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  --data '{"model": "'"$AISIX_MODEL"'", "messages": [{"role": "user", "content": "What are the steps to reset a password?"}]}'

If the similarity clears the threshold, the response comes from the semantic layer without an upstream call:

x-aisix-cache: hit
x-aisix-cache-layer: semantic
x-aisix-cache-similarity: 0.9714

x-aisix-cache-similarity reports the cosine similarity of the matched entry. If the paraphrase misses instead, lower the threshold carefully or verify that both requests use identical sampling parameters.

A semantic hit also backfills the exact layer for the new wording, so repeating the paraphrase hits the exact layer directly.

Tune the Threshold

Start at 0.92 and adjust with real traffic:

Wrong answers served (different questions treated as the same): raise the threshold toward 0.97–0.99.
Paraphrases missing (same question re-answered upstream): lower it gradually, watching quality. Avoid going below 0.9 for open-ended traffic.
The right value depends on the embedding model. When switching embedding models, re-tune the threshold — scores are not comparable across models. Changing the embedding model also invalidates existing semantic entries, since vectors from different models cannot be compared.

Use the x-aisix-cache-similarity header to calibrate: send known-equivalent and known-different prompt pairs and observe their scores.

The policy's scope selects who may share cached entries. It applies to both matching layers:

Scope	Behavior
`api_key`	Default. Entries are private to the caller API key that stored them — one caller's answers are never replayed to another.
`env`	Every caller API key in the environment shares the entries.

Keep the default api_key scope unless every caller is equivalent. Semantic matching makes cross-caller reuse riskier than exact matching: a paraphrase of another user's question can surface a response generated for that user's context. Choose env only for shared-knowledge traffic such as FAQ or documentation Q&A, where cross-caller reuse is the point.

Bypass the Cache Per Request

Standard Cache-Control request directives change cache behavior for one request without touching the policy. no-cache skips both matching layers and refreshes the stored entry. no-store leaves both lookups enabled but suppresses a write after a miss. See Bypass the Cache Per Request.

Purge Cached Entries

Purging invalidates every entry stored under a policy — both layers, on every gateway instance — without deleting the policy:

curl -sS -X POST \
  "$AISIX_CP/environments/$ENV_ID/cache_policies/$CACHE_POLICY_ID/purge" \
  -H "Authorization: Bearer $AISIX_TOKEN" | jq

The response returns the policy with an incremented purge_generation. Gateways treat entries from earlier generations as gone the moment the bumped configuration propagates; storage is reclaimed in the background. The dashboard exposes the same action as the Purge button on each policy row.

For resources-file deployments, add a purge_generation field to the existing policy entry and increase it on each purge:

resources.yaml (cache purge)
cache_policies:
  - name: default-chat-cache
    # ...
    purge_generation: 1

Keep this value monotonically increasing. Never lower or omit it after a purge; doing so can make older exact-cache entries reachable again until they expire. See Purge Cached Entries for the full requirement.

On the memory backend, each gateway instance keeps its own entries. To share both cache layers across instances, use backend: redis with the gateway's cache.redis startup configuration, as described in Share Cache Entries with Redis.

The semantic layer has additional Redis requirements:

Redis 8 or later (or a Redis Stack deployment with the search module). The gateway stores embeddings in a vector index and runs KNN similarity queries.
single or sentinel mode. Redis Cluster is not supported for semantic entries.
RESP2. Do not add protocol=resp3 to a cache.redis URL. The semantic cache probe rejects RESP3 because its vector-search reply parser requires RESP2.

At startup, the gateway probes the Redis deployment for vector-search and RESP2 support. An older Redis version, a missing module, cluster mode, or RESP3 causes the probe to fail. Policies on that backend then keep serving exact matches only and log a warning. Exact-layer sharing keeps working; only the similarity layer is disabled.

The exact and semantic layers stay consistent on Redis: entries are stored under the same key space, purge generations apply to both, and TTLs match the policy.

Observability

Cache behavior is visible at three levels:

Response headers: x-aisix-cache (hit / miss / bypass), x-aisix-cache-layer (exact / semantic on hits), and x-aisix-cache-similarity (semantic hits only). See Headers and Error Codes.
Prometheus metrics: aisix_cache_requests_total counts outcomes per policy (hit_exact / hit_semantic / miss / bypass), and the aisix_cache_semantic_* series track embedding latency and failure modes. See the Metrics Reference for hit-rate queries.
Usage events: cached responses record the serving layer in cache_hit_layer and the matched similarity in cache_similarity, so exported records can attribute saved upstream calls.

Next Steps

You have enabled semantic caching and verified layer-attributed cache hits. Next:

Review Response Caching for backend selection, applies_to scoping, and the exact-layer fingerprint rules that still gate semantic matches.
Watch the semantic hit rate with the Metrics Reference and tune the threshold before widening a policy's scope.

How Semantic Caching Works​

Embedding Failures Fail Open​

Prerequisites​

Configure a Semantic Cache Policy​

AISIX Cloud​

Open-Source AISIX Gateway​

Optional Semantic Settings​

Verify Semantic Hits​

Tune the Threshold​

Choose the Sharing Scope​

Bypass the Cache Per Request​

Purge Cached Entries​

Share Semantic Entries with Redis​

Observability​

Next Steps​