Semantic Caching

Semantic caching lets AISIX serve a cached response when a new request means the same thing as an earlier one, even when the wording differs. It extends Response Caching: the exact-match layer is always checked first, and a semantic layer answers requests that miss it.

In this guide, you will add a semantic block to a cache policy, verify exact and semantic hits, and purge cached entries. You will also learn the Redis requirements for sharing semantic entries across gateway instances.

How Semantic Caching Works

A cache policy with a semantic block runs two matching layers on each eligible request:

1. Exact layer. The request fingerprint — model alias, normalized messages, sampling settings, and other response-affecting options — is looked up first. An identical repeat request hits here without any embedding call.
2. Semantic layer. On an exact miss, AISIX embeds the request's message text with the policy's embedding model and compares it against stored entries. The nearest entry at or above the similarity threshold is served; otherwise the request goes upstream and the response is stored with its embedding.

Only the messages content is matched by similarity. The model alias, sampling parameters, and other fingerprint fields must still match exactly — a paraphrased prompt sent with a different temperature does not reuse the entry. Requests whose messages contain images, audio, or tool results never match semantically; they keep using the exact layer only.

A semantic hit also refreshes nothing: entries expire on the policy TTL like exact entries.

Embedding Failures Fail Open

Semantic matching is an optimization, never a gate. If the embedding call fails or times out, the request proceeds to the upstream uncached, and the failure is counted on the cache metrics. If the embedding model is deleted or misconfigured, the policy logs a warning and keeps serving exact matches only.

Prerequisites

Before starting, prepare the following:

• A working Response Caching setup: a cache policy path (AISIX Cloud or a declarative resources file), a model alias, and a caller API key.
• An embedding model in the same environment. In AISIX Cloud this is a model of kind embedding; in a resources file it is a model entry carrying an embedding block. The embedding model embeds request text for similarity comparison; its dimensions value fixes the vector size for the policy's entries.
• For Redis-backed policies: a Redis deployment with vector search — see Share Semantic Entries with Redis.

Export the shared connection values:

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="gpt-4o-mini"

Configure a Semantic Cache Policy

The semantic block enables the similarity layer on a cache policy. The threshold is required: it is the minimum cosine similarity, in [0, 1], for a stored entry to be served. Higher is stricter. Values below about 0.9 noticeably increase the risk of serving a wrong answer; 0.92 is a good starting point.

AISIX Cloud

Export the control-plane connection details:

export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_BASE_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Look up the embedding model's ID, then create the policy with a semantic block. The Admin API references the embedding model by resource ID:

export EMBED_MODEL_ID=$(curl -sS "$AISIX_CP/environments/$ENV_ID/models" \
  -H "Authorization: Bearer $AISIX_TOKEN" |
  jq -r '.data[] | select(.kind == "embedding") | .id' | head -1)

curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/cache_policies" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF | jq
  {
    "name": "semantic-chat-cache",
    "enabled": true,
    "backend": "memory",
    "applies_to": "model:${AISIX_MODEL}",
    "ttl_seconds": 3600,
    "semantic": {
      "embedding_model_id": "${EMBED_MODEL_ID}",
      "threshold": 0.92
    }
  }
EOF

The same block can be managed on the Cache Policies page of the AISIX Cloud dashboard: enable Semantic matching, pick the embedding model, and adjust the threshold.

To turn the semantic layer off later without recreating the policy, send "semantic": null in a PATCH request. Omitting the field leaves the configuration unchanged.

Open-Source AISIX Gateway

In a resources file, the semantic block references the embedding model by its display_name:

resources.yaml

models:
  - display_name: text-embedder
    provider: openai
    model_name: text-embedding-3-small
    provider_key: openai-prod
    embedding:
      dimensions: 1536

cache_policies:
  - name: semantic-chat-cache
    enabled: true
    backend: memory
    applies_to: "model:gpt-4o-mini"
    ttl_seconds: 3600
    semantic:
      embedding_model: text-embedder
      threshold: 0.92

Validate the complete resources file, then reload the gateway.

Optional Semantic Settings

Field	Default	Behavior
max_entries	1000	Upper bound on stored entries per policy on the memory backend (1–10,000); the oldest entry is evicted first. Shared backends bound growth by TTL and ignore this value.
embedding_timeout_ms	none	Per-call deadline for the embedding request. On timeout, the request proceeds upstream uncached.

Verify Semantic Hits

Send a request, then repeat the same meaning in different words. The response headers distinguish the layers:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  --data '{"model": "'"$AISIX_MODEL"'", "messages": [{"role": "user", "content": "How do I reset my password?"}]}'

The first request reports a miss and stores the response:

x-aisix-cache: miss

Repeat the identical request. It hits the exact layer:

x-aisix-cache: hit
x-aisix-cache-layer: exact

Now paraphrase the prompt:

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  --data '{"model": "'"$AISIX_MODEL"'", "messages": [{"role": "user", "content": "What are the steps to reset a password?"}]}'

If the similarity clears the threshold, the response comes from the semantic layer without an upstream call:

x-aisix-cache: hit
x-aisix-cache-layer: semantic
x-aisix-cache-similarity: 0.9714

x-aisix-cache-similarity reports the cosine similarity of the matched entry. If the paraphrase misses instead, lower the threshold carefully or verify that both requests use identical sampling parameters.

A semantic hit also backfills the exact layer for the new wording, so repeating the paraphrase hits the exact layer directly.

Tune the Threshold

Start at 0.92 and adjust with real traffic:

• Wrong answers served (different questions treated as the same): raise the threshold toward 0.97–0.99.
• Paraphrases missing (same question re-answered upstream): lower it gradually, watching quality. Avoid going below 0.9 for open-ended traffic.
• The right value depends on the embedding model. When switching embedding models, re-tune the threshold — scores are not comparable across models. Changing the embedding model also invalidates existing semantic entries, since vectors from different models cannot be compared.

Use the x-aisix-cache-similarity header to calibrate: send known-equivalent and known-different prompt pairs and observe their scores.

Choose the Sharing Scope

The policy's scope selects who may share cached entries. It applies to both matching layers:

Scope	Behavior
api_key	Default. Entries are private to the caller API key that stored them — one caller's answers are never replayed to another.
env	Every caller API key in the environment shares the entries.

Keep the default api_key scope unless every caller is equivalent. Semantic matching makes cross-caller reuse riskier than exact matching: a paraphrase of another user's question can surface a response generated for that user's context. Choose env only for shared-knowledge traffic such as FAQ or documentation Q&A, where cross-caller reuse is the point.

Bypass the Cache Per Request

Standard Cache-Control request directives let a caller skip the cache for one request without touching the policy — see Bypass the Cache Per Request. no-cache skips both matching layers and refreshes the stored entry; no-store additionally suppresses the write.

Purge Cached Entries

Purging invalidates every entry stored under a policy — both layers, on every gateway instance — without deleting the policy:

curl -sS -X POST \
  "$AISIX_CP/environments/$ENV_ID/cache_policies/$CACHE_POLICY_ID/purge" \
  -H "Authorization: Bearer $AISIX_TOKEN" | jq

The response returns the policy with an incremented purge_generation. Gateways treat entries from earlier generations as gone the moment the bumped configuration propagates; storage is reclaimed in the background. The dashboard exposes the same action as the Purge button on each policy row.

For resources-file deployments, add a purge_generation field to the policy entry and increase it on each purge:

resources.yaml

cache_policies:
  - name: semantic-chat-cache
    # ...
    purge_generation: 1

Share Semantic Entries with Redis

On the memory backend, each gateway instance keeps its own entries. To share both cache layers across instances, use backend: redis with the gateway's cache.redis startup configuration, as described in Share Cache Entries with Redis.

The semantic layer has additional Redis requirements:

• Redis 8 or later (or a Redis Stack deployment with the search module). The gateway stores embeddings in a vector index and runs KNN similarity queries.
• single or sentinel mode. Redis Cluster is not supported for semantic entries.

At startup, the gateway probes the Redis deployment for vector-search support. If the probe fails — an older Redis, a missing module, or cluster mode — policies on that backend keep serving exact matches only and a warning is logged. Exact-layer sharing keeps working; only the similarity layer is disabled.

The exact and semantic layers stay consistent on Redis: entries are stored under the same key space, purge generations apply to both, and TTLs match the policy.

Observability

Cache behavior is visible at three levels:

• Response headers: x-aisix-cache (hit / miss / bypass), x-aisix-cache-layer (exact / semantic on hits), and x-aisix-cache-similarity (semantic hits only). See Headers and Error Codes.
• Prometheus metrics: aisix_cache_requests_total counts outcomes per policy (hit_exact / hit_semantic / miss / bypass), and the aisix_cache_semantic_* series track embedding latency and failure modes. See the Metrics Reference for hit-rate queries.
• Usage events: cached responses record the serving layer in cache_hit_layer and the matched similarity in cache_similarity, so exported records can attribute saved upstream calls.

Next Steps

You have enabled semantic caching and verified layer-attributed cache hits. Next:

• Review Response Caching for backend selection, applies_to scoping, and the exact-layer fingerprint rules that still gate semantic matches.
• Watch the semantic hit rate with the Metrics Reference and tune the threshold before widening a policy's scope.