Response Caching

Response caching lets AISIX reuse an earlier non-streaming Chat Completions response when a later request has the same cache key. It is gateway-side response caching, not provider prompt caching or the AISIX Cloud configuration snapshot cache. For provider-side caching on Anthropic models, see Anthropic Prompt Caching. To also serve cached responses for similar — not identical — prompts, add Semantic Caching on top of a policy.

In this guide, you will configure a model-scoped cache policy through AISIX Cloud or a declarative resources file. You will then verify cache misses and hits, choose the sharing scope, bypass and purge cached entries, and optionally share cached responses across gateway instances with Redis.

How Response Caching Works

AISIX caches exact non-streaming Chat Completions responses. Streaming responses and other proxy API families do not use this response-cache path.

Cache policies can apply to all eligible requests, one model alias, or one caller API key. Each policy stores responses in the gateway's in-process memory cache or in Redis when Redis is configured at startup.

Before AISIX can cache a response, the gateway must have the selected backend available and the request must match an enabled cache policy.

Cache Key Matching

AISIX creates the cache key from the normalized Chat Completions request that affects the upstream response. Two requests share a cache entry only when the model alias, message roles and normalized content, sampling settings, response length, and extra OpenAI-compatible request options match.

JSON object key order does not affect cache matching, including in nested objects. Array order does affect matching, so two requests with the same tool definitions in a different order use different cache entries.

For a multi-target model, the cache key uses the alias the caller requested, not the target model that served the miss. The request ID and request headers are not part of the cache key.

Whether the caller API key partitions the cache is controlled by the policy's scope field. With the default scope: api_key, entries are private to the API key that stored them; scope: env shares entries across every caller in the environment. See Choose the Sharing Scope.

Choose a Cache Backend

Each cache policy chooses where matching responses are stored:

Backend	Behavior
Memory	Default. Uses the in-process cache on the gateway instance that handled the miss.
Redis	Uses the shared Redis cache configured at gateway startup.

Use memory for single-instance deployments or when node-local cache entries are acceptable. Use Redis when several gateway instances should share cached responses for the same policy. AISIX supports a single Redis endpoint, Redis Cluster, and Redis Sentinel.

Configure an In-Memory Cache Policy

The following example creates a model-scoped policy with the default in-process memory backend. Choose either the AISIX Cloud or open-source configuration path, then use the shared verification procedure. If another enabled cache policy already matches the example requests, disable or delete it before continuing.

Prerequisites

Before starting, prepare the following:

One of these configuration paths:
- AISIX Cloud with an environment, an attached gateway, and a write-scoped admin token. For On-Premises, follow the AISIX Cloud Quickstart. To request Hybrid Cloud access, contact API7.
- An open-source AISIX gateway that loads a declarative resources.yaml file.
A working model alias and caller API key that can send non-streaming Chat Completions requests.
curl. The AISIX Cloud path also uses jq.

Export the gateway connection and resource values used by both paths:

# AISIX_PROXY has no trailing slash or endpoint path such as /v1.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="gpt-4o-mini"
export CACHE_PROMPT="cache-check-$(date +%s)"

AISIX Cloud

Export the control-plane connection details:

# AISIX_CP includes /api and has no trailing slash.
# The local On-Premises quickstart uses http://localhost:8080/api.
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_BASE_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"

Create an in-memory cache policy for the example model alias. Capture the policy ID because the Redis procedure replaces this policy later:

CACHE_POLICY_RESPONSE=$(curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/cache_policies" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "name": "default-chat-cache",
    "enabled": true,
    "backend": "memory",
    "applies_to": "model:${AISIX_MODEL}",
    "ttl_seconds": 3600
  }
EOF
)

export CACHE_POLICY_ID=$(printf '%s' "$CACHE_POLICY_RESPONSE" | jq -r '.cache_policy.id')
printf '%s' "$CACHE_POLICY_RESPONSE" | jq

The policy projects to the attached gateways automatically. The response includes the policy ID, environment ID, selected backend, scope, TTL, and timestamps.

Open-Source AISIX Gateway

Add the cache policy to the resources file that already defines the example model and caller API key:

resources.yaml
cache_policies:
  - name: default-chat-cache
    enabled: true
    backend: memory
    applies_to: "model:gpt-4o-mini"
    ttl_seconds: 3600

The model name in applies_to must match the model's caller-facing display_name. Validate the complete resources file, then reload the gateway. See Reload a Resources File for the runnable Docker workflow.

Verify Cache Miss and Hit Behavior

Define a helper that sends the same request body on each call:

send_cache_request() {
  curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
    -H "Authorization: Bearer ${AISIX_API_KEY}" \
    -H "Content-Type: application/json" \
    --data-binary @- <<EOF
    {
      "model": "${AISIX_MODEL}",
      "messages": [{"role": "user", "content": "${CACHE_PROMPT}"}]
    }
EOF
}

Send the request once:

send_cache_request

The first matching request should include this response header:

x-aisix-cache: miss

The miss means AISIX called the upstream provider and wrote the response into the cache. Repeat the request without changing the model or body:

send_cache_request

The repeated request should include:

x-aisix-cache: hit

The hit means AISIX served the stored response without calling the upstream provider. Change the prompt to confirm that the request body participates in the cache key:

CACHE_PROMPT="${CACHE_PROMPT}-different" send_cache_request

The changed request should return x-aisix-cache: miss.

Redis lets several gateway instances share entries. Make sure Redis is running and reachable from every participating gateway, then add its connection details to config.yaml:

config.yaml
cache:
  redis:
    mode: single
    url: redis://127.0.0.1:6379/

Start or restart each gateway after adding the Redis configuration. Configuring a policy with backend: redis does not establish the Redis connection. If a matching policy selects Redis but cache.redis is absent, AISIX disables caching for those requests instead of silently falling back to memory.

After Redis is available, replace the in-memory policy with a Redis-backed policy through the same management path you used above.

AISIX Cloud

The backend of an existing AISIX Cloud cache policy cannot be changed. Delete the in-memory policy, then create its Redis-backed replacement:

curl -sS -X DELETE \
  "$AISIX_CP/environments/$ENV_ID/cache_policies/$CACHE_POLICY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" | jq

curl -sS -X POST "$AISIX_CP/environments/$ENV_ID/cache_policies" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF | jq
  {
    "name": "shared-chat-cache",
    "enabled": true,
    "backend": "redis",
    "applies_to": "model:${AISIX_MODEL}",
    "ttl_seconds": 3600
  }
EOF

Open-Source AISIX Gateway

Change the policy's backend in resources.yaml:

resources.yaml
cache_policies:
  - name: shared-chat-cache
    enabled: true
    backend: redis
    applies_to: "model:gpt-4o-mini"
    ttl_seconds: 3600

Validate the complete resources file, then reload the gateway before testing the Redis-backed policy.

Verify Redis Cache Behavior

Set a fresh prompt, then run send_cache_request twice. The first request should report a miss and the second should report a hit:

export CACHE_PROMPT="redis-cache-check-$(date +%s)"

send_cache_request
send_cache_request

To confirm that Redis is shared, send the first request to one gateway instance and the identical request to another. Both instances must load the same cache policy and connect to the same Redis deployment.

export AISIX_PROXY_A="https://gateway-a.example.com"
export AISIX_PROXY_B="https://gateway-b.example.com"
export CACHE_PROMPT="shared-cache-check-$(date +%s)"

AISIX_PROXY="$AISIX_PROXY_A" send_cache_request
AISIX_PROXY="$AISIX_PROXY_B" send_cache_request

The request through gateway A should report a miss. The request through gateway B should report a hit. With the memory backend, the second gateway would have its own empty cache and report another miss.

Tune Policy Scope

After verifying the basic cache behavior, tune the policy scope for the traffic you want to cache.

The applies_to field controls which requests match a cache policy:

Value	Scope
`all`	Every eligible non-streaming Chat Completions request.
`model:<alias>`	Requests that use the caller-visible model alias.
`api_key:<id>`	Requests authenticated with the caller API key resource ID.

For AISIX Cloud, use the caller API key ID returned by the Admin API. In resources.yaml, use the key's deterministic derived ID; the resources loader does not resolve a caller API key display_name in this field.

Start with a narrow policy, such as a model-scoped or caller-key-scoped policy. Use a global policy only when every eligible Chat Completions request should participate in response caching.

The applies_to selector controls eligibility — which requests a policy covers. Whether covered requests share entries with each other is the separate scope field, described next.

Avoid overlapping enabled policies for the same requests. When more than one policy matches, AISIX uses the first matching policy to choose the backend and TTL, so overlapping policies can make cache behavior harder to reason about.

Use only the supported matcher forms. The gateway treats an unrecognized prefix as all, which can make a mistyped policy broader than intended regardless of which management path created it.

The policy's scope field selects the sharing boundary for cached entries:

Scope	Behavior
`api_key`	Default. Entries are private to the caller API key that stored them. One caller's responses are never replayed to another caller.
`env`	Every caller API key in the environment shares the entries. Identical eligible requests reuse one entry across callers.

Keep the default api_key scope for general traffic: responses often embed caller-specific context, and per-key partitioning prevents one caller's answer from leaking to another. Choose env for shared-knowledge traffic — FAQ bots, documentation Q&A — where cross-caller reuse is the point and the hit rate benefits from a shared pool.

The scope applies to a semantic layer the same way: with scope: env, a caller can receive an entry stored for another caller's similar — not just identical — request, which widens the review needed before enabling it.

Scope is mutable, so switch the policy you already created rather than adding a second, overlapping one (set CACHE_POLICY_ID to the ID of whichever policy currently covers the model):

curl -sS -X PATCH \
  "$AISIX_CP/environments/$ENV_ID/cache_policies/$CACHE_POLICY_ID" \
  -H "Authorization: Bearer $AISIX_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"scope": "env"}' | jq

In resources.yaml, set the same field on the policy entry (scope: env). Existing policies without the field use api_key.

Bypass the Cache Per Request

A caller can skip the cache for one request with standard Cache-Control request directives — no policy change needed:

Directive	Behavior
`Cache-Control: no-cache`	Skips the cache lookup. The fresh upstream response still refreshes the stored entry.
`Cache-Control: no-store`	Skips the lookup and does not write the response to the cache.

curl -sSi -X POST "$AISIX_PROXY/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Cache-Control: no-cache" \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "model": "${AISIX_MODEL}",
    "messages": [{"role": "user", "content": "${CACHE_PROMPT}"}]
  }
EOF

A bypassed request reports x-aisix-cache: bypass instead of hit or miss. Use no-cache to force-refresh a stale answer; use no-store for requests whose responses should never be cached.

Purge Cached Entries

Purging invalidates every entry stored under a policy — on every gateway instance and, for a policy with a semantic layer, in both matching layers — without deleting or disabling the policy:

curl -sS -X POST \
  "$AISIX_CP/environments/$ENV_ID/cache_policies/$CACHE_POLICY_ID/purge" \
  -H "Authorization: Bearer $AISIX_TOKEN" | jq

The response includes the policy's incremented purge_generation. Entries stored under earlier generations stop being served as soon as the updated configuration reaches each gateway; storage is reclaimed in the background. The AISIX Cloud dashboard exposes the same action as the Purge button on each policy row.

In resources.yaml, add a purge_generation field to the policy entry and increase it whenever you need to invalidate the policy's entries:

resources.yaml
cache_policies:
  - name: default-chat-cache
    enabled: true
    backend: memory
    applies_to: "model:gpt-4o-mini"
    ttl_seconds: 3600
    purge_generation: 1

Purge when cached answers have gone stale ahead of their TTL — after a system-prompt change, a provider-side model update, or a corrected upstream document.

Next Steps

You have configured response caching and verified cache miss and hit behavior. Next:

Add Semantic Caching to also serve cached responses for similar — not identical — prompts.
Enable Anthropic Prompt Caching when you want provider-side discounts for repeated prompt prefixes instead of reusing whole responses.
Continue with Guardrails to add request and response checks before and after provider calls.

How Response Caching Works​

Cache Key Matching​

Choose a Cache Backend​

Configure an In-Memory Cache Policy​

Prerequisites​

AISIX Cloud​

Open-Source AISIX Gateway​

Verify Cache Miss and Hit Behavior​

Share Cache Entries with Redis​

AISIX Cloud​

Open-Source AISIX Gateway​

Verify Redis Cache Behavior​

Verify Cache Sharing Across Gateway Instances​

Tune Policy Scope​

Choose the Sharing Scope​

Bypass the Cache Per Request​

Purge Cached Entries​

Next Steps​

How Response Caching Works

Cache Key Matching

Choose a Cache Backend

Configure an In-Memory Cache Policy

Prerequisites

AISIX Cloud

Open-Source AISIX Gateway

Verify Cache Miss and Hit Behavior

Share Cache Entries with Redis

AISIX Cloud

Open-Source AISIX Gateway

Verify Redis Cache Behavior

Verify Cache Sharing Across Gateway Instances

Tune Policy Scope

Choose the Sharing Scope

Bypass the Cache Per Request

Purge Cached Entries

Next Steps