Skip to main content
API7.ai Logo

ai-cache

The ai-cache plugin caches responses from LLM services so that repeated requests are served from the cache instead of calling the upstream model again. This reduces response latency and upstream token usage for repeated prompts.

The plugin supports exact-match caching, where a response is reused only when the normalized request is identical to a previously cached one. It can also use a semantic cache layer that compares prompt embeddings through RediSearch after an exact miss.

Behavior by Request Format​

The plugin keeps each detected request format in separate cache entries.

The gateway identifies each request by checking URI-specific rules before body-only rules:

  • Bedrock Converse requires a URI ending in /converse and a messages array.
  • Anthropic Messages requires a URI ending in /v1/messages.
  • Responses API requires a URI ending in /v1/responses and an input field.
  • Chat Completions uses a messages array.
  • Embeddings uses input after the earlier rules do not match.
  • Other non-empty JSON objects use passthrough after none of the earlier rules match.
Request formatExact-match cacheSemantic cache
Bedrock ConverseSupportedBypassed
Anthropic MessagesSupportedBypassed
Responses APISupportedBypassed
Chat CompletionsSupportedSupported
EmbeddingsSupportedBypassed
Other JSON (passthrough)SupportedBypassed

Exact-match caching was introduced in API7 Enterprise 3.9.16 and 3.10.2, and APISIX 3.18.0.

Semantic and streaming response caching were introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.

How It Works​

The ai-cache plugin must be used together with the ai-proxy or ai-proxy-multi plugin on the same route, because it caches the LLM traffic those plugins proxy.

On each request, the plugin computes a cache key from the detected request format, the request body, and the selected AI instance's configuration. The key is scoped as configured by cache_key. Exact cache entries are stored in Redis with a configurable time-to-live.

For Chat Completions requests, semantic caching runs after an exact cache miss. The plugin embeds the configured prompt window and queries a RediSearch vector index for a sufficiently similar cached response.

The plugin sets the X-AI-Cache-Status response header to one of the following:

  • HIT - a valid cached response was found and is returned directly, without calling the upstream. The X-AI-Cache-Age header reports the age of the cached entry in seconds. Semantic hits also return X-AI-Cache-Similarity.
  • MISS - no cached response was found. The request is proxied to the upstream, and a successful (HTTP 200) response within max_cache_body_size is cached for future requests.
  • BYPASS - caching is skipped for this request, for example because it matches a bypass_on rule, no AI instance was selected, or the response cannot be safely captured.

Complete SSE streaming responses can be cached and replayed with their streaming content type. A stream is cached only after the plugin receives the client protocol's terminal event, such as [DONE] for OpenAI Chat Completions, message_stop for Anthropic Messages, or response.completed for the OpenAI Responses API. Interrupted or limit-truncated streams are not cached. Streaming and non-streaming requests use separate entries, and a cached stream is replayed immediately rather than with its original token timing. Streams that use another framing format, such as Bedrock ConverseStream's AWS event-stream format, bypass the cache.

caution

Cached prompts and responses can contain sensitive data. Restrict access to Redis, choose an appropriate cache TTL, and use cache_key.include_consumer or cache_key.include_vars when cached responses should not be shared across consumers or request contexts.

Examples​

The following example uses OpenAI as the upstream LLM service and a Redis instance to store the cache. Before proceeding, create an OpenAI account and an API key, and make sure a Redis instance is reachable from the gateway. You can optionally save the key to an environment variable:

export OPENAI_API_KEY=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   # replace with your API key

If you are working with other LLM providers, please refer to the provider's documentation to obtain an API key.

Cache LLM Responses​

The following example demonstrates how to configure ai-cache together with ai-proxy so that repeated, identical requests are served from Redis.

Create a route that proxies to OpenAI with ai-proxy and caches responses with ai-cache:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-cache-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4"
        }
      },
      "ai-cache": {
        "redis_host": "127.0.0.1",
        "redis_port": 6379,
        "exact": {
          "ttl": 3600
        }
      }
    }
  }
EOF

❶ Attach the OpenAI API key in the Authorization header as a Bearer token.

❷ Specify the name of the model.

❸ Point the cache at your Redis instance.

❹ Cache each response for one hour.

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

The first request is a cache miss and is proxied to OpenAI. You should receive an HTTP/1.1 200 OK response that includes the following header:

X-AI-Cache-Status: MISS

Send the same request again. This time the response is served from the cache without calling the upstream, and includes the cache status and age headers:

X-AI-Cache-Status: HIT
X-AI-Cache-Age: 2

Cache Semantically Similar Prompts​

Semantic caching performs an embedding request after an exact-cache miss and uses RediSearch to find a sufficiently similar cached prompt. This example uses OpenAI for both the LLM and embedding requests.

caution

Semantic caching requires Redis Stack with the RediSearch module. A standard Redis server can store exact cache entries but cannot perform the vector search required by the semantic layer.

Create a route that enables both cache layers and configures the embedding service:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-cache-semantic-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4o-mini"
        }
      },
      "ai-cache": {
        "redis_host": "127.0.0.1",
        "layers": ["exact", "semantic"],
        "semantic": {
          "similarity_threshold": 0.92,
          "embedding": {
            "openai": {
              "model": "text-embedding-3-small",
              "api_key": "$OPENAI_API_KEY"
            }
          },
          "vector_search": {
            "redis": {
              "index": "ai-cache"
            }
          }
        }
      }
    }
  }
EOF

Send an initial request:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "What is Apache APISIX?"
      }
    ]
  }'

The first request is an exact and semantic cache miss. The plugin proxies it to the LLM, embeds the prompt, and stores both entries. You should receive HTTP/1.1 200 OK with the following header:

X-AI-Cache-Status: MISS

Send a semantically similar request with different wording:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Can you explain what Apache APISIX is?"
      }
    ]
  }'

If the similarity score meets the configured threshold, the request misses the exact cache but is served from the semantic cache. You should receive HTTP/1.1 200 OK with headers similar to the following:

X-AI-Cache-Status: HIT
X-AI-Cache-Age: 12
X-AI-Cache-Similarity: 0.9487

The similarity value depends on the embedding model and prompts. If the request is a miss, lower the threshold only after comparing scores for prompts that should and should not share responses.