Skip to main content
API7.ai Logo

ai-cache

The ai-cache plugin caches responses from LLM services so that repeated requests are served from the cache instead of calling the upstream model again. This reduces response latency and upstream token usage for repeated prompts.

The plugin supports exact-match caching, where a response is reused only when the normalized request is identical to a previously cached one. It can also use a semantic cache layer that compares prompt embeddings through Redis Search after an exact miss.

Behavior by Request Format​

The plugin keeps each detected request format in separate cache entries.

The gateway identifies each request by checking URI-specific rules before body-only rules:

  • Bedrock Converse requires a URI ending in /converse and a messages array.
  • Anthropic Messages requires a URI ending in /v1/messages.
  • Responses API requires a URI ending in /v1/responses and an input field.
  • Chat Completions uses a messages array.
  • Embeddings uses input after the earlier rules do not match.
  • Other non-empty JSON objects use passthrough after none of the earlier rules match.
Request formatExact-match cacheSemantic cache
Bedrock ConverseSupportedBypassed
Anthropic MessagesSupportedBypassed
Responses APISupportedBypassed
Chat CompletionsSupportedSupported
EmbeddingsSupportedBypassed
Other JSON (passthrough)SupportedBypassed

Exact-match caching was introduced in API7 Enterprise 3.9.16 and 3.10.2, and APISIX 3.18.0.

Semantic and streaming response caching were introduced in API7 Enterprise 3.9.16 and 3.10.3, and APISIX 3.18.0.

How It Works​

The ai-cache plugin must be used together with the ai-proxy or ai-proxy-multi plugin on the same route, because it caches the LLM traffic those plugins proxy.

On each request, the plugin computes a cache key from the detected request format, the request body, and the selected AI instance's configuration. The key is scoped as configured by cache_key. Exact cache entries are stored in Redis with a configurable time-to-live.

For the passthrough protocol, the key also includes the request method, URI, and query string. The method reflects changes made by earlier plugins such as proxy-rewrite.

This isolation was introduced in API7 Enterprise 3.9.20 and 3.10.7. APISIX supports it from 3.19.0.

Requests with the same body but different method, URI, or query values use different exact cache keys. Other protocols keep their existing key structure.

For Chat Completions requests, semantic caching runs after an exact cache miss. The plugin embeds the configured prompt window and queries a Redis Search vector index for a sufficiently similar cached response.

The plugin sets the X-AI-Cache-Status response header to one of the following:

  • HIT - a valid cached response was found and is returned directly, without calling the upstream. The X-AI-Cache-Age header reports the age of the cached entry in seconds. Semantic hits also return X-AI-Cache-Similarity.
  • MISS - no cached response was found. The request is proxied to the upstream, and a successful (HTTP 200) response within max_cache_body_size is cached for future requests.
  • BYPASS - caching is skipped for this request, for example because it matches a bypass_on rule, no AI instance was selected, or the response cannot be safely captured.

Complete SSE streaming responses can be cached and replayed with their streaming content type. A stream is cached only after the plugin receives the client protocol's terminal event, such as [DONE] for OpenAI Chat Completions, message_stop for Anthropic Messages, or response.completed for the OpenAI Responses API. Interrupted or limit-truncated streams are not cached. Streaming and non-streaming requests use separate entries, and a cached stream is replayed immediately rather than with its original token timing. Streams that use another framing format, such as Bedrock ConverseStream's AWS event-stream format, bypass the cache.

caution

Cached prompts and responses can contain sensitive data. Restrict access to Redis, choose an appropriate cache TTL, and use cache_key.include_consumer or cache_key.include_vars when cached responses should not be shared across consumers or request contexts.

Examples​

The following examples use OpenAI as the upstream LLM service and Redis to store cached responses. Before proceeding, create an OpenAI account and an API key. Save the key to an environment variable:

export OPENAI_API_KEY=replace-with-openai-api-key

If you are working with another LLM provider, refer to that provider's documentation to obtain an API key.

The ADC examples define a conventional service upstream because ADC services require one. The ai-proxy plugin handles the matching LLM requests.

Start Redis​

Redis 8 includes Redis Search, so the same server supports both exact and semantic cache examples. Choose the environment that matches the gateway deployment.

Set GATEWAY_CONTAINER to the running APISIX or API7 Gateway container. Create a dedicated network and connect the gateway to it:

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-redis-net
docker network connect gateway-redis-net "$GATEWAY_CONTAINER"

Start Redis on the shared network with an example password:

docker run -d \
  --name redis-server \
  --network gateway-redis-net \
  redis:8.10.2-alpine \
  redis-server --requirepass redis-password

Verify that Redis accepts authenticated connections:

docker exec -e REDISCLI_AUTH=redis-password redis-server redis-cli ping

The command should return PONG.

Set the Redis hostname used by the Admin API and ADC examples:

export REDIS_HOST=redis-server

These evaluation fixtures use tutorial passwords and internal service addresses. For production deployments, restrict network access, manage credentials as secrets, and configure redis_ssl and redis_ssl_verify when the Redis service supports TLS.

Cache LLM Responses​

The following example demonstrates how to configure ai-cache together with ai-proxy so that repeated, identical requests are served from Redis.

Create a route that proxies to OpenAI with ai-proxy and caches responses with ai-cache:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-cache-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4o-mini"
        }
      },
      "ai-cache": {
        "redis_host": "$REDIS_HOST",
        "redis_port": 6379,
        "redis_password": "redis-password",
        "exact": {
          "ttl": 3600
        }
      }
    }
  }
EOF

❶ Attach the OpenAI API key in the Authorization header as a Bearer token.

❷ Specify the name of the model.

❸ Point the cache at your Redis instance.

❹ Cache each response for one hour.

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

The first request is a cache miss and is proxied to OpenAI. You should receive an HTTP/1.1 200 OK response that includes the following header:

X-AI-Cache-Status: MISS

Send the same request again. This time the response is served from the cache without calling the upstream, and includes the cache status and age headers:

X-AI-Cache-Status: HIT
X-AI-Cache-Age: 2

Cache Semantically Similar Prompts​

Semantic caching performs an embedding request after an exact-cache miss and uses Redis Search to find a sufficiently similar cached prompt. This example uses OpenAI for both the LLM and embedding requests.

Semantic caching requires Redis Search. The Redis 8 server started in the previous section includes this capability.

Create a route that enables both cache layers and configures the embedding service:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-cache-semantic-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4o-mini"
        }
      },
      "ai-cache": {
        "redis_host": "$REDIS_HOST",
        "redis_password": "redis-password",
        "layers": ["exact", "semantic"],
        "semantic": {
          "similarity_threshold": 0.92,
          "embedding": {
            "openai": {
              "model": "text-embedding-3-small",
              "api_key": "$OPENAI_API_KEY"
            }
          },
          "vector_search": {
            "redis": {
              "index": "ai-cache"
            }
          }
        }
      }
    }
  }
EOF

Send an initial request:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "What is Apache APISIX?"
      }
    ]
  }'

The first request is an exact and semantic cache miss. The plugin proxies it to the LLM, embeds the prompt, and stores both entries. You should receive HTTP/1.1 200 OK with the following header:

X-AI-Cache-Status: MISS

Send a semantically similar request with different wording:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Can you explain what Apache APISIX is?"
      }
    ]
  }'

If the similarity score meets the configured threshold, the request misses the exact cache but is served from the semantic cache. You should receive HTTP/1.1 200 OK with headers similar to the following:

X-AI-Cache-Status: HIT
X-AI-Cache-Age: 12
X-AI-Cache-Similarity: 0.9487

The similarity value depends on the embedding model and prompts. If the request is a miss, lower the threshold only after comparing scores for prompts that should and should not share responses.