ai-cache Enterprise

The ai-cache plugin caches responses from LLM services so that repeated requests are served from the cache instead of calling the upstream model again. This reduces response latency and upstream token usage for repeated prompts.

The plugin supports exact-match caching, where a response is reused only when the normalized request is identical to a previously cached one. It can also use a semantic cache layer that compares prompt embeddings through RediSearch after an exact miss.

Behavior by Request Format​

The plugin keeps each detected request format in separate cache entries.

The gateway identifies each request by checking URI-specific rules before body-only rules:

• Bedrock Converse requires a URI ending in /converse and a messages array.
• Anthropic Messages requires a URI ending in /v1/messages.
• Responses API requires a URI ending in /v1/responses and an input field.
• Chat Completions uses a messages array.
• Embeddings uses input after the earlier rules do not match.
• Other non-empty JSON objects use passthrough after none of the earlier rules match.

Exact-match caching is available in API7 Enterprise 3.9.16+ or 3.10.2+, and in APISIX 3.18.0+.

Semantic and streaming response caching are available in API7 Enterprise 3.9.16+ or 3.10.3+.

How It Works​

The ai-cache plugin must be used together with the ai-proxy or ai-proxy-multi plugin on the same route, because it caches the LLM traffic those plugins proxy.

On each request, the plugin computes a cache key from the detected request format, the request body, and the selected AI instance's configuration. The key is scoped as configured by cache_key. Exact cache entries are stored in Redis with a configurable time-to-live.

For Chat Completions requests, semantic caching runs after an exact cache miss. The plugin embeds the configured prompt window and queries a RediSearch vector index for a sufficiently similar cached response.

The plugin sets the X-AI-Cache-Status response header to one of the following:

• HIT - a valid cached response was found and is returned directly, without calling the upstream. The X-AI-Cache-Age header reports the age of the cached entry in seconds. Semantic hits also return X-AI-Cache-Similarity.
• MISS - no cached response was found. The request is proxied to the upstream, and a successful (HTTP 200) response within max_cache_body_size is cached for future requests.
• BYPASS - caching is skipped for this request, for example because it matches a bypass_on rule, no AI instance was selected, or the response cannot be safely captured.

Complete SSE streaming responses can be cached and replayed with their streaming content type. Streams that use another framing format, such as Bedrock ConverseStream's AWS event-stream format, bypass the cache.

Example​

The following example uses OpenAI as the upstream LLM service and a Redis instance to store the cache. Before proceeding, create an OpenAI account and an API key, and make sure a Redis instance is reachable from the gateway. You can optionally save the key to an environment variable:

export OPENAI_API_KEY=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   # replace with your API key

If you are working with other LLM providers, please refer to the provider's documentation to obtain an API key.

Cache LLM Responses​

The following example demonstrates how to configure ai-cache together with ai-proxy so that repeated, identical requests are served from Redis.

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-cache-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "gpt-4"
        }
      },
      "ai-cache": {
        "redis_host": "127.0.0.1",
        "redis_port": 6379,
        "exact": {
          "ttl": 3600
        }
      }
    }
  }
EOF

services:
  - name: ai-cache-service
    routes:
      - name: ai-cache-route
        uris:
          - /anything
        methods:
          - POST
        plugins:
          ai-proxy:
            provider: openai
            auth:
              header:
                Authorization: "Bearer ${OPENAI_API_KEY}"
            options:
              model: gpt-4
          ai-cache:
            redis_host: 127.0.0.1
            redis_port: 6379
            exact:
              ttl: 3600

adc sync -f adc.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: ai-cache-plugin-config
spec:
  plugins:
    - name: ai-proxy
      config:
        provider: openai
        auth:
          header:
            Authorization: "Bearer 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"
        options:
          model: gpt-4
    - name: ai-cache
      config:
        redis_host: 127.0.0.1
        redis_port: 6379
        exact:
          ttl: 3600
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: ai-cache-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
          method: POST
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: ai-cache-plugin-config

kubectl apply -f ai-cache-ic.yaml

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

The first request is a cache miss and is proxied to OpenAI. You should receive an HTTP/1.1 200 OK response that includes the following header:

X-AI-Cache-Status: MISS

Send the same request again. This time the response is served from the cache without calling the upstream, and includes the cache status and age headers:

X-AI-Cache-Status: HIT
X-AI-Cache-Age: 2