ai-cache

The ai-cache plugin caches responses from LLM services so that repeated requests are served from the cache instead of calling the upstream model again. This reduces response latency and upstream token usage for repeated prompts.

The plugin supports exact-match caching, where a response is reused only when the normalized request is identical to a previously cached one. On the API7 Enterprise 3.9 line from version 3.9.16, it can also use a semantic cache layer that compares prompt embeddings through RediSearch after an exact miss.

Exact-match caching is available in API7 Enterprise from version 3.9.16 on the 3.9 line and from version 3.10.2 on the 3.10 line. It will be available in APISIX from version 3.18.0. Semantic caching and streaming response caching are available in API7 Enterprise from version 3.9.16 on the 3.9 line.

How It Works​

The ai-cache plugin must be used together with the ai-proxy or ai-proxy-multi plugin on the same route, because it caches the LLM traffic those plugins proxy.

On each request, the plugin computes a cache key from the request body (messages and model) and the selected AI instance's configuration, scoped as configured by cache_key. Exact cache entries are stored in Redis with a configurable time-to-live. When semantic caching is enabled and the exact cache misses, the plugin embeds the configured prompt window, queries a RediSearch vector index, and can reuse a sufficiently similar cached response.

The plugin sets the X-AI-Cache-Status response header to one of the following:

• HIT - a valid cached response was found and is returned directly, without calling the upstream. The X-AI-Cache-Age header reports the age of the cached entry in seconds. Semantic hits also return X-AI-Cache-Similarity.
• MISS - no cached response was found. The request is proxied to the upstream, and a successful (HTTP 200) response within max_cache_body_size is cached for future requests.
• BYPASS - caching is skipped for this request, for example because it matches a bypass_on rule, no AI instance was selected, or the response cannot be safely captured.

Streaming responses can be cached after a complete stream is received. Cached streaming responses are replayed as SSE and keep their streaming content type.

Example​

The following example uses OpenAI as the upstream LLM service and a Redis instance to store the cache. Before proceeding, create an OpenAI account and an API key, and make sure a Redis instance is reachable from the gateway. You can optionally save the key to an environment variable:

export OPENAI_API_KEY=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   # replace with your API key

If you are working with other LLM providers, please refer to the provider's documentation to obtain an API key.

Cache LLM Responses​

The following example demonstrates how to configure ai-cache together with ai-proxy so that repeated, identical requests are served from Redis.

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ai-cache-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer '"$OPENAI_API_KEY"'"
          }
        },
        "options": {
          "model": "gpt-4"
        }
      },
      "ai-cache": {
        "redis_host": "127.0.0.1",
        "redis_port": 6379,
        "exact": {
          "ttl": 3600
        }
      }
    }
  }'

services:
  - name: ai-cache-service
    routes:
      - name: ai-cache-route
        uris:
          - /anything
        methods:
          - POST
        plugins:
          ai-proxy:
            provider: openai
            auth:
              header:
                Authorization: "Bearer ${OPENAI_API_KEY}"
            options:
              model: gpt-4
          ai-cache:
            redis_host: 127.0.0.1
            redis_port: 6379
            exact:
              ttl: 3600

adc sync -f adc.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: ai-cache-plugin-config
spec:
  plugins:
    - name: ai-proxy
      config:
        provider: openai
        auth:
          header:
            Authorization: "Bearer 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"
        options:
          model: gpt-4
    - name: ai-cache
      config:
        redis_host: 127.0.0.1
        redis_port: 6379
        exact:
          ttl: 3600
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: ai-cache-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
          method: POST
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: ai-cache-plugin-config

kubectl apply -f ai-cache-ic.yaml

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

The first request is a cache miss and is proxied to OpenAI. You should receive an HTTP/1.1 200 OK response that includes the following header:

X-AI-Cache-Status: MISS

Send the same request again. This time the response is served from the cache without calling the upstream, and includes the cache status and age headers:

X-AI-Cache-Status: HIT
X-AI-Cache-Age: 2