Skip to main content
Version: 1.11.0

Anthropic Prompt Caching

Automatic Anthropic prompt caching lets AISIX add cache markers to eligible requests for a direct Claude model on the Anthropic API, Amazon Bedrock, or Vertex AI. Callers can receive provider cache discounts without adding the markers themselves. This is provider-side caching for repeated prompt prefixes, not gateway-side Response Caching, which stores and reuses complete responses.

Enable automatic prompt caching on a direct Claude model, choose a lifetime, then send two requests with a stable prefix to confirm cache creation and reads.

How Automatic Prompt Caching Works​

Claude prompt caching requires a cache_control marker on each eligible request. Callers can place markers themselves, or AISIX can add them automatically for a configured model. The provider can serve a marked prefix from cache at a reduced input rate on later requests; without a marker, it processes the prompt at the standard input rate.

When automatic prompt caching is enabled on a model, AISIX adds up to two markers to requests that contain no caller-supplied markers:

  • If the request has a nonempty system block, AISIX marks its last block to cache the tools and system prefix.
  • If the final message has a nonempty last content block, AISIX marks that block to cache the conversation prefix. Because the marker is attached to the final turn, the cached prefix advances and new content is cached incrementally as the conversation grows.

AISIX adds at most these two markers, and only to a request that carried none of its own, so the provider's four-breakpoint-per-request limit is not exceeded.

If the caller already sent any cache_control marker, AISIX respects the caller's strategy and adds nothing. Provider adapters can still translate marker fields as described in Scope and Limitations.

Prerequisites​

Before starting, prepare the following:

  • A provider key for Anthropic, AWS Bedrock, or Gemini Enterprise Agent Platform, a direct Claude model on it, and a caller API key that is allowed to use the model.
  • A real provider credential for verifying provider cache creation and reads. The configuration shape can be validated without sending a live provider request.
  • curl and jq for the verification requests.
  • For AISIX Enterprise or AISIX Cloud, prepare an environment with an attached gateway. For AISIX Enterprise, follow the AISIX Enterprise getting-started guide; to request AISIX Cloud access, contact API7.
  • For Dashboard workflows, sign in with permission to update models.
  • For Admin API workflows, obtain the direct model's resource ID and a write-scoped token from an identity allowed to update models.
  • For AISIX OSS, prepare a gateway that loads a complete declarative resources.yaml file. For verification through exported telemetry, configure an observability exporter; alternatively, inspect the OpenAI-compatible response fields described below.

Enable Automatic Prompt Caching​

Automatic prompt caching is a model-level setting and is disabled by default. Choose either the control-plane or AISIX OSS configuration path below.

Automatic prompt caching is supported only on direct Claude models:

  • On the Anthropic API, a model whose provider key uses the anthropic adapter.
  • On Amazon Bedrock, a model whose model_name starts with anthropic. after any region or cross-region inference-profile prefix such as us. or global.. For an ARN, AISIX reads the model ID in its last segment.
  • On Vertex AI, a model whose model_name starts with claude-.

AISIX Enterprise and AISIX Cloud reject automatic prompt caching with HTTP 400 for unsupported models. These include Amazon Nova, other Bedrock and Vertex AI publishers, and routing, ensemble, semantic, or embedding models. An open-source gateway ignores the setting on those models. A gateway at 1.6.0 or earlier loads the setting but adds markers only for the Anthropic API.

Update the direct Claude model that serves the application.

  1. Open the environment and select Models.
  2. Select Edit on the direct Claude model, then expand Automatic prompt caching.
  3. Select Enable automatic prompt caching. Keep the default 5 minutes lifetime for this example.
Automatic prompt caching enabled on a direct Claude model with a five-minute lifetime
  1. Select Save changes.

To disable it later, edit the model, expand the same section, and clear Enable automatic prompt caching.

Automatic prompt caching disabled on a direct Claude model

Select Save changes.

Configuration Fields​

Both configuration paths use the fields below. enabled is required whenever the object contains configuration fields.

FieldTypeDescription
enabledbooleanWhether AISIX injects prompt-cache markers for this model.
ttlstringCache lifetime for injected markers: 5m (default when omitted) or 1h. Some Claude models on Amazon Bedrock accept only the 5-minute cache, and Bedrock rejects a request to such a model that asks for 1h.

Choose a Cache Lifetime​

Anthropic supports two cache lifetimes, and AISIX injects the one you select:

LifetimettlCache creation costCache read cost
5 minutes (default)5m1.25x the base input rateModel-specific discounted input rate
1 hour1h2x the base input rateModel-specific discounted input rate

For a given model, the cache-read rate is the same for both lifetimes. Check the provider pricing for its model-specific discount. The difference is the cache creation cost and how long the prefix remains cached. A cache read also refreshes the entry, extending its lifetime while the prefix remains active.

Use 5m by default. It has the lower cache creation cost and breaks even after a single cache read. Choose 1h only when the same prefix is reused after gaps longer than five minutes. For example, the higher creation cost can be worthwhile for long agent sessions with idle periods between turns because it avoids repeatedly creating the cache entry.

Change Cache lifetime to 1 hour in the model's Automatic prompt caching section.

Automatic prompt caching enabled with a one-hour lifetime

Select Save changes.

Some Claude models on Amazon Bedrock support only the 5-minute lifetime. Bedrock rejects ttl: 1h for those models, whether the caller supplied it or AISIX injected it. AISIX forwards the configured lifetime without changing it.

When the provider reports a lifetime split, the usage event records how many cache creation tokens were written at each lifetime.

For a Claude-family platform price, AISIX Enterprise and AISIX Cloud bill one-hour writes at twice the input rate unless that price defines a one-hour write rate. A sale or model-group price without its own one-hour rate instead uses its active cache-write rate. See Model Pricing and Cache Creation by Lifetime.

For current multipliers and supported lifetimes, see the Anthropic prompt caching documentation.

Verify Cache Creation and Reads​

Automatic prompt caching pays off when a large, stable prefix is reused. To verify it, send two requests that share the same prefix and inspect the cache creation and read tokens.

Anthropic requires a model-specific minimum prompt length for caching. A shorter prefix is processed without caching and returns no error, so use a prefix that meets the minimum for the selected Claude model.

Export the gateway connection and request values:

# AISIX_PROXY has no trailing slash or endpoint path such as /v1.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="YOUR_DIRECT_MODEL_ALIAS"

Send the first request with a sufficiently long, stable system prompt. Replace the placeholder with a real prompt, and use the exact same text in the second request:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
-H "Authorization: Bearer ${AISIX_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"model": "'"${AISIX_MODEL}"'",
"messages": [
{"role": "system", "content": "<a stable system prompt that meets the model minimum>"},
{"role": "user", "content": "First question"}
]
}' | jq

The first request should record cache creation tokens. Send a second request that keeps the exact same system prompt:

curl -sS -X POST "$AISIX_PROXY/v1/chat/completions" \
-H "Authorization: Bearer ${AISIX_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"model": "'"${AISIX_MODEL}"'",
"messages": [
{"role": "system", "content": "<a stable system prompt that meets the model minimum>"},
{"role": "user", "content": "Second question"}
]
}' | jq

The second request should usually record cache read tokens. On Amazon Bedrock, provider placement or overlapping requests can produce another cache write instead of a hit; see Prompt caching for faster model inference.

OpenAI-compatible clients can inspect the response directly. Chat Completions returns the counters under usage.prompt_tokens_details:

Chat Completions usage (partial)
{
"prompt_tokens_details": {
"cached_tokens": 800,
"cache_creation_tokens": 200
}
}

On /v1/responses, the same fields appear under usage.input_tokens_details. These fields project the provider's counters into the client response. Exported usage records retain the provider's accounting; see Token Usage.

Open the environment's Logs page and expand each request. The first request shows Cache creation tokens, while the second shows Cache read tokens. A request can report both when it reads a cached prefix and creates a new suffix. Counters appear when their values are greater than zero.

Request details showing positive cache read and creation token counters

The token_type="total" gateway metric also includes Anthropic cache creation and read tokens, but it does not separate them.

AISIX records cache creation and read tokens separately from ordinary input tokens. In AISIX Enterprise and AISIX Cloud, provider-discounted cache reads are reflected in budgets and spend reporting. Both cache token types count toward AISIX token-based rate limits.

Scope and Limitations​

Request path or providerPrompt-cache behavior
/v1/chat/completions or /v1/responses to supported Claude modelsAISIX can add automatic markers to streaming and non-streaming requests.
/v1/messages to Claude on Anthropic, Bedrock, or Vertex AIAISIX does not add automatic markers. Caller-supplied markers pass through the platform conversion described below.
Claude through Amazon BedrockAISIX keeps only type and ttl on each caller-supplied or injected marker.
Claude through Vertex AIAISIX removes the marker's scope field.
Bedrock ConverseA marker becomes a cachePoint after the marked content. Amazon Nova receives checkpoints in system and messages without a ttl; other publishers receive none. Automatic prompt caching does not cover Nova.
  • Anthropic isolates caches by its organization and workspace boundaries, not by AISIX organization or caller API key. Callers that share one Anthropic provider key can therefore share the same provider-side cache. For cache isolation between tenants, use provider keys from separate Anthropic workspaces.
  • Cache read tokens do not count toward Anthropic's input-token-per-minute rate limit. This provider behavior is separate from AISIX token-based rate limits, which include cache read and cache creation tokens.

Next Steps​

You have enabled automatic prompt caching and verified cache creation and reads. Next:

  • In AISIX Enterprise and AISIX Cloud, track the resulting cost in Budgets.
  • Add gateway-side Response Caching when identical requests can reuse the complete response.