Traffic Controls

After the request path is working, traffic controls let you decide who can use it and how it should behave under real application traffic. They bind callers to access rules, protect shared model capacity, enforce spend and content policy, and cache reusable work.

For model traffic, start with Caller API Keys to give each application a governed identity and model access. Applications that already use an identity provider can present a JWT that AISIX resolves to one of those keys.

Some controls run before AISIX calls the provider. Others run after the provider returns a response. The request path looks like this:

CallerApplication request

AISIX AI Gateway

Before the provider

Authenticate the callerCaller API key and model access

Resolve the modelModel alias, IP rules, and route selection

Check input guardrailsBlock or redact early, before spending quota

Apply budget and rate limitsAISIX Cloud budget check, then local limits

Look up the cacheNon-streaming Chat Completions; a hit skips the provider

Cache miss

Upstream · outside the gatewayProvider model APIRetries and failover; model limits re-checked per attempt

After the provider

Check output guardrailsRuns on provider responses and cache hits

Meter and recordUsage metering, cache store, access logs

Streaming pathSkips the cache; enforcing output guardrails scan the buffered stream before release

CallerResponse returned

Request-side controls can stop a request before AISIX calls the provider. Rate limits, AISIX Cloud budgets, and input guardrails decide whether the request is allowed to continue. Cache lookup can return a stored response on a hit; on a miss, AISIX calls the provider.

After the provider returns, output guardrails can inspect the response before AISIX returns it to the caller.

Choose a Control

Start with caller identity, then add the controls required for the workload.

Establish Caller Identity and Access

Caller API Keys authenticate applications and define the models, MCP tools, A2A agents, and passthrough routes they can use. They also carry caller-specific rate limits and provide the principal used for policy matching and usage attribution.

JWT Authentication lets an application present a short-lived token from an OIDC identity provider instead of a long-lived caller key. AISIX verifies the token and resolves it to a caller API key, so the key's access and traffic controls still apply. Use JWT Claim Mappings when groups, departments, or other verified claims should select a shared policy key without maintaining one direct subject binding per identity.

When an internal upstream must authorize on the end user rather than on the gateway, forward the caller's own headers to it after authentication. No caller credential is forwarded by default, and a public model provider should never be sent one.

Protect Capacity and Spend

Goal	Control	Behavior
Limit request volume, token usage, or concurrent work on one key or model	API Key and Model Rate Limits	Request and concurrency limits run before the provider call. Token usage settles after the response reports it.
Apply conditional quotas across teams, members, API keys, models, model names, or providers	Rate Limit Policies	Every matching policy applies alongside the limits configured directly on API keys and models.
Cap AI spend across organizational scopes and application identities	Budgets	AISIX Cloud evaluates matching organization, environment, caller API key, provider key, team, and member budgets before the provider call.

Enforce Content Policy

Guardrails detect, block, or mask sensitive and unsafe content. Input guardrails run before the provider request, output guardrails inspect returned content, and attachments determine which traffic each guardrail covers.

Reuse Provider Work

Choose the cache according to what should be reused:

Goal	Control	Behavior
Reuse an eligible response for the same Chat Completions request	Response Caching	AISIX returns an exact gateway-side cache hit without contacting the provider.
Reuse an eligible response for a meaningfully similar prompt	Semantic Caching	An embedding comparison extends response caching beyond exact request matches.
Let Anthropic reuse a repeated prompt prefix	Anthropic Prompt Caching	AISIX adds provider cache markers to eligible direct Claude requests; Anthropic still generates each response.

Controls compose on the same request. For example, a Chat Completions request can authenticate a caller, resolve its model, evaluate input guardrails, and check an AISIX Cloud budget when configured. AISIX can then reserve rate-limit capacity and look for a cached response. Configure each control independently, then verify the combined path with the same caller API key and model alias the application will use.