API Key and Model Rate Limits

Rate limits protect provider capacity and prevent one caller or model from consuming a shared quota. Configure a limit on a caller API key when the quota belongs to that caller, or on a model when every caller of that model should share the quota.

These limits belong to the API key or model on which they are configured. Rate limit policies are separate rules that can match teams, members, API keys, models, model names, or providers and divide matching traffic into independent quota buckets. A request can match API key limits, model limits, and policies at the same time. Every matching limit applies, and the gateway returns 429 before calling the provider if any bucket is exhausted.

This guide shows how to configure limits on API keys and models, verify enforcement, and choose shared counter storage. AISIX Cloud and the open-source AISIX gateway support the same capabilities through different management paths.

Prerequisites

Before starting, prepare the following:

• A working model alias and caller API key that can send proxy requests. If you have not configured them yet, see Provider Keys, Model Aliases, and Caller API Keys.
• For AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. Follow the AISIX Cloud Quickstart for a local evaluation, or contact API7 to request Hybrid Cloud access.
• For the open-source AISIX gateway, a declarative resources file that you can validate and reload. See Reload a Resources File.

Export the gateway request values used to verify any example on this page:

# AISIX_PROXY has no trailing slash or endpoint path.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_MODEL="gpt-4o-prod"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"

For the AISIX Cloud examples, also export the control-plane connection values and the resource IDs you want to limit:

# AISIX_CP includes /api and has no trailing slash.
# The local On-Premises quickstart uses http://localhost:8080/api.
export AISIX_CP="YOUR_AISIX_CLOUD_ADMIN_API_BASE_URL"
export AISIX_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="YOUR_ENVIRONMENT_ID"
# API_KEY_ID identifies the caller key whose plaintext is AISIX_API_KEY.
export API_KEY_ID="YOUR_CALLER_API_KEY_ID"
# MODEL_ID identifies the model whose alias is AISIX_MODEL.
export MODEL_ID="YOUR_MODEL_ID"

Choose a Rate-Limit Configuration

You can configure a rate limit in two ways:

• Attach a rate_limit block directly to a caller API key or model. Use a caller limit when one application or tenant needs its own quota, or a model limit when every caller of an alias should share the quota.
• Create a rate-limit policy when you need a separately managed quota, conditional matching, or independent buckets for teams, members, API keys, models, or providers.

A caller API key can also carry a separate request and concurrency limit for each MCP server it calls. See MCP Rate Limits and Budgets.

Every matching limit applies. For example, a request can consume capacity from its caller API key, resolved model, team, and member policies at the same time. The request continues only when every matching bucket has capacity.

Configure a Caller API Key Limit

The following examples limit one caller API key to one request per minute.

AISIX Cloud

In the dashboard, open API keys, expand the caller API key, and use its Rate limits controls. The Admin API provides the same configuration for automation.

Update the caller API key with a one-request-per-minute limit:

curl -sS -X PATCH "${AISIX_CP}/environments/${ENV_ID}/api_keys/${API_KEY_ID}" \
  -H "Authorization: Bearer ${AISIX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "rate_limit": {
      "rpm": 1
    }
  }'

The change projects to attached gateways automatically.

Open-Source AISIX Gateway

Add the same limit to your existing caller API key entry in the declarative resources file. Keep its current display_name, key_env, and allowed_models values. The following example uses the values from the caller API key guide:

resources.yaml

api_keys:
  - display_name: chat-app
    key_env: CALLER_API_KEY
    allowed_models:
      - gpt-4o-prod
    rate_limit:
      rpm: 1

Make the caller credential available to the validation command:

# Use the same plaintext value as the verification requests.
export CALLER_API_KEY="${AISIX_API_KEY}"

If AISIX is installed on the host, validate the complete resources file before reloading the gateway:

aisix validate --resources resources.yaml

For the Docker setup in the open-source quickstart, use the container-based validation and reload commands. The quickstart already makes CALLER_API_KEY available to the container.

Configure a Model Limit

Use a model limit when every caller of one alias should share the same quota. The following examples allow one request per minute across all caller API keys that use gpt-4o-prod.

AISIX Cloud

In the dashboard, open Models, edit the model, and use the Rate limits fields. The Admin API provides the same configuration for automation.

Update the model:

curl -sS -X PATCH "${AISIX_CP}/environments/${ENV_ID}/models/${MODEL_ID}" \
  -H "Authorization: Bearer ${AISIX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "rate_limit": {
      "rpm": 1
    }
  }'

Open-Source AISIX Gateway

Add the limit to the direct model entry and retain its existing provider and provider-key settings:

resources.yaml

models:
  - display_name: gpt-4o-prod
    provider: openai
    model_name: gpt-4o
    provider_key: openai-prod
    rate_limit:
      rpm: 1

Validate the complete resources file and reload the gateway. Requests through different caller API keys now draw from the same model bucket.

Verify Rate Limiting

The preceding caller and model examples use the same one-request-per-minute limit. Configure and test one direct limit at a time so the 429 response identifies the limit under test. Use the same request sequence to verify a rate-limit policy.

After the AISIX Cloud change reaches the gateway, or after you reload the open-source resources file, send three requests:

for i in 1 2 3; do
  printf "request %s: " "${i}"
  curl -sS -o /dev/null -w "%{http_code}\n" -X POST "${AISIX_PROXY}/v1/chat/completions" \
    -H "Authorization: Bearer ${AISIX_API_KEY}" \
    -H "Content-Type: application/json" \
    --data-binary @- <<EOF
    {
      "model": "${AISIX_MODEL}",
      "messages": [
        {
          "role": "user",
          "content": "Hello from AISIX."
        }
      ]
    }
EOF
done

The first request should reach the upstream model. The remaining requests should exceed the limit during the same fixed window:

request 1: 200
request 2: 429
request 3: 429

An OpenAI-compatible rate-limit rejection returns 429 Too Many Requests with the proxy error format:

{
  "error": {
    "message": "request limit exceeded (requests)",
    "type": "rate_limit_exceeded"
  }
}

Request-count and token-limit rejections include Retry-After because their fixed windows provide a reset time. A concurrency rejection has no window-based retry hint. Successful Chat Completions responses can also include caller-key rate-limit state; see Headers and Error Codes.

Rate Limit Fields and Counter Behavior

Caller API keys and model aliases use the same rate_limit fields. Each field is optional. When a field is omitted, AISIX does not enforce that dimension.

Field	Limit	Fixed Window
rps	Requests per second	1 second
rpm	Requests per minute	60 seconds
rph	Requests per hour	3,600 seconds
rpd	Requests per day	86,400 seconds
tpm	Tokens per minute	60 seconds
tpd	Tokens per day	86,400 seconds
concurrency	In-flight requests	Not windowed

AISIX checks and records the three limit dimensions at different points while processing a request:

Dimension	When AISIX Checks It	When AISIX Records Usage
Request count	Before the provider call	During the pre-provider checks; an earlier matching layer can increment before a later layer rejects the request
Tokens	Before the provider call, using tokens already recorded in the current window	After the upstream response reports usage
Concurrency	Before the provider call	Held until the response or stream completes, then released

Token limits include prompt and completion tokens. For Anthropic traffic, they also include prompt cache creation and cache read tokens, which Anthropic reports separately from input tokens. OpenAI cached tokens are already included in prompt tokens, so AISIX does not count them twice.

Because the provider reports usage after generating a response, the request that crosses a token limit can finish successfully. Its token usage can exhaust the current window and cause a later request to be rejected.

Token limits attached directly to caller API keys or models support minute and day windows only. There is no per-second or per-hour token field for these resources. Classic rate-limit policies enforce token caps only with a minute window. AISIX Cloud rejects token caps with other windows. For the open-source AISIX gateway, the resources schema accepts them, but the runtime does not enforce the cap.

How Model Limits Apply

If a request matches both caller and model limits, AISIX reserves capacity in both buckets. Exhausting either bucket rejects the request.

Model limits also apply when a direct model serves as the target of a routing model or semantic router. Routing dispatch skips a target that is over its own limit and continues with the remaining targets; when every target is over its limit, the request returns 429. Semantic dispatch enforces the selected target's limit and returns 429 when that target has no capacity. Direct and routed requests to the same model share its limit bucket.

Model limits also apply on provider passthrough when the request body names the model by its alias or provider-native name. Passthrough shares the same limit bucket as modeled routes and enforces request-count and concurrency limits. It relays response bodies verbatim and does not parse provider-reported token usage, so tpm and tpd never count passthrough traffic. An exhausted token window still rejects passthrough requests until it resets.

Choose Rate-Limit Counter Storage

Gateway startup configuration determines whether rate-limit counters are local to one process or shared across gateway instances. The setting applies to limits attached directly to resources and to reusable policies enforced by that gateway.

Backend	Counter Scope	Use When
Memory, the default	One gateway process	A single instance enforces the limit, or per-instance quotas are acceptable.
Redis	Every gateway instance using the same Redis backend	A deployment must enforce one shared request, token, or concurrency quota.

With the memory backend, each gateway instance counts only the traffic it handles. In an evenly distributed deployment with multiple instances, the aggregate traffic admitted during a window can therefore exceed the configured per-process limit. Consistent routing can reduce that variation for one caller or tenant, but Redis is the shared-counter option.

Configure Redis when several gateway instances must enforce one quota:

config.yaml

ratelimit:
  backend: redis
  redis:
    mode: single
    url: redis://127.0.0.1:6379/

The gateway requires the ratelimit.redis block when ratelimit.backend is redis. Startup fails when the selected Redis configuration is missing or Redis cannot be reached.

After a successful startup, a later Redis outage degrades rate limiting to process-local counters. Requests remain protected by per-process limits, but the deployment does not enforce one cluster-wide quota during the outage. Outage-time counts are not copied back to Redis, so counters can remain different until the active windows roll over after recovery.

Redis-backed concurrency slots are released when requests complete. concurrency_ttl_secs reclaims a slot left behind by a crashed instance or interrupted request and defaults to 300 seconds.

Redis Connection Modes

The ratelimit.redis.mode field selects the Redis deployment type and determines which connection fields are required. The preceding example uses single mode with one url.

Use cluster for Redis Cluster seed nodes:

config.yaml

ratelimit:
  backend: redis
  redis:
    mode: cluster
    nodes:
      - redis://10.0.0.1:6379/
      - redis://10.0.0.2:6379/

Use sentinel for a Sentinel-managed master:

config.yaml

ratelimit:
  backend: redis
  redis:
    mode: sentinel
    sentinels:
      - redis://10.0.0.1:26379/
      - redis://10.0.0.2:26379/
    master_name: mymaster

The required fields are:

Mode	Required Connection Fields
single	url
cluster	One or more nodes entries
sentinel	One or more sentinels entries and master_name

For cluster and sentinel modes, set username and password when Redis data nodes require ACL authentication. In sentinel mode, sentinel-node credentials belong in the sentinel URLs, while username, password, and database apply to the discovered Redis master.

Troubleshooting

Start with the rejected request's scope and then identify every limit that can match it.

Symptom	What to Check
A configured limit does not apply.	For AISIX Cloud, confirm that the change has projected to the gateway. For the open-source gateway, confirm that the resources file was reloaded. Then verify that the request uses the expected caller API key and model.
A request is rejected below the expected caller limit.	Check for a model limit and rate-limit policies that match the API key, model, team, member, or provider. Every matching layer must pass.
Aggregate traffic exceeds the configured limit across gateway instances.	Check ratelimit.backend. The memory backend keeps an independent counter in every process; use Redis for one shared quota.
A token-limited request succeeds and the next request is rejected.	This is expected when the successful response consumes the remaining token capacity. Token usage is recorded after the provider reports it.

Next Steps

Configure Rate-Limit Policies when direct caller and model limits are not expressive enough. Continue with Response Caching to reuse eligible Chat Completions responses, or review Metrics to monitor rate-limit rejections.