# API Key and Model Rate Limits

Rate limits protect provider capacity and prevent one caller or model from consuming a shared quota. Configure a limit on a caller API key when the quota belongs to that caller, or on a model when every caller of that model should share the quota.

These limits belong to the API key or model on which they are configured. [Rate limit policies](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limit-policies.md) are separate rules that can match teams, members, API keys, models, model names, or providers and divide matching traffic into independent quota buckets. A request can match API key limits, model limits, and policies at the same time. Every matching limit applies, and the gateway returns `429` before calling the provider if any bucket is exhausted.

This guide shows how to configure limits on API keys and models, verify enforcement, and choose shared counter storage. Use the control-plane path for AISIX Enterprise or AISIX Cloud, or the resources-file path for AISIX OSS. Both paths configure the same gateway capabilities.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

Before starting, prepare the following:

* A working model alias and caller API key that can send proxy requests. If you have not configured them yet, see [Provider Keys](https://docs.api7.ai/ai-gateway/models/provider-keys.md), [Model Aliases](https://docs.api7.ai/ai-gateway/models/model-aliases.md), and [Caller API Keys](https://docs.api7.ai/ai-gateway/traffic-controls/caller-api-keys.md).
* For AISIX Enterprise or AISIX Cloud, an environment with an attached gateway and a write-scoped admin token. Follow the [AISIX Enterprise getting-started guide](https://docs.api7.ai/ai-gateway/getting-started/aisix-cloud-quickstart.md) for a local evaluation, or [contact API7](https://api7.ai/contact) to request AISIX Cloud access.
* For the open-source AISIX gateway, a declarative resources file that you can validate and reload. See [Reload a Resources File](https://docs.api7.ai/ai-gateway/deployment/configuration-propagation.md#reload-a-resources-file).

Export the gateway request values used to verify any example on this page:

```
# AISIX_PROXY has no trailing slash or endpoint path.

# The local quickstarts use http://127.0.0.1:3000.

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"

export AISIX_MODEL="gpt-4o-prod"

export AISIX_API_KEY="YOUR_CALLER_API_KEY"
```

For the control-plane examples, also export the control-plane connection values and the resource IDs you want to limit:

```
# AISIX_CP includes /api and has no trailing slash.

# The local AISIX Enterprise quickstart uses http://localhost:8080/api.

export AISIX_CP="YOUR_AISIX_CONTROL_PLANE_API_URL"

export AISIX_TOKEN="YOUR_ADMIN_TOKEN"

export ENV_ID="YOUR_ENVIRONMENT_ID"

# API_KEY_ID identifies the caller key whose plaintext is AISIX_API_KEY.

export API_KEY_ID="YOUR_CALLER_API_KEY_ID"

# MODEL_ID identifies the model whose alias is AISIX_MODEL.

export MODEL_ID="YOUR_MODEL_ID"
```

## Choose a Rate-Limit Configuration[​](#choose-a-rate-limit-configuration "Direct link to Choose a Rate-Limit Configuration")

You can configure a rate limit in two ways:

* Attach a `rate_limit` block directly to a caller API key or model. Use a caller limit when one application or tenant needs its own quota, or a model limit when every caller of an alias should share the quota.
* Create a [rate-limit policy](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limit-policies.md) when you need a separately managed quota, conditional matching, or independent buckets for teams, members, API keys, models, or providers.

A caller API key can also carry a separate request and concurrency limit for each MCP server it calls. See [MCP Rate Limits and Budgets](https://docs.api7.ai/ai-gateway/mcp-gateway/traffic-controls.md#limit-a-caller-per-mcp-server).

Every matching limit applies. For example, a request can consume capacity from its caller API key, resolved model, team, and member policies at the same time. The request continues only when every matching bucket has capacity.

## Configure a Caller API Key Limit[​](#configure-a-caller-api-key-limit "Direct link to Configure a Caller API Key Limit")

The following examples limit one caller API key to one request per minute.

### AISIX Enterprise or AISIX Cloud[​](#aisix-enterprise-or-aisix-cloud "Direct link to AISIX Enterprise or AISIX Cloud")

In the dashboard, open **API keys**, expand the caller API key, and use its **Rate limits** controls. The Admin API provides the same configuration for automation.

Update the caller API key with a one-request-per-minute limit:

```
curl -sS -X PATCH "${AISIX_CP}/environments/${ENV_ID}/api_keys/${API_KEY_ID}" \

  -H "Authorization: Bearer ${AISIX_TOKEN}" \

  -H "Content-Type: application/json" \

  -d '{

    "rate_limit": {

      "rpm": 1

    }

  }'
```

The change projects to attached gateways automatically.

### AISIX OSS[​](#aisix-oss "Direct link to AISIX OSS")

Replace the existing caller API key entry to add the same limit. Keep its current `display_name`, `key_env`, and `allowed_models` values:

resources.yaml (caller rate limit)

```
api_keys:

  - display_name: chat-app

    key_env: CALLER_API_KEY

    allowed_models:

      - gpt-4o-prod

    rate_limit:

      rpm: 1
```

Make the caller credential available to the validation command:

```
# Use the same plaintext value as the verification requests.

export CALLER_API_KEY="${AISIX_API_KEY}"
```

If AISIX is installed on the host, validate the complete resources file before reloading the gateway:

```
aisix validate --resources resources.yaml
```

For the Docker setup in the open-source quickstart, use the [container-based validation and reload commands](https://docs.api7.ai/ai-gateway/deployment/configuration-propagation.md#reload-a-resources-file). The quickstart already makes `CALLER_API_KEY` available to the container.

## Configure a Model Limit[​](#configure-a-model-limit "Direct link to Configure a Model Limit")

Use a model limit when every caller of one alias should share the same quota. The following examples allow one request per minute across all caller API keys that use `gpt-4o-prod`.

### AISIX Enterprise or AISIX Cloud[​](#aisix-enterprise-or-aisix-cloud-1 "Direct link to AISIX Enterprise or AISIX Cloud")

In the dashboard, open **Models**, edit the model, and use the **Rate limits** fields. The Admin API provides the same configuration for automation.

Update the model:

```
curl -sS -X PATCH "${AISIX_CP}/environments/${ENV_ID}/models/${MODEL_ID}" \

  -H "Authorization: Bearer ${AISIX_TOKEN}" \

  -H "Content-Type: application/json" \

  -d '{

    "rate_limit": {

      "rpm": 1

    }

  }'
```

### AISIX OSS[​](#aisix-oss-1 "Direct link to AISIX OSS")

Add the limit to the direct model entry and retain its existing provider and provider-key settings:

resources.yaml (model rate limit)

```
models:

  - display_name: gpt-4o-prod

    provider: openai

    model_name: gpt-4o

    provider_key: openai-prod

    rate_limit:

      rpm: 1
```

Validate the complete resources file and reload the gateway. Requests through different caller API keys now draw from the same model bucket.

## Verify Rate Limiting[​](#verify-rate-limiting "Direct link to Verify Rate Limiting")

The preceding caller and model examples use the same one-request-per-minute limit. Configure and test one direct limit at a time so the `429` response identifies the limit under test. Use the same request sequence to [verify a rate-limit policy](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limit-policies.md#verify-the-policy).

After the control-plane change reaches the gateway, or after you reload the open-source resources file, send three requests:

```
for i in 1 2 3; do

  printf "request %s: " "${i}"

  curl -sS -o /dev/null -w "%{http_code}\n" -X POST "${AISIX_PROXY}/v1/chat/completions" \

    -H "Authorization: Bearer ${AISIX_API_KEY}" \

    -H "Content-Type: application/json" \

    --data-binary @- <<EOF

    {

      "model": "${AISIX_MODEL}",

      "messages": [

        {

          "role": "user",

          "content": "Hello from AISIX."

        }

      ]

    }

EOF

done
```

The first request should reach the upstream model. The remaining requests should exceed the limit, provided all three fall within the same clock minute:

```
request 1: 200

request 2: 429

request 3: 429
```

A minute window resets at the top of each UTC minute, so a sequence that straddles that boundary can see a later request succeed.

An OpenAI-compatible rate-limit rejection returns `429 Too Many Requests` with the proxy error format:

```
{

  "error": {

    "message": "request limit exceeded (requests)",

    "type": "rate_limit_exceeded"

  }

}
```

A rejection from a caller API key limit, a model limit, or a rate-limit policy carries `Retry-After` plus the `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, and `X-RateLimit-Scope` headers. They describe the one limit that refused the request:

```
retry-after: 43

x-ratelimit-limit: 1

x-ratelimit-remaining: 0

x-ratelimit-reset: 43

x-ratelimit-scope: rpm
```

Request-count and token-limit rejections count down to the next window boundary, such as the top of the minute, or 00:00 UTC for a daily limit. A concurrency rejection has no window — a slot frees when some in-flight request finishes — so it reports a fixed 60-second hint. Successful Chat Completions responses carry a different, per-dimension header family instead. See [Rate-Limit Rejection Headers](https://docs.api7.ai/ai-gateway/reference/headers-and-error-codes.md#rate-limit-rejection-headers).

## Rate Limit Fields and Counter Behavior[​](#rate-limit-fields-and-counter-behavior "Direct link to Rate Limit Fields and Counter Behavior")

Caller API keys and model aliases use the same `rate_limit` fields. Each field is optional. When a field is omitted, AISIX does not enforce that dimension.

| Field         | Limit               | Window         |
| ------------- | ------------------- | -------------- |
| `rps`         | Requests per second | 1 second       |
| `rpm`         | Requests per minute | 60 seconds     |
| `rph`         | Requests per hour   | 3,600 seconds  |
| `rpd`         | Requests per day    | 86,400 seconds |
| `tpm`         | Tokens per minute   | 60 seconds     |
| `tpd`         | Tokens per day      | 86,400 seconds |
| `concurrency` | In-flight requests  | Not windowed   |

Each windowed field counts in a fixed window aligned to the UTC clock, not to a caller's first request. The count resets as a whole at each boundary: every second for `rps`, the top of every minute for `rpm` and `tpm`, the top of every hour for `rph`, and 00:00 UTC for `rpd` and `tpd`, which is 08:00 in UTC+8. Because the whole count resets at once, a burst at the end of one window and another at the start of the next can together admit up to twice the limit within one window's length. For example, with `rpm: 100`, 100 requests at 10:00:50 and another 100 at 10:01:05 are all admitted.

AISIX checks and records the three limit dimensions at different points while processing a request:

| Dimension     | When AISIX Checks It                                                                                                    | When AISIX Records Usage                                                                                         |
| ------------- | ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Request count | Before the provider call                                                                                                | During the pre-provider checks; an earlier matching layer can increment before a later layer rejects the request |
| Tokens        | Before the provider call, using only tokens already recorded in the current window. Nothing is reserved for the request | After the response completes, using the usage the upstream reports                                               |
| Concurrency   | Before the provider call                                                                                                | Held until the response or stream completes, then released                                                       |

Token limits include prompt and completion tokens. For Anthropic traffic, they also include prompt cache creation and cache read tokens, which Anthropic reports separately from input tokens. OpenAI cached tokens are already included in prompt tokens, so AISIX does not count them twice.

Because the provider reports usage after generating a response, AISIX records a request's tokens only once it completes. Admission checks the tokens recorded so far, so the requests in flight are not counted against the limit. Usage can therefore exceed the limit, as described in [Token Limit Overruns](#token-limit-overruns).

Only token usage that the upstream response reports advances the token counters. Some media calls report none, so they add nothing to `tpm` or `tpd`: `whisper-1` transcriptions and translations, every text-to-speech request, `dall-e-2` and `dall-e-3` image generations and edits, and video requests. Models that do report token usage, such as `gpt-4o-transcribe` and `gpt-image-1`, are counted normally. Audio duration is recorded for cost calculation only and does not count toward token limits. A call that adds no tokens is still rejected when an applicable `tpm` or `tpd` window is already exhausted, whatever its endpoint.

Token limits attached directly to caller API keys or models support minute and day windows only. There is no per-second or per-hour token field for these resources. [Classic rate-limit policies](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limit-policies.md#manage-classic-single-scope-policies) also enforce token caps with minute and day windows. The AISIX Cloud Admin API rejects token caps with second or hour windows. For the open-source AISIX gateway, the resources schema accepts those combinations, but the runtime does not enforce the token cap.

### Token Limit Overruns[​](#token-limit-overruns "Direct link to Token Limit Overruns")

When recorded usage reaches a token limit, AISIX rejects new requests, but every request it has already admitted still completes and is counted. Under concurrency, the overrun is therefore not limited to the one request that crossed the limit. It is roughly the number of requests in flight when the limit is reached, multiplied by the largest token usage of one request.

For example, with 10 requests in flight that each use at most 40 tokens, usage can exceed a `tpm` limit by about 400 tokens.

The number of requests in flight depends on the counter backend:

* With Redis, count the in-flight requests across all gateway instances that share the backend.
* With the memory backend, each instance counts only its own traffic, so multiply the overrun by the number of instances. The limit itself also applies per instance, as described in [Choose Rate-Limit Counter Storage](#choose-rate-limit-counter-storage).

A `concurrency` limit caps the requests in flight, which reduces the overrun. It rejects the excess requests with `429` rather than queueing them.

AISIX does not enforce a strict token cap. Use this figure to size a limit, not as a guarantee.

## How Model Limits Apply[​](#how-model-limits-apply "Direct link to How Model Limits Apply")

If a request matches both caller and model limits, AISIX counts it against the request-count and concurrency limits of both buckets. Exhausting either bucket rejects the request. Tokens are not reserved in either bucket; see [Token Limit Overruns](#token-limit-overruns).

Model limits also apply when a direct model serves as the target of a [routing model](https://docs.api7.ai/ai-gateway/routing/routing-and-failover.md) or [semantic router](https://docs.api7.ai/ai-gateway/routing/semantic-routing.md). Routing dispatch skips a target that is over its own limit and continues with the remaining targets; when every target is over its limit, the request returns `429`. Semantic dispatch enforces the selected target's limit and returns `429` when that target has no capacity. Direct and routed requests to the same model share its limit bucket.

Inject-mode [passthrough routes](https://docs.api7.ai/ai-gateway/endpoints/provider-passthrough.md#rate-limits) can also reserve model limits. AISIX does this when the request body names a model of the route's provider key's provider, by its alias or provider-native name. The matched model shares the same request-count and concurrency buckets as modeled routes; `forward_client` routes never consult the model probe. Caller-level request-count and concurrency layers apply to every route.

An applicable `tpm` or `tpd` window that is already exhausted still rejects a passthrough request. Passthrough usage does not advance those token counters, however, even when AISIX detects a recognized envelope and records token usage on the request's usage event.

## Choose Rate-Limit Counter Storage[​](#choose-rate-limit-counter-storage "Direct link to Choose Rate-Limit Counter Storage")

Gateway startup configuration determines whether rate-limit counters are local to one process or shared across gateway instances. The setting applies to limits attached directly to resources and to reusable policies enforced by that gateway.

| Backend             | Counter Scope                                       | Use When                                                                     |
| ------------------- | --------------------------------------------------- | ---------------------------------------------------------------------------- |
| Memory, the default | One gateway process                                 | A single instance enforces the limit, or per-instance quotas are acceptable. |
| Redis               | Every gateway instance using the same Redis backend | A deployment must enforce one shared request, token, or concurrency quota.   |

With the memory backend, each gateway instance counts only the traffic it handles. In an evenly distributed deployment with multiple instances, the aggregate traffic admitted during a window can therefore exceed the configured per-process limit. Consistent routing can reduce that variation for one caller or tenant, but Redis is the shared-counter option.

Configure Redis when several gateway instances must enforce one quota:

config.yaml

```
ratelimit:

  backend: redis

  redis:

    mode: single

    url: redis://127.0.0.1:6379/
```

The gateway requires the `ratelimit.redis` block when `ratelimit.backend` is `redis`, and startup fails when that block is missing. A Redis the gateway cannot reach at startup no longer fails startup, but the bounded startup connection attempts can delay all listeners. After those attempts finish, Redis no longer blocks listener startup, and the gateway logs one warning with the backend, configured endpoint values, reason, and connection error. When the proxy listener becomes available, the gateway counts per replica. The warning is restated every five minutes, and a background task attaches the shared backend as soon as Redis answers. The gateway never switches permanently to the `memory` backend, and cluster-wide limits are not enforced while it is degraded.

Redis endpoint values can expose credentials

The warning includes the configured endpoint values verbatim: `url` in single mode, `nodes` in cluster mode, and `sentinels` in sentinel mode. Any credentials embedded in those values are logged too. Use the separate `username` and `password` fields for data-node credentials. Sentinel-node credentials remain part of the sentinel URLs, so restrict access to gateway logs.

Startup fails for local configuration and resolution errors that cannot recover in the background. That includes an unparsable `url`; TLS material AISIX cannot read or parse; a client certificate or key configured without its matching key or certificate; `ratelimit.concurrency_ttl_secs: 0`; a missing `url`, `nodes`, `sentinels`, or `master_name` for the selected mode; `timeout_secs: 0`; or a hostname that resolves to no address. A Redis that answers and refuses the connection settings takes the degraded path instead. That covers a rejected credential, a `NOAUTH` or `DENIED` answer, and a `database` the server does not have. The warning carries `reason=refused` rather than `reason=unreachable`, and the five-minute restatement carries the same word. The gateway keeps re-attaching, so a credential corrected on the Redis side is adopted without a restart.

A Redis outage degrades rate limiting to process-local counters, whether the outage is already under way when the gateway starts or begins later. Requests remain protected by per-process limits, but the deployment does not enforce one cluster-wide quota during the outage. Outage-time counts are not copied back to Redis, so counters can remain different until the active windows roll over after recovery.

`ratelimit.redis.timeout_secs` sets how long the gateway waits before that fail-open path takes over. It defaults to `5` seconds, with a minimum of `1`, and `0` is rejected at startup naming the field. Shared counting resumes on its own once Redis answers again.

`timeout_secs` bounds a single Redis round trip or connection attempt, including each connection attempt the gateway makes at startup. A timeout or connectivity failure makes everything that subsystem does next short-circuit for 30 seconds instead of waiting again. A server-reported Redis error fails the current operation but does not open the short circuit. The cache's exact and vector connections count as one subsystem, while the rate limiter is another. The 30-second cool-off is sized to outlast an upstream call, because a request's cache read and its cache write straddle that call. A request therefore pays the budget once for the cache and once for the rate limiter, rather than once per Redis operation. A request whose upstream leg runs longer than 30 seconds can still pay a second budget on the write. In `cluster` and `sentinel` modes, the gateway wraps the whole startup topology walk in an aggregate budget of `timeout_secs` multiplied by the number of configured endpoints plus one. That product bounds the whole startup connect, not each attempt within it; each attempt keeps the single `timeout_secs` bound. In `single` mode, the whole startup connect has one `timeout_secs` budget.

A background task, not a business request, decides when the cool-off ends: when the 30 seconds expire it sends a Redis `PING`, closes the short circuit on success, and re-arms another 30 seconds on failure. Requests keep short-circuiting until it closes, so no request pays the timeout to discover that Redis is still down. Commands resume only after a successful background probe. Recovery can take more than one cool-off window after Redis becomes reachable again.

Local concurrency slots are released when requests complete. An SSE relay retains every matching slot until the response body closes or is cancelled. A Redis-backed relay renews each live lease while Redis is reachable. Stale slots are recovered after `concurrency_ttl_secs`, which defaults to `300` seconds. If Redis reports that a live lease no longer exists, AISIX ends a passthrough SSE relay.

Mixed-version rollout

During a mixed-version rollout, treat earlier replicas according to how they hold each stream reservation. Where an earlier replica retains a reservation for the stream lifetime but cannot renew it, set `ratelimit.concurrency_ttl_secs` longer than the longest possible reservation, including the upstream-header wait and stream lifetime. A longer TTL also delays recovery of a slot left by a replica that crashes.

Passthrough SSE requires a different rollout. Replicas that release a passthrough reservation when they return the response head cannot be protected by a longer TTL. Stop sending those replicas new passthrough SSE requests before newer replicas begin enforcing the shared limit. Existing passthrough streams on the earlier replicas remain outside that concurrency count until they finish or are terminated.

### Redis Connection Modes[​](#redis-connection-modes "Direct link to Redis Connection Modes")

The `ratelimit.redis.mode` field selects the Redis deployment type and determines which connection fields are required. The preceding example uses `single` mode with one `url`.

Use `cluster` for Redis Cluster seed nodes:

config.yaml

```
ratelimit:

  backend: redis

  redis:

    mode: cluster

    nodes:

      - redis://10.0.0.1:6379/

      - redis://10.0.0.2:6379/
```

Use `sentinel` for a Sentinel-managed master:

config.yaml

```
ratelimit:

  backend: redis

  redis:

    mode: sentinel

    sentinels:

      - redis://10.0.0.1:26379/

      - redis://10.0.0.2:26379/

    master_name: mymaster
```

The required fields are:

| Mode       | Required Connection Fields                        |
| ---------- | ------------------------------------------------- |
| `single`   | `url`                                             |
| `cluster`  | One or more `nodes` entries                       |
| `sentinel` | One or more `sentinels` entries and `master_name` |

Set `username` and `password` when Redis requires authentication. They apply in every mode, and they override a credential embedded in the `url`. A value supplied through the environment is therefore not outranked by a stale one left in the URL. Set them as a pair: providing either half replaces both, because composing a login from two sources produces one that exists in neither. `database` applies in `single` and `sentinel` modes; Redis Cluster has only DB 0. In sentinel mode, sentinel-node credentials belong in the sentinel URLs, while `username`, `password`, and `database` apply to the discovered Redis master.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

Start with the rejected request's scope and then identify every limit that can match it.

| Symptom                                                                                        | What to Check                                                                                                                                                                                                                                                                   |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A configured limit does not apply.                                                             | For AISIX Enterprise or AISIX Cloud, confirm that the change has projected to the gateway. For the open-source AISIX gateway, confirm that the resources file was reloaded. Then verify that the request uses the expected caller API key and model.                            |
| A request is rejected below the expected caller limit.                                         | Check for a model limit and [rate-limit policies](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limit-policies.md) that match the API key, model, team, member, or provider. Every matching layer must pass.                                                            |
| Aggregate traffic exceeds the configured limit across gateway instances.                       | Check `ratelimit.backend`. The memory backend keeps an independent counter in every process; use Redis for one shared quota.                                                                                                                                                    |
| A token-limited request succeeds and the next request is rejected, or usage exceeds the limit. | This is expected when the successful responses consume the remaining token capacity. Token usage is recorded after the provider reports it, so every request in flight when the limit is reached still completes and counts. See [Token Limit Overruns](#token-limit-overruns). |

## Next Steps[​](#next-steps "Direct link to Next Steps")

Configure [Rate-Limit Policies](https://docs.api7.ai/ai-gateway/traffic-controls/rate-limit-policies.md) when direct caller and model limits are not expressive enough. Continue with [Response Caching](https://docs.api7.ai/ai-gateway/traffic-controls/caching.md) to reuse eligible Chat Completions responses, or review [Metrics](https://docs.api7.ai/ai-gateway/reference/metrics.md) to monitor rate-limit rejections.