Rate Limits

Rate limits protect provider capacity and prevent one caller, model, team, or member from consuming a shared quota. AISIX checks every limit that matches a request. If any matching limit is exhausted, the gateway returns 429 before calling the provider.

Self-hosted gateways attach limits directly to caller API keys or model aliases. Managed deployments support those limits and add policies for API keys, models, teams, members, and each member inside a team.

Choose a Rate-Limit Scope

In self-hosted gateways, attach inline rate limits to caller API keys or models. Choose the narrowest scope that represents the traffic you want to control:

• Use a caller API key when one application or tenant should have its own quota.
• Use a model when every caller of a model alias should share one quota.

Managed deployments support these inline limits and standalone policies with additional scopes. See Managed Rate-Limit Policies for those options.

The self-hosted procedure below protects one application by attaching a limit to its caller API key.

Configure a Caller Limit

This example creates a self-hosted caller API key that can send one request per minute to one model alias.

Prerequisites

Prepare the following resources and access:

• A self-hosted AISIX gateway with the admin and proxy listeners available.
• The admin key from the gateway config.yaml.
• A working model alias that can serve proxy requests. If you have not created one, configure Provider Credentials and Model Aliases first.

Set the Example Values

Set the admin key, plaintext caller API key, and model alias. Hash the caller API key before storing it in the gateway:

export AISIX_ADMIN_KEY="YOUR_ADMIN_KEY"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="gpt-4o-prod"

AISIX_API_KEY_HASH=$(printf '%s' "${AISIX_API_KEY}" | shasum -a 256 | awk '{print $1}')

Create the Caller API Key

Create the caller API key resource through the AISIX Admin API. The rate_limit object applies a one-request-per-minute limit:

curl -sS -X POST "http://127.0.0.1:3001/admin/v1/api_keys" \
  -H "Authorization: Bearer ${AISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "key_hash": "'"${AISIX_API_KEY_HASH}"'",
    "allowed_models": ["'"${AISIX_MODEL}"'"],
    "rate_limit": {
      "rpm": 1
    }
  }'

The response contains the stored resource ID, key hash, configured limit, and resource revision:

{
  "id": "4ae2b1b8-5e2c-4f44-8d8a-2f6a6f5ef7f8",
  "value": {
    "key_hash": "4b4f91305bd7f14a04ef6c850b3f4d0a8ce9ac67bc63f8b342ccdfd0d2f5b8f8",
    "allowed_models": [
      "gpt-4o-prod"
    ],
    "rate_limit": {
      "rpm": 1
    }
  },
  "revision": 1
}

The example shows only the fields needed for this task. See the AISIX Admin API for the complete caller API key schema.

Verify Rate Limiting

Send three requests with the rate-limited caller API key:

for i in 1 2 3; do
  printf "request %s: " "${i}"
  curl -sS -o /dev/null -w "%{http_code}\n" -X POST "http://127.0.0.1:3000/v1/chat/completions" \
    -H "Authorization: Bearer ${AISIX_API_KEY}" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "'"${AISIX_MODEL}"'",
      "messages": [
        {
          "role": "user",
          "content": "Hello from AISIX."
        }
      ]
    }'
done

The first request should reach the upstream model. The remaining requests should exceed the limit during the same fixed window:

request 1: 200
request 2: 429
request 3: 429

An OpenAI-compatible rate-limit rejection returns 429 Too Many Requests with the proxy error format:

{
  "error": {
    "message": "request limit exceeded (requests)",
    "type": "rate_limit_exceeded"
  }
}

Request-count and token-limit rejections include Retry-After because their fixed windows provide a reset time. A concurrency rejection has no window-based retry hint. Successful Chat Completions responses can also include caller-key rate-limit state; see Headers and Error Codes.

Rate Limit Fields

Caller API keys and model aliases use the same rate_limit fields. Each field is optional. When a field is omitted, AISIX does not enforce that dimension.

Field	Limit	Fixed Window
rps	Requests per second	1 second
rpm	Requests per minute	60 seconds
rph	Requests per hour	3,600 seconds
rpd	Requests per day	86,400 seconds
tpm	Tokens per minute	60 seconds
tpd	Tokens per day	86,400 seconds
concurrency	In-flight requests	Not windowed

The three limit dimensions are accounted at different points in the request path:

Dimension	When AISIX Checks It	When AISIX Records Usage
Request count	Before the provider call	During the pre-provider checks; an earlier matching layer can increment before a later layer rejects the request
Tokens	Before the provider call, using tokens already recorded in the current window	After the upstream response reports usage
Concurrency	Before the provider call	Held until the response or stream completes, then released

Token limits include prompt and completion tokens. For Anthropic traffic, they also include prompt cache creation and cache read tokens, which Anthropic reports separately from input tokens. OpenAI cached tokens are already included in prompt tokens, so AISIX does not count them twice.

Because the provider reports usage after generating a response, the request that crosses a token limit can finish successfully. Its token usage can exhaust the current window and cause a later request to be rejected.

Token limits support minute and day windows only. There is no per-second or per-hour token field for limits attached directly to caller API keys or models.

Configure a Model Limit

Attach the same rate_limit object to a model when all callers of that alias should share a limit. Add it when you create or update the model through /admin/v1/models.

The following model allows up to 300 requests per minute and 20 concurrent requests:

{
  "display_name": "gpt-4o-prod",
  "provider": "openai",
  "model_name": "gpt-4o",
  "provider_key_id": "YOUR_PROVIDER_KEY_ID",
  "rate_limit": {
    "rpm": 300,
    "concurrency": 20
  }
}

If a request matches both caller and model limits, AISIX reserves capacity in both buckets. Exhausting either bucket rejects the request.

Choose Counter Storage for Self-Hosted Gateways

Gateway startup configuration determines whether rate-limit counters are local to one process or shared across gateway instances. The setting applies to inline limits and managed policies enforced by that gateway.

Backend	Counter Scope	Use When
Memory, the default	One gateway process	A single instance enforces the limit, or per-instance quotas are acceptable.
Redis	Every gateway instance using the same Redis backend	A deployment must enforce one shared request, token, or concurrency quota.

With the memory backend, each gateway instance counts only the traffic it handles. In an evenly distributed deployment with multiple instances, the aggregate traffic admitted during a window can therefore exceed the configured per-process limit. Consistent routing can reduce that variation for one caller or tenant, but Redis is the shared-counter option.

Configure Redis when several gateway instances must enforce one quota:

config.yaml

ratelimit:
  backend: redis
  redis:
    mode: single
    url: redis://127.0.0.1:6379/

The gateway requires the ratelimit.redis block when ratelimit.backend is redis. Startup fails when the selected Redis configuration is missing or Redis cannot be reached.

After a successful startup, a later Redis outage degrades rate limiting to process-local counters. Requests remain protected by per-process limits, but the deployment does not enforce one cluster-wide quota during the outage. Outage-time counts are not copied back to Redis, so counters can remain different until the active windows roll over after recovery.

Redis-backed concurrency slots are released when requests complete. concurrency_ttl_secs reclaims a slot left behind by a crashed instance or interrupted request and defaults to 300 seconds.

Redis Connection Modes

The ratelimit.redis.mode field selects the Redis deployment type and determines which connection fields are required. The preceding example uses single mode with one url.

Use cluster for Redis Cluster seed nodes:

config.yaml

ratelimit:
  backend: redis
  redis:
    mode: cluster
    nodes:
      - redis://10.0.0.1:6379/
      - redis://10.0.0.2:6379/

Use sentinel for a Sentinel-managed master:

config.yaml

ratelimit:
  backend: redis
  redis:
    mode: sentinel
    sentinels:
      - redis://10.0.0.1:26379/
      - redis://10.0.0.2:26379/
    master_name: mymaster

The required fields are:

Mode	Required Connection Fields
single	url
cluster	One or more nodes entries
sentinel	One or more sentinels entries and master_name

For cluster and sentinel modes, set username and password when Redis data nodes require ACL authentication. In sentinel mode, sentinel-node credentials belong in the sentinel URLs, while username, password, and database apply to the discovered Redis master.

Managed Rate-Limit Policies

The AISIX managed control plane supports the caller API key and model limits described above. It also projects shared rate-limit policies for API keys, models, teams, members, and per-member limits inside a team.

A managed request can match several limits at the same time. These can include inline caller API key and model limits, plus policies for its API key, model, team, or member. Every matching limit must have capacity for the request to continue.

These policy scopes differ in what they count:

Policy Scope	Matching Traffic	Bucket Behavior
API key	Requests authenticated with the selected key	One bucket for that key
Model	Requests resolved to the selected model	One bucket shared by callers of that model
Team	Requests whose caller API key is bound to the selected team	One shared team bucket
Member	Requests whose caller API key is bound to the selected member	One member bucket across that member's keys
Each member in a team	Requests whose caller API key is bound to both the selected team and a member	One independent bucket for each member of the team

A team policy gives the whole team one shared bucket. A per-member limit inside a team gives every member an independent bucket of the same size, so one member cannot consume another member's quota. Team membership by itself does not select either policy; matching depends on the team and member bindings carried by the caller API key.

Use a shared policy when you need a second or hour request-count window, or when the quota should apply to a team or member scope. Shared policies use window with max_requests, max_tokens, or both. Request-count limits can use second, minute, or hour windows. Token caps require a minute window.

Configure a Policy in the Dashboard

Create a policy in the environment whose gateway traffic it should control:

1. Open the environment and select Rate limits.
2. Select New policy.
3. Enter a Name, choose the Scope, and select the target resource.
4. Choose the Window and enter Max requests, Max tokens, or both.
5. Select Create policy.

At least one maximum is required. An environment can contain only one policy for the same scope and target. The scope and target cannot be changed after creation; delete and recreate the policy to target a different resource.

Caller API key and model pages also expose inline limit editors for the seven rate_limit fields described earlier. Those limits belong to the resource itself. The Rate limits page manages the standalone policies described in this section.

Use the Cloud Admin API

Use an organization admin token with write scope to create policies from automation. Set the control-plane base URL, environment ID, and target resource ID:

export AISIX_CP_API="https://<your-cp-api-host>/api"
export AISIX_ADMIN_TOKEN="YOUR_ADMIN_TOKEN"
export ENV_ID="11111111-1111-4111-8111-111111111111"
export TEAM_ID="22222222-2222-4222-8222-222222222222"

Create a policy that limits the selected team to exactly 1,000,000 tokens per minute:

curl -sS -X POST "${AISIX_CP_API}/environments/${ENV_ID}/rate_limits" \
  -H "Authorization: Bearer ${AISIX_ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "team-acme-tpm",
    "scope": "team",
    "scope_ref": "'"${TEAM_ID}"'",
    "window": "minute",
    "max_tokens": 1000000
  }'

A successful request returns 201 Created with the new policy:

{
  "rate_limit_policy": {
    "id": "33333333-3333-4333-8333-333333333333",
    "env_id": "11111111-1111-4111-8111-111111111111",
    "name": "team-acme-tpm",
    "scope": "team",
    "scope_ref": "22222222-2222-4222-8222-222222222222",
    "window": "minute",
    "max_requests": null,
    "max_tokens": 1000000,
    "created_at": "2026-07-16T08:00:00Z",
    "updated_at": "2026-07-16T08:00:00Z"
  }
}

The API returns 409 Conflict if the environment already has a policy for the same scope and target. See the Cloud Admin API Reference for list, get, update, and delete operations.

When a shared policy includes both request and token caps, select a minute window so AISIX can enforce both values. Second and hour policies accept max_requests only.

Shared policies are managed-control-plane resources. The self-hosted AISIX Admin API does not expose CRUD routes for them.

Troubleshoot Rate Limits

Start with the rejected request's scope and then identify every limit that can match it.

Symptom	What to Check
A request is rejected below the expected caller limit.	Check for a model limit and managed API key, model, team, or member policies. Every matching layer must pass.
Aggregate traffic exceeds the configured limit across gateway instances.	Check ratelimit.backend. The memory backend keeps an independent counter in every process; use Redis for one shared quota.
A token-limited request succeeds and the next request is rejected.	This is expected when the successful response consumes the remaining token capacity. Token usage is recorded after the provider reports it.
A per-member limit inside a team does not apply.	Confirm that the caller API key carries both the target team binding and a member binding.
Redis-backed limits do not start.	Check the connection fields required by the selected Redis mode and verify that the gateway can reach the configured endpoints.
Limits become less strict during a Redis outage.	Restore Redis connectivity. Runtime Redis failures fall back to process-local counters, so cluster-wide enforcement is temporarily unavailable.
A concurrency rejection has no Retry-After header.	Concurrency is not a fixed window, so AISIX cannot calculate a reset time. Retry when an in-flight request completes.

Next Steps

Continue with Caching to reuse eligible Chat Completions responses, or review Metrics to monitor rate-limit rejections.