Thread-per-Core Workers
The AISIX gateway serves traffic from a pool of worker threads. Two startup settings control that pool: proxy.workers sets how many worker threads serve traffic, and proxy.thread_per_core selects how those workers share the proxy listener and the upstream connections. This page explains both settings and their defaults, shows how to verify which mode a running gateway uses, and covers the behaviors that follow from serving with independent workers.
Understand the Serving Modes
In thread-per-core serving, each worker is an independent thread with its own runtime, its own listener on proxy.addr, and its own upstream connection pools. The workers share the proxy address through the SO_REUSEPORT socket option, and the kernel assigns each new client connection to one worker by hashing the connection's addresses and ports. A request is then accepted, processed, and answered on a single thread. A model request that dispatches on one of the worker's own pools also makes its upstream call from that thread, while some upstream calls use pools shared across the process instead, as Account for Per-Worker Connection Pools describes.
With proxy.thread_per_core: false, the gateway serves from one shared runtime instead: a single listener accepts connections, and the worker threads balance tasks among themselves through work stealing. A request can then migrate between threads during processing, for example when an upstream response arrives on a different thread from the one that started the request. Each migration costs a thread wake-up and a context switch. Thread-per-core serving removes those handoffs for requests whose upstream call uses a per-worker pool.
Thread-per-core serving is on by default on Linux and off by default elsewhere, because the connection spreading it relies on is Linux kernel behavior. In the validation benchmarks for this mode, it sustained 54-88% more requests per second than the shared runtime on a 4-worker x86 virtual machine, and 28-42% more on Arm (AWS Graviton) hardware. Those benchmarks drove the gateway from a closed-loop load generator at a fixed number of concurrent connections, from 16 to 768, against a local mock upstream that answered immediately or, at 768 connections, after a 10-millisecond time to first token. The gateway was pinned to 4 cores, plus an 8-core run on Graviton. At 128 or more concurrent connections, the p99 latency roughly halved in both cases; at 16 and 32 connections the p99 improvement was smaller. The difference is gateway throughput capacity; provider latency dominates end-to-end request time in either mode.
Configure the Worker Pool
Both settings live in the proxy block of the startup configuration:
proxy:
addr: "0.0.0.0:3000"
thread_per_core: true
workers: 4
| Field | Default | Description |
|---|---|---|
proxy.thread_per_core | true on Linux, false elsewhere | Serve from independent workers with per-worker listeners and upstream pools. Set false to serve from one shared runtime on any platform. |
proxy.workers | Parallelism available to the process | Number of worker threads that serve traffic, in either mode. Must be at least 1. |
Both settings are read once at process start. Changing either one takes effect at the next gateway restart.
Leave proxy.workers unset in most deployments. The default follows the parallelism actually available to the process, so a container CPU limit, a cgroup quota, or a taskset affinity mask sizes the pool without restating the count in configuration. A gateway limited to 4 vCPUs starts 4 workers even on a 16-core host. Set an explicit count only when the gateway should use fewer threads than it may run on, for example when it shares its CPU allowance with a sidecar.
AISIX rejects proxy.workers: 0 at startup, because zero workers would bind no listener at all. Omit the field to use the default instead.
Both fields follow the standard environment-override form, with double underscores between nested field names:
export AISIX_PROXY__THREAD_PER_CORE=false
export AISIX_PROXY__WORKERS=8
Use these variables when the deployment manages startup settings through the process environment. For the api7/aisix Helm chart, set config.proxy.thread_per_core and config.proxy.workers in the values file instead; see Set Any Other Gateway Configuration. See Environment Variables for the override mechanism.
Verify the Active Mode
The serving mode is visible without any dedicated endpoint.
With log_level: "info", a gateway in thread-per-core mode logs one aisix listening (http, thread-per-core) line per worker at startup, each carrying its worker index, or (https, thread-per-core) when the proxy listener terminates TLS. The shared runtime logs its proxy listener as aisix listening (http), or aisix listening (https) with TLS, without the thread-per-core marker. The admin and metrics listeners, when enabled, log that unmarked form in both modes. The default warn level writes none of these lines.
On a host or debug image with ps and pgrep, list the process's threads. If those tools are unavailable in your gateway image, use the startup log lines above. PID 1 in the published container is the tini wrapper rather than the gateway process:
ps -T -p "$(pgrep -x aisix)"
In thread-per-core mode, the worker threads are named tpc-0 through tpc-<N-1>, one per worker:
PID SPID TTY TIME CMD
23110 23110 ? 00:00:00 aisix
23110 23111 ? 00:00:00 tokio-rt-worker
23110 23112 ? 00:00:00 tokio-rt-worker
23110 23115 ? 00:00:41 tpc-0
23110 23116 ? 00:00:40 tpc-1
23110 23117 ? 00:00:41 tpc-2
23110 23118 ? 00:00:39 tpc-3
In this mode, a small control runtime carries the metrics listener, signal handling, and background export work. Its tokio-rt-worker threads appear alongside the workers and do not count toward proxy.workers. With thread_per_core: false, no tpc- threads exist, and all serving threads carry the default runtime thread name tokio-rt-worker.
Account for Per-Worker Connection Pools
In thread-per-core serving, each worker keeps its own upstream connection pools for model requests that dispatch on the gateway's default HTTP client and whose provider key sets no connection overrides: one pool for regular model requests and a separate one for passthrough requests. Connections in these per-worker pools are never handed between workers: a worker that holds an idle connection to a provider reuses it, and a worker that does not opens its own.
Other outbound connections use pools shared by the whole process, and a connection in one of them can be reused by any worker. These include the pools for provider keys that set their own tls or resolve_addresses, the shared pool a worker falls back to if its own pool cannot be built, and the clients for guardrails and MCP servers. Amazon Bedrock calls made through the AWS SDK also use one process-wide client, which upstream.pool_max_idle_per_host does not limit.
Two sizing consequences follow:
- For the per-worker pools,
upstream.pool_max_idle_per_hostapplies per worker and per pool. A gateway with 8 workers andpool_max_idle_per_host: 32can hold up to 256 idle connections to one provider host in its regular per-worker pools, and as many again in its passthrough pools. To budget idle connections for one kind of per-worker pool, divide the intended total by the worker count; the process-wide pools come on top, so this is not a limit on every connection the process holds. - The idle connections the per-worker pools hold scale with the worker count. When a provider, a NAT gateway, or a corporate egress proxy limits connections per client, count both the per-worker and the process-wide pools.
The pool timeouts keep their meaning unchanged; see Tune the Upstream Connection Layer for the pool settings themselves.
Plan for Low-Concurrency Traffic
The kernel spreads client connections across workers, not individual requests. That spreading is even across many connections and uneven across few. Below about four client connections per worker, some workers can sit idle while their siblings carry several connections each. In that range, throughput can fall below the shared runtime, which balances individual requests instead of connections.
What counts is the number of connections the gateway itself accepts, not the number of end clients. Direct traffic from many clients sits far above the threshold. However, an L7 load balancer or ingress in front of the gateway, as Network and Security recommends for the caller-facing port, can pool requests from many clients onto a small number of gateway-side connections. Benchmarks that drive the gateway from a handful of persistent connections sit in the same range: a load test with 8 connections against 8 workers measures connection placement, not gateway capacity.
If the gateway receives its traffic over only a few long-lived connections, raise the number of connections the tier in front of it keeps toward the gateway, lower proxy.workers so that each worker still receives several connections, or set proxy.thread_per_core: false.
Understand Listener Sharing
Thread-per-core workers share one proxy address through the SO_REUSEPORT socket option. Two properties of that option are worth knowing when operating or auditing a gateway host.
Port conflicts still fail loudly at startup. Before binding its workers' listeners, the gateway probes the address with a regular bind. Starting a gateway while another process holds the address, including a second gateway, therefore fails at startup, exactly as a single-listener process does. The probe leaves one theoretical gap: two gateways started in the same instant can both pass their probe and bind the same address. Sequential restarts, as orchestrators perform them, never hit this window; avoid deliberately starting two gateways on one address at the same moment.
A same-user process can join the listener while the gateway runs. While the gateway is serving, any process running under the same effective user ID that itself sets SO_REUSEPORT can bind the proxy address and receive a share of new connections. The kernel restricts joining to the same effective user ID, so this is not a cross-user risk. It is a property of the mode worth remembering when auditing what runs on a gateway host: if some connections appear to bypass the gateway, check for another process bound to the proxy port, for example with ss -tlpn.
Next Steps
Continue with Network and Security to protect the proxy and metrics listeners, or return to Performance and Sizing to size CPU for your request volume.