Rate limiting at the proxy

Rate limiting belongs at the proxy because that is the cheapest place in the stack to shed load: the proxy can refuse a request after parsing the header block, before a connection is borrowed from the upstream pool and before a thread or database handle is committed. That holds only if the proxy can tell one client from another, which makes rate limiting entirely downstream of the client IP problem. A limiter keyed on a value the client controls is worse than no limiter; a limiter keyed on a value thousands of unrelated users share is an outage generator. Choose the key first, the algorithm second, the implementation third.

The algorithms, and what each does to a burst#

The fixed window boundary problem is why nobody serious ships it: at 100 per minute, a client sends 100 requests at 12:00:59.9 and 100 more at 12:01:00.0, delivering 200 in 200 milliseconds without ever violating the stated limit. Token bucket and leaky bucket make nearly the same admission decision and differ only in what happens to an excess request. A token bucket rejects it; a leaky bucket queues it and releases it at the drain rate. That distinction is why nginx behaves as it does.

nginx: limit_req is a leaky bucket#

The nginx documentation says so directly. Excess requests are not refused, they are held until the bucket drains enough to admit them, and only when the number held exceeds burst is one terminated with an error. The symptom of an over-limit client is latency, not a wall of 429s.

http {
    limit_req_zone  $binary_remote_addr zone=api:10m   rate=10r/s;
    limit_req_zone  $binary_remote_addr zone=login:5m  rate=30r/m;
    limit_conn_zone $binary_remote_addr zone=conns:10m;

    limit_req_status    429;    # default 503
    limit_conn_status   429;    # default 503
    limit_req_log_level warn;   # default error; delays log one level lower

    server {
        location /api/ {
            limit_req  zone=api burst=20 nodelay;
            limit_conn conns 20;
            proxy_pass http://app;
        }
        location = /login {
            limit_req zone=login burst=3;   # no nodelay: excess is delayed
            proxy_pass http://app;
        }
        location /downloads/ {
            limit_rate_after 2m;            # full speed for the first 2 MB
            limit_rate       512k;          # then 512 kB/s, per request
            proxy_pass http://files;
        }
    }
}

A client firing 25 requests at /api/ gets 21 immediately (allowance plus burst=20, released without delay) and 429 for the rest, each logging one limiting requests line at warn. At /login the excess is held instead, released one every two seconds until four are queued and the fifth is rejected. limit_rate is per request, so two connections get twice the configured bandwidth; it defaults to 0 and accepts variables since nginx 1.17.0.

The rate syntax, and the thing everyone gets wrong#

rate=30r/m does not mean "thirty requests allowed in any given minute". nginx expresses sub-1r/s rates in requests per minute purely as notation: 30r/m is half a request per second, which is one request every two seconds. The bucket drains continuously and never refills in a lump at the top of the minute, so a client that waits 59 seconds then sends 30 requests has 29 delayed or rejected. burst is the only knob that expresses "allow a clump".

delay= is the underrated one: burst=20 delay=5 gives a browser room for a five-way page-load fan-out while still smoothing a script that opens twenty. Size every new limit in dry-run mode from a week of real excess: values before enforcing.

Zone sizing, and the number people copy from the wrong page#

Per the documentation, a limit_req state occupies 64 bytes on 32-bit platforms and 128 bytes on 64-bit, so one megabyte holds about 16 thousand 64-byte states or about 8 thousand 128-byte states. A limit_conn state occupies 32 or 64 bytes on 32-bit and 64 bytes on 64-bit, so one megabyte holds about 32 thousand or about 16 thousand states.

The modules also differ when full, which matters more than the arithmetic. An exhausted limit_req zone evicts the least recently used state and errors only if it still cannot allocate; an exhausted limit_conn zone returns the error to all further requests. Undersizing limit_conn is a total outage; undersizing limit_req is a silent accuracy loss as heavy talkers evict each other.

HAProxy: stick tables#

HAProxy has no rate limit directive. It has stick tables, a general keyed counter store, and rate limiting is one thing you build with them: one table tracks request rate, error rate, connection rate and arbitrary flags, and any of them can drive any action.

backend st_clients
    stick-table type ipv6 size 1m expire 10m \
        store http_req_rate(10s),http_err_rate(10s),conn_rate(10s),gpc0

frontend fe_https
    bind :443 ssl crt /etc/haproxy/site.pem

    # Resolve the real client first, or you will track the CDN.
    acl from_edge src -f /etc/haproxy/trusted-proxies.lst
    http-request set-src hdr_ip(x-forwarded-for,-1) if from_edge

    # Track a /32 for IPv4 and a /64 for IPv6.
    http-request track-sc0 src,ipmask(32,64) table st_clients

    acl abusive  sc_http_req_rate(0) gt 200
    acl erroring sc_http_err_rate(0) gt 50
    acl flagged  sc_get_gpc0(0)      gt 0

    http-request sc-inc-gpc0(0) if erroring
    http-request silent-drop    if flagged
    http-request deny deny_status 429 if abusive
    default_backend be_app

http_req_rate(10s) is a decaying rate over a ten second period, so it reacts in seconds rather than at a window boundary, and gpc0 is a counter independent of those windows, which is how you say "flagged once, stays flagged until the entry expires".

The ordering is load bearing. tcp-request content track-sc0 src runs during content inspection, before any http-request rule and therefore before set-src has rewritten the source from X-Forwarded-For, so behind a CDN it buckets every user of a PoP into one entry. Use it for connection-rate control on the socket peer and http-request track-sc0 for anything keyed on the resolved client, per configuring trusted proxies.

silent-drop is right less often than it looks: it makes the client wait out its own timeout, which for a browser is a hung tab rather than a fast error.

Envoy: local filter versus global service#

The local filter (envoy.filters.http.local_ratelimit) is a token bucket held in the Envoy process:

name: envoy.filters.http.local_ratelimit
typed_config:
  "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
  stat_prefix: http_local_rate_limiter
  token_bucket:
    max_tokens: 1000
    tokens_per_fill: 100
    fill_interval: 1s
  filter_enforced:
    default_value: { numerator: 100, denominator: HUNDRED }

It costs nothing, adds no latency and cannot fail. It is also per instance: with local_rate_limit_per_downstream_connection at its default false the bucket is shared across that process's workers, but nothing crosses processes, so twenty replicas give twenty times the configured limit. Rejected requests get 429 and x-envoy-ratelimited.

The global filter (envoy.filters.http.ratelimit) calls an external gRPC RateLimitService, keyed by a domain and by descriptors built from route configuration. It gives one true limit across every replica and puts a network round trip in front of every matching request. failure_mode_deny defaults to false, so an unreachable limit service means unlimited traffic; setting it true returns 500 for everything instead, converting a limiter outage into a site outage.

The decision rule: local filter for protective limits where "roughly this much per instance" is the real requirement, global service only for contractual quotas that must be exact. Either way that call is a hop with its own timeout and belongs in the timeout budget for the chain.

Traefik and Caddy#

Traefik's rateLimit middleware is a token bucket of average per period (default 1s) with burst (default 1). sourceCriterion selects the key: ipStrategy (with depth, excludedIPs and, usefully, ipv6Subnet), requestHeaderName, or requestHost, defaulting to the remote address. Without Redis the state is in memory and therefore per instance.

Caddy ships no rate limiting in the standard build. The usual option is the third-party caddy-ratelimit module (xcaddy build --with github.com/mholt/caddy-ratelimit). It implements a sliding window, returns 429 with Retry-After, exposes ipv4_prefix and ipv6_prefix, and has a distributed block that exchanges state through Caddy's storage, documented as eventually consistent and approximate.

Choosing the key#

Every key is a guess about identity, and every guess is defeated by something specific.

Two population effects wreck address keys in opposite directions. CGNAT and corporate NAT collapse many users into one key: a carrier shares one public address across a subscriber pool (typically 100.64.0.0/10 internally), and an office gateway does the same, so a limit that is generous for one human throttles an entire building. IPv6 does the reverse: a residential subscriber is routinely delegated a /64, often a /56 or /48, can source from 2^64 addresses inside it, and rotates through them by default for privacy.

The rule: limit on the strongest identity the request carries, and fall back to a prefix, never to an address. Authenticated traffic gets an account key; unauthenticated traffic gets a prefix key sized for a shared NAT, plus a much tighter limit on the endpoints that mint identity (login, signup, password reset, token exchange). Whatever you pick, validate it before you key on it: an unverified header, an unsigned token or X-Forwarded-For on an untrusted path produces a limit that never fires on abusers and fires constantly for real users. Check what your proxy actually derives with the client IP resolver, read the semantics in the X-Forwarded-For header, and work the surrounding hardening from the reverse proxy security checklist.

Distributed limits: divide or share#

Every in-process limiter multiplies by replica count: N instances at limit L give an effective global limit of N x L. Two honest responses exist.

Divide by the replica count. Set each instance to L/N. Free, no latency, cannot fail, and wrong during every deployment: a rolling update that briefly runs 2N pods doubles the effective limit, a scale-down halves it, and uneven load balancing saturates some instances while others idle.

Share a counter. Redis, an Envoy rate limit service, or the periodic state exchange Traefik and caddy-ratelimit implement. One true limit, paid for with a round trip on the request path, a new failure domain, and a fail-open or fail-closed choice that is unpleasant either way. Periodic exchange sits in between and is explicitly approximate: between syncs every instance works from stale counts.

What to return#

Return 429 Too Many Requests (RFC 6585 section 4) with Retry-After (RFC 9110 section 10.2.3), as delta-seconds or an HTTP-date. Jitter it across clients, or every rejected caller returns at the same instant and you have built a synchronised thundering herd.

For quota communication the IETF HTTPAPI working group has draft-ietf-httpapi-ratelimit-headers. As of revision 11 (23 May 2026) this is an active Internet-Draft, not a published RFC, so treat the field names as provisional. It defines two structured fields:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
RateLimit-Policy: "burst";q=100;w=60, "daily";q=1000;w=86400
RateLimit: "burst";r=0;t=30

RateLimit-Policy advertises the policies in force (q quota, w window in seconds, qu quota unit, pk partition key); RateLimit reports remaining quota (r) and seconds until the window resets (t). Earlier revisions used three separate fields, RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset, which is what most deployed servers emit and most client libraries parse, and the appendix notes the widespread non-standard X-RateLimit-* triple. Emit a legacy triple for compatibility today and revisit when the draft becomes an RFC.

Why 503 is wrong and 429 is right#

nginx defaults limit_req_status and limit_conn_status to 503. Change both. A 5xx means the server failed; a rate limit is the server working correctly and telling one client to slow down, which is what a 4xx expresses.

The practical damage is worse than the semantics, because generic retry machinery keys on status class. Envoy's retry_on: 5xx and gateway-error retry a 503; nginx's proxy_next_upstream http_503 sends the request to the next upstream, spreading the load you were containing; CDNs treat origin 5xx as an origin fault and may retry or serve stale. A 503 therefore makes the infrastructure between you and the client generate extra attempts for every request you refused, which is exactly the amplification the limiter exists to prevent, and it charges successful policy decisions against your availability SLO.

Failure modes#

Every user shares one bucket, and the bucket is the CDN. Symptom: limiting requests, excess: lines naming a handful of addresses, users 429ed on their first request of the day. Cause: the key resolved to the edge address, because set_real_ip_from is missing, real_ip_recursive is off with two appending hops, or tracking runs at tcp-request content before set-src. Fix: resolve the client first, track second, and log peer and derived address side by side.

limit_conn zone exhaustion takes the site down. Symptom: sudden blanket rejections for all clients, unrelated to any client's rate, with limiting connections entries. Cause: the zone is full, and unlike limit_req that module errors on all further requests rather than evicting. Fix: size for the real distinct-key count and alert on it, because there is no partial degradation.

Health checks get rate limited. Symptom: backends flap under load and the outage widens as the pool shrinks. Cause: probes arrive frequently from few addresses, so on a busy node the checker is your heaviest single talker. Fix: exempt the checker ranges, for example an nginx geo block yielding an empty key, since requests with an empty key value are not accounted. See health checks and upstream failover.

The limiter makes a retry storm worse. Symptom: an upstream slows slightly, rejections begin, and total volume rises. Cause: clients and intermediaries retry immediately, so each refused request becomes two or three. Fix: return 429 so generic 5xx retry policies do not fire, send a jittered Retry-After, and retry at exactly one layer.

Frequently asked questions#

Why does nginx delay requests instead of returning 429?#

Because limit_req implements a leaky bucket. Excess requests within burst are queued and released at the configured rate rather than refused, and rejection happens only once the number queued exceeds burst. Add nodelay to serve the burst immediately and reject beyond it, or delay=N to release the first N excess immediately and delay the rest.

What does rate=30r/m actually mean in nginx?#

One request every two seconds, not thirty requests at any point within a minute. nginx expresses sub-1r/s rates in requests per minute purely as notation, and the bucket drains continuously rather than refilling in a lump at the start of a minute. To allow a clump of thirty you need rate=30r/m burst=30 nodelay.

Should a rate limit return 429 or 503?#

1. A rate limit is not a server failure, and a 5xx makes retry machinery treat it as one: Envoy's retry_on: 5xx, nginx's proxy_next_upstream http_503 and CDN origin-error handling all generate extra attempts for a request you just refused. nginx defaults both limit_req_status and limit_conn_status to 503, so set them explicitly.

How do I rate limit IPv6 clients?#

Key on a prefix, not an address. A single subscriber typically holds a /64 and can rotate through 2^64 source addresses, so a per-address limit is trivially evaded and fills the state zone with useless entries. Use /64 as the minimum aggregation and /56 against a determined subscriber: HAProxy has src,ipmask(32,64), Traefik has ipStrategy.ipv6Subnet, caddy-ratelimit has ipv6_prefix.

How big should a limit_req zone be?#

Multiply the expected distinct key count by 128 bytes on 64-bit platforms, then add headroom. Per the nginx documentation a limit_req state is 64 bytes on 32-bit and 128 bytes on 64-bit, so one megabyte holds about 16 thousand or about 8 thousand states respectively. The often-quoted "16,000 per megabyte" comes from the limit_conn page, whose states are half the size, so it undersizes a request-rate zone twofold.

How do I limit users behind CGNAT without throttling all of them?#

Stop keying on the address for that traffic. Use the strongest identity the request carries, such as an API key or a verified session, and reserve prefix keys for genuinely unauthenticated endpoints with a limit sized for a shared egress point rather than for one person. Keep a much tighter limit on the endpoints that mint identity, since those are what an attacker needs first.

Does rate limiting work across multiple proxy instances?#

Not by default. Every in-process limiter (nginx limit_req, Envoy's local filter, Traefik without Redis) is per instance, so N replicas give N times the configured limit. Either divide the limit by the replica count, accepting that deployments and scaling events change it, or use a shared counter and accept the latency and the new failure domain. Fail open on that counter for protective limits and fail closed only for contractual ones: Envoy's failure_mode_deny defaults to false, and setting it true turns a limiter outage into a total outage.