Timeout budgets across a proxy chain

In a chain of proxies, the client-facing timeout must be the longest and every hop closer to the work must be shorter. Not the other way round. The layer nearest the actual work is the layer that knows why the request is slow, so it must be the one that gives up first, emit the error, and report it upward. Every outer layer then just relays a real diagnosis instead of manufacturing a generic one.

Get the order backwards and the edge times out while every inner layer is still working. The client sees 504 Gateway Timeout, the application log contains nothing, and the request that "failed" completes successfully seconds later against a socket nobody is reading. That is the most common shape of an unfixable-looking timeout incident.

The ladder rule, stated precisely#

For a chain client -> A -> B -> C -> work:

timeout(client) > timeout(A) > timeout(B) > timeout(C) > expected(work)

with a real margin between each rung, not one second. The margin covers the outer layer's own overhead: connection setup, queueing, and the time to serialise and forward the inner error. A 2s gap per hop is workable; 5s is safer and costs nothing, since these values only matter in the failure case.

Three corollaries people miss:

1. The innermost bound is a work estimate, not a timeout. The database statement_timeout or the internal RPC deadline must sit above the slowest legitimate query plus headroom, otherwise the ladder has no floor.
2. Retries live inside a rung, not between rungs. If layer B retries, B's budget is attempts x per-try timeout, and that total is what must fit under A's timeout.
3. Idle keep-alive timeouts run in the opposite direction, which is a different family of bugs entirely.

Defaults you are inheriting#

HAProxy is the only entry on that list that warns you when the important timeouts are unset:

[WARNING] config : missing timeouts for frontend 'public'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.

Gunicorn's --timeout is a worker liveness check, not a request timeout: the arbiter kills a worker that has not heartbeated. With sync workers that is equivalent to a request timeout; with gevent or eventlet workers it is not, because the worker keeps heartbeating while a request hangs. Deployments that changed worker class and kept the number silently lost their innermost rung. Similarly, uWSGI's harakiri is disabled by default, so a stock uWSGI deployment has no request time limit at all and the ladder has no floor.

nginx has no total request timeout#

proxy_read_timeout is defined as the timeout between two successive read operations, not for the whole response. An upstream that emits one byte every 59 seconds keeps the connection alive indefinitely under the default. There is no proxy_total_timeout or equivalent in the HTTP proxy module.

Two consequences. Slow-drip responses are never caught by nginx, so an absolute cap has to come from a layer that has one (Envoy's max_stream_duration, a load balancer idle timeout the drip interval exceeds, or the application). And proxy_read_timeout should be sized against the longest legitimate gap between bytes, not the longest legitimate request: for an endpoint emitting a token every 300ms, proxy_read_timeout 10s; is generous even if the stream runs for ten minutes. That is why streaming and response buffering have to be reasoned about together, since buffering changes what the gap actually is.

The retry arithmetic#

A timeout without a retry count is not a budget. The worst case is:

effective = attempts x per_try_timeout + (attempts - 1) x backoff

nginx's defaults are the dangerous ones. With a three-server upstream group and the default proxy_next_upstream_tries 0 and proxy_next_upstream_timeout 0, a request against three dead-slow backends consumes 3 x 60s = 180s while the config appears to say "60 seconds". Always set both:

proxy_connect_timeout 3s;
proxy_read_timeout    20s;
proxy_next_upstream   error timeout;
proxy_next_upstream_tries   2;
proxy_next_upstream_timeout 35s;   # hard ceiling across all attempts

Envoy is the opposite: because route timeout includes retries, setting per_try_timeout equal to timeout means the second attempt can never run. Pair them deliberately:

route:
  cluster: app
  timeout: 21s                 # total budget for this route
  retry_policy:
    retry_on: "5xx,reset,connect-failure"
    num_retries: 2             # 3 attempts total
    per_try_timeout: 6s        # 3 x 6s = 18s, leaving 3s of slack

A worked budget for a real chain#

Chain: browser, CloudFront, ALB, nginx, gunicorn application, PostgreSQL. Target: the slowest legitimate request is a report that takes about 6 seconds of database time.

Check the ladder in one line: 60 > 45 > 40 > 35 > 33 > 28 > 18 > 10 > 6. Every gap is at least 2 seconds and most are 5 or more. Run the same chain through the timeout ladder checker, which flags inversions and rungs whose retry-multiplied budget breaks the layer above.

Note which layer produces the useful error: PostgreSQL cancels the statement at 10s, the application catches it and logs the offending query, gunicorn is never involved, and the browser sees a real cause. Nobody manufactures a 504.

The keep-alive ladder runs the other way#

Request timeouts run longest-outside to shortest-inside. Idle keep-alive timeouts run shortest-outside to longest-inside, because the party that initiated the connection should be the party that closes it. If the inner side closes an idle pooled connection first, the outer side eventually writes a request onto a socket that is already closing and returns a 502.

That last row is the classic AWS 502. Gunicorn's default 2 second keep-alive and Node's default 5 second keepAliveTimeout are both far below the ALB's 60 second idle timeout, so the load balancer reuses connections the backend has already discarded. The full mechanism is in keep-alive and upstream connection pooling.

What each layer returns when a timer fires#

The pairing to memorise: 504 means somebody waited and got nothing; 502 means somebody got something unusable, usually a closed connection. A 503 in HAProxy and Envoy commonly means the connection was never established at all. 502 vs 503 vs 504 works through the distinctions in detail.

Connect, read, and absolute timeouts are not interchangeable#

• Connect timeout covers TCP (and TLS) establishment only. Inside one VPC it belongs at 1 to 3 seconds: a large value does not make a healthy backend more reachable, it only delays discovering that an unhealthy one is not. DNS resolution is frequently outside it, since nginx has a separate resolver_timeout (default 30s) that no proxy timeout accounts for.
• Read and send timeouts are idle timers between successive I/O operations in nginx and HAProxy. They do not bound total duration.
• Absolute timeouts do: Envoy's route timeout and max_stream_duration, PostgreSQL's statement_timeout, gunicorn's --timeout with sync workers.

A chain built only from idle timers has no ceiling anywhere, which is how requests survive for hours. At least one rung must be absolute, and it should be as close to the work as possible.

Failure modes#

The 60 second cliff. Symptom: a cluster of failures at exactly 60 seconds, with different status codes from different clients. Cause: nginx proxy_read_timeout, nginx client_body_timeout, and the AWS ALB idle timeout all default to exactly 60s, so three layers fire simultaneously and none of them is clearly first. Fix: never leave two adjacent hops on the same default. Move each rung to a distinct value so the log timestamps tell you who won.

ALB idle timeout above the target's keep-alive. Symptom: sporadic 502s in the ALB access log with target_status_code -, uncorrelated with load and often worse when traffic is low. Cause: the backend closes idle keep-alive connections before the ALB does. Fix: raise the target's keep-alive above the ALB idle timeout (gunicorn --keep-alive 75, Node server.keepAliveTimeout = 65000 with headersTimeout above it).

Retries turning one slow request into a load spike. Symptom: a backend slows slightly, then load doubles or triples and it collapses. Cause: every layer retries independently, so n layers each retrying twice produce up to 2^n attempts for one client request. Fix: retry at exactly one layer, cap the attempts, bound the total, and add a retry budget or circuit breaker so retries cannot exceed a fraction of live traffic.

Streaming endpoints truncated at a round number. Symptom: an SSE or WebSocket connection dies at exactly 60s or 300s. Cause: an idle timer sized for request/response traffic applied to a long-lived stream. Fix: raise proxy_read_timeout on that location only, set HAProxy's timeout tunnel or Envoy's stream_idle_timeout, and heartbeat from the application at well under the smallest idle timer in the chain.

A 504 with a healthy application. Symptom: the edge returns 504, the application log shows the request completing successfully a few seconds later. Cause: inverted ladder. Fix: walk the chain inward comparing each rung to the one outside it, and find the first non-decreasing pair.

Health checks sharing the request timeout. Symptom: during a slowdown, backends are marked unhealthy en masse and the outage widens. Cause: health check timeouts inherited from request timeouts, so a slow-but-working backend fails its checks. Fix: give health checks an independent budget, as discussed in health checks and upstream failover.

Frequently asked questions#

Should timeouts get shorter or longer as you move toward the client?#

Longer. The client-facing timeout is the longest in the chain, and each hop closer to the actual work has a shorter one. This ensures the layer with the most context about the failure is the one that gives up first and produces the error, rather than an outer layer synthesising a generic 504.

Why do I get a 504 with nothing in the application log?#

Because an outer layer's timeout fired before the application's did. The application saw a client disconnect rather than a timeout, so it logged a broken pipe, a 499, or nothing. Compare each hop's timeout to the hop outside it and find the first pair that is not strictly decreasing.

Does nginx have a total request timeout?#

No. proxy_read_timeout and proxy_send_timeout measure the gap between two successive I/O operations, not the total duration, so a slowly trickling response can run indefinitely. If you need an absolute ceiling, it must come from Envoy's max_stream_duration, a load balancer idle timeout, or the application itself.

How do I calculate a timeout budget with retries?#

Multiply. The worst case for a rung is attempts x per-try timeout plus backoff, and that total must fit under the timeout of the layer above it. Envoy's route timeout already includes retries, so per_try_timeout should be roughly timeout / attempts. nginx does not bound the total unless you set proxy_next_upstream_timeout, which defaults to 0 (unlimited).

Why does my ALB return 502 with target_status_code "-"?#

Because the target closed a keep-alive connection that the ALB then tried to reuse. The ALB's default 60 second idle timeout is longer than gunicorn's default 2 second and Node's default 5 second keep-alive. Raise the target's keep-alive above the ALB idle timeout.

What is the difference between an idle timeout and an absolute timeout?#

An idle timeout resets whenever data moves, so a connection that keeps trickling bytes never expires. An absolute timeout bounds total duration regardless of activity. Most proxy timeouts (nginx proxy_read_timeout, HAProxy timeout server, ALB idle timeout) are idle timers, so a chain built only from them has no ceiling at all.

Do timeouts need to be different at every hop?#

They need to differ enough to be distinguishable. Two adjacent hops sharing a default (the 60s that nginx and AWS ALB both ship with) fire together, so the logs cannot tell you which failed first. Give each rung a distinct value with a margin of at least a couple of seconds, and verify the whole chain with the ladder checker.