502 vs 503 vs 504: reading proxy errors correctly

Three statuses, three different experiences at the proxy. 502 means the proxy reached an upstream but the reply was unusable or the connection broke before a complete response header arrived. 503 means the proxy had no usable upstream to try, or refused the request itself (rate limit, queue full, maintenance mode, no healthy backend). 504 means the proxy successfully engaged an upstream and then gave up waiting, so a timer it owns expired.

502 is a reply problem, 503 is a selection problem, 504 is a clock problem. The status names which of the proxy's three jobs failed, not what is wrong with the application.

Decision table#

The trap is assuming the status has the same meaning in every proxy. It does not.

First branch: intermittent or constant#

Before reading a single log line, establish which of two entirely different investigations you are in.

Constant (every request fails, immediately and identically) means a static misconfiguration or a hard-down upstream. It reproduces from the proxy host in one command, and the fix is in the proxy config, DNS, a firewall rule, a listener port, or a process that is not running.

Intermittent (a small percentage, bursty, often correlated with load or a deploy) means a race, an exhaustion, or a timeout sitting close to real latency. Nothing run by hand reproduces it, so log correlation is the only tool. The four generators, in rough order of frequency:

1. Keep-alive idle race. The upstream closes a pooled connection as the proxy writes a request onto it. Gives 502 with upstream prematurely closed connection or recv() failed (104: Connection reset by peer).
2. A timeout near the p99. The endpoint answers in 45s and proxy_read_timeout is 60s, so the tail crosses the line. Gives 504 in a pattern that tracks load.
3. Capacity edges. Upstream pool saturation, maxconn queues, or a proxy connection cap. Gives 503 and 504 together, in bursts.
4. Rolling deploys. Instances leave rotation after they stop accepting connections rather than before. Gives a short spike aligned with deploy timestamps.

An intermittent 502 that hits POST, PUT, PATCH and DELETE while GET looks clean is the keep-alive race and nothing else. proxy_next_upstream defaults to error timeout, and nginx will not retry a non-idempotent request unless non_idempotent is added. The identical socket race is therefore retried and hidden for reads, and surfaced as a 502 for writes. The asymmetric error rate by method is the fingerprint.

nginx: error_log signatures and the status each produces#

nginx writes the cause to error_log and the status to access_log; the pairing identifies the fault. Each line carries a while ... clause naming the phase, and that clause is the important half.

Defaults behind most of these: proxy_connect_timeout 60s, proxy_read_timeout 60s, proxy_send_timeout 60s, max_fails 1, fail_timeout 10s. The nginx documentation notes that proxy_connect_timeout "cannot usually exceed 75 seconds", which is the operating system's own TCP connect limit rather than a cap nginx enforces.

Worked example: the keep-alive 502#

upstream app {
    server 10.0.3.11:8080;
    keepalive 32;                # upstream keepalive_timeout defaults to 60s (nginx 1.15.3+)
}                                # keepalive_requests defaults to 1000 (nginx 1.19.10+)

server {
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;  # required, or nginx opens a new connection per request
        proxy_set_header Connection "";
    }
}

The upstream is a Node service whose server.keepAliveTimeout is 5 seconds, while nginx believes it may hold an idle connection for 60. Between second 5 and second 60 the upstream sends FIN on idle sockets. A request written into that window produces:

2026/09/08 11:04:22 [error] 812#812: *91043 upstream prematurely closed connection
while reading response header from upstream, client: 198.51.100.7,
server: api.example.com, request: "POST /v1/orders HTTP/1.1",
upstream: "http://10.0.3.11:8080/v1/orders", host: "api.example.com"

and a 502 in access_log, with nothing in the application log because the request never completed a round trip. The fix belongs on the upstream: its idle timeout must exceed the proxy's, because only the side that closes first can lose the race. Full treatment in keep-alive and upstream connection pooling.

HAProxy: termination state flags#

HAProxy's log line contains a four-character termination state such as sH--. Only the first two characters carry the diagnosis: character one is what ended the session, character two is which phase it was in. Characters three and four describe persistence cookies and are usually --.

For the two case pairs the manual defines, the convention is worth internalising: C/S mean an abort by the client or the server, c/s mean a timeout on that side. S versus s is the difference between "the server said no" and "the server said nothing". The rule does not generalise to the other first characters, which name a cause rather than a party: P is the proxy blocking or denying, L is HAProxy answering locally, R is a resource on the proxy exhausted, I an internal error, D and U connections killed because a server went down or a primary came back, and K an administrative kill.

HAProxy ships no defaults for timeout connect, timeout client and timeout server: leaving any of them unset gives an infinite timeout and a startup warning. retries defaults to 3 (CONN_RETRIES in the source; the manual documents no default), and option redispatch is what lets those retries land on a different server, which converts many SC events into successful requests.

Envoy: response flags#

Envoy puts a compact flag set in %RESPONSE_FLAGS% in the access log, and also writes a human-readable reason into the response body, which is unusual and useful.

Two Envoy specifics catch people out. The route-level timeout defaults to 15 seconds, far shorter than nginx's 60, and it is the usual cause of unexpected UT after moving to Envoy or to a mesh built on it. And NR is a 404, not a 5xx, so a routing mistake presents as "not found": when users report 404s the application never logged, check NR before the application router. Configuration context is in Envoy listeners, routes and clusters.

A systematic diagnosis procedure#

Run these in order. Each step either fixes the problem or eliminates a layer.

1. Reproduce from the proxy host, against the upstream, with no proxy in the path. curl -sS -o /dev/null -w '%{http_code} %{time_connect} %{time_starttransfer}\n' http://10.0.3.11:8080/healthz. Identical failure means the proxy is reporting the truth and the investigation moves upstream. Success means the fault is in the proxy's view of the upstream: resolution, port, TLS expectations, source-address firewall rules, or health-check state. Flag syntax is in curl through a proxy.
2. Check whether the upstream logged the request at all. The highest-value single step. Present in the upstream log with a 200 and a short duration means the proxy connected, sent, and the application answered, so the failure is in reading the response: header size, framing, or a connection torn down mid-response. Absent from the upstream log means it never arrived: selection, connect, or admission control. The two branches share almost no root causes, so one query halves the search space.
3. Compare every timeout in the chain against measured latency. Each hop's timeout must be shorter than that of the hop in front of it, or the outer hop reports 504 while the inner one keeps working on a request nobody is waiting for. Work through timeout budgets across a proxy chain and check the ladder with the timeout ladder checker.
4. Check the keep-alive idle race on any intermittent 502. The application server's idle timeout must exceed the proxy's. Confirm with the method asymmetry above.
5. Check header size limits. Large Set-Cookie sets, SAML assertions, JWTs and verbose CORS policies routinely exceed a 4k or 8k header buffer. Symptom: a 502 only for authenticated users.
6. Check upstream saturation and health checks. A worker pool with N slots serving requests of duration T handles N/T requests per second and queues the rest; queued time is invisible to the application's own duration metric and fully visible to the proxy timeout, which is why 504 rates climb while application latency percentiles look flat. Separately, a 503 with all backends "down" is often a health check stricter than the traffic it gates: a path requiring auth, a Host header the vhost does not match, or a 2s timeout against a 3s warm-up. See health checks and upstream failover.

Failure modes worth recognising on sight#

502 on exactly one endpoint, 200 everywhere else. Almost always response header size, or a response the proxy cannot frame (a Content-Length that disagrees with the body, or both Content-Length and Transfer-Encoding). Measure the header block with curl -sD - -o /dev/null against the upstream.

504 at a suspiciously round number of seconds. 15s is an Envoy route default. 30s is a common HAProxy defaults block. 60s is nginx proxy_read_timeout. 75s is the practical ceiling on proxy_connect_timeout, imposed by the OS TCP connect timeout rather than by nginx. If the observed duration is one of these, you have identified the hop that gave up without reading any logs.

503 with UO in Envoy while the upstream is idle. A circuit breaker limit (max_connections, max_pending_requests, max_requests) is at its low default. The backend has capacity; Envoy is refusing to use it.

Alternating 502 and 504 for the same endpoint. Two upstreams behind one name, one refusing connections and one hanging. Log $upstream_addr and group by it.

Frequently asked questions#

What is the difference between 502 and 504?#

A 502 means the proxy received something from the upstream that it could not use, including an abrupt disconnect, so a byte exchange happened and went wrong. A 504 means the proxy received nothing in time and abandoned the request when its own timer expired. Failing fast points at 502; failing after a fixed duration points at 504.

Why does nginx return 502 instead of 503 when all upstreams are down?#

Because nginx maps its "no live upstreams" condition to NGX_HTTP_BAD_GATEWAY. When every server in an upstream block has been marked failed by max_fails inside fail_timeout, nginx logs no live upstreams while connecting to upstream and returns 502, where HAProxy and Envoy return 503.

What does "upstream prematurely closed connection" mean in nginx?#

It means the upstream accepted the connection and then sent a FIN before nginx had read a complete response header. The two dominant causes are an application worker that crashed or was recycled mid-request, and a keep-alive connection the upstream closed on its idle timer at the moment nginx reused it.

What does the HAProxy termination state sH mean?#

sH means HAProxy's timeout server expired while it was waiting for the backend to send response headers, and HAProxy returned 504. The lowercase s indicates a server-side timeout rather than an abort by the server, and H indicates the header-waiting phase. The request did reach the backend, so it usually appears in the backend log with a long duration or no completion.

Should clients retry a 502, 503 or 504?#

Retrying 503 is usually safe and often correct, especially when the response carries Retry-After, because the request typically never reached the application. Retrying 504 risks duplicating work that is still running upstream. Retrying 502 is safe only for idempotent methods, since the proxy cannot know whether the upstream processed the request before the connection broke.

Why do I only see 502 errors on POST requests?#

Because nginx's proxy_next_upstream will not retry non-idempotent methods by default, so the same underlying connection race that is transparently retried for GET is surfaced as an error for POST, PUT, PATCH and DELETE. The error rate difference by method is diagnostic: it identifies a connection-reuse race rather than an application fault. Fixing the upstream's idle timeout removes both, whereas adding non_idempotent to proxy_next_upstream only hides it and risks duplicate writes.