Keep-alive and upstream connection pooling

A reverse proxy maintains two entirely separate connection lifetimes: the one between the client and the proxy, and the one between the proxy and the upstream. They have different directives, different defaults, and different failure modes. The single most common production mistake is assuming that because keepalive_timeout 75s; appears in an nginx config, nginx is reusing upstream connections. That directive says nothing about the upstream leg. On nginx before 1.29.7 it is worse than uninformative: nginx opened a fresh TCP connection to the upstream for every single request unless you configured an upstream block with the keepalive directive and set proxy_http_version 1.1 and cleared the hop-by-hop Connection header. nginx 1.29.7 changed the defaults so that upstream connection caching is active (keepalive 32 local), proxy_http_version defaults to 1.1, and the Connection proxy header is no longer sent, so a stock configuration now reuses connections.

On anything older, that default costs a TCP handshake (and a TLS handshake, if the upstream leg is encrypted) per request, and it burns one ephemeral port per request that then sits in TIME_WAIT for 60 seconds. Because long-term-support distributions ship older nginx for years, check the version you are actually running rather than assuming either behaviour.

The two lifetimes, and why they get conflated#

The naming collision drives most of the confusion: keepalive_timeout exists in two modules with two meanings and two defaults (75s client-side, 60s inside upstream). Reading a config top to bottom you cannot tell them apart without checking which block they sit in.

The nginx trap, and the exact config that fixes it#

Before 1.29.7, nginx sent HTTP/1.0 to upstreams by default (proxy_http_version 1.0), and HTTP/1.0 has no persistent connections without Connection: keep-alive. It also passed through a client's Connection: keep-alive or Connection: close header if you did not clear it, which can cause the upstream to close a connection nginx wanted to keep. On 1.29.7 and later all three of these are already the default, but writing them explicitly is harmless and keeps a config portable across versions:

upstream app {
    server 10.0.2.11:8080;
    server 10.0.2.12:8080;

    # 1. Size the pool. Before 1.29.7 there was no default and the absence
    #    of this line meant no pooling at all; from 1.29.7 the default is
    #    "keepalive 32 local". This is idle connections PER WORKER PROCESS,
    #    not a connection limit.
    keepalive 64;

    # Optional but recommended: bound reuse in requests and in wall-clock time.
    keepalive_requests 1000;   # default 1000 since nginx 1.19.10 (was 100)
    keepalive_timeout  60s;    # nginx closes an idle pooled conn after this
    keepalive_time     1h;     # nginx 1.19.10+, max lifetime of a reused conn
}

server {
    location / {
        proxy_pass http://app;

        # 2. HTTP/1.1 is required for persistent connections.
        #    Default since 1.29.7; was 1.0 before that.
        proxy_http_version 1.1;

        # 3. Clear the client's hop-by-hop Connection header.
        #    Empty value means nginx omits the header entirely.
        #    From 1.29.7 nginx no longer sends it by default.
        proxy_set_header Connection "";

        proxy_set_header Host $host;
    }
}

Two details that bite people. First, keepalive 64; is per worker process: with worker_processes auto; on a 16-core box that is up to 1024 idle connections held against the upstream group, so size it against the upstream's connection limit rather than your request rate. Second, on nginx before 1.29.7, if you proxy_pass to a literal address rather than a named upstream (proxy_pass http://10.0.2.11:8080;) there is no upstream block, therefore no pool, therefore no reuse, a common side effect of the URI-rewriting patterns in nginx proxy_pass and the trailing slash.

Defaults by implementation#

Caddy and Traefik both sit on Go's net/http transport, which matters if you write your own Go proxy: http.DefaultTransport uses DefaultMaxIdleConnsPerHost = 2, so a hand-rolled httputil.ReverseProxy on the default transport churns connections under any real concurrency even though keep-alive is nominally enabled. Caddy and Traefik override that value; your code does not unless you say so.

What HAProxy's http-reuse safe actually protects against#

HAProxy has offered four reuse strategies since 1.9, and the default became safe in HAProxy 2.0. The four are not a simple aggressiveness dial:

What safe protects is retryability, not correctness. When a request is written onto a pooled connection the server is closing at the same instant, somebody must retry it. Browsers universally retry a request that failed on a connection where no response was ever received, but only reliably for the first request of that connection. safe therefore arranges for the risky first-use case to land on the one party that already cleans up after it. It is not preventing the race, it is placing the race where it is already handled.

Related knobs: pool-max-conn <n> on a server line caps idle connections held for that server (0 disables idle pooling), and pool-purge-delay (default 5s) controls how quickly unused idle connections are reaped.

Envoy's model: one pool per cluster, per worker#

Envoy has no "enable keep-alive" switch because it is always on. Each worker thread holds its own connection pool per cluster and per priority, so a multi-threaded Envoy holds more sockets than a naive reading of a single max_connections circuit breaker suggests. The two fields to know:

clusters:
- name: app
  connect_timeout: 2s
  typed_extension_protocol_options:
    envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
      "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
      common_http_protocol_options:
        idle_timeout: 30s          # default 1h
        max_requests_per_connection: 1000   # default 0 = unlimited
      explicit_http_config:
        http_protocol_options: {}

max_requests_per_connection matters when an intermediate L4 load balancer sits between Envoy and the real backends. With the default of 0 (unlimited), a pooled connection pinned to one backend stays pinned forever and newly added backends receive no traffic at all. A finite value forces periodic redial and rebalancing. The same reasoning applies to nginx's keepalive_requests and keepalive_time, and it is the main reason to lower those rather than raise them. See health checks and upstream failover for the companion problem of a backend that is up but useless.

The idle-close race that produces intermittent 502s#

This is the single most reported keep-alive failure. Symptom in the nginx error log:

2026/09/08 11:04:19 [error] 1123#1123: *88213 upstream prematurely closed
connection while reading response header from upstream, client: 10.1.0.7,
server: api.example.com, request: "POST /v1/orders HTTP/1.1",
upstream: "http://10.0.2.11:8080/v1/orders", host: "api.example.com"

The client sees a 502 Bad Gateway. It happens for a small fraction of requests, is not reproducible on demand, and correlates with periods of low traffic rather than high, because idle connections are the ones at risk.

The mechanism: the upstream decides an idle pooled connection is dead and sends FIN; within the same round trip the proxy pulls that connection from its pool and writes a request onto it; the request lands on a closing socket and no response header is ever read. There is no way to eliminate this race in HTTP/1.1, because there is no in-band "I am about to close" signal that arrives early enough. You can only make it rare and make the residue harmless:

Rule 1: the proxy must close idle connections before the upstream does. Whoever closes an idle connection when nothing is in flight causes no error. Whoever closes it under a request causes a 502. So set the upstream application's idle keep-alive timeout longer than the proxy's pooled idle timeout, with a comfortable margin. nginx's upstream keepalive_timeout defaults to 60s, so the app needs something above that. This is the same ordering discipline as the timeout ladder, applied to idle time instead of request time.

Rule 2: retry idempotent requests. nginx's proxy_next_upstream defaults to error timeout, which does cover this case. But since nginx 1.9.13, requests with non-idempotent methods (POST, LOCK, PATCH) are not retried once the request has been sent, unless you explicitly add the non_idempotent parameter. That default is correct and you should generally leave it: retrying a POST that may have been processed is worse than a 502.

proxy_next_upstream error timeout http_502;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 10s;

Envoy expresses the same intent with retry_on: reset, HAProxy with retry-on conn-failure empty-response. In every implementation a retry multiplies worst-case latency, which is why proxy_next_upstream_timeout and Envoy's per_try_timeout exist.

Ephemeral port exhaustion arithmetic#

If pooling is off, every request needs a fresh source port on the proxy host, and the proxy is the side that initiates the close, so the socket enters TIME_WAIT and holds the four-tuple.

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	60999

That range is 28,232 ports. On Linux, TIME_WAIT duration is a compile-time constant of 60 seconds (TCP_TIMEWAIT_LEN), not a sysctl. The sustainable ceiling is therefore:

28,232 ports / 60 seconds ≈ 470 new connections per second to a single destination (IP, port) pair.

The critical qualifier is per destination pair: the kernel only needs the four-tuple (src IP, src port, dst IP, dst port) to be unique, so 470/s is the limit to one upstream socket address. Ten upstream servers give roughly ten times the headroom, which is exactly why this hides in staging and appears in production behind a single internal load balancer VIP. EADDRNOTAVAIL surfaces in nginx as:

[crit] 1123#1123: *44219 connect() to 10.0.2.5:8080 failed
(99: Cannot assign requested address) while connecting to upstream

Note the severity: crit, not error, and errno 99. That is not a backend problem, that is your own host out of source ports.

Observing it#

# Summary: total sockets, and the timewait count
ss -s

# Count TIME_WAIT sockets against one specific upstream
ss -tan state time-wait dst 10.0.2.5:8080 | wc -l

# Distribution across upstreams, which shows a single hot destination fast
ss -tan state time-wait | awk '{print $5}' | sort | uniq -c | sort -rn | head

# Kernel counters, for rate rather than instantaneous depth
nstat -az | grep -iE 'TcpExtTW|TcpActiveOpens'

If ss -s shows a timewait count in the tens of thousands against one peer and TcpActiveOpens is climbing at hundreds per second, you have found it.

Mitigations, in order of correctness#

1. Enable upstream keep-alive. This is the fix. It removes the churn rather than making the churn survivable.
2. Widen the port range: net.ipv4.ip_local_port_range = 1024 65535 raises the ceiling to roughly 1,075 per second per destination. You are now able to allocate source ports that collide with listening ports on the same host, so keep the floor at 1024.
3. net.ipv4.tcp_tw_reuse = 1 lets the kernel reuse a TIME_WAIT socket for a new outbound connection when TCP timestamps make it safe. Reasonable on a busy proxy; it does nothing for inbound.
4. Do not use tcp_tw_recycle. It broke clients behind NAT and was removed from Linux in 4.12. A runbook recommending it predates 2017.
5. Add upstream addresses. More destination pairs, more four-tuples. A workaround, not a fix.

Failure modes#

Reuse silently disabled by a missing directive. Symptom: upstream connection counts equal request counts, and ss -s shows a large TIME_WAIT figure. Cause: on nginx before 1.29.7, keepalive present but proxy_http_version still 1.0, or Connection not cleared. Verify by counting TcpActiveOpens over a 10s window and comparing to requests served.

keepalive set too high against a small upstream. Symptom: the upstream hits its own connection limit and refuses, producing 502s under low load, because idle pooled connections still occupy upstream slots. Cause: keepalive N is per worker; compute N * worker_processes.

Connections pinned behind an intermediate L4 balancer. Symptom: newly added backends receive no traffic and load stays uneven. Cause: unbounded reuse. Fix: cap requests per connection and connection lifetime so the pool redials.

Upgrade requests forced through the pool. Symptom: WebSocket handshakes fail or hang after enabling proxy_set_header Connection ""; globally. Cause: an Upgrade request needs Connection: upgrade, not an empty Connection header. Fix: scope the empty Connection header to non-upgrade locations, or use a map. Full treatment in WebSockets through a reverse proxy.

HTTP/2 to the client hides upstream churn. Symptom: client-side connection metrics look excellent, upstream metrics look terrible. Cause: HTTP/2 multiplexing on the client leg is unrelated to the upstream leg, which is usually still HTTP/1.1. See HTTP/2 and HTTP/3 through proxies.

502s only during deploys. Symptom: a burst of upstream prematurely closed connection at every rolling restart. Cause: pooled connections to a terminating instance. Fix: drain by responding Connection: close on in-flight requests before the instance stops accepting, and set the termination grace period above the proxy's pooled idle timeout.

Frequently asked questions#

Does nginx use keep-alive to upstream servers by default?#

It depends on the version. From nginx 1.29.7 it does: upstream connection caching defaults to keepalive 32 local, proxy_http_version defaults to 1.1, and the Connection proxy header is no longer sent. On anything older it does not, and nginx opens a new TCP connection to the upstream for every request unless you define an upstream block containing the keepalive directive, set proxy_http_version 1.1, and clear the Connection header with proxy_set_header Connection "";. In either case the client-facing keepalive_timeout directive has no effect on the upstream leg.

What does the nginx keepalive number actually limit?#

It sets the maximum number of idle keep-alive connections to the upstream group retained per worker process. It is not a connection limit and not a concurrency limit: nginx will open more connections than that when it needs them, and close the excess down to keepalive N once they go idle. Multiply by worker_processes to get the fleet-wide idle footprint.

Why do I get random 502s with "upstream prematurely closed connection"?#

Because the upstream closed an idle pooled connection at the same moment the proxy wrote a request onto it. It is a race inherent to HTTP/1.1 keep-alive. Make it rare by setting the upstream application's idle timeout longer than the proxy's pooled idle timeout, and make the remainder harmless by retrying idempotent requests. See 502 vs 503 vs 504 for distinguishing this from other gateway errors.

What changed about keepalive_requests in nginx 1.19.10?#

nginx 1.19.10 raised the default keepalive_requests from 100 to 1000, in both the client-facing core module and the upstream module, and introduced the keepalive_time directive with a default of 1 hour to bound total connection lifetime independently of request count. On nginx older than 1.19.10 the effective default is still 100 requests per connection.

How many connections per second can one host make before ephemeral ports run out?#

Roughly 470 per second to a single destination IP and port, using the Linux default net.ipv4.ip_local_port_range of 32768-60999 (28,232 ports) and the fixed 60 second TIME_WAIT. The limit is per destination four-tuple, so more upstream addresses raise it proportionally. Widening the range to 1024-65535 gets you to about 1,075 per second.

Should I set net.ipv4.tcp_tw_reuse or tcp_tw_recycle?#

tcp_tw_reuse=1 is safe for outbound connections on a proxy and helps when you are near the port ceiling. tcp_tw_recycle must never be used: it broke clients behind NAT and was removed from the Linux kernel in version 4.12. Neither is a substitute for enabling upstream keep-alive.