Health checks and upstream failover

A reverse proxy decides where to send a request using two independent signals: active health checks, where the proxy periodically probes each backend on its own schedule, and passive health checks, where the proxy infers health from the outcome of real user requests. The single most consequential fact when choosing a proxy is that nginx open source only has passive checks; the health_check directive is a commercial nginx Plus feature. HAProxy, Envoy, Traefik and Caddy all include active checking in the free product.

Active vs passive in one paragraph each#

Passive checking costs nothing when idle and reacts to exactly the failures users hit, including ones no synthetic probe would notice. Its weakness is that it needs a victim: a dead backend is discovered only when real requests fail against it, and it returns to rotation on a timer rather than on evidence of recovery. At low traffic it can sit undetected for minutes.

Active checking detects failure without harming a user, and detects recovery positively rather than by optimistically retrying. Its weaknesses: the probe may not exercise the code path users do (a static /healthz returning 200 while the request thread pool is deadlocked), and under load it competes with real traffic and times out, ejecting a backend that is merely busy. The two are complements, not alternatives, which Envoy states most clearly by exposing active health_checks and passive outlier_detection separately.

How the five proxies configure checks#

Two readings of that table matter. The defaults differ by an order of magnitude: HAProxy probes every 2 seconds, Traefik and Caddy every 30, so a failure HAProxy notices in 6 seconds takes Traefik 30 or more. And nginx's passive-only model defines "failure" as whatever proxy_next_upstream says, which by default is only error timeout. A backend returning 500 to every request is, to default nginx, perfectly healthy, which is why a broken deploy behind nginx OSS serves errors forever instead of failing over. Adding http_500 http_502 http_503 changes that, at the cost of retrying non-idempotent requests unless you also set non_idempotent.

upstream app {
    server 10.0.1.10:8080 max_fails=3 fail_timeout=15s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=15s;
    server 10.0.1.12:8080 backup;
}

server {
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_connect_timeout 2s;
    }
}

max_fails=3 fail_timeout=15s reads as: three failures within 15 seconds marks the server unavailable for 15 seconds. Both windows are the same parameter, which is the common misreading. max_fails=0 disables the mechanism and keeps the server permanently in rotation.

Liveness, readiness, and why deep checks cause outages#

Two questions need two endpoints.

• Liveness: is this process irrecoverably broken and in need of a restart? Check only in-process state, never a dependency, because restarting a healthy process does not fix someone else's database.
• Readiness: should this instance receive traffic right now? May fail for warm-up not finished, pool exhausted, or deliberate draining.

The load balancer must probe readiness; the orchestrator probes liveness.

The rule that follows: a health check should only report failures that failing over to another instance can fix. A shared dependency being down is by definition not one of those. Monitor shared dependencies with alerting, not with the endpoint that controls routing, and if you need the deep check, degrade rather than fail: return 200 with a degraded-status body and alert on that separately.

The counter-case is a genuinely per-instance dependency: a local cache that failed to load, a per-shard connection this replica alone cannot make, a config file that failed to parse. Those are correct things to fail readiness on, because a peer really can serve the request.

Fail open or fail closed?#

When every backend is marked unhealthy, what should the proxy do? Two answers exist and the choice should be deliberate.

Envoy fails open by default via the panic threshold, which defaults to 50%: if fewer than half the hosts in a cluster are healthy, Envoy ignores health status entirely and balances across all hosts. The reasoning is that mass unhealthiness usually means the checking is wrong (a bad probe path, a partition between proxy and backends) rather than that every backend died, and possibly-working backends beat nowhere.

nginx and HAProxy fail closed: with no available upstream nginx logs no live upstreams while connecting to upstream and returns 502, and HAProxy returns 503 with NOSRV in the log. Both are covered in HAProxy configuration for HTTP reverse proxying. That is safer for correctness (you never route to a backend that failed a check) and worse for availability during a checking bug. Which status you see and what it implies is covered in 502 vs 503 vs 504.

Pick fail-open when the check is shallow and a wrong ejection is expensive. Pick fail-closed when routing to a bad backend causes damage a 503 does not, such as corrupting data or double-charging.

A worked timeline: how a rollout marks every backend down#

This is the interaction that produces "the deploy succeeded and the site was down for 20 seconds". Four backends, an L7 proxy with interval 10s and healthy_threshold 2 (so 20 seconds from process start to being marked up), and an orchestrator that considers a pod ready 5 seconds after start using its own faster probe, then proceeds to the next pod.

Between roughly t=15 and t=25 every process is up and serving while the proxy believes nothing is healthy. Users get 503s, unless you are on Envoy and panic mode engages below 50% healthy, which is the one case where fail-open earns its keep.

Slow start, draining and shutdown order#

Slow start solves the mirror-image problem: a backend just marked healthy receives its full share of traffic instantly, with an empty cache, a cold JIT and an unwarmed pool, then times out and gets ejected again. HAProxy's slowstart ramps a server's weight linearly over a configured period; Envoy has slow_start_config on the load balancing policy with a configurable aggression curve (see Envoy listeners, routes and clusters); nginx Plus has slow_start. Caddy and Traefik have no equivalent, so the application must warm itself before passing readiness.

Graceful shutdown must happen in this order, and the order is what people get wrong:

1. Receive the termination signal. Do not close the listener yet.
2. Start failing the readiness endpoint.
3. Keep serving normally for longer than the proxy's detection window (interval x unhealthy_threshold plus config propagation delay). This is the step that is almost always missing.
4. Stop accepting new connections, and send Connection: close on keep-alive responses so pooled connections are retired rather than reused.
5. Wait for in-flight requests, up to a grace period, then exit.

Skipping step 3 is the classic source of deploy-time 502s: the process closes its listener the instant it gets SIGTERM while the proxy still has it marked healthy and holds pooled keep-alive connections to it. Because those connections are pooled, the damage outlasts the check interval; see keep-alive and upstream connection pooling for why a pooled connection can be handed a request microseconds after the peer decided to close it. Long-lived connections make step 5 unbounded, which is why streaming backends need a forced cutoff and client-side reconnect.

Outlier detection is the passive complement. In Envoy, consecutive_5xx (default 5) or success_rate deviation ejects a host for base_ejection_time (default 30s), doubling on repeat ejections, with max_ejection_percent (default 10%) capping how much of the cluster can be ejected at once. That cap stops cascading ejection: if a shared dependency makes every host return 5xx, uncapped detection would eject everything.

Failure modes#

Flapping. A backend oscillates between up and down every few intervals. Usually the check timeout sits close to the backend's p99 latency under load, so probes fail whenever the instance is busy, which ejects it, which loads the survivors, which makes them slow, which ejects them. Fix by raising the check timeout well above p99, raising fall or unhealthy_threshold so one slow probe cannot eject, and using a check endpoint that does no real work.

Thundering herd on recovery. All backends are marked healthy in the same second, every backing-off client retries at once, caches are cold, and the cluster falls over again. Mitigate with slow start, jittered client retry and staggered probe start times.

Health check traffic in your metrics. HAProxy at inter 2000ms against 20 servers generates 600 probe requests per minute into your access logs and dashboards. It inflates request counts, drags average latency down (probes are fast) and hides a real drop in user traffic. Probe a dedicated port or path and exclude it from access logging, for example access_log off; inside the nginx health check location.

The check bypasses the layer that is broken. A probe hitting /healthz on the application server directly, or a route registered before the authentication middleware, passes while every authenticated request fails. The probe should traverse the same middleware stack as real traffic minus the parts requiring credentials, and if it must skip auth, keep that skip to one narrow path that is not publicly reachable: health endpoints routinely leak version numbers, dependency status and hostnames.

Affinity fights failover. Ejecting a backend moves its pinned users elsewhere and they lose session state. See sticky sessions and session affinity for how affinity behaves when the backend set changes.

Frequently asked questions#

Does nginx open source support active health checks?#

No. nginx open source only supports passive checks through the max_fails and fail_timeout parameters on an upstream server, which infer health from real request outcomes. The health_check directive that performs active probing is part of nginx Plus.

What does max_fails=3 fail_timeout=15s actually mean in nginx?#

It means three failed attempts occurring within a 15 second window mark the server unavailable, and it stays unavailable for that same 15 seconds before nginx tries it again with live traffic. The one parameter defines both the counting window and the exclusion period. Defaults are max_fails=1 and fail_timeout=10s.

Should a health check query the database?#

Not the check your load balancer routes on. A shared dependency failing makes every backend fail at the same moment, leaving the proxy with zero healthy upstreams and turning a partial failure into a full outage. Check only conditions that failing over to another instance would actually resolve.

What is the difference between liveness and readiness probes?#

Liveness answers "should this process be restarted", and must only inspect in-process state. Readiness answers "should this instance receive traffic right now", and may fail temporarily during warm-up or draining. The load balancer must use readiness; the orchestrator uses liveness for restarts.

How long should I wait after SIGTERM before closing the listener?#

Longer than the proxy's detection window, which is the check interval multiplied by the unhealthy threshold, plus any configuration propagation delay. Fail the readiness endpoint immediately but keep serving during that wait, otherwise the proxy is still routing traffic to a socket you have already closed.

Why do I get 502s during every deploy even though the deploy succeeds?#

Almost always because instances close their listeners immediately on SIGTERM while the proxy still has them marked healthy and holds pooled keep-alive connections to them. Add a drain delay before closing the listener, and make sure connection pools are retired with Connection: close rather than reused.

What is Envoy's panic threshold?#

It is the point, 50% of hosts healthy by default, below which Envoy stops honouring health status and balances across all hosts. The assumption is that mass unhealthiness more often means a broken health check than genuinely dead backends, so failing open preserves more availability than returning 503 to everything.