HAProxy configuration for HTTP reverse proxying

An HAProxy configuration is a small number of sections with strict inheritance: global configures the process (privileges, tuning, the stats socket), defaults supplies values inherited by every proxy section declared after it, frontend binds sockets and decides which backend a request goes to, and backend holds the servers and the balancing policy. listen fuses a frontend and a backend into one block for cases with a single backend. The two things that most often go wrong are not syntax, they are rule evaluation order and the timeout ladder, so both get their own tables below.

Section reference#

A defaults section applies to every proxy defined below it until the next defaults. That is a positional rule, not a lexical one, so a second defaults block halfway down a file silently changes the behaviour of everything after it.

mode http versus mode tcp#

mode decides how much of the byte stream HAProxy understands, and it changes far more than header access.

Use mode tcp when you must not terminate TLS (see TLS termination, passthrough and re-encryption) and route on SNI instead, or for non-HTTP protocols. Everything else belongs in mode http, because per-request balancing and observability are worth more than the small parsing cost.

Rule evaluation order#

This is the part that is not obvious from reading a config top to bottom, because rules run in category order first and declaration order second.

1. tcp-request connection (frontend, before anything is read; the only place src is guaranteed pre-PROXY-protocol semantics matter)
2. tcp-request session (frontend, after the handshake, so after PROXY protocol and TLS)
3. tcp-request content (frontend, after inspect-delay has buffered payload)
4. http-request (frontend), in declaration order
5. use_backend rules in declaration order, then default_backend
6. tcp-request content (backend)
7. http-request (backend), in declaration order
8. Server selection: balance, stickiness, queueing
9. Request sent; on the way back, http-response (backend) then http-response (frontend), then http-after-response

ACLs are named boolean tests evaluated lazily at the point a rule references them, and they are scoped to the section that declares them. Multiple conditions on one rule are ANDed; or must be written explicitly.

    acl is_api      path_beg /api/
    acl is_health   path      /healthz
    acl from_office src       203.0.113.0/24
    acl is_websocket hdr(Upgrade) -i websocket

    http-request deny if is_api !from_office !{ req.hdr(Authorization) -m found }
    use_backend be_ws  if is_websocket
    use_backend be_api if is_api
    default_backend be_web

path_beg /api/ is a prefix match on the path only; path is exact. Anonymous ACLs in braces ({ req.hdr(...) -m found }) are convenient inline but cannot be reused.

Header manipulation#

option forwardfor is the exception that does not follow those semantics: it appends the connection's source address to X-Forwarded-For. Two parameters matter:

• if-none adds the header only when the request has none. Use it when an upstream CDN is authoritative for the client IP.
• except <network> skips the header for sources in that network, typically a local health checker.

    option forwardfor except 127.0.0.0/8 if-none

Whether you should append or overwrite depends entirely on whether the peer is trusted, which is the subject of configuring trusted proxies; and getting it wrong is how client IP spoofing succeeds. To overwrite unconditionally, drop option forwardfor and write http-request set-header X-Forwarded-For %[src].

Balance algorithms#

The rule of thumb: roundrobin for uniform short requests, leastconn as soon as request durations vary by more than an order of magnitude, and random(2) when several HAProxy nodes balance over the same servers, because independent leastconn instances all pick the same "least loaded" server at the same instant.

For affinity that must survive server churn, cookie-based stickiness beats hashing; see sticky sessions and session affinity.

Server options and health checks#

With check alone the check is a TCP connect. option httpchk upgrades it to L7. Since HAProxy 2.2 the composable form is preferred and is much easier to extend:

    option httpchk
    http-check send meth GET uri /healthz ver HTTP/1.1 hdr Host app.internal
    http-check expect status 200

slowstart 30s matters more than it looks: a JIT-compiled or cache-cold application that returns UP instantly will be handed its full share of traffic and time out, flap DOWN, and repeat. The interaction between check intervals and failover is covered in health checks and upstream failover.

Timeouts and the log flag they produce#

HAProxy has no safe defaults here: timeout connect, timeout client and timeout server have no built-in value and HAProxy warns at startup if they are missing. The table below is the diagnostic shortcut, because option httplog prints a two-character termination state (termination_state) that identifies exactly which timer fired.

Reading the flags: the first character is who or what ended the session (c client timeout, s server timeout, C client abort, S server abort or reset, P proxy denied, R resource exhaustion, - normal), the second is which phase it was in (R reading the request, Q queued, C connecting, H reading response headers, D data transfer, L last data push, - complete). So sH is unambiguous: timeout server expired while waiting for response headers, and HAProxy synthesised a 504. SH with a capital S is a different bug entirely: the server closed or reset while sending headers, and you get a 502.

timeout http-keep-alive has a documented fallback chain worth knowing: if it is unset, timeout http-request applies to the wait for the next request, and if that is also unset, timeout client applies. That is why an unset keep-alive timeout with a generous timeout client quietly pins connections for minutes.

Worked example#

global
    log         /dev/log local0
    user        haproxy
    group       haproxy
    chroot      /var/lib/haproxy
    maxconn     40000
    # Runtime API. level admin is required for state changes.
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    mode                http
    log                 global
    option              httplog
    option              dontlognull
    # Do not retry non-idempotent requests after the response started.
    retries             2
    option              redispatch
    timeout connect     5s
    timeout client      30s
    timeout server      30s
    timeout http-request 10s      # slowloris guard
    timeout http-keep-alive 4s
    timeout queue       10s
    timeout tunnel      1h        # WebSockets outlive timeout server
    option              forwardfor except 127.0.0.0/8 if-none

frontend fe_https
    bind :443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    bind :80
    http-request redirect scheme https code 301 unless { ssl_fc }

    # Frontend response rules apply to denials too, so they belong here.
    http-response set-header Strict-Transport-Security "max-age=63072000"
    http-request set-header X-Forwarded-Proto %[ssl_fc,iif(https,http)]
    http-request set-header X-Request-Id %[uuid()] unless { req.hdr(X-Request-Id) -m found }

    acl is_api       path_beg /api/
    acl is_ws        hdr(Upgrade) -i websocket
    acl is_metrics   path        /metrics
    acl internal_src src         10.0.0.0/8

    http-request deny deny_status 404 if is_metrics !internal_src

    use_backend be_ws  if is_ws
    use_backend be_api if is_api
    default_backend be_web

backend be_api
    balance leastconn
    option httpchk
    http-check send meth GET uri /healthz ver HTTP/1.1 hdr Host api.internal
    http-check expect status 200
    # Backend rules run after the frontend's and know the chosen backend.
    http-request set-header X-Backend be_api
    default-server check inter 3s fastinter 1s rise 2 fall 3 maxconn 200 slowstart 30s
    server api1 10.0.1.11:8080
    server api2 10.0.1.12:8080
    server api3 10.0.1.13:8080

backend be_ws
    balance leastconn
    timeout tunnel 2h
    server ws1 10.0.2.11:8080 check
    server ws2 10.0.2.12:8080 check

backend be_web
    balance roundrobin
    cookie SRV insert indirect nocache httponly secure
    option httpchk GET /healthz
    default-server check inter 2s maxconn 500
    server web1 10.0.3.11:8080 cookie w1
    server web2 10.0.3.12:8080 cookie w2

Two details in that config are deliberate. timeout tunnel is set globally to 1h and raised to 2h only in the WebSocket backend, because timeout server would otherwise cut an idle upgraded connection at 30s. And option redispatch with retries 2 lets a request that failed to connect be retried on a different server; it does not retry a request whose response had already started, which is the correct default for non-idempotent traffic.

Draining a server without a reload#

Reloading HAProxy is graceful but leaves the old process alive until its connections finish, which makes reload-per-deployment a poor tool for taking one server out of rotation. The runtime API on the stats socket does it instantly and statefully.

# Inspect
echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18,19
echo "show servers state" | socat stdio /run/haproxy/admin.sock

# Stop new load-balanced traffic, keep existing and sticky sessions
echo "set server be_api/api2 state drain" | socat stdio /run/haproxy/admin.sock

# Watch it empty out (scur column)
echo "show stat" | socat stdio /run/haproxy/admin.sock | grep be_api

# Full maintenance: also drops stickiness and stops checks counting
echo "set server be_api/api2 state maint" | socat stdio /run/haproxy/admin.sock

# Kill anything still attached, then return to service
echo "shutdown sessions server be_api/api2" | socat stdio /run/haproxy/admin.sock
echo "set server be_api/api2 state ready"   | socat stdio /run/haproxy/admin.sock

The drain and maint distinction is the useful one: drain removes the server from the load balancing algorithm but continues to serve requests that carry its persistence cookie, so a session-bound user finishes their work; maint removes it entirely and stops health checks from being reported. For a rolling deploy you want drain, a wait, then maint.

set weight be_api/api2 10% gives partial drains for canaries. Runtime state is lost on restart unless you dump it with show servers state into the file referenced by server-state-file and enable load-server-state-from-file global, which is how you keep a drained server drained across a reload.

Version notes#

Claims here that are version dependent:

• The composable http-check send / http-check expect syntax arrived in 2.2. The older single-line option httpchk <method> <uri> <version> still parses.
• The legacy reqrep, reqadd, rspadd header keywords were deprecated in 1.9 in favour of http-request and http-response, and removed in later 2.x releases. Configurations copied from pre-1.9 material will fail to parse.
• balance random was added in 1.9.
• Master-worker mode with the master CLI (-W -S) is the supported way to run HAProxy under a supervisor from 1.8 onward, and expose-fd listeners on the stats socket enables socket handover for seamless reloads.
• HAProxy 2.8 and 3.0 are LTS branches. 3.0 introduced the crt-store section for managing certificates independently of bind lines.

For how HAProxy's design choices compare with the alternatives, see nginx vs HAProxy vs Envoy vs Caddy vs Traefik.

Failure modes#

Frequently asked questions#

What is the difference between a frontend, a backend and a listen section?#

A frontend binds listening sockets and decides which backend handles a request; a backend holds the servers and the load balancing policy. listen combines both in one section, which is convenient when there is exactly one backend behind one bind, such as a stats page or a simple TCP relay.

In what order does HAProxy evaluate rules?#

Category first, then declaration order within a category. Frontend tcp-request rules run first, then frontend http-request, then use_backend, then the backend's tcp-request and http-request rules. Response rules run in reverse: the backend's http-response before the frontend's.

What are HAProxy's default timeout values?#

timeout connect, timeout client and timeout server have no defaults and HAProxy warns at startup when they are missing from defaults. timeout http-keep-alive falls back to timeout http-request, which in turn falls back to timeout client. Health check interval inter defaults to 2000ms, with rise 2 and fall 3.

How do I take a server out of rotation without reloading HAProxy?#

Send set server <backend>/<server> state drain to the runtime API socket. The server stops receiving new load-balanced traffic but still serves requests carrying its persistence cookie. Follow with state maint once the connection count reaches zero, and state ready to return it to service.

When should I use leastconn instead of roundrobin?#

Use leastconn when request durations vary widely, for example a mix of fast API calls and long uploads, or for WebSockets and database protocols where connections are long-lived. Use roundrobin when requests are short and uniform, because it distributes more evenly and costs less to compute.

Why is HAProxy returning 503 when the servers are up?#

The three common causes are all visible in the termination flags: sC means the connection to the server timed out, sQ means the request waited past timeout queue for a maxconn slot, and a log line naming the backend with no server available means every server failed its health check. Check the health check's request path and Host header first.

Does mode tcp let me set headers?#

No. In mode tcp HAProxy never parses HTTP, so http-request and http-response rules are not available and no header can be added, removed or inspected. Routing decisions must use connection-level fetches such as req.ssl_sni, which requires a tcp-request inspect-delay.

How does option forwardfor differ from set-header X-Forwarded-For?#

option forwardfor appends the connection source address to any existing X-Forwarded-For, preserving upstream entries, and can be limited with except and if-none. http-request set-header X-Forwarded-For %[src] deletes everything that was there and writes a single value, which is the correct behaviour when the peer is untrusted.