HTTP/2 and HTTP/3 through proxies

HTTP version is negotiated independently on every hop. A browser can speak HTTP/3 to your edge, the edge HTTP/2 to an internal proxy, and that proxy HTTP/1.1 to the application, all in one request. That is not a misconfiguration, it is the normal deployment: nginx terminates HTTP/2 from clients while forwarding HTTP/1.1 upstream through proxy_pass. Almost every surprising HTTP/2 bug in a proxy chain is a translation artefact at one of those version boundaries rather than a fault in HTTP/2 itself. "We support HTTP/2" is therefore an ambiguous claim: it has to be qualified by leg, by transport, and by version of the proxy.

Support matrix by leg#

h2c here means HTTP/2 over cleartext TCP by prior knowledge.

The column that decides most architectures is upstream h2. It used to be the column where nginx was the odd one out, and on any build older than 1.29.4 it still is: grpc_pass is the only path that emits HTTP/2 upstream, so end-to-end HTTP/2 for anything else means HAProxy or Envoy at that hop. The wider trade-offs are set out in the reverse proxy comparison.

How the version is actually chosen#

Over TLS, by ALPN. RFC 7301 adds an application_layer_protocol_negotiation extension to the ClientHello carrying an ordered list of identifiers, of which h2, http/1.1 and h3 matter here. The server chooses from the client's list and echoes one value back. There is no in-band fallback afterwards: if the server selects h2, the first bytes the client sends must be the HTTP/2 connection preface.

Over cleartext, by prior knowledge. RFC 9113 removed the Upgrade: h2c handshake that RFC 7540 defined, so the only interoperable cleartext method is for the client to assume the server speaks HTTP/2 and send the preface immediately:

50 52 49 20 2a 20 48 54 54 50 2f 32 2e 30 0d 0a   PRI * HTTP/2.0..
0d 0a 53 4d 0d 0a 0d 0a                           ..SM....

Those 24 octets deliberately look like a malformed HTTP/1.1 request so an HTTP/1.1-only server rejects the connection rather than misparsing it. Envoy's codec_type: AUTO uses exactly this signature to decide whether an inbound cleartext connection is h2c or HTTP/1.1, which is why an Envoy sidecar accepts both on one port and nginx (whose listener is configured one way) does not.

For HTTP/3, by advertisement. A client does not try QUIC first. It connects over TCP and the origin advertises an alternative with Alt-Svc: h3=":443"; ma=86400 (RFC 7838), or the client learns it from an HTTPS DNS resource record carrying alpn=h3. Later requests race QUIC against TCP, and if QUIC wins the client remembers the alternative for ma seconds.

Connection coalescing and why routing gets weird#

RFC 9113 permits a client to reuse an existing HTTP/2 connection for a different origin when the new origin resolves to an address the connection already reaches and the server's certificate is authoritative for that host. Browsers do this aggressively. One wildcard or multi-SAN certificate covering app.example.com, api.example.com and static.example.com behind a single address means the browser opens one connection and sends all three hosts' requests over it, distinguished only by the :authority pseudo-header.

The failure is subtle because it never reproduces with curl, which does not coalesce across hostnames by default. Symptoms: requests for api.example.com served by the app.example.com backend, but only from browsers and only sometimes; access logs showing a Host that does not match the SNI recorded at TLS time; an intermittent 421 Misdirected Request for a request that succeeds on reload.

The root cause is a layer mismatch. The connection was routed once at TLS time by SNI, and the requests on it are addressed per stream by :authority. Any component that decides routing from SNI and then forwards the whole connection is wrong the moment coalescing happens, which is the specific hazard in SNI-based routing designs that steer TCP without terminating it. Fixes, in order of preference: route on :authority at the HTTP layer; return 421 Misdirected Request (RFC 9110, section 15.5.20) when the authority is not one this connection can serve, which tells the client to retry on a fresh connection; or prevent coalescing by splitting certificates or addresses, which is blunt and costs connection reuse.

The downgrade hazards: HTTP/2 to HTTP/1.1#

A proxy accepting HTTP/2 and forwarding HTTP/1.1 reconstructs a text protocol from a binary one. HTTP/2 is stricter, so the translation must add validation rather than merely re-encode.

Field names are lowercase. RFC 9113 section 8.2.1 requires lowercase field names on the wire and treats a message containing an uppercase name as malformed. Application code that compares header names case-sensitively breaks when a deployment moves to HTTP/2 on the client leg, and the downgrade preserves whatever case the proxy emits, so X-Forwarded-For on one path can arrive as x-forwarded-for on another.

Pseudo-headers replace the request line. :method, :scheme, :authority and :path carry what used to be the start line, must precede all regular fields, and must not appear in trailers. An intermediary converting to HTTP/1.1 generates Host from :authority, falling back to a Host field if :authority is absent. If both are present and disagree, the message is malformed and must be rejected, not reconciled.

Connection-specific fields are illegal. Connection, Keep-Alive, Proxy-Connection, Transfer-Encoding and Upgrade must not appear in HTTP/2, and TE is permitted only with the exact value trailers. So the HTTP/1.1 Upgrade: websocket handshake does not exist in HTTP/2 (extended CONNECT replaces it), and a proxy that copies fields verbatim into an HTTP/1.1 request can smuggle framing directives upstream.

H2.TE and H2.CL smuggling. If the front end accepts a content-length or transfer-encoding field inside an HTTP/2 request and passes it through, the upstream HTTP/1.1 parser frames the body using that field rather than the byte count the front end actually forwarded, and the two ends disagree about where the request ends. HTTP/2 frames bodies with DATA and END_STREAM, so the only correct behaviour is to regenerate framing headers from the real length and reject inbound transfer-encoding. See HTTP request smuggling and proxy desync.

Zero-length field names. HPACK's wire format can encode a header with an empty name. RFC 9113's malformed-message rules require rejecting such a message, but a proxy that omits the check and downgrades it emits an HTTP/1.1 line beginning with a colon, which downstream parsers interpret inconsistently. Validate that every field name is non-empty and contains only permitted characters before serialising to HTTP/1.1.

HTTP/3 and QUIC at the proxy#

HTTP/3 (RFC 9114) runs over QUIC (RFC 9000), which runs over UDP. That single fact changes proxy operations more than the HTTP semantics do, which are essentially unchanged from HTTP/2.

• You are now running a UDP service. Sockets, receive buffers, connection tracking and DDoS posture all change. QUIC stacks frequently need net.core.rmem_max and net.core.wmem_max raised, and an undersized receive buffer shows up as packet loss under load rather than as an error.
• Connections are identified by connection ID, not by the 4-tuple. A client that changes network or gets a new NAT binding keeps the same QUIC connection by presenting the same connection ID from a new address. An L4 balancer that hashes source IP and port sends those packets to a backend with no state for the connection, which can only drop them or send a stateless reset; the client sees a stall and then a fresh handshake. Correct behaviour requires parsing the QUIC header and routing on the destination connection ID, with the server encoding routing information into the IDs it issues (the approach the IETF QUIC working group's load balancer draft standardises). Many general purpose L4 balancers do not do this.
• The handshake needs a 1200-byte path. QUIC requires datagrams carrying Initial packets to be padded to at least 1200 bytes. A tunnel or overlay that lowers the effective MTU below that, with ICMP fragmentation-needed messages filtered, makes the handshake fail silently.
• CONNECT still exists. HTTP/3 carries CONNECT per stream, and extended CONNECT (RFC 9220) adds :protocol, which is how CONNECT-UDP and the wider MASQUE family proxy UDP and IP inside HTTP/3. These tunnels look like ordinary QUIC on UDP/443, so a policy that assumes tunnels are TCP CONNECT stops detecting them. See HTTP CONNECT tunnelling.

Worked example: HTTP/2 in, HTTP/1.1 out, gRPC excepted#

# nginx 1.25.1 or later
server {
    listen 443 ssl;
    listen 443 quic reuseport;   # nginx 1.25.0+ for HTTP/3
    http2 on;
    http3 on;

    ssl_certificate     /etc/ssl/example.pem;
    ssl_certificate_key /etc/ssl/example.key;

    # Advertise h3 only from a listener that actually serves it
    add_header Alt-Svc 'h3=":443"; ma=86400' always;

    location / {
        # This leg is HTTP/1.1 regardless of what the client used
        proxy_http_version 1.1;      # default since 1.29.7; explicit before that
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_pass http://app_pool;
    }

    location /grpc.Service/ {
        # The nginx path that emits HTTP/2 upstream on any version
        grpc_pass grpc://grpc_pool;
    }
}

proxy_http_version 1.1 is the default from nginx 1.29.7 onward; before 1.29.7 proxy_pass defaulted to HTTP/1.0, so the directive had to be set explicitly. Likewise proxy_set_header Connection "" removes the close nginx would otherwise send, and since 1.29.7 nginx no longer sends the Connection proxy header by default. On anything older, without both, the upstream keepalive pool is silently useless, as covered in keep-alive and upstream connection pooling. The grpc_pass location exists because gRPC mandates HTTP/2 end to end, covered in gRPC through a reverse proxy.

Observable behaviour: curl --http2 https://example.com/ reports HTTP/2 200 while the upstream access log records HTTP/1.1, and curl --http1.1 produces an identical upstream request. The upstream cannot tell which version the client used unless you tell it, for example with proxy_set_header X-Forwarded-Proto-Version $server_protocol;.

Failure modes#

421 Misdirected Request on a request that works on reload. Coalescing sent the request over a connection whose server is not authoritative for that authority. That is the correct outcome, and the client retries on a new connection. It becomes a bug when the proxy does not emit 421 and serves the wrong backend instead. Reproduce in a browser, or force it with curl --resolve pinning both hostnames to one address.

ERR_HTTP2_PROTOCOL_ERROR after a proxy change. Almost always an illegal field: an uppercase name, a Connection header, a Transfer-Encoding in HTTP/2, or a pseudo-header in trailers. Inspect the frame trace with nghttp -v or curl --http2 --trace.

Streams reset in bulk (rapid reset, CVE-2023-44487). A client opens a stream with HEADERS and immediately sends RST_STREAM. The stream stops counting against SETTINGS_MAX_CONCURRENT_STREAMS, but the proxy has usually already dispatched work upstream, so one connection generates unbounded backend requests. Defensively: keep SETTINGS_MAX_CONCURRENT_STREAMS modest (nginx's http2_max_concurrent_streams defaults to 128), count RST_STREAM frames per connection and close connections above a rate threshold, bound requests per connection, cap upstream concurrency independently, and patch. Fixes shipped across the ecosystem in October 2023, including Go 1.21.3 and 1.20.10.

HTTP/3 silently never used. The site advertises Alt-Svc: h3 but every request still uses TCP. Causes, in rough order: UDP/443 blocked by a firewall or corporate policy, an L4 balancer that does not forward UDP, a path MTU below 1200 bytes, or a listener without reuseport. Because the client races and falls back, no error appears anywhere; the only signal is an HTTP/3 request counter stuck at zero. Test with curl --http3-only and treat failure there as hard failure.

Stalled uploads or slow tunnels. HTTP/2 flow control is per stream and per connection, with SETTINGS_INITIAL_WINDOW_SIZE defaulting to 65,535 octets. A proxy that does not update windows aggressively throttles bulk transfer on an idle network, which looks like a slow backend and is not.

Client multiplexing turning into upstream queueing. Multiplexing exists only on the client leg. Sixty concurrent HTTP/2 streams become sixty HTTP/1.1 upstream requests needing sixty pooled connections; a smaller pool queues them. Size the pool against client stream concurrency, not request rate.

Frequently asked questions#

Does nginx support HTTP/2 to upstream servers?#

Since nginx 1.29.4, yes: ngx_http_proxy_module accepts proxy_http_version 2, which requires ngx_http_v2_module to be built in. On any earlier build the answer is no, because proxy_pass offers only HTTP/1.0 (the default before 1.29.7) and HTTP/1.1, and the single exception is grpc_pass, which uses HTTP/2 because gRPC requires it. If you are on an older nginx and need HTTP/2 to a backend for anything other than gRPC, put HAProxy (proto h2 on the server line) or Envoy at that hop.

Is end-to-end HTTP/2 worth it?#

Usually not. The client leg benefits from multiplexing because it is long, lossy and high latency; the upstream leg is typically a low latency network where an HTTP/1.1 keepalive pool performs comparably. End-to-end HTTP/2 matters when the protocol demands it (gRPC, trailers) or when backend connection count is itself the constraint.

What is h2c and when should I use it?#

h2c is HTTP/2 over cleartext TCP. Since RFC 9113 removed the Upgrade: h2c handshake it works by prior knowledge: the client sends the connection preface immediately and assumes the server understands it. Use it inside a trusted segment or a service mesh where TLS is handled separately, not on the public internet where there is no negotiation to fall back on.

Why does one hostname sometimes get served by another host's backend?#

Because the browser coalesced the connections. If two hostnames resolve to the same address and one certificate covers both, an HTTP/2 client may send both hosts' requests over one connection, so any routing decision taken from TLS SNI is wrong for half of them. Route on :authority, or answer 421 Misdirected Request.

Why is my HTTP/3 traffic falling back to TCP with no error?#

Because fallback is by design and is silent. The client races QUIC against TCP and uses whichever completes; a blocked UDP path, a path MTU under 1200 bytes, or a balancer that drops UDP simply means TCP always wins. Diagnose by forcing the protocol (curl --http3-only) rather than by looking for errors in normal traffic.

How do I stop an HTTP/2 rapid reset flood?#

Bound what a single connection can cost you: keep SETTINGS_MAX_CONCURRENT_STREAMS modest, track RST_STREAM frames per connection and terminate connections that exceed a rate threshold, limit total requests per connection, and make sure upstream concurrency is capped independently so a burst of resets cannot be amplified into backend load. Then apply the vendor patches released for CVE-2023-44487, since the accounting fix belongs in the HTTP/2 stack itself.

Can an L4 load balancer sit in front of HTTP/3?#

Only if it routes on the QUIC connection ID. QUIC connections survive address changes, so hashing source IP and port sends migrated packets to a backend with no state for the connection, which drops them or emits a stateless reset. Balancers without connection ID awareness are safe only where clients never migrate, which you cannot assume on mobile networks.