gRPC through a reverse proxy

gRPC is not a new protocol at the proxy layer: it is HTTP/2 with a fixed set of conventions, and a reverse proxy either speaks HTTP/2 end to end and forwards trailers, or it cannot carry gRPC at all. That single constraint explains almost every gRPC proxy failure. The second constraint is load balancing: because gRPC multiplexes many calls over one long-lived connection, an L4 balancer distributes connections rather than calls and pins all of a client's traffic to one backend.

What gRPC requires on the wire#

A unary gRPC call is one HTTP/2 stream. The request headers are ordinary HTTP/2 pseudo-headers plus a small fixed set:

:method = POST
:scheme = https
:path = /helloworld.Greeter/SayHello
:authority = api.example.com
content-type = application/grpc+proto
te = trailers
grpc-timeout = 5S
grpc-encoding = identity

The body is a sequence of length-prefixed messages. Each message is exactly 5 bytes of prefix followed by the payload: one byte of compressed flag, then a 4 byte big-endian length.

00                  compressed flag: 0 = not compressed
00 00 00 07         message length: 7 bytes
0a 05 77 6f 72 6c 64 ...   protobuf payload

The response is where proxies fall over. The server sends response headers (:status 200, content-type: application/grpc), then the message frames, then HTTP/2 trailers carrying the actual outcome:

grpc-status = 0
grpc-message =

:status is 200 even for failures. An RPC that fails with NOT_FOUND still returns HTTP 200, and the error lives in the grpc-status trailer as the integer 5. If the RPC fails before any messages, the server may send a Trailers-Only response: a single HEADERS frame with END_STREAM containing :status, content-type and grpc-status together.

Why trailers and buffering break naive proxies#

Two design assumptions in older proxies are fatal here.

Trailers. HTTP/1.1 supports trailers only with chunked transfer encoding, and in practice almost nothing implements them. A proxy that downgrades HTTP/2 to HTTP/1.1 upstream, which is what nginx's proxy_pass does unless you set proxy_http_version 2 (available since 1.29.4), has nowhere to put grpc-status. The client then sees the stream end with no status, which the gRPC runtime reports as code = Internal desc = server closed the stream without sending trailers. grpc_pass remains the supported way to carry gRPC through nginx, because it implements gRPC framing and trailers rather than merely speaking HTTP/2.

Buffering. A proxy that buffers the whole response before forwarding it converts a server-streaming RPC into a single delivery at the end. The RPC still succeeds, so nothing errors; the client just receives every message at once after the stream closes, and any latency-sensitive streaming design silently stops working. For bidirectional streaming, request buffering is worse: the server never receives the first client message until the client half-closes, so the two sides deadlock and the call hangs until a timeout. The same mechanics are covered generally in proxy buffering and streaming responses.

Configuration by proxy#

nginx#

nginx has proxied gRPC since 1.13.10 through a dedicated module. Use grpc_pass, never proxy_pass.

server {
    listen 443 ssl;
    http2 on;                     # nginx 1.25.1+ directive form

    location /helloworld.Greeter/ {
        grpc_pass grpc://grpc_backend;      # grpcs:// for TLS to the upstream
        grpc_read_timeout  3600s;
        grpc_send_timeout  3600s;
        grpc_socket_keepalive on;
    }
}

upstream grpc_backend {
    server 10.0.1.10:50051;
    server 10.0.1.11:50051;
}

grpc_read_timeout and grpc_send_timeout both default to 60s, which is the same value as proxy_read_timeout and for the same reason. A long-lived server-streaming or bidirectional RPC that is quiet for 60 seconds is killed by the proxy, and the client reports code = Unavailable. Location matching is on the fully qualified method path, so location /helloworld.Greeter/ routes one service and location / routes everything.

Envoy#

Envoy is the natural fit because it was designed around HTTP/2 and it understands gRPC semantics rather than just carrying the bytes. A cluster needs explicit HTTP/2 upstream options:

clusters:
- name: grpc_backend
  type: STRICT_DNS
  lb_policy: ROUND_ROBIN
  typed_extension_protocol_options:
    envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
      "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
      explicit_http_config:
        http2_protocol_options: {}
  load_assignment:
    cluster_name: grpc_backend
    endpoints:
    - lb_endpoints:
      - endpoint: { address: { socket_address: { address: grpc.svc, port_value: 50051 }}}

Two filters are worth knowing. envoy.filters.http.grpc_web translates gRPC-Web (which browsers can speak over HTTP/1.1, with trailers encoded into the body) into real gRPC for the backend, which is how you serve a browser client without changing the service. envoy.filters.http.grpc_json_transcoder accepts REST/JSON requests and transcodes them to gRPC using the compiled proto descriptor set, giving you a JSON API and a gRPC API from one service definition. Neither exists in nginx. See Envoy listeners, routes and clusters for how the filter chain fits together.

HAProxy#

HAProxy 2.0 and later speak HTTP/2 on both sides in mode http. The only gRPC-specific part is telling HAProxy the backend is h2.

frontend fe_grpc
    bind :443 ssl crt /etc/ssl/api.pem alpn h2,http/1.1
    mode http
    default_backend be_grpc

backend be_grpc
    mode http
    timeout server 1h
    balance roundrobin
    server g1 10.0.1.10:50051 proto h2 check
    server g2 10.0.1.11:50051 proto h2 check

alpn h2,http/1.1 on the bind line lets the client negotiate HTTP/2 over TLS. On the server line, proto h2 selects cleartext h2c to the backend; for TLS to the backend use ssl alpn h2 instead. Omit these and HAProxy downgrades to HTTP/1.1 upstream, and you are back to the trailers problem. More on the surrounding config in HAProxy configuration for HTTP reverse proxying.

Traefik#

Traefik selects the upstream protocol from the service scheme. For a cleartext gRPC backend, set the scheme to h2c:

labels:
  - "traefik.http.services.grpc-svc.loadbalancer.server.scheme=h2c"
  - "traefik.http.services.grpc-svc.loadbalancer.server.port=50051"

Without h2c Traefik dials the backend with HTTP/1.1 and the RPC fails immediately. With TLS to the backend, use https and ensure the backend negotiates h2 via ALPN.

Caddy#

Caddy needs the h2c transport declared explicitly, because h2c cannot be negotiated (there is no ALPN on a cleartext connection):

grpc.example.com {
    reverse_proxy 10.0.1.10:50051 {
        transport http {
            versions h2c 2
        }
    }
}

Proxy support matrix#

The last row is the important one, and it includes a plain Kubernetes ClusterIP Service, which balances at the connection level via kube-proxy.

Load balancing is the real problem#

In HTTP/1.1 a client opens a connection per concurrent request, so balancing connections happens to balance requests. gRPC breaks that equivalence. A gRPC channel opens one HTTP/2 connection to the resolved address and multiplexes every subsequent RPC over it, keeping it alive indefinitely. An L4 balancer sees exactly one connection, sends it to one backend, and every RPC that client ever makes lands there. Add ten backends and traffic stays on the one the connection happened to hit.

The observable symptom is characteristic: one pod at 90% CPU, nine idle, and a load balancer dashboard showing perfectly even connection counts. Scaling up does nothing because new pods receive no traffic until clients reconnect.

There are exactly two fixes.

1. L7 proxy balancing per stream. Terminate HTTP/2 at a proxy that understands streams (every row above except the last) and let it distribute individual RPCs across backends. Cost: an extra hop and an extra HTTP/2 termination.
2. Client-side load balancing. The gRPC client resolves all backend addresses (a headless Service in Kubernetes, or DNS returning multiple A records), opens a subchannel to each, and applies a policy such as round_robin or xDS-driven policies. Cost: clients must be able to reach every backend, and every language runtime needs configuring.

Either way, add MAX_CONNECTION_AGE on the server (with MAX_CONNECTION_AGE_GRACE). It makes the server send a GOAWAY after a bounded lifetime so clients re-resolve and reconnect, which is what lets newly added backends ever receive traffic. Without it, a connection established before a scale-up survives forever and the imbalance is permanent. This is the same class of issue as connection reuse in keep-alive and upstream connection pooling.

Failure modes#

rpc error: code = Unavailable desc = ... (status 14) is the transport-level catch-all: connection refused, TLS failure, GOAWAY, proxy idle timeout, or a proxy that could not establish HTTP/2 upstream. It tells you the RPC never reached application code. Check, in order: is the upstream actually speaking h2c or h2, did an idle timeout fire (grpc_read_timeout 60s in nginx), and is a middlebox in the path terminating HTTP/2.

code = Internal desc = server closed the stream without sending trailers. The proxy delivered the response body but not the grpc-status trailer. Root cause is almost always an HTTP/1.1 hop: proxy_pass instead of grpc_pass, a Traefik service without h2c, or a HAProxy server line without proto h2.

nginx error log lines of the form upstream sent ... http2 ... at error level mean nginx's gRPC module got something it could not parse as HTTP/2 from the upstream. The usual cause is grpc_pass pointing at a plain HTTP/1.1 server or at a TLS listener without grpcs://.

Envoy upstream connect error or disconnect/reset before headers. reset reason: protocol error. The cluster is not configured for HTTP/2, so Envoy dialled HTTP/1.1 into an HTTP/2-only backend. Add http2_protocol_options to the cluster.

HTTP 464. AWS Application Load Balancer documents this status for a protocol version mismatch between the listener and the target group, for example an HTTP/1.1 request arriving for a gRPC target group, or a gRPC request to a target group configured for HTTP/1. If you see a 464 rather than a gRPC status, the request died at the load balancer before gRPC semantics applied.

GOAWAY with ENHANCE_YOUR_CALM and debug data too_many_pings. The client is sending HTTP/2 PING frames more often than the server permits. gRPC servers enforce a minimum interval between client pings (the default enforcement policy is 5 minutes, and pings without active streams are not permitted by default). A client configured with a 10 second keepalive against a default server gets disconnected repeatedly. Fix by raising the client keepalive interval or lowering the server's MinTime and setting PermitWithoutStream on both sides consistently. Note that proxies in the path have their own HTTP/2 flood protections, so a client keepalive that the origin tolerates may still trip an intermediary.

code = ResourceExhausted desc = grpc: received message larger than max. The gRPC default maximum receive message size is 4 MiB. This is a library default, not a proxy default, but it appears after a proxy change because compression settings differed. Check whether the proxy stripped or altered grpc-encoding.

Deadline exceeded on long streams that used to work. Compare grpc-timeout from the client with the proxy's read timeout. Whichever is shorter wins, and the proxy will not report a gRPC status when it is the one that gave up. Put both on the same ladder as described in timeout budgets across a proxy chain.

Frequently asked questions#

Why does nginx proxy_pass not work for gRPC?#

proxy_pass speaks HTTP/1.x to the upstream by default, and HTTP/1.1 has no practical way to carry the HTTP/2 trailers that hold grpc-status. Use grpc_pass (nginx 1.13.10 and later), which keeps HTTP/2 end to end and understands gRPC framing.

Why is all my gRPC traffic hitting one backend?#

Because gRPC multiplexes every call over a single long-lived HTTP/2 connection, and an L4 load balancer distributes connections rather than calls. Use an L7 proxy that balances per stream, or client-side load balancing with a round_robin policy, and set MAX_CONNECTION_AGE on the server so connections are periodically rebalanced.

What does grpc-status 14 UNAVAILABLE mean?#

It means the RPC failed at the transport level and never reached the service implementation. Common causes are a proxy that could not establish HTTP/2 to the upstream, a proxy idle timeout closing a quiet stream, connection refused, or a GOAWAY from the server.

Do I need h2c or TLS for gRPC through a proxy?#

Either works, but the choice must be explicit on every hop. h2c is cleartext HTTP/2 and cannot be negotiated by ALPN, so proxies need to be told (proto h2 in HAProxy, h2c scheme in Traefik, versions h2c 2 in Caddy). Over TLS, ALPN must advertise h2.

What is gRPC-Web and do I need it?#

gRPC-Web is a variant that browsers can speak, because browser JavaScript cannot control HTTP/2 frames or read trailers. It encodes trailers into the response body and works over HTTP/1.1. You need a translating proxy such as Envoy's grpc_web filter between the browser and a normal gRPC service.

Why do I get ENHANCE_YOUR_CALM too_many_pings?#

The client is sending HTTP/2 keepalive pings more frequently than the server's enforcement policy allows, so the server sends a GOAWAY with error code ENHANCE_YOUR_CALM. Align the client keepalive interval with the server's minimum ping interval, and enable PermitWithoutStream on both sides if you need pings on idle connections.

Can I put a WAF or CDN in front of gRPC?#

Only if it terminates HTTP/2, forwards trailers and does not buffer bodies, which excludes many edge products. gRPC bodies are binary protobuf, so signature-based inspection produces false positives. The equivalent constraint for the other long-lived protocol is discussed in WebSockets through a reverse proxy.

How do I see gRPC errors in proxy logs?#

You have to log the grpc-status trailer, because :status is 200 even for failed RPCs. Envoy's grpc_stats filter exposes per-method status counters; other proxies generally require the application or a sidecar to report gRPC status codes.