Topic cluster
Troubleshooting proxies
Read 502, 503 and 504 correctly, drive curl through a proxy, understand no_proxy, survive a corporate proxy, and bisect a chain methodically.
Most proxy debugging goes wrong in the first minute, because the investigation starts at the layer that reported the error rather than the layer that caused it. A proxy reports what it experienced, which is usually a symptom of something one hop further in.
The method that works is always the same: draw the chain, bisect it, and find the innermost hop that still shows the fault. A proxy debugging playbook is the long form of that method, and everything else in this cluster is a specific case of it.
By symptom#
| Symptom | Start here |
|---|---|
| 502, 503 or 504 | 502 vs 503 vs 504 |
| Intermittent 502 under load | Keep-alive and upstream connection pooling |
| 504 with no upstream log line | Timeout budgets |
499 in the nginx access log, ALB 460, Envoy DC | What nginx 499 means, and its HAProxy, Envoy and ALB names |
connect() failed (111) to [::1] with proxy_pass http://localhost | Why nginx tries ::1 when proxy_pass says localhost |
SSL_do_handshake() failed ... while SSL handshaking to upstream, or upstream SSL certificate verify error | Why nginx fails the TLS handshake to an HTTPS upstream |
connect() to 127.0.0.1:3000 failed (13: Permission denied) while connecting to upstream on RHEL, Rocky, Alma or Fedora | Why nginx gets 13: Permission denied connecting to upstream (SELinux) |
connect() failed (111) to 127.0.0.1, or host not found in upstream "host.docker.internal", from nginx in a container | Why nginx in Docker returns 502 for proxy_pass localhost |
502 Bad Gateway from Nginx Proxy Manager, or could not be resolved (3: Host not found) in /data/logs/proxy-host-*_error.log | Why Nginx Proxy Manager shows 502 Bad Gateway for a proxy host |
Caddy 502 with dial tcp ... connect: connection refused, no such host, i/o timeout or tls: first record does not look like a TLS handshake | Why Caddy reverse_proxy returns 502 with dial tcp: connection refused |
Envoy or Istio 503 with upstream connect error or disconnect/reset before headers. reset reason: connection termination, or flag UC | What reset reason: connection termination means in Envoy and Istio |
Apache AH01102: error reading status line from remote server, AH00898: Error reading from remote server, or 502 Proxy Error now and then | Why Apache logs AH01102: error reading status line from remote server |
HAProxy 503 No server is available to handle this request, <NOSRV> in the log, or backend 'X' has no server available! | Why HAProxy returns 503 No server is available to handle this request |
HAProxy Server X is DOWN, reason: Layer4 connection problem, Layer4 timeout, Layer6 invalid response or Layer7 wrong status | HAProxy health check failed: what each Layer4, Layer6 and Layer7 reason means |
| 502 after restarting a Docker container | nginx 502 after a Docker container restart |
worker_connections are not enough, or Too many open files | What worker_connections should be on a reverse proxy |
Access-Control-Allow-Origin contains multiple values | Why the CORS header appears twice behind a proxy |
Large downloads stop early: ERR_CONTENT_LENGTH_MISMATCH, curl error 18 | Why nginx cuts off large downloads |
upstream sent no valid HTTP/1.0 header, status 009, or Received HTTP/0.9 when not allowed | Why nginx logs no valid HTTP/1.0 header |
| A POST arrives as a GET, or with an empty body, after a 301 or 302 redirect | Why a POST becomes a GET after a redirect behind nginx |
400 The plain HTTP request was sent to HTTPS port, or error_page 497 not redirecting | Why nginx says the plain HTTP request was sent to HTTPS port |
A custom header such as X_Api_Key never reaches the backend, or client sent invalid header line | Why nginx drops request headers with underscores |
| Uploads through Traefik fail at exactly 60 seconds with 499, 502 or 504 | Why large uploads through Traefik fail after 60 seconds |
413 Request Entity Too Large, or client intended to send too large body after raising client_max_body_size | Why nginx returns 413 Request Entity Too Large |
X-Cache-Status: MISS on every request, or an empty $upstream_cache_status | Why is nginx proxy_cache always MISS? |
upstream sent too big header | Header and body size limits |
| Wrong client IP in the logs | Configuring trusted proxies |
| Infinite redirect loop over HTTPS | X-Real-IP, X-Forwarded-Proto, Host and Port |
| 404 only for one path prefix | nginx proxy_pass and the trailing slash |
| Some tools bypass the proxy, others do not | The no_proxy environment variable |
unable to get local issuer certificate | TLS interception and corporate root CAs |
| npm, pip, git or docker failing at work | Corporate proxies and developer tooling |
Squid logs TCP_DENIED/403 for every request, or X-Squid-Error: ERR_ACCESS_DENIED 0 | Why does Squid log TCP_DENIED/403 for every request? |
ERR_TUNNEL_CONNECTION_FAILED, ERR_PROXY_CONNECTION_FAILED or ERR_PROXY_AUTH_UNSUPPORTED in Chrome or Edge | What ERR_TUNNEL_CONNECTION_FAILED means, and what the proxy answered |
wrong version number or Your proxy appears to only use HTTP through a proxy | Should HTTPS_PROXY start with http:// or https://? |
407 from every dev tool behind an NTLM or Kerberos proxy | Cntlm or Px: which local auth proxy to use |
| Windows proxy set in Settings, a service or tool still goes direct | netsh winhttp vs Windows proxy settings |
The one tool to learn properly#
curl through a proxy is worth reading even if you already use curl daily. The --write-out timing breakdown alone will localise most latency problems to a single hop in one command.
Every guide in this cluster#
- 502 vs 503 vs 504: reading proxy errors correctlyWhat the proxy actually experienced behind 502, 503 and 504, with nginx error_log strings, HAProxy termination flags and Envoy response flags decoded.
- Apache AH01102 error reading status line: fix the 502AH01102 means mod_proxy_http got no status line back. Usually a pooled keep-alive connection the backend closed. The log lines, the ttl fix and the 502 timeout case.
- Caddy 502 dial tcp connection refused: each causeCaddy logs the Go dial error word for word, and each wording has its own cause. Tested on Caddy 2.11.7: refused, no such host, i/o timeout, EOF, TLS, unix sockets.
- ERR_TUNNEL_CONNECTION_FAILED: what the proxy answeredThe proxy answered CONNECT with something other than 200, and Chrome hid it. Each proxy answer mapped to the Chrome error it produces, tested on Chromium 142.
- Cntlm vs Px: which one for an NTLM or Kerberos proxy?Px for Windows single sign-on with no stored password, Cntlm for a 0.3 MB C daemon. Which builds really have Kerberos, and how to wire tools to 3128.
- Corporate proxies and developer toolingHow to point curl, git, npm, pip, Java, Go, Docker, apt, Maven and VS Code at a corporate proxy and its root CA, with the gotcha for each tool.
- Access-Control-Allow-Origin multiple values: the fixTwo layers both add the CORS header, so the browser sees two values and blocks the response. How to prove it with curl and pick one owner in nginx, Traefik or Caddy.
- curl through a proxyEvery curl proxy flag that matters, how to read -v output by phase, and a --write-out timing string that shows whether the proxy or the origin is slow.
- Envoy reset reason: connection termination, every causeEnvoy's 503 body names why the upstream failed. Tested on Envoy 1.36.13: each reset reason, its flag and details, and the idle timeout pairs behind UC.
- HAProxy 503 No server is available: read the log lineHAProxy wrote that 503 itself and never reached a server. Tested on HAProxy 3.2.26: the log line for each cause, from all servers down to a Host port.
- HAProxy health check failed: every reason decodedThe reason in an HAProxy health check line is a check status. Tested on HAProxy 3.2.26: the exact line for L4CON to L7TOUT, its usual causes and the one fix.
- Header and body size limits at the proxyWhy requests die at the proxy before the app sees them: nginx buffer directives, 400/413/431/494/502, the upstream-sent-too-big-header fix, and a defaults table.
- HTTPS_PROXY: http:// or https://? Tested in 12 clientsAlmost always http://. The scheme says how to reach the proxy, not what it carries. What 12 clients do with each form, the exact errors, and how to check.
- netsh winhttp vs Windows proxy settings: who reads which?Settings, netsh winhttp or environment variables: which proxy browsers, services, Windows Update, PowerShell, .NET, curl, pip and Go read on Windows.
- nginx 413 Request Entity Too Large: find the limitThe body is over client_max_body_size (1m by default) in the location that serves it. Why raising it often fails, and which layer really sent your 413.
- nginx 499: what it means, and its ALB, HAProxy, Envoy names499 means the client hung up before nginx sent a response header. The ALB 460, HAProxy CH and Envoy DC equivalents, and how to tell which of four causes you have.
- nginx 502 after Docker restart: fix the stale IPnginx resolves upstream names once at load, so a recreated container on a new IP gets 502. The 1.27.3+ resolve fix, the variable fallback and what each costs.
- nginx in Docker: 502 on proxy_pass localhost, fixedInside a container, localhost is the container itself. What to put in proxy_pass for a backend on the host or in another container, and the bind address each needs.
- nginx drops headers with underscores: why, and the fixnginx discards X_Api_Key before proxying, and so do Caddy 2.11.4+, Gunicorn and Werkzeug. Why, where underscores_in_headers must go, and when enabling it is unsafe.
- nginx 400 plain HTTP request sent to HTTPS port: fixA client spoke plain HTTP to a listen ... ssl port. Where it comes from, the error_page 497 line that fixes it, and the three ways that line silently fails.
- nginx proxy_cache always MISS: every cause, testedEvery request shows X-Cache-Status MISS. The response headers, config and request details that stop nginx storing it, each tested, and how to find yours in one look.
- Nginx Proxy Manager 502 Bad Gateway: find your causeEach NPM proxy host field mapped to the nginx line it generates and the error log line a wrong value leaves. Tested on NPM 2.16.0: localhost, ports, networks, scheme.
- nginx connect() failed to [::1] with localhost: the fixlocalhost becomes two addresses, ::1 and 127.0.0.1, and a backend on only one of them fails on the other. Which builds see it, the 10 second lockout it adds, and the fix.
- nginx (13: Permission denied) to upstream: SELinux fixSELinux blocks nginx from every upstream port by default on RHEL, Rocky, Alma and Fedora. Which ports each boolean opens, when semanage helps, Unix sockets.
- nginx SSL handshaking to upstream errors: fix each oneEach upstream TLS error nginx logs, from alert 112 to verify error 20, mapped to its cause and fix. Includes the Let's Encrypt chain that breaks proxy_ssl_verify_depth 1.
- nginx cuts off large downloads: find which of 5 causesA proxied download that stops early is logged by nginx as a 200. The five causes, the error log line that names each one, and the fix that matches it.
- nginx upstream sent no valid HTTP/1.0 header: the fixThe backend's first bytes were not an HTTP status line, so nginx passed them on as HTTP/0.9 and logged 009, not 502. What causes it, what does not, and how to find it.
- nginx worker_connections: the right number for a proxyEvery proxied request holds two connections in one worker, idle upstream pools hold more, and the open-files limit caps it all. The sizing formula, worked through.
- The no_proxy environment variableHow curl, Go, Python requests, urllib, wget and Java each interpret no_proxy, the portable subset that works everywhere, and the suffix-matching trap.
- POST becomes GET after a redirect: proxies and clientsA 301, 302 or 303 lets the client resend a POST as a GET with no body. Which code each proxy sends by default, and what 11 clients do with each, tested.
- A proxy debugging playbookA repeatable method for proxy faults: draw the topology, bisect the chain, correlate with a request ID, read timers in order, reproduce, then capture packets.
- Squid TCP_DENIED/403 on every request: the http_access fixA fresh Squid allows only localhost: allow localnet ships commented out. The rule order, each distro's default, and how to see which line denied you.
- Traefik upload fails after 60 seconds: the readTimeout fixThe entryPoint readTimeout (60s since v2.11.2) bounds the whole request body. Which entryPoint needs it, what HTTP/1.1, HTTP/2 and HTTP/3 clients get, tested.