Topic cluster
Troubleshooting proxies
Read 502, 503 and 504 correctly, drive curl through a proxy, understand no_proxy, survive a corporate proxy, and bisect a chain methodically.
Most proxy debugging goes wrong in the first minute, because the investigation starts at the layer that reported the error rather than the layer that caused it. A proxy reports what it experienced, which is usually a symptom of something one hop further in.
The method that works is always the same: draw the chain, bisect it, and find the innermost hop that still shows the fault. A proxy debugging playbook is the long form of that method, and everything else in this cluster is a specific case of it.
By symptom#
| Symptom | Start here |
|---|---|
| 502, 503 or 504 | 502 vs 503 vs 504 |
| Intermittent 502 under load | Keep-alive and upstream connection pooling |
| 504 with no upstream log line | Timeout budgets |
499 in the nginx access log, ALB 460, Envoy DC | What nginx 499 means, and its HAProxy, Envoy and ALB names |
connect() failed (111) to [::1] with proxy_pass http://localhost | Why nginx tries ::1 when proxy_pass says localhost |
SSL_do_handshake() failed ... while SSL handshaking to upstream, or upstream SSL certificate verify error | Why nginx fails the TLS handshake to an HTTPS upstream |
| 502 after restarting a Docker container | nginx 502 after a Docker container restart |
worker_connections are not enough, or Too many open files | What worker_connections should be on a reverse proxy |
Access-Control-Allow-Origin contains multiple values | Why the CORS header appears twice behind a proxy |
Large downloads stop early: ERR_CONTENT_LENGTH_MISMATCH, curl error 18 | Why nginx cuts off large downloads |
upstream sent no valid HTTP/1.0 header, status 009, or Received HTTP/0.9 when not allowed | Why nginx logs no valid HTTP/1.0 header |
upstream sent too big header | Header and body size limits |
| Wrong client IP in the logs | Configuring trusted proxies |
| Infinite redirect loop over HTTPS | X-Real-IP, X-Forwarded-Proto, Host and Port |
| 404 only for one path prefix | nginx proxy_pass and the trailing slash |
| Some tools bypass the proxy, others do not | The no_proxy environment variable |
unable to get local issuer certificate | TLS interception and corporate root CAs |
| npm, pip, git or docker failing at work | Corporate proxies and developer tooling |
wrong version number or Your proxy appears to only use HTTP through a proxy | Should HTTPS_PROXY start with http:// or https://? |
407 from every dev tool behind an NTLM or Kerberos proxy | Cntlm or Px: which local auth proxy to use |
| Windows proxy set in Settings, a service or tool still goes direct | netsh winhttp vs Windows proxy settings |
The one tool to learn properly#
curl through a proxy is worth reading even if you already use curl daily. The --write-out timing breakdown alone will localise most latency problems to a single hop in one command.
Every guide in this cluster#
- 502 vs 503 vs 504: reading proxy errors correctlyWhat the proxy actually experienced behind 502, 503 and 504, with nginx error_log strings, HAProxy termination flags and Envoy response flags decoded.
- Cntlm vs Px: which one for an NTLM or Kerberos proxy?Px for Windows single sign-on with no stored password, Cntlm for a 0.3 MB C daemon. Which builds really have Kerberos, and how to wire tools to 3128.
- Corporate proxies and developer toolingHow to point curl, git, npm, pip, Java, Go, Docker, apt, Maven and VS Code at a corporate proxy and its root CA, with the gotcha for each tool.
- Access-Control-Allow-Origin multiple values: the fixTwo layers both add the CORS header, so the browser sees two values and blocks the response. How to prove it with curl and pick one owner in nginx, Traefik or Caddy.
- curl through a proxyEvery curl proxy flag that matters, how to read -v output by phase, and a --write-out timing string that shows whether the proxy or the origin is slow.
- Header and body size limits at the proxyWhy requests die at the proxy before the app sees them: nginx buffer directives, 400/413/431/494/502, the upstream-sent-too-big-header fix, and a defaults table.
- HTTPS_PROXY: http:// or https://? Tested in 12 clientsAlmost always http://. The scheme says how to reach the proxy, not what it carries. What 12 clients do with each form, the exact errors, and how to check.
- netsh winhttp vs Windows proxy settings: who reads which?Settings, netsh winhttp or environment variables: which proxy browsers, services, Windows Update, PowerShell, .NET, curl, pip and Go read on Windows.
- nginx 499: what it means, and its ALB, HAProxy, Envoy names499 means the client hung up before nginx sent a response header. The ALB 460, HAProxy CH and Envoy DC equivalents, and how to tell which of four causes you have.
- nginx 502 after Docker restart: fix the stale IPnginx resolves upstream names once at load, so a recreated container on a new IP gets 502. The 1.27.3+ resolve fix, the variable fallback and what each costs.
- nginx connect() failed to [::1] with localhost: the fixlocalhost becomes two addresses, ::1 and 127.0.0.1, and a backend on only one of them fails on the other. Which builds see it, the 10 second lockout it adds, and the fix.
- nginx SSL handshaking to upstream errors: fix each oneEach upstream TLS error nginx logs, from alert 112 to verify error 20, mapped to its cause and fix. Includes the Let's Encrypt chain that breaks proxy_ssl_verify_depth 1.
- nginx cuts off large downloads: find which of 5 causesA proxied download that stops early is logged by nginx as a 200. The five causes, the error log line that names each one, and the fix that matches it.
- nginx upstream sent no valid HTTP/1.0 header: the fixThe backend's first bytes were not an HTTP status line, so nginx passed them on as HTTP/0.9 and logged 009, not 502. What causes it, what does not, and how to find it.
- nginx worker_connections: the right number for a proxyEvery proxied request holds two connections in one worker, idle upstream pools hold more, and the open-files limit caps it all. The sizing formula, worked through.
- The no_proxy environment variableHow curl, Go, Python requests, urllib, wget and Java each interpret no_proxy, the portable subset that works everywhere, and the suffix-matching trap.
- A proxy debugging playbookA repeatable method for proxy faults: draw the topology, bisect the chain, correlate with a request ID, read timers in order, reproduce, then capture packets.