curl through a proxy

curl -x http://proxy.example.com:3128 https://api.example.com/v1/health sends a request through a proxy. The detail is where the debugging value sits: the scheme in front of the proxy host decides who resolves DNS, whether curl issues a CONNECT, and which TLS handshake -k applies to. Below is the flag reference plus the two recipes worth memorising, reading -v output by phase and using --write-out to say whether the proxy or the origin is slow.

If no port is given in the proxy string curl assumes 1080; if no scheme is given it assumes http://.

The proxy scheme decides the protocol and who resolves DNS#

The socks5:// versus socks5h:// split is the most common SOCKS misconfiguration: with socks5:// a hostname that only exists in the remote network fails with Could not resolve host even though the proxy is reachable and would have resolved it happily. The legacy spellings --socks5 and --socks5-hostname mean the same as socks5:// and socks5h://. Protocol trade-offs are in SOCKS5 versus HTTP proxy.

Authentication flags#

--proxy-anyauth is right when you do not know the scheme in advance and wrong in a script that runs at volume, because the mandatory 407 round trip doubles connection setup. Scheme mechanics are in proxy authentication.

-p is --proxytunnel, and it is not --proxy#

-p is the short form of --proxytunnel. It has nothing to do with setting a proxy or a port. It tells curl to reach the destination with CONNECT through the HTTP proxy named by -x, instead of asking the proxy to fetch the URL on its behalf.

You need it when the URL scheme is not https. For https:// URLs curl already issues CONNECT automatically, because it must run its own TLS handshake with the origin. For http:// URLs, curl by default sends an absolute-form request to the proxy:

GET http://example.com/path HTTP/1.1
Host: example.com

With -p, curl issues CONNECT example.com:80 and sends an ordinary origin-form request inside the tunnel. That matters when the proxy must stay out of the HTTP layer, when testing whether a proxy permits CONNECT to a non-443 port, or when running a non-HTTP URL scheme through an HTTP proxy. Tunnel mechanics are in HTTP CONNECT tunnelling.

TLS flags for HTTPS proxies#

When -x https://... is in use there are two TLS sessions: curl to proxy, and curl to origin inside the tunnel. Every ordinary TLS flag has a --proxy- twin that applies to the outer session only.

--preproxy [scheme://]host[:port] (curl 7.52.0 and later) chains a SOCKS proxy in front of the HTTP or HTTPS proxy given by -x. curl reaches the SOCKS proxy first, tunnels to the HTTP proxy through it, then talks HTTP to that proxy. Ordering is fixed: SOCKS is always the outer hop.

Bypassing the proxy: --noproxy#

--noproxy takes a comma-separated list of hosts that must not go through the proxy, and --noproxy "*" disables the proxy entirely for that invocation, which is the quickest way to test whether the proxy is implicated at all. Since curl 7.86.0 the list also accepts IP ranges in CIDR notation. curl's matching is label-aware, so example.com matches api.example.com but not badexample.com; that is not true of every implementation, which is the subject of the no_proxy environment variable and the no_proxy tester.

Environment variables, and the uppercase that curl ignores#

curl reads http_proxy, HTTPS_PROXY, ALL_PROXY and NO_PROXY (and their case variants) when no -x is given. One asymmetry is deliberate:

curl reads lowercase http_proxy but not uppercase HTTP_PROXY.

The reason is CGI. A CGI environment maps request headers into environment variables by upper-casing them and prefixing HTTP_, so a client sending a Proxy: evil.example.com header causes HTTP_PROXY=evil.example.com to appear in the CGI script's environment. Honouring the uppercase form would let a remote client redirect the server's own outbound HTTP through an attacker-controlled host. That is the httpoxy issue (CVE-2016-5385 and companions, 2016). HTTPS_PROXY is unaffected because there is no Proxys: header, so both cases are honoured there.

-x on the command line overrides all of them. --noproxy overrides no_proxy.

Reading -v output by phase#

Every proxy failure happens in one of three phases, and -v labels the boundaries. Annotated output for curl -v -x http://proxy.example.com:3128 https://api.example.com/health:

*   Trying 10.10.0.9:3128...                      <- phase 1: TCP to the PROXY
* Connected to proxy.example.com (10.10.0.9) port 3128
* allocate connect buffer
* Establish HTTP proxy tunnel to api.example.com:443
> CONNECT api.example.com:443 HTTP/1.1            <- phase 2: CONNECT to the proxy
> Host: api.example.com:443
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established             <- proxy reached the origin
* CONNECT phase completed
* TLSv1.3 (OUT), TLS handshake, Client hello (1): <- phase 3: TLS with the ORIGIN
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* Server certificate:
*  subject: CN=api.example.com
> GET /health HTTP/1.1                            <- phase 4: the actual request
> Host: api.example.com

Read it as a checklist:

• Failure before Connected to proxy is your network to the proxy. The proxy has not been involved yet, and the URL is irrelevant.
• Failure between CONNECT and the < HTTP/1.1 200 is the proxy's problem: policy denial (403), authentication (407), or a failure by the proxy to reach the origin (502, 504).
• Failure in the TLS block after CONNECT phase completed is between you and the origin, with the proxy acting as a pipe. A certificate error here is the origin's certificate, or an interceptor's substitute for it.
• Failure after GET /health is an ordinary HTTP problem that has nothing to do with proxying.

For a plain http:// URL through a proxy there is no CONNECT, and the tell is the request line: > GET http://example.com/path HTTP/1.1 in absolute form. If you see origin form (> GET /path) while -x is set, curl is tunnelling.

Localising latency with --write-out#

Save this as ~/.curl-format and use curl -w @~/.curl-format:

namelookup:  %{time_namelookup}s\n
connect:     %{time_connect}s\n
appconnect:  %{time_appconnect}s\n
pretransfer: %{time_pretransfer}s\n
starttransfer: %{time_starttransfer}s\n
total:       %{time_total}s\n
http_code:   %{http_code}   connect_code: %{http_connect}\n
remote_ip:   %{remote_ip}   num_connects: %{num_connects}\n

The variables are cumulative seconds from the start of the transfer, so the useful numbers are the differences:

The decision rule: a large time_appconnect - time_connect means the proxy's path to the origin or the TLS negotiation is slow; a large time_starttransfer - time_pretransfer means the proxy has done its job and the origin application is slow. Running the same command with --noproxy "*" from the proxy host brackets the fault with two measurements. %{http_connect} is separately valuable: it records the status the proxy returned to CONNECT even when the transfer later fails for another reason.

Worked examples#

Test an upstream the way the proxy would see it. From the proxy host, take the proxy out and measure the origin directly:

curl -sS -o /dev/null --noproxy '*' \
  -w 'code=%{http_code} conn=%{time_connect} ttfb=%{time_starttransfer}\n' \
  https://api.internal.example.com/v1/health

Decide whether the proxy or the origin is slow. Same URL, twice, with and without the proxy:

for p in "-x http://proxy.example.com:3128" "--noproxy *"; do
  curl -sS -o /dev/null $p \
    -w "$p ttfb=%{time_starttransfer} total=%{time_total}\n" \
    https://api.example.com/v1/health
done

A large delta with similar time_starttransfer values means queueing or buffering at the proxy. A large time_starttransfer in both means the origin.

Force a specific origin IP while keeping SNI and Host. To hit one backend of a pool without editing /etc/hosts:

curl -sv --resolve api.example.com:443:203.0.113.42 https://api.example.com/v1/health

--resolve overrides only the DNS answer, so the TLS SNI, the certificate check and the Host header all remain api.example.com. --connect-to api.example.com:443:backend-7.internal:443 does the same thing but takes a name rather than an address, and it likewise leaves SNI and Host untouched. Note the interaction with proxies: when -x is set, curl's TCP connection is to the proxy, so neither option changes where the origin connection lands. Use them from the proxy host with the proxy disabled, or in place of a proxy for origin-side testing.

Common curl proxy errors#

Exit code 97 is worth knowing on its own: it exists precisely so scripts can distinguish "the proxy failed" from "the origin failed", which exit code 56 cannot express.

Failure modes#

A proxy set in the environment that you forgot about. curl announces it at the top of -v with * Uses proxy env variable http_proxy == 'http://...'. Any unexplained routing behaviour starts there; --noproxy '*' neutralises it for one command.

socks5:// where socks5h:// was needed. Symptom: Could not resolve host: internal.example.com while the same name resolves fine on the proxy host.

Credentials in shell history and process lists. -U user:pass is visible in ps output. Use -U user and let curl prompt, or put the credentials in ~/.netrc with --netrc, which matches proxy hosts too.

Testing a CONNECT policy with an http:// URL. Without -p, an http:// URL is fetched by the proxy rather than tunnelled, so a CONNECT ACL is never exercised and the test proves nothing. More sequencing advice in a proxy debugging playbook.

Frequently asked questions#

How do I make curl use a proxy?#

Pass -x (or --proxy) with the proxy address, for example curl -x http://proxy.example.com:3128 https://example.com/. If no scheme is given curl assumes http://, and if no port is given it assumes 1080. Setting the http_proxy, https_proxy or ALL_PROXY environment variables has the same effect for all curl invocations in that shell.

What is the difference between socks5 and socks5h in curl?#

With socks5://, curl resolves the target hostname itself and sends an address to the proxy. With socks5h://, curl sends the hostname and the proxy resolves it. Use socks5h:// whenever the destination name only resolves inside the proxy's network, which is the usual case for SSH tunnels and bastion hosts.

Why does curl ignore HTTP_PROXY in uppercase?#

Because in a CGI environment, a client-supplied Proxy: request header becomes the HTTP_PROXY environment variable, so honouring it would let a remote client redirect the server's outbound requests. curl therefore reads only lowercase http_proxy. This is the httpoxy issue from 2016, and other clients including Go's net/http adopted equivalent mitigations.

How do I tell whether the proxy or the origin server is slow?#

Use --write-out and compare intervals. %{time_appconnect} minus %{time_connect} covers the proxy's connection to the origin plus TLS, while %{time_starttransfer} minus %{time_pretransfer} is the origin's time to first byte. A large second interval means the application is slow; a large first interval means the proxy's path to the origin is.

What does "Proxy CONNECT aborted" mean?#

It means the proxy closed the TCP connection before returning a complete response to the CONNECT request. It is not an HTTP status, so there is no code to look up. Common causes are an ACL that resets instead of replying, a TLS-inspecting appliance that failed to build a certificate, and an idle timeout shorter than the proxy's own connect time to the origin.

Does --resolve work when a proxy is configured?#

Not for the origin connection. With -x set, curl connects to the proxy and the proxy connects to the origin, so overriding the origin's DNS answer locally has no effect on where the request lands. --resolve and --connect-to are tools for testing origin behaviour without a proxy, typically run from the proxy host itself.

How do I send a header to the proxy but not to the origin?#

Use --proxy-header. On a tunnelled request it is placed on the CONNECT rather than on the request inside the tunnel, which is what proxy-specific routing or tracing headers need. Ordinary -H headers go to the origin.