Why does nginx cut off large downloads with ERR_CONTENT_LENGTH_MISMATCH or curl error 18?

A large download that stops partway through nginx, with ERR_CONTENT_LENGTH_MISMATCH in Chrome or curl: (18) transfer closed with N bytes remaining to read, means nginx sent a 200 status line and then could not deliver the body. There are five causes, and the nginx error log names the one you have: the temp directory is not writable (13: Permission denied), the temp disk is full (28: No space left on device), the upstream stopped sending (upstream prematurely closed connection or upstream timed out ... while reading upstream), or the client stopped reading (client timed out ... while sending to client, which is only logged at info). The access log is no help: every one of these is recorded as a 200.

The common advice splits two ways, "fix the permissions on proxy_temp" and "turn off proxy_buffering". Both are right for some causes and useless for the others, which is why the advice keeps contradicting itself. Match the log line first:

The "while sending to client" and "while reading upstream" suffixes are the part to search for, because the same upstream timed out text with while reading response header from upstream is a 504, which is a different problem covered in 502 vs 503 vs 504.

proxy_buffering off (or proxy_max_temp_file_size 0) fixes only the first two rows. Its cost is the part the circulating answers leave out: nginx then reads the upstream only as fast as the client drains, so it holds the upstream connection for as long as the slowest client takes, which on an application server with a fixed number of workers means one worker tied up per slow download. If that upstream also gives up on slow writes, the temp file cap stall described in cause 3 now starts at the first byte instead of at 1 GB.

Why nginx logs a 200 for a broken download#

The status line and headers leave nginx as soon as it has read the upstream's response header. The body follows. When the body fails partway, the 200 is already on the wire and HTTP has no way to take it back, so nginx closes the connection and records the status it sent. The client knows only because the byte count is wrong. RFC 9112 is explicit: "A message that uses a valid Content-Length is incomplete if the size of the message body received (in octets) is less than the value given by Content-Length", and a chunked body "is incomplete if the zero-sized chunk that terminates the encoding has not been received". A client "MUST record the message as incomplete".

That gives the two client-side shapes, and they are the two errors people search for:

libcurl documents error 18, CURLE_PARTIAL_FILE, as the case where "the server first reports an expected transfer size, and then delivers data that does not match". It is the same event as the Chrome error, seen from a different client.

How the cases were reproduced#

Every observation on this page comes from nginx 1.27.5 and 1.30.5 in the official Alpine images, proxying to a small Python upstream that could send a fixed Content-Length or chunked body, stop at a chosen byte, pause, or give up on a slow write. The client was curl 7.68.0, rate-limited with --limit-rate where the case needed a slow reader. Every case run on both versions behaved the same; the temp file cap case and the HTTP/2 run were on 1.30.5 only, and the $request_completion values are from 1.27.5. The access log format was:

log_format trunc 'rc=$request_completion cl=$sent_http_content_length '
                 'ubr=$upstream_bytes_received $request_uri status=$status '
                 'sent=$body_bytes_sent ust=$upstream_status '
                 'urt=$upstream_response_time rt=$request_time';

The numbers below are from loopback, where socket buffers are large. On a real network fewer bytes are in flight when nginx closes the connection, so the cut-off point will be earlier, but the log lines are the same.

Cause 1: the temp directory is not writable#

Its signature is distinctive: small files download, large files to slow clients stop at roughly the same place every time, and the same download over a fast link succeeds.

With proxy_buffering on (the default), nginx reads the upstream as fast as it can and holds what the client has not taken yet. Memory comes first, proxy_buffers 8 4k|8k per request, so 32 or 64 KB. What does not fit is written to a temporary file under proxy_temp_path, up to proxy_max_temp_file_size (1024m by default). A client that keeps up never needs the file. A slow client with a large response always does, and that is the first moment the worker tries to create a file in the temp directory. Proxy buffering and streaming responses covers the buffer arithmetic in full.

When the create fails:

[crit] 27#27: *3 open() "/var/cache/nginx/locked/2/0000000002" failed (13: Permission denied) while reading upstream, client: 127.0.0.1, request: "GET /perm/?size=10485760 HTTP/1.1"
rc= cl=10485760 ubr=2658304 /perm/?size=10485760 status=200 sent=2621336 ust=200 urt=0.003 rt=0.003

A 200, 2,621,336 bytes of a 10,485,760-byte body, in 3 milliseconds. Everything the client received was already in the kernel's socket buffers when nginx gave up.

Why the directory is wrong when nginx "fixes" it at startup. At startup, nginx's ngx_create_paths() creates each temp directory if it is missing, and if the worker user does not own it, runs chown and adds owner read, write and execute. It does this for the configured path only. The temp files go into level subdirectories below it (the official image's default proxy_temp uses two levels; a spill file landed at /var/cache/nginx/proxy_temp/4/00/0000000004 in the test), and those are created later by whichever worker first needs them, owned by that worker's user. The startup fix never touches them. In the lab, a proxy_temp_path ... 1 directory owned by the right user but containing subdirectories owned by root with mode 700 reproduced the failure on every request large enough to spill, and a restart did not repair it.

That points at the real-world triggers:

• The user directive changed, for example when moving to the official image, which runs workers as nginx (its build sets --user=nginx), while a temp tree created under the old user stayed on disk.
• A volume or host path was reused across images whose nginx user has a different UID.
• The master is not root. nginx sets a worker user only when the master runs as root (ngx_core_module_init_conf checks geteuid() == 0, and the user directive is otherwise "ignored" with a warning). With no worker user, the ownership fix is skipped completely, so in a rootless or user: 1000 container the whole temp tree must already be writable by that UID.

The fix is to make the tree belong to the worker user, all of it:

nginx -V 2>&1 | tr ' ' '\n' | grep temp-path      # compiled-in defaults
nginx -T 2>/dev/null | grep -E 'proxy_temp_path|^user'   # overrides
find /var/cache/nginx -not -user nginx -ls        # anything the worker cannot own
chown -R nginx: /var/cache/nginx/proxy_temp

The alternative is to stop writing to disk at all, with proxy_max_temp_file_size 0. With the same unwritable directory and that one line, the 10 MB download completed and nginx held the upstream connection for 2.8 seconds instead of 3 milliseconds while the client drained it. That is the trade: no disk, but a slow client now occupies an upstream connection and whatever worker or thread sits behind it.

Cause 2: the temp disk is full#

Same mechanism, different errno. With proxy_temp_path on a 2 MB tmpfs:

[crit] 27#27: *5 pwritev() "/mnt/tiny/0000000003" failed (28: No space left on device) while reading upstream
rc= cl=20971520 ubr=4689920 /full/?size=20971520 status=200 sent=2555800 ust=200 urt=0.005 rt=0.005

When the filesystem was already full before the request started, the result was identical: a 200 status line and a truncated body, not a 502, because the headers had been sent before the first temp write.

The options are a dedicated volume for proxy_temp_path sized for concurrent slow downloads, a smaller proxy_max_temp_file_size so that no one response can take the whole disk, or 0 to stop spilling.

Cause 3: the temp file cap stalls the upstream until it gives up#

This is the "downloads stop at about 1 GB" report. When a spill file reaches proxy_max_temp_file_size, nginx stops reading from the upstream and reads only as fast as the client drains. Maxim Dounin explained the consequence on the nginx mailing list: reaching the limit "causes relative long delay in reading from upstream server; this may cause timeout on upstream server, seen as 'truncation'." nginx is not timing out. The upstream is, because its writes stop making progress.

The lab version used proxy_max_temp_file_size 1m, an upstream that abandoned any single 64 KB write that took over one second, and a client reading at 30 KB/s:

upstream: upstream write timeout at 11927552
[error] 27#27: *23 upstream prematurely closed connection while reading upstream
/max/?size=20971520&wt=1 status=200 sent=11927552 ust=200 urt=315.126 rt=315.125
curl: (18) transfer closed with 9043968 bytes remaining to read

The control, the same request with the default 1024m cap, delivered all 20,971,520 bytes: nginx spooled the whole response to disk in 27 milliseconds and released the upstream, and the client took ten minutes to finish. So the size at which downloads break is the cap plus whatever the buffers and sockets hold, and the time at which they break is set by the upstream's write timeout, not by anything in nginx.

The error log line is the same one a crashing upstream produces, which is why this cause is misdiagnosed. Two details separate them. A crash or a restart is not tied to the response size; this happens at nearly the same size, and only for slow clients. And $upstream_response_time is long, here 315 seconds, because nginx was still connected to the upstream while it waited for the client.

Fixes, in order of preference: raise the upstream's own write or send timeout, whatever the application server calls it, above the longest stall a slow client can cause; raise proxy_max_temp_file_size above the largest response on that location so the spill absorbs it; or serve large files without proxying the bytes, with X-Accel-Redirect or a direct object storage URL.

Cause 4: the upstream stops sending#

Two variants, both visible in the error log with while reading upstream.

The upstream closes the connection. The application crashed, was restarted mid-response, hit its own timeout, or a pooled connection was cut by something between nginx and the upstream:

[error] 27#27: *9 upstream prematurely closed connection while reading upstream
rc=OK cl=10485760 ubr=3000104 /a/?size=10485760&at=3000000 status=200 sent=3000000 ust=200 urt=0.003 rt=0.005

With proxy_buffering off the result was identical: same error line, same 200, same 3,000,000 bytes. Turning buffering off does not help here, because the bytes were never sent to nginx. An application restarted while a long download is in progress produces exactly this line; keep-alive and upstream connection pooling covers the related case of a pooled connection closing at the wrong moment.

The upstream pauses for longer than proxy_read_timeout. The timeout applies "only between two successive read operations, not for the transmission of the whole response", so a slow but steady response never trips it. A pause does. An upstream that sent 3,000,000 bytes and then paused for six seconds, against proxy_read_timeout 3s:

[error] 27#27: *7 upstream timed out (110: Operation timed out) while reading upstream
rc=OK cl=10485760 ubr=3000104 /rt/... status=200 sent=2998168 ust=200 urt=3.005 rt=3.005

An upstream that pauses mid-body, for example while it generates the next part of an export, trips it. Either make the upstream send something steadily, or raise proxy_read_timeout for that location only, keeping the order described in timeout budgets across a proxy chain so that the hop in front of nginx still waits longer than nginx does.

Cause 5: the client is too slow for send_timeout#

send_timeout (60s default) is, like the read timeout, a gap: "If the client does not receive anything within this time, the connection is closed." A client that stops reading for a minute, a laptop going to sleep, a phone losing signal, a stalled mobile network, is disconnected with the transfer incomplete. With send_timeout 3s and a client draining slowly:

[info] 27#27: *15 client timed out (110: Operation timed out) while sending to client
rc= cl=10485760 ubr=10485864 /slow/?size=10485760 status=200 sent=5207835 ust=200 urt=0.006 rt=3.385

That line is at info. The default error_log level is error, and the official image's nginx.conf sets notice, so with either configuration this cause leaves no error log line at all, only an access log entry where sent is less than cl. Note also that $request_time is when nginx gave up, not when the client stopped receiving: in the 1.30.5 run nginx closed the connection at 4.5 seconds, and curl went on reading data already sitting in socket buffers until 118 seconds.

This is the one cause that is usually not a server fault: the client, or the network in front of it, stopped taking data. The client that disconnects on purpose before the header is different again, and is logged as 499; what nginx 499 means covers it, including the point that a client leaving after the header is a 200 with a short $body_bytes_sent, which is the same signature as everything on this page.

Finding truncations in the access log#

Because the status is 200, a truncation has to be found by comparing byte counts. The tempting variable is $request_completion, documented as "OK if a request has completed, or an empty string otherwise". It does not do this job:

Two rules come out of that table. The comparison that catches every case where the upstream set a Content-Length is $body_bytes_sent against $sent_http_content_length. And $upstream_bytes_received separates the slow-client case from the others: a full count with a short sent means the problem was on the client side of nginx.

A one-line check over a log in the trunc format above:

awk '{for(i=1;i<=NF;i++){split($i,kv,"="); f[kv[1]]=kv[2]}
      if (f["cl"] ~ /^[0-9]+$/ && f["sent"] < f["cl"]) print}' access.log

HEAD requests and 206 range responses can make this misfire, since Content-Length there is not the full object, so filter on $request_method and status if your traffic has many of either. Chunked responses carry no length to compare against; for those, the error log is the only record.

Failure modes#

Raising proxy_buffers "fixed" it, briefly. Larger memory buffers only raise the size at which a response spills to disk. Downloads under the new threshold now succeed and larger ones still fail, which makes the fault look intermittent. Look for the crit line instead.

Downloads break only behind the CDN or load balancer. Then nginx may not be the hop cutting them off. A truncation at an outer hop looks the same to the client. Test the origin directly with curl through a proxy techniques (--resolve to pin the origin, -w '%{size_download}' to count bytes), and if the origin is clean, bisect the chain as described in the proxy debugging playbook.

A cached location still truncates with proxy_max_temp_file_size 0. The nginx documentation notes that the restriction "does not apply to responses that will be cached or stored on disk", so a proxy_cache location still writes to disk and still needs a writable cache path.

Where this answer stops applying#

The behaviour above was observed on nginx 1.27.5 and 1.30.5 with HTTP/1.1 clients, plus one HTTP/2 run on 1.30.5; browsers were not tested, so the Chrome error names come from Chromium's error definitions, not from observation, and the HTTP/2 browser error for a reset stream was not checked. The byte counts depend on loopback socket buffer sizes and will differ on a real network. The cause 3 reproduction used a deliberately small cap and an upstream with a one-second write timeout so that the event could be reproduced in minutes; the mechanism is the one Maxim Dounin described, but how long your application server tolerates a stalled write is its own setting, not nginx's. Nothing here applies to uploads, where a request body that stops early is a different event with different log lines.

Frequently asked questions#

Why does nginx log 200 when the download was incomplete?#

Because the 200 status line had already been sent when the body failed. HTTP cannot change a status after it is sent, so nginx closes the connection and logs the status it sent. Compare $body_bytes_sent with the Content-Length to find these.

What does "open() proxy_temp failed (13: Permission denied) while reading upstream" mean?#

The nginx worker could not create a temporary file for a response that did not fit in its memory buffers. The usual cause is level subdirectories under the temp path owned by another user. chown -R the temp directory to the worker user, or set proxy_max_temp_file_size 0 to avoid the disk.

Does proxy_buffering off fix ERR_CONTENT_LENGTH_MISMATCH?#

Only when the error log shows a temp file failure: permission denied or a full disk. It does not help when the upstream closed the connection or timed out, and it makes slow clients hold upstream connections, which can turn an upstream write timeout into more truncations, not fewer.

Why do nginx downloads stop at around 1 GB?#

The temp file cap, proxy_max_temp_file_size, defaults to 1024m. When a slow client lets the spill file reach it, nginx stops reading from the upstream, and the upstream's own write timeout can then close the connection. Raise the upstream's timeout or the cap.

What does curl error 18 "transfer closed with bytes remaining to read" mean?#

The server announced a body length, sent fewer bytes, and closed the connection. libcurl calls it CURLE_PARTIAL_FILE. Behind nginx, the error log line for the same request names the cause.

Is $request_completion a reliable way to detect truncated responses?#

No. In testing it was OK for responses cut short by the upstream, both a premature close and a read timeout. It was empty for temp file failures and client timeouts. Comparing $body_bytes_sent with $sent_http_content_length catches all of them when a length was sent.