Why does Apache log "AH01102: error reading status line from remote server"?

Apache logs AH01102: error reading status line from remote server when its proxy module sent a request to the backend and got nothing back. There was no first line of a reply, not even a broken one. The usual cause is not a crashed app. It is the keep-alive race. Apache kept an idle connection to the backend in its pool. The backend closed it for being idle, at the very moment Apache reused it. That is why the error is rare, never repeats on reload, and clusters after quiet spells. The fix is to make Apache drop idle connections before the backend does, with ttl= on the ProxyPass line.

Apache's own manual describes this race. Under the proxy-initial-not-pooled variable it says the setting "avoids the 'proxy: error reading status line from remote server' error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend."

The same AH01102 line has a few variants. The lines logged next to it tell them apart. Each row below follows the httpd 2.4.x source of mod_proxy_http.c:

The rest of this page explains each row, then the fixes.

What the status line is, and what AH01102 says#

A status line is the first line of every HTTP/1.1 response, such as HTTP/1.1 200 OK. A proxy reads it before anything else. In Apache, the module that speaks HTTP to a backend is mod_proxy_http. The backend is the app server behind Apache: Gunicorn, Node, Tomcat, or another proxy.

AH01102 is that module saying it read that first line and found nothing. Here is the code path in mod_proxy_http.c on the 2.4.x branch, near line 1100:

1. Apache reads one line from the backend connection.
2. If the line is empty, it reads one more. The source comment says: "handle one potential stray CRLF".
3. If that read is also empty, or nothing came at all, it logs AH01102 at error level. The message names the backend: error reading status line from remote server app:8000.

So one blank line before the status line is tolerated. Two blank lines are not, and they give AH01102 too. That is rare, since a real server sends the status line first. nginx is stricter: it does not accept even one leading blank line, as shown in why nginx logs no valid HTTP/1.0 header. A reply that has bytes but is not a valid status line takes a different branch further down, with a different message. AH01102 is only for an empty read.

The text in brackets before AH01102 is the reason the read stopped. It comes from APR, the library Apache uses for sockets. For a timeout it is (70007)The timeout specified has expired. APR defines APR_TIMEUP as status base 70000 plus 7. For a closed or reset connection it is a different code. The one that matters most is 70007, because it splits the timeout case from the race.

The AH00898 line and the 502 Proxy Error page#

When Apache gives up on the request, it calls its shared error helper in proxy_util.c. That helper does three things:

• It logs AH00898: Error reading from remote server returned by /path at error level. The path is the client's request URI.
• It sets the status line to 502 Proxy Error. That is why the browser tab says "Proxy Error" and not "Bad Gateway".
• It adds the reason to the error page: "The proxy server could not handle the request. Reason: Error reading from remote server".

So a log with AH01102 then AH00898 on the same client is one event, not two. The first line says what failed at the socket. The second says what the client was told. Other proxies report the same event under other names, and 502 vs 503 vs 504 maps them.

Why it is intermittent: the keep-alive race#

Apache keeps connections to the backend open after a response and reuses them. That is connection pooling, and HTTP keep-alive is what lets a connection carry more than one request. It saves a TCP handshake per request, and a TLS handshake too on an HTTPS backend.

The backend has its own idle timer. When a connection sits idle long enough, the backend closes it. Apache does not know the backend's timer. By default it sets no limit of its own. The ttl parameter in the ProxyPass table has no default value.

Pooled connections cause a different error when Apache is the backend. A front proxy that reuses one TLS connection for two host names gets a 421, as in why Apache returns 421 behind a proxy. Here Apache is the proxy, and the problem is simpler. Apache does check a pooled connection before it reuses it. In proxy_util.c, a connection that the backend already closed is caught, logged at debug as AH00951: backend socket is disconnected, and replaced with a new one. No error reaches anyone. So most idle closes are harmless.

The failure needs exact timing. Apache checks the connection, finds it open, and sends the request. In that same instant the backend's idle timer fires and it closes the socket. The request arrives at a socket that is going away. Nothing is read back, so Apache logs AH01102. The manual's wording is precise: the backend closed the connection "after the connection check by the proxy and before data sent by the proxy reached the backend."

The source agrees. When the failed connection had served a request before, a comment in mod_proxy_http.c says Apache "might just have run into a keepalive race". It assumes the backend is fine and does not mark it as failed.

That explains the pattern people see:

• It is rare. The window is only the time a request takes to reach the backend.
• It comes after quiet spells. Busy connections never sit idle long enough to hit the backend's timer. A connection idle for just under that timer is the one at risk.
• A reload works. The retry goes out on a new connection.
• The backend logs nothing. It never read a request on that connection.

The same race in nginx, Caddy and HAProxy is covered in keep-alive and upstream connection pooling. The rule there holds here too: whoever closes an idle connection with nothing in flight causes no error, so the proxy must close first. Envoy logs the same event as a 503, explained in what reset reason: connection termination means.

Backend idle timeouts that Apache has to beat#

These are the documented defaults for common backends behind Apache. Apache's ttl needs to sit below the one you run:

A few notes on those rows:

• Gunicorn's docs say the sync worker "does not support persistent connections and will ignore this option". With that worker there is no idle connection to race. The threaded and async workers keep connections for 2 seconds.
• Node added keepAliveTimeoutBuffer in 24.6.0 and 22.19.0. Its docs say the buffer "helps reduce connection reset (ECONNRESET) errors". The real close then comes at 6 seconds. On older Node it comes at 5. The createServer() options list on the same page shows 65000 for keepAliveTimeout. The server.keepAliveTimeout entry itself says 5000, and that entry is what this table uses.
• Tomcat's keepAliveTimeout defaults to connectionTimeout. The docs give that a default of 60000 ms, "but note that the standard server.xml that ships with Tomcat sets this to 20000".

What Apache tells the client depends on the client's connection#

This part of the source surprises people. The 502 page is not always sent.

When the status line read fails and it was not a timeout, mod_proxy_http.c checks the client side. If the client's connection to Apache has already carried a request (its keepalive count is above zero), Apache logs AH01104 at debug and closes the client connection without any reply. The source comment says this is done "to trigger a retry by the client if allowed (as for idempotent requests)". A client that retries gets its answer on a new connection. Then only the error log shows that anything happened.

If this was the client's first request on its connection, Apache cannot do that. The comment says some browsers "only display an empty page in this case and do not do a retry". So Apache logs AH01105: NOT Closing connection to client, sends 502 Proxy Error, and logs AH00898.

This is the reason proxy-initial-not-pooled exists. It covers exactly the case where Apache has no quiet way out: the client's first request.

Fix 1: set ttl below the backend's idle timeout#

The ProxyPass manual describes ttl as the time to live for inactive connections. "Once reaching this limit, a connection will not be used again; it will be closed at some later time." The pooling section adds that ttl "can be used to avoid using a connection which is subject to closing because of the backend server's keep-alive timeout." That is this exact case.

# Node 24 behind Apache: Node closes idle connections at 5 s + 1 s
ProxyPass        "/" "http://app:3000/" ttl=4
ProxyPassReverse "/" "http://app:3000/"

# Stock Tomcat: keepAliveTimeout falls back to connectionTimeout 20000 ms
ProxyPass "/shop/" "http://tomcat:8080/shop/" ttl=15

# Uvicorn: 5 s
ProxyPass "/api/" "http://uvicorn:8000/api/" ttl=3

ttl is in whole seconds, so a 2 second Gunicorn timeout leaves little room. Raise Gunicorn's --keep-alive instead, for example to 10, and set ttl=5. Leave a margin of a second or more, since no timer is exact.

One limit is in the manual. With the prefork MPM, backend connections are not pooled. The acquire and ttl parameters "have no effect when using prefork". Check your MPM with apachectl -V. On prefork, use fix 2 or fix 3.

The other direction also works: raise the backend's idle timeout above Apache's ttl. On its own, without a ttl, there is no value high enough. Apache would hold the connection for as long as it is not reused.

Fix 2: SetEnv proxy-initial-not-pooled 1#

<Location "/">
    ProxyPass "http://app:3000/"
    SetEnv proxy-initial-not-pooled 1
</Location>

The manual says that with this set, "no pooled connection will be reused if the client request is the initial request on the frontend connection." In the source, the check applies to reverse proxy requests where the client connection has no prior request. It marks the backend connection to close, and the connect step then drops any pooled socket and opens a new one.

So the 502 case from the table can no longer happen on a reused connection. The silent-close case (AH01104) still can, and that one the client can retry. The manual names the price: "setting this variable downgrades performance, especially with HTTP/1.0 clients." Every new visitor connection costs a new backend connection.

SetEnv is fine here. The manual warns that SetEnv "is not evaluated early enough" for the no-proxy variable, but its own protocol example sets proxy-nokeepalive with SetEnv. These variables are read when the request is proxied, which is late enough.

Fix 3: disablereuse=On, the blunt fix#

ProxyPass "/" "http://app:3000/" disablereuse=On

The manual says disablereuse makes mod_proxy "immediately close a connection to the backend after being used". It names two cases where that helps: a firewall that "tends to silently drop connections", and backends "under round-robin DNS". With no pooled connections there is nothing to race. The cost is one new TCP connection per request, plus a TLS handshake if the backend is HTTPS. The proxy-nokeepalive variable does the same per location: it "forces the proxy to close the backend connection after each request."

Use this when the backend's idle timer is unknown or shorter than one second, or when something between Apache and the backend drops idle connections without telling either side.

Settings that do not fix the race#

keepalive=On sounds right and is not. It turns on TCP keep-alive probes. The manual says it is for "a firewall between your Apache httpd and the backend server, which tends to drop inactive connections". The probe interval "depends on global OS settings, and may be as high as 2 hours". It does nothing about an app that closes idle connections after a few seconds.

ping= sends Expect: 100-continue to the backend and waits for the reply. The manual describes it as a test of the connection "before forwarding the request". In the 2.4.x source, though, the ping only runs for reverse proxy requests that have a body. A GET never gets it. And when the ping times out, the result is a 503 with the reason Timeout on 100-Continue, not a fixed connection.

A longer timeout= only changes the read timeout case below. The race fails at once, long before any timeout fires.

The read timeout variant: AH01103 and a 502, not a 504#

If the AH01102 line starts with (70007)The timeout specified has expired, this is not the race. The backend accepted the request and did not send a status line within the socket timeout. With LogLevel at debug for proxy_http, the next line is AH01103: read timeout.

The timeout comes from the timeout= parameter on ProxyPass. Its default is ProxyTimeout, and ProxyTimeout defaults to the core TimeOut, which is 60 seconds.

Many people expect a 504 Gateway Timeout here. httpd 2.4's mod_proxy_http does not send one. On this path the source sets a proxy_timedout note and then returns HTTP_BAD_GATEWAY. The client gets 502 Proxy Error with the same "Error reading from remote server" reason as the race. The status cannot tell the two apart. Only the (70007) prefix and the delay can. The one exception is the ping case above, which returns 503.

Apache does send 504 on other paths. In proxy_util.c, a timeout while sending the request body returns HTTP_GATEWAY_TIME_OUT. mod_proxy_fcgi and mod_proxy_ajp have their own 504 returns. So a 504 from Apache points away from the status line read.

The fix for a real read timeout is the slow endpoint, or a longer timeout= for that path only. Inside a balancer, the failontimeout option reads the same proxy_timedout note and puts the worker into error state. Timeout budgets shows how to order the timers on every hop so the right one fires first.

How to tell which one you have#

Turn up logging for the proxy modules only. The core manual shows the per-module form, LogLevel info ssl:warn. The same works here:

LogLevel warn proxy:debug proxy_http:debug

Then read the lines around one AH01102:

• (70007) in front, AH01103: read timeout next, and the gap matches timeout=: a slow backend.
• No 70007, and the failure is instant: the backend closed or reset the connection.
• Rare, after quiet spells, never on a busy connection: the race. Fix 1.
• Every request fails, even the first on a fresh connection: the backend is not speaking HTTP on that port, or it closes on purpose. Check the backend's own log. That is not a race, and ttl will not help.

Switch the level back afterwards. At debug, mod_proxy logs lines for every proxied request.

Worked case: Gunicorn behind Apache, failing a few times a day#

Take an Apache 2.4 server with the event MPM, in front of a Flask app on Gunicorn. Gunicorn runs the gthread worker with its default --keep-alive of 2 seconds. The Apache config is one line, with no parameters:

ProxyPass "/api/" "http://app:8000/api/"

Suppose the error log shows AH01102 a few times a day, each followed by AH00898 for a different path. There is no (70007) prefix, so it is not a timeout. Users say a page failed once and worked on reload. Walk it through.

1. Apache sets no ttl, so a pooled connection to Gunicorn stays in the pool until it is reused or found closed.
2. Gunicorn closes any connection idle for 2 seconds.
3. Most idle connections sit longer than 2 seconds. Apache's pre-use check finds them closed and opens a new one. No error.
4. Now suppose a request comes 1.999 seconds after the last one on a connection. Apache's check passes. The request leaves. Gunicorn's 2 second timer fires before it arrives. Nothing comes back, and AH01102 is logged.
5. If the visitor's browser had already made a request on its connection, Apache closes it quietly, and the browser can retry. If this was the browser's first request, the visitor gets 502 Proxy Error.

The fix follows from step 2. Raise Gunicorn's keep-alive and set Apache's ttl well below it:

# gunicorn --worker-class gthread --keep-alive 10 ...
ProxyPass "/api/" "http://app:8000/api/" ttl=5

Now Apache stops reusing any connection idle for 5 seconds. Gunicorn would not close it for 10. The window in step 4 no longer exists. If the Gunicorn command line cannot change, disablereuse=On removes the race at the cost of a new connection per request.

Where this answer stops applying#

This page reads the httpd 2.4.x source and the 2.4 manual. Line numbers move between releases, so search for the APLOGNO numbers rather than a line. HTTP/2 backends through mod_proxy_http2, AJP and FastCGI backends use other modules and other messages. The balancer adds retries and worker error states not covered here. An HTTPS backend can fail in the TLS handshake first, with its own messages. If every request fails, or the failures follow a deploy or a crash, the backend is closing on purpose or dying, and no ttl fixes that. A firewall or NAT that drops idle connections silently is a different timer. The manual's answer for that is disablereuse or keepalive=On. The defaults in the backend table are the documented ones. Check your own config, since a platform image or a framework can set its own.

Frequently asked questions#

What does AH01102 error reading status line from remote server mean?#

Apache's mod_proxy_http sent the request to the backend and read back no status line, the first line of a response. It tolerates one stray blank line, then gives up. The usual cause is a pooled connection that the backend closed at the moment Apache reused it.

Why do I get 502 Proxy Error: Error reading from remote server only sometimes?#

That pattern is the keep-alive race. Apache reused an idle connection just as the backend's idle timer closed it. It is rare because the window is tiny, and it clusters after quiet spells. Set ttl= on the ProxyPass line below the backend's keep-alive timeout.

What is the AH00898 line after AH01102?#

AH00898: Error reading from remote server returned by /path is Apache's shared proxy error helper reporting what it told the client. It names the request path and sets the 502 Proxy Error status. It follows AH01102 for the same request.

Does Apache return 504 when the backend times out?#

Not for this error. When mod_proxy_http times out reading the status line, httpd 2.4 logs (70007)The timeout specified has expired with AH01102, then returns 502. Apache returns 504 on other paths, such as a timeout while sending the request body.

What does SetEnv proxy-initial-not-pooled 1 do?#

It stops Apache from reusing a pooled backend connection for a client's first request on its connection. That is the one case where a failed reuse becomes a 502 page. The manual warns that it lowers performance, since each new visitor connection opens a new backend connection.

Is disablereuse=On or ttl better for AH01102?#

ttl is better when you know the backend's idle timeout. It keeps pooling and only retires connections before the backend would close them. disablereuse=On opens a new connection for every request. Use it when the timeout is unknown, very short, or a firewall drops idle connections.