Why does Apache return 503 with "AH01114: HTTP: failed to make connection to backend"?

Apache logs AH01114: HTTP: failed to make connection to backend when it could not open a TCP connection to the app behind it. It then answers the client with 503 Service Unavailable. The line just before AH01114 gives the reason, such as (111)Connection refused. Whether the 503 keeps coming after the app is back up depends on how the backend is set up. A plain ProxyPass to one address tries again on the next request. A BalancerMember is shut out for retry seconds, 60 by default, and the log says so with AH00959 ... disabling worker ... for 60s. Set retry=0 on the member, or add a second member, and the window goes away.

Here is what each setup did after one refused connect, then after the backend came back. Each row was run on httpd 2.4.69 (official Docker image, event MPM). The rows marked † were run again on 2.4.63, with the same result.

The sections below explain each row. The short version: if you see 503s for a full minute after a restart, look for balancer:// and forcerecovery=Off in the config.

What AH01114 means#

Apache's reverse proxy is built from a few modules. mod_proxy holds the shared logic. mod_proxy_http speaks HTTP to the backend, the app server behind Apache. Each backend address that Apache proxies to is a worker: one scheme://host:port, with its own pool of connections and its own settings. A ProxyPass line creates one worker. A BalancerMember line creates one worker inside a balancer.

Each request goes through a few steps in mod_proxy_http.c. Step two is to make the connection. If that step fails, the code near line 2054 on the 2.4.x branch logs AH01114 at error level and sets the status:

ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, APLOGNO(01114)
              "HTTP: failed to make connection to backend: %s",
              backend->hostname);
status = HTTP_SERVICE_UNAVAILABLE;

So AH01114 always means the TCP connect failed. Nothing was sent to the backend, and nothing was read back. The host name at the end of the line is the one in your config, not the address Apache tried.

Why it is 503 and not 502#

HTTP_SERVICE_UNAVAILABLE is 503. The client sees Apache's stock page: "Service Unavailable. The server is temporarily unable to service your request due to maintenance downtime or capacity problems."

This is unlike nginx and Caddy, which both send 502 when the backend refuses the connection. In Apache a 502 means something else. The connection was made, and the reply was missing or broken. The most common case is the keep-alive race in why Apache logs AH01102: error reading status line. So in Apache, the status alone tells you which step failed. 503 points at the connect. 502 points at the response. The full mapping across proxies is in 502 vs 503 vs 504.

The line before AH01114 says why#

AH01114 does not give a reason. The line just before it does. In proxy_util.c, each failed connect attempt logs AH00957. The text in brackets is the error from the operating system:

[proxy:error] (111)Connection refused: AH00957: http: attempt to connect to 127.0.0.1:8001 (127.0.0.1:8001) failed
[proxy_http:error] [client 127.0.0.1:55420] AH01114: HTTP: failed to make connection to backend: 127.0.0.1

The two you will see most often:

• (111)Connection refused. The address answered, but nothing listens on that port. The app is down, still starting, or bound to another address or port. This is the same refusal Caddy logs as dial tcp ... connect: connection refused. The usual Docker causes are the same as in why Caddy returns 502 with connection refused.
• (13)Permission denied. On RHEL, Rocky, Alma and Fedora with SELinux enforcing, the kernel blocked the connect before it left the host. Apache runs in the httpd_t domain. The policy source declares httpd_can_network_connect with a default of false, and describes it as "Allow HTTPD scripts and modules to connect to the network using TCP". nginx runs in the same domain, so the same booleans and port labels apply. Why nginx gets 13: Permission denied connecting to upstream lists which ports each boolean opens and why semanage port alone is not enough.

A 13 is not a network fault, so retrying will never fix it. A 111 fixes itself once the app is listening. Both go down the same code path after the bracket, and both can put the worker in error state, as the next section shows.

What AH00959 does: the error state and the retry timer#

When every address for a worker has failed, ap_proxy_connect_backend in proxy_util.c decides what to do with the worker. Unless the worker has the ignore-errors flag, it records the time and sets the worker's error state:

[proxy:error] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1:8004) for 60s

A worker in error state is skipped. Each time a request wants it, Apache checks whether retry seconds have passed since that failure. If not, it logs AH00933: too soon to retry worker at debug level and does not connect. Once the time is up, it logs AH00932: has been marked for retry (also debug) and tries again.

The timer starts at the failure that set the error state. Requests that arrive during the window do not restart it. With retry=10, a backend that came back one second after the failure got its first request about ten seconds after the failure.

The retry default is 60 seconds. It is PROXY_WORKER_DEFAULT_RETRY 60 in mod_proxy.h, and the manual's table says the same: "If the connection pool worker to the backend server is in the error state, Apache httpd will not forward any requests to that server until the timeout expires." It adds: "A value of 0 means always retry workers in an error state with no timeout."

Which workers get disabled: the 60-second claim, corrected#

Most write-ups of this error say Apache disables the backend for 60 seconds after any failed connect. That is true for balancer members. It has not been true for a plain ProxyPass worker since httpd 2.4.23.

In the 2.4.x source, the function that defines a worker ends like this:

if (!balancer) {
    wshared->status |= PROXY_WORKER_IGNORE_ERRORS;
}

So every worker that is not inside a balancer starts in ignore-errors mode. That is the same flag status=+I sets. The AH00959 branch checks for that flag and skips the error state. That is why the first row of the table logged AH01114 and a 503, but no AH00959, and the next request after the app came back got 200. The 2.4.20 source does not have these lines, and the 2.4.23 source does. Before that, a plain ProxyPass worker could be disabled like a balancer member.

The claim lives on for two reasons. Old logs and old answers still show AH00959 and AH00940 for single backends. And balancer members still behave exactly as the claim says.

What AH00940 means now#

AH00940: http: disabled connection for (host:port) is logged when a request is handed a worker that is in error state and too soon to retry. The request gets 503 at once, with no connect attempt. On current releases, a plain worker only reaches that state if you remove its ignore-errors flag with status=-I.

With status=-I, one more detail showed up. A plain worker's error state lives in the child process that saw the failure. In the test, one child logged AH00959 and then AH00940 with a 503. Two seconds later, another child served the same URL with a 200. Balancer members keep their state in shared memory, so every child sees the same error state.

Balancer members: failover, forcerecovery and AH01170#

A balancer is a group of workers declared with <Proxy "balancer://name"> and BalancerMember. When a member fails to connect, mod_proxy marks it as failed and tries another member for the same request. In the two-member test, the client got 200 every time. The log still showed AH00957, AH00959 and AH01114 for the dead member. So AH01114 in the log does not always mean a client saw a 503.

The cost comes later. The member that failed is skipped until retry runs out. In the test, the first member came back within a second. The next 20 requests all went to the other member. Only after 60 seconds did it get traffic again. Nobody sees an error, but the capacity you just restored sits idle for a minute.

Things change when every member is in error state. mod_proxy_balancer.c then checks the balancer's forcerecovery setting. The manual describes it as "Force the immediate recovery of all workers without considering the retry parameter of the workers if all workers of a balancer are in error state." The default is On, available since 2.4.2.

• forcerecovery=On (default): the next request clears the error state on every member and tries again (AH01165: Forcing recovery for worker, debug). A one-member balancer therefore acts much like a plain worker: 200 as soon as the app is back.
• forcerecovery=Off: nothing is forced. Each request is refused at once with AH01170: balancer://name: All workers are in error state and a 503 until retry runs out. With the default retry, that is the full 60 seconds of 503 after the app is back. The manual's reason for this setting is that "an already overloaded backend can get into deeper trouble" if all members are forced back at once.

So on a current httpd, a full minute of 503 needs three things: a balancer, every member in error state, and forcerecovery=Off. Apache's balancer manager and health check modules can also set states. The comparison with the passive checks in other proxies is in health checks and failover.

Fix 1: retry=0, or a shorter retry#

<Proxy "balancer://app">
    BalancerMember "http://app1:8000" retry=0
    BalancerMember "http://app2:8000" retry=0
</Proxy>
ProxyPass "/" "balancer://app/"

With retry=0, Apache still logs AH00959, but it reads disabling worker ... for 0s. The next request tries the member again. Failover still works, because the member is marked as failed for the request that hit it, and that request moves on to another member.

The price is one refused connect per request while a member is down. In the two-member test with one member down, all 12 requests got 200. The log held 12 AH00957 lines for the dead member, so each of those requests tried it first. On a refused port that costs well under a millisecond. On a backend that does not answer at all, each attempt waits for the connect timeout. connectiontimeout defaults to timeout, which defaults to ProxyTimeout. If that is a risk, use a short retry such as retry=5 rather than 0, and set connectiontimeout to a few seconds.

retry is whole seconds. A negative value is refused at startup with "Retry must be a positive value".

Fix 2: status=+I, but only for a single member#

<Proxy "balancer://app">
    BalancerMember "http://app1:8000" status=+I
</Proxy>

The manual says I puts the worker in "ignore-errors mode", and it "will always be considered available". The AH00959 branch is skipped, so the worker is never disabled. With one member, that gives the same result as a plain ProxyPass: 200 on the first request after the app is back.

Do not use it on one member of a larger balancer. A member that is never marked as failed keeps being picked. In the two-member test with +I on the dead member, requests alternated 503 200 503 200. Six of 12 failed, because no failover happened. For a group of two or more, use retry instead.

Fix 3: leave forcerecovery on, or turn it back on#

If the config sets forcerecovery=Off and you see a minute of 503 after every deploy, that setting is the cause. Remove it, or set ProxySet forcerecovery=On inside the <Proxy "balancer://..."> block. Keep it off only if your backends really cannot take a burst of traffic the moment they come back. If you keep it off, lower retry on the members so the window is shorter.

localhost, ::1 and the line that names 127.0.0.1#

With localhost in the URL, Apache asks the C library for every address of that name and tries them in order. For each address but the last, a failed attempt is logged at debug level. Only the last failure is logged at error level. At the default LogLevel warn, you never see the first try.

Two images gave two different results:

So on Alpine, a backend that listens only on 127.0.0.1 costs a refused [::1] attempt on every new connection, logged where you cannot see it. On Debian inside a container, ::1 is dropped from the list. A backend bound only to ::1 then fails with an error line that names 127.0.0.1, which looks like the wrong address. When every address fails, AH00959 names the worker as (localhost:8010), not an address.

The worker goes into error state only when all addresses fail. A refused [::1] followed by a working 127.0.0.1 is a success. That differs from nginx, which turns localhost into a group of two servers, each with its own failure count. Why nginx tries ::1 when proxy_pass says localhost covers that case, where the result is 10 seconds of no live upstreams. In Apache, the fix is the same: write 127.0.0.1 (or [::1]) in the URL, or bind the backend to both.

How the cases were reproduced#

Apache ran from the official httpd:2.4.69 and httpd:2.4.63 images and httpd:2.4-alpine (also 2.4.69), each with the stock event MPM and LogLevel warn proxy:debug proxy_http:debug proxy_balancer:debug. Each backend was a Python http.server in a second container sharing Apache's network namespace, so a stopped backend gave a clean (111) refusal on 127.0.0.1. Each worker had its own port, so no two cases shared a worker. A loop sent one request, started the backend, then sent a request every two seconds and noted the first 200. The (13) SELinux case was not run here. It comes from the policy source and from the Rocky Linux test in the nginx article linked above.

Find which case you have#

Turn up logging for the proxy modules only:

LogLevel warn proxy:debug proxy_balancer:debug

Then read the lines around one AH01114:

• No AH00959 after AH00957: a plain ProxyPass worker, or a member with +I. The 503 lasts only as long as the backend is down. If 503s go on after the app is up, the app is not where the URL points. Check the bracketed error.
• AH00959 ... for 60s, then 200s: a balancer that failed over or forced recovery. Nothing to fix for clients, but the member sat out a minute. Lower retry.
• AH00959, then AH01170: All workers are in error state: forcerecovery=Off. This is the 60 seconds of 503. Fix 3, or a lower retry.
• AH00940: disabled connection: a plain worker with status=-I, or an httpd older than 2.4.23.
• (13)Permission denied in AH00957: SELinux. No retry setting helps.

Switch debug off afterwards. At debug level, mod_proxy logs many lines for every request.

Worked case: a rolling restart behind a two-member balancer#

Take two app servers, app1 and app2, behind this config, with every setting left at its default:

<Proxy "balancer://app">
    BalancerMember "http://app1:8000"
    BalancerMember "http://app2:8000"
    ProxySet forcerecovery=Off
</Proxy>
ProxyPass "/" "balancer://app/"

Someone copied forcerecovery=Off from an old example. Now suppose a deploy restarts app1 at 10:00:00 and app2 at 10:00:30, and each takes five seconds to start. Walk it through with the rules above.

1. At 10:00:00 a request picks app1. The connect is refused. Apache logs AH00957, AH00959 ... for 60s and AH01114, then sends the same request to app2. The client gets 200.
2. app1 is back at 10:00:05. It stays in error state until 10:01:00, so app2 takes every request.
3. At 10:00:30 app2 restarts. The next request picks app2, is refused, and app2 goes into error state until 10:01:30. There is no member left to fail over to, so that client gets 503.
4. Both members are now in error. With forcerecovery=Off, nothing forces them back. Every request gets 503 with AH01170: All workers are in error state, even though app1 has been up since 10:00:05.
5. At 10:01:00 app1's retry runs out, and requests succeed again.

That is 30 seconds of 503 with a working backend sitting idle, which is the symptom in the question. With the default forcerecovery=On, step 4 does not happen: the first request after both members fail clears the error state on both, so app1 takes requests again from that moment. With retry=5 on each member, app1 would be back in use at 10:00:05 and step 3 would fail over to it. The deploy then causes no 503 at all.

Where this answer stops applying#

This page covers mod_proxy_http and the workers it uses on httpd 2.4. The table rows were run on 2.4.69 and 2.4.63 in Docker, on loopback, with refused connects. They did not include a backend that drops packets, so the connect timeout path was not timed. The version boundary for plain workers (2.4.20 without the default, 2.4.23 with it) comes from the tagged upstream source. Linux distributions patch their httpd builds, so check yours if it is older than 2.4.23. mod_proxy_hcheck, the balancer manager, failonstatus and failontimeout can also put a member in error state. They are not covered here. AJP, FastCGI and HTTP/2 backends use other modules, which have their own messages for a failed connect. The SELinux row was not run on this setup. Check it with getsebool httpd_can_network_connect on your own host.

Frequently asked questions#

What does AH01114 HTTP failed to make connection to backend mean?#

Apache's mod_proxy_http could not open a TCP connection to the backend, so nothing was sent. Apache returns 503 Service Unavailable. The AH00957 line just before it gives the operating system's reason, such as (111)Connection refused or (13)Permission denied.

Why does Apache return 503 and not 502 when the backend is down?#

The source sets status = HTTP_SERVICE_UNAVAILABLE when the connect fails, and that is 503. Apache keeps 502 for a connection that was made but returned no valid response. nginx and Caddy send 502 for the same refused connect.

Why does Apache keep returning 503 after the backend is back?#

On httpd 2.4.23 and later, that happens when the backend is a BalancerMember in error state and the balancer cannot force recovery. Look for AH01170: All workers are in error state and forcerecovery=Off. The member is skipped until retry runs out, 60 seconds by default.

What does AH00959 disabling worker for 60s mean?#

The connect to that worker failed, so Apache marked it as failed and will not try it again for retry seconds. On current releases, only balancer members and workers set to status=-I get this line. With retry=0 it reads for 0s, and the next request tries again.

Does retry=0 on ProxyPass fix AH01114?#

No. retry only shortens the time a worker stays disabled. AH01114 is the connect failure itself, and it will repeat until the backend is listening. On a plain ProxyPass worker in httpd 2.4.23 and later, retry has no visible effect, because the worker is never disabled.

Should I use status=+I or retry=0 on a BalancerMember?#

Use retry=0, or a short retry, when the balancer has two or more members, because it keeps failover. status=+I means the member is never marked as failed, so requests that pick it while it is down get a 503. In a two-member test, half the requests failed that way.

How do I fix (13)Permission denied in AH00957 on RHEL or Rocky Linux?#

That is SELinux blocking the httpd_t domain from connecting. setsebool -P httpd_can_network_connect 1 allows all TCP ports. A narrower fix uses httpd_can_network_relay plus a port label. Changing retry does not help, because every attempt is denied.