What does "reset reason: connection termination" mean in Envoy and Istio?

upstream connect error or disconnect/reset before headers. reset reason: connection termination means Envoy had already sent your request to the backend, and the backend closed the connection before it sent back a single response header. Envoy did not time out, and it did not fail to connect. The backend hung up. Envoy then answers the client itself, with 503, the response flag UC and the details upstream_reset_before_response_started{connection_termination}. The usual cause is not a crash. It is an idle keep-alive connection that the backend closed at the same moment Envoy reused it. Envoy keeps idle connections for an hour, and most app servers keep them for two to six seconds.

The words after reset reason: change with the cause. Every row below was produced on Envoy 1.36.13:

The general Envoy flags table, with UH, UT and NR, is in 502 vs 503 vs 504. This page covers only the message in the body.

What the message says, word by word#

An upstream is the server Envoy forwards to: your app, or another proxy. Envoy's word for a group of them is a cluster. The message is one fixed sentence with a reason on the end, and each part has a meaning.

• upstream connect error covers a failed connection: the dial did not work.
• disconnect/reset covers a connection that worked and then broke. A disconnect is a normal close, a TCP FIN. A reset is an abrupt close, a TCP RST. Envoy uses one phrase for both.
• before headers means Envoy had not received the response headers yet. So it still owned the reply, and it wrote its own 503 or 502 with this text as the body.
• reset reason is Envoy's name for why the upstream stream ended. For a refused connection it adds transport failure reason: and the operating system's error.

Envoy writes the same reason into the access log in a form with no spaces. The response code details page lists it as upstream_reset_before_response_started{details}: "The upstream connection was reset before a response was started". The substitution formatter page says the details string "will not contain any whitespaces, which will be converted to underscore". That is why the body says connection termination and the log says connection_termination. The flag reference gives UC the long name UpstreamConnectionTermination: "Upstream connection termination in addition to 503 response code".

The default access log does not show the details. Add the two fields to your format. The format used in these tests was:

[%START_TIME%] %REQ(:METHOD)% %REQ(:PATH)% %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% tfr=%UPSTREAM_TRANSPORT_FAILURE_REASON% %DURATION%ms up=%UPSTREAM_HOST% local=%UPSTREAM_LOCAL_ADDRESS% attempts=%UPSTREAM_REQUEST_ATTEMPT_COUNT%

%UPSTREAM_LOCAL_ADDRESS% is Envoy's own side of the upstream connection. It shows whether a request went on a new connection or a reused one, which is the whole question for UC.

How the cases were reproduced#

Everything here was observed on 3 November 2026 unless a document is quoted. Envoy 1.36.13 ran from the envoyproxy/envoy:v1.36-latest image on Docker Engine 28.1.1, Linux host, one listener, one cluster per case, and no retry policy unless stated. Backends ran in other containers on the same Docker network. A small Python socket server could close, reset, stall or answer with junk on cue. Node 24.22.0 served HTTP/1.1 and HTTP/2. Gunicorn 26.2.0 (gthread worker) and Uvicorn 0.54.0 ran on Python 3.13.17. Requests came from curl and from a Python client. The race tests ran Envoy with --concurrency 1, so one worker thread held one connection pool. With the default of one worker per core, each request can land in a different pool, and reuse is harder to steer. Istio was not run. Everything said about Istio comes from istio.io.

connection termination: four causes, one string#

A backend that read the request and closed the connection gave connection termination. So did a backend that read it and sent a TCP reset. Envoy 1.36.13 did not tell FIN from RST on an HTTP/1.1 upstream. Both lines read:

GET /fin 503 UC upstream_reset_before_response_started{connection_termination} tfr=- 1ms
GET /rst 503 UC upstream_reset_before_response_started{connection_termination} tfr=- 0ms

A backend container killed with SIGKILL while it held a request gave the same line. The request had waited 488 ms when the process died. An HTTP/2 backend that sent GOAWAY and closed the session also gave connection termination, with flag UC. It did not give remote reset.

In every case the cluster counter upstream_cx_destroy_remote_with_active_rq went up by one. That stat means the upstream closed a connection that had a request on it. Envoy's admin endpoint shows it at /stats. When this counter climbs, the upstream is closing connections under requests, and the question is why.

The four causes differ in timing, so ask when the 503s happen:

• Every request to that cluster fails. The backend closes on purpose. It may not be an HTTP server on that port, or it rejects something about the connection. Look in the backend's own log.
• 503s come in a burst at a deploy or a crash. Requests were in flight on an instance that stopped. Check the backend's restarts and its shutdown handling.
• One 503 now and then, after a quiet spell, and it never repeats on a retry. That is the keep-alive race, below. It is the most common cause by far.
• A 503 on a long request at a fixed duration. Something between Envoy and the app closed it on a timer: a sidecar, a NAT or a firewall. This case was not reproduced here. Timeout budgets shows how to line up every timer on the path so the right one fires first.

The keep-alive race: Envoy's one hour against the app's few seconds#

Envoy reuses upstream connections. After a response, the connection goes back to the pool and waits for the next request. Envoy closes it when it has been idle for idle_timeout. The protocol options reference defines that as "the period in which there are no active requests" and says: "If not specified, this defaults to 1 hour." For upstream connections the field lives in common_http_protocol_options, inside the cluster's typed_extension_protocol_options.

The app closes idle connections too, much sooner:

So the backend always closes first. Mostly that is harmless. The backend sends FIN, Envoy sees it while the connection is idle, and Envoy drops it from the pool. The next request opens a new connection. Gunicorn, Uvicorn and Node did close 104, 62 and 31 connections this way during the tests without one failure.

The failure needs one more thing: Envoy picks the connection and sends a request at the same moment the backend closes it. The request is on the wire before the FIN reaches Envoy. The backend's kernel answers the request with a reset, and Envoy logs connection termination. Node's own docs say the buffer exists to "reduce connection reset (ECONNRESET) errors", which is this race seen from the other side.

To find the window, a client sent a request, waited a set gap, and sent another. The gap moved in 5 to 10 ms steps around each backend's timeout. On the Docker network, with well under a millisecond between Envoy and the backend, the window was too narrow to hit. There were no errors in 243 tries across the three backends. With 20 ms of delay added on Envoy's outbound traffic (tc qdisc ... netem delay 20ms), each backend failed where its timeout said it would:

The failing gaps sit just under each timeout because the client measured its gap from when it got the first response, which reached it after the delay. Node failed at six seconds, not five. That is the buffer at work. It does not remove the race. It moves it.

The window is the time a request takes to reach the backend. A request that crosses a node, a zone or a sidecar takes longer than one on a Docker bridge, so production hits it more often than a laptop does. Low traffic makes it worse, because idle gaps near the backend's timeout are more common. This is why the error shows up overnight and on quiet services.

Keep-alive and upstream connection pooling sets out the general rule for every proxy: whoever closes an idle connection with nothing in flight causes no error, so the proxy must close first. Caddy and Go show the same race as a 502 on POSTs, described in the Caddy dial error guide.

The fix: make Envoy close idle connections first#

Set the cluster's idle timeout below the backend's keep-alive timeout:

clusters:
- name: app
  typed_extension_protocol_options:
    envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
      "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
      common_http_protocol_options:
        idle_timeout: 4s        # backend keeps idle connections for 5s
      explicit_http_config:
        http_protocol_options: {}

With idle_timeout at 1 s for Gunicorn and 4 s for Uvicorn and Node, the same 20 ms delay and the same gaps around each timeout gave 0 errors in 83 tries. The stat upstream_cx_idle_timeout counted 85 closes, and upstream_cx_destroy_remote stayed at 0. Envoy closed every idle connection itself, before the backend could.

The other direction also works: raise the app's keep-alive above Envoy's. But Envoy's default is an hour, and an app holding idle sockets for an hour is a poor trade. Lowering Envoy's timer is the change to make. Leave a margin of a second or more, since a timer is not exact. Gunicorn's sync worker never reuses a connection, so it cannot race. The gthread, gevent and eventlet workers can.

Envoy listeners, routes and clusters shows where typed_extension_protocol_options sits in a full bootstrap.

Retrying the race: retry_on reset#

With no retry policy, Envoy does not retry. The router reference says: "If a retry policy is not configured and x-envoy-retry-on or x-envoy-retry-grpc-on headers are not specified, Envoy will not retry a failed request." Every UC line in these tests showed attempts=1.

The retry condition that names this event is reset. The router reference defines it: "Envoy will attempt a retry if the upstream server does not respond at all (disconnect/reset/read timeout.)" The broader 5xx condition covers it too, since it also retries when the upstream "does not respond at all". With retry_policy: {retry_on: reset, num_retries: 1} on the route, a backend that closed every reused connection gave 200 on every request. The log showed attempts=2, the second attempt on a new local port, and upstream_rq_retry_success counted each one. The retried GETs took 15 to 25 ms in all. The reference gives a "default base interval of 25ms" for the jittered back-off, so a first retry waits up to 24 ms.

Envoy retried a POST on reset the same way, and the POST got 200. That is a real difference from Go's transport, which retries only idempotent requests. In the race, the backend never read the request, so a retry is safe. When a backend is killed mid-request, it may have done the work, and a retried POST runs it twice. Envoy cannot tell these apart from connection termination. Two narrower choices exist:

• reset-before-request: "Equivalent to reset but will only retry requests that have not been sent to the upstream server (i.e. the headers have not been sent)." It does not catch the race, where the headers were sent.
• reset on read-only routes only, and the idle timeout fix for the rest.

Fix the timeout first. A retry hides the 503 but still pays the round trip.

Circuit breaker overflow: Envoy refused before sending#

reset reason: overflow is not about the backend at all. Envoy's circuit breakers cap connections, pending requests and active requests per cluster. When a cap is full, Envoy fails the request at once without sending it. The circuit breaking overview says they "are enabled by default and have modest default values, e.g. 1024 connections per cluster".

With max_connections, max_pending_requests and max_requests all at 1, and a backend taking 3 s per reply, three requests 0.2 s apart gave 200, 503, 503. Both 503s took 0 ms and named no local address, because no connection was used. The response carried x-envoy-overloaded: true, which the router reference says is set "if a request was dropped due to either maintenance mode or upstream circuit breaking". The counter upstream_rq_pending_overflow counted 2.

So overflow with an idle backend means the limits are too low for the traffic, not that the backend is sick. Raise the thresholds in circuit_breakers, or in Istio the DestinationRule connectionPool fields. The DestinationRule reference gives http1MaxPendingRequests and http2MaxRequests a default of 2^32-1, so in Istio a low limit was set by someone.

protocol error is a 502, not a 503#

A cluster set to HTTP/2 (explicit_http_config: {http2_protocol_options: {}}) in front of an HTTP/1.1-only backend gave reset reason: protocol error, flag UPE, details upstream_reset_before_response_started{protocol_error}, and status 502. A backend that answered HELLO instead of an HTTP status line gave the same. It is the only reason in this set that returned 502. upstream_cx_protocol_error counted each one.

If you see this after a config change, check which protocol the cluster speaks. An HTTP/2 cluster needs an HTTP/2 backend, or h2c on a plain port. gRPC is the usual reason a cluster is HTTP/2 in the first place; gRPC through proxies covers the setup.

remote reset and remote refused stream reset: HTTP/2 backends#

On an HTTP/2 upstream, a backend can end one stream and keep the connection. A Node HTTP/2 server that closed each stream with INTERNAL_ERROR gave reset reason: remote reset, flag UR, 503. The same server closing with REFUSED_STREAM gave remote refused stream reset, also UR, 503. The difference matters for retries. The router reference says a REFUSED_STREAM reset "indicates that a request is safe to retry", and the refused-stream condition is in Istio's default list.

The connect failures: UF#

These are dial errors. Nothing was sent.

• A closed port gave remote connection failure, transport failure reason: delayed connect error: Connection refused in 0 ms. %UPSTREAM_TRANSPORT_FAILURE_REASON% held delayed_connect_error:_Connection_refused.
• An address with nothing behind it and connect_timeout: 1s gave connection timeout after 996 ms.
• A cluster with a TLS transport socket pointed at a plain HTTP port gave connection timeout too, after 5000 ms, the cluster's connect_timeout. The TLS handshake never finished, so Envoy counted it as a slow connect. The transport failure field was empty. A UF that always takes exactly connect_timeout is worth checking for a TLS mismatch before blaming the network.

Istio: the same timer and the same retry, in mesh fields#

Istio's sidecars and gateways are Envoy, so the bodies and flags above are what Istio shows. The settings have Istio names.

Idle timeout. The DestinationRule's trafficPolicy.connectionPool.http.idleTimeout is described as "The idle timeout for upstream connection pool connections. The idle timeout is defined as the period in which there are no active requests. If not set, the default is 1 hour." That is Envoy's definition and Envoy's default. Set it below the app's keep-alive:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: api
spec:
  host: api.default.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        idleTimeout: 4s

There is a second field, connectionPool.tcp.idleTimeout. It counts bytes rather than requests ("no bytes sent or received"), also defaults to 1 hour, and the reference warns it is "a property of a listener, not a cluster" when weighted destinations are used. For the HTTP keep-alive race, http.idleTimeout is the one that matches.

Retries. The VirtualService reference states the mesh default: "attempts: 2 retryOn: 'connect-failure,refused-stream,unavailable,cancelled'". That covers UF and REFUSED_STREAM. It has no reset, so a connection termination is not retried by default. To retry it, add reset:

http:
- route:
  - destination:
      host: api.default.svc.cluster.local
  retries:
    attempts: 2
    retryOn: connect-failure,refused-stream,unavailable,cancelled,reset

Do not write 503 and expect it to catch this. The reference says: "if a connection is reset, Istio will translate this to 503 for it's response. However, the destination did not return a 503 error, so this would not match '503' (it would, however, match 'reset')." The POST warning above applies in Istio too.

Worked case: one UC traced to its timer#

Take a Uvicorn app behind Envoy with stock settings. Suppose a few requests a day fail with reset reason: connection termination, mostly at night, and a retry by the client always works. Walk it through.

1. Read the access log line. With %RESPONSE_CODE_DETAILS% in the format, suppose it says 503 UC upstream_reset_before_response_started{connection_termination}. So the request was sent, and the backend closed.
2. Look at %UPSTREAM_LOCAL_ADDRESS%. Suppose the same local port appears on an earlier line for that host, a few seconds before. The failing request went out on a reused connection.
3. Check /stats on the admin port. A rising cluster.<name>.upstream_cx_destroy_remote_with_active_rq confirms the backend closed under a request.
4. Compare the timers. Envoy's idle_timeout is unset, so it is 1 hour. Uvicorn's --timeout-keep-alive is 5 s. Envoy will happily reuse a connection that Uvicorn is about to close.
5. Set idle_timeout: 4s on the cluster, or connectionPool.http.idleTimeout: 4s in Istio. Then upstream_cx_idle_timeout should rise and upstream_cx_destroy_remote_with_active_rq should stop.

If step 2 showed a new local port every time, the race is not the cause. Go back to the timing list and check the backend for restarts or for closing every connection.

Failure modes#

idle_timeout set in the wrong block. The cluster field is common_http_protocol_options inside envoy.extensions.upstreams.http.v3.HttpProtocolOptions. The HTTP connection manager has a field with the same name. Its reference says it holds settings "for HTTP requests handled by the connection manager", which are the client's requests. Setting idle_timeout there leaves the upstream pool at 1 hour.

retryOn: 503 expected to catch it in Istio. The 503 is made by Envoy, not sent by the app, so a status code match misses it. Istio's reference says so in as many words. Use reset, or 5xx, whose Envoy definition includes "does not respond at all".

Fixed in testing, back in production. The window is the network delay between Envoy and the app. A test on one host may show no errors while a cross-zone hop shows many. Test with delay, as above, or watch the stats after the change.

Where this answer stops applying#

The bodies, flags and statuses here are Envoy 1.36.13 with static configuration, HTTP/1.1 clients and HTTP/1.1 or HTTP/2 upstreams. Other Envoy releases can word a reason differently, so read your own log. Istio was not run, and its fields are taken from istio.io alone. In a mesh, find which proxy wrote the line before changing a timer, since each hop has its own pool. LR (local reset) and upstream_reset_after_response_started, where the backend fails after the headers and the client gets a cut-off body rather than this message, were not reproduced. The race windows were measured with a fixed 20 ms of added delay, so they show where the race sits, not how often it hits on your network. The app server defaults are for Gunicorn 26.2.0, Uvicorn 0.54.0 and Node 24.22.0. Node before 24.6.0 has no keepAliveTimeoutBuffer and closes at 5 s.

Frequently asked questions#

What does reset reason: connection termination mean?#

The backend closed the TCP connection after Envoy sent the request and before any response header came back. Envoy answers 503 with flag UC. The usual cause is an idle keep-alive connection that the backend closed just as Envoy reused it.

Why do I get upstream connect error or disconnect/reset before headers in Istio?#

Istio's sidecars are Envoy, so the body is Envoy's. Read the words after reset reason:. connection termination is usually the idle timeout mismatch: set the DestinationRule's connectionPool.http.idleTimeout below the app's keep-alive timeout.

What is Envoy's default upstream idle timeout?#

One hour. The Envoy protocol options reference and the Istio DestinationRule reference both give 1 hour when idle_timeout or idleTimeout is not set. Most app servers close idle connections within 2 to 6 seconds, so Envoy's timer should be set lower.

Does Envoy retry a connection termination by default?#

No. With no retry policy, Envoy does not retry anything, and Istio's default retryOn list does not include reset. Add retry_on: reset in Envoy or reset to retryOn in Istio. Envoy retried a POST that way too, so limit it to requests that are safe to repeat.

What is the difference between UC and UF in Envoy?#

UF means Envoy could not open the connection, so nothing was sent. UC means the connection worked, the request went out, and the backend closed it before replying. A UF is safe to retry. A UC may have reached the app.

Why does reset reason: protocol error return 502 and the others 503?#

In these tests on Envoy 1.36.13, protocol error (flag UPE) was the only reason that returned 502. It means the backend's reply did not fit the protocol Envoy spoke, most often an HTTP/2 cluster in front of an HTTP/1.1-only app.