Why does nginx drop request headers with underscores?

nginx drops request headers with underscores because it classes a name such as X_Api_Key as invalid, and its default ignore_invalid_headers on discards invalid headers while it reads the request, before any location, proxy_set_header or proxy_pass runs. The backend never sees the header, nginx answers normally, and the only trace is an info-level line that the default error log does not record. HTTP itself allows the underscore: RFC 9110 defines a field name as a token, and the tchar list includes both _ and .. The rule comes from CGI, where X-Api-Key and X_Api_Key become the same variable. The fix, if you keep the underscore, is one directive in the right place:

http {
    underscores_in_headers on;   # http block, or the default server for the port
    # ...
}

The better fix, when you control the client, is to send X-Api-Key instead. Several other layers now drop underscores too, and turning the directive on in front of the wrong backend reopens the hole it exists to close.

How the cases were reproduced#

Everything below was observed on 20 October 2026 unless it is attributed to documentation or source code. nginx 1.27.5 and 1.30.6 ran from the official Alpine images and behaved identically. HAProxy 3.2.26, Caddy 2.11.7 and 2.11.4, Traefik 3.7.15, Envoy 1.36.13 and Apache httpd 2.4.69 with mod_proxy each sat in front of a Node 24 server that echoed the raw header names it received. Gunicorn 26.2.0, Uvicorn 0.54.0, the Werkzeug 3.1.9 development server and PHP-FPM 8.4.26 were tested as backends. The client was curl 8.23.0, plus a raw socket script where header order mattered. The ingress-nginx row comes from its documentation and was not run.

What nginx does with the header#

nginx's core module documentation states the rule in two directives. underscores_in_headers (default off): "When the use of underscores is disabled, request header fields whose names contain underscores are marked as invalid and become subject to the ignore_invalid_headers directive." ignore_invalid_headers (default on): "Valid names are composed of English letters, digits, hyphens, and possibly underscores." Both are allowed only in the http and server contexts, so there is no per-location switch.

The consequence is wider than underscores. A header named X.Api.Dot was dropped by the stock configuration too, while X-Api-Ok sent alongside it arrived. Any name with a character outside letters, digits and hyphens goes the same way.

The drop happens while the request headers are parsed, which has three effects people do not expect:

• No variable sees it. With the default settings, $http_x_api_key was empty in the access log for a request carrying X_Api_Key: under. You cannot rescue the header with proxy_set_header X-Api-Key $http_x_api_key;, because by the time that runs the header is gone.
• The client gets a normal response. Status 200, no warning header, nothing in the body. The backend simply behaves as if the client had not sent it, which usually shows up as a 401 from an API key check or a feature flag that never switches on.
• The default error log is silent. nginx logs the drop at info:

[info] 30#30: *1 client sent invalid header line: "X_Api_Key: under" while reading client request headers, client: 10.201.13.5, server: , request: "GET / HTTP/1.1", host: "n1.30"

Over HTTP/2 the wording changes to client sent invalid header: "x_api_key". The official nginx image sets error_log ... notice;, and the compiled default is error, so neither shows info lines. To confirm the cause, raise the level temporarily (error_log /var/log/nginx/error.log info;) and grep for invalid header.

Response headers are a different path. A backend that answered with X_Resp_Under: r1 had that header delivered to the client by nginx unchanged; the directives apply only to "client request header fields". So an application can send underscore headers out through nginx but cannot receive them.

Where underscores_in_headers has to go#

The directive's own documentation adds a sentence most answers leave out: "If the directive is specified on the server level, the value from the default server can be used." The server names page explains why. A connection starts in the default server's context, and the server is chosen again after the request line and after the Host header. For ignore_invalid_headers, large_client_header_buffers and underscores_in_headers, the effect "additionally depends whether the server configuration was updated according to the request line or the Host header field".

In practice, headers parsed before nginx knows the server name are judged by the default server's setting. The test used a default server with no directive and a second server, server_name app.test, with underscores_in_headers on;. The same request, sent with a different header order:

Both nginx versions logged the failing request under server_name app.test with an empty $http_x_api_key, so the access log points at the server that did have the directive. curl and browsers put Host first, which is why a server-level setting appears to work in testing and then fails for an SDK, a load balancer health check or a hand-built request that orders headers differently. When the directive is confirmed to be in place and the header still goes missing, the problem is usually another hop: nginx trac ticket 2251, where the reporter had it in both the http and server blocks, was closed as "worksforme" after the maintainer showed the header reaching his backend.

The rule that follows: set underscores_in_headers on; in the http block, or in the server marked default_server for every listen address that needs it. A named server on its own is not enough. The same applies to ignore_invalid_headers off.

Why nginx drops them: two names, one variable#

RFC 3875, the CGI specification, says how a header becomes a variable: "The HTTP header field name is converted to upper case, has all occurrences of "-" replaced with "_" and has "HTTP_" prepended." X-Api-Key and X_Api_Key both become HTTP_X_API_KEY. The same RFC says that multiple fields with the same name must be rewritten "as a single value having the same semantics", but these are not the same field name in HTTP, so each server picks its own resolution.

That ambiguity is a security problem when one layer makes a trust decision on the exact header name and a later layer folds the two together. The case that made Caddy change its defaults, advisory GHSA-f59h-q822-g45g, is the textbook example. Caddy's forward_auth deleted the client's Remote-User before copying the authenticated value from the auth service, but a client-sent Remote_User survived the delete, and php_fastcgi then mapped both to HTTP_REMOTE_USER. The advisory's summary: "a remote client can inject or sometimes override identity/group headers trusted by PHP/FastCGI applications behind Caddy." The same pattern applies to every forward auth setup in nginx, Traefik or Caddy that passes identity headers to a CGI-style backend.

The collision is easy to reproduce with nginx in front of PHP-FPM 8.4.26 through fastcgi_pass and the stock fastcgi_params. The script printed $_SERVER['HTTP_X_API_KEY']:

With the directive on, the last header in the request won. If anything in front of PHP trusts X-Api-Key, a client that appends X_Api_Key replaces the value PHP sees. With the default off, the hyphenated header is the only one that can arrive, and the ambiguity cannot happen. The drop is still not an access control. As mTLS through proxies puts it, it accidentally protects X_Client_DN but not X-Client-DN: a client can always send the hyphenated name, and only an explicit overwrite at the proxy removes that. The proxy security checklist lists handling underscore headers deliberately as its own item for that reason.

nginx's own variables have the same folding. The documentation defines $http_name as the field name "converted to lower case with dashes replaced by underscores", so $http_x_api_key matches both spellings. With underscores_in_headers on and both headers sent, the access log showed $http_x_api_key as under, dash or dash, under, joined in arrival order, while the Node backend received two separate headers. A map or if on $http_x_api_key cannot tell which name the client used.

Caddy, Gunicorn and Werkzeug drop them too#

Moving the underscore header past nginx often only moves the problem to the next hop, because the same CGI argument has been applied elsewhere.

Caddy 2.11.4 and later drops underscore headers before any handler runs. In 2.11.4 the drop was unconditional, in the server's request path, with a comment citing GHSA-f59h-q822-g45g, and the drop is logged only at debug as dropping header containing underscore. In testing, 2.11.4 dropped X_Api_Key and still passed X.Api.Dot. From 2.11.5 (tagged 30 September 2026; 2.11.6 on 1 October is the first published release that includes it) dotted names are dropped too, citing advisory GHSA-49wc-4hcv-v58q about backends such as PHP's $_SERVER that fold dots, and two allowlists were added: expected_underscore_headers and expected_dot_headers. On 2.11.7 the stock reverse_proxy passed neither X_Api_Key nor X.Api.Dot. 2.11.3 and earlier have no such code, so a Caddy upgrade can break a working integration with no configuration change. The page on Caddy as a reverse proxy covers its other defaults.

{
    servers {
        expected_underscore_headers X_Api_Key
    }
}

The allowlist is stricter than nginx's switch, and the source comments in server.go say how. "When this list is configured, only the specified headers are kept; their hyphenated variants are actively dropped to prevent confusion." In testing, a request with both X_Api_Key and X-Api-Key delivered only X_Api_Key. "If an allowlisted header arrives with multiple values (repeated field), all values are dropped": two X_Api_Key lines delivered none. A trailing * works as a prefix glob. The option is marked experimental in the source.

Gunicorn 22.0.0 and later drops them in the application server. The header_map setting defaults to drop. Its documentation says that mapping different names into one environ key "will dangerously confuse applications as to which is which", that drop will "silently drop headers that cannot be unambiguously mapped", that refuse "will return an error if a request contains any such header", and that dangerous restores the old folding. Gunicorn 26.2.0 behaved as described: drop gave an environ with no HTTP_X_API_KEY for X_Api_Key alone, and kept dash when both were sent; refuse returned 400 Bad Request for every request containing an underscore header; dangerous produced under,dash in arrival order. So a Django or Flask app under Gunicorn loses the header even after nginx is fixed. The Werkzeug 3.1.9 development server did the same as Gunicorn's drop, so the header also vanishes under flask run.

Uvicorn and Node pass both names separately. ASGI keeps raw header names, so Uvicorn 0.54.0 handed the app x_api_key and x-api-key as two entries. Node 24's rawHeaders had both. These backends have no collision to protect against, which is the one case where turning underscores on is safe.

What passes them, and what can be made to drop them#

HAProxy 3.2.26, Traefik 3.7.15, Envoy 1.36.13 and Apache 2.4.69's mod_proxy all forwarded X_Api_Key and X.Api.Dot unchanged in content. HAProxy and Envoy sent the names in lower case over HTTP/1.1; Traefik, through Go's header canonicalisation, sent X_api_key. Header names are case-insensitive, so neither change matters to a correct backend, but code that compares raw names will see it.

Envoy is the only one with a documented switch. headers_with_underscores_action in the HTTP protocol options defaults to ALLOW. DROP_HEADER removed X_Api_Key in testing (the documentation says it happens "before the filter chain is invoked", and counts each drop in dropped_headers_with_underscores), and REJECT_REQUEST answered 400 Bad Request (HTTP/2 requests "end with the stream reset", per the documentation). Envoy listeners, routes and clusters shows where the HTTP connection manager options live.

Apache passing the header through mod_proxy is not the same as Apache's CGI behaviour. Its environment variable documentation says that when headers become variables for CGI scripts "only dashes are replaced with underscores; if the header contains any other (invalid) character, the whole header is silently dropped", a 2.4 change made because the old translation "allowed for some potential cross-site-scripting attacks via header injection". So mod_php, mod_cgi and similar modules inside Apache apply the drop themselves, even though mod_proxy does not.

ingress-nginx exposes the nginx directives as ConfigMap keys: enable-underscores-in-headers, default "false", and ignore-invalid-headers, default "true". They are controller-wide, not per Ingress. The controller is retired and the annotations that replace it in Gateway API are covered in moving off ingress-nginx; any replacement needs its own decision about underscore headers.

Which fix to use#

You control the client: rename the header. X-Api-Key passes every layer in the tables above with no configuration, and it removes the ambiguity rather than choosing a winner. A header name is part of an API contract, so keep accepting the old name at the application during a transition only if no proxy in the path drops it.

You cannot change the client, and the backend keeps names apart (Node and Uvicorn did): enable it at the right level. Put underscores_in_headers on; in the http block, or in the default server for the port. On Caddy 2.11.4 or later, list the exact names in expected_underscore_headers rather than a broad glob. Check the backend's own server: Gunicorn and the Werkzeug development server will still drop it.

The backend is PHP, FastCGI or CGI: do not enable it for headers anything trusts. Where a trusted value travels in a header, set the variable yourself at the proxy. With underscores_in_headers on and fastcgi_param HTTP_X_API_KEY "set-by-proxy"; in the location, PHP-FPM saw set-by-proxy whether the client sent X_Api_Key, X-Api-Key or both, so the explicit parameter won over every client spelling. Remember the replace-not-merge inheritance of proxy_set_header and fastcgi_param described in nginx as a reverse proxy: a location with its own set loses the server's.

Do not use ignore_invalid_headers off as the fix. It passed X_Api_Key, but it also passed X.Api.Dot and every other name with characters outside the valid set, which is the wider version of the same collision problem for backends that fold dots.

Failure modes#

A custom header works with curl against the app and is missing behind nginx. The name contains an underscore or a dot. Raise error_log to info and look for client sent invalid header line.

underscores_in_headers on is set and the header still sometimes disappears. The directive is in a named server, not the default server, and some clients send the header before Host. Move it to the http block.

The header vanished after a Caddy upgrade. Caddy 2.11.4 started dropping underscore headers and 2.11.5 dotted ones. Add the name to expected_underscore_headers or expected_dot_headers, or rename the header.

nginx passes it, the Django or Flask app still does not see it. Gunicorn 22.0.0 and later (default header_map drop) or the Werkzeug development server removed it. Rename the header.

Every request with the header returns 400 from the backend. Gunicorn runs with --header-map refuse, or Envoy has REJECT_REQUEST.

A PHP app sees a value the proxy did not set. underscores_in_headers on lets a client-sent underscore spelling collide with the trusted hyphenated header in $_SERVER. Turn it off, or overwrite the variable explicitly at the proxy.

Where this answer stops applying#

The nginx results were identical on 1.27.5 and 1.30.6 with HTTP/1.1 and HTTP/2; HTTP/3 was not tested. The header order effect needs a non-default server with the directive and a client that sends the header before Host; most browsers and curl do not. Caddy results are for 2.11.4 and 2.11.7 and, for 2.11.5 and later, the 2.11.7 source; the allowlist is marked experimental and may change. Gunicorn, Uvicorn, Werkzeug and PHP-FPM were tested only at the versions listed. Apache's CGI behaviour is from its documentation; only mod_proxy was run. Managed load balancers and CDNs were not tested, and each has its own rules for header names.

Frequently asked questions#

Why does nginx ignore headers with underscores?#

Because underscores_in_headers is off by default, which marks names containing _ as invalid, and ignore_invalid_headers is on by default, which drops invalid headers. The reason is CGI: X-Api-Key and X_Api_Key both become HTTP_X_API_KEY in a CGI-style backend, so allowing both lets a client collide with a header a proxy trusts.

How do I allow underscores in headers in nginx?#

Add underscores_in_headers on; to the http block, or to the default_server for the port. Then reload. A setting only in a named server block is applied too late for headers sent before Host.

Why is underscores_in_headers on not working?#

Usually because it is in a server block that is not the default server, and the client sends the header before Host, or because another layer drops it: Caddy 2.11.4 and later, Gunicorn 22.0.0 and later, the Werkzeug development server, or ingress-nginx's controller-wide default. Test each hop with curl to find which one removes it.

Is it safe to enable underscores_in_headers?#

It is safe when the backend keeps X_Api_Key and X-Api-Key as different headers, as Node 24 and Uvicorn did in testing. It is unsafe in front of PHP, FastCGI or CGI backends, where the client's underscore spelling can override a hyphenated header set by an auth proxy.

Are headers with underscores valid HTTP?#

Yes. RFC 9110 defines a field name as a token, and the token characters include _ and .. Dropping them is a choice made by nginx, Caddy, Gunicorn and Apache's CGI layer, not a protocol requirement.

Does nginx drop response headers with underscores?#

No. The directives apply to client request headers. A backend response header named X_Resp_Under was passed to the client unchanged by nginx 1.27.5 and 1.30.6.