Grafana origin not allowed behind a reverse proxy
Grafana returns 403 origin not allowed when the proxy rewrites Host. The nginx, Traefik and Caddy fixes, the Live WebSocket check, and the csrf settings.
Key points
- Grafana's CSRF check compares the host name in the browser's
Originheader with the host name in theHostheader it receives. nginx sendsHost: $proxy_hostby default, the address fromproxy_pass, so the names differ and Grafana answers403 origin not allowed. - Pages load and saving fails because browsers send
OriginonPOST,PUTandDELETEbut not on a same-originGET. Grafana skips the check when there is noOrigin. Saving a dashboard (POST /api/dashboards/db) and panel queries (POST /api/ds/query) are checked. - The fix for nginx is
proxy_set_header Host $host;in every location that reaches Grafana. Ports and schemes are ignored by this check, so$hostand$http_hostboth pass it. Apache needsProxyPreserveHost On. - Grafana Live, the WebSocket at
/api/live/ws, runs its own origin check. It needs the$connection_upgrademap, and anOriginthat matchesHostexactly (port included) or the scheme and host ofroot_url. A failure there is a 403 logged asRequest Origin is not authorized. - When the proxy cannot send the public
Host, Grafana 9.1.0 and later accept it another way:csrf_trusted_origins(bare host names, not URLs) orcsrf_additional_headers = X-Forwarded-Host.
Grafana says "origin not allowed" because the Host header it receives does not name the same server as the browser's Origin header. Behind nginx this is almost always the default Host value. nginx replaces the client's Host with the address written in proxy_pass, such as 127.0.0.1:3000. The browser's Origin says https://grafana.example.com. The two host names differ, so Grafana refuses the request with a 403. Add proxy_set_header Host $host; to each location that proxies to Grafana, and the check passes.
If you also use Grafana Live (dashboards that update without a refresh), the WebSocket at /api/live/ws needs the upgrade headers and the same Host fix. Set root_url to the public URL as well. The rest of this page explains both checks, why only some actions fail, and what to do when the proxy cannot send the public Host.
The fix for nginx#
This is the configuration from the Grafana tutorial, with root_url added. It covers both checks.
# Needed for the Grafana Live WebSocket.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream grafana {
server 127.0.0.1:3000;
}
server {
listen 443 ssl;
server_name grafana.example.com;
location / {
proxy_set_header Host $host;
proxy_pass http://grafana;
}
# Grafana Live WebSocket connections.
location /api/live/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_pass http://grafana;
}
}[server]
domain = grafana.example.com
root_url = https://grafana.example.com/Two details trip people up. First, nginx inherits proxy_set_header lines from the level above only if the current level has none. So a location that sets any header of its own must repeat Host. Second, root_url defaults to %(protocol)s://%(domain)s:%(http_port)s/. With no changes that is http://localhost:3000/, which is not the URL anyone uses behind a proxy. The general nginx proxy settings are covered in nginx as a reverse proxy.
What "origin" means here#
An origin is the scheme, host and port a web page came from. For a page at https://grafana.example.com/d/abc, the origin is https://grafana.example.com. Browsers put it in an Origin request header so a server can tell where a request started.
CSRF stands for cross-site request forgery. In this attack, a page on another site makes your browser send a request to Grafana. The browser attaches your Grafana login cookie, so the request runs as you. A CSRF check stops this by asking where the request came from. If the Origin is some other site, the server refuses.
Grafana's check does not compare the Origin with a configured list by default. It compares the Origin with the Host header of the same request. Without a proxy, those two always agree, because the browser writes both. With a proxy, the browser writes Origin and the proxy writes Host. That is where they drift apart.
Worked example: one dashboard save through default nginx#
Take a common setup and follow one request through it. Grafana listens on 127.0.0.1:3000. nginx terminates TLS for grafana.example.com and has only proxy_pass http://127.0.0.1:3000; in its location. Grafana's grafana.ini is untouched.
You open a dashboard and click Save. The browser sends:
POST /api/dashboards/db HTTP/1.1
Host: grafana.example.com
Origin: https://grafana.example.com
Cookie: grafana_session=...
Content-Type: application/jsonThe nginx documentation says that by default the client's Host is not passed on. nginx sends Host: $proxy_host instead, which is the host and port from proxy_pass. So Grafana receives:
POST /api/dashboards/db HTTP/1.1
Host: 127.0.0.1:3000
Origin: https://grafana.example.com
Cookie: grafana_session=...Now walk it through the check in csrf.go. The request carries the grafana_session cookie, so the check runs. POST is not on the skip list. Grafana strips the port from Host and gets 127.0.0.1. It takes the host name from Origin and gets grafana.example.com. These differ, and no trusted origin is configured. Grafana answers 403 with the plain text body origin not allowed, and the UI shows it as a red message.
Add proxy_set_header Host $host; and run the same request again. Grafana now receives Host: grafana.example.com. Both names are grafana.example.com, and the save goes through.
Now the Live WebSocket on the same default setup. The browser opens wss://grafana.example.com/api/live/ws with Upgrade: websocket and Connection: Upgrade. nginx drops both, because they are hop-by-hop headers, and sends Connection: close. The WebSocket library inside Grafana rejects that with 400 Bad Request, since there is no upgrade token in Connection. Add the map and the two headers, but leave Host alone, and the request reaches the origin check. Origin host grafana.example.com does not equal Host 127.0.0.1:3000. The fallback compares the Origin with root_url, here http://localhost:3000/. The scheme and host both differ, so the answer is 403 Forbidden. With Host $host the first comparison matches, and the socket opens.
Why page loads work and saving fails#
The CSRF check skips a request in four cases. These are read from the current csrf.go:
- No login cookie. If the request has no cookie named by
login_cookie_name(defaultgrafana_session), the check is skipped. Grafana 10.0.0 addedcsrf_always_check(defaultfalse) to run the check anyway. - A "safe" method.
HEAD,OPTIONSandTRACEare skipped.GETis not on this list. The source comment says it "may have side effects in some APIs". - A safe endpoint. Grafana code can register paths that skip the check. You cannot set these from the configuration file.
- No
Originheader. IfOriginis empty, the request passes.
The last rule is the one that explains the symptom. The Fetch Standard says when a browser adds Origin. It always does for cross-origin (CORS) requests and WebSockets. For other requests it adds Origin only when the method "is neither GET nor HEAD". Grafana's own pages fetch from the same origin, so their GET requests carry no Origin and skip the check.
So the login page loads, dashboards list, and settings pages open. Then any write fails: saving a dashboard, adding a data source, creating a folder. Panel queries fail too on current versions, because they go through POST /api/ds/query. That is why reports describe Grafana as working until they try to change something.
One more consequence: a curl test without an Origin header passes. To reproduce the 403 from a shell, send the cookie and an Origin header yourself.
What the check compares, exactly#
The details matter when you choose a fix.
- Host names only. Grafana strips the port from
Host. It takes only the host name fromOrigin. The scheme is never compared. SoHost: grafana.example.com:3000matchesOrigin: https://grafana.example.com. $hostand$http_hostboth pass. nginx's$hostis the request's host name without a port.$http_hostis the client'sHostheader as sent, port included. Since the port is ignored, either one passes the CSRF check. The Grafana tutorial uses$host; the Live documentation uses$http_host. The difference matters only for Live, below.X-Forwarded-Hostis not read unless you name it incsrf_additional_headers. A 2022 pull request to read it by default (grafana#46418) was closed without being merged.- An empty
Hostname always fails, even with trusted origins set.
This is not CORS. Adding Access-Control-Allow-Origin in nginx changes nothing, because Grafana's check reads request headers, not response headers. Mixing the two up can also add a second CORS header and break the page further; see the duplicate Access-Control-Allow-Origin error.
When the proxy cannot send the public Host#
Sometimes the Host must stay as something else. A common case is nginx in front of an internal load balancer that routes on Host. Another is an HTTPS backend whose certificate names only the internal host. From Grafana 9.1.0, two settings in [security] cover this. Both were absent in 9.0.0 and present in 9.1.0 in the source.
csrf_trusted_origins is a list of extra host names to accept. The documentation calls it a "list of additional allowed URLs". The code, though, compares each entry with the host name taken from Origin, with no scheme. So write bare host names. An entry like https://grafana.example.com never matches.
[security]
csrf_trusted_origins = grafana.example.com grafana.internal.example.comcsrf_additional_headers names request headers whose value Grafana treats as another Host. It strips the port and compares the host name with Origin. The usual value is X-Forwarded-Host, which Caddy and Traefik already send.
[security]
csrf_additional_headers = X-Forwarded-HostBoth lists are split on spaces, not commas. Both can be set from the environment as GF_SECURITY_CSRF_TRUSTED_ORIGINS and GF_SECURITY_CSRF_ADDITIONAL_HEADERS. On 8.3.5 to 9.0.x neither exists, and the only fix is the Host header.
If you use csrf_additional_headers behind nginx, set the header in nginx with proxy_set_header X-Forwarded-Host $host;. nginx does not send it by default. Which forwarded headers to trust in general is covered in X-Real-IP, X-Forwarded-Proto, Host and Port.
The second check: Grafana Live#
Grafana Live is the WebSocket connection that pushes updates to open dashboards. It runs at /api/live/ws and needs a signed-in user. It has its own origin check in live.go, separate from the CSRF code, and it is stricter.
The Live check passes a request in these cases, in this order:
- No
Originheader. allowed_originsis exactly*.- The
Originhost equals theHostheader, port included, ignoring case. - The
Originmatchesroot_url. The scheme must match. If theOriginhas no port, its host must equal the host inroot_url. If it has a port, host and port must both match. - The
Originmatches a pattern in[live] allowed_origins. This is a comma-separated list. Patterns are matched against the wholeOrigin, scheme included, and support*, as inhttps://*.example.com.
Point 3 is where $host and $http_host differ. Say users reach Grafana at https://grafana.example.com:8443. The Origin then carries :8443. With Host $host nginx sends no port, so point 3 fails. The request then depends on root_url being https://grafana.example.com:8443/. With Host $http_host point 3 passes on its own. On the default ports 80 and 443 the browser leaves the port out of Origin, and $host passes too.
A Live rejection looks different from the CSRF one. The response is 403 with the body Forbidden. Grafana logs a warning, Request Origin is not authorized, with the origin, host, appUrl and allowedOrigins it compared. The words "origin not allowed" appear only in a debug-level log line from the WebSocket library: websocket: request origin not allowed by Upgrader.CheckOrigin. In the browser you see a failed WebSocket in the console, and live updates stop. Saving still works.
The upgrade headers are a separate requirement. Without them the socket fails with 400 before any origin check runs. Why proxies drop those headers, and the idle timeouts that close sockets later, are covered in WebSockets through a reverse proxy. The Grafana tutorial also notes that many Live users can exhaust nginx's worker_connections; nginx worker_connections are not enough explains that limit.
Version history: the 8.3.5 wave#
The CSRF check arrived on 8 February 2022 in Grafana 8.3.5 and 7.5.15. It fixed CVE-2022-21703, a CSRF flaw affecting every version from 3.0-beta1. The advisory warned that behind a reverse proxy "you need to make sure that you are passing the original Host and Origin headers". It gave proxy_set_header Host $http_host; for nginx and ProxyPreserveHost on for Apache.
Many admins missed that note. Issue grafana#45117, "Unable to Create/Save Dashboard after v8.3.5 Update", was opened the next day. Grafana closed it as not a bug, pointing to the release note. Most of the forum threads that rank for this error date from that week.
The 8.3.5 code was simpler than today's. It split Host on the first colon, which broke Host values that are IPv6 literals, such as [::1]:3000. It had no trusted origin settings. The file is identical in 8.3.6 and 8.4.0, so upgrading to those changed nothing. In 8.5.0 the Host is parsed with a proper host and port split. The two trust settings came in 9.1.0, and csrf_always_check in 10.0.0.
Traefik, Caddy and Apache#
Traefik forwards the client's Host by default. The documentation says "By default, passHostHeader is true." It also forwards WebSocket upgrades with no extra settings. So a plain Traefik setup passes both checks once root_url is right. Setting passHostHeader to false stops Traefik forwarding the client's Host, and the mismatch returns.
Caddy also "passes through incoming headers, including Host" by default, and proxies WebSockets with no configuration. One exception catches Grafana users. Since Caddy 2.11.0, when the upstream is https://, Caddy sets Host to the upstream's address so it matches the TLS server name. If Grafana serves its own TLS and Caddy proxies to https://grafana:3000, Grafana sees Host: grafana:3000 and the CSRF check fails. Caddy's documented opt-out is header_up Host {hostport}. Or keep Caddy's default and set csrf_additional_headers = X-Forwarded-Host, since Caddy sets that header itself.
grafana.example.com {
reverse_proxy https://grafana:3000 {
header_up Host {hostport}
}
}With {hostport} the upstream TLS check uses Grafana's certificate, which must then be valid for the public name. Otherwise use the X-Forwarded-Host route.
Apache mod_proxy sends the backend's name unless ProxyPreserveHost On is set, as the CVE advisory says.
Which fix for which setup#
| Setup | What Grafana sees | Fix |
|---|---|---|
nginx with only proxy_pass | Host from proxy_pass | proxy_set_header Host $host; in every Grafana location |
nginx location with its own proxy_set_header lines | Host from proxy_pass (not inherited) | Repeat Host $host in that location |
| nginx, saves work, live updates stop | Live Origin fails points 3 and 4 | Upgrade map, Host on /api/live/, correct root_url |
| Public port is not 80 or 443 | Live: Host $host lacks the port | Host $http_host, or root_url with the port |
Apache mod_proxy | Backend name as Host | ProxyPreserveHost On |
Traefik, passHostHeader true (default) | Client Host | Nothing for CSRF; set root_url |
Caddy to an https:// Grafana (2.11.0+) | Upstream address as Host | header_up Host {hostport}, or csrf_additional_headers = X-Forwarded-Host |
Proxy must send an internal Host (Grafana 9.1.0+) | Internal name as Host | csrf_trusted_origins with the public host name, plus root_url for Live |
Grafana served under /grafana/ | Same checks, different paths | The fixes above, plus root_url with the path |
For the sub-path row, Grafana's tutorial uses location /grafana/ and location /grafana/api/live/. root_url must end in the sub-path. Then either strip the prefix in nginx or set serve_from_sub_path = true. The general pattern is in running an app under a subpath.
Failure modes#
proxy_set_header Host $host;was added and saves still fail. Look for a second location, often/api/live/or/api/, that sets its own headers. It does not inheritHostfrom the server block. Check also for a second proxy in front, such as a cloud load balancer, that rewritesHostagain.csrf_trusted_originsis set and nothing changed. Check for a scheme in the entries. Removehttps://and write host names. Check also that the entries are separated by spaces, and that the version is 9.1.0 or later.- Live shows
Request Origin is not authorizedwithappUrlset tohttp://localhost:3000/.root_urlwas never set. Set it to the public URL, scheme included. - The browser console shows the WebSocket failed with 400. The upgrade headers did not reach Grafana. Add the map and the
UpgradeandConnectionheaders to the/api/live/location. - Grafana answers 400 instead of 403 on every write. The current CSRF code returns 400 when it cannot parse
Host,Originor a header named incsrf_additional_headers. Check those values for a stray scheme or path. - Saves fail when users browse to Grafana by IPv6 address, on an old release. The 8.3.5 to 8.4.0 code split
Hoston the first colon, so[2001:db8::1]:3000became[2001and never matched. Upgrade, or use a host name.
What this page does not cover#
The checks described here are read from Grafana's source on the main branch and at the tagged releases named above. They were not run against a live Grafana. Grafana Cloud, Amazon Managed Grafana and other hosted offerings run their own proxies, and their settings are not covered. Auth proxy, OAuth and SAML logins have their own redirect and callback rules that depend on root_url; this page covers only the origin checks. The Caddy https:// behaviour is from the Caddy 2.11 documentation. Older Caddy versions passed the client's Host to HTTPS upstreams too. Any setting that widens the trusted list weakens the protection CVE-2022-21703 added. allowed_origins = * turns off the Live check entirely.
Frequently asked questions#
Why does Grafana say origin not allowed when I save a dashboard?#
Because the Host header Grafana receives names a different server than the browser's Origin header. Behind nginx, Host defaults to the proxy_pass address. Add proxy_set_header Host $host; to every location that proxies to Grafana.
Why do Grafana dashboards load but saving gives a 403?#
Browsers send an Origin header on POST, PUT and DELETE, but not on same-origin GET requests. Grafana skips its CSRF check when Origin is missing, so reads pass and writes are checked. Fix the Host header and both work.
Should I use $host or $http_host for Grafana?#
Either one passes the CSRF check, because Grafana ignores the port there. For Grafana Live on a non-standard public port, $http_host is safer, because it keeps the port the browser sent. On ports 80 and 443 they behave the same.
What is the format of csrf_trusted_origins?#
A space-separated list of bare host names, such as grafana.example.com. Grafana compares each entry with the host name from the Origin header, so an entry that includes https:// never matches. The setting exists from Grafana 9.1.0.
Does adding CORS headers fix Grafana origin not allowed?#
No. Grafana's check compares two request headers, Origin and Host. CORS response headers such as Access-Control-Allow-Origin play no part in it.
Why does Grafana Live fail behind nginx when saving works?#
Live runs a separate origin check on its WebSocket. It needs the Upgrade and Connection headers forwarded, and an Origin that matches Host exactly or matches root_url. Grafana logs failures as Request Origin is not authorized.
Did Grafana 8.3.6 fix origin not allowed?#
No. The CSRF code is identical in 8.3.5, 8.3.6 and 8.4.0. The error is the intended result of the CVE-2022-21703 fix when a proxy rewrites Host. The fix is in the proxy configuration.
Primary sources#
Every normative claim on this page is checked against the specification or the vendor documentation listed here. Where behaviour is version dependent, the version is named in the text.
- Grafana source, pkg/middleware/csrf/csrf.go (main)
- Grafana source, pkg/middleware/csrf.go at v8.3.5 (first CSRF check)
- Grafana source, pkg/services/live/live.go (Live origin check)
- Grafana source, conf/defaults.ini (root_url, login_cookie_name, allowed_origins)
- Grafana documentation, Configure Grafana (root_url, csrf_trusted_origins, csrf_additional_headers, csrf_always_check, allowed_origins)
- Grafana tutorial, Run Grafana behind a reverse proxy
- Grafana documentation, Set up Grafana Live (request origin check)
- Grafana security advisory GHSA-cmf4-h3xc-jw8w (CVE-2022-21703)
- Grafana issue 45117, Unable to Create/Save Dashboard after v8.3.5 Update
- Centrifuge v0.39.0 source, internal/websocket/server.go (Upgrader errors)
- nginx ngx_http_proxy_module (proxy_set_header defaults)
- WHATWG Fetch Standard, append a request Origin header
- Traefik HTTP services, load balancer (passHostHeader)
- Caddy documentation, reverse_proxy directive (header defaults, HTTPS Host)
Found something wrong, or behaviour that differs on your version? Report it with the version number and a primary source. Anything substantive is fixed in the page and logged on the corrections page. See editorial standards for how pages are researched, sourced and reviewed.
More in reverse proxy configuration#
- Apache 421 Misdirected Request after 2.4.64: the fix
- Caddy reverse proxy
- Custom error page when the backend is down: nginx and more
- Envoy listeners, routes and clusters
- nginx auth_request vs ForwardAuth: why the 302 fails
- gRPC through a reverse proxy
- HAProxy configuration for HTTP reverse proxying
- Health checks and upstream failover
- ingress-nginx retired: proxy annotation equivalents
- nginx proxy_pass and the trailing slash
- nginx redirect drops the port or scheme: which fix
- nginx vs HAProxy vs Envoy vs Caddy vs Traefik
- Sticky sessions and session affinity
- Traefik routers, services and middlewares