Traefik routers, services and middlewares

Traefik splits a reverse proxy into five objects: an entryPoint is a listening socket, a router binds a rule to a service and an optional middleware chain, a service is the load balancer over backend servers, a middleware transforms the request or response, and a provider is where the routers, services and middlewares come from (Docker labels, Kubernetes CRDs, a watched file, Consul, and others). Static configuration (entryPoints, providers, certificate resolvers) is set by CLI flags, environment variables or traefik.yml and requires a restart. Dynamic configuration (routers, services, middlewares) is watched and applied live, with no reload signal and no dropped connections.

Every dynamic object is namespaced by its provider. A middleware defined by Docker labels is name@docker, one from the file provider is name@file, and Traefik's own dashboard API is the built-in service api@internal. Referencing a middleware from a different provider without the suffix is one of the most common configuration mistakes.

Rule syntax, and what changed in v3#

Rules are boolean expressions over matchers, combined with &&, || and parentheses. Traefik v3 also supports ! for negation. The canonical form is:

Host(`api.example.com`) && PathPrefix(`/v1/`)

Traefik v3 tightened several matchers. The changes below are the ones to check first when a v2 configuration stops matching after an upgrade.

The same rules written out, v2 first and v3 second:

HostRegexp(`{sub:[a-z]+}.example.com`) && Query(`env=prod`) && Headers(`X-Env`, `prod`)
HostRegexp(`^[a-z]+\.example\.com$`) && Query(`env`, `prod`) && Header(`X-Env`, `prod`)

Traefik v3 keeps an escape hatch: core.defaultRuleSyntax (and a per-router ruleSyntax option) lets you keep evaluating rules with v2 semantics while you migrate them one at a time. Use it to decouple the binary upgrade from the rule rewrite rather than doing both in one change window. Confirm the exact matcher list against the migration guide for your version before you rely on any single line above.

Middlewares are chained per router, in order#

A middleware is a definition, not an attachment. It only runs when a router lists it, and the list order is the execution order for the request (and the reverse order for the response). Two routers pointing at the same service can have completely different chains, which is how you apply authentication to /admin without touching /.

Order matters in ways that are easy to get wrong:

• stripPrefix before forwardAuth means the auth service sees the rewritten path. If the auth service makes decisions on the path, put forwardAuth first.
• redirectScheme or redirectRegex should come first in the chain. Anything after a redirect middleware that returns a 301 never executes for that request.
• rateLimit placed after an expensive forwardAuth still costs you the auth call on every rejected request.

Providers: labels, CRDs and files#

The Docker provider reads labels from running containers, resolves the container's IP on a shared Docker network, and builds routers and services from them. The Kubernetes CRD provider reads IngressRoute, Middleware, ServersTransport and TLSOption custom resources, which expose the same object model with none of the string-parsing that labels require. The file provider watches a YAML or TOML file or directory and is the right place for anything not tied to a workload: TLS options, shared middlewares, static backends outside the container platform.

Mixing providers is normal and supported. A typical layout is Docker labels for application routers, plus a file provider for TLSOption definitions and shared middlewares that the labels reference as secure-headers@file.

Worked example: docker-compose with labels#

services:
  traefik:
    image: traefik:v3.3
    command:
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --providers.file.directory=/etc/traefik/dynamic
      - --entryPoints.web.address=:80
      - --entryPoints.websecure.address=:443
      - --entryPoints.web.http.redirections.entryPoint.to=websecure
      - --entryPoints.web.http.redirections.entryPoint.scheme=https
      - --entryPoints.websecure.forwardedHeaders.trustedIPs=10.0.0.0/8
      - --certificatesresolvers.le.acme.email=ops@example.com
      - --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.le.acme.httpchallenge=true
      - --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
      - --accesslog=true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./dynamic:/etc/traefik/dynamic:ro
      - letsencrypt:/letsencrypt
    networks: [edge]

  api:
    image: example/api:1.4
    networks: [edge]
    labels:
      - traefik.enable=true
      - traefik.docker.network=edge
      - traefik.http.routers.api.rule=Host(`example.com`) && PathPrefix(`/api/`)
      - traefik.http.routers.api.entrypoints=websecure
      - traefik.http.routers.api.tls.certresolver=le
      - traefik.http.routers.api.middlewares=api-strip@docker,api-ratelimit@docker
      - traefik.http.middlewares.api-strip.stripprefix.prefixes=/api
      - traefik.http.middlewares.api-ratelimit.ratelimit.average=50
      - traefik.http.middlewares.api-ratelimit.ratelimit.burst=100
      - traefik.http.services.api.loadbalancer.server.port=8080
      - traefik.http.services.api.loadbalancer.healthcheck.path=/healthz

volumes:
  letsencrypt:

networks:
  edge:

Observable behaviour: https://example.com/api/users reaches the container as GET /users on port 8080, after the rate limit and prefix strip run in that order. Plain http://example.com/api/users gets a 301 to HTTPS from the entryPoint redirection, and the ACME HTTP challenge still completes because Traefik's own challenge router on the web entryPoint outranks the redirect. Remove traefik.docker.network=edge while the container is attached to two networks and Traefik may pick the wrong IP, producing a 502 with no useful log line on the application side.

loadbalancer.server.port is not optional guesswork: Traefik needs it whenever the container exposes zero or more than one port. Omitting it in the multi-port case is a silent misroute to whichever port Traefik picked.

TLS: entryPoint defaults versus router TLS#

There are two places TLS gets configured and they are not equivalent.

• Router level: traefik.http.routers.api.tls=true and ...tls.certresolver=le. This router terminates TLS and requests a certificate for the domains in its Host() rule.
• EntryPoint level: entryPoints.websecure.http.tls.certResolver=le in static config. This supplies a default for every router on that entryPoint that does not set its own.

A router attached to websecure with no TLS configuration at either level still accepts the connection, and Traefik serves its built-in self-signed certificate. The symptom is a browser or curl error naming TRAEFIK DEFAULT CERT, which is the single most diagnostic string in Traefik TLS debugging: it means the request reached Traefik, TLS was attempted, and no resolver or configured certificate covered the requested SNI. Compare this against the broader options in TLS termination, passthrough and re-encryption.

The ACME account key and every issued certificate live in one JSON file, by default acme.json. Three rules about it:

1. It must be on a persistent volume. Rebuilding a container without persisting it means re-issuing every certificate, and Let's Encrypt enforces rate limits per registered domain that a redeploy loop will hit.
2. It must be mode 0600. Traefik refuses to use a file with broader permissions and logs an error about the permissions being too open.
3. File storage assumes a single Traefik instance. Two replicas sharing one volume will fight over it. Highly available ACME needs a distributed store, which the open-source distribution does not provide, so the usual pattern is one certificate-terminating instance or certificates issued outside Traefik.

Trusting the proxy in front of Traefik#

If Traefik sits behind a cloud load balancer or a CDN, the client IP it sees is the load balancer's. Two settings fix that, per entryPoint:

• forwardedHeaders.trustedIPs: a list of CIDRs. When the immediate peer is in the list, Traefik preserves incoming X-Forwarded-For, X-Forwarded-Proto and friends and appends to them. When it is not, Traefik overwrites them with its own view, which is exactly the behaviour you want against spoofing. forwardedHeaders.insecure=true trusts everyone and should not be used on an internet-facing entryPoint. The evaluation order is covered in configuring trusted proxies, and you can check a chain against the client IP resolver.
• proxyProtocol.trustedIPs: enables PROXY protocol parsing on that entryPoint, for an L4 load balancer that preserves the client address out of band. This is the better option when TLS is passed through rather than terminated upstream, since there are no headers to rewrite. See the PROXY protocol and the PROXY protocol decoder.

Enabling proxyProtocol on an entryPoint that receives plain TCP breaks every connection, because Traefik expects a PROXY header that is not there. Enable it on a dedicated entryPoint and cut traffic over, rather than flipping it on a live port.

Failure modes#

The dashboard case deserves emphasis. api.insecure is convenient in a compose file on a laptop and dangerous the moment that compose file is deployed, because the dashboard reveals every router, service, backend address and middleware, which is a complete map of the internal network. The secure form is a normal router on websecure with a Host() rule, a basicAuth middleware, and service api@internal, remembering that the dashboard path needs a trailing slash (/dashboard/).

Frequently asked questions#

Why does Traefik return 404 page not found?#

Because no router matched the request. In order of likelihood: the container is missing traefik.enable=true while exposedByDefault is false, the rule does not match the Host or path actually sent, the router is attached to a different entryPoint than the port that received the request, or a referenced middleware failed to resolve and disabled the router. The dashboard's router list shows which of these it is.

What changed in Traefik v3 rule syntax?#

The main changes are that PathPrefix no longer accepts regular expressions (use PathRegexp), HostRegexp takes a plain Go regular expression instead of named-group syntax and needs its own ^ and $ anchors, Query takes a key and value as separate arguments, and Headers and HeadersRegexp were renamed to Header and HeaderRegexp. Setting core.defaultRuleSyntax to v2 lets you upgrade the binary before rewriting rules.

How does Traefik decide which router wins when two rules match?#

Each router gets a default priority equal to the length of its rule string, and the highest priority is evaluated first, so longer and more specific rules naturally win. An explicit priority value on a router overrides the default, and the higher number wins.

Where does Traefik store Let's Encrypt certificates?#

In the file named by certificatesresolvers.<name>.acme.storage, conventionally /letsencrypt/acme.json. It holds the ACME account key and every issued certificate, must be on a volume that survives redeploys, and must have 0600 permissions or Traefik will refuse to use it.

Do I need to configure anything for WebSockets in Traefik?#

No. Traefik proxies WebSocket upgrades through a normal HTTP router without extra configuration. If a WebSocket connection drops at a fixed interval, look at the entryPoint's respondingTimeouts and at any load balancer in front, not at Traefik's routing.

How do I get the real client IP behind a load balancer?#

Set forwardedHeaders.trustedIPs on the entryPoint to the CIDR of the load balancer, so Traefik preserves the incoming X-Forwarded-For instead of overwriting it. If the upstream speaks PROXY protocol, use proxyProtocol.trustedIPs on a dedicated entryPoint instead.

Is exposing the Docker socket to Traefik safe?#

Read-only access to the Docker socket still grants effective root on the host to anything that can reach it, so treat the Traefik container as a privileged component. Reduce the blast radius by running a socket proxy that only permits the container and event endpoints Traefik needs, or by using the Kubernetes CRD or file providers instead.