traefik: Let's encrypt TLS Challenge failing when behind a traefik TCP Router

Welcome!

  • Yes, I’ve searched similar issues on GitHub and didn’t find any.
  • Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.

What did you do?

I have multiple servers in my Network and one public static IP. The firewall is forwarding all traffic to Server 1 on ports 80 and 443. Server 1 has multiple TCP Router in place to send the traffic to Server 2 and Server 3 which look live this:

tcp:
    routers:
        server_2:
            entrypoints:
                - websecure
            rule: HostSNIRegexp(`^.+\.server2\.subdomain\.example\.com$`)
            tls:
                passthrough: true
            service: server_2
    services:
        server_2:
            loadBalancer:
                proxyProtocol:
                    version: 2
                servers:
                    - address: 172.16.10.12:443

and

tcp:
    routers:
        server_3:
            entrypoints:
                - websecure
            rule: HostSNIRegexp(`^.+\.server3\.subdomain\.example\.com$`)
            tls:
                passthrough: true
            service: server_3
    services:
        server_3:
            loadBalancer:
                proxyProtocol:
                    version: 2
                servers:
                    - address: 172.16.10.13:443

On Server 2 and Server 3 lets’s encrypt is configures like this:

certificatesResolvers:
    letsencrypt-tls:
        acme:
            email: le@example.com
            storage: /letsencrypt/acme.json
            tlsChallenge: true 

If try to create a Certificate on Server 2 or Server 3 for example www.app.server2.subdomain.example.com and app.server2.subdomain.example.com it always fails for the domain with the www prefix. With a error 400 code.

Edit it looks like all certification request are failing not only the one with the www prefix, which was reported because it was the first to process.

On Server 1 which is facing directly to the internet the problem is not present.

What did you see instead?


What version of Traefik are you using?

Version: 3.0.0 Codename: beaufort Go version: go1.22.2 Built: 2024-04-29T14:25:59Z OS/Arch: linux/amd64

What is your environment & configuration?

Server 1

log:
    level: INFO

accessLog:
    filePath: /var/log/traefik/access.log
    format: json
    bufferingSize: 100
    fields:
        defaultMode: keep
        headers:
            defaultMode: keep

api:
    dashboard: true

providers:
    docker:
        endpoint: "unix:///var/run/docker.sock"
        exposedByDefault: false
        network: external
    file:
        directory: /etc/traefik/rules
        watch: true

entryPoints:
    web:
        address: :80
        forwardedHeaders:
            trustedIPs:
                - 172.16.10.11
        http:
            middlewares:
                - crowdsec-bouncer@file
            redirections:
                entryPoint:
                    to: websecure
                    scheme: https
    websecure:
        address: :443
        forwardedHeaders:
            trustedIPs:
                - 172.16.10.11
        http:
            middlewares:
                - crowdsec-bouncer@file
        http3:
            advertisedPort: 443

certificatesResolvers:
    letsencrypt-tls:
        acme:
            email: le@example.com
            storage: /letsencrypt/acme.json
            tlsChallenge: true 

metrics:
    influxDB2:
        address: https://influxdb.sg.t-k-f.ch
        token: ***
        org: ***
        bucket: traefik
        addEntryPointsLabels: true
        addRoutersLabels: true
        addServicesLabels: true
        additionalLabels:
            host: traefik.example.com

Server 2

log:
    level: INFO

accessLog:
    filePath: /var/log/traefik/access.log
    format: json
    bufferingSize: 100
    fields:
        defaultMode: keep
        headers:
            defaultMode: keep

api:
    dashboard: true

providers:
    docker:
        endpoint: "unix:///var/run/docker.sock"
        exposedByDefault: false
        network: external
    file:
        directory: /etc/traefik/rules
        watch: true

entryPoints:
    websecure:
        address: :443
        forwardedHeaders:
            trustedIPs:
                - 172.16.10.12
        proxyProtocol:
            trustedIPs:
                - 172.16.10.11
        http:
            middlewares:
                - crowdsec-bouncer@file
        http3: true

certificatesResolvers:
    letsencrypt-tls:
        acme:
            email: le@example.com
            storage: /letsencrypt/acme.json
            tlsChallenge: true 

metrics:
    influxDB2:
        address: https://influxdb.sg.t-k-f.ch
        token: ***
        org: ***
        bucket: traefik
        addEntryPointsLabels: true
        addRoutersLabels: true
        addServicesLabels: true
        additionalLabels:
            host: traefik.example.com

If applicable, please paste the log output in DEBUG level

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 28 (8 by maintainers)

Most upvoted comments

We think this was caused by #10536. TCP routers marked as passthrough should pass through all traffic to the backend service, including ACME TLS.

👋 @rtribotte Just letting you know that I’ve tested your dev branch and the passthrough is working like a charm! Well done!

Hello @CybotTM @acul009, we discussed about the option among maintainers during the triage session, and we decided to bring also the static option, so I am going to rework #10981.

Looks like we have the same issue for 2.11.2

While I understand the intent when using TLS-termination, is it really intended to block the downstream challenge verification when using tls-passthrough? I‘d feel like the entire point of the tls-passthrough would be not to interfere and make exactly these things possible.

Either way, I‘d simply be happy to have the option back. Calling it insecure doesn‘t really bother me - although I‘d be interested in the possible attack vector from this. When using passthrough the downstream target already has a valid certificate, no?

Thank you for your hard work on traefik 😃

@CybotTM We introduced this breaking change as a fix because of security concerns we had regarding potential hijacking of certificate negotiation in a multi-tenant environment. I have to admit that we probably moved too fast on this issue, because in the end we came to the conclusion that Traefik should not be considered secure “out of the box” when used to expose routes of multiple clients in a multi-tenant environment, simply because Traefik does not provide/enforce client isolation.

So you’re blocking this because in theory a compromised downstream application that is not supposed to have access to TLS certificates should not be able to generate a valid certificate that could then be used for additional attacks?

If I got this wrong, I’d be interested to hear more about your concerns. As far as I understand, the TLS challenge is already only handed through to the routers which actually match the SNI. Which means I can only try to obtain certificates for my own domain, just like the HTTP challenge. And that also only works then TLS passthrough is on. When it’s off the TLS tunnel is terminated before reaching the target, thus blocking of the alpn request, right?

@acul009 In the setup you shared, it seems doable not to declare the TLS challenge resolver on the “gateway” Network Traefik instance, WDYT?

In theory yes, but as I’ve said: my traefik configurations share their configuration and this would make matters more complicated than they already are. This also breaks apart if I had 3 traefik instances in a row.

I think the best option would still be the insecure option. Although I’d name it more explicitly. Maybe allow_acme_challenge or passthrough_acme_challenge?

In a full example this could look like this:

tcp:
    routers:
        server_3:
            entrypoints:
                - websecure
            rule: HostSNIRegexp(`^.+\.server3\.example\.com$`)
            tls:
                passthrough:
                    allow_acme_challenge: true
            service: server_3

If passthrough is set to true you just fall back to your default options setting allow_acme_challenge to false.


If that is to much work, how about adding a static option or environment variable to bypass the check from #10536 for all routers?

Maybe something like tls_passthrough_allow_acme_challenge ?

People like me could just enable this option to get the old behaviour back - since I’m not in a multi-tentant environment I won’t have any problems.

Edit: The HTTP Option also doesn’t work when only passing through Port 443 from the internet, which I personally like to do if I can - less attack surface is always a win.

@acul009 Thanks for your feedback. Are you using a TLS resolver with Traefik? We are wondering if simply allowing the TLS challenge passthrough when no TLS resolver is configured is enough in your situation.

Not really, since my configuration is shared between pretty much all my traefik instances. The insecure flag for passthing through tls challenges as well would actually better suite my needs.

I’m using traefik both as a reverse proxy for my docker application as well as a network wide reverse proxy to allow things like a NAS to obtain a Let’s encrypt Certificate.

To me the TLS-Passthrough is one of traefiks killer-features. It’s just so convinient to share Port 443 without much reconfiguration needed on third party vm images. Some applications just assume they will handle TLS themselves and I actually like having those connections encrypted, even if it is inside my “secure” local network.

Here’s a little diagram about how I use traefik:

flowchart LR
    
internet[Internet]
network-traefik[Network Traefik]
internet -->|Port 443 / 80| network-traefik
network-traefik -->|Port 443\nTLS-Passthrough| docker-traefik1[Docker Traefik 1]
network-traefik -->|Port 443\nTLS-Passthrough| docker-traefik2[Docker Traefik 2]
network-traefik -->|Port 443\nTLS-Passthrough| docker-traefik3[Docker Traefik 3]

network-traefik -->|Port 80\nPathprefix: /.well-known/acme-challenge| nas[NAS]
network-traefik -->|Port 80 / 443\nTLS-Passthrough\nPathprefix: /.well-known/acme-challenge| non-docker-host[Non Docker Host]

To make this easier I actually built myself a very basic Web-UI to configure routes so I don’t have to use nano in the command line.

https://github.com/Rahn-IT/traefik-gui (Warning: very janky)

I really love traefik because it automagically handles my docker routing with just a few labels and I can easily share my external Port 443 without breaking the TLS encryption inside my local network - I mean how cool is that? No other reverse proxy combines this feature set with such elegance as traefik.

This Behaviour was introduced with v3.0.0-rc4 downgrading to v3.0.0-rc3 resolves the problem