traefik: Let's encrypt TLS Challenge failing when behind a traefik TCP Router
Welcome!
- Yes, I’ve searched similar issues on GitHub and didn’t find any.
- Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.
What did you do?
I have multiple servers in my Network and one public static IP. The firewall is forwarding all traffic to Server 1 on ports 80 and 443. Server 1 has multiple TCP Router in place to send the traffic to Server 2 and Server 3 which look live this:
tcp:
routers:
server_2:
entrypoints:
- websecure
rule: HostSNIRegexp(`^.+\.server2\.subdomain\.example\.com$`)
tls:
passthrough: true
service: server_2
services:
server_2:
loadBalancer:
proxyProtocol:
version: 2
servers:
- address: 172.16.10.12:443
and
tcp:
routers:
server_3:
entrypoints:
- websecure
rule: HostSNIRegexp(`^.+\.server3\.subdomain\.example\.com$`)
tls:
passthrough: true
service: server_3
services:
server_3:
loadBalancer:
proxyProtocol:
version: 2
servers:
- address: 172.16.10.13:443
On Server 2 and Server 3 lets’s encrypt is configures like this:
certificatesResolvers:
letsencrypt-tls:
acme:
email: le@example.com
storage: /letsencrypt/acme.json
tlsChallenge: true
If try to create a Certificate on Server 2 or Server 3 for example www.app.server2.subdomain.example.com and app.server2.subdomain.example.com it always fails for the domain with the www prefix. With a error 400 code.
Edit
it looks like all certification request are failing not only the one with the www prefix, which was reported because it was the first to process.
On Server 1 which is facing directly to the internet the problem is not present.
What did you see instead?
What version of Traefik are you using?
Version: 3.0.0 Codename: beaufort Go version: go1.22.2 Built: 2024-04-29T14:25:59Z OS/Arch: linux/amd64
What is your environment & configuration?
Server 1
log:
level: INFO
accessLog:
filePath: /var/log/traefik/access.log
format: json
bufferingSize: 100
fields:
defaultMode: keep
headers:
defaultMode: keep
api:
dashboard: true
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false
network: external
file:
directory: /etc/traefik/rules
watch: true
entryPoints:
web:
address: :80
forwardedHeaders:
trustedIPs:
- 172.16.10.11
http:
middlewares:
- crowdsec-bouncer@file
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: :443
forwardedHeaders:
trustedIPs:
- 172.16.10.11
http:
middlewares:
- crowdsec-bouncer@file
http3:
advertisedPort: 443
certificatesResolvers:
letsencrypt-tls:
acme:
email: le@example.com
storage: /letsencrypt/acme.json
tlsChallenge: true
metrics:
influxDB2:
address: https://influxdb.sg.t-k-f.ch
token: ***
org: ***
bucket: traefik
addEntryPointsLabels: true
addRoutersLabels: true
addServicesLabels: true
additionalLabels:
host: traefik.example.com
Server 2
log:
level: INFO
accessLog:
filePath: /var/log/traefik/access.log
format: json
bufferingSize: 100
fields:
defaultMode: keep
headers:
defaultMode: keep
api:
dashboard: true
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false
network: external
file:
directory: /etc/traefik/rules
watch: true
entryPoints:
websecure:
address: :443
forwardedHeaders:
trustedIPs:
- 172.16.10.12
proxyProtocol:
trustedIPs:
- 172.16.10.11
http:
middlewares:
- crowdsec-bouncer@file
http3: true
certificatesResolvers:
letsencrypt-tls:
acme:
email: le@example.com
storage: /letsencrypt/acme.json
tlsChallenge: true
metrics:
influxDB2:
address: https://influxdb.sg.t-k-f.ch
token: ***
org: ***
bucket: traefik
addEntryPointsLabels: true
addRoutersLabels: true
addServicesLabels: true
additionalLabels:
host: traefik.example.com
If applicable, please paste the log output in DEBUG level
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 6
- Comments: 28 (8 by maintainers)
We think this was caused by #10536. TCP routers marked as passthrough should pass through all traffic to the backend service, including ACME TLS.
👋 @rtribotte Just letting you know that I’ve tested your dev branch and the passthrough is working like a charm! Well done!
Hello @CybotTM @acul009, we discussed about the option among maintainers during the triage session, and we decided to bring also the static option, so I am going to rework #10981.
Looks like we have the same issue for 2.11.2
While I understand the intent when using TLS-termination, is it really intended to block the downstream challenge verification when using tls-passthrough? I‘d feel like the entire point of the tls-passthrough would be not to interfere and make exactly these things possible.
Either way, I‘d simply be happy to have the option back. Calling it insecure doesn‘t really bother me - although I‘d be interested in the possible attack vector from this. When using passthrough the downstream target already has a valid certificate, no?
Thank you for your hard work on traefik 😃
So you’re blocking this because in theory a compromised downstream application that is not supposed to have access to TLS certificates should not be able to generate a valid certificate that could then be used for additional attacks?
If I got this wrong, I’d be interested to hear more about your concerns. As far as I understand, the TLS challenge is already only handed through to the routers which actually match the SNI. Which means I can only try to obtain certificates for my own domain, just like the HTTP challenge. And that also only works then TLS passthrough is on. When it’s off the TLS tunnel is terminated before reaching the target, thus blocking of the alpn request, right?
In theory yes, but as I’ve said: my traefik configurations share their configuration and this would make matters more complicated than they already are. This also breaks apart if I had 3 traefik instances in a row.
I think the best option would still be the
insecureoption. Although I’d name it more explicitly. Maybeallow_acme_challengeorpassthrough_acme_challenge?In a full example this could look like this:
If passthrough is set to
trueyou just fall back to your default options settingallow_acme_challengetofalse.If that is to much work, how about adding a static option or environment variable to bypass the check from #10536 for all routers?
Maybe something like
tls_passthrough_allow_acme_challenge?People like me could just enable this option to get the old behaviour back - since I’m not in a multi-tentant environment I won’t have any problems.
Edit: The HTTP Option also doesn’t work when only passing through Port 443 from the internet, which I personally like to do if I can - less attack surface is always a win.
Not really, since my configuration is shared between pretty much all my traefik instances. The insecure flag for passthing through tls challenges as well would actually better suite my needs.
I’m using traefik both as a reverse proxy for my docker application as well as a network wide reverse proxy to allow things like a NAS to obtain a Let’s encrypt Certificate.
To me the TLS-Passthrough is one of traefiks killer-features. It’s just so convinient to share Port 443 without much reconfiguration needed on third party vm images. Some applications just assume they will handle TLS themselves and I actually like having those connections encrypted, even if it is inside my “secure” local network.
Here’s a little diagram about how I use traefik:
To make this easier I actually built myself a very basic Web-UI to configure routes so I don’t have to use nano in the command line.
https://github.com/Rahn-IT/traefik-gui (Warning: very janky)
I really love traefik because it automagically handles my docker routing with just a few labels and I can easily share my external Port 443 without breaking the TLS encryption inside my local network - I mean how cool is that? No other reverse proxy combines this feature set with such elegance as traefik.
This Behaviour was introduced with v3.0.0-rc4 downgrading to v3.0.0-rc3 resolves the problem