traefik: Incorrect X-FORWARDED-PROTO, X-FORWARDED-PORT for external TLS termination and proxy protocol
Welcome!
- Yes, I’ve searched similar issues on GitHub and didn’t find any.
- Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.
What did you do?
I think I’ve discovered an edge case (or myself not completely understanding the docs) that relates to setting the X-FORWARDED-PROTO and -PORT when proxy protocol is enabled.
I was able to reproduce the bug with this minimal setup:
- my browser @
80.1.2.3- connects to the public ip of the external load balancer
- an external load balancer service, does TLS termination,
10.11.0.200- wireshark shows that it correctly sets
X-Forwarded-Proto https,X-Forwarded-Port 443 - proxies to Traefik on port 8181, proxy protocol enabled
- wireshark shows that it correctly sets
- Traefik
--entrypoints.web.address=:8181/tcp --entrypoints.web.forwardedHeaders.trustedIPs=10.11.0.200 --entrypoints.web.proxyProtocol.trustedIPs=10.11.0.200- service that connects to the sample app
- Sample App, simply dumps all request headers in the response
I’d expect the sample app to receive those headers:
x-forwarded-for 80.1.2.3, 10.11.0.200
x-forwarded-host hostname.example
x-forwarded-port 443
x-forwarded-proto https
x-forwarded-server traefik-node-name
x-real-ip 80.1.2.3
What did you see instead?
Traefik forwards the headers, except for X-FORWARDED-PROTO and -PORT. The sample app receives:
x-forwarded-for 80.1.2.3
x-forwarded-host hostname.example
x-forwarded-port 80
x-forwarded-proto http
x-forwarded-server traefik-node-name
x-real-ip 80.1.2.3
I can get the expected x-forwarded-proto and x-forwarded-port by:
- disabling proxy protocol on both Traefik and the external load balancer
- setting
forwardedHeaders.trustedIPs=0.0.0.0/0(orforwardedHeaders.insecure=true) - setting
forwardedHeaders.trustedIPs=80.1.2.3(my client IP)
None of this is really suitable for production.
As it seems, traefik uses the client ip as the source ip when determining whether it is a trusted ip for header forwarding, when proxy protocol is enabled. I’d still expect traefik to use the actual source ip (load balancer).
What version of Traefik are you using?
Version: 2.9.6 Codename: banon Go version: go1.19.4 Built: 2022-12-07T14:17:58Z OS/Arch: linux/amd64
What is your environment & configuration?
See description and cases above
If applicable, please paste the log output in DEBUG level
No response
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 7
- Comments: 16 (8 by maintainers)
Hi, sorry for being quiet, we ended up re-structuring our setup to mitigate this issue and make some other changes, I wasn’t in touch with this issue since then and didn’t have a setup to reproduce it.
As far as I remember: we opted to enable ProxyProtocol on our load balancer for various reasons (mainly due to having non-HTTP services where we needed the the real source IP). AFAIK ProxyProtocol only conveys information about the original remote IP and Port. The problem here was that the connection between the load balancer and traefik was pure HTTP (no HTTPS), which caused traefik to set the
X-Forwarded-Protoheader by itself tohttpin this case (causing issues with the end application) even though the external load balancer did SSL termination. We couldn’t find a quick way to ensure thatX-Forwarded-Protowas either not present or correctly set tohttps, raising the need for combined ProxyProtocol and X-Forwarded*-Headers from the load balancer.However, I don’t mind any outcome at this point, it is not relevant for us anymore. Maybe a note in the docs (
X-Forwarded* and ProxyProtocol don't work well together) would suffice.Regards, Peter
Thanks for the tip @polarathene, it is helping me a lot!
Hi @mpl,
sorry for the confusion - I wrote this issue after a long day of debugging symptoms that eventually resulted in this specific problem and had way too many snippets in my clipboard.
Yes, it was a typo, I changed it in the initial comment
I didn’t really think about this (yet), for now I just cared about
protoandport. It doesn’t really matter for my setup if the external load balancer (10.11.0.200) is included inX-Forwarded-Forsince it’s part of a trusted environment. However, I do agree that it probably should be part ofX-Forwarded-Forin general.I checked
X-Forwarded-Foronce again:X-Forwarded-For: 80.1.2.3, 10.11.0.200(as expected)forwardedHeaders.insecure=true:X-Forwarded-For: 80.1.2.3, 80.1.2.3(duplicate client ip)forwardedHeaders.trustedIPs=10.11.0.200:X-Forwarded-For 80.1.2.3(along with port 80 and http protocol)Cheers!
I am a bit late, but you can inject your own PROXY protocol header via plain HTTP requests:
Within a Docker Compose config, you can quite easily setup containers all within the same network for better IP assignments and minimizing gotchas.
For example since you sent the
curlrequest tolocalhost:4443, Docker routed that through the subnet gateway IP172.18.100.1(although I guess this is irrelevant due to using PROXY protocol 😅 ).I wrote a little proxy CLI program to do similar to the
printfabove, but for any connection over TCP. Not sure if it’d be useful to anyone but I could share it if useful.If no trusted IP is configured, raw TCP is forwarded, so untrusted clients can sneak a header through but I think that’s intentional/expected to blindly forward and not make assumptions (since there’s a small possibility of the data randomly being a PROXY header but legitimate traffic routed to a destination that doesn’t expect a PROXY header, thus discarding by default may be bad?).
The docs don’t mention this though and the two config options available imply:
insecure=true, consumes the PROXY header)trustedIPs=..., consume the PROXY header or if not trusted discard)Which I think requires adding an IP that shouldn’t be a client (but not the subnet gateway IP either of course), if you want to discard any incoming PROXY headers for TCP routers?
This is how I created the test environment (I hope I didn’t over-complicated it). I created three Docker containers:
proxyProtocol.truestedIPsconfiguration.The following figure illustrate the setup:
Notice that the elb has two network interfaces. I used mkcert to set HTTPS support in the elb.
So, when I execute curl this is what I get:
I also double check that the PROXY Header was being sent using
tcpdumpinside the elb container. This is what I get:If I understood correctly, the issue is that I should get this answer instead:
Could you confirm @peterkappelt, @mpl, @rtribotte ?
To reproduce my steps, here the files I am using:
issue-9757.zip
I didn’t include the SSL certificate, but one can be easily generated using
mkcert.yeah ok, I think there’s a bug.
What I believe is happening is that Proxy Protocol changes the result/value of the RemoteAddr() call that is used at some point. In your case, it will return
80.1.2.3, instead of10.11.0.200.So then, when we get in
forwardedHeaders, and we ask the question “hey, is the last hop I’m talking to a trusted IP?” we use RemoteAddr() (maybe we shouldn’t, or maybe the result of RemoteAddr() itself should be fixed), which tells us that the last hop is80.1.2.3(which of course factually isn’t, it should be10.11.0.200). And since traefik is configured to trust only10.11.0.200(and not80.1.2.3), it goes into untrusted mode, erases all the X-Forwarded headers (as it should in these circumstances), and writes to them whatever it thinks the last hop was. It explains the behaviour regarding X-Forwarded-For, and probably for the others as well (I haven’t looked into them in detail, but I would be surprised if it’s not the same root cause).Hopefully we’ll look further into a fix soon.