traefik: Incorrect X-FORWARDED-PROTO, X-FORWARDED-PORT for external TLS termination and proxy protocol

Welcome!

  • Yes, I’ve searched similar issues on GitHub and didn’t find any.
  • Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.

What did you do?

I think I’ve discovered an edge case (or myself not completely understanding the docs) that relates to setting the X-FORWARDED-PROTO and -PORT when proxy protocol is enabled.

I was able to reproduce the bug with this minimal setup:

  • my browser @ 80.1.2.3
    • connects to the public ip of the external load balancer
  • an external load balancer service, does TLS termination, 10.11.0.200
    • wireshark shows that it correctly sets X-Forwarded-Proto https, X-Forwarded-Port 443
    • proxies to Traefik on port 8181, proxy protocol enabled
  • Traefik
    • --entrypoints.web.address=:8181/tcp --entrypoints.web.forwardedHeaders.trustedIPs=10.11.0.200 --entrypoints.web.proxyProtocol.trustedIPs=10.11.0.200
    • service that connects to the sample app
  • Sample App, simply dumps all request headers in the response

I’d expect the sample app to receive those headers:

x-forwarded-for 80.1.2.3, 10.11.0.200
x-forwarded-host hostname.example
x-forwarded-port 443
x-forwarded-proto https
x-forwarded-server traefik-node-name
x-real-ip 80.1.2.3

What did you see instead?

Traefik forwards the headers, except for X-FORWARDED-PROTO and -PORT. The sample app receives:

x-forwarded-for 80.1.2.3 
x-forwarded-host hostname.example
x-forwarded-port 80
x-forwarded-proto http
x-forwarded-server traefik-node-name
x-real-ip 80.1.2.3

I can get the expected x-forwarded-proto and x-forwarded-port by:

  • disabling proxy protocol on both Traefik and the external load balancer
  • setting forwardedHeaders.trustedIPs=0.0.0.0/0 (or forwardedHeaders.insecure=true)
  • setting forwardedHeaders.trustedIPs=80.1.2.3 (my client IP)

None of this is really suitable for production.

As it seems, traefik uses the client ip as the source ip when determining whether it is a trusted ip for header forwarding, when proxy protocol is enabled. I’d still expect traefik to use the actual source ip (load balancer).

What version of Traefik are you using?

Version: 2.9.6 Codename: banon Go version: go1.19.4 Built: 2022-12-07T14:17:58Z OS/Arch: linux/amd64

What is your environment & configuration?

See description and cases above

If applicable, please paste the log output in DEBUG level

No response

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 7
  • Comments: 16 (8 by maintainers)

Most upvoted comments

Hi, sorry for being quiet, we ended up re-structuring our setup to mitigate this issue and make some other changes, I wasn’t in touch with this issue since then and didn’t have a setup to reproduce it.

As far as I remember: we opted to enable ProxyProtocol on our load balancer for various reasons (mainly due to having non-HTTP services where we needed the the real source IP). AFAIK ProxyProtocol only conveys information about the original remote IP and Port. The problem here was that the connection between the load balancer and traefik was pure HTTP (no HTTPS), which caused traefik to set the X-Forwarded-Proto header by itself to http in this case (causing issues with the end application) even though the external load balancer did SSL termination. We couldn’t find a quick way to ensure that X-Forwarded-Proto was either not present or correctly set to https, raising the need for combined ProxyProtocol and X-Forwarded*-Headers from the load balancer.

However, I don’t mind any outcome at this point, it is not relevant for us anymore. Maybe a note in the docs (X-Forwarded* and ProxyProtocol don't work well together) would suffice.

Regards, Peter

Thanks for the tip @polarathene, it is helping me a lot!

Hi @mpl,

sorry for the confusion - I wrote this issue after a long day of debugging symptoms that eventually resulted in this specific problem and had way too many snippets in my clipboard.

Btw, you say that your browser is 8.1.2.3, but then everywhere else I see 80.1.2.3. I assume that’s a typo, and these two are in fact the same IP?

Yes, it was a typo, I changed it in the initial comment

However, you said that you would expect x-forwarded-for 80.1.2.3 (and that you did receive that on the backend). Was that a typo? If not, I think that in itself is already problematic, isn’t it?

I didn’t really think about this (yet), for now I just cared about proto and port. It doesn’t really matter for my setup if the external load balancer (10.11.0.200) is included in X-Forwarded-For since it’s part of a trusted environment. However, I do agree that it probably should be part of X-Forwarded-For in general.

I checked X-Forwarded-For once again:

  • with proxy protocol disabled: X-Forwarded-For: 80.1.2.3, 10.11.0.200 (as expected)
  • with proxy protocol enabled and forwardedHeaders.insecure=true: X-Forwarded-For: 80.1.2.3, 80.1.2.3 (duplicate client ip)
  • with proxy protocol enabled and forwardedHeaders.trustedIPs=10.11.0.200: X-Forwarded-For 80.1.2.3 (along with port 80 and http protocol)

Cheers!

I am having problems to reproduce it locally

I am a bit late, but you can inject your own PROXY protocol header via plain HTTP requests:

# Use `printf` command to format a raw HTTP request with plaintext PROXY header prepended,
# then pipe into netcat to send to `localhost:80`:
printf "PROXY TCP4 10.20.30.40 172.16.13.37 12345 44380\r\nGET / HTTP/1.1\r\nHost: http-without-redirect.test\r\n\r\n" | nc -w 1 localhost 80

Within a Docker Compose config, you can quite easily setup containers all within the same network for better IP assignments and minimizing gotchas.

For example since you sent the curl request to localhost:4443, Docker routed that through the subnet gateway IP 172.18.100.1 (although I guess this is irrelevant due to using PROXY protocol 😅 ).


I wrote a little proxy CLI program to do similar to the printf above, but for any connection over TCP. Not sure if it’d be useful to anyone but I could share it if useful.

If no trusted IP is configured, raw TCP is forwarded, so untrusted clients can sneak a header through but I think that’s intentional/expected to blindly forward and not make assumptions (since there’s a small possibility of the data randomly being a PROXY header but legitimate traffic routed to a destination that doesn’t expect a PROXY header, thus discarding by default may be bad?).

The docs don’t mention this though and the two config options available imply:

  • “trust everyone” (insecure=true, consumes the PROXY header)
  • “trust only these IP” (trustedIPs=..., consume the PROXY header or if not trusted discard)

Which I think requires adding an IP that shouldn’t be a client (but not the subnet gateway IP either of course), if you want to discard any incoming PROXY headers for TCP routers?

This is how I created the test environment (I hope I didn’t over-complicated it). I created three Docker containers:

  1. external_elb: This is suppose to be the external load balancer. I choose HAproxy for this.
  2. traefik: Traefik container running version 2.11. It only has one entry-point with Proxy Protocol enabled. To do this I set the proxyProtocol.truestedIPs configuration.
  3. whoami: The service for testing.

The following figure illustrate the setup:

Proxy

Notice that the elb has two network interfaces. I used mkcert to set HTTPS support in the elb.

So, when I execute curl this is what I get:

Hostname: whoami
IP: 127.0.0.1
IP: 10.150.100.15
RemoteAddr: 10.150.100.5:40536
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: curl/8.6.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.18.100.1
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik
X-Real-Ip: 172.18.100.1

I also double check that the PROXY Header was being sent using tcpdump inside the elb container. This is what I get:

proxy-header request

If I understood correctly, the issue is that I should get this answer instead:

Hostname: whoami
IP: 127.0.0.1
IP: 10.150.100.15
RemoteAddr: 10.150.100.5:40536
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: curl/8.6.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.18.100.1, 10.150.30.10
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik
X-Real-Ip: 172.18.100.1

Could you confirm @peterkappelt, @mpl, @rtribotte ?

To reproduce my steps, here the files I am using:

issue-9757.zip

I didn’t include the SSL certificate, but one can be easily generated using mkcert.

yeah ok, I think there’s a bug.

What I believe is happening is that Proxy Protocol changes the result/value of the RemoteAddr() call that is used at some point. In your case, it will return 80.1.2.3, instead of 10.11.0.200.

So then, when we get in forwardedHeaders, and we ask the question “hey, is the last hop I’m talking to a trusted IP?” we use RemoteAddr() (maybe we shouldn’t, or maybe the result of RemoteAddr() itself should be fixed), which tells us that the last hop is 80.1.2.3 (which of course factually isn’t, it should be 10.11.0.200). And since traefik is configured to trust only 10.11.0.200 (and not 80.1.2.3), it goes into untrusted mode, erases all the X-Forwarded headers (as it should in these circumstances), and writes to them whatever it thinks the last hop was. It explains the behaviour regarding X-Forwarded-For, and probably for the others as well (I haven’t looked into them in detail, but I would be surprised if it’s not the same root cause).

Hopefully we’ll look further into a fix soon.