traefik-crowdsec-bouncer: Bouncer doesn't correctly process X-Forwarded-For headers

Hi guys! I think I have similar or the same problem. I tried the latest version from both repos.

I am running crowdsec and this bouncer on a k3s cluster with traefik as ingress. When I access the website from my browser and look at the bouncer’s logs, this is what I get:

2022-02-25T04:17:52Z DBG Handling forwardAuth request ClientIP=10.42.0.123 X-Forwarded-For=10.42.1.0 X-Real-Ip=10.42.1.0
2022-02-25T04:17:52Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec-service.crowdsec:8080/v1/decisions?type=ban&ip=10.42.0.123
2022-02-25T04:17:52Z DBG No decision for IP "10.42.0.123". Accepting
{"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"10.42.0.123","latency":36.3995,"user_agent":"Mozilla/5.0  (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101  Firefox/98.0","time":"2022-02-25T04:17:52Z","message":"Request"}

when I check where that ip comes from I get that is from the traefik pod

> kubectl get pods -o wide --all-namespaces | grep "123"
kube-system          traefik-ffd48d555-wfpg4         1/1     Running       0                9h      10.42.0.123     main-node01

Am I missing something or this bug still present?

Thanks

It’s mainly an issue with your setup of traefik entry point. Within Kubernetes, Traefik Agents act as Network Load Balancers, so you must configure the EntryPoint Option “Forwarded Headers”.

https://doc.traefik.io/traefik/routing/entrypoints/#forwarded-headers

In addition, if you have another reverse proxy in front of Traefik, you must:

1. Ensure your front Reverse Proxy uses Proxy Protocol (Automatic with CloudFlare, setting up send-proxy-v2 within backend service of HAProxy)
2. Configure your Traefik entrypoint with ProxyProtocol

https://doc.traefik.io/traefik/routing/entrypoints/#proxyprotocol

Last but not least, make sure your Traefik middleware “FowardAuth” uses the option “trustForwardHeaders” https://doc.traefik.io/traefik/middlewares/http/forwardauth/#trustforwardheader

additionalArguments:
  - --providers.file.filename=/data/traefik-config.yaml
  - --entrypoints.websecure.http.middlewares=hardening@file
  - --entrypoints.websecure.http.tls=true
  - --entrypoints.websecure.http.tls.certresolver=leresolver
  - --entrypoints.websecure.http.tls.domains[0].main=yourdomain.com
  - --entrypoints.websecure.http.tls.domains[0].sans=*.yourdomain.com
  - --entrypoints.websecure.proxyProtocol.insecure=true
  - --entrypoints.websecure.forwardedHeaders.insecure
  - --certificatesresolvers.leresolver.acme.dnschallenge.provider=cloudflare
  - --certificatesresolvers.leresolver.acme.dnschallenge.resolvers=8.8.8.8
  - --certificatesresolvers.leresolver.acme.email=cloudflare-email-account@domain.com
  - --certificatesresolvers.leresolver.acme.storage=/certs/acme.json
ports:
  web:
    redirectTo: websecure
  dnsudp:
    port: 5553
    hostPort: 5553
    expose: true
    exposedPort: 53
    protocol: UDP
  dnstls:
    port: 5853
    hostPort: 5853
    expose: true
    exposedPort: 853
    protocol: TCP
  metrics:
    port: 9100
    expose: true
    exposedPort: 9205
    protocol: TCP
env:
  - name: CF_DNS_API_TOKEN
    valueFrom:
      secretKeyRef:
        key: apiKey
        name: cloudflare-api-credentials
  - name: CLOUDFLARE_PROPAGATION_TIMEOUT
    value: "300"
ingressRoute:
  dashboard:
    enabled: false
providers:
  kubernetesCRD:
    allowCrossNamespace: true
persistence:
  enabled: true
  path: /certs
  size: 128Mi
volumes:
  - mountPath: /data
    name: traefik-config
    type: configMap
pilot:
  enabled: true
  token: PILOT-TOKEN
experimental:
  plugins:
    enabled: true
logs:
  general:
    level: ERROR
  access:
    enabled: true

---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-credentials
  namespace: kube-system
type: Opaque
stringData:
  email: cloudflare-email-account@domain.com
  apiKey: cloudflare-api-token
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-config
  namespace: kube-system
data:
  traefik-config.yaml: |
    http:
      middlewares:
        hardening:
          chain:
            middlewares:
              - request-limits@file
              - secure-headers@file
              - crowdsec-bouncer@file
        request-limits:
          ratelimit:
            average: 100
            burst: 50
        secure-headers:
          headers:
            browserXssFilter: true
            contentSecurityPolicy: "default-src 'self'; connect-src * 'self'; img-src * 'self' data: https:; object-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https:; style-src 'self' 'unsafe-inline' https:; frame-src 'self' https:; child-src 'self' https:; font-src 'self' data:"
            contentTypeNosniff: true
            customresponseheaders:
              Permissions-Policy: "geolocation=(self), microphone=(), camera=(), fullscreen=(self)"
              Server: ""
              X-Powered-By: ""
            permissionsPolicy: connect-src * 'self'; "vibrate 'self'; geolocation 'self'; midi 'self'; notifications 'self'; push 'self'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; speaker 'none'; vibrate 'self'; fullscreen 'self'; frame-src 'self' https:; child-src 'self' https:; font-src 'self' data:"
            forceSTSHeader: true
            frameDeny: false
            referrerPolicy: same-origin
            stsIncludeSubdomains: true
            stsSeconds: 31536000
        crowdsec-bouncer:
          forwardauth:
            address: http://crowdsec-traefik-bouncer.crowdsec:8080/api/v1/forwardAuth
            trustForwardHeader: true

Quite honestly you don’t need to set the trusted proxy setting unless you’re setting it up on a different machine or something. Traefik will handle which are trusted before it hits this one. Keeping it blank will set it to trust all forwarding traffic. You could set it to trust docker networks if you really want to. The setting is about trusting the proxy in front of the bouncer really not necessarily cloudflare but you’d need to do testing.

You do need to set cloudflare CDN as trusted in the traefik config though.

I’ve opened a pull request for this, happy to talk through it or other concerns. I’ve tested it working for me (specifically sitting behind Cloudflare CDN) but I don’t have much golang experience so there might be formatting or other issues with it.

@youngt2, don’t worry about the ProxyProtocol from Cloudflare it’s automated. I’m using it too with a free account for my lab without any additional settings to make it works.

Only if you have a LoadBalancer or a Reverse Proxy in between your Kubernetes infrastructure and your internet router.

@SpaceComet, it’s the “k3s.service” file describing the service. If your OS uses systemd it should be located here: /etc/systemd/system/k3s.service

I was actually concern about not having HA.

Mainly, if you don’t have a Traefik Pod Instance per node, using “externaltrafficpolicy: Local” on the service will give you some wired routing situations (black hole, traffic routed on single node,…)

Well detailed explanations: https://technotes.adelerhof.eu/containers/kubernetes/externaltrafficpolicy/ Official Kubernetes Warnings: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip

What I noticed with my current setup is that traefik only deply one pod and 3 others as load balancer but if the node where traefik lives dies then my cluster go down

In that case, as you are using Helm, that’s pretty easy, play with the “replicas” option into your helm chart values file.

However, according to Traefik documentations, certificate management in HA way is not implemented anymore (that feature was present in v1.x), only in the enterprise edition. (cf. https://doc.traefik.io/traefik-enterprise/features/#compare)

So, if you want to do it, make sure your SSL certificates are managed externally of Traefik, example by using “cert-manager”. (eg. https://www.scaleway.com/en/docs/tutorials/traefik-v2-cert-manager/)

So if I install calico this should be solved right ?

By using Calico, you will have a better control of your ingress network and less pains with reverse proxies source ips. Then it should be solved if all points discussed previously are in place.

Hi. I have been working on this problem the last 3 days. First of all, thank you guys for taking the time to help me!

@vherrlein, at first I tried adding the forwardedHeaders.insecure CLI options into my HelmChartConfig file (traefik-config.yaml). Like here

additionalArguments:
      - --entrypoints.websecure.proxyProtocol.insecure=true
      - --entrypoints.websecure.forwardedHeaders.insecure

Also, my Traefik middleware “FowardAuth” uses the option “trustForwardHeaders.”

For some reason this didn’t work. Or at least not just by adding those extra flags. Then I read what @fbonalair posted. At the beginning I didn’t know how to do it until I found this.

So I also added this to my file:

service:
      spec:
        externalTrafficPolicy: Local

I confirm too, that’s working fine now, even behind 2 reverse proxies .

https://github.com/fbonalair/traefik-crowdsec-bouncer/pull/14 fixed my problems. Thanks and great job @pewter77!

So the X-Forwarded-For header is the defacto standard for source (and intermediate) IP address recording. Proxies and other intercepting services are permitted to append to the X-Forwarded-For header so that you have a “paper trail” of the hops the request has passed through, but the first value should be the originating IP. X-Real-IP doesn’t support appending and while it may show the originating IP there’s no way to know it hasn’t been overwritten along the way (technically this is also true with X-Forwarded-For but it at least allows appending). Consequently, it’s much more reliable to use the X-Forwarded-For header whenever there’s a possibility of more than one proxy between the origin and the destination server.

That’s what I have concluded too. I just did not have time to fix it until now.

I’ve opened a pull request for this, happy to talk through it or other concerns. I’ve tested it working for me (specifically sitting behind Cloudflare CDN) but I don’t have much golang experience so there might be formatting or other issues with it.

Thanks for your work! I will merge it, it’s an additional feature for the project. Don’t worry about golang, I’m a noob myself, I have used this project to work on the language actually ^^.

Should just be the Traefik CIDRs as that’s the immediate upstream source (Traefik should already trust the Cloudflare addresses).

Exactly, by default the Web framework I’m using is trusting everything by default. For this service and its scope, it should be fine.

This trusted proxy issue should be fixed from version 0.3.4 of the container.

In the interim I’ve forked the repo here https://github.com/thespad/traefik-crowdsec-bouncer and merged @pewter77 's PR.

Go is very much not my strong suit but I’ve been trying to get a handle on why the behaviour isn’t as expected and I think I’ve got a sense of it now.

So the X-Forwarded-For header is the defacto standard for source (and intermediate) IP address recording. Proxies and other intercepting services are permitted to append to the X-Forwarded-For header so that you have a “paper trail” of the hops the request has passed through, but the first value should be the originating IP. X-Real-IP doesn’t support appending and while it may show the originating IP there’s no way to know it hasn’t been overwritten along the way (technically this is also true with X-Forwarded-For but it at least allows appending). Consequently, it’s much more reliable to use the X-Forwarded-For header whenever there’s a possibility of more than one proxy between the origin and the destination server.

For example, in this request:

crowdsec-bouncer-traefik  | 2022-01-25T11:45:06Z DBG Handling forwardAuth request X-Forwarded-For="34.220.8.237, 162.158.107.233" X-Real-Ip=162.158.107.233
crowdsec-bouncer-traefik  | 2022-01-25T11:45:06Z DBG Request Crowdsec's decision Local API method=GET url=http://crowdsec:8080/v1/decisions?type=ban&ip=162.158.107.233
crowdsec-bouncer-traefik  | {"level":"info","status":200,"method":"GET","path":"/api/v1/forwardAuth","ip":"172.18.0.16","latency":2.872559,"user_agent":"Amazon Music Podcast","time":"2022-01-25T11:45:06Z","message":"Request"}
crowdsec-bouncer-traefik  | 2022-01-25T11:45:06Z DBG No decision for IP "162.158.107.233". Accepting

Here, 162.158.107.233 is a Cloudflare address, showing up as both the whole X-Real-IP header and the last entry in X-Forwarded-For. 34.220.8.237 is the originating IP, which also shows up in the X-Forwarded-For header as the first address.

Traefik’s own logging shows:

34.220.8.237 - - [25/Jan/2022:11:45:06 +0000] "GET /feed/dungeons/ HTTP/2.0" 200 52493 "-" "Amazon Music Podcast" 46091 "https-tsp-all@docker" "http://172.18.0.23:80" 138ms

Your current code is using the X-Real-IP header to do its auth checks, which is only going to show the last hop before Traefik, which in my case is Cloudflare’s edge.

I think it should be as straightforward as using the X-Forwarded-For header and extracting the first address from the list.