ingress-nginx: Whitelist-source-range not working properly with PROXY protocol

I can confirm this issue. It appears that 0.25.0 is matching the wrong IP against the whitelist when using the PROXY protocol. It matches the load balancer IP against cidrs defined by the ingress annotation. Afterwards, it logs the correct client ip though.

This is a redacted part of our logs with 0.25.0:

2019/07/30 14:16:36 [error] 115#115: *2569 access forbidden by rule, client: <LOAD BALANCER IP>, server: <HOST>, request: "GET / HTTP/2.0", host: "<HOST>"
<REAL CLIENT IP> - [<REAL CLIENT IP>] - - [30/Jul/2019:14:16:36 +0000] "GET / HTTP/2.0" 403 150 "-" "curl/7.52.1" 47 0.000 [<TARGET SERVICE>] [] - - - - b09fd9ea322b26b20463ff9952dc65f6

If my Ingress is annotated with nginx.ingress.kubernetes.io/whitelist-source-range=<REAL CLIENT IP>/32, this is the result. If I update the annotation to nginx.ingress.kubernetes.io/whitelist-source-range=<LOAD BALANCER IP>/32, it allows access (but for every client, since it’s the load balancer’s IP).

Our setup is not unusual, it’s a TCP HAProxy with PROXY protocol enabled in front of nginx-ingress. This issue might break white- and blacklists for everyone running a similiar setup.

Edit: Our Helm values for the stable/nginx-ingress Chart; Chart version 1.11.4:

controller:
  config:
    use-proxy-protocol: "True"
    proxy-real-ip-cidr: "<LOAD BALANCER IP>/32"
    server-tokens: "false"
  kind: DaemonSet
  service:
    type: NodePort
    nodePorts:
      http: 31080
      https: 31443

@aledbf knock knock… sorry to bother you but we are growing 👯‍♂

yeah, we are getting this also.

I did no investigation because I did not have the time, but the new version (0.25.0) I get 403’d but the old version works fine(0.24.0).

Maybe the end of the week.

To those affected by this issue, please use the image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev (current master) that contains the refactoring #4557

I’m ashamed to admit it, but at least from my side this issue does not apply anymore as well, but not because of #4557 but rather because I finally added externalTrafficPolicy: Local to my Helm values. Classic “didn’t fully read the docs” on my side, sorry.

That being said something changed regarding that from 0.24.x to 0.25.x because it worked before. Just as an interesting tidbit. I guess it should have not worked in the first place.

I’ve the same symptoms on my side. What could I provide to help you ?

I’m using OVH Manages Kubernetes with an HAProxy in front of the nginx-ingress-controller.

Confirm that it work with 0.24.1 version.

@aledbf hi, have you tried reproducing this with the Helm Chart config (https://github.com/kubernetes/ingress-nginx/issues/4305#issuecomment-516442084) and HAProxy config (https://github.com/kubernetes/ingress-nginx/issues/4305#issuecomment-519393468) I shared?

This issue is still present for us and prevents us from updating to 0.25.1 and therefore incorporating the CVE patches.

Hi @aledbf are you able to reproduce this with a static HAProxy configuration? I posted my Helm Chart values above, this is a minimal snippet for HAProxy that mirrors our setup:

frontend kubernetes_ingress_HTTP
    bind 0.0.0.0:80
    mode tcp

    default_backend k8s_ingress_http

frontend kubernetes_ingress_HTTPS
    bind 0.0.0.0:443
    mode tcp

    default_backend k8s_ingress_https

backend k8s_ingress_http
    mode tcp

    default-server inter 1s rise 2 fall 3
    server node01 <node01 IP>:31080 send-proxy
    server node02 <node02 IP>:31080 send-proxy
    server node03 <node03 IP>:31080 send-proxy
    
backend k8s_ingress_https
    mode tcp

    default-server inter 1s rise 2 fall 3
    server node01 <node01 IP>:31443 send-proxy
    server node02 <node02 IP>:31443 send-proxy
    server node03 <node03 IP>:31443 send-proxy

(our setup works pre 0.25.0 and stops working when updating to 0.25.0 without any other changes)