ingress-nginx: DO Proxy Protocol broken header

Hi, A workaround is to add the service.beta.kubernetes.io/do-loadbalancer-hostname annotation to the service with a hostname pointing to the loadbalancer IP.

See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
    service.beta.kubernetes.io/do-loadbalancer-hostname: "do-k8s.example.com"
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

It works for me using the helm chart with the following values:

## nginx configuration
## https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml

## https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol
## https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer
controller:
  config:
    use-proxy-protocol: "true"

  service:
    externalTrafficPolicy: "Local"
    annotations:
      # https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/
      # https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

Can confirm this issue, same configuration here across the board.

However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)

@tlaverdure that’s because you are no specifying the flag --haproxy-protocol in the curl command. If you enable proxy protocol in the ingress controller you need to decode it.

I’m a bit unclear whether this issue is the same one we just encountered, but perhaps the following is helpful:

We have a microservice architecture, where services talk to each other via the axios library, all inside the same cluster. What we’d misconfigured was the URL by which the services talk to each other. We had one service talk to the other via the external dns record by which the target service was known e.g foo.domain.com, causing traffic for it to go all the way out and back into the cluster again. When nginx tried to handle the request, the header looked broken because the request wasn’t preceded by the instruction PROXY TCP4 10.8.0.18 [TARGET_IP] 55966 443 (which is what happens when you curl --haproxy-protocol, and is what happens to all inbound traffic handled by the ingress controller when you enable the “Proxy Protocol” setting)

By changing the URL of the target service to the internal dns record e.g http://[container].[namespace].svc.cluster.local by which it was known, traffic was sent directly to the target, not back through the ingress controller.

For users of DigitalOcean’s managed Kubernetes offering (DOKS) and possibly others: you are most likely running into a specific bypassing behavior of kube-proxy that causes requests to never leave the cluster but go straight to the internal pod. For receiving pods that expect a certain protocol to be implemented by the load balancer (like proxy protocol or TLS termination), this breaks work flows.

Pointers to the related upstream issue and workarounds are described in our CCM README. The tracking issue to address the problem long-term for DOKS is at digitalocean/DOKS#8.

This is precisely the problem we are facing and the workaround was to add the hostname as described. I can confirm that this also works with wildcard DNS matches.

Thank you service.beta.kubernetes.io/do-loadbalancer-hostname seems to work. No more broken headers using proxy protocol

adding service.beta.kubernetes.io/do-loadbalancer-hostname works for DO. But it’s not clear what to do at the first time when reading DO docs (from here and here)

Basically it contains the following steps:

1. First download nginx-ingress yaml file to local (https://kubernetes.github.io/ingress-nginx/deploy/#digital-ocean), DON’T apply directly from remote URL.
2. Apply the downloaded yaml script
3. After Nginx ingress controller successfully started and has its External IP (check its status by running kubectl get svc --namespace=ingress-nginx). Then go to your domain provider and point your domain to its External IP, after that wait for few mins for changes to get updated and try access the domain, make sure it reaches to Ingress controller (usually show ingress status page, 404… something)
4. Open the Yaml file in step 1, find service with name ingress-nginx-controller (it has a comment Source: ...controller-service.yaml), and add 1 more annotation with value:

service.beta.kubernetes.io/do-loadbalancer-hostname: "your_domain_name"

After this you can create your Ingress resource and use Cert Manager to issue SSL certificate as usual. Hope this help

The same issue happens in Kubernetes cluster on top of OpenStack cloud (using openstack-cloud-controller-manager).

The Ingress service could be accessed from outside the cluster, but not from the cluster node or in the pod.

Hello,

I am running EKS with kubernetes nginx ingress on NLB with SSL termination but the proxy protocol annotations was not working.

controller:
  electionID: external-ingress-controller-leader
  ingressClassResource:
    name: external-nginx
    enabled: true
    default: false
    controllerValue: "k8s.io/external-ingress-nginx"
  config: 
    use-proxy-protocol": "true" 
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "environment=dev,owner=platform-engineering"
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:<REDACTED>:certificate/<REDACTED>
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-<REDACTED>, subnet-<REDACTED>
      
      # below annotations were not working
      # service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      # service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: "preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true"

But once the NLB and ingress objects are created, the services wont be reachable because the proxy protocol on the NLB target groups was not turned on automatically even with the provided annotations. I got the below error in the controller pod logs,

2022/11/03 20:50:23 [error] 32#32: *26209 broken header: "GET /index.html HTTP/1.1
Host: nginx.<REDACTED>
Connection: keep-alive
Cache-Control: max-age=0
sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
DNT: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,ta;q=0.8

" while reading PROXY protocol, client: <REDACTED>, server: 0.0.0.0:80

Once I turned them on manually through console (NLB -> Listeners -> Target Group attributes), it works (I get 200 response in the same controller logs).

But both the annotations in the helm values does not seem to work.

Can anyone help on this ?

I ran into this and at one point, I deleted the ingress service and recreated it and it worked. I got the broken headers issue when I had the nginx configmap set, but not the annotations on the ingress service that created the DO LB. Manually configuring “Proxy Protocol” on the LB w/ the Web UI didn’t work for me.

Anyways, here is a config that worked for me:

mandatory.yaml

# Please note this file has been customized from the original at: 
# https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml  

#... stuff in the mandatory.yaml file that doesn't need to be customized.

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
# https://www.digitalocean.com/community/questions/how-to-set-up-nginx-ingress-for-load-balancers-with-proxy-protocol-support?answer=50244
data:
  use-forwarded-headers: "true"
  compute-full-forwarded-for: "true"
  use-proxy-protocol: "true"

# ... more stuff in the mandatory.yaml file that doesn't need to be customized.

cloud-generic.yaml

# Please note this file has been customized from the original at: 
# https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml 

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    # https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: https
      port: 443
      targetPort: https

ingress.yaml (I wanted a whitelist for only CloudFlare)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: allup
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "2400:cb00::/32, 2606:4700::/32, 2803:f800::/32, 2405:b500::/32, 2405:8100::/32, 2a06:98c0::/29, 2c0f:f248::/32, 173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, 141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20, 197.234.240.0/22, 198.41.128.0/17, 162.158.0.0/15, 104.16.0.0/12, 172.64.0.0/13, 131.0.72.0/22"
spec:
  tls:
    - secretName: cloudflare-tls-cert
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: app-www-service
              servicePort: http

Stupid check, but if you go to the load balancer’s settings in the DO admin panel, it’s enabled, right? Because if it’s not while use-proxy-protocol = true you end up with the same kind of encoding mismatch.

@w1ndy no. You need a custom template for that https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/