ingress-nginx: use-forwarded-headers config is ignored/bugged for X-Forwarded-Proto header

Hi,

I just stumbled upon this and I think it was once working with PR #4958, but got then reverted with PR #6170. The configuration option use-forwarded-headers is ignored / unvoluntarily not being used for the transfered X-Forwarded-Proto header.

This is because in

https://github.com/kubernetes/ingress-nginx/blob/922e27fea7a2409f2a23a741b939ddfe01130324/rootfs/etc/nginx/template/nginx.tmpl#L1286

the variable $pass_access_scheme is used. While the comment of #6170 says it is working like that assuming the value is properly set already with

https://github.com/kubernetes/ingress-nginx/blob/4dd206b31af977a484bd7b9f52a8f6c5367cd17e/rootfs/etc/nginx/lua/lua_ingress.lua#L119

the code here in this repo is overriding it beforehand, making the assumption wrong, I think:

https://github.com/kubernetes/ingress-nginx/blob/922e27fea7a2409f2a23a741b939ddfe01130324/rootfs/etc/nginx/template/nginx.tmpl#L1182

Would be really nice to get it fixed (again) by maybe reverting the reverted PR #4958.

Leaving the remaining stuff empty, as it is obviously something independent of the following…

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

Kubernetes version (use kubectl version):

Environment:

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
- Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
Basic cluster related info:
- kubectl version
- kubectl get nodes -o wide
How was the ingress-nginx-controller installed:
- If helm was used then please show output of helm ls -A | grep -i ingress
- If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
- If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
- if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
Current State of the controller:
- kubectl describe ingressclasses
- kubectl -n <ingresscontrollernamespace> get all -A -o wide
- kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
- kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Current state of ingress object, if applicable:
- kubectl -n <appnnamespace> get all,ing -o wide
- kubectl -n <appnamespace> describe ing <ingressname>
- If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
Others:
- Any other related information like ;
  - copy/paste of the snippet (if applicable)
  - kubectl describe ... of any custom configmap(s) created and in use
  - Any other related information that may help

What happened:

What you expected to happen:

How to reproduce it:

Anything else we need to know:

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 5
Comments: 19 (4 by maintainers)

Most upvoted comments

Is there a way to force the header without changing the global configuration? I tried with this ingress configuration:

    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-Proto https;

But it instead appends the value to the header:

X-Forwarded-Proto http, https

This causes the backend which relies on the header to go in an infinite redirect loop in my case.

jonathan-boudreau-work on Apr 19, 2022

Using proxySetHeaders doesn’t work either as then both headers are set:

controller:
  proxySetHeaders:
    X-Forwarded-Proto: https

Request headers:

Hypertext Transfer Protocol
    ...
    X-Forwarded-Port: 80\r\n
    X-Forwarded-Proto: http\r\n
    X-Forwarded-Scheme: http\r\n
    X-Scheme: http\r\n
    X-Forwarded-Proto: https\r\n

lyz-code on Feb 28, 2022

Commenting in hope this will help someone, as I have spent way too much time trying to solve this.

I had the same issue, my FE requests are going through a node proxy, which was setting the x-forwarded headers:

headers['x-forwarded-for'] = proxiedReq.socket.remoteAddress;
headers['x-forwarded-proto'] = proxiedReq.socket.encrypted ? 'https' : 'http';
headers['x-forwarded-host'] = proxiedReq.headers[':authority'] || proxiedReq.headers.host;

This was then hitting the ingress domain, for which I tried various workarounds on my ingress, like:

          annotations:
            nginx.ingress.kubernetes.io/ssl-redirect: "false"

            nginx.ingress.kubernetes.io/configuration-snippet: |
              proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
              proxy_set_header x-forwarded-host $http_x_forwarded_host;
              proxy_set_header x-forwarded-proto $http_x_forwarded_proto;
              proxy_set_header x-forwarded-scheme $http_x_forwarded_scheme;

Nothing worked, ingress-nginx-controller was still appending its own values to those, and the upstream (Ruby) was not detecting the scheme correctly, where header was set as HTTP_X_FORWARDED_PROTO = http, https

Thanks to @armujahid’s comment, I have added the use-forwarded-headers config map option to the helm chart (ingress-nginx-4.4.0):

      values:
        controller:
          config:
            use-forwarded-headers: true
          ...

And it works great, I was also able to remove all annotations from the ingress and it works just fine.

gustaff-weldon on Nov 25, 2022

@awwithro for me currently use-forwarded-headers: “true” is working fine. I am using helm chart ingress-nginx-4.3.0 with chart aws-load-balancer-controller-1.4.5 and everything is working fine without any hack.

The hack I was referring in my previous comment is no longer required. I was using custom nginx template at that time with helm chart using customTempale helm value.

  # customTemplate:
  #   configMapName: "nginx-template"
  #   configMapKey: "nginx.tmpl"

armujahid on Nov 23, 2022