ingress-nginx: Problem with HTTPS, CloudFlare and X-Forwarded-Port header

Here is illustration of a full working solution with Helm and ingress-nginx chart version 4.0.13 / K8s 1.21.

First, add new template, ingress-nginx/templates/controller-configmap-lua.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    {{- include "ingress-nginx.labels" . | nindent 4 }}
    app.kubernetes.io/component: controller
    {{- with .Values.controller.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
  name: {{ include "ingress-nginx.fullname" . }}-custom-lua
  namespace: {{ .Release.Namespace }}
data:
  main.lua: |
    local ngx = ngx

    local _M = {}

    function _M.rewrite()
      ngx.var.pass_port = 443
      ngx.var.pass_access_scheme = "https"
    end

    return _M

Then add custom values.yaml:

  plugins: "rewrite_fwd_headers"

  extraVolumeMounts:
    ## Additional volumeMounts to the controller main container.
    - name: cm-volume-lua-plugin
      mountPath: /etc/nginx/lua/plugins/rewrite_fwd_headers

  extraVolumes:
    ## Additional volumes to the controller pod.
    - name: cm-volume-lua-plugin
      configMap:
        name: ingress-nginx-custom-lua

Your upstream apps should now see

• X-Forwarded-Proto -> https
• X-Forwarded-Scheme -> https
• X-Forwarded-Port -> 443

Note there’s a slight defect in this setup, in that values.yaml hardcodes the configmap name even though that name is configurable through the template. (It could just be hardcoded there also.)

having the same problem here (and with keycloak too)… the solution still this this ‘custom plugin’, really?

This problem is driving me nuts. Any solution yet?

Changing the value of the header X-Forwarded-Port without using a custom template can be done with a plugin. https://github.com/kubernetes/ingress-nginx/tree/master/rootfs/etc/nginx/lua/plugins

The content of such a thing is trivial:

local ngx = ngx

local _M = {}

function _M.rewrite()
  if ngx.var.http_cf_connecting_ip then
    ngx.log(ngx.ERR, "Changing x-forwarded-port to 443")
    ngx.var.pass_port = 443
  end
end

return _M

curl localhost -H 'CF-Connecting-IP: 1.1.1'


Hostname: http-svc-6b7fcd49cc-jb27g

Pod Information:
	node name:	kind-control-plane
	pod name:	http-svc-6b7fcd49cc-jb27g
	pod namespace:	default
	pod IP:	10.244.0.11

Server values:
	server_version=nginx: 1.12.2 - lua: 10010

Request Information:
	client_address=10.244.0.7
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://localhost:8080/

Request Headers:
	accept=*/*
	cf-connecting-ip=1.1.1
	host=localhost
	user-agent=curl/7.68.0
	x-forwarded-for=172.18.0.1
	x-forwarded-host=localhost
	x-forwarded-port=443
	x-forwarded-proto=http
	x-real-ip=172.18.0.1
	x-request-id=dc00f8bf88bc383b786a227b7d2657e4
	x-scheme=http

Request Body:
	-no body in request-

The condition to change the variable can check for any other header (like limiting the change to a particular host) Using a configmap is possible to mount the plugin as a file https://github.com/kubernetes/ingress-nginx/blob/master/charts/ingress-nginx/values.yaml#L396-L404

Is there any plan for this error? X-Forwarded-Port should be configurable from the annotations. I saw that the protocol and the scheme can be set with the annotations, but don’t know why the port is missed out. Currently we have to hardcode the forward header port within the template file. It would be not friendly with the mixed http and https traffic.

                                ### hardcoded x-forwarded-port for Keycloak
                                #{{ $proxySetHeader }} X-Forwarded-Port       $pass_port;
                                #{{ $proxySetHeader }} X-Forwarded-Proto      $pass_access_scheme;
                                #{{ $proxySetHeader }} X-Forwarded-Scheme     $pass_access_scheme;
                                {{ $proxySetHeader }} X-Forwarded-Port       "443";
                                {{ $proxySetHeader }} X-Forwarded-Proto      "https";
                                {{ $proxySetHeader }} X-Forwarded-Scheme     "https";
                                ### End of modification

@aledbf

Changing the value of the header X-Forwarded-Port without using a custom template can be done with a plugin. https://github.com/kubernetes/ingress-nginx/tree/master/rootfs/etc/nginx/lua/plugins

The content of such a thing is trivial:

Slapping the word ‘trivial’ on something doesn’t make it trivial. Asking people to write a Lua plugin is not trivial nor should it be necessary to set a single proxy header for an upstream app.

I just wanted to mention that with the most recent ingress-nginx helm chart this solution needs to be modified to work correctly. In “values.yaml” plugins: "rewrite_fwd_headers" needs to be a child of controller.config. Thanks @brsolomon-deloitte for this workaround!

Just to add a bit to this, using the bitnami chart, this is what we did:

controller:
  config:
    plugins: "rewrite_fwd_headers"
  extraVolumeMounts:
    - name: cm-volume-lua-plugin
      mountPath: /etc/nginx/lua/plugins/rewrite_fwd_headers
  extraVolumes:
    - name: cm-volume-lua-plugin
      configMap:
        name: ingress-nginx-custom-lua

Here is illustration of a full working solution with Helm and ingress-nginx chart version 4.0.13 / K8s 1.21.

First, add new template, ingress-nginx/templates/controller-configmap-lua.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    {{- include "ingress-nginx.labels" . | nindent 4 }}
    app.kubernetes.io/component: controller
    {{- with .Values.controller.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
  name: {{ include "ingress-nginx.fullname" . }}-custom-lua
  namespace: {{ .Release.Namespace }}
data:
  main.lua: |
    local ngx = ngx

    local _M = {}

    function _M.rewrite()
      ngx.var.pass_port = 443
      ngx.var.pass_access_scheme = "https"
    end

    return _M

Then add custom values.yaml:

  plugins: "rewrite_fwd_headers"

  extraVolumeMounts:
    ## Additional volumeMounts to the controller main container.
    - name: cm-volume-lua-plugin
      mountPath: /etc/nginx/lua/plugins/rewrite_fwd_headers

  extraVolumes:
    ## Additional volumes to the controller pod.
    - name: cm-volume-lua-plugin
      configMap:
        name: ingress-nginx-custom-lua

Your upstream apps should now see

* `X-Forwarded-Proto` -> https

* `X-Forwarded-Scheme` -> https

* `X-Forwarded-Port` -> 443

Note there’s a slight defect in this setup, in that values.yaml hardcodes the configmap name even though that name is configurable through the template. (It could just be hardcoded there also.)

I just wanted to mention that with the most recent ingress-nginx helm chart this solution needs to be modified to work correctly. In “values.yaml” plugins: "rewrite_fwd_headers" needs to be a child of controller.config. Thanks @brsolomon-deloitte for this workaround!

Hi !

I found a way to do a tcpdump in nginx-ingress-controllers.

CloudFlare doesn’t send a X-Forwarded-Port Header.

I think this is related to this : https://github.com/kubernetes/ingress-nginx/blob/fb6a03ffb42cfbf682c3c7f400f46fb4802caa1c/rootfs/etc/nginx/template/nginx.tmpl#L1123

This variable is set to the server port.

More critical is that I think this variable is not overwritable in general server-snippet nor in annotation in the Ingress.

for example, nginx.conf when I added this variable to annotation :

        server {
                server_name myApp.mydomain.com ;
                listen 80;
                listen [::]:80;
                set $proxy_upstream_name "-";
                proxy_set_header X-Forwarded-Port "443";
                location / {
                        proxy_set_header X-Forwarded-Port       $pass_port;

@brsolomon-deloitte thanks this is working fine in the newer 4.0.19 chart. The plugins didn’t seem to load unless i explicitly set the plugins in the controllers configmap.

From what i can see by browsing the nginx.conf generated, This issue seems to arise because all the X-Forwarded headers are set after the server/configuration snippet. Would moving these X-Forwarded header definition above those definitions solve them?

Hi @tpoindessous @kolorful @Uysim, can you guys can confirm that the issue still exists? Also with newer versions of ingress-nginx?

Still exists as of most recent chart/controller (chart 4.0.13). It is virtually impossible to set X-Forwarded-Proto for upstream apps that need to know the original client made an HTTPS request.

Hi @tpoindessous @kolorful @Uysim, can you guys can confirm that the issue still exists? Also with newer versions of ingress-nginx?

@toredash if you use a snippet to set the X-Forwarded-Port header it ends up appending instead of replacing.

Ideally if the upstream loadbalancer is not sending an X-Forwarded-Port header and ingress-nginx is configured to use-forwarded-headers then either it would 1) not send an X-Forwarded-Port at all or 2) derive the X-Forwarded-Port from the X-Forwarded-Proto. Right now it is sending an X-Forwarded-Port that is the port NGINX itself is listening on, which means the resulting X-Forwarded-* headers are a mixture of NGINX and the upstream loadbalancer.

I stumbled into this issue while troubleshooting the closely related problem. We use custom SSL port 5443 for ingress-controller instead of standard 443, but ingress-nginx always sends X-Forwarded-Port: 443 to our backend. Sad thing is that even sending this header from the client to nginx doesn’t help, it always returns a constant 443 value, regardless of $server_port. I believe the problem is in the following part of Lua code: https://github.com/kubernetes/ingress-nginx/blob/6c729e9cc76ca33ecd1b33c36b931c0aa27aa34f/rootfs/etc/nginx/lua/lua_ingress.lua#L168-L169

I tried to comprehend this code but I really don’t understand why the value is set to 443 and not to config.listen_ports.https (or left as is, because it is already set to the proper value at line 163).

When I tried to play with nginx.conf template on a live server, I figured out these things:

As others already mentioned there’s no way how to override this value by configuration.

Any help would be appreciated.

Maybe we should add more flexibility to users to allow overriding / setting those headers to solve such particular scenarios. WDYT ?

@Dohbedoh if there is a way to override or set those headers that would work for me.