ingress-nginx: HSTS header missing on responses from a custom errors backend

NGINX Ingress controller version:

Release: v0.34.1 Build: v20200715-ingress-nginx-2.11.0-8-gda5fa45e2 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.19.1

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:“1”, Minor:“16+”, GitVersion:“v1.16.13-gke.1”, GitCommit:“688c6543aa4b285355723f100302d80431e411cc”, GitTreeState:“clean”, BuildDate:“2020-07-21T02:37:26Z”, GoVersion:“go1.13.9b4”, Compiler:“gc”, Platform:“linux/amd64”}

However, also tried with 1.18.3 on minikube and its the same problem.

Environment:

• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release): Alpine Linux v3.11 (3.11.6)
• Kernel (e.g. uname -a): Linux ingress-nginx-controller-5f68d8cb5f-zlmgv 4.19.112+ #1 SMP Thu May 21 12:32:38 PDT 2020 x86_64 Linux
• Install tools: kubectl / kustomize

What happened:

I use the default HSTS settings (enabled). When hitting an endpoint that exists, I get the Strict-Transport-Security as expected. However, when hitting the ingress (host match) with no path matches, the response does not contain the Strict-Transport-Security as expected.

What you expected to happen:

I would expect that the Strict-Transport-Security header is included on all responses, including 404s.

How to reproduce it:

1. Install minikube 1.12.3, and start with 1.16.13: minikube start --kubernetes-version=v1.16.13 (the K8S version doesn’t seem to matter though, tried it on 1.18.3 and its the same problem)
2. Install the ingress controller

   kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml
   

3. Create a custom error backend service at ingress-nginx/nginx-errors – tboerger/errors:latest seems to work for the purposes of this repro.

   kubectl run nginx-errors --image=tboerger/errors:latest --namespace ingress-nginx
   
   echo "
     apiVersion: v1
     kind: Service
     metadata:
       name: nginx-errors
       namespace: ingress-nginx
       labels:
         app.kubernetes.io/name: nginx-errors
         app.kubernetes.io/instance: ingress-nginx
     spec:
       selector:
         run: nginx-errors
       ports:
       - port: 80
         targetPort: 8080
         name: http
   " | kubectl apply -f -
   

4. Edit the ingress-nginx-controller Deployment and set the value of the --default-backend-service flag to ingress-nginx/nginx-errors.
5. Edit the ingress-nginx-controller ConfigMap and create the key custom-http-errors with a value of 404.
6. Create a backend:

   kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/http-svc.yaml
   

7. Create an ingress that responds on /foo:

   echo "
     apiVersion: networking.k8s.io/v1beta1
     kind: Ingress
     metadata:
       name: foo-bar
     spec:
       rules:
       - host: foo.bar
         http:
           paths:
           - backend:
               serviceName: http-svc
               servicePort: 80
             path: /foo
   " | kubectl apply -f -
   
   POD_NAME=$(kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o NAME | grep controller)
   kubectl exec -it -n ingress-nginx $POD_NAME -- curl -k -v -H 'Host: foo.bar' https://localhost/baz
   

8. Note requests to /baz or / 404, and are served by the nginx-errors service as expected – however these responses do not have the expected HSTS header. Requests to /foo have the correct header. Deleting the svc backend results in a 503 error – this is served by nginx itself, not the errors backend, and has the correct header.

Anything else we need to know:

• The header is added when no custom error backend is defined, or the code is not routed to the custom backend.
• On version 0.26.1, this worked fine.

/kind bug