ingress-nginx: Permanent Redirect with Cert-Manager conflicts with Well-Known ACME validator

NGINX Ingress controller version:

NGINX Ingress controller
  Release:       v0.41.2
  Build:         d8a93551e6e5798fc4af3eb910cef62ecddc8938
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.4

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.12-gke.1205", GitCommit:"160736941f00e54acb1c0a4647166b6f6eb211d9", GitTreeState:"clean", BuildDate:"2021-01-05T23:44:13Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GKE with private nodes
• OS (e.g. from /etc/os-release): ID=COS, 13310.1041.24
• Kernel (e.g. uname -a): Linux gke-foo-prod-foo-prod-n-bbdae2b2-4bfo 5.4.49+ #1 SMP Sun Oct 18 19:43:35 PDT 2020 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/Linux

What happened:

We’re deploying an Ingress with annotation to permantently redirect to a new domain, instead of using the actual backend webserver for this.

Additionally, ssl certs are managed using cert-manager.

Unfortunately, ingress-nginx does create the well-known ingress, but includes the 301 redirect from the annotation nevertheless. As such, proxy_pass to the ACME solver pod is essentially ignored.

As the 301 redirect occurs, Let’s Encrypt is unable to validate the certificate request.

What you expected to happen:

While the annotations should be used on the locations, applying a redirect on the well-known URL location isn’t what Let’s Encrypt expects.

Instead, requests to the well-known ACME Challenges should pass to the ACME solver pod.

How to reproduce it:

Minimum ingress YAML:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    cert-manager.io/issue-temporary-certificate: "true"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/permanent-redirect: https://example.ch$uri$is_args$args
  labels:
    app: live-ch-ch
  name: shopware-live-ch-ch-redirects
  namespace: shopware-platform-385-live-ch
spec:
  rules:
  - host: www.example.ch
    http:
      paths:
      - backend:
          serviceName: shopware-live-ch-ch
          servicePort: 80
        path: /
        pathType: ImplementationSpecific
  - host: shop.example.ch
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-pgkqx
          servicePort: 8089
        path: /.well-known/acme-challenge/xxx
        pathType: ImplementationSpecific
      - backend:
          serviceName: cm-acme-http-solver-pgkqx
          servicePort: 8089
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - www.example.ch
    - shop.example.ch
    secretName: live-ch-ch-redirect-tls

This generates the following ingress nginx.conf (excerpt):

        ## start server shop.example.ch
        server {
                server_name shop.example.ch ;

                listen 80  ;
                listen 443  ssl http2 ;

                set $proxy_upstream_name "-";

                ssl_certificate_by_lua_block {
                        certificate.call()
                }

                location /.well-known/acme-challenge/xxx/ {

                        set $namespace      "shopware-platform-385-live-ch";
                        set $ingress_name   "shopware-live-ch-ch-redirects";
                        set $service_name   "";
                        set $service_port   "";
                        set $location_path  "/.well-known/acme-challenge/xxx/";
(...)
                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_timeout             0;
                        proxy_next_upstream_tries               3;

                        return 301 https://example.ch$uri$is_args$args;

                        proxy_pass http://upstream_balancer;

                        proxy_redirect                          off;

                }

CURLing the endpoint, results in a redirect instead of serving the ACME validation response.

curl -i -k 'https://shop.example.ch/.well-known/acme-challenge/xxx'
HTTP/2 301
date: Mon, 08 Feb 2021 12:18:47 GMT
content-type: text/html
content-length: 162
location: https://example.ch/.well-known/acme-challenge/xxx
strict-transport-security: max-age=15724800; includeSubDomains

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

After removing the redirect-annotation, cert-manager and subsequently Let’s Encrypt were able to validate the request without issues. Afterwards, adding the redirect back did also work as expected.

I traced this back to nginx.tmpl#L1329.

Most likely, this could be solved by ensuring that a redirect is not added if the location starts with “.well-known/acme-challenge”. I could draft a PR, but my go-template-skills are to be improved.

Additionally (not related to this…), a lot of configuration could be removed if a redirect is present: there’s no need for proxy_pass directives or similar.

Anything else we need to know:

Switching to DNS-solver instead of HTTP would solve the underlying validation issue. Due to corporate constraints, we’re unable to use DNS to validate the ACME requests.

/kind bug