cert-manager: Ingress Priority - CM Ingress being ignored

Describe the bug: Attempting to create a certificate via annotations on an ingress via Helm chart.

Expected behaviour: A valid certificate

Steps to reproduce the bug: Install a web service using a Helm chart, having the Ingress resource created with the annotations to have the certificate automatically created by a ClusterIssuer

Anything else we need to know?:

Environment details::

Kubernetes version (e.g. v1.18.2): v1.18.2
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): kubeadm
cert-manager version (e.g. v0.4.0): 0.15.1
Install method (e.g. helm or static manifests): Helm

When I am installing a Flask App using a Helm chart, I enable ingress and TLS with the annotations on the Ingress. cert-manager then attempts to grab certs from letsencrypt, but I end up in a world of issues.

Currently, the most common issue I get is ultimately a 301, and see within nginx-ingress logs this

Error retrieving endpoints for the service cm-acme-http-solver-k8dkn: No endpoints for target port 8089 in service cm-acme-http-solver-k8dkn
2020/06/06 05:44:02 [emerg] 282#282: cannot load certificate "/etc/nginx/secrets/flask-test-tls": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
E0606 05:44:02.041765       1 controller.go:407] Error updating endpoints for [flask-test/flask-test]: Error reloading NGINX when updating endpoints: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "nginx: [emerg] cannot load certificate \"/etc/nginx/secrets/flask-test-tls\": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)\n"
finished with error: exit status 1
10.44.0.5 - - [06/Jun/2020:05:48:50 +0000] "GET /.well-known/acme-challenge/pFHMCJrmn3Cbl_KUtHGlfbvMlpdJ3j-AjWB3W9b1QIo HTTP/1.1" 301 169 "-" "Go-http-client/1.1" "-"

I can sometimes end up eventually getting a certificate by manually deleting the created ingress for the cm-acme and service, at random, I’ll eventually get a valid cert through this process. But if I leave failed, which is does 95% of the time, it will stay that way.

If I manually create the certificate resource in kubernetes with a manifest, instead of trying to do it through the ingress annotations, it works flawlessly, but this is less than ideal since annotations on Ingress are really the proper way to do it with Helm Charts.

/kind bug

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 17 (7 by maintainers)

Most upvoted comments

After some more tests i finally got it working.

As you mentioned, the main problem is that two nginx configurations are generated with identical host names.

I first started looking for a solution to prevent this and found the nginx-ingress supports Mergable-Ingress via annotations. This requires to set annotations on the main ingress to mark it the “master” and on the cert-manager created ingress to mark it as “minion”. This can be achieved via ingressTemplate on the Issuer as described.

Using this setup i got it working to generate only a single nginx config.

However i saw this error on the nginx reload which prevented the config to be applied:

cannot load certificate \"/etc/nginx/secrets/default-demo-cert\": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)

#2712 #2707 describe what might be the cause of this error.

By looking for a solution i stumbled across the “cert-manager.io/issue-temporary-certificate” and “acme.cert-manager.io/http01-edit-in-place” annotations.

“Edit in place” seems to be exactly what we need for the nginx maintaned ingress.

I ended up with the following ingress configuration that supports creating new certificates and successfully completing the acme-challenge. Note: with this ingress configs no Mergable-Ingress annotations are needed

apiVersion: networking.k8s.io/v1beta1 # for versions before 1.14 use extensions/v1beta1
kind: Ingress
metadata:
  name: cloud-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: letsencrypt-prod-issuer  # reference to the ClusterIssuer resource
    cert-manager.io/issue-temporary-certificate: "true" # temporary cert required for nginx to be able to reload
    acme.cert-manager.io/http01-edit-in-place: "true" # important to merge with existing ingress resource into a single nginx config file
    #ingress.kubernetes.io/ssl-redirect: "false" # avoid http > https redirect ( acme-challenge was still successful even with the redirect enabled )
spec:
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nginx-svc
              servicePort: 80
  tls: # < placing a host in the TLS config will indicate a certificate should be created
    - secretName: example-prod-cert # < cert-manager will store the created certificate in this secret.
      hosts:
        - www.example.com

I was not able to test the certificate renew process, but i would guess now that the ingress cert issue process is working it should also be supported.

michaelhaessig on Jul 7, 2020

The annotation works for me.

jslay88 on Jul 21, 2020