cert-manager: TLS secret is not being created when using ClusterIssuer and ingress-shim

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened: I have chart “cert-manager-0.2.4” installed. I’ve installed it without rbac:

helm install --namespace kube-system --set rbac.create=false stable/cert-manager

I’ve created a cluster wide issuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-cluster-issuer
spec:
  acme:
    server: https://acme-staging.api.letsencrypt.org/directory
    email: me@somedomain.com
    privateKeySecretRef:
      name: letsencrypt-staging
    http01: {}

I have the following ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging-cluster-issuer
    ingress.kubernetes.io/proxy-body-size: 2048m
    nginx.org/proxy-connect-timeout: 30s
    nginx.org/proxy-read-timeout: 300s
  creationTimestamp: 2018-03-22T10:43:00Z
  generation: 16
  labels:
    app: goodly-kangaroo-letsencrypt-test
    chart: letsencrypt-test-0.1.5
    heritage: Tiller
    release: goodly-kangaroo
  name: goodly-kangaroo-httpserver
  namespace: test
  resourceVersion: "33812443"
  selfLink: /apis/extensions/v1beta1/namespaces/test/ingresses/goodly-kangaroo-httpserver
  uid: ca5b839c-2dbd-11e8-b51c-fa163e7911c3
spec:
  rules:
  - host: cert-test.mydomain.eu
    http:
      paths:
      - backend:
          serviceName: goodly-kangaroo-httpserver
          servicePort: 8080
        path: /
      - backend:
          serviceName: cm-test-letsencrypt-ssl-secret-enasc
          servicePort: 8089
        path: /.well-known/acme-challenge/IEpp9_YzfkasRZP8dLQVUf51YbN-Bgqo2x_yNv1Kc4g
  tls:
  - hosts:
    - cert-test.mydomain.eu
    secretName: test-letsencrypt-ssl-secret
status:
  loadBalancer:
    ingress:
    - ip: 192.168.0.220

http://cert-test.mydomain.eu works fine. http://cert-test.mydomain.eu/.well-known/acme-challenge/IEpp9_YzfkasRZP8dLQVUf51YbN-Bgqo2x_yNv1Kc4g also works fine BUT https://cert-test.mydomain.eu/ does not work (have no certificate). I can see the followings in the logs.

nginx ingress pod:

W0322 13:35:02.170376   25961 backend_ssl.go:46] error obtaining PEM from secret test/test-letsencrypt-ssl-secret: secret named test/test-letsencrypt-ssl-secret does not exist

cert-manager container:

I0322 13:35:38.565077       7 sync.go:242] Error preparing issuer for certificate: error waiting for key to be available for domain "cert-test.mydomain.eu": context deadline exceeded
E0322 13:35:38.621398       7 sync.go:190] [test/test-letsencrypt-ssl-secret] Error getting certificate 'test-letsencrypt-ssl-secret': secret "test-letsencrypt-ssl-secret" not found
E0322 13:35:38.621476       7 controller.go:196] certificates controller: Re-queuing item "test/test-letsencrypt-ssl-secret" due to error processing: error waiting for key to be available for domain "cert-test.mydomain.eu": context deadline exceeded
I0322 13:35:38.621555       7 controller.go:187] certificates controller: syncing item 'test/test-letsencrypt-ssl-secret'
I0322 13:35:38.621661       7 sync.go:107] Error checking existing TLS certificate: secret "test-letsencrypt-ssl-secret" not found
I0322 13:35:38.621734       7 sync.go:238] Preparing certificate with issuer
I0322 13:35:38.622728       7 prepare.go:239] Compare "" with "https://acme-staging.api.letsencrypt.org/acme/reg/5581886"

ingress-shim container:

I0322 13:35:38.567766      19 controller.go:147] ingress-shim controller: syncing item 'test/goodly-kangaroo-httpserver'
I0322 13:35:38.569372      19 sync.go:85] Certificate "test-letsencrypt-ssl-secret" for ingress "goodly-kangaroo-httpserver" already exists, not re-creating
I0322 13:35:38.569407      19 controller.go:161] ingress-shim controller: Finished processing work item "test/goodly-kangaroo-httpserver"
I0322 13:35:39.342743      19 controller.go:147] ingress-shim controller: syncing item 'test/goodly-kangaroo-httpserver'
I0322 13:35:39.342807      19 sync.go:85] Certificate "test-letsencrypt-ssl-secret" for ingress "goodly-kangaroo-httpserver" already exists, not re-creating
I0322 13:35:39.342829      19 controller.go:161] ingress-shim controller: Finished processing work item "test/goodly-kangaroo-httpserver"

Am I doing something wrong? My understanding is that the “test-letsencrypt-ssl-secret” secret should be created by the cert-manager/shim. Is that correct?

I couldn’t find instructions in the documentation about how to define the ingress. First I tried without the tls clause as I thought that the cert manager will add that part automatically. Then I added the tls part, but without the secretName definition… I think it would be good to have an example ingress definition in the doc.

What you expected to happen:

Certificate is automatically created through Let’s Encrypt for each new ingress definition that has the certmanager.k8s.io/cluster-issuer: "letsencrypt-staging-cluster-issuer" annotation.

How to reproduce it (as minimally and precisely as possible):

Set up a ClusterIssuer and use it through shim.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“8”, GitVersion:“v1.8.2”, GitCommit:“bdaeafa71f6c7c04636251031f93464384d54963”, GitTreeState:“clean”, BuildDate:“2017-10-24T19:48:57Z”, GoVersion:“go1.8.3”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“7”, GitVersion:“v1.7.3+coreos.0”, GitCommit:“42de91f04e456f7625941a6c4aaedaa69708be1b”, GitTreeState:“clean”, BuildDate:“2017-08-07T19:44:31Z”, GoVersion:“go1.8.3”, Compiler:“gc”, Platform:“linux/amd64”}

  • Cloud provider or hardware configuration**: OpenStack Newton

  • Install tools: kubeadm, helm v2.8.1

  • Others: The httpserver I’m using in the test chart is: adejonge/helloworld:latest

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 10
  • Comments: 15

Most upvoted comments

@novakg What I did is is kubectl exec -it -n <namespace> <cert-manager-pod> cert-manager sh and tried nslookup test.example.com which in my case yielded an unknown / bad address. After fixing that DNS issue I did the same and coupled that with a wget and all was 👌

Hi thanks for the reply. I cannot curl the test from within the cert-manager pod and it says nslookup: can't resolve '(null)': Name does not resolve

However in another pod in same cluster with same resolv.conf it resolves without issue. Unsure what is different about the helm chart that prevents it and I saw others above say they solved with DNS config but no one actually said what they did.

One person on Kubernetes Slack workspace suggested adding this to helm chart: --set extraArgs={"--dns01-recursive-nameservers=8.8.8.8:53,4.4.4.4:53"}

I can confirm that the problem was in my network set-up, not in cert manager: the cert manager was trying to check the exposed token through the external IP of our system which was routed incorrectly from inside the network. Cert manager works fine after I defined some DNS views in our DNS server with correct routings that make cert manager see the exposed tokens correctly.

Thanks everyone for the help.

I have solved this problem. As @vdbxavier correctly pointed out, you should make sure that cert-manager is able to connect to your service.

In my case it was that AWS Route53 had a Public and a Private Hosted zone (managed via TerraForm) that shared the domain example.com. When trying to do a challenge for test.example.com it was bouncing against the private one not correctly resolving the address.