cert-manager: TLS secret is not being created when using ClusterIssuer and ingress-shim
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened: I have chart “cert-manager-0.2.4” installed. I’ve installed it without rbac:
helm install --namespace kube-system --set rbac.create=false stable/cert-manager
I’ve created a cluster wide issuer:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging-cluster-issuer
spec:
acme:
server: https://acme-staging.api.letsencrypt.org/directory
email: me@somedomain.com
privateKeySecretRef:
name: letsencrypt-staging
http01: {}
I have the following ingress definition:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-staging-cluster-issuer
ingress.kubernetes.io/proxy-body-size: 2048m
nginx.org/proxy-connect-timeout: 30s
nginx.org/proxy-read-timeout: 300s
creationTimestamp: 2018-03-22T10:43:00Z
generation: 16
labels:
app: goodly-kangaroo-letsencrypt-test
chart: letsencrypt-test-0.1.5
heritage: Tiller
release: goodly-kangaroo
name: goodly-kangaroo-httpserver
namespace: test
resourceVersion: "33812443"
selfLink: /apis/extensions/v1beta1/namespaces/test/ingresses/goodly-kangaroo-httpserver
uid: ca5b839c-2dbd-11e8-b51c-fa163e7911c3
spec:
rules:
- host: cert-test.mydomain.eu
http:
paths:
- backend:
serviceName: goodly-kangaroo-httpserver
servicePort: 8080
path: /
- backend:
serviceName: cm-test-letsencrypt-ssl-secret-enasc
servicePort: 8089
path: /.well-known/acme-challenge/IEpp9_YzfkasRZP8dLQVUf51YbN-Bgqo2x_yNv1Kc4g
tls:
- hosts:
- cert-test.mydomain.eu
secretName: test-letsencrypt-ssl-secret
status:
loadBalancer:
ingress:
- ip: 192.168.0.220
http://cert-test.mydomain.eu works fine. http://cert-test.mydomain.eu/.well-known/acme-challenge/IEpp9_YzfkasRZP8dLQVUf51YbN-Bgqo2x_yNv1Kc4g also works fine BUT https://cert-test.mydomain.eu/ does not work (have no certificate). I can see the followings in the logs.
nginx ingress pod:
W0322 13:35:02.170376 25961 backend_ssl.go:46] error obtaining PEM from secret test/test-letsencrypt-ssl-secret: secret named test/test-letsencrypt-ssl-secret does not exist
cert-manager container:
I0322 13:35:38.565077 7 sync.go:242] Error preparing issuer for certificate: error waiting for key to be available for domain "cert-test.mydomain.eu": context deadline exceeded
E0322 13:35:38.621398 7 sync.go:190] [test/test-letsencrypt-ssl-secret] Error getting certificate 'test-letsencrypt-ssl-secret': secret "test-letsencrypt-ssl-secret" not found
E0322 13:35:38.621476 7 controller.go:196] certificates controller: Re-queuing item "test/test-letsencrypt-ssl-secret" due to error processing: error waiting for key to be available for domain "cert-test.mydomain.eu": context deadline exceeded
I0322 13:35:38.621555 7 controller.go:187] certificates controller: syncing item 'test/test-letsencrypt-ssl-secret'
I0322 13:35:38.621661 7 sync.go:107] Error checking existing TLS certificate: secret "test-letsencrypt-ssl-secret" not found
I0322 13:35:38.621734 7 sync.go:238] Preparing certificate with issuer
I0322 13:35:38.622728 7 prepare.go:239] Compare "" with "https://acme-staging.api.letsencrypt.org/acme/reg/5581886"
ingress-shim container:
I0322 13:35:38.567766 19 controller.go:147] ingress-shim controller: syncing item 'test/goodly-kangaroo-httpserver'
I0322 13:35:38.569372 19 sync.go:85] Certificate "test-letsencrypt-ssl-secret" for ingress "goodly-kangaroo-httpserver" already exists, not re-creating
I0322 13:35:38.569407 19 controller.go:161] ingress-shim controller: Finished processing work item "test/goodly-kangaroo-httpserver"
I0322 13:35:39.342743 19 controller.go:147] ingress-shim controller: syncing item 'test/goodly-kangaroo-httpserver'
I0322 13:35:39.342807 19 sync.go:85] Certificate "test-letsencrypt-ssl-secret" for ingress "goodly-kangaroo-httpserver" already exists, not re-creating
I0322 13:35:39.342829 19 controller.go:161] ingress-shim controller: Finished processing work item "test/goodly-kangaroo-httpserver"
Am I doing something wrong? My understanding is that the “test-letsencrypt-ssl-secret” secret should be created by the cert-manager/shim. Is that correct?
I couldn’t find instructions in the documentation about how to define the ingress. First I tried without the tls clause as I thought that the cert manager will add that part automatically. Then I added the tls part, but without the secretName definition… I think it would be good to have an example ingress definition in the doc.
What you expected to happen:
Certificate is automatically created through Let’s Encrypt for each new ingress definition that has the certmanager.k8s.io/cluster-issuer: "letsencrypt-staging-cluster-issuer"
annotation.
How to reproduce it (as minimally and precisely as possible):
Set up a ClusterIssuer and use it through shim.
Anything else we need to know?:
Environment:
-
Kubernetes version (use
kubectl version
): Client Version: version.Info{Major:“1”, Minor:“8”, GitVersion:“v1.8.2”, GitCommit:“bdaeafa71f6c7c04636251031f93464384d54963”, GitTreeState:“clean”, BuildDate:“2017-10-24T19:48:57Z”, GoVersion:“go1.8.3”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“7”, GitVersion:“v1.7.3+coreos.0”, GitCommit:“42de91f04e456f7625941a6c4aaedaa69708be1b”, GitTreeState:“clean”, BuildDate:“2017-08-07T19:44:31Z”, GoVersion:“go1.8.3”, Compiler:“gc”, Platform:“linux/amd64”} -
Cloud provider or hardware configuration**: OpenStack Newton
-
Install tools: kubeadm, helm v2.8.1
-
Others: The httpserver I’m using in the test chart is: adejonge/helloworld:latest
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 10
- Comments: 15
@novakg What I did is is
kubectl exec -it -n <namespace> <cert-manager-pod> cert-manager sh
and triednslookup test.example.com
which in my case yielded an unknown / bad address. After fixing that DNS issue I did the same and coupled that with awget
and all was 👌Hi thanks for the reply. I cannot curl the test from within the
cert-manager
pod and it saysnslookup: can't resolve '(null)': Name does not resolve
However in another pod in same cluster with same
resolv.conf
it resolves without issue. Unsure what is different about the helm chart that prevents it and I saw others above say they solved with DNS config but no one actually said what they did.One person on Kubernetes Slack workspace suggested adding this to helm chart:
--set extraArgs={"--dns01-recursive-nameservers=8.8.8.8:53,4.4.4.4:53"}
I can confirm that the problem was in my network set-up, not in cert manager: the cert manager was trying to check the exposed token through the external IP of our system which was routed incorrectly from inside the network. Cert manager works fine after I defined some DNS views in our DNS server with correct routings that make cert manager see the exposed tokens correctly.
Thanks everyone for the help.
I have solved this problem. As @vdbxavier correctly pointed out, you should make sure that
cert-manager
is able to connect to your service.In my case it was that
AWS Route53
had a Public and a Private Hosted zone (managed via TerraForm) that shared the domainexample.com
. When trying to do a challenge fortest.example.com
it was bouncing against the private one not correctly resolving the address.