cert-manager: certificate never appears ready

kubectl get cert

NAME                   READY   SECRET                     AGE
example.domain.com   False   example-tls   11m

The certificate is actually working but the status is not updated.

Logs show this in a perpetual loop:

kubectl -n cert-manager logs -l app=cert-manager

I1115 23:55:41.521223       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example.domain.com-2545306439" "related_resource_namespace"="namespace1" "resource_kind"="Certificate" "resource_name"="example.domain.com" "resource_namespace"="namespace1"
I1115 23:55:41.720351       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="namespace1/example.domain.com"

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 15 (4 by maintainers)

Most upvoted comments

This issue still seems to exist for me on v0.13.0

saltenhub on Jan 22, 2020

I am having the same issue on v0.12 using DNS01 verification with acmeDNS and Istio v1.4. The endless loop described above is spamming our logs with about 30 lines/sec, even though the certificate is already working and valid:

I1202 17:42:08.180354       1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.180379       1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.180571       1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.180600       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.372284       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.373234       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.374438       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.374573       1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.374622       1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.374984       1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.375054       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.577480       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.577540       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.580228       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.580594       1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.580717       1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.581894       1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.582025       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.775820       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.775882       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.777220       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.777405       1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.777475       1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.777901       1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.777962       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.972866       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.972943       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:08.974481       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.974630       1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.974688       1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.975061       1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:08.975121       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:09.177992       1 controller.go:135] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:09.178049       1 controller.go:129] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="istio-system/example_cert.example.com" 
I1202 17:42:09.179199       1 sync.go:379] cert-manager/controller/certificates "level"=0 "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:09.179345       1 sync.go:442] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest is in a Ready state, issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:09.179396       1 sync.go:445] cert-manager/controller/certificates "level"=0 "msg"="decoding certificate data" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:09.179745       1 sync.go:453] cert-manager/controller/certificates "level"=0 "msg"="checking if certificate stored on CertificateRequest is up to date" "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system" 
I1202 17:42:09.179817       1 sync.go:466] cert-manager/controller/certificates "level"=0 "msg"="CertificateRequest contains a valid certificate for issuance. Issuing certificate..." "related_resource_kind"="CertificateRequest" "related_resource_name"="example_cert.example.com-38051xxxx" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="example_cert.example.com" "resource_namespace"="istio-system"

EDIT: The issue seems to be that the controller is waiting for the CertificateRequest to complete, even though the CertificateRequest already states Certificate fetched from issuer successfully, but the Certificate itself states Waiting for CertificateRequest [...] to complete. If you need any additional info / log outputs etc. let me know, I am happy to help.

saltenhub on Dec 2, 2019