cert-manager: requestmanager_controller got stuck in a loop and stopped generating new certificates afterward

Describe the bug: At some point, it seems that the communication between the cert-manager-cainjector and ServerAPI stopped working (we received few EOF logs and subsequently “Successfully Reconciled” logs in the cert-manager-cainjector). However, after the communication restarted, we started receiving:

1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item  due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="default/stan-client-tls"

After another while (like 10s), the controller moved further in the processing of items, but outputted this log for all the previous logs:

I0112 15:48:53.799058       1 requestmanager_controller.go:196] cert-manager/controller/CertificateRequestManager "msg"="Multiple matching CertificateRequest resources exist, delete one of them. This is likely an error and should be reported on the issue tracker!" "key"="default/stan-client-tls"

Afterward, the generation of this certificate stopped altogether.

In the Kubernetes environment, we could see that multiple CertificateRequest objects have been generated for stan-client-tls Certificate with the same revision number. So probably, the client interface (https://github.com/jetstack/cert-manager/blob/cdc53b65cbd344dbef64f0c5c22e6070e79c5b5c/pkg/controller/certificates/requestmanager/requestmanager_controller.go#L339) was fully working and creating new instances, while certificateRequestLister was unable to get proper current state (https://github.com/jetstack/cert-manager/blob/cdc53b65cbd344dbef64f0c5c22e6070e79c5b5c/pkg/controller/certificates/requestmanager/requestmanager_controller.go#L165).

Expected behaviour: The controller should probably delete the unused CertificateRequests objects and continue with creating new ones until one of them succeeds.

Environment details::

Kubernetes version: 1.17.9
Cloud-provider/provisioner: Azure
cert-manager version: 1.0.3
Install method: Helm

/kind bug

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 3
Comments: 24 (1 by maintainers)

Most upvoted comments

For the record, for other OVH users having this issue and looking for a workaround/quick fix :

restart the API Server (using POST /cloud/project/{serviceName}/kube/{kubeId}/restart)
delete extra CertificateRequests that are duplicate

pdesgarets on Oct 26, 2021

I’m facing the same issue in OVH… Multiple CertificateRequests created and multiple entries like this in cert-manager pod log:

E0330 17:07:08.164430       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="cert-manager-test/selfsigned-cert"

marinoborges on Mar 30, 2021

Same issue on OVH cloud provider, as the certmanager controller continues to spawn new CertificateRequest objects, without ever detecting them.

Is there any progress ?

devlifealways on Aug 8, 2021

Same issue for me, infinete creation of certificaterequest, one every 30-40 seconds; no orders and no challenges created. Every certificate request has zero events and no status. Some days before I’ve create three certificates with success.

This is a sample log from cert-manager:

W0405 14:31:53.353021       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
I0405 14:35:36.753993       1 conditions.go:173] Setting lastTransitionTime for Certificate "tls-cert" condition "Issuing" to 2021-04-05 14:35:36.753984598 +0000 UTC m=+1191568.543064875
I0405 14:35:36.754048       1 conditions.go:173] Setting lastTransitionTime for Certificate "tls-cert" condition "Ready" to 2021-04-05 14:35:36.754044213 +0000 UTC m=+1191568.543124449
E0405 14:35:36.864108       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="evo-seven/tls-cert"
I0405 14:35:36.864205       1 conditions.go:173] Setting lastTransitionTime for Certificate "tls-cert" condition "Ready" to 2021-04-05 14:35:36.864201839 +0000 UTC m=+1191568.653282102
E0405 14:35:37.098787       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tls-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="evo-seven/tls-cert"
E0405 14:35:42.134625       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:35:48.154531       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:35:55.184975       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:36:04.213143       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:36:17.238978       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:36:38.258443       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:37:13.303129       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:37:48.337118       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:38:23.372114       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:38:58.527014       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
W0405 14:39:33.357965       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
E0405 14:39:33.550267       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:40:08.575066       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:40:44.057994       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:41:19.102416       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:41:54.138793       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:42:29.167208       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:43:04.189638       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:43:39.488843       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:44:14.517500       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:44:50.404372       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:45:25.445090       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:46:00.480316       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:46:35.577328       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
W0405 14:46:46.361263       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
E0405 14:47:10.598888       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:47:45.876135       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:48:21.060154       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:48:56.403248       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:49:31.791474       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:50:07.610210       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:50:42.690564       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:51:17.790211       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:51:52.813480       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:52:29.144392       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:53:04.905653       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:53:40.435520       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:54:16.079388       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:54:51.104798       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
W0405 14:54:58.364606       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
E0405 14:55:26.209327       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:56:01.233293       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:56:36.265480       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:57:11.321807       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:57:46.349915       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:58:21.374360       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:58:56.406887       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 14:59:31.430364       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:00:06.458617       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:00:41.821982       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:01:17.112874       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:01:52.158603       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
W0405 15:01:56.367271       1 warnings.go:67] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
E0405 15:02:27.200101       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:03:02.262696       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:03:37.368137       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:04:12.416447       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:04:47.666610       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:05:22.699331       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:05:57.726778       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"
E0405 15:06:32.751335       1 controller.go:158] cert-manager/controller/CertificateRequestManager "msg"="re-queuing item due to error processing" "error"="failed whilst waiting for CertificateRequest to exist - this may indicate an apiserver running slowly. Request will be retried" "key"="evo-seven/tls-cert"

cainjector is not involved, no special logs inside.

frabe1579 on Apr 5, 2021