cert-manager: Error initializing issuer: context deadline exceeded

Describe the bug:

Failed to create clusterissuer, the following is the error log of cert-manager:

$ kubectl -n cert-manager logs -f cert-manager-59d959c87c-qsxbc
I0914 17:33:32.129086       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I0914 17:33:32.129113       1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-venafi" 
I0914 17:33:32.129130       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-revision-manager" 
I0914 17:33:32.129143       1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-selfsigned" 
I0914 17:33:32.129656       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-key-manager" 
I0914 17:33:32.129744       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I0914 17:33:32.133919       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I0914 17:33:32.133980       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-approver" 
I0914 17:33:32.133996       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I0914 17:33:32.134022       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-issuing" 
I0914 17:33:32.134044       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I0914 17:33:32.134078       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-readiness" 
I0914 17:33:32.236494       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I0914 17:33:32.236658       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-trigger" 
I0914 17:33:32.236691       1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-request-manager" 
I0914 17:35:53.546363       1 setup.go:219] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 
E0914 17:36:03.547090       1 setup.go:259] cert-manager/controller/clusterissuers "msg"="failed to register an ACME account" "error"="context deadline exceeded" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 
I0914 17:36:03.547134       1 conditions.go:95] Setting lastTransitionTime for Issuer "letsencrypt-staging" condition "Ready" to 2021-09-14 17:36:03.547122659 +0800 CST m=+151.486801748
E0914 17:36:03.547165       1 sync.go:60] cert-manager/controller/clusterissuers "msg"="error setting up issuer" "error"="context deadline exceeded" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 
E0914 17:36:03.548101       1 controller.go:163] cert-manager/controller/clusterissuers "msg"="re-queuing item due to error processing" "error"="context deadline exceeded" "key"="letsencrypt-staging" 
I0914 17:36:08.549436       1 setup.go:219] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 
E0914 17:36:18.550061       1 setup.go:259] cert-manager/controller/clusterissuers "msg"="failed to register an ACME account" "error"="context deadline exceeded" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 
I0914 17:36:18.550114       1 conditions.go:95] Setting lastTransitionTime for Issuer "letsencrypt-staging" condition "Ready" to 2021-09-14 17:36:18.550108288 +0800 CST m=+166.489787378
E0914 17:36:18.550143       1 sync.go:60] cert-manager/controller/clusterissuers "msg"="error setting up issuer" "error"="context deadline exceeded" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 
E0914 17:36:18.550260       1 controller.go:163] cert-manager/controller/clusterissuers "msg"="re-queuing item due to error processing" "error"="context deadline exceeded" "key"="letsencrypt-staging" 
I0914 17:36:28.551558       1 setup.go:219] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1" 

$ kubectl -n cert-manager logs -f cert-manager-cainjector-6ff65c66fd-z8tbw
I0915 10:48:50.725518       1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1" 
E0915 10:48:50.733174       1 controller.go:175] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="unable to update target object with new CA data" "error"="Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"cert-manager-webhook\": the object has been modified; please apply your changes to the latest version and try again" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1" 
E0915 10:48:50.733316       1 controller.go:304] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Reconciler error" "error"="Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"cert-manager-webhook\": the object has been modified; please apply your changes to the latest version and try again" "name"="cert-manager-webhook" "namespace"="" 
I0915 10:48:50.736253       1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1" 
I0915 10:48:50.741147       1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1" 

$ kubectl -n cert-manager logs -f cert-manager-webhook-6984c6cbbc-zhwjw 
W0915 10:48:33.783799       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0915 10:48:33.785307       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0915 10:48:33.785502       1 webhook.go:70] cert-manager/webhook "msg"="using dynamic certificate generating using CA stored in Secret resource"  "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I0915 10:48:33.788419       1 server.go:138] cert-manager/webhook "msg"="listening for insecure healthz connections"  "address"=":6080"
I0915 10:48:33.788536       1 server.go:169] cert-manager/webhook "msg"="listening for secure connections"  "address"=":10250"
I0915 10:48:33.789029       1 server.go:201] cert-manager/webhook "msg"="registered pprof handlers"  
I0915 10:48:34.819424       1 dynamic_source.go:272] cert-manager/webhook "msg"="Updated serving TLS certificate" 

Anything else we need to know?:

cat staging-clusterissuer.yaml 
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: admin@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class:  nginx

$ kubectl create -f staging-clusterissuer.yaml 
clusterissuer.cert-manager.io/letsencrypt-staging created

$ kubectl -n cert-manager describe clusterissuer letsencrypt-staging    
Name:         letsencrypt-staging
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2021-09-14T09:47:49Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:acme:
          .:
          f:email:
          f:privateKeySecretRef:
            .:
            f:name:
          f:server:
          f:solvers:
    Manager:         kubectl
    Operation:       Update
    Time:            2021-09-14T09:47:49Z
  Resource Version:  270254
  Self Link:         /apis/cert-manager.io/v1/clusterissuers/letsencrypt-staging
  UID:               dcb7e1b4-145e-4a3f-9bca-d663738699c9
Spec:
  Acme:
    Email:            admin@example.com
    Preferred Chain:  
    Private Key Secret Ref:
      Name:  letsencrypt-staging
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Events:
  Type     Reason         Age                  From          Message
  ----     ------         ----                 ----          -------
  Warning  ErrInitIssuer  83s (x7 over 7m38s)  cert-manager  Error initializing issuer: context deadline exceeded

Environment details::

  • Kubernetes version: v1.18.18
  • cert-manager version: 1.5.3
  • Install method: static manifests

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 25 (3 by maintainers)

Most upvoted comments

We’ve now published cert-manager 1.8.2 and 1.7.3 which include these fixes. @omBratteng mentioned on slack that they tested and it worked as expected with Sectigo!

Thanks very much to everyone who was involved in this issue - I’ll close it now as it seems to be completed ❤️

This should be fixed by #5226 which is backported into cert-manager 1.8 and 1.7 by #5231 and #5232.

We should hopefully be able to release new versions of cert-manager with these fixes soon, either today, tomorrow or at the start of next week 👍

same issue with zerossl: Error initializing issuer: context deadline exceeded

Environment details:: Kubernetes version: v1.21.4 cert-manager version: 1.8.0 Install method: helm chart

I also have the same issue, and I don’t even have that many domains to be migrated from Let’s Encrypt to ZeroSSL. The EAB registration itself is failing. I tried this on a different cluster few days ago, and after few hours of not working it automatically did.

Today I asked the ZeroSSL Support team regarding the issue and was given the following response:

This issue is caused by downtimes on our provider’s side. We have a ticket raised with them, and we have already pointed out the urgency of solving the problem. The outages seem to happen periodically when the provider faces higher loads. Unfortunately, we cannot impact this situation nor the occasions when they happen.

So its possible that the increasing timeout to a larger value might only decrease the probability of failure condition but not completely eliminate it.

I’m running into the same issue with ZeroSSL.

Yes, a configureable context deadline would solve it.

I will verify the actual duration it took tomorrow, but I think it was 25 seconds.

@hadogenes can you please tell us what exactly you corrected on your DNS config ?