cert-manager: Operation cannot be fulfilled

Same issue exists in V1.1.0

v1.1.0 is released now

Reopened

@kaypeter87 surfacing of rate limit issues is improved in v1.1

It appears my certificate having the issue has successfully been issued. I can safely say that it took 48 hours to issue and it appears to be a rate limiting problem. What worries me is that there were no indication of rate limiting in the debug logs and other new certificates were being issued during this problem. I cannot say, as there are many variables, if let’s encrypt or cert-manager was failing. I do know that a certificate was failing to be issued when there were no indications of rate limiting.

I appear to be seeing the same behavior as @kaypeter87 above, and thought that I had a workaround to offer. Instead, this behavior quickly resurfaced and appears to be intermittent?

Due to the intermittent nature of this problem I am starting to suspect that it stems from rate-limiting, but can’t offer any concrete evidence. letsdebug.net reports no problems with any subdomain + DNS-01, and I’m not sure how else to debug for rate-limiting.

As a side note: does letsdebug.net just always say “OK” no matter what?? I have yet to see it report a problem…

Verbose details can be found below

---

I have a fresh Kubernetes 1.19.2 cluster with a small-scale application running in the default namespace. I have also installed cert-manager in a namespace with the same name. I’ve configured an ACMEDNS + LetsEncrypt ClusterIssuer that appears to be working in the default namespace for issuing wildcard certs via LetsEncrypt DNS-01.

$ kubectl get cert
NAME               READY   SECRET             AGE
ndslabs-tls-auth   True    ndslabs-tls-auth   27h
ndslabs-tls-open   True    ndslabs-tls-open   27h
ndslabs-tls-root   True    ndslabs-tls-root   27h

In another namespace, I have a similar ingress rule set up that seems to be stuck:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: acmedns-issuer-prod
    kubernetes.io/ingress.class: nginx
    ndslabs.org/updated: 2020-09-27 04:34:10.235391921 +0000 UTC m=+96392.482403672
    nginx.ingress.kubernetes.io/auth-signin: https://www.hub.cheesehub.org/login/
    nginx.ingress.kubernetes.io/auth-url: https://www.hub.cheesehub.org/cauth/auth
  name: s2mvbc-sqlinjection-ingress
  namespace: sigite22
spec:
  rules:
  - host: s2mvbc-sqlinjection.hub.cheesehub.org
    http:
      paths:
      - backend:
          serviceName: s2mvbc-sqlinjection
          servicePort: 8888
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - '*.hub.cheesehub.org'
    secretName: s2mvbc-sqlinjection-tls

The Certificate / CertificateRequest are stuck with an Order that has no events and no Challenge resources are created:

$ kubectl get cert,certificaterequest,order,challenge -n sigite22
I0927 05:32:41.383477   23697 request.go:645] Throttling request took 1.110302164s, request: GET:https://192.168.0.4:6443/apis/autoscaling/v1?timeout=32s
I0927 05:32:51.383573   23697 request.go:645] Throttling request took 3.794775254s, request: GET:https://192.168.0.4:6443/apis/certificates.k8s.io/v1beta1?timeout=32s
I0927 05:33:01.388882   23697 request.go:645] Throttling request took 5.119265571s, request: GET:https://192.168.0.4:6443/apis/coordination.k8s.io/v1beta1?timeout=32s
I0927 05:33:11.389011   23697 request.go:645] Throttling request took 7.795506954s, request: GET:https://192.168.0.4:6443/apis/acme.cert-manager.io/v1alpha3?timeout=32s
NAME                                                  READY   SECRET                    AGE
certificate.cert-manager.io/s2mvbc-sqlinjection-tls   False   s2mvbc-sqlinjection-tls   59m

NAME                                                               READY   AGE
certificaterequest.cert-manager.io/s2mvbc-sqlinjection-tls-7w42b   False   59m

NAME                                                                  STATE   AGE
order.acme.cert-manager.io/s2mvbc-sqlinjection-tls-7w42b-2903943313           59m

No Challenge resource appears to be created, and the Order resource does not appear to have any events associated with it:

$ kubectl describe order -n sigite22
Name:         s2mvbc-sqlinjection-tls-7w42b-2903943313
Namespace:    sigite22
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: s2mvbc-sqlinjection-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: s2mvbc-sqlinjection-tls-r2c6m
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2020-09-27T04:34:10Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  s2mvbc-sqlinjection-tls-7w42b
    UID:                   87f7af68-f598-4085-9d62-8709bfbb66b4
  Resource Version:        406367
  Self Link:               /apis/acme.cert-manager.io/v1/namespaces/sigite22/orders/s2mvbc-sqlinjection-tls-7w42b-2903943313
  UID:                     6240089a-c5e8-433f-9135-c21bde6af1aa
Spec:
  Dns Names:
    *.hub.cheesehub.org
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   acmedns-issuer-prod
  Request:  <REDACTED>
Events:     <none>

As a test, I added the root domain alongside the wildcard in the TLS section and that initially appeared to fix things but the behavior came back:

  tls:
  - hosts:
    - hub.cheesehub.org
    - '*.hub.cheesehub.org'