cert-manager: cert-manager stops reconciling certificate expiries

Describe the bug:

cert-manager doesn’t seem to reconcile certificate expiries, leading to certs expiring. This manifests for certificates issued by lets encrypt, and also a self-signed issuer. Note that all the following logs/snippets are for a certificate which has already expired.

Ingress object:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  name: happa
  namespace: giantswarm
  rules:
  - host: happa.mydomain.com
    http:
      paths:
      - backend:
          serviceName: happa
          servicePort: 8000
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - happa.mydomain.com
    secretName: happa-certs

Certificate object:

apiVersion: cert-manager.io/v1alpha3
kind: Certificate
metadata:
  name: happa-certs
  namespace: giantswarm
spec:
  dnsNames:
  - happa.domain.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-giantswarm
  secretName: happa-certs
status:
  conditions:
  - lastTransitionTime: "2020-09-14T14:32:49Z"
    message: Certificate is up to date and has not expired
    reason: Ready
    status: "True"
    type: Ready
  notAfter: "2020-11-19T14:22:14Z"

Certificate details at the time:

kg secret happa-certs -o "jsonpath={.data['tls\.crt']}" | base64 -d | openssl x509 -noout --enddate
notAfter=Nov 19 14:22:14 2020 GMT

Controller logs whilst adding renewBefore to the Certificate object:

I1124 11:37:16.133801       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133818       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133823       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133836       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133843       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133846       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133879       1 controller.go:152] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="giantswarm/happa"
I1124 11:37:16.133883       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133813       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133909       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.133944       1 sync.go:167] cert-manager/controller/ingress-shim "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="happa-certs" "related_resource_namespace"="giantswarm" "resource_kind"="Ingress" "resource_name"="happa" "resource_namespace"="giantswarm"
I1124 11:37:16.133976       1 sync.go:180] cert-manager/controller/ingress-shim "msg"="certificate resource is already up to date for ingress" "related_resource_kind"="Certificate" "related_resource_name"="happa-certs" "related_resource_namespace"="giantswarm" "resource_kind"="Ingress" "resource_name"="happa" "resource_namespace"="giantswarm"
I1124 11:37:16.134015       1 controller.go:162] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="giantswarm/happa"
I1124 11:37:16.134363       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.134382       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.134610       1 conditions.go:162] Found status change for Certificate "happa-certs" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2020-11-24 11:37:16.134603286 +0000 UTC m=+1798840.579961992
I1124 11:37:16.143905       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.143908       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.143932       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.143936       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.143942       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.143936       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.143971       1 controller.go:152] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="giantswarm/happa"
I1124 11:37:16.143977       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.144027       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.144049       1 sync.go:167] cert-manager/controller/ingress-shim "msg"="certificate already exists for ingress resource, ensuring it is up to date" "related_resource_kind"="Certificate" "related_resource_name"="happa-certs" "related_resource_namespace"="giantswarm" "resource_kind"="Ingress" "resource_name"="happa" "resource_namespace"="giantswarm"
I1124 11:37:16.144061       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.144080       1 sync.go:180] cert-manager/controller/ingress-shim "msg"="certificate resource is already up to date for ingress" "related_resource_kind"="Certificate" "related_resource_name"="happa-certs" "related_resource_namespace"="giantswarm" "resource_kind"="Ingress" "resource_name"="happa" "resource_namespace"="giantswarm"
I1124 11:37:16.144130       1 controller.go:162] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="giantswarm/happa"
I1124 11:37:16.144089       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="giantswarm/happa-certs"
I1124 11:37:16.144411       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.144423       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="giantswarm/happa-certs"
I1124 11:37:16.153140       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="giantswarm/happa-certs"

Certificate status after adding renewBefore:

Status:
  Conditions:
    Last Transition Time:  2020-11-19T15:30:17Z
    Message:               Certificate expired on Thu, 19 Nov 2020 14:22:14 UTC
    Reason:                Expired
    Status:                False
    Type:                  Ready
  Not After:               2020-11-19T14:22:14Z

Deleting the secret is the only way to get the certificate updated

Expected behaviour:

Certificates are renewed 30d before expiry (as renewBefore is not configured).

Steps to reproduce the bug:

Anything else we need to know?:

We have seen this behaviour across multiple clusters/cert-manager versions, and also for different certificates. This issue also manifests itself for KIAM certs which are created by a self-signed issuer.

Environment details::

• Kubernetes version: 1.17.13, 1.18.10 (probably others inbetween too)
• Cloud-provider/provisioner: custom - GiantSwarm
• cert-manager version: 0.9.0, 0.16.1, 1.0.2
• Install method: helm (custom, based on offical chart)

/kind bug