cert-manager: ca.crt is empty after generating tls secret

Bugs should be filed for issues encountered whilst operating cert-manager. You should first attempt to resolve your issues through the community support channels, e.g. Slack, in order to rule out individual configuration errors. Please provide as much detail as possible.

Describe the bug: Hi there,

I created an issuer with letsencrypt staging and cloudflare acme dns01 provider, and later on I created a certificate for it. When it finished generating the certificate, I got the following error in the deployment logs:

0926 02:04:50.409208       1 logger.go:43] Calling GetOrder
I0926 02:04:50.562787       1 sync.go:76] cert-manager/controller/orders "level"=0 "msg"="updating Order resource status" "resource_kind"="Order" "resource_name"="test-cert-vlcntest-com-1823914796" "resource_namespace"="expr" 
E0926 02:04:50.576234       1 sync.go:79] cert-manager/controller/orders "msg"="failed to update status" "error"="error finalizing order: acme: urn:ietf:params:acme:error:orderNotReady: Order's status (\"valid\") is not acceptable for finalization" "resource_kind"="Order" "resource_name"="test-cert-vlcntest-com-1823914796" "resource_namespace"="expr" 
E0926 02:04:50.576266       1 controller.go:131] cert-manager/controller/orders "msg"="re-queuing item  due to error processing" "error"="[error finalizing order: acme: urn:ietf:params:acme:error:orderNotReady: Order's status (\"valid\") is not acceptable for finalization, Operation cannot be fulfilled on orders.certmanager.k8s.io \"test-cert-vlcntest-com-1823914796\": the object has been modified; please apply your changes to the latest version and try again]" "key"="expr/test-cert-vlcntest-com-1823914796" 

Expected behaviour: I expect that the ca.crt is present and not empty in the corresponding certificate secret.

Steps to reproduce the bug: I used the following configuration:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: admin@mycompany.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: mycompany-issuer-account-key
    dns01:
      providers:
        - name: cf-dns
          cloudflare:
            email: admin@mycompany.com
            apiKeySecretRef:
              name: cloudflare-api-key
              key: apiKey
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: test-cert-mycompany-com
  namespace: expr
spec:
  secretName: test-cert-mycompany-com-tls
  renewBefore: 360h # 15d
  commonName: test-cert.mycompany.com
  dnsNames:
  - test-cert.mycompany.com
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  acme:
    config:
    - dns01:
        provider: cf-dns
      domains:
      - test-cert.mycompany.com

Environment details:: Kubernetes:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud-provider/provisioner: AWS EK
  • cert-manager version (e.g. v0.4.0): cert-manager v0.10
  • Install method (e.g. helm or static manifests): via helm with default configuration

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 9
  • Comments: 25 (1 by maintainers)

Commits related to this issue

Most upvoted comments

Seems related to this: https://github.com/jetstack/cert-manager/issues/1571 (tls.crt is already the full chain in the case for letsencrypt issuer I believe)

And in one of the threads I found this snippet, which splits it out properly … not ideal but 😕

kubectl patch secret \
  -n <namespace> <secret name> \
  -p="{\"data\":{\"ca.crt\": \"$(kubectl get secret \
  -n <namespace> <secret name> \
  -o json -o=jsonpath="{.data.tls\.crt}" \
  | base64 -d | awk 'f;/-----END CERTIFICATE-----/{f=1}' - | base64 -w 0)\"}}"

Having same issue:

# kubectl get secret -n myapp my-domain -o yaml 
apiVersion: v1
data:
  ca.crt: ""
  tls.crt: ""
  tls.key: LS0tLS1CRUdJTiBSU0E...ZAo4VnJBS1NXL0lxM29qQk1sbjczSTNzeVpXNlBKbXBtVHBJa1hDQnhiTU1mNElreVpJc009Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
  annotations:
    cert-manager.io/certificate-name: my-domain
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: letsencrypt
  creationTimestamp: 2019-12-07T16:19:28Z
  name: my-domain
  namespace: myapp
  resourceVersion: "17355"
type: kubernetes.io/tls

And nginx can’t use it:

W1207 16:19:31.891802       7 controller.go:1125] Error getting SSL certificate "ga-monitor/gamonitor2-midtrans-work": local SSL certificate myapp/mydomain was not found. Using default certificate

So my app is served with Kubernetes Ingress Controller Fake Certificate

@fsniper

This bug, having the secret with only tls.key first, and then updating with the tls.crt effects behaviour of pods.

We’ve just introduced a new certificate controller implementation that no longer behaves like this, and will instead atomically issue the certificate and private key once both are ready.

You can test this out by using the recent v0.15.0-alpha.1 release and enabling the ExperimentalCertificateControllers feature gate with --feature-gates=ExperimentalCertificateControllers=true as a flag passed to the controller.

@javiertoledos when using the ACME issuer type, we are unable to retrieve the certificate of the CA that signed your certificate, hence why this field is empty. As this is working as intended, I think this issue can be closed 😃 feel free to re-open if you think there’s anything else actionable here however.

featureGates and installCRDs worked for me.

I confirm, v0.15.0-alpha.1 installed with Helm option --set featureGates="ExperimentalCertificateControllers=true" did worked for me, thank you

I had the same problem and found this issue. Tried v0.15.0-alpha.1 as suggested and it fixed the problem!

After upgrading to the alpha version, I removed the old Secret and Certificate and created the Certificate again. The Secret was created correctly by cert-manager a few seconds later.

For anyone using helm, enable that feature-gate by adding --set featureGates="ExperimentalCertificateControllers=true" to your install or upgrade command.

Thanks @munnerz !

I ran into this issue as well. In my case, I changed DNS records AFTER publishing the ingress. I deleted the namespace and re-applied after DNS was changed and it all worked the second time around.

Same issue.

➜ xooa-charts git:(nk-1.4.1) ✗ kubectl get secret -n testorg5 hlf-testorg5-tls -o yaml apiVersion: v1 data: ca.crt: “” tls.crt: “” tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNS (deleted for security reasons) kind: Secret metadata:

Do we have a fix for it yet?

This same set of helm charts was working for me in my old cluster. On my new cluster which has upgraded cert manager, I am getting this issue.

Same issue:

Name:         wildcard-ost-ai-tls
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: wildcard-ost-ai
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-prod

Type:  kubernetes.io/tls

Data
====
ca.crt:   0 bytes
tls.crt:  0 bytes
tls.key:  1675 bytes