cert-manager: The certificate request has failed to complete and will be retried: Failed to decode returned certificate: error decoding certificate PEM block

I can confirm this is a bug in these functions introduced in cert-manager 1.4. It can be reproduced in unit tests with the following patch applied against the master branch:

diff --git a/pkg/util/pki/parse_test.go b/pkg/util/pki/parse_test.go
index 1b49b2081..2e86d5c4e 100644
--- a/pkg/util/pki/parse_test.go
+++ b/pkg/util/pki/parse_test.go
@@ -237,11 +237,11 @@ func mustCreateBundle(t *testing.T, issuer *testBundle, name string) *testBundle
 
 func TestParseSingleCertificateChain(t *testing.T) {
 	root := mustCreateBundle(t, nil, "root")
-	intA1 := mustCreateBundle(t, root, "intA-1")
-	intA2 := mustCreateBundle(t, intA1, "intA-2")
+	intA1 := mustCreateBundle(t, root, "root")
+	intA2 := mustCreateBundle(t, intA1, "root")
 	intB1 := mustCreateBundle(t, root, "intB-1")
 	intB2 := mustCreateBundle(t, intB1, "intB-2")
-	leaf := mustCreateBundle(t, intA2, "leaf")
+	leaf := mustCreateBundle(t, intA2, "root")
 	random := mustCreateBundle(t, nil, "random")
 
 	joinPEM := func(first []byte, rest ...[]byte) []byte {

cert-manager 1.4.1 has just been released, which includes the fix for this issue 👍

Thank you to everyone involved in raising it and providing the details we needed to diagnose it and fix it!

For reference, the actual bug has been in cert-manager since Jan 2019:

https://github.com/jetstack/cert-manager/blob/f489c7df0210c5066a057344265cf614d72ba0ee/pkg/util/pki/csr.go#L464-L479

specifically these lines.

cert-manager 1.4 just started using this older function in other places, which surfaced the old bug.

I’ll raise a PR to fix; we’ll almost certainly release cert-manager 1.4.1 as a bugfix release for this

Hi @maelvls, thanks a lot for looking into the issue. For me the issue is happening not due to using vault. I have a self-signed issuer created root CA, which then use to sign leaf certificate. Example manifest:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: default
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: root-ca-cert
  namespace: default
spec:
  isCA: true
  commonName: cert-cn
  duration: 2160h
  renewBefore: 360h
  dnsNames: 
  - cert-cn
  secretName: root-ca-cert
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 3072
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-issuer
  namespace: default
spec:
  ca:
    secretName: root-ca-cert
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: leaf-cert
  namespace: default
spec:
  commonName: cert-cn
  duration: 2160h
  renewBefore: 360h
  dnsNames: 
  - cert-cn
  secretName: leaf-cert
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 3072
  issuerRef:
    name: ca-issuer
    kind: Issuer

But when I changed the ‘commonName’ between CA and the certificate, I’m able to create the certificate.

BTW, as already mentioned here using different ‘commonName’ is only a worst case option for us.

@rumanzo I was able to reproduce the issue using Vault v1.7.3 and cert-manager v1.4.0. I also tested with cert-manager v1.3.1 and it does not have the issue. That looks like a regression.

Steps I took:

k3d cluster create
helm upgrade --install cert-manager jetstack/cert-manager --version 1.4.0 --namespace cert-manager --set installCRDs=true --create-namespace --wait
helm upgrade --install vault hashicorp/vault --set server.dev.enabled=true --set global.tlsDisable=true --wait

kubectl exec vault-0 -i -- vault secrets enable -path linkerd pki
kubectl exec vault-0 -i -- vault secrets tune -max-lease-ttl=175200h linkerd
kubectl exec vault-0 -i -- vault write linkerd/root/generate/internal common_name=identity.linkerd.cluster.local key_type=ec key_bits=256 ttl=175200h
kubectl exec vault-0 -i -- vault write linkerd/config/urls \
    issuing_certificates="http://vault.default:8200/v1/linkerd/ca" \
    url_distribution_points="http://vault.default:8200/v1/linkerd/crl"
kubectl exec vault-0 -i -- vault write linkerd/roles/linkerd-trust-anchor allowed_domains=cluster.local allow_subdomains=true max_ttl=48h key_type=ec key_bits=256

kubectl exec vault-0 -i -- vault auth enable kubernetes
kubectl exec vault-0 -i -- sh -c 'vault write auth/kubernetes/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'

kubectl exec vault-0 -i -- vault policy write linkerd-trust-anchor - <<EOF
path "linkerd*"                        { capabilities = ["read", "list"] }
path "linkerd/roles/linkerd-trust-anchor"   { capabilities = ["create", "update"] }
path "linkerd/sign/linkerd-trust-anchor"    { capabilities = ["create", "update"] }
path "linkerd/issue/linkerd-trust-anchor"   { capabilities = ["create"] }
EOF
kubectl exec vault-0 -i -- vault write auth/kubernetes/role/linkerd-trust-anchor bound_service_account_names=linkerd-trust-anchor bound_service_account_namespaces=linkerd policies=linkerd-trust-anchor ttl=20m

Then, the manifests:

kubectl apply -f- <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: linkerd
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: linkerd-trust-anchor
  namespace: linkerd
---
apiVersion: v1
kind: Secret
metadata:
  name: linkerd-trust-anchor-secret
  namespace: linkerd
  annotations:
    kubernetes.io/service-account.name: linkerd-trust-anchor
type: kubernetes.io/service-account-token
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linkerd-trust-anchor
  namespace: linkerd
spec:
  vault:
    path: linkerd/sign/linkerd-trust-anchor
    server: http://vault.default.svc.cluster.local:8200
    auth:
      kubernetes:
        role: linkerd-trust-anchor
        mountPath: /v1/auth/kubernetes
        secretRef:
          name: linkerd-trust-anchor-secret
          key: token
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
spec:
  secretName: linkerd-identity-issuer
  duration: 48h
  renewBefore: 25h
  issuerRef:
    name: linkerd-trust-anchor
    kind: Issuer
  commonName: identity.linkerd.cluster.local
  dnsNames:
  - identity.linkerd.cluster.local
  isCA: true
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
  - cert sign
  - crl sign
  - server auth
  - client auth
EOF

The error:

$ kubectl cert-manager status certificate linkerd-identity-issuer -n linkerd
  Events:
    Type     Reason           Age                    From          Message
    ----     ------           ----                   ----          -------
    Normal   cert-manager.io  4m36s                  cert-manager  Certificate request has been approved by cert-manager.io
    Warning  DecodeError      4m36s (x2 over 4m36s)  cert-manager  Failed to decode returned certificate: error decoding certificate PEM block

As @SgtCoDFish suggested during this morning’s standup, we need to “know” what Vault is returning in order to investigate. Here is the call made by cert-manager to vault (using mitmproxy):

POST /v1/linkerd/sign/linkerd-trust-anchor HTTP/1.1
Host: vault.default.svc.cluster.local:8200
User-Agent: Go-http-client/1.1
Content-Length: 709
X-Vault-Token: s.GVqpWaTNIzijhEkfPYD6ifXz
Accept-Encoding: gzip

{"alt_names":"identity.linkerd.cluster.local","common_name":"identity.linkerd.cluster.local","csr":"-----BEGIN CERTIFICATE REQUEST-----\nMIIBSzCB8wIBADApMScwJQYDVQQDEx5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIu\nbG9jYWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATljAk9qrLRtw6LTMEQ+yEF\nKOLAnlp6vRnH2j223gXvYylDPcneB9RukAwjA6alm/nptmzuNp22EdZTPk22N/Ag\noGgwZgYJKoZIhvcNAQkOMVkwVzApBgNVHREEIjAggh5pZGVudGl0eS5saW5rZXJk\nLmNsdXN0ZXIubG9jYWwwCwYDVR0PBAQDAgEGMB0GA1UdJQQWMBQGCCsGAQUFBwMB\nBggrBgEFBQcDAjAKBggqhkjOPQQDAgNHADBEAiA6bbbJF/9TA0XkD2MndzpSW7d2\nlcSqStgVRGWI8+XpHwIget4hH3oVf8/cXG+/8xELPVLVKZJI4cbO7cwJd0rD9KY=\n-----END CERTIFICATE REQUEST-----\n","exclude_cn_from_sans":"true","ip_sans":"","ttl":"48h0m0s","uri_sans":""}

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Wed, 21 Jul 2021 12:44:35 GMT
Content-Length: 2048

{"request_id":"8b3e379d-38e2-517a-6ad9-e71325fce447","lease_id":"","renewable":false,"lease_duration":0,"data":{"certificate":"-----BEGIN CERTIFICATE-----\nMIICNzCCAd2gAwIBAgIUAm9x7b8ABMk8VN1LpiTMd7Vc5icwCgYIKoZIzj0EAwIw\nKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIx\nMDcyMTEyNDQwNVoXDTIxMDcyMzEyNDQzNVowKTEnMCUGA1UEAxMeaWRlbnRpdHku\nbGlua2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\n5YwJPaqy0bcOi0zBEPshBSjiwJ5aer0Zx9o9tt4F72MpQz3J3gfUbpAMIwOmpZv5\n6bZs7jadthHWUz5NtjfwIKOB4jCB3zAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYw\nFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBTY1k8c7pnx0UcvoZMsOMCI\nwXwmkzAfBgNVHSMEGDAWgBQUyeJXut4A5BMMMu8bReAsTXi9qTBDBggrBgEFBQcB\nAQQ3MDUwMwYIKwYBBQUHMAKGJ2h0dHA6Ly92YXVsdC5kZWZhdWx0OjgyMDAvdjEv\nbGlua2VyZC9jYTApBgNVHREEIjAggh5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIu\nbG9jYWwwCgYIKoZIzj0EAwIDSAAwRQIgVC2y8eSPj2rNEvcbjX+QhoT0K7Q1Hyxl\n43/1bk8FMhQCIQCX1DMKiqqSxGDZPJwKp7gZRV5PUltauN+LesoB9UZdbw==\n-----END CERTIFICATE-----","expiration":1627044275,"issuing_ca":"-----BEGIN CERTIFICATE-----\nMIIB5DCCAYqgAwIBAgIUCCBwT8uFI2tc47ffANIJxopQ0VMwCgYIKoZIzj0EAwIw\nKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIx\nMDcyMTExMDcxN1oXDTQxMDcxNjExMDc0N1owKTEnMCUGA1UEAxMeaWRlbnRpdHku\nbGlua2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\nprcufmWRfo0CLkScF4+oqJ7zg7xyjduu8/IvXdutm4pDgQB8bts/keHOAunBzt2g\nc8mdat+x2s6751MI8iGkOKOBjzCBjDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/\nBAUwAwEB/zAdBgNVHQ4EFgQUFMniV7reAOQTDDLvG0XgLE14vakwHwYDVR0jBBgw\nFoAUFMniV7reAOQTDDLvG0XgLE14vakwKQYDVR0RBCIwIIIeaWRlbnRpdHkubGlu\na2VyZC5jbHVzdGVyLmxvY2FsMAoGCCqGSM49BAMCA0gAMEUCIAHgIy00Q5IWr9np\nzVikSAFkCftsoxD60R5UcfBMWCoLAiEA+lgqdQiDzEMqr6DCT4UDNnQdipzYeLz3\nObE1DQ3r30A=\n-----END CERTIFICATE-----","serial_number":"02:6f:71:ed:bf:00:04:c9:3c:54:dd:4b:a6:24:cc:77:b5:5c:e6:27"},"wrap_info":null,"warnings":["the common_name field was provided but the role is set with \"use_csr_common_name\" set to true","the alt_names field was provided but the role is set with \"use_csr_sans\" set to true"],"auth":null}

The returned certificate seems OK:

jq .data.certificate -r <<<'{"request_id":"8b3e379d-38e2-517a-6ad9-e71325fce447","lease_id":"","renewable":false,"lease_duration":0,"data":{"certificate":"-----BEGIN CERTIFICATE-----\nMIICNzCCAd2gAwIBAgIUAm9x7b8ABMk8VN1LpiTMd7Vc5icwCgYIKoZIzj0EAwIw\nKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIx\nMDcyMTEyNDQwNVoXDTIxMDcyMzEyNDQzNVowKTEnMCUGA1UEAxMeaWRlbnRpdHku\nbGlua2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\n5YwJPaqy0bcOi0zBEPshBSjiwJ5aer0Zx9o9tt4F72MpQz3J3gfUbpAMIwOmpZv5\n6bZs7jadthHWUz5NtjfwIKOB4jCB3zAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYw\nFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBTY1k8c7pnx0UcvoZMsOMCI\nwXwmkzAfBgNVHSMEGDAWgBQUyeJXut4A5BMMMu8bReAsTXi9qTBDBggrBgEFBQcB\nAQQ3MDUwMwYIKwYBBQUHMAKGJ2h0dHA6Ly92YXVsdC5kZWZhdWx0OjgyMDAvdjEv\nbGlua2VyZC9jYTApBgNVHREEIjAggh5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIu\nbG9jYWwwCgYIKoZIzj0EAwIDSAAwRQIgVC2y8eSPj2rNEvcbjX+QhoT0K7Q1Hyxl\n43/1bk8FMhQCIQCX1DMKiqqSxGDZPJwKp7gZRV5PUltauN+LesoB9UZdbw==\n-----END CERTIFICATE-----","expiration":1627044275,"issuing_ca":"-----BEGIN CERTIFICATE-----\nMIIB5DCCAYqgAwIBAgIUCCBwT8uFI2tc47ffANIJxopQ0VMwCgYIKoZIzj0EAwIw\nKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIx\nMDcyMTExMDcxN1oXDTQxMDcxNjExMDc0N1owKTEnMCUGA1UEAxMeaWRlbnRpdHku\nbGlua2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\nprcufmWRfo0CLkScF4+oqJ7zg7xyjduu8/IvXdutm4pDgQB8bts/keHOAunBzt2g\nc8mdat+x2s6751MI8iGkOKOBjzCBjDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/\nBAUwAwEB/zAdBgNVHQ4EFgQUFMniV7reAOQTDDLvG0XgLE14vakwHwYDVR0jBBgw\nFoAUFMniV7reAOQTDDLvG0XgLE14vakwKQYDVR0RBCIwIIIeaWRlbnRpdHkubGlu\na2VyZC5jbHVzdGVyLmxvY2FsMAoGCCqGSM49BAMCA0gAMEUCIAHgIy00Q5IWr9np\nzVikSAFkCftsoxD60R5UcfBMWCoLAiEA+lgqdQiDzEMqr6DCT4UDNnQdipzYeLz3\nObE1DQ3r30A=\n-----END CERTIFICATE-----","serial_number":"02:6f:71:ed:bf:00:04:c9:3c:54:dd:4b:a6:24:cc:77:b5:5c:e6:27"},"wrap_info":null,"warnings":["the common_name field was provided but the role is set with \"use_csr_common_name\" set to true","the alt_names field was provided but the role is set with \"use_csr_sans\" set to true"],"auth":null}' \
  | openssl x509 -noout -text
issuer=CN = identity.linkerd.cluster.local
subject=CN = identity.linkerd.cluster.local

The issuing_ca:

jq .data.issuing_ca -r <<<'{"request_id":"8b3e379d-38e2-517a-6ad9-e71325fce447","lease_id":"","renewable":false,"lease_duration":0,"data":{"certificate":"-----BEGIN CERTIFICATE-----\nMIICNzCCAd2gAwIBAgIUAm9x7b8ABMk8VN1LpiTMd7Vc5icwCgYIKoZIzj0EAwIw\nKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIx\nMDcyMTEyNDQwNVoXDTIxMDcyMzEyNDQzNVowKTEnMCUGA1UEAxMeaWRlbnRpdHku\nbGlua2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\n5YwJPaqy0bcOi0zBEPshBSjiwJ5aer0Zx9o9tt4F72MpQz3J3gfUbpAMIwOmpZv5\n6bZs7jadthHWUz5NtjfwIKOB4jCB3zAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYw\nFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBTY1k8c7pnx0UcvoZMsOMCI\nwXwmkzAfBgNVHSMEGDAWgBQUyeJXut4A5BMMMu8bReAsTXi9qTBDBggrBgEFBQcB\nAQQ3MDUwMwYIKwYBBQUHMAKGJ2h0dHA6Ly92YXVsdC5kZWZhdWx0OjgyMDAvdjEv\nbGlua2VyZC9jYTApBgNVHREEIjAggh5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIu\nbG9jYWwwCgYIKoZIzj0EAwIDSAAwRQIgVC2y8eSPj2rNEvcbjX+QhoT0K7Q1Hyxl\n43/1bk8FMhQCIQCX1DMKiqqSxGDZPJwKp7gZRV5PUltauN+LesoB9UZdbw==\n-----END CERTIFICATE-----","expiration":1627044275,"issuing_ca":"-----BEGIN CERTIFICATE-----\nMIIB5DCCAYqgAwIBAgIUCCBwT8uFI2tc47ffANIJxopQ0VMwCgYIKoZIzj0EAwIw\nKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIx\nMDcyMTExMDcxN1oXDTQxMDcxNjExMDc0N1owKTEnMCUGA1UEAxMeaWRlbnRpdHku\nbGlua2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\nprcufmWRfo0CLkScF4+oqJ7zg7xyjduu8/IvXdutm4pDgQB8bts/keHOAunBzt2g\nc8mdat+x2s6751MI8iGkOKOBjzCBjDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/\nBAUwAwEB/zAdBgNVHQ4EFgQUFMniV7reAOQTDDLvG0XgLE14vakwHwYDVR0jBBgw\nFoAUFMniV7reAOQTDDLvG0XgLE14vakwKQYDVR0RBCIwIIIeaWRlbnRpdHkubGlu\na2VyZC5jbHVzdGVyLmxvY2FsMAoGCCqGSM49BAMCA0gAMEUCIAHgIy00Q5IWr9np\nzVikSAFkCftsoxD60R5UcfBMWCoLAiEA+lgqdQiDzEMqr6DCT4UDNnQdipzYeLz3\nObE1DQ3r30A=\n-----END CERTIFICATE-----","serial_number":"02:6f:71:ed:bf:00:04:c9:3c:54:dd:4b:a6:24:cc:77:b5:5c:e6:27"},"wrap_info":null,"warnings":["the common_name field was provided but the role is set with \"use_csr_common_name\" set to true","the alt_names field was provided but the role is set with \"use_csr_sans\" set to true"],"auth":null}' \
  | openssl x509 -noout -subject -issuer
subject=CN = identity.linkerd.cluster.local
issuer=CN = identity.linkerd.cluster.local

We are also facing the same issue. Unfortunately changing the CN between CA and certificate is not an option for us. Is it possible to fix this issue in a coming version?

@rumanzo I am not sure though if that exact part of the code is causing the issue as it’s quite old

You are right, it’s old part of code. But in version <v1.4.0 this part not used with vault issuer

@rumanzo not sure what you mean. Isn’t issuer and CA the same thing in your case? Anyways, as far as I know CN must be different within the same subject (obviously to distinguish between different certs).

Looks like I’ve stumbled on the same issue. 1.3.1 worked, but I still don’t understand what’s the reason. @rumanzo you’ve posted a link where RawIssuer and RawSubject are checked - does it mean that it compares issuer subject and request subject? I’m not providing commonName at all and use only dnsNames.

Edit: did some tests and everything confirmed, if subject is the same for issuer and requested cert it doesn’t work and this is the only reason.

And finally, this is correct (different common_name)

vault write linkerd/root/generate/internal common_name=root.linkerd.cluster.local key_type=ec key_bits=256 ttl=175200h

It’is not a bug, but feature request to improve logging