cert-manager: Certificates issued by vault with isCa: true are missing CA:TRUE in certificate

Describe the bug: I’m trying to set up linkerd with cert-manager and vault. I’ve created a root CA and an intermediate CA for my cluster with Terraform and I’m using the intermediate certificate as a trust anchor for linkerd. Linkerd’s identity container does not startup with the following message:

time="2021-12-10T09:39:23Z" level=fatal msg="Failed to initialize identity service: failed to verify issuer certificate: it must be an intermediate-CA, but it is not"

After checking the certificates and trying around I found the following behavior:

For testing purposes, I have created two ClusterIssuers,

A self signed cluster issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cert-manager.io/v1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"linkerd-self-signed-issuer"},"spec":{"selfSigned":{}}}
  creationTimestamp: "2021-12-10T09:32:10Z"
  generation: 1
  name: linkerd-self-signed-issuer
  resourceVersion: "50506"
  uid: ab029ca9-d992-4b32-95db-3f910738d363
spec:
  selfSigned: {}
status:
  conditions:
  - lastTransitionTime: "2021-12-10T09:32:10Z"
    observedGeneration: 1
    reason: IsReady
    status: "True"
    type: Ready

and one cluster issuer for vault

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  creationTimestamp: "2021-12-08T14:01:00Z"
  generation: 1
  name: vault-cluster-issuer
  resourceVersion: "1731"
  uid: 6ef7b5f8-efbe-4b03-a912-7943316abf2c
spec:
  vault:
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes_cluster1_local
        role: cluster1-cert-manager
        secretRef:
          key: token
          name: cert-manager-token-wfg85
    path: pki_intermediate_cluster1_local/sign/cluster1-local.mydomain.lo
    server: http://192.168.59.110:31891
status:
  conditions:
  - lastTransitionTime: "2021-12-08T14:01:02Z"
    message: Vault verified
    observedGeneration: 1
    reason: VaultVerified
    status: "True"
    type: Ready

My certificate resource :

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: asd
  namespace: platform-cert-manager
spec:
  isCA: true
  commonName: asd.platform-linkerd.cluster1-local.mydomain.lo
  secretName: asd
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: linkerd-self-signed-issuer
    kind: ClusterIssuer
    group: cert-manager.io

When I use the linkerd-self-signed-issuer to create the certificate with isCa: true, the generated certificate looks like the following:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            8c:aa:11:4f:07:01:1d:dd:12:34:81:db:2c:52:1d:6b
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=asd.platform-linkerd.cluster1-local.mydomain.lo
        Validity
            Not Before: Dec 10 09:32:10 2021 GMT
            Not After : Mar 10 09:32:10 2022 GMT
        Subject: CN=asd.platform-linkerd.cluster1-local.mydomain.lo
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:fc:bb:2a:45:d6:48:92:bb:6e:c0:94:15:c2:8b:
                    79:d9:47:05:98:f3:53:c8:31:b7:ff:fc:97:2b:9b:
                    42:9a:78:de:3f:7f:f1:b4:3a:be:62:aa:6e:81:ee:
                    7a:0c:ad:d5:7a:d7:45:5b:aa:5f:ab:25:17:ef:d3:
                    c9:e9:ce:d4:82:d8:18:60:53:d6:e8:7d:08:3c:27:
                    8c:57:24:c2:79:45:54:3b:13:5d:05:c6:5f:54:56:
                    44:f7:5d:dc:73:c7:a9:9a:97:e5:56:93:54:76:73:
                    d6:56:3d:18:a0:83:63:f6:92:69:97:7b:75:f7:76:
                    53:c3:39:7e:74:31:9f:54:83:2d:86:87:04:d3:a9:
                    40:17:87:6a:da:38:a8:40:7b:c5:df:bd:53:d6:ae:
                    82:e9:1b:c9:bc:56:cb:6c:5f:82:0a:27:07:ff:d6:
                    7d:41:38:47:75:34:b9:e7:14:66:f4:fd:13:ec:cc:
                    f3:d8:69:9a:a0:f3:b4:62:02:43:e9:21:f3:d4:db:
                    0b:79:56:9a:6d:bc:fc:30:81:5b:25:56:59:63:32:
                    40:8a:0a:d6:c2:ac:e5:68:c5:23:fe:fb:52:7f:d4:
                    84:4e:c5:0f:ef:17:df:30:19:1b:14:06:62:01:d0:
                    70:6b:ef:0e:fd:0c:52:b6:79:8b:d0:48:ca:75:58:
                    4a:7d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                15:0C:E8:0C:D6:4B:A5:35:70:BC:46:C8:9F:5D:DA:31:25:C5:8D:36
    Signature Algorithm: sha256WithRSAEncryption
         64:5e:b4:b2:cd:c6:ab:d1:99:23:bf:f8:db:78:5f:b6:0b:11:
         fa:85:d3:61:28:bc:4d:f4:9a:dc:ce:33:d5:39:ea:76:f7:49:
         be:4e:71:22:12:7e:fa:c9:d3:ab:8e:e1:7d:8b:60:e0:9d:31:
         82:06:76:e8:18:89:cd:14:81:29:a4:d2:33:8c:5b:ca:c3:03:
         9f:90:21:8a:fa:d5:65:75:60:c6:2b:47:a2:62:4e:18:3a:1f:
         0f:b0:27:df:c5:e2:99:77:90:3d:c0:e9:f6:66:42:36:70:cb:
         a8:54:86:67:6e:e3:65:6a:da:a2:13:d0:56:45:35:4a:89:ec:
         c8:4d:4d:07:63:26:2c:55:11:b1:ec:8b:d3:12:68:b5:4c:76:
         12:2e:3a:ca:1d:09:89:25:c6:bf:ce:b4:18:24:e4:aa:4b:93:
         85:9f:94:73:fd:4e:19:37:a6:75:f5:f4:69:34:b0:56:e0:8a:
         de:4c:59:91:88:ca:5c:d3:d1:9b:d5:f1:c0:85:43:05:63:b7:
         8f:2f:b9:05:e7:06:4c:02:5d:ab:05:fd:6b:df:03:cf:96:9a:
         c7:fc:f2:5c:c2:33:87:e5:b4:ee:85:30:30:20:22:b3:bb:40:
         eb:93:c8:99:ef:1a:c7:c3:f5:7b:07:03:38:6d:93:58:2c:5f:
         f1:6d:6d:76

The x509v3 extensions include CA:TRUE

If I change the issuerRef.name to my vault issuer vault-cluster-issuer, the x509v3 extensions looks completely different, and CA:TRUE is missing.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            27:c3:70:81:90:d6:ef:07:49:cd:67:08:61:fb:eb:a3:b4:02:96:5a
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=cluster1-local.mydomain.lo, OU=cluster1-local.mydomain.lo, CN=cluster1-local.mydomain.lo
        Validity
            Not Before: Dec 10 09:01:01 2021 GMT
            Not After : Mar 10 09:01:31 2022 GMT
        Subject: CN=asd.platform-linkerd.cluster1-local.mydomain.lo
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:be:80:63:d9:ef:c9:88:00:bb:61:ea:e9:95:96:
                    5e:9e:ba:79:0f:4b:d0:a3:3e:7c:d7:b9:7d:da:8c:
                    9d:77:6c:8b:c4:8c:66:bd:c9:3a:b0:1e:7c:4a:66:
                    cc:c6:f5:c2:0a:7f:5c:f1:17:3d:8b:71:6b:2e:37:
                    40:87:5c:b4:f7:1f:07:19:44:ec:b1:d6:71:76:6e:
                    bf:89:41:98:cb:04:96:bc:49:fc:d0:fe:d4:95:a1:
                    05:1f:1a:8f:b5:80:47:ba:97:09:31:e2:20:23:22:
                    37:e4:ef:d7:33:f7:58:1a:2b:0c:3e:f0:31:71:3d:
                    db:66:f7:d9:01:2c:fe:6f:60:74:8d:39:78:23:04:
                    82:4b:ba:8f:a7:3e:08:c8:df:7d:20:8e:7f:84:f6:
                    93:98:29:2b:1f:ff:64:57:fd:37:21:23:53:ea:d8:
                    43:53:12:85:cd:b1:95:ba:a9:81:88:05:96:0d:3a:
                    85:f0:33:85:5a:95:9a:31:1d:65:e6:c1:e7:68:1a:
                    19:47:53:d2:20:cb:0c:3b:a4:20:95:a4:af:98:ef:
                    33:9e:4a:4a:72:be:27:cf:35:2c:26:52:4d:3e:72:
                    ee:54:ee:b2:fa:44:f6:68:bf:a1:61:4e:1f:6f:4d:
                    53:c6:be:0f:03:bd:f9:56:f4:14:a9:04:2a:3f:4c:
                    b7:73
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                16:B0:FA:3B:5B:03:AA:FF:F7:7C:73:30:3D:DF:26:9D:3A:F5:13:0C
            X509v3 Authority Key Identifier:
                keyid:22:5B:0B:83:32:C6:4F:44:00:2B:A9:01:00:D8:B2:55:03:AE:C8:F6

    Signature Algorithm: sha256WithRSAEncryption
         2c:e9:ab:3e:4d:dd:7a:ab:fd:31:52:09:3b:3e:d9:51:96:ba:
         18:8d:1c:db:1e:bf:57:f4:0a:67:9e:f2:17:d9:26:ec:49:93:
         cd:67:53:55:12:eb:8d:a5:ba:a3:01:53:ea:1b:ec:03:77:bc:
         4d:5b:bd:f3:bd:53:c1:d2:9d:d2:82:57:24:42:5e:11:fb:bb:
         12:8c:2c:fc:e4:a7:2d:0f:24:56:30:56:b9:3c:fc:49:51:25:
         0c:3e:2b:c4:0d:fa:11:36:90:6f:62:7f:83:13:3a:fd:e2:38:
         b8:f1:d5:9e:91:20:6d:71:01:ff:11:e9:c2:9c:9d:10:37:bd:
         68:15:93:f8:46:58:89:dc:de:eb:84:e9:5b:ff:bd:76:7b:93:
         d0:e6:d8:49:9d:d6:8e:9b:d1:02:d7:9d:89:1b:df:d7:fd:c1:
         9e:c1:db:5a:f9:ed:f2:63:32:aa:51:b4:2d:73:b6:a4:3f:9b:
         2a:29:b1:99:59:68:bb:1b:9a:1e:04:0a:01:d7:a3:74:6d:92:
         29:eb:84:72:0a:f5:16:73:e0:8c:f5:c6:5c:87:bf:a1:f4:7a:
         ff:46:70:ed:dc:72:00:b4:af:d6:85:1b:85:5d:c5:58:c1:e4:
         55:c4:27:c5:65:f6:42:2a:1c:e7:75:ca:d8:2a:35:ac:9c:25:
         8b:48:1e:68

Expected behaviour: I would expect, that the certificate extensions for the resulting certificate are the same for both ClusterIssuers and that CA:TRUE is set for the vault-generated certificate.

Steps to reproduce the bug:

Anything else we need to know?:

After looking into the code I found that there is a validation function which suggests, that vault is not capable to create the X509v3 Basic Constraints extension. From what I can see in Github, the method is only been referenced inside the tests. Maybe @munnerz can explain why vault does not support this?

https://github.com/jetstack/cert-manager/blob/538683bdf38fb9e273cc5e91a1141a712a93057a/internal/apis/certmanager/validation/certificate_for_issuer.go#L73

https://github.com/jetstack/cert-manager/blob/538683bdf38fb9e273cc5e91a1141a712a93057a/internal/apis/certmanager/validation/certificate_for_issuer.go#L27-L45

Environment details::

Kubernetes version: 1.22.3
Cloud-provider/provisioner: vault: 1.9.0
cert-manager version: 1.6.1
Install method: e.g. helm

/kind bug

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (3 by maintainers)

Most upvoted comments

Updating the path worked for me

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: istio-system
spec:
  vault:
    path: pki_int/root/sign-intermediate
    server: $VAULT_ADDR
    auth:
      tokenSecretRef:
          name: vault-token
          key: token

+10

nmnellis on Jul 11, 2022

Thank you sooo much!

I can confirm that @nmnellis worked.

Summary:

Update Vault Policy to give the role permission to that path

Terraform snippet path "<pki_engine_path>/root/sign-intermediate" { capabilities = ["create", "update"] }

Issuer spec path as described by @nmnellis, though I’m using a ClusterIssuer and the Kubernetes auth method.
isCA: true on the certificate spec

Duration is a little short, but I left that as the same default as my other certificates. No reason it couldn’t be a year or longer.

@SgtCoDFish there’s still the issue of no obvious error if you don’t use the right path, but it looks like the capability is there after all. I’ll be using this setup with some k8s operators (ECK and possibly Strimzi), and will report back here if there are issues that arise out of this.

Result:

Certificate: Data: Version: 3 (0x2) Serial Number: 61:13:f6:38:53:03:fd:35:b1:15:4f:91:e1:2d:b8:fa:b1:61:1f:97 Signature Algorithm: sha256WithRSAEncryption Issuer: O = OrgName, OU = SASS, CN = OrgName Intermediate CA Validity Not Before: Jul 12 13:07:37 2022 GMT Not After : Oct 10 13:08:07 2022 GMT Subject: CN = my-es-cluster Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:a1:89:90:ba:b0:4e:c6:a5:1a:59:b2:7a:27:ea: 5e:01:26:f1:ae:56:70:a8:3d:03:02:6b:a2:43:6c: (…) c4:53 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:2 X509v3 Subject Key Identifier: C8:42:4D:5C:F7:7B:26:65:7C:A4:9C:46:19:6A:13:C0:58:71:18:32 X509v3 Authority Key Identifier: keyid:F5:95:94:CF:D9:0D:E8:7C:A1:4E:1B:58:7E:12:0C:AB:39:8B:80:7A

benjdaun on Jul 12, 2022

Hi,

I ran into this exact issue trying to get Vault and cert-manager to issue the required CA certs for Strimzi. Everything behaves normally, except that the certificate returned by cert-manager doesn’t have CA set to true. I’ll discuss with my team, but we might be motivated enough to pick this up.

benjdaun on Jun 22, 2022