cert-manager: (Cluster)Issuer with vault auth and serviceAccountRef is not accepted by cluster due to audience

I have kubernetes 1.25, cert-manager 1.12.1, external Vault 1.12 and I am trying to use new feature from cert-manager 1.12: kubernetes auth in Vault without reviewer token. I followed documentation how to set this up (https://cert-manager.io/docs/configuration/vault/#secretless-authentication-with-a-service-account).

I have ClusterIssuer named vault-issuer and service account name created named cert-manager-vault. I created all RoleBindings just as in documentation. Also, I added ClusterRole auth-delegator to cert-manager-vault so it can work without reviewer token configured in Vault. (https://developer.hashicorp.com/vault/docs/auth/kubernetes#use-the-vault-client-s-jwt-as-the-reviewer-jwt)

I am having following errors:

ClusterIssuer: Error initializing issuer: while requesting a Vault token using the Kubernetes auth: error calling Vault server: Error making API request. Code: 403. Errors: * permission denied

So, I captured request to Vault, decoded token, looks fine, maybe except “aud” field, but still this is fine according to docs. Looks like Vault is trying to call back kubernetes API, lets see:

kube-api: [authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, token audiences ["vault://vault-issuer"] is invalid for the target audiences ["https://kubernetes.default.svc.cluster.my.domain"]]"

And here I am stuck. How to fix this? By the way, I got same setup working for external-secrets project (https://external-secrets.io/v0.8.3/provider/hashicorp-vault/#kubernetes-authentication) - but there in jwt “aud” field is set to “https://kubernetes.default.svc.cluster.my.domain/” so kube-api accepts it and validates.

I think both Vault validation of audience and API validation could be satisfied if there were two audiences in JWT token:

vault://namespace/issuer (this is for Vault to validate)
https://kubernetes.default.svc.cluster.my.domain/ (this is for API to validate).

JWT tokens have “aud” as array, so both would fit. kube-api docs state that only one of audiences must match cluster audience to be OK. Not sure how Vault validates this though, but I suspect that only one from list is sufficient as well to pass validation.

@maelvls would this work?

My vault auth config:

    disable_iss_validation:        True
    disable_local_ca_jwt:        False
    issuer:
    kubernetes_ca_cert:
        -----BEGIN CERTIFICATE-----
        (...)
        -----END CERTIFICATE-----
    kubernetes_host:        https://kubernetes.default.svc.cluster.my.domain
    pem_keys: []

Cluster issuer config:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
  namespace: cert-manager
spec:
  vault:
    server: http://active.vault.service.my.domain:8200
    caBundle: 
    path: pki_int/sign/platform.role
    auth:
      # https://cert-manager.io/docs/configuration/vault/#secretless-authentication-with-a-service-account
      kubernetes:
        role: "k8s-cert-manager-role"
        mountPath: "/v1/auth/k8s-sd"
        serviceAccountRef:
          name: "cert-manager-vault"

About this issue

Original URL
State: open
Created a year ago
Reactions: 6
Comments: 15 (4 by maintainers)

Most upvoted comments

It appears that this issue arises from the fact that I haven’t accounted for one of the ways to configure the field token_reviewer_jwt when configuring Vault’s Kubernetes Auth. This option is the one used when configuring Vault:

vault write auth/kubernetes/config \
    token_reviewer_jwt="<your reviewer service account JWT>" \
    kubernetes_host=https://192.168.99.100:<your TCP port or leave it blank for 443> \
    kubernetes_ca_cert=@ca.crt

There are three scenarios with regards to the use of token_reviewer_jwt:

✅ In-Cluster Vault with no token_reviewer_jwt: In this situation, Vault picks up the pod’s service account token, which comes with the API server’s audience. Vault can use that token to authenticate with the Kubernetes API server when making a TokenReview request.
✅ Out-of-Cluster Vault with token_reviewer_jwt: In this situation, the token is manually created using the old service account token mechanism. Unlike the newer “bound” service account tokens, old service account tokens never expire. Depending on how your Kubernetes API server is set up, the audience of that token should be something like https://kubernetes.default.svc.cluster.local. This way, Vault can authenticate with the API server when making a TokenReview request.
❌ Out-of-Cluster Vault with no token_reviewer_jwt: (“secretless”) In this case, the token meant to be reviewed in the TokenReview call is also used to authenticate the request. This is where things go wrong: since cert-manager creates a service account token with a calculated audience (e.g., vault://default/issuer-1), Vault fails to authenticate with the API server. I have reproduced the issue, instructions are available in https://hackmd.io/@maelvls/S1tYFmegp.

This is an oversight on my part; I didn’t consider scenario (3) when I implemented https://github.com/cert-manager/cert-manager/pull/5502. I’m not sure how I would have made it work, though, since allowing the user to choose a custom audience, such as https://kubernetes.default.svc.cluster.local, would expose cert-manager to risks A and B. After a discussion with @SpectralHiss, I realize that I may have been overly conservative in the risk assessment: the Issuer object might not be the attack vector I thought it would be.

For users of cert-manager 1.12 and 1.13, I would recommend defaulting to scenario (2) by generating an old “static” Kubernetes token and passing it to token_reviewer_jwt. It’s not ideal because it defeats the purpose of not having any statically generated tokens, which isn’t great from a security perspective, but it’s the best option for now.

Let’s explore how we can address this in a future version of cert-manager.

maelvls on Nov 20, 2023

It should be possible to create token with 2 audiences: one that you create now (vault://…) and additional one that will satisfy review api (same that is needed in point 2). Such token would be validated in 2 places then: in Vault (service account+namespace+audience1), kube-api (audience2 and role bindings from RBAC for service account). Seems pretty tight in my opinion. Your thoughts? Feasible?

Garagoth on Sep 25, 2023

Same issue here, I’m running a vault in a cluster and i try to configure an issuer in another cluster. It is working if I run a fork of cert-manager where I add https://kubernetes.default.svc in audience array.

duizabojul on Jun 22, 2023