cert-manager: Vault issuer with Kubernetes auth: claim "iss" is invalid due to the use of default service account tokens

🚧 Update 29 July 2022 🚧 If you are seeing the message “claim iss is invalid”, the workaround is to disable iss validation in Vault as explained in Problem 3: Incohenrent iss.

In Kubernetes v1.20+, kube-apiserver requires the flag --service-account-issuer to be an URI (see documentation).

When I deployed the Vault Issuer with Helm, I got the following error on vault login api:

claim "iss" is invalid

I found in (vault.go)[https://github.com/jetstack/cert-manager/blob/master/pkg/internal/vault/vault.go#L313-L315] that cert-manager sends the JWT token as is, but Vault expects a JWT with an issuer equal to the value of the flag --service-account-issuer.

For example, when kube-apiserver is running with the flag:

--service-account-issuer=https://kubernetes.default.svc.cluster.local

then the JWT token used by cert-manager in the Vault issuer should have a payload that looks like this:

{
    "iss": "https://kubernetes.default.svc.cluster.local"
}

Instead of this, the JWT token looks like this:

{
    "iss": "kubernetes/serviceaccount"
}

My current workaround is to disable JWT Issuer Validation on my PKI config in Vault.

Steps to reproduce the bug: Use latest version of kubernetes Define service-account-issuer in kube-apiserver config Setup vault issuer with this doc use latest vault and create pki with JWT Issuer Validation enabled try to create vault issuer

Environment details::

Kubernetes version: v1.21.0
cert-manager version: v1.4.0
Install method: e.g. helm

/kind bug

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 10
Comments: 32 (13 by maintainers)

Most upvoted comments

To recap the state of this issue, and possible workarounds: There is a clean and secure permanent bounded service account support that would solve this issue in Draft PR

Meanwwhile, we tested two possible workarounds:

Disable JWT issuer validation in path
Make two kubernetes auth engines, one for all cert-manager vault issuer, using non bounded service accounts, and another for the rest of pods which in EKS 1.21 that speak to Vault using bounded serviceaccounts with oidc iss.

Workaround 1 is a compromise in security but it’s very easy to do and understand, you be the judge of how it affects your threat model most likely not a good idea.

To test number 2 we followed the steps here: We start with on an EKS 1.21 with cert-manager and internal Vault (done following setup shown by @maelvls here).

The only difference is that we set the iss on the /v1/auth/kubernetes engine to kubernetes/serviceaccount 🔥 ( and not https://kubernetes.default.svc.cluster.local). In EKS 1.21 all static service accounts that are not bounded (meaning not tied to pod and permanent) will have that fixed iss.

We then created the issuer with the same exact config in that comment and it showed itself ready (avoiding any iss error). If there is any trouble make sure all the policies and engine -> sa auth role config are binding the vault-sa in the last step in comment paying attention to bound_service_account_name.

This validates that we can use existing cert-manager mechanism for Vault issuers with static sa on EKS 1.21. What about all other workloads that might need to talk to vault that will use bounded service accounts?

For those, we tested added a different kubenetes auth engine at a different path v1/auth/kubernetes-oidc-iss. we set the iss to the correct cluster oidc endpoint. we then tied a different service account vault-user which we tied to a pod, in EKS 1.21 they will automatically be using a bounded service account, we then checked auth to vault works:

// Add bounded sa engine:

kubectl exec vault-0 -i -- sh -c 'vault write auth/kubernetes-oidc-iss/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
   issuer=https://oidc.eks.eu-west-2.amazonaws.com/id/REDACTED' # 🔥

// authorise a vault-user for a bounded pod sa: kubectl exec vault-0 -i -- vault write auth/kubernetes-oidc-iss/role/vault-sa bound_service_account_names=vault-user 🔥 bound_service_account_namespaces=default policies=vault-sa ttl=20m

From vault-user pod, we check authentication works successfully using (bounded token):

 curl -X POST $VAULT_ADDR/v1/auth/kubernetes-oidc-iss/login -d @req.json
{
  "role": "dev-role",
  "jwt": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." < (loaded from /var/run/secrets/kubernetes.io/serviceaccount/token))
}
response:
{"request_id":"4912b13c-a157-20ac-568b-6200850acd9e","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"REDACTED","accessor":"REDACTED","policies":["default","vault-sa"],"token_policies":["default","vault-sa"],"metadata":{"role":"vault-sa","service_account_name":"vault-user","service_account_namespace":"default","service_account_secret_name":"","service_account_uid":"7d103c6f-86ee-43b5-9b9c-53d23970642d"},"lease_duration":1200,"renewable":true,"entity_id":"322e2fa7-18d5-c88f-69a6-157b1fa36152","token_type":"service","orphan":true}}

We double checked the iss of the JWT on the pod has the correct oidc iss, and it was a bounded token with service account name and an expiry.

Of course for workaround 2 having to manage two auth engines is not ideal, and more importantly it would be better to have cert-manager issuers have the more secure bounded sa support through https://github.com/jetstack/cert-manager/pull/4524, this is how to do it until the feature lands and you are able to upgrade.

SpectralHiss on Nov 4, 2021

/retitle Make it possible to give a projected service account token to the Vault Issuer instead of a service account Secret

maelvls on Sep 30, 2021

Hi!

I was able to reproduce the issue! Back in https://github.com/jetstack/cert-manager/issues/4144#issuecomment-884275324, I made a tiny mistake that was setting iss to kubernetes/serviceaccount instead of the issuer I was giving. I get the same error message.

For anyone looking at this thread, here is where we are at:

KEP-2799 plans on deprecating the Token Controller which creates a Secret token for each service account. The deprecation may happen as of Kubernetes 1.24.
The “new way” became stable in Kubernetes 1.20 and is described in bound-service-account-tokens.md. The principle is to use a “bound” (also referred as to “projected”) service account token; by “bound”, we mean that the token is time-bound and audience-bound, unlike Secret tokens that aren’t bound.
cert-manager does not know how to use bound tokens with the Vault Issuer; it is only able to use old Secret tokens with the secretRef field.

Idea 1: let cert-manager request the token using the TokenRequest API, similarly as to how you can ask a Pod to mount a projected serviceAccountToken; we could imagine a new field projectedServiceAccount:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-sa
spec:
  vault:
    path: pki/sign/vault-sa
    server: http://vault.default.svc.cluster.local:8200
    auth:
      kubernetes:
        role: vault-sa
        mountPath: /v1/auth/kubernetes
        projectedServiceAccount: # ✨
          name: vault-sa
          expirationSeconds: 7200
          audience: cert-manager

The cert-manager Pod will thus need to be able to request tokens, which we can give permission to with the following role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tokenrequest
rules:
- apiGroups: [""]
  resources:
  - "serviceaccounts/token"
  - "serviceaccounts"
  verbs:
  - "create"
  - "get"

Idea 2: a simpler idea could be instead to mount a projected volume with a projection of the vault-sa service account into the cert-manager Pod and configure the Issuer with a tokenFile field. The downside is that would have to edit cert-manager’s Deployment spec anytime you want to use a Vault Issuer. It could look like this:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-sa
spec:
  vault:
    path: pki/sign/vault-sa
    server: http://vault.default.svc.cluster.local:8200
    auth:
      kubernetes:
        role: vault-sa
        mountPath: /v1/auth/kubernetes
        tokenFile: /var/run/secrets/tokens/vault-sa/token # ✨
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cert-manager
spec:
  template:
    spec:
      containers:
      - args:
        - --v=2
        - --cluster-resource-namespace=$(POD_NAMESPACE)
        - --leader-election-namespace=kube-system
        image: quay.io/jetstack/cert-manager-controller:v1.4.0
        name: cert-manager

        volumeMounts:
        - mountPath: /var/run/secrets/tokens
          name: vault-sa
    - mountPath: 
      name: vault-sa
      volumes:
      - name: vault-sa
        projected:
          sources:
          - serviceAccountToken:
              path: vault-sa
              expirationSeconds: 7200
              audience: vault

maelvls on Oct 15, 2021

FWIW @hawksight, that’s exactly what HashiCorp seem to be suggesting to get Kuberetes auth working - https://learn.hashicorp.com/tutorials/vault/agent-kubernetes?in=vault%2Fauth-methods#kubernetes-1-24-only=

stevegore on Jul 14, 2022

It looks like Kubernetes 1.24 doesn’t create a secret storing the service account token by default. We need to document this breaking change ASAP. There has been a lot of noise on Twitter due to this change.

@maelvls - the change you are referring to above is the following?: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#urgent-upgrade-notes

Specifically:

The LegacyServiceAccountTokenNoAutoGeneration feature gate is beta, and enabled by default. When enabled, Secret API objects containing service account tokens are no longer auto-generated for every ServiceAccount. Use the TokenRequest API to acquire service account tokens, or if a non-expiring token is required, create a Secret API object for the token controller to populate with a service account token by following this guide. (https://github.com/kubernetes/kubernetes/pull/108309, @zshihang)

So in the cert-manager docs we would need to update here to accommodate?

Instead of:

Create service account
Get the name of the token secret auto generated

We would have:

Create a service accounts

Create a secret of type kubernetes.io/service-account-token eg.

apiVersion: v1
kind: Secret
metadata:
  name: vault-issuer-token
  annotations:
    kubernetes.io/service-account.name: "vault-issuer"
type: kubernetes.io/service-account-token
data: {}

Configure your issuer / cluster-issuer as normal with the secret name, vault-issuer-token in the above example.

With that, cert-manager will still support vault with service account auth? Just the process for getting that static token is a little different?

hawksight on Jul 12, 2022

tl;dr:

cert-manager -> Vault auth method	Supported by cert-manager in the built-in Vault Issuer?
Kubernetes auth	✅
JWT auth	❌
OIDC auth	❌

You are correct, with Vault 1.9.0, the OIDC auth method does not work with the “default” JWT token used by cert-manager using the OIDC Discovery URL. As per the ID Token Validation section of the OIDC spec, the “default” JWT token does not abide by the spec for the claims iss, sub, aud, exp, and iat.

I wondered if you could still use the JWT auth method using static keys or with the JWKS bundle URI (Kubernetes 1.20+). For example, imagining that your “default” JWT token looks like this:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "cert-manager",
  "kubernetes.io/serviceaccount/secret.name": "cert-manager-token-rbgp6",
  "kubernetes.io/serviceaccount/service-account.name": "cert-manager",
  "kubernetes.io/serviceaccount/service-account.uid": "72ae1e66-da9a-48b9-a512-3200cd876009",
  "sub": "system:serviceaccount:cert-manager:cert-manager"
}

It would not work either since the following JWT auth configuration (imagining your cluster is on EKS) would require an audience to exist on the “default” JWT token, and Vault requires bound_audiences to be set with the role type jwt:

vault write auth/jwt/config \
    jwks_url="https://oidc.eks.eu-west-1.amazonaws.com/id/***/.well-known/jwks.json" \
    bound_issuer="kubernetes/serviceaccount"
    default_role="demo"

vault write auth/jwt/role/demo -<<EOF
{
  "policies": "demo",
  "role_type": "jwt",
  "ttl": "1h",
  "user_claim": "sub",
  "bound_audiences": "❌"
  "bound_claims": {
    "kubernetes.io/serviceaccount/namespace": ["cert-manager"],
    "kubernetes.io/serviceaccount/service-account.name": ["cert-manager"],
  }
}
EOF

Until we can get bound tokens to work with the Vault issuer in cert-manager, the JWT and OIDC auth methods will not work. The alternative is to use the Kubernetes auth.

maelvls on Apr 11, 2022

Based on Vault docs from here

In response to the issuer changes, Kubernetes auth has been updated in Vault 1.9.0 to not validate the issuer by default.

as a result both disable_iss_validation and issuer parameters will be deprecated, ref.

avoidik on Jan 12, 2022

@maelvls In our case, we’re using an OIDC provider to connect IAM users with serviceAccounts, so we can give specific permissions to every pod if necessary, described here.
I assume because we’re using this, EKS automatically configure the necessary parameters for service account volume projection, described here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection. It’s a managed solution, so we can’t change the kube-api server parameters, which means, the token volume projection and service account issuer discovery are enabled by default.

To use iss: kubernetes/serviceaccount, we should set automountServiceAccountToken to false and manually mount the serviceAccount token as a secret volume to avoid projection. It would work, but require more manual maintenance, changing the mount if the serviceAccount secret name changes or adding manually a secret with a fixed name to an already existing serviceAccount. We don’t want to do that, because multiple pods calling Vault with serviceAccount token auth and this would become a maintenance nightmare quickly. Or we should use 2 separate Vault as @tafkam said, which is cumbersome, I agree.

ncsibra on Oct 11, 2021

@maelvls other applications (e.g. kubernetes-external-secrets, any microservices you are running) are using the projected tokens by default to authenticate in Vault since k8s 1.21. Configuring iss: kubernetes/serviceaccount would require two different Vault Kubernetes authentication backends for a single cluster. One for legacy serviceaccount tokens, and one for projected tokens since iss: kubernetes/serviceaccount will break anything trying to connect to Vault with a projected token. This is rather cumbersome when managing alot of k8s clusters.

That said, cert-manager seems to be in the minority of apps still using legacy tokens, in regard of the Vault issuer use-case.

tafkam on Oct 8, 2021

Hi,

I have the same issue, using AWS EKS.

EKS version: 1.21 Vault version: 1.5.4 Cert manager version: 1.5.3

First, here’s the relevant configs/tokens:

Vault:

/ $ vault read auth/kubernetes/config
Key                       Value
---                       -----
disable_iss_validation    false
disable_local_ca_jwt      false
issuer                    https://oidc.eks.eu-west-1.amazonaws.com/id/***
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
***
-----END CERTIFICATE-----
kubernetes_host           https://kubernetes.default.svc.cluster.local
pem_keys                  []

Vault pod mounted (projected) token:

{
  "aud": [
    "https://kubernetes.default.svc"
  ],
  "exp": 1664348742,
  "iat": 1632812742,
  "iss": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",
  "kubernetes.io": {
    "namespace": "sbx",
    "pod": {
      "name": "vault-0",
      "uid": "0417d0fc-9866-4f0c-b019-792ba9b968a3"
    },
    "serviceaccount": {
      "name": "vault",
      "uid": "5aec8a35-ba81-42d2-80ba-e7808ef4f387"
    },
    "warnafter": 1632816349
  },
  "nbf": 1632812742,
  "sub": "system:serviceaccount:sbx:vault"
}

Cert manager pod mounted (projected) token:

{
  "aud": [
    "https://kubernetes.default.svc"
  ],
  "exp": 1664275577,
  "iat": 1632739577,
  "iss": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",
  "kubernetes.io": {
    "namespace": "infra",
    "pod": {
      "name": "cert-manager-55bddfd78b-g2ccs",
      "uid": "4e334b33-6cf5-4fbb-a032-aacd80fae771"
    },
    "serviceaccount": {
      "name": "cert-manager",
      "uid": "707c95c2-8f4d-43be-8045-2f5f2aa40123"
    },
    "warnafter": 1632743184
  },
  "nbf": 1632739577,
  "sub": "system:serviceaccount:infra:cert-manager"
}

An example Issuer:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-dliver-fluentd-aggregator
  namespace: infra
spec:
  vault:
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: dliver-fluentd-aggregator-issuer
        secretRef:
          key: token
          name: dliver-fluentd-aggregator-issuer-token-rbgp6
    path: service_pki__dliver-fluentd-aggregator/sign/service
    server: http://vault.sbx.svc.cluster.local:8200

The ServiceAccount token passed to the issuer (dliver-fluentd-aggregator-issuer-token-rbgp6):

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "infra",
  "kubernetes.io/serviceaccount/secret.name": "dliver-fluentd-aggregator-issuer-token-rbgp6",
  "kubernetes.io/serviceaccount/service-account.name": "dliver-fluentd-aggregator-issuer",
  "kubernetes.io/serviceaccount/service-account.uid": "72ae1e66-da9a-48b9-a512-3200cd876009",
  "sub": "system:serviceaccount:infra:dliver-fluentd-aggregator-issuer"
}

The Issuer status:

status:
  conditions:
  - lastTransitionTime: "2021-09-23T10:18:46Z"
    message: |-
      Failed to initialize Vault client: error reading Kubernetes service account token from dliver-fluentd-aggregator-issuer-token-rbgp6: error calling Vault server: Error making API request.

      URL: POST http://vault.sbx.svc.cluster.local:8200/v1/auth/kubernetes/login
      Code: 500. Errors:

      * claim "iss" is invalid
    observedGeneration: 3
    reason: VaultError
    status: "False"
    type: Ready

So on the pods level, everything is correct, they’re using the projected token, which has the correct “iss”, which is the oidc URL. We have some other services that call Vault through its REST API and they are working correctly too, because they are using the projected token.

The issue is, that the cert-manager Issuer CRD is not able to use projected tokens, just a simple secret token, based on the API reference: https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.VaultKubernetesAuth. As @maelvls said, “By default, the kube-apiserver flag --service-account-issuer is not used to fill in the iss of service account tokens. The default “iss”: “kubernetes/serviceaccount” is always used when mounting the service account.”, so it will never match with the Vault issuer. I didn’t find any documentation for Kubernetes to change the “iss” field for the ServiceAccount token. Based on this, I think it should be handled by cert-manager somehow because currently it just sends the token as is, the related code linked in the first comment by @rumanzo.

@maelvls Can you reproduce the info based on this information? Or do you have any idea how to give a correct, projected token to the Issuer CRD?

ncsibra on Sep 28, 2021