vault-helm: Vault agent can't authenticate using k8s 1.21

Describe the bug Since I updated kind to 0.11, which by default uses k8s 1.21, my sidecar vault-agents can’t authenticate with Vault showing the following error: [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s

Using kind 0.11 and k8s 1.20, vault-agent authenticates correctly. The only fix I could find using k8s 1.21 is configuring the auth method specifying the issuer as follows:

vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="$K8S_HOST" \
kubernetes_ca_cert="$SA_CA_CRT" \
issuer="https://kubernetes.default.svc.cluster.local"

Disabling the issuer validation with disable_iss_validation=true also works. I think the issue may be related to the following change introduced on k8s 1.21: The ServiceAccountIssuerDiscovery feature has graduated to GA, and is unconditionally enabled.

To Reproduce Steps to reproduce the behavior:

Install vault helm chart 0.13 in a kind 0.11 cluster using k8s 1.21
Configure an auth method as documented here: https://www.vaultproject.io/docs/auth/kubernetes. 2.1 Enable auth method: vault auth enable kubernetes 2.2 Configure the auth endpoint:

vault write auth/kubernetes/config \
    token_reviewer_jwt="<your reviewer service account JWT>" \
    kubernetes_host=https://192.168.99.100:<your TCP port or blank for 443> \
    kubernetes_ca_cert=@ca.crt

2.3 Create a named role:

vault write auth/kubernetes/role/demo \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces=default \
    policies=default \
    ttl=1h

2.4 Configure Kubernetes creeating a ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default

Deploy a pod with vault-agent as sidecar using the configured auth method.
Check logs of vault-agent. You should see a periodic log that looks like: [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s

Expected behavior The documented steps to configure the auth method (https://www.vaultproject.io/docs/auth/kubernetes) should be working despite I am using k8s 1.21.

Environment

Kubernetes version:
- Kind 0.11 with k8s 1.21
vault-helm version:
- Vault Helm Chart 0.13

Chart values:

USER-SUPPLIED VALUES:
server:
  extraLabels:
    vault-server: "true"
  ha:
    config: "ui = true\n\nlistener \"tcp\" {\n  tls_disable = 1\n  address = \"[::]:8200\"\n
      \ cluster_address = \"[::]:8201\"\n}\n\nstorage \"consul\" {\n  path = \"vault/\"
      \ \n  address = \"consul-consul-server:8500\"\n}\n\nservice_registration \"kubernetes\"
      {}\n"
    enabled: true
    replicas: 2

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 14
Comments: 15 (5 by maintainers)

Commits related to this issue

Add HashiCorp Vault vs Kubernetes 1.21 bug workaround and documentation of disable_iss_validation=true (#237) - move HC tests into their own larger class so that we can test the engines too - workar... — committed to nuodb/nuodb-helm-charts by deleted user 3 years ago

Most upvoted comments

Hi, We encountered the same error after upgrading from AWS EKS to 1.21.

We solved the error by setting issuer for Kubernetes authentication method:

kubectl proxy &

issuer="$(curl --silent http://127.0.0.1:8001/api/v1/namespaces/default/serviceaccounts/default/token \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{"apiVersion": "authentication.k8s.io/v1", "kind": "TokenRequest"}' \
  | jq -r '.status.token' \
  | cut -d. -f2 \
  | base64 -D | jq .iss)"

vault write auth/kubernetes/config token_reviewer_jwt=$tr_account_token kubernetes_host=$k8s_host kubernetes_ca_cert=$k8s_cacert issuer=$issuer

+31

HakimG on Jul 21, 2021

I’ve tried with the following on EKS 1.21 without success:

vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  issuer="https://kubernetes.default.svc.cluster.local" \

The only way I got it working is by adding disable_iss_validation=true. I’m not sure of what I need to put as issuer. Should https:// be included ? And should I use the public AWS API endpoint like REDACTED.gr3.eu-west-1.eks.amazonaws.com` ?

Edit: found thanks to previous post, issuer is actually the OIDC provider for EKS cluster: https://oidc.eks.eu-west-1.amazonaws.com/id/REDACTED

ArchiFleKs on Aug 4, 2021

this is sorted as it was due to istio sidecar blocking the connections for auth.

rusanki on Jul 18, 2022

Hi @prasanjitshome, these are probably good questions for our discuss forums: https://discuss.hashicorp.com/c/vault

As for the question a new vault-helm release, we don’t typically release a new version of vault-helm for every Vault release, since they aren’t that tightly coupled version-wise. We recommend setting a specific Vault version in the user-specified overrides in the chart values when running in production: https://www.vaultproject.io/docs/platform/k8s/helm/configuration#image-1

So for example you can use vault-helm v0.19.0 and set server.image.tag=1.9.4 to use the latest version of Vault.

My guess about not seeing disable_iss_validation is you might be on an older version of Vault where that option doesn’t exist. The vault write command just ignores options it doesn’t recognize. I think it was added in 1.5.0: https://github.com/hashicorp/vault/blob/main/CHANGELOG.md#150

auth/kubernetes: Allow disabling iss validation [GH-91]

tvoran on Mar 10, 2022

As of Vault 1.9, the issuer and disable_iss_validation parameters are considered deprecated, and the disable_iss_validation default is now true for new configs: https://www.vaultproject.io/api-docs/auth/kubernetes#disable_iss_validation. (More details around that decision can be found here.)

We’ve also detailed the options for configuring K8s auth with respect to the changes in K8s 1.21 here: https://www.vaultproject.io/docs/auth/kubernetes#kubernetes-1-21

And we also have some changes to K8s auth coming soon to make it easier to run Vault in K8s with short-lived tokens: https://github.com/hashicorp/vault-plugin-auth-kubernetes/pull/122

Closing this for now, thanks for all your input!

tvoran on Jan 7, 2022

@Marc-Pons Glad you got it working on AWS! And yes, disable_iss_validation=true is the default, but only for new k8s auth configs. So if a k8s auth config was created on Vault 1.8, and then Vault is upgraded to Vault 1.9, the old default of disable_iss_validation=false could still be set.

tvoran on Jan 11, 2022

Hi @Marc-Pons, this is related to https://github.com/hashicorp/vault/issues/11953 and may be helpful to fix your authentication problems. Hope that helps!

jasonodonnell on Jul 20, 2021