vault-k8s: Unable to inject Vault secrets into Kubernetes pods using vault-k8s
Describe the bug Unable to inject Vault secrets into Kubernetes pods using vault-k8s. Following the steps at: https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar
To Reproduce Steps to reproduce the behavior:
- Install Vault using Vault Helm Chart.
- Configure Vault policy.
kubectl exec -ti vault-anubbhav-0 /bin/sh
cat <<EOF > /home/vault/app-policy.hcl
path "secret*" {
capabilities = ["read"]
}
EOF
vault policy write app /home/vault/app-policy.hcl
- Configure the Vault Kubernetes Auth method and attach our newly recreated policy to our applications service account.
vault auth enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
vault write auth/kubernetes/role/myapp \
bound_service_account_names=app-anubbhav \
bound_service_account_namespaces=default \
policies=app \
ttl=1h
- Create an example username and password in Vault using the KV Secrets Engine.
vault kv put secret/helloworld username=foobaruser password=foobarbazpass
- Use below .yaml configuration file for running a demo application and creating service account. Application deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
labels:
app: vault-agent-demo
spec:
selector:
matchLabels:
app: vault-agent-demo
replicas: 1
template:
metadata:
annotations:
labels:
app: vault-agent-demo
spec:
serviceAccountName: app-anubbhav
containers:
- name: app
image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-anubbhav
labels:
app: vault-agent-demo
- Apply annotations patch to our running example application’s pod configuration.
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
vault.hashicorp.com/role: "myapp"
kubectl patch deployment app --patch "$(cat patch-basic-annotations.yaml)"
Other useful info to include: kubectl get pods
app-5bdc46bb58-rv8hh 1/1 Running 0 14m
vault-anubbhav-0 1/1 Running 0 3h1m
vault-anubbhav-agent-injector-76748c7c47-l54j2 1/1 Running 0 3h1m
Expected behavior
kubectl exec -ti app-5bdc46bb58-rv8hh -c app -- cat /vault/secrets/helloworld should return the vault secret value. However the following error is received instead.
cat: can't open '/vault/secrets/helloworld': No such file or directory
command terminated with exit code 1
Environment
- Kubernetes version:
- Distribution or cloud vendor (GKE): 1.17.14-gke.1600
- vault-k8s version: Vault Helm Chart 0.8.0
Additional context The code snippet for the various steps are the same as used in the demo: https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 30 (15 by maintainers)
@anubbhavm I think since extraEnvironmentVars is a map, you can use ‘.’ instead of ‘=’ to --set the variable:
Glad it worked! And thanks for pointing out the blog tutorial. I think we can just update the chart so that it works in dev mode like it used to, but otherwise we can certainly update the docs.
@jasonodonnell @tvoran The fix worked! The app pod is now able to retrieve the vault secrets. Thanks a ton for your help and being patient!
Since I was following the tutorial at: https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar and it doesn’t mention anything about resetting the
VAULT_DEV_LISTEN_ADDRESSenv variable, is it possible to update the tutorial so that other people don’t run into this problem going forward?@jasonodonnell I disabled istio and made the firewall rules less restrictive.
I’m able to see a tmpfs mount
/vault/secretsin theappcontainer, which is a part of theapp-c45774854-9fbz6pod. However this pod is stuck in theInit:0/1status because of which I’m unable to access the tmpfs mount. Please see below for command outputs.Thanks for the extra info, @anubbhavm. Everything looks setup properly now, so this is either (hopefully):
anubbhavnamespace.