bank-vaults: [vault-secrets-webhook] Container fails to start after CA certificate in Kubernetes secret renewed
Describe the bug: So my environment uses the vault-secrets-webhook and a certificate (self signed by cert-manager) mounted from a secret. Helm chart setup:
certificate:
useCertManager: true
generate: false
Annotations on my container:
vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: "vault-server-tls"
Today my certificate got renewed by cert-manager and now if one of my pods crashes and is restarted I get an error like this:
OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/f60f1274-34e3-4c86-8c5d-317e1818b17b/volume-subpaths/vault-tls/cms-api/2\\\" to rootfs \\\"/var/lib/docker/overlay2/d76402adf3a76689023c7de2fdb06fd75b26f9017b002575043b7fb82202d703/merged\\\" at \\\"/var/lib/docker/overlay2/d76402adf3a76689023c7de2fdb06fd75b26f9017b002575043b7fb82202d703/merged/vault/tls/ca.crt\\\" caused \\\"no such file or directory\\\"\"": unknown
Deleting the pod and letting the Deployment recreate it fixes the issue. The secret shows as 60 days old and has all the expected files.
Expected behaviour: Certificate gets renewed, secret is updated with the new certificate files, containers continue to work as expected.
Steps to reproduce the bug: Run a container using a vault-secrets-webhook injected secret (environment variable) and have the vault CA certificate stored in a secret. Renew this
Additional context: Add any other context about the problem here.
Environment details:
- Kubernetes version: v1.15.10
- Cloud-provider/provisioner: EKS:
- bank-vaults version (e.g. 0.4.17): banzaicloud/vault-secrets-webhook:0.9.0
- Install method (e.g. helm or static manifests): Helm
- Logs from the misbehaving component (and any other relevant logs): See above
- Resource definition (possibly in YAML format) that caused the issue, without sensitive data: Hopefully shared enough above.
/kind bug
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (19 by maintainers)
Confirmed fixed in my environment 👍 Thanks!
This landed in master, please pull the latest master image from the webhook. When you test it please make sure that the application pod mounts the vault CA Secret from a projected volume instead directly from the Secret.
If you face any issues again feel free to reopen this issue. Thanks!
Great. Will do!