bank-vaults: [vault-secrets-webhook] Container fails to start after CA certificate in Kubernetes secret renewed

Describe the bug: So my environment uses the vault-secrets-webhook and a certificate (self signed by cert-manager) mounted from a secret. Helm chart setup:

certificate:
  useCertManager: true
  generate: false

Annotations on my container:

  vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
  vault.security.banzaicloud.io/vault-tls-secret: "vault-server-tls"

Today my certificate got renewed by cert-manager and now if one of my pods crashes and is restarted I get an error like this:

OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/f60f1274-34e3-4c86-8c5d-317e1818b17b/volume-subpaths/vault-tls/cms-api/2\\\" to rootfs \\\"/var/lib/docker/overlay2/d76402adf3a76689023c7de2fdb06fd75b26f9017b002575043b7fb82202d703/merged\\\" at \\\"/var/lib/docker/overlay2/d76402adf3a76689023c7de2fdb06fd75b26f9017b002575043b7fb82202d703/merged/vault/tls/ca.crt\\\" caused \\\"no such file or directory\\\"\"": unknown

Deleting the pod and letting the Deployment recreate it fixes the issue. The secret shows as 60 days old and has all the expected files.

Expected behaviour: Certificate gets renewed, secret is updated with the new certificate files, containers continue to work as expected.

Steps to reproduce the bug: Run a container using a vault-secrets-webhook injected secret (environment variable) and have the vault CA certificate stored in a secret. Renew this

Additional context: Add any other context about the problem here.

Environment details:

  • Kubernetes version: v1.15.10
  • Cloud-provider/provisioner: EKS:
  • bank-vaults version (e.g. 0.4.17): banzaicloud/vault-secrets-webhook:0.9.0
  • Install method (e.g. helm or static manifests): Helm
  • Logs from the misbehaving component (and any other relevant logs): See above
  • Resource definition (possibly in YAML format) that caused the issue, without sensitive data: Hopefully shared enough above.

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (19 by maintainers)

Most upvoted comments

Confirmed fixed in my environment 👍 Thanks!

This landed in master, please pull the latest master image from the webhook. When you test it please make sure that the application pod mounts the vault CA Secret from a projected volume instead directly from the Secret.

If you face any issues again feel free to reopen this issue. Thanks!

Great. Will do!