secrets-store-csi-driver: Don't rely on pod creation to mount to Kubernetes secrets
Describe the solution you’d like
Currently, when you choose to sync to Kubernetes secrets, pulling secrets/certs/keys from Key vault always happen on pod creation and you also need to mount SecretProviderClass to specific pod to make it work.
If we have issues with Key Vault access (managed identity is not working, things are deleted accidently in Key Vault or Key Vault API is unavailable), things will start to fail when pod gets restart/recreated. And it’s even worse for cronjobs, it will try to retrieve secrets/certs/keys from Key Vault every time a new pod is created under this cronjob, but as Key Vault is down, you can no longer create ANY new pod in production.
I hope this library can simply sync secrets/certs/keys to Kubernetes secrets at deploy time and don’t need to rely on any pod to mount them, thus I don’t have to worry about Key Vault availability after deployment.
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
- Secrets Store CSI Driver version: (use the image tag):
- Kubernetes version: (use
kubectl version
):
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 86
- Comments: 34 (7 by maintainers)
Any update regarding this issue?
I agree with jemag on this – not being able to sync any of the secrets from a vault into a secret means effectively having to manage secrets in two ways for most systems, because pretty much every Helm chart and 3rd party component manages its pods via an Operator or other mechanism, and you can’t control how they access secrets. They just expect them to be present.
That leaves administrators having to choose between not using a vault implementation, managing secrets two ways, or using a hacky process of priming the system periodically with a pod to get things synced as a secret. We settled on the latter for now, but its definitely tricky to manage – the CSI configuration and that pod configuration have to be kept in sync, and if someone misses something, its not always obvious.
Given that the CSI mechanism wakes up when a SecretProviderClass is created or updated, and it knows what secrets are being requested to be synced, it seems like its the best place to create a transient pod to trigger the synchronization.
I think this is a reasonable suggestion given the fact that the main use case for the secrets-store-csi-driver is the integration with some kind of vault.
There are a lot of resources that rely on native kubernetes secrets. Let’s say that I have 100+ resources depending on a native kubernetes secret. Right now, that would mean that I need to create a SecretProviderClass, then add relatively useless volumeMounts and volumes for those 100+ resources (useless in the sense that the resources will not use the mount itself but the reference to the kubernetes secret). If I do not add that huge amount of boiler plate code for some resources, there could be a risk where no previous resources tried to mount it and the secret itself is unavailable.
If that feature were to be implemented, I would instead only need to create the SecretProviderClass, and reference the kubernetes secret in the resources that need it. This would mean a lot less overhead and cleaner yaml.
I think this could be a big improvement for all the vault integrations that depend on this project, unless there is already some way to get around this with the current setup that I missed.
I think for some use cases described in the comments External Secrets Operator (https://external-secrets.io) could be a reasonable alternative. AFAIK it can just sync external secret with k8s native secret.
/lifecycle frozen
Marking this as frozen so it doesn’t go to stale. We’re exploring the option of decoupling the sync as k8s secret feature from the CSI driver mount.
Although
External Secrets Operator
can solve this particular problem, would be nice if we could avoid having two controllers solving similar problems.Another use case is image pull secrets; there is a chicken and egg problem if creating the secret to pull the pod’s image requires the pod to be running.
@aramase may I know if there is any progress regarding this? Thanks🙇
==================
Found some update from the meeting notes. If there is ETA about the feature, it would be great
Are there any potential issues with using a gcr.io/google_containers/pause pod to define the mount then have other pods reference the secret?
Another quick feedback: I also find very cumbersome the necessity to start a pod in order for a secret to be created. I would say even counter intuitive when looking at this project for the first time, although I understand the initial scope of the driver was different.
As a user, I was hoping for a solution that easily bridges GCP Secret Manager and Kubernetes secrets, without affecting my existing workloads definitions.
+1 on this, will provide a detailed use case shortly.