bank-vaults: better coverage and more reliable secrets mutation needed as race condition exists and mutations fail...
Is your feature request related to a problem? Please describe. We use the vault-secrets-webhook (VSW) heavily in our environment. One problem we are seeing, however, is that if the VSW is not ready then when dependent resources startup and attempt to mutate secrets, if the VSW is not ready then those resources, which are typically either some kind of service or a configmap/secret for a service fail to properly function but the fact that the lack of secrets mutation is a quiet problem which when a service fails requires manual intervention to investigate the problem and secrets mutation failure is often overlooked.
Describe the solution you’d like
Could we get better functionality out of the sidecar and out of VSW which could assist in:
- mitigating the issue of secrets not mutating as a result of lack of readiness in the service or
- if inability to mutate, some proper output to that fact as a side-car
- as a service, vsw should at least expose metrics for secrets failures, attempt, successes, etc
- it would be nice that the side-car continues to attempt to mutate failed secrets – most important aspect in my estimation
Describe alternatives you’ve considered
so far the only thing we’ve come up with is running a script that exec into pods where we use VSW for environmentally stored secrets to validate if secrets mutated by looking through /proc – this is slow and not very reliable.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 5
- Comments: 15 (11 by maintainers)
we’ve been discussing getting off banzaicloud products because of lack of support and maintenance. More to come on that.
I got rid of creating docker secrets via VSW at the time being. We spotted a bottle neck in storage IOPS for 2/3 nodes. 2 of them use a CEPH volume with spinning disks which seemed much to slow for Raft to work properly. We meanwhile switched to SSDs and this made vault much more stable again.
But why secrets ended up non mutated on the system I think is a race condition between our GitOps tool Flux and the mutating webhook. And because vault was not functioning properly the VSW get time outs for the mutation and the non mutated Secret ends up on the cluster as defined by Flux.
I guess instead of parsing ConfigMap or Secret directly, I bank-vaults should introduce 2 CRDs for VaultSecret or VaultConfigMap and define those in GitOps and let the mutating webhook/operator create the actual parsed Secret or ConfigMap so there won’t be a race condition between gitops tooling and bank-vaults.
In the past few months I haven’t had the time for the project, but now it is getting focus again, I will try to to look into this in the coming days, I think webhook invocation auto restart tuning and pod disruption budget can help here.