bank-vaults: better coverage and more reliable secrets mutation needed as race condition exists and mutations fail...

Is your feature request related to a problem? Please describe. We use the vault-secrets-webhook (VSW) heavily in our environment. One problem we are seeing, however, is that if the VSW is not ready then when dependent resources startup and attempt to mutate secrets, if the VSW is not ready then those resources, which are typically either some kind of service or a configmap/secret for a service fail to properly function but the fact that the lack of secrets mutation is a quiet problem which when a service fails requires manual intervention to investigate the problem and secrets mutation failure is often overlooked.

Describe the solution you’d like

Could we get better functionality out of the sidecar and out of VSW which could assist in:

  1. mitigating the issue of secrets not mutating as a result of lack of readiness in the service or
  2. if inability to mutate, some proper output to that fact as a side-car
  3. as a service, vsw should at least expose metrics for secrets failures, attempt, successes, etc
  4. it would be nice that the side-car continues to attempt to mutate failed secrets – most important aspect in my estimation

Describe alternatives you’ve considered so far the only thing we’ve come up with is running a script that exec into pods where we use VSW for environmentally stored secrets to validate if secrets mutated by looking through /proc – this is slow and not very reliable.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 5
  • Comments: 15 (11 by maintainers)

Most upvoted comments

we’ve been discussing getting off banzaicloud products because of lack of support and maintenance. More to come on that.

dockersecrets end up non mutated on the system. kills, seems like like snowballs my vault setup, gitops is also racing with the object and relying on mutationg which then not works. also all automated images use the imagepull secret. so both automation and deployments. and keep harassing vault. deleting the secret and restarting the webhook worked initially. currently getting rid of vault secrets webhook dealing with docker secrets for all my clusters as all my systems break. and i can only put the webhook as the culprit at this stage as there are 15x more tokens than any other policy. spending already a couple of days on this. webhook should never create non mutated secrets. especially not update earlier mutated secrets with contents of vault refs in etcd. as i understand this ticket is also about.

I’m not sure how you’ve got VSW set up, but we’ve never seen the kinds of problems you’re describing. It sounds like maybe your setup is in need of some tweaking? Other than a few things we’ve identified with the ecosystem, it works pretty nicely, but the things we’ve identified are pretty big deals. The things we spoke of above are one of the main problems we’ve identified. Another one is the inability for VSW ecosystem to update/mutate secrets in running pods when secrets are changed in vault.

I got rid of creating docker secrets via VSW at the time being. We spotted a bottle neck in storage IOPS for 2/3 nodes. 2 of them use a CEPH volume with spinning disks which seemed much to slow for Raft to work properly. We meanwhile switched to SSDs and this made vault much more stable again.

But why secrets ended up non mutated on the system I think is a race condition between our GitOps tool Flux and the mutating webhook. And because vault was not functioning properly the VSW get time outs for the mutation and the non mutated Secret ends up on the cluster as defined by Flux.

I guess instead of parsing ConfigMap or Secret directly, I bank-vaults should introduce 2 CRDs for VaultSecret or VaultConfigMap and define those in GitOps and let the mutating webhook/operator create the actual parsed Secret or ConfigMap so there won’t be a race condition between gitops tooling and bank-vaults.

In the past few months I haven’t had the time for the project, but now it is getting focus again, I will try to to look into this in the coming days, I think webhook invocation auto restart tuning and pod disruption budget can help here.