flux2: Kustomization and HelmRelease objects are stuck if they fail to apply on first install
Describe the bug
Kustomization and HelmRelease objects can’t be updated/deleted if they fail to be applied into the cluster when they are configured for first time.
To Reproduce
Steps to reproduce the behaviour:
- Install a HelmRelease to create a pod with a wrong configuration to prevent kubernetes creates the pod (bad node selector, bigger resource requests than allow, not allowed PodSecurityPolicy). Install kustomization to apply a bad git repository, with duplicated resources or resources using a non installed CRD.
- Delete or update the objects to fix detected issues
Expected behavior
The object should be deleted/updated and cancel stuck reconciliation process. The same process is working on flux v1 with almost no delay.
Additional context
There are no logs until the HR is eventually deleted (30 minutes after I deleted in my latest try)
Logs from Helm-controller when the HR is deleted (I rewrote the HR and namespace names)
{"level":"error","ts":"2021-07-01T09:17:24.183Z","logger":"controller.helmrelease","msg":"unable to update status after reconciliation","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"my-release\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2021-07-01T09:17:24.183Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"my-release\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2021-07-01T09:17:24.503Z","logger":"controller.helmrelease","msg":"uninstalled Helm release for deleted resource","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace"}
{"level":"info","ts":"2021-07-01T09:17:24.535Z","logger":"controller.helmrelease","msg":"uninstalled Helm release for deleted resource","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace"}
{"level":"error","ts":"2021-07-01T09:17:24.540Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"my-release\": StorageError: invalid object, Code: 4, Key: /registry/helm.toolkit.fluxcd.io/helmreleases/my-namespace/my-release, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 74bcb564-45b8-401a-b3a2-f3267b2b2ac1, UID in object meta: "}
$ flux check
► checking prerequisites
✔ kubectl 1.21.2 >=1.18.0-0
✔ Kubernetes 1.21.2 >=1.16.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.10.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.12.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.12.2
✔ all checks passed
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 26
- Comments: 28 (15 by maintainers)
yes, I know that it is not a Kustomization issue, because I know where the root cause is. Just the behaviour of the kustomization, it’s kinda stressing me out 😄, all I can do is just deleting it, or sometime I have to
flux uninstalland bootstrap again 😢. By the way, could you please point out where should I put thespec.timeoutvalue you was saying about? TYIA.@stefanprodan I broke the kustomization applying a bad git repository.
In my case I fail to migrate the HelmRelease from fluxv1 CRD (I set a wrong chart definition) and the kustomization was failing to apply all resources from the repository.
Kustomization object was alsy trying to apply same git changeset all the time, even after I fixed it and GitRepository object was already synced with the new version.
One issue that comes up often in our development environments is HelmReleases ending up in a broken state because the
helm-controllerfails to reconcile it multiple times. There seems to be some threshold for how many times it can fail before thehelm-controllersimply starts ignoring it (it seems).The only solution that I have found to remediate this problem is to delete the HelmRelease. I’ve never had an issue with deleting them, although sometimes it can take a while.
Ok this has nothing to do with kustomize-controller, please open an issue in helm-controller my guess is that your chart has some random generator that trips the controller to update it every time.
The threshold is configurable, see:
It works! Thank you so much! 🤟