flux2: Kustomization and HelmRelease objects are stuck if they fail to apply on first install

Describe the bug

Kustomization and HelmRelease objects can’t be updated/deleted if they fail to be applied into the cluster when they are configured for first time.

To Reproduce

Steps to reproduce the behaviour:

  1. Install a HelmRelease to create a pod with a wrong configuration to prevent kubernetes creates the pod (bad node selector, bigger resource requests than allow, not allowed PodSecurityPolicy). Install kustomization to apply a bad git repository, with duplicated resources or resources using a non installed CRD.
  2. Delete or update the objects to fix detected issues

Expected behavior

The object should be deleted/updated and cancel stuck reconciliation process. The same process is working on flux v1 with almost no delay.

Additional context

There are no logs until the HR is eventually deleted (30 minutes after I deleted in my latest try)

Logs from Helm-controller when the HR is deleted (I rewrote the HR and namespace names)

{"level":"error","ts":"2021-07-01T09:17:24.183Z","logger":"controller.helmrelease","msg":"unable to update status after reconciliation","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"my-release\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2021-07-01T09:17:24.183Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"my-release\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2021-07-01T09:17:24.503Z","logger":"controller.helmrelease","msg":"uninstalled Helm release for deleted resource","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace"}
{"level":"info","ts":"2021-07-01T09:17:24.535Z","logger":"controller.helmrelease","msg":"uninstalled Helm release for deleted resource","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace"}
{"level":"error","ts":"2021-07-01T09:17:24.540Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"my-release","namespace":"my-namespace","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"my-release\": StorageError: invalid object, Code: 4, Key: /registry/helm.toolkit.fluxcd.io/helmreleases/my-namespace/my-release, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 74bcb564-45b8-401a-b3a2-f3267b2b2ac1, UID in object meta: "}

$ flux check

► checking prerequisites
✔ kubectl 1.21.2 >=1.18.0-0
✔ Kubernetes 1.21.2 >=1.16.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.10.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.12.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.12.2
✔ all checks passed

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 26
  • Comments: 28 (15 by maintainers)

Most upvoted comments

yes, I know that it is not a Kustomization issue, because I know where the root cause is. Just the behaviour of the kustomization, it’s kinda stressing me out 😄, all I can do is just deleting it, or sometime I have to flux uninstall and bootstrap again 😢. By the way, could you please point out where should I put the spec.timeout value you was saying about? TYIA.

@stefanprodan I broke the kustomization applying a bad git repository.

In my case I fail to migrate the HelmRelease from fluxv1 CRD (I set a wrong chart definition) and the kustomization was failing to apply all resources from the repository.

Kustomization object was alsy trying to apply same git changeset all the time, even after I fixed it and GitRepository object was already synced with the new version.

One issue that comes up often in our development environments is HelmReleases ending up in a broken state because the helm-controller fails to reconcile it multiple times. There seems to be some threshold for how many times it can fail before the helm-controller simply starts ignoring it (it seems).

The only solution that I have found to remediate this problem is to delete the HelmRelease. I’ve never had an issue with deleting them, although sometimes it can take a while.

Ok this has nothing to do with kustomize-controller, please open an issue in helm-controller my guess is that your chart has some random generator that trips the controller to update it every time.

It works! Thank you so much! 🤟