argo-cd: Kustomize build ComparisonError (unable to make loader)
Describe the bug App templated with kustomize ends in error state (ComparisonError)
We encountered problem with kustomize (v4.5.7 and v5.1.1) and argocd v2.8.0.
First, restarting argocd-repo-server
solved the problem. However, the error reappeared and the restart did not fix it.
I did not find anything useful in the debug
logs. With argocd v2.7.x everything worked fine.
Did not help:
- disabling redis compression
- update kustomize to v5
- restarting controllers
To Reproduce
The problem appears with kustomize templated apps. I have not been able to find the key to reproduce accurately, but with a hard refresh
action, the error should appear. But it depends on how long the repo-server is running.
In my case, app ends in the error state after hard-refresh triggered between 15-60minutes after repo-server restart.
Once kustomize build get permission denied
, it never recovered without restarting repo-server.
Expected behavior
Kustomize apps do not fall in the error state after (hard)refresh.
Version
v2.8.0+804d4b8
Logs
Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `kustomize build <path to cached source>/path/to/my/app` failed exit status 1: 2023/08/15 12:31:20 unable to make loader at '.'; not a valid directory: abs path error on '.' : stat .: permission denied
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 22 (4 by maintainers)
Commits related to this issue
- fix(kustomize): set build dir (#15057) Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com> — committed to crenshaw-dev/argo-cd by crenshaw-dev 8 months ago
@crenshaw-dev After a few days, I don’t see any problem with kustomize. I think you can safely integrate this fix into v2.8 and v2.9. Thanks a lot!
I built ArgoCD v2.8.2 with the fix. So far no problems with kustomize. But after restarting the repo-server it always worked for some random time (5-120min). So I wouldn’t give a final verdict until tomorrow.
Argocd version: v2.8.4+c279299 Applications randomly enter this sync state. Sometimes it resolves on hard refresh. But more often they don’t. We have restart the repo server pods to resolve this.