kustomize-controller: kustomize-controller hangs frequently

We use flux in 5 clusters, and in just one of them (the newest, smallest one) the kustomize controller keeps hanging so bad I have to kill the pod. I don’t see anything interesting in the logs before this happens, and otherwise there’s no difference between this and other deploys. I upgraded to the latest version, but still seeing the same behaviour.

$ kubectl -n flux-system exec -it source-controller-7b66c9d497-r4d68 -- sh
~ $ du -sh /data/*
3.9M	/data/gitrepository

As it hangs, I see CPU max out at 1.5GB. NB in all of the clusters I’ve had to increase the ~memory~CPU requests and limits to around 1.2/1.5GB, but only in this cluster does the controller hang.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 22 (9 by maintainers)

Most upvoted comments

$ kubectl -n flux-system top pods
NAME                                           CPU(cores)   MEMORY(bytes)
helm-controller-69689858db-xfpml               2m           85Mi
image-automation-controller-5dc8dd5444-4jvjn   1m           19Mi
image-reflector-controller-b678b766b-5fkv8     4m           50Mi
kustomize-controller-7c5dcfc76d-2d9zm          1501m        164Mi
notification-controller-5f54dc6c8b-lv52m       1m           25Mi
source-controller-6979d6967d-xfkx9             1m           76Mi