source-controller: Source controller fails to correctly fetch the git repositories
Describe the bug
The bug manifests with errors on reconciliations such as apply failed: error: no objects passed to apply
❯ flux get kustomizations -A
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system kube-prometheus-stack-crds False apply failed: error: no objects passed to apply kube-prometheus-stack-15.1.3/8e4a0fed3c1ea36293f0e5af4001ccc8c4110293 False
or
kustomization path not found: stat /tmp/system-flux132511846/cluster/base: no such file or directory
It is due to the source controller not fetching the repository correctly (32 bytes tarball)
/data/gitrepository/flux-system/kube-prometheus-stack-crds $ ls -al
total 16
drwxr-sr-x 2 controll 1337 4096 Apr 22 22:47 .
drwxr-sr-x 6 controll 1337 4096 Apr 22 20:18 ..
-rw-r--r-- 1 controll 1337 32 Apr 22 22:33 531ce5c7d64da93901b643a2b737d36117a2776a.tar.gz
lrwxrwxrwx 1 controll 1337 106 Apr 22 22:33 latest.tar.gz -> /data/gitrepository/flux-system/kube-prometheus-stack-crds/531ce5c7d64da93901b643a2b737d36117a2776a.tar.gz
Despite reporting success
❯ flux get source git -A
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system kube-prometheus-stack-crds True Fetched revision: kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a False
To Reproduce
Have a GitRepository Reconcile it Check the source controller data
Expected behavior
Repository not empty
Additional context
This bug was introduces with v0.13.0, previous version worked fine.
$ flux --version
flux version 0.13.0
$ flux check
► checking prerequisites
✔ kubectl 1.21.0 >=1.18.0-0
✔ Kubernetes 1.20.6+k3s1 >=1.16.0-0
► checking controllers
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.11.1
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.10.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.13.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.12.0
✔ all checks passed
$ kubectl -n <namespace> get all
NAME READY STATUS RESTARTS AGE
pod/kustomize-controller-78bdd99b66-d2nqm 1/1 Running 0 11h
pod/helm-controller-69667f94bc-jk9bj 1/1 Running 0 11h
pod/notification-controller-5c4d48f476-7k9rh 1/1 Running 0 11h
pod/source-controller-6c458cdb8d-jlzdn 1/1 Running 0 11h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notification-controller ClusterIP 10.1.123.23 <none> 80/TCP 28h
service/source-controller ClusterIP 10.1.234.45 <none> 80/TCP 28h
service/webhook-receiver ClusterIP 10.1.45.56 <none> 80/TCP 28h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kustomize-controller 1/1 1 1 28h
deployment.apps/helm-controller 1/1 1 1 28h
deployment.apps/notification-controller 1/1 1 1 28h
deployment.apps/source-controller 1/1 1 1 28h
NAME DESIRED CURRENT READY AGE
replicaset.apps/source-controller-785dd8f8b9 0 0 0 28h
replicaset.apps/kustomize-controller-78bdd99b66 1 1 1 11h
replicaset.apps/kustomize-controller-97dbdcb99 0 0 0 28h
replicaset.apps/helm-controller-69667f94bc 1 1 1 11h
replicaset.apps/helm-controller-5bbd94c75 0 0 0 28h
replicaset.apps/notification-controller-5c4d48f476 1 1 1 11h
replicaset.apps/notification-controller-d66bdc787 0 0 0 28h
replicaset.apps/source-controller-6c458cdb8d 1 1 1 11h
$ kubectl -n flux-system logs deploy/source-controller
{"level":"info","ts":"2021-04-23T06:48:06.075Z","logger":"controller.gitrepository","msg":"Reconciliation finished in 690.693629ms, next run in 30m0s","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"kube-prometheus-stack-crds","namespace":"flux-system"}
kubectl -n flux-system logs deploy/kustomize-controller
{"level":"error","ts":"2021-04-23T07:48:01.553Z","logger":"controller.kustomization","msg":"Reconciliation failed after 422.112691ms, next try in 30m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"kube-prometheus-stack-crds","namespace":"flux-system","revision":"kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a","error":"apply failed: error: no objects passed to apply\n"}
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (8 by maintainers)
Any chance the https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/.sourceignore is still the same, and excluding your directory?
Pulling in
v0.12.1of the source-controller should solve this issue,fluxrelease is building.Have identified the core of the issue, will work on a patch that should be released later today.