source-controller: Source controller fails to correctly fetch the git repositories

Describe the bug

The bug manifests with errors on reconciliations such as apply failed: error: no objects passed to apply

❯ flux get kustomizations -A
NAMESPACE      NAME                                        READY    MESSAGE                                                            REVISION                                                                 SUSPENDED
flux-system    kube-prometheus-stack-crds                  False    apply failed: error: no objects passed to apply                    kube-prometheus-stack-15.1.3/8e4a0fed3c1ea36293f0e5af4001ccc8c4110293    False

or

kustomization path not found: stat /tmp/system-flux132511846/cluster/base: no such file or directory

It is due to the source controller not fetching the repository correctly (32 bytes tarball)

/data/gitrepository/flux-system/kube-prometheus-stack-crds $ ls -al
total 16
drwxr-sr-x    2 controll 1337          4096 Apr 22 22:47 .
drwxr-sr-x    6 controll 1337          4096 Apr 22 20:18 ..
-rw-r--r--    1 controll 1337            32 Apr 22 22:33 531ce5c7d64da93901b643a2b737d36117a2776a.tar.gz
lrwxrwxrwx    1 controll 1337           106 Apr 22 22:33 latest.tar.gz -> /data/gitrepository/flux-system/kube-prometheus-stack-crds/531ce5c7d64da93901b643a2b737d36117a2776a.tar.gz

Despite reporting success

❯ flux get source git -A
NAMESPACE      NAME                                        READY    MESSAGE                                                                                    REVISION                                                                 SUSPENDED
flux-system    kube-prometheus-stack-crds                  True     Fetched revision: kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a    kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a    False  

To Reproduce

Have a GitRepository Reconcile it Check the source controller data

Expected behavior

Repository not empty

Additional context

This bug was introduces with v0.13.0, previous version worked fine.

$ flux --version
flux version 0.13.0
$ flux check
► checking prerequisites
✔ kubectl 1.21.0 >=1.18.0-0
✔ Kubernetes 1.20.6+k3s1 >=1.16.0-0
► checking controllers
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.11.1
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.10.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.13.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.12.0
✔ all checks passed
$ kubectl -n <namespace> get all
NAME                                           READY   STATUS    RESTARTS   AGE
pod/kustomize-controller-78bdd99b66-d2nqm      1/1     Running   0          11h
pod/helm-controller-69667f94bc-jk9bj           1/1     Running   0          11h
pod/notification-controller-5c4d48f476-7k9rh   1/1     Running   0          11h
pod/source-controller-6c458cdb8d-jlzdn         1/1     Running   0          11h

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/notification-controller   ClusterIP   10.1.123.23   <none>        80/TCP    28h
service/source-controller         ClusterIP   10.1.234.45    <none>        80/TCP    28h
service/webhook-receiver          ClusterIP   10.1.45.56    <none>        80/TCP    28h

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kustomize-controller      1/1     1            1           28h
deployment.apps/helm-controller           1/1     1            1           28h
deployment.apps/notification-controller   1/1     1            1           28h
deployment.apps/source-controller         1/1     1            1           28h

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/source-controller-785dd8f8b9         0         0         0       28h
replicaset.apps/kustomize-controller-78bdd99b66      1         1         1       11h
replicaset.apps/kustomize-controller-97dbdcb99       0         0         0       28h
replicaset.apps/helm-controller-69667f94bc           1         1         1       11h
replicaset.apps/helm-controller-5bbd94c75            0         0         0       28h
replicaset.apps/notification-controller-5c4d48f476   1         1         1       11h
replicaset.apps/notification-controller-d66bdc787    0         0         0       28h
replicaset.apps/source-controller-6c458cdb8d         1         1         1       11h
$ kubectl -n flux-system logs deploy/source-controller
{"level":"info","ts":"2021-04-23T06:48:06.075Z","logger":"controller.gitrepository","msg":"Reconciliation finished in 690.693629ms, next run in 30m0s","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"kube-prometheus-stack-crds","namespace":"flux-system"}
kubectl -n flux-system logs deploy/kustomize-controller
{"level":"error","ts":"2021-04-23T07:48:01.553Z","logger":"controller.kustomization","msg":"Reconciliation failed after 422.112691ms, next try in 30m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"kube-prometheus-stack-crds","namespace":"flux-system","revision":"kube-prometheus-stack-15.2.0/531ce5c7d64da93901b643a2b737d36117a2776a","error":"apply failed: error: no objects passed to apply\n"}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

Any chance the https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/.sourceignore is still the same, and excluding your directory?

Pulling in v0.12.1 of the source-controller should solve this issue, flux release is building.

Have identified the core of the issue, will work on a patch that should be released later today.