argo-cd: App of Apps not working when using kustomize applications

I think I have finally found the trigger point.

All kustomization.yaml files will be ignored, or kind: Kustomization configs will throw the error above, if the application source directory recurse: true is set. recurse: false or omitting recurse works without issue.

Failing scenario:

kind: Application
spec:
  source:
    directory:
      recurse: true

Successful scenario:

kind: Application
spec:
  source:
    directory:
      recurse: false

I have the same error w/ a Helm App of Apps. All application manifests specify namespace: argocd.

edit: for anyone who found themselves here while trying out the App of App patterns. If your child apps are deploying to a different cluster, make sure your parent app has destination server: https://kubernetes.default.svc as that is where the child ArgoCD Application resources are created and tracked.

I think I have finally found the trigger point.

All kustomization.yaml files will be ignored, or kind: Kustomization configs will throw the error above, if the application source directory recurse: true is set. recurse: false or omitting recurse works without issue.

Failing scenario:

kind: Application
spec:
  source:
    directory:
      recurse: true

Successful scenario:

kind: Application
spec:
  source:
    directory:
      recurse: false

Is there any way around this? I’m specifically trying to use a number of kustomize.yml configs in my repo and rely on all applications being recursively deployed from a kustomize file but I’m seeing the same sync issues unless I disable recursion

In my case I’m trying to run a bootstrap that then adds an app-of-apps to argocd where I expect my source of truth for everything to be.

.
├── argocd
│  ├── argocd
│  │  └── kustomization.yml
│  └── sealed-secrets
│     └── kustomization.yml
├── bootstrap
│  └── kustomization.yml
├── juju
│  ├── deployment.yml
│  └── README.md
├── kustomize
│  ├── argocd
│  │  ├── app.yml
│  │  ├── cm.rbac.yml
│  │  ├── cm.repos.yml
│  │  ├── cm.sso.yml
│  │  ├── kustomization.yml
│  │  ├── overlay.argocd.yml
│  │  ├── overlay.secrets.yml
│  │  ├── resource.ingress.yml
│  │  ├── resource.namespace.yml
│  │  ├── resource.project.yml
│  │  └── resource.secrets.yml
│  └── sealed-secrets
│     ├── app.yml
│     └── kustomization.yml
└── README.md

# argocd/argocd/kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../kustomize/argocd/

# kustomize/argocd/app.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  source:
    path: argocd
    directory:
      recurse: true
...

is this really still an issue? I’m POC’ing ArgoCD now but this bug makes it seem like App of Apps configuration is broken and so what’s the point

Same problem as above with argo 1.5.4 and kustomize templates. However, mine says sync is successful, however no application pods are created. Works fine if i create app directly. I have a full repository available if someone can double check:

https://github.com/VirtoCommerce/vc-deployments/tree/3e831b31db71fe8f9867bd6257ea69c0560c9471

After more experiements, I was able to fix the issue by adding namespace under yaml definition in the apps folder

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: webstore-demo
  **namespace: argocd**

It has to have argocd namespace, anything else and it doesn’t work. In my deployment I add everything into “demo” namespace and argocd runs under default argocd namespace.

It does seem to work, but there are some gotchas along the way. The one that had us really twisted is that you can NOT set up the app to be recursive or it doesn’t properly detect what is going on and ArgoCD gets terribly confused.

One thing I’ve discovered in my research is the API endpoint yourargocd.com/api/v1/repositories/{repo}/apps. It appears that if if hit, with the problematic {repo} name, it would result in some context for where the issue manifests. It is document on your ArgoCD UI at the path yourargocd.com/swagger-ui#operation/ListApps.

I’m wondering, is it that ArgoCD isn’t recognizing Kustomize is involved so trying to actually add the Kustomization type to the cluster rather than processing it? If so, how does ArgoCD get triggered on this?

In the interest of discovering that information, but not being a Golang guy, I found these hopefully relevant bits. There is apparently two ways this is supposed to be triggered. There is some documentation in the Tool Detection Doc, but it left me with some questions and clearly it isn’t working exactly as documented.

First, auto-detection based on the contents of the repo.

So what does the code use to determine if a directory of code should be handled as a Kustomize appType? The answer appears to be on line 168 of kustomize.go.

var KustomizationNames = []string{"kustomization.yaml", "kustomization.yml", "Kustomization"}

This is then used on line 181 in the isKustomization function. It appears very consistent with the docs.

This info is used by reposerver/repository/repository.go on line 100 in the ListApps function. There is also a ListApps function in server/repository/repository.go on line 155, but this one doesn’t use the discovery.Discover function, but instead calling the API ListApps endpoint.

Second, explicit identification in the Application resource as spec.source.kustomize. I’m not clear on what this should be if I just want to explicitly say this should use kustomize, but not change other characteristics. Maybe spec.source.kustomize: []? pkg/apis/application/v1alpha1/types.go line 2260 seems to be involved and best I can tell anything non-nil would be acceptable.

The two methods of detection appear to come together in reposerver/repository/repository.go function GetAppSourceType where it first checks ExplicitType, then if that comes back blank, uses discovery.AppType to tell GenerateManifests which type of application to output manifests for and hopefully hits line 378 where it gets processed by Kustomize.

My suspicion is that we aren’t making it this far, but I don’t know how to watch the code at work.

I’m facing the same issue and can’t figure out why it isn’t working. I’m sure it has worked at some point.

See here my app of apps.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: opa-policies
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: opa-policies
    server: {{ .Values.spec.destination.server }}
  project: my-project
  source:
    path: opa-policies
    repoURL: {{ .Values.spec.source.repoURL }}
    targetRevision: {{ .Values.spec.source.targetRevision }}
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Then in the folder opa-policies I have the following Kustomize.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# namespace: gatekeeper-system

resources:
  # - https://github.com/sigstore/cosign-gatekeeper-provider/raw/main/manifest/deployment.yaml
  # - https://github.com/sigstore/cosign-gatekeeper-provider/raw/main/manifest/service.yaml
  # - https://github.com/sigstore/cosign-gatekeeper-provider/raw/main/manifest/provider.yaml
  - templates/container-resources-template.yaml
  - templates/image-tagging-template.yaml
  - templates/cosign-template.yaml
  - constraints/image-must-not-have-latest-tag-constraint.yaml
  - constraints/image-must-have-limits-and-requests-constraint.yaml
  - constraints/image-signed-constraint.yaml

# patches:
#   - target:
#       kind: Provider
#       group: externaldata.gatekeeper.sh
#       version: v1alpha1
#       name: cosign-gatekeeper-provider
#     patch: |-
#       - op: add
#         path: /metadata/annotations/argocd.argoproj.io~1sync-wave
#         value: -2
#       - op: add
#         path: /spec/insecureTLSSkipVerify
#         value: true
#       - op: replace
#         path: /spec/timeout
#         value: 15
#       - op: replace
#         path: /spec/url
#         value: http://cosign-gatekeeper-provider.gatekeeper-system:8090/validate
#   - target:
#       kind: Deployment
#       group: apps
#       version: v1
#       name: cosign-gatekeeper-provider
#     patch: |-
#       - op: add
#         path: /metadata/annotations/argocd.argoproj.io~1sync-wave
#         value: -2
#   - target:
#       kind: Service
#       version: v1
#       name: cosign-gatekeeper-provider
#     patch: |-
#       - op: add
#         path: /metadata/annotations/argocd.argoproj.io~1sync-wave
#         value: -2

The commented lines I have tested with to figure out if it is related to any kustomize specifics in combination with ArgoCD.

The result is that ArgoCD thinks all is synchronized, however the App is not created.

time="2022-08-16T09:13:35Z" level=info msg="Applying resource Config/config in cluster: https://172.20.0.1:443, namespace: gatekeeper-system"
time="2022-08-16T09:13:35Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'config.config.gatekeeper.sh/config unchanged'" application=my-policies kind=Config name=config namespace=gatekeeper-system phase=Sync syncId=00052-MtTBa
time="2022-08-16T09:13:35Z" level=info msg="Applying resource Application/opa-policies in cluster: https://172.20.0.1:443, namespace: argocd"
time="2022-08-16T09:13:35Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'application.argoproj.io/opa-policies unchanged'" application=my-policies kind=Application name=opa-policies namespace=argocd phase=Sync syncId=00052-MtTBa

See this screenshot all is synced according to ArgoCD, but when clicking the red circled icon the app doesn’t actually exist, neither it is applied.

See here the screenshot showing the app that pretends to by synced actually isn’t.

Strange thing is there are no errors, and nothing in the logs.

I suppose it makes sense that this wouldn’t work, since there is ambiguity in how to handle a directory containing both an Application manifest as well as a Kustomize entrypoint in some sub-directory with recursion enabled - do we deploy the Kustomize manifest as part of the base app or as part of the dependent app?

It would be nice if Argo was “smarter” about how to deploy the app of apps when using Kustomize, i.e., allow recursive directories, but only for Application manifests. Maybe implementing some kind of globbing filter would help:

spec:
  directory:
    recurse: true
    pattern: **/app.yaml

For the time being, my seed app is set to directory.recurse: false and pointing to a flat directory of app manifests (clusters/cluster-1). Each app manifest points to its own sub-directory (clusters/cluster-1/my-app), containing Kustomize manifests or Helm charts. Recursion can be independently configured for each app as needed.

That issue still exist I m using 2.0.5 and in the middle of the POC with kustomization app of apps repo. I already tried @Woland2k and @FearTheBadger recommendation but still have no luck child apps stuck at OutOfSync

argocd: v2.0.5+4c94d88.dirty BuildDate: 2021-07-23T05:12:02Z GitCommit: 4c94d886f56bcb2f9d5b3251fdc049c2d1354b88 GitTreeState: dirty GoVersion: go1.16.6 Compiler: gc Platform: darwin/amd64