argo-cd: App of apps sync-waves not working in 1.8.1

Describe the bug

Using 1.8.1 - app of app sync waves are not working - all the apps sync at once. If I delete them, and sync again, sync waves then work. Can delete/re-sync, several times after, and each time sync-waves work. But if I make a new commit to the target brach, delete all the apps and sync - same thing happens - all the apps sync at once. Delete them, and re-sync - then sync waves work. Seems there is an issue honoring sync-waves when the repo is first cloned and/or refreshed.

I am using a mono repo and have the new path annotation on each app. Auto sync is on for all the child apps, and I manually issue the sync command for the bootstrap app.

I have added the required resource.customizations block to the argocd config map.

  resource.customizations: |
    argoproj.io/Application:
      health.lua: |
        hs = {}
        hs.status = "Healthy"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.health ~= nil then
            hs.status = obj.status.health.status
            hs.message = obj.status.health.message
          end
        end
        return hs

To Reproduce

Create an app of apps.
Add child apps with auto sync on and place them in different waves using the sync-wave annotation.
Issue argocd app create on the bootstrap manifest. You will see the bootstrap app get created.
Then argocd app sync on the bootstrap app. You will see the bootstrap app sync, and then all the child apps will begin to sync without waiting for the previous wave to sync first and become healthy.
Do an argocd app delete on the bootstrap app (or just do it all in the UI) and wait for the everything to get removed).
Then do the the steps again and sync waves will work.

Expected behavior

Sync waves should wrk the first time around

Version

$ argocd version
argocd: v1.8.1+c2547dc.dirty
  BuildDate: 2020-12-10T04:44:20Z
  GitCommit: c2547dca95437fdbb4d1e984b0592e6b9110d37f
  GitTreeState: dirty
  GoVersion: go1.15.5
  Compiler: gc

  Platform: darwin/amd64
argocd-server: v1.8.1+c2547dc
  BuildDate: 2020-12-10T02:59:21Z
  GitCommit: c2547dca95437fdbb4d1e984b0592e6b9110d37f
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.8.1 2020-07-16T00:58:46Z
  Helm Version: v3.4.1+gc4e7485
  Kubectl Version: v1.17.8
  Jsonnet Version: v0.17.0

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 15
Comments: 35 (9 by maintainers)

Commits related to this issue

trying to reproduce https://github.com/argoproj/argo-cd/issues/5146 — committed to alexmt/argocd-example-apps by alexmt 4 years ago
Rollback ArgoCD to 1.7.11 because of https://github.com/argoproj/argo-cd/issues/5146 — committed to mcanevet/devops-stack by mcanevet 3 years ago
Rollback ArgoCD to 1.7.11 because of https://github.com/argoproj/argo-cd/issues/5146 — committed to camptocamp/devops-stack by mcanevet 3 years ago
Rollback ArgoCD to 1.7.11 because of https://github.com/argoproj/argo-cd/issues/5146 — committed to pburgisser/devops-stack by mcanevet 3 years ago

Most upvoted comments

Sync-waves have also broken for us after upgrading to 1.8.1. However our setup is slightly different:

no app of apps
autosync disabled

All the outofsync resources in the screenshot had sync-waves set with negative numbers so they would get applied before the other resources. Screen Shot 2021-01-12 at 10 19 14 AM

devonhk on Jan 12, 2021

@alexmt 1.8.5 does not seem to fix this issue. I was able to reproduce this using version 1.8.7, while the sync-wave feature works OK with version 1.7.14.

ArieLevs on Mar 6, 2021

I can join in and say that this does not work for me either as of today. Version 1.8.2

Thanks for acting so swiftly @alexmt!

luddskunk on Jan 22, 2021

I’m sorry. Got distracted by meetings and did not have time to work on it. Keep trying to reproduce the issue today.

alexmt on Jan 21, 2021

After further tests, I found that version 1.7.11 works as expected.

Versions 1.8.0 and later don’t.

irizzant on Jan 19, 2021

I have the same issue when migrating to 1.7.11 to 1.8.7 (same with 2.0) with app of apps and argocd app custom health check. At parent’s app creation all the child’s apps are created simultaneously despite wave annotation.

When argocd create the first child app, it is directly healthy and after move back to processing but the others child apps (with different wave) have been created and start to sync theirs content.

After some digging, i found a bad initialization in lua script. By default, the status is set to Healthy but when the argocd app is created the app status is empty and skip the if block to return the default status.

I updated the default status to Progressing and the issue disapear.

The final lua script:

resource.customizations: |
    argoproj.io/Application:
      health.lua: |
        hs = {}
        hs.status = "Progressing"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.health ~= nil then
            hs.status = obj.status.health.status
            hs.message = obj.status.health.message
          end
        end
        return hs

LucasBoisserie on May 20, 2021