argo-cd: AutoSync doesn't work anymore

Describe the bug

When I change my bootstrap-application in git it doesn’t get synced anymore to kuberntes, although AutoSync in argocd is enabled. Only when I click ‘Refresh’ in argocd webconsole, the application gets OutOfSync and Sync process starts.

To Reproduce

  1. change something in the git-repo of my bootstrap-application
  2. wait the AutoSync interval of 3 minutes, nothing happens (also waited for an hour)
  3. clicked ‘refresh’ in the webconsole of argocd for this application
  4. application gets OutOfSync and starts synchronization process

Example

I am not able to create an example, because it is vey specific to our environment. However, maybe I can narrow down to a basic example. Right now I do not know if that happens with every application in our environment or just with our boostrap-application.

Expected behavior

AutoSync should sync the application without any user interaction.

Screenshots

Version

1.2.0

Logs

logs from application-controller, when git-repo and kubernetes is already out-of-sync, but argocd doesn’t recognize it (since we have lots of applications defined, I tried to filter a little bit for the bootstrap-application):

time="2019-09-22T09:41:12Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=bootstrap-application
W0922 09:41:12.393879       1 listers.go:77] can not retrieve list of objects using index : Index with name namespace does not exist
time="2019-09-22T09:41:12Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=bootstrap-application
time="2019-09-22T09:41:12Z" level=info msg="Skipping auto-sync: application status is Synced" application=bootstrap-application
time="2019-09-22T09:41:12Z" level=info msg="Update successful" application=bootstrap-application
time="2019-09-22T09:41:12Z" level=info msg="Reconciliation completed" application=bootstrap-application fields.level=1 time_ms=281.376147
time="2019-09-22T09:41:12Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=bootstrap-application
W0922 09:41:12.674738       1 listers.go:77] can not retrieve list of objects using index : Index with name namespace does not exist
time="2019-09-22T09:41:12Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=bootstrap-application
time="2019-09-22T09:41:12Z" level=info msg="Skipping auto-sync: application status is Synced" application=bootstrap-application
time="2019-09-22T09:41:12Z" level=info msg="Update successful" application=bootstrap-application
time="2019-09-22T09:41:12Z" level=info msg="Reconciliation completed" application=bootstrap-application fields.level=1 t

log when I click Refresh:

time="2019-09-22T09:43:57Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=bootstrap-application
time="2019-09-22T09:43:57Z" level=info msg="copying credentials" credUrl="https://gitlab.e-bk.m086/" repoURL="https://gitlab.e-bk.m086/arz-container-plattform/k8s-platform.git"
time="2019-09-22T09:43:58Z" level=info msg="Initiated automated sync to 'd1ac81ea342f68997be69ff2befbf2b0ec1b045f'" application=bootstrap-application dest-namespace=argocd dest-server="https://kubernetes.default.svc" reason=OperationStarted type=Normal
time="2019-09-22T09:43:58Z" level=info msg="Initiated automated sync to 'd1ac81ea342f68997be69ff2befbf2b0ec1b045f'" application=bootstrap-application
time="2019-09-22T09:43:58Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=bootstrap-application dest-namespace=argocd dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2019-09-22T09:43:58Z" level=info msg="updated 'bootstrap-application' operation (phase: Running)"
time="2019-09-22T09:43:58Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:d1ac81ea342f68997be69ff2befbf2b0ec1b045f,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[],Source:nil,Manifests:[],}}" application=bootstrap-application
time="2019-09-22T09:43:58Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=bootstrap-application
time="2019-09-22T09:43:58Z" level=info msg="copying credentials" credUrl="https://gitlab.e-bk.m086/" repoURL="https://gitlab.e-bk.m086/arz-container-plattform/k8s-platform.git"
time="2019-09-22T09:43:58Z" level=info msg="Update successful" application=bootstrap-application

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 4
  • Comments: 40 (9 by maintainers)

Commits related to this issue

Most upvoted comments

So, 2023. Issue still exists. Argo version: v2.6.0+352e51f.

We’ve just upgraded to 1.2.4 and sadly didn’t fix the issue where the change in the git repo isn’t picked up. I’m short on time right now to debug it but probably around the 1.3 release I can really sit down and do some tests.

Looks like this is a bug related to git optimization. Controller incorrectly decides to skip fetching new version from Git if application is big and target state changes frequently. This definitely deserves patch release. hope to get it today/tomorrow

Keeping the issue open until getting confirmation that the bug is fixed.

Release v1.2.5 is available https://github.com/argoproj/argo-cd/releases/tag/v1.2.5 . Please give it a try.

Finally found the real reason ( at least I’m 99% sure about it 😃 ) . The controller incorrectly updates the state.reconciledAt field even if app reconciliation was performed using cached manifests.

PR is in review: https://github.com/argoproj/argo-cd/pull/2581

Will create v1.2.5 once it is merged.

I also encounterd this issue. I’m using ArgoCD v1.2.3 and kustomize. In v1.2.3, has this issue completely resolved? In my case, this sometimes occurred using CronJob, however, when trying to sync other resoueces(e.g. Deployment, Ingress), no problems did.

thanks @amarrella . I don’t know the exact reason for my problem. After restarting the application controller everything works fine. Could be that it got in some error situation after I experimented around due to https://github.com/argoproj/argo-cd/issues/2257 . But thanks for the tip with the timeout! looks very useful!

I think I encountered the same problem recently. I don’t know the size of your boostrap application, but what happened with us is that rendering the manifest in argocd-repo-server took longer than the default timeout (60s) and therefore application controller wasn’t able to detect a change.

Try to increase --repo-server-timeout-seconds and see if this helps