argo-cd: level=fatal msg="Operation has completed with phase: Running

Argocd sync action fail with the below error.

Command - argocd app sync test-appplication Error log - time=“2021-02-24T06:46:56Z” level=fatal msg=“Operation has completed with phase: Running”

Version Paste the output from argocd version here. argocd: v1.8.1+c2547dc.dirty BuildDate: 2020-12-10T04:45:44Z GitCommit: c2547dca95437fdbb4d1e984b0592e6b9110d37f GitTreeState: dirty GoVersion: go1.15.5 Compiler: gc Platform: darwin/amd64

Logs

time="2021-02-24T06:40:31Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:40:31Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=21 fields.level=1 git_ms=6 health_ms=0 live_ms=1 settings_ms=0 sync_ms=0 time_ms=69
time="2021-02-24T06:40:52Z" level=info msg="Refreshing app status (normal refresh requested), level (2)" application=test-application
time="2021-02-24T06:40:52Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:40:52Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:40:52Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:40:52Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:40:52Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:40:52Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=7 version_ms=0
time="2021-02-24T06:40:52Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:40:52Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:40:52Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=32 fields.level=1 git_ms=7 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=86
time="2021-02-24T06:40:53Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=942 unmarshal_ms=942 version_ms=0
time="2021-02-24T06:40:53Z" level=info msg="Skipping auto-sync: application status is Synced" application=test-application
time="2021-02-24T06:40:53Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:40:53Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:40:53Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:40:53Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=7 version_ms=0
time="2021-02-24T06:40:53Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:40:53Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=13 fields.level=2 git_ms=942 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=978
time="2021-02-24T06:40:53Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:40:53Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=20 fields.level=1 git_ms=7 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=40
time="2021-02-24T06:42:00Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:42:00Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:42:00Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:42:00Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:42:00Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:51Z" level=info msg="updated 'test-application' operation (phase: Running)"
time="2021-02-24T06:46:51Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:bbc22125b61e602efb84e9a53a13450eb8789834,Prune:false,DryRun:false,SyncStrategy:&SyncStrategy{Apply:nil,Hook:&SyncStrategyHook{SyncStrategyApply:SyncStrategyApply{Force:false,},},},Resources:[]SyncOperationResource{},Source:nil,Manifests:[],SyncOptions:[CreateNamespace=true],} {admin false} [] {0 nil}}" application=test-application
time="2021-02-24T06:46:51Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:51Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:51Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:51Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:51Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=22 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=70
time="2021-02-24T06:46:51Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:51Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:51Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:51Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:51Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:51Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:51Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=21 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=68
time="2021-02-24T06:46:53Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=1414 unmarshal_ms=1414 version_ms=0
time="2021-02-24T06:46:53Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:53Z" level=info msg=Syncing application=test-application skipHooks=false started=false syncId=00022-yxHpZ
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=11 unmarshal_ms=11 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Skipping auto-sync: another operation is in progress" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Updated health status: Healthy -> Progressing" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=18 fields.level=1 git_ms=7 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=78
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=11 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=75
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=8 unmarshal_ms=8 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Initiated automated sync to 'e160b33c8e760e85dbedc221938b5d312fe166b1'" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=OperationStarted type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Initiated automated sync to 'e160b33c8e760e85dbedc221938b5d312fe166b1'" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Updated health status: Healthy -> Progressing" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=26 fields.level=1 git_ms=8 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=85
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=21 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=107
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=23
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Skipping auto-sync: another operation is in progress" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=72
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=15 unmarshal_ms=15 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=15 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=59
time="2021-02-24T06:46:54Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=37 fields.level=1 git_ms=15 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=67
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=24 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=41
time="2021-02-24T06:46:54Z" level=info msg="sync/terminate complete" application=test-application duration=1.355674294s syncId=00022-yxHpZ
time="2021-02-24T06:46:54Z" level=info msg="updated 'test-application' operation (phase: Succeeded)"
time="2021-02-24T06:46:54Z" level=info msg="Sync operation to bbc22125b61e602efb84e9a53a13450eb8789834 succeeded" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=OperationCompleted type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=7 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:54Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:54Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=7 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=63
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=8 unmarshal_ms=8 version_ms=0
time="2021-02-24T06:46:54Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=26 fields.level=1 git_ms=8 health_ms=2 live_ms=0 settings_ms=0 sync_ms=0 time_ms=57
time="2021-02-24T06:46:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:54Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:54Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:55Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=21 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=40
time="2021-02-24T06:46:55Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:55Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:46:55Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:55Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=27 fields.level=1 git_ms=7 health_ms=0 live_ms=1 settings_ms=0 sync_ms=0 time_ms=73
time="2021-02-24T06:46:55Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:55Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:55Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=910 unmarshal_ms=910 version_ms=0
time="2021-02-24T06:46:55Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=20 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=38
time="2021-02-24T06:46:55Z" level=info msg="Skipping auto-sync: most recent sync already to bbc22125b61e602efb84e9a53a13450eb8789834" application=test-application
time="2021-02-24T06:46:55Z" level=info msg="Updated health status: Progressing -> Healthy" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:55Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:55Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=11 fields.level=2 git_ms=910 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=977
time="2021-02-24T06:46:55Z" level=info msg="Refreshing app status (normal refresh requested), level (2)" application=test-application
time="2021-02-24T06:46:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:55Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:55Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:55Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=4 version_ms=0
time="2021-02-24T06:46:55Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:55Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=35 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=138
time="2021-02-24T06:46:55Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:55Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:55Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:56Z" level=info msg="updated 'test-application' operation (phase: Running)"
time="2021-02-24T06:46:56Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:bbc22125b61e602efb84e9a53a13450eb8789834,Prune:false,DryRun:false,SyncStrategy:nil,Resources:[]SyncOperationResource{},Source:nil,Manifests:[],SyncOptions:[CreateNamespace=true],} { true} [] {5 nil}}" application=test-application
time="2021-02-24T06:46:56Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:56Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:56Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=7 version_ms=0
time="2021-02-24T06:46:56Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:56Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=20 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=39
time="2021-02-24T06:46:56Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:56Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:56Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:56Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:56Z" level=info msg=Syncing application=test-application skipHooks=false started=false syncId=00023-BlFyD
time="2021-02-24T06:46:56Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:56Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:56Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=27 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=48
time="2021-02-24T06:46:56Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=914 unmarshal_ms=914 version_ms=0
time="2021-02-24T06:46:56Z" level=error msg="Failed to initiate auto-sync to bbc22125b61e602efb84e9a53a13450eb8789834: rpc error: code = FailedPrecondition desc = another operation is already in progress" application=test-application
time="2021-02-24T06:46:56Z" level=info msg="Updated health status: Progressing -> Healthy" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:56Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:56Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=11 fields.level=2 git_ms=914 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=965
time="2021-02-24T06:46:56Z" level=info msg="Refreshing app status (normal refresh requested), level (2)" application=test-application
time="2021-02-24T06:46:56Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:56Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:56Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:56Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:57Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:57Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=20 fields.level=1 git_ms=7 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=85
time="2021-02-24T06:46:57Z" level=info msg="sync/terminate complete" application=test-application duration=1.175340765s syncId=00023-BlFyD
time="2021-02-24T06:46:57Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:57Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:57Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:57Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:57Z" level=info msg="updated 'test-application' operation (phase: Succeeded)"
time="2021-02-24T06:46:57Z" level=info msg="Sync operation to bbc22125b61e602efb84e9a53a13450eb8789834 succeeded" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=OperationCompleted type=Normal
time="2021-02-24T06:46:57Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:57Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=20 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=37
time="2021-02-24T06:46:57Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:57Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:57Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:57Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:46:57Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:57Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=18 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=38
time="2021-02-24T06:46:57Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=1035 unmarshal_ms=1035 version_ms=0
time="2021-02-24T06:46:57Z" level=info msg="Skipping auto-sync: another operation is in progress" application=test-application
time="2021-02-24T06:46:57Z" level=info msg="Updated sync status: OutOfSync -> Synced" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:57Z" level=info msg="Updated health status: Healthy -> Progressing" application=test-application dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:57Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:57Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=17 fields.level=2 git_ms=1036 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=1098
time="2021-02-24T06:46:57Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" application=test-application
time="2021-02-24T06:46:57Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:57Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: test-application)" application=test-application
time="2021-02-24T06:46:58Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:58Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:58Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:58Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:46:58Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:46:58Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:46:58Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=78
time="2021-02-24T06:46:58Z" level=info msg="getRepoObjs stats" application=test-application build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=890 unmarshal_ms=890 version_ms=0
time="2021-02-24T06:46:58Z" level=info msg="Skipping auto-sync: application status is Synced" application=test-application
time="2021-02-24T06:46:58Z" level=info msg="Update successful" application=test-application
time="2021-02-24T06:46:58Z" level=info msg="Reconciliation completed" application=test-application dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=11 fields.level=2 git_ms=890 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=926
time="2021-02-24T06:46:58Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:46:58Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:46:58Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:46:58Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:46:58Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:46:58Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=27 fields.level=1 git_ms=7 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=47
time="2021-02-24T06:47:07Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:47:07Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:47:07Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:47:07Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:47:07Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:47:07Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:47:08Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:47:08Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:47:08Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:47:08Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:47:08Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:47:08Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:47:08Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:47:08Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:47:08Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:47:08Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=63
time="2021-02-24T06:47:09Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:47:09Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:47:09Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=6
time="2021-02-24T06:47:09Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:47:09Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:47:09Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:47:09Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:47:09Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:47:09Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=37
time="2021-02-24T06:47:25Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:47:25Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:47:25Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:47:26Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:47:26Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:47:26Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:47:26Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=9 unmarshal_ms=9 version_ms=0
time="2021-02-24T06:47:26Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:47:26Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:47:26Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=9 health_ms=0 live_ms=1 settings_ms=0 sync_ms=0 time_ms=76
time="2021-02-24T06:47:39Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:47:39Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:47:39Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:47:43Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:47:43Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:47:43Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:47:43Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:47:43Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:47:44Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:47:44Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=22 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=66
time="2021-02-24T06:47:54Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:47:54Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:47:54Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:47:55Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:47:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:47:55Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:47:55Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:47:55Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:47:55Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:47:55Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=24 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=77
time="2021-02-24T06:48:00Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:48:00Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:48:00Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=
time="2021-02-24T06:48:01Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=21 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=122
time="2021-02-24T06:48:05Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:48:05Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:48:05Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:48:06Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:48:06Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:48:06Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:48:06Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:48:06Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:48:06Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=23 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=48
time="2021-02-24T06:48:07Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:48:07Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:48:07Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:48:07Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:48:07Z" level=info msg="No status changes. Skipping patch" application=xyz
time="2021-02-24T06:48:07Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=19 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=38
time="2021-02-24T06:48:13Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=xxxxx
time="2021-02-24T06:48:13Z" level=info msg="No status changes. Skipping patch" application=xxxxx
time="2021-02-24T06:48:13Z" level=info msg="Reconciliation completed" application=xxxxx dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=xxxxx dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:48:14Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:48:14Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:48:14Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:48:14Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:48:14Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:48:25Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:48:25Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=20 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=63
time="2021-02-24T06:48:44Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:48:44Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:48:44Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:48:45Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:48:45Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:48:45Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:48:45Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2021-02-24T06:48:45Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:48:45Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:48:45Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=25 fields.level=1 git_ms=5 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=69
time="2021-02-24T06:48:56Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:48:56Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:48:56Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:48:57Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:48:57Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:48:57Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:48:57Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=7 version_ms=0
time="2021-02-24T06:48:57Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:48:57Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:48:57Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=18 fields.level=1 git_ms=7 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=69
time="2021-02-24T06:49:56Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=xxxxx
time="2021-02-24T06:49:56Z" level=info msg="No status changes. Skipping patch" application=xxxxx
time="2021-02-24T06:49:56Z" level=info msg="Reconciliation completed" application=xxxxx dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=xxxxx dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:49:57Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:49:57Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:49:57Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:49:57Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:49:57Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal
time="2021-02-24T06:49:57Z" level=info msg="Update successful" application=xyz
time="2021-02-24T06:49:57Z" level=info msg="Reconciliation completed" application=xyz dedup_ms=0 dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" diff_ms=20 fields.level=1 git_ms=6 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=69
time="2021-02-24T06:50:05Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=test-application
time="2021-02-24T06:50:05Z" level=info msg="No status changes. Skipping patch" application=test-application
time="2021-02-24T06:50:05Z" level=info msg="Reconciliation completed" application=test-application dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=test-application dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:50:07Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=xxxxx
time="2021-02-24T06:50:07Z" level=info msg="No status changes. Skipping patch" application=xxxxx
time="2021-02-24T06:50:07Z" level=info msg="Reconciliation completed" application=xxxxx dest-name="arn:aws:eks:us-west-2:******:cluster/******" dest-namespace=xxxxx dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" fields.level=0 time_ms=4
time="2021-02-24T06:50:09Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=xyz
time="2021-02-24T06:50:09Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-02-24T06:50:09Z" level=info msg="Comparing app state (cluster: https://******.gr7.us-west-2.eks.amazonaws.com, namespace: argocd)" application=xyz
time="2021-02-24T06:50:09Z" level=info msg="getRepoObjs stats" application=xyz build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=6 version_ms=0
time="2021-02-24T06:50:09Z" level=info msg="Updated health status: Degraded -> Healthy" application=xyz dest-namespace=argocd dest-server="https://******.gr7.us-west-2.eks.amazonaws.com" reason=ResourceUpdated type=Normal

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 37 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Yes, for me that is the case. I’m triggering a manual sync and then do a wait in my cd pipeline (just like mentioned in the docs)

@jochemvankessel As a workaround, to simulate a “manual sync” in a cd pipeline you can first get with --refresh, then wait. Something like this: argocd --grpc-web app get --refresh my-app > /dev/null && argocd --grpc-web app wait my-app --sync --health

@alexmt and I were able to reliably reproduce this today - it appeared to be a combination of having auto sync enabled and then attempting to do a manual sync on the application. Is that the case for other folks here?

Yes, for me that is the case. I’m triggering a manual sync and then do a wait in my cd pipeline (just like mentioned in the docs)

I have seen this issue as well w/o having 2 application controllers

@alexmt and I were able to reliably reproduce this today - it appeared to be a combination of having auto sync enabled and then attempting to do a manual sync on the application. Is that the case for other folks here?

I think I understand the issue. The problem will happen more frequently with auto sync enabled, and a second sync operation, which due to timing reasons, can initiate a new operation immediately after previous operation completed. I can also see slow controllers contributing to the issue, as well as webhooks that may fast-track operation initiations.

The sequence of events to cause this:

  1. sync operation 1 is started and is Running
  2. waitOnApplicationStatus is called against the app to wait for operation 1 to complete
  3. user makes a git commit which will cause a new sync operation 2 to eventually happen
  4. operation 1 completes Successful
  5. while we are still in waitOnApplicationStatus, we refresh the app. Historically, this refresh is done because we want the sync status to be up-to-date before returning to the caller. This was added because we have separate operation and status reconcilers, and so it is possible for an operation to complete but the app’s sync status to be inaccurate.
  6. due to the refresh in step 5, we might trigger operation 2, which transition the app immediately back from Successful to Running. We return that app with the Running operationState, despite that Running state being for operation 2.
  7. The app wait caller examines the returned app, and sees that operation phase is not Successful and fails

So in summary, the fact that we refresh the app right before we return from waitOnApplicationStatus, could cause a new operation to happen and so we return the app in a Running state, even though we just finished the previous operation.

I think the solution should be:

  1. waitOnApplicationStatus should return as a second return value, the operationState of the operation it was monitoring at the time of the wait. Because this operationState may be different than the status.operationState reflected in the app object that we return.
  2. The callers to waitOnApplicationStatus (argocd app wait, argocd app sync) should examine this new operationState returned by waitOnApplicationStatus, and not rely on the status.operationState of the app object (since it cannot be trusted due to a possible second operation).

Still happening in 2.1.5

I’m experiencing this issue as well without 2 application controllers, specifically when using postSync hooks.

Any updates?

Same here still facing the issue in 2.4.11

Same issue here with clean install of 2.1.3. Checked, only one controller pod is running.

Hi, the problem was supposed to be fixed in ArgoCD 2.1.0 but it’s still happening for me frequently with v2.1.1+aab9542. Any suggestions?

@alexmt I have this issue again

Same here still facing the issue in 2.4.12

still have the same issue in 2.4.11

@jochemvankessel As a workaround, to simulate a “manual sync” in a cd pipeline you can first get with --refresh, then wait. Something like this: argocd --grpc-web app get --refresh my-app > /dev/null && argocd --grpc-web app wait my-app --sync --health

I followed your suggestion here and it worked in v2.1.6. Thanks!