argo-cd: waiting for completion of hook and hook never succeds

HI,

We are seeing this issue quite often where app sync is getting stuck in “waiting for completion of hook” and these hooks are never getting completed

As you can see the below application got stuck on secret creation phase and some how that secret never got created

image

Stripped out all un-necessary details. Now this is how the secret is created and used by the job.

apiVersion: v1
kind: Secret
metadata:
  name: {{ include "xxx.fullname" . }}-migrations-{{ .Chart.AppVersion }}
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
    helm.sh/hook-weight: "-5"
type: Opaque
data:
  xxxx

apiVersion: batch/v1
kind: Job
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation
    helm.sh/hook-weight: "-4"
spec:
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
        - name: app-settings
          configMap:
            name: {{ include "xxx.fullname" . }}-migrations-{{ .Chart.AppVersion }}
        - name: app-secrets
          secret:
            secretName: {{ include "xxx.fullname" . }}-migrations-{{ .Chart.AppVersion }}

kubectl -n argocd logs argocd-server-768f46f469-j98h6 | grep xxx-migrations - No matching logs kubectl -n argocd logs argocd-repo-server-57bdbf899c-9lxhr | grep xxx-migrations - No matching logs kubectl -n argocd logs argocd-repo-server-57bdbf899c-7xvs7 | grep xxx-migrations - No matching logs kubectl -n argocd logs argocd-server-768f46f469-tqp8p | grep xxx-migrations - No matching logs

[testadmin@server0 ~]$ kubectl -n argocd logs argocd-application-controller-0 | grep orchestrator-migrations time=“2021-08-02T02:16:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:16:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:19:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:19:26Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:22:17Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:22:17Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:22:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:25:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:25:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:28:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:28:26Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:31:25Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx time=“2021-08-02T02:31:26Z” level=info msg=“Resuming in-progress operation. phase: Running, message: waiting for completion of hook /Secret/xxx-migrations-0.0.19-private4.1784494” application=xxx

Environment:

  • 3 Node RKE2 Cluster
  • OS: RHEL 8.4
  • K8’s setup on Azure VM’s

ArgoCD Version: 2.0.1

Please let me know in case of any other info required

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 50
  • Comments: 121 (27 by maintainers)

Commits related to this issue

Most upvoted comments

For folks waiting for this: we expect the fix to ship with 3.0.

I just figured out what was causing Argo to freeze on the hook. In my case the specific hook had ttlSecondsAfterFinished: 0 defined in the spec. Through Kustomize I removed this field:

patches:
  - target:
      name: pre-hook
      kind: Job
    path: patches/hook.yaml
# patches/hook.yaml
- path: "/spec/ttlSecondsAfterFinished"
  op: remove

Afterwards the chart finally went through! It’s still a bug that should be addressed, I’m just sharing this for others to work around it.

I have the same problem in version 2.2.0.

Is there a clear solution?

I’m facing sync issue from ingress-nginx 1.12.3 controlled by ArgoCD v2.13.4. The ingress-nginx-admission-create job remains in “Pending deletion” status indefinitely even after completion, and the ingress-nginx application also waits indefinitely in “Syncing” status.

ingress-nginx sync status in Argo UI:

Image
waiting for completion of hook batch/Job/ingress-nginx-admission-create

Issue still exists in argocd:v2.10.1 image

@younsl see https://github.com/kubernetes/ingress-nginx/pull/13483.

On my side, issues with nginx-ingress charts v4.11.7 & v4.12.2, where https://github.com/kubernetes/ingress-nginx/pull/13408 was added.

Rollback to v4.11.6 resolves temporarily my issue.

Using ArgoCD v2.14.11+8283115

I figured it out - envoy gateway allows setting this parameter directly on helm values - no need for kustomize:

      certgen:
        job:
          ttlSecondsAfterFinished: ~

Or set it to 60 and it doesn’t bug ArgoCD too.

still an issue. hoping for a fix.

Hello Argo community 😃

I am fairly familiar with ArgoCD codebase and API, and I’d happily try to repay you for building such an awesome project by trying to have a stab at this issue, if there are no objections?

@jkleinlercher Every now and then this thread makes me laugh out load, Thanks for this time ♥

It’s funny, because (this bug); it’s sad and or disappointing… 😞

https://github.com/argoproj/argo-cd/issues/6880#issuecomment-1996077818

Running into this issue on v2.10.1+a79e0ea as well

I just figured out what was causing Argo to freeze on the hook. In my case the specific hook had ttlSecondsAfterFinished: 0 defined in the spec. Through Kustomize I removed this field:

patches:
  - target:
      name: pre-hook
      kind: Job
    path: patches/hook.yaml
# patches/hook.yaml
- path: "/spec/ttlSecondsAfterFinished"
  op: remove

Afterwards the chart finally went through! It’s still a bug that should be addressed, I’m just sharing this for others to work around it.

@boedy You’re a Saint. I’ve been staring at envoyproxy/gateway for two weeks.

I’m experiencing the same problem with latest argocd chart version “7.8.7”

Hey @jkleinlercher,

@crenshaw-dev had good comments about generalizing the approach (also thanks for the finalizer idea ❤️ ), so I am in the process of refactoring it.

After that, if you’d like a preview, you can build it from my fork.

Still happens in v2.10.2+fcf5d8c

I’m seeing this issue with v2.6.1+3f143c9

To add some information here, we are running into the same issue (“waiting for completion of hook” when the hook has already completed), and it happens when we are attempting to sync to a revision that is not the targetRevision for the app. When we sync an app with hooks to the same revision as the targetRevision, we do not run into this.

expand for argo version
{
    "Version": "v2.3.4+ac8b7df",
    "BuildDate": "2022-05-18T11:41:37Z",
    "GitCommit": "ac8b7df9467ffcc0920b826c62c4b603a7bfed24",
    "GitTreeState": "clean",
    "GoVersion": "go1.17.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KsonnetVersion": "v0.13.1",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.0+gd141386",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

We are running 2 application-controller replicas in HA setup as per https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/ . I have verified we do not have a leftover instance of argo before it used stateful-sets.

I can confirm the error was fixed on 2.0.3. We recently upgraded to 2.3.3 and we are experiencing the error again.

@rverchere 👋 I can confirm that upgrading ArgoCD to v3.0.6 fixes the issue on nginx-ingress v4.11.7.

Since ArgoCD helm chart now has also a webhook for redis we get this problem also with ArgoCD managing ArgoCD. It stays OutOfSync with this

waiting for completion of hook batch/Job/argocd-redis-secret-init

        - group: batch
          hookPhase: Running
          hookType: PreSync
          kind: Job
          message: job.batch/sx-argocd-redis-secret-init created
          name: sx-argocd-redis-secret-init
          namespace: argocd
          syncPhase: PreSync
          version: v1

that is really a problem …

FYI - Running into the same issue with longhorn on ArgoCD v2.10.1+a79e0ea.
image

edit: found this relevant to the longhorn chart. it seems they added a value that can be set to false in order to work with argocd

The same with envoy-gateway

I’m using v2.11.0+f287dab and hit the same problem with whatever version of kube-prometheus-stack from 45.0.0 on. Last testing was with the most recent one, 58.0.0, and still facing the issue. Sadly, no workaround worked for me yet

Same issue in v1.3.3.7 and also in version v6.9… This issue was opened on Aug 2, 2021, we are now at 2023, please bump this comment via emoji so I can see it in my inbox in 2042

in all seriousness, still happens at version 2.7.7

We’re seeing a similar issue on the syncfailed hook which means we can’t actually terminate the sync action.

The job doesn’t exist in the target namespace, and we’ve tried to trick argo by creating a job with the same name, namespace, and annotations as we’d expect to see with a simple echo "done’ action but nothing is helping.

image

ArgoCD Version;

{“Version”:“v2.3.4+ac8b7df”,“BuildDate”:“2022-05-18T11:41:37Z”,“GitCommit”:“ac8b7df9467ffcc0920b826c62c4b603a7bfed24”,“GitTreeState”:“clean”,“GoVersion”:“go1.17.10”,“Compiler”:“gc”,“Platform”:“linux/amd64”,“KsonnetVersion”:“v0.13.1”,“KustomizeVersion”:“v4.4.1 2021-11-11T23:36:27Z”,“HelmVersion”:“v3.8.0+gd141386”,“KubectlVersion”:“v0.23.1”,“JsonnetVersion”:“v0.18.0”}

We started experiencing this issue after upgrading to 2.3.3. Before that we were on 2.2.3. I am not 100% sure but I do not recall we had any issue with 2.2.3.

@alexmt - We are using the below version of ArgoCD and seeing the same issue with Contour helm. Application is waiting for PreSync Job to complete whereas on a cluster I can see the job is completed.

{ “Version”: “v2.1.3+d855831”, “BuildDate”: “2021-09-29T21:51:21Z”, “GitCommit”: “d855831540e51d8a90b1006d2eb9f49ab1b088af”, “GitTreeState”: “clean”, “GoVersion”: “go1.16.5”, “Compiler”: “gc”, “Platform”: “linux/amd64”, “KsonnetVersion”: “v0.13.1”, “KustomizeVersion”: “v4.2.0 2021-06-30T22:49:26Z”, “HelmVersion”: “v3.6.0+g7f2df64”, “KubectlVersion”: “v0.21.0”, “JsonnetVersion”: “v0.17.0” }

I find it working only with manual intervention. First terminate sync then click on ConfigMap (there is only one) and sync only that ConfigMap. It will sync and everything else will immediately start syncing. Tho, I’m not sure why it works and would it help to put ConfigMap in some sync stage/wave to sync as early as possible

For me this is not solved in argocd 3.0.6 @dejanzele ingress-nginx helm 4.12.3 and argocd 3.0.6

waiting for completion of hook batch/Job/ingress-nginx-admission-create

same think with “batch/Job/ingress-nginx-admission-create” and “batch/Job/ingress-nginx-admission-path” but for nginx-controller v4.11.7

Image

Ver: v3.0.3+a14b012

@younsl as far as I understand, the solution is to upgrade to ArgoCD 3.

still having this issue in 3.0

@David2011Hernandez hook objects now have a finalizer which gets removed when the hook phase terminates.

This is the fix: https://github.com/argoproj/gitops-engine/pull/646

It’s too invasive to cherry-pick, so it has to wait for 3.0.

Hello awesome argo community,

I played around with this bug and implemented a proof-of-concept PR which proposes a fix for the stuck hook issue.

I kindly ask for feedback from the community and maintainers, as I would very much like to upstream it.

Issue - https://github.com/argoproj/argo-cd/issues/21055 PR - https://github.com/argoproj/gitops-engine/pull/646

Would overriding ttlSecondsAfterFinished or Argo side to be > 0 if it’s set to 0 solve the problem?

No, even if ttlSecondsAfterFinished is set to a value > 0, the Job remains in an incomplete state in Argo CD.

Had the same issue although for me the service account was not being deleted with the same hooks. My job ttlSecondsAfterFinished was > 0. Unsure if this is the actual root issue but I added this annotations to my job/sa argocd.argoproj.io/sync-options: PruneLast=false My reasoning being that the sync wave for pruning was going to be Last under the default. The chart needs the deletion to happen before any resources are upgraded. I have not noticed the issue on installation of charts. Per the argocd docs pruneLast would occur after the other resources have been deployed and become healthy so my assumption is argo holds on to the resource causing the sync to be delayed indefinitely. Been testing for a week without issue

Because I merged the PR that someone else wrote which has magic words on it so GitHub closed this issue.

Reporting the same problem on a k3s cluster trying to install the kube-prometheus-stack Helm chart latest version

image

Looking at what appears to be the same issue on ArgoCD v2.6.7. I’ve killed all controller pods, server and repo server at the same time, hoping that ArgoCD would start behaving. But to no avail. I believe the reason for it to start behaving like this is in the first place was a ImagePullBackOff on the Job image.

We had to completely exclude all jobs from argo cd via resource exclusion global config: https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#resource-exclusioninclusion

And we migrated all jobs on repo to cronjobs with suspend: true BUT fair warninig, Due to a k8s bug sometimes cronjobs may be triggered when changing spec, INCLUDING changing suspend: false to suspend: true - yes it’s stupid like that…

I think it’s this one, but there are others as well… https://github.com/kubernetes/kubernetes/issues/63371

Same here on 2.6.4

Observing the same issue on v2.6.2. Post-Sync hook never completes even though the corresponding pod exited successfully.

We also had this issue and it was resolved once we set ARGOCD_CONTROLLER_REPLICAS.

Instructions here: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-application-controller

If the controller is managing too many clusters and uses too much memory then you can shard clusters across multiple controller replicas. To enable sharding increase the number of replicas in argocd-application-controller StatefulSet and repeat number of replicas in ARGOCD_CONTROLLER_REPLICAS environment variable. The strategic merge patch below demonstrates changes required to configure two controller replicas.

Hello. I am still seeing this in v2.2.4. PreSync hook is scheduled, Job starts, runs to completion, Argo sits there spinning “Progressing” until terminated. To work around it, we are terminating the op and using ‘sync --strategy=apply’ (disabling the hook) and running our job out of band.

Kube events during the sync confirm the job success. I no longer see the job/pod (per those events) if I check the namespace directly.

LAST SEEN TYPE REASON OBJECT MESSAGE 22m Normal Scheduled pod/dcs-artifact-promoter0ba458b-presync-1645132298-hqm9l Successfully assigned dcs-prodemea-ns/dcs-artifact-promoter0ba458b-presync-1645132298-hqm9l to gke-service-nap-e2-standard-8-1oj503q-5bf9adda-f9t6 22m Normal Pulling pod/dcs-artifact-promoter0ba458b-presync-1645132298-hqm9l Pulling image "gcr.io/dremio-1093/accept-release:v3" 21m Normal Pulled pod/dcs-artifact-promoter0ba458b-presync-1645132298-hqm9l Successfully pulled image "gcr.io/dremio-1093/accept-release:v3" in 48.040095979s 21m Normal Created pod/dcs-artifact-promoter0ba458b-presync-1645132298-hqm9l Created container dcs-artifact 21m Normal Started pod/dcs-artifact-promoter0ba458b-presync-1645132298-hqm9l Started container dcs-artifact 22m Normal SuccessfulCreate job/dcs-artifact-promoter0ba458b-presync-1645132298 Created pod: dcs-artifact-promoter0ba458b-presync-1645132298-hqm9l

image

Let me know if I can provide any diagnostics to help.

I suspect this is fixed by https://github.com/argoproj/argo-cd/pull/6294 . The fix is available in https://github.com/argoproj/argo-cd/releases/tag/v2.0.3 . Can you try upgrading please?