argo-cd: Argo hook not running on auto sync only on manual sync

Checklist:

• I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I’ve included steps to reproduce the bug.
• I’ve pasted the output of argocd version.

Describe the bug

We have some jobs that use argo hooks to kick off a job but the hook only gets triggered when we run the app sync from the cli. The app itself is auto-syncing and will show as synced in the UI (with the commit hash of the latest change) but the hook doesn’t get triggered so the job never actually runs unless we run argocd app sync migrations-mssql-dev in the terminal. Not sure if this is related but I have noticed that I can’t manually trigger the sync using the UI because it doesn’t seem to recognize any resources (this isn’t as important cause i can use the cli, but I don’t know if it’s indicative of some other issue).

The hooks we are using are:

  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

but we also tried Sync, and PostSync, and changing the deletion policy on HookSucceeded. We also tried using the generated name instead of metadata.name.

To Reproduce

here is the application (we have a couple but they are pretty much the same just different images):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: migrations-mssql-dev
  namespace: argocd
spec:
  destination:
    namespace: dev-engineering
    server: https://xxxx.xxx.us-xxx.eks.amazonaws.com/
  project: default
  source:
    path: migrations-mssql
    repoURL: https://github.com/xxx/xxx.git
    targetRevision: main
    helm:
      valueFiles:
        - values.yaml
      parameters:
        - name: imageTag
          value: staged-552138452accb9a66b6f40cbd1c0e8800f62a5d1-1656015181
  syncPolicy:
    automated: {}

then under path migrations-mssql/Chart.yaml

apiVersion: v1
kind: application
description: Mssql migration Helm chart for Kubernetes
name: migrations-mssql
version: 1.1.1

and a job migrations-msslq/templates/job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}
  labels:
    app: {{ template "migrations-mssql.name" . }}
    chart: {{ template "migrations-mssql.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        {{- include "migrations-mssql.selectorLabels" . | nindent 8 }}
    spec:
      restartPolicy: Never
      containers:
        - name: migrations-mssql
          image: {{ .Values.imageRegistry }}:{{ .Values.imageTag }}
          imagePullPolicy: Always
          command: ["/bin/sh","-c"]
          args: 
            - /mssql-scripting/migrate-db-all.sh $(mssqlServer) \
              $(mssqlPort) \
              $(mssqlUser) \
              $(mssqlPassword) \
              $(FlywayEnvironment)
          env:
            - name: mssqlServer
              value: {{ .Values.mssqlServer | quote }}
            - name: mssqlPort
              value: {{ .Values.mssqlPort | quote }}
            - name: FlywayEnvironment
              value: {{ .Release.Namespace }}
            - name: mssqlPassword
              valueFrom:
                secretKeyRef:
                  key: mssqlPassword
                  name: {{ .Values.envFrom.secretRef.name }}
            - name: mssqlUser
              valueFrom:
                secretKeyRef:
                  key: mssqlUser
                  name: {{ .Values.envFrom.secretRef.name }}

When the image tag on the application is updated at helm.parameters.value that triggers the auto-sync and the application gets marked as sync and points to the newest image tag but the job never runs unless we manually sync in the cli.

Expected behavior

Pre-sync and sync hooks should run on auto-sync and not just when manually triggering a sync.

Screenshots

job last ran 5 days ago (when I ran the app sync command manually in the cli)

but the current sync status says the last sync was an hour ago (when the image was updated in github) but the job didn’t run:

Version

Paste the output from `argocd version` here.
argocd: v2.3.2+ecc2af9.dirty
  BuildDate: 2022-03-23T05:19:12Z
  GitCommit: ecc2af9dcaa12975e654cde8cbbeaffbb315f75c
  GitTreeState: dirty
  GoVersion: go1.18
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.3.4+ac8b7df

Logs

Logs show that the sync was successful but the job was never triggered

time="2022-06-29T19:29:09Z" level=info msg="Update successful" application=migrations-mssql-prod
time="2022-06-29T19:29:09Z" level=info msg="Reconciliation completed" application=migrations-mssql-prod dedup_ms=0 dest-name= dest-namespace=prod-engineering dest-server="https://xxx.gr7.us-east-1.eks.amazonaws.com" diff_ms=3 fields.level=2 git_ms=66 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=305