argo-workflows: UI does not show all running workflows (it used to do so) / UI and argo list differ (missing running workflows)

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

When I am submitting a workflow in the UI it is shown in the list of running workflows. When refreshing this page all workflows should be listed.

We just upgraded to argo 3.4.0 and not all running workflows are shown in the UI. When you wait long enough eventually workflows pop up. In our day to day work we noticed that workflows seem to be added to the UI once a transition happens, i.e. when a new step is entered in the workflow.

In short: The output of argo list and the workflows shown in the UI are not the same. Some running flows are missing. They are being added once they transition from one node to another while the UI is shown.

Version

3.4.0

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: zargo-debug-child
  namespace: argo
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: node1
            template: work
        - - name: node2
            template: work
    - name: work
      script:
        image: bash:5.2.0-alpine3.15
        command:
          - bash
        source: |
          echo "Doing some work for 60 seconds"
          sleep 60

Logs from the workflow controller

time="2022-09-27T15:57:49.577Z" level=info msg="Processing workflow" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.591Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.593Z" level=info msg="Steps node zargo-debug-child-vcmgq initialized Running" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.593Z" level=info msg="StepGroup node zargo-debug-child-vcmgq-3865638504 initialized Running" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.593Z" level=info msg="Pod node zargo-debug-child-vcmgq-3922648685 initialized Pending" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.628Z" level=info msg="Created pod: zargo-debug-child-vcmgq[0].node1 (zargo-debug-child-vcmgq-work-3922648685)" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.628Z" level=info msg="Workflow step group node zargo-debug-child-vcmgq-3865638504 not yet completed" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.628Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.628Z" level=info msg=reconcileAgentPod namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:49.639Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=10435392 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:59.629Z" level=info msg="Processing workflow" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:59.629Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:59.629Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=zargo-debug-child-vcmgq-3922648685 old.message= old.phase=Pending old.progress=0/1 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:59.629Z" level=info msg="Workflow step group node zargo-debug-child-vcmgq-3865638504 not yet completed" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:59.629Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:59.629Z" level=info msg=reconcileAgentPod namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:57:59.726Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=10435445 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.782Z" level=info msg="Processing workflow" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.782Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.782Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.782Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=zargo-debug-child-vcmgq-3922648685 old.message= old.phase=Running old.progress=0/1 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.782Z" level=info msg="Step group node zargo-debug-child-vcmgq-3865638504 successful" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.782Z" level=info msg="node zargo-debug-child-vcmgq-3865638504 phase Running -> Succeeded" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.782Z" level=info msg="node zargo-debug-child-vcmgq-3865638504 finished: 2022-09-27 15:59:01.78294768 +0000 UTC" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.787Z" level=info msg="StepGroup node zargo-debug-child-vcmgq-2858834269 initialized Running" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.787Z" level=info msg="SG Outbound nodes of zargo-debug-child-vcmgq-3922648685 are [zargo-debug-child-vcmgq-3922648685]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.788Z" level=info msg="Pod node zargo-debug-child-vcmgq-3116902745 initialized Pending" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.818Z" level=info msg="Created pod: zargo-debug-child-vcmgq[1].node2 (zargo-debug-child-vcmgq-work-3116902745)" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.818Z" level=info msg="Workflow step group node zargo-debug-child-vcmgq-2858834269 not yet completed" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.818Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.818Z" level=info msg=reconcileAgentPod namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.828Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=10435675 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:01.834Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/zargo-debug-child-vcmgq-work-3922648685/labelPodCompleted
time="2022-09-27T15:59:11.820Z" level=info msg="Processing workflow" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:11.820Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:11.820Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=zargo-debug-child-vcmgq-3116902745 old.message= old.phase=Pending old.progress=0/1 workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:11.820Z" level=info msg="SG Outbound nodes of zargo-debug-child-vcmgq-3922648685 are [zargo-debug-child-vcmgq-3922648685]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:11.821Z" level=info msg="Workflow step group node zargo-debug-child-vcmgq-2858834269 not yet completed" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:11.821Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:11.821Z" level=info msg=reconcileAgentPod namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T15:59:11.834Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=10435730 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="Processing workflow" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=zargo-debug-child-vcmgq-3116902745 old.message= old.phase=Running old.progress=0/1 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="SG Outbound nodes of zargo-debug-child-vcmgq-3922648685 are [zargo-debug-child-vcmgq-3922648685]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="Step group node zargo-debug-child-vcmgq-2858834269 successful" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="node zargo-debug-child-vcmgq-2858834269 phase Running -> Succeeded" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="node zargo-debug-child-vcmgq-2858834269 finished: 2022-09-27 16:00:13.417764045 +0000 UTC" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="Outbound nodes of zargo-debug-child-vcmgq-3116902745 is [zargo-debug-child-vcmgq-3116902745]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="Outbound nodes of zargo-debug-child-vcmgq is [zargo-debug-child-vcmgq-3116902745]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="node zargo-debug-child-vcmgq phase Running -> Succeeded" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="node zargo-debug-child-vcmgq finished: 2022-09-27 16:00:13.41780197 +0000 UTC" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="Checking daemoned children of zargo-debug-child-vcmgq" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg=reconcileAgentPod namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.417Z" level=info msg="Running OnExit handler: b72ca92baf93-exit-handler" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.419Z" level=info msg="Steps node zargo-debug-child-vcmgq-4002967377 initialized Running" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.419Z" level=info msg="StepGroup node zargo-debug-child-vcmgq-3620866933 initialized Running" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.419Z" level=info msg="Pod node zargo-debug-child-vcmgq-1020052388 initialized Pending" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.438Z" level=info msg="Created pod: zargo-debug-child-vcmgq.onExit[0].success (zargo-debug-child-vcmgq-b72ca92baf93-send-slack-1020052388)" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.438Z" level=info msg="Skipping zargo-debug-child-vcmgq.onExit[0].failure: when 'Succeeded != Succeeded' evaluated false" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.438Z" level=info msg="Skipped node zargo-debug-child-vcmgq-1275290133 initialized Skipped (message: when 'Succeeded != Succeeded' evaluated false)" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.438Z" level=info msg="Workflow step group node zargo-debug-child-vcmgq-3620866933 not yet completed" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.448Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=10436003 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:13.455Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/zargo-debug-child-vcmgq-work-3116902745/labelPodCompleted
time="2022-09-27T16:00:23.440Z" level=info msg="Processing workflow" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.440Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.440Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.440Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=zargo-debug-child-vcmgq-1020052388 old.message= old.phase=Pending old.progress=0/1 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg=reconcileAgentPod namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Running OnExit handler: b72ca92baf93-exit-handler" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Step group node zargo-debug-child-vcmgq-3620866933 successful" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="node zargo-debug-child-vcmgq-3620866933 phase Running -> Succeeded" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="node zargo-debug-child-vcmgq-3620866933 finished: 2022-09-27 16:00:23.441636451 +0000 UTC" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Outbound nodes of zargo-debug-child-vcmgq-1020052388 is [zargo-debug-child-vcmgq-1020052388]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Outbound nodes of zargo-debug-child-vcmgq-1275290133 is [zargo-debug-child-vcmgq-1275290133]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Outbound nodes of zargo-debug-child-vcmgq-4002967377 is [zargo-debug-child-vcmgq-1020052388 zargo-debug-child-vcmgq-1275290133]" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="node zargo-debug-child-vcmgq-4002967377 phase Running -> Succeeded" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="node zargo-debug-child-vcmgq-4002967377 finished: 2022-09-27 16:00:23.44170012 +0000 UTC" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Checking daemoned children of zargo-debug-child-vcmgq-4002967377" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Marking workflow completed" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.441Z" level=info msg="Checking daemoned children of " namespace=argo workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.447Z" level=info msg="cleaning up pod" action=deletePod key=argo/zargo-debug-child-vcmgq-1340600742-agent/deletePod
time="2022-09-27T16:00:23.462Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=10436076 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.646Z" level=info msg="archiving workflow" namespace=argo uid=cdd6c373-96d9-4c98-b894-3ef1f02b18a2 workflow=zargo-debug-child-vcmgq
time="2022-09-27T16:00:23.650Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/zargo-debug-child-vcmgq-b72ca92baf93-send-slack-1020052388/labelPodCompleted
time="2022-09-27T16:00:23.669Z" level=info msg="Queueing Succeeded workflow argo/zargo-debug-child-vcmgq for delete in 18h0m0s due to TTL"

Logs from in your workflow’s wait container

time="2022-09-27T15:58:52.538Z" level=info msg="Creating minio client using static credentials" endpoint=s3.amazonaws.com
time="2022-09-27T15:58:52.544Z" level=info msg="Saving file to s3" bucket=incrmntal-argo-artifactory endpoint=s3.amazonaws.com key=incrmntal-prod-two/argo/zargo-debug-child-vcmgq/2022-09-27-zargo-debug-child-vcmgq-work-3922648685/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-09-27T15:58:52.587Z" level=info msg="Save artifact" artifactName=main-logs duration=49.392744ms error="<nil>" key=incrmntal-prod-two/argo/zargo-debug-child-vcmgq/2022-09-27-zargo-debug-child-vcmgq-work-3922648685/main.log
time="2022-09-27T15:58:52.587Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-09-27T15:58:52.587Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-09-27T15:58:52.599Z" level=info msg="Create workflowtaskresults 403"
time="2022-09-27T15:58:52.599Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2022-09-27T15:58:52.618Z" level=info msg="Patch pods 200"
time="2022-09-27T15:58:52.621Z" level=info msg="Deadline monitor stopped"
time="2022-09-27T15:58:52.621Z" level=info msg="Alloc=7002 TotalAlloc=16520 Sys=24018 NumGC=5 Goroutines=9"