argo-workflows: 3.5.5+: Workflow stuck Running when failed resource template

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

When running a workflow that includes a Kubernetes Resource step, if that step fails the workflow is stuck in Running phase even while using continueOn: error and continueOn: failed.

This issue started occurring when we upgraded from v3.4.8 to v3.5.8. I have confirmed the issue is not preset on v3.4.8. Edit: on further investigation the issue seems to have started since v3.5.6 (v3.5.5 is working fine)

I have reproduced the issue using the latest image for the controller.

Example workflow using continueOn: failed: image

Metadata present on the workflow stuck in the Running phase

The following labels are present on the Workflow: workflows.argoproj.io/completed: 'false' workflows.argoproj.io/phase: Running

The following label is present on the WorkflowTaskResult: workflows.argoproj.io/report-outputs-completed: 'false'

The conditions have only one entry: PodRunning: false

progress: 1/2
taskResultsCompletionStatus:
    sligo-bug-2486207007: false
    sligo-bug-2605594679: true

All pods created for each step are removed from the namespace

The full yaml for the node running the kubernetes resource step is:

    sligo-bug-2486207007:
      boundaryID: sligo-bug
      children:
        - sligo-bug-2988043323
      displayName: check-k8-resource
      finishedAt: '2024-07-04T09:49:44Z'
      hostNodeName: aks-default-65748190-vmss000008
      id: sligo-bug-2486207007
      message: Error (exit code 1)
      name: sligo-bug[0].check-k8-resource
      outputs:
        exitCode: '1'
      phase: Failed
      progress: 0/1
      resourcesDuration:
        cpu: 0
        memory: 2
      startedAt: '2024-07-04T09:49:39Z'
      templateName: check-mongo-ready
      templateScope: local/sligo-bug
      type: Pod

Comparison using container step instead of kubernetes resource step

If we replace the template for the kubernetes resource step check-mongo-ready with a container step such as intentional-fail taken from examples the workflow finishes fine (it does not get stuck in Running phase):

  - name: intentional-fail
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo intentional failure; exit 1"]

Version

v3.5.8

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

To avoid RBAC issues with this workflow, you need these permissions:

  - verbs:
      - get
    apiGroups:
      - apps
    resources:
      - statefulsets
metadata:
  name: sligo-bug
  namespace: argo-wf
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
      - - name: check-k8-resource
          template: check-mongo-ready
          arguments: {}
          continueOn:
            error: true
            failed: true
      - - name: B
          template: finish
          arguments: {}
  - name: check-mongo-ready
    resource:
      action: get
      manifest: |
        apiVersion: apps/v1
        kind: StatefulSet
        metadata:
          name: myapp-mongodb
      # we expect this to fail because the resource does not exist
      successCondition: status.readyReplicas > 0
      failureCondition: status.readyReplicas < 1
  - name: finish
    container:
      name: ''
      image: docker/whalesay:latest
      command:
        - cowsay
      args:
        - hello world; exit 0
  ttlStrategy:
    secondsAfterCompletion: 300
  podGC:
    strategy: OnPodCompletion

Logs from the workflow controller

{"Phase":"","ResourceVersion":"1708512","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:49:39.515Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":0,"time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"info","msg":"Updated phase  -\u003e Running","namespace":"argo-wf","time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo-wf","time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"argo-wf","time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"info","msg":"Steps node sligo-bug initialized Running","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"info","msg":"StepGroup node sligo-bug-3055300894 initialized Running","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"info","msg":"Pod node sligo-bug-2486207007 initialized Pending","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"info","msg":"Created pod: sligo-bug[0].check-k8-resource (sligo-bug-check-mongo-ready-2486207007)","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow step group node sligo-bug-3055300894 not yet completed","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708515","time":"2024-07-04T09:49:39.601Z","workflow":"sligo-bug"}
{"Phase":"Running","ResourceVersion":"1708515","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:49:49.590Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":1,"time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2486207007","time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"displayName":"check-k8-resource","level":"info","msg":"Pod failed: Error (exit code 1)","namespace":"argo-wf","pod":"sligo-bug-check-mongo-ready-2486207007","templateName":"check-mongo-ready","time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"level":"info","msg":"node changed","namespace":"argo-wf","new.message":"Error (exit code 1)","new.phase":"Failed","new.progress":"0/1","nodeID":"sligo-bug-2486207007","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"level":"info","msg":"Step group node sligo-bug-3055300894 successful","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-3055300894 phase Running -\u003e Succeeded","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-3055300894 finished: 2024-07-04 09:49:49.592112732 +0000 UTC","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"StepGroup node sligo-bug-2988043323 initialized Running","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"SG Outbound nodes of sligo-bug-2486207007 are [sligo-bug-2486207007]","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"Pod node sligo-bug-2605594679 initialized Pending","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"Created pod: sligo-bug[1].B (sligo-bug-finish-2605594679)","namespace":"argo-wf","time":"2024-07-04T09:49:49.614Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow step group node sligo-bug-2988043323 not yet completed","namespace":"argo-wf","time":"2024-07-04T09:49:49.614Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:49:49.615Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:49:49.615Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708590","time":"2024-07-04T09:49:49.629Z","workflow":"sligo-bug"}
{"action":"deletePod","key":"argo-wf/sligo-bug-check-mongo-ready-2486207007/deletePod","level":"info","msg":"cleaning up pod","time":"2024-07-04T09:49:54.631Z"}
{"Phase":"Running","ResourceVersion":"1708590","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:49:59.616Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":2,"time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2486207007","time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2605594679","time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"node changed","namespace":"argo-wf","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"sligo-bug-2605594679","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"SG Outbound nodes of sligo-bug-2486207007 are [sligo-bug-2486207007]","namespace":"argo-wf","time":"2024-07-04T09:49:59.619Z","workflow":"sligo-bug"}
{"level":"info","msg":"Step group node sligo-bug-2988043323 successful","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-2988043323 phase Running -\u003e Succeeded","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-2988043323 finished: 2024-07-04 09:49:59.620324699 +0000 UTC","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"Outbound nodes of sligo-bug-2605594679 is [sligo-bug-2605594679]","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"Outbound nodes of sligo-bug is [sligo-bug-2605594679]","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug phase Running -\u003e Succeeded","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug finished: 2024-07-04 09:49:59.620749704 +0000 UTC","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708676","time":"2024-07-04T09:49:59.636Z","workflow":"sligo-bug"}
{"action":"deletePod","key":"argo-wf/sligo-bug-finish-2605594679/deletePod","level":"info","msg":"cleaning up pod","time":"2024-07-04T09:50:04.637Z"}
{"Phase":"Running","ResourceVersion":"1708676","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:50:14.667Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":2,"time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2486207007","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2605594679","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708676","time":"2024-07-04T09:50:14.685Z","workflow":"sligo-bug"}

Logs from in your workflow’s wait container

Edit: There is no wait container for Resource type pods. These logs are for the step that uses a container

{"level":"info","msg":"Starting Workflow Executor","time":"2024-07-04T09:55:13.015Z","version":"v3.5.8"}
{"Duration":1000000000,"Factor":1.6,"Jitter":0.5,"Steps":5,"level":"info","msg":"Using executor retry strategy","time":"2024-07-04T09:55:13.018Z"}
{"deadline":"2024-07-05T01:55:00Z","includeScriptOutput":false,"level":"info","msg":"Executor initialized","namespace":"argo-wf","podName":"sligo-bug-vj8v4-finish-1000479478","templateName":"finish","time":"2024-07-04T09:55:13.018Z","version":"\u0026Version{Version:v3.5.8,BuildDate:2024-06-18T03:43:17Z,GitCommit:3bb637c0261f8c08d4346175bb8b1024719a1f11,GitTag:v3.5.8,GitTreeState:clean,GoVersion:go1.21.10,Compiler:gc,Platform:linux/amd64,}"}
W0704 09:55:13.045324       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
{"level":"info","msg":"Starting deadline monitor","time":"2024-07-04T09:55:13.046Z"}
{"error":null,"level":"info","msg":"Main container completed","time":"2024-07-04T09:55:16.047Z"}
{"level":"info","msg":"No Script output reference in workflow. Capturing script output ignored","time":"2024-07-04T09:55:16.047Z"}
{"level":"info","msg":"No output parameters","time":"2024-07-04T09:55:16.047Z"}
{"level":"info","msg":"No output artifacts","time":"2024-07-04T09:55:16.047Z"}
W0704 09:55:16.061106       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
W0704 09:55:16.068093       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
{"level":"info","msg":"Alloc=8106 TotalAlloc=13731 Sys=19557 NumGC=4 Goroutines=8","time":"2024-07-04T09:55:16.068Z"}
W0704 09:55:16.074405       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 18 (5 by maintainers)

Most upvoted comments

Oh dear, I see the problem. We’re not using client-go at all, but straight up calling the kubectl code as a library, which, as a CLI, will of course have exits 😕

If we want to keep doing that, we’d have to do what their tests do and catch exits. But tbh, I don’t think we should run like that, I would think using client-go directly would make more sense. It would also shrink the size of the Executor binary to not include kubectl code. One caveat is idk if the flags field would work the same that way, which would be a breaking change if not 😕

@jswxstw yes, I already mentioned FinalizeOutputs not being run in my very first comment here. But, per that comment, it seemed like it should still run unless there’s a panic in HandleError? Unless I’m misunderstanding how Go defer works

In any case, FinalizeOutputs must still run if there’s an error. It shouldn’t be stuck ever (unless there is a bug).

I’ve checked again, I might have been mistaken earlier looking at incorrect WorkflowTaskResult.

In 3.5.5 the WorkflowTaskResult does still have workflows.argoproj.io/report-outputs-completed: 'false'

@BarryOwensDigital ok, that makes a lot more sense. That means the code from #12544 has a bug in it in the error case.

Also would you mind adding syntax highlighting to your code blocks? i.e. ```yaml. I’ve been editing all your comments to add it 😅

So the problem is that it’s not being marked complete for some reason in 3.5.6+

I’ve checked again, I might have been mistaken earlier looking at incorrect WorkflowTaskResult.

In 3.5.5 the WorkflowTaskResult does still have workflows.argoproj.io/report-outputs-completed: 'false'

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTaskResult
metadata:
  generation: 1
  labels:
    workflows.argoproj.io/report-outputs-completed: 'false'
    workflows.argoproj.io/workflow: sligo-bug

Does this part happen in 3.5.5 too?

~Seems like in 3.5.5 we get workflows.argoproj.io/report-outputs-completed: 'true'. The WorkflowTaskResult is removed quickly on this version~

Edit: Yes we get the following in the WorkflowTaskResult for the resource template step: workflows.argoproj.io/report-outputs-completed: 'false'

Logs from the main container (v3.5.8). We expect the “myapp-mongodb not found” which is shown here:

time="2024-07-04T16:27:32 UTC" level=info msg="Starting Workflow Executor" version=v3.5.8
time="2024-07-04T16:27:32 UTC" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-07-04T16:27:32 UTC" level=info msg="Executor initialized" deadline="2024-07-05 08:27:28 +0000 UTC" includeScriptOutput=false namespace=jenkins podName=sligo-bug2-check-mongo-ready-2523367045 templateName=check-mongo-ready version="&Version{Version:v3.5.8,BuildDate:2024-06-18T03:43:17Z,GitCommit:3bb637c0261f8c08d4346175bb8b1024719a1f11,GitTag:v3.5.8,GitTreeState:clean,GoVersion:go1.21.10,Compiler:gc,Platform:linux/amd64,}"
W0704 16:27:32.689976      14 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
time="2024-07-04T16:27:32 UTC" level=info msg="Loading manifest to /tmp/manifest.yaml"
time="2024-07-04T16:27:32 UTC" level=info msg="kubectl get -f /tmp/manifest.yaml -o json"
Warning: Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
Error from server (NotFound): statefulsets.apps "myapp-mongodb" not found
{"argo":true,"error":null,"level":"info","msg":"sub-process exited","time":"2024-07-04T16:27:33.620Z"}
Error: exit status 1

Also agree this might be related to https://github.com/argoproj/argo-workflows/pull/12544, when I enable the debug logs we get this log on the controller: ...msg="taskresults of workflow are incomplete or still have daemon nodes, so can't mark workflow completed" fromPhase=Running namespace=jenkins toPhase=Succeeded ...