argo-workflows: 3.5.5+: Workflow stuck Running when failed resource template
Pre-requisites
- I have double-checked my configuration
- I have tested with the
:latestimage tag (i.e.quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on:latest. If not, I have explained why, in detail, in my description below. - I have searched existing issues and could not find a match for this bug
- I’d like to contribute the fix myself (see contributing guide)
What happened/what did you expect to happen?
When running a workflow that includes a Kubernetes Resource step, if that step fails the workflow is stuck in Running phase even while using continueOn: error and continueOn: failed.
This issue started occurring when we upgraded from v3.4.8 to v3.5.8. I have confirmed the issue is not preset on v3.4.8. Edit: on further investigation the issue seems to have started since v3.5.6 (v3.5.5 is working fine)
I have reproduced the issue using the latest image for the controller.
Example workflow using continueOn: failed:
Metadata present on the workflow stuck in the Running phase
The following labels are present on the Workflow:
workflows.argoproj.io/completed: 'false'
workflows.argoproj.io/phase: Running
The following label is present on the WorkflowTaskResult:
workflows.argoproj.io/report-outputs-completed: 'false'
The conditions have only one entry: PodRunning: false
progress: 1/2
taskResultsCompletionStatus:
sligo-bug-2486207007: false
sligo-bug-2605594679: true
All pods created for each step are removed from the namespace
The full yaml for the node running the kubernetes resource step is:
sligo-bug-2486207007:
boundaryID: sligo-bug
children:
- sligo-bug-2988043323
displayName: check-k8-resource
finishedAt: '2024-07-04T09:49:44Z'
hostNodeName: aks-default-65748190-vmss000008
id: sligo-bug-2486207007
message: Error (exit code 1)
name: sligo-bug[0].check-k8-resource
outputs:
exitCode: '1'
phase: Failed
progress: 0/1
resourcesDuration:
cpu: 0
memory: 2
startedAt: '2024-07-04T09:49:39Z'
templateName: check-mongo-ready
templateScope: local/sligo-bug
type: Pod
Comparison using container step instead of kubernetes resource step
If we replace the template for the kubernetes resource step check-mongo-ready with a container step such as intentional-fail taken from examples the workflow finishes fine (it does not get stuck in Running phase):
- name: intentional-fail
container:
image: alpine:latest
command: [sh, -c]
args: ["echo intentional failure; exit 1"]
Version
v3.5.8
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.
To avoid RBAC issues with this workflow, you need these permissions:
- verbs:
- get
apiGroups:
- apps
resources:
- statefulsets
metadata:
name: sligo-bug
namespace: argo-wf
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: check-k8-resource
template: check-mongo-ready
arguments: {}
continueOn:
error: true
failed: true
- - name: B
template: finish
arguments: {}
- name: check-mongo-ready
resource:
action: get
manifest: |
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: myapp-mongodb
# we expect this to fail because the resource does not exist
successCondition: status.readyReplicas > 0
failureCondition: status.readyReplicas < 1
- name: finish
container:
name: ''
image: docker/whalesay:latest
command:
- cowsay
args:
- hello world; exit 0
ttlStrategy:
secondsAfterCompletion: 300
podGC:
strategy: OnPodCompletion
Logs from the workflow controller
{"Phase":"","ResourceVersion":"1708512","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:49:39.515Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":0,"time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"info","msg":"Updated phase -\u003e Running","namespace":"argo-wf","time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo-wf","time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"argo-wf","time":"2024-07-04T09:49:39.526Z","workflow":"sligo-bug"}
{"level":"info","msg":"Steps node sligo-bug initialized Running","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"info","msg":"StepGroup node sligo-bug-3055300894 initialized Running","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"info","msg":"Pod node sligo-bug-2486207007 initialized Pending","namespace":"argo-wf","time":"2024-07-04T09:49:39.527Z","workflow":"sligo-bug"}
{"level":"info","msg":"Created pod: sligo-bug[0].check-k8-resource (sligo-bug-check-mongo-ready-2486207007)","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow step group node sligo-bug-3055300894 not yet completed","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:49:39.589Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708515","time":"2024-07-04T09:49:39.601Z","workflow":"sligo-bug"}
{"Phase":"Running","ResourceVersion":"1708515","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:49:49.590Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":1,"time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2486207007","time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"displayName":"check-k8-resource","level":"info","msg":"Pod failed: Error (exit code 1)","namespace":"argo-wf","pod":"sligo-bug-check-mongo-ready-2486207007","templateName":"check-mongo-ready","time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"level":"info","msg":"node changed","namespace":"argo-wf","new.message":"Error (exit code 1)","new.phase":"Failed","new.progress":"0/1","nodeID":"sligo-bug-2486207007","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-07-04T09:49:49.591Z","workflow":"sligo-bug"}
{"level":"info","msg":"Step group node sligo-bug-3055300894 successful","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-3055300894 phase Running -\u003e Succeeded","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-3055300894 finished: 2024-07-04 09:49:49.592112732 +0000 UTC","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"StepGroup node sligo-bug-2988043323 initialized Running","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"SG Outbound nodes of sligo-bug-2486207007 are [sligo-bug-2486207007]","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"Pod node sligo-bug-2605594679 initialized Pending","namespace":"argo-wf","time":"2024-07-04T09:49:49.592Z","workflow":"sligo-bug"}
{"level":"info","msg":"Created pod: sligo-bug[1].B (sligo-bug-finish-2605594679)","namespace":"argo-wf","time":"2024-07-04T09:49:49.614Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow step group node sligo-bug-2988043323 not yet completed","namespace":"argo-wf","time":"2024-07-04T09:49:49.614Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:49:49.615Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:49:49.615Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708590","time":"2024-07-04T09:49:49.629Z","workflow":"sligo-bug"}
{"action":"deletePod","key":"argo-wf/sligo-bug-check-mongo-ready-2486207007/deletePod","level":"info","msg":"cleaning up pod","time":"2024-07-04T09:49:54.631Z"}
{"Phase":"Running","ResourceVersion":"1708590","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:49:59.616Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":2,"time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2486207007","time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2605594679","time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"node changed","namespace":"argo-wf","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"sligo-bug-2605594679","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-07-04T09:49:59.618Z","workflow":"sligo-bug"}
{"level":"info","msg":"SG Outbound nodes of sligo-bug-2486207007 are [sligo-bug-2486207007]","namespace":"argo-wf","time":"2024-07-04T09:49:59.619Z","workflow":"sligo-bug"}
{"level":"info","msg":"Step group node sligo-bug-2988043323 successful","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-2988043323 phase Running -\u003e Succeeded","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug-2988043323 finished: 2024-07-04 09:49:59.620324699 +0000 UTC","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"Outbound nodes of sligo-bug-2605594679 is [sligo-bug-2605594679]","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"Outbound nodes of sligo-bug is [sligo-bug-2605594679]","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug phase Running -\u003e Succeeded","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"node sligo-bug finished: 2024-07-04 09:49:59.620749704 +0000 UTC","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:49:59.620Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708676","time":"2024-07-04T09:49:59.636Z","workflow":"sligo-bug"}
{"action":"deletePod","key":"argo-wf/sligo-bug-finish-2605594679/deletePod","level":"info","msg":"cleaning up pod","time":"2024-07-04T09:50:04.637Z"}
{"Phase":"Running","ResourceVersion":"1708676","level":"info","msg":"Processing workflow","namespace":"argo-wf","time":"2024-07-04T09:50:14.667Z","workflow":"sligo-bug"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo-wf","numObjs":2,"time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2486207007","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"task-result changed","namespace":"argo-wf","nodeID":"sligo-bug-2605594679","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo-wf","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo-wf","time":"2024-07-04T09:50:14.668Z","workflow":"sligo-bug"}
{"level":"info","msg":"Workflow update successful","namespace":"argo-wf","phase":"Running","resourceVersion":"1708676","time":"2024-07-04T09:50:14.685Z","workflow":"sligo-bug"}
Logs from in your workflow’s wait container
Edit: There is no wait container for Resource type pods. These logs are for the step that uses a container
{"level":"info","msg":"Starting Workflow Executor","time":"2024-07-04T09:55:13.015Z","version":"v3.5.8"}
{"Duration":1000000000,"Factor":1.6,"Jitter":0.5,"Steps":5,"level":"info","msg":"Using executor retry strategy","time":"2024-07-04T09:55:13.018Z"}
{"deadline":"2024-07-05T01:55:00Z","includeScriptOutput":false,"level":"info","msg":"Executor initialized","namespace":"argo-wf","podName":"sligo-bug-vj8v4-finish-1000479478","templateName":"finish","time":"2024-07-04T09:55:13.018Z","version":"\u0026Version{Version:v3.5.8,BuildDate:2024-06-18T03:43:17Z,GitCommit:3bb637c0261f8c08d4346175bb8b1024719a1f11,GitTag:v3.5.8,GitTreeState:clean,GoVersion:go1.21.10,Compiler:gc,Platform:linux/amd64,}"}
W0704 09:55:13.045324 1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
{"level":"info","msg":"Starting deadline monitor","time":"2024-07-04T09:55:13.046Z"}
{"error":null,"level":"info","msg":"Main container completed","time":"2024-07-04T09:55:16.047Z"}
{"level":"info","msg":"No Script output reference in workflow. Capturing script output ignored","time":"2024-07-04T09:55:16.047Z"}
{"level":"info","msg":"No output parameters","time":"2024-07-04T09:55:16.047Z"}
{"level":"info","msg":"No output artifacts","time":"2024-07-04T09:55:16.047Z"}
W0704 09:55:16.061106 1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
W0704 09:55:16.068093 1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
{"level":"info","msg":"Alloc=8106 TotalAlloc=13731 Sys=19557 NumGC=4 Goroutines=8","time":"2024-07-04T09:55:16.068Z"}
W0704 09:55:16.074405 1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 18 (5 by maintainers)
Oh dear, I see the problem. We’re not using
client-goat all, but straight up calling thekubectlcode as a library, which, as a CLI, will of course have exits 😕If we want to keep doing that, we’d have to do what their tests do and catch exits. But tbh, I don’t think we should run like that, I would think using
client-godirectly would make more sense. It would also shrink the size of the Executor binary to not includekubectlcode. One caveat is idk if theflagsfield would work the same that way, which would be a breaking change if not 😕@jswxstw yes, I already mentioned
FinalizeOutputsnot being run in my very first comment here. But, per that comment, it seemed like it should still run unless there’s a panic inHandleError? Unless I’m misunderstanding how GodeferworksIn any case,
FinalizeOutputsmust still run if there’s an error. It shouldn’t be stuck ever (unless there is a bug).@BarryOwensDigital ok, that makes a lot more sense. That means the code from #12544 has a bug in it in the error case.
Also would you mind adding syntax highlighting to your code blocks? i.e. ```yaml. I’ve been editing all your comments to add it 😅
I’ve checked again, I might have been mistaken earlier looking at incorrect WorkflowTaskResult.
In 3.5.5 the WorkflowTaskResult does still have
workflows.argoproj.io/report-outputs-completed: 'false'~Seems like in 3.5.5 we get
workflows.argoproj.io/report-outputs-completed: 'true'. The WorkflowTaskResult is removed quickly on this version~Edit: Yes we get the following in the WorkflowTaskResult for the resource template step:
workflows.argoproj.io/report-outputs-completed: 'false'Logs from the main container (v3.5.8). We expect the “myapp-mongodb not found” which is shown here:
Also agree this might be related to https://github.com/argoproj/argo-workflows/pull/12544, when I enable the debug logs we get this log on the controller:
...msg="taskresults of workflow are incomplete or still have daemon nodes, so can't mark workflow completed" fromPhase=Running namespace=jenkins toPhase=Succeeded ...