argo-workflows: During upgrade: `Maximum recursion depth exceeded` error when not using recursion

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

never saw this error in 3.4.11

but after upgrading to 3.5.1 got error in entry template execution: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth on 2 different small (1 step) workflows

Version

3.5.1

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

n/a

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow’s wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 36 (24 by maintainers)

Commits related to this issue

Most upvoted comments

There does appear to be an issue with this specifically in 3.5.1 (not in 3.5.0). Our CI is failing on it, and definitely uses fewer than 100 recursions.

I’m working on something small and replicatable.

I think the problem is that not actually recursion is counted, but rather any “open” call to executeTemplate. So a workflow that doesn’t recurse (that much) but does fan out into a lot of parallel steps can also hit this limit. (At least this seems to be the case for us, we’ve disabled the feature now.)

edit: not true, fan out only increases concurrency by 1

This problem is about workflows running during an upgrade.

If you have non recursive workflows that you can start on 3.5.1 or later that erroneously create this error message please can you show me one that we can use to reproduce it?

3.5.1 introduced new nodeFlag to the node status block in #11839.

@isubasinghe has proved the recursion is happening in retry nodes. I suspect the 3.5.1+ code is making incorrect assumptions on retry nodes that cause this recursion, because they don’t have the Retried nodeFlag set. Workflows started in 3.5.0 or 3.4 and continued in 3.5.1+ with retry nodes are probably exhibiting this issue.

We don’t use steps at Pipekit mate 😃

I don’t recommend anybody setting DISABLE_MAX_RECURSION=true, because that will end up with stack overflow panics when you start hitting the problem in a workflow, and the argo workflow controller will keep restarting until you delete the problematic workflow.

We were still facing this issue in Argo 3.5.2, to debug the problem, I baked an image to do the mentioned log changed above and promoted its level from debug to info. Then in a cluster where we were having this issue, I switched to this image, setting DISABLE_MAX_RECURSION=false at the same time, and I got the following logs:

# while it was `DISABLE_MAX_RECURSION=true`
2024-05-14T00:04:35.20947308Z,"time=""2024-05-14T00:04:35.209Z"" level=info msg=""Processing workflow"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.252261778Z,"time=""2024-05-14T00:04:35.252Z"" level=info msg=""Task-result reconciliation"" namespace=default numObjs=0 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.252807048Z,"time=""2024-05-14T00:04:35.252Z"" level=info msg=""Pod failed: Error (exit code 1)"" displayName=""run-train(0)"" namespace=default pod=rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527 templateName=run-train workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.252924701Z,"time=""2024-05-14T00:04:35.252Z"" level=warning msg=""workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.252990251Z,"time=""2024-05-14T00:04:35.252Z"" level=warning msg=""workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.253160097Z,"time=""2024-05-14T00:04:35.253Z"" level=info msg=""node changed"" namespace=default new.message=""Error (exit code 1)"" new.phase=Failed new.progress=0/1 nodeID=rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527 old.message= old.phase=Running old.progress=0/1 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.253304318Z,"time=""2024-05-14T00:04:35.253Z"" level=info msg=""Workflow pod is missing"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0)"" nodePhase=Running recentlyStarted=false workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.253395075Z,"time=""2024-05-14T00:04:35.253Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-794976774 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.253462751Z,"time=""2024-05-14T00:04:35.253Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-794976774 message: pod deleted"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T00:04:35.253543399Z,"time=""2024-05-14T00:04:35.253Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-794976774 finished: 2024-05-14 00:04:35.253496244 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
# ... repeating the same pattern until we set `DISABLE_MAX_RECURSION` from true to false at 2024-05-14T22:12:57, then:
2024-05-14T22:12:57.868454091Z,"time=""2024-05-14T22:12:57.864Z"" level=info msg=""Processing workflow"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.905766307Z,"time=""2024-05-14T22:12:57.905Z"" level=info msg=""Task-result reconciliation"" namespace=default numObjs=0 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.90807108Z,"time=""2024-05-14T22:12:57.906Z"" level=info msg=""Pod failed: Error (exit code 1)"" displayName=""run-train(0)"" namespace=default pod=rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527 templateName=run-train workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912209745Z,"time=""2024-05-14T22:12:57.912Z"" level=warning msg=""workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912220182Z,"time=""2024-05-14T22:12:57.912Z"" level=warning msg=""workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912294118Z,"time=""2024-05-14T22:12:57.912Z"" level=info msg=""node changed"" namespace=default new.message=""Error (exit code 1)"" new.phase=Failed new.progress=0/1 nodeID=rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527 old.message= old.phase=Running old.progress=0/1 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912446978Z,"time=""2024-05-14T22:12:57.912Z"" level=info msg=""Workflow pod is missing"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0)"" nodePhase=Running recentlyStarted=false workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912465037Z,"time=""2024-05-14T22:12:57.912Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-794976774 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912484951Z,"time=""2024-05-14T22:12:57.912Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-794976774 message: pod deleted"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912544446Z,"time=""2024-05-14T22:12:57.912Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-794976774 finished: 2024-05-14 22:12:57.912498416 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912617332Z,"time=""2024-05-14T22:12:57.912Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644: template: *v1alpha1.WorkflowStep (ml-k-module-xxx-ads-categorize-train), boundaryID: , currentStackDepth: 0"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.912878499Z,"time=""2024-05-14T22:12:57.912Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1: template: *v1alpha1.DAGTask (exit-handler-1), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644, currentStackDepth: 1"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.913926903Z,"time=""2024-05-14T22:12:57.913Z"" level=warning msg=""was unable to obtain the node for rr-ml-k-module-xxx-74gmh-1-1379393644-2682863345, taskName run-mlflow-log"""
2024-05-14T22:12:57.914737228Z,"time=""2024-05-14T22:12:57.914Z"" level=warning msg=""was unable to obtain the node for rr-ml-k-module-xxx-74gmh-1-1379393644-3044613789, taskName run-finalize"""
2024-05-14T22:12:57.914752884Z,"time=""2024-05-14T22:12:57.914Z"" level=warning msg=""was unable to obtain the node for rr-ml-k-module-xxx-74gmh-1-1379393644-2946699175, taskName list-all-trainings"""
2024-05-14T22:12:57.920401028Z,"time=""2024-05-14T22:12:57.920Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(0...<LONG PARAM LIST>...): template: *v1alpha1.DAGTask (for-loop-5), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619, currentStackDepth: 2"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.921250032Z,"time=""2024-05-14T22:12:57.921Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(1...<LONG PARAM LIST>...): template: *v1alpha1.DAGTask (for-loop-5), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619, currentStackDepth: 2"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.922065485Z,"time=""2024-05-14T22:12:57.921Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(2...<LONG PARAM LIST>...): template: *v1alpha1.DAGTask (for-loop-5), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619, currentStackDepth: 2"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.922882744Z,"time=""2024-05-14T22:12:57.922Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...): template: *v1alpha1.DAGTask (for-loop-5), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619, currentStackDepth: 2"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:57.926746957Z,"time=""2024-05-14T22:12:57.923Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...).run-train: template: *v1alpha1.DAGTask (run-train), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2035740812, currentStackDepth: 3"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
# ... repeating ~100 times
2024-05-14T22:12:58.46761812Z,"time=""2024-05-14T22:12:58.453Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...).run-train: template: *v1alpha1.DAGTask (run-train), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2035740812, currentStackDepth: 99"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.472900557Z,"time=""2024-05-14T22:12:58.471Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...).run-train"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.472931908Z,"time=""2024-05-14T22:12:58.471Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-562365516 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.472935313Z,"time=""2024-05-14T22:12:58.471Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-562365516 message: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.47293766Z,"time=""2024-05-14T22:12:58.471Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-562365516 finished: 2024-05-14 22:12:58.471853015 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.473096379Z,"time=""2024-05-14T22:12:58.471Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...).run-train(0)"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.473106159Z,"time=""2024-05-14T22:12:58.471Z"" level=error msg=""node is already fulfilled"" fromPhase=Failed namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...).run-train(0)"" toPhase=Error workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.473112838Z,"time=""2024-05-14T22:12:58.472Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527 phase Failed -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.473115521Z,"time=""2024-05-14T22:12:58.472Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527 message: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.473162192Z,"time=""2024-05-14T22:12:58.472Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...).run-train(0)"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
# ... repeating ~100 times
2024-05-14T22:12:58.517350212Z,"time=""2024-05-14T22:12:58.517Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(3...<LONG PARAM LIST>...).run-train(0)"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.517624319Z,"time=""2024-05-14T22:12:58.517Z"" level=info msg=""Outbound nodes of rr-ml-k-module-xxx-74gmh-1-1379393644-2035740812 set to [rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527]"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.517658723Z,"time=""2024-05-14T22:12:58.517Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2035740812 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.517662915Z,"time=""2024-05-14T22:12:58.517Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2035740812 finished: 2024-05-14 22:12:58.517618731 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.52270363Z,"time=""2024-05-14T22:12:58.522Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...): template: *v1alpha1.DAGTask (for-loop-5), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619, currentStackDepth: 2"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.523771669Z,"time=""2024-05-14T22:12:58.523Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train: template: *v1alpha1.DAGTask (run-train), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327, currentStackDepth: 3"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.864898413Z,"time=""2024-05-14T22:12:58.864Z"" level=info msg=""Created pod: rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0) (rr-ml-k-module-xxx-74gmh-1-1379393644-794976774)"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:58.865015582Z,"time=""2024-05-14T22:12:58.864Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train: template: *v1alpha1.DAGTask (run-train), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327, currentStackDepth: 4"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.418885046Z,"time=""2024-05-14T22:12:59.418Z"" level=info msg=""Failed pod rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0) (rr-ml-k-module-xxx-74gmh-1-1379393644-794976774) creation: already exists"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.41905412Z,"time=""2024-05-14T22:12:59.418Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train: template: *v1alpha1.DAGTask (run-train), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327, currentStackDepth: 5"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.671258908Z,"time=""2024-05-14T22:12:59.671Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train: template: *v1alpha1.DAGTask (run-train), boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327, currentStackDepth: 99"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
# ... repeating ~100 times
2024-05-14T22:12:59.671361012Z,"time=""2024-05-14T22:12:59.671Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.671370981Z,"time=""2024-05-14T22:12:59.671Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-4042243083 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.671377684Z,"time=""2024-05-14T22:12:59.671Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-4042243083 message: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.671380425Z,"time=""2024-05-14T22:12:59.671Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-4042243083 finished: 2024-05-14 22:12:59.671315687 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.671445639Z,"time=""2024-05-14T22:12:59.671Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0)"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.671451251Z,"time=""2024-05-14T22:12:59.671Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-794976774 message: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.671485433Z,"time=""2024-05-14T22:12:59.671Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0)"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
# ... repeating ~100 times
024-05-14T22:12:59.693476433Z,"time=""2024-05-14T22:12:59.678Z"" level=error msg=""Mark error node"" error=""Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth"" namespace=default nodeName=""rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0)"" workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693480732Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""Outbound nodes of rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327 set to [rr-ml-k-module-xxx-74gmh-1-1379393644-794976774]"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693483374Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.69348586Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327 finished: 2024-05-14 22:12:59.678392409 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693487846Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2704183877 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693489799Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2704183877 finished: 2024-05-14 22:12:59.678511785 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693492096Z,"time=""2024-05-14T22:12:59.678Z"" level=warning msg=""was unable to obtain the node for rr-ml-k-module-xxx-74gmh-1-1379393644-2946699175, taskName list-all-trainings"""
2024-05-14T22:12:59.693494139Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""Skipped node rr-ml-k-module-xxx-74gmh-1-1379393644-2946699175 initialized Omitted (message: omitted: depends condition not met)"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693496191Z,"time=""2024-05-14T22:12:59.678Z"" level=warning msg=""was unable to obtain the node for rr-ml-k-module-xxx-74gmh-1-1379393644-3044613789, taskName run-finalize"""
2024-05-14T22:12:59.693499875Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""Skipped node rr-ml-k-module-xxx-74gmh-1-1379393644-3044613789 initialized Omitted (message: omitted: depends condition not met)"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693502123Z,"time=""2024-05-14T22:12:59.678Z"" level=warning msg=""was unable to obtain the node for rr-ml-k-module-xxx-74gmh-1-1379393644-2682863345, taskName run-mlflow-log"""
2024-05-14T22:12:59.693504133Z,"time=""2024-05-14T22:12:59.678Z"" level=info msg=""Skipped node rr-ml-k-module-xxx-74gmh-1-1379393644-2682863345 initialized Omitted (message: omitted: depends condition not met)"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693507768Z,"time=""2024-05-14T22:12:59.679Z"" level=info msg=""Outbound nodes of rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619 set to [rr-ml-k-module-xxx-74gmh-1-1379393644-2682863345]"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693509795Z,"time=""2024-05-14T22:12:59.679Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693511871Z,"time=""2024-05-14T22:12:59.679Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644-2150580619 finished: 2024-05-14 22:12:59.679122842 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693584338Z,"time=""2024-05-14T22:12:59.679Z"" level=info msg=""Outbound nodes of rr-ml-k-module-xxx-74gmh-1-1379393644 set to [rr-ml-k-module-xxx-74gmh-1-1379393644-2682863345]"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693586783Z,"time=""2024-05-14T22:12:59.689Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644 phase Running -> Error"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693589171Z,"time=""2024-05-14T22:12:59.689Z"" level=info msg=""node rr-ml-k-module-xxx-74gmh-1-1379393644 finished: 2024-05-14 22:12:59.689573024 +0000 UTC"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693591599Z,"time=""2024-05-14T22:12:59.689Z"" level=info msg=""TaskSet Reconciliation"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693593733Z,"time=""2024-05-14T22:12:59.689Z"" level=info msg=reconcileAgentPod namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693595694Z,"time=""2024-05-14T22:12:59.689Z"" level=info msg=""Running OnExit handler: exit-handlers-train"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693598475Z,"time=""2024-05-14T22:12:59.689Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.onExit: template: *v1alpha1.WorkflowStep (exit-handlers-train), boundaryID: , currentStackDepth: 0"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693600994Z,"time=""2024-05-14T22:12:59.689Z"" level=warning msg=""Node was nil, will be initialized as type Skipped"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693604096Z,"time=""2024-05-14T22:12:59.691Z"" level=info msg=""was unable to obtain node for , letting display name to be nodeName"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693606295Z,"time=""2024-05-14T22:12:59.691Z"" level=info msg=""Retry node rr-ml-k-module-xxx-74gmh-1-1379393644-1646786213 initialized Running"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693608247Z,"time=""2024-05-14T22:12:59.692Z"" level=info msg=""was unable to obtain node for , letting display name to be nodeName"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:12:59.693610282Z,"time=""2024-05-14T22:12:59.692Z"" level=info msg=""Pod node rr-ml-k-module-xxx-74gmh-1-1379393644-1649482756 initialized Pending"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:00.291666651Z,"time=""2024-05-14T22:13:00.291Z"" level=info msg=""Created pod: rr-ml-k-module-xxx-74gmh-1-1379393644.onExit(0) (rr-ml-k-module-xxx-74gmh-1-1379393644-1649482756)"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:00.347960727Z,"time=""2024-05-14T22:13:00.347Z"" level=info msg=""Workflow update successful"" namespace=default phase=Running resourceVersion=2131354266 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:00.351118572Z,"time=""2024-05-14T22:13:00Z"" level=info msg=""Processing object (rr-ml-k-module-xxx-74gmh-1-1379393644): object has no owner."" Workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:00.358677615Z,"time=""2024-05-14T22:13:00.358Z"" level=info msg=""cleaning up pod"" action=labelPodCompleted key=default/rr-ml-k-module-xxx-74gmh-1-1379393644-3334131527/labelPodCompleted"
2024-05-14T22:13:00.358858972Z,"time=""2024-05-14T22:13:00.358Z"" level=info msg=""cleaning up pod"" action=labelPodCompleted key=default/rr-ml-k-module-xxx-74gmh-1-1379393644-794976774/labelPodCompleted"
2024-05-14T22:13:06.379389483Z,"time=""2024-05-14T22:13:06.377Z"" level=info msg=""Processing workflow"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.441437157Z,"time=""2024-05-14T22:13:06.441Z"" level=info msg=""Task-result reconciliation"" namespace=default numObjs=0 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.441726468Z,"time=""2024-05-14T22:13:06.441Z"" level=info msg=""node changed"" namespace=default new.message=""Unschedulable: 0/33 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/role: mem-8xlarge}, 11 node(s) had untolerated taint {example.com/dedicated: cluster-management}, 12 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint {example.com/dedicated: ml-mgmt}, 3 node(s) had untolerated taint {example.com/dedicated: airflow-jobs}, 4 node(s) had untolerated taint {example.com/dedicated: bd-mgmt}. preemption: 0/33 nodes are available: 33 Preemption is not helpful for scheduling.."" new.phase=Pending new.progress=0/1 nodeID=rr-ml-k-module-xxx-74gmh-1-1379393644-1649482756 old.message= old.phase=Pending old.progress=0/1 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.441856674Z,"time=""2024-05-14T22:13:06.441Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644: template: *v1alpha1.WorkflowStep (ml-k-module-xxx-ads-categorize-train), boundaryID: , currentStackDepth: 0"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.44220717Z,"time=""2024-05-14T22:13:06.442Z"" level=info msg=""TaskSet Reconciliation"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.44228502Z,"time=""2024-05-14T22:13:06.442Z"" level=info msg=reconcileAgentPod namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.442342693Z,"time=""2024-05-14T22:13:06.442Z"" level=info msg=""Running OnExit handler: exit-handlers-train"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.442414045Z,"time=""2024-05-14T22:13:06.442Z"" level=info msg=""Evaluating node rr-ml-k-module-xxx-74gmh-1-1379393644.onExit: template: *v1alpha1.WorkflowStep (exit-handlers-train), boundaryID: , currentStackDepth: 0"" namespace=default workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"
2024-05-14T22:13:06.557979115Z,"time=""2024-05-14T22:13:06.557Z"" level=info msg=""Workflow update successful"" namespace=default phase=Running resourceVersion=2131355212 workflow=rr-ml-k-module-xxx-74gmh-1-1379393644"

In status.nodes, that was the state for one of the failing retry step:

status:
  # ...
  nodes:
    # ...
    rr-ml-k-module-xxx-74gmh-1-1379393644-4042243083:
      boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327
      children:
      - rr-ml-k-module-xxx-74gmh-1-1379393644-794976774
      displayName: run-train
      finishedAt: "2024-05-14T22:12:59Z"
      id: rr-ml-k-module-xxx-74gmh-1-1379393644-4042243083
      inputs:
        parameters:
        # ...
      message: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth
      name: rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train
      phase: Error
      progress: 0/1
      startedAt: "2024-05-03T05:30:54Z"
      templateName: run-train
      templateScope: local/rr-ml-k-module-xxx-74gmh-1-1379393644
      type: Retry
    rr-ml-k-module-xxx-74gmh-1-1379393644-794976774:
      boundaryID: rr-ml-k-module-xxx-74gmh-1-1379393644-2396377327
      children:
      - rr-ml-k-module-xxx-74gmh-1-1379393644-2946699175
      displayName: run-train(0)
      finishedAt: "2024-05-14T22:12:57Z"
      hostNodeName: ip-10-0-30-148.eu-west-1.compute.internal
      id: rr-ml-k-module-xxx-74gmh-1-1379393644-794976774
      inputs:
        parameters:
        # ...
      message: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth
      name: rr-ml-k-module-xxx-74gmh-1-1379393644.exit-handler-1.for-loop-5(4...<LONG PARAM LIST>...).run-train(0)
      phase: Error
      progress: 0/1
      startedAt: "2024-05-03T05:30:54Z"
      templateName: run-train
      templateScope: local/rr-ml-k-module-xxx-74gmh-1-1379393644
      type: Pod

I noticed that we don’t have the following expected part in the retried Pod step:

      nodeFlag:
        retried: true

To repeat the problem, I prepared a minimal workflow like that:

# reproduce-max-recursion-depth-exceeded.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: reproduce-max-recursion-depth-exceeded
spec:
  entrypoint: sleep-long-and-exit-peacefully-on-sigint
  templates:
  - name: sleep-long-and-exit-peacefully-on-sigint
    container:
      args:
      - |
        trap 'echo exiting 0; exit 0' INT
        sleep 3600
      command:
      - ash
      - -c
      image: alpine:latest
      name: sleep-and-exit
    retryStrategy:
      limit: 1

then:

kubectl apply -f reproduce-max-recursion-depth-exceeded.yaml
kubectl edit workflow reproduce-max-recursion-depth-exceeded # in status, delete the `nodeFlag` field
kubectl exec -it "$(kubectl get pods | grep reproduce-max-recursion-depth-exceeded | grep Running | awk '{print $1}')" -- ash -c 'kill -INT 1'

and voila:

image

Unfortunately I couldn’t find a natural way yet to repeat the same issue without manually changing the status field, but I believe this is a case that is hit during a coinciding argo workflow controller restart.

O, yeah, I have to retract that statement. It’s not true. We tried to reproduce it too and a fan out does only increase it by 1. Sorry. Still trying to find out what does cause it. I tried to sneak in an extra log line for the concurrency so far. But I hadn’t time to look at this after anymore (just disabled the max recursion check).

@biochimia, can you see if it reoccurs after a workflow-controller restart? All cases I’ve had reported have been on a fresh upgrade only, but a controller restart has made them go away.

This seems to be the case now with several upgrades to 3.5, in flight upgrades seem to cause this to happen, but afterwards it is fine.

I agree with @agilgur5, it would be bad to try and hide the bugs this way, we should either have it as a working feature or get rid of it. It’s still working fine for @tico24 at the moment.

I’m looking into it, workflows that reproduce it would be helpful though.