argo-workflows: During upgrade: `Maximum recursion depth exceeded` error when not using recursion
Pre-requisites
- I have double-checked my configuration
- I can confirm the issues exists when I tested with
:latest - I’d like to contribute the fix myself (see contributing guide)
What happened/what you expected to happen?
never saw this error in 3.4.11
but after upgrading to 3.5.1
got error in entry template execution: Maximum recursion depth exceeded. See https://argoproj.github.io/argo-workflows/scaling/#maximum-recursion-depth on 2 different small (1 step) workflows
Version
3.5.1
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.
n/a
Logs from the workflow controller
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
Logs from in your workflow’s wait container
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 36 (24 by maintainers)
Commits related to this issue
- fix: provide fallback for 3.4 to 3.5 transition with absent `NodeFlag`. Fixes #12162 (#13504) Signed-off-by: isubasinghe <isitha@pipekit.io> — committed to argoproj/argo-workflows by isubasinghe 2 years ago
- fix: provide fallback for 3.4 to 3.5 transition with absent `NodeFlag`. Fixes #12162 (#13504) Signed-off-by: isubasinghe <isitha@pipekit.io> — committed to argoproj/argo-workflows by isubasinghe 2 years ago
There does appear to be an issue with this specifically in 3.5.1 (not in 3.5.0). Our CI is failing on it, and definitely uses fewer than 100 recursions.
I’m working on something small and replicatable.
I think the problem is that not actually recursion is counted, but rather any “open” call to
executeTemplate. So a workflow that doesn’t recurse (that much) but does fan out into a lot of parallel steps can also hit this limit. (At least this seems to be the case for us, we’ve disabled the feature now.)edit: not true, fan out only increases concurrency by 1
This problem is about workflows running during an upgrade.
If you have non recursive workflows that you can start on 3.5.1 or later that erroneously create this error message please can you show me one that we can use to reproduce it?
3.5.1 introduced new
nodeFlagto the node status block in #11839.@isubasinghe has proved the recursion is happening in retry nodes. I suspect the 3.5.1+ code is making incorrect assumptions on retry nodes that cause this recursion, because they don’t have the
RetriednodeFlagset. Workflows started in 3.5.0 or 3.4 and continued in 3.5.1+ with retry nodes are probably exhibiting this issue.We don’t use steps at Pipekit mate 😃
I don’t recommend anybody setting
DISABLE_MAX_RECURSION=true, because that will end up with stack overflow panics when you start hitting the problem in a workflow, and the argo workflow controller will keep restarting until you delete the problematic workflow.We were still facing this issue in Argo 3.5.2, to debug the problem, I baked an image to do the mentioned log changed above and promoted its level from debug to info. Then in a cluster where we were having this issue, I switched to this image, setting
DISABLE_MAX_RECURSION=falseat the same time, and I got the following logs:In
status.nodes, that was the state for one of the failing retry step:I noticed that we don’t have the following expected part in the retried Pod step:
To repeat the problem, I prepared a minimal workflow like that:
then:
and voila:
Unfortunately I couldn’t find a natural way yet to repeat the same issue without manually changing the status field, but I believe this is a case that is hit during a coinciding argo workflow controller restart.
O, yeah, I have to retract that statement. It’s not true. We tried to reproduce it too and a fan out does only increase it by 1. Sorry. Still trying to find out what does cause it. I tried to sneak in an extra log line for the concurrency so far. But I hadn’t time to look at this after anymore (just disabled the max recursion check).
@biochimia, can you see if it reoccurs after a workflow-controller restart? All cases I’ve had reported have been on a fresh upgrade only, but a controller restart has made them go away.
This seems to be the case now with several upgrades to 3.5, in flight upgrades seem to cause this to happen, but afterwards it is fine.
I agree with @agilgur5, it would be bad to try and hide the bugs this way, we should either have it as a working feature or get rid of it. It’s still working fine for @tico24 at the moment.
I’m looking into it, workflows that reproduce it would be helpful though.