argo-workflows: Dependencies and Enhanced Depends is broken if you call a task 'split'
Pre-requisites
- I have double-checked my configuration
- I can confirm the issues exists when I tested with
:latest - I’d like to contribute the fix myself (see contributing guide)
What happened/what you expected to happen?
If you happen to call a task split, you’ll break anything that depends on it.
This is broken:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: broken-
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: split
template: foo
- name: map
template: foo
depends: split
- name: foo
container:
image: alpine
command:
- sh
- -c
- |
echo "foo"
This is the same workflow, but with split renamed to bar. It works fine:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: working-
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: bar
template: foo
- name: map
template: foo
depends: bar
- name: foo
container:
image: alpine
command:
- sh
- -c
- |
echo "foo"
The resulting error in the UI is
unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': type func(...interface {}) (interface {}, error)[string] is undefined (1:8) | (split.Succeeded || split.Skipped || split.Daemoned) | .......^
This used to work on 3.4.11 and earlier. I’m even convinced it worked on both 3.5.0 release candidates.
Version
v3.5.0
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.
See above
Logs from the workflow controller
time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2364469670, taskName map" time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-4039980688 phase Running -> Succeeded" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-4039980688 finished: 2023-10-19 07:11:28.513582048 +0000 UTC" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2364469670, taskName map"
time="2023-10-19T07:11:28.513Z" level=info msg="Skipped node wat-fpdsp-2364469670 initialized Error (message: unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': unabletime="2023-10-19T07:11:28.513Z" level=info msg="Outbound nodes of wat-fpdsp-2263525054 set to [wat-fpdsp-2364469670]" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-2263525054 phase Running -> Error" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-2263525054 finished: 2023-10-19 07:11:28.513781073 +0000 UTC" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:28.513Z" level=info msg="1 child nodes of wat-fpdsp failed. Trying again..." namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:28.513Z" level=info msg="DAG node wat-fpdsp-2733151291 initialized Running" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map" time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-942064205, taskName split"
time="2023-10-19T07:11:28.514Z" level=warning msg="was unable to obtain the node for wat-fpdsp-942064205, taskName split" time="2023-10-19T07:11:28.514Z" level=info msg="All of node wat-fpdsp(1).split dependencies [] completed" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.514Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:28.514Z" level=info msg="Retry node wat-fpdsp-942064205 initialized Running" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.514Z" level=info msg="Pod node wat-fpdsp-2693710620 initialized Pending" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:28.537Z" level=info msg="Created pod: wat-fpdsp(1).split(0) (wat-fpdsp-foo-2693710620)" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.537Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map" time="2023-10-19T07:11:28.537Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:28.537Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:28.537Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.537Z" level=info msg="Workflow to be dehydrated" Workflow Size=3285 time="2023-10-19T07:11:28.575Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536869 workflow=wat-fpdsp
time="2023-10-19T07:11:31.534Z" level=info msg="Processing workflow" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:31.535Z" level=info msg="Task-result reconciliation" namespace=ci numObjs=0 workflow=wat-fpdsp
time="2023-10-19T07:11:31.535Z" level=info msg="node changed" namespace=ci new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=wat-fpdsp-2693710620 old.message= old.phase=Pending old.ptime="2023-10-19T07:11:31.535Z" level=info msg="node unchanged" namespace=ci nodeID=wat-fpdsp-2072654003 workflow=wat-fpdsp
time="2023-10-19T07:11:31.535Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map" time="2023-10-19T07:11:31.535Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:31.535Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map" time="2023-10-19T07:11:31.535Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:31.535Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:31.535Z" level=info msg="Workflow to be dehydrated" Workflow Size=3343
time="2023-10-19T07:11:31.555Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536902 workflow=wat-fpdsp time="2023-10-19T07:11:34.536Z" level=info msg="Processing workflow" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:34.537Z" level=info msg="Task-result reconciliation" namespace=ci numObjs=0 workflow=wat-fpdsp time="2023-10-19T07:11:34.537Z" level=info msg="node changed" namespace=ci new.message= new.phase=Running new.progress=0/1 nodeID=wat-fpdsp-2693710620 old.message=PodInitializing old.phase=Pending old.p
time="2023-10-19T07:11:34.537Z" level=info msg="node unchanged" namespace=ci nodeID=wat-fpdsp-2072654003 workflow=wat-fpdsp time="2023-10-19T07:11:34.538Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:34.538Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map" time="2023-10-19T07:11:34.538Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:34.538Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:34.538Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:34.538Z" level=info msg="Workflow to be dehydrated" Workflow Size=3332 time="2023-10-19T07:11:34.543Z" level=info msg="cleaning up pod" action=terminateContainers key=ci/wat-fpdsp-foo-2693710620/terminateContainers
time="2023-10-19T07:11:34.553Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536924 workflow=wat-fpdsp time="2023-10-19T07:11:37.619Z" level=info msg="Processing workflow" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.619Z" level=info msg="Task-result reconciliation" namespace=ci numObjs=0 workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=info msg="node changed" namespace=ci new.message= new.phase=Succeeded new.progress=0/1 nodeID=wat-fpdsp-2693710620 old.message= old.phase=Running old.progress=0/1 w
time="2023-10-19T07:11:37.620Z" level=info msg="node unchanged" namespace=ci nodeID=wat-fpdsp-2072654003 workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-942064205 phase Running -> Succeeded" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-942064205 finished: 2023-10-19 07:11:37.62036697 +0000 UTC" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map" time="2023-10-19T07:11:37.620Z" level=info msg="Skipped node wat-fpdsp-2348667151 initialized Error (message: unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': unable
time="2023-10-19T07:11:37.620Z" level=info msg="Outbound nodes of wat-fpdsp-2733151291 set to [wat-fpdsp-2348667151]" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-2733151291 phase Running -> Error" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-2733151291 finished: 2023-10-19 07:11:37.620538442 +0000 UTC" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="2 child nodes of wat-fpdsp failed. Trying again..." namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="DAG node wat-fpdsp-49276704 initialized Running" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-911175476, taskName map"
time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-4145173754, taskName split" time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-4145173754, taskName split"
time="2023-10-19T07:11:37.620Z" level=info msg="All of node wat-fpdsp(2).split dependencies [] completed" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="Retry node wat-fpdsp-4145173754 initialized Running" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.620Z" level=info msg="Pod node wat-fpdsp-3447821985 initialized Pending" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.655Z" level=info msg="Created pod: wat-fpdsp(2).split(0) (wat-fpdsp-foo-3447821985)" namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.655Z" level=warning msg="was unable to obtain the node for wat-fpdsp-911175476, taskName map"
time="2023-10-19T07:11:37.655Z" level=warning msg="was unable to obtain the node for wat-fpdsp-911175476, taskName map" time="2023-10-19T07:11:37.656Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.656Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp time="2023-10-19T07:11:37.656Z" level=info msg="Workflow to be dehydrated" Workflow Size=4503
time="2023-10-19T07:11:37.676Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536948 workflow=wat-fpdsp
Logs from in your workflow’s wait container
none relevant
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 29 (13 by maintainers)
Commits related to this issue
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> — committed to argoproj/argo-workflows by isubasinghe 3 years ago
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> — committed to isubasinghe/argo-workflows by isubasinghe 3 years ago
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> — committed to isubasinghe/argo-workflows by isubasinghe 3 years ago
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> — committed to isubasinghe/argo-workflows by isubasinghe 3 years ago
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> — committed to isubasinghe/argo-workflows by isubasinghe 3 years ago
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isubasinghe@student... — committed to isubasinghe/argo-workflows by isubasinghe 3 years ago
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> — committed to isubasinghe/argo-workflows by isubasinghe 3 years ago
- fix: upgrade expr-lang. Fixes #12037 (#12573) Signed-off-by: isubasinghe <isitha@pipekit.io> Signed-off-by: Isitha Subasinghe <isitha@pipekit.io> — committed to isubasinghe/argo-workflows by isubasinghe 3 years ago
There’s a PR out above: #12573. That will almost certainly make it into the next patch release, 3.5.5
Thanks @antonmedv, appreciate your quick response
I see. Let me refactor expr so env variables take precedence over built-in function (plus some syntax to still be able to call top overridden built-ins).
joinalso appears to be a problemReleased new expr version https://github.com/expr-lang/expr/releases/tag/v1.16.0
Will release today or tomorrow.
After looking at the expr-lang/expr code, I am surprised how this ever worked actually. expr-lang has two phases, a compile phase and a run phase. The eval function first compiles the stringly typed expression into a expr-langs VM instructions, then the run phase kicks in (this is where reads from the env happen).
Our error happens in the compile phase (and rightly so) because split is of the wrong type. I suspect a commit in expr-lang introduced stricter type checking and this might have been the reason it worked in the past. If this is the case, the reason our code worked in the past was because of a bug (or lack of a feature(type checking)) in expr-lang.
The fix for this should be from our end ultimately (imo), it’s quite simple to do, replace all “${keyword}” expressions with “${argo_wf_keyword}” and then perform this replace in the env as well.
We could also ask expr-lang to special case the Eval function, where we leak the env to the
Compilefunction, where it overrides it’s own definition of “${keyword}”.If that’s the case that 3.4.x worked with Argo variables overriding built-in functions, then it’s possibly a regression in
expr: that built-ins now have priority over added context variables?It also breaks if you call a task
map