argo-workflows: Dependencies and Enhanced Depends is broken if you call a task 'split'

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

If you happen to call a task split, you’ll break anything that depends on it.

This is broken:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: broken-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: split
            template: foo
          - name: map
            template: foo
            depends: split

    - name: foo
      container:
        image: alpine
        command:
          - sh
          - -c
          - |
            echo "foo"

This is the same workflow, but with split renamed to bar. It works fine:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: working-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: bar
            template: foo
          - name: map
            template: foo
            depends: bar

    - name: foo
      container:
        image: alpine
        command:
          - sh
          - -c
          - |
            echo "foo"

The resulting error in the UI is

unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': type func(...interface {}) (interface {}, error)[string] is undefined (1:8) | (split.Succeeded || split.Skipped || split.Daemoned) | .......^

This used to work on 3.4.11 and earlier. I’m even convinced it worked on both 3.5.0 release candidates.

Version

v3.5.0

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

See above

Logs from the workflow controller

time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2364469670, taskName map"                                                                                  time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-4039980688 phase Running -> Succeeded" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-4039980688 finished: 2023-10-19 07:11:28.513582048 +0000 UTC" namespace=ci workflow=wat-fpdsp                                              time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2364469670, taskName map"
time="2023-10-19T07:11:28.513Z" level=info msg="Skipped node wat-fpdsp-2364469670 initialized Error (message: unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': unabletime="2023-10-19T07:11:28.513Z" level=info msg="Outbound nodes of wat-fpdsp-2263525054 set to [wat-fpdsp-2364469670]" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-2263525054 phase Running -> Error" namespace=ci workflow=wat-fpdsp                                                                         time="2023-10-19T07:11:28.513Z" level=info msg="node wat-fpdsp-2263525054 finished: 2023-10-19 07:11:28.513781073 +0000 UTC" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=ci workflow=wat-fpdsp                                                                    time="2023-10-19T07:11:28.513Z" level=info msg="1 child nodes of wat-fpdsp failed. Trying again..." namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=ci workflow=wat-fpdsp                                                      time="2023-10-19T07:11:28.513Z" level=info msg="DAG node wat-fpdsp-2733151291 initialized Running" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"                                                                                  time="2023-10-19T07:11:28.513Z" level=warning msg="was unable to obtain the node for wat-fpdsp-942064205, taskName split"
time="2023-10-19T07:11:28.514Z" level=warning msg="was unable to obtain the node for wat-fpdsp-942064205, taskName split"                                                                                 time="2023-10-19T07:11:28.514Z" level=info msg="All of node wat-fpdsp(1).split dependencies [] completed" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.514Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=ci workflow=wat-fpdsp                                                                     time="2023-10-19T07:11:28.514Z" level=info msg="Retry node wat-fpdsp-942064205 initialized Running" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.514Z" level=info msg="Pod node wat-fpdsp-2693710620 initialized Pending" namespace=ci workflow=wat-fpdsp                                                                        time="2023-10-19T07:11:28.537Z" level=info msg="Created pod: wat-fpdsp(1).split(0) (wat-fpdsp-foo-2693710620)" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.537Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"                                                                                  time="2023-10-19T07:11:28.537Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:28.537Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp                                                                                                   time="2023-10-19T07:11:28.537Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:28.537Z" level=info msg="Workflow to be dehydrated" Workflow Size=3285                                                                                                             time="2023-10-19T07:11:28.575Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536869 workflow=wat-fpdsp
time="2023-10-19T07:11:31.534Z" level=info msg="Processing workflow" namespace=ci workflow=wat-fpdsp                                                                                                      time="2023-10-19T07:11:31.535Z" level=info msg="Task-result reconciliation" namespace=ci numObjs=0 workflow=wat-fpdsp
time="2023-10-19T07:11:31.535Z" level=info msg="node changed" namespace=ci new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=wat-fpdsp-2693710620 old.message= old.phase=Pending old.ptime="2023-10-19T07:11:31.535Z" level=info msg="node unchanged" namespace=ci nodeID=wat-fpdsp-2072654003 workflow=wat-fpdsp
time="2023-10-19T07:11:31.535Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"                                                                                  time="2023-10-19T07:11:31.535Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:31.535Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"                                                                                  time="2023-10-19T07:11:31.535Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:31.535Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp                                                                                                          time="2023-10-19T07:11:31.535Z" level=info msg="Workflow to be dehydrated" Workflow Size=3343
time="2023-10-19T07:11:31.555Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536902 workflow=wat-fpdsp                                                       time="2023-10-19T07:11:34.536Z" level=info msg="Processing workflow" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:34.537Z" level=info msg="Task-result reconciliation" namespace=ci numObjs=0 workflow=wat-fpdsp                                                                                     time="2023-10-19T07:11:34.537Z" level=info msg="node changed" namespace=ci new.message= new.phase=Running new.progress=0/1 nodeID=wat-fpdsp-2693710620 old.message=PodInitializing old.phase=Pending old.p
time="2023-10-19T07:11:34.537Z" level=info msg="node unchanged" namespace=ci nodeID=wat-fpdsp-2072654003 workflow=wat-fpdsp                                                                               time="2023-10-19T07:11:34.538Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:34.538Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"                                                                                  time="2023-10-19T07:11:34.538Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:34.538Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp                                                                                                   time="2023-10-19T07:11:34.538Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:34.538Z" level=info msg="Workflow to be dehydrated" Workflow Size=3332                                                                                                             time="2023-10-19T07:11:34.543Z" level=info msg="cleaning up pod" action=terminateContainers key=ci/wat-fpdsp-foo-2693710620/terminateContainers
time="2023-10-19T07:11:34.553Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536924 workflow=wat-fpdsp                                                       time="2023-10-19T07:11:37.619Z" level=info msg="Processing workflow" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.619Z" level=info msg="Task-result reconciliation" namespace=ci numObjs=0 workflow=wat-fpdsp                                                                                     time="2023-10-19T07:11:37.620Z" level=info msg="node changed" namespace=ci new.message= new.phase=Succeeded new.progress=0/1 nodeID=wat-fpdsp-2693710620 old.message= old.phase=Running old.progress=0/1 w
time="2023-10-19T07:11:37.620Z" level=info msg="node unchanged" namespace=ci nodeID=wat-fpdsp-2072654003 workflow=wat-fpdsp                                                                               time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"
time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-942064205 phase Running -> Succeeded" namespace=ci workflow=wat-fpdsp                                                                      time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-942064205 finished: 2023-10-19 07:11:37.62036697 +0000 UTC" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-2348667151, taskName map"                                                                                  time="2023-10-19T07:11:37.620Z" level=info msg="Skipped node wat-fpdsp-2348667151 initialized Error (message: unable to evaluate expression '(split.Succeeded || split.Skipped || split.Daemoned)': unable
time="2023-10-19T07:11:37.620Z" level=info msg="Outbound nodes of wat-fpdsp-2733151291 set to [wat-fpdsp-2348667151]" namespace=ci workflow=wat-fpdsp                                                     time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-2733151291 phase Running -> Error" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="node wat-fpdsp-2733151291 finished: 2023-10-19 07:11:37.620538442 +0000 UTC" namespace=ci workflow=wat-fpdsp                                              time="2023-10-19T07:11:37.620Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="2 child nodes of wat-fpdsp failed. Trying again..." namespace=ci workflow=wat-fpdsp                                                                       time="2023-10-19T07:11:37.620Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="DAG node wat-fpdsp-49276704 initialized Running" namespace=ci workflow=wat-fpdsp                                                                          time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-911175476, taskName map"
time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-4145173754, taskName split"                                                                                time="2023-10-19T07:11:37.620Z" level=warning msg="was unable to obtain the node for wat-fpdsp-4145173754, taskName split"
time="2023-10-19T07:11:37.620Z" level=info msg="All of node wat-fpdsp(2).split dependencies [] completed" namespace=ci workflow=wat-fpdsp                                                                 time="2023-10-19T07:11:37.620Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.620Z" level=info msg="Retry node wat-fpdsp-4145173754 initialized Running" namespace=ci workflow=wat-fpdsp                                                                      time="2023-10-19T07:11:37.620Z" level=info msg="Pod node wat-fpdsp-3447821985 initialized Pending" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.655Z" level=info msg="Created pod: wat-fpdsp(2).split(0) (wat-fpdsp-foo-3447821985)" namespace=ci workflow=wat-fpdsp                                                            time="2023-10-19T07:11:37.655Z" level=warning msg="was unable to obtain the node for wat-fpdsp-911175476, taskName map"
time="2023-10-19T07:11:37.655Z" level=warning msg="was unable to obtain the node for wat-fpdsp-911175476, taskName map"                                                                                   time="2023-10-19T07:11:37.656Z" level=info msg="TaskSet Reconciliation" namespace=ci workflow=wat-fpdsp
time="2023-10-19T07:11:37.656Z" level=info msg=reconcileAgentPod namespace=ci workflow=wat-fpdsp                                                                                                          time="2023-10-19T07:11:37.656Z" level=info msg="Workflow to be dehydrated" Workflow Size=4503
time="2023-10-19T07:11:37.676Z" level=info msg="Workflow update successful" namespace=ci phase=Running resourceVersion=175536948 workflow=wat-fpdsp

Logs from in your workflow’s wait container

none relevant

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 29 (13 by maintainers)

Commits related to this issue

Most upvoted comments

There’s a PR out above: #12573. That will almost certainly make it into the next patch release, 3.5.5

Thanks @antonmedv, appreciate your quick response

that built-ins now have priority over added context variables?

I see. Let me refactor expr so env variables take precedence over built-in function (plus some syntax to still be able to call top overridden built-ins).

join also appears to be a problem

Will release today or tomorrow.

After looking at the expr-lang/expr code, I am surprised how this ever worked actually. expr-lang has two phases, a compile phase and a run phase. The eval function first compiles the stringly typed expression into a expr-langs VM instructions, then the run phase kicks in (this is where reads from the env happen).

Our error happens in the compile phase (and rightly so) because split is of the wrong type. I suspect a commit in expr-lang introduced stricter type checking and this might have been the reason it worked in the past. If this is the case, the reason our code worked in the past was because of a bug (or lack of a feature(type checking)) in expr-lang.

The fix for this should be from our end ultimately (imo), it’s quite simple to do, replace all “${keyword}” expressions with “${argo_wf_keyword}” and then perform this replace in the env as well.

We could also ask expr-lang to special case the Eval function, where we leak the env to the Compile function, where it overrides it’s own definition of “${keyword}”.

If that’s the case that 3.4.x worked with Argo variables overriding built-in functions, then it’s possibly a regression in expr: that built-ins now have priority over added context variables?

It also breaks if you call a task map