argo-workflows: Invalid artifact mount path in windows

Checklist:

  • I’ve included the version.
  • I’ve included reproduction steps.
  • I’ve included the workflow YAML.
  • I’ve included the logs.

What happened: Tried to mount an artifact in windows produced an invalid volume specification.

What you expected to happen: Volume specification worked as intended and mounting works.

How to reproduce it (as minimally and precisely as possible):

  1. have a k8s cluster setup with a windows node pool
  2. deploy following workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-directory-
spec:
  entrypoint: artifact-example
  templates:
    - name: artifact-example
      steps:
        - - name: generate-artifact
            template: whalesay
        - - name: consume-artifact
            template: print-message
            arguments:
              artifacts:
                # bind message to the hello-art artifact
                # generated by the generate-artifact step
                - name: message
                  from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

    - name: whalesay
      container:
        image: docker/whalesay:latest
        command: [sh, -c]
        args:
          [
            "mkdir -p /tmp/foobar; cowsay hello world | tee /tmp/foobar/hello_world.txt",
          ]
      outputs:
        artifacts:
          # generate hello-art artifact from /tmp/foobar/hello_world.txt
          # artifacts can be directories as well as files
          - name: hello-art
            path: /tmp/foobar/

    - name: print-message
      inputs:
        artifacts:
          # unpack the message input artifact
          # and put it at C:\message
          - name: message
            path: "C:\\message"
      nodeSelector:
        kubernetes.io/os: windows
      container:
        image: mcr.microsoft.com/windows/nanoserver:1809
        command: ["cmd", "/c"]
        args: ["type C:\\message"]

See the following error when describing the windows pod:

Events:
  Type     Reason     Age   From                                                          Message
  ----     ------     ----  ----                                                          -------
  Normal   Scheduled  94s   default-scheduler                                             Successfully assigned argo/artifact-passing-directory-sscmx-414621924 to gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm
  Normal   Pulled     86s   kubelet, gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm  Container image "argoproj/argoexec:v2.10.0-rc5" already present on machine
  Normal   Created    85s   kubelet, gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm  Created container init
  Normal   Started    57s   kubelet, gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm  Started container init
  Normal   Pulled     17s   kubelet, gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm  Container image "argoproj/argoexec:v2.10.0-rc5" already present on machine
  Warning  Failed     17s   kubelet, gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm  Error: Error response from daemon: invalid volume specification: 'c:\var\lib\kubelet\pods\c99aa2da-9328-41b2-a5b0-3a369052aa8d\volumes\kubernetes.io~empty-dir\input-artifacts\message:/mainctrfs/C:\message'
  Normal   Pulling    17s   kubelet, gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm  Pulling image "mcr.microsoft.com/windows/nanoserver:1809"

Environment:

  • Argo version:
argo: v2.10.0-rc5+e9ca55e.dirty
  BuildDate: 2020-08-03T22:45:45Z
  GitCommit: e9ca55ec1cdbf37a43ee68da756ac91abb4edf73
  GitTreeState: dirty
  GitTag: v2.10.0-rc5
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64
  • Kubernetes version :
clientVersion:
  buildDate: "2020-07-16T00:04:31Z"
  compiler: gc
  gitCommit: dff82dc0de47299ab66c83c626e08b245ab19037
  gitTreeState: clean
  gitVersion: v1.18.6
  goVersion: go1.14.4
  major: "1"
  minor: "18"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-06-25T22:55:26Z"
  compiler: gc
  gitCommit: baccd25d44f1a0d06ad3190eb508784efbb990a5
  gitTreeState: clean
  gitVersion: v1.16.11-gke.5
  goVersion: go1.13.9b4
  major: "1"
  minor: 16+
  platform: linux/amd64

Other debugging information (if applicable):

  • workflow result:
DEBU[0000] CLI version                                   version="{v2.10.0-rc5+e9ca55e.dirty 2020-08-03T22:45:45Z e9ca55ec1cdbf37a43ee68da756ac91abb4edf73 v2.10.0-rc5 dirty go1.13.4 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{ false false}  0x2308ed0 <nil> 0x2308f20}"
Name:                artifact-passing-directory-x6nx6
Namespace:           argo
ServiceAccount:      default
Status:              Running
Created:             Thu Aug 06 15:33:03 +0200 (2 minutes ago)
Started:             Thu Aug 06 15:33:03 +0200 (2 minutes ago)
Duration:            2 minutes 24 seconds

STEP                                 TEMPLATE          PODNAME                                      DURATION  MESSAGE
 ● artifact-passing-directory-x6nx6  artifact-example
 ├---✔ generate-artifact             whalesay          artifact-passing-directory-x6nx6-1448276824  3s
 └---◷ consume-artifact              print-message     artifact-passing-directory-x6nx6-2036393080  2m        PodInitializing
  • executor logs:
time="2020-08-06T13:33:27.915Z" level=info msg="Starting Workflow Executor" version=latest+e9ca55e.dirty
time="2020-08-06T13:33:28.143Z" level=info msg="Creating a docker executor"
time="2020-08-06T13:33:28.143Z" level=info msg="Executor (version: latest+e9ca55e.dirty, build_date: 2020-08-03T22:54:29Z) initialized (pod: argo/artifact-passing-directory-x6nx6-2036393080) with template:\n{\"name\":\"print-message\",\"arguments\":{},\"inputs\":{\"artifacts\":[{\"name\":\"message\",\"path\":\"C:\\\\message\",\"gcs\":{\"bucket\":\"testbucket\",\"serviceAccountKeySecret\":{\"name\":\"argo-gcs-artifact-secret\",\"key\":\"googleStorageCredentials\"},\"key\":\"artifact-passing-directory-x6nx6/artifact-passing-directory-x6nx6-1448276824/hello-art.tgz\"}}]},\"outputs\":{},\"nodeSelector\":{\"kubernetes.io/os\":\"windows\"},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"mcr.microsoft.com/windows/nanoserver:1809\",\"command\":[\"cmd\",\"/c\"],\"args\":[\"type C:\\\\message\"],\"resources\":{}}}"
time="2020-08-06T13:33:28.143Z" level=info msg="Start loading input artifacts..."
time="2020-08-06T13:33:28.143Z" level=info msg="Downloading artifact: message"
time="2020-08-06T13:33:28.147Z" level=info msg="GCS Load path: /argo/inputs/artifacts/message.tmp, key: artifact-passing-directory-x6nx6/artifact-passing-directory-x6nx6-1448276824/hello-art.tgz"
time="2020-08-06T13:33:28.746Z" level=info msg="Detecting if /argo/inputs/artifacts/message.tmp is a tarball"
time="2020-08-06T13:33:28.747Z" level=info msg="Successfully download file: /argo/inputs/artifacts/message"
time="2020-08-06T13:33:28.748Z" level=info msg="Alloc=5307 TotalAlloc=13517 Sys=10886 NumGC=5 Goroutines=6"

full describe pod (without Events part):

Name:         artifact-passing-directory-sscmx-414621924
Namespace:    argo
Priority:     0
Node:         gke-foo-cluster-autoscaling-pool-window-737a8e8c-scbm/10.30.0.34
Start Time:   Thu, 06 Aug 2020 16:11:01 +0200
Labels:       workflows.argoproj.io/completed=false
              workflows.argoproj.io/workflow=artifact-passing-directory-sscmx
Annotations:  workflows.argoproj.io/node-name: artifact-passing-directory-sscmx[1].consume-artifact
              workflows.argoproj.io/template:
                {"name":"print-message","arguments":{},"inputs":{"artifacts":[{"name":"message","path":"C:\\message","gcs":{"bucket":"k8s-foo-clusterelopmen...
Status:       Pending
IP:           10.33.5.10
IPs:
  IP:           10.33.5.10
Controlled By:  Workflow/artifact-passing-directory-sscmx
Init Containers:
  init:
    Container ID:  docker://1a1974c595520d2b8b97d64b8680b475b637ff4aa7a11a26eb318bb58002a79c
    Image:         argoproj/argoexec:v2.10.0-rc5
    Image ID:      docker-pullable://argoproj/argoexec@sha256:268d604b5dc45746fe253bcd7b802fc02adafaca78041a54edde711f307fe027
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 06 Aug 2020 16:11:38 +0200
      Finished:     Thu, 06 Aug 2020 16:12:13 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:  artifact-passing-directory-sscmx-414621924 (v1:metadata.name)
    Mounts:
      /argo/inputs/artifacts from input-artifacts (rw)
      /argo/podmetadata from podmetadata (rw)
      /argo/secret/argo-gcs-artifact-secret from argo-gcs-artifact-secret (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rgcfb (ro)
Containers:
  wait:
    Container ID:
    Image:         argoproj/argoexec:v2.10.0-rc5
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:  artifact-passing-directory-sscmx-414621924 (v1:metadata.name)
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /argo/secret/argo-gcs-artifact-secret from argo-gcs-artifact-secret (ro)
      /mainctrfs/C:\message from input-artifacts (rw,path="message")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rgcfb (ro)
      \\.\pipe\docker_engine from docker-sock (rw)
  main:
    Container ID:
    Image:         mcr.microsoft.com/windows/nanoserver:1809
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      cmd
      /c
    Args:
      type C:\message
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rgcfb (ro)
      C:\message from input-artifacts (rw,path="message")
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          \\.\pipe\docker_engine
    HostPathType:
  argo-gcs-artifact-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  argo-gcs-artifact-secret
    Optional:    false
  input-artifacts:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-rgcfb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rgcfb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/os=windows
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/os=windows:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute for 300s

The issue seems to be: common/common.go#L81 workflow/controller/workflowpod.go#L797

plus in a few encounters of that constant in workflow/executor/executor.go.

Not sure if it works when we just not join the filepath with ExecutorMainFilesystemDir on windows?

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 16 (16 by maintainers)

Commits related to this issue

Most upvoted comments

@mweibel Just use /message as a path. This gets automatically mapped to C:\message. I changed the step as following and this works for me without a problem:

# ...
    - name: print-message
      inputs:
        artifacts:
          # unpack the message input artifact
          # and put it at C:\message
          - name: message
            path: "/message"   # <-------------- change the path
      nodeSelector:
        kubernetes.io/os: windows
      container:
        image: mcr.microsoft.com/windows/nanoserver:1809
        command: ["cmd", "/c"]
        args: ["dir C:\\message"]   # <------------- I used dir instead as type gives me an access denied error, you may wanted to use "type C:\\message\\hello_world.txt" which also works fine

I would really appreciate if you would create a PR to document this 😃

+1
lippertmarkus on Aug 7, 2020
Read more comments on GitHub
← argo-workflows: `inputs` or `workflow` is nil when used in sprig expression that has more than 1 parameter (`withParam` loop)
argo-workflows: Invalid node IDs in workflow.status →