kubeflow: Argo (&Pipelines) not working well with microk8s.

CC @elmarhaussmann

Microk8s versions tested: 1.12.4 and 1.13.2. Kubeflow versions tested: 0.3.5, 0.4.1 Microk8s was installed with snap using the guide: https://www.kubeflow.org/docs/started/getting-started-multipass/, here, we tried with and without multipass, both to same affect.

Running anything with outputs will fail on argo/kfp and in the wait container you’ll see something like this:

time="2019-01-29T23:48:19Z" level=error msg="`[sh -c docker cp -a 3b16f4eb7b1b3c53b3a5321f25841a88af8080b676af6c656545dc67c34db79d:/tmp/output - | tar -ax -O]` stderr:\nError: No such container:path: 3b16f4eb7b1b3c53b3a5321f25841a88af8080b676af6c656545dc67c34db79d:/tmp/output\ntar: This does not look like a tar archive\ntar: Exiting with failure status due to previous errors\n"

The docker here, refers to the docker in the host (or vm) instead of what microk8s.docker is using (/var/snap/microk8s/common/var/lib/docker):

Expected

Argo uses docker from microk8’s, not the host/vm.

Argo source volume mount definitions

These are hardcoded: https://github.com/argoproj/argo/blob/master/workflow/controller/workflowpod.go#L56 :

    ...
    Path: "/var/run/docker.sock",
    ...

Mounts of the wait container

    volumeMounts:
    - mountPath: /argo/podmetadata
      name: podmetadata
    - mountPath: /var/lib/docker
      name: docker-lib
      readOnly: true
    - mountPath: /var/run/docker.sock
      name: docker-sock
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: pipeline-runner-token-wgt4c
      readOnly: true

Dirty fix

To align these, in your host or VM (when using multipass) you can link to what argo’s mounting eg

sudo ln -s /var/snap/microk8s/common/var/lib/docker /var/lib/docker
sudo ln -s /var/snap/microk8s/current/docker.sock /var/run/docker.sock