moby: rootless docker in kubernetes: "getting the final child's pid from pipe caused \"EOF\"": unknown"

I am trying to run rootless docker-in-docker deployed on Kubernetes, where the docker daemon is deployed using this manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: rootless
  name: rootless
spec:
  replicas: 1
  selector:
    matchLabels:
      run: rootless
  template:
    metadata:
      labels:
        run: rootless
    spec:
      containers:
      - image: docker:19.03.1-dind-rootless
        name: rootless
        command:
        - sh
        - -c
        - dockerd-entrypoint.sh --experimental --storage-driver=vfs
        securityContext:
          runAsUser: 1000
          allowPrivilegeEscalation: true
          privileged: true
        resources: {}
status: {}

The daemon seems to start up fine but kubectl exec’ing into the pod, setting the DOCKER_HOST and attempting to run a container seems to break

kubctl create -f rootless.yml
kubectl exec -it rootless-7c94dcc5d7-gplvm sh
/ $ export DOCKER_HOST=unix:///run/user/1000/docker.sock
/ $ docker run --rm -it  busybox sh
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:303: getting the final child's pid from pipe caused \"EOF\"": unknown.
/ $ command terminated with exit code 125

The dockerd daemon logs are collected in this gist

https://gist.github.com/lukasheinrich/e23c00240afed83066978dab8c354f52#file-logs

kubectl version gives

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"archive", BuildDate:"2019-06-18T20:40:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

cc @AkihiroSuda @rochaporto

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 7
Comments: 41 (22 by maintainers)

Most upvoted comments

Great thread, thanks @GreeFine & all. Have the same need (rootless dind on k8s) and solved the issue for me as well.

In summary: Preparation

wget https://github.com/containers/crun/releases/download/0.13/crun-0.13-static-x86_64 
# verify as needed
chmod 755 crun-0.13-static-x86_64

Dockerfile

FROM docker:19.03.7-dind-rootless
RUN mkdir -p /usr/local/bin/
COPY crun-0.13-static-x86_64 /usr/local/bin/crun

build & push.

k8s configuration as per above, personally feeling more comfortable with

securityContext:
  privileged: true
args: ["--experimental", "--default-runtime", "crun", "--add-runtime", "crun=/usr/local/bin/crun"]

jappenzesr on Mar 13, 2020

@AkihiroSuda working for me with the last build of crun ! Kubernetes DinD Rootless 😄

The working crun version:

crun --version
crun version 0.12.2.1.29-0402
commit: 0402a428c1f2c92be021e77d7b735360bae5f77e
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

GreeFine on Feb 27, 2020

Great thread, thanks @GreeFine & all. Have the same need (rootless dind on k8s) and solved the issue for me as well.

In summary: Preparation
wget https://github.com/containers/crun/releases/download/0.13/crun-0.13-static-x86_64 
# verify as needed
chmod 755 crun-0.13-static-x86_64
Dockerfile
FROM docker:19.03.7-dind-rootless
RUN mkdir -p /usr/local/bin/
COPY crun-0.13-static-x86_64 /usr/local/bin/crun
build & push.

k8s configuration as per above, personally feeling more comfortable with
securityContext:
  privileged: true
args: ["--experimental", "--default-runtime", "crun", "--add-runtime", "crun=/usr/local/bin/crun"]

I just did but this is what he shows me when trying to run some container:

docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/a01fa450e7243e0cf24ce9421d32cb17f9aa3c18cfdd53b31bed936715d3cf18/log.json: no such file or directory): fork/exec /usr/local/bin/crun: permission denied: <nil>: unknown.

Tried to run this command:

docker run -it --rm curlimages/curl:latest sh

Edit: Nevermind, I had to chmod +x /usr/local/bin/crun

FROM docker:20.10.3-dind-rootless
USER 0
RUN mkdir -p /usr/local/bin/
COPY crun-0.13-static-x86_64 /usr/local/bin/crun
RUN chmod +x /usr/local/bin/crun
USER 1000

kfirfer on Feb 6, 2021

Can you open an issue at https://github.com/containers/crun/issues for crun issue

AkihiroSuda on Feb 27, 2020

yes, containers/crun@ac173ad2c6d752814e4b7970612c24dc2db35117 works

AkihiroSuda on Nov 15, 2019

Not likely, sorry.

Maybe --runtime=crun works or prints more helpful error?

AkihiroSuda on Nov 15, 2019