buildkit: Rootless mode doesn't work on Google Container-Optimized OS kernel (CONFIG_SECURITY_CHROMIUMOS_NO_UNPRIVILEGED_UNSAFE_MOUNTS?)

On GCOS kernel

Yes, latest GKE with cos_containerd image. I got fully functional rootless buildkit with resource definitions from examples/kubernetes with only added a emptyDir/hostPath volumeMount for /home/user/.local/share/buildkit.

Isn’t this VOLUME working by default? 🤔

Yes, and that’s the problem - default volumes are mounted with nosuid,nodev flags, which cause Permission denied error trying to remount this volume without this flags. See details in an excellent investigation from @bcressey there in linked bottlerocket issue.

Not only the issue in snapshotter

$ git diff
diff --git a/vendor/github.com/containerd/containerd/mount/mount_linux.go b/vendor/github.com/containerd/containerd/mount/mount_linux.go
index a7edd455..526640be 100644
--- a/vendor/github.com/containerd/containerd/mount/mount_linux.go
+++ b/vendor/github.com/containerd/containerd/mount/mount_linux.go
@@ -93,7 +93,10 @@ func (m *Mount) Mount(target string) error {
        const broflags = unix.MS_BIND | unix.MS_RDONLY
        if oflags&broflags == broflags {
                // Remount the bind to apply read only.
-               return unix.Mount("", target, "", uintptr(oflags|unix.MS_REMOUNT), "")
+               unix.Mount("", target, "", uintptr(oflags|unix.MS_REMOUNT), "")
+               // DO-NOT-MERGE:
+               // ignore err here to avoid hitting https://github.com/moby/buildkit/issues/879#issuecomment-473396544
+               // How can we ensure target to be read-only?
        }
        return nil
 }

$ buildctl --addr=kube-pod://buildkitd build --frontend dockerfile.v0 --local dockerfile=. --local context=
.
[+] Building 6.1s (5/6)                                                                                                         
 => [internal] load build definition from Dockerfile                                                                       0.2s
 => => transferring dockerfile: 109B                                                                                       0.2s
 => [internal] load .dockerignore                                                                                          0.2s
 => => transferring context: 2B                                                                                            0.1s
 => [internal] load metadata for docker.io/library/alpine:latest                                                           3.3s
 => [1/3] FROM docker.io/library/alpine@sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d            2.1s
 => => resolve docker.io/library/alpine@sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d            0.0s
 => => sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d 1.64kB / 1.64kB                             0.0s
 => => sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45 528B / 528B                                 0.0s
 => => sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 2.80MB / 2.80MB                             1.2s
 => => sha256:e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a 1.51kB / 1.51kB                             0.0s
 => => unpacking docker.io/library/alpine@sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d          0.1s
 => ERROR [2/3] RUN apk add --no-cache figlet                                                                              0.1s
------
 > [2/3] RUN apk add --no-cache figlet:
#5 0.084 container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.g
o:58: mounting \\\"/home/user/.local/share/buildkit/runc-native/executor/resolv.conf\\\" to rootfs \\\"/home/user/.local/share/b
uildkit/runc-native/executor/c9qbj5rmvwnjixos72ek7k7ko/rootfs\\\" at \\\"/home/user/.local/share/buildkit/runc-native/executor/c
9qbj5rmvwnjixos72ek7k7ko/rootfs/etc/resolv.conf\\\" caused \\\"operation not permitted\\\"\""
------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c apk add --no-cache figlet]: buildki
t-runc did not terminate successfully

@ei-grad On GCOS kernel? 👀

Using an idea from https://github.com/bottlerocket-os/bottlerocket/issues/1934 I added an emptyDir volume to /home/user/.local/share/buildkit and it worked.

v0.4.0-rootless (both overlay and native; both w/ and w/o privileged) works with GKE Ubuntu nodes (kernel 4.15.0-1026-gcp #27-Ubuntu, kube v1.11.7-gke.4, Ubuntu 18.04.1, docker://17.3.2).

Seems an issue on Google COS.

No, even v0.3.0-rootless w/ securityContext: privileged does not work now.

This is rather likely to be a regression in GKE, although I don’t have any evidence that v0.3.0-rootless had been working on GKE.