buildkit: Invalid layer content in rootless mode

Hello,

We are starting to run buildkit in rootless mode and in some cases, we seem to be having issues with the snapshotter. For instance, with this Dockerfile

FROM golang:latest as builder

# https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/commit/93915e330be4291f209b13dc6df063729bd4d9c4
ENV TAG "93915e330be4291f209b13dc6df063729bd4d9c4"
ENV GOOS "linux"
ENV CGO_ENABLED "0"

WORKDIR src/sigs.k8s.io/sig-storage-local-static-provisioner

RUN git clone https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner.git .  && ls -l vendor/k8s.io/utils/io/ && find . | wc -l
RUN git checkout $TAG && ls -l vendor/k8s.io/utils/io/ && find | wc -l

RUN echo "Checking content" && ls -l vendor/k8s.io/utils/io/ && find | wc -l

We get this with buildkit in rootless mode:

#6 [3/5] RUN git clone https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner.git .  && ls -l vendor/k8s.io/utils/io/ && find . | wc -l
#6 sha256:aff0ee315ab635aa458b8e553f083158f525c06590d39b21212716100211e4fa
#6 0.073 Cloning into '.'...
#6 3.166 total 8
#6 3.166 -rw-r--r-- 1 root root  134 Sep 25 09:58 README.md
#6 3.166 -rw-r--r-- 1 root root 2982 Sep 25 09:58 read.go
#6 3.195 6787
#6 DONE 3.4s

#7 [4/5] RUN git checkout 93915e330be4291f209b13dc6df063729bd4d9c4 && ls -l vendor/k8s.io/utils/io/ && find . | wc -l
#7 sha256:712f0ac1353ca45de25e50eb4e17134c9865712a83f756c893ee1fde0750e0c8
#7 1.068 Note: switching to '93915e330be4291f209b13dc6df063729bd4d9c4'.
#7 1.072 total 4
#7 1.072 -rw-r--r-- 1 root root 1748 Sep 25 09:58 consistentread.go
#7 1.106 5573
#7 DONE 1.3s

#8 [5/5] RUN echo "Checking content" && ls -l vendor/k8s.io/utils/io/ && find . | wc -l
#8 sha256:600ad17eb12cb7bd7a2f3154b8eec4ce13dd60bd40b2cabf4b95320e00fd45df
#8 0.120 Checking content
#8 0.121 total 12
#8 0.121 -rw-r--r-- 1 root root  134 Sep 25 09:58 README.md
#8 0.121 -rw-r--r-- 1 root root 1748 Sep 25 09:58 consistentread.go
#8 0.121 -rw-r--r-- 1 root root 2982 Sep 25 09:58 read.go
#8 0.157 5605
#8 DONE 0.2s

Of course we’d expect content of the layer at step #8 to be the same as the content at the end of step #7

We tested building in non-rootless mode and the problem disappeared. After looking into past issues, we found https://github.com/moby/buildkit/issues/1792 which felt a bit similar, so we tried in rootless mode with --oci-worker-snapshotter=native and the problem also disappeared.

So the problem only happens in rootless mode with overlayfs snapshotter. Here is the configuration where we see this behavior:

• Ubuntu 20.04 with kernel 5.8
• image: buildkit:v0.9.0-rootless
• parameters: --oci-worker-no-process-sandbox
• non-privileged with seccomp and apparmor set to unconfined

Is there a way to diagnose what is failing when we use the overlayfs snapshotter? I could not find the impact of using the native snapshotter instead of the overlayfs one. I assume slower build?

Of course, we are more than happy to perform additional tests to gather more data