moby: runc run failed: stat: no such file or directory with warm BuildKit cache after 23.0 upgrade

Description

I have some images built with BUILDKIT_INLINE_CACHE=1. Modify the Dockerfile and rebuild it.

docker build \
--tag gcr.io/fuzzbench/base-image \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--file docker/base-image/Dockerfile \
.
[+] Building 0.7s (7/13)

 => [internal] load .dockerignore                                                                                    0.2s
 => => transferring context: 118B                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                                 0.3s
 => => transferring dockerfile: 2.54kB                                                                               0.0s
 => [internal] load metadata for docker.io/library/ubuntu:focal                                                      0.0s
 => [1/9] FROM docker.io/library/ubuntu:focal                                                                        0.0s
 => [internal] load build context                                                                                    0.2s
 => => transferring context: 38B                                                                                     0.0s
 => CACHED [2/9] RUN apt-get update     && apt-get install -y ca-certificates     && sed -i "s@http://.*archive.ubu  0.0s
 => ERROR [3/9] RUN ls                                                                                               0.4s
------

 > [3/9] RUN ls:
#0 0.341 runc run failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory

Images without BUILDKIT_INLINE_CACHE=1 seem fine.

It seems that it is not related to https://github.com/moby/moby/issues/44918

Reproduce

Build some images with BUILDKIT_INLINE_CACHE=1 in 20.10 and upgrade to 23.0.0. Modify the Dockerfile and rebuild it.

Expected behavior

To work as it worked before 23.0.0.

docker version

Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:49:08 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.0
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       d7573ab
  Built:            Wed Feb  1 17:49:08 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 56
  Running: 1
  Paused: 0
  Stopped: 55
 Images: 723
 Server Version: 23.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.13.0-37-generic
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 96
 Total Memory: 502.6GiB
 Name: cocoa
 ID: 6AXB:OEBV:VSR6:AYW3:N23Z:4LFF:4BRU:JUKX:4KOH:ROLW:OEQG:CP54
 Docker Root Dir: /mnt/data/docker
 Debug Mode: false
 HTTP Proxy: http://xxx
 HTTPS Proxy: http://xxx
 No Proxy: localhost,127.0.0.1
 Username: xxx
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 15 (10 by maintainers)

Most upvoted comments

Some more updates: I don’t think this is based on base image like postulated in https://github.com/moby/moby/issues/44947; instead it appears to be based on a hot/cold cache.

This can be mitigated by dropping the cache: docker builder prune -a. A regular docker builder prune does not suffice.

It’s worth noting that it appears to me that BUILDKIT_INLINE_CACHE=1 is not necessary here – instead, it looks like only some base images trigger this behavior.