moby: runc run failed: stat: no such file or directory with warm BuildKit cache after 23.0 upgrade
Description
I have some images built with BUILDKIT_INLINE_CACHE=1
. Modify the Dockerfile and rebuild it.
docker build \
--tag gcr.io/fuzzbench/base-image \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--file docker/base-image/Dockerfile \
.
[+] Building 0.7s (7/13)
=> [internal] load .dockerignore 0.2s
=> => transferring context: 118B 0.0s
=> [internal] load build definition from Dockerfile 0.3s
=> => transferring dockerfile: 2.54kB 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:focal 0.0s
=> [1/9] FROM docker.io/library/ubuntu:focal 0.0s
=> [internal] load build context 0.2s
=> => transferring context: 38B 0.0s
=> CACHED [2/9] RUN apt-get update && apt-get install -y ca-certificates && sed -i "s@http://.*archive.ubu 0.0s
=> ERROR [3/9] RUN ls 0.4s
------
> [3/9] RUN ls:
#0 0.341 runc run failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory
Images without BUILDKIT_INLINE_CACHE=1
seem fine.
It seems that it is not related to https://github.com/moby/moby/issues/44918
Reproduce
Build some images with BUILDKIT_INLINE_CACHE=1
in 20.10 and upgrade to 23.0.0. Modify the Dockerfile and rebuild it.
Expected behavior
To work as it worked before 23.0.0.
docker version
Client: Docker Engine - Community
Version: 23.0.0
API version: 1.42
Go version: go1.19.5
Git commit: e92dd87
Built: Wed Feb 1 17:49:08 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 23.0.0
API version: 1.42 (minimum version 1.12)
Go version: go1.19.5
Git commit: d7573ab
Built: Wed Feb 1 17:49:08 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.16
GitCommit: 31aa4358a36870b21a992d3ad2bef29e1d693bec
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.15.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Containers: 56
Running: 1
Paused: 0
Stopped: 55
Images: 723
Server Version: 23.0.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 5.13.0-37-generic
Operating System: Ubuntu 20.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 96
Total Memory: 502.6GiB
Name: cocoa
ID: 6AXB:OEBV:VSR6:AYW3:N23Z:4LFF:4BRU:JUKX:4KOH:ROLW:OEQG:CP54
Docker Root Dir: /mnt/data/docker
Debug Mode: false
HTTP Proxy: http://xxx
HTTPS Proxy: http://xxx
No Proxy: localhost,127.0.0.1
Username: xxx
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional Info
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 15 (10 by maintainers)
Fixed in https://github.com/moby/moby/pull/44959
Some more updates: I don’t think this is based on base image like postulated in https://github.com/moby/moby/issues/44947; instead it appears to be based on a hot/cold cache.
This can be mitigated by dropping the cache:
docker builder prune -a
. A regulardocker builder prune
does not suffice.It’s worth noting that it appears to me that
BUILDKIT_INLINE_CACHE=1
is not necessary here – instead, it looks like only some base images trigger this behavior.