buildkit: IO ERROR after enabling BuildKit

We enabled buildkit in our project by adding DOCKER_BUILDKIT=1 to our builds in docker-ce, and for one build we consistently get this error with it enabled (and no errors when buildkit is disabled):

#6       digest: sha256:30da00d34747eb29aac15c9bbd7386f198eea43c10752373b55a253ffd914d73
#6         name: "[2/3] RUN apk add --no-cache curl>=7.61.1-r1 openssh>=7.7_p1-r3 rsync>=3.1.3-r1     && mkdir -p ~/.ssh     && chmod 700 ~/.ssh"
#6      started: 2018-12-10 13:56:21.911458752 +0000 UTC
#6 0.709 fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
#6 0.710 WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: IO ERROR
#6 0.710 fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
#6 0.711 WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: IO ERROR
#6 0.712 ERROR: unsatisfiable constraints:
#6 0.712   curl (missing):
#6 0.712     required by: world[curl>=7.61.1-r1]
#6 0.712   openssh (missing):
#6 0.712     required by: world[openssh>=7.7_p1-r3]
#6 0.712   rsync (missing):
#6 0.712     required by: world[rsync>=3.1.3-r1]
#6    completed: 2018-12-10 13:56:22.825157977 +0000 UTC
#6     duration: 913.699225ms
#6        error: "executor failed running [/bin/sh -c apk add --no-cache 'curl>=7.61.1-r1' 'openssh>=7.7_p1-r3' 'rsync>=3.1.3-r1'     && mkdir -p ~/.ssh     && chmod 700 ~/.ssh]: exit code: 3"

Do you have any ideas why, and how this can be solved ? Can I help anyhow to debug this ?

Thank you in advance !

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 16 (4 by maintainers)

Most upvoted comments

I don’t think it is generic docker issue, more likely a specific alpine image issue. I was able to constantly reproduce it by just trying to install packages(specifically g++/gcc) using the latest docker-ce on updated ubuntu 18.04:

$ docker run -it alpine
 Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
6c40cc604d8e: Already exists 
Digest: sha256:b3dbf31b77fd99d9c08f780ce6f5282aba076d70a513a8be859d8d3a4d0c92b8
Status: Downloaded newer image for alpine:latest
/ # apk add gcc g++
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/13) Installing libgcc (8.2.0-r2)
(2/13) Installing libstdc++ (8.2.0-r2)
(3/13) Installing binutils (2.31.1-r2)
(4/13) Installing gmp (6.1.2-r1)
(5/13) Installing isl (0.18-r0)
(6/13) Installing libgomp (8.2.0-r2)
(7/13) Installing libatomic (8.2.0-r2)
(8/13) Installing mpfr3 (3.1.5-r1)
(9/13) Installing mpc1 (1.0.3-r1)
(10/13) Installing gcc (8.2.0-r2)
ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/8.2.0/lto1: I/O error
ERROR: gcc-8.2.0-r2: IO ERROR
(11/13) Installing musl-dev (1.1.20-r3)
(12/13) Installing libc-dev (0.7.1-r0)
(13/13) Installing g++ (8.2.0-r2)
ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/8.2.0/cc1plus: I/O error
ERROR: g++-8.2.0-r2: IO ERROR
Executing busybox-1.29.3-r10.trigger
2 errors; 30 MiB in 25 packages
/ #

The above problem is a permission issue, those files gives I/O error due to missing write permissions on this folders. I was able to test the theory by giving full write permissions to the /usr folder and trying the exact same command which was successfully due to the change. The problem is obviously in alpine image having improper permissions/ownership of the /usr folder.

sciffer on Feb 27, 2019