buildkit: BuildKit builds (via Docker) are broken if /etc/hosts or /etc/resolv.conf is replaced

My Dockerfile:

from archlinux:20191105
run pacman -Sy --noconfirm filesystem

The filesystem package contains most of the base system files, as can be seen at https://www.archlinux.org/packages/core/x86_64/filesystem/. If it’s upgraded, it tries to write /etc/hosts and /etc/resolv.conf, which leads to errors with BuildKit (“classical” Docker build is fine):

$ DOCKER_BUILDKIT=0 docker build .
Sending build context to Docker daemon  159.8MB
Step 1/2 : from archlinux:20191105
20191105: Pulling from library/archlinux
Digest: sha256:3fcb6f0c3a1266b579f7d5a89cbb66db1530e8dd533794b9c9588b630255b754
Status: Downloaded newer image for archlinux:20191105
 ---> 5ee688d008f4
Step 2/2 : run pacman -Sy --noconfirm filesystem
 ---> Running in f092dfd264b8
:: Synchronizing package databases...
downloading core.db...
downloading extra.db...
downloading community.db...
resolving dependencies...
looking for conflicting packages...

Packages (1) filesystem-2019.10-2

Total Download Size:   0.03 MiB
Total Installed Size:  0.04 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
downloading filesystem-2019.10-2-x86_64.pkg.tar.xz...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
upgrading filesystem...
warning: directory permissions differ on /srv/ftp/
filesystem: 755  package: 555
:: Running post-transaction hooks...
(1/4) Creating system user accounts...
(2/4) Applying kernel sysctl settings...
  Skipped: Current root is not booted.
(3/4) Creating temporary files...
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
(4/4) Arming ConditionNeedsUpdate...
Removing intermediate container f092dfd264b8
 ---> a7fa3f88205d
Successfully built a7fa3f88205d

docker build .
[+] Building 16.4s (5/5) FINISHED                                                                                                    
 => [internal] load build definition from Dockerfile                                                                            0.1s
 => => transferring dockerfile: 105B                                                                                            0.0s
 => [internal] load .dockerignore                                                                                               0.1s
 => => transferring context: 2B                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/archlinux:20191105                                                           0.5s
 => CACHED [1/2] FROM docker.io/library/archlinux:20191105@sha256:3fcb6f0c3a1266b579f7d5a89cbb66db1530e8dd533794b9c9588b630255  0.0s
 => ERROR [2/2] RUN pacman -Sy --noconfirm filesystem                                                                          15.7s
------                                                                                                                               
 > [2/2] RUN pacman -Sy --noconfirm filesystem:                                                                                      
#5 0.578 :: Synchronizing package databases...                                                                                       
#5 1.578 downloading core.db...                                                                                                      
#5 3.764 downloading extra.db...                                                                                                     
#5 7.726 downloading community.db...                                                                                                 
#5 14.71 resolving dependencies...
#5 14.72 looking for conflicting packages...
#5 14.72 
#5 14.72 Packages (1) filesystem-2019.10-2
#5 14.72 
#5 14.72 Total Download Size:   0.03 MiB
#5 14.72 Total Installed Size:  0.04 MiB
#5 14.72 Net Upgrade Size:      0.00 MiB
#5 14.72 
#5 14.72 :: Proceed with installation? [Y/n] 
#5 14.72 :: Retrieving packages...
#5 15.30 downloading filesystem-2019.10-2-x86_64.pkg.tar.xz...
#5 15.51 checking keyring...
#5 15.56 checking package integrity...
#5 15.59 loading package files...
#5 15.59 checking for file conflicts...
#5 15.59 checking available disk space...
#5 15.59 error: Partition /etc/resolv.conf is mounted read only
#5 15.59 error: Partition /etc/hosts is mounted read only
#5 15.59 error: not enough free disk space
#5 15.59 error: failed to commit transaction (not enough free disk space)
#5 15.60 Errors occurred, no packages were upgraded.
------

About this issue

Original URL
State: open
Created 5 years ago
Reactions: 8
Comments: 19 (6 by maintainers)

Links to this issue

Docker build freezes on command - Stack Overflow

Commits related to this issue

BAU: Set nginx-tls image /etc/hosts owner in entrypoint Since we started building the nginx-tls image in the pipeline with the vito/oci-build-task image, the `build-nginx-tls` Concourse job has been ... — committed to alphagov/verify-infrastructure by Wynndow 3 years ago
Fix docker build on BuildKit now default on GHA Runners use Docker v24 now, which uses BuildKit by default. This broke the build when reinstalling `filesystem` due to https://github.com/moby/buildkit... — committed to ian-h-chamberlain/holo-docker by ian-h-chamberlain 9 months ago

Most upvoted comments

Just for anybody who finds this useful, i stumbled across the OP issue when using GitHub Actions and attempting to build a Arch Linux Docker image ‘from scratch’ using the buildx step:-

        uses: docker/setup-buildx-action@v1

if i then used the following ‘uses’ action then it blew up with the same read only issue for /etc/hostsand /etc/resolv.conf as OP.

        uses: docker/build-push-action@v2

the solution for me is simply to ignore the filesystem package during upgrade by doing the following:-

sed -i -e 's~#IgnorePkg.*~IgnorePkg = filesystem~g' '/etc/pacman.conf'

pacman then happily ignores the upgrade of filesystem and completes, this is obviously not ideal but it works and im happy enough to share this as a workaround for now until when/if the filesystem package stops attempting to modify the two files mentioned above.

binhex on Jan 29, 2021

A lot of posts here mention --add-host as the proper way to change /etc/hosts, but what about /etc/resolv.conf? My organization uses different nameservers on the host than we do during build, so what we’d probably use ideally is the --dns option that docker run has. However there appears to be no such option for docker build, so for now using BuildKit doesn’t seem to be an option for us.

Is there any way to explicitly set nameservers for a single docker build command?

ian-h-chamberlain on Sep 15, 2022

@binhex correct; BuildKit currently mounts these read-only;

With BuildKit:

DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain -<<EOF
FROM busybox
RUN mount | grep /etc
RUN cat /etc/hosts
EOF

#5 [2/2] RUN mount | grep /etc
#5 sha256:30dbb0d55d6b8a70aaf13d4655662c2ee23283d7fcbccceb70ff0a011adef4f1
#5 0.277 /dev/vda1 on /etc/resolv.conf type ext4 (ro,nosuid,nodev,noexec,relatime)
#5 0.277 /dev/vda1 on /etc/hosts type ext4 (ro,nosuid,nodev,noexec,relatime)
#5 DONE 0.4s

Without BuildKit:

DOCKER_BUILDKIT=0 docker build --no-cache --progress=plain -<<EOF
FROM busybox
RUN mount | grep /etc
EOF

Step 2/2 : RUN mount | grep /etc
 ---> Running in fbe072913354
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime)

If your intent is to install the filesystem package to build a custom/more recent “base image” for arch linux, please refer to https://docs.docker.com/develop/develop-images/baseimages/

thaJeztah on Jan 29, 2021

@thaJeztah sorry to labour the issue, but am i right in saying that there is still no workaround for the original issue, where installing arch linux package ‘filesystem’ causes a read only error due to bind mounts to the host?, i have gone through the links above but cannot see a way around this issue that relate to installation of a package that needs write access to /etc/hosts and /etc/resolv.conf please correct me if i’m wrong here.

binhex on Jan 27, 2021

I don’t think there are plans to “fix” this. As I explained before these files are not modifiable in containers and managed by the runtime. Changing them leads to confusing behavior and broken images. --add-host works as intended also in BuildKit.

It’s a tricky one; I agree that for users that want to customise during build, --add-host should be the recommendation; I guess (reading the discussion above) the problem, is that some packages (for whatever reason) appear to want to have these files writable. While it’s worth looking why they want to do so, generally this would be out of control for the user that performs the build.

I saw https://github.com/moby/moby/issues/11950 was linked above; probably relevant here are https://github.com/moby/moby/issues/2267, and https://github.com/moby/moby/pull/5129, which (although not “persisting”), makes both of these writable by mounting the files.

I do share the concerns about performance though, so not sure what’s best.

thaJeztah on Jan 27, 2021