buildah: Builds fail when doing `RUN` after `VOLUME` with empty storage driver
Description
After upgrading from podman 3.0.1 to 3.1.0, certain Dockerfile
s/Containerfile
s fail to build.
For some reason this is only the case when not specifying a storage driver in the config.
I could see this only with podman build
, because buildah bud
refuses to do anything without that config set.
Originally I saw this error when building in OBS, where the error message is actually much weirder:
[ 80s] STEP 23: RUN obs-docker-support --uninstall
[ 82s] time="2021-04-16T04:30:29Z" level=error msg="The storage 'driver' option must be set in /etc/containers/storage.conf, guarantee proper operation."
[ 82s] [ 70.671121] overlayfs: unrecognized mount option " guarantee proper operation."
[ 82s] [ 70.671121] /srv/www/htdocs" or missing value
[ 82s] error running container: error creating container for [/bin/sh -c obs-docker-support --uninstall]: time="2021-04-16T04:30:30Z" level=error msg="container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:60: mounting \"/var/lib/containers/storage/overlay-containers/0aef943f1188d994fa4ba7beb20e87c3ceb5efb2ab832bfd9babd8309eaef1ac/userdata/overlay/281018080/merge\" to rootfs at \"/var/tmp/buildah132004737/mnt/rootfs/time=\\\"2021-04-16T04:30:27Z\\\" level=error msg=\\\"The storage 'driver' option must be set in /etc/containers/storage.conf, guarantee proper operation.\\\"\\n/srv/www/htdocs\" caused: invalid argument"
[ 82s] : exit status 1
[ 82s] time="2021-04-16T04:30:30Z" level=error msg="unable to write build event: \"write unixgram @00003->/run/systemd/journal/socket: sendmsg: no such file or directory\""
[ 82s] Error: error building at STEP "RUN obs-docker-support --uninstall": error while running runtime: exit status 1
Which looks like it somehow passed part of the error message into overlayfs mount options? I’m not actually sure whether the issue I can reproduce locally is even the same.
Edit: I should clarify that I removed that option from my local storage.conf
to reproduce the OBS problem
Steps to reproduce the issue:
f241:/tmp # cat Dockerfile
FROM scratch
VOLUME /srv/www/htdocs
RUN nothing
f241:/tmp # podman build -t h .
ERRO[0000] The storage 'driver' option must be set in /etc/containers/storage.conf, guarantee proper operation.
STEP 1: FROM scratch
STEP 2: VOLUME /srv/www/htdocs
--> Using cache 25a0b9b68c12848bb955480a42776572bf1e1008d8bfeeded7bb5b1189ee0a3b
--> 25a0b9b68c1
STEP 3: RUN nothing
Error: error building at STEP "RUN nothing": error resolving mountpoints for container "5e515df94cc7b8f151170e49593d34b0c9c8954d7371cfc727e165b25858571c": stat /var/lib/containers/storage/overlay/ac7d6fa150bfaff50246d797ffdcc861807604570a9a0f0088182115df9ef9e0/merged/time="2021-04-14T17:17:16Z" level=error msg="The storage 'driver' option must be set in /etc/containers/storage.conf, guarantee proper operation."
/srv/www/htdocs: no such file or directory
Output of rpm -q buildah
or apt list buildah
:
buildah-1.20.0-1.1.x86_64
Output of buildah version
:
Version: 1.20.0
Go Version: go1.15.10
Image Spec: 1.0.1-dev
Runtime Spec: 1.0.2-dev
CNI Spec: 0.4.0
libcni Version: v0.8.1
image Version: 5.10.5
Git Commit: unknown
Built: Mon Mar 29 00:00:00 2021
OS/Arch: linux/amd64
Output of podman version
if reporting a podman build
issue:
Version: 3.1.0
API Version: 3.1.0
Go Version: go1.13.15
Built: Fri Apr 9 00:00:00 2021
OS/Arch: linux/amd64
Output of cat /etc/*release
:
NAME="openSUSE MicroOS"
# VERSION="20210408"
ID="opensuse-microos"
ID_LIKE="suse opensuse opensuse-tumbleweed"
VERSION_ID="20210408"
PRETTY_NAME="openSUSE MicroOS"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:microos:20210408"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:MicroOS"
LOGO="distributor-logo"
Output of uname -a
:
Linux f241 5.11.11-1-default #1 SMP Tue Mar 30 17:57:52 UTC 2021 (dbc4a02) x86_64 x86_64 x86_64 GNU/Linux
Output of cat /etc/containers/storage.conf
:
# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]
# Default Storage Driver
driver = ""
# Temporary storage location
runroot = "/var/run/containers/storage"
# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"
[storage.options]
# Storage options to be passed to underlying storage drivers
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]
# Size is used to set a maximum size of the container image. Only supported by
# certain container storage drivers.
size = ""
# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"
# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
# override_kernel_check = "false"
# mountopt specifies comma separated list of extra mount options
# mountopt = "nodev"
# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs. Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536
# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file. Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"
# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""
# Set to skip a PRIVATE bind mount on the storage home directory. Only supported by
# certain container storage drivers
# skip_mount_home = "false"
[storage.options.thinpool]
# Storage Options for thinpool
# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"
# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"
# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"
# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"
# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""
# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"
# fs specifies the filesystem type to use for the base device.
# fs="xfs"
# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"
# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"
# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""
# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"
# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"
# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (11 by maintainers)
Commits related to this issue
- Update podman to version 3.0.1 / rev 78 https://build.opensuse.org/package/rdiff/openSUSE:Factory/podman?linkrev=base&rev=78 by user dimstar_suse https://github.com/containers/buildah/issues/3153 — committed to bmwiedemann/openSUSE by deleted user 3 years ago
- Update libcontainers-common to version 20210112 / rev 35 via SR 886688 https://build.opensuse.org/request/show/886688 by user RBrownSUSE + dimstar_suse - Force overlay as default storage driver if sy... — committed to bmwiedemann/openSUSE by deleted user 3 years ago
Ok I have figured this out, the issue is the change to copier (Internal to buildah) is interpreting the output to stdout/stderr and is causing a failure.
@nalind @vrothberg
I added a fmt.Println(“DAN WAS HERE”) to containers/storage and this blows up
./bin/podman build --no-cache /tmp/ DAN WAS HERE STEP 1: FROM alpine STEP 2: RUN mkdir -p /mnt Error: error building at STEP “RUN mkdir -p /mnt”: error decoding response from copier subprocess: invalid character ‘D’ looking for beginning of value