buildah: buildah bud failing on COPY where docker build works

Description

I am trying to create a container with the following Docker file:

FROM websphere-liberty:webProfile7
ARG FILE
ARG DB_DRIVER_JAR
USER root
RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
RUN chown -R default:root /opt/ibm/jars /opt/ssfs/runtime
USER default
COPY --chown=default:root $FILE /config/dropins
ADD --chown=default:root $DB_DRIVER_JAR /opt/ibm/jars

and the following command:

buildah bud -t mycluster.icp:8500/ibmcom/om-app-svt:ubi1 --build-arg FILE=ear.dir --build-arg DB_DRIVER_JAR=db2jcc4.jar -f /opt/ssfs/runtime/docker/docker-app/Dockerfile.om-app-os /opt/ssfs/runtime/docker/docker-app

I have also tried with podman build -t and that also fails.

Steps to reproduce the issue:

1. Using the Docker file above, try to build a container with buildah bud or podman build. Note the failure.
2. Try the same with docker build, note the success.

Describe the results you received:

The output from the above command is below. I have tried both with and without cache and it doesn’t matter.

STEP 1: FROM websphere-liberty:webProfile7
STEP 2: ARG FILE
STEP 3: ARG DB_DRIVER_JAR
STEP 4: USER root
STEP 5: RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
STEP 6: RUN chown -R default:root /opt/ibm/jars /opt/ssfs/runtime
STEP 7: USER default
STEP 8: COPY --chown=default:root $FILE /config/dropins
error building at step {Env:[PATH=/opt/ibm/wlp/bin:/opt/ibm/helpers/build:/opt/ibm/java/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JAVA_VERSION=1.8.0_sr5fp31 JAVA_HOME=/opt/ibm/java/jre IBM_JAVA_OPTIONS=-XX:+UseContainerSupport LIBERTY_VERSION=19.0.0_03 LOG_DIR=/logs WLP_OUTPUT_DIR=/opt/ibm/wlp/output RANDFILE=/tmp/.rnd JVM_ARGS=-Xshareclasses:name=liberty,nonfatal,cacheDir=/output/.classCache/ KEYSTORE_REQUIRED=true FILE=ear.dir DB_DRIVER_JAR=db2jcc4.jar] Command:copy Args:[ear.dir /config/dropins] Flags:[--chown=default:root] Attrs:map[] Message:COPY --chown=default:root $FILE /config/dropins Original:COPY --chown=default:root $FILE /config/dropins}: error creating directory "/var/lib/containers/storage/overlay/9119a7a96ed918dd6d3ad94a785cda5c1bc06b7e1b23f8ddb15ae6c9269c92d1/merged/config/dropins": chown /var/lib/containers/storage/overlay/9119a7a96ed918dd6d3ad94a785cda5c1bc06b7e1b23f8ddb15ae6c9269c92d1/merged/config/dropins: no such file or directory
[root@dub385002 docker]# echo ${COMMAND} -t mycluster.icp:8500/ibmcom/om-app-svt:ubi1 --build-arg FILE=ear.dir --build-arg DB_DRIVER_JAR=db2jcc4.jar -f ${ROOT_FOLDER}/docker-app/Dockerfile.om-app-os ${ROOT_FOLDER}/docker-app
buildah bud -t mycluster.icp:8500/ibmcom/om-app-svt:ubi1 --build-arg FILE=ear.dir --build-arg DB_DRIVER_JAR=db2jcc4.jar -f /opt/ssfs/runtime/docker/docker-app/Dockerfile.om-app-os /opt/ssfs/runtime/docker/docker-app

Describe the results you expected:

When I run the same command but with docker build, it works and I see this:

Sending build context to Docker daemon  718.4MB
Step 1/8 : FROM websphere-liberty:webProfile7
webProfile7: Pulling from library/websphere-liberty
Digest: sha256:ac7d913863021465f4a4370bfbff79eb47cb2f2dbd6efda4447637df31cb6a46
Status: Image is up to date for websphere-liberty:webProfile7
 ---> e6995314a475
Step 2/8 : ARG FILE
 ---> Running in ac5a4d00d3ea
Removing intermediate container ac5a4d00d3ea
 ---> 65488246bf3a
Step 3/8 : ARG DB_DRIVER_JAR
 ---> Running in a95d1f6884d8
Removing intermediate container a95d1f6884d8
 ---> b565260297df
Step 4/8 : USER root
 ---> Running in d14f6f30839e
Removing intermediate container d14f6f30839e
 ---> f158eeb65055
Step 5/8 : RUN mkdir -p /opt/ibm/jars && mkdir -p /opt/ssfs/runtime && chown -R default:root /opt/ibm/jars && chown -R default:root /opt/ssfs/runtime
 ---> Running in be66c3b5ea55
Removing intermediate container be66c3b5ea55
 ---> 1d6402cb2d0f
Step 6/8 : USER default
 ---> Running in 63f06e9ffda6
Removing intermediate container 63f06e9ffda6
 ---> 2b21ed712685
Step 7/8 : COPY --chown=default:root $FILE /config/dropins
 ---> b70a46bb60ea
Step 8/8 : ADD --chown=default:root $DB_DRIVER_JAR /opt/ibm/jars
 ---> 58c6e94fb346
Successfully built 58c6e94fb346
Successfully tagged mycluster.icp:8500/ibmcom/om-app-svt:test1

Output of rpm -q buildah or apt list buildah:

buildah-1.5-2.gite94b4f9.el7.x86_64

Output of buildah version:

Version:         1.5
Go Version:      go1.10.3
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-alpha1
Git Commit:      e94b4f9
Built:           Mon Dec 10 12:15:42 2018
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:       0.12.1.2
Go Version:    go1.10.3
OS/Arch:       linux/amd64

Output of cat /etc/*release:

LSB_VERSION=core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Red Hat Enterprise Linux Server release 7.6 (Maipo)

Output of uname -a:

Linux hostname.com 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""

# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"

# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"