buildah: buildah bud fails at commit stage for layered images from private registry with internal error
Description
Whenever I’m building a layered image from my private registry (OpenShift Origin 1.5 registry), it breaks at the end with weird messages like this:
STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@ac887cd1c3698c3247f983681a24daeb2e35e9497f0a9e77fc120a310c43cf7b
error copying layers and metadata: Error initializing source containers-storage:centos-buildah-working-container: internal error: history lists 5 non-empty layers, but we have 10 layers on disk
Steps to reproduce the issue:
- Set up Origin 1.5
- Build a container in it
- Use
buildah budto build a container layered on top of that one.
Example “base” container:
FROM centos:7
RUN yum upgrade -y && yum -y install buildah && rm -rf /var/cache/yum/*
CMD /bin/bash
Example “layered” container:
FROM registry.localhost.localdomain/test/centos-buildah
RUN yum -y install docker && rm -rf /var/cache/yum/*
CMD /bin/bash
Describe the results you received: It fails to build due to internal error with history of layers, as noted in the description.
Describe the results you expected: I expected the commit stage to succeed and the image to build.
Output of rpm -q buildah:
On my CentOS 7 box:
buildah-0.11-3.git49095a8.el7.x86_64
On my Fedora 28 box:
buildah-0.16-1.git82529cd.fc28.x86_64
Output of buildah version:
On my CentOS 7 box:
Version: 0.11
Go Version: go1.8.3
Image Spec: 1.0.0
Runtime Spec: 1.0.0
Git Commit: 49095a8
Built: Wed Mar 7 17:03:34 2018
OS/Arch: linux/amd64
On my Fedora 28 box:
Version: 0.16
Go Version: go1.10
Image Spec: 1.0.0
Runtime Spec: 1.0.0
Git Commit: 82529cd
Built: Wed Apr 4 10:22:20 2018
OS/Arch: linux/amd64
Output of cat /etc/*release:
CentOS 7 box:
CentOS Linux release 7.4.1708 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.4.1708 (Core)
CentOS Linux release 7.4.1708 (Core)
Fedora 28 box:
Fedora release 28 (Twenty Eight)
NAME=Fedora
VERSION="28 (Twenty Eight)"
ID=fedora
VERSION_ID=28
PLATFORM_ID="platform:f28"
PRETTY_NAME="Fedora 28 (Twenty Eight)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:28"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=28
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=28
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
Fedora release 28 (Twenty Eight)
Fedora release 28 (Twenty Eight)
Output of uname -a:
CentOS 7 box:
Linux localhost 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Fedora 28 box:
Linux localhost 4.16.4-300.fc28.x86_64 #1 SMP Tue Apr 24 22:15:13 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Output of cat /etc/containers/storage.conf:
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]
# Default Storage Driver
driver = "overlay"
# Temporary storage location
runroot = "/var/run/containers/storage"
# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"
[storage.options]
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]
# Size is used to set a maximum size of the container image. Only supported by
# certain container storage drivers.
size = ""
# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"
Configuration is same for both CentOS and Fedora.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 52 (17 by maintainers)
Commits related to this issue
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to mtrmac/image by mtrmac 6 years ago
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to mtrmac/image by mtrmac 6 years ago
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to mtrmac/image by mtrmac 6 years ago
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to mtrmac/image by mtrmac 6 years ago
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to mtrmac/image by mtrmac 6 years ago
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to mtrmac/image by mtrmac 6 years ago
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to Jamesjaj2dc6j/lucascalsilvaa by Jamesjaj2dc6j in 4 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to mrhyperbit23z0d/thomasdarimont by mrhyperbit23z0d in 4 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to easyriderk0c9g/rnin9 by easyriderk0c9g in 4 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to oguzturker8ijm8l/diwir by oguzturker8ijm8l in 4 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to wesnowm/MatfOOP- by wesnowm in 4 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to gaveen3/KimJeongHoon3x by gaveen3 in 4 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to gaveen3/KimJeongHoon3x by gaveen3 in 4 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to Manuel-Suarez-Abascal80n9y/knimeu by Manuel-Suarez-Abascal80n9y in 5 years
- Don't physically store "empty layers" in v2s1 images buildah image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which re... — committed to straiforos8bsh5n/tokingsq by straiforos8bsh5n in 5 years
@mtrmac is still digging. From the last I heard he’d solved the mainline issue, but ran into some issues in other places. He’s trying to squish all the legs of this nasty bug.
@pixdrift, didn’t get to this until just a bit ago, but am able to duplicate the issue with your reproducer. We’d a similar issue before with too many slashes, I’ll look over the weekend to see if I can dig it up.
The original “base” image was built by Origin 1.5 and stored in its registry.
Here’s
buildah inspectoutput: