moby: overlayfs fails to run container with a strange file checksum error

There is a yum patch, simply add this in your Dockerfile :

RUN yum install -y yum-plugin-ovl

Tested in Centos 7/6 + RHEL 7/6

http://man7.org/linux/man-pages/man1/yum-ovl.1.html

Enjoy Docker/overlay 😃

Found that running rpm --rebuilddb before any yum installs is a decent workaround for this issue as @porjo has mentioned above. Also, using AUFS I have not seen this problem.

I have the same issue with Docker version 17.03.1-ce, build c6d412e and CENTOS 7

RUN rpm --rebuilddb && yum install -y XXX => This solution has resolved this issue

If you’re unable to install the yum plugin because of dependency conflicts (e.g. an older base image), You can work around this with touch /var/lib/rpm/*. See the bug report for details.

I use CoreOS and have just switched from Btrfs to Ext4 and I having the same issue. I was also seeing errors like this:

error: rpmdb: BDB0689 Packages page 122 is on free list with type 7
error: rpmdb: BDB0061 PANIC: Invalid argument
error: db5 error(-30973) from dbcursor->c_put: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: error(-30973) adding header #163 record
error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery
error: db5 error(-30973) from dbcursor->c_close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery
error: db5 error(-30973) from db->sync: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

I found that inserting RUN rpm --rebuilddb prior to yum install -y <pkg> resolved both errors, however that should not be necessary.

@xavierbaude I also get the checksum error when install yum-plugin-ovl 😢

@jasonmp85 I’ve just pushed an updated for the OL6 and OL7 latest images. In future, ping me if you have any issues with the OL base and I can address them. Usually our turnaround time is pretty quick, if we’re made aware of an issue. 😃

So what’s the current guidance for fixing this? Still touch /var/lib/rpm/*?

It’s one thing when I’m building images on my local box, but for automated builds in Docker Hub, this is really annoying, and I have no clue how to fix it.

@jakirkham Please try this.

#!/usr/bin/env python
# Workaround for CentOS 6 yum overlay issue (docker/docker#10180)
# Based on yum-utils-1.1.31-34.el7 (GPLv2+)
#
# Example Dockerfile:
#  FROM centos:centos6
#  ADD this.py /
#  RUN /this.py && yum install -y pkg1
#  RUN /this.py && yum install -y pkg2

from os import walk, path, fstat

def _stat_ino_fp(fp):
    """
    Get the inode number from file descriptor
    """
    return fstat(fp.fileno()).st_ino


def get_file_list(rpmpath):
    """
    Enumerate all files in a directory
    """
    for root, _, files in walk(rpmpath):
        for f in files:
            yield path.join(root, f)


def for_each_file(files, cb, m='rb'):
    """
    Open each file with mode specified in `m`
    and invoke `cb` on each of the file objects
    """
    if not files or not cb:
        return []
    ret = []
    for f in files:
        with open(f, m) as fp:
            ret.append(cb(fp))
    return ret


def do_detect_copy_up(files):
    """
    Open the files first R/O, then R/W and count unique
    inode numbers
    """
    num_files = len(files)
    lower = for_each_file(files, _stat_ino_fp, 'rb')
    upper = for_each_file(files, _stat_ino_fp, 'ab')
    diff = set(lower + upper)
    return len(diff) - num_files

def main():
    rpmdb_path = '/var/lib/rpm'
    try:
        files = list(get_file_list(rpmdb_path))
        copied_num = do_detect_copy_up(files)
        print("ovl: Copying up (%i) files from OverlayFS lower layer" % copied_num)
    except Exception as e:
        print("ovl: Error while doing RPMdb copy-up:\n%s" % e)

if __name__ == '__main__':
    main()

The latest CentOS 7 image includes the workaround yum-plugin-ovl:

$ docker run -it --rm library/centos@sha256:8dcd2ec6183f3f4a94d4f9552ce76091624760edefcaa39a9e04441f9e2ad9f6 sh -c "rpm -qa | grep yum"
yum-plugin-fastestmirror-1.1.31-34.el7.noarch
yum-utils-1.1.31-34.el7.noarch
yum-metadata-parser-1.1.4-10.el7.x86_64
yum-3.4.3-132.el7.centos.0.1.noarch
yum-plugin-ovl-1.1.31-34.el7.noarch

Note that the latest tagged image not:

$ docker run -it --rm centos:centos7.2.1511 sh -c "rpm -qa | grep yum"
yum-metadata-parser-1.1.4-10.el7.x86_64
yum-3.4.3-132.el7.centos.0.1.noarch
yum-plugin-fastestmirror-1.1.31-34.el7.noarch

I’ll open a PR that adds a note to the documentation.

@thaJeztah Do you think this issue is closable after that PR?

meet this issue on Linux fedora 4.0.4-301.fc22.x86_64

The original “Rpmdb checksum is invalid” occurs because of an overlayfs limitation.

In general, data inconsistency can be demonstrated by doing the following:

• pick a file that has not yet been copied up to the upper layer
• open it twice: once in O_RDONLY (fd-ro) and once in O_RDWR (fd-rw) ==> Writing to fd-rw, and reading from fd-ro results in data mismatch.

The problem with yum occurs because a file is opened from /var/lib/rpm in RO mode (from ‘lower’ layer), the same file is subsequently opened in r/w mode (resulting in a copy up to ‘upper’) and is updated with new data. The contents of the file when read from the RO fd don’t match the new data resulting in the ‘checksum is invalid’ complaint. Subsequent operations with yum work because /var/lib/rpm files how exist in the ‘upper’ layer and all the fds come from the ‘upper’ layer.

We’ve created a kernel patch for overlayfs to solve this problem and are currently testing it. Please give it a whirl and see if it solves your problem - appreciate your feedback.

The source is here: https://github.com/portworx/overlayfs