moby: docker run fails with "invalid argument" when using overlay driver on top of xfs

Fixed…

from docker official site.

To configure Docker to use the overlay storage driver your Docker host must be running version 3.18 of the Linux kernel (preferably newer) with the overlay kernel module loaded. OverlayFS can operate on top of most supported Linux filesystems. However, ext4 is currently recommended for use in production environments.

so, update kernel from 3.10.0 to 3.18.0 + fixed the issue.

presently is not supported. Red Hat is tracking getting xfs to support overlayfs here: https://bugzilla.redhat.com/show_bug.cgi?id=1158888

I wanted to share some internal findings. We’ve tried linux 4.9 and found this in dmesg:

ivan@foo:~$ dmesg | tail
[  184.223910] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[  209.364018] overlayfs: upper fs needs to support d_type.
[  209.582764] overlayfs: upper fs needs to support d_type.
[  209.944933] overlayfs: upper fs needs to support d_type.
[  210.110592] overlayfs: upper fs needs to support d_type.
[  210.317340] overlayfs: upper fs needs to support d_type.
[  210.844979] overlayfs: upper fs needs to support d_type.
[  211.055778] overlayfs: upper fs needs to support d_type.
[  211.305498] overlayfs: upper fs needs to support d_type.
[  212.146527] overlayfs: upper fs needs to support d_type.

Let’s see what’s that: https://patchwork.kernel.org/patch/8377611/

In some instances xfs has been created with ftype=0 and there if a file
on lower fs is removed, overlay leaves a whiteout in upper fs but that
whiteout does not get filtered out and is visible to overlayfs users.

To see what we have:

ivan@foo:~$ sudo xfs_info /dev/md127
meta-data=/dev/md127             isize=256    agcount=32, agsize=5491456 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=175724928, imaxpct=25
         =                       sunit=128    swidth=384 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=85808, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

http://man7.org/linux/man-pages/man8/mkfs.xfs.8.html

                   ftype=value
                          This feature allows the inode type to be stored in
                          the directory structure so that the readdir(3) and
                          getdents(2) do not need to look up the inode to
                          determine the inode type.

                          The value is either 0 or 1, with 1 signifiying
                          that filetype information will be stored in the
                          directory structure. The default value is 0.

                          When CRCs are enabled via -m crc=1, the ftype
                          functionality is always enabled. This feature can
                          not be turned off for such filesystem
                          configurations.

CRC sounds useful and should be cheap on modern CPUs.

                   crc=value
                          This is used to create a filesystem which
                          maintains and checks CRC information in all
                          metadata objects on disk. The value is either 0 to
                          disable the feature, or 1 to enable the use of
                          CRCs.

                          CRCs enable enhanced error detection due to
                          hardware issues, whilst the format changes also
                          improves crash recovery algorithms and the ability
                          of various tools to validate and repair metadata
                          corruptions when they are found.  The CRC
                          algorithm used is CRC32c, so the overhead is
                          dependent on CPU architecture as some CPUs have
                          hardware acceleration of this algorithm.
                          Typically the overhead of calculating and checking
                          the CRCs is not noticeable in normal operation.

                          By default, mkfs.xfs will enable metadata CRCs.

I’ve reformatted disk with crc=1 and it fixed the issue we were facing. It may have been a different issue, but a very similar one.

Centos 7.2 + extfs ===> docker: Error response from daemon: mkdir /var/lib/docker/overlay/xxxx-init/merged/dev/shm: invalid argument.

Hello,

we also experiment the same issue with Centos 7.2 + extfs [root]# docker info > Server Version: 1.11.1 > Storage Driver: overlay > Backing Filesystem: extfs > Kernel Version: 3.10.0-327.4.5.el7.x86_64 > Operating System: CentOS Linux 7 (Core)

Would somebody know a workaround on Centos 7.2 + extfs ?

Regards

PS : I recommend you this test from https://github.com/docker/docker/issues/15668 that reveales the OverlayFS regression: mkdir lower upper work merged mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged touch lower/foo rm merged/foo mkdir merged/foo <- Invalid argument

@bobrik that’s a known issue indeed; in future, docker will refuse to use overlay in that situation, see https://github.com/docker/docker/blob/v1.13.0-rc4/docs/deprecated.md#backing-filesystem-without-d_type-support-for-overlayoverlay2, and https://github.com/docker/docker/issues/27358

Someone want to update the issue title to mention ext4? Sounds like overlay on xfs works, but overlay on ext4 doesn’t.

I have test in centos 7 with 3.10.0-229 kernel， I change the fs/namei.c --> vfs_rename() 4085 line number:

rename2 = get_rename2_iop(old_dir);
if (!old_dir->i_op->rename && !rename2)
    return -EPERM;

if (flags && !rename2)
    return -EINVAL; 

to：

rename2 = get_rename2_iop(old_dir);
if (!old_dir->i_op->rename && !rename2)
    return -EPERM;

if (flags && (!(rename2 || old_dir->i_op->rename))){
       return -EINVAL;
 }

and it work ok， overlayfs can work in 3.10.0-229. I check that overlayfs use ovl_do_rename() ， for example

ovl_create_over_whiteout() --> ovl_do_rename(wdir, newdentry, udir, upper,
                    RENAME_EXCHANGE); 

the inode wdir has not rename2 member. I don’t know why, but I check that 3.18.22 ，when create /dev/shm ， ovl_rename2() is not called, so I think this place should not just test the rename2 function.

Confirmed the issue using home made ext4 image on amazon with Cent OS 7. Note: this is a locally built ova that I use as a base for some vagrant boxes, which don’t have the problem. As well, I have an opensuse host that doesn’t have the issue.

Update: I am not sure why this does not cause a problem in my virtual box instance, based on the same image, but I noticed that the documentation specifically mentions overlayfs only having been merged, officially, in 3.18.0.

Checking uname -r reveled that centos 7.1 is still using 3.10 and I could not find anything online about centos having back ported. Installing the newer 4.2 kernel got things working as expected.

@dserodio afaik, CentOS / RHEL 7.2, (and the kernel that comes with that). According to the release notes. In version 7.1, EXT4 was still recommended