moby: file system corruption with devicemapper/aufs

I am filing this issue to try to get a single place to document what I see as an extremely severe issue: when using Docker, sometimes your container filesystems and/or files become seriously corrupted.

There are at least two mailing list threads pointing to the problem: https://groups.google.com/forum/#!topic/docker-dev/Xzwm5GRYCLo https://groups.google.com/forum/#!topic/docker-user/35Z0-g8sObY

And one previously filed issue (now closed) #6368

The problem manifests itself in a few different ways. It is unclear to me if these are all related.

Known symptoms:

• Corrupt files (docker run myimage md5sum /some/file returns different results on different hosts)
• Corrupt filesystems, example messages:
  • EXT4-fs error (device dm-2): ext4_lookup:1437: inode #132310: comm docker: deleted inode referenced: 134547
  • [271394.160211] EXT4-fs warning (device dm-10): ext4_end_bio:317: I/O error writing to inode 402137 (offset 0 size 0 starting block 169271) [271394.160214] Buffer I/O error on device dm-10, logical block 169271
  • htree_dirblock_to_tree: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0

So far this has only been reported against the devicemapper and aufs backends, not btrfs. It is possible that this is a kernel bug but since it is so disastrous to Docker workloads I believe it deserves a bug here regardless of the root cause.

I have verified that this happens against Ubuntu 14.04 3.13.0-24-generic with Docker 1.0.0.