moby: unexpected file permission error in container

We are seeing a very similar issue with Overlay.

Our current guess is that it is caused by the bug fixed in kernel 4.4.6 by this commit: https://lkml.org/lkml/2016/1/31/82

We have not yet managed to test with 4.4.6.

@jberkus thanks for the info. I was able to reproduce it on Ubuntu 14.04 on kernel 3.19.something. Would be nice to know which version of aufs its solved in, possibly the Docker Mac team can bump the version to include the patch that’s needed

rm /etc/puppet/* didn’t work for me, instead I found that deleting individual files on the same layer works perfectly well:

find /etc/pappet -type f | xargs -L1 rm -f

Here is link to my Gist

This is a kind of bug in AUFS. When a directory has a given permission mask in a lower layer, the upper layers cannot have a broader mask. Well, they can, but the more restrictive permission mask will be enforced anyways.

The rationale is the following:

• suppose that you have directory /secret with permissions 0700, containing file /secret/key.pem
• in an upper layer, you give /secret permissions 0755
• now /secret/key.pem could become reachable

Multiple behaviors could be considered “acceptable” in this scenario:

• give access to the file anyway (but this option was vetoed because it was deemed insecure)
• prevent access
• place a kind of “tombstone” or “opaque whiteout” for the whole directory so that the directory below becomes “opaqued” or “whited out”, and the new one takes precedence

My understanding is that the last solution should be used, but for some reason, AUFS doesn’t behave correctly. It might be because the directory exists in a lower layer, then doesn’t exist anymore (because of the rm), then exists again (because of the ADD).

I’m willing to take a guess: the logic that decides whether or not to do a whiteout is not exactly the same as the one looking up permissions; so the first one stops when /etc/puppet is marked as non-existent in the middle layer, while the latter goes bottom up.

Anyway!

As a workaround, you can rm /etc/puppet/* instead of rm /etc/puppet, and that will do the trick.

I opened an issue in the internal issue tracker as well (number #2693)

I can confirm encountering this issue on the the docker os x beta-15 (current version). I have switched back to docker-machine in the meantime.

@antoineco I was able to reproduce on Docker for OS X, switching to overlay resolved it, so it’s an aufs issue indeed. Can you report this issue through beta-feedback@docker.com ?

Here’s a workaround I use in a postgres dockerfile

RUN mkdir /etc/ssl/private-copy; mv /etc/ssl/private/* /etc/ssl/private-copy/; rm -r /etc/ssl/private; mv /etc/ssl/private-copy /etc/ssl/private; chmod -R 0700 /etc/ssl/private; chown -R postgres /etc/ssl/private

Just got bit by this too. We’ve got sequential builds, and at several points we need to obliterate a directory that was ADDed in a previous build step. I was surprised that the following command silently fails:

RUN rm -Rf /var/shared/sites/coursecal

I’m not 100% sure why, but the workaround suggested by @jpetazzo above seems to work:

RUN rm -Rf /var/shared/sites/coursecal/* /var/shared/sites/coursecal/.*git

The only reason we have to do the “rm” in the first place is because the following line is really a tar x, and it leaves existing files around, as documented here

ADD . /var/shared/sites/coursecal

Hmmm… What about a “KNOWN BUGS AND ISSUES” section in the documentation? /cc @metalivedev

@jpetazzo what do you think?

Since this is documented aufs behavior, we can 1) close this as wontfix, 2) close + document the behavior in the docker docs, or 3) other?

Labelling as aufs-related.