moby: inotify does not work when mounting volumes to docker daemons running in virtual machines

Description of problem:

Many tools used for local development include restart scripts which watch the file system and restart the app when a change occurs. Virtual machines do not propagate filesystem events from mounted volumes, preventing containers with mounted volumes running in virtual machines from seeing these events (read, boot2docker via docker-machine).

Many of us use Docker as a tool for rapidly iterating on code, using restart scripts watching mounted volumes. But this workflow breaks down when sharing our tooling with team members who run OSx or Windows (since they are relying on virtual machines).

While this is a shortcoming with Virtualbox and similar virtualization tools, the maintainers have stated they will not be implementing fs events on mounted volumes. For more background, refer to the Additional Info section below.

Since it appears that the virtual machines will not be supporting this anytime soon, would it be worth solving this at the docker level?

Proposed Solution

A popular current solution is to rsync the contents of the volume on loop to the virtualmachine and mount the volume from there, avoiding the shared volume all together. This is essentially polling and is expensive.

I propose that the docker tooling take on this responsibility. When running docker run with a mounted volume on a remote docker daemon do the following:

1. sync the contents of the volume to the remote daemon
2. Mount the remote directory to the container.
3. Establish a recursive inotify/kqueue/ReadDirectoryChangesW on the local directory.
4. When a change to the directory is detected, re-run sync.

This avoids the need of shared volumes for the virtual machine, allowing for the use of filesystem watches within containers

Downsides to the proposed solution

1. Requires a daemon like process to establish and maintain inotify events during the lifetime of the remote container.
2. Potentially expensive operation when syncing large files.
3. May fill remove VM’s hard disk with synced files.
4. Need to implement a form of garbage collection for remote directories when the container is removed.

docker version`

Docker version 1.9.0 build 76d6bc9

docker-machine -v

docker-machine version 0.5.0 (04cfa58)

docker info

Containers: 0
Images: 4457
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 4712
 Dirperm1 Supported: true
Execution Driver: native-0.2
aLogging Driver: json-file
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 8
Total Memory: 31.37 GiB
Name: BigBlu
ID: ABYD:SSMM:VFCM:VX2N:ZJA3:IDFP:6UA2:67XD:5J5J:IDSX:UXYB:63PW
Username: wblankenship
Registry: https://index.docker.io/v1/

uname -a

Linux BigBlu 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

vboxmanage --version

4.3.18_Debianr96516

How reproducible:

100%

Steps to Reproduce:

1. docker machine create -d virtualbox inotify
2. eval "$(docker machine env inotify)"
3. touch test
4. docker run -v ${PWD}/test:/usr/src/test -it debian:jessie bash -c “apt-get update && apt-get install -y inotify-tools && inotifywait /usr/src/test”
5. Finally, open another terminal and touch test

Actual Results:

Script doesn’t exit

Expected Results:

inotifywait should have returned test OPEN and the script exited.

Additional Info:

Closed tickets on the Virtualbox bug tracker referring to this short coming:

• https://www.virtualbox.org/ticket/10660
• https://www.virtualbox.org/ticket/14234

This may be a duplicate issue, I searched around across the docker, swarm, and machine repos and found nothing. Here are similar issues that I found:

• https://www.virtualbox.org/ticket/14234
• https://github.com/docker/docker/issues/4023
• https://github.com/docker/docker/issues/1044
• https://github.com/docker/docker/issues/4213
• https://github.com/docker/machine/issues/179

This seems to be the state of the art solution currently, from @brikis98 et. al.:

https://github.com/brikis98/docker-osx-dev