moby: inotify does not work when mounting volumes to docker daemons running in virtual machines
Description of problem:
Many tools used for local development include restart scripts which watch the file system and restart the app when a change occurs. Virtual machines do not propagate filesystem events from mounted volumes, preventing containers with mounted volumes running in virtual machines from seeing these events (read, boot2docker via docker-machine).
Many of us use Docker as a tool for rapidly iterating on code, using restart scripts watching mounted volumes. But this workflow breaks down when sharing our tooling with team members who run OSx or Windows (since they are relying on virtual machines).
While this is a shortcoming with Virtualbox and similar virtualization tools, the maintainers have stated they will not be implementing fs events on mounted volumes. For more background, refer to the Additional Info
section below.
Since it appears that the virtual machines will not be supporting this anytime soon, would it be worth solving this at the docker level?
Proposed Solution
A popular current solution is to rsync the contents of the volume on loop to the virtualmachine and mount the volume from there, avoiding the shared volume all together. This is essentially polling and is expensive.
I propose that the docker tooling take on this responsibility. When running docker run
with a mounted volume on a remote docker daemon do the following:
- sync the contents of the volume to the remote daemon
- Mount the remote directory to the container.
- Establish a recursive
inotify
/kqueue
/ReadDirectoryChangesW
on the local directory. - When a change to the directory is detected, re-run sync.
This avoids the need of shared volumes for the virtual machine, allowing for the use of filesystem watches within containers
Downsides to the proposed solution
- Requires a daemon like process to establish and maintain
inotify
events during the lifetime of the remote container. - Potentially expensive operation when syncing large files.
- May fill remove VM’s hard disk with synced files.
- Need to implement a form of garbage collection for remote directories when the container is removed.
docker version`
Docker version 1.9.0 build 76d6bc9
docker-machine -v
docker-machine version 0.5.0 (04cfa58)
docker info
Containers: 0
Images: 4457
Server Version: 1.9.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 4712
Dirperm1 Supported: true
Execution Driver: native-0.2
aLogging Driver: json-file
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 8
Total Memory: 31.37 GiB
Name: BigBlu
ID: ABYD:SSMM:VFCM:VX2N:ZJA3:IDFP:6UA2:67XD:5J5J:IDSX:UXYB:63PW
Username: wblankenship
Registry: https://index.docker.io/v1/
uname -a
Linux BigBlu 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
Environment details (AWS, VirtualBox, physical, etc.):
vboxmanage --version
4.3.18_Debianr96516
How reproducible:
100%
Steps to Reproduce:
docker machine create -d virtualbox inotify
eval "$(docker machine env inotify)"
touch test
- docker run -v ${PWD}/test:/usr/src/test -it debian:jessie bash -c “apt-get update && apt-get install -y inotify-tools && inotifywait /usr/src/test”
- Finally, open another terminal and
touch test
Actual Results:
Script doesn’t exit
Expected Results:
inotifywait
should have returned test OPEN
and the script exited.
Additional Info:
Closed tickets on the Virtualbox bug tracker referring to this short coming:
This may be a duplicate issue, I searched around across the docker, swarm, and machine repos and found nothing. Here are similar issues that I found:
- https://www.virtualbox.org/ticket/14234
- https://github.com/docker/docker/issues/4023
- https://github.com/docker/docker/issues/1044
- https://github.com/docker/docker/issues/4213
- https://github.com/docker/machine/issues/179
This seems to be the state of the art solution currently, from @brikis98 et. al.:
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 42
- Comments: 43 (14 by maintainers)
In Mac, like @jeremyVignelles says above, this has been solved in
docker run
with Docker for Mac’s release, where filesystem events will fire inside and out of container.I’m going ahead and close this issue;
@cglewis inotify not working in VirtualBox is a limitation of VirtualBox guest additions, and was closed as a “won’t fix” by the VirtualBox team (see the issues linked from the top description) - as @friism explained, this was one of the reasons for Docker to develop Docker for Mac, which does not have that limitation
@tframesqui and what’s supposed to be auto-reloading your app? If you get rid of pyinotify, it looks like it’ll poll: https://docs.djangoproject.com/en/1.10/ref/django-admin/#runserver
Any news about this on Windows?
@thaJeztah Here’s our bug report, same issue on two Docker Captain’s machines plus others: https://github.com/docker/for-mac/issues/148.
I agree swarm mode on Mac is mostly an edge case, but in our use case we’re talking to the swarm api to create swarm containers with mounts that need fs-events in/out.
Semi related: your excellent
run
vsservice
epic https://github.com/docker/docker/issues/25303I know it won’t work if you’ve enabled swarm mode and use
docker service create
but I’ve always seen it work withdocker run
. You’ve got evidence of it otherwise @rjsteinert ?@duro I haven’t had time to code it yet.
For now, I’ve been using nodemon’s legacy watcher. It supports non-node applications and it doesn’t rely on file system events.
If you go with this solution and you have large directories (like
node_modules
in Node.js) that rarely change, I would recommend adding them to nodemon’s exclude list. Checking all of those files for changes is extremely taxing.