moby: [1.11.2] Mounted directory in new container out of sync with host directory

Seeing some odd behavior, where mounting a directory into a container is bringing in some other unknown directory, and not the one from the host container that I expect:

$ pwd
/vol/elasticsearch

$ ls
config  data  logs

$ docker run -it --rm -v /vol/elasticsearch:/vol/elasticsearch elasticsearch:2.3.3 ls /vol/elasticsearch
config  config2  data  logs

Note the existence of “config2” in the container, but not the host. The “config2” directory was created at some point by me trying to debug this thing.

The actual failure I’m getting is that elasticsearch is failing to start because inside the config directory, there is an elasticsearch.yml file, which is a normal file on the host, but is inexplicably a directory when mounted:

$ ls -la /vol/elasticsearch/config/
total 8
drwxr-xr-x 3 root root  62 Aug 10 23:26 .
drwxr-xr-x 5 root root  41 Aug  8 22:32 ..
-rw-r--r-- 1 root root 449 Aug 10 16:44 elasticsearch.yml
-rw-r--r-- 1 root root 468 Aug  8 21:48 logging.yml
drwxr-xr-x 2 root root   6 Aug  8 21:53 scripts

$ docker run -it --rm -v /vol/elasticsearch:/vol/elasticsearch elasticsearch:2.3.3 ls -la /vol/elasticsearch/config
total 12
drwxr-xr-x 3 root root 4096 Aug  5 22:01 .
drwxr-xr-x 6 root root 4096 Aug  8 22:29 ..
drwxr-xr-x 2 root root 4096 Aug  5 22:01 elasticsearch.yml

How that elasticsearch.yml file turned into a directory is beyond me. It could be something weird that the elasticsearch image is doing, but regardless, the mounted volume is just entirely out of sync.

Restarting docker fixed the issue for me while writing up this bug report, but I have also seen it not be fixed on restart:

$ service docker restart
Stopping docker:                                           [  OK  ]
Starting docker:                                       [  OK  ]

$ docker run -it --rm -v /vol/elasticsearch:/vol/elasticsearch elasticsearch:2.3.3 ls -la /vol/elasticsearch/config
total 8
drwxr-xr-x 3 root root  62 Aug 10 23:26 .
drwxr-xr-x 5 root root  41 Aug  8 22:32 ..
-rw-r--r-- 1 root root 449 Aug 10 16:44 elasticsearch.yml
-rw-r--r-- 1 root root 468 Aug  8 21:48 logging.yml
drwxr-xr-x 2 root root   6 Aug  8 21:53 scripts

Not really sure what "inode"s are, but in reading some other issues, it sounded like this info could be useful:

$ stat -c %i /vol/elasticsearch/config
33554560

$ docker run -it --rm -v /vol/elasticsearch:/vol/elasticsearch elasticsearch:2.3.3 stat -c %i /vol/elasticsearch/config
270154

Possibly related to #25326

Output of docker version:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   b9f10c9/1.11.2
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   b9f10c9/1.11.2
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 4
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-202:1-266559-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.432 GB
 Data Space Total: 107.4 GB
 Data Space Available: 4.813 GB
 Metadata Space Used: 3.187 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.15-25.57.amzn1.x86_64
Operating System: Amazon Linux AMI 2016.03
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.956 GiB
Name: crumble
ID: 4QFV:AKU3:QQN2:MM7H:HA53:SGL2:62OK:VABQ:CQCK:6WRP:65LN:QV4A
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

Additional environment details (AWS, VirtualBox, physical, etc.):

• AWS server using Amazon Linux 2016.03 (latest)
• Deployed into EC2 Container Service
• Using Chef and Opsworks to provision server

Steps to reproduce the issue: Happens daily, but I can’t reproduce on demand. Will work more on this. If it is something specific to my environment, it seems likely that it could be happening when Chef runs occasionally, but not sure.

Describe the results you received:

Directory mounted into container has different contents than the host directory, even for a brand new container.

Describe the results you expected:

Expect the directory to have the same contents on the host and container.

Additional information you deem important (e.g. issue happens only occasionally):

The /vol/elasticsearch directory and all its contents are created using Chef, which runs occasionally and examines the directories/files to make sure they are what they should be.