moby: Docker 1.8 corrupts /etc/hosts files in containers

related: #16619

@mavenugo - We have a problem.

We’ve been experiencing periodic end-to-end test failures where some containers can not resolve “localhost”. Killing the container and letting it restart usually fixes the problem. Today we got it into this state and pinned down the following:

The container in question has a /etc/hosts file that is several hundred lines long. That /etc/hosts file does NOT have localhost or the container’s own IP. If I add those 2 lines the tests pass.

Of course I thought of #16619 . So we looked at /etc/hosts of every container on this machine - 84 in total. Those files range in size from 175 lines (correct: 7 boilerplate + 2 per container) to 1700+ lines. Looking at the offending files, there are a lot of duplicate IPs. It looks like old containers have not been removed, but localhost, “self”, and the ipv6 boilerplate has been removed.

Even looking at my local test machine (far less churn) I see the same. I have 4 containers with network namespace running, I should have 8+7=15 lines, right?

# wc -l /var/lib/docker/containers/*/hosts
  15 /var/lib/docker/containers/3fc23782814513d1934ce21c0f1b096263df1fcacee6787208c3cda715822356/hosts
  15 /var/lib/docker/containers/4b7e8432a368eeb7f1dc49230f16c1d7c788b471e4dfa7d8fbf709c3495f430f/hosts
  19 /var/lib/docker/containers/5715aa09fc091ee0ff3d4dcf2ee300d8e52603dafcde8f24aaf1def960cf4a28/hosts
  15 /var/lib/docker/containers/6e7fe564dece829de3c335400357b2ef327a1a44c638045350f0b49e5324a189/hosts
  15 /var/lib/docker/containers/7845351acbacf0588142a1ebfa3fa481c3191da48a188b20c5d5fdae689cef1e/hosts
  17 /var/lib/docker/containers/ba23cbabfa90700f4d9ba80c5723afb1d7479a08b12efdded74e5874991ce76b/hosts
  17 /var/lib/docker/containers/c1e64038b35a0527b6aef911dad002278569724263cc922efd8e715747e83538/hosts
  19 /var/lib/docker/containers/fc7c5475fd923b75a0595431dcaf6b6db5fa8a18409d0df4e6282e3c0bd8aedf/hosts

Diffing a 15 and 19 line file shows:

# diff -u -U 100 <(sort -n /var/lib/docker/containers/3fc23782814513d1934ce21c0f1b096263df1fcacee6787208c3cda715822356/hosts) <(sort -n /var/lib/docker/containers/5715aa09fc091ee0ff3d4dcf2ee300d8e52603dafcde8f24aaf1def960cf4a28/hosts)
--- /dev/fd/63  2015-10-19 23:59:20.618879795 +0000
+++ /dev/fd/62  2015-10-19 23:59:20.618879795 +0000
@@ -1,15 +1,19 @@
 ::1    localhost ip6-localhost ip6-loopback
 fe00::0    ip6-localnet
 ff00::0    ip6-mcastprefix
 ff02::1    ip6-allnodes
 ff02::2    ip6-allrouters
+10.244.0.10    distracted_ritchie
+10.244.0.10    distracted_ritchie.bridge
 10.244.0.2 k8s_POD-7be6d81d_fluentd-cloud-logging-kubernetes-minion-isfq_kube-system_dcfde077753b8b6908952cfea6a14200_c4f00d1d
 10.244.0.2 k8s_POD-7be6d81d_fluentd-cloud-logging-kubernetes-minion-isfq_kube-system_dcfde077753b8b6908952cfea6a14200_c4f00d1d.bridge
 10.244.0.3 k8s_POD-7be6d81d_heapster-v10-lx8q7_kube-system_2a3fc6d6-6ddb-11e5-9249-42010af00002_cbe79e10
 10.244.0.3 k8s_POD-7be6d81d_heapster-v10-lx8q7_kube-system_2a3fc6d6-6ddb-11e5-9249-42010af00002_cbe79e10.bridge
 10.244.0.4 k8s_POD-c5371ceb_monitoring-influxdb-grafana-v2-koh3j_kube-system_2a43f315-6ddb-11e5-9249-42010af00002_76d3cb92
 10.244.0.4 k8s_POD-c5371ceb_monitoring-influxdb-grafana-v2-koh3j_kube-system_2a43f315-6ddb-11e5-9249-42010af00002_76d3cb92.bridge
 10.244.0.5 k8s_POD-9db2f941_kube-ui-v2-v4yd6_kube-system_2a42e6da-6ddb-11e5-9249-42010af00002_69a0a0c7
 10.244.0.5 k8s_POD-9db2f941_kube-ui-v2-v4yd6_kube-system_2a42e6da-6ddb-11e5-9249-42010af00002_69a0a0c7.bridge
-10.244.0.5 kube-ui-v2-v4yd6
+10.244.0.6 k8s_POD-6e934112_kube-dns-v9-ee38a_kube-system_2a4c2763-6ddb-11e5-9249-42010af00002_86e49fbc
+10.244.0.6 k8s_POD-6e934112_kube-dns-v9-ee38a_kube-system_2a4c2763-6ddb-11e5-9249-42010af00002_86e49fbc.bridge
+10.244.0.6 kube-dns-v9-ee38a
 127.0.0.1  localhost

Is there any reason the hosts files would be different for different containers on the same node?

I know docker is live-editing the hosts file (which is absolutely unsafe, but should only be unsafe for the users in the container). Any clues on what’s going on?

@artfulcoder