rancher: kubelet - failed to collect filesystem stats - rootDiskErr: du command failed

Rancher Versions: Server:1.4.0 healthcheck: ipsec: network-services: scheduler: kubernetes (if applicable): rancher/k8s:v1.5.2-rancher1-2 Docker Version: 1.12.6 OS and where are the hosts located? (cloud, bare metal, etc): RHEL7.3, vmware Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) single node rancher, external mysql db Environment Type: (Cattle/Kubernetes/Swarm/Mesos) k8s

2/9/2017 11:54:51 AME0209 17:54:51.517354   25464 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /mnt/dockerdata/overlay/a7a4ef47300a0b3fe397070b096505a624f6b4ec292929e505ea109ffa7f801c with output stdout: 943940	/mnt/dockerdata/overlay/a7a4ef47300a0b3fe397070b096505a624f6b4ec292929e505ea109ffa7f801c
2/9/2017 11:54:51 AM, stderr: du: cannot access '/mnt/dockerdata/overlay/a7a4ef47300a0b3fe397070b096505a624f6b4ec292929e505ea109ffa7f801c/merged/proc/17285/task/17285/fd/4': No such file or directory
2/9/2017 11:54:51 AMdu: cannot access '/mnt/dockerdata/overlay/a7a4ef47300a0b3fe397070b096505a624f6b4ec292929e505ea109ffa7f801c/merged/proc/17285/task/17285/fdinfo/4': No such file or directory
2/9/2017 11:54:51 AMdu: cannot access '/mnt/dockerdata/overlay/a7a4ef47300a0b3fe397070b096505a624f6b4ec292929e505ea109ffa7f801c/merged/proc/17285/fd/4': No such file or directory
2/9/2017 11:54:51 AMdu: cannot access '/mnt/dockerdata/overlay/a7a4ef47300a0b3fe397070b096505a624f6b4ec292929e505ea109ffa7f801c/merged/proc/17285/fdinfo/4': No such file or directory
2/9/2017 11:54:51 AM - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 17

Most upvoted comments

@pulberg - I attach a new device to my machine, mounted it on /mnt/dockerdata

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        64G  3.6G   61G   6% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G   33M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           799M     0  799M   0% /run/user/1022
/dev/dm-2        10G  256M  9.8G   3% /var/lib/docker/devicemapper/mnt/b365d57bf058ddfb4e8a4bf1894c96d7f44f01ea838433c59c788e997bf37444
shm              64M     0   64M   0% /var/lib/docker/containers/147a6f070303c57abd06ccaccbe104550b5f2335af75e8052c01dc9554d4a73a/shm
/dev/dm-3        10G  263M  9.8G   3% /var/lib/docker/devicemapper/mnt/f6c22f2d4f3a6696345e06d6529531b988453891aeb8453394a66c1aebb5b4e9
shm              64M     0   64M   0% /var/lib/docker/containers/81b91129ffda1d4d4a19c1659343b864b597db89aaa2213046e78b44b5f53a37/shm
/dev/dm-4        10G  268M  9.8G   3% /var/lib/docker/devicemapper/mnt/8c283d3e16804e2abdb284e6683d0b363400bd1ef3656bd187df08ea3412cd5b
shm              64M     0   64M   0% /var/lib/docker/containers/5c34bd4802979f2124dfef55a37f6a08ae3703b53b3e2f1f613f05c1f37de41d/shm
/dev/dm-7        10G  285M  9.8G   3% /var/lib/docker/devicemapper/mnt/7651e9d1001f680fb41e91b387c1e2c40c4db17dcfa7749d71fbb20b79aaf893
shm              64M     0   64M   0% /var/lib/docker/containers/9ade55e85bc7951746261e691377fa260c5aff4f23577c47df64eea28da07d56/shm
/dev/dm-6        10G  261M  9.8G   3% /var/lib/docker/devicemapper/mnt/41eac21070c75c273b009f2fe6920458994dcd8152147ab7d4e8d0ef82ea9e60
shm              64M     0   64M   0% /var/lib/docker/containers/0a2751bca1be4769386803db8a191a726d5f8a7cf958af62106c1a4ff19c3611/shm
/dev/dm-5        10G  401M  9.6G   4% /var/lib/docker/devicemapper/mnt/f8cfd561ef55618cc474b1adc90c31a4ce5e77148d8588f009a9dfa2ab22681b
shm              64M     0   64M   0% /var/lib/docker/containers/b392c09ee88aae63666029fdbe1d1f632ad965bacb2ccc044b20a96909189806/shm
/dev/dm-1        10G  283M  9.8G   3% /var/lib/docker/devicemapper/mnt/d3cd353d810ceb6b09442b720fc592e371e4ef674b611c3ce969a467a26b2c1d
shm              64M     0   64M   0% /var/lib/docker/containers/a8797c3578b33b6704d39a8162ccaf31600f90d9b64c29767afd03d285a35311/shm
/dev/dm-9        10G  285M  9.8G   3% /var/lib/docker/devicemapper/mnt/36b9a99d194e66c8828c9e38496b805af508f0deb459c4bdf069d34f1e6e774b
shm              64M     0   64M   0% /var/lib/docker/containers/e6dc3a35e5f7a05fdc2c788dabfd871bf7f09119e2090a6efde0b300dcf46f6c/shm
/dev/dm-8        10G  929M  9.1G  10% /var/lib/docker/devicemapper/mnt/be206c7c79f63730494b9f678f093d5983925962312c4a26d1a278191e81ccef
/dev/dm-10       10G  929M  9.1G  10% /var/lib/docker/devicemapper/mnt/c581c2ab4431fcd5f84b0a99020306ad534d1aa8d2796692bb54c756eaed2f8a
shm              64M     0   64M   0% /var/lib/docker/containers/100d2c7683dc635ee1c693b0a2c372167a58ecf86e679a6590872333f035647d/shm
tmpfs           799M     0  799M   0% /run/user/0
/dev/sdb        976M  1.3M  908M   1% /mnt/dockerdata

and started a pod with this spec

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-service
spec:
  replicas: 1
  selector:
    k8s-app: nginx-service
  template:
    metadata:
      labels:
        k8s-app: nginx-service
    spec:
      nodeSelector:
        redhat: "true"
      terminationGracePeriodSeconds: 60
      containers:
      - name: nginx-service
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: "tester"
          mountPath: "/test"
      volumes:
      - name: "tester"
        hostPath: 
          path: "/mnt/dockerdata/tester"

My only log lines are

2/22/2017 2:35:03 PMI0222 22:35:03.633025    4278 docker_manager.go:2236] Determined pod ip after infra change: "nginx-service-tp5rz_default(26152753-f94f-11e6-8625-02a3313a30e8)": "10.42.183.102"
2/22/2017 2:35:03 PME0222 22:35:03.635925    4278 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
2/22/2017 2:35:03 PME0222 22:35:03.638259    4278 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
2/22/2017 2:35:04 PMI0222 22:35:04.732925    4278 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/26152753-f94f-11e6-8625-02a3313a30e8-default-token-6d3fx" (spec.Name: "default-token-6d3fx") pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8").
2/22/2017 2:35:05 PMI0222 22:35:05.738080    4278 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/26152753-f94f-11e6-8625-02a3313a30e8-default-token-6d3fx" (spec.Name: "default-token-6d3fx") pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8").
2/22/2017 2:35:18 PME0222 22:35:18.749115    4278 generic.go:238] PLEG: Ignoring events for pod nginx-service-tp5rz/default: Error response from daemon: {"message":"devmapper: Unknown device e99ffc6e1856b08c21366be51384ceebf782ff046b9c5f019d34295afb18923e"}
2/22/2017 2:35:19 PMW0222 22:35:19.761542    4278 pod_container_deletor.go:77] Container "8f09cf2323947f152725ecf02fc8ba2677257f2b67bcd33d56d6d3d938213351" not found in pod's containers
2/22/2017 2:35:19 PMI0222 22:35:19.880329    4278 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/26152753-f94f-11e6-8625-02a3313a30e8-default-token-6d3fx" (spec.Name: "default-token-6d3fx") pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8").
2/22/2017 2:35:21 PMI0222 22:35:21.912733    4278 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/26152753-f94f-11e6-8625-02a3313a30e8-default-token-6d3fx" (spec.Name: "default-token-6d3fx") pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8").
2/22/2017 2:35:38 PMW0222 22:35:38.989793    4278 docker_manager.go:1611] No ref for pod '"456bbbf66618c908f60b28cb5c3f7b59784b8b046fbc6c0911db8a8739fc5f2b nginx-service default/nginx-service-tp5rz"'
2/22/2017 2:35:39 PMW0222 22:35:39.003757    4278 docker_manager.go:1611] No ref for pod '"655aa71b905c6c5ac128059596d34b4b4b8c134687647f1d3a82acad33070d7f default/nginx-service-tp5rz"'
2/22/2017 2:35:39 PMI0222 22:35:39.101211    4278 reconciler.go:189] UnmountVolume operation started for volume "kubernetes.io/secret/26152753-f94f-11e6-8625-02a3313a30e8-default-token-6d3fx" (spec.Name: "default-token-6d3fx") from pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8").
2/22/2017 2:35:39 PMI0222 22:35:39.101283    4278 reconciler.go:189] UnmountVolume operation started for volume "kubernetes.io/host-path/26152753-f94f-11e6-8625-02a3313a30e8-tester" (spec.Name: "tester") from pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8").
2/22/2017 2:35:39 PMI0222 22:35:39.101332    4278 operation_executor.go:992] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/26152753-f94f-11e6-8625-02a3313a30e8-tester" (OuterVolumeSpecName: "tester") pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8"). InnerVolumeSpecName "tester". PluginName "kubernetes.io/host-path", VolumeGidValue ""
2/22/2017 2:35:39 PMI0222 22:35:39.104847    4278 operation_executor.go:992] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26152753-f94f-11e6-8625-02a3313a30e8-default-token-6d3fx" (OuterVolumeSpecName: "default-token-6d3fx") pod "26152753-f94f-11e6-8625-02a3313a30e8" (UID: "26152753-f94f-11e6-8625-02a3313a30e8"). InnerVolumeSpecName "default-token-6d3fx". PluginName "kubernetes.io/secret", VolumeGidValue "

i deleted the pod using docker kill, stopped the rc. I couldn’t still reproduce the issue.

However, I was able to find out that the error you are seeing is from cadvisor and it is generally caused by docker not cleaning up cgroups after deleting containers.

https://github.com/kubernetes/kubernetes/issues/16651 https://github.com/kubernetes/kubernetes/issues/21022

wlan0 on Feb 22, 2017