kubernetes: Failed to collect filesystem stats, no such file or directory

Running into this on my AWS cluster. Seems to happen every ~10 seconds.

Nov 02 01:42:57 ip-172-20-0-49 kubelet[13540]: I1102 01:42:57.265901   13540 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/b3f6e3297e2821bf03f6b22f4858d6c92f2187032eed873a3430ac36f2d5a5b5 with output du: cannot access ‘/var/lib/docker/aufs/diff/b3f6e3297e2821bf03f6b22f4858d6c92f2187032eed873a3430ac36f2d5a5b5’: No such file or directory

Perhaps it’s trying to check something that was deleted? It should probably handle this and stop trying to check after it discovers the diff has been deleted.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 37 (36 by maintainers)

Most upvoted comments

@yujuhong @HardySimpson i think there are two reasons about the high cpu usage(v1.1.1 version):

  1. cadvisor itself, of course, when the node In idle state, the cadvisor works just fine, once the node in full load(there are many pods), the cadvisor will take up a lot of resources, you can see the code as the pprof graph: cadvisor_cpu_1 cadvisor_cpu_2
  2. docker or kubelet has not clear resources which related to pods thoroughly(cgroup resources, mount point and network .etcd), of course, in the normal state, it will not reach this problem(we just find this problem in the density test): cgroup

network

+1
magicwang-cn on Nov 26, 2015
Read more comments on GitHub
← kubernetes: Failed to add label for pod security policy limt
kubernetes: failed to create patch: failed to create two-way merge patch: unable to find api field in struct HorizontalPodAutoscalerSpec for the json field "behavior" →