kubernetes: fsHandler.go:121: failed to collect filesystem stats

/kind bug /sig node

What happened: I run kubelet at physical machine and find below logs with systemctl status kubelet:

kubelet[182959]: E0720 01:06:15.454048  182959 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff with output stdout: 4085180        /software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff
kubelet[182959]: , stderr: du: cannot access ‘/software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff/usr/share/elasticsearch/data/nodes/0/indices/rls2BZc3TEaxM0dufaYDRQ/0/index/_kbf.si’: No such file or directory
kubelet[182959]: - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>
kubelet[182959]: E0720 01:27:19.828180  182959 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff with output stdout: 4160724        /software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff
kubelet[182959]: , stderr: du: cannot access ‘/software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff/usr/share/elasticsearch/data/nodes/0/indices/KhMTs8WhTwyahUoPEuHQHQ/0/index/_1gs7_Lucene50_0.tip’: No such file or directory
kubelet[182959]: du: cannot access ‘/software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff/usr/share/elasticsearch/data/nodes/0/indices/KhMTs8WhTwyahUoPEuHQHQ/0/index/_1gs7.nvm’: No such file or directory
kubelet[182959]: du: cannot access ‘/software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff/usr/share/elasticsearch/data/nodes/0/indices/KhMTs8WhTwyahUoPEuHQHQ/0/index/_1gs7.dim’: No such file or directory
kubelet[182959]: du: cannot access ‘/software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff/usr/share/elasticsearch/data/nodes/0/indices/KhMTs8WhTwyahUoPEuHQHQ/0/index/_1gs7.fnm’: No such file or directory
kubelet[182959]: - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>
kubelet[182959]: E0720 05:58:03.368413  182959 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff with output stdout: 4267452        /software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff
kubelet[182959]: , stderr: du: cannot access ‘/software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff/usr/share/elasticsearch/data/nodes/0/indices/3qK3SN_zToGdmqJ5PdEkQA/3/index/_1efi.cfs’: No such file or directory
kubelet[182959]: - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>

/software/docker/overlay2/b9bb924fa49e4f1414894eba364c2c80f6e3924122bcc01d09b26a300ba559a9/diff exists, but .cfs,.dim files and so on do not exist.

But i check all my pods, they are all fine.

I really do not know should i ignore them or not.

I look some issues like above but do not get useful info.

I tired to restart kubelet service, the logs miss, but after a while, above logs occur again.

My environment is CentOS7.4, and k8s version is:

$ kubelet --version
Kubernetes v1.9.7
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-18T23:58:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 24 (8 by maintainers)

Most upvoted comments

same with k8s 1.11.7 on AWS with EBS root volume

Feb 11 22:28:35 ip-10-40-11-167 kubelet[2100]: E0211 22:28:35.210519    2100 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: failed to exec du - fork/exec /usr/bin/ionice: resource temporarily unavailable, rootInodeErr: failed to exec cmd [ionice -c3 nice -n 19 find /var/lib/docker/overlay2/33b025d9a22abb7cceb62b6901f26aec006c985d7608302fcc1e6a588c6a408b/diff -xdev -printf .] - fork/exec /usr/bin/ionice: resource temporarily unavailable; stderr: , extraDiskErr: failed to exec du - fork/exec /usr/bin/ionice: resource temporarily unavailable

edit: after researching a lot, I finally figured out what the actual issue here was. I don’t think it’s related. One docker container created too many threads (over 20k) and so the node ran into an issue with no more pid’s available, so fork couln’t create new processes checking the file system usage. this SO article helped me

m1schka on Feb 12, 2019

same problem with k8s v1.18.3 E0623 22:03:40.147960 35377 fsHandler.go:118] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/docker/overlay2/ea530e7d335a9f8dcafead409294cf42476eb57d86c7b0b0e34d281a53393032/diff" to get inode usage: stat /var/lib/docker/overlay2/ea530e7d335a9f8dcafead409294cf42476eb57d86c7b0b0e34d281a53393032/diff: no such file or directory, extraDiskErr: could not stat "/var/lib/docker/containers/b256dd634c306c260e53eaa6ef42a4d7db0d376b66ea31141e98d0b9db4a7af8" to get inode usage: stat /var/lib/docker/containers/b256dd634c306c260e53eaa6ef42a4d7db0d376b66ea31141e98d0b9db4a7af8: no such file or directory

takakawa on Jun 23, 2020

same problem with k8s v1.14.2

Dionysusio on May 29, 2019