kubernetes: cri stats provider: unable to find data in memory cache. Result in InvalidDiskCapacity warning.

What happened?

/sig node

Addresses:
  InternalIP:  10.6.182.198
  Hostname:    ubuntu
Capacity:
  cpu:                4
  ephemeral-storage:  164026132Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8140292Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  151166483001
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8037892Ki
  pods:               110

  Warning  InvalidDiskCapacity      7m3s               kubelet     invalid capacity 0 on image filesystem

kubelet log below

Oct 14 06:08:31 ubuntu kubelet[107256]: E1014 06:08:31.900518 107256 cri_stats_provider.go:452] “Failed to get the info of the filesystem with mountpoint” err=“unable to find data in memory cache” mountpoint=“/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs” Oct 14 06:08:31 ubuntu kubelet[107256]: E1014 06:08:31.900553 107256 kubelet.go:1317] “Image garbage collection failed once. Stats initialization may not have completed yet” err=“invalid capacity 0 on image filesystem” Oct 14 06:08:31 ubuntu kubelet[107256]: I1014 06:08:31.900588 107256 event.go:294] “Event occurred” object=“ubuntu” fieldPath=“” kind=“Node” apiVersion=“” type=“Warning” reason=“InvalidDiskCapacity” message=“invalid capacity 0 on image filesystem” Oct 14 06:08:31 ubuntu kubelet[107256]: I1014 06:08:31.901754 107256 hostutil_linux.go:216] Directory /var/lib/kubelet is already on a shared mount Oct 14 06:08:31 ubuntu kubelet[107256]: I1014 06:08:31.901992 107256 fs_resource_analyzer.go:67] “Starting FS ResourceAnalyzer”

[root@paco ~]# df -h
文件系统                 容量  已用  可用 已用% 挂载点
devtmpfs                 3.9G     0  3.9G    0% /dev
tmpfs                    3.9G     0  3.9G    0% /dev/shm
tmpfs                    3.9G  427M  3.5G   11% /run
tmpfs                    3.9G     0  3.9G    0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   12G   39G   24% /
/dev/mapper/centos-home  102G   33M  102G    1% /home
/dev/sda1               1014M  194M  821M   20% /boot

[root@paco ~]# nerdctl info
Client:
 Namespace:	k8s.io
 Debug Mode:	false

Server:
 Server Version: v1.6.0-721-gafc79c380.m
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: json-file
  Storage: native overlayfs
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.62.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.638GiB
 Name: paco
 ID: 23efba04-6c3c-43c7-a011-4ddd29ecdd5d

What did you expect to happen?

No InvalidDiskCapacity warning.

unable to find data in memory cache?

How can we reproduce it (as minimally and precisely as possible)?

I join a new node of ubuntu with new kernel 5.15 and the node get this warning.

Anything else we need to know?

After some investigations and searching, I found there are several issues similar

• https://github.com/kubernetes/kubernetes/issues/109663
• #113003
• #94825, and #100448 is PR that tries to fix it
• https://discuss.kubernetes.io/t/kubernetes-cluster-doesnt-start-after-installation-i-am-clueless/20340
• https://github.com/opsnull/follow-me-install-kubernetes-cluster/issues/620
• https://github.com/kubernetes/kubernetes/issues/71704
• #108700
• #106977
• #106420
• #112360
• https://github.com/kubernetes/kubernetes/issues/92640

Some suggests to disable LocalStorageCapacityIsolation as a workaround, but it is GAed in v1.25. See https://github.com/kubernetes/kubernetes/issues/71704#issuecomment-901622661 and https://github.com/opsnull/follow-me-install-kubernetes-cluster/issues/620#issuecomment-760654953.

Kubernetes version

Kubernetes 1.25.0 + kubelet v1.25.2

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:38:15Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

vsphere

OS version

ubuntu kernel 5.15

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux ubuntu 5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

kubeadm

Container runtime (CRI) and version (if applicable)

containerd v1.6.8

Related plugins (CNI, CSI, …) and versions (if applicable)

calico v3.24.0 installed by tigera operator.