kubernetes: Kubelet Is Not Reporting Containers Memory Usage

What happened?

I had a problem with metric-server and I figured out the problem originated from kubelet (or maybe cAdvisor). both /metrics/resource and /stats/summary kubelet API endpoints are reporting zero memory working set for containers, however, for pods they are reporting memory usage:

kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/stats/summary

CPU usage values are shown correctly and non-zero.

What did you expect to happen?

non-zero values for containers memory usage.

How can we reproduce it (as minimally and precisely as possible)?

kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/metrics/resource > metrics-resource.log kubelet-metrics-resource.log

kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/stats/summary > stat-summary.log stats-summary.log

Anything else we need to know?

I have installed kubernetes on a single node. Screen Shot 2022-06-16 at 1 21 48 AM

Kubernetes version

$ kubectl version
# paste output here
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
# paste output here
Linux khavari 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

kubeadm

Container runtime (CRI) and version (if applicable)

``` docker://20.10.16 ```

Related plugins (CNI, CSI, …) and versions (if applicable)

flannel

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (6 by maintainers)

Most upvoted comments

Both containerd and CRI-O can run Docker-formatted (actually OCI-formatted) images, they just do it without having to use the docker command or the Docker daemon.

Yes, containerd can run docker images. See the blog Dockershim Deprecation FAQ for more details. @ahmetemrebasakcioglu

mengjiao-liu on Jun 24, 2022

After migrated to containerd now, everything works fine, thanks @mengjiao-liu @mohammadkhavari

ahmetemrebasakcioglu on Jun 24, 2022

@ahmetemrebasakcioglu I didn’t use helm but it’s heartwarming that im not the only one who had faced this issue. By the way I didn’t understand well, does kubectl top pod work for you?

Oh sorry, i was written wrong, it doesnt work.

ahmetemrebasakcioglu on Jun 21, 2022