kubernetes: kubelet_running_pod_count metric is not correct

What happened: kubelet_running_pod_count is larger than the pod actually running on the node, as follows:

Getting metic on prometheus:

kubelet_running_pod_count{instance="k8s-node01"}

result is:

kubelet_running_pod_count{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-node01",job="kubernetes-nodes",kubernetes_io_hostname="k8s-node01"}  98

But using kubectl describe node:

kubectl  describe node k8s-node01

result is:

Non-terminated Pods:         (95 in total)

kubelet_running_pod_count is 98 but actually there is only 95 pod running on the node.

What you expected to happen: kubelet_running_pod_count should equal the pod actually running on the node

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.13.2
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release): centos linux 7
• Kernel (e.g. uname -a): 4.14.67-2.el7.centos.x86_64
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others: