kubernetes: `kubectl get pod` is very slow, it takes about 15 seconds，is there any way to improve it？

The issue should be reopened as I can’t find any solution too!

We sloved this problem，kubelctl cache the result from apiserver but it’s really slow because it fsync everytime ! So we link .kube/cache and http_cache dir to /dev/shm .After that everything works really good!

I am not sure why but this is something I am facing with k3s running on openSUSE Leap 15.2. Reinstalling didn’t work till now

@rogperez What worked for me was to clean up the host nodes and reinstalling kubernetes (but i was using rke). Here were the steps you must do for all your nodes:

1. Clean up Docker

docker rm -f $(docker ps -qa)
docker rmi -f $(docker images -q)
docker volume rm $(docker volume ls -q)

1. Unmount Volumes (need su permission for it)

for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done

1. Remove all these directories

sudo rm -rf /etc/ceph \
       /etc/cni \
       /etc/kubernetes \
       /opt/cni \
       /opt/rke \
       /run/secrets/kubernetes.io \
       /run/calico \
       /run/flannel \
       /var/lib/calico \
       /var/lib/etcd \
       /var/lib/cni \
       /var/lib/kubelet \
       /var/lib/rancher/rke/log \
       /var/log/containers \
       /var/log/pods \
       /var/run/calico

1. Remove Interface

ip address show
ip link delete flannel.1

1. Clean up your .kube folder (basically delete and make dir again)
2. Reinstall kubernetes

Like i said i was using Rancher - RKE for my setup and this worked. So everytime I have installed k8s on a fresh VM (or bare-metal) I have always had this issue of kubectl being slow and fortunately the only solution was to clean up and reinstall on the same machines.

version 1.27

[#] time kubectl get pod -o wide | wc -l 3

real 0m0.044s user 0m0.042s sys 0m0.023s

[#] time kubectl get pod -n prod -o wide | wc -l 3073

real 0m3.452s user 0m3.615s sys 0m0.308s

Just to provide some details, it seems to me that is cause by the difference in the kubectl version and the k8s cluster. Observed the same issue with v1.16.13-eks-2ba888. It’s also notable that the slowdown is mostly visible when executing get all and get pods commands, but it’s missing when execute get nodes.

Here are the results from the experiments that I did:

# time /tmp/kubectl_1.20.2 get all >/dev/null
real	0m6.451s
user	0m0.661s
sys	0m0.087s

# time /tmp/kubectl_1.19.7 get all >/dev/null
real	0m7.661s
user	0m0.659s
sys	0m0.106s

# time /tmp/kubectl_1.18.15 get all >/dev/null
real	0m6.451s
user	0m0.649s
sys	0m0.085s

# time /tmp/kubectl_1.17.17 get all >/dev/null
real	0m6.433s
user	0m0.631s
sys	0m0.100s

# time /tmp/kubectl_1.16.15 get all >/dev/null
real	0m2.560s
user	0m0.627s
sys	0m0.082s

# time /tmp/kubectl_1.15.12 get all >/dev/null
real	0m2.632s
user	0m0.647s
sys	0m0.058s

Here is the same test using get nodes

# time /tmp/kubectl_1.20.2 get nodes >/dev/null
real	0m1.243s
user	0m0.564s
sys	0m0.086s

# time /tmp/kubectl_1.19.7 get nodes >/dev/null
real	0m1.199s
user	0m0.580s
sys	0m0.077s

# time /tmp/kubectl_1.18.15 get nodes >/dev/null
real	0m1.159s
user	0m0.576s
sys	0m0.074s

# time /tmp/kubectl_1.17.17 get nodes >/dev/null
real	0m1.175s
user	0m0.610s
sys	0m0.048s

# time /tmp/kubectl_1.16.15 get nodes >/dev/null
real	0m1.145s
user	0m0.562s
sys	0m0.085s

# time /tmp/kubectl_1.15.12 get nodes >/dev/null
real	0m1.272s
user	0m0.598s
sys	0m0.060s