k3s: High CPU and Memory Load on single node k3s cluster

HI,

i have an ubuntu 20.02 machine, which has k3s installe. After installing rancher, the memory and cpu consumption of the k3s service is skyrocketing.

versions: “Ubuntu 20.04.1 LTS”

- export INSTALL_K3S_VERSION=v1.19.2+k3s1; curl -sfL https://get.k3s.io | sh -

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
715 root      20   0 4148060   3.0g  52692 S 100.0  38.9  30:03.70 k3s-server

the server running the k3s is a normal x86 vserver with 4 cores and 8 gb ram.

 # install k3s
    - export INSTALL_K3S_VERSION=v1.19.2+k3s1; curl -sfL https://get.k3s.io | sh -

    # install helm 
    - curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
    - starting=true ;while $starting ; do kubectl get ns && starting=false;sleep 1;done

    # install cert-manager
    - kubectl create namespace cert-manager
    - helm repo add jetstack https://charts.jetstack.io
    - helm repo update
    - KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm install cert-manager jetstack/cert-manager --wait --namespace cert-manager --version v1.0.2 --set installCRDs=true

    # install rancher
    - helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
    - helm repo update
    - kubectl create namespace cattle-system
    - KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm install rancher rancher-latest/rancher  --wait --namespace cattle-system --set replicas=1

● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-10-12 09:58:20 CEST; 10min ago
       Docs: https://k3s.io
    Process: 2277 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 2279 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 2280 (k3s-server)
      Tasks: 191
     Memory: 3.3G

any idea what could cause the high load?

kind regards Philipp

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 30 (11 by maintainers)

Most upvoted comments

I’ve just done a

Fresh install of ubuntu server 20.04
Fresh install of k3s
No k8s/k3s workloads running. No other workloads, either.

I am also seeing 20% CPU usage on an intel i5. If k3s is suitable for raspberry pi level hardware, why does it need 20% constant cpu to do nothing?

Hats off for a super slick installation procedure, though!

pletnes on Jan 26, 2022

Hey, This “bug” seems to be the most relatable one to my issue. Since upgrading to >=1.21 .0 the cpu usage on my raspberry is ~20% higher. I have tried up to v1.21.2+k3s1 but eventually rolled back to v1.20.8+k3s1. I’m on armv7 and I remember that there was a regression with kubernetes and the go version build for armv7 (I can’t remember exactly when that was). I had that issue before and was happy when it finally dissipated with a newer version. with 1.21 the issue seems to be back. I’ll attach a screenshot of the cpu usage. (The timeframes are quite large because I do not spend too much time up/downgrading). Upgrade to 1.21: Downgrade back to 1.20 downgrade to 1.20

transacid on Jul 15, 2021

The perf regression in 1.21 is an upstream issue, ref:

https://github.com/kubernetes/kubernetes/issues/101989

brandond on Jul 28, 2021

I can confirm that I’ve had the exact same experience as @transacid. Downgrading immediately rectified the problem. To rule other factors out, I’ve even attempted a clean install, a reinstall, and a full node restart. All in isolation. The downgrade made the biggest impact.

EDIT: I was wrong. It appears that downgrading only alleviates the problem, but over time, the CPU peaks again.

MrSaints on Aug 27, 2021

Load average doesn’t really correspond to much in terms of resource utilization. There’s a reason it’s not used for scheduling. Just keep an eye on your CPU and memory.

brandond on Oct 13, 2020