microk8s: Nodes crashed. Kubelet keeps restarting. k8s-dqlite 100% cpu. node.kubernetes.io/unreachable:NoSchedule taint

Summary

Ubuntu 20.04.5 LTS microk8s 1.26.1 installed via snap 3 nodes HA clusters

I came back one morning and my cluster that was working fine the previous day was completely down. Unlike my previous similar experience, this time the disks still have space remaining.

microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.

At this point microk8s inspect would just freeze at the Gathering system information step after printing that the services were running.

I then launched a microk8s start, nothing for a few minutes, then exit with no message. status still say not running.

I tried to check the logs using sudo journalctl -u snap.microk8s.daemon-kubelite but there’s too much stuff, couldn’t find anything relevant.

I then rebooted the nodes sudo reboot and they actually came back online status Ready.

The condition look good:

    Conditions:
    Type Status LastHeartbeatTime LastTransitionTime Reason Message

    NetworkUnavailable False Thu, 23 Feb 2023 04:36:12 +0000 Thu, 23 Feb 2023 04:36:12 +0000 CalicoIsUp Calico is running on this node
    MemoryPressure False Mon, 20 Mar 2023 20:38:48 +0000 Mon, 20 Mar 2023 20:10:26 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
    DiskPressure False Mon, 20 Mar 2023 20:38:48 +0000 Mon, 20 Mar 2023 20:10:26 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
    PIDPressure False Mon, 20 Mar 2023 20:38:48 +0000 Mon, 20 Mar 2023 20:10:26 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
    Ready True Mon, 20 Mar 2023 20:38:48 +0000 Mon, 20 Mar 2023 20:10:26 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled

But there is a Taint preventing any workload to get scheduled on them:

    Taints: http://node.kubernetes.io/unreachable:NoSchedule

Events on pods:

    0/3 nodes are available: 3 node(s) had untolerated taint {http://node.kubernetes.io/unreachable: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling…

Events seems to crash the kubelet service on loop:

    Warning InvalidDiskCapacity 9m38s kubelet invalid capacity 0 on image filesystem
    Normal NodeHasSufficientPID 9m38s kubelet Node jldocker-1 status is now: NodeHasSufficientPID
    Normal NodeAllocatableEnforced 9m38s kubelet Updated Node Allocatable limit across pods
    Normal NodeHasNoDiskPressure 9m38s kubelet Node jldocker-1 status is now: NodeHasNoDiskPressure
    Normal NodeHasSufficientMemory 9m38s kubelet Node jldocker-1 status is now: NodeHasSufficientMemory
    Normal Starting 9m38s kubelet Starting kubelet.
    Normal Starting 7m7s kubelet Starting kubelet.
    Warning InvalidDiskCapacity 7m7s kubelet invalid capacity 0 on image filesystem
    Normal NodeHasNoDiskPressure 7m6s kubelet Node jldocker-1 status is now: NodeHasNoDiskPressure
    Normal NodeAllocatableEnforced 7m6s kubelet Updated Node Allocatable limit across pods
    Normal NodeHasSufficientPID 7m6s kubelet Node jldocker-1 status is now: NodeHasSufficientPID
    Normal NodeHasSufficientMemory 7m6s kubelet Node jldocker-1 status is now: NodeHasSufficientMemory
    Normal Starting 2m57s kubelet Starting kubelet.
    Warning InvalidDiskCapacity 2m57s kubelet invalid capacity 0 on image filesystem
    Normal NodeHasSufficientMemory 2m57s kubelet Node jldocker-1 status is now: NodeHasSufficientMemory
    Normal NodeHasNoDiskPressure 2m57s kubelet Node jldocker-1 status is now: NodeHasNoDiskPressure
    Normal NodeHasSufficientPID 2m57s kubelet Node jldocker-1 status is now: NodeHasSufficientPID
    Normal NodeAllocatableEnforced 2m57s kubelet Updated Node Allocatable limit across pods

The “Invalid Disk Capacity” part seems to be normal, because the pods handling the disk CSI (Longhorn) are not running because of the node taint.

Any command I send to microk8s kubectl print some error message about memcache.go:

    E0320 20:41:19.730002 50241 memcache.go:255] couldn’t get resource list for http://metrics.k8s.io/v1beta1: the server is currently unable to handle the request
    E0320 20:41:19.733484 50241 memcache.go:106] couldn’t get resource list for http://metrics.k8s.io/v1beta1: the server is currently unable to handle the request
    E0320 20:41:19.735017 50241 memcache.go:106] couldn’t get resource list for http://metrics.k8s.io/v1beta1: the server is currently unable to handle the request
    E0320 20:41:19.738322 50241 memcache.go:106] couldn’t get resource list for http://metrics.k8s.io/v1beta1: the server is currently unable to handle the request
    E0320 20:41:20.827654 50241 memcache.go:106] couldn’t get resource list for http://metrics.k8s.io/v1beta1: the server is currently unable to handle the request
    E0320 20:41:20.830214 50241 memcache.go:106] couldn’t get resource list for http://metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Any idea what is causing this? From my point of view, nodes seems healthy, ram, cpu and disk are good, networking between them is also functional.

Introspection Report

Here is my microk8s inspect tarball file hosted on my Google Drive: https://drive.google.com/file/d/1LEaCI3O2Y82OI4Q2quo-uIBD-zV8rZBA/view?usp=sharing

About this issue

Original URL
State: closed
Created a year ago
Comments: 20 (15 by maintainers)

Most upvoted comments

Yes, the whole sale pitch why we spent so much development effort moving to dockers and k8s is stability and resilience, along with providing easy zero downtime updates. Splitting the work load on multiple machines is secondary. Our application don’t have that much processing to do or concurrent requests to process, but just can’t afford to go down.

Our main server is cloud hosted, so no problem there. But we still have a few customers who require an airgapped system not exposed on internet and local data. So with microk8s, the goal was for them just to provide a few Linux machines, and we would handle installation and management.

The current cluster is our dev/staging environment so is not critical, and was used as a test if microk8s was stable enough to use on customers networks. But with the results here of kubelite crashing and dqlite cpu hog issue, I’m not confident at all.

It’s pretty certain it’s due to the host vm itself, but unless I have a way to proves it is flawed in some way, I can’t confirm that.

And now, I still have the coredns issue to fix, which I don’t even know how to start. I have to get my staging environment back on track.

Dunge on Mar 24, 2023