k3s: kubelet does not start even when the agent node is unable to connect to the server node
I have an agent node with some static pods (I’ve set the --kubelet-arg=pod-manifest-path=... argument) but sometimes that node goes offline and could get some reboots. The problem is when the node starts after a reboot and still is offline, the k3s doesn’t start anything (i.e. containerd, kubelet, …) and with that, don’t start the static pods.
Is there any solution or workaround for this use case?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 18 (6 by maintainers)
Want to +1 this.
K3S’s on the homepage
certified Kubernetes distribution built for IoT & Edge computingbut IoT & Edge computing isn’t a data-center. Edge devices almost exclusively have intermittent connectivity. I want pod’s to run on IoT & Edge even if my thing isn’t connected… because unlike a webapp my container/pod is still doing something productive even when it’s not ‘online’.Static pods without a control plane is a great way to get managed app’s on top of firmware. Kubelet also supports this behavior without issue. It’s just really hard to justify putting a 200Mb bin(s) on a embedded device. K3S would be an amazing solution in that place if this issue wasn’t present.
As a device engineer I’d like to see K3S support offline static manifest so I can use it to run edge app’s via pod’s and manage them as I see fit, via manifest updates when they are online and run as last configure, via manifest when they aren’t.
my use case is run specific pods (workloads) on some agent nodes, even the nodes are offline and disconnected from the master. I don’t want DaemonSets, because I want to create/schedule specific pods into specific agent nodes and despite that, to start the pods of a DaemonSet in an agent node that is offline or disconnected from the master, the agent node must have connection to the api-server (or the node must be visible as Ready by the api-server). With static pods that is not necessary, only kubelet running is needed to start static pod on an offline node.
Yeah, having the agent run without server connectivity would be great. Bonus points if the kubelet on server instances would start up independent of the apiserver so that we could do thinks like running etcd or mysql as a static pod.
as described in https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/: “Static Pods are managed directly by the kubelet daemon on a specific node, without the API server observing them”
With that, IMO, I think that kubelet on the k3s process must be started before the agent node tries to connect to the server node.