microk8s: calicp-kube-controllers can't get API Server: context deadline exceeded
I have the problem after having installed microk8s on my Ubuntu 21.10 server:
sudo snap install microk8s --channel=1.23 --classic
Checked the pods and saw that one crushed:
$ microk8s.kubectl get pods -n kube-system
calico-node-c7h46 1/1 Running 1 (7m38s ago) 10m
calico-kube-controllers-5ddf994775-gp8cv 0/1 CrashLoopBackOff 7 (34s ago) 10m
In logs , I see that something is wrong with the API Server:
$ microk8s.kubectl logs calico-kube-controllers-5ddf994775-gp8cv -n kube-system
2022-04-22 13:26:15.311 [INFO][1] main.go 88: Loaded configuration from environment config=&config.Config{LogLevel:"info", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", DatastoreType:"kubernetes"}
W0422 13:26:15.312587 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2022-04-22 13:26:15.313 [INFO][1] main.go 109: Ensuring Calico datastore is initialized
2022-04-22 13:26:25.313 [ERROR][1] client.go 261: Error getting cluster information config ClusterInformation="default" error=Get "https://10.152.183.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": context deadline exceeded
2022-04-22 13:26:25.313 [FATAL][1] main.go 114: Failed to initialize Calico datastore error=Get "https://10.152.183.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": context deadline exceeded
Result of microk8s.inspect
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 46 (12 by maintainers)
Hi could you please try this.
/etc/modulesand add in there a new line:br_netfilter. This will loadbr_netfilterat boot time.sudo microk8s stopto stop MicroK8s services/var/snap/microk8s/current/args/kube-proxyand remove the--proxy-modecompletely.sudo modprobe br_netfilterto load thebr_netfilterif not already loaded.sudo microk8s startto start MicroK8s servicesHey folks, we’ll take a look at this right away. Thank you all for raising this!
The CNI used by default in MicroK8s is calico. Calico works best with the
br_netfilterkernel module loaded. When MicroK8s starts it tries to load thebr_netfiltermodule, if it fails it sets theproxy-modetouserspace. Userspace routing means that the routing is taken care in userspace instead of via iptable rules. This proxy-mode is the oldest mode and is kept for compatibility reasons. The issue you are seeing is that both MicroK8s fails to load the kernel module and calico fails to play well with the userspace routing. Reproducing this issue is not straight forward. I see it happening under certain conditions on Ubuntu 21.10 but not on any of the 18.04, 20.04, 22.04. Maybe some combination of libraries is at fault here that I only happen to find in 21.10.In any case, we will be shipping a patch in the following days for this issue. We would appreciate if you could verify that the edge channel of the track you are using works for you. You can test this by doing a fresh install or refreshing to the respective channel, eg assuming you are on the 1.23 track you can do
sudo snap refresh microk8s --channel=1.23/edge. Thank you and apologies any the trouble we may have caused.Here is a source of errors I would appreciate if you could please eliminate.
When the calico CNI sets up the network it needs to select a network interface through which it will route traffic. In
/var/snap/microk8s/current/args/cni-network/cni.yamlsearch forIP_AUTODETECTION_METHODand you will see that calico will use by default the “first-found” interface to route traffic. It is possible this interface auto-detection method is selecting an inappropriate interface (eg an interface of lxd). Let’s try to provide a hint on which interface it should be used. Edit/var/snap/microk8s/current/args/cni-network/cni.yamland replacefirst-foundwithcan-reach=<IP_IN_NETWORK_TO_BE USED>with<IP_IN_NETWORK_TO_BE USED>being an IP of a machine in the network we want to use for routing traffic. I think that could be the public facing IP of the host. Then reapply thecni.yamlwithmicrok8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml. In case of multi-node clusters we are able to identify where we should route traffic through because we know where the join node is reached from so this problem should not be present in multi-node clusters.Yes, appears as if the error was reintroduced. Added
br_netfilteras well to the /etc/modules file and restarted entire system. No resolution. k3s works smoothly. All pods up and running without restart.ENVIRONMENT
MicroK8s v1.26.0 revision 4390 on a NUC Intel Celeron N5095, 16GB RAM, 1TB SSD MicroK8s v1.26.0 revision 4390 on a NUC Intel Celeron N3350, 4GB RAM, 512GB SSD
Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy
Reproduce:
kc logs pod/calico-kube-controllers-7874bcdbb4-5ftc2 -n kube-system -f
This is happening to me in 1.24, ubuntu 22.04 on raspberry pi
I did
kubectl rollout restart deployment -n kube-system calico-kube-controllersThe pod is still in crashloop.Here is the output of
ethtool --show-offload vxlan.calico