k3s: CrashLoopBackOff / Error

Hello World!

I just installed k3s (Single Master Install), yet seeing Error and/or CrashLoopBackOff for few deployments (that came out of the box)

# k3s kubectl get pods --all-namespaces=true
NAMESPACE     NAME                                      READY   STATUS             RESTARTS   AGE
kube-system   coredns-57d8bbb86-jzhmv                   1/1     Running            0          29m
kube-system   local-path-provisioner-58fb86bdfd-bwhsj   0/1     CrashLoopBackOff   10         29m
kube-system   helm-install-traefik-nmvrz                0/1     CrashLoopBackOff   10         29m
# 

# k3s -v
k3s version v0.10.2 (8833bfd9)
# 

# cat /etc/redhat-release 
CentOS Linux release 8.0.1905 (Core) 
# uname -a
Linux X.X.X 4.18.0-80.11.2.el8_0.x86_64 rancher/k3s#1 SMP Tue Sep 24 11:32:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
#

Please advise.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 10
  • Comments: 30 (9 by maintainers)

Most upvoted comments

I finally got K3S working on Centos 7 and docker, it wasn’t working even after replace firewalld by iptables.

Facts:

So I just added following rule to iptables in order to fix it:

iptables -A INPUT -s 10.42.0.0/16 -d <host_internal_ip>/32 -j ACCEPT

Where 10.42.0.0/16 is the default k3s network CIDR, change it if you defined a different value on --cluster-cidr flag.

The step by step solution:

# Stop and remove k3s
k3s-killall.sh
k3s-uninstall.sh

# Stop and remove all k3s containers
docker stop $(docker ps -a -q --filter "name=k8s_") | xargs docker rm

Replace firewalld by iptables

systemctl stop firewalld
systemctl disable firewalld
yum install iptables-services
systemctl start iptables
systemctl enable iptables

Configure iptables rules, my setup is based on this article: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-basic-iptables-firewall-on-centos-6

# Erase iptables rules
iptables -F

# Block the most common attacks
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP

# Enable outgoing connections
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -P OUTPUT ACCEPT

# Open Traffik http and https ports
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

# Open SSH port
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

# Block everything else
iptables -P INPUT DROP

Here is where the magic happens, enable connections from k3s pods to your host internal ip:

iptables -A INPUT -s 10.42.0.0/16 -d <host_internal_ip>/32 -j ACCEPT

Save changes and restart iptables

iptables-save | sudo tee /etc/sysconfig/iptables
service iptables restart

Install and run k3s

curl -sfL https://get.k3s.io | sh -s - --docker
+18
Lohann on Mar 16, 2020

I’m seeing the same issue, with centos8 and k3s v1.0. Seems like the only way to get it working is to disable/stop firewalld (systemctl stop firewalld) but that’s not really a solution.

+5
ethanmdavidson on Dec 19, 2019

For what it’s worth I wrote a ufw application profile for this - I didn’t feel like turning the firewall off was a solution. Turned out my pods needed to have 6443 (TCP) and 8472 (UDP) as well as access to the host 443 (TCP).

# file: /etc/ufw/applications.d/k3s
# Used to configure the ufw definition for Lightweight Kubernetes (k3s)

[k3s]
title=Lightweight Kubernetes (k3s)
description=K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.
ports=443,6443/tcp|8472/udp

To use this make sure you’ve got your CIDR identified and then run the following (after creating the file above):

sudo ufw app update k3s
sudo ufw allow from $CIDR to any app k3s

This will allow access within the cluster. Seems to have fixed my problems, well at least some of them… 😉

+4
darth-veitcher on Mar 27, 2020

On Ubuntu 20.04.2 LTS confirmed: Error and CrashLoopBackOff

k3s -v k3s version v1.21.2+k3s1 (5a67e8dc) go version go1.16.4

Solution

  1. Disable IPv6 sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 && sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 && sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
  2. Disable firewall (I do have a dedicated IDS/IPS firewall within in my network) sudo ufw disable
  3. Reboot sudo reboot
+3
remkolems on Jun 27, 2021

Sorry? This doesn’t seem like a problem that k3s can easily fix. The root of the problem is with iptables and how they decided to implement the upgrade. If Kubernetes/containerization has issues with nftables support you would think that RHEL would provide legacy support, maybe should file an issue there? Any container which uses legacy iptables will cause problems if the host has switched to nftables.

+2
erikwilson on Dec 2, 2019

I finally got K3S working on Centos 7 and docker, it wasn’t working even after replace firewalld by iptables.

Facts:

So I just added following rule to iptables in order to fix it:

iptables -A INPUT -s 10.42.0.0/16 -d <host_internal_ip>/32 -j ACCEPT

Where 10.42.0.0/16 is the default k3s network CIDR, change it if you defined a different value on --cluster-cidr flag.

The step by step solution:

# Stop and remove k3s
k3s-killall.sh
k3s-uninstall.sh

# Stop and remove all k3s containers
docker stop $(docker ps -a -q --filter "name=k8s_") | xargs docker rm

Replace firewalld by iptables

systemctl stop firewalld
systemctl disable firewalld
yum install iptables-services
systemctl start iptables
systemctl enable iptables

Configure iptables rules, my setup is based on this article: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-basic-iptables-firewall-on-centos-6

# Erase iptables rules
iptables -F

# Block the most common attacks
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP

# Enable outgoing connections
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -P OUTPUT ACCEPT

# Open Traffik http and https ports
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

# Open SSH port
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

# Block everything else
iptables -P INPUT DROP

Here is where the magic happens, enable connections from k3s pods to your host internal ip:

iptables -A INPUT -s 10.42.0.0/16 -d <host_internal_ip>/32 -j ACCEPT

Save changes and restart iptables

iptables-save | sudo tee /etc/sysconfig/iptables
service iptables restart

Install and run k3s

curl -sfL https://get.k3s.io | sh -s - --docker

this doesn’t seem to be working on Ubuntu 22.04. Any ideas on how to proceed?

+1
cguertin14 on Apr 28, 2022

@agilob and @criscola both of you appear to be on Raspberry Pi, whereas this issue appears to be reported against CentOS on amd64. I suggest one or both of you create new issues to track your problem; pods crash-looping is a generic symptom of many potential problems and we can’t troubleshoot them all in the same place.

+1
brandond on Mar 16, 2021

I’m having the same problem with a raspberry pi 4, I tried a couple of times with a fresh install but I’m still getting the error each time:

$ k3s kubectl logs local-path-provisioner-5ff76fc89d-7g5qk -n kube-system
time="2021-03-16T14:57:18Z" level=fatal msg="Error starting daemon: Cannot start Provisioner: failed to get Kubernetes server version: Get https://10.43.0.1:443/version?timeout=32s: dial tcp 10.43.0.1:443: connect: connection refused"

OS is Raspberry Pi OS 64bit

PS If I execute the kill-all script and reboot the server with k3s server start it starts working again, but this is inconvenient.

+1
criscola on Mar 16, 2021

I am seeing this error on a k3s install onto a RPi cluster using Rasbian:

The logs of the process:

kubectl logs local-path-provisioner-58b55cb6b6-fhn4k -n local-path-storage -f time=“2020-03-03T09:33:20Z” level=debug msg=“Applied config: {"nodePathMap":[{"node":"DEFAULT_PATH_FOR_NON_LISTED_NODES","paths":["/opt/local-path-provisioner"]}]}” time=“2020-03-03T09:33:20Z” level=debug msg=“Provisioner started” I0303 09:33:20.827795 1 leaderelection.go:242] attempting to acquire leader lease local-path-storage/rancher.io-local-path… I0303 09:33:47.163803 1 leaderelection.go:252] successfully acquired lease local-path-storage/rancher.io-local-path I0303 09:33:47.164705 1 controller.go:773] Starting provisioner controller rancher.io/local-path_local-path-provisioner-58b55cb6b6-fhn4k_7a01cc7c-f9b5-4015-b401-efeb8a1e86a7! I0303 09:33:47.166570 1 event.go:281] Event(v1.ObjectReference{Kind:“Endpoints”, Namespace:“local-path-storage”, Name:“rancher.io-local-path”, UID:“02dc6406-3334-4c75-a9c2-c388373e6403”, APIVersion:“v1”, ResourceVersion:“46969”, FieldPath:“”}): type: ‘Normal’ reason: ‘LeaderElection’ local-path-provisioner-58b55cb6b6-fhn4k_7a01cc7c-f9b5-4015-b401-efeb8a1e86a7 became leader I0303 09:33:47.265028 1 controller.go:822] Started provisioner controller rancher.io/local-path_local-path-provisioner-58b55cb6b6-fhn4k_7a01cc7c-f9b5-4015-b401-efeb8a1e86a7! I0303 09:39:12.199569 1 leaderelection.go:288] failed to renew lease local-path-storage/rancher.io-local-path: failed to tryAcquireOrRenew context deadline exceeded F0303 09:39:12.199642 1 controller.go:851] leaderelection lost

+1
stevef1uk on Mar 3, 2020
Read more comments on GitHub
← k3s: Crash loop due to apiserver healthcheck error with poststarthook/rbac/bootstrap-roles
k3s: CSI plugin functionality is broken: "attacher.Attach failed: volumeattachments.storage.k8s.io is forbidden" →