k3s: Container networking is broken on hosts with default-deny iptables rules after upgrading to v1.26.3

Environmental Info: K3s Version:

k3s version v1.26.3+k3s1 (01ea3ff2)
go version go1.19.7

Node(s) CPU architecture, OS, and Version: Linux CENSORED 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-1 (2023-03-19) x86_64 GNU/Linux

Cluster Configuration:

Local development environment using k3s

Describe the bug:

After upgrading to k3s 1.26.x (version above) from 1.25.x, nothing would come up, even after wiping all k3s config data and starting a fresh cluster. After digging into logs, the issue traced backwards from “coredns not responding” to “coredns stuck waiting on kubernetes service” to the kubernetes service failing to be initialized on the first cluster start attempt, and never attempting repair thereafter:

E0403 15:02:19.425921 1727565 controller.go:156] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocate IP 10.43.0.1: cannot allocate resources of type serviceipallocations at this time

Steps To Reproduce:

Installed K3s: Download binary from github and drop in /usr/local/bin
Start k3s: sudo k3s server --write-kubeconfig-mode 644 --docker --kube-apiserver-arg=service-node-port-range=1024-32767 --tls-san=0.0.0.0

Expected behavior: It should be able to start a cluster

Actual behavior: It fails to start the cluster

Additional context / logs:

Call-out that, due to developer environment, we are using --docker instead of containerd. Unless --docker is no longer supported, please don’t just say “you should use containerd / nerdctl”. We’ve evaluated that, and it is not an easy replacement for our workflows & machine setups right now.

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 64 (32 by maintainers)

Most upvoted comments

I’ll try to give this a test tomorrow, the diff to kube-router looks hopeful

Confirmed this latest fix works for me too, thank you!

mgabeler-lee-6rs on Apr 14, 2023

If you’re going to try to build from source, I would recommend doing so on a host with Docker, and just do git clean -xffd && SKIP_VALIDATE=true make ci

brandond on Apr 3, 2023

The primary differences likely to impact you are updates to flannel and kube-router, the rest of the stuff in there isn’t going to make much difference.

Just out of curiosity, you might try starting k3s with --prefer-bundled-bin, on the off chance there are some problems with the version of iptables your hosts have?

brandond on Apr 3, 2023