k3s: Raspbian 10 fresh install has broken routing (iptables/nf_tables detection)
Version: k3s version v1.17.4+k3s1 (3eee8ac3) on a raspberry pi 4 running Raspbian 10.
K3s arguments:
curl -sfL https://get.k3s.io | sh -
Describe the bug
On a fresh install, no traffic is routed inside the cluster, even for core services. Resolution is to uninstall the default iptables v1.8.2 (nf_tables), and install nftables.
On a fresh install, no traffic is routed inside the cluster. nodes cannot reach each other or coredns. Core services can’t reach each other or the api. Ports are not opened on the physical host.
The host is not listening on port 80. Traefik LB reports that it is listening on port 80, but sudo netstat -tlp |grep 80 disagrees. External hosts cannot access created ingresses.
To Reproduce
- Install k3s on Raspbian 10.
- Run shell in a dnsutils container:
kubectl run -it --rm --restart=Never dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 sh - Inside that container, run
wget -O- github.com, orwget -O- kubernetes.defaultand observe “invalid name” errors. Try pinging any IP you please - the DNS server, external IPs - and observe failures.
Expected behavior Traffic inside the cluster should be routed.
Actual behavior No traffic is routed inside the cluster. Services (even kube-system) can’t reach each other, nothing can reach the API server, etc.
First symptom I noticed was that ingresses failed to open port 80, and services couldn’t reach their pods.
Additional context / logs
Fresh uninstall/reinstall on a raspbian host with IP 192.168.1.41:
$ sudo kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/metrics-server-6d684c7b5-w7swt 1/1 Running 0 25m
pod/coredns-6c6bb68b64-tqs4d 1/1 Running 0 25m
pod/helm-install-traefik-pvsvx 0/1 Completed 0 25m
pod/svclb-traefik-pr5cn 2/2 Running 0 22m
pod/traefik-7b8b884c8-826lt 1/1 Running 0 22m
pod/local-path-provisioner-58fb86bdfd-cdjwd 1/1 Running 2 25m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 25m
service/metrics-server ClusterIP 10.43.106.186 <none> 443/TCP 25m
service/traefik-prometheus ClusterIP 10.43.118.104 <none> 9100/TCP 22m
service/traefik LoadBalancer 10.43.138.141 192.168.1.41 80:30192/TCP,443:31737/TCP 22m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/svclb-traefik 1 1 1 1 1 <none> 22m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/metrics-server 1/1 1 1 25m
deployment.apps/coredns 1/1 1 1 25m
deployment.apps/traefik 1/1 1 1 22m
deployment.apps/local-path-provisioner 1/1 1 1 25m
NAME DESIRED CURRENT READY AGE
replicaset.apps/metrics-server-6d684c7b5 1 1 1 25m
replicaset.apps/coredns-6c6bb68b64 1 1 1 25m
replicaset.apps/traefik-7b8b884c8 1 1 1 22m
replicaset.apps/local-path-provisioner-58fb86bdfd 1 1 1 25m
NAME COMPLETIONS DURATION AGE
job.batch/helm-install-traefik 1/1 2m24s 25m
$ sudo kubectl describe service traefik -n kube-system
Name: traefik
Namespace: kube-system
Labels: app=traefik
chart=traefik-1.81.0
heritage=Helm
release=traefik
Annotations: <none>
Selector: app=traefik,release=traefik
Type: LoadBalancer
IP: 10.43.138.141
LoadBalancer Ingress: 192.168.1.41
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30192/TCP
Endpoints: 10.42.0.6:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31737/TCP
Endpoints: 10.42.0.6:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
$ sudo kubectl describe ingress nginx
Name: nginx
Namespace: default
Address: 192.168.1.41
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
nginx.cluster.vert
/ nginx:80 (<none>)
Annotations:
Events: <none>
$ sudo netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:8125 0.0.0.0:* LISTEN 461/netdata
tcp 0 0 0.0.0.0:19999 0.0.0.0:* LISTEN 461/netdata
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 2754/k3s
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 2754/k3s
tcp 0 0 127.0.0.1:6444 0.0.0.0:* LISTEN 2754/k3s
tcp 0 0 127.0.0.1:10256 0.0.0.0:* LISTEN 2754/k3s
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 523/sshd
tcp 0 0 127.0.0.1:10010 0.0.0.0:* LISTEN 2821/containerd
tcp6 0 0 ::1:8125 :::* LISTEN 461/netdata
tcp6 0 0 :::19999 :::* LISTEN 461/netdata
tcp6 0 0 :::10250 :::* LISTEN 2754/k3s
tcp6 0 0 :::10251 :::* LISTEN 2754/k3s
tcp6 0 0 :::6443 :::* LISTEN 2754/k3s
tcp6 0 0 :::10252 :::* LISTEN 2754/k3s
tcp6 0 0 :::30192 :::* LISTEN 2754/k3s
tcp6 0 0 :::22 :::* LISTEN 523/sshd
tcp6 0 0 :::31737 :::* LISTEN 2754/k3s
This all started with a power loss/reboot of my working pi cluster, after an apt update.
See my eventual resolution. Seems to me that it was applying rules to both nftables and iptables-legacy, and there was some conflict.
UPDATE: changed focus now that I know routing is completely borked. UPDATE 2: rewrite title/description after I discovered/resolved the problem. Left open because I believe it will affect other Raspbian 10 users and could probably use a PR to improve iptables vs nftables behavior.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 16 (1 by maintainers)
Got it! W00tarz!
So here’s the problem, for future frustrated folk:
Raspbian 10 comes with an iptables wrapper around nf_tables in the kernel. So the command iptables exists, but only as a simlink to iptables_nft. It returns version string
iptables v1.8.2 (nf_tables)which seems like it should be correctly handled in check-config.sh. Still, I found firewall entries both iniptables -L(ie nf_tables) andiptables-legacy -L.The fix was to remove the iptables wrapper and explicitly install nftables:
sudo apt remove iptables -y && sudo apt install nftablesI then reinstalled with a reboot for good measure. And hey presto, everything works!
I’m leaving this issue open and re-titling/describing, because I believe this should be common to all recently updated Raspbian 10 installs, and it probably indicates something to be improved in the installer.
@ohthehugemanatee I followed your guidance and ran
sudo apt remove iptables -y && sudo apt install nftables, followed by a reboot than an install of k3s, all on a fresh install of raspbian. Now, k3s won’t start.:~ k3s config-check returnsDid you also setup alternatives as well? It seems like k3s needs to be able to locate an alternative to iptables, but just installing nftables isn’t enough.
ohthehugemanatee you are amazing - I burned up a couple days trying to figure out why this wasn’t working on my Pi Bramble. Your solution worked a charm.
I encountered similar issues. seems like the
kubernetes.default/ 10.43.0.1 route is broken after reboot / further deployment. My temp workaround: