k3s: the overlay network between nodes seems to be broken in Debian 11 (bullseye)

Environmental Info:

K3s Version:

k3s version v1.21.3+k3s1 (1d1f220f)
go version go1.16.6

Node(s) CPU architecture, OS, and Version:

root@s1:~# uname -a
Linux s1 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux
root@s1:~# kubectl get nodes -o wide
NAME   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
s1     Ready    control-plane,master   17m   v1.21.3+k3s1   10.11.0.101   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-8-amd64   containerd://1.4.8-k3s1
a1     Ready    <none>                 14m   v1.21.3+k3s1   10.11.0.201   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-8-amd64   containerd://1.4.8-k3s1

Cluster Configuration:

1 server (without worker role) and 1 agent as configured by my playground at https://github.com/rgl/k3s-vagrant/tree/debian-11-wip.

Describe the bug:

the traefik dashboard is installed as https://github.com/rgl/k3s-vagrant/blob/debian-11-wip/provision-k3s-server.sh#L62-L167 and is available in all the cluster nodes at:

when running in Debian 10, accessing both addresses works fine (they return the expected web page). but when running in Debian 11, the a1 node address does not work (the request times out).

it does not even work when trying a wget http://10.11.0.201:9000/dashboard/ from the s1 node (but works inside the a1 node).

while running tcpdump at the s1 and a1 nodes, I can see the SYN packets leave s1 to a1, but a1 never replies.

do you have any idea why this is happening or what might be blocking this? or any clue how to make it work?

PS when launching the server with --flannel-backend 'host-gw' things seem to work. so it seems there’s something going on with the vxlan backend.

Steps To Reproduce:

  1. install k3s like https://github.com/rgl/k3s-vagrant/blob/wip-vxlan/provision-k3s-server.sh#L46-L61

PS: That is running in KVM VMs (using virtio nic) on a Ubuntu 20.04 host. The code is at https://github.com/rgl/k3s-vagrant/tree/wip-vxlan.

Expected behavior:

expected it to work in Debian 11, like it does in Debian 10.

Actual behavior:

traffic between nodes in the overlay network does not seem to be working correctly.

Additional context / logs:

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 21 (6 by maintainers)

Most upvoted comments

I’m still seeing this problem on v1.21.4+k3s1 on bulleseye, in my case 2 Hetzner cloud instances. For testing I just increased coredns to 2 replicas to have one on each node, then I tested DNS resolution via gcr.io/kubernetes-e2e-test-images/dnsutils:1.3, querying both the local node endpoint and the remote node endpoint.

kubectl --context test-hcloud-k3s get pods -A -o wide

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
kube-system   local-path-provisioner-5ff76fc89d-pv8nx   1/1     Running   0          10m     10.42.0.2   test-hc-k3s-master-0   <none>           <none>
kube-system   coredns-7448499f4d-tcb8g                  1/1     Running   0          10m     10.42.0.4   test-hc-k3s-master-0   <none>           <none>
kube-system   metrics-server-86cbb8457f-8rq25           1/1     Running   0          10m     10.42.0.3   test-hc-k3s-master-0   <none>           <none>
kube-system   coredns-7448499f4d-vdkv5                  1/1     Running   0          6m21s   10.42.1.2   test-hc-k3s-agent-0    <none>           <none>
default       dnsutils                                  1/1     Running   0          5m55s   10.42.0.5   test-hc-k3s-master-0   <none>           <none>

kubectl --context test-hcloud-k3s describe endpoints kube-dns -n kube-system

Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
              objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-08-22T13:44:03+02:00
Subsets:
  Addresses:          10.42.0.4,10.42.1.2
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP
    metrics  9153  TCP

Events:  <none>

Now when I try to query the pod on the master node from the dnsutils container on the master node everything is working fine: kubectl --context test-hcloud-k3s exec -it dnsutils -- dig +search kube-dns.kube-system @10.42.0.4

; <<>> DiG 9.11.6-P1 <<>> +search kube-dns.kube-system @10.42.0.4
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49634
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: fd2f9e9d0725fd6b (echoed)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.cluster.local.	IN A

;; ANSWER SECTION:
kube-dns.kube-system.svc.cluster.local.	5 IN A	10.43.0.10

;; Query time: 0 msec
;; SERVER: 10.42.0.4#53(10.42.0.4)
;; WHEN: Sun Aug 22 11:51:21 UTC 2021
;; MSG SIZE  rcvd: 133

root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

13:51:21.306554 vethe10710cb P   IP 10.42.0.5.58375 > 10.42.0.4.53: 8851+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:51:21.306624 veth5e515acc Out IP 10.42.0.5.58375 > 10.42.0.4.53: 8851+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:51:21.307048 veth5e515acc P   IP 10.42.0.4.53 > 10.42.0.5.58375: 8851 NXDomain*- 0/1/1 (180)
13:51:21.307068 vethe10710cb Out IP 10.42.0.4.53 > 10.42.0.5.58375: 8851 NXDomain*- 0/1/1 (180)
13:51:21.307295 vethe10710cb P   IP 10.42.0.5.40611 > 10.42.0.4.53: 49634+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
13:51:21.307322 veth5e515acc Out IP 10.42.0.5.40611 > 10.42.0.4.53: 49634+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
13:51:21.307477 veth5e515acc P   IP 10.42.0.4.53 > 10.42.0.5.40611: 49634*- 1/0/1 A 10.43.0.10 (133)
13:51:21.307484 vethe10710cb Out IP 10.42.0.4.53 > 10.42.0.5.40611: 49634*- 1/0/1 A 10.43.0.10 (133)

Now let’s repeat the same, querying the pod on the agent node from the dnstest container on the master node: kubectl --context test-hcloud-k3s exec -it dnsutils -- dig +search kube-dns.kube-system @10.42.1.2

; <<>> DiG 9.11.6-P1 <<>> +search kube-dns.kube-system @10.42.1.2
;; global options: +cmd
;; connection timed out; no servers could be reached
command terminated with exit code 9

root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

13:57:15.906884 vethe10710cb P   IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.906884 cni0  In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.906942 flannel.1 Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.909608 flannel.1 P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.906768 vethe10710cb P   IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.906768 cni0  In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.906877 flannel.1 Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.907807 flannel.1 P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.906785 vethe10710cb P   IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.906785 cni0  In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.906828 flannel.1 Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908381 flannel.1 P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)

root@test-hc-k3s-agent-0:~# tcpdump -i any -nn port 53

13:57:15.907722 flannel.1 In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.907779 cni0  Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.907789 vethb0b8faff Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.909179 vethb0b8faff P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:15.909196 cni0  In  IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:15.909239 flannel.1 Out IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.907678 flannel.1 In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.907698 cni0  Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.907707 vethb0b8faff Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.908064 vethb0b8faff P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.908097 cni0  In  IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.908113 flannel.1 Out IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.908036 flannel.1 In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908062 cni0  Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908074 vethb0b8faff Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908900 vethb0b8faff P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.908918 cni0  In  IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.908962 flannel.1 Out IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)

It does seem to be an improvement over v1.21.3+k3s1 though as with v1.21.3+k3s1 I don’t even see a response in tcpdump at all:

kubectl --context test-hcloud-k3s get pods -A -o wide

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
kube-system   coredns-7448499f4d-zb92q                  1/1     Running   0          8m31s   10.42.0.2   test-hc-k3s-master-0   <none>           <none>
kube-system   local-path-provisioner-5ff76fc89d-5gw2r   1/1     Running   0          8m31s   10.42.0.3   test-hc-k3s-master-0   <none>           <none>
kube-system   metrics-server-86cbb8457f-gsghc           1/1     Running   0          8m31s   10.42.0.4   test-hc-k3s-master-0   <none>           <none>
default       dnsutils                                  1/1     Running   0          6m26s   10.42.0.5   test-hc-k3s-master-0   <none>           <none>
kube-system   coredns-7448499f4d-qpwk8                  1/1     Running   0          5m43s   10.42.1.2   test-hc-k3s-agent-0    <none>           <none>

kubectl --context test-hcloud-k3s describe endpoints kube-dns -n kube-system

Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
              objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-08-22T14:04:53+02:00
Subsets:
  Addresses:          10.42.0.2,10.42.1.2
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP
    metrics  9153  TCP

Events:  <none>

local request within master node: root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

14:05:32.404481 veth7c207c5b P   IP 10.42.0.5.53914 > 10.42.0.2.53: 63669+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:32.404578 vethdb968d68 Out IP 10.42.0.5.53914 > 10.42.0.2.53: 63669+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:32.405599 vethdb968d68 P   IP 10.42.0.2.53 > 10.42.0.5.53914: 63669 NXDomain*- 0/1/1 (180)
14:05:32.405626 veth7c207c5b Out IP 10.42.0.2.53 > 10.42.0.5.53914: 63669 NXDomain*- 0/1/1 (180)
14:05:32.407124 veth7c207c5b P   IP 10.42.0.5.46851 > 10.42.0.2.53: 18114+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
14:05:32.407200 vethdb968d68 Out IP 10.42.0.5.46851 > 10.42.0.2.53: 18114+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
14:05:32.407557 vethdb968d68 P   IP 10.42.0.2.53 > 10.42.0.5.46851: 18114*- 1/0/1 A 10.43.0.10 (133)
14:05:32.407568 veth7c207c5b Out IP 10.42.0.2.53 > 10.42.0.5.46851: 18114*- 1/0/1 A 10.43.0.10 (133)

request from master node to agent node: root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

14:05:38.526888 veth7c207c5b P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:38.526888 cni0  In  IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:38.526958 flannel.1 Out IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.526755 veth7c207c5b P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.526755 cni0  In  IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.526803 flannel.1 Out IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.527231 veth7c207c5b P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.527231 cni0  In  IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.527369 flannel.1 Out IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)

root@test-hc-k3s-agent-0:~# tcpdump -i any -nn port 53

14:05:38.519772 flannel.1 P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.519393 flannel.1 P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.520172 flannel.1 P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
+2
Nothing4You on Aug 22, 2021
Read more comments on GitHub
← k3s: systemd-shutdown hangs on containerd-shim when k3s-agent running
k3s: Too many open files →