usernetes: VXLAN doesn't seem to work on GCP (while works on AWS and Azure); probably related to MTU

VXLAN doesn’t seem to work on GCP, while it works on AWS and Azure

$ kubectl taint nodes --all node-role.kubernetes.io/control-plane-
$ ./hack/test-smoke.sh 
[INFO] Waiting for nodes to be ready
node/u7s-suda-tmp-1 condition met
node/u7s-suda-tmp-2 condition met
[INFO] Creating StatefulSet "dnstest" and headless Service "dnstest"
service/dnstest created
statefulset.apps/dnstest created
[INFO] Waiting for 3 replicas to be ready
Waiting for 3 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 3 new pods have been updated...
[INFO] Connecting to dnstest-{0,1,2}.dnstest.default.svc.cluster.local
If you don't see a command prompt, try pressing enter.
wget: bad address 'dnstest-0.dnstest.default.svc.cluster.local'
pod "dnstest-shell" deleted
pod default/dnstest-shell terminated (Error)

Likely to be related to MTU.

  • GCP: 1460
  • AWS: 9001
  • Azure: 1500

Version: Usernetes gen2-v20230906.0, Rootless Docker 24.0.6, on Ubuntu 22.04.

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Comments: 21 (6 by maintainers)

Most upvoted comments

/proc/sys/net/bridge/bridge-nf-call-iptables does not exist

You need to modprobe br_netfilter https://github.com/rootless-containers/usernetes/blob/4f81b6e34d331e27ef0b427ed4a7cb819b8d42cb/init-host/init-host.root.sh#L18-L22

Confirmed that VXLAN is functional on GCP with https://github.com/rootless-containers/usernetes/commit/462ccf05dd4931d664ff7cbb3325123a29246dee 🎉

Is it possible to change it, and if so, how?

https://github.com/rootless-containers/usernetes/blob/462ccf05dd4931d664ff7cbb3325123a29246dee/hack/init-host.root.sh#L24-L30

(Also you have to run systemctl --user restart docker.service )