weave: Weave Net breaks when host OS uses iptables 1.8

What happened?

I installed Weave Net on a test cluster where the host OS is Debian Buster (aka Debian testing, rolling distro with the ~latest version of everything). After installing Weave, pod<>pod and pod<>internet communication is completely broken, no traffic flows at all, even between pods on the same machine.

I root-caused it to an incompatibility in iptables versions between the weave pods and the host OS. Debian Buster now ships iptables 1.8. It has a major change, which is that the iptables command is now a translating facade on top of nftables, i.e. creating rules with iptables or iptables-restore actually programs nf_tables in the kernel.

OTOH, the weave pod contains iptables 1.6, the previous stable release which programs the “classic” iptables netfilter stack. So, docker on the host OS ends up programming nf_tables rules (because it uses the host iptables 1.8), and weave ends up programming legacy iptables rules (because it uses iptables 1.6). For some reason I don’t fully understand, having both programmed causes packets to get dropped instead of forwarded on the host, before the packets get transmitted to the target container.

How to reproduce it?

I filed extensive reproduction steps in a sibling bug with the Calico folks, please refer to https://github.com/projectcalico/calico/issues/2322#issuecomment-443402871 . The only changes for weave are to use Weave’s pod-network-cidr in kubeadm, obviously install weave instead of calico, and then some of the output changes slightly like different interface names. Everything else, including the core failure mode and the fix, plays out the same.

Additionally, the hacky steps to verify that iptables 1.8 is the problem are at https://github.com/projectcalico/calico/issues/2322#issuecomment-443405705 - basically hackily overwrite the iptables binaries with the ones from debian stable (which still uses 1.6), reboot the machine, and Weave starts working perfectly again.

Anything else we need to know?

There’s also a bug tracking similar problems in core k8s, at https://github.com/kubernetes/kubernetes/issues/71305 . In core k8s this mismatch breaks kube-proxy, but it’s the exact same root cause, mismatched iptables versions.

Versions:

$ weave version

Whichever the latest one is - my test harness is running calico atm, can’t look it up.

$ docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:43 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:06 2018
  OS/Arch:          linux/amd64
  Experimental:     false

$ uname -a

Linux cluster1-controller 4.18.0-2-amd64 #1 SMP Debian 4.18.10-2 (2018-11-02) x86_64 GNU/Linux

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}