weave: weave-npc blocking connections in Kubernetes with no NetworkPolicies (2.6.0)

What you expected to happen?

weave-npc to allow all network connections

What happened?

We upgraded weave in our Kubernetes cluster from version 2.4.1 to 2.6.0 ~2 weeks ago. We’ve been experiencing intermittent network issues since then. In the pod logs I’ve noticed multiple applications unable to communicate with other pods in the cluster. When I checked weave-npc logs, I noticed a ton of blocked connections:

WARN: 2020/01/27 18:45:42.924357 TCP connection from 172.20.34.171:38490 to 100.104.0.5:18000 blocked by Weave NPC.
WARN: 2020/01/27 18:45:42.924378 TCP connection from 172.20.47.38:47620 to 100.104.0.15:80 blocked by Weave NPC.
WARN: 2020/01/27 18:45:45.100358 TCP connection from 172.20.0.14:33848 to 100.117.0.5:8080 blocked by Weave NPC.
WARN: 2020/01/27 18:45:45.100381 TCP connection from 172.20.34.171:38510 to 100.104.0.5:18000 blocked by Weave NPC.
WARN: 2020/01/27 18:45:45.100390 TCP connection from 172.20.0.14:33848 to 100.117.0.5:8080 blocked by Weave NPC.
WARN: 2020/01/27 18:45:46.796350 TCP connection from 172.20.34.171:38510 to 100.104.0.5:18000 blocked by Weave NPC.
WARN: 2020/01/27 18:45:47.884357 TCP connection from 172.20.47.38:47620 to 100.104.0.15:80 blocked by Weave NPC.
WARN: 2020/01/27 18:45:47.884380 TCP connection from 172.20.0.14:33848 to 100.117.0.5:8080 blocked by Weave NPC.
WARN: 2020/01/27 18:45:47.884387 TCP connection from 172.20.34.171:38510 to 100.104.0.5:18000 blocked by Weave NPC.
WARN: 2020/01/27 18:45:49.260360 UDP connection from 100.97.0.8:38592 to 100.118.0.12:53 blocked by Weave NPC.
WARN: 2020/01/27 18:45:49.260381 UDP connection from 100.97.0.8:38592 to 100.118.0.12:53 blocked by Weave NPC.
WARN: 2020/01/27 18:45:49.260389 UDP connection from 100.97.0.7:42516 to 100.118.0.12:53 blocked by Weave NPC.
WARN: 2020/01/27 18:45:49.260395 UDP connection from 100.97.0.7:42516 to 100.118.0.12:53 blocked by Weave NPC.
WARN: 2020/01/27 18:45:49.260401 UDP connection from 100.97.0.4:52150 to 100.118.0.12:53 blocked by Weave NPC.

Note that we don’t have any network policies in our cluster:

$ kubectl get networkpolicies --all-namespaces
No resources found.

How to reproduce it?

I’m unsure how to actually “trigger” the blocked connections. As soon as I saw the timeouts I downgraded weave to 2.4.1 in our cluster to get it back up and running. We don’t observe this behavior in 2.4.1.

It should be noted that we went to 2.4.1 instead of 2.5.2 since we suffered from the default-deny of traffic until network policies are validated which was introduced in 2.5 (and discussed in this issue: https://github.com/weaveworks/weave/issues/3464).

Anything else we need to know?

Kubernetes v1.15.6 managed w/ Kops running in AWS

Versions:

$ weave version: 2.6.0
$ docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:28:26 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.3-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       d7080c1
  Built:            Wed Feb 20 02:26:51 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ uname -a
Linux ip-172-20-118-147 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux

$ kubectl version:
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Not that I don’t have Logs or IP tables output since I don’t have any nodes running v2.6.0 currently since I’ve downgraded everything. If needed I can see about updating back to 2.6.0.

It should be noted that other than the numerous blocked connections I didn’t see anything remarkable in the weave or weave-npc container logs. I did not inspect the IP tables.