weave: Kube-dns crashed when weave plugin used

What you expected to happen?

Expect that kube-dsn is up after deployment of weave plugin.

What happened?

E0205 15:05:28.519183       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0205 15:05:28.519486       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0205 15:05:29.018365       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:29.518414       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:30.018394       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:30.518408       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:31.018377       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:31.518468       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:32.018393       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:32.518415       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:33.018413       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:33.518439       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:34.018393       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:34.518386       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:35.018409       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:35.518429       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:36.018444       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:36.518384       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:37.018413       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:37.518474       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:38.018362       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:38.518355       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:39.018387       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:39.518432       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:40.018414       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:40.518397       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:41.018433       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:41.518866       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:42.018433       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:42.518424       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:43.018396       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:43.518485       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:44.018403       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:44.518406       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:45.018414       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:45.518460       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:46.018373       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:46.518308       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:47.018392       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:47.518437       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:48.018438       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:48.518390       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:49.018386       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:49.518444       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:50.018406       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:50.518386       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:51.018364       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:51.518433       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:52.018375       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:52.518384       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:53.018387       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:53.518378       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:54.018393       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:54.518388       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:55.018362       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:55.520461       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:56.018395       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:56.518402       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:57.018370       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:57.518387       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0205 15:05:58.018384       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
F0205 15:05:58.518384       1 dns.go:167] Timeout waiting for initialization

How to reproduce it?

Deploy k8s via kubeadm # kubeadm init --pod-network-cidr=10.244.0.0/16 --token abcdef.1234567890123456 # export KUBECONFIG=/etc/kubernetes/admin.conf # kubever=$(kubectl version | base64 | tr -d '\n') # kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"

Anything else we need to know?

Versions:

weave version
weaveworks/weave-kube:2.2.0
weaveworks/weave-npc:2.2.0

$ docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.centos.1.x86_64
 Go version:      go1.8.3
 Git commit:      3e8e77d/1.12.6
 Built:           Tue Jan 30 09:17:00 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.centos.1.x86_64
 Go version:      go1.8.3
 Git commit:      3e8e77d/1.12.6
 Built:           Tue Jan 30 09:17:00 2018
 OS/Arch:         linux/amd64

$ uname -a
Linux master 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Logs:

weave_0.log weave-npc_0.log

Network:

$ ip route
# ip route
default via 192.168.121.1 dev eth0  proto static  metric 100 
10.32.0.0/12 dev weave  proto kernel  scope link  src 10.32.0.1 
169.254.0.0/16 dev eth1  scope link  metric 1003 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.121.0/24 dev eth0  proto kernel  scope link  src 192.168.121.52  metric 100 
192.168.200.0/24 dev eth1  proto kernel  scope link  src 192.168.200.2

$ ip -4 -o addr
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 192.168.121.52/24 brd 192.168.121.255 scope global dynamic eth0\       valid_lft 1442sec preferred_lft 1442sec
3: eth1    inet 192.168.200.2/24 brd 192.168.200.255 scope global eth1\       valid_lft forever preferred_lft forever
4: docker0    inet 172.17.0.1/16 scope global docker0\       valid_lft forever preferred_lft forever
6: weave    inet 10.32.0.1/12 brd 10.47.255.255 scope global weave\       valid_lft forever preferred_lft forever

$ sudo iptables-save
# Generated by iptables-save v1.4.21 on Mon Feb  5 15:31:55 2018
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [2:120]
:POSTROUTING ACCEPT [2:120]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-P2TFMN4YKJKFJKNH - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:WEAVE - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -j WEAVE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-P2TFMN4YKJKFJKNH -s 192.168.121.52/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-P2TFMN4YKJKFJKNH -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-P2TFMN4YKJKFJKNH --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.121.52:6443
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-P2TFMN4YKJKFJKNH --mask 255.255.255.255 --rsource -j KUBE-SEP-P2TFMN4YKJKFJKNH
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-P2TFMN4YKJKFJKNH
-A WEAVE -s 10.32.0.0/12 -d 224.0.0.0/4 -j RETURN
-A WEAVE ! -s 10.32.0.0/12 -d 10.32.0.0/12 -j MASQUERADE
-A WEAVE -s 10.32.0.0/12 ! -d 10.32.0.0/12 -j MASQUERADE
COMMIT
# Completed on Mon Feb  5 15:31:55 2018
# Generated by iptables-save v1.4.21 on Mon Feb  5 15:31:55 2018
*filter
:INPUT ACCEPT [239:39497]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [227:39605]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -o weave -m comment --comment "NOTE: this must go before \'-j KUBE-FORWARD\'" -j WEAVE-NPC
-A FORWARD -o weave -m state --state NEW -j NFLOG --nflog-group 86
-A FORWARD -o weave -j DROP
-A FORWARD -i weave ! -o weave -j ACCEPT
-A FORWARD -o weave -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m comment --comment "kubernetes forward rules" -j KUBE-FORWARD
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -d 224.0.0.0/4 -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC -m set ! --match-set weave-local-pods dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-?b%zl9GIe0AET1(QI^7NWe*fO dst -m comment --comment "DefaultAllow isolation for namespace: kube-system" -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-0EHD/vdN#O4]V?o4Tx7kS;APH dst -m comment --comment "DefaultAllow isolation for namespace: kube-public" -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-E.1.0W^NGSp]0_t5WwH/]gX@L dst -m comment --comment "DefaultAllow isolation for namespace: default" -j ACCEPT
COMMIT