flannel: kube-flannel: host->remote-pod networking does not work on first node

I’ve been trying to use kube-flannel as part of the bootkube, but I’m still seeing an issue on bootstrap.

What I’m seeing is that on some nodes host-network --> remote pod (seemingly) hits some kind of race condition when bootstrapping using kube-flannel.

pod-to-pod (on same node and to remote nodes) works fine, this just affects host to remote pod, and it begins working after restarting kube-flannel on the remote node.

Repro steps:

Launch a cluster:

go get github.com/kubernetes-incubator/bootkube
git remote add aaronlevy https://github.com/aaronlevy/bootkube
cd $GOPATH/src/github.com/kubernetes-incubator/bootkube
git fetch aaronlevy
git checkout -b v1.4.0+flannel aaronlevy/v1.4.0+flannel
make clean all
cd hack/multi-node
./bootkube-up

Start nginx pods for testing:

kubectl --kubeconfig=cluster/auth/kubeconfig run n1 --image=nginx --replicas=2

You should will have one nginx pod on each node, but can verify by checking nodeIP of each pod:

kubectl --kubeconfig=cluster/auth/kubeconfig get pods -owide

Make note of the podIP for each of the pods above, and which node they are assigned to. E.g.:

NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE       IP             NODE
default       n1-1110676790-e0w0c                        1/1       Running   0          12m       10.2.1.2       172.17.4.201
default       n1-1110676790-s3qfi                        1/1       Running   0          12m       10.2.0.5       172.17.4.101

• The 10.2.0.5 is assigned to the “controller” (c1) node (172.17.4.101)
• The 10.2.1.2 is assinged to the “worker” (w1) node (172.17.4.201)

Test the routability from each host:

• note: This seems to fail in different ways sometimes (e.g. both sides, or just one side). Typically from what I’ve seen the common case is master (host) --> worker (pod) fails.

From controller node (route to remote pod likely doesn’t work):

vagrant ssh c1
$ curl 10.2.0.5 # same node, should work
$ curl 10.2.1.2 # different node, probably doesn't work

From worker node (all routes should work):

vagrant ssh w1
$ curl 10.2.0.5 # different node, probably works
$ curl 10.2.1.2 # same node, should work

Now to resolve the issue:

Kill the kube-flannel pod on the failing “remote destination” side (in this case, worker)

kubectl --kubeconfig=cluster/auth/kubeconfig --namespace=kube-system delete pod kube-flannel-abcde

Wait until the kube-flannel daemonset re-launches a replacment pod, then you should be able to re-do the tests above and all networking should work.