flannel: L3 miss and Route not found loop

Hi there, I’m running Flannel v0.5.5 in vxlan mode as part of a Kubernetes cluster. I seem to keep getting this repeating:

<Timestamp> <host> flanneld[873]: I0407 18:36:51.705743 00873 vxlan.go:345] L3 miss: <Service's IP>
<Timestamp> <host> flanneld[873]: I0407 18:36:51.705865 00873 vxlan.go:349] Route for <Service's IP> not found

It only happens to one node in the cluster at a time. It would seem like it’s not aware a route has changed/removed and it is still looking for it.

It all was working fine for awhile, then I updated one service and then I keep getting this.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 26 (7 by maintainers)

Most upvoted comments

@eyakubovich is there any way to reduce log level to hide the Ignoring not a miss messages?

+10

jsoriano on Sep 19, 2016

@tomdee can we reopen this issue? I use kargo to deploy a kubernetes env with flannel, the flannel subnet changed by some unknown reason, which is a serious issue.

garyyang85 on Jul 10, 2017

Hi, i have a similar issue with CoreOS 1010.5.0, flannel v0.5.5 in vxLan mode and Kubernetes 1.2.4. My setup is 1 schedulable master and 2 nodes on virtualbox - installed with the “bare metal” guide.

<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:340] Ignoring not a miss: <Mac:address>, <Pod-IP>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:340] Ignoring not a miss: <Mac:address>, <Pod-IP>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:340] Ignoring not a miss: <Mac:address>, <Pod-IP>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:340] Ignoring not a miss: <Mac:address>, <Pod-IP>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:248] Subnet removed: <Pod-Subnet>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 device.go:176] calling NeighDel: <k8s-master-IP>, <Mac:address>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:345] L3 miss: <Pod-IP>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:349] Route for <Pod-IP> not found
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:345] L3 miss: <Pod-IP>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:349] Route for <Pod-IP> not found
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:345] L3 miss: <Pod-IP>
<Timestamp> <host> sdnotify-proxy[770]: <Timestamp> 00001 vxlan.go:349] Route for <Pod-IP> not found
... looping ...

My cluster was running fine. Systemd docker service requires flannel, both did not show any error logs. The issue appeared after some time. Subnets started getting deleted on all nodes: pods lost all connection outside the host node. The machines had been rebooted a lot, but the issue did NOT appear after a reboot or a systemd restart - in which case a race could be considered.

It could be linked to heavy CPU and memory load. I experienced this with several kubernetes and coreos versions, and although I cannot give any evidence, it seems to me that it always happened on a over-loaded cluster. Maybe network load to be considered too ?

After restarting flannel everything seems fine from the pods, but I still see a LOT of Ignoring not a miss: <Mac:address>, <Some-Pod-IP> in flanneld logs.

sen-lgtm on Jun 23, 2016