flannel: flannel network issue --- cannot ping pod on other nodes

Hi there,

I have a kubernetes cluster with version 1.3.10, now pod on one node cannot ping pod on different node, but can ping other’s docker port.

for example:

root@pcnbj-sto024:/etc/default# kubectl get pods -o wide
NAME                        READY     STATUS    RESTARTS   AGE       IP            NODE
my-nginx-2395715568-4ai66   1/1       Running   2          1d        172.31.11.4   10.148.13.236
my-nginx-2395715568-dyr5w   1/1       Running   1          7h        172.31.15.3   10.140.140.29
my-nginx-2395715568-kpd19   1/1       Running   2          1d        172.31.15.2   10.140.140.29
my-nginx-2395715568-z3k3q   1/1       Running   0          7h        172.31.9.2    10.140.35.98

u see IP of pod on 10.148.13.236 is 172.31.11.4, I cannot ping it from 10.140.140.29:

root@pcnbj-cp027:/var/lib# ping 172.31.11.4
PING 172.31.11.4 (172.31.11.4) 56(84) bytes of data.
^C

but docker0 port on 10.148.13.236 is reachable from 10.140.140.29:

IP address on 10.148.13.236:

root@pcnbj-sto024:/etc/default# ifconfig docker0
docker0   Link encap:Ethernet  HWaddr 02:42:61:43:c7:9a  
          inet addr:172.31.11.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:61ff:fe43:c79a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:57466 errors:0 dropped:0 overruns:0 frame:0
          TX packets:57847 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:11560525 (11.5 MB)  TX bytes:6530376 (6.5 MB)

root@pcnbj-sto024:/etc/default# ifconfig flannel.1
flannel.1 Link encap:Ethernet  HWaddr e6:e0:c3:ee:7a:ac  
          inet addr:172.31.11.0  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::e4e0:c3ff:feee:7aac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7930 errors:0 dropped:0 overruns:0 frame:0
          TX packets:94 errors:0 dropped:11 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:631824 (631.8 KB)  TX bytes:8844 (8.8 KB)

root@pcnbj-cp027:/var/lib# ping 172.31.11.1
PING 172.31.11.1 (172.31.11.1) 56(84) bytes of data.
64 bytes from 172.31.11.1: icmp_seq=1 ttl=64 time=0.267 ms
64 bytes from 172.31.11.1: icmp_seq=2 ttl=64 time=0.234 ms
64 bytes from 172.31.11.1: icmp_seq=3 ttl=64 time=0.231 ms
^C

I also checked flannel ip forward feature are enabled on both nodes, please help me to fingure out what happend, thanks all.

root@pcnbj-cp027:/var/lib# sysctl -a | grep 'net.ipv4.conf.flannel/1.forwarding'
net.ipv4.conf.flannel/1.forwarding = 1