origin: After running oc cluster up and creating a pod/service, the newly created pod can't connect outbound to its own service IP:port

Running on the latest origin vagrant VM + dnf -y update, with latest origin master, after running oc cluster up and creating a pod/service, the newly created pod can’t connect outbound to its own service IP:port.

Version

[root@openshiftdev ~]# uname -a
Linux openshiftdev 4.8.10-100.fc23.x86_64 #1 SMP Mon Nov 21 20:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@openshiftdev ~]# docker version
Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-1.10.3-45.gite03ddb8.fc23.x86_64
 Go version:      go1.5.4
 Git commit:      e03ddb8/1.10.3
 Built:           
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-1.10.3-45.gite03ddb8.fc23.x86_64
 Go version:      go1.5.4
 Git commit:      e03ddb8/1.10.3
 Built:           
 OS/Arch:         linux/amd64
[root@openshiftdev ~]# oc version
oc v1.5.0-alpha.0+7b90443-191
kubernetes v1.4.0+776c994
features: Basic-Auth

Server https://192.168.121.216:8443
openshift v1.5.0-alpha.0+3b2bbe5
kubernetes v1.4.0+776c994

Steps To Reproduce

1. oc cluster up
2. oc create -f examples/gitserver/gitserver-ephemeral.yaml
3. oc rsh git-1-XXXXX
4. sh-4.2$ curl git:8080/_/healthz

Current Result

hangs

Expected Result

outputs the text “ok”

Additional Information

[root@openshiftdev ~]# oc get pods -o wide
NAME          READY     STATUS    RESTARTS   AGE       IP           NODE
git-1-it8uk   1/1       Running   0          9m        172.17.0.3   192.168.121.216
[root@openshiftdev ~]# oc get svc
NAME      CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
git       172.30.211.219   <none>        8080/TCP   9m

[root@openshiftdev ~]# curl 172.17.0.3:8080/_/healthz
ok
[root@openshiftdev ~]# curl 172.30.211.219:8080/_/healthz
ok
[root@openshiftdev ~]# oc rsh git-1-it8uk
sh-4.2$ curl -k https://172.30.0.1:443/healthz
ok
sh-4.2$ curl 172.17.0.3:8080/_/healthz
ok
sh-4.2$ curl 172.30.211.219:8080/_/healthz
<HANGS>

Two interesting additional pieces of information:

1. If I run ifconfig docker0 promisc on the host, things start working fully as expected
2. If I scale up to 2 pods providing the back end service, things “sort of” work, but with irregular long delays:

[root@openshiftdev ~]# oc scale dc/git --replicas=2
deploymentconfig "git" scaled
[root@openshiftdev ~]# oc rsh git-1-it8uk
sh-4.2$ time $ curl 172.30.211.219:8080/_/healthz
ok
real	0m0.003s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m3.072s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m1.052s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m3.092s
user	0m0.001s
sys	0m0.003s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m0.003s
user	0m0.001s
sys	0m0.001s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m0.003s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m0.003s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m0.003s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m0.003s
user	0m0.002s
sys	0m0.001s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m0.003s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m1.013s
user	0m0.001s
sys	0m0.002s
sh-4.2$ time curl 172.30.211.219:8080/_/healthz
ok
real	0m7.133s
user	0m0.000s
sys	0m0.003s