podman: Containers cannot access published ports of other containers on same host

/kind bug

Description

Containers cannot access published ports of other containers on same host. The port can be accessed from the host itself (via localhost and external ip) and from other hosts.

Steps to reproduce the issue:

1. Start a server and expose the port on the machine: sudo podman run -d --rm -p 8080:80 nginx:alpine

2. Start another pod and try to access the server via the server name / server IP:

   $ sudo podman run -it --rm alpine
   / # wget my-server.my-domain:8080
   Connecting to my-server.my-domain:8080 (10.0.1.4:8080)
   

   The request hangs. Other ports on the host system can be accessed.

Describe the results you received:

The second pod cannot access the service on port 8080. But it can access all other ports on the host system.

Netstat shows that SYN packed is sent.

/ # netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      1 10.88.0.11:48968        10.0.1.4:8080           SYN_SENT

Other ports can be accessed:

/ # nc 10.0.1.4 22
SSH-2.0-OpenSSH_7.4

Describe the results you expected:

The second pod / container should be able to access the published port on the host.

Additional information you deem important (e.g. issue happens only occasionally):

This can be reproduced every time.

I also tried to use the gateway IP to access the published port - this does not work either. Other ports can also be accessed this way:

/ # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.88.0.1       0.0.0.0         UG    0      0        0 eth0
10.88.0.0       *               255.255.0.0     U     0      0        0 eth0
/ # wget 10.88.0.1:8080
Connecting to 10.88.0.1:8080 (10.88.0.1:8080)
/ # nc 10.88.0.1 22
SSH-2.0-OpenSSH_7.4

Output of podman version:

Version:            1.2.0
RemoteAPI Version:  1
Go Version:         go1.10.2
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.2
  podman version: 1.2.0
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2-2.git3bd528e.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: 345710c5d359e8d5b126906e24615d6a3e28c131-dirty'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 6445105152
  MemTotal: 8353083392
  OCIRuntime:
    package: runc-1.0.0-60.dev.git2abd837.el7.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: podman-test
  kernel: 3.10.0-862.11.6.el7.x86_64
  os: linux
  rootless: false
  uptime: 1h 25m 15.91s (Approximately 0.04 days)
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.): Tried on Centos 7.6 on Azure and RHEL 7.6 on-premise (VMWare)