podman: podman 3.0.1 rootless network issues: Connection reset by peer

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I want run a django application in a rootless podman environment. The application starts up and even connects to a DB that is running in the same named network. It even looks like the port is exposed. But when i do a curl localhost:8080 i get a (56) Recv failure: Connection reset by peer

The setup worked with podman 2.2.1 it only fails with 3.0.1

Steps to reproduce the issue:

1. create named network

2. run application with -p “8080:8080”

3. curl localhost:800

when i execute podman exec -it app curl localhost:8080 it shows what i expext.

This is the systemd unit I am using: I removed all the application specific values.

[Unit]
Description=Podman app.service
Wants=network.target
Requires=app-db.service
After=app-db.service
StartLimitIntervalSec=10
StartLimitBurst=5


[Service]
User=podman-user
Group=podman-user
Environment=PODMAN_SYSTEMD_UNIT=%n
TimeoutStartSec=3m

Restart=on-failure
ExecStartPre=/usr/bin/podman run \
        --network=app-network \
    registry.gitlab.com/path/to/image:main \
    python manage.py migrate
ExecStart=/usr/bin/podman run \
    --name app \
        --network=app-network \
            -p "8080:8080" \
    registry.gitlab.com/path/to/image:main \
    waitress-serve --port=8080 app.wsgi:application
ExecStop=/usr/bin/podman stop \
    --ignore \
    -t 10 \
    app
ExecStopPost=/usr/bin/podman rm \
    --ignore \
    -f \
    app

[Install]
WantedBy=multi-user.target default.target

Describe the results you received:

When everything is up and i do a curl localhost:8080 i get a connection reset.

The output of netstat is a bit weird:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      484/sshd            
tcp6       0      0 :::8080                 :::*                    LISTEN      1324/containers-roo 
tcp6       0      0 :::22                   :::*                    LISTEN      484/sshd            

In the podman 2.2.1 setup there is an entry for 0.0.0.0:8080

There is also something else going on. When I manually restart the container with podman restart i first get a 0.0.0.0:8080 is already in use message and then a lot of these:

ERRO[0000] error joining network namespace for container 4f2e652d70422c0d01da082b41ec80e40bc4419b68cb8bd615d80852d61d92e2: error retrieving network namespace at /proc/3210/ns/net: failed to Statfs "/proc/3210/ns/net": no such file or directory

And then I have trouble bringing the container up agasin

Describe the results you expected:

I expect that the exposed port gives me access to the application

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.14
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

(paste your output here)
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.26, commit: '
  cpus: 2
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: debian
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.19.0-14-amd64
  linkmode: dynamic
  memFree: 878088192
  memTotal: 2091941888
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2144333824
  swapTotal: 2144333824
  uptime: 28m 23.71s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 100:3.0.1-2 amd64 [residual-config]
podman/unknown 100:3.0.1-2 arm64
podman/unknown 100:3.0.1-2 armhf
podman/unknown 100:3.0.1-2 ppc64el
podman/unknown 100:3.0.1-2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

I also tried all of the suggestions I could find that are listed in issues that seemed to be related. Like adding --net=slirp4netns:port_handler=slirp4netns

Additional environment details (AWS, VirtualBox, physical, etc.):

qemu+KVM