podman: firewall-cmd port-forward wont forward to rootless container

/kind bug Description

I don’t get traffic into pod using firewall-cmd por forwarding to rootless pod. I try running pihole in rootless container, and as it needs low ports, I need to do port forward.

Steps to reproduce the issue:

1. start pod as a user:

   cat .config/systemd/user/pihole-container-pod-pihole.service 
   [Unit]
   Description=pihole Podman Container
   After=network.target
   
   [Service]
   Type=simple
   TimeoutStartSec=15
   ExecStartPre=-/usr/bin/rm -f %T/%n-pid %T/%n-cid
   
   ExecStart=/usr/bin/podman run --name pihole \
    --rm -p 1153:53/tcp -p 1153:53/udp -p 1167:67/udp -p 1180:80/tcp -p 1443:443/tcp -v "/var/lib/containers/exported_volumes/pihole-etc:/etc/pihole:Z" -v "/var/lib/containers/exported_volumes/pihole-dnsmasq.d:/etc/dnsmasq.d:Z" --hostname="pihole.ikenet" --memory=512M -e "TZ=Europe/Helsinki" --dns 127.0.0.1,192.168.117.1 --label "io.containers.autoupdate=image" \
    --conmon-pidfile  %T/%n-pid --cidfile %T/%n-cid \
    docker.io/pihole/pihole:latest 
   ExecStop=/usr/bin/sh -c "/usr/bin/podman stop -t "15" `cat %T/%n-cid`"
   ExecStop=/usr/bin/sh -c "/usr/bin/podman rm -f `cat %T/%n-cid`"
   Restart=on-failure
   RestartSec=30
   KillMode=mixed
   PIDFile=%T/%n-pid
   Environment=PODMAN_SYSTEMD_UNIT=%n
   
   [Install]
   WantedBy=default.target
   
   systemctl --user start pihole-container-pod-pihole.service
   

2. open all ports and do the necessary port forward to localhost:1180

   firewall-cmd --list-ports
   firewall-cmd --add-port 80/tcp
   firewall-cmd --add-port 1180/tcp
   firewall-cmd --add-forward-port=port=80:proto=tcp:toaddr=127.0.0.1:toport=1180
   firewall-cmd --list-ports
   8091/tcp 8080/tcp 8443/tcp 3478/udp 10001/udp 10080/tcp 10022/tcp 3000/tcp 8090/tcp 8099/tcp 80/tcp 1180/tcp
   firewall-cmd --list-forward-port
   port=80:proto=tcp:toport=1180:toaddr=127.0.0.1
   

3. try with curl

Describe the results you received:

[root@fediot ~]# firewall-cmd --list-ports
8091/tcp 8080/tcp 8443/tcp 3478/udp 10001/udp 10080/tcp 10022/tcp 3000/tcp 8090/tcp 8099/tcp 80/tcp 1180/tcp
[root@fediot ~]# firewall-cmd --list-forward-port
port=80:proto=tcp:toport=1180:toaddr=127.0.0.1
[root@fediot ~]# curl http://127.0.0.1:1180/admin
[root@fediot ~]# curl http://127.0.0.1/admin
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

Describe the results you expected:

The port 80 should get forwarded to 1180 and into pod, and return OK as the 1180 does.

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 16:37:50 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:                                                                                                          
  arch: amd64                                                                                                  
  buildahVersion: 1.18.0                                                                                       
  cgroupManager: systemd                                                                                       
  cgroupVersion: v2                                                                                            
  conmon:                                                                                                      
    package: conmon-2.0.21-3.fc33.x86_64                                                                       
    path: /usr/bin/conmon                                                                                      
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'                         
  cpus: 2                                                                                                      
  distribution:                                                                                                
    distribution: fedora                                                                                       
    version: "33"                                                                                              
  eventLogger: journald  
  hostname: fediot.ikenet 
  idMappings:                                          
    gidmap:
    - container_id: 0
      host_id: 1023                                    
      size: 1                                          
    - container_id: 1  
      host_id: 165536
      size: 65536
    uidmap:                                                                                                    
    - container_id: 0
      host_id: 1023
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536         
  kernel: 5.10.9-201.fc33.x86_64
  linkmode: dynamic       
  memFree: 612728832                                   
  memTotal: 8337133568                                 
  ociRuntime:    
    name: crun                                         
    package: crun-0.16-3.fc33.x86_64
    path: /usr/bin/crun                                
    version: |-                                        
      crun version 0.16                                                                                        
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0                                      
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux                                            
  remoteSocket:                                        
    path: /run/user/1023/podman/podman.sock
  rootless: true
  slirp4netns:                                         
    executable: /usr/bin/slirp4netns                                                                           
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-    
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 1073737728
  swapTotal: 1073737728
  uptime: 21h 42m 31.35s (Approximately 0.88 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/pihole/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/pihole/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1023/containers
  volumePath: /var/home/pihole/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 16:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.1-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No

Additional environment details (AWS, VirtualBox, physical, etc.):

VM Fedora-IoT