podman: podman can not run a container with a symlinked /etc dir

/kind bug Description

while trying out https://github.com/remote-android/redroid-doc (a GPU accelerated AIC (Android In Container) solution) running on podman (rootful), podman throws an error:

Error: error creating mtab directory: mkdir /var/lib/containers/storage/overlay/e80f.../merged/etc: file exists

A look inside the container revealed that redroid uses symlinks for:

bin -> /system/bin
etc -> /system/etc
init -> /system/bin/init

and some android related symlinks in the root directory including the symlink of /etc to /system/etc.

while tracing the error message to the corresponding source code i found that it looks like this error message was introduced since https://github.com/containers/podman/pull/10270 which fixed https://github.com/containers/podman/issues/10263 .

the corresponding code creates the directory of /etc/mtab but somehow fails if /etc exists and is a symlink.

it is still in the current main branch (which includes the original modification and also the later added pull request https://github.com/containers/podman/pull/10765) , see: https://github.com/containers/podman/blob/main/libpod/container_internal.go#L1583

Steps to reproduce the issue:

1. podman run --name=redroid --userns=auto --volume=redroid_data:/data docker.io/redroid/redroid:11.0.0-latest

Describe the results you received: podman[3459841]: Error: error creating mtab directory: mkdir /var/lib/containter…

Describe the results you expected: podman starts the container

Additional information you deem important (e.g. issue happens only occasionally): this was probably working before https://github.com/containers/podman/pull/10270 introduced creation of /etc inside the container.

Output of podman version:

Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:                                                                                                                                                                                              
  arch: amd64                                                                                                                                                                                      
  buildahVersion: 1.22.3                                                                                                                                                                           
  cgroupControllers:                                                                                                                                                                               
  - cpuset                                                                                                                                                                                         
  - cpu                                                                                                                                                                                            
  - io                                                                                                                                                                                             
  - memory                                                                                                                                                                                         
  - hugetlb                                                                                                                                                                                          - pids                                                                                                                                                                                             cgroupManager: systemd                                                                                                                                                                             cgroupVersion: v2                                            
  conmon:                                                                                                                                                                                              package: 'conmon: /usr/libexec/podman/conmon'                                                                                                                                                      path: /usr/libexec/podman/conmon                                                                                                                                                                   version: 'conmon version 2.0.30, commit: '                                                                                                                                                       cpus: 4                                                                                                                                                                                            distribution:                                                                                                                                                                                        distribution: ubuntu                                                    
    version: "20.04"                                                                                                                                                                                 eventLogger: journald                                                                                                                                                                              hostname: zap                                                                                                                                                                                      idMappings:                                                                                                                                                                                          gidmap: null                                                                                                                                                                                       uidmap: null                                               
  kernel: 5.11.0-38-generic                                                                                                                                                                          linkmode: dynamic                                                                                                                                                                                  memFree: 956403712                                                                                                                                                                                 memTotal: 16710602752                                                                                                                                                                              ociRuntime:                                                                                                                                                                                          name: crun                                                 
    package: 'crun: /usr/bin/crun'                                                                                                                                                                     path: /usr/bin/crun                                                                                                                                                                                version: |-                                                                                                                                                                                          crun version UNKNOWN                                                                                                                                                                               commit: ea1fe3938eefa14eb707f1d22adff4db670645d6                                                                                                                                                   spec: 1.0.0                                                                                                                                                                                        +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL                                                                                                                                    os: linux                                                                                                                                                                                          remoteSocket:                                                                                                                                                                                        path: /run/podman/podman.sock                                                                                                                                                                    security:                                                                                                                                                                                            apparmorEnabled: true              
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
 serviceIsRemote: false                                                                                                                                                                           
  slirp4netns:                                  
    executable: /usr/bin/slirp4netns                                                             
    package: 'slirp4netns: /usr/bin/slirp4netns'                                                 
    version: |-                                 
      slirp4netns version 1.1.8                                                                  
      commit: unknown                           
      libslirp: 4.3.1-git                       
      SLIRP_CONFIG_VERSION_MAX: 3                                                                
      libseccomp: 2.4.3                         
  swapFree: 9656070144                          
  swapTotal: 9661575168                         
  uptime: 100h 38m 24.05s (Approximately 4.17 days)                                              
registries:                                     
  search:                                       
  - docker.io                                   
  - quay.io                                     
store:                                          
  configFile: /etc/containers/storage.conf                                                       
  containerStore:                               
    number: 5                                   
    paused: 0                                   
    running: 4                                  
    stopped: 1                                  
  graphDriverName: overlay                      
  graphOptions:                                 
    overlay.mount_program:                      
      Executable: /usr/bin/fuse-overlayfs                                                        
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'                                         
      Version: |-                               
        fusermount3 version: 3.9.0                                                               
        fuse-overlayfs: version 1.5                                                              
        FUSE library version 3.9.0                                                               
        using FUSE kernel interface version 7.31                                                 
    overlay.mountopt: nodev,metacopy=on                                                          
  graphRoot: /var/lib/containers/storage                                                         
  graphStatus:                                  
    Backing Filesystem: zfs                     
    Native Overlay Diff: "false"                                                                 
    Supports d_type: "true"                     
    Using metacopy: "false"                     
  imageStore:                                   
    number: 9                                   
  runRoot: /run/containers/storage                                                               
  volumePath: /var/lib/containers/storage/volumes                                                
version:                                        
  APIVersion: 3.3.1                             
  Built: 0                                      
  BuiltTime: Thu Jan  1 01:00:00 1970                                                            
  GitCommit: "" 
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 100:3.3.1-1 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md) No (as the github states 3.4.1 as latest and the ppa is only at 3.3.1)