podman: Cannot set up namespace using /usr/bin/newuidmap - Rocky 9.2 Kernel 5.14 Podman 4.4.1

Issue Description

Describe your issue

When running any podman command (aiming for rootless podman), we get: ERRO[0000] running /usr/bin/newuidmap <pid> 0 83060 1 1 608752 65536 Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

I’ve been getting this issue for a couple of weeks now, since updating my server to Rocky 9.2. We have another server that is configured by someone else that is also running Rocky 9.2, this one seems to be working. But the error we are seeing is reproducible on a rebuild of the server.

Steps to reproduce the issue

Steps to reproduce the issue

1. Execute any podman command. Including podman system migrate

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

Expected podman commands to run, e.g podman ps, would list running containers

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

We cannot run podman info as a user, but as root, we get:

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.el9_2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: e6cdc9a4d6319e039efa13e532c1e58b713c904d'
  cpuUtilization:
    idlePercent: 99.76
    systemPercent: 0.07
    userPercent: 0.18
  cpus: 32
  distribution:
    distribution: '"rocky"'
    version: "9.2"
  eventLogger: journald
  hostname: <not needed>
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-284.11.1.el9_2.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 400244948992
  memTotal: 403770892288
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.el9_2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 0h 39m 45.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 211281186816
  graphRootUsed: 1769046016
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1683632637
  BuiltTime: Tue May  9 11:43:57 2023
  GitCommit: ""
  GoVersion: go1.19.6
  Os: limux
  OsArch: linux/amd64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Running on a Rocky 9.2 box Kernel 5.14 Podman 4.4.1

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting