podman: Failing to run MPI Podman example on HPC cluster with subuid/subgid mapping restrictions

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

We’re trying to test out Podman on one of our HPC clusters at ORNL, and we are setting up rootless Podman to do it. For security and administrative reasons, we can’t set up and maintain the userid mappings in /etc/subuid and /etc/subgid. We’ve gotten to the point where we can get a single container running or multiple containers running as a job with Slurm started with mpirun. But these are containers that are separate from each other i.e. not talking to each other. Now we’re trying to figure out how to do MPI where the containers are talking to each other. I’m following along with this tutorial. However, I am getting the error seen in the ‘Describe your results’ section. This is similar to this other Github issue but there their issue seems to be resolved with setting up the mapping in the /etc/subuid /etc/subgid files. However, that is not a mapping that we can maintain due to the administrative overhead of having to maintain a mapping for hundreds of users on each of the nodes in the cluster (plus adding new ones all the time). And providing setuid capabilities for the newuidmap and newgidmap is something our security folks have pushed back against.

Is there a way to provide the MPI functionality with rootless Podman with these restrictions in mind? And if you can point me to other centers that have deployed Podman successfully with these restrictions, that would also be helpful.

For reference, the program I am testing with is an MPI Ring program, and the following command in the job script to run, similar to what is described in the tutorial blog post.

mpirun -np 4 podman -v --cgroup-manager=cgroupfs run --userns=keep-id --env-host -v /tmp/podman_mpi_tmp:/tmp/podman_mpi_tmp --net=host --pid=host --ipc=host localhost/centosmpi /home/mpi_ring

Steps to reproduce the issue:

Describe the results you received:

time="2020-12-02T17:27:19-05:00" level=error msg="cannot find UID/GID for user subil: No subuid ranges found for user \"subil\" in /etc/subuid - check rootless mode in man pages."
time="2020-12-02T17:27:19-05:00" level=error msg="cannot find UID/GID for user subil: No subuid ranges found for user \"subil\" in /etc/subuid - check rootless mode in man pages."
time="2020-12-02T17:27:19-05:00" level=error msg="cannot find UID/GID for user subil: No subuid ranges found for user \"subil\" in /etc/subuid - check rootless mode in man pages."
time="2020-12-02T17:27:19-05:00" level=error msg="cannot find UID/GID for user subil: No subuid ranges found for user \"subil\" in /etc/subuid - check rootless mode in man pages."
time="2020-12-02T17:27:19-05:00" level=error msg="cannot find UID/GID for user subil: No subuid ranges found for user \"subil\" in /etc/subuid - check rootless mode in man pages."
time="2020-12-02T17:27:19-05:00" level=error msg="cannot find UID/GID for user subil: No subuid ranges found for user \"subil\" in /etc/subuid - check rootless mode in man pages."
Error: chown /run/user/15377/containers/overlay-containers/6cd5784303d85eb01cfc931102243fb640fc2cb139e349cb395d2040641d3ef9/userdata: invalid argument
Error: chown /run/user/15377/containers/overlay-containers/ec5cb0449b5bb835803a18b93c9205767bc96e153c07e9749cdb277d4c109fe2/userdata: invalid argument
Error: chown /run/user/15377/containers/overlay-containers/c690c88eda9a75076e4371a9d46b0e181ec4ad0775c46e4fe11e8d44f1d7666d/userdata: invalid argument
time="2020-12-02T17:27:19-05:00" level=error msg="cannot find UID/GID for user subil: No subuid ranges found for user \"subil\" in /etc/subuid - check rootless mode in man pages."
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[28556,1],0]
Exit code: 125
--------------------------------------------------------------------------

Describe the results you expected: Proper output of the mpi ring program with four processes

Process 1 received token -1 from process 0
Process 2 received token -1 from process 1
Process 3 received token -1 from process 2
Process 0 received token -1 from process 3

Additional information you deem important (e.g. issue happens only occasionally):

We are setting ignore_chown_errors to true in the storage.conf

# storage.conf
[storage]
driver = "overlay"
graphroot = "/tmp/subil-containers"
#rootless_storage_path = "$HOME/.local/share/containers/storage"
rootless_storage_path = "/tmp/subil-containers-storage"

[storage.options]
additionalimagestores = [
]

[storage.options.overlay]
ignore_chown_errors = "true"
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "nodev,metacopy=on"

[storage.options.thinpool]

The dockerfile for the localhost/centosmpi image

FROM centos:8

RUN yum -y install openmpi-devel
ENV PATH="/usr/lib64/openmpi/bin:$PATH"
ENV LD_LIBRARY_PATH="/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH"


COPY mpi_ring /home/mpi_ring

Output of podman version:

Version: 2.0.2
API Version: 1
Go Version: go1.13.4
Built: Wed Dec 31 19:00:00 1969
OS/Arch: linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: 4726cba6f219c7479c28f0687868bd2ffe894869'
  cpus: 32
  distribution:
    distribution: '"rhel"'
    version: "8.1"
  eventLogger: file
  hostname: andes-login1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 27008
      size: 1
    uidmap:
    - container_id: 0
      host_id: 15377
      size: 1
  kernel: 4.18.0-147.8.1.el8_1.x86_64
  linkmode: dynamic
  memFree: 235466366976
  memTotal: 270055858176
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.el8.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 88886aef25302adfd40a9335372bbc2b970c8ae5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/15377/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.el8.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 0
  swapTotal: 0
  uptime: 800h 24m 25.67s (Approximately 33.33 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /autofs/nccs-svm1_home1/subil/.config/containers/storage.conf
  containerStore:
    number: 14
    paused: 0
    running: 0
    stopped: 14
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.el8.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /tmp/subil-containers-storage
  graphStatus:
    Backing Filesystem: tmpfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 5
  runRoot: /run/user/15377/containers
  volumePath: /tmp/subil-containers-storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.13.4
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.2-2.el8.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No

Additional environment details (AWS, VirtualBox, physical, etc.):

Using the Andes HPC cluster at ORNL.

% cat /etc/*release
NAME="Red Hat Enterprise Linux"
VERSION="8.1 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.1"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.1"
Red Hat Enterprise Linux release 8.1 (Ootpa)
Red Hat Enterprise Linux release 8.1 (Ootpa)