podman: XDG_RUNTIME_DIR directory "/run/user/1000" is not owned by the current user

/kind bug

Description

I am attempting to run podman in rootless mode with lingering. When I attempt to start the systemctl --user service podman fails with the error in the title. In my case, I am user 1002 and the error states UID 1002, but I replaced it with 1000 as I expect that to be most common and should help with other users searching for the same error. In my case the directory is in fact owned by the current user and is writable by the current user.

EDIT: Whenever I reboot and need to test from scratch I create a new user, so the UIDs don’t necessarily match, but when applicable, they do match the user currently being utilized for testing.

Steps to reproduce the issue:

As a test, I created a new user. sudo su - test -c 'podman info' works

So I enable linger and it fails, there is a note about this on the troubleshooting page, so let’s login to the console.

sudo loginctl enable-linger test
sudo su - test -c 'podman info'
ERRO[0000] XDG_RUNTIME_DIR directory "/run/user/1006" is not owned by the current user

sudo ls -lah /run/user/1006
total 0
drwx------ 7 test test 180 Feb 24 08:13 .
drwxr-xr-x 5 root root 100 Feb 24 08:13 ..
srw-rw-rw- 1 test test   0 Feb 24 08:13 bus
drwx------ 2 test test  40 Feb 24 08:13 containers
drwx------ 2 test test 140 Feb 24 08:13 gnupg
drwx-----T 2 test test  40 Feb 24 08:13 libpod
srw-rw-rw- 1 test test   0 Feb 24 08:13 pk-debconf-socket
drwxr-xr-x 2 test test  60 Feb 24 08:13 podman
drwxr-xr-x 4 test test 120 Feb 24 08:13 systemd

On the troubleshooting page it states that I need to create a login session, so I login from the console. I still get the error.

test@2006-ct:~$ podman info
ERRO[0000] XDG_RUNTIME_DIR directory "/run/user/1006" is not owned by the current user

Ok, let’s tests with the machinectl method.

sudo machinectl shell test@
Connected to the local host. Press ^] three times within 1s to exit session.
podman info
ERRO[0000] XDG_RUNTIME_DIR directory "/run/user/1006" is not owned by the current user

Let’s reboot for good measure… sudo su - test -c 'podman info' works!

So in summary, after enabling lingering we need to reboot the server for podman to operate as that user.

Describe the results you received:

Podman does not work as a lingering user until the host is rebooted as shown above. Command: podman info Error: ERRO[0000] XDG_RUNTIME_DIR directory "/run/user/1006" is not owned by the current user

Describe the results you expected:

Running podman commands should work as expected after a user has been granted lingering without rebooting the host.

Additional information you deem important (e.g. issue happens only occasionally):

I am working with Ansible on this and it is repeatable for every run and every instance when the host is recreated. I am running a Debian CT on Proxmox. The Proxmox filesystem is ZFS so I am using the VFS driver in the CT.

Proxmox:

pveversion
pve-manager/7.1-8/5b267f33 (running kernel: 5.13.19-2-pve)

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye

Container:

cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye

Ansible:

- name: "{{ systemd_user }} | User prerequisites"
  block: 

  - name: "{{ systemd_user }} | Get home directory"
    ansible.builtin.user:
      name: "{{ systemd_user }}"
      state: present
    register: user_info

  - name: "{{ systemd_user }} | Set Systemd directory"
    ansible.builtin.set_fact:
      user_systemd: "{{ user_info.home }}/.config/systemd/user"

  - name: "{{ systemd_user }} | Create Systemd directory"
    ansible.builtin.file:
      path: "{{ user_systemd }}"
      state: directory
      mode: '0750'
      # recurse: yes

  - name: "{{ systemd_user }} | Fix Systemd connection"
    ansible.builtin.lineinfile:
      dest: "~/.bashrc"
      regexp: "{{ item.regexp }}"
      line: "{{ item.line }}"
    loop:
      - regexp: "^#?export XDG_RUNTIME_DIR"
        line: "export XDG_RUNTIME_DIR=\"${XDG_RUNTIME_DIR:-/run/user/$UID}\""
      - regexp: "^#?export DBUS_SESSION_BUS_ADDRESS"
        line: "export DBUS_SESSION_BUS_ADDRESS=\"${DBUS_SESSION_BUS_ADDRESS:-unix:path=${XDG_RUNTIME_DIR}/bus}\""
  
  become_user: "{{ systemd_user }}"
  become: true
  when: systemd_user != "root"

- name: "{{ systemd_user }} | Enable lingering"
  block:

  - name: "{{ systemd_user }} | Check if lingering is enabled"
    ansible.builtin.stat: 
      path: "/var/lib/systemd/linger/{{ systemd_user }}"
    register: linger

  - name: "{{ systemd_user }} | Enable lingering"
    ansible.builtin.command: "loginctl enable-linger {{ systemd_user }}"
    when: 
      - not linger.stat.exists
      - systemd_config.enable_linger | default('yes')
  
  # - name: "{{ systemd_user }} | Get user info"
  #   getent:
  #     database: passwd
  #     key: "{{ systemd_user }}"
  #   when: not linger.stat.exists

  # - name: Debug user info
  #   debug:
  #     var: ansible_facts.getent_passwd[{{ systemd_user }}]
  #   when: not linger.stat.exists

  # - name: Restart user Systemd service # Doesn't work...
  #   ansible.builtin.systemd:
  #     name: "user@{{ ansible_facts.getent_passwd[systemd_user].1 }}"
  #     state: restarted
  #   when: not linger.stat.exists

  # - name: "{{ systemd_user }} | Restart Systemd to apply lingering" # Doesn't work...
  #   ansible.builtin.systemd:
  #     daemon_reexec: yes
  #   when:
  #     - not linger.stat.exists | default('no')
  #     - systemd_config.enable_linger | default('yes')

  - name: "{{ systemd_user }} | Reboot to apply linger (There has got to be a better way!)"
    ansible.builtin.reboot:
    when: linger.changed
  
  become: true
  when: 
    - systemd_user != "root"
    - systemd_config.enable_linger | default('yes')

Output of podman version:

podman version
Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.9
Built:        Wed Dec 31 18:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

podman info --debug
host:
  arch: amd64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: 2006-ct
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1006
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1006
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
  kernel: 5.13.19-2-pve
  linkmode: dynamic
  memFree: 8145469440
  memTotal: 8589934592
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1006/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 0
  swapTotal: 0
  uptime: 10m 18.12s
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: [REDACTED]
    - Insecure: false
      Location: mirror.gcr.io
    Prefix: docker.io
  search:
  - docker.io
store:
  configFile: /home/test/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/test/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 0
  runRoot: /tmp/containers-user-1006/containers
  volumePath: /home/test/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Wed Dec 31 18:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.9
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

apt list podman
Listing... Done
podman/stable,now 3.0.1+dfsg1-3+b2 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

I am using the latest version of Podman available for Debian 11, I have referenced the Podman Troubleshooting Guide.

Additional environment details (AWS, VirtualBox, physical, etc.):

Proxmox CT.