WSL: `systemd status` fails output on Ubuntu

Version

10.0.22621.521

WSL Version

  • WSL 2
  • WSL 1

Kernel Version

5.15.62.1

Distro Version

Ubuntu 22.10

Other Software

No response

Repro Steps

install Ubuntu (Preview), store id 9P7BDVKVNXZ6. This version has systemd already enabled.

Expected Behavior

systemctl status command to work

Actual Behavior

toor@XPS:~$ systemctl status
Failed to dump process list for 'XPS', ignoring: Input/output error
● XPS
    State: degraded
    Units: 318 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 1 units
    Since: Sat 2022-09-24 08:43:15 PDT; 14min ago
  systemd: 251.4-1ubuntu6
  Tainted: cgroupsv1
   CGroup: /

It doesn’t work on ArchLinux also:

[toor@XPS ~]$ systemctl status
Failed to dump process list for 'XPS', ignoring: Input/output error
● XPS
    State: running
    Units: 243 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Sat 2022-09-24 08:51:04 PDT; 14s ago
  systemd: 251.4-1-arch
  Tainted: cgroupsv1
   CGroup: /

Both WSL instances run systemd-251.4, so this could be a systemd issue. It works okay on Ubuntu 22.04 and systemd-249.11.

Diagnostic Logs

No response

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 13
  • Comments: 40 (1 by maintainers)

Commits related to this issue

Most upvoted comments

It now works for me

> systemctl status
● metro
    State: running
    Units: 403 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Sat 2024-07-20 22:53:46 CEST; 1 day 23h ago
  systemd: 255.4-1ubuntu8.2
  Tainted: cgroupsv1
   CGroup: /
           ├─init.scope
           │ ├─     1 /sbin/init
           │ ├─     2 /init
           │ ├─     8 plan9 --control-socket 6 --log-level 4 --server-fd 7 --pipe-fd 9 --log-truncate
           │ ├─   558 /init
...
> wsl --version
WSL version: 2.2.4.0
Kernel version: 5.15.153.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5326
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26091.1-240325-1447.ge-release
Windows version: 10.0.22631.3880
> systemctl version
systemd 255 (255.4-1ubuntu8.2)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

I’ve applied the following workaround to my systemd on WSL, and it makes the output look like (I think) it’s supposed to look:

diff -Naur systemd-stable-251.8.orig/src/core/dbus-unit.c systemd-stable-251.8/src/core/dbus-unit.c
--- systemd-stable-251.8.orig/src/core/dbus-unit.c      2022-12-13 23:27:56.133926327 +0100
+++ systemd-stable-251.8/src/core/dbus-unit.c   2022-12-13 23:39:03.772622555 +0100
@@ -1288,6 +1288,8 @@
                  * “cgroup.procs” in a threaded domain cgroup contains the PIDs of all processes in
                  * the subtree and is not readable in the subtree proper. */
                 r = cg_read_pid(f, &pid);
+                if (r == -EIO)
+                        continue;
                 if (IN_SET(r, 0, -EOPNOTSUPP))
                         break;
                 if (r < 0)

It’s most likely not a proper fix or a good idea to do, but it gets stuff working for me in the meantime.

@themasch the latest stable version of systemd 255 is 255.7 not 255.4. So you have to wait for FC to upgrade systemd.

This is fixed in the latest systemd stable version of 255, so someone just has to get Ubuntu to update.

This issue is reproducible on WSL-based Arch Linux with systemd 254 (254.5-1-arch).

$ sudo SYSTEMD_LOG_LEVEL=7 SYSTEMD_PAGER= systemctl status
Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/systemd/private...
Bus n/a: changing state OPENING → AUTHENTICATING
Bus n/a: changing state AUTHENTICATING → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=1 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=1 reply_cookie=1 signature=a{sv} error-name=n/a error-message=n/a
● MSI-B660M
    State: running
    Units: 252 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2023-11-09 11:18:50 PST; 25min ago
  systemd: 254.5-1-arch
  Tainted: cgroupsv1
   CGroup: /
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnitProcesses cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=error sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=2 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.IOError error-message=Input/output error
Failed to dump process list for 'MSI-B660M', ignoring: Input/output error
Bus n/a: changing state RUNNING → CLOSED

Still an issue to date.

Versão do WSL: 1.2.0.0
Versão do kernel: 5.15.90.1
Versão do WSLg: 1.0.51
Versão do MSRDC: 1.2.3770
Versão do Direct3D: 1.608.2-61064218
Versão do DXCore: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Versão do Windows: 10.0.25336.1010

Also tested with pre-release 1.2.4. Both using cgroups v2. Of the Ubuntu apps, that situation still only affects the Ubuntu-Preview for now, but eventually other releases will be at systemd version 2.50 or later, so that will be a problem.

A suitable fix seems to be to create and assign a cgroup and a cgroup namespace to every distro instance. Handling the cgroup2 mount point properly before starting systemd, it should then see that as its root cgroup. There will be no PID 0’s inside the cgroup.procs file, because it will be populated only by the PIDs of the child processes of systemd. Also, care must be taken to ensure the 'login" shell also goes to the that cgroup (and namespaces).

For the sake of demonstration, if instead of enabling systemd through the proper settings in wsl.conf we use the hack described below, then we get systemctl status behaving correctly, with the downside of the root cgroup being displayed not as /.

An extension to an older hack script to start systemd in its own namespace and cgroup.

#!/bin/sh
set -eu

# Ensure that we don't clean up /tmp setup by the Microsoft init process.
mkdir -p /run/tmpfiles.d
ln -s /dev/null /run/tmpfiles.d/tmp.conf

umount /proc/sys/fs/binfmt_misc

# Create the distro cgroup
mkdir -p /sys/fs/cgroup/distros/ubuntu-preview

# Start systemd in its own namespace. Notice the `-C` for cgroup
env -i /usr/bin/unshare -Cm --fork --mount-proc --pid --propagation unchanged -- sh -c '
echo $PPID > /sys/fs/cgroup/distros/ubuntu-preview/cgroup.procs && exec /lib/systemd/systemd --unit=multi-user.target
' &

DISCLAIMER: By no means the above is a solution. It has many drawbacks, such as not showing the systemd root cgroup as / as well as requiring more hacks to run processes (such as a shell) under the systemd namespaces.

Configuring the boot command parameter to run that script and starting a shell under systemd namespaces allows us to see the following systemctl output.

/etc/wsl.conf

[boot]
command=<absolute path of the script>
carlos@Zero01:~$ systemctl status
● Zero01
    State: running
    Units: 312 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2023-04-19 15:18:12 -03; 5s ago
  systemd: 252.5-2ubuntu3
   CGroup: /distros/ubuntu-preview
           ├─init.scope
           │ └─1 /lib/systemd/systemd --unit=multi-user.target
           └─system.slice
             ├─console-getty.service
             │ └─125 /sbin/agetty -o "-p -- \\u" --noclear --keep-baud - 115200,38400,9600 vt220
             ├─cron.service
             │ └─111 /usr/sbin/cron -f -P
             ├─dbus.service
             │ └─112 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─rsyslog.service
             │ └─124 /usr/sbin/rsyslogd -n -iNONE
             ├─snap-core22-607.mount
             │ └─62 snapfuse /var/lib/snapd/snaps/core22_607.snap /snap/core22/607 -o ro,nodev,allow_other,suid
             ├─snap-snapd-18596.mount
             │ └─63 snapfuse /var/lib/snapd/snaps/snapd_18596.snap /snap/snapd/18596 -o ro,nodev,allow_other,suid
             ├─snap-snapd-18933.mount
             │ └─65 snapfuse /var/lib/snapd/snaps/snapd_18933.snap /snap/snapd/18933 -o ro,nodev,allow_other,suid
             ├─snap-subiquity-4692.mount
             │ └─64 snapfuse /var/lib/snapd/snaps/subiquity_4692.snap /snap/subiquity/4692 -o ro,nodev,allow_other,suid
             ├─snap-subiquity-4714.mount
             │ └─61 snapfuse /var/lib/snapd/snaps/subiquity_4714.snap /snap/subiquity/4714 -o ro,nodev,allow_other,suid
             ├─snap.subiquity.subiquity-server.service
             │ ├─117 /bin/sh /snap/subiquity/4714/usr/bin/subiquity-server
             │ └─186 /snap/subiquity/4714/usr/bin/python3.10 -m subiquity.cmd.server
             ├─snapd.service
             │ └─119 /usr/lib/snapd/snapd

I’ve applied the following workaround to my systemd on WSL, and it makes the output look like (I think) it’s supposed to look:

diff -Naur systemd-stable-251.8.orig/src/core/dbus-unit.c systemd-stable-251.8/src/core/dbus-unit.c
--- systemd-stable-251.8.orig/src/core/dbus-unit.c      2022-12-13 23:27:56.133926327 +0100
+++ systemd-stable-251.8/src/core/dbus-unit.c   2022-12-13 23:39:03.772622555 +0100
@@ -1288,6 +1288,8 @@
                  * “cgroup.procs” in a threaded domain cgroup contains the PIDs of all processes in
                  * the subtree and is not readable in the subtree proper. */
                 r = cg_read_pid(f, &pid);
+                if (r == -EIO)
+                        continue;
                 if (IN_SET(r, 0, -EOPNOTSUPP))
                         break;
                 if (r < 0)

It’s most likely not a proper fix or a good idea to do, but it gets stuff working for me in the meantime.

Works and was dead easy to get into NixOS:

nixpkgs.overlays = [
  (final: prev: with final; {
    systemd-wsl = final.systemd.overrideAttrs ({ patches, ... }: {
      patches = patches ++ [
        ./systemd-systemctl-status-wsl.patch
      ];
    });
  })
];

systemd.package = pkgs.systemd-wsl;

I guess this issue will get more attention over time with more distos update systemd to >= 250. At the moment it seems that the only supported distro with such an systemd version is Kali (but I did not verify this).

Still an issue also on the latest 1.0.0

$ wsl --version
Versione WSL: 1.0.0.0
Versione kernel: 5.15.74.2
Versione WSLg: 1.0.47
Versione MSRDC: 1.2.3575
Versione Direct3D: 1.606.4
Versione DXCore: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
versione Windows: 10.0.19045.2311

I think it is 3; that’s the same root cause @nullpo-head figured out for the problem when it was reported under distrod (https://github.com/nullpo-head/wsl-distrod/issues/31#issuecomment-1079726209), with the same fix (put systemd in a new cgroup). Of course, with systemd now running as pid 1 without a hack, the fix’ll need to be in WSL proper.

I’ve done some investigating: This regression was introduced by systemd/systemd@038cae098b6d85ad9fb06594d36f49915d32f000, which comes from this PR systemd/systemd#21172. It’s possible this is related to systemd/systemd#22089 but I’m not sure as it’s slightly different symptoms and isn’t fixed on main. That commit was a change to systemctl status to use the systemd dbus instead of “directly via cgroupfs” (I’m not sure what that means). I can replicate this bug by calling the dbus directly with busctl call -- org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager GetUnitProcesses 's' '-.slice'. Some code diving suggests that the cause of the IO error is that systemd is not expecting to see zero PIDs in /sys/fs/cgroup/unified/cgroup.procs, which I have a bunch of, I believe they’re there because systemd is in a PID namespace but not in a new cgroup.

I see 3 possible places the bug might be:

  1. That systemctl status code: it possibly shouldn’t be making that dbus call in that way, however, I’m skeptical as that API sounds sensible from its name.
  2. The systemd dbus handler for that method: systemd/systemd:src/core/dbus-unit.c:1310, specifically this line that generates the IO error systemd/systemd:src/basic/cgroup-util.c, that generates an error because of the previously mentioned zeros in /sys/fs/cgroup/unified/cgroup.procs.
  3. The WSL handling of cgroups: it’s possible that systemd shouldn’t observe zeros in its root cgroup. The systemd container interface suggests that systemd should be running in a new cgroup, but I don’t know if that is required.