moby: docker top fails when large PIDs exceed 5 figures

Two quick questions:

1. Is there a target release for this fix? I’m hitting this issue regularly with Docker for Windows 18.03.1-ce-win64 (17438) using the official Jenkins image (i.e. jenkins/jenkins:lts-slim) when leveraging Docker containers in the pipeline.
2. For a “universal” solution, did you look into using the position of the “USER” or “UID” label in the column headers to determine where to cutoff the PID? I’m not sure if it is the same for all distros, but seems like in most that I’ve used the label and values are usually both left-aligned with the same starting horizontal position.

Todays Docker Desktop Edge release should have the fix. Should also be in the next stable release

For the time being - those who wish to work around it much like I had, this should work…

Start a new privileged throwaway container:

docker run --privileged --rm -it ubuntu

Identify if your pid_max is over five digits long:

cat /proc/sys/kernel/pid_max

Change it to 99999:

echo 99999 > /proc/sys/kernel/pid_max

This should allow use cases, such as Jenkins running via Docker-in-Docker (on Windows) to work reliably afterwards.

It’s very likely due to a bug/presentation issue in busybox; docker uses the output of ps on the host (https://github.com/moby/moby/issues/34282#issuecomment-323077673), which has this issue if the host is using busybox (which is the case on docker for mac/linux), but not if it’s running on Ubuntu

Until there’s a proper fix for the parsing in place, can’t the Linux VM (mobylinuxvm) just set (on boot) /proc/sys/kernel/pid_max to 99999 each time to avoid this issue? I’ve been doing this by hand, it does work but the fix has to be manually reapplied every time the host or the VM restarts.

For a “universal” solution, did you look into using the position of the “USER” or “UID” label in the column headers to determine where to cutoff the PID? I’m not sure if it is the same for all distros, but seems like in most that I’ve used the label and values are usually both left-aligned with the same starting horizontal position.

I think that may actually be a good solution; one other option that may work is to customize the COLUMN-headers; not all variations of ps seem to support -o column:<width>, but custom names seem to be supported (even by busybox); from https://unix.stackexchange.com/a/313470

Column width will increase as needed for wide headers; this may be used to widen up columns such as WCHAN (ps -o pid,wchan=WIDE-WCHAN-COLUMN -o comm)

So doing something like;

 ps -o pid=PID______________________ -o user=USER______________________ -o time -o comm

Will make the columns wider, accommodating the underscores;

PID______________________ USER______________________ TIME   COMMAND
                        1 root                         0:00 sh
                        9 root                         0:00 ps

Yes, this is where things could get complicated.

Assumptions:

• PID is always digits
• We can’t guarantee which column will be after PID. Next column could start with a digit. -ps -o pid so far seems to be dependable way to get the pids on the few distros I’ve tried so far.

For this case, the delimiter withUSER can be found with unicode.IsLetter(). That doesn’t help if the user specified arguments that would have values starting with a digit for the column after PID (for example TIME, CPU, etc).

One idea that I’m still working with is that ps -o pid will return only the pids. This will give a something to compare the value at fieldsASCII(line)[pidIndex] to. If there is a match from ps -o pid with whats in the broken line we will be able to figure out at what index the PID field ends.

A quick update, if someone wants to look into this (don’t have time myself at this moment);

@mbentley sent me the output of ps -ef on the host, which is what docker is running behind the scenes to collect the information (see daemon/top_unix.go#L118-L155, and daemon/top_unix.go#L61-L111)

PID   USER     TIME   COMMAND
    1 root       0:00 /sbin/init
    2 root       0:00 [kthreadd]

...

100676root       0:00 [kworker/0:1]
100785root       0:00 [kworker/0:3]
100787root       0:00 docker-containerd-shim 605da7454597d536380f61dabc9e6d5fc9fbf99a65c9357140035ab46b350a0a /var/run/docker/libcontainerd/605da7454597d536380f61dabc9e6d5fc9fbf99a65c9357140035ab46b350a0a docker-runc
100805root       0:00 nginx: master process nginx -g daemon off;
100848openvpn    0:00 nginx: worker process
100998mbentley   0:00 ps -ef

The problem here is that that output (at least by default) glues together the PID and USER columns, therefore docker fails to split them (which causes the error).

I did a quick search if there’s a portable way to change the format, so that we can guarantee that the columns are always separated. This ServerFault discussion provides some possible options; https://serverfault.com/a/157618

However, if there’s an option, we need to be sure it’s supported on all platforms/distros, given that docker shells out to the ps binary on the host, and we don’t know what that ps binary supports on each distro.

The docker top command is known to be problematic for the same reason, as the output (and options) are not standardised; ideally docker wouldn’t shell out, and obtain the same information other ways, but that will be a bigger change.