toolbox: Container doesn't start because user.Current() fails inside the entry point

Describe the bug When trying to enter the container fedora-toolbox-35 created using

toolbox create --distro fedora --release 35

I get the error

$ toolbox enter fedora-toolbox-35
Error: invalid entry point PID of container fedora-toolbox-35

To have a better error message I tried to attach to the container using podman instead of toolbox and got the error:

$ podman start --attach --interactive fedora-toolbox-35
Error: failed to get the current user: user: lookup userid 0: invalid argument

This error is returned by toolbox whenever Golang’s user.Current() gets an error when calling getpwuid_r.

Steps how to reproduce the behaviour

toolbox create --distro fedora --release 35
toolbox enter fedora-toolbox-35

Expected behaviour toolbox should start the default user shell inside a container.

Actual behaviour toolbox exits with error code 1 and prints the error Error: invalid entry point PID of container fedora-toolbox-35.

Screenshots If applicable, add screenshots to help explain your problem.

Output of toolbox --version (v0.0.90+)

toolbox version 0.0.99.3

Toolbox package info (rpm -q toolbox)

toolbox-0.0.99.3-2.fc34.x86_64

Output of podman version

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.8
Built:        Sun Nov 14 00:16:48 2021
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)

podman-3.4.2-1.fc34.x86_64

Info about your OS Fedora Silverblue 34

Additional context Add any other context about the problem here. When did the issue start occurring? After an update (what packages were updated)? If the issue is about operating with containers/images (creating, using, deleting,…), share here what image you used. If you’re unsure, share here the output of toolbox list -i (shows all toolbox images on your system).

IMAGE ID      IMAGE NAME                                    CREATED
ab8bc106d4a7  <none>                                        2 months ago
d8a734db8c5c  registry.fedoraproject.org/fedora-toolbox:34  7 months ago
40b181c70b73  registry.fedoraproject.org/fedora-toolbox:35  4 weeks ago

If you see an error message saying: Error: invalid entry point PID of container <name-of-container>, add to the ticket output of command podman start --attach <name-of-container>.

About this issue

Original URL
State: open
Created 2 years ago
Comments: 19 (8 by maintainers)

Most upvoted comments

Hey, @angiglesias , my apologies, I didn’t have to time get back to this, but thanks for continuing to dig into it.

This is expected:

[root@toolbox /]# LD_LIBRARY_PATH=/run/host/lib64/:/lib64 toolbox --log-level debug init-container --gid 1000 --home /var/home/test-vm --shell /bin/bash --uid 1000 --user test-vm --monitor-host --home-link --media-link --mnt-link
...
...     
DEBU Binding /etc/machine-id to /run/host/etc/machine-id 
mount: /run/host/lib64/libc.so.6: version `GLIBC_2.34' not found (required by mount)
Error: failed to bind /etc/machine-id to /run/host/etc/machine-id

… because Fedora 35 inside the container uses glibc-2.34. Binaries, in this case mount(8), that were built against it can’t be run against the Fedora 34 host with glibc-2.33.

debarshiray on Mar 8, 2023

As far as I can make out, we are going through current() in go.git/src/os/user/cgo_lookup_unix.go, which ends up calling getpwuid_r (0, ...). So, you could try to write a small C program that calls getpwuid_r (0, ...) and run it as root:root inside the container and see what’s going on.

debarshiray on Oct 25, 2022

To test the container, I created a new container with podman with the same arguments used by toolbox (using podman inspect on the container made by toolbox). The only difference should be that I didn’t use the default toolbox init-container as the container cmdline and instead used an interactive shell and invoked toolbox there.

Yes, that’s a smart way to test!

We could conditionalize this line to only run on the host:

currentUser, err = user.Current()

… because we don’t use currentUser inside the container. However, before changing anything, I want to understand exactly what’s going on here.

It makes sense that it’s trying to look up the user for UID 0, because toolbox init-container is running as root:root inside the container. The question is, why does user.Current() think that it’s an invalid argument.

Given that this is rootless Toolbx, UID 0 inside the container isn’t the same UID 0 as on the host. There’s a user namespace and some other non-0 UID from the host has been mapped to 0 inside the container. I am wondering if there’s something wrong there.

Could you please show me the /etc/subuid and /etc/subgid files from your host?

And the UID/GID mappings for the namespace? Inside the container:

$ cat /proc/self/uid_map
...
$ cat /proc/self/gid_map
...

… and from the host:

$ cat /proc/<PID of entry point>/uid_map
...
$ cat /proc/<PID of entry point>/gid_map
...

debarshiray on Oct 25, 2022

As far as I know, the container starts as root and then init-container pivots to the user’s uid, is that right?

The toolbox init-container entry point is the first process started inside the container by podman start as root:root in terms of the container’s user namespace. It keeps running as root:root, and doesn’t change to something else.

The interactive shells offered by toolbox enter are invoked separately through podman exec and those always run as the same user on the host, which is most likely not root.

debarshiray on Oct 25, 2022

@patriziobrunops did something go wrong with your comment? 😃

debarshiray on Oct 24, 2022