toolbox: /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc

Hello

I am using an Fedora Silverblur Rawhide 35 and after toolbox update to version 0.0.99.2 I cannot start any container created before or after the upgrade, because of error: invalid entry point PID of container

I am attaching a logs with verbose:

[jakub@NBJSIERZEGA sbpwsdlvalidator]$ toolbox enter -vvv
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user jakub 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /var/home/jakub/.config/toolbox 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called version.PersistentPreRunE(podman --log-level debug version --format json) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called version.PersistentPostRunE(podman --log-level debug version --format json) 
DEBU Current Podman version is 3.3.0-dev          
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.3.0-dev              
DEBU Migration not needed: Podman version 3.3.0-dev is unchanged 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-35'               
DEBU Image: 'fedora-toolbox:35'                   
DEBU Release: '35'                                
DEBU Checking if container fedora-toolbox-35 exists 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called exists.PersistentPreRunE(podman --log-level debug container exists fedora-toolbox-35) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called exists.PersistentPostRunE(podman --log-level debug container exists fedora-toolbox-35) 
DEBU Inspecting mounts of container fedora-toolbox-35 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU Starting container fedora-toolbox-35         
DEBU Inspecting entry point of container fedora-toolbox-35 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-35 is toolbox (PID=0) 
Error: invalid entry point PID of container fedora-toolbox-35

Please help for resolving the issue.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 7
Comments: 17 (16 by maintainers)

Commits related to this issue

libc-wrappers: Added wrappers for pthread_attr_getstacksize, pthread_create and pthread_detach. Fixes #821 — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Added wrappers for __libc_start_main, pthread_attr_getstacksize, pthread_create and pthread_detach. Fixes #821 — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Added wrappers for __libc_start_main, pthread_attr_getstacksize, pthread_create and pthread_detach. Fixes #821 — committed to olivergs/toolbox by olivergs 3 years ago
playbooks/setup-env: Show version of glibc An upgrade of glibc has caused an issue on Fedora Rawhide[0]. We need a clear indicator that a change in glibc could cause it. [0] https://github.com/conta... — committed to HarryMichal/toolbox by HarryMichal 3 years ago
playbooks/setup-env: Show version of glibc An upgrade of glibc has caused an issue on Fedora Rawhide[0]. We need a clear indicator that a change in glibc could cause it. [0] https://github.com/conta... — committed to HarryMichal/toolbox by HarryMichal 3 years ago
playbooks/setup-env: Show version of glibc An upgrade of glibc has caused an issue on Fedora Rawhide[0]. We need a clear indicator that a change in glibc could cause it. [0] https://github.com/conta... — committed to containers/toolbox by HarryMichal 3 years ago
build: Split out the names of the wrapped C library functions A subsequent commit will add more wrappers to cover new symbol versions introduced in glibc-2.34. This will make it easier to read becaus... — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Add a comment A subsequent commit will add more wrappers to cover new symbol versions introduced in glibc-2.34. This will make it easier to read by clearly segregating the file into se... — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Ensure binaries built on Fedora 35 run on older Fedoras https://github.com/containers/toolbox/issues/821 — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Ensure binaries built on Fedora 35 run on older Fedoras https://github.com/containers/toolbox/issues/821 — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Add a comment A subsequent commit will add more wrappers to cover new symbol versions introduced in glibc-2.34. This will make it easier to read by clearly segregating the file into se... — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Avoid new versions of pthread_* symbols from glibc-2.34 Recently, glibc-2.34, which is used by Fedora 35 onwards, added new versions of the pthread_attr_getstacksize [1], pthread_creat... — committed to olivergs/toolbox by olivergs 3 years ago
libc-wrappers: Avoid the new __libc_start_main from glibc-2.34 https://github.com/containers/toolbox/issues/821 — committed to olivergs/toolbox by olivergs 3 years ago
tmpfiles.d: Style fix The subsequent commit will add an entry to create a /run/host symbolic link on the host that points to /, and it will require explicitly skipping some of the columns. Doing the ... — committed to debarshiray/toolbox by debarshiray 3 years ago
build: Ensure that binaries are run against their build-time ABI https://github.com/containers/toolbox/issues/821 — committed to debarshiray/toolbox by debarshiray 3 years ago
build: Ensure that binaries are run against their build-time ABI https://github.com/containers/toolbox/issues/821 — committed to debarshiray/toolbox by debarshiray 3 years ago
build: Ensure that binaries are run against their build-time ABI The /usr/bin/toolbox binary is not only used to interact with toolbox containers and images from the host. It's also used as the entry... — committed to debarshiray/toolbox by debarshiray 3 years ago
build: Ensure that binaries are run against their build-time ABI https://github.com/containers/toolbox/issues/821 — committed to debarshiray/toolbox by debarshiray 3 years ago
build: Ensure that binaries are run against their build-time ABI The /usr/bin/toolbox binary is not only used to interact with toolbox containers and images from the host. It's also used as the entry... — committed to debarshiray/toolbox by debarshiray 3 years ago
build: Ensure that binaries are run against their build-time ABI The /usr/bin/toolbox binary is not only used to interact with toolbox containers and images from the host. It's also used as the entry... — committed to debarshiray/toolbox by debarshiray 3 years ago

Most upvoted comments

Filed as https://github.com/containers/toolbox/issues/832 for tracking.

+10

travier on Jul 7, 2021

There is a new candidate-registry.fedoraproject.org/fedora-toolbox:35 image now and also fedora-toolbox:36 finally.

juhp on Sep 9, 2021

This should be fixed by https://github.com/containers/toolbox/pull/897

debarshiray on Oct 21, 2021

In the meantime, would it help if we had an image that actually contained the newer glibc?

nanonyme on Aug 19, 2021

It might make sense to create a separate, fully statically linked binary to use inside the toolbox for setup operations. This would thus be completely independent of the version of the glibc available inside the container. Some container images may not even have glibc installed at all.

travier on Jul 7, 2021

https://bugzilla.redhat.com/show_bug.cgi?id=1995439 is proposed as an F35 blocker.

cheese on Sep 11, 2021

Do you have a doc somewhere explaining why we need to have toolbox as the entrypoint of the container? Thanks!

There’s a paragraph in toolbox-init-container(1) that touches the topic:

OCI containers are inherently immutable. Configuration options passed through podman create are baked into the definition of the OCI container, and can't be changed later. This means that changes and improvements made in newer versions of Toolbox can't be applied to pre-existing toolbox containers created by older versions of Toolbox. This is avoided by using the entry point to configure the container at runtime.

HarryMichal on Jul 7, 2021