podman: podman-3.2.1 in container: cannot clone: Operation not permitted Error: cannot re-exec process
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Following was working prior to release of podman 3.2.1 (i.e. worked until last week under podman 3.1.2)
Trying to build a centos-8-based container with podman and skopeo installed in a docker-in-docker (Jenkins agent running in kubernetes) environment… The larger context is Jenkins build of Jenkins jnlp agent with podman, docker, compilers, build tools etc, we use in our CICD pipelines to build containers, but the problem is reproducible with a small dockerfile running locally in docker (see below).
Seems similar to https://github.com/containers/podman/pull/10692 where the fix was to use podman instead of docker – that’s not an option for us at this time – podman isn’t mature enough to support the wide variety of container builds we support in our CICD pipelines. We’ve tried, but it’s currently far from a drop-in replacement from docker – we’re trying! 😃
Steps to reproduce the issue:
- create Dockerfile to build a centos 8 container with latest podman
FROM centos:8
USER root
RUN dnf -y module disable container-tools \
&& dnf -y install 'dnf-command(copr)' \
&& dnf -y copr enable rhcontainerbot/container-selinux \
&& curl -sSL -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo
RUN dnf -y install podman skopeo
#RUN podman --storage-driver=vfs version
RUN podman --storage-driver=vfs info --debug
- docker build -t test .
Describe the results you received:
Full output of the docker build is included below, but the error in question is:
Step 5/5 : RUN podman --storage-driver=vfs info
---> Running in dc4ea7a56855
cannot clone: Operation not permitted
Error: cannot re-exec process
Describe the results you expected:
expected typical podman info output
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
Step 5/6 : RUN podman --storage-driver=vfs version
---> Running in 90096abfcf93
cannot clone: Operation not permitted
Error: cannot re-exec process
Output of podman info --debug:
Step 5/5 : RUN podman --storage-driver=vfs info --debug
---> Running in e3f5463e8f14
cannot clone: Operation not permitted
Error: cannot re-exec process
Package info (e.g. output of rpm -q podman or apt list podman):
podman-3.2.1-1.el8.4.1.x86_64
podman-plugins-3.2.1-1.el8.4.1.x86_64
skopeo-2:1.3.0-1.el8.1.1.x86_64
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
can be reproduced locally with docker build running in a centos-8 based docker container (docker-in-docker):
(base) [jenkins@100533c486d2 ~]$ docker build -t test .
Sending build context to Docker daemon 324.6MB
Step 1/5 : FROM centos:8
8: Pulling from library/centos
7a0437f04f83: Pull complete
Digest: sha256:5528e8b1b1719d34604c87e11dcd1c0a20bedf46e83b5632cdeac91b8c04efc1
Status: Downloaded newer image for centos:8
---> 300e315adb2f
Step 2/5 : USER root
---> Running in 4fb8aa267174
Removing intermediate container 4fb8aa267174
---> 13c2987e0a87
Step 3/5 : RUN dnf -y module disable container-tools && dnf -y install 'dnf-command(copr)' && dnf -y copr enable rhcontainerbot/container-selinux && curl -sSL -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo
---> Running in 271b9fde57ef
CentOS Linux 8 - AppStream 407 kB/s | 7.5 MB 00:18
CentOS Linux 8 - BaseOS 622 kB/s | 2.6 MB 00:04
CentOS Linux 8 - Extras 24 kB/s | 9.6 kB 00:00
Dependencies resolved.
================================================================================
Package Architecture Version Repository Size
================================================================================
Disabling modules:
container-tools
Transaction Summary
================================================================================
Complete!
Last metadata expiration check: 0:00:01 ago on Mon Jun 28 16:26:34 2021.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
dnf-plugins-core noarch 4.0.18-4.el8 baseos 69 k
Installing dependencies:
dbus-glib x86_64 0.110-2.el8 baseos 127 k
python3-dateutil noarch 1:2.6.1-6.el8 baseos 251 k
python3-dbus x86_64 1.2.4-15.el8 baseos 134 k
python3-dnf-plugins-core noarch 4.0.18-4.el8 baseos 234 k
python3-six noarch 1.11.0-8.el8 baseos 38 k
Transaction Summary
================================================================================
Install 6 Packages
Total download size: 854 k
Installed size: 2.3 M
Downloading Packages:
(1/6): dnf-plugins-core-4.0.18-4.el8.noarch.rpm 170 kB/s | 69 kB 00:00
(2/6): dbus-glib-0.110-2.el8.x86_64.rpm 272 kB/s | 127 kB 00:00
(3/6): python3-dateutil-2.6.1-6.el8.noarch.rpm 441 kB/s | 251 kB 00:00
(4/6): python3-dbus-1.2.4-15.el8.x86_64.rpm 614 kB/s | 134 kB 00:00
(5/6): python3-dnf-plugins-core-4.0.18-4.el8.no 1.2 MB/s | 234 kB 00:00
(6/6): python3-six-1.11.0-8.el8.noarch.rpm 394 kB/s | 38 kB 00:00
--------------------------------------------------------------------------------
Total 961 kB/s | 854 kB 00:00
warning: /var/cache/dnf/baseos-f6a80ba95cf937f2/packages/dbus-glib-0.110-2.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
CentOS Linux 8 - BaseOS 1.6 MB/s | 1.6 kB 00:00
Importing GPG key 0x8483C65D:
Userid : "CentOS (CentOS Official Signing Key) <security@centos.org>"
Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : python3-six-1.11.0-8.el8.noarch 1/6
Installing : python3-dateutil-1:2.6.1-6.el8.noarch 2/6
Installing : dbus-glib-0.110-2.el8.x86_64 3/6
Running scriptlet: dbus-glib-0.110-2.el8.x86_64 3/6
Installing : python3-dbus-1.2.4-15.el8.x86_64 4/6
Installing : python3-dnf-plugins-core-4.0.18-4.el8.noarch 5/6
Installing : dnf-plugins-core-4.0.18-4.el8.noarch 6/6
Running scriptlet: dnf-plugins-core-4.0.18-4.el8.noarch 6/6
Verifying : dbus-glib-0.110-2.el8.x86_64 1/6
Verifying : dnf-plugins-core-4.0.18-4.el8.noarch 2/6
Verifying : python3-dateutil-1:2.6.1-6.el8.noarch 3/6
Verifying : python3-dbus-1.2.4-15.el8.x86_64 4/6
Verifying : python3-dnf-plugins-core-4.0.18-4.el8.noarch 5/6
Verifying : python3-six-1.11.0-8.el8.noarch 6/6
Installed:
dbus-glib-0.110-2.el8.x86_64
dnf-plugins-core-4.0.18-4.el8.noarch
python3-dateutil-1:2.6.1-6.el8.noarch
python3-dbus-1.2.4-15.el8.x86_64
python3-dnf-plugins-core-4.0.18-4.el8.noarch
python3-six-1.11.0-8.el8.noarch
Complete!
Repository successfully enabled.
Enabling a Copr repository. Please note that this repository is not part
of the main distribution, and quality may vary.
The Fedora Project does not exercise any power over the contents of
this repository beyond the rules outlined in the Copr FAQ at
<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,
and packages are not held to any quality or security level.
Please do not file bug reports about these packages in Fedora
Bugzilla. In case of problems, contact the owner of this repository.
Removing intermediate container 271b9fde57ef
---> b87b175466e8
Step 4/5 : RUN dnf -y install podman skopeo
---> Running in 33f12077193f
Copr repo for container-selinux owned by rhcont 3.6 kB/s | 1.4 kB 00:00
Stable Releases of Upstream github.com/containe 50 kB/s | 66 kB 00:01
Dependencies resolved.
====================================================================================================
Package Arch Version Repository Size
====================================================================================================
Installing:
podman x86_64 3.2.1-1.el8.4.1 devel_kubic_libcontainers_stable 13 M
skopeo x86_64 2:1.3.0-1.el8.1.1 devel_kubic_libcontainers_stable 7.2 M
Upgrading:
iptables-libs x86_64 1.8.4-17.el8 baseos 107 k
Installing dependencies:
conmon x86_64 2:2.0.29-1.el8.3.4 devel_kubic_libcontainers_stable 50 k
containernetworking-plugins x86_64 1.0.0-0.2.rc1.el8.6.1 devel_kubic_libcontainers_stable 21 M
containers-common noarch 4:1-17.el8.17.3 devel_kubic_libcontainers_stable 60 k
crun x86_64 0.20.1-1.el8.3.1 devel_kubic_libcontainers_stable 194 k
dnsmasq x86_64 2.79-15.el8 appstream 318 k
fuse-common x86_64 3.2.1-12.el8 baseos 21 k
fuse3 x86_64 3.2.1-12.el8 baseos 50 k
fuse3-libs x86_64 3.2.1-12.el8 baseos 94 k
iptables x86_64 1.8.4-17.el8 baseos 586 k
jansson x86_64 2.11-3.el8 baseos 46 k
libnetfilter_conntrack x86_64 1.0.6-5.el8 baseos 65 k
libnfnetlink x86_64 1.0.1-13.el8 baseos 33 k
libnftnl x86_64 1.1.5-4.el8 baseos 83 k
libslirp x86_64 4.3.1-4.el8.4.7 devel_kubic_libcontainers_stable 73 k
nftables x86_64 1:0.9.3-18.el8 baseos 313 k
yajl x86_64 2.1.0-10.el8 appstream 41 k
Installing weak dependencies:
catatonit x86_64 0.1.5-6.el8.3.7 devel_kubic_libcontainers_stable 290 k
fuse-overlayfs x86_64 1.5.0-1.el8.1.4 devel_kubic_libcontainers_stable 73 k
podman-plugins x86_64 3.2.1-1.el8.4.1 devel_kubic_libcontainers_stable 3.4 M
slirp4netns x86_64 1.1.8-4.el8.7.8 devel_kubic_libcontainers_stable 55 k
Transaction Summary
====================================================================================================
Install 22 Packages
Upgrade 1 Package
Total download size: 47 M
Downloading Packages:
(1/23): yajl-2.1.0-10.el8.x86_64.rpm 114 kB/s | 41 kB 00:00
(2/23): dnsmasq-2.79-15.el8.x86_64.rpm 575 kB/s | 318 kB 00:00
(3/23): fuse-common-3.2.1-12.el8.x86_64.rpm 28 kB/s | 21 kB 00:00
(4/23): fuse3-3.2.1-12.el8.x86_64.rpm 53 kB/s | 50 kB 00:00
(5/23): jansson-2.11-3.el8.x86_64.rpm 175 kB/s | 46 kB 00:00
(6/23): fuse3-libs-3.2.1-12.el8.x86_64.rpm 78 kB/s | 94 kB 00:01
(7/23): libnetfilter_conntrack-1.0.6-5.el8.x86_ 238 kB/s | 65 kB 00:00
(8/23): libnfnetlink-1.0.1-13.el8.x86_64.rpm 125 kB/s | 33 kB 00:00
(9/23): libnftnl-1.1.5-4.el8.x86_64.rpm 302 kB/s | 83 kB 00:00
(10/23): iptables-1.8.4-17.el8.x86_64.rpm 335 kB/s | 586 kB 00:01
(11/23): nftables-0.9.3-18.el8.x86_64.rpm 438 kB/s | 313 kB 00:00
(12/23): catatonit-0.1.5-6.el8.3.7.x86_64.rpm 257 kB/s | 290 kB 00:01
(13/23): conmon-2.0.29-1.el8.3.4.x86_64.rpm 51 kB/s | 50 kB 00:00
(14/23): containers-common-1-17.el8.17.3.noarch 205 kB/s | 60 kB 00:00
(15/23): crun-0.20.1-1.el8.3.1.x86_64.rpm 522 kB/s | 194 kB 00:00
(16/23): fuse-overlayfs-1.5.0-1.el8.1.4.x86_64. 161 kB/s | 73 kB 00:00
(17/23): libslirp-4.3.1-4.el8.4.7.x86_64.rpm 254 kB/s | 73 kB 00:00
(18/23): containernetworking-plugins-1.0.0-0.2. 11 MB/s | 21 MB 00:01
(19/23): podman-plugins-3.2.1-1.el8.4.1.x86_64. 4.2 MB/s | 3.4 MB 00:00
(20/23): podman-3.2.1-1.el8.4.1.x86_64.rpm 12 MB/s | 13 MB 00:01
(21/23): slirp4netns-1.1.8-4.el8.7.8.x86_64.rpm 191 kB/s | 55 kB 00:00
(22/23): skopeo-1.3.0-1.el8.1.1.x86_64.rpm 9.8 MB/s | 7.2 MB 00:00
(23/23): iptables-libs-1.8.4-17.el8.x86_64.rpm 105 kB/s | 107 kB 00:01
--------------------------------------------------------------------------------
Total 7.3 MB/s | 47 MB 00:06
warning: /var/cache/dnf/devel_kubic_libcontainers_stable-37b272243bc11f7c/packages/catatonit-0.1.5-6.el8.3.7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 75060aa4: NOKEY
Stable Releases of Upstream github.com/containe 3.0 kB/s | 1.1 kB 00:00
Importing GPG key 0x75060AA4:
Userid : "devel:kubic OBS Project <devel:kubic@build.opensuse.org>"
Fingerprint: 2472 D6D0 D2F6 6AF8 7ABA 8DA3 4D64 3903 7506 0AA4
From : https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/repodata/repomd.xml.key
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : iptables-libs-1.8.4-17.el8.x86_64 1/24
Installing : libnftnl-1.1.5-4.el8.x86_64 2/24
Running scriptlet: libnftnl-1.1.5-4.el8.x86_64 2/24
Installing : libnfnetlink-1.0.1-13.el8.x86_64 3/24
Running scriptlet: libnfnetlink-1.0.1-13.el8.x86_64 3/24
Installing : libnetfilter_conntrack-1.0.6-5.el8.x86_64 4/24
Running scriptlet: libnetfilter_conntrack-1.0.6-5.el8.x86_64 4/24
Running scriptlet: iptables-1.8.4-17.el8.x86_64 5/24
Installing : iptables-1.8.4-17.el8.x86_64 5/24
Running scriptlet: iptables-1.8.4-17.el8.x86_64 5/24
Installing : libslirp-4.3.1-4.el8.4.7.x86_64 6/24
Installing : slirp4netns-1.1.8-4.el8.7.8.x86_64 7/24
Installing : containernetworking-plugins-1.0.0-0.2.rc1.el8.6.1. 8/24
Installing : conmon-2:2.0.29-1.el8.3.4.x86_64 9/24
Installing : catatonit-0.1.5-6.el8.3.7.x86_64 10/24
Installing : jansson-2.11-3.el8.x86_64 11/24
Installing : nftables-1:0.9.3-18.el8.x86_64 12/24
Running scriptlet: nftables-1:0.9.3-18.el8.x86_64 12/24
Installing : fuse3-libs-3.2.1-12.el8.x86_64 13/24
Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 13/24
Installing : fuse-common-3.2.1-12.el8.x86_64 14/24
Installing : fuse3-3.2.1-12.el8.x86_64 15/24
Installing : fuse-overlayfs-1.5.0-1.el8.1.4.x86_64 16/24
Running scriptlet: fuse-overlayfs-1.5.0-1.el8.1.4.x86_64 16/24
Installing : yajl-2.1.0-10.el8.x86_64 17/24
Installing : crun-0.20.1-1.el8.3.1.x86_64 18/24
Installing : containers-common-4:1-17.el8.17.3.noarch 19/24
Running scriptlet: dnsmasq-2.79-15.el8.x86_64 20/24
Installing : dnsmasq-2.79-15.el8.x86_64 20/24
Running scriptlet: dnsmasq-2.79-15.el8.x86_64 20/24
Installing : podman-3.2.1-1.el8.4.1.x86_64 21/24
Installing : podman-plugins-3.2.1-1.el8.4.1.x86_64 22/24
Installing : skopeo-2:1.3.0-1.el8.1.1.x86_64 23/24
Cleanup : iptables-libs-1.8.4-15.el8.x86_64 24/24
Running scriptlet: iptables-libs-1.8.4-15.el8.x86_64 24/24
Verifying : dnsmasq-2.79-15.el8.x86_64 1/24
Verifying : yajl-2.1.0-10.el8.x86_64 2/24
Verifying : fuse-common-3.2.1-12.el8.x86_64 3/24
Verifying : fuse3-3.2.1-12.el8.x86_64 4/24
Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/24
Verifying : iptables-1.8.4-17.el8.x86_64 6/24
Verifying : jansson-2.11-3.el8.x86_64 7/24
Verifying : libnetfilter_conntrack-1.0.6-5.el8.x86_64 8/24
Verifying : libnfnetlink-1.0.1-13.el8.x86_64 9/24
Verifying : libnftnl-1.1.5-4.el8.x86_64 10/24
Verifying : nftables-1:0.9.3-18.el8.x86_64 11/24
Verifying : catatonit-0.1.5-6.el8.3.7.x86_64 12/24
Verifying : conmon-2:2.0.29-1.el8.3.4.x86_64 13/24
Verifying : containernetworking-plugins-1.0.0-0.2.rc1.el8.6.1. 14/24
Verifying : containers-common-4:1-17.el8.17.3.noarch 15/24
Verifying : crun-0.20.1-1.el8.3.1.x86_64 16/24
Verifying : fuse-overlayfs-1.5.0-1.el8.1.4.x86_64 17/24
Verifying : libslirp-4.3.1-4.el8.4.7.x86_64 18/24
Verifying : podman-3.2.1-1.el8.4.1.x86_64 19/24
Verifying : podman-plugins-3.2.1-1.el8.4.1.x86_64 20/24
Verifying : skopeo-2:1.3.0-1.el8.1.1.x86_64 21/24
Verifying : slirp4netns-1.1.8-4.el8.7.8.x86_64 22/24
Verifying : iptables-libs-1.8.4-17.el8.x86_64 23/24
Verifying : iptables-libs-1.8.4-15.el8.x86_64 24/24
Upgraded:
iptables-libs-1.8.4-17.el8.x86_64
Installed:
catatonit-0.1.5-6.el8.3.7.x86_64
conmon-2:2.0.29-1.el8.3.4.x86_64
containernetworking-plugins-1.0.0-0.2.rc1.el8.6.1.x86_64
containers-common-4:1-17.el8.17.3.noarch
crun-0.20.1-1.el8.3.1.x86_64
dnsmasq-2.79-15.el8.x86_64
fuse-common-3.2.1-12.el8.x86_64
fuse-overlayfs-1.5.0-1.el8.1.4.x86_64
fuse3-3.2.1-12.el8.x86_64
fuse3-libs-3.2.1-12.el8.x86_64
iptables-1.8.4-17.el8.x86_64
jansson-2.11-3.el8.x86_64
libnetfilter_conntrack-1.0.6-5.el8.x86_64
libnfnetlink-1.0.1-13.el8.x86_64
libnftnl-1.1.5-4.el8.x86_64
libslirp-4.3.1-4.el8.4.7.x86_64
nftables-1:0.9.3-18.el8.x86_64
podman-3.2.1-1.el8.4.1.x86_64
podman-plugins-3.2.1-1.el8.4.1.x86_64
skopeo-2:1.3.0-1.el8.1.1.x86_64
slirp4netns-1.1.8-4.el8.7.8.x86_64
yajl-2.1.0-10.el8.x86_64
Complete!
Removing intermediate container 33f12077193f
---> d3c0eaff56b3
Step 5/5 : RUN podman --storage-driver=vfs info --debug
---> Running in 976f61eebfea
cannot clone: Operation not permitted
Error: cannot re-exec process
The command '/bin/sh -c podman --storage-driver=vfs info --debug' returned a non-zero code: 125
(base) [jenkins@100533c486d2 ~]$
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 37 (15 by maintainers)
Yes, my steps were:
Basically…what Dan said. IIRC, you (@mheon) and I had an IRC chat in which you confirmed: We deliberately bypass lots of stuff for the version sub-command. Apparently we still need to clone though 😞 In this particular case (with docker), I’m afraid the answer might be building with
--privileged(assuming that’s even a thing). The only way I was able to get around the problem was to usepodman(I understand that’s not possible here) 😢No, we’re using podman in a docker (kubernetes pod) container “back end”. Our CICD infrastructure is Jenkins hosted in IBM Cloud – our Jenkins agents are Centos 8 containers running as IBM Cloud kubernetes pods. These Centos 8 containers have podman installed (and docker and other tools). Within these pods, we perform docker builds. So, we have a container running in kubernetes that does docker builds (docker-in-docker) that need to execute podman commands. As of podman 3.2.1, we found this no longer works (see above) for example of docker build that no longer works when run in a container. We can’t run a docker build with a Dockerfile that installs and executes podman commands. Follow up testing shows it’s not jenkins or kubernetes – or docker build. There seems to be a new requirement for podman 3.2.1 running in containers to have privileges that weren’t needed before – privileges we can’t seem to grant to a docker build.
Simplest example, we can’t
docker buildthe following to bootstrap our podman build image with podman 3.2.1 (worked with 3.1.2):podman/buildah is not a mature, drop-in replacement for docker for container builds in jenkins pipelines. We support a common Jenkins CICD pipeline used across numerous teams to build hundreds of projects, most of which are not interested in making the switch to podman. We will need to support docker as a container build tool for some time as we try to increase podman adoption.
Until the 3.2.x versions, we’ve been able to maintain a single Jenkins agent with both docker and podman (as well as a long list of other build tools) – if we can’t support docker and podman side-by-side, it’s going to be difficult to increase podman adoption. For example, we use docker for most container builds, but have adopted Pod Manager container signing (skopeo copy --sign-by, podman push --sign-by) for image signing and verification.
If running podman in docker/kubernetes isn’t a supported configuration, we’ve got some rethinking to do. Was this a known/intentional change in 3.2.x?