sysbox: Sysbox v0.6.1 fails on Fedora 37 with `moby-engine` due to default overlay2/btrfs backing and `--selinux-enabled` in docker daemon

Apr 13 18:27:19 gh-amd64-0001 sh[1019]: sysbox-runc
Apr 13 18:27:19 gh-amd64-0001 sh[1019]:         edition:        Community Edition (CE)
Apr 13 18:27:19 gh-amd64-0001 sh[1019]:         version:        0.6.1
Apr 13 18:27:19 gh-amd64-0001 sh[1019]:         commit:         278997aab055ad6eec9e48a555b90eef877596b7
Apr 13 18:27:19 gh-amd64-0001 sh[1019]:         built at:       Sat Apr  8 06:08:15 UTC 2023
Apr 13 18:27:19 gh-amd64-0001 sh[1019]:         built by:       Rodny Molina
Apr 13 18:27:19 gh-amd64-0001 sh[1019]:         oci-specs:      1.0.2-dev
Apr 13 18:27:19 gh-amd64-0001 sh[1025]: sysbox-mgr
Apr 13 18:27:19 gh-amd64-0001 sh[1025]:         edition:        Community Edition (CE)
Apr 13 18:27:19 gh-amd64-0001 sh[1025]:         version:        0.6.1
Apr 13 18:27:19 gh-amd64-0001 sh[1025]:         commit:         ba99c0e7088f1e1ab51f95551f50de9524176655
Apr 13 18:27:19 gh-amd64-0001 sh[1025]:         built at:       Sat Apr  8 06:08:57 UTC 2023
Apr 13 18:27:19 gh-amd64-0001 sh[1025]:         built by:       Rodny Molina
Apr 13 18:27:19 gh-amd64-0001 sh[1030]: sysbox-fs
Apr 13 18:27:19 gh-amd64-0001 sh[1030]:         edition:        Community Edition (CE)
Apr 13 18:27:19 gh-amd64-0001 sh[1030]:         version:        0.6.1
Apr 13 18:27:19 gh-amd64-0001 sh[1030]:         commit:         a2631f69c62722c67dfd3aa97a8412b5c4db6a8a
Apr 13 18:27:19 gh-amd64-0001 sh[1030]:         built at:       Sat Apr  8 06:08:45 UTC 2023
Apr 13 18:27:19 gh-amd64-0001 sh[1030]:         built by:       Rodny Molina
Apr 13 18:29:19 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:29:19" level=info msg="registered new container 1e0d92eceac1"
Apr 13 18:29:19 gh-amd64-0001 sysbox-fs[994]: time="2023-04-13 18:29:19" level=info msg="Container pre-registration completed: id = 1e0d92eceac1"
Apr 13 18:29:19 gh-amd64-0001 sysbox-fs[994]: time="2023-04-13 18:29:19" level=info msg="Container unregistration completed: id = 1e0d92eceac1"
Apr 13 18:29:20 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:29:20" level=info msg="unregistered container 1e0d92eceac1"
Apr 13 18:29:20 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:29:20" level=info msg="released resources for container 1e0d92eceac1"
Apr 13 18:29:53 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:29:53" level=info msg="registered new container 70b9768314c4"
Apr 13 18:29:53 gh-amd64-0001 sysbox-fs[994]: time="2023-04-13 18:29:53" level=info msg="Container pre-registration completed: id = 70b9768314c4"
Apr 13 18:29:53 gh-amd64-0001 sysbox-fs[994]: time="2023-04-13 18:29:53" level=info msg="Container registration completed: id = 70b9768314c4, initPid = 1700, uid:gid = 524288:524288"
Apr 13 18:29:54 gh-amd64-0001 sysbox-fs[994]: time="2023-04-13 18:29:54" level=info msg="Container unregistration completed: id = 70b9768314c4"
Apr 13 18:29:54 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:29:54" level=info msg="unregistered container 70b9768314c4"
Apr 13 18:29:54 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:29:54" level=info msg="released resources for container 70b9768314c4"
Apr 13 18:32:42 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:32:42" level=info msg="registered new container d86053195247"
Apr 13 18:32:42 gh-amd64-0001 sysbox-fs[994]: time="2023-04-13 18:32:42" level=info msg="Container pre-registration completed: id = d86053195247"
Apr 13 18:32:43 gh-amd64-0001 sysbox-fs[994]: time="2023-04-13 18:32:43" level=info msg="Container unregistration completed: id = d86053195247"
Apr 13 18:32:43 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:32:43" level=info msg="unregistered container d86053195247"
Apr 13 18:32:43 gh-amd64-0001 sysbox-mgr[767]: time="2023-04-13 18:32:43" level=info msg="released resources for container d86053195247"

[root@gh-amd64-0001 ~]# docker run --runtime=sysbox-runc --rm nestybox/ubuntu-jammy-systemd-docker:latest /bin/bash
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:424: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: failed to mount /var/lib/docker/overlay2/a3b391dab2ee67356935a5a05e9dc94f9f1f0232d927ae8f20b8a28b0d15d155/merged: invalid argument: unknown.
[root@gh-amd64-0001 ~]# docker run --runtime=sysbox-runc --rm nestybox/ubuntu-jammy-systemd-docker:latest /bin/bash
[root@gh-amd64-0001 ~]# docker run --runtime=sysbox-runc --rm nestybox/ubuntu-jammy-systemd-docker:latest /bin/bash
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:424: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: failed to mount /var/lib/docker/overlay2/9a329ecd0e8b2f3e2f8535bfd34cc225c5e4b1dff78d98e573b41ac9fceae568/merged: invalid argument: unknown.
[root@gh-amd64-0001 ~]# docker run --runtime=sysbox-runc -it --rm nestybox/ubuntu-jammy-systemd-docker:latest /bin/bash
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:424: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: failed to mount /var/lib/docker/overlay2/1dfe1eca360eeb6526e2c55f16ed9cc644ee66eb9b08c2b7f7448a20d33e4d9c/merged: invalid argument: unknown.
[root@gh-amd64-0001 ~]# docker run --runtime=sysbox-runc -it --rm nestybox/ubuntu-jammy-systemd-docker:latest /bin/bash

Welcome to Ubuntu 22.04.1 LTS!

Failed to create /init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
[root@gh-amd64-0001 ~]#

About this issue

Original URL
State: closed
Created a year ago
Comments: 56 (16 by maintainers)

Most upvoted comments

Perfect, thanks @arcivanov. Glad we got to the bottom of it.

Closing this issue now.

ctalledo on Apr 19, 2023

So strange, worked for me without any issues on the Fedora 37 host.

Please go to the cloned Sysbox repo, and do:
$ make test-shell
...
root@sysbox-test:~  docker run --runtime=sysbox-runc -it --rm nestybox/ubuntu-jammy-systemd-docker:latest
The make test-shell will launch a container, inside of which Docker + Sysbox are installed and running. It works like a sandbox environment to play around with Sysbox.

I am starting to think it’s an SELinux (or some other security module) issue.

That worked!

Let me see if it’s SELinux.

arcivanov on Apr 19, 2023

@ctalledo same issue built from source

I was actually able to get it working, and even enabled support for BTRFS; testing it now. Once the test suite runs, I’ll post the steps so you can get it working on your side.

I am on a GCP machine with Fedora 37 and kernel 6.0:

$ uname -a
Linux fedora-37.us-central1-a.c.amazing-hub-320818.internal 6.0.7-301.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 4 18:35:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

ctalledo on Apr 18, 2023

# docker run --runtime=sysbox-runc -it --rm nestybox/alpine-docker
/ # ls -l /var/lib | grep docker 
drwx------    2 root     root          4096 Apr 18 17:23 docker
/ #

That looks good … Let me try to repro on my side (I spawned a Fedora 37 on GCP)

ctalledo on Apr 18, 2023

Regarding the FS errors in the kernel log:

[ 128.715787] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
[ 128.715832] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.

These are not ideal but expected, and harmless.

What’s happening is that Sysbox needs to enable “ID-mapping” on the container’s rootfs. In order to do that, it needs to unmount the overlayfs mount setup by Docker, ID-map the lower layers, and remount it. That unmount/remount operation occurs inside the container’s mount namespace. This causes the kernel to detect that the same overlayfs upper/work dirs are active in two mounts (the init mount done by Docker and the remount by Sysbox), which then causes the warning. However it’s harmless because only the container will use that mount, so there is no risk of data corruption.

ctalledo on Apr 14, 2023

Sysbox-mgr log looks fine.

Which one would you recommend: ext4 or xfs?

Please try ext4 first.

Also, make sure /var/lib/sysbox is also on ext4. You can configure this via the sysbox-mgr’s --data-root=<path> option, by modifying the systemd unit service in /lib/systemd/system/sysbox-mgr.service.

ctalledo on Apr 14, 2023