sysbox: Unable to start container with sysbox runtime after kernel update.
Running Ubuntu 22.04, and just received kernel update from 5.15.0-47 to 5.15.0-48, matching this security advisory, and It seems that containers can no-longer be started with the runtime:-
https://ubuntu.com/security/notices/USN-5624-1
# docker run --runtime sysbox-runc -it nestybox/ubuntu-focal-docker:latest /bin/bash
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:425: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:366: failed to mkdirall /var/lib/sysbox/shiftfs/002a816d-d852-4e38-ac0d-d6b37bbdd8ea/var/lib/rancher/rke2: mkdir /var/lib/sysbox/shiftfs/002a816d-d852-4e38-ac0d-d6b37bbdd8ea/var/lib/rancher: value too large for defined data type caused: mkdir /var/lib/sysbox/shiftfs/002a816d-d852-4e38-ac0d-d6b37bbdd8ea/var/lib/rancher: value too large for defined data type: unknown.
Rolled back to 5.15.0-47 and it seems to be working again.
Installed from “sysbox-ce_0.5.2-0.linux_amd64.deb” - Any thoughts would be appreciated.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 28 (15 by maintainers)
FYI: commit with the fix for shiftfs in Ubuntu: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/lunar/commit/fs/shiftfs.c?h=master-next&id=cfe3544e11cc53e0038410a2199ee6afeea3687f
Should be present in the upcoming Ubuntu 23.04 release (Lunar Lobster), due April 2023.
NOTE: the upcoming release of Sysbox (v0.6.0) will automatically check if shiftfs works on the host or not, and adjust accordingly. In platforms where it works, it will use it as needed. In platforms where it does not work, it will use an alternative mechanism. The new Sysbox release will also automatically check if the kernel supports ID-mapped mounts (kernel 5.12+) and overlayfs on ID-mapped mounted lower dirs (kernel 5.19+), and use both of these features. The latter one really makes shiftfs unnecessary going forward.
Updating to 5.19.0-28 fixed the problem for me 😃
It’s broken since 5.15-0-48 I believe, and I believe in 5.17 and possibly 5.19 too; we don’t know when the fix is coming so it’s hard to tie it to a kernel version.
@felipecrs
This is a good idea, especially with so many kernel changes going on these days in this area. During sysbox-mgr’s initialization we could attempt to mount a shiftfs resource and decide to enable/disable shiftfs based on this.
Oh yeah, I edited it. Thanks!
That’s great, thanks for digging that up @philipzgithub.
Thank you @ctalledo for your response, you have saved me a lot of time and effort.
Here’s a one-liner to disable shiftfs:
I wonder if it’s worthwhile to add a check in the sysbox-mgr to automatically disable using shiftfs in known broken kernels.
This does help; thanks!
Hi @netlore , thanks for using Sysbox.
The error is for sure caused by incompatibility between shiftfs and the kernel (nothing in sysbox per-se).
Just yesterday someone else reported this issue too: https://github.com/nestybox/sysbox/issues/595
There must be something in the 5.15.0-48 kernel that is causing the incompatibility with the shiftfs module. Speculating a bit, maybe the kernel is missing a Ubuntu patch required for overlayfs to work with shiftfs, or maybe the shiftfs module needs updating to work with this kernel.
We would need to dig deep into the commits of 5.15.0-48 to see what’s going on.
If rolling back to the prior kernel is not an option for you, as a workaround you can try using a newer kernel (maybe 5.18?) or configuring Sysbox to not use shiftfs (it will instead use an alternative mechanism called ID-mapped-mounts in the kernel). To do the latter, modify the sysbox systemd service for the sysbox-mgr and pass the
--disable-shiftfsflag to it.Disabling shiftfs is not ideal, but things should still work without it.