secureblue: [BUG] rpm-ostree can't deploy on Cosmic images
Describe the bug On the cosmic-main-hardened and cosmic-main-userns-hardened images (and probably nvidia images too), rpm-ostree can’t write new deployments. Attempting to install or uninstall overlays or set kargs fails, leaving the current deployment as the latest, and prints this error:
error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.
Here are some highlights from my journalctl -xe:
Aug 12 15:49:46 localhost rpm-ostree[13779]: g_atomic_ref_count_dec: assertion ‘old_value > 0’ failed Aug 12 15:49:46 localhost rpm-ostree[13779]: fatal allocator error: detected write after free Aug 12 15:49:46 localhost systemd[1]: rpm-ostreed.service: Main process exited, code=dumped, status=6/ABRT Aug 12 15:49:46 localhost systemd[1]: rpm-ostreed.service: Failed with result ‘core-dump’.
To Reproduce Steps to reproduce the behavior:
- boot the cosmic-main-hardened or cosmic-main-userns-hardened image
- run
rpm-ostree install,rpm-ostree uninstall,rpm-ostree kargs --append, etc. - See error
Expected behavior rpm-ostree should be able to write new deployments, just like on stable desktops.
Actual behavior rpm-ostree does not write new deployments, and reports an error.
Your current image
● ostree-image-signed:docker://ghcr.io/secureblue/cosmic-main-userns-hardened:latest Digest: sha256:498ee00ea9cfe33f0322092bae28053605f272657cb9dc4c92ff4b01646a511e Version: 40.1 (2024-08-12T05:34:07Z)
If you’re using a secureblue Bluefin image
- n/a
For all images
- I cannot reproduce on ostree-image-signed:docker://ghcr.io/ublue-os/cosmic:40-amd64
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 87 (31 by maintainers)
@idnovic I already tried locally layering those. The symbols still aren’t present
This is an anti-pattern, they are excluded by default for a reason
This is the place I got stuck too
We are slowly removing the workarounds. Setting it twice does not seem to cause problems
We also need to verify the files have the correct caps first
As for setcaps causing corruption, do not worry about it for now if it is not responsible.
The coredump will contain the responsible file somewhere. If you use debug symbols, you can do
brthen go to the stack that picks the file and doinfo locals. This will spit the file.When rechunk is used, it seals the whole current container in an OSTree commit, including permissions, xattrs, SELinux, everything.
If you build your image using the 24 runner, podman will maintain those permissions while youre building your container too, so in the end rechunk will see and maintain those perms.
The only thing that changed when rechunk was used is that now the OSTree commit contains a bunch of new files, and one of those files causes a small memory leak.
I tried to handle it, but I am not going to mess with trying to add debug symbols. Below is a link to one of the coredumps without debug symbols. https://we.tl/t-OcYO0dshLO
With the ubuntu 24 runner and rechunk they persist
@idnovic if you’re doing that at build time, it’s not actually working. caps don’t persist into the image, which is why we use a service
The log I posted is not the error log. ostree ls -CXR = list Checksum Xattrs Recursive
I think partial commits are what remains after you do the --delete operation.
So unless you cleanup your store they will remain.
If you have the error where files have a different hash, you can use
ostree ls ... | grep <hash>to show the file it belongs to.However, as this error happens during staging, I doubt it corrupts the ostree store.
I’m not at the computer any more. I’ll check it out in a couple of hours
Yes indeed different configuration. But we are about to deploy this to thousands of users so caution is justified
In fact we have already deployed it as part of bazzite
There is a file with an xattr in your image where when it is copied if it is done with the secure malloc it complains.
You can freely rebase from a hardened image to a nonhardened image and from a nonhardened image to a hardened image. The problem is going from a hardened image to a hardened image.
There is no filesystem corruption. You just need to do some clever rebasing to get around it.
As for which file it is. I do not know, there are no logs, so perhaps you need to do a clever lsof or something to see which was the latest file touched by ostree before it died.
Once that happens, if appropriate you can open an issue at the ostree repo so that it can be fixed. And if applicable I will add a quirk to rechunk so that it is fixed for following images.