secureblue: [BUG] rpm-ostree can't deploy on Cosmic images

Describe the bug On the cosmic-main-hardened and cosmic-main-userns-hardened images (and probably nvidia images too), rpm-ostree can’t write new deployments. Attempting to install or uninstall overlays or set kargs fails, leaving the current deployment as the latest, and prints this error:

error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.

Here are some highlights from my journalctl -xe:

Aug 12 15:49:46 localhost rpm-ostree[13779]: g_atomic_ref_count_dec: assertion ‘old_value > 0’ failed Aug 12 15:49:46 localhost rpm-ostree[13779]: fatal allocator error: detected write after free Aug 12 15:49:46 localhost systemd[1]: rpm-ostreed.service: Main process exited, code=dumped, status=6/ABRT Aug 12 15:49:46 localhost systemd[1]: rpm-ostreed.service: Failed with result ‘core-dump’.

To Reproduce Steps to reproduce the behavior:

  1. boot the cosmic-main-hardened or cosmic-main-userns-hardened image
  2. run rpm-ostree install, rpm-ostree uninstall, rpm-ostree kargs --append, etc.
  3. See error

Expected behavior rpm-ostree should be able to write new deployments, just like on stable desktops.

Actual behavior rpm-ostree does not write new deployments, and reports an error.

Your current image

● ostree-image-signed:docker://ghcr.io/secureblue/cosmic-main-userns-hardened:latest Digest: sha256:498ee00ea9cfe33f0322092bae28053605f272657cb9dc4c92ff4b01646a511e Version: 40.1 (2024-08-12T05:34:07Z)

If you’re using a secureblue Bluefin image

  • n/a

For all images

  • I cannot reproduce on ostree-image-signed:docker://ghcr.io/ublue-os/cosmic:40-amd64

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 87 (31 by maintainers)

Most upvoted comments

@idnovic I already tried locally layering those. The symbols still aren’t present

This is something we could add for everyone. It simplifies debug operations.

This is an anti-pattern, they are excluded by default for a reason

This is the place I got stuck too

We are slowly removing the workarounds. Setting it twice does not seem to cause problems

We also need to verify the files have the correct caps first

As for setcaps causing corruption, do not worry about it for now if it is not responsible.

The coredump will contain the responsible file somewhere. If you use debug symbols, you can do br then go to the stack that picks the file and do info locals. This will spit the file.

When rechunk is used, it seals the whole current container in an OSTree commit, including permissions, xattrs, SELinux, everything.

If you build your image using the 24 runner, podman will maintain those permissions while youre building your container too, so in the end rechunk will see and maintain those perms.

The only thing that changed when rechunk was used is that now the OSTree commit contains a bunch of new files, and one of those files causes a small memory leak.

I think these are multiple bugs. 2 or 3? Can you two @antheas @qoijjj handle the memory issue with hmalloc? I can test our setcaps thing

I tried to handle it, but I am not going to mess with trying to add debug symbols. Below is a link to one of the coredumps without debug symbols. https://we.tl/t-OcYO0dshLO

@idnovic if you’re doing that at build time, it’s not actually working. caps don’t persist into the image, which is why we use a service

With the ubuntu 24 runner and rechunk they persist

@idnovic if you’re doing that at build time, it’s not actually working. caps don’t persist into the image, which is why we use a service

@idnovic those all have something in common:

https://github.com/secureblue/secureblue/blob/live/files/system/usr/bin/setcapsforunsuidbinaries

we set caps on all of those files to replace suid.

The error says the same:

“security.capability”

This seems like a red herring at best.

The log I posted is not the error log. ostree ls -CXR = list Checksum Xattrs Recursive

I think partial commits are what remains after you do the --delete operation.

So unless you cleanup your store they will remain.

If you have the error where files have a different hash, you can use ostree ls ... | grep <hash> to show the file it belongs to.

However, as this error happens during staging, I doubt it corrupts the ostree store.

I’m not at the computer any more. I’ll check it out in a couple of hours

Yes indeed different configuration. But we are about to deploy this to thousands of users so caution is justified

In fact we have already deployed it as part of bazzite

image

There is a file with an xattr in your image where when it is copied if it is done with the secure malloc it complains.

You can freely rebase from a hardened image to a nonhardened image and from a nonhardened image to a hardened image. The problem is going from a hardened image to a hardened image.

There is no filesystem corruption. You just need to do some clever rebasing to get around it.

As for which file it is. I do not know, there are no logs, so perhaps you need to do a clever lsof or something to see which was the latest file touched by ostree before it died.

Once that happens, if appropriate you can open an issue at the ostree repo so that it can be fixed. And if applicable I will add a quirk to rechunk so that it is fixed for following images.