Bug Report

I recently was experimenting with the quay.io/fedora/fedora-coreos:next-devel container on one of my systems. I rebased it using the following command and left it like that for a few releases (manually running rpm-ostree upgrade each cycle).

sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:next-devel

I then decided to go back to having automatic updates working (zincati + OSTree repo) so I rebased back to what I had to begin with:

sudo rpm-ostree rebase fedora:fedora/x86_64/coreos/next

and ended up with:

[core@weevm ~]$ rpm-ostree status 
State: idle
Deployments:
● fedora:fedora/x86_64/coreos/next
                  Version: 37.20221003.1.0 (2022-10-03T17:40:38Z)
                   Commit: df296c944c11261f69f0e3b26d04b7e4016e3f5dd38cd3cc4cd05deab72f7474
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A

  ostree-unverified-registry:quay.io/fedora/fedora-coreos:next-devel
                   Digest: sha256:10e906cc2514a8098343eb8e813d59e549af47fb43ecbc2345f3c40a0d3ba702
                Timestamp: 2022-09-29T19:39:44Z

However I noticed zincati fails to start now:

Oct 11 19:09:43 weevm systemd[1]: Starting zincati.service - Zincati Update Agent...
Oct 11 19:09:43 weevm zincati[442167]: [INFO  zincati::cli::agent] starting update agent (zincati 0.0.24)
Oct 11 19:09:43 weevm zincati[442167]: [ERROR zincati] critical error: failed to assemble configuration settings
Oct 11 19:09:43 weevm zincati[442167]: [ERROR zincati]  -> failed to validate agent identity configuration
Oct 11 19:09:43 weevm zincati[442167]: [ERROR zincati]  -> failed to build default identity
Oct 11 19:09:43 weevm zincati[442167]: [ERROR zincati]  -> missing field `coreos-assembler.basearch` at line 77 column 7
Oct 11 19:09:43 weevm systemd[1]: zincati.service: Main process exited, code=exited, status=1/FAILURE
Oct 11 19:09:43 weevm systemd[1]: zincati.service: Failed with result 'exit-code'.
Oct 11 19:09:43 weevm systemd[1]: Failed to start zincati.service - Zincati Update Agent.
Oct 11 19:09:54 weevm systemd[1]: zincati.service: Scheduled restart job, restart counter is at 52512.
Oct 11 19:09:54 weevm systemd[1]: Stopped zincati.service - Zincati Update Agent.

This is because there is apparently some data missing about the container deployment that causes zincati to barf and not continue best effort. I would assume that since the booted deployment is good zincati should be able to continue.

After cleaning up the rollback deployment with sudo rpm-ostree rollback -r zincati was able to start.

Environment

QEMU - FCOS at 37.20221003.1.0

Expected Behavior

Able to start zincati.

Actual Behavior

zincati fails to start up.

Reproduction Steps

1. boot system
2. sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:next and reboot
3. sudo rpm-ostree rebase fedora:fedora/x86_64/coreos/next and reboot
4. zincati.service should fail to start now because rollback deployment doesn’t conform.

Other Information