kanidm: Database migrations/restore failing, Uuid referenced not found in database

I did this

I upgraded kanidm-server to 1.1.0-alpha12 and attempted to start it, but this resulted in the following errors at startup:

May 03 15:58:42 petra.fuwafuwatime.moe kanidmd[626988]: 015add86-a8d2-4473-9f23-84adcb46b3ad ERROR       ┝━ 🚨 [error]: UUID reference set size differs from query result size <fast path, no uuid info available> | event_tag_id: 1
May 03 15:58:42 petra.fuwafuwatime.moe kanidmd[626988]: 015add86-a8d2-4473-9f23-84adcb46b3ad ERROR       ┝━ 🚨 [error]: Post-Modify operation failed (plugin), Plugin(ReferentialIntegrity("Uuid referenced not found in database")) | event_tag_id: 1
May 03 15:58:42 petra.fuwafuwatime.moe kanidmd[626988]: 015add86-a8d2-4473-9f23-84adcb46b3ad ERROR       ┕━ 🚨 [error]: initialise_idm p3 -> result | event_tag_id: 1 | res: Err(Plugin(ReferentialIntegrity("Uuid referenced not found in database")))
May 03 15:58:42 petra.fuwafuwatime.moe kanidmd[626988]: 00000000-0000-0000-0000-000000000000 ERROR    🚨 [error]: Unable to setup query server or idm server -> Plugin(ReferentialIntegrity("Uuid referenced not found in database"))
May 03 15:58:42 petra.fuwafuwatime.moe kanidmd[626988]: 00000000-0000-0000-0000-000000000000 ERROR    🚨 [error]: Failed to start server core!

Subsequent attempts to start kanidm-server after these errors resulted in these errors:

May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: ddf4ebc5-e705-40d4-8048-3fa441f71e27 WARN     ┝━ 🚧 [warn]: starting 9 to 10 migration. | event_tag_id: 2
May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: ddf4ebc5-e705-40d4-8048-3fa441f71e27 INFO     ┝━ ｉ [info]: allowed 5 entries ✅ | event_tag_id: 10
May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: ddf4ebc5-e705-40d4-8048-3fa441f71e27 INFO     ┝━ ｉ [info]: migrate_10_to_11 no entries to migrate, complete | event_tag_id: 3
May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: ddf4ebc5-e705-40d4-8048-3fa441f71e27 WARN     ┝━ 🚧 [warn]: starting 11 to 12 migration. | event_tag_id: 2
May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: ddf4ebc5-e705-40d4-8048-3fa441f71e27 INFO     ┝━ ｉ [info]: allowed 1 entries ✅ | event_tag_id: 10
May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: ddf4ebc5-e705-40d4-8048-3fa441f71e27 ERROR    ┕━ 🚨 [error]: Failed to convert api_token_session from session -> apitoken
May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: 00000000-0000-0000-0000-000000000000 ERROR    🚨 [error]: Unable to setup query server or idm server -> InvalidValueState
May 03 15:59:09 petra.fuwafuwatime.moe kanidmd[627235]: 00000000-0000-0000-0000-000000000000 ERROR    🚨 [error]: Failed to start server core!

It turns out that all of my database backups going back about 4 weeks are producing the same first batch of errors when trying to restore the database, even on version 1.1.0-alpha11, so this isn’t related to a problematic upgrade.

I expected the following

Database migrations should have completed successfully and all of the database backups that are currently causing errors should be restoreable without problems.

Kanidm version (and git commit)

I can reproduce this on both 1.1.0-alpha11 (d3a2a6b) and 1.1.0-alpha12 (bcdbb18).

Operating System / Version

Gentoo Linux

Linux petra.fuwafuwatime.moe 6.1.20-gentoo-hardened1 #1 SMP Fri Mar 24 08:05:53 EDT 2023 x86_64 Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz GenuineIntel GNU/Linux

Any other comments

N/A