sops-nix: Secrets not showing up in /run/secrets after reboot

Just ran into this issue. We are using AWS KMS for keys with instance roles for access. Had to depend on network-online target, it seems just network is still too early for loading AWS role creds. Decided to use setupSecrets from activation scripts as it simple and allows to use all sops-nix features. Would be great to have native option for using service instead of activation script. Here is the code:

{
  systemd.services.decrypt-sops = {
    description = "Decrypt sops secrets";
    wantedBy = [ "multi-user.target" ];
    after = [ "network-online.target" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      # in network is not ready
      Restart = "on-failure";
      RestartSec = "2s";
    };
    script = config.system.activationScripts.setupSecrets.text;
   };
}

I’d like to use this on a system with impermanence, which does some bind-mounting between different partitions/{btr,z}fs volumes to make persistent data not disappear during reboots. This limitation is a bit awkward for that, and I’m sure many other use cases.

You’ll need to ensure that the persistent filesystems/datasets/whatevers, including the one(s) containing the sops key(s) used for decryption, have neededForBoot set so that they’re mounted early enough. Aside from that, impermanence runs using activation scripts* and doesn’t depend on mount units, so all that’s possibly-needed there is to ensure that sops’ activation script runs after impermanence’s. Enjoy:

{ config, lib, ... }:
let
  inherit (lib) filterAttrs mkIf;
  regularSecrets = filterAttrs (n: v: !v.neededForUsers) config.sops.secrets;
in
{
  # Ensure non-users-secrets from sops are only initialised *after*
  # impermanence's persistence module has linked files into place, otherwise we
  # likely do not have the decryption key (which is most-frequently the ssh
  # host key).
  config = mkIf (regularSecrets != {} && config.environment.persistence != {}) {
    system.activationScripts.setupSecrets.deps = [ "persist-files" ];
  };
}

*: It additionally uses per-file service units, but it does initial setup of the links/mounts via an activation script, so should be fine.

Ok. Unfortunately we cannot handle this usecase right now since we need to run before any additional mounts trigged by zfs or by systemd mount units are loaded.

Why is this the case? Just some silly ordering specific to how NixOS mounts filesystems, or something more annoying?

I’d like to use this on a system with impermanence, which does some bind-mounting between different partitions/{btr,z}fs volumes to make persistent data not disappear during reboots. This limitation is a bit awkward for that, and I’m sure many other use cases.