talos: cryptsetup failing in 1.9.4, successful in 1.8.3

Bug Report

Description

After upgrade from 1.8.3 to 1.9.4, we noticed that encrypted PVs couldn’t be mounted anymore on the Talos node that was upgraded. Other nodes (still on 1.8.3) were still capable of mounting the same PVs.

Booted the upgraded node back to 1.8.3, and mounting the volumes worked again. Booted again into 1.9.4, the issue returned. Decided to rollback to 1.8.3 for the time being. I couldn’t relate it to any part of the release notes, so therefore this issue. And as in Rook and Ceph versions nothing changed, and they’re at their latest versions, I’d expect it to be a Talos issue.

Configuration

Encrypted storageclass using:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
...
parameters:
...
    encrypted: "true"
    encryptionKMSID: example-secret-name
...

Logs

Pod event logs:

MountVolume.MountDevice failed for volume "pvc-ba495824-4fd2-470e-a13b-c389d5cb0a60" : rpc error: code = Internal desc = an error (exit status 1) occurred while running cryptsetup args: [luksOpen /dev/rbd4 luks-rbd-0001-0009-rook-ceph-0000000000000008-e1890dfc-b7e1-4692-a46d-7dc871010d67 --disable-keyring -d /dev/stdin]

csi-rbdplugin logs:

2025-03-05T09:42:11.843928203Z W0305 09:42:11.843768  247407 rbd_attach.go:238] nbd modprobe failed (an error (exit status 1) occurred while running modprobe args: [nbd]): "modprobe: ERROR: could not insert 'nbd': Operation not permitted\n"
2025-03-05T09:42:48.289039584Z E0305 09:42:48.288918  247407 crypto.go:283] ID: 14 Req-ID: 0001-0009-rook-ceph-0000000000000008-b7d268cc-e8bf-450e-8765-f4e3fc125954 failed to open device "/dev/rbd5" (an error (exit status 1) occurred while running cryptsetup args: [luksOpen /dev/rbd5 luks-rbd-0001-0009-rook-ceph-0000000000000008-b7d268cc-e8bf-450e-8765-f4e3fc125954 --disable-keyring -d /dev/stdin]): Failed to open key file.
2025-03-05T09:42:48.289077695Z E0305 09:42:48.288962  247407 encryption.go:300] ID: 14 Req-ID: 0001-0009-rook-ceph-0000000000000008-b7d268cc-e8bf-450e-8765-f4e3fc125954 failed to open device replicatedpool/csi-vol-b7d268cc-e8bf-450e-8765-f4e3fc125954: an error (exit status 1) occurred while running cryptsetup args: [luksOpen /dev/rbd5 luks-rbd-0001-0009-rook-ceph-0000000000000008-b7d268cc-e8bf-450e-8765-f4e3fc125954 --disable-keyring -d /dev/stdin]
2025-03-05T09:42:48.383675741Z E0305 09:42:48.383560  247407 utils.go:271] ID: 14 Req-ID: 0001-0009-rook-ceph-0000000000000008-b7d268cc-e8bf-450e-8765-f4e3fc125954 GRPC error: rpc error: code = Internal desc = an error (exit status 1) occurred while running cryptsetup args: [luksOpen /dev/rbd5 luks-rbd-0001-0009-rook-ceph-0000000000000008-b7d268cc-e8bf-450e-8765-f4e3fc125954 --disable-keyring -d /dev/stdin]

Environment

  • Talos version: 1.9.4 / 1.8.3
  • Kubernetes version: 1.31.3
  • Platform: Rook 1.16.4 / Ceph v19.2.1-20250202

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 28 (8 by maintainers)

Most upvoted comments

PR was merged into ceph-csi devel branch and will be included in the next release (latest release is currently 3.14.0, so probably 3.14.1 or 3.15.0). 🥳

For other Rook users, like me, that means:

  1. Wait for the Ceph CSI release
  2. Wait until Rook includes the new Ceph CSI release (Rook 1.17.0/1.17.1 contain 3.14.0)
  3. Upgrade to that Rook release
  4. Finally upgrading to a Talos release newer than 1.8.x

I tried creating the issue at Ceph-side; see https://github.com/ceph/ceph-csi/issues/5306

Great that the problem is solved now in Longhorn, should a similar issue be created at Rook/Ceph side?

1.9.0 of Longhorn fixes this issue, we can likely close it.

This issue is for Rook/Ceph, not for Longhorn. I’m glad that Longhorn users have a solution earlier 😆 but I don’t quite agree that the issue has been completely resolved yet. And I still can’t really understand why this changed between Talos 1.8 and 1.9. I can imagine, for future changes, it’d be useful to find out what caused this to break, even though the current behavior of both Ceph and Longhorn was not 100% according to specs.

Longhorn uses -d /dev/stdin because -d - requires interactive input. If you run the following command directly for testing, it should fail due to the missing piped input into /dev/stdin:

This is not true. The proper way is to follow cryptsetup documentation:

/ # echo "passphrase" | nsenter -t 1 -m cryptsetup  -q luksFormat --type luks2 --ciphe
r aes-xts-plain64 --hash sha256 --key-size 256 --pbkdf argon2i /dev/loop1  -d /dev/std
in
Failed to open key file.
/ # echo -n "passphrase" | nsenter -t 1 -m cryptsetup  -q luksFormat --type luks2 --ciphe
r aes-xts-plain64 --hash sha256 --key-size 256 --pbkdf argon2i /dev/loop1  -d -
/ # (successfully encrypted)

The -d /dev/stdin simply doesn’t work.

As mentioned in issue #10469, it appears there is a problem with cryptsetup in Talos 1.9.4 (or possibly an earlier version). Longhorn volume encryption does not work in 1.9.4, although it functions correctly in 1.8.4.