talos: Unable to mount PVC in a pod after provisioning zfs-generic-nvmeof volume with democratic-csi

Sorry if this is the wrong place to post this, it may not be a bug in Talos Linux but I’m not sure where to look.

Bug Report

I’m using democratic-csi with ZFS on Linux and can successfully create a PVC bound to a PV. Now when I attempt to mount the volume in a pod, it’s stuck with the error MountVolume.MountDevice failed for volume "pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7" : rpc error: code = Unknown desc = unable to attach any nvme devices

Description

I believe Talos nodes already have the nvme modules built in to connect to an external nvme over TCP mount. In my case I’m running a separate TrueNAS Scale instance and I manually configured it as the root user for nvmeof. I tested manually from a separate Linux VM and was able to add the nvme share and mount it, along with reading/writing files to it.

With democratic-csi, I can create a PVC and it successfully provisions a PV and binds it:

root@localvm:~/# k get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
testpvc   Bound    pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7   5Gi        RWO            truenas-nvmeof   <unset>                 45m
root@localvm:~/# k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS     VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7   5Gi        RWO            Delete           Bound    default/testpvc   truenas-nvmeof   <unset>
   45m

But running a test pod, it’s stuck on the error message above. Here’s the pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: testlogger
spec:
    containers:
    - name: testlogger
      image: alpine:3.20
      command: ["/bin/ash"]
      args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
      volumeMounts:
      - name: testvol
        mountPath: /mnt
    volumes:
    - name: testvol
      persistentVolumeClaim:
        claimName: testpvc

Logs

MountVolume.MountDevice failed for volume "pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7" : rpc error: code = Unknown desc = unable to attach any nvme devices

Environment

  • Talos version:
Client:
        Tag:         v1.7.5
        SHA:         47731624
        Built:
        Go version:  go1.22.4
        OS/Arch:     linux/amd64
Server:
        NODE:        10.0.50.23
        Tag:         v1.7.4
        SHA:         cb3a8308
        Built:
        Go version:  go1.22.3
        OS/Arch:     linux/amd64
        Enabled:     RBAC
  • Kubernetes version:
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
  • Platform: Proxmox 8

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (12 by maintainers)

Most upvoted comments

I doesn’t relate in any way, it just shows there is no issue with Talos itself.

Mounting /dev in a container is always required, it’s not related to Talos.

The issue should be tracked from the CSI side - what exactly fails there.

My guess so far it’s something new and not available yet in Linux 6.6 Talos is using.