talos: Unable to mount PVC in a pod after provisioning zfs-generic-nvmeof volume with democratic-csi
Sorry if this is the wrong place to post this, it may not be a bug in Talos Linux but I’m not sure where to look.
Bug Report
I’m using democratic-csi with ZFS on Linux and can successfully create a PVC bound to a PV. Now when I attempt to mount the volume in a pod, it’s stuck with the error MountVolume.MountDevice failed for volume "pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7" : rpc error: code = Unknown desc = unable to attach any nvme devices
Description
I believe Talos nodes already have the nvme modules built in to connect to an external nvme over TCP mount. In my case I’m running a separate TrueNAS Scale instance and I manually configured it as the root user for nvmeof. I tested manually from a separate Linux VM and was able to add the nvme share and mount it, along with reading/writing files to it.
With democratic-csi, I can create a PVC and it successfully provisions a PV and binds it:
root@localvm:~/# k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
testpvc Bound pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7 5Gi RWO truenas-nvmeof <unset> 45m
root@localvm:~/# k get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7 5Gi RWO Delete Bound default/testpvc truenas-nvmeof <unset>
45m
But running a test pod, it’s stuck on the error message above. Here’s the pod yaml:
apiVersion: v1
kind: Pod
metadata:
name: testlogger
spec:
containers:
- name: testlogger
image: alpine:3.20
command: ["/bin/ash"]
args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
volumeMounts:
- name: testvol
mountPath: /mnt
volumes:
- name: testvol
persistentVolumeClaim:
claimName: testpvc
Logs
MountVolume.MountDevice failed for volume "pvc-13b4ff6a-90ee-48e1-be6a-f011824f63c7" : rpc error: code = Unknown desc = unable to attach any nvme devices
Environment
- Talos version:
Client:
Tag: v1.7.5
SHA: 47731624
Built:
Go version: go1.22.4
OS/Arch: linux/amd64
Server:
NODE: 10.0.50.23
Tag: v1.7.4
SHA: cb3a8308
Built:
Go version: go1.22.3
OS/Arch: linux/amd64
Enabled: RBAC
- Kubernetes version:
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
- Platform: Proxmox 8
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (12 by maintainers)
I doesn’t relate in any way, it just shows there is no issue with Talos itself.
Mounting
/devin a container is always required, it’s not related to Talos.The issue should be tracked from the CSI side - what exactly fails there.
My guess so far it’s something new and not available yet in Linux 6.6 Talos is using.