democratic-csi: GrpcError when mounting iscsi disk - Nomad
Hi @travisghansen, Thank you for the great support you provide for this storage plugin!
I ran across an issue recently when I was updating my systems. Now when I deploy a container with iscsi storage it fails with the following error:
Setup Failure: failed to setup alloc: pre-run hook "csi_hook" failed: mounting volumes: rpc error: code = InvalidArgument desc = unknown/unsupported node_attach_driver: undefined
When I look at the node logs I see this:
Node logs
{"host":"nomad1","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeGetCapabilities response: {\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME_STATS\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}}]}","service":"democratic-csi","timestamp":"2023-07-25T18:07:53.412Z"}
{"host":"nomad1","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeStageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.53.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{},\"volume_id\":\"nvme.traefik\",\"staging_target_path\":\"/local/csi/staging/nvme.traefik/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}},\"cancelled\":false}","service":"democratic-csi","timestamp":"2023-07-25T18:08:16.164Z"}
{"host":"nomad1","level":"error","message":"handler error - driver: FreeNASApiDriver method: NodeStageVolume error: {\"name\":\"GrpcError\",\"code\":3,\"message\":\"unknown/unsupported node_attach_driver: undefined\"}","service":"democratic-csi","timestamp":"2023-07-25T18:08:16.166Z"}
{"host":"nomad1","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnpublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.53.0\"]},\"request\":{\"volume_id\":\"nvme.traefik\",\"target_path\":\"/local/csi/per-alloc/9b518b5e-c1a8-a6ed-00dc-7964c1d103ef/nvme.traefik/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2023-07-25T18:08:16.174Z"}
executing mount command: findmnt --mountpoint /local/csi/per-alloc/9b518b5e-c1a8-a6ed-00dc-7964c1d103ef/nvme.traefik/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
retry - failed condition, not trying again
{"host":"nomad1","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeUnpublishVolume response: {}","service":"democratic-csi","timestamp":"2023-07-25T18:08:16.212Z"}
{"host":"nomad1","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnstageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.53.0\"]},\"request\":{\"volume_id\":\"nvme.traefik\",\"staging_target_path\":\"/local/csi/staging/nvme.traefik/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2023-07-25T18:08:16.216Z"}
executing mount command: findmnt --mountpoint /local/csi/staging/nvme.traefik/rw-file-system-single-node-writer/block_device --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/staging/nvme.traefik/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/staging/nvme.traefik/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: rmdir /local/csi/staging/nvme.traefik/rw-file-system-single-node-writer
{"host":"nomad1","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeUnstageVolume response: {}","service":"democratic-csi","timestamp":"2023-07-25T18:08:16.335Z"}
{"host":"nomad1","level":"info","message":"new request - driver: FreeNASApiDriver method: Probe call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.53.0\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2023-07-25T18:08:23.148Z"}
{"host":"nomad1","level":"info","message":"new response - driver: FreeNASApiDriver method: Probe response: {\"ready\":{\"value\":true}}","service":"democratic-csi","timestamp":"2023-07-25T18:08:23.148Z"}
From the node logs it looks like it is complaining that a grpc variable is undefined? Not sure what changed with the grpc options between Nomad 1.5.6 and 1.6.1. I have not tried downgrading Nomad yet but I can do that if you think that is the right direction.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 37 (16 by maintainers)
Sheesh, thanks for your diligence in getting it figured out! Once a solution is discovered let’s get it documented.
Oh right, sorry, too many things going on and I get wires crossed 😦
Are you running the node job with host network and full privileges etc?
I think this is a popular option: https://github.com/backube/volsync
some folks like velero.
Ok something is quite strange if originally the map was empty, then you added a single data point and now it has all the other values 😦
It’s not something that would get added to the deployment at all. It’s data that typically would be stored by nomad as part of the return from the volume create process.
In the first comment which has the node logs you can see the
volume_contextattribute in theNodeStageVolumerequest logs. For iscsi volumes that map should have several attributes includingnode_attach_driver(which should always be included in that call no matter the volume type), but also the target portals, lun, iqn, etc.