proxmox-csi-plugin: Volume won't resize after original node is deleted from the cluster
Bug Report
Description
As the volume handle in the pv stores the original node name, for instance:
homelab/bismarck/iscsi-shared-volume/vm-9999-pvc-ad46c3c9-f8d4-42a3-a094-706615eb60ce
If the node (bismarck in this case) is no longer in the cluster, but the shared volume (iSCSI here) exists clusterwide, the controller will keep trying to address the old node. Additionally, it also seems to keep old Kubernetes nodes in memory as well.
Note that this PV is actually assigned to a different node currently.
We would expect that the plugin does not store the node name into the volume handle if the storage is clusterwide, but instead looks up the current assignment when resizing.
Logs
Controller: [kubectl logs -c proxmox-csi-plugin-controller proxmox-csi-plugin-controller-...]
I0531 20:54:25.630370 1 controller.go:649] "ControllerExpandVolume: called" args="{\"capacity_range\":{\"required_bytes\":53687091200},\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITE
R\"},\"mount\":{\"fs_type\":\"xfs\"}},\"volume_id\":\"homelab/bismarck/iscsi-shared-volume/vm-9999-pvc-ad46c3c9-f8d4-42a3-a094-706615eb60ce\"}"
I0531 20:54:28.594339 1 controller.go:520] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8te-worker-2\",\"volume_id\":\"homelab//iscsi-shared-volume/vm-9999-pvc-0de5d743-6a86-4840-99c9-d56355e8fdd5
\"}"
I0531 20:54:28.594343 1 controller.go:520] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8te-worker-2\",\"volume_id\":\"homelab/bismarck/fast-lvm/vm-9999-pvc-da5523e7-9bb6-4edc-8d33-20b3a73f913d\"}
"
I0531 20:54:28.594576 1 controller.go:520] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8te-worker-2\",\"volume_id\":\"homelab/bismarck/fast-lvm/vm-9999-pvc-8d887b82-3292-43e9-9360-0f70171f2617\"}
"
E0531 20:54:28.601999 1 main.go:101] "GRPC error" err="rpc error: code = InvalidArgument desc = nodes \"k8te-worker-2\" not found"
E0531 20:55:31.725229 1 controller.go:687] "ControllerExpandVolume: failed to verify the existence of the PVC" err="failed to get storage list: 500 hostname lookup 'bismarck' failed - failed to get address inf
o for: bismarck: Temporary failure in name resolution" cluster="homelab" volumeID="homelab/bismarck/iscsi-shared-volume/vm-9999-pvc-ad46c3c9-f8d4-42a3-a094-706615eb60ce"
E0531 20:55:31.725257 1 main.go:101] "GRPC error" err="rpc error: code = NotFound desc = failed to get storage list: 500 hostname lookup 'bismarck' failed - failed to get address info for: bismarck: Temporary
failure in name resolution"
(in this example both Proxmox node bismarck and Kubernetes node k8te-worker-2 were removed in the past.
Node: [kubectl logs -c proxmox-csi-plugin-node proxmox-csi-plugin-node-...]
Environment
- Plugin version: ghcr.io/sergelogvinov/proxmox-csi-controller:v0.11.0, helm chart 0.3.7
- Kubernetes version: [
kubectl version --short] v1.29.4+k3s1 - CSI capasity: [
kubectl get csistoragecapacities -ocustom-columns=CLASS:.storageClassName,AVAIL:.capacity,ZONE:.nodeTopology.matchLabels -A] - CSI resource on the node: [
kubectl get CSINode <node> -oyaml] - Node describe: [
kubectl describe node <node>] - OS version [
cat /etc/os-release]
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 22 (12 by maintainers)
Seems to work perfectly, after upgrading ControllerUnpublishVolume got triggered on all the volumes that were mounted when the node was removed, then it logged the ‘node does not exist’ once. After, the logs did not come back again (unlike before).
Log example:
Let’s make the bugfix release. I will close this issue as both issues are resolved in the next version.
Sidenote: one question remains for me though: why did expand fail on the ‘wrong’ volume handles while normal attachment and detachment had no issues. Are you treating the volume handle in your code in two separate ways? This manual changeover is fine, but I am curious if this would have been preventable without it.
==========
For the next person that runs into a similar issue with deleted nodes causing issues with resizing (on shared storage):
{clustername}//{storagename}/{pvc id}for shared storage).kubectl create -fand after removing all annotations, versions, uids, status fields. If you are in doubt, reference https://github.com/sergelogvinov/proxmox-csi-plugin/blob/main/docs/faq.md#create-a-pvpvc-with-already-existing-disk for an example. Remove the node name from the volume handle before putting the backup back.Thanks, I found how to fix it…
I found that out myself too, removing the claimRef is enough to get them to stay in the cluster, but to make ArgoCD happy you should clean them fully (remove all annotations, claimRef, versions, uids etc) and use kubectl create. Similar to: https://github.com/sergelogvinov/proxmox-csi-plugin/blob/main/docs/faq.md#create-a-pvpvc-with-already-existing-disk
I’ve done it to all of my old handles now and everything works (except the Kubernetes node thing).
Give me a few days - I think we can fix the “node does not exist” error.