proxmox-csi-plugin: Volume won't resize after original node is deleted from the cluster

Bug Report

Description

As the volume handle in the pv stores the original node name, for instance:

homelab/bismarck/iscsi-shared-volume/vm-9999-pvc-ad46c3c9-f8d4-42a3-a094-706615eb60ce

If the node (bismarck in this case) is no longer in the cluster, but the shared volume (iSCSI here) exists clusterwide, the controller will keep trying to address the old node. Additionally, it also seems to keep old Kubernetes nodes in memory as well.

Note that this PV is actually assigned to a different node currently.

We would expect that the plugin does not store the node name into the volume handle if the storage is clusterwide, but instead looks up the current assignment when resizing.

Logs

Controller: [kubectl logs -c proxmox-csi-plugin-controller proxmox-csi-plugin-controller-...]

I0531 20:54:25.630370       1 controller.go:649] "ControllerExpandVolume: called" args="{\"capacity_range\":{\"required_bytes\":53687091200},\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITE
R\"},\"mount\":{\"fs_type\":\"xfs\"}},\"volume_id\":\"homelab/bismarck/iscsi-shared-volume/vm-9999-pvc-ad46c3c9-f8d4-42a3-a094-706615eb60ce\"}"
I0531 20:54:28.594339       1 controller.go:520] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8te-worker-2\",\"volume_id\":\"homelab//iscsi-shared-volume/vm-9999-pvc-0de5d743-6a86-4840-99c9-d56355e8fdd5
\"}"
I0531 20:54:28.594343       1 controller.go:520] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8te-worker-2\",\"volume_id\":\"homelab/bismarck/fast-lvm/vm-9999-pvc-da5523e7-9bb6-4edc-8d33-20b3a73f913d\"}
"
I0531 20:54:28.594576       1 controller.go:520] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8te-worker-2\",\"volume_id\":\"homelab/bismarck/fast-lvm/vm-9999-pvc-8d887b82-3292-43e9-9360-0f70171f2617\"}
"
E0531 20:54:28.601999       1 main.go:101] "GRPC error" err="rpc error: code = InvalidArgument desc = nodes \"k8te-worker-2\" not found"
E0531 20:55:31.725229       1 controller.go:687] "ControllerExpandVolume: failed to verify the existence of the PVC" err="failed to get storage list: 500 hostname lookup 'bismarck' failed - failed to get address inf
o for: bismarck: Temporary failure in name resolution" cluster="homelab" volumeID="homelab/bismarck/iscsi-shared-volume/vm-9999-pvc-ad46c3c9-f8d4-42a3-a094-706615eb60ce"
E0531 20:55:31.725257       1 main.go:101] "GRPC error" err="rpc error: code = NotFound desc = failed to get storage list: 500 hostname lookup 'bismarck' failed - failed to get address info for: bismarck: Temporary
failure in name resolution"

(in this example both Proxmox node bismarck and Kubernetes node k8te-worker-2 were removed in the past.

Node: [kubectl logs -c proxmox-csi-plugin-node proxmox-csi-plugin-node-...]

Environment

  • Plugin version: ghcr.io/sergelogvinov/proxmox-csi-controller:v0.11.0, helm chart 0.3.7
  • Kubernetes version: [kubectl version --short] v1.29.4+k3s1
  • CSI capasity: [kubectl get csistoragecapacities -ocustom-columns=CLASS:.storageClassName,AVAIL:.capacity,ZONE:.nodeTopology.matchLabels -A]
  • CSI resource on the node: [kubectl get CSINode <node> -oyaml]
  • Node describe: [kubectl describe node <node>]
  • OS version [cat /etc/os-release]

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 22 (12 by maintainers)

Most upvoted comments

Hi, could you please check the edge version? I’m planning to make a bugfix release if everything looks good.

Seems to work perfectly, after upgrading ControllerUnpublishVolume got triggered on all the volumes that were mounted when the node was removed, then it logged the ‘node does not exist’ once. After, the logs did not come back again (unlike before).

Log example:

I0610 10:01:43.546941       1 controller.go:547] "ControllerUnpublishVolume: called" args="{\"node_id\":\"k8te-worker-2\",\"volume_id\":\"homelab//iscsi-shared-volume/vm-9999-pvc-0de5d743-6a86-4840-99c9-d56355e8fdd5\"}"
I0610 10:01:43.553263       1 controller.go:589] "ControllerUnpublishVolume: node does not exist" volumeID="homelab//iscsi-shared-volume/vm-9999-pvc-0de5d743-6a86-4840-99c9-d56355e8fdd5" nodeID="k8te-worker-2"

Let’s make the bugfix release. I will close this issue as both issues are resolved in the next version.

Sidenote: one question remains for me though: why did expand fail on the ‘wrong’ volume handles while normal attachment and detachment had no issues. Are you treating the volume handle in your code in two separate ways? This manual changeover is fine, but I am curious if this would have been preventable without it.

==========

For the next person that runs into a similar issue with deleted nodes causing issues with resizing (on shared storage):

  1. Make backup of all your PVC and PV resources.
  2. Add the annotation csi.proxmox.sinextra.dev/lifecycle: “keep” to the PV resources that have wrong volume handles (shared storage, but still include node name; handles should look like: {clustername}//{storagename}/{pvc id} for shared storage).
  3. Scale down deployments that use those PVCs to 0
  4. Delete the PVC (and PV)
  5. Recreate the PV and PVC from your backup, by applying them with kubectl create -f and after removing all annotations, versions, uids, status fields. If you are in doubt, reference https://github.com/sergelogvinov/proxmox-csi-plugin/blob/main/docs/faq.md#create-a-pvpvc-with-already-existing-disk for an example. Remove the node name from the volume handle before putting the backup back.
  6. Scale the deployments back up
  7. Everything should work again, including resizing disks.

Thanks, I found how to fix it…

I think I might know what happened. When you deleted the PV and recreated it from the backup, did you also remove the claimRef and any sidecar-related annotations (most of it)? These can affect the behavior of the sidecars, as they often use the annotations as priority value.

Sorry, I forgot to mention that earlier. we have the same logic here

proxmox-csi-plugin/cmd/pvecsictl/utils.go

Lines 75 to 82 in 85a0e89 newPV := pv.DeepCopy() newPV.ObjectMeta.UID = “” newPV.ObjectMeta.ResourceVersion = “” delete(newPV.ObjectMeta.Annotations, csi.DriverName+“/migrate”) delete(newPV.ObjectMeta.Annotations, csi.DriverName+“/migrate-node”) newPV.Spec.ClaimRef = nil newPV.Status = corev1.PersistentVolumeStatus{} newPV.Spec.CSI.VolumeHandle = volume.NewVolume(vol.Region(), node, vol.Storage(), vol.Disk()).VolumeID()

Can you clean all annotations in PV resources?

I found that out myself too, removing the claimRef is enough to get them to stay in the cluster, but to make ArgoCD happy you should clean them fully (remove all annotations, claimRef, versions, uids etc) and use kubectl create. Similar to: https://github.com/sergelogvinov/proxmox-csi-plugin/blob/main/docs/faq.md#create-a-pvpvc-with-already-existing-disk

I’ve done it to all of my old handles now and everything works (except the Kubernetes node thing).

Give me a few days - I think we can fix the “node does not exist” error.