cloudstack: Cloudstack volume is not deleted when PVC is deleted

ISSUE TYPE

• Bug Report

COMPONENT NAME

Storage volumes, Kubernetes Service

CLOUDSTACK VERSION

4.17.2.0

CONFIGURATION

• Advanced networking with vxlan isolation of guest networks
• Kubernetes service enabled
• Cloudstack CSI driver deployed

OS / ENVIRONMENT

Rocky Linnux 8.7

SUMMARY

When a Kubernetes persistent volume claim is deleted, the Cloudstack volume is not deleted, and an error message is written to the management log.

STEPS TO REPRODUCE

• Install the CSI driver in the K8s cluster using the command :

kubectl apply -f https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml

• Deploy this storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cloudstack-custom
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
  csi.cloudstack.apache.org/disk-offering-id: <id of custom disk offering>

• Deploy this pvs and pod:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-pvc
spec:
  storageClassName: cloudstack-custom
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
    - name: example
      image: busybox
      volumeMounts:
        - mountPath: "/data"
          name: example-volume
      stdin: true
      stdinOnce: true
      tty: true
  volumes:
    - name: example-volume
      persistentVolumeClaim:
        claimName: example-pvc

• Delete the pod and pvc

EXPECTED RESULTS

I expected the Cloudstack storage volume backing the PVC would be deleted.

ACTUAL RESULTS

The storage volume was not deleted, and the following error was printed to the management log:

2023-03-05 14:18:39,144 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658 ctx-d6957fbc) (logid:fd682bcf) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment
2023-03-05 14:18:39,144 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658 ctx-d6957fbc) (logid:fd682bcf) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment
2023-03-05 14:18:39,144 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658) (logid:fd682bcf) Done with run of VM work job: com.cloud.vm.VmWorkDetachVolume for VM 188, job origin: 1657
2023-03-05 14:18:39,144 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-92:ctx-3f7cf487 job-1657/job-1658) (logid:fd682bcf) Unable to complete AsyncJobVO: {id:1658, userId: 4, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkDetachVolume, cmdInfo: rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrRGV0YWNoVm9sdW1lhvReA-s81AMCAAFMAAh2b2x1bWVJZHQAEExqYXZhL2xhbmcvTG9uZzt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAAEAAAAAAAAALx0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAvA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 128437426857854, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sun Mar 05 14:18:36 EST 2023, removed: null}, job origin:1657
com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume pvc-0e82d1a8-9af2-40b3-b037-ebcf151febd3 from VM testk8s-node-186b0939378; com.cloud.exception.InternalErrorException: Could not detach volume. Probably the VM is in boot state at the moment
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:2793)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:4404)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4435)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy231.handleVmWorkJob(Unknown Source)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)