vsphere-csi-driver: PV stays hanging in released state

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: I deploy a stateful set with 3 replicas and 3 PVCs (via storageclass). When I delete the statefulset and immediately delete the PVCs, most of the PVs stays hanging in status Released. When I wait a some seconds before I delete the PVCs, this problem does not occur. This problem does also not happen with csi-driver 1.0.2. In vCenter I constantly see the error “The operation is not allowed in the current state”. It seems that the driver tries to delete the storage object before it has been detached from the node.

A workaound to remove the hanging PVs is to remove the PV finalizers: kubectl patch pv pvc-*** -p ‘{“metadata”:{“finalizers”:null}}’

What you expected to happen: PVs do not hang in the Released status and are removed.

How to reproduce it (as minimally and precisely as possible): Deploy a stateful set with 3 replicas and 3 PVCs (via storageclass). Delete the statefulset and immediately delete the PVCs.

Anything else we need to know?: csi-attacher logs

I0505 12:15:09.079852 1 csi_handler.go:428] Saving detach error to "csi-50820c7e61c23397a1ef7a282e237cae50eeebc6f416cf22fb04a951d21398e2"
I0505 12:15:09.089540 1 csi_handler.go:439] Saved detach error to "csi-50820c7e61c23397a1ef7a282e237cae50eeebc6f416cf22fb04a951d21398e2"
I0505 12:15:09.089591 1 csi_handler.go:99] Error processing "csi-50820c7e61c23397a1ef7a282e237cae50eeebc6f416cf22fb04a951d21398e2": failed to detach: rpc error: code = Aborted desc = pending
I0505 12:15:09.089942 1 controller.go:141] Ignoring VolumeAttachment "csi-50820c7e61c23397a1ef7a282e237cae50eeebc6f416cf22fb04a951d21398e2" change
I0505 12:15:14.026312 1 csi_handler.go:428] Saving detach error to "csi-d2da7a09f4310e2889cdd422ac8e6a8e3cb30f1eb93c1a35d023872acdeb1a9c"
I0505 12:15:14.033809 1 controller.go:141] Ignoring VolumeAttachment "csi-d2da7a09f4310e2889cdd422ac8e6a8e3cb30f1eb93c1a35d023872acdeb1a9c" change
I0505 12:15:14.035294 1 csi_handler.go:439] Saved detach error to "csi-d2da7a09f4310e2889cdd422ac8e6a8e3cb30f1eb93c1a35d023872acdeb1a9c"
I0505 12:15:14.035351 1 csi_handler.go:99] Error processing "csi-d2da7a09f4310e2889cdd422ac8e6a8e3cb30f1eb93c1a35d023872acdeb1a9c": failed to detach: rpc error: code = Aborted desc = pending

csi-controller logs

{"level":"error","time":"2020-05-05T12:15:06.865649379Z","caller":"volume/manager.go:433","msg":"failed to delete volume: \"8290b052-db75-4e06-a181-5ebcf4bdea4c\", fault: \"(*types.LocalizedMethodFault)(0xc000b3e2c0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n BaseMethodFault: (types.BaseMethodFault) <nil>,\\n Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"076c157f\"","TraceId":"910d62cb-2091-4e5f-ab36-d9e40f6b2349","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/volume.(*defaultManager).DeleteVolume\n\t/build/pkg/common/cns-lib/volume/manager.go:433\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/common.DeleteVolumeUtil\n\t/build/pkg/csi/service/common/vsphereutil.go:349\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:449\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/
rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2020-05-05T12:15:06.865869018Z","caller":"common/vsphereutil.go:351","msg":"failed to delete disk 8290b052-db75-4e06-a181-5ebcf4bdea4c with error failed to delete volume: \"8290b052-db75-4e06-a181-5ebcf4bdea4c\", fault: \"(*types.LocalizedMethodFault)(0xc000b3e2c0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n BaseMethodFault: (types.BaseMethodFault) <nil>,\\n Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"076c157f\"","TraceId":"910d62cb-2091-4e5f-ab36-d9e40f6b2349","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/common.DeleteVolumeUtil\n\t/build/pkg/csi/service/common/vsphereutil.go:351\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:449\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218
\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2020-05-05T12:15:06.865947275Z","caller":"vanilla/controller.go:452","msg":"failed to delete volume: \"8290b052-db75-4e06-a181-5ebcf4bdea4c\". Error: failed to delete volume: \"8290b052-db75-4e06-a181-5ebcf4bdea4c\", fault: \"(*types.LocalizedMethodFault)(0xc000b3e2c0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n BaseMethodFault: (types.BaseMethodFault) <nil>,\\n Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"076c157f\"","TraceId":"910d62cb-2091-4e5f-ab36-d9e40f6b2349","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:452\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.
2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2020-05-05T12:15:07.529234683Z","caller":"volume/manager.go:433","msg":"failed to delete volume: \"38f57ff1-aa92-40dc-8e8a-371624fce197\", fault: \"(*types.LocalizedMethodFault)(0xc000b3fea0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n BaseMethodFault: (types.BaseMethodFault) <nil>,\\n Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"076c1580\"","TraceId":"22936af7-e73c-43c2-839c-823d26f844f8","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/volume.(*defaultManager).DeleteVolume\n\t/build/pkg/common/cns-lib/volume/manager.go:433\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/common.DeleteVolumeUtil\n\t/build/pkg/csi/service/common/vsphereutil.go:349\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:449\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/
rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2020-05-05T12:15:07.529710938Z","caller":"common/vsphereutil.go:351","msg":"failed to delete disk 38f57ff1-aa92-40dc-8e8a-371624fce197 with error failed to delete volume: \"38f57ff1-aa92-40dc-8e8a-371624fce197\", fault: \"(*types.LocalizedMethodFault)(0xc000b3fea0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n BaseMethodFault: (types.BaseMethodFault) <nil>,\\n Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"076c1580\"","TraceId":"22936af7-e73c-43c2-839c-823d26f844f8","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/common.DeleteVolumeUtil\n\t/build/pkg/csi/service/common/vsphereutil.go:351\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:449\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218
\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2020-05-05T12:15:07.529904148Z","caller":"vanilla/controller.go:452","msg":"failed to delete volume: \"38f57ff1-aa92-40dc-8e8a-371624fce197\". Error: failed to delete volume: \"38f57ff1-aa92-40dc-8e8a-371624fce197\", fault: \"(*types.LocalizedMethodFault)(0xc000b3fea0)({\\n DynamicData: (types.DynamicData) {\\n },\\n Fault: (types.CnsFault) {\\n BaseMethodFault: (types.BaseMethodFault) <nil>,\\n Reason: (string) (len=63) \\\"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n },\\n LocalizedMessage: (string) (len=79) \\\"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\\\n\\\"\\n})\\n\", opID: \"076c1580\"","TraceId":"22936af7-e73c-43c2-839c-823d26f844f8","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).DeleteVolume\n\t/build/pkg/csi/service/vanilla/controller.go:452\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5164\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).deleteVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:183\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/serialvolume/serial_volume_locker.go:92\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1
.2.1/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.1/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5166\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}

vshpere-syncer logs

{"level":"error","time":"2020-05-05T12:15:07.274001589Z","caller":"syncer/util.go:43","msg":"Error getting Persistent Volume Claim for volume www with err: persistentvolumeclaim \"www-nginx-topology-0\" not found","TraceId":"5afee8cc-c778-40ab-84fb-6701a1969341","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/syncer.IsValidVolume\n\t/build/pkg/syncer/util.go:43\nsigs.k8s.io/vsphere-csi-driver/pkg/syncer.csiUpdatePod\n\t/build/pkg/syncer/metadatasyncer.go:772\nsigs.k8s.io/vsphere-csi-driver/pkg/syncer.updatePodMetadata\n\t/build/pkg/syncer/metadatasyncer.go:511\nsigs.k8s.io/vsphere-csi-driver/pkg/syncer.podDeleted\n\t/build/pkg/syncer/metadatasyncer.go:503\nsigs.k8s.io/vsphere-csi-driver/pkg/syncer.InitMetadataSyncer.func7\n\t/build/pkg/syncer/metadatasyncer.go:191\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/controller.go:209\nk8s.io/client-go/tools/cache.(*processorListener).run.func1.1\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/shared_informer.go:556\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/wait/wait.go:265\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/shared_informer.go:548\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/wait/wait.go:88\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+
incompatible/tools/cache/shared_informer.go:546\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/wait/wait.go:71"}

csi-provisioner logs

E0505 12:15:06.866375       1 controller.go:1334] delete "pvc-70aa8526-bca6-4f6d-835e-1073eaddb01f": volume deletion failed: rpc error: code = Internal desc = failed to delete volume: "8290b052-db75-4e06-a181-5ebcf4bdea4c". Error: failed to delete volume: "8290b052-db75-4e06-a181-5ebcf4bdea4c", fault: "(*types.LocalizedMethodFault)(0xc000b3e2c0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.CnsFault) {\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\n  Reason: (string) (len=63) \"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n },\n LocalizedMessage: (string) (len=79) \"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n})\n", opID: "076c157f"
E0505 12:15:07.530972       1 controller.go:1334] delete "pvc-f2ed636e-3190-4368-8c10-d7256b5700aa": volume deletion failed: rpc error: code = Internal desc = failed to delete volume: "38f57ff1-aa92-40dc-8e8a-371624fce197". Error: failed to delete volume: "38f57ff1-aa92-40dc-8e8a-371624fce197", fault: "(*types.LocalizedMethodFault)(0xc000b3fea0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.CnsFault) {\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\n  Reason: (string) (len=63) \"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n },\n LocalizedMessage: (string) (len=79) \"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n})\n", opID: "076c1580"
W0505 12:15:07.531196       1 controller.go:936] Retrying syncing volume "pvc-f2ed636e-3190-4368-8c10-d7256b5700aa", failure 0
E0505 12:15:07.531316       1 controller.go:954] error syncing volume "pvc-f2ed636e-3190-4368-8c10-d7256b5700aa": rpc error: code = Internal desc = failed to delete volume: "38f57ff1-aa92-40dc-8e8a-371624fce197". Error: failed to delete volume: "38f57ff1-aa92-40dc-8e8a-371624fce197", fault: "(*types.LocalizedMethodFault)(0xc000b3fea0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.CnsFault) {\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\n  Reason: (string) (len=63) \"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n },\n LocalizedMessage: (string) (len=79) \"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n})\n", opID: "076c1580"
I0505 12:15:07.531433       1 event.go:255] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-f2ed636e-3190-4368-8c10-d7256b5700aa", UID:"73287ac1-5db7-4551-a3a4-37fdcb6498b1", APIVersion:"v1", ResourceVersion:"61687007", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = Internal desc = failed to delete volume: "38f57ff1-aa92-40dc-8e8a-371624fce197". Error: failed to delete volume: "38f57ff1-aa92-40dc-8e8a-371624fce197", fault: "(*types.LocalizedMethodFault)(0xc000b3fea0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.CnsFault) {\n  BaseMethodFault: (types.BaseMethodFault) <nil>,\n  Reason: (string) (len=63) \"CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n },\n LocalizedMessage: (string) (len=79) \"CnsFault error: CNS: Failed to delete disk:Fault cause: vim.fault.InvalidState\\n\"\n})\n", opID: "076c1580"

Environment:

• csi-vsphere version: 2.0.0
• vsphere-cloud-controller-manager version: 1.1.0
• Kubernetes version: 1.14.6
• vSphere version: 6.7u3
• OS (e.g. from /etc/os-release): RancherOS v1.5.1
• Kernel (e.g. uname -a): 4.14.85-rancher
• Install tools: rke
• Others: I used this manifests to deploy the driver https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/master/manifests/v2.0.0/vsphere-67u3/vanilla