kubernetes: Failing tests: [sig-storage] CSI Volumes [Driver: csi-hostpath] *

Which jobs are failing:

gce-master-scale-correctness

Which test(s) are failing:

Around ~50 tests with prefix "Kubernetes e2e suite: [sig-storage] CSI Volumes [Driver: csi-hostpath] " Example tests:

Kubernetes e2e suite: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume
Kubernetes e2e suite: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount
Kubernetes e2e suite: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it 
Kubernetes e2e suite: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly] 
Kubernetes e2e suite: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]
...

Since when has it been failing:

05-27-2021

Testgrid link:

https://testgrid.k8s.io/sig-release-master-informing#gce-master-scale-correctness

Reason for failure:

I’ve checked logs for pod “pod-bd3fa2c8-2279-446b-98e5-a268aa308126” in test “Kubernetes e2e suite: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it”. https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/1398972946927587328

kubelet logs:

./kubelet.log:523:I0530 12:24:09.838798    2113 kubelet.go:1938] "SyncLoop ADD" source="api" pods=[volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./kubelet.log:527:I0530 12:24:10.052174    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:528:E0530 12:24:10.052298    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:10.55116941 +0000 UTC m=+858.587378827 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:529:I0530 12:24:10.052358    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcvv\" (UniqueName: \"kubernetes.io/projected/49cbe128-3778-4623-b37a-f3980e709f5b-kube-api-access-fxcvv\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:530:I0530 12:24:10.153707    2113 reconciler.go:269] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcvv\" (UniqueName: \"kubernetes.io/projected/49cbe128-3778-4623-b37a-f3980e709f5b-kube-api-access-fxcvv\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:531:I0530 12:24:10.198699    2113 operation_generator.go:698] MountVolume.SetUp succeeded for volume "kube-api-access-fxcvv" (UniqueName: "kubernetes.io/projected/49cbe128-3778-4623-b37a-f3980e709f5b-kube-api-access-fxcvv") pod "pod-bd3fa2c8-2279-446b-98e5-a268aa308126" (UID: "49cbe128-3778-4623-b37a-f3980e709f5b")
./kubelet.log:532:I0530 12:24:10.556368    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:533:E0530 12:24:10.556764    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:11.556689246 +0000 UTC m=+859.592898644 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:534:I0530 12:24:11.574846    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:535:E0530 12:24:11.575005    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:13.574938703 +0000 UTC m=+861.611148093 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:536:I0530 12:24:13.612979    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:537:E0530 12:24:13.651348    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:17.651267361 +0000 UTC m=+865.687476745 (durationBeforeRetry 4s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:539:I0530 12:24:17.977866    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:540:E0530 12:24:17.978407    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:25.978353055 +0000 UTC m=+874.014562445 (durationBeforeRetry 8s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:637:I0530 13:38:17.842036    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:640:E0530 13:38:21.236943    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 13:38:34.739730301 +0000 UTC m=+5322.775939691 (durationBeforeRetry 16s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./kubelet.log:667:E0530 13:38:22.336502    2113 kubelet.go:1707] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-fxcvv]: timed out waiting for the condition" pod="volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126"
./kubelet.log:668:E0530 13:38:22.337195    2113 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-fxcvv]: timed out waiting for the condition" pod="volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126" podUID=49cbe128-3778-4623-b37a-f3980e709f5b
./kubelet.log:1091:I0530 13:38:41.638765    6839 kubelet.go:1938] "SyncLoop ADD" source="api" pods=[kube-system/metadata-proxy-v0.1-wxc2j kube-system/kube-proxy-gce-scale-cluster-minion-group-1-clw2 volume-expand-235-7191/csi-hostpathplugin-0 volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./kubelet.log:1319:I0530 13:38:50.594330    6839 kubelet.go:1954] "SyncLoop DELETE" source="api" pods=[volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./kubelet.log:1320:I0530 13:38:50.596477    6839 kubelet.go:1948] "SyncLoop REMOVE" source="api" pods=[volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./kubelet.log:1322:I0530 13:38:50.597705    6839 kubelet.go:2157] "Failed to delete pod" pod="volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126" err="pod not found"

systemd logs:

./systemd.log:2452:May 30 12:24:09.839101 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:09.838798    2113 kubelet.go:1938] "SyncLoop ADD" source="api" pods=[volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./systemd.log:2456:May 30 12:24:10.052465 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:10.052174    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2457:May 30 12:24:10.052465 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 12:24:10.052298    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:10.55116941 +0000 UTC m=+858.587378827 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2458:May 30 12:24:10.052465 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:10.052358    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcvv\" (UniqueName: \"kubernetes.io/projected/49cbe128-3778-4623-b37a-f3980e709f5b-kube-api-access-fxcvv\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2459:May 30 12:24:10.154040 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:10.153707    2113 reconciler.go:269] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcvv\" (UniqueName: \"kubernetes.io/projected/49cbe128-3778-4623-b37a-f3980e709f5b-kube-api-access-fxcvv\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2460:May 30 12:24:10.198744 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:10.198699    2113 operation_generator.go:698] MountVolume.SetUp succeeded for volume "kube-api-access-fxcvv" (UniqueName: "kubernetes.io/projected/49cbe128-3778-4623-b37a-f3980e709f5b-kube-api-access-fxcvv") pod "pod-bd3fa2c8-2279-446b-98e5-a268aa308126" (UID: "49cbe128-3778-4623-b37a-f3980e709f5b")
./systemd.log:2461:May 30 12:24:10.556483 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:10.556368    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2462:May 30 12:24:10.556796 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 12:24:10.556764    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:11.556689246 +0000 UTC m=+859.592898644 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2463:May 30 12:24:11.574902 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:11.574846    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2464:May 30 12:24:11.575558 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 12:24:11.575005    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:13.574938703 +0000 UTC m=+861.611148093 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2474:May 30 12:24:13.613019 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:13.612979    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2475:May 30 12:24:13.662561 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 12:24:13.651348    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:17.651267361 +0000 UTC m=+865.687476745 (durationBeforeRetry 4s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2477:May 30 12:24:17.977899 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 12:24:17.977866    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2478:May 30 12:24:18.059339 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 12:24:17.978407    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 12:24:25.978353055 +0000 UTC m=+874.014562445 (durationBeforeRetry 8s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2777:May 30 13:38:20.689877 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: I0530 13:38:17.842036    2113 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2780:May 30 13:38:21.236983 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 13:38:21.236943    2113 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2 podName: nodeName:}" failed. No retries permitted until 2021-05-30 13:38:34.739730301 +0000 UTC m=+5322.775939691 (durationBeforeRetry 16s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\" (UniqueName: \"kubernetes.io/csi/csi-hostpath-volume-expand-235^eeb4e7d3-c141-11eb-9364-4a26c59411e2\") pod \"pod-bd3fa2c8-2279-446b-98e5-a268aa308126\" (UID: \"49cbe128-3778-4623-b37a-f3980e709f5b\") "
./systemd.log:2807:May 30 13:38:22.337063 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 13:38:22.336502    2113 kubelet.go:1707] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-fxcvv]: timed out waiting for the condition" pod="volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126"
./systemd.log:2808:May 30 13:38:22.337235 gce-scale-cluster-minion-group-1-clw2 kubelet[2113]: E0530 13:38:22.337195    2113 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[volume1], unattached volumes=[volume1 kube-api-access-fxcvv]: timed out waiting for the condition" pod="volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126" podUID=49cbe128-3778-4623-b37a-f3980e709f5b
./systemd.log:3341:May 30 13:38:41.638822 gce-scale-cluster-minion-group-1-clw2 kubelet[6839]: I0530 13:38:41.638765    6839 kubelet.go:1938] "SyncLoop ADD" source="api" pods=[kube-system/metadata-proxy-v0.1-wxc2j kube-system/kube-proxy-gce-scale-cluster-minion-group-1-clw2 volume-expand-235-7191/csi-hostpathplugin-0 volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./systemd.log:3696:May 30 13:38:50.594374 gce-scale-cluster-minion-group-1-clw2 kubelet[6839]: I0530 13:38:50.594330    6839 kubelet.go:1954] "SyncLoop DELETE" source="api" pods=[volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./systemd.log:3697:May 30 13:38:50.596510 gce-scale-cluster-minion-group-1-clw2 kubelet[6839]: I0530 13:38:50.596477    6839 kubelet.go:1948] "SyncLoop REMOVE" source="api" pods=[volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126]
./systemd.log:3699:May 30 13:38:50.597755 gce-scale-cluster-minion-group-1-clw2 kubelet[6839]: I0530 13:38:50.597705    6839 kubelet.go:2157] "Failed to delete pod" pod="volume-expand-235/pod-bd3fa2c8-2279-446b-98e5-a268aa308126" err="pod not found"

Looks like reason why it fails is Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9096038e-4455-4c1c-a596-5b10a03ffb47\"

Anything else we need to know: