kubernetes: Failing tests: [sig-storage] Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV

Which jobs are failing:

pull-kubernetes-unit

Which test(s) are failing:

Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV

Since when has it been failing:

Testgrid link:

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/102162/pull-kubernetes-unit/1399622097709305856

Reason for failure:

race detected during execution of test

Anything else we need to know:

k8s.io/kubernetes/pkg/controller/volume/attachdetach: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV expand_less0s=== RUN   Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV W0601 07:16:12.365733   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.365889   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.365967   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366036   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366107   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366349   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366419   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. E0601 07:16:12.370442   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin I0601 07:16:12.373449   58041 shared_informer.go:240] Waiting for caches to sync for attach detach E0601 07:16:12.376683   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin E0601 07:16:12.378222   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin E0601 07:16:12.379574   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin I0601 07:16:12.380788   58041 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1"  I0601 07:16:12.381084   58041 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode"  ================== WARNING: DATA RACE Write at 0x00c0005a4138 by goroutine 181:   k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fixture.go:521 +0x68   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fake.go:145 +0x32e   k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:85 +0x246   k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher.go:117 +0xb13   k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:342 +0x330   k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/types/types.go:79 +0x195   k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159  Previous read at 0x00c0005a4138 by goroutine 49:   k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fixture.go:521 +0x68   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fake.go:145 +0x32e   k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:58 +0x273   k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/listwatch.go:106 +0x94   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager/pager.go:40 +0x94   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager/pager.go:91 +0x1f4   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7  Goroutine 181 (running) created at:   k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x572   k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x5e5   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x199   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f  Goroutine 49 (finished) created at:   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:360 +0x304   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x53   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x2b5   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:218 +0x54   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d ================== | k8s.io/kubernetes/pkg/controller/volume/attachdetach: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV expand_less | 0s | === RUN   Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV W0601 07:16:12.365733   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.365889   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.365967   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366036   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366107   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366349   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366419   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. E0601 07:16:12.370442   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin I0601 07:16:12.373449   58041 shared_informer.go:240] Waiting for caches to sync for attach detach E0601 07:16:12.376683   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin E0601 07:16:12.378222   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin E0601 07:16:12.379574   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin I0601 07:16:12.380788   58041 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1"  I0601 07:16:12.381084   58041 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode"  ================== WARNING: DATA RACE Write at 0x00c0005a4138 by goroutine 181:   k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fixture.go:521 +0x68   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fake.go:145 +0x32e   k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:85 +0x246   k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher.go:117 +0xb13   k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:342 +0x330   k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/types/types.go:79 +0x195   k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159  Previous read at 0x00c0005a4138 by goroutine 49:   k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fixture.go:521 +0x68   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fake.go:145 +0x32e   k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:58 +0x273   k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/listwatch.go:106 +0x94   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager/pager.go:40 +0x94   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager/pager.go:91 +0x1f4   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7  Goroutine 181 (running) created at:   k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x572   k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x5e5   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x199   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f  Goroutine 49 (finished) created at:   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:360 +0x304   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x53   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x2b5   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:218 +0x54   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d ==================
k8s.io/kubernetes/pkg/controller/volume/attachdetach: Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV expand_less | 0s
=== RUN   Test_ADC_VolumeAttachmentRecovery/Deleted_Pod_with_migrated_PV W0601 07:16:12.365733   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.365889   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.365967   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366036   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366107   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366349   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. W0601 07:16:12.366419   58041 mutation_detector.go:53] Mutation detector is enabled, this will result in memory leakage. E0601 07:16:12.370442   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode" : no volume plugin matched name: kubernetes.io/testPlugin I0601 07:16:12.373449   58041 shared_informer.go:240] Waiting for caches to sync for attach detach E0601 07:16:12.376683   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-2" : no volume plugin matched name: kubernetes.io/testPlugin E0601 07:16:12.378222   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-3" : no volume plugin matched name: kubernetes.io/testPlugin E0601 07:16:12.379574   58041 reconciler.go:228] attacherDetacher.DetachVolume failed to start for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : DetachVolume.FindAttachablePluginByName failed for volume "nil" (UniqueName: "kubernetes.io/testPlugin/lostVolumeName") on node "mynode-4" : no volume plugin matched name: kubernetes.io/testPlugin I0601 07:16:12.380788   58041 reconciler.go:219] attacherDetacher.DetachVolume started for volume "nil" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/vol1") on node "mynode-1"  I0601 07:16:12.381084   58041 reconciler.go:295] attacherDetacher.AttachVolume started for volume "pd.csi.storage.gke.io-pdName" (UniqueName: "kubernetes.io/csi/pd.csi.storage.gke.io^projects/UNSPECIFIED/zones/UNSPECIFIED/disks/pdName") from node "mynode"  ================== WARNING: DATA RACE Write at 0x00c0005a4138 by goroutine 181:   k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func7()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing/testvolumespec.go:185 +0x207   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fixture.go:521 +0x68   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fake.go:145 +0x32e   k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).Create()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:85 +0x246   k8s.io/kubernetes/pkg/volume/csi.(*csiAttacher).Attach()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher.go:117 +0xb13   k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateAttachVolumeFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:342 +0x330   k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/types/types.go:79 +0x195   k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:183 +0x159  Previous read at 0x00c0005a4138 by goroutine 49:   k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing.CreateTestClient.func6()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing/testvolumespec.go:179 +0x104   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fixture.go:521 +0x68   k8s.io/kubernetes/vendor/k8s.io/client-go/testing.(*Fake).Invokes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/testing/fake.go:145 +0x32e   k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake.(*FakeVolumeAttachments).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes/typed/storage/v1/fake/fake_volumeattachment.go:58 +0x273   k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1.NewFilteredVolumeAttachmentInformer.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/informers/storage/v1/volumeattachment.go:64 +0x23c   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/listwatch.go:106 +0x94   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:277 +0xf4   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager/pager.go:40 +0x94   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.(*ListPager).List()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager/pager.go:91 +0x1f4   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:302 +0x2b7  Goroutine 181 (running) created at:   k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:178 +0x572   k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationExecutor).AttachVolume()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:675 +0x348   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).attachDesiredVolumes()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:293 +0x5e5   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconcile()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:233 +0x199   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).reconciliationLoopFunc.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:110 +0x5e   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x114   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x89   k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.(*reconciler).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:101 +0x2f  Goroutine 49 (finished) created at:   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:268 +0x36f   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:360 +0x304   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x53   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xba   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x2b5   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:218 +0x54   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:56 +0x45   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()       /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x6d ==================