kubernetes: [Failing Test] bazel-test-master (periodic-kubernetes-bazel-test-master)

Which jobs are failing:

bazel-test-master (periodic-kubernetes-bazel-test-master)

Which test(s) are failing:

//cmd/kube-scheduler/app/go_default_test:run_1_of_3
//cmd/kube-scheduler/app/go_default_test:run_3_of_3
//pkg/util/iptables/go_default_test:run_2_of_3
//cmd/kube-scheduler/app/go_default_test:run_2_of_3
...

Since when has it been failing:

09-04-20 01:03 CDT

Testgrid link:

https://testgrid.k8s.io/sig-release-master-blocking#bazel-test-master

Reason for failure:

Errors differ between each of the failing tests. Looks like failure started between 26d0f8460 and fe8aa6eba.

Anything else we need to know:

/cc @kubernetes/ci-signal /priority critical-urgent /milestone v1.20

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (20 by maintainers)

Most upvoted comments

Fixes identified:

cmd/kube-scheduler/app: TestSetup - @liggitt, https://github.com/kubernetes/kubernetes/pull/94529
pkg/probe/http: TestHTTPProbeProxy - @liggitt, https://github.com/kubernetes/kubernetes/pull/94529
k8s.io/apiserver/pkg/server/filters: TestClientReceivedGOAWAY, TestGOAWAYConcurrency - @knight42 , https://github.com/kubernetes/kubernetes/pull/94530, https://github.com/kubernetes/kubernetes/pull/94529
pkg/util/iptables - TestRestoreAllGrabOldLock, @knight42, https://github.com/kubernetes/kubernetes/pull/94534
k8s.io/apiserver/pkg/server/options, @liggitt, https://github.com/kubernetes/kubernetes/pull/94544
pkg/kubelet/cm/devicemanager - @liggitt, https://github.com/kubernetes/kubernetes/pull/94539
pkg/client/tests - @liggitt, https://github.com/kubernetes/kubernetes/pull/94543
pkg/kubelet/cm/cpumanager/state, @liggitt, https://github.com/kubernetes/kubernetes/pull/94541
cmd/kubeadm/app/phases/upgrade - @liggitt, https://github.com/kubernetes/kubernetes/pull/94535
bug - k8s.io/legacy-cloud-providers/azure/cache - @knight42, https://github.com/kubernetes/kubernetes/pull/94537
k8s.io/client-go/tools/watch, @knight42, https://github.com/kubernetes/kubernetes/pull/94554
pkg/kubelet/kuberuntime, @knight42, https://github.com/kubernetes/kubernetes/pull/94549
k8s.io/client-go/util/workqueue, @knight42, #94545, https://github.com/kubernetes/kubernetes/pull/94556
k8s.io/client-go/tools/events
- TestEventSeriesf timeout - https://github.com/kubernetes/kubernetes/issues/94569
pkg/util/iptables
- TestRestoreAllWaitOldIptablesRestore, @knight42, https://github.com/kubernetes/kubernetes/pull/94553
k8s.io/client-go/tools/portforward - @knight42, https://github.com/kubernetes/kubernetes/pull/94594
k8s.io/apiserver/pkg/storage/cacher - https://github.com/kubernetes/kubernetes/issues/95394, @wojtek-t

Unresolved flakes (grouped by sig):

sig-storage
- pkg/kubelet/volumemanager/reconciler
  - Test_UncertainVolumeMountState/failed_operation_should_result_in_not-mounted_volume - https://github.com/kubernetes/kubernetes/issues/94567
  - Test_Run_Positive_VolumeMountControllerAttachEnabledRace data race (https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/periodic-kubernetes-bazel-test-master/1302035081157677056) - https://github.com/kubernetes/kubernetes/issues/94568
- pkg/controller/volume/attachdetach/reconciler
  - Test_Run_OneVolumeAttachAndDetachMultipleNodesWithReadWriteMany, https://github.com/kubernetes/kubernetes/issues/93902
sig-node
- pkg/kubelet/cm/devicemanager (after 2020-09-04)
  - TestDevicePluginReRegistrationProbeMode #94547
- pkg/kubelet/config
  - TestWatchFileChanged “file_linux_test.go:396: expected empty update, got types.PodUpdate{Pods:[]*v1.Pod{(*v1.Pod)(0xc00010b400)}, Op:0, Source:“file”}” - https://github.com/kubernetes/kubernetes/issues/99845
sig-api-machinery
- ~~k8s.io/apiserver/pkg/admission/plugin/webhook/validating~~ - cannot reproduce
- k8s.io/apiserver/pkg/admission/plugin/webhook/mutating
sig-cloud-provider
- k8s.io/legacy-cloud-providers/vsphere
  - TestSecretUpdated/Secret_with_a_different_data “vsphere_test.go:1252: Unexpected log error messages for secrets”

Query excluding unresolved/unmerged flakes (investigate anything flaking after 2020-09-05): https://testgrid.k8s.io/sig-release-master-blocking#bazel-test-master&width=20&sort-by-failures=&exclude-filter-by-regex=Overall|k8s.io/client-go/util/workqueue|k8s.io/client-go/tools/events|pkg/kubelet/volumemanager/reconciler|k8s.io/legacy-cloud-providers/vsphere|pkg/kubelet/config|pkg/controller/volume/attachdetach/reconciler|pkg/util/iptables|pkg/kubelet/cm/devicemanager|k8s.io/client-go/tools/portforward|k8s.io/apiserver/pkg/admission/plugin/webhook/validating|k8s.io/apiserver/pkg/storage/cacher|k8s.io/apiserver/pkg/admission/plugin/webhook/mutating

liggitt on Mar 5, 2021

bazel job is removed in favor of the pull-kubernetes-unit job; I opened individual issues for the remaining flakes I saw referenced from here and added to the 1.21 milestone

/close

liggitt on Mar 5, 2021

@rphillips can you have a look at the sig-node failure in https://github.com/kubernetes/kubernetes/issues/94528#issuecomment-687208028 ?

deads2k on Oct 29, 2020

working on pkg/util/iptables

knight42 on Sep 4, 2020

Almost certainly https://github.com/kubernetes/kubernetes/pull/93605, which means we have remaining unit test flakes

liggitt on Sep 4, 2020