kubernetes: Failing Test : persistent-volume-upgrade [sig-storage]
Failing Job
sig-release-master-upgrade#gce-new-master-upgrade-master
Gubernator Logs
Error
Expected error:
<*errors.errorString | 0xc420a4b980>: {
s: "failed to get logs from pvc-tester-j4c4l for write-pod: the server does not allow this method on the requested resource (get pods pvc-tester-j4c4l)",
}
failed to get logs from pvc-tester-j4c4l for write-pod: the server does not allow this method on the requested resource (get pods pvc-tester-j4c4l)
not to have occurred
k8s.io/kubernetes/test/e2e/framework.(*Framework).testContainerOutputMatcher(0xc42097b340, 0x443f662, 0xf, 0xc42026a000, 0x0, 0xc420090e98, 0x1, 0x1, 0x45d0920)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2350 +0x216
k8s.io/kubernetes/test/e2e/framework.(*Framework).TestContainerOutput(0xc42097b340, 0x443f662, 0xf, 0xc42026a000, 0x0, 0xc420090e98, 0x1, 0x1)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:475 +0x8a
k8s.io/kubernetes/test/e2e/upgrades/storage.(*PersistentVolumeUpgradeTest).testPod(0x6c17120, 0xc42097b340, 0x449c1c2, 0x20)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/storage/persistent_volumes.go:100 +0x10e
k8s.io/kubernetes/test/e2e/upgrades/storage.(*PersistentVolumeUpgradeTest).Test(0x6c17120, 0xc42097b340, 0xc42151eae0, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/storage/persistent_volumes.go:82 +0x8f
k8s.io/kubernetes/test/e2e/lifecycle.(*chaosMonkeyAdapter).Test(0xc4217d7640, 0xc421382b80)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:447 +0x1da
k8s.io/kubernetes/test/e2e/lifecycle.(*chaosMonkeyAdapter).Test-fm(0xc421382b80)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:516 +0x34
k8s.io/kubernetes/test/e2e/chaosmonkey.(*chaosmonkey).Do.func1(0xc421382b80, 0xc42157c8a0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76
created by k8s.io/kubernetes/test/e2e/chaosmonkey.(*chaosmonkey).Do
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:86 +0xa8
Triage
This test is a persistent fail in this one test bucket only, and aside from one flake it appears to consistently pass everywhere else in sig-release-master-upgrade and in sig-release-1.12-all.
/kind bug /priority failing-test /sig storage @kubernetes/sig-storage-bugs /assign @msau42 /cc @saad-ali @childsb /milestone v1.12
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 61 (61 by maintainers)
I don’t believe the kubelet has the ability to update its taints anymore. Assuming this is the case, this seems like a version skew problem between master components. I.E. the 1.11 controller manager isn’t compatible with the 1.13 scheduler because it doesn’t add taints that the scheduler requires. This means that if the scheduler is upgraded before the controller manager (or before it has the chance to perform updates), we can run into issues.
I think making the scheduler respect conditions in 1.13, and thus making it compatible with the 1.11 controller manager, seems like the best remedy to this. Regardless of this issue, user pods would need to include both the critical annotation as well as the toleration in order to maintain the same behavior across releases that transition from conditions to taints.
/assign
I’m going to contribute in my daytime 😃
Don’t think #68492 will help. It looks specific to gpu labeling
It is interesting that in some places this test is always failing, some it is always succeeding, and even in one it flakes but with a high success rate: https://k8s-testgrid.appspot.com/sig-release-master-upgrade#gce-new-master-upgrade-cluster