kubernetes: AfterSuite failures in gce-(large/scale)-correctness tests

I’m seeing the jobs go through fine, but failing during AfterSuite action. E.g:

From the build-log of the first one, I’m seeing the following:

I1118 06:40:42.351] [91m[1m[AfterSuite] AfterSuite [0m
I1118 06:40:42.351] [37m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:254[0m
I1118 06:40:42.351] 
I1118 06:40:42.351]   [91m[1mTest Panicked[0m
I1118 06:40:42.351]   [91mclose of closed channel[0m
I1118 06:40:42.351]   /usr/local/go/src/runtime/panic.go:491
I1118 06:40:42.352] 
I1118 06:40:42.352]   [91mFull Stack Trace[0m
I1118 06:40:42.352]   	/usr/local/go/src/runtime/panic.go:491 +0x283
I1118 06:40:42.352]   k8s.io/kubernetes/test/e2e/framework.(*containerResourceGatherer).StopAndSummarize(0xc420d06690, 0xc426e97290, 0x3, 0x3, 0xc4210c0930, 0x0, 0x0, 0x0)
I1118 06:40:42.352]   	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:292 +0x47
I1118 06:40:42.352]   k8s.io/kubernetes/test/e2e/framework.(*Framework).AfterEach(0xc42118a7e0)
I1118 06:40:42.352]   	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:324 +0xbc1
...

The panic suggests that it was attempted to close a closed channel, more specifically it seems to be the stopCh of the resource-gatherer from the stack-trace:

https://github.com/kubernetes/kubernetes/blob/2aaab817de5c90340cf94913df166f389d8d6aca/test/e2e/framework/resource_usage_gatherer.go#L291-L293

cc @kubernetes/sig-scalability-misc @porridge

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 17 (17 by maintainers)

Most upvoted comments

I cannot say it with 100% certainty, but I think #56128 should have fixed this one too. The reason we fail in containerResourceGatherer.StopAndSummarize (which is invoked earlier by AfterEach) rather than framework.AllNodesReady is most likely that resource gathering is not enabled for most jobs, but is enabled for scalability jobs. /close

porridge on Nov 22, 2017