cilium: CI: Travis: ENISuite.TestNodeManagerManyNodes fails
CI failure
level=info msg="Synchronized ENI information" numInstances=100 numSecurityGroups=2 numSubnets=4 numVPCs=1 subsys=eni
----------------------------------------------------------------------
FAIL: node_manager_test.go:563: ENISuite.TestNodeManagerManyNodes
node_manager_test.go:602:
c.Errorf("Node %s allocation mismatch. expected: %d allocated: %d", s.name, minAllocate, node.Stats().AvailableIPs)
... Error: Node node53 allocation mismatch. expected: 10 allocated: 18
node_manager_test.go:602:
c.Errorf("Node %s allocation mismatch. expected: %d allocated: %d", s.name, minAllocate, node.Stats().AvailableIPs)
... Error: Node node59 allocation mismatch. expected: 10 allocated: 18
node_manager_test.go:617:
c.Assert(metricsapi.AllocatedIPs("available"), check.Equals, numNodes*minAllocate)
... obtained int = 1016
... expected int = 1000
...
OOPS: 17 passed, 1 FAILED
--- FAIL: Test (2.60s)
FAIL
coverage: 4.3% of statements in ./...
FAIL github.com/cilium/cilium/pkg/aws/eni 2.684s
FAIL
Makefile:204: recipe for target 'unit-tests' failed
make: *** [unit-tests] Error 1
The command "./.travis/build.sh" exited with 2.
Done. Your build exited with 1.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 23 (22 by maintainers)
Commits related to this issue
- eni: Fix node manager test The test was failing due resulting values being greater then expected, however it is evident that `minAllocate` is a minimum value and should be treated as such. Fixes: #1... — committed to cilium/cilium by errordeveloper 4 years ago
- eni: Fix node manager test The test was failing due resulting values being greater then expected, however it is evident that `minAllocate` is a minimum value and should be treated as such. Fixes: #1... — committed to cilium/cilium by errordeveloper 4 years ago
- eni: Fix node manager test [ upstream commit f7198934f299d535586b0bf255bdbdfb480291e8 ] The test was failing due resulting values being greater then expected, however it is evident that `minAllocate... — committed to cilium/cilium by errordeveloper 4 years ago
- eni: Fix node manager test [ upstream commit f7198934f299d535586b0bf255bdbdfb480291e8 ] The test was failing due resulting values being greater then expected, however it is evident that `minAllocate... — committed to cilium/cilium by errordeveloper 4 years ago
- ipam: Fix flaky test TestNodeManagerManyNodes TestNodeManagerManyNodes had been flaky before ported from aws/eni, and was eventually disabled in aws tests: https://github.com/cilium/cilium/issues/115... — committed to ctripcloud/cilium by jaffcheng a year ago
- ipam: Fix race in NodeManager.Resync TestNodeManagerManyNodes had been flaky before ported from aws/eni, and was eventually disabled in aws tests: https://github.com/cilium/cilium/issues/11560 One o... — committed to ctripcloud/cilium by jaffcheng a year ago
- ipam: Fix race in NodeManager.Resync TestNodeManagerManyNodes had been flaky before ported from aws/eni, and was eventually disabled in aws tests: https://github.com/cilium/cilium/issues/11560 One o... — committed to cilium/cilium by jaffcheng a year ago
Hmm, there’s definitely some raciness going on. I ran the tests with
-raceenabled, and it found many, many data races. A decent amount of data races are in theTestNodeManagerManyNodesso this might explain this Travis failure. I’m working on a PR to fix some data races under justTestNodeManagerManyNodes, as there are many data races around this code.Removing @christarazi 's stale assignment here. If you plan on working on this, feel free to reassign it back to yourself.
@gandro I restarted the Travis CI build where it happened. I’ll keep it next time and ping here.