moby: Optimize slow bottleneck tests

The time the integration test suite takes to finish is getting out of control. We would like to ask your help in optimizing some of the longest test cases. This is another of those “non-pleasant” tasks of which, if we all do just one, we can get it all done extra fast.

If you would like to help out, please comment on this issue with the name of the test you would like to take care of; we’ll add your Github handle next to the test name so that others can tell which tests still need an owner.

UPDATE: optimizing a test consists of making sure the test code is reasonably efficient. However if the slowdown is a sign of not efficient code in Docker itself, then that would need to be addressed too.

I’m shamelessly stealing the following from @estesp’s gist https://gist.github.com/estesp/40a056a8262eafb62dd1:

Sampled from a 1/12/2016 run of standard Linux x64 engine janky run.

Total time of these 25 tests adds up to 663.8 seconds

These 25 make up 2.2% of total tests, but ~29% of total test runtime.

DockerDaemonSuite.TestDaemonNoSpaceleftOnDeviceError            107.143s DockerTrustSuite.TestTrustedPushWithFailingServer               95.405s
DockerSuite.TestRunSeccompDefaultProfile                        81.711s
DockerSuite.TestEventsImagePull                                 43.321s
DockerSuite.TestBuildEnvUsage2                                  32.791s
DockerSuite.TestDaemonStartWithDaemonCommand                    25.648s
DockerHubPullSuite.TestPullAllTagsFromCentralRegistry           24.504s
DockerSuite.TestEventsOOMDisableTrue                            20.288s
DockerHubPullSuite.TestPullFromCentralRegistryImplicitRefParts  19.389s
DockerSuite.TestBuildHistory                                    18.234s
DockerSuite.TestBuildUsersAndGroups                             16.550s
DockerSuite.TestRunUnshareProc                                  16.196s
DockerSuite.TestVolumeCliCreate                                 15.118s
DockerSuite.TestApiStatsNetworkStatsVersioning                  14.699s
DockerSuite.TestBuildDockerignoringWildDirs                     14.443s
DockerSuite.TestApiStatsNetworkStats                            13.200s
DockerSuite.TestBuildSixtySteps                                 12.663s
DockerHubPullSuite.TestPullNonExistingImage                     12.259s
DockerDaemonSuite.TestDaemonRestartWithContainerRunning         12.110s
DockerDaemonSuite.TestDaemonCorruptedLogDriverAddress           12.007s
DockerSuite.TestDaemonIPv6FixedCIDR                             11.564s
DockerSuite.TestInfoDiscoveryInvalidAdvertise                   11.504s
DockerSuite.TestDaemonIPv6FixedCIDRAndMac                       11.451s
DockerSuite.TestHelpTextVerify                                  11.039s
DockerSuite.TestEventsAttach                                    10.635s

Additional tests (suggested by @aaronlehmann)

DockerSuite.TestEventsContainerFailStartDie 40.018s
DockerSuite.TestRunSeccompAllowPrivCloneUserns  40.302s
DockerSuite.TestRunSeccompProfileDenyCloneUserns        45.066s
DockerSuite.TestRunSeccompUnconfinedCloneUserns 40.056s

And more (suggested by @wenchma)

DockerNetworkSuite.TestDockerNetworkInternalMode   11.268s

TODO:

DockerDaemonSuite.TestDaemonNoSpaceleftOnDeviceError (#19551 @aaronlehmann)
DockerTrustSuite.TestTrustedPushWithFailingServer (#23217 @clnperez)
DockerSuite.TestRunSeccompDefaultProfile (#19725 @WeiZhang555)
DockerSuite.TestEventsImagePull (test is now 2.273s)
DockerSuite.TestBuildEnvUsage2 (#19600 @aaronlehmann)
DockerSuite.TestDaemonStartWithDaemonCommand (#20476 @wenchma)
DockerHubPullSuite.TestPullAllTagsFromCentralRegistry
DockerSuite.TestEventsOOMDisableTrue (#19619 @WeiZhang555)
DockerHubPullSuite.TestPullFromCentralRegistryImplicitRefParts
DockerSuite.TestBuildHistory (#20880 @yongtang)
DockerSuite.TestBuildUsersAndGroups (#21056 @askb)
DockerSuite.TestRunUnshareProc (#20965 @yongtang)
DockerSuite.TestVolumeCliCreate (@askb)
DockerSuite.TestApiStatsNetworkStatsVersioning (#19800 @wenchma)
DockerSuite.TestBuildDockerignoringWildDirs (#20929 @yongtang)
DockerSuite.TestApiStatsNetworkStats (#19800 @wenchma)
DockerSuite.TestBuildSixtySteps
DockerHubPullSuite.TestPullNonExistingImage
DockerDaemonSuite.TestDaemonRestartWithContainerRunning (#19663 @mountkin)
DockerDaemonSuite.TestDaemonCorruptedLogDriverAddress (#19708 @wenchma)
DockerSuite.TestDaemonIPv6FixedCIDR (#19658 @wenchma)
DockerSuite.TestInfoDiscoveryInvalidAdvertise
DockerSuite.TestDaemonIPv6FixedCIDRAndMac (#19658 @wenchma)
DockerSuite.TestHelpTextVerify (@sainath14)
DockerSuite.TestEventsAttach (#19703 @mountkin)
DockerSuite.TestEventsContainerFailStartDie (#19713 @WeiZhang555)
DockerSuite.TestRunSeccompAllowPrivCloneUserns
DockerSuite.TestRunSeccompProfileDenyCloneUserns
DockerSuite.TestRunSeccompUnconfinedCloneUserns
DockerNetworkSuite.TestDockerNetworkInternalMode (#20903 @wenchma )

About this issue

Original URL
State: open
Created 8 years ago
Comments: 60 (55 by maintainers)

Commits related to this issue

Slight TestBuildEnvUsage2 speedup TestBuildEnvUsage2 was flagged in https://github.com/docker/docker/issues/19425 as one of the slowest integration tests. It's slow because it has some comprehensive ... — committed to aaronlehmann/docker by aaronlehmann 8 years ago
Optimize slow integration test Optimize `TestEventsOOMDisableTrue` performance It's part of #19425 issue. Signed-off-by: Zhang Wei <zhangwei555@huawei.com> — committed to WeiZhang555/moby by WeiZhang555 8 years ago
Slight TestBuildEnvUsage2 speedup TestBuildEnvUsage2 was flagged in https://github.com/docker/docker/issues/19425 as one of the slowest integration tests. It's slow because it has some comprehensive ... — committed to aditirajagopal/docker by aaronlehmann 8 years ago
Optimize slow integration test Optimize `TestEventsOOMDisableTrue` performance It's part of #19425 issue. Signed-off-by: Zhang Wei <zhangwei555@huawei.com> — committed to aditirajagopal/docker by WeiZhang555 8 years ago
Optimize slow bottleneck tests of TestDaemonStartWithDaemonCommand * Remvoe integration test of TestDaemonStartWithDaemonCommand * Rewrite as unit test Related issue #19425 Signed-off-by: Wen Cheng... — committed to wenchma/docker by deleted user 8 years ago
Optimized integration test case DockerSuite.TestBuildUsersAndGroups for #19425 Removed unnecessary RUN statements and combined some of the RUN statement into a single line. The runtime performance i... — committed to moby/moby by askb 8 years ago
Merge pull request #21056 from askb/optimize-TestBuildUsersAndGroups Optimized integration test case DockerSuite.TestBuildUsersAndGroups fixes #19425 — committed to moby/moby by calavera 8 years ago

Most upvoted comments

Ok, I wrote up a script and put it into a docker image

docker run --rm dnephin/docker-testsuite-report 31507

Elapsed: 1:07:31
Elapsed integration suite: 0:50:21
Sum test times: 0:32:09
Time to first integration test: 0:10:02 
Time after last integration test: 0:07:08
Test count: 1429

Slowest tests:
   PASS: docker_cli_push_test.go:666: DockerRegistryAuthTokenSuite.TestPushMisconfiguredTokenServiceResponseError   50.144s
   PASS: docker_cli_run_test.go:3105: DockerSuite.TestRunUnshareProc    43.859s
   PASS: docker_cli_pull_test.go:201: DockerHubPullSuite.TestPullAllTagsFromCentralRegistry 33.525s
   PASS: docker_cli_run_unix_test.go:1105: DockerSuite.TestRunSeccompDefaultProfileNS   31.771s
   PASS: docker_cli_network_unix_test.go:1621: DockerSuite.TestDockerNetworkInternalMode    30.940s
   PASS: docker_cli_network_unix_test.go:958: DockerNetworkSuite.TestDockerNetworkDriverUngracefulRestart   27.392s
   PASS: docker_api_swarm_test.go:624: DockerSwarmSuite.TestApiSwarmRaftQuorum  25.981s
   PASS: docker_cli_run_unix_test.go:1076: DockerSuite.TestRunSeccompDefaultProfileAcct 25.896s
   PASS: docker_api_swarm_test.go:694: DockerSwarmSuite.TestApiSwarmNodeRemove  24.595s
   PASS: docker_api_swarm_test.go:802: DockerSwarmSuite.TestApiSwarmLeaveOnPendingJoin  22.170s

Something that I found interested is this:

Elapsed integration suite: 0:50:21
Sum test times: 0:32:09

We spend over 1/3 of the test time in setup/teardown fixtures.

dnephin on Aug 30, 2016

I was taking a look at DockerHubPullSuite.TestPullAllTagsFromCentralRegistry, which apparently takes a long time because it is verifying --all-tags functionality using busybox. I was wondering, would it make sense to host a dedicated dummy repo for this test that has say, 2-3 tiny images? I’m sure there are pre-existing repos with fewer tags/images, but it seems to me it would be nice to have a long-term guarantee that the repo used in this test will have favorable characteristics.

Please let me know if I’ve said anything dumb, I’m just starting my journey of understanding the project and contributing 😃

samwagg on Oct 13, 2016