moby: Optimize slow bottleneck tests
The time the integration test suite takes to finish is getting out of control. We would like to ask your help in optimizing some of the longest test cases. This is another of those “non-pleasant” tasks of which, if we all do just one, we can get it all done extra fast.
If you would like to help out, please comment on this issue with the name of the test you would like to take care of; we’ll add your Github handle next to the test name so that others can tell which tests still need an owner.
UPDATE: optimizing a test consists of making sure the test code is reasonably efficient. However if the slowdown is a sign of not efficient code in Docker itself, then that would need to be addressed too.
I’m shamelessly stealing the following from @estesp’s gist https://gist.github.com/estesp/40a056a8262eafb62dd1:
Sampled from a 1/12/2016 run of standard Linux x64 engine janky run.
Total time of these 25 tests adds up to 663.8 seconds
These 25 make up 2.2% of total tests, but ~29% of total test runtime.
DockerDaemonSuite.TestDaemonNoSpaceleftOnDeviceError 107.143s DockerTrustSuite.TestTrustedPushWithFailingServer 95.405s
DockerSuite.TestRunSeccompDefaultProfile 81.711s
DockerSuite.TestEventsImagePull 43.321s
DockerSuite.TestBuildEnvUsage2 32.791s
DockerSuite.TestDaemonStartWithDaemonCommand 25.648s
DockerHubPullSuite.TestPullAllTagsFromCentralRegistry 24.504s
DockerSuite.TestEventsOOMDisableTrue 20.288s
DockerHubPullSuite.TestPullFromCentralRegistryImplicitRefParts 19.389s
DockerSuite.TestBuildHistory 18.234s
DockerSuite.TestBuildUsersAndGroups 16.550s
DockerSuite.TestRunUnshareProc 16.196s
DockerSuite.TestVolumeCliCreate 15.118s
DockerSuite.TestApiStatsNetworkStatsVersioning 14.699s
DockerSuite.TestBuildDockerignoringWildDirs 14.443s
DockerSuite.TestApiStatsNetworkStats 13.200s
DockerSuite.TestBuildSixtySteps 12.663s
DockerHubPullSuite.TestPullNonExistingImage 12.259s
DockerDaemonSuite.TestDaemonRestartWithContainerRunning 12.110s
DockerDaemonSuite.TestDaemonCorruptedLogDriverAddress 12.007s
DockerSuite.TestDaemonIPv6FixedCIDR 11.564s
DockerSuite.TestInfoDiscoveryInvalidAdvertise 11.504s
DockerSuite.TestDaemonIPv6FixedCIDRAndMac 11.451s
DockerSuite.TestHelpTextVerify 11.039s
DockerSuite.TestEventsAttach 10.635s
Additional tests (suggested by @aaronlehmann)
DockerSuite.TestEventsContainerFailStartDie 40.018s
DockerSuite.TestRunSeccompAllowPrivCloneUserns 40.302s
DockerSuite.TestRunSeccompProfileDenyCloneUserns 45.066s
DockerSuite.TestRunSeccompUnconfinedCloneUserns 40.056s
And more (suggested by @wenchma)
DockerNetworkSuite.TestDockerNetworkInternalMode 11.268s
TODO:
- DockerDaemonSuite.TestDaemonNoSpaceleftOnDeviceError (#19551 @aaronlehmann)
- DockerTrustSuite.TestTrustedPushWithFailingServer (#23217 @clnperez)
- DockerSuite.TestRunSeccompDefaultProfile (#19725 @WeiZhang555)
- DockerSuite.TestEventsImagePull (test is now 2.273s)
- DockerSuite.TestBuildEnvUsage2 (#19600 @aaronlehmann)
- DockerSuite.TestDaemonStartWithDaemonCommand (#20476 @wenchma)
- DockerHubPullSuite.TestPullAllTagsFromCentralRegistry
- DockerSuite.TestEventsOOMDisableTrue (#19619 @WeiZhang555)
- DockerHubPullSuite.TestPullFromCentralRegistryImplicitRefParts
- DockerSuite.TestBuildHistory (#20880 @yongtang)
- DockerSuite.TestBuildUsersAndGroups (#21056 @askb)
- DockerSuite.TestRunUnshareProc (#20965 @yongtang)
- DockerSuite.TestVolumeCliCreate (@askb)
- DockerSuite.TestApiStatsNetworkStatsVersioning (#19800 @wenchma)
- DockerSuite.TestBuildDockerignoringWildDirs (#20929 @yongtang)
- DockerSuite.TestApiStatsNetworkStats (#19800 @wenchma)
- DockerSuite.TestBuildSixtySteps
- DockerHubPullSuite.TestPullNonExistingImage
- DockerDaemonSuite.TestDaemonRestartWithContainerRunning (#19663 @mountkin)
- DockerDaemonSuite.TestDaemonCorruptedLogDriverAddress (#19708 @wenchma)
- DockerSuite.TestDaemonIPv6FixedCIDR (#19658 @wenchma)
- DockerSuite.TestInfoDiscoveryInvalidAdvertise
- DockerSuite.TestDaemonIPv6FixedCIDRAndMac (#19658 @wenchma)
- DockerSuite.TestHelpTextVerify (@sainath14)
- DockerSuite.TestEventsAttach (#19703 @mountkin)
- DockerSuite.TestEventsContainerFailStartDie (#19713 @WeiZhang555)
- DockerSuite.TestRunSeccompAllowPrivCloneUserns
- DockerSuite.TestRunSeccompProfileDenyCloneUserns
- DockerSuite.TestRunSeccompUnconfinedCloneUserns
- DockerNetworkSuite.TestDockerNetworkInternalMode (#20903 @wenchma )
About this issue
- Original URL
- State: open
- Created 8 years ago
- Comments: 60 (55 by maintainers)
Commits related to this issue
- Slight TestBuildEnvUsage2 speedup TestBuildEnvUsage2 was flagged in https://github.com/docker/docker/issues/19425 as one of the slowest integration tests. It's slow because it has some comprehensive ... — committed to aaronlehmann/docker by aaronlehmann 8 years ago
- Optimize slow integration test Optimize `TestEventsOOMDisableTrue` performance It's part of #19425 issue. Signed-off-by: Zhang Wei <zhangwei555@huawei.com> — committed to WeiZhang555/moby by WeiZhang555 8 years ago
- Slight TestBuildEnvUsage2 speedup TestBuildEnvUsage2 was flagged in https://github.com/docker/docker/issues/19425 as one of the slowest integration tests. It's slow because it has some comprehensive ... — committed to aditirajagopal/docker by aaronlehmann 8 years ago
- Optimize slow integration test Optimize `TestEventsOOMDisableTrue` performance It's part of #19425 issue. Signed-off-by: Zhang Wei <zhangwei555@huawei.com> — committed to aditirajagopal/docker by WeiZhang555 8 years ago
- Optimize slow bottleneck tests of TestDaemonStartWithDaemonCommand * Remvoe integration test of TestDaemonStartWithDaemonCommand * Rewrite as unit test Related issue #19425 Signed-off-by: Wen Cheng... — committed to wenchma/docker by deleted user 8 years ago
- Optimized integration test case DockerSuite.TestBuildUsersAndGroups for #19425 Removed unnecessary RUN statements and combined some of the RUN statement into a single line. The runtime performance i... — committed to moby/moby by askb 8 years ago
- Merge pull request #21056 from askb/optimize-TestBuildUsersAndGroups Optimized integration test case DockerSuite.TestBuildUsersAndGroups fixes #19425 — committed to moby/moby by calavera 8 years ago
Ok, I wrote up a script and put it into a docker image
Something that I found interested is this:
We spend over 1/3 of the test time in setup/teardown fixtures.
I was taking a look at
DockerHubPullSuite.TestPullAllTagsFromCentralRegistry
, which apparently takes a long time because it is verifying --all-tags functionality using busybox. I was wondering, would it make sense to host a dedicated dummy repo for this test that has say, 2-3 tiny images? I’m sure there are pre-existing repos with fewer tags/images, but it seems to me it would be nice to have a long-term guarantee that the repo used in this test will have favorable characteristics.Please let me know if I’ve said anything dumb, I’m just starting my journey of understanding the project and contributing 😃