swift-testing: Compiling many Swift Testing tests is significantly slower than XCTest tests
Description
Compiling a large amount of Swift Testing tests is significantly slower than compiling the same amount of tests using XCTest on a clean build.
At 1000 tests in a single file building a test suite with Swift Testing is 24x slower than an almost identical suite using XCTest.
Here’s a repo with a project to benchmark: https://github.com/marcosgriselli/swift-testing-performance. A few results from that project:
1000 Tests in a single file:
➜ swift-testing-performance git:(main) ✗ ./benchmark.py --test_count 1000 --runs 5
Benchmark 1: XCTest (1000 tests)
Time (mean ± σ): 3.059 s ± 0.090 s [User: 0.658 s, System: 0.214 s]
Range (min … max): 2.965 s … 3.183 s 5 runs
Benchmark 2: SwiftTesting (1000 tests)
Time (mean ± σ): 73.940 s ± 0.341 s [User: 0.747 s, System: 0.285 s]
Range (min … max): 73.603 s … 74.427 s 5 runs
Summary
XCTest (1000 tests) ran
24.17 ± 0.72 times faster than SwiftTesting (1000 tests)
Performance significantly improves when splitting the tests into different files, still 10x slower than XCTest.
1000 tests split in 10 files:
➜ swift-testing-performance git:(main) ./benchmark.py --test_count 1000 --test_files 10
Benchmark 1: XCTest (1000 tests in 10 files)
Time (mean ± σ): 2.010 s ± 0.166 s [User: 0.665 s, System: 0.216 s]
Range (min … max): 1.855 s … 2.185 s 3 runs
Benchmark 2: SwiftTesting (1000 tests in 10 files)
Time (mean ± σ): 20.769 s ± 0.351 s [User: 0.792 s, System: 0.295 s]
Range (min … max): 20.531 s … 21.171 s 3 runs
Summary
XCTest (1000 tests in 10 files) ran
10.33 ± 0.87 times faster than SwiftTesting (1000 tests in 10 files)
I’m aware of the macros compilation build time issues outside of swift-syntax and that the team is working on addressing these, is there anything we can do in the meantime to reduce the impact while adopting Swift Testing?
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
Run benchmarks on the shared repo to measure performance.
./benchmarks.py --test_count 1000 --test_files 1 --runs 5
./benchmarks.py --test_count 1000 --test_files 10 --runs 5
swift-testing version/commit hash
Xcode 16.0 (16A5171c)
Swift & OS version (output of swift --version ; uname -a)
swift-driver version: 1.109.2 Apple Swift version 6.0 (swiftlang-6.0.0.3.300 clang-1600.0.20.10) Target: arm64-apple-macosx14.0 Darwin Marcoss-MBP.attlocal.net 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 8
- Comments: 18 (13 by maintainers)
I’m actively looking into this issue:
So far, these patches improves the
swift-testingtest compilation performance by 2.6x or so (tested with a file with 500@Test/#expectcases) . I’m exploring other areas for further improvements.Sorry for the false alarm, this turned out to be a GitHub issue that happened right as we merged our migration: https://github.com/actions/runner-images/issues/11827#issuecomment-2734028720
Thank you for this report @marcosgriselli! The example project and data are very helpful. I am investigating this along with @rintaro.
Tracked internally as rdar://130478685
Thanks for those statistics! I agree it sounds like we can close this as resolved at this point. The timing between the two libraries will never be identical, but 1.55x is a reasonable baseline and is something we can iteratively improve over time.
It includes the entire job, so yes including runtime both before and after.
We were fully serial before, but we did enable parallelization for a lot of the Swift Testing suites. So if anything runtime should have gotten faster.
We can wait to transfer. The relevant people know about the issue already.
At this point I do believe it should be transferred to the swiftlang/swift repo. I just tried doing that though, and it appears that you cannot transfer issues across organization boundaries, and due to the pending apple → swiftlang organization transition we’ll need to wait until swift-testing makes the transition. We expect this to happen before too long.