swift: Significant build regressions on `swift:6.0-noble` compared to `5.10-noble`
Description
Absolutely massive Significant build regressions on linux , worsened by swift-testing
EDIT: Please also read my next comment which includes more info.
Environment
- GitHub Actions
- 8 CPU x 16GB RAM c7x EC2 Instance (we use RunsOn)
- Migrating from
swift:5.10-nobletoswift:6.0-noble - Tried with and without using a previous
.buildcache. No behavior difference noticed at all. - 200K-300K LOC project
- Updated to
// swift-tools-version:6.0inPackage.swift - All targets still have Swift 5 language mode enabled.
- No actual tests migrated to swift-testing just yet.
What Happened?
- Our Tests CI started getting stuck after the toolchain update.
- The CI machines were getting killed by AWS. Considering this has been a common problem specially in the Server Side Swift ecosystem, which also has been brought up for ages and numerous times, my first guess was that the machine is running out of RAM when building the Swift project.
- After a lot of tries, I noticed a
c7x-family machine with 64 CPU x 128 GB RAM (8x larger than before) runs the tests as well as they were being run before, on swift 5.10. - My first guess was that maybe Swift testing is causing an issue, so tried
--disable-swift-testing. - After that, our tests have been running on an only 2x larger machine, and the rest of the things are back to normal in tests CI.
- With this bigger machine, it still almost takes as long as before for the tests CI to complete (we run tests in parallel with 64 workers).
- So this means that not only Swift 6 has a significant build regression, but also swift-testing makes it go from
significanttoabsolutely massive. - This is borderline unusable for the Swift-Server ecosystem. I hope this problem requires more specific situations to happen that what meets the eye.
- Still trying things out with the deployment CI. even 2x CPU and 4x RAM isn’t proving helpful even though I did throw the
--disable-swift-testingflag into the mix.
Reproduction
Not sure.
Expected behavior
No regressions. Preferably even faster than previous Swift versions.
Environment
Mentioned above.
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 70 (30 by maintainers)
I just reran 2 of the most important CIs in terms of knowing the situation, in the old repo that all the benchmarks above are related to.
On a 2cpu 8ram machine it appears noble images are marginally faster than the jammy images.
This confirms the build regressions, at least those I reported, are gone.
@MaxDesiatov ah no, I believe the other issues are resolved already, mostly thanks to @finagolfin . So only the SwiftPM issue remains as far as I can tell.
Feel free to reopen if there are repro steps that show it’s not specific to SwiftPM and is actually a compiler frontend bug.
Filed this: https://github.com/swiftlang/swift-package-manager/issues/8275
Running some CIs with command
swift build --build-tests -v -j 1:It appears the compiler regressions don’t react to more CPU or RAM. Looks like the compiler is not doing multithreaded stuff which makes sense I guess with
-j 1. However, I could not build this project with less than 8GB of ram as it would simply go OOM.Will continue the investigation.
TBH, because I’m always suspicious of the linker when build time is involved. As I said, I have no evidence at this time to back up that suspicion.
I’ve never tried to track down such a compiler speed regression, but what you can try is to run your build with each compiler in verbose mode with parallelism turned off
-v -j 1, as Erik says, then check the timestamps of the object files to get an idea of how long the build took:I don’t know if the two build tools will stick to the same build order between versions, but you can use the verbose build log and the object file timestamps to find which commands are regressing the most. Once you have the slower compilation command, there is some doc on what flags to use to try and further track down compiler performance issues.
Try that out and let us know what you find. 😸
I’ll need to redo the specific CIs to be sure, but yeah it does seem Swift 6.0.3 has solved the issues. In smaller projects like vapor/penny-bot I noticed no significant discrepancies on Swift jammy vs noble. If anything noble was <5 s faster.
To the best of my knowledge, we’ve found that 6.0.3 has largely resolved the regressions referenced here; I’ll defer to @MahdiBM for a more definitive statement.
@finagolfin There are some improvements in the more-constrained environments:
<kbd>
</kbd>
<kbd>
</kbd>
Reminder:
Also the name is wrong, this is just a
swift build --build-tests.The actual build-only steps took 9:45s vs 10:28s in the smaller machine in one of the 2 tries, which indicates a 7% improvement.
Alright, @MahdiBM, got something for you to try: a new Swift 6.0 Oct. 8 snapshot tag was just released with the linker deduplication change for Foundation. Compare it with the last 6.0 Sep. 17 snapshot tag and see how much it improves your build. Comparing it to the 6.0.1 release may not help, as that has other stuff turned off for release builds.
Ever since the last comment, I’ve seen 2 more users encountering this problem on Heroku. So a total of 3, and that’s only those who cared to ask about their weird build failures before reverting back.
Interesting bit is that one person was consistently having problems on Swift 6 jammy, but their project would consistently build fine on Swift 6 noble, in Heroku buildpack builders. This is unexpected to me, but shows that the regressions are even more complicated and have happened on multiple different fronts.
So:
I was unable to find specs of Heroku builders (also asked around a bit, no luck) to see if anything has changed between their Ubuntu 22 and 24 stacks.
First victim: Heroku can’t build Swift apps that it previously could: https://github.com/vapor-community/heroku-buildpack/issues/78#issuecomment-2370945753
Heroku is a pretty popular platform for SSS apps, considering it once had a free tier. This problem will be have a decent negative impact.
It was the CNIO errors. Couldn’t get them fixed although i made some decent effort.
Of course there is no problem on the released Swift 6 😅. This whole issue is about CI slowness. We run CI on push, PR, etc… and they’ve all passed.
Does simply building your code with the Swift 6.0 release still work? If so, I’d try to instrument the build to figure out the bottlenecks, as I suggested before. In particular, if you’re building a large executable at the end, that might be taking the most time.
As I said yesterday, you could try dumping all the commands that
swift buildis running with the-vflag, check if the final linker command is repeating-lFoundationEssentials -l_FoundationICUover and over again, then manually run that link alone twice to measure the timing: once exactly as it was to get the baseline, then a second time with those library flags de-duplicated to see if that helps. Ideally, you’d do this on the same lower-RAM hardware where you were seeing the largest build-time regressions before.@MahdiBM, I have submitted that linker fix for the next 6.0.1 patch release, but the branch managers would like some confirmation from you that this is fixed in trunk first. Specifically, you should compare the Aug. 29 trunk 6.1 snapshot build from last month to the Sep. 4 or subsequent trunk builds.
Yes, I pinged you on that last month, but never got a response. A fix was just merged into
release/6.0in #76504, so it will not be released until 6.0.2 or 6.0.3.@jmschonfeld or @shahmishal, can that fix be prioritized to get into the next patch release?