swift: Significant build regressions on `swift:6.0-noble` compared to `5.10-noble`

Description

Absolutely massive Significant build regressions on linux , worsened by swift-testing

EDIT: Please also read my next comment which includes more info.

Environment

  • GitHub Actions
  • 8 CPU x 16GB RAM c7x EC2 Instance (we use RunsOn)
  • Migrating from swift:5.10-noble to swift:6.0-noble
  • Tried with and without using a previous .build cache. No behavior difference noticed at all.
  • 200K-300K LOC project
  • Updated to // swift-tools-version:6.0 in Package.swift
  • All targets still have Swift 5 language mode enabled.
  • No actual tests migrated to swift-testing just yet.

What Happened?

  • Our Tests CI started getting stuck after the toolchain update.
  • The CI machines were getting killed by AWS. Considering this has been a common problem specially in the Server Side Swift ecosystem, which also has been brought up for ages and numerous times, my first guess was that the machine is running out of RAM when building the Swift project.
  • After a lot of tries, I noticed a c7x-family machine with 64 CPU x 128 GB RAM (8x larger than before) runs the tests as well as they were being run before, on swift 5.10.
  • My first guess was that maybe Swift testing is causing an issue, so tried --disable-swift-testing.
  • After that, our tests have been running on an only 2x larger machine, and the rest of the things are back to normal in tests CI.
  • With this bigger machine, it still almost takes as long as before for the tests CI to complete (we run tests in parallel with 64 workers).
  • So this means that not only Swift 6 has a significant build regression, but also swift-testing makes it go from significant to absolutely massive.
  • This is borderline unusable for the Swift-Server ecosystem. I hope this problem requires more specific situations to happen that what meets the eye.
  • Still trying things out with the deployment CI. even 2x CPU and 4x RAM isn’t proving helpful even though I did throw the --disable-swift-testing flag into the mix.

Reproduction

Not sure.

Expected behavior

No regressions. Preferably even faster than previous Swift versions.

Environment

Mentioned above.

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 70 (30 by maintainers)

Most upvoted comments

I just reran 2 of the most important CIs in terms of knowing the situation, in the old repo that all the benchmarks above are related to.

On a 2cpu 8ram machine it appears noble images are marginally faster than the jammy images.

This confirms the build regressions, at least those I reported, are gone.

@MaxDesiatov ah no, I believe the other issues are resolved already, mostly thanks to @finagolfin . So only the SwiftPM issue remains as far as I can tell.

Feel free to reopen if there are repro steps that show it’s not specific to SwiftPM and is actually a compiler frontend bug.

Running some CIs with command swift build --build-tests -v -j 1:

Machine swift:5.10-jammy swift:6.0-jammy regression %
AWS EC2 m7a 32cpu 128ram 17m 12s 31m 50s 85%
AWS EC2 m7a 2cpu 8ram 18m 2s 33m 2s 83%

It appears the compiler regressions don’t react to more CPU or RAM. Looks like the compiler is not doing multithreaded stuff which makes sense I guess with -j 1. However, I could not build this project with less than 8GB of ram as it would simply go OOM.

Will continue the investigation.

Why do you still suspect the final linker run, considering 6.0.3 now de-duplicates most of the Swift runtime library names passed to the linker?

TBH, because I’m always suspicious of the linker when build time is involved. As I said, I have no evidence at this time to back up that suspicion.

I’ve never tried to track down such a compiler speed regression, but what you can try is to run your build with each compiler in verbose mode with parallelism turned off -v -j 1, as Erik says, then check the timestamps of the object files to get an idea of how long the build took:

> find .build/aarch64-unknown-linux-android24/debug/ -name "*.o" | xargs ls -l --full-time | sort
-rw-------. 1 u0_a511 u0_a511    4664 2025-01-18 13:59:41.531677245 +0130 .build/aarch64-unknown-linux-android24/debug/ArgumentParserGenerateManualTests.build/CountLinesGenerateManualTests.swift.o
-rw-------. 1 u0_a511 u0_a511    4816 2025-01-18 13:59:40.587677245 +0130 .build/aarch64-unknown-linux-android24/debug/ArgumentParserExampleTests.build/CountLinesExampleTests.swift.o

I don’t know if the two build tools will stick to the same build order between versions, but you can use the verbose build log and the object file timestamps to find which commands are regressing the most. Once you have the slower compilation command, there is some doc on what flags to use to try and further track down compiler performance issues.

Try that out and let us know what you find. 😸

I’ll need to redo the specific CIs to be sure, but yeah it does seem Swift 6.0.3 has solved the issues. In smaller projects like vapor/penny-bot I noticed no significant discrepancies on Swift jammy vs noble. If anything noble was <5 s faster.

To the best of my knowledge, we’ve found that 6.0.3 has largely resolved the regressions referenced here; I’ll defer to @MahdiBM for a more definitive statement.

@finagolfin There are some improvements in the more-constrained environments:

<kbd> Screenshot 2024-10-12 at 9 55 05 PM </kbd>

<kbd> Screenshot 2024-10-12 at 11 05 48 PM </kbd>

Reminder:

          - name: "medium" # 16gb 8cpu c7x
            arch: amd64
          - name: "large" # 32gb 16cpu c7x
            arch: amd64

Also the name is wrong, this is just a swift build --build-tests.

The actual build-only steps took 9:45s vs 10:28s in the smaller machine in one of the 2 tries, which indicates a 7% improvement.

Alright, @MahdiBM, got something for you to try: a new Swift 6.0 Oct. 8 snapshot tag was just released with the linker deduplication change for Foundation. Compare it with the last 6.0 Sep. 17 snapshot tag and see how much it improves your build. Comparing it to the 6.0.1 release may not help, as that has other stuff turned off for release builds.

Ever since the last comment, I’ve seen 2 more users encountering this problem on Heroku. So a total of 3, and that’s only those who cared to ask about their weird build failures before reverting back.

Interesting bit is that one person was consistently having problems on Swift 6 jammy, but their project would consistently build fine on Swift 6 noble, in Heroku buildpack builders. This is unexpected to me, but shows that the regressions are even more complicated and have happened on multiple different fronts.

So:

  • Swift noble, even on 5.10 has build-time regressions.
  • Swift 6 itself contains some regressions as well.
  • jammy / noble contain their own regressions as well?!

I was unable to find specs of Heroku builders (also asked around a bit, no luck) to see if anything has changed between their Ubuntu 22 and 24 stacks.

First victim: Heroku can’t build Swift apps that it previously could: https://github.com/vapor-community/heroku-buildpack/issues/78#issuecomment-2370945753

Heroku is a pretty popular platform for SSS apps, considering it once had a free tier. This problem will be have a decent negative impact.

It was the CNIO errors. Couldn’t get them fixed although i made some decent effort.

Of course there is no problem on the released Swift 6 😅. This whole issue is about CI slowness. We run CI on push, PR, etc… and they’ve all passed.

Does simply building your code with the Swift 6.0 release still work? If so, I’d try to instrument the build to figure out the bottlenecks, as I suggested before. In particular, if you’re building a large executable at the end, that might be taking the most time.

As I said yesterday, you could try dumping all the commands that swift build is running with the -v flag, check if the final linker command is repeating -lFoundationEssentials -l_FoundationICU over and over again, then manually run that link alone twice to measure the timing: once exactly as it was to get the baseline, then a second time with those library flags de-duplicated to see if that helps. Ideally, you’d do this on the same lower-RAM hardware where you were seeing the largest build-time regressions before.

@MahdiBM, I have submitted that linker fix for the next 6.0.1 patch release, but the branch managers would like some confirmation from you that this is fixed in trunk first. Specifically, you should compare the Aug. 29 trunk 6.1 snapshot build from last month to the Sep. 4 or subsequent trunk builds.

Is there any chance we’re still failing to deduplicate linker command lines fully?

Yes, I pinged you on that last month, but never got a response. A fix was just merged into release/6.0 in #76504, so it will not be released until 6.0.2 or 6.0.3.

@jmschonfeld or @shahmishal, can that fix be prioritized to get into the next patch release?