swift-collections: Fails to compile (Abort trap: 6)

Update for anyone affected by this: I figured out a way to reliably reproduce this issue, and I’m developing a workaround that has minimal effect on future development.

The assertion is a (known, rare) issue with swiftc’s -merge-modules action (used by SwiftPM’s default debug configuration), which by itself would be easy to work around. However, the issue seems to be somehow only triggered by some specific orderings in which the OrderedSet source files happen to be listed in the Sources/OrderedCollections/OrderedSet directory: the assertion is reproducible with a small subset of orderings, but not others. If this is indeed the trigger, this neatly explains why this issue only hits certain machines, and why it’s so reliably reproducible on them. (If this was caused by something accidentally iterating over an unsorted hash table within SwiftPM or the compiler, then I’d expect the bug to trigger with the same frequency on all machines.)

After we have the workaround in place, and we have a working swift-collection release, I’ll spend some time on tracking down the exact source of this nondeterminism, and fixing it so that future compilers won’t produce such unpleasant surprises. The order of directory entries should have no effect on the build process of a Swift package.

(The actual -merge-modules assertion is going to get fixed by replacing it with a better mechanism. This work is already underway, but it will take some time.)

One amusing aspect of this is that filesystem directories are just another dictionary implementation – so it seems the OrderedCollections module, including the OrderedDictionary implementation, may have been broken by some code accidentally relying on the order of items in an unordered dictionary – exactly the sort of thing that OrderedDictionary is designed to prevent.

workaround solves it for me as well, thank you for tracking down that subtle issue and getting us a fix!

@lorentey it works! 🎉 good job!

Test Suite 'swift-collectionsPackageTests.xctest' passed at 2021-04-10 11:26:11.636.
	 Executed 268 tests, with 0 failures (0 unexpected) in 85.478 (85.532) seconds
Test Suite 'All tests' passed at 2021-04-10 11:26:11.636.
	 Executed 268 tests, with 0 failures (0 unexpected) in 85.478 (85.533) seconds

OK, I merged a workaround for the build assertion.

The exact source of the nondeterminism still eludes me, so this is something of an “it works on my machine” situation, but I’ve let my shuffle-the-sources-then-do-a-clean-rebuild script run for a day now, and it is yet to surface any issues.

@Sajjon, @heckj: Thanks for all the helpful assistance (& patience!). When convenient, could you please check if the current main branch (as of commit 2d719d75a2065f213e58a5164384a3d2fcf9b59a) resolves the issue for you? If so, I’ll tag a release, and we can move on to happier matters. 😝

@lorentey trivial PR: https://github.com/apple/swift-collections/pull/15

I just checked my system, and I’ve got the same list as @Sajjon.

@lorentey seems I get the same ouput that you get:

$ git --no-pager log --decorate=short --pretty=oneline -n1
5a2a6841fd60d01ea2f0365314a062672304d0b1 (HEAD -> main, upstream/main, origin/main, origin/HEAD) Update OrderedSet+ReserveCapacity.swift

~/Developer/Swift/Apple/swift-collections on  main! ⌚ 7:16:09
$ ls -f ./Sources/OrderedCollections/OrderedSet | cat
.
..
OrderedSet+RandomAccessCollection.swift
OrderedSet+Sequence.swift
OrderedSet+UnstableInternals.swift
OrderedSet+Invariants.swift
OrderedSet+Hashable.swift
OrderedSet+CustomDebugStringConvertible.swift
OrderedSet+CustomStringConvertible.swift
OrderedSet+Insertions.swift
OrderedSet+Codable.swift
OrderedSet+Equatable.swift
OrderedSet+Initializers.swift
OrderedSet.swift
OrderedSet+UnorderedView.swift
OrderedSet+ExpressibleByArrayLiteral.swift
OrderedSet+SubSequence.swift
OrderedSet+Partial MutableCollection.swift
OrderedSet+Testing.swift
OrderedSet+CustomReflectable.swift
OrderedSet+ReserveCapacity.swift
OrderedSet+Partial RangeReplaceableCollection.swift
OrderedSet+Partial SetAlgebra+Basics.swift
OrderedSet+Partial SetAlgebra+Predicates.swift
OrderedSet+Partial SetAlgebra+Operations.swift

~/Developer/Swift/Apple/swift-collections on  main! ⌚ 7:16:14

Want me to check something else?

Btw, seems like Package.resolved isn’t git commited? using latest commit on main, 5a2a6841fd60d01ea2f0365314a062672304d0b1 when I run swift test (CLI), git status shows change in Package.resolved:

      {
        "package": "swift-collections-benchmark",
        "repositoryURL": "https://github.com/apple/swift-collections-benchmark",
        "state": {
          "branch": null,
          "revision": "49297647d1c0855eb33ec18cd6ee1181727c159b",
          "version": "0.0.1"
        }
      },

Awesome, great job @lorentey as always! I’m very much looking forward to playing around with Swift-Collections, but Benchmark package specifically!

Feel free to ping me if you want me to try some release candidate or something since I seem to hit this bug every time.

Have a nice weekend!

@lorentey some more details on my machine/environment

Machine

macOS Big Sure 11.2.3 (20D91).
MacBook Pro (16-inch, 2019)
2,3 GHz 8-Core Intel Core i9
64 GB 2667 MHz DDR4
Intel UHD Graphics 630 1536 MB

Environment

$ swift --version
Apple Swift version 5.3.2 (swiftlang-1200.0.45 clang-1200.0.32.28)
Target: x86_64-apple-darwin20.3.0

$ clang --version
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I hit the same - reported is as bug https://bugs.swift.org/browse/SR-14448.

Others have reported this too. Evidently it is only triggered on some unknown condition but when it is triggered, it remains easily reproducible. We’re looking into it!

https://forums.swift.org/t/compiler-failure-when-trying-to-build-the-new-swift-collections/47170

I get the same error, also using Xcode 12.4 (12D4e), but on macOS 11.2.3 (20D91).