MINGW-packages: i686 lld crashes linking, mostly on Github runners

I have been building packages for clang32/i686 for several months now, and have not seen this until I started trying to build them in a Github actions workflow/hosted runner. It seems to happen intermittently on different packages, but seems to be consistently happening when linking python, which uses LTO and PGO (this is the profile-instr-generate phase).

libc++abi: terminating with uncaught exception of type std::__1::system_error: thread constructor failed: Exec format error
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.	Program arguments: D:/a/_temp/msys/msys64/clang32/bin/ld.lld -m i386pe --shared -Bdynamic -e _DllMainCRTStartup@12 --enable-auto-image-base -o libpython3.8.dll D:/a/_temp/msys/msys64/clang32/i686-w64-mingw32/lib/dllcrt2.o D:/a/_temp/msys/msys64/clang32/i686-w64-mingw32/lib/crtbegin.o -LD:/a/_temp/msys/msys64/clang32/i686-w64-mingw32/lib -LD:/a/_temp/msys/msys64/clang32/lib -LD:/a/_temp/msys/msys64/clang32/i686-w64-mingw32/sys-root/mingw/lib -LD:/a/_temp/msys/msys64/clang32/lib/clang/11.0.0/lib/windows --enable-auto-image-base --dynamicbase --no-seh --dynamicbase --no-seh --out-implib=libpython3.8.dll.a Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/token.o Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/c...
 #0 0x6df45608 HandleAbort (D:\a\_temp\msys\msys64\clang32\bin\libLLVM.dll+0x125608)
 msys2/CLANG-packages#1 0x754a5b22 (C:\Windows\System32\ucrtbase.dll+0xa5b22)
 msys2/CLANG-packages#2 0x6dd97a98 abort_message (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x77a98)
 msys2/MINGW-packages#9044 0x6dd7a616 __cxa_throw_bad_array_new_length (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x5a616)
 msys2/CLANG-packages#4 0x6ddabf20 (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x8bf20)
 msys2/CLANG-packages#5 0x6ddabfa8 (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x8bfa8)
 msys2/CLANG-packages#6 0x0250e1a4 
 msys2/CLANG-packages#7 0x0003fda4 
 msys2/CLANG-packages#8 0x6dd022ac _ZN9libunwind12UnwindCursorINS_17LocalAddressSpaceENS_13Registers_x86EE4stepEv (D:\a\_temp\msys\msys64\clang32\bin\libunwind.dll+0x22ac)
 msys2/CLANG-packages#9 0x6dd05db6 _Unwind_RaiseException (D:\a\_temp\msys\msys64\clang32\bin\libunwind.dll+0x5db6)
msys2/CLANG-packages#10 0x75443dfc (C:\Windows\System32\ucrtbase.dll+0x43dfc)
msys2/CLANG-packages#11 0x011cdcfc _ZN3lld4coff12LinkerDriver11enqueuePathEN4llvm9StringRefEbb (D:\a\_temp\msys\msys64\clang32\bin\ld.lld.exe+0xdcfc)
msys2/CLANG-packages#12 0x6dd782cb _ZNSt3__120__throw_system_errorEiPKc (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x582cb)
msys2/CLANG-packages#13 0x75642b66 (C:\Windows\System32\KERNELBASE.dll+0x112b66)
msys2/CLANG-packages#14 0x778ee697 (C:\Windows\SYSTEM32\ntdll.dll+0x3e697)
msys2/CLANG-packages#15 0x7564b627 (C:\Windows\System32\KERNELBASE.dll+0x11b627)
msys2/CLANG-packages#16 0x6dd7a315 _ZNSt3__121__libcpp_execute_onceEPPvPFvvE (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x5a315)
msys2/CLANG-packages#17 0x6dd7a49a _ZNSt3__116__libcpp_tls_getEl (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x5a49a)
msys2/CLANG-packages#18 0x6dd96b42 __cxa_get_globals (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x76b42)
msys2/CLANG-packages#19 0x6dd96e85 _ZSt11__terminatePFvvE (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x76e85)
msys2/CLANG-packages#20 0x6dd997d7 __cxa_throw (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x797d7)
msys2/CLANG-packages#21 0x6dd99762 __cxa_throw (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x79762)
msys2/CLANG-packages#22 0x6dd782ba _ZNSt3__120__throw_system_errorEiPKc (D:\a\_temp\msys\msys64\clang32\bin\libc++.dll+0x582ba)
msys2/CLANG-packages#23 0x011cdcfc _ZN3lld4coff12LinkerDriver11enqueuePathEN4llvm9StringRefEbb (D:\a\_temp\msys\msys64\clang32\bin\ld.lld.exe+0xdcfc)
msys2/CLANG-packages#24 0x011cd994 _ZN3lld4coff12LinkerDriver11enqueuePathEN4llvm9StringRefEbb (D:\a\_temp\msys\msys64\clang32\bin\ld.lld.exe+0xd994)
msys2/CLANG-packages#25 0x6df393d0 _ZN4llvm3sys2fs11getUniqueIDENS_5TwineERNS1_8UniqueIDE (D:\a\_temp\msys\msys64\clang32\bin\libLLVM.dll+0x1193d0)
msys2/MINGW-packages#9045 0x011c8025 _ZN3lld4coff12LinkerDriver4linkEN4llvm8ArrayRefIPKcEE (D:\a\_temp\msys\msys64\clang32\bin\ld.lld.exe+0x8025)
msys2/CLANG-packages#27 0x77908450 (C:\Windows\SYSTEM32\ntdll.dll+0x58450)
msys2/MINGW-packages#9046 0x77908450 (C:\Windows\SYSTEM32\ntdll.dll+0x58450)
msys2/MINGW-packages#9047 0x778ed925 (C:\Windows\SYSTEM32\ntdll.dll+0x3d925)
msys2/CLANG-packages#30 0x778ed49e (C:\Windows\SYSTEM32\ntdll.dll+0x3d49e)
msys2/CLANG-packages#31 0x778ed1e2 (C:\Windows\SYSTEM32\ntdll.dll+0x3d1e2)
msys2/CLANG-packages#32 0x7564081e (C:\Windows\System32\KERNELBASE.dll+0x11081e)
msys2/CLANG-packages#33 0x0002ed3a

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 42 (34 by maintainers)

Commits related to this issue

Allow CLANG32 jobs to fail. Linker errors (#34) are being troublesome. — committed to jeremyd2019/CLANG-packages by jeremyd2019 3 years ago
Allow CLANG32 jobs to fail. Linker errors (#34) are being troublesome. — committed to jeremyd2019/CLANG-packages by jeremyd2019 3 years ago
Enable --large-address-aware on 32-bit. LLD needs all the address space it can get. This might help with #34 — committed to jeremyd2019/CLANG-packages by jeremyd2019 3 years ago
Enable --large-address-aware on 32-bit. LLD needs all the address space it can get. This might help with #34 — committed to jeremyd2019/CLANG-packages by jeremyd2019 3 years ago
disable 32-bit workarounds for #34. Update clang patches. — committed to jeremyd2019/CLANG-packages by jeremyd2019 3 years ago
clang: disable std::launch::async on 32-bit There seems to be issues there with a large number of threads. Fixes msys2/CLANG-packages#34 — committed to jeremyd2019/MINGW-packages by jeremyd2019 3 years ago
clang: disable std::launch::async on 32-bit There seems to be issues there with a large number of threads. Fixes msys2/CLANG-packages#34 — committed to jeremyd2019/MINGW-packages by jeremyd2019 3 years ago
clang: use ninja for clang32. Now that #9048 is fixed this is no longer necessary. — committed to jeremyd2019/MINGW-packages by jeremyd2019 3 years ago
clang: use ninja for clang32. Now that #9048 is fixed this is no longer necessary. — committed to jeremyd2019/MINGW-packages by jeremyd2019 3 years ago

Most upvoted comments

My patch for this is now committed upstream: https://github.com/llvm/llvm-project/commit/7a7da69fbe288de088bfee47d2f7c21da2132085

jeremyd2019 on Jul 7, 2021

wxWidgets

wxWidgets3.1

xalan-c

assimp

I was able to build all of these but xalan-c locally. Case of the cursed github runner again?

jeremyd2019 on Jun 23, 2021

Building clang with large-address-aware is still a good thing to do, but doesn’t seem to have solved this particular issue.

jeremyd2019 on Apr 25, 2021

I’m just going to do the clang PKGBUILD for now. My plan is to run through bootstrapping on my fork with my hacks around this issue removed and see if this solves it. Then I can see about a PR to MINGW-packages

jeremyd2019 on Apr 24, 2021

vs just doing in in the clang PKGBUILD? Or vs changing the default in LLD?

vs changing the default. my 2c, from what I see this has the potential to break things, and unlike with ASLR we don’t get the benefit of MSVC having this enabled by default, so bugs might surface. And rare/hard to reproduce bugs on top of that.

lazka on Apr 24, 2021

Would something like editbin /largeaddressaware lld.exe help here? Afair that bumps the limit on virtual address space from 2GB to 3GB or so

Hm, I guess it would postpone the issue a bit at least.

mstorsjo on Apr 24, 2021

I think the GHA windows-latest runner is cursed with respect to 32-bit address space. I build msys2/i686 packages mostly using them too, and python build always has fork issues, even though it works fine on my local Windows VM.

jeremyd2019 on Apr 15, 2021