godot: Godot 4 compiled with MinGW is crashing with a white screen [GCC 11.x bug fixed in 11.3]
Godot version
4.0 custom build (bdcc8741e)
System information
Windows 10 21H1
Issue description
The editor is crashing with a white screen on startup, with or without a project. I have been tried some combinations of compilers and options: MSVC with debug -> OK MSVC with release_debug -> OK MinGW with debug -> OK MinGW with release_debug -> Crash
This issue has been introduced with the 99fe462452be44efa618e83ad9bbecd722ae6ecd commit.
Output of drmingw
godot.windows.opt.tools.64.exe caused an Access Violation at location 00007FF73A4C0000 DEP violation at location 00007FF73A4C0000.
AddrPC Params
00007FF73A4C0000 00000215616FA200 0000003CE61FF5A8 0000003CE61FF5B0
00007FF77C559842 0000000000400000 00007FF77CA33F03 0000000000400000 godot.windows.opt.tools.64.exe!RenderingServerDefault::RenderingServerDefault [E:/path/core/os/thread.h @ 90]
88: _FORCE_INLINE_ ID get_id() const { return id; }
89: // get the ID of the caller thread
> 90: _FORCE_INLINE_ static ID get_caller_id() { return caller_id; }
91: // get the ID of the main thread
92: _FORCE_INLINE_ static ID get_main_id() { return main_thread_id; }
00007FF77A4DE90B 0000021561579CC0 0000021560C1F690 00000215000000FF godot.windows.opt.tools.64.exe!Main::setup2 [E:/path/main/main.cpp @ 1590]
1588: /* Initialize Rendering Server */
1589:
> 1590: rendering_server = memnew(RenderingServerDefault(OS::get_singleton()->get_render_thread_mode() == OS::RENDER_SEPARATE_THREAD));
1591:
1592: rendering_server->init();
00007FF77A4E479B 0000021560C20C30 0000000000000001 000002155F450860 godot.windows.opt.tools.64.exe!Main::setup [E:/path/main/main.cpp @ 1388]
1386:
1387: if (p_second_phase) {
> 1388: return setup2();
1389: }
1390:
00007FF77A4C18A7 000002155F153296 0000003CE61FFD4C 0000000000000001 godot.windows.opt.tools.64.exe!widechar_main [E:/path/platform/windows/godot_windows.cpp @ 151]
149: TEST_MAIN_PARAM_OVERRIDE(argc, argv_utf8)
150:
> 151: Error err = Main::setup(argv_utf8[0], argc - 1, &argv_utf8[1]);
152:
153: if (err != OK) {
00007FF77A4C19A0 0000000000000000 0000000000000047 00007FF7811164D8 godot.windows.opt.tools.64.exe!_main [E:/path/platform/windows/godot_windows.cpp @ 185]
183: }
184:
> 185: result = widechar_main(argc, wc_argv);
186:
187: LocalFree(wc_argv);
00007FF77A4C13B1 0000000000000000 0000000000000000 0000000000000000 godot.windows.opt.tools.64.exe!__tmainCRTStartup [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c @ 321]
00007FF77A4C14E6 0000000000000000 0000000000000000 0000000000000000 godot.windows.opt.tools.64.exe!mainCRTStartup [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c @ 202]
00007FF8F8997034 0000000000000000 0000000000000000 0000000000000000 KERNEL32.DLL!BaseThreadInitThunk
00007FF8FA1C2651 0000000000000000 0000000000000000 0000000000000000 ntdll.dll!RtlUserThreadStart
Log of the crashing MinGW release_debug build
Godot Engine v4.0.dev.custom_build.bdcc8741e - https://godotengine.org
Using "ICU / HarfBuzz / Graphite" text server...
Vulkan API 1.2.162
ERROR: GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
loader_get_json: Failed to open JSON file C:\ProgramData\GOG.com\Galaxy\redists\overlay\injected\galaxy_overlay_vklayer_x64.json
Objects - 1
Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 2934791902880
at: _debug_messenger_callback (drivers\vulkan\vulkan_context.cpp:157)
WARNING: GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
ReadDataFilesInRegistry: Registry lookup failed to get layer manifest files.
Objects - 1
Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 2934791902880
at: _debug_messenger_callback (drivers\vulkan\vulkan_context.cpp:154)
Using Vulkan Device #0: NVIDIA - NVIDIA GeForce RTX 2080
- Vulkan multiview supported:
max view count: 32
max instances: 134217727
- Vulkan subgroup:
size: 32
stages: STAGE_VERTEX, STAGE_TESSELLATION_CONTROL, STAGE_TESSELLATION_EVALUATION, STAGE_GEOMETRY, STAGE_FRAGMENT, STAGE_COMPUTE, STAGE_RAYGEN_KHR, STAGE_ANY_HIT_KHR, STAGE_CLOSEST_HIT_KHR, STAGE_MISS_KHR, STAGE_INTERSECTION_KHR, STAGE_CALLABLE_KHR, STAGE_TASK_NV, STAGE_MESH_NV
supported ops: FEATURE_BASIC, FEATURE_VOTE, FEATURE_ARITHMETIC, FEATURE_BALLOT, FEATURE_SHUFFLE, FEATURE_SHUFFLE_RELATIVE, FEATURE_CLUSTERED, FEATURE_QUAD, FEATURE_PARTITIONED_NV
quad operations in all stages
Using present mode: VK_PRESENT_MODE_FIFO_KHR
Using "winink" pen tablet driver...
Log of the working MinGW debug build
Godot Engine v4.0.dev.custom_build.bdcc8741e - https://godotengine.org
Using "ICU / HarfBuzz / Graphite" text server...
Vulkan API 1.2.162
ERROR: GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
loader_get_json: Failed to open JSON file C:\ProgramData\GOG.com\Galaxy\redists\overlay\injected\galaxy_overlay_vklayer_x64.json
Objects - 1
Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 1835716830080
at: _debug_messenger_callback (drivers\vulkan\vulkan_context.cpp:157)
WARNING: GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
ReadDataFilesInRegistry: Registry lookup failed to get layer manifest files.
Objects - 1
Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 1835716830080
at: _debug_messenger_callback (drivers\vulkan\vulkan_context.cpp:154)
Using Vulkan Device #0: NVIDIA - NVIDIA GeForce RTX 2080
- Vulkan multiview supported:
max view count: 32
max instances: 134217727
- Vulkan subgroup:
size: 32
stages: STAGE_VERTEX, STAGE_TESSELLATION_CONTROL, STAGE_TESSELLATION_EVALUATION, STAGE_GEOMETRY, STAGE_FRAGMENT, STAGE_COMPUTE, STAGE_RAYGEN_KHR, STAGE_ANY_HIT_KHR, STAGE_CLOSEST_HIT_KHR, STAGE_MISS_KHR, STAGE_INTERSECTION_KHR, STAGE_CALLABLE_KHR, STAGE_TASK_NV, STAGE_MESH_NV
supported ops: FEATURE_BASIC, FEATURE_VOTE, FEATURE_ARITHMETIC, FEATURE_BALLOT, FEATURE_SHUFFLE, FEATURE_SHUFFLE_RELATIVE, FEATURE_CLUSTERED, FEATURE_QUAD, FEATURE_PARTITIONED_NV
quad operations in all stages
Using present mode: VK_PRESENT_MODE_FIFO_KHR
Using "winink" pen tablet driver...
Shader 'VoxelGiSdfShaderRD' SHA256: b6972a55781e21cfbac0dfc035381ee1f65e972c1f7e0bcc6004bfb74cc9dc9e
Shader 'ParticlesShaderRD' SHA256: cbb6a5ce1933875a8e2df4f0cde4684286dedb6737cfbcfdeba745c978a2cc45
Shader 'ParticlesCopyShaderRD' SHA256: 6bd2ea6fca4c3a51f1e9bcd5068315e63820ec68fd8596e20480174c943bb9df
Shader 'CanvasSdfShaderRD' SHA256: 692328d90ee7d404503d304e2464af3f3ac466ebd287762fd8883248f3e2a046
Shader 'SkeletonShaderRD' SHA256: 8f03a2ebc895d833120c3b2341c16e971fb1404dd3ae9e8030bc4bbf63e67fd5
Shader 'CanvasShaderRD' SHA256: a30a00a53b17f4e413dfb1fe109868063f2d67900aaeec66b0f50964108e216b
Shader 'CanvasOcclusionShaderRD' SHA256: fd08d5c92b6537852fb4b8e5fba27f28518faad41a5c21ac9bd51bf49807824c
Shader 'ClusterRenderShaderRD' SHA256: a2924afb04064a1dc42ed9d63721e7d5dd0466f245037ff85b5bef8bd251f515
Shader 'ClusterStoreShaderRD' SHA256: 64c437e6bebba9faa4766b524281310d934fd2a28f26e9a8a7834ca07f09d2dc
Shader 'ClusterDebugShaderRD' SHA256: 0f66ac91d81e6f440ffa804d473579cb5d7ac135b375dce6d120186a65b54204
Shader 'SkyShaderRD' SHA256: dd0dbf614e400cc2f58360d292f85f19237fdd9746b9a46c6fd7c50fe8a2ae43
Shader 'VoxelGiShaderRD' SHA256: 51aa13996be8fb6071160d3ea73d752edcd67ef81e1aec1ec3623a13eee94d0a
Shader 'VoxelGiDebugShaderRD' SHA256: 7fa6be49a459f8891a6e756165f2e9ca7f3ba261f53edf36c2041dc2acbf2a96
Shader 'SdfgiPreprocessShaderRD' SHA256: 056a9e6eaae9cc98093ceec6142bcb4582be0f3443e18402c1e25c293899b104
Shader 'SdfgiDirectLightShaderRD' SHA256: 0707e534eb11aeb7566a94913fe168d96b66896b440650f768f92ec6ed5cc842
Shader 'SdfgiIntegrateShaderRD' SHA256: deacba90b0c075f22f4000295f6a0b846957d2602b5d54b631c089792288e7df
Shader 'GiShaderRD' SHA256: c1d65ee198e82b396759752f26b64ae98d7bb8691aed1bc07924617cf25872c5
Shader 'SdfgiDebugShaderRD' SHA256: 5e9f1e4d567be5c04dacec2ddbddb98638086a1fe7dc1d0260f3a58e93856a0e
Shader 'SdfgiDebugProbesShaderRD' SHA256: a6a825191c0869482f513376305eb79fa16189e65068bf76a484a0e1eb602c1b
Shader 'VolumetricFogShaderRD' SHA256: bc8fa35c7d92fd0353d5458af7d19438bf0c46f8c873b1b5e45f328f988f2d34
Shader 'SceneForwardClusteredShaderRD' SHA256: 300732714cbaf090328e341566d4d056b1bf83a2786d31de56778f6c929d1987
Shader 'CopyShaderRD' SHA256: 4ea51535b5cdcba7ece20ebd074657d05cbde47d1d0e87d3c84a2a1c28aa976f
Shader 'CopyToFbShaderRD' SHA256: 6972e6ea6ece204edb2197ac92f699ad4e7c4b22cac23422dd816a2a9fc11ce3
Shader 'CubemapRoughnessShaderRD' SHA256: 083d83f629fde69ed8e4c482638f30df40144b9e05c8d8243317b1866ff97897
Shader 'TonemapShaderRD' SHA256: cd4d7156689aa910864bd876e2c439c8f2b49dd281df707f93145fec99b9bd67
Shader 'LuminanceReduceShaderRD' SHA256: f8db4abaa10d52974aef5a0443e69f8838eb05f4b4eee120c19ca3b0f8ef6353
Shader 'CubeToDpShaderRD' SHA256: 1ac5646aaebfed0f2fa62c6faa2f87981b8a5acf4748191482448d7fe516cc78
Shader 'BokehDofShaderRD' SHA256: 7ae11ecc5f3863fff19a28c9e41a9ba8016c53f54cc182daa15e6a1e080ce3c6
Shader 'SsaoDownsampleShaderRD' SHA256: 4db35346ca9c82436e993cde3466bee499a8f66f6d58e39d9bed78ee7d835e1f
Shader 'SsaoShaderRD' SHA256: db7ede5f05458aaa5e03db5e62804bcd7e3b4b3eeb556e9eb027a958a96b6bd7
Shader 'SsaoImportanceMapShaderRD' SHA256: a2305d0dc9bed6da701e58b6d6a28bb2a664aedb9de4460c390b21a7f989e8eb
Shader 'SsaoBlurShaderRD' SHA256: 9158ebb0e50201b64d1c1009317e7cf5de0792a82a4d06b393e5db5ec351b54f
Shader 'SsaoInterleaveShaderRD' SHA256: e8d5eda46653d75788db067517688755630bcb40ff6da19c5f039db86df234c3
Shader 'RoughnessLimiterShaderRD' SHA256: fe30b9757ac0d3bbb53ab48bc7e0d48cc760f8212fc723c588c30dc9466539d4
Shader 'CubemapDownsamplerShaderRD' SHA256: 8184770cb634e8d5b7f7d4e801b864caa2972a7a98181a62c196ffa7b882edc3
Shader 'CubemapFilterShaderRD' SHA256: 5c27e7fe5c0fce79c147c7a8ae6984a0a7ae2dd90af07e35a4a34934698a91ca
Shader 'SpecularMergeShaderRD' SHA256: f6dc7afac1a2715cc2f134daff1bddfd6b7ca715d655aca8cafa32798395dec2
Shader 'ScreenSpaceReflectionShaderRD' SHA256: d56b77268d5c239c3886ca85201d733c47ea7e6542969c5eb6d5350b483778ef
Shader 'ScreenSpaceReflectionFilterShaderRD' SHA256: 4cd6afe65db3d97e059daa77702faf2a6e6cccb9c552c61af6055cb5be448b5f
Shader 'ScreenSpaceReflectionScaleShaderRD' SHA256: 89c672345ad85d25a8e9cbc814aac4007cb5fcb29cb8262d0185abd23fdccb9f
Shader 'SubsurfaceScatteringShaderRD' SHA256: c633c5beb82a7102c2849bec06732e4c6c98ebfc17b013f0844089d30beaef52
Shader 'ResolveShaderRD' SHA256: d0c9281d4f856dc19c5c6d14e0b256b28ca5455371db93744c010476572dbf25
Shader 'SortShaderRD' SHA256: 2192a5f1b643af9d53a2a609c7d2a715237ea16c083317a1611af42c6845b99c
Shader 'BlitShaderRD' SHA256: 3c3477999355f834dd2a898d94df5b40ae97299bb1f5e9de2dd2b6caf46055f5
WASAPI: wFormatTag = 65534
WASAPI: nChannels = 8
WASAPI: nSamplesPerSec = 48000
WASAPI: nAvgBytesPerSec = 1536000
WASAPI: nBlockAlign = 32
WASAPI: wBitsPerSample = 32
WASAPI: cbSize = 22
WASAPI: detected 8 channels
WASAPI: audio buffer frames: 1962 calculated latency: 44ms
ERROR: Attempted to free invalid ID: 2100239007746
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
CORE API HASH: 12701809910231588744
EDITOR API HASH: 7657489752925521230
Using present mode: VK_PRESENT_MODE_FIFO_KHR
Loaded builtin certs
EditorSettings: Save OK!
Using present mode: VK_PRESENT_MODE_FIFO_KHR
Using present mode: VK_PRESENT_MODE_FIFO_KHR
Using present mode: VK_PRESENT_MODE_FIFO_KHR
EditorSettings: Save OK!
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: Attempted to free invalid ID: 0
at: _free_internal (drivers\vulkan\rendering_device_vulkan.cpp:8313)
ERROR: 1 RID allocations of type 'N17RendererStorageRD7TextureE' were leaked at exit.
WARNING: 2 RIDs of type "Texture" were leaked.
at: finalize (drivers\vulkan\rendering_device_vulkan.cpp:9055)
StringName: 173 unclaimed string names at exit.
Summary of tried combinations and result of it:
| Distribution | MinGW version | GCC version | binutils version | Works |
|---|---|---|---|---|
| Fedora 34 | 8.0.0 | 10.3.1 | 2.34 | yes |
| Fedora 35 | 9.0.0 | 11.2.1 | 2.37 | no |
| Debian Sid | 8.0.0 | 10.2.1 | 2.37 | no |
| OpenSuse TW | 9.0.0 | 9.2.0 | 2.33 | yes |
| MSYS2 | 9.0.0.6346.6cc97775a-1 | 11.2.0 | 2.37 | no |
| Mageia 8 | 8.0.0 | 10.2.1 | 2.34 | yes[1] |
| MSYS2 | 9.0.0.6454.b4445ee52 | 11.2.0 | 2.38 | no |
| MSYS2 | 10.0.0.r0.gaa08f56da-1 | 11.2.0 | 2.38 | no |
[1] The icon from the executable was missing
Steps to reproduce
For a not working build:
scons platform=windows target=release_debug use_mingw=yes -j12
For a working build:
scons platform=windows target=debug use_mingw=yes -j12
Minimal reproduction project
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 28 (28 by maintainers)
For the record, I now think that this issue might have been a GCC regression in 11.2 (maybe 11.1 too), which got fixed in later versions. Because that seems to be the only significant between the mingw-headers, mingw-gcc and mingw-binutils package of Fedora 35 and Fedora 36.
So I suspect this was found and fixed in the 11.x stable branch after 11.2 was released. There was a 11.3 release earlier this year which should also have the fix I suppose.
TL;DR: Avoid GCC 11.1 or 11.2, unless it’s a snapshot more recent than the July 2021 11.2 release. GCC 11.3 or 12.1+ should be fine.
So after spending some time chasing the wrong issue in #61176, I finally realized that Mono builds for Godot 3.x seem to be affected by the same issue you describe. Windows binaries built on Fedora 35 are crashing, while Fedora 36 is fine.
Thanks a lot for all the debugging you’ve done so far and documenting your findings. I’m still not sure what we can do to get this actually fixed in the upstream MinGW distributions, but at least we can make sure to use Fedora 36+ / skip Fedora 35 for official builds. Since the bug is fixed in recent toolchains, we can assume that upstream projects did find the regression and got it fixed in the relevant affected projects (but that didn’t get backported all the way to past releases).
For the record, I did a couple test builds to confirm that I get a crash with Fedora 35 and a working binary with Fedora 36 in the current
masterbranch:With the release of the alpha 8 I noticed that the Windows binaries works, although it was compiled on Fedora 36 which has binutils 2.37. This is not in line with the previous assumptions/findings, so I tested again on various distributions.
11.2.0-1: Not works12.1.0-1: WorksI have looked through the patches of MSYS and the only patch could be related is this, but it was released on 2021 Jul 28, so it is included in all of my test with MSYS (therefore not related). Another thing is the whole binutils package haven’t been touched since 2022 Mar 1, the build didn’t work on 2022 Apr 20, but it magically started working today (I tested it today, so I do not know when precisely).
Arch currently have
mingw-w64-gccwith version11.2.0-1inCommunityand have12.1.0-1inCommunity-Testing. If I compiled and installed the newer gcc fromCommunity-Testing, the issue went away.I think this issue is neither binutils’ nor gcc’s solely fault, rather something of a compatibility issue between them, which seems to be resolving itself as the distributions releasing the newer packages.
@akien-mga I have some good news! I have bisected the binutils and found out this is the first bad commit: https://sourceware.org/git/?p=binutils-gdb.git;h=514b4e191d5f46de8e142fe216e677a35fa9c4bb It was first included in 2.36.
I found a way* to make it work. (* two ways. See follow-up comment) Compile binutils with reverted
NT_EXE_IMAGE_BASEand use link flags to get correct result. Scons command:Patch here: MmAaXx500/binutils-gdb-godot@b02b5836d58c1fcef7179fff91ac4e59ca432d03
Below, some of my inexperienced observations:
The
--disable-high-entropy-vais redundant because the--disable-dynamicbasedisables this as well. (this is a fact, not an observation: source)The flags are required because the above-mentioned commit changes the default dll characteristics from
0to enable dynamic base, high entropy va and nx compat. The nx compat flag left out from the command on purpose because it does not affect the binary in a way related to this issue.I think this issue is related somehow to the (dynamic) base address, because if I do not revert the
NT_EXE_IMAGE_BASEand disable the dynamic base or do the revert but do not disable the dynamic base the result is the same: this issueI’d like to note that I don’t have much knowledge in this area, so I cannot determine who is “at fault” or what exactly causing this issue. So, I highly appreciate if someone can dig deeper in that and found out the root cause of the issue.
Another workaround without modifying binutils. Just pass the base address as a parameter.
So yeah binutils sounds like a likely culprit. I’ll see if I can use Fedora 35’s mingw-binutils on Fedora 34 to see if that produces a crashing binary. And then we’d have to try to bisect that with intermediate releases and report the issue upstream, though we’d need to identify a bit more precisely how binutils relates to this (thread-related?) crash.
MSYS2 have 2.37
I think I found something. F34 uses 8.0.0 and F35 uses 9.0.0 mingw (assuming
mingw64-headerspackage version reflects mingw version). But, in order to make the situation less clear, OpenSuse TW which uses 9.0.0 mingw (mingw64-headers) it runs fine, but Debian uses 8.0.0 mingw (mingw-w64-common) and not works.Just to summarize, here is a table: (at mingw version, I’m assuming package version reflects this)
(after I wrote the table, I’m not sure about I really found something)
Tried with Fedora 34, it’s works as expected, but with Fedora 35 I can reproduce this issue. (so do not update the official build server 😄) Fedora 34 has gcc 10.3.1 Fedora 35 has gcc 11.2.1
Fedora 34 gcc -v
Fedora 35 gcc -v