godot: Godot 4.0 is very slow to start on macOS compared to 3.x

I ran benchmarks of all versions since Godot 3.0.1 up to 4.0.alpha10:

Specifications

OS: Fedora 36 CPU: Intel Core i7-6700K @ 4.4 GHz RAM: 2×16 GB DDR4-3000 (dual channel) GPU: NVIDIA GeForce GTX 1080 (NVIDIA 510.68.02) SSD: Samsung 850 EVO 1 TB

Using official Godot Linux x86_64 release binaries downloaded from TuxFamily. An average of 25 runs is performed.

Benchmark source and data files (CSV, JSON, Markdown) are available at: https://github.com/Calinou/godot-startup-times

Results

Cold project manager start + exit

editor_data/ and project folder are removed before every run.

Warm project manager start + exit

Cold editor start + exit

editor_data/ and project folder are removed before every run.

Warm editor start + exit

Conclusion

The main takeaway is that 4.0.alpha10 is much slower to startup and shutdown compared to 4.0.alpha9, especially in cold runs. This suggests that the shader cache has more work to do since more shader variants have to be compiled, presumably due to TAA. cc @JFonS

4.0.alpha6 also has a noticeable regression in startup/shutdown speed compared to 4.0.alpha5 (both cold and warm), but mainly in the project manager.

I have this problem as well and looked into it a bit. I don’t have a solution but my findings so far is that one call to vkCreateGraphicsPipelines (for the shader ClusterRenderShaderRD:0) takes 30s. What happens is that the separate MTLCompilerService process crashes, and the caller times out & continues after 10 seconds. (It does 3 retries however, hence a 30s total delay). So it doesn’t seem to be a shader cache performance issue per se like @Calinou investigated, but a bug, possibly in the MoltenVK/Metal backend.

As for why it’s crashing is a bit more unclear. The callstack of the crashed thread in MTLCompilerService is:

0   libsystem_kernel.dylib        	    0x7ff819438dba __abort_with_payload + 10
1   libsystem_kernel.dylib        	    0x7ff81943a877 abort_with_payload_wrapper_internal + 80
2   libsystem_kernel.dylib        	    0x7ff81943a827 abort_with_reason + 19
3   MTLCompiler                   	    0x7ffa32221d4f fatalErrorHandler(void*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) + 635
4   libLLVM.dylib                 	    0x7ffa31e86986 llvm::report_fatal_error(llvm::Twine const&, bool) + 323
5   libLLVM.dylib                 	    0x7ffa31e869c0 llvm::report_fatal_error(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) + 29
6   libIGIL-Metal.dylib           	    0x7ffa295242dd IGILErrorMessageHandlers::reportUnsupportedFunctionReferenceError(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 42
7   libIGIL-Metal.dylib           	    0x7ffa2954532f (anonymous namespace)::BiFInliner::runOnModule(llvm::Module&) + 3877
8   libLLVM.dylib                 	    0x7ffa31af3c15 llvm::legacy::PassManagerImpl::run(llvm::Module&) + 559
9   libMTLIntelCompilerPlugin.dylib	       0x10291acb9 MTLIntelCompiler::generateIGIL(llvm::Module*, MTLIntelFunctionType, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) + 669
10  libMTLIntelCompilerPlugin.dylib	       0x10290fa3b MTLCompilerBuildRequestWithOptions + 523
11  MTLCompiler                   	    0x7ffa3221276a MTLCompilerPluginInterface::compilerBuildRequest(bool, unsigned int, void const*, unsigned long, unsigned int, void const*, BackendCompilationOutput&) + 276
12  MTLCompiler                   	    0x7ffa3220cd11 MTLCompilerObject::backendCompileModule(MTLCompilerObject::BinaryRequestData&, BackendCompilationOutput&, unsigned long long, std::__1::vector<CompileTimeData, std::__1::allocator<CompileTimeData> >&) + 159
13  MTLCompiler                   	    0x7ffa322129f7 MTLCompilerObject::backendCompileExecutableRequest(MTLCompilerObject::BinaryRequestData&) + 517
14  MTLCompiler                   	    0x7ffa32221658 MTLCompilerObject::buildRequest(unsigned int, unsigned int, void const*, unsigned long, void (unsigned int, void const*, unsigned long, char const*) block_pointer) + 258
15  MTLCompiler                   	    0x7ffa321f822d split_stack_call + 13
16  MTLCompiler                   	    0x7ffa32212cef MTLCodeGenServiceBuildRequest + 265
17  MTLCompilerService            	       0x1026d57e3 invocation function for block in MTLCompilerServiceHandleEvent(NSObject<OS_xpc_object>*) + 793
18  libxpc.dylib                  	    0x7ff819191b70 _xpc_connection_call_event_handler + 56
19  libxpc.dylib                  	    0x7ff819190956 _xpc_connection_mach_event + 1413
20  libdispatch.dylib             	    0x7ff81929b3b1 _dispatch_client_callout4 + 9
21  libdispatch.dylib             	    0x7ff8192b4041 _dispatch_mach_msg_invoke + 445
22  libdispatch.dylib             	    0x7ff8192a11cd _dispatch_lane_serial_drain + 342
23  libdispatch.dylib             	    0x7ff8192b4b77 _dispatch_mach_invoke + 484
24  libdispatch.dylib             	    0x7ff8192a11cd _dispatch_lane_serial_drain + 342
25  libdispatch.dylib             	    0x7ff8192a1dfd _dispatch_lane_invoke + 366
26  libdispatch.dylib             	    0x7ff8192abeee _dispatch_workloop_worker_thread + 753
27  libsystem_pthread.dylib       	    0x7ff81944efd0 _pthread_wqthread + 326
28  libsystem_pthread.dylib       	    0x7ff81944df57 start_wqthread + 15

It seems to me like it encounters an error, but fails to properly report it and crashes instead. reportUnsupportedFunctionReferenceError is seems like the biggest hint to to the cause to me. I have not been able to attach to the process with a debugger to try to extract more info. (The process is a bit transient.)

While compiling it also emits [MTLCompiler] Error: undefined reference to `air_simd_is_helper_thread()' to the system log. This happens more times than for the failing call though so I’m not sure it’s related.

I don’t know if there are any more Vulkan/Metal logs to check, but let me know if there’s some place I should have a peek.

I think there needs to be a new issue since a lot of people seem to be reporting on other issues some graphic driver related slowdowns. I think the issue is from Vulkan implementation in Godot that is causing slowdowns probably because there is something that’s not being properly initialized and the application waits for it?

This is already being tracked in https://github.com/godotengine/godot/issues/43351. In general, Vulkan is expected to be slower to initialize compared to OpenGL – it’s a more complex rendering backend.

Are there any useful benchmarking scenes readily available somewhere?

You can use https://github.com/Calinou/godot-reflection for this purpose 🙂

I’ve tested it and can confirm it works on 4.0.beta3.

Ok so I did a bit of a deep dive on this one, and basically yes the Metal compiler crashes and [MTLCompiler] Error: undefined reference to air_simd_is_helper_thread()` is the closest to a detailed error there is. Unfortunately this is in internal proprietary code and not much to work with so that’s a bit of a dead end.

However, with a bit of testing I figured out that what actually triggers this error is the usage of gl_HelperInvocation in cluster_render.glsl:

https://github.com/godotengine/godot/blob/c660cc4adc7032c11f74dfce5fcb7b5a02f6d097/servers/rendering/renderer_rd/shaders/cluster_render.glsl#L164

In my case it’s taking the non-USE_SUBGROUPSpath and removing those two conditionals (i.e. running the atomicOr always) fixes the compiler failure and Godot starts fast.

I’m not sure making a pull request of that makes sense because those conditionals are obviously there for a reason, and I’m not well versed enough in the shaders to know what the best alternative is. (As I understand it they’re there to avoid doing unnecessary calculations when the shader is run in a helper mode that doesn’t actually output anything, so removing it likely doesn’t cause any errors but may have performance effects. If I read the docs correctly atomicOr doesn’t actually do anything when in a helper invocation though so possibly it’s just basically a free NOP anyway, but someone who actually knows what they’re talking about should chime in.)

As an extra reference, this specific code is actually discussed in a separate but related discussion about subgroups + helper invocations in a Khronos GLSL issue, mentioning @reduz : https://github.com/KhronosGroup/GLSL/issues/35

I’d be happy to test alternative fixes on this computer where it normally fails, just let me know.

@kdada This is an unrelated issue. Also, your project is using the Forward Mobile backend, which is more compatible with lower-end desktop GPUs (including old Macs and MoltenVK in general). On the other hand, the project manager always uses the Forward Plus backend, even on GPUs that don’t support it well.

The project manager should probably always use the Forward Mobile backend (or Compatibility on GPUs that don’t support Vulkan), as there is no point in using the Forward Plus backend for it to my knowledge.

There’s a fair chance this issue is a MoltenVK bug, which we unfortunately can’t fix on our end.

Yes, unfortunately. It’s maybe possible something in the shader triggers the error and can be avoided by adjusting it somehow, but the error reports I currently have available aren’t that much to work with.

For me this is around 30 seconds usually on 4.0. Before on 3.3 it launched under a couple of seconds.

I’m on Windows and I get close to 10 seconds of boot time significantly slower than 3.x. Also the entire experience is very sluggish until I turn off multi window mode then is fine.