godot: Very slow project/editor opening in Godot 4

Godot version: 4.0.dev.custom_build. 494bf38c7 GLES 2 - tested 14.01.2020

OS/device including version: Ubuntu 20.04 Nvidia 970 440.83, i7 4770 4/8 Hotspot -v1.3.0-27-g61f1314

Issue description: Opening and closing empty project in editor with command godot -e -q take a lot of more time than previous versions.

2020 11 results

RESULTS FROM 06.11.2020

Version	Opening Time	Memory usage in editor
Godot 3.2	4.2s	135 MB
Godot 4.0 GLES 2	8.2s	403 MB
Godot 4.0 (494bf38c7)	13.2s	340 MB
Godot 3.2 + Sanitizers	13.1s	-
Godot 4.0 (494bf38c7) + Sanitizers	41s	-

25.07.2020 results

RESULTS FROM 25.07.2020 |Version|Opening Time| |-|-| |Godot 3.4 (6d58ea6ce)| 5.2s | |Godot 3.4 (6d58ea6ce) + Sanitizers| 13.7s | |Godot 4.0 (b918c4c3c) - first run | 13.9s | |Godot 4.0 (b918c4c3c) - second run | 10.4s | |Godot 4.0 (b918c4c3c) + Sanitizers - first run| 53,4s | |Godot 4.0 (b918c4c3c) + Sanitizers - second run| 33s |

RESULTS FROM 28.11.2022 - opening and closing empty project in editor

Dev debug builds

Version	Opening Time
Godot 3.6 (6d58ea6ce) GLES3	5.2s
Godot 3.6 (6d58ea6ce) GLES3 + Sanitizers	13.7s
Godot 4.0 (71f5bb391) Vulkan - first run	47.4s
Godot 4.0 (71f5bb391) Vulkan - second run	27.4s
Godot 4.0 (71f5bb391) Headless	23s
Godot 4.0 (71f5bb391) Vulkan + Sanitizers - first run	129s
Godot 4.0 (71f5bb391) Vulkan + Sanitizers - second run	110s

optimized builds(downloaded from official site)

Version	Opening Time
Godot 3.5.1 GLES3	7.7s
Godot 4.0 beta 6 Vulkan	10.8s
Godot 4.0 beta 6 Headless	8.2s

This makes it very difficult to test and develop the engine, especially when you start it with Address Sanitizer for example.

Most time spends on compiling shaders, which seems for me that partially can be avoidable by caching it or even compiling it when compiling Godot(not sure if this is possible).

Running Valgrind in one small project take almost 10 minutes

2020 11 results

CPU Cycles in Godot 3.2 Zrzut ekranu z 2020-11-06 09-58-37

CPU Cycles in Godot 4.0 Zrzut ekranu z 2020-11-06 09-56-46

CPU Cycles in Godot 4.0 GLES 2 Zrzut ekranu z 2021-01-14 14-52-31

Graphs Godot 4 Beta 6 Dev Hadless:

Screenshot from 2022-11-28 08-56-29

Godot 4 Beta 6 Dev Vulkan Screenshot from 2022-11-28 09-19-58

Godot 3.6 - 6d58ea6ce GLES 3 Screenshot from 2022-11-28 09-30-51

Steps to reproduce: To get perf data just execute this command(path must be changed)

perf record -o /home/rafal/perf.data --call-graph dwarf --aio --sample-cpu /usr/bin/godot4 -e -q

And then open it in latest version of Hotspot- https://github.com/KDAB/hotspot/releases

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 25
Comments: 27 (22 by maintainers)

Commits related to this issue

mbedtls: Backport PR 4819 to fix certs parsing speed Fixes engine startup time regression described here: https://github.com/godotengine/godot/issues/43351#issuecomment-847466783 — committed to akien-mga/godot by akien-mga 3 years ago
mbedtls: Backport PR 4819 to fix certs parsing speed Fixes engine startup time regression described here: https://github.com/godotengine/godot/issues/43351#issuecomment-847466783 (cherry picked from... — committed to godotengine/godot by akien-mga 3 years ago
mbedtls: Backport PR 4819 to fix certs parsing speed Fixes engine startup time regression described here: https://github.com/godotengine/godot/issues/43351#issuecomment-847466783 (cherry picked from... — committed to sairam4123/godot by akien-mga 3 years ago
mbedtls: Backport PR 4819 to fix certs parsing speed Fixes engine startup time regression described here: https://github.com/godotengine/godot/issues/43351#issuecomment-847466783 (cherry picked from... — committed to KoderaSoftwareUnlimited/godot by akien-mga 3 years ago
mbedtls: Backport PR 4819 to fix certs parsing speed Fixes engine startup time regression described here: https://github.com/godotengine/godot/issues/43351#issuecomment-847466783 (cherry picked from... — committed to akien-mga/godot by akien-mga 3 years ago

Most upvoted comments

Not sure if I should be putting this here or in #62322 but if I’m understanding this correctly and used the Instruments software properly (I really don’t know what I’m doing) it looks like it’s just doing a whole lot of nothing. Macbook pro late 2013. It takes around 45 seconds to launch the project list, another 45 to launch the editor, and 45 more when I run the project to test. This could be entirely unrelated but I hope this helps because launcher v3.4.4 only takes 2 seconds max. Screen Shot 2022-08-08 at 8 20 18 pm edit I know it’s alpha not beta that’s my bad

ffernn-dev on Aug 8, 2022

Pointing out something I noticed the past few days… When I run my project standalone (e.g. --path my/project) it starts in like a second. However, if I run another instance right after, it takes like 5-6 seconds to start. When using the editor, the project seems to always take 5-6 seconds. There could be a bug with the shader caching with multiple instances.

Edit: Reported at https://github.com/godotengine/godot/issues/63419

nathanfranke on Jul 25, 2022

There is a slight increase in start-up time for 4.0 optimized compared to 3.x optimized, but it looks like the main regression is the debug (unoptimized) start-up times.

3.4 tests done on v3.4.beta.custom_build [8d0b2edfc] 4.0 tests done on v4.0.dev.custom_build [95432893e]

Task	3.4 Debug	3.4 Release	4.0 Debug	4.0 Release
Project Manager (Initial)	2.201 s ± 0.029 s	1.491 s ± 0.009 s	4.131 s ± 0.023 s	1.558 s ± 0.053 s
Project Manager (Subsequent)	2.222 s ± 0.037 s	1.505 s ± 0.062 s	2.112 s ± 0.045 s	1.109 s ± 0.065 s
Editor (Initial)	6.417 s ± 0.045 s	3.751 s ± 0.013 s	13.195 s ± 0.372 s	5.269 s ± 0.462 s
Editor (Subsequent)	6.411 s ± 0.119 s	3.777 s ± 0.092 s	8.222 s ± 0.426 s	4.444 s ± 0.322 s
Empty Project	2.164 s ± 0.135 s	1.497 s ± 0.106 s	2.121 s ± 0.317 s	1.599 s ± 0.091 s
TPS Demo*	4.127 s ± 0.373 s	11.882 s ± 0.332 s

* Time is aggregated from these steps: Load Main Menu, Load Main Scene, Wait 3 Frames, Exit Game.

nathanfranke on Oct 8, 2021

I ran a profiler with https://github.com/godotengine/godot/pull/49050 merged locally. The project manager was opened with the --quit command line argument, which makes Godot exit after one frame was rendered:

Project manager with no projects in the list

Linux perf file (can be extracted and opened in HotSpot): perf.data.zip

Top-down with cycle-intensive methods unfolded

Flame graph

Caller/callee

With a few dozen projects in the project list

Linux perf file (can be extracted and opened in HotSpot): perf.data.zip

Top-down with cycle-intensive methods unfolded

Flame graph

It looks like mbedTLS of all things is causing the slowness, which is surprising at first. cc @Faless

Calinou on Jul 20, 2021

#48978 (comment) shows the same for 3.x yeah. Earlier comparisons in this issue did not, but that must have been introduced in 3.x with the backport of the crypto stuff.

Good catch! I’ll investigate.

Faless on May 25, 2021

Does it also happen in 3.x? The code should be the same as far as I recall.

https://github.com/godotengine/godot/issues/48978#issuecomment-846599528 shows the same for 3.x yeah. Earlier comparisons in this issue did not, but that must have been introduced in 3.x with the backport of the crypto stuff.

That would likely best be tracked in a separate issue for 3.x + 4.0.

akien-mga on May 25, 2021

From my perspective, now I don’t check whether the bug is reproducible on master any more when submitting a bug report because creating a test project and edit means double the long wait time (project manager + editor).

Use this command to create a new project in a temporary folder, bypassing the project manager:

# Assuming you're in the root folder of a Godot Git clone:
mkdir -p /tmp/a
touch /tmp/a/project.godot
bin/godot.x11.tools.64 /tmp/a/project.godot

This works with any Godot version. You can save a script like this one in your PATH to simplify this to a single command.

Calinou on Apr 2, 2021