godot: Editor crashes on project load: "pure virtual method called"

Godot version

4.0 dev d11117d9d8ae32d6acc025a7495d4ae58b9c37b4

System information

Arch Linux, KDE Plasma, Radeon RX6800XT, amdgpu driver, Vulkan

Issue description

For more than a week now I have been unable to open a project I’m working on.

Godot editor crashes while loading the project. I have bisected by project commits to maybe see what in my project has triggered this engine bug, but what I found doesn’t seem to make any sense, and when I tried it again to checkout the last known good commit - it crashes as well, so I don’t think this will be helpful unfortunately.

What’s interesting is if the assets are already imported, the game will run but not load in the editor. If the assets are not imported, the editor crashes before any of them are - the .godot/imported/ folder stays empty.

I’ve been testing Calinou’s nightly builds every couple of days, hoping this will go away. The crashes didn’t stop, but the crash messages are changing here and there.

I’ve been asking for help with this issue on Godot Contributors Chat. I have tried clearing all possible cache - it’s one of the first things I try in such cases. I have tried removing recently created shader assets from my project since shader compilation errors were present as well. I have tried switching to single-threaded rendering, following a suggestion from @RandomShaper . I compiled Godot myself with a a patch from @bruvzg and I thought it solved the problem, but it was a just fluke.

Before finally posting here I have just compiled Godot 4.0-dev from source and tested to confirm this has not been fixed. Here’s the current backtrace:

[Thread 0x7fffa8ffd6c0 (LWP 1677949) exited]

Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:373
#2  0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:404
#3  0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f1f90, p1=...) at servers/rendering/rendering_server_default.h:916
#4  0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#5  0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619f0288, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#6  0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619f0288, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#7  0x000055555b486125 in ShaderCompiler::compile (this=0x5555619f0288, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffc7a0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#8  0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557bb35710, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#9  0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613b9780, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#10 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb81d1b20) at ./core/templates/command_queue_mt.h:322
#11 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:373
#12 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f21d0) at ./core/templates/command_queue_mt.h:404
#13 0x000055555b44103d in RenderingServerDefault::free (this=0x5555612f1f90, p_rid=...) at servers/rendering/rendering_server_default.h:931
#14 0x000055555a0127db in CanvasItem::~CanvasItem (this=0x5555755bb370, __in_chrg=<optimized out>) at scene/main/canvas_item.cpp:1183
#15 0x00005555582dd348 in Control::~Control (this=0x5555755bb370, __in_chrg=<optimized out>) at ./scene/gui/control.h:45
#16 0x000055555a337278 in Range::~Range (this=0x5555755bb370, __in_chrg=<optimized out>) at scene/gui/range.cpp:335
#17 0x000055555a332de4 in ProgressBar::~ProgressBar (this=0x5555755bb370, __in_chrg=<optimized out>) at scene/gui/progress_bar.h:36
#18 0x0000555557f2033f in memdelete<Node> (p_class=0x5555755bb370) at ./core/os/memory.h:109
--Type <RET> for more, q to quit, c to continue without paging--
#19 0x000055555a06ace7 in Node::_notification (this=0x5555755bbbb0, p_notification=1) at scene/main/node.cpp:169
#20 0x0000555557c1345c in Node::_notificationv (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at scene/main/node.h:45
#21 0x00005555582db2ad in CanvasItem::_notificationv (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at ./scene/main/canvas_item.h:45
#22 0x00005555582db9e9 in Control::_notificationv (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at scene/gui/control.h:46
#23 0x000055555c0b7bbf in Object::notification (this=0x5555755bbbb0, p_notification=1, p_reversed=true) at core/object/object.cpp:735
#24 0x000055555c0b56bd in Object::_predelete (this=0x5555755bbbb0) at core/object/object.cpp:195
#25 0x000055555c0c1df3 in predelete_handler (p_object=0x5555755bbbb0) at core/object/object.cpp:1777
#26 0x0000555557f2031f in memdelete<Node> (p_class=0x5555755bbbb0) at ./core/os/memory.h:105
#27 0x000055555a06ace7 in Node::_notification (this=0x555575718480, p_notification=1) at scene/main/node.cpp:169
#28 0x0000555557c1345c in Node::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at scene/main/node.h:45
#29 0x00005555582db2ad in CanvasItem::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/main/canvas_item.h:45
#30 0x00005555582db9e9 in Control::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at scene/gui/control.h:46
#31 0x00005555582dc239 in Container::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/gui/container.h:37
#32 0x00005555582dc975 in BoxContainer::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/gui/box_container.h:37
#33 0x00005555582dd263 in HBoxContainer::_notificationv (this=0x555575718480, p_notification=1, p_reversed=true) at ./scene/gui/box_container.h:72
#34 0x000055555c0b7bbf in Object::notification (this=0x555575718480, p_notification=1, p_reversed=true) at core/object/object.cpp:735
#35 0x000055555c0b56bd in Object::_predelete (this=0x555575718480) at core/object/object.cpp:195
#36 0x000055555c0c1df3 in predelete_handler (p_object=0x555575718480) at core/object/object.cpp:1777
#37 0x000055555922dbe3 in memdelete<HBoxContainer> (p_class=0x555575718480) at ./core/os/memory.h:105
--Type <RET> for more, q to quit, c to continue without paging--c
#38 0x000055555922a9b0 in BackgroundProgress::_end_task (this=0x555569a357d0, p_task=...) at editor/progress_dialog.cpp:94
#39 0x0000555557ad1d21 in call_with_variant_args_helper<__UnexistingClass, String const&, 0ul> (p_instance=0x555569a357d0, p_method=(void (__UnexistingClass::*)(__UnexistingClass * const, const String &)) 0x55555922a8d2 <BackgroundProgress::_end_task(String const&)>, p_args=0x7fffffffd2a0, r_error=...) at ./core/variant/binder_common.h:262
#40 0x0000555557ad190e in call_with_variant_args_dv<__UnexistingClass, String const&> (p_instance=0x555569a357d0, p_method=(void (__UnexistingClass::*)(__UnexistingClass * const, const String &)) 0x55555922a8d2 <BackgroundProgress::_end_task(String const&)>, p_args=0x7fffffffd4b0, p_argcount=1, r_error=..., default_values=...) at ./core/variant/binder_common.h:409
#41 0x0000555557ad14c2 in MethodBindT<String const&>::call (this=0x555569a36410, p_object=0x555569a357d0, p_args=0x7fffffffd4b0, p_arg_count=1, r_error=...) at ./core/object/method_bind.h:320
#42 0x000055555c0b77fb in Object::callp (this=0x555569a357d0, p_method=..., p_args=0x7fffffffd4b0, p_argcount=1, r_error=...) at core/object/object.cpp:678
#43 0x000055555bd77c9c in Callable::callp (this=0x7fffe62b8038, p_arguments=0x7fffffffd4b0, p_argcount=1, r_return_value=..., r_call_error=...) at core/variant/callable.cpp:62
#44 0x000055555c0b19a0 in MessageQueue::_call_function (this=0x5555602b2880, p_callable=..., p_args=0x7fffe62b8050, p_argcount=1, p_show_error=false) at core/object/message_queue.cpp:229
#45 0x000055555c0b1cef in MessageQueue::flush (this=0x5555602b2880) at core/object/message_queue.cpp:275
#46 0x000055555a0a53aa in SceneTree::physics_process (this=0x555563bda020, p_time=0.016666666666666666) at scene/main/scene_tree.cpp:420
#47 0x0000555557ab1dea in Main::iteration () at main/main.cpp:2855
#48 0x0000555557a5a87b in OS_LinuxBSD::run (this=0x7fffffffd840) at platform/linuxbsd/os_linuxbsd.cpp:537
#49 0x0000555557a55250 in main (argc=4, argv=0x7fffffffdd68) at platform/linuxbsd/godot_linuxbsd.cpp:72

I’ve noticed there’s a mention of instance shader uniforms, so I commented out a line in the codebase that uses that.

Now the crash occurs exactly the same way, but the output is a bit different:

Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x000055555c10ddaa in StringName::StringName (this=<error reading variable: Cannot access memory at address 0x7fffff7feff8>, p_name=<error reading variable: Cannot access memory at address 0x7fffff7feff0>, p_static=<error reading variable: Cannot access memory at address 0x7fffff7fefec>) at core/string/string_name.cpp:200
200     StringName::StringName(const char *p_name, bool p_static) {
(gdb) bt
#0  0x000055555c10ddaa in StringName::StringName (this=<error reading variable: Cannot access memory at address 0x7fffff7feff8>, p_name=<error reading variable: Cannot access memory at address 0x7fffff7feff0>, p_static=<error reading variable: Cannot access memory at address 0x7fffff7fefec>)
    at core/string/string_name.cpp:200
#1  0x000055555983dd4d in ShaderLanguage::VaryingFunctionNames::VaryingFunctionNames (this=0x7fffff7ff348) at ./servers/rendering/shader_language.h:346
#2  0x000055555983dedb in ShaderLanguage::ShaderCompileInfo::ShaderCompileInfo (this=0x7fffff7ff300) at ./servers/rendering/shader_language.h:1113
#3  0x000055555b486073 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff7ff640, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1317
#4  0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#5  0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#6  0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#7  0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#8  0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#9  0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#10 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#11 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#12 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#13 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff800650, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#14 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#15 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#16 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#17 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#18 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#19 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#20 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#21 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#22 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#23 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff801660, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#24 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#25 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#26 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#27 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#28 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#29 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#30 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#31 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#32 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#33 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff802670, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#34 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#35 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#36 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#37 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#38 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#39 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
#40 0x000055555b486010 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#41 0x000055555b4b85ca in ShaderLanguage::_parse_shader (this=0x5555619a1418, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8292
#42 0x000055555b4c54b9 in ShaderLanguage::compile (this=0x5555619a1418, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9677
#43 0x000055555b486125 in ShaderCompiler::compile (this=0x5555619a1418, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff803680, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#44 0x000055555b5e3fc9 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x5555797c4090, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#45 0x000055555b61ebfa in RendererRD::MaterialStorage::shader_set_code (this=0x5555613bd0a0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#46 0x000055555b477c7f in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffb8336d48) at ./core/templates/command_queue_mt.h:322
#47 0x000055555b428e49 in CommandQueueMT::_flush (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:373
#48 0x000055555b428ede in CommandQueueMT::flush_if_pending (this=0x5555612f6150) at ./core/templates/command_queue_mt.h:404
#49 0x000055555b440cf6 in RenderingServerDefault::global_shader_uniform_get_type (this=0x5555612f5f10, p1=...) at servers/rendering/rendering_server_default.h:916
(...)

This goes on for 20k lines now. Here’s the full thing: full backtrace.zip

I’ve also tried doing an completely clean git repo clone, as in the past that has proven to “solve” some issues that nothing else helped with. How does that work - I have absolutely no idea, but it does. I guess sometimes a repo clone on my disk just gets cursed and I need to get a clean one, checkout where I was and keep going 😝

So this is where I am at the moment.

Steps to reproduce

If you’d like to test my project on your own (unfortunately it’s definitely not a minimal one - clean repo clone is 2.2 GB right now), here’s how:

  1. Clone my project repo: https://codeberg.org/unfa/Liblast
  2. Initialize LFS: git lfs install (it seems there is no need to do git lfs pull but it won’t hurt).
  3. Check out server-refactor branch
  4. Try to open the project located in Liblast/Game in the Godot editor.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 25 (15 by maintainers)

Most upvoted comments

My suggestion would be to start migrating away from using raw Object* (which will continue to have dangling pointer issues due to lack of any ownership semantics), and move into using something like a std::weak_ptr, which can be tested and converted into a strong reference before usage, ex before: (obj->callp) which will guarantee lifetime beyond the function call.

An async processed MessageQueue (godot’s main thread message queue) would be a good example of where something like std::weak_ptr is a good design choice for how the queue stores the actual object pointers in it’s internal callable queue. This would also make callable’s faster, as there would be no lookup cost and no global locking (there is a global spinlock in ObjectDB::get_instance).

Ahhh, ok. So digging a bit further, here is potential root source of error (still related to call_deferred)… (This is also why thread safety is tricky to get right).

So Callable doesn’t actually store (well in most cases) an Object*, but an Object ID. Which is OK as it does a lookup to retrieve the object by ID before actually jumping to the callable exec (see below screencap).

However, the lines between the object retrieval and the exec function on the object (line 52 and line 62), are not thread-safe. I.e. line 52 ObjectDB::get_instance(ObjectID(object)) can retrieve a valid Object* but the Object* could be freed by another thread before line 62 is executed. (We know this particular issue has object destruction on a different thread as reported by asan).

Screen Shot 2022-08-10 at 1 01 55 AM

The correct solution for this is not that trivial as it has implication for how Callable's work, and for how locking in the ObjectDB is done. But, the current design is not thread-safe. To make Callable’s thread-safe, they need to ensure the lifetime of the object extends past actually executing obj->callp (line 62).

So… I’m almost willing to bet, this is the cause (in texture.cpp):

Screen Shot 2022-08-10 at 12 13 31 AM

If this queue_update API is called on an object from ANY non-main thread, and the object is destructed before the MessageQueue on the main thread is flushed (call_deferred pushes that onto this queue for processing), guess what? Kablam…

I really have a problem with the call_deferred design, because it’s really hacky and easily leads to bugs like this which hinders the ability to do multithreaded things in the engine safely.

I’m not sure what the specific fix is for this issue, but overall call_deferred should probably warn or do something when it’s called from a non-main thread, or if it’s used from a Resource or any reference counted object (Ref<>), increment the reference count (delays object destruction), and decrement after callable is flushed from the main thread’s MessageQueue. It’s always dangerous pushing something like an Object* across threads into an deferred async processing queue which has no pointer ownership or sharing semantics… But will be up to engine maintainers to decide how to proceed with any overall design change.

Your asan report looks like a standard use after free, combined with multithreading issue and focus there… Use after free can continue on in certain circumstances and continue to cause more memory corruption which can lead to weird things happening, so start with fixing the use after free memory error reported by asan before debugging further with gdb in non-asan builds.

From your asan attached report: 1): The engine is allocing a GradientTexture1D ON THREAD T50 here: (This thread looks like it is a EditorFileSystem::scan thread).

#1 0x561ce6be6538 in Memory::alloc_static(unsigned long, bool) core/os/memory.cpp:75
    #2 0x561ce6be6456 in operator new(unsigned long, char const*) core/os/memory.cpp:40
    #3 0x561cde94889a in Object* ClassDB::creator<GradientTexture1D>() (/opt/godot-master/bin/godot.linuxbsd.opt.tools.64.san+0xa44389a)
    #4 0x561ce825cbbb in ClassDB::instantiate(StringName const&) core/object/class_db.cpp:338
    #5 0x561ce1af3537 in ResourceLoaderText::load() scene/resources/resource_format_text.cpp:565

2): It is being free on the same editor scan thread (T50) here:

    #1 0x561ce6be710e in Memory::free_static(void*, bool) core/os/memory.cpp:168
    #2 0x561cd5d4dfc2 in void memdelete<Texture2D>(Texture2D*) core/os/memory.h:112
    #3 0x561cd5d44347 in Ref<Texture2D>::unref() core/object/ref_counted.h:221
    #4 0x561cd5d38f79 in Ref<Texture2D>::~Ref() core/object/ref_counted.h:233
    #5 0x561ce19a53a3 in ParticlesMaterial::~ParticlesMaterial() scene/resources/particles_material.cpp:1867

3): And it is being used again on main thread (T0) here:

    #0 0x561ce203577f in GradientTexture1D::_update() scene/resources/texture.cpp:2240
    #1 0x561cd6213be7 in void call_with_variant_args_helper<__UnexistingClass>(__UnexistingClass*, void (__UnexistingClass::*)(), Variant const**, Callable::CallError&, IndexSequence<>) core/variant/binder_common.h:262
    #2 0x561cd6204492 in void call_with_variant_args_dv<__UnexistingClass>(__UnexistingClass*, void (__UnexistingClass::*)(), Variant const**, int, Callable::CallError&, Vector<Variant> const&) core/variant/binder_common.h:409
    #3 0x561cd61f0d7e in MethodBindT<>::call(Object*, Variant const**, int, Callable::CallError&) core/object/method_bind.h:320
    #4 0x561ce82cb5cd in Object::callp(StringName const&, Variant const**, int, Callable::CallError&) core/object/object.cpp:678
    #5 0x561ce74c5457 in Callable::callp(Variant const**, int, Variant&, Callable::CallError&) const core/variant/callable.cpp:62
    #6 0x561ce82ae53c in MessageQueue::_call_function(Callable const&, Variant const*, int, bool) core/object/message_queue.cpp:229
    #7 0x561ce82aef6b in MessageQueue::flush() core/object/message_queue.cpp:275

So some callable, to GradientTexture1D::_update( is queued in the MessageQueue for processing on the main thread, which is executing on the GradientTexture1D (which was already freed from the originating T50 thread).

In your particular case, it looks to be caused by something in a particles material which can at least start to point you in some direction:

#5 0x561ce19a53a3 in ParticlesMaterial::~ParticlesMaterial() scene/resources/particles_material.cpp:1867

I’ve done more testing. All Godot 4 official alpha releases between 1 and 14 segfault on loading my project.

Also - the patch makes Godot crash consistently when I try to load any scene. I am not sure it helps any - the crashing is delayed, but it’s not any more useful unfortunately.

I’ve moved the .godot folder hoping I could get back this branch to a working state but I still get the same crash, it just ends the loop on a different line of code:

Thread 1 "godot" received signal SIGSEGV, Segmentation fault.
0x000055555be06eac in String::operator+= (this=<error reading variable: Cannot access memory at address 0x7fffff7fefe8>, p_char=<error reading variable: Cannot access memory at address 0x7fffff7fefe4>) at core/string/ustring.cpp:583
583     core/string/ustring.cpp: No such file or directory.
(gdb) bt
#0  0x000055555be06eac in String::operator+= (this=<error reading variable: Cannot access memory at address 0x7fffff7fefe8>, p_char=<error reading variable: Cannot access memory at address 0x7fffff7fefe4>) at core/string/ustring.cpp:583
#1  0x000055555b1a03cb in ShaderLanguage::_get_token (this=0x55556148ade8) at servers/rendering/shader_language.cpp:805
#2  0x000055555b1c58c5 in ShaderLanguage::_parse_shader (this=0x55556148ade8, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:7803
#3  0x000055555b1d6417 in ShaderLanguage::compile (this=0x55556148ade8, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#4  0x000055555b197c57 in ShaderCompiler::compile (this=0x55556148ade8, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff7ffc90, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#5  0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557611bab0, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#6  0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x555561097fe0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#7  0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffac13b558) at ./core/templates/command_queue_mt.h:322
#8  0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fbc970) at ./core/templates/command_queue_mt.h:373
#9  0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fbc970) at ./core/templates/command_queue_mt.h:404
#10 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fbc730, p1=...) at servers/rendering/rendering_server_default.h:916

After I rolled back my merge with git reset --soft HEAD~1 and then did git rest --hard HEAD I still have a crash on project load, but it’s a bit different:

#0  0x000055555bde9b50 in StringName::StringName (this=<error reading variable: Cannot access memory at address 0x7fffff7fefe8>, p_name=<error reading variable: Cannot access memory at address 0x7fffff7fefe0>, 
    p_static=<error reading variable: Cannot access memory at address 0x7fffff7fefdc>) at core/string/string_name.cpp:200
#1  0x00005555595fddfe in ShaderLanguage::VaryingFunctionNames::VaryingFunctionNames (this=0x7fffff7ff348) at ./servers/rendering/shader_language.h:346
#2  0x00005555595fdf87 in ShaderLanguage::ShaderCompileInfo::ShaderCompileInfo (this=0x7fffff7ff300) at ./servers/rendering/shader_language.h:1113
#3  0x000055555b197ba5 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff7ff6a0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1317
#4  0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#5  0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#6  0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#7  0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#8  0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#9  0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#10 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#11 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#12 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#13 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff800710, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#14 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#15 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#16 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#17 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#18 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#20 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#21 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#22 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#23 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff801780, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#24 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#25 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#26 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#27 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#28 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#29 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#30 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#31 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#32 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#33 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff8027f0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#34 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#35 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#36 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#37 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#38 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#39 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#40 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#41 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#42 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#43 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffff803860, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#44 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#45 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#46 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#47 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#48 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#49 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916

This goes on for 20k lines and ends with:

#19894 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#19895 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#19896 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#19897 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#19898 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19899 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#19900 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#19901 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#19902 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#19903 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffbd40, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#19904 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#19905 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#19906 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#19907 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#19908 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19909 0x000055555b15304b in RenderingServerDefault::global_shader_uniform_get_type (this=0x555560fca7c0, p1=...) at servers/rendering/rendering_server_default.h:916
#19910 0x000055555b197b42 in ShaderCompiler::_get_variable_type (p_type=...) at servers/rendering/shader_compiler.cpp:1312
#19911 0x000055555b1c97a4 in ShaderLanguage::_parse_shader (this=0x555561574d48, p_functions=..., p_render_modes=..., p_shader_types=...) at servers/rendering/shader_language.cpp:8330
#19912 0x000055555b1d6417 in ShaderLanguage::compile (this=0x555561574d48, p_code=..., p_info=...) at servers/rendering/shader_language.cpp:9715
#19913 0x000055555b197c57 in ShaderCompiler::compile (this=0x555561574d48, p_mode=RenderingServer::SHADER_SPATIAL, p_code=..., p_actions=0x7fffffffcdb0, p_path=..., r_gen_code=...) at servers/rendering/shader_compiler.cpp:1323
#19914 0x000055555b2eb413 in RendererSceneRenderImplementation::SceneShaderForwardClustered::ShaderData::set_code (this=0x55557ad72040, p_code=...) at servers/rendering/renderer_rd/forward_clustered/scene_shader_forward_clustered.cpp:143
#19915 0x000055555b3244a5 in RendererRD::MaterialStorage::shader_set_code (this=0x5555610a5ff0, p_shader=..., p_code=...) at servers/rendering/renderer_rd/storage_rd/material_storage.cpp:2383
#19916 0x000055555b189ae3 in CommandQueueMT::Command2<RendererMaterialStorage, void (RendererMaterialStorage::*)(RID, String const&), RID, String>::call (this=0x7fffbc14cc78) at ./core/templates/command_queue_mt.h:322
#19917 0x000055555b13b5b9 in CommandQueueMT::_flush (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:373
#19918 0x000055555b13b646 in CommandQueueMT::flush_if_pending (this=0x555560fcaa00) at ./core/templates/command_queue_mt.h:404
#19919 0x000055555b13d965 in RenderingServerDefault::material_set_shader (this=0x555560fca7c0, p1=..., p2=...) at servers/rendering/rendering_server_default.h:242
#19920 0x000055555a89a10f in BaseMaterial3D::_update_shader (this=0x7fffbc0aa940) at scene/resources/material.cpp:1438
#19921 0x000055555a89aab9 in BaseMaterial3D::flush_changes () at scene/resources/material.cpp:1445
#19922 0x0000555559e1bbf8 in SceneTree::_call_idle_callbacks (this=0x555563fd6330) at scene/main/scene_tree.cpp:1344
#19923 0x0000555559e13c07 in SceneTree::process (this=0x555563fd6330, p_time=0.016719999999999999) at scene/main/scene_tree.cpp:500
#19924 0x0000555557940bed in Main::iteration () at main/main.cpp:2885
#19925 0x00005555578f5dce in OS_LinuxBSD::run (this=0x7fffffffd780) at platform/linuxbsd/os_linuxbsd.cpp:540
#19926 0x00005555578f1260 in main (argc=4, argv=0x7fffffffdca8) at platform/linuxbsd/godot_linuxbsd.cpp:72

I guess we can see where this endless loop has started. Is it #19921 0x000055555a89aab9 in BaseMaterial3D::flush_changes () at scene/resources/material.cpp:1445 ? Is that a hint?

I wonder why this persists even though code on my disk is now rolled back…

I was able to reproduce the issue. Here’s the ASAN report: asan_report.txt

The reports shows an use-after-free, but in particular what I’m seeing with gdb is that there’s a CommandBase casted object that seems to get overwritten at some point. It’s function pointer at cmd->call gets zeroed out and that’s what’s causing a segfault:

Thread 1 "godot-master" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) up
#1  0x0000555555f4f9ac in CommandQueueMT::_flush (this=0x55556287fac0) at ./core/templates/command_queue_mt.h:373
373                             cmd->call(); //execute the function