godot: Radeon Vulkan (radv) driver crash on Linux with -O1 or -O2 optimizations (target=release_debug build with GCC 10)

Godot version: master (0cd98ec7e13038d09a77cf821e930be79026f943)

OS/device including version: Mageia 8 (Linux)

CPU:       Topology: Quad Core model: Intel Core i7-8705G bits: 64 type: MT MCP L2 cache: 8192 KiB 
Graphics:  Device-1: Intel HD Graphics 630 driver: i915 v: kernel 
           Device-2: Advanced Micro Devices [AMD/ATI] Polaris 22 XL [Radeon RX Vega M GL] driver: amdgpu v: kernel 
           Display: x11 server: Mageia X.org 1.20.8 driver: intel,v4l resolution: 1: 1920x1080 2: 1920x1080 
           OpenGL: renderer: AMD VEGAM (DRM 3.37.0 5.7.9-desktop-1.mga8 LLVM 10.0.0) v: 4.6 Mesa 20.1.1

Issue specific to the AMD GPU, using the radv driver from Mesa 20.1.1.

Compiling with:

gcc (Mageia 10.2.0-0.RC.1.mga8) 10.2.0

Issue description: While I can run Godot fine with a tools=yes target=debug build (default parameters), an optimized tools=yes target=release_debug build crashes on the AMD Radeon RX Vega M GPU with Mesa’s radv.

If I force the use of the Intel GPU, it doesn’t crash.

Backtrace (I built with target=debug debug_symbols=full to have more debug info):

Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x00007ffff78a6ce4 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff78a6ce4 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
#1  0x00007ffff54c97ef in memcpy (__len=<optimized out>, __src=0x6226a8c, __dest=<optimized out>) at /usr/include/bits/string_fortified.h:34
#2  radv_flush_constants (cmd_buffer=cmd_buffer@entry=0x62262a0, stages=<optimized out>, stages@entry=31) at ../src/amd/vulkan/radv_cmd_buffer.c:2410
#3  0x00007ffff54ca615 in radv_upload_graphics_shader_descriptors (cmd_buffer=cmd_buffer@entry=0x62262a0, pipeline_is_dirty=pipeline_is_dirty@entry=true) at ../src/amd/vulkan/radv_cmd_buffer.c:2651
#4  0x00007ffff54cd6e0 in radv_draw (cmd_buffer=0x62262a0, info=info@entry=0x7fffffffc720) at ../src/amd/vulkan/radv_cmd_buffer.c:4837
#5  0x00007ffff54cf69e in radv_CmdDrawIndexed (commandBuffer=<optimized out>, indexCount=<optimized out>, instanceCount=<optimized out>, firstIndex=<optimized out>, vertexOffset=<optimized out>, 
    firstInstance=<optimized out>) at ../src/amd/vulkan/radv_cmd_buffer.c:4901
#6  0x0000000002becb1d in RasterizerCanvasRD::_render_item (this=0x7fffe8427020, p_draw_list=<optimized out>, p_item=<optimized out>, p_framebuffer_format=<optimized out>, p_canvas_transform_inverse=..., 
    current_clip=<optimized out>, p_lights=<optimized out>, p_pipeline_variants=<optimized out>) at servers/rendering/rasterizer_rd/rasterizer_canvas_rd.cpp:864
#7  0x0000000002bee717 in RasterizerCanvasRD::_render_items (this=0x7fffe8427020, p_to_render_target=..., p_item_count=<optimized out>, p_canvas_transform_inverse=..., p_lights=0x0, p_screen_uniform_set=...)
    at servers/rendering/rasterizer_rd/rasterizer_canvas_rd.cpp:1362
#8  0x0000000002bef23a in RasterizerCanvasRD::canvas_render_items (this=0x7fffe8427020, p_to_render_target=..., p_item_list=<optimized out>, p_modulate=..., p_light_list=0x0, p_canvas_transform=...)
    at servers/rendering/rasterizer_rd/rasterizer_canvas_rd.cpp:1524
#9  0x0000000002bda8ff in RenderingServerCanvas::render_canvas (this=<optimized out>, p_render_target=..., p_canvas=p_canvas@entry=0x73bdf70, p_transform=..., p_lights=0x0, 
    p_masked_lights=p_masked_lights@entry=0x0, p_clip_rect=...) at servers/rendering/rendering_server_canvas.cpp:265
#10 0x0000000002a382f9 in RenderingServerViewport::_draw_viewport (this=0x62ab370, p_viewport=0x7479bb0, p_eye=XRInterface::EYE_MONO) at servers/rendering/rendering_server_viewport.cpp:257
#11 0x0000000002a39709 in RenderingServerViewport::draw_viewports (this=0x62ab370) at servers/rendering/rendering_server_viewport.cpp:413
#12 0x0000000002a0a7ca in RenderingServerRaster::draw (this=0x5ba65d0, p_swap_buffers=<optimized out>, frame_step=0.39811098575592041) at servers/rendering/rendering_server_raster.cpp:112
#13 0x00000000007d4e3e in Main::iteration () at main/main.cpp:2204
#14 0x00000000007b5359 in OS_LinuxBSD::run (this=this@entry=0x7fffffffd1d0) at platform/linuxbsd/os_linuxbsd.cpp:238
#15 0x00000000007b0cf3 in main (argc=1, argv=0x7fffffffd6c8) at platform/linuxbsd/godot_linuxbsd.cpp:55

This is likely a Mesa driver bug, but opening an issue here to track it and see if it might be triggered by something specific we do.

The only difference between target=debug and target=release_debug is the optimization level (-O2) and the use of -rdynamic. I suspect that the optimization might be the problem, I’ll do some tests to figure it out, and open a Mesa bug report.

Steps to reproduce:

scons p=linuxbsd tools=yes target=release_debug debug_symbols=full
Run it, poof

Minimal reproduction project: Crashes even in the project manager.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (20 by maintainers)

Most upvoted comments

Here you create a command buffer:

https://github.com/godotengine/godot/blob/2efba270899b2852fa67cad05a55fc81decb24cd/drivers/vulkan/rendering_device_vulkan.cpp#L7186

maybe the buffer is in a bad state and Mesa than reads more that it ought to? I would dump its content at the place it crashes.

marxin on Jul 28, 2020

valgrind does seem to find some invalid reads:

$ valgrind ./bin/godot.linuxbsd.tools.64 
==441176== Memcheck, a memory error detector
==441176== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==441176== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==441176== Command: ./bin/godot.linuxbsd.tools.64
==441176== 
Godot Engine v4.0.dev.custom_build.5700429e4 - https://godotengine.org
DisplayServer::_create_window 0 want rect: 448, 240, 1024, 600 got rect 0, 0, 1024, 600
DisplayServer::_window_changed: 0 rect: 896, 29, 1024, 600
 
==441176== Conditional jump or move depends on uninitialised value(s)
==441176==    at 0xE33B68: DisplayServerX11::process_events() (display_server_x11.cpp:2490)
==441176==    by 0xE240F6: OS_LinuxBSD::run() (os_linuxbsd.cpp:234)
==441176==    by 0xE23064: main (godot_linuxbsd.cpp:58)
==441176== 
==441176== Invalid read of size 2
==441176==    at 0x65FA2EF: memmove (vg_replace_strmem.c:1270)
==441176==    by 0x9ED48FE: UnknownInlinedFun (string_fortified.h:34)
==441176==    by 0x9ED48FE: radv_flush_constants (radv_cmd_buffer.c:2433)
==441176==    by 0x9ED5724: radv_upload_graphics_shader_descriptors (radv_cmd_buffer.c:2674)
==441176==    by 0x9ED88AF: radv_draw (radv_cmd_buffer.c:4860)
==441176==    by 0x9EDA84D: radv_CmdDrawIndexed (radv_cmd_buffer.c:4924)
==441176==    by 0x1A496A6: vkCmdDrawIndexed (trampoline.c:1698)
==441176==    by 0x19FB870: RenderingDeviceVulkan::draw_list_draw(long, bool, unsigned int, unsigned int) (rendering_device_vulkan.cpp:6228)
==441176==    by 0x334FE9F: RasterizerCanvasRD::_render_item(long, RasterizerCanvas::Item const*, long, Transform2D const&, RasterizerCanvas::Item*&, RasterizerCanvas::Light*, RasterizerCanvasRD::PipelineVariants*) (rasterizer_canvas_rd.cpp:788)
==441176==    by 0x3351D22: RasterizerCanvasRD::_render_items(RID, int, Transform2D const&, RasterizerCanvas::Light*, RID) (rasterizer_canvas_rd.cpp:1362)
==441176==    by 0x3352988: RasterizerCanvasRD::canvas_render_items(RID, RasterizerCanvas::Item*, Color const&, RasterizerCanvas::Light*, Transform2D const&) (rasterizer_canvas_rd.cpp:1524)
==441176==    by 0x3341D59: RenderingServerCanvas::_render_canvas_item_tree(RID, RenderingServerCanvas::Canvas::ChildItem*, int, RenderingServerCanvas::Item*, Transform2D const&, Rect2 const&, Color const&, RasterizerCanvas::Light*) (rendering_server_canvas.cpp:71)
==441176==    by 0x3341EBC: RenderingServerCanvas::render_canvas(RID, RenderingServerCanvas::Canvas*, Transform2D const&, RasterizerCanvas::Light*, RasterizerCanvas::Light*, Rect2 const&) (rendering_server_canvas.cpp:265)
==441176==  Address 0x92a3cc0 is 0 bytes after a block of size 3,744 alloc'd
==441176==    at 0x65F3751: malloc (vg_replace_malloc.c:307)
==441176==    by 0x9ED8E09: UnknownInlinedFun (vk_alloc.h:36)
==441176==    by 0x9ED8E09: UnknownInlinedFun (vk_alloc.h:44)
==441176==    by 0x9ED8E09: radv_create_cmd_buffer (radv_cmd_buffer.c:275)
==441176==    by 0x9ED8E09: radv_AllocateCommandBuffers (radv_cmd_buffer.c:3349)
==441176==    by 0x1A49517: vkAllocateCommandBuffers (trampoline.c:1516)
==441176==    by 0x1A0261C: RenderingDeviceVulkan::initialize(VulkanContext*, bool) (rendering_device_vulkan.cpp:7186)
==441176==    by 0xE38F40: DisplayServerX11::DisplayServerX11(String const&, DisplayServer::WindowMode, unsigned int, Vector2i const&, Error&) (display_server_x11.cpp:3662)
==441176==    by 0xE39441: DisplayServerX11::create_func(String const&, DisplayServer::WindowMode, unsigned int, Vector2i const&, Error&) (display_server_x11.cpp:3188)
==441176==    by 0x2F930C8: DisplayServer::create(int, String const&, DisplayServer::WindowMode, unsigned int, Vector2i const&, Error&) (display_server.cpp:599)
==441176==    by 0xE44E20: Main::setup2(unsigned long) (main.cpp:1409)
==441176==    by 0xE4F349: Main::setup(char const*, int, char**, bool) (main.cpp:1331)
==441176==    by 0xE2300D: main (godot_linuxbsd.cpp:51)
==441176== 
==441176== Invalid read of size 2
==441176==    at 0x65FA2E0: memmove (vg_replace_strmem.c:1270)
==441176==    by 0x9ED48FE: UnknownInlinedFun (string_fortified.h:34)
==441176==    by 0x9ED48FE: radv_flush_constants (radv_cmd_buffer.c:2433)
==441176==    by 0x9ED5724: radv_upload_graphics_shader_descriptors (radv_cmd_buffer.c:2674)
==441176==    by 0x9ED88AF: radv_draw (radv_cmd_buffer.c:4860)
==441176==    by 0x9EDA84D: radv_CmdDrawIndexed (radv_cmd_buffer.c:4924)
==441176==    by 0x1A496A6: vkCmdDrawIndexed (trampoline.c:1698)
==441176==    by 0x19FB870: RenderingDeviceVulkan::draw_list_draw(long, bool, unsigned int, unsigned int) (rendering_device_vulkan.cpp:6228)
==441176==    by 0x334FE9F: RasterizerCanvasRD::_render_item(long, RasterizerCanvas::Item const*, long, Transform2D const&, RasterizerCanvas::Item*&, RasterizerCanvas::Light*, RasterizerCanvasRD::PipelineVariants*) (rasterizer_canvas_rd.cpp:788)
==441176==    by 0x3351D22: RasterizerCanvasRD::_render_items(RID, int, Transform2D const&, RasterizerCanvas::Light*, RID) (rasterizer_canvas_rd.cpp:1362)
==441176==    by 0x3352988: RasterizerCanvasRD::canvas_render_items(RID, RasterizerCanvas::Item*, Color const&, RasterizerCanvas::Light*, Transform2D const&) (rasterizer_canvas_rd.cpp:1524)
==441176==    by 0x3341D59: RenderingServerCanvas::_render_canvas_item_tree(RID, RenderingServerCanvas::Canvas::ChildItem*, int, RenderingServerCanvas::Item*, Transform2D const&, Rect2 const&, Color const&, RasterizerCanvas::Light*) (rendering_server_canvas.cpp:71)
==441176==    by 0x3341EBC: RenderingServerCanvas::render_canvas(RID, RenderingServerCanvas::Canvas*, Transform2D const&, RasterizerCanvas::Light*, RasterizerCanvas::Light*, Rect2 const&) (rendering_server_canvas.cpp:265)
==441176==  Address 0x92a3cc6 is 6 bytes after a block of size 3,744 alloc'd
==441176==    at 0x65F3751: malloc (vg_replace_malloc.c:307)
==441176==    by 0x9ED8E09: UnknownInlinedFun (vk_alloc.h:36)
==441176==    by 0x9ED8E09: UnknownInlinedFun (vk_alloc.h:44)
==441176==    by 0x9ED8E09: radv_create_cmd_buffer (radv_cmd_buffer.c:275)
==441176==    by 0x9ED8E09: radv_AllocateCommandBuffers (radv_cmd_buffer.c:3349)
==441176==    by 0x1A49517: vkAllocateCommandBuffers (trampoline.c:1516)
==441176==    by 0x1A0261C: RenderingDeviceVulkan::initialize(VulkanContext*, bool) (rendering_device_vulkan.cpp:7186)
==441176==    by 0xE38F40: DisplayServerX11::DisplayServerX11(String const&, DisplayServer::WindowMode, unsigned int, Vector2i const&, Error&) (display_server_x11.cpp:3662)
==441176==    by 0xE39441: DisplayServerX11::create_func(String const&, DisplayServer::WindowMode, unsigned int, Vector2i const&, Error&) (display_server_x11.cpp:3188)
==441176==    by 0x2F930C8: DisplayServer::create(int, String const&, DisplayServer::WindowMode, unsigned int, Vector2i const&, Error&) (display_server.cpp:599)
==441176==    by 0xE44E20: Main::setup2(unsigned long) (main.cpp:1409)
==441176==    by 0xE4F349: Main::setup(char const*, int, char**, bool) (main.cpp:1331)
==441176==    by 0xE2300D: main (godot_linuxbsd.cpp:51)
==441176== 
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib64/libc.so.6(+0x3a3f0) [0x6d163f0] (??:0)
[2] /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so(_vgr20181ZZ_libcZdsoZa_memmove+0x15f) [0x65fa2ef] (??:0)
[3] /usr/lib64/libvulkan_radeon.so(+0xba8ff) [0x9ed48ff] (??:0)
[4] /usr/lib64/libvulkan_radeon.so(+0xbb725) [0x9ed5725] (??:0)
[5] /usr/lib64/libvulkan_radeon.so(+0xbe8b0) [0x9ed88b0] (??:0)
[6] /usr/lib64/libvulkan_radeon.so(+0xc084e) [0x9eda84e] (??:0)
[7] ./bin/godot.linuxbsd.tools.64(vkCmdDrawIndexed+0xd) [0x1a496a7] (/home/akien/Projects/godot/godot.git/thirdparty/vulkan/loader/trampoline.c:1699)
[8] RenderingDeviceVulkan::draw_list_draw(long, bool, unsigned int, unsigned int) (/home/akien/Projects/godot/godot.git/drivers/vulkan/rendering_device_vulkan.cpp:6228 (discriminator 2))
[9] RasterizerCanvasRD::_render_item(long, RasterizerCanvas::Item const*, long, Transform2D const&, RasterizerCanvas::Item*&, RasterizerCanvas::Light*, RasterizerCanvasRD::PipelineVariants*) (/home/akien/Projects/godot/godot.git/servers/rendering/rasterizer_rd/rasterizer_canvas_rd.cpp:1289)
[10] RasterizerCanvasRD::_render_items(RID, int, Transform2D const&, RasterizerCanvas::Light*, RID) (/home/akien/Projects/godot/godot.git/servers/rendering/rasterizer_rd/rasterizer_canvas_rd.cpp:1362)
[11] RasterizerCanvasRD::canvas_render_items(RID, RasterizerCanvas::Item*, Color const&, RasterizerCanvas::Light*, Transform2D const&) (/home/akien/Projects/godot/godot.git/servers/rendering/rasterizer_rd/rasterizer_canvas_rd.cpp:1529)
[12] RenderingServerCanvas::_render_canvas_item_tree(RID, RenderingServerCanvas::Canvas::ChildItem*, int, RenderingServerCanvas::Item*, Transform2D const&, Rect2 const&, Color const&, RasterizerCanvas::Light*) (/home/akien/Projects/godot/godot.git/servers/rendering/rendering_server_canvas.cpp:72 (discriminator 5))
[13] RenderingServerCanvas::render_canvas(RID, RenderingServerCanvas::Canvas*, Transform2D const&, RasterizerCanvas::Light*, RasterizerCanvas::Light*, Rect2 const&) (/home/akien/Projects/godot/godot.git/servers/rendering/rendering_server_canvas.cpp:265)
[14] RenderingServerViewport::_draw_viewport(RenderingServerViewport::Viewport*, XRInterface::Eyes) (/home/akien/Projects/godot/godot.git/servers/rendering/rendering_server_viewport.cpp:257)
[15] RenderingServerViewport::draw_viewports() (/home/akien/Projects/godot/godot.git/servers/rendering/rendering_server_viewport.cpp:415)
[16] RenderingServerRaster::draw(bool, double) (/home/akien/Projects/godot/godot.git/servers/rendering/rendering_server_raster.cpp:113)
[17] RenderingServerWrapMT::draw(bool, double) (/home/akien/Projects/godot/godot.git/servers/rendering/rendering_server_wrap_mt.cpp:93)
[18] Main::iteration() (/home/akien/Projects/godot/godot.git/main/main.cpp:2377)
[19] OS_LinuxBSD::run() (/home/akien/Projects/godot/godot.git/platform/linuxbsd/os_linuxbsd.cpp:238)
[20] ./bin/godot.linuxbsd.tools.64(main+0xf3) [0xe23065] (/home/akien/Projects/godot/godot.git/platform/linuxbsd/godot_linuxbsd.cpp:60)
[21] /lib64/libc.so.6(__libc_start_main+0xea) [0x6d02cda] (??:0)
[22] ./bin/godot.linuxbsd.tools.64(_start+0x2a) [0xe22eca] (??:?)
-- END OF BACKTRACE --
==441176== 
==441176== Process terminating with default action of signal 6 (SIGABRT): dumping core
==441176==    at 0x6D16380: raise (in /usr/lib64/libc-2.31.so)
==441176==    by 0x6D01526: abort (in /usr/lib64/libc-2.31.so)
==441176==    by 0xE23E34: handle_crash(int) (crash_handler_linuxbsd.cpp:120)
==441176==    by 0x6D163EF: ??? (in /usr/lib64/libc-2.31.so)
==441176==    by 0x65FA2EE: memmove (vg_replace_strmem.c:1270)
==441176==    by 0x9ED48FE: UnknownInlinedFun (string_fortified.h:34)
==441176==    by 0x9ED48FE: radv_flush_constants (radv_cmd_buffer.c:2433)
==441176==    by 0x9ED5724: radv_upload_graphics_shader_descriptors (radv_cmd_buffer.c:2674)
==441176==    by 0x9ED88AF: radv_draw (radv_cmd_buffer.c:4860)
==441176==    by 0x9EDA84D: radv_CmdDrawIndexed (radv_cmd_buffer.c:4924)
==441176==    by 0x1A496A6: vkCmdDrawIndexed (trampoline.c:1698)
==441176==    by 0x19FB870: RenderingDeviceVulkan::draw_list_draw(long, bool, unsigned int, unsigned int) (rendering_device_vulkan.cpp:6228)
==441176==    by 0x334FE9F: RasterizerCanvasRD::_render_item(long, RasterizerCanvas::Item const*, long, Transform2D const&, RasterizerCanvas::Item*&, RasterizerCanvas::Light*, RasterizerCanvasRD::PipelineVariants*) (rasterizer_canvas_rd.cpp:788)
==441176== 
==441176== HEAP SUMMARY:
==441176==     in use at exit: 103,100,524 bytes in 216,800 blocks
==441176==   total heap usage: 4,079,899 allocs, 3,863,099 frees, 2,057,016,341 bytes allocated
==441176== 
==441176== LEAK SUMMARY:
==441176==    definitely lost: 7,752 bytes in 63 blocks
==441176==    indirectly lost: 10,392 bytes in 63 blocks
==441176==      possibly lost: 78,787,894 bytes in 200,823 blocks
==441176==    still reachable: 24,294,438 bytes in 15,849 blocks
==441176==                       of which reachable via heuristic:
==441176==                         newarray           : 262,808 bytes in 20 blocks
==441176==                         multipleinheritance: 3,744 bytes in 6 blocks
==441176==         suppressed: 48 bytes in 2 blocks
==441176== Rerun with --leak-check=full to see details of leaked memory
==441176== 
==441176== Use --track-origins=yes to see where uninitialised values come from
==441176== For lists of detected and suppressed errors, rerun with: -s
==441176== ERROR SUMMARY: 318642 errors from 3 contexts (suppressed: 0 from 0)
Aborted (core dumped)

akien-mga on Jul 26, 2020