thorvg: Random crashes due data races

When converting some svg with svg2png tool, I sometimes have crash. When I check this files manually only once, almost always this works fine, but when I convert them in loop 100-1000 times, then I have very frequently multiple crashes.

Example file: https://user-images.githubusercontent.com/41945903/232339294-4afce7ae-2af2-459a-b2f5-c30172a19c42.svg

Try to run conversion in loop (and compile app with asan sanitizer)

 for i in {0..1000}; do /home/rafal/test/thorvg/build/src/bin/svg2png/svg2png /home/rafal/Desktop/SVGBigPack/KOSZ_20043.svg -r 400x400; done;

this may happens very rarely, so I suggest to use https://github.com/qarmin/SVG-regression-finder in which problem is more visible(tool automatically will use all avaialble threads)

Ubsan/Asan Backtraces:

==87591==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000001bde at pc 0x7fb0ad6b96b4 bp 0x7fff8c432850 sp 0x7fff8c432840
READ of size 1 at 0x603000001bde thread T0
    #0 0x7fb0ad6b96b3 in _decomposeOutline ../src/lib/sw_engine/tvgSwRle.cpp:733
    #1 0x7fb0ad6b96b3 in _genRle ../src/lib/sw_engine/tvgSwRle.cpp:767
    #2 0x7fb0ad6bbd38 in rleRender(SwRleData*, SwOutline const*, SwBBox const&, bool) ../src/lib/sw_engine/tvgSwRle.cpp:963
    #3 0x7fb0ad6c138a in shapeGenRle(SwShape*, tvg::RenderShape const*, bool) ../src/lib/sw_engine/tvgSwShape.cpp:517
    #4 0x7fb0ad6b1158 in SwShapeTask::run(unsigned int) ../src/lib/sw_engine/tvgSwRenderer.cpp:122
    #5 0x7fb0ad6adfa6 in tvg::Task::done(unsigned int) ../src/lib/tvgTaskScheduler.h:63
    #6 0x7fb0ad6a8bc0 in tvg::SwRenderer::renderShape(void*) ../src/lib/sw_engine/tvgSwRenderer.cpp:386
    #7 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #8 0x7fb0ad657cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #9 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #10 0x7fb0ad657cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #11 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #12 0x7fb0ad657cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #13 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #14 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #15 0x7fb0ad630664 in tvg::Canvas::Impl::draw() ../src/lib/tvgCanvasImpl.h:120
    #16 0x7fb0ad62e9b5 in tvg::Canvas::draw() ../src/lib/tvgCanvas.cpp:61
    #17 0x55f4554051f9 in Renderer::render(char const*, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ../src/bin/svg2png/svg2png.cpp:145
    #18 0x55f455406000 in App::renderFile(char const*) ../src/bin/svg2png/svg2png.cpp:342
    #19 0x55f4554078a4 in App::setup(int, char**) ../src/bin/svg2png/svg2png.cpp:264
    #20 0x55f4553fffaa in main ../src/bin/svg2png/svg2png.cpp:413
    #21 0x7fb0ac42350f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #22 0x7fb0ac4235c8 in __libc_start_main_impl ../csu/libc-start.c:381
    #23 0x55f4554000a4 in _start (/home/rafal/test/thorvg/build/src/bin/svg2png/svg2png+0x9e0a4)

0x603000001bde is located 14 bytes inside of 19-byte region [0x603000001bd0,0x603000001be3)
freed by thread T1 here:
    #0 0x7fb0adabe720 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7fb0ad6bcf2c in _growOutlinePoint ../src/lib/sw_engine/tvgSwShape.cpp:93
    #2 0x7fb0ad6bfb6f in _genOutline ../src/lib/sw_engine/tvgSwShape.cpp:434
    #3 0x7fb0ad6c0cdf in shapePrepare(SwShape*, tvg::RenderShape const*, tvg::Matrix const*, SwBBox const&, SwBBox&, SwMpool*, unsigned int, bool) ../src/lib/sw_engine/tvgSwShape.cpp:484
    #4 0x7fb0ad6b07ac in SwShapeTask::run(unsigned int) ../src/lib/sw_engine/tvgSwRenderer.cpp:105
    #5 0x7fb0ad668703 in tvg::Task::operator()(unsigned int) ../src/lib/tvgTaskScheduler.h:81
    #6 0x7fb0ad66a289 in tvg::TaskSchedulerImpl::run(unsigned int) (/home/rafal/test/thorvg/build/src/bin/svg2png/../../libthorvg.so.0+0x26a289)
    #7 0x7fb0ad0dc3a2  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc3a2)

previously allocated by thread T0 here:
    #0 0x7fb0adabe720 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7fb0ad6bcf2c in _growOutlinePoint ../src/lib/sw_engine/tvgSwShape.cpp:93
    #2 0x7fb0ad6bfb6f in _genOutline ../src/lib/sw_engine/tvgSwShape.cpp:434
    #3 0x7fb0ad6c0cdf in shapePrepare(SwShape*, tvg::RenderShape const*, tvg::Matrix const*, SwBBox const&, SwBBox&, SwMpool*, unsigned int, bool) ../src/lib/sw_engine/tvgSwShape.cpp:484
    #4 0x7fb0ad6b07ac in SwShapeTask::run(unsigned int) ../src/lib/sw_engine/tvgSwRenderer.cpp:105
    #5 0x7fb0ad6adfa6 in tvg::Task::done(unsigned int) ../src/lib/tvgTaskScheduler.h:63
    #6 0x7fb0ad6a8bc0 in tvg::SwRenderer::renderShape(void*) ../src/lib/sw_engine/tvgSwRenderer.cpp:386
    #7 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #8 0x7fb0ad657cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #9 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #10 0x7fb0ad657cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #11 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #12 0x7fb0ad657cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #13 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #14 0x7fb0ad63cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #15 0x7fb0ad630664 in tvg::Canvas::Impl::draw() ../src/lib/tvgCanvasImpl.h:120
    #16 0x7fb0ad62e9b5 in tvg::Canvas::draw() ../src/lib/tvgCanvas.cpp:61
    #17 0x55f4554051f9 in Renderer::render(char const*, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ../src/bin/svg2png/svg2png.cpp:145
    #18 0x55f455406000 in App::renderFile(char const*) ../src/bin/svg2png/svg2png.cpp:342
    #19 0x55f4554078a4 in App::setup(int, char**) ../src/bin/svg2png/svg2png.cpp:264
    #20 0x55f4553fffaa in main ../src/bin/svg2png/svg2png.cpp:413
    #21 0x7fb0ac42350f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

Thread T1 created by T0 here:
    #0 0x7fb0ada4af75 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
    #1 0x7fb0ad0dc478 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc478)
    #2 0x7fb0ad67083e in void std::vector<std::thread, std::allocator<std::thread> >::_M_realloc_insert<tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}>(__gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >, tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}&&) (/home/rafal/test/thorvg/build/src/bin/svg2png/../../libthorvg.so.0+0x27083e)
    #3 0x7fb0ad667acd in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}>(tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}&&) /usr/include/c++/12/bits/vector.tcc:123
    #4 0x7fb0ad667acd in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}>(tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}&&) /usr/include/c++/12/bits/vector.tcc:111
    #5 0x7fb0ad667acd in tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int) ../src/lib/tvgTaskScheduler.cpp:113
    #6 0x7fb0ad667acd in tvg::TaskScheduler::init(unsigned int) ../src/lib/tvgTaskScheduler.cpp:170
    #7 0x7fb0ad632f13 in tvg::Initializer::init(tvg::CanvasEngine, unsigned int) ../src/lib/tvgInitializer.cpp:112
    #8 0x55f455402bfb in Renderer::createCanvas() ../src/bin/svg2png/svg2png.cpp:176
    #9 0x55f4554039ef in Renderer::render(char const*, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ../src/bin/svg2png/svg2png.cpp:76
    #10 0x55f455406000 in App::renderFile(char const*) ../src/bin/svg2png/svg2png.cpp:342
    #11 0x55f4554078a4 in App::setup(int, char**) ../src/bin/svg2png/svg2png.cpp:264
    #12 0x55f4553fffaa in main ../src/bin/svg2png/svg2png.cpp:413
    #13 0x7fb0ac42350f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
WRITE of size 240 at 0x61b000000700 thread T1
    #0 0x7f8b0d049845 in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
    #1 0x7f8b0ccc625f in memcpy /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
    #2 0x7f8b0ccc625f in _exportBorderOutline ../src/lib/sw_engine/tvgSwStroke.cpp:785
    #3 0x7f8b0cccfcf8 in strokeExportOutline(SwStroke*, SwMpool*, unsigned int) ../src/lib/sw_engine/tvgSwStroke.cpp:936
    #4 0x7f8b0ccc35c6 in shapeGenStrokeRle(SwShape*, tvg::RenderShape const*, tvg::Matrix const*, SwBBox const&, SwBBox&, SwMpool*, unsigned int) ../src/lib/sw_engine/tvgSwShape.cpp:597
    #5 0x7f8b0ccb1c97 in SwShapeTask::run(unsigned int) ../src/lib/sw_engine/tvgSwRenderer.cpp:137
    #6 0x7f8b0cc68703 in tvg::Task::operator()(unsigned int) ../src/lib/tvgTaskScheduler.h:81
    #7 0x7f8b0cc6a289 in tvg::TaskSchedulerImpl::run(unsigned int) (/home/rafal/test/thorvg/build/src/bin/svg2png/../../libthorvg.so.0+0x26a289)
    #8 0x7f8b0c6dc3a2  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc3a2)
    #9 0x7f8b0ba90401 in start_thread nptl/pthread_create.c:442
    #10 0x7f8b0bb1f58f in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

0x61b000000700 is located 0 bytes to the right of 1664-byte region [0x61b000000080,0x61b000000700)
allocated by thread T0 here:
    #0 0x7f8b0d0be720 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:85
    #1 0x7f8b0cccfa34 in strokeExportOutline(SwStroke*, SwMpool*, unsigned int) ../src/lib/sw_engine/tvgSwStroke.cpp:927
    #2 0x7f8b0ccc35c6 in shapeGenStrokeRle(SwShape*, tvg::RenderShape const*, tvg::Matrix const*, SwBBox const&, SwBBox&, SwMpool*, unsigned int) ../src/lib/sw_engine/tvgSwShape.cpp:597
    #3 0x7f8b0ccb1c97 in SwShapeTask::run(unsigned int) ../src/lib/sw_engine/tvgSwRenderer.cpp:137
    #4 0x7f8b0ccadfa6 in tvg::Task::done(unsigned int) ../src/lib/tvgTaskScheduler.h:63
    #5 0x7f8b0cca8bc0 in tvg::SwRenderer::renderShape(void*) ../src/lib/sw_engine/tvgSwRenderer.cpp:386
    #6 0x7f8b0cc3cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #7 0x7f8b0cc57cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #8 0x7f8b0cc3cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #9 0x7f8b0cc57cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #10 0x7f8b0cc3cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #11 0x7f8b0cc57cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #12 0x7f8b0cc3cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #13 0x7f8b0cc57cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #14 0x7f8b0cc3cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #15 0x7f8b0cc57cb3 in tvg::Scene::Impl::render(tvg::RenderMethod&) ../src/lib/tvgSceneImpl.h:134
    #16 0x7f8b0cc3cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #17 0x7f8b0cc3cdb0 in tvg::Paint::Impl::render(tvg::RenderMethod&) ../src/lib/tvgPaint.cpp:177
    #18 0x7f8b0cc30664 in tvg::Canvas::Impl::draw() ../src/lib/tvgCanvasImpl.h:120
    #19 0x7f8b0cc2e9b5 in tvg::Canvas::draw() ../src/lib/tvgCanvas.cpp:61
    #20 0x555a389b81f9 in Renderer::render(char const*, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ../src/bin/svg2png/svg2png.cpp:145
    #21 0x555a389b9000 in App::renderFile(char const*) ../src/bin/svg2png/svg2png.cpp:342
    #22 0x555a389ba8a4 in App::setup(int, char**) ../src/bin/svg2png/svg2png.cpp:264
    #23 0x555a389b2faa in main ../src/bin/svg2png/svg2png.cpp:413
    #24 0x7f8b0ba2350f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

Thread T1 created by T0 here:
    #0 0x7f8b0d04af75 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
    #1 0x7f8b0c6dc478 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc478)
    #2 0x7f8b0cc7083e in void std::vector<std::thread, std::allocator<std::thread> >::_M_realloc_insert<tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}>(__gnu_cxx::__normal_iterator<std::thread*, std::vector<std::thread, std::allocator<std::thread> > >, tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}&&) (/home/rafal/test/thorvg/build/src/bin/svg2png/../../libthorvg.so.0+0x27083e)
    #3 0x7f8b0cc67acd in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}>(tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}&&) /usr/include/c++/12/bits/vector.tcc:123
    #4 0x7f8b0cc67acd in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}>(tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int)::{lambda()#1}&&) /usr/include/c++/12/bits/vector.tcc:111
    #5 0x7f8b0cc67acd in tvg::TaskSchedulerImpl::TaskSchedulerImpl(unsigned int) ../src/lib/tvgTaskScheduler.cpp:113
    #6 0x7f8b0cc67acd in tvg::TaskScheduler::init(unsigned int) ../src/lib/tvgTaskScheduler.cpp:170
    #7 0x7f8b0cc32f13 in tvg::Initializer::init(tvg::CanvasEngine, unsigned int) ../src/lib/tvgInitializer.cpp:112
    #8 0x555a389b5bfb in Renderer::createCanvas() ../src/bin/svg2png/svg2png.cpp:176
    #9 0x555a389b69ef in Renderer::render(char const*, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ../src/bin/svg2png/svg2png.cpp:76
    #10 0x555a389b9000 in App::renderFile(char const*) ../src/bin/svg2png/svg2png.cpp:342
    #11 0x555a389ba8a4 in App::setup(int, char**) ../src/bin/svg2png/svg2png.cpp:264
    #12 0x555a389b2faa in main ../src/bin/svg2png/svg2png.cpp:413
    #13 0x7f8b0ba2350f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 15 (8 by maintainers)

Commits related to this issue

Most upvoted comments

does Godot currently have a memory corruption issue unrelated to ThorVG?

Yes, the handling of operating system windows and devices has timing relevant lock and crash issues, but that can be avoided so it’s irrelevant for this test case.

@capnm I updated the patch, could you please test again with applying them?

@hermet I don’t know if it might help, after #1397 is the behaviour different.

In lock-down by Godot exit I see 2 threads running thorvg code.

The one is waiting in TaskSchedulerImpl::~TaskSchedulerImpl() and the other in tvg::TaskSchedulerImpl::run(unsigned int): (*task)(i + 1);tvg::Task::operator()(unsigned int):lock_guard<mutex> lock(mtx);

Crash now mostly happens by a allocating corrupt memory by Godot itself, one svg thread is e.g. running ImageLoaderSVG::create_image_from_utf8_buffer(this=<unavailable>, p_image=Ref<Image> @ 0x000000009f107200, p_buffer=<unavailable>, p_scale=<unavailable>, p_upsample=<unavailable>) at image_loader_svg.cpp:132

and the other waiting in

    frame #0: 0x00007ffff7f92170 libpthread.so.0`__lll_lock_wait(futex=0x000055555f85e2b8, private=0) at lowlevellock.c:52:7
    frame #1: 0x00007ffff7f8a0a3 libpthread.so.0`__GI___pthread_mutex_lock(mutex=0x000055555f85e2b8) at pthread_mutex_lock.c:80:7
    frame #2: 0x000055555758d80d godot.linuxbsd.editor.dev.x86_64`tvg::Task::operator()(unsigned int) [inlined] __gthread_mutex_lock(__mutex=0x000055555f85e2b8) at gthr-default.h:749:12
    frame #3: 0x000055555758d7fb godot.linuxbsd.editor.dev.x86_64`tvg::Task::operator()(unsigned int) [inlined] std::mutex::lock(this=<unavailable>) at std_mutex.h:100
    frame #4: 0x000055555758d7fb godot.linuxbsd.editor.dev.x86_64`tvg::Task::operator()(unsigned int) [inlined] std::lock_guard<std::mutex>::lock_guard(__m=<unavailable>) at std_mutex.h:159
    frame #5: 0x000055555758d7fb godot.linuxbsd.editor.dev.x86_64`tvg::Task::operator(this=<unavailable>, tid=<unavailable>)(unsigned int) at tvgTaskScheduler.h:76
    frame #6: 0x000055555758d5a9 godot.linuxbsd.editor.dev.x86_64`tvg::TaskSchedulerImpl::run(this=<unavailable>, i=<unavailable>) at tvgTaskScheduler.cpp:138:13
    frame #7: 0x000055555a90b794 godot.linuxbsd.editor.dev.x86_64`execute_native_thread_routine + 20
    frame #8: 0x00007ffff7f87609 libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:477:8
    frame #9: 0x00007ffff7d57133 libc.so.6`__GI___clone at clone.S:95

It could be a Godot problem. Currently thread handling is buggy in many places. I’ll try to check 4.0 or older versions.

For me problem from issue is fixed

There is also weekly CI job, that runs thorvg on 50000 svg files once per week, and such crashes are clearly visible there

https://github.com/qarmin/SVG-regression-finder/actions -> choose latest action -> test_sanitizer -> Check log from - Run tests