tt-metal: Race condition in FDMeshCommandQueue

Describe the bug

From time to time I see crashes like this one below. It’s not reproducible, but shows up from time to time. It might be something on our end, might be something in tt-metal. But these are our tests, which are not doing anything sophisticated, just serial execution of isolated ops. Any ideas how to debug this are welcome. If you have any switches to enable any “sanitizers” for this kind of race conditions probably?

terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_FATAL @ /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/tt_metal/impl/event/dispatch.cpp:192: event_completed == event_descriptor.event_id
info:
Event Order Issue: expected to read back completion signal for event 25 but got 1065369472!
backtrace:
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x2bad8b) [0x7316fc4bad8b]
 --- tt::tt_metal::event_dispatch::read_events_from_completion_queue(tt::tt_metal::ReadEventDescriptor&, int, unsigned short, unsigned char, tt::tt_metal::SystemMemoryManager&)
 --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue_event(tt::tt_metal::distributed::MeshReadEventDescriptor&)
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x729c70) [0x7316fc929c70]
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x729c3d) [0x7316fc929c3d]
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x729bed) [0x7316fc929bed]
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x729acd) [0x7316fc929acd]
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x729a16) [0x7316fc929a16]
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x72862b) [0x7316fc92862b]
 --- tt::tt_metal::distributed::FDMeshCommandQueue::read_completion_queue()
 --- void std::__invoke_impl<void, void (tt::tt_metal::distributed::FDMeshCommandQueue::*)(), tt::tt_metal::distributed::FDMeshCommandQueue*>(std::__invoke_memfun_deref, void (tt::tt_metal::distributed::FDMeshCommandQueue::*&&)(), tt::tt_metal::distributed::FDMeshCommandQueue*&&)
 --- std::__invoke_result<void (tt::tt_metal::distributed::FDMeshCommandQueue::*)(), tt::tt_metal::distributed::FDMeshCommandQueue*>::type std::__invoke<void (tt::tt_metal::distributed::FDMeshCommandQueue::*)(), tt::tt_metal::distributed::FDMeshCommandQueue*>(void (tt::tt_metal::distributed::FDMeshCommandQueue::*&&)(), tt::tt_metal::distributed::FDMeshCommandQueue*&&)
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x72f992) [0x7316fc92f992]
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x72f955) [0x7316fc92f955]
 --- /workspaces/oxpython/components/backends/tenstorrent/externals/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x72f839) [0x7316fc92f839]
 --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe62b3) [0x7316fd2792b3]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7316fcf17ac3]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7316fcfa9850]

Steps to reproduce the issue

No easy way to reproduce this and that’s the problem.

Expected behavior

No response

Please complete the following environment information

  • Ubuntu 22.04
  • tt-metal stack of fairly recent version (several day behind main).

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 27 (24 by maintainers)

Commits related to this issue

Most upvoted comments

I’m running in the Docker container and it doesn’t have rights to change ASLR. I disabled ASLR on the host and rerun the build, now it works.

Running the test now. It runs noticeably slower and doesn’t fail so far. I’ll keep running it.