tt-metal: WH Hangs after test crash

Add an assert False (after device open) to any test and it seems that WH device is not cleaned up gracefully, the subsequent test run will hang indefinitely trying to read device memory.

For example while developing I ran into a host side error:

RuntimeError: TT_THROW @ tt_metal/impl/allocator/allocator.cpp:107: tt::exception
info:
Out of Memory: Not enough space to allocate {} B {} buffer across {} banks, where each bank needs to store {} B

On a subsequent run of this test it hangs indefinitely on device open:

  * frame #0: 0x00007fffcbe7f9f0 libtt_metal.so`memcpy_from_device(void*, void const*, unsigned long) + 256
    frame #1: 0x00007fffcbe7fbf7 libtt_metal.so`read_block(TTDevice*, unsigned int, unsigned int, unsigned char*, unsigned int) + 423
    frame #2: 0x00007fffcbe8f126 libtt_metal.so`tt_SiliconDevice::read_device_memory(void*, tt_cxy_pair, unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
 const&) + 662
    frame #3: 0x00007fffcbea47b2 libtt_metal.so`tt_SiliconDevice::read_from_device(void*, tt_cxy_pair, unsigned long, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&) + 402
    frame #4: 0x00007fffcc04449c libtt_metal.so`tt::Cluster::read_core(void*, unsigned int, tt_cxy_pair, unsigned long, bool) const at tt_cluster.cpp:463:47
    frame #5: 0x00007fffcc04dda6 libtt_metal.so`tt::llrt::read_hex_vec_from_core(int, tt_xy_pair const&, unsigned long, unsigned int) at llrt.cpp:140:38
    frame #6: 0x00007fffcc04ec87 libtt_metal.so`std::_Function_handler<bool (unsigned long), tt::llrt::internal_::check_if_riscs_on_specified_core_done(int, tt_xy_pair const&, int)::'lambda'(unsigned long)>::_M_i
nvoke(std::_Any_data const&, unsigned long&&) at llrt.cpp:288:58
    frame #7: 0x00007fffcc04f810 libtt_metal.so`tt::llrt::internal_::wait_until_cores_done(int, int, std::unordered_set<tt_xy_pair, std::hash<tt_xy_pair>, std::equal_to<tt_xy_pair>, std::allocator<tt_xy_pair> >&)
 at std_function.h:688:14
    frame #8: 0x00007fffcbf8dc2a libtt_metal.so`tt::tt_metal::Device::initialize_and_launch_firmware() at device.cpp:244:43
    frame #9: 0x00007fffcbf99acf libtt_metal.so`tt::tt_metal::Device::initialize(std::vector<unsigned int, std::allocator<unsigned int> > const&) at device.cpp:793:41
    frame #10: 0x00007fffcbf99e93 libtt_metal.so`tt::tt_metal::Device::Device(int, unsigned char, std::vector<unsigned int, std::allocator<unsigned int> > const&) at device.cpp:65:21
    frame #11: 0x00007fffcbe5cc52 libtt_metal.so`tt::tt_metal::CreateDevice(int, unsigned char, std::vector<unsigned int, std::allocator<unsigned int> > const&) at tt_metal.cpp:635:67
    frame #12: 0x00007fffcbd432ea libttnn.so`ttnn::device::open_device(int) at device.cpp:23:55
  • Note that this issue is tracking the subsequent hang, the former OOM error isn’t a bug I just tried allocating a huge tensor.
  • I can workaround this issue with a warm reset

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (12 by maintainers)

Commits related to this issue

Most upvoted comments

No hang with Grayskull.

It seems to appear when a test/app dies before device is closed. @joelsmithTT I saw some messages on slack where you were looking into this, could you summarize your experiments?

It does, but I’ve worked around it by just having it auto-reset in my script workflow. I appreciate that this isn’t top priority or have someone to look into it, also it doesn’t seem like too many folks are complaining about it, demoting to P2.