tt-metal: Matmul hangs on blackhole (p150) with device profiling calls and quick_push

Describe the bug

The matmul test in tests/didt/test_ff1_matmul.py::test_ff1_matmul hangs when device profiling calls (i.e. DeviceZoneScopedN) are added in the data movement kernel along with a quick_push in the profiler to flush them from L1 to DRAM. This issue is specific to blackhole devices.

Steps to reproduce the issue

  1. Checkout the following branch with the required changes: https://github.com/sohaibnadeemTT/tt-metal/tree/snadeem/device_profiling_hang

  2. Run the following test in tt-metal: pytest ./tests/didt/test_ff1_matmul.py::test_ff1_matmul -k "without_gelu and 1chips" --didt-workload-iterations 1. This test runs normally and passes.

  3. Run again but with device profiling enabled, it will hang: TT_METAL_DEVICE_PROFILER=1 pytest ./tests/didt/test_ff1_matmul.py::test_ff1_matmul -k "without_gelu and 1chips" --didt-workload-iterations 1

Image

Expected behavior

No response

Please complete the following environment information

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

Applying the following diff to the branch removes the hang

--- a/tt_metal/tools/profiler/kernel_profiler.hpp
+++ b/tt_metal/tools/profiler/kernel_profiler.hpp
@@ -319,6 +319,14 @@ __attribute__((noinline)) void quick_push() {
     defined(COMPILE_FOR_BRISC) || defined(COMPILE_FOR_NCRISC) || defined(COMPILE_FOR_ERISC) || \
     defined(COMPILE_FOR_IDLE_ERISC))

+    uint32_t read_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, read_cmd_buf, NOC_CTRL);
+    uint32_t write_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, write_cmd_buf, NOC_CTRL);
+    uint32_t write_reg_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, write_reg_cmd_buf, NOC_CTRL);
+    uint32_t write_at_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, write_at_cmd_buf, NOC_CTRL);
+    if ((read_buf_reg >> 6 & 0x1) || (write_buf_reg >> 6 & 0x1) ||\
+            (write_reg_buf_reg >> 6 & 0x1) || (write_at_buf_reg >> 6 & 0x1)) {return;}
+
+
     SrcLocNameToHash("PROFILER-NOC-QUICK-SEND");
     mark_time_at_index_inlined(wIndex, hash);
     wIndex += PROFILER_L1_MARKER_UINT32_SIZE;

@sohaibnadeemTT It would be good to give the original hang a try as well with this. Just to confirm it on the noc-tracing side as well.

Confirmed that it does not hang with noc tracing on this test as well after applying the changes above.

Applying the following diff to the branch removes the hang

--- a/tt_metal/tools/profiler/kernel_profiler.hpp
+++ b/tt_metal/tools/profiler/kernel_profiler.hpp
@@ -319,6 +319,14 @@ __attribute__((noinline)) void quick_push() {
     defined(COMPILE_FOR_BRISC) || defined(COMPILE_FOR_NCRISC) || defined(COMPILE_FOR_ERISC) || \
     defined(COMPILE_FOR_IDLE_ERISC))

+    uint32_t read_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, read_cmd_buf, NOC_CTRL);
+    uint32_t write_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, write_cmd_buf, NOC_CTRL);
+    uint32_t write_reg_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, write_reg_cmd_buf, NOC_CTRL);
+    uint32_t write_at_buf_reg = NOC_CMD_BUF_READ_REG(noc_index, write_at_cmd_buf, NOC_CTRL);
+    if ((read_buf_reg >> 6 & 0x1) || (write_buf_reg >> 6 & 0x1) ||\
+            (write_reg_buf_reg >> 6 & 0x1) || (write_at_buf_reg >> 6 & 0x1)) {return;}
+
+
     SrcLocNameToHash("PROFILER-NOC-QUICK-SEND");
     mark_time_at_index_inlined(wIndex, hash);
     wIndex += PROFILER_L1_MARKER_UINT32_SIZE;

@sohaibnadeemTT It would be good to give the original hang a try as well with this. Just to confirm it on the noc-tracing side as well.

Some debugging done so far:

  • gdb shows we are waiting for the matmul op to finish on device and a tracy trace comparison (with vs without profiling) also shows that the last op we reach is matmul. Initially, the likely suspects were either L1/DRAM corruption from logging profiling events but these have been ruled out

    • L1 corruption (ruled out)
      • we see the issue even with low L1 usage (smaller matmuls)
      • Increasing the reserved space in L1 (to reduce chance of L1 corruption) has no effect (we still see a hang)
    • DRAM corruption (ruled out)
      • flushing profiling data to the same address every time on DRAM still shows a hang
      • Tried different sizes of push and to different offsets in DRAM (as shown in the code snippet above)

That leaves an issue with the NOC write itself:

  • commenting out both the posted write call and flush call does get rid of the hang, so it related to the noc write
  • changing to a non posted write did not help
  • removing some of the tracing calls (around noc_semaphore_set, noc_semaphore_wait) did make the hang go away
    • changing quick_push to a happen randomly also made the hang go away occasionally
    • adding random delays before quick push did not seem to help though
  • Watcher’s noc address sanitize feature didn’t reveal anything
  • Watcher and gdb on device show that some cores are stuck on NSW (noc_semaphore_wait) while others are in the flush for the posted write in quick_push (profiler_noc_async_flush_posted_write)
  • Commenting out wall clock reg read did not help (known prior issue that reg read followed by noc push caused hang)
  • Updating profiler_noc_async_writes_flushed to match similar function in dataflow_api.h did not help
  • Skipping the first quick_push does makes hang go away:
diff --git a/tt_metal/tools/profiler/kernel_profiler.hpp b/tt_metal/tools/profiler/kernel_profiler.hpp
index 4f7359bfd7..80b5b57411 100644
--- a/tt_metal/tools/profiler/kernel_profiler.hpp
+++ b/tt_metal/tools/profiler/kernel_profiler.hpp
@@ -38,6 +38,7 @@ namespace kernel_profiler {
 
 extern uint32_t wIndex;
 extern uint32_t stackSize;
+uint32_t pushIndex = 0;
 
 extern uint32_t sums[SUM_COUNT];
 extern uint32_t sumIDs[SUM_COUNT];
@@ -319,6 +320,12 @@ __attribute__((noinline)) void quick_push() {
     defined(COMPILE_FOR_BRISC) || defined(COMPILE_FOR_NCRISC) || defined(COMPILE_FOR_ERISC) || \
     defined(COMPILE_FOR_IDLE_ERISC))
 
+    pushIndex++;
+    if (pushIndex == 1) {
+        wIndex = CUSTOM_MARKERS;
+        return;
+    }
+
     SrcLocNameToHash("PROFILER-NOC-QUICK-SEND");
     mark_time_at_index_inlined(wIndex, hash);
     wIndex += PROFILER_L1_MARKER_UINT32_SIZE;

Using a stripped down version of quick_push where there is only a fixed small noc write to a fixed address in DRAM (no address calculation, reading wall clock registers), the test still hangs:

__attribute__((noinline)) void quick_push() {
#if (defined(COMPILE_FOR_NCRISC))  
    const InterleavedAddrGen<true> s = {
        .bank_base_address = 64,
        .page_size = 16 * sizeof(uint32_t) };

    uint64_t dram_bank_dst_noc_addr = s.get_noc_addr(0, 0);

    profiler_noc_async_write_posted(
        reinterpret_cast<uint32_t>(profiler_data_buffer[0]),
        dram_bank_dst_noc_addr,
        4 * sizeof(uint32_t));

    profiler_noc_async_flush_posted_write();

    wIndex = CUSTOM_MARKERS;
#endif
}