tt-metal: Matmul hangs on blackhole (p150) with device profiling calls and quick_push
Describe the bug
The matmul test in tests/didt/test_ff1_matmul.py::test_ff1_matmul hangs when device profiling calls (i.e. DeviceZoneScopedN) are added in the data movement kernel along with a quick_push in the profiler to flush them from L1 to DRAM. This issue is specific to blackhole devices.
Steps to reproduce the issue
-
Checkout the following branch with the required changes: https://github.com/sohaibnadeemTT/tt-metal/tree/snadeem/device_profiling_hang
-
Run the following test in tt-metal:
pytest ./tests/didt/test_ff1_matmul.py::test_ff1_matmul -k "without_gelu and 1chips" --didt-workload-iterations 1. This test runs normally and passes. -
Run again but with device profiling enabled, it will hang:
TT_METAL_DEVICE_PROFILER=1 pytest ./tests/didt/test_ff1_matmul.py::test_ff1_matmul -k "without_gelu and 1chips" --didt-workload-iterations 1
Expected behavior
No response
Please complete the following environment information
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- #22578: Fix blackhole hangs on matmul when device profiling enabled — committed to tenstorrent/tt-metal by bgrady-tt a year ago
- #22578: Fix blackhole hangs on matmul when device profiling enabled — committed to tenstorrent/tt-metal by bgrady-tt a year ago
- Fix Blackhole profiler quick push hang when linked multicast is used (#23349) ### Ticket #22578 : Fix blackhole hangs when device profiling with noc tracing is enabled ### Problem description Enabli... — committed to tenstorrent/tt-metal by bgrady-tt a year ago
- Fix Blackhole profiler quick push hang when linked multicast is used (#23349) ### Ticket #22578 : Fix blackhole hangs when device profiling with noc tracing is enabled ### Problem description Enabli... — committed to tenstorrent/tt-metal by bgrady-tt a year ago
- [DM]: Reintroducing BH profiler quick push (#23424) ### Ticket [Link to Github Issue](https://github.com/tenstorrent/tt-metal/issues/22578) ### Problem description The quick push fix, introduced [he... — committed to tenstorrent/tt-metal by atatuzunerTT a year ago
Confirmed that it does not hang with noc tracing on this test as well after applying the changes above.
Applying the following diff to the branch removes the hang
@sohaibnadeemTT It would be good to give the original hang a try as well with this. Just to confirm it on the noc-tracing side as well.
Some debugging done so far:
gdb shows we are waiting for the matmul op to finish on device and a tracy trace comparison (with vs without profiling) also shows that the last op we reach is matmul. Initially, the likely suspects were either L1/DRAM corruption from logging profiling events but these have been ruled out
That leaves an issue with the NOC write itself:
Using a stripped down version of quick_push where there is only a fixed small noc write to a fixed address in DRAM (no address calculation, reading wall clock registers), the test still hangs: