tt-metal: Blackhole: Unpacker hanging in `unpack_tilizeA_B_block`
unpack_tilizeA_B_block calls the LLK: llk_unpack_tilizeA_B_block<true, true>(...).
Branch: asarje/bh-rn50-20241205-test
Build: ./build_metal.sh --debug
Run:
TT_METAL_DPRINT_RISCVS=TR TT_METAL_DPRINT_CORES=0,0 pytest "tests/ttnn/unit_tests/operations/test_maxpool2d.py::test_run_max_pool[dtype=DataType.BFLOAT8_B-dilation=(1, 1)-stride=(2, 2)-padding=(1, 1)-kernel_size=(3, 3)-act_shape=[1, 64, 112, 112]-device_params={'l1_small_size': 24576}]"
This test will hang in a call to the above function (works for initial few iterations). I was able to narrow that down using debug printing. I have limited the grid to 64 cores to match WH, and I couldn’t get smaller grid size as either data doesn’t fit, or the issue does not appear.
Relevant kernels:
compute = ttnn/cpp/ttnn/operations/pool/generic/device/kernels/compute/max_pool_multi_core.cpp
reader = ttnn/cpp/ttnn/operations/pool/generic/device/kernels/dataflow/reader_max_pool_2d_multi_core_sharded_with_halo_v2.cpp
Likely some recent update broke it because this test was working on BH a about week ago.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (16 by maintainers)
Commits related to this issue
- #15756 Pull in BH LLK fix for maxpool hang — committed to tenstorrent/tt-metal by nvelickovicTT 2 years ago
- #15756 Pull in BH LLK fix for maxpool hang — committed to tenstorrent/tt-metal by nvelickovicTT 2 years ago
- #15756 Pull in BH LLK fix for maxpool hang (#16663) ### Ticket [Link to Github Issue](https://github.com/tenstorrent/tt-metal/issues/15756) ### Problem description Maxpool OP on BH hangs. #... — committed to tenstorrent/tt-metal by nvelickovicTT 2 years ago
- #15756 Pull in BH LLK fix for maxpool hang (#16663) ### Ticket [Link to Github Issue](https://github.com/tenstorrent/tt-metal/issues/15756) ### Problem description Maxpool OP on BH hangs. #... — committed to tenstorrent/tt-metal by nvelickovicTT 2 years ago
@ttmtrajkovic there were recent changes in tilize/untilize functions to fix attention matmul kernel, however, it shouldn’t cause different behavior between iterations of a kernel. That part is weird, and test being multicore makes it more difficult to debug.
@mywoodstock if you can pinpoint the commit in tt-metal where this started failing, it would help us figure out what went wrong.