tt-metal: Blackhole: Unpacker hanging in `unpack_tilizeA_B_block`

unpack_tilizeA_B_block calls the LLK: llk_unpack_tilizeA_B_block<true, true>(...).

Branch: asarje/bh-rn50-20241205-test Build: ./build_metal.sh --debug Run:

TT_METAL_DPRINT_RISCVS=TR TT_METAL_DPRINT_CORES=0,0 pytest "tests/ttnn/unit_tests/operations/test_maxpool2d.py::test_run_max_pool[dtype=DataType.BFLOAT8_B-dilation=(1, 1)-stride=(2, 2)-padding=(1, 1)-kernel_size=(3, 3)-act_shape=[1, 64, 112, 112]-device_params={'l1_small_size': 24576}]"

This test will hang in a call to the above function (works for initial few iterations). I was able to narrow that down using debug printing. I have limited the grid to 64 cores to match WH, and I couldn’t get smaller grid size as either data doesn’t fit, or the issue does not appear.

Relevant kernels: compute = ttnn/cpp/ttnn/operations/pool/generic/device/kernels/compute/max_pool_multi_core.cpp reader = ttnn/cpp/ttnn/operations/pool/generic/device/kernels/dataflow/reader_max_pool_2d_multi_core_sharded_with_halo_v2.cpp

Likely some recent update broke it because this test was working on BH a about week ago.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (16 by maintainers)

Commits related to this issue

Most upvoted comments

@ttmtrajkovic there were recent changes in tilize/untilize functions to fix attention matmul kernel, however, it shouldn’t cause different behavior between iterations of a kernel. That part is weird, and test being multicore makes it more difficult to debug.

@mywoodstock if you can pinpoint the commit in tt-metal where this started failing, it would help us figure out what went wrong.