tt-metal: BH MMul Hang in Resnet

@ncvetkovicTT With removing the matmul_block workaround and adding your workaround commit, the model still hangs.

I am using this branch: asarje/rn50-bh-largebatch Test: pytest "tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=32-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]"

This branch currently has the matmul workaround skipped in the matmul kernel, but not in the conv kernel. The conv kernel is: ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize_col_major_out_blocks.cpp Line 271 (https://github.com/tenstorrent/tt-metal/blob/asarje/rn50-bh-largebatch/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize_col_major_out_blocks.cpp#L271-L277)

Originally posted by @mywoodstock in #16439

This instance is successful when the workaround is applied; but workaround is not confirmed to solve the root cause.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (14 by maintainers)

Commits related to this issue

Most upvoted comments

OK removing the matmul block workarounds from both matmul and conv kernels looks good! All three cases with batch=16, 20 and 32 work without hangs. Here is the latest PR with these updates: https://github.com/tenstorrent/tt-metal/pull/17985

The issue https://github.com/tenstorrent/tt-metal/issues/17099 gets fixed only when compiler option for sfpi is changed and the disable_gathering() related changes are made, one does not suffice, so I tested this issue with those changes and like that issue I see a similar failure related to assert.

tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=32-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}] 2025-02-26 09:51:30.358 | DEBUG    | ttnn:manage_config:91 - Set ttnn.CONFIG.report_name to tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=32-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]: 2025-02-26 09:51:30 (UTC)
                 Device | INFO     | Opening user mode device driver
  Detecting chips (found 1)
2025-02-26 09:51:30.387 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.29.0, IOMMU: disabled
2025-02-26 09:51:30.493 | INFO     | SiliconDriver   - Detected PCI devices: [0]
2025-02-26 09:51:30.493 | INFO     | SiliconDriver   - Using local chip ids: {0} and remote chip ids {}
2025-02-26 09:51:31.028 | INFO     | SiliconDriver   - Device: 0 Mapped iATU region 0 from 0x0 to 0x3fffffff to 0x200000000
                  Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                 Device | INFO     | For Blackhole hardcode AICLK to 800 MHz due to lack of ARC message support
                  Metal | INFO     | AI CLK for device 0 is:   800 MHz
                 Always | FATAL    | Event Order Issue: expected to read back completion signal for event 4 but got 640258074!
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_ASSERT @ /proj_sw/user_dev/amahmud/tt-metal/tt_metal/impl/dispatch/hardware_command_queue.cpp:648: event_completed == read_descriptor.event_id
info:
Event Order Issue: expected to read back completion signal for event 4 but got 640258074!
backtrace:

@amahmudTT Can you confirm that what you did is the following:

  1. Go to Abhinav’s branch from above
  2. Comment out the workaround (nop loop) in ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize_col_major_out_blocks.cpp
  3. Put disable_gathering() in trisc.cc
  4. Comment out disable_gathering() and enable_gathering() inside both definitions of load_replay_buf
  5. Run pytest "tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=32-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]"
  6. Observe the hang ?

The reason I ask is that I get a pass in this case on yyzo-bh-15, although I do get a hang for some other cases. See the table for details: https://docs.google.com/spreadsheets/d/1i0bfPhjZcUI1ce_CTE44a-VMnvquQjY-kRPAb5oMXow/edit?gid=0#gid=0

Okay so here is what I realized, if I encounter a hang, then no matter what I do from the above mentioned steps, i encounter a hang if I do not restart the card, but if I restart then following the steps outlined does not produce a hang.

Cannot reproduce a hang on yyzo-bh-15. I checked out the latest asarje/rn50-bh-largebatch, have a fresh build and ran pytest "tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=32-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]" but the test passes quickly.

EDIT: I didn’t comment out the workaround in ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize_col_major_out_blocks.cpp

But the hang persisted.

What if you do disable_gathering() in trisc.cc and then comment out the two calls in to disable/enable gathering in load_replay_buf?

Did not work