tt-metal: BH MMul Hang in Resnet
@ncvetkovicTT With removing the
matmul_blockworkaround and adding your workaround commit, the model still hangs.I am using this branch:
asarje/rn50-bh-largebatchTest:pytest "tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=32-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]"This branch currently has the matmul workaround skipped in the matmul kernel, but not in the conv kernel. The conv kernel is:
ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize_col_major_out_blocks.cppLine 271 (https://github.com/tenstorrent/tt-metal/blob/asarje/rn50-bh-largebatch/ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize_col_major_out_blocks.cpp#L271-L277)
Originally posted by @mywoodstock in #16439
This instance is successful when the workaround is applied; but workaround is not confirmed to solve the root cause.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (14 by maintainers)
OK removing the matmul block workarounds from both matmul and conv kernels looks good! All three cases with batch=16, 20 and 32 work without hangs. Here is the latest PR with these updates: https://github.com/tenstorrent/tt-metal/pull/17985
The issue https://github.com/tenstorrent/tt-metal/issues/17099 gets fixed only when compiler option for sfpi is changed and the disable_gathering() related changes are made, one does not suffice, so I tested this issue with those changes and like that issue I see a similar failure related to assert.
Okay so here is what I realized, if I encounter a hang, then no matter what I do from the above mentioned steps, i encounter a hang if I do not restart the card, but if I restart then following the steps outlined does not produce a hang.
Cannot reproduce a hang on
yyzo-bh-15. I checked out the latestasarje/rn50-bh-largebatch, have a fresh build and ranpytest "tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[pretrained_weight_false-batch_size=32-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_params={'l1_small_size': 24576}]"but the test passes quickly.EDIT: I didn’t comment out the workaround in
ttnn/cpp/ttnn/operations/conv/conv2d/device/kernels/conv_bmm_tilize_col_major_out_blocks.cppDid not work