tt-metal: BH P100 only: Non-deterministic matmul hang/test failure on blackhole post-commit
Started seeing ND unit test failure beginning Mar 4, first instance occurring on scheduled blackhole workflow:
https://github.com/tenstorrent/tt-metal/actions/runs/13667169213/job/38210795340 Running commit: https://github.com/tenstorrent/tt-metal/commit/ce8bbbadd52d505cd420ed879d9599d8282210ee
[sweep tests: tests/tt_eager/python_api_testing/sweep_tests/pytests/tt_dnn/test_matmul.py#L61](https://github.com/tenstorrent/tt-metal/commit/ce8bbbadd52d505cd420ed879d9599d8282210ee#annotation_32538102265)
test_run_matmul_test[BFLOAT16-input_shapes8]
AssertionError: matmul test failed with input shape [[1, 2, 4416, 1024], [1, 1, 1024, 6048]]. Max ATOL Delta: 1.1097561234743666e+23, Max RTOL Delta: inf, PCC: 0.0002924033844466211, PCC check failed
Also exhibits itself as a hang that times out after 45min https://github.com/tenstorrent/tt-metal/actions/runs/13688234850/job/38276945280
Thu, 06 Mar 2025 05:34:23 GMT
tests/tt_eager/python_api_testing/sweep_tests/pytests/tt_dnn/test_matmul.py::test_run_matmul_test[BFLOAT16-input_shapes3] Metal | INFO | Initializing device 0. Program cache is NOT enabled
Thu, 06 Mar 2025 05:34:23 GMT
Device | INFO | For Blackhole hardcode AICLK to 1350 MHz due to lack of ARC message support
Thu, 06 Mar 2025 05:34:23 GMT
Metal | INFO | AI CLK for device 0 is: 1350 MHz
Thu, 06 Mar 2025 05:34:23 GMT
2025-03-06 05:34:23.331 | INFO | tests.tt_eager.python_api_testing.sweep_tests.run_pytorch_ci_tests:run_single_pytorch_test:68 - Running with shape: [[1, 2, 4608, 1024], [1, 1, 1024, 6144]] on device: 0
Thu, 06 Mar 2025 06:16:13 GMT
Error: The action has timed out.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 77 (74 by maintainers)
Commits related to this issue
- Fix matmul ND hang due to packer race condition (#20415) ### Ticket [Link to Github Issue ](https://github.com/tenstorrent/tt-metal/issues/18779#issuecomment-2782834310) ### Problem description Mat... — committed to tenstorrent/tt-metal by amahmudTT 2 years ago
- Fix matmul ND hang due to packer race condition (#20415) ### Ticket [Link to Github Issue ](https://github.com/tenstorrent/tt-metal/issues/18779#issuecomment-2782834310) ### Problem description Mat... — committed to tenstorrent/tt-metal by amahmudTT 2 years ago
Applying test FW
fw_pack_80.16.0.124_p100a_harvesting.fwbundleon a P100A (yyz-lab-125) and ran the repro case for many hours without seeing any hangs. This might be 👻 issue being chased on a Scrappy.REG2FLOPis definitely a THCON instruction. The# FIXME: should this be CFG? ckernels STALLWAIT against CFG to block thiscomment should be deleted, as all it does is sow confusion and doubt.I was able to reduce the repro case further, just to a single matmul.
32 run
to force L1 to have bad state and then run the matmul. Might need to run a few times, although I always ran into it within one or two tries.