tt-metal: BH P100 only: Non-deterministic matmul hang/test failure on blackhole post-commit

Image https://superset.tenstorrent.com/superset/dashboard/59/?native_filters_key=_D0HOibZYvdKZqAmmPSpNlYc84O49jnnrClpEICsnO93dTgLKbdqYvikMkzTBYBU

Started seeing ND unit test failure beginning Mar 4, first instance occurring on scheduled blackhole workflow:

https://github.com/tenstorrent/tt-metal/actions/runs/13667169213/job/38210795340 Running commit: https://github.com/tenstorrent/tt-metal/commit/ce8bbbadd52d505cd420ed879d9599d8282210ee

[sweep tests: tests/tt_eager/python_api_testing/sweep_tests/pytests/tt_dnn/test_matmul.py#L61](https://github.com/tenstorrent/tt-metal/commit/ce8bbbadd52d505cd420ed879d9599d8282210ee#annotation_32538102265)
test_run_matmul_test[BFLOAT16-input_shapes8]

AssertionError: matmul test failed with input shape [[1, 2, 4416, 1024], [1, 1, 1024, 6048]]. Max ATOL Delta: 1.1097561234743666e+23, Max RTOL Delta: inf, PCC: 0.0002924033844466211, PCC check failed

Also exhibits itself as a hang that times out after 45min https://github.com/tenstorrent/tt-metal/actions/runs/13688234850/job/38276945280

Thu, 06 Mar 2025 05:34:23 GMT
tests/tt_eager/python_api_testing/sweep_tests/pytests/tt_dnn/test_matmul.py::test_run_matmul_test[BFLOAT16-input_shapes3]                   Metal | INFO     | Initializing device 0. Program cache is NOT enabled
Thu, 06 Mar 2025 05:34:23 GMT
                 Device | INFO     | For Blackhole hardcode AICLK to 1350 MHz due to lack of ARC message support
Thu, 06 Mar 2025 05:34:23 GMT
                  Metal | INFO     | AI CLK for device 0 is:   1350 MHz
Thu, 06 Mar 2025 05:34:23 GMT
2025-03-06 05:34:23.331 | INFO     | tests.tt_eager.python_api_testing.sweep_tests.run_pytorch_ci_tests:run_single_pytorch_test:68 - Running with shape: [[1, 2, 4608, 1024], [1, 1, 1024, 6144]] on device: 0
Thu, 06 Mar 2025 06:16:13 GMT
Error: The action has timed out.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 77 (74 by maintainers)

Commits related to this issue

Most upvoted comments

Applying test FW fw_pack_80.16.0.124_p100a_harvesting.fwbundle on a P100A (yyz-lab-125) and ran the repro case for many hours without seeing any hangs. This might be 👻 issue being chased on a Scrappy.

In Wormhole we update the register using REG2FLOP which is a THCON instruction (although there is a question asked as comments in the assembly asking whether it is THCON or CFG)

REG2FLOP is definitely a THCON instruction. The # FIXME: should this be CFG? ckernels STALLWAIT against CFG to block this comment should be deleted, as all it does is sow confusion and doubt.

I was able to reduce the repro case further, just to a single matmul.

  1. change the 0 in clear_l1_state() to 0xdeadbeef (and compile in release build and disable watcher)
diff --git a/tt_metal/impl/device/device.cpp b/tt_metal/impl/device/device.cpp
index a600e1e4ae..b5e67d65dc 100644
--- a/tt_metal/impl/device/device.cpp
+++ b/tt_metal/impl/device/device.cpp
@@ -899,12 +899,30 @@ void Device::clear_l1_state() {
     // Clear all clearable Tensix and Eth L1
     CoreCoord logical_grid_size = this->logical_grid_size();
     TT_ASSERT(this->l1_size_per_core() % sizeof(uint32_t) == 0);
-    std::vector<uint32_t> zero_vec(this->l1_size_per_core() / sizeof(uint32_t), 0);
+    std::vector<uint32_t> zero_vec(this->l1_size_per_core() / sizeof(uint32_t), 0xdeadbeef);

32 run

TT_METAL_CLEAR_L1=1 pytest tests/tt_eager/python_api_testing/sweep_tests/pytests/tt_dnn/test_matmul.py::test_run_matmul_test[BFLOAT16-input_shapes8]

to force L1 to have bad state and then run the matmul. Might need to run a few times, although I always ran into it within one or two tries.