tt-metal: Large DRAM-sharded matmuls reliably hang on wormhole

Describe the bug Large DRAM-sharded matmuls cause wormhole to hang after a few iterations. Reducing the number of columns in the weight matrix to below 32k seems to work around the issue. Sometimes the board seems to need to be reset twice before it can be used again.

To Reproduce Steps to reproduce the behavior:

  1. Build metal 2.pytest dram_sharded_hang.py

Expected behavior Progress bar reaches 100 and pytest passes

Screenshots

                  Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 0 is:   1000 MHz
                  Metal | INFO     | MMIO Device 0 : Tunnel 0 : Device 0
                  Metal | INFO     | MMIO Device 0 : Tunnel 0 : Device 4
                  Metal | INFO     | Enabling program cache on device 0
  5%|█▌                          | 5/100 [00:01<00:17,  5.58it/s]

Please complete the following environment information:

  • Reproduced on aus-wh-01
  • main (4f9d153d3c22517d5f1f15c486f601ee9a152382 but no particular reason to think this is a recent regression)

Additional context Doesn’t happen on every iteration, Jasmina suggested we file as a suspected di/dt issue but could also be a race in DRAM reading.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (23 by maintainers)

Commits related to this issue

Most upvoted comments

I tried the above changes on yugao/mm and it passes without hanging. When this hits main we can close the issue!

Data Movement team found an issue with the transaction id usage. I pushed a fix to yugao/mm. I tried the above hang unit test and it seems passing.

@yieldthought is there a model that reliably hang I can test?