tt-metal: Large DRAM-sharded matmuls reliably hang on wormhole
Describe the bug Large DRAM-sharded matmuls cause wormhole to hang after a few iterations. Reducing the number of columns in the weight matrix to below 32k seems to work around the issue. Sometimes the board seems to need to be reset twice before it can be used again.
To Reproduce Steps to reproduce the behavior:
- Build metal
2.
pytest dram_sharded_hang.py
Expected behavior Progress bar reaches 100 and pytest passes
Screenshots
Metal | INFO | Initializing device 0. Program cache is NOT enabled
Metal | INFO | AI CLK for device 0 is: 1000 MHz
Metal | INFO | MMIO Device 0 : Tunnel 0 : Device 0
Metal | INFO | MMIO Device 0 : Tunnel 0 : Device 4
Metal | INFO | Enabling program cache on device 0
5%|█▌ | 5/100 [00:01<00:17, 5.58it/s]
Please complete the following environment information:
- Reproduced on aus-wh-01
- main (4f9d153d3c22517d5f1f15c486f601ee9a152382 but no particular reason to think this is a recent regression)
Additional context Doesn’t happen on every iteration, Jasmina suggested we file as a suspected di/dt issue but could also be a race in DRAM reading.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (23 by maintainers)
Commits related to this issue
- #13668: Fix FAKE_DEVICE=T3K, 3b functional on n150, n300, t3k — committed to tenstorrent/tt-metal by yieldthought 2 years ago
- #13668: Added deallocate to test model — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Add missing deallocates. This time without breaking accuracy — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Fix single llama single-chip — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Fix Llama 1B for N150/N300/T3k — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Llama 1/3/8B text models working for N150/N300/T3K in a single codebase — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Add TT module for Tile Position Embeddings. Note, load_hook and dynamic_resize are not implemented. Tests pass for llama23 shapes. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #13668: Fix FAKE_DEVICE=T3K, 3b functional on n150, n300, t3k — committed to tenstorrent/tt-metal by yieldthought 2 years ago
- #13668: Added deallocate to test model — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Add missing deallocates. This time without breaking accuracy — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Fix single llama single-chip — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Fix Llama 1B for N150/N300/T3k — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Llama 1/3/8B text models working for N150/N300/T3K in a single codebase — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Add TT module for Tile Position Embeddings. Note, load_hook and dynamic_resize are not implemented. Tests pass for llama23 shapes. — committed to tenstorrent/tt-metal by avoraTT 2 years ago
- #13668: Fix FAKE_DEVICE=T3K, 3b functional on n150, n300, t3k — committed to tenstorrent/tt-metal by yieldthought 2 years ago
- #13668: Added deallocate to test model — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Add missing deallocates. This time without breaking accuracy — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Fix single llama single-chip — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Fix Llama 1B for N150/N300/T3k — committed to tenstorrent/tt-metal by mtairum 2 years ago
- #13668: Llama 1/3/8B text models working for N150/N300/T3K in a single codebase — committed to tenstorrent/tt-metal by mtairum 2 years ago
I tried the above changes on
yugao/mmand it passes without hanging. When this hits main we can close the issue!Data Movement team found an issue with the transaction id usage. I pushed a fix to yugao/mm. I tried the above hang unit test and it seems passing.
@yieldthought is there a model that reliably hang I can test?