tt-metal: ND hang of SD unit tests on N300 device
Running SD unit tests with WH_ARCH_YAML on N300 devices non-deterministically hangs.
To repro the issue, switch to main branch and run the following on N300 device:
WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml pytest tests/ttnn/integration_tests/stable_diffusion
EDIT:
Running the same test with enabling watcher in the fast-dispatch CI raises the std::runtime_error below on tests/ttnn/integration_tests/stable_diffusion/test_cross_attn_up_block_2d.py (full log):
terminate called after throwing an instance of 'std::runtime_error'
what(): Read 0xffffffff from ARC scratch[6]: auto-reset succeeded.
Fatal Python error: Aborted
Thread 0x00007f3744ff9700 (most recent call first):
File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 306 in wait
File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 558 in wait
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run
File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 890 in _bootstrap
Thread 0x00007f38db2c1740 (most recent call first):
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/decorators.py", line 410 in call_wrapper
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/decorators.py", line 616 in call_wrapper
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/decorators.py", line 693 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attention.py", line 306 in time_sharded_attention
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attention.py", line 471 in get_attention_scores_opt
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attention.py", line 706 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_basic_transformer_block.py", line 90 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_transformer_2d.py", line 298 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attn_upblock.py", line 153 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/tests/ttnn/integration_tests/stable_diffusion/test_cross_attn_up_block_2d.py", line 321 in test_cross_attn_up_block_2d_512x512
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 1789 in runtest
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 260 in <lambda>
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 131 in runtestprotocol
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 167 in main
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 190 in console_main
File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/bin/pytest", line 8 in <module>
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 47 (35 by maintainers)
Commits related to this issue
- #7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue #7560 — committed to tenstorrent/tt-metal by vtangTT 3 years ago
- #7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue #7560 — committed to tenstorrent/tt-metal by vtangTT 3 years ago
- #7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue #7560 — committed to tenstorrent/tt-metal by vtangTT 3 years ago
- #7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue #7560 — committed to tenstorrent/tt-metal by vtangTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Add SD tests to FD nightly with legacy pass — committed to tenstorrent/tt-metal by mtatsumiTT 3 years ago
- #7560: Slow down MMs when running non-perf tests to avoid ND hang. No hang observed on BM machine where perf tests are run. — committed to tenstorrent/tt-metal by AleksKnezevic 3 years ago
- #7560: Slow down MMs when running non-perf tests to avoid ND hang. No hang observed on BM machine where perf tests are run. — committed to tenstorrent/tt-metal by AleksKnezevic 3 years ago
- #7560 Revert temp chane. — committed to tenstorrent/tt-metal by AleksKnezevic 3 years ago
- #7560: Slow down MMs when running non-perf tests to avoid ND hang. No hang observed on BM machine where perf tests are run. — committed to tenstorrent/tt-metal by AleksKnezevic 3 years ago
There was an assert that was commented out later on: https://github.com/tenstorrent/tt-metal/commit/d46d435e9a3cbe5d802fc5d6bb02fb261c52838cm, I just commented it out, and it worked.
Overall, the new fw helps a lot, most of the repro examples we had are fixed, but we still have a couple of remaining issues still hanging less frequently.
Regarding SD specifically, the small repro examples we had from this thread are fixed, however I was unable to reproduce a hang running a full demo even with the old firmware (80.8.12) on the https://github.com/tenstorrent/tt-metal/commit/e7a43ccf0852eedfda983441abe6853892148143 commit. So I can’t really say nd hangs from demo are resolved with new firmware, when I couldn’t catch the nd hang in the old setup either.
I am taking a look into why this commit would cause issues specifically for SD, unless it is causing timing issues for a different bug/problem or SD is hitting some kind of very specific corner case. Currently it is not ideal to try and revert these changes.
Yes I’ll create an issue to track it 👍 Unless some says otherwise, I will close this once I create the 2 follow on issues.
Assigning myself to this since I’m taking ownership of SD for now. I’ll test out @tt-aho’s fix to see if we can re-enable these tests on CI.
The fix is on main. So retesting/re-enabling of SD tests can be done
After the fix is merged, I guess it should be checked if the ND hangs still exist or not with the latest fw. However, we won’t be doing these checks and removing workarounds for all models, model owners will need to check if they can remove workarounds from the models. We will let everybody know once the software fix and fw testing is done, so that model owners can addess this.
I have a fix for this in this pr #10911. Didn’t enable the test in CI though. Will you enable it after doing di/dt testing on latest main @s-jovic ?
I’ve isolated the specific change that is causing the hang, though still unknown why it’s causing it.
This commit changed which FD core is used for what, specifically eth core (0, 4) was used for dispatcher, and (0, 5) was used for prefetcher. Was changed so that (0, 4) is now prefetcher and (0, 5) is now dispatcher. This should have no real change in functionality and other tests/models are functional, so something weird is happening as a result of this (potentially some timing/race issue).
I am saying https://github.com/tenstorrent/tt-metal/commit/e7a43ccf0852eedfda983441abe6853892148143 passes (commit before https://github.com/tenstorrent/tt-metal/commit/3e7273c45b196881d39c9fe5693bc2c70706bcd2), I assumed the two commits (https://github.com/tenstorrent/tt-metal/commit/3e7273c45b196881d39c9fe5693bc2c70706bcd2 and https://github.com/tenstorrent/tt-metal/commit/db25a356488a5e4b10752cdb87189c0e33aa238d) were a part of the same change, I wasn’t clear.
Yup, those are 2 changes we needed to apply to get rid of di/dt (subblocks and grid size reduction) (Just realised that ‘furthermore’ is a bad choice of words for what I was trying to say 😄)
Hang is still present on both tests. I rebased and pushed
aknezevic/repro_MM_hangThe watcher log of the previous test is different than the one we saw previously. Perhaps a different style of hang? That one can be reproed using a 5-6 op unit test on the same branch:
WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml TT_METAL_WATCHER=1 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 pytest tests/ttnn/integration_tests/stable_diffusion/test_sharded_attention.py::test_time_sharded_attnention -k 4096@jliangTT, can you please coordinate?
I have found a way to reliably repro on the submodule, trying to further isolate.