tt-metal: ND multi-chip WH llama/Mistral hangs on CI
Update
Commit https://github.com/tenstorrent/tt-metal/commit/25e8e02f74b45dfd71a5be1ba78cf94fb3c52274 seems to be the culprit @TT-BrianLiu
Reproduce
Run the TT-Transformers demo with batch=32 for many iterations (ci config). Saw hanging for Llama 8B and Mistral7B on N300/T3K.
For llama8B
If using a cloud machine: /mnt/MLPerf/tt_dnn-models/llama/Meta-Llama-3.1-8B-Instruct/ otherwise should be somewhere under proj_sw/user_dev/...
LLAMA_DIR=<PATH_TO_LLAMA_WEIGHTS> WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml pytest models/tt_transformers/demo/simple_text_demo.py::test_demo_text -k performance-ci-32
For Mistral-7B check @pprajapatiTT comment: https://github.com/tenstorrent/tt-metal/issues/21519#issuecomment-2846082576
Original description
N300/T3K Llama models are ND hanging on CI during decode. Unclear when it started happening, appears to have been an issue for multiple days already.
Ex t3k demos: https://github.com/tenstorrent/tt-metal/actions/runs/14768713913/job/41465090817#step:9:2350 https://github.com/tenstorrent/tt-metal/actions/runs/14767852291/job/41462881724#step:9:8419
Ex n300 demos: https://github.com/tenstorrent/tt-metal/actions/runs/14775010169/job/41481778903#step:9:43742 https://github.com/tenstorrent/tt-metal/actions/runs/14721424475/job/41316050235#step:9:44035
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (22 by maintainers)
Steps to reproduce using Mistral7B:
Take this PR
https://github.com/tenstorrent/tt-metal/pull/21306Run the Mistral7B model,
2.1. Reserve N300 machine using IRD 2.2 You need build tt-metal (build_metal.sh) and have python env created with create_venv.sh 2.3 set your <user dir> and get hugging-face token with access to https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 2.4 Run run following commands.
pytest models/tt_transformers/demo/simple_text_demo.py -k performance-ci-32I have seen it hangs every single time. After reverting the patch the test passed.
I am running more cycles of the test to make sure reverting the patch works everytime, stay tuned.
@pprajapatiTT did you run multiple iterations to confirm since this looked ND? If so, please post the repro instructions (test command) here for @TT-BrianLiu so he can try to reproduce.