tt-metal: ND multi-chip WH llama/Mistral hangs on CI

Update

Commit https://github.com/tenstorrent/tt-metal/commit/25e8e02f74b45dfd71a5be1ba78cf94fb3c52274 seems to be the culprit @TT-BrianLiu

Reproduce

Run the TT-Transformers demo with batch=32 for many iterations (ci config). Saw hanging for Llama 8B and Mistral7B on N300/T3K.

For llama8B If using a cloud machine: /mnt/MLPerf/tt_dnn-models/llama/Meta-Llama-3.1-8B-Instruct/ otherwise should be somewhere under proj_sw/user_dev/...

LLAMA_DIR=<PATH_TO_LLAMA_WEIGHTS> WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml pytest models/tt_transformers/demo/simple_text_demo.py::test_demo_text -k performance-ci-32

For Mistral-7B check @pprajapatiTT comment: https://github.com/tenstorrent/tt-metal/issues/21519#issuecomment-2846082576

Original description

N300/T3K Llama models are ND hanging on CI during decode. Unclear when it started happening, appears to have been an issue for multiple days already.

Ex t3k demos: https://github.com/tenstorrent/tt-metal/actions/runs/14768713913/job/41465090817#step:9:2350 https://github.com/tenstorrent/tt-metal/actions/runs/14767852291/job/41462881724#step:9:8419

Ex n300 demos: https://github.com/tenstorrent/tt-metal/actions/runs/14775010169/job/41481778903#step:9:43742 https://github.com/tenstorrent/tt-metal/actions/runs/14721424475/job/41316050235#step:9:44035

cc @mtairum @uaydonat

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (22 by maintainers)

Most upvoted comments

Steps to reproduce using Mistral7B:

  1. Take this PR https://github.com/tenstorrent/tt-metal/pull/21306

  2. Run the Mistral7B model,

    2.1. Reserve N300 machine using IRD 2.2 You need build tt-metal (build_metal.sh) and have python env created with create_venv.sh 2.3 set your <user dir> and get hugging-face token with access to https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 2.4 Run run following commands.

                deactivate
		cd /proj_sw/user_dev/<user dir>/tt-metal
		source python_env/bin/activate

		export ARCH_NAME=wormhole_b0
		export TT_METAL_HOME=/proj_sw/user_dev/<user dir>/tt-metal
		export PYTHONPATH=/proj_sw/user_dev/<user dir>/tt-metal
		export HF_HOME=/proj_sw/user_dev/hf_data
		export WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml
		cd $PYTHONPATH

		export HF_TOKEN=<your token>
		export HF_MODEL=mistralai/Mistral-7B-Instruct-v0.3
  1. run the test with this command

pytest models/tt_transformers/demo/simple_text_demo.py -k performance-ci-32

I have seen it hangs every single time. After reverting the patch the test passed.

I am running more cycles of the test to make sure reverting the patch works everytime, stay tuned.

@pprajapatiTT did you run multiple iterations to confirm since this looked ND? If so, please post the repro instructions (test command) here for @TT-BrianLiu so he can try to reproduce.