tt-metal: [Bug Report] v0 drop of Llama3.1-70B on LoudBox hangs on latest firmware [80.13.1.0] when running full dataset evaluation
Describe the bug When running the latest drop (v0) of Llama3.1-70B on LoudBox with firmware 80.13.1.0 we experience a hang (presumably di/dt). Running on firmware with downgraded clock of 900MHz, there is no hang. Validated on 80.13.1.1 and 80.10.1.0.
To Reproduce Steps to reproduce the behavior:
- Install tt-metal + vllm for the latest drop of Llama3.1-70B on LoudBox following the instructions from: https://github.com/tenstorrent/vllm/tree/dev/tt_metal (this includes downloading the model weights)
- With the vllm environment from step 1. active, from your home directory, run:
git clone https://github.com/tenstorrent/tt-inference-server.git
cd tt-inference-server/vllm-tt-metal-llama3-70b/src
pip install pyjwt==2.7.0
export JWT_SECRET=password
WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python run_vllm_api_server.py
- In a new terminal, activate the environment from step 1., and after step 2. has successfully deployed, run:
export JWT_SECRET=password
export CACHE_ROOT=$PWD
python example_openai_client_alpaca_eval.py
If running on 80.13.1.0 (1000MHz), the model will eventually hang. This usually occurs at a random point.
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software (tt-metal): 3859041
- Version of software (vllm): 384f179
- TT-KMD: 1.29
- Firmware:
Additional context Tested this on a few systems, all experiencing the same behaviour. Going from 1000MHz -> 900MHz was able to stabilize.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 33 (30 by maintainers)
Closing since this is now a stale issue. We have not seen any issues reported with reworked cards in the field. Firmware releases have been updated since.
I’ll reach out to Zlatomir in Serbia and we’ll track down someone on the Software team who knows. Stand by.
Update: After looking at Milan’s server logs from the crash, I believe the hang occurs during decode. In an attempt to simplify the repro, I am currently running a traced, decode-only 24-hour stress test on the following systems:
sjc-snva-t3010f10cs01f10cs02All of these systems have been flashed to
80.13.1.0.