tt-metal: [Bug Report] v0 drop of Llama3.1-70B on LoudBox hangs on latest firmware [80.13.1.0] when running full dataset evaluation

Describe the bug When running the latest drop (v0) of Llama3.1-70B on LoudBox with firmware 80.13.1.0 we experience a hang (presumably di/dt). Running on firmware with downgraded clock of 900MHz, there is no hang. Validated on 80.13.1.1 and 80.10.1.0.

To Reproduce Steps to reproduce the behavior:

  1. Install tt-metal + vllm for the latest drop of Llama3.1-70B on LoudBox following the instructions from: https://github.com/tenstorrent/vllm/tree/dev/tt_metal (this includes downloading the model weights)
  2. With the vllm environment from step 1. active, from your home directory, run:
git clone https://github.com/tenstorrent/tt-inference-server.git
cd tt-inference-server/vllm-tt-metal-llama3-70b/src
pip install pyjwt==2.7.0
export JWT_SECRET=password
WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python run_vllm_api_server.py
  1. In a new terminal, activate the environment from step 1., and after step 2. has successfully deployed, run:
export JWT_SECRET=password
export CACHE_ROOT=$PWD
python example_openai_client_alpaca_eval.py

If running on 80.13.1.0 (1000MHz), the model will eventually hang. This usually occurs at a random point.

Please complete the following environment information:

Additional context Tested this on a few systems, all experiencing the same behaviour. Going from 1000MHz -> 900MHz was able to stabilize.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 33 (30 by maintainers)

Most upvoted comments

Closing since this is now a stale issue. We have not seen any issues reported with reworked cards in the field. Firmware releases have been updated since.

@milank94 are you able to run any workloads on the systems in Serbia? There are 75+ reworked boards deployed and ready to go.

Definitely. If we have a LoudBox configuration (4x n300) with reworked cards I can jump on that. Who should I reach out to for access?

I’ll reach out to Zlatomir in Serbia and we’ll track down someone on the Software team who knows. Stand by.

Update: After looking at Milan’s server logs from the crash, I believe the hang occurs during decode. In an attempt to simplify the repro, I am currently running a traced, decode-only 24-hour stress test on the following systems:

  1. sjc-snva-t3010
  2. f10cs01
  3. f10cs02

All of these systems have been flashed to 80.13.1.0.