tt-metal: [6U Llama3-70B] Hangs on Stress test - g05glx6u02/g05glx6u04

Description

During stress testing on g05glx6u02 and g05glx6u04, the system hangs in the 3rd pytest run between iterations 1700–1800 and 5200+ respectively. Due to suppressed logging (extensive printing mitigation), progress is only shown every 100 iterations, making precise hang detection difficult.

Background

What is stress test:

A bash script (run_stress_test_in_a_loop_6u.sh) that runs decode-demo 5 times in a row, generating 20.000 tokens per run.

Why testing it on 6U

Group of 4U galaxies were hanging while another group was passing consistently without noticeable difference. Further Investigation paused on 4U; now verifying if the same issue occurs on 6U.

Current 4U CI

This test is not in CI due to resource constraints and also some 4U CI machines were observing hangs while running this test. A mini-stress-test (2K tokens, single run) was added recently and passes reliably.

Steps to reproduce

chmod +x run_stress_test_in_a_loop_6u.sh
./run_stress_test_in_a_loop_6u.sh g05glx6u02

Known Issue

In case a run hangs reset will probably not work so you won’t be able to run for full 5 iterations but only up to the one where it hangs. Once reset sequence works properly script will be able to recover from hang.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (5 by maintainers)

Most upvoted comments

@uaydonat 's team has tested

  • g05glx6u01
  • g05glx6u04
  • g11glx6u02
  • g11glx6u03

for Llama hang stress testing, and all passed.

g05glx6u02 has not been tested.