tt-metal: [6U Llama3-70B] Hangs on Stress test - g05glx6u02/g05glx6u04
Description
During stress testing on g05glx6u02 and g05glx6u04, the system hangs in the 3rd pytest run between iterations 1700–1800 and 5200+ respectively. Due to suppressed logging (extensive printing mitigation), progress is only shown every 100 iterations, making precise hang detection difficult.
Background
What is stress test:
A bash script (run_stress_test_in_a_loop_6u.sh) that runs decode-demo 5 times in a row, generating 20.000 tokens per run.
Why testing it on 6U
Group of 4U galaxies were hanging while another group was passing consistently without noticeable difference. Further Investigation paused on 4U; now verifying if the same issue occurs on 6U.
Current 4U CI
This test is not in CI due to resource constraints and also some 4U CI machines were observing hangs while running this test. A mini-stress-test (2K tokens, single run) was added recently and passes reliably.
Steps to reproduce
Branch: https://github.com/tenstorrent/tt-metal/tree/divanovic/6u_galaxy_stress_testing- Ensure
tt-metalis compiled in Release mode. - Run:
chmod +x run_stress_test_in_a_loop_6u.sh
./run_stress_test_in_a_loop_6u.sh g05glx6u02
Known Issue
In case a run hangs reset will probably not work so you won’t be able to run for full 5 iterations but only up to the one where it hangs. Once reset sequence works properly script will be able to recover from hang.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (5 by maintainers)
@uaydonat 's team has tested
g05glx6u01g05glx6u04g11glx6u02g11glx6u03for Llama hang stress testing, and all passed.
g05glx6u02has not been tested.