tt-metal: [llama] llama 3 70B hangs after multiple batches of full context decode generation
Describe the bug
Consistent hang running llama3 70B demo after several batches of 32x prompts are completed, i.e. 2048 tokens decoded for 32x users multiple times. The hang sometimes occurs on the 2nd batch of 32 prompts, sometimes after 10 or more. The model forward call hangs and does not terminate.
To Reproduce Steps to reproduce the behavior: 1 build tt-metal on commit f0534b4 (https://github.com/tenstorrent/tt-metal/tree/f0534b4467eb7ae994dfa8dbecbbabb2f6893806) 2. run llama 3 demo with looping (attached patch file: llama3_70b_demo_looping.patch )
Expected behavior The demo should complete all loops of decode token generation.
logs
pytest -svv models/demos/t3000/llama3_70b/demo/demo.py::test_LlamaModel_demo[wormhole_b0-True-short_context-check_disabled-sampling-tt-70b-T3000-80L-decode_only-text_completion-llama3]
...
2024-07-26 09:28:10.337 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 2045
2024-07-26 09:28:10.451 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 2046
2024-07-26 09:28:10.564 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 2047
2024-07-26 09:28:10.566 | INFO | models.demos.t3000.llama2_70b.demo.demo:latency_printout:303 - Overall throughput: 3.4 ms @ 292.8 tokens/s
2024-07-26 09:28:10.566 | INFO | models.demos.t3000.llama2_70b.demo.demo:latency_printout:304 - Tokens per second per user: 9.2 tokens/s/u
2024-07-26 09:28:10.566 | INFO | models.demos.t3000.llama2_70b.demo.demo:latency_printout:305 - User latency: 109.3 ms @ 9.2 tokens/s
2024-07-26 09:28:10.574 | INFO | models.demos.t3000.llama2_70b.demo.demo:main:108 - batch_idx: 1
2024-07-26 09:28:10.574 | WARNING | models.demos.t3000.llama2_70b.demo.demo:run_decode:237 - Requested more output tokens than allowed by model. Truncating to 2000 output tokens.
2024-07-26 09:28:10.674 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 1
2024-07-26 09:28:10.773 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 2
2024-07-26 09:28:10.872 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 3
...
2024-07-26 09:29:05.200 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 523
2024-07-26 09:29:05.309 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 524
2024-07-26 09:29:05.417 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 525
2024-07-26 09:29:05.526 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 526
2024-07-26 09:29:05.634 | INFO | models.demos.t3000.llama2_70b.demo.demo:run_decode:261 - Loop 527
-> hung for over 10 minutes before I killed process
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software: f0534b4467eb7ae994dfa8dbecbbabb2f6893806
- firmware bundle: 80.10.0.0
- tt-kmd: 1.28
Additional context Possible issues with Ethernet on latest firmware.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (17 by maintainers)
Testing using the experimental firmware bundle 80.10.1.0 (https://github.com/tenstorrent/tt-firmware/blob/6154c3f999b22072d6f6ac193559e73f1e490116/experiments/fw_pack-80.10.1.0.fwbundle) with decrease max AICLK to 900MHz appears to resolve the issue.
I tested running 15x25 + 23 = 398 batches (12,736 sample prompt completions, ~26M decode loops):
An apparent performance degration of ~5% is observed (9.5 t/s/u to 9.0 t/s/u). This is anticipated given the reduced frequency.
If you haven’t, you may need a cold boot to bring all 4 devices back
I am very happy to report that after rebuilding on https://github.com/tenstorrent/tt-metal/commit/1b62f170c35e29a66c37c1a824e7b7682779b81f I no longer get the hang after running 10x batches and counting.