tt-metal: [Blackhole - Llama3-8B] demo hang on 1xP150
Main Llama3-on-BH issue: https://github.com/tenstorrent/tt-metal/issues/18135
13 May
-
Llama-8B - Batch-1 is much more stable since the mailbox/semaphore reordering fix on the LLK (https://github.com/tenstorrent/tt-llk/pull/185).
- Stress tested different seqlens (128, 256, 1k, 2k) for 20k iterations. All-pass ✅
- Seqlen sweep: 64K fails due to issues with tt-metal cahe:
/home/mtairum/.cache/tt-metal-cache/a56bd6e297/1056768/kernels/writer_fill_cache_interleaved/13402343708871966648/brisc/ - All other seqlens work without hangs
- Running seqlen 128 for 100k hanged on iteration 32669.
-
Batch-32 still hangs (watcher: watcher_12May_llama_batch32.log)
16 April
Update 16 April.
On latest main (commit 26655054ad5ce63d369950247e4d95878e9b16fe) which includes @bbradelTT fix, I see the full sweep passing consistently with chunk size = 128.
I tested sequence lengths from 128 to 64k. 👍
However, when trying back a chunk size of 256, like the one used in Wormhole, I see hangs start at 1K sequence length. Updating the grid size of the op to (7, 7) makes the hangs go away for chunks == 256.
This is the change, in file models/tt_transformers/tt/model_config.py:
self.model_config["SDPA_DECODE_PROGCFG"] = ttnn.SDPAProgramConfig(
compute_with_storage_grid_size=(7, 7),
exp_approx_mode=False,
q_chunk_size=256,
k_chunk_size=256,
)
9 April
Tested on Commit: 7c596285567db10854220befbc96a4183c47e954
With the following two changes I see the tests passing. ❗ The first test (small 128 prompt) we need to have the SDPA chunk change, otherwise it will hang. ❗
max_seq_len=131072
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_128.json” --max_seq_len $max_seq_len
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_256.json” --max_seq_len $max_seq_len
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_1k.json” --max_seq_len $max_seq_len
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_2k.json” --max_seq_len $max_seq_len
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_4k.json” --max_seq_len $max_seq_len
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_8k.json” --max_seq_len $max_seq_len
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_16k.json” --max_seq_len $max_seq_len
✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_32k.json” --max_seq_len $max_seq_len
Anil’s changes
--- a/tt_llk_blackhole/common/inc/cpack_common.h
+++ b/tt_llk_blackhole/common/inc/cpack_common.h
@@ -465,6 +465,7 @@ inline void select_packer_dest_registers()
inline void program_packer_destination(uint32_t addr)
{
uint32_t new_l1_addr = (1 << 31) | addr;
+ TTI_STALLWAIT(p_stall::STALL_TDMA, p_stall::PACK);
TT_SETDMAREG(0, LOWER_HALFWORD(addr), 0, LO_16(p_gpr_pack::OUTPUT_ADDR));
TT_SETDMAREG(0, UPPER_HALFWORD(new_l1_addr), 0, HI_16(p_gpr_pack::OUTPUT_ADDR));
@@ -472,6 +473,8 @@ inline void program_packer_destination(uint32_t addr)
TTI_WRCFG(p_gpr_pack::OUTPUT_ADDR, 0, THCON_SEC0_REG1_L1_Dest_addr_ADDR32);
TT_SETDMAREG(0, UPPER_HALFWORD(addr), 0, HI_16(p_gpr_pack::OUTPUT_ADDR));
+ TTI_DMANOP;
+ TTI_DMANOP;
}
// RT: If multiple contexts are used, for issue #https://github.com/tenstorrent/tt-llk-bh/issues/20
@@ -504,6 +507,8 @@ inline void program_packer_untilized_destination(const uint32_t addr, const uint
inline void program_packer_dest_offset_registers(uint32_t dest_tile_offset)
{
+ TTI_STALLWAIT(p_stall::STALL_TDMA, p_stall::PACK);
+ TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::PACK);
TT_SETDMAREG(0, LOWER_HALFWORD(dest_tile_offset), 0, LO_16(p_gpr_pack::TEMP_TILE_OFFSET));
TT_SETDMAREG(0, UPPER_HALFWORD(dest_tile_offset), 0, HI_16(p_gpr_pack::TEMP_TILE_OFFSET));
TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::THCON);
SDPA chunk change
This one is require for the small 128 seqlen test, otherwise it will hang!
--- a/models/tt_transformers/tt/model_config.py
+++ b/models/tt_transformers/tt/model_config.py
@@ -712,8 +712,8 @@ class ModelArgs:
self.model_config["SDPA_DECODE_PROGCFG"] = ttnn.SDPAProgramConfig(
compute_with_storage_grid_size=(8, 8),
exp_approx_mode=False,
- q_chunk_size=128 if self.arch_name == "blackhole" else 256,
- k_chunk_size=128 if self.arch_name == "blackhole" else 256,
+ q_chunk_size=256 if self.arch_name == "blackhole" else 256,
+ k_chunk_size=256 if self.arch_name == "blackhole" else 256,
)
28 March
The tests done on this date include the SDPA chunk change https://github.com/tenstorrent/tt-metal/pull/19794 on top of main commit: bd878ad7944c01e8acc54b38e02e36be29b3ecc9.
For all tests the AI clk was set at 800MHz.
The goal of these tests is to sweep the array of combinations we can run our demo on. The small demo we run is pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1"
Notes
-
We still have to export
TT_METAL_DISABLE_L1_DATA_CACHE_RISCVS=BR,NC,TR*,ER*otherwise we still see hangs, even in the smaller demo.- Here’s the watcher file for the small demo and running without this envvar: 28mar_llama8_demo.log
-
Increasing the max length supported leads to further hangs. This makes sense as this increases considerable the size of cache-related matmuls (like SDPA).
Status
- If tests are run with
max_seq_len > 32768the test will hang. - if
max_seq_len is range(4096, 32768)tests with input length >= 2K hang.
Full sweep reproduction code
#!/bin/bash
max_seq_len=1024
max_seq_len=2048
# max_seq_len=4096
# max_seq_len=8192
# max_seq_len=16384
# max_seq_len=32768
# max_seq_len=65536
# max_seq_len=131072
# [April] On latest versions we don't require this flag anymore
#flags="TT_METAL_DISABLE_L1_DATA_CACHE_RISCVS=BR,NC,TR*,ER*"
# Run a sweep of batch-1 LLama3 demo for different sequence lengths up to the maximum of $max_seq_len
echo "### Running seqlen of 128..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_128.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 256..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_256.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 1k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_1k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 2k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_2k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 4k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_4k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 8k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_8k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 16k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_16k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 32k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_32k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 64k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_64k.json" --max_seq_len $max_seq_len
# echo "### Running seqlen of 128k... [FAILING]"
# $flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_128k.json" --max_seq_len $max_seq_len
18 March
On latest main (Seeing this since 14th March on main) Llama3 is hanging on Blackhole single chip (P150). The hang is happening after prefill,after compiling the decode trace.
Notes on the issue:
- Only happens on test_model and demo. 1L is enough to replicate.
- decoder test and smaller ones are ok
- Cannot be replicated on x2 chips. It passes on 2xP150
- Watcher log for demo: watcher_log_llama3_bh_17mar.log
- Watcher log for 19 March: watcher-llama-demo-1xP150.log
- Seems to be around SDPA 🤔
- Watcher log for
test_model.py: watcher_log_llama3_model-test_bh_17mar.log
Watcher settings used:
export TT_METAL_WATCHER=30
export TT_METAL_WATCHER_DISABLE_ASSERT=1
export TT_METAL_WATCHER_DISABLE_PAUSE=1
export TT_METAL_WATCHER_DISABLE_RING_BUFFER=1
export TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1
export TT_METAL_WATCHER_DISABLE_DISPATCH=1
To Replicate
On a BH machine you should have the weights on device already:
export LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct
pytest models/tt_transformers/demo/simple_text_demo.py
In case you don’t have weights, you can run dummy weights with the test model:
pytest models/tt_transformers/tests/test_model.py -k "performance and quick"
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 167 (165 by maintainers)
Commits related to this issue
- #19201: add missing barrier and flushed in sdpa decode and adjust brisc stack size — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add missing barrier and flushed in sdpa decode and adjust brisc stack size — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add missing barrier and flushed in sdpa decode and adjust brisc stack size — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add missing barrier and flushed in sdpa decode and adjust brisc stack size — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add missing barrier and flushed in sdpa decode and adjust brisc stack size (#20663) ### Ticket Link to Github Issue #19201 ### Problem description - llama-3-8B demo hangs on BH ### What's ... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add missing barrier and flushed in sdpa decode and adjust brisc stack size (#20663) ### Ticket Link to Github Issue #19201 ### Problem description - llama-3-8B demo hangs on BH ### What's ... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: disable replay buffer for SFPI on BH — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: disable replay buffer for SFPI on BH (#20993) ### Ticket Link to Github Issue #19201 ### Problem description - re-enabling replay buffer for SFPI on BH broke llama3 on BH ### What's change... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add missing barrier and flushed in sdpa decode and adjust brisc stack size (#20663) ### Ticket Link to Github Issue #19201 ### Problem description - llama-3-8B demo hangs on BH ### What's ... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: disable replay buffer for SFPI on BH (#20993) ### Ticket Link to Github Issue #19201 ### Problem description - re-enabling replay buffer for SFPI on BH broke llama3 on BH ### What's change... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add extra syncs to sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add extra syncs to sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201 add in cb_wait_front for non-BH in sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add BH hang workaround comment for such lines in sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add extra syncs to sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201 add in cb_wait_front for non-BH in sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add BH hang workaround comment for such lines in sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: fix typo in sdpa decode in previous intermediate commit — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: revert of add of cb_wait_front for non-BH in sdpa decode — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #19201: add extra syncs to sdpa decode (#21240) ### Ticket Link to Github Issue #19201 ### Problem description The BH llama tests were still hanging in SDPA decode after several other changes and re... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
Steps to attach gdb.
For me this is changing the chunks to 256 (that’s where the hang currently is)
then running
tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-gdbbut something changed and it is no longer there for me.0x88b0 for trisc0
check generated/watcher/kernel_elf_paths.txt for kernel path and remove slash
Something like
Identify core and risc you want to attach to: e.g.
Run
attach 2Stuff will show up on tt-lens window. Should be okay to ignore.
Run the following based on the watcher info.
Output will be something like
kill %quitin gdbexitin tt-exalens1st night of stress testing status:
Example command:
All seqlens tested passing for 2h. (Batch-1).
This night I’ll repeat the test for Batch=32. FIY @ejouretTT
Latest status - 13 May
Llama-8B - Batch-1 seems to have been fixed by the mailbox/semaphore reordering on the LLK (https://github.com/tenstorrent/tt-llk/pull/185)
Stress tested different seqlens (128, 256, 1k, 2k) for 20k iterations. All-pass ✅
Seqlen sweep: 64K fails due to issues with tt-metal cahe:
/home/mtairum/.cache/tt-metal-cache/a56bd6e297/1056768/kernels/writer_fill_cache_interleaved/13402343708871966648/brisc/Running seqlen 128 for 100k iterations…
Batch-32 still hangs (Check watcher in previous comment above)
[Update]
watcher_12May_llama_batch32.log
Here’s the watcher for batch-32 run with @amahmudTT fixes, that hangs at iteration 897. I think it always hanged at this iteration but needs confirmation.
Run command:
pytest models/tt_transformers/demo/simple_text_demo.py -k performance-batch-32 --max_seq_len 131072 --max_generated_tokens 20000 --stop_at_eos 0@bbradelTT if you have the time, can you try on that branch to see if you can reproduce the hang?
@mtairum investigating a minimal size version of this.
@mtairum could you please double check what I found?
128 is fine w/ me provided CI passes
The tracing error was thrown on host when attempting to add op synchronization. The noc transaction being in flight (
NKFW) was seen when op synchronization was disabled (default behaviour)@mtairum would know if this is ND or not.
Now that I can finally repro the issue, I’m planning to look at
The error with trace is valid. Disabling fast runtime mode and enabling logging enables synchronization (which calls EnqueueRecordEvent) between ops but this breaks trace because we only support EnqueueProgram while tracing.
@bbradelTT if you want to disable fast runtime mode and enable logging I suggest disabling Tracing from the model test.
enable_trace = False # Use tracing for better perfinmodels/tt_transformers/demo/simple_text_demo.pyI started running on a P100 (not P100a) machine with the main branch since it now has the fix for matmul hangs which affect P100 (as of now). The only hang I got was during the capturing of trace. AFAIK capturing trace does not execute the kernels exactly rather makes a list of what is to be executed. Doesn’t that mean its more of a runtime issue ?
I added some prints and these were the output before the hang.
This is where I added the prints.
@amahmudTT
Also checked the configs of the sharded matmuls. The attention and MLP MMs on the decode side are all DRAM-sharded. On the prefill side they’re interleaved.
This signature looks like #18943 (trisc0/2 hung), fyi @jbaumanTT