tt-metal: [Blackhole - Llama3-8B] demo hang on 1xP150

Main Llama3-on-BH issue: https://github.com/tenstorrent/tt-metal/issues/18135

13 May

  • Llama-8B - Batch-1 is much more stable since the mailbox/semaphore reordering fix on the LLK (https://github.com/tenstorrent/tt-llk/pull/185).

    • Stress tested different seqlens (128, 256, 1k, 2k) for 20k iterations. All-pass ✅
    • Seqlen sweep: 64K fails due to issues with tt-metal cahe: /home/mtairum/.cache/tt-metal-cache/a56bd6e297/1056768/kernels/writer_fill_cache_interleaved/13402343708871966648/brisc/
    • All other seqlens work without hangs
    • Running seqlen 128 for 100k hanged on iteration 32669.
  • Batch-32 still hangs (watcher: watcher_12May_llama_batch32.log)

16 April

Update 16 April.

On latest main (commit 26655054ad5ce63d369950247e4d95878e9b16fe) which includes @bbradelTT fix, I see the full sweep passing consistently with chunk size = 128.

I tested sequence lengths from 128 to 64k. 👍

However, when trying back a chunk size of 256, like the one used in Wormhole, I see hangs start at 1K sequence length. Updating the grid size of the op to (7, 7) makes the hangs go away for chunks == 256.

This is the change, in file models/tt_transformers/tt/model_config.py:

self.model_config["SDPA_DECODE_PROGCFG"] = ttnn.SDPAProgramConfig(
                compute_with_storage_grid_size=(7, 7),
                exp_approx_mode=False,
                q_chunk_size=256,
                k_chunk_size=256,
            )

9 April

Tested on Commit: 7c596285567db10854220befbc96a4183c47e954

With the following two changes I see the tests passing. ❗ The first test (small 128 prompt) we need to have the SDPA chunk change, otherwise it will hang. ❗

max_seq_len=131072

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_128.json” --max_seq_len $max_seq_len

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_256.json” --max_seq_len $max_seq_len

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_1k.json” --max_seq_len $max_seq_len

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_2k.json” --max_seq_len $max_seq_len

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_4k.json” --max_seq_len $max_seq_len

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_8k.json” --max_seq_len $max_seq_len

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_16k.json” --max_seq_len $max_seq_len

✅ pytest models/tt_transformers/demo/simple_text_demo.py -k “performance-batch-1” --input_prompts “models/tt_transformers/demo/sample_prompts/input_data_long_32k.json” --max_seq_len $max_seq_len

Anil’s changes

--- a/tt_llk_blackhole/common/inc/cpack_common.h
+++ b/tt_llk_blackhole/common/inc/cpack_common.h
@@ -465,6 +465,7 @@ inline void select_packer_dest_registers()
 inline void program_packer_destination(uint32_t addr)
 {
     uint32_t new_l1_addr = (1 << 31) | addr;
+    TTI_STALLWAIT(p_stall::STALL_TDMA, p_stall::PACK);
     TT_SETDMAREG(0, LOWER_HALFWORD(addr), 0, LO_16(p_gpr_pack::OUTPUT_ADDR));
     TT_SETDMAREG(0, UPPER_HALFWORD(new_l1_addr), 0, HI_16(p_gpr_pack::OUTPUT_ADDR));
 
@@ -472,6 +473,8 @@ inline void program_packer_destination(uint32_t addr)
     TTI_WRCFG(p_gpr_pack::OUTPUT_ADDR, 0, THCON_SEC0_REG1_L1_Dest_addr_ADDR32);
 
     TT_SETDMAREG(0, UPPER_HALFWORD(addr), 0, HI_16(p_gpr_pack::OUTPUT_ADDR));
+    TTI_DMANOP;
+    TTI_DMANOP;
 }
 
 // RT: If multiple contexts are used, for issue #https://github.com/tenstorrent/tt-llk-bh/issues/20
@@ -504,6 +507,8 @@ inline void program_packer_untilized_destination(const uint32_t addr, const uint
 
 inline void program_packer_dest_offset_registers(uint32_t dest_tile_offset)
 {
+    TTI_STALLWAIT(p_stall::STALL_TDMA, p_stall::PACK);
+    TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::PACK);
     TT_SETDMAREG(0, LOWER_HALFWORD(dest_tile_offset), 0, LO_16(p_gpr_pack::TEMP_TILE_OFFSET));
     TT_SETDMAREG(0, UPPER_HALFWORD(dest_tile_offset), 0, HI_16(p_gpr_pack::TEMP_TILE_OFFSET));
     TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::THCON);

SDPA chunk change

This one is require for the small 128 seqlen test, otherwise it will hang!

--- a/models/tt_transformers/tt/model_config.py
+++ b/models/tt_transformers/tt/model_config.py
@@ -712,8 +712,8 @@ class ModelArgs:
             self.model_config["SDPA_DECODE_PROGCFG"] = ttnn.SDPAProgramConfig(
                 compute_with_storage_grid_size=(8, 8),
                 exp_approx_mode=False,
-                q_chunk_size=128 if self.arch_name == "blackhole" else 256,
-                k_chunk_size=128 if self.arch_name == "blackhole" else 256,
+                q_chunk_size=256 if self.arch_name == "blackhole" else 256,
+                k_chunk_size=256 if self.arch_name == "blackhole" else 256,
             )

28 March

The tests done on this date include the SDPA chunk change https://github.com/tenstorrent/tt-metal/pull/19794 on top of main commit: bd878ad7944c01e8acc54b38e02e36be29b3ecc9.

For all tests the AI clk was set at 800MHz.

The goal of these tests is to sweep the array of combinations we can run our demo on. The small demo we run is pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1"

Notes

  • We still have to export TT_METAL_DISABLE_L1_DATA_CACHE_RISCVS=BR,NC,TR*,ER*otherwise we still see hangs, even in the smaller demo.

  • Increasing the max length supported leads to further hangs. This makes sense as this increases considerable the size of cache-related matmuls (like SDPA).

Status

  • If tests are run with max_seq_len > 32768 the test will hang.
  • if max_seq_len is range(4096, 32768) tests with input length >= 2K hang.

Full sweep reproduction code

#!/bin/bash

max_seq_len=1024
max_seq_len=2048
# max_seq_len=4096
# max_seq_len=8192
# max_seq_len=16384
# max_seq_len=32768
# max_seq_len=65536
# max_seq_len=131072

# [April] On latest versions we don't require this flag anymore 
#flags="TT_METAL_DISABLE_L1_DATA_CACHE_RISCVS=BR,NC,TR*,ER*"

# Run a sweep of batch-1 LLama3 demo for different sequence lengths up to the maximum of $max_seq_len
echo "### Running seqlen of 128..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_128.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 256..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_256.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 1k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_1k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 2k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_2k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 4k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_4k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 8k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_8k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 16k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_16k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 32k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_32k.json" --max_seq_len $max_seq_len
echo "### Running seqlen of 64k..."
$flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_64k.json" --max_seq_len $max_seq_len
# echo "### Running seqlen of 128k... [FAILING]"
# $flags pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_128k.json" --max_seq_len $max_seq_len


18 March

On latest main (Seeing this since 14th March on main) Llama3 is hanging on Blackhole single chip (P150). The hang is happening after prefill,after compiling the decode trace.

Notes on the issue:

Watcher settings used:

export TT_METAL_WATCHER=30       

export TT_METAL_WATCHER_DISABLE_ASSERT=1
export TT_METAL_WATCHER_DISABLE_PAUSE=1
export TT_METAL_WATCHER_DISABLE_RING_BUFFER=1
export TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1
export TT_METAL_WATCHER_DISABLE_DISPATCH=1

To Replicate

On a BH machine you should have the weights on device already:

export LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct
pytest models/tt_transformers/demo/simple_text_demo.py

In case you don’t have weights, you can run dummy weights with the test model:

pytest models/tt_transformers/tests/test_model.py -k "performance and quick"

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 167 (165 by maintainers)

Commits related to this issue

Most upvoted comments

Steps to attach gdb.

  1. Set up - export env vars, get the right branch (update: code should now be in main), and compile:
export TT_METAL_WATCHER=12
export TT_METAL_WATCHER_DISABLE_PAUSE=1
export TT_METAL_WATCHER_DISABLE_RING_BUFFER=1
export TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1
export TT_METAL_WATCHER_DISABLE_WAYPOINT=1
export TT_METAL_WATCHER_DISABLE_STACK_USAGE=1
export TT_METAL_WATCHER_TEXT_START=1

export TT_METAL_RISCV_DEBUG_INFO=1
rm -rf /home/bbradel/.cache/tt-metal-cache/
git checkout jbauman/watcherpath2 # update: code in branch has been merged to main since the initial post, no need to use this branch
git submodule sync
git submodule update --init --recursive
./build_metal.sh --clean
ccache -C
rm -rf .cpmcache/
./build_metal.sh -p
  1. start test in first window

For me this is changing the chunks to 256 (that’s where the hang currently is)

diff --git a/models/tt_transformers/tt/model_config.py b/models/tt_transformers/tt/model_config.py
index 838a3dc32f..f977baf7b6 100644
--- a/models/tt_transformers/tt/model_config.py
+++ b/models/tt_transformers/tt/model_config.py
@@ -765,8 +765,8 @@ class ModelArgs:
             self.model_config["SDPA_DECODE_PROGCFG"] = ttnn.SDPAProgramConfig(
                 compute_with_storage_grid_size=(8, 8),
                 exp_approx_mode=False,
-                q_chunk_size=128 if self.arch_name == "blackhole" else 256,
-                k_chunk_size=128 if self.arch_name == "blackhole" else 256,
+                q_chunk_size=256 if self.arch_name == "blackhole" else 256,
+                k_chunk_size=256 if self.arch_name == "blackhole" else 256,
             )

then running

export max_seq_len=131072
                    
echo "INPUT 128"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_128.json" --max_seq_len $max_seq_len
echo "INPUT 256"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_questions_prefill_256.json" --max_seq_len $max_seq_len
echo "INPUT 1k"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_1k.json" --max_seq_len $max_seq_len
echo "INPUT 2k"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_2k.json" --max_seq_len $max_seq_len
echo "INPUT 4k"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_4k.json" --max_seq_len $max_seq_len
echo "INPUT 8k"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_8k.json" --max_seq_len $max_seq_len
echo "INPUT 16k"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_16k.json" --max_seq_len $max_seq_len
echo "INPUT 32k"
LLAMA_DIR=/proj_sw/user_dev/hf_data/Llama3/Llama3.1-8B-Instruct pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-batch-1" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_32k.json" --max_seq_len $max_seq_len
  1. start tt-exalens in second window
pip install git+https://github.com/tenstorrent/tt-exalens.git
tt-exalens
# the following from within exalens
gdb start --port 6767
  1. run gdb in third window Note: you need to obtain this version of gdb first. Should be in tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-gdb but something changed and it is no longer there for me.
sfpi/compiler/bin/riscv32-unknown-elf-gdb
  1. wait for hang and then get watcher info in fourth window check generated/watcher/watcher.log to see where is hang and text offsets. probably BNT on core with sdpa kernels running
Device 0 worker core(x= 0,y= 0) virtual(x= 1,y= 2): rmsg:D1G|BNT h_id:2777088 smsg:DGGG k_ids:161|160|159 text_start:0x6ee0|0x7f60|0x88b0|0xc160|0x10170

0x88b0 for trisc0

check generated/watcher/kernel_elf_paths.txt for kernel path and remove slash

Something like

/home/bbradel/.cache/tt-metal-cache/b7e3706454/1056768/kernels/sdpa_flash_decode/7779109927252611913/trisc0/trisc0.elf
  1. in gdb window
target extended-remote localhost:6767
info os processes

Identify core and risc you want to attach to: e.g.

2          limited    2          0          worker     1-2 (0,0)  TRISC0     None       1

Run attach 2

Stuff will show up on tt-lens window. Should be okay to ignore.

Run the following based on the watcher info.

add-symbol-file /home/bbradel/.cache/tt-metal-cache/b7e3706454/1056768/kernels/sdpa_flash_decode/7779109927252611913/trisc0/trisc0.elf 0x88b0
# press y for yes
bt

Output will be something like

Reading symbols from /home/bbradel/.cache/tt-metal-cache/b7e3706454/1056768/kernels/sdpa_flash_decode/7779109927252611913/trisc0/trisc0.elf...
(gdb) bt
#0  ckernel::semaphore_read (index=3 '\003')
    at /localdev/bbradel/tt-metal/tt_metal/third_party/tt_llk/tt_llk_blackhole/common/inc/ckernel.h:149
#1  _llk_unpack_release_tile_<true, true> ()
    at /localdev/bbradel/tt-metal/tt_metal/third_party/tt_llk/tt_llk_blackhole/llk_lib/llk_unpack_common.h:59
#2  llk_unpack_release_tile<true, true> (operand=<optimized out>)
    at /localdev/bbradel/tt-metal/tt_metal/hw/ckernels/blackhole/metal/llk_api/llk_unpack_common_api.h:47
#3  ckernel::cb_release_tile (cb_id=8) at /localdev/bbradel/tt-metal/tt_metal/include/compute_kernel_api/cb_api.h:162
#4  chlkc_unpack::unpack_main ()
    at /localdev/bbradel/tt-metal/ttnn/cpp/ttnn/operations/transformer/sdpa_decode/device/kernels/compute/sdpa_flash_decode.cpp:111
#5  0x00005d58 in ?? ()
  1. shut everything down
  • ctrl-z on pytest and kill %
  • quit in gdb
  • exit in tt-exalens

1st night of stress testing status:

Example command:

timeout --signal=SIGINT 2h pytest models/tt_transformers/demo/simple_text_demo.py -k "performance-long-context-64k" --input_prompts "models/tt_transformers/demo/sample_prompts/input_data_long_1k.json" --max_seq_len $max_seq_len --stop_at_eos 0

All seqlens tested passing for 2h. (Batch-1).

  • ✅ 128 seqlen
  • ✅ 256 seqlen
  • ✅ 1K seqlen
  • ✅ 2K seqlen
  • ✅ 4K seqlen
  • ✅ 8K seqlen
  • ✅ 16K seqlen
  • ✅ 32K seqlen

This night I’ll repeat the test for Batch=32. FIY @ejouretTT

Latest status - 13 May

Llama-8B - Batch-1 seems to have been fixed by the mailbox/semaphore reordering on the LLK (https://github.com/tenstorrent/tt-llk/pull/185)

  • Stress tested different seqlens (128, 256, 1k, 2k) for 20k iterations. All-pass ✅

  • Seqlen sweep: 64K fails due to issues with tt-metal cahe: /home/mtairum/.cache/tt-metal-cache/a56bd6e297/1056768/kernels/writer_fill_cache_interleaved/13402343708871966648/brisc/

    • All other seqlens work without hangs
  • Running seqlen 128 for 100k iterations…

  • Batch-32 still hangs (Check watcher in previous comment above)

[Update]

  • Stress testing 128 seqlen still hangs at iteration 32669.

watcher_12May_llama_batch32.log

Here’s the watcher for batch-32 run with @amahmudTT fixes, that hangs at iteration 897. I think it always hanged at this iteration but needs confirmation.

Run command: pytest models/tt_transformers/demo/simple_text_demo.py -k performance-batch-32 --max_seq_len 131072 --max_generated_tokens 20000 --stop_at_eos 0

@bbradelTT if you have the time, can you try on that branch to see if you can reproduce the hang?

@mtairum investigating a minimal size version of this.

@mtairum could you please double check what I found?

128 is fine w/ me provided CI passes

@abhullar-tt why is this a device assert? this is an error in using the API, right? can you file an issue against the runtime to error out on the host when violating the constraints of the API?

The tracing error was thrown on host when attempting to add op synchronization. The noc transaction being in flight (NKFW) was seen when op synchronization was disabled (default behaviour)

@mtairum would know if this is ND or not.

Now that I can finally repro the issue, I’m planning to look at

  • adding synchronization and dprints to see where the hang is at a finer granularity and possibly
  • see if a smaller test case can be created with tracing

The error with trace is valid. Disabling fast runtime mode and enabling logging enables synchronization (which calls EnqueueRecordEvent) between ops but this breaks trace because we only support EnqueueProgram while tracing.

@bbradelTT if you want to disable fast runtime mode and enable logging I suggest disabling Tracing from the model test.

enable_trace = False # Use tracing for better perf in models/tt_transformers/demo/simple_text_demo.py

I started running on a P100 (not P100a) machine with the main branch since it now has the fix for matmul hangs which affect P100 (as of now). The only hang I got was during the capturing of trace. AFAIK capturing trace does not execute the kernels exactly rather makes a list of what is to be executed. Doesn’t that mean its more of a runtime issue ?

I added some prints and these were the output before the hang.

2025-04-10 06:11:12.850 | INFO     | models.tt_transformers.tt.generator:_capture_trace_text:282 - Done Compiling Model
2025-04-10 06:11:12.850 | INFO     | models.tt_transformers.tt.generator:_capture_trace_text:296 - Begin Capturing Trace
2025-04-10 06:11:12.850 | INFO     | models.tt_transformers.tt.generator:_capture_trace_text:300 - Begin Transform Trace Capture
2025-04-10 06:11:12.851 | INFO     | models.tt_transformers.tt.generator:_capture_trace_text:303 - Begin Decode Trace Capture

This is where I added the prints.

    def _capture_trace_text(
        self,
        tokens,
        current_pos,
        page_table=None,
        kv_cache=None,
        argmax_on_device=False,
    ):

       ------- REMOVED FOR BREVITY -------


        logger.info("Begin Capturing Trace")
        trace_id = ttnn.begin_trace_capture(self.mesh_device, cq_id=0)
        for i in range(self.data_parallel):
            user_kv_cache = kv_cache[i] if kv_cache is not None else None
            logger.info("Begin Transform Trace Capture")
            transformed_inputs = self.model[i].transform_decode_inputs_device(*(device_inputs[i]))

            logger.info("Begin Decode Trace Capture")
            tt_out_trace.append(
                self.model[i].ttnn_decode_forward(
                    *transformed_inputs, kv_cache=user_kv_cache, argmax_on_device=argmax_on_device
                )
            )

        ttnn.end_trace_capture(self.mesh_device, trace_id, cq_id=0)

        logger.info("Done Capturing Decode Trace")
        return trace_id, tt_out_trace, *device_inputs

@amahmudTT

Also checked the configs of the sharded matmuls. The attention and MLP MMs on the decode side are all DRAM-sharded. On the prefill side they’re interleaved.

Decode

- xqkv_fused_sharded
> in_0: Shape([1, 1, 32, 4096])
  > mem_config: MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::L1,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=3)]},shape={32, 128},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt)) 
> in_1: Shape([1, 1, 4096, 6144])
  > mem_config: MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::DRAM,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=0)]},shape={4096, 768},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt))
> MC: L1 width sharded
> PC: MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig(in0_block_w=4,per_core_M=1,per_core_N=6,fused_activation=std::nullopt)


- w1_out / w3_out
> in_0: Shape([1, 1, 32, 4096])
  > MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::L1,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=7)]},shape={32, 64},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt))
> in_1: Shape([4096, 14336])
  > MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::DRAM,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=0)]},shape={4096, 1792},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt))
> MC: same as in_0
> PC: MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig(in0_block_w=2,per_core_M=1,per_core_N=7,fused_activation=std::nullopt)
(Pdb) 

- w2_out
> in_0: Shape([1, 1, 32, 14336])
  > MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::L1,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=7)]},shape={32, 224},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt))
> in_1: Shape([14336, 4096])
  > MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::DRAM,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=0)]},shape={14336, 512},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt))
> MC: L1_WIDTH_SHARDED_MEMORY_CONFIG
> PC: MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig(in0_block_w=7,per_core_M=1,per_core_N=2,fused_activation=std::nullopt)
(Pdb) 

This signature looks like #18943 (trisc0/2 hung), fyi @jbaumanTT