tt-metal: [Blackhole] Llama3 test decoder hang
models/demos/llama3/tests/test_llama_decoder.py is failing on latest main on blackhole.
I’ve tested this on bh-30.
Update
Run with watcher.
From the terminal I see the following:
Always | WARNING | Device 0 worker core(x= 8,y= 2) virtual(x=11,y= 4): ncrisc tripped an assert on line 62. Current kernel: ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/dataflow/reader_binary_interleaved_start_id.cpp. Note that file name reporting is not yet implemented, and the reported line number for the assert may be from a different file.
Always | INFO | Last waypoint: NTW,NKFW, W, W, W
Always | INFO | While running kernels:
Always | INFO | brisc : ttnn/cpp/ttnn/operations/eltwise/unary/device/kernels/dataflow/writer_unary_interleaved_start_id.cpp
Always | INFO | ncrisc: ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/dataflow/reader_binary_interleaved_start_id.cpp
Always | INFO | triscs: ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/compute/eltwise_binary_kernel.cpp
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /localdev/mtairum/tt-metal/tt_metal/impl/debug/watcher_device_reader.cpp:545: tt::exception
info:
Watcher detected tripped assert and stopped device.
The assert being tripped is line 62 in tt_metal/hw/firmware/src/ncrisck.cc:
ASSERT(ncrisc_noc_nonposted_atomics_flushed(NOC_INDEX));
The watcher log for this will be:
Original
The issue seems to be in the MLP layer In the w2 matmul. The decoder test will pass the first iteration, but hang after the MLP while trying to read its output tensor on the 2nd iteration.
models.demos.llama3.tests.test_llama_decoder:test_llama_decoder_inference:196 - PCC: 0.9999965799944094
models.demos.llama3.tests.test_llama_decoder:test_llama_decoder_inference:199 - Llama Decoder Block Passed!
models.demos.llama3.tests.test_llama_decoder:test_llama_decoder_inference:157 - [Decoder] Generating token 1
.... < HANG >
However, if I add a sync ttnn.synchronize_device(self.mesh_device.get_devices()[0]) the MLP layer, I don’t observe the hangs anymore. Specifically if I add it to right before the w2 ttnn.linear() op inmodels/demos/llama3/tt/llama_mlp.py the test passes.
Weirdly, this issue is not observed in the MLP tests pytest models/demos/llama3/tests/test_llama_mlp.py -k 32-1.
This is the MM in question:
ttnn.synchronize_device(self.mesh_device.get_devices()[0])
w2_out = ttnn.linear(
w2_in,
self.w2,
compute_kernel_config=self.args.compute_kernel_config_hifi2_fp16,
dtype=self.args.ccl_dtype if TG else ttnn.bfloat16,
program_config=pc_2,
memory_config=(
(ttnn.L1_WIDTH_SHARDED_MEMORY_CONFIG if mode == "decode" else ttnn.DRAM_MEMORY_CONFIG)
if TG
else w2_in.memory_config()
),
core_grid=None, # FIXME: validate on TG ttnn.CoreGrid(y=8, x=8) if not pc_2 else None,
)
-
w2_in.shape =
[1, 1, 32, 14336] -
w2_in.memory_config() =
MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::L1,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=7)]},shape={32, 224},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt)) -
self.w2.shape=
[14336, 4096] -
self.w2.memory_config() =
MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::DRAM,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=0)]},shape={14336, 512},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt)) -
program_config =
MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig(in0_block_w=7,per_core_M=1,per_core_N=2,fused_activation=std::nullopt)
Related issues
- Main issue tracking llama3 e2e: https://github.com/tenstorrent/tt-metal/issues/18135
- BH Mmul hang parent issue: https://github.com/tenstorrent/tt-metal/issues/18064
- Potentially related: https://github.com/tenstorrent/tt-metal/issues/18094
Reproduce
I’m using branch mtairum/llama3-blackhole which has some modifications to support running our llama3 tests on blackhole.
Steps:
git checkout mtairum/llama3-blackhole
# Test 1B weight - Faster to replicate
export LLAMA_DIR=Llama3.2-1B/
# Test 8B weights
# export LLAMA_DIR=Llama3.1-8B-Instruct
export TT_METAL_WATCHER=8
pytest models/demos/llama3/tests/test_llama_decoder.py -k paged
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (13 by maintainers)
Yeah, in that commit I just replaced
post_atomic_incrementswith true in brisc.cc, but it might be wrong of course