tt-metal: [Blackhole] Llama3 test decoder hang

models/demos/llama3/tests/test_llama_decoder.py is failing on latest main on blackhole. I’ve tested this on bh-30.

Update

Run with watcher.

From the terminal I see the following:

                 Always | WARNING  | Device 0 worker core(x= 8,y= 2) virtual(x=11,y= 4): ncrisc tripped an assert on line 62. Current kernel: ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/dataflow/reader_binary_interleaved_start_id.cpp. Note that file name reporting is not yet implemented, and the reported line number for the assert may be from a different file.
                 Always | INFO     | Last waypoint:  NTW,NKFW,   W,   W,   W 
                 Always | INFO     | While running kernels:
                 Always | INFO     |  brisc : ttnn/cpp/ttnn/operations/eltwise/unary/device/kernels/dataflow/writer_unary_interleaved_start_id.cpp
                 Always | INFO     |  ncrisc: ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/dataflow/reader_binary_interleaved_start_id.cpp
                 Always | INFO     |  triscs: ttnn/cpp/ttnn/operations/eltwise/binary/device/kernels/compute/eltwise_binary_kernel.cpp
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /localdev/mtairum/tt-metal/tt_metal/impl/debug/watcher_device_reader.cpp:545: tt::exception
info:
Watcher detected tripped assert and stopped device.

The assert being tripped is line 62 in tt_metal/hw/firmware/src/ncrisck.cc:

        ASSERT(ncrisc_noc_nonposted_atomics_flushed(NOC_INDEX));

The watcher log for this will be:

watcher.log

Original

The issue seems to be in the MLP layer In the w2 matmul. The decoder test will pass the first iteration, but hang after the MLP while trying to read its output tensor on the 2nd iteration.

models.demos.llama3.tests.test_llama_decoder:test_llama_decoder_inference:196 - PCC: 0.9999965799944094
models.demos.llama3.tests.test_llama_decoder:test_llama_decoder_inference:199 - Llama Decoder Block Passed!
models.demos.llama3.tests.test_llama_decoder:test_llama_decoder_inference:157 - [Decoder] Generating token 1
.... < HANG >

However, if I add a sync ttnn.synchronize_device(self.mesh_device.get_devices()[0]) the MLP layer, I don’t observe the hangs anymore. Specifically if I add it to right before the w2 ttnn.linear() op inmodels/demos/llama3/tt/llama_mlp.py the test passes.

Weirdly, this issue is not observed in the MLP tests pytest models/demos/llama3/tests/test_llama_mlp.py -k 32-1.

This is the MM in question:

ttnn.synchronize_device(self.mesh_device.get_devices()[0])
w2_out = ttnn.linear(
            w2_in,
            self.w2,
            compute_kernel_config=self.args.compute_kernel_config_hifi2_fp16,
            dtype=self.args.ccl_dtype if TG else ttnn.bfloat16,
            program_config=pc_2,
            memory_config=(
                (ttnn.L1_WIDTH_SHARDED_MEMORY_CONFIG if mode == "decode" else ttnn.DRAM_MEMORY_CONFIG)
                if TG
                else w2_in.memory_config()
            ),
            core_grid=None,  # FIXME: validate on TG ttnn.CoreGrid(y=8, x=8) if not pc_2 else None,
        )
  • w2_in.shape = [1, 1, 32, 14336]

  • w2_in.memory_config() = MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::L1,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=7)]},shape={32, 224},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt))

  • self.w2.shape= [14336, 4096]

  • self.w2.memory_config() = MemoryConfig(memory_layout=TensorMemoryLayout::WIDTH_SHARDED,buffer_type=BufferType::DRAM,shard_spec=ShardSpec(grid={[(x=0,y=0) - (x=7,y=0)]},shape={14336, 512},orientation=ShardOrientation::ROW_MAJOR,mode=ShardMode::PHYSICAL,physical_shard_shape=std::nullopt))

  • program_config = MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig(in0_block_w=7,per_core_M=1,per_core_N=2,fused_activation=std::nullopt)

Related issues

Reproduce

I’m using branch mtairum/llama3-blackhole which has some modifications to support running our llama3 tests on blackhole.

Steps:

git checkout mtairum/llama3-blackhole

# Test 1B weight - Faster to replicate
export LLAMA_DIR=Llama3.2-1B/

# Test 8B weights
# export LLAMA_DIR=Llama3.1-8B-Instruct

export TT_METAL_WATCHER=8

pytest models/demos/llama3/tests/test_llama_decoder.py -k paged

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (13 by maintainers)

Commits related to this issue

Most upvoted comments

Yeah, in that commit I just replaced post_atomic_increments with true in brisc.cc, but it might be wrong of course