tt-metal: [Bug Report] ttnn.linear performs out of bounds writes to dram

Describe the bug With specific shapes, ttnn.linear clobbers dram values outside of the output tensor.

To Reproduce Minimal test case, reproduced on single device with 8x8 coregrid:

def test_padded_matmul(device):
    '''
    out_1BNX shape: ttnn.Shape([1, 1, 44520[44544], 3072])
    self.proj_x shape: ttnn.Shape([3072, 384])
    self.proj_x_bias shape: ttnn.Shape([1[32], 384])
    out_joint_1BLX shape: ttnn.Shape([1, 1, 118[128], 384])
    '''
    S = 44520
    D = 1536
    torch.manual_seed(0)
    pcc = 0.99
    torch_act = torch.rand([1, 1, S, D], dtype=torch.float32)
    torch_weight = torch.rand([1, 1, D, D], dtype=torch.float32)
    torch_bias = torch.rand([1, 1, 1, D], dtype=torch.float32)
    # Allocate tensors above and below where the output will be
    dummy_lower = torch.full([1, 1, 2**15, 2**15], 2)
    dummy_out = torch.zeros([1, 1, S, D])
    dummy_upper = torch.full([1, 1, 2**15, 2**15], 4)

    act = ttnn.from_torch(torch_act, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
    weight = ttnn.from_torch(torch_weight, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
    bias = ttnn.from_torch(torch_bias, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
    lower_tt = ttnn.from_torch(dummy_lower, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
    out_tt = ttnn.from_torch(dummy_out, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
    upper_tt = ttnn.from_torch(dummy_upper, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
    # Free up dummy output tensor so linear will allocate output there
    ttnn.deallocate(out_tt)
    output1 = ttnn.linear(act, weight, bias=bias, core_grid=ttnn.CoreGrid(x=8, y=8), compute_kernel_config=ttnn.WormholeComputeKernelConfig(math_fidelity=ttnn.MathFidelity.HiFi2, math_approx_mode=False, fp32_dest_acc_en=True, packer_l1_acc=False))
    lower = ttnn.to_torch(lower_tt).float()
    upper = ttnn.to_torch(upper_tt).float()
    # Check that the tensors above and below the output are unchanged
    assert torch.all(lower == 2)
    assert torch.all(upper == 4)

Expected behavior I expect the asserts to pass but they throw. Instead, I get the following error:

(Pdb) p upper
TorchTensor([[[[350., 370., 362.,  ...,   4.,   4.,   4.],
               [350., 360., 352.,  ...,   4.,   4.,   4.],
               [358., 364., 348.,  ...,   4.,   4.,   4.],
               ...,
               [  4.,   4.,   4.,  ...,   4.,   4.,   4.],
               [  4.,   4.,   4.,  ...,   4.,   4.,   4.],
               [  4.,   4.,   4.,  ...,   4.,   4.,   4.]]]])
(Pdb) n
> /home/cglagovich/tt-metal/tests/ttnn/unit_tests/operations/test_matmul.py(2193)test_padded_matmul()
-> assert torch.all(upper == 4)
(Pdb) n
AssertionError: assert TorchTensor(False)
 +  where TorchTensor(False) = <built-in method all of type object at 0x7f9bb6cb68c0>(TorchTensor([[[[350., 370., 362.,  ...,   4.,   4.,   4.],\n               [350., 360., 352.,  ...,   4.,   4.,   4.],\n               [358., 364., 348.,  ...,   4.,   4.,   4.],\n               ...,\n               [  4.,   4.,   4.,  ...,   4.,   4.,   4.],\n               [  4.,   4.,   4.,  ...,   4.,   4.,   4.],\n               [  4.,   4.,   4.,  ...,   4.,   4.,   4.]]]]) == 4)
 +    where <built-in method all of type object at 0x7f9bb6cb68c0> = torch.all
> /home/cglagovich/tt-metal/tests/ttnn/unit_tests/operations/test_matmul.py(2193)test_padded_matmul()
-> assert torch.all(upper == 4)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

I looked through the code.

We only trim the last out_block_h/w. This behaviour seems common for 1D, 2D, and DRAM code.

Using ttnn/cpp/ttnn/operations/matmul/device/matmul_op_multi_core_reuse_mcast_2d_program_factory.cpp as an example.

What is happening is that we pass in the number of blocks for w and h via compile time args.

    constexpr uint32_t in1_block_num_tiles = get_compile_time_arg_val(1);
    // in0/in1 common args
    constexpr uint32_t num_blocks_inner_dim = get_compile_time_arg_val(2);
    constexpr uint32_t num_blocks_w_dim = get_compile_time_arg_val(3);
    constexpr uint32_t num_blocks_h_dim = get_compile_time_arg_val(4);

    uint32_t in0_num_blocks_y = per_core_M / out_block_h;
    uint32_t in1_num_blocks_x = per_core_N / out_block_w;
    uint32_t out_num_blocks_x = in1_num_blocks_x;
    uint32_t out_num_blocks_y = in0_num_blocks_y;
    ...

        (std::uint32_t)in1_block_w * in0_block_w,  // in1_block_num_tiles
        // in0/in1 common args
        (std::uint32_t)num_blocks,  // num_blocks
        (std::uint32_t)out_num_blocks_x,
        (std::uint32_t)out_num_blocks_y,

                .compile_args = in1_receiver_writer_compile_time_args,

in0_num_blocks_y is 16 and in1_num_blocks_x is 1

We also adjust the last block’s sizes:

    // Parameters for last row, col, or block
    uint32_t last_per_core_M = M % per_core_M == 0 ? per_core_M : M % per_core_M; // 144
    uint32_t last_per_core_N = N % per_core_N == 0 ? per_core_N : N % per_core_N;
    uint32_t last_out_block_h = last_per_core_M % out_block_h == 0 ? out_block_h : last_per_core_M % out_block_h;
    uint32_t last_out_block_w = last_per_core_N % out_block_w == 0 ? out_block_w : last_per_core_N % out_block_w;
    uint32_t last_block_num_nonzero_subblocks_h = (last_out_block_h - 1) / out_subblock_h + 1;
    uint32_t last_block_num_nonzero_subblocks_w = (last_out_block_w - 1) / out_subblock_w + 1;

With the example’s numbers:

M = 43*32 = 1376 (Mt)
per_core_M = 176
M % per_core_M != 0
last_per_core_M = 144
out_block_h = 11
last_out_block_h = 1 (144-13*11)

And then the kernels go through all 16 y blocks, with the last one processing only one tile.

That’s 15*11+1 = 166, which is the behaviour that I saw.

Instead, what needs to happen is that the last behaviour needs to extend to the num blocks.

144 / 11 is 13 1/3, which means that 14 y blocks should be processed, 13 full blocks of 11 sub blocks, and the last out block having 1 (meaning that the last out subblock logic might already work if the num blocks are adjusted.

Started looking into this issue. Changed 2**15 to 2**8 because otherwise test took too long to run with debug build, and the issue is still there.

Also tried linear without bias and with matmul. Looks like everything is affected.