tt-metal: [Bug Report] ttnn.linear performs out of bounds writes to dram
Describe the bug
With specific shapes, ttnn.linear clobbers dram values outside of the output tensor.
To Reproduce Minimal test case, reproduced on single device with 8x8 coregrid:
def test_padded_matmul(device):
'''
out_1BNX shape: ttnn.Shape([1, 1, 44520[44544], 3072])
self.proj_x shape: ttnn.Shape([3072, 384])
self.proj_x_bias shape: ttnn.Shape([1[32], 384])
out_joint_1BLX shape: ttnn.Shape([1, 1, 118[128], 384])
'''
S = 44520
D = 1536
torch.manual_seed(0)
pcc = 0.99
torch_act = torch.rand([1, 1, S, D], dtype=torch.float32)
torch_weight = torch.rand([1, 1, D, D], dtype=torch.float32)
torch_bias = torch.rand([1, 1, 1, D], dtype=torch.float32)
# Allocate tensors above and below where the output will be
dummy_lower = torch.full([1, 1, 2**15, 2**15], 2)
dummy_out = torch.zeros([1, 1, S, D])
dummy_upper = torch.full([1, 1, 2**15, 2**15], 4)
act = ttnn.from_torch(torch_act, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
weight = ttnn.from_torch(torch_weight, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
bias = ttnn.from_torch(torch_bias, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
lower_tt = ttnn.from_torch(dummy_lower, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
out_tt = ttnn.from_torch(dummy_out, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
upper_tt = ttnn.from_torch(dummy_upper, layout=ttnn.TILE_LAYOUT, device=device, dtype=ttnn.bfloat16)
# Free up dummy output tensor so linear will allocate output there
ttnn.deallocate(out_tt)
output1 = ttnn.linear(act, weight, bias=bias, core_grid=ttnn.CoreGrid(x=8, y=8), compute_kernel_config=ttnn.WormholeComputeKernelConfig(math_fidelity=ttnn.MathFidelity.HiFi2, math_approx_mode=False, fp32_dest_acc_en=True, packer_l1_acc=False))
lower = ttnn.to_torch(lower_tt).float()
upper = ttnn.to_torch(upper_tt).float()
# Check that the tensors above and below the output are unchanged
assert torch.all(lower == 2)
assert torch.all(upper == 4)
Expected behavior I expect the asserts to pass but they throw. Instead, I get the following error:
(Pdb) p upper
TorchTensor([[[[350., 370., 362., ..., 4., 4., 4.],
[350., 360., 352., ..., 4., 4., 4.],
[358., 364., 348., ..., 4., 4., 4.],
...,
[ 4., 4., 4., ..., 4., 4., 4.],
[ 4., 4., 4., ..., 4., 4., 4.],
[ 4., 4., 4., ..., 4., 4., 4.]]]])
(Pdb) n
> /home/cglagovich/tt-metal/tests/ttnn/unit_tests/operations/test_matmul.py(2193)test_padded_matmul()
-> assert torch.all(upper == 4)
(Pdb) n
AssertionError: assert TorchTensor(False)
+ where TorchTensor(False) = <built-in method all of type object at 0x7f9bb6cb68c0>(TorchTensor([[[[350., 370., 362., ..., 4., 4., 4.],\n [350., 360., 352., ..., 4., 4., 4.],\n [358., 364., 348., ..., 4., 4., 4.],\n ...,\n [ 4., 4., 4., ..., 4., 4., 4.],\n [ 4., 4., 4., ..., 4., 4., 4.],\n [ 4., 4., 4., ..., 4., 4., 4.]]]]) == 4)
+ where <built-in method all of type object at 0x7f9bb6cb68c0> = torch.all
> /home/cglagovich/tt-metal/tests/ttnn/unit_tests/operations/test_matmul.py(2193)test_padded_matmul()
-> assert torch.all(upper == 4)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- #17491: add out of bounds matmul unit tests — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #17491: add tile count to padded 2d matmul test — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #17491: add block sharded matmul test — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #17491: add side to padded 1d matmul test — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #17491: adjust padded matmul tests — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #0: fix uneven split on height/width of out tensor in Matmul (#18113) ### Ticket [Link to Github Issue](https://github.com/tenstorrent/tt-metal/issues/17491#event-16386901268) ### Problem descriptio... — committed to tenstorrent/tt-metal by yugaoTT 2 years ago
I looked through the code.
We only trim the last out_block_h/w. This behaviour seems common for 1D, 2D, and DRAM code.
Using ttnn/cpp/ttnn/operations/matmul/device/matmul_op_multi_core_reuse_mcast_2d_program_factory.cpp as an example.
What is happening is that we pass in the number of blocks for w and h via compile time args.
in0_num_blocks_y is 16 and in1_num_blocks_x is 1
We also adjust the last block’s sizes:
With the example’s numbers:
And then the kernels go through all 16 y blocks, with the last one processing only one tile.
That’s
15*11+1 = 166, which is the behaviour that I saw.Instead, what needs to happen is that the
lastbehaviour needs to extend to the num blocks.144 / 11 is 13 1/3, which means that 14 y blocks should be processed, 13 full blocks of 11 sub blocks, and the last out block having 1 (meaning that the last out subblock logic might already work if the num blocks are adjusted.
Started looking into this issue. Changed
2**15to2**8because otherwise test took too long to run with debug build, and the issue is still there.Also tried linear without bias and with matmul. Looks like everything is affected.