tt-metal: [Bug Report] conv2d failed with `bias_ntiles == weight_matrix_width_ntiles`
Describe the bug
conv2d failed with bias_ntiles == weight_matrix_width_ntiles info for certain shapes
To Reproduce
Run the example below
import torch
import ttnn
from tests.ttnn.utils_for_testing import assert_with_pcc
def main(device):
input_shape, weight_shape, stride, padding, dilation, transposed, groups = (
[1, 1024, 45, 80],
[2048, 1024, 1, 1],
[2, 2],
[0, 0],
[1, 1],
False,
1,
)
n, in_c, in_h, in_w = input_shape
out_c, _, k_h, k_w = weight_shape
n, in_c, in_h, in_w = input_shape
out_c0, out_c1, k_h, k_w = weight_shape
out_c = out_c1 if transposed else out_c0
torch_input = torch.rand(input_shape, dtype=torch.bfloat16)
torch_weight = torch.rand(weight_shape, dtype=torch.bfloat16)
torch_bias = torch.rand((out_c,), dtype=torch.bfloat16)
input_tensor = ttnn.from_torch(torch_input.permute((0, 2, 3, 1)), layout=ttnn.TILE_LAYOUT, device=device)
weight_tensor = ttnn.from_torch(torch_weight, layout=ttnn.ROW_MAJOR_LAYOUT)
bias_tensor = ttnn.from_torch(torch_bias.reshape(1, 1, 1, -1), layout=ttnn.ROW_MAJOR_LAYOUT)
expected_output = torch.convolution(
torch_input, torch_weight, torch_bias, stride, padding, dilation, transposed, (0, 0), groups
)
expected_output = expected_output.permute((0, 2, 3, 1)).reshape(1, 1, -1, out_c)
output_tensor = ttnn.conv2d(
input_tensor=input_tensor,
weight_tensor=weight_tensor,
bias_tensor=bias_tensor,
batch_size=n,
in_channels=in_c,
out_channels=out_c,
input_height=in_h,
input_width=in_w,
kernel_size=(k_h, k_w),
stride=stride,
padding=padding,
dilation=dilation,
groups=groups,
device=device,
)
assert_with_pcc(expected_output, ttnn.to_torch(output_tensor), 0.99)
if __name__ == "__main__":
try:
device = ttnn.open_device(device_id=0, l1_small_size=16384)
main(device)
finally:
ttnn.close_device(device)
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software a8a812beba9230df1196fe9ac11bb7769f5f1d90
- Card: N300
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (15 by maintainers)
@nvukobratTT @mstojkovicTT Ok in short if you pass in weights tensor on device conv assumes that it’s prepared, so this is not the correct usage. Try some of the options above, from my point of view we should close this one.
@sankarmanoj-tt so basically you are saying that they are calling conv with unprepared weights, but with weights tensor on device right? So they have options:
Is my assessment correct?
The conv op applies some preprocessing on the weights tensor before using it. It determines if the weights have been preprocessed by looking at the storage type. By manually placing the weights/bias on device, the op assumes the weights have already been preprocessed.
This code can be fixed by keeping the weights / bias tensors on host.
@nvukobratTT I’m counting on you guys to provide the repro
@sankarmanoj-tt pls try to check what is going on here
@nvukobratTT pls add repros.