tt-metal: [Bug Report] Conv2d allocated buffers clashing with L1 buffers
Describe the bug The convolution is:
- Input shape (1, 28, 28, 512) (NHWC)
- Kernel shape (1024, 512, 1, 1)
- Strides: 2x2
- Padding: None
- Dilation: 1
- Bias: None
To Reproduce
Run the following ttnn pytest:
@pytest.mark.parametrize("device_params", [{"l1_small_size": 32768}], indirect=True)
def test_conv(device):
batch_size = 1
output_channels = 1024
input_channels = 512
input_height = 28
input_width = 28
filter_height = 1
filter_width = 1
stride_h = 2
stride_w = 2
groups = 1
pad_h = 0
pad_w = 0
dilation = 1
activations_dtype = ttnn.bfloat16
weights_dtype = ttnn.bfloat16
has_bias = False
input_torch = torch.rand(batch_size, input_channels, input_height, input_width)
weight_torch = torch.rand(output_channels, input_channels, filter_height, filter_width)
ground_truth = torch.nn.functional.conv2d(
input_torch,
weight_torch,
bias=None,
stride=(stride_h, stride_w),
padding=(pad_h, pad_w),
dilation=(dilation, dilation),
groups=groups,
).transpose(-3, -2).transpose(-2, -1) # For comparison with channel-last conv from ttnn
# Format input as channel last for ttnn
tt_input_tensor = ttnn.from_torch(
input_torch.transpose(-3, -2).transpose(-2, -1), activations_dtype,
)
tt_weight_tensor = ttnn.from_torch(
weight_torch, weights_dtype
)
[tt_output_tensor_on_device, out_height, out_width, weights_device, bias_device] = ttnn.conv2d(
input_tensor=tt_input_tensor,
weight_tensor=tt_weight_tensor,
in_channels=input_channels,
out_channels=output_channels,
device=device,
bias_tensor=None,
kernel_size=(filter_height, filter_width),
stride=(stride_h, stride_w),
padding=(pad_h, pad_w),
dilation=(dilation, dilation),
batch_size=batch_size,
input_height=input_height,
input_width=input_width,
# conv_config=conv_config,
# conv_op_cache=reader_patterns_cache,
# debug=debug,
groups=groups,
)
tt_output_tensor = ttnn.from_device(tt_output_tensor_on_device)
torch_output_tensor = ttnn.to_torch(tt_output_tensor).reshape(batch_size, out_height, out_width, output_channels)
pcc = 0.99
assert_with_pcc(torch_output_tensor, ground_truth, pcc=pcc)
Additional context Tensorflow resnet50 contains this conv. This is blocking the bringup through forge-fe --> MLIR --> ttnn runtime
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (14 by maintainers)
So far… no.
@mywoodstock Using block sharding for this conv worked. Thanks!
@LPanosTT I think we want to dump the output memory config that the op picks to double check this is something sane. It might have picked something pretty non-optimal by default.