tt-metal: [Bug Report] Conv2d allocated buffers clashing with L1 buffers

Describe the bug The convolution is:

  • Input shape (1, 28, 28, 512) (NHWC)
  • Kernel shape (1024, 512, 1, 1)
  • Strides: 2x2
  • Padding: None
  • Dilation: 1
  • Bias: None

To Reproduce

Run the following ttnn pytest:

@pytest.mark.parametrize("device_params", [{"l1_small_size": 32768}], indirect=True)
def test_conv(device):
    batch_size = 1
    output_channels = 1024
    input_channels = 512
    input_height = 28
    input_width = 28
    filter_height = 1
    filter_width = 1
    stride_h = 2
    stride_w = 2
    groups = 1
    pad_h = 0
    pad_w = 0
    dilation = 1
    activations_dtype = ttnn.bfloat16
    weights_dtype = ttnn.bfloat16
    has_bias = False

    input_torch = torch.rand(batch_size, input_channels, input_height, input_width)
    weight_torch = torch.rand(output_channels, input_channels, filter_height, filter_width)

    ground_truth = torch.nn.functional.conv2d(
        input_torch,
        weight_torch,
        bias=None,
        stride=(stride_h, stride_w),
        padding=(pad_h, pad_w),
        dilation=(dilation, dilation),
        groups=groups,
    ).transpose(-3, -2).transpose(-2, -1) # For comparison with channel-last conv from ttnn

    # Format input as channel last for ttnn

    tt_input_tensor = ttnn.from_torch(
        input_torch.transpose(-3, -2).transpose(-2, -1), activations_dtype,
    )

    tt_weight_tensor = ttnn.from_torch(
        weight_torch, weights_dtype
    )

    [tt_output_tensor_on_device, out_height, out_width, weights_device, bias_device] = ttnn.conv2d(
        input_tensor=tt_input_tensor,
        weight_tensor=tt_weight_tensor,
        in_channels=input_channels,
        out_channels=output_channels,
        device=device,
        bias_tensor=None,
        kernel_size=(filter_height, filter_width),
        stride=(stride_h, stride_w),
        padding=(pad_h, pad_w),
        dilation=(dilation, dilation),
        batch_size=batch_size,
        input_height=input_height,
        input_width=input_width,
        # conv_config=conv_config,
        # conv_op_cache=reader_patterns_cache,
        # debug=debug,
        groups=groups,
    )

    tt_output_tensor = ttnn.from_device(tt_output_tensor_on_device)
    torch_output_tensor = ttnn.to_torch(tt_output_tensor).reshape(batch_size, out_height, out_width, output_channels)

    pcc = 0.99
    assert_with_pcc(torch_output_tensor, ground_truth, pcc=pcc)

Additional context Tensorflow resnet50 contains this conv. This is blocking the bringup through forge-fe --> MLIR --> ttnn runtime

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (14 by maintainers)

Most upvoted comments

Please remind me, @LPanosTT did you hit some bigger issues when pushing conv op to work on DRAM? That way we’ll for sure escape memory management issues and sharding.

So far… no.

@mywoodstock Using block sharding for this conv worked. Thanks!

@LPanosTT I think we want to dump the output memory config that the op picks to double check this is something sane. It might have picked something pretty non-optimal by default.