tt-metal: [Bug Report] ttnn.concat - Tile padding along concatenated dim is not supported

Describe the bug The ttnn.concat op throws Tile padding along concatenated dim (3) not supported for concat yet (tensor: 0) error when the tuple of two input tensor of same shape (1, 32, 12, 50) are passed with concat dim = -1. The above error is thrown while validating the input tensor of the concat op in the ttnn::operations::data_movement::ConcatDeviceOperation::validate function.

For more context, here is the exact error message:

E       RuntimeError: TT_FATAL @ ../ttnn/cpp/ttnn/operations/data_movement/concat/device/concat_device_operation.cpp:43: !in_ref.get_shape().has_tile_padding(this->dim)
E       info:
E       Tile padding along concatenated dim (3) not supported for concat yet (tensor: 0).

To Reproduce Run the following test:

import torch
import ttnn
from tests.ttnn.utils_for_testing import assert_with_pcc

def test_concat_rotary_embedding(device):
    concat_dim = 3
    torch_input_tensor_a = torch.rand((1, 32, 12, 50), dtype=torch.float32)
    torch_input_tensor_b = torch.rand((1, 32, 12, 50), dtype=torch.float32)
    torch_output_tensor = torch.concat([torch_input_tensor_a, torch_input_tensor_b], dim=concat_dim)

    input_tensor_a = ttnn.from_torch(torch_input_tensor_a, layout=ttnn.TILE_LAYOUT, device=device)
    input_tensor_b = ttnn.from_torch(torch_input_tensor_b, layout=ttnn.TILE_LAYOUT, device=device)

    output = ttnn.concat([input_tensor_a, input_tensor_b], dim=concat_dim)
    output = ttnn.to_torch(output)

    assert_with_pcc(torch_output_tensor, output, 0.9999)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

Should be resolved by https://github.com/tenstorrent/tt-metal/pull/15513 once merged. I’ve added the requested shapes to make sure we don’t regress on this.

@dgolubovicTT spoke to @jaykru-tt , he is on it, I think early next week is when he can have it done since Thu/Fri is off

just wanted to give context on this bug on the turn around time.

Concat allocates really big tensors and this shows up an unrelated bug due to allocation that @jaykru-tt should have a fix for. Unfortunately it only shows up in a large e2e test that shows up in the SD tests that takes 1.5 hrs to run, even a smaller unit test doesn’t recreate this with the same inputs to concat as its dependent on previous op allocations.

@jaykru-tt is working hard at this and we want to thank you for your patience.

@jaykru-tt is back to work on this. He has seen unexpected bugs related to perf tests of models due to concat allocating large tensors. His eta to get this fixed and in is tomorrow.

Thanks for addressing this so quickly @jaykru-tt, much appreciated!

Should be resolved by #14306. Let me know if you’re still seeing problems! @dgolubovicTT @tt-mpantic

@ntarafdar let’s update the Forge team on the status on this issue. 😃

During the bugs meeting this morning, the Forge team reported that this is top priority for them to unblock a single model they want to push through. It’s not a generality and sweep milestone.

Setting back to P0, as per Forge request.