tt-metal: ttnn::transpose data mismatch when using BFLOAT16 data format

Describe the bug ttnn::transpose has data mismatch with torch::transpose with bfloat16 dataformat on specific shapes:

Input shape (1,12,32,100) and dims for transpose (-3,-2) - Tensor mismatch

Input shape (32,12,100) and dims for transpose (-1,-2) - Tensor mismatch

These two shapes are expact transposes we have in Llama attention, so this mismatch causes data mismatch in Llama attention.

There also some transpose tests failing with bfloat16:

Input shape (18, 65) and dims for transpose (1, 0) - Tensor mismatch

Input shape (18, 64) and dims for transpose (1, 0) - Tensor match
Input shape (6, 33, 34) and dims for transpose (1, 0) - Tensor mismatch

Input shape (6, 33, 32) and dims for transpose (1, 0) - Tensor match

Expected behavior It is expected for transpose to work for bfloat16 inputs without tensor missmatch.

Additional context I am attaching ttnn IR for transpose tests that are failing with tensor mismatch. Transpose_tensor_mismatch_2.txt Transpose_tensor_mismatch_1.txt

Here is a ttnn repro test:

@pytest.mark.parametrize("b", [1])
@pytest.mark.parametrize("h", [18])
@pytest.mark.parametrize("w", [65])
@pytest.mark.parametrize("dim0", [1])
@pytest.mark.parametrize("dim1", [2])
def test_transpose(device, b, h, w, dim0, dim1):
    torch.manual_seed(1234)

    torch_input_tensor = torch_random((b, h, w), -0.1, 0.1, dtype=torch.bfloat16)
    torch_output_tensor = torch_input_tensor.transpose(dim0, dim1)

    input_tensor = ttnn.to_device(ttnn.from_torch(torch_input_tensor), device, memory_config=ttnn.DRAM_MEMORY_CONFIG)
    input_tensor = ttnn.to_layout(input_tensor, layout=ttnn.TILE_LAYOUT)
    output_tensor = ttnn.transpose(input_tensor, dim0, dim1, memory_config=ttnn.DRAM_MEMORY_CONFIG)
    output_tensor = ttnn.from_device(output_tensor)
    output_tensor = ttnn.to_layout(output_tensor, layout=ttnn.ROW_MAJOR_LAYOUT)
    output_tensor = ttnn.to_torch(output_tensor)

    assert_with_pcc(torch_output_tensor, output_tensor)

I tried reproing this with much smaller tensor shape:

@pytest.mark.parametrize("b", [1])
@pytest.mark.parametrize("h", [2])
@pytest.mark.parametrize("w", [3])
@pytest.mark.parametrize("dim0", [-1])
@pytest.mark.parametrize("dim1", [-2])
def test_transpose(device, b, h, w, dim0, dim1):
    torch.manual_seed(1234)

    torch_input_tensor = torch_random((1, b, h, w), -0.1, 0.1, dtype=torch.bfloat16)
    torch_output_tensor = torch_input_tensor.transpose(dim0, dim1)

    input_tensor = ttnn.to_device(ttnn.from_torch(torch_input_tensor), device, memory_config=ttnn.DRAM_MEMORY_CONFIG)
    input_tensor = ttnn.to_layout(input_tensor, layout=ttnn.TILE_LAYOUT)
    output_tensor = ttnn.transpose(input_tensor, dim0, dim1, memory_config=ttnn.DRAM_MEMORY_CONFIG)
    output_tensor = ttnn.from_device(output_tensor)
    output_tensor = ttnn.to_layout(output_tensor, layout=ttnn.ROW_MAJOR_LAYOUT)
    output_tensor = ttnn.to_torch(output_tensor)
    print("torch_input_tensor: ")
    print(torch_input_tensor)
    print("output_tensor: ")
    print(output_tensor)
    print("torch_output_tensor: ")
    print(torch_output_tensor)

    assert_with_pcc(torch_output_tensor, output_tensor)

Here is the output:

torch_input_tensor: 
tensor([[[[-0.0635,  0.0649, -0.0703],
          [-0.0586,  0.0593, -0.0094]]]], dtype=torch.bfloat16)
output_tensor: 
TorchTensor([[[[-6.3477e-02,  5.9326e-02],
               [ 6.4941e-02, -9.3994e-03],
               [-7.0312e-02,  3.2300e+38]]]], dtype=torch.bfloat16)
torch_output_tensor: 
tensor([[[[-0.0635, -0.0586],
          [ 0.0649,  0.0593],
          [-0.0703, -0.0094]]]], dtype=torch.bfloat16)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 26 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Just to clarify, this is the bug for one of the core models on Forge, therefore we set P0 for it 😃)

@ntarafdar @sjameelTT thanks for looking into this one! ❤

@nvukobratTT fyi. Transpose causes data mismatch on Llama attention, therefore P0.