tt-metal: ttnn::transpose data mismatch when using BFLOAT16 data format
Describe the bug ttnn::transpose has data mismatch with torch::transpose with bfloat16 dataformat on specific shapes:
Input shape (1,12,32,100) and dims for transpose (-3,-2) - Tensor mismatch
Input shape (32,12,100) and dims for transpose (-1,-2) - Tensor mismatch
These two shapes are expact transposes we have in Llama attention, so this mismatch causes data mismatch in Llama attention.
There also some transpose tests failing with bfloat16:
Input shape (18, 65) and dims for transpose (1, 0) - Tensor mismatch
Input shape (18, 64) and dims for transpose (1, 0) - Tensor match
Input shape (6, 33, 34) and dims for transpose (1, 0) - Tensor mismatch
Input shape (6, 33, 32) and dims for transpose (1, 0) - Tensor match
Expected behavior It is expected for transpose to work for bfloat16 inputs without tensor missmatch.
Additional context I am attaching ttnn IR for transpose tests that are failing with tensor mismatch. Transpose_tensor_mismatch_2.txt Transpose_tensor_mismatch_1.txt
Here is a ttnn repro test:
@pytest.mark.parametrize("b", [1])
@pytest.mark.parametrize("h", [18])
@pytest.mark.parametrize("w", [65])
@pytest.mark.parametrize("dim0", [1])
@pytest.mark.parametrize("dim1", [2])
def test_transpose(device, b, h, w, dim0, dim1):
torch.manual_seed(1234)
torch_input_tensor = torch_random((b, h, w), -0.1, 0.1, dtype=torch.bfloat16)
torch_output_tensor = torch_input_tensor.transpose(dim0, dim1)
input_tensor = ttnn.to_device(ttnn.from_torch(torch_input_tensor), device, memory_config=ttnn.DRAM_MEMORY_CONFIG)
input_tensor = ttnn.to_layout(input_tensor, layout=ttnn.TILE_LAYOUT)
output_tensor = ttnn.transpose(input_tensor, dim0, dim1, memory_config=ttnn.DRAM_MEMORY_CONFIG)
output_tensor = ttnn.from_device(output_tensor)
output_tensor = ttnn.to_layout(output_tensor, layout=ttnn.ROW_MAJOR_LAYOUT)
output_tensor = ttnn.to_torch(output_tensor)
assert_with_pcc(torch_output_tensor, output_tensor)
I tried reproing this with much smaller tensor shape:
@pytest.mark.parametrize("b", [1])
@pytest.mark.parametrize("h", [2])
@pytest.mark.parametrize("w", [3])
@pytest.mark.parametrize("dim0", [-1])
@pytest.mark.parametrize("dim1", [-2])
def test_transpose(device, b, h, w, dim0, dim1):
torch.manual_seed(1234)
torch_input_tensor = torch_random((1, b, h, w), -0.1, 0.1, dtype=torch.bfloat16)
torch_output_tensor = torch_input_tensor.transpose(dim0, dim1)
input_tensor = ttnn.to_device(ttnn.from_torch(torch_input_tensor), device, memory_config=ttnn.DRAM_MEMORY_CONFIG)
input_tensor = ttnn.to_layout(input_tensor, layout=ttnn.TILE_LAYOUT)
output_tensor = ttnn.transpose(input_tensor, dim0, dim1, memory_config=ttnn.DRAM_MEMORY_CONFIG)
output_tensor = ttnn.from_device(output_tensor)
output_tensor = ttnn.to_layout(output_tensor, layout=ttnn.ROW_MAJOR_LAYOUT)
output_tensor = ttnn.to_torch(output_tensor)
print("torch_input_tensor: ")
print(torch_input_tensor)
print("output_tensor: ")
print(output_tensor)
print("torch_output_tensor: ")
print(torch_output_tensor)
assert_with_pcc(torch_output_tensor, output_tensor)
Here is the output:
torch_input_tensor:
tensor([[[[-0.0635, 0.0649, -0.0703],
[-0.0586, 0.0593, -0.0094]]]], dtype=torch.bfloat16)
output_tensor:
TorchTensor([[[[-6.3477e-02, 5.9326e-02],
[ 6.4941e-02, -9.3994e-03],
[-7.0312e-02, 3.2300e+38]]]], dtype=torch.bfloat16)
torch_output_tensor:
tensor([[[[-0.0635, -0.0586],
[ 0.0649, 0.0593],
[-0.0703, -0.0094]]]], dtype=torch.bfloat16)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (15 by maintainers)
Commits related to this issue
- #15099: add tests for forge transpose case #12705: enable odd inner-dim RM bfloat16 untilize and add some unit tests #14227: add tests for odd inner-dim RM bfloat16 transpose #13749: unaligned RM tran... — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #15099: add tests for forge transpose case #12705: enable odd inner-dim RM bfloat16 untilize and add some unit tests #14227: add tests for odd inner-dim RM bfloat16 transpose #13749: unaligned RM tran... — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #15099: add tests for forge transpose case #12705: enable odd inner-dim RM bfloat16 untilize and add some unit tests #14227: add tests for odd inner-dim RM bfloat16 transpose #13749: unaligned RM tran... — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #15099: add tests for forge transpose case #12705: enable odd inner-dim RM bfloat16 untilize and add some unit tests #14227: add tests for odd inner-dim RM bfloat16 transpose #13749: unaligned RM tran... — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #15099: add tests for forge transpose case #12705: enable odd inner-dim RM bfloat16 untilize and add some unit tests #14227: add tests for odd inner-dim RM bfloat16 transpose #13749: unaligned RM tran... — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- Add tests for untilize, transpose, and tilize on non-4B aligned row-dim Row Major tensors + enable support for these tensors on untilize (#15347) ### Ticket - #15099 - #12705 - #14227 - #13749 ... — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #15099: Add test transpose and test embedding to try and repro tensor mismatches occuring on forge. tests/ttnn/unit_tests/operations/test_embedding.py — committed to tenstorrent/tt-metal by dgolubovicTT 2 years ago
Just to clarify, this is the bug for one of the core models on Forge, therefore we set P0 for it 😃)
@ntarafdar @sjameelTT thanks for looking into this one! ❤
@nvukobratTT fyi. Transpose causes data mismatch on Llama attention, therefore P0.