tt-metal: [Bug Report] ttnn.mean op - Data Mismatch
Describe the bug The ttnn.mean throws assertion error because of data mismatch between PyTorch and TTNN output and the pcc is dropped to 0.72 when the input tensor of (1, 12, 3200) and dim = -1 is passed to ttnn.mean op. For more context, here is the exact error message
def assert_with_pcc(expected_pytorch_result, actual_pytorch_result, pcc=0.9999):
assert list(expected_pytorch_result.shape) == list(
actual_pytorch_result.shape
), f"list(expected_pytorch_result.shape)={list(expected_pytorch_result.shape)} vs list(actual_pytorch_result.shape)={list(actual_pytorch_result.shape)}"
pcc_passed, pcc_message = comp_pcc(expected_pytorch_result, actual_pytorch_result, pcc)
> assert pcc_passed, construct_pcc_assert_message(pcc_message, expected_pytorch_result, actual_pytorch_result)
E AssertionError: 0.7203957195745748
To Reproduce
Run the following test:
import torch
import ttnn
from tests.ttnn.utils_for_testing import assert_with_pcc
from models.utility_functions import torch_random
def test_mean_pcc_issue(device):
torch.manual_seed(0)
input_shape = (1, 12, 3200)
reduce_dim = -1
torch_input_tensor = torch.rand(input_shape, dtype=torch.float32)
torch_output_tensor = torch.mean(torch_input_tensor, dim=reduce_dim, keepdim=True, dtype=torch.float32)
input_tensor = ttnn.from_torch(torch_input_tensor, dtype=ttnn.float32, layout=ttnn.TILE_LAYOUT, device=device)
output_tensor = ttnn.mean(input_tensor, dim=reduce_dim)
output_tensor = ttnn.to_torch(output_tensor)
assert_with_pcc(torch_output_tensor, output_tensor)
Expected behavior The data mismatch between PyTorch and TTNN output should be resolved.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 50 (40 by maintainers)
Commits related to this issue
- #13621: enable default fp32 acc for reduce — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #13621: enable default fp32 acc for reduce — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #13621: enable default fp32 acc for reduce — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #13621: adjust reduce, including tree add for reduce sum — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #13621: enable default fp32 acc for reduce — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #13621: enable default fp32 acc for reduce (#15665) ### Ticket Link to Github Issue #13621 ### Problem description reduce sum is not very accurate because fp32 acc for reduce was not enabled by... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
@dgolubovicTT Thats Right The reduce op input shape is (1, 12, 3200) Here, 1 - refers to batch_size 12 -> refers to number of token in the sequences (It will vary any number less than max sequence length) 3200 -> embedding dimension (Fixed for open llama) As @dgolubovicTT mentioned, we need ttnn.mean to work generally
I don’t think there’s anything in hardware that would help with accuracy. The pairwise summation algorithm is probably going to be easier to implement.
I think the easiest would be to change
output_tensor = tt::tt_metal::reduce(input_tensor, tt::tt_metal::ReduceOpMath::SUM...calls in ttnn/cpp/ttnn/operations/reduction/generic/generic_reductions.cppfor reduction along a single dimension to make sure that the scaling factor is 1 and then split the tensor via something like the below and then apply reduce across the sub-tensors.
Note: The split has to occur at tile boundaries. Therefore you can’t do the following:
Removing the scaling factor and then calling ttnn::multiply resulted in the following.
With all ones, increased the accuracy:
However, with random inputs, lowered PCC:
Will look at CBs to see what happened.
Update for today:
Looking at different shapes:
Next steps: As far as I can tell, the floating point errors from reduce_tile are accumulative. Therefore the next step is to try to remove the scaling factor (leave it as 1) from reduce_tile and apply it afterwards at the end. I will need to analyze how that helps PCC. That will have a performance hit, but hopefully will improve PCC.
cc @rtawfik01 for LLK visibility
Spoke to Jasmine, and @bbradelTT is for now taking over reductions. I’m reassigning this to him.