tt-metal: [Bug Report] ttnn.mean op - Data Mismatch

Describe the bug The ttnn.mean throws assertion error because of data mismatch between PyTorch and TTNN output and the pcc is dropped to 0.72 when the input tensor of (1, 12, 3200) and dim = -1 is passed to ttnn.mean op. For more context, here is the exact error message

def assert_with_pcc(expected_pytorch_result, actual_pytorch_result, pcc=0.9999):
        assert list(expected_pytorch_result.shape) == list(
            actual_pytorch_result.shape
        ), f"list(expected_pytorch_result.shape)={list(expected_pytorch_result.shape)} vs list(actual_pytorch_result.shape)={list(actual_pytorch_result.shape)}"
        pcc_passed, pcc_message = comp_pcc(expected_pytorch_result, actual_pytorch_result, pcc)
>       assert pcc_passed, construct_pcc_assert_message(pcc_message, expected_pytorch_result, actual_pytorch_result)
E       AssertionError: 0.7203957195745748

To Reproduce

Run the following test:

import torch
import ttnn
from tests.ttnn.utils_for_testing import assert_with_pcc
from models.utility_functions import torch_random
def test_mean_pcc_issue(device):
    torch.manual_seed(0)

    input_shape = (1, 12, 3200)
    reduce_dim = -1
    
    torch_input_tensor = torch.rand(input_shape, dtype=torch.float32)
    torch_output_tensor = torch.mean(torch_input_tensor, dim=reduce_dim, keepdim=True, dtype=torch.float32)

    input_tensor = ttnn.from_torch(torch_input_tensor, dtype=ttnn.float32, layout=ttnn.TILE_LAYOUT, device=device)
    
    output_tensor = ttnn.mean(input_tensor, dim=reduce_dim)
    output_tensor = ttnn.to_torch(output_tensor)
    
    assert_with_pcc(torch_output_tensor, output_tensor)

Expected behavior The data mismatch between PyTorch and TTNN output should be resolved.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 50 (40 by maintainers)

Commits related to this issue

Most upvoted comments

@dgolubovicTT Thats Right The reduce op input shape is (1, 12, 3200) Here, 1 - refers to batch_size 12 -> refers to number of token in the sequences (It will vary any number less than max sequence length) 3200 -> embedding dimension (Fixed for open llama) As @dgolubovicTT mentioned, we need ttnn.mean to work generally

I don’t think there’s anything in hardware that would help with accuracy. The pairwise summation algorithm is probably going to be easier to implement.

I think the easiest would be to change output_tensor = tt::tt_metal::reduce(input_tensor, tt::tt_metal::ReduceOpMath::SUM... calls in ttnn/cpp/ttnn/operations/reduction/generic/generic_reductions.cpp

for reduction along a single dimension to make sure that the scaling factor is 1 and then split the tensor via something like the below and then apply reduce across the sub-tensors.

    y = torch.ones((1, 1, 1024, 8192), dtype=torch.float32)
    y_tt = ttnn.from_torch(y, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)
    a = ttnn.slice(y_tt, [0, 0, 0, 0], [1, 1, 1024, 4096])
    b = ttnn.slice(y_tt, [0, 0, 0, 4096], [1, 1, 1024, 8192])

Note: The split has to occur at tile boundaries. Therefore you can’t do the following:


Something like the following will not work:
z = torch.ones((1, 1, 50, 70), dtype=torch.float32)
z_tt = ttnn.from_torch(z, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)
c = ttnn.slice(z_tt, [0, 0, 0, 0], [1, 1, 50, 35])
d = ttnn.slice(z_tt, [0, 0, 0, 35], [1, 1, 50, 70])

You would need to split it at 32 or 64 instead of 35.

If the split is across two dimensions, then the PCCs will continue to be as they currently are. However, that would take longer and is probably best addressed in the future. This will take at least several days to implement. I'll continue daily updates, although they will be short.

Removing the scaling factor and then calling ttnn::multiply resulted in the following.

With all ones, increased the accuracy:

  • 1,1,12,800 output is 0.9961
  • 1,1,12,3200 output is 0.9961
  • 1,1,12,2048 output is still 1

However, with random inputs, lowered PCC:

expected_pytorch_result = tensor([[[0.4979],
         [0.4969],
         [0.5080],
         [0.5029],
         [0.5012],
         [0.5046],
         [0.4993],
         [0.5034],
         [0.5109],
         [0.4984],
         [0.4972],
         [0.4963]]])
actual_pytorch_result = TorchTensor([[[0.4961],
              [0.4980],
              [0.5000],
              [0.4961],
              [0.5039]...            [0.5039],
              [0.5078],
              [0.5000],
              [0.5078],
              [0.4961]]])
pcc = 0.9999

    def assert_with_pcc(expected_pytorch_result, actual_pytorch_result, pcc=0.9999):
        assert list(expected_pytorch_result.shape) == list(
            actual_pytorch_result.shape
        ), f"list(expected_pytorch_result.shape)={list(expected_pytorch_result.shape)} vs list(actual_pytorch_result.shape)={list(actual_pytorch_result.shape)}"
        pcc_passed, pcc_message = comp_pcc(expected_pytorch_result, actual_pytorch_result, pcc)
>       assert pcc_passed, construct_pcc_assert_message(pcc_message, expected_pytorch_result, actual_pytorch_result)
E       AssertionError: 0.3941431853119553

Will look at CBs to see what happened.

Update for today:

  • instrumented compute kernel to print out input CBs and output CB and used all ones as input.
  • Used ones instead of random values.
  • There is a scalar factor that is 1/n. When n is not a power of 2, there is error introduced.
  • The reduce_tile operation is called for each tile of the reduced operation, multiplying the error.

Looking at different shapes:

  • 1,1,12,32 scalar is exact, output is 1
  • 1,1,12,800 scalar is 0.00124359 vs .00125 (.6% off), output is 0.988281
  • 1,1,12,3200 scalar is 0.000310898 vs 0.0003125 (.5% off), output is 0.917969
  • 1,1,12,2048 scalar is exact, output is 1

Next steps: As far as I can tell, the floating point errors from reduce_tile are accumulative. Therefore the next step is to try to remove the scaling factor (leave it as 1) from reduce_tile and apply it afterwards at the end. I will need to analyze how that helps PCC. That will have a performance hit, but hopefully will improve PCC.

cc @rtawfik01 for LLK visibility

Spoke to Jasmine, and @bbradelTT is for now taking over reductions. I’m reassigning this to him.