tt-metal: [Bug Report] Inconsistent Values When Converting Torch Tensor to TTNN Tensor with bfloat8_b Type

Describe the bug When creating a ttnn tensor with the bfloat8_b type from a Torch tensor containing random numbers between 0 and 1, the ttnn tensor does not match the Torch tensor values exactly, even though all numbers should have the same exponent (0). There are small differences in the printed values, which is unexpected.

To Reproduce Steps to reproduce the behavior:

  1. Create a Torch tensor with random numbers between 0 and 1, using the torch.bfloat16 data type.
  2. Convert the Torch tensor to a ttnn tensor with the bfloat8_b type.
  3. Print the values of both the ttnn tensor and the original Torch tensor.
  4. Compare the printed results to observe differences.
npu_dtype = ttnn.bfloat8_b
cpu_dtype = torch.bfloat16
npu_layout = ttnn.TILE_LAYOUT

torch_input = torch.rand(input_shape, dtype=cpu_dtype, requires_grad=True) 

tt_input = ttnn.from_torch(torch_input, dtype=npu_dtype, layout=npu_layout, device=device)
tt_input_cpu = ttnn.to_torch(tt_input)

print(tt_input_cpu)
print(torch_input)

Expected behavior The values in the ttnn tensor should exactly match the values in the original Torch tensor, as both tensors represent numbers between 0 and 1 with the same exponent (0).

Screenshots Here are the printed results showing the differences:

torchTensor([[[[0.3438, 0.1016, 0.7734, 0.7188, 0.6328, 0.8594, 0.0156, 0.4531,
                0.2031, 0.8359, 0.3984, 0.8750, 0.0859, 0.5703, 0.9375, 0.7109,
                0.6875, 0.3047, 0.1562, 0.7734, 0.8438, 0.9453, 0.5156, 0.2266,
                0.1094, 0.8203, 0.4297, 0.9375, 0.7812, 0.2500, 0.9453],...
ttnnTensor([[[[0.3398, 0.0977, 0.7734, 0.7148, 0.6289, 0.8594, 0.0117, 0.4531,
           0.2031, 0.8359, 0.3945, 0.8750, 0.0859, 0.5664, 0.9336, 0.7070,
           0.6836, 0.3008, 0.1523, 0.7695, 0.8438, 0.9453, 0.5117, 0.2266,
           0.1094, 0.8164, 0.4297, 0.9375, 0.7773, 0.2461, 0.9414],...

Please complete the following environment information:

  • OS: Ubuntu 22.04.4
  • Version of software: 567c64b
  • Device: n150

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (12 by maintainers)

Most upvoted comments

For BFP8, we also additionally store the hidden bit in the MSB of mantissa (which is always 1 for BF16 and is not stored). So we end up losing the LSB, which accounts for the difference (even if exponents are the exact same). I updated this slide here: Image

Starting on this today!

Update: There was a misunderstanding on my part. The random numbers should actually be between 1 and 2 to share the same exponent. I’ve updated the test accordingly, but there are still some differences between the two tensors. You can take a look at this file to help with debugging.