tt-metal: [Bug Report] Incorrect rounding when using `bfp8` in `ttnn.from_torch`.
Describe the bug
There is incorrect rounding when using bfp8 in ttnn.from_torch.
To Reproduce Steps to reproduce the behavior:
import pytest
import torch
import ttnn
def test_example(device):
# bfp8_b rounding issue.
# mantissa for 64.5 is 0000 001
# According to the "Round to nearest, ties to even" rule, the mantissa should be 0.
value = 64.5
torch_input = torch.tensor([value], dtype=torch.float)
tt_input = ttnn.from_torch(torch_input, layout=ttnn.TILE_LAYOUT, dtype=ttnn.bfloat8_b, device=device)
tt_output = ttnn.to_torch(tt_input)
print("python round", round(value)) # 64
print("tt_output ", tt_output) # 65. this is not ties to even
print("===================================")
# bfp8_b rounding issue2, mantissa overflow was not handled properly
# this is the case all mantissa is 1 and sholud round up
value = 7.9999
torch_input = torch.tensor([value], dtype=torch.float)
tt_input = ttnn.from_torch(torch_input, layout=ttnn.TILE_LAYOUT, dtype=ttnn.bfloat8_b, device=device)
tt_output = ttnn.to_torch(tt_input)
torch_output = torch_input.to(dtype=torch.bfloat16)
print("torch_output", torch_input.round()) # 8
print("tt_output ", tt_output) # 7.9375
- Run above comment
- See result
The first case is an example where 64.5 is converted to the bfp8_b type, and rounding occurs. According to the “Round to nearest, ties to even” rule, 64.5 should round to 64, but for some unknown reason, it has rounded to 65.
The second case is when 7.9999 is rounded. 7.9999 should round to 8, but it seems that bfp8 has an issue handling the case where the mantissa is all 1s, which prevents it from rounding to 8.
As a result, instead of 8, the value becomes 7.9375, leading to an error of 0.0625, when the expected error should have been 0.0001.
Expected behavior
The results of ttnn.from_torch for bfp8 should follow the “Round to nearest, ties to even” rule.
Please complete the following environment information:
- OS: [e.g. Ubuntu 20.04]
- Version of software (eg. commit 23ce00c668d7bff14c98657d1245f58a65fe0cef)
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (17 by maintainers)
Commits related to this issue
- #15889: Fix handling of mantissa rounding to respect round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bfloat8... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to nikileshx/tt-metal by TT-BrianLiu 2 years ago
- #15889: Fix handling of mantissa rounding to respect ties round to even - Add tt_metal gtest to test convert_u32_to_bfp rounding logic for bfp8 - Remove unused functions in tt_metal/api/tt-metalium/bf... — committed to tenstorrent/tt-metal by TT-BrianLiu 2 years ago
Folks, thanks for the changes and comments! Closing the issue as completed 😃
@TT-BrianLiu @hschoi4448
We just confirmed with the HW team that WH and BH both DO NOT support rounding to even, only rounding up in case of ties. This means the original implementation was accurately modelling HW.
Regarding the mantissa overflow case, this is also correctly modelling what the HW does in this case. The HW also would not increment the exponent, but round down to 7.9375 instead.
In both cases we are dealing with HW limitations for deterministic rounding. If you want more even rounding, you would have to sacrifice determinism and use stochastic rounding.