tt-metal: BH TTNN Unit failure in test for reduction
https://github.com/tenstorrent/tt-metal/actions/runs/14918580796/job/41909745136
FAILED tests/ttnn/unit_tests/operations/reduce/test_reduction.py::test_2d_topk[dtype=DataType.BFLOAT16-largest=True-k=50-dim=1-dim2=50257-dim1=1] - AssertionError: Cosine similarity between topk values and gather from indices is less than 0.99
assert TorchTensor(0.980, dtype=torch.bfloat16) > 0.99
0.98 is close to 0.99.
Might be other errors in test_reduction.py as well.
Need to be investigated.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- Mark topk tests as xfail on blackhole until issue is resolved (#21931) ### Ticket https://github.com/tenstorrent/tt-metal/issues/21881 ### Problem description Pipeline runs use `-x` option with pyte... — committed to tenstorrent/tt-metal by nsorabaTT 2 years ago
- Fix llk_unpack_A_hw_configure CB index. Fixes #21881. The CB index was hardcoded to 0, which meant it would not disable flush denormals to zero in the case of uint16 for CB index != 0. — committed to jasondavies/tt-metal by jasondavies a year ago
- transpose_wh_init: fix incorrect llk_unpack_A_hw_configure CB index. (#23246) ### Ticket #21881 ### Problem description `transpose_wh_init` used a hardcoded CB index of 0 for `llk_unpack_A_hw_conf... — committed to tenstorrent/tt-metal by jasondavies a year ago
- Address random failures in topk unit tests (#23469) ### Ticket https://github.com/tenstorrent/tt-metal/issues/21881 ### Problem description Topk unit test with 2D tensor fails only on blackhole. On ... — committed to tenstorrent/tt-metal by nsorabaTT a year ago
@nsorabaTT Fixed in #23246.
@jasondavies I have updated the branch with a fix to the unit tests so that it fails for all seeds. This will reliably reproduce the error on both blackhole and wormhole.