tt-metal: sum/reduce op does not work for random inputs
Describe the bug
The sum op (reduce) has large error for non integer inputs. Currently in test_sum.py only the summation of integers is tested - this is not sufficient and fails for random fp32 and bfloat16 inputs.
I added a test that uses random inputs and different data formats, and compares using comp_allclose. This test should pass.
What I see:
BF16 shape = (1, 1, 32, 32) ref vs tt = Max ATOL Delta: 0.0625, Max RTOL Delta: 0.068359375, PCC: 0.9999956409523759, Allclose check failed
FP32 shape = (1, 1, 32, 32) ref vs tt = Max ATOL Delta: 0.05738258361816406, Max RTOL Delta: 0.02300325594842434, PCC: 0.9999959183719673, Allclose check failed
BF16 shape = (1, 1, 32, 1024) ref vs tt = Max ATOL Delta: 2.75, Max RTOL Delta: 0.279296875, PCC: 0.9998845309940628, Allclose check failed
FP32 shape = (1, 1, 32, 1024) ref vs tt = Max ATOL Delta: 1.9558944702148438, Max RTOL Delta: 0.3209826350212097, PCC: 0.9998569160517634, Allclose check failed
Even adding two numbers fails:
Two input numbers: 1.9297, 1.4844 (all other numbers in tensor are zero)
Sum over that dim:
Torch reference (bf16): 3.4062
Torch reference (fp32): 3.4141
FP32: 3.4219
BF16: 3.4219
BFP8_B: 3.4375
BFP4_B: 3.0
For larger shapes, e.g. [32, 1024] or [2048, 1024], this error accumulates to very large numbers.
To Reproduce Steps to reproduce the behavior:
- Checkout
jrock/reduce-issue - Run
pytest tests/tt_eager/python_api_testing/unit_testing/misc/test_sum.py::test_sum
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 25 (25 by maintainers)
Commits related to this issue
- #9908: Add reduce/sum test with random inputs — committed to tenstorrent/tt-metal by johanna-rock-tt 2 years ago
- #9908: Add reduce/sum test with random inputs — committed to tenstorrent/tt-metal by johanna-rock-tt 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Apply PR feedback — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Apply PR feedback — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Apply PR feedback 2 — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Typo — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is workedaround by... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #9908: Use mm tiles for reduce sum on w dim (#10933) LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is wo... — committed to tenstorrent/tt-metal by pavlejosipovic 2 years ago
- #9908: Use mm tiles for reduce sum on w dim (#10933) LLK impl for reduce row has precision issues. Reduce column doesn't seem to have the same issue. For reduce sum on row(w) dim the issue is wo... — committed to tenstorrent/tt-metal by pavlejosipovic 2 years ago
I’m still looking into this. My new conclusion is that reduce sum works ok on dim 2 (reduce column), but reduce sum on dim 3 (reduce row) has issues with 32 bit accumulation and is loosing precision. So depending on your use case if you are using dim 2 precision should be ok if you enable fp32 acc. If you are using dim 3 there are precision issue and I’m working with @ttmtrajkovic to see if we can do something about it. Problem is on the LLK level.
So Pavles hack was basically two things: (A) use fp32 accumulation and (B) use fp32 as reduce scalar.
(A) is (somewhat) helpful for both bf16 inputs and fp32 intputs but we’ve tried this also in e.g. distirbuted layernorm and the error is still large and more than we’d expect. So this is already used in other kernels and was only missing in the reduce op itself (but is used already in distributed layernorm and I believe in SDPA too).
(B) only helps in case of fp32 inputs but not for bf16 inputs.
I think we should (1) At some point add Pavles hack for fp32 inputs (using the fp32 reduce scalar in that case improved the error 5x), but in my opinion that’s not super high priority because we are not really interested in using fp32 inputs. (2) Investigate further why we are still seeing so large error for both fp32 and especially bf16 inputs
https://github.com/tenstorrent/tt-metal/tree/pjosipovic/reduce32_hack I’ve hacked here an implementation for 32 bit (other data formats won’t work), I’m getting significantly better absolute error with this one. So our FPU supports only TF32 on input which is 19bit so you will get some error here converting from 32 -> 19 bits, but accumulation for sum will be in 32 bit.