tt-metal: sum/reduce op does not work for random inputs

Describe the bug The sum op (reduce) has large error for non integer inputs. Currently in test_sum.py only the summation of integers is tested - this is not sufficient and fails for random fp32 and bfloat16 inputs.

I added a test that uses random inputs and different data formats, and compares using comp_allclose. This test should pass.

What I see:

BF16 shape = (1, 1, 32, 32) ref vs tt = Max ATOL Delta: 0.0625, Max RTOL Delta: 0.068359375, PCC: 0.9999956409523759, Allclose check failed

FP32 shape = (1, 1, 32, 32) ref vs tt = Max ATOL Delta: 0.05738258361816406, Max RTOL Delta: 0.02300325594842434, PCC: 0.9999959183719673, Allclose check failed

BF16 shape = (1, 1, 32, 1024) ref vs tt = Max ATOL Delta: 2.75, Max RTOL Delta: 0.279296875, PCC: 0.9998845309940628, Allclose check failed

FP32 shape = (1, 1, 32, 1024) ref vs tt = Max ATOL Delta: 1.9558944702148438, Max RTOL Delta: 0.3209826350212097, PCC: 0.9998569160517634, Allclose check failed

Even adding two numbers fails:

Two input numbers: 1.9297, 1.4844 (all other numbers in tensor are zero)

Sum over that dim:
Torch reference (bf16): 3.4062
Torch reference (fp32): 3.4141
FP32: 3.4219
BF16: 3.4219
BFP8_B: 3.4375 
BFP4_B: 3.0

For larger shapes, e.g. [32, 1024] or [2048, 1024], this error accumulates to very large numbers.

To Reproduce Steps to reproduce the behavior:

  1. Checkout jrock/reduce-issue
  2. Run pytest tests/tt_eager/python_api_testing/unit_testing/misc/test_sum.py::test_sum

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (25 by maintainers)

Commits related to this issue

Most upvoted comments

I’m still looking into this. My new conclusion is that reduce sum works ok on dim 2 (reduce column), but reduce sum on dim 3 (reduce row) has issues with 32 bit accumulation and is loosing precision. So depending on your use case if you are using dim 2 precision should be ok if you enable fp32 acc. If you are using dim 3 there are precision issue and I’m working with @ttmtrajkovic to see if we can do something about it. Problem is on the LLK level.

So Pavles hack was basically two things: (A) use fp32 accumulation and (B) use fp32 as reduce scalar.

(A) is (somewhat) helpful for both bf16 inputs and fp32 intputs but we’ve tried this also in e.g. distirbuted layernorm and the error is still large and more than we’d expect. So this is already used in other kernels and was only missing in the reduce op itself (but is used already in distributed layernorm and I believe in SDPA too).

(B) only helps in case of fp32 inputs but not for bf16 inputs.

I think we should (1) At some point add Pavles hack for fp32 inputs (using the fp32 reduce scalar in that case improved the error 5x), but in my opinion that’s not super high priority because we are not really interested in using fp32 inputs. (2) Investigate further why we are still seeing so large error for both fp32 and especially bf16 inputs

  • @pavlejosipovic is looking into this right now, he has learned that the reduce uses a matmul under the hood (and not SFPU as previously assumed) so it essentially does a dot product with all ones. This means there’s more complexity than just adding two numbers including more configuration possibilities (e.g. HiFi settings make a difference there while they don’t on SFPU). He’s looking at a case where all numbers should be representable though bf16/fp32 and with fp32 accumulation there should in theory be no loss in precision - but he’s also there seeing non negligible error there.

https://github.com/tenstorrent/tt-metal/tree/pjosipovic/reduce32_hack I’ve hacked here an implementation for 32 bit (other data formats won’t work), I’m getting significantly better absolute error with this one. So our FPU supports only TF32 on input which is 19bit so you will get some error here converting from 32 -> 19 bits, but accumulation for sum will be in 32 bit.