tt-metal: [Bug Report] Invalid rounding with reduce_tile REDUCE_SCALAR

Describe the bug While debugging recently, I noticed that the result of reduce_scalar seemed strange, and I discovered unusual behavior.

The result of the reduce operation varies depending on the position of the values.

    constexpr auto cb_in0 = tt::CBIndex::c_0;
    {
        cb_reserve_back(cb_in0, 1);
        float* ptr = reinterpret_cast<float*>(get_write_ptr(cb_in0));
        memset(ptr, 0, 1024 * sizeof(float));
        // this case result is 2.0
        ptr[0] = 1.0019531;
        ptr[16] = 1.0f;

        // this case result is 2.0019531250
        // ptr[0] = 1.0019531;
        // ptr[1] = 1.0f;

        cb_push_back(cb_in0, 1);
    }

    {
        constexpr auto cb_in2 = tt::CBIndex::c_2;
        cb_reserve_back(cb_in2, 1);
        float* ptr = reinterpret_cast<float*>(get_write_ptr(cb_in2));
        memset(ptr, 0, 1024 * sizeof(float));
        for (int i = 0; i < 1024; i++) {
            ptr[i] = 1;
        }
        cb_push_back(cb_in2, 1);
    }

I prepared both the data and scalar values in the reader kernel for the reduce operation. And the scalar CB values are all filled with 1.0f. The test was conducted by adding the values 1.0019531 and 1.0.

if ptr[0] = 1.0019531, ptr[16] = 1.0. than reduce result is 2.0. if ptr[0] = 1.0019531, ptr[1] = 1.0. than reduce result is 2.0019531250

I think that the fact the position of the values affects the result is a bug.

  • If there were many values, it would be understandable that the calculation order could affect the result. However, since there are only two values, it’s odd that the order has an impact on the result.

related kernel code:

To Reproduce Steps to reproduce the behavior:

  1. Go to https://github.com/tenstorrent/tt-metal/tree/hyungsuk/reduce_bug_example_scalar
  2. Run cmd: pytest ./tests/ttnn/unit_tests/operations/test_moreh_softmax.py
  3. See DPRINT result

Expected behavior I hope the issue with the result can be fixed.

Screenshots If applicable, add screenshots to help explain your problem.

Please complete the following environment information:

  • OS: Ubuntu 22.04.4 LTS
  • Version of software (eg. commit f191d02edac2b821b1333c74fd239ab0830cc71e)
  • Arch: wormhole_b0

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (11 by maintainers)

Most upvoted comments

@ihnpark551 can you please review the above comments with the Moreh team. Thanks

@rdjogoTT expecting to review simulation waveforms early this week.

@hschoi4448 our team is stretched at the moment due to large number of high priority issues. @fvranicTT investigated this a bit more and realized that the error is not accumulated. That is, if he adds an additional number to the sum which shows an error the result is correct. Add another one and the error appears again. So, the error is capped with the value represented with the LSB and is appearing whenever we try to represent a value requiring more than 8 bits of mantissa.

My understanding is this issue is not blocking any progress, so we will circle back to fully clarifying whether the cause is something ill programmed in the LLK or just the artifact of how HW works when we get more bandwidth.

Position of values will always affect numerical precision, since a smaller number added to a bigger number may be ignored partly and how much is ignored is dependent on the bigger number, so in general the best way to add a sequence of numbers is to start from the smallest and go to the largest. But then again there could be other bugs causing this, but a certain amount of difference based on the position is expected.

Yes, I understand that when there are many numbers, the position of the value can affect the result. However, in this case, there are only two numbers.

1.0019531 and 1.0f; and all zeros

I believe this is a bug in this case. @amahmudTT