tt-metal: The `mul_tiles` results are randomly incorrect.

Describe the bug

Recently, I observed that the results of mul_tiles are randomly incorrect.

So I proceeded with debugging and found that the issue is highly likely to be reproduced with the code below.

The operation performed in the code below is as follows.

  1. The prim.example I modified takes an input tensor and uses copy_tiles to copy it to the output tensor.
  2. And the modified example_multiple_return multiplies input and other using mul_tiles, and returns the result as output.
  3. In the pytest, an input tensor with the value 2 and an other tensor with the value 0 are created. The other tensor is copied into output1. Then, the input and output1 are multiplied, and the result is compared against 0.
  • 2 * 0 should always be 0. But sometimes, it is not 0.
import pytest
import torch
import ttnn

def test_example(device):
    torch.manual_seed(0)

    shape = [5 * 32, 32]

    torch_input = torch.ones(shape, dtype=torch.bfloat16) * 2
    torch_other = torch.zeros(shape, dtype=torch.bfloat16)

    expected_otput = torch_input * torch_other

    for x in range(100):
        print("iter", x)
        tt_input = ttnn.from_torch(torch_input, layout=ttnn.TILE_LAYOUT, device=device)
        tt_other = ttnn.from_torch(torch_other, layout=ttnn.TILE_LAYOUT, device=device)

        tt_output1 = ttnn.empty(shape, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)
        tt_output = ttnn.empty(shape, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)

        ttnn.prim.example(tt_other, tt_output1) # copy 0 to 0
        ttnn.prim.example_multiple_return(tt_input, tt_output1, output=tt_output) # mul_tiles 2 * 0

        actual_output = ttnn.to_torch(tt_output)

        ret = torch.allclose(actual_output, expected_otput, atol=0, rtol=0)
        if not ret:
            print("tt_input", tt_input)
            print("tt_output1", tt_output1)
            print("actual_output", actual_output)
            print("expected_otput", expected_otput)
        assert torch.allclose(actual_output, expected_otput, atol=0, rtol=0)

This issue does not occur if the unnecessary code, which was added just to reproduce the problem, is removed from eltwise_sfpu.cpp in the example_multiple_return.

#include "hostdevcommon/kernel_structs.h" // code for reduce error
#include "compute_kernel_api.h"
#include "compute_kernel_api/eltwise_binary.h"
#include "debug/dprint.h" 

uint32_t cb_srca = 256;  // code for reduce error
uint32_t cb_srcb = 256;  // code for reduce error

constexpr std::uint32_t onetile = 1; // code for reduce error

template <bool to_from_int8 = false> // code for reduce error
ALWI void test_func(const uint32_t srca_new_operand, const uint32_t srcb_new_operand) { // code for reduce error
    DPRINT << "AAAAA 1\n"; // code for reduce error
} // code for reduce error

inline bool dummy_compare(std::uint32_t old_operand, std::uint32_t new_operand) { // code for reduce error
    return (unpack_src_format[old_operand] != unpack_src_format[new_operand]) || // code for reduce error
           (unpack_dst_format[old_operand] != unpack_dst_format[new_operand]); // code for reduce error
} // code for reduce error

namespace NAMESPACE {
void MAIN {
    // Circular Buffers
    constexpr auto cb_in = tt::CBIndex::c_0;
    constexpr auto cb_other = tt::CBIndex::c_1;
    constexpr auto cb_out = tt::CBIndex::c_2;

    // Variables
    constexpr uint32_t input_dst_reg = 0;

    binary_op_init_common(cb_in, cb_other, cb_out);

    tile_regs_acquire();
    cb_wait_front(cb_in, onetile);
    cb_wait_front(cb_other, onetile);

   // code for reduce error
    {
        DPRINT << "DST_ACCUM_MODE " << static_cast<uint32_t>(DST_ACCUM_MODE) << "\n";
        if (cb_srca == 256 || cb_srcb == 256) {
            // do nothing
            test_func<DST_ACCUM_MODE>(cb_in, cb_other);
        } else {
            // not executed
            UNPACK(if (dummy_compare(cb_srca, cb_in)) { TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::UNPACK0); }) 
        }

        cb_srca = cb_in; // code for reduce error
        cb_srcb = cb_other; // code for reduce error
    }

    mul_tiles_init(cb_in, cb_other);
    mul_tiles(cb_in, cb_other, 0, 0, 0);

    cb_pop_front(cb_in, onetile);
    cb_pop_front(cb_other, onetile);
    tile_regs_commit();

    tile_regs_wait();
    cb_reserve_back(cb_out, onetile);
    pack_tile(0, cb_out);
    cb_push_back(cb_out, onetile);
    tile_regs_release();
}
}  // namespace NAMESPACE

Steps to reproduce the issue

  1. checkout https://github.com/tenstorrent/tt-metal/tree/hyungsuk/mul_random_bug
  2. build on debug mode
  3. Set Env (Important)
export TT_METAL_LOGGER_LEVEL=Debug
export TT_METAL_LOGGER_TYPES=Test
export TT_METAL_DPRINT_CORES="(0,0)"
  1. go to tests/ttnn/unit_tests/operations
  2. run pytest ./test_examples.py
  3. See error

Expected behavior

2 * 0 must be 0. always.

Please complete the following environment information

device: wormhole_b0

Updated reproducer instructions (July 4) - by Anil Mahmud

  1. Checkout tt-metal branch origin/amahmud/mul_random_bug commit id https://github.com/tenstorrent/tt-metal/commit/c94f33d219567e5178ebb2173f10c54da32417f0

  2. You do not need to set any debug build/DPRINT core/ LOG level

  3. go to tests/ttnn/unit_tests/operations

  4. run pytest ./test_examples.py

  5. See the failure. The failure will happen probablistically after a random number of iterations (around 5-10, explained inside the test case). The test runs on a single core.

Note : The issue is not related to any dvalid signals (as confirmed by simulations)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 75 (66 by maintainers)

Most upvoted comments

I checked with @rdjogoTT’s changes and the test passes as expected. The test case I used has a high rate of reproducing the error but after more than 1000 attempts it did not fail. I am making the issue Radomir is working on as the parent issue and when his PR is merged, I will close this issue.

EDIT : Will not close the issue, will wait for green signal from @hschoi4448

The issue lies in the race on the ALU_ACC_CTRL_Zero_Flag_disabled_src bit (datacopy needs this bit set while mul needs it cleared), in the failing iteration this bit fails to clear (gets cleared for only a couple of cycles and then restored) causing wrong FPU results. The cause of the race is that the register that stores that bit also stores some packer bits meaning that the same register is accessed from both the unpacker and the packer thread. Unpacker thread uses tensix RMW instructions to modify that bit (cunpack_common.h:294) while packer uses explicit risc mmio read modify write sequence outside of the appropriate mutex guards (cpack_common.h:554) if the unpack RMW happens between the pack read and write the unpack change will be lost. @rdjogoTT has come to the same conclusion while debugging some reduce issues, so the fix is available at rd/reduce_w_fp32. @amahmudTT will pick up those changes and test on current repros.

@ihnpark551 , @amahmudTT will take a look from the LLK side to debug this behavior. He just started looking into this yesterday.

The LLK team (@nvelickovicTT ) will have a look at this shortly

@hschoi4448 the relevant PR has been merged, you can test your test cases and close this issue if you think everything is okay.

I tested on main branch and it seems like problem fixed.

Thank you.

@amahmudTT

Yes, (but you need to fix the factory related bug where I saw 1 instead of i, to not get wrong results in differing scenarios)

Thank you, I’ll wait for the parent issue to be resolved.

However, from what I gathered, it seems that the root cause has been identified, and once the parent issue(tenstorrent/tt-llk#122) is merged, my issue will be resolved as well — is my understanding correct? @amahmudTT

Yes, (but you need to fix the factory related bug where I saw 1 instead of i, to not get wrong results in differing scenarios)

@lpremovicTT I did both because there is another reg being programmed using tensix RMW so I put them together. It’s also how I see it done in cunpack_common.h. We should figure out why that is.

Was able to reproduce the issue in versim for a 2 tile problem, just that in the simulation one tile had zeroes and the other tile had nans, but in silicon one tile has zeroes and the other tile has -1s.

For a single tile problem i could not reproduce the issue in versim, but then again single tile one only fails on silicon if I have run a two tile version and failed beforehand. Next step is to analyze the versim files with somebody adept in looking at waveforms 😃

I am currently trying to remove the extra dummy code and replace them with variable number of nops and check if the error could be made deterministic.

@amahmudTT is starting to look at this from the LLK side.

Is it possible to create a failing test that does not require dprint?

I tried a few things to create a failing test without DPRINT, but the issue didn’t occur in my old branch mul_random_bug.

If it can’t be reproduced easily, I think I’ll need to go back to the model level, check if random loss occurs, and start debugging from there. But that would take more than a week, so it might be difficult to proceed with that.

Would it be possible for you to continue debugging with DPRINT? @abhullar-tt @ihnpark551

Yeah I did capture a simulation with dprint enabled. We can wait on LLK team’s analysis to determine if another repro case is needed

The branch includes some hacks to enable simulator + dprint at the same time but it isn’t fully robust. @ihnpark551 / @hschoi4448 is it possible to create a failing test that does not require dprint?

I’m not fully following the conversation, so I’m not sure — but could you clarify why reproducing the issue without DPRINT is necessary? ~Was DPRINT the cause of the problem and has now been fixed, and you’re trying to confirm whether the issue still reproduces after that?~

to enable simulator + dprint at the same time but it isn’t fully robust

I found request context

@abhullar-tt

Now that versim is functional for WH, we will try to simulate this test as a next step

  1. @nathan-TT will take a look to see what might be going on
  2. a fix was merged into tt-exalens to fix the write issue

@amahmudTT @zzigler-tt If @amahmudTT should deal with other issue, could any other TT member can be assigned to this issue?

I hope to be back soon ~(a week), and in the meantime AFAIK @bbradelTT was going to have a short look from the op side.

I found printing out the dest before and after multiles, in the erroneous cases the dest starts with zeroes but ends up with -1s. Just to make sure I am not getting lost in the printouts, If I print only before mum_tiles i never get -1s even though the result is -1 but when I print after mul_tiles i do get -1s in dest when result is -1s. So we can rule out the issue of zeroacc. The problem is coming from somewhere else. Althought the zeroacc should be used anyways (until and unless somewhere else somebody is doing it (the packer sometimes does it).

Next step would be to use perturbation in timing to reduce the test case and try to create a single core repro.

@hschoi4448 I checked on my end with your example, and here is what I got:

1. Both DEST and L1 memory of the cb_out are all 0's even when there is non 0 value in the output. I used tt-exalens to read L1 after I hit the issue. Can you try the same experiment on your end?

2. The only non-zero value I am seeing is 1.1755e-38, which represented in BFLOAT16 is 0 00000001 0000000. So the exponent somehow gets incremented.

3. Adding a delay in the writer_unary.cpp kernel after `noc_async_` calls solves the issue. Which leads me to think it is a race of some sort, but not between compute and data movement, since I add the delay at the end of the writer kernel.

4. I then printed `tt_output` tensor from python and it also seems to be all 0's.

From this it seems to me that the issue is happening in ttnn.to_torch() call. @hschoi4448 could you also check on your side as well whether TTNN tensor is fine. Before iit is converted to torch tensor.

Thank you for debugging problem! But I found that there is an incorrect value in cb_out using DPRINT in UNPACK.

So I updated test code for easier debugging.

What I changed

  1. Change test number to -1.7014118e38 * 0, and invalid output is -1.
  • invalid output is much bigger, so easy to check without SETPRECISION
  1. Add dprint cb_out in compute kernel

Now you need change TT_METAL_DPRINT for better check.

export TT_METAL_LOGGER_LEVEL=Debug
export TT_METAL_LOGGER_TYPES=Test
export TT_METAL_DPRINT_CORES="(0,0)-(7,6)"

if you run test, you can see dprint like below.

Image

@nvelickovicTT

@hschoi4448 I checked on my end with your example, and here is what I got:

  1. Both DEST and L1 memory of the cb_out are all 0’s even when there is non 0 value in the output. I used tt-exalens to read L1 after I hit the issue. Can you try the same experiment on your end?
  2. The only non-zero value I am seeing is 1.1755e-38, which represented in BFLOAT16 is 0 00000001 0000000. So the exponent somehow gets incremented.
  3. Adding a delay in the writer_unary.cpp kernel after noc_async_ calls solves the issue. Which leads me to think it is a race of some sort, but not between compute and data movement, since I add the delay at the end of the writer kernel.
  4. I then printed tt_output tensor from python and it also seems to be all 0’s.

From this it seems to me that the issue is happening in ttnn.to_torch() call. @hschoi4448 could you also check on your side as well whether TTNN tensor is fine. Before iit is converted to torch tensor.

@cmaryanTT @ayerofieiev-tt can you please review and triage. TY