tt-metal: The `mul_tiles` results are randomly incorrect.
Describe the bug
Recently, I observed that the results of mul_tiles are randomly incorrect.
So I proceeded with debugging and found that the issue is highly likely to be reproduced with the code below.
The operation performed in the code below is as follows.
- The
prim.exampleI modified takes an input tensor and usescopy_tilesto copy it to the output tensor. - And the modified
example_multiple_returnmultipliesinputandotherusingmul_tiles, and returns the result asoutput. - In the pytest, an
inputtensor with the value 2 and anothertensor with the value 0 are created. Theothertensor is copied intooutput1. Then, theinputandoutput1are multiplied, and the result is compared against 0.
- 2 * 0 should always be 0. But sometimes, it is not 0.
import pytest
import torch
import ttnn
def test_example(device):
torch.manual_seed(0)
shape = [5 * 32, 32]
torch_input = torch.ones(shape, dtype=torch.bfloat16) * 2
torch_other = torch.zeros(shape, dtype=torch.bfloat16)
expected_otput = torch_input * torch_other
for x in range(100):
print("iter", x)
tt_input = ttnn.from_torch(torch_input, layout=ttnn.TILE_LAYOUT, device=device)
tt_other = ttnn.from_torch(torch_other, layout=ttnn.TILE_LAYOUT, device=device)
tt_output1 = ttnn.empty(shape, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)
tt_output = ttnn.empty(shape, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)
ttnn.prim.example(tt_other, tt_output1) # copy 0 to 0
ttnn.prim.example_multiple_return(tt_input, tt_output1, output=tt_output) # mul_tiles 2 * 0
actual_output = ttnn.to_torch(tt_output)
ret = torch.allclose(actual_output, expected_otput, atol=0, rtol=0)
if not ret:
print("tt_input", tt_input)
print("tt_output1", tt_output1)
print("actual_output", actual_output)
print("expected_otput", expected_otput)
assert torch.allclose(actual_output, expected_otput, atol=0, rtol=0)
This issue does not occur if the unnecessary code, which was added just to reproduce the problem, is removed from eltwise_sfpu.cpp in the example_multiple_return.
#include "hostdevcommon/kernel_structs.h" // code for reduce error
#include "compute_kernel_api.h"
#include "compute_kernel_api/eltwise_binary.h"
#include "debug/dprint.h"
uint32_t cb_srca = 256; // code for reduce error
uint32_t cb_srcb = 256; // code for reduce error
constexpr std::uint32_t onetile = 1; // code for reduce error
template <bool to_from_int8 = false> // code for reduce error
ALWI void test_func(const uint32_t srca_new_operand, const uint32_t srcb_new_operand) { // code for reduce error
DPRINT << "AAAAA 1\n"; // code for reduce error
} // code for reduce error
inline bool dummy_compare(std::uint32_t old_operand, std::uint32_t new_operand) { // code for reduce error
return (unpack_src_format[old_operand] != unpack_src_format[new_operand]) || // code for reduce error
(unpack_dst_format[old_operand] != unpack_dst_format[new_operand]); // code for reduce error
} // code for reduce error
namespace NAMESPACE {
void MAIN {
// Circular Buffers
constexpr auto cb_in = tt::CBIndex::c_0;
constexpr auto cb_other = tt::CBIndex::c_1;
constexpr auto cb_out = tt::CBIndex::c_2;
// Variables
constexpr uint32_t input_dst_reg = 0;
binary_op_init_common(cb_in, cb_other, cb_out);
tile_regs_acquire();
cb_wait_front(cb_in, onetile);
cb_wait_front(cb_other, onetile);
// code for reduce error
{
DPRINT << "DST_ACCUM_MODE " << static_cast<uint32_t>(DST_ACCUM_MODE) << "\n";
if (cb_srca == 256 || cb_srcb == 256) {
// do nothing
test_func<DST_ACCUM_MODE>(cb_in, cb_other);
} else {
// not executed
UNPACK(if (dummy_compare(cb_srca, cb_in)) { TTI_STALLWAIT(p_stall::STALL_CFG, p_stall::UNPACK0); })
}
cb_srca = cb_in; // code for reduce error
cb_srcb = cb_other; // code for reduce error
}
mul_tiles_init(cb_in, cb_other);
mul_tiles(cb_in, cb_other, 0, 0, 0);
cb_pop_front(cb_in, onetile);
cb_pop_front(cb_other, onetile);
tile_regs_commit();
tile_regs_wait();
cb_reserve_back(cb_out, onetile);
pack_tile(0, cb_out);
cb_push_back(cb_out, onetile);
tile_regs_release();
}
} // namespace NAMESPACE
Steps to reproduce the issue
- checkout https://github.com/tenstorrent/tt-metal/tree/hyungsuk/mul_random_bug
- build on debug mode
- Set Env (Important)
export TT_METAL_LOGGER_LEVEL=Debug
export TT_METAL_LOGGER_TYPES=Test
export TT_METAL_DPRINT_CORES="(0,0)"
- go to
tests/ttnn/unit_tests/operations - run
pytest ./test_examples.py - See error
Expected behavior
2 * 0 must be 0. always.
Please complete the following environment information
device: wormhole_b0
Updated reproducer instructions (July 4) - by Anil Mahmud
-
Checkout tt-metal branch origin/amahmud/mul_random_bug commit id https://github.com/tenstorrent/tt-metal/commit/c94f33d219567e5178ebb2173f10c54da32417f0
-
You do not need to set any debug build/DPRINT core/ LOG level
-
go to
tests/ttnn/unit_tests/operations -
run
pytest ./test_examples.py -
See the failure. The failure will happen probablistically after a random number of iterations (around 5-10, explained inside the test case). The test runs on a single core.
Note : The issue is not related to any dvalid signals (as confirmed by simulations)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 75 (66 by maintainers)
I checked with @rdjogoTT’s changes and the test passes as expected. The test case I used has a high rate of reproducing the error but after more than 1000 attempts it did not fail. I am making the issue Radomir is working on as the parent issue and when his PR is merged, I will close this issue.
EDIT : Will not close the issue, will wait for green signal from @hschoi4448
The issue lies in the race on the
ALU_ACC_CTRL_Zero_Flag_disabled_srcbit (datacopy needs this bit set while mul needs it cleared), in the failing iteration this bit fails to clear (gets cleared for only a couple of cycles and then restored) causing wrong FPU results. The cause of the race is that the register that stores that bit also stores some packer bits meaning that the same register is accessed from both the unpacker and the packer thread. Unpacker thread uses tensix RMW instructions to modify that bit (cunpack_common.h:294) while packer uses explicit risc mmio read modify write sequence outside of the appropriate mutex guards (cpack_common.h:554) if the unpack RMW happens between the pack read and write the unpack change will be lost. @rdjogoTT has come to the same conclusion while debugging some reduce issues, so the fix is available at rd/reduce_w_fp32. @amahmudTT will pick up those changes and test on current repros.@ihnpark551 , @amahmudTT will take a look from the LLK side to debug this behavior. He just started looking into this yesterday.
The LLK team (@nvelickovicTT ) will have a look at this shortly
I tested on main branch and it seems like problem fixed.
Thank you.
@amahmudTT
Thank you, I’ll wait for the parent issue to be resolved.
Yes, (but you need to fix the factory related bug where I saw 1 instead of i, to not get wrong results in differing scenarios)
@lpremovicTT I did both because there is another reg being programmed using tensix RMW so I put them together. It’s also how I see it done in cunpack_common.h. We should figure out why that is.
Was able to reproduce the issue in versim for a 2 tile problem, just that in the simulation one tile had zeroes and the other tile had nans, but in silicon one tile has zeroes and the other tile has -1s.
For a single tile problem i could not reproduce the issue in versim, but then again single tile one only fails on silicon if I have run a two tile version and failed beforehand. Next step is to analyze the versim files with somebody adept in looking at waveforms 😃
I am currently trying to remove the extra dummy code and replace them with variable number of nops and check if the error could be made deterministic.
@amahmudTT is starting to look at this from the LLK side.
Yeah I did capture a simulation with dprint enabled. We can wait on LLK team’s analysis to determine if another repro case is needed
I’m not fully following the conversation, so I’m not sure — but could you clarify why reproducing the issue without DPRINT is necessary? ~Was DPRINT the cause of the problem and has now been fixed, and you’re trying to confirm whether the issue still reproduces after that?~
I found request context
@abhullar-tt
Now that versim is functional for WH, we will try to simulate this test as a next step
I hope to be back soon ~(a week), and in the meantime AFAIK @bbradelTT was going to have a short look from the op side.
I found printing out the dest before and after multiles, in the erroneous cases the dest starts with zeroes but ends up with -1s. Just to make sure I am not getting lost in the printouts, If I print only before mum_tiles i never get -1s even though the result is -1 but when I print after mul_tiles i do get -1s in dest when result is -1s. So we can rule out the issue of zeroacc. The problem is coming from somewhere else. Althought the zeroacc should be used anyways (until and unless somewhere else somebody is doing it (the packer sometimes does it).
Next step would be to use perturbation in timing to reduce the test case and try to create a single core repro.
Thank you for debugging problem! But I found that there is an incorrect value in
cb_outusing DPRINT inUNPACK.So I updated test code for easier debugging.
What I changed
SETPRECISIONcb_outin compute kernelNow you need change TT_METAL_DPRINT for better check.
if you run test, you can see dprint like below.
@nvelickovicTT
@hschoi4448 I checked on my end with your example, and here is what I got:
noc_async_calls solves the issue. Which leads me to think it is a race of some sort, but not between compute and data movement, since I add the delay at the end of the writer kernel.tt_outputtensor from python and it also seems to be all 0’s.From this it seems to me that the issue is happening in
ttnn.to_torch()call. @hschoi4448 could you also check on your side as well whether TTNN tensor is fine. Before iit is converted to torch tensor.@cmaryanTT @ayerofieiev-tt can you please review and triage. TY