tt-metal: Insufficient reset of state/config/context in LLK after transpose_wh_dest
Describe the bug
Something is happening that breaks normal usage of ttnn after running int32 permute tests.
I’ve narrowed it down to running int32 permute an odd number of times, which indicates there’s some kind of context or config state that doesn’t get reset properly afterwards, or there’s a missing reset that needs to occur at the beginning of non-int32 normal usage.
Steps to reproduce the issue
- Switch to https://github.com/jasondavies/tt-metal/tree/permute-int32-bfloat16-failure-leak
- Run
pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_4d_other_permutations[dtype=DataType.INT32-perm=[0, 3, 1, 2]-shape=[1, 1, 32, 32]]'and observe failure - Run
pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_4d_other_permutations[dtype=DataType.INT32-perm=[0, 3, 1, 2]-shape=[1, 1, 32, 32]]'again and observe success. - Go to 2.
Expected behavior
Bfloat16 should work normally and not be affected by using int32 beforehand by a completely separate process.
Please complete the following environment information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 27 (26 by maintainers)
@pmilenkovicTT Sounds good. Will try these changes and report back.
Now the issue is that making these changes causes a hang when transpose_dest is used on its own, i.e. in the new
DeviceFixture.TensixComputeTransposeWHDestthat I added (i.e. the tt-metal APItranspose_wh_dest). I think the overall issue here is:transpose_wh_dest, then you do want SrcA/SrcB flipped viallk_unpack_set_srcb_dummy_validand then in MATH viaSETRWC.transpose_wh_tilethen we only want SrcB flipped.Hey @jasondavies - we’re onto something! It seems that the test is continually passing when I make two small modifications:
TTI_SETRWC(p_setrwc::CLR_AB, 0, 0, 0, 0, p_setrwc::SET_ABD);to only clear Source B (p_setrwc::CLR_B)TTI_UNPACR_NOP(SrcA, p_unpacr_nop::UNP_SET_DVALID);line from_llk_unpack_set_srcb_dummy_valid_Can you try it out locally and run the tests that you intended to perform when you encountered the issue? We can go from there. Note: I am also running the
test_permute_4d_other_permutationswith BFLOAT16 and it passes every time with these modifications, while it used to depend on the INT32 test prior to these changes. Bonus points: I also ran the wholetest_permute.pyon your branch twice, and it seems like these modifications fix more stuff than I expected.Quick update: The behavior is the same even if we change the path for bfloat16 inputs (what Lazar suggested) - will keep looking into it.
Hey @jasondavies, we’ve looked into this issue a bit. Current findings (including yours) are that, without running any int32 tests, everything behaves regularly. After running the permute test for int32 dtype, if the test passes, bfloat16 test behaves regularly. However, if the Int32 tests leave the board in a “weird” state, bfloat16 fails every time. When hitting the int32 data path, we’ve discovered that commenting out the line:
TTI_UNPACR_NOP(SrcA, p_unpacr_nop::UNP_SET_DVALID);in the_llk_unpack_set_srcb_dummy_valid_function does not change the behavior, while doing the same thing for SrcB causes a hang, leading us to believe that there is an instruction that sets DVALID bit for the SrcA one more time than expected. I’ll keep looking into this, and if the solution is not obvious, we’ll perform a simulation to figure out what is going on.@vmilicevicTT who is the best person to assign this item to?