tt-metal: Insufficient reset of state/config/context in LLK after transpose_wh_dest

Describe the bug

Something is happening that breaks normal usage of ttnn after running int32 permute tests.

I’ve narrowed it down to running int32 permute an odd number of times, which indicates there’s some kind of context or config state that doesn’t get reset properly afterwards, or there’s a missing reset that needs to occur at the beginning of non-int32 normal usage.

Steps to reproduce the issue

  1. Switch to https://github.com/jasondavies/tt-metal/tree/permute-int32-bfloat16-failure-leak
  2. Run pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_4d_other_permutations[dtype=DataType.INT32-perm=[0, 3, 1, 2]-shape=[1, 1, 32, 32]]' and observe failure
  3. Run pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_4d_other_permutations[dtype=DataType.INT32-perm=[0, 3, 1, 2]-shape=[1, 1, 32, 32]]' again and observe success.
  4. Go to 2.

Expected behavior

Bfloat16 should work normally and not be affected by using int32 beforehand by a completely separate process.

Please complete the following environment information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 27 (26 by maintainers)

Commits related to this issue

Most upvoted comments

@pmilenkovicTT Sounds good. Will try these changes and report back.

Now the issue is that making these changes causes a hang when transpose_dest is used on its own, i.e. in the new DeviceFixture.TensixComputeTransposeWHDest that I added (i.e. the tt-metal API transpose_wh_dest). I think the overall issue here is:

  • if transpose_dest is used alone in MATH via transpose_wh_dest, then you do want SrcA/SrcB flipped via llk_unpack_set_srcb_dummy_valid and then in MATH via SETRWC.
  • if transpose_dest is used in combination with unpack_A, e.g. in the new transpose_wh_tile then we only want SrcB flipped.

Hey @jasondavies - we’re onto something! It seems that the test is continually passing when I make two small modifications:

  1. Modify the line TTI_SETRWC(p_setrwc::CLR_AB, 0, 0, 0, 0, p_setrwc::SET_ABD); to only clear Source B (p_setrwc::CLR_B)
  2. Remove TTI_UNPACR_NOP(SrcA, p_unpacr_nop::UNP_SET_DVALID); line from _llk_unpack_set_srcb_dummy_valid_

Can you try it out locally and run the tests that you intended to perform when you encountered the issue? We can go from there. Note: I am also running the test_permute_4d_other_permutations with BFLOAT16 and it passes every time with these modifications, while it used to depend on the INT32 test prior to these changes. Bonus points: I also ran the whole test_permute.py on your branch twice, and it seems like these modifications fix more stuff than I expected.

Quick update: The behavior is the same even if we change the path for bfloat16 inputs (what Lazar suggested) - will keep looking into it.

Hey @jasondavies, we’ve looked into this issue a bit. Current findings (including yours) are that, without running any int32 tests, everything behaves regularly. After running the permute test for int32 dtype, if the test passes, bfloat16 test behaves regularly. However, if the Int32 tests leave the board in a “weird” state, bfloat16 fails every time. When hitting the int32 data path, we’ve discovered that commenting out the line: TTI_UNPACR_NOP(SrcA, p_unpacr_nop::UNP_SET_DVALID); in the _llk_unpack_set_srcb_dummy_valid_ function does not change the behavior, while doing the same thing for SrcB causes a hang, leading us to believe that there is an instruction that sets DVALID bit for the SrcA one more time than expected. I’ll keep looking into this, and if the solution is not obvious, we’ll perform a simulation to figure out what is going on.

@vmilicevicTT who is the best person to assign this item to?