tt-metal: Reproducible permute/transpose failures on single core (possible race condition)
Describe the bug
As part of #19950 I found that some permute tests failed intermittently when working on the new support for int32.
However, we also found that even the existing bfloat16 implementation fails predictably, when switching to single core:
@sjameelTT found this:
I noticed when switching to single core, that some of the other tests in the suite have PCC drops.
tests/ttnn/unit_tests/operations/test_permute.py::test_permute_5d_blockedhas several. It seems like with enough iterations of the LLK PCC ends up dropping.
Steps to reproduce the issue
- Switch to @sjameelTT’s branch with a two-line change that makes permute RM work on single core only: https://github.com/tenstorrent/tt-metal/compare/main...sjameel/transpose_single_core
- Optionally pull in latest tt-llk changes with the reset config context fix - but this doesn’t appear to affect anything
- Run:
pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_5d_blocked[dtype=DataType.BFLOAT16-memory_config=MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt)-perm=(4, 0, 3, 2, 1)-shape=(1, 6, 256, 20, 50)]'
The above fails every time for me. You can also run:
pytest tests/ttnn/unit_tests/operations/test_permute.py::test_permute_5d_blocked
To see some other failures.
Expected behavior
No failures.
Please complete the following environment information
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 26 (26 by maintainers)
Commits related to this issue
- Add support for int32 to transpose_wh_tile (fixes ttnn.{permute,transpose}). (#21270) ### Ticket Closes #19950 ### Problem description ttnn.permute and ttnn.transpose don't work for int32. The ro... — committed to tenstorrent/tt-metal by jasondavies a year ago
- Second attempt: add support for int32 to transpose_wh_tile. (#23415) ### Ticket Second attempt at closing #19950. First attempt was reverted due to failing BH tests. ### Problem description ttnn.p... — committed to tenstorrent/tt-metal by jasondavies a year ago
- Second attempt: add support for int32 to transpose_wh_tile. (#23415) ### Ticket Second attempt at closing #19950. First attempt was reverted due to failing BH tests. ### Problem description ttnn.p... — committed to tenstorrent/tt-metal by jasondavies a year ago
@sjameelTT considering the issue in #21927 was deterministic and not a race, this probably isn’t related to it.
Reading back the output from DRAM once the logic is complete shows me that it’s actually correct. I think it’s a readback issue, similar to your other issue #22336. I can personally recreate it with
shape = (1, 96*96, 32*(228))(output_shape = shape = (1, 96*96, 32*(228)), which is actually smaller. I made a new branchsjameel/transpose_readbackwhich can print the output after writing it out to DRAM, but before it’s readback on host. The last tile is wrong in our final output, which is on host, but correct in DRAM at the end of the kernel.TT_METAL_DPRINT_ONE_FILE_PER_RISC=1 TT_METAL_DPRINT_CORES="0,0" pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_jd'That should print the final output on host (in python) and the DPRINTs for the last 32 tiles, read back from DRAM after we write out the values.
generated/dprint/device-0_worker-core-0-0_BRISC.txt should contain the following at the very end:
It takes a while to generate so might take a bit. The 0s are the padded values and the actual 1s are the real data, which do not appear on host.
I’ll take a look today.
That’s true for the input tensor, but when you permute the first two dimensions, I think dimension 0 gets padded out to 32, so it ends up using
32 * 160MB ~= 5.12GB. I haven’t verified this in depth, but this is my sense from fiddling with shapes.