tt-metal: Reproducible permute/transpose failures on single core (possible race condition)

Describe the bug

As part of #19950 I found that some permute tests failed intermittently when working on the new support for int32.

However, we also found that even the existing bfloat16 implementation fails predictably, when switching to single core:

@sjameelTT found this:

I noticed when switching to single core, that some of the other tests in the suite have PCC drops.

tests/ttnn/unit_tests/operations/test_permute.py::test_permute_5d_blocked has several. It seems like with enough iterations of the LLK PCC ends up dropping.

Steps to reproduce the issue

  1. Switch to @sjameelTT’s branch with a two-line change that makes permute RM work on single core only: https://github.com/tenstorrent/tt-metal/compare/main...sjameel/transpose_single_core
  2. Optionally pull in latest tt-llk changes with the reset config context fix - but this doesn’t appear to affect anything
  3. Run:
pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_5d_blocked[dtype=DataType.BFLOAT16-memory_config=MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::DRAM,shard_spec=std::nullopt)-perm=(4, 0, 3, 2, 1)-shape=(1, 6, 256, 20, 50)]'

The above fails every time for me. You can also run:

pytest tests/ttnn/unit_tests/operations/test_permute.py::test_permute_5d_blocked

To see some other failures.

Expected behavior

No failures.

Please complete the following environment information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 26 (26 by maintainers)

Commits related to this issue

Most upvoted comments

@sjameelTT considering the issue in #21927 was deterministic and not a race, this probably isn’t related to it.

Reading back the output from DRAM once the logic is complete shows me that it’s actually correct. I think it’s a readback issue, similar to your other issue #22336. I can personally recreate it with shape = (1, 96*96, 32*(228)) (output_shape = shape = (1, 96*96, 32*(228)), which is actually smaller. I made a new branch sjameel/transpose_readback which can print the output after writing it out to DRAM, but before it’s readback on host. The last tile is wrong in our final output, which is on host, but correct in DRAM at the end of the kernel.

TT_METAL_DPRINT_ONE_FILE_PER_RISC=1 TT_METAL_DPRINT_CORES="0,0" pytest 'tests/ttnn/unit_tests/operations/test_permute.py::test_permute_jd'

That should print the final output on host (in python) and the DPRINTs for the last 32 tiles, read back from DRAM after we write out the values.

generated/dprint/device-0_worker-core-0-0_BRISC.txt should contain the following at the very end:

page i: 2101247 in output buffer
======
0 : 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
6 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
10 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
11 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
14 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
23 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
24 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
25 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
26 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
27 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
28 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
29 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
30 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
31 : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
++++++

It takes a while to generate so might take a bit. The 0s are the padded values and the actual 1s are the real data, which do not appear on host.

I’ll take a look today.

Here the tensor is much smaller that 4GB, unless I am missing something. 1*96*96*96*96*2bytes ~ 160MBs.

That’s true for the input tensor, but when you permute the first two dimensions, I think dimension 0 gets padded out to 32, so it ends up using 32 * 160MB ~= 5.12GB. I haven’t verified this in depth, but this is my sense from fiddling with shapes.