tt-metal: [$3000 Bounty] int32 permute produces incorrect data

The following test will fail:

def test_integer_permute(device):

    torch_input = torch.randint(-10, 10, (1, 2, 1, 1))
    torch_output = torch.permute(torch_input, [0, 2, 3, 1])

    input_tensor = ttnn.from_torch(torch_input, layout=ttnn.TILE_LAYOUT, dtype=ttnn.int32, device=device)
    output_tensor = ttnn.permute(input_tensor, [0, 2, 3, 1])
    output_tensor = ttnn.to_torch(output_tensor)

    assert torch_output.shape == output_tensor.shape
    assert torch.all(torch_output == output_tensor)

If permute isn’t intended to be supported for int32 data yet, an informative error should be thrown stating so instead.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 42 (40 by maintainers)

Commits related to this issue

Most upvoted comments

Congrats Jaon on completing another bounty. Keep an eye on your inbox while we reach out for the payment.

@nvelickovicTT Just rebased my tt-metal PR #21270 against latest main. This contains the important transpose_wh_tile change that checks for 32-bit inputs:

ALWI void transpose_wh_tile(uint32_t icb, uint32_t itile, uint32_t idst) { #if defined(TRISC_MATH) || defined(TRISC_UNPACK) const std::uint32_t src_format = get_operand_src_format(icb); const bool is_32bit = (src_format & 0xf) == (std::uint32_t)DataFormat::Int32; if (is_32bit) { copy_tile_init(icb); copy_tile(icb, itile, idst);

    transpose_wh_dest_init_short<true>();
    transpose_wh_dest<true>(idst);
} else {
    UNPACK((llk_unpack_A<BroadcastType::NONE, false>(icb, itile, false)));
    MATH((llk_math_eltwise_unary_datacopy<A2D, BroadcastType::NONE, DST_ACCUM_MODE>(idst)));
}

#endif } Though it does feel a bit like this could be simplified e.g. reference specific LLKs rather than calling copy_tile_init and copy_tile, it does solve the issue for all users of transpose_wh_tile.

Suggestions welcome for improvements on this front.

It feels like these checks should be pushed down to LLK somehow, and leave this API level oblivious to these details. I’ll take a look, think a bit and share my thoughts on the PR.

@jasondavies that’s great news. Thanks for pursuing this debug. Blackhole implementation uses 2 contexts, so no need for a fix like this. Wormhole implementation had some issues with the 2nd context for unpack to dest tilize for int32, so that is why it diverges from the standard approach of using 2 config contexts. We will revisit this for Wormhole at some point, but it is deprioritized at the moment.

That being said, it’s obvious LLK code architecture doesn’t gracefully handle mixed usage of 1 and 2 config contexts. It’s something to think about on our side.

@nvelickovicTT FYI this is my latest test for the above copy_tile modification (so it doesn’t actually do a transpose).

It does one test with bfloat16, and then another with int32, printing the non-matching elements and the first two dimensions. This shows that everything in ttnn_output[1] and ttnn_output[3] doesn’t match the expected output for int32.

import ttnn import torch

torch.manual_seed(0) device = ttnn.open_device(device_id=0)

def test_integer_permute(device, is_32bit): permutation = [0, 2, 1] if is_32bit: dtype = ttnn.int32 shape = (5, 32, 32) torch_input = torch.randint(0, 10, shape) else: dtype = ttnn.bfloat16 shape = (2 * 5, 32, 32) torch_input = torch.randint(-10, 0, shape).bfloat16()

torch_output = torch_input # torch.permute(torch_input, permutation)

input_tensor = ttnn.from_torch(torch_input, layout=ttnn.ROW_MAJOR_LAYOUT, dtype=dtype, device=device)
ttnn_output = ttnn.permute(input_tensor, permutation)
ttnn_output = ttnn.to_torch(ttnn_output)

if not is_32bit:
    ttnn_output = ttnn_output.bfloat16()

for i, a in enumerate(torch_output):
    for j, row in enumerate(a):
        if not torch.all(ttnn_output[i][j] == row):
            print(i, j, ttnn_output[i][j], row)

assert torch_output.shape == ttnn_output.shape
assert torch.all(torch_output == ttnn_output)

try: test_integer_permute(device, 0) test_integer_permute(device, 1) finally: ttnn.close_device(device)

@jasondavies I’ve pushed a PR that fixes this test (https://github.com/tenstorrent/tt-llk/pull/205). The issue was that unpack to dest with tilize uses just 1 config context, and it wasn’t explicitly setting that. Other kernels usually use 2 config contexts, so it happens that unpack to dest with tilize reads from config context that it didn’t set during init. This is only the case for int32 format.

This should explain why you were seeing problems only with row major case, because that needs to first tilize the input which for int32 used this implementation.

You can pull in the PR and give it a try, and I will merge it once all the test pass.

Since this is turning out to be more difficult than what we expected so we are increasing the reward amount for this one to $3000. All the best Jason!

OK, will take a look at the test tomorrow. Thanks for sharing.

And if the SrcB transpose route doesn’t work, I think another viable route is:

  1. Use unpackers to copy a tile of 32b data from L1 to Dst.
  2. Use some sneaky packer configuration to perform the transpose whilst copying the data from Dst back to L1.

I think step 2 in the above would involve 64 single-cycle pack instructions per 16x16 face, which is a lot faster than the 1280 cycles needed to do the operation purely on the baby RISCs, but probably still slower than the SrcB transpose route.

The issue with a baby RISC implementation is that the maximum write throughput is 32b per 5 cycles, which would mean 1280 cycles for each 16x16 face.

For reference, this is that baby RISCs on Wormhole, when writing directly to L1 using RISC sw instructions, have a maximum sustained write throughput of one 32b write every 5 cycles (whereas sw instructions to addresses outside of L1 have a much more reasonable maximum sustained write throughput of one 32b write every cycle). One potential workaround for write-heavy workloads, when running as a compute kernel (rather than a data movement kernel) is to use the GPRs of the Tensix Scalar (ThCon) unit: an aligned 128b write to L1 can be done in five cycles; four cycles of sw instructions to write data to Tensix Scalar GPRs, followed by one cycle to issue a Tensix Scalar 128b write instruction (which’ll move from Tensix Scalar GPRs to L1). A bunch of really subtle synchronisation is required to make this work though (there are some non-subtle ways of doing the synchronisation, but they’ll tank performance), so I’d only recommend exploring this approach if the two 16b planes approach doesn’t work.

I can do this. Can you explain how to approach this with NOC APIs as an alternative to RISC-V?