tt-metal: [$3000 Bounty] int32 permute produces incorrect data
The following test will fail:
def test_integer_permute(device):
torch_input = torch.randint(-10, 10, (1, 2, 1, 1))
torch_output = torch.permute(torch_input, [0, 2, 3, 1])
input_tensor = ttnn.from_torch(torch_input, layout=ttnn.TILE_LAYOUT, dtype=ttnn.int32, device=device)
output_tensor = ttnn.permute(input_tensor, [0, 2, 3, 1])
output_tensor = ttnn.to_torch(output_tensor)
assert torch_output.shape == output_tensor.shape
assert torch.all(torch_output == output_tensor)
If permute isn’t intended to be supported for int32 data yet, an informative error should be thrown stating so instead.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 42 (40 by maintainers)
Commits related to this issue
- Add workaround that prevents permute from operating on integers (#2774) ### Ticket tt-metal issue: https://github.com/tenstorrent/tt-metal/issues/19950 ### Problem description ttnn.permute is not su... — committed to tenstorrent/tt-mlir by LPanosTT 2 years ago
- Add support for int32 to transpose_wh_tile (fixes ttnn.{permute,transpose}). (#21270) ### Ticket Closes #19950 ### Problem description ttnn.permute and ttnn.transpose don't work for int32. The ro... — committed to tenstorrent/tt-metal by jasondavies a year ago
- Second attempt: add support for int32 to transpose_wh_tile. (#23415) ### Ticket Second attempt at closing #19950. First attempt was reverted due to failing BH tests. ### Problem description ttnn.p... — committed to tenstorrent/tt-metal by jasondavies a year ago
- Second attempt: add support for int32 to transpose_wh_tile. (#23415) ### Ticket Second attempt at closing #19950. First attempt was reverted due to failing BH tests. ### Problem description ttnn.p... — committed to tenstorrent/tt-metal by jasondavies a year ago
Congrats Jaon on completing another bounty. Keep an eye on your inbox while we reach out for the payment.
It feels like these checks should be pushed down to LLK somehow, and leave this API level oblivious to these details. I’ll take a look, think a bit and share my thoughts on the PR.
@jasondavies that’s great news. Thanks for pursuing this debug. Blackhole implementation uses 2 contexts, so no need for a fix like this. Wormhole implementation had some issues with the 2nd context for unpack to dest tilize for int32, so that is why it diverges from the standard approach of using 2 config contexts. We will revisit this for Wormhole at some point, but it is deprioritized at the moment.
That being said, it’s obvious LLK code architecture doesn’t gracefully handle mixed usage of 1 and 2 config contexts. It’s something to think about on our side.
@jasondavies I’ve pushed a PR that fixes this test (https://github.com/tenstorrent/tt-llk/pull/205). The issue was that unpack to dest with tilize uses just 1 config context, and it wasn’t explicitly setting that. Other kernels usually use 2 config contexts, so it happens that unpack to dest with tilize reads from config context that it didn’t set during init. This is only the case for int32 format.
This should explain why you were seeing problems only with row major case, because that needs to first tilize the input which for int32 used this implementation.
You can pull in the PR and give it a try, and I will merge it once all the test pass.
Since this is turning out to be more difficult than what we expected so we are increasing the reward amount for this one to $3000. All the best Jason!
OK, will take a look at the test tomorrow. Thanks for sharing.
And if the SrcB transpose route doesn’t work, I think another viable route is:
I think step 2 in the above would involve 64 single-cycle pack instructions per 16x16 face, which is a lot faster than the 1280 cycles needed to do the operation purely on the baby RISCs, but probably still slower than the SrcB transpose route.
For reference, this is that baby RISCs on Wormhole, when writing directly to L1 using RISC
swinstructions, have a maximum sustained write throughput of one 32b write every 5 cycles (whereasswinstructions to addresses outside of L1 have a much more reasonable maximum sustained write throughput of one 32b write every cycle). One potential workaround for write-heavy workloads, when running as a compute kernel (rather than a data movement kernel) is to use the GPRs of the Tensix Scalar (ThCon) unit: an aligned 128b write to L1 can be done in five cycles; four cycles ofswinstructions to write data to Tensix Scalar GPRs, followed by one cycle to issue a Tensix Scalar 128b write instruction (which’ll move from Tensix Scalar GPRs to L1). A bunch of really subtle synchronisation is required to make this work though (there are some non-subtle ways of doing the synchronisation, but they’ll tank performance), so I’d only recommend exploring this approach if the two 16b planes approach doesn’t work.I can do this. Can you explain how to approach this with NOC APIs as an alternative to RISC-V?