tt-metal: DRAM-sharded matmuls do not implement padding
Describe the bug 1d and 2d matmuls support padding in their outputs e.g. N does not need to exactly divide into tiles and cores. dram-sharded matmuls have some of the code for padding support but do not implement it, leading to downstream problems like L1 memory corruption.
We can see this in matmul_op_multi_core_reuse_mcast_dram_sharded_program_factory.cpp - it defines padding variables such as last_block_padded_block_tiles_w_skip but then never uses them, whereas matmul_op_multi_core_reuse_mcast_2d_program_factory.cpp and matmul_op_multi_core_reuse_mcast_1d_program_factory.cpp actually do.
To Reproduce This is probably the cause of L1 corruption in 14902, see there for the reproducer. Simple inspection of the code linked above shows that the last block padding is neither implemented nor an exact match asserted. Presumably the entire last block is written over in L1, sometimes writing garbage over address space used for other tensors or CBs.
Expected behavior
As with other matmaul ops, the dram-sharded matmul should allow per-core-N to be the next largest multiple e.g. as the program config generation code in matmul_op.cc expects.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (15 by maintainers)
Commits related to this issue
- #15004: minimal test case repro 1 device for dram sharded matmul bug — committed to tenstorrent/tt-metal by caixunshiren 2 years ago
- Fix s2i op when shard grid is larger than actual used grid (#15113) ### Ticket https://github.com/tenstorrent/tt-metal/issues/15004#issuecomment-2476975904 https://github.com/tenstorrent/tt-metal... — committed to tenstorrent/tt-metal by yugaoTT 2 years ago
It turns out it’s the s2i op causing L1 errors, fix is here https://github.com/tenstorrent/tt-metal/pull/15113
@caixunshiren I just ran some changes to get rid of the sharded to interleaved op after linear. I tried directly call to_torch for tt_input, test pass. I tried call reshard for tt_input, and do the add op, test pass. It seems the interaction wiht sharded to interleaved is causing problems. investigating further