tt-metal: Failing upsample multi core tests on blackhole

When running pytest tests/ttnn/unit_tests/operations/pool/test_upsample.py::test_bilinear_multi_core 16 tests fail with PCC error.

When this is closed, please remove remove temporary skip from the test.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (18 by maintainers)

Commits related to this issue

Most upvoted comments

Commit https://github.com/tenstorrent/tt-metal/issues/18200#:~:text=…-,2c719aa,-Add%20a%20comment from branch nkpatel/bilinear-fix-BH on top of Anil’s change makes all test pass for BH.

There is one more issue I found is that while filling values for scalar in reader kernel only 4 scalars are filled, instead of repeated scalar for each row. I shall add that change.. As per https://tenstorrent.slack.com/archives/C03MBJVDAKG/p1747767781662009?thread_ts=1747763408.861799&cid=C03MBJVDAKG , in case of GAPOOL, there should be broadcast of each scalar along row.

Ok here is the thing.

On Wormhole, during the init phase, multiple arguments are passed and the mop for doing the reduction is configured using those arguments.

<bool neginf_srcA = true, bool zero_srcA_reduce = false>
ALWI void tilizeA_B_reduce_init(

But in Wormhole some of the tasks are not done using the mop and are done through explicit tensix instructions when the mop is called, but that would mean that the call to

ALWI void unpack_tilizeA_B_block(
    uint32_t icb0,
    uint32_t icb1,
    uint32_t block,
    uint32_t tile_idx_b,
    uint32_t num_faces = 4,
    uint32_t srca_face_r_dim = 16) {
    UNPACK((llk_unpack_tilizeA_B_block<true, true>(icb0, icb1, block, tile_idx_b, num_faces, srca_face_r_dim)));
}

Should have a method of passing those arguments which it should forward to llk_unpack_tilize_A_B_block, but as we can see no such provision is there and the hardcoded <true, true> is passed. But for our purposes, for the function

template <bool neginf_srcA = false, std::uint32_t reload_srcB = false, bool zero_srcA = false>
inline void llk_unpack_tilizeA_B_block(

The template arguments should be like this

neginf_srcA = false
reload_srcB  = true
zero_srcA = true

This is to match the arguments of tilizeA_B_reduce_init. Note inside tilizeA_B_reduce_init we call Moreover, there is no code to execute the corresponding task when zero_srcA is true in llk_unpack_tilizeA_B_block, that needs to be added too.

template <
    bool neginf_srcA = false,
    std::uint32_t reload_srcB = false,
    bool zero_srcA = false,
    bool zero_srcA_reduce = false>
inline void llk_unpack_tilizeA_B_init(

So there is some difference between zero_srcA and zero_srcA_reduce but for calling llk_unpack_tilizeA_B_block, there is no such difference and there is one such argument that is zero_srcA, do we want to rename it or do we want to add an extra unused argument that is something of a debate.

Now due to this what ends up happening is that neginf_srcA is set to true and that puts negative infiniteis in srcA register. Given the reduction in effect does a matmul using the GAPOOL instruction which can be described as

This instruction is almost identical to MVMUL except that it lacks the BroadcastSrcBRow option (BroadcastSrcBRow is always taken to be false), and Dst / SrcB are both 4x16 matrices rather than 8x16 matrices. The alignment restriction on SrcB is the same as for MVMUL though, so the 4x16 SrcB matrix is the top half of an aligned 8x16 matrix. The alignment restriction on Dst is however slighty laxer; it only needs to be an aligned 4x16 matrix. There are a few cases where this instruction can do something that MVMUL can't, but outside of those, software is encouraged to use MVMUL instead, as MVMUL achieves 2x the operations per second of GAPOOL.

Now if there is infinities in srcA and non-zeroes in srcB they would get into the dest register. So I think the assumption was that srcA should be zeroed out and hence the call to tilizeA_B_reduce_init was called with <neginf_srcA=false, zero_srcA_reduce=true>. @nkpatel-tt might add some detail regarding this.

The last commit https://github.com/tenstorrent/tt-metal/commit/ae3ad5e9981bf1d17210f52706bf149908ded769 of branch https://github.com/tenstorrent/tt-metal/tree/amahmud/upsample_main fixes the issue for the test case above.

This is for BH-WH functional parity - not blocking anything critical yet.