tt-metal: Failing upsample multi core tests on blackhole
When running pytest tests/ttnn/unit_tests/operations/pool/test_upsample.py::test_bilinear_multi_core 16 tests fail with PCC error.
When this is closed, please remove remove temporary skip from the test.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 21 (18 by maintainers)
Commits related to this issue
- #18200: Fix for bug, by adding arguments to pass zero_srcA and corresponding code to zero out srcA register — committed to tenstorrent/tt-metal by amahmudTT a year ago
- Fix window size for bilinear upsample and enable test cases for blackhole. Fixes #18200 — committed to tenstorrent/tt-metal by nkpatel-tt a year ago
- #18200:Add and pass correct template arguments to unpack_tilizeA_B_block and fix number of input rows to tilize reduction — committed to tenstorrent/tt-metal by amahmudTT a year ago
- #18200:Add and pass correct template arguments to unpack_tilizeA_B_block and fix number of input rows to tilize reduction — committed to tenstorrent/tt-metal by amahmudTT a year ago
- #18200:Add and pass correct template arguments to unpack_tilizeA_B_block and fix number of input rows to tilize reduction — committed to tenstorrent/tt-metal by amahmudTT a year ago
- #18200:Add and pass correct template arguments to unpack_tilizeA_B_block and fix number of input rows to tilize reduction — committed to tenstorrent/tt-metal by amahmudTT a year ago
- #18200:Add and pass correct template arguments to unpack_tilizeA_B_block and fix number of input rows to tilize reduction — committed to tenstorrent/tt-metal by amahmudTT a year ago
- #18200:Add and pass correct template arguments to unpack_tilizeA_B_block and fix number of input rows to tilize reduction — committed to tenstorrent/tt-metal by amahmudTT a year ago
- #18200:Add missing template arguments to unpack_tilizeA_B_block and correct number of input rows to tilize reduce (#22642) ### Ticket https://github.com/tenstorrent/tt-metal/issues/18200 ### Problem... — committed to tenstorrent/tt-metal by amahmudTT a year ago
- #18200:Add missing template arguments to unpack_tilizeA_B_block and correct number of input rows to tilize reduce (#22642) ### Ticket https://github.com/tenstorrent/tt-metal/issues/18200 ### Problem... — committed to tenstorrent/tt-metal by amahmudTT a year ago
Commit https://github.com/tenstorrent/tt-metal/issues/18200#:~:text=…-,2c719aa,-Add%20a%20comment from branch nkpatel/bilinear-fix-BH on top of Anil’s change makes all test pass for BH.
There is one more issue I found is that while filling values for scalar in reader kernel only 4 scalars are filled, instead of repeated scalar for each row. I shall add that change.. As per https://tenstorrent.slack.com/archives/C03MBJVDAKG/p1747767781662009?thread_ts=1747763408.861799&cid=C03MBJVDAKG , in case of GAPOOL, there should be broadcast of each scalar along row.Ok here is the thing.
On Wormhole, during the init phase, multiple arguments are passed and the mop for doing the reduction is configured using those arguments.
But in Wormhole some of the tasks are not done using the mop and are done through explicit tensix instructions when the mop is called, but that would mean that the call to
Should have a method of passing those arguments which it should forward to
llk_unpack_tilize_A_B_block, but as we can see no such provision is there and the hardcoded <true, true> is passed. But for our purposes, for the functionThe template arguments should be like this
This is to match the arguments of tilizeA_B_reduce_init. Note inside
tilizeA_B_reduce_initwe call Moreover, there is no code to execute the corresponding task when zero_srcA is true inllk_unpack_tilizeA_B_block, that needs to be added too.So there is some difference between
zero_srcAandzero_srcA_reducebut for calling llk_unpack_tilizeA_B_block, there is no such difference and there is one such argument that iszero_srcA, do we want to rename it or do we want to add an extra unused argument that is something of a debate.Now due to this what ends up happening is that neginf_srcA is set to true and that puts negative infiniteis in srcA register. Given the reduction in effect does a matmul using the GAPOOL instruction which can be described as
Now if there is infinities in srcA and non-zeroes in srcB they would get into the dest register. So I think the assumption was that srcA should be zeroed out and hence the call to
tilizeA_B_reduce_initwas called with<neginf_srcA=false, zero_srcA_reduce=true>. @nkpatel-tt might add some detail regarding this.The last commit https://github.com/tenstorrent/tt-metal/commit/ae3ad5e9981bf1d17210f52706bf149908ded769 of branch https://github.com/tenstorrent/tt-metal/tree/amahmud/upsample_main fixes the issue for the test case above.
This is for BH-WH functional parity - not blocking anything critical yet.