tt-metal: Reinitialization of Tilize/Untilize Operations Fails When Changing # Out Tiles
Describe the bug When processing tensors with C dimensions that are larger than 8 but not multiples of 8 tiles, the resulting output is wrong. This seems to be due to incomplete re-initialization of the tilize/untilize operations in the compute kernel since for instance if we have 12 tiles, we need to first process 8 tiles, then 4 tiles.
To Reproduce
- checkout
wransom/TwelveTileBugRebase - build the code
./build_metal.sh --debug - run the max pool unit test
pytest tests/ttnn/unit_tests/operations/test_maxpool2d.pywhich runs a simple test case on 1 core - the test should fail, but should also print register output from right after the reduce operation (see line 80 of
max_pool_multi_core.cpp)
From here we need to set the stage a bit to understand the current debugging information. Because we have 12 C tiles, reduce_h_fused in max_pool_multi_core.cpp will be called twice for each window position. The first time it should process the first 8 tiles, the second time it should process the remaining 4 tiles. (both C = 4 tiles and C = 16 tile cases work, but C = 12 does not). Furthermore, we have HW=2x2, but a window size of 3x3. Thus, our tensor is padded such that H_pad, W_pad = 4. So, given our stride of 1x1 in a padded HW of 4x4 and a window size of 3x3, this results in 4 window positions. So, we will see a total of 8 tiles printed to the console from the reduce output registers, 2 for each window position times 4 window positions. In the branches default state it is set to print register data from tile 0 so, for the 12 tile test case we are working with, we will see all 8 printed tiles with a top row equal to “4” (the correct value). If we change line 80 of max_pool_multi_core.cpp to print tile 4 instead, we will see every other tile printed with 0 as the top row. This is what we should expect since we only have 12 tiles and are printing tile index 4, which on the second pass of reduce_h_fused exceeds the total of 12 tiles.
So all this seems to imply that the output of reduce is correct, and that the issue may lie in the untilize procedure.
Note in the branches default state, we are not reinitializing the tilize operation (we are only reinitializing the utilize operation (see lines 138, 144, 150, 156 of max_pool_multi_core.cpp. Also note that the commented out calls to tilizeA_B_reduce_init_short on lines 138 and 150 reference a newly implemented, untested function which may not behave as intended. Nonetheless it is expected that the tilize operation will need to be re-initialized as well, but the results are the same at least for this simple case whether these calls to tilize are used or not.
Expected behavior With the updates in the referenced branch, a tensor with a 12 tile C dimensions should work.
Please complete the following environment information:
- This issue was encountered on a wormhole system
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (18 by maintainers)
@amahmudTT I have rebased my branch off main, so now instead of using
wransom/TwelveTileBug, please usewransom/TwelveTileBugRebase. A head’s up though, Abhinav tells me there is an LLK issue on the current main for blackhole. So it is likely the rebased version still will not run on blackhole. Let me know if you need to get access to a wormhole machine to work on this. It’s a very high priority for my team so maybe we can help get you a WH reservation. Thanks again, and let me know if you need anything!@amahmudTT So I realized I kind of botched the explanation of why we will see 8 tiles outputted to the console. This comes down to some confusing logic about the stride, padding and such. I have re-worded this explanation in the original description. But I don’t think this was actually the source of your confusion.
So, what I meant when I say this:
“If we change line 80 of max_pool_multi_core.cpp to print tile 4 instead, we will see every other tile printed with 0 as the top row. This is what we should expect since we only have 12 tiles and are printing tile index 4, which on the second pass of reduce_h_fused exceeds the total of 12 tiles.”
Is that if we are printing tile index 0, then for a given window position, we should see both tiles come out of reduce with the value populated, because we are printing tiles 0 and 8 out of 12. However, if we print tile index 4 instead, then we are printing tiles 4 and 12 out of 12. Since index 12 exceeds the max valid index of 11, the second tile which is printed should not have values populated, we should just see padding there. The same would be true if we printed index 5. In this case tiles 5 and 13 would be printed, and again since index 13 exceeds the max index of 11, we again would see valid values in the first tile, but just padding in the second.
Let me know if a call would be better to discuss this, it’s pretty confusing, and thanks for taking this on!