tt-metal: [Bug Report] Low PCC of `ttnn.global_avg_pool2d` after running `ttnn.max_pool2d`
Describe the bug
The PCC of ttnn.global_avg_pool2d becomes very low (0.7) if we run ttnn.max_pool2d first for some shapes
To Reproduce
Run the example below:
import torch
import ttnn
from tests.ttnn.utils_for_testing import assert_with_pcc
def test_maxpool(device, input_shape, kernel_size, stride, padding, dilation):
torch_input = torch.rand(input_shape, dtype=torch.bfloat16)
batch_size, in_c, in_h, in_w = input_shape
input_tensor = torch.permute(torch_input, (0, 2, 3, 1))
input_tensor = torch.reshape(input_tensor, (1, 1, -1, in_c))
input_tensor = ttnn.from_torch(input_tensor, layout=ttnn.ROW_MAJOR_LAYOUT, device=device)
output_tensor = ttnn.max_pool2d(
input_tensor,
batch_size,
in_h,
in_w,
in_c,
kernel_size,
stride,
padding,
dilation,
)
expected_output = torch.nn.functional.max_pool2d(torch_input, kernel_size, stride, padding)
output_tensor = ttnn.to_torch(output_tensor)
_, out_c, out_h, out_w = expected_output.shape
output_tensor = torch.reshape(output_tensor, (batch_size, out_h, out_w, out_c))
output_tensor = torch.permute(output_tensor, (0, 3, 1, 2))
assert torch.allclose(output_tensor, expected_output), "mismatch"
def test_global_avg_pool(device, input_shape):
torch_input_tensor = torch.randn(input_shape, dtype=torch.bfloat16)
torch_output_tensor = torch.nn.functional.adaptive_avg_pool2d(torch_input_tensor, (1, 1))
input_tensor = torch.permute(torch_input_tensor, (0, 2, 3, 1))
input_tensor = ttnn.from_torch(input_tensor, layout=ttnn.TILE_LAYOUT, device=device)
output_tensor = ttnn.global_avg_pool2d(input_tensor)
output_tensor = ttnn.to_torch(output_tensor)
output_tensor = torch.permute(output_tensor, (0, 3, 1, 2))
assert_with_pcc(torch_output_tensor, output_tensor)
if __name__ == "__main__":
try:
device = ttnn.open_device(device_id=0, l1_small_size=4096)
test_maxpool(device, (1, 192, 56, 56), (2, 2), (2, 2), (0, 0), (1, 1))
test_global_avg_pool(device, (1, 64, 1, 32))
finally:
ttnn.close_device(device)
Expected behavior
The PCC check of ttnn.global_avg_pool2d should pass (> 0.9999)
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software 612744d7e9d8810f20c1458710c7f5eab361bb7e
- Card: N300
Additional context
The PCC issue of ttnn.global_avg_pool2d seems to be persistent once ttnn.max_pool2d is run, unless the card is reset
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 36 (35 by maintainers)
Commits related to this issue
- #15824: Add dprints to reduce_h kernel — committed to tenstorrent/tt-metal by ncvetkovicTT 2 years ago
- #15824: Add dprints to reduce_h kernel — committed to tenstorrent/tt-metal by ncvetkovicTT 2 years ago
- #15824 Workaround LLK issue in max_pool pack_untilize_dst is causing issues on GS and WH in case where reduce op is called after max_pool. Workaround is to use default value for block_ct_dim in pack... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #15824 Workaround LLK issue in max_pool pack_untilize_dst is causing issues on GS and WH in case where reduce op is called after max_pool. Workaround is to use default value for block_ct_dim in pack... — committed to tenstorrent/tt-metal by deleted user 2 years ago
- #15824: Added a comment describing the workaround — committed to tenstorrent/tt-metal by ncvetkovicTT 2 years ago
- #15824 Workaround LLK issue in max_pool (#16849) pack_untilize_dst is causing issues on GS and WH in case where reduce op is called after max_pool. Workaround is to use default value for block_ct... — committed to tenstorrent/tt-metal by pavlejosipovic 2 years ago
- #15824 Workaround LLK issue in max_pool (#16849) pack_untilize_dst is causing issues on GS and WH in case where reduce op is called after max_pool. Workaround is to use default value for block_ct... — committed to tenstorrent/tt-metal by pavlejosipovic 2 years ago
Yes that is correct. Let me elaborate
for (uint32_t i = 0; i < nsticks_per_core; ++i)) at line 114 in compute kernel of max pool, one output stick is calculated. One output stick is1 X input_channels.unpack_tilizeA_B_blockat line no. 61 in compute kernel contains kernel_h X kernel_w rows of input. The output expected is 1 row.6*32 = 192) and window_size_hw(kernel_h*kernel_w) are relevant for config changeI did several experiments with example in this thread and by borys here and for all of the following configurations the result is same, every 4th iteration the PCC is good.
My set-up
export WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yamlbefore doing experiment),@mywoodstock , @ncvetkovicTT please make sure to post the new ticket here
@ncvetkovicTT Yes, we suspect there may be same issue with the other kernel, but don’t have a repro yet. SO yes lets close this issue and open a new one to track the proper LLK fix. Thanks!
I ran eltwise ADD C++ test on Vincent’s branch, it works on silicon but doesn’t on simulator, breaks with the following error:
Any ideas @vtangTT how I can continue? Ideally I would like to make the simulator work with my branch: ncvetkovic/15824-max_pool_and_reduce_fail
Also, I
ran test_pool.pyon that branch as well, butmax_poolhad some other usage back then so it yields an error, andmeanfails with 0.995 PCC, so I didn’t even run it on silicon.FYI @nvelickovicTT
Ah okay that’s what I initially thought the arguments to be, thanks. I will try simulating in the morning. This case fails always btw, not on every Nth attempt, which is good for simulating.
Raising to P0 to ensure this is resolved and not delayed further. Thank you!
@ncvetkovicTT maxpool is @mywoodstock 's area of expertise. I don’t know the details.
@ncvetkovicTT in regards to what this is related to, I think it’s related to
reduce_init/reduce_tileFor the basic reduce unit tests
ttnn/cpp/ttnn/operations/reduction/generic/device/kernels/compute/reduce_w.cppusesmm_init/matmul_tiles. When that happens, when run after max_pool, the tests pass. However, when I commented out that option and forcedreduce_init/reduce_tileto be used, I saw the same behaviour.That is:
You should be able to test via the code in https://github.com/tenstorrent/tt-metal/issues/6991#issuecomment-2559885872 by changing
to
In terms of what else uses reduce_tile, running
grep 'reduce_tile(' -r *in tt-metal shows other ops use reduce_tile as well. E.g. layernorm, groupnorm, and softmax. There are also a lot of tests:Trying this other stuff out might be a big time sink though.
One bug reported by while enabling max pool. It seems this is also in same line. https://github.com/tenstorrent/tt-metal/issues/15481
Narrowed it down further:
Commented out
tt_metal/include/compute_kernel_api/pack_untilize.h
and reduce/test_global_avg_pool worked.