tt-metal: [Bug Report] Low PCC of `ttnn.global_avg_pool2d` after running `ttnn.max_pool2d`

Describe the bug The PCC of ttnn.global_avg_pool2d becomes very low (0.7) if we run ttnn.max_pool2d first for some shapes

To Reproduce

Run the example below:

import torch
import ttnn
from tests.ttnn.utils_for_testing import assert_with_pcc


def test_maxpool(device, input_shape, kernel_size, stride, padding, dilation):
    torch_input = torch.rand(input_shape, dtype=torch.bfloat16)
    batch_size, in_c, in_h, in_w = input_shape

    input_tensor = torch.permute(torch_input, (0, 2, 3, 1))
    input_tensor = torch.reshape(input_tensor, (1, 1, -1, in_c))
    input_tensor = ttnn.from_torch(input_tensor, layout=ttnn.ROW_MAJOR_LAYOUT, device=device)
    output_tensor = ttnn.max_pool2d(
        input_tensor,
        batch_size,
        in_h,
        in_w,
        in_c,
        kernel_size,
        stride,
        padding,
        dilation,
    )

    expected_output = torch.nn.functional.max_pool2d(torch_input, kernel_size, stride, padding)

    output_tensor = ttnn.to_torch(output_tensor)
    _, out_c, out_h, out_w = expected_output.shape
    output_tensor = torch.reshape(output_tensor, (batch_size, out_h, out_w, out_c))
    output_tensor = torch.permute(output_tensor, (0, 3, 1, 2))

    assert torch.allclose(output_tensor, expected_output), "mismatch"


def test_global_avg_pool(device, input_shape):
    torch_input_tensor = torch.randn(input_shape, dtype=torch.bfloat16)
    torch_output_tensor = torch.nn.functional.adaptive_avg_pool2d(torch_input_tensor, (1, 1))

    input_tensor = torch.permute(torch_input_tensor, (0, 2, 3, 1))
    input_tensor = ttnn.from_torch(input_tensor, layout=ttnn.TILE_LAYOUT, device=device)
    output_tensor = ttnn.global_avg_pool2d(input_tensor)
    output_tensor = ttnn.to_torch(output_tensor)
    output_tensor = torch.permute(output_tensor, (0, 3, 1, 2))

    assert_with_pcc(torch_output_tensor, output_tensor)


if __name__ == "__main__":
    try:
        device = ttnn.open_device(device_id=0, l1_small_size=4096)
        test_maxpool(device, (1, 192, 56, 56), (2, 2), (2, 2), (0, 0), (1, 1))
        test_global_avg_pool(device, (1, 64, 1, 32))
    finally:
        ttnn.close_device(device)

Expected behavior The PCC check of ttnn.global_avg_pool2d should pass (> 0.9999)

Please complete the following environment information:

  • OS: Ubuntu 20.04
  • Version of software 612744d7e9d8810f20c1458710c7f5eab361bb7e
  • Card: N300

Additional context The PCC issue of ttnn.global_avg_pool2d seems to be persistent once ttnn.max_pool2d is run, unless the card is reset

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 36 (35 by maintainers)

Commits related to this issue

Most upvoted comments

@nkpatel-tt Thank you for trying to explain, but I am not sure I fully follow. If I put input_shape to be (1, 192, 56, 56), in compute kernel I get:

in_ntiles_hw = 1 in_ntiles_c = 6 window_size_hw = 4 out_h = 28 out_w = 28 nsticks_per_core = 13

but for (1, 192, 64, 64) I get:

in_ntiles_hw = 1 in_ntiles_c = 6 window_size_hw = 4 out_h = 32 out_w = 32 nsticks_per_core = 16

I can’t see the difference in parameters between the two shapes in LLKs, that would help me understand what gets configured poorly.

Yes that is correct. Let me elaborate

  • In each iteration of for loop (for (uint32_t i = 0; i < nsticks_per_core; ++i)) at line 114 in compute kernel of max pool, one output stick is calculated. One output stick is 1 X input_channels.
  • The buffer passed to unpack_tilizeA_B_block at line no. 61 in compute kernel contains kernel_h X kernel_w rows of input. The output expected is 1 row.
  • So each core is given several outputs to find max_pool of and reduce operation is called on them one after another.
  • The configurations can only change in case there is change in input channels, kernel size. In your prints above, in_ntiles_c(6*32 = 192) and window_size_hw(kernel_h*kernel_w) are relevant for config change

I did several experiments with example in this thread and by borys here and for all of the following configurations the result is same, every 4th iteration the PCC is good.

  • test_maxpool(device, (1, 192, 56, 56), (2, 2), (2, 2), (0, 0), (1, 1)
  • test_maxpool(device, (1, 192, 64, 64), (2, 2), (2, 2), (0, 0), (1, 1))
  • test_maxpool(device, (1, 192, 32, 32), (2, 2), (2, 2), (0, 0), (1, 1))
  • test_maxpool(device, (1, 64, 32, 32), (2, 2), (2, 2), (0, 0), (1, 1))

My set-up

  • latest main(2a86ff7fdde8561ec6c60f1635937ac2b13e122f)
  • Board N300 (ran export WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml before doing experiment),

@mywoodstock , @ncvetkovicTT please make sure to post the new ticket here

@ncvetkovicTT Yes, we suspect there may be same issue with the other kernel, but don’t have a repro yet. SO yes lets close this issue and open a new one to track the proper LLK fix. Thanks!

I ran eltwise ADD C++ test on Vincent’s branch, it works on silicon but doesn’t on simulator, breaks with the following error:

libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_ASSERT @ ../tt_metal/third_party/umd/device/simulation/tt_simulation_device.cpp:82: cmd == DEVICE_COMMAND_EXIT
info:
Did not receive expected command from remote.
backtrace:
 --- tt_SimulationDevice::start_device(tt_device_params const&)
 --- tt::Cluster::initialize_device_drivers()
 --- tt::Cluster::Cluster()
 --- tt::Cluster::instance()
 --- tt::tt_metal::v0::GetNumAvailableDevices()
 --- ./build/test/tt_metal/unit_tests(+0x4d3ed) [0x55aa221fb3ed]
 --- ./build/test/tt_metal/unit_tests(+0x69a79) [0x55aa22217a79]
 --- testing::internal::UnitTestImpl::PostFlagParsingInit()
 --- void testing::internal::InitGoogleTestImpl<char>(int*, char**)
 --- /localdev/ncvetkovic/tt-metal/build_Release/lib/libgtest_main.so.1.13.0(main+0x2e) [0x7f20012a016e]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f2000d05083]
 --- ./build/test/tt_metal/unit_tests(+0x4cc6e) [0x55aa221fac6e]

Aborted (core dumped)

Any ideas @vtangTT how I can continue? Ideally I would like to make the simulator work with my branch: ncvetkovic/15824-max_pool_and_reduce_fail

Also, I ran test_pool.py on that branch as well, but max_pool had some other usage back then so it yields an error, and mean fails with 0.995 PCC, so I didn’t even run it on silicon.

FYI @nvelickovicTT

Ah okay that’s what I initially thought the arguments to be, thanks. I will try simulating in the morning. This case fails always btw, not on every Nth attempt, which is good for simulating.

Raising to P0 to ensure this is resolved and not delayed further. Thank you!

@ncvetkovicTT maxpool is @mywoodstock 's area of expertise. I don’t know the details.

@ncvetkovicTT in regards to what this is related to, I think it’s related to reduce_init/reduce_tile

For the basic reduce unit teststtnn/cpp/ttnn/operations/reduction/generic/device/kernels/compute/reduce_w.cpp uses mm_init/matmul_tiles. When that happens, when run after max_pool, the tests pass. However, when I commented out that option and forced reduce_init/reduce_tile to be used, I saw the same behaviour.

That is:

  1. max_pool + reduce_w using matmul - passes on the first attempt
  2. max_pool + reduce_w using reduce - fails on the first attempt (I did not try 4+ attempts, although I’m guessing the same pattern would exist)

You should be able to test via the code in https://github.com/tenstorrent/tt-metal/issues/6991#issuecomment-2559885872 by changing

test_mean(device, 1, 32, 32, -2)

to

test_mean(device, 1, 32, 32, -1)

In terms of what else uses reduce_tile, running grep 'reduce_tile(' -r * in tt-metal shows other ops use reduce_tile as well. E.g. layernorm, groupnorm, and softmax. There are also a lot of tests:

tests/tt_metal/tt_metal/test_kernels/compute/layernorm.cpp:                reduce_tile(cb_x, cb_scaler, wt + j, scaler0, dst0);
tests/tt_metal/tt_metal/test_kernels/compute/layernorm.cpp:                reduce_tile(cb_xmm2, cb_scaler, wt + wtr, scaler0, dst0);
tests/tt_metal/tt_metal/test_kernels/compute/layernorm.cpp:            // reduce_tile(cb_xmm, cb_scaler, wt+wtr, scaler0, dst0);
tests/tt_metal/tt_metal/test_kernels/compute/max_pool.cpp:            reduce_tile(in_cb_id, in_scalar_cb_id, tile_i, 0, dst_i);
tests/tt_metal/tt_metal/test_kernels/compute/rmsnorm.cpp:                reduce_tile(cb_x2, cb_scaler, wt + wtr, scaler0, dst0);
tests/tt_metal/tt_metal/test_kernels/compute/rmsnorm.cpp:            // reduce_tile(cb_xmm, cb_scaler, wt+wtr, scaler0, dst0);
tests/tt_metal/tt_metal/test_kernels/compute/max_pool_multi_core.cpp:            reduce_tile(in_cb_id, in_scalar_cb_id, tile_i, 0, dst_i);
tests/tt_metal/tt_metal/test_kernels/compute/reduce_h.cpp:                reduce_tile(tt::CBIndex::c_0, tt::CBIndex::c_2, 0, 0, reduce_dst_idx);
tests/tt_metal/tt_metal/test_kernels/compute/reduce_hw.cpp:                reduce_tile(tt::CBIndex::c_0, tt::CBIndex::c_2, 0, 0, reduce_dst_idx);
tests/tt_metal/tt_metal/test_kernels/compute/softmax.cpp:            reduce_tile(cb_exps, cb_bcast_scaler, wt, bcast_scaler0, dst0);
tests/tt_metal/tt_metal/test_kernels/compute/reduce_w.cpp:                reduce_tile(tt::CBIndex::c_0, tt::CBIndex::c_2, 0, 0, reduce_dst_idx);

Trying this other stuff out might be a big time sink though.

More context: This issue is being tracked together with #6991. They happen only on WH. In the example from the mentioned issue there’s the same pattern as explained in #6991, where max_pool+mean passes on every 4th attempt. In order to see if the pattern is related to mean, or irrelevant of the op, we should try with a different op after max_pool.

One bug reported by while enabling max pool. It seems this is also in same line. https://github.com/tenstorrent/tt-metal/issues/15481

Narrowed it down further:

Commented out

tt_metal/include/compute_kernel_api/pack_untilize.h

PACK((llk_pack_untilize_init<block_ct_dim, full_ct_dim, diagonal, narrow_row, row_num_datums>(
    ocb, face_r_dim, num_faces)));

and reduce/test_global_avg_pool worked.