tt-metal: Conv fails with OOM issue in unit tests

Describe the bug On unit testing the conv ops of model_k, for the input resolution 256x256:

  • Run the following command to reproduce the issue: pytest tests/ttnn/unit_tests/operations/test_new_conv2d.py::test_model_k_256x256_failing_convs
  • Among 7 convs, 3 convs passed. 4 convs fail with Out of Memory issue.
E       RuntimeError: TT_THROW @ ../tt_metal/impl/allocator/allocator.cpp:143: tt::exception
E       info:
E       Out of Memory: Not enough space to allocate 25477120 B L1 buffer across 5 banks, where each bank needs to store 5095424 B

On unit testing the conv ops of model_k, for the input resolution 128x128:

    • Run the following command to reproduce the issue: pytest tests/ttnn/unit_tests/operations/test_new_conv2d.py::test_model_k_128x128_failing_convs
  • Among 7 convs, 4 convs passed. 3 convs fail with Out of Memory issue.
E       RuntimeError: TT_THROW @ ../tt_metal/impl/allocator/allocator.cpp:143: tt::exception
E       info:
E       Out of Memory: Not enough space to allocate 4636672 B L1 buffer across 1 banks, where each bank needs to store 4636672 B

To Reproduce Steps to reproduce the behavior:

  1. Checkout to the branchharini/model_k_failing_convs
  2. Run the respective commands in each input resolution to reproduce the issues.

Expected behavior To run conv op for the specific input configurations without error.

Please complete the following environment information:

  • OS: [e.g. Ubuntu 20.04]
  • Device: n150

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

Fix is merged