tt-metal: Multiple calls to ttnn.fold do not work when program cache is enabled

It looks like ttnn.fold doesn’t work when called multiple times with tensors that have same setup.

Found when looking into https://github.com/tenstorrent/tt-metal/issues/19344

Repro:

import pytest
import torch
import ttnn

def test_fold(device):
    ttnn.enable_program_cache(device)
    torch.manual_seed(0)
    input_shape = (8, 224, 14, 64)
    stride_h = 16
    stride_w = 1
    torch_input_tensor = torch.rand(input_shape, dtype=torch.bfloat16)
    memory_config = ttnn.MemoryConfig(
                memory_layout=ttnn.TensorMemoryLayout.HEIGHT_SHARDED,
                buffer_type=ttnn.BufferType.L1,
                shard_spec=ttnn.ShardSpec(
                    ttnn.CoreRangeSet({ttnn.CoreRange(ttnn.CoreCoord(0, 0), ttnn.CoreCoord(7, 0))}),
                    (3136, 64),
                    ttnn.ShardOrientation.ROW_MAJOR,
                    ttnn.ShardMode.PHYSICAL,
                ),
            )
    input_tensor = ttnn.from_torch(torch_input_tensor, dtype=ttnn.bfloat16, device=device, layout=ttnn.ROW_MAJOR_LAYOUT, memory_config=memory_config)
    input_tensor2 = ttnn.from_torch(torch_input_tensor, dtype=ttnn.bfloat16, device=device, layout=ttnn.ROW_MAJOR_LAYOUT, memory_config=memory_config)
    output_tensor = ttnn.fold(input_tensor, stride_h, stride_w)
    output_tensor.deallocate() # added
    output_tensor2 = ttnn.fold(input_tensor2, stride_h, stride_w)

Expected:

  • Passes

Actual:

With the deallocate:

    @wraps(function)
    def call_wrapper(*function_args, **function_kwargs):
>       output = function(*function_args, **function_kwargs)
E       RuntimeError: TT_THROW @ /localdev/bbradel/tt-metal/tt_metal/impl/program/program.cpp:645: tt::exception
E       info:
E       No circular buffer with id 3883580496 exists in Program 1
E       backtrace:
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0xd71e39) [0x7fb2539fae39]
E        --- tt::tt_metal::detail::ProgramImpl::get_circular_buffer(unsigned long) const
E        --- tt::tt_metal::detail::GetCircularBuffer(tt::tt_metal::Program const&, unsigned long)

Without the deallocate (this is probably okay):

self = FastOperation(python_fully_qualified_name='ttnn.fold', function=<ttnn._ttnn.operations.data_movement.fold_t object at ...uts=None, golden_function=None, postprocess_golden_function_outputs=None, is_cpp_operation=True, is_experimental=False)
function_args = (ttnn.Tensor([[[[ 0.67188,  0.18359,  ...,  0.79297,  0.44531],
               [ 0.71484,  0.10938,  ...,  0.19141,  0......,  0.94531,  0.30469]]]], shape=Shape([8, 224, 14, 64]), dtype=DataType::BFLOAT16, layout=Layout::ROW_MAJOR), 16, 1)
function_kwargs = {}

    def __call__(self, *function_args, **function_kwargs):
>       return self.function(*function_args, **function_kwargs)
E       RuntimeError: TT_THROW @ /localdev/bbradel/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
E       info:
E       Out of Memory: Not enough space to allocate 3211264 B L1 buffer across 8 banks, where each bank needs to store 401408 B
E       backtrace:
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x591065) [0x7fb4e0b98065]
E        --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::__1::optional<unsigned int>)
E        --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::Buffer*)
E        --- tt::tt_metal::Buffer::allocate_impl()
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x4c8251) [0x7fb4e0acf251]
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x4c8225) [0x7fb4e0acf225]
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x4c81e5) [0x7fb4e0acf1e5]
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x4c81bd) [0x7fb4e0acf1bd]
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x4c74b9) [0x7fb4e0ace4b9]
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x488c02) [0x7fb4e0a8fc02]
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x4887e5) [0x7fb4e0a8f7e5]
E        --- /localdev/bbradel/tt-metal/build_Debug/lib/libtt_metal.so(+0x47531e) [0x7fb4e0a7c31e]
E        --- tt::tt_metal::Device::push_work(std::__1::function<void ()>, bool)
E        --- tt::tt_metal::Buffer::create(tt::tt_metal::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::__1::optional<tt::tt_metal::ShardSpecBuffer> const&, std::__1::optional<bool>, std::__1::optional<tt::stl::StrongType<unsigned char, tt::tt_metal::SubDeviceIdTag>>)
E        --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::IDevice*, tt::tt_metal::TensorSpec const&)
E        --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::IDevice*)
E        --- ttnn::operations::data_movement::Fold::create_output_tensors(ttnn::operations::data_movement::Fold::operation_attributes_t const&, ttnn::operations::data_movement::Fold::tensor_args_t const&)
E        --- ttnn::operations::data_movement::Fold::tensor_return_value_t ttnn::device_operation::detail::launch_on_single_device<ttnn::operations::data_movement::Fold>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::data_movement::Fold::operation_attributes_t const&, ttnn::operations::data_movement::Fold::tensor_args_t const&)
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eb3f8) [0x7fb4e4d303f8]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eb3a8) [0x7fb4e4d303a8]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eb360) [0x7fb4e4d30360]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eb308) [0x7fb4e4d30308]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eb080) [0x7fb4e4d30080]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eafe2) [0x7fb4e4d2ffe2]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eaf44) [0x7fb4e4d2ff44]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35eaeed) [0x7fb4e4d2feed]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35ea379) [0x7fb4e4d2f379]
E        --- ttnn::operations::data_movement::Fold::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::data_movement::Fold>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::data_movement::Fold::operation_attributes_t const&, ttnn::operations::data_movement::Fold::tensor_args_t const&)
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35e99bd) [0x7fb4e4d2e9bd]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35e943d) [0x7fb4e4d2e43d]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x35d44b4) [0x7fb4e4d194b4]
E        --- ttnn::operations::data_movement::FoldOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, unsigned int, unsigned int, bool, std::__1::optional<tt::tt_metal::Shape const> const&, unsigned int, unsigned int, unsigned int, std::__1::optional<CoreRangeSet> const&, std::__1::optional<tt::tt_metal::MemoryConfig> const&)
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314d073) [0x7fb4e4892073]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314cd91) [0x7fb4e4891d91]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314ccd5) [0x7fb4e4891cd5]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314c798) [0x7fb4e4891798]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314c113) [0x7fb4e4891113]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314bff8) [0x7fb4e4890ff8]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314bf88) [0x7fb4e4890f88]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314bf38) [0x7fb4e4890f38]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314b29c) [0x7fb4e489029c]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x4dcdafd) [0x7fb4e6512afd]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x4db8598) [0x7fb4e64fd598]
E        --- void tt::tt_metal::operation::launch_op_func<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(std::__1::function<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> (std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&)> const&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>>, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>>)
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x314a748) [0x7fb4e488f748]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x3149c3f) [0x7fb4e488ec3f]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x3148d61) [0x7fb4e488dd61]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x3148719) [0x7fb4e488d719]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x3148551) [0x7fb4e488d551]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x3134456) [0x7fb4e4879456]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x31342e6) [0x7fb4e48792e6]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x313402c) [0x7fb4e487902c]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x3133f12) [0x7fb4e4878f12]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x3133d49) [0x7fb4e4878d49]
E        --- /localdev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x1d7f3cb) [0x7fb4e34c43cb]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x18ae12) [0x55c6638a9e12]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_MakeTpCall+0x25b) [0x55c6638a075b]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x198bc0) [0x55c6638b7bc0]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x29cbcf) [0x55c6639bbbcf]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(PyObject_Call+0xbb) [0x55c6638b83cb]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x55c663896cf3]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_FastCallDictTstate+0xc4) [0x55c66389f934]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_Call_Prepend+0x5c) [0x55c6638b48ec]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x29cba4) [0x55c6639bbba4]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_MakeTpCall+0x25b) [0x55c6638a075b]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x5f66) [0x55c66389a1d6]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(PyObject_Call+0x122) [0x55c6638b8432]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x55c663896cf3]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x55c663896cf3]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x55c6638998b2]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x198791) [0x55c6638b7791]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x55c6638998b2]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x55c66389f9dd]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_Call_Prepend+0x5c) [0x55c6638b48ec]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x29cba4) [0x55c6639bbba4]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_MakeTpCall+0x25b) [0x55c6638a075b]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x66f2) [0x55c66389a962]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x55c663894a74]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x55c663896cf3]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x55c6638998b2]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x198791) [0x55c6638b7791]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x55c6638998b2]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x55c66389f9dd]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyObject_Call_Prepend+0x5c) [0x55c6638b48ec]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(+0x29cba4) [0x55c6639bbba4]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(PyObject_Call+0xbb) [0x55c6638b83cb]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x55c663896cf3]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyFunction_Vectorcall+0x7c) [0x55c6638aa66c]
E        --- /localdev/bbradel/tt-metal/python_env/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x55c66389492f]

ttnn/ttnn/decorators.py:333: RuntimeError

WH N150 commit a769d7e4f9

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (19 by maintainers)

Commits related to this issue

Most upvoted comments

@mbahnasTT can you sync with @yugi957 maybe even in person to get to the bottom of this (both of you are in the SC office).

@yugi957 let me share another test from Stable Diffusion 3.5, that will need T3K (N300 x4)

hi @mbahnasTT , @yugi957 will look at this this week, have a solution by end of week to early next week.