tt-metal: [Bug Report] Matmul crashing
Describe the bug Matmul with this parameters results in crash due to limitation of L1 cache. This should never happen.
ttnn::linear(
tensor->get_value(), // assume it's just ttnn::Tensor
weight->get_value(), // assume it's just ttnn::Tensor
bias->get_value(), // assume it's just ttnn::Tensor
/* transpose_a */ false,
/* tranpose_b */ true,
/* memory_config */ std::nullopt,
/* dtype */ std::nullopt,
/* program_config */ std::nullopt,
/* activation */ std::nullopt,
/* compute_kernel_config */ core::ComputeKernelConfig::matmul(),
/* core_grid */ ttnn::CoreGrid{7, 8})
ttnn::WormholeComputeKernelConfig ComputeKernelConfig::matmul() {
ttnn::WormholeComputeKernelConfig config;
config.fp32_dest_acc_en = false;
config.math_approx_mode = false;
config.math_fidelity = MathFidelity::HiFi2;
config.packer_l1_acc = true;
return config;
}
BOUND MATMUL: A shape SimpleShape([64, 1, 1024, 768]) B shape SimpleShape([1, 1, 2304, 768]) transpose A false transpose B true
Always | FATAL | Statically allocated circular buffers on core range [(x=0,y=0) - (x=6,y=7)] grow to 6148640 B which is beyond max L1 size of 1499136 B
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /home/ubuntu/tt-metal/tt_metal/impl/program/program.cpp:819: tt::exception
info:
Statically allocated circular buffers on core range [(x=0,y=0) - (x=6,y=7)] grow to 6148640 B which is beyond max L1 size of 1499136 B
backtrace:
--- /home/ubuntu/tt-metal/build/lib/libtt_metal.so(+0x12e1bb) [0x7f06ffa301bb]
--- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::Device const*)
--- tt::tt_metal::v0::CommandQueue::run_command_impl(tt::tt_metal::CommandInterface const&)
--- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::v0::CommandQueue&, tt::tt_metal::v0::Program&, bool)
--- tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>::tensor_return_value_t ttnn::device_operation::detail::launch_on_single_device<tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>>(unsigned char, tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>::tensor_args_t const&)
--- /home/ubuntu/tt-metal/build/lib/_ttnn.so(+0x1d7c05a) [0x7f06feb8b05a]
--- tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>>(unsigned char, tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>::tensor_args_t const&)
--- /home/ubuntu/tt-metal/build/lib/_ttnn.so(+0x1d7b3cb) [0x7f06feb8a3cb]
--- /home/ubuntu/tt-metal/build/lib/_ttnn.so(+0x1d7820b) [0x7f06feb8720b]
--- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, unsigned char)
--- /home/ubuntu/tt-metal/build/lib/_ttnn.so(+0x13a9e41) [0x7f06fe1b8e41]
--- /home/ubuntu/tt-metal/build/lib/_ttnn.so(+0x13aab65) [0x7f06fe1b9b65]
--- ttnn::operations::matmul::matmul(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, std::__1::optional<tt::tt_metal::Tensor const> const&, ttnn::operations::matmul::Matmul const&, unsigned char)
--- ttnn::operations::matmul::bound_matmul(tt::tt_metal::Tensor const&, tt::tt_metal::Tensor const&, std::__1::optional<tt::tt_metal::Tensor const> const&, ttnn::operations::matmul::Matmul const&, unsigned char const&)
--- /home/ubuntu/tt-metal/build/lib/_ttnn.so(_ZN4ttnn10operations6matmul15LinearOperation6invokeERKN2tt8tt_metal6TensorES7_RKNSt3__18optionalIS6_EEbbRKNS9_IKNS4_12MemoryConfigEEENS9_IKNS4_8DataTypeEEERKNS9_IKNS8_7variantIJNS1_28MatmulMultiCoreProgramConfigENS1_45MatmulMultiCoreNonOptimizedReuseProgramConfigENS1_33MatmulMultiCoreReuseProgramConfigENS1_42MatmulMultiCoreReuseMultiCastProgramConfigENS1_44MatmulMultiCoreReuseMultiCast1DProgramConfigENS1_53MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfigEEEEEERKNS9_IKNS8_12basic_stringIcNS8_11char_traitsIcEENS8_9allocatorIcEEEEEENS9_IKNSL_IJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEEEENS9_IKNS_5types8CoreGridEEERKNS9_IKNS4_4TileEEE+0x2cd) [0x7f06fe188f2d]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0x207326) [0x555e9e605326]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0x202111) [0x555e9e600111]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0x200ec5) [0x555e9e5feec5]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0x194c3a) [0x555e9e592c3a]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0x1951c8) [0x555e9e5931c8]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0x193c11) [0x555e9e591c11]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0x18db1e) [0x555e9e58bb1e]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0xf7f3b) [0x555e9e4f5f3b]
--- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f06fc42c083]
--- /home/ubuntu/tt-metal/tt-train/build/sources/examples/nano_gpt/nano_gpt(+0xf413e) [0x555e9e4f213e]
Aborted (core dumped)
To Reproduce There are two ways to reproduce issue.
- Run
tt-trainwith configtraining_shakespear_gpt2s.yaml(increase batch size to 64). - Implement minimalistic python program which multiplies specified matrices (with given cores and compute config parameters)
Expected behavior If one of the algorithms not able to perform due to hardware limitations it should fallback to one that works.
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software: 84401ed5f3c516798d978c3fcc64d843b4fbec82
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 37 (34 by maintainers)
Commits related to this issue
- #17532: Add relevant tests to matmul trace sweeps (#15850) ### Ticket Link to Github Issue #15732 ### Problem description Need tests for various shapes to verify that they pass after making cha... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #17532: Add relevant tests to matmul trace sweeps (#15850) ### Ticket Link to Github Issue #15732 ### Problem description Need tests for various shapes to verify that they pass after making cha... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: add matmul block h/w parameter processing — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: add matmul block h/w parameter processing — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: update gpt2s tests to expect success and adjust matmul block_h/w alg — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: add matmul block h/w parameter processing — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: update gpt2s tests to expect success and adjust matmul block_h/w alg — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: adjust criteria for setting in0_block_w when choosing matmul program config — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: use auto to get a pair of variables in some matmul code — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: add matmul block h/w parameter processing — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: update gpt2s tests to expect success and adjust matmul block_h/w alg — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: adjust criteria for setting in0_block_w when choosing matmul program config — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: use auto to get a pair of variables in some matmul code — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: check fp32 dest acc for matmul subblock params and remove non opt config usage — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: add matmul block h/w parameter processing — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: update gpt2s tests to expect success and adjust matmul block_h/w alg — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: adjust criteria for setting in0_block_w when choosing matmul program config — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: use auto to get a pair of variables in some matmul code — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: check fp32 dest acc for matmul subblock params and remove non opt config usage — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #15732: update 1d matmul bias size — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
Found the issue. The amount to reserve by was not adjusted to be block sized, but the CB size was adjusted. Therefore there was not enough space to reserve the requested number of tiles.
The following fixes the problem:
Next steps:
Update:
Hey @bbradelTT! Thanks for looking into the trace. I’ve also parsed the trace. Please take a look at
ttnn::matmulandttnn::linear(shape for second input should be transposed). It contains all unique operation calls.https://docs.google.com/spreadsheets/d/1bkeeVeW_2YCMmO9DokcgyY1XNDckgoyYIbKxBxrXVCw/edit?gid=1071015504#gid=1071015504
@bbradelTT I don’t think setting program config is an easy solution/task. Here is an example of all configs that
groqmodels (found inmodels) use (just for matmul):This is a blocker for TT-train, does P0
fyi @TT-BrianLiu
I was able to re-create the issue via a pytest:
I’ll work on checking expected CB sizes vs L1 and adjust parameters accordingly in more scenarios than we currently do.
@dmakoviichuk-tt @davorchap FYI