tt-metal: [Bug Report] Qwen 2.5 72B Out of Memory for 32k+ context prefill

Describe the bug

Qwen 2.5 72B Instruct has Out of Memory error when attempting to run 1 prompt with input sequence length (ISL) of 32000+. Prompts between ISL=1 and ISL=16384 work fine.

libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
info:
Out of Memory: Not enough space to allocate 268435456 B DRAM buffer across 12 banks, where each bank needs to store 22370304 B

This is on v0.56.0-rc33 which is listed as the tt-metal release for Qwen 2.5 72B (TP=8): https://github.com/tenstorrent/tt-metal?tab=readme-ov-file#llms

To Reproduce Steps to reproduce the behavior:

  1. build tt-metal: v0.56.0-rc33
  2. run long context demo.py for qwen/Qwen2.5
export HF_MODEL=qwen/Qwen2.5-72B-Instruct
pytest models/demos/llama3/demo/demo.py -k "performance and long"

Expected behavior The model should be able to run with 32k+ context prefill. Running 32x users with 4k+ context works as expected. If the model cannot run with 32k+ context prefills an error message should be raised instead of crashing.

Logs

$ /tt-metal$ pytest models/demos/llama3/demo/demo.py -k "performance and long"
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.8.10, pytest-7.2.2, pluggy-1.5.0 -- /tt-metal/python_env/bin/python3
cachedir: .pytest_cache
rootdir: /tt-metal, configfile: pytest.ini
plugins: dash-2.15.0, xdist-3.6.1, timeout-2.2.0, split-0.8.2, anyio-4.5.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collecting ... 2025-03-30 15:24:27.276 | WARNING  | models.perf.benchmarking_utils:<module>:16 - Skipping import of pydantic_models for benchmarking since not running in CI environment
                 Device | INFO     | Opening user mode device driver
  Detecting chips (found 8)
2025-03-30 15:24:27.670 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.673 | INFO     | SiliconDriver   - Opened PCI device 1; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.676 | INFO     | SiliconDriver   - Opened PCI device 2; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.678 | INFO     | SiliconDriver   - Opened PCI device 3; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.683 | INFO     | SiliconDriver   - Detected PCI devices: [0, 1, 2, 3]
2025-03-30 15:24:27.683 | INFO     | SiliconDriver   - Using local chip ids: {0, 1, 2, 3} and remote chip ids {4, 5, 6, 7}
2025-03-30 15:24:27.845 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 0)
2025-03-30 15:24:27.845 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 1)
2025-03-30 15:24:27.845 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 2)
2025-03-30 15:24:27.845 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 3)
2025-03-30 15:24:27.846 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 4)
2025-03-30 15:24:27.846 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 5)
2025-03-30 15:24:27.846 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 6)
2025-03-30 15:24:27.847 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 7)
collected 8 items / 7 deselected / 1 selected

models/demos/llama3/demo/demo.py::test_llama_demo[wormhole_b0-True-8-device_params0-performance-long-context]                   Metal | INFO     | Initializing device 0. Program cache is NOT enabled
           BuildKernels | INFO     | Skipping deleting built cache
                  Metal | INFO     | AI CLK for device 0 is:   1000 MHz
                  Metal | INFO     | Initializing device 1. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 1 is:   1000 MHz
                  Metal | INFO     | Initializing device 2. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 2 is:   1000 MHz
                  Metal | INFO     | Initializing device 3. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 3 is:   1000 MHz
                  Metal | INFO     | Initializing device 4. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 4 is:   1000 MHz
                  Metal | INFO     | Initializing device 5. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 5 is:   1000 MHz
                  Metal | INFO     | Initializing device 6. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 6 is:   1000 MHz
                  Metal | INFO     | Initializing device 7. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 7 is:   1000 MHz
                  Metal | INFO     | While initializing device 0, ethernet dispatch core (x=20,y=16) on Device 0 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 0, ethernet dispatch core (x=23,y=16) on Device 0 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 4, ethernet dispatch core (x=20,y=16) on Device 4 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 4, ethernet dispatch core (x=23,y=16) on Device 4 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 1, ethernet dispatch core (x=20,y=16) on Device 1 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 1, ethernet dispatch core (x=23,y=16) on Device 1 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 5, ethernet dispatch core (x=20,y=16) on Device 5 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 5, ethernet dispatch core (x=23,y=16) on Device 5 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 2, ethernet dispatch core (x=20,y=16) on Device 2 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 2, ethernet dispatch core (x=23,y=16) on Device 2 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 6, ethernet dispatch core (x=20,y=16) on Device 6 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 6, ethernet dispatch core (x=23,y=16) on Device 6 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 3, ethernet dispatch core (x=20,y=16) on Device 3 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 3, ethernet dispatch core (x=23,y=16) on Device 3 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 7, ethernet dispatch core (x=20,y=16) on Device 7 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 7, ethernet dispatch core (x=23,y=16) on Device 7 detected as still running, issuing exit signal.
                  Metal | INFO     | Enabling program cache on device 4
                  Metal | INFO     | Enabling program cache on device 0
                  Metal | INFO     | Enabling program cache on device 3
                  Metal | INFO     | Enabling program cache on device 7
                  Metal | INFO     | Enabling program cache on device 6
                  Metal | INFO     | Enabling program cache on device 2
                  Metal | INFO     | Enabling program cache on device 1
                  Metal | INFO     | Enabling program cache on device 5
2025-03-30 15:24:30.938 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:207 - Start profiler
2025-03-30 15:24:30.938 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:211 - Reading inputs...
2025-03-30 15:24:31.349 | INFO     | models.demos.llama3.demo.demo:load_and_cache_context:46 - Downloaded and cached context: https://www.gutenberg.org/cache/epub/84/pg84.txt
2025-03-30 15:24:31.350 | INFO     | models.demos.llama3.demo.demo:load_and_cache_context:57 - Clipped the context text to 260000 characters
2025-03-30 15:24:31.351 | INFO     | models.demos.llama3.tt.model_config:__init__:153 - Checkpoint directory: /home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct
2025-03-30 15:24:31.351 | INFO     | models.demos.llama3.tt.model_config:__init__:154 - Tokenizer file: /home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct/tokenizer.model
2025-03-30 15:24:31.351 | INFO     | models.demos.llama3.tt.model_config:__init__:155 - Cache directory: /home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct/T3K
2025-03-30 15:24:31.352 | INFO     | models.demos.llama3.tt.model_config:_set_params_from_dict:1034 - PAD_MLP_CORES=32, padding hidden dim from 29568 to 32768
2025-03-30 15:24:32.654 | INFO     | models.demos.llama3.tt.model_config:__init__:890 - Attention grid: ttnn.CoreGrid(x=8, y=4)
2025-03-30 15:24:32.654 | INFO     | models.demos.llama3.tt.model_config:__init__:891 - MLP grid: ttnn.CoreGrid(x=8, y=8)
2025-03-30 15:24:32.654 | INFO     | models.demos.llama3.tt.model_config:__init__:892 - MLP prefill grids @ 32: w1/w3: (1, 8), w2: (1, 8)
2025-03-30 15:24:32.654 | INFO     | models.demos.llama3.tt.model_config:__init__:893 - MLP prefill grids @ max_seq_len(65536): w1/w3: (8, 8), w2: (8, 8)
2025-03-30 15:24:32.654 | INFO     | models.demos.llama3.tt.model_config:__init__:896 - LM head grid: ttnn.CoreGrid(x=8, y=8)
2025-03-30 15:24:32.654 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:250 - Loading weights...
2025-03-30 15:24:33.626 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:274 - Loading weights to device...
  0%|                                                                                                                                                                                    | 0/80 [00:00<?, ?it/s]                 Always | FATAL    | Cannot open "/home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct/T3K/tensor_cache_instruct_bfp8/kvcache_torch.Size([1, 1, 65536, 128])_multi_device_dtype_BFLOAT8_B_layout_TILE.bin"
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80/80 [03:02<00:00,  2.28s/it]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 37/37 [00:30<00:00,  1.21it/s]
2025-03-30 15:28:28.559 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:295 - Finished loading weights to device.
2025-03-30 15:28:28.559 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:299 - Starting inference...
2025-03-30 15:28:28.560 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:301 - Processing batch 0
2025-03-30 15:28:28.795 | INFO     | models.demos.llama3.demo.demo:preprocess_inputs_prefill:109 - Encoded prompt lengths:59803
2025-03-30 15:28:28.796 | INFO     | models.demos.llama3.demo.demo:preprocess_inputs_prefill:149 - # of users: 1
2025-03-30 15:28:28.944 | INFO     | models.demos.llama3.demo.demo:run_llama3_demo:333 - Starting prefill...
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24. First unused index: 2. Kernels: writer_unary_interleaved_start_id_blocked, reader_unary_interleaved_ln_rm_gb_pre_allgather, layernorm_pre_allgather
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5,16,24,26,27,28,30. First unused index: 3. Kernels: writer_unary_interleaved_start_id_blocked, reader_unary_interleaved_ln_rm_gb_post_allgather, layernorm_post_allgather
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_receiver_writer_padding, reader_bmm_tile_layout_in0_receiver, bmm_large_block_zm_fused_bias_activation
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_receiver_writer_padding, reader_bmm_tile_layout_in0_sender_padding, bmm_large_block_zm_fused_bias_activation
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_sender_writer_padding, reader_bmm_tile_layout_in0_receiver, bmm_large_block_zm_fused_bias_activation
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_sender_writer_padding, reader_bmm_tile_layout_in0_sender_padding, bmm_large_block_zm_fused_bias_activation
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: reader_unary_interleaved_start_id
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: writer_unary_stick_layout_split_rows_multicore, reader_unary_interleaved_start_id, untilize
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: reader_unary_pad_dims_split_rows_multicore
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: writer_unary_interleaved_start_id, reader_unary_pad_dims_split_rows_multicore, tilize
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 1. First unused index: 0. Kernels: writer_tm_tile_layout_nlp_create_qkv_heads, reader_tm_tile_layout_nlp_create_qkv_heads
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24,25,26. First unused index: 4. Kernels: reader_rotary_embedding_llama_interleaved_start_id
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24,25,26. First unused index: 4. Kernels: writer_rotary_embedding_llama_interleaved_start_id, reader_rotary_embedding_llama_interleaved_start_id, rotary_embedding_llama
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 2. First unused index: 1. Kernels: reader_unary_interleaved_start_id
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 2. First unused index: 1. Kernels: writer_unary_interleaved_start_id, reader_unary_interleaved_start_id, eltwise_sfpu
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24,25,26,27,28,29,30,31. First unused index: 6. Kernels: writer_interleaved, reader_interleaved, sdpa
                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B

                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
                 Always | FATAL    | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
info:
Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
backtrace:
 --- /tt-metal/build/lib/libtt_metal.so(+0x1dfbd5) [0x7f68df013bd5]
 --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::__1::optional<unsigned int>)
 --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*)
 --- tt::tt_metal::v0::Buffer::allocate_impl()
 --- /tt-metal/build/lib/libtt_metal.so(+0x1a5f70) [0x7f68defd9f70]
 --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::__1::optional<tt::tt_metal::ShardSpecBuffer> const&, std::__1::optional<bool>, std::__1::optional<tt::tt_metal::SubDeviceId>)
 --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&)
 --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*)
 --- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations6matmul6MatmulEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKNSt3__16vectorINS0_6TensorENSD_9allocatorISF_EEEERKNSE_INSD_8optionalISF_EENSG_ISM_EEEE+0x319) [0x7f68e0fd18a9]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17ccea9) [0x7f68e1025ea9]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x3a) [0x7f68e27b210a]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5904a) [0x7f68e27b204a]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7f68e27b1a07]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f583bb) [0x7f68e27b13bb]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5809b) [0x7f68e27b109b]
 --- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>)
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17cc6a8) [0x7f68e10256a8]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5fa55) [0x7f68e27b8a55]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f6002f) [0x7f68e27b902f]
 --- /tt-metal/build/lib/libtt_metal.so(+0x190c9b) [0x7f68defc4c9b]
 --- /tt-metal/build/lib/libtt_metal.so(+0x190f6b) [0x7f68defc4f6b]
 --- /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6951259609]
 --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6951393353]

Fatal Python error: Aborted

Thread 0x00007f63ca7fc700 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 306 in wait
  File "/usr/lib/python3.8/threading.py", line 558 in wait
  File "/tt-metal/python_env/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007f69510a8740 (most recent call first):
  File "/tt-metal/models/demos/llama3/tt/llama_mlp.py", line 204 in forward
  File "/tt-metal/models/demos/llama3/tt/llama_decoder.py", line 141 in forward
  File "/tt-metal/models/common/lightweightmodule.py", line 12 in __call__
  File "/tt-metal/models/demos/llama3/tt/llama_model.py", line 342 in forward
  File "/tt-metal/models/common/lightweightmodule.py", line 12 in __call__
  File "/tt-metal/models/demos/llama3/demo/demo.py", line 364 in run_llama3_demo
  File "/tt-metal/models/demos/llama3/demo/demo.py", line 996 in test_llama_demo
  File "/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/tt-metal/python_env/lib/python3.8/site-packalibc++abi: geterminating due to uncaught exception of type std::runtime_error: TT_THROW @ /tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
info:
Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
backtrace:
 --- /tt-metal/build/lib/libtt_metal.so(+0x1dfbd5) [0x7f68df013bd5]
 --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::__1::optional<unsigned int>)
 --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*)
 --- tt::tt_metal::v0::Buffer::allocate_impl()
 --- /tt-metal/build/lib/libtt_metal.so(+0x1a5f70) [0x7f68defd9f70]
 --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::__1::optional<tt::tt_metal::ShardSpecBuffer> const&, std::__1::optional<bool>, std::__1::optional<tt::tt_metal::SubDeviceId>)
 --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&)
 --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*)
 --- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations6matmul6MatmulEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKNSt3__16vectorINS0_6TensorENSD_9allocatorISF_EEEERKNSE_INSD_8optionalISF_EENSG_ISM_EEEE+0x319) [0x7f68e0fd18a9]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17ccea9) [0x7f68e1025ea9]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x3a) [0x7f68e27b210a]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5904a) [0x7f68e27b204a]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7f68e27b1a07]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f583bb) [0x7f68e27b13bb]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5809b) [0x7f68e27b109b]
 --- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>)
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17cc6a8) [0x7f68e10256a8]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5fa55) [0x7f68e27b8a55]
 --- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f6002f) [0x7f68e27b902f]
 --- /tt-metal/build/lib/libtt_metal.so(+0x190c9b) [0x7f68defc4c9b]
 --- /tt-metal/build/lib/libtt_metal.so(+0x190f6b) [0x7f68defc4f6b]
 --- /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6951259609]
 --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6951393353]
s
/pluggy/_callers.py", line 103 in _multicall
  File "/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tt-metalAborted (core dumped)

Scheenshots tt-smi firmware / topology:

Image Image

Please complete the following environment information:

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 16 (14 by maintainers)

Most upvoted comments

Thanks @skhorasganiTT, using MAX_PREFILL_CHUNK_SIZE=16 I was able to make successful requests with seqlen > 32k: https://github.com/tenstorrent/tt-inference-server/issues/127#issuecomment-3079508996

We discussed offline about if this should be reflected in model_config.py. If I measure the long-context perf and it doesn’t take a huge hit, we will change the prefill chunk size to 16 instead of 32

Suggested to @bgoelTT offline to try setting MAX_PREFILL_CHUNK_SIZE=16.

  • In terms of kv cache size, the 64k demo uses {"page_block_size": 64, "page_max_num_blocks_per_dp": 2048}, while in vLLM the block size is 64 and the number of blocks is slightly higher (roughly 2%) than 2048 (see determine_num_available_blocks in https://github.com/tenstorrent/vllm/blob/dev/vllm/worker/tt_worker.py#L211 for details). The exact number of blocks is printed in the server logs in a format similar to [executor_base.py:112] # tt blocks: 1034, # CPU blocks: 0.
  • Furthermore, for prefilling large sequence lengths we do chunked prefill, where the max chunk sizes are set in model config (MAX_PREFILL_CHUNK_SIZES_DIV1024) - https://github.com/tenstorrent/tt-metal/blob/main/models/tt_transformers/tt/model_config.py#L550. For Qwen72B-T3K it does chunks of 32k by default (i.e. the 64k demo is running two chunks of 32k). It could be the case that that is very tight and the slight number of extra blocks that vLLM allocates for the cache causes it to run out of memory, so suggested reducing max chunk size to 16k (which may reduce prefill perf for > 16k though will be necessary if it doesn’t fit and if we don’t want to change model data formats).