tt-metal: [Bug Report] Qwen 2.5 72B Out of Memory for 32k+ context prefill
Describe the bug
Qwen 2.5 72B Instruct has Out of Memory error when attempting to run 1 prompt with input sequence length (ISL) of 32000+. Prompts between ISL=1 and ISL=16384 work fine.
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
info:
Out of Memory: Not enough space to allocate 268435456 B DRAM buffer across 12 banks, where each bank needs to store 22370304 B
This is on v0.56.0-rc33 which is listed as the tt-metal release for Qwen 2.5 72B (TP=8): https://github.com/tenstorrent/tt-metal?tab=readme-ov-file#llms
To Reproduce Steps to reproduce the behavior:
- build tt-metal: v0.56.0-rc33
- run long context demo.py for qwen/Qwen2.5
export HF_MODEL=qwen/Qwen2.5-72B-Instruct
pytest models/demos/llama3/demo/demo.py -k "performance and long"
Expected behavior The model should be able to run with 32k+ context prefill. Running 32x users with 4k+ context works as expected. If the model cannot run with 32k+ context prefills an error message should be raised instead of crashing.
Logs
$ /tt-metal$ pytest models/demos/llama3/demo/demo.py -k "performance and long"
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.8.10, pytest-7.2.2, pluggy-1.5.0 -- /tt-metal/python_env/bin/python3
cachedir: .pytest_cache
rootdir: /tt-metal, configfile: pytest.ini
plugins: dash-2.15.0, xdist-3.6.1, timeout-2.2.0, split-0.8.2, anyio-4.5.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collecting ... 2025-03-30 15:24:27.276 | WARNING | models.perf.benchmarking_utils:<module>:16 - Skipping import of pydantic_models for benchmarking since not running in CI environment
Device | INFO | Opening user mode device driver
Detecting chips (found 8)
2025-03-30 15:24:27.670 | INFO | SiliconDriver - Opened PCI device 0; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.673 | INFO | SiliconDriver - Opened PCI device 1; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.676 | INFO | SiliconDriver - Opened PCI device 2; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.678 | INFO | SiliconDriver - Opened PCI device 3; KMD version: 1.31.0, IOMMU: disabled
2025-03-30 15:24:27.683 | INFO | SiliconDriver - Detected PCI devices: [0, 1, 2, 3]
2025-03-30 15:24:27.683 | INFO | SiliconDriver - Using local chip ids: {0, 1, 2, 3} and remote chip ids {4, 5, 6, 7}
2025-03-30 15:24:27.845 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 0)
2025-03-30 15:24:27.845 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 1)
2025-03-30 15:24:27.845 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 2)
2025-03-30 15:24:27.845 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 3)
2025-03-30 15:24:27.846 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 4)
2025-03-30 15:24:27.846 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 5)
2025-03-30 15:24:27.846 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 6)
2025-03-30 15:24:27.847 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 7)
collected 8 items / 7 deselected / 1 selected
models/demos/llama3/demo/demo.py::test_llama_demo[wormhole_b0-True-8-device_params0-performance-long-context] Metal | INFO | Initializing device 0. Program cache is NOT enabled
BuildKernels | INFO | Skipping deleting built cache
Metal | INFO | AI CLK for device 0 is: 1000 MHz
Metal | INFO | Initializing device 1. Program cache is NOT enabled
Metal | INFO | AI CLK for device 1 is: 1000 MHz
Metal | INFO | Initializing device 2. Program cache is NOT enabled
Metal | INFO | AI CLK for device 2 is: 1000 MHz
Metal | INFO | Initializing device 3. Program cache is NOT enabled
Metal | INFO | AI CLK for device 3 is: 1000 MHz
Metal | INFO | Initializing device 4. Program cache is NOT enabled
Metal | INFO | AI CLK for device 4 is: 1000 MHz
Metal | INFO | Initializing device 5. Program cache is NOT enabled
Metal | INFO | AI CLK for device 5 is: 1000 MHz
Metal | INFO | Initializing device 6. Program cache is NOT enabled
Metal | INFO | AI CLK for device 6 is: 1000 MHz
Metal | INFO | Initializing device 7. Program cache is NOT enabled
Metal | INFO | AI CLK for device 7 is: 1000 MHz
Metal | INFO | While initializing device 0, ethernet dispatch core (x=20,y=16) on Device 0 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 0, ethernet dispatch core (x=23,y=16) on Device 0 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 4, ethernet dispatch core (x=20,y=16) on Device 4 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 4, ethernet dispatch core (x=23,y=16) on Device 4 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 1, ethernet dispatch core (x=20,y=16) on Device 1 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 1, ethernet dispatch core (x=23,y=16) on Device 1 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 5, ethernet dispatch core (x=20,y=16) on Device 5 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 5, ethernet dispatch core (x=23,y=16) on Device 5 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 2, ethernet dispatch core (x=20,y=16) on Device 2 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 2, ethernet dispatch core (x=23,y=16) on Device 2 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 6, ethernet dispatch core (x=20,y=16) on Device 6 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 6, ethernet dispatch core (x=23,y=16) on Device 6 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 3, ethernet dispatch core (x=20,y=16) on Device 3 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 3, ethernet dispatch core (x=23,y=16) on Device 3 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 7, ethernet dispatch core (x=20,y=16) on Device 7 detected as still running, issuing exit signal.
Metal | INFO | While initializing device 7, ethernet dispatch core (x=23,y=16) on Device 7 detected as still running, issuing exit signal.
Metal | INFO | Enabling program cache on device 4
Metal | INFO | Enabling program cache on device 0
Metal | INFO | Enabling program cache on device 3
Metal | INFO | Enabling program cache on device 7
Metal | INFO | Enabling program cache on device 6
Metal | INFO | Enabling program cache on device 2
Metal | INFO | Enabling program cache on device 1
Metal | INFO | Enabling program cache on device 5
2025-03-30 15:24:30.938 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:207 - Start profiler
2025-03-30 15:24:30.938 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:211 - Reading inputs...
2025-03-30 15:24:31.349 | INFO | models.demos.llama3.demo.demo:load_and_cache_context:46 - Downloaded and cached context: https://www.gutenberg.org/cache/epub/84/pg84.txt
2025-03-30 15:24:31.350 | INFO | models.demos.llama3.demo.demo:load_and_cache_context:57 - Clipped the context text to 260000 characters
2025-03-30 15:24:31.351 | INFO | models.demos.llama3.tt.model_config:__init__:153 - Checkpoint directory: /home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct
2025-03-30 15:24:31.351 | INFO | models.demos.llama3.tt.model_config:__init__:154 - Tokenizer file: /home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct/tokenizer.model
2025-03-30 15:24:31.351 | INFO | models.demos.llama3.tt.model_config:__init__:155 - Cache directory: /home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct/T3K
2025-03-30 15:24:31.352 | INFO | models.demos.llama3.tt.model_config:_set_params_from_dict:1034 - PAD_MLP_CORES=32, padding hidden dim from 29568 to 32768
2025-03-30 15:24:32.654 | INFO | models.demos.llama3.tt.model_config:__init__:890 - Attention grid: ttnn.CoreGrid(x=8, y=4)
2025-03-30 15:24:32.654 | INFO | models.demos.llama3.tt.model_config:__init__:891 - MLP grid: ttnn.CoreGrid(x=8, y=8)
2025-03-30 15:24:32.654 | INFO | models.demos.llama3.tt.model_config:__init__:892 - MLP prefill grids @ 32: w1/w3: (1, 8), w2: (1, 8)
2025-03-30 15:24:32.654 | INFO | models.demos.llama3.tt.model_config:__init__:893 - MLP prefill grids @ max_seq_len(65536): w1/w3: (8, 8), w2: (8, 8)
2025-03-30 15:24:32.654 | INFO | models.demos.llama3.tt.model_config:__init__:896 - LM head grid: ttnn.CoreGrid(x=8, y=8)
2025-03-30 15:24:32.654 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:250 - Loading weights...
2025-03-30 15:24:33.626 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:274 - Loading weights to device...
0%| | 0/80 [00:00<?, ?it/s] Always | FATAL | Cannot open "/home/container_app_user/cache_root/model_weights/Qwen2.5-72B-Instruct/T3K/tensor_cache_instruct_bfp8/kvcache_torch.Size([1, 1, 65536, 128])_multi_device_dtype_BFLOAT8_B_layout_TILE.bin"
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80/80 [03:02<00:00, 2.28s/it]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 37/37 [00:30<00:00, 1.21it/s]
2025-03-30 15:28:28.559 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:295 - Finished loading weights to device.
2025-03-30 15:28:28.559 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:299 - Starting inference...
2025-03-30 15:28:28.560 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:301 - Processing batch 0
2025-03-30 15:28:28.795 | INFO | models.demos.llama3.demo.demo:preprocess_inputs_prefill:109 - Encoded prompt lengths:59803
2025-03-30 15:28:28.796 | INFO | models.demos.llama3.demo.demo:preprocess_inputs_prefill:149 - # of users: 1
2025-03-30 15:28:28.944 | INFO | models.demos.llama3.demo.demo:run_llama3_demo:333 - Starting prefill...
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24. First unused index: 2. Kernels: writer_unary_interleaved_start_id_blocked, reader_unary_interleaved_ln_rm_gb_pre_allgather, layernorm_pre_allgather
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5,16,24,26,27,28,30. First unused index: 3. Kernels: writer_unary_interleaved_start_id_blocked, reader_unary_interleaved_ln_rm_gb_post_allgather, layernorm_post_allgather
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_receiver_writer_padding, reader_bmm_tile_layout_in0_receiver, bmm_large_block_zm_fused_bias_activation
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_receiver_writer_padding, reader_bmm_tile_layout_in0_sender_padding, bmm_large_block_zm_fused_bias_activation
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_sender_writer_padding, reader_bmm_tile_layout_in0_receiver, bmm_large_block_zm_fused_bias_activation
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 4,5. First unused index: 2. Kernels: reader_bmm_tile_layout_in1_sender_writer_padding, reader_bmm_tile_layout_in0_sender_padding, bmm_large_block_zm_fused_bias_activation
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: reader_unary_interleaved_start_id
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: writer_unary_stick_layout_split_rows_multicore, reader_unary_interleaved_start_id, untilize
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: reader_unary_pad_dims_split_rows_multicore
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16. First unused index: 1. Kernels: writer_unary_interleaved_start_id, reader_unary_pad_dims_split_rows_multicore, tilize
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 1. First unused index: 0. Kernels: writer_tm_tile_layout_nlp_create_qkv_heads, reader_tm_tile_layout_nlp_create_qkv_heads
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24,25,26. First unused index: 4. Kernels: reader_rotary_embedding_llama_interleaved_start_id
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24,25,26. First unused index: 4. Kernels: writer_rotary_embedding_llama_interleaved_start_id, reader_rotary_embedding_llama_interleaved_start_id, rotary_embedding_llama
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 2. First unused index: 1. Kernels: reader_unary_interleaved_start_id
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 2. First unused index: 1. Kernels: writer_unary_interleaved_start_id, reader_unary_interleaved_start_id, eltwise_sfpu
Metal | WARNING | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 16,24,25,26,27,28,29,30,31. First unused index: 6. Kernels: writer_interleaved, reader_interleaved, sdpa
Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
Always | FATAL | Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
info:
Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
backtrace:
--- /tt-metal/build/lib/libtt_metal.so(+0x1dfbd5) [0x7f68df013bd5]
--- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::__1::optional<unsigned int>)
--- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*)
--- tt::tt_metal::v0::Buffer::allocate_impl()
--- /tt-metal/build/lib/libtt_metal.so(+0x1a5f70) [0x7f68defd9f70]
--- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::__1::optional<tt::tt_metal::ShardSpecBuffer> const&, std::__1::optional<bool>, std::__1::optional<tt::tt_metal::SubDeviceId>)
--- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&)
--- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*)
--- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations6matmul6MatmulEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKNSt3__16vectorINS0_6TensorENSD_9allocatorISF_EEEERKNSE_INSD_8optionalISF_EENSG_ISM_EEEE+0x319) [0x7f68e0fd18a9]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17ccea9) [0x7f68e1025ea9]
--- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x3a) [0x7f68e27b210a]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5904a) [0x7f68e27b204a]
--- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7f68e27b1a07]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f583bb) [0x7f68e27b13bb]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5809b) [0x7f68e27b109b]
--- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>)
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17cc6a8) [0x7f68e10256a8]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5fa55) [0x7f68e27b8a55]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f6002f) [0x7f68e27b902f]
--- /tt-metal/build/lib/libtt_metal.so(+0x190c9b) [0x7f68defc4c9b]
--- /tt-metal/build/lib/libtt_metal.so(+0x190f6b) [0x7f68defc4f6b]
--- /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6951259609]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6951393353]
Fatal Python error: Aborted
Thread 0x00007f63ca7fc700 (most recent call first):
File "/usr/lib/python3.8/threading.py", line 306 in wait
File "/usr/lib/python3.8/threading.py", line 558 in wait
File "/tt-metal/python_env/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap
Thread 0x00007f69510a8740 (most recent call first):
File "/tt-metal/models/demos/llama3/tt/llama_mlp.py", line 204 in forward
File "/tt-metal/models/demos/llama3/tt/llama_decoder.py", line 141 in forward
File "/tt-metal/models/common/lightweightmodule.py", line 12 in __call__
File "/tt-metal/models/demos/llama3/tt/llama_model.py", line 342 in forward
File "/tt-metal/models/common/lightweightmodule.py", line 12 in __call__
File "/tt-metal/models/demos/llama3/demo/demo.py", line 364 in run_llama3_demo
File "/tt-metal/models/demos/llama3/demo/demo.py", line 996 in test_llama_demo
File "/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
File "/tt-metal/python_env/lib/python3.8/site-packalibc++abi: geterminating due to uncaught exception of type std::runtime_error: TT_THROW @ /tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
info:
Out of Memory: Not enough space to allocate 536870912 B DRAM buffer across 12 banks, where each bank needs to store 44740608 B
backtrace:
--- /tt-metal/build/lib/libtt_metal.so(+0x1dfbd5) [0x7f68df013bd5]
--- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::__1::optional<unsigned int>)
--- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*)
--- tt::tt_metal::v0::Buffer::allocate_impl()
--- /tt-metal/build/lib/libtt_metal.so(+0x1a5f70) [0x7f68defd9f70]
--- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::__1::optional<tt::tt_metal::ShardSpecBuffer> const&, std::__1::optional<bool>, std::__1::optional<tt::tt_metal::SubDeviceId>)
--- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&)
--- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*)
--- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations6matmul6MatmulEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKNSt3__16vectorINS0_6TensorENSD_9allocatorISF_EEEERKNSE_INSD_8optionalISF_EENSG_ISM_EEEE+0x319) [0x7f68e0fd18a9]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17ccea9) [0x7f68e1025ea9]
--- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x3a) [0x7f68e27b210a]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5904a) [0x7f68e27b204a]
--- /tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7f68e27b1a07]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f583bb) [0x7f68e27b13bb]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5809b) [0x7f68e27b109b]
--- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>)
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x17cc6a8) [0x7f68e10256a8]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f5fa55) [0x7f68e27b8a55]
--- /tt-metal/ttnn/ttnn/_ttnn.so(+0x2f6002f) [0x7f68e27b902f]
--- /tt-metal/build/lib/libtt_metal.so(+0x190c9b) [0x7f68defc4c9b]
--- /tt-metal/build/lib/libtt_metal.so(+0x190f6b) [0x7f68defc4f6b]
--- /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6951259609]
--- /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6951393353]
s
/pluggy/_callers.py", line 103 in _multicall
File "/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/tt-metalAborted (core dumped)
Scheenshots tt-smi firmware / topology:
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of tt-metal: v0.56.0-rc33
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16 (14 by maintainers)
Thanks @skhorasganiTT, using
MAX_PREFILL_CHUNK_SIZE=16I was able to make successful requests with seqlen > 32k: https://github.com/tenstorrent/tt-inference-server/issues/127#issuecomment-3079508996We discussed offline about if this should be reflected in
model_config.py. If I measure the long-context perf and it doesn’t take a huge hit, we will change the prefill chunk size to 16 instead of 32Suggested to @bgoelTT offline to try setting
MAX_PREFILL_CHUNK_SIZE=16.{"page_block_size": 64, "page_max_num_blocks_per_dp": 2048}, while in vLLM the block size is 64 and the number of blocks is slightly higher (roughly 2%) than 2048 (seedetermine_num_available_blocksin https://github.com/tenstorrent/vllm/blob/dev/vllm/worker/tt_worker.py#L211 for details). The exact number of blocks is printed in the server logs in a format similar to[executor_base.py:112] # tt blocks: 1034, # CPU blocks: 0.MAX_PREFILL_CHUNK_SIZES_DIV1024) - https://github.com/tenstorrent/tt-metal/blob/main/models/tt_transformers/tt/model_config.py#L550. For Qwen72B-T3K it does chunks of 32k by default (i.e. the 64k demo is running two chunks of 32k). It could be the case that that is very tight and the slight number of extra blocks that vLLM allocates for the cache causes it to run out of memory, so suggested reducing max chunk size to 16k (which may reduce prefill perf for > 16k though will be necessary if it doesn’t fit and if we don’t want to change model data formats).