tt-metal: Error: "No more available dispatch cores" on 8-chip Wormhole system running simple_text_demo
Hi Team,
We are encountering a persistent error when trying to run the multi-device transformer demo on our new-bought 8-chip system.
Environment:
- Hardware: 4x Tenstorrent n300s cards (Wormhole, 8 chips total)
- System: Using the official
tt-metalium-ubuntu-22.04-amd64-release:latest-rcDocker image. - Drivers/FW: KMD version: 1.32.0, Ethernet FW version: 6.10.0
- Topology: Configured using
tt-topology(resulting in a 2x4 grid, appears successful).
Problem:
When running the 8-device test using pytest:
pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1"
The execution fails during the device initialization phase with the following error:
terminate called after throwing an instance of 'std::runtime_error'
what(): TT_THROW @ /work/tt_metal/impl/dispatch/dispatch_core_manager.cpp:319: tt::exception
info:
No more available dispatch cores on device 0 to assign. Expand dispatch cores specified in core descriptor YAML
Troubleshooting Steps Taken:
We have experimented with different SoC descriptor YAML files(modified on “Nebula X2” settings) by setting the WH_ARCH_YAML environment variable:
wormhole_b0_80_arch_eth_dispatch.yaml: Failed with the same “No more available dispatch cores” error, likely because it only defines ethernet dispatch cores (dispatch_core_type: "ethernet").wormhole_b0_80_arch.yaml: Failed withCluster Generation Failed / Ethernet map mismatcherror, indicating an incorrect hardware topology description (e.g., compute grid size mismatch).- Custom YAML (based on
_eth_dispatch.yaml): We copied_eth_dispatch.yaml, changeddispatch_core_typeto"tensix", and progressively increased the number of valid core coordinates listed indispatch_cores(trying 2, 7, 9, and finally 16 valid cores per chip using the last two rows). Despite successfully detecting 8 chips with this approach, we still get the exact same “No more available dispatch cores on device 0 to assign” error, even with 16 cores specified.
Request:
Could you please provide guidance on the correct SoC descriptor YAML file to use for an 8-chip Nebula X2 (Wormhole B0) system with KMD 1.32.0? Alternatively, is there a specific configuration or a minimum number of dispatch cores required for this demo that we might be missing in our custom YAML attempts? Or it may not be the problem of YAML .
Any help would be greatly appreciated! Let me know if you need more detailed logs.
This full log
docker run -itd --name gjz2 --ulimit memlock=-1:-1 --shm-size=8G -v /dev/hugepages-1G:/dev/hugepages-1G -v /dev/hugepages:/dev/hugepages -v /home:/home /
--device /dev/tenstorrent --privileged registry.cn-hangzhou.aliyuncs.com/gjz-im1/tt-metalium-ubuntu-22.04-amd64-release:latest-rc
root@tqs-NF5468-M7-A0-R0-00:~# docker exec -it gjz2 bash
root@5412ee4c022f:/# cd /home/gjz/tt-metal
root@5412ee4c022f:/home/gjz/tt-metal# export PYTHONPATH=$(pwd)
root@5412ee4c022f:/home/gjz/tt-metal# export HF_MODEL=/home/models
root@5412ee4c022f:/home/gjz/tt-metal# export WH_ARCH_YAML=/home/gjz/tt-metal/tt_metal/core_descriptors/wormhole_b0_80_arch_eth_dispatch.yaml
root@5412ee4c022f:/home/gjz/tt-metal# pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1"
2025-04-14 06:17:28.580 | DEBUG | ttnn.library_tweaks:prepare_dir_as_metal_home:54 - Existing installation of 0.57.0rc60+any detected
2025-04-14 06:17:28.661 | DEBUG | ttnn:<module>:83 - Initial ttnn.CONFIG:
Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.10.12, pytest-7.2.2, pluggy-1.5.0 -- /opt/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/gjz/tt-metal, configfile: pytest.ini
plugins: dash-2.15.0, split-0.8.2, github-actions-annotate-failures-0.3.0, xdist-3.6.1, anyio-4.9.0, timeout-2.2.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
collecting ... 2025-04-14 06:17:29.564 | WARNING | models.perf.benchmarking_utils:<module>:16 - Skipping import of pydantic_models for benchmarking since not running in CI environment
Device | INFO | Opening user mode device driver
2025-04-14 06:17:29.565 | INFO | SiliconDriver - Opened PCI device 0; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.566 | INFO | SiliconDriver - Opened PCI device 1; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.567 | INFO | SiliconDriver - Opened PCI device 2; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.567 | INFO | SiliconDriver - Opened PCI device 3; KMD version: 1.32.0, IOMMU: disabled
Detecting chips (found 8)
2025-04-14 06:17:29.899 | INFO | SiliconDriver - Opened PCI device 0; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.899 | INFO | SiliconDriver - Opened PCI device 1; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.900 | INFO | SiliconDriver - Opened PCI device 2; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.901 | INFO | SiliconDriver - Opened PCI device 3; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.910 | INFO | SiliconDriver - Harvesting mask for chip 0 is 0x210 (physical layout: 0x201, logical: 0x210, simulated harvesting mask: 0x0).
2025-04-14 06:17:29.919 | INFO | SiliconDriver - Opened PCI device 0; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.991 | INFO | SiliconDriver - Harvesting mask for chip 1 is 0x48 (physical layout: 0xc0, logical: 0x48, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.001 | INFO | SiliconDriver - Opened PCI device 1; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:30.005 | INFO | SiliconDriver - Harvesting mask for chip 2 is 0x300 (physical layout: 0x5, logical: 0x300, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.014 | INFO | SiliconDriver - Opened PCI device 2; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:30.020 | INFO | SiliconDriver - Harvesting mask for chip 3 is 0x204 (physical layout: 0x21, logical: 0x204, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.029 | INFO | SiliconDriver - Opened PCI device 3; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:30.033 | INFO | SiliconDriver - Harvesting mask for chip 4 is 0x202 (physical layout: 0x9, logical: 0x202, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.043 | INFO | SiliconDriver - Harvesting mask for chip 5 is 0x280 (physical layout: 0x11, logical: 0x280, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.053 | INFO | SiliconDriver - Harvesting mask for chip 6 is 0x81 (physical layout: 0x12, logical: 0x81, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.065 | INFO | SiliconDriver - Harvesting mask for chip 7 is 0x201 (physical layout: 0x3, logical: 0x201, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.074 | INFO | SiliconDriver - Detected PCI devices: [0, 1, 2, 3]
2025-04-14 06:17:30.074 | INFO | SiliconDriver - Using local chip ids: {0, 1, 2, 3} and remote chip ids {4, 5, 6, 7}
2025-04-14 06:17:30.082 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 0)
2025-04-14 06:17:30.082 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 1)
2025-04-14 06:17:30.082 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 2)
2025-04-14 06:17:30.082 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 3)
2025-04-14 06:17:30.082 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 4)
2025-04-14 06:17:30.082 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 5)
2025-04-14 06:17:30.083 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 6)
2025-04-14 06:17:30.083 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 7)
collected 22 items / 21 deselected / 1 selected
models/tt_transformers/demo/simple_text_demo.py::test_demo_text[wormhole_b0-8-device_params0-performance-batch-1] Metal | INFO | Initializing device 0. Program cache is NOT enabled
Metal | INFO | AI CLK for device 0 is: 1000 MHz
Metal | INFO | Initializing device 1. Program cache is NOT enabled
Metal | INFO | AI CLK for device 1 is: 1000 MHz
Metal | INFO | Initializing device 2. Program cache is NOT enabled
Metal | INFO | AI CLK for device 2 is: 1000 MHz
Metal | INFO | Initializing device 3. Program cache is NOT enabled
Metal | INFO | AI CLK for device 3 is: 1000 MHz
Metal | INFO | Initializing device 4. Program cache is NOT enabled
Metal | INFO | AI CLK for device 4 is: 1000 MHz
Metal | INFO | Initializing device 5. Program cache is NOT enabled
Metal | INFO | AI CLK for device 5 is: 1000 MHz
Metal | INFO | Initializing device 6. Program cache is NOT enabled
Metal | INFO | AI CLK for device 6 is: 1000 MHz
Metal | INFO | Initializing device 7. Program cache is NOT enabled
Metal | INFO | AI CLK for device 7 is: 1000 MHz
Metal | INFO | While initializing Device 0, active ethernet core (x=22,y=16) on Device 0 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 0, active ethernet core (x=21,y=16) on Device 0 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 0, active ethernet core (x=25,y=17) on Device 0 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 0, active ethernet core (x=18,y=17) on Device 0 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 0, active ethernet core (x=22,y=17) on Device 0 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 0, active ethernet core (x=21,y=17) on Device 0 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
Metal | INFO | While initializing Device 4, active ethernet core (x=25,y=16) on Device 4 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 4, active ethernet core (x=18,y=16) on Device 4 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 4, active ethernet core (x=22,y=16) on Device 4 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 4, active ethernet core (x=21,y=16) on Device 4 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
Metal | INFO | While initializing Device 1, active ethernet core (x=22,y=16) on Device 1 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 1, active ethernet core (x=21,y=16) on Device 1 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 1, active ethernet core (x=25,y=17) on Device 1 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 1, active ethernet core (x=18,y=17) on Device 1 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 1, active ethernet core (x=22,y=17) on Device 1 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 1, active ethernet core (x=21,y=17) on Device 1 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
Metal | INFO | While initializing Device 6, active ethernet core (x=25,y=16) on Device 6 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 6, active ethernet core (x=18,y=16) on Device 6 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 6, active ethernet core (x=22,y=16) on Device 6 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 6, active ethernet core (x=21,y=16) on Device 6 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
Metal | INFO | While initializing Device 2, active ethernet core (x=21,y=17) on Device 2 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 2, active ethernet core (x=22,y=17) on Device 2 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 2, active ethernet core (x=18,y=16) on Device 2 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 2, active ethernet core (x=25,y=16) on Device 2 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 2, active ethernet core (x=25,y=17) on Device 2 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 2, active ethernet core (x=18,y=17) on Device 2 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
Metal | INFO | While initializing Device 5, active ethernet core (x=25,y=16) on Device 5 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 5, active ethernet core (x=18,y=16) on Device 5 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 5, active ethernet core (x=22,y=16) on Device 5 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 5, active ethernet core (x=21,y=16) on Device 5 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
Metal | INFO | While initializing Device 3, active ethernet core (x=21,y=17) on Device 3 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 3, active ethernet core (x=22,y=17) on Device 3 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 3, active ethernet core (x=18,y=16) on Device 3 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 3, active ethernet core (x=25,y=16) on Device 3 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 3, active ethernet core (x=25,y=17) on Device 3 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 3, active ethernet core (x=18,y=17) on Device 3 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
Metal | INFO | While initializing Device 7, active ethernet core (x=25,y=16) on Device 7 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 7, active ethernet core (x=18,y=16) on Device 7 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 7, active ethernet core (x=22,y=16) on Device 7 detected as still running, issuing exit signal.
Metal | INFO | While initializing Device 7, active ethernet core (x=21,y=16) on Device 7 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
Always | WARNING | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
terminate called after throwing an instance of 'std::runtime_error'
what(): TT_THROW @ /work/tt_metal/impl/dispatch/dispatch_core_manager.cpp:319: tt::exception
info:
No more available dispatch cores on device 0 to assign. Expand dispatch cores specified in core descriptor YAML
backtrace:
--- void tt::assert::tt_throw_impl<fmt::v11::basic_format_string<char, int const&>, int>(char const*, int, char const*, char const*, fmt::v11::basic_format_string<char, int const&> const&, int const&)
--- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x2690f3) [0x74943e18d0f3]
--- tt::tt_metal::dispatch_core_manager::mux_core(int, unsigned short, unsigned char)
--- EthRouterKernel::GenerateStaticConfigs()
--- tt::tt_metal::populate_cq_static_args(tt::tt_metal::IDevice*)
--- tt::DevicePool::initialize_active_devices() const
--- tt::DevicePool::init_firmware_on_active_devices() const
--- tt::DevicePool::initialize(std::vector<int, std::allocator<int> > const&, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::stl::Span<unsigned int const, 18446744073709551615ul>)
--- tt::tt_metal::detail::CreateDevices(std::vector<int, std::allocator<int> > const&, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, std::vector<unsigned int, std::allocator<unsigned int> > const&)
--- tt::tt_metal::distributed::MeshDevice::ScopedDevices::ScopedDevices(std::vector<int, std::allocator<int> > const&, unsigned long, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&)
--- tt::tt_metal::distributed::MeshDevice::ScopedDevices::ScopedDevices(unsigned long, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::tt_metal::distributed::MeshDeviceConfig const&)
--- tt::tt_metal::distributed::MeshDevice::create(tt::tt_metal::distributed::MeshDeviceConfig const&, unsigned long, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::stl::Span<unsigned int const, 18446744073709551615ul>)
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x18c3596) [0x7494401f5596]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x18c33ed) [0x7494401f53ed]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0xd59785) [0x74943f68b785]
--- /opt/venv/bin/python3(+0x18ae12) [0x590319545e12]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
--- /opt/venv/bin/python3(+0x198a6b) [0x590319553a6b]
--- /opt/venv/bin/python3(_PyObject_Call+0x118) [0x5903195545d8]
--- /opt/venv/bin/python3(+0x19512b) [0x59031955012b]
--- /opt/venv/bin/python3(+0x181afb) [0x59031953cafb]
--- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0xd525cb) [0x74943f6845cb]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x66f2) [0x590319536962]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(PyObject_Call+0x122) [0x590319554432]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
--- /opt/venv/bin/python3(+0x2297f5) [0x5903195e47f5]
--- /opt/venv/bin/python3(+0x18b8c9) [0x5903195468c9]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
--- /opt/venv/bin/python3(+0x198791) [0x590319553791]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59031953b9dd]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x5903195508ec]
--- /opt/venv/bin/python3(+0x29cba4) [0x590319657ba4]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x66f2) [0x590319536962]
--- /opt/venv/bin/python3(+0x198791) [0x590319553791]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x18d3) [0x590319531b43]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
--- /opt/venv/bin/python3(+0x198791) [0x590319553791]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59031953b9dd]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x5903195508ec]
--- /opt/venv/bin/python3(+0x29cba4) [0x590319657ba4]
--- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x5903195543cb]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
--- /opt/venv/bin/python3(+0x198791) [0x590319553791]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x18d3) [0x590319531b43]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x18d3) [0x590319531b43]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
--- /opt/venv/bin/python3(+0x198791) [0x590319553791]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59031953b9dd]
--- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x5903195508ec]
--- /opt/venv/bin/python3(+0x29cba4) [0x590319657ba4]
--- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x66f2) [0x590319536962]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
--- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
--- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
--- /opt/venv/bin/python3(+0x198791) [0x590319553791]
Fatal Python error: Aborted
Current thread 0x0000749506a0c1c0 (most recent call first):
File "/opt/venv/lib/python3.10/site-packages/ttnn/distributed/distributed.py", line 160 in open_mesh_device
File "/home/gjz/tt-metal/conftest.py", line 226 in mesh_device
File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 901 in call_fixture_func
File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 1129 in pytest_fixture_setup
File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 1075 in execute
File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 677 in _compute_fixture_value
File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 591 in _get_active_fixturedef
File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 569 in getfixturevalue
File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 550 in _fillfixtures
File "/opt/venv/lib/python3.10/site-packages/_pytest/python.py", line 1792 in setup
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 492 in setup
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 155 in pytest_runtest_setup
File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 260 in <lambda>
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 339 in from_call
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 125 in runtestprotocol
File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 324 in _main
File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 270 in wrap_session
File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/opt/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 167 in main
File "/opt/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 190 in console_main
File "/opt/venv/bin/pytest", line 8 in <module>
Extension modules: markupsafe._speedups, zmq.backend.cython._zmq, tornado.speedups, psutil._psutil_linux, psutil._psutil_posix, _brotli, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imaging, PIL._imagingft, yaml._yaml, regex._regex (total: 33)
Aborted (core dumped)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (11 by maintainers)
Commits related to this issue
- [skip ci] Fix Qwen 2.5 paths in LLM table to point to correct directory for older release (#21323) ### Ticket [#20561](https://github.com/tenstorrent/tt-metal/issues/20561) ### Problem description -... — committed to tenstorrent/tt-metal by skhorasganiTT 2 years ago
- # This is a combination of 751 commits. # This is the 1st commit message: Add check for correct formatting of defines # This is the commit message #2: moved file to requested location # This is th... — committed to jdiaz97/tt-metal by jdiaz97 2 years ago
@milank94 I have changed the paths for Qwen7B/72B to
v0.56.0-rc33/models/demos/llama3in the LLM table in #21323, thanks for raising this. We’ll change it back tomodels/tt_transformersonce the issue on main has been resolved.Hi @milank94,
Thanks for your suggestions!
I tried
export ARCH_NAME=wormhole_b0and got the errorRegarding
MESH_DEVICE=T3K, I haven’t tried that specific value. However, we managed to resolve the original “No more available dispatch cores” error by settingMESH_DEVICE=N300and using the originalwormhole_b0_80_arch_eth_dispatch.yamlfile (specified just by filename, not absolute path) forWH_ARCH_YAML.Here are the commands that successfully got past the dispatch core issue:
It seems setting
MESH_DEVICE=N300might trigger specific configurations for this hardware, potentially overriding or correctly interpreting the dispatch core needs even with the_eth_dispatch.yamlfile. Using the absolute path for the YAML file still resulted in the dispatch core error.Unfortunately, while this resolved the initial problem, we now encounter a new fatal error during the model’s prefill stage when running the
simple_text_demo.pywith the Qwen2.5-7B model:This appears to be an assertion failure within a Matmul operation, indicating that the inner dimension (K) of a tensor, after tiling, is not divisible by the required block width (
in0_block_w). This happens during the actual model execution, specifically within the prefill phase according to the call stack.Regarding the
tt-smioutput you asked for: just to clarify, thett-smitool itself is installed on the host machine, not directly within the Docker container where we are running thepytestcommands.However, I can provide the output from the host system as requested. Please find the screenshot attached below:
So, we’ve overcome the dispatch core hurdle (likely thanks to
MESH_DEVICE=N300), but are now blocked by this Matmul dimension incompatibility (Kt must be divisible by in0_block_w) when running the Qwen2.5-7B model on the 8-chip N300 mesh.I will paste the full running result in the below.Any insights or suggestions on how to resolve this Matmul error would be greatly appreciated!
Thanks again for your help.