tt-metal: Error: "No more available dispatch cores" on 8-chip Wormhole system running simple_text_demo

Hi Team,

We are encountering a persistent error when trying to run the multi-device transformer demo on our new-bought 8-chip system.

Environment:

  • Hardware: 4x Tenstorrent n300s cards (Wormhole, 8 chips total)
  • System: Using the official tt-metalium-ubuntu-22.04-amd64-release:latest-rc Docker image.
  • Drivers/FW: KMD version: 1.32.0, Ethernet FW version: 6.10.0
  • Topology: Configured using tt-topology (resulting in a 2x4 grid, appears successful).

Problem:

When running the 8-device test using pytest:

pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1"

The execution fails during the device initialization phase with the following error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_THROW @ /work/tt_metal/impl/dispatch/dispatch_core_manager.cpp:319: tt::exception
info:
No more available dispatch cores on device 0 to assign. Expand dispatch cores specified in core descriptor YAML

Troubleshooting Steps Taken:

We have experimented with different SoC descriptor YAML files(modified on “Nebula X2” settings) by setting the WH_ARCH_YAML environment variable:

  1. wormhole_b0_80_arch_eth_dispatch.yaml: Failed with the same “No more available dispatch cores” error, likely because it only defines ethernet dispatch cores (dispatch_core_type: "ethernet").
  2. wormhole_b0_80_arch.yaml: Failed with Cluster Generation Failed / Ethernet map mismatch error, indicating an incorrect hardware topology description (e.g., compute grid size mismatch).
  3. Custom YAML (based on _eth_dispatch.yaml): We copied _eth_dispatch.yaml, changed dispatch_core_type to "tensix", and progressively increased the number of valid core coordinates listed in dispatch_cores (trying 2, 7, 9, and finally 16 valid cores per chip using the last two rows). Despite successfully detecting 8 chips with this approach, we still get the exact same “No more available dispatch cores on device 0 to assign” error, even with 16 cores specified.

Request:

Could you please provide guidance on the correct SoC descriptor YAML file to use for an 8-chip Nebula X2 (Wormhole B0) system with KMD 1.32.0? Alternatively, is there a specific configuration or a minimum number of dispatch cores required for this demo that we might be missing in our custom YAML attempts? Or it may not be the problem of YAML .

Any help would be greatly appreciated! Let me know if you need more detailed logs.

This full log

docker run -itd --name gjz2  --ulimit memlock=-1:-1  --shm-size=8G  -v /dev/hugepages-1G:/dev/hugepages-1G -v /dev/hugepages:/dev/hugepages  -v /home:/home  /
--device /dev/tenstorrent  --privileged  registry.cn-hangzhou.aliyuncs.com/gjz-im1/tt-metalium-ubuntu-22.04-amd64-release:latest-rc

root@tqs-NF5468-M7-A0-R0-00:~# docker exec -it gjz2 bash
root@5412ee4c022f:/# cd /home/gjz/tt-metal
root@5412ee4c022f:/home/gjz/tt-metal# export PYTHONPATH=$(pwd)
root@5412ee4c022f:/home/gjz/tt-metal# export HF_MODEL=/home/models
root@5412ee4c022f:/home/gjz/tt-metal# export WH_ARCH_YAML=/home/gjz/tt-metal/tt_metal/core_descriptors/wormhole_b0_80_arch_eth_dispatch.yaml
root@5412ee4c022f:/home/gjz/tt-metal# pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1"
2025-04-14 06:17:28.580 | DEBUG    | ttnn.library_tweaks:prepare_dir_as_metal_home:54 - Existing installation of 0.57.0rc60+any detected
2025-04-14 06:17:28.661 | DEBUG    | ttnn:<module>:83 - Initial ttnn.CONFIG:
Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.10.12, pytest-7.2.2, pluggy-1.5.0 -- /opt/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/gjz/tt-metal, configfile: pytest.ini
plugins: dash-2.15.0, split-0.8.2, github-actions-annotate-failures-0.3.0, xdist-3.6.1, anyio-4.9.0, timeout-2.2.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
collecting ... 2025-04-14 06:17:29.564 | WARNING  | models.perf.benchmarking_utils:<module>:16 - Skipping import of pydantic_models for benchmarking since not running in CI environment
                 Device | INFO     | Opening user mode device driver
2025-04-14 06:17:29.565 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.566 | INFO     | SiliconDriver   - Opened PCI device 1; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.567 | INFO     | SiliconDriver   - Opened PCI device 2; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.567 | INFO     | SiliconDriver   - Opened PCI device 3; KMD version: 1.32.0, IOMMU: disabled
  Detecting chips (found 8)                                                                                                                                                 
2025-04-14 06:17:29.899 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.899 | INFO     | SiliconDriver   - Opened PCI device 1; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.900 | INFO     | SiliconDriver   - Opened PCI device 2; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.901 | INFO     | SiliconDriver   - Opened PCI device 3; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.910 | INFO     | SiliconDriver   - Harvesting mask for chip 0 is 0x210 (physical layout: 0x201, logical: 0x210, simulated harvesting mask: 0x0).
2025-04-14 06:17:29.919 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:29.991 | INFO     | SiliconDriver   - Harvesting mask for chip 1 is 0x48 (physical layout: 0xc0, logical: 0x48, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.001 | INFO     | SiliconDriver   - Opened PCI device 1; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:30.005 | INFO     | SiliconDriver   - Harvesting mask for chip 2 is 0x300 (physical layout: 0x5, logical: 0x300, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.014 | INFO     | SiliconDriver   - Opened PCI device 2; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:30.020 | INFO     | SiliconDriver   - Harvesting mask for chip 3 is 0x204 (physical layout: 0x21, logical: 0x204, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.029 | INFO     | SiliconDriver   - Opened PCI device 3; KMD version: 1.32.0, IOMMU: disabled
2025-04-14 06:17:30.033 | INFO     | SiliconDriver   - Harvesting mask for chip 4 is 0x202 (physical layout: 0x9, logical: 0x202, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.043 | INFO     | SiliconDriver   - Harvesting mask for chip 5 is 0x280 (physical layout: 0x11, logical: 0x280, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.053 | INFO     | SiliconDriver   - Harvesting mask for chip 6 is 0x81 (physical layout: 0x12, logical: 0x81, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.065 | INFO     | SiliconDriver   - Harvesting mask for chip 7 is 0x201 (physical layout: 0x3, logical: 0x201, simulated harvesting mask: 0x0).
2025-04-14 06:17:30.074 | INFO     | SiliconDriver   - Detected PCI devices: [0, 1, 2, 3]
2025-04-14 06:17:30.074 | INFO     | SiliconDriver   - Using local chip ids: {0, 1, 2, 3} and remote chip ids {4, 5, 6, 7}
2025-04-14 06:17:30.082 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 0)
2025-04-14 06:17:30.082 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 1)
2025-04-14 06:17:30.082 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 2)
2025-04-14 06:17:30.082 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 3)
2025-04-14 06:17:30.082 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 4)
2025-04-14 06:17:30.082 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 5)
2025-04-14 06:17:30.083 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 6)
2025-04-14 06:17:30.083 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.10.0 (Device 7)
collected 22 items / 21 deselected / 1 selected                                                                                                                            

models/tt_transformers/demo/simple_text_demo.py::test_demo_text[wormhole_b0-8-device_params0-performance-batch-1]                   Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 0 is:   1000 MHz
                  Metal | INFO     | Initializing device 1. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 1 is:   1000 MHz
                  Metal | INFO     | Initializing device 2. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 2 is:   1000 MHz
                  Metal | INFO     | Initializing device 3. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 3 is:   1000 MHz
                  Metal | INFO     | Initializing device 4. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 4 is:   1000 MHz
                  Metal | INFO     | Initializing device 5. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 5 is:   1000 MHz
                  Metal | INFO     | Initializing device 6. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 6 is:   1000 MHz
                  Metal | INFO     | Initializing device 7. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 7 is:   1000 MHz
                  Metal | INFO     | While initializing Device 0, active ethernet core (x=22,y=16) on Device 0 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 0, active ethernet core (x=21,y=16) on Device 0 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 0, active ethernet core (x=25,y=17) on Device 0 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 0, active ethernet core (x=18,y=17) on Device 0 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 0, active ethernet core (x=22,y=17) on Device 0 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 0, active ethernet core (x=21,y=17) on Device 0 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                  Metal | INFO     | While initializing Device 4, active ethernet core (x=25,y=16) on Device 4 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 4, active ethernet core (x=18,y=16) on Device 4 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 4, active ethernet core (x=22,y=16) on Device 4 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 4, active ethernet core (x=21,y=16) on Device 4 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                  Metal | INFO     | While initializing Device 1, active ethernet core (x=22,y=16) on Device 1 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 1, active ethernet core (x=21,y=16) on Device 1 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 1, active ethernet core (x=25,y=17) on Device 1 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 1, active ethernet core (x=18,y=17) on Device 1 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 1, active ethernet core (x=22,y=17) on Device 1 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 1, active ethernet core (x=21,y=17) on Device 1 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                  Metal | INFO     | While initializing Device 6, active ethernet core (x=25,y=16) on Device 6 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 6, active ethernet core (x=18,y=16) on Device 6 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 6, active ethernet core (x=22,y=16) on Device 6 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 6, active ethernet core (x=21,y=16) on Device 6 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                  Metal | INFO     | While initializing Device 2, active ethernet core (x=21,y=17) on Device 2 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 2, active ethernet core (x=22,y=17) on Device 2 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 2, active ethernet core (x=18,y=16) on Device 2 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 2, active ethernet core (x=25,y=16) on Device 2 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 2, active ethernet core (x=25,y=17) on Device 2 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 2, active ethernet core (x=18,y=17) on Device 2 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                  Metal | INFO     | While initializing Device 5, active ethernet core (x=25,y=16) on Device 5 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 5, active ethernet core (x=18,y=16) on Device 5 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 5, active ethernet core (x=22,y=16) on Device 5 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 5, active ethernet core (x=21,y=16) on Device 5 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                  Metal | INFO     | While initializing Device 3, active ethernet core (x=21,y=17) on Device 3 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 3, active ethernet core (x=22,y=17) on Device 3 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 3, active ethernet core (x=18,y=16) on Device 3 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 3, active ethernet core (x=25,y=16) on Device 3 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 3, active ethernet core (x=25,y=17) on Device 3 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 3, active ethernet core (x=18,y=17) on Device 3 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                  Metal | INFO     | While initializing Device 7, active ethernet core (x=25,y=16) on Device 7 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 7, active ethernet core (x=18,y=16) on Device 7 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 7, active ethernet core (x=22,y=16) on Device 7 detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing Device 7, active ethernet core (x=21,y=16) on Device 7 detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_THROW @ /work/tt_metal/impl/dispatch/dispatch_core_manager.cpp:319: tt::exception
info:
No more available dispatch cores on device 0 to assign. Expand dispatch cores specified in core descriptor YAML
backtrace:
 --- void tt::assert::tt_throw_impl<fmt::v11::basic_format_string<char, int const&>, int>(char const*, int, char const*, char const*, fmt::v11::basic_format_string<char, int const&> const&, int const&)
 --- /opt/venv/lib/python3.10/site-packages/ttnn/build/lib/libtt_metal.so(+0x2690f3) [0x74943e18d0f3]
 --- tt::tt_metal::dispatch_core_manager::mux_core(int, unsigned short, unsigned char)
 --- EthRouterKernel::GenerateStaticConfigs()
 --- tt::tt_metal::populate_cq_static_args(tt::tt_metal::IDevice*)
 --- tt::DevicePool::initialize_active_devices() const
 --- tt::DevicePool::init_firmware_on_active_devices() const
 --- tt::DevicePool::initialize(std::vector<int, std::allocator<int> > const&, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::stl::Span<unsigned int const, 18446744073709551615ul>)
 --- tt::tt_metal::detail::CreateDevices(std::vector<int, std::allocator<int> > const&, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, std::vector<unsigned int, std::allocator<unsigned int> > const&)
 --- tt::tt_metal::distributed::MeshDevice::ScopedDevices::ScopedDevices(std::vector<int, std::allocator<int> > const&, unsigned long, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&)
 --- tt::tt_metal::distributed::MeshDevice::ScopedDevices::ScopedDevices(unsigned long, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::tt_metal::distributed::MeshDeviceConfig const&)
 --- tt::tt_metal::distributed::MeshDevice::create(tt::tt_metal::distributed::MeshDeviceConfig const&, unsigned long, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::stl::Span<unsigned int const, 18446744073709551615ul>)
 --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x18c3596) [0x7494401f5596]
 --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0x18c33ed) [0x7494401f53ed]
 --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0xd59785) [0x74943f68b785]
 --- /opt/venv/bin/python3(+0x18ae12) [0x590319545e12]
 --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
 --- /opt/venv/bin/python3(+0x198a6b) [0x590319553a6b]
 --- /opt/venv/bin/python3(_PyObject_Call+0x118) [0x5903195545d8]
 --- /opt/venv/bin/python3(+0x19512b) [0x59031955012b]
 --- /opt/venv/bin/python3(+0x181afb) [0x59031953cafb]
 --- /opt/venv/lib/python3.10/site-packages/ttnn/_ttnn.cpython-310-x86_64-linux-gnu.so(+0xd525cb) [0x74943f6845cb]
 --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x66f2) [0x590319536962]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(PyObject_Call+0x122) [0x590319554432]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
 --- /opt/venv/bin/python3(+0x2297f5) [0x5903195e47f5]
 --- /opt/venv/bin/python3(+0x18b8c9) [0x5903195468c9]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
 --- /opt/venv/bin/python3(+0x198791) [0x590319553791]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59031953b9dd]
 --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x5903195508ec]
 --- /opt/venv/bin/python3(+0x29cba4) [0x590319657ba4]
 --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x66f2) [0x590319536962]
 --- /opt/venv/bin/python3(+0x198791) [0x590319553791]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x18d3) [0x590319531b43]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x804) [0x590319530a74]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
 --- /opt/venv/bin/python3(+0x198791) [0x590319553791]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59031953b9dd]
 --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x5903195508ec]
 --- /opt/venv/bin/python3(+0x29cba4) [0x590319657ba4]
 --- /opt/venv/bin/python3(PyObject_Call+0xbb) [0x5903195543cb]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
 --- /opt/venv/bin/python3(+0x198791) [0x590319553791]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x18d3) [0x590319531b43]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x6bf) [0x59031953092f]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x18d3) [0x590319531b43]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
 --- /opt/venv/bin/python3(+0x198791) [0x590319553791]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyObject_FastCallDictTstate+0x16d) [0x59031953b9dd]
 --- /opt/venv/bin/python3(_PyObject_Call_Prepend+0x5c) [0x5903195508ec]
 --- /opt/venv/bin/python3(+0x29cba4) [0x590319657ba4]
 --- /opt/venv/bin/python3(_PyObject_MakeTpCall+0x25b) [0x59031953c75b]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x66f2) [0x590319536962]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x2a83) [0x590319532cf3]
 --- /opt/venv/bin/python3(_PyFunction_Vectorcall+0x7c) [0x59031954666c]
 --- /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0x5642) [0x5903195358b2]
 --- /opt/venv/bin/python3(+0x198791) [0x590319553791]

Fatal Python error: Aborted

Current thread 0x0000749506a0c1c0 (most recent call first):
  File "/opt/venv/lib/python3.10/site-packages/ttnn/distributed/distributed.py", line 160 in open_mesh_device
  File "/home/gjz/tt-metal/conftest.py", line 226 in mesh_device
  File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 901 in call_fixture_func
  File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 1129 in pytest_fixture_setup
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 1075 in execute
  File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 677 in _compute_fixture_value
  File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 591 in _get_active_fixturedef
  File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 569 in getfixturevalue
  File "/opt/venv/lib/python3.10/site-packages/_pytest/fixtures.py", line 550 in _fillfixtures
  File "/opt/venv/lib/python3.10/site-packages/_pytest/python.py", line 1792 in setup
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 492 in setup
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 155 in pytest_runtest_setup
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 339 in from_call
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 125 in runtestprotocol
  File "/opt/venv/lib/python3.10/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 324 in _main
  File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/opt/venv/lib/python3.10/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/opt/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/opt/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/opt/venv/bin/pytest", line 8 in <module>

Extension modules: markupsafe._speedups, zmq.backend.cython._zmq, tornado.speedups, psutil._psutil_linux, psutil._psutil_posix, _brotli, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imaging, PIL._imagingft, yaml._yaml, regex._regex (total: 33)
Aborted (core dumped)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@milank94 I have changed the paths for Qwen7B/72B to v0.56.0-rc33/models/demos/llama3 in the LLM table in #21323, thanks for raising this. We’ll change it back to models/tt_transformers once the issue on main has been resolved.

Hi @milank94,

Thanks for your suggestions!

I tried export ARCH_NAME=wormhole_b0 and got the error

FAILED models/tt_transformers/demo/simple_text_demo.py::test_demo_text[wormhole_b0-8-device_params0-performance-batch-1] - AssertionError: Qwen2.5-7B is only supported on 2 or 4 devices, run on an N300 or use MESH_DEVICE=N150x4

Regarding MESH_DEVICE=T3K, I haven’t tried that specific value. However, we managed to resolve the original “No more available dispatch cores” error by setting MESH_DEVICE=N300 and using the original wormhole_b0_80_arch_eth_dispatch.yaml file (specified just by filename, not absolute path) for WH_ARCH_YAML.

Here are the commands that successfully got past the dispatch core issue:

cd /home/gjz/tt-metal
export PYTHONPATH=$(pwd)
export MESH_DEVICE=N300 # Key addition
export HF_MODEL=/home/models/Qwen/Qwen2.5-7B-Instruct
export WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml # Using filename only
pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1"

It seems setting MESH_DEVICE=N300 might trigger specific configurations for this hardware, potentially overriding or correctly interpreting the dispatch core needs even with the _eth_dispatch.yaml file. Using the absolute path for the YAML file still resulted in the dispatch core error.

Unfortunately, while this resolved the initial problem, we now encounter a new fatal error during the model’s prefill stage when running the simple_text_demo.py with the Qwen2.5-7B model:

terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_FATAL @ /work/ttnn/cpp/ttnn/operations/matmul/device/matmul_op.cpp:1991: (input_tensor_a.get_padded_shape()[-1] / in0_tile_shape[1]) % program_config.in0_block_w == 0
info:
Kt must be divisible by in0_block_w

This appears to be an assertion failure within a Matmul operation, indicating that the inner dimension (K) of a tensor, after tiling, is not divisible by the required block width (in0_block_w). This happens during the actual model execution, specifically within the prefill phase according to the call stack.

Regarding the tt-smi output you asked for: just to clarify, the tt-smi tool itself is installed on the host machine, not directly within the Docker container where we are running the pytest commands.

However, I can provide the output from the host system as requested. Please find the screenshot attached below:

Image

So, we’ve overcome the dispatch core hurdle (likely thanks to MESH_DEVICE=N300), but are now blocked by this Matmul dimension incompatibility (Kt must be divisible by in0_block_w) when running the Qwen2.5-7B model on the 8-chip N300 mesh.I will paste the full running result in the below.

Any insights or suggestions on how to resolve this Matmul error would be greatly appreciated!

Thanks again for your help.