tt-metal: tt-metal v0.59 East-West Bug

Describe the bug

I recently moved to metal v0.59 and my 4 n300s are grouped by: (2xn300 with 2 tt-fly bridge) x 2. They do NOT have qsfp cables between them. When i try to run programming examples it crashes. I tried with v0.56 and it works

[2025-05-30 19:17:18.663] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 1; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.667] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 2; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.672] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 3; KMD version: 1.33.0, IOMMU: disabled
  Detecting chips (found 8)                                                                                                                                       
                 Device | INFO     | Opening user mode device driver
[2025-05-30 19:17:18.950] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 0; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.954] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 1; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.958] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 2; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.962] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 3; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.982] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 0; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.986] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 1; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.990] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 2; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:18.994] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 3; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:19.015] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 3 is 0x208 (physical layout: 0x81, logical: 0x208, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.025] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 3; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:19.172] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 2 is 0x201 (physical layout: 0x3, logical: 0x201, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.182] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 2; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:19.190] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 1 is 0x240 (physical layout: 0x41, logical: 0x240, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.200] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 1; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:19.209] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 0 is 0x201 (physical layout: 0x3, logical: 0x201, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.218] [info] [pci_device.cpp:191] [SiliconDriver] Opened PCI device 0; KMD version: 1.33.0, IOMMU: disabled
[2025-05-30 19:17:19.227] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 7 is 0x202 (physical layout: 0x9, logical: 0x202, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.237] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 6 is 0x201 (physical layout: 0x3, logical: 0x201, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.247] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 5 is 0x202 (physical layout: 0x9, logical: 0x202, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.257] [info] [cluster.cpp:272] [SiliconDriver] Harvesting mask for chip 4 is 0x240 (physical layout: 0x41, logical: 0x240, simulated harvesting mask: 0x0).
[2025-05-30 19:17:19.266] [info] [cluster.cpp:147] [SiliconDriver] Opening local chip ids/pci ids: {0, 1, 2, 3}/[0, 1, 2, 3] and remote chip ids {4, 5, 6, 7}
[2025-05-30 19:17:19.322] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 0)
[2025-05-30 19:17:19.322] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 1)
[2025-05-30 19:17:19.322] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 2)
[2025-05-30 19:17:19.322] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 3)
[2025-05-30 19:17:19.323] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 4)
[2025-05-30 19:17:19.324] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 5)
[2025-05-30 19:17:19.324] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 6)
[2025-05-30 19:17:19.325] [info] [cluster.cpp:1013] [SiliconDriver] Software version 6.0.0, Ethernet FW version 6.14.0 (Device 7)
                  Metal | INFO     | AI CLK for device 0 is:   1000 MHz
           BuildKernels | INFO     | Skipping deleting built cache
                  Metal | INFO     | AI CLK for device 1 is:   1000 MHz
                  Metal | INFO     | AI CLK for device 2 is:   1000 MHz
                  Metal | INFO     | AI CLK for device 3 is:   1000 MHz
                  Metal | INFO     | AI CLK for device 4 is:   1000 MHz
                  Metal | INFO     | AI CLK for device 5 is:   1000 MHz
                  Metal | INFO     | AI CLK for device 6 is:   1000 MHz
                  Metal | INFO     | AI CLK for device 7 is:   1000 MHz
                  Metal | WARNING  | Opening subset of mmio devices slows down UMD read/write to remote chips. If opening more devices, consider using CreateDevices API.
                  Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                  Metal | INFO     | While initializing device 0, active ethernet dispatch core (x=18,y=17) detected as still running, issuing exit signal.
                  Metal | INFO     | While initializing device 0, active ethernet dispatch core (x=25,y=17) detected as still running, issuing exit signal.
Read unexpected run_mailbox value: 0x40 (expected 0x80 or 0x0)
                 Always | WARNING  | Detected dispatch kernels still running but failed to complete an early exit. This may happen from time to time following a reset, continuing to FW intialization...
                   Test | ERROR    | Test failed with exception!
                   Test | ERROR    | TT_FATAL @ /home/howard/tt-metal/tt_metal/fabric/control_plane.cpp:309: physical_chip_ids.size() == mesh_ew_size
info:
Did not find edge with expected number of East-West chips 4
backtrace:
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x483a0f) [0x7f2b56f07a0f]
 --- tt::tt_fabric::ControlPlane::get_mesh_physical_chip_ids(unsigned int, unsigned int, int) const
 --- tt::tt_fabric::ControlPlane::get_physical_chip_mapping_from_mesh_graph_desc_file(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
 --- tt::tt_fabric::ControlPlane::ControlPlane(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x96f338) [0x7f2b573f3338]
 --- tt::Cluster::initialize_control_plane()
 --- tt::Cluster::get_control_plane()
 --- tt::tt_metal::CreateDataMovementKernel(tt::tt_metal::Program&, tt::tt_metal::KernelSource const&, CoreRangeSet const&, tt::tt_metal::DataMovementConfig const&)
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebede) [0x7f2b5756fede]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebe6d) [0x7f2b5756fe6d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebe2d) [0x7f2b5756fe2d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebded) [0x7f2b5756fded]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebd0d) [0x7f2b5756fd0d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebccb) [0x7f2b5756fccb]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebc2d) [0x7f2b5756fc2d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebbfa) [0x7f2b5756fbfa]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xae8226) [0x7f2b5756c226]
 --- tt::tt_metal::CreateKernel(tt::tt_metal::Program&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::variant<tt::umd::xy_pair, CoreRange, CoreRangeSet> const&, std::__1::variant<tt::tt_metal::DataMovementConfig, tt::tt_metal::ComputeConfig, tt::tt_metal::EthernetConfig> const&)
 --- tt::tt_metal::FDKernel::configure_kernel_variant(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int>> const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>, bool, bool, bool, tt::tt_metal::KernelBuildOptLevel)
 --- tt::tt_metal::PrefetchKernel::CreateKernel()
 --- tt::tt_metal::create_and_compile_cq_program(tt::tt_metal::IDevice*)
 --- tt::tt_metal::Device::compile_command_queue_programs()
 --- tt::tt_metal::Device::init_command_queue_device()
 --- tt::DevicePool::initialize_active_devices() const
 --- tt::DevicePool::initialize_fabric_and_dispatch_fw() const
 --- tt::DevicePool::init_firmware_on_active_devices() const
 --- tt::DevicePool::initialize(std::__1::vector<int, std::__1::allocator<int>> const&, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::stl::Span<unsigned int const, 18446744073709551615ul>, unsigned long, bool, bool, bool)
 --- tt::tt_metal::CreateDevice(int, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int>> const&, unsigned long)
 --- ./build/programming_examples/eltwise_sfpu(+0x218b1) [0x55d426a098b1]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2b55ee2d90]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f2b55ee2e40]
 --- ./build/programming_examples/eltwise_sfpu(+0x21735) [0x55d426a09735]

libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_FATAL @ /home/howard/tt-metal/tt_metal/fabric/control_plane.cpp:309: physical_chip_ids.size() == mesh_ew_size
info:
Did not find edge with expected number of East-West chips 4
backtrace:
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x483a0f) [0x7f2b56f07a0f]
 --- tt::tt_fabric::ControlPlane::get_mesh_physical_chip_ids(unsigned int, unsigned int, int) const
 --- tt::tt_fabric::ControlPlane::get_physical_chip_mapping_from_mesh_graph_desc_file(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
 --- tt::tt_fabric::ControlPlane::ControlPlane(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0x96f338) [0x7f2b573f3338]
 --- tt::Cluster::initialize_control_plane()
 --- tt::Cluster::get_control_plane()
 --- tt::tt_metal::CreateDataMovementKernel(tt::tt_metal::Program&, tt::tt_metal::KernelSource const&, CoreRangeSet const&, tt::tt_metal::DataMovementConfig const&)
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebede) [0x7f2b5756fede]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebe6d) [0x7f2b5756fe6d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebe2d) [0x7f2b5756fe2d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebded) [0x7f2b5756fded]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebd0d) [0x7f2b5756fd0d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebccb) [0x7f2b5756fccb]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebc2d) [0x7f2b5756fc2d]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xaebbfa) [0x7f2b5756fbfa]
 --- /home/howard/tt-metal/build_Debug/tt_metal/libtt_metal.so(+0xae8226) [0x7f2b5756c226]
 --- tt::tt_metal::CreateKernel(tt::tt_metal::Program&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::variant<tt::umd::xy_pair, CoreRange, CoreRangeSet> const&, std::__1::variant<tt::tt_metal::DataMovementConfig, tt::tt_metal::ComputeConfig, tt::tt_metal::EthernetConfig> const&)
 --- tt::tt_metal::FDKernel::configure_kernel_variant(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int>> const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>, bool, bool, bool, tt::tt_metal::KernelBuildOptLevel)
 --- tt::tt_metal::PrefetchKernel::CreateKernel()
 --- tt::tt_metal::create_and_compile_cq_program(tt::tt_metal::IDevice*)
 --- tt::tt_metal::Device::compile_command_queue_programs()
 --- tt::tt_metal::Device::init_command_queue_device()
 --- tt::DevicePool::initialize_active_devices() const
 --- tt::DevicePool::initialize_fabric_and_dispatch_fw() const
 --- tt::DevicePool::init_firmware_on_active_devices() const
 --- tt::DevicePool::initialize(std::__1::vector<int, std::__1::allocator<int>> const&, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, tt::stl::Span<unsigned int const, 18446744073709551615ul>, unsigned long, bool, bool, bool)
 --- tt::tt_metal::CreateDevice(int, unsigned char, unsigned long, unsigned long, tt::tt_metal::DispatchCoreConfig const&, std::__1::vector<unsigned int, std::__1::allocator<unsigned int>> const&, unsigned long)
 --- ./build/programming_examples/eltwise_sfpu(+0x218b1) [0x55d426a098b1]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2b55ee2d90]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f2b55ee2e40]
 --- ./build/programming_examples/eltwise_sfpu(+0x21735) [0x55d426a09735]

[1]    294647 IOT instruction (core dumped)  ./build/programming_examples/eltwise_sfpu```

### Steps to reproduce the issue

Remove qsfp cables from loudbox setup and run programming examples with v0.59. Ubuntu 20.04 amd64

### Expected behavior

_No response_

### Please complete the following environment information

_No response_

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 41 (40 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks for the log! It looks like I was able to reproduce a similar error/log running on a 2 N300 system, so I will continue debugging there.

@daminakaTT I can take over looking at this since it’s related to control plane, and we’ve had some discussions about how to handle custom topologies.

@csirak just to get a bit more info about your use case, the current error is related to trying to just use a single device, which we have some proposed workarounds/solutions for, but will you also be expecting to run ccls/tt-fabric on this device configuration at a later time?