tt-metal: Blackhole quietbox qb-gh01 error - Did not find edge with expected number of East-West chips 2
Seen in most tests running on the baremetal host: https://github.com/tenstorrent/tt-metal/actions/runs/15233245337/job/42843785876
E.g. https://github.com/tenstorrent/tt-metal/actions/runs/15233245337/job/42843785876#step:4:2089
terminate called after throwing an instance of 'std::runtime_error'
what(): TT_THROW @ /work/tests/tt_metal/tt_metal/test_compile_sets_kernel_binaries.cpp:327: tt::exception
Test | ERROR | TT_FATAL @ /work/tt_metal/fabric/control_plane.cpp:309: physical_chip_ids.size() == mesh_ew_size
info:
Test Failed
info:
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_metal/test_compile_sets_kernel_binaries(+0x11908) [0x55e167618908]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_metal/test_compile_sets_kernel_binaries(+0x1025b) [0x55e16761725b]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb6ad6d6d90]
--- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fb6ad6d6e40]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_metal/test_compile_sets_kernel_binaries(+0xc825) [0x55e167613825]
Did not find edge with expected number of East-West chips 2
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x191fa0) [0x7fb6ae307fa0]
--- tt::tt_fabric::ControlPlane::get_mesh_physical_chip_ids(unsigned int, unsigned int, int) const
--- tt::tt_fabric::ControlPlane::get_physical_chip_mapping_from_mesh_graph_desc_file(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
--- tt::tt_fabric::ControlPlane::ControlPlane(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
--- tt::Cluster::initialize_control_plane()
--- tt::Cluster::get_control_plane()
--- tt::tt_metal::CreateDataMovementKernel(tt::tt_metal::Program&, tt::tt_metal::KernelSource const&, CoreRangeSet const&, tt::tt_metal::DataMovementConfig const&)
--- tt::tt_metal::CreateKernel(tt::tt_metal::Program&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::variant<tt::umd::xy_pair, CoreRange, CoreRangeSet> const&, std::variant<tt::tt_metal::DataMovementConfig, tt::tt_metal::ComputeConfig, tt::tt_metal::EthernetConfig> const&)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_metal/test_compile_sets_kernel_binaries(+0xd92e) [0x55e16761492e]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_metal/test_compile_sets_kernel_binaries(+0xe452) [0x55e167615452]
--- /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb6ad6d6d90]
--- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fb6ad6d6e40]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_metal/test_compile_sets_kernel_binaries(+0xc825) [0x55e167613825]
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17 (15 by maintainers)
It seems like chips 0 and 2 have one less link than expected (7 vs 8). I think this might be causing downstream Control Plane issues.
FWIW I also see the octo-bh assigned to me have different number of links come up. This isn’t seen on bh-31 which is 4 chip networked P150 on IRD (links here are stable and the correct number come up).
FYI @bkeith-TT
Querying size for a host channel that does not exist.this looks like some hugepage set up on the machine.I think we expect 8 links to be up per chip. not sure why they aren’t all up. Could you see if they come up after multiple retries? and also confirm with syseng the number of expected links up