tt-metal: Assertion Failure in Cluster::set_tunnels_from_mmio_device() When Two WH_B0 Cards are Installed in Host System

Describe the bug To improve development productivity, two WH_B0 cards were installed in the server so that two users can each use one WH_B0 card simultaneously for development. However, when running a program on this system, an assertion failure occurs in the Cluster::set_tunnels_from_mmio_device() function within the tt-metal runtime. (This function was merged in #9515). I would appreciate it if you could let me know if there are any settings I might have missed.

To Reproduce Steps to reproduce the behavior:

  1. Install two WH_B0 cards in the host.
  2. Build the latest version of tt-metal.
  3. Execute any test program.

Expected behavior

  • Users should be able to create and use devices corresponding to each card.

Screenshots

  • Logs of the program execution and screenshots of tt-smi are attached.
(python_env) dongjin@ttdev01:~/tt-metal$ ./build/test/tt_eager/ops/test_bmm_op
                 Device | INFO     | Opening user mode device driver
  Detecting chips (found 4)
2024-07-16 22:04:46.074 | INFO     | SiliconDriver   - Detected 2 PCI devices : [0, 1]
2024-07-16 22:04:46.158 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 0)
2024-07-16 22:04:46.165 | INFO     | SiliconDriver   - Detected 2 PCI devices : [0, 1]
2024-07-16 22:04:46.169 | WARNING  | SiliconDriver   - NumHostMemChannels: 2 used for device_id: 0x401e less than target: 3. Workload will fail if it exceeds NumHostMemChannels. Increase Number of Hugepages.
2024-07-16 22:04:46.197 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 1)
2024-07-16 22:04:46.198 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 2)
2024-07-16 22:04:46.199 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 3)
                 Always | FATAL    | Loop Exit Error.
libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_ASSERT @ /home/dongjin/tt-metal/tt_metal/llrt/tt_cluster.cpp:656: tunneled_device_hit || (it == device_ids.end())
info:
Loop Exit Error.
backtrace:
 --- void tt::assert::tt_assert<char [17]>(char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool, char const*, char const (&) [17])
 --- tt::Cluster::set_tunnels_from_mmio_device()
 --- tt::Cluster::Cluster()
 --- tt::Cluster::instance()
 --- tt::DevicePool::DevicePool(std::__1::vector<int, std::__1::allocator<int>>, unsigned char, unsigned long, unsigned long, std::__1::vector<unsigned int, std::__1::allocator<unsigned int>> const&)
 --- /home/dongjin/tt-metal/build/tt_metal/libtt_metal.so(+0x449919) [0x7fa4c4baa919]
 --- tt::tt_metal::CreateDevice(int, unsigned char, unsigned long, unsigned long, std::__1::vector<unsigned int, std::__1::allocator<unsigned int>> const&)
 --- ./build/test/tt_eager/ops/test_bmm_op(main+0x5b) [0x5636fc9295db]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fa4c3954083]
 --- ./build/test/tt_eager/ops/test_bmm_op(+0x1f4be) [0x5636fc9294be]

Aborted (core dumped)
(python_env) dongjin@ttdev01:~/tt-metal/build/test/tt_metal$ TT_METAL_SLOW_DISPATCH_MODE=1 ./unit_tests_frequent
Running main() from /home/dongjin/tt-metal/.cpmcache/googletest/96129d89f45386492ae46d6bb8c027bc3df5f949/googletest/src/gtest_main.cc
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from Common
[ RUN      ] Common.AllCoresRunManyTimes
                   Test | INFO     | Running iteration #0
                 Device | INFO     | Opening user mode device driver
  Detecting chips (found 4)
2024-07-16 22:05:43.962 | INFO     | SiliconDriver   - Detected 2 PCI devices : [0, 1]
2024-07-16 22:05:44.049 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 0)
2024-07-16 22:05:44.056 | INFO     | SiliconDriver   - Detected 2 PCI devices : [0, 1]
2024-07-16 22:05:44.060 | WARNING  | SiliconDriver   - NumHostMemChannels: 2 used for device_id: 0x401e less than target: 3. Workload will fail if it exceeds NumHostMemChannels. Increase Number of Hugepages.
2024-07-16 22:05:44.088 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 1)
2024-07-16 22:05:44.089 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 2)
2024-07-16 22:05:44.090 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 3)
                 Always | FATAL    | Loop Exit Error.
unknown file: Failure
C++ exception with description "TT_ASSERT @ /home/dongjin/tt-metal/tt_metal/llrt/tt_cluster.cpp:656: tunneled_device_hit || (it == device_ids.end())
info:
Loop Exit Error.
backtrace:
 --- void tt::assert::tt_assert<char [17]>(char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool, char const*, char const (&) [17])
 --- tt::Cluster::set_tunnels_from_mmio_device()
 --- tt::Cluster::Cluster()
 --- tt::Cluster::instance()
 --- tt::tt_metal::GetNumAvailableDevices()
 --- ./unit_tests_frequent(+0xbd44) [0x5589a093bd44]
 --- void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)
 --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)
 --- testing::Test::Run()
 --- testing::TestInfo::Run()
 --- testing::TestSuite::Run()
 --- testing::internal::UnitTestImpl::RunAllTests()
 --- bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)
 --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)
 --- testing::UnitTest::Run()
 --- RUN_ALL_TESTS()
 --- /home/dongjin/tt-metal/build/lib/libgtest_main.so.1.13.0(main+0x3d) [0x7fbc7018f18d]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fbc6facb083]
 --- ./unit_tests_frequent(+0xaa5e) [0x5589a093aa5e]
" thrown in the test body.
[  FAILED  ] Common.AllCoresRunManyTimes (258 ms)
[----------] 1 test from Common (258 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (258 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] Common.AllCoresRunManyTimes

 1 FAILED TEST
  • tt-smi Screenshot 2024-07-16 at 8 56 25 PM Screenshot 2024-07-16 at 8 56 32 PM
Screenshot 2024-07-16 at 8 56 43 PM

Please complete the following environment information:

  • OS: [e.g. Ubuntu 20.04] : Ubuntu 20.04
  • Version of software (eg. commit) :
commit 354761a4cad252a45335854fa9466ff7b73904dc (HEAD -> main, origin/main, origin/HEAD)
Author: Bharane AB <bharane.amutha@multicorewareinc.com>
Date:   Tue Jul 16 06:18:24 2024 +0000

    #9911: rebase to latest main

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 23 (20 by maintainers)

Commits related to this issue

Most upvoted comments

Dear all,

The hardware setup has been completed recently, allowing me to perform the tests. I connected two standalone N300 cards to the mainboard using PCIe slots and configured tt-topology as follows:

tt-topology -l linear

 Detected Chips: 2
 Starting flash on pcie chips to default state.
 Completed default flash for board 0: 10001451182202d
 Completed default flash for board 1: 1000145117220c7
 Sleeping for 15s ...
 Finished flashing pcie chips to default state.
 Initiating reset on chips at pcie interface: [0, 1]
 Starting PCI link reset on WH devices at PCI indices: 0, 1
 Finishing PCI link reset on WH devices at PCI indices: 0, 1
 Completed reset on 2 chips
 Detected Chips: 4
 Post reset detected : 4 chips
 Generated connection map:
 0  :  [(2, 'X')]
 1  :  [(3, 'X')]
 2  :  [(0, 'X')]
 3  :  [(1, 'X')]
 Coordinates for linear layout:  {}
 Completed coord flash for board 10001451182202d
 Completed coord flash for board 1000145117220c7
 Sleeping for 15s ...
 Finished flashing chips to generated coordinates.
 Initiating reset on chips at pcie interface: [0, 1]
 Starting PCI link reset on WH devices at PCI indices: 0, 1
 Finishing PCI link reset on WH devices at PCI indices: 0, 1
 Detected Chips: 4
 Completed reset on 4 chips

 0
 Saved json log file to /home/dongjin/tt_topology_logs/12-08-2024_20:20:24_log.json

tt-topology -ls

 Detected Chips: 4
 Devices on system:
 0: n300 L 10001451182202d - (0, 0)
 1: n300 L 1000145117220c7 - (0, 0)
 2: n300 R 10001451182202d - (1, 0)
 3: n300 R 1000145117220c7 - (1, 0)
 Configuration: Not flashed into a configuration

tt-smi

$ tt-smi
Device Information ───────────────────────────────────────────────────────────────────

 #     Bus ID     Board Type     Board ID         Coords     DRAM Trained  DRAM Speed

 0  0000:01:00.0    n300 L    10001451182202d  (0, 0, 0, 0)       Y           12G
 1  0000:06:00.0    n300 L    1000145117220c7  (0, 0, 0, 0)       Y           12G
 2      N/A         n300 R    10001451182202d  (1, 0, 0, 0)       Y           12G
 3      N/A         n300 R    1000145117220c7  (1, 0, 0, 0)       Y           12G

I ran the matmul_multi_core example from programming_examples in tt-metal (commit ID: 8ee1f823cc, 24/12/6) by changing the device_id from 0 to 3, and it worked well!

./build/programming_examples/matmul_multi_core 1
                 Device | INFO     | Opening user mode device driver
  Detecting chips (found 4)
2024-12-08 20:33:47.056 | INFO     | SiliconDriver   - Detected 2 PCI devices : [0, 1] # two devices are found
2024-12-08 20:33:47.057 | INFO     | SiliconDriver   - Opened PCI device 1; KMD version: 1.29.0, IOMMU: disabled
2024-12-08 20:33:47.086 | WARNING  | SiliconDriver   - NumHostMemChannels: 2 used for device_id: 0x401e less than target: 4. Workload will fail if it exceeds NumHostMemChannels. Increase Number of Hugepages.
2024-12-08 20:33:47.088 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.29.0, IOMMU: disabled
2024-12-08 20:33:47.089 | WARNING  | SiliconDriver   - NumHostMemChannels: 2 used for device_id: 0x401e less than target: 4. Workload will fail if it exceeds NumHostMemChannels. Increase Number of Hugepages.
2024-12-08 20:33:47.094 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 0)
2024-12-08 20:33:47.094 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 1)
2024-12-08 20:33:47.094 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 2)
2024-12-08 20:33:47.095 | INFO     | SiliconDriver   - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 3)
                  Metal | INFO     | Initializing device 1. Program cache is NOT enabled  # open device id 1
                  Metal | INFO     | AI CLK for device 1 is:   1000 MHz
                  Metal | INFO     | MMIO Device 1 : Tunnel 0 : Device 1
                  Metal | INFO     | MMIO Device 1 : Tunnel 0 : Device 3
                  Verif | INFO     | Output vector of size 409600
                  Verif | INFO     | Metalium vs Golden -- PCC = 0.99910563
                  Metal | INFO     | Closing device 1  # close device id 1
                  Metal | INFO     | Disabling and clearing program cache on device 1  
                   Test | INFO     | Test Passed
                 Device | INFO     | Closing user mode device drivers

The issue seems to be resolved so that I will close it.

Thank you all for your kind support.

UMD should be supporting multiple unconnected cards (not thoroughly tested) when this gets merged https://github.com/tenstorrent/tt-umd/pull/306. Such problems as investigated in this issue wouldn’t be seen anymore.

I will be moving the issue to tt_metal, since there is still an item on their part to properly allow users to open only a part of the the available system, which should already be available through UMD’s constructor: https://github.com/tenstorrent/tt-metal/blob/main/tt_metal/llrt/tt_cluster.cpp#L259 .

I would expect problems to arise if that is not done, and if two processes both try opening all the available devices. I’m also not completely sure if everything will work well even when that is done, UMD might still mishandle something (hugepages for example). We haven’t tested multiple hosts on same machine.

As a side question, do you guys have any plan to implement a feature similar to CUDA_VISIBLE_DEVICES environment variable?

I think this would be more of a question for tt_metal, and how would they implement selecting a subset of devices for the client. The UMD currently supports selecting a subset through the API. If there is a request to also be able to select this from ENV variable, we could do that easily.

The cluster description from .umd/cluster_desc.yaml assigns the same coordinates and ethernet connections to multiple chips.

arch: {
   0: Wormhole,
   1: Wormhole,
   2: Wormhole,
   3: Wormhole,
}

chips: {
   0: [0,0,0,0],
   1: [0,0,0,0],
   2: [1,0,0,0],
   3: [1,0,0,0],
}

ethernet_connections: [
   [{chip: 0, chan: 8}, {chip: 2, chan: 0}],
   [{chip: 0, chan: 9}, {chip: 2, chan: 1}],
   [{chip: 1, chan: 8}, {chip: 3, chan: 0}],
   [{chip: 1, chan: 9}, {chip: 3, chan: 1}],
]

chips_with_mmio: [
   0: 2,
   1: 3,
]

# harvest_mask is the bit indicating which tensix row is harvested. So bit 0 = first tensix row; bit 1 = second tensix row etc...
harvesting: {
   0: {noc_translation: true, harvest_mask: 65},
   1: {noc_translation: true, harvest_mask: 65},
   2: {noc_translation: true, harvest_mask: 513},
   3: {noc_translation: true, harvest_mask: 17},
}

# This value will be null if the boardtype is unknown, should never happen in practice but to be defensive it would be useful to throw an error on this case.
boardtype: {
   0: n300,
   1: n300,
   2: n300,
   3: n300,

I’m going to let @nhuang-tt do the rest of the investigation, but let me know if there’s any info you could use.