tt-metal: Blackhole P150 VMs crash when running tests
Related Cloud Issue: https://github.com/tenstorrent/cloud/issues/4185
All VMs are running:
- kmd ver 1.31
- fw ver 80.15.2.0
- tt-smi ver 3.0.10
In CI this shows up as the following error:
The self-hosted runner: tt-metal-ci-vm-quad-01 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
Seen on all P150 VMs that have been provisioned: https://github.com/tenstorrent/tt-metal/actions/runs/14000681209/job/39217317830
The workflow runs one job at a time (serial) to prevent resource contention on the shared host where the VMs are running.
Steps to reproduce on a VM:
git clone https://github.com/tenstorrent/tt-metal.git --recurse-submodules
cd tt-metal
./create_venv.sh
source python_env/bin/activate
export TT_METAL_HOME=$(pwd)
export PYTHONPATH=$(pwd)
export ARCH_NAME=blackhole
export LD_LIBRARY_PATH=$(pwd)/build/lib
./build_metal.sh --build-tests --build-programming-examples
tt-smi-metal -r0
sleep 10
pytest tests/tt_eager/python_api_testing/unit_testing/ -xvvv --splits 7 --group 1
Example of observed crash:
(python_env) ubuntu@tt-metal-ci-vm-203:~/tt-metal$ pytest tests/tt_eager/python_api_testing/unit_testing/ -xvvv --splits 7 --group 1
2025-03-19 23:14:49.142 | DEBUG | ttnn:<module>:83 - Initial ttnn.CONFIG:
Config{cache_path=/home/ubuntu/.cache/ttnn,model_cache_path=/home/ubuntu/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
================================================================ test session starts =================================================================
platform linux -- Python 3.8.10, pytest-7.2.2, pluggy-1.5.0 -- /home/ubuntu/tt-metal/python_env/bin/python3
cachedir: .pytest_cache
rootdir: /home/ubuntu/tt-metal, configfile: pytest.ini
plugins: dash-2.15.0, anyio-4.5.2, split-0.8.2, github-actions-annotate-failures-0.3.0, timeout-2.2.0, xdist-3.6.1
timeout: 300.0s
timeout method: signal
timeout func_only: False
collecting 7392 items
[pytest-split] Splitting tests with algorithm: duration_based_chunks
[pytest-split] Running group 1/7 (estimated duration: 1082.00s)
collected 7568 items / 6486 deselected / 1082 selected
tests/tt_eager/python_api_testing/unit_testing/test_prod_all.py::test_prod[shapes0] SKIPPED (Hangs on Blackhole, see #12349)
tests/tt_eager/python_api_testing/unit_testing/test_prod_all.py::test_prod[shapes1] SKIPPED (Hangs on Blackhole, see #12349)
tests/tt_eager/python_api_testing/unit_testing/test_prod_all.py::test_prod[shapes2] SKIPPED (Hangs on Blackhole, see #12349)
tests/tt_eager/python_api_testing/unit_testing/test_prod_nc.py::test_prod_dims[0-2, 3, TILE_HEIGHT * 6 - 1, TILE_WIDTH * 7 - 1] Device | INFO | Opening user mode device driver
2025-03-19 23:14:53.381 | INFO | SiliconDriver - Opened PCI device 0; KMD version: 1.29.0, IOMMU: disabled
2025-03-19 23:14:53.538 | INFO | SiliconDriver - Opened PCI device 0; KMD version: 1.29.0, IOMMU: disabled
2025-03-19 23:14:53.693 | INFO | SiliconDriver - Harvesting mask for chip 0 is 0x0 (physical layout: 0x0, logical: 0x0, simulated harvesting mask: 0x0).
2025-03-19 23:14:53.697 | INFO | SiliconDriver - Opened PCI device 0; KMD version: 1.29.0, IOMMU: disabled
2025-03-19 23:14:53.754 | INFO | SiliconDriver - Detected PCI devices: [0]
2025-03-19 23:14:53.754 | INFO | SiliconDriver - Using local chip ids: {0} and remote chip ids {}
2025-03-19 23:14:53.773 | WARNING | SiliconDriver - init_detect_tt_device_numanodes(): Could not determine NumaNodeSet for TT device (physical_device_id: 0 pci_bus_id: 0000:05:00.0)
2025-03-19 23:14:53.773 | WARNING | SiliconDriver - Could not find NumaNodeSet for TT Device (physical_device_id: 0 pci_bus_id: 0000:05:00.0)
2025-03-19 23:14:53.885 | WARNING | SiliconDriver - bind_area_memory_nodeset(): Unable to determine TT Device to NumaNode mapping for physical_device_id: 0. Skipping membind.
2025-03-19 23:14:53.886 | WARNING | SiliconDriver - ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 0 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893).
2025-03-19 23:14:53.891 | INFO | SiliconDriver - Device: 0 Mapped iATU region 0 from 0x0 to 0x3fffffff to 0x1c0000000
Metal | INFO | Initializing device 0. Program cache is NOT enabled
BuildKernels | INFO | Skipping deleting built cache
Device | INFO | For Blackhole hardcode AICLK to 1350 MHz due to lack of ARC message support
Metal | INFO | AI CLK for device 0 is: 1350 MHz
Read from remote host 172.27.45.71: Connection reset by peer
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 33 (24 by maintainers)
@abhullar-tt @williamlyTT VMs get the devices with pcie pass-through, the VM guest is not aware of any other hardware, so the KMD enumeration will start from 0. This is unlike with containers (IRD) where the host device might be mounted into the container using same file name as it is on the host (that KMD assigned it).
The PR (https://github.com/tenstorrent/tt-kmd/pull/52) linked by SYS-1286 has been merged, can this be closed?
@williamlyTT If needed, please file a support ticket to Cloud so we can track this configuration change separately. Can you let us know which VM system should be used for this purpose as all systems are reserved for Metalium right now. Somebody from the team needs to decide which one should be converted as there are no additional cards/systems available. CC @bkeith-TT
Running the following instructions on the
quad-[01-04]VMs and the hoste03cs07to reproduce the failures.e03cs07 - ✅ ✅
Ran the tests twice on all 4 cards and was not able to reproduce the crash.
quad-01 - 04 - ❌ ❌ ❌ ❌
Tried running on the VMs one at a time with the other VMs idle and I was able to reproduce the crash. Also it seems to be random which VMs crash since I was able to get all tests to pass on one VM but the same VM also crashed on another boot.
Not sure what the next steps are for this. We need to involve @TTDRosen and other syseng folks.
Shutting down all 4 VMs to debug this on the host:
tt-metal-ci-vm-quad-[01-04]I was thinking that it could be some init issue that results in a NOC hang (i.e. undetected failed DRAM training). I assumed that this particular VM was pulled from CI but that is a really good thought and something we definitely need to confirm!
@williamlyTT could you help me get access to one of these VMs?