tt-metal: TG - Deterministic hang after device initialisation in Llama70b CCL unit tests
After execution 1 or more tests successfully it seems like Fabric is not torn properly in previous run so just after device initialisation is finished hang is observed. It happens in TG model perf pipeline. Currently it’s on a branch but I will provide lately pipeline run from latest main as well.
Here are couple of failed runs on different machines on last week main: (branch: divanovic/test_lower_dram_bw_with_choose_pipeline)
- g15cs02
- g14cs03
- g07cs01
Test where fail is observed in CI is added as well but I believe running couple of tests in sequence would reveal this bug!
tests/ttnn/unit_tests/operations/ccl/test_ccl_async_TG_llama.py::test_all_gather_tg_llama[silicon_arch_name=wormhole_b0-device_params={'trace_region_size': 17068032, 'fabric_config': <FabricConfig.FABRIC_1D: 1>}-8x4_grid-replication_factor=8-binary_mult-shard_grid_orientation=ShardOrientation.ROW_MAJOR-num_iters=75-warmup_iters=10-input_dtype=DataType.BFLOAT16-num_devices=4]
Let me know if something is missing!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (15 by maintainers)
@mo-tenstorrent @djordje-tt
Remove device-host profiler sync requirement for DRAM buffers + Make active mesh device use FD for reading/writing profiler buffers
This command is passing 3 times in a row on aus-glx-11. Main with these two PRs cherry-picked.
pytest models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ag_tg_llama_perfThat warning/info message isn’t an issue. In debug log I can see the tunnelers have exited properly. Also, this prints with/without perf collection and shows up on other systems like T3K.