tt-metal: TG - Deterministic hang after device initialisation in Llama70b CCL unit tests

After execution 1 or more tests successfully it seems like Fabric is not torn properly in previous run so just after device initialisation is finished hang is observed. It happens in TG model perf pipeline. Currently it’s on a branch but I will provide lately pipeline run from latest main as well.

Here are couple of failed runs on different machines on last week main: (branch: divanovic/test_lower_dram_bw_with_choose_pipeline)

Test where fail is observed in CI is added as well but I believe running couple of tests in sequence would reveal this bug!

tests/ttnn/unit_tests/operations/ccl/test_ccl_async_TG_llama.py::test_all_gather_tg_llama[silicon_arch_name=wormhole_b0-device_params={'trace_region_size': 17068032, 'fabric_config': <FabricConfig.FABRIC_1D: 1>}-8x4_grid-replication_factor=8-binary_mult-shard_grid_orientation=ShardOrientation.ROW_MAJOR-num_iters=75-warmup_iters=10-input_dtype=DataType.BFLOAT16-num_devices=4]

Let me know if something is missing!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (15 by maintainers)

Commits related to this issue

Most upvoted comments

@mo-tenstorrent @djordje-tt

Remove device-host profiler sync requirement for DRAM buffers + Make active mesh device use FD for reading/writing profiler buffers

This command is passing 3 times in a row on aus-glx-11. Main with these two PRs cherry-picked. pytest models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ag_tg_llama_perf

PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ag_tg_llama_perf[sdpa-15-12.9]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ag_tg_llama_perf[binary_mult-15-12.54]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ag_tg_llama_perf[layernorm-15-5.4]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ar_tg_llama_perf[ff2-15-18.6]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ar_tg_llama_perf[qkv-15-11.9]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ar_tg_llama_perf[ff1-15-19.2]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_ar_tg_llama_perf[lm_head-15-61.8]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_rms_perf[rms-10-25]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_fused_all_gather_concat_perf[5-17]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_fused_all_reduce_create_heads_perf[5-14.6]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_reduce_scatter_perf[5-9.7]
PASSED models/demos/llama3_subdevices/tests/tg_perf_unit_tests/test_ccl_async_perf_TG_llama.py::test_rs_create_heads_perf[5-9.6]
======================= 12 passed in 1799.85s (0:29:59) ========================

That warning/info message isn’t an issue. In debug log I can see the tunnelers have exited properly. Also, this prints with/without perf collection and shows up on other systems like T3K.