tt-metal: All Gather test hangs after many runs in loop

This hang was discovered when trying a potential workaround (disabling program caching) for #6363.

snijjar/issue-6363

Disable program caching

The following config hangs reliably after about 120 loop iterations

tests/tt_eager/python_api_testing/unit_testing/misc/test_all_gather.py::test_all_gather_on_t3000_post_commit_looping[500-mem_config1-input_dtype0-8-1-input_shape2-3-layout2] (you may need to manually update the num_iters argument parametrization to support 500)

This hang was seen during a local run of the post-commit test cases: pytest tests/tt_eager/python_api_testing/unit_testing/misc/test_all_gather.py -k "post_commit"

It may be reproducible with the standalone test command listed first.

Enabling watcher avoids the hang.

Note on priority. There are three concerns with this:

• This isn’t strictly a blocker for merging this development branch into maiin as I can artificially reduce the number of loop iterations we run for the all-gather test
• This has a reasonable probability of hitting in real workloads given back to back commands being run in sequence
• There is a “workaround” but I don’t think it’s one we can reliably apply. It doesn’t hang with watcher enabled but we’ll then want to enable watcher on all models until this is fixed (or maybe just conditionally on hang), hypothetically.

UPDATE:

Bidirectional all-gather support was temporarily disabled on main. For testing with fast-dispatch 2, please re-enable this feature. You can revert the line change in tt_eager/tt_dnn/op_library/all_gather/all_gather_op.hpp from commit b989c269a64c6d40a7523a94f3c4ba16b1eefd20 or ask me for assistance, before retesting.