tt-metal: [blackhole] intermittent hang on CommandQueueProgramFixture.TensixTestRandomizedProgram

Seen on multiple blackhole runners

yyzo-bh-gh01 : https://github.com/tenstorrent/tt-metal/actions/runs/13159101219/job/36723384163

yyzo-bh-gh06 : https://github.com/tenstorrent/tt-metal/actions/runs/13051947418/job/36414290972 https://github.com/tenstorrent/tt-metal/actions/runs/12881076360/job/35911467562 https://github.com/tenstorrent/tt-metal/actions/runs/12846254233/job/35821313384 (debugged and repro’d in issue https://github.com/tenstorrent/tt-metal/issues/17178)

tt-metal-ci-vm-178 : https://github.com/tenstorrent/tt-metal/actions/runs/12981824287/job/36201252650

Hanging on CommandQueueProgramFixture.TensixTestRandomizedProgram

[----------] 1 test from CommandQueueProgramFixture
[ RUN      ] CommandQueueProgramFixture.TensixTestRandomizedProgram
                  Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                 Device | INFO     | For Blackhole hardcode AICLK to 800 MHz due to lack of ARC message support
                  Metal | INFO     | AI CLK for device 0 is:   800 MHz
           BuildKernels | INFO     | GIT_COMMIT_HASH not found
                   Test | INFO     | Using Test Seed: 0
                   Test | INFO     | Starting compile of 100 programs now.
                   Test | INFO     | Compiling program 1 of 100
                   Test | INFO     | Compiling program 11 of 100
                   Test | INFO     | Compiling program 21 of 100
                   Test | INFO     | Compiling program 31 of 100
                   Test | INFO     | Compiling program 41 of 100
                   Test | INFO     | Compiling program 51 of 100
                   Test | INFO     | Compiling program 61 of 100
                   Test | INFO     | Compiling program 71 of 100
                   Test | INFO     | Compiling program 81 of 100
                   Test | INFO     | Compiling program 91 of 100
                   Test | INFO     | Running 100 programs for cache warmup.
                   Test | INFO     | Running 100 programs for 500 iterations now.
                   Test | INFO     | Enqueueing 100 programs for iter: 1/500 now.
                   Test | INFO     | Enqueueing 100 programs for iter: 51/500 now.
                   Test | INFO     | Enqueueing 100 programs for iter: 101/500 now.
                   Test | INFO     | Enqueueing 100 programs for iter: 151/500 now.
                   Test | INFO     | Enqueueing 100 programs for iter: 201/500 now.
Error: The action has timed out.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (17 by maintainers)

Most upvoted comments

@williamlyTT is there any additional output showing what / where / why / how it hung? I can see the Test | INFO | Enqueueing 100 programs for iter: 401/500 now. and then the cancel. I’m assuming the runner is still connected and all, can you add some dumping on that error (dmesg might be immensely helpful honestly) if it hits a timeout like that?

@abhullar-tt are you using yyzo-bh-gh06? If not I can flash to new fw ver 80.15.0.0 which includes some fixes for BH and see if I can still reproduce the hang.

no Im not, you an go ahead and re-flash it