tt-metal: [blackhole] intermittent hang on CommandQueueProgramFixture.TensixTestRandomizedProgram
Seen on multiple blackhole runners
yyzo-bh-gh01 :
https://github.com/tenstorrent/tt-metal/actions/runs/13159101219/job/36723384163
yyzo-bh-gh06 :
https://github.com/tenstorrent/tt-metal/actions/runs/13051947418/job/36414290972
https://github.com/tenstorrent/tt-metal/actions/runs/12881076360/job/35911467562
https://github.com/tenstorrent/tt-metal/actions/runs/12846254233/job/35821313384
(debugged and repro’d in issue https://github.com/tenstorrent/tt-metal/issues/17178)
tt-metal-ci-vm-178 :
https://github.com/tenstorrent/tt-metal/actions/runs/12981824287/job/36201252650
Hanging on CommandQueueProgramFixture.TensixTestRandomizedProgram
[----------] 1 test from CommandQueueProgramFixture
[ RUN ] CommandQueueProgramFixture.TensixTestRandomizedProgram
Metal | INFO | Initializing device 0. Program cache is NOT enabled
Device | INFO | For Blackhole hardcode AICLK to 800 MHz due to lack of ARC message support
Metal | INFO | AI CLK for device 0 is: 800 MHz
BuildKernels | INFO | GIT_COMMIT_HASH not found
Test | INFO | Using Test Seed: 0
Test | INFO | Starting compile of 100 programs now.
Test | INFO | Compiling program 1 of 100
Test | INFO | Compiling program 11 of 100
Test | INFO | Compiling program 21 of 100
Test | INFO | Compiling program 31 of 100
Test | INFO | Compiling program 41 of 100
Test | INFO | Compiling program 51 of 100
Test | INFO | Compiling program 61 of 100
Test | INFO | Compiling program 71 of 100
Test | INFO | Compiling program 81 of 100
Test | INFO | Compiling program 91 of 100
Test | INFO | Running 100 programs for cache warmup.
Test | INFO | Running 100 programs for 500 iterations now.
Test | INFO | Enqueueing 100 programs for iter: 1/500 now.
Test | INFO | Enqueueing 100 programs for iter: 51/500 now.
Test | INFO | Enqueueing 100 programs for iter: 101/500 now.
Test | INFO | Enqueueing 100 programs for iter: 151/500 now.
Test | INFO | Enqueueing 100 programs for iter: 201/500 now.
Error: The action has timed out.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (17 by maintainers)
@williamlyTT is there any additional output showing what / where / why / how it hung? I can see the
Test | INFO | Enqueueing 100 programs for iter: 401/500 now.and then the cancel. I’m assuming the runner is still connected and all, can you add some dumping on that error (dmesgmight be immensely helpful honestly) if it hits a timeout like that?no Im not, you an go ahead and re-flash it