tt-metal: BH P100a whisper_performance demo test hangs on first iteration with harvested fw+sw

https://github.com/tenstorrent/tt-metal/actions/runs/14653158459/job/41159933205#step:8:199 p100a VM FW ver 80.18.0.0 TT-KMD 1.33 Running blackhole post commit on williamly/bh-virtual branch with Almeet’s changes to enable noc translation, which is required to enable fw ver 80.18.0.0 on VMs with harvested p100a/p150a cards.

Model hangs on first iteration on a harvested p100a (not seen in p150a)

models/demos/whisper/demo/demo.py::test_demo_for_conditional_generation[device_params0-True-2-models.demos.whisper.tt.ttnn_optimized_functional_whisper]                   Metal | INFO     | Initializing device 0. Program cache is NOT enabled
           BuildKernels | INFO     | Skipping deleting built cache
                  Metal | INFO     | AI CLK for device 0 is:   1350 MHz
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2025-04-24 23:08:55.633 | WARNING  | ttnn.model_preprocessing:from_torch:529 - ttnn: model cache can be enabled by passing model_name argument to preprocess_model[_parameters] and setting env variable TTNN_CONFIG_OVERRIDES='{"enable_model_cache": true}'
2025-04-24 23:08:57.989 | INFO     | models.demos.whisper.tt.ttnn_optimized_functional_whisper:init_kv_cache:34 - Initializing KV cache with max batch size: 1 and max sequence length: 512
2025-04-24 23:08:58.041 | INFO     | models.demos.whisper.demo.demo:_model_pipeline:249 - Running model on audio data with duration 6.960s
2025-04-24 23:09:12.209 | INFO     | models.demos.whisper.demo.demo:run_generate:122 - Time to encoder states: 14168.323ms

Decode inference iterations:   0%|          | 0/448 [00:00<?, ?it/s]
Decode inference iterations:   0%|          | 1/448 [00:12<1:33:51, 12.60s/it]
Error: The action has timed out.

@skhorasganiTT can you take a look? I can provide access to a p100a VM for debug. Please re-assign if needed as well, thank you.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (20 by maintainers)

Commits related to this issue

Most upvoted comments

@abhullar-tt @ejouretTT If we remove all p100 scrappys we’ll lose all end to end testing of Whisper on BH and can no longer consider it functional. If it’s possible to keep a couple around to ensure the model does not have other regressions on BH that would be highly preferable.

@ejouretTT FYI this issue blocks swapping the 4 p100scrappy cards to p100a

it looks like the same SDPA hang signature seen in Llama hangs which is actively being debugged.

The hang looks to be in SDPA kernel and “goes away” when I run with Watcher + bbradel-19201_fence fix for SDPA. Without Watcher I still see the hang

fyi @amorrisonTT this issue has reappeared (only on Almeet’s branch though), not sure if the root cause is due to reshape but seems like #20993 possibly only dodged the hang and didn’t fix it

It wasn’t clear to me that it was caused by reshape in the prior case, see https://tenstorrent.slack.com/archives/C08JBSWSCHH/p1745345908480069

Yes I can take a look