tt-metal: BH P100a whisper_performance demo test hangs on first iteration with harvested fw+sw
https://github.com/tenstorrent/tt-metal/actions/runs/14653158459/job/41159933205#step:8:199
p100a VM
FW ver 80.18.0.0
TT-KMD 1.33
Running blackhole post commit on williamly/bh-virtual branch with Almeet’s changes to enable noc translation, which is required to enable fw ver 80.18.0.0 on VMs with harvested p100a/p150a cards.
Model hangs on first iteration on a harvested p100a (not seen in p150a)
models/demos/whisper/demo/demo.py::test_demo_for_conditional_generation[device_params0-True-2-models.demos.whisper.tt.ttnn_optimized_functional_whisper] Metal | INFO | Initializing device 0. Program cache is NOT enabled
BuildKernels | INFO | Skipping deleting built cache
Metal | INFO | AI CLK for device 0 is: 1350 MHz
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2025-04-24 23:08:55.633 | WARNING | ttnn.model_preprocessing:from_torch:529 - ttnn: model cache can be enabled by passing model_name argument to preprocess_model[_parameters] and setting env variable TTNN_CONFIG_OVERRIDES='{"enable_model_cache": true}'
2025-04-24 23:08:57.989 | INFO | models.demos.whisper.tt.ttnn_optimized_functional_whisper:init_kv_cache:34 - Initializing KV cache with max batch size: 1 and max sequence length: 512
2025-04-24 23:08:58.041 | INFO | models.demos.whisper.demo.demo:_model_pipeline:249 - Running model on audio data with duration 6.960s
2025-04-24 23:09:12.209 | INFO | models.demos.whisper.demo.demo:run_generate:122 - Time to encoder states: 14168.323ms
Decode inference iterations: 0%| | 0/448 [00:00<?, ?it/s]
Decode inference iterations: 0%| | 1/448 [00:12<1:33:51, 12.60s/it]
Error: The action has timed out.
@skhorasganiTT can you take a look? I can provide access to a p100a VM for debug. Please re-assign if needed as well, thank you.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (20 by maintainers)
Commits related to this issue
- Add cache invalidations for whisper on p100a (#22001) ### Ticket https://github.com/tenstorrent/tt-metal/issues/21210 ### Problem description Whisper on P100A was hanging on top of f74db1f8d545a77ce... — committed to tenstorrent/tt-metal by abhullar-tt 2 years ago
- #21390: [skip ci] Run blackhole demo tests on p100a cards (#21790) ### Ticket Resolves https://github.com/tenstorrent/tt-metal/issues/21390 and resolves https://github.com/tenstorrent/tt-metal/issues... — committed to tenstorrent/tt-metal by williamlyTT 2 years ago
Fix for this on https://github.com/tenstorrent/tt-metal/pull/21670
https://github.com/tenstorrent/tt-metal/actions/runs/14849914967 (ran on P100a)
@abhullar-tt @ejouretTT If we remove all p100 scrappys we’ll lose all end to end testing of Whisper on BH and can no longer consider it functional. If it’s possible to keep a couple around to ensure the model does not have other regressions on BH that would be highly preferable.
@ejouretTT FYI this issue blocks swapping the 4 p100scrappy cards to p100a
it looks like the same SDPA hang signature seen in Llama hangs which is actively being debugged.
The hang looks to be in SDPA kernel and “goes away” when I run with Watcher + bbradel-19201_fence fix for SDPA. Without Watcher I still see the hang
It wasn’t clear to me that it was caused by reshape in the prior case, see https://tenstorrent.slack.com/archives/C08JBSWSCHH/p1745345908480069
Yes I can take a look