tt-metal: BH Whisper (P150) demo stress test crashes after running for > 2 hours

When running a stress test version of the BH Whisper demo on P150 (tested on bh-30 IRD machine), it was observed that at iteration 10340 (after ~2.5 hours), the encoder took 8 hours (instead of ~100ms) and after a couple decode iterations the device closed abruptly (see error below). Also ran the experiment a second time and it stopped on the same iteration without any error log.

Branch: skhorasgani/whisper_bh_stress Command: pytest --disable-warnings --input-path="models/demos/whisper/demo/dataset/conditional_generation" models/demos/whisper/demo/demo.py::test_demo_for_conditional_generation

stress test iters:  26%|██▌       | 10340/40000 [2:33:59<7:29:56,  1.10it/s]2025-04-15 06:56:52.781 | INFO     | models.demos.whisper.demo.demo:_model_pipeline:249 - Running model on audio data with duration 5.280s
2025-04-15 06:56:52.895 | INFO     | models.demos.whisper.demo.demo:run_generate:122 - Time to encoder states: 114.163ms


Decode inference iterations:   0%|          | 0/448 [00:00<?, ?it/s]

Decode inference iterations:   2%|▏         | 7/448 [00:00<00:07, 61.95it/s]

Decode inference iterations:   3%|▎         | 14/448 [00:00<00:07, 61.32it/s]

Decode inference iterations:   5%|▍         | 21/448 [00:00<00:06, 61.37it/s]
Decode inference iterations:   5%|▍         | 22/448 [00:00<00:07, 58.87it/s]
2025-04-15 06:56:53.270 | INFO     | models.demos.whisper.demo.demo:_run_generate:212 - Time to first token: 131.468ms
2025-04-15 06:56:53.271 | INFO     | models.demos.whisper.demo.demo:_run_generate:213 - Total decode time: 0.350s
2025-04-15 06:56:53.271 | INFO     | models.demos.whisper.demo.demo:_run_generate:214 - Total generate time: 0.489s
2025-04-15 06:56:53.271 | INFO     | models.demos.whisper.demo.demo:_run_generate:215 - Average decode throughput: 65.624 t/s/u
2025-04-15 06:56:53.271 | INFO     | models.demos.whisper.demo.demo:run_demo_whisper_for_conditional_generation_inference:404 - Model Output (Input 1):  Although three people were inside the house when the car impacted it, none of them were hurt.
2025-04-15 06:56:53.271 | INFO     | models.demos.whisper.demo.demo:_model_pipeline:249 - Running model on audio data with duration 5.220s
2025-04-15 15:29:06.494 | INFO     | models.demos.whisper.demo.demo:run_generate:122 - Time to encoder states: 30733222.604ms


Decode inference iterations:   0%|          | 0/448 [00:00<?, ?it/s]

Decode inference iterations:   2%|▏         | 10/448 [00:00<00:04, 95.31it/s]

Decode inference iterations:   3%|▎         | 12/448 [02:25<00:04, 95.31it/s]
Decode inference iterations:   3%|▎         | 12/448 [02:25<1:28:16, 12.15s/it]

stress test iters:  26%|██▌       | 10340/40000 [11:08:39<31:58:00,  3.88s/it]
FAILED                  Metal | INFO     | Disabling and clearing program cache on device 0
                  Metal | INFO     | Closing device 0
Read unexpected run_mailbox value: 0xbc (expected 0x80 or 0x0)

models/demos/whisper/demo/demo.py::test_demo_for_conditional_generation[device_params0-True-2-models.demos.whisper.tt.ttnn_optimized_functional_whisper] ERROR

==================================== ERRORS ====================================
_ ERROR at teardown of test_demo_for_conditional_generation[device_params0-True-2-models.demos.whisper.tt.ttnn_optimized_functional_whisper] _

request = <SubRequest 'device' for <Function test_demo_for_conditional_generation[device_params0-True-2-models.demos.whisper.tt.ttnn_optimized_functional_whisper]>>
device_params = {'l1_small_size': 1024}

    @pytest.fixture(scope="function")
    def device(request, device_params):
        import ttnn
    
        device_id = request.config.getoption("device_id")
        request.node.pci_ids = [ttnn.GetPCIeDeviceID(device_id)]
    
        num_devices = ttnn.GetNumPCIeDevices()
        assert device_id < num_devices, "CreateDevice not supported for non-mmio device"
        updated_device_params = get_updated_device_params(device_params)
        device = ttnn.CreateDevice(device_id=device_id, **updated_device_params)
        ttnn.SetDefaultDevice(device)
    
        yield device
    
        ttnn.DumpDeviceProfiler(device)
>       ttnn.close_device(device)

conftest.py:119: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

device = <ttnn._ttnn.device.Device object at 0x7a36fbbd4470>

    def close_device(device: "ttnn.device.Device"):
        """
        Close the device and remove it from the device cache.
    
        Args:
            device (ttnn.device.Device): The device to close.
    
        Returns:
            `None`: the device is closed.
    
        Example:
            >>> device_id = 0
            >>> device = ttnn.open_device(device_id = device_id)
            >>> success = ttnn.close_device(device)
            Closing device 0
    
        """
        synchronize_device(device)
>       ttnn._ttnn.device.close_device(device)
E       RuntimeError: TT_FATAL @ /localdev/skhorasgani/repos/metal/tt-metal/tt_metal/llrt/llrt.cpp:226: run == run_state || run == RUN_MSG_DONE
E       info:
E       Read unexpected run_mailbox value from core (x=16,y=3)
E       backtrace:
E        --- /localdev/skhorasgani/repos/metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xc63340) [0x7a3743063340]
E        --- tt::llrt::internal_::wait_until_cores_done(int, int, std::__1::unordered_set<tt::umd::xy_pair, std::__1::hash<tt::umd::xy_pair>, std::__1::equal_to<tt::umd::xy_pair>, std::__1::allocator<tt::umd::xy_pair>>&, int)
E        --- tt::tt_metal::Device::close()
E        --- tt::DevicePool::close_device(int)
E        --- /localdev/skhorasgani/repos/metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xcf8939) [0x7a37430f8939]

cc @uaydonat @ejouretTT

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 31 (30 by maintainers)

Most upvoted comments

yeah I think it can be closed, @ruizhangTT had confirmed on a branch that he wasn’t able to repro with Nigels PR

Probably the optimizations are causing the line number to be incorrect. You can try to look at the actual assembly using

build/libexec/tt-metalium/runtime/sfpi/compiler/bin/riscv32-tt-elf-objdump -dSxC < /localdev/ruizhang/.cache/tt-metal-cache/1fd98cbd9d/1052672/kernels/reader_unary_sharded_blocks_interleaved_start_id/10099691848132593013/ncrisc/ncrisc.elf > out.S and looking for offset 0x1a0 into the text section.

Alternatively, if you attach gdb to exalens then you should be able to see the exact instruction that’s running. Instructions:

  1. In exalens, do gdb start --port 6767
  2. Simultaneously, run build/libexec/tt-metalium/runtime/sfpi/compiler/bin/riscv32-tt-elf-gdb. All remaining commands are in gdb
  3. Do target extended-remote localhost:6767
  4. Doinfo os processes to find the pid associated with the core.
  5. Do attach <PID>
  6. Do add-symbol-file <filename> <base>, where the filename in this case is /localdev/ruizhang/.cache/tt-metal-cache/1fd98cbd9d/1052672/kernels/reader_unary_sharded_blocks_interleaved_start_id/10099691848132593013/ncrisc/ncrisc.elf and base is 0x11180, from the “Kernel path” and “base” columns of tt-triage, respectively
  7. Disassemble or backtrace