tt-mlir: Llama Device Perf Failed using TTRT
When I run ttrt on some flatbuffer generated by forge-fe and compiled by ttmlir tools, I get this error:
2025-03-28 13:23:38,216 - ERROR - ERROR: test=out.ttnn experienced an error with exception=Device data mismatch: Expected 1601 but received 1372 ops on device 0. Device is showing op ID 1365 when host is showing op ID 1335
Traceback (most recent call last):
File "/opt/ttforge-toolchain/venv/lib/python3.10/site-packages/ttrt/common/perf.py", line 486, in _execute
process_ops(None, None, False)
File "/opt/ttforge-toolchain/venv/lib/python3.10/site-packages/ttrt/runtime/tt_metal/tools/profiler/process_ops_logs.py", line 798, in process_ops
deviceOps, traceOps = append_device_data(ops, traceReplays, logFolder, analyze_noc_traces)
File "/opt/ttforge-toolchain/venv/lib/python3.10/site-packages/ttrt/runtime/tt_metal/tools/profiler/process_ops_logs.py", line 339, in append_device_data
assert False, (
AssertionError: Device data mismatch: Expected 1601 but received 1372 ops on device 0. Device is showing op ID 1365 when host is showing op ID 1335
This flow how I compile TTIR file and then run flatbuffer using TTRT
ttrt query --save-artifacts
ttmlir-opt --tt-register-device="system-desc-path=ttrt-artifacts/system_desc.ttsys" --ttir-to-ttnn-backend-pipeline ttir_out.mlir -o ttnn.mlir
ttmlir-translate --ttnn-to-flatbuffer ttnn.mlir -o out.ttnn
ttrt perf out.ttnn
The TTIR file is given, here.
Note: TTIR has extension .txt, because I couldn’t upload the file with extension .mlir.
@tapspatel Can you, please, take a look?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (21 by maintainers)
Commits related to this issue
- Enable device perf collection on CI for Llama benchmark model. (#1840) Now, we are able to run device run on Llama benchmark model, too. So, we enable it. The issue with TTRT is solved. There is the ... — committed to tenstorrent/tt-forge-fe by vcanicTT 2 years ago
- Implementing runtime API dump_device_profile_results() with workaround reshape (#3581) ### Ticket [Link to Github Issue](https://github.com/tenstorrent/tt-mlir/issues/3428) [Link to Github Issue](htt... — committed to tenstorrent/tt-mlir by jgrimTT a year ago
@odjuricicTT @jgrimTT is looking into a workaround, else we will raise this as P0 for metal.
Sure thing @vcanicTT, I’ll take a look this morning.