tt-mlir: Llama Device Perf Failed using TTRT

When I run ttrt on some flatbuffer generated by forge-fe and compiled by ttmlir tools, I get this error:

2025-03-28 13:23:38,216 - ERROR - ERROR: test=out.ttnn experienced an error with exception=Device data mismatch: Expected 1601 but received 1372 ops on device 0. Device is showing op ID 1365 when host is showing op ID 1335
Traceback (most recent call last):
  File "/opt/ttforge-toolchain/venv/lib/python3.10/site-packages/ttrt/common/perf.py", line 486, in _execute
    process_ops(None, None, False)
  File "/opt/ttforge-toolchain/venv/lib/python3.10/site-packages/ttrt/runtime/tt_metal/tools/profiler/process_ops_logs.py", line 798, in process_ops
    deviceOps, traceOps = append_device_data(ops, traceReplays, logFolder, analyze_noc_traces)
  File "/opt/ttforge-toolchain/venv/lib/python3.10/site-packages/ttrt/runtime/tt_metal/tools/profiler/process_ops_logs.py", line 339, in append_device_data
    assert False, (
AssertionError: Device data mismatch: Expected 1601 but received 1372 ops on device 0. Device is showing op ID 1365 when host is showing op ID 1335

This flow how I compile TTIR file and then run flatbuffer using TTRT

ttrt query --save-artifacts
ttmlir-opt --tt-register-device="system-desc-path=ttrt-artifacts/system_desc.ttsys" --ttir-to-ttnn-backend-pipeline ttir_out.mlir -o ttnn.mlir
ttmlir-translate --ttnn-to-flatbuffer ttnn.mlir -o out.ttnn
ttrt perf out.ttnn

The TTIR file is given, here.

Note: TTIR has extension .txt, because I couldn’t upload the file with extension .mlir.

@tapspatel Can you, please, take a look?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (21 by maintainers)

Commits related to this issue

Most upvoted comments

@odjuricicTT @jgrimTT is looking into a workaround, else we will raise this as P0 for metal.

Sure thing @vcanicTT, I’ll take a look this morning.