tt-metal: Some matmul sweep tests are failing

Describe the bug Some matmul sweep tests are failing

Issue purpose

  1. Analyze errors
  2. Fix low hanging fruit
  3. Spawn off separate issues for harder to deal with problems

To Reproduce Steps to reproduce the behaviour: Run matmul sweeps

Expected behavior All sweeps pass

Please complete the following environment information:

  • OS: Ubuntu 20.04
  • Version of software: main

Additional context

The sweep tests not listed below passed.

Tests run via e.g. python tests/ttnn/sweep_tests/run_sweeps.py --include matmul/short/matmul.py

matmul/short/matmul.py

  • 8 PCC too low failures

matmul/full/matmul_default_interleaved.py

  • 8 no space for circular buffer failures

matmul/full/matmul_default_height_sharded.py

  • 416 shape failures: Matmul program config could not be determined for given input shapes!

matmul/full/matmul_default_width_sharded.py

  • 356 shape failures
  • 7 no space for circular buffer failures
  • 464 Kt % in0_block_w == 0 assertion failures

matmul/full/matmul_default_block_sharded.py

  • 8012 shape failures

  • 990 no space for circular buffer failures

  • 280 per_core_N must be greater than 0! failures

  • 6160 Num cores along x must match provided grid size! failures

  • 392 Num total blocks must be smaller than num blocks y and num blocks x failures

  • #9449

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 21 (20 by maintainers)

Commits related to this issue

Most upvoted comments

After https://github.com/tenstorrent/tt-metal/pull/9447 block: 9712 passed, 8240 not passed

That means that PR fixed ~4k tests.

The crashes for Block Sharded can be grouped into the following. The big issue that’s left is “1D mcast for in0 or in1 is not implemented yet.” which @TT-BrianLiu is looking into.

sqlite> select substr(exception,instr(exception,"TT_FATAL @"),150), count(*) from table_146740210 where status = "crashed" and instr(exception,"TT_FATAL @")>1  group by substr(exception,instr(exception,"TT_FATAL @"),150);
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/bmm_op.cpp:1029: input_tensor_a.memory_config().memory_layout == TensorMemoryLayout::HEIGHT_SHARDED
backt|536
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/bmm_op.cpp:1212: (input_tensor_a.get_legacy_shape()[-1] / TILE_WIDTH) % program_config.in0_block_w == 0
i|448
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/bmm_op.cpp:988: input_tensor_a.memory_config().memory_layout == TensorMemoryLayout::WIDTH_SHARDED
backtra|852
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/multi_core_reuse_mcast_2d_optimized/bmm_op_multi_core_reuse_mcast_2d_optimized.cpp:1299: num_blocks_x <= |50
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/multi_core_reuse_mcast_2d_optimized/bmm_op_multi_core_reuse_mcast_2d_optimized.cpp:1301: num_blocks_y <= |50
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/multi_core_reuse_mcast_2d_optimized/bmm_op_multi_core_reuse_mcast_2d_optimized.cpp:1343: false
info:
1D m|4196
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/multi_core_reuse_mcast_2d_optimized/bmm_op_multi_core_reuse_mcast_2d_optimized.cpp:1345: false
info:
Grid|242
sqlite> select substr(exception,instr(exception,"TT_THROW @"),57), count(*) from table_146740210 where status = "crashed" and instr(exception,"TT_FATAL @")<=1  group by substr(exception,instr(exception,"TT_THROW @"),57);
TT_THROW @ ../tt_metal/impl/allocator/allocator.cpp:118: |380
TT_THROW @ ../tt_metal/impl/program/program.cpp:496: tt::|940
TT_THROW @ ../tt_metal/impl/program/program.cpp:505: tt::|546

For Width Sharded:

sqlite> select substr(exception,instr(exception,"TT_FATAL @"),150), count(*) from table_170595335 where status = "crashed" and instr(exception,"TT_FATAL @")>1  group by substr(exception,instr(exception,"TT_FATAL @"),150);
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/bmm_op.cpp:1024: program_config.out_subblock_w == per_core_N || program_config.out_subblock_h == 1
backtr|53
sqlite> select substr(exception,instr(exception,"TT_THROW @"),57), count(*) from table_170595335 where status = "crashed" and instr(exception,"TT_FATAL @")<=1  group by substr(exception,instr(exception,"TT_THROW @"),57);
TT_THROW @ ../tt_metal/impl/allocator/allocator.cpp:118: |7
TT_THROW @ ../tt_metal/impl/program/program.cpp:496: tt::|75

For Height Sharded:

sqlite> select substr(exception,instr(exception,"TT_FATAL @"),150), count(*) from table_503190626 where status = "crashed" and instr(exception,"TT_FATAL @")>1  group by substr(exception,instr(exception,"TT_FATAL @"),150);
TT_FATAL @ ../tt_eager/tt_dnn/op_library/bmm/bmm_op.cpp:1060: program_config.out_subblock_w == per_core_N || program_config.out_subblock_h == 1
backtr|40

Across the 4 full sweeps:

Interleaved: 2000 passed Width: 2848 passed, 185 failed, 135 crashed (3168 total) Height: 2392 passed, 40 crashed (2432 total) Block: 9712 passed, 8240 crashed (17952 total)

After the latest change from @TT-BrianLiu (https://github.com/tenstorrent/tt-metal/pull/9388) the number of failing/crashing tests has decreased to ~14.5k:

interleaved: 1992 passed 0 failed 8 crashed height: 2376 passed 0 failed 56 crashed width: 2732 passed 264 failed 172 crashed block: 3880 passed 56 failed 14016 crashed