tt-metal: Some matmul sweep tests are failing
Describe the bug Some matmul sweep tests are failing
Issue purpose
- Analyze errors
- Fix low hanging fruit
- Spawn off separate issues for harder to deal with problems
To Reproduce Steps to reproduce the behaviour: Run matmul sweeps
Expected behavior All sweeps pass
Please complete the following environment information:
- OS: Ubuntu 20.04
- Version of software: main
Additional context
The sweep tests not listed below passed.
Tests run via e.g. python tests/ttnn/sweep_tests/run_sweeps.py --include matmul/short/matmul.py
matmul/short/matmul.py
- 8 PCC too low failures
matmul/full/matmul_default_interleaved.py
- 8 no space for circular buffer failures
matmul/full/matmul_default_height_sharded.py
- 416 shape failures:
Matmul program config could not be determined for given input shapes!
matmul/full/matmul_default_width_sharded.py
- 356 shape failures
- 7 no space for circular buffer failures
- 464
Kt % in0_block_w == 0assertion failures
matmul/full/matmul_default_block_sharded.py
-
8012 shape failures
-
990 no space for circular buffer failures
-
280
per_core_N must be greater than 0!failures -
6160
Num cores along x must match provided grid size!failures -
392
Num total blocks must be smaller than num blocks y and num blocks xfailures
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 21 (20 by maintainers)
Commits related to this issue
- #9059: Update matmul test pcc — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Update matmul test pcc — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Update matmul test pcc — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Update matmul test pcc (#9061) — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Add fallback for getting matmul program config — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Add fallback for getting matmul program config — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Add fallback for getting matmul program config (#9077) — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: adjust matmul parameters for rounding up in some scenarios — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: adjust matmul parameters for rounding up in some scenarios — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Adjust some matmul parameters to use div_up — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: adjust matmul parameters for rounding up in some scenarios — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Adjust some matmul parameters to use div_up — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: adjust matmul parameters for rounding up in some scenarios — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Adjust some matmul parameters to use div_up — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: adjust matmul parameters for rounding up in some scenarios (#9105) * #9059: adjust matmul parameters for rounding up in some scenarios * #9059: Adjust some matmul parameters to use div_up — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Fix matmul for single core grid — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Adjust width matmul sweep pcc and add comment to kernel — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Use in0_mcast_num_cores for matmul kernel test and adjust width sweep pcc condition — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Fix matmul for single core grid (#9341) * #9059: Fix matmul for single core grid * #9059: Adjust width matmul sweep pcc and add comment to kernel * #9059: Use in0_mcast_num_cores for mat... — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #9059: Avoid a couple of fatals in matmul — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
After https://github.com/tenstorrent/tt-metal/pull/9447 block: 9712 passed, 8240 not passed
That means that PR fixed ~4k tests.
The crashes for Block Sharded can be grouped into the following. The big issue that’s left is “1D mcast for in0 or in1 is not implemented yet.” which @TT-BrianLiu is looking into.
For Width Sharded:
For Height Sharded:
Across the 4 full sweeps:
Interleaved: 2000 passed Width: 2848 passed, 185 failed, 135 crashed (3168 total) Height: 2392 passed, 40 crashed (2432 total) Block: 9712 passed, 8240 crashed (17952 total)
After the latest change from @TT-BrianLiu (https://github.com/tenstorrent/tt-metal/pull/9388) the number of failing/crashing tests has decreased to ~14.5k:
interleaved: 1992 passed 0 failed 8 crashed height: 2376 passed 0 failed 56 crashed width: 2732 passed 264 failed 172 crashed block: 3880 passed 56 failed 14016 crashed