tt-metal: 2D matmul with specific shapes gives bad PCC on a TG
Describe the bug A 2D fractured matmul with specific shapes gives bad PCC when run on a galaxy. Shape is for llama3-405B FF1 for prefill with sequence length = 512, both activation and weight are in DRAM interleaved format. The matmul works with lower sequence lengths (=M), e.g. 128 or 256.
M = 512, K = 16 * 1024, N = 52 * 1024
The PCC is (so far) always around zero but still non-deterministic, e.g. in three runs:
PCC: 0.0002399902230190794
PCC: -0.00038166219156071467
PCC: -0.00017789431408447355
To Reproduce Steps to reproduce the behavior:
- Reproduce on a TG
- Checkout branch
jrock/llama3-405b - Run unit test
pytest tests/ttnn/multichip_unit_tests/test_multidevice_TG.py::test_galaxy_matmul_2d_fracture[silicon_arch_name=wormhole_b0-silicon_arch_wormhole_b0=True-Llama3-405B_prefill_seq512_FF1-4x8_grid]
Expected behavior Passes the test with 0.99 PCC target.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 39 (35 by maintainers)
Commits related to this issue
- #10936: Update matmul_multi_core_reuse to support mixed precision (#11947) Co-authored-by: Austin Ho <aho@tenstorrent.com> — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #10936: Update matmul_multi_core_reuse to support mixed precision (#11947) Co-authored-by: Austin Ho <aho@tenstorrent.com> — committed to tenstorrent/tt-metal by bbradelTT 2 years ago
- #10936: Enable llama tg unit test — committed to tenstorrent/tt-metal by johanna-rock-tt 2 years ago
- #10936: Enable llama tg unit test — committed to tenstorrent/tt-metal by johanna-rock-tt 2 years ago
- #10936: Enable llama tg unit test (#12263) — committed to tenstorrent/tt-metal by johanna-rock-tt 2 years ago
Interesting. The kernels used by MatmulMultiCoreNonOptimized may not be configured properly to handle bfloat4_b that shows up when rounding is involved.
I’ll have to look into that.
Couple of things:
ttnn.matmultakes. These paths become more and more involved as we try to support more behaviour.@johanna-rock-tt I think once the tensors get big enough there is overflow and rounding happening, where order of operation has a big impact.
The tensor has some common info for bfloat4_b and bfloat8_b. Based on what I saw, probably at least for the mantissa.
E.g. I would get sets of values such as 0.50000, 0.25000, …, 0.12500, 0.25000 or 512.00000, 384.00000, …, 128.00000, 896.00000, with nothing between set boundaries.
If that common info is set based on inputs or some criteria that can’t handle the right sets of ranges, and then there is enough addition, the values probably drift off.
It’d be good to have a reference for bfloat4_b, and then we could do more than speculate.
yes the matmul is running on 32 devices when it fails. I’ll check on t3k