tt-metal: 2D matmul with specific shapes gives bad PCC on a TG

Describe the bug A 2D fractured matmul with specific shapes gives bad PCC when run on a galaxy. Shape is for llama3-405B FF1 for prefill with sequence length = 512, both activation and weight are in DRAM interleaved format. The matmul works with lower sequence lengths (=M), e.g. 128 or 256.

M = 512, K = 16 * 1024, N = 52 * 1024

The PCC is (so far) always around zero but still non-deterministic, e.g. in three runs:

PCC: 0.0002399902230190794
PCC: -0.00038166219156071467
PCC: -0.00017789431408447355

To Reproduce Steps to reproduce the behavior:

  1. Reproduce on a TG
  2. Checkout branch jrock/llama3-405b
  3. Run unit test pytest tests/ttnn/multichip_unit_tests/test_multidevice_TG.py::test_galaxy_matmul_2d_fracture[silicon_arch_name=wormhole_b0-silicon_arch_wormhole_b0=True-Llama3-405B_prefill_seq512_FF1-4x8_grid]

Expected behavior Passes the test with 0.99 PCC target.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 39 (35 by maintainers)

Commits related to this issue

Most upvoted comments

Interesting. The kernels used by MatmulMultiCoreNonOptimized may not be configured properly to handle bfloat4_b that shows up when rounding is involved.

I’ll have to look into that.

Couple of things:

  • Shouldn’t draw any conclusions out of tests that have 0.000 PCC. The output is probably complete garbage. “ND” could come from reading/writing to undefined space which might change per run.
  • Try to use a specific variant instead of default matmul in models. You can bypass the automatic paths that ttnn.matmul takes. These paths become more and more involved as we try to support more behaviour.
  • Try to run your matmul for single chip (with properly scaled down specs) and see if that works. If PCC is good, then go to multi-chip

@johanna-rock-tt I think once the tensors get big enough there is overflow and rounding happening, where order of operation has a big impact.

The tensor has some common info for bfloat4_b and bfloat8_b. Based on what I saw, probably at least for the mantissa.

E.g. I would get sets of values such as 0.50000, 0.25000, …, 0.12500, 0.25000 or 512.00000, 384.00000, …, 128.00000, 896.00000, with nothing between set boundaries.

If that common info is set based on inputs or some criteria that can’t handle the right sets of ranges, and then there is enough addition, the values probably drift off.

It’d be good to have a reference for bfloat4_b, and then we could do more than speculate.

yes the matmul is running on 32 devices when it fails. I’ll check on t3k