tt-metal: attn_matmul fail with short init

While working on #16439 and other related issues, we noticed an error in ttnn/cpp/ttnn/operations/experimental/matmul/attn_matmul/device/kernels/compute/transformer_attn_matmul.cpp where using mm_init_short_with_dt(cb_in0, cb_in1, cb_intermed2, transpose_hw); caused an error in. Blackhole, shown here: https://github.com/tenstorrent/tt-metal/actions/runs/14012386191/job/39234931852

That’s why we have put a workaround for it to pass, and used mm_init(cb_in0, cb_in1, cb_intermed0, transpose_hw); instaed.

The purpose of this issue is to investigate why does this failure occur and to fix it, since using mm_init_short_with_dt is correct from the design perpsective.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 17 (16 by maintainers)

Most upvoted comments

@bbradelTT hopefully the comment above the change is a good enough signal that this is not intended usage, and deters people from using it. If you feel stronger comment should be added, it can be added in a separate PR. Let’s just get this workaround in so it is easier for people to consume. My guess of what happened here is that there is a race somewhere which got exposed with the change @ncvetkovicTT introduced. I understand this is important to figure out and let’s prioritize accordingly in our regular syncs.

Of course! These conditions are intended to make the issue as reproducible as possible, I would say perhaps it deviates by 1 at most. I don’t think the described situation is likely to happen unless the test is run under different conditions like disabling watcher/inserting nops/using the full init.

To double check, I enabled watcher and ran my modified test on a p100a 20 times. The test failed every single time on iteration 8. And for clarity the iteration count starts at zero, so it is actually the output of the ninth time the op runs that will show the error.

It showed up while debugging matmul hang, but I wouldn’t include it in BH Hang group of issues.

EDIT: It doesn’t contribute to the hang, it makes some failing test pass.

@ncvetkovicTT I was told by the LLK team that a full init can take hundreds of cycles and does a bunch of stuff that assumes that the three TRISCs are not busy doing other work. And that this meant that putting in a full init anywhere other than at the start can lead to errors.

I still need to identify the perf impact, but at this point I’m worried about correctness and people possibly mimicking this pattern as well.