tt-metal: TopK op is causing some multidevice bad state

Issue Description

This behaviour was first spotted in T3000 CI tests. The issue persists after device close and seemingly can only be fixed with device reset.

I managed to triage the issue to the ttnn.experimental.operations.primary.topk.

When a module with the TopK op is executed on multi-device before Falcon40B decode test, the output PCC of falcon is greatly lowered (around 0.8 instead of 0.99).

This has been observed in multiple SJC T3000 machines.

Reproduction steps

  1. Run a small unit test of topK op. I have tested both the shapes used in our Mixtral MoE module below as well as the tests inside tests/tt_eager/python_api_testing/unit_testing/misc/test_topk.py:

test_topk.py

import torch
import ttnn

def test_topK(t3k_device_mesh, use_program_cache, reset_seeds):
    gate_shape = [1,1,32,64]
    gate_torch = torch.randn(gate_shape)
    gate_logits = ttnn.from_torch(
        gate_torch,
        dtype=ttnn.bfloat16,
        layout=ttnn.TILE_LAYOUT,
        memory_config=ttnn.L1_MEMORY_CONFIG,
        device=t3k_device_mesh,
        mesh_mapper=ttnn.ReplicateTensorToMesh(t3k_device_mesh),
    )
    ttnn.experimental.operations.primary.topk(gate_logits, 32)
  1. Run falcon40b decode test. It will download the falcon weights if non-existent in your machine. pytest models/demos/t3000/falcon40b/tests/test_falcon_decoder.py::test_FalconDecoder_inference[BFLOAT8_B-SHARDED-falcon_40b-layer_0-decode_batch32-8chips]

The output of falcon40b test will be something like:

Output: Max ATOL Delta: 58.7618408203125, Max RTOL Delta: inf, PCC: 0.8449884518114068, PCC check failed
K Cache: Max ATOL Delta: 0.22574996948242188, Max RTOL Delta: inf, PCC: 0.9999446038705259
V Cache: Max ATOL Delta: 0.019847989082336426, Max RTOL Delta: inf, PCC: 0.9999691527555423
K Cache new token: Max ATOL Delta: 0.22574996948242188, Max RTOL Delta: inf, PCC: 0.9998440004560603
V Cache new token: Max ATOL Delta: 0.019847989082336426, Max RTOL Delta: inf, PCC: 0.9997933878705257

The correct output PCC should be instead PCC: 0.9984268799778373

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (18 by maintainers)

Commits related to this issue

Most upvoted comments

https://github.com/tenstorrent/tt-metal/actions/runs/9467796636

CI run with radomir’s change and moving mixtral back before falcon

I just tried commit from @rdjogoTT d5312d7c1f997c59cbaeb12facb65c3866c026c1 isolated (cherry-picked on latest main) and PCC is good as well=0.998

Would there be a perf impact in choosing one of the solutions over the other? I.e. make other ops init to the normal mode or just init to the normal mode at the end of topk.

Perf impact should be small either way, but technically initializing back to normal mode after topk would add fewer new init calls. However, my commit implements the other solution as that is what we have in Buda, it just wasn’t captured by the submodule migration and that’s why it’s missing from metal.

Talked to @rdjogoTT and it seems like the topk_tile_init() puts the sfpu into a special mode needed for copy the swaps needed for sorting to the indices. The fix is to either make other ops init to the normal mode or just init to the normal mode at the end of topk.

The fix should be to put config back to default state, other ops should not know about what has run before them

Yeah I’m the right person for this. I’ll take a look.