tt-metal: TopK op is causing some multidevice bad state
Issue Description
This behaviour was first spotted in T3000 CI tests. The issue persists after device close and seemingly can only be fixed with device reset.
I managed to triage the issue to the ttnn.experimental.operations.primary.topk.
When a module with the TopK op is executed on multi-device before Falcon40B decode test, the output PCC of falcon is greatly lowered (around 0.8 instead of 0.99).
This has been observed in multiple SJC T3000 machines.
Reproduction steps
- Run a small unit test of topK op. I have tested both the shapes used in our Mixtral MoE module below as well as the tests inside
tests/tt_eager/python_api_testing/unit_testing/misc/test_topk.py:
test_topk.py
import torch
import ttnn
def test_topK(t3k_device_mesh, use_program_cache, reset_seeds):
gate_shape = [1,1,32,64]
gate_torch = torch.randn(gate_shape)
gate_logits = ttnn.from_torch(
gate_torch,
dtype=ttnn.bfloat16,
layout=ttnn.TILE_LAYOUT,
memory_config=ttnn.L1_MEMORY_CONFIG,
device=t3k_device_mesh,
mesh_mapper=ttnn.ReplicateTensorToMesh(t3k_device_mesh),
)
ttnn.experimental.operations.primary.topk(gate_logits, 32)
- Run falcon40b decode test. It will download the falcon weights if non-existent in your machine.
pytest models/demos/t3000/falcon40b/tests/test_falcon_decoder.py::test_FalconDecoder_inference[BFLOAT8_B-SHARDED-falcon_40b-layer_0-decode_batch32-8chips]
The output of falcon40b test will be something like:
Output: Max ATOL Delta: 58.7618408203125, Max RTOL Delta: inf, PCC: 0.8449884518114068, PCC check failed
K Cache: Max ATOL Delta: 0.22574996948242188, Max RTOL Delta: inf, PCC: 0.9999446038705259
V Cache: Max ATOL Delta: 0.019847989082336426, Max RTOL Delta: inf, PCC: 0.9999691527555423
K Cache new token: Max ATOL Delta: 0.22574996948242188, Max RTOL Delta: inf, PCC: 0.9998440004560603
V Cache new token: Max ATOL Delta: 0.019847989082336426, Max RTOL Delta: inf, PCC: 0.9997933878705257
The correct output PCC should be instead PCC: 0.9984268799778373
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (18 by maintainers)
Commits related to this issue
- #9206: try mixtral falcon with dram states cleared — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try mixtral falcon with dram states cleared — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try mixtral falcon with dram states cleared — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try mixtral falcon with dram states cleared — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try mixtral falcon with dram states cleared — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: try topk then falcon with dram states cleared in between — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
- #9206: add sfpu config reg init to llk sfpu inits — committed to tenstorrent/tt-metal by rdjogoTT 2 years ago
- #9206: move mixtral above falcon in pipelines — committed to tenstorrent/tt-metal by sjameelTT 2 years ago
https://github.com/tenstorrent/tt-metal/pull/9358 merged
https://github.com/tenstorrent/tt-metal/actions/runs/9467796636
CI run with radomir’s change and moving mixtral back before falcon
Perf impact should be small either way, but technically initializing back to normal mode after topk would add fewer new init calls. However, my commit implements the other solution as that is what we have in Buda, it just wasn’t captured by the submodule migration and that’s why it’s missing from metal.
The fix should be to put config back to default state, other ops should not know about what has run before them
Yeah I’m the right person for this. I’ll take a look.