tt-metal: Pushing BFP8 tensors to device is twice as slow as BF16
Describe the bug We need to push attn_mask to device every token and its in bfp8 to save memory, we found that the calling torch2tt_tensor to DRAM interleaved is twice as slow for bfp8 tensors than bf16 tensors of the same size.
To Reproduce Steps to reproduce the behavior:
- checkout this branch:
- Take a look at this test file that creates two tensors in diffrent data format and profiles the time for torch2tt_tensor
- run this command after building tt-metal :
pytest -svv models/demos/llama2_70b/tests/unit_tests/test_BFP8_to_device.py::test_bfp8_to_device[32-1] - observe results shown by the profiler
Expected behavior For tensor of size (32, 1, 32, 512)
pushing_attn_mask_to_DRAM_bf16: 0.017s
pushing_attn_mask_to_DRAM_bf8: 0.034s
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (12 by maintainers)
We will include that in our next merge to main. Should be able to speed up e2e time by a lot.
@jliangTT we arent looking at end to end performance very closely right now so P2 should be good? But this could also speed up pushing weights so potentially time saving for a lot of teams.