tt-metal: [Bug Report] Softmax op L1 memory issue
Describe the bug
When apply softmax op to tensors of (32, 32000) dimension, the following error arise:
RuntimeError: TT_THROW @ /tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception
info:
Statically allocated circular buffers on core range [(x=0,y=0) - (x=7,y=7)] grow to 2287328 B which is beyond max L1 size of 1499136 B
This also happens for other dimensions (128, 32000), (512, 32000), (2048, 32000), (32, 128000), (128, 128000), (512, 128000), (2048, 128000). Also when bfloat16 dtype is used.
To Reproduce Steps to reproduce the behavior:
import ttnn
import torch
device_id = 0
device = ttnn.open_device(device_id=device_id)
logits_torch = torch.rand((32, 32000), dtype=torch.float32)
logits_ttnn = ttnn.from_torch(logits_torch, layout=ttnn.TILE_LAYOUT, device=device)
ttnn.softmax(input_tensor=logits_ttnn, dim=-1)
Expected behavior Output of softmax should be similar to torch:
tensor([[2.4504e-05, 2.4076e-05, 3.8839e-05, ..., 2.4552e-05, 2.2156e-05,
4.6891e-05],
[2.2884e-05, 2.0268e-05, 2.6400e-05, ..., 4.3591e-05, 2.4748e-05,
4.6146e-05],
[4.6252e-05, 2.0523e-05, 2.1745e-05, ..., 4.6246e-05, 3.0338e-05,
4.3000e-05],
...,
[2.6454e-05, 3.9702e-05, 3.7524e-05, ..., 2.1065e-05, 2.8300e-05,
2.8593e-05],
[3.9618e-05, 2.3662e-05, 2.4006e-05, ..., 1.9505e-05, 3.7952e-05,
3.4435e-05],
[1.9118e-05, 1.9010e-05, 2.2848e-05, ..., 2.1335e-05, 1.8789e-05,
3.5355e-05]])
Please complete the following environment information:
- Version of software (eg. commit): https://github.com/tenstorrent/tt-metal/commit/077162206d7f1b7cb5df6294991454e017d3e719
Additional context This issue is relevant for LLM model training as it arises during CrossEntropy loss calculation. Specifically for Llama models (openllama 3B and Llama 3.2 1B), but it might happen for other LLM models with large vocabulary sizes.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (12 by maintainers)
Commits related to this issue
- Vsuresh tt/softmax l1ovf pr1 (#23130) ### Ticket Link to Github Issue https://github.com/tenstorrent/tt-metal/issues/18997 https://github.com/tenstorrent/tt-metal/issues/20099 ### Problem descriptio... — committed to tenstorrent/tt-metal by vsureshTT a year ago
- Vsuresh tt/softmax l1ovf pr1 (#23130) ### Ticket Link to Github Issue https://github.com/tenstorrent/tt-metal/issues/18997 https://github.com/tenstorrent/tt-metal/issues/20099 ### Problem descriptio... — committed to tenstorrent/tt-metal by vsureshTT a year ago
I thought I would add an update before signing off. Made some good progress on adding the scale/attention mask to softmax, and things seem to be going at the expected schedule.
Hi @pmarkovicTT, whenever you have the chance could you try testing this branch for your use cases https://github.com/tenstorrent/tt-metal/tree/origin/vsureshTT/softmaxL1overflow/stable
You can test the larger softmax shapes with
cd tt-metal/tests/ttnn/unit_tests/operations/fusedand then runningpytest test_softmax.py::test_large_softmaxThis test should pass on this branch. Please note that this branch is experimental and only supports base softmax.
Right now, I am working on adding numerical stability, fused scale mask and casual mask to softmax.
So far numerical stability is working, and I am looking to finish up the rest by the end of this week and the goal is to get a PR merged in early next week.
Please let me know if have any issues testing this branch
@bbradelTT I still have to add support for all the control paths like the different masking options, but should work with a standard softmax call. I’ll update this branch today to get it into a decent state. https://github.com/tenstorrent/tt-metal/tree/vsureshTT/softmaxL1ovf. The PCC I believe was around 92%, which seems a little too low to merge into main.
Hi Sapna, a fix is currently in development. It should be done by end of next week.