tt-metal: [Bug Report] Softmax op L1 memory issue

Describe the bug When apply softmax op to tensors of (32, 32000) dimension, the following error arise:

RuntimeError: TT_THROW @ /tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception
info:
Statically allocated circular buffers on core range [(x=0,y=0) - (x=7,y=7)] grow to 2287328 B which is beyond max L1 size of 1499136 B

This also happens for other dimensions (128, 32000), (512, 32000), (2048, 32000), (32, 128000), (128, 128000), (512, 128000), (2048, 128000). Also when bfloat16 dtype is used.

To Reproduce Steps to reproduce the behavior:

import ttnn
import torch


device_id = 0
device = ttnn.open_device(device_id=device_id)

logits_torch = torch.rand((32, 32000), dtype=torch.float32)
logits_ttnn = ttnn.from_torch(logits_torch, layout=ttnn.TILE_LAYOUT, device=device)

ttnn.softmax(input_tensor=logits_ttnn, dim=-1)

Expected behavior Output of softmax should be similar to torch:

tensor([[2.4504e-05, 2.4076e-05, 3.8839e-05,  ..., 2.4552e-05, 2.2156e-05,
         4.6891e-05],
        [2.2884e-05, 2.0268e-05, 2.6400e-05,  ..., 4.3591e-05, 2.4748e-05,
         4.6146e-05],
        [4.6252e-05, 2.0523e-05, 2.1745e-05,  ..., 4.6246e-05, 3.0338e-05,
         4.3000e-05],
        ...,
        [2.6454e-05, 3.9702e-05, 3.7524e-05,  ..., 2.1065e-05, 2.8300e-05,
         2.8593e-05],
        [3.9618e-05, 2.3662e-05, 2.4006e-05,  ..., 1.9505e-05, 3.7952e-05,
         3.4435e-05],
        [1.9118e-05, 1.9010e-05, 2.2848e-05,  ..., 2.1335e-05, 1.8789e-05,
         3.5355e-05]])

Please complete the following environment information:

Additional context This issue is relevant for LLM model training as it arises during CrossEntropy loss calculation. Specifically for Llama models (openllama 3B and Llama 3.2 1B), but it might happen for other LLM models with large vocabulary sizes.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (12 by maintainers)

Commits related to this issue

Most upvoted comments

I thought I would add an update before signing off. Made some good progress on adding the scale/attention mask to softmax, and things seem to be going at the expected schedule.

Hi @pmarkovicTT, whenever you have the chance could you try testing this branch for your use cases https://github.com/tenstorrent/tt-metal/tree/origin/vsureshTT/softmaxL1overflow/stable

You can test the larger softmax shapes with cd tt-metal/tests/ttnn/unit_tests/operations/fused and then running pytest test_softmax.py::test_large_softmax

This test should pass on this branch. Please note that this branch is experimental and only supports base softmax.

Right now, I am working on adding numerical stability, fused scale mask and casual mask to softmax.

So far numerical stability is working, and I am looking to finish up the rest by the end of this week and the goal is to get a PR merged in early next week.

Please let me know if have any issues testing this branch

@bbradelTT I still have to add support for all the control paths like the different masking options, but should work with a standard softmax call. I’ll update this branch today to get it into a decent state. https://github.com/tenstorrent/tt-metal/tree/vsureshTT/softmaxL1ovf. The PCC I believe was around 92%, which seems a little too low to merge into main.

@vsureshTT Do we have any update on this bug?

Hi Sapna, a fix is currently in development. It should be done by end of next week.