tt-metal: Incorrect ttnn.linear result for activations with shape Mx1xN
When the activation tensor has shape Mx1xN and a bias of rank 2 is present, the result of ttnn.linear is incorrect.
We were able to work around the issue by making the bias a rank 3 tensor by adding another dimension of size 1. While this solution works flawlessly for our use-case, I believe that a bias of rank 3 should ideally not be required when calling ttnn.linear, as it makes the output always have at least rank 3.
Test:
import pytest
import torch
import ttnn
@pytest.mark.parametrize("seed", range(10))
@pytest.mark.parametrize("inner_dim", [1, 2])
def test_linear_inner_dim_one(*, inner_dim: int, seed: int, device: ttnn.Device):
torch.manual_seed(seed)
k = 32
m = 32
x = torch.zeros([2, inner_dim, k])
w = torch.zeros([k, m])
b = torch.randn([1, m])
tt_x = ttnn.from_torch(x, layout=ttnn.TILE_LAYOUT, device=device)
tt_w = ttnn.from_torch(w, layout=ttnn.TILE_LAYOUT, device=device)
tt_b = ttnn.from_torch(b, layout=ttnn.TILE_LAYOUT, device=device)
tt_result = ttnn.linear(tt_x, tt_w, bias=tt_b)
result = ttnn.to_torch(tt_result)
assert torch.allclose(result[0], result[1])
The test runs ttnn.linear with a batch size of two and compares the outcome of the two batches. They should match, since the activations and weights are all zero. There seems to be access to uninitialized memory involved, so the problem may not always be visible. Therefore, the test is run multiple times. On our setup, the test succeeds with inner_dim=2 (as it should) and fails with inner_dim=1.
Environment:
- tt-metal version: main branch, commit https://github.com/tenstorrent/tt-metal/commit/5d9e93f6d4
- OS: Linux
- Distro: Ubuntu 20.04.6 LTS
- Kernel: 5.4.0-196-generic
- Platform: x86_64
- Python: 3.8.10
- FW Bundle Version: 80.10.0.0
- TT-Flash Version: 0.2.2.0
- CM FW Version: 2.27.0.0
- ETH FW Version: 6.9.0
- BM BL Version: 129.2.0.0
- BM App Version: 5.9.0.2
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (18 by maintainers)
Commits related to this issue
- Disable bias broadcast cases until resolution of issue #16599 — committed to tenstorrent/tt-metal by nsorabaTT 2 years ago
- Disable bias broadcast cases until resolution of issue #16599 — committed to tenstorrent/tt-metal by nsorabaTT 2 years ago
@bbradelTT Issued created https://github.com/tenstorrent/tt-metal/issues/24316
Testing on a commit from this week: 813c49f51f
The pytest in the description passes. test_add_error from https://github.com/tenstorrent/tt-metal/issues/16599#issuecomment-2828991210 still fails:
The original test in the description is still failing for me. Reopening.
@dchenTT what is the status of binary ng and turning it on for the add in linear?
@mbahnasTT to confirm
We currently do not support uneven sharding size when use_legacy=false. That explains it. We can easily support uneven shard size for HEIGHT sharding though. As when to support WIDTH and BLOCK, we shall discuss with @cmaryanTT