tt-metal: ttnn.layer_norm causes low accuracy in mixer_b16_224.goog_in21k model in pytorch2.0_ttnn project
Describe the bug Several layer_norm operations have low pcc in mixer_b16_224.goog_in21k model which causes low accuracy overall in the model (tested by replacing every layer_norm with aten.native_layer_norm). Inside the model, 23 ttnn.layer_norm operations have pcc bellow 0.999. Input specs which are common for all layer_norm ops:
- input_shape: [1, 196, 768]
- weight_shape: [768]
- bias_shape: [768]
- epsilon: 1e-6
CSV files of inputs: mixer_layernorm_inputs.zip
To Reproduce Run unit test (contains function for loading tensors from csv file):
# SPDX-FileCopyrightText: © 2023 Tenstorrent Inc.
# SPDX-License-Identifier: Apache-2.0
from loguru import logger
import random
import pytest
import torch
import ttnn
import traceback
from tests.ttnn.utils_for_testing import assert_with_pcc
aten = torch.ops.aten
def load_tensor_from_csv(filename, shape):
import pandas as pd
df = pd.read_csv(filename)
data = df.values
tensor = torch.tensor(data)
tensor = tensor.view(shape)
return tensor
def run_layer_norm_tests(
file_path,
input_shape,
dtype,
dlayout,
epsilon,
device,
):
x = load_tensor_from_csv(file_path[0], input_shape[0]).to(torch.bfloat16)
weight = load_tensor_from_csv(file_path[1], input_shape[1]).to(torch.bfloat16)
bias = load_tensor_from_csv(file_path[2], input_shape[2]).to(torch.bfloat16)
try:
ref_value = aten.native_layer_norm.default(x, [768], weight, bias, epsilon)[0]
tt_x = ttnn.from_torch(x, dtype=dtype[0], layout=dlayout[0], device=device)
tt_weight = ttnn.from_torch(weight, dtype=dtype[0], layout=dlayout[0], device=device)
tt_bias = ttnn.from_torch(bias, dtype=dtype[0], layout=dlayout[0], device=device)
tt_result = ttnn.layer_norm(tt_x, epsilon=epsilon, weight=tt_weight, bias=tt_bias)
tt_result = ttnn.to_torch(tt_result)
inds = torch.where(ref_value != tt_result)
except Exception as e:
logger.warning(f"Test execution crashed: {e}")
print(traceback.format_exc())
raise e
assert len(tt_result.shape) == len(ref_value.shape)
assert tt_result.shape == ref_value.shape
assert_with_pcc(ref_value, tt_result, 0.999)
test_sweep_args = [
(
["/home/ubuntu/pytorch2.0_ttnn/mixer_ln1_input.csv",
"/home/ubuntu/pytorch2.0_ttnn/mixer_ln1_weight.csv",
"/home/ubuntu/pytorch2.0_ttnn/mixer_ln1_bias.csv"],
[(1, 196, 768), (768), (768)],
[ttnn.bfloat16],
[ttnn.TILE_LAYOUT],
1e-6,
),
]
for i in range(3, 25):
test_sweep_args.append(
(
[f"/home/ubuntu/pytorch2.0_ttnn/mixer_ln{i}_input.csv",
f"/home/ubuntu/pytorch2.0_ttnn/mixer_ln{i}_weight.csv",
f"/home/ubuntu/pytorch2.0_ttnn/mixer_ln{i}_bias.csv"],
[(1, 196, 768), (768), (768)],
[ttnn.bfloat16],
[ttnn.TILE_LAYOUT],
1e-6,
)
)
@pytest.mark.parametrize(
"file_path, input_shape, dtype, dlayout, epsilon",
(test_sweep_args),
)
def test_layer_norm(file_path, input_shape, dtype, dlayout, epsilon, device):
run_layer_norm_tests(file_path, input_shape, dtype, dlayout, epsilon, device)
You can also follow instructions from issue: https://github.com/tenstorrent/pytorch2.0_ttnn/issues/789.
Additional content
Function used for extracting tensor values and saving them to csv file:
def save_tensor_to_csv(tensor, filename):
import pandas as pd
tensor = tensor.contiguous()
if tensor.dtype == torch.bfloat16:
tensor = tensor.to(torch.float32)
if tensor.ndim == 0:
df = pd.DataFrame([tensor.item()])
elif tensor.ndim == 1:
df = pd.DataFrame(tensor.numpy())
elif tensor.ndim == 2:
df = pd.DataFrame(tensor.numpy())
elif tensor.ndim == 3:
# Reshape to 2D for saving
df = pd.DataFrame(tensor.view(-1, tensor.size(-1)).numpy())
elif tensor.ndim == 4:
# Reshape to 2D for saving
df = pd.DataFrame(tensor.view(-1, tensor.size(-1)).numpy())
else:
raise ValueError("Unsupported tensor dimension")
df.to_csv(filename, index=False)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 41 (40 by maintainers)
Commits related to this issue
- #18548: use reduce fp32 support for layernorm — committed to tenstorrent/tt-metal by bbradelTT a year ago
- #18548: use reduce fp32 support for layernorm — committed to tenstorrent/tt-metal by bbradelTT a year ago
- #18548: use reduce fp32 support for layernorm — committed to tenstorrent/tt-metal by bbradelTT a year ago
Hey @FL33TW00D,
Sorry it seems like my testing was faulty, and more fixes are needed to fix the PCC errors. I’ll continue to work on it and update the thread with my progress.
Best Vash
Can recreate with 2 tiles:
with diff
@jasondavies They’re what’s in main. I just added prints between the last two lines that set ulp_delta and return.
models/utility_functions.py comp_ulp
@vsureshTT
On your branch:
On v0.57.0-rc60
repro: https://gist.github.com/FL33TW00D/ab314343c2b1855e071cdf1784480f1e
@vsureshTT apologies to jump in where I have limited knowledge of the hardware, but is there a reason you’re not using Welfords algorithm? https://fleetwood.dev/posts/layernorm-as-fast-as-possible
Awesome, this is super useful. Thank you for the links!
referencing this issue regarding reduce_sum correctness. takeaway: using matmul for reduce_sum can improve correctness https://github.com/tenstorrent/tt-metal/issues/9908
if you go down this route, here’s an example of how to use matmul for reduce_sum in layernorm https://github.com/tenstorrent/tt-metal/commit/11144b872c823cf36d7871df6fb996d541907c6d
@vsureshTT please take a look to see if you can find out why the PCC is low.