tt-metal: ttnn.layer_norm causes low accuracy in mixer_b16_224.goog_in21k model in pytorch2.0_ttnn project

Describe the bug Several layer_norm operations have low pcc in mixer_b16_224.goog_in21k model which causes low accuracy overall in the model (tested by replacing every layer_norm with aten.native_layer_norm). Inside the model, 23 ttnn.layer_norm operations have pcc bellow 0.999. Input specs which are common for all layer_norm ops:

  • input_shape: [1, 196, 768]
  • weight_shape: [768]
  • bias_shape: [768]
  • epsilon: 1e-6

CSV files of inputs: mixer_layernorm_inputs.zip

To Reproduce Run unit test (contains function for loading tensors from csv file):

# SPDX-FileCopyrightText: © 2023 Tenstorrent Inc.

# SPDX-License-Identifier: Apache-2.0

from loguru import logger
import random
import pytest
import torch
import ttnn
import traceback

from tests.ttnn.utils_for_testing import assert_with_pcc
aten = torch.ops.aten

    
def load_tensor_from_csv(filename, shape):
    import pandas as pd
    df = pd.read_csv(filename)
    data = df.values
    tensor = torch.tensor(data)
    tensor = tensor.view(shape)
    return tensor


def run_layer_norm_tests(
    file_path,
    input_shape, 
    dtype, 
    dlayout,
    epsilon,
    device,
):
    x = load_tensor_from_csv(file_path[0], input_shape[0]).to(torch.bfloat16)
    weight = load_tensor_from_csv(file_path[1], input_shape[1]).to(torch.bfloat16)
    bias = load_tensor_from_csv(file_path[2], input_shape[2]).to(torch.bfloat16)

    try:
        ref_value = aten.native_layer_norm.default(x, [768], weight, bias, epsilon)[0]
        
        tt_x = ttnn.from_torch(x, dtype=dtype[0], layout=dlayout[0], device=device)
        tt_weight = ttnn.from_torch(weight, dtype=dtype[0], layout=dlayout[0], device=device)
        tt_bias = ttnn.from_torch(bias, dtype=dtype[0], layout=dlayout[0], device=device)

        tt_result = ttnn.layer_norm(tt_x, epsilon=epsilon, weight=tt_weight, bias=tt_bias)
        tt_result = ttnn.to_torch(tt_result)
        
        inds = torch.where(ref_value != tt_result)
    except Exception as e:
        logger.warning(f"Test execution crashed: {e}")
        print(traceback.format_exc())
        raise e

    assert len(tt_result.shape) == len(ref_value.shape)
    assert tt_result.shape == ref_value.shape
    assert_with_pcc(ref_value, tt_result, 0.999)


test_sweep_args = [
    (
        ["/home/ubuntu/pytorch2.0_ttnn/mixer_ln1_input.csv", 
         "/home/ubuntu/pytorch2.0_ttnn/mixer_ln1_weight.csv", 
         "/home/ubuntu/pytorch2.0_ttnn/mixer_ln1_bias.csv"],
        [(1, 196, 768), (768), (768)],
        [ttnn.bfloat16],
        [ttnn.TILE_LAYOUT],
        1e-6,
    ),

]
for i in range(3, 25):
    test_sweep_args.append(
        (
            [f"/home/ubuntu/pytorch2.0_ttnn/mixer_ln{i}_input.csv", 
            f"/home/ubuntu/pytorch2.0_ttnn/mixer_ln{i}_weight.csv", 
            f"/home/ubuntu/pytorch2.0_ttnn/mixer_ln{i}_bias.csv"],
            [(1, 196, 768), (768), (768)],
            [ttnn.bfloat16],
            [ttnn.TILE_LAYOUT],
            1e-6,
        )
    )

@pytest.mark.parametrize(
    "file_path, input_shape, dtype, dlayout, epsilon",
    (test_sweep_args),
)
def test_layer_norm(file_path, input_shape, dtype, dlayout, epsilon, device):
    run_layer_norm_tests(file_path, input_shape, dtype, dlayout, epsilon, device)

You can also follow instructions from issue: https://github.com/tenstorrent/pytorch2.0_ttnn/issues/789.

Additional content

Function used for extracting tensor values and saving them to csv file:

  def save_tensor_to_csv(tensor, filename):
    import pandas as pd
    tensor = tensor.contiguous()
    if tensor.dtype == torch.bfloat16:
        tensor = tensor.to(torch.float32)
    if tensor.ndim == 0: 
        df = pd.DataFrame([tensor.item()])
    elif tensor.ndim == 1: 
        df = pd.DataFrame(tensor.numpy())
    elif tensor.ndim == 2:  
        df = pd.DataFrame(tensor.numpy())
    elif tensor.ndim == 3:  
        # Reshape to 2D for saving
        df = pd.DataFrame(tensor.view(-1, tensor.size(-1)).numpy())
    elif tensor.ndim == 4:  
        # Reshape to 2D for saving
        df = pd.DataFrame(tensor.view(-1, tensor.size(-1)).numpy())
    else:
        raise ValueError("Unsupported tensor dimension")
    df.to_csv(filename, index=False)

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 41 (40 by maintainers)

Commits related to this issue

Most upvoted comments

@vsureshTT

On your branch:

Comparison Results:
torch.allclose(rtol=0.01, atol=0.001): False
Mean Absolute Error (MAE):           200.81968689
Mean Squared Error (MSE):            98403.03906250
Maximum Absolute Difference:         2458.21875000

Top 5 Largest Differences:
Index (0, 3, 1420): Ref=5.781250, TTNN=2464.000000, Diff=2458.218750, Rel=425.205405
Index (0, 4, 1643): Ref=8.187500, TTNN=2304.000000, Diff=2295.812500, Rel=280.404580
Index (0, 5, 1098): Ref=8.687500, TTNN=2304.000000, Diff=2295.312500, Rel=264.208633
Index (0, 4, 418): Ref=-7.281250, TTNN=-2288.000000, Diff=2280.718750, Rel=313.231760
Index (0, 0, 586): Ref=6.812500, TTNN=2192.000000, Diff=2185.187500, Rel=320.761468

On v0.57.0-rc60

Comparison Results:
torch.allclose(rtol=0.01, atol=0.001): False
Mean Absolute Error (MAE):           0.02234062
Mean Squared Error (MSE):            0.00127915
Maximum Absolute Difference:         0.43750000

Top 5 Largest Differences:
Index (0, 1, 2023): Ref=11.687500, TTNN=12.125000, Diff=0.437500, Rel=0.037433
Index (0, 3, 419): Ref=5.312500, TTNN=5.625000, Diff=0.312500, Rel=0.058824
Index (0, 1, 107): Ref=-7.968750, TTNN=-8.250000, Diff=0.281250, Rel=0.035294
Index (0, 2, 707): Ref=-8.687500, TTNN=-8.937500, Diff=0.250000, Rel=0.028777
Index (0, 0, 1651): Ref=-7.937500, TTNN=-8.187500, Diff=0.250000, Rel=0.031496

repro: https://gist.github.com/FL33TW00D/ab314343c2b1855e071cdf1784480f1e

Hey @FL33TW00D,

Sorry it seems like my testing was faulty, and more fixes are needed to fix the PCC errors. I’ll continue to work on it and update the thread with my progress.

Best Vash

Can recreate with 2 tiles:

pytest tests/tt_eager/python_api_testing/unit_testing/misc/test_layernorm.py::test_layer_norm_4D_llama
...
(tensor(False), 'Max ULP Delta: 72351744.0')

with diff

--- a/tests/tt_eager/python_api_testing/unit_testing/misc/test_layernorm.py
+++ b/tests/tt_eager/python_api_testing/unit_testing/misc/test_layernorm.py
@@ -10,7 +10,7 @@ import torch
 import ttnn
 
 
-from models.utility_functions import pad_by_zero, torch2tt_tensor, comp_pcc, is_grayskull
+from models.utility_functions import pad_by_zero, torch2tt_tensor, comp_pcc, is_grayskull, comp_ulp
 
 
 def ref_layernorm(x, gamma, beta, eps):
@@ -223,19 +223,30 @@ def test_layernorm_mix_precision(test_id, in_dtype, gamma_dtype, in0_mem_config,
     run_layernorm_mix_precision_tests(test_id, in_dtype, gamma_dtype, in0_mem_config, out_mem_config, device)
 
 
-@pytest.mark.parametrize("h", [1632, 8192, 16384])
-@pytest.mark.parametrize("w", [1280])
-@pytest.mark.parametrize("num_chunks", [1, 4])
+#@pytest.mark.parametrize("h", [1632, 8192, 16384])
+#@pytest.mark.parametrize("w", [1280])
+@pytest.mark.parametrize("h", [32])
+@pytest.mark.parametrize("w", [64])
+@pytest.mark.parametrize("num_chunks", [1,])# 4])
 def test_layer_norm_4D_llama(device, h, w, num_chunks):
     """
     Test specific shapes for LLama and other LLMs
     """
     torch.manual_seed(0)
-
-    torch_input_tensor = torch.rand((1, num_chunks, h, w), dtype=torch.bfloat16)
-    torch_weight = torch.rand((w,), dtype=torch.bfloat16)
-    torch_bias = torch.rand((w,), dtype=torch.bfloat16)
-
+    n = num_chunks * h * w
+    offset = 1e-8
+    torch_input_tensor = (torch.arange(n).reshape([1,num_chunks,h,w]).type(torch.bfloat16))/10 - offset
+    #torch_input_tensor = torch.rand((1, num_chunks, h, w), dtype=torch.bfloat16)
+    torch_weight = torch.ones((w,), dtype=torch.bfloat16)
+    torch_bias = 0 * torch.rand((w,), dtype=torch.bfloat16)
+
+    compute_kernel_config = ttnn.init_device_compute_kernel_config(
+            device.arch(),
+            math_fidelity=ttnn.MathFidelity.HiFi4,
+            math_approx_mode=False,
+            fp32_dest_acc_en=False,
+            packer_l1_acc=False,
+        )
     torch_output_tensor = torch.nn.functional.layer_norm(
         torch_input_tensor, normalized_shape=[w], weight=torch_weight, bias=torch_bias
     )
@@ -245,9 +256,12 @@ def test_layer_norm_4D_llama(device, h, w, num_chunks):
     bias = ttnn.from_torch(torch_bias, layout=ttnn.TILE_LAYOUT, device=device)
 
     output_tensor = ttnn.layer_norm(input_tensor, weight=weight, bias=bias)
-    output_tensor = ttnn.to_layout(output_tensor, ttnn.ROW_MAJOR_LAYOUT)
+    #output_tensor = ttnn.to_layout(output_tensor, ttnn.ROW_MAJOR_LAYOUT)
     output_tensor = ttnn.from_device(output_tensor)
     output_tensor = ttnn.to_torch(output_tensor)
 
-    passing, output = comp_pcc(torch_output_tensor, output_tensor)
+    ulp_threshold = 2
+    allow_nonfinite = True
+    print(comp_ulp(torch_output_tensor, output_tensor, ulp_threshold))
+    passing, output = comp_pcc(torch_output_tensor, output_tensor,pcc=1)
     assert passing, output

@jasondavies They’re what’s in main. I just added prints between the last two lines that set ulp_delta and return.

models/utility_functions.py comp_ulp

    ulp_delta = torch.max(torch.abs(calculated - golden) / ulp_value)
    print(f'calc {calculated}\ngold {golden}\n abs {torch.abs(calculated - golden)}\n  ulp_v {ulp_value}\ndel {ulp_delta} {torch.abs(calculated - golden) / ulp_value}\nshape {golden.shape}')
    s = golden.shape
    for n in range(s[0]):
        for c in range(s[1]):
            for w in range(s[2]):
                for h in range(s[3]):
                    d = torch.abs(calculated[n,c,w,h] - golden[n,c,w,h]) / ulp_value[n,c,w,h]
                    if d > 10000000:
                        print(f'{n},{c},{w},{h} -> {d} : {calculated[n,c,w,h]} {golden[n,c,w,h]} {ulp_value[n,c,w,h]}') 
    
    return (ulp_delta <= ulp_threshold, f"Max ULP Delta: {ulp_delta}")

@vsureshTT

On your branch:

Comparison Results:
torch.allclose(rtol=0.01, atol=0.001): False
Mean Absolute Error (MAE):           200.81968689
Mean Squared Error (MSE):            98403.03906250
Maximum Absolute Difference:         2458.21875000

Top 5 Largest Differences:
Index (0, 3, 1420): Ref=5.781250, TTNN=2464.000000, Diff=2458.218750, Rel=425.205405
Index (0, 4, 1643): Ref=8.187500, TTNN=2304.000000, Diff=2295.812500, Rel=280.404580
Index (0, 5, 1098): Ref=8.687500, TTNN=2304.000000, Diff=2295.312500, Rel=264.208633
Index (0, 4, 418): Ref=-7.281250, TTNN=-2288.000000, Diff=2280.718750, Rel=313.231760
Index (0, 0, 586): Ref=6.812500, TTNN=2192.000000, Diff=2185.187500, Rel=320.761468

On v0.57.0-rc60

Comparison Results:
torch.allclose(rtol=0.01, atol=0.001): False
Mean Absolute Error (MAE):           0.02234062
Mean Squared Error (MSE):            0.00127915
Maximum Absolute Difference:         0.43750000

Top 5 Largest Differences:
Index (0, 1, 2023): Ref=11.687500, TTNN=12.125000, Diff=0.437500, Rel=0.037433
Index (0, 3, 419): Ref=5.312500, TTNN=5.625000, Diff=0.312500, Rel=0.058824
Index (0, 1, 107): Ref=-7.968750, TTNN=-8.250000, Diff=0.281250, Rel=0.035294
Index (0, 2, 707): Ref=-8.687500, TTNN=-8.937500, Diff=0.250000, Rel=0.028777
Index (0, 0, 1651): Ref=-7.937500, TTNN=-8.187500, Diff=0.250000, Rel=0.031496

repro: https://gist.github.com/FL33TW00D/ab314343c2b1855e071cdf1784480f1e

@vsureshTT apologies to jump in where I have limited knowledge of the hardware, but is there a reason you’re not using Welfords algorithm? https://fleetwood.dev/posts/layernorm-as-fast-as-possible

referencing this issue regarding reduce_sum correctness. takeaway: using matmul for reduce_sum can improve correctness #9908

if you go down this route, here’s an example of how to use matmul for reduce_sum in layernorm 11144b8

Awesome, this is super useful. Thank you for the links!

referencing this issue regarding reduce_sum correctness. takeaway: using matmul for reduce_sum can improve correctness https://github.com/tenstorrent/tt-metal/issues/9908

if you go down this route, here’s an example of how to use matmul for reduce_sum in layernorm https://github.com/tenstorrent/tt-metal/commit/11144b872c823cf36d7871df6fb996d541907c6d

@vsureshTT please take a look to see if you can find out why the PCC is low.