nncf: NNCF2.5 When quantizing the model, an error occurred: "RuntimeError: Could not find the bias value of the node."

I have an ONNX model that contains convolutional layers but no fully connected layers. Upon inspection with Netron, I found that if a convolutional layer is not directly followed by a BatchNormalization layer, then the convolutional layer has both weights and biases. However, if a convolutional layer is directly followed by a BatchNormalization layer, it only has weights, and the BatchNormalization layer carries the bias. This is the structure of my model. I want to quantize it to int8 using NNCF. Currently, in the get_bias_value function in the nncf/quantization/algorithms/fast_bias_correction/onnx_backend.py code, I am encountering an error that says ‘Could not find the bias value of the node’. Do I now have to add a bias of 0 to all convolutional layers that do not have a bias, in order to avoid this error during quantization?

The code for quantization is:

import torch
import numpy as np
import onnx
from torchvision import datasets
from torchvision import transforms

import nncf

model_path = "/home/fp32_mainbody_onnx_bs/const_shape_pp_main_body.onnx"

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_dataset = datasets.ImageFolder(
    root=f"/home/resize_images_400",
    transform=transforms.Compose(
        [
            transforms.Resize(640),
            transforms.ToTensor(),
            normalize,
        ]
    ),
)

val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1, shuffle=False)
model = onnx.load(model_path)

def transform_fn(data_item):
    images, _ = data_item
    scale = np.array([640 / 1800, 640 / 1800], dtype='float32').reshape(1, 2)
    return {input_img_name: images.numpy(),input_scale_name: scale}
    
calibration_dataset = nncf.Dataset(val_loader, transform_fn)
onnx_quantized_model = nncf.quantize(model, calibration_dataset, subset_size=400)

int8_model_path = f"/home/int8_nncf_quant/int8_main_body.onnx"
onnx.save(onnx_quantized_model, int8_model_path)

The error message is:

Traceback (most recent call last): File “nncf_quant_mainbody.py”, line 40, in <module> onnx_quantized_model = nncf.quantize(model, calibration_dataset, subset_size=400) File “/home/.local/lib/python3.8/site-packages/nncf/quantization/quantize_model.py”, line 93, in quantize return quantize_impl( File “/home/.local/lib/python3.8/site-packages/nncf/telemetry/decorator.py”, line 71, in wrapped retval = fn(*args, **kwargs) File “/home/.local/lib/python3.8/site-packages/nncf/onnx/quantization/quantize_model.py”, line 68, in quantize_impl quantized_model = quantization_algorithm.apply(model, dataset=calibration_dataset) File “/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/algorithm.py”, line 58, in apply return self._apply(model, statistic_points=None, dataset=dataset) File “/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/post_training/algorithm.py”, line 188, in _apply modified_model = algorithm.apply(modified_model, statistic_points) File “/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/algorithm.py”, line 63, in apply return self._apply(model, statistic_points) File “/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/fast_bias_correction/algorithm.py”, line 136, in _apply for node, bias_value in tqdm(list(node_and_bias_value), desc=“Biases correction”): File “/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/fast_bias_correction/algorithm.py”, line 128, in <genexpr> (node, self._backend_entity.get_bias_value(node, nncf_graph, model)) File “/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/fast_bias_correction/onnx_backend.py”, line 86, in get_bias_value return get_bias_value(node, model) File “/home/.local/lib/python3.8/site-packages/nncf/onnx/graph/node_utils.py”, line 60, in get_bias_value raise RuntimeError(“Could not find the bias value of the node”) RuntimeError: Could not find the bias value of the node

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 38 (3 by maintainers)

Commits related to this issue

Most upvoted comments

thanks @alexsu52

but nncf.quantize(model, calibration_dataset, preset=nncf.QuantizationPreset.MIXED,fast_bias_correction=False,subset_size=2000,advanced_parameters=nncf.AdvancedQuantizationParameters(activations_range_estimator_params=nncf.RangeEstimatorParametersSet.MINMAX)this code is throwing an error:module 'nncf' has no attribute 'RangeEstimatorParametersSet' as same as activations_range_estimator_params=RangeEstimatorParametersSet.MINMAX

And I randomly selected representative images. Below is the download link.

https://drive.google.com/file/d/1CWwR4BzJSstEow4hEkNX9DvO5lgtO5Ys/view?usp=sharing

fp32 model: https://drive.google.com/file/d/14Xh9Uj4kEjxuWmGewGyT9b9dIHGjUp8V/view?usp=drive_link

Here is the quantization code I have, which ignores the post-processing layer of quantization:

import torch
import numpy as np
import onnx
from onnx.version_converter import convert_version
from torchvision import datasets
from torchvision import transforms
from PIL import Image
import nncf

model_path = "/home/1/fp32_mainbody_onnx_bs/const_shape_pp_main_body.onnx"

class StoreOriginalSize:
    def __init__(self):
        self.sizes = []  # a list to store original sizes

    def __call__(self, img):
        # This transform does not change the image, it only stores its original size
        self.sizes.append(img.size)  # save original size (width, height)
        return img

store_original_size = StoreOriginalSize()

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_dataset = datasets.ImageFolder(
    root=f"/home/1/resize_images_2000",
    transform=transforms.Compose(
        [
            store_original_size,  # this will store original sizes before resizing
            transforms.Resize((640, 640)),
            transforms.ToTensor(),
            normalize,
        ]
    ),
)

val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1, shuffle=False)
old_model = onnx.load(model_path)

model = convert_version(old_model, target_version=13)

onnx.checker.check_model(model)

input_names = [input.name for input in model.graph.input]
input_img_name = input_names[0]
input_scale_name = input_names[1]

def transform_fn(data_item):
    images, _ = data_item
    origin_width, origin_height = store_original_size.sizes.pop(0)
    scale = np.array([640 / origin_height, 640 / origin_width], dtype='float32').reshape(1, 2)
    return {input_img_name: images.numpy(),input_scale_name: scale}
    
calibration_dataset = nncf.Dataset(val_loader, transform_fn)
onnx_quantized_model = nncf.quantize(model, calibration_dataset, preset=nncf.QuantizationPreset.MIXED
, fast_bias_correction=False,subset_size=492,
        ignored_scope=nncf.IgnoredScope(
            #types=["Reshape","Slice","Concat"],  # ignore operations
            names=[
                "p2o.Slice.1",  # in the post-processing subgraph
                "p2o.Reshape.33",
                "p2o.Reshape.32",
                "p2o.Reshape.31",
                "p2o.Concat.24",
                "p2o.Slice.0",
                "p2o.Reshape.29",
                "p2o.Reshape.30",
                "p2o.Gather.0",
                "p2o.Mul.196",
                "p2o.Gather.2",
                "p2o.NonMaxSuppression.0",
                "p2o.Reshape.28",
                "p2o.Mul.194",
                "p2o.Add.208",
                "p2o.Add.206",
                "p2o.Mul.192",
                "p2o.Reshape.12",
                "p2o.MatMul.0",
                "p2o.Reshape.11",
                "p2o.Reshape.17",
                "p2o.MatMul.2",
                "p2o.Reshape.16",
                "p2o.Reshape.22",
                "p2o.MatMul.4",
                "p2o.Reshape.21",
                "p2o.Reshape.27",
                "p2o.MatMul.6",
                "p2o.Reshape.26",
                "p2o.Reshape.10",
                "p2o.Reshape.15",
                "p2o.TopK.0",
                "p2o.ReduceMin.0",
                "p2o.Gather.8",
                "p2o.Gather.10",
                "p2o.Div.94",
            ],
        ),
    )

int8_model_path = f"/home/1/int8_nncf_quant/int8_main_body_new_branch_bs_1_2000.onnx"
onnx.save(onnx_quantized_model, int8_model_path)