opencv: Error while loading pytorch model in opencv

While reading the CRNN model from meijieru in OpenCV, I got the following error

import cv2
net = cv2.dnn.readNetFromTorch("crnn.pth")
error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\dnn\src\torch\torch_importer.cpp:1022: error: (-213:The function/feature is not implemented) Unsupported Lua type in function 'cv::dnn::dnn4_v20191202::TorchImporter::readObject'

model link How to fix this error?

System information (version)
  • OpenCV = 4.2.0
  • Operating System / Platform => Windows 64 Bit

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 33 (26 by maintainers)

Most upvoted comments

amazing news 😉

using the model from meijieru (and resp. preprocessing) i could reproduce results:

crnn

    dnn::Net net = dnn::readNet("crnn.onnx");
    Mat in = imread("demo.png",0);
    in.convertTo(in,CV_32F,1.0/255);
    in -= 0.5;
    in /= 0.5;
    resize(in,in,Size(100, 32)); // fixed !
    Mat blob = blobFromImage(in);
    net.setInput(blob);
    Mat res = net.forward(); // 26 x 1 x 37
    string alphabet = "0123456789abcdefghijklmnopqrstuvwxyz"; // 10 nums + 26 letters + 1 = 37

    for (size_t r=0; r<res.size[0]; r++) {
        Mat slice = Mat(1,res.size[2],CV_32F,res.ptr<float>(r)); // 37 one-hot encoded pins
        Point p; double m;
        minMaxLoc(slice, 0, &m, 0, &p);
        char c = p.x>0 ? alphabet[p.x-1] : '-'; // '-' == no detection
        cout << r << "\t" << c << "\t" << p << "\t" << m << endl;
    }

0       a       [11, 0] -84.5621
1       -       [0, 0]  -64.7243
2       -       [0, 0]  -57.1287
3       -       [0, 0]  -41.7201
4       -       [0, 0]  -55.4314
5       -       [0, 0]  -63.6334
6       v       [32, 0] -100.808
7       -       [0, 0]  -46.9045
8       -       [0, 0]  -60.1601
9       a       [11, 0] -79.15
10      -       [0, 0]  -56.0937
11      i       [19, 0] -83.8074
12      -       [0, 0]  -60.5409
13      l       [22, 0] -88.1379
14      -       [0, 0]  -69.5504
15      a       [11, 0] -74.0695
16      -       [0, 0]  -61.5555
17      b       [12, 0] -85.9853
18      b       [12, 0] -35.0758
19      -       [0, 0]  -77.5764
20      l       [22, 0] -73.1142
21      l       [22, 0] -67.7525
22      e       [15, 0] -71.5772
23      -       [0, 0]  -61.0475
24      -       [0, 0]  -60.4686
25      -       [0, 0]  -60.5497

imho, we can close this issue 😉

@saraansh1999, I believe that this fix will enable the crnn model so nothing will remain for the summer 😃

i just tried https://github.com/opencv/opencv/pull/16817 and it can at least read the crnn.onnx correctly (including bidirectional LSTM) 😉

thanks, @dkurt !!

@sunice-nyy please have a look at https://github.com/opencv/opencv/pull/16817 , which will add ConstantOfShape and LSTM import from onnx

torch.load() loads a state_dict, not a model. (you have to construct the model first, then upload its state_dict) assuming this repo , try:

import torch
import models.crnn as crnn

model = crnn.CRNN(32, 1, 37, 256)
model.load_state_dict(torch.load("crnn.pth"))

dummy_input = torch.randn(1, 1, 37, 256)
torch.onnx.export(model, dummy_input, "crnn.onnx")

import cv2
net=cv2.dnn.readNet("crnn.onnx")
error: OpenCV(4.2.0) /io/opencv/modules/dnn/src/dnn.cpp:562: error: (-2:Unspecified error) Can't create layer "82" of type "ConstantOfShape" in function 'getLayerInstance'

Please read the documentation for readNetFromTorch method: https://docs.opencv.org/master/d6/d0f/group__dnn.html#ga65a1da76cb7d6852bdf7abbd96f19084. It is used for Torch models, not PyTorch. For PyTorch you need to convert it to ONNX and use readNet.