opencv_contrib: sfm::reconstruct error with python error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

Hi Everyone,

I’m messing around with the sfm module and doing a 3D reconstruction from a video file. I’ve compiled the OpenCV project with the contributed modules on my machine (Mac OS X 10.11.6, python 3.6 Anaconda) with python bindings for sfm. Compilation was successful, and I am able to successfully run the example_sfm_scene_reconstruction program.

For the problem I’m having, the general program is as follows:

Extract the location of ORB features for the first 8 images from the video
Compile the features into a python list, where each element of the list is a 2XN (where N is the number of features detected in the image, typically 100 points so 2X100) numpy array.
Load the camera calibration matrix from a previously stored file
Call the sfm::reconstruct(InputArrayOfArrays points2d, OutputArray Ps, OutputArray points3d, InputOutputArray K, bool is_projective) via the python binding.

Code is as follows

# get all the keypoints, save them in a pandas dataframe
all_keypoints = []

for idx, im in tqdm(zip(range(idx_start, idx_stop),images),desc='gettingkeypoints',total=len(images)):
#     kps = fast_detector.detect(im)
    kps = orb_detector.detect(im)
    for kp in kps:
        all_keypoints.append({'frame':idx, 'x':kp.pt[0], 'y':kp.pt[1]})

# get the keypoints for the first 8 frames
samped_frames = list(set(kp_df.loc[:, 'frame'].values.tolist()))
samped_frames = samped_frames[0:8]
kps_for_recon = []
for idx in samped_frames:
    kps_data = kp_df.loc[kp_df['frame']==idx, ['x','y']]
    kps_for_recon.append(np.matrix(kps_data.values.transpose().astype(np.float32).copy()))
    
# get the camera calibration data
lcm = pd.read_csv(left_cam_cal_file, index_col=0)
lcm = lcm.values.astype(np.float32).copy()

# run scene reconstruction

cv2.sfm.reconstruct(kps_for_recon, lcm, None, None, True)

The code starts to run, then I get an error: error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

Based on how I understand the sfm::reconstruct function works, I think the error is coming from the line linked below.

https://github.com/opencv/opencv_contrib/blob/21e1c8b4480c89270193f688c4078e15ba1deb92/modules/sfm/src/reconstruct.cpp#L141

Camera matrix:

1685.72249 | 0.000000 | 689.894140
0.00000 | 1749.486859 | 767.726025
0.00000 | 0.000000 | 1.000000

Any help much appreciated. I’m really not sure where the error is coming from.

About this issue

Original URL
State: open
Created 6 years ago
Reactions: 2
Comments: 22 (10 by maintainers)

Commits related to this issue

Apply fix from https://github.com/opencv/opencv_contrib/issues/1675\#issuecomment-438413290 — committed to anibali/opencv_contrib by anibali 5 years ago

Most upvoted comments

Ok, strange behavior but I got it to work. I had to change the reconstruct.hpp file as such:

CV_EXPORTS_W //changed to allow python bindings to be built
void
reconstruct(InputArrayOfArrays points2d, OutputArrayOfArrays Ps, OutputArrayOfArrays points3d, InputOutputArray K, bool is_projective = false);

Seems like the object type of the input argument wasn’t correct, and hence the python bindings were generating the incorrect object types for the function. I’ve updated the code on my forked version of the repo. @berak are you a maintainer for the sfm module in the opencv_contrib repo?

themantalope on Jun 27, 2018

Ok found it, it’s interesting:

static PyObject* pyopencv_cv_sfm_reconstruct(PyObject* , PyObject* args, PyObject* kw)
{
    using namespace cv::sfm;

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    Mat Ps;
    PyObject* pyobj_points3d = NULL;
    Mat points3d;
    PyObject* pyobj_K = NULL;
    Mat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }
    PyErr_Clear();

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    UMat Ps;
    PyObject* pyobj_points3d = NULL;
    UMat points3d;
    PyObject* pyobj_K = NULL;
    UMat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }

    return NULL;
}

It appears that the order of the keyword arguments in the python wrapper don’t align with the order of arguments (passed?) to the C++ sfm::reconstruct function. However, this doesn’t have any bearing on functionality and another error is thrown. Based on what I see, it looks like the python wrapper correctly maps the python keyword arguments to the C++ arguments.

I’m wondering if the problem is that I’m using the latest version of OpenCV (4.0.0-pre) and OpenCV contrib modules, and the behavior of some of the core functions is a bit different.

themantalope on Jun 27, 2018

Ok, so I made the changes suggested and rebuilt. There are now modifications to the pyopencv_generated_funcs.h file:

static PyObject* pyopencv_cv_sfm_reconstruct(PyObject* , PyObject* args, PyObject* kw)
{
    using namespace cv::sfm;

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    vector_Mat Ps;
    PyObject* pyobj_points3d = NULL;
    Mat points3d;
    PyObject* pyobj_K = NULL;
    Mat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }
    PyErr_Clear();

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    vector_Mat Ps;
    PyObject* pyobj_points3d = NULL;
    UMat points3d;
    PyObject* pyobj_K = NULL;
    UMat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }

    return NULL;
}

And Psis now a vector mat. However, re-running the python code still gives the same error…

themantalope on Jun 27, 2018

Hey @berak

Here is the output of help(cv2.sfm.reconstruct):

Help on built-in function reconstruct:

reconstruct(...)
    reconstruct(points2d, K[, Ps[, points3d[, is_projective]]]) -> Ps, points3d, K
    .   @brief Reconstruct 3d points from 2d correspondences while performing autocalibration.
    .   @param points2d Input vector of vectors of 2d points (the inner vector is per image).
    .   @param Ps Output vector with the 3x4 projections matrices of each image.
    .   @param points3d Output array with estimated 3d points.
    .   @param K Input/Output camera matrix \f$K = \vecthreethree{f_x}{0}{c_x}{0}{f_y}{c_y}{0}{0}{1}\f$. Input parameters used as initial guess.
    .   @param is_projective if true, the cameras are supposed to be projective.
    .   
    .   This method calls below signature and extracts projection matrices from estimated K, R and t.
    .   
    .   @note
    .   - Tracks must be as precise as possible. It does not handle outliers and is very sensible to them.

Changing the input arguments to named arguments resulted in the same error:

cv2.sfm.reconstruct(kps_for_recon, K=lcm,is_projective=True)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-96-c9a12494b6e8> in <module>()
----> 1 cv2.sfm.reconstruct(kps_for_recon, K=lcm,is_projective=True)

error: OpenCV(4.0.0-pre) /Users/<path>/opencv/modules/core/src/matrix_wrap.cpp:1735: error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

themantalope on Jun 27, 2018