opencv_contrib: Using Facemark API (Python), Version 4.0.0 - pre : bad alloc error

finally:

https://github.com/berak/opencv_contrib/blob/b6e631bdc87c40260995080abce5f52cdbd453d9/modules/face/src/facemarkLBF.cpp#L375-L397

and all my tests pass (travis code here)

but it’s ugly. fixing the deficiancies of the python bindings from the inside of opencv code – can’t be really the answer. however, it’s far less invasive, than changing the interfaces.

thanks for all your help, btw !

it’s weird, roimat.type() is CV_32SC4 when the input comes from c++ or java, but CV_32S, when it’s from python. i guess, it’s better, to check

 roimat.depth() == CV_32S

instead.

(that’s why it returned an empty landmarks list)

hi, @krshrimali

well, i added the python / java bindings, but could only test the java ones properly, so this is probably my bad here.

indeed the roi.getObj() code is the problem:

https://github.com/opencv/opencv_contrib/blob/2231018c839d728811a39556ec83741bf9a27614/modules/face/src/facemarkLBF.cpp#L373-L376

the generated python wrappers for the fit() function produce this:

static PyObject* pyopencv_cv_face_face_Facemark_fit(PyObject* self, PyObject* args, PyObject* kw)
{
    using namespace cv::face;

    cv::face::Facemark* _self_ = NULL;
    if(PyObject_TypeCheck(self, &pyopencv_face_Facemark_Type))
        _self_ = dynamic_cast<cv::face::Facemark*>(((pyopencv_face_Facemark_t*)self)->v.get());
    if (_self_ == NULL)
        return failmsgp("Incorrect type of self (must be 'face_Facemark' or its derivative)");
    {
    PyObject* pyobj_image = NULL;
    Mat image;
    PyObject* pyobj_faces = NULL;
    Mat faces;
    PyObject* pyobj_landmarks = NULL;
    vector_Mat landmarks;
    bool retval;

    const char* keywords[] = { "image", "faces", "landmarks", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|O:face_Facemark.fit", (char**)keywords, &pyobj_image, &pyobj_faces, &pyobj_landmarks) &&
        pyopencv_to(pyobj_image, image, ArgInfo("image", 0)) &&
        pyopencv_to(pyobj_faces, faces, ArgInfo("faces", 0)) &&
        pyopencv_to(pyobj_landmarks, landmarks, ArgInfo("landmarks", 1)) )
    {
        ERRWRAP2(retval = _self_->fit(image, faces, landmarks));
        return Py_BuildValue("(NN)", pyopencv_from(retval), pyopencv_from(landmarks));
    }

see the Mat faces there ? the situation is similar to this:

// vector<Rect> f1; // would be ok.
Mat f1;            // problem !
InputArray f2(f1);
void *o = f2.getObj(); // o has the address of a Mat, not a vector<Rect>
std::vector<Rect> & faces = *(std::vector<Rect> *)o;  // oh noes, bad cast!
cout << faces.size(); // entire garbage

the java wrappers are a bit more lucky here:

        std::vector<Rect> faces;
        Mat& faces_mat = *((Mat*)faces_mat_nativeObj);
        Mat_to_vector_Rect( faces_mat, faces ); // explicit copy

but they had to be told, to do so:

https://github.com/opencv/opencv_contrib/blob/2231018c839d728811a39556ec83741bf9a27614/modules/face/misc/java/gen_dict.json#L12-L13

solution ? i have no immediate one. maybe the python wrapppers can be made smarter ? maybe the interface can be refactored (swap InputArray for a vector<Rect> ) ? maybe we can find a way to retrieve a vector from an InputArray without a horrible cast ? none of it nice or easy.

@tegusi – almost there !

can you check the kazemi version, again ?

Maybe you miss the face_landmark_model.dat download_link? The code has passed in my environment.

Besides it’s my fault that I should have made a new branch… If it’s possible to change the existing pr?

@berak Sorry, I just submitted the pr1 for opencv_contrib and pr2 for opencv, which should solve the problem and works fine on your tests. Since I’m not quite familiar with the whole process of pull request and it may take some time to merge into the master branch, you can try the one on my github page.

Thanks a lot for the hints you provided!

@berak : Thanks. Let me see if I can ask a student interested in GSOC to pick up this problem.

@krshrimali usually a pr (making a proper solution available to everyone else) should close this, please rather leave this open for now (mainly, – so other folks with the same problem have a better chance getting here).

till there is a sweeter solution!

exactly. it’s a duct-tape fix. it’s mending the symptom (segfaults all over), not the problem (python bindings are not smart enough)

well, no solution yet.

while

Mat roimat = roi.getMat();
std::vector<Rect> faces = roimat.reshape(4,roimat.rows);
if (faces.empty()) return false;

would fix problem #1 ,it still segfaults with the landmarks now (similar problem).

c++ and java use a vector<vector<Point2f>>, while python has a vector_Mat there (look at the gennerated code above). and it segfaults after leaving the fit() function, in pyopencv_to().

there are even correct pyopencv_to() versions for vector_Rect and vector_vector_Point2f in cv2.cpp , but as long as the parser generates wrong signatures, they can’t do the right thing.

yes, i get the segfault too, still inquiring. java & python wrappers are handling std::vector differently (unfortunately)

for a single face, i get a [1 x 1] Mat with type CV_32SC4, while from python i get [4 x 1] with type CV_32S (so, channels vs. columns)

and the python version crashes then, because the MatIterator tries to makes 4 Rects of it, and goesout of bounds this way.

i’ll try with a reshape(4,matroi.rows)

Update:

Earlier I was using cascade detector to detect faces, this time I used getFacesHAAR() function and since it returns a tuple, I had to input list(faces)[1] to fit function.

This returns segmentation fault though, but passes faces.empty() stage.

EDIT : The segmentation fault is in fitImpl() function, in the beginning where it accesses landmarks.size(). And there is further segmentation fault in:

landmarks = Mat(shape.reshape(2) + Scalar(min_x, min_y)

i might have found a solution: just accept, that we have a Mat here, and treat it like that.

instead of casting a void pointer:

 bool FacemarkLBFImpl::fit( InputArray image, InputArray roi, OutputArrayOfArrays  _landmarks ) 
 { 
     // FIXIT 
    std::vector<Rect> & faces = *(std::vector<Rect> *)roi.getObj(); 

retrieve the vector<Rect> from a Mat:

bool FacemarkLBFImpl::fit( InputArray image, InputArray roi, OutputArrayOfArrays  _landmarks )
{
    std::vector<Rect> faces;
    Mat roimat = roi.getMat(); // see issue #1661
    if ((!roimat.empty()) && (roimat.type()==CV_32SC4))
        faces.insert(faces.begin(), roimat.begin<Rect>(), roimat.end<Rect>());
    if (faces.empty()) return false;

this would work with both vector<Rect> and Mat also add a bit of type safety, imho. (ofc. the AAM and kazemi classes would need the same treatment)

@krshrimali , could you try it ? also, i’d be glad about any comments !