opencv: Charuco board detection is not suitable for calibration?

System Information

OpenCV python version: 4.7.0 Operating System / Platform: Debian GNU/Linux 11 (bullseye) Python version: 3.9.2

Detailed description

It’s my first time using charuco and I did not expect to encounter so many problems with this method.

According to the OpenCV docs, the idea of introducing Charuco was to make the classic chessboard pattern robust to partial view.

But when I ran the charuco corners detection, I realized that the detected chessboard corners did not match the real ones

According to the documentation and source code of cv::CharucoDetector::detectBoard(), this function doesn’t detect chessboard corners. Instead, it detects inner ArUco markers and uses them to estimate the camera pose and project back the chessboard corners. It has two options: when you have camera intrinsic parameters and when you don’t (it’s our option). For the second option, it uses a homography matrix to estimate the chessboard corners, which means that this method does not account for distortion. Therefore, the reprojected positions of corners may be incorrect. After that, the method calls the cv::cornerSubPix() method to refine the corners’ positions, and if the initial reprojection position is not far away from the real one, this call could produce accurate detection. That’s why not all corners are found incorrectly.

Although this method is mentioned in many tutorials, it is not suitable for calibration.

Steps to reproduce

I got this problem running my own code, but for baseline I ran the official example

./example_aruco_calibrate_camera_charuco -v=<...>/frame-%03d.png -d 0 -h=6 -w=4 --sl=0.3175 --ml=0.2540 --sc --rs

Issue submission checklist

I report the issue, it’s not a question
I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
I updated to the latest OpenCV version and the issue is still there
There is reproducer code and related data files (videos, images, onnx, etc)

About this issue

Original URL
State: open
Created a year ago
Reactions: 4
Comments: 20 (16 by maintainers)

Most upvoted comments

@victor1234, I found the problem. You are using the deprecated ChArUco template. Just add this code to fix your problem:

board.setLegacyPattern(True)

out

Charuco templates was updated in https://github.com/opencv/opencv_contrib/pull/3174. The behavior has changed only for templates with an even number of rows. This was done to make the ChArUco patterns compatible with the chessboard. Use gen_pattern.py to create new pattern or use setLegacyPattern.

Also OpenCV team tested CharucoDetector with another camera: The results were also quite good. Nevertheless, we need to make a big comparison of findChessboardCorners vs CharucoDetector. If the CharucoDetector does not work well enough, the OpenCV team can try to improve the algorithm using cornerHarris or other methods. But we need a large dataset and a community request.

AleksandrPanov on Jul 5, 2023

Hi,

I am also facing some problems with ChAruCo detection. This is the board used (opencv-python, 4.8)

marker_length: 7.35294117647 #mm
square_length: 12.5 #mm
pattern_size: [20,10] 
aruco_dict: 1 #aruco.DICT_4X4_100

code used to detect corners:

charuco_detector_left = cv2.aruco.CharucoDetector(board, charucoParams_left, detectorParams, refineParams)
...
charucoCorners_left_i, charucoIds_left_i, _ ,_  =  charuco_detector_left.detectBoard(image_left)

Images:

19l

19l_detected encv/opencv/assets/46029153/8635fa6e-6832-40ce-b947-2ec8ef951498)

If you would like, we can supply a lot more images

Thanks, Alon

AlonSpinner on Aug 1, 2023

@victor1234 Thanks for the feedback. We are looking on options how to rework Charuco pipeline and data with corner cases will be useful. Could you share some images with us?

#24117

victor1234 on Aug 7, 2023

The script to generate the ChArUco board:

cols = 4
rows = 6
square_size = 100
marker_size = 0.8*square_size
aruco_dict = cv.aruco.getPredefinedDictionary(0) # cv.aruco.DICT_4X4_250
board = cv.aruco.CharucoBoard((cols, rows), square_size, marker_size, aruco_dict)
board.setLegacyPattern(True) # comment this line to create the new template
from_cv_img = board.generateImage((cols*square_size, rows*square_size))
cv.imshow("board_6_4_dict_0", from_cv_img)
cv.imwrite("board_6_4_dict_0.png", from_cv_img)

The deprecated ChArUco template: board_6_4_dict_0

The new ChArUco template (this template is compatible with the chessboard): board_6_4_dict_0

AleksandrPanov on Jul 4, 2023