image processing - Camera calibration and extrinsic parameters for perspective transformation

I'm working in a photogrammetry camera and laser application where I need to measure a displacement in a specific plane, this is a quick diagram of the setup:

In order to get proper measurements I'm trying to do a projection from the 3d points to the 2d camera plane proceeding like this:

• First I'm doing the Camera calibration using opencv and a chessboard, I'm taking a few chessboard shots in different angles and applying the function initUndistortRectifyMap, I´ll have the distortion coefficients, intrinsic and extrinsic parameters.


• With the intrinsic parameters and the coefficients I'll apply lens correction.


• The extrinsic parameters of the previous step are for each chessboard position and I assume that they won't be valid for my particular application, so I'm placing a chessboard in the above marked area as "Measurement Plane" and running again the initUndistortRectifyMap function which this time it's giving me, I think, correct extrinsic parameters of the ROI.


• Once I've got the "correct" intrinsic and extrinsic parameters I´m applying the camera matrix to calculate the 2d points (u,v) projection:

My questions are:

Is it the correct way to proceed for this application?

The camera system only reads X,Y pixel points each time, but to perform a 2d projection seems like I'll need the Z, how could I have this last coordinate of the 3d point?

Answer

You can bypass all the OpenCV stuff, and create a custom mapping from pixel location to real world location of the form:

$$ x = A_x m^2 + B_x mn + C_x n^2 + D_x m + E_x n + F_x $$ $$ y = A_y m^2 + B_y mn + C_y n^2 + D_y m + E_y n + F_y $$

Where $(x,y)$ is the location on the plane using a grid on the plane, and $(m,n)$ is the pixel location in the image.

To find the coefficients simply requires a set of calibration points and some linear algebra. You will need at least six points, but more points spread out over you entire viewing field of operation would be better.

Establish a grid on your plane. A point on the plane will be $(x_i,y_i)$ which corresponds to a point $(m_i,n_i)$ on the image. N is the number of calibration points.

Construct a matrix equation like this:

$$ \left[ \begin{array}{ccccccc} m_1^2 & m_1 n_1 & n_1^2 & m_1 & n_1 & 1 \\ m_2^2 & m_2 n_2 & n_2^2 & m_2 & n_2 & 1 \\ .&.&.&.&.&.\\ .&.&.&.&.&.\\ m_i^2 & m_i n_i & n_i^2 & m_i & n_i & 1 \\ .&.&.&.&.&.\\ .&.&.&.&.&.\\ m_N^2 & m_N n_i & n_N^2 & m_N & n_N & 1 \\ \end{array} \right] \cdot \left[ \begin{array}{cc} A_x & A_y \\ B_x & B_y \\ C_x & C_y \\ D_x & D_y \\ E_x & E_y \\ F_x & F_y \\ \end{array} \right] = \left[ \begin{array}{cc} x_1 & y_1 \\ x_2 & y_2 \\ .&.\\ x_i & y_i \\ .&.\\ x_N & y_N \\ \end{array} \right] $$

Which is in the form:

$$ M \cdot U = R $$

Multiply both sides by $ M^T $.

$$ M^TM U = M^T R $$

The product, $M^TM$, is a 6x6 matrix. Find its inverse and multiply both sides by it.

$$ U = (M^TM)^{-1}M^T R $$

The coefficients you are seeking are the elements of U.

The same technique can be used to create a mapping in the reverse direction as well.

Hope this helps.

Ced