computer vision - Edge detection on a color image

I understand the process of using a Sobel kernel for edge detection in greyscale images. The input is a greyscale image, and the output is a greyscale image. I'm having trouble, however, figuring out how to apply the Sobel kernel to a color image.

Do I find the color gradient for each color channel (R, G, and B) then average each value to get the final color gradient value? Or do I average the color channels then find the color gradient of that image?

My goal is to render a raytraced image, find the edges, then re-render only the high-contrast regions with anti-aliasing and leave the low-contrast regions alone.

Answer

Finding edges in a color image can be done by decomposing the image into its channels, finding the gradients separately and fusing them somehow. However, such approach doesn't incorporate the color components in a joint model. Luckily, there is a better way to do this, which is the structure tensor representation.

The color structure tensor describes the bi-dimensional first order differential structure at a certain point in the image. It is specified by:

$ S= \left[ {\begin{array}{cc} R_x^2+G_x^2+B_x^2 & R_xR_y+G_xG_y+B_xB_y \\ R_xR_y+G_xG_y+B_xB_y & R_y^2+G_y^2+B_y^2 \\ \end{array} } \right] $

where subscripts denote the partial derivatives. This is a more precise description of the local gradients. Eigen-decomposition is then applied to the structure tensor matrix $S$ to form the eigenvalues and eigenvectors. The larger eigenvalue shows the strength of the local image edges (gradient magnitude) and the corresponding eigenvector points across the edge (in gradient direction). In other words, eigenvalues encode the gradient magnitude while the eigenvectors contain the gradient orientation information. Note that the eigenvectors and eigenvalues can be computed analytically for a structure tensor.

There are many other usecases of the structure tensor, so it is good to know. Last but not least, here is a MATLAB code, which uses this information to compute the gradients:

http://www.mathworks.com/matlabcentral/fileexchange/28114-fast-edges-of-a-color-image-actual-color-not-converting-to-grayscale/content/coloredges.m

After obtaining the gradients, you could further carry on your edge detection in a standard fashion (canny etc.).