openCV -- 6: apply the mouse event

As one of the most important part of GUI, the interaction with the hardware such as keyboard and  mouse is always relavant for the user. OpenCV is able to offer some basic interaction functions for the mouse event's reaction. Mouse events are handled by a typical callback mechanism. So to enable response to mouse clicks, we must first write a callback routine that OpenCV can call whenever a mouse event occurs. Once we have done that, the callback will be registered with OpenCV, thereby informing OpenCV that this is the correct function to use whenever the user does something with the mouse over a particular window. With an example I'd like to introduce how to apply this mouse controlling event methods. The application is to use mouse drawing circles on the screen:

#include <cv.h>
#include <highgui.h>
#include <cmath>

void my_mouse_callback(
 int event, int x, int y, int flags, void* param
);

CvPoint center;
int radius;
bool drawing_circle = false;


void draw_circle(IplImage* img)
{
  cvCircle(
      img,
      center,
      radius,
      cvScalar(0x00,0xff,0x00)
      );
}

int main( int argc, char* argv[] ) {

IplImage* image = cvCreateImage(
  cvSize(500,500),
  IPL_DEPTH_8U,
  3
);
cvZero( image );
IplImage* temp = cvCloneImage( image );

cvNamedWindow( "circle Example" );

cvSetMouseCallback(
  "circle Example",
  my_mouse_callback,
  (void*) image
);


while( 1 ) {

  cvCopyImage( image, temp );
  if( drawing_circle ) draw_circle( temp);
  cvShowImage( "circle Example", temp );

  if( cvWaitKey( 15 )==27 ) break;
}

cvReleaseImage( &image );
cvReleaseImage( &temp );
cvDestroyWindow( "circle Example" );
}


void my_mouse_callback(
 int event, int x, int y, int flags, void* param )
{

IplImage* image = (IplImage*) param;

switch( event ) {
  case CV_EVENT_MOUSEMOVE: {
    if( drawing_circle ) {
      radius = int(sqrt(double((x-center.x)*(x-center.x)
               + (y - center.y)*(y - center.y))));
    }
  }
  break;
  case CV_EVENT_LBUTTONDOWN: {
    drawing_circle = true;
    center.x = x;
    center.y = y;
  }
  break; 
  case CV_EVENT_LBUTTONUP: {
    drawing_circle = false;
    draw_circle( image );
  }
  break; 
}
}

The function my_mouse_callback() is installed to respond to mouse events, and it uses the event to determine what to do when it is called. First it detects the event of mouse whether left button click, right button click or with the clicked button moving. According to the diverse cases different actions will be carried out for drawing a circle, e.g determing the center of a circle or calculating the radius of a circle or directly drawing the circle. The method cvCircle() is used to draw a circle with determined center and radius. The source code is available here, and below is a stupid bear with several circles:

Read more »

OpenCV -- 5: handling the subimage by ROI function

Today I'd like to introduce another interesting image processing function to handle a subimage of a picture to realize the image part replacement. This has been done by lots of image processing softwares, such as photoshop. Of cource they are much more professional. Here the example is just to demonstrate how to choose a region of a picture and process it as will. In my example I use the michael Jordan's head picture to take place of mine. Although the visual effect is not perfect, we could lernen the function of this ROI(region of interest) application. The code is below:

#include <cv.h>
#include <highgui.h>
#include <stdio.h>

int main(int argc, char** argv)
{
  IplImage *src1, *src2;
  if( argc == 9 && ((src1=cvLoadImage("lzsNew.jpg",1)) != 0
      )&&((src2=cvLoadImage("jordan.jpg",1)) != 0 ))//load the images
  {
      int x = 340;//set the beginning of region
      int y = 215;
      int width = 105;//set the size of regino
      int height = 105;
      double alpha = 0.25;//set how strong will original image influence
                          //the result
      double beta  = 0.75;//set how strong will added image influence
                          //the result
      cvSetImageROI(src1, cvRect(x,y,width,height));
      cvSetImageROI(src2, cvRect(20,5,width,height));
      cvAddWeighted(src1, alpha, src2, beta,0.0,src1);
      cvResetImageROI(src1);
      cvNamedWindow( "Alpha_blend", 1 );
      cvShowImage( "Alpha_blend", src1 );
  cvSaveImage("lzsReplace.jpg",src1);
      cvWaitKey();
  }
  else
    printf("Couldn't load one or both of %s, %s\n",argv[1],argv[2]);
  return 0;
}

As we can see above, the main job to implement it is as follows:

first, loading the two images by cvLoadImage(); and then set the range of the interesting region, including the beginning and the size as welll as the weighting parameters of each image. After that with the help of cvSetImageROI() we are able to active the region of the image which we just want to process, with this function during the matrix adding or other operations, just the actived region will take place, the others keep the same state as before. At last after the weighted added function cvAddWeighted(), we achieve the final result and save it as an image file. The final visual result is below. And the original code is here:

Read more »

OpenCV -- 4: operations of Matrix

Usually, OpenCV is used for computer vision applications. The friends who are familiar with computer vision must know how important is the linear algebra for computer vision, specially the matrix operations. As we know, by C++ programming, the assit about matrix calculation is always scarce. We are able to use array or vector data types to handle the matrix operation but at any rate it is not a efficient and intuitive method. But luckily with OpenCV library we are in the position to do the most of matrix calculations like adding matrix, getting the dimension or size of a matrix and so on. Here is a short introduction of some relevant matrix operations in OpenCV. Today I will give an example of how to calculata a SVD(singular value decomposision) of a matrix.

In linear algebra, SVD operation is used very commonly, like to achieve the pseudoinverse of a matrix, solving homogeneous linear operations and so forth. Before I have mentioned how to achieve a real rotation matrix with the help of SVD. And as I know lots of 3D pose estimation algorithms are based on the SVD calculations. Without the help of OpenCV library if we just implement it in C++ by ourselves it could be a large work. But just apply the function cvSVD(), everything will be solved. Here the example is about how to modify a matrix into a rotation matirx. The theory is descibed in the previous article(link above), and the C++ codes are written as follows:

#include "cv.h"
#include <stdio.h>
void main()
{
CvMat* mat = cvCreateMat(3,3,CV_32FC1);
cvZero(mat);

CV_MAT_ELEM( *mat, float, 0, 0 ) = 0.23839f;
CV_MAT_ELEM( *mat, float, 0, 1 ) = 0.957069f;
CV_MAT_ELEM( *mat, float, 0, 2 ) = 0.16489f;
CV_MAT_ELEM( *mat, float, 1, 0 ) = -0.101391f;
CV_MAT_ELEM( *mat, float, 1, 1 ) = 0.934907f;
CV_MAT_ELEM( *mat, float, 1, 2 ) = 0.340102f;
CV_MAT_ELEM( *mat, float, 2, 0 ) = 0.171344f;
CV_MAT_ELEM( *mat, float, 2, 1 ) = -0.0977953f;
CV_MAT_ELEM( *mat, float, 2, 2 ) = 0.31991f;

CvMat* U  = cvCreateMat(3,3,CV_32FC1);
CvMat* D  = cvCreateMat(3,3,CV_32FC1);
CvMat* V  = cvCreateMat(3,3,CV_32FC1);
CvMat* Result  = cvCreateMat(3,3,CV_32FC1);
cvSVD(mat, D, U, V, CV_SVD_V_T); // A = U D V^T
cvMatMul(U,V,Result);

float element1 = CV_MAT_ELEM(*Result,float,0,0);
printf("%f\n",element1);
float element2 = CV_MAT_ELEM(*Result,float,0,1);
printf("%f\n",element2);
float element3 = CV_MAT_ELEM(*Result,float,0,2);
printf("%f\n",element3);
float element4 = CV_MAT_ELEM(*Result,float,1,0);
printf("%f\n",element4);
float element5 = CV_MAT_ELEM(*Result,float,1,1);
printf("%f\n",element5);
float element6 = CV_MAT_ELEM(*Result,float,1,2);
printf("%f\n",element6);
float element7 = CV_MAT_ELEM(*Result,float,2,0);
printf("%f\n",element7);
float element8 = CV_MAT_ELEM(*Result,float,2,1);
printf("%f\n",element8);
float element9 = CV_MAT_ELEM(*Result,float,2,2);
printf("%f\n",element9);
}

Here in the codes there are several things which should be paid more attention:

cvMat* cvCreateMat(int rows, int cols, int type) is used to create a new two-dimensional matrix whereat the cvMat is a structure of a matrix data type, whose definition is carried out as follows:

typedef struct CvMat
{
   int type;
   int step;

   /* for internal use only */
   int* refcount;
   int hdr_refcount;

   union
   {
       uchar* ptr;
       short* s;
       int* i;
       float* fl;
       double* db;
   } data;

Actually it is just one of the ways to create a matrix, but applied commenly. The way to access a data in the matrix is also various, here I used the easy method with CV_MAT_ELEM() macro. This macro takes the matrix, the type of element to be retrieved, and the row and column numbers and then returns the element. cvSDV() is of course used to apply the SVD operation with the mark CV_SVD_V_T implied the output v is actually the transpose of v. Finally with the cvMatMul() for the multiplication of U and V the rotation matrix Result is obtained. The code is available here.

Read more »

Posted in: Codes,openCV

Demo of POSIT one camera tracking

Before I have mentioned the single camera tracking problem, and recently I have been engaged in this research with POSIT algorithm. With the A.R.T infrared camera tracking system (of course with single camera), I have implemented a test platform in C++ programing, with the 600 frames in about 10 seconds data. The brute-force method has been choosen here to find the best correspondence as well as the achievement of reconstructed transformation matrix. I have tested 4 times today, with 600 frames twice were perfect to find all the 600 frames correspondence, and one time found 579, another one was with 591. The reconstructed 3D errors are on average 19.8 mm with the distance between target and camera 1.6 meter. Below is a video I have made about this try, enjoy it!

Read more »

Posted in: tracking technology,Videos,Virtual Reality

OpenCV -- 3: two interesting image processing methods

Yesterday I have written an article about how to use openCV to process the video, including opening, processing and closing. In the processing part we have seen that actually it is just a process of every frame, that is to say, in the final analysis it is about image processing. And here I'd like to introduce some interesting and common image processing applications by OpenCV. Today I just focus on three functions: change the color image into gray image, Gaussian Pyramid Filter and the canny edge detection. What I want to process is the image "lzsNew.jpg".

It is obvious that it is a color image with three color channels R, G and B. Here if we want to achieve the gray image, the mission is to convert the image from color space to a gray space by the function cvCvtColor() with the parameter CV_RGB2GRAY or CV_BGR2GRAY. To generate a canny image in openCV the function cvCanny() is available, which finds the edges on the input image image and marks them in the output image edges using the Canny algorithm. The smallest of threshold1 and threshold2 is used for edge linking, the largest - to find initial segments of strong edges. To obtain a Gaussian Pyramid Filter cvPyrDown() and cvPyrUp() are workable. Here we choose the first one to decrease the image size as half. And the functions to load a image, release a image, create a window to show and destroy it, I have already introduced before. And below is the sample code:

#include "cv.h"
#include "highgui.h"

IplImage* doCanny(
IplImage* in,
double    lowThresh,
double    highThresh,
double    aperture)
{
IplImage* out = cvCreateImage(
cvGetSize( in ),
in->depth, //IPL_DEPTH_8U,    
1);
cvCanny( in, out, lowThresh, highThresh, aperture );
return( out );
};

IplImage* doPyrDown(
IplImage* in,
int       filter = IPL_GAUSSIAN_5x5)
{

// Best to make sure input image is divisible by two.
//
assert( in->width%2 == 0 && in->height%2 == 0 );

IplImage* out = cvCreateImage(
cvSize( in->width/2, in->height/2 ),
in->depth,
in->nChannels
);
cvPyrDown( in, out );
return( out );
};

int main( int argc, char** argv )
{
IplImage* img_rgb  = cvLoadImage( "lzsNew.jpg" );
IplImage* img_gry  = cvCreateImage( cvSize( img_rgb->width,img_rgb->height ),
                                 img_rgb->depth, 1);
cvCvtColor(img_rgb, img_gry ,CV_BGR2GRAY);
IplImage* img_pyr  = doPyrDown( img_gry, IPL_GAUSSIAN_5x5 );
IplImage* img_cny  = doCanny( img_pyr, 10, 100, 3 );

cvNamedWindow("Example Gray", CV_WINDOW_AUTOSIZE );
cvNamedWindow("Example Pyr", CV_WINDOW_AUTOSIZE );
cvNamedWindow("Example Canny", CV_WINDOW_AUTOSIZE );
cvShowImage("Example Gray", img_gry );
cvShowImage("Example Pyr", img_pyr );
cvShowImage("Example Canny", img_cny );
cvWaitKey(0);
cvSaveImage("lzsGray.jpg",img_gry);
cvSaveImage("lzsGrayDown.jpg",img_pyr);
cvSaveImage("lzsCanny.jpg",img_cny);
cvReleaseImage( &img_rgb);
cvReleaseImage( &img_gry);
cvReleaseImage( &img_pyr);
cvReleaseImage( &img_cny);
cvDestroyWindow("Example Gray");
cvDestroyWindow("Example Pyr");
cvDestroyWindow("Example Canny");
}

And after the program compilation we are able to get then three images below.

Gray Image

Gray Down

Canny Detection

With the same theory which is mentioned at the beginning that video is composed of several frames, so the video is also able to process by a Gaussian Pyramid Filter or canny detection. And yesterday I have posted a sample video, this one is the canny detection effect of that video.

Here is the code of today's image processing.

Read more »

Posted in: Codes,openCV,Videos

OpenCV -- 2: how to read, process and write a video

In the previous article I have written something about openCV. At that time it is about how to set the Visual Studio to use OpenCV library and a simple example about how to read an image by cvLoadImage() function. Besides statical image processing, openCV is also able to handle dynamical images -- video. Here I'd like to use a example to explain some relevant functions. Below is the video about a short introduction of the code and the visual result about how to open an available video and process it in a determined way as well as store another video in avi form:

from the video we have seen the structure of the codes. The thought of this code is first generating proper number of windows to demonstrate the videos. Actually video is composed of every single image, so the method to process the image is in fact the core of processing. The whole processing is just done during one while loop until all frames are handled or press the "ESC" button. Of course to implement the functions there are more details for us to care about, such as achieving the infromation of original image and set them for the new constructed video. And for different purposes the image processing methods are also diverse. And here are the codes of the implementation:

#include "cv.h"
#include "highgui.h"
#include <stdio.h>



int main( int argc, char* argv[] ) {
 cvNamedWindow( "Color_Image", CV_WINDOW_AUTOSIZE );
 cvNamedWindow( "Gray_Image", CV_WINDOW_AUTOSIZE );
 cvNamedWindow( "Log_Polar", CV_WINDOW_AUTOSIZE );
 CvCapture* capture = cvCreateFileCapture( "lzs.avi" );
 if (!capture){
     return -1;
 }
 IplImage* bgr_frame;
 double fps = cvGetCaptureProperty (
     capture,
     CV_CAP_PROP_FPS
 );
 printf("fps=%d\n",(int)fps);

 CvSize size = cvSize(
     (int)cvGetCaptureProperty( capture,
                                CV_CAP_PROP_FRAME_WIDTH),
     (int)cvGetCaptureProperty( capture,
                                CV_CAP_PROP_FRAME_HEIGHT)
 );

 printf("frame (w, h) = (%d, %d)\n",size.width,size.height);
 CvVideoWriter* writer = cvCreateVideoWriter(
     "cope.avi",                            
     CV_FOURCC('D','X','5','0'), 
     fps,
     size
 );

 IplImage* logpolar_frame = cvCreateImage(
     size,
     IPL_DEPTH_8U,
     3
 );

 IplImage* gray_frame = cvCreateImage(
     size,
     IPL_DEPTH_8U,
     1
 );

 while( (bgr_frame=cvQueryFrame(capture)) != NULL ) {
     cvShowImage( "Color_Image", bgr_frame );
     cvConvertImage(
         bgr_frame,
         gray_frame,
         0
     );
  
     cvLogPolar( bgr_frame, logpolar_frame,
                 cvPoint2D32f(bgr_frame->width/4,
                 bgr_frame->height/4),
                 60,
                 CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS );
     cvShowImage("Gray_Image", gray_frame);
     cvShowImage( "Log_Polar", logpolar_frame );
     cvWriteToAVI( writer, logpolar_frame );
     char c = cvWaitKey(30);
     if( c == 27 ) break;
 }

 cvReleaseVideoWriter( &writer );
 cvReleaseImage( &gray_frame );
 cvReleaseImage( &logpolar_frame );
 cvReleaseCapture( &capture );
}

Inside it the meaning of some significant functions are like this:

cvNamedWindow( "Color_Image", CV_WINDOW_AUTOSIZE ): it is used to generate a new window to illustrate a image or video;

CvCapture* capture = cvCreateFileCapture( "lzs.avi" ): it is used to capture or read a video from the determind path;

CvVideoWriter* writer = cvCreateVideoWriter("cope.avi",CV_FOURCC('D','X','5','0'), fps,size): it is used to create a new video with fixed name and determinded characters;

IplImage* gray_frame = cvCreateImage( size,IPL_DEPTH_8U,1): it is used to create a new image with the expected requirements, here the processed image will be stored here for a further processing;

cvConvertImage() and cvLogPolar() are two special functions in openCV to process the images, in the articles in the future, I will introduce them in details;

cvShowImage("Gray_Image", gray_frame): is the function to demonstrate the image or video;

cvWriteToAVI( writer, logpolar_frame ): write down the new generated video into the disk;

Because the names of the functions are able to imply the application and meanings very obviously, so I will just attach my project code here for you. The detailed description you can find in openCV wiki or reference site here.

Read more »

Posted in: Codes,openCV,Videos