Virtual Reality Therapy

A couple days ago, one AR&VR friend talked with me online about the use of augmented reality as well as virtual reality. Sure in many aspects of our world they are able to be used. What we mainly discussed is in medical aspects. We both kown the AR technique is widely used in Surgical Navigation Systems which is truly grandios. But except it anything practical else? Well, suddenly I remembered a video clip I saw in Youtube about the Virtual Reality therapy for the injured soldiers who suffered a lot from the war. He was partly burn during the attack and suffered from the burn pain. So the doctors and researchers designed and generated a funny game about shooting penguin in a snow-covered environment. Certainly as a VR-application the patient will require a HMD as the view device and a joystick,too, as a interaction controller. It is said the white snow view and "cold" wind sound as well as the game fun can reduce the pain dramatically.

So I searched out the video and found its scientifical name -- "Virtual Reality Therapy" (VRT). And of course in wiki there is a detailed description(here). Simply to say it is a kind of psychotherapy with virtual reality technique. The patients have some fears of something, e.g. acrophobia. And the solution is absolutely let them confront this situation and gain the self-confidence. However they won't. Therefore a simulated case is generated as real as possible, and the patients konw that ok, this is the hell but I'm not dead. So they have the chance to face what they scare. According to the report the success rate of this therapy is 90%. Amazing!

In my opinion this therapy has its own advantages, like you don't really have to talk to your doctor during the therapy and you don't have to take pains to generate any illusion to pass the doctors' words. And it's more intuitive, cooler and more fancy. But the animation plays an important role. A nice, natural and not overacted animation can truly makes a peaceful immersion. But I'm afraid the opposite example could deteriorate the sickness. :p I know little knowledge about psychotherapy, but I believe the power of VR for it!

I dug three videos of VRT in Youtube, two of which are about the such therapy for the military applications:

Read more »

Posted in: Videos,Virtual Reality

Engineering passions

I have worked as a engineer since 1 year. To be honest, I do have improved my technical skills a lot, however the joy of my engineering job gets less and less. It seems something is wrong with me, but not my work.

I analysed myself as well as my life for months and read lots of books to find the causes. Until one day I remembered the words from Michael Jordan that, only if you love this game, you'd love to contribute your everything to it.

Oh, sure! Where is my passion for the technology I liked. To be a good engineer is as hard as to be a super NBA star. Without passion I don't think I'm able to go further like learning more about computer vision stuffs or writing a better code. The moment when I began to ignore the power of passion, every amazing part of the engineering mission looks faint and absolutely it is not what I want. Unfortunately the passion of engineering is not so obvious as a football match. How and where can I find as well as strengthen this passion is my next question.

Read more »

Posted in: engineering skills

Augmented Reality + Magic = ?

The magic is a fascinating show which can always bring us into mysterious imagination. However as introduced before, there is also an amazing technique which is able to produce the strong visual impact. Sure it is the Augmented Reality. So what could happen if merge these two stuffs? The next two videos can tell us:






The first video makes use of the markerless tracking tech. to determine the pose of an object, e.g. the card, and manipulate the animation based on the known pose. Certainly the guy who demonstrated the whole magic, is also a nice performer, however to take part in such a magic show the participant requires a HUD to observe the amazing animation.

In contrast, no extra hardwares are required for the participants. What you should do is just hold that board and enjoy the magic. But in my opinion the pose tracking of the board is realized by the inner coplanar markers, which means there must be an extra tracking instrument, e.g. an infrared camera, to determine the markers' pose. And if I didn't guess wrongly, the optimal performance time for this magic is night, which can guarantee the reflected light strength.

Anyway, they are two interesting magic Augmented Reality videos.

Read more »

Posted in: Augmented Reality,Videos,Virtual Reality

Parallel Tracking and Mapping for small AR Workspaces

Recently I read a paper about how to map the future points and determine the space coordinate system as well as track an object from university of Oxford. Actually since my thesis I have looked for the approaches estimating the pose and feature points-image points correspondence simutaneously. Some good candidates are like SoftPOSIT, RANSAC and Blind PnP. However during the practical implementation, I have found that the convergence ratio and run time are always the stumbling block for the practical applications, especially for the small number of feature points (between 4 to 11). Although this PTAM method requires large number of points, too, this is really an efficient and robust algorithm which could be applied in real time. Here let's have a look at a demo video and feel the power of this approach.



From this video I think most of us can have the intuitive experience of this PTAM approach. This method is mainly used for the hand-held camera pose estimation and it makes use of the dual-core computer to carry out the tracking and feature points mapping simutaneously. What surprises me more is that it requires no prior map before the tracking and a frame with 660 successful observations just used 18 ms. The implementation of this method for mapping and tracking respectively is certainly complecated, especially for the mapping. But if such an amazing application runs on your cellphone, it is still worth while. And here is another video of PTAM applications on IPhone.




And the paper of this approach is here.

Read more »

Posted in: Augmented Reality,Videos,Virtual Reality

Servus! I'm back.

Hello, everybody. It has been four months since I left my blog. During these four months it happend too much to me. I graduated from the uni and found a job which is about VR and AR. In addition I have set up a forum which is mainly about virtual reality and augmented reality in chinese language. Anyway I'd like to come back and manage my homepage sooner or later. And now I'd like to begin my new article. In the future I would like to share more info and source code about VR and AR in my homepage with the friends who have the interest. And welcome to contact with me by my new E-mail(linzsarvr@hotmail.com).

In anycase

Read more »

A quasi-Augmented Reality video

Naja, just found a very interesting vid. which I called a quasi augmented reality. In fact it is made totally in the reality world, but visible effect is like a virtual animation on their T-shirts. Two funny guys and an ingenious team PLUS a nice video. So enjoy it!!

Read more »

Posted in: Augmented Reality,Videos

Manipulation Interface(Input devices) for Virtual Reality

For Virtual Reality applications, or for every computer-based applications the input device plays always an important role in system implementation. In case of V.R. the input device is also called Manipulation Interface. And Grigore C. gave a nice definition: A manipulation interface is a device that allows the interactive change of the view to the virtual environment and exploration through the selection and manipulation of a virtual object of interest.

A classical input device are of course the quite familiar hardwares: keyborad and mouse. But as a manipulation interface, they are truly ....................





















Although so awful they are absolutely not, the limitation of space movement and less intuitive interaction way truly make us hard to see their perspective in the futures application of Virtual Reality. And a more intuitive device Trackball was developed and further improved as an important interface device for computer applications. The ball which is regarded as main function part of the device locates in a socket. And the socket is equiped with sensors which is able to detect the rotation of the ball and estimate the movement and final position of objects animated by computer. It could be manipulated by users more intuitively with diverse fingers' movement and through the combination of different function buttons, more functions can be realized more flexibly. However the location of this device is also fixed and the user's motion space is still limited.

In the recent ten years more and more powerful interaction devices are available with the special features and based on different implementation principles. They are either the trackers which has been introduced in my previous article i.e. Wii Remote, which I have also described here. It is used for the Nintendo's video game as a pose tracker. The user hold it and the approximate position as well as the rotation of Wii Remote could be detected and calculated by the signal receiver. Besides the Nintendo platform this remote device is also able to be used under the regular computer with proper setup. And as you know if it is connected with computer, unbelieable things will happen. I have found some interesting videos about it, first is how to set up the Wii Remote for computer applications and second one is a funny application about Head pose determination. Enjoy them:)





In addition to these devices I have also read about some unusual tools which are perhaps still under research period. One example is the so-called cubic mouse. It consists of a plastic cubic structure of dimensions 9cm x 9cm x 9cm. This plastic cube houses a Polhemus Fastrack, three mutually perpendicular translating rod, and six application-programmable control buttons. The buttons are placed on the face that normally is oriented upward, while wires from the tracker, the sensors embedded in the translating rods, and the buttons exit through the opposite face. A nice demo video is here:



Recently the movie "Avatar" is truly hot, the animation of this film is a typical V.R. application. Exactly speaking, it belongs to motion capture. Therefore the body suits for motion capture is another relevant input device for virtual reality. It is more likely a combination of a number of trackers, i.e. in the next video it is a optical-tracker-based suit. But the true principle is definitely not so simple just as a combination. It requires knowledge about biomechanics, photogrammetric methods and so on. Naja, it is a relatively large research aspect. Another interesting example is called Virtusphere, which is a spherical device with big size that could guarantee a common people move himself freely inside it. It rotates as the user walks, allowing for an unlimited plane upon which the user can walk and it is connected with computer to merge the human's movement into the virtual envirenmont. Oooook, have a look of them.....

Read more »

Posted in: Videos,Virtual Reality

Brave thinking leads to future!

Yesterday introduced by my friend, I saw the video about "Sixth Sense Tech." by Pranav Mistrywho does the research about Gestural Interface which is absolutely related to V.R. In this conference presentation-like video he has explained and shown his sixth sense system which bases on the camera tracking and remote projector illustrate the required images or dynamic visualization onto any possible flat surface or maybe any curved surface. What wonders me is not just his system or creative idea, but rather his attitude to his research and the courage to prove his thought, to realize his technical dream and to love what he is doing. In the last blog I have attended a dreamlike video of virtual reality application. Most of my friends told me it is impossible in reality, for example ........

Sure, if we don't make our first step to approach to our imagination, that amazing scene will be always far away from us ......

Read more »

Posted in: Videos,Virtual Reality

Nice video about Virtual Reality application in the future

Recently because of the thesis implementation the homepage has been ignored for a long time. I will be back soon after the presentation and maybe bring some modifications of my homepage layout. Anyway today I saw an amazing video of the Virtual Reality with the name "Build World" and this is the first time I was shocked by the V.R's application not only visually but also spiritually. Enjoy it together!

Read more »

Posted in: Videos,Virtual Reality

openCV -- 6: apply the mouse event

As one of the most important part of GUI, the interaction with the hardware such as keyboard and  mouse is always relavant for the user. OpenCV is able to offer some basic interaction functions for the mouse event's reaction. Mouse events are handled by a typical callback mechanism. So to enable response to mouse clicks, we must first write a callback routine that OpenCV can call whenever a mouse event occurs. Once we have done that, the callback will be registered with OpenCV, thereby informing OpenCV that this is the correct function to use whenever the user does something with the mouse over a particular window. With an example I'd like to introduce how to apply this mouse controlling event methods. The application is to use mouse drawing circles on the screen:

#include <cv.h>
#include <highgui.h>
#include <cmath>

void my_mouse_callback(
 int event, int x, int y, int flags, void* param
);

CvPoint center;
int radius;
bool drawing_circle = false;


void draw_circle(IplImage* img)
{
  cvCircle(
      img,
      center,
      radius,
      cvScalar(0x00,0xff,0x00)
      );
}

int main( int argc, char* argv[] ) {

IplImage* image = cvCreateImage(
  cvSize(500,500),
  IPL_DEPTH_8U,
  3
);
cvZero( image );
IplImage* temp = cvCloneImage( image );

cvNamedWindow( "circle Example" );

cvSetMouseCallback(
  "circle Example",
  my_mouse_callback,
  (void*) image
);


while( 1 ) {

  cvCopyImage( image, temp );
  if( drawing_circle ) draw_circle( temp);
  cvShowImage( "circle Example", temp );

  if( cvWaitKey( 15 )==27 ) break;
}

cvReleaseImage( &image );
cvReleaseImage( &temp );
cvDestroyWindow( "circle Example" );
}


void my_mouse_callback(
 int event, int x, int y, int flags, void* param )
{

IplImage* image = (IplImage*) param;

switch( event ) {
  case CV_EVENT_MOUSEMOVE: {
    if( drawing_circle ) {
      radius = int(sqrt(double((x-center.x)*(x-center.x)
               + (y - center.y)*(y - center.y))));
    }
  }
  break;
  case CV_EVENT_LBUTTONDOWN: {
    drawing_circle = true;
    center.x = x;
    center.y = y;
  }
  break; 
  case CV_EVENT_LBUTTONUP: {
    drawing_circle = false;
    draw_circle( image );
  }
  break; 
}
}

The function my_mouse_callback() is installed to respond to mouse events, and it uses the event to determine what to do when it is called. First it detects the event of mouse whether left button click, right button click or with the clicked button moving. According to the diverse cases different actions will be carried out for drawing a circle, e.g determing the center of a circle or calculating the radius of a circle or directly drawing the circle. The method cvCircle() is used to draw a circle with determined center and radius. The source code is available here, and below is a stupid bear with several circles:

Read more »

OpenCV -- 5: handling the subimage by ROI function

Today I'd like to introduce another interesting image processing function to handle a subimage of a picture to realize the image part replacement. This has been done by lots of image processing softwares, such as photoshop. Of cource they are much more professional. Here the example is just to demonstrate how to choose a region of a picture and process it as will. In my example I use the michael Jordan's head picture to take place of mine. Although the visual effect is not perfect, we could lernen the function of this ROI(region of interest) application. The code is below:

#include <cv.h>
#include <highgui.h>
#include <stdio.h>

int main(int argc, char** argv)
{
  IplImage *src1, *src2;
  if( argc == 9 && ((src1=cvLoadImage("lzsNew.jpg",1)) != 0
      )&&((src2=cvLoadImage("jordan.jpg",1)) != 0 ))//load the images
  {
      int x = 340;//set the beginning of region
      int y = 215;
      int width = 105;//set the size of regino
      int height = 105;
      double alpha = 0.25;//set how strong will original image influence
                          //the result
      double beta  = 0.75;//set how strong will added image influence
                          //the result
      cvSetImageROI(src1, cvRect(x,y,width,height));
      cvSetImageROI(src2, cvRect(20,5,width,height));
      cvAddWeighted(src1, alpha, src2, beta,0.0,src1);
      cvResetImageROI(src1);
      cvNamedWindow( "Alpha_blend", 1 );
      cvShowImage( "Alpha_blend", src1 );
  cvSaveImage("lzsReplace.jpg",src1);
      cvWaitKey();
  }
  else
    printf("Couldn't load one or both of %s, %s\n",argv[1],argv[2]);
  return 0;
}

As we can see above, the main job to implement it is as follows:

first, loading the two images by cvLoadImage(); and then set the range of the interesting region, including the beginning and the size as welll as the weighting parameters of each image. After that with the help of cvSetImageROI() we are able to active the region of the image which we just want to process, with this function during the matrix adding or other operations, just the actived region will take place, the others keep the same state as before. At last after the weighted added function cvAddWeighted(), we achieve the final result and save it as an image file. The final visual result is below. And the original code is here:

Read more »

OpenCV -- 4: operations of Matrix

Usually, OpenCV is used for computer vision applications. The friends who are familiar with computer vision must know how important is the linear algebra for computer vision, specially the matrix operations. As we know, by C++ programming, the assit about matrix calculation is always scarce. We are able to use array or vector data types to handle the matrix operation but at any rate it is not a efficient and intuitive method. But luckily with OpenCV library we are in the position to do the most of matrix calculations like adding matrix, getting the dimension or size of a matrix and so on. Here is a short introduction of some relevant matrix operations in OpenCV. Today I will give an example of how to calculata a SVD(singular value decomposision) of a matrix.

In linear algebra, SVD operation is used very commonly, like to achieve the pseudoinverse of a matrix, solving homogeneous linear operations and so forth. Before I have mentioned how to achieve a real rotation matrix with the help of SVD. And as I know lots of 3D pose estimation algorithms are based on the SVD calculations. Without the help of OpenCV library if we just implement it in C++ by ourselves it could be a large work. But just apply the function cvSVD(), everything will be solved. Here the example is about how to modify a matrix into a rotation matirx. The theory is descibed in the previous article(link above), and the C++ codes are written as follows:

#include "cv.h"
#include <stdio.h>
void main()
{
CvMat* mat = cvCreateMat(3,3,CV_32FC1);
cvZero(mat);

CV_MAT_ELEM( *mat, float, 0, 0 ) = 0.23839f;
CV_MAT_ELEM( *mat, float, 0, 1 ) = 0.957069f;
CV_MAT_ELEM( *mat, float, 0, 2 ) = 0.16489f;
CV_MAT_ELEM( *mat, float, 1, 0 ) = -0.101391f;
CV_MAT_ELEM( *mat, float, 1, 1 ) = 0.934907f;
CV_MAT_ELEM( *mat, float, 1, 2 ) = 0.340102f;
CV_MAT_ELEM( *mat, float, 2, 0 ) = 0.171344f;
CV_MAT_ELEM( *mat, float, 2, 1 ) = -0.0977953f;
CV_MAT_ELEM( *mat, float, 2, 2 ) = 0.31991f;

CvMat* U  = cvCreateMat(3,3,CV_32FC1);
CvMat* D  = cvCreateMat(3,3,CV_32FC1);
CvMat* V  = cvCreateMat(3,3,CV_32FC1);
CvMat* Result  = cvCreateMat(3,3,CV_32FC1);
cvSVD(mat, D, U, V, CV_SVD_V_T); // A = U D V^T
cvMatMul(U,V,Result);

float element1 = CV_MAT_ELEM(*Result,float,0,0);
printf("%f\n",element1);
float element2 = CV_MAT_ELEM(*Result,float,0,1);
printf("%f\n",element2);
float element3 = CV_MAT_ELEM(*Result,float,0,2);
printf("%f\n",element3);
float element4 = CV_MAT_ELEM(*Result,float,1,0);
printf("%f\n",element4);
float element5 = CV_MAT_ELEM(*Result,float,1,1);
printf("%f\n",element5);
float element6 = CV_MAT_ELEM(*Result,float,1,2);
printf("%f\n",element6);
float element7 = CV_MAT_ELEM(*Result,float,2,0);
printf("%f\n",element7);
float element8 = CV_MAT_ELEM(*Result,float,2,1);
printf("%f\n",element8);
float element9 = CV_MAT_ELEM(*Result,float,2,2);
printf("%f\n",element9);
}

Here in the codes there are several things which should be paid more attention:

cvMat* cvCreateMat(int rows, int cols, int type) is used to create a new two-dimensional matrix whereat the cvMat is a structure of a matrix data type, whose definition is carried out as follows:

typedef struct CvMat
{
   int type;
   int step;

   /* for internal use only */
   int* refcount;
   int hdr_refcount;

   union
   {
       uchar* ptr;
       short* s;
       int* i;
       float* fl;
       double* db;
   } data;

Actually it is just one of the ways to create a matrix, but applied commenly. The way to access a data in the matrix is also various, here I used the easy method with CV_MAT_ELEM() macro. This macro takes the matrix, the type of element to be retrieved, and the row and column numbers and then returns the element. cvSDV() is of course used to apply the SVD operation with the mark CV_SVD_V_T implied the output v is actually the transpose of v. Finally with the cvMatMul() for the multiplication of U and V the rotation matrix Result is obtained. The code is available here.

Read more »

Posted in: Codes,openCV

Demo of POSIT one camera tracking

Before I have mentioned the single camera tracking problem, and recently I have been engaged in this research with POSIT algorithm. With the A.R.T infrared camera tracking system (of course with single camera), I have implemented a test platform in C++ programing, with the 600 frames in about 10 seconds data. The brute-force method has been choosen here to find the best correspondence as well as the achievement of reconstructed transformation matrix. I have tested 4 times today, with 600 frames twice were perfect to find all the 600 frames correspondence, and one time found 579, another one was with 591. The reconstructed 3D errors are on average 19.8 mm with the distance between target and camera 1.6 meter. Below is a video I have made about this try, enjoy it!

Read more »

Posted in: tracking technology,Videos,Virtual Reality

OpenCV -- 3: two interesting image processing methods

Yesterday I have written an article about how to use openCV to process the video, including opening, processing and closing. In the processing part we have seen that actually it is just a process of every frame, that is to say, in the final analysis it is about image processing. And here I'd like to introduce some interesting and common image processing applications by OpenCV. Today I just focus on three functions: change the color image into gray image, Gaussian Pyramid Filter and the canny edge detection. What I want to process is the image "lzsNew.jpg".

It is obvious that it is a color image with three color channels R, G and B. Here if we want to achieve the gray image, the mission is to convert the image from color space to a gray space by the function cvCvtColor() with the parameter CV_RGB2GRAY or CV_BGR2GRAY. To generate a canny image in openCV the function cvCanny() is available, which finds the edges on the input image image and marks them in the output image edges using the Canny algorithm. The smallest of threshold1 and threshold2 is used for edge linking, the largest - to find initial segments of strong edges. To obtain a Gaussian Pyramid Filter cvPyrDown() and cvPyrUp() are workable. Here we choose the first one to decrease the image size as half. And the functions to load a image, release a image, create a window to show and destroy it, I have already introduced before. And below is the sample code:

#include "cv.h"
#include "highgui.h"

IplImage* doCanny(
IplImage* in,
double    lowThresh,
double    highThresh,
double    aperture)
{
IplImage* out = cvCreateImage(
cvGetSize( in ),
in->depth, //IPL_DEPTH_8U,    
1);
cvCanny( in, out, lowThresh, highThresh, aperture );
return( out );
};

IplImage* doPyrDown(
IplImage* in,
int       filter = IPL_GAUSSIAN_5x5)
{

// Best to make sure input image is divisible by two.
//
assert( in->width%2 == 0 && in->height%2 == 0 );

IplImage* out = cvCreateImage(
cvSize( in->width/2, in->height/2 ),
in->depth,
in->nChannels
);
cvPyrDown( in, out );
return( out );
};

int main( int argc, char** argv )
{
IplImage* img_rgb  = cvLoadImage( "lzsNew.jpg" );
IplImage* img_gry  = cvCreateImage( cvSize( img_rgb->width,img_rgb->height ),
                                 img_rgb->depth, 1);
cvCvtColor(img_rgb, img_gry ,CV_BGR2GRAY);
IplImage* img_pyr  = doPyrDown( img_gry, IPL_GAUSSIAN_5x5 );
IplImage* img_cny  = doCanny( img_pyr, 10, 100, 3 );

cvNamedWindow("Example Gray", CV_WINDOW_AUTOSIZE );
cvNamedWindow("Example Pyr", CV_WINDOW_AUTOSIZE );
cvNamedWindow("Example Canny", CV_WINDOW_AUTOSIZE );
cvShowImage("Example Gray", img_gry );
cvShowImage("Example Pyr", img_pyr );
cvShowImage("Example Canny", img_cny );
cvWaitKey(0);
cvSaveImage("lzsGray.jpg",img_gry);
cvSaveImage("lzsGrayDown.jpg",img_pyr);
cvSaveImage("lzsCanny.jpg",img_cny);
cvReleaseImage( &img_rgb);
cvReleaseImage( &img_gry);
cvReleaseImage( &img_pyr);
cvReleaseImage( &img_cny);
cvDestroyWindow("Example Gray");
cvDestroyWindow("Example Pyr");
cvDestroyWindow("Example Canny");
}

And after the program compilation we are able to get then three images below.

Gray Image

Gray Down

Canny Detection

With the same theory which is mentioned at the beginning that video is composed of several frames, so the video is also able to process by a Gaussian Pyramid Filter or canny detection. And yesterday I have posted a sample video, this one is the canny detection effect of that video.

Here is the code of today's image processing.

Read more »

Posted in: Codes,openCV,Videos