Jelle Pingen

Webcam-controlled 3D environment in Processing

Introduction

Added on February 21, 2011

Those of you who play videogames, and in particular first person (shooter) games, probably know what I’m talking out. When you reach a corner or a small doorway and want to know who’s hiding behind it, you automatically tilt your head sideways. Even though you know it won’t affect the game, since you can’t actually peek around the corner. I thought, why not?! Sure it must be technically possible, and I’m sure it will add a lot to the gameplay if your ‘real life’ gestures would actually mean something. Kind of like Microsoft’s Kinect, but without all the jumping up and down and waving your arms. Just small, natural gestures.

So when I saw an old webcam lying around, it gave me an idea. I want to able to peek around corners, or look around a 3D environment using just my webcam as a sensor.

Progress update #1: color tracking

Added on February 22, 2011

I decided upon the Processing language, since we had just learned some basic programming in Processing at university. Since it was my first time using a webcam, or even using video in combination with Processing, I started things off by trying some basic color tracking. I found a little green plastic piece, probably from an old toy or something, which I wanted to track. Because it was green (a color not present in the rest of my room) I thought it would be a good thing to track.

The basic idea about how I wanted the code to function is this: Import the video stream from the webcam first through USB. Then, for every incoming frame, scan every pixel and check that pixel’s RGB color values against the predefined color (in my case the green from the plastic piece). If the color is closer to the previous closest color, store this pixel’s coordinates. In the end, this would mean that the coordinates of the pixel with the color closest to the green plastic piece were stored. Then, draw a little red circle around this pixel.

This gave a pretty good result, where the red circle would actually follow the green piece when I moved it around. I refined the code a bit more though, to exclude pixels that were too different from the predifined color, to filter out random ‘jumps’ that would occur. If you want to have a look at the code, please leave a comment. I’d be happy to share it with anyone who is interested!

Progress update #2: face tracking!

Added on February 23, 2011

Now, the move from the basic color tracking I had to full face recognition was maybe a bit more difficult than I thought at first. I tried many different things, but I didn’t manage to program my own custom face recognition algorithm. After a lot of Googlin’ on how one would do something like that, I decided that for the time being it would be better to use a library to handle the face recognition. I stumbled upon the nice OpenCV library for Processing which has some great functions, one of which is face recognition. I played around with it a bit, and then coded the face tracking control.

The OpenCV library basically returns an X and Y coordinate of the detected face. I then mapped these coordinates to an angle between -40° and 40°. Lastly, the 3D environment is then rotated and translated according to the calculated angle. Sounds easy, right? Again, for those interested, I’m willing to share the code. Just leave a comment below!

Final product!

Added on February 24, 2011

It was then time to build the ‘final’ product. Well, not really final ofcourse. There is still a lot of room for improvements which I will discuss at the end of this post, but for now I will leave it as is. Maybe I might return to this project some day and go over all the code again, optimizing every last bit of it. Anyways, to demonstrate what I have built until now I made a really basic 3D room in Processing. It is just a cube, with some boxes floating in the center you can peek around. I colored the back box red, so you can clearly see the effect.

The video below has 2 screens to demonstrate how it works. The left screen is the 3D environment the user will see. Instead of a boring grey box, this can also be a highly-detailed game environment. The right screen is to show what the webcam ‘sees’. The red rectangle is drawn to show you how the face recognition works.

And that’s it!

Now I know there’s a lot to be improved. Firstly, it is quite slow. It has to scan every separate pixel of every frame to calculate where the face is. Since I’m using the OpenCV library, I cannot change this. However, if I ever find the time and courage I might try again to program my own face recognition algorithm. There are also a lot of jerky movements which can be filtered out. That’s also on the to-do list. Any more comments or suggestions are always welcome! When these bugs, and more, are fixed I can try to incorporate this in a real game. Someday…

Copyright © 2012 - Jelle Pingen
Proudly powered by WordPress MyRule on deviantART J_Pingen on Twitter