Webcam-based Head Tracking with linux-track

There’s a new breed of RC hobbyist these days – the First Person View (FPV) pilot/driver.  These folks install small cameras on their remote control vehicles and pilot/drive them by looking at a video monitor (generally a head mounted display ie. video goggles) rather than by watching the vehicle directly. A couple of example videos are here and here. It’s pretty cool, and if I was into RC I might try to set up something like this.  The part that caught my attention from a robotics standpoint was the head tracking aspect.  These guys use video goggles with a gyro that reports their head position back to the camera on the aircraft, which is mounted on a pan-tilt assembly using servos.  This allows them to remotely pan or tilt the camera just by turning their head.  Pilot looks to their right, camera swivels to the right.  Neat!

I did some research and discovered that there are a bunch of different methods of head tracking.  You can use gyros like the FPV pilots do, or you could try an optical method of head tracking, ie. with a camera.  There are commercial solutions like the TrackIR, which is mainly intended for video gaming. (Did I mention that head tracking is huge in the flightsim and FPS gaming world now too?)  The bad news is that the TrackIR costs $150, which in my mind is altogether too much to pay for what is essentially a webcam.  So why not just use a normal webcam for face tracking?  Well, it turns out that you can do exactly that – software like FaceAPI and FreeTrack can use any video source to track and quantify your head movement.  A little more digging and I found Linux-track, a FreeTrack clone for linux. Perfect, let’s play with it!

First we will add linux-track’s repository to apt, then we’ll update apt and install the package. (I’m using Ubuntu 11.10 – your sources.list entry will change based on your Ubuntu version, see this page for details.)

wget --quiet -O - | sudo apt-key add -
echo "deb oneiric main"  | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install linuxtrack

Linux-track should now be installed in the “games” menu.

OK, let’s take a break from the software side and take a look at some of the hardware requirements of head tracking.  Some software (ie FaceAPI) observes facial features and relative head size to calculate pose information.  Linux-track is not so smart. It needs a “point model” – a series of LEDs or IR reflectors in a known configuration that it can track.  With three “points” in a known pattern it can determine your head pose in 6 degrees of freedom (rotation, pitch, tilt, movement in X, Y and Z axes).  For my purposes of pan and tilt, I need only 2DOF (rotation and pitch), and so I can get away with a single-point model – just 1 LED or IR reflector.  I found a cheap clip-on LED keychain in the garage which will be perfect for this.

Because I’m only interested in seeing the light emitted from my LED “point model” and the LED will emit lots of IR as well as visible spectrum, I chose to add an IR filter to my webcam.  A small piece of exposed film negative (man, was that tough to find!) taped over the lens of the webcam will act as an effective IR filter.  There are also other options out there for DIY IR filters.  Finally, clipping the LED light to the brim of a ball cap finishes off the hardware side of this project.

Back in the linux-track config GUI, I created a new point model, specifying a single-point LED model.  I played with the sensitivity settings and visible blob threshold until I had a smooth-moving point.  All your preferences are stored in .linuxtrack/linuxtrack.conf in your home folder.

In my next post, I’ll be using ltr-pipe to create a virtual joystick that will drive a remote pan-tilt camera mount.  Stay tuned for that.