Please see the HandVu page for up-to-date information about this topic.

Vision-based Hand Gesture Recognition and User Interface

Overview

With almost any color camera and sufficient processing power, this software collection implements a hand gesture interface. It detects the hand in a standard posture, then tracks it and recognizes key postures - all in real-time and without the need for camera or user calibration. The output is accessible through a client-server infrastructure in a custom format and as OpenSound Control packets, allowing control of music applications that use this format.
This video (in WMV format) shows the tracking component in action.

Requirements

Intel PC with 1GHz or faster CPU
Windows 2k or XP
DirectShow - alpha2 was compiled with DX9
Administrator rights (to register COM objects)
Intel Image Processing Library (IPL) binaries; these are not freely available anymore but can probably be substituted with IPP; future versions (scheduled for end of March) will not depend on Intel products
OpenCV binaries
Magick binaries (alpha2 was compiled with 5.5.7-Q8)
Camera with DirectShow source filter (most FireWire and USB cameras will work)

Download

The current version as of 1/13/2004 is alpha2: it_alpha2.tar.gz
The old version alpha1 (11/19/2003) has no OSC support: it_alpha1.tar.gz

Installation

Get all the required packages (see Requirements above).
Download the latest version into a directory of your choice.
Unzip and untar the package that you downloaded.
Run install.bat and read the instructions.

If they are not in the path, copy all of the following runtime libraries into the local directory

OpenCV:	cv.dll, cvaux.dll, cvcam.dll, highgui.dll, OptCV*.dll
IPL:	Cpuinf32.dll, ipl*.dll
Magick:	CORE_RL_*.dll, X11.dll

Recognize Gestures!

Quick Start Guide

Execute runme.bat in the directory with all the files.
Click on the "conductor" symbol in the toolbar and load hmd.conductor.
Connect a camera to your computer and click on the camera symbol in the toolbar. The live video will appear in the application window.
Perform the symbolized posture at the suggested location. After recognition of your hand, tracking will start and a number of key postures will be recognized.

Advanced features

Hit "o" to see verbose output: a white square in which the initial posture is recognized, the color segmentation and KLT feature locations during tracking, and discrete, recognized posture locations (green rectangles).
If your hand is overexposed a lot, check the "Auto-control camera" option from the menu if your camera supports this interface. The exposure will be adjusted automatically within the recognition area.
Open the Configuration from the Options menu (or just hit "c") and note the event distribution settings. From a command prompt or a terminal on another machine, call "telnet your.machine.name port", where port is the gesture server port. You will receive a stream of recognized gestures along with the relative image location of its centroid. Changes in the settings will only have effect if done before loading a vision conductor.
If the OSC settings in the Configuration are set, one UDP packet in OSC format will be sent for each frame the hand is detected or tracked. Changes in the settings will only have effect if done before loading a vision conductor.
F11 toggles between windowed and full-screen mode.

Matz' home page