the

HandVu
vision-based hand gesture interface
homepage | index | basics | common | ARtk demo | troubleshooting

HandVu Application Documentation

This document assumes that you have successfully installed HandVu and at least one of the sample applications on your system. Please read at least the basics and common functionality. While a lot of functionality is common to all sample applications, some key shortcuts do not work in all applications, indicated with a (*).

Basics: Setup and Hand Postures

Camera:

HandVu is designed for a camera that views the space in front of a sitting or standing person from a top-down view. The camera can be ceiling-mounted or head-worn. It should deliver at least a 320x240 resolution.

Hand detection:

The hand is detected only in a standard posture and orientaentirelytion with respect to the camera, called the closed posture: recognized postures. This is necessary to avoid inadvertent gesture commands and to speed the image processing.

Hand tracking:

Once the hand has been detected, you can move the hand around in any posture. The better the lighting conditions are (uniform without harsh shadows) and the less brightness variation exists in the background, the better tracking will work. Avoid all too rapid movements or quick posture changes if you experience problems.

Posture recognition:

All of the six recognized postures can be performed at any time during tracking and they will be recognized. Note that all postures are to be performed in a plane parallel to the imaging plane, facing upwards in the image, and with no more than 15 degrees counter-clockwise rotation (to the left). You will have to practice the gestures a few times until you achieve a good recognition rate.

Common Functionality

GestureServer

A "GestureServer" is automatically started in the current version. It accepts TCP/IP connections on port 7045 and returns GestureEvents in ASCII format. To see the recognition results, connect to the Server at port 7045 with an ordinary telnet session. (For example, go to Start->Run... and type "telnet localhost 7045")
If of interest, please read the more detailed description of the GestureEvent format. The OSC server will be back in the next release or earlier if you beg me.

VisionConductor:

Most of the settings of HandVu can be controlled with a configuration file, called a VisionConductor. Please see the following document for details: VisionConductor file format.

Verbosity Level:

The key shortcuts 0,1,2,3 select different verbosity levels. If you are experiencing trouble with the recognition, please select level 3. The white rectangle is the initial detection area. The closed posture has to be performed entirely within these bounds. The minimum and maximum sizes of the hand are indicated by the blue rectangles.
Shown in some of the verbosity levels are: a white rectangle in which the initial posture is recognized, the color segmentation and KLT feature locations during tracking, and discrete, recognized posture locations (green rectangles) together with the foreground color probability.

Restart Detection

Pressing "r" restarts the detection in the initial area.

Software Exposure

(*) If your hand is overexposed a lot, enable the auto exposure option by pressing "e". The exposure will be adjusted automatically within the recognition area. This will only work on cameras whose drivers export a standard control interface.

Miscellaneous

(*) F11 or "f" toggles between windowed and full-screen mode.
Pressing "t" takes a snapshot and saves it in a folder called "hv_pics".
Distortion correction for several camera lenses can be turned on with "u", using the calibration file set in the conductor.

HandVu Demo for ARToolKit

This little demo lets you play with virtual objects in an augmented-reality environment. A multi marker is the anchor point for a bunch of objects which can both be manipulated with hand gestures. Hold the multi-maker with your left hand at the larger grey section and don't obstruct the symbols with the right hand.
As with all applications, the "closed" posture is the only one possible for initialization. "Lback" is used to select for dragging. "Lpalm" is used to select for rotation. "open" ends either of these operations. "sidepoint" changes the color of the object under the cursor once a second. (this might happen during wrong recognitions as well...) Command line flags: 
--snapshots     - take a snapshot of the scanned area without any OpenGL
                   renderings, at most twice a second, and store them
                   in $HOME/hv_pics (will be created)
--fullsnaps     - instead of infrequent, small snapshots, this switch
                   makes it take a full frame shot every frame, after
                   all OpenGL rendering
--pointerdist   - from tracked hand position to grabbing location, in 
                   screen pixel units
--fixedworld    - start in "fixed world" setup, which means: multiMarker
                   is used to control a single object (another teapot),
                   while the other two, formerly anchor-controlled, 
                   objects stay where they are
--threshold     - ARtk threshold
Runtime key shortcuts. 
m    - toggle show markers [default: on]
e    - exposure control by HandVu software, vs camera auto exposure;
         use when the hand is too bright.
         [default set in the ARtk.conductor file]
f    - toggle fixed world (basically, what the multiMarker is used for)
w    - reset world: two objects in fixed locations, the third one only 
         visible if the multiMarker is there and fixedworld is off
s    - toggle take snapshots (fullsize mode as selected at command line);
         no snapshots are taken if an overlay level>0 is selected
+/-  - increase, decrease ARtk threshold  ("=" works as +, too)

0-3  - HandVu overlay level; note that no snapshots will be taken 
         regardless of the setting if an overlay level>0 is selected
         (that's a feature: otherwise, the images are cluttered with
         debugging output)

Troubleshooting & Tips