Title

A Fast and Robust Fingertips Tracking Algorithm for Vision-Based

Multi-touch Interaction

Introduction

This paper propose a fast and robust fingertip 

tracking algorithm based on geometry structure model of 

hand. 

Compared with existing methods, this algorithm attempts to track movements of multiple fingertips not only in 2D but also in 3D space without using any marks. A stereovision system is set up to retrieve depth information of the scene. The algorithm detects the hand region using skin color filter as well as depth images reconstructed from the stereovision system, and then calculates the position of palm center. Finally based on the observation that the geometry structure of hands is almost identical, it’s possible utilizing the geometry relation between palm center and hand

contour to determine the fingertip locations on the contour. 

1.  Hand Localization

In order to track the movements of fingertips, hand region should be segmented from background first. Hand localization, however, is still a challenging problem due to its high degree of freedom (DOF) and flexible shapes. For the purpose of improving system efficiency, this approach handles the problem in a simpler way. The hand is considered as the closest object with skin color in front of the camera. The hand region can be detected by a two step method. First, a skin-color filter is applied to locate the candidates of hand region.Then the hand is segmented from depth images using depth clipping and region grow algorithm.

1.1). Skin colour filter

Skin-color has been proven to be an effective cue for extracting hand and face regions from background. Basically, skin color detection is to define decision rules and build a skin color classifier. The main difficulty is to find both appropriate color space and adequate decision rules.

In our algorithm, we choose YCbCr color space for skincolor segmentation. YCbCr color space separates the color information to three channels: luminance, chrominance and compactness, and is appropriate for skin-color segmentation. In addition, a parametric model, the Gaussian Mixture

Model is also employed to describe the skin-color distribute. For a single Gaussian model, skin-color probability distribution p(c|skin) is defined as follow

Here c is a color vector. μs and Σs are the model parameters which can be estimated from training data as follows: 

where n is the number of training samples. Finally, the Gaussian Mixture Model is defined by

where k denotes the number of mixture components, λi denotes the weight of each Gaussian model which satisfy kΣ i=1 λi = 1. The P(c|skin) can be used directly as the measure of how ”skin-like” the color is. In this paper we set k to 5 and the Gaussian Mixture Model parameters are evaluated by

the well-known Expectation Maximization (EM) algorithm. After extracting skin-like regions from background, apply morphological operations on the extracted regions for the purpose of removing noises. In this stage, some objects with skin-like color are also extracted as candidates of hand

region.

1.2) Hand segmentation from depth image

As mentioned before, hand is assumed to be the closest object with skin-color in front of the camera. Therefore, the hand can be identified from all the candidates by depth clipping technique and region grow algorithm. Note that the hand region in depth image is continuously distributed and extends within a limited 3D space, so the points with minimum depth are picked as seeds. By applying

region grow algorithm, the hand region is segmented from the background.

(a) Original hand image; (b) Depth image of (a); (c) Extracted region after skin-color filtering and noise removal; (d) Extracted region after applying depth clipping and region grow algorithm;

(e) Hand region: intersection of (c) and (d).

2. Fingertip Tracking

In this section, we will describe a fast and robust fingertip track algorithm. Different from most existing approaches which are usually based on template matching, curvature analysis or color marks, this approach utilizes the geometry information of hand shape to determine the positions of

fingertips. Due to its simplicity and effectiveness, this method is expected to make faster and more stable tracking on. 

2.1) Palm center localization

In order to find positions of fingertips, the center and the size of palm should be estimated first.

We notice the palm is a rectangle-like region, more or less. Based on this observation, a projection based algorithm is 

The basic idea is to project the hand region in all directions. If the projection line goes through only one block of the hand region, that implies this block is a candidate of palm region and should be preserved. After projections along all directions, the intersection of all candidates will make a final palm region. This algorithm is formulated in this algorithm. 

Normally it’s unnecessary to project the image in all directions because the extracted region will not change much when the projection angle interval is smaller than a certain value. Here this does the projection every 450 from 00 to ±1800 experientially. Compared with shape model based methods,

this algorithm is faster and simpler

Palm center C0 is defined as the point in the palm region which has maximum distance from the closest palm boundary. 

(a) Original hand image;

(b) Extracted hand region; 

(c) Preserved hand region when the projection angleθ =00;

(d) Preserved hand region when the projection angle θ =900;

(e) Extracted palm region: intersection of all preserved hand regions when projection angle varies from 00 to ±1800 at an interval equals to 450. The red point is the palm center.

where P denotes the point inside palm region Rp, PB denotes the boundary B of palm region, and d2 is the function for calculating 2D Euclidean distance between two points. The size of palm R is defined as the distance between C0 and the closest boundary.

2.2) Fingertip localization 

The fingertip is considered as the point with maximum distance to the palm center on the contour of each finger. Here utilize another two-step method to locate the fingertips. Contour for each finger is extracted first. Then the point on the contour with maximum distance to the palm center is picked as fingertip. Finger contours can be regarded as subsets of hand contour. The contour which makes the maximum area is considered as the hand border. Then the distances between contour points and palm center C0 are calculated. If a distance is larger than a predefined threshold, this contour point will be put into a candidate set of fingertips immediately, as formulated in the following equation

Here P denotes a contour point, d2(P;C0) denotes the distance between palm center C0 and P, α is a scale factor, set it to 1:2 empirically. F is the candidate set of the fingertips.

Firstly we need to determine the contour points for each finger in the candidate set. This can be fulfilled by tracing all the contour points and compared it with the points in the candidate set. In order to decrease the computing time, this present a more effective solution here. The idea is to assign

an index to each point in the candidate set, then sort the set by the index. A function ϕ is defined for calculating the index.

Here Pf is the point in the candidate set F and θPf is the index of Pf. The candidate set is sorted by θPf in ascending order (clockwise). Then the distances between successive points are calculated as follows to determine the start and end point of several subsets (the contour points for each finger) of the candidate set.

Here, Dpi denotes the distance between successive points Pi and Pi+1. If Dpi is greater than a predefined threshold δ (in this system, it is set to 2.5), Pi is considered as the start or end point of a subset. Therefore, all points in the candidate set can be divided into several subsets which are disconnected on the hand contour. Meanwhile each of them corresponds to one fingertip contour. Finally we compute the distance from each point in the subset to the palm center. The one with

maximum distance is identified as a fingertip.

(a) Extracted Hand region; (b) Hand contour; (c) Candidate set of fingertips on the hand contour.

(d) Calculation of index θpf for each point in the candidate set. The set is sorted by the index. The

indexes are illustrated using a color coding scheme from pure green (the minima) to pure red (the maxima). The points are sorted in ascending order (clockwise); (e) Extracted fingertips (green points) and the start and end point for each fingertip region (red points).

Applications

In this paper, proposed a fast and robust method to track the fingertips. Different from existing approaches, this vision based method needs neither pressure sensing devices nor extra marks for fingertip localization. With the help of a stereovision system, 3D positions of the fingertips

are recovered by an efficient algorithm based on skin color detection and geometry model analysis. 

we can use their palm center detection methods and fingertip detection methods for our system.