Abstract: In view of the man-machine interaction in mobile phone platform and the time-domain problem in gesture recognition, a visual based gesture trajectory recognition is proposed as a way of interaction between human and mobile phone. Fourier descriptors are used to characterize the gesture trajectory, and the dynamic gesture trajectory is classified combined with confidence coefficient. In this paper, the motion detection of gesture video is firstly carried out, and the hand tracking is carried out by using the color-based particle filter to obtain the trajectory of gesture center. The obtained trajectory is closed so that the trajectory becomes a closed curve set, and the closed curve set is Fourier transformed to make the trajectory have the invariance of translation, scaling and rotation. The obtained Fourier descriptors are matched with template and have good stability in a certain range. The deformation coefficient is proposed to describe the degree of distortion of the sampled trajectory. The experimental results show that the method has high accuracy when the deformation coefficient is less than 0.3.