Finding targets in a complex background is important for visual servo systems. From feedback control perspective, it represents reliable sensor information to prevent the servo systems from tracking a wrong target. Prior to target identification, it is necessary to localize the target within the picture so as to enhance the identification accuracy. Most image processing methods applied for target localization dealt with static pictures. In other words, information regarding the temporal behavior of the video sequence is seldom utilized. In this paper, a statistical framework using Gaussian Mixture Model (GMM) is investigated to localize a moving target without knowing the information about the target. Then, support vector machine (SVM) is applied to be a classifier to identify the moving target. A vision servo system is constructed by the scheme that incorporates Gaussian Mixture Model (GMM) and Support Vector Machine (SVM). Experimental results are shown to demonstrate the effectiveness of the proposed method.