In this paper, we propose a new mean-shift tracking algorithm based on a novel similarity measure function. The joint spatial-color feature is used as our basic model elements. The target image is modeled with the kernel density estimation and the new similarity measure functions is developed using the expectation of the estimated kernel density. With these new similarity measure functions, two similarity-based mean-shift tracking algorithms are derived. To enhance the robustness, the weighted background information is added into the proposed tracking algorithm. In order to solve the object deformation problem, the principal component analysis is used to update the orientation of the tracking object, and corresponding eigenvalues are used to monitor the scale of the object. The experimental results show that the new similarity-based tracking algorithms can be implemented in real-time and are able to track the moving object with an automatic update of the orientation and scale.