In this paper, we propose a heterogeneous tracking model, referred to as HTM, to efficiently mine object moving patterns and track objects. Specifically, we use a variable memory Markov model to exploit the dependencies among object movements. Furthermore, due to the hierarchical nature of HTM, multi-resolution object moving patterns are provided. The proposed HTM is able to accurately predict the movements of objects and thus reduces the energy consumption for object tracking. Simulation results show that HTM not only is able to effectively mine object moving patterns but also save energy in tracking objects.