In recent years, multimedia content processing has become a hot topic with the rapid development of information technology and popularity of World Wide Web. Among the emerging research topics, content-based video retrieval is an attractive and challenging one since query-by-text cannot provide the users with good support in finding the desired videos effectively. In addition, query-by-image also fails in reducing the gap between the image and user's interest. In this paper, we propose an innovative method for achieving effective content-based video retrieval by mining the temporal patterns in the video contents. Based on the temporal patterns, an efficient indexing technique is proposed to reduce the computation cost in searching videos. Experimental results show that our approach delivers excellent performance for content-based video retrieval in terms of efficiency and effectiveness.