In this paper, we present an adaptive foreground object extraction algorithm for real-time video surveillance, in conjunction with a human detection technique applied in the extracted foreground regions by using AdaBoost learning algorithm and Histograms of Oriented Gradient (HOG) descriptors. Furthermore, a RANSAC-based temporal tracking algorithm is also applied to refine and trace the detected human windows in order to increase the detection accuracy and reduce the false alarm rate. The traditional background subtraction technique usually cannot work well for situations with lighting variations in the scene. The proposed algorithm employs a two-stage foreground/background classification procedure to perform background subtraction and remove the undesirable subtraction results due to shadow, automatic white balance, and sudden illumination change. Experimental results on some real surveillance video are shown to demonstrate the good performance of the proposed adaptive foreground extraction algorithm under a variety of different environments with lighting variations and human detection system.