This paper proposes a robust object segmentation by the means of the probability-based background extraction algorithm. The color background images can be extracted efficiently and quickly from color image sequences by the proposed background extraction algorithm. After the background extraction algorithm, the intrusive objects can be segmented correctly and immediately by the robust object segmentation. The background extraction algorithm calculates the color probabilities of each pixel and uses a convergent value to decide the background pixel color, whose probability is the maximum one and greater than the convergent value. The extracted background is updated in real-time to overcome the variation of the illuminative condition. Experimental results using different types of video sequences are presented to demonstrate the robustness, accuracy, and time responses of the proposed algorithm.