A video-based incident detection system for monitoring the urban road is investigated in this paper. The developed surveillance system can monitor not only vehicles and motorcycles on the road surface but also pedestrians on the walkway and the prohibitive zones in the image. Several different kinds of incident can be detected in the presented system, such as the congestion, the illegally parking, the lane-changing vehicle, the falling object, the pedestrian across the road, and the pedestrian appearing in the prohibitive zone. The proposed method is based on the background subtraction. Therefore, the background image and lane markings are estimated in the beginning. Then the foreground image is obtained from the difference between the current image and the background image. In order to have isolated objects, the objects in the foreground image are separated by lane marking. The occlusion caused by the vehicles and the pedestrians can also be handled by the temprol-spacial analysis for the exact detection. After that, the tracking is applied to track the targets. Finally the proposed system can detect the incident by the integrated information from the pre-defined information and the tracking. Several challenge video are performed to verify and the experimental results demonstrate that our system is satisfying and effective.