This paper presents a novel edge labeling scheme for detecting lanes from videos in real time. Firstly, pairs of edge pixels with different edge types are grouped using the labeling technique. Then, different lane hypotheses can be generated for lane modeling. Then, a lane geometrical constraint is derived from the pinhole camera geometry for filtering out impossible lane hypotheses. Since the constraint is invariant to shadows and lighting changes, each desired lane can be robustly detected even though different occlusions and shadows are included in the analyzed scenes. After filtering, a kernel-based modeling technique is then proposed for modeling different lane properties. With the modeling, different lanes can be effectively detected and tracked even though they are fragmented into pieces of segments or occluded by shadows. The proposed scheme works very well to analyze lane conditions with night vision. With the lane information, different dangerous driving behaviors like lane departure can be directly analyzed from road scenes. Experimental results show that the proposed scheme is powerful in lane detection. The average accuracy rate of vehicle detection is 95%.