This paper presents a novel color-based method to detect road signs directly from videos. A road sign usually has specific colors and high contrast to its background. Traditional color-based approaches need to train different color detectors for detecting road signs if their colors are different. This paper presents a novel color model derived from Karhunen-Loeve(KL) transform to detect road sign color pixels from the background. The proposed color transform model is invariant to different perspective effects and occlusions. Furthermore, only one color model is needed to detect various road signs. After transformation into the proposed color space, a RBF (Radial Basis Function) network is trained for finding all possible road sign candidates. Then, a verification process is applied to these candidates according to their edge maps. Due to the filtering effect and discriminative ability of the proposed color model, different road signs can be very efficiently detected from videos. Experiment results have proved that the proposed method is robust, accurate, and powerful in road sign detection.