General 3D interactive system based on embedded optical sensor had been reported for supporting single user or single touch. However, some actions like catch, zoom or rotate have not been proposed. In this paper, a 3D multi-interactive system was proposed to enhance the interaction between multi-users and 3D images. The concept of the proposed system is to use color filter as a band-pass filter to separate the illuminating properties of various light sources. Consequently, by using different color LEDs as the light-pen, this system can detect 3D positions (x, y and z) of multi-light sources. Finally, it can be applied for multi-user or multi-touch on 3D interactive display.