This paper is to present the development and the verification of a compact optical sensor system for measuring the deflection of the cantilever in a tapping-mode atomic force microscope (AFM). An optical pickup head of commercial digital versatile disc read only memory (DVD-ROM) drive is applied in this sensor system. In order to satisfy the strict measuring requirements of sensibility and reliability, the build-in detection system is replaced with a four-quadrant photodiode of traditional optical-lever technique. In addition, an inertial motor composed by a piezo-actuator is used as the sample approaching mechanism and the scanner in the vertical direction. Thus, the volume of hardware structure is decreased, and the sensing variance due to temperature change will be minimized. The developed AFM system is well functionally verified by measuring a calibration grating with step height 19.5 nm, and the vertical resolution is ±8 nm.