The development of digital aerial camera provides the possibility of acquiring highly overlapped aerial images with high spatial resolution. In addition to its high spatial resolution that improves the capability of image interpretation, the highly overlapped images provide favorable geometrical configuration with high redundancy. The high similarity of stereo images is, thus, beneficial to the reliable image matching. Hence, the 3-D point clouds from the image matching have the great potential in 3-D modeling. The topographic maps provide the distinct boundaries for building modeling. The strategies of building reconstruction with existing topographic maps may improve the quality, cost, and efficiency for building modeling. The objective of this investigation is to integrate highly overlapped aerial images and building boundaries from topographic map. The proposed semi-automatic method includes 3-D lines extraction and polyhedral model generation. In the beginning, the operator locates the initial lines by a graphical user interface. The initial lines are refined by stereo images. The precise 3-D lines are processed into a 3-D modeling by an inference engine. The experimental results indicate that the proposed method may reconstruct the 3-D building model effectively.