The objective of this investigation is to reconstruct building models by the integration of high overlapping aerial images and building boundaries. The proposed method includes four major parts: (1) polygon tracing, (2) 3-D building polygon generation, (3) 3-D structural lines extraction, and (4) Split-Merge-Shape building modeling. First, the operator locates the initial lines by a graphical user interface. The initial lines are refined by image matching of high overlapping images. Finally, the precise 3-D lines and 3-D polygons are processed into a 3-D modeling by an inference engine. The experimental results indicate that the proposed method may reconstruct the 3-D building model effectively.