We proposed a surface-based vacant parking space detection system. Unlike many car-oriented or space-oriented methods, the proposed system is parking-lot-oriented. In the system, we treat the whole parking lot as a structure consisting of plentiful surfaces. A surface-based hierarchical framework is then proposed to integrate the 3-D scene information with the patch-based image observation for the inference of vacant space. To be robust, the feature vector of each image patch is extracted based on the Histogram of Oriented Gradients (HOG) approach. By incorporating these texture features into the proposed probabilistic models, we could systematically infer the optimal hypothesis of parking statuses while dealing with occlusion effect, shadow effect, perspective distortion, and fluctuation of lighting condition in both day time and night time.