This paper proposes a novel Shift with Importance Sampling (SIS) scheme to improve the efficiency in DPM-based object detection but maintain its high accuracy. For fast and efficient object detection, the cascade-boosting structure is the commonly-used approach in the literature. However, its detection performance is quite lower due to non-robust features and a fully-scanning on image especially when deformable part models are adopted. Another powerful method 'deformable part model' is commonly adopted to deal with the above problem. However, its full combinations of parts to represent an object make its inefficiency in the scanning process which needs to check all possible object positions. The proposed SIS scheme can avoid many redundant positions and thus significantly improve the efficiency of the DPM scheme up to a time order. Firstly, various interest points are first detected and then clustered via the ISO-data clustering scheme to produce potential candidates. Since each candidate will not exactly locate in the center of the detected target, it will be shifted according to the weights of its eight neighborhoods. The importance of each neighbor is scored by the DPM-classifier. Once it is shifted, the size of search window to find its positions will be narrowed to only quarter. Thus, the proposed SIS scanning scheme can quickly find the correct location of each pedestrian with minimum tries and tests. After analysis, the time complexity of scanning is reduced from O(n2) to O(logn), where the frame dimension is n×n. After that, the particle filter is adopted to track targets if they are missed. Experimental results show the superiority of our SIS method in pedestrian detection (evaluated on different famous datasets).