The quadtree-based inter-prediction in the latest High Efficiency Video Coding (HEVC) enables highly flexible structure and good coding efficiency, but also introduces significant computational complexity for hardware design. Conventional approach determines the best inter-prediction mode after computation of both integer-pel (IME) and fractional-pel motion estimation (FME). However, the best prediction unit (PU) after IME and FME are highly correlated according to our analysis. Thus, this paper proposes a fast PU selection method that computes possible FME PU size only in the same quadtree depth as that of the best IME PU instead of exhaustive search. The simulation results show that the proposed approach can save up to 79% computational complexity of FME, with only up to 1.34% BD-rate increases when compared with the reference software HM6.0.