In the present study, a comprehensive computational fluid dynamics model is developed for the internal ballistics study of hybrid rocket motors. A hybrid rocket motor with vortex generating mixing enhancer is proposed to increase the overall mixing and combustion efficiency, which also results in the allowance of reducing the chamber port length in the motor designs. The present 3-D computational tool with finite-rate chemistry and real-fluid models is employed to numerically assess the mixing effectiveness and combustion efficiency of a hybrid rocket motor. An extended two-equation turbulence model is employed to simulate the turbulence effects. The computational results of the present model are validated with experimental hot-fire test data, including a single-port hybrid rocket motor with or without a mixing enhancement mechanism installed. The internal ballistics and flame structures in the hybrid rocket combustion chamber are analyzed.