One main physical feature of hybrid rocket combustion is its diffusion flame structure that requires excessively long solid grain port which often leads to undesirable large slenderness of a rocket configuration. The diffusion flame also results in generally low combustion efficiency of hybrid rockets. Some remedial designs have used liquefying solid grain, such as paraffin, or mixing enhancement mechanisms to boost the overall combustion efficiency. Thus, shortened combustion chamber can be used to deliver reasonable thrust performance of hybrid rockets. In addition to the study of multi-stage mixing enhancer effects, a compact hybrid rocket motor design concept is also proposed in the present study to provide better form factors for hybrid rocket engine designs. This design concept features in vertical-flow structures such that greatly improved combustion efficiency is obtained. A 3D computational model with finite-rate chemistry and radiative heat transfer capabilities is employed to assess the mixing effectiveness and combustion efficiency of the new design concept. The present computational model is validated for a wide range of rocket propulsion design problems, including a single-port hybrid rocket motor with and without using a mixing enhancement mechanism. The internal ballistics and flame structures in the hybrid rocket engine with mixing enhancement designs are analyzed.