Recently, the hybrid rocket propulsion has become attractive to the research community and has developed the trend to become an alternative to the conventional liquid and solid rockets. Among available hybrid systems, the N2O (Nitrous Oxide)-HTPB (Hydroxyl-Terminated PolyButadiene) hybrid propulsion represents the simplest but sufficiently efficient design. To date, research in developing hybrid N2O-HTPB propulsion system, despite some available fuel regression rate correlations, still strongly depends on experimental trials-and-errors, which are time-consuming and expensive. Thus, detailed understanding of the fundamental combustion processes that are involved in the N2O-HTPB propulsion system can greatly impact the research community in this field. This may further facilitate the successful modeling of the combustion processes and help improving the design of N2O-HTPB propulsion system in the future. A comprehensive numerical model with real-fluid properties and finite-rate chemistry is developed in this research to predict the combustion flowfield inside a N2O-HTPB hybrid rocket system. The effects of a mixing enhancer design are demonstrated experimentally and numerically in boosting the overall thrust performance of a hybrid rocket motor. Good numerical predictions as compared to experimental performance data are presented. Computation of the N2O flow rate control with a ball valve shows smooth variation behavior which is desirable in propulsion system designs.