We demonstrate the theoretical and experimental results of using a single prism in the rapid-scanning optical delay line of an optical coherence tomography (OCT) system for compensating the mismatches of the first- and second-order group delay dispersion (GDD) between the reference and sample arms. The analytical expressions for the first- and second-order GDD are derived based on the typically designed system configuration. Numerical results in varying various parameters are shown. An optimized set of parameters for efficient dispersion compensation in a practical fiber-based OCT system is obtained. The numerical result of the dispersion compensation is demonstrated. Also, the experimental implementation of such a dispersion compensation method is illustrated with the conditions similar to the numerical calculations. The compensation result is quite satisfactory.