Oxygen vacancy formation and migration in ceria is critical to its electrochemical and catalytic properties in systems for chemical and energy transformation, but its quantification is rather challenging especially at atomic-scale because of disordered distribution. Here we report a rational approach to track oxygen vacancy diffusion in single grains of pure and Sm-doped ceria at −20 °C to 160 °C using in situ (scanning) transmission electron microscopy ((S)TEM). To create a gradient in oxygen vacancy concentration, a small region (∼30 nm in diameter) inside a ceria grain is reduced to the C-type CeO1.68 phase by the ionization or radiolysis effect of a high-energy electron beam. The evolution in oxygen vacancy concentration is then mapped through lattice expansion measurement using scanning nano-beam diffraction or 4D STEM at a spatial resolution better than 2 nm; this allows direct determination of local oxygen vacancy diffusion coefficients in a very small domain inside pure and Sm-doped ceria at different temperatures. Further, the activation energies for oxygen transport are determined to be 0.59, 0.66, 1.12, and 1.27 eV for pure CeO2, Ce0.94Sm0.06O1.97, Ce0.89Sm0.11O1.945, and Ce0.8Sm0.2O1.9, respectively, implying that activation energy increases due to impurity scattering. The results are qualitatively supported by density functional theory (DFT) calculations. In addition, our in situ TEM investigation reveals that dislocations impede oxygen vacancy diffusion by absorbing oxygen vacancies from the surrounding areas and pinning them locally. With more oxygen vacancies absorbed, dislocations show extended strain fields with local tensile zone sandwiched between the compressed ones. Therefore, dislocation density should be reduced in order to minimize the resistance to oxygen vacancy diffusion at low temperatures.