High-κ cerium oxide (CeO2) was applied to AlGaN/GaN high-electron-mobility transistors (HEMTs) as a gate insulator and a passivation layer by molecular beam deposition (MBD) for high-power applications. From capacitance-voltage (C-V) measurement results, the dielectric constant of the CeO2 film was 25.2. The C-V curves showed clear accumulation and depletion behaviors with a small hysteresis (20 mV). Moreover, the interface trap density (Dit) was calculated to be 5.5 × 1011 eV-1 cm-2 at 150 °C. A CeO2 MOS-HEMT was fabricated and demonstrated a low subthreshold swing (SS) of 87mV/decade, a high ON/OFF drain current ratio (ION/IOFF) of 1.14 × 109, and a low gate leakage current density (Jleakage) of 2.85 × 10-9Acm-2 with an improved dynamic ON-resistance (RON), which is about one order of magnitude lower than that of a conventional HEMT.