Because of its 360° field of view, an omni-directional camera is suitable for detecting and tracking environmental features in mobile robot navigation applications. This study aims to investigate simultaneous localization and mapping (SLAM) of a mobile robot using omni-directional images. A switching method of visual reference scans is proposed to facilitate fast visual scan matching in the SLAM design. In this method, new reference scans can be added to a database and an existent reference scan can be switched to be current reference scan automatically in SLAM calculation. Visual reference scans can be used repeatedly to reduce the computation complexity of extended Kalman filter (EKF) in the SLAM algorithm. Experimental results show that the correct matching rate of landmark features is 92.6%. Indoor navigation experiments validate the proposed localization algorithm. Average localization error of 10cm has been achieved in a 30m travel in an indoor environment using omni-directional images.