This study presents a method of controlling a mobile robot of omni-directional wheels. Two optical flow sensors are installed on the chassis of the robot to obtain motion information relative to the world coordinate. Due to wheel slip, the kinematic relation from the wheel speed to the robot speed no longer holds. An extremum seeking algorithm is proposed to cope with the unknown friction when the robot tries to follow a prescribed trajectory. Further, a spatial tracking algorithm is proposed to modify the velocity commands to achieve a better performance in trajectory following. Lastly, a simple obstacle avoidance algorithm is also proposed based on the optical flow sensor measurement. Experiments are conducted for various driving conditions including different road surfaces to show the effectiveness of the proposed methods.