This study proposes a method for image-based surgical instrument pose estimation. In minimally invasive surgery (MIS), stereo vision can be used for surgical tool recognition and pose estimation. We use color markers to indicate the instrument's joints and detect them with a custom-made stereo camera to obtain their 3D coordinates. Kalman filter is adopted to attain the position of the given instrument's joints. We propose a kinematic algorithm to calculate the 6-Dof pose of the surgical instrument in the Cartesian coordinate system and the rotation angles of its joints. The proposed method is verified on a lab-built experimental platform, consisting of a 6-DoF HIWIN manipulator and a motorized surgical instrument. The experiment involves the instrument's tracking with a stereo camera in MIS pick-and-place routine. Experimental results show that the visual system has a maximum absolute error lower than 2.7 millimeters at the endpoint and the MAE of joint angles is less than 0.11 radians. The developed image-based pose estimation system can be used for the surgeon's training in MIS and may be applied to autonomous surgical operations.