In this paper, we develop a simple immersed boundary method to simulate the dynamics of three-dimensional axisymmetric inextensible vesicles in Navier-Stokes flows. Instead of introducing a Lagrange's multiplier to enforce the vesicle inextensibility constraint, we modify the model by adopting a spring-like tension to make the vesicle boundary nearly inextensible so that solving for the unknown tension can be avoided. We also derive a new elastic force from the modified vesicle energy and obtain exactly the same form as the originally unmodified one. In order to represent the vesicle boundary, we use Fourier spectral approximation so we can compute the geometrical quantities on the interface more accurately. A series of numerical tests on the present scheme have been conducted to illustrate the applicability and reliability of the method. We first perform the accuracy check of the geometrical quantities of the interface, and the convergence check for different stiffness numbers as well as fluid variables. Then we study the vesicle dynamics in quiescent flow and in gravity. Finally, the shapes of vesicles in Poiseuille flow are investigated in detail to study the effects of the reduced volume, the confinement, and the mean flow velocity. The numerical results are shown to be in good agreement with those obtained in literature.