A method for analysis of spectroscopic ellipsometry data is demonstrated to extract the optical conductivity of mono- and bilayer chemical-vapor-deposited graphene. We model graphene as a truly two-dimensional (2D) material with a sheet conductivity, rather than a phenomenological effective refractive index as has been used in the literature. This technique measures both the real and imaginary part of the optical conductivity, which is important for graphene optoelectronics and metamaterials. Using this method, we obtain broadband measurements of the complex optical conductivity for mono- and bilayer graphene from ultraviolet to mid-infrared wavelengths. We also study how chemical doping with nitric acid modifies the complex optical conductivity.