The results of a detailed characterization study on a systematic set of InAs/GaAs self-assembled quantum dot infrared photodetectors are presented. A simple physical picture is also discussed to account for the main observed features. Photoresponse characteristics in a wide spectral region from the mid- to far-infrared are reported. Clear polarization behaviors with a dominant P-polarized response in the mid-infrared and a strong S-response in the far-infrared regions are shown. These behaviors can be qualitatively understood in view of the quantum dot shape of a large in-plane diameter and a small height in the growth direction. With a set of three samples, effects of the number of electrons per quantum dot on the spectra are investigated.