This paper describes a new algorithm for detecting people using a single downward-viewing fisheye camera. People detection from images taken with projective cameras has been studied extensively in recent years. On the other hand, researches on people detection from fisheye camera images are very limited, and existing techniques are either designed for very simplistic and uncluttered environments, or are very time-consuming. Given that fisheye cameras have a number of advantages in application such as people counting and visual surveillance, including less occlusion among peoples and larger views, the objective here is to propose a new technique of people counting using fisheye cameras that is both practical to realistic environments and efficient enough for real-time applications. The main innovation of our method is to take advantage of fisheye geometries to estimate the expected sizes of people as functions of image locations. An adaptive set of elliptic templates are pre-computed to expedite processing. Given the different appearances of people at different distances to the image center, a set of support vector machines (SVMs) are used to classify different templates as people or not. We also describe a tracking algorithm for people counting and tracking in indoor environments.