Traffic surveillance systems have been widely used for traffic monitoring. If the degree of traffic congestion can be evaluated from the surveillance videos immediately, the drivers can choose alternate routes to avoid traffic jam when traffic congestion arises. Compared to daytime surveillance, some tough factors such as poor visibility and higher noise increase the difficulty in video understanding under nighttime environments. In this paper, we propose a framework of traffic congestion classification for nighttime surveillance videos. The framework consists of three steps: the first one is to detect headlights based on three salient headlight features. Second, headlights are grouped into individual vehicles by evaluating their correlations. Third, a virtual detection line is adopted to gather the traffic information for traffic congestion evaluation. Then the traffic congestion is classified into five levels: jam, heavy, medium, mild and low in real-time. We use freeway nighttime surveillance videos to demonstrate the performances on accuracy and computation. Satisfactory experimental results validate the effectiveness of the proposed framework.