The deployment of small cells with carrier aggregation (CA) technique is a significant feature of next-generation cellular networks. The potential benefits of dense base stations (BSs) deployment are provided by using proper interference management mechanisms. Meanwhile, the user equipments (UEs) can simultaneously access multiple component carriers (CCs) to meet the dramatically increased traffic demand by means of CA. In this paper, an optimization problem consists of UE-BS association, CC configuration, frequency resources scheduling, and power allocation with adaptive modulation schemes is investigated to enhance energy efficiency, focusing on the quality of service (QoS)-aware CA. To obtain the near optimal solutions as well as the consideration of computational complexity, a cross entropy- based algorithm (CEA) is proposed to jointly solve the combinatorial problem. Compared to the traditional heuristic algorithms, the merits of proposed scheme can be observed via simulation results.