Recently, the development of Unmanned Aerial Vehicle (UAV) has been nearly matured and widely used in various fields. The combination of UAV and communication technologies, such as UAV Base Station (UAV-BS), can significantly increase the flexibility and scalability of the overall communication networks to provide more efficient communication services. While the UAV-BS improves the network service efficiency, the quality of services (QoS) in the air-to-ground communication link is highly affected unless the right users are unknown. In this paper, we propose the learning-based downlink user selection algorithm. The 3D downlink channel can be fast identified to judiciously select the users subset. In our proposed framework, we combine the k-means clustering and Convolutional Neural Network (CNN) that can increase the estimation accuracy of 3D wireless channels to enhance the communication service efficiency of the UAV-BS network. The field measurement results show that proposed method can achieve an average bit error rate (BER) of 3.56x10-7, which is better than the distance-based selection scheme that has an average of BER 2.88x10-3. The feasibility and effectiveness of the proposed method in real environment are proved, experimentally.