To satisfy future explosive demands of mobile traffic, millimeter wave (mmWave) technology has been widely considered in the next generation wireless networks thanks to its large spectrum and spatial resources. MmWave Hybrid beamforming (HBF) with multi-input multi-output (MIMO) system is a critical technique to support multi-beam operation and reduce the cost and complexity. In this paper, we define inter-user interference in the mainlobe and the sidelobe. With the consideration of quality of service (QoS) requirements, we design a mainlobe interference avoidance (MIA) scheduling algorithm to maximize data throughput under the existence of inter-user interferences. Our proposed scheme allocates system resources by jointly considering not only the subband resources but also the antenna beamwidth and beam direction. Simulation results demonstrate that the proposed scheme outperforms existing heuristic algorithms both in the uniform and hotspot scenarios.