A hybrid architecture of radio frequency (RF) beamforming (BF) and baseband (BB) signal processing is presented for downlink (DL) multi-user (MU) multiple input multiple output (MIMO) systems operating in millimeter wave (mmWave) channels. To avoid the strong spatial correlation and the transmission or reception dead zones caused by the use of single-polarized antenna arrays in mmWave radio, the hybrid architecture is implemented on dual-polarized (DP) planar antenna arrays (PAA). To balance the quality of each data stream of each user under a limited power consumption, an iterative algorithmfor jointly designing the RF beamformers (BFer), BB precoder, BB equalizers, and the power allocation is proposed for the DL MU-MIMO system of using the hybrid architecture. Simulation results show that the signal to interference-plus-noise ratio (SINR) achieved with the MU-MIMO system that uses the proposed hybrid architecture is only 5∼6 dB lower than that of the conventional one which uses 8 times more BB modules for the same 4 × 4 DP-PAA. Furthermore, the RF BFers obtained by the proposed iterative algorithm provide a better SINR than that of directly steering the RF BFs towards the line-of-sight transmission path.