Recently, discrete Fourier transform (DFT)-based discrete multitone modulation (DMT) systems have been widely applied to various applications. In this paper, we study a broader class of DMT systems using more general unitary matrices instead of DFT matrices. For this class, we will show how to design the optimal DMT systems over frequency-selective channels with colored noise. In addition, asymptotical performance of DFT-based and optimal DMT systems will be studied and shown to be equivalent. However, for a moderate number of bands, the optimal DMT system offers significant gain over the DFT-based DMT system, as will be demonstrated by examples.