It is anticipated that the generic CMOS device will continue to shrink for some generations before the end of the Moore's law and that the gate length of the transistor will be eventually shortened to some few nanometers. On the road towards this ultimate size there are a lot of challenges regarding the device structure, material engineering, as well as the fabrication technology. Significant device performance degradation, such as the increase of the "off current" because of the source-drain punchthrough and the gate leakage in decananometer devices, has already been encountered. New device structures such as FinFET, nanowire FET, or other 3D configurations have to be employed in the future CMOS technology. From the physical point of view, these structures do provide better performance and better scalability. However, their realization requires some challenging atomic-scale fabrication technologies and needs to introduce some new materials as well. Metal gate/high-k stack will still be the only solution for channel current control in the transistors. However, the equivalent oxide thickness (EOT) of the high-k gate dielectric has to be scaled down to half nanometer range. We may prefer even aggressive thinning of the gate dielectric in order to suppress the "off current" more effectively. Unfortunately the actual downsizing rate of the gate oxide is far slower to date than that predicted by the technology roadmap a couple years ago. The major difficulty is that the EOT is now mainly governed by both the silicon/high-k and the metal gate/high-k interfaces which are not directly scalable. In this review, these issues will be critically discussed from the device physics and fabrication technology points of view. Possible scenarios for device structure and fabrication procedures for the ultimate nano-CMOS technology at the end of Moore's law will be highlighted.