Realizing the real-time gaze redirection system with convolutional neural network

Chih Fan Hsu, Yu Cheng Chen, Yu-Shuen Wang, Chin Laung Lei, Kuan Ta Chen

研究成果: Conference contribution同行評審

摘要

Retaining eye contact of remote users is a critical issue in video conferencing systems because of parallax caused by the physical distance between a screen and a camera. To achieve this objective, we present a real-time gaze redirection system called Flx-gaze to post-process each video frame before sending it to the remote end. Specifically, we relocate and relight the pixels representing eyes by using a convolutional neural network (CNN). To prevent visual artifacts during manipulation, we minimize not only the L2 loss function but also four novel loss functions when training the network. Two of them retain the rigidity of eyeballs and eyelids; and the other two prevent color discontinuity on the eye peripheries. By leveraging the CPU and the GPU resources, our implementation achieves real-time performance (i.e., 31 frames per second). Experimental results show that the gazes redirected by our system are of high quality under this restrict time constraint.We also conducted an objective evaluation of our system by measuring the peak signal-to-noise ratio (PSNR) between the real and the synthesized images.

原文English
主出版物標題Proceedings of the 9th ACM Multimedia Systems Conference, MMSys 2018
發行者Association for Computing Machinery, Inc
頁面509-512
頁數4
ISBN(電子)9781450351928
DOIs
出版狀態Published - 12 六月 2018
事件9th ACM Multimedia Systems Conference, MMSys 2018 - Amsterdam, Netherlands
持續時間: 12 六月 201815 六月 2018

出版系列

名字Proceedings of the 9th ACM Multimedia Systems Conference, MMSys 2018

Conference

Conference9th ACM Multimedia Systems Conference, MMSys 2018
國家Netherlands
城市Amsterdam
期間12/06/1815/06/18

指紋 深入研究「Realizing the real-time gaze redirection system with convolutional neural network」主題。共同形成了獨特的指紋。

引用此