Realizing the real-time gaze redirection system with convolutional neural network

Chih Fan Hsu, Yu Cheng Chen, Yu-Shuen Wang, Chin Laung Lei, Kuan Ta Chen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Retaining eye contact of remote users is a critical issue in video conferencing systems because of parallax caused by the physical distance between a screen and a camera. To achieve this objective, we present a real-time gaze redirection system called Flx-gaze to post-process each video frame before sending it to the remote end. Specifically, we relocate and relight the pixels representing eyes by using a convolutional neural network (CNN). To prevent visual artifacts during manipulation, we minimize not only the L2 loss function but also four novel loss functions when training the network. Two of them retain the rigidity of eyeballs and eyelids; and the other two prevent color discontinuity on the eye peripheries. By leveraging the CPU and the GPU resources, our implementation achieves real-time performance (i.e., 31 frames per second). Experimental results show that the gazes redirected by our system are of high quality under this restrict time constraint.We also conducted an objective evaluation of our system by measuring the peak signal-to-noise ratio (PSNR) between the real and the synthesized images.

Original languageEnglish
Title of host publicationProceedings of the 9th ACM Multimedia Systems Conference, MMSys 2018
PublisherAssociation for Computing Machinery, Inc
Pages509-512
Number of pages4
ISBN (Electronic)9781450351928
DOIs
StatePublished - 12 Jun 2018
Event9th ACM Multimedia Systems Conference, MMSys 2018 - Amsterdam, Netherlands
Duration: 12 Jun 201815 Jun 2018

Publication series

NameProceedings of the 9th ACM Multimedia Systems Conference, MMSys 2018

Conference

Conference9th ACM Multimedia Systems Conference, MMSys 2018
CountryNetherlands
CityAmsterdam
Period12/06/1815/06/18

Keywords

  • Convolutional Neural Network
  • Gaze Manipulation

Fingerprint Dive into the research topics of 'Realizing the real-time gaze redirection system with convolutional neural network'. Together they form a unique fingerprint.

Cite this