Deep 360 pilot: Learning a deep agent for piloting through 360° sports videos

Hou Ning Hu, Yen Chen Lin, Ming Yu Liu, Hsien Tzu Cheng, Yung-Ju Chang, Min Sun

研究成果: Conference contribution同行評審

65 引文 斯高帕斯(Scopus)

摘要

Watching a 360° sports video requires a viewer to continuously select a viewing angle, either through a sequence of mouse clicks or head movements. To relieve the viewer from this "360 piloting" task, we propose "deep 360 pilot" - a deep learning-based agent for piloting through 360° sports videos automatically. At each frame, the agent observes a panoramic image and has the knowledge of previously selected viewing angles. The task of the agent is to shift the current viewing angle (i.e. action) to the next preferred one (i.e., goal). We propose to directly learn an online policy of the agent from data. Specifically, we leverage a state-of-the-art object detector to propose a few candidate objects of interest (yellow boxes in Fig. 1). Then, a recurrent neural network is used to select the main object (green dash boxes in Fig. 1). Given the main object and previously selected viewing angles, our method regresses a shift in viewing angle to move to the next one. We use the policy gradient technique to jointly train our pipeline, by minimizing: (1) a regression loss measuring the distance between the selected and ground truth viewing angles, (2) a smoothness loss encouraging smooth transition in viewing angle, and (3) maximizing an expected reward of focusing on a foreground object. To evaluate our method, we built a new 360-Sports video dataset consisting of five sports domains. We trained domain-specific agents and achieved the best performance on viewing angle selection accuracy and users' preference compared to [53] and other baselines.

原文English
主出版物標題Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
發行者Institute of Electrical and Electronics Engineers Inc.
頁面1396-1405
頁數10
ISBN(電子)9781538604571
DOIs
出版狀態Published - 6 十一月 2017
事件30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 - Honolulu, United States
持續時間: 21 七月 201726 七月 2017

出版系列

名字Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
2017-January

Conference

Conference30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
國家United States
城市Honolulu
期間21/07/1726/07/17

指紋 深入研究「Deep 360 pilot: Learning a deep agent for piloting through 360° sports videos」主題。共同形成了獨特的指紋。

引用此