DLWV2: A Deep Learning-Based Wearable Vision-System with Vibrotactile-Feedback for Visually Impaired People to Reach Objects

Meng Li Shih, Yi Chun Chen, Chia Yu Tung, Cheng Sun, Ching Ju Cheng, Li-Wei Chan, Srenivas Varadarajan, Min Sun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

We develop a Deep Learning-based Wearable Vision-system with Vibrotactile-feedback (DLWV2)to guide Blind and Visually Impaired (BVI)people to reach objects. The system achieves high accuracy in object detection and tracking in 3-D using an extended deep learning-based 2.5-D detector and a 3-D object tracker with the ability to track 3-D object locations even outside the camera field-of-view. We train our detector with a large number of images with 2.5-D object ground-truth (i.e., 2-D object bounding boxes and distance from the camera to objects). A novel combination of HTC Vive Tracker with our system enables us to automatically obtain the ground-truth labels for training while requiring very little human effort to set up the system. Moreover, our system processes frames in real-time through a client-server computing platform such that BVI people can receive realtime vibrotactile guidance. We conduct a thorough user study on 12 BVI people in new environments with object instances which are unseen during training. Our system outperforms the non-assistive guiding strategy with statistic significance in both time and the number of contacting irrelevant objects. Finally, the interview with BVI users confirms that our system with distance-based vibrotactile feedback is mostly preferred, especially for objects requiring gentle manipulation such as a bottle with water inside.

Original languageEnglish
Title of host publication2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages7904-7911
Number of pages8
ISBN (Electronic)9781538680940
DOIs
StatePublished - 27 Dec 2018
Event2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018 - Madrid, Spain
Duration: 1 Oct 20185 Oct 2018

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

Conference2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018
CountrySpain
CityMadrid
Period1/10/185/10/18

Fingerprint Dive into the research topics of 'DLWV2: A Deep Learning-Based Wearable Vision-System with Vibrotactile-Feedback for Visually Impaired People to Reach Objects'. Together they form a unique fingerprint.

  • Cite this

    Shih, M. L., Chen, Y. C., Tung, C. Y., Sun, C., Cheng, C. J., Chan, L-W., Varadarajan, S., & Sun, M. (2018). DLWV2: A Deep Learning-Based Wearable Vision-System with Vibrotactile-Feedback for Visually Impaired People to Reach Objects. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018 (pp. 7904-7911). [8593711] (IEEE International Conference on Intelligent Robots and Systems). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IROS.2018.8593711