Smart self-checkout carts based on deep learning for shopping activity recognition

Hong Chuan Chi, Muhammad Atif Sarwar, Yousef Awwad Daraghmi, Kuan Wen Lin, Tsi-Ui Ik*, Yih-Lang Li

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Fast and reliable communication plays a major role in the success of smart shopping applications. In a 'Just Walk Out' shopping scenario, a video camera is installed on the cart to monitor shopping activities and transmit images to the cloud for processing so that items in the cart can be tracked and checked out. This paper proposes a prototype of a smart shopping cart based on image-based action recognition. Firstly, deep learning networks such as Faster R-CNN, YOLOv2, and YOLOv2-Tiny are utilized to analyze the content of each video frame. Frames are classified into three classes: No Hand, Empty Hand, and Holding Items. The classification accuracy based on Faster R-CNN, YOLOv2, or YOLOv2-Tiny is between 93.0% and 90.3%, and the processing speed of the three networks can be up to 5 fps, 39 fps, and 50 fps, respectively. Secondly, based on the sequence of frame classes, the timeline is divided into No Hand intervals, Empty Hand intervals, and Holding Items intervals. The accuracy of action recognition is 96%, and the time error is 0.119s on average. Finally, we categorize the events into four cases: No Change, placing, Removing, and Swapping. Even including the correctness of the item recognition, the accuracy of shopping event detection is 97.9%, which is higher than the minimal requirement to deploy such a system in a smart shopping environment. A demo of the system and a link to download the data set used in the paper are in Smart Shopping Cart Prototype or found at this URL: https://hackmd.io/abEiC83rQoqxz7zpL4Kh2w.

Original languageEnglish
Title of host publicationAPNOMS 2020 - 2020 21st Asia-Pacific Network Operations and Management Symposium
Subtitle of host publicationTowards Service and Networking Intelligence for Humanity
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages185-190
Number of pages6
ISBN (Electronic)9788995004388
DOIs
StatePublished - Sep 2020
Event21st Asia-Pacific Network Operations and Management Symposium, APNOMS 2020 - Daegu, Korea, Republic of
Duration: 22 Sep 202025 Sep 2020

Publication series

NameAPNOMS 2020 - 2020 21st Asia-Pacific Network Operations and Management Symposium: Towards Service and Networking Intelligence for Humanity

Conference

Conference21st Asia-Pacific Network Operations and Management Symposium, APNOMS 2020
CountryKorea, Republic of
CityDaegu
Period22/09/2025/09/20

Keywords

  • Action recognition
  • Faster R-CNN
  • Frame classification
  • Smart shopping cart
  • YOLOv2
  • YOLOv2-Tiny

Fingerprint Dive into the research topics of 'Smart self-checkout carts based on deep learning for shopping activity recognition'. Together they form a unique fingerprint.

Cite this