Multi-View and Multi-Modal Action Recognition with Learned Fusion

Sandy Ardianto; Hsueh-Ming Hang

doi:10.23919/APSIPA.2018.8659539

Multi-View and Multi-Modal Action Recognition with Learned Fusion

Sandy Ardianto, Hsueh-Ming Hang

Institute of Electronics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Scopus citations

Abstract

In this paper, we study multi-modal and multi-view action recognition system based on the deep-learning techniques. We extended the Temporal Segment Network with additional data fusion stage to combine information from different sources. In this research, we use multiple types of information from different modality such as RGB, depth, infrared data to detect predefined human actions. We tested various combinations of these data sources to examine their impact on the final detection accuracy. We designed 3 information fusion methods to generate the final decision. The most interested one is the Learned Fusion Net designed by us. It turns out the Learned Fusion structure has the best results but requires more training.

Original language	English
Title of host publication	2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1601-1604
Number of pages	4
ISBN (Electronic)	9789881476852
DOIs	https://doi.org/10.23919/APSIPA.2018.8659539
State	Published - 4 Mar 2019
Event	10th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Honolulu, United States Duration: 12 Nov 2018 → 15 Nov 2018

Publication series

Name	2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings

Conference

Conference	10th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018
Country	United States
City	Honolulu
Period	12/11/18 → 15/11/18

Keywords

deep learning
human action recognition
information fusion
multi-modal video
multi-view video
neural nets

Access to Document

10.23919/APSIPA.2018.8659539

Link to publication in Scopus

Fingerprint Dive into the research topics of 'Multi-View and Multi-Modal Action Recognition with Learned Fusion'. Together they form a unique fingerprint.

Cite this

Ardianto, S., & Hang, H-M. (2019). Multi-View and Multi-Modal Action Recognition with Learned Fusion. In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings (pp. 1601-1604). [8659539] (2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.23919/APSIPA.2018.8659539

Ardianto, Sandy ; Hang, Hsueh-Ming. / Multi-View and Multi-Modal Action Recognition with Learned Fusion. 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 1601-1604 (2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings).

@inproceedings{fd5247a8b4ef4581ab36d4dbe762781c,

title = "Multi-View and Multi-Modal Action Recognition with Learned Fusion",

abstract = "In this paper, we study multi-modal and multi-view action recognition system based on the deep-learning techniques. We extended the Temporal Segment Network with additional data fusion stage to combine information from different sources. In this research, we use multiple types of information from different modality such as RGB, depth, infrared data to detect predefined human actions. We tested various combinations of these data sources to examine their impact on the final detection accuracy. We designed 3 information fusion methods to generate the final decision. The most interested one is the Learned Fusion Net designed by us. It turns out the Learned Fusion structure has the best results but requires more training.",

keywords = "deep learning, human action recognition, information fusion, multi-modal video, multi-view video, neural nets",

author = "Sandy Ardianto and Hsueh-Ming Hang",

year = "2019",

month = mar,

day = "4",

doi = "10.23919/APSIPA.2018.8659539",

language = "English",

series = "2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1601--1604",

booktitle = "2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings",

note = "null ; Conference date: 12-11-2018 Through 15-11-2018",

}

Ardianto, S & Hang, H-M 2019, Multi-View and Multi-Modal Action Recognition with Learned Fusion. in 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings., 8659539, 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings, Institute of Electrical and Electronics Engineers Inc., pp. 1601-1604, 10th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018, Honolulu, United States, 12/11/18. https://doi.org/10.23919/APSIPA.2018.8659539

Multi-View and Multi-Modal Action Recognition with Learned Fusion. / Ardianto, Sandy; Hang, Hsueh-Ming.

2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1601-1604 8659539 (2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Multi-View and Multi-Modal Action Recognition with Learned Fusion

AU - Ardianto, Sandy

AU - Hang, Hsueh-Ming

PY - 2019/3/4

Y1 - 2019/3/4

N2 - In this paper, we study multi-modal and multi-view action recognition system based on the deep-learning techniques. We extended the Temporal Segment Network with additional data fusion stage to combine information from different sources. In this research, we use multiple types of information from different modality such as RGB, depth, infrared data to detect predefined human actions. We tested various combinations of these data sources to examine their impact on the final detection accuracy. We designed 3 information fusion methods to generate the final decision. The most interested one is the Learned Fusion Net designed by us. It turns out the Learned Fusion structure has the best results but requires more training.

AB - In this paper, we study multi-modal and multi-view action recognition system based on the deep-learning techniques. We extended the Temporal Segment Network with additional data fusion stage to combine information from different sources. In this research, we use multiple types of information from different modality such as RGB, depth, infrared data to detect predefined human actions. We tested various combinations of these data sources to examine their impact on the final detection accuracy. We designed 3 information fusion methods to generate the final decision. The most interested one is the Learned Fusion Net designed by us. It turns out the Learned Fusion structure has the best results but requires more training.

KW - deep learning

KW - human action recognition

KW - information fusion

KW - multi-modal video

KW - multi-view video

KW - neural nets

UR - http://www.scopus.com/inward/record.url?scp=85063546883&partnerID=8YFLogxK

U2 - 10.23919/APSIPA.2018.8659539

DO - 10.23919/APSIPA.2018.8659539

M3 - Conference contribution

AN - SCOPUS:85063546883

T3 - 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings

SP - 1601

EP - 1604

BT - 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 12 November 2018 through 15 November 2018

ER -

Ardianto S, Hang H-M. Multi-View and Multi-Modal Action Recognition with Learned Fusion. In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1601-1604. 8659539. (2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2018 - Proceedings). https://doi.org/10.23919/APSIPA.2018.8659539