Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data

Akshi Kumar*, Kathiravan Srinivasan, Wen Huang Cheng, Albert Y. Zomaya

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

26 Scopus citations

Abstract

Detecting sentiments in natural language is tricky even for humans, making its automated detection more complicated. This research proffers a hybrid deep learning model for fine-grained sentiment prediction in real-time multimodal data. It reinforces the strengths of deep learning nets in combination to machine learning to deal with two specific semiotic systems, namely the textual (written text) and visual (still images) and their combination within the online content using decision level multimodal fusion. The proposed contextual ConvNet-SVMBoVW model, has four modules, namely, the discretization, text analytics, image analytics, and decision module. The input to the model is multimodal text, m ε {text, image, info-graphic}. The discretization module uses Google Lens to separate the text from the image, which is then processed as discrete entities and sent to the respective text analytics and image analytics modules. Text analytics module determines the sentiment using a hybrid of a convolution neural network (ConvNet) enriched with the contextual semantics of SentiCircle. An aggregation scheme is introduced to compute the hybrid polarity. A support vector machine (SVM) classifier trained using bag-of-visual-words (BoVW) for predicting the visual content sentiment. A Boolean decision module with a logical OR operation is augmented to the architecture which validates and categorizes the output on the basis of five fine-grained sentiment categories (truth values), namely ‘highly positive,’ ‘positive,’ ‘neutral,’ ‘negative’ and ‘highly negative.’ The accuracy achieved by the proposed model is nearly 91% which is an improvement over the accuracy obtained by the text and image modules individually.

Original languageEnglish
Article number102141
JournalInformation Processing and Management
Volume57
Issue number1
DOIs
StatePublished - Jan 2020

Keywords

  • BoVW
  • Context
  • Deep learning
  • Multimodal
  • Sentiment analysis

Fingerprint Dive into the research topics of 'Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data'. Together they form a unique fingerprint.

Cite this