Sentiment analysis of social images via hierarchical deep fusion of content and links

Sentiment analysis is crucial for many social media analytic tasks. Earlier researches mainly focus on single modality, e.g., text description or visual content. Recently, more and more works pay attention to the incorporation of multiple modalities. Different from the traditional image database, social images usually interconnect with each other, which makes the sentiment analysis nontrivial. Most existing methods consider different images independently, which cannot be directly applied to the interconnected images. In this paper, we propose a novel Hierarchical Deep Fusion (HDF)model to explore the cross-modal correlations among images, texts, and their social links, which can learn comprehensive and complementary features for more effective sentiment analysis. Specifically, we combine the visual content with different semantic fragments of textual content through a three-level hierarchical LSTMs (H-LSTMs)to learn the inter-modal correlations between image and text at different levels. To exploit the link information effectively, the linkages among social images are modeled by a weighted relation network and each node is embedded into a distributed vector. Then, the extracted image–text features and node embeddings are fused by a Multi-Layer Perceptron (MLP)to further capture the non-linear cross-modal correlations for sentiment prediction. Comprehensive experiments are conducted to demonstrate the effectiveness of our approach on both machine weakly labeled and manually labeled datasets.