MCFM: Multimodal Competitive Fusion Mechanism for Sentiment Analysis

With the popularity of social media, users are able to express their opinions in multiple forms, such as text, audio, and video. Traditional unimodal sentiment analysis methods can no longer meet the processing requirements of such multisource heterogeneous data, which makes multimodal sentiment analysis a research hotspot. However, most existing methods rely on simple feature splicing or weighted fusion, neglecting the differences in the reliability of different modalities and failing to fully explore the intermodality consistency and difference information. In this article, we propose a multimodal competitive fusion mechanism and construct multimodal competitive fusion model (MCFM). The model first dynamically evaluates the reliability of each modality through the competition mechanism and adaptively assigns weights accordingly. Then it decomposed the modality representations into similar and dissimilar features through modality feature decomposition, supplemented by the overlap of orthogonal traffic channel attention constraints, to achieve the collaborative learning of consistency and dissimilarity features. We evaluate the proposed model on several datasets. In our experiments, we used textual modality data from the dataset with audio modality data for the experiments. The experimental results show that MCFM has 2%–3% higher binary accuracy (ACC2) than the baseline model on the sentiment classification task (with 2% higher binary accuracy under the negative/nonnegative metrics and 3% higher binary accuracy under the positive/negative metrics), and that on the regression task, MCFM’s mean absolute error on the test dataset is 3% lower than that of the baseline model.