TY - GEN
T1 - Continuous emotion recognition in videos by fusing facial expression, head pose and eye gaze
AU - Wu, Suowei
AU - Du, Zhengyin
AU - Li, Weixin
AU - Huang, Di
AU - Wang, Yunhong
N1 - Publisher Copyright:
© 2019 Association for Computing Machinery.
PY - 2019/10/14
Y1 - 2019/10/14
N2 - Continuous emotion recognition is of great significance in affective computing and human-computer interaction. Most of existing methods for video based continuous emotion recognition utilize facial expression. However, besides facial expression, other clues including head pose and eye gaze are also closely related to human emotion, but have not beenwell explored in continuous emotion recognition task. On the one hand, head pose and eye gaze could result in different degrees of credibility of facial expression features. On the other hand, head pose and eye gaze carry emotional clues themselves, which are complementary to facial expression. Accordingly, in this paper we propose two ways to incorporate these two clues into continuous emotion recognition. They are respectively an attention mechanism based on head pose and eye gaze clues to guide the utilization of facial features in continuous emotion recognition, and an auxiliary line which helps extract more useful emotion information from head pose and eye gaze. Experiments are conducted on the Recola dataset, a database for continuous emotion recognition, and the results show that our framework outperforms other stateof- the-art methods due to the full use of head pose and eye gaze clues in addition to facial expression for continuous emotion recognition.
AB - Continuous emotion recognition is of great significance in affective computing and human-computer interaction. Most of existing methods for video based continuous emotion recognition utilize facial expression. However, besides facial expression, other clues including head pose and eye gaze are also closely related to human emotion, but have not beenwell explored in continuous emotion recognition task. On the one hand, head pose and eye gaze could result in different degrees of credibility of facial expression features. On the other hand, head pose and eye gaze carry emotional clues themselves, which are complementary to facial expression. Accordingly, in this paper we propose two ways to incorporate these two clues into continuous emotion recognition. They are respectively an attention mechanism based on head pose and eye gaze clues to guide the utilization of facial features in continuous emotion recognition, and an auxiliary line which helps extract more useful emotion information from head pose and eye gaze. Experiments are conducted on the Recola dataset, a database for continuous emotion recognition, and the results show that our framework outperforms other stateof- the-art methods due to the full use of head pose and eye gaze clues in addition to facial expression for continuous emotion recognition.
KW - Attention
KW - Continuous Emotion Recognition
KW - Eye gaze
KW - Facial Expression
KW - Head Pose
UR - https://www.scopus.com/pages/publications/85074905343
U2 - 10.1145/3340555.3353739
DO - 10.1145/3340555.3353739
M3 - 会议稿件
AN - SCOPUS:85074905343
T3 - ICMI 2019 - Proceedings of the 2019 International Conference on Multimodal Interaction
SP - 40
EP - 48
BT - ICMI 2019 - Proceedings of the 2019 International Conference on Multimodal Interaction
A2 - Gao, Wen
A2 - Ling Meng, Helen Mei
A2 - Turk, Matthew
A2 - Fussell, Susan R.
A2 - Schuller, Bjorn
A2 - Schuller, Bjorn
A2 - Song, Yale
A2 - Yu, Kai
PB - Association for Computing Machinery, Inc
T2 - 21st ACM International Conference on Multimodal Interaction, ICMI 2019
Y2 - 14 October 2019 through 18 October 2019
ER -