Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR

Yang Shen; Ming Li; Yu Li; Yang Gao; Junjun Pan

doi:10.1109/ISMAR-Adjunct60411.2023.00153

Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR

Yang Shen
, Ming Li
, Yu Li
, Yang Gao
, Junjun Pan^*

^*Corresponding author for this work

School of Computer Science and Engineering

Beijing Normal University
Beihang University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

During VR cognitive rehabilitation, user performance in cognitive tasks is influenced by their emotional states. However, most current research primarily focuses on emotional valence, lacking a comprehensive examination of the impact of emotions on cognitive activities and their underlying physiological mechanisms based on an arousal-valence dimensional model. To address this gap, we have conducted an experiment in which participants are induced with four distinct emotions using VR videos, followed by cognitive tasks. Concurrently, we collect participants' EEG, ECG, and GSR signals. The results indicate that high arousal and high valence emotional states significantly improve user performance in cognitive tasks (shorter reaction times).

Original language	English
Title of host publication	Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023
Editors	Gerd Bruder, Anne-Helene Olivier, Andrew Cunningham, Evan Yifan Peng, Jens Grubert, Ian Williams
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	714-715
Number of pages	2
ISBN (Electronic)	9798350328912
DOIs	https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00153
State	Published - 2023
Event	2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023 - Sydney, Australia Duration: 16 Oct 2023 → 20 Oct 2023

Publication series

Name	Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023

Conference

Conference	2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023
Country/Territory	Australia
City	Sydney
Period	16/10/23 → 20/10/23

Keywords

Human computer interaction (HCI)
Human-centered computing
Interaction paradigms
Virtual reality
Visualization
Visualization design and evaluation methods

Access to Document

10.1109/ISMAR-Adjunct60411.2023.00153

Cite this

Shen, Y., Li, M., Li, Y., Gao, Y., & Pan, J. (2023). Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR. In G. Bruder, A.-H. Olivier, A. Cunningham, E. Y. Peng, J. Grubert, & I. Williams (Eds.), Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023 (pp. 714-715). (Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00153

Shen, Yang ; Li, Ming ; Li, Yu et al. / Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR. Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023. editor / Gerd Bruder ; Anne-Helene Olivier ; Andrew Cunningham ; Evan Yifan Peng ; Jens Grubert ; Ian Williams. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 714-715 (Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023).

@inproceedings{ff578c5786ac44ba86bb4106c1b1007d,

title = "Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR",

abstract = "During VR cognitive rehabilitation, user performance in cognitive tasks is influenced by their emotional states. However, most current research primarily focuses on emotional valence, lacking a comprehensive examination of the impact of emotions on cognitive activities and their underlying physiological mechanisms based on an arousal-valence dimensional model. To address this gap, we have conducted an experiment in which participants are induced with four distinct emotions using VR videos, followed by cognitive tasks. Concurrently, we collect participants' EEG, ECG, and GSR signals. The results indicate that high arousal and high valence emotional states significantly improve user performance in cognitive tasks (shorter reaction times).",

keywords = "Human computer interaction (HCI), Human-centered computing, Interaction paradigms, Virtual reality, Visualization, Visualization design and evaluation methods",

author = "Yang Shen and Ming Li and Yu Li and Yang Gao and Junjun Pan",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023 ; Conference date: 16-10-2023 Through 20-10-2023",

year = "2023",

doi = "10.1109/ISMAR-Adjunct60411.2023.00153",

language = "英语",

series = "Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "714--715",

editor = "Gerd Bruder and Anne-Helene Olivier and Andrew Cunningham and Peng, \{Evan Yifan\} and Jens Grubert and Ian Williams",

booktitle = "Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023",

address = "美国",

}

Shen, Y, Li, M, Li, Y, Gao, Y & Pan, J 2023, Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR. in G Bruder, A-H Olivier, A Cunningham, EY Peng, J Grubert & I Williams (eds), Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023. Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023, Institute of Electrical and Electronics Engineers Inc., pp. 714-715, 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023, Sydney, Australia, 16/10/23. https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00153

Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR. / Shen, Yang; Li, Ming; Li, Yu et al.
Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023. ed. / Gerd Bruder; Anne-Helene Olivier; Andrew Cunningham; Evan Yifan Peng; Jens Grubert; Ian Williams. Institute of Electrical and Electronics Engineers Inc., 2023. p. 714-715 (Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR

AU - Shen, Yang

AU - Li, Ming

AU - Li, Yu

AU - Gao, Yang

AU - Pan, Junjun

PY - 2023

Y1 - 2023

N2 - During VR cognitive rehabilitation, user performance in cognitive tasks is influenced by their emotional states. However, most current research primarily focuses on emotional valence, lacking a comprehensive examination of the impact of emotions on cognitive activities and their underlying physiological mechanisms based on an arousal-valence dimensional model. To address this gap, we have conducted an experiment in which participants are induced with four distinct emotions using VR videos, followed by cognitive tasks. Concurrently, we collect participants' EEG, ECG, and GSR signals. The results indicate that high arousal and high valence emotional states significantly improve user performance in cognitive tasks (shorter reaction times).

AB - During VR cognitive rehabilitation, user performance in cognitive tasks is influenced by their emotional states. However, most current research primarily focuses on emotional valence, lacking a comprehensive examination of the impact of emotions on cognitive activities and their underlying physiological mechanisms based on an arousal-valence dimensional model. To address this gap, we have conducted an experiment in which participants are induced with four distinct emotions using VR videos, followed by cognitive tasks. Concurrently, we collect participants' EEG, ECG, and GSR signals. The results indicate that high arousal and high valence emotional states significantly improve user performance in cognitive tasks (shorter reaction times).

KW - Human computer interaction (HCI)

KW - Human-centered computing

KW - Interaction paradigms

KW - Virtual reality

KW - Visualization

KW - Visualization design and evaluation methods

UR - https://www.scopus.com/pages/publications/85180361376

U2 - 10.1109/ISMAR-Adjunct60411.2023.00153

DO - 10.1109/ISMAR-Adjunct60411.2023.00153

M3 - 会议稿件

AN - SCOPUS:85180361376

T3 - Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023

SP - 714

EP - 715

BT - Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023

A2 - Bruder, Gerd

A2 - Olivier, Anne-Helene

A2 - Cunningham, Andrew

A2 - Peng, Evan Yifan

A2 - Grubert, Jens

A2 - Williams, Ian

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023

Y2 - 16 October 2023 through 20 October 2023

ER -

Shen Y, Li M, Li Y, Gao Y , Pan J. Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR. In Bruder G, Olivier AH, Cunningham A, Peng EY, Grubert J, Williams I, editors, Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 714-715. (Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2023). doi: 10.1109/ISMAR-Adjunct60411.2023.00153

Multimodal Physiological Analysis of Cognition and the Correlations with Emotion in VR

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Cite this