TAGformer: A Multimodal Physiological Signals Fusion Network for Pilot Stress Recognition

Pilot stress recognition is crucial for safe and smooth flight, while heightened stress can significantly impede pilots’ capacity to respond to potential dangers. Recent research has witnessed the success of deep learning (DL) models using multimodal physiological signals in achieving high classification accuracy. However, these models often overlook the intricate dependencies among these signals, especially as they vary with different stress levels. Explicitly modeling, these dependencies can enhance feature extraction efficiency using a more compact network model, thus improving the accuracy of stress recognition. Therefore, we propose a novel DL model for pilot stress recognition based on 14 pilots’ physiological data, including electrocardiography (ECG), electromyography (EMG), heart rate (HR), respiration (RESP), and skin temperature (SKT). Handcrafted features from these physiological data are initially organized into a graph and fused using a topology adaptive graph convolutional module (TAGCM). Then, the extracted features are fed into a transformer encoder followed by a multilayer perceptron (MLP) for recognizing stress. The multistage gated average fusion (MGAF) was employed to fuse features from different modules. Experiments were conducted using a self-collected and organized dataset for pilots’ stress recognition as well as a publicly available dataset for drivers’ stress recognition. The experimental results show that the proposed model could achieve better results in terms of feature extraction ability and classification accuracy of different pilots’ stress levels than other baseline methods. Moreover, the outcomes obtained from the experiment on the public dataset underscore the potential of the proposed model in effectively recognizing stress across diverse scenarios.