A Cross-Modal Network Integrating the Spatial-Frequency Relationship of EEG for State Recognition

The accurate recognition of pilots' cognitive states is critical for ensuring the effectiveness of aerial operations. To better capture inter-modal correlations in physiological signals and reduce the complexity of high-frequency raw data, we propose a novel model that hierarchically extracts spatial-frequency-temporal features from EEG signals while explicitly modeling cross-modal dependencies for the simultaneous recognition of pilot vigilance and stress. Specifically, multi-channel EEG signals are transformed into a 4D feature representation, and features from multimodal signals of varying frequencies are aligned to a unified low-frequency domain. We then employ a cross-modal interactive Trans-former to explore latent multimodal relationships and facilitate mutual enhancement among modalities. Evaluated on a dataset on pilot physiological data from simulated combat scenarios, our method significantly outperforms both simple feature concatenation and isolated cross-modal fusion in cognitive state recognition. The model's effectiveness is further validated on the public DEAP dataset.