Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network with Multiscale Feature Fusion

Reconstructing visual stimulus from human brain activity measured with functional magnetic resonance imaging (fMRI) is a challenging decoding task for revealing the visual system. Recent deep learning approaches commonly neglect the relationship between hierarchical image features and different regions of the visual cortex, and fail to use global and local image features in reconstructing visual stimulus. To address these issues, in this article, a novel neural decoding framework is proposed by using a dual attention (DA) hierarchical latent generative network with multiscale feature fusion (DA-HLGN-MSFF) method. Specifically, the fMRI data are first encoded to hierarchical features of our image encoder network, which employs a multikernel convolution block to extract the multiscale spatial information of images. In order to reconstruct the perceived images and further improve the performance of our generator network, a DA block based on the channel-spatial attention mechanism is then proposed to exploit the interchannel relationships and spatial long-range dependencies of features. Moreover, a multiscale feature fusion block is finally adopted to aggregate the global and local information of features at different scales and synthesize the final reconstructed images in the generator network. Competitive experimental results on two public fMRI data sets demonstrate that our method is able to achieve promising reconstructing performance compared with the state-of-the-art methods. The codes of our proposed DA-HLGN-MSFF method will be open access on https://github.com/ljbuaa/HLDAGN.