Intelligent gravitational wave detection with pulsar timing array using complex-valued convolutional neural network

The detection of nanohertz gravitational waves using pulsar timing arrays (PTAs) represents a frontier in astrophysics, crucial for probing dynamical processes of supermassive black hole binaries (SMBHBs) and early universe cosmology. However, the extraction of these faint signals remains profoundly challenging due to the exceptionally low signal-to-noise ratio (SNR) in PTA datasets, where gravitational wave signatures are often obscured by correlated noise and instrumental artifacts. Conventional analysis techniques, which rely on stringent statistical assumptions and template matching, exhibit limited sensitivity in such regimes. To overcome these limitations, we propose a novel intelligent detection framework that integrates a complex-valued convolutional neural network (CV-CNN) with ant colony optimization (ACO). The ACO algorithm first optimizes the spatial arrangement of pulsars to enhance the intrinsic coherence of gravitational wave signals across the array, effectively exploiting the quadrupole spatial correlation pattern expected from gravitational waves. The data are then transformed into the frequency domain, effectively concentrating discriminative features into low frequencies while reducing dimensionality. This preprocessed data is fed into a CV-CNN, which leverages complex-valued operations to capture phase-sensitive patterns and non-linear relationships that are critical for detecting stochastic gravitational wave backgrounds. Simulation results indicate our method demonstrates superior performance, achieving over 3.6% higher detection sensitivity under SNR conditions below 6.4121 × 10⁻². Furthermore, the framework demonstrates robust performance across different astrophysical population models of SMBHBs. The framework not only offers a powerful tool for imminent nanohertz gravitational astronomy but also establishes a new paradigm for machine learning-based signal processing in high-noise environments, with potential applications in other areas of low-frequency astrophysical data analysis.