Nonlinear Response Analysis in the Multi-Pass Cavity SERF Comagnetometer

Enhancing the light-atom interaction is a critical approach to improving the sensitivity of the spin-based quantum sensors. However, as the optical rotation increases, the output signal exhibits nonlinear characteristics that consequently impair the critical operation of the SERF comagnetometer, thereby negating the sensitivity gain. To address this issue, the nonlinear response model of the SERF comagnetometer is established, and the evolution of the coupled spin ensemble near the self-compensation state is analyzed based on this model. Based on the analysis of nonlinear responses under different nuclear spin self-compensation capabilities, a quasi-static parameter measurement method is proposed, which minimizes nonlinear response disturbance while decoupling probe light intensity noise, thereby enhancing measurement sensitivity. Experimental validation on a SERF comagnetometer with an integrated 14-pass cell confirms the accuracy of the nonlinear model. The self-compensation state was successfully determined under a 19.8-fold optical rotation amplification, and the suppression of probe laser intensity noise through frequency fitting achieving a 1.4-fold sensitivity improvement at 1 Hz. This study provides a significant advancement in light-atom interaction enhancement systems and enhances the sensitivity of SERF comagnetometers.