Single-beam atomic magnetometry with sub-nanotesla heading error based on hyperfine alignment resonances

Heading error is a major limitation for resonance-based atomic magnetometers operating in the geomagnetic-field range, where the magnetic-field direction often changes with respect to the sensor. Here, we investigate the physical origin of heading error in a single-beam amplitude-modulated nonlinear magneto-optical rotation magnetometer based on the hyperfine alignment resonances of the (Formula presented) (Formula presented) Rb D1 line in a buffer-gas vapor cell. Unlike orientation-based magnetometers, where nonlinear Zeeman effects usually constitute a major source of heading error, the (Formula presented) (Formula presented) alignment resonance spectrum maintains a stable center frequency under different magnetic-field orientations when the individual resonance components are unresolved. In practical single-beam measurements, the observed heading error is associated with an orientation-dependent residual spectral background, which breaks the apparent symmetry of the alignment-resonance profile and biases the extracted center frequency. To suppress this background-induced bias, we propose a multi-point frequency extraction method that retrieves the resonance center from distorted single-beam spectra. In unshielded geomagnetic-field measurements, the equal-weight pooled peak-to-peak heading-error estimate for the (Formula presented) (Formula presented) manifold is reduced to 0.32 nT, with a one-sided 95% upper coverage limit of 0.90 nT. Separately, an estimated photon-shot-noise-limited sensitivity of about (Formula presented) (Formula presented) is obtained. These results provide a compact route to low-heading-error and high-sensitivity atomic magnetometry for practical geomagnetic-field quantum sensing.