MHQA: Enhanced nnU-Net via multi-head and 3D quadruplet attention for tooth instance segmentation in CBCT images

Precise segmentation of dental structures in cone-beam computed tomography (CBCT) is essential for computer-aided diagnosis and treatment planning. However, anatomical symmetry, high morphological similarity between adjacent teeth, and imaging noise significantly hinder accurate multi-class tooth segmentation. To address these challenges, we propose MHQA, a novel segmentation framework built upon nnU-Net that integrates a newly designed three-dimensional Quadruple Attention (3D-QA) module as the core component, complemented by a multi-head self-attention (MHSA) module. The proposed 3D-QA module explicitly models spatial–channel interactions in 3D feature space and enhances boundary-aware representations, enabling more accurate delineation of adjacent teeth in densely packed and low-contrast regions. The auxiliary MHSA module facilitates global context modeling, reducing confusion between morphologically similar and contralaterally symmetric teeth while maintaining computational efficiency. The proposed method is evaluated on two CBCT datasets, including the public CUI dataset and a newly constructed TDT dataset with 32-class annotations and pronounced anatomical variability. Experimental results demonstrate that MHQA consistently outperforms state-of-the-art CNN- and Transformer-based methods, achieving Dice Similarity Coefficients of 81.53% on CUI and 67.52% on TDT. Ablation studies further confirm the effectiveness of the proposed attention modules, indicating that MHQA effectively balances fine-grained local detail extraction and global semantic context modeling for complex CBCT tooth segmentation.