MMDFusion: Multimodal Deformable Fusion for Robust 3-D Detection in Unstructured Road Environments

Three-dimensional object detection in unstructured environments, such as open-pit mines, faces severe challenges due to rugged terrain and complex environmental conditions. In particular, the accuracy of detection in these scenarios is significantly degraded by imprecise image depth estimation, point cloud sparsity, and extrinsic calibration errors caused by vehicle vibrations. To address these challenges, this article proposes MMDFusion, a robust object detection framework tailored for unstructured road environments. In the feature extraction stage, an image-guided LiDAR feature enhancement (IGLFE) module is designed to augment the sparse bird’s-eye-view (BEV) features derived from point clouds. Subsequently, in the feature fusion stage, the framework introduces an adaptive deformable fusion (ADF) module, which leverages deformable convolutions to dynamically align and fuse BEV features from both images and LiDAR, thereby enhancing robustness against calibration errors. In addition, by constructing dense depth ground truth and designing a collaborative depth supervision loss (CoDSL), the framework fully exploits the geometric constraints within the common-view area of stereo cameras to refine detection performance. The framework was validated on UnStruct3D, a new real-world dataset collected from autonomous mining trucks. Experiments show that MMDFusion achieves a state-of-the-art (SOTA) 88.67 % mean average precision (mAP) and 80.51 % nuScenes detection score (NDS), outperforming the BEVFusion baseline by 8.17 % mAP. Significantly, MMDFusion demonstrates exceptional resilience to extrinsic calibration errors. Subject to simulated noise, it maintains high detection accuracy with a marginal mAP drop of 1.29 %, significantly outperforming the 2.4 % drop observed in the baseline, thereby validating its effectiveness for robust perception in rugged environments.