STEM-DETR: multimodal remote sensing object detection based on improved spatial-temporal feature enhancement

Object detection remains a key task in remote sensing image processing. When applying the task to multimodal and sequential images, the problem of fusion of reciprocal information from multimodal sources, as well as the problem of temporal information extraction from sequential images, remain difficult yet rewarding. Current object detection networks mostly cannot process images that are both multimodal and sequential. The processed result also suffers from the variant object sizes found in remote sensing images, the unalignment and redundancy between modalities, and the difficulty in preserving long-range temporal information. Designed to tackle these problems, this research proposes a multimodal remote sensing object detection method based on improved spatial-temporal feature enhancement. The model proposed, called Spatial-Temporal Enhanced Multimodal DETR or STEM-DETR, supports object detection on RGB-T multimodal sequential images. We iterated on the typical end-to-end object detection pipeline of DETR by designing two unique modules, namely the RGB-T mixed attention merging module and the global spatial-temporal enhancement module. The RGB-T mixed attention merging module facilitates feature-level fusion between modalities, while the global spatial-temporal enhancement module builds on the concept of object queries by filtering high-confidence ones in the temporal sequence to enhance others. To validate the effectiveness of our method, thorough ablation study and comparison experiments are conducted. Within experiments, STEM-DETR achieved a maximum of 75.3 AP50 on our custom dataset, surpassing that of YOLOV++, SuperYOLO and TransVOD. These statistics are also supported by visual representations of the model's output. The results show that our method is both effective and adaptable.