Brimory: Bringing Humanoid Memory Into Trajectory Prediction Model for Autonomous Driving

Predicting the future trajectories of moving agents serves as a pivotal endeavor in advancing safe autonomous driving forward within the context of the evolving Internet of Things technology. Despite significant progress driven by deep learning methods, a gap in predictive capability remains when compared to experienced human drivers, particularly in scenarios that demand heightened perception and broader situational understanding. In this study, we propose Brimory, a novel trajectory prediction approach inspired by human driving, designed to equip autonomous vehicles (AVs) with humanoid memory systems. The core concept of Brimory is to emulate human driving processes by enabling the model to deeply comprehend traffic scenes, establish working memory, and iteratively accumulate to form driving experience. Brimory first introduces the working memory generator, which processes multiple interactions between the scene elements in the driving environment. It is noted that unlike traditional models confined to the time domain, Brimory also extracts underlying dependencies in the frequency domain. The long-term memory builder module is then developed to adaptively retrieve information from working memory and gradually accumulate driving experience with the help of the designed iterative updating mechanism. To mitigate the impact of irrelevant components on long-term memory effectiveness, we devise a co-modulation strategy that retains meaningful representations in a learnable manner. Finally, the combined humanoid memories are leveraged by the multimodal trajectory predictor to forecast future motions. Extensive experiments on benchmark datasets demonstrate the superiority, feasibility, and efficiency of Brimory. The results underscore the potential of humanoid memory frameworks in trajectory prediction, offering a promising path toward the realization of safer AVs.