Close-range Air Combat Autonomous Maneuver Decision-Making of UAV Based on TD3-Transformer

Reinforcement Learning (RL) is a promising approach for autonomous air combat decision-making of Unmanned Aerial Vehicles (UAVs). However, extracting complete and effective features from the complex air combat environment remains a significant challenge for RL agents. The air combat trajectory inherently contains deep-level features, such as motion dynamics and implicit combat intentions. To exploit this wealth of historical information, we propose a Twin Delayed Deep Deterministic Policy Gradient with Transformer (TD3-Transformer) algorithm. Unlike traditional RL methods that rely solely on the current state, our approach employs a Transformer-based actor to process historical state-action sequences, thereby capturing deep dynamic features. To preserve the crucial order of events within these trajectories, a dedicated timestep embedding mechanism is introduced, enabling superior situational awareness. Numerical experiments demonstrate that TD3-Transformer significantly outperforms baseline models in combat effectiveness. Crucially, the framework exhibits strong transfer learning capabilities for rapid adaptation to new adversaries and superior robustness against observational disturbances, critical attributes for successful simulation-to-reality transitions. Visualized self-attention weights provide interpretable insights into the agent’s decision-making logic, enhancing its trustworthiness for critical applications. The proposed framework presents a robust and generalizable approach for sequential decision-making in complex aerospace systems.