Real-Time optimal entry guidance based on deep reinforcement learning

Entry guidance system plays an important role in ensuring a successful atmospheric entry of hypersonic vehicles. The entry guidance problems are essentially optimal control problems and are traditionally solved by direct methods or indirect methods. However, both two kinds of methods suffer the shortcomings of high computational burden and poor real-time property for on-board applications. In this study, an intelligent entry guidance approach is proposed to achieve real-time optimal control for entry flight based on Deep Reinforcement Learning (DRL). This study focuses on the following three contributions. First, the entry problem in longitudinal channel is formulated as a Markov decision process based on an improved bank corridor, and a constraint management technique is developed to help ensure the satisfaction of path constraints. Second, the Deep Deterministic Policy Gradient (DDPG) algorithm is employed to learn an optimal control policy for bank decision-making and optimize a long-term reward by interacting with flight dynamics. In order to facilitate the learning effect, relevant network sizes and reward function design are optimized. Third, a neural network is trained in a supervised manner based on the crossranges collected during the DDPG training, and help determine the bank reversal according to a lateral crossrange decrement strategy. On these bases, an intelligent, multi-constrained entry guidance algorithm is developed to achieve real-time and optimal entry flight control. Since the entry problem on longer needs to be solved on-board, the proposed DRL-based algorithm successful overcomes the long-standing challenge of the existing numerical optimal control methods in real-time performance. Simulations are conducted through comparing with the state-of-the-art numerical optimal control methods, and the results demonstrate the optimality and real-time performance of the obtained policy and substantiate the effectiveness of the proposed entry guidance algorithm.