Bio-inspired searching and soaring strategies in drifting thermal updrafts for UAV based on deep reinforcement learning

Improving the airborne endurance of small Unmanned Aerial Vehicles (UAVs) is critical to enhancing their operational capabilities and mission performance. Tropical migratory birds, which share similar wing loading and size characteristics with small UAVs, can skillfully exploit thermal updrafts and trade wind zones to achieve long-distance soaring flights without landing. As a result, mimicking the flight strategies of these birds has become a key research focus in recent years. The implementation of such flight strategies relies on the control algorithms of UAVs. However, traditional algorithms for designing UAV flight strategies primarily depend on deterministic decision-making methods, which struggle to effectively cope with complex environmental variables and dynamic uncertainties. In response to this challenge, this study introduces a comprehensive environmental model that incorporates stochastic thermal updrafts and crosswind fields, along with a bio-inspired UAV dynamics model. Based on this foundation, a deep reinforcement learning framework is employed and an innovative method for designing a modular reward function is proposed, by introducing different sub-rewards, agent can effectively balancing exploration and energy acquisition during flight. This approach enables the training of holistic flight strategy in complex environments. The strategy allows UAV to achieve sustained flight in trade wind zones by utilizing energy from thermal updrafts. Through the analysis of the strategy, the mechanisms of energy acquisition in drifting thermal updrafts are further revealed. Simulation tests validate the adaptability of the proposed strategies to various wind field parameters, including crosswind speeds and the duration and frequency of thermal updrafts, demonstrating the effectiveness and broad application potential of this method.