Multi-goal forced convection heat transfer control for cylinder under online and offline scenarios via deep reinforcement learning

Deep reinforcement learning (DRL)-based forced convection heat transfer control encounters challenges when facing complex control objectives and scenarios. This study focuses on a two-dimensional cylindrical heat source within a narrow cavity. First, DRL is applied to perform the energy-efficient cooling task with an energy penalty term incorporated into the reward. Results show that the continuous DRL algorithm achieves the best performance, with a comprehensive gain exceeding the baseline by 25.1%. To handle temperature control tasks with variable goals, a goal-oriented reinforcement learning (GoRL) framework is established. Results show that control accuracy decreases as the goal space expands, however, it can be improved by employing hindsight experience replay with a future method and designing the observation space such that the reward becomes a function of the observation. The GoRL framework achieves precise variable-goal temperature control across a broad continuous goal space of 20 K with a single training process, yielding a temperature variance of 0.1071 while consuming only 5% computational resources required by traditional DRL frameworks. Furthermore, considering scenarios where online DRL learning and interaction are restricted due to safety or cost constraints, a continuous goal-oriented conservative Q -learning (GoCQL) framework is proposed, aiming to rapidly accomplish variable-goal temperature control solely based on the randomized offline dataset. Results demonstrate that GoCQL agent exhibits excellent generalization ability, achieving a temperature variance of 0.1572. Moreover, the time consumption of offline learning is less than 1% of that of online learning under a complete training session, enabling fast acquisition of control strategies in scenarios where online interaction is impractical.