An Integrated Framework of Lateral and Longitudinal Behavior Decision-Making for Autonomous Driving Using Reinforcement Learning

Lateral lane-changing and longitudinal car-following behavior decision-making is crucial for the implementation of autonomous driving (AD) in complex and dynamic traffic environment. Reinforcement learning (RL) has been increasingly applied to these behaviors due to its powerful environmental adaptability. However, typical RL-based research focused solely on one directional decision-making or output discrete actions in both lateral and longitudinal directions, which is not sufficiently flexible for AD. In view of the close interaction between lane-changing and car-following tasks, this study proposed an integrated framework to realize coupled decision-making using RL. Specifically, double deep Q-network (DDQN) was used to make the lateral lane change command decision-making, and twin delay deep deterministic policy gradient (TD3) used to make longitudinal car-following acceleration decision-making. The integration of DDQN and TD3 was deployed to tackle the problem of discrete-continuous hybrid action space in RL. The lateral decision-making was integrated into the actor-critic network for longitudinal decision-making to generate accurate acceleration in lane-changing or lane-keeping conditions. A series of simulation experiments were conducted in typical two-lane straight driving scenarios, validating the feasibility and applicability of the proposed integrated framework. The results reveal that the RL-based coupled decision-making helps enhance the implementation of AD.