Optimizing same-day delivery with vehicles and drones: A hierarchical deep reinforcement learning approach

The advent of same-day delivery services has achieved immense popularity, driven by escalating customer expectations on fast shipping and the need for market competitiveness. To optimize such services, the use of heterogeneous fleets with vehicles and drones has proven effective in reducing the resource requirements needed for delivery. This paper focuses on investigating the same-day delivery dispatching and routing problem with a fleet of multiple vehicles and drones. In this problem, stochastic and dynamic requests, coupled with their stringent time constraints, require dispatchers to make real-time decisions about optimally assigning vehicles and drones, ensuring both efficiency and effectiveness in delivery operations while taking into account the routing. To tackle this complex problem, we model it with a route-based Markov decision process and develop a novel hierarchical decision approach based on deep reinforcement learning (HDDRL). The first level of the hierarchy is tasked with determining the departure times of vehicles, balancing the trade-offs between the delivery frequency and efficiency. The second level of the hierarchy is dedicated to determining the most suitable delivery mode for each request, whether by vehicles or drones. The third level is responsible for planning routes for vehicles and drones, thereby enhancing route efficiency. These three levels in the hierarchical framework collaborate to solve the problem in a synchronized manner, with the objective of maximizing the service requests within a day. Empirical results from computational experiments highlight the superiority of the HDDRL over benchmark methods, demonstrating not only its enhanced efficacy but also its robust generalization across diverse data distributions and fleet sizes. This underscores the HDDRL's potential as a powerful tool for enhancing operational efficiency in same-day delivery services.