Dynamic Edge Caching via Online Meta-RL

The content request patterns perceived by edge devices are becoming highly dynamic, especially for emerging short video platforms compared to traditional video platforms. This calls for caching policies that can continuously adapt to dynamic environments, challenging previously popular reinforcement learning (RL)-based policies. A straightforward solution, i.e., repeatedly restarting and training RL agents, would fail to converge timely while meeting the observed adaptation process. Offering transferable knowledge is considered a possible method to speed up the adaptation process. Unfortunately, it fails to outperform the RL-based approach as an alternative solution in these scenarios. To alleviate this drawback, we 1) design a sequential-pair meta-learning for edge caching that captures the meta-knowledge of dynamic changes from sequential-pair-wise intervals, which are segmentations from the whole dynamic episode, and 2) develop an online meta-RL-based solution called Online Meta Actor-Critic (OMAC), which updates the meta-knowledge in an online manner. To evaluate the proposed framework, we conduct trace-driven experiments to demonstrate the effectiveness of our design: it improves the average cache hit rate by up to 37.4% (normalized) compared with other baselines.