DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models

Although knowledge distillation (KD) is an effec-tive approach to improve the performance of a smaller LLM (i.e., the student model) by trans-ferring knowledge from a large LLM (i.e., the teacher model), it still suffers from high training cost. Existing LLM distillation methods ignore the difficulty difference among different samples, making the distillation of easy samples unnec-essary. This leads to high distillation cost. In this paper, we propose difficulty-aware knowl-edge distillation (DA-KD) framework for efficient knowledge distillation, in which we dynamically adjust the distillation dataset based on the diffi-culty of samples. We further observe existing KD loss cannot perform well when most of samples are difficult in the distillation dataset because of unstable optimization and the neglect of hard sam-ples. Therefore, we also propose a new KD loss called bidirectional discrepancy loss (BDL) for effective KD. Extensive experiments demonstrate that our DA-KD framework is effective and ef-ficient. Without bells and whistles, DA-KD can outperform existing state-of-the-art KD methods by 2% with half training cost and even surpass the teacher model with 4.7× compression.