Knowledge Distilled Group Prompts Learning for HOI Detection with Large Vision-Language Models

Large vision-language models (VLMs) have significantly advanced human-object interaction (HOI) detection. However, existing VLM-based HOI detectors primarily rely on simple text prompt paradigms, specifically in relation to knowledge hallucination, with limited exploration of the intrinsic attributes or extrinsic context. In this paper, we propose a knowledge distilled group prompts learning method for HOI detection, termed GPL-HOI, which transfer knowledge from vision-language models via group prompts and knowledge distillation. Specifically, we design visual-textual group prompts by combining scene-aware, region-aware, and pose-aware prompt to guide knowledge transfer from VLMs. Additionally, we introduce a cross-modal group distillation module,which aligns the semantic features of both the vision and text models via KL divergence, encouraging the visual encoder to generate similar probability distributions to the text encoder through the learnable prompts. Extensive experiments demonstrate that our method surpasses state-of-the-art approaches in both conventional and zero-shot settings, achieving improvements of +2.04 mAP and +1.84 mAP on HICO-DET, respectively. Code will be available at https://github.com/hxqstree/GPL-HOI.