CLIP-HOI: Vision-Language Semantic Alignment for Human-Object Interaction Detection

We propose CLIP-HOI, a vision-language enhanced framework for end-to-end Human-Object Interaction (HOI) detection. While transformer-based HOI methods have shown strong performance, they still struggle with rare and complex interactions that demand deeper semantic understanding beyond visual cues. To address this limitation, we integrate Contrastive Language-Image Pre-Training (CLIP) into the HoiTransformer, leveraging its powerful vision-language alignment capability. Specifically, CLIP-HOI adopts a dual-stream architecture: the original HoiTransformer branch encodes instance-level visual and spatial features, while a parallel CLIP branch extracts global, context-aware semantic representations from image-text alignment space. A cross-attention fusion module adaptively refines interaction features by infusing CLIP-guided semantic cues into visual reasoning. Extensive experiments on the HICO-DET benchmark demonstrate that CLIP-HOI consistently out-performs the baseline HoiTransformer. Our approach shows that integrating vision-language knowledge substantially improves the comprehension of human-object interactions within an end-to-end framework.