Cross-modal collaboration with side temporal adaptation for open-vocabulary temporal action localization

Temporal Action Localization (TAL) is a crucial task in video understanding that aims to detect and identify human action segments within untrimmed videos. Although existing TAL methods have demonstrated strong performance on various datasets, they still face challenges in handling the complexity and diversity of human actions in real-world scenarios, particularly failing to handle action categories absent from training data. To address this challenge, we propose a cross-modal collaboration framework for open-vocabulary TAL to achieve robust action localization in both seen and unseen action categories. It leverages Vision-Language Models (VLMs) to align action videos and annotation texts in a unified vocabulary space, enabling generalized action understanding that transcends predefined category constraints. In this framework, we propose a Side Temporal Adaptation (STA) method that captures temporal relationships in videos while mitigating the domain gap between static image-pretrained VLMs and dynamic human action representations. The STA incorporates VLM features in a unidirectional manner, facilitating temporal association learning without disrupting the original image-text representation space. During inference, we propose a classification ensemble strategy that integrates in-vocabulary prediction from the STA with out-of-vocabulary prediction from the VLM. This approach improves classification accuracy for seen categories while maintaining robust generalization for unseen categories. We evaluate our method on the ActivityNet-1.3 and THUMOS’14 datasets. Experimental results demonstrate that our method achieves high-quality performance and generalization capability on the open-vocabulary TAL task.