AdaSeViLA: Adaptive Dynamic Temporal Window and Object-Aware Frame Selection for Video Question Answering

Video question answering remains a challenging task that requires a sophisticated understanding of both visual content and temporal dynamics across video sequences. Current approaches typically rely on fixed temporal processing strategies and uniform frame-selection mechanisms, which fail to adapt to the diverse requirements of different question types and may overlook critical visual information. We propose AdaSeViLA, an adaptive framework that enhances video understanding through two key innovations: Adaptive Temporal Window Selection (ATWS) that dynamically adjusts the number of processed frames (3–12 frames) based on question-type classification, and Object-importance-Aware Frame Selection (OAFS) that combines global relevance with local visual saliency for enhanced frame identification. Our approach intelligently allocates computational resources based on question complexity while maintaining high accuracy through improved frame-selection mechanisms. Extensive experiments on three challenging VideoQA benchmarks demonstrate that AdaSeViLA achieves superior performance: 87.4% accuracy on MM-AU (+2.7% over SeViLA), 73.6% on NExT-QA (+0.4% improvement), and 61.6% on STAR (+0.6% gain), while providing up to 4× computational speedup for short-term tasks. These results validate the effectiveness of adaptive temporal processing and object-aware selection in advancing video question answering capabilities.