跳到主要导航 跳到搜索 跳到主要内容

Multimodal Reflective Synergy: Orchestrating Large-Small Model Collaboration for Knowledge-Based Visual Question Answering

  • Zhongfan Sun
  • , Kan Guo*
  • , Yongli Hu
  • , Jiapu Wang
  • , Daxin Tian
  • , Baocai Yin
  • *此作品的通讯作者
  • Faculty of Information Technology
  • Beijing University of Technology
  • School of Computer Science and Engineering
  • Nanjing University of Science and Technology

科研成果: 期刊稿件 › 文章 › 同行评审

摘要

Large Vision-Language Models (VLMs) have demonstrated exceptional performance in knowledge-based visual question answering, while small-scale VLMs, owing to their distinct training paradigms, retain complementary strengths that large VLMs lack. Consequently, collaboration between large and small VLMs has emerged as a promising research direction. However, existing collaborative frameworks predominantly rely on unidirectional prompting, which limits the depth of synergistic reasoning across model scales. To address this limitation, we propose SyMRR, a large-small model Synergy framework based on Multimodal Reflective Reasoning, which enables deep, reflective, and visually grounded collaboration between compact and large VLMs. Specifically, we introduce a multimodal reflective chain-of-thought mechanism that guides the large VLM to critically assess the small model's predictions, generating both textual rationales and visually grounded evidence. These reflective signals are then integrated back into the small model's features via a dual reflective interaction strategy, utilizing modality-specific cross-attention to refine the initial multimodal cues. The resulting interaction-enhanced representations guide the large VLM toward more reliable and interpretable answer generation. Notably, SyMRR adopts a parameter-efficient design by freezing both backbone models and optimizing only lightweight interaction modules. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SyMRR achieves state-of-the-art performance among collaborative frameworks, highlighting the effectiveness of deep reflective interaction for complex, knowledge-intensive multimodal reasoning.

源语言英语
期刊IEEE Transactions on Multimedia
DOI
出版状态已接受/待刊 - 2026

学术指纹

探究 'Multimodal Reflective Synergy: Orchestrating Large-Small Model Collaboration for Knowledge-Based Visual Question Answering' 的科研主题。它们共同构成独一无二的学术指纹。

引用此