Skip to main navigation Skip to search Skip to main content

Multimodal Reflective Synergy: Orchestrating Large-Small Model Collaboration for Knowledge-Based Visual Question Answering

  • Zhongfan Sun
  • , Kan Guo*
  • , Yongli Hu
  • , Jiapu Wang
  • , Daxin Tian
  • , Baocai Yin
  • *Corresponding author for this work
  • Faculty of Information Technology
  • Beijing University of Technology
  • School of Computer Science and Engineering
  • Nanjing University of Science and Technology

Research output: Contribution to journal › Article › peer-review

Abstract

Large Vision-Language Models (VLMs) have demonstrated exceptional performance in knowledge-based visual question answering, while small-scale VLMs, owing to their distinct training paradigms, retain complementary strengths that large VLMs lack. Consequently, collaboration between large and small VLMs has emerged as a promising research direction. However, existing collaborative frameworks predominantly rely on unidirectional prompting, which limits the depth of synergistic reasoning across model scales. To address this limitation, we propose SyMRR, a large-small model Synergy framework based on Multimodal Reflective Reasoning, which enables deep, reflective, and visually grounded collaboration between compact and large VLMs. Specifically, we introduce a multimodal reflective chain-of-thought mechanism that guides the large VLM to critically assess the small model's predictions, generating both textual rationales and visually grounded evidence. These reflective signals are then integrated back into the small model's features via a dual reflective interaction strategy, utilizing modality-specific cross-attention to refine the initial multimodal cues. The resulting interaction-enhanced representations guide the large VLM toward more reliable and interpretable answer generation. Notably, SyMRR adopts a parameter-efficient design by freezing both backbone models and optimizing only lightweight interaction modules. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SyMRR achieves state-of-the-art performance among collaborative frameworks, highlighting the effectiveness of deep reflective interaction for complex, knowledge-intensive multimodal reasoning.

Original languageEnglish
JournalIEEE Transactions on Multimedia
DOIs
StateAccepted/In press - 2026

Keywords

  • Dual Reflective Interaction
  • Knowledge-Based Visual Question Answering
  • Large-Small Model Synergy
  • Multimodal Reflective Chain-of-Thought

Fingerprint

Dive into the research topics of 'Multimodal Reflective Synergy: Orchestrating Large-Small Model Collaboration for Knowledge-Based Visual Question Answering'. Together they form a unique fingerprint.

Cite this