用于病理图像分类的混合CLIP专家模型

Objective Vision-language models (VLMs), such as contrastive language image pretraining (CLIP), achieve cross-modal image-text alignment by employing large-scale contrastive pretraining and enable prompt-driven zero-shot classification: thus they overcome the limitations of closed category spaces in traditional supervised classification models. However, their generalization performance is constrained by distribution shifts between pretraining and downstream tasks, as well as by inherent knowledge boundaries, particularly in specialized domains with scarce labeled data (e. g. pathology). Various CLIP variants and extensions have emerged; these variants exhibit complementary but inconsistent downstream performance which is attributed to their distinct model architectures and pretraining datasets. Although various parameter-efficient fine-tuning techniques have been proposed for adapting individual CLIP models to downstream tasks. existing studies have mainly focused on optimizing single pretrained models: thus they fail to effectively exploit the complementary advantages of heterogeneous models. This study aimed to exploit the complementary strengths of heterogeneous pretrained CLIP models for pathology image classification, Specifically, we conduct a systematic comparison among ensemble strategies at both the model output and middle-layer feature levels; additionally, we propose a novel feature-level ensemble framework termed Mix-of-CLIP-Experts (MoCE), [Methods] Initially, we evaluated multiple pretrained CLIP models for pathology image classification tasks under the zero-shot setting to demonstrate their complementary strengths and weaknesses across different datasets. Next, we designed and evaluated various ensemble strategies, At the output level, we investigated simple averaging and a weighted combination of predictions based on model confidence scores or learned gating networks. At the feature level, we applied the proposed MoCE method to the fusion of image features obtained from heterogeneous CLIP models. The main challenge in feature-level CLIP model ensembling is the misalignment of embedding across incompatible cross-modal spaces of different CLIP models. To address this challenge, we combined MOCE adapter-based fine-tuning with the mix-of-experts (MoE) framework. Using this approach, the pretrained models can be simultaneously adapted to the downstream pathology task and their image (and aligned text) features can be projected onto a unified em bedding space. A learned router was employed to dynamically weight and aggregate these aligned image features to generate a fused representation: this representation was subsequently compared against text prompts, which were encoded using a single text encoder to perform the final classification. This process reduces computational redundancy by eliminating the need for multiple text encoders; additionally, it improves downstream performance via adapter-based fine-tuning and MoE routing to fully exploit model complementarity. Results] We comprehensively evaluated the proposed framework and baseline ensemble strategies using multiple public pathology datasets under various few-shot settings. The results showed that MOCE consistently outperformed single-model fine-tuning baselines and output-level ensemble methods, demonstrating the advantages of feature-level model ensembling via adapter-hased alignment and dynamic routing. Detailed ablation studies validated the effectiveness of the proposed MoCE framework and its specific components. [Conclusions To the best of our knowledge. MoCE is the first feature-level ensembling framework for heterogeneous pretrained CLIP models. By combining adapter-based cross-model feature alignment with MoCE routing, effective fusion of diverse CLIP backbones can be achieved, additionally, the pathology image classification under limited-data regimes can be substantially improved. The parameter-efficient cross-model alignment and dynamic expert fusion mechanisms are broadly applicable beyond pathology to other specialized domains. In future work, we will focus on scaling to larger model pools, applying model distillation for inference efficiency, and extending the framework to dense prediction tasks such as object detection and image segmentation.