Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection

Background and objectives: With the development of deep learning techniques, the computer-assisted pathology diagnosis plays a crucial role in clinical diagnosis. An important task within this field is report generation, which provides doctors with text descriptions of whole slide images (WSIs). Report generation from WSIs presents significant challenges due to the structural complexity and pathological diversity of tissues, as well as the large size and high information density of WSIs. The objective of this study is to design a histopathology report generation method that can efficiently generate reports from WSIs and is suitable for clinical practice. Methods: In this paper, we propose a novel approach for generating pathology reports from WSIs, leveraging knowledge retrieval and multi-level regional feature selection. To deal with the uneven distribution of pathological information in WSIs, we introduce a multi-level regional feature encoding network and a feature selection module that extracts multi-level region representations and filters out region features irrelevant to the diagnosis, enabling more efficient report generation. Moreover, we design a knowledge retrieval module to improve the report generation performance that can leverage the diagnostic information from historical cases. Additionally, we propose an out-of-domain application mode based on large language model (LLM). The use of LLM enhances the scalability of the generation model and improves its adaptability to data from different sources. Results: The proposed method is evaluated on a public datasets and one in-house dataset. On the public GastricADC (991 WSIs), our method outperforms state-of-the-art text generation methods and achieved 0.568 and 0.345 on metric Rouge-L and Bleu-4, respectively. On the in-house Gastric-3300 (3309 WSIs), our method achieved significantly better performance with Rouge-L of 0.690, which surpassed the second-best state-of-the-art method Wcap 6.3%. Conclusions: We present an advanced method for pathology report generation from WSIs, addressing the key challenges associated with the large size and complex pathological structures of these images. In particular, the multi-level regional feature selection module effectively captures diagnostically significant regions of varying sizes. The knowledge retrieval-based decoder leverages historical diagnostic data to enhance report accuracy. Our method not only improves the informativeness and relevance of the generated pathology reports but also outperforms the state-of-the-art techniques.