A multimodal large language model and knowledge graph-driven approach for intelligent product service

As industrial equipment grows in scale and complexity, product service systems increasingly demand unified perception and highly reliable decision support. In practical engineering environments, the coexistence of heterogeneous data—such as visual, acoustic, time-series, and textual information—poses significant challenges for cross-modal reasoning. Although multimodal large language models (MLLMs) offer strong perception and generation capabilities, their application in industrial scenarios is severely limited by a lack of domain-specific knowledge and uncontrollable reasoning, which often leads to hallucinations. To address these issues, this study proposes an intelligent product service approach driven by MLLMs and a knowledge graph. First, a top-down domain ontology is constructed to explicitly structure industrial knowledge. Building on this schema, a graph-driven late fusion strategy is proposed to align visual, acoustic, and time-series representations within a unified semantic space. Furthermore, a multimodal knowledge graph-enhanced retrieval-augmented generation (MM-KG-RAG) mechanism is developed. By utilizing structured entity relations to mathematically bound the generative search space of the LLM, this mechanism effectively mitigates hallucinations and ensures interpretable maintenance decisions. The proposed approach is validated through an end-to-end fault diagnosis and decision task using a complex railway equipment case study. Quantitative results demonstrate that the MM-KG-RAG framework significantly outperforms both retrieval-free and text-only retrieval-augmented baselines in fault identification accuracy and procedure matching, confirming its engineering feasibility and high reliability for complex product service systems.