IDR-SQE: Combining semantic and statistical relations to refine documents

As the dominant tool to access information online, Web search engines still struggle to improve the quality of search results using various methods. Among these methods, query expansion and document refinement are particularly useful and effective. Query expansion aims at generating an expanded query by adding new terms and phrases, but document refinement keeps Web queries invariant and ceaselessly refines every document to judge whether it is relevant. So document refinement is considered to be much closer to human thinking habit of information seeking than query expansion. However, existing document refinement approaches only consider the semantic expansion of Web queries, thus there might be some key terms ignored, although they play an important role in discovering information needs of users. This paper presents an improved document refinement approach, which takes into account not only semantic similarity between terms, but statistical correlation between terms as well. The combination of these two factors creates an underlying model for robust and accurate similarity calculation. Experimental results show that our proposed approach can lead to much improved Web search accuracy over traditional methods.