Deep Bayesian Active Learning for Learning to Rank: A Case Study in Answer Selection

Given a question and a set of candidate answers, answer selection is the task of identifying the best answer, which can be viewed as a kind of learning-to-rank tasks. Learning to rank arises in many information retrieval applications, where deep learning models can achieve inspiring results. Training a deep learning model often requires large scale annotated data that are expensive and time-consuming to obtain. Active learning presents a promising approach to this problem by selecting more informative training data to reduce the amount of labelling efforts required. Because traditional active learning methods cannot be directly used for deep learning, researchers have proposed multiple deep active learning methods. However, none of the previous research efforts on deep active learning algorithms presents a specific framework for learning-to-rank tasks. In this work, we introduce a novel deep active learning framework based on Deep Expected Loss Optimization (DELO) for the answer selection task. It adopts a data acquisition function based on model uncertainty with Bayesian deep learning and the expected loss optimization. Moreover, a two-step batch-mode procedure, combining DELO and other data acquisition strategies is proposed to further improve the performance of active learning. Experimental results verify the effectiveness of the proposed framework.