A decision tree based quality control framework for multi-phase tasks in crowdsourcing

In crowdsourcing, there exists an important category of tasks that comprise an ordered sequence of subtasks, which we refer to as Multi-phase Tasks (MPTs) - e.g. travel planning, translation and micro-writing. Existing result inference methods are ineffective for processing MPTs. The constrained relationships among phase-level subtasks of MPT cannot be ignored for two reasons. First, it is ineffective to conduct a MPT without phase-processing, e.g. for travel planning, recommending a complete route of travel planning, and using existing methods to infer the final result generated by an individual worker can hardly meet various requirements due to the lack of flexibility. Second, although a MPT consists of a set of phaselevel subtasks, it is unsuitable to simply split a MPT into subtasks and use top-k methods to recommend final results; because this will not only increase costs but also lose the constrained relationships among the phases. Thus it calls for a new approach to handle MPTs. This research first introduces the concept of MPT to identify these special tasks. Second, a decision tree based framework is provided to control task generation and final result combination in the crowdsourcing cooperative workflow for MPTs. Third, a probabilistic graphical model is proposed to characterize the subtasks of each MPT phase and a maximum likelihood based method is designed for result inference. Finally, extensive experiments were conducted based on real-world travel planning tasks and experimental results demonstrate the superiority of this approach in comparison with the state-of-the-art methods.