HiBERT: Detecting the illogical patterns with hierarchical BERT for multi-turn dialogue reasoning

Dialogue reasoning is a new task beyond the traditional dialogue system, because it requires recognizing not only semantic relevance but also logical consistency between the candidate response and the dialogue history. Like “all happy families are happy alike, all unhappy families are unhappy in their own way”, various illogical patterns exist in the data. For example, some candidate responses use many similar words but with contradicted meanings with history; while some candidates may employ totally different words but convey consistent meanings. Therefore, an ideal dialogue reasoning model should gather clues from both coarse-grained utterance-level and fine-grained word-level to determine the logical relation between candidates and the dialogue history. However, traditional models mainly rely on the widely used BERT to read all the history and candidates word by word but ignore the utterance-level signals, which cannot well capture various illogical patterns in this task. To tackle this problem, we propose a novel Hierarchical BERT (HiBERT) to recognize both utterance-level and word-level illogical patterns in this paper. Specifically, BERT is firstly utilized to encode the dialogue history and each candidate response as the contextualized representation. Secondly, hierarchical reasoning architecture is conducted with this contextualized representation to obtain the word-level and the utterance-level attention distributions, respectively. In detail, we utilize the word-grained attention mechanism to obtain the word-level representation, and propose two different types of attention function, i.e, hard attention and soft attention, to obtain the utterance-grained representation. Finally, we fuse both the word-grained representation and the utterance-grained representation to calculate the logical ranking scores for the given candidate. Experimental results on two public dialogue datasets show that our model obtains higher ranking measures than the widely used BERT model, validating the effectiveness of hierarchical reading of HiBERT. Further analysis on the impact of context length and attention weights shows that the HiBERT actually has the ability to recognize different illogical patterns.