Document Based Question Answering Heuristics

The task of Document Based Question Answering(DBQA) is to answer questions by selecting one or multiple sentences from a set of unstructured documents as answers. Formally, given an utterance $Q$ and a document set $D$ , the document-based chatbot engine retrieves response $R$ based on the following three steps:

Response Retrieval, which retrieves response candidates $C$ from $D$ based on $Q$ :
$C = Retrieve(Q, D)$
Each S ∈ C is a sentence existing in D.
Response Ranking, which ranks all response candidates in C and selects the most possible response candidate as $\hat S$ :
$\hat S = argmax_{S∈C}Rank(S, Q)$
Response Triggering, which decides whether it is confident enough to response $Q$ using $\hat{S}$ :
$I = T rigger(\hat S, Q)$
where $I$ is a binary value. When $I$ equals to true, let the response $R = \hat S$ and output $R$ ; otherwise, output nothing.

Response Retrieval

Search Engine 101

Index

The purpose of storing an index is to optimize speed and performance in finding relevant documents for a search query. Without an index, the search engine would scan every document in the corpus, which would require considerable time and computing power.
The most commonly used indexing method is Inverted Indexing. An inverted index consists of a list of all the unique words that appear in any document, and for each word, we attach a list of the documents in which it appears.

inverted index

Search

Given user query $q$ , we first tokenize $q$ , such that $q = (w_{1,q}, w_{2, q}, ..., w_{n, q})$ , for each word $w_{k, q}$ , we can read their corresponding inverted index and get a list of documents $d_{k}$ contain the word $w_{k,q}$ . The final candidate document set is $(d_{1}, d_{2}, ..., d_{n})$ .
We could have more constrains on the candidate document set, for example, a binary boolean operator(should, must) could be specified on each word to indicate this world must be contained by each of the final candidate documents or not. The operator and the word constitute a boolean clause, clause can be integrated into a more complex clause.

Ranking

The candidate document set could be large as 10,000, we need a facility to score these documents to find out the most relevance ones.

Vector Space Model and TF-IDF statistics

Say we have a vocabulary represents the uniq terms in the corpus(all documents in search engine). Each query and document can be represented as a vocab-size vector, such that we can calculate the similarity score between $q$ and a candidate document $\hat d$
$cosine\_similarity(q, \hat d) = \dfrac{q \cdot \hat d}{||q||\cdot||\hat d||}$

Each dimension $i$ of the vector is the weight of $i$ -th term in the vocabulary. The most simple weighting strategy is Boolean weighting, if term $i$ appear in document, so set $\hat d[i] = 1$ . Documents share more common terms with th query have higher score.

Boolean weighting has its own weakness. Terms in query are not identical important. TF-IDF(Term Frequency–Inverse Document Frequency) weighting believes that the more some term $t$ appears in the document, and the less $t$ is contained by other document, the more important $t$ is.
$\begin{align} TF(t, d) &= 0.5 + 0.5 * \dfrac{f_{t, d}}{max\{f'_{t', d}:t' \in d\}}\\ IDF(t) &= log \dfrac{|corpus|}{|d \in D, t \in d|} \\ TF-IDF(t, d) &= TF(t, d) * IDF(t) \end{align}$

Response Ranking

Given a user utterance Q and a response candidate $S$ , the ranking function $Rank(S, Q)$ is designed as an ensemble of individual matching features:
$Rank(S, Q) = \sum_k λ_k · h_k(S, Q)$
where $h_k(·)$ denotes the $k$ -th feature function, $λ_k$ denotes $h_k(·)$ ’s corresponding weight.

Framework

We need to design multi-level(Paraphrase, Causality, Discourse Relationship, etc.) semantic and syntactic features to rank $<Q, S>$ pair precisely, which is often laborious. Deep learning approaches have gained a lot of attention from the research community and industry for their ability to automatically learn optimal feature representation for a given task, while claiming state-of-the-art performance in many tasks.

Deep QA

Data

正例

Q:吃了罗红霉素分散片可以怀孕吗？
A: 怀孕三十八周可以口服罗红霉素分散片的，但在怀孕期间不能盲目用药，盲目用药是会影响胎儿发育的，建议你在医生的指导下用药为宜。
Q:拍胸片后多久可以怀孕？
A:因此 , 妇女平时应尽量减少 x光的照射机会 , 怀孕前 4 周内必须禁忌照射 X光
Q:剖腹产后多久可以做瑜伽？
A:那剖腹产后多久可以做瑜伽？

正例噪声很大， QA库里面的答案不一定是全部跟问题相关。

负例

Q：孕前优生优育如何检查？
A: 宝宝补锌吃什么好？
Q: 哺乳期用什么护肤品好？
A: 头围是指绕胎头一周的最大长度。

负例太负，跟实际的数据分布不一致。实际数据是一篇篇的文章，这些文章里面的句子都是相关的。我们实际需要的是一些看起来相近的负例。比如问题是问因果，答案是说为什么。

Demo

DEMO

Cases:

怀孕可以锻炼吗（G）
宝宝吃手指（G）
宝宝吃手指怎么办（G）
宝宝发烧身上出红疹子怎么办 (N)
宝宝晚上不睡觉 (N)
宝宝感冒怎么办（B）