Enabling Semi-Structured Knowledge Access via a Question-Answering Module in Task-oriented Dialogue Systems

Users of task-oriented dialogue systems are often limited to ‘in-schema queries’, i.e., questions constrained by a predefined database structure. Providing access to additional semi- or unstructured knowledge could enable users to enter a wider range of queries answerable by the system. To this end, we have integrated a Question-Answering (QA)-module in an interactive restaurant search system and evaluated its impact using a crowd-sourced user evaluation. The QA-module includes knowledge selection and response generation components, both driven by fine-tuned GPT-2 language models, and a method to prevent responses unrelated to a user question (‘off-topic responses’). The results show that systems with QA-module are significantly preferred over the baseline without QA-module. Moreover, while the off-topic response prevention method was correctly triggered in 98.1% of questions not covered in the knowledge base, users showed more preference to the system that can retrieve information irrespective of whether it is relevant or not.