Enabling Semi-Structured Knowledge Access via a Question-Answering Module in Task-oriented Dialogue Systems
Authors: 
Norbert Braunschweiler, Rama Sanand Doddipatla, Simon Keizer, Svetlana StoyanchevAuthors Info & Claims
Article No.: 36, Pages 1 - 11
Abstract
Users of task-oriented dialogue systems are often limited to ‘in-schema queries’, i.e., questions constrained by a predefined database structure. Providing access to additional semi- or unstructured knowledge could enable users to enter a wider range of queries answerable by the system. To this end, we have integrated a Question-Answering (QA)-module in an interactive restaurant search system and evaluated its impact using a crowd-sourced user evaluation. The QA-module includes knowledge selection and response generation components, both driven by fine-tuned GPT-2 language models, and a method to prevent responses unrelated to a user question (‘off-topic responses’). The results show that systems with QA-module are significantly preferred over the baseline without QA-module. Moreover, while the off-topic response prevention method was correctly triggered in 98.1% of questions not covered in the knowledge base, users showed more preference to the system that can retrieve information irrespective of whether it is relevant or not.
References
[1]
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 5016–5026. https://doi.org/10.18653/v1/D18-1547
[2]
Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, and William Yang Wang. 2022. KETOD: Knowledge-Enriched Task-Oriented Dialogue. https://doi.org/10.48550/ARXIV.2205.05589
[3]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423
[4]
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-Powered Conversational Agents. In International Conference on Learning Representations. https://openreview.net/forum?id=r1l73iRqKm
[5]
Michel Galley, Chris Brockett, Xiang Gao, Jianfeng Gao, and B. Dolan. 2019. Grounded Response Generation Task at DSTC7. In Proceedings of the AAAI-19 Workshop on Dialog System Technology Challenges.
[6]
Silin Gao, Ryuichi Takanobu, Wei Peng, Qun Liu, and Minlie Huang. 2021. HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online, 1591–1602. https://doi.org/10.18653/v1/2021.findings-acl.139
[7]
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019. 1891–1895. https://doi.org/10.21437/Interspeech.2019-3079
[8]
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The Second Dialog State Tracking Challenge. In Proceedings of the 15th Annual SIGdial Meeting on Discourse and Dialogue. 263–272.
[9]
Seokhwan Kim, Mihail Eric, Karthik Gopalakrishnan, Behnam Hedayatnia, Yang Liu, and Dilek Hakkani-Tur. 2020. Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, 1st virtual meeting, 278–289.
[10]
Seokhwan Kim, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, and Dilek Hakkani-Tur. 2021. Beyond domain APIs: Task-oriented conversational modeling with unstructured knowledge access track in DSTC9. arXiv preprint arXiv:2101.09276 (2021).
[11]
Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papangelis, Karthik Gopalakrishnan, Behnam Hedayatnia, and Dilek Hakkani-Tur. 2021. "How robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations. arxiv:2109.13489 [cs.CL]
[12]
Longxuan Ma, Mingda Li, Wei-Nan Zhang, Jiapeng Li, and Ting Liu. 2022. Unstructured Text Enhanced Open-Domain Dialogue System: A Systematic Survey. ACM Trans. Inf. Syst. 40, 1 (2022), 9:1–9:44. https://doi.org/10.1145/3464377
[13]
Jan Nehring, Nils Feldhus, Harleen Kaur, and Akhyar Ahmed. 2021. Combining Open Domain Question Answering with a Task-Oriented Dialog System. In Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021). Association for Computational Linguistics, Online, 38–45. https://doi.org/10.18653/v1/2021.dialdoc-1.5
[14]
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. ArXiv abs/1901.04085 (2019).
[15]
OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt/
[16]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. https://doi.org/10.48550/ARXIV.2203.02155
[17]
Suraj Pandey, Svetlana Stoyanchev, and Rama Doddipatla. 2021. Towards Handling Unconstrained User Preferences in Dialogue. In 12th International Workshop on Spoken Dialog System Technology (IWSDS). https://arxiv.org/abs/2109.08650
[18]
Alexandros Papangelis, Mahdi Namazifar, Chandra Khatri, Yi-Chia Wang, Piero Molino, and Gokhan Tur. 2020. Plato Dialogue System: A Flexible Conversational AI Research Platform. arxiv:2001.06463 [cs.HC]
[19]
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
[20]
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34(05). 8689–8696. https://doi.org/10.1609/aaai.v34i05.6394
[21]
Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. FAQ retrieval using query-question similarity and BERT-based query-answer relevance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1113–1116.
[22]
Svetlana Stoyanchev, Simon Keizer, and Rama Doddipatla. 2021. Action State Update Approach to Dialogue Management. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 7398–7402.
[23]
Svetlana Stoyanchev, Suraj Pandey, Simon Keizer, Norbert Braunschweiler, and Rama Doddipatla. 2022. Combining Structured and Unstructured Knowledge in an Interactive Search Dialogue System. In SIGdial 2022 Conference. 531–540.
[24]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems abs/1706.03762 (2017), 6000–6010.
[25]
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 5881–5885.
[26]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6
[27]
Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017. Building Task-Oriented Dialogue Systems for Online Shopping. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence(AAAI’17). AAAI Press, San Francisco, California, USA, 4618–4625.
[28]
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense Knowledge Aware Conversation Generation with Graph Attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18. International Joint Conferences on Artificial Intelligence Organization, 4623–4629. https://doi.org/10.24963/ijcai.2018/643
[29]
Kangyan Zhou, Shrimai Prabhumoye, and Alan W. Black. 2018. A Dataset for Document Grounded Conversations. CoRR abs/1809.07358 (2018). arXiv:1809.07358http://arxiv.org/abs/1809.07358
Index Terms
- Enabling Semi-Structured Knowledge Access via a Question-Answering Module in Task-oriented Dialogue Systems
Recommendations
Learning to Extract Conditional Knowledge for Question Answering using Dialogue
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge ManagementKnowledge based question answering (KBQA) has attracted much attention from both academia and industry in the field of Artificial Intelligence. However, many existing knowledge bases (KBs) are built by static triples. It is hard to answer user questions ...
From vocal to multimodal dialogue management
ICMI '06: Proceedings of the 8th international conference on Multimodal interfacesMultimodal, speech-enabled systems pose different research problems when compared to unimodal, voice-only dialogue systems. One of the important issues is the question of how a multimodal interface should look like in order to make the multimodal ...
Can machines talk? Comparison of Eliza with modern dialogue systems
To find if current dialogue systems use the same, psychotherapist questioning technique as Joseph Weizenbaum's 1960 natural language understanding programme, Eliza, the authors carried out an original experiment comparing five successful artificial ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2023
504 pages
ISBN:9798400700149
DOI:10.1145/3571884
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 July 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
CUI '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 34 of 100 submissions, 34%
Upcoming Conference
CUI '25
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 107Total Downloads
- Downloads (Last 12 months)46
- Downloads (Last 6 weeks)3
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format