UnAnswGen: A Systematic Approach for Generating Unanswerable Questions in Machine Reading Comprehension
Authors: 
Hadiseh Moradisani, Fattane Zarrinkalam, Julien Serbanescu, Zeinab NoorianAuthors Info & Claims
Pages 280 - 286
Abstract
This paper introduces a configurable software workflow to automatically generate and publicly share a dataset of multi-labeled unanswerable questions for Machine Reading Comprehension (MRC). Unlike existing datasets like SQuAD2.0, which do not account for the reasons behind question unanswerability, our method fills a critical gap by systematically transforming answerable questions into their unanswerable counterparts across various linguistic dimensions including entity swap, number swap, negation, antonym, mutual exclusion, and no information. These candidate unanswerable questions are evaluated using advanced MRC models to ensure their context-based unanswerability, with the final selection based on a majority consensus mechanism. Our approach addresses the scarcity of multi-labeled datasets like SQuAD2-CR, enabling comprehensive evaluation of MRC systems' ability to handle unanswerable queries and facilitating the exploration of solutions such as query reformulation. The resulting UnAnswGen dataset and associated software workflow are made publicly available to advance research in machine reading comprehension, offering researchers a standardized toolset for evaluating and enhancing MRC systems' robustness and performance.
References
[1]
Seohyun Back, Sai Chetan Chinthakindi, Akhil Kedia, Haejun Lee, and Jaegul Choo. 2020. NeurQuRI: Neural question requirement inspector for answerability prediction in machine reading comprehension. In International Conference on Learning Representations.
[2]
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ''O'Reilly Media, Inc.''.
[3]
Chang Nian Chuy, Qinmin Vivian Hu, and Chen Ding. 2023. One Stop Shop for Question-Answering Dataset Selection. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3115--3119.
[4]
Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723 (2017).
[5]
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In International Conference on Learning Representations.
[6]
Christiane Fellbaum. 1998. WordNet: An electronic lexical database.
[7]
Ge Gao, Hung-Ting Chen, Yoav Artzi, and Eunsol Choi. 2023. Continually Improving Extractive QA via Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 406--423.
[8]
Vagrant Gautam, Miaoran Zhang, and Dietrich Klakow. 2023. A Lightweight Method to Generate Unanswerable Questions in English. In Findings of the Association for Computational Linguistics: EMNLP 2023. 7349--7360.
[9]
Kilem L Gwet. 2011. On the Krippendorff's alpha coefficient. Manuscript submitted for publication. Retrieved October, Vol. 2, 2011 (2011), 2011.
[10]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020).
[11]
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. 2018. DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications. In Proceedings of the Workshop on Machine Reading for Question Answering. 37--46.
[12]
Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, and Dongsheng Li. 2019. Read verify: Machine reading comprehension with unanswerable questions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 6529--6537.
[13]
Kevin Huang, Yun Tang, Jing Huang, Xiaodong He, and Bowen Zhou. 2019. Relation module for non-answerable predictions on reading comprehension. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). 747--756.
[14]
Yunjie Ji, Liangyu Chen, Chenxiao Dou, Baochang Ma, and Xiangang Li. 2022. To answer or not to answer? Improving machine reading comprehension model with span-based contrastive learning. arXiv preprint arXiv:2208.01299 (2022).
[15]
Souvik Kundu and Hwee Tou Ng. 2018. A nil-aware answer extraction framework for question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 4243--4252.
[16]
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, Vol. 7 (2019), 453--466.
[17]
Gyeongbok Lee, Seung-won Hwang, and Hyunsouk Cho. 2020. SQuAD2-CR: Semi-supervised annotation for cause and rationales for unanswerability in SQuAD 2.0. In Proceedings of the Twelfth Language Resources and Evaluation Conference. 5425--5432.
[18]
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115 (2017).
[19]
Jinzhi Liao, Xiang Zhao, Jianming Zheng, Xinyi Li, Fei Cai, and Jiuyang Tang. 2022. Ptau: Prompt tuning for attributing unanswerable questions. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1219--1229.
[20]
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Jiancheng Lv, Nan Duan, and Ming Zhou. 2020. Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5798--5810.
[21]
Qian Liu, Rui Mao, Xiubo Geng, and Erik Cambria. 2023. Semantic matching in machine reading comprehension: An empirical study. Information Processing & Management, Vol. 60, 2 (2023), 103145.
[22]
Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2018. Stochastic Answer Networks for Machine Reading Comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1694--1704.
[23]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
[24]
Dheeraj Mekala, Jason Wolfe, and Subhro Roy. 2023. ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 5792--5799.
[25]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human-generated machine reading comprehension dataset. (2016).
[26]
Hariom A Pandya and Brijesh S Bhatt. 2021. Question answering survey: Directions, challenges, datasets, evaluation matrices. arXiv preprint arXiv:2112.03572 (2021).
[27]
Wei Peng, Yue Hu, Jing Yu, Luxi Xing, and Yuqiang Xie. 2021. APER: adaptive evidence-driven reasoning network for machine reading comprehension with unanswerable questions. Knowledge-Based Systems, Vol. 229 (2021), 107364.
[28]
Rifki Afina Putri and Alice Oh. 2022. IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 6918--6933.
[29]
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018).
[30]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016).
[31]
Amrita Saha, Rahul Aralikatte, Mitesh M Khapra, and Karthik Sankaranarayanan. 2018. DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1683--1693.
[32]
Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang, Weifeng Lv, and Ming Zhou. 2018. I know there is no answer: Modeling answer validation for machine reading comprehension. In Natural Language Processing and Chinese Computing: 7th CCF International Conference, NLPCC 2018, Hohhot, China, August 26--30, 2018, Proceedings, Part I 7. Springer, 85--97.
[33]
Son Quoc Tran, Gia-Huy Do, Phong Nguyen-Thuan Do, Matt Kretchmar, and Xinya Du. 2023. AGent: A Novel Pipeline for Automatically Creating Unanswerable Questions. arXiv preprint arXiv:2309.05103 (2023).
[34]
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830 (2016).
[35]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2369--2380.
[36]
Changchang Zeng, Shaobo Li, Qin Li, Jie Hu, and Jianjun Hu. 2020. A survey on machine reading comprehension-tasks, evaluation metrics and benchmark datasets. Applied Sciences, Vol. 10, 21 (2020), 7640.
[37]
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020. SG-Net: Syntax-guided machine reading comprehension. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 9636--9643.
[38]
Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2021. Retrospective reader for machine reading comprehension. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 14506--14514.
[39]
Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019. Learning to Ask Unanswerable Questions for Machine Reading Comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 4238--4248.
Index Terms
- UnAnswGen: A Systematic Approach for Generating Unanswerable Questions in Machine Reading Comprehension
Recommendations
Sentence Extraction-Based Machine Reading Comprehension for Vietnamese
Knowledge Science, Engineering and ManagementAbstractThe development of natural language processing (NLP) in general and machine reading comprehension in particular has attracted the great attention of the research community. In recent years, there are a few datasets for machine reading ...
MA-MRC: A Multi-answer Machine Reading Comprehension Dataset
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information RetrievalMachine reading comprehension (MRC) is an essential task for many question-answering applications. However, existing MRC datasets mainly focus on data with single answer and overlook multiple answers, which are common in the real world. In this paper, we ...
I Know There Is No Answer: Modeling Answer Validation for Machine Reading Comprehension
Natural Language Processing and Chinese ComputingAbstractExisting works on machine reading comprehension mostly focus on extracting text spans from passages with the assumption that the passage must contain the answer to the question. This assumption usually cannot be satisfied in real-life ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2024
328 pages
ISBN:9798400707247
DOI:10.1145/3673791
- General Chairs:
- Tetsuya Sakai,
- Emi Ishita,
- Hiroaki Ohshima,
- Program Chairs:
- Faegheh Radboud University, Netherlands,
- Jiaxin Mao,
- Joemon Jose
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 December 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SIGIR-AP 2024
Sponsor:
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 17Total Downloads
- Downloads (Last 12 months)17
- Downloads (Last 6 weeks)17
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in