<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sen-saffari-2020-models">
    <titleInfo>
        <title>What do Models Learn from Question Answering Datasets?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Priyanka</namePart>
        <namePart type="family">Sen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Amir</namePart>
        <namePart type="family">Saffari</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Bonnie</namePart>
            <namePart type="family">Webber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Trevor</namePart>
            <namePart type="family">Cohn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yulan</namePart>
            <namePart type="family">He</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yang</namePart>
            <namePart type="family">Liu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>While models have reached superhuman performance on popular question answering (QA) datasets such as SQuAD, they have yet to outperform humans on the task of question answering itself. In this paper, we investigate if models are learning reading comprehension from QA datasets by evaluating BERT-based models across five datasets. We evaluate models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. We find that no single dataset is robust to all of our experiments and identify shortcomings in both datasets and evaluation methods. Following our analysis, we make recommendations for building future QA datasets that better evaluate the task of question answering through reading comprehension. We also release code to convert QA datasets to a shared format for easier experimentation at https://github.com/amazon-research/qa-dataset-converter</abstract>
    <identifier type="citekey">sen-saffari-2020-models</identifier>
    <identifier type="doi">10.18653/v1/2020.emnlp-main.190</identifier>
    <location>
        <url>https://aclanthology.org/2020.emnlp-main.190</url>
    </location>
    <part>
        <date>2020-11</date>
        <extent unit="page">
            <start>2429</start>
            <end>2438</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T What do Models Learn from Question Answering Datasets?
%A Sen, Priyanka
%A Saffari, Amir
%Y Webber, Bonnie
%Y Cohn, Trevor
%Y He, Yulan
%Y Liu, Yang
%S Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F sen-saffari-2020-models
%X While models have reached superhuman performance on popular question answering (QA) datasets such as SQuAD, they have yet to outperform humans on the task of question answering itself. In this paper, we investigate if models are learning reading comprehension from QA datasets by evaluating BERT-based models across five datasets. We evaluate models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. We find that no single dataset is robust to all of our experiments and identify shortcomings in both datasets and evaluation methods. Following our analysis, we make recommendations for building future QA datasets that better evaluate the task of question answering through reading comprehension. We also release code to convert QA datasets to a shared format for easier experimentation at https://github.com/amazon-research/qa-dataset-converter
%R 10.18653/v1/2020.emnlp-main.190
%U https://aclanthology.org/2020.emnlp-main.190
%U https://doi.org/10.18653/v1/2020.emnlp-main.190
%P 2429-2438

Markdown (Informal)

[What do Models Learn from Question Answering Datasets?](https://aclanthology.org/2020.emnlp-main.190) (Sen & Saffari, EMNLP 2020)

• What do Models Learn from Question Answering Datasets? (Sen & Saffari, EMNLP 2020)

ACL

• Priyanka Sen and Amir Saffari. 2020. What do Models Learn from Question Answering Datasets?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2429–2438, Online. Association for Computational Linguistics.