<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="youssef-etal-2024-queen">
    <titleInfo>
        <title>The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Paul</namePart>
        <namePart type="family">Youssef</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jörg</namePart>
        <namePart type="family">Schlötterer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christin</namePart>
        <namePart type="family">Seifert</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matthew</namePart>
            <namePart type="family">Purver</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Factual knowledge encoded in Pre-trained Language Models (PLMs) enriches their representations and justifies their use as knowledge bases. Previous work has focused on probing PLMs for factual knowledge by measuring how often they can correctly predict an _object_ entity given a subject and a relation, and improving fact retrieval by optimizing the prompts used for querying PLMs. In this work, we consider a complementary aspect, namely the coherency of factual knowledge in PLMs, i.e., how often can PLMs predict the _subject_ entity given its initial prediction of the object entity. This goes beyond evaluating how much PLMs know, and focuses on the internal state of knowledge inside them. Our results indicate that PLMs have low coherency using manually written, optimized and paraphrased prompts, but including an evidence paragraph leads to substantial improvement. This shows that PLMs fail to model inverse relations and need further enhancements to be able to handle retrieving facts from their parameters in a coherent manner, and to be considered as knowledge bases.</abstract>
    <identifier type="citekey">youssef-etal-2024-queen</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-eacl.155</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>2342</start>
            <end>2354</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs
%A Youssef, Paul
%A Schlötterer, Jörg
%A Seifert, Christin
%Y Graham, Yvette
%Y Purver, Matthew
%S Findings of the Association for Computational Linguistics: EACL 2024
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F youssef-etal-2024-queen
%X Factual knowledge encoded in Pre-trained Language Models (PLMs) enriches their representations and justifies their use as knowledge bases. Previous work has focused on probing PLMs for factual knowledge by measuring how often they can correctly predict an _object_ entity given a subject and a relation, and improving fact retrieval by optimizing the prompts used for querying PLMs. In this work, we consider a complementary aspect, namely the coherency of factual knowledge in PLMs, i.e., how often can PLMs predict the _subject_ entity given its initial prediction of the object entity. This goes beyond evaluating how much PLMs know, and focuses on the internal state of knowledge inside them. Our results indicate that PLMs have low coherency using manually written, optimized and paraphrased prompts, but including an evidence paragraph leads to substantial improvement. This shows that PLMs fail to model inverse relations and need further enhancements to be able to handle retrieving facts from their parameters in a coherent manner, and to be considered as knowledge bases.
%U https://aclanthology.org/2024.findings-eacl.155
%P 2342-2354

Markdown (Informal)

[The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs](https://aclanthology.org/2024.findings-eacl.155) (Youssef et al., Findings 2024)

• The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs (Youssef et al., Findings 2024)

ACL

• Paul Youssef, Jörg Schlötterer, and Christin Seifert. 2024. The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2342–2354, St. Julian’s, Malta. Association for Computational Linguistics.