<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kovacs-etal-2024-mitigating">
    <titleInfo>
        <title>Mitigating Metric Bias in Minimum Bayes Risk Decoding</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Geza</namePart>
        <namePart type="family">Kovacs</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Daniel</namePart>
        <namePart type="family">Deutsch</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Markus</namePart>
        <namePart type="family">Freitag</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Ninth Conference on Machine Translation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Barry</namePart>
            <namePart type="family">Haddow</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tom</namePart>
            <namePart type="family">Kocmi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Philipp</namePart>
            <namePart type="family">Koehn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christof</namePart>
            <namePart type="family">Monz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as any improvement might simply be due to reward hacking rather than reflecting real quality improvements. In this work we demonstrate that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.</abstract>
    <identifier type="citekey">kovacs-etal-2024-mitigating</identifier>
    <identifier type="doi">10.18653/v1/2024.wmt-1.109</identifier>
    <location>
        <url>https://aclanthology.org/2024.wmt-1.109</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>1063</start>
            <end>1094</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Mitigating Metric Bias in Minimum Bayes Risk Decoding
%A Kovacs, Geza
%A Deutsch, Daniel
%A Freitag, Markus
%Y Haddow, Barry
%Y Kocmi, Tom
%Y Koehn, Philipp
%Y Monz, Christof
%S Proceedings of the Ninth Conference on Machine Translation
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F kovacs-etal-2024-mitigating
%X While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as any improvement might simply be due to reward hacking rather than reflecting real quality improvements. In this work we demonstrate that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.
%R 10.18653/v1/2024.wmt-1.109
%U https://aclanthology.org/2024.wmt-1.109
%U https://doi.org/10.18653/v1/2024.wmt-1.109
%P 1063-1094

Markdown (Informal)

[Mitigating Metric Bias in Minimum Bayes Risk Decoding](https://aclanthology.org/2024.wmt-1.109) (Kovacs et al., WMT 2024)

• Mitigating Metric Bias in Minimum Bayes Risk Decoding (Kovacs et al., WMT 2024)

ACL

• Geza Kovacs, Daniel Deutsch, and Markus Freitag. 2024. Mitigating Metric Bias in Minimum Bayes Risk Decoding. In Proceedings of the Ninth Conference on Machine Translation, pages 1063–1094, Miami, Florida, USA. Association for Computational Linguistics.