Article

Free access

A machine learning approach to the automatic evaluation of machine translation

Authors:

Simon Corston-Oliver,

Michael Gamon,

Chris BrockettAuthors Info & Claims

ACL '01: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics

Pages 148 - 155

https://doi.org/10.3115/1073012.1073032

Published: 06 July 2001 Publication History

PDF eReader

Abstract

We present a machine learning approach to evaluating the well-formedness of output of a machine translation system, using classifiers that learn to distinguish human reference translations from machine translations. This approach can be used to evaluate an MT system, tracking improvements over time; to aid in the kind of failure analysis that can help guide system development; and to select among alternative output strings. The method presented is fully automated and independent of source language, target language and domain.

References

[1]

Alshawi, H., S. Bangalore, and S. Douglas. 1998. Automatic acquisition of hierarchical transduction models for machine translation. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics, Montreal Canada, Vol. 1:41--47.

Digital Library

Google Scholar

[2]

Bangalore, S., O. Rambow, and S. Whittaker. 2000. Evaluation Metrics for Generation. In Proceedings of the International Conference on Natural Language Generation (INLG 2000), Mitzpe Ramon, Israel. 1-13.

Digital Library

Google Scholar

[3]

Chickering, D. M., D. Heckerman, and C. Meek. 1997. A Bayesian approach to learning Bayesian networks with local structure. In Geiger, D. and P. Punadlik Shenoy (Eds.), Uncertainty in Artificial Intelligence: Proceedings of the Thirteenth Conference. 80--89.

Digital Library

Google Scholar

[4]

Clarkson, P. and R. Rosenfeld. 1997. Statistical Language Modeling Using the CMU-Cambridge Toolkit. Proceedings of Eurospeech97. 2707--2710.

Google Scholar

[5]

Heckerman, D., D. M. Chickering, C. Meek, R. Rounthwaite, and C. Kadie. 2000. Dependency networks for inference, collaborative filtering and data visualization. Journal of Machine Learning Research 1:49--75.

Digital Library

Google Scholar

[6]

Heidorn, G. E., 2000. Intelligent writing assistance. In R. Dale, H. Moisl and H. Somers (Eds.). Handbook of Natural Language Processing. New York, NY. Marcel Dekker. 181--207.

Google Scholar

[7]

Langkilde, I., and K. Knight. 1998. Generation that exploits corpus-based statistical knowledge. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics, and 17th International Conference on Computational Linguistics, Montreal, Canada. 704--710.

Digital Library

Google Scholar

[8]

Nyberg, E. H., T. Mitamura, and J. G. Carbonnell. 1994. Evaluation Metrics for Knowledge-Based Machine Translation. In Proceedings of the 15th International Conference on Computational Linguistics, Kyoto, Japan (Coling 94). 95--99.

Digital Library

Google Scholar

[9]

Platt, J., N. Cristianini, J. Shawe-Taylor. 2000. Large margin DAGs for multiclass classification. In Advances in Neural Information Processing Systems 12, MIT Press. 547--553.

Digital Library

Google Scholar

[10]

Richardson, S., B. Dolan, A. Menezes, and J. Pinkham. 2001. Achieving commercial-quality translation with example-based methods. Submitted for review.

Google Scholar

[11]

Ringger, E., M. Corston-Oliver, and R. Moore. 2001. Using Word-Perplexity for Automatic Evaluation of Machine Translation. Manuscript.

Google Scholar

[12]

Su, K., M. Wu, and J. Chang. 1992. A new quantitative quality measure for machine translation systems. In Proceedings of COLING-92, Nantes, France. 433--439.

Digital Library

Google Scholar

[13]

Vapnik, V. 1998. Statistical Learning Theory, Wiley-Interscience, New York.

Digital Library

Google Scholar

Cited By

View all

Huang MZhu XGao J(2020)Challenges in Building Intelligent Open-domain Dialog SystemsACM Transactions on Information Systems10.1145/338312338:3(1-32)Online publication date: 9-Apr-2020
https://dl.acm.org/doi/10.1145/3383123
Sharif NWhite LBennamoun MLiu WShah S(2019)LCEval: Learned Composite Metric for Caption EvaluationInternational Journal of Computer Vision10.1007/s11263-019-01206-z127:10(1586-1610)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s11263-019-01206-z
Yang MSun SZhu JLi SZhao TZhu X(2018)Improvement of machine translation evaluation by simple linguistically motivated featuresJournal of Computer Science and Technology10.5555/1991836.199184326:1(57-67)Online publication date: 21-Dec-2018
https://dl.acm.org/doi/10.5555/1991836.1991843
Show More Cited By

A machine learning approach to the automatic evaluation of machine translation

Recommendations

Dependency-based automatic evaluation for machine translation
SSST '07: Proceedings of the NAACL-HLT 2007/AMTA Workshop on Syntax and Structure in Statistical Translation

We present a novel method for evaluating the output of Machine Translation (MT), based on comparing the dependency structures of the translation and reference rather than their surface string forms. Our method uses a treebank-based, widecoverage, ...
Evaluation of machine translation
ICWET '11: Proceedings of the International Conference & Workshop on Emerging Trends in Technology

Machine Translation (MT) refers to the use of a machine for performing translation task which converts text or speech from one Natural Language (NL) into another Natural Language. Machine Translation is an important technology for localization, and is ...
N-gram-based statistical machine translation versus syntax augmented machine translation: comparison and system combination
EACL '09: Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics

In this paper we compare and contrast two approaches to Machine Translation (MT): the CMU-UKA Syntax Augmented Machine Translation system (SAMT) and UPC-TALP N-gram-based Statistical Machine Translation (SMT). SAMT is a hierarchical syntax-driven ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ACL '01: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics

July 2001

562 pages

General Chair:
Bonnie Lynn Webber

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 06 July 2001

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 85 of 443 submissions, 19%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
783
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Huang MZhu XGao J(2020)Challenges in Building Intelligent Open-domain Dialog SystemsACM Transactions on Information Systems10.1145/338312338:3(1-32)Online publication date: 9-Apr-2020
https://dl.acm.org/doi/10.1145/3383123
Sharif NWhite LBennamoun MLiu WShah S(2019)LCEval: Learned Composite Metric for Caption EvaluationInternational Journal of Computer Vision10.1007/s11263-019-01206-z127:10(1586-1610)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s11263-019-01206-z
Yang MSun SZhu JLi SZhao TZhu X(2018)Improvement of machine translation evaluation by simple linguistically motivated featuresJournal of Computer Science and Technology10.5555/1991836.199184326:1(57-67)Online publication date: 21-Dec-2018
https://dl.acm.org/doi/10.5555/1991836.1991843
Li MWang M(2018)Optimizing Automatic Evaluation of Machine Translation with the ListMLE ApproachACM Transactions on Asian and Low-Resource Language Information Processing10.1145/322604518:1(1-18)Online publication date: 12-Nov-2018
https://dl.acm.org/doi/10.1145/3226045
Ayala BChen JMcDonald RWorby NJatowt AMarshall CMilligan I(2017)A machine learning approach to evaluating translation qualityProceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries10.5555/3200334.3200373(281-282)Online publication date: 19-Jun-2017
https://dl.acm.org/doi/10.5555/3200334.3200373
Rubino RFoster JWagner JRoturier JKaljahi RHollowood F(2012)DCU-symantec submission for the WMT 2012 quality estimation taskProceedings of the Seventh Workshop on Statistical Machine Translation10.5555/2393015.2393034(138-144)Online publication date: 7-Jun-2012
https://dl.acm.org/doi/10.5555/2393015.2393034
Amigó EGonzalo JGiménez JVerdejo FMerlo PBarzilay RJohnson M(2011)Corroborating text evaluation results with heterogeneous measuresProceedings of the Conference on Empirical Methods in Natural Language Processing10.5555/2145432.2145485(455-466)Online publication date: 27-Jul-2011
https://dl.acm.org/doi/10.5555/2145432.2145485
Song XCohn TCallison-Burch CKoehn PMonz CZaidan O(2011)Regression and ranking based optimisation for sentence level machine translation evaluationProceedings of the Sixth Workshop on Statistical Machine Translation10.5555/2132960.2132975(123-129)Online publication date: 30-Jul-2011
https://dl.acm.org/doi/10.5555/2132960.2132975
Parton KTetreault JMadnani NChodorow MCallison-Burch CKoehn PMonz CZaidan O(2011)e-rating machine translationProceedings of the Sixth Workshop on Statistical Machine Translation10.5555/2132960.2132973(108-115)Online publication date: 30-Jul-2011
https://dl.acm.org/doi/10.5555/2132960.2132973
Nenkova AChae JLouis APitler E(2010)Structural features for predicting the linguistic quality of textEmpirical methods in natural language generation10.5555/1880370.1880386(222-241)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.5555/1880370.1880386
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Recommendations

Dependency-based automatic evaluation for machine translation

Evaluation of machine translation

N-gram-based statistical machine translation versus syntax augmented machine translation: comparison and system combination

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations