Abstract
The improvement in machine translation quality is creating a constantly increasing number of post-editing jobs. As a result, research geared toward ensuring an efficient translation process for post-editors has become more important than ever. To this end, being able to measure and predict the effort involved during the post-editing activity is essential. This work aims to assess whether simple post-editing effort metrics associated with the three effort dimensions (temporal, cognitive, and technical) correlate among themselves. Also, it seeks to examine whether these simple metrics are able to capture the variation in effort involved in addressing different error types. To address these objectives, we asked professional translators to post-edit a test suite of sentences that include one pre-selected error each and used a set of simple metrics to measure the post-editing effort. Results seem to indicate that the correlation between the metrics is rather low, which suggests that the use of a single metric to measure the effort might produce biased measurements. We also observe that, overall, metrics report very similar effort values for the different error types but some distinctions are noticeable, which allow us to rank error types per difficulty.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
See Do Carmo (this volume, Chap. 1) for a discussion on the terminological confusion of different definitions of HTER.
- 2.
https://translate.google.com/.
- 3.
- 4.
The differences are not statistically significant for the following pairs: total time and pause time; total time and editing pause time; total time and first pause; total time and last pause; editing time and pause count; editing time and editing pause count; editing time and keystrokes; pause time and editing pause time; pause time and first pause; pause time and last pause.
References
Alves F, Szpak K, Luiz Gonçalve J, Sekino K, Aquino M, Castro R, Koglin A, Fonseca N, Mesa-Lao B (2016) Investigating cognitive effort in post-editing: a relevance-theoretical approach. In: Hansen-Schirra S, Grucza S (eds) Eye-tracking and applied linguistics. Language Science Press, pp 109–142
Aranberri N (2017) What do professional translators do when post-editing for the first time? First insight into the Spanish-Basque language pair. HERMES J Lang Commun Bus 56:89–110
Aranberri N, Pascual JA (2018) Towards a post-editing recommendation system for Spanish–Basque machine translation. In: Proceedings of the 21st annual conference of the European association for machine translation. European Association for Machine Translation, pp 21–30
Arnold D, Moffat D, Sadler L, Way A (1993) Automatic test suite generation. Mach Transl 8(1–2):29–38
Aziz W, Castilho S, Specia L (2012) PET: a tool for post-editing and assessing machine translation. In: Eighth international conference on language resources and evaluation. European Language Resources Association (ELRA), pp 3982–3987
Blain F, Senellart J, Schwenk H, Plitt M, Roturier J (2011) Qualitative analysis of post-editing for high quality machine translation. In: Proceedings of the XIII machine translation summit. Asia-Pacific Association for Machine Translation, pp 164–171
Burchardt A, Harris K, Rehm G, Uszkoreit H (2016) Towards a systematic and human-informed paradigm for high-quality machine translation. In: Proceedings of the LREC 2016 workshop “translation evaluation – from fragmented tools and data sets to an integrated ecosystem”. European Language Resources Association (ELRA), pp 35–42
Carl M (2012) Translog-II: a program for recording user activity data for empirical reading and writing research. In: Eighth international conference on language resources and evaluation. European Language Resources Association (ELRA), pp 4108–4112
Carl M, Schaeffer MJ (2017) Models of the translation process. The handbook of translation and cognition, pp 50–70
Carl M, Dragsted B, Elming J, Hardt D, Jakobsen AL (2011) The process of post-editing: a pilot study. Copenhagen Stud Lang 41:131–142
Daems J, Vandepitte S, Hartsuiker R, Macken L (2015) The impact of machine translation error types on post-editing effort indicators. In: 4th workshop on post-editing technology and practice (WPTP4). Association for Machine Translation in the Americas, pp 31–45
Daems J, Vandepitte S, Hartsuiker R, Macken L (2017) Identifying the machine translation error types with the greatest impact on post-editing effort. Front Psychol 8:1282
De Almeida G (2013) Translating the post-editor: an investigation of post-editing changes and correlations with professional experience across two romance languages. Ph.D. thesis, Dublin City University
De Almeida G, O’Brien S (2010) Analysing post-editing performance: correlations with years of translation experience. In: Proceedings of the 14th annual conference of the European association for machine translation, pp 26–28
Dragsted B, Hansen I (2009) Exploring translation and interpreting hybrids. the case of sight translation. Meta: journal des traducteurs/Meta: Translators’ Journal 54(3):588–604
Flanagan M, Christensen TP (2014) Testing post-editing guidelines: how translation trainees interpret them and how to tailor them for translator training purposes. Interpreter Transl Trainer 8(2):257–275
Gaspari F, Almaghout H, Doherty S (2015) A survey of machine translation competences: insights for translation technology educators and practitioners. Perspect Stud Translatology 23:1–26
de Gibert O, Aranberri N (2019) Estrategia multidimensional para la selección de candidatos de traducción automática para posedición. Linguamática 11(2):3–16
Guerberof A (2009) Productivity and quality in mt post-editing. In: Proceedings of the MT summit XII-Workshop: beyond translation memories: new tools for translators MT. Association for Machine Translation in the Americas, pp 1–9
Guerberof A (2013) What do professional translators think about post-editing? J Specialised Transl 19:75–95
Guillou L, Hardmeier C (2016) Protest: a test suite for evaluating pronouns in machine translation. In: Proceedings of the tenth international conference on language resources and evaluation (LREC’16). European Language Resources Association (ELRA), pp 636–643
Hu K, Cadwell P (2016) A comparative study of post-editing guidelines. Baltic J Modern Comput 2:346–353
Koponen M (2012) Comparing human perceptions of post-editing effort with post-editing operations. In: Proceedings of the seventh workshop on statistical machine translation. Association for Computational Linguistics, pp 181–190
Koponen M (2016) Machine translation post-editing and effort: Empirical studies on the post-editing process. University of Helsinki, Helsinki
Koponen M, Salmi L, Nikulin M (2019) A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output. Mach Transl 33:61–90
Krings H (2001) Repairing texts: empirical investigations of machine translation post-editing processes. The Kent State University Press
Lacruz I, Shreve G (2914) Pauses and cognitive effort in post-editing. In Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia (Eds.), Post-editing of Machine Translation: Processes and Applications. Cambridge Scholars Publishing, pp 246–272
Lacruz I, Shreve G, Angelone E (2012) Average pause ratio as an indicator of cognitive effort in post-editing: A case study. In: Proceedings of the workshop on post-editing technology and practice (WPTP). Association for Machine Translation in the Americas, pp 1–10
Lacruz I, Denkowski M, Lavie A (2014) Cognitive demand and cognitive effort in post-editing. In: Proceedings of the third workshop on post-editing technology and practice (WPTP-3). Association for Machine Translation in the Americas, pp 73–84
Martínez-Gómez P, Han D, Carl M, Aizawa A (2018) Recognition and characterization of translator attributes using sequences of fixations and keystrokes. Eye tracking and multidisciplinary studies on translation, pp 97–120
Massardo I, van der Meer J, O’Brien S, Hollowood F, Aranberri N, Drescher K (2016) MT post-editing guidelines. TAUS Signature Editions
Mesa-Lao B (2013) Eye-tracking post-editing behaviour in an interactive translation prediction environment. J Eye Mov Res 6(3):541
Moorkens J (2018) Eye tracking as a measure of cognitive effort for post-editing of machine translation. In: Walker C, Federici F (eds) Eye tracking and multidisciplinary studies on translation. John Benjamins, pp 55–70
Moorkens J, O’Brien S, da Silva I, de Lima Fonseca N, Alves F (2015) Correlation of perceived post-editing effort with measurements of actual effort. Mach Transl 29:267–284
Nitzke J, Oster K (2016) Comparing translation and post-editing: an annotation schema for activity units. In: Carl M, Srinivas B, Moritz S (eds) New directions in empirical translation process research. Springer, pp 293–308
O’Brien S (2005) Methodologies for measuring the correlations between post-editing effort and machine translatability. Mach Transl 19:37–58
O’Brien S (2006a) Eye-tracking and translation memory matches. Perspect Stud Translatol 14:185–205
O’Brien S (2006b) Pauses as indicators of cognitive effort in post-editing machine translation output. Across Lang Cult 7(1):1–21
O’Brien S (2011) Towards predicting post-editing productivity. Mach Transl 25:197–215
O’Brien S, Winther Balling L, Carl M, Simard M, Specia L (eds) (2014) Post-editing of machine translation: processes and applications. Cambridge Schoolars Publishing
Parra Escartín C, Arcedillo M (2015) Living on the edge: productivity gain thresholds in machine translation evaluation metrics. In: Proceedings of 4th workshop on post-editing technology and practice (WPTP4). Association for Machine Translation in the Americas, pp 46–56
Plitt M, Masselot F (2010) A productivity test of statistical machine translation post-editing in a typical localisation context. Prague Bull Math Clin Linguist 93:7–16
Popovic M, Lommel A, Burchardt A, Avramidis E, Uszkoreit H (2014) Relations between different types of post-editing operations, cognitive effort and temporal effort. In: Proceedings of the 17th annual conference of the european association for machine translation, pp 191–198
Probst A (2017) The effect of error type on pause length in post-editing machine translation output. Master’s thesis, Tilburg University
Schaeffer M, Nitzke J, Tardel A, oster K, Gutermuth S, Hansen-Schirra S (2019) Eye-tracking revision processes of translation students and professional translators. Perspectives Studies in Translation Theory and Practice 27(4): 589–603
da Silva IAL, Alves F, Schmaltz M, Pagano A, Wong D, Chao L, Leal ALV, Quaresma P, Garcia C, da Silva GE (2017) Translation, post-editing and directionality. Transl Transit Between Cogn Comput Technol 133:107–134
Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. In: Proceedings of the 7th conference of the association for machine translation in the Americas. Association for Machine Translation in the Americas, pp 223–231
Specia L (2011) Exploiting objective annotations for measuring translation post-editing effort. In: Proceedings of the 15th conference of the European association for machine translation. European Association for Machine Translation, pp 73–80
Specia L, Farzindar A (2010) Estimating machine translation post-editing effort with hter. In: Proceedings of the second joint EM+/CNGL workshop: bringing MT to the user: research on integrating MT in the translation industry (JEG 10), pp 33–41
Tatsumi M (2009) Correlation between automatic evaluation metric scores, post-editing speed, and some other factors. In: Proceedings of the XII machine translation summit, pp 332–339
Tatsumi M, Roturier J (2010) Source text characteristics and technical and temporal post-editing effort: what is their relationship. In: Proceedings of the second joint EM+/CNGL workshop bringing MT to the user: research on integrating MT in the translation industry (JEC 10), pp 43–51
Temnikova I (2010) Cognitive evaluation approach for a controlled language post–editing experiment. In: Proceedings of the seventh international conference on language resources and evaluation. European Language Resources Association, pp 3485–3490
Vieira LN (2016) How do measures of cognitive effort relate to each other? a multivariate analysis of post-editing process data. Mach Transl 30:41–62
Wisniewski G, Singh AK, Segal N, Yvon F (2013) Design and analysis of a large corpus of post-edited translations: quality estimation, failure analysis and the variability of post-edition. In: Proceedings of the XIV machine translation summit. European Association for Machine Translation, pp 117–124
Acknowledgements
The research leading to this work was partially funded by the Spanish MEIC and MCIU (UnsupNMT TIN2017-91692-EXP and DOMINO PGC2018-102041-B-I00, co-funded by EU FEDER), and the BigKnowledge project (BBVA foundation grant 2018).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Cumbreño, C., Aranberri, N. (2021). What Do You Say? Comparison of Metrics for Post-editing Effort. In: Carl, M. (eds) Explorations in Empirical Translation Process Research. Machine Translation: Technologies and Applications, vol 3. Springer, Cham. https://doi.org/10.1007/978-3-030-69777-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-69777-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69776-1
Online ISBN: 978-3-030-69777-8
eBook Packages: EducationEducation (R0)