Abstract
Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Ball Linden J, Eger N, Stevens R, Dodd J (2006) Applying the post-experience eye-tracked protocol (PEEP) method in usability testing. Interfaces 67: 15–19
Caffrey C (2008) Using pupillometric, fixation-based and subjective measures to measure the processing effort experienced when viewing subtitled TV anime with pop-up gloss. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 125–144
Daneman M, Carpenter P (1980) Individual differences in working memory and reading. J Verbal Learn Verbal Behav 19(4): 450–466
Duchowski A (2003) Eye-tracking methodology—theory and practice. Springer-Verlag, London
Göpferich, S, Jakobsen, A, Mees, I (eds) (2009) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 125–144
Hess E, Polt J (1964) Pupil size in relation to mental activity in simple problem solving. Science 143: 1190–1192
Hyönä J, Nurminen AM (2006) Do adult readers know how they read? Evidence from eye movement patterns and verbal reports. Br J Psychol 97: 31–50
Iqbal S, Adamzyck P, Zheng X, Bailey P (2005) Towards an index of opportunity: understanding changes in mental workload during task execution. In: Human factors in computing systems: proceedings of CHI’05. ACM Press, New York, pp 311–320
Jakobsen AL, Jensen K (2009) Eye movement behaviour across four different types of reading task. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 103–124
Jakobsen AL (1999) Logging target text production with Translog. In: Hansen G (eds) Probing the process in translation: methods and results. Copenhagen Studies in Language 24. Samfundslitteratur, Copenhagen, pp 9–20
Kaakinen JK, Hyönä J (2005) Perspective effects on expository text comprehension: evidence from think-aloud protocols, eyetracking, and recalls. Discourse Process 40: 239–257
Kaakinen JK, Hyönä J, Keenan J (2003) How prior knowledge, WMC, and relevance of information affect eye fixations in expository text. J Exp Psychol 29(3): 447–457
Kintsch W (1998) Comprehension: a paradigm for cognition. Cambridge University Press, Cambridge, England
Nakayama M, Koji T, Yasutaka S (2002) The act of task difficulty and eye-movement frequency for the oculo-motor indices. In: Proceedings of the symposium on eye tracking research and application, New Orleans, Louisiana, pp 37–42
O’Brien S (forthcoming) Controlled language and readability. In: Shreve G, Angelone E (eds) Translation and cognition. American Translators Association Scholarly Monograph Series. John Benjamins, Amsterdam
O’Brien S (2008) Processing fuzzy matches in translation memory tools—an eye-tracking analysis. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 79–102
O’Brien S (2006) Eye-tracking and translation memory matches. Perspectives 14(3): 185–205
Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124: 372–422
Roturier J (2006) An investigation into the impact of controlled English rules on the comprehensibility, usefulness, and acceptability of machine-translated technical documentation for French and German users. PhD Dissertation, Dublin City University
Roturier J (2009) Deploying novel MT technology to raise the bar for quality: a review of key advantages and challenges. MT Summit XII: proceedings of the twelfth Machine Translation Summit, August 26–30, Ottawa, Ontario, Canada
Schultheis H, Jameson A (2004) Assessing cognitive load in adaptive hypermedia systems: Physiological and behavioural methods. In: Neijdlw W, Bra P (eds) Adaptive hypermedia and adaptive web-based systems. Springer Verlag, Eindhoven, pp 18–24
Stanford Poynter Project: http://www.poynterextra.org/et/i.htm [Last accessed: 29/04/2009]
Van Gog T, Kester L, Nievelstein F, Giesbers B, Paas F (2009) Uncovering cognitive processes: different techniques that can contribute to cognitive load research and instruction. Comput Hum Behav 25: 325–331
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Doherty, S., O’Brien, S. & Carl, M. Eye tracking as an MT evaluation technique. Machine Translation 24, 1–13 (2010). https://doi.org/10.1007/s10590-010-9070-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10590-010-9070-9