More Web Proxy on the site http://driver.im/

article

Project ernestine: validating a GOMS analysis for predicting and explaining real-world task performance

Authors:

Bonnie E. John,

Michael E. AtwoodAuthors Info & Claims

Human-Computer Interaction, Volume 8, Issue 3

Pages 237 - 309

https://doi.org/10.1207/s15327051hci0803_3

Published: 01 September 1993 Publication History

Abstract

Project Ernestine served a pragmatic as well as a scientific goal: to compare the worktimes of telephone company toll and assistance operators on two different workstations and to validate a GOMS analysis for predicting and explaining real-world performance. Contrary to expectations, GOMS predicted and the data confirmed that performance with the proposed workstation was slower than with the current one. Pragmatically, this increase in performance time translates into a cost of almost $2 million a year to NYNEX. Scientifically, the GOMS models predicted performance with exceptional accuracy.

The empirical data provided us with three interesting results: proof that the new workstation was slower than the old one, evidence that this difference was not constant but varied with call category, and (in a trial that spanned 4 months and collected data on 72,450 phone calls) proof that performance on the new workstation stabilized after the first month. The GOMS models predicted the first two results and explained all three.

In this article, we discuss the process and results of model building as well as the design and outcome of the field trial. We assess the accuracy of GOMS predictions and use the mechanisms of the models to explain the empirical results. Last, we demonstrate how the GOMS models can be used to guide the design of a new workstation and evaluate design decisions before they are implemented.

References

[1]

Atwood, M. E., Gray, W. D., & John, B. E. (in press). Project Ernestine: Analytic and empirical method. applied to a real-world CHI problem. San Mateo, CA: Morgan-Kaufmann.

[2]

Benigni, D., Yao, S., & Hevner, A. R. (Eds.). (1984). A guide to performance evaluations of database systems (National Bureau of Standards Special Publication 500-118) Washington, DC: U.S. Government Printing Office.

[3]

Benigni, D., Yao, S., & Hevner, A. R. (Eds.). (1985). Benchmark analysis of database architectures: A case study (National Bureau of Standards Special Publication 500-132). Washington, DC: U.S. Government Printing Office.

Digital Library

[4]

Bovair, S., Kieras, D. E., & Polson, P. G. (1990). The acquisition and performance of text-editing skill: A cognitive complexity analysis. Human-Computer Interaction, 5, 1-48.

Digital Library

[5]

Card, S. K., Moran, T. P., & Newell, A. (1980a). Computer text editing: An information processing analysis of a routine cognitive skill. Cognitive Psychology, 12, 32-74.

[6]

Card, S. K., Moran, T. P., & Newell, A. (1980b). The keystroke-level model for user performance time with interactive systems. Communications of the ACM, 23, 396-410.

Digital Library

[7]

Card, S. K., Moran, T. P., & Newell, A. (1983). The psychology of human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Digital Library

[8]

Doyle, A. C. (1986). Silver blaze. Sherlock Holmes: The complete novels and stories (Vol. 1, pp. 455-477). New York: Bantam. (Original work published 1892).

[9]

Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement Journal of Experimental Psychology, 47, 381-391.

[10]

Gray, W. D., John, B. E., & Atwood, M. E. (1992). The préécis of Project Ernestine or an overview of a validation of GOMS. Proceedings of the CHI '92 Conference on Human Factors in Computing Systems, 307-312. New York: Addison-Wesley.

Digital Library

[11]

Gray, W. D., John, B. E., Lawrence, D., Stuart, R., & Atwood, M. E. (1989, May). GOMS meets the phone company, or, can 8,400,000 unit-tasks be wrong? Poster presented at the CHI '89 Conference on Human Factors in Computing Systems, Austin, TX.

[12]

Gray, W. D., John, B. E., Stuart, R., Lawrence, D., & Atwood, M. E. (1990). GOMS meets the phone company: Analytic modeling applied to real-world problems. Proceedings of Human-Computer Information-INTERACT '90, 29-34. New York: Elsevier.

Digital Library

[13]

Hillelsohn, M. J. (1984). Benchmarking authoring systems. Journal of Computer-Based Instruction, 11, 95-97.

[14]

John, B. E. (1988). Contributions to engineering models of human-computer interaction. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh.

Digital Library

[15]

John, B. E. (1990). Extensions of GOMS analyses to expert performance requiring perception of dynamic visual and auditory information. Proceedings of the CHI'90 Conference on Human Factors in Computing Systems, 107-115. New York: ACM.

Digital Library

[16]

John, B. E., & Gray, W. D. (1992, May). GOMS analyses for parallel activities. Tutorial presented at the CHI '92 Conference on Human Factors in Computing Systems, Monterey, CA.

Digital Library

[17]

John, B. E., & Newell, A. (1989a). Cumulating the science of HCI: From S-R compatibility to transcription typing. Proceedings of the CHI '89 Conference on Human Factors in Computing Systems, 109-114. New York: ACM.

Digital Library

[18]

John, B. E., & Newell, A. (1989b). Toward an engineering model of stimulus-response compatibility. In R. W. Proctor & T. G. Reeve (Eds.), Stimulus-response compatibility: An integrated perspective (pp. 427-479). New York: Elsevier.

[19]

John, B. E., Rosenbloom, P. S., & Newell, A. (1985). A theory of stimulus-response compatibility applied to human-computer interaction. Proceedings of the CHI '85 Conference on Human Factors in Computing Systems, 212-219. New York: ACM.

Digital Library

[20]

Judd, C. M., & McClelland, G. H. (1989). Data analysis: A model-comparison approach. New York: Harcourt Brace.

[21]

Keppel, G. (1973). Design and analysis: A researcher's handbook. Englewood Cliffs, NJ: Prentice-Hall.

[22]

Lawrence, D., & Dews, S. (1992, May). Natural dialog in a time-sensitive setting: A study of telephone operators. Short talk presented at the CHI '92 Conference on Human Factors in Computing Systems, Monterey, CA.

Digital Library

[23]

Lerch, F. J., Mantei, M. M., & Olson, J. R. (1989). Skilled financial planning: The cost of translating ideas into action. Proceedings of the CHI '89 Conference on Human Factors in Computing Systems, 121-126. New York: ACM.

Digital Library

[24]

Newell, A., & Card, S. K. (1985). The prospects for psychological science in human-computer interaction. Human-Computer Interaction, 1, 209-242.

Digital Library

[25]

Newell, A., & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition and the law of practice. In J. R. Anderson (Ed.), Cognitive skills and their acquisition (pp. 1-55). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

[26]

Norwine, A. C., & Murphy, O. J. (1938). Characteristic time intervals in telephone conversation. Bell System Technical Journal, 17, 281-291.

[27]

Olson, J. R., & Olson, G. M. (1990). The growth of cognitive modeling in human-computer interaction since GOMS. Human-Computer Interaction, 5, 221-265.

Digital Library

[28]

Roberts, T. L., & Moran, T. P. (1983). The evaluation of text editors: Methodology and empirical results. Communications of the ACM, 26, 265-282.

Digital Library

[29]

Snedecor, G. W., & Cochran, W. G. (1967). Statistical methods (6th ed.). Ames: Iowa State University Press.

[30]

Stevens, J. (1986). Applied multivariate statistics for the social sciences. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Digital Library

[31]

Stuart, R., & Gabrys, G. (1993, May). A speech compression proposal for directory assistance: GOMS predictions. Short talk presented at the INTERCHI '93 Conference on Human Factors in Computing Systems, Amsterdam.

Digital Library

Cited By

Khan SDevlen CManno MHou D(2024)Mouse Dynamics Behavioral Biometrics: A SurveyACM Computing Surveys10.1145/364031156:6(1-33)Online publication date: 24-Jan-2024
https://dl.acm.org/doi/10.1145/3640311
Inga JRuess MRobens JNelius TRothfuß SKille SDahlinger PLindenmann AThomaschke RNeumann GMatthiesen SHohmann SKiesel A(2023)Human-machine symbiosisInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2022.102926170:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2022.102926
Yuan HLi SRusconi P(2021)CogTool+ACM Transactions on Computer-Human Interaction10.1145/344753428:2(1-38)Online publication date: 17-Apr-2021
https://dl.acm.org/doi/10.1145/3447534
Show More Cited By

Index Terms

Project ernestine: validating a GOMS analysis for predicting and explaining real-world task performance

Recommendations

Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
ESEC/FSE '09: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering

Prediction of software defects works well within projects as long as there is a sufficient amount of data available to train any models. However, this is rarely the case for new software projects and for many companies. So far, only a few have studies ...
Improving Cross-Project Defect Prediction Methods with Data Simplification
SEAA '15: Proceedings of the 2015 41st Euromicro Conference on Software Engineering and Advanced Applications

Context: Cross-project defect prediction (CPDP) research has been popular and many CPDP methods were proposed. While these methods used cross-project data as is for their inputs, useless or noisy information in the cross-project data can cause the ...
Recalling the "imprecision" of cross-project defect prediction
FSE '12: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering

There has been a great deal of interest in defect prediction: using prediction models trained on historical data to help focus quality-control resources in ongoing development. Since most new projects don't have historical data, there is interest in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Human-Computer Interaction

Human-Computer Interaction Volume 8, Issue 3

September 1993

117 pages

ISSN:0737-0024

EISSN:1532-7051

Issue’s Table of Contents

Publisher

L. Erlbaum Associates Inc.

United States

Publication History

Published: 01 September 1993

Revised: 05 May 1993

Received: 06 September 1991

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

101
Total Citations
View Citations
228
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khan SDevlen CManno MHou D(2024)Mouse Dynamics Behavioral Biometrics: A SurveyACM Computing Surveys10.1145/364031156:6(1-33)Online publication date: 24-Jan-2024
https://dl.acm.org/doi/10.1145/3640311
Inga JRuess MRobens JNelius TRothfuß SKille SDahlinger PLindenmann AThomaschke RNeumann GMatthiesen SHohmann SKiesel A(2023)Human-machine symbiosisInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2022.102926170:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2022.102926
Yuan HLi SRusconi P(2021)CogTool+ACM Transactions on Computer-Human Interaction10.1145/344753428:2(1-38)Online publication date: 17-Apr-2021
https://dl.acm.org/doi/10.1145/3447534
Li NHuang JFeng Y(2020)Human performance modeling and its uncertainty factors affecting decision making: a surverySoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-019-04659-z24:4(2851-2871)Online publication date: 1-Feb-2020
https://dl.acm.org/doi/10.1007/s00500-019-04659-z
Ritter FLarkin J(2018)Developing process models as summaries of HCI action sequencesHuman-Computer Interaction10.1207/s15327051hci0903%264_49:4(345-383)Online publication date: 26-Dec-2018
https://dl.acm.org/doi/10.1207/s15327051hci0903%25264_4
West RWard LDudzik KNagy NKarimi F(2018)Micro and Macro Predictions: Using SGOMS to Predict Phone App Game Playing and Emergency Operations Centre ResponsesEngineering Psychology and Cognitive Ergonomics10.1007/978-3-319-91122-9_41(501-519)Online publication date: 15-Jul-2018
https://dl.acm.org/doi/10.1007/978-3-319-91122-9_41
Yuan HLi SRusconi PAljaffan N(2017)When Eye-Tracking Meets Cognitive Modeling: Applications to Cyber Security SystemsHuman Aspects of Information Security, Privacy and Trust10.1007/978-3-319-58460-7_17(251-264)Online publication date: 9-Jul-2017
https://dl.acm.org/doi/10.1007/978-3-319-58460-7_17
Leu GAbbass H(2016)A multi-disciplinary review of knowledge acquisition methodsKnowledge-Based Systems10.1016/j.knosys.2016.02.012105:C(1-22)Online publication date: 1-Aug-2016
https://dl.acm.org/doi/10.1016/j.knosys.2016.02.012
Paik JKim JRitter FReitter D(2015)Predicting User Performance and Learning in Human--Computer Interaction with the Herbal CompilerACM Transactions on Computer-Human Interaction10.1145/277689122:5(1-26)Online publication date: 24-Aug-2015
https://dl.acm.org/doi/10.1145/2776891
Hua Guo Gomez SZiemkiewicz CLaidlaw D(2015)A Case Study Using Visualization Interaction Logs and Insight Metrics to Understand How Analysts Arrive at InsightsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2015.246761322:1(51-60)Online publication date: 27-Oct-2015
https://dl.acm.org/doi/10.1109/TVCG.2015.2467613
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents