CogTool+: Modeling Human Performance at Large Scale
Article No.: 15, Pages 1 - 38
Abstract
Cognitive modeling tools have been widely used by researchers and practitioners to help design, evaluate, and study computer user interfaces (UIs). Despite their usefulness, large-scale modeling tasks can still be very challenging due to the amount of manual work needed. To address this scalability challenge, we propose CogTool+, a new cognitive modeling software framework developed on top of the well-known software tool CogTool. CogTool+ addresses the scalability problem by supporting the following key features: (1) a higher level of parameterization and automation; (2) algorithmic components; (3) interfaces for using external data; and (4) a clear separation of tasks, which allows programmers and psychologists to define reusable components (e.g., algorithmic modules and behavioral templates) that can be used by UI/UX researchers and designers without the need to understand the low-level implementation details of such components. CogTool+ also supports mixed cognitive models required for many large-scale modeling tasks and provides an offline analyzer of simulation results. In order to show how CogTool+ can reduce the human effort required for large-scale modeling, we illustrate how it works using a pedagogical example, and demonstrate its actual performance by applying it to large-scale modeling tasks of two real-world user-authentication systems.
References
[1]
ACT-R Research Group, Department of Psychology, Carnegie Mellon University. [n.d.]. ACT-R. Retrieved from http://act-r.psy.cmu.edu/.
[2]
John R. Anderson. 2007. How Can the Human Mind Occur in the Physical Universe?Oxford University Press.
[3]
John R. Anderson, Daniel Bothell, Michael D. Byrne, Scott Douglass, Christian Lebiere, and Yulin Qin. 2004. An integrated theory of the mind. Psychological Review 111, 4 (2004), 1036–1060.
[4]
Rachel Bellamy, Bonnie John, John Richards, and John Thomas. 2010. Using CogTool to model programming tasks. In Proceedings of the 2010 ACM SIGPLAN Workshop on Evaluation and Usability of Programming Languages and Tools. ACM, Article 1.
[5]
Rachel Bellamy, Bonnie E. John, and Sandra L. Kogan. 2011. Deploying CogTool: Integrating quantitative usability assessment into real-world software development. In Proceedings of 2011 33rd International Conference on Software Engineering. IEEE, 691–700.
[6]
Michael D. Byrne. 2001. ACT-R/PM and menu selection: Applying a cognitive architecture to HCI. International Journal of Human-Computer Studies 55, 1 (2001), 41–84.
[7]
Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. The keystroke-level model for user performance time with interactive systems. Communications of the ACM 23, 7 (1980), 396–410.
[8]
Stuart K. Card, Allen Newell, and Thomas P. Moran. 1983. The Psychology of Human-Computer Interaction. L. Erlbaum Associates Inc.
[9]
Gray Wayne D., John Bonnie E., and Atwood Michael E.1993. Project Ernestine: Validating a GOMS analysis for predicting and explaining real-world task performance. Human Computer Interaction 8, 3 (1993), 237–309.
[10]
Alexander De Luca, Katja Hertzschuch, and Heinrich Hussmann. 2010. ColorPIN: Securing PIN entry through indirect input. In Proceedings of the 2010 SIGCHI Conference on Human Factors in Computing Systems. ACM, 1103–1106.
[11]
Sebastian Feuerstack and Bertram Wortelen. 2015. Revealing differences in designers’ and users’ perspectives: A tool-supported process for visual attention prediction for designing HMIs for maritime monitoring tasks. In Proceedings of the 15th IFIP TC 13 International Conference on Human-Computer Interaction.Lecture Notes in Computer Science, Vol. 9299, Springer, 105–122.
[12]
Paul M. Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47, 6 (1954), 381–391.
[13]
Daniel Gartenberg, Ross Thornton, Mortazavi Masood, Dustin Pfannenstiel, Daniel Taylor, and Raja Parasuraman. 2013. Collecting health-related data on the smart phone: Mental models, cost of collection, and perceived benefit of feedback. Personal and Ubiquitous Computing 17, 3 (2013), 561–570.
[14]
Bonnie E. John. [n.d.]. CogTool | Cognitive Crash Dummies: Predictive human performance modeling for UI design. Retrieved from https://cogtool.wordpress.com/.
[15]
Bonnie E. John and David E. Kieras. 1996. The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Transactions on Computer-Human Interaction 3, 4 (1996), 320–351.
[16]
Bonnie E. John, Konstantine Prevas, Dario D. Salvucci, and Ken Koedinger. 2004. Predictive human performance modeling made easy. In Proceedings of the 2004 SIGCHI Conference on Human Factors in Computing Systems. ACM, 455–462.
[17]
Siwan Kim, Hyunyi Yi, and Jyun Yi. 2014. FakePIN: Dummy key based mobile user authentication scheme. In Ubiquitous Information Technologies and Applications: CUTE 2013 (Lecture Notes in Electrical Engineering), Vol. 280. Springer, 157–164.
[18]
John E. Laird. 2012. The Soar Cognitive Architecture. MIT Press.
[19]
Ximing Liu, Yingjiu Li, Robert H. Deng, Bing Chang, and Shujun Li. 2019. When human cognitive modeling meets PINs: User-independent inter-keystroke timing attacks. Computers & Security 80 (2019), 90–107.
[20]
Lu Luo and Bonnie E. John. 2005. Predicting task execution time on handheld devices using the keystroke-level model. In Proceedings of the Extended Abstracts on Human Factors in Computing Systems. ACM, 1605–1608.
[21]
Bao N. Nguyen, Bryan Robbins, Ishan Banerjee, and Atif Memon. 2014. GUITAR: An innovative tool for automated testing of GUI-driven software. Automated Software Engineering 21, 1 (2014), 65–105.
[22]
Object Refinery Limited. [n.d.]. JFreeChart. Retrieved from http://www.jfree.org/jfreechart/.
[23]
Nihan Ocak and Kursat Cagiltay. 2017. Comparison of cognitive modeling and user performance analysis for touch screen mobile interface design. International Journal of Human-Computer Interaction 33, 8 (2017), 633–641.
[24]
Jaehyon Paik, Jong W. Kim, Frank E. Ritter, and David Reitter. 2015. Predicting user performance and learning in human-computer interaction with the herbal compiler. ACM Transactions on Computer-Human Interaction 22, 5 (2015), 1–25.
[25]
Evan W. Patton and Wayne D. Gray. 2010. SANLab-CM: A tool for incorporating stochastic operations into activity network modeling. Behavior Research Methods 42, 3 (2010), 877–883. available at https://github.com/CogWorks/SANLab-CM.
[26]
Toni Perković, Shujun Li, Asma Mumtaz, Syed Ali Khayam, Yousra Javed, and Mario Čagalj. 2011. Breaking undercover: Exploiting design flaws and nonuniform human behavior. In Proceedings of the 7th Symposium on Usable Privacy and Security. ACM, Article 5.
[27]
Casey Reas and Ben Fry. 2006. Processing: Programming for the media arts. AI & Society: Knowledge, Culture and Communication 20, 4 (2006), 526–538.
[28]
Volker Roth, Kai Richter, and Rene Freidinger. 2004. A PIN-entry method resilient against shoulder surfing. In Proceedings of the 11th ACM Conference on Computer and Communications Security. ACM, 236–245.
[29]
Dario D. Salvucci. 2001. Predicting the effects of in-car interfaces on driver behavior using a cognitive architecture. In Proceedings of the 2001 SIGCHI Conference on Human Factors in Computing Systems. ACM, 120–127.
[30]
Hirokazu Sasamoto, Nicolas Christin, and Eiji Hayashi. 2008. Undercover: Authentication usable in front of prying eyes. In Proceedings of the 2008 SIGCHI Conference on Human Factors in Computing Systems. ACM, 183–192.
[31]
M. Angela Sasse, Michelle Steves, Kat Krol, and Dana Chisnell. 2014. The great authentication fatigue – and how to overcome it. In Proceedings of the 6th International Conference on Cross-Cultural Design.Lecture Notes in Computer Science, Vol. 8528, Springer, 228–239.
[32]
Anil Shankar, Honray Lin, Hans-Frederick Brown, and Colson Rice. 2015. Rapid usability assessment of an enterprise application in an agile environment with CogTool. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 719–726.
[33]
Soar Research Groups, University of Michigan. [n.d.]. Home - Soar Cognitive Architecture. Retrieved from http://soar.eecs.umich.edu/.
[34]
Amanda Swearngin, Myra Cohen, Bonnie John, and Rachel Bellamy. 2012. Easing the generation of predictive human performance models from legacy systems. In Proceedings of the 2012 SIGCHI Conference on Human Factors in Computing Systems. ACM, 2489–2498.
[35]
Amanda Swearngin, Myra B. Cohen, Bonnie E. John, and Rachel K.E. Bellamy. 2013. Human performance regression testing. In Proceedings of the 2013 International Conference on Software Engineering. IEEE, 152--161.
[36]
Leonghwee Teo and Bonnie E. John. 2008. CogTool-explorer: Towards a tool for predicting user interaction. In Proceedings of the Extended Abstracts on Human Factors in Computing Systems. ACM, 2793--2798.
[37]
The MITRE Corporation. [n.d.]. Cogulator: A Cognitive Calculator. Retrieved from http://cogulator.io/.
[38]
Anne Treisman and Janet Souther. 1985. Search asymmetry: A diagnostic for preattentive processing of separable features. Journal of Experimental Psychology: General 114, 3 (Sept. 1985), 285--310.
[39]
Emanuel von Zezschwitz, Alexander De Luca, Bruno Brunkow, and Heinrich Hussmann. 2015. SwiPIN: Fast and secure PIN-entry on smartphones. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1403--1406.
[40]
Wikimedia Foundation, Inc.[n.d.]. Lisp (programming language). Retrieved from https://en.wikipedia.org/wiki/Lisp_(programming_language).
[41]
Jeremy M. Wolfe. 2001. Asymmetries in visual search: An introduction. Perception & Psychophysics 63, 3 (Apr. 2001), 381--389.
[42]
Geoffrey F. Woodman and Marvin M. Chun. 2006. The role of working memory and long-term memory in visual search. Visual Cognition 14, 4-8 (2006), 808--830.
[43]
Geoffrey F. Woodman and Steven J. Luck. 2004. Visual search is slowed when visuospatial working memory is occupied. Psychonomic Bulletin & Review 11, 2 (2004), 269--274.
[44]
Haiyue Yuan, Shujun Li, Patrice Rusconi, and Nouf Aljaffan. 2017. When eye-tracking meets cognitive modeling: Applications to cyber security systems. In Proceedings of the 5th International Conference on Human Aspects of Information Security, Privacy and Trust.Lecture Notes in Computer Science, Vol. 10292, Springer, 251--264.
Index Terms
- CogTool+: Modeling Human Performance at Large Scale
Recommendations
When Eye-Tracking Meets Cognitive Modeling: Applications to Cyber Security Systems
Human Aspects of Information Security, Privacy and TrustAbstractHuman cognitive modeling techniques and related software tools have been widely used by researchers and practitioners to evaluate the effectiveness of user interface (UI) designs and related human performance. However, they are rarely used in the ...
Rapid Usability Assessment of an Enterprise Application in an Agile Environment with CogTool
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing SystemsLab based user testing with participants quickly becomes a bottleneck for UX teams in the industry that exist in an Agile software development environment characterized by frequent release cycles and continually changing requirements. For such teams to ...
Research of Human-Machine Interface Evaluation Based on Cogtool
Human Interface and the Management of Information. Visual Information and Knowledge ManagementAbstractAlong with the development of interactive interface design technology, interface display becomes increasingly dynamic, multi-functional and logically-complex that increases the difficulty of interactive interface evaluation. The traditional ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2021
264 pages
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 April 2021
Accepted: 01 January 2021
Revised: 01 December 2020
Received: 01 December 2018
Published in TOCHI Volume 28, Issue 2
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Funding Sources
- Engineering and Physical Sciences Research Council (EPSRC)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 1,264Total Downloads
- Downloads (Last 12 months)265
- Downloads (Last 6 weeks)51
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in