Looking Back: Retrospective Study Methods for HCI

• Akers, D., Simpson, M., Jeffries, R., & Winograd, T. (2009, April). Undo and erase events as indicators of usability problems. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (pp. 659–668). New York, NY: ACM.

  Google Scholar 

• Al-Qaimari, G., & McRostie, D. (1999). KALDI: A computer-aided usability engineering tool for supporting testing and analysis of human computer interaction. In J. Vanderdonckt & A. Puerta (Eds.), Proceedings of the 3rd International Conference on Computer-Aided Design of User Interfaces (CADUI‘99), Dordrecht, October, 1999. Louvain-la-Neuve: Kluwer.

  Google Scholar 

• Baauw, E., & Markopoulous, P. (2004, June). A comparison of think-aloud and post-task interview for usability testing with children. In Proceedings of the 2004 Conference on Interaction Design and Children: Building a Community (pp. 115–116). New York, NY: ACM.

  Google Scholar 

• Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311.

  Article  Google Scholar 

• Blackwell, A., Jones, R., Milic-Frayling, N., & Rodden, K. (2005, April). Combining logging with interviews to investigate web browser usage in the workplace. Position paper for Workshop Usage Analysis: Combining Logging and Qualitative Methods, ACM Conference on Human Factors in Computing Systems (ACM CHI 2005).

  Google Scholar 

• Brandt, J., Weiss, H., & Klemmer, S. (2007). txt 4l8r: Lowering the burden for diary studies under mobile conditions. In CHI ‘07, April 23–May 3, 2007, San Jose, California.

  Google Scholar 

• Brewer, W. F. (1986). What is autobiographical memory? In D. Rubin (Ed.), Autobiographical memory (pp. 25–49). Cambridge, UK: Cambridge University Press.

  Chapter  Google Scholar 

• Brewer, W. F. (1988). Qualitative analysis of the recalls of randomly sampled autobiographical events. In M. M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory: Current research and issues (Vol. 1, pp. 263–268). Chichester, UK: Wiley.

  Google Scholar 

• Brush, A. J., Ames, M., & Davis, J. (2004, April). A comparison of synchronous remote and local usability studies for an expert interface. In CHI‘04 Extended Abstracts on Human Factors in Computing Systems (pp. 1179–1182). New York, NY: ACM.

  Google Scholar 

• Capra, M. (2002). Contemporaneous versus retrospective user-reported critical incidents in usability evaluation. In Proc. Human Factors 2002, HFES, 1973–1977.

  Google Scholar 

• Chi, E. H., Pirolli, P., & Pitkow, J. (2000). The scent of a site: A system for analyzing and predicting information scent, usage, and usability of a web site. In Proc. CHI 2000 (pp. 161–167). New York, NY: ACM Press.

  Google Scholar 

• Collier, J. (1967). Visual anthropology: Photography as a research method. New York, NY: Holt, Rinehart and Winston.

  Google Scholar 

• Czerwinski, M., Horvitz, E., & Wilhite, S. (2004). A diary study of task switching and interruptions. In CHI (pp. 175–182). New York, NY: ACM Press.

  Google Scholar 

• Dickson, J., McLennan, J., & Omodei, M. M. (2000). Effects of concurrent verbalization on a time-critical, dynamic decision-making task. The Journal of General Psychology, 127(2), 217–228.

  Article  Google Scholar 

• Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. In K. A. Ericsson, N. Charness, P. Feltovich, & R. R. Hoffman (Eds.), Cambridge handbook of expertise and expert performance (pp. 223–242). Cambridge, UK: Cambridge University Press.

  Chapter  Google Scholar 

• Ericsson, K. A., & Simon, H. A. (1985). Protocol analysis. Cambridge, MA: MIT Press.

  Google Scholar 

• Guan, Z., Lee, S., Cuddihy, E., & Ramey, J. (2006). The validity of the stimulated retrospective think-aloud method as measured by eye tracking. In Proc. CHI 2006 (pp. 1253–1262). New York, NY: ACM Press.

  Google Scholar 

• Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Experience sampling method: Measuring the quality of everyday life. Thousand Oaks, CA: Sage Publications.

  Google Scholar 

• Holbrook, A. L., Green, M. C., & Krosnick, J. A. (2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability response bias. Public Opinion Quarterly, 67(1), 79–125.

  Article  Google Scholar 

• Höysniemi, J., Hämäläinen, P., & Turkki, L. (2003). Using peer tutoring in evaluating the usability of a physically interactive computer game with children. Interacting with Computers, 15(2), 203–225.

  Article  Google Scholar 

• Hyrskykari, A., Ovaska, S., Räihä, K., Majaranta, P., & Lehtinen, M. (2008). Gaze path stimulation in retrospective think-aloud. Journal of Eye Movement Research, 2(4), 1–18.

  Google Scholar 

• Intille, S. S., Kukla, C., & Ma, X. (2002). Eliciting user preferences using image-based experience sampling and reflection. In Proc. CHI ‘02 Extended Abstracts on Human Factors in Computing Systems (pp. 738–739). New York, NY: ACM Press.

  Google Scholar 

• Ivory, M. Y., & Hearst, M. A. (2001). The state of the art in automated usability evaluation of user interfaces. ACM Computing Surveys, 33(4), 470–516.

  Article  Google Scholar 

• Jain, J., & Boyce, S. (2012). Case study: Longitudinal comparative analysis for analyzing user behavior. In Proceedings of the 2012 ACM Annual Conference Extended Abstracts on Human Factors in Computing Systems Extended Abstracts. New York, NY: ACM.

  Google Scholar 

• Jones, R., Milic-Frayling, N., Rodden, K., & Blackwell, A. (2007). Contextual method for the redesign of existing software products. International Journal of Human-Computer Interaction, 22(1–2), 81–101.

  Article  Google Scholar 

• Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A. (2004). A survey method for characterizing daily life experience: The day reconstruction method. Science, 306(5702), 1776.

  Article  Google Scholar 

• Karapanos, E., Martens, J.-B., & Hassenzahl, M. (2009). Reconstructing experiences through sketching. Arxiv preprint, arXiv:0912.5343.

  Google Scholar 

• Karapanos, E., Martens, J., & Hassenzahl, M. (2010). On the retrospective assessment of users’ experiences over time: Memory or actuality?. In Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing systems. New York, NY: ACM.

  Google Scholar 

• Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. B. (2009). User experience over time: An initial framework. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (pp. 729–738). New York, NY: ACM.

  Google Scholar 

• Kellar, M., Watters, C., & Shepherd, M. (2006). A goal-based classification of web information tasks. In Proceedings of the Annual Meeting of the American Society for Information Science and Technology, Austin, TX (ASIS&T).

  Google Scholar 

• Kuniavsky, M. (2003). Observing the user experience: A practioner’s guide to user research. New York, NY: Morgan Kaufman.

  Google Scholar 

• Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113(3), 387–404.

  Article  Google Scholar 

• Lamming, M., Brown, P., Carter, K., Eldridge, M., Flynn, M., Louie, G., et al. (1994). The design of a human memory prosthesis. Computer Journal, 37(3), 153–163.

  Article  Google Scholar 

• Larson, R., & Csikszentmihalyi, M. (1983). The experience sampling method. In H. T. Reis (Ed.), Naturalistic approaches to studying social interaction: New directions for methodology of social and behavioral science. San Francisco, CA: Jossey-Bass.

  Google Scholar 

• Loftus, E. F. (1996). Memory distortion and false memory creation. Journal of the American Academy of Psychiatry and the Law Online, 24(3), 281–295.

  Google Scholar 

• Loftus, E. F. (2005). Planting misinformation in the human mind: A 30-year investigation of the malleability of memory. Learning & Memory, 12(4), 361–366.

  Article  Google Scholar 

• McCarney, R., Warner, J., Iliffe, S., van Haselen, R., Griffin, M., & Fisher, P. (2007). The Hawthorne effect: A randomised, controlled trial. BMC Medical Research Methodology, 7, 30.

  Article  Google Scholar 

• Memon, A., Wark, L., Holley, A., Bull, R., & Koehnken, G. (1997). Eyewitness performance in cognitive and structured interviews. Memory, 5(5), 639.

  Article  Google Scholar 

• Muralidharan, A., Gyongyi, Z., & Chi, E. (2012). Social annotations in web search. In Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems. New York, NY: ACM.

  Google Scholar 

• Norman, D. (2009). Memory is more important that actuality. Interactions, 16(2), 24–26.

  Article  MathSciNet  Google Scholar 

• Novick, D. G., Santaella, B., Cervantes, A., & Andrade, C. (2012, October). Short-term methodology for long-term usability. In Proceedings of the 30th ACM International Conference on Design of Communication (pp. 205–212). New York, NY: ACM.

  Google Scholar 

• Rieman, J. (1993). The diary study: A workplace-oriented research tool to guide laboratory efforts. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems (pp. 321–326).

  Google Scholar 

• Roediger, H. L., & McDermott, K. B. (1995). Creating false memories: Remembering words that were not presented in lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 803–814.

  Google Scholar 

• Russell, D. M., & Grimes, C. (2007). Assigned tasks are not the same as self-chosen Web search tasks. In System Sciences, 2007. HICSS 2007. 40th Annual Hawaii International Conference on (pp. 83-83). IEEE.

  Google Scholar 

• Russell, D. M., & Oren, M. (2009, January). Retrospective cued recall: A method for accurately recalling previous user behaviors. In System Sciences, 2009, HICSS‘09. 42nd Hawaii International Conference on (pp. 1–9). IEEE.

  Google Scholar 

• Schacter, D. L. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. American Psychologist, 54(3), 182–203.

  Article  Google Scholar 

• Schacter, D. L. (2001). The seven sins of memory: How the mind forgets and remembers. Boston, MA: Houghton Mifflin.

  Google Scholar 

• Schwarz, N., Kahneman, D., Xu, J., Belli, R., Stafford, F., & Alwin, D. (2009). Global and episodic reports of hedonic experience. In R. Belli, D. Alwin, & F. Stafford (Eds.), Using calendar and diary methods in life events research (pp. 157–174). Newbury Park, CA: Sage Publishing.

  Google Scholar 

• Shiffman, S., Hufford, M., Hickcox, M., Paty, J. A., Gnys, M., & Kassel, J. D. (1997). Remember that? A comparison of real-time versus retrospective recall of smoking lapses. Journal of Consulting and Clinical Psychology, 65, 292.

  Article  Google Scholar 

• Siochi, A. C., & Hid, D. (1991). A study of computer-supported user interface evaluation using maximal repeating pattern analysis. In Proceedings of ACM CHI ‘91 (pp. 301–305).

  Google Scholar 

• Steele-Johnson, D. (2000). Goal orientation and task demand effects on motivation, affect, and performance. The Journal of Applied Psychology, 85(5), 724–738.

  Article  Google Scholar 

• Teevan, J., & Karger, D. (2005). The research engine: Helping people return to information on the Web. Paper presented at the Proceedings of the ACM Symposium on User Interface Software and Technology (UIST‘05), Seattle, WA.

  Google Scholar 

• Underwood, B. J. (1965). False recognition produced by implicit verbal responses. Journal of Experimental Psychology, 70, 122–129.

  Article  Google Scholar 

• Van Boven, L., Kane, J., & McGraw, A. P. (2009). Temporally asymmetric constraints on mental simulation: Retrospection is more constrained than prospection. The handbook of imagination and mental simulation (131–147).

  Google Scholar 

• van den Haak, M., De Jong, M., & Schellens, P. J. (2003). Retrospective versus concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22, 339–351.

  Article  Google Scholar 

• Van Gog, T., Paas, F., & Van Merriënboer, J. J. G. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244.

  Google Scholar 

• Van House, N. (2006). Interview viz: Visualization-assisted photo elicitation. Ext. Abstracts CHI 2006 (pp. 1463–1468). New York, NY: ACM Press.

  Google Scholar 

• van Kesteren, I. E., Bekker, M. M., Vermeeren, A. P., & Lloyd, P. A. (2003). Assessing usability evaluation methods on their effectiveness to elicit verbal comments from children subjects. In Proceedings of the 2003 Conference on Interaction Design and Children (pp. 41–49). New York, NY: ACM.

  Google Scholar 

• Weisberg, H., Krosnick, J. A., & Bowen, B. D. (1996). An introduction to survey research, polling, and data analysis. Thousand Oaks, CA: Sage Publications.

  Google Scholar 

• Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications.

  Google Scholar 

• Wilson, B. A., Evans, J. J., Emslie, H., & Malinek, V. (1997). Evaluation of NeuroPage: A new memory aid. Journal of Neurology, Neurosurgery and Psychiatry, 63, 113–115.

  Article  Google Scholar 

• Wilson, T. D. (1994). The Proper Protocol: Validity and Completeness of Verbal Reports. Psychological Science, 5(5), 249–252.

  Google Scholar 

Download references