More Web Proxy on the site http://driver.im/

research-article

Open access

HaTS: large-scale in-product measurement of user attitudes & experiences with <u>ha</u>ppiness <u>t</u>racking <u>s</u>urveys

Authors:

Hendrik Müller,

Aaron SedleyAuthors Info & Claims

OzCHI '14: Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design

Pages 308 - 315

https://doi.org/10.1145/2686612.2686656

Published: 02 December 2014 Publication History

Abstract

With the rise of Web-based applications, it is both important and feasible for human-computer interaction practitioners to measure a product's user experience. While quantifying user attitudes at a small scale has been heavily studied, in this industry case study, we detail best Happiness Tracking Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time. This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products' user bases. This case study of HaTS goes beyond the design of the questionnaire to also suggest best practices for appropriate sampling, invitation techniques, and its data analysis. HaTS has been deployed successfully across dozens of Google's products to measure progress towards product goals and to inform product decisions; its sensitivity to product changes has been demonstrated widely. We are confident that teams in other organizations will be able to embrace HaTS as well, and, if necessary, adapt it for their unique needs.

References

[1]

J. Brooke. SUS: A quick and dirty usability scale. Usability evaluation in industry, 189:194, 1996.

[2]

J. P. Chin, V. A. Diehl, and K. L. Norman. Development of an instrument measuring user satisfaction of the human-computer interface. In Proceedings of the SIGCHI, CHI '88, pages 213--218. ACM, 1988.

Digital Library

[3]

M. Couper. Designing effective web surveys. Cambridge University Press Cambridge, 2008.

Digital Library

[4]

D. B. Grisaffe. Questions about the ultimate question: Conceptual considerations in evaluating reichheld's net promoter score (NPS). In Journal of Consumer Satisfaction, Dissatisfaction, and Complaining Behavior: CS/D&CB, pages 36--53, 2007.

[5]

R. M. Groves, E. Singer, J. M. Lepkowski, S. G. Heeringa, and D. F. Alwin. Survey methodology. The University of Michigan Press, 2004.

[6]

M. Hassenzahl, M. Burmester, and F. Koller. Attrakdiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In Mensch & Computer 2003, pages 187--196. Springer, 2003.

[7]

J. Kirakowski and M. Corbett. SUMI: The software usability measurement inventory. British journal of educational technology, 24(3): 210--212, 1993.

[8]

J. Kirakowski and A. Dillion. The computer user satisfaction inventory. Proceedings from the IEE: Evaluation Techniques for Interactive System Design, 1987.

[9]

J. A. Krosnick. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5: 213--236, 1991.

[10]

J. A. Krosnick. Survey research. Annual review of psychology, 50(1): 537--567, 1999.

[11]

J. A. Krosnick. The causes of no-opinion responses to attitude measures in surveys: They are rarely what they appear to be. Survey nonresponse, pages 87--100, 2002.

[12]

J. A. Krosnick and D. F. Alwin. A test of the form-resistant correlation hypothesis ratings, rankings, and the measurement of values. Public Opinion Quarterly, 52(4): 526--538, 1988.

[13]

J. A. Krosnick and L. R. Fabrigar. Designing rating scales for effective measurement in surveys. Survey measurement and process quality, pages 141--164, 1997.

[14]

J. A. Krosnick, S. Narayan, and W. R. Smith. Satisficing in surveys: Initial evidence. New Directions for Evaluation, 1996(70): 29--44, 1996.

[15]

J. A. Krosnick and S. Presser. Question and questionnaire design. Handbook of Survey Research. 2nd edition. Bingley, UK: Emerald, pages 263--314, 2010.

[16]

J. A. Krosnick and A. M. Tahk. The optimal length of rating scales to maximize reliability and validity. Unpublished manuscript, Stanford University, 2008.

[17]

E. L. Landon. Order bias, the ideal rating, and the semantic differential. Journal of Marketing Research, 8(3): 375--378, 1971.

[18]

R. Larson and M. Csikszentmihalyi. The experience sampling method. In H. T. Reis, editor, Naturalistic Approaches to Studying Social Interaction, volume 15 of New Directions for Methodology of Social and Behavioral Science, pages 41--56. Jossey-Bass, San Francisco, CA, USA, 1983.

[19]

J. R. Lewis. Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ. SIGCHI Bulletin, 23(1): 78--81, 1991.

Digital Library

[20]

J. R. Lewis. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int. J. Hum.-Comput. Interact., 7(1): 57--78, 1995.

Digital Library

[21]

H. Müller, A. Sedley, and E. Ferrall-Nunge. Survey research in HCI. In J. Olson and W. Kellogg, editors, Ways of Knowing in HCI, pages 229--266. Springer, New York, NY, USA, 2014.

[22]

F. F. Reichheld. The one number you need to grow. Harvard Business Review, 2003.

[23]

K. Rodden, H. Hutchinson, and X. Fu. Measuring the user experience on a large scale: User-centered metrics for web applications. In Proceedings of the SIGCHI, CHI '10, pages 2395--2398. ACM, 2010.

Digital Library

[24]

W. E. Saris, J. A. Krosnick, and E. M. Shaeffer. Comparing questions with agree/disagree response options to questions with construct-specific response options. Unpublished manuscript, University of Amsterdam, 2005.

[25]

N. C. Schaeffer and S. Presser. The science of asking questions. Annual Review of Sociology, 29: 65--88, 2003.

[26]

B. R. Schlenker and M. F. Weigold. Goals and the self-identification process: Constructing desired identities. Goal concepts in personality and social psychology, pages 243--290, 1989.

[27]

A. Sedley and H. Müller. Minimizing change aversion for the Google Drive launch. In CHI '13 Extended Abstracts, CHI EA '13, pages 2351--2354. ACM, 2013.

Digital Library

[28]

D. H. Smith. Correcting for social desirability response sets in opinion-attitude survey research. The Public Opinion Quarterly, 31(1): 87--94, 1967.

[29]

R. Tourangeau. Cognitive science and survey methods, volume 73. National Academy Press Washington, 1984.

[30]

R. Tourangeau, M. P. Couper, and F. Conrad. Spacing, position, and order interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3): 368--393, 2004.

Cited By

Pielot MCallegaro M(2024)Did You Misclick? Reversing 5-Point Satisfaction Scales Causes Unintended ResponsesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642397(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642397
St. Thomas BChandar PHosey CDiaz FZhu FChin Ooi BMiao CWang HSkrypnyk IHsu WChawla S(2021)Mixed Method Development of Evaluation MetricsProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining10.1145/3447548.3470802(4070-4071)Online publication date: 14-Aug-2021
https://dl.acm.org/doi/10.1145/3447548.3470802
Yildiz ESafyan JHarper MGupta RLiu YShah MRajan STang JPrakash B(2020)User Sentiment as a Success MetricProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3394486.3403340(2891-2899)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1145/3394486.3403340
Show More Cited By

Index Terms

HaTS: large-scale in-product measurement of user attitudes & experiences with happiness tracking surveys
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Becoming experts: measuring attitude development in introductory computer science
SIGCSE '13: Proceeding of the 44th ACM technical symposium on Computer science education

We have begun the process of examining how students perceive the field of computer science by employing a novice-to-expert continuum framework. As part of this exploration we have developed and are validating the Computing Attitudes Survey (CAS). In ...
Attitudes of individuals with visual impairments towards distance education

The aims of this work were to examine the attitudes of individuals with visual impairments towards distance education (DE) and the relationships between attitudes and participants' personal characteristics. Forty-one adults with visual impairments, who ...
Toward a validated computing attitudes survey
ICER '12: Proceedings of the ninth annual international conference on International computing education research

The Computing Attitudes Survey (CAS) is a newly designed instrument, adapted from the Colorado Learning Attitudes about Science Survey (CLASS), for measuring novice to expert-like perceptions about computer science. In this paper we outline the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

OzCHI '14: Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design

December 2014

689 pages

ISBN:9781450306539

DOI:10.1145/2686612

Conference Chair:
Tuck Leong
University of Technology Sydney

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

U1: U1 PTY LTD
IDHuP: Interaction Design and Human Practice Lab
UTS-HCTDRS: The UTS Human Centred Technology Design Research Strength
CSIRO
QUT
HFESA: Human Factors and Ergonomics Society of Australia Inc.
University of Technology Sydney
IDF: The Interaction Design Foundation
CHISIG: Computer-Human Interaction Special Interest Group, Human Factors & Ergonomics Society of Australia

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 December 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

OzCHI '14

Sponsor:

U1
IDHuP
UTS-HCTDRS
HFESA
IDF
CHISIG

OzCHI '14: the Future of Design

December 2 - 5, 2014

New South Wales, Sydney, Australia

Acceptance Rates

OzCHI '14 Paper Acceptance Rate 85 of 176 submissions, 48%;

Overall Acceptance Rate 362 of 729 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
1,413
Total Downloads

Downloads (Last 12 months)482
Downloads (Last 6 weeks)30

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pielot MCallegaro M(2024)Did You Misclick? Reversing 5-Point Satisfaction Scales Causes Unintended ResponsesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642397(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642397
St. Thomas BChandar PHosey CDiaz FZhu FChin Ooi BMiao CWang HSkrypnyk IHsu WChawla S(2021)Mixed Method Development of Evaluation MetricsProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining10.1145/3447548.3470802(4070-4071)Online publication date: 14-Aug-2021
https://dl.acm.org/doi/10.1145/3447548.3470802
Yildiz ESafyan JHarper MGupta RLiu YShah MRajan STang JPrakash B(2020)User Sentiment as a Success MetricProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3394486.3403340(2891-2899)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1145/3394486.3403340
Kohavi RTang DXu Y(2020)Trustworthy Online Controlled Experiments10.1017/9781108653985Online publication date: 13-Mar-2020
https://doi.org/10.1017/9781108653985
Garcia-Gathright JSt. Thomas BHosey CNazari ZDiaz FCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Understanding and Evaluating User Satisfaction with Music DiscoveryThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210049(55-64)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210049
Callegaro MYang Y(2017)The Role of Surveys in the Era of “Big Data”The Palgrave Handbook of Survey Research10.1007/978-3-319-54395-6_23(175-192)Online publication date: 23-Oct-2017
https://doi.org/10.1007/978-3-319-54395-6_23

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents