research-article

Authentication Melee: A Usability Analysis of Seven Web Authentication Systems

Authors:

Scott Ruoti,

Brent Roberts,

Kent SeamonsAuthors Info & Claims

WWW '15: Proceedings of the 24th International Conference on World Wide Web

Pages 916 - 926

https://doi.org/10.1145/2736277.2741683

Published: 18 May 2015 Publication History

Get Access

Abstract

Passwords continue to dominate the authentication landscape in spite of numerous proposals to replace them. Even though usability is a key factor in replacing passwords, very few alternatives have been subjected to formal usability studies, and even fewer have been analyzed using a standard metric. We report the results of four within-subjects usability studies for seven web authentication systems. These systems span federated, smartphone, paper tokens, and email-based approaches. Our results indicate that participants prefer single sign-on systems. We report several insightful findings based on participants' qualitative responses: (1) transparency increases usability but also leads to confusion and a lack of trust, (2) participants prefer single sign-on but wish to augment it with site-specific low-entropy passwords, and (3) participants are intrigued by biometrics and phone-based authentication. We utilize the Systems Usability Scale (SUS) as a standard metric for empirical analysis and find that it produces reliable, replicable results. SUS proves to be an accurate measure of baseline usability. We recommend that new authentication systems be formally evaluated for usability using SUS, and should meet a minimum acceptable SUS score before receiving serious consideration.

References

[1]

Steam Guard. https://support.steampowered.com/kb_article.php?ref=4020-ALZM-5519. {Online; accessed 2014/11/20}.

Google Scholar

[2]

A. Bangor, P. Kortum, and J. Miller. An empirical evaluation of the System Usability Scale. International Journal of Human--Computer Interaction, 24(6):574--594, 2008.

Crossref

Google Scholar

[3]

A. Bangor, P. Kortum, and J. Miller. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3):114--123, 2009.

Digital Library

Google Scholar

[4]

A. Bianchi, I. Oakley, and D. S. Kwon. Using mobile device screens for authentication. In Proceedings of the 23rd Australian Computer-Human Interaction Conference, OzCHI '11, pages 50--53, New York, NY, USA, 2011. ACM.

Digital Library

Google Scholar

[5]

J. Bonneau, E. W. Felten, P. Mittal, and A. Narayanan. Privacy concerns of implicit secondary factors for web authentication. In SOUPS Workshop on "Who are you?!": Adventures in Authentication, 2014.

Google Scholar

[6]

J. Bonneau, C. Herley, P. C. Van Oorschot, and F. Stajano. The quest to replace passwords: A framework for comparative evaluation of web authentication schemes. In Symposium on Security and Privacy, pages 553--567. IEEE, 2012.

Digital Library

Google Scholar

[7]

J. Brooke. SUS | a quick and dirty usability scale. In Usability Evaluation in Industry. CRC Press, 1996.

Google Scholar

[8]

J. Brooke. SUS: A retrospective. Journal of Usability Studies, 8(2):29--40, 2013.

Digital Library

Google Scholar

[9]

S. Chiasson, E. Stobert, A. Forget, R. Biddle, and P. C. Van Oorschot. Persuasive cued click-points: Design, implementation, and evaluation of a knowledge-based authentication mechanism. IEEE Transactions on Dependable and Secure Computing, 9(2):222--235, 2012.

Digital Library

Google Scholar

[10]

S. Chiasson, P. C. van Oorschot, and R. Biddle. A usability study and critique of two password managers. In USENIX Security, 2006.

Digital Library

Google Scholar

[11]

R. Dhamija and A. Perrig. Deja Vu | a user study: Using images for authentication. In USENIX Security, 2000.

Digital Library

Google Scholar

[12]

B. Dodson, D. Sengupta, D. Boneh, and M. S. Lam. Secure, consumer-friendly web authentication and payments with a phone. In International Conference on Mobile Computing, Applications, and Services, pages 17--38. Springer, 2012.

Crossref

Google Scholar

[13]

S. L. Garfinkel. Email-based identification and authentication: An alternative to PKI? In Symposium on Security and Privacy, pages 20--26. IEEE, 2003.

Digital Library

Google Scholar

[14]

E. Hayashi, B. Pendleton, F. Ozenc, and J. Hong. WebTicket: Account management using printable tokens. In SIGCHI Conference on Human Factors in Computing Systems, pages 997--1006. ACM, 2012.

Digital Library

Google Scholar

[15]

R. Jhawar, P. Inglesant, N. Courtois, and M. A. Sasse. Make mine a quadruple: Strengthening the security of graphical one-time PIN authentication. In International Conference on Network and System Security, pages 81--88. IEEE, 2011.

Crossref

Google Scholar

[16]

K. A. Juang, S. Ranganayakulu, and J. S. Greenstein. Using system-generated mnemonics to improve the usability and security of password authentication. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 56, pages 506--510. SAGE Publications, 2012.

Crossref

Google Scholar

[17]

J. R. Lewis. Ibm computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1):57--78, 1995.

Digital Library

Google Scholar

[18]

C. Robison, S. Ruoti, T. W. van der Horst, and K. E. Seamons. Private facebook chat. In International Conference on Privacy, Security, Risk and Trust and International Confernece on Social Computing, pages 451--460. IEEE, 2012.

Digital Library

Google Scholar

[19]

S. Ruoti. Authentication melee: A usability analysis of seven web authentication systems. Master's thesis, Brigham Young University, 2015. http://scholarsarchive.byu.edu/etd/4376/.

Google Scholar

[20]

S. Ruoti, N. Kim, B. Burgon, T. Van Der Horst, and K. Seamons. Confused Johnny: When automatic encryption leads to confusion and mistakes. In Symposium on Usable Privacy and Security. ACM, 2013.

Digital Library

Google Scholar

[21]

F. Schaub, M. Walch, B. Konings, and M. Weber. Exploring the design space of graphical passwords on smartphones. In Symposium on Usable Privacy and Security. ACM, 2013.

Digital Library

Google Scholar

[22]

S. E. Schechter, R. Dhamija, A. Ozment, and I. Fischer. The Emperor's new security indicators. In Security and Privacy, pages 51--65. IEEE, 2007.

Digital Library

Google Scholar

[23]

S.-T. Sun, E. Pospisil, I. Muslukhov, N. Dindar, K. Hawkey, and K. Beznosov. What makes users refuse web single sign-on?: An empirical investigation of OpenID. In Symposium on Usable Privacy and Security. ACM, 2011.

Digital Library

Google Scholar

[24]

H. Tao. Pass-Go, A New Graphical Password Scheme. PhD thesis, University of Ottawa, 2006.

Google Scholar

[25]

R. Tassabehji and M. A. Kamala. Evaluating biometrics for online banking: The case for usability. International Journal of Information Management, 32(5):489--494, 2012.

Digital Library

Google Scholar

[26]

S. Trewin, C. Swart, L. Koved, J. Martino, K. Singh, and S. Ben-David. Biometric authentication on a mobile device: a study of user effort, error and task disruption. In Proceedings of the 28th Annual Computer Security Applications Conference, pages 159--168. ACM, 2012.

Digital Library

Google Scholar

[27]

T. S. Tullis and J. N. Stetson. A comparison of questionnaires for assessing website usability. Presented at Usability Professional Association Conference, 2004.

Google Scholar

[28]

T. W. van der Horst and K. E. Seamons. Simple authentication for the web. In International Conference on Security and Privacy in Communications Networks and the Workshops, pages 473--482. IEEE, 2007.

Google Scholar

[29]

D. Weinshall. Cognitive authentication schemes safe against spyware. In Symposium on Security and Privacy, pages 295--300. IEEE, 2006.

Digital Library

Google Scholar

Cited By

View all

Rivera-Dourado MGestal MPazos AVázquez-Naya J(2024)A Novel Protocol Using Captive Portals for FIDO2 Network AuthenticationApplied Sciences10.3390/app1409361014:9(3610)Online publication date: 24-Apr-2024
https://doi.org/10.3390/app14093610
Leguesse YVella MColombo CHernandez-Castro J(2024)A Metric to Assess the Reliability of Crowd-sourced SUS Scores: A Case Study on the PoPLar Authentication ToolProceedings of the 2024 European Symposium on Usable Security10.1145/3688459.3688470(309-321)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3688459.3688470
Kruzikova AMuzik MKnapova LDedkova LSmahel DMatyas V(2024)Two-factor authentication timeComputers and Security10.1016/j.cose.2023.103667138:COnline publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1016/j.cose.2023.103667
Show More Cited By

Index Terms

Authentication Melee: A Usability Analysis of Seven Web Authentication Systems
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
2. Security and privacy
  1. Security services
    1. Authentication

Recommendations

Patient Preferences for Authentication and Security: A Comparison Study of Younger and Older Patients
SIGMIS-CPR'18: Proceedings of the 2018 ACM SIGMIS Conference on Computers and People Research

We examine authentication and security preferences of younger versus older patients in the healthcare domain. Previous research has investigated users' perception of the acceptability of various forms of authentication in non-healthcare domains, but not ...
A Secure and Efficient Deniable Authentication Protocol
ICIE '09: Proceedings of the 2009 WASE International Conference on Information Engineering - Volume 02

A deniable authentication can be used to provide secure negotiation on the Internet. Although many deniable authentication protocols have been proposed, most of them are interactive or vulnerable to various cryptanalytic attacks. To find a secure and ...
A usability evaluation of the Google Home with non-native English speakers using the system usability scale

Advancements in multidisciplinary research have made voice user interfaces (VUIs) become a reality in both research and commercialisation. Although the existing research has indicated that VUIs are useful in people's daily lives (e.g., personal assistant),...

Reviews

Reviewer: Andre C. M. Marien

Authentication systems must be strong enough to ensure confidentiality, accountability, and integrity for a reasonable cost-and be user friendly. Good, publicly available guidance does exist; for instance, see [1]. Usability, on the other hand, is mostly treated informally. The authors report that only four of the 23 publications report the results of an empirical usability study, and none of the systems are analyzed using a standard usability metric. This paper uses a formal approach, systems usability scale (SUS), to assess the usability of seven different web authentication systems. It provides reliable, replicable results. The seven web authentication systems are split into three groups: federated single sign-on, email-based, and QR code-based. For each of the groups, the authors conduct a separate usability study, and the system with the highest SUS score in each study is selected as a winner. The three winners are then compared again. Of these seven systems, federated and smartphone-based single sign-on receive the best overall usability ratings. The authors also collected feedback and proposals. These provide more insight into what makes an authentication system usable. Note that the study was executed with young people with a medium level of IT skills, somewhat more males than females, in a lab setting. Single sign-on (SSO) is preferred, but there are security concerns related to the SSO provider. Combining it with low-entropy passwords per site is suggested. They also like transparency, but too much raises suspicion: is this really secure__?__ Both of these findings indicate that we may underestimate the users' willingness to assist and be involved with secure access. New, more innovative authentication mechanisms were found attractive, and biometric mechanisms (not in the test) were suggested. Given the age of most participants in the test, this might not be generally so. The mean time to authenticate did not seem to play a role, whereas this is a common measure currently. It may be because it was a lab test, not one with people dealing with numerous systems each day. When using a physical token (that is, a WebTicket or smartphone), participants want to have a fallback mechanism. This concern is well known. I can only welcome more scientific approaches in security for a key pillar: the user. SUS seems to be fit for the purpose. The significant improvement of one of the systems based on the analysis should inspire authentication product creators. The classic idea of users just wanting to get the job done and not caring about the authentication seems to be a mistake: less hiding and more explaining may be a new path to consider. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

WWW '15: Proceedings of the 24th International Conference on World Wide Web

May 2015

1460 pages

ISBN:9781450334693

General Chairs:
Aldo Gangemi
National Research Council, Italy & Paris 13 University-CNRS, France
,
Stefano Leonardi
Sapienza University of Rome, Italy
,
Alessandro Panconesi
Sapienza University of Rome, Italy

In-Cooperation

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

International World Wide Web Conferences Steering Committee

Republic and Canton of Geneva, Switzerland

Publication History

Published: 18 May 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WWW '15

Sponsor:

IW3C2

WWW '15: 24th International World Wide Web Conference

May 18 - 22, 2015

Florence, Italy

Acceptance Rates

WWW '15 Paper Acceptance Rate 131 of 929 submissions, 14%;

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
1,010
Total Downloads

Downloads (Last 12 months)132
Downloads (Last 6 weeks)18

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Rivera-Dourado MGestal MPazos AVázquez-Naya J(2024)A Novel Protocol Using Captive Portals for FIDO2 Network AuthenticationApplied Sciences10.3390/app1409361014:9(3610)Online publication date: 24-Apr-2024
https://doi.org/10.3390/app14093610
Leguesse YVella MColombo CHernandez-Castro J(2024)A Metric to Assess the Reliability of Crowd-sourced SUS Scores: A Case Study on the PoPLar Authentication ToolProceedings of the 2024 European Symposium on Usable Security10.1145/3688459.3688470(309-321)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3688459.3688470
Kruzikova AMuzik MKnapova LDedkova LSmahel DMatyas V(2024)Two-factor authentication timeComputers and Security10.1016/j.cose.2023.103667138:COnline publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1016/j.cose.2023.103667
Petrie HSreekumar GShahandashti S(2024)Understanding Users’ Mental Models of Federated Identity Management (FIM): Use of a New Tangible Elicitation MethodHuman Aspects of Information Security and Assurance10.1007/978-3-031-72559-3_21(308-322)Online publication date: 28-Nov-2024
https://doi.org/10.1007/978-3-031-72559-3_21
Smith GYadav TDutson JRuoti SSeamons KCalandrino JTroncoso C(2023)"If i could do this, i feel anyone could:"Proceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620266(499-516)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620266
Gardner DTanenbaum T(2023)The access control double bind: How everyday interfaces regulate access and privacy, enable surveillance, and enforce identityConvergence: The International Journal of Research into New Media Technologies10.1177/1354856523119370630:3(1186-1218)Online publication date: 19-Aug-2023
https://doi.org/10.1177/13548565231193706
Röse MKablo EArias-Cabarcos P(2023)Overcoming Theory: Designing Brainwave Authentication for the Real WorldProceedings of the 2023 European Symposium on Usable Security10.1145/3617072.3617120(175-191)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3617072.3617120
Arias-Cabarcos PFallahi MHabrich TSchulze KBecker CStrufe T(2023)Performance and Usability Evaluation of Brainwave Authentication Techniques with Consumer DevicesACM Transactions on Privacy and Security10.1145/3579356Online publication date: 18-Jan-2023
https://doi.org/10.1145/3579356
Pöhn DGruschka NZiegler LBüttner A(2023)A framework for analyzing authentication risks in account networksComputers and Security10.1016/j.cose.2023.103515135:COnline publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1016/j.cose.2023.103515
Kruzikova AMatyas VBroz M(2023)Authentication of IT Professionals in the Wild – A SurveySecurity Protocols XXVIII10.1007/978-3-031-43033-6_5(43-56)Online publication date: 21-Oct-2023
https://doi.org/10.1007/978-3-031-43033-6_5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Patient Preferences for Authentication and Security: A Comparison Study of Younger and Older Patients

A Secure and Efficient Deniable Authentication Protocol

A usability evaluation of the Google Home with non-native English speakers using the system usability scale

Reviews

Access critical reviews of Computing literature here