Characterizing Usability Issue Discussions in Open Source Software Projects
Article No.: 30, Pages 1 - 26
Abstract
Usability is a crucial factor but one of the most neglected concerns in open source software (OSS). While far from an ideal approach, a common practice that OSS communities adopt to collaboratively address usability is through discussions on issue tracking systems (ITSs). However, there is little knowledge about the extent to which OSS community members engage in usability issue discussions, the aspects of usability they frequently target, and the characteristics of their collaboration around usability issue discussions. This knowledge is important for providing practical recommendations and research directions to better support OSS communities in addressing this important topic and improve OSS usability in general. To help achieve this goal, we performed an extensive empirical study on issues discussed in five popular OSS applications: three data science notebook projects (Jupyter Lab, Google Colab, and CoCalc) and two code editor projects (VSCode and Atom). Our results indicated that while usability issues are extensively discussed in the OSS projects, their scope tended to be limited to efficiency and aesthetics. Additionally, these issues are more frequently posted by experienced community members and display distinguishable characteristics, such as involving more visual communication and more participants. Our results provide important implications that can inform the OSS practitioners to better engage the community in usability issue discussion and shed light on future research efforts toward collaboration techniques and tools for discussing niche topics in diverse communities, such as the usability issues in the OSS context.
Supplemental Material
ZIP File
This repository includes the dataset, and the data analysis code accompanying the submitted paper, titled "Characterizing Usability Issue Discussions in Open Source Software Projects". Please see README.md for details.
- Download
- 19.11 MB
References
[1]
Rabe Abdalkareem, Suhaib Mujahid, and Emad Shihab. 2020. A Machine Learning Approach to Improve the Detection of CI Skip Commits. IEEE Transactions on Software Engineering, Vol. 47, 12 (2020), 1--1. https://doi.org/10.1109/TSE.2020.2967380
[2]
Vishakha Agrawal, Yong-Han Lin, and Jinghui Cheng. 2022. Understanding the Characteristics of Visual Contents in Open Source Issue Discussions: A Case Study of Jupyter Notebook. In The International Conference on Evaluation and Assessment in Software Engineering 2022 (Gothenburg, Sweden). ACM, New York, NY, USA, 1--7. https://doi.org/10.1145/3530019.3534082
[3]
Deeksha Arya, Wenting Wang, Jin L. C. Guo, and Jinghui Cheng. 2019. Analysis and Detection of Information Types of Open Source Software Issue Discussions. In Proceedings of the 41st International Conference on Software Engineering (ICSE '19). IEEE Press, New York, NY, USA, 454--464. https://doi.org/10.1109/ICSE.2019.00058
[4]
Paula M Bach and John M Carroll. 2009. FLOSS UX design: An analysis of user experience design in Firefox and OpenOffice. org. In Open Source Ecosystems: Diverse Communities Interacting. Springer Berlin Heidelberg, Berlin, Heidelberg, 237--250.
[5]
Paula M Bach, Robert DeLine, and John M Carroll. 2009. Designers wanted: participation and the user experience in open source software development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, Inc., Boston MA USA, 985--994.
[6]
Paula M Bach and Michael Twidale. 2010. Involving reflective users in design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 2037--2040. https://doi.org/10.1145/1753326.1753636
[7]
Mehdi Bagherzadeh and Raffi Khatchadourian. 2019. Going big: a large-scale study on what big data developers ask. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 432--442. https://doi.org/10.1145/3338906.3338939
[8]
Olga Baysal, Reid Holmes, and Michael W. Godfrey. 2014. No Issue Left behind: Reducing Information Overload in Issue Tracking. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China) (FSE 2014). Association for Computing Machinery, New York, NY, USA, 666--677. https://doi.org/10.1145/2635868.2635887
[9]
Dane Bertram, Amy Voida, Saul Greenberg, and Robert Walker. 2010. Communication, collaboration, and bugs: the social nature of issue tracking in small, collocated teams. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. Association for Computing Machinery, New York, NY, USA, 291--300.
[10]
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. J. Mach. Learn. Res., Vol. 3, null (mar 2003), 993--1022.
[11]
Judith M. Brown, Gitte Lindgaard, and Robert Biddle. 2012. Joint Implicit Alignment Work of Interaction Designers and Software Developers. In Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design (Copenhagen, Denmark) (NordiCHI '12). Association for Computing Machinery, New York, NY, USA, 693--702. https://doi.org/10.1145/2399016.2399121
[12]
Fabio Calefato, Filippo Lanubile, Federico Maiorano, and Nicole Novielli. 2018. Sentiment Polarity Detection for Software Development. Empirical Software Engineering, Vol. 23, 3 (jun 2018), 1352--1382. https://doi.org/10.1007/s10664-017--9546--9
[13]
Stephanie Chamberlain, Helen Sharp, and Neil Maiden. 2006. Towards a framework for integrating agile development and user-centred design. In International Conference on Extreme Programming and Agile Processes in Software Engineering. Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 143--153.
[14]
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009. Reading Tea Leaves: How Humans Interpret Topic Models. In Advances in Neural Information Processing Systems, Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Eds.), Vol. 22. Curran Associates, Inc., Red Hook, NY, USA, 288--296.
[15]
Yan Chen, Sang Won Lee, and Steve Oney. 2021. CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 416, 14 pages. https://doi.org/10.1145/3411764.3445573
[16]
Jinghui Cheng and Jin L.C. Guo. 2018. How Do the Open Source Communities Address Usability and UX Issues? An Exploratory Study. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA '18). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/3170427.3188467
[17]
Jinghui Cheng and Jin L.C. Guo. 2019. Activity-Based Analysis of Open Source Software Contributors: Roles and Dynamics. In 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE Press, New York, NY, USA, 11--18. https://doi.org/10.1109/CHASE.2019.00011
[18]
Elizabeth F. Churchill. 2019. Scaling UX with Design Systems. Interactions, Vol. 26, 5 (aug 2019), 22--23. https://doi.org/10.1145/3352681
[19]
Isabella Ferreira, Jinghui Cheng, and Bram Adams. 2021. The “Shut the f** k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1--35.
[20]
Stuart Geman and Donald Geman. 1984. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-6, 6 (1984), 721--741. https://doi.org/10.1109/TPAMI.1984.4767596
[21]
Petra Heck and Andy Zaidman. 2017. A framework for quality assessment of just-in-time requirements: the case of open source feature requests. Requirements Engineering, Vol. 22, 4 (2017), 453--473.
[22]
Jazlyn Hellman, Jiahao Chen, Md Sami Uddin, Jinghui Cheng, and Jin LC Guo. 2022. Characterizing User Behaviors in Open-Source Software User Forums: An Empirical Study. In 2022 IEEE/ACM 15th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, New York, NY, USA, 46--55.
[23]
Jazlyn Hellman, Jinghui Cheng, and Jin L.C. Guo. 2021. Facilitating Asynchronous Participatory Design of Open Source Software: Bringing End Users into the Loop. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA '21). Association for Computing Machinery, New York, NY, USA, Article 438, 7 pages. https://doi.org/10.1145/3411763.3451643
[24]
Qiao Huang, David Lo, Xin Xia, Qingye Wang, and Shanping Li. 2017. Which packages would be affected by this bug report?. In 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE). IEEE, New York, NY, USA, 124--135. https://doi.org/10.1109/ISSRE.2017.24
[25]
Netta Iivari. 2011. Participatory design in OSS development: interpretive case studies in company and community OSS development contexts. Behaviour & Information Technology, Vol. 30, 3 (2011), 309--323.
[26]
Netta Iivari. 2013. `Configuring the User and the Designer' -- A Critical Inquiry on Usability Work in the Company Open Source Software Development Context. In Nordic Contributions in IS Research, Margunn Aanestad and Tone Bratteteig (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1--17.
[27]
Finn Kensing and Jeanette Blomberg. 1998. Participatory Design: Issues and Concerns. Computer Supported Cooperative Work (CSCW), Vol. 7, 3 (1998), 167--185. https://doi.org/10.1023/A:1008689307411
[28]
Amy J. Ko and Parmit K. Chilana. 2010. How Power Users Help and Hinder Open Bug Reporting. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 1665--1674. https://doi.org/10.1145/1753326.1753576
[29]
Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications, California, USA.
[30]
Yassine Lamine and Jinghui Cheng. 2022. Understanding and supporting the design systems practice. Empirical Software Engineering, Vol. 27, 6 (2022), 146. https://doi.org/10.1007/s10664-022--10181-y
[31]
Germán Leiva, Nolwenn Maudet, Wendy Mackay, and Michel Beaudouin-Lafon. 2019. Enact: Reducing Designer--Developer Breakdowns When Prototyping Custom Interactions. ACM Trans. Comput.-Hum. Interact., Vol. 26, 3, Article 19 (may 2019), 48 pages. https://doi.org/10.1145/3310276
[32]
Paul Luo Li, Andrew J. Ko, and Andrew Begel. 2017. Cross-Disciplinary Perspectives on Collaborations with Software Engineers. In Proceedings of the 10th International Workshop on Cooperative and Human Aspects of Software Engineering (Buenos Aires, Argentina) (CHASE '17). IEEE, New York, NY, USA, 2--8. https://doi.org/10.1109/CHASE.2017.3
[33]
Renee Li, Pavitthra Pandurangan, Hana Frluckaj, and Laura Dabbish. 2021. Code of conduct conversations in open source software projects on Github. Proceedings of the ACM on Human-computer Interaction, Vol. 5, CSCW1 (2021), 1--31.
[34]
Malin Lundström, Johan rAberg, and Johan Blomkvist. 2015. Perceptions of Software Developers' Empathy with Designers. In Proceedings of the 2015 British HCI Conference (Lincoln, Lincolnshire, United Kingdom) (British HCI '15). Association for Computing Machinery, New York, NY, USA, 239--246. https://doi.org/10.1145/2783446.2783563
[35]
Andrew Kachites McCallum. 2002. MALLET: A Machine Learning for Language Toolkit. (2002). http://www.cs.umass.edu/ mccallum/mallet.
[36]
Itzel Morales-Ramirez, Fitsum Meshesha Kifetew, and Anna Perini. 2019. Speech-acts based analysis for requirements discovery from online discussions. Information Systems, Vol. 86 (2019), 94--112.
[37]
David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic Evaluation of Topic Coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Los Angeles, California) (HLT '10). Association for Computational Linguistics, USA, 100--108.
[38]
David M Nichols and Michael B Twidale. 2006. Usability processes in open source projects. Software Process: Improvement and Practice, Vol. 11, 2 (2006), 149--162.
[39]
Jakob Nielsen. 1994. Usability engineering. Morgan Kaufmann, San Francisco, CA, USA.
[40]
Jakob Nielsen. 2020. Ten usability heuristics. http://www.nngroup.com/articles/ten-usability-heuristics/
[41]
J. Noble, J. Ferreira, and R. Biddle. 2007. Agile Development Iterations and UI Design. In Agile 2007. IEEE Computer Society, Los Alamitos, CA, USA, 50--58. https://doi.org/10.1109/AGILE.2007.8
[42]
Nicole Novielli, Fabio Calefato, Filippo Lanubile, and Alexander Serebrenik. 2021. Assessment of off-the-shelf SE-specific sentiment analysis tools: An extended replication study. Empirical Software Engineering, Vol. 26, 4 (2021), 77. https://doi.org/10.1007/s10664-021-09960-w
[43]
Moses Openja, Bram Adams, and Foutse Khomh. 2020. Analysis of Modern Release Engineering Topics : -- A Large-Scale Study using StackOverflow --. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, New York, NY, USA, 104--114. https://doi.org/10.1109/ICSME46990.2020.00020
[44]
Fatih Kursat Ozenc, Miso Kim, John Zimmerman, Stephen Oney, and Brad Myers. 2010. How to Support Designers in Getting Hold of the Immaterial Material of Software. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 2513--2522. https://doi.org/10.1145/1753326.1753707
[45]
Sangameshwar Patil and Balaraman Ravindran. 2020. Predicting software defect type using concept-based classification. Empirical Software Engineering, Vol. 25, 2 (2020), 1341--1378.
[46]
Mikko Rajanen and Netta Iivari. 2015. Power, Empowerment and Open Source Usability. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 3413--3422. https://doi.org/10.1145/2702123.2702441
[47]
Michael Rath and Patrick M"ader. 2020. Request for Comments: Conversation Patterns in Issue Tracking Systems of Open-Source Projects. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. Association for Computing Machinery, New York, NY, USA, 1414--1417. https://doi.org/10.1145/3341105.3374056
[48]
Arif Raza, Luiz F. Capretz, and Faheem Ahmed. 2010. Improvement of Open Source Software Usability: An Empirical Evaluation from Developers' Perspective. Adv. Soft. Eng., Vol. 2010, Article 2 (jan 2010), 12 pages. https://doi.org/10.1155/2010/517532
[49]
Margarete Sandelowski. 1995. Qualitative analysis: What it is and how to begin. Research in nursing & health, Vol. 18, 4 (1995), 371--375.
[50]
Arghavan Sanei, Jinghui Cheng, and Bram Adams. 2021. The Impacts of Sentiments and Tones in Community-Generated Issue Discussions. In 2021 2021 IEEE/ACM 13th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, New York, NY, USA, 1--10. https://doi.org/10.1109/CHASE52884.2021.00009
[51]
Daniel Schwartz and Allen Gunn. 2009. Integrating User Experience into Free/Libre Open Source Software: CHI 2009 Special Interest Group. In CHI '09 Extended Abstracts on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 2739--2742. https://doi.org/10.1145/1520340.1520395
[52]
Julius Sim and Norma Reid. 1999. Statistical Inference by Confidence Intervals: Issues of Interpretation and Utilization. Physical Therapy, Vol. 79, 2 (02 1999), 186--195. https://doi.org/10.1093/ptj/79.2.186
[53]
Chade-Meng Tan, Yuan-Fang Wang, and Chan-Do Lee. 2002. The Use of Bigrams to Enhance Text Categorization. Information processing & management, Vol. 38, 4 (jul 2002), 529--546. https://doi.org/10.1016/S0306--4573(01)00045-0
[54]
Michael Terry, Matthew Kay, and Ben Lafreniere. 2010. Perceptions and Practices of Usability in the Free/Open Source Software (FoSS) Community. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 999--1008. https://doi.org/10.1145/1753326.1753476
[55]
Anthony J. Viera and Joanne M. Garrett. 2005. Understanding interobserver agreement: the kappa statistic. Family Medicine, Vol. 37.5 (2005), 360--363.
[56]
Giovanni Viviani, Calahan Janik-Jones, Michalis Famelis, Xin Xia, and Gail C. Murphy. 2018. What Design Topics Do Developers Discuss?. In Proceedings of the 26th Conference on Program Comprehension (Gothenburg, Sweden) (ICPC '18). Association for Computing Machinery, New York, NY, USA, 328--331. https://doi.org/10.1145/3196321.3196357
[57]
Wenting Wang, Deeksha Arya, Nicole Novielli, Jinghui Cheng, and Jin L.C. Guo. 2020. ArguLens: Anatomy of Community Opinions On Usability Issues Using Argumentation Models. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376218
[58]
Wenting Wang, Jinghui Cheng, and Jin L.C. Guo. 2022. How Do Open Source Software Contributors Perceive and Address Usability?: Valued Factors, Practices, and Challenges. IEEE Software, Vol. 39, 1 (2022), 76--83. https://doi.org/10.1109/MS.2020.3009514
[59]
Xukun Wang, Matthias Lee, Angie Pinchbeck, and Fatemeh Fard. 2019. Where Does LDA Sit for GitHub?. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW). IEEE, New York, NY, USA, 94--97. https://doi.org/10.1109/ASEW.2019.00037
[60]
Xin Xia, David Lo, Ying Ding, Jafar M. Al-Kofahi, Tien N. Nguyen, and Xinyu Wang. 2017. Improving Automated Bug Triaging with Specialized Topic Model. IEEE Transactions on Software Engineering, Vol. 43, 3 (2017), 272--297. https://doi.org/10.1109/TSE.2016.2576454
[61]
Peifeng Yin, Zhe Liu, Anbang Xu, and Taiga Nakamura. 2017. Tone Analyzer for Online Customer Service: An Unsupervised Model with Interfered Training. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (Singapore, Singapore) (CIKM '17). Association for Computing Machinery, New York, NY, USA, 1887--1895. https://doi.org/10.1145/3132847.3132864
[62]
Nor Shahida Mohamad Yusop, John Grundy, and Rajesh Vasa. 2017. Reporting Usability Defects: A Systematic Literature Review. IEEE Transactions on Software Engineering, Vol. 43, 9 (2017), 848--867. https://doi.org/10.1109/TSE.2016.2638427
[63]
Joyce Zhu, Jeremy Warner, Mitchell Gordon, Jeffery White, Renan Zanelatto, and Philip J. Guo. 2015. Toward a domain-specific visual discussion forum for learning computer programming: An empirical study of a popular MOOC forum. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, Now York, NY, USA, 101--109. https://doi.org/10.1109/VLHCC.2015.7357204
Index Terms
- Characterizing Usability Issue Discussions in Open Source Software Projects
Recommendations
Usability and free/libre/open source software SIG: HCI expertise and design rationale
CHI EA '07: CHI '07 Extended Abstracts on Human Factors in Computing SystemsThe usability of Free/Libre/Open Source Software (FLOSS) is a new challenge for HCI professionals. Although HCI professionals are working on usability issues in FLOSS, the CHI community has not yet organized with respect to FLOSS. The purpose of this ...
Introducing usability activities into open source software development projects: a participative approach
NordiCHI '12: Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through DesignUsability is an important quality characteristic of software products and information systems. Different approaches for introducing usability activities into open source software (OSS) development have not yet been fully evaluated. This paper ...
An Empirical Study of Open Source Software Usability: The Industrial Perspective
Recent years have seen a sharp increase in the use of open source projects by common novice users; Open Source Software OSS is thus no longer a reserved arena for software developers and computer gurus. Although user-centered designs are gaining ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 April 2024
Published in PACMHCI Volume 8, Issue CSCW1
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 149Total Downloads
- Downloads (Last 12 months)149
- Downloads (Last 6 weeks)13
Reflects downloads up to 06 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in