More Web Proxy on the site http://driver.im/

research-article

On the subjectivity of emotions in software projects: : How reliable are pre-labeled data sets for sentiment analysis?

Authors:

Larissa Chazette,

Jil KlünderAuthors Info & Claims

Volume 193, Issue C

https://doi.org/10.1016/j.jss.2022.111448

Published: 01 November 2022 Publication History

Abstract

Social aspects of software projects become increasingly important for research and practice. Different approaches analyze the sentiment of a development team, ranging from simply asking the team to so-called sentiment analysis on text-based communication. These sentiment analysis tools are trained using pre-labeled data sets from different sources, including GitHub and Stack Overflow.

In this paper, we investigate if the labels of the statements in the data sets coincide with the perception of potential members of a software project team. Based on an international survey, we compare the median perception of 94 participants with the pre-labeled data sets as well as every single participant’s agreement with the predefined labels. Our results point to three remarkable findings: (1) Although the median values coincide with the predefined labels of the data sets in 62.5% of the cases, we observe a huge difference between the single participant’s ratings and the labels; (2) there is not a single participant who totally agrees with the predefined labels; and (3) the data set whose labels are based on guidelines performs better than the ad hoc labeled data set.

Highlights

•

Study about the perceptions of software developers relating sentiments in statements.

•

Agreement analysis of labels from 94 participants with labels from original authors.

•

Comparison of guidelines-based labeled data and ad hoc labeled data.

References

[1]

Ahmed T., Bosu A., Iqbal A., Rahimi S., SentiCR: A customized sentiment analysis tool for code review interactions, in: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, Piscataway, NJ, USA, 2017, pp. 106–111,.

[2]

Calefato F., Lanubile F., Maiorano F., Novielli N., Sentiment polarity detection for software development, Empir. Softw. Eng. 23 (2018) 1352–1382,.

Digital Library

[3]

Chen Z., Cao Y., Yao H., Lu X., Peng X., Mei H., Liu X., Emoji-powered sentiment and emotion detection from software developers’ communication data, ACM Trans. Softw. Eng. Methodol. 30 (2) (2021),.

Digital Library

[4]

Cohen J., A coefficient of agreement for nominal scales, Educ. Psychol. Meas. 20 (1) (1960) 37–46,.

[5]

Ding J., Sun H., Wang X., Liu X., Entity-level sentiment analysis of issue comments, in: Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering, in: SEmotion ’18, Association for Computing Machinery, New York, NY, USA, 2018, pp. 7–13,.

Digital Library

[6]

Fleiss J.L., Measuring nominal scale agreement among many raters, Psychol. Bull. 76 (5) (1971) 378–-382,.

[7]

Graziotin D., Wang X., Abrahamsson P., Happy software developers solve problems better: psychological measurements in empirical software engineering, PeerJ 2 (2014),.

[8]

Graziotin D., Wang X., Abrahamsson P., How do you feel, developer? An explanatory theory of the impact of affects on programming performance, Peer J Comput. Sci. 1 (2015),.

[9]

Harris C.R., Millman K.J., van der Walt S.J., Gommers R., Virtanen P., Cournapeau D., Wieser E., Taylor J., Berg S., Smith N.J., Kern R., Picus M., Hoyer S., van Kerkwijk M.H., Brett M., Haldane A., del Río J.F., Wiebe M., Peterson P., Gérard-Marchant P., Sheppard K., Reddy T., Weckesser W., Abbasi H., Gohlke C., Oliphant T.E., Array programming with NumPy, Nature 585 (7825) (2020) 357–362,.

[10]

Haynes W., Bonferroni correction, in: Encyclopedia of Systems Biology, Springer, New York, NY, USA, 2013, p. 154,.

[11]

Herrmann M., Klünder J., From textual to verbal communication: Towards applying sentiment analysis to a software project meeting, in: 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), IEEE, Piscataway, NJ, USA, 2021, pp. 371–376,.

[12]

Herrmann M., Obaidi M., Chazette L., Klünder J., Dataset: On the subjectivity of emotions in software projects: How reliable are pre-labeled data sets for sentiment analysis?, 2022,.

[13]

Hunter J.D., Matplotlib: A 2D graphics environment, Comput. Sci. Eng. 9 (3) (2007) 90–95,.

Digital Library

[14]

Imtiaz N., Middleton J., Girouard P., Murphy-Hill E., Sentiment and politeness analysis tools on developer discussions are unreliable, but so are people, in: Proceedings of the Third International Workshop on Emotion Awareness in Software Engineering, Association for Computing Machinery, New York, NY, USA, 2018, pp. 55–61,.

Digital Library

[15]

Islam M.R., Zibran M.F., Leveraging automated sentiment analysis in software engineering, in: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), IEEE, 2017, pp. 203–214,.

Digital Library

[16]

Islam M., Zibran M., SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text, J. Syst. Softw. 145 (2018) 125–146,.

[17]

Klünder J., Horstmann J., Karras O., Identifying the mood of a software development team by analyzing text-based communication in chats with machine learning, in: International Conference on Human-Centred Software Engineering, Springer International Publishing, Heidelberg, BW, DE, 2020, pp. 133–151.

[18]

Kraut R.E., Streeter L.A., Coordination in software development, Commun. ACM 38 (3) (1995) 69–81,.

Digital Library

[19]

Landis J., Koch G., The measurement of observer agreement for categorical data, Biometrics 33 1 (1977) 159–174,.

[20]

Lin B., Zampetti F., Bavota G., Di Penta M., Lanza M., Oliveto R., Sentiment analysis for software engineering: How far can we go?, in: Proceedings of the 40th International Conference on Software Engineering, in: ICSE ’18, Association for Computing Machinery, New York, NY, USA, 2018, pp. 94–104,.

Digital Library

[21]

Marjaie, S., Rathod, U., 2011. Communication in agile software projects: qualitative analysis using grounded theory in system dynamics. In: Proc. Int’L Conf. of the System Dynamics Society 2011.

[22]

McChesney I.R., Gallagher S., Communication and co-ordination practices in software engineering projects, Inf. Softw. Technol. 46 (7) (2004) 473–489,.

[23]

Mohammad S., A practical guide to sentiment annotation: Challenges and solutions, in: Proceedings of the 7th Workshop on Computational Approaches To Subjectivity, Sentiment and Social Media Analysis, Association for Computational Linguistics, 2016, pp. 174–179,.

[24]

Murgia A., Tourani P., Adams B., Ortu M., Do developers feel emotions? An exploratory analysis of emotions in software artifacts, in: Proceedings of the 11th Working Conference on Mining Software Repositories, in: MSR 2014, Association for Computing Machinery, New York, NY, USA, 2014, pp. 262–271,.

Digital Library

[25]

Niinimäki T., Piri A., Lassenius C., Paasivaara M., Reflecting the choice and usage of communication tools in global software development projects with media synchronicity theory, J. Softw. Evol. Process 24 (6) (2012) 677–692,.

[26]

Novielli N., Calefato F., Dongiovanni D., Girardi D., Lanubile F., Can we use SE-specific sentiment analysis tools in a cross-platform setting?, in: Proceedings of the 17th International Conference on Mining Software Repositories, Association for Computing Machinery, New York, NY, USA, 2020, pp. 158–168,.

Digital Library

[27]

Novielli N., Calefato F., Lanubile F., A gold standard for emotion annotation in stack overflow, in: Proceedings of the 15th International Conference on Mining Software Repositories, in: MSR ’18, Association for Computing Machinery, New York, NY, USA, 2018, pp. 14–17,.

Digital Library

[28]

Novielli N., Girardi D., Lanubile F., A benchmark study on sentiment analysis for software engineering research, in: 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), in: MSR ’18, Association for Computing Machinery, 2018, pp. 364–375,.

Digital Library

[29]

Obaidi M., Klünder J., Development and application of sentiment analysis tools in software engineering: A systematic literature review, in: International Conference on Evaluation and Assessment in Software Engineering, Association for Computing Machinery, ACM, New York, NY, USA, 2021, pp. 80–89,.

Digital Library

[30]

Parrott W.G., Emotions in Social Psychology: Essential Readings, psychology Press, 2001.

[31]

Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., Blondel M., Prettenhofer P., Weiss R., Dubourg V., Vanderplas J., Passos A., Cournapeau D., Brucher M., Perrot M., Duchesnay E., Scikit-learn: Machine learning in python, J. Mach. Learn. Res. 12 (2011) 2825–2830.

[32]

Rey D., Neuhäuser M., Wilcoxon-signed-rank test, in: International Encyclopedia of Statistical Science, Springer, Berlin, Heidelberg, 2011, pp. 1658–1659,.

[33]

Robson C., McCartan K., Real World Research, John Wiley & Sons, Inc, New York, NY, USA, 2015.

[34]

Shapiro S.S., Wilk M.B., An analysis of variance test for normality (complete samples), Biometrika 52 (3–4) (1965) 591–611,.

[35]

Shaver P.R., Schwartz J.C., Kirson D., O’Connor C., Emotion knowledge: further exploration of a prototype approach, J. Personal. Soc. Psychol. 52 6 (1987) 1061–1086.

[36]

Student P.R., The probable error of a mean, Biometrika 6 (1) (1908) 1–25,.

[37]

Uddin G., Khomh F., Automatic mining of opinions expressed about APIs in stack overflow, IEEE Trans. Softw. Eng. 47 (3) (2021) 522–559,.

[38]

Wes McKinney G., Data structures for statistical computing in python, in: Proceedings of the 9th Python in Science Conference, SciPy, Austin, TX, USA, 2010, pp. 56–61,.

[39]

Wu J., Ye C., Zhou H., BERT for sentiment classification in software engineering, in: 2021 International Conference on Service Science (ICSS), 2021, pp. 115–121,.

[40]

Zhang T., Xu B., Thung F., Haryono S.A., Lo D., Jiang L., Sentiment analysis for software engineering: How far can pre-trained transformer models go?, in: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, Adelaide, SA, Australia, 2020, pp. 70–80,.

Cited By

Anwar ZAfzal HKadry SCheng X(2024)Semantic Web Approaches in Stack OverflowInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.35861720:1(1-61)Online publication date: 9-Nov-2024
https://dl.acm.org/doi/10.4018/IJSWIS.358617
Specht AObaidi MNagel LStess MKlünder J(2024)What is Needed to Apply Sentiment Analysis in Real Software Projects: A Feasibility Study in IndustryHuman-Centered Software Engineering10.1007/978-3-031-64576-1_6(105-129)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1007/978-3-031-64576-1_6
Obaidi MHolm HSchneider KKlünder J(2022)On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform SettingProduct-Focused Software Process Improvement10.1007/978-3-031-21388-5_8(108-123)Online publication date: 21-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-21388-5_8

Index Terms

On the subjectivity of emotions in software projects: How reliable are pre-labeled data sets for sentiment analysis?

Index terms have been assigned to the content through auto-classification.

Recommendations

Sentence Subjectivity Analysis in Social Domains
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 01

Subjectivity analysis recognizes the contextual polarity of opinions, attitudes, emotions, feelings etc. regarding products, services, topics, or issues. Subjectivity classification categorizes the given text as subjective or objective. While an ...
Semi-supervised probabilistic sentiment analysis: merging labeled sentences with unlabeled reviews to identify sentiment
ASIST '13: Proceedings of the 76th ASIS&T Annual Meeting: Beyond the Cloud: Rethinking Information Boundaries

Document level sentiment analysis, the task of determining whether the sentiment expressed in a document is positive or negative, is commonly performed by supervised methods. As with all supervised tasks, obtaining training data for these methods can be ...
The challenge of understanding the flow of sentiments in social media documents
SMUC '11: Proceedings of the 3rd international workshop on Search and mining user-generated contents

This talk is focused on a key task in the area of Opinion Mining and Sentiment Analysis: polarity classification of social media documents (e.g. blog posts). Estimating polarity is much more demanding than estimating topicality. As a matter of fact, the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Systems and Software

Journal of Systems and Software Volume 193, Issue C

Nov 2022

400 pages

ISSN:0164-1212

Issue’s Table of Contents

Elsevier Inc.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 November 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Anwar ZAfzal HKadry SCheng X(2024)Semantic Web Approaches in Stack OverflowInternational Journal on Semantic Web & Information Systems10.4018/IJSWIS.35861720:1(1-61)Online publication date: 9-Nov-2024
https://dl.acm.org/doi/10.4018/IJSWIS.358617
Specht AObaidi MNagel LStess MKlünder J(2024)What is Needed to Apply Sentiment Analysis in Real Software Projects: A Feasibility Study in IndustryHuman-Centered Software Engineering10.1007/978-3-031-64576-1_6(105-129)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1007/978-3-031-64576-1_6
Obaidi MHolm HSchneider KKlünder J(2022)On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform SettingProduct-Focused Software Process Improvement10.1007/978-3-031-21388-5_8(108-123)Online publication date: 21-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-21388-5_8

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents