More Web Proxy on the site http://driver.im/

short-paper

On Using Active Learning and Self-training when Mining Performance Discussions on Stack Overflow

Authors:

Iben Lennerstad,

Elizabeth BjarnasonAuthors Info & Claims

EASE '17: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering

Pages 308 - 313

https://doi.org/10.1145/3084226.3084273

Published: 15 June 2017 Publication History

Abstract

Abundant data is the key to successful machine learning. However, supervised learning requires annotated data that are often hard to obtain. In a classification task with limited resources, Active Learning (AL) promises to guide annotators to examples that bring the most value for a classifier. AL can be successfully combined with self-training, i.e., extending a training set with the unlabelled examples for which a classifier is the most certain. We report our experiences on using AL in a systematic manner to train an SVM classifier for Stack Overflow posts discussing performance of software components. We show that the training examples deemed as the most valuable to the classifier are also the most difficult for humans to annotate. Despite carefully evolved annotation criteria, we report low inter-rater agreement, but we also propose mitigation strategies. Finally, based on one annotator's work, we show that self-training can improve the classification accuracy. We conclude the paper by discussing implication for future text miners aspiring to use AL and self-training.

References

[1]

M. Allamanis and C. Sutton. 2013. Why, When, and What: Analyzing Stack Overflow Questions by Topic, Type, and Code. In Proc. of the 10th Working Conference on Mining Software Repositories. 53--56.

Digital Library

[2]

M. Banko and E. Brill. 2001. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In Proc. of the 39th Annual Meeting on Association for Computational Linguistics. 26--33.

Digital Library

[3]

A. Barua, S. Thomas, and A. Hassan. 2012. What are Developers Talking About? An Analysis of Topics and Trends in Stack Overflow. Empirical Software Engineering 19, 3 (2012), 619--654.

Digital Library

[4]

M. Borg. 2016. TuneR: A Framework for Tuning Software Engineering Tools with Hands-On Instructions in R. Journal of Software: Evolution and Process 28, 6 (2016), 427--459.

Digital Library

[5]

S. Chowdhury and A. Hindle. 2015. Mining StackOverflow to Filter out Off-topic IRC Discussion. In Proc. of the 12th Working Conference on Mining Software Repositories. 422--425.

Digital Library

[6]

A. Cicchetti, M. Borg, S Sentilles, K. Wnuk, J. Carlsson, and E. Papatheocharous. 2016. Towards Software Assets Origin Selection Supported by a Knowledge Repository. In Proc. of the 1st International Workshop on Decision Making in Software Architecture.

[7]

G.Feng. 2015. Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 11, 1 (2015), 13--22.

[8]

J. Hannay and M. Jørgensen. 2008. The Role of Deliberate Artificial Design Elements in Software Engineering Experiments. IEEE Transactions on Software Engineering 34, 2 (2008), 242--259.

Digital Library

[9]

A. Hassan and T. Xie. 2010. Software Intelligence: The Future of Mining Software Engineering Data. In Proc. of the FSE/SDP Workshop on Future of Software Engineering Research. 161--166.

Digital Library

[10]

Y. Lin, C. Sun, W Xiaolong, and W Xuan. 2010. Combining Self Learning and Active Learning for Chinese Named Entity Recognition. Journal of Software 5, 5 (2010), 530--537.

[11]

H. Lu and B. Cukic. 2012. An Adaptive Approach with Active Learning in Software Fault Prediction. In Proc. of the 8th International Conference on Predictive Models in Software Engineering. 79--88.

Digital Library

[12]

H. Lu, E. Kocaguneli, and B. Cukic. 2014. Defect Prediction between Software Versions with Active Learning and Dimensionality Reduction. In Proc. of the 25th International Symposium on Software Reliability Engineering. 312--322.

Digital Library

[13]

T. Menzies, C. Bird, T. Zimmermann, W. Schulte, and E. Kocaganeli. 2011. The Inductive Software Engineering Manifesto: Principles for Industrial Data Mining. In Proc. of the International Workshop on Machine Learning Technologies in Software Engineering. 19--26.

Digital Library

[14]

T. Mitra, C. Hutto, and E. Gilbert. 2015. Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. In Proc. of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1345--1354.

Digital Library

[15]

F. Pedregosa, G. Varoquaux, A. Gramfort, V Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, and V Dubourg. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct (2011), 2825--2830.

Digital Library

[16]

F. Pereira, P. Norvig, and A. Halevy. 2009. The Unreasonable Effectiveness of Data. IEEE Intelligent Systems 24, 2 (2009), 8--12.

Digital Library

[17]

L. Ponzanelli, G. Bavota, M. Di Penta, R. Oliveto, and M. Lanza. 2014. Mining StackOverflow to Turn the IDE into a Self-confident Programming Prompter. In Proc. of the 11th Working Conference on Mining Software Repositories. 102--111.

Digital Library

[18]

J. Richarz, S. Vajda, and G. Fink. 2012. Annotating Handwritten Characters with Minimal Human Involvement in a Semi-supervised Learning Strategy. In Proc. of the International Conference on Frontiers in Handwriting Recognition. 23--28.

Digital Library

[19]

P. Runeson, M. Höst, A. Rainer, and B. Regnell. 2012. Case Study Research in Software Engineering. Guidelines and Examples. Wiley.

Digital Library

[20]

B. Settles. 2010. Active Learning Literature Survey. Technical Report Computer Sciences Technical Report 1648. University of Wisconsin-Madison.

[21]

B. Settles. 2011. Closing the Loop: Fast, Interactive Semi-supervised Annotation with Queries on Features and Instances. In Proc. of the Conference on Empirical Methods in Natural Language Processing. 1467--1478.

Digital Library

[22]

B. Settles. 2011. From Theories to Queries: Active Learning in Practice. In Proc. of the JMRL Workshop on Active Learning and Experimental Design.

[23]

M. Soliman, M. Galster, A. Salama, and M. Riebisch. 2016. Architectural Knowledge for Technology Decisions in Developer Communities: An Exploratory Study with StackOverflow. In Proc. of the 13th Working IEEE/IFIP Conference on Software Architecture. 128--133.

[24]

M. Stikic, K. Van Laerhoven, and B. Schiele. 2008. Exploring semi-supervised and active learning for activity recognition. In Proc. of the 12th International Symposium on Wearable Computers. 81--88.

Digital Library

[25]

B. Sun, G. Shu, A. Podgurski, and S. Ray. 2012. CARIAL: Cost-Aware Software Reliability Improvement with Active Learning. In Proc. of the 5th International Conference on Software Testing, Verification and Validation. 360--369.

Digital Library

[26]

C. Treude, O. Barzilay, and M. Storey. 2011. How do Programmers Ask and Answer Questions on the Web?: NIER track. In Proc. of the 33rd International Conference on Software Engineering. 804--807.

Digital Library

[27]

V. Vapnik. 2000. The Nature of Statistical Learning Theory. Springer New York.

Digital Library

[28]

C. Wohlin, K. Wnuk, D. Smite, U. Franke, D. Badampudi, and A. Cicchetti. 2016. Supporting Strategic Decision-Making for Selection of Software Assets. In Software Business, A. Maglyas and AL. Lamprecht (Eds.). Number 240 in Lecture Notes in Business Information Processing. Springer, Cham, 1--15.

[29]

D. Yarowsky. 1995. Unsupervised Word Sense Disambiguation Rivaling Supervised Methods. In Proc. of the 33rd Annual Meeting on Association for Computational Linguistics. 189--196.

Digital Library

[30]

A. Ying. 2015. Mining Challenge 2015: Comparing and Combining Different Information Sources on the Stack Overflow Data Set. In Proc. of the 12th Working Conference on Mining Software Repositories.

[31]

Z. Zhang, E. Pasolli, M. Crawford, and J. C. Tilton. 2016. An Active Learning Framework for Hyperspectral Image Classification Using Hierarchical Segmentation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 9, 2 (2016), 640--654.

Cited By

Arshid ZMustafa EShuja JShah F(2024)Using Active and Self-Training to Improve Sentiment Prediction on App Reviews2024 1st International Conference on Innovative Engineering Sciences and Technological Research (ICIESTR)10.1109/ICIESTR60916.2024.10798138(1-6)Online publication date: 14-May-2024
https://doi.org/10.1109/ICIESTR60916.2024.10798138
Lidfeldt AIsaksson DHedlund LÅberg SBorg MLarsson E(2020)Enabling Image Recognition on Constrained Devices Using Neural Network Pruning and a CycleGANCompanion Proceedings of the 10th International Conference on the Internet of Things10.1145/3423423.3423437(1-14)Online publication date: 6-Oct-2020
https://dl.acm.org/doi/10.1145/3423423.3423437
Khan JLee Y(2019)LeSSA: A Unified Framework based on Lexicons and Semi-Supervised Learning Approaches for Textual Sentiment ClassificationApplied Sciences10.3390/app92455629:24(5562)Online publication date: 17-Dec-2019
https://doi.org/10.3390/app9245562
Show More Cited By

Index Terms

On Using Active Learning and Self-training when Mining Performance Discussions on Stack Overflow
1. Software and its engineering
  1. Software creation and management

Recommendations

Self-Training with Selection-by-Rejection
ICDM '12: Proceedings of the 2012 IEEE 12th International Conference on Data Mining

Practical machine learning and data mining problems often face shortage of labeled training data. Self-training algorithms are among the earliest attempts of using unlabeled data to enhance learning. Traditional self-training algorithms label unlabeled ...
Improving Graph Neural Networks by combining active learning with self-training
Abstract
In this paper, we propose a novel framework, called STAL, which makes use of unlabeled graph data, through a combination of Active Learning and Self-Training, in order to improve node labeling by Graph Neural Networks (GNNs). GNNs have been shown ...
Text classification method based on self-training and LDA topic models

A novel text classification method for learning from very small labeled set.The method uses a text representation based on the LDA topic model.Self-training is used to enlarge labeled set from unlabeled instances.A model for setting methods parameters ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '17: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering

June 2017

405 pages

ISBN:9781450348041

DOI:10.1145/3084226

Conference Chair:
Emilia Mendes,
Program Chairs:
Steve Counsell,
Kai Petersen

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

School of Computing, BTH: Blekinge Institute of Technology - School of Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

EASE'17

EASE'17: Evaluation and Assessment in Software Engineering

June 15 - 16, 2017

Karlskrona, Sweden

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
150
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Arshid ZMustafa EShuja JShah F(2024)Using Active and Self-Training to Improve Sentiment Prediction on App Reviews2024 1st International Conference on Innovative Engineering Sciences and Technological Research (ICIESTR)10.1109/ICIESTR60916.2024.10798138(1-6)Online publication date: 14-May-2024
https://doi.org/10.1109/ICIESTR60916.2024.10798138
Lidfeldt AIsaksson DHedlund LÅberg SBorg MLarsson E(2020)Enabling Image Recognition on Constrained Devices Using Neural Network Pruning and a CycleGANCompanion Proceedings of the 10th International Conference on the Internet of Things10.1145/3423423.3423437(1-14)Online publication date: 6-Oct-2020
https://dl.acm.org/doi/10.1145/3423423.3423437
Khan JLee Y(2019)LeSSA: A Unified Framework based on Lexicons and Semi-Supervised Learning Approaches for Textual Sentiment ClassificationApplied Sciences10.3390/app92455629:24(5562)Online publication date: 17-Dec-2019
https://doi.org/10.3390/app9245562
Nguyen PDi Rocco JDi Ruscio DAli SGarousi V(2019)Enabling heterogeneous recommendations in OSS developmentProceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering10.1145/3319008.3319353(326-331)Online publication date: 15-Apr-2019
https://dl.acm.org/doi/10.1145/3319008.3319353
Raulamo-Jurvanen PHosio SMäntylä MAli SGarousi V(2019)Practitioner Evaluations on Software Testing ToolsProceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering10.1145/3319008.3319018(57-66)Online publication date: 15-Apr-2019
https://dl.acm.org/doi/10.1145/3319008.3319018

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents