A novel hybrid CNN-LSTM approach for assessing StackOverflow post quality

1 Introduction

StackOverflow is the largest community question answering (CQA) site for programmers. About 30% of the questions posted on StackOverflow remain unanswered. The reason behind this is that the questioner is unable to attract experts to answer his question [1]. Content quality [2] is important to attract new users and retain existing users [3]. Moderators assess the quality of questions posted on StackOverflow and delete the low-quality questions. The daily volume of users’ posts is quite high; therefore, assessing the quality of content manually is not practical. To overcome this, an automated system is required to determine the quality of content.

Generally, many answers are posted on CQA sites for a question but only a few answers are of worth [4]. All answers are not posted by domain experts, and the moderator can judge the quality of the answer, but it is not practical because it is based on the moderator’s subjective assessment [3]. Assessing the quality of Q&As on CQA sites can be divided into three problems, i.e. determining the quality of questions [5], determining the quality of the answers [6,7], and determining the quality of both Q&A [8]. The post quality system has various applications like finding experts [9,10], information seeking [11], education [12], knowledge sharing [2,13], and question routing [14].

Accessing the quality of content posted on online communities is an active research area. In the existing research, user post quality can be determined by using features-based classification [8,11,15] and non-features-based classification [4,16]. Both of these techniques have their merits and demerits. However, feature-based techniques are most widely used for this purpose. For example, both Tóth et al. [17] and Baltadzhieva and Chrupallla [18] found that linguistic and semantic features, such as the presence of code snippets and the use of certain terms, can influence question quality. Mi et al. [19] found that the number of tags and code snippets is the most discriminative feature for identifying high-quality questions. Arora et al. [20] proposed a method for classifying questions based on retrieving similar questions previously asked in the same forum. These findings suggest that question quality on Stack Overflow can be assessed using various features and that automated methods can be used to improve the effectiveness of question quality moderation.

Furthermore, most of the researchers use binary classification to classify users’ posts into good and bad categories [11,15]. The accuracy of these techniques is 74–76% [5,17]. But while analysing the StackOverflow posts, it is observed that the post should be categorized into four categories. Very good post has a high score and also accepted answer. A good question has a high score but no accepted answer. Similarly, a bad question has low score, and a very bad question has a low score and is also closed by the moderator. It is also observed that existing multi-classification techniques for user post quality assessment has a low accuracy. For example, the work of Kopp et al. has 41.6% accuracy [21]. Researchers also used different categories of features, including textual, format, community, and readability features [22]. Using the community features like reputation of user and score of question is biased towards the high reputation users [5]. The limitations of existing techniques are given in Table 1.

StackOverflow has developed a subjective assessment-based mechanism to determine the quality of Q&A. In subjective assessment score of posts rated by users is used to measure the quality of post. However, the objective mechanism that is based on different features that are calculated from post is better than the subjective mechanism as it is not biased [28]. While there have been previous attempts to evaluate the quality of user posts, these have various limitations as identified earlier. To overcome these limitations, a novel hybrid model for assessing the quality of StackOverflow posts is proposed. The proposed model combines the strengths of both convolutional neural networks (CNNs) and long short-term memory (LSTM) networks. The proposed model is trained on a set of 20 different features that are extracted from the data. The feature extraction process is a critical step in accurately evaluating the quality of StackOverflow posts. Therefore, we selected different features and group these features into five different sets. Extensive experiments were performed to select the best features set. Finally, the best feature set is used to train the proposed classifier. The proposed hybrid model classifies the StackOverflow questions into four categories. For this task, already published and publicly available StackOverflow dataset [26] is used. Accuracy, precision, recall, and F1-score are used to evaluate models. In addition to this, the results of the proposed CNN + LSTM model are also compared with the existing state of the art and validated with various machine learning and deep learning models implemented and trained by authors. After the analysis of results it is observed that the proposed CNN + LSTM model gives the best accuracy on unseen test dataset that is considerably much higher than the baseline techniques.

The major contributions are enlisted as follows:

• Quality assessment of StackOverflow questions is performed using a novel hybrid deep learning architecture that combines the benefits of CNN and LSTM in one model. The proposed model and dataset is shared online for future research.

• Extensive experiments were performed, and a set of 20 features is proposed to access the quality of user posts.

• The performance of the proposed model has shown an improvement of 21–23% in accuracy as compared to the existing state of the art.

The remaining sections of this article are organized as follows. A review of related work is made in Section 2. Research methodology is explained in Section 3. Experimental results and analysis is given in Section 4. Validation of the proposed model is performed in Section 5. The proposed method is compared with state of the art tools in Section 6, and Section 7 concludes this article.

2 Related works

Extensive survey of literature is performed to conduct this research. In this section, articles that are most relevant to this research are reported. The authors highlighted and presented the tools and techniques, features, results, and limitations of existing research in this section.

Ponzanelli et al. [15] classified SO questions as good and bad quality questions. Readability metrics, author’s popularity, and simple textual features of Stack Overflow are used. Classification is performed using decision trees and genetic algorithms. It is observed that decision trees are good to classify only on the author’s popularity. Stanford NLP Parser is used to remove code snippets from questions. JGAP is used to implement genetic algorithm (GA). Important features are identified that determine the quality of questions. A precision of 80–97 is achieved using GA. It is concluded that the identification of bad-quality questions can be automated partially.

In another article, Ponzanelli et al. [11] proposed an approach to remove good-quality questions from the review queue, thus decreasing the workload of the moderators. Votes alone do not tell about the quality of the answer. Therefore, Omondiagbe et al. [29] used Random forest and neural network classifiers with various settings of features to determine the best feature set that can predict the accepted answer. It is observed that the length of code in the answer, the reputation of the user, the similarity of text between Q&A, and, the time lag between question & answer can best predict the accepted and unaccepted answers.

Ellmann and Schnecke [8] studied different measures that will increase the quality of question and answer. StackOverflow data dump of 2014 is used in this research. Error, discrepancy, and how-to question types are extracted from the dataset for further research. Naive Bayes with six features, i.e., Flesh-Reading-Ease score, Code-to-Text ratio, word count, image exists, listing exists, quote exists, and the number of tags are used to predict when a question will be answered or voted.

In another article [23], a co-prediction method based on regression and classification is proposed to predict the quality of questions and answers. The correlation between the quality of the question and answer is predicted. Features of question, answer, and author are used for this purpose. The model is trained on one million questions and 1.5 million answers. For the evaluation of model correlation, effectiveness, efficiency, root mean square error, and prediction error is used. This method improves 13.13% prediction error.

Rychwalska et al. [30] researched the association between communication and article quality at Wikipedia. The projects with denser communication between the editors are of high quality. For this purpose, the authors collected private communication between editors.

CQA sites evolved and contain a huge number of questions and answers. Reading all answers about a particular topic is not feasible and requires a huge effort. To reduce the number of answers, a technique based on grey wolf optimizer is proposed. The Biterm topic model is used to model the answer. This technique performs well to extract core answers [31].

There are three possibilities to create quality prediction systems, i.e. by learning features by using handcrafted features, by using deep learning to learn features, and by using a mixture of handcrafted and deep learning. The author uses CNN to learn the features and uses handcrafter features and CNN-generated features to learn deep neural network. This approach is called a deep fusion network. The authors use a total of 601 features. This technique is evaluated on SemEval-15 and SemEval-16, which consist of three and two classes, respectively. Accuracy of 75.24 and 76.67 is achieved on SemEval-15 and SemEval-16 datasets, respectively [16].

In a recent article [4], BERT is used to predict the quality of question and answer. MSDN dataset is used in the experiments, and many experiments using BERT are conducted. BERT as features extractor achieved an accuracy of 0.589, the BERT fine-tuning model achieved an accuracy of 0.696, and BERT pre-training model an achieved an accuracy of 0.774 on the test set.

Xiang et al. [32] developed two deep learning-based models for automatically tagging the answers in the SemEval-2015 dataset. The first model is based on CNN + LSTM and conditional random field. This model archives 58.96% F1 score. The second model is simple and is based on CNN and Bi-LSTM. The second model achieves 58.29% F1 score.

In another research [21], a dataset of 4,575 questions was manually labelled to judge the question quality posted on the intelligent learning system. The same dataset was processed by several NLP tools, and various NLP indices were calculated. These indices were used by the machine learning algorithm to predict the question quality. The results of the prediction were compared with the manually labelled dataset. Accuracy of 41.6% was achieved by using four classes and accuracy improved to 61.6% by predicting two classes of questions.

About 30% of the questions on StackOverflow are unanswered. To reduce this number and help the questioner to get answers to their question, Hsieh explored the literature, unanswered questions, and successful questions and collected features that make questions successful and unsuccessful. Three features named as code snippet presence, median length, and presence of attempt signifying words were selected by the author to train binary logistic regression classifier. A total of 75.59% accuracy, 76.92% precision, and 17.24% recall were attained. The author developed a chrome plugin using this approach. This plugin proposes answerability of the question before posting it to StackOverflow [5].

In a recent article, Roy and Singh [33] started by identifying several features that are indicative of a question being closed, such as the length of the question, the presence of certain words or phrases, and the number of tags associated with the question. The authors performed experiment with CNN and LSTM and then proposed a method that uses a CNN to automatically learn these features from the text of the question. The authors conducted experiments using a dataset from the Stack Exchange CQA site to evaluate their method. The results showed that their method outperformed existing methods for predicting closed questions, achieving an accuracy of over 80%. This method can help the users to improve the quality of their questions before posting to CQA sites.

Roy et al. [34] in another article identified several factors that are indicative of a question being closed, such as the length of the question, the number of words in all caps, and the presence of certain keywords. They then proposed a method that uses a Random Forest classifier to automatically learn these factors from the text of the question. The authors evaluate the performance of their method using a dataset from the Stack Exchange site and show that their method achieves an accuracy of over 80% in predicting question closability. They also conduct experiments to analyse the impact of different factors on question closability and find that factors such as the number of tags associated with the question and the length of the title have a significant impact on question closability.

In another article [17], researchers used linguistic and semantic features to train deep learning classifier to predict StackOverflow questions into good and bad quality. Title, body, and tags of questions are used. Embedding for Gated Rectifier Unit (GRU) was created by using Word2Vector and Doc2Vector. The authors performed four experiments by inputting different numbers of questions for training and gained a maximum of 74% accuracy.

A system based on an SVM classifier is proposed to find good and bad quality posts on Nabble.com. This approach uses surface, lexical, syntactic, and forum-specific features to determine the quality. This technique achieves 89% accuracy in categorizing posts into two classes. The authors experiment with different feature sets and found that without using forum-specific features, 82% accuracy can be achieved [22].

Ruseti et al. [35], used various variants of recurrent neural network (RNN) with different embedding to predict the quality of questions asked by students in an intelligent tutoring system. The accuracy of LSTM was low as compared to GRU and BiGRU. The best results in terms of accuracy, i.e., 81.22%, were obtained by using FastText embedding with the BiGRU network.

Selleras in his thesis [24] developed a model for identifying high quality and low answer in StackOverflow. The model consists of partial least square (PLS), natural language processing (NLP), and binomial logistic regression (BLR). Data for this research were extracted using stack exchange data explorer by using queries. Six features of the question and five features of the answer were selected. The regression equation was generated and the three Q&A were selected to test the model. The model predicted the optimal answer with 80% accuracy.

Recently, Mondal et al. [25] identified various reasons why edit on posts are rejected. To automate the edit quality assessment process, the authors trained four machine learning classifiers and developed a plugin. The accuracy of the proposed technique is 68%.

In another article, Mondal et al. [28] compared the results of subjective and objective assessments of post-quality. Subjective assessment is the default method adopted by StackOverflow in which users vote for the posts, and posts with a high score are considered as good quality. The authors identified various features and implemented five machine-learning algorithms to determine the objective quality of posts. The accuracy of the machine learning algorithm ranges from 68 to 86%.

The comparison of existing research in the form of their advantages and disadvantages is given in Table 2. Most of the existing researches [5,8,15,17,23,24] is based on binary classification. Classifying the user post quality into two classes increases the accuracy of the classifier, but the binary classifier may struggle to compute the quality of questions with accepted answers and closed questions. Therefore, a classifier that can predict the quality of users’ posts into four classes is more suitable. The accuracy of the existing classifiers [11,15] is low. To overcome these limitations, CNN, and LSTM-based hybrid model is developed for the prediction of user quality into four categories.

3 Proposed research methodology

The proposed research methodology is divided into five major steps: data collection from StackOverflow, data pre-processing & cleaning, features extraction, developing & training the proposed model, and making predictions from trained models on new test data.

3.1 Data collection

StackOverflow data can also be accessed by using Google BigQuery [36]. By using BigQuery, one does not need to download all the data dump and the required data can be extracted easily and quickly. For this research, the data were extracted from the StackOverflow posts table. The extracted fields and their description are given in Table 3, and the query used to extract data is given in Figure 1.

Table 3

Description of data fields extracted using BigQuery and used to predict quality

Field Name	Description
Id	Unique identifier of posts/questions
Accepted Answer Id	Unique identifier of accepted answer
Score	Score of question given by community member.
View count	Total views of question
Body	Body/text of the question
Title	Title of the question
Tags	Tags of question
Answer count	Total number of answer for specific question
Comment count	Total number of comments posted for specific answer
Favorite count	Number the question was favourite
Close date	Date on which question was closed

Figure 1

Google BigQuery used to extract data from Stackoverflow.

To make this research comparable and replicate the results of the existing state-of-the-art methods, the same query is used to extract data as used in [26,27,37]. A total of 200,000 questions were extracted. 70/30 rule is used to randomly split the dataset into training and testing sets. 10% of training data is reserved for validation.

3.4 Proposed CNN, and LSTM-based classification model for StackOverflow post quality assessment

To further improve the accuracy of StackOverflow questions quality assessment, a hybrid model of CNN and LSTM is proposed and implemented. The intuition behind the selection of this hybrid model is that after various experiments with machine learning and deep learning model results of this model are quite high. Furthermore, this model achieved state-of-the-art accuracy on different tasks and StackOverflow dataset [40,41]. By using the hybrid of CNN and LSTM advantages of both models can be gained. CNN model helps us to extract the features and LSTM helps to capture the long-term dependencies in the data [42]. The architecture of the proposed model is given in Figure 2. The input to the model is set to 20 × selected features. The model consists of two Conv1D layers with 64 filters and 3 kernel size. After Conv1D layers, dropout of 0.5 is added to avoid over-fitting. The Maxpooling layer after the dropout is also added to the down-sample input representation. Maxpooling layer is followed by the LSTM layer with 100 units. The next layer is a dense layer with 100 units. All of these layers are activated by the relu activation function. The last layer is dense with 4 units which is the output layer. The output layer is activated by the softmax activation function. The model is compiled with Adam optimizer and the loss function is categorical cross-entropy.

Figure 2

Proposed CNN + LSTM model-based architecture for StackOverflow question quality assessment.

4 Results and discussion

The proposed model of CNN and LSTM is implemented in Google Colab [43] environment by using Keras with Tensorflow backend. Several experiments with different numbers of CNN and LSTM layers were conducted, and the accuracy of each experiment was determined. The best results were gained by the model as given in Figure 2. This model is used to report the results. As stated earlier, the optimal set of features was identified, and the final model using the 20 features is trained on processed training data and tested on processed test data. The dataset is explained in Section 3.1, and data processing steps are explained in Section 3.2. The output of the model is the quality of the question and four metrics, i.e. accuracy, precision, recall, and F1-score that are used to analyse the results of the prediction model. The equations to calculate these metrics are given as follows:

In the aforementioned equations, TP is true positive, TN is true negative, FP is false positive, and FN is false negative.

The results of the proposed hybrid model that is given in Table 7. The accuracy of the model is 80.45%, which is considerably high and indicates that the model makes 80.45% correct predictions, which is a signification portion of the dataset. The precision of the model is 57.65%, which means the model is making only 57.65% true positive predictions from the overall positive predictions. Again the recall of the proposed model is 57.65%, which means the model is correctly predicting 57.65% of actual positive classes. The F1-score of the model indicates the balance between precision and recall, but in this case, the model is struggling to find the right trade-off between precision and recall. The accuracy, precision, recall, and F1-score of the model are given in Figure 3.

Figure 3

Comparison between accuracy of different models for post quality assessment.

The accuracy of the model is acceptable and higher than all of the models implemented and proposed in the literature. However, precision, recall, and F1-score indicate that there is room for significant improvement in the model. The possible improvement in the model can be achieved by optimizing the model hyperparameters that are already tested by using different numbers of layers, dropout, and max pooling. The second option is to select different feature sets that are already experimented with by different combinations of features. The third possibility is choosing a different model that is also experimented with by four different deep learning models whose detail is given in the model validation section. Finally, the balanced dataset can be used for training the model which may improve the performance of the model.

5 Model validation

Several machine learning and deep learning classifiers are trained to predict the quality of questions and validate the proposed model. The input of each model is 20 features, and the output is a class of questions defined as Very Good, Good, Bad, and Very Bad. A total of 70% data is used for training, 30% data for testing, and 10% data for validation of the model. A detail of each model that is trained with hyper-parameters, and results are given in Appendix A.

The results of all the nine models that are used in experiments are given in Table 7. From the results, one can observe that SVM, KNN, and simple MLP,models are not suitable for judging the quality of StackOverflow questions because of low accuracy. The accuracy of all of these three models is low as compared to deep learning models. But when the number of layers is increased in MLP, there is a significant improvement in the accuracy that can be observed by analysing the Figure 3. When complex models are trained the accuracy is also improving. For example, the accuracy of the MLP-TF model is 78%, whereas, the accuracy of the CNN + LSTM model is 80.45%. The best accuracy of 80.45% is achieved by using the CNN + LSTM model.

From the results, one can observe that SVM, KNN, and simple MLP models are not suitable for judging the quality of StackOverflow questions because of low accuracy. The accuracy of all of these three models is low as compared to deep learning models. But when the number of layers is increased in MLP, there is a significant improvement in the accuracy that can be observed by analysing the Figure 3. When complex models are trained, the accuracy is also improving. For example, the accuracy of the MLP-TF model is 78%, whereas the accuracy of the CNN + LSTM model is 80.45%. The best accuracy of 80.45% is achieved by using the CNN + LSTM model.

After analysing the precision, recall, and F1-score of the models, one can observe that these metrics of the four deep learning models are the same. There is a sudden increase in precision, recall, and F1-score when complex deep learning models are trained, but after changing the models and using hybrid model, values of these metrics do not change. The precision, recall, and F1-score of MLP-TF, CNN, GRU+CNN, and LSTM+CNN models is 57.65%. The recall and F1-score of RNN models are high as compared to other deep learning models. But accuracy and precision of RNN models are lower than the other deep learning models. Therefore, one can say that the best model for the assessment of StackOverflow questions quality is the LSTM+CNN model as the accuracy of this model is high, and precision, recall, and F1-score of this model is also consistent with other deep learning models, i.e. TF Seq., CNN, and GRU+CNN.

6 Comparison and evaluation

The proposed hybrid model of CNN + LSTM for the quality assessment of StackOverflow questions is compared with various existing models reported in the literature [5,8, 11,15,17, 23–26]. Most of the existing techniques for quality assessment are developed by using binary classifiers. The maximum precision of 80–90% is reported by Ponzanelli et al. [15] with binary classification. The results of the comparison are given in the Table 8. In most of the binary classification techniques, machine learning classifiers are used. Only one researcher used a deep learning model, i.e. gated recurrent unit (GRU), and obtained 74% accuracy.

In the literature, only three techniques [11,26,27] uses multi-class classifier (four classes) to classify the question’s quality. These techniques are similar to this work, but out of these two techniques, Ponzanelli et al. [11] used the features of answers to judge the quality of the question. This is based on the assumption that good questions are always answered by community members. This technique is good only for classifying the old questions because answers to new questions are not always available to extract features. During analysis of the literature, it is observed that a technique that can classify the questions at the time of posting is more valuable and will help the moderators to make decisions about the question. Therefore, the focus of this research is to classify the questions without using the features of the answer. The rest of the two techniques, i.e. the studies by Vakil and Mahajan [26,27], are more relevant to this work and use the same dataset and features to assess the quality of StackOverflow questions.

In the first technique, Vakil [26] used three classifiers, i.e. MLP, KNN, and SVM to assess the quality of StackOverflow questions. The best results were achieved by using MLP with 56.53% accuracy, 30.19% precision, 27.49% recall, and 26.39% F1-score. As compared to these results, our proposed model is 23.92% accurate, and the F1-score of the proposed model is also 31.26% better than the results of Vakil.

In the second technique [27], NN, SVM, and logistic regression are used for the assessment of StackOverflow questions quality. This technique also uses the subset of the dataset that is used in this research. The authors use seven features, i.e. ARI, FRE, CLI, gunning index, body length, presence of URL, code to text ratio and title body similarity in their research. Six of these features are also used in this research. The additional features that are used by authors are title and body similarity. The authors used cross-validation to implement all three algorithms and reported the accuracy of their algorithms. The maximum average accuracy of 58.75% is achieved by using SVM and NN classifiers on training data. Results on the test dataset are not reported by the authors. The accuracy of the author’s model is 21.7% lower than accuracy of our proposed CNN + LSTM method, which is computed on the test dataset.

After the comparison, it can be observed that existing models are based on traditional machine learning models and the accuracy of these models is also low. This research is novel in the sense that the existing problem is solved by developing a novel hybrid model of CNN and LSTM. The results of the proposed model are also much higher than the existing techniques.

7 Conclusion and future work

In this research, firstly, an extensive literature review of various articles related to the quality assessment of users’ posts is performed to get insight into the latest trends and find the research gap. From the existing research, it is observed that features-based classification techniques are most popular for the quality assessment of user posts, but an optimal set of features for this purpose is not known. Therefore, the first goal of this research is to select the optimal set of features. Different features that are used in the existing research studies were listed and divided into various categories. After this extensive experiments were performed for the selection of an optimal set of features. After experiments, a set of 20 features were selected for further implementation. In the literature, most of the techniques implemented for content quality assessment are binary classification techniques. While deeply analysing the StackOverflow posts, it is observed that StackOverflow posts quality is classified into four classes. Therefore, existing binary classifiers do not correctly reflect the actual quality of StackOverflow posts. Above in view, a hybrid multi-classifier is implemented for the classification of SO posts. In this article, various experiments with different features and machine learning algorithms are performed to classify StackOverflow questions based on the quality of the questions. After various experiments, 20 features are selected and a hybrid deep learning model that is a combination of CNN and LSTM is developed to classify the questions. The model’s performance is compared with the various other deep learning and machine learning models. The accuracy of the model is higher than all of the other models and state-of-the-art methods proposed in the literature. The proposed model is 23.92% accurate as compared with [26] and 21.7% accurate as compared to [37].

In the future, plan to further improve the accuracy of the proposed model by using different features, classifiers, and data balance methods as there is huge potential to further improve the accuracy. This will improve the recall and F1-score. Testing the performance of the proposed model on other software engineering community question-answering datasets like Quora and MSDN questions is also included in our future plan.