Migrating birds optimization-based feature selection for text classification

Feature selection with filter methods

In Labani, Moradi & Jalili (2020), a novel multi-objective algorithm called Multi-Objective Relative Discriminative Criterion (MORDC) for text feature selection is introduced. A balance between minimizing redundant features and maximizing relevance to the target class is achieved by MORDC. The solution space is explored utilizing a multi-objective evolutionary framework. The relevance of text features to the target class is evaluated by the first objective function, while the correlation among features is assessed by the second function. Importantly, the selected features are evaluated without utilizing learning-based methods, rendering it a multivariate filter approach. Experiments are conducted on three real-world datasets, including WebKB, 20-Newsgroups, Reuters-21578, to evaluate the effectiveness of MORDC. Comparative analyses against state-of-the-art feature selection methods demonstrate that, in most cases, superior classification performance is achieved by MORDC.

In Paniri, Dowlatshahi & Nezamabadi-Pour (2020), a novel multi-label feature selection method named MLACO, based on Ant Colony Optimization (ACO), is introduced by the researchers. MLACO aims to identify the most promising features in the feature space by addressing both relevance and redundancy aspects. This is achieved by incorporating unsupervised and supervised heuristic functions and conducting multiple iterations. To expedite the convergence of the algorithm, the initial pheromone of each ant is set using the normalized cosine similarity between features and class labels. It is important to note that the proposed method does not rely on any learning algorithm and can be categorized as a filter-based approach. To assess its performance, MLACO is compared against five well-known and state-of-the-art feature selection methods using the ML-KNN classifier. Experimental results conducted on commonly employed datasets including 20-Newsgroups, Bibtex, Chemistry, CS, and Cooking demonstrate the superiority of MLACO in terms of various multi-label evaluation measures and run time.

In the study (Uysal & Gunal, 2012), a novel filter-based probabilistic feature selection method named the distinguishing feature selector (DFS) for text classification is introduced. The proposed method is compared to well-known filter approaches such as chi-square, information gain, Gini index, and deviation from the Poisson distribution across various datasets, classification algorithms, and evaluation metrics. The experimental results clearly demonstrate that DFS achieves competitive performance when compared to the aforementioned approaches, as evidenced by its classification accuracy, dimension reduction rate, and processing time.

Network clustering is considered a foundational unsupervised technique in the realm of knowledge discovery, aimed at grouping similar nodes together without supervision or foreknowledge regarding cluster characteristics. Amidst diverse clustering methodologies, semi-supervised clustering detection emerges as a particularly promising avenue for community detection, owing to its capacity to leverage supplementary information to glean deeper insights into network topology. Nevertheless, prior endeavors face twin challenges: reliance on linear techniques for dimensionality reduction and arbitrary selection of auxiliary information, resulting in diminished efficacy of semi-supervised community detection methodologies. To address these shortcomings, an end-to-end framework termed deep semi-supervised community detection is proposed for intricate networks. A novel learning objective (Berahmand, Li & Xu, 2023) is introduced that integrates a semi-autoencoder augmented with a predefined pair-wise constraint matrix grounded on point-wise mutual information within the representation layer, thereby facilitating precise feature acquisition. Furthermore, in the clustering layer, a pair-wise constraint is incorporated as a term to minimize intra-cluster distances while augmenting inter-cluster disparities. Empirical evaluations highlight the remarkable performance of the proposed approach compared to prevailing state-of-the-art community detection methodologies in complex network settings.

Feature selection with wrapper models

In Forman (2004), it is revealed that Information Gain, alongside other existing methods, fails to produce satisfactory results when applied to an industrial text classification problem. Upon further analysis, a common flaw is discovered in a large class of feature scoring methods: they tend to prioritize highly predictive features for certain classes while neglecting features crucial for discriminating challenging classes. Even in a relatively uniform text classification task, this flaw negatively impacts performance. To address this issue, solutions inspired by round-robin scheduling are proposed that effectively circumvent this flaw without resorting to computationally expensive wrapper methods. Through empirical evaluation on a set of 19 multi-class text classification tasks, the proposed method demonstrates significant improvements in performance.

In Hu et al. (2015), a hybrid filter-wrapper approach for feature selection in short-term load forecasting (STLF) is introduced. The current proposition leverages the combined strengths of filter and wrapper methods. Initially, the partial mutual information (PMI) filter method is employed to eliminate irrelevant and redundant features. Subsequently, a wrapper method, employing a firefly algorithm, is applied to further reduce redundancies while maintaining the accuracy of the forecasting process. The established support vector regression model is chosen to implement this innovative hybrid feature selection scheme. To assess the efficacy of the proposed approach, real-world electricity load datasets from a North-American electric utility and the Global Energy Forecasting Competition 2012 are employed. Experimental results conclusively demonstrate the superiority of the proposed approach over comparable methodologies.

In Günal (2012), a comprehensive analysis is conducted to investigate the redundancy or relevancy of text features selected by different methods in various scenarios, including different feature set sizes, dataset characteristics, classifiers, and success measures. To achieve this, a hybrid feature selection strategy is proposed, encompassing both filter and wrapper feature selection steps. The findings from the experimental investigation highlight the superior effectiveness of combining features selected from multiple methods, as opposed to relying solely on features chosen by a single selection method. Nevertheless, the composition of this feature combination is influenced by factors such as dataset characteristics, the selection of the classification algorithm, and the success measure employed.

EGA is proposed in the study (Ghareb, Bakar & Hamdan, 2016), through the enhancement of crossover and mutation operators. The crossover operation is executed by partitioning the chromosome (feature subset) based on the term and document frequencies of chromosome entries (features). Meanwhile, the mutation operation is performed by considering the classifier performance of the original parents and feature importance. Consequently, the crossover and mutation operations utilize valuable information rather than relying on probability and random selection. Moreover, a hybrid approach is developed by integrating six renowned filter feature selection methods with EGA. Within this hybrid approach, EGA is applied to various feature subsets of different sizes, which are prioritized in descending order based on their importance, followed by dimension reduction. The EGA operations are specifically targeted towards the most significant features with higher ranks. The effectiveness of the proposed approach is evaluated by employing naïve Bayes and associative classification techniques on three distinct collections of Arabic text datasets. The experimental results unequivocally demonstrate the superiority of EGA over genetic algorithm (GA). Comparative analyses between GA and EGA reveal that the latter achieves superior outcomes in terms of dimensionality reduction, time efficiency, and categorization performance. Additionally, six hybrid feature selection approaches are introduced, comprising a combination of a filter method and EGA, which are applied to diverse feature subsets. The results exhibit that these hybrid approaches surpass individual filter methods in terms of dimensionality reduction, as they achieve a higher reduction rate without sacrificing categorization precision in most scenarios.

An enhanced binary grey wolf optimizer (GWO) is proposed within a wrapper feature selection (FS) approach to address Arabic text classification problems in Chantar et al. (2020). The proposed binary GWO algorithm is employed as a wrapper-based FS technique. The performance of the proposed method is evaluated using various learning models, such as decision trees, K-nearest neighbour, naïve Bayes, and SVM classifiers. Three Arabic public datasets, namely Alwatan, Akhbar-Alkhaleej, and Al-jazeera-News, are utilized to assess the effectiveness of different wrapper methods based on the binary GWO algorithm. The results and analysis indicate that the SVM-based feature selection technique with the proposed binary GWO optimizer, utilizing an elite-based crossover scheme, demonstrates superior efficacy in addressing Arabic text classification problems compared to other approaches in the field.

Feature selection with embedded methods

In Keyvanpour, Zandian & Abdolhosseini (2022), an embedded approach for feature selection is adopted, utilizing the Chi-square (CHI) feature selector as a filter step to discard less discriminative features. In the subsequent wrapper step, a novel algorithm that combines the fast global search capability of the GA with the positive feedback mechanism of ant colony optimization (ACO) is proposed. To validate the proposed approach, a series of experiments are conducted using the Reuters-21578 corpus, and the results are compared against other well-known techniques. The evaluation outcomes demonstrate that their method outperforms the alternative approaches in the majority of cases, highlighting its effectiveness.

In Nafis & Awang (2020), an embedded feature selection technique that combines Term Frequency-Inverse Document Frequency (TF-IDF) and support vector machine-recursive feature elimination (SVM-RFE) for text classification in unstructured and high-dimensional data is proposed. The proposed technique aims to assess the importance of features in high-dimensional text documents and enhance the efficiency of feature selection, leading to improved text classification accuracy. Initially, TF-IDF is employed as a filter approach to measure the importance of features in the text documents. Subsequently, SVM-RFE is utilized in a backward feature elimination scheme to recursively eliminate insignificant features from the filtered feature subsets. The study conducts experiments using a benchmark repository of Twitter posts, where relevant features are extracted through pre-processing. The pre-processed features are then divided into training and testing datasets. Feature selection is performed on the training dataset by calculating TF-IDF scores for each feature, followed by SVM-RFE for feature ranking. Only the top-ranked features are selected for text classification using the SVM classifier. The experimental results demonstrate that the proposed technique achieves a remarkable accuracy of 98%, surpassing other existing techniques. In conclusion, the proposed technique effectively selects significant features in unstructured and high-dimensional text documents.

In Behera et al. (2021), a hybrid approach is proposed that combines two deep learning architectures, namely convolutional neural network (CNN) and long short term memory (LSTM) using word embedding, for the sentiment classification of reviews across various domains. The suggested Co-LSTM model aims to achieve two primary objectives in sentiment analysis. Firstly, it offers high adaptability for analyzing large-scale social data while considering scalability. Secondly, unlike conventional machine learning methods, it is not limited to a specific domain. The experimental evaluation involves training the model on four diverse review datasets to capture different types of dependencies commonly found in posts. The experimental results indicate that the proposed ensemble model surpasses other machine learning approaches in terms of accuracy and other performance metrics for sentiment analysis task.

An Ontology and Latent Dirichlet Allocation (OLDA)-based approach for sentiment classification is proposed by Ali et al. (2019). The system, developed using Web Ontology Language and Java, retrieves transportation content from social networks and applies OLDA to extract meaningful information by generating topics and features. Additionally, word embedding techniques are employed to represent documents, and lexicon-based approaches are utilized to enhance the accuracy of the word embedding model. The proposed approach is evaluated using machine learning classifiers, demonstrating an accuracy of 93% and confirming its effectiveness in sentiment classification.

The effectiveness of numerical word representation models is investigated in Stein, Jaques & Valiati (2019) through experimentation and analysis. The application of various models and algorithms is examined to assess their performance. Classification models are trained using prominent machine learning algorithms such as fastText, XGBoost, SVM, and CNN. Noteworthy word embedding methods, including GloVe, word2vec, and fastText, are employed for generating word embeddings. Publicly available data is utilized for training and evaluation, employing measures specifically suitable for the hierarchical context. F1 score of 0.893 is achieved by fastText on a single-labeled version of the RCV1 dataset. The analysis indicates that employing word embeddings and their variations presents a highly promising approach for hierarchical text classification.

In the study (Berahmand et al., 2024), authors attempt to furnish such an exhaustive survey, commencing with an elucidation of the foundational principles underlying conventional autoencoders and their developmental trajectory. Subsequently, a systematic classification of autoencoders based on their architectural configurations and operational principles is proffered, meticulously scrutinizing and deliberating upon associated models. Additionally, a comprehensive review of autoencoder applications across diverse domains such as machine vision, natural language processing, complex network analysis, recommender systems, speech processing, anomaly detection, among others, is undertaken. Finally, the current constraints inherent in autoencoder algorithms are synthesized and prospective avenues for future advancements within this realm are deliberated upon.

Feature selection with heuristic optimization-based techniques

In Purushothaman, Rajagopalan & Dhandapani (2020), a technique demonstrating a high level of maturity in convergence rate and requiring minimal computational time is proposed, thereby avoiding entrapment in local minima within a low-dimensional space. The input comprises text data, which undergoes initial preprocessing steps within the document. Subsequently, text feature selection is carried out by identifying local optima from the text document, and then the best global optima are selected from these local optima using a hybrid GWO-GOA approach. Additionally, the selected optima are subjected to clustering through the utilization of the Fuzzy c-means (FCM) clustering algorithm. This algorithm enhances reliability and minimizes computational time expenditure. The proposed algorithm is tested using eight datasets, exhibiting efficient performance. Evaluation metrics, including accuracy, precision, recall, F-measure, sensitivity, and specificity, are employed to evaluate text feature selection and text clustering. Comparisons with various other algorithms reveal that the proposed methodology exhibits superior quality, surpassing GWO, GOA, and the hybrid GWO-GOA algorithm, with an efficiency rate of 87.6%.

In Moh’d Mesleh & Kanaan (2008), the implementation of a text classifier utilizing the support vector machine (SVM) model for Arabic articles has been undertaken. Furthermore, a novel feature subset selection (FSS) method based on Ant Colony Optimization (ACO) and Chi-square statistic has been developed and implemented. The proposed ACO-Based FSS method incorporates Chi-square statistic as heuristic information and leverages the efficacy of the SVM classifier as a guide to enhance feature selection for each category. In comparison to six prevailing FSS methods, the ACO-Based FSS algorithm demonstrates superior effectiveness in text classification. The evaluation utilizes an internally developed corpus for Arabic text classification, comprising 1,445 documents that have been independently categorized into nine distinct classes. The experimental results are presented in terms of macro-averaging precision, macro-averaging recall, and macro-averaging F1 measures.

In Jain & Kashyap (2023), the sentiment categorization of text is presented, encompassing pre-processing, feature vector extraction, feature selection, and classification procedures. The utilization of the grey wolf technique is employed for feature selection, extracting salient text features from the feature vector. Additionally, a deep learning neural network is employed for classifying the text into positive and negative sentiments. The proposed model is validated using publicly available datasets from Twitter (sentiment140) and movie reviews (IMDb). Remarkable achievements are observed, with precision, accuracy, recall, and F-measure reaching the highest levels of 94.16%, 95.81%, 92.87%, and 95.22%, respectively.

In Hosseinalipour et al. (2021), the integration of the Black Widow Optimization (BWO) algorithm into a binary algorithm for addressing discrete problems is initially presented. The utilization of opposition-based inputs is employed to expedite the attainment of optimal solutions. Furthermore, to tackle the property selection problem, which is characterized by multiple objectives, the algorithm is transformed into a multi-objective algorithm. To evaluate the performance of the proposed method, 23 well-known functions are utilized. The results demonstrate favorable outcomes. Additionally, a practical example involving the application of the proposed method to various emotion datasets is considered. The findings indicate the remarkable efficacy of the proposed method in the domain of text psychology.

A new method called meta-heuristic feature optimization (MHFO) for data security in the campus workplace with robotic assistance is proposed in Gong, Dinesh Jackson Samuel & Pandian (2021). Firstly, the terms of the space vector model are mapped to the concepts of data protection ontology, which enables the calculation of conceptual frequency weights by considering various weights assigned to each term. Moreover, based on the designs of the data protection ontology, the weight of theoretical identification is assigned. The combination of standard frequency weights and weights derived from the data protection ontology significantly reduces the dimensionality of functional areas. Additionally, this process integrates semantic knowledge. The results demonstrate that the advancement of the characteristics within this process leads to a notable improvement in secure text mining in the campus workplace. In conclusion, the experimental findings indicate that enhancing the features of the concept hierarchy structure process substantially enhances the data security of campus workplace text mining when robotic assistance is employed.

Migration birds optimization

The MBO model (Duman, Uysal & Alkaya, 2012) is inspired by the behavior of bird flocks during migration, specifically N birds, with N being an odd number, flying in a V-shaped formation (Fig. 1). Each bird represents a potential solution, undergoing refinement through the course of flights. A random bird is chosen as the leader, guiding the entire flock, while the remaining birds follow in two orderly lines, maintaining the V-shaped formation (Fig. 1). This innovative approach leverages the dynamics of a bird flock to enhance the exploration and exploitation capabilities of the optimization algorithm.

In Table 1, the detailed explanation of each parameter used in the study is given to provide a better understanding of the proposed framework. Algorithm 1 outlines the MBO, which receives an input of $M\times N$ data and the number of birds in the flock, and strives to enhance the classification outcome by eliminating less significant features.

In the initial step, it calculates the fitness of the input data, which is a way to see how well the data works within the context of different solutions (is discussed in “Fitness function” in detail), denoted as F where $0\le F\le 1$ (line $1$ of Algorithm 1). In line $2$, we assign the initial fitness F value to 3 different variables in order to observe the improvement of the fitness value. More specifically, $F_i$ ( $1\le F\le 3$) retains the fitness value from the fitness values of the iteration $i$ steps earlier (line $20$). Hence, when the difference between $F_1$ and $F_3$ reaches $0$ (line $8$), it indicates that the same result has been achieved in the last three steps. At this point, we conclude the algorithm, presuming that no additional improvement can be achieved.

At line $6$, the flock is initialized with an odd number of birds. An illustration of a flock with seven birds is presented in Fig. 1. The bird positioned at the forefront, i.e., at the corner of the V-formation, is designated as the leader of the flock. The flock iteratively engages in flight and landing cycles, wherein optimizations are applied to enhance the flock’s performance. After completing a cycle, a new leader is chosen based on the bird with the highest fitness value, and the position of the current leader is replaced with the newly selected leader (line $19$) while the positions of the remaining birds unchanged. Subsequently, the flock initiates another flight, following the same sequence of steps, and this process continues iteratively until the flock reaches its desired destination. This iterative process contributes to the ongoing refinement of the solutions explored by the bird flock.

We depict the steps carried out between lines $8$– $22$ in Fig. 2. Each step within the upper large box of Fig. 2 corresponds to lines $11$ in Algorithm 1. In these steps, we make some changes in the solutions which will be explained in detail in the following paragraph, with the goal of discovering more optimized solutions, referred to as birds, throughout the flight. The number of changes is decided by the number of $change$ variable as given in line $9$. For instance, if the $change$ is $3$, we make three changes in the current solution. If an optimization could have been made, the global best fitness value (lines $14$– $17$) is updated. After $10$ steps, a number determined through previous experiments, we reorder the flock by replacing the bird with the highest fitness value with the leader (line $19$). The algorithm terminates when either no optimization is possible, i.e., the last three steps produces the same results, or it reaches max number of iterations, that is $100$ (line $8$). Then, the best solution found so far $F_{max}$ is returned as output.

Each flock fly operation (line $10$), i.e., shown as arrows in the Fig. 2, corresponds to the operations given in Fig. 3. We, for simplicity, elaborate a single iteration (line $10$) with a hypothetical flock having five birds $\{b_1,b_2,b_3,b_4,b_5\}$ where $b_1$ is the leader, $b_2$ and $b_3$ follows the leader and $b_4$ and $b_5$ follows $b_2$ and $b_3$, respectively, in V-shaped formation as illustrated in Fig. 3A. In Fig. 3B, every bird $b_i$ generates a unique set of neighbors $B_i$ by creating similar solutions as itself through a small number of changes (neighbor generation). Then, in Fig. 3C, the leader $b_1$ replaces itself with the best solution selected from the set $\{b_1\}\cup B_1$ and selects two more best solutions, i.e., the second ${b_1}^{\mathrm{\prime }\mathrm{\prime }}$ and the third best ${b_1}^{\mathrm{\prime }\mathrm{\prime }\mathrm{\prime }}$ solutions, and ${b_1}^{\mathrm{\prime }\mathrm{\prime }}$ is added to its left set of birds and the ${b_1}^{\mathrm{\prime }\mathrm{\prime }\mathrm{\prime }}$ is added to its right set of birds. Thus, the set of birds on the left side of the leader becomes $\{b_2\}\cup B_2\cup \{{b_1}^{\mathrm{\prime }\mathrm{\prime }}\}$ and the set of birds on the right side of the leader becomes $\{b_3\}\cup B_3\cup \{{b_1}^{\mathrm{\prime }\mathrm{\prime }\mathrm{\prime }}\}$ (Fig. 3C). In the fourth step (Fig. 3D), the birds $b_2$ and $b_3$ replace themselves with the best solution selected from the sets $\{b_2\}\cup B_2\cup \{{b_1}^{\mathrm{\prime }\mathrm{\prime }}\}$ and $\{b_3\}\cup B_3\cup \{{b_1}^{\mathrm{\prime }\mathrm{\prime }\mathrm{\prime }}\}$, respectively. Subsequently, both $b_2$ and $b_3$ selects their second best solutions, e.g., ${b_2}^{\mathrm{\prime }\mathrm{\prime }}$ and ${b_3}^{\mathrm{\prime }\mathrm{\prime }}$, and they are added to the left and right set of birds, respectively. In the last step, the birds $b_4$ and $b_5$ replaces themselves with the best solutions selected from the sets $\{b_4\}\cup B_4\cup \{{b_2}^{\mathrm{\prime }\mathrm{\prime }}\}$ and $\{b_5\}\cup B_5\cup \{{b_3}^{\mathrm{\prime }\mathrm{\prime }}\}$, respectively. Finally, the new flock contains { ${b_1}^{\mathrm{\prime }}$, ${b_2}^{\mathrm{\prime }}$, ${b_3}^{\mathrm{\prime }}$, ${b_4}^{\mathrm{\prime }}$, ${b_5}^{\mathrm{\prime }}$} birds.

Modeling the feature selection as heuristic problem

In this section, we are not solely approaching the feature selection problem as an MBO problem. Instead, we present a generic modeling approach. This approach enables the utilization of various heuristic optimization techniques, provided they meet the specified requirements, for the feature selection.

Traditional heuristic methods typically require two key components: a fitness function and a neighbor generation function. The fitness function serves as a crucial evaluation metric, quantifying the quality or effectiveness of a potential solution within the problem space. It assigns a numerical value to each solution, indicating its performance with respect to the optimization goal. On the other hand, the neighbor generation function determines how neighboring solutions are generated. This function plays an important role in the exploration of the solution space, influencing the search strategy by defining the set of neighboring solutions considered for potential improvements. Together, the fitness function and the neighbor generation function form the core elements guiding the heuristic search process, allowing the algorithm to iteratively evaluate and refine solutions in the pursuit of optimal outcomes.

Therefore, we first introduce how we define a solution for feature selection problem and then present novel fitness and neighbor approaches for the problem in the following sections.

MBO as feature selection

MBO can be regarded as a conventional heuristic optimization technique, incorporating standard practices for fitness functions and neighbor utilization. Therefore, MBO is well-suited to serve as an effective approach for addressing the feature selection problem. This section outlines our utilization of MBO to tackle the feature selection problem.

We start with employing Info Gain (Quinlan, 1986; Karegowda, Manjunath & Jayaram, 2010) feature selection algorithm to reduce the feature set to a manageable size. This initial reduction is essential as MBO’s efficiency is exponentially influenced by the number of features. The reduced data resulting from Info Gain serves as an input for MBO. Subsequently, we feed this reduced data into MBO, allowing heuristic operations to identify and select the most beneficial features among the remaining set. This approach ensures that, in the worst-case scenario, our selection aligns with the features identified by Info Gain. The Info Gain algorithm selects all features contributing to classification, unless the total selected features surpasses 2,500 which is a limitation imposed for performance reasons of MBO. We, then, use the reduced data as input for MBO.

We already explained the details of Algorithm 1 in this section. It takes this reduced M × N data as input and produces a M′ × N solution by selecting M′ best features out of initial M features. In Fig. 4, the block diagram of MBO algorithm is given for the purpose of demonstrating the feature selection process.

Data

In the experiments, we utilized a diverse set of data, each characterized by specific features, instances, and classes. The details of the data are outlined in Table 5. The table presents key information regarding five different data: 20News-18828 (Mitchell, 1997), aahaber (Tantuğ, 2010), hurriyet (Kilimci & Akyokus, 2018), mininews (Kilimci & Akyokus, 2018), and webkb4 (Craven et al., 1998).

In Table 5, the “no of feature” column indicates the number of features or attributes associated with each data. The “no of instances” column denotes the total number of instances or data points in each data. The “no of classes” column represents the number of classes or categories present in each data. The “avg. word count per instance” column provides the average word count per data instance whereas the “avg. word length” column indicates the average length of words in each data. And lastly, the language column gives the language of the text used in the study. For instance, the data “hurriyet” contains 71,071 features, 6,006 instances, and six classes (or categories). The average word count per instance and average word length in data are $158.3$ and $7.1$, respectively. The text language is in Turkish.

We initially acquired the data in its raw textual format. During the data preprocessing phase, we employ tokenization to break down the text into individual units, followed by the removal of stop words to enhance the efficiency of subsequent analyses. Then, we utilized Term Frequency-Inverse Document Frequency (TF-IDF) (Joachims, 1997) to evaluate the importance of terms in a document relative to a collection of documents. Finally, we construct feature vectors, representing the processed data in the feature vector format to further computational operations.

These data with varying characteristics provide a rich set of examples for our experiments, allowing us to explore and analyze the performance of different models across diverse data scenarios.

Models

In this section, the models employed in this research are briefly introduced. In the scope of this work, Information Gain (Quinlan, 1986), PSO (Abualigah, Khader & Hanandeh, 2018) and MBO algorithms are assessed as feature selection techniques while naïve Bayes, Decision Tree, multilayer perceptron, and support vector machine, are evaluated as machine learning techniques. In each model creation, use used five-fold cross-validation technique.

The naïve Bayes algorithm, widely employed for classification tasks, operates based on the principles of Bayes’ theorem (Dale, 2005; Xu, 2018), assuming conditional independence of features given the class label. The algorithm’s functioning entails the preparation of labeled data for training, estimation of probabilities using Bayes’ theorem, and prediction of class labels based on the highest probability.

Decision Tree algorithm (Quinlan, 1986; Charbuty & Abdulazeez, 2021) involves recursively partitioning the input data based on feature values to create a hierarchical structure resembling a tree. At each internal node of the tree, a decision rule is applied to determine the optimal feature to split the data, maximizing the homogeneity within each resulting subset. The process continues until a stopping criterion is met, typically when the data subsets reach a predetermined purity threshold or a specified depth is reached. The resulting tree can then be used to make predictions for new, unseen instances by traversing the tree from the root node to a leaf node, following the decision rules at each internal node.

In multilayer perceptron (MLP), information moves forward from one layer to the next, starting with the input nodes, then progressing through hidden layers (which can be one or more layers), and finally reaching the nodes in the output layer (Tuan Hoang et al., 2021). These networks are made up of a mix of simpler models known as sigmoid neurons. MLPs consist of many nodes, and sigmoidal functions are commonly used as activation functions. MLPs have the ability to grasp the complex and nonlinear decision boundaries from data. They typically have one input layer, one output layer, and multiple hidden layers positioned between the input and output layers. The inclusion of more hidden layers enables MLPs to understand more intricate nonlinear relationships between the input and output.

Particle Swarm Optimization (PSO), introduced by Kennedy & Eberhart (1995), represents an emergent population-based meta-heuristic. This method emulates social behaviors, specifically the phenomenon of birds flocking towards a promising position to achieve precise objectives within a multidimensional space. Similar to evolutionary algorithms, PSO conducts searches utilizing a population, referred to as a swarm, composed of individuals termed particles. These particles undergo iterative updates to enhance their positions. In the pursuit of discovering the optimal solution, each particle adjusts its search direction based on two influential factors: its individual best previous experience and the collective best, incorporating both cognitive and social aspects. It is characterized by its straightforward implementation requiring few parameters, and it has found extensive application in addressing optimization problems, including those related to feature selection (Lin et al., 2008).

Support vector machines algorithm (Vapnik, 2013; Pisner & Schnyer, 2020) involves identifying an optimal hyperplane in a high-dimensional feature space to separate different classes of data points. SVM achieves this by maximizing the margin, which is the distance between the hyperplane and the nearest data points from each class, known as support vectors. This optimal hyperplane is determined by solving a convex optimization problem that aims to minimize classification errors while maximizing the margin.

Information Gain revolves around quantifying the relevance of features in a data by measuring their contribution to the reduction of uncertainty during the classification process (Quinlan, 1986; Azhagusundari & Thanamani, 2013). It calculates the information gain by evaluating the difference in entropy (or impurity) before and after the feature is considered. Features with higher information gain are deemed more informative and are thus selected for further analysis.

Research limitations

Despite aforementioned significant contributions, this proposed framework has some limitations.

• Limited adaptability to diverse data and problem contexts: While MBO-NB demonstrates effectiveness in feature reduction and classification accuracy improvement, its adaptability to diverse datasets and problem contexts may be limited. The current approach may not be optimized for specific domains or may exhibit varying performance across different types of data.

• Dependency on preprocessing techniques: The success of MBO-NB relies on preprocessing techniques, such as the Information Gain algorithm, to strategically reduce the feature count. However, the efficacy of these preprocessing steps may vary depending on the nature and characteristics of the dataset. There may be scenarios where the chosen preprocessing techniques are not optimal, leading to suboptimal feature selection outcomes.

• Lack of robustness to noisy or imbalanced data: MBO-NB may face challenges in handling noisy or imbalanced datasets, which are common in real-world applications. The method may not adequately account for the presence of noise or imbalance in the data, potentially leading to biased or unreliable classification results.

• Limited exploration of advanced techniques: While MBO-NB integrates Migrating Birds Optimization with naïve Bayes, it may not leverage more advanced techniques, such as deep learning algorithms and text-based transformer approaches. These advanced methods have shown significant promise in text classification tasks and may offer opportunities for further improving the performance and robustness of MBO-NB.

By addressing these weaknesses and exploring potential areas for improvement, future research can enhance the robustness, versatility, and applicability of MBO-NB in text classification tasks. Additionally, further investigation into the integration of advanced techniques and the development of more adaptive and scalable methodologies can contribute to the advancement of feature selection methods in text classification.

Potential future research

The results of this research emphasize the effectiveness of MBO-NB as a feature selection algorithm. The careful selection of classifiers, coupled with strategic preprocessing steps, contributes to the algorithm’s efficiency and performance. The findings presented in this study provide valuable insights into the application of MBO-NB in diverse scenarios, with implications for data-driven decision-making processes.

As we move forward, future research endeavors should thoroughly explore specific domains, examining the adaptability of MBO-NB to varying data and problem contexts. The insights gained from this research contribute to the changing nature of feature selection methodologies, opening avenues for further exploration and refinement. Furthermore, another potential area for future investigation could be the integration of deep learning algorithms and text-based transformer approaches within the MBO framework, alongside the incorporation of word embedding techniques, presents intriguing avenues for future investigation, promising advancements in the field of text classification.

Based on the limitation of this article and the computational results, we propose some potential directions for future research.

• Hybridization with other optimization algorithms: Investigating the potential benefits of hybridizing MBO with other optimization algorithms, such as genetic algorithms or simulated annealing, can be a promising direction. Hybrid approaches may leverage the strengths of multiple algorithms to overcome individual weaknesses and achieve superior feature selection outcomes in text classification.

• Integration with deep learning models: Considering the recent advancements in deep learning models for natural language processing, future research can explore the integration of MBO-based feature selection methods with deep learning architectures, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs). This integration can leverage the expressive power of deep learning models while harnessing the efficiency of MBO for feature selection.

• Adaptation to multimodal data: With the increasing availability of multimodal data, including text, images, and audio, future research can investigate the adaptation of MBO-based feature selection methods to handle multimodal classification tasks. This involves extending the methodology to select relevant features from diverse modalities and integrating them into a unified classification framework.

• Scalability and efficiency: Addressing the scalability and computational efficiency of MBO-based feature selection methods is crucial for handling large-scale text classification tasks. Future research can investigate strategies to parallelize and distribute the computation, optimize memory usage, and develop scalable implementations to accommodate massive datasets and high-dimensional feature spaces.

By exploring these potential research directions, future studies can advance the state-of-the-art in MBO-based feature selection methods for text classification, leading to more effective, efficient, and interpretable solutions for real-world applications.