2.2.1. Inclusion Criteria

Studies were included if they:

1. Investigated feature selection involving the explicit selection, ranking, weighting, masking, or removal of original input variables in a high-dimensional dataset.
2. Evaluated statistical, computational, machine-learning, optimisation-based, or hybrid feature-selection method.
3. Reported an empirical assessment using real-world, publicly available, simulated, or recognised benchmark data.
4. Applied feature selection within a classification, regression, survival-analysis, or related predictive-modelling task.
5. Published as peer-reviewed English-language journal research articles between January 1, 2020 and December 31, 2025.

2.2.2. Exclusion Criteria

Studies were excluded if they:

1. Applied only feature transformation or latent representation methods, such as PCA, autoencoders, or representation learning, without explicitly selecting, ranking, weighting, masking, or removing original variables.
2. Presented a theoretical, conceptual, or methodological proposal without empirical evaluation.
3. Did not focus on high-dimensional data or predictive modelling.
4. Were review articles, conference papers, editorials, book chapters, protocols, short communications, or non-peer-reviewed manuscripts.
5. Did not provide sufficient methodological information to determine how the original variables were selected or evaluated.

2.3. Data Extraction

A structured data-extraction protocol was applied to all included studies to improve consistency and comparability. Information was extracted concerning study characteristics, feature-selection methodology, application context, data structure, evaluation design, and reported findings. The following information was recorded for each study:

• Authors, publication year, and publication source.
• Feature-selection method and conventional methodological category.
• Statistical, computational, machine-learning, or optimisation procedures.
• Application domain.
• Dataset source and relevant data characteristics.
• Predictive learner and evaluation design, where reported.
• Performance measures, including accuracy, AUC, F1-score, MCC, RMSE, MAE, C-index, recall, or error rate.
• Direction and interpretation of the reported result relative to the study’s stated comparator.

This structured extraction supported a systematic comparison of methodological categories, application domains, evaluation practices, data-regime considerations, and the direction of reported findings across the included studies (Table 2).

This structured extraction allowed systematic comparing of cross-studies and synergy among classes of methods.

2.4. Screening and Quality Assessment

The screening process was conducted in two stages.

1. The titles and abstracts of the 369 unique records were assessed against the predefined eligibility criteria.
2. The 94 full-text reports retrieved were evaluated to determine final eligibility. Two reviewers participated in the screening and full-text assessment processes, and disagreements were resolved through item-by-item discussion and consensus.

Methodological quality was assessed using a customised checklist informed by systematic-review reporting principles and machine-learning evaluation considerations. The assessment examined the clarity of study objectives, appropriateness of the feature-selection method, adequacy of experimental validation, integrity of the feature-selection pipeline, transparency of evaluation reporting, and reproducibility of the analysis.

Inter-reviewer agreement was evaluated using the paired decisions presented in Table 3. The resulting Cohen’s kappa coefficient was κ = 0.6082. Under commonly used interpretation thresholds, this value lies at the boundary between moderate and substantial agreement and should not be described unconditionally as “strong agreement.” All disagreements were resolved through discussion before the final inclusion decisions were confirmed.

The agreement matrix contains 50 paired reviewer decisions. The manuscript should clearly state whether these decisions represent a documented subsample or a particular screening stage. If the 50 decisions do not constitute the actual agreement dataset, Cohen’s kappa should be recalculated using the complete retained reviewer-decision log.

2.5. Evidence-Synthesis Strategy

The included studies differed substantially in their application domains, datasets, sample sizes, dimensionality ratios, outcome prevalence, predictor structures, predictive learners, comparator methods, hyperparameter-tuning procedures, validation designs, and reported performance measures. The studies were therefore not treated as statistically exchangeable. Accuracy, AUC, F1-score, MCC, RMSE, MAE, C-index, recall, error rate, and other reported outcomes were retained in their original forms and were not pooled or converted into a common performance scale.

The evidence synthesis combined five complementary components:

1. Structured study tabulation
2. Direction-of-effect vote counting
3. Thematic synthesis
4. Methodological quality appraisal
5. Domain-by-method evidence mapping

A favourable direction of effect indicated that an included study reported improvement relative to at least one comparator evaluated within that study. Vote counting was used only to describe the direction of published findings. It was not used to estimate a pooled effect size, calculate statistical significance across studies, or establish the superiority of one feature-selection category over another.

No cross-study means, weighted averages, pooled standard deviations, normalised performance scores, ANOVA tests, or category-level inferential comparisons were calculated. When an individual study reported results for multiple datasets, classifiers, or feature-selection configurations, those findings were described within the context of that study rather than reduced to a single aggregated value. This approach prevented studies with numerous data sets or experimental combinations from exerting disproportionate influence over the synthesis.

The conclusions of the review therefore concern recurring methodological patterns, practical trade-offs, validation quality, stability, causal awareness, and suitability for different data regimes. They do not represent numerical rankings of feature-selection categories.

2.6. Quality and Risk-of-Bias Assessment

Each study was rated from 0 to 2 across five domains:

• Validation quality
• Dataset quality
• Selection-pipeline integrity
• Reproducibility
• Reporting and risk of bias

The detailed scoring rules used for the five quality-appraisal domains are presented in Table 4. These criteria were applied consistently across all included studies to support a structured and transparent assessment of methodological quality and risk of bias.

Using the criteria defined in Table 4, each included study was assigned a score for validation quality, dataset quality, selection-pipeline integrity, reproducibility, and reporting or risk of bias. The resulting domain-level scores, total scores, and overall methodological-quality judgements are presented in Table 5.

2.7. Inter-Reviewer Agreement

Inter-reviewer agreement was evaluated using Cohen’s kappa. The calculated coefficient was κ = 0.6082. Under the commonly cited Landis and Koch interpretation thresholds, values from 0.41 to 0.60 are generally described as moderate agreement, whereas values from 0.61 to 0.80 are described as substantial agreement. The obtained coefficient lies close to the boundary between these categories and is therefore interpreted as moderate-to-substantial agreement rather than unequivocally strong agreement. Disagreements were reviewed individually and resolved through discussion and consensus before the final study-selection decisions were established.

3. FEATURE SELECTION METHODS FOR HIGH-DIMENSIONAL DATA

One of the key preprocessing tasks in predictive modelling, usually in a high-dimensional predictive situation where the number of variables greatly outweighs the number of observations, is feature selection. High-dimensional data is often present in areas like genomics, bioinformatics, medical diagnostics and sensor-based system where features that are redundant, irrelevant, or noisy can grossly diminish model performance. The objective of feature selection is to identify an informative subset of variables while reducing redundancy, computational burden, and the risk of overfitting.

In predictive modelling, feature selection has numerous applications. First, it increases model interpretability by lessening complexity thus facilitating better understanding of relationships between predictors and the outcomes. Second, it enhances the generalization performance through the curse of dimensionality mitigation and avoiding over-fitting, particularly when operating in small sample size and high dimensional contexts. Third, it enhances computational efficiency which is essential in large scale or real time. In the recent research, including the works of [8, 10], it is confirmed that successful feature selection may greatly contribute to the increase of classification accuracy and a decrease in the complexity of models.

The need to design robust, adaptive and scalable feature selection methods that can process noise, correlated and heterogeneous data are becoming important in the research of feature selection in contemporary times. Consequently, there exists a broad spectrum of techniques both in the statistical, machine learning, and evolutionary paradigms that have been devised and utilized in various areas of application.

3.1. Analytical Refinement of Feature Selection Taxonomy

Although the classical taxonomy of feature selection methods into filter, wrapper, embedded and hybrid categories offers a good organizational grounding, it falls short in explaining behavior in performance in modern high-dimensional learning settings. More specifically, taxonomy is primarily procedural rather than theoretical, classifying methods according to how they are conducted, as opposed to why they perform well or poorly under the particular statistical conditions.

The failure of the classical taxonomy to describe cross-category convergence is a critical weakness of the taxonomy. As an example, the current embedded techniques using regularization (e.g. LASSO) implicitly adopt filtering by shrinking coefficients, and the hybrid techniques often impose stability constraints or causal priors and operate without depending on a wrapper-based optimization approach. Through this, there is blurring of strict categorical boundaries in practice.

To overcome these shortcomings, this review suggests an improved analytic taxonomy using statistical behavior instead of algorithm structure. In particular, the methods of feature selection can be reclassified in the three orthogonal dimensions:

(i) Methods that are aware of stability, and take explicit control over selection variability to perturbation of the data (e.g. stability selection, ensemble-based filters)

(ii) Causal-conscious approaches that seek to detect features whose relevance is structural and not merely predictive (e.g. causal feature selection, treatment-effect-oriented selection)

(iii) Data-regime-adaptive methods, which change their selection mechanism depending on the sample size and number of dimensions compared to noise (p ≫ n), structure of noise, and how much the features are correlated.

Within this framework, the traditional filter, wrapper, embedded, and hybrid methods are not mutually exclusive classes but implementation-level realizations of more profound statistical goals. This approach allows making a more principled comparison between studies and will not result in the repetition of taxonomy.

3.2. Categories of Feature Selection Methods

Methods of feature selection are generally described as either filter, wrapper, embedded or hybrid, depending on the way that feature relevance is measured and incorporated into the learning process. However, as machine learning evolves and datasets become increasingly complex, these categories need to be refined and extended to account for the growing diversity in real-world applications. We propose the following extended taxonomy:

3.2.1. Filter Methods

Filter methods consider features without any predictive model based on the statistical measures of relevance. The typical measures can be correlation coefficients, mutual information, chi-square and entropy-based measures. While they are computationally efficient, they assume feature independence and do not account for feature interactions. These methods can be extended to include stability-aware filters, where feature importance is evaluated by stability over multiple runs of different data splits or noise perturbations.

A number of reviewed studies employed filter-based approaches, for example, [6] employed correlation-based ranking and Matthews Correlation Coefficient (MCC) to rank informative features in high-dimensional gene expression data, and found that it was stable in small-sample settings. Equally, [28] used a variation of mutual information criteria to improve the COVID-19 prediction accuracy using clinical data. Although filter methods are scalable and simple, they do not in many cases resolve feature dependencies which may restrict predictive effectiveness in complicated datasets.

3.2.2. Wrapper Methods

Wrapper methods also consider sets of features by direct maximization of the performance of a predictive model. These methods usually entail the use of iterative methods of searching, including genetic algorithms, particle swarm optimization or recursive feature elimination. The wrapper methods can be more accurate than the filter methods due to the ability of the former to consider the interactions between features and since they also have a higher computational cost.

A number of the studies in the examined corpus use wrapper-based approaches. As an example, [6] used classification accuracy as the feature selection evaluation metric based on a recursive ensemble-based feature selection. Likewise, [21] also introduced a Binary Salp Swarm Algorithm (BSSA), which combines swarm intelligence and wrapper evaluation, and demonstrated a good performance on various benchmark datasets. But wrapper methods are computationally costly, and are likely to be overfit, especially when used on small or noisy data.

3.2.3. Embedded Methods

Embedded methods involve the model training process that is directly embedded with feature selection. Intrinsic techniques like LASSO, Elastic Net, and tree-based methods are techniques that do variable selection by penalizing the complexity of the model or by estimating the importance of features during training.

Many studies embrace embedded strategies because of their efficiency and predictive ability. As an illustration, [7] used regularized regression in their highMLR model to choose informative genomic features to use in survival analysis. In a similar manner, the embedded mechanism that [18] utilized in a multigranularity fuzzy autoencoder was to detect discriminative features in biomedical data. Such techniques have the advantage of being less computationally expensive and having better generalization, but in many cases are model-specific and less applicable to different problem domains.

3.2.4. Hybrid Methods

Hybrid feature selection algorithms have a combination of two or more algorithms- generally filter and wrapper- to use their two-way benefits. These approaches are meant to provide a balance between computational efficiency and predictive accuracy and strength.

The hybrid strategies are a big percentage of recent studies. An example of this is the case of [3], who combined cooperative co-evolution and feature selection to efficiently search high-dimensional search spaces. On the same note, [8] suggested a hybrid AI system that incorporates statistical filtering and machine learning selection in enhancing the accuracy of classification across various datasets. Hybrid techniques have been especially useful in dealing with a large number of dimensions and removing redundancy without losing model interpretability.

3.3. Proposed Analytical Framework

The review proposes a complementary conceptual framework consisting of three dimensions:

• Stability awareness: whether a method evaluates or controls variation in selected features across resamples, perturbations, repeated fits, or parameter settings.
• Causal awareness: whether relevance is defined through causal structure, treatment effects, adjustment requirements, or intervention-related reasoning rather than predictive association alone.
• Data-regime adaptivity: whether the method explicitly addresses a particular statistical condition, including , predictor correlation, small samples, class imbalance, censoring, multimodal data, streaming observations, structured predictors, or unequal costs.

Table 6 applies the proposed three-dimensional analytical framework to the 40 included studies. It identifies each study’s conventional methodological category and indicates whether the method demonstrates stability awareness, causal awareness, or adaptation to a particular data regime, together with the principal basis for its classification.

3.4. Statistical Foundations of Feature Selection

The techniques of feature selection are based on the statistical learning theory, specifically, the notions of the bias-variance trade-off, regularization and information theory. Normalization L1 (LASSO) and L2 (Ridge) methods are regularization which promote sparsity and discourage overfitting, by sending a penalty to huge coefficients. Some of the reviewed papers, such as the ones by [4, 30], used LASSO-based frameworks to improve the generalization performance in medical prediction tasks.

Mutual information and entropy are some of the information-theoretic measures used extensively to estimate the dependency between features and target variables. Most of the filter-based and hybrid methods are based on these measures, including the modified mutual information method suggested by [28]. Also, Bayesian techniques, such as Bayesian networks and probabilistic graphical models, can be used to quantify uncertainty as well as estimate feature relevance, which is shown by [14].

More recently the use of ensemble methods as well as metaheuristic methods, including genetic algorithms, particle swarm optimization and swarm intelligence, have been increasingly prevalent because of their capability to effectively search large search spaces. [4, 37] emphasizes the knowledge on such techniques to enhance strong performance and stability in high-dimensional feature selections.

3.5. Bias–Variance Trade-off and Statistical Limitations of Feature Selection

The bias-variance tradeoff is a major concern when it comes to defining the usefulness of feature selection methods in high-dimensional environments but is commonly addressed only in broad strokes. In practice, feature selection algorithms are biased because they reduce dimensionality and aim to decrease the variance of estimators caused by redundant or noisy predictors. These two effects are extremely sensitive to both regime of data and model assumptions. Regularization-based embedded regression techniques, like LASSO, minimize variance by imposing sparsity with an ℓ1 penalty, but this advantage depends on linearity and fails when there is strong collinearity among the predictors. In p ≫ n regimes, LASSO can randomly choose one feature of a set of correlated features, resulting in unstable and non-identifiable solutions.

Moreover, ℓ₁ regularization is not statistically consistent in the presence of irrepresentable condition, which restricts its asymptotic accuracy in correlated high-dimensional data. The use of information-theoretic filter methods, especially those based on mutual information, inject bias due to finite-sample estimation. Mutual information estimators are also sensitive to discretization strategy, bandwidth of the kernel and the sample size, as they tend to have an upward bias in small sample regimes. The curse of dimensionality also progressively worsens the estimator consistency as dimensionality grows and in turn makes relevance scores unreliable without bias correction or ensemble estimation techniques. These failure modes underline why strategies that perform well empirically can deteriorate drastically when they are not in their optimal operating regions. Therefore, comparisons of performance that fail to correct for estimator bias, variance inflation and identifiability constraints, are likely to overestimate generalizability.

3.6. Emerging Trends and Research Directions

Recent studies point to an apparent movement towards the hybrid and learning-based feature selection models where the statistical rigor is combined with adaptive learning functions. Autoencoders and attention mechanisms are also deep learning-based methods that are being used to automatically discover the levels of feature importance in data. Research, like deep architectures, includes [12, 18], is used to illustrate the role of deep architecture in modeling the complex non-linear relationship of high-dimensional spaces.

The incorporation of stability and robustness analysis is another new development, especially in biomedical and clinical applications where reproducibility is paramount. Stability selection, ensemble learning, and repeated cross-validation are some of the techniques that are being applied to reduce variance and enhance reliability.

Also, explainable AI (XAI) and interpretability-conscious feature selection are becoming increasingly popular, fulfilling the increasing need for transparency in high-stakes areas of healthcare and finance. The balance between predictivity performance and explainability is highlighted in recent publications, in the form of hybrid frameworks, which incorporate explainable models in deep learning structures.

4. PERFORMANCE EVALUATION AND COMPARATIVE ANALYSIS

The study provides a critical analysis of feature selection methods of high-dimension predictive modeling, based on the empirical evidence of the 40 studies assessed. The performance metrics, computational effectiveness, interpretability, and domain-specific effectiveness are aspects of the analysis and provide trends, strong points, and weaknesses of the various feature selection paradigms through synthesizing findings over heterogeneous datasets and different approaches.

4.1. Predictive Outcomes

The included studies reported multiple predictive-performance measures, including classification accuracy, AUC, F1-score, MCC, recall, RMSE, MAE, C-index, error rate, sparsity, and convergence rate. These measures represent different statistical properties and cannot be treated as interchangeable. Accordingly, AUC was not pooled with accuracy, F1-score, MCC, RMSE, MAE, C-index, or other outcomes.

Because the included studies used heterogeneous datasets, predictive tasks, outcome distributions, performance metrics, comparators, and validation procedures, no performance metric was normalised or converted to a common scale. Accuracy values were retained in their originally reported form, whether expressed as percentages or proportions. AUC, F1-score, MCC, RMSE, MAE, C-index, recall, error rate, sparsity, convergence rate, and other outcomes were interpreted separately because they represent different statistical properties. Findings were assessed only relative to the comparators and experimental conditions reported within each original study. Accordingly, no pooled performance estimates, cross-study means, between-study standard deviations, category-level error bars, or inferential rankings were calculated.

Reported findings were interpreted relative to the comparators, datasets, classifiers, and validation procedures used within each original study. Salhi et al. [8] reported substantial variation across classifier-method combinations, including very high performance in some configurations and considerably lower accuracy in others. Such variation demonstrates why reducing a study to one category-level average could misrepresent its findings. Therefore, quantitative results were retained as study-specific evidence rather than pooled estimates of method performance.

4.2. Performance Metrics

The effectiveness of feature-selection methods was evaluated across the included studies using predictive performance, computational requirements, feature-set size, selection stability, interpretability, and reproducibility. However, these dimensions were not reported consistently. Consequently, the review provides a qualitative and study-specific synthesis rather than a pooled numerical comparison.

4.2.1. Predictive Accuracy

Predictive outcomes were retained in the form reported by each original study. A favourable result indicates that a method performed better than at least one comparator under the conditions examined in that study. It does not demonstrate superiority across other datasets, domains, classifiers, or validation procedures.

Hybrid, ensemble, embedded, evolutionary, and reinforcement-learning approaches frequently report favourable within-study findings. Nevertheless, the magnitude and direction of their performance varied according to dataset characteristics, sample size, dimensionality, class distribution, predictive learner, comparator strength, and tuning procedure. These findings suggest that adaptive and combined approaches may be beneficial in particular settings, but they do not establish consistent or universal superiority over simpler feature-selection methods.

Some individual studies reported high accuracy or AUC values. These numerical findings should be interpreted cautiously because they originated from specific experimental settings and were not directly comparable across the review corpus. High performance in a particular benchmark or clinical dataset may reflect class balance, test-set size, feature construction, model complexity, validation design, or overfitting. Several hybrid, ensemble, and reinforcement-learning studies reported favourable results within their respective experimental settings [8, 17, 33]. However, these findings were not combined numerically because the studies differed substantially in dataset characteristics, dimensionality, outcome prevalence, predictive learners, comparator methods, hyperparameter-tuning procedures, and validation designs. The reported performance values were therefore retained as study-specific evidence and were not normalised, averaged, or used to construct cross-study error bars. These results demonstrate the potential usefulness of adaptive feature-selection methods under experimental conditions but do not establish their general superiority across datasets or application domains.

It is reported by [8] for classification accuracy higher than 95% with a hybrid AI-based feature selection framework, which is 6-10% higher than baseline models. Similarly, [17] have shown that the feature selection process using deep reinforcement learning had a 6-9% improvement with absolute AUC values ranging from 0.84 to 0.92 relative to a baseline of 0.85. This overcomes the traditional feature selection methods used in medical diagnosis problems. Other studies that used ensemble learning reported high accuracies of over 98% using feature selection and ensemble classifiers as used by [33].

Simpler filter-based algorithms, e.g., mutual information or correlation-based ranking, in contrast, are computationally efficient but can commonly perform poorly in predicting when the features interact in a complex manner, and are useful for large datasets. [28] found that mutual information-based selection was able to perform competitively but worse than wrapper and hybrid selections with imbalanced clinical data.

4.2.2. Computational Efficiency and Runtime Considerations

Computational requirements are important when high-dimensional datasets contain thousands or millions of candidate variables. Filter methods generally require fewer model fits and are therefore commonly suitable for large-scale screening or resource-constrained applications. Nevertheless, runtime comparisons were inconsistently reported across the included studies, and no pooled runtime analysis was undertaken [6, 29].

Wrapper, evolutionary, swarm-based, reinforcement-learning, and hybrid methods typically require repeated model training or iterative subset evaluation. These procedures may provide flexible search capabilities but usually involve greater computational cost and a larger hyperparameter space. Deep-learning-based selectors may require additional training time, memory, and specialised hardware. The evidence therefore supports a qualitative trade-off: simpler methods commonly provide greater scalability, whereas more adaptive search procedures may model complex interactions at the cost of computational burden and tuning complexity.

4.2.3. Interpretability and Model Transparency

Interpretability is particularly important in high-stakes applications such as healthcare, finance, and public decision-making. This review distinguishes intrinsic interpretability from post-hoc explainability. Intrinsic interpretability concerns whether the model or selection process can be understood directly from its structure, as in sparse linear models, rule-based methods, or transparent statistical filters. Post-hoc explainability refers to auxiliary techniques, such as Shapley values, saliency measures, or attention visualisations, that approximate feature influence after model training [28, 30].

Filter methods and sparse embedded models can often provide relatively transparent feature rankings or coefficient-based interpretations. By contrast, deep-learning, reinforcement-learning, evolutionary, and complex hybrid selection mechanisms may remain opaque even when post-hoc explanation methods are applied. Reducing the number of input variables does not automatically make the internal selection process interpretable. Claims for complex hybrid or deep methods improve interpretability and should therefore be framed as improvements in explainability or dimensional simplicity unless the decision structure itself is transparent [40].

4.3. Comparative Interpretation of Feature-Selection Approaches

The studies included were not treated as statistically exchangeable. They differed in application domain, dataset size, dimensionality, predictor dependence, outcome prevalence, predictive learner, comparator strength, hyperparameter tuning, validation procedure, and reported metric. Consequently, no meta-analysis, cross-study average, pooled standard deviation, ANOVA, category-level significance test, or confirmatory ranking of feature-selection approaches were conducted.

The comparative synthesis instead focused on recurring methodological trade-offs. Filter methods generally offered scalability, transparency, and model independence but could overlook nonlinear interactions or conditional relevance. Wrapper methods directly evaluated feature subsets using a predictive learner and could capture model-specific interactions, although they involved greater computational cost and a higher risk of overfitting. Embedded approaches integrated feature selection with model estimation and often provided a practical balance between efficiency and predictive modelling, but their performance remained dependent on model assumptions and predictor structure.

Hybrid, ensemble, evolutionary, reinforcement-learning, and other adaptive approaches combined complementary search or evaluation mechanisms. Several studies reported favourable results for these methods within their respective experimental settings. However, additional flexibility also introduced greater tuning requirements, computational burden, opacity, instability, and potential leakage risk. These methods should therefore be interpreted as potentially advantageous under suitable conditions rather than universally superior.

Terms such as “significant”, “superior”, and “outperformed” are used in this review only when referring to a formal comparison conducted within an individual included study. They are not used to imply statistical superiority across the review corpus. The strengths, limitations, and appropriate interpretation of the main methodological approaches are summarised in Table 7.

4.3.1. Biomedical and Healthcare Applications

Hybrid, embedded, ensemble, and learning-based methods frequently reported favourable results in biomedical and healthcare studies. However, the evidence varied across clinical cohorts, imaging datasets, diagnostic tasks, sample sizes, outcome prevalence, and validation procedures. These findings support selecting methods according to dimensionality, sample size, interpretability requirements, stability, and external-validation availability rather than claiming that one methodological category performs best.

4.3.2. Bioinformatics and Genomics

Evolutionary, embedded, filter-based, ensemble, and multi-objective methods were commonly evaluated in genomic settings characterised by large numbers of predictors and relatively small samples. Several studies reported useful trade-offs between predictive performance and feature-set sparsity. Nevertheless, the scientific value of selected genes or biomarkers depends on stability across resamples, biological plausibility, independent validation, and adequate protection against overfitting.

4.3.3. Industrial and Engineering Applications

Wrapper, hybrid, reinforcement-learning, evolutionary, and ensemble methods were evaluated in industrial and engineering datasets containing noise, heterogeneous variables, or complex interactions. Some studies reported favourable within-study findings, although increased search flexibility was generally accompanied by greater computational requirements and tuning sensitivity. Leakage-controlled, repeated, and preferably external validation is therefore important when applying these methods.

4.3.4. General Machine-Learning Benchmarks

Benchmark studies frequently reported favourable results for hybrid, swarm, ensemble, and evolutionary methods. However, benchmark outcomes are sensitive to dataset selection, classifier choice, hyperparameter tuning, comparator strength, and validation design. These findings demonstrate potential effectiveness under specific experimental conditions but do not support a universal ranking of feature-selection methods. Because the domain-specific studies were heterogeneous, no numerical domain scores, normalised comparisons, error bars, or ANOVA tests were calculated. Domain-level evidence was synthesised narratively.

4.4. Discussion of Findings

The evidence does not identify one feature-selection category as universally superior. Instead, effectiveness depends on the alignment among the data regime, predictive model, selection objective, available computational resources, and validation design. Filter methods provide scalability and transparency but may fail to capture nonlinear or conditional relationships. Wrapper, evolutionary, swarm-based, and reinforcement-learning methods explore more complex subsets but introduce greater computational cost, tuning sensitivity, and overfitting risk. Embedded methods frequently offer a practical compromise but remain dependent on the assumptions and inductive biases of the selected predictive learner.

The most defensible cross-study conclusion concerns trade-offs rather than rankings. Increased search flexibility may improve predictive fit within a particular study while simultaneously increasing instability, runtime, opacity, and susceptibility to leakage. These concerns are especially important in small-sample, high-dimensional genomic and clinical datasets, where different samples can produce different selected subsets despite similar predictive scores.

The proposed reclassification indicates that data-regime adaptation is widely represented and stability awareness is moderately represented, whereas explicit causal awareness remains rare. Most included studies identified predictors associated with an outcome rather than variables shown to have structural, mechanistic, or intervention-relevant effects. Predictive feature importance should therefore not be interpreted as evidence of causality.

The quality appraisal further indicates that only a minority of studies combined strong validation, protected feature-selection pipelines, reproducible implementation, and balanced reporting. Highly favourable predictive results warrant particular caution when test samples are small, feature selection is conducted before data splitting, nested validation is absent, or numerous selector-classifier combinations are evaluated without adequate control for optimisation bias.

Overall, feature-selection performance is conditioned by three interacting considerations: the data regime, model capacity, and selection objective. The data regime influences estimator reliability and selection stability; model capacity affects the bias-variance trade-off, interaction modelling, and interpretability; and the selection objective determines whether evaluation should prioritise prediction, stability, sparsity, computational cost, or causal relevance. Feature-selection methods are most defensible when these three considerations are aligned.

5. PRACTICAL APPLICATIONS FOR FEATURE SELECTION

Feature selection has become a key focus of the current generation of data-driven applications which have the challenge of high dimensionality, redundancy, and noise as major factors that affect the model performance and interpretability. In fields like bioinformatics, finance, healthcare, and medical imaging, efficient feature selection can be used to facilitate more accurate prediction and can be used with lower computational cost and better model clarity. In this section, an overview of the practical applications of feature selection methods in major areas of application has been synthesized based on the empirical evidence provided by the studies reviewed.

5.1. Applications in Bioinformatics

Bioinformatics is one of the most highly visible areas of application of feature selection because of the high dimensionality of biological data, especially in genomics, transcriptomics, and proteomics. The datasets used in this field have thousands of features (genes or biomarkers) and relatively few samples, which are highly overfitted. The role of feature selection is therefore important in discovering biologically important variables, as well as minimizing noise and redundancy.

Studies provide evidence of the usefulness of sophisticated feature selection techniques in biological and biomedical studies. Here, [7] used feature selection framework based on machine learning (highMLR) to select prognostic genes in cancer survival prediction with better predictive capabilities and interpretability. Equally, [6] used correlation-based and ensemble feature ranking methods to the high-dimensional gene expression datasets and was able to obtain strong classification results even in small sample settings.

Genomic data analysis has also been performed through transformation and hybrid solutions. It was shown by [4, 24] that multi-objective evolutionary feature selection techniques are an effective balance between accuracy and sparsity in situations where the number of genomic features is thousands. They have found applications, especially in omics studies where it is important to determine small sets of biologically meaningful genes that can be used to perform downstream analysis and clinically make decisions.

In addition, feature selection algorithms that are based on deep learning, including the ones suggested by [18, 12], are based on autoencoders and representation learning to discover intricate nonlinear interactions in high dimensional biological data. However, as much as such approaches have high predictive power, they are not easy to interpret, which supports the significance of hybrid and explainable methods in biomedical studies.

5.2. Applications in Finance

In finance, feature selection is valuable for handling high-dimensional, noisy, and correlated data that are generated from transactional records, market data, and customer behavioral information. In the use of such applications as credit scoring, fraud detection, and risk assessment, the choice of informative features is important to boost predictive accuracy without affecting the model interpretability and regulatory compliance.

Some of the studies reviewed can show that feature selection is effective in a financial setting. As an example, [30] used LASSO-based feature selection to credit risk modelling, which is capable of identifying the main financial indicators with a high level of reliability, but minimized the complexity of the model. On the same note, hybrid feature selection procedures which incorporate filter and wrapper methods have been depicted to enhance prediction power in financial classification tasks with the concurrent computational efficiency.

Research findings like those by [8, 35] also show that hybrid and ensemble-based feature selection frameworks can be useful in dealing with heterogeneous financial data, such as transactional, demographic and behavioral features. These methods not only improve prediction but also help in regulatory transparency because they facilitate the ability to understand the choice of risk factors.

Besides, feature selection has been demonstrated to minimize overfitting and enhance generalization in continually changing market conditions in algorithmic trading and financial forecasting. The capability to predictive features is especially important in financial applications where the model decisions may carry strong economic implications.

5.3. Applications in Healthcare and Medical Imaging

Healthcare is one of the most influential application areas of feature selection, especially in medical diagnosis, prognosis and decision support systems. Clinical data in high dimension tend to harbor redundant and noisy information, including electronic health record, medical imaging, and genomics samples which may obstruct significant patterns.

Some of the studies that have been reviewed prove the usefulness of feature selection in enhancing diagnostic accuracy. As an illustration, [13] used machine learning-based feature selection to forecast the outcome of chronic pain, and they reported significant enhancement in model performance. Equally, [34] found that deep learning models compared to conventional machine learning methods in predicting COVID-19 mortality were more predictive and the judiciously chosen features enhanced predictive reliability.

In medical imaging case, feature selection is important in dimension reduction and interpretability. Such methods as convolutional feature extraction with selection mechanisms including attention or saliency-based selection have been demonstrated to enhance diagnostic accuracy and decrease computing costs. The effectiveness of the hybrid representation learning and selective feature pruning methods can be proven with references to studies that use deep feature selection techniques, similar to those provided by [18].

5.4. Emerging Applications and Future Directions

In addition to more conventional fields, feature selection is being used in more emerging applications in the Internet of Things (IoT), remote sensing and smart city analytics. These applications include high dimensional and heterogeneous data, which are often streaming data, and require scalable and adaptive feature selection methods.

Real-time feature selection can be used in IoT settings to process sensor data efficiently and reduce the use of energy and communication overheads. Experiments done using evolutionary and reinforcement learning based feature selection show promising outcomes in dynamic settings where the data distributions change with time. Equally important, feature selection is useful in remote sensing and analysis of satellite images to minimize spectral redundancy and improve the accuracy of land-cover classification.

6. CHALLENGES AND FUTURE DIRECTIONS

The procedure of feature selection is an extensive but developing part of high-dimensional data analysis. Although notable achievements have been achieved in terms of superior statistical, machine, and hybrid methods, various methodological and practical issues persist. This part overviews the major weaknesses that have been noted throughout the studies reviewed and the research directions that are emerging in the future of feature selection.

6.1. Standardised Benchmarking

Future research should use benchmark suites that vary systematically in sample size, dimensionality ratio, predictor correlation, noise, imbalance, and interaction structure. Reporting only the best-performing dataset or classifier combination should be avoided.

6.2. Leakage-Controlled Validation

Feature selection, preprocessing, imputation, resampling, and hyperparameter tuning must be conducted exclusively within training folds. Applying selection to the complete dataset before cross-validation introduces information leakage and overestimates generalisation.

6.3. Stability Reporting

Studies should report the frequency with which each feature is selected across repeated resamples, together with an overall subset-stability measure such as the Jaccard index, Kuncheva index, or related agreement statistic.

6.4. Causal Feature Selection

Only one included study explicitly focused on causal or treatment-effect objectives. Future work should investigate how causal discovery, domain knowledge, structural assumptions, and predictive selection can be combined without conflating association with relevance.

6.5. External and Temporal Validation

Many studies rely on public benchmarks or single datasets. Independent external, temporal, multicentre, and cross-domain validation should receive greater emphasis.

6.6. Reproducibility

Authors should release code, data-access instructions, random seeds, search-space definitions, stopping criteria, preprocessing steps, feature-selection timing, and complete hyperparameters.

6.7. Interpretability

Studies should distinguish intrinsic transparency from post-hoc explanation. A model does not become intrinsically interpretable merely because it operates on fewer variables or because SHAP values are calculated after trainings.

6.8. Research Gaps

Despite extensive research, there remain a few gaps in the existing literature on feature selection. The absence of standard benchmarking frameworks is one of the primary weaknesses. All research studies use various datasets, measures and experimental designs which makes it hard to directly compare the methods. Despite the fact that a few works have tried to cross-dataset validate such as those reported by [2, 8], there is no single benchmark similar to ImageNet that features selection researchers can use.

Another such area of weakness is the thin domain feature selection frameworks. Although generic approaches work fairly, domain-neutral adaptations can be much better. Medical and genomic data are one such example, medical needs approaches that are sensitive to biological importance and clinical meaning, whereas financial needs are more sensitive to withstand noise and time decay. Research conducted by [30, 35] reveal the necessity of tailored feature selection strategies that would be based on the domain knowledge in the selection process.

Also, the combination of causal inference in the process of selecting features is under-researched. The relevance of most of the current methods is based on correlation and not causality, and this reduces the potential use of the methods in making critical decisions. Research is also emerging indicating that causal discovery plus feature selection would enhance model robustness and interpretability especially in the healthcare and policy-focused fields.

6.9. Emerging Research Trends

Future research directions reflect recent advances and include several promising directions. A key trend is the combination of feature selection with the use of deep learning where neural networks are directly trained to be feature relevant either by direct attention or by regularization techniques. Research works like [12, 18] prove that deep learning models are able to find complex nonlinear correlations in high-dimensional data.

The other developing field is the transfer learning and meta-learning to select features where information earned in one task or domain are used to enhance feature selection in another. The method is especially helpful in the data-sparse fields, e.g. the diagnosis of rare diseases or industry-specific tasks. Early indications indicate that transfer-based feature selection can be very effective in terms of minimizing the time of training and still achieving accuracy.

Lastly, a new and important line of research is beginning to emerge, that of multi-modal and multi-task feature selection. The present-day usage is frequently characterized by the heterogeneity of data sources, including a combination of clinical records, data of imaging, and genomic profiles. The methods which combine the selection of features between two or more different modalities or tasks, like the ones suggested by [4, 18], can provide promising directions in the creation of more holistic and robust prediction models.

This systematic review makes five contributions:

1. It establishes an explicit boundary between feature selection, which retains original variables and feature transformation, which constructs latent representations.
2. It integrates conventional algorithmic categories with an author-proposed analytical framework based on stability awareness, causal awareness, and adaptation to the data regime.
3. It reclassifies the 40 included studies using this framework to determine which methodological objectives are well represented and which remain under-investigated.
4. It applies structured quality appraisal, direction-of-effect vote counting, and thematic synthesis rather than interpreting heterogeneous accuracy values as directly comparable effect estimates.
5. It develops practitioner-oriented guidance linking common data conditions and decision requirements to appropriate starting methods, validation procedures, and implementation cautions. These contributions shift the review from a catalogue of algorithms toward a critical framework for method selection, evidence interpretation, and future research design.