Wrappers In Machine Learning
A wrapper is a strategy or meta-method that uses a machine learning model as a sub-component to solve a bigger task.
It wraps around the model and perform operations such as feature selection, model tuning, model ensembling and evaluation under constraints.
Wrappers for Feature Selection
See MML - Feature Selection - Wrapper Methods.
They evaluate subsets of features by training and testing a model on each subset. Examples:
- Forward selection: start with no features, add the best one at each step
- Backward elimination: start with all features, remove the worst feature
- Recursive Feature Elimination (RFE) with SVM or Random Forest
Hyperparameter Optimization Wrappers
These wrappers repeatedly train a model with different hyperparameters to find which one has the best performance.
Examples: Grid Search, Random Search, Bayesian Optimization (e.g., Tree Parzen Estimators).
Ensemble Wrappers
Some ensemble methods are wrappers because they repeatedly train a base model on different configurations of the data.
Examples:
- Bagging: Random Forest wraps around decision trees
- Boosting: AdaBoost wraps around weak learners
- Stacking: a meta-model wraps around multiple base models
Cross-validation
Cross-validation is itself a wrapper:
- It trains a model on multiple splits of the data
- It aggregates performance
This improves reliability and prevents overfitting.