The Overfitting Trap
In computer vision, adding more data and features is almost always better. In quantitative trading, adding more features usually leads to catastrophic overfitting.
We started with a dataset of 500 alpha factors (momentum indicators, mean-reversion signals, and alternative data embeddings). Our XGBoost model looked amazing in cross-validation, sporting a Sharpe ratio of 3.5. When we deployed it to live paper trading, it bled money instantly. The model had memorized the noise.
Purging and SHAP
We implemented a strict feature selection pipeline. First, we applied Temporal Purging to our cross-validation splits to ensure absolutely zero lookahead bias.
Then, we used SHAP (SHapley Additive exPlanations) values to determine true feature importance.
import shap
import xgboost as xgb
# Train a model on a subset of data
model = xgb.train(params, dtrain)
# Calculate SHAP values to find true feature importance
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_train)
# Plot summary to identify which features actually move the prediction
shap.summary_plot(shap_values, X_train)We aggressively dropped highly correlated features and removed any feature that didn't have a consistently high SHAP value across all market regimes. We reduced the feature set from 500 down to just 15 orthogonal signals. The live performance immediately stabilized. In finance, less is more.