Model Analysis, Diagnosis and Improvement

LightGBM model trained on quarterly financial reports to predict whether a stock will outperform the S&P 500 over the following year. Binary target: positive when stock_change_365 minus sp500_change_365 is greater than zero. At each execution date, the model scores all tickers and selects a top_n = 10 portfolio, evaluated against an equal-weight S&P 500 baseline.

Initial models showed roughly 11% outperformance over the benchmark, immediately flagged as suspicious. Diagnosis used learning curves plotting binary logloss across boosted trees for train and validation: the median loss decreased then increased, confirming overfitting typical of low-data financial problems.

• arrow_forwardSimplify first, then add complexity incrementally
• arrow_forwardLow learning rates to control variance
• arrow_forwardNo manual parameter search, it risks overfitting the validation set by hand

Feature importance via permutation importance and SHAP revealed close_0 (stock price at prediction time) as suspiciously dominant. Historical prices are adjusted retroactively for reverse splits, so a high close_0 signals a future reverse split, not fundamental strength. The model had learned accounting mechanics, not business signal.

• arrow_forwardFix: remove all technical and price features, retrain on fundamentals only
• arrow_forwardPost-fix: most important features shifted to business ratios and financial statement data

• arrow_forwardOutperforms the benchmark in 46% of periods
• arrow_forwardDelivers roughly 5% higher mean return than the S&P 500
• arrow_forwardMedian excess return is negative; consistent outperformance requires a long investment horizon to let the high-return outlier periods dominate