The risk of overfitting is serious and can lead to significant losses. It has been discussed in previous posts. In this edition, we revisit the topic, given its continued relevance to quantitative strategy development.
Formal Study of Overfitting in Trading System Design
A serious problem when designing a trading system is the overfitting phenomenon, wherein the system is excessively tuned to historical data. Overfitting occurs when a trading strategy performs exceptionally well on past data but fails to generalize to new, unseen data. This can lead to false positives and inflated expectations, as the system may appear profitable due to chance rather than true predictive power.
Reference [1] formally studied this issue, using analytical approximations for the in-sample and out-of-sample Sharpe ratios of portfolios.
Findings
-The paper analyzes how the in-sample performance of trading strategies based on linear predictive models deteriorates out-of-sample due to overfitting.
-It develops closed-form approximations for both in-sample and out-of-sample Sharpe ratios by modeling the means and variances of strategy PnLs.
-The results show that strategies using a large number of assets and weak signals experience a significant decline in out-of-sample performance.
-In contrast, strategies relying on fewer but stronger signals tend to exhibit more stable and replicable results.
-Increasing the size of the training dataset improves the out-of-sample replication ratio and reduces overfitting risk.
-Signals with low true Sharpe ratios are particularly prone to overfitting, leading to inflated in-sample performance that does not persist.
-Simulation and empirical studies, including applications to commodity futures, confirm the magnitude and robustness of these effects.
-The findings also show that incorporating more realistic signal dynamics does not materially alter the main conclusions.
-The replication ratio is largely determined by the true out-of-sample Sharpe ratio rather than specific model assumptions.
-Overall, the study suggests that controlling model complexity and maximizing data usage are key to mitigating overfitting in predictive trading strategies.
In summary, the paper formally demonstrated that to minimize the risk of overfitting, one should,
- Keep models as simple as possible,
- Use the longest sensible backtest period available,
- Develop systems with high Sharpe ratios, and
- Rely on fewer signals.
From our experience, we have reservations about points #3 and #4, while agreeing with points #1 and #2. What do you think?
Reference
[1] Antoine Jacquier, Johannes Muhle-Karbe, Joseph Mulligan, In-Sample and Out-of-Sample Sharpe Ratios for Linear Predictive Models, 2025, arXiv:2501.03938
Avoiding Overfitting: Searching for Parameter Plateau
To mitigate the risk of overfitting, system developers often employ techniques such as cross-validation and out-of-sample testing to ensure that their strategies remain robust across various market conditions and time periods.
Another technique to prevent overfitting involves selecting a parameter region, often referred to as a “plateau,” where the trading system maintains stable performance. Reference [2] introduced a method for quantifying this plateau and utilized particle-swarm optimization to search for it.
Findings
-The study highlights that quantitative trading performance depends heavily on parameter selection and is vulnerable to overfitting.
-It introduces the concept of a parameter plateau to identify stable and robust parameter regions rather than single optimal points.
-A plateau score algorithm is developed to replace the conventional approach of selecting the best in-sample parameters.
-The results show that parameters with high plateau scores exhibit more stable and consistent out-of-sample performance.
-The approach helps avoid “parameter islands” that perform well in-sample but fail out-of-sample.
-To improve search efficiency, the study applies particle swarm optimization instead of brute-force methods.
-Particle swarm optimization enables faster exploration of high-dimensional parameter spaces.
-Experiments demonstrate that the combined plateau and optimization approach improves both robustness and profitability.
-The method remains effective as strategy complexity increases from low- to high-dimensional parameter settings.
-The study also proposes suitable hyperparameter ranges for particle swarm optimization in this framework.
In short, the extent of plateau stability is quantified, and an efficient optimization algorithm is utilized to search for it. The out-of-sample test results show promise.
Reference
[2] Jimmy Ming-Tai Wu, Wen-Yu Lin, Ko-Wei Huang, Mu-En Wu, On the design of searching algorithm for parameter plateau in quantitative trading strategies using particle swarm optimization, Knowledge-Based Systems, Volume 293, 7 June 2024, 111630
Closing Thoughts
Taken together, these studies highlight that both model design and parameter selection are key sources of fragility in quantitative strategies. Overfitting arises not only from using too many weak signals but also from selecting unstable parameter configurations that fail to generalize out-of-sample. Approaches such as reducing model complexity, increasing data, and focusing on stable parameter regions through the concept of parameter plateaus offer practical ways to improve robustness. Overall, the evidence suggests that consistent performance depends less on optimizing in-sample results and more on ensuring stability across regimes and datasets.