Category: statistical arbitrage

Markets have been behaving unusually lately. In May, equity indices rose while volatility and skew also increased, a relatively rare occurrence historically. Since last week, the same phenomenon has emerged again, with the spot/volatility correlation turning positive.

Is this still a rare occurrence? We don’t know. But one thing is clear: regime detection is becoming increasingly important in today’s markets. In this post, we explore a couple of approaches for detecting market regimes.

A Regime Classification Framework for Mean-Reverting and Trending Markets

Regime classification is important in asset and risk management. Traditional approaches classify regimes based on direction, bullish or bearish, and volatility, high or low.

Reference [1] departs from this framework and instead classifies markets as mean-reverting or trending. Specifically, it uses return thresholds of 0.5%, 0.75%, and 1% to define regimes and examines SPY, QQQ, DIA, and IWM over the period 2000 to 2024.

Findings

-The study evaluates Random Forest and Neural Network classifiers using macroeconomic announcement indicators and technical features, including VIX, RSI, and ATR.

-It uses 25 years of daily data from 2000–2024 for IWM, SPY, QQQ, and DIA.

-The study frames next-day ETF behavior as a binary classification problem between “oscillating” and “trending” days.

-Oscillating days are defined using intraday movement thresholds of 0.5%, 0.75%, and 1%, with movements exceeding these thresholds classified as trending.

-At the 0.5% threshold, Neural Networks outperform a naive classifier by 13.4% for IWM, 15.4% for SPY, 4.7% for QQQ, and 3.2% for DIA.

-SPY produces the strongest results, with AUC values reaching 0.67–0.74 at the 0.75% and 1% thresholds.

-IWM shows improvements of 5.7%–13.4% across thresholds, with evidence of predictive power at the 0.5% and 0.75% thresholds.

-QQQ shows improvements of 4.7%–6.1%, but its predictive performance is weaker at lower thresholds.

-The results show that predictive performance varies materially across ETFs and oscillation thresholds, with some configurations providing limited discriminatory power.

In summary, the results show that the best case achieves a 15.4% improvement in prediction over a naive strategy for SPY using a neural network with a 0.5% threshold; although in many cases the improvement is more modest, in the range of 1 to 5%, and varies significantly across ETFs.

While the study has several limitations, it points to a more relevant research direction: predicting the magnitude-based regime appears slightly easier than predicting direction, and machine learning is effective as a risk or regime filter rather than as a direct alpha-generating signal.

Reference

[1] Azizi, S. (2026), Leveraging Machine Learning for Financial Forecasting: Distinguishing Market Trends from Oscillations in ETFs, Journal of Risk and Financial Management, 19(4), 262.

Entropy-Based Regime Detection of Tail Risks

Reference [2] proposes an alternative regime classification by distinguishing between “normal” and heavy-tailed regimes. Specifically, the study develops a nonparametric method to detect financial market regimes using differential entropy rather than volatility alone. The underlying idea is that while volatility measures dispersion, entropy captures the full distributional uncertainty, including tail behavior, which becomes particularly important during crisis periods.

The authors estimate entropy using a kernel density estimator with a heavy-tailed kernel in rolling windows and compare entropy with variance. When markets behave approximately Gaussian, i.e., normally, entropy and variance move together; during turbulent periods, the relationship breaks down, revealing heavy-tailed regimes that volatility alone cannot identify.

Findings

-The study develops a differential entropy approach to identify financial market regimes through changes in distributional complexity rather than variance alone.

-The method uses a data-adaptive heavy-tailed kernel and combines entropy with tail-index analysis within a moving-window framework.

-Monte Carlo experiments show that the approach is robust and sensitive to changes in tail behavior.

-Applied to the Ibovespa, S&P 500, Nikkei, and SSE Composite from 1998 to 2025, the method identifies heavy-tailed regimes associated with major periods of market turbulence.

-These periods include the Dot-com Bubble, Global Financial Crisis, COVID-19 shock, and the 2025 tariff-related crisis.

-Gaussian regimes correspond to periods of relative stability and market efficiency.

-The results show that variance and entropy do not necessarily move together during crises.

-While volatility measures dispersion, entropy captures broader uncertainty and tail risk, providing a complementary measure of systemic instability.

In short, the paper developed a regime detection method based on entropy, which provides an alternative regime indicator that captures tail risk and structural shifts that standard volatility measures may miss.

This represents an important contribution to the literature, particularly in the context of managing tail risks and risk management more broadly.

Reference

[2] Raul Matsushita, Iuri Nobre, Sergio Da Silva, Beyond volatility: Using differential entropy to detect financial market regimes, Chaos, Solitons and Fractals 202 (2026) 117553

Closing Thoughts

Together, these studies highlight two different approaches to market regime detection. The first uses machine learning to classify next-day ETF behavior as oscillating or trending, while the second uses differential entropy to identify shifts in market uncertainty and tail behavior. Both demonstrate that market regimes can be characterized using information beyond conventional volatility measures, although their effectiveness varies across markets, thresholds, and conditions.

Advances in computing power, declining hardware costs, and the rapid rise of machine learning and algorithmic trading have fundamentally transformed modern financial markets. While these technologies have improved market efficiency and execution, they have also introduced new challenges and risks.

In this post, we examine research on the impact of algorithmic trading, from its influence on corporate behavior and stock price crash risk to the role of high-frequency trading in liquidity, volatility, and overall market quality.

How Algorithmic Trading Impacts the Markets

Algorithmic trading is a method of executing trades using algorithms, or sets of predetermined rules, to make trading decisions. These algorithms are designed to take into account a variety of market conditions, such as price, volume, and timing. Algorithmic trading is often used by large institutional investors, such as hedge funds and investment banks, to execute trades quickly and efficiently. Algorithmic trading is also becoming increasingly popular with individual investors who have access to sophisticated trading software.

Algorithmic trading has a number of advantages over traditional methods of trading. First, algorithms can take into account a wider range of market data and make better-informed decisions. Second, algorithms can execute trades faster than humans, which can be especially important in fast-moving markets. Third, algorithmic trading can help to reduce costs by eliminating the need for human traders.

Algorithmic trading has grown enormously in the last two decades to become the dominant type of trading in the capital markets. Reference [1] studies the impact that algorithmic trading has on the markets.

Findings

-The study examines whether algorithmic trading (AT) increases firm-specific stock price crash risk.

-The authors argue that the short-term focus of algorithmic traders encourages managers to prioritize short-term earnings and delay the disclosure of bad news.

-The empirical results show that higher levels of algorithmic trading are associated with greater future stock price crash risk.

-The study finds that firms with more algorithmic trading are more likely to exhibit opportunistic financial reporting and disclosure practices.

-The relationship between algorithmic trading and crash risk is stronger when managers have greater incentives or the ability to withhold bad news.

-The findings suggest that algorithmic trading may reduce monitoring by fundamental investors, allowing bad news to accumulate over time.

-The results are supported by both instrumental-variable analysis and evidence from the SEC’s 2016 Tick Size Pilot Program.

In short, the authors conclude that increased algorithmic trading can contribute to higher firm-specific crash risk, with potentially adverse consequences for shareholders.

Reference

[1] Ahmed, Anwer S. and Li, Yiwen and McMartin, Andrew Stephen and Xu, Nina, The Rise of Machines: Algorithmic Trading and Stock Price Crash Risk, SSRN 4203738

The Role of HFT in Modern Financial Markets

High-frequency trading (HFT) is a type of algorithmic trading that uses computer programs to place orders at very fast speeds. High-frequency traders use sophisticated algorithms to analyze market data and make trades based on their predictions. These traders typically trade in large volumes of shares and use very short-term strategies.

While the previous article examined the broader impact of algorithmic trading on market behavior and stock price crash risk, Reference [2] focuses specifically on high-frequency trading. Rather than analyzing managerial incentives, it investigates how HFT affects market quality, providing direct evidence on its role in liquidity provision, volatility, and overall market efficiency.

Findings

-The study investigates the impact of high-frequency trading (HFT) by examining a major exchange infrastructure failure that temporarily prevented low-latency trading.

-The outage provides a natural experiment for assessing the role of HFT in modern financial markets.

-The authors find that the disruption has only a modest effect on trading volume and the number of trades.

-However, liquidity deteriorates significantly when high-frequency traders lose low-latency access.

-Market volatility also increases during the outage, although the effect is less pronounced than the decline in liquidity.

-The results suggest that investments in HFT infrastructure generate positive spillover benefits for all market participants by improving overall market quality.

-The findings support earlier research showing that HFT enhances market liquidity and, to a lesser extent, reduces volatility.

The authors conclude that markets remain functional without HFT, but trading becomes more expensive, and market quality deteriorates when high-frequency traders cannot operate at low latency.

Reference

[2] Benjamin Clapham, Martin Haferkorn and Kai Zimmermann, The Impact of High-Frequency Trading on Modern Securities Markets, Bus Inf Syst Eng, 2022

Closing Thoughts

Taken together, these two papers illustrate that algorithmic trading is neither inherently beneficial nor harmful; its impact depends on the aspect of the market being examined. While algorithmic trading may encourage short-term corporate behavior and increase stock price crash risk, high-frequency trading appears to enhance market quality by improving liquidity and reducing transaction costs. As algorithmic trading continues to evolve, understanding its diverse effects on market efficiency, stability, and price formation remains an important area of research.

Today, AI and machine learning techniques are evolving at a rapid pace, making the development of trading systems increasingly accessible. Generating signals, building models, and testing ideas is easier than ever. As a result, the challenge is no longer simply developing a trading strategy, but determining whether it is genuinely robust or merely the product of overfitting and data mining.

In this post, we discuss several frameworks for trading system validation and examine how researchers assess the reliability of systematic strategies before deploying them in live markets.

What Are the Correct Methods for Evaluating a Trading Strategy?

With the rapid advancement in computing power, quantitative researchers can now develop trading strategies quickly, employing multiple variables and methodologies. These approaches extend beyond traditional time-series and statistical models to include machine learning and AI-based techniques.

However, such models often deliver impressive in-sample results but fail in live trading, largely due to overfitting. While researchers still seek to exploit increased computing power, the key challenge remains how to address this overfitting problem.

Reference [1] addresses this problem by introducing a framework for evaluating trading strategies in the presence of multiple testing.

Findings

-The paper argues that many trading strategies appear profitable simply because researchers test a large number of ideas and select the best-performing results.

-Traditional statistical methods often ignore multiple testing, which can significantly inflate Sharpe ratios, t-statistics, and the perceived profitability of trading strategies.

-The paper discusses several multiple-testing frameworks, including Bonferroni, Holm, and Benjamini-Hochberg-Yekutieli (BHY), to reduce the likelihood of false discoveries.

-The authors show that a seemingly attractive strategy can emerge purely by chance when hundreds of strategies are tested simultaneously.

-To address this problem, they propose “haircutting” Sharpe ratios to account for data mining and multiple testing.

-In an example involving 200 randomly generated strategies, a strategy with a Sharpe ratio of 0.92 becomes statistically insignificant after multiple-testing adjustments.

-Applying the methodology to a database of 484 equity strategies results in substantial reductions in reported Sharpe ratios, suggesting that many apparent alphas are overstated.

-The paper also discusses the trade-off between false discoveries and missed discoveries, concluding that reducing false positives is more important than retaining marginal signals.

-The paper concludes that many published factors, anomalies, and trading strategies are likely false discoveries and that the traditional two-sigma threshold is no longer sufficient for strategy evaluation.

This is a foundational paper that brought the issue of strategy validation to the forefront of quantitative finance. It highlighted the dangers of data mining and multiple testing, and helped raise awareness that many seemingly profitable trading strategies may simply be statistical artifacts rather than genuine sources of alpha.

Reference

[1] Harvey, Campbell R. and Liu, Yan, Evaluating Trading Strategies, SSRN 2474755

Toward a Validation Framework for Data-Driven Trading Strategies

Reference [2] proposes what the authors describe as a rigorous walk-forward validation framework. In this approach, trading systems are developed using machine learning techniques and then tested 34 times over a 10-year sample, with each test period independent and trained solely on past data.

Findings

-The paper’s primary contribution is a rigorous validation framework for quantitative trading research rather than a new trading strategy.

-The proposed framework is designed to prevent look-ahead bias, incorporate realistic transaction costs, maintain interpretability, and support a wide range of hypothesis-generation methods, including large language models.

-The framework is evaluated through 34 independent out-of-sample tests spanning a 10-year period.

-The tested strategies generate modest but realistic performance, with an annualized return of 0.55% and a Sharpe ratio of 0.33.

-Despite modest returns, the framework exhibits strong downside protection, with a maximum drawdown of only -2.76% compared with -23.8% for SPY.

-The aggregate returns are not statistically significant, and the authors present this result transparently rather than relying on p-hacking or selective reporting.

-The key empirical finding is that market microstructure signals derived from daily OHLCV data are highly regime-dependent.

-These signals perform well during high-volatility periods but perform poorly during stable market environments.

-The results suggest that daily-data trading signals are most effective when information flow and trading activity are elevated.

-The paper emphasizes the importance of robust validation procedures and honest performance reporting in quantitative finance research.

While the initiative is commendable and highlights the need for more research on system validation, several limitations remain. We observe the following,

• First, the reported performance is rather modest.
• Second, rather than employing traditional rolling or anchored walk-forward analysis, the authors perform repeated out-of-sample tests using independent, non-overlapping data periods. This is the main contribution of the paper.
• Third, a critical unaddressed issue is that although the full sample spans multiple market regimes, the choice of the number of intervals and the length of each data window is itself arbitrary and should be treated as random variables. As a result, the reported trading performance is also conditional on these design choices and may be materially affected by them, undermining the claimed rigor of the validation framework.

Reference

[2] Gagan Deep, Akash Deep, William Lamptey, Interpretable Hypothesis-Driven Trading: A Rigorous Walk-Forward Validation Framework for Market Microstructure Signals,  arXiv:2512.12924

Closing Thoughts

Taken together, these papers emphasize that rigorous validation is at least as important as model development. The first paper shows that many seemingly successful trading strategies may be false discoveries arising from multiple testing and data mining, while the second demonstrates that even carefully validated signals can be highly regime-dependent and deliver only modest performance out of sample.

The message is clear: robust validation frameworks, realistic assumptions, and transparent reporting are essential for distinguishing genuine alpha from statistical artifacts and for building trading systems that can survive changing market environments.

Regression is one of the oldest and widely used statistical techniques. It has found applications across the social sciences, engineering, natural sciences, and finance. Despite the rapid rise of machine learning and AI, regression remains a useful tool for modeling relationships, making forecasts, and extracting signals from data.

In this post, we revisit regression-based trading systems and examine whether simple linear and logistic regression models can still generate useful predictive signals in today’s increasingly complex financial markets.

Is Linear Regression Still a Good Prediction Method?

Forecasting stock prices is a challenge due to the non-stationary nature of price time series and the noisy data inherent in these price sequences. Linear regression was a frequently used prediction method, but recent advancements in computing technologies have given rise to more sophisticated approaches like Long Short-Term Memory (LSTM), Artificial Neural Networks (ANN), Recurrent Neural Networks (RNN), etc.

Does the linear regression method still have its place amongst these advanced techniques?

Reference [1] examines the effectiveness of the linear regression method by applying it to a set of US stocks, using it for predicting closing prices and 10-day moving averages.

Findings

-The study develops a stock prediction framework based on historical prices, economic indicators, and linear regression techniques.

-The authors construct two models: one for stock price forecasting and another for predicting the 10-day Exponential Moving Average (EMA_10).

-The methodology includes data cleaning, feature selection, model training using Ordinary Least Squares (OLS), and performance evaluation using RMSE and MAE metrics.

-Both models achieve low prediction errors and high explanatory power, as reflected by favorable RMSE, MAE, and R-squared statistics.

-The results suggest that the models provide accurate forecasts of stock prices and short-term trend indicators.

-The proposed trading strategy generates profitable results while also reducing portfolio risk.

-The study concludes that simple linear regression models can provide useful insights into future stock price movements and market trends.

In summary, linear regression is still an effective prediction method.  It remains a viable method due to its

• Simplicity and interpretability,
• Efficiency with smaller datasets,
• Ability to mitigate excessive overfitting.

Reference

[1] S. Sanapala, V. A. Reddy, S. Sinha Choudhury, V. V. Akshaya and V. Maheedhar Varma, Optimising Trading Strategies using Linear Regression on Stock Prices, 2023 International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE), Chennai, India, 2023, pp. 1-6.

Evaluating a Logistic Regression Trading Framework

Reference [2] employs logistic regression, which is particularly suited for modeling binary outcomes, to predict stock price movements based on historical returns.

The author uses cumulative returns over the past 20 days and the past 12 months as predictive variables, capturing short-term and long-term momentum effects. Logistic regression is then applied to classify whether a stock’s return in the upcoming month exceeds that month’s median return. The procedure is implemented on S&P 500 stocks from January 1985 to July 2024 using survivorship-bias-free data.

Findings

-The paper evaluates a Logistic Regression-Based Systematic Trading (LRST) strategy applied to S&P 500 stocks from 1983 to 2023.

-The strategy uses logistic regression to predict future stock price direction based on historical returns and frames the problem as a binary classification task.

-The model employs a rolling 10-year estimation window, allowing it to adapt to changing market conditions over time.

-Over the full sample, the strategy achieves an annualized return of 24.61%, outperforming the S&P 500 during several periods, particularly in the 1990s and early 2000s.

-Despite strong historical returns, the strategy exhibits substantial risk, with an annualized volatility of 26.11% and a Sharpe ratio of 0.77.

-Recent performance from 2021 to 2024 is notably weak, with the strategy failing to participate in much of the market’s gains.

-The results suggest that structural market changes, including the growth of algorithmic trading and shifting macroeconomic conditions, may have reduced the strategy’s effectiveness.

-The study highlights the importance of adapting systematic trading models to evolving market environments.

-The authors suggest that incorporating machine learning methods, sentiment indicators, and macroeconomic variables could improve robustness and future performance.

In short, the paper shows that the logistic regression-based strategy delivers an annualized return of 24.61%, outperforming the S&P 500, but its high volatility and Sharpe ratio of 0.77 indicate substantial risk and room for improvement in its risk-return profile. Its recent underperformance may reflect structural weaknesses amid the rise of algorithmic trading and shifting macroeconomic conditions, underscoring the need for adaptation.

This article is insightful as it demonstrates that,

• Even a basic regression framework can serve as a useful predictive tool within a trading system, although further refinement is necessary.
• There might be structural changes in market dynamics, driven by the increasing prevalence of algorithmic trading and artificial intelligence, implying that traders must adapt accordingly.

Reference

[2] Conrad O. Voigt, Logistic Regression-Based Systematic Trading: Performance on the S&P 500, 2026, github

Closing Thoughts

Taken together, these studies suggest that simple regression techniques, whether linear or logistic, remain useful tools for systematic trading even in the modern era. Despite the rapid growth of machine learning and AI, relatively straightforward models can still generate meaningful predictive signals and attractive historical performance.

However, the papers also highlight that refinement is necessary, as the effectiveness of these models depends on market conditions, structural changes, and the choice of predictive variables. Continuous adaptation and model improvement remain essential for maintaining performance over time.

The risk of overfitting is serious and can lead to significant losses. It has been discussed in previous posts. In this edition, we revisit the topic, given its continued relevance to quantitative strategy development.

Formal Study of Overfitting in Trading System Design

A serious problem when designing a trading system is the overfitting phenomenon, wherein the system is excessively tuned to historical data. Overfitting occurs when a trading strategy performs exceptionally well on past data but fails to generalize to new, unseen data. This can lead to false positives and inflated expectations, as the system may appear profitable due to chance rather than true predictive power.

Reference [1] formally studied this issue, using analytical approximations for the in-sample and out-of-sample Sharpe ratios of portfolios.

Findings

-The paper analyzes how the in-sample performance of trading strategies based on linear predictive models deteriorates out-of-sample due to overfitting.

-It develops closed-form approximations for both in-sample and out-of-sample Sharpe ratios by modeling the means and variances of strategy PnLs.

-The results show that strategies using a large number of assets and weak signals experience a significant decline in out-of-sample performance.

-In contrast, strategies relying on fewer but stronger signals tend to exhibit more stable and replicable results.

-Increasing the size of the training dataset improves the out-of-sample replication ratio and reduces overfitting risk.

-Signals with low true Sharpe ratios are particularly prone to overfitting, leading to inflated in-sample performance that does not persist.

-Simulation and empirical studies, including applications to commodity futures, confirm the magnitude and robustness of these effects.

-The findings also show that incorporating more realistic signal dynamics does not materially alter the main conclusions.

-The replication ratio is largely determined by the true out-of-sample Sharpe ratio rather than specific model assumptions.

-Overall, the study suggests that controlling model complexity and maximizing data usage are key to mitigating overfitting in predictive trading strategies.

In summary, the paper formally demonstrated that to minimize the risk of overfitting, one should,

1. Keep models as simple as possible,
2. Use the longest sensible backtest period available,
3. Develop systems with high Sharpe ratios, and
4. Rely on fewer signals.

From our experience, we have reservations about points #3 and #4, while agreeing with points #1 and #2. What do you think?

Reference

[1] Antoine Jacquier, Johannes Muhle-Karbe, Joseph Mulligan, In-Sample and Out-of-Sample Sharpe Ratios for Linear Predictive Models, 2025, arXiv:2501.03938

Avoiding Overfitting: Searching for Parameter Plateau

To mitigate the risk of overfitting, system developers often employ techniques such as cross-validation and out-of-sample testing to ensure that their strategies remain robust across various market conditions and time periods.

Another technique to prevent overfitting involves selecting a parameter region, often referred to as a “plateau,” where the trading system maintains stable performance. Reference [2] introduced a method for quantifying this plateau and utilized particle-swarm optimization to search for it.

Findings

-The study highlights that quantitative trading performance depends heavily on parameter selection and is vulnerable to overfitting.

-It introduces the concept of a parameter plateau to identify stable and robust parameter regions rather than single optimal points.

-A plateau score algorithm is developed to replace the conventional approach of selecting the best in-sample parameters.

-The results show that parameters with high plateau scores exhibit more stable and consistent out-of-sample performance.

-The approach helps avoid “parameter islands” that perform well in-sample but fail out-of-sample.

-To improve search efficiency, the study applies particle swarm optimization instead of brute-force methods.

-Particle swarm optimization enables faster exploration of high-dimensional parameter spaces.

-Experiments demonstrate that the combined plateau and optimization approach improves both robustness and profitability.

-The method remains effective as strategy complexity increases from low- to high-dimensional parameter settings.

-The study also proposes suitable hyperparameter ranges for particle swarm optimization in this framework.

In short, the extent of plateau stability is quantified, and an efficient optimization algorithm is utilized to search for it. The out-of-sample test results show promise.

Reference

[2] Jimmy Ming-Tai Wu, Wen-Yu Lin, Ko-Wei Huang, Mu-En Wu, On the design of searching algorithm for parameter plateau in quantitative trading strategies using particle swarm optimization, Knowledge-Based Systems, Volume 293, 7 June 2024, 111630

Closing Thoughts

Taken together, these studies highlight that both model design and parameter selection are key sources of fragility in quantitative strategies. Overfitting arises not only from using too many weak signals but also from selecting unstable parameter configurations that fail to generalize out-of-sample. Approaches such as reducing model complexity, increasing data, and focusing on stable parameter regions through the concept of parameter plateaus offer practical ways to improve robustness. Overall, the evidence suggests that consistent performance depends less on optimizing in-sample results and more on ensuring stability across regimes and datasets.

I just returned from a two-day conference in New York, FutureAlpha (formerly QuantStrats). This year, the theme focused largely on data, machine learning, and AI. While some speakers were very enthusiastic about the potential of AI to generate alpha, our panel was more conservative. The consensus among the panelists was to use ML and AI to enhance and improve risk management. Along this theme, in this post, I discuss the use of generative AI in trading.

Integrating Structured and Unstructured Data with LLMs and RAG

Traditional quantitative methods often rely on structured data, such as time series. With the emergence of Large Language Models (LLMs), it is now possible to process unstructured data. A new line of research focuses on integrating unstructured data analysis into traditional frameworks.

Along this line, Reference [1] proposed the use of LLMs together with retrieval-augmented generation (RAG) to process both structured and unstructured data concurrently. Specifically, the authors developed a system that first applies LLMs to detect regime shifts using time-series techniques, then employs RAG to integrate external knowledge into the model’s decision-making process. By retrieving relevant information from a vector database and combining it with the model’s capabilities, RAG improves both the interpretability and effectiveness of trading strategies.

Findings

-The paper studies methods for fine-tuning open-source Large Language Models to enhance quantitative trading strategies.

-It integrates numerical data, such as prices and technical indicators, with textual data, including news and sentiment.

-The approach uses Retrieval-Augmented Generation with a vector database to process and contextualize textual information.

-The study focuses on fully fine-tuning smaller models to achieve cost efficiency and scalability.

-It proposes a hybrid framework that combines LLM capabilities with traditional quantitative methods.

-The framework incorporates real-time data pipelines and adaptive model tuning.

-The results show improvements in predictive accuracy and risk-adjusted returns.

-The integration of multimodal data helps address challenges in combining structured and unstructured information.

-Fine-tuned smaller models improve regime detection and trading decision accuracy while maintaining efficiency.

-Additional techniques enhance model performance and robustness, supporting practical applications in quantitative finance.

In short, incorporating RAG into the framework enhances the model’s ability to understand complex macroeconomic environments and adapt trading strategies as conditions evolve. Experimental results show significant gains in predictive accuracy and risk-adjusted returns, demonstrating the practical value of these fine-tuning methods in finance.

Reference

[1] Li, C., Chan, C.H.R., Huang, S.H., Choi, P.M.S. (2025). Integrating LLM-Based Time Series and Regime Detection with RAG for Adaptive Trading Strategies and Portfolio Management. In: Choi, P.M.S., Huang, S.H. (eds) Finance and Large Language Models. Blockchain Technologies. Springer, Singapore.

Can AI Trade? Modeling Investors with Large Language Models

The previous paper focuses on improving trading performance by integrating LLMs with quantitative models and data, while another line of research explores how LLMs behave as autonomous agents within market environments.

Reference [2] utilized LLMs to construct trading agents in the financial markets. Specifically, the author used LLMs to emulate various types of investors: value investors, momentum traders, market makers, retail traders, etc.

Findings

-The paper develops a simulated stock market in which large language models act as heterogeneous trading agents.

-The framework includes realistic market features such as an order book, market and limit orders, partial fills, dividends, and equilibrium clearing.

-Agents operate with different strategies, information sets, and endowments, and communicate decisions using structured outputs while explaining reasoning in natural language.

-The results show that LLMs can consistently follow instructions and implement strategies such as value investing, momentum trading, and market making.

-LLM agents process market information and respond meaningfully to prices, dividends, and historical data.

-The simulated market exhibits realistic dynamics, including price discovery, bubbles, underreaction, and liquidity provision.

-The framework enables controlled analysis of agent behavior under different market conditions, similar to interpretability methods in machine learning.

-It provides a cost-effective way to test financial theories that lack closed-form solutions.

-The study highlights that LLM behavior is highly sensitive to prompts, which can lead to correlated actions across agents.

-This correlation may amplify volatility and introduce systemic risks, emphasizing the need for careful testing before real-world deployment.

In short, the article concluded that trading strategies generated by large language models are effective, but could introduce new systemic risks to financial markets because these agents would act in a correlated manner.

Reference

[2] Alejandro Lopez-Lira, Can Large Language Models Trade? Testing Financial Theories with LLM Agents in Market Simulations, arXiv:2504.10789

Closing Thoughts

In this issue, the discussion highlights two complementary directions in applying LLMs to finance. On one hand, integrating LLMs with quantitative models and multimodal data can improve predictive accuracy and risk-adjusted returns. On the other hand, treating LLMs as autonomous trading agents reveals how their behavior can shape market dynamics, including liquidity, price discovery, and potential instability. Taken together, the results suggest that while LLMs offer meaningful opportunities in trading and risk management, their impact depends critically on implementation, prompting, and control of system-wide behavior.

Applications of machine learning in finance continue to evolve rapidly. In previous posts, we discussed both the uses and the challenges of applying machine learning in financial markets. In this installment, we continue that discussion by highlighting new research on machine learning approaches for pricing complex derivatives and identifying signals that may precede major market downturns.

Speeding Up Derivatives Pricing Using Machine Learning

A financial derivative is a contract whose value depends on the price of an underlying asset such as a stock, bond, commodity, or index. Accurate valuation of financial derivatives and their associated sensitivity factors is important for both investment and hedging purposes. However, many complex derivatives exhibit path-dependency and early-exercise features, which means that closed-form solutions rarely exist, and numerical methods must be used.

The issue with numerical methods is that they are often slow. As a result, efforts are being made to improve the efficiency of numerical techniques for valuing financial derivatives. Reference [1] proposed a fast valuation method based on machine learning. It developed a hybrid two-stage valuation framework that applies a machine learning algorithm to highly accurate derivative valuations incorporating full volatility surfaces. The volatility surface is parameterized, and a Gaussian Process Regressor (GPR) is trained to learn the nonlinear mapping from the complete set of pricing inputs directly to the valuation outputs. Once trained, the GPR delivers near-instantaneous valuation results.

Findings

-The study develops a machine learning framework for pricing derivative products whose valuation depends on volatility surfaces.

-Volatility surfaces are parameterized using the five-parameter SVI model with a one-factor term structure adjustment to generate realistic synthetic market scenarios.

-High-accuracy valuations for variance swaps and American put options are computed using conventional numerical methods and used to create training and testing datasets.

-A Gaussian Process Regressor is trained to learn the nonlinear relationship between input risk factors, such as volatility surface parameters, strike, and interest rate, and valuation outputs including prices and Greeks.

-The trained model achieves high accuracy, with approximately 0.5% relative error for variance swap fair strikes and 1.7–3.5% relative error for American put prices and first-order Greeks.

-The model is less accurate for the Gamma Greek due to discontinuities in the strike dimension.

-After training, the machine learning model produces valuations almost instantly, achieving a speed improvement of three to four orders of magnitude compared with traditional numerical methods.

-The results demonstrate that machine learning can enable real-time risk analytics, dynamic hedging, and large-scale scenario analysis for derivatives.

-The framework is general and can be extended to other path-dependent derivatives with early exercise features.

In summary, the authors developed an efficient method to price complex financial derivatives using a machine learning technique. However, it is noted that GPR’s performance in valuing higher-order greeks is noticeably less accurate. Additionally, the study was conducted using synthetic data, so it would be useful to see the method applied to real-world scenarios.

Reference

[1] Lijie Ding, Egang Lu, Kin Cheung,  Fast Derivative Valuation from Volatility Surfaces using Machine Learning, arXiv:2505.22957

Forecasting Market Crashes with Machine Learning Techniques

Reference [2] examines how machine learning can be used to predict market crashes within the Adaptive Market Hypothesis framework.

The study considers three categories of factors:

1. Internal factors, such as technical indicators designed to capture endogenous market dynamics, including momentum, trend strength, and money flow arising from investor behavior and adaptive learning;
2. External factors, including macroeconomic and commodity variables that proxy for systematic, exogenous risks affecting fundamental valuations; and
3. Volatility features that quantify market fear and uncertainty.

The author evaluates the performance of three predictive models—logistic regression, random forest, and a long short-term memory (LSTM) network.

Findings

-While the Efficient Market Hypothesis suggests crashes cannot be predicted, the Adaptive Market Hypothesis allows for temporary periods of predictability as market conditions evolve.

-The analysis compares a traditional econometric model, Logistic Regression, with machine learning approaches, including Random Forest and LSTM.

-The models use a feature set combining technical, macroeconomic, and volatility-based indicators.

-Model performance is evaluated using metrics designed for imbalanced classification problems, where crash events are rare but economically significant.

-Empirical results show that the LSTM provides the best balance between precision and recall, although Logistic Regression remains competitive.

-The findings highlight that simpler models can still perform effectively, supporting the value of model parsimony in turbulent market environments.

-The results also support the Adaptive Market Hypothesis by showing that market predictability evolves over time and depends on changing conditions.

-Logistic Regression performs well as an early-warning system due to its high recall, although it generates many false positives.

-The LSTM model improves precision while maintaining strong recall, suggesting that capturing temporal patterns in financial data enhances predictive performance.

-Overall, the study concludes that market crashes are not entirely random, but their prediction depends on the appropriate balance between model complexity and practical application.

In short, the study concludes that market crashes are difficult to forecast but not entirely random, and different models capture different aspects of predictability. Logistic regression functions well as a high-recall early warning tool, while LSTM models provide more balanced signals.

Reference

[2] Michele Della Mura, Predicting Stock Market Crashes, A Comparative Analysis of Econometric and Machine Learning Models, Politecnico di Torino, 2025

Closing Thoughts

Taken together, these studies illustrate the expanding role of machine learning in modern quantitative finance. One line of research demonstrates how machine learning models can dramatically accelerate the pricing of complex derivatives while maintaining high accuracy, enabling real-time risk management and large-scale scenario analysis. Another line of work explores the ability of both traditional econometric methods and advanced machine learning models to identify signals that may precede market crashes. Collectively, these findings show that machine learning is reshaping financial modeling, though simpler approaches can still play a meaningful role.

Herding behavior has been extensively studied and is well understood in equity markets, but far less so in other asset classes such as commodities and cryptocurrencies. In this post, we explore key aspects of herding behavior in crypto and commodity markets.

Investor Behavior in Crypto During Geopolitical Shocks

Herd behavior refers to the tendency of investors to follow the actions of a larger group, often ignoring their own analysis or information. This collective movement can lead to asset bubbles during bull markets and sharp sell-offs during downturns. Understanding herd behavior is essential for identifying potential mispricings and avoiding emotionally driven decisions.

Herding behavior has been well studied in the equity markets, but less so in the cryptocurrency market. One might expect stronger herding in crypto due to the prevalence of young, inexperienced traders and the fact that crypto markets are under-regulated, less transparent, and highly volatile. However, existing studies have produced inconclusive results.

Reference [1] extends the research on herding in the crypto space by examining behavior during major geopolitical events, such as the COVID-19 pandemic and the Russia–Ukraine war.

Findings

-The study finds strong evidence of market-wide herding behavior in cryptocurrency markets by analyzing the relationship between return dispersion and market returns.

-Geopolitical risk (GPR) significantly amplifies herding, with severe herding detected across nearly all model specifications.

-The GPR Threat index has a stronger impact on herding than the GPR Act index, indicating that perceived geopolitical threats matter more than realized events.

-Herding behavior is asymmetric, occurring more intensely during bearish market conditions than bullish ones.

-Imitative trading is particularly pronounced during periods of market stress, confirming the presence of asymmetric herding.

-The strongest herding effects are observed during extreme geopolitical and global events, notably the COVID-19 pandemic and the Russia–Ukraine war.

-The findings suggest that herding in cryptocurrency markets is largely intentional, reflecting low information symmetry, weak disclosure, and limited information quality.

-Actual geopolitical events (GPR Act) tend to lose explanatory power because market participants rapidly process and price in the information once it is released.

-When realized geopolitical shocks exceed investor expectations, uncertainty rises sharply and herding intensifies.

In short, the authors found that herding intensifies during such events and is clearly present throughout these periods.

Reference

[1] Phasin Wanidwaranan, Jutamas Wongkantarakorn, Chaiyuth Padungsaksawasdi, Geopolitical risk, herd behavior, and cryptocurrency market, The North American Journal of Economics and Finance Volume 80, September 2025, 102487

Does Herding Behavior Exist in the Commodity Markets?

Herding behavior has been shown to exist in equity markets. Reference [2] examines the herding behavior in the commodity markets.

Findings

-The study investigates herding behavior in commodity ETFs using high-frequency microstructure data and a GARCH model that incorporates cross-sectional and market volatility at 15-, 30-, 45-, and 60-minute intervals.

-During periods of market instability and the COVID-19 pandemic, agricultural and metal-based ETFs generally exhibit weaker herding behavior, while energy-based ETFs tend to herd more.

-Under normal market conditions, herding typically emerges at frequencies longer than 30 minutes.

-Broad basket commodity ETFs and energy-based ETFs display herding behavior across multiple frequencies rather than at a single time scale.

-A notable exception is agricultural ETFs during the COVID-19 pandemic, where herding is observed across all frequencies, representing a key and unusual finding.

-Correlation analysis shows that commodity ETFs become less correlated with each other as time progresses.

-Lower observation frequencies are associated with weaker correlations across ETFs, except in the energy sector.

-The results suggest that herding behavior varies significantly by commodity type, market regime, and observation frequency.

The findings provide insights for investors, economists, and policymakers, particularly for designing diversification, hedging strategies and mitigating risks such as asset price bubbles and financial instability.

Reference

[2] Ah Mand, Abdollah and Sifat, Imtiaz and Ang, Wei Kee and Choo, Jian Jing, Herding Behavior in Commodity Markets. SSRN 4502804

Closing Thoughts

Taken together, these two studies show that herding behavior extends well beyond equity markets and plays a meaningful role in both cryptocurrencies and commodity ETFs, particularly under stress. In crypto markets, herding is strongly amplified by geopolitical risk, bearish conditions, and extreme events. In commodity ETFs, herding is more nuanced and highly dependent on asset class, market regime, and trading frequency, with energy and broad commodity baskets exhibiting persistent herding, while agricultural and metal ETFs remain relatively resilient except during extreme volatility.

Overall, the evidence suggests that herding is regime-dependent, frequency-specific, and asset-class-specific, with important implications for risk management, diversification, and the design of trading and hedging strategies during periods of market stress.

Pairs trading, or statistical arbitrage (stat arb), is a classic, well-established quantitative trading strategy, and it is still in use today. I discussed its profitability in a previous post, and in this installment, we continue that discussion.

Pairs Selection Methods

Reference [1] provides a thorough review of the pairs trading literature between 2016 and 2023.

Pair selection is a critical step in pairs trading, and the paper offers a comprehensive review of the various pair selection methods used in practice. They are:

1-Distance Methods

Use SSE/SAE of normalized price differences to identify co-moving assets. Simple, intuitive, and historically profitable across markets, even after costs.

2-Cointegration Methods

Exploit long-run equilibrium relationships. Strong empirical support across equities and bonds, with advances in regime switching and external-factor integration.

3-Stochastic Control Methods

Model pairs trading as a continuous-time optimization problem. Incorporate jumps, regime changes, and stochastic volatility, showing strong performance but facing practical frictions.

4-Time Series Methods

Use GARCH, OU, and fractional OU to model short-term dynamics and volatility clustering. Adaptive thresholds improve returns; hybrid models are an emerging area.

5-Other Methods

Copulas capture tail dependence; Hurst exponent methods capture long memory; entropic approaches address model uncertainty. These improve robustness under nonlinear dynamics.

Overall, the review helps practitioners adapt stat-arb techniques to new markets and regimes. While simple methods once worked well, today’s competitive environment often requires more sophisticated approaches, though success still depends on model design, data quality, and market regime.

Profitability of Pairs Trading

There is an ongoing debate in the literature—some argue that “pairs trading is dead,” while others maintain that it remains profitable. From this review paper [1], we learn the following.

1- Pairs trading remains profitable, but returns are weaker and more conditional

The survey explicitly notes that profitability persists, but is not uniform and depends on market conditions, costs, and implementation details:

Empirical evidence consistently shows that distance-based pairs trading can be profitable across different markets, asset classes, and time horizons.

However, this is immediately tempered elsewhere by declining performance stability:

Performance is not uniform over time: profitability tends to vary with market volatility, and Sharpe ratios decline in certain subperiods.

2. Transaction costs and competition materially erode profits

Modern profitability survives only after careful cost control, unlike the early 2000s results:

Even after accounting for realistic transaction costs, the strategy remains profitable in several markets.

3. Advanced methods outperform naïve approaches

The paper makes clear that simple Gatev-style [2] implementations are no longer sufficient:

The apparent simplicity of GGR’s strategy becomes less evident as more sophisticated models and techniques have been introduced.

And later:

Regime-switching structures … demonstrate superior performance, particularly under frequent or pronounced regime shifts.

In short, the paper does not argue that pairs trading has stopped working, but it makes clear that the simple, mechanical versions that worked in the 1990s and early 2000s no longer deliver robust returns. Profitability today is weaker, highly dependent on market regimes, and much more sensitive to transaction costs and execution. What survives is not the original Gatev–Goetzmann–Rouwenhorst method, but more adaptive, model-driven implementations that account for changing volatility, correlations, and liquidity.

Reference

[1] Sun, Y. (2025). A survey of statistical arbitrage pairs trading strategies with non-machine learning methods, 2016-2023. WNE Working Papers, 19/2025 (482). Faculty of Economic Sciences, University of Warsaw

[2] Gatev, E., Goetzmann, W., & Rouwenhorst, K. G. (2006). Journal of Financial Economics, 81(1), 105–141.

Closing Thoughts

The paper provides a thorough review of all existing pair selection methods, which are critical to pairs trading. It also concludes that current profitability is weaker, highly dependent on market regimes, and significantly more sensitive to transaction costs and execution.

Most research in portfolio management focuses on alpha generation; however, another critical component of portfolio construction is position sizing. In this post, we examine key considerations in position sizing, including the Kelly criterion and the martingale betting system.

Does Kelly Portfolio Outperform the Market?

A method for capital allocation and position sizing is to employ the Kelly criterion. The Kelly criterion aims to optimize the expected growth rate of capital, maximizing the anticipated value of the logarithm of wealth. This strategy is rooted in John Kelly’s paper, “A New Interpretation of Information Rate.” According to Kelly, in repeated bets, a bettor should act to maximize the expected growth rate of capital, thus maximizing expected wealth at the end.

Reference [1] applies Thorp’s approach, as outlined in “The Kelly Criterion in Blackjack, Sports Betting and the Stock Market,” [2]  to construct a portfolio in the Norwegian stock market. The formula computes the optimal investment fraction in a set of assets, considering the expected excess returns of the assets and the inverse of the variance-covariance matrix.

Findings

-The study evaluates the performance of a growth-optimal Kelly portfolio in the Norwegian stock market over the period February 2003 to December 2022.

-It assesses abnormal performance using the CAPM, Fama–French three-factor model, and Carhart four-factor model.

-The Kelly portfolio achieves a higher compound annual growth rate (14.1%) and higher ending wealth than the benchmark index, which grows at 12%.

-It also outperforms a Markowitz portfolio, which delivers lower growth and final wealth.

-The Kelly portfolio and the benchmark exhibit similar Sharpe ratios (0.58), while the Kelly portfolio attains a higher Sortino ratio (0.95).

-Factor regressions indicate an annualized alpha of 16.8% for the Kelly portfolio, statistically significant at the 1% level before transaction costs.

-However, the factor models display very low explanatory power, suggesting that the estimated alpha may be overstated.

-Once transaction costs are incorporated, the Kelly portfolio no longer outperforms the benchmark in terms of final wealth.

-After costs, the alpha remains only marginally significant at the 10% level, implying limited real-world risk-adjusted excess returns.

This paper presents several interesting findings,

-First, the correlation of the Kelly portfolio with the market is nearly zero.

-Second, the performance is sensitive to transaction costs. We believe that with lower transaction costs, the Kelly portfolio has the potential to outperform the market and display zero correlation with it.

-Third, the Kelly portfolio surpasses the Markowitz mean-variance portfolio in performance.

We also concur with the author that the utilization of options can further enhance the risk-adjusted return.

Reference

[1] Jon Endresen and Erik Grødem, The Kelly criterion, an empricial study of the growth optimal Kelly portfolio, backtested on the Oslo Stock Exchange, 2023, Norwegian School of Economics.

[2] Thorp, E. O., The Kelly Criterion in Blackjack Sports Betting and the Stock Market, in: Zenios, S.A. & Ziemba, W.T., Handbook of Asset and Liability Management, Volume 1, 387–428, 2006

Enhanced Martingale Betting System with Stop Policy

The martingale betting system is a popular gambling strategy that involves doubling one’s wager after each loss in the pursuit of recovering previous losses and securing a profit equal to the original bet. The underlying idea is that, statistically, a win will eventually occur, allowing the player to recoup losses and gain a net profit equal to the initial stake. While simple in concept, the martingale system carries inherent risks, as it assumes unlimited funds for doubling bets and disregards the fact that losing streaks can persist longer than expected. Thus, this system will eventually result in bankruptcy.

Reference [3] however argues that different perspectives exist regarding whether stock price movements adhere strictly to a random walk, often modeled as a geometric Brownian motion. This suggests a potential for enhancement in the martingale betting system. The author has subsequently introduced an enhanced martingale betting system that includes a stop policy.

Findings

-The paper proposes an Improved Martingale Betting System (IMBS) by modifying the traditional martingale strategy with a stop policy and adapting it from casino gambling to intraday trading.

-The IMBS is empirically tested using TAIEX (TX) futures across three intraday trading strategies.

-Results show that the IMBS delivers strong performance and is applicable to TX intraday trading and related markets.

-The study finds that returns increase with leverage up to a certain threshold, beyond which traditional martingale strategies face a high probability of bankruptcy.

-By controlling key parameters—specifically leverage scaling (a), the number of steps (n), and total leverage—the IMBS significantly outperforms both the Equal-Weight Betting System (EWBS) and the traditional Martingale Betting System (MBS).

-The inclusion of a stop-loss mechanism further improves performance and risk control.

-Empirical tests indicate that IMBS performs particularly well when combined with price breakout strategies, which are identified as the most profitable approach for TX intraday trading.

In short, after testing on real data, the article concludes that

-The conventional martingale betting system inevitably leads to bankruptcy,

-With the integration of a stop policy, the new and improved martingale betting system demonstrates enhanced efficacy.

Reference

[3] Ting-Yuan Chen, and Szu-Lang Liao, Improved Martingale Betting System for Intraday Trading in Index Futures—Evidence of TAIEX Futures, Asian Journal of Economics and Business, Year:2023, Vol.4 (2), PP.339-366

Closing Thoughts

Taken together, the two studies highlight the trade-off between growth maximization and risk control in position sizing. The Kelly-based approach demonstrates strong theoretical and empirical growth performance, but its apparent alpha weakens once transaction costs and model limitations are accounted for, raising questions about real-world applicability. By contrast, the Improved Martingale Betting System shows that disciplined leverage control and stop policies can materially improve intraday trading outcomes relative to naive martingale schemes, especially when combined with breakout strategies. Overall, both strands of research suggest that position sizing is as critical as signal generation, and that practical constraints, parameter calibration, and market frictions ultimately determine whether theoretically attractive sizing rules translate into sustainable performance.