Variance ratio as a measure of backtest reliability

March 16, 2015 12:00 PM

A complete trading system embodies the principal trading strategy or trade trigger rule, risk management, and a standardized evaluation process of system efficiency, which is generally confugured with backtesting.

These primary blocks are mutually connected: We can’t configure the trading system rules without backtesting. The cycle of backtesting and algorithm optimization is closed. Traders manually interrupt it or introduce self-learning modules for automated trading. However, in both cases, questions arise. What volume of historical data is needed to set parameters properly? How much should be chosen to optimize strategies, yield/risk relation, risk per trade, etc.?

Let’s select at least three criteria that are supported by successful investors:


Psychological significance: What is the minimal amount of data needed to be confident about a trading approach? Hundreds of price bars, thousands or even millions? In part, the answer depends on the psychological makeup of the trader.

Homogeneity of market conditions: A qualitative change of market structure introduces updates of system parameters. Can we trade forex in the same way if a balanced approach has been replaced by expansionary monetary policy? Most would say no because  market properties have changed explosively. However, long-term backtesting typically ignores such historical shifts and is based on continuous testing. This is despite simple logic that suggests a mean-reversion strategy may work well in one scenario, while a trend-following system might work better in another.

Sustainability of results: This criterion is probably the most important. It lets us satisfy the subjective requirements of a trader and to discover hidden market conversions, such as volatility gaps, company mergers or central bank actions. Sustainability may be interpreted as model settings conservancy under the condition of a slow volume change. Will the optimal yield/risk relationship change significantly if you add 1% of historical data volume to the backtest analysis? If the answer is yes, then trading system sustainability is doubtful.

Backtesting allows researchers to obtain the empirical distribution of relative yield. The key statistics, and others such as mean yield and maximum drawdown, can be derived from this distribution. Each yield interval corresponds to a certain number of executed trades. If the whole interval of yield data lies between -10% and 40%, an empirical distribution may be represented in the form of a table (see “Empirical distribution,” right).

In the table, the trade’s fraction is defined on the basis of a whole number of trades; N - five subintervals are used. A corresponding number of subintervals depends on required accuracy for yield definition. 

Page 1 of 3
About the Author

Sergey Kamenshchikov graduated with a PhD from Moscow State University, where he remains a research associate. He currently works for IFC Markets in London and is a principal analyst of financial markets.