Backtesting: what it does and does not prove

Backtesting: what it does and does not prove

Anyone reading about a systematic investment strategy eventually asks the question that matters most: does it work? The usual answer is a backtest, which applies a strategy’s rules to historical prices to see how it would have performed. A backtest is a useful tool, but it proves less than the charts often suggest. This article explains what a backtest does show, where it falls short and why we deliberately publish no results on this page.

What a backtest is

A backtest is essentially a reconstruction: you apply a strategy’s rules to past prices and work out which trades the system would have made at the time. This works for a systematic strategy precisely because its rules are fixed in advance. An investor who trades on instinct cannot recreate their own past decisions; they simply do not know what they would have done. A rule-based system does not have that problem: every trade follows from the rules and can be reproduced.

Why a backtest is valuable

Being able to reproduce every trade has two benefits. First, backtesting forces precision: to test a strategy against history, every rule has to be defined exactly in advance, including the edge cases. Vague intentions do not survive that process. Second, a backtest shows how a strategy behaves in sustained rallies, in sharp sell-offs and in the directionless periods that tend to be hardest for trend-following systems. This is where the rules described in our article on trend following can be tested against historical market data.

Where a backtest falls short

A backtest also has structural weaknesses. Five of them come up most often, and anyone evaluating a backtest should understand them.

Overfitting: fitting the past too closely

The biggest pitfall is overfitting, also called curve-fitting: adjusting the rules until they match the historical data almost perfectly. The result looks impressive, but the rules then describe the quirks of one particular period rather than a lasting pattern. The more settings a system has, and the more often its designer adjusts them, the greater the risk that the backtest mainly confirms itself. Future markets will not reproduce that particular set of conditions.

Look-ahead bias: using information you did not yet have

A subtler error is look-ahead bias: using information that was not available at the moment the decision was made. Using a day’s closing price for a trade that would have been executed earlier that same day, for example, quietly lets the future leak into the test. In a spreadsheet this is easy to miss; in the results it can make a large difference.

Survivorship bias: the coins that disappeared

Test a strategy only on the crypto assets that exist today and you leave out every coin that disappeared along the way. This is survivorship bias: only the survivors remain in the dataset. In crypto this is far from a minor issue; projects fail, lose their liquidity or are delisted from exchanges. A test on survivors alone flatters the results, because the biggest losers are missing.

Costs and slippage: theory versus practice

On paper, a system trades for free and at exactly the quoted price. In practice every trade incurs transaction costs, and there is slippage: the difference between the expected price and the price at which the trade is actually executed. For a strategy that trades regularly, these differences add up. A backtest that ignores costs and slippage will systematically overstate performance.

In-sample versus out-of-sample

The last distinction may be the most important. In-sample means tested on the data used to design the rules. Out-of-sample means tested on data the system has never seen. The second gives a better indication of whether a strategy holds up beyond its design; the first mainly shows how well the designer knows the dataset. Even a favourable out-of-sample result can still be down to chance.

Stronger evidence than an impressive backtest

An impressive backtest is therefore no proof that a strategy will perform the same way in the future. Treat it as an entrance exam rather than a final verdict. Anyone assessing a strategy seriously should look at four things beyond the simulation itself. Simplicity of the rules: the fewer settings, the less room to mimic the past. Robustness out of sample: the approach should hold up on data it was not designed on. Execution in practice: rules only prove themselves when they are actually followed in turbulent markets, after all transaction costs. Live performance is an important additional test here, though it should always be weighed against the length of the live period and the market conditions it covered. And drawdowns and risk: how deep the interim losses were matters as much as the gains. Anyone who talks only about gains and says nothing about losses deserves a critical reading. Our article on drawdowns explains why those losing periods are part of a trend-following approach.

Why we show no figures here

You may have noticed that this article contains no performance figures. That is deliberate. We do not publish returns in isolation in an educational article, but in the context of the fund itself, with the period, the source of the figures and the risk warning alongside them. A percentage without that context says little and can easily suggest more than it supports. If you want to understand the strategy in more depth, the fund documentation is the place to start a conversation with us.

Investing in crypto involves substantial risk; you may lose part or all of your investment.

What an independent party establishes

A backtest describes a simulated series. The actual value of the fund is a separate matter: at Callisto Capital it is determined monthly by an independent administrator, outside the model.

Callisto Capital is a Dutch fund for joint account; its manager is registered with the AFM under the AIFMD-light regime and is not subject to ongoing supervision. Participation is open from an investment of 100,000 euro. This article is for information only and does not constitute investment advice.