Backtesting: what makes a backtest worth trusting

What a backtest is actually worth

In quant and systematic trading, a backtest is the first attempt to answer a genuinely hard question: if this process had existed earlier, how would it have behaved through real market conditions? Most strategy ideas sound plausible before they meet data. The backtest forces the model to confront a record of trend reversals, volatility shocks, liquidity squeezes, crowded positioning, and the long quiet stretches when very little works.

What you have at the end of that exercise, however, is a controlled reconstruction of the past. Not evidence of what comes next.

For serious investors, the value of a backtest is narrower than pitch decks usually imply. It can help assess process credibility, drawdown character, turnover burden, and whether a strategy still looks economically plausible after the basic frictions of live trading are added back in. It cannot certify future returns. It cannot make up for sloppy work upstream.

A good backtest tells you less than people think. The point is not to admire what it shows. The point is to figure out how much trust the result actually deserves.

The base rates behind that caution are measured, not rhetorical. Chague, De-Losso and Giovannetti tracked 19,646 Brazilians who day traded mini-Ibovespa futures between 2013 and 2015; among those who persisted for more than 300 sessions, 97% lost money (Day Trading for a Living?, SSRN, 2020).

A backtest cannot move anyone into the profitable 3%. What disciplined testing changes is where the losing happens: in simulation, where the tuition is cheap, rather than in a live account, where it is not.

What a backtest can usefully remove is naivety. Strategies that look promising in theory often look quite different once a researcher has to specify exactly when trades would have executed, at what prices, with what costs, on which dates, and with what information actually available at the time.

The act of specification has value of its own. Vague ideas cannot be backtested. Honest results emerge only when someone is forced to be precise about every assumption.

Here is the part that gets understated. The integrity of a backtest is mostly determined before the first line of test code runs. By the data. By the universe definition. By the cost model. By the discipline applied to specifying what the strategy actually is. After all that, the simulation itself is mostly arithmetic. The interesting story sits in everything that came before.

What the research says about backtest reliability

Most articles about backtesting describe what it is. The more useful question is how reliable backtests tend to be in live use. The academic record on that question is considerably less flattering than the average pitch deck implies.

Bailey and López de Prado’s 2014 paper in the Notices of the American Mathematical Society made the case bluntly. Many quantitative claims rest on bad statistics, narrative overreach, and weak inference dressed up to look like serious work.

The paper formalized what experienced fund selectors already observed informally: a strategy that has survived a research process involving many specifications, parameter sweeps, and window selections is not the same thing as a strategy that has been tested.

The backtest reflects the best result from a search. Not the expected result from the strategy. That distinction is almost never disclosed in a presentation.

Harvey, Liu, and Zhu’s 2016 paper in the Review of Financial Studies went further. By the time it was published, academic finance had documented several hundred supposedly distinct equity factors. The paper argued that the standard significance hurdle, a t-statistic of 2.0, was no longer adequate given how exhaustively the factor space had been explored.

The authors proposed raising the bar to roughly 3.0 for newly published factors. By Harvey and Liu’s own 2019 census, the count of published factors in top journals had passed 400, with subsequent work documenting more than 450 by the early 2020s.

What happened next was more interesting than the proliferation itself. Hou, Xue, and Zhang published a large replication study in 2020 that re-ran the catalogued anomalies on cleaner samples and tighter methodology. The result was striking. Around 60% of the published findings could not be replicated at the conventional t-statistic of 1.96. Once the data-snooping-adjusted hurdle of 2.78 was applied, roughly 80% became insignificant. Not just weak. Insignificant.

The most recent work with direct relevance for serious investors has gone in the opposite direction: not finding more, but compressing what survives. Swade, Hanauer, Lohre, and Blitz, writing in the Journal of Portfolio Management‘s 2024 Quantitative Special Issue, asked how many factors are needed to span the available alpha across the entire zoo.

Their answer was around 15. The rest, by their analysis, are redundant. Different labels on a small number of underlying economic exposures.

The Deflated Sharpe Ratio, published by López de Prado and Bailey in 2014 in the Journal of Portfolio Management, gave professional investors a working tool through this whole period. It corrects a reported Sharpe for non-normal return distributions and, more importantly, for the number of trials run before the strategy was selected.

A Sharpe of 1.2 from a single test means something different from a Sharpe of 1.2 picked out of fifty parameter combinations. Both will print the same number on a tear sheet.

One practical convention deserves a note. Many professional investors apply a flat 50% haircut to a reported backtest Sharpe before taking it seriously. Harvey and Liu themselves, in their 2015 Backtesting paper, argued this is a serious mistake.

Their work shows the appropriate haircut is non-linear: marginal-Sharpe strategies should be discounted heavily or thrown out entirely, while genuinely high backtest Sharpes are penalised more lightly because they are more likely to reflect real signal. The 50% rule is convenient. It is also wrong in both directions, depending on where on the distribution the strategy actually sits.

What the body of work adds up to is a simple message that almost never makes it into a sales conversation: backtests are evidence, sometimes weak evidence, and the burden of proof should sit with the manager presenting them. Not with the investor evaluating them.

The way most pitch processes work is precisely the opposite. The backtest is presented as the credibility, and the investor is expected to spot what is missing without being told.

That asymmetry is the problem.

A research Sharpe of 2.0 settles near 1.1 after launch, and trading costs take two to three points a year before anything else goes wrong.

Where backtesting earns its keep, and where it lies

Backtesting is worth doing, but not for the reason most tutorials give. Its real value is disqualification: a rigorous test kills bad strategies cheaply, before they consume real capital. As a prediction tool it is weak. As a rejection tool it is close to irreplaceable.

That inversion changes how the whole exercise should be read. A backtest can demonstrate that a strategy is broken, because the evidence of breakage is concrete: costs eat the signal, the edge turns out to live entirely in three months of 2020, performance collapses when a parameter moves by 10%.

It cannot demonstrate that a strategy is good. Passing a backtest only means one kind of failure did not appear, on one stretch of history, under one set of assumptions.

Worth saying directly: we consider a backtest that rejects a strategy more informative than one that flatters it. The rejection is usually durable. The flattery rarely is.

Where does backtesting lie? Mostly where it stops being a test and becomes a search.

A researcher who tries eighty variants and presents the strongest is not reporting an experiment, whatever the slide says; he is reporting the luckiest draw from a distribution the audience never sees, which is why the disclosure of discarded variants matters more than the shape of the curve itself.

The lie is rarely in the arithmetic. It is in the selection.

For a serious investor, the working posture follows from all of this. Treat backtesting as the filter that clears the ground for harder questions about capacity, costs, and governance. Strategies that fail it deserve no further time. Strategies that pass it have earned exactly that. Further time.

The anatomy of a fragile backtest

The biases that undermine a backtest are almost never declared. They accumulate quietly inside a research process, each one individually defensible, all of them collectively distorting. Understanding where they hide tends to be more useful than memorizing a checklist of what to avoid.

Survivorship bias is not obvious when a researcher is using a vendor’s universe; that is just the universe available. Look-ahead bias often enters through a corporate action field that was revised after the date in question, not through any intentional choice. Optimization bias often feels like thoroughness: testing many parameter values to find the best specification.

Each of these has an honest explanation in isolation. The trouble is what happens when they accumulate.

Seven of these biases, layered together, can produce an equity curve that looks institutional and is in fact fragile. The diagram below names them on a stylized chart. The shape tends to be familiar: the in-sample period is flattering, the out-of-sample stretch holds up just well enough to seem credible, and then live trading proceeds to disappoint in ways the simulation never imagined.

Each bias on the chart is fixable. Most are reasonably easy to detect, once an investor knows where to look. The hard part is that the same research process that produces the bias often also produces a confident narrative explaining why the bias does not really apply here. That narrative is usually polished. It is rarely accurate.

Where backtests break in live trading

In the decks that cross our desk, the first sign that a backtest may be fragile is rarely technical. It is the absence of bad years that ought to be there. Every market-making strategy had a difficult August 2015. Every statistical arbitrage book had an awkward February 2018. Every trend-following program went through a long stretch of underperformance from 2011 to 2014. If those years look smooth in a presentation, the next reasonable question is what the assumptions were during those periods.

The places where backtests and live trading diverge tend to follow a pattern. Costs are understated in research; live trading reveals them larger than the model assumed.

Execution prices that simulations treat as the midpoint or close turn into queue position, adverse selection, and partial fills the moment real money flows. And capacity, which most backtests treat as effectively infinite, starts to bite at AUM levels well below where the marketing materials suggested.

The August 2007 episode is worth studying because the failure mode there was not a bad model. The models were fine. The failure was that backtesting cannot account for market structure. Specifically, it cannot account for what happens when many independent researchers make similar discoveries, allocate to similar trades, and run into liquidity problems on the same morning. A beautifully tested individual backtest can be a crowded trade at scale. No backtest methodology resolves that.

Out-of-sample evidence: what a backtest can and cannot show

In-sample testing is where a model is built. Out-of-sample testing is where credibility starts to accumulate, though never where it ends. That distinction matters because a strategy can look excellent on the data used to construct it and disappoint almost everywhere else.

Serious research teams stress a model outside the period that shaped it. They test across regimes, rebalance assumptions, cost ranges, and parameter perturbations. They want to know whether the signal stays directionally intact when conditions become less flattering.

Not every result has to look strong. In fact, uniform strength in every regime is itself a warning sign. Strategies that genuinely worked through 2008 usually look different from strategies that genuinely worked through 2017, and a model that performs identically in both is more likely a fitted curve than a durable process.

A question worth asking when reviewing any research presentation: does the team distinguish between strategies that failed out-of-sample and strategies they chose not to show? The answer matters. A team that ran forty variations and is showing you the best three is presenting something different from a team that ran three variations and is showing you all three. Both presentations look identical on a slide.

Winton has acknowledged in its public research that live trend-following products tend to underperform their backtests after launch. The acknowledgment is unusual because most managers do not say it out loud. It is also worth taking seriously, because Winton has had thirty years to study the gap, and the firms that have been doing this longest tend to be the most candid about how persistent the gap is.

We have not met a serious quantitative manager who claims to have fully solved the live-versus-backtest problem. The ones who speak most specifically about it tend to be the ones managing it most carefully.

Do professional traders backtest?

Yes, universally, and far more skeptically than retail tutorials suggest. Professional desks treat backtesting as one stage in a validation pipeline: point-in-time data, out-of-sample discipline, cost models, capacity tests, and a research log. The backtest that convinces an institutional committee is the one that survived attempts to kill it.

The difference from the retail version is not the software. It is the intent. A retail tutorial backtests to confirm an idea someone already likes; an institutional research group backtests to refute one, and staffs the process with people whose job security does not depend on the strategy surviving.

Man AHL has run systematic programs through three decades of regime shifts, which means three decades of measured gaps between what simulations promised and what markets paid.

A pattern we have learned to look for in manager materials: where the backtest sits in the deck. Retail-grade presentations offer it as the conclusion. Institutional-grade presentations offer it as one exhibit among several, alongside the research log, the cost model by turnover band, and the record of variants tried and discarded. One of those decks can be interrogated. The other can only be admired.

The professional habit most worth copying is unglamorous. Write down what was tried. A validation process that cannot show its failures is not a validation process. It is a highlight reel.

Can ChatGPT backtest a trading strategy?

It can write backtest code in seconds. It cannot be trusted to run an honest test. When QuantPedia put ChatGPT through a documented backtesting exercise in October 2023, it hallucinated confidently, miscalculated volatility, and produced results that changed between runs. Useful assistant. Unreliable examiner.

The failure modes matter because they are silent: the code runs, the curve renders, and the errors sit underneath, where a hurried researcher will not look. We examine the tool properly, including what happens when its strategies meet clean out-of-sample data, in our evaluation of ChatGPT for trading.

Due diligence questions for any backtest

Most due diligence questions about backtests sit on a checklist somewhere. The ones that matter rarely do. The questions below are not technical audits. They are indicators of how a research team actually thinks. That second category is what determines whether a backtest can be trusted.

Start with the data. Was every input point-in-time, meaning the values used on each date were the ones actually available on that date, or were revisions, restatements, and corporate actions allowed to leak backwards into the historical record?

Was the trade universe defined using only information that existed at the time, or did the screening rules quietly include knowledge of which names later became important? These are not pedantic questions. The answers reveal whether the team understands what survivorship bias and look-ahead bias actually are.

Then turn to costs. What cost model was used in the backtest, and how did it vary by turnover, liquidity, and position size? Were spreads, commissions, financing, and market impact each modelled separately, or were they collapsed into a flat percentage that has the elegance of being almost certainly wrong?

At what level of AUM does the strategy start to degrade, and was that tested directly rather than asserted in conversation?

Robustness is the third pillar. How many parameter variations were tested before the final specification was selected? What happened on the variations that did not get selected? Across which specific market regimes did the strategy struggle, and does the pattern of struggle match what the underlying economic logic would have predicted? A team that cannot answer the last question has trouble. The strategy worked, but they cannot tell you why.

Finally, governance. Is there a research log showing what was changed and when? Can the team reconcile current live performance against their backtest expectations at the trade or sleeve level? Who has the authority to override the model during a drawdown, and is that authority written down anywhere?

The backtest is rarely the weak-looking part. The weak part is usually what sits around it: assumptions, exclusions, friction, and the explanation of what changed after live trading began.

Backtesting questions investors ask