Backtests: five ways they fool you

A backtest replays a set of rules on past prices to see how they would have done. It is the best tool we have for testing an idea cheaply, and also very easy to get wrong. Every backtest on this site has the limits below.

1. Survivorship bias

Our test uses today's list of NSE stocks. Companies that were delisted, merged or went bankrupt in the last two years are missing, and many of those were losing trades. So the real results would have been a little worse.

2. Overfitting

Try enough rule variations and one will look brilliant on the past by luck alone. The fix: decide the rules first, test once on many trades, and be suspicious of settings that are oddly precise (a 13-day average that works but 12 and 14 do not).

3. Costs and slippage

Brokerage, STT, exchange fees, GST, stamp duty and the gap between the price you see and the price you get. Our test takes 0.1% off each trade. A stop on a thin stock can fill far below its level; that is why the test ignores stocks trading less than ₹5 crore a day.

4. Look-ahead bias

Using information that was not available at the time, for example today's company results to judge a trade from last year. Our test uses only prices known at the close of each day, and buys the next day.

5. Too few trades

Ten trades say almost nothing. As a rough guide, wait for 30+ closed trades before trusting a number, and many more before trusting a small difference between two methods.

So why backtest at all?

Because the alternative is trusting a story. A test with known limits tells you far more than a few memorable winners on a chart.