What is overfitting in a backtest?
A backtest replays a set of rules against historical prices. Any historical period contains two things mixed together: patterns that may recur, and random noise that will not. Overfitting is what happens when a strategy is shaped to match both, including the noise. The result is a strategy that fits the past almost perfectly and has little to say about anything else.
A useful comparison is a student who memorizes last year's exam answers instead of learning the subject. The memorized answers score perfectly on last year's exam and poorly on a new one. An overfit strategy does the same with price history.
Why does overfitting happen?
Three causes show up again and again.
- Too many parameters. Every adjustable setting, such as a moving-average length, a threshold, a stop distance, or a time-of-day filter, gives the strategy another way to bend toward the data. With enough settings, almost any historical curve can be reproduced.
- Repeated testing on the same data. Each time you run a test, look at the result, tweak the rules, and run it again on the same history, information from that history leaks into your design. After enough rounds, the data has effectively been used to write the strategy.
- Data snooping and multiple comparisons. If you search through many ideas and keep the one that looks strongest, the winner is partly selected for luck. The next section explains why this matters so much.
How many variations have you tried?
This is the most important question to ask about any backtest result, and it is the one most often skipped.
Suppose you flip a coin ten times and get eight heads. That is mildly surprising. Now suppose you flip 1,000 coins ten times each and report only the coin that landed eight or more heads. That result is not surprising at all, because with enough tries something will look unusual by chance alone. This is the multiple-testing problem.
Strategy search works the same way. Trying 50 combinations of settings and reporting the top-ranked one is not the same as testing one idea. A rough rule of thumb: the more variations you try, the higher the bar a result must clear before you take it seriously. A result that survives after you honestly count every attempt is far more meaningful than one that does not.
A hypothetical example
The numbers below are made up to illustrate the pattern. They are not real results.
Imagine a simple moving-average crossover rule. You test 40 combinations of fast and slow lengths on 2016 to 2020 data, then pick the combination with the highest annualized return. You then run that same combination on 2021 to 2024 data that was set aside and not used for tuning.
| Hypothetical test | Annualized return |
|---|---|
| Average of all 40 combinations, tuning period | 3% |
| Top-ranked combination, tuning period | 14% |
| Same combination, unseen period | 1% |
Hypothetical example for illustration only.
The top-ranked combination looks impressive in the tuning period, but part of that is selection: out of 40 tries, one was bound to look good. On unseen data, the result falls back near the ordinary level. The gap between 14% and 1% is the cost of overfitting, and you only see it if you kept data in reserve.
What are the warning signs?
- Unusually high fit on history. Very smooth, steep results with tiny drawdowns deserve suspicion, not celebration. Real strategies are usually messier.
- Results collapse with tiny parameter changes. If moving a setting from 20 to 22 turns a strong result into a weak one, the strategy likely found a narrow quirk in the data.
- Too many rules. Long lists of conditions, filters, and exceptions often exist because each one patched a past failure.
- No economic rationale. If you cannot explain in a sentence or two why the rule should work, such as a behavioral tendency or a structural feature of the market, it may simply be a coincidence.
- A short sample. A handful of trades, or a single market regime, is too little evidence to separate skill from luck.
How do you reduce it?
- Use fewer parameters. Every setting you remove is one less way to fit noise. If a rule needs eight settings to work, be skeptical.
- Test out of sample. Split history into a tuning period and a holdout period. Tune on the first, then evaluate on the second, ideally once. Looking repeatedly at the holdout and adjusting turns it into tuning data.
- Use walk-forward testing. Tune on one window, test on the window that follows, then roll both forward and repeat. This checks whether the tuning process itself holds up across time. See walk-forward validation for more.
- Check parameter stability. Test settings near your chosen values. A sound idea tends to work across a neighborhood of similar settings, not at one isolated point.
- Include realistic costs. Commissions, spreads, and slippage can erase a thin edge. Strategies that trade often are especially sensitive to this.
- Use more data and more regimes. Cover rising, falling, and sideways markets, and calm and volatile periods, so one environment cannot carry the result.
- Prefer simple rules with a stated reason. Write down why the rule should work before you test it. This limits the temptation to invent explanations afterward.
Where Thulo fits
Thulo is an analytical platform that lets you describe a strategy in plain English and backtest it against historical data. The habits above apply to any backtesting tool: keep the idea simple, hold data back, and keep count of how many versions you try. Thulo does not provide investment advice, and the decisions about what to test and what to conclude remain yours.