Why can one backtest mislead you?
A backtest replays a set of trading rules over historical prices. Most strategies have adjustable settings: the length of a moving average, the distance of a stop, the threshold that triggers an entry. If you try many combinations and keep the one with the strongest historical result, you are selecting the settings that happened to suit that particular stretch of history, including its random noise. This is called overfitting, or curve fitting. The reported result then describes how well the settings were fitted, not how well the underlying idea works.
The problem grows with the number of combinations tried. Test enough variations of a rule on purely random data and some will look impressive by chance alone. A single backtest gives you no way to tell a real pattern from a lucky one, because the same data both chose the settings and graded them.
What is the difference between in-sample and out-of-sample data?
In-sample data is the history you use to build or tune a strategy. Out-of-sample data is history you hold back and do not look at while tuning. Out-of-sample results are more informative because the settings could not have been fitted to them.
The simplest version is a single split: tune on the first 70 percent of the history, then test once on the remaining 30 percent. This is better than no split, but it has two weaknesses. It produces only one verdict, which may depend heavily on which period happened to land in the test portion. And if you look at the out-of-sample result, change the rules, and test again, that data has quietly become in-sample.
How does walk-forward validation work?
Walk-forward validation repeats the in-sample and out-of-sample split many times, moving through history in order, the way a real trader would have experienced it:
- Choose an optimization window (the data used for tuning) and a test window (the data used for scoring).
- Tune the strategy's settings using only the first optimization window.
- Freeze those settings and run the strategy on the test window that immediately follows. Record the result.
- Slide both windows forward by the length of the test window and repeat from step 2.
- Join all the test-window results into one continuous out-of-sample record and compare it with the in-window results.
There are two common variants. A rolling walk-forward keeps the optimization window the same length and drops the oldest data as it moves. An anchored walk-forward keeps the start date fixed, so the optimization window grows. Rolling emphasizes recent conditions; anchored uses more data at each step.
Hypothetical example
The numbers below are made up to show the mechanics. They do not describe any real strategy or market. Suppose you have 10 years of daily data, a 3-year optimization window, and a 1-year test window. Rolling forward gives seven steps, because years 1 to 3 are used only for the first round of tuning.
| Step | Optimization years | Test year | In-window return | Out-of-window return |
|---|---|---|---|---|
| 1 | 1 to 3 | 4 | 14% | 6% |
| 2 | 2 to 4 | 5 | 12% | -3% |
| 3 | 3 to 5 | 6 | 13% | 8% |
| 4 | 4 to 6 | 7 | 11% | 2% |
| 5 | 5 to 7 | 8 | 15% | 5% |
| 6 | 6 to 8 | 9 | 12% | -4% |
| 7 | 7 to 9 | 10 | 13% | 7% |
| Simple average | 12.9% | 3.0% | ||
Hypothetical annual returns for illustration only. In-window returns are the average yearly return over each 3-year optimization window.
A single backtest over the full 10 years, with settings tuned on all of it, would probably show something close to the in-window figures. The walk-forward record shows what happened each time the tuned settings met a year they had not seen: about 3.0% on average, positive in five of seven windows, with two losing years. The gap between 12.9% and 3.0% is the cost of fitting to the past, and it is information a single backtest hides.
How do you choose window lengths?
There is no universal answer, only trade-offs.
- Optimization window: it needs enough trades to estimate how the rules behave. A rule that trades a few times a year needs a much longer window than one that trades daily. Some practitioners make the optimization window several times longer than the test window, such as three to one or more, but that is a convention rather than a law.
- Test window: short enough that tuned settings are still relevant when applied, long enough to contain a meaningful number of trades. A test window with only two or three trades says very little.
- Number of windows: more windows give a more varied record, but each one must still hold enough trades to mean something.
Decide these lengths before you look at any results. Adjusting window lengths until the output looks good is just another way of overfitting.
What does a good walk-forward result look like, and what are its limits?
Signs that a result deserves more attention include:
- Out-of-window results that are positive in most windows, rather than depending on one exceptional year.
- Out-of-window performance that is reasonably close to in-window performance. Some decay is normal. A very large gap points to overfitting.
- Tuned settings that stay similar from one window to the next. Settings that jump around suggest the rules are fitting noise.
- Losses in the test windows that are small enough that you could have tolerated them.
The limits are real. Walk-forward validation still uses one body of history, so if you test fifty different strategy ideas and keep the one that passes, you may have selected a lucky result. It does not repair lookahead bias or survivorship bias in the underlying data. Trading costs and slippage need to be included or the figures will be flattering. Finally, it describes how a method behaved in the past; markets change, and it cannot predict what comes next.
Where Thulo fits
Thulo is an analytical platform. It lets you describe a strategy in plain English and backtest it against historical data. Techniques such as walk-forward validation are ways of reading and questioning those results. Thulo does not provide investment advice or tell you what to trade.