Way of the Turtle — Chapter 5: Backtesting & Out-of-Sample Validation
阅读中文版 (with Audio)Way of the Turtle Chapter 5: Recognizing overfitting, the necessity of out-of-sample validation, and why a beautiful backtest is often the most dangerous signal.
🔊 Listen to Article (Chinese Audio)
Way of the Turtle — Chapter 5: Backtesting & Out-of-Sample Validation
"You can always find a perfect system in historical data. The problem is that it's only perfect in history." — Curtis Faith
Financial Context
Faith devotes considerable space to a counterintuitive phenomenon: the more beautiful the backtest, the more likely the system fails in live trading.
The reason is that when you adjust parameters repeatedly until historical performance is optimal, you have not discovered a market regularity — you have merely described what happened to occur in that particular stretch of history. This is overfitting: with enough parameters you can explain any past perfectly while retaining no predictive power at all.
Wall Street Application
1. Four Warning Signs of Overfitting
- Too many parameters: A system with seven or eight tunable inputs is almost certainly fitted. The Turtle system has three core parameters: breakout period, the N lookback window, and the stop multiple.
- Extreme parameter sensitivity: If a 20-day breakout is highly profitable while 19-day and 21-day both lose, you have found noise rather than structure.
- Too few trades: An "edge" validated on 15 trades carries no statistical meaning.
- A flawless equity curve: Real trend systems necessarily contain long flat periods and deep drawdowns. A smoothly rising backtest curve is itself an alarm.
2. Parameter Plateau vs. Parameter Spike
This is the chapter's most practical diagnostic:
- Parameter spike (dangerous): 20-day breakout returns 30% annually; 19-day returns 8%; 21-day returns 6%.
- Parameter plateau (reliable): Breakout periods from 18 to 25 days all return between 18% and 24%.
Conclusion: prefer a suboptimal parameter in the middle of a plateau over the optimal parameter on a spike. A plateau means the system is insensitive to parameter choice and therefore more likely to have captured genuine market structure.
3. Standard Out-of-Sample Practice
- Split history into in-sample (about 70%) and out-of-sample (about 30%).
- All parameter selection and rule adjustment happens only in-sample.
- Out-of-sample data may be used exactly once. If out-of-sample results are materially worse, discard the system — going back to "tweak it and retest" immediately contaminates the out-of-sample set.
4. Do Not Forget Costs
The most common fatal omission in backtesting is transaction cost. A system trading 200 times per year with 0.15% round-trip cost (commission plus slippage) surrenders 30 percentage points of gross return annually. Many "25% annualized" high-frequency systems have negative expectancy once costs are included.
Risk Management Rules
- Set aside out-of-sample data before you start: Seal the most recent 30% before looking at any result. This is the only reliable defense against self-deception.
- Choose plateaus, not spikes: Abandon any system whose performance swings wildly on small parameter changes.
- Estimate costs pessimistically: Model slippage at twice observed fills. Only a system with positive expectancy under pessimistic cost assumptions deserves real capital.
Relevance to a Retirement Portfolio
For retirement investors this chapter is primarily defensive: it explains why quantitative products, signal subscriptions, and trading courses advertising "30% backtested returns" are usually untrustworthy. Learning to recognize overfitting lets you reject the large majority of products you should never touch — which protects retirement assets more than finding any single good strategy.