Overfitting is when a model memorises the noise in past data instead of learning the pattern. It looks brilliant on history and falls apart on anything new — which is exactly the failure mode that makes a spectacular backtest worth nothing.
Imagine a student who prepares for an exam by memorising the answers to last year's paper. On last year's paper they score 100%. On this year's paper they are lost, because they never learned the subject — they learned the answer key. That student is overfit.
A model does the same thing when it has enough freedom to bend itself around every wiggle in the historical data. Every market has coincidences in it: a stock that happened to rise on Tuesdays, a pattern that happened to precede three rallies. A model with enough parameters will happily "discover" those coincidences and present them as insight. They will not repeat, because there was never anything there to repeat.
Example (illustrative): you test 1,000 randomly generated trading rules against ten years of history. Purely by chance, a handful of them will show a superb hit rate — that is what "1 in 1,000" means. If you then publish the best one and show its backtest curve, the chart will look fantastic and the strategy will be worthless. Nothing was faked; the selection itself created the illusion. This is why "we tested many ideas and this one worked" is a warning sign, not a credential, unless the tester also tells you how many ideas they tested.
The standard defences are all versions of "make the model prove itself on data it has never seen." Hold out a slice of history the model was never fitted on and test there. Prefer simpler models with fewer knobs, because fewer knobs means less room to memorise noise. And check calibration, not just accuracy: a model that says "90% confident" should be right about 90% of the time, and a model that is overfit usually turns out to be far more confident than it has any right to be.
This is why we publish a calibration page showing how often our confidence intervals actually contained the realised price — including the horizons and tickers where coverage is poor. A page that only showed the wins would be exactly the selection effect described above. The related question of whether AI stock predictions are accurate at all is worth reading alongside this one.
Knowing about overfitting does not let you detect it from the outside. When someone shows you a backtest, you generally cannot tell how many variants they tried before landing on that one. The only real defence available to you as a reader is to ask for out-of-sample or forward results and to distrust any curve that looks too clean. Apply that scepticism to us too — that is the point of publishing backtest accuracy with its failures visible. None of this is investment advice.
Because the backtest was run on the same data the model was tuned on. Any sufficiently flexible model can be bent to fit the past perfectly. The past performance was, in a sense, an input rather than a result.
You cannot prove it is not, but the strongest evidence is performance on data that did not exist when the model was built — a genuine forward-looking track record — plus calibration checks that show its stated confidence matches its real hit rate.
Not always, but they are harder to overfit, because they have fewer parameters to bend around noise. The trade-off is that a model too simple to capture the real pattern is underfit, which is its own failure — the craft is finding the balance.
Browse all S&P 500 tickers to see this metric applied to individual companies.
Educational research only — not investment advice.