What is a Brier score?

The Brier score grades how well-calibrated a probabilistic forecast is — 0 is perfect, 0.25 is a coin flip, and higher is worse.

Why a lower Brier score beats a bolder-sounding claim

A model that always shouts "90% chance" sounds confident but posts a terrible Brier score the moment reality disagrees even a third of the time — squaring the error punishes overconfident misses hard. A model quoting a modest 55% when the true edge really is marginal, and proving right about that edge over many trials, scores well even though no single call looks impressive.

Where this shows up at Quantustik

Brier score is used on the Fear & Greed backtest, scoring each candidate model's P(index > 50) at each day of the forecast horizon, averaged, against what the index actually did. It is a companion to per-ticker backtest CI90 accuracy: that metric grades whether a price band contained the outcome; Brier score grades whether a stated probability was honest.

Reading a Brier score correctly

Treat 0.25 as the free baseline every model must beat, not a passing grade. Because the committed backtest window is finite, small differences between two models' Brier scores (say, 0.21 vs. 0.19) are closer to noise than to a meaningful edge — look for scores that are consistently, not marginally, below 0.25 across multiple independent windows before treating a model's calibration as proven.

Frequently asked questions

Is a lower Brier score always better?

Yes — 0 is perfect foresight and 0.25 is the coin-flip baseline for a binary forecast, so any score meaningfully below 0.25 beats guessing.

How is Brier score different from a hit rate?

A hit rate only checks whether the final call was right; Brier score also grades whether the stated confidence level matched how often that outcome actually happened.

Where does Quantustik report Brier score?

On the Fear & Greed backtest, scoring each model's stated probability that the index will be above 50 at the forecast horizon against what actually happened.