Calibration vs. directional accuracy: why we publish one and not the other

"Is this model accurate?" usually means two different questions at once: does its confidence band cover reality at the rate it claims, and does its forecast point in the direction the market actually went? Quantustik answers the first question in public, continuously. It does not yet answer the second in public, and this page explains why that is a deliberate choice, not an oversight.

Two different questions wearing one word

"Accurate" gets used for two unrelated claims. Calibration asks: when we say a price has a given chance of landing in a range, does reality agree with us that often, over many forecasts? Directional accuracy asks: when the forecast pointed up or down, did the market actually move that way — what fraction of the calls got the sign right? A model can score well on one and poorly on the other — they measure different things.

A wide, well-calibrated band can cover the real outcome almost by construction, without ever committing to a useful direction. A narrow, directionally lucky model can look impressive on a small sample while its stated confidence levels are badly miscalibrated. Neither number alone tells you whether a trade based on the forecast would have worked.

The calibration page answers the first question, continuously and in public, including the misses.

Why we do not publish a win-rate yet

Publishing a performance number after the fact is easy to get wrong in a way that flatters the publisher: pick a favorable window, a favorable subset of tickers, or restate the criteria after seeing the results, and almost any track record can be made to look good. The only defense against that is deciding the measurement rules before the results exist and refusing to touch them afterward.

That's what the track record page is: a pre-registered set of criteria and a live clock. Until the tracked outcomes accumulate to the minimum sample the criteria call for, we do not publish a public win-rate or return figure — not because the number would necessarily be bad, but because a number published before its own gate clears is not evidence, it is a promise, and promises about money are exactly what we are not allowed to sell.

Calibrated does not mean profitable

Say it plainly: a model can be well-calibrated and still make you no money, or lose you money. Calibration tells you the band is honest about its own uncertainty. It does not tell you the band is narrow enough to act on, that the direction inside the band is the one worth betting on, or that the cost of being wrong is one you can afford. Those are trade-plan questions — sizing, entries, exits, invalidation — that calibration alone cannot answer.

The same discipline applies to directional accuracy itself: a hit-rate is not a P&L. A model that gets the sign right 55% of the time can still lose money if the average loss outweighs the average win, and one that is right only 45% of the time can be profitable with the asymmetry reversed. Coverage, hit-rate, and profit are three different numbers, and only the first is on our public record today.

This is also why "well-calibrated" is not a marketing claim here. It is the minimum bar for a forecast to be worth reading at all, not proof that reading it will make you money.

Where this fails

This page is itself a limitation statement: we are telling you we do not publish a win-rate, and why. A visitor who wants proof the model makes money will not find it here — that is the point. A perfectly calibrated model can still lose you money if you trade on it mechanically without position sizing, entries, and exits; calibration is necessary context for a trade plan, not a substitute for one.

See the evidence

Frequently asked questions

Educational research only — not investment advice.