A checklist for vetting any forecasting tool (ours included)

What accuracy really means, how to read a calibration table, how to spot overfitting, and how track records get filtered — here it all comes together as a practical checklist you can run against any forecasting product before you pay, including this one.

The vetting checklist

Which kind of accuracy is being claimed? Directional hit-rate, calibration, or an error metric — and is a hit-rate quoted against its base rate, or on its own? A number without its base rate is marketing. Is there a calibration table? When it says “90% confident,” do 90% of outcomes actually land in the band? Was it tested out-of-sample, on data the model never saw during tuning? Is the track record complete — a fixed, pre-declared set with the misses shown next to the hits? Does it show where it’s weak? Every real model fails somewhere. And what exactly are you paying for — access and convenience are honest to sell; a promise of profit or “guaranteed returns” is always a red flag.

Running the checklist on Quantustik itself

It would be hypocritical to teach this and exempt ourselves, so here’s how we score against our own list. Quantustik publishes a live calibration page showing stated-vs-actual coverage per horizon; on the committed TOP-20 backtest, measured against the model as served, the 90% bands have covered roughly 90.6% (3-month), 90.9% (6-month) and 90.2% (1-year) of outcomes — all three within a point of the 90% target and inside the well-calibrated 85–95% range. Those figures come from held-out evaluation, not the data the model was tuned on, and the same page discloses where coverage is poor: individual stocks vary widely, and names with a structural business-model break are weakly covered at the 1-year horizon. The track record is live and auto-updating, failures included. We sell access, convenience, and history depth — never a promise of profit. That’s the standard; hold every tool, including this one, to it.

Why this matters for your money

You don’t need to be a statistician to avoid the worst tools — you need to ask six questions and notice which ones go unanswered. The tools worth paying for tend to answer readily and show their failures; the ones to avoid deflect to a single impressive number. That instinct — “show me the receipts, including the misses” — is the most durable investing skill this whole path is trying to build.

This lesson is investor education, not personalized advice. The checklist helps you avoid the worst tools; it does not endorse any tool or imply an expected return. Quantustik sells access and convenience, never a promise of profit.

Where this comes from

Frequently asked questions

What’s the single most useful question to ask a forecasting tool?

“Show me where you’re wrong.” A tool that publishes its misses — a calibration table, a complete track record with losers included, disclosed weak spots — has earned a look. One that deflects to a single flattering number has not.

Does Quantustik pass its own checklist?

It aims to: it publishes a live calibration page with stated-vs-actual coverage per horizon, a live track record with failures included, and it discloses where the model is weak. It sells access, convenience, and history — never a promise of profit. You’re encouraged to run the checklist and check for yourself.

Do I need to be a statistician to use this checklist?

No. The checklist is six plain-language questions. You don’t have to compute anything — you just have to notice which questions a tool answers openly and which ones it dodges. The dodges are the signal.

Related glossary terms

Continue this course

Investor education only — not investment advice, and never a promise of profit. Every investment can lose value.