Testing & risk
Reading a probability forecast: confidence, calibration and the misses
A forecast that says “62 % up” is only useful if things it calls 62 % happen about 62 % of the time. That is calibration — and it has to be shown, not claimed.
A probability, not an order
A probabilistic forecast does not say what will happen. It says how often, in similar situations, one outcome has followed. A 62 % forecast that turns out wrong was not necessarily a bad forecast — it was the 38 % case.
Calibration and hit rate
- Hit rate. How often the forecast direction was right overall.
- Calibration. Whether forecasts of 60 % come true about 60 % of the time, forecasts of 70 % about 70 %.
- Out of sample. Only forecasts made before the outcome count. Fitted history proves nothing.
Why misses must stay on the record
A model that only shows its hits is advertising, not evidence. A complete, permanent record of every out-of-sample forecast — right and wrong — is the only way to judge whether it adds anything.
TakeawayTrust a forecast as far as its published, out-of-sample record allows — no further.
In Drift Engine
Drift forecast states a direction per currency as a probability with confidence. The ML lab keeps every out-of-sample forecast on record, and Edge & calibration shows the hit rate and calibration openly, misses included.
Educational content. Not investment advice and not a recommendation to buy or sell any financial instrument. Trading currencies involves a high risk of loss.