A calibrated weather model, judged by its forecasts
Daily-high forecasts for US cities, trained on CRPS (a score for whole forecasts) and promoted to live only through pre-registered, leakage-safe tests.
Built with Python, CRPS, NWP and AI weather models, walk-forward backtesting, block bootstrap.
- 20US cities covered
About this number
Cities with daily high-temperature contracts the model forecasts. As of Oct 2026.
Why it mattered
Kalshi lists CFTC-regulated contracts on each day’s high temperature in 20 US cities. A forecast is only useful there if its probabilities are calibrated. That means a bracket priced at thirty percent should happen about thirty percent of the time.
This page covers how good the forecasts are and how the model is tested. I leave profit, positions and trade sizes out.
Problem
Raw weather-model forecasts are not good enough on their own, because each model is biased differently by city and by season.
Approach
A model trained on CRPS blends several national numerical weather prediction models and AI weather models into calibrated probabilities for each temperature bracket.
Every model change is decided by a pre-registered, leakage-safe walk-forward backtest with block-bootstrap confidence intervals. A statistical gate decides when a shadow model is promoted to live. Execution fails closed, which means it reconciles every order with the exchange and stops on any mismatch.
What I got wrong
- What I assumed going in
- That backtest skill would carry over to live without a shadow period.
- What I'd measure first next time
- Reliability with block-bootstrap bands, from the first week.
- What it can't do yet
- Days with fast-moving fronts stay poorly calibrated.