Aditya Rathi

Personal project, 2026

A calibrated weather model, judged by its forecasts

Daily-high forecasts for US cities, trained on CRPS (a score for whole forecasts) and promoted to live only through pre-registered, leakage-safe tests.

Built with Python, CRPS, NWP and AI weather models, walk-forward backtesting, block bootstrap.

  • 20US cities covered
    About this number

    Cities with daily high-temperature contracts the model forecasts. As of Oct 2026.

How the weather model is built and judgedNumerical weather prediction models and AI weather models feed a blend trained on CRPS, which gives a probability for each daily-high temperature bracket. When the day settles, the outcome scores the forecasts for calibration and CRPS. A model change goes live only after pre-registered, leakage-safe walk-forward backtests and a statistical gate on a shadow model. Numbers shown: 20 US cities covered.Numerical weatherpredictionseveral nationalmodelsAI weather modelsBlend trained on CRPSeach source is biased its own way by city andseasonBracket probabilitiesa probability for each daily-high bracket20 US cities coveredSettled days score the forecastscalibration and CRPSStatistical gatea change goes live only after pre-registered,leakage-safe walk-forward backtests and a shadowperiodpromotes a new blend
Forecast quality and the tests a change must pass.

Why it mattered

Kalshi lists CFTC-regulated contracts on each day’s high temperature in 20 US cities. A forecast is only useful there if its probabilities are calibrated. That means a bracket priced at thirty percent should happen about thirty percent of the time.

This page covers how good the forecasts are and how the model is tested. I leave profit, positions and trade sizes out.

Problem

Raw weather-model forecasts are not good enough on their own, because each model is biased differently by city and by season.

Approach

A model trained on CRPS blends several national numerical weather prediction models and AI weather models into calibrated probabilities for each temperature bracket.

Every model change is decided by a pre-registered, leakage-safe walk-forward backtest with block-bootstrap confidence intervals. A statistical gate decides when a shadow model is promoted to live. Execution fails closed, which means it reconciles every order with the exchange and stops on any mismatch.

What I got wrong

What I assumed going in
That backtest skill would carry over to live without a shadow period.
What I'd measure first next time
Reliability with block-bootstrap bands, from the first week.
What it can't do yet
Days with fast-moving fronts stay poorly calibrated.