How the weather model is built and tested
This page covers how the forecast model works and how every change is tested before it ships. It also covers the live-trading safety rails and calibration.
The forecast
The target is each city’s official daily high, as the National Weather Service settles it. Its day runs from midnight to midnight in local standard time all year, so in summer it ends an hour after midnight by the clock.
Each run reads 8 forecasts of that high: US and European physics models, the national blend, the local forecast office, a European ensemble, and two AI weather models.
The model is EMOS (ensemble model output statistics): a Student-t distribution with 7 degrees of freedom. Its center is a weighted sum of the forecasts plus a per-city correction; the weights are shared across cities and cannot go negative. Its width can grow when ensemble members disagree, but today the fit gives that term no weight, so the width is one fitted value, never below 2.5 °F.
It is refit every day on the previous 30 days, strictly before the day it forecasts. The fit minimizes CRPS, a score that covers the whole distribution. The fit starts from 8 points and keeps the best, so it doesn’t flip between solutions from one day to the next.
Kalshi settles on whole degrees, rounded half up, so each bracket covers the true high from half a degree below its first value to half a degree above its last. A bracket’s probability is the distribution’s mass between those edges.
This model is the system’s only source of probabilities, live included, so there is no fallback forecast. If the fit is too old, or too much of its weight would rest on missing inputs, that city sits the day out.
How a change is tested
Every change is pre-registered. The options to try, the score, the sanity floors, the bootstrap and the rule for promotion are written down and committed before any run; results land in a later commit. If a test cannot tell two options apart, it counts as a failure, and I do not reopen the argument.
The hardest lesson came from an audit. It found that some stored forecasts had been re-scored after the day’s peak and some inputs had been overwritten in place, so past forecasts could not be rebuilt as they were at the time. Tests now use only what was known at decision time, which means inputs fetched before a fixed cutoff that falls ahead of any city’s afternoon peak.
Testing walks forward through time. Each step fits on the days before and scores the next day, then moves on by one day. I never use shuffled folds, because they would let the model learn from the future.
Uncertainty comes from a block bootstrap, which resamples whole dates in runs of 7 days and does this 1,000 times. Every city on a date stays together, because neighboring days and neighboring cities share weather.
Every model is compared with three baselines: climatology (the long-run normal), the weather models’ plain average, and the market’s own prices.
Safety rails
An order can go out only if the exchange’s own status check said yes in this run. The gate fails closed, so a missing or malformed answer counts as no, as does an exchange it cannot reach.
Live trading needs its own credentials and an explicit live setting. Anything ambiguous falls back to shadow, which logs what it would do and sends nothing.
I count the exchange as the source of truth. An order that might exist is treated as existing, and any mismatch between the exchange and my records halts trading and pages me.
An hourly health check pages me when something drifts, and a single file on disk stops all new orders.
A model change goes through the pre-registered test, then a side-by-side shadow run, then a sealed forward test, then a go or no-go by a person. A release goes through a pinned checkout, frozen dependencies, parity tests and a canary before it runs.
Calibration, explained
A forecast is calibrated when its probabilities come true as often as they say. Of all the brackets given thirty percent, about thirty percent should happen.
PIT: where each observed high falls within its own forecast distribution. If the forecasts are calibrated, those positions spread evenly; a hump in the middle means the forecasts are too wide, piles at the edges mean too narrow.
Coverage: how often the high lands inside the forecast’s central ranges (half, four in five, and nineteen in twenty of its probability).
The reliability curve plots forecast probability against how often it happened, fitted by isotonic regression, which also splits the Brier score into miscalibration, discrimination and uncertainty.
The scores tracked are the Brier score, a ranked probability score across the brackets, and log loss.