NFL props: how our numbers have held up

# NFL props board: pre-registered grading rule

Written 2026-10-19, before the grader existed and before any NFL props row had been compared with a box score by this
tool. `tools/nfl_props_grade.py` prints this file's sha256 and the commit that added it into every report, so the rule a
report was graded under can be checked against what was written here first. Mirrors `docs/CFB_PROPS_PREREG.md`.

Unlike the CFB rule there is no pass/fail bar and no "not yet" verdict. The record is public proof of how the board has
held up, so it is published whatever it says, misses included. The sections below fix how it is computed.

## Question

How close are the ATO model's NFL stat projections (pass yards, pass TDs, rush yards, receiving yards, catches, anytime
TD) to what happened, compared with the market line the board showed next to them, and how often does the model's side of
the line (over or under) land?

## What is graded

- The board used for a game is the last archived build whose `generated` time is before that game's kickoff
  (`~/archives/nfl-props-snapshots/<season>-weekNN/props_*.json`). Later builds never replace it. Kickoff comes from the
  nflverse schedule (`gameday` and `gametime`, Eastern time, converted to UTC). A board belongs to a game when its
  `season` and `week` equal the game's and the row's team and opponent are the game's two teams (opponent codes LAR, WSH
  and JAC are mapped to nflverse's LA, WAS, JAX, the same mapping the board builder uses).
- One row per player, stat and game. A row counts only if all of these hold:
  1. the player's `status` on that build is not OUT, DOUBTFUL, IR or D (the builder already drops these; the check is kept
     so the rule does not depend on it);
  2. it has the market in question (see Markets);
  3. the game is final in nflverse (the schedule has a score and the weekly player stats hold rows for that `game_id`);
  4. the player has a stats row for that game, matched on game, normalised name and team. Normalisation is
     `tools.nfl_props_history.norm_name`, the rule the board builder uses (lower case, suffixes Jr/Sr/II/III/IV dropped,
     letters only). A player with no row is a void prop (inactive, or no recorded stat in any category) and is left out of
     the primary numbers, as sportsbooks void those props.
- Sensitivity, reported next to the primary and never instead of it: players with no stats row counted as 0.
- Rows are counted and reported at every drop: no board before kickoff, game not final, status, no market, void.
- The reported actuals are nflverse weekly player stats (`stats_player_week_<season>.parquet`, CC-BY-4.0): `passing_yards`,
  `passing_tds`, `rushing_yards`, `receiving_yards`, `receptions`. Anytime TD is `rushing_tds + receiving_tds >= 1`, the
  same quantity the model's probability is built from. Sensitivity: also counting `special_teams_tds`.

## Markets (kept separate, never pooled)

- **Books**: the consensus line `bk_line` (the line most books hang, at least 2 books, `bk_books >= 2`) and its de-vigged
  over chance `bk_over`. Anytime TD: `bk_over` is the average Yes price across books converted to a chance, so it keeps the
  books' margin; this is stated wherever it is shown.
- **Underdog**: `ud_line` and `ud_over` where the board carried them (they were removed from the page on 2026-10-15, so
  the Underdog arm covers earlier weeks only).
- Kalshi and Novig fields are not graded: Kalshi's terms bar public display and Novig is display-only on the board.

A row can appear in both arms; each arm is analysed on its own rows.

## Comparisons (all with n)

1. **Accuracy, line stats** (pass yards, rush yards, receiving yards, catches, pass TDs). By stat: mean absolute error of
   ours and of the market line, bias (prediction minus actual) for each, the paired mean of (|ours - actual| minus
   |line - actual|) with a 95% bootstrap interval, and the share of rows where ours was closer. The bootstrap resamples
   whole GAMES (players in one game share script and weather, so rows are not independent), 2,000 resamples, seed
   20261019. `ours` is the board's `ours` field: the model's per-game expected stat line scaled to its adjusted projection;
   for pass TDs it is an expected count, the market line is a median-style line, and the report says so.
2. **Lean.** Over when ours is above the line, Under when below; equal means no lean. Pass TDs: Over when the model's
   chance of clearing the line (`ours_over`) is above the market's (`bk_over` or `ud_over`), the comparison the board
   itself shows for that stat. A lean hits when the actual lands on that side. A push (actual equals the line) is reported
   separately and is in neither hits nor misses. Reported overall and by stat for all leans, and for the large-gap bucket:
   **gap percent >= 20 in absolute value**, where gap percent = (ours - line) / max(line, floor) with floors 50 pass yards,
   15 rush or receiving yards, 2 catches, recomputed per market arm; for pass TDs the bucket is a gap of at least 20
   percentage points between the two chances. Each rate carries its Wilson 95% interval and the exact one-sided binomial
   p-value against 50%.
3. **Baselines for the lean.** The market's own lean (Over when its de-vigged over chance is above 50%) on the same rows
   where that chance exists; always Over.
4. **Anytime TD** is a probability and is not graded by MAE. Brier score and log loss (probabilities clipped to
   [0.001, 0.999]) of ours and of the market's implied chance, the paired mean of (Brier ours minus Brier market) with the
   same game-cluster bootstrap, the Brier of a constant equal to the sample's scoring rate (context), and a calibration
   table by ours-probability band (under 10%, 10-20%, 20-35%, 35-50%, 50% and over).
5. **By week and by stat** the same numbers are produced for each NFL week, so the page can show the record week by week.
6. **Largest misses and hits.** In the large-gap bucket on the Books arm: the ten leans that missed by the widest margin
   and the ten that hit by the widest, margin = |actual - line| on the lean side. For anytime TD: the five rows where the
   model's chance beat the market's by the most and the player did not score, and the five where it was lower by the most
   and he scored.

X = 20 is fixed here and is not tuned after seeing results.

## Display rule

A rate or interval whose n is under 30 is not shown on the public page; it says "n under 30, not shown" and the count.
The JSON keeps n for every number and marks a rate with `small: true`.

## Known limits, stated in advance

- The board's lines are only as fresh as its last sportsbook pull before the build; the board can show a line that moved
  later. That can only make the market look worse than its closing line.
- The model does not use the prop lines as an input. It is fitted on earlier games, so weeks 1-6 are out of sample for the
  weekly projections, but the role adjustment on `ours` is the board's own and was changed on 2026-10-10 and 2026-10-17;
  earlier boards were graded as they stood, not re-built.
- Weeks 1 and 2 have thinner coverage: the archive starts at 2026-09-13 15:48Z, so games before that have no board, and
  early builds carried no sportsbook lines on some rows.
- Rows within a game are correlated; intervals resample games, but six weeks is still a small sample.
- Lines are mostly x.5, so pushes are rare; whole-number Underdog lines can push and are counted as such.
- A void prop (player did not play) is left out; a player who played and recorded nothing may also lack a stats row and
  is treated as void in the primary numbers. The sensitivity run counts those as 0.