Every predict_* is a ranked outcome — there is no predictive distribution for a SCORE #48

New Issue

2026-09-07T21:26:50Z

logaritmisk commented

2026-09-07 21:26:50 +00:00

predict_quality, predict_win_probabilities, predict_outcome and predict_ranking all answer "who wins, and with what probability". A consumer that records Outcome::scores_with_sigma never asks that question. It wants: what will this competitor score against this opposition, and how wide is that interval?

There is no such call, so a score-based consumer has to build one — and doing it from marginals goes wrong exactly where it matters most.

What it costs

ustat records every round as a scored duel and reimplements this as a hand-fitted noise law: predicted mean from the node means, predicted variance as floor + node_weight × node_var, with both constants fitted empirically on held-out rounds. Measured over ~200 held-out duels:

cohort	n	quoted σ	residual sd	within 68%
repeat appearances	133	5.83	~5.8	61.7%
first appearances	72	5.83	12.44	36.1%

A first appearance gets the same interval as a well-known competitor, because the fitted node_weight came out at 0.0 — so the quoted sigma is constant whether the model's own node_var is 12.7 or 44.9. The interval collapses precisely where the model knows least. For this app that is the headline feature: predicting a course nobody has played.

The information is in the model, and the consumer cannot get at it. Refitting the same law on both cohorts rather than one:

corr(node_var, residual²) is −0.066 among repeats and +0.401 among first appearances
a law that carries node_var beats the flat one by −0.314 ± 0.141 paired NLPD, costing almost nothing on repeats (3.105 vs 3.084) and gaining a lot on first appearances (3.888 vs 4.693)

So the model's own uncertainty does track its accuracy. It only looks worthless if you fit the relationship on the cohort where every competitor is equally well known — which is the mistake a consumer makes when the crate gives it no guidance.

What would help

/// Predictive distribution of `teams[i]`'s score, priors and node
/// uncertainty included, for a matchup that may involve unseen competitors.
fn predict_score<Q>(&self, teams: &[&[&Q]], options: &GameOptions)
    -> Result<Vec<Gaussian>, InferenceError>;

Two properties matter more than the exact shape:

It must widen for an unseen or thinly-evidenced competitor. That is the whole point; a call that returns a constant interval is the bug above with a nicer name.
It must handle unseen keys rather than returning None. "I have never seen this competitor, here is the prior-informed answer" is a legitimate and useful reply, and it is what a consumer will otherwise fake badly.

Related: #46 — a predictive interval over a combination of competitors needs the joint anyway, so these two probably want to land together.

`predict_quality`, `predict_win_probabilities`, `predict_outcome` and `predict_ranking` all answer "who wins, and with what probability". A consumer that records `Outcome::scores_with_sigma` never asks that question. It wants: **what will this competitor score against this opposition, and how wide is that interval?** There is no such call, so a score-based consumer has to build one — and doing it from marginals goes wrong exactly where it matters most. ## What it costs `ustat` records every round as a scored duel and reimplements this as a hand-fitted noise law: predicted mean from the node means, predicted variance as `floor + node_weight × node_var`, with both constants fitted empirically on held-out rounds. Measured over ~200 held-out duels: | cohort | n | quoted σ | residual sd | within 68% | |---|---|---|---|---| | repeat appearances | 133 | 5.83 | ~5.8 | 61.7% | | **first appearances** | 72 | **5.83** | **12.44** | **36.1%** | A first appearance gets the *same* interval as a well-known competitor, because the fitted `node_weight` came out at 0.0 — so the quoted sigma is constant whether the model's own `node_var` is 12.7 or 44.9. The interval collapses precisely where the model knows least. For this app that is the headline feature: predicting a course nobody has played. **The information is in the model, and the consumer cannot get at it.** Refitting the same law on both cohorts rather than one: - `corr(node_var, residual²)` is **−0.066** among repeats and **+0.401** among first appearances - a law that carries `node_var` beats the flat one by **−0.314 ± 0.141 paired NLPD**, costing almost nothing on repeats (3.105 vs 3.084) and gaining a lot on first appearances (3.888 vs 4.693) So the model's own uncertainty *does* track its accuracy. It only looks worthless if you fit the relationship on the cohort where every competitor is equally well known — which is the mistake a consumer makes when the crate gives it no guidance. ## What would help ```rust /// Predictive distribution of `teams[i]`'s score, priors and node /// uncertainty included, for a matchup that may involve unseen competitors. fn predict_score<Q>(&self, teams: &[&[&Q]], options: &GameOptions) -> Result<Vec<Gaussian>, InferenceError>; ``` Two properties matter more than the exact shape: 1. **It must widen for an unseen or thinly-evidenced competitor.** That is the whole point; a call that returns a constant interval is the bug above with a nicer name. 2. **It must handle unseen keys rather than returning `None`.** "I have never seen this competitor, here is the prior-informed answer" is a legitimate and useful reply, and it is what a consumer will otherwise fake badly. Related: #46 — a predictive interval over a combination of competitors needs the joint anyway, so these two probably want to land together.

logaritmisk added the api enhancement labels 2026-09-07 21:26:50 +00:00

logaritmisk referenced this issue

2026-09-07 21:27:06 +00:00

`expected_information_gain` is discrete-outcome only, so scored consumers cannot ask "which matchup next?" #49

logaritmisk referenced this issue from a commit

2026-09-07 23:28:54 +00:00

feat: add UnknownKeys::Prior, and explain why there is no Skip

logaritmisk referenced this issue from a commit

2026-09-07 23:46:40 +00:00

feat: add History::posterior_of for a linear combination of competitors

logaritmisk referenced this issue from a commit

2026-09-08 00:02:21 +00:00