logaritmisk/trueskill-tt
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Activity

feat!: name the unknown key, expose tail probabilities, flag short fits

Browse Source 
Three issues from two downstream consumers, all small, all sharing a
theme: the crate had the information and would not hand it over.

#44 — `UnknownKey { team: 0, member: 0 }` did not say which key. A
consumer upgrading 0.1.2 -> 0.4.1 had every one of 5591 predictions
return this error, fell back to a neutral 0.5, and lost its entire
metadata model for a day. Nothing crashed and nothing logged; it was
found by sweeping an unrelated parameter and noticing the output did not
move. The 0.4.0 change that made unknown keys an error was right — the
error was just too anonymous to act on. It now carries the key's `Debug`
rendering, and its `Display` says what to do about it. The precondition
is documented on every prediction entry point, which the reporter said
would alone have saved the day.

#43 — `cdf` was `pub(crate)`, so a consumer asking "is this competitor
below the cutoff" approximated it with a `mu + z * sigma` band and had no
way to say what confidence any `z` bought. Adds
`Gaussian::probability_below` / `probability_above`. The second is
separate on purpose: `1 - cdf` collapses to exactly zero past ~8.3
sigma, and a stopping rule is evaluated precisely there. Both route
through the survival function added in 0.4.1, so this is visibility
rather than new numerics.

#50 — `ConvergenceReport` was not `#[must_use]`, so the one signal that
a fit stopped short was trivially discarded. It now is, and that
immediately found 78 sites doing exactly that — including this crate's
own ATP example, which was capped at 10 sweeps when the history needs
30. The example now reads the report and says so.

`ITERATIONS = 30` is documented as the floor it is, with the three
measurements to hand: 400 events over 100 competitors already stops
there at ~7e-3 against a 1e-6 tolerance, the ATP example needs 30 at a
much looser one, and a consumer's 2000-node model needs 76 to 161.

BREAKING CHANGE: `InferenceError::UnknownKey` gains a `key` field, and
the prediction methods now require `K: Debug` in order to fill it.

Closes #43, #50. Refs #44 — its third ask, an opt-in `UnknownKeys::Skip`
mode, is a live API question and deliberately not answered here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
logaritmisk
2026-09-07 23:41:03 +02:00
co-authored by Claude Opus 5
parent 901f60972e
commit c12bc830a5
21 changed files with 396 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/history.rs
+105 -33
View File
@@ -583,7 +583,10 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
/// reported rather than dropped — silently skipping them would turn a team
/// of strangers into a confident-looking prediction about nobody, which is
/// the failure this replaced.
fn member_skills(&self, teams: &[&[&K]]) -> Result<Vec<Vec<Gaussian>>, InferenceError> {
fn member_skills(&self, teams: &[&[&K]]) -> Result<Vec<Vec<Gaussian>>, InferenceError>
where
K: std::fmt::Debug,
{
if teams.len() < 2 {
return Err(InferenceError::NotEnoughTeams { got: teams.len() });
}
@@ -600,6 +603,7 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
let unknown = InferenceError::UnknownKey {
team: team_idx,
member: member_idx,
key: format!("{key:?}"),
};
let index = self.keys.get(*key).ok_or(unknown.clone())?;
members.push(
@@ -625,7 +629,10 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
/// # Errors
///
/// As [`History::member_skills`].
fn performances(&self, teams: &[&[&K]]) -> Result<(Vec<Gaussian>, Vec<usize>), InferenceError> {
fn performances(&self, teams: &[&[&K]]) -> Result<(Vec<Gaussian>, Vec<usize>), InferenceError>
where
K: std::fmt::Debug,
{
let skills = self.member_skills(teams)?;
let performances = skills
@@ -669,10 +676,23 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
/// "is this matchup *informative*" — the two coincide for two evenly
/// matched teams and diverge elsewhere.
///
/// # Preconditions
///
/// Every key must already be known to the history — that is, must have
/// appeared in an ingested event. An unknown key is `UnknownKey`, not a
/// silently dropped member. If your caller cannot guarantee that, pre-filter
/// with [`History::lookup`] or [`History::current_skill`]; treating the
/// error as "no information" and substituting a neutral value turns a
/// whole-team miss into a plausible constant, which is invisible to any
/// test that does not assert on variation.
///
/// # Errors
///
/// `NotEnoughTeams`, `EmptyTeam`, or `UnknownKey`.
pub fn predict_quality(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> {
pub fn predict_quality(&self, teams: &[&[&K]]) -> Result<f64, InferenceError>
where
K: std::fmt::Debug,
{
let groups = self.member_skills(teams)?;
let group_refs: Vec<&[Gaussian]> = groups.iter().map(Vec::as_slice).collect();
Ok(crate::quality(&group_refs, self.beta))
@@ -697,9 +717,22 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
///
/// # Errors
///
/// # Preconditions
///
/// Every key must already be known to the history — that is, must have
/// appeared in an ingested event. An unknown key is `UnknownKey`, not a
/// silently dropped member. If your caller cannot guarantee that, pre-filter
/// with [`History::lookup`] or [`History::current_skill`]; treating the
/// error as "no information" and substituting a neutral value turns a
/// whole-team miss into a plausible constant, which is invisible to any
/// test that does not assert on variation.
///
/// As [`History::member_skills`], plus `TooManyTeams` and anything
/// inference returns for a hypothetical outcome.
pub fn expected_information_gain(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> {
pub fn expected_information_gain(&self, teams: &[&[&K]]) -> Result<f64, InferenceError>
where
K: std::fmt::Debug,
{
let skills = self.member_skills(teams)?;
let ratings: Vec<Vec<Rating<T, D>>> = skills
@@ -737,10 +770,23 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
/// not factorially. Prefer this to [`History::predict_outcome`] when you
/// only need to know who wins.
///
/// # Preconditions
///
/// Every key must already be known to the history — that is, must have
/// appeared in an ingested event. An unknown key is `UnknownKey`, not a
/// silently dropped member. If your caller cannot guarantee that, pre-filter
/// with [`History::lookup`] or [`History::current_skill`]; treating the
/// error as "no information" and substituting a neutral value turns a
/// whole-team miss into a plausible constant, which is invisible to any
/// test that does not assert on variation.
///
/// # Errors
///
/// `NotEnoughTeams`, `EmptyTeam`, or `UnknownKey`.
pub fn predict_win_probabilities(&self, teams: &[&[&K]]) -> Result<Vec<f64>, InferenceError> {
pub fn predict_win_probabilities(&self, teams: &[&[&K]]) -> Result<Vec<f64>, InferenceError>
where
K: std::fmt::Debug,
{
let (performances, sizes) = self.performances(teams)?;
Ok(crate::predict::win_probabilities(
&performances,
@@ -770,10 +816,23 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
/// use [`History::predict_ranking`], and when you only need the winner use
/// [`History::predict_win_probabilities`]; both stay cheap at any size.
///
/// # Preconditions
///
/// Every key must already be known to the history — that is, must have
/// appeared in an ingested event. An unknown key is `UnknownKey`, not a
/// silently dropped member. If your caller cannot guarantee that, pre-filter
/// with [`History::lookup`] or [`History::current_skill`]; treating the
/// error as "no information" and substituting a neutral value turns a
/// whole-team miss into a plausible constant, which is invisible to any
/// test that does not assert on variation.
///
/// # Errors
///
/// `NotEnoughTeams`, `EmptyTeam`, `UnknownKey`, or `TooManyTeams`.
pub fn predict_outcome(&self, teams: &[&[&K]]) -> Result<Prediction, InferenceError> {
pub fn predict_outcome(&self, teams: &[&[&K]]) -> Result<Prediction, InferenceError>
where
K: std::fmt::Debug,
{
if teams.len() > crate::MAX_PREDICTED_TEAMS {
return Err(InferenceError::TooManyTeams {
got: teams.len(),
@@ -800,9 +859,22 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
///
/// # Errors
///
/// # Preconditions
///
/// Every key must already be known to the history — that is, must have
/// appeared in an ingested event. An unknown key is `UnknownKey`, not a
/// silently dropped member. If your caller cannot guarantee that, pre-filter
/// with [`History::lookup`] or [`History::current_skill`]; treating the
/// error as "no information" and substituting a neutral value turns a
/// whole-team miss into a plausible constant, which is invisible to any
/// test that does not assert on variation.
///
/// `NotEnoughTeams`, `EmptyTeam`, `UnknownKey`, or `MismatchedShape` if
/// `ranks` does not have one entry per team.
pub fn predict_ranking(&self, teams: &[&[&K]], ranks: &[u32]) -> Result<f64, InferenceError> {
pub fn predict_ranking(&self, teams: &[&[&K]], ranks: &[u32]) -> Result<f64, InferenceError>
where
K: std::fmt::Debug,
{
if ranks.len() != teams.len() {
return Err(InferenceError::MismatchedShape {
kind: "ranks vs teams",
@@ -1513,7 +1585,7 @@ mod tests {
epsilon = 1e-6
);
h1.converge().unwrap();
let _ = h1.converge().unwrap();
assert_ulps_eq!(
h1.time_slices[0].skills.get(a).unwrap().posterior(),
@@ -1558,7 +1630,7 @@ mod tests {
epsilon = 1e-6
);
h2.converge().unwrap();
let _ = h2.converge().unwrap();
assert_ulps_eq!(
h2.time_slices[2].skills.get(a).unwrap().posterior(),
@@ -1591,7 +1663,7 @@ mod tests {
&[5, 6, 7],
);
h.add_events(events).unwrap();
h.converge().unwrap();
let _ = h.converge().unwrap();
let lc_a = h.learning_curve("a");
let lc_c = h.learning_curve("c");
@@ -1633,7 +1705,7 @@ mod tests {
&[1, 2, 3],
);
h.add_events(events).unwrap();
h.converge().unwrap();
let _ = h.converge().unwrap();
let a = h.keys.get("a").unwrap();
let b = h.keys.get("b").unwrap();
@@ -1724,7 +1796,7 @@ mod tests {
let evidence_third_event = h.log_evidence_internal(false, &[a]).exp() * 2.0;
assert_ulps_eq!(0.669885, evidence_third_event, epsilon = 1e-6);
h.converge().unwrap();
let _ = h.converge().unwrap();
let loocv_hat = h.log_evidence_internal(false, &[]).exp();
let p_d_m_hat = h.log_evidence_internal(true, &[]).exp();
@@ -1789,7 +1861,7 @@ mod tests {
let b = h.keys.get("b").unwrap();
let c = h.keys.get("c").unwrap();
h.converge().unwrap();
let _ = h.converge().unwrap();
assert_eq!(h.time_slices[2].skills.get(b).unwrap().elapsed, 2);
assert_eq!(h.time_slices[2].skills.get(c).unwrap().elapsed, 1);
@@ -1838,7 +1910,7 @@ mod tests {
]
);
h.converge().unwrap();
let _ = h.converge().unwrap();
assert_ulps_eq!(
h.time_slices[0].skills.get(a).unwrap().posterior(),
@@ -1886,7 +1958,7 @@ mod tests {
let b = h.keys.get("b").unwrap();
let c = h.keys.get("c").unwrap();
h.converge().unwrap();
let _ = h.converge().unwrap();
assert_eq!(h.time_slices[2].skills.get(b).unwrap().elapsed, 2);
assert_eq!(h.time_slices[2].skills.get(c).unwrap().elapsed, 1);
@@ -1935,7 +2007,7 @@ mod tests {
]
);
h.converge().unwrap();
let _ = h.converge().unwrap();
assert_ulps_eq!(
h.time_slices[0].skills.get(a).unwrap().posterior(),
@@ -2001,7 +2073,7 @@ mod tests {
epsilon: EPSILON,
alpha: 1.0,
};
h.converge().unwrap();
let _ = h.converge().unwrap();
let loocv_approx_2 = h.log_evidence_internal(false, &[]).exp().sqrt();
@@ -2052,7 +2124,7 @@ mod tests {
&[0, 10, 20],
);
h.add_events(events).unwrap();
h.converge().unwrap();
let _ = h.converge().unwrap();
let a = h.keys.get("a").unwrap();
let b = h.keys.get("b").unwrap();
@@ -2116,7 +2188,7 @@ mod tests {
assert_eq!(h.time_slices[0].skills.get(b).unwrap().elapsed, 0);
assert_eq!(h.time_slices[end].skills.get(b).unwrap().elapsed, 5);
h.converge().unwrap();
let _ = h.converge().unwrap();
assert_ulps_eq!(
h.time_slices[0].skills.get(b).unwrap().posterior(),
@@ -2155,7 +2227,7 @@ mod tests {
&[0, 10, 20],
);
h2.add_events(events).unwrap();
h2.converge().unwrap();
let _ = h2.converge().unwrap();
let a = h2.keys.get("a").unwrap();
let b = h2.keys.get("b").unwrap();
@@ -2219,7 +2291,7 @@ mod tests {
assert_eq!(h2.time_slices[0].skills.get(b).unwrap().elapsed, 0);
assert_eq!(h2.time_slices[end].skills.get(b).unwrap().elapsed, 5);
h2.converge().unwrap();
let _ = h2.converge().unwrap();
assert_ulps_eq!(
h2.time_slices[0].skills.get(b).unwrap().posterior(),
@@ -2294,7 +2366,7 @@ mod tests {
epsilon = 1e-6
);
h.converge().unwrap();
let _ = h.converge().unwrap();
let lc_a = h.learning_curve("a");
let lc_b = h.learning_curve("b");
@@ -2368,11 +2440,11 @@ mod tests {
})
.build();
events_for(&mut h_capped);
h_capped.converge().unwrap();
let _ = h_capped.converge().unwrap();
let mut h_full: History<i64, _, _, &'static str> = History::builder().build();
events_for(&mut h_full);
h_full.converge().unwrap();
let _ = h_full.converge().unwrap();
let curves_capped = h_capped.learning_curves();
let curves_full = h_full.learning_curves();
@@ -2409,7 +2481,7 @@ mod tests {
let mut h_undamped: History<i64, _, _, &'static str> = History::builder().build();
events_for(&mut h_undamped);
h_undamped.converge().unwrap();
let _ = h_undamped.converge().unwrap();
let mut h_damped: History<i64, _, _, &'static str> = History::builder()
.convergence(ConvergenceOptions {
@@ -2419,7 +2491,7 @@ mod tests {
})
.build();
events_for(&mut h_damped);
h_damped.converge().unwrap();
let _ = h_damped.converge().unwrap();
let curves_u = h_undamped.learning_curves();
let curves_d = h_damped.learning_curves();
@@ -2453,7 +2525,7 @@ mod tests {
outcome: Outcome::scores_with_sigma([3.0, 1.0], 0.5),
}])
.unwrap();
h_a.converge().unwrap();
let _ = h_a.converge().unwrap();
// Path B: history-wide default 0.5, no per-event override.
let mut h_b = crate::History::builder().score_sigma(0.5).build();
@@ -2466,7 +2538,7 @@ mod tests {
outcome: Outcome::scores([3.0, 1.0]),
}])
.unwrap();
h_b.converge().unwrap();
let _ = h_b.converge().unwrap();
// Inheritance: posteriors must be bit-equal.
let curves_a = h_a.learning_curves();
@@ -2495,7 +2567,7 @@ mod tests {
outcome: Outcome::scores_with_sigma([3.0, 1.0], 2.0),
}])
.unwrap();
h_a.converge().unwrap();
let _ = h_a.converge().unwrap();
// Path B: history-wide default 2.0, no per-event override.
let mut h_b = crate::History::builder().score_sigma(2.0).build();
@@ -2508,7 +2580,7 @@ mod tests {
outcome: Outcome::scores([3.0, 1.0]),
}])
.unwrap();
h_b.converge().unwrap();
let _ = h_b.converge().unwrap();
// Override == default-set-to-the-override-value: bit-equal.
let curves_a = h_a.learning_curves();
@@ -2532,7 +2604,7 @@ mod tests {
outcome: Outcome::scores([3.0, 1.0]),
}])
.unwrap();
h_c.converge().unwrap();
let _ = h_c.converge().unwrap();
let curves_c = h_c.learning_curves();
let mut max_diff: f64 = 0.0;
@@ -2561,7 +2633,7 @@ mod tests {
.scores_with_sigma([3.0, 1.0], 2.0)
.commit()
.unwrap();
h_a.converge().unwrap();
let _ = h_a.converge().unwrap();
// Path B: same outcome via the explicit Outcome constructor.
let mut h_b = crate::History::builder().score_sigma(0.5).build();
@@ -2574,7 +2646,7 @@ mod tests {
outcome: Outcome::scores_with_sigma([3.0, 1.0], 2.0),
}])
.unwrap();
h_b.converge().unwrap();
let _ = h_b.converge().unwrap();
let curves_a = h_a.learning_curves();
let curves_b = h_b.learning_curves();