Three issues from two downstream consumers, all small, all sharing a theme: the crate had the information and would not hand it over. #44 — `UnknownKey { team: 0, member: 0 }` did not say which key. A consumer upgrading 0.1.2 -> 0.4.1 had every one of 5591 predictions return this error, fell back to a neutral 0.5, and lost its entire metadata model for a day. Nothing crashed and nothing logged; it was found by sweeping an unrelated parameter and noticing the output did not move. The 0.4.0 change that made unknown keys an error was right — the error was just too anonymous to act on. It now carries the key's `Debug` rendering, and its `Display` says what to do about it. The precondition is documented on every prediction entry point, which the reporter said would alone have saved the day. #43 — `cdf` was `pub(crate)`, so a consumer asking "is this competitor below the cutoff" approximated it with a `mu + z * sigma` band and had no way to say what confidence any `z` bought. Adds `Gaussian::probability_below` / `probability_above`. The second is separate on purpose: `1 - cdf` collapses to exactly zero past ~8.3 sigma, and a stopping rule is evaluated precisely there. Both route through the survival function added in 0.4.1, so this is visibility rather than new numerics. #50 — `ConvergenceReport` was not `#[must_use]`, so the one signal that a fit stopped short was trivially discarded. It now is, and that immediately found 78 sites doing exactly that — including this crate's own ATP example, which was capped at 10 sweeps when the history needs 30. The example now reads the report and says so. `ITERATIONS = 30` is documented as the floor it is, with the three measurements to hand: 400 events over 100 competitors already stops there at ~7e-3 against a 1e-6 tolerance, the ATP example needs 30 at a much looser one, and a consumer's 2000-node model needs 76 to 161. BREAKING CHANGE: `InferenceError::UnknownKey` gains a `key` field, and the prediction methods now require `K: Debug` in order to fill it. Closes #43, #50. Refs #44 — its third ask, an opt-in `UnknownKeys::Skip` mode, is a live API question and deliberately not answered here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
167 lines
5.4 KiB
Rust
167 lines
5.4 KiB
Rust
//! `quality()` beyond two rating groups.
|
|
//!
|
|
//! The historical golden (two equal singletons) is asserted in
|
|
//! `src/lib.rs::tests::test_quality`. These cover the N-group generalisation,
|
|
//! which previously panicked with an out-of-bounds index at 3+ groups.
|
|
|
|
use trueskill_tt::{Gaussian, quality};
|
|
|
|
const BETA: f64 = 25.0 / 3.0 / 2.0;
|
|
|
|
fn rating(mu: f64, sigma: f64) -> Gaussian {
|
|
Gaussian::from_ms(mu, sigma)
|
|
}
|
|
|
|
#[test]
|
|
fn three_equal_groups_is_finite_and_in_range() {
|
|
let r = rating(25.0, 3.0);
|
|
let q = quality(&[&[r], &[r], &[r]], BETA);
|
|
|
|
assert!(q.is_finite(), "quality must be finite, got {q}");
|
|
assert!((0.0..=1.0).contains(&q), "quality out of range: {q}");
|
|
}
|
|
|
|
#[test]
|
|
fn quality_supports_many_groups() {
|
|
let r = rating(25.0, 3.0);
|
|
for n in 2..=8 {
|
|
let holders: Vec<[Gaussian; 1]> = (0..n).map(|_| [r]).collect();
|
|
let groups: Vec<&[Gaussian]> = holders.iter().map(|g| g.as_slice()).collect();
|
|
let q = quality(&groups, BETA);
|
|
assert!(q.is_finite(), "n={n}: quality must be finite, got {q}");
|
|
assert!((0.0..=1.0).contains(&q), "n={n}: out of range: {q}");
|
|
}
|
|
}
|
|
|
|
/// Equal-strength groups are the best-matched case: introducing a skill gap
|
|
/// must lower quality.
|
|
#[test]
|
|
fn imbalance_lowers_quality() {
|
|
let strong = rating(40.0, 3.0);
|
|
let average = rating(25.0, 3.0);
|
|
|
|
let balanced = quality(&[&[average], &[average], &[average]], BETA);
|
|
let lopsided = quality(&[&[strong], &[average], &[average]], BETA);
|
|
|
|
assert!(
|
|
lopsided < balanced,
|
|
"expected imbalanced quality {lopsided} < balanced {balanced}"
|
|
);
|
|
}
|
|
|
|
/// Quality is a property of the multiset of groups, not their order.
|
|
#[test]
|
|
fn quality_is_permutation_invariant() {
|
|
let a = rating(30.0, 2.0);
|
|
let b = rating(25.0, 3.0);
|
|
let c = rating(20.0, 4.0);
|
|
|
|
let forward = quality(&[&[a], &[b], &[c]], BETA);
|
|
let reversed = quality(&[&[c], &[b], &[a]], BETA);
|
|
|
|
assert!(
|
|
(forward - reversed).abs() < 1e-9,
|
|
"permutation changed quality: {forward} vs {reversed}"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn multi_player_groups_work() {
|
|
let r = rating(25.0, 3.0);
|
|
let q = quality(&[&[r, r], &[r, r], &[r, r]], BETA);
|
|
assert!(q.is_finite());
|
|
assert!((0.0..=1.0).contains(&q));
|
|
}
|
|
|
|
#[test]
|
|
fn uneven_group_sizes_work() {
|
|
let r = rating(25.0, 3.0);
|
|
let q = quality(&[&[r, r], &[r], &[r, r, r]], BETA);
|
|
assert!(q.is_finite(), "got {q}");
|
|
assert!((0.0..=1.0).contains(&q), "got {q}");
|
|
}
|
|
|
|
#[test]
|
|
#[should_panic(expected = "at least 2 rating groups")]
|
|
fn single_group_panics_with_clear_message() {
|
|
let r = rating(25.0, 3.0);
|
|
let _ = quality(&[&[r]], BETA);
|
|
}
|
|
|
|
#[test]
|
|
#[should_panic(expected = "at least 2 rating groups")]
|
|
fn zero_groups_panics_with_clear_message() {
|
|
let _ = quality(&[], BETA);
|
|
}
|
|
|
|
#[test]
|
|
#[should_panic(expected = "non-empty")]
|
|
fn empty_group_panics_with_clear_message() {
|
|
let r = rating(25.0, 3.0);
|
|
let _ = quality(&[&[r], &[]], BETA);
|
|
}
|
|
|
|
#[test]
|
|
fn history_predict_quality_supports_three_teams() {
|
|
use trueskill_tt::History;
|
|
|
|
let mut h = History::default();
|
|
h.record_winner(&"a", &"b", 1).unwrap();
|
|
h.record_winner(&"b", &"c", 2).unwrap();
|
|
let _ = h.converge().unwrap();
|
|
|
|
let q = h.predict_quality(&[&[&"a"], &[&"b"], &[&"c"]]).unwrap();
|
|
assert!(
|
|
q.is_finite(),
|
|
"3-team predict_quality must be finite, got {q}"
|
|
);
|
|
assert!((0.0..=1.0).contains(&q), "out of range: {q}");
|
|
}
|
|
|
|
/// `quality()` for N identical teams has a closed form, which pins the N-group
|
|
/// determinant path across the whole range rather than at a single golden.
|
|
///
|
|
/// For two identical single-player teams the standard result is
|
|
/// `sqrt(2b^2 / (2b^2 + s1^2 + s2^2))`. With the conventional parameters
|
|
/// (`sigma = 25/3`, `beta = 25/6`) that ratio is exactly `1/5`, and the N-group
|
|
/// generalisation is `(1/5)^((n-1)/2)` — one factor per adjacent pair.
|
|
///
|
|
/// The n=3 and n=5 values this produces (0.200 and 0.040) are also what the
|
|
/// `trueskill` Python package returns for the same configuration, so this
|
|
/// doubles as the cross-implementation check the README asked for.
|
|
#[test]
|
|
fn quality_of_identical_teams_follows_its_closed_form() {
|
|
let g = Gaussian::from_ms(25.0, 25.0 / 3.0);
|
|
let beta = 25.0 / 6.0;
|
|
|
|
for n in 2..=10usize {
|
|
let groups: Vec<Vec<Gaussian>> = (0..n).map(|_| vec![g]).collect();
|
|
let refs: Vec<&[Gaussian]> = groups.iter().map(Vec::as_slice).collect();
|
|
|
|
let got = quality(&refs, beta);
|
|
let expected = 0.2f64.powf((n - 1) as f64 / 2.0);
|
|
|
|
assert!(
|
|
(got - expected).abs() / expected < 1e-9,
|
|
"n={n}: quality {got}, closed form {expected}"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// Spot-check against the two values the `trueskill` Python package is known
|
|
/// to produce for this configuration, stated as literals so a future change to
|
|
/// the closed-form reasoning above cannot quietly take these with it.
|
|
#[test]
|
|
fn quality_matches_the_reference_implementation() {
|
|
let g = Gaussian::from_ms(25.0, 25.0 / 3.0);
|
|
let beta = 25.0 / 6.0;
|
|
|
|
let three: Vec<Vec<Gaussian>> = (0..3).map(|_| vec![g]).collect();
|
|
let refs: Vec<&[Gaussian]> = three.iter().map(Vec::as_slice).collect();
|
|
assert!((quality(&refs, beta) - 0.200).abs() < 1e-9);
|
|
|
|
let five: Vec<Vec<Gaussian>> = (0..5).map(|_| vec![g]).collect();
|
|
let refs: Vec<&[Gaussian]> = five.iter().map(Vec::as_slice).collect();
|
|
assert!((quality(&refs, beta) - 0.040).abs() < 1e-9);
|
|
}
|