Three issues from two downstream consumers, all small, all sharing a theme: the crate had the information and would not hand it over. #44 — `UnknownKey { team: 0, member: 0 }` did not say which key. A consumer upgrading 0.1.2 -> 0.4.1 had every one of 5591 predictions return this error, fell back to a neutral 0.5, and lost its entire metadata model for a day. Nothing crashed and nothing logged; it was found by sweeping an unrelated parameter and noticing the output did not move. The 0.4.0 change that made unknown keys an error was right — the error was just too anonymous to act on. It now carries the key's `Debug` rendering, and its `Display` says what to do about it. The precondition is documented on every prediction entry point, which the reporter said would alone have saved the day. #43 — `cdf` was `pub(crate)`, so a consumer asking "is this competitor below the cutoff" approximated it with a `mu + z * sigma` band and had no way to say what confidence any `z` bought. Adds `Gaussian::probability_below` / `probability_above`. The second is separate on purpose: `1 - cdf` collapses to exactly zero past ~8.3 sigma, and a stopping rule is evaluated precisely there. Both route through the survival function added in 0.4.1, so this is visibility rather than new numerics. #50 — `ConvergenceReport` was not `#[must_use]`, so the one signal that a fit stopped short was trivially discarded. It now is, and that immediately found 78 sites doing exactly that — including this crate's own ATP example, which was capped at 10 sweeps when the history needs 30. The example now reads the report and says so. `ITERATIONS = 30` is documented as the floor it is, with the three measurements to hand: 400 events over 100 competitors already stops there at ~7e-3 against a 1e-6 tolerance, the ATP example needs 30 at a much looser one, and a consumer's 2000-node model needs 76 to 161. BREAKING CHANGE: `InferenceError::UnknownKey` gains a `key` field, and the prediction methods now require `K: Debug` in order to fill it. Closes #43, #50. Refs #44 — its third ask, an opt-in `UnknownKeys::Skip` mode, is a live API question and deliberately not answered here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
424 lines
12 KiB
Rust
424 lines
12 KiB
Rust
//! Degenerate, boundary, and error-path coverage.
|
|
//!
|
|
//! These run in both debug and release: the defects they pin were all
|
|
//! guarded only by `debug_assert!`, so a debug-only suite never saw them.
|
|
|
|
mod common;
|
|
|
|
use common::assert_finite;
|
|
use trueskill_tt::{
|
|
ConstantDrift, ConvergenceOptions, Game, GameOptions, Gaussian, History, InferenceError,
|
|
NullObserver, Outcome, Rating,
|
|
};
|
|
|
|
type R = Rating<i64, ConstantDrift>;
|
|
|
|
fn rating() -> R {
|
|
R::new(
|
|
Gaussian::from_ms(25.0, 25.0 / 3.0),
|
|
25.0 / 6.0,
|
|
ConstantDrift(25.0 / 300.0),
|
|
)
|
|
}
|
|
|
|
#[test]
|
|
fn record_draw_without_draw_probability_is_rejected() {
|
|
let mut h = History::default();
|
|
let err = h.record_draw(&"a", &"b", 1).unwrap_err();
|
|
assert!(matches!(
|
|
err,
|
|
InferenceError::TieWithoutDrawProbability { .. }
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn builder_draw_without_draw_probability_is_rejected() {
|
|
let mut h = History::default();
|
|
let err = h
|
|
.event(1)
|
|
.team(["a"])
|
|
.team(["b"])
|
|
.draw()
|
|
.commit()
|
|
.unwrap_err();
|
|
assert!(matches!(
|
|
err,
|
|
InferenceError::TieWithoutDrawProbability { .. }
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn draw_with_positive_draw_probability_is_finite() {
|
|
let mut h = History::builder().p_draw(0.25).build();
|
|
h.record_draw(&"a", &"b", 1).unwrap();
|
|
let report = h.converge().unwrap();
|
|
|
|
assert_finite(h.current_skill("a").unwrap(), "drawn competitor skill");
|
|
assert_finite(h.current_skill("b").unwrap(), "drawn competitor skill");
|
|
assert!(report.log_evidence.is_finite());
|
|
assert!(report.converged);
|
|
}
|
|
|
|
#[test]
|
|
fn game_ranked_rejects_tie_without_draw_probability() {
|
|
let a = [rating()];
|
|
let b = [rating()];
|
|
let teams: Vec<&[R]> = vec![&a, &b];
|
|
let err = Game::ranked(&teams, Outcome::draw(2), &GameOptions::default()).unwrap_err();
|
|
assert!(matches!(
|
|
err,
|
|
InferenceError::TieWithoutDrawProbability { .. }
|
|
));
|
|
}
|
|
|
|
/// `Outcome::winner(w, n)` ties every loser, so any n >= 3 free-for-all hits
|
|
/// the tie path even though the caller never asked for a draw.
|
|
#[test]
|
|
fn winner_of_three_or_more_requires_draw_probability() {
|
|
let a = [rating()];
|
|
let b = [rating()];
|
|
let c = [rating()];
|
|
let teams: Vec<&[R]> = vec![&a, &b, &c];
|
|
|
|
let err = Game::ranked(&teams, Outcome::winner(0, 3), &GameOptions::default()).unwrap_err();
|
|
assert!(matches!(
|
|
err,
|
|
InferenceError::TieWithoutDrawProbability { .. }
|
|
));
|
|
|
|
let opts = GameOptions {
|
|
p_draw: 0.1,
|
|
..GameOptions::default()
|
|
};
|
|
let game = Game::ranked(&teams, Outcome::winner(0, 3), &opts).unwrap();
|
|
for team in game.posteriors() {
|
|
for skill in team {
|
|
assert_finite(skill, "3-team winner posterior");
|
|
}
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn full_ranking_without_ties_needs_no_draw_probability() {
|
|
let a = [rating()];
|
|
let b = [rating()];
|
|
let c = [rating()];
|
|
let teams: Vec<&[R]> = vec![&a, &b, &c];
|
|
let game = Game::ranked(&teams, Outcome::ranking([0, 1, 2]), &GameOptions::default()).unwrap();
|
|
|
|
for team in game.posteriors() {
|
|
for skill in team {
|
|
assert_finite(skill, "strict ranking posterior");
|
|
}
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn empty_history_converges_trivially() {
|
|
let mut h = History::default();
|
|
let report = h.converge().unwrap();
|
|
assert_eq!(report.iterations, 0);
|
|
assert!(report.converged);
|
|
}
|
|
|
|
/// Issue #27's exact reproduction: a non-default key type reaching `converge`
|
|
/// with no events at all. The underflow it reported trapped in debug and
|
|
/// indexed out of bounds in release, so this must run in both profiles.
|
|
#[test]
|
|
fn converge_on_an_empty_history_with_owned_keys() {
|
|
let mut history: History<i64, ConstantDrift, NullObserver, String> =
|
|
History::builder_with_key().score_sigma(5.0).build();
|
|
|
|
let report = history.converge().unwrap();
|
|
|
|
assert_eq!(report.iterations, 0);
|
|
assert!(report.converged);
|
|
}
|
|
|
|
/// A weights/team length mismatch used to be a `debug_assert!`, so release
|
|
/// builds ingested the event with the weights silently unapplied. This file's
|
|
/// CI job runs in release too, which is the point of pinning it here.
|
|
#[test]
|
|
fn event_builder_rejects_a_weights_length_mismatch() {
|
|
let mut h = History::default();
|
|
|
|
let err = h
|
|
.event(1)
|
|
.team(["a"])
|
|
.weights([1.0, 2.0])
|
|
.team(["b"])
|
|
.winner(0)
|
|
.commit()
|
|
.unwrap_err();
|
|
|
|
assert!(
|
|
matches!(
|
|
err,
|
|
InferenceError::MismatchedShape {
|
|
kind: "weights",
|
|
expected: 1,
|
|
got: 2,
|
|
}
|
|
),
|
|
"expected a weights MismatchedShape, got {err:?}"
|
|
);
|
|
}
|
|
|
|
/// The mismatch must not be applied even partially — a half-weighted team
|
|
/// reaching the history would be worse than the error.
|
|
#[test]
|
|
fn event_builder_weights_mismatch_leaves_the_history_untouched() {
|
|
let mut h = History::default();
|
|
|
|
// Two teams, so ingestion would otherwise succeed — a one-team event is
|
|
// rejected for an unrelated reason and would pass this vacuously.
|
|
let _ = h
|
|
.event(1)
|
|
.team(["a"])
|
|
.weights([1.0, 2.0])
|
|
.team(["b"])
|
|
.winner(0)
|
|
.commit();
|
|
|
|
assert!(h.learning_curve("a").is_empty());
|
|
}
|
|
|
|
#[test]
|
|
fn empty_event_stream_then_converge() {
|
|
let mut h = History::default();
|
|
h.add_events(std::iter::empty()).unwrap();
|
|
let report = h.converge().unwrap();
|
|
assert_eq!(report.iterations, 0);
|
|
}
|
|
|
|
#[test]
|
|
fn empty_history_queries_do_not_panic() {
|
|
let h = History::default();
|
|
assert!(h.learning_curves().is_empty());
|
|
assert!(h.learning_curve("nobody").is_empty());
|
|
assert!(h.current_skill("nobody").is_none());
|
|
}
|
|
|
|
#[test]
|
|
fn single_event_history_converges() {
|
|
let mut h = History::default();
|
|
h.record_winner(&"a", &"b", 1).unwrap();
|
|
let report = h.converge().unwrap();
|
|
assert!(report.converged);
|
|
assert_finite(h.current_skill("a").unwrap(), "single-event skill");
|
|
}
|
|
|
|
#[test]
|
|
fn scored_event_rejects_non_positive_sigma() {
|
|
let mut h = History::builder().score_sigma(2.0).build();
|
|
let err = h
|
|
.event(1)
|
|
.team(["a"])
|
|
.team(["b"])
|
|
.scores_with_sigma([3.0, 1.0], f64::NAN)
|
|
.commit()
|
|
.unwrap_err();
|
|
assert!(matches!(
|
|
err,
|
|
InferenceError::InvalidParameter {
|
|
name: "score_sigma",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn convergence_reports_are_finite_across_many_teams() {
|
|
let opts = GameOptions {
|
|
p_draw: 0.1,
|
|
convergence: ConvergenceOptions::default(),
|
|
..GameOptions::default()
|
|
};
|
|
let holders: Vec<[R; 1]> = (0..12).map(|_| [rating()]).collect();
|
|
let teams: Vec<&[R]> = holders.iter().map(|t| t.as_slice()).collect();
|
|
let game = Game::ranked(&teams, Outcome::ranking(0..12), &opts).unwrap();
|
|
|
|
assert!(
|
|
game.log_evidence().is_finite(),
|
|
"12-team log-evidence must be finite, got {}",
|
|
game.log_evidence()
|
|
);
|
|
for team in game.posteriors() {
|
|
for skill in team {
|
|
assert_finite(skill, "12-team posterior");
|
|
}
|
|
}
|
|
}
|
|
|
|
/// A long diff chain underflows a linear evidence product: each link
|
|
/// contributes a probability in (0, 1], so ~1000 links flush the product to
|
|
/// exactly 0.0 and `ln(0.0)` is `-inf`. Accumulating in log space keeps it
|
|
/// finite.
|
|
#[test]
|
|
fn log_evidence_survives_a_long_diff_chain() {
|
|
let holders: Vec<[R; 1]> = (0..1200).map(|_| [rating()]).collect();
|
|
let teams: Vec<&[R]> = holders.iter().map(|t| t.as_slice()).collect();
|
|
let game = Game::ranked(
|
|
&teams,
|
|
Outcome::ranking(0..holders.len() as u32),
|
|
&GameOptions::default(),
|
|
)
|
|
.unwrap();
|
|
|
|
let log_evidence = game.log_evidence();
|
|
assert!(
|
|
log_evidence.is_finite(),
|
|
"1200-team log-evidence must be finite, got {log_evidence}"
|
|
);
|
|
assert!(
|
|
log_evidence < 0.0,
|
|
"log-evidence of a probability must be negative, got {log_evidence}"
|
|
);
|
|
}
|
|
|
|
/// A near-certain outcome rounds the losing tail to exactly zero in the
|
|
/// `erfc` approximation; the evidence floor keeps `ln` finite.
|
|
#[test]
|
|
fn log_evidence_finite_for_near_certain_outcome() {
|
|
let overwhelming = R::new(Gaussian::from_ms(5_000.0, 0.5), 1.0, ConstantDrift(0.0));
|
|
let hopeless = R::new(Gaussian::from_ms(-5_000.0, 0.5), 1.0, ConstantDrift(0.0));
|
|
let a = [overwhelming];
|
|
let b = [hopeless];
|
|
let teams: Vec<&[R]> = vec![&a, &b];
|
|
|
|
let game = Game::ranked(&teams, Outcome::winner(0, 2), &GameOptions::default()).unwrap();
|
|
assert!(
|
|
game.log_evidence().is_finite(),
|
|
"got {}",
|
|
game.log_evidence()
|
|
);
|
|
|
|
// And the reverse — a colossal upset — must also stay finite.
|
|
let upset = Game::ranked(&teams, Outcome::winner(1, 2), &GameOptions::default()).unwrap();
|
|
assert!(
|
|
upset.log_evidence().is_finite(),
|
|
"upset log-evidence must be finite, got {}",
|
|
upset.log_evidence()
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn empty_history_has_no_filtered_estimates() {
|
|
let history: History = History::builder().build();
|
|
|
|
assert_eq!(history.filtered_log_evidence(), 0.0);
|
|
|
|
assert!(history.filtered_learning_curves().is_empty());
|
|
|
|
assert!(history.filtered_learning_curve("nobody").is_empty());
|
|
}
|
|
|
|
// --- Boundary inputs (#26) ----------------------------------------------
|
|
|
|
fn tight() -> ConvergenceOptions {
|
|
ConvergenceOptions {
|
|
max_iter: 2_000,
|
|
epsilon: 1e-12,
|
|
..ConvergenceOptions::default()
|
|
}
|
|
}
|
|
|
|
fn assert_curve_finite(h: &History, keys: &[&str], what: &str) {
|
|
for key in keys {
|
|
for (time, g) in h.learning_curve(*key) {
|
|
assert!(
|
|
g.mu().is_finite() && g.sigma().is_finite(),
|
|
"{what}: non-finite posterior for {key} at t={time} (mu={} sigma={})",
|
|
g.mu(),
|
|
g.sigma()
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
/// A zero weight reaches `(m - performance.exclude(..)) * (1.0 / w)`, i.e. a
|
|
/// division by zero. The commit is accepted today, so this pins that the
|
|
/// resulting posterior is still finite rather than quietly NaN.
|
|
#[test]
|
|
fn zero_weight_does_not_produce_a_non_finite_posterior() {
|
|
let mut h = History::builder().build();
|
|
|
|
h.event(1)
|
|
.team(["a"])
|
|
.weights([0.0])
|
|
.team(["b"])
|
|
.winner(0)
|
|
.commit()
|
|
.expect("a zero weight is accepted today; update this test if that changes");
|
|
|
|
let _ = h.converge().unwrap();
|
|
|
|
assert_curve_finite(&h, &["a", "b"], "zero weight");
|
|
}
|
|
|
|
#[test]
|
|
fn negative_weight_does_not_produce_a_non_finite_posterior() {
|
|
let mut h = History::builder().build();
|
|
|
|
h.event(1)
|
|
.team(["a"])
|
|
.weights([-1.0])
|
|
.team(["b"])
|
|
.winner(0)
|
|
.commit()
|
|
.expect("a negative weight is accepted today; update this test if that changes");
|
|
|
|
let _ = h.converge().unwrap();
|
|
|
|
assert_curve_finite(&h, &["a", "b"], "negative weight");
|
|
}
|
|
|
|
/// Events supplied newest-first must land in the same slices as oldest-first:
|
|
/// ingestion sorts by time rather than trusting arrival order.
|
|
#[test]
|
|
fn out_of_order_timestamps_converge_to_the_same_answer() {
|
|
fn build(descending: bool) -> History {
|
|
let mut h = History::builder().convergence(tight()).build();
|
|
|
|
let mut times: Vec<i64> = (1..=6).collect();
|
|
if descending {
|
|
times.reverse();
|
|
}
|
|
|
|
for time in times {
|
|
h.record_winner(&"a", &"b", time).unwrap();
|
|
}
|
|
|
|
let _ = h.converge().unwrap();
|
|
h
|
|
}
|
|
|
|
let ascending = build(false);
|
|
let descending = build(true);
|
|
|
|
let one = ascending.current_skill("a").unwrap();
|
|
let other = descending.current_skill("a").unwrap();
|
|
|
|
assert!(
|
|
(one.mu() - other.mu()).abs() < 1e-8 && (one.sigma() - other.sigma()).abs() < 1e-8,
|
|
"arrival order changed the answer: ascending mu={} sigma={}, descending mu={} sigma={}",
|
|
one.mu(),
|
|
one.sigma(),
|
|
other.mu(),
|
|
other.sigma()
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn extreme_beta_and_sigma_stay_finite() {
|
|
for (beta, sigma) in [(1e-6, 1e-6), (1e6, 1e6), (1e-6, 1e6), (1e6, 1e-6)] {
|
|
let mut h = History::builder().beta(beta).sigma(sigma).build();
|
|
|
|
h.record_winner(&"a", &"b", 1).unwrap();
|
|
h.record_winner(&"a", &"b", 2).unwrap();
|
|
let _ = h.converge().unwrap();
|
|
|
|
assert_curve_finite(&h, &["a", "b"], &format!("beta={beta} sigma={sigma}"));
|
|
}
|
|
}
|