Three issues from two downstream consumers, all small, all sharing a theme: the crate had the information and would not hand it over. #44 — `UnknownKey { team: 0, member: 0 }` did not say which key. A consumer upgrading 0.1.2 -> 0.4.1 had every one of 5591 predictions return this error, fell back to a neutral 0.5, and lost its entire metadata model for a day. Nothing crashed and nothing logged; it was found by sweeping an unrelated parameter and noticing the output did not move. The 0.4.0 change that made unknown keys an error was right — the error was just too anonymous to act on. It now carries the key's `Debug` rendering, and its `Display` says what to do about it. The precondition is documented on every prediction entry point, which the reporter said would alone have saved the day. #43 — `cdf` was `pub(crate)`, so a consumer asking "is this competitor below the cutoff" approximated it with a `mu + z * sigma` band and had no way to say what confidence any `z` bought. Adds `Gaussian::probability_below` / `probability_above`. The second is separate on purpose: `1 - cdf` collapses to exactly zero past ~8.3 sigma, and a stopping rule is evaluated precisely there. Both route through the survival function added in 0.4.1, so this is visibility rather than new numerics. #50 — `ConvergenceReport` was not `#[must_use]`, so the one signal that a fit stopped short was trivially discarded. It now is, and that immediately found 78 sites doing exactly that — including this crate's own ATP example, which was capped at 10 sweeps when the history needs 30. The example now reads the report and says so. `ITERATIONS = 30` is documented as the floor it is, with the three measurements to hand: 400 events over 100 competitors already stops there at ~7e-3 against a 1e-6 tolerance, the ATP example needs 30 at a much looser one, and a consumer's 2000-node model needs 76 to 161. BREAKING CHANGE: `InferenceError::UnknownKey` gains a `key` field, and the prediction methods now require `K: Debug` in order to fill it. Closes #43, #50. Refs #44 — its third ask, an opt-in `UnknownKeys::Skip` mode, is a live API question and deliberately not answered here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
223 lines
6.8 KiB
Rust
223 lines
6.8 KiB
Rust
//! `Member::with_prior` / `with_drift_scale` — competitor configuration.
|
|
//!
|
|
//! Both were previously consumed only on the branch that *creates* a
|
|
//! competitor, so configuration supplied for a key the history already knew was
|
|
//! dropped with no error. `with_prior` had no coverage in this directory at
|
|
//! all, which is how that survived.
|
|
|
|
use smallvec::smallvec;
|
|
use trueskill_tt::{
|
|
ConvergenceOptions, Event, Gaussian, History, InferenceError, Member, Outcome, Team,
|
|
};
|
|
|
|
const CONVERGENCE: ConvergenceOptions = ConvergenceOptions {
|
|
max_iter: 2_000,
|
|
epsilon: 1e-12,
|
|
alpha: 1.0,
|
|
};
|
|
|
|
fn history() -> History {
|
|
History::builder()
|
|
.mu(25.0)
|
|
.sigma(25.0 / 3.0)
|
|
.beta(25.0 / 6.0)
|
|
.p_draw(0.0)
|
|
.convergence(CONVERGENCE)
|
|
.build()
|
|
}
|
|
|
|
/// One event, optionally configuring `a`.
|
|
fn bout(
|
|
a: &'static str,
|
|
b: &'static str,
|
|
time: i64,
|
|
prior: Option<Gaussian>,
|
|
scale: Option<f64>,
|
|
) -> Event<i64, &'static str> {
|
|
let mut member = Member::new(a);
|
|
if let Some(p) = prior {
|
|
member = member.with_prior(p);
|
|
}
|
|
if let Some(s) = scale {
|
|
member = member.with_drift_scale(s);
|
|
}
|
|
|
|
Event {
|
|
time,
|
|
teams: smallvec![
|
|
Team::with_members([member]),
|
|
Team::with_members([Member::new(b)]),
|
|
],
|
|
outcome: Outcome::winner(0, 2),
|
|
}
|
|
}
|
|
|
|
fn skill_of(h: &History, key: &str) -> Gaussian {
|
|
h.current_skill(&key).expect("key in history")
|
|
}
|
|
|
|
/// Baseline: the mechanism works at all on a competitor's first appearance.
|
|
#[test]
|
|
fn a_prior_applies_to_a_new_competitor() {
|
|
let seeded = Gaussian::from_ms(40.0, 1.0);
|
|
|
|
let mut with = history();
|
|
with.add_events(vec![bout("a", "b", 0, Some(seeded), None)])
|
|
.unwrap();
|
|
let _ = with.converge().unwrap();
|
|
|
|
let mut without = history();
|
|
without
|
|
.add_events(vec![bout("a", "b", 0, None, None)])
|
|
.unwrap();
|
|
let _ = without.converge().unwrap();
|
|
|
|
assert!(
|
|
(skill_of(&with, "a").mu() - skill_of(&without, "a").mu()).abs() > 1.0,
|
|
"a seeded prior should move the fit"
|
|
);
|
|
}
|
|
|
|
/// The defect in #10: a prior supplied for a competitor the history already
|
|
/// knows was silently discarded, and the caller got output computed from the
|
|
/// default prior with no indication anything had been dropped.
|
|
#[test]
|
|
fn a_prior_applies_to_a_competitor_the_history_already_knows() {
|
|
let seeded = Gaussian::from_ms(40.0, 1.0);
|
|
|
|
let mut late = history();
|
|
late.add_events(vec![bout("a", "b", 0, None, None)])
|
|
.unwrap();
|
|
// "a" now exists. Configuring it here used to do nothing whatsoever.
|
|
late.add_events(vec![bout("a", "b", 1, Some(seeded), None)])
|
|
.unwrap();
|
|
let _ = late.converge().unwrap();
|
|
|
|
let mut never = history();
|
|
never
|
|
.add_events(vec![
|
|
bout("a", "b", 0, None, None),
|
|
bout("a", "b", 1, None, None),
|
|
])
|
|
.unwrap();
|
|
let _ = never.converge().unwrap();
|
|
|
|
assert!(
|
|
(skill_of(&late, "a").mu() - skill_of(&never, "a").mu()).abs() > 1.0,
|
|
"a late prior must not be silently dropped: {} vs {}",
|
|
skill_of(&late, "a").mu(),
|
|
skill_of(&never, "a").mu()
|
|
);
|
|
}
|
|
|
|
/// Configuration is competitor-scoped, not event-scoped, and `converge` refits
|
|
/// from competitor state — so seeding late reaches the same fit as seeding from
|
|
/// the start. This is the documented scope, asserted rather than assumed.
|
|
#[test]
|
|
fn a_prior_is_whole_history_scoped_not_per_event() {
|
|
let seeded = Gaussian::from_ms(40.0, 1.0);
|
|
|
|
let mut late = history();
|
|
late.add_events(vec![bout("a", "b", 0, None, None)])
|
|
.unwrap();
|
|
late.add_events(vec![bout("a", "b", 1, Some(seeded), None)])
|
|
.unwrap();
|
|
let _ = late.converge().unwrap();
|
|
|
|
let mut early = history();
|
|
early
|
|
.add_events(vec![
|
|
bout("a", "b", 0, Some(seeded), None),
|
|
bout("a", "b", 1, Some(seeded), None),
|
|
])
|
|
.unwrap();
|
|
let _ = early.converge().unwrap();
|
|
|
|
let (l, e) = (skill_of(&late, "a"), skill_of(&early, "a"));
|
|
assert!(
|
|
(l.mu() - e.mu()).abs() < 1e-9 && (l.sigma() - e.sigma()).abs() < 1e-9,
|
|
"late seeding should refit the whole history: {l:?} vs {e:?}"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn repeating_the_same_prior_is_inert() {
|
|
let seeded = Gaussian::from_ms(40.0, 1.0);
|
|
|
|
let mut once = history();
|
|
once.add_events(vec![
|
|
bout("a", "b", 0, Some(seeded), None),
|
|
bout("a", "b", 1, None, None),
|
|
])
|
|
.unwrap();
|
|
let _ = once.converge().unwrap();
|
|
|
|
let mut every_time = history();
|
|
every_time
|
|
.add_events(vec![
|
|
bout("a", "b", 0, Some(seeded), None),
|
|
bout("a", "b", 1, Some(seeded), None),
|
|
])
|
|
.unwrap();
|
|
let _ = every_time.converge().unwrap();
|
|
|
|
let (o, e) = (skill_of(&once, "a"), skill_of(&every_time, "a"));
|
|
assert!(
|
|
(o.mu() - e.mu()).abs() < 1e-12 && (o.sigma() - e.sigma()).abs() < 1e-12,
|
|
"declaring the same prior repeatedly changed the fit: {o:?} vs {e:?}"
|
|
);
|
|
}
|
|
|
|
/// Events within a batch have no order, so two different values for one
|
|
/// competitor have no well-defined winner. Rejecting is what keeps the answer
|
|
/// independent of iteration order.
|
|
#[test]
|
|
fn a_batch_declaring_two_different_priors_is_rejected() {
|
|
let mut h = history();
|
|
let err = h
|
|
.add_events(vec![
|
|
bout("a", "b", 0, Some(Gaussian::from_ms(40.0, 1.0)), None),
|
|
bout("a", "b", 1, Some(Gaussian::from_ms(10.0, 1.0)), None),
|
|
])
|
|
.expect_err("two different priors for one competitor in one batch");
|
|
|
|
assert!(
|
|
matches!(
|
|
err,
|
|
InferenceError::ConflictingCompetitorConfig { field: "prior", .. }
|
|
),
|
|
"got {err:?}"
|
|
);
|
|
}
|
|
|
|
/// A member setting only `drift_scale` must not also assert the default prior,
|
|
/// or it would silently undo a prior seeded earlier. This is why the collected
|
|
/// configuration tracks each field separately rather than a merged `Rating`.
|
|
#[test]
|
|
fn setting_one_field_late_leaves_the_other_alone() {
|
|
let seeded = Gaussian::from_ms(40.0, 1.0);
|
|
|
|
let mut h = history();
|
|
h.add_events(vec![bout("a", "b", 0, Some(seeded), None)])
|
|
.unwrap();
|
|
// Only the scale this time — the prior above must survive.
|
|
h.add_events(vec![bout("a", "b", 1, None, Some(0.5))])
|
|
.unwrap();
|
|
let _ = h.converge().unwrap();
|
|
|
|
let mut both_upfront = history();
|
|
both_upfront
|
|
.add_events(vec![
|
|
bout("a", "b", 0, Some(seeded), Some(0.5)),
|
|
bout("a", "b", 1, None, None),
|
|
])
|
|
.unwrap();
|
|
let _ = both_upfront.converge().unwrap();
|
|
|
|
let (a, b) = (skill_of(&h, "a"), skill_of(&both_upfront, "a"));
|
|
assert!(
|
|
(a.mu() - b.mu()).abs() < 1e-9 && (a.sigma() - b.sigma()).abs() < 1e-9,
|
|
"setting drift_scale late clobbered the earlier prior: {a:?} vs {b:?}"
|
|
);
|
|
}
|