fix!: no prediction path answers from a fit it cannot answer from
`converge` refuses to report a NaN fit. Nothing stopped a caller from
ignoring that error and predicting anyway, and every prediction path was
differently wrong when they did. Measured on a point-mass-prior history
with `beta(0.0)`, after `converge` returned `NonFiniteResult`:
predict_quality = Ok(NaN)
predict_outcome().total() = NaN
predict_win_probabilities = Ok([0.0, 0.0])
The third is the dangerous one: finite, plausible, and summing to zero
against a doc that promises one at `p_draw == 0`. A caller checking
`total() ≈ 1` catches the second and misses it.
The same parameters on a *scored* event converge cleanly and leave
legitimate point-mass posteriors. There `predict_quality` **panicked** —
"cannot invert a singular matrix", out of a method returning `Result` —
because the contrast covariance `beta²AᵀA + AᵀSA` is exactly singular,
and `predict_win_probabilities` again returned `Ok([0.0, 0.0])`. That
promise assumes continuous performances, where an exact tie has measure
zero; point masses break the assumption, not the arithmetic.
Both checks now live at `member_skills`, the one gate every prediction
path reads skills through, rather than being repeated per method.
The finiteness check is on `mu` / `sigma`, not on the natural parameters.
The first attempt checked `pi` and `tau`, and measurement showed it
rejected a *legitimate* point mass — `pi = inf`, `mu = 0`, `sigma = 0` —
turning a working prediction into an error. The question is whether the
usable moments exist, and those are what predictions consume.
Docs: `converge_partial` omitted the drift-variance `InvalidParameter` it
validates before sweeping, and the free `expected_information_gain`
omitted `GridTooCoarse`, which comes from `outcome_distribution` and so
is not covered by its "anything `Game::ranked` returns" clause.
Refs #78 (parts 1 and 2; the layering and `predict_quality` rename
questions are still open).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
@@ -0,0 +1,152 @@
|
||||
//! No prediction path may answer from a fit it cannot answer from.
|
||||
//!
|
||||
//! `converge` grew a `NonFiniteResult` guard; nothing stopped a caller from
|
||||
//! ignoring that error and predicting anyway. The three failures that produced
|
||||
//! were each differently wrong: `Ok(NaN)`, a panic out of a `Result`-returning
|
||||
//! method, and `Ok([0.0, 0.0])` — finite, plausible, summing to zero against a
|
||||
//! doc that promises one.
|
||||
//!
|
||||
//! Every test here has a healthy control, so none can pass by everything
|
||||
//! returning `Err`.
|
||||
|
||||
use trueskill_tt::{
|
||||
ConstantDrift, Event, Gaussian, History, InferenceError, Member, NullObserver, Outcome, Team,
|
||||
};
|
||||
|
||||
type H = History<i64, ConstantDrift, NullObserver, &'static str>;
|
||||
|
||||
fn build(beta: f64, prior: Option<Gaussian>, outcome: Outcome) -> H {
|
||||
let mut h: H = History::builder()
|
||||
.beta(beta)
|
||||
.drift(ConstantDrift::new(0.0))
|
||||
.build();
|
||||
|
||||
let member = |k: &'static str| match prior {
|
||||
Some(p) => Member::new(k).with_prior(p),
|
||||
None => Member::new(k),
|
||||
};
|
||||
|
||||
let _ = h.add_events(vec![Event {
|
||||
time: 1,
|
||||
teams: [
|
||||
Team::with_members([member("a")]),
|
||||
Team::with_members([member("b")]),
|
||||
]
|
||||
.into_iter()
|
||||
.collect(),
|
||||
outcome,
|
||||
}]);
|
||||
h
|
||||
}
|
||||
|
||||
/// Point-mass priors with `beta(0.0)` on a *ranked* event: `converge` reports
|
||||
/// `NonFiniteResult` and the stored posteriors are `pi: NaN, tau: NaN`.
|
||||
fn nan_poisoned() -> H {
|
||||
let mut h = build(
|
||||
0.0,
|
||||
Some(Gaussian::from_ms(0.0, 0.0)),
|
||||
Outcome::winner(0, 2),
|
||||
);
|
||||
let err = h.converge().expect_err("this fixture must not converge");
|
||||
assert!(
|
||||
matches!(err, InferenceError::NonFiniteResult { .. }),
|
||||
"{err:?}"
|
||||
);
|
||||
h
|
||||
}
|
||||
|
||||
/// The same degenerate parameters on a *scored* event, where inference
|
||||
/// converges cleanly and leaves legitimate point-mass posteriors behind. The
|
||||
/// fit is fine; it is prediction that has nothing to work with.
|
||||
fn degenerate_but_converged() -> H {
|
||||
let mut h = build(
|
||||
0.0,
|
||||
Some(Gaussian::from_ms(0.0, 0.0)),
|
||||
Outcome::scores([1.0, 0.0]),
|
||||
);
|
||||
h.converge().expect("this fixture converges");
|
||||
h
|
||||
}
|
||||
|
||||
fn healthy() -> H {
|
||||
let mut h = build(1.0, None, Outcome::winner(0, 2));
|
||||
h.converge().expect("control converges");
|
||||
h
|
||||
}
|
||||
|
||||
macro_rules! all_predictions {
|
||||
($h:ident, $f:expr) => {{
|
||||
let teams: &[&[&&'static str]] = &[&[&"a"], &[&"b"]];
|
||||
let f = $f;
|
||||
f("predict_quality", $h.predict_quality(teams).map(|_| ()));
|
||||
f(
|
||||
"predict_win_probabilities",
|
||||
$h.predict_win_probabilities(teams).map(|_| ()),
|
||||
);
|
||||
f("predict_outcome", $h.predict_outcome(teams).map(|_| ()));
|
||||
f(
|
||||
"predict_ranking",
|
||||
$h.predict_ranking(teams, &[0, 1]).map(|_| ()),
|
||||
);
|
||||
f(
|
||||
"expected_information_gain",
|
||||
$h.expected_information_gain(teams).map(|_| ()),
|
||||
);
|
||||
}};
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_nan_poisoned_fit_is_refused_by_every_prediction_path() {
|
||||
let h = nan_poisoned();
|
||||
|
||||
all_predictions!(h, |name: &str, r: Result<(), InferenceError>| {
|
||||
match r {
|
||||
Err(InferenceError::NonFiniteResult { .. }) => {}
|
||||
other => panic!("{name} answered from a NaN fit: {other:?}"),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn degenerate_performances_are_refused_rather_than_answered_wrongly() {
|
||||
let h = degenerate_but_converged();
|
||||
|
||||
// The fit itself is sound — the posteriors are point masses, not NaN.
|
||||
let skill = h.current_skill("a").expect("a played");
|
||||
assert_eq!(skill.sigma(), 0.0);
|
||||
assert!(skill.mu().is_finite());
|
||||
|
||||
// `predict_quality` previously PANICKED here, out of a method that returns
|
||||
// `Result`: the contrast covariance is exactly singular when beta is zero
|
||||
// and every skill is a point mass.
|
||||
all_predictions!(h, |name: &str, r: Result<(), InferenceError>| {
|
||||
match r {
|
||||
Err(InferenceError::InvalidParameter { .. }) => {}
|
||||
other => panic!("{name} predicted from a degenerate fit: {other:?}"),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_control_history_answers_every_prediction() {
|
||||
let h = healthy();
|
||||
|
||||
all_predictions!(h, |name: &str, r: Result<(), InferenceError>| {
|
||||
assert!(r.is_ok(), "{name} failed on a healthy history: {r:?}");
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn win_probabilities_sum_to_one_on_the_control() {
|
||||
// The promise the `Ok([0.0, 0.0])` case broke. Asserted on the control so
|
||||
// the guard above cannot be "fixed" by making every path error.
|
||||
let h = healthy();
|
||||
let p = h
|
||||
.predict_win_probabilities(&[&[&"a"], &[&"b"]])
|
||||
.expect("control predicts");
|
||||
let total: f64 = p.iter().sum();
|
||||
assert!(
|
||||
(total - 1.0).abs() < 1e-6,
|
||||
"win probabilities sum to {total}"
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user