feat: add UnknownKeys::Prior, and explain why there is no Skip
#44's third ask was an opt-in mode so a caller with partially-known teams need not pre-filter. The requested shape was `Skip` — drop unknown members. Measured, that is the wrong mode to build. A team's performance is the *sum* of its members, so dropping one drops its variance too. On a two-member team with one unknown: SKIP (drop the member) : performance sigma 2.37 PRIOR (member at prior) : performance sigma 6.53 (2.76x wider) Skipping makes the model *more* certain because it knows *less*, which is backwards. `Prior` is also the answer the model already gives for a competitor it knows about but has no evidence for — measured, such a competitor sits at sigma 4.99 against the prior's 6.0 — so it corresponds to a state the model can actually be in. Skipping does not. So the enum is `Reject` (default, unchanged) and `Prior`, and it is `#[non_exhaustive]` in case a real use for skipping turns up later. Placed on `HistoryBuilder` rather than per-call. Neither consumer wants it to vary between queries: one scores thousands of candidate matchups in a loop, the other's headline feature is predicting a competitor nobody has faced. That makes it a property of how the model is being used, and keeps five prediction signatures unchanged. This also gives #48 the semantics it asked for — "I have never seen this competitor, here is the prior-informed answer" — which it needs for predicting a course nobody has played. `an_unknown_member_widens_its_team_rather_than_narrowing_it` pins the property that ruled `Skip` out, so a future convenience cannot quietly reintroduce it. Closes #44. Refs #48 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
@@ -337,3 +337,81 @@ fn unknown_key_names_the_key_it_could_not_find() {
|
||||
"Display should say what to do about it: {rendered}"
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// UnknownKeys policy
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
fn history_with_policy(names: &[&'static str], policy: trueskill_tt::UnknownKeys) -> History {
|
||||
let mut h = History::builder().unknown_keys(policy).build();
|
||||
for pair in names.windows(2) {
|
||||
h.record_winner(&pair[0], &pair[1], 1).unwrap();
|
||||
}
|
||||
let _ = h.converge().unwrap();
|
||||
h
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reject_is_the_default() {
|
||||
let h = history_with(&["a", "b"], 0.0);
|
||||
assert!(matches!(
|
||||
h.predict_outcome(&[&[&"a"], &[&"ghost"]]),
|
||||
Err(InferenceError::UnknownKey { .. })
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prior_answers_instead_of_erroring() {
|
||||
let h = history_with_policy(&["a", "b"], trueskill_tt::UnknownKeys::Prior);
|
||||
let p = h
|
||||
.predict_outcome(&[&[&"a"], &[&"ghost"]])
|
||||
.expect("Prior should answer rather than reject");
|
||||
assert!((p.total() - 1.0).abs() < 1e-6);
|
||||
}
|
||||
|
||||
/// Two competitors the model has never seen are genuinely a coin flip. The
|
||||
/// point is that this is now *derived* rather than a constant a caller
|
||||
/// substitutes after swallowing an error.
|
||||
#[test]
|
||||
fn two_unknown_competitors_are_an_honest_coin_flip() {
|
||||
let h = history_with_policy(&["a", "b"], trueskill_tt::UnknownKeys::Prior);
|
||||
let wins = h
|
||||
.predict_win_probabilities(&[&[&"nobody"], &[&"no_one"]])
|
||||
.unwrap();
|
||||
assert!((wins[0] - 0.5).abs() < 1e-9, "{wins:?}");
|
||||
assert!((wins[1] - 0.5).abs() < 1e-9, "{wins:?}");
|
||||
}
|
||||
|
||||
/// The property that rules out a `Skip` mode: an unknown member must make a
|
||||
/// team *less* certain, never more. Skipping would drop the member's variance
|
||||
/// from the sum and narrow the team, which is backwards.
|
||||
#[test]
|
||||
fn an_unknown_member_widens_its_team_rather_than_narrowing_it() {
|
||||
let h = history_with_policy(&["a", "b", "c"], trueskill_tt::UnknownKeys::Prior);
|
||||
|
||||
// "a" alone against "b" — then "a" plus an unknown partner against "b".
|
||||
let solo = h.predict_win_probabilities(&[&[&"a"], &[&"b"]]).unwrap();
|
||||
let with_unknown = h
|
||||
.predict_win_probabilities(&[&[&"a", &"stranger"], &[&"b"]])
|
||||
.unwrap();
|
||||
|
||||
// Adding an unknown partner pulls the outcome toward even, because the
|
||||
// team's performance spread grew.
|
||||
assert!(
|
||||
(with_unknown[0] - 0.5).abs() < (solo[0] - 0.5).abs(),
|
||||
"an unknown partner should make the result less certain: solo {solo:?}, \
|
||||
with unknown {with_unknown:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prior_reaches_every_prediction_entry_point() {
|
||||
let h = history_with_policy(&["a", "b"], trueskill_tt::UnknownKeys::Prior);
|
||||
let teams: &[&[&&str]] = &[&[&"a"], &[&"ghost"]];
|
||||
|
||||
assert!(h.predict_quality(teams).is_ok());
|
||||
assert!(h.predict_win_probabilities(teams).is_ok());
|
||||
assert!(h.predict_outcome(teams).is_ok());
|
||||
assert!(h.predict_ranking(teams, &[0, 1]).is_ok());
|
||||
assert!(h.expected_information_gain(teams).is_ok());
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user