Two accessors answered a question about a key the history had never seen with a well-formed value indistinguishable from a real answer. `log_evidence_for` filter_map'd unknown keys away. An empty target list means "no restriction" downstream, so a list of *entirely* unknown keys returned the whole-history evidence: measured on a two-cohort fixture, `log_evidence_for(["typo"])` returned exactly `log_evidence()`. On the one workload it is documented for — leave-one-out cross-validation — that is the un-held-out score, a plausible number that silently invalidates the comparison it was computed for. It now returns `Err(UnknownKey)` naming the offending position. `learning_curve` and `filtered_learning_curve` returned an empty `Vec` both for a typo'd key and for a competitor who is registered but has not played yet. They now return `Option`, so `None` is "never heard of it" and `Some(vec![])` is "known, no appearances". Tests carry a control case in each direction, so they cannot pass by everything returning the same thing. Closes #66, closes #70. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
195 lines
5.9 KiB
Rust
195 lines
5.9 KiB
Rust
//! `EventBuilder::members` must reach exactly what the typed path reaches.
|
|
//!
|
|
//! Before this existed, `EventBuilder` could set weights and nothing else, so
|
|
//! `prior` and `drift_scale` were expressible only through `Event`/`Team`/
|
|
//! `Member` + `add_events`. Which ingestion route a competitor arrived through
|
|
//! decided whether it could be configured at all.
|
|
|
|
use smallvec::smallvec;
|
|
use trueskill_tt::{
|
|
ConstantDrift, ConvergenceOptions, Event, Gaussian, History, InferenceError, Member, Outcome,
|
|
Team,
|
|
};
|
|
|
|
type H = History<i64, ConstantDrift, trueskill_tt::NullObserver, &'static str>;
|
|
|
|
fn history() -> H {
|
|
History::builder()
|
|
.mu(0.0)
|
|
.sigma(6.0)
|
|
.beta(1.0)
|
|
.score_sigma(2.0)
|
|
.drift(ConstantDrift::new(0.5))
|
|
.convergence(ConvergenceOptions {
|
|
max_iter: 20_000,
|
|
epsilon: 1e-13,
|
|
alpha: 1.0,
|
|
})
|
|
.build()
|
|
}
|
|
|
|
const PRIOR: Gaussian = Gaussian::from_ms(3.0, 1.5);
|
|
|
|
/// The contract that makes the escape hatch worth having: same configuration,
|
|
/// same fit, bit for bit.
|
|
#[test]
|
|
fn members_matches_the_typed_path_exactly() {
|
|
let mut typed = history();
|
|
typed
|
|
.add_events(vec![Event {
|
|
time: 1,
|
|
teams: smallvec![
|
|
Team::with_members([Member::new("player")]),
|
|
Team::with_members([Member::new("layout_7")
|
|
.with_drift_scale(0.0)
|
|
.with_prior(PRIOR)]),
|
|
],
|
|
outcome: Outcome::scores([5.0, 2.0]),
|
|
}])
|
|
.unwrap();
|
|
assert!(typed.converge().unwrap().converged);
|
|
|
|
let mut fluent = history();
|
|
fluent
|
|
.event(1)
|
|
.team(["player"])
|
|
.members([Member::new("layout_7")
|
|
.with_drift_scale(0.0)
|
|
.with_prior(PRIOR)])
|
|
.scores([5.0, 2.0])
|
|
.commit()
|
|
.unwrap();
|
|
assert!(fluent.converge().unwrap().converged);
|
|
|
|
for key in ["player", "layout_7"] {
|
|
let a = typed.current_skill(&key).unwrap();
|
|
let b = fluent.current_skill(&key).unwrap();
|
|
assert_eq!(a.pi(), b.pi(), "{key} pi");
|
|
assert_eq!(a.tau(), b.tau(), "{key} tau");
|
|
}
|
|
}
|
|
|
|
/// The configuration has to actually take effect, not merely round-trip: a
|
|
/// competitor pinned with `drift_scale = 0.0` must not move across slices,
|
|
/// where an unpinned one does.
|
|
///
|
|
/// The comparison is against a control rather than against a fixed epsilon.
|
|
/// Pinned marginals are not bit-identical across slices — each slice combines
|
|
/// its own forward and backward messages, so the arithmetic order differs and
|
|
/// the last bit moves. What "pinned" promises is that no drift variance
|
|
/// accumulates, and the control is what makes that measurable.
|
|
#[test]
|
|
fn a_drift_scale_set_through_members_is_applied() {
|
|
fn spread(h: &H, key: &'static str) -> f64 {
|
|
let curve = h.learning_curve(&key).unwrap();
|
|
assert!(curve.len() >= 2, "{key}: expected several appearances");
|
|
let (lo, hi) = curve.iter().fold((f64::MAX, f64::MIN), |(lo, hi), (_, g)| {
|
|
(lo.min(g.sigma()), hi.max(g.sigma()))
|
|
});
|
|
(hi - lo) / hi
|
|
}
|
|
|
|
let mut h = history();
|
|
for t in 1..=4 {
|
|
h.event(t)
|
|
.team(["player"])
|
|
.members([Member::new("pinned").with_drift_scale(0.0)])
|
|
.scores([5.0, 2.0])
|
|
.commit()
|
|
.unwrap();
|
|
// Same shape, no pinning: the control.
|
|
h.event(t)
|
|
.team(["rival"])
|
|
.team(["drifting"])
|
|
.scores([5.0, 2.0])
|
|
.commit()
|
|
.unwrap();
|
|
}
|
|
assert!(h.converge().unwrap().converged);
|
|
|
|
let pinned = spread(&h, "pinned");
|
|
let drifting = spread(&h, "drifting");
|
|
assert!(pinned < 1e-9, "pinned competitor moved: {pinned:e}");
|
|
assert!(
|
|
drifting > 1e-3,
|
|
"control did not move, so the test proves nothing: {drifting:e}"
|
|
);
|
|
}
|
|
|
|
/// `weights` still applies to a team added through `members`, and still
|
|
/// records a mismatch rather than partially applying it.
|
|
#[test]
|
|
fn weights_still_guards_a_members_team() {
|
|
let mut h = history();
|
|
let err = h
|
|
.event(1)
|
|
.team(["a"])
|
|
.members([Member::new("b"), Member::new("c")])
|
|
.weights([1.0])
|
|
.winner(0)
|
|
.commit()
|
|
.unwrap_err();
|
|
assert!(
|
|
matches!(
|
|
err,
|
|
InferenceError::MismatchedShape {
|
|
kind: "weights",
|
|
expected: 2,
|
|
got: 1,
|
|
..
|
|
}
|
|
),
|
|
"{err:?}"
|
|
);
|
|
assert!(h.current_skill(&"b").is_none(), "nothing may reach history");
|
|
}
|
|
|
|
/// An invalid `drift_scale` surfaces from `commit`, not from a panic and not
|
|
/// silently.
|
|
#[test]
|
|
fn an_invalid_drift_scale_surfaces_from_commit() {
|
|
for bad in [-1.0, f64::NAN, f64::INFINITY] {
|
|
let mut h = history();
|
|
let err = h
|
|
.event(1)
|
|
.team(["a"])
|
|
.members([Member::new("b").with_drift_scale(bad)])
|
|
.winner(0)
|
|
.commit()
|
|
.unwrap_err();
|
|
assert!(
|
|
matches!(
|
|
err,
|
|
InferenceError::InvalidParameter {
|
|
name: "drift_scale",
|
|
..
|
|
}
|
|
),
|
|
"{bad}: {err:?}"
|
|
);
|
|
assert!(h.current_skill(&"b").is_none(), "{bad} reached the history");
|
|
}
|
|
}
|
|
|
|
/// `members` and `team` compose in either order.
|
|
#[test]
|
|
fn members_and_team_interleave() {
|
|
let mut h = history();
|
|
h.event(1)
|
|
.members([Member::new("a").with_prior(PRIOR)])
|
|
.team(["b"])
|
|
.scores([3.0, 1.0])
|
|
.commit()
|
|
.unwrap();
|
|
h.event(2)
|
|
.team(["b"])
|
|
.members([Member::new("c").with_prior(PRIOR)])
|
|
.scores([2.0, 4.0])
|
|
.commit()
|
|
.unwrap();
|
|
assert!(h.converge().unwrap().converged);
|
|
for key in ["a", "b", "c"] {
|
|
assert!(h.current_skill(&key).is_some(), "{key} missing");
|
|
}
|
|
}
|