feat: add expected information gain for active matchup selection
`quality()` answers "is this matchup fair". Callers picking which
comparison to run next need "is this matchup informative", and the two
coincide only for two evenly matched competitors. Without a principled
alternative, downstream code was reaching for hand-rolled heuristics
like `quality * sigma_a^2 * sigma_b^2`, which double-counts uncertainty:
the two factors are not independent.
Adds `expected_information_gain`, the outcome-weighted divergence
between current beliefs and the beliefs each result would produce:
EIG = SUM P(outcome) * KL(posterior_after(outcome) || prior)
Available standalone over `Rating`s, and as
`History::expected_information_gain` using current skills and the
history's own beta, drift and p_draw — so the outcomes it weighs are the
ones that would actually be fitted.
This is the mutual information between the outcome and the skills, which
gives an analytic ceiling: gain cannot exceed the entropy of the thing
being observed, so at most `ln k` nats for k outcomes. That bound is the
sharpest test available, because an acquisition function is unusually
exposed to returning finite, plausible, monotone numbers while being
wrong — it would simply select slightly worse matchups forever. A
prototype of this returned 4.77 nats from a sign error while passing
every monotonicity check; `never_exceeds_the_entropy_of_the_outcome`
catches that class unconditionally.
Measured against the ceiling the values are meaningful rather than
vacuous: 0.382 nats for an even matchup between diffuse priors against
an 0.693 ceiling, falling to 0.013 for a lopsided one and 0.000 for a
hopeless one.
`disagrees_with_the_quality_times_variance_heuristic` pins down that
this is not a monotone transform of the heuristic it replaces — the two
rank a lopsided matchup and a confident even one in opposite orders — so
a later "simplification" cannot quietly revert to it.
Cost is one inference pass per possible outcome, documented on the
public API alongside the shortlist-then-score pattern, so callers do not
discover it in production.
Also folds the duplicated key-gathering in `predict_quality` and
`performances` into one validated `member_skills`.
Refs #39
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
@@ -164,6 +164,75 @@ h.event(1)
|
|||||||
h.converge().unwrap();
|
h.converge().unwrap();
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Prediction
|
||||||
|
|
||||||
|
`predict_outcome` gives the full distribution over finishing orders. Each entry
|
||||||
|
is a rank vector in the same shape `Outcome::ranking` takes — equal ranks mean a
|
||||||
|
tie — so an outcome feeds straight back into inference.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use trueskill_tt::History;
|
||||||
|
|
||||||
|
let mut h = History::builder().p_draw(0.1).build();
|
||||||
|
h.record_winner(&"alice", &"bob", 1).unwrap();
|
||||||
|
h.converge().unwrap();
|
||||||
|
|
||||||
|
let p = h.predict_outcome(&[&[&"alice"], &[&"bob"]]).unwrap();
|
||||||
|
|
||||||
|
// Probabilities are exhaustive and disjoint, so they sum to one.
|
||||||
|
assert!((p.total() - 1.0).abs() < 1e-6);
|
||||||
|
|
||||||
|
let (best, likelihood) = p.most_likely().unwrap();
|
||||||
|
println!("most likely: {best:?} at {likelihood:.3}");
|
||||||
|
println!("draw: {:.3}", p.probability_of(&[0, 0]));
|
||||||
|
```
|
||||||
|
|
||||||
|
Supports any number of teams. Because the outcome space grows factorially, the
|
||||||
|
full distribution is capped at `MAX_PREDICTED_TEAMS`; two cheaper entry points
|
||||||
|
stay available at any size:
|
||||||
|
|
||||||
|
- `predict_win_probabilities(teams)` — `P(team i finishes strictly first)`,
|
||||||
|
quadratic in team count.
|
||||||
|
- `predict_ranking(teams, ranks)` — one specific finishing order.
|
||||||
|
|
||||||
|
Unknown keys are an error, not a silent omission: a team the history has never
|
||||||
|
seen cannot produce a confident-looking probability.
|
||||||
|
|
||||||
|
## Which match to play next
|
||||||
|
|
||||||
|
`quality()` measures whether a matchup is *fair*. That is not the same as
|
||||||
|
whether it is *informative*, and the two only coincide for two evenly matched
|
||||||
|
competitors. When each observation costs something, ask
|
||||||
|
`expected_information_gain` instead — the outcome-weighted divergence between
|
||||||
|
what you believe now and what you would believe afterwards.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use trueskill_tt::History;
|
||||||
|
|
||||||
|
let mut h = History::builder().build();
|
||||||
|
for t in 1..=10 {
|
||||||
|
h.record_winner(&"veteran", &"regular", t).unwrap();
|
||||||
|
h.record_winner(&"regular", &"veteran", t + 100).unwrap();
|
||||||
|
}
|
||||||
|
h.record_winner(&"veteran", &"newcomer", 500).unwrap();
|
||||||
|
h.converge().unwrap();
|
||||||
|
|
||||||
|
let settled = h.expected_information_gain(&[&[&"veteran"], &[&"regular"]]).unwrap();
|
||||||
|
let unknown = h.expected_information_gain(&[&[&"veteran"], &[&"newcomer"]]).unwrap();
|
||||||
|
|
||||||
|
// Playing the newcomer teaches you more than replaying a settled rivalry.
|
||||||
|
assert!(unknown > settled);
|
||||||
|
```
|
||||||
|
|
||||||
|
The result is in nats, and is bounded by the entropy of the outcome: at most
|
||||||
|
`ln 2 ≈ 0.693` for a two-way result, `ln 3` once draws are possible, `ln k` for
|
||||||
|
`k` outcomes. A value near zero means you already know how it ends.
|
||||||
|
|
||||||
|
This costs one full inference pass **per possible outcome**, so it is far more
|
||||||
|
expensive than `quality()`. Scoring every pairing among `n` competitors is
|
||||||
|
`O(n² × outcomes)` passes — shortlist with `quality()` or
|
||||||
|
`predict_win_probabilities` first, then score only the shortlist.
|
||||||
|
|
||||||
## Todo
|
## Todo
|
||||||
|
|
||||||
- [x] Implement approx for Gaussian
|
- [x] Implement approx for Gaussian
|
||||||
@@ -172,6 +241,7 @@ h.converge().unwrap();
|
|||||||
- [x] Add examples (`examples/atp.rs`, `examples/scored.rs`)
|
- [x] Add examples (`examples/atp.rs`, `examples/scored.rs`)
|
||||||
- [x] Add Observer (`Observer` / `NullObserver`)
|
- [x] Add Observer (`Observer` / `NullObserver`)
|
||||||
- [x] Benchmark the inference loop (`benches/batch.rs`, `benches/history_converge.rs`, `benches/ingest.rs`)
|
- [x] Benchmark the inference loop (`benches/batch.rs`, `benches/history_converge.rs`, `benches/ingest.rs`)
|
||||||
|
- [x] N-team `predict_outcome` with draw mass, and `expected_information_gain`
|
||||||
- [ ] Cross-check `quality()` against [sublee/trueskill](https://github.com/sublee/trueskill/tree/master) — N-group support works and is covered by invariants, but no reference values are asserted
|
- [ ] Cross-check `quality()` against [sublee/trueskill](https://github.com/sublee/trueskill/tree/master) — N-group support works and is covered by invariants, but no reference values are asserted
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|||||||
@@ -0,0 +1,352 @@
|
|||||||
|
//! Active learning: which comparison teaches you the most.
|
||||||
|
//!
|
||||||
|
//! [`quality`](crate::quality) answers "is this matchup *fair*". That is a
|
||||||
|
//! different question from "is this matchup *informative*", and the two
|
||||||
|
//! coincide only for two evenly matched competitors. When each observation
|
||||||
|
//! costs something — a human click, a scheduled fixture — the question worth
|
||||||
|
//! asking is the second one.
|
||||||
|
//!
|
||||||
|
//! The quantity here is expected information gain: the outcome-weighted
|
||||||
|
//! divergence between what you believe now and what you would believe after
|
||||||
|
//! seeing the result.
|
||||||
|
//!
|
||||||
|
//! ```text
|
||||||
|
//! EIG(matchup) = SUM P(outcome) * KL( posterior_after(outcome) || prior )
|
||||||
|
//! outcome
|
||||||
|
//! ```
|
||||||
|
//!
|
||||||
|
//! It is the mutual information between the observed outcome and the skills,
|
||||||
|
//! which is worth remembering because it pins the scale: information gain
|
||||||
|
//! cannot exceed the entropy of the thing you are about to observe. A contest
|
||||||
|
//! with `k` distinguishable outcomes can teach you at most `ln k` nats,
|
||||||
|
//! whatever the ratings. That ceiling is the sharpest available test of an
|
||||||
|
//! implementation — see [`expected_information_gain`].
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
GameOptions, Gaussian, InferenceError, Outcome, Rating, drift::Drift, predict, time::Time,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Outcomes below this probability contribute nothing measurable and are not
|
||||||
|
/// worth an inference pass.
|
||||||
|
///
|
||||||
|
/// The contribution of an outcome is `P * KL`, and `KL` is bounded in practice
|
||||||
|
/// by tens of nats, so a probability this small moves the total by less than
|
||||||
|
/// the quadrature error already present in `P` itself.
|
||||||
|
const NEGLIGIBLE: f64 = 1e-12;
|
||||||
|
|
||||||
|
/// `KL(q || p)` for two univariate Gaussians, in nats.
|
||||||
|
///
|
||||||
|
/// Both arguments are proper posteriors from inference, so the degenerate
|
||||||
|
/// cases guarded here (zero or infinite variance) indicate that inference has
|
||||||
|
/// broken down rather than anything a caller did.
|
||||||
|
fn kl_divergence(q: Gaussian, p: Gaussian) -> f64 {
|
||||||
|
let (var_q, var_p) = (q.sigma().powi(2), p.sigma().powi(2));
|
||||||
|
|
||||||
|
if !(var_q.is_finite() && var_p.is_finite()) || var_q <= 0.0 || var_p <= 0.0 {
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mean_gap = q.mu() - p.mu();
|
||||||
|
0.5 * ((var_p / var_q).ln() + (var_q + mean_gap * mean_gap) / var_p - 1.0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Expected information gain of a hypothetical matchup, in nats.
|
||||||
|
///
|
||||||
|
/// Enumerates the outcomes this matchup could have, runs inference for each to
|
||||||
|
/// get the belief it would produce, and weights the resulting divergence by
|
||||||
|
/// that outcome's probability. A higher value means the result would teach you
|
||||||
|
/// more.
|
||||||
|
///
|
||||||
|
/// # Interpreting the value
|
||||||
|
///
|
||||||
|
/// Nats. The upper bound is the entropy of the outcome variable: at most
|
||||||
|
/// `ln 2 ≈ 0.693` for a two-way result, `ln 3 ≈ 1.099` once draws are
|
||||||
|
/// possible, `ln k` for `k` outcomes. A value near the ceiling means the
|
||||||
|
/// result is close to a coin flip *and* would move the posteriors a long way;
|
||||||
|
/// a value near zero means you already know what will happen, or that the
|
||||||
|
/// result would barely change your beliefs if you saw it.
|
||||||
|
///
|
||||||
|
/// This is not a monotone transform of [`quality`](crate::quality). A lopsided
|
||||||
|
/// matchup between two uncertain competitors scores well on quality-times-
|
||||||
|
/// variance heuristics and poorly here, because the near-certain outcome
|
||||||
|
/// carries almost no information.
|
||||||
|
///
|
||||||
|
/// # Cost
|
||||||
|
///
|
||||||
|
/// One full inference pass per possible outcome, so this is far more expensive
|
||||||
|
/// than `quality()` — which is one closed-form evaluation. The outcome count
|
||||||
|
/// grows quickly with team count (3 outcomes for two teams that can draw, 13
|
||||||
|
/// for three, 75 for four), and scoring every candidate pairing among `n`
|
||||||
|
/// competitors is `O(n² × outcomes)` inference passes.
|
||||||
|
///
|
||||||
|
/// For a selector over many candidates, shortlist with the cheap
|
||||||
|
/// [`quality`](crate::quality) or
|
||||||
|
/// [`predict_win_probabilities`](crate::History::predict_win_probabilities)
|
||||||
|
/// first and score only the shortlist here. The expected-variance-reduction
|
||||||
|
/// proxy sometimes suggested as a cheaper alternative is *not* cheaper: it
|
||||||
|
/// needs the same hypothetical posteriors, so it shares the dominant cost.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
///
|
||||||
|
/// - `NotEnoughTeams` if fewer than two teams are supplied.
|
||||||
|
/// - `EmptyTeam` if any team has no members.
|
||||||
|
/// - `TooManyTeams` if the outcome space is too large to enumerate; see
|
||||||
|
/// [`MAX_PREDICTED_TEAMS`](crate::MAX_PREDICTED_TEAMS).
|
||||||
|
/// - `InvalidProbability` if `options.p_draw` is outside `[0.0, 1.0)`.
|
||||||
|
/// - Anything [`Game::ranked`](crate::Game::ranked) returns for a hypothetical
|
||||||
|
/// outcome.
|
||||||
|
pub fn expected_information_gain<T: Time, D: Drift<T>>(
|
||||||
|
teams: &[&[Rating<T, D>]],
|
||||||
|
options: &GameOptions,
|
||||||
|
) -> Result<f64, InferenceError> {
|
||||||
|
if teams.len() < 2 {
|
||||||
|
return Err(InferenceError::NotEnoughTeams { got: teams.len() });
|
||||||
|
}
|
||||||
|
if teams.len() > crate::MAX_PREDICTED_TEAMS {
|
||||||
|
return Err(InferenceError::TooManyTeams {
|
||||||
|
got: teams.len(),
|
||||||
|
max: crate::MAX_PREDICTED_TEAMS,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if !(0.0..1.0).contains(&options.p_draw) {
|
||||||
|
return Err(InferenceError::InvalidProbability {
|
||||||
|
value: options.p_draw,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
for (idx, team) in teams.iter().enumerate() {
|
||||||
|
if team.is_empty() {
|
||||||
|
return Err(InferenceError::EmptyTeam { team: idx });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Prediction runs on performances: skill inflated by each member's beta.
|
||||||
|
let performances: Vec<Gaussian> = teams
|
||||||
|
.iter()
|
||||||
|
.map(|team| {
|
||||||
|
team.iter()
|
||||||
|
.fold(crate::N00, |acc, rating| acc + rating.performance())
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
// Draw margins per pair, derived from the teams' betas exactly as
|
||||||
|
// inference derives them, so the outcomes weighted here are the outcomes
|
||||||
|
// that would actually be fitted.
|
||||||
|
let beta_sq: Vec<f64> = teams
|
||||||
|
.iter()
|
||||||
|
.map(|team| team.iter().map(|r| r.beta().powi(2)).sum())
|
||||||
|
.collect();
|
||||||
|
let p_draw = options.p_draw;
|
||||||
|
let margins = predict::Margins::new(teams.len(), |i, j| {
|
||||||
|
if p_draw == 0.0 {
|
||||||
|
0.0
|
||||||
|
} else {
|
||||||
|
crate::compute_margin(p_draw, (beta_sq[i] + beta_sq[j]).sqrt())
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut gain = 0.0;
|
||||||
|
|
||||||
|
for (ranks, probability) in predict::outcome_distribution(&performances, &margins) {
|
||||||
|
if probability <= NEGLIGIBLE {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let game = crate::Game::ranked(teams, Outcome::ranking(ranks), options)?;
|
||||||
|
let posteriors = game.posteriors();
|
||||||
|
|
||||||
|
// Beliefs factorise across competitors, so the joint divergence is the
|
||||||
|
// sum of the per-competitor ones.
|
||||||
|
let divergence: f64 = teams
|
||||||
|
.iter()
|
||||||
|
.zip(&posteriors)
|
||||||
|
.flat_map(|(team, posterior)| team.iter().zip(posterior))
|
||||||
|
.map(|(rating, &after)| kl_divergence(after, rating.prior()))
|
||||||
|
.sum();
|
||||||
|
|
||||||
|
gain += probability * divergence;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(gain)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::{BETA, ConstantDrift, GAMMA};
|
||||||
|
|
||||||
|
type R = Rating<i64, ConstantDrift>;
|
||||||
|
|
||||||
|
fn rating(mu: f64, sigma: f64) -> R {
|
||||||
|
R::new(Gaussian::from_ms(mu, sigma), BETA, ConstantDrift(GAMMA))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn options(p_draw: f64) -> GameOptions {
|
||||||
|
GameOptions {
|
||||||
|
p_draw,
|
||||||
|
..GameOptions::default()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn eig(teams: &[&[R]], p_draw: f64) -> f64 {
|
||||||
|
expected_information_gain(teams, &options(p_draw)).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The analytic ceiling. Information gain is the mutual information between
|
||||||
|
/// the outcome and the skills, so it cannot exceed the entropy of the
|
||||||
|
/// outcome variable — whatever the ratings. This is the check a subtly
|
||||||
|
/// wrong implementation fails while still returning plausible numbers: an
|
||||||
|
/// early prototype of this returned 4.77 nats from a sign error and passed
|
||||||
|
/// every monotonicity test.
|
||||||
|
#[test]
|
||||||
|
fn never_exceeds_the_entropy_of_the_outcome() {
|
||||||
|
let ceiling_two = std::f64::consts::LN_2;
|
||||||
|
|
||||||
|
for (a, b) in [
|
||||||
|
(rating(0.0, 6.0), rating(0.0, 6.0)),
|
||||||
|
(rating(0.0, 0.5), rating(0.0, 0.5)),
|
||||||
|
(rating(12.0, 6.0), rating(-12.0, 6.0)),
|
||||||
|
(rating(40.0, 1.0), rating(-40.0, 1.0)),
|
||||||
|
(rating(3.0, 6.0), rating(-2.0, 0.1)),
|
||||||
|
(rating(0.0, 25.0), rating(0.0, 25.0)),
|
||||||
|
] {
|
||||||
|
let g = eig(&[&[a], &[b]], 0.0);
|
||||||
|
assert!(
|
||||||
|
g >= 0.0 && g <= ceiling_two,
|
||||||
|
"EIG {g} outside [0, ln 2] for mu=({}, {}) sigma=({}, {})",
|
||||||
|
a.prior().mu(),
|
||||||
|
b.prior().mu(),
|
||||||
|
a.prior().sigma(),
|
||||||
|
b.prior().sigma()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// With draws enabled there are three outcomes, so the ceiling rises to
|
||||||
|
/// `ln 3` — and the two-outcome bound no longer applies.
|
||||||
|
#[test]
|
||||||
|
fn the_ceiling_follows_the_outcome_count() {
|
||||||
|
let ceiling_three = 3.0f64.ln();
|
||||||
|
for sigma in [0.5, 3.0, 6.0, 25.0] {
|
||||||
|
let g = eig(&[&[rating(0.0, sigma)], &[rating(0.0, sigma)]], 0.25);
|
||||||
|
assert!(
|
||||||
|
g >= 0.0 && g <= ceiling_three,
|
||||||
|
"EIG {g} outside [0, ln 3] at sigma {sigma}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// An even matchup between uncertain competitors is the informative one.
|
||||||
|
/// A hopelessly lopsided matchup teaches you almost nothing, because you
|
||||||
|
/// already know how it ends.
|
||||||
|
#[test]
|
||||||
|
fn an_even_matchup_beats_a_lopsided_one() {
|
||||||
|
let even = eig(&[&[rating(0.0, 6.0)], &[rating(0.0, 6.0)]], 0.0);
|
||||||
|
let lopsided = eig(&[&[rating(12.0, 6.0)], &[rating(-12.0, 6.0)]], 0.0);
|
||||||
|
assert!(
|
||||||
|
even > lopsided,
|
||||||
|
"even {even} should beat lopsided {lopsided}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Certainty is the thing information gain is measuring the absence of:
|
||||||
|
/// the less you know, the more there is to learn.
|
||||||
|
#[test]
|
||||||
|
fn gain_falls_as_certainty_rises() {
|
||||||
|
let mut previous = f64::INFINITY;
|
||||||
|
for sigma in [12.0, 6.0, 3.0, 1.0, 0.5, 0.1] {
|
||||||
|
let g = eig(&[&[rating(0.0, sigma)], &[rating(0.0, sigma)]], 0.0);
|
||||||
|
assert!(
|
||||||
|
g < previous,
|
||||||
|
"sigma {sigma}: {g} did not fall below {previous}"
|
||||||
|
);
|
||||||
|
previous = g;
|
||||||
|
}
|
||||||
|
assert!(previous >= 0.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The heuristic this replaces is `quality * sigma_a^2 * sigma_b^2`. It is
|
||||||
|
/// not a monotone transform of information gain — it ranks a lopsided
|
||||||
|
/// matchup above a confident even one, and EIG ranks them the other way.
|
||||||
|
/// Pinning the disagreement down is what stops a future "simplification"
|
||||||
|
/// from quietly reverting to the heuristic.
|
||||||
|
#[test]
|
||||||
|
fn disagrees_with_the_quality_times_variance_heuristic() {
|
||||||
|
let heuristic = |a: &R, b: &R| {
|
||||||
|
crate::quality(&[&[a.prior()], &[b.prior()]], BETA)
|
||||||
|
* a.prior().sigma().powi(2)
|
||||||
|
* b.prior().sigma().powi(2)
|
||||||
|
};
|
||||||
|
|
||||||
|
let (confident_a, confident_b) = (rating(0.0, 0.5), rating(0.0, 0.5));
|
||||||
|
let (lopsided_a, lopsided_b) = (rating(12.0, 6.0), rating(-12.0, 6.0));
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
heuristic(&lopsided_a, &lopsided_b) > heuristic(&confident_a, &confident_b),
|
||||||
|
"the heuristic should prefer the lopsided matchup"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
eig(&[&[confident_a], &[confident_b]], 0.0) > eig(&[&[lopsided_a], &[lopsided_b]], 0.0),
|
||||||
|
"information gain should prefer the even matchup"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn supports_more_than_two_teams() {
|
||||||
|
let teams: Vec<Vec<R>> = vec![
|
||||||
|
vec![rating(0.0, 6.0)],
|
||||||
|
vec![rating(0.0, 6.0)],
|
||||||
|
vec![rating(0.0, 6.0)],
|
||||||
|
];
|
||||||
|
let refs: Vec<&[R]> = teams.iter().map(Vec::as_slice).collect();
|
||||||
|
let g = expected_information_gain(&refs, &options(0.0)).unwrap();
|
||||||
|
// Six distinguishable orderings with no draws.
|
||||||
|
assert!(
|
||||||
|
g > 0.0 && g <= 6.0f64.ln(),
|
||||||
|
"three-team EIG {g} out of range"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn multi_member_teams_are_supported() {
|
||||||
|
let a = [rating(0.0, 6.0), rating(1.0, 4.0)];
|
||||||
|
let b = [rating(0.0, 6.0)];
|
||||||
|
let g = expected_information_gain(&[&a, &b], &options(0.0)).unwrap();
|
||||||
|
assert!(g > 0.0 && g <= std::f64::consts::LN_2, "{g}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn degenerate_shapes_are_errors() {
|
||||||
|
let a = [rating(0.0, 6.0)];
|
||||||
|
assert!(matches!(
|
||||||
|
expected_information_gain(&[&a], &options(0.0)),
|
||||||
|
Err(InferenceError::NotEnoughTeams { got: 1 })
|
||||||
|
));
|
||||||
|
let empty: [R; 0] = [];
|
||||||
|
assert!(matches!(
|
||||||
|
expected_information_gain(&[&a, &empty], &options(0.0)),
|
||||||
|
Err(InferenceError::EmptyTeam { team: 1 })
|
||||||
|
));
|
||||||
|
assert!(matches!(
|
||||||
|
expected_information_gain(&[&a, &a], &options(1.5)),
|
||||||
|
Err(InferenceError::InvalidProbability { .. })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn kl_divergence_is_zero_for_identical_beliefs() {
|
||||||
|
let g = Gaussian::from_ms(3.0, 2.0);
|
||||||
|
assert!(kl_divergence(g, g).abs() < 1e-15);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn kl_divergence_is_non_negative_and_grows_with_separation() {
|
||||||
|
let prior = Gaussian::from_ms(0.0, 3.0);
|
||||||
|
let mut previous = 0.0;
|
||||||
|
for mu in [0.0, 0.5, 1.0, 2.0, 4.0] {
|
||||||
|
let d = kl_divergence(Gaussian::from_ms(mu, 3.0), prior);
|
||||||
|
assert!(d >= 0.0, "negative divergence at mu {mu}: {d}");
|
||||||
|
assert!(d >= previous, "not increasing at mu {mu}");
|
||||||
|
previous = d;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+77
-42
@@ -538,10 +538,7 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
|
|||||||
.sum()
|
.sum()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Each team's performance Gaussian, and its member count.
|
/// Every team's member skills, validated.
|
||||||
///
|
|
||||||
/// Performance is skill inflated by `beta`: the question a prediction
|
|
||||||
/// answers is "how will they do today", not "how good are they".
|
|
||||||
///
|
///
|
||||||
/// # Errors
|
/// # Errors
|
||||||
///
|
///
|
||||||
@@ -549,39 +546,60 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
|
|||||||
/// reported rather than dropped — silently skipping them would turn a team
|
/// reported rather than dropped — silently skipping them would turn a team
|
||||||
/// of strangers into a confident-looking prediction about nobody, which is
|
/// of strangers into a confident-looking prediction about nobody, which is
|
||||||
/// the failure this replaced.
|
/// the failure this replaced.
|
||||||
fn performances(&self, teams: &[&[&K]]) -> Result<(Vec<Gaussian>, Vec<usize>), InferenceError> {
|
fn member_skills(&self, teams: &[&[&K]]) -> Result<Vec<Vec<Gaussian>>, InferenceError> {
|
||||||
if teams.len() < 2 {
|
if teams.len() < 2 {
|
||||||
return Err(InferenceError::NotEnoughTeams { got: teams.len() });
|
return Err(InferenceError::NotEnoughTeams { got: teams.len() });
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut performances = Vec::with_capacity(teams.len());
|
let mut gathered = Vec::with_capacity(teams.len());
|
||||||
let mut sizes = Vec::with_capacity(teams.len());
|
|
||||||
|
|
||||||
for (team_idx, team) in teams.iter().enumerate() {
|
for (team_idx, team) in teams.iter().enumerate() {
|
||||||
if team.is_empty() {
|
if team.is_empty() {
|
||||||
return Err(InferenceError::EmptyTeam { team: team_idx });
|
return Err(InferenceError::EmptyTeam { team: team_idx });
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut total = crate::N00;
|
let mut members = Vec::with_capacity(team.len());
|
||||||
for (member_idx, key) in team.iter().enumerate() {
|
for (member_idx, key) in team.iter().enumerate() {
|
||||||
let unknown = InferenceError::UnknownKey {
|
let unknown = InferenceError::UnknownKey {
|
||||||
team: team_idx,
|
team: team_idx,
|
||||||
member: member_idx,
|
member: member_idx,
|
||||||
};
|
};
|
||||||
let index = self.keys.get(*key).ok_or(unknown.clone())?;
|
let index = self.keys.get(*key).ok_or(unknown.clone())?;
|
||||||
let skill = self
|
members.push(
|
||||||
.time_slices
|
self.time_slices
|
||||||
.iter()
|
.iter()
|
||||||
.rev()
|
.rev()
|
||||||
.find_map(|ts| ts.skills.get(index).map(|s| s.posterior()))
|
.find_map(|ts| ts.skills.get(index).map(|s| s.posterior()))
|
||||||
.ok_or(unknown)?;
|
.ok_or(unknown)?,
|
||||||
total = total + skill.forget(self.beta.powi(2));
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
performances.push(total);
|
gathered.push(members);
|
||||||
sizes.push(team.len());
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
Ok(gathered)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Each team's performance Gaussian, and its member count.
|
||||||
|
///
|
||||||
|
/// Performance is skill inflated by `beta`: the question a prediction
|
||||||
|
/// answers is "how will they do today", not "how good are they".
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
///
|
||||||
|
/// As [`History::member_skills`].
|
||||||
|
fn performances(&self, teams: &[&[&K]]) -> Result<(Vec<Gaussian>, Vec<usize>), InferenceError> {
|
||||||
|
let skills = self.member_skills(teams)?;
|
||||||
|
|
||||||
|
let performances = skills
|
||||||
|
.iter()
|
||||||
|
.map(|team| {
|
||||||
|
team.iter()
|
||||||
|
.fold(crate::N00, |acc, s| acc + s.forget(self.beta.powi(2)))
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
let sizes = skills.iter().map(Vec::len).collect();
|
||||||
|
|
||||||
Ok((performances, sizes))
|
Ok((performances, sizes))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -618,38 +636,55 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
|
|||||||
///
|
///
|
||||||
/// `NotEnoughTeams`, `EmptyTeam`, or `UnknownKey`.
|
/// `NotEnoughTeams`, `EmptyTeam`, or `UnknownKey`.
|
||||||
pub fn predict_quality(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> {
|
pub fn predict_quality(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> {
|
||||||
let mut groups: Vec<Vec<Gaussian>> = Vec::with_capacity(teams.len());
|
let groups = self.member_skills(teams)?;
|
||||||
|
|
||||||
for (team_idx, team) in teams.iter().enumerate() {
|
|
||||||
if team.is_empty() {
|
|
||||||
return Err(InferenceError::EmptyTeam { team: team_idx });
|
|
||||||
}
|
|
||||||
let mut members = Vec::with_capacity(team.len());
|
|
||||||
for (member_idx, key) in team.iter().enumerate() {
|
|
||||||
let unknown = InferenceError::UnknownKey {
|
|
||||||
team: team_idx,
|
|
||||||
member: member_idx,
|
|
||||||
};
|
|
||||||
let index = self.keys.get(*key).ok_or(unknown.clone())?;
|
|
||||||
members.push(
|
|
||||||
self.time_slices
|
|
||||||
.iter()
|
|
||||||
.rev()
|
|
||||||
.find_map(|ts| ts.skills.get(index).map(|s| s.posterior()))
|
|
||||||
.ok_or(unknown)?,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
groups.push(members);
|
|
||||||
}
|
|
||||||
|
|
||||||
if groups.len() < 2 {
|
|
||||||
return Err(InferenceError::NotEnoughTeams { got: groups.len() });
|
|
||||||
}
|
|
||||||
|
|
||||||
let group_refs: Vec<&[Gaussian]> = groups.iter().map(Vec::as_slice).collect();
|
let group_refs: Vec<&[Gaussian]> = groups.iter().map(Vec::as_slice).collect();
|
||||||
Ok(crate::quality(&group_refs, self.beta))
|
Ok(crate::quality(&group_refs, self.beta))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Expected information gain of running this matchup, in nats.
|
||||||
|
///
|
||||||
|
/// Answers "which comparison should I run next" rather than "who will
|
||||||
|
/// win": the outcome-weighted divergence between current beliefs and the
|
||||||
|
/// beliefs each possible result would produce. Higher means the result
|
||||||
|
/// would teach you more.
|
||||||
|
///
|
||||||
|
/// Uses each competitor's current skill as the prior, and the history's
|
||||||
|
/// own `beta`, `drift` and `p_draw`, so the outcomes weighted here are the
|
||||||
|
/// ones that would actually be fitted if the matchup were played and
|
||||||
|
/// recorded.
|
||||||
|
///
|
||||||
|
/// Distinct from [`History::predict_quality`], which measures *fairness*.
|
||||||
|
/// The two coincide for two evenly matched competitors and diverge
|
||||||
|
/// elsewhere. See [`expected_information_gain`](crate::expected_information_gain)
|
||||||
|
/// for the scale, the analytic `ln k` ceiling, and the cost.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
///
|
||||||
|
/// As [`History::member_skills`], plus `TooManyTeams` and anything
|
||||||
|
/// inference returns for a hypothetical outcome.
|
||||||
|
pub fn expected_information_gain(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> {
|
||||||
|
let skills = self.member_skills(teams)?;
|
||||||
|
|
||||||
|
let ratings: Vec<Vec<Rating<T, D>>> = skills
|
||||||
|
.iter()
|
||||||
|
.map(|team| {
|
||||||
|
team.iter()
|
||||||
|
.map(|&skill| Rating::new(skill, self.beta, self.drift))
|
||||||
|
.collect()
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
let team_refs: Vec<&[Rating<T, D>]> = ratings.iter().map(Vec::as_slice).collect();
|
||||||
|
|
||||||
|
crate::expected_information_gain(
|
||||||
|
&team_refs,
|
||||||
|
&crate::GameOptions {
|
||||||
|
p_draw: self.p_draw,
|
||||||
|
score_sigma: self.score_sigma,
|
||||||
|
convergence: self.convergence,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
/// `P(team i finishes strictly first)`, for every team.
|
/// `P(team i finishes strictly first)`, for every team.
|
||||||
///
|
///
|
||||||
/// Supports any number of teams. Because performances are independent
|
/// Supports any number of teams. Because performances are independent
|
||||||
|
|||||||
@@ -110,6 +110,7 @@ pub(crate) mod arena;
|
|||||||
mod time;
|
mod time;
|
||||||
mod time_slice;
|
mod time_slice;
|
||||||
pub use time_slice::{EventKind, TimeSlice};
|
pub use time_slice::{EventKind, TimeSlice};
|
||||||
|
mod acquisition;
|
||||||
mod color_group;
|
mod color_group;
|
||||||
mod competitor;
|
mod competitor;
|
||||||
mod convergence;
|
mod convergence;
|
||||||
@@ -132,6 +133,7 @@ mod rating;
|
|||||||
pub(crate) mod schedule;
|
pub(crate) mod schedule;
|
||||||
pub mod storage;
|
pub mod storage;
|
||||||
|
|
||||||
|
pub use acquisition::expected_information_gain;
|
||||||
pub use competitor::Competitor;
|
pub use competitor::Competitor;
|
||||||
pub use convergence::{ConvergenceOptions, ConvergenceReport};
|
pub use convergence::{ConvergenceOptions, ConvergenceReport};
|
||||||
pub use drift::{ConstantDrift, Drift};
|
pub use drift::{ConstantDrift, Drift};
|
||||||
|
|||||||
@@ -212,3 +212,80 @@ fn team_size_affects_the_prediction() {
|
|||||||
assert!((p.total() - 1.0).abs() < 1e-6, "total = {}", p.total());
|
assert!((p.total() - 1.0).abs() < 1e-6, "total = {}", p.total());
|
||||||
assert!(p.probability_of(&[0, 0]) > 0.0);
|
assert!(p.probability_of(&[0, 0]) > 0.0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Expected information gain
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// The whole point of #39: "which comparison should I run next?" is a
|
||||||
|
/// different question from "who will win?" or "is this fair?".
|
||||||
|
#[test]
|
||||||
|
fn information_gain_prefers_the_uncertain_pairing() {
|
||||||
|
let mut h = History::builder().build();
|
||||||
|
|
||||||
|
// "known" and "rival" have played a lot; "newcomer" has played once.
|
||||||
|
for t in 1..=15 {
|
||||||
|
h.record_winner(&"known", &"rival", t).unwrap();
|
||||||
|
h.record_winner(&"rival", &"known", t + 100).unwrap();
|
||||||
|
}
|
||||||
|
h.record_winner(&"known", &"newcomer", 500).unwrap();
|
||||||
|
h.converge().unwrap();
|
||||||
|
|
||||||
|
let settled = h
|
||||||
|
.expected_information_gain(&[&[&"known"], &[&"rival"]])
|
||||||
|
.unwrap();
|
||||||
|
let unknown = h
|
||||||
|
.expected_information_gain(&[&[&"known"], &[&"newcomer"]])
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
unknown > settled,
|
||||||
|
"pairing against the newcomer should teach more: {unknown} vs {settled}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The analytic ceiling, through the `History` entry point rather than the
|
||||||
|
/// standalone one.
|
||||||
|
#[test]
|
||||||
|
fn information_gain_respects_the_entropy_ceiling() {
|
||||||
|
let h = history_with(&["a", "b", "c"], 0.0);
|
||||||
|
|
||||||
|
let two = h.expected_information_gain(&[&[&"a"], &[&"b"]]).unwrap();
|
||||||
|
assert!(
|
||||||
|
(0.0..=std::f64::consts::LN_2).contains(&two),
|
||||||
|
"two-team EIG {two} outside [0, ln 2]"
|
||||||
|
);
|
||||||
|
|
||||||
|
let three = h
|
||||||
|
.expected_information_gain(&[&[&"a"], &[&"b"], &[&"c"]])
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
(0.0..=6.0f64.ln()).contains(&three),
|
||||||
|
"three-team EIG {three} outside [0, ln 6]"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn information_gain_reports_unknown_keys() {
|
||||||
|
let h = history_with(&["a", "b"], 0.0);
|
||||||
|
assert_eq!(
|
||||||
|
h.expected_information_gain(&[&[&"a"], &[&"ghost"]])
|
||||||
|
.unwrap_err(),
|
||||||
|
InferenceError::UnknownKey { team: 1, member: 0 }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A draw-enabled history has three outcomes to weigh rather than two, so the
|
||||||
|
/// draw branch must actually be reachable through this path.
|
||||||
|
#[test]
|
||||||
|
fn information_gain_accounts_for_draws() {
|
||||||
|
let with_draws = history_with(&["a", "b"], 0.25);
|
||||||
|
let g = with_draws
|
||||||
|
.expected_information_gain(&[&[&"a"], &[&"b"]])
|
||||||
|
.unwrap();
|
||||||
|
assert!(g > 0.0 && g <= 3.0f64.ln(), "{g}");
|
||||||
|
|
||||||
|
// The draw outcome carries mass, so it is genuinely being weighed.
|
||||||
|
let dist = with_draws.predict_outcome(&[&[&"a"], &[&"b"]]).unwrap();
|
||||||
|
assert!(dist.probability_of(&[0, 0]) > 0.0);
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user