feat: add expected information gain for active matchup selection

`quality()` answers "is this matchup fair". Callers picking which
comparison to run next need "is this matchup informative", and the two
coincide only for two evenly matched competitors. Without a principled
alternative, downstream code was reaching for hand-rolled heuristics
like `quality * sigma_a^2 * sigma_b^2`, which double-counts uncertainty:
the two factors are not independent.

Adds `expected_information_gain`, the outcome-weighted divergence
between current beliefs and the beliefs each result would produce:

    EIG = SUM P(outcome) * KL(posterior_after(outcome) || prior)

Available standalone over `Rating`s, and as
`History::expected_information_gain` using current skills and the
history's own beta, drift and p_draw — so the outcomes it weighs are the
ones that would actually be fitted.

This is the mutual information between the outcome and the skills, which
gives an analytic ceiling: gain cannot exceed the entropy of the thing
being observed, so at most `ln k` nats for k outcomes. That bound is the
sharpest test available, because an acquisition function is unusually
exposed to returning finite, plausible, monotone numbers while being
wrong — it would simply select slightly worse matchups forever. A
prototype of this returned 4.77 nats from a sign error while passing
every monotonicity check; `never_exceeds_the_entropy_of_the_outcome`
catches that class unconditionally.

Measured against the ceiling the values are meaningful rather than
vacuous: 0.382 nats for an even matchup between diffuse priors against
an 0.693 ceiling, falling to 0.013 for a lopsided one and 0.000 for a
hopeless one.

`disagrees_with_the_quality_times_variance_heuristic` pins down that
this is not a monotone transform of the heuristic it replaces — the two
rank a lopsided matchup and a confident even one in opposite orders — so
a later "simplification" cannot quietly revert to it.

Cost is one inference pass per possible outcome, documented on the
public API alongside the shortlist-then-score pattern, so callers do not
discover it in production.

Also folds the duplicated key-gathering in `predict_quality` and
`performances` into one validated `member_skills`.

Refs #39

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
2026-09-07 15:08:12 +02:00
co-authored by Claude Opus 5
parent 507894dae7
commit 3c2f9ac64c
5 changed files with 581 additions and 45 deletions
+70
View File
@@ -164,6 +164,75 @@ h.event(1)
h.converge().unwrap(); h.converge().unwrap();
``` ```
## Prediction
`predict_outcome` gives the full distribution over finishing orders. Each entry
is a rank vector in the same shape `Outcome::ranking` takes — equal ranks mean a
tie — so an outcome feeds straight back into inference.
```rust
use trueskill_tt::History;
let mut h = History::builder().p_draw(0.1).build();
h.record_winner(&"alice", &"bob", 1).unwrap();
h.converge().unwrap();
let p = h.predict_outcome(&[&[&"alice"], &[&"bob"]]).unwrap();
// Probabilities are exhaustive and disjoint, so they sum to one.
assert!((p.total() - 1.0).abs() < 1e-6);
let (best, likelihood) = p.most_likely().unwrap();
println!("most likely: {best:?} at {likelihood:.3}");
println!("draw: {:.3}", p.probability_of(&[0, 0]));
```
Supports any number of teams. Because the outcome space grows factorially, the
full distribution is capped at `MAX_PREDICTED_TEAMS`; two cheaper entry points
stay available at any size:
- `predict_win_probabilities(teams)``P(team i finishes strictly first)`,
quadratic in team count.
- `predict_ranking(teams, ranks)` — one specific finishing order.
Unknown keys are an error, not a silent omission: a team the history has never
seen cannot produce a confident-looking probability.
## Which match to play next
`quality()` measures whether a matchup is *fair*. That is not the same as
whether it is *informative*, and the two only coincide for two evenly matched
competitors. When each observation costs something, ask
`expected_information_gain` instead — the outcome-weighted divergence between
what you believe now and what you would believe afterwards.
```rust
use trueskill_tt::History;
let mut h = History::builder().build();
for t in 1..=10 {
h.record_winner(&"veteran", &"regular", t).unwrap();
h.record_winner(&"regular", &"veteran", t + 100).unwrap();
}
h.record_winner(&"veteran", &"newcomer", 500).unwrap();
h.converge().unwrap();
let settled = h.expected_information_gain(&[&[&"veteran"], &[&"regular"]]).unwrap();
let unknown = h.expected_information_gain(&[&[&"veteran"], &[&"newcomer"]]).unwrap();
// Playing the newcomer teaches you more than replaying a settled rivalry.
assert!(unknown > settled);
```
The result is in nats, and is bounded by the entropy of the outcome: at most
`ln 2 ≈ 0.693` for a two-way result, `ln 3` once draws are possible, `ln k` for
`k` outcomes. A value near zero means you already know how it ends.
This costs one full inference pass **per possible outcome**, so it is far more
expensive than `quality()`. Scoring every pairing among `n` competitors is
`O(n² × outcomes)` passes — shortlist with `quality()` or
`predict_win_probabilities` first, then score only the shortlist.
## Todo ## Todo
- [x] Implement approx for Gaussian - [x] Implement approx for Gaussian
@@ -172,6 +241,7 @@ h.converge().unwrap();
- [x] Add examples (`examples/atp.rs`, `examples/scored.rs`) - [x] Add examples (`examples/atp.rs`, `examples/scored.rs`)
- [x] Add Observer (`Observer` / `NullObserver`) - [x] Add Observer (`Observer` / `NullObserver`)
- [x] Benchmark the inference loop (`benches/batch.rs`, `benches/history_converge.rs`, `benches/ingest.rs`) - [x] Benchmark the inference loop (`benches/batch.rs`, `benches/history_converge.rs`, `benches/ingest.rs`)
- [x] N-team `predict_outcome` with draw mass, and `expected_information_gain`
- [ ] Cross-check `quality()` against [sublee/trueskill](https://github.com/sublee/trueskill/tree/master) — N-group support works and is covered by invariants, but no reference values are asserted - [ ] Cross-check `quality()` against [sublee/trueskill](https://github.com/sublee/trueskill/tree/master) — N-group support works and is covered by invariants, but no reference values are asserted
## License ## License
+352
View File
@@ -0,0 +1,352 @@
//! Active learning: which comparison teaches you the most.
//!
//! [`quality`](crate::quality) answers "is this matchup *fair*". That is a
//! different question from "is this matchup *informative*", and the two
//! coincide only for two evenly matched competitors. When each observation
//! costs something — a human click, a scheduled fixture — the question worth
//! asking is the second one.
//!
//! The quantity here is expected information gain: the outcome-weighted
//! divergence between what you believe now and what you would believe after
//! seeing the result.
//!
//! ```text
//! EIG(matchup) = SUM P(outcome) * KL( posterior_after(outcome) || prior )
//! outcome
//! ```
//!
//! It is the mutual information between the observed outcome and the skills,
//! which is worth remembering because it pins the scale: information gain
//! cannot exceed the entropy of the thing you are about to observe. A contest
//! with `k` distinguishable outcomes can teach you at most `ln k` nats,
//! whatever the ratings. That ceiling is the sharpest available test of an
//! implementation — see [`expected_information_gain`].
use crate::{
GameOptions, Gaussian, InferenceError, Outcome, Rating, drift::Drift, predict, time::Time,
};
/// Outcomes below this probability contribute nothing measurable and are not
/// worth an inference pass.
///
/// The contribution of an outcome is `P * KL`, and `KL` is bounded in practice
/// by tens of nats, so a probability this small moves the total by less than
/// the quadrature error already present in `P` itself.
const NEGLIGIBLE: f64 = 1e-12;
/// `KL(q || p)` for two univariate Gaussians, in nats.
///
/// Both arguments are proper posteriors from inference, so the degenerate
/// cases guarded here (zero or infinite variance) indicate that inference has
/// broken down rather than anything a caller did.
fn kl_divergence(q: Gaussian, p: Gaussian) -> f64 {
let (var_q, var_p) = (q.sigma().powi(2), p.sigma().powi(2));
if !(var_q.is_finite() && var_p.is_finite()) || var_q <= 0.0 || var_p <= 0.0 {
return 0.0;
}
let mean_gap = q.mu() - p.mu();
0.5 * ((var_p / var_q).ln() + (var_q + mean_gap * mean_gap) / var_p - 1.0)
}
/// Expected information gain of a hypothetical matchup, in nats.
///
/// Enumerates the outcomes this matchup could have, runs inference for each to
/// get the belief it would produce, and weights the resulting divergence by
/// that outcome's probability. A higher value means the result would teach you
/// more.
///
/// # Interpreting the value
///
/// Nats. The upper bound is the entropy of the outcome variable: at most
/// `ln 2 ≈ 0.693` for a two-way result, `ln 3 ≈ 1.099` once draws are
/// possible, `ln k` for `k` outcomes. A value near the ceiling means the
/// result is close to a coin flip *and* would move the posteriors a long way;
/// a value near zero means you already know what will happen, or that the
/// result would barely change your beliefs if you saw it.
///
/// This is not a monotone transform of [`quality`](crate::quality). A lopsided
/// matchup between two uncertain competitors scores well on quality-times-
/// variance heuristics and poorly here, because the near-certain outcome
/// carries almost no information.
///
/// # Cost
///
/// One full inference pass per possible outcome, so this is far more expensive
/// than `quality()` — which is one closed-form evaluation. The outcome count
/// grows quickly with team count (3 outcomes for two teams that can draw, 13
/// for three, 75 for four), and scoring every candidate pairing among `n`
/// competitors is `O(n² × outcomes)` inference passes.
///
/// For a selector over many candidates, shortlist with the cheap
/// [`quality`](crate::quality) or
/// [`predict_win_probabilities`](crate::History::predict_win_probabilities)
/// first and score only the shortlist here. The expected-variance-reduction
/// proxy sometimes suggested as a cheaper alternative is *not* cheaper: it
/// needs the same hypothetical posteriors, so it shares the dominant cost.
///
/// # Errors
///
/// - `NotEnoughTeams` if fewer than two teams are supplied.
/// - `EmptyTeam` if any team has no members.
/// - `TooManyTeams` if the outcome space is too large to enumerate; see
/// [`MAX_PREDICTED_TEAMS`](crate::MAX_PREDICTED_TEAMS).
/// - `InvalidProbability` if `options.p_draw` is outside `[0.0, 1.0)`.
/// - Anything [`Game::ranked`](crate::Game::ranked) returns for a hypothetical
/// outcome.
pub fn expected_information_gain<T: Time, D: Drift<T>>(
teams: &[&[Rating<T, D>]],
options: &GameOptions,
) -> Result<f64, InferenceError> {
if teams.len() < 2 {
return Err(InferenceError::NotEnoughTeams { got: teams.len() });
}
if teams.len() > crate::MAX_PREDICTED_TEAMS {
return Err(InferenceError::TooManyTeams {
got: teams.len(),
max: crate::MAX_PREDICTED_TEAMS,
});
}
if !(0.0..1.0).contains(&options.p_draw) {
return Err(InferenceError::InvalidProbability {
value: options.p_draw,
});
}
for (idx, team) in teams.iter().enumerate() {
if team.is_empty() {
return Err(InferenceError::EmptyTeam { team: idx });
}
}
// Prediction runs on performances: skill inflated by each member's beta.
let performances: Vec<Gaussian> = teams
.iter()
.map(|team| {
team.iter()
.fold(crate::N00, |acc, rating| acc + rating.performance())
})
.collect();
// Draw margins per pair, derived from the teams' betas exactly as
// inference derives them, so the outcomes weighted here are the outcomes
// that would actually be fitted.
let beta_sq: Vec<f64> = teams
.iter()
.map(|team| team.iter().map(|r| r.beta().powi(2)).sum())
.collect();
let p_draw = options.p_draw;
let margins = predict::Margins::new(teams.len(), |i, j| {
if p_draw == 0.0 {
0.0
} else {
crate::compute_margin(p_draw, (beta_sq[i] + beta_sq[j]).sqrt())
}
});
let mut gain = 0.0;
for (ranks, probability) in predict::outcome_distribution(&performances, &margins) {
if probability <= NEGLIGIBLE {
continue;
}
let game = crate::Game::ranked(teams, Outcome::ranking(ranks), options)?;
let posteriors = game.posteriors();
// Beliefs factorise across competitors, so the joint divergence is the
// sum of the per-competitor ones.
let divergence: f64 = teams
.iter()
.zip(&posteriors)
.flat_map(|(team, posterior)| team.iter().zip(posterior))
.map(|(rating, &after)| kl_divergence(after, rating.prior()))
.sum();
gain += probability * divergence;
}
Ok(gain)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{BETA, ConstantDrift, GAMMA};
type R = Rating<i64, ConstantDrift>;
fn rating(mu: f64, sigma: f64) -> R {
R::new(Gaussian::from_ms(mu, sigma), BETA, ConstantDrift(GAMMA))
}
fn options(p_draw: f64) -> GameOptions {
GameOptions {
p_draw,
..GameOptions::default()
}
}
fn eig(teams: &[&[R]], p_draw: f64) -> f64 {
expected_information_gain(teams, &options(p_draw)).unwrap()
}
/// The analytic ceiling. Information gain is the mutual information between
/// the outcome and the skills, so it cannot exceed the entropy of the
/// outcome variable — whatever the ratings. This is the check a subtly
/// wrong implementation fails while still returning plausible numbers: an
/// early prototype of this returned 4.77 nats from a sign error and passed
/// every monotonicity test.
#[test]
fn never_exceeds_the_entropy_of_the_outcome() {
let ceiling_two = std::f64::consts::LN_2;
for (a, b) in [
(rating(0.0, 6.0), rating(0.0, 6.0)),
(rating(0.0, 0.5), rating(0.0, 0.5)),
(rating(12.0, 6.0), rating(-12.0, 6.0)),
(rating(40.0, 1.0), rating(-40.0, 1.0)),
(rating(3.0, 6.0), rating(-2.0, 0.1)),
(rating(0.0, 25.0), rating(0.0, 25.0)),
] {
let g = eig(&[&[a], &[b]], 0.0);
assert!(
g >= 0.0 && g <= ceiling_two,
"EIG {g} outside [0, ln 2] for mu=({}, {}) sigma=({}, {})",
a.prior().mu(),
b.prior().mu(),
a.prior().sigma(),
b.prior().sigma()
);
}
}
/// With draws enabled there are three outcomes, so the ceiling rises to
/// `ln 3` — and the two-outcome bound no longer applies.
#[test]
fn the_ceiling_follows_the_outcome_count() {
let ceiling_three = 3.0f64.ln();
for sigma in [0.5, 3.0, 6.0, 25.0] {
let g = eig(&[&[rating(0.0, sigma)], &[rating(0.0, sigma)]], 0.25);
assert!(
g >= 0.0 && g <= ceiling_three,
"EIG {g} outside [0, ln 3] at sigma {sigma}"
);
}
}
/// An even matchup between uncertain competitors is the informative one.
/// A hopelessly lopsided matchup teaches you almost nothing, because you
/// already know how it ends.
#[test]
fn an_even_matchup_beats_a_lopsided_one() {
let even = eig(&[&[rating(0.0, 6.0)], &[rating(0.0, 6.0)]], 0.0);
let lopsided = eig(&[&[rating(12.0, 6.0)], &[rating(-12.0, 6.0)]], 0.0);
assert!(
even > lopsided,
"even {even} should beat lopsided {lopsided}"
);
}
/// Certainty is the thing information gain is measuring the absence of:
/// the less you know, the more there is to learn.
#[test]
fn gain_falls_as_certainty_rises() {
let mut previous = f64::INFINITY;
for sigma in [12.0, 6.0, 3.0, 1.0, 0.5, 0.1] {
let g = eig(&[&[rating(0.0, sigma)], &[rating(0.0, sigma)]], 0.0);
assert!(
g < previous,
"sigma {sigma}: {g} did not fall below {previous}"
);
previous = g;
}
assert!(previous >= 0.0);
}
/// The heuristic this replaces is `quality * sigma_a^2 * sigma_b^2`. It is
/// not a monotone transform of information gain — it ranks a lopsided
/// matchup above a confident even one, and EIG ranks them the other way.
/// Pinning the disagreement down is what stops a future "simplification"
/// from quietly reverting to the heuristic.
#[test]
fn disagrees_with_the_quality_times_variance_heuristic() {
let heuristic = |a: &R, b: &R| {
crate::quality(&[&[a.prior()], &[b.prior()]], BETA)
* a.prior().sigma().powi(2)
* b.prior().sigma().powi(2)
};
let (confident_a, confident_b) = (rating(0.0, 0.5), rating(0.0, 0.5));
let (lopsided_a, lopsided_b) = (rating(12.0, 6.0), rating(-12.0, 6.0));
assert!(
heuristic(&lopsided_a, &lopsided_b) > heuristic(&confident_a, &confident_b),
"the heuristic should prefer the lopsided matchup"
);
assert!(
eig(&[&[confident_a], &[confident_b]], 0.0) > eig(&[&[lopsided_a], &[lopsided_b]], 0.0),
"information gain should prefer the even matchup"
);
}
#[test]
fn supports_more_than_two_teams() {
let teams: Vec<Vec<R>> = vec![
vec![rating(0.0, 6.0)],
vec![rating(0.0, 6.0)],
vec![rating(0.0, 6.0)],
];
let refs: Vec<&[R]> = teams.iter().map(Vec::as_slice).collect();
let g = expected_information_gain(&refs, &options(0.0)).unwrap();
// Six distinguishable orderings with no draws.
assert!(
g > 0.0 && g <= 6.0f64.ln(),
"three-team EIG {g} out of range"
);
}
#[test]
fn multi_member_teams_are_supported() {
let a = [rating(0.0, 6.0), rating(1.0, 4.0)];
let b = [rating(0.0, 6.0)];
let g = expected_information_gain(&[&a, &b], &options(0.0)).unwrap();
assert!(g > 0.0 && g <= std::f64::consts::LN_2, "{g}");
}
#[test]
fn degenerate_shapes_are_errors() {
let a = [rating(0.0, 6.0)];
assert!(matches!(
expected_information_gain(&[&a], &options(0.0)),
Err(InferenceError::NotEnoughTeams { got: 1 })
));
let empty: [R; 0] = [];
assert!(matches!(
expected_information_gain(&[&a, &empty], &options(0.0)),
Err(InferenceError::EmptyTeam { team: 1 })
));
assert!(matches!(
expected_information_gain(&[&a, &a], &options(1.5)),
Err(InferenceError::InvalidProbability { .. })
));
}
#[test]
fn kl_divergence_is_zero_for_identical_beliefs() {
let g = Gaussian::from_ms(3.0, 2.0);
assert!(kl_divergence(g, g).abs() < 1e-15);
}
#[test]
fn kl_divergence_is_non_negative_and_grows_with_separation() {
let prior = Gaussian::from_ms(0.0, 3.0);
let mut previous = 0.0;
for mu in [0.0, 0.5, 1.0, 2.0, 4.0] {
let d = kl_divergence(Gaussian::from_ms(mu, 3.0), prior);
assert!(d >= 0.0, "negative divergence at mu {mu}: {d}");
assert!(d >= previous, "not increasing at mu {mu}");
previous = d;
}
}
}
+80 -45
View File
@@ -538,10 +538,7 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
.sum() .sum()
} }
/// Each team's performance Gaussian, and its member count. /// Every team's member skills, validated.
///
/// Performance is skill inflated by `beta`: the question a prediction
/// answers is "how will they do today", not "how good are they".
/// ///
/// # Errors /// # Errors
/// ///
@@ -549,39 +546,60 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
/// reported rather than dropped — silently skipping them would turn a team /// reported rather than dropped — silently skipping them would turn a team
/// of strangers into a confident-looking prediction about nobody, which is /// of strangers into a confident-looking prediction about nobody, which is
/// the failure this replaced. /// the failure this replaced.
fn performances(&self, teams: &[&[&K]]) -> Result<(Vec<Gaussian>, Vec<usize>), InferenceError> { fn member_skills(&self, teams: &[&[&K]]) -> Result<Vec<Vec<Gaussian>>, InferenceError> {
if teams.len() < 2 { if teams.len() < 2 {
return Err(InferenceError::NotEnoughTeams { got: teams.len() }); return Err(InferenceError::NotEnoughTeams { got: teams.len() });
} }
let mut performances = Vec::with_capacity(teams.len()); let mut gathered = Vec::with_capacity(teams.len());
let mut sizes = Vec::with_capacity(teams.len());
for (team_idx, team) in teams.iter().enumerate() { for (team_idx, team) in teams.iter().enumerate() {
if team.is_empty() { if team.is_empty() {
return Err(InferenceError::EmptyTeam { team: team_idx }); return Err(InferenceError::EmptyTeam { team: team_idx });
} }
let mut total = crate::N00; let mut members = Vec::with_capacity(team.len());
for (member_idx, key) in team.iter().enumerate() { for (member_idx, key) in team.iter().enumerate() {
let unknown = InferenceError::UnknownKey { let unknown = InferenceError::UnknownKey {
team: team_idx, team: team_idx,
member: member_idx, member: member_idx,
}; };
let index = self.keys.get(*key).ok_or(unknown.clone())?; let index = self.keys.get(*key).ok_or(unknown.clone())?;
let skill = self members.push(
.time_slices self.time_slices
.iter() .iter()
.rev() .rev()
.find_map(|ts| ts.skills.get(index).map(|s| s.posterior())) .find_map(|ts| ts.skills.get(index).map(|s| s.posterior()))
.ok_or(unknown)?; .ok_or(unknown)?,
total = total + skill.forget(self.beta.powi(2)); );
} }
performances.push(total); gathered.push(members);
sizes.push(team.len());
} }
Ok(gathered)
}
/// Each team's performance Gaussian, and its member count.
///
/// Performance is skill inflated by `beta`: the question a prediction
/// answers is "how will they do today", not "how good are they".
///
/// # Errors
///
/// As [`History::member_skills`].
fn performances(&self, teams: &[&[&K]]) -> Result<(Vec<Gaussian>, Vec<usize>), InferenceError> {
let skills = self.member_skills(teams)?;
let performances = skills
.iter()
.map(|team| {
team.iter()
.fold(crate::N00, |acc, s| acc + s.forget(self.beta.powi(2)))
})
.collect();
let sizes = skills.iter().map(Vec::len).collect();
Ok((performances, sizes)) Ok((performances, sizes))
} }
@@ -618,38 +636,55 @@ impl<T: Time, D: Drift<T>, O: Observer<T>, K: Eq + Hash + Clone> History<T, D, O
/// ///
/// `NotEnoughTeams`, `EmptyTeam`, or `UnknownKey`. /// `NotEnoughTeams`, `EmptyTeam`, or `UnknownKey`.
pub fn predict_quality(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> { pub fn predict_quality(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> {
let mut groups: Vec<Vec<Gaussian>> = Vec::with_capacity(teams.len()); let groups = self.member_skills(teams)?;
for (team_idx, team) in teams.iter().enumerate() {
if team.is_empty() {
return Err(InferenceError::EmptyTeam { team: team_idx });
}
let mut members = Vec::with_capacity(team.len());
for (member_idx, key) in team.iter().enumerate() {
let unknown = InferenceError::UnknownKey {
team: team_idx,
member: member_idx,
};
let index = self.keys.get(*key).ok_or(unknown.clone())?;
members.push(
self.time_slices
.iter()
.rev()
.find_map(|ts| ts.skills.get(index).map(|s| s.posterior()))
.ok_or(unknown)?,
);
}
groups.push(members);
}
if groups.len() < 2 {
return Err(InferenceError::NotEnoughTeams { got: groups.len() });
}
let group_refs: Vec<&[Gaussian]> = groups.iter().map(Vec::as_slice).collect(); let group_refs: Vec<&[Gaussian]> = groups.iter().map(Vec::as_slice).collect();
Ok(crate::quality(&group_refs, self.beta)) Ok(crate::quality(&group_refs, self.beta))
} }
/// Expected information gain of running this matchup, in nats.
///
/// Answers "which comparison should I run next" rather than "who will
/// win": the outcome-weighted divergence between current beliefs and the
/// beliefs each possible result would produce. Higher means the result
/// would teach you more.
///
/// Uses each competitor's current skill as the prior, and the history's
/// own `beta`, `drift` and `p_draw`, so the outcomes weighted here are the
/// ones that would actually be fitted if the matchup were played and
/// recorded.
///
/// Distinct from [`History::predict_quality`], which measures *fairness*.
/// The two coincide for two evenly matched competitors and diverge
/// elsewhere. See [`expected_information_gain`](crate::expected_information_gain)
/// for the scale, the analytic `ln k` ceiling, and the cost.
///
/// # Errors
///
/// As [`History::member_skills`], plus `TooManyTeams` and anything
/// inference returns for a hypothetical outcome.
pub fn expected_information_gain(&self, teams: &[&[&K]]) -> Result<f64, InferenceError> {
let skills = self.member_skills(teams)?;
let ratings: Vec<Vec<Rating<T, D>>> = skills
.iter()
.map(|team| {
team.iter()
.map(|&skill| Rating::new(skill, self.beta, self.drift))
.collect()
})
.collect();
let team_refs: Vec<&[Rating<T, D>]> = ratings.iter().map(Vec::as_slice).collect();
crate::expected_information_gain(
&team_refs,
&crate::GameOptions {
p_draw: self.p_draw,
score_sigma: self.score_sigma,
convergence: self.convergence,
},
)
}
/// `P(team i finishes strictly first)`, for every team. /// `P(team i finishes strictly first)`, for every team.
/// ///
/// Supports any number of teams. Because performances are independent /// Supports any number of teams. Because performances are independent
+2
View File
@@ -110,6 +110,7 @@ pub(crate) mod arena;
mod time; mod time;
mod time_slice; mod time_slice;
pub use time_slice::{EventKind, TimeSlice}; pub use time_slice::{EventKind, TimeSlice};
mod acquisition;
mod color_group; mod color_group;
mod competitor; mod competitor;
mod convergence; mod convergence;
@@ -132,6 +133,7 @@ mod rating;
pub(crate) mod schedule; pub(crate) mod schedule;
pub mod storage; pub mod storage;
pub use acquisition::expected_information_gain;
pub use competitor::Competitor; pub use competitor::Competitor;
pub use convergence::{ConvergenceOptions, ConvergenceReport}; pub use convergence::{ConvergenceOptions, ConvergenceReport};
pub use drift::{ConstantDrift, Drift}; pub use drift::{ConstantDrift, Drift};
+77
View File
@@ -212,3 +212,80 @@ fn team_size_affects_the_prediction() {
assert!((p.total() - 1.0).abs() < 1e-6, "total = {}", p.total()); assert!((p.total() - 1.0).abs() < 1e-6, "total = {}", p.total());
assert!(p.probability_of(&[0, 0]) > 0.0); assert!(p.probability_of(&[0, 0]) > 0.0);
} }
// ---------------------------------------------------------------------------
// Expected information gain
// ---------------------------------------------------------------------------
/// The whole point of #39: "which comparison should I run next?" is a
/// different question from "who will win?" or "is this fair?".
#[test]
fn information_gain_prefers_the_uncertain_pairing() {
let mut h = History::builder().build();
// "known" and "rival" have played a lot; "newcomer" has played once.
for t in 1..=15 {
h.record_winner(&"known", &"rival", t).unwrap();
h.record_winner(&"rival", &"known", t + 100).unwrap();
}
h.record_winner(&"known", &"newcomer", 500).unwrap();
h.converge().unwrap();
let settled = h
.expected_information_gain(&[&[&"known"], &[&"rival"]])
.unwrap();
let unknown = h
.expected_information_gain(&[&[&"known"], &[&"newcomer"]])
.unwrap();
assert!(
unknown > settled,
"pairing against the newcomer should teach more: {unknown} vs {settled}"
);
}
/// The analytic ceiling, through the `History` entry point rather than the
/// standalone one.
#[test]
fn information_gain_respects_the_entropy_ceiling() {
let h = history_with(&["a", "b", "c"], 0.0);
let two = h.expected_information_gain(&[&[&"a"], &[&"b"]]).unwrap();
assert!(
(0.0..=std::f64::consts::LN_2).contains(&two),
"two-team EIG {two} outside [0, ln 2]"
);
let three = h
.expected_information_gain(&[&[&"a"], &[&"b"], &[&"c"]])
.unwrap();
assert!(
(0.0..=6.0f64.ln()).contains(&three),
"three-team EIG {three} outside [0, ln 6]"
);
}
#[test]
fn information_gain_reports_unknown_keys() {
let h = history_with(&["a", "b"], 0.0);
assert_eq!(
h.expected_information_gain(&[&[&"a"], &[&"ghost"]])
.unwrap_err(),
InferenceError::UnknownKey { team: 1, member: 0 }
);
}
/// A draw-enabled history has three outcomes to weigh rather than two, so the
/// draw branch must actually be reachable through this path.
#[test]
fn information_gain_accounts_for_draws() {
let with_draws = history_with(&["a", "b"], 0.25);
let g = with_draws
.expected_information_gain(&[&[&"a"], &[&"b"]])
.unwrap();
assert!(g > 0.0 && g <= 3.0f64.ln(), "{g}");
// The draw outcome carries mass, so it is genuinely being weighed.
let dist = with_draws.predict_outcome(&[&[&"a"], &[&"b"]]).unwrap();
assert!(dist.probability_of(&[0, 0]) > 0.0);
}