expected_information_gain is discrete-outcome only, so scored consumers cannot ask "which matchup next?" #49

New Issue

2026-09-07T21:27:06Z

logaritmisk commented

2026-09-07 21:27:06 +00:00

Follow-up to #39, which asked for active-learning primitives and was closed by expected_information_gain. That call enumerates discrete outcomes — its own docs give the ceiling as "ln 2 ≈ 0.693 for a two-way result, ln 3 once draws are possible", and it costs one full inference pass per possible outcome.

A consumer recording Outcome::scores_with_sigma has no discrete outcome to enumerate. The observation is a continuous score, so #39's question is still unanswerable for that half of the API.

Why it is worth having

This is the difference between a caveat and advice.

ustat publishes a block whose entire job is to contradict its own standings: pairs of players whose rating gap is smaller than the uncertainty in that gap. It currently says, correctly, "we cannot order these two, and the fix is more rounds from either of them" — because every competitor duels the course rather than another player, so the pair need not play together to become separable.

What it cannot say is which round would help most. That is exactly expected information gain, and for a scored model it should be cheaper than the ranked version rather than more expensive: with a Gaussian observation the posterior update is closed-form, so the expected reduction in variance for a target functional ought to be computable without enumerating anything.

Concretely, the useful question is not "how much would this event teach me in general" but "how much would it reduce the variance of this quantity" — where the quantity is usually a difference between two competitors:

/// Expected reduction in the variance of a target functional, were this
/// scored matchup observed.
fn expected_variance_reduction<Q>(
    &self,
    teams: &[&[&Q]],
    target: &[(&Q, f64)],   // the same linear functional as #46
    options: &GameOptions,
) -> Result<f64, InferenceError>;

That target is deliberately the linear-functional shape proposed in #46, so the two share a concept rather than inventing two.

Honest caveat

The value of this is lower than #46 and #48 — those two fix numbers that are currently wrong, whereas this adds a capability. It is also the one most likely to be a research question rather than an afternoon's work, since "expected variance reduction under EP" may not have a clean closed form even for Gaussian likelihoods. Filing it so the gap left by #39 is recorded, not to suggest it is next.

Follow-up to #39, which asked for active-learning primitives and was closed by `expected_information_gain`. That call enumerates **discrete outcomes** — its own docs give the ceiling as "`ln 2 ≈ 0.693` for a two-way result, `ln 3` once draws are possible", and it costs one full inference pass per possible outcome. A consumer recording `Outcome::scores_with_sigma` has no discrete outcome to enumerate. The observation is a continuous score, so #39's question is still unanswerable for that half of the API. ## Why it is worth having This is the difference between a caveat and advice. `ustat` publishes a block whose entire job is to contradict its own standings: pairs of players whose rating gap is smaller than the uncertainty in that gap. It currently says, correctly, "we cannot order these two, and the fix is more rounds from either of them" — because every competitor duels the *course* rather than another player, so the pair need not play together to become separable. What it cannot say is **which** round would help most. That is exactly expected information gain, and for a scored model it should be cheaper than the ranked version rather than more expensive: with a Gaussian observation the posterior update is closed-form, so the expected reduction in variance for a target functional ought to be computable without enumerating anything. Concretely, the useful question is not "how much would this event teach me in general" but "how much would it reduce the variance of *this* quantity" — where the quantity is usually a difference between two competitors: ```rust /// Expected reduction in the variance of a target functional, were this /// scored matchup observed. fn expected_variance_reduction<Q>( &self, teams: &[&[&Q]], target: &[(&Q, f64)], // the same linear functional as #46 options: &GameOptions, ) -> Result<f64, InferenceError>; ``` That `target` is deliberately the linear-functional shape proposed in #46, so the two share a concept rather than inventing two. ## Honest caveat The value of this is lower than #46 and #48 — those two fix numbers that are currently *wrong*, whereas this adds a capability. It is also the one most likely to be a research question rather than an afternoon's work, since "expected variance reduction under EP" may not have a clean closed form even for Gaussian likelihoods. Filing it so the gap left by #39 is recorded, not to suggest it is next.

logaritmisk added the api enhancement labels 2026-09-07 21:27:06 +00:00

logaritmisk referenced this issue from a commit

2026-09-08 00:08:36 +00:00

feat: add expected_variance_reduction for scored active learning

logaritmisk closed this issue

2026-09-08 00:08:36 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logaritmisk/trueskill-tt#49

expected_information_gain is discrete-outcome only, so scored consumers cannot ask "which matchup next?" #49

Why it is worth having

Honest caveat

`expected_information_gain` is discrete-outcome only, so scored consumers cannot ask "which matchup next?" #49