No public way to ask "what is P(skill below x)?" — cdf is pub(crate), so callers approximate it with a z-multiplier #43

New Issue

2026-09-07T21:20:04Z

logaritmisk commented

2026-09-07 21:20:04 +00:00

The question a caller cannot ask

lester-stash is building a cull workflow whose entire decision rule is one question about a posterior:

What is the probability this scene's true skill lies below the cutoff?

The crate computes exactly that internally — cdf(x, mu, sigma) at src/lib.rs:302 — but it is pub(crate), so downstream code cannot reach it. pdf (lib.rs:417) is private too.

What we do instead, and why it is worse

pub fn cull_decision(mu: f64, sigma: f64, cutoff: f64, z: f64) -> Decision {
    if mu + z * sigma < cutoff      { Decision::Cull }
    else if mu - z * sigma > cutoff { Decision::Keep }
    else                            { Decision::Undecided }
}

A z-multiplier stand-in for a probability. It works, but z is an opaque knob: a sweep at 1.5 / 1.8 / 2.0 / 2.2 produced precision 0.918 / 0.926 / 0.941 / 0.939, and there is no way to say what confidence any of those settings actually buys. The user-facing question is "how sure do you want to be before deleting something", and the parameter cannot express it.

With a public CDF the rule becomes what it always meant:

if posterior.probability_below(cutoff) > 0.95 { Decision::Cull }

The threshold is then a stated probability rather than a magic number, and it stays meaningful as sigma changes — a z-band silently means different confidence at different uncertainties, which is precisely the regime a stopping rule operates in.

Suggested shape

A method on Gaussian reads better than a free function, since the caller already holds the posterior:

impl Gaussian {
    /// `P(X < x)` under this Gaussian.
    #[must_use]
    pub fn probability_below(&self, x: f64) -> f64;

    /// `P(X > x)`. Equivalent to `1.0 - probability_below(x)`, but avoids
    /// cancellation in the far tail.
    #[must_use]
    pub fn probability_above(&self, x: f64) -> f64;
}

Exposing the free cdf/pdf would also work and is a smaller change; the method form just avoids callers re-deriving (mu, sigma) from a Gaussian only to hand them straight back.

Why this is a fair ask rather than scope creep

The maths is already implemented, tested and now routed through libm (0.4.2) — this is visibility, not new numerics.
It is the natural complement to what 0.4 already added. expected_information_gain answers "which comparison should I run"; this answers "have I learned enough to stop". A crate that helps you choose matchups but not to stop running them is only half of an active-learning loop.
Every alternative is worse: reimplementing erfc downstream duplicates the precision work 0.4.2 just did, and Gaussian::forget/exclude being pub(crate) blocks deriving it from the existing arithmetic.

Note on the tail

probability_above is listed separately on purpose. 1 - cdf loses precision in the upper tail, and a stopping rule is evaluated exactly there — the interesting cases are the ones near certainty. If only one method lands, the docs should say which tail it is accurate in.

## The question a caller cannot ask lester-stash is building a cull workflow whose entire decision rule is one question about a posterior: > What is the probability this scene's true skill lies below the cutoff? The crate computes exactly that internally — `cdf(x, mu, sigma)` at `src/lib.rs:302` — but it is `pub(crate)`, so downstream code cannot reach it. `pdf` (`lib.rs:417`) is private too. ## What we do instead, and why it is worse ```rust pub fn cull_decision(mu: f64, sigma: f64, cutoff: f64, z: f64) -> Decision { if mu + z * sigma < cutoff { Decision::Cull } else if mu - z * sigma > cutoff { Decision::Keep } else { Decision::Undecided } } ``` A z-multiplier stand-in for a probability. It works, but `z` is an opaque knob: a sweep at 1.5 / 1.8 / 2.0 / 2.2 produced precision 0.918 / 0.926 / 0.941 / 0.939, and there is no way to say what confidence any of those settings actually buys. The user-facing question is "how sure do you want to be before deleting something", and the parameter cannot express it. With a public CDF the rule becomes what it always meant: ```rust if posterior.probability_below(cutoff) > 0.95 { Decision::Cull } ``` The threshold is then a stated probability rather than a magic number, and it stays meaningful as sigma changes — a z-band silently means different confidence at different uncertainties, which is precisely the regime a stopping rule operates in. ## Suggested shape A method on `Gaussian` reads better than a free function, since the caller already holds the posterior: ```rust impl Gaussian { /// `P(X < x)` under this Gaussian. #[must_use] pub fn probability_below(&self, x: f64) -> f64; /// `P(X > x)`. Equivalent to `1.0 - probability_below(x)`, but avoids /// cancellation in the far tail. #[must_use] pub fn probability_above(&self, x: f64) -> f64; } ``` Exposing the free `cdf`/`pdf` would also work and is a smaller change; the method form just avoids callers re-deriving `(mu, sigma)` from a `Gaussian` only to hand them straight back. ## Why this is a fair ask rather than scope creep - The maths is already implemented, tested and now routed through libm (0.4.2) — this is visibility, not new numerics. - It is the natural complement to what 0.4 already added. `expected_information_gain` answers "which comparison should I run"; this answers "have I learned enough to stop". A crate that helps you choose matchups but not to stop running them is only half of an active-learning loop. - Every alternative is worse: reimplementing erfc downstream duplicates the precision work 0.4.2 just did, and `Gaussian::forget`/`exclude` being `pub(crate)` blocks deriving it from the existing arithmetic. ## Note on the tail `probability_above` is listed separately on purpose. `1 - cdf` loses precision in the upper tail, and a stopping rule is evaluated exactly there — the interesting cases are the ones near certainty. If only one method lands, the docs should say which tail it is accurate in.

logaritmisk referenced this issue from a commit

2026-09-07 21:41:11 +00:00

feat!: name the unknown key, expose tail probabilities, flag short fits

logaritmisk closed this issue

2026-09-07 21:41:11 +00:00

logaritmisk added the api enhancement labels 2026-09-07 21:41:37 +00:00

logaritmisk commented

2026-09-08 05:47:10 +00:00

Adopted in lester-stash on 0.5.0 — thank you. One correction to this issue, since the reasoning in it was partly wrong and I'd rather not leave that standing.

The claim that a z-band "silently means different confidence at different uncertainties" is false. For a Gaussian posterior the two forms are algebraically identical:

mu + z*sigma < cutoff   <=>   z < (cutoff - mu)/sigma   <=>   phi(z) < P(X < cutoff)

So mu + z*sigma < cutoff is P(X < cutoff) > phi(z), at every sigma. The z-rule was already a confidence rule; it just spelled the confidence as a multiplier. I asserted otherwise and it was the main argument in the issue.

I pinned this rather than just noting it — a grid over z ∈ {1.0, 1.5, 1.8, 2.0, 2.2, 3.0} × sigma ∈ {0.5 … 40} × 101 means, ~3600 comparisons, asserting the new form with confidence = phi(z) picks exactly what the old margin rule picked.

One real difference did show up, at exact ties. Where mu ± z*sigma == cutoff the two disagree in the last bits, because Gaussian::from_ms round-trips mu and sigma through 1/sigma^2 while the margin form compares in mu-space directly. Measure-zero and not worth defending against — the grid skips ties — but worth knowing if anyone else pins this.

The change was still worth making, for the reason I put third rather than first: the knob is now interpretable. lester's cull UI has to tell someone how sure the system is before it deletes their file, and "at least 95% sure it's below the cutoff" is a sentence; "z = 1.5" is not. The default is 0.9332 = phi(1.5), which reproduces every prior measurement exactly.

probability_above's survival-function implementation is also the right call, though for this particular rule it changes no decisions — by the time the tail underflows, both forms have long since agreed.

The parts of the issue that held up: the maths was already implemented and tested, and this is the natural complement to expected_information_gain — choosing matchups without being able to stop is half a loop.

Adopted in lester-stash on 0.5.0 — thank you. One correction to this issue, since the reasoning in it was partly wrong and I'd rather not leave that standing. **The claim that a z-band "silently means different confidence at different uncertainties" is false.** For a Gaussian posterior the two forms are algebraically identical: ``` mu + z*sigma < cutoff <=> z < (cutoff - mu)/sigma <=> phi(z) < P(X < cutoff) ``` So `mu + z*sigma < cutoff` *is* `P(X < cutoff) > phi(z)`, at every sigma. The z-rule was already a confidence rule; it just spelled the confidence as a multiplier. I asserted otherwise and it was the main argument in the issue. I pinned this rather than just noting it — a grid over z ∈ {1.0, 1.5, 1.8, 2.0, 2.2, 3.0} × sigma ∈ {0.5 … 40} × 101 means, ~3600 comparisons, asserting the new form with `confidence = phi(z)` picks exactly what the old margin rule picked. **One real difference did show up, at exact ties.** Where `mu ± z*sigma == cutoff` the two disagree in the last bits, because `Gaussian::from_ms` round-trips mu and sigma through `1/sigma^2` while the margin form compares in mu-space directly. Measure-zero and not worth defending against — the grid skips ties — but worth knowing if anyone else pins this. **The change was still worth making, for the reason I put third rather than first:** the knob is now interpretable. lester's cull UI has to tell someone how sure the system is before it deletes their file, and "at least 95% sure it's below the cutoff" is a sentence; "z = 1.5" is not. The default is 0.9332 = phi(1.5), which reproduces every prior measurement exactly. `probability_above`'s survival-function implementation is also the right call, though for this particular rule it changes no decisions — by the time the tail underflows, both forms have long since agreed. The parts of the issue that held up: the maths was already implemented and tested, and this is the natural complement to `expected_information_gain` — choosing matchups without being able to stop is half a loop.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logaritmisk/trueskill-tt#43

No public way to ask "what is P(skill below x)?" — cdf is pub(crate), so callers approximate it with a z-multiplier #43

The question a caller cannot ask

What we do instead, and why it is worse

Suggested shape

Why this is a fair ask rather than scope creep

Note on the tail

No public way to ask "what is P(skill below x)?" — `cdf` is pub(crate), so callers approximate it with a z-multiplier #43