predict_outcome returns probabilities greater than 1: the grid clamps with no detection #55

New Issue

2026-09-09T14:50:24Z

logaritmisk commented

2026-09-09 14:50:24 +00:00

src/predict.rs:184. grid_shape asks for 12 nodes per narrowest sigma and then clamps to MAX_GRID_POINTS = 262_144 with no detection that the request was not met. Once step / sigma_min exceeds about 1.7 the trapezoid rule stops resolving the density and the result is unbounded.

Measured on outcome_distribution, two teams, mu = (1, 0), sigma_b = 25:

 sigma_a   step/sig_a   P(a first)     exact        total
 2.0e-3      0.86       0.515953      0.515953    1.000000
 1.0e-3      1.72       0.517185      0.515953    1.002388
 3.0e-4      5.72       1.147353      0.515953    2.223758
 1.0e-4     17.17       2.791336      0.515953    5.410065
 5.0e-5     34.33       2.752306      0.515953    5.334418

A probability of 2.79 and a Prediction::total() of 5.41. The reference agrees to 18 digits from two independent routes (closed form, and mpmath quad at 40 dps).

Reachable through the public API, on a documented pattern

History::predict_outcome, with a pinned reference competitor — Member::with_prior(Gaussian::from_ms(1.0, 1e-5)), the "bot at a known strength" the crate documents as supported:

beta=1e-5, pinned sigma=1e-5   (performance sigma ratio 9.98e4)
  predict_outcome   P(bot first) = 0.016619953553   total = 0.022513522543
  predict_win_prob  P(bot first) = 0.738228412330
  exact closed form              = 0.738228412328

The two public predictors disagree by 44x on identical inputs, and predict_outcome is the wrong one. predict_win_probabilities goes through adaptive Gauss-Kronrod and is accurate to 1e-13 on these very inputs, so the defect is the grid path specifically.

The failure is not monotone: 0.738 → 2.79 → 0.017 depending on where the grid nodes land relative to the spike. No smooth sanity check catches it.

Onset measured at a performance-sigma ratio of roughly 1.5e4; materially wrong from 2.5e4; catastrophic from 8e4.

The cliff is sharp because trapezoid error on an analytic Gaussian is ~exp(-2*pi^2*(sigma/h)^2): 1e-12 at h/sigma = 0.86, 1e-3 at 1.7, O(1) at 17. The clamp puts the code on the wrong side of it silently.

Why the existing checks cannot see it

Prediction::total() is the crate's strongest self-check and it is structurally blind here — the same grid produces every entry, so they stay mutually consistent while being individually wrong. Note total is 5.41 in the worst row, so a total ≈ 1 assertion would catch this one — but the companion defect in win_probabilities (see the sibling issue on the quadrature tolerance floor) keeps total = 0.999999999999996 while individual probabilities collapse to 0.0. Between them, no normalisation check is sufficient.

ranking_probability shares the same Sampled grid and matches outcome_distribution to 0.00e0, so the agreement test between them cannot detect this either.

Fix

grid_shape must report, not silently clamp. Either return an error when the requested resolution exceeds MAX_GRID_POINTS, or fall back to the adaptive quadrature path that win_probabilities already uses and that is measured accurate in exactly this regime.

`src/predict.rs:184`. `grid_shape` asks for 12 nodes per narrowest sigma and then clamps to `MAX_GRID_POINTS = 262_144` **with no detection that the request was not met**. Once `step / sigma_min` exceeds about 1.7 the trapezoid rule stops resolving the density and the result is unbounded. Measured on `outcome_distribution`, two teams, `mu = (1, 0)`, `sigma_b = 25`: ``` sigma_a step/sig_a P(a first) exact total 2.0e-3 0.86 0.515953 0.515953 1.000000 1.0e-3 1.72 0.517185 0.515953 1.002388 3.0e-4 5.72 1.147353 0.515953 2.223758 1.0e-4 17.17 2.791336 0.515953 5.410065 5.0e-5 34.33 2.752306 0.515953 5.334418 ``` **A probability of 2.79 and a `Prediction::total()` of 5.41.** The reference agrees to 18 digits from two independent routes (closed form, and mpmath `quad` at 40 dps). ## Reachable through the public API, on a documented pattern `History::predict_outcome`, with a pinned reference competitor — `Member::with_prior(Gaussian::from_ms(1.0, 1e-5))`, the "bot at a known strength" the crate documents as supported: ``` beta=1e-5, pinned sigma=1e-5 (performance sigma ratio 9.98e4) predict_outcome P(bot first) = 0.016619953553 total = 0.022513522543 predict_win_prob P(bot first) = 0.738228412330 exact closed form = 0.738228412328 ``` The two public predictors disagree by **44x** on identical inputs, and `predict_outcome` is the wrong one. `predict_win_probabilities` goes through adaptive Gauss-Kronrod and is accurate to 1e-13 on these very inputs, so the defect is the grid path specifically. The failure is **not monotone**: 0.738 → 2.79 → 0.017 depending on where the grid nodes land relative to the spike. No smooth sanity check catches it. Onset measured at a performance-sigma ratio of roughly **1.5e4**; materially wrong from 2.5e4; catastrophic from 8e4. The cliff is sharp because trapezoid error on an analytic Gaussian is `~exp(-2*pi^2*(sigma/h)^2)`: 1e-12 at `h/sigma = 0.86`, 1e-3 at 1.7, O(1) at 17. The clamp puts the code on the wrong side of it silently. ## Why the existing checks cannot see it `Prediction::total()` is the crate's strongest self-check and it is **structurally blind** here — the same grid produces every entry, so they stay mutually consistent while being individually wrong. Note `total` is 5.41 in the worst row, so a `total ≈ 1` assertion *would* catch this one — but the companion defect in `win_probabilities` (see the sibling issue on the quadrature tolerance floor) keeps `total = 0.999999999999996` while individual probabilities collapse to `0.0`. Between them, no normalisation check is sufficient. `ranking_probability` shares the same `Sampled` grid and matches `outcome_distribution` to `0.00e0`, so the agreement test between them cannot detect this either. ## Fix `grid_shape` must report, not silently clamp. Either return an error when the requested resolution exceeds `MAX_GRID_POINTS`, or fall back to the adaptive quadrature path that `win_probabilities` already uses and that is measured accurate in exactly this regime. ## Related `expected_information_gain` inherits this — it weights `gain += probability * divergence` by these probabilities and consequently exceeds its own `ln 2` ceiling. Filed separately since it is a free function needing no fixture. Found by a floating-point audit, 2026-09-09.

logaritmisk referenced this issue

2026-09-09 14:50:43 +00:00

`expected_information_gain` returns 4.67x its own `ln k` ceiling #56

logaritmisk added the bug numerics labels 2026-09-09 14:53:39 +00:00

logaritmisk referenced this issue from a commit

2026-09-09 16:10:51 +00:00

fix!: report an unresolvable prediction grid instead of clamping

logaritmisk closed this issue

2026-09-09 16:10:51 +00:00

logaritmisk referenced this issue from a commit

2026-09-09 16:10:52 +00:00

Merge branch 'fix/numerics-critical'