test: calibrate the marginals against the exact posterior
Investigation for #46 and #47, before touching either. A scored history is linear-Gaussian, so its true joint posterior has a closed form and the crate can be checked against ground truth. Measured on five competitors: means marginal sd (crate / exact) tree (star) exact 1.000 loopy (robin) exact 0.502 On a tree the crate is exact in both. With cycles the means stay exact — the standard Gaussian-BP result, and the property ratings rely on — while marginal variances come out about half the true width. That is the opposite direction from what #47 reports, so whatever is happening in that consumer's model, the crate being conservative is not it. It also means #46 cannot be implemented as an added covariance accessor. The exact correlation between two nodes here is +0.857, so ignoring it overstates the width of a difference — but the too-narrow marginals partially cancel that, leaving 1.327x rather than 2.646x. Adding true correlations to these marginals without correcting them would give 0.765 against a true 1.524: overconfident, which is the direction the reporter specifically called unsafe. Pins the two real invariants (exactness on a tree, exact means with cycles) and deliberately only records the variance gap, since closing it is what #46 proposes. Also records the working rules this project has converged on: investigate before implementing, fix the root issue, and scout crates.io on measured accuracy rather than adoption. Refs #46, #47 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
@@ -24,6 +24,20 @@ is where several defects have hidden — a debug-only run is not evidence.
|
||||
- `approx` — `approx::AbsDiffEq` etc. for `Gaussian`. Most numerical goldens need it.
|
||||
- `rayon` — opt-in parallel within-slice sweep and per-slice query passes.
|
||||
|
||||
## Working rules
|
||||
|
||||
- **Investigate before implementing.** Measure the actual behaviour first —
|
||||
against an analytic reference where one exists. Several "obvious" fixes in
|
||||
this repo turned out to be wrong in sign or unnecessary, and the measurement
|
||||
is what caught them.
|
||||
- **Fix the root issue, not the symptom.** A clamp that hides an underflow, or
|
||||
a tolerance loosened to make a test pass, is a defect deferred.
|
||||
- **Scout crates.io before hand-rolling numerics.** Check accuracy against an
|
||||
independent reference rather than trusting downloads: `puruspe` has 1.4M
|
||||
downloads and is 346 ULP off in the tail, where `libm` is 1. Fewer
|
||||
dependencies is preferable, not mandatory — take the dependency when it is
|
||||
measurably better.
|
||||
|
||||
## Architecture
|
||||
|
||||
A Rust port of [TrueSkillThroughTime.py](https://github.com/glandfried/TrueSkillThroughTime.py):
|
||||
|
||||
Reference in New Issue
Block a user