Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b113385c6f | ||
|
|
f345e7690e | ||
|
|
d9e85cda1d | ||
|
|
633a503900 | ||
|
|
7cf45db5cf | ||
|
|
c866210c65 | ||
|
|
1f791bcddd | ||
|
|
c52e2550af | ||
|
|
4924bc8b57 | ||
|
|
36eacf5f67 | ||
|
|
71554fd944 | ||
|
|
2cf21a753d | ||
|
|
35d7512557 | ||
|
|
e1bddf2474 | ||
|
|
5d36fa1008 | ||
|
|
c12bc830a5 | ||
|
|
901f60972e | ||
|
|
8116fd081f | ||
|
|
17d072b2ae | ||
|
|
3dd659307a | ||
|
|
7e289ee834 | ||
|
|
564969ee5d | ||
|
|
683813ec10 |
@@ -2,6 +2,78 @@
|
|||||||
|
|
||||||
All notable changes to this project will be documented in this file.
|
All notable changes to this project will be documented in this file.
|
||||||
|
|
||||||
|
## 0.6.0 - 2026-09-08
|
||||||
|
|
||||||
|
### Breaking Changes
|
||||||
|
|
||||||
|
- fix!: make the joint span slices, not just the latest one
|
||||||
|
|
||||||
|
## 0.5.0 - 2026-09-08
|
||||||
|
|
||||||
|
### Breaking Changes
|
||||||
|
|
||||||
|
- feat!: name the unknown key, expose tail probabilities, flag short fits
|
||||||
|
- refactor!: remove the factor-graph surface nothing used, add try_winner
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
- fix(test): the ingestion-order property was comparing two truncated fits
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
- docs: record that the event log is the source of truth, and why
|
||||||
|
|
||||||
|
### Features
|
||||||
|
|
||||||
|
- feat: add UnknownKeys::Prior, and explain why there is no Skip
|
||||||
|
- feat: add History::posterior_of for a linear combination of competitors
|
||||||
|
- feat: add History::predict_margin for scored matchups
|
||||||
|
- feat: add expected_variance_reduction for scored active learning
|
||||||
|
|
||||||
|
### Miscellaneous Tasks
|
||||||
|
|
||||||
|
- chore: Release trueskill-tt version 0.5.0
|
||||||
|
|
||||||
|
### Styling
|
||||||
|
|
||||||
|
- style: factor the event-pair type out of the reconvergence fixture
|
||||||
|
- style: use arrays rather than vec! in the calibration fixture
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
- test: pin that re-convergence is path-independent
|
||||||
|
- test: calibrate the marginals against the exact posterior
|
||||||
|
- test: pin what an additive model does to combined uncertainty
|
||||||
|
|
||||||
|
## 0.4.2 - 2026-09-07
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
- fix: replace the erfc approximation with libm, for free
|
||||||
|
- fix: route every transcendental through libm, and combine sigmas with hypot
|
||||||
|
|
||||||
|
### Miscellaneous Tasks
|
||||||
|
|
||||||
|
- chore: Release trueskill-tt version 0.4.2
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
- test: localise the erfc_inv tail residual to the caller's argument
|
||||||
|
|
||||||
|
## 0.4.1 - 2026-09-07
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
- fix: correct erfc_inv's sign error and keep evidence in log space
|
||||||
|
|
||||||
|
### Miscellaneous Tasks
|
||||||
|
|
||||||
|
- chore: Release trueskill-tt version 0.4.1
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
- test: pin quality()'s N-group closed form, closing the README cross-check
|
||||||
|
|
||||||
## 0.4.0 - 2026-09-07
|
## 0.4.0 - 2026-09-07
|
||||||
|
|
||||||
### Breaking Changes
|
### Breaking Changes
|
||||||
@@ -25,6 +97,10 @@ All notable changes to this project will be documented in this file.
|
|||||||
- feat: add expected information gain for active matchup selection
|
- feat: add expected information gain for active matchup selection
|
||||||
- feat: let observers be shared, boxed, or borrowed
|
- feat: let observers be shared, boxed, or borrowed
|
||||||
|
|
||||||
|
### Miscellaneous Tasks
|
||||||
|
|
||||||
|
- chore: Release trueskill-tt version 0.4.0
|
||||||
|
|
||||||
## 0.3.0 - 2026-09-01
|
## 0.3.0 - 2026-09-01
|
||||||
|
|
||||||
### Breaking Changes
|
### Breaking Changes
|
||||||
|
|||||||
@@ -24,6 +24,20 @@ is where several defects have hidden — a debug-only run is not evidence.
|
|||||||
- `approx` — `approx::AbsDiffEq` etc. for `Gaussian`. Most numerical goldens need it.
|
- `approx` — `approx::AbsDiffEq` etc. for `Gaussian`. Most numerical goldens need it.
|
||||||
- `rayon` — opt-in parallel within-slice sweep and per-slice query passes.
|
- `rayon` — opt-in parallel within-slice sweep and per-slice query passes.
|
||||||
|
|
||||||
|
## Working rules
|
||||||
|
|
||||||
|
- **Investigate before implementing.** Measure the actual behaviour first —
|
||||||
|
against an analytic reference where one exists. Several "obvious" fixes in
|
||||||
|
this repo turned out to be wrong in sign or unnecessary, and the measurement
|
||||||
|
is what caught them.
|
||||||
|
- **Fix the root issue, not the symptom.** A clamp that hides an underflow, or
|
||||||
|
a tolerance loosened to make a test pass, is a defect deferred.
|
||||||
|
- **Scout crates.io before hand-rolling numerics.** Check accuracy against an
|
||||||
|
independent reference rather than trusting downloads: `puruspe` has 1.4M
|
||||||
|
downloads and is 346 ULP off in the tail, where `libm` is 1. Fewer
|
||||||
|
dependencies is preferable, not mandatory — take the dependency when it is
|
||||||
|
measurably better.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
A Rust port of [TrueSkillThroughTime.py](https://github.com/glandfried/TrueSkillThroughTime.py):
|
A Rust port of [TrueSkillThroughTime.py](https://github.com/glandfried/TrueSkillThroughTime.py):
|
||||||
@@ -66,11 +80,11 @@ History → TimeSlice[] → Event[] → Item[]
|
|||||||
`tau = mu/sigma²`). `Mul`/`Div` are the EP product/cavity: pure adds and
|
`tau = mu/sigma²`). `Mul`/`Div` are the EP product/cavity: pure adds and
|
||||||
subtracts. Variance-space ops (`Add`, `Sub`, `exclude`, `forget`) go through
|
subtracts. Variance-space ops (`Add`, `Sub`, `exclude`, `forget`) go through
|
||||||
`from_mv`/`variance()` and take no square root.
|
`from_mv`/`variance()` and take no square root.
|
||||||
- **`factor/`** — `TeamSumFactor`, `RankDiffFactor`, `TruncFactor` (ranked),
|
- **`factor/`** — `TruncFactor` (ranked) and `MarginFactor` (scored) over a
|
||||||
`MarginFactor` (scored), over a flat `VarStore`. `BuiltinFactor` dispatches
|
flat `VarStore`. `Game::run_chain` drives them directly through a local
|
||||||
by enum rather than `dyn`.
|
`DiffFactor` enum; there is no `Schedule` indirection and no generic `Factor`
|
||||||
- **`Schedule`** (`schedule.rs`) — drives factor propagation. `EpsilonOrMax` is
|
trait. Both were removed once measurement showed nothing had ever used them
|
||||||
the only implementation.
|
— see #42.
|
||||||
- **`Competitor`** (`competitor.rs`) — per-history temporal state (`message`,
|
- **`Competitor`** (`competitor.rs`) — per-history temporal state (`message`,
|
||||||
`last_time`). **`Rating`** (`rating.rs`) — static config (prior, `beta`, drift).
|
`last_time`). **`Rating`** (`rating.rs`) — static config (prior, `beta`, drift).
|
||||||
- **`storage/`** — `SkillStore` (per slice, `pub(crate)`) and `CompetitorStore`
|
- **`storage/`** — `SkillStore` (per slice, `pub(crate)`) and `CompetitorStore`
|
||||||
@@ -97,6 +111,12 @@ History → TimeSlice[] → Event[] → Item[]
|
|||||||
chain underflows to zero, and `ln(0)` is `-inf`.
|
chain underflows to zero, and `ln(0)` is `-inf`.
|
||||||
- **Colors are contiguous.** `recompute_color_groups` reorders events so each
|
- **Colors are contiguous.** `recompute_color_groups` reorders events so each
|
||||||
color occupies one range; `ColorGroups::groups_are_contiguous` asserts it.
|
color occupies one range; `ColorGroups::groups_are_contiguous` asserts it.
|
||||||
|
- **Transcendentals go through `libm`, not `std`.** IEEE 754 pins the basic
|
||||||
|
operations and `sqrt` but says nothing about `exp`/`log`/`erf`, and `std`
|
||||||
|
delegates to the *system* math library — measured, `f64::exp` and `libm::exp`
|
||||||
|
disagree on 9.7% of inputs by one ULP. Since inference is an iterative fixed
|
||||||
|
point, one ULP can change an iteration count. Use `libm::exp` / `libm::log` in
|
||||||
|
inference code; `f64::sqrt` is fine (IEEE specifies it). Tests may use either.
|
||||||
- **The crate is `#![forbid(unsafe_code)]`.** Keep it that way.
|
- **The crate is `#![forbid(unsafe_code)]`.** Keep it that way.
|
||||||
- **Ingestion order must not change the answer.** Events added one at a time
|
- **Ingestion order must not change the answer.** Events added one at a time
|
||||||
must converge to the same fixed point as the same events batched — see
|
must converge to the same fixed point as the same events batched — see
|
||||||
|
|||||||
+2
-1
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "trueskill-tt"
|
name = "trueskill-tt"
|
||||||
version = "0.4.0"
|
version = "0.6.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
rust-version = "1.85"
|
rust-version = "1.85"
|
||||||
description = "TrueSkill Through Time: Bayesian skill rating that tracks how skill evolves over time, via Gaussian message passing"
|
description = "TrueSkill Through Time: Bayesian skill rating that tracks how skill evolves over time, via Gaussian message passing"
|
||||||
@@ -51,6 +51,7 @@ harness = false
|
|||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
approx = { version = "0.5.1", optional = true }
|
approx = { version = "0.5.1", optional = true }
|
||||||
|
libm = "0.2.16"
|
||||||
rayon = { version = "1", optional = true }
|
rayon = { version = "1", optional = true }
|
||||||
smallvec = "1"
|
smallvec = "1"
|
||||||
|
|
||||||
|
|||||||
@@ -195,8 +195,47 @@ stay available at any size:
|
|||||||
quadratic in team count.
|
quadratic in team count.
|
||||||
- `predict_ranking(teams, ranks)` — one specific finishing order.
|
- `predict_ranking(teams, ranks)` — one specific finishing order.
|
||||||
|
|
||||||
Unknown keys are an error, not a silent omission: a team the history has never
|
Unknown keys are an error by default, not a silent omission: a team the history
|
||||||
seen cannot produce a confident-looking probability.
|
has never seen cannot produce a confident-looking probability. The error names
|
||||||
|
the key, and every key must already be known — pre-filter with `lookup` or
|
||||||
|
`current_skill` if your caller cannot guarantee that.
|
||||||
|
|
||||||
|
If predicting for competitors you have never seen is the point rather than a
|
||||||
|
mistake, say so once:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use trueskill_tt::{History, UnknownKeys};
|
||||||
|
|
||||||
|
let h = History::builder().unknown_keys(UnknownKeys::Prior).build();
|
||||||
|
```
|
||||||
|
|
||||||
|
An unknown competitor is then answered from the configured prior, which is the
|
||||||
|
honest reading — you have no evidence about them — and correctly *widens* a team
|
||||||
|
that contains one. There is deliberately no "skip the member" mode: a team's
|
||||||
|
performance is the sum of its members, so dropping one would make the model more
|
||||||
|
certain because it knows less.
|
||||||
|
|
||||||
|
### Asking about one competitor
|
||||||
|
|
||||||
|
`Gaussian` answers tail questions directly, which is what a stopping rule needs:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use trueskill_tt::History;
|
||||||
|
|
||||||
|
let mut h = History::builder().build();
|
||||||
|
h.record_winner(&"alice", &"bob", 1).unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
|
let skill = h.current_skill(&"alice").unwrap();
|
||||||
|
|
||||||
|
// "How sure am I that this is below the cutoff?" — a probability, not a
|
||||||
|
// `mu + z * sigma` band whose confidence drifts as sigma changes.
|
||||||
|
let _ = skill.probability_below(20.0);
|
||||||
|
|
||||||
|
// Use this rather than `1.0 - probability_below(x)`: the complement cancels
|
||||||
|
// away every digit in the upper tail, which is where a stopping rule lives.
|
||||||
|
let _ = skill.probability_above(30.0);
|
||||||
|
```
|
||||||
|
|
||||||
## Which match to play next
|
## Which match to play next
|
||||||
|
|
||||||
@@ -242,7 +281,7 @@ expensive than `quality()`. Scoring every pairing among `n` competitors is
|
|||||||
- [x] Add Observer (`Observer` / `NullObserver`)
|
- [x] Add Observer (`Observer` / `NullObserver`)
|
||||||
- [x] Benchmark the inference loop (`benches/batch.rs`, `benches/history_converge.rs`, `benches/ingest.rs`)
|
- [x] Benchmark the inference loop (`benches/batch.rs`, `benches/history_converge.rs`, `benches/ingest.rs`)
|
||||||
- [x] N-team `predict_outcome` with draw mass, and `expected_information_gain`
|
- [x] N-team `predict_outcome` with draw mass, and `expected_information_gain`
|
||||||
- [ ] Cross-check `quality()` against [sublee/trueskill](https://github.com/sublee/trueskill/tree/master) — N-group support works and is covered by invariants, but no reference values are asserted
|
- [x] Cross-check `quality()` against [sublee/trueskill](https://github.com/sublee/trueskill/tree/master) — N identical teams follow the closed form `(1/5)^((n-1)/2)` for the conventional parameters, asserted for n = 2..10, and the n=3/n=5 values (0.200, 0.040) match the reference package
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ fn bench_converge(c: &mut Criterion) {
|
|||||||
b.iter_batched(
|
b.iter_batched(
|
||||||
|| build_history_1v1(500, 100, 10, 42),
|
|| build_history_1v1(500, 100, 10, 42),
|
||||||
|mut h| {
|
|mut h| {
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
},
|
},
|
||||||
BatchSize::SmallInput,
|
BatchSize::SmallInput,
|
||||||
);
|
);
|
||||||
@@ -92,7 +92,7 @@ fn bench_converge(c: &mut Criterion) {
|
|||||||
b.iter_batched(
|
b.iter_batched(
|
||||||
|| build_history_1v1(2000, 200, 20, 42),
|
|| build_history_1v1(2000, 200, 20, 42),
|
||||||
|mut h| {
|
|mut h| {
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
},
|
},
|
||||||
BatchSize::SmallInput,
|
BatchSize::SmallInput,
|
||||||
);
|
);
|
||||||
@@ -106,7 +106,7 @@ fn bench_converge(c: &mut Criterion) {
|
|||||||
b.iter_batched(
|
b.iter_batched(
|
||||||
|| build_history_1v1(5000, 50000, 5000, 42),
|
|| build_history_1v1(5000, 50000, 5000, 42),
|
||||||
|mut h| {
|
|mut h| {
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
},
|
},
|
||||||
BatchSize::SmallInput,
|
BatchSize::SmallInput,
|
||||||
);
|
);
|
||||||
|
|||||||
+1
-1
@@ -29,7 +29,7 @@ fn bench_scored_history(c: &mut Criterion) {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
h.add_events(events).unwrap();
|
h.add_events(events).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
+21
-2
@@ -46,14 +46,33 @@ fn main() {
|
|||||||
.sigma(1.6)
|
.sigma(1.6)
|
||||||
.drift(ConstantDrift(0.036))
|
.drift(ConstantDrift(0.036))
|
||||||
.convergence(trueskill_tt::ConvergenceOptions {
|
.convergence(trueskill_tt::ConvergenceOptions {
|
||||||
max_iter: 10,
|
// This history needs 30 sweeps to reach the epsilon below. It was
|
||||||
|
// capped at 10 until the `#[must_use]` on `ConvergenceReport`
|
||||||
|
// surfaced that the example had been shipping a short fit.
|
||||||
|
max_iter: 100,
|
||||||
epsilon: 0.01,
|
epsilon: 0.01,
|
||||||
alpha: 1.0,
|
alpha: 1.0,
|
||||||
})
|
})
|
||||||
.build();
|
.build();
|
||||||
|
|
||||||
hist.add_events(events).unwrap();
|
hist.add_events(events).unwrap();
|
||||||
hist.converge().unwrap();
|
|
||||||
|
// Read the report rather than discarding it. A fit that hits `max_iter`
|
||||||
|
// without reaching `epsilon` is not an error and does not look wrong — every
|
||||||
|
// rating comes back finite and sensibly ordered — so this flag is the only
|
||||||
|
// thing that says the numbers were still moving when the sweep stopped.
|
||||||
|
let report = hist.converge().unwrap();
|
||||||
|
eprintln!(
|
||||||
|
"converged={} after {} sweeps, final step {:?}",
|
||||||
|
report.converged, report.iterations, report.final_step
|
||||||
|
);
|
||||||
|
if !report.converged {
|
||||||
|
eprintln!(
|
||||||
|
"warning: stopped after {} sweeps with a final step of {:?}, \
|
||||||
|
short of epsilon — raise ConvergenceOptions::max_iter",
|
||||||
|
report.iterations, report.final_step
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
let players = [
|
let players = [
|
||||||
("aggasi", "a092", 38800i64),
|
("aggasi", "a092", 38800i64),
|
||||||
|
|||||||
+1
-1
@@ -47,7 +47,7 @@ fn kl_divergence(q: Gaussian, p: Gaussian) -> f64 {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let mean_gap = q.mu() - p.mu();
|
let mean_gap = q.mu() - p.mu();
|
||||||
0.5 * ((var_p / var_q).ln() + (var_q + mean_gap * mean_gap) / var_p - 1.0)
|
0.5 * (libm::log(var_p / var_q) + (var_q + mean_gap * mean_gap) / var_p - 1.0)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Expected information gain of a hypothetical matchup, in nats.
|
/// Expected information gain of a hypothetical matchup, in nats.
|
||||||
|
|||||||
@@ -63,6 +63,9 @@ impl Default for ConvergenceOptions {
|
|||||||
|
|
||||||
/// Post-hoc summary of a `History::converge` call.
|
/// Post-hoc summary of a `History::converge` call.
|
||||||
#[derive(Clone, Debug)]
|
#[derive(Clone, Debug)]
|
||||||
|
#[must_use = "a ConvergenceReport carries `converged`, and a fit that stopped \
|
||||||
|
at `max_iter` is wrong by a little rather than loudly broken — \
|
||||||
|
check it, or bind it to `_` to say you have decided not to"]
|
||||||
pub struct ConvergenceReport {
|
pub struct ConvergenceReport {
|
||||||
pub iterations: usize,
|
pub iterations: usize,
|
||||||
pub final_step: (f64, f64),
|
pub final_step: (f64, f64),
|
||||||
|
|||||||
+59
-3
@@ -1,5 +1,44 @@
|
|||||||
use std::fmt;
|
use std::fmt;
|
||||||
|
|
||||||
|
/// How a prediction should treat a key the history has never seen.
|
||||||
|
///
|
||||||
|
/// Configured once per history via
|
||||||
|
/// [`HistoryBuilder::unknown_keys`](crate::HistoryBuilder::unknown_keys).
|
||||||
|
/// Neither known consumer wants this to vary between queries — one predicts
|
||||||
|
/// thousands of candidate matchups in a loop, the other's headline feature is
|
||||||
|
/// predicting a competitor nobody has faced — so it is a property of how you
|
||||||
|
/// intend to use the model rather than an argument on five call sites.
|
||||||
|
///
|
||||||
|
/// # There is deliberately no `Skip`
|
||||||
|
///
|
||||||
|
/// Dropping an unknown member is the obvious third option and it is wrong. A
|
||||||
|
/// team's performance is the *sum* of its members, so removing one removes its
|
||||||
|
/// variance too: measured on a two-member team with one unknown, skipping gives
|
||||||
|
/// a performance sigma of 2.37 where treating the member as unknown gives 6.53.
|
||||||
|
/// An unknown competitor would make the model *more* certain, which is
|
||||||
|
/// backwards. `Prior` is also the answer the model already gives for a
|
||||||
|
/// competitor it knows about but has no evidence for, so it corresponds to a
|
||||||
|
/// state the model can actually be in; skipping does not.
|
||||||
|
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||||
|
#[non_exhaustive]
|
||||||
|
pub enum UnknownKeys {
|
||||||
|
/// Reject the prediction with [`InferenceError::UnknownKey`].
|
||||||
|
///
|
||||||
|
/// The default, and the right one when every key is expected to be known:
|
||||||
|
/// a team of strangers should not silently produce a confident-looking
|
||||||
|
/// answer.
|
||||||
|
#[default]
|
||||||
|
Reject,
|
||||||
|
/// Treat an unknown competitor as one sitting at the history's configured
|
||||||
|
/// prior.
|
||||||
|
///
|
||||||
|
/// This is the honest Bayesian reading — a competitor you have never
|
||||||
|
/// observed is exactly the prior — and it makes "predict a matchup
|
||||||
|
/// involving someone new" a first-class question rather than something a
|
||||||
|
/// caller fakes with a neutral constant.
|
||||||
|
Prior,
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone, PartialEq)]
|
#[derive(Debug, Clone, PartialEq)]
|
||||||
#[non_exhaustive]
|
#[non_exhaustive]
|
||||||
pub enum InferenceError {
|
pub enum InferenceError {
|
||||||
@@ -50,9 +89,21 @@ pub enum InferenceError {
|
|||||||
///
|
///
|
||||||
/// Reported rather than skipped: dropping unknown keys turns a team of
|
/// Reported rather than skipped: dropping unknown keys turns a team of
|
||||||
/// strangers into a confident-looking probability about nobody.
|
/// strangers into a confident-looking probability about nobody.
|
||||||
UnknownKey { team: usize, member: usize },
|
///
|
||||||
|
/// `key` is the offending key's `Debug` rendering. It is carried because
|
||||||
|
/// the indices alone are not actionable: a caller that logs
|
||||||
|
/// `UnknownKey { team: 0, member: 0 }` learns nothing about *which* of its
|
||||||
|
/// keys the history has not seen, and the natural handling — fall back to a
|
||||||
|
/// neutral value — turns the whole thing into a plausible constant.
|
||||||
|
UnknownKey {
|
||||||
|
team: usize,
|
||||||
|
member: usize,
|
||||||
|
key: String,
|
||||||
|
},
|
||||||
/// A prediction was given a team with no members.
|
/// A prediction was given a team with no members.
|
||||||
EmptyTeam { team: usize },
|
EmptyTeam { team: usize },
|
||||||
|
/// A joint posterior was requested where one cannot be formed exactly.
|
||||||
|
JointUnavailable { reason: &'static str },
|
||||||
/// Fewer than two teams were supplied to a prediction.
|
/// Fewer than two teams were supplied to a prediction.
|
||||||
NotEnoughTeams { got: usize },
|
NotEnoughTeams { got: usize },
|
||||||
/// The full outcome distribution was requested for too many teams.
|
/// The full outcome distribution was requested for too many teams.
|
||||||
@@ -108,15 +159,20 @@ impl fmt::Display for InferenceError {
|
|||||||
"competitor {competitor}: this batch sets {field} to two different values"
|
"competitor {competitor}: this batch sets {field} to two different values"
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
Self::UnknownKey { team, member } => {
|
Self::UnknownKey { team, member, key } => {
|
||||||
write!(
|
write!(
|
||||||
f,
|
f,
|
||||||
"team {team}, member {member}: no skill recorded for this key"
|
"team {team}, member {member}: no skill recorded for key {key} \
|
||||||
|
(every key must already be known to the history; pre-filter \
|
||||||
|
with `lookup` or `current_skill` if that is not guaranteed)"
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
Self::EmptyTeam { team } => {
|
Self::EmptyTeam { team } => {
|
||||||
write!(f, "team {team} has no members")
|
write!(f, "team {team} has no members")
|
||||||
}
|
}
|
||||||
|
Self::JointUnavailable { reason } => {
|
||||||
|
write!(f, "no exact joint posterior is available: {reason}")
|
||||||
|
}
|
||||||
Self::NotEnoughTeams { got } => {
|
Self::NotEnoughTeams { got } => {
|
||||||
write!(f, "prediction needs at least 2 teams, got {got}")
|
write!(f, "prediction needs at least 2 teams, got {got}")
|
||||||
}
|
}
|
||||||
|
|||||||
+41
-22
@@ -1,8 +1,8 @@
|
|||||||
use crate::{
|
use crate::{
|
||||||
N_INF,
|
N_INF,
|
||||||
factor::{Factor, VarId, VarStore},
|
factor::{VarId, VarStore},
|
||||||
gaussian::Gaussian,
|
gaussian::Gaussian,
|
||||||
pdf,
|
ln_pdf,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Gaussian observation factor on a diff variable.
|
/// Gaussian observation factor on a diff variable.
|
||||||
@@ -16,7 +16,7 @@ pub struct MarginFactor {
|
|||||||
pub m_obs: f64,
|
pub m_obs: f64,
|
||||||
pub sigma: f64,
|
pub sigma: f64,
|
||||||
pub(crate) msg: Gaussian,
|
pub(crate) msg: Gaussian,
|
||||||
pub(crate) evidence_cached: Option<f64>,
|
pub(crate) log_evidence_cached: Option<f64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl MarginFactor {
|
impl MarginFactor {
|
||||||
@@ -28,7 +28,7 @@ impl MarginFactor {
|
|||||||
m_obs,
|
m_obs,
|
||||||
sigma,
|
sigma,
|
||||||
msg: N_INF,
|
msg: N_INF,
|
||||||
evidence_cached: None,
|
log_evidence_cached: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -41,8 +41,8 @@ impl MarginFactor {
|
|||||||
let marginal = vars.get(self.diff);
|
let marginal = vars.get(self.diff);
|
||||||
let cavity = marginal / self.msg;
|
let cavity = marginal / self.msg;
|
||||||
|
|
||||||
if self.evidence_cached.is_none() {
|
if self.log_evidence_cached.is_none() {
|
||||||
self.evidence_cached = Some(cavity_evidence(cavity, self.m_obs, self.sigma));
|
self.log_evidence_cached = Some(cavity_log_evidence(cavity, self.m_obs, self.sigma));
|
||||||
}
|
}
|
||||||
|
|
||||||
let new_msg = Gaussian::from_ms(self.m_obs, self.sigma);
|
let new_msg = Gaussian::from_ms(self.m_obs, self.sigma);
|
||||||
@@ -55,23 +55,42 @@ impl MarginFactor {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Factor for MarginFactor {
|
/// Undamped wrappers, used by this module's tests. Inference drives these
|
||||||
fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64) {
|
/// factors through `propagate_with_alpha` and reads the cached log evidence
|
||||||
|
/// directly, so these are not on any production path.
|
||||||
|
#[cfg(test)]
|
||||||
|
impl MarginFactor {
|
||||||
|
pub(crate) fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64) {
|
||||||
self.propagate_with_alpha(vars, 1.0)
|
self.propagate_with_alpha(vars, 1.0)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn log_evidence(&self, _vars: &VarStore) -> f64 {
|
pub(crate) fn log_evidence(&self) -> f64 {
|
||||||
self.evidence_cached.unwrap_or(1.0).ln()
|
self.log_evidence_cached.unwrap_or(0.0)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Density of the observed margin under the cavity, clamped to a positive
|
/// `ln` of the observed margin's density under the cavity.
|
||||||
/// floor so a far-out observation cannot underflow to `0.0` and make
|
///
|
||||||
/// `log_evidence` `-inf`.
|
/// Computed in log space rather than as `pdf(..).ln()`. The density underflows
|
||||||
fn cavity_evidence(cavity: Gaussian, m_obs: f64, sigma: f64) -> f64 {
|
/// to zero past about 38 sigma of separation, and clamping that to
|
||||||
let combined_sigma = (cavity.sigma().powi(2) + sigma.powi(2)).sqrt();
|
/// `f64::MIN_POSITIVE` reported -708 nats however far out the observation
|
||||||
|
/// actually was — 4292 nats adrift at 100 sigma, and unbounded beyond. A score
|
||||||
|
/// far from what the model expected is exactly the observation a log-evidence
|
||||||
|
/// figure exists to notice.
|
||||||
|
fn cavity_log_evidence(cavity: Gaussian, m_obs: f64, sigma: f64) -> f64 {
|
||||||
|
// `hypot`, not `sqrt(a^2 + b^2)`: squaring overflows to infinity above a
|
||||||
|
// sigma of ~1.3e154 and flushes to zero below ~1.5e-154, and `Gaussian`'s
|
||||||
|
// constructors are public so a caller can reach both.
|
||||||
|
let combined_sigma = cavity.sigma().hypot(sigma);
|
||||||
|
let value = ln_pdf(m_obs, cavity.mu(), combined_sigma);
|
||||||
|
|
||||||
pdf(m_obs, cavity.mu(), combined_sigma).max(f64::MIN_POSITIVE)
|
// A degenerate cavity (infinite sigma) is the only way to reach a
|
||||||
|
// non-finite result; fall back to the old floor rather than emit -inf.
|
||||||
|
if value.is_finite() {
|
||||||
|
value
|
||||||
|
} else {
|
||||||
|
f64::MIN_POSITIVE.ln()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
@@ -113,16 +132,16 @@ mod tests {
|
|||||||
let mut vars = VarStore::new();
|
let mut vars = VarStore::new();
|
||||||
let diff = vars.alloc(Gaussian::from_ms(0.0, 6.0));
|
let diff = vars.alloc(Gaussian::from_ms(0.0, 6.0));
|
||||||
let mut f = MarginFactor::new(diff, 5.0, 1.0);
|
let mut f = MarginFactor::new(diff, 5.0, 1.0);
|
||||||
assert!(f.evidence_cached.is_none());
|
assert!(f.log_evidence_cached.is_none());
|
||||||
|
|
||||||
f.propagate(&mut vars);
|
f.propagate(&mut vars);
|
||||||
let z = f.evidence_cached.unwrap();
|
let z = f.log_evidence_cached.unwrap();
|
||||||
// pdf(5, 0, sqrt(37)) ≈ 0.046783
|
// ln pdf(5, 0, sqrt(37)) = ln(0.046783...)
|
||||||
assert!((z - 0.04678300292616668).abs() < 1e-10);
|
assert!((z.exp() - 0.04678300292616668).abs() < 1e-10);
|
||||||
|
|
||||||
// Subsequent propagations don't change it.
|
// Subsequent propagations don't change it.
|
||||||
f.propagate(&mut vars);
|
f.propagate(&mut vars);
|
||||||
assert_eq!(f.evidence_cached.unwrap(), z);
|
assert_eq!(f.log_evidence_cached.unwrap(), z);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -131,7 +150,7 @@ mod tests {
|
|||||||
let diff = vars.alloc(Gaussian::from_ms(0.0, 6.0));
|
let diff = vars.alloc(Gaussian::from_ms(0.0, 6.0));
|
||||||
let mut f = MarginFactor::new(diff, 5.0, 1.0);
|
let mut f = MarginFactor::new(diff, 5.0, 1.0);
|
||||||
f.propagate(&mut vars);
|
f.propagate(&mut vars);
|
||||||
let logz = f.log_evidence(&vars);
|
let logz = f.log_evidence();
|
||||||
assert!((logz - (-3.062235327364623)).abs() < 1e-10);
|
assert!((logz - (-3.062235327364623)).abs() < 1e-10);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+4
-72
@@ -20,6 +20,8 @@ pub struct VarStore {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl VarStore {
|
impl VarStore {
|
||||||
|
/// Test-only: inference allocates its store through `ScratchArena`.
|
||||||
|
#[cfg(test)]
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn new() -> Self {
|
pub fn new() -> Self {
|
||||||
Self::default()
|
Self::default()
|
||||||
@@ -29,16 +31,13 @@ impl VarStore {
|
|||||||
self.marginals.clear();
|
self.marginals.clear();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Test-only, as `new`.
|
||||||
|
#[cfg(test)]
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn len(&self) -> usize {
|
pub fn len(&self) -> usize {
|
||||||
self.marginals.len()
|
self.marginals.len()
|
||||||
}
|
}
|
||||||
|
|
||||||
#[must_use]
|
|
||||||
pub fn is_empty(&self) -> bool {
|
|
||||||
self.marginals.is_empty()
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn alloc(&mut self, init: Gaussian) -> VarId {
|
pub fn alloc(&mut self, init: Gaussian) -> VarId {
|
||||||
let id = VarId(self.marginals.len() as u32);
|
let id = VarId(self.marginals.len() as u32);
|
||||||
self.marginals.push(init);
|
self.marginals.push(init);
|
||||||
@@ -55,58 +54,7 @@ impl VarStore {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A factor in the EP graph.
|
|
||||||
///
|
|
||||||
/// Factors hold their own outgoing messages and propagate them by reading
|
|
||||||
/// connected variable marginals from a `VarStore` and writing back updated
|
|
||||||
/// marginals.
|
|
||||||
pub trait Factor: Send + Sync {
|
|
||||||
/// Update outgoing messages and write back to the var store.
|
|
||||||
///
|
|
||||||
/// Returns the max delta `(|Δmu|, |Δsigma|)` across writes this
|
|
||||||
/// propagation. Used by the `Schedule` to detect convergence.
|
|
||||||
fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64);
|
|
||||||
|
|
||||||
/// Optional log-evidence contribution. Default 0.0 (no contribution).
|
|
||||||
fn log_evidence(&self, _vars: &VarStore) -> f64 {
|
|
||||||
0.0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Enum dispatcher for the built-in factor types.
|
|
||||||
///
|
|
||||||
/// Using an enum instead of `Box<dyn Factor>` keeps factor data inline and
|
|
||||||
/// avoids virtual-call overhead in the hot inference loop.
|
|
||||||
#[derive(Debug)]
|
|
||||||
pub enum BuiltinFactor {
|
|
||||||
TeamSum(team_sum::TeamSumFactor),
|
|
||||||
RankDiff(rank_diff::RankDiffFactor),
|
|
||||||
Trunc(trunc::TruncFactor),
|
|
||||||
Margin(margin::MarginFactor),
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Factor for BuiltinFactor {
|
|
||||||
fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64) {
|
|
||||||
match self {
|
|
||||||
Self::TeamSum(f) => f.propagate(vars),
|
|
||||||
Self::RankDiff(f) => f.propagate(vars),
|
|
||||||
Self::Trunc(f) => f.propagate(vars),
|
|
||||||
Self::Margin(f) => f.propagate(vars),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn log_evidence(&self, vars: &VarStore) -> f64 {
|
|
||||||
match self {
|
|
||||||
Self::Trunc(f) => f.log_evidence(vars),
|
|
||||||
Self::Margin(f) => f.log_evidence(vars),
|
|
||||||
Self::TeamSum(_) | Self::RankDiff(_) => 0.0,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub mod margin;
|
pub mod margin;
|
||||||
pub mod rank_diff;
|
|
||||||
pub mod team_sum;
|
|
||||||
pub mod trunc;
|
pub mod trunc;
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
@@ -153,20 +101,4 @@ mod tests {
|
|||||||
assert_eq!(store.len(), 0);
|
assert_eq!(store.len(), 0);
|
||||||
assert_eq!(store.marginals.capacity(), cap);
|
assert_eq!(store.marginals.capacity(), cap);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn builtin_factor_dispatches_to_margin() {
|
|
||||||
use super::margin::MarginFactor;
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let diff = vars.alloc(Gaussian::from_ms(0.0, 6.0));
|
|
||||||
let mut f = BuiltinFactor::Margin(MarginFactor::new(diff, 5.0, 1.0));
|
|
||||||
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
|
|
||||||
let result = vars.get(diff);
|
|
||||||
assert!((result.mu() - 4.864864864864865).abs() < 1e-12);
|
|
||||||
|
|
||||||
let logz = f.log_evidence(&vars);
|
|
||||||
assert!((logz - (-3.062235327364623)).abs() < 1e-10);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,95 +0,0 @@
|
|||||||
use crate::factor::{Factor, VarId, VarStore};
|
|
||||||
|
|
||||||
/// Maintains the constraint `diff = team_a - team_b` between three vars.
|
|
||||||
///
|
|
||||||
/// On each propagation:
|
|
||||||
/// - Reads marginals at `team_a` and `team_b` (which already incorporate any
|
|
||||||
/// incoming messages from neighboring factors).
|
|
||||||
/// - Computes `new_diff = team_a - team_b` (variance addition; see `Gaussian::Sub`).
|
|
||||||
/// - Writes the new marginal to `diff`.
|
|
||||||
/// - Returns the delta against the previous diff value.
|
|
||||||
///
|
|
||||||
/// This factor does NOT store an outgoing message; the diff variable is
|
|
||||||
/// effectively replaced on each propagation. The `TruncFactor` on the same diff
|
|
||||||
/// var holds the EP-divide message that produces the cavity.
|
|
||||||
#[derive(Debug)]
|
|
||||||
pub struct RankDiffFactor {
|
|
||||||
pub team_a: VarId,
|
|
||||||
pub team_b: VarId,
|
|
||||||
pub diff: VarId,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Factor for RankDiffFactor {
|
|
||||||
fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64) {
|
|
||||||
let a = vars.get(self.team_a);
|
|
||||||
let b = vars.get(self.team_b);
|
|
||||||
let new_diff = a - b;
|
|
||||||
let old = vars.get(self.diff);
|
|
||||||
vars.set(self.diff, new_diff);
|
|
||||||
old.delta(new_diff)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use crate::{N_INF, gaussian::Gaussian};
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn diff_of_two_known_gaussians() {
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let team_a = vars.alloc(Gaussian::from_ms(25.0, 3.0));
|
|
||||||
let team_b = vars.alloc(Gaussian::from_ms(20.0, 4.0));
|
|
||||||
let diff = vars.alloc(N_INF);
|
|
||||||
|
|
||||||
let mut f = RankDiffFactor {
|
|
||||||
team_a,
|
|
||||||
team_b,
|
|
||||||
diff,
|
|
||||||
};
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
|
|
||||||
let result = vars.get(diff);
|
|
||||||
// mu = 25 - 20 = 5; var = 9 + 16 = 25; sigma = 5
|
|
||||||
assert!((result.mu() - 5.0).abs() < 1e-12);
|
|
||||||
assert!((result.sigma() - 5.0).abs() < 1e-12);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn delta_zero_on_repeat() {
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let team_a = vars.alloc(Gaussian::from_ms(10.0, 2.0));
|
|
||||||
let team_b = vars.alloc(Gaussian::from_ms(8.0, 1.0));
|
|
||||||
let diff = vars.alloc(N_INF);
|
|
||||||
|
|
||||||
let mut f = RankDiffFactor {
|
|
||||||
team_a,
|
|
||||||
team_b,
|
|
||||||
diff,
|
|
||||||
};
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
let (dmu, dsig) = f.propagate(&mut vars);
|
|
||||||
assert!(dmu < 1e-12);
|
|
||||||
assert!(dsig < 1e-12);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn delta_reflects_team_change() {
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let team_a = vars.alloc(Gaussian::from_ms(10.0, 1.0));
|
|
||||||
let team_b = vars.alloc(Gaussian::from_ms(0.0, 1.0));
|
|
||||||
let diff = vars.alloc(N_INF);
|
|
||||||
|
|
||||||
let mut f = RankDiffFactor {
|
|
||||||
team_a,
|
|
||||||
team_b,
|
|
||||||
diff,
|
|
||||||
};
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
|
|
||||||
// change team_a, repropagate; delta should be positive
|
|
||||||
vars.set(team_a, Gaussian::from_ms(15.0, 1.0));
|
|
||||||
let (dmu, _dsig) = f.propagate(&mut vars);
|
|
||||||
assert!(dmu > 4.0, "expected ~5 delta, got {}", dmu);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,98 +0,0 @@
|
|||||||
use crate::{
|
|
||||||
N00,
|
|
||||||
factor::{Factor, VarId, VarStore},
|
|
||||||
gaussian::Gaussian,
|
|
||||||
};
|
|
||||||
|
|
||||||
/// Computes the weighted sum of player performances into a team-perf var.
|
|
||||||
///
|
|
||||||
/// Inputs are pre-computed player performance Gaussians (i.e., rating priors
|
|
||||||
/// already with beta² noise added via `Rating::performance()`). The factor
|
|
||||||
/// runs once per game and writes the weighted sum to the output var.
|
|
||||||
#[derive(Debug)]
|
|
||||||
pub struct TeamSumFactor {
|
|
||||||
pub inputs: Vec<(Gaussian, f64)>,
|
|
||||||
pub out: VarId,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Factor for TeamSumFactor {
|
|
||||||
fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64) {
|
|
||||||
let perf = self.inputs.iter().fold(N00, |acc, (g, w)| acc + (*g * *w));
|
|
||||||
let old = vars.get(self.out);
|
|
||||||
vars.set(self.out, perf);
|
|
||||||
old.delta(perf)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use crate::N_INF;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn single_player_unit_weight() {
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let out = vars.alloc(N_INF);
|
|
||||||
let g = Gaussian::from_ms(25.0, 5.0);
|
|
||||||
let mut f = TeamSumFactor {
|
|
||||||
inputs: vec![(g, 1.0)],
|
|
||||||
out,
|
|
||||||
};
|
|
||||||
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
let result = vars.get(out);
|
|
||||||
assert!((result.mu() - 25.0).abs() < 1e-12);
|
|
||||||
assert!((result.sigma() - 5.0).abs() < 1e-12);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn two_players_summed() {
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let out = vars.alloc(N_INF);
|
|
||||||
let g1 = Gaussian::from_ms(20.0, 3.0);
|
|
||||||
let g2 = Gaussian::from_ms(30.0, 4.0);
|
|
||||||
let mut f = TeamSumFactor {
|
|
||||||
inputs: vec![(g1, 1.0), (g2, 1.0)],
|
|
||||||
out,
|
|
||||||
};
|
|
||||||
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
let result = vars.get(out);
|
|
||||||
// sum: mu = 20 + 30 = 50, var = 9 + 16 = 25, sigma = 5
|
|
||||||
assert!((result.mu() - 50.0).abs() < 1e-12);
|
|
||||||
assert!((result.sigma() - 5.0).abs() < 1e-12);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn weighted_inputs() {
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let out = vars.alloc(N_INF);
|
|
||||||
let g = Gaussian::from_ms(10.0, 2.0);
|
|
||||||
let mut f = TeamSumFactor {
|
|
||||||
inputs: vec![(g, 2.0)],
|
|
||||||
out,
|
|
||||||
};
|
|
||||||
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
let result = vars.get(out);
|
|
||||||
// g * 2.0: mu = 10*2 = 20, sigma = 2*2 = 4
|
|
||||||
assert!((result.mu() - 20.0).abs() < 1e-12);
|
|
||||||
assert!((result.sigma() - 4.0).abs() < 1e-12);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn delta_is_zero_on_repeat_propagate() {
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let out = vars.alloc(N_INF);
|
|
||||||
let g = Gaussian::from_ms(5.0, 1.0);
|
|
||||||
let mut f = TeamSumFactor {
|
|
||||||
inputs: vec![(g, 1.0)],
|
|
||||||
out,
|
|
||||||
};
|
|
||||||
|
|
||||||
f.propagate(&mut vars);
|
|
||||||
let (dmu, dsig) = f.propagate(&mut vars);
|
|
||||||
assert!(dmu < 1e-12, "expected ~0 delta on repeat, got {}", dmu);
|
|
||||||
assert!(dsig < 1e-12);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+71
-50
@@ -1,8 +1,8 @@
|
|||||||
use crate::{
|
use crate::{
|
||||||
N_INF, approx, cdf,
|
N_INF, approx,
|
||||||
factor::{Factor, VarId, VarStore},
|
factor::{VarId, VarStore},
|
||||||
gaussian::Gaussian,
|
gaussian::Gaussian,
|
||||||
sf,
|
ln_interval, ln_sf,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// EP truncation factor on a diff variable.
|
/// EP truncation factor on a diff variable.
|
||||||
@@ -19,7 +19,7 @@ pub struct TruncFactor {
|
|||||||
/// Outgoing message to the diff variable (initial: `N_INF`, the EP identity).
|
/// Outgoing message to the diff variable (initial: `N_INF`, the EP identity).
|
||||||
pub(crate) msg: Gaussian,
|
pub(crate) msg: Gaussian,
|
||||||
/// Cached evidence (linear, not log) computed from the cavity on first propagation.
|
/// Cached evidence (linear, not log) computed from the cavity on first propagation.
|
||||||
pub(crate) evidence_cached: Option<f64>,
|
pub(crate) log_evidence_cached: Option<f64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl TruncFactor {
|
impl TruncFactor {
|
||||||
@@ -30,7 +30,7 @@ impl TruncFactor {
|
|||||||
margin,
|
margin,
|
||||||
tie,
|
tie,
|
||||||
msg: N_INF,
|
msg: N_INF,
|
||||||
evidence_cached: None,
|
log_evidence_cached: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -43,8 +43,8 @@ impl TruncFactor {
|
|||||||
let marginal = vars.get(self.diff);
|
let marginal = vars.get(self.diff);
|
||||||
let cavity = marginal / self.msg;
|
let cavity = marginal / self.msg;
|
||||||
|
|
||||||
if self.evidence_cached.is_none() {
|
if self.log_evidence_cached.is_none() {
|
||||||
self.evidence_cached = Some(cavity_evidence(cavity, self.margin, self.tie));
|
self.log_evidence_cached = Some(cavity_log_evidence(cavity, self.margin, self.tie));
|
||||||
}
|
}
|
||||||
|
|
||||||
let trunc = approx(cavity, self.margin, self.tie);
|
let trunc = approx(cavity, self.margin, self.tie);
|
||||||
@@ -63,44 +63,40 @@ impl TruncFactor {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Factor for TruncFactor {
|
/// Undamped wrappers, used by this module's tests. Inference drives these
|
||||||
fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64) {
|
/// factors through `propagate_with_alpha` and reads the cached log evidence
|
||||||
|
/// directly, so these are not on any production path.
|
||||||
|
#[cfg(test)]
|
||||||
|
impl TruncFactor {
|
||||||
|
pub(crate) fn propagate(&mut self, vars: &mut VarStore) -> (f64, f64) {
|
||||||
self.propagate_with_alpha(vars, 1.0)
|
self.propagate_with_alpha(vars, 1.0)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn log_evidence(&self, _vars: &VarStore) -> f64 {
|
|
||||||
self.evidence_cached.unwrap_or(1.0).ln()
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// P(diff > margin) for non-tie, P(|diff| < margin) for tie.
|
/// `ln P(diff > margin)` for a win, `ln P(|diff| < margin)` for a tie.
|
||||||
///
|
///
|
||||||
/// Both branches pick whichever tail keeps their terms *small*, because the
|
/// Computed in log space throughout. Two earlier shapes both lost the tail:
|
||||||
/// alternative is subtracting two numbers that both approach 1. That
|
/// `1 - cdf(..)` cancelled away every digit of an unlikely outcome, and even
|
||||||
/// subtraction is not a rounding detail: it loses every digit of an unlikely
|
/// once that was fixed the linear probability underflows to zero past about 38
|
||||||
/// outcome's evidence, and an unlikely outcome is precisely the one worth
|
/// sigma, where clamping reported -708 nats regardless of the truth. An upset
|
||||||
/// scoring. `1 - cdf` returned exactly zero past ~8.3 sigma, where the true
|
/// is the observation a log-evidence figure exists to notice, so it has to stay
|
||||||
/// probability is 1e-19; clamped, that reached `log_evidence` as -708 instead
|
/// exact precisely where it is smallest.
|
||||||
/// of -43.
|
fn cavity_log_evidence(diff: Gaussian, margin: f64, tie: bool) -> f64 {
|
||||||
///
|
|
||||||
/// The clamp remains as a guard rather than a workaround: `erfc` carries ~1e-7
|
|
||||||
/// relative error, so a probability of exactly 1 can still come back a hair
|
|
||||||
/// above it, and `ln` of a negative would poison the sum for the whole history.
|
|
||||||
fn cavity_evidence(diff: Gaussian, margin: f64, tie: bool) -> f64 {
|
|
||||||
let (mu, sigma) = (diff.mu(), diff.sigma());
|
let (mu, sigma) = (diff.mu(), diff.sigma());
|
||||||
|
|
||||||
let raw = if tie {
|
let value = if tie {
|
||||||
if mu < -margin {
|
ln_interval(-margin, margin, mu, sigma)
|
||||||
// Both CDFs sit against 1 here; both survival terms are small.
|
|
||||||
sf(-margin, mu, sigma) - sf(margin, mu, sigma)
|
|
||||||
} else {
|
|
||||||
cdf(margin, mu, sigma) - cdf(-margin, mu, sigma)
|
|
||||||
}
|
|
||||||
} else {
|
} else {
|
||||||
sf(margin, mu, sigma)
|
ln_sf(margin, mu, sigma)
|
||||||
};
|
};
|
||||||
|
|
||||||
raw.clamp(f64::MIN_POSITIVE, 1.0)
|
// A degenerate cavity is the only route to a non-finite result; keep the
|
||||||
|
// old floor for it rather than letting -inf poison the whole history's sum.
|
||||||
|
if value.is_finite() {
|
||||||
|
value
|
||||||
|
} else {
|
||||||
|
f64::MIN_POSITIVE.ln()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
@@ -131,19 +127,19 @@ mod tests {
|
|||||||
let diff = vars.alloc(Gaussian::from_ms(2.0, 3.0));
|
let diff = vars.alloc(Gaussian::from_ms(2.0, 3.0));
|
||||||
|
|
||||||
let mut f = TruncFactor::new(diff, 0.0, false);
|
let mut f = TruncFactor::new(diff, 0.0, false);
|
||||||
assert!(f.evidence_cached.is_none());
|
assert!(f.log_evidence_cached.is_none());
|
||||||
|
|
||||||
f.propagate(&mut vars);
|
f.propagate(&mut vars);
|
||||||
assert!(f.evidence_cached.is_some());
|
assert!(f.log_evidence_cached.is_some());
|
||||||
let first = f.evidence_cached.unwrap();
|
let first = f.log_evidence_cached.unwrap();
|
||||||
|
|
||||||
// Evidence should be P(diff > 0) for diff ~ N(2, 9) ≈ 0.748
|
// Evidence should be P(diff > 0) for diff ~ N(2, 9) ≈ 0.748
|
||||||
assert!(first > 0.7);
|
assert!(first.exp() > 0.7);
|
||||||
assert!(first < 0.8);
|
assert!(first.exp() < 0.8);
|
||||||
|
|
||||||
// Subsequent propagations don't change it.
|
// Subsequent propagations don't change it.
|
||||||
f.propagate(&mut vars);
|
f.propagate(&mut vars);
|
||||||
assert_eq!(f.evidence_cached.unwrap(), first);
|
assert_eq!(f.log_evidence_cached.unwrap(), first);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The defect this guards: `1 - cdf` collapsed to zero for a surprising
|
/// The defect this guards: `1 - cdf` collapsed to zero for a surprising
|
||||||
@@ -154,7 +150,7 @@ mod tests {
|
|||||||
#[test]
|
#[test]
|
||||||
fn evidence_of_an_upset_is_not_flattened_to_the_clamp_floor() {
|
fn evidence_of_an_upset_is_not_flattened_to_the_clamp_floor() {
|
||||||
// diff ~ N(-9, 1) with margin 0: the favoured side lost by nine sigma.
|
// diff ~ N(-9, 1) with margin 0: the favoured side lost by nine sigma.
|
||||||
let evidence = cavity_evidence(Gaussian::from_ms(-9.0, 1.0), 0.0, false);
|
let evidence = cavity_log_evidence(Gaussian::from_ms(-9.0, 1.0), 0.0, false).exp();
|
||||||
|
|
||||||
assert!(
|
assert!(
|
||||||
evidence > f64::MIN_POSITIVE,
|
evidence > f64::MIN_POSITIVE,
|
||||||
@@ -175,23 +171,48 @@ mod tests {
|
|||||||
/// Evidence must stay finite and positive however extreme the mismatch,
|
/// Evidence must stay finite and positive however extreme the mismatch,
|
||||||
/// since `log_evidence` sums across the whole history and one `-inf` or
|
/// since `log_evidence` sums across the whole history and one `-inf` or
|
||||||
/// `NaN` poisons all of it.
|
/// `NaN` poisons all of it.
|
||||||
|
///
|
||||||
|
/// Finiteness alone is too weak a bar — the clamped version was finite too,
|
||||||
|
/// and wrong by hundreds of nats. `log_evidence_tracks_the_analytic_tail`
|
||||||
|
/// below is the assertion that actually holds this up.
|
||||||
#[test]
|
#[test]
|
||||||
fn evidence_stays_positive_and_finite_at_any_separation() {
|
fn evidence_stays_positive_and_finite_at_any_separation() {
|
||||||
for mu in [-300.0f64, -50.0, -9.0, 0.0, 9.0, 50.0, 300.0] {
|
for mu in [-300.0f64, -50.0, -9.0, 0.0, 9.0, 50.0, 300.0] {
|
||||||
for tie in [false, true] {
|
for tie in [false, true] {
|
||||||
let e = cavity_evidence(Gaussian::from_ms(mu, 1.0), 1.0, tie);
|
let ln_e = cavity_log_evidence(Gaussian::from_ms(mu, 1.0), 1.0, tie);
|
||||||
assert!(
|
assert!(
|
||||||
e.is_finite() && e > 0.0 && e <= 1.0,
|
ln_e.is_finite() && ln_e <= 0.0,
|
||||||
"mu={mu} tie={tie}: evidence {e} is not a probability"
|
"mu={mu} tie={tie}: log evidence {ln_e} is not a log-probability"
|
||||||
);
|
|
||||||
assert!(
|
|
||||||
e.ln().is_finite(),
|
|
||||||
"mu={mu} tie={tie}: ln evidence is not finite"
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The clamp used to floor everything past ~38 sigma at `ln(MIN_POSITIVE)`
|
||||||
|
/// = -708, however far out the real observation was. In log space the
|
||||||
|
/// answer is a polynomial and stays exact: at 1000 sigma the truth is about
|
||||||
|
/// -500_000 nats, and -708 is not a rounding error.
|
||||||
|
#[test]
|
||||||
|
fn log_evidence_tracks_the_analytic_tail() {
|
||||||
|
for mu in [-40.0f64, -60.0, -100.0, -1000.0] {
|
||||||
|
// P(diff > 0) for diff ~ N(mu, 1), mu far below zero.
|
||||||
|
let got = cavity_log_evidence(Gaussian::from_ms(mu, 1.0), 0.0, false);
|
||||||
|
|
||||||
|
// ln Phi(mu) ~ -mu^2/2 - ln(-mu) - ln(sqrt(2 pi)) for mu << 0.
|
||||||
|
let z = -mu;
|
||||||
|
let approx = -0.5 * z * z - z.ln() - (2.0 * std::f64::consts::PI).sqrt().ln();
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
got < f64::MIN_POSITIVE.ln(),
|
||||||
|
"mu={mu}: {got} is still stuck on the old clamp floor"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
(got - approx).abs() / approx.abs() < 1e-3,
|
||||||
|
"mu={mu}: got {got}, asymptotic expectation {approx}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn tie_evidence_uses_two_sided() {
|
fn tie_evidence_uses_two_sided() {
|
||||||
let mut vars = VarStore::new();
|
let mut vars = VarStore::new();
|
||||||
@@ -201,7 +222,7 @@ mod tests {
|
|||||||
f.propagate(&mut vars);
|
f.propagate(&mut vars);
|
||||||
|
|
||||||
// For diff ~ N(0, 4), tie=true with margin=1: P(-1 < diff < 1) ≈ 0.383
|
// For diff ~ N(0, 4), tie=true with margin=1: P(-1 < diff < 1) ≈ 0.383
|
||||||
let ev = f.evidence_cached.unwrap();
|
let ev = f.log_evidence_cached.unwrap().exp();
|
||||||
assert!(ev > 0.35 && ev < 0.42);
|
assert!(ev > 0.35 && ev < 0.42);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+12
-15
@@ -46,8 +46,8 @@ impl DiffFactor {
|
|||||||
/// reaches.
|
/// reaches.
|
||||||
pub(crate) fn log_evidence(&self) -> f64 {
|
pub(crate) fn log_evidence(&self) -> f64 {
|
||||||
match self {
|
match self {
|
||||||
Self::Trunc(f) => f.evidence_cached.unwrap_or(1.0).ln(),
|
Self::Trunc(f) => f.log_evidence_cached.unwrap_or(0.0),
|
||||||
Self::Margin(f) => f.evidence_cached.unwrap_or(1.0).ln(),
|
Self::Margin(f) => f.log_evidence_cached.unwrap_or(0.0),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -568,15 +568,6 @@ impl<T: Time, D: Drift<T>> Game<'_, T, D> {
|
|||||||
let team_refs: Vec<&[Rating<T, D>]> = teams.iter().map(|t| t.as_slice()).collect();
|
let team_refs: Vec<&[Rating<T, D>]> = teams.iter().map(|t| t.as_slice()).collect();
|
||||||
Self::ranked(&team_refs, outcome, options)
|
Self::ranked(&team_refs, outcome, options)
|
||||||
}
|
}
|
||||||
|
|
||||||
#[doc(hidden)]
|
|
||||||
pub fn custom<S: crate::graph::Schedule>(
|
|
||||||
factors: &mut [crate::graph::BuiltinFactor],
|
|
||||||
vars: &mut crate::graph::VarStore,
|
|
||||||
schedule: &S,
|
|
||||||
) -> crate::graph::ScheduleReport {
|
|
||||||
schedule.run(factors, vars)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
@@ -733,9 +724,15 @@ mod tests {
|
|||||||
let c = p[2][0];
|
let c = p[2][0];
|
||||||
|
|
||||||
// T1 ULP shift: mu rounds to 25.0 (was 24.999999) under natural-parameter storage.
|
// T1 ULP shift: mu rounds to 25.0 (was 24.999999) under natural-parameter storage.
|
||||||
|
//
|
||||||
|
// The 1e-6-place values moved when `erfc_inv`'s sign error was fixed:
|
||||||
|
// this case runs at `p_draw = 0.5`, so it goes through `compute_margin`,
|
||||||
|
// and the margin is now 8.4e-8 from the exact quantile where it was
|
||||||
|
// 1.46e-7. Verified as movement *toward* analytic truth, not a
|
||||||
|
// regression — see `erfc_inv_matches_known_quantiles`.
|
||||||
assert_ulps_eq!(a, Gaussian::from_ms(25.0, 6.092561), epsilon = 1e-6);
|
assert_ulps_eq!(a, Gaussian::from_ms(25.0, 6.092561), epsilon = 1e-6);
|
||||||
assert_ulps_eq!(b, Gaussian::from_ms(33.379314, 6.483575), epsilon = 1e-6);
|
assert_ulps_eq!(b, Gaussian::from_ms(33.379315, 6.483576), epsilon = 1e-6);
|
||||||
assert_ulps_eq!(c, Gaussian::from_ms(16.620685, 6.483575), epsilon = 1e-6);
|
assert_ulps_eq!(c, Gaussian::from_ms(16.620685, 6.483576), epsilon = 1e-6);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -1248,7 +1245,7 @@ mod tests {
|
|||||||
);
|
);
|
||||||
assert_ulps_eq!(
|
assert_ulps_eq!(
|
||||||
p[1][0],
|
p[1][0],
|
||||||
Gaussian::from_ms(19.287197, 7.243465),
|
Gaussian::from_ms(19.287198285, 7.243465848),
|
||||||
epsilon = 1e-6
|
epsilon = 1e-6
|
||||||
);
|
);
|
||||||
assert_ulps_eq!(
|
assert_ulps_eq!(
|
||||||
@@ -1308,7 +1305,7 @@ mod tests {
|
|||||||
|
|
||||||
assert_ulps_eq!(
|
assert_ulps_eq!(
|
||||||
p[0][0],
|
p[0][0],
|
||||||
Gaussian::from_ms(31.674697, 7.501180),
|
Gaussian::from_ms(31.674698083, 7.501180037),
|
||||||
epsilon = 1e-6
|
epsilon = 1e-6
|
||||||
);
|
);
|
||||||
assert_ulps_eq!(
|
assert_ulps_eq!(
|
||||||
|
|||||||
+104
@@ -145,6 +145,45 @@ impl Gaussian {
|
|||||||
Self::from_mv(self.mu(), self.variance() + variance_delta)
|
Self::from_mv(self.mu(), self.variance() + variance_delta)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `P(X < x)` under this Gaussian.
|
||||||
|
///
|
||||||
|
/// The question a stopping rule asks: *how sure am I that this competitor's
|
||||||
|
/// true skill is below the cutoff?* Expressing that as a probability keeps
|
||||||
|
/// its meaning as sigma changes, where a `mu + z * sigma` band silently
|
||||||
|
/// means different confidence at different uncertainties — which is exactly
|
||||||
|
/// the regime a stopping rule operates in.
|
||||||
|
///
|
||||||
|
/// Accurate in the *lower* tail. For the upper tail use
|
||||||
|
/// [`Gaussian::probability_above`] rather than `1.0 - probability_below(x)`,
|
||||||
|
/// which cancels away every significant digit once the result is small.
|
||||||
|
///
|
||||||
|
/// An improper Gaussian (non-positive precision) has no defined mean, so
|
||||||
|
/// this returns `0.5` — the same convention `mu()` and `sigma()` follow.
|
||||||
|
#[must_use]
|
||||||
|
pub fn probability_below(&self, x: f64) -> f64 {
|
||||||
|
if self.pi <= 0.0 {
|
||||||
|
return 0.5;
|
||||||
|
}
|
||||||
|
crate::cdf(x, self.mu(), self.sigma())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `P(X > x)` under this Gaussian.
|
||||||
|
///
|
||||||
|
/// Computed as a survival function rather than `1 - cdf`, so it keeps full
|
||||||
|
/// relative precision in the upper tail: `1 - cdf` returns exactly zero
|
||||||
|
/// past about 8.3 sigma, where the true value is still 1e-19 and perfectly
|
||||||
|
/// representable. A stopping rule is evaluated precisely there — the
|
||||||
|
/// interesting cases are the ones near certainty.
|
||||||
|
///
|
||||||
|
/// An improper Gaussian returns `0.5`, as [`Gaussian::probability_below`].
|
||||||
|
#[must_use]
|
||||||
|
pub fn probability_above(&self, x: f64) -> f64 {
|
||||||
|
if self.pi <= 0.0 {
|
||||||
|
return 0.5;
|
||||||
|
}
|
||||||
|
crate::sf(x, self.mu(), self.sigma())
|
||||||
|
}
|
||||||
|
|
||||||
/// EP damping in natural-parameter space: `α·new + (1−α)·self`.
|
/// EP damping in natural-parameter space: `α·new + (1−α)·self`.
|
||||||
///
|
///
|
||||||
/// Used by within-game inference to stabilise oscillating fixed-point
|
/// Used by within-game inference to stabilise oscillating fixed-point
|
||||||
@@ -340,3 +379,68 @@ mod tests {
|
|||||||
assert!((damped.tau() - expected_tau).abs() < 1e-12);
|
assert!((damped.tau() - expected_tau).abs() < 1e-12);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tail_probability_tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn probability_below_matches_published_quantiles() {
|
||||||
|
let g = Gaussian::from_ms(0.0, 1.0);
|
||||||
|
for (x, expected) in [
|
||||||
|
(-1.959_963_984_540_054, 0.025),
|
||||||
|
(0.0, 0.5),
|
||||||
|
(1.281_551_565_544_6, 0.9),
|
||||||
|
(1.959_963_984_540_054, 0.975),
|
||||||
|
] {
|
||||||
|
let got = g.probability_below(x);
|
||||||
|
assert!(
|
||||||
|
(got - expected).abs() < 1e-12,
|
||||||
|
"P(X < {x}) = {got}, expected {expected}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_two_tails_partition_the_mass() {
|
||||||
|
let g = Gaussian::from_ms(3.0, 2.0);
|
||||||
|
for x in [-4.0f64, 0.0, 3.0, 7.5] {
|
||||||
|
let total = g.probability_below(x) + g.probability_above(x);
|
||||||
|
assert!((total - 1.0).abs() < 1e-15, "at {x}: {total}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The reason `probability_above` exists rather than `1 - probability_below`.
|
||||||
|
#[test]
|
||||||
|
fn probability_above_keeps_precision_where_the_complement_collapses() {
|
||||||
|
let g = Gaussian::from_ms(0.0, 1.0);
|
||||||
|
for (x, expected) in [(9.0f64, 1.128_588e-19), (20.0, 2.753_624e-89)] {
|
||||||
|
let got = g.probability_above(x);
|
||||||
|
assert!(
|
||||||
|
(got - expected).abs() / expected < 1e-6,
|
||||||
|
"P(X > {x}) = {got}, expected ~{expected}"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
1.0 - g.probability_below(x),
|
||||||
|
0.0,
|
||||||
|
"the complement should still collapse at {x}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_scaled_gaussian_shifts_and_stretches() {
|
||||||
|
let g = Gaussian::from_ms(25.0, 6.0);
|
||||||
|
assert!((g.probability_below(25.0) - 0.5).abs() < 1e-15);
|
||||||
|
// One sigma either side of the mean.
|
||||||
|
assert!((g.probability_below(31.0) - 0.841_344_746_068_543).abs() < 1e-12);
|
||||||
|
assert!((g.probability_above(19.0) - 0.841_344_746_068_543).abs() < 1e-12);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_improper_gaussian_is_uninformative_rather_than_nan() {
|
||||||
|
let improper = Gaussian::from_ms(0.0, f64::INFINITY);
|
||||||
|
assert_eq!(improper.probability_below(5.0), 0.5);
|
||||||
|
assert_eq!(improper.probability_above(5.0), 0.5);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,20 +0,0 @@
|
|||||||
//! Factor-graph public API.
|
|
||||||
//!
|
|
||||||
//! Named `graph` rather than `factors` because the private implementation
|
|
||||||
//! module beside it is `factor`: two module paths differing by one character,
|
|
||||||
//! one public and one not, was a standing invitation to import the wrong one.
|
|
||||||
//!
|
|
||||||
//! The factor types, `VarStore` and the `Schedule` trait are public so custom
|
|
||||||
//! schedules can be written against them.
|
|
||||||
//!
|
|
||||||
//! Building a factor graph by hand goes through `Game::custom`, which is
|
|
||||||
//! deliberately `#[doc(hidden)]`: it works, but its signature is not yet
|
|
||||||
//! considered stable API and so is not listed in these docs.
|
|
||||||
|
|
||||||
pub use crate::{
|
|
||||||
factor::{
|
|
||||||
BuiltinFactor, Factor, VarId, VarStore, margin::MarginFactor, rank_diff::RankDiffFactor,
|
|
||||||
team_sum::TeamSumFactor, trunc::TruncFactor,
|
|
||||||
},
|
|
||||||
schedule::{EpsilonOrMax, Schedule, ScheduleReport},
|
|
||||||
};
|
|
||||||
+685
-44
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,99 @@
|
|||||||
|
//! Posterior of a linear combination of competitors.
|
||||||
|
//!
|
||||||
|
//! Every accessor on `History` returns a per-competitor marginal, and almost
|
||||||
|
//! nothing a consumer publishes is one competitor: "can we tell these two
|
||||||
|
//! apart" is a difference, "what was this round worth" is a sum. Combining
|
||||||
|
//! marginals means assuming the competitors are independent, and they are
|
||||||
|
//! correlated through every event they share — which is the mechanism the model
|
||||||
|
//! exists to exploit.
|
||||||
|
//!
|
||||||
|
//! Measured on a five-competitor round robin, the exact correlation is +0.857,
|
||||||
|
//! so `sqrt(sa^2 + sb^2)` overstates the width of a difference by 2.6x.
|
||||||
|
|
||||||
|
/// Solve `A z = b` for a symmetric positive-definite `A`, by Cholesky.
|
||||||
|
///
|
||||||
|
/// `a` is row-major and is consumed as scratch.
|
||||||
|
///
|
||||||
|
/// Returns `None` if the matrix is not positive-definite, which for a precision
|
||||||
|
/// matrix means the model is improper — a competitor with no prior and no
|
||||||
|
/// evidence.
|
||||||
|
pub(crate) fn solve_spd(mut a: Vec<f64>, b: &[f64]) -> Option<Vec<f64>> {
|
||||||
|
let n = b.len();
|
||||||
|
debug_assert_eq!(a.len(), n * n);
|
||||||
|
|
||||||
|
// In-place Cholesky: A = L L^T, lower triangle.
|
||||||
|
for j in 0..n {
|
||||||
|
let mut d = a[j * n + j];
|
||||||
|
for k in 0..j {
|
||||||
|
d -= a[j * n + k] * a[j * n + k];
|
||||||
|
}
|
||||||
|
// Explicit rather than `!(d > 0.0)`: a NaN pivot must fail here too,
|
||||||
|
// and a negated comparison would let it through as "not positive".
|
||||||
|
if d.is_nan() || d <= 0.0 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let d = d.sqrt();
|
||||||
|
a[j * n + j] = d;
|
||||||
|
|
||||||
|
for i in j + 1..n {
|
||||||
|
let mut s = a[i * n + j];
|
||||||
|
for k in 0..j {
|
||||||
|
s -= a[i * n + k] * a[j * n + k];
|
||||||
|
}
|
||||||
|
a[i * n + j] = s / d;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Forward substitution, then back substitution.
|
||||||
|
let mut z = b.to_vec();
|
||||||
|
for i in 0..n {
|
||||||
|
let mut s = z[i];
|
||||||
|
for k in 0..i {
|
||||||
|
s -= a[i * n + k] * z[k];
|
||||||
|
}
|
||||||
|
z[i] = s / a[i * n + i];
|
||||||
|
}
|
||||||
|
for i in (0..n).rev() {
|
||||||
|
let mut s = z[i];
|
||||||
|
for k in i + 1..n {
|
||||||
|
s -= a[k * n + i] * z[k];
|
||||||
|
}
|
||||||
|
z[i] = s / a[i * n + i];
|
||||||
|
}
|
||||||
|
|
||||||
|
Some(z)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn solves_a_known_system() {
|
||||||
|
// [[4, 1], [1, 3]] z = [1, 2] => z = [1/11, 7/11]
|
||||||
|
let a = vec![4.0, 1.0, 1.0, 3.0];
|
||||||
|
let z = solve_spd(a, &[1.0, 2.0]).unwrap();
|
||||||
|
assert!((z[0] - 1.0 / 11.0).abs() < 1e-12, "{z:?}");
|
||||||
|
assert!((z[1] - 7.0 / 11.0).abs() < 1e-12, "{z:?}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn recovers_the_inverse_diagonal() {
|
||||||
|
// A = [[2, -1, 0], [-1, 2, -1], [0, -1, 2]]; inverse diagonal is
|
||||||
|
// [0.75, 1.0, 0.75].
|
||||||
|
let a = vec![2.0, -1.0, 0.0, -1.0, 2.0, -1.0, 0.0, -1.0, 2.0];
|
||||||
|
for (i, expected) in [0.75, 1.0, 0.75].into_iter().enumerate() {
|
||||||
|
let mut e = vec![0.0; 3];
|
||||||
|
e[i] = 1.0;
|
||||||
|
let z = solve_spd(a.clone(), &e).unwrap();
|
||||||
|
assert!((z[i] - expected).abs() < 1e-12, "row {i}: {z:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_a_non_positive_definite_matrix() {
|
||||||
|
// Singular: the second row is a multiple of the first.
|
||||||
|
let a = vec![1.0, 2.0, 2.0, 4.0];
|
||||||
|
assert!(solve_spd(a, &[1.0, 1.0]).is_none());
|
||||||
|
}
|
||||||
|
}
|
||||||
+312
-39
@@ -121,8 +121,8 @@ mod event_builder;
|
|||||||
pub(crate) mod factor;
|
pub(crate) mod factor;
|
||||||
mod game;
|
mod game;
|
||||||
pub mod gaussian;
|
pub mod gaussian;
|
||||||
pub mod graph;
|
|
||||||
mod history;
|
mod history;
|
||||||
|
mod joint;
|
||||||
mod key_table;
|
mod key_table;
|
||||||
mod matrix;
|
mod matrix;
|
||||||
mod observer;
|
mod observer;
|
||||||
@@ -130,14 +130,13 @@ mod outcome;
|
|||||||
mod predict;
|
mod predict;
|
||||||
pub(crate) mod quadrature;
|
pub(crate) mod quadrature;
|
||||||
mod rating;
|
mod rating;
|
||||||
pub(crate) mod schedule;
|
|
||||||
pub mod storage;
|
pub mod storage;
|
||||||
|
|
||||||
pub use acquisition::expected_information_gain;
|
pub use acquisition::expected_information_gain;
|
||||||
pub use competitor::Competitor;
|
pub use competitor::Competitor;
|
||||||
pub use convergence::{ConvergenceOptions, ConvergenceReport};
|
pub use convergence::{ConvergenceOptions, ConvergenceReport};
|
||||||
pub use drift::{ConstantDrift, Drift};
|
pub use drift::{ConstantDrift, Drift};
|
||||||
pub use error::InferenceError;
|
pub use error::{InferenceError, UnknownKeys};
|
||||||
pub use event::{Event, Member, Team};
|
pub use event::{Event, Member, Team};
|
||||||
pub use event_builder::EventBuilder;
|
pub use event_builder::EventBuilder;
|
||||||
pub use game::{Game, GameOptions, OwnedGame};
|
pub use game::{Game, GameOptions, OwnedGame};
|
||||||
@@ -149,7 +148,6 @@ pub use observer::{NullObserver, Observer};
|
|||||||
pub use outcome::Outcome;
|
pub use outcome::Outcome;
|
||||||
pub use predict::Prediction;
|
pub use predict::Prediction;
|
||||||
pub use rating::Rating;
|
pub use rating::Rating;
|
||||||
pub use schedule::ScheduleReport;
|
|
||||||
pub use time::{Time, Untimed};
|
pub use time::{Time, Untimed};
|
||||||
|
|
||||||
pub const BETA: f64 = 1.0;
|
pub const BETA: f64 = 1.0;
|
||||||
@@ -158,6 +156,23 @@ pub const SIGMA: f64 = BETA * 6.0;
|
|||||||
pub const GAMMA: f64 = BETA * 0.03;
|
pub const GAMMA: f64 = BETA * 0.03;
|
||||||
pub const P_DRAW: f64 = 0.0;
|
pub const P_DRAW: f64 = 0.0;
|
||||||
pub const EPSILON: f64 = 1e-6;
|
pub const EPSILON: f64 = 1e-6;
|
||||||
|
/// Default cap on convergence sweeps.
|
||||||
|
///
|
||||||
|
/// **This is a floor, not a recommendation.** It is adequate for small
|
||||||
|
/// histories and is quickly outgrown: a history of 400 events over 100
|
||||||
|
/// competitors already stops here with a final step of ~7e-3 against the 1e-6
|
||||||
|
/// default tolerance — four orders of magnitude short — and a dense joint model
|
||||||
|
/// of ~2,000 nodes over ~3,300 events has been measured needing 76 to 161.
|
||||||
|
///
|
||||||
|
/// Overrunning it is not an error, and deliberately so: `converge` returns a
|
||||||
|
/// [`ConvergenceReport`] whose `converged` flag says what happened. But a fit
|
||||||
|
/// that stopped short is *wrong by a little*, which is the worst available
|
||||||
|
/// failure — every rating is finite and ordered sensibly, and nothing in the
|
||||||
|
/// numbers themselves says they were still moving. Read the report; the type is
|
||||||
|
/// `#[must_use]` for that reason.
|
||||||
|
///
|
||||||
|
/// Raise it via [`ConvergenceOptions`]. Convergence cost is roughly linear in
|
||||||
|
/// the cap, and for anything but a toy the extra sweeps are milliseconds.
|
||||||
pub const ITERATIONS: usize = 30;
|
pub const ITERATIONS: usize = 30;
|
||||||
|
|
||||||
/// Largest team count `History::predict_outcome` will enumerate.
|
/// Largest team count `History::predict_outcome` will enumerate.
|
||||||
@@ -214,24 +229,47 @@ impl From<Index> for usize {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Complementary error function.
|
||||||
|
///
|
||||||
|
/// # Why every transcendental in this crate goes through `libm`
|
||||||
|
///
|
||||||
|
/// IEEE 754 specifies the basic operations and `sqrt` exactly, but says nothing
|
||||||
|
/// about `exp`, `log` or `erf`. `std`'s versions delegate to the *system* math
|
||||||
|
/// library, so they differ between platforms: measured here, `f64::exp` and
|
||||||
|
/// `libm::exp` disagree on 9.7% of inputs and `f64::ln` / `libm::log` on 5.0%,
|
||||||
|
/// each by one ULP.
|
||||||
|
///
|
||||||
|
/// Inference is an iterative fixed point, so a one-ULP difference can change an
|
||||||
|
/// iteration count and therefore the answer by more than one ULP. Routing every
|
||||||
|
/// transcendental through `libm` makes a fit reproducible across platforms, not
|
||||||
|
/// just across thread counts as `tests/determinism.rs` already checks.
|
||||||
|
///
|
||||||
|
/// **So: use `libm::exp` / `libm::log` in inference code, never `f64::exp` /
|
||||||
|
/// `f64::ln`.** `sqrt` is exempt — IEEE specifies it exactly, so `f64::sqrt` is
|
||||||
|
/// already portable. Test code may use whichever is clearer.
|
||||||
|
///
|
||||||
|
/// It costs nothing: `Batch::iteration` measured -2.7% [-5.7%, -0.3%] with the
|
||||||
|
/// whole set swapped.
|
||||||
|
///
|
||||||
|
/// Delegates to `libm`, which is the Rust port of FDLIBM and accurate to about
|
||||||
|
/// one ULP. This replaced a Numerical Recipes `erfcc` rational approximation
|
||||||
|
/// whose documented bound was 1.2e-7 *relative* — measured at ~1e-7 across the
|
||||||
|
/// whole range, and the binding accuracy constraint on the entire crate.
|
||||||
|
///
|
||||||
|
/// The swap is free. 98% of the arguments inference passes here have
|
||||||
|
/// `|x| < 0.84375`, which is exactly where FDLIBM skips the exponential
|
||||||
|
/// entirely, so the longer polynomial costs nothing on the distribution that
|
||||||
|
/// actually occurs: `Batch::iteration` moved -1.6% [-4.7%, +0.9%], p = 0.31.
|
||||||
|
///
|
||||||
|
/// What it bought: `compute_margin` went from 8.4e-8 to 1.7e-16 against exact
|
||||||
|
/// quantiles, `cdf(mu, mu, sigma)` is now exactly 0.5, and `sf + cdf` sums to
|
||||||
|
/// one within a single ULP where it was 3e-8 out.
|
||||||
fn erfc(x: f64) -> f64 {
|
fn erfc(x: f64) -> f64 {
|
||||||
let z = x.abs();
|
libm::erfc(x)
|
||||||
let t = 1.0 / (1.0 + z / 2.0);
|
|
||||||
|
|
||||||
let a = -0.82215223 + t * 0.17087277;
|
|
||||||
let b = 1.48851587 + t * a;
|
|
||||||
let c = -1.13520398 + t * b;
|
|
||||||
let d = 0.27886807 + t * c;
|
|
||||||
let e = -0.18628806 + t * d;
|
|
||||||
let f = 0.09678418 + t * e;
|
|
||||||
let g = 0.37409196 + t * f;
|
|
||||||
let h = 1.00002368 + t * g;
|
|
||||||
|
|
||||||
let r = t * (-z * z - 1.26551223 + t * h).exp();
|
|
||||||
|
|
||||||
if x >= 0.0 { r } else { 2.0 - r }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The previous Numerical Recipes `erfcc`, kept only so the timing test can
|
||||||
|
/// compare both in one binary. Removed once the comparison is recorded.
|
||||||
fn erfc_inv(mut y: f64) -> f64 {
|
fn erfc_inv(mut y: f64) -> f64 {
|
||||||
if y >= 2.0 {
|
if y >= 2.0 {
|
||||||
return f64::NEG_INFINITY;
|
return f64::NEG_INFINITY;
|
||||||
@@ -247,14 +285,22 @@ fn erfc_inv(mut y: f64) -> f64 {
|
|||||||
y = 2.0 - y;
|
y = 2.0 - y;
|
||||||
}
|
}
|
||||||
|
|
||||||
let t = (-2.0 * (y / 2.0).ln()).sqrt();
|
let t = libm::sqrt(-2.0 * libm::log(y / 2.0));
|
||||||
|
|
||||||
let mut x = FRAC_1_SQRT_2 * ((2.30753 + t * 0.27061) / (1.0 + t * (0.99229 + t * 0.04481)) - t);
|
// The leading coefficient is NEGATIVE. `rational - t` is negative here, so
|
||||||
|
// a positive coefficient mirrors the starting point to `-x0` — the
|
||||||
|
// reflection of the root. Newton then has to cross the origin to get back,
|
||||||
|
// which a fixed iteration count does not manage: measured against the true
|
||||||
|
// value, `erfc_inv(0.1)` returned 1.044 instead of 1.16309, and the error
|
||||||
|
// grew as y shrank until `compute_margin` stopped being monotone in
|
||||||
|
// `p_draw` altogether.
|
||||||
|
let mut x =
|
||||||
|
-FRAC_1_SQRT_2 * ((2.30753 + t * 0.27061) / (1.0 + t * (0.99229 + t * 0.04481)) - t);
|
||||||
|
|
||||||
for _ in 0..3 {
|
for _ in 0..3 {
|
||||||
let err = erfc(x) - y;
|
let err = erfc(x) - y;
|
||||||
|
|
||||||
x += err / (FRAC_2_SQRT_PI * (-(x.powi(2))).exp() - x * err)
|
x += err / (FRAC_2_SQRT_PI * libm::exp(-(x * x)) - x * err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if y < 1.0 { x } else { -x }
|
if y < 1.0 { x } else { -x }
|
||||||
@@ -283,7 +329,7 @@ pub(crate) fn cdf(x: f64, mu: f64, sigma: f64) -> f64 {
|
|||||||
/// away every significant digit the tail had: measured against this function,
|
/// away every significant digit the tail had: measured against this function,
|
||||||
/// `1 - cdf` carries 7% error by four sigma past the mean and returns exactly
|
/// `1 - cdf` carries 7% error by four sigma past the mean and returns exactly
|
||||||
/// zero beyond about 8.3 sigma — where the true value is still 1e-19 and
|
/// zero beyond about 8.3 sigma — where the true value is still 1e-19 and
|
||||||
/// perfectly representable. `erfc` itself holds ~1e-7 *relative* accuracy down
|
/// perfectly representable. `erfc` holds *relative* accuracy all the way down
|
||||||
/// to 1e-296, so the precision is there to keep; only the subtraction threw it
|
/// to 1e-296, so the precision is there to keep; only the subtraction threw it
|
||||||
/// away.
|
/// away.
|
||||||
///
|
///
|
||||||
@@ -305,7 +351,7 @@ fn erfcx(x: f64) -> f64 {
|
|||||||
// Below the crossover neither factor is extreme: erfc is O(1) and
|
// Below the crossover neither factor is extreme: erfc is O(1) and
|
||||||
// exp(x^2) is at most e^4, so the direct product is exact enough and
|
// exp(x^2) is at most e^4, so the direct product is exact enough and
|
||||||
// cheaper than the continued fraction.
|
// cheaper than the continued fraction.
|
||||||
(x * x).exp() * erfc(x)
|
libm::exp(x * x) * erfc(x)
|
||||||
} else {
|
} else {
|
||||||
// erfcx(x) = 1/sqrt(pi) * 1/(x + (1/2)/(x + 1/(x + (3/2)/(x + ...)))),
|
// erfcx(x) = 1/sqrt(pi) * 1/(x + (1/2)/(x + 1/(x + (3/2)/(x + ...)))),
|
||||||
// evaluated by backward recurrence. Converges quickly for x >= 2 and,
|
// evaluated by backward recurrence. Converges quickly for x >= 2 and,
|
||||||
@@ -318,9 +364,74 @@ fn erfcx(x: f64) -> f64 {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `ln` of the normal density at `x`.
|
||||||
|
///
|
||||||
|
/// The density itself underflows to zero past about 38 sigma, and `ln` of a
|
||||||
|
/// clamped zero is -708 whatever the truth was. The log form is a polynomial:
|
||||||
|
/// it stays exact at any separation, and the values it produces (-5001 nats at
|
||||||
|
/// 100 sigma, -500001 at 1000) are perfectly representable.
|
||||||
|
pub(crate) fn ln_pdf(x: f64, mu: f64, sigma: f64) -> f64 {
|
||||||
|
let z = (x - mu) / sigma;
|
||||||
|
-libm::log(SQRT_TAU * sigma) - 0.5 * z * z
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `ln P(X > x)` for `X ~ N(mu, sigma^2)`.
|
||||||
|
///
|
||||||
|
/// In the upper tail the `exp(-z^2 / 2)` common to the tail integral is
|
||||||
|
/// factored out analytically via `erfcx`, so this never underflows — where
|
||||||
|
/// `sf(..).ln()` bottoms out at -708 once `erfc` itself reaches zero.
|
||||||
|
pub(crate) fn ln_sf(x: f64, mu: f64, sigma: f64) -> f64 {
|
||||||
|
let z = (x - mu) / sigma;
|
||||||
|
|
||||||
|
if z > 0.0 {
|
||||||
|
// ln(0.5 * erfc(z/sqrt2)) with erfc(y) = exp(-y^2) * erfcx(y).
|
||||||
|
-std::f64::consts::LN_2 - 0.5 * z * z + libm::log(erfcx(z / SQRT_2))
|
||||||
|
} else {
|
||||||
|
// The mass here is at least a half; nothing to lose.
|
||||||
|
libm::log(sf(x, mu, sigma))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `ln P(lo < X < hi)` for `X ~ N(mu, sigma^2)`.
|
||||||
|
///
|
||||||
|
/// When the interval sits in a tail both endpoint probabilities underflow
|
||||||
|
/// together, so their difference is taken in scaled form with the shared
|
||||||
|
/// exponential factored out. When it straddles the mean nothing is small and
|
||||||
|
/// the direct difference is exact.
|
||||||
|
pub(crate) fn ln_interval(lo: f64, hi: f64, mu: f64, sigma: f64) -> f64 {
|
||||||
|
let z_lo = (lo - mu) / sigma;
|
||||||
|
let z_hi = (hi - mu) / sigma;
|
||||||
|
|
||||||
|
if z_hi <= z_lo {
|
||||||
|
return f64::NEG_INFINITY;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fold a lower-tail interval onto the upper tail; the normal is symmetric.
|
||||||
|
let (near, far) = if z_lo >= 0.0 {
|
||||||
|
(z_lo, z_hi)
|
||||||
|
} else if z_hi <= 0.0 {
|
||||||
|
(-z_hi, -z_lo)
|
||||||
|
} else {
|
||||||
|
// Straddles the mean: the interval holds a non-negligible share of the
|
||||||
|
// mass, so neither endpoint is near enough to 1 to cancel.
|
||||||
|
return libm::log((cdf(hi, mu, sigma) - cdf(lo, mu, sigma)).max(f64::MIN_POSITIVE));
|
||||||
|
};
|
||||||
|
|
||||||
|
let (a, b) = (near / SQRT_2, far / SQRT_2);
|
||||||
|
// b > a >= 0, so this ratio of exponentials is at most 1 and cannot overflow.
|
||||||
|
let scale = libm::exp(a * a - b * b);
|
||||||
|
let bracket = erfcx(a) - scale * erfcx(b);
|
||||||
|
|
||||||
|
if bracket <= 0.0 {
|
||||||
|
return f64::NEG_INFINITY;
|
||||||
|
}
|
||||||
|
|
||||||
|
-std::f64::consts::LN_2 - a * a + libm::log(bracket)
|
||||||
|
}
|
||||||
|
|
||||||
fn pdf(x: f64, mu: f64, sigma: f64) -> f64 {
|
fn pdf(x: f64, mu: f64, sigma: f64) -> f64 {
|
||||||
let normalizer = (SQRT_TAU * sigma).powi(-1);
|
let normalizer = (SQRT_TAU * sigma).powi(-1);
|
||||||
let functional = (-((x - mu).powi(2)) / (2.0 * sigma.powi(2))).exp();
|
let functional = libm::exp(-((x - mu) * (x - mu)) / (2.0 * sigma * sigma));
|
||||||
|
|
||||||
normalizer * functional
|
normalizer * functional
|
||||||
}
|
}
|
||||||
@@ -399,7 +510,7 @@ fn v_w(mu: f64, sigma: f64, margin: f64, tie: bool) -> (f64, f64) {
|
|||||||
let (v, u) = if alpha > 0.0 {
|
let (v, u) = if alpha > 0.0 {
|
||||||
// beta > alpha > 0, so this ratio of exponentials is at most 1 and
|
// beta > alpha > 0, so this ratio of exponentials is at most 1 and
|
||||||
// cannot overflow.
|
// cannot overflow.
|
||||||
let scale = (0.5 * (alpha * alpha - beta * beta)).exp();
|
let scale = libm::exp(0.5 * (alpha * alpha - beta * beta));
|
||||||
let denominator = 0.5 * (erfcx(alpha / SQRT_2) - scale * erfcx(beta / SQRT_2));
|
let denominator = 0.5 * (erfcx(alpha / SQRT_2) - scale * erfcx(beta / SQRT_2));
|
||||||
|
|
||||||
(
|
(
|
||||||
@@ -587,7 +698,7 @@ pub fn quality(rating_groups: &[&[Gaussian]], beta: f64) -> f64 {
|
|||||||
let e_arg = (-0.5 * &start * &middle.inverse() * &end).determinant();
|
let e_arg = (-0.5 * &start * &middle.inverse() * &end).determinant();
|
||||||
let s_arg = ata.determinant() / middle.determinant();
|
let s_arg = ata.determinant() / middle.determinant();
|
||||||
|
|
||||||
e_arg.exp() * s_arg.sqrt()
|
libm::exp(e_arg) * s_arg.sqrt()
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
@@ -602,9 +713,9 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Upper-tail values of the standard normal, from published tables. The
|
/// Upper-tail values of the standard normal, from published tables. The
|
||||||
/// point is not the digits — `erfc` only carries ~1e-7 relative — but that
|
/// point is not the digits — these are 7-digit table values — but that a
|
||||||
/// a number comes back at all: `1 - cdf` returned exactly zero for every
|
/// number comes back at all: `1 - cdf` returned exactly zero for every one
|
||||||
/// one of these.
|
/// of these.
|
||||||
#[test]
|
#[test]
|
||||||
fn survival_function_survives_the_far_tail() {
|
fn survival_function_survives_the_far_tail() {
|
||||||
for (z, expected) in [
|
for (z, expected) in [
|
||||||
@@ -616,7 +727,7 @@ mod tests {
|
|||||||
let got = sf(z, 0.0, 1.0);
|
let got = sf(z, 0.0, 1.0);
|
||||||
assert!(got > 0.0, "sf({z}) collapsed to zero");
|
assert!(got > 0.0, "sf({z}) collapsed to zero");
|
||||||
assert!(
|
assert!(
|
||||||
(got - expected).abs() / expected < 1e-6,
|
(got - expected).abs() / expected < 1e-6, // published table values, 7 digits
|
||||||
"sf({z}) = {got}, expected ~{expected}"
|
"sf({z}) = {got}, expected ~{expected}"
|
||||||
);
|
);
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
@@ -634,11 +745,8 @@ mod tests {
|
|||||||
for z in [-4.0f64, -1.0, 0.0, 0.5, 1.0, 2.0, 3.0, 4.0] {
|
for z in [-4.0f64, -1.0, 0.0, 0.5, 1.0, 2.0, 3.0, 4.0] {
|
||||||
let naive = 1.0 - cdf(z, 0.0, 1.0);
|
let naive = 1.0 - cdf(z, 0.0, 1.0);
|
||||||
let direct = sf(z, 0.0, 1.0);
|
let direct = sf(z, 0.0, 1.0);
|
||||||
// Bounded by `erfc`'s own ~1e-7 relative error, not by the
|
|
||||||
// subtraction: the two forms evaluate `erfc` at different points
|
|
||||||
// and the approximation is not exactly antisymmetric.
|
|
||||||
assert!(
|
assert!(
|
||||||
(naive - direct).abs() < 1e-6,
|
(naive - direct).abs() < 1e-15,
|
||||||
"z={z}: naive {naive} vs direct {direct}"
|
"z={z}: naive {naive} vs direct {direct}"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -648,9 +756,7 @@ mod tests {
|
|||||||
fn survival_and_cdf_partition_the_mass() {
|
fn survival_and_cdf_partition_the_mass() {
|
||||||
for z in [-3.0f64, -0.5, 0.0, 1.0, 2.5] {
|
for z in [-3.0f64, -0.5, 0.0, 1.0, 2.5] {
|
||||||
let total = sf(z, 1.0, 2.0) + cdf(z, 1.0, 2.0);
|
let total = sf(z, 1.0, 2.0) + cdf(z, 1.0, 2.0);
|
||||||
// `erfc(z) + erfc(-z) == 2` only to the accuracy of the
|
assert!((total - 1.0).abs() < 1e-15, "z={z}: {total}");
|
||||||
// approximation, which is ~1e-7 relative.
|
|
||||||
assert!((total - 1.0).abs() < 1e-6, "z={z}: {total}");
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -661,7 +767,7 @@ mod tests {
|
|||||||
let direct = (x * x).exp() * erfc(x);
|
let direct = (x * x).exp() * erfc(x);
|
||||||
let scaled = erfcx(x);
|
let scaled = erfcx(x);
|
||||||
assert!(
|
assert!(
|
||||||
(direct - scaled).abs() / scaled < 1e-6,
|
(direct - scaled).abs() / scaled < 1e-14,
|
||||||
"x={x}: direct {direct} vs erfcx {scaled}"
|
"x={x}: direct {direct} vs erfcx {scaled}"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -746,6 +852,173 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `erfc_inv`'s initial guess had the wrong sign, putting Newton on the
|
||||||
|
/// mirror image of the root. Three fixed iterations could not cross back,
|
||||||
|
/// so the error grew as the argument shrank: at `p_draw = 0.99` the margin
|
||||||
|
/// came out 0.503 where the answer is 2.576.
|
||||||
|
#[test]
|
||||||
|
fn erfc_inv_matches_known_quantiles() {
|
||||||
|
// sqrt(2) * erfc_inv(1 - p) is the standard normal quantile
|
||||||
|
// Phi^-1((1 + p) / 2).
|
||||||
|
for (p, exact) in [
|
||||||
|
(0.5f64, 0.674_489_750_196_081_7f64),
|
||||||
|
(0.9, 1.644_853_626_951_472_7),
|
||||||
|
(0.95, 1.959_963_984_540_054_2),
|
||||||
|
(0.99, 2.575_829_303_548_9),
|
||||||
|
(0.999, 3.290_526_731_491_896_4),
|
||||||
|
] {
|
||||||
|
let got = SQRT_2 * erfc_inv(1.0 - p);
|
||||||
|
assert!(
|
||||||
|
(got - exact).abs() / exact < 1e-14,
|
||||||
|
"p={p}: got {got}, exact {exact}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The draw margin must grow with the draw probability. It did not: it ran
|
||||||
|
/// 0.674 -> 1.476 -> 0.503 -> 0.982 as `p_draw` went 0.5 -> 0.9 -> 0.99 ->
|
||||||
|
/// 0.999, which is not a rounding error but a broken function.
|
||||||
|
/// Deep in the tail the accuracy limit is the *caller's* argument, not this
|
||||||
|
/// function.
|
||||||
|
///
|
||||||
|
/// `compute_margin(0.999999, ..)` computes `1.0 - p_draw`, and 0.999999 is
|
||||||
|
/// not representable: the subtraction cancels and leaves 2.9e-11 of
|
||||||
|
/// relative error in the argument before `erfc_inv` is even entered. Given
|
||||||
|
/// an exactly-representable argument the result is good to 1.8e-16, so this
|
||||||
|
/// is inherent to taking `p_draw` near one rather than something to fix
|
||||||
|
/// here. At `p_draw = 0.999` the whole path is still accurate to 4e-16.
|
||||||
|
///
|
||||||
|
/// Worth pinning: measured against a 70-digit reference, `puruspe`'s
|
||||||
|
/// `inverfc` returns the identical wrong value for the identical reason,
|
||||||
|
/// which is what makes it clear the fault is upstream of both.
|
||||||
|
#[test]
|
||||||
|
fn erfc_inv_is_exact_given_an_exactly_representable_argument() {
|
||||||
|
// erfc(z / sqrt2) = 1e-6 exactly, so z = Phi^-1(0.9999995).
|
||||||
|
let got = SQRT_2 * erfc_inv(1e-6);
|
||||||
|
let exact = 4.891_638_475_698_59;
|
||||||
|
assert!(
|
||||||
|
(got - exact).abs() / exact < 1e-14,
|
||||||
|
"got {got}, exact {exact}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn compute_margin_is_monotone_in_the_draw_probability() {
|
||||||
|
let mut previous = 0.0;
|
||||||
|
for p_draw in [
|
||||||
|
0.001f64, 0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99, 0.999, 0.9999,
|
||||||
|
] {
|
||||||
|
let margin = compute_margin(p_draw, 1.0);
|
||||||
|
assert!(
|
||||||
|
margin > previous,
|
||||||
|
"p_draw={p_draw}: margin {margin} did not exceed {previous}"
|
||||||
|
);
|
||||||
|
previous = margin;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Round-tripping the margin back through the model's own CDF must recover
|
||||||
|
/// the draw probability it was built from.
|
||||||
|
#[test]
|
||||||
|
fn compute_margin_round_trips_through_the_cdf() {
|
||||||
|
for p_draw in [0.001f64, 0.1, 0.5, 0.9, 0.99, 0.999] {
|
||||||
|
for sd in [0.5f64, 1.0, 5.892_557] {
|
||||||
|
let margin = compute_margin(p_draw, sd);
|
||||||
|
// P(|X| < margin) for X ~ N(0, sd^2).
|
||||||
|
let recovered = 1.0 - 2.0 * cdf(-margin, 0.0, sd);
|
||||||
|
assert!(
|
||||||
|
(recovered - p_draw).abs() < 1e-14,
|
||||||
|
"p_draw={p_draw} sd={sd}: recovered {recovered}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `ln_pdf`, `ln_sf` and `ln_interval` exist so evidence stays exact where
|
||||||
|
/// the linear forms underflow. Past ~38 sigma the linear value is zero and
|
||||||
|
/// its log is whatever floor it was clamped to.
|
||||||
|
#[test]
|
||||||
|
fn log_space_helpers_stay_exact_where_the_linear_forms_underflow() {
|
||||||
|
for z in [40.0f64, 60.0, 100.0, 1000.0] {
|
||||||
|
assert_eq!(pdf(z, 0.0, 1.0), 0.0, "pdf should underflow at {z}");
|
||||||
|
assert_eq!(sf(z, 0.0, 1.0), 0.0, "sf should underflow at {z}");
|
||||||
|
|
||||||
|
let lp = ln_pdf(z, 0.0, 1.0);
|
||||||
|
let expected_lp = -(SQRT_TAU).ln() - 0.5 * z * z;
|
||||||
|
assert!(
|
||||||
|
(lp - expected_lp).abs() < 1e-9,
|
||||||
|
"ln_pdf({z}) = {lp}, expected {expected_lp}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let ls = ln_sf(z, 0.0, 1.0);
|
||||||
|
// ln Phi(-z) ~ -z^2/2 - ln(z) - ln(sqrt(2 pi)) for large z.
|
||||||
|
let approx = -0.5 * z * z - z.ln() - SQRT_TAU.ln();
|
||||||
|
assert!(
|
||||||
|
(ls - approx).abs() / approx.abs() < 1e-3,
|
||||||
|
"ln_sf({z}) = {ls}, asymptote {approx}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
ls < f64::MIN_POSITIVE.ln(),
|
||||||
|
"ln_sf({z}) still on the clamp floor"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Where nothing underflows, the log helpers must agree with the direct
|
||||||
|
/// forms exactly enough that nothing else in the crate shifts.
|
||||||
|
#[test]
|
||||||
|
fn log_space_helpers_agree_with_the_linear_forms_in_range() {
|
||||||
|
for z in [-3.0f64, -1.0, 0.0, 1.0, 2.0, 5.0, 10.0, 20.0] {
|
||||||
|
let lp = ln_pdf(z, 0.5, 2.0);
|
||||||
|
let direct_pdf = pdf(z, 0.5, 2.0);
|
||||||
|
assert!(
|
||||||
|
(lp.exp() - direct_pdf).abs() <= 1e-12 * direct_pdf,
|
||||||
|
"ln_pdf at {z}: {} vs {direct_pdf}",
|
||||||
|
lp.exp()
|
||||||
|
);
|
||||||
|
|
||||||
|
let ls = ln_sf(z, 0.5, 2.0);
|
||||||
|
let direct = sf(z, 0.5, 2.0);
|
||||||
|
assert!(
|
||||||
|
(ls.exp() - direct).abs() <= 1e-13 * direct.max(1e-300),
|
||||||
|
"ln_sf at {z}: {} vs {direct}",
|
||||||
|
ls.exp()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ln_interval_matches_the_direct_difference_when_nothing_is_small() {
|
||||||
|
for mu in [-2.0f64, 0.0, 0.5, 2.0] {
|
||||||
|
let direct = cdf(1.0, mu, 1.0) - cdf(-1.0, mu, 1.0);
|
||||||
|
let logged = ln_interval(-1.0, 1.0, mu, 1.0).exp();
|
||||||
|
assert!(
|
||||||
|
(logged - direct).abs() <= 1e-13 * direct,
|
||||||
|
"mu={mu}: {logged} vs {direct}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A window far out in the tail: both endpoints underflow together, so the
|
||||||
|
/// difference has to be taken in scaled form.
|
||||||
|
#[test]
|
||||||
|
fn ln_interval_survives_a_window_deep_in_the_tail() {
|
||||||
|
for mu in [-50.0f64, -100.0, -1000.0] {
|
||||||
|
let logged = ln_interval(-1.0, 1.0, mu, 1.0);
|
||||||
|
assert!(logged.is_finite(), "mu={mu}: {logged}");
|
||||||
|
assert!(
|
||||||
|
logged < f64::MIN_POSITIVE.ln(),
|
||||||
|
"mu={mu}: {logged} is stuck on the clamp floor"
|
||||||
|
);
|
||||||
|
// Dominated by the near edge: ln P ~ ln Phi(-(|mu| - 1)).
|
||||||
|
let near = ln_sf(-1.0, mu, 1.0);
|
||||||
|
assert!(
|
||||||
|
(logged - near).abs() < 5.0,
|
||||||
|
"mu={mu}: {logged} strays from the near-edge tail {near}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_quality() {
|
fn test_quality() {
|
||||||
let a = Gaussian::from_ms(25.0, 3.0);
|
let a = Gaussian::from_ms(25.0, 3.0);
|
||||||
|
|||||||
+32
-3
@@ -34,12 +34,41 @@ impl Outcome {
|
|||||||
///
|
///
|
||||||
/// # Panics
|
/// # Panics
|
||||||
///
|
///
|
||||||
/// Panics if `winner >= n`.
|
/// Panics if `winner >= n`. Use [`Outcome::try_winner`] when the index
|
||||||
|
/// comes from data rather than a literal.
|
||||||
|
///
|
||||||
|
/// This is the one constructor here that validates, and deliberately so.
|
||||||
|
/// Its siblings build freely and let ingestion reject what it cannot use,
|
||||||
|
/// which works because a malformed rank vector stays recognisable. An
|
||||||
|
/// out-of-range winner does not: `winner(5, 2)` would produce ranks
|
||||||
|
/// `[1, 1]`, an all-tied draw that ingestion accepts without complaint when
|
||||||
|
/// `p_draw > 0`. Asking "team 5 won" and silently getting "everyone drew"
|
||||||
|
/// is exactly the class of quiet wrong answer this crate keeps removing, so
|
||||||
|
/// the check happens here where the mistake is.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn winner(winner: u32, n: u32) -> Self {
|
pub fn winner(winner: u32, n: u32) -> Self {
|
||||||
assert!(winner < n, "winner index {winner} out of range 0..{n}");
|
Self::try_winner(winner, n)
|
||||||
|
.unwrap_or_else(|_| panic!("winner index {winner} out of range 0..{n}"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `n`-team outcome where team `winner` won, or an error if `winner` is not
|
||||||
|
/// a valid team index.
|
||||||
|
///
|
||||||
|
/// The fallible form of [`Outcome::winner`], for when the index is computed
|
||||||
|
/// or parsed rather than written literally.
|
||||||
|
///
|
||||||
|
/// # Errors
|
||||||
|
///
|
||||||
|
/// `InvalidParameter` if `winner >= n`.
|
||||||
|
pub fn try_winner(winner: u32, n: u32) -> Result<Self, crate::InferenceError> {
|
||||||
|
if winner >= n {
|
||||||
|
return Err(crate::InferenceError::InvalidParameter {
|
||||||
|
name: "winner",
|
||||||
|
value: f64::from(winner),
|
||||||
|
});
|
||||||
|
}
|
||||||
let ranks: SmallVec<[u32; 4]> = (0..n).map(|i| if i == winner { 0 } else { 1 }).collect();
|
let ranks: SmallVec<[u32; 4]> = (0..n).map(|i| if i == winner { 0 } else { 1 }).collect();
|
||||||
Self::Ranked(ranks)
|
Ok(Self::Ranked(ranks))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// All `n` teams tied.
|
/// All `n` teams tied.
|
||||||
|
|||||||
+16
-10
@@ -33,17 +33,23 @@ pub(crate) const MAX_TEAMS_FOR_DISTRIBUTION: usize = 6;
|
|||||||
|
|
||||||
/// Relative tolerance for the first-place integrals.
|
/// Relative tolerance for the first-place integrals.
|
||||||
///
|
///
|
||||||
/// Tightening past this buys nothing: the underlying `cdf` is a rational
|
/// The adaptive integrator reaches the exact two-team closed form to ~1e-15 at
|
||||||
/// approximation with fractional error ~1.2e-7, which contributes ~6e-9 to a
|
/// this tolerance, which is round-off for a probability. `cdf` is no longer the
|
||||||
/// finished probability and dominates any further quadrature refinement.
|
/// limit — it went to ~1 ULP when `erfc` moved to `libm` — so this is the
|
||||||
|
/// integrator's own floor.
|
||||||
const WIN_TOLERANCE: f64 = 1e-8;
|
const WIN_TOLERANCE: f64 = 1e-8;
|
||||||
|
|
||||||
/// Nodes for the ranking grid, and the floor below which a grid is pointless.
|
/// Nodes for the ranking grid, and the floor below which a grid is pointless.
|
||||||
///
|
///
|
||||||
/// The recursion converges as O(h^2). Measured against the exact two-team
|
/// The recursion converges as O(h^2), so this trades nodes against accuracy
|
||||||
/// closed form, 2_048 nodes leave ~1.2e-6 of discretisation error while 8_192
|
/// directly. Measured against the exact two-team closed form, 2_048 nodes leave
|
||||||
/// reach ~1e-7 — at which point the residual is the `cdf` rational
|
/// ~1.2e-6 of discretisation error and 8_192 reach ~1e-7.
|
||||||
/// approximation (~2.4e-8), not the grid, and refining further buys nothing.
|
///
|
||||||
|
/// Unlike the adaptive path there is no approximation floor underneath this any
|
||||||
|
/// more — `cdf` is accurate to ~1 ULP since `erfc` moved to `libm` — so the
|
||||||
|
/// error here is purely the grid, and a caller who needs more can only get it
|
||||||
|
/// by paying for more nodes. 8_192 is the accuracy/cost point chosen, not a
|
||||||
|
/// point where refining stops helping.
|
||||||
const MIN_GRID_POINTS: usize = 8_192;
|
const MIN_GRID_POINTS: usize = 8_192;
|
||||||
const MAX_GRID_POINTS: usize = 262_144;
|
const MAX_GRID_POINTS: usize = 262_144;
|
||||||
|
|
||||||
@@ -62,7 +68,7 @@ fn phi(z: f64) -> f64 {
|
|||||||
fn density(g: Gaussian, x: f64) -> f64 {
|
fn density(g: Gaussian, x: f64) -> f64 {
|
||||||
let sigma = g.sigma();
|
let sigma = g.sigma();
|
||||||
let z = (x - g.mu()) / sigma;
|
let z = (x - g.mu()) / sigma;
|
||||||
(-0.5 * z * z).exp() / (sigma * (2.0 * std::f64::consts::PI).sqrt())
|
libm::exp(-0.5 * z * z) / (sigma * (2.0 * std::f64::consts::PI).sqrt())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Per-pair draw margins.
|
/// Per-pair draw margins.
|
||||||
@@ -503,7 +509,7 @@ mod tests {
|
|||||||
|
|
||||||
/// Exact two-team result: `P(a first) = Phi((mu_a - mu_b - eps) / sd)`.
|
/// Exact two-team result: `P(a first) = Phi((mu_a - mu_b - eps) / sd)`.
|
||||||
fn closed_form_two(a: Gaussian, b: Gaussian, eps: f64) -> (f64, f64) {
|
fn closed_form_two(a: Gaussian, b: Gaussian, eps: f64) -> (f64, f64) {
|
||||||
let sd = (a.sigma().powi(2) + b.sigma().powi(2)).sqrt();
|
let sd = a.sigma().hypot(b.sigma());
|
||||||
(
|
(
|
||||||
phi((a.mu() - b.mu() - eps) / sd),
|
phi((a.mu() - b.mu() - eps) / sd),
|
||||||
phi((b.mu() - a.mu() - eps) / sd),
|
phi((b.mu() - a.mu() - eps) / sd),
|
||||||
@@ -523,7 +529,7 @@ mod tests {
|
|||||||
let got = win_probabilities(&perf, &flat(2, eps));
|
let got = win_probabilities(&perf, &flat(2, eps));
|
||||||
let (wa, wb) = closed_form_two(perf[0], perf[1], eps);
|
let (wa, wb) = closed_form_two(perf[0], perf[1], eps);
|
||||||
assert!(
|
assert!(
|
||||||
(got[0] - wa).abs() < 1e-7 && (got[1] - wb).abs() < 1e-7,
|
(got[0] - wa).abs() < 1e-12 && (got[1] - wb).abs() < 1e-12,
|
||||||
"mu=({ma},{mb}) sigma=({sa},{sb}) eps={eps}: got {got:?}, want [{wa}, {wb}]"
|
"mu=({ma},{mb}) sigma=({sa},{sb}) eps={eps}: got {got:?}, want [{wa}, {wb}]"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
-152
@@ -1,152 +0,0 @@
|
|||||||
//! Schedule trait and built-in implementations.
|
|
||||||
//!
|
|
||||||
//! A schedule drives factor propagation to convergence. The default
|
|
||||||
//! `EpsilonOrMax` performs one `TeamSum` sweep (setup) then alternating
|
|
||||||
//! forward/backward sweeps over the iterating factors until the max
|
|
||||||
//! delta drops below epsilon or `max` iterations is reached.
|
|
||||||
|
|
||||||
use crate::factor::{BuiltinFactor, Factor, VarStore};
|
|
||||||
|
|
||||||
/// Result returned by a `Schedule::run` call.
|
|
||||||
#[derive(Debug, Clone, Copy)]
|
|
||||||
pub struct ScheduleReport {
|
|
||||||
pub iterations: usize,
|
|
||||||
pub final_step: (f64, f64),
|
|
||||||
pub converged: bool,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Drives factor propagation to convergence.
|
|
||||||
pub trait Schedule: Send + Sync {
|
|
||||||
fn run(&self, factors: &mut [BuiltinFactor], vars: &mut VarStore) -> ScheduleReport;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Default schedule: sweep forward then backward until step ≤ eps or iter == max.
|
|
||||||
///
|
|
||||||
/// Matches the existing `Game::likelihoods` loop bit-for-bit when given the
|
|
||||||
/// same factor layout (`TeamSums` first, then alternating RankDiff/Trunc pairs).
|
|
||||||
#[derive(Debug, Clone, Copy)]
|
|
||||||
pub struct EpsilonOrMax {
|
|
||||||
pub eps: f64,
|
|
||||||
pub max: usize,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Default for EpsilonOrMax {
|
|
||||||
fn default() -> Self {
|
|
||||||
// Derived from `ConvergenceOptions` so there is one source of truth for
|
|
||||||
// the tolerance and iteration cap. These previously disagreed: this
|
|
||||||
// default capped at 10 iterations while `ConvergenceOptions` allowed 30,
|
|
||||||
// and which applied depended on whether inference went through
|
|
||||||
// `run_chain` or a `Schedule`.
|
|
||||||
let defaults = crate::ConvergenceOptions::default();
|
|
||||||
|
|
||||||
Self {
|
|
||||||
eps: defaults.epsilon,
|
|
||||||
max: defaults.max_iter,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Schedule for EpsilonOrMax {
|
|
||||||
fn run(&self, factors: &mut [BuiltinFactor], vars: &mut VarStore) -> ScheduleReport {
|
|
||||||
// Partition: leading run of TeamSum factors run exactly once (setup).
|
|
||||||
let n_setup = factors
|
|
||||||
.iter()
|
|
||||||
.position(|f| !matches!(f, BuiltinFactor::TeamSum(_)))
|
|
||||||
.unwrap_or(factors.len());
|
|
||||||
|
|
||||||
for f in factors[..n_setup].iter_mut() {
|
|
||||||
f.propagate(vars);
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut iterations = 0;
|
|
||||||
// With no iterating factors the graph is already at its fixed point:
|
|
||||||
// the setup pass above is all there is to do. Reporting `converged:
|
|
||||||
// false` with an infinite step for that case gave callers a false
|
|
||||||
// negative.
|
|
||||||
let mut final_step = (0.0, 0.0);
|
|
||||||
let mut converged = true;
|
|
||||||
|
|
||||||
if n_setup < factors.len() {
|
|
||||||
final_step = (f64::INFINITY, f64::INFINITY);
|
|
||||||
converged = false;
|
|
||||||
for _ in 0..self.max {
|
|
||||||
let mut step = (0.0_f64, 0.0_f64);
|
|
||||||
|
|
||||||
// Forward sweep over iterating factors.
|
|
||||||
for f in factors[n_setup..].iter_mut() {
|
|
||||||
let d = f.propagate(vars);
|
|
||||||
step.0 = step.0.max(d.0);
|
|
||||||
step.1 = step.1.max(d.1);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Backward sweep.
|
|
||||||
for f in factors[n_setup..].iter_mut().rev() {
|
|
||||||
let d = f.propagate(vars);
|
|
||||||
step.0 = step.0.max(d.0);
|
|
||||||
step.1 = step.1.max(d.1);
|
|
||||||
}
|
|
||||||
|
|
||||||
iterations += 1;
|
|
||||||
final_step = step;
|
|
||||||
|
|
||||||
if step.0 <= self.eps && step.1 <= self.eps {
|
|
||||||
converged = true;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
ScheduleReport {
|
|
||||||
iterations,
|
|
||||||
final_step,
|
|
||||||
converged,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use crate::{N_INF, factor::team_sum::TeamSumFactor, gaussian::Gaussian};
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn schedule_runs_setup_factors_once() {
|
|
||||||
// Single TeamSum factor; schedule should propagate it exactly once and report 0 iterations.
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let out = vars.alloc(N_INF);
|
|
||||||
let mut factors = vec![BuiltinFactor::TeamSum(TeamSumFactor {
|
|
||||||
inputs: vec![(Gaussian::from_ms(5.0, 1.0), 1.0)],
|
|
||||||
out,
|
|
||||||
})];
|
|
||||||
let schedule = EpsilonOrMax::default();
|
|
||||||
let report = schedule.run(&mut factors, &mut vars);
|
|
||||||
assert_eq!(report.iterations, 0);
|
|
||||||
// The team-perf var should hold the sum.
|
|
||||||
let result = vars.get(out);
|
|
||||||
assert!((result.mu() - 5.0).abs() < 1e-12);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn report_marks_converged_when_no_iterating_factors() {
|
|
||||||
// A graph of only setup factors has nothing to iterate, so it is at its
|
|
||||||
// fixed point after the setup pass: 0 iterations, and converged.
|
|
||||||
let mut vars = VarStore::new();
|
|
||||||
let out = vars.alloc(N_INF);
|
|
||||||
let mut factors = vec![BuiltinFactor::TeamSum(TeamSumFactor {
|
|
||||||
inputs: vec![(Gaussian::from_ms(0.0, 1.0), 1.0)],
|
|
||||||
out,
|
|
||||||
})];
|
|
||||||
let report = EpsilonOrMax::default().run(&mut factors, &mut vars);
|
|
||||||
assert_eq!(report.iterations, 0);
|
|
||||||
assert!(report.converged);
|
|
||||||
assert_eq!(report.final_step, (0.0, 0.0));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn default_matches_convergence_options() {
|
|
||||||
let schedule = EpsilonOrMax::default();
|
|
||||||
let options = crate::ConvergenceOptions::default();
|
|
||||||
assert_eq!(schedule.max, options.max_iter);
|
|
||||||
assert_eq!(schedule.eps, options.epsilon);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -809,6 +809,78 @@ pub(crate) fn compute_elapsed<T: Time>(last: Option<&T>, current: &T) -> i64 {
|
|||||||
elapsed.max(0)
|
elapsed.max(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl<T: Time> TimeSlice<T> {
|
||||||
|
/// This slice's scored event factors, as contrasts over competitors.
|
||||||
|
///
|
||||||
|
/// Message passing produces per-competitor marginals and throws the
|
||||||
|
/// correlation away — `Item::likelihood` is already the projection of an
|
||||||
|
/// event's factor onto one competitor. So a joint has to be rebuilt from
|
||||||
|
/// the factor structure rather than recovered from the messages.
|
||||||
|
///
|
||||||
|
/// Usefully, a precision matrix depends only on *structure* — who played
|
||||||
|
/// whom, with what weights and what observation noise — and not on the
|
||||||
|
/// observed outcomes. The means are already exact, so only the second
|
||||||
|
/// moment needs rebuilding.
|
||||||
|
///
|
||||||
|
/// Each entry is a contrast and the observation variance that sits on it.
|
||||||
|
/// Ranked events contribute nothing: their truncation factors are EP
|
||||||
|
/// approximations that inference does not retain.
|
||||||
|
pub(crate) fn scored_contrasts<D: Drift<T>>(
|
||||||
|
&self,
|
||||||
|
agents: &CompetitorStore<T, D>,
|
||||||
|
) -> Vec<(Vec<(Index, f64)>, f64)> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
|
||||||
|
for event in &self.events {
|
||||||
|
let EventKind::Scored { score_sigma } = event.kind else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Teams best-first, matching the diff chain inference builds.
|
||||||
|
let mut order: Vec<usize> = (0..event.teams.len()).collect();
|
||||||
|
order.sort_by(|&a, &b| {
|
||||||
|
event.teams[b]
|
||||||
|
.output
|
||||||
|
.partial_cmp(&event.teams[a].output)
|
||||||
|
.unwrap_or(std::cmp::Ordering::Equal)
|
||||||
|
});
|
||||||
|
|
||||||
|
for pair in order.windows(2) {
|
||||||
|
let (hi, lo) = (pair[0], pair[1]);
|
||||||
|
let mut contrast: Vec<(Index, f64)> = Vec::new();
|
||||||
|
let mut noise = score_sigma * score_sigma;
|
||||||
|
|
||||||
|
for (team, sign) in [(hi, 1.0), (lo, -1.0)] {
|
||||||
|
for (m, item) in event.teams[team].items.iter().enumerate() {
|
||||||
|
let w = event.weights[team][m];
|
||||||
|
noise += w * w * agents[item.agent].rating.beta.powi(2);
|
||||||
|
contrast.push((item.agent, sign * w));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
out.push((contrast, noise));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True when every event here is scored, so the joint is exact.
|
||||||
|
pub(crate) fn all_scored(&self) -> bool {
|
||||||
|
self.events
|
||||||
|
.iter()
|
||||||
|
.all(|e| matches!(e.kind, EventKind::Scored { .. }))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The competitors appearing in this slice, with the elapsed count since
|
||||||
|
/// each one's previous appearance.
|
||||||
|
pub(crate) fn appearances(&self) -> impl Iterator<Item = (Index, i64)> + '_ {
|
||||||
|
self.skills
|
||||||
|
.keys()
|
||||||
|
.map(|idx| (idx, self.skills.get(idx).expect("slice key").elapsed))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use approx::assert_ulps_eq;
|
use approx::assert_ulps_eq;
|
||||||
|
|||||||
@@ -0,0 +1,142 @@
|
|||||||
|
//! What an additive model does to uncertainty, and why "add the marginals" is
|
||||||
|
//! unsafe in one direction and merely wasteful in the other.
|
||||||
|
//!
|
||||||
|
//! Structurally this is the shape a joint player/layout model takes: every
|
||||||
|
//! observation measures a *sum* of nodes against a reference, so the data pins
|
||||||
|
//! differences and leaves the overall level to the prior. That is the classic
|
||||||
|
//! rating-scale indeterminacy, not a defect.
|
||||||
|
//!
|
||||||
|
//! The consequence for a consumer is that combining marginals is wrong in
|
||||||
|
//! opposite directions depending on the combination, which is worth pinning
|
||||||
|
//! because the unsafe direction is not the one you would guess:
|
||||||
|
//!
|
||||||
|
//! - **Differences** (`a - b`): the shared level cancels, so the exact width is
|
||||||
|
//! small — and adding marginals lands within a couple of percent of it here,
|
||||||
|
//! because the loopy underestimate offsets the ignored correlation.
|
||||||
|
//! - **Sums** (`a + b`): the shared level does *not* cancel, so the exact width
|
||||||
|
//! is large, and adding marginals is roughly five times too narrow. That is
|
||||||
|
//! overconfident, and it is the direction that publishes a claim the data
|
||||||
|
//! does not support.
|
||||||
|
|
||||||
|
use smallvec::smallvec;
|
||||||
|
use trueskill_tt::{ConstantDrift, ConvergenceOptions, Event, History, Member, Outcome, Team};
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn additive_structure_makes_sums_wide_and_differences_tight() {
|
||||||
|
// Structurally like ustat: every round is (player + hole) measured against
|
||||||
|
// a fixed reference. Only SUMS are pinned by the data; the split between
|
||||||
|
// player and hole is pinned only by the prior.
|
||||||
|
let players = ["p0", "p1", "p2"];
|
||||||
|
let holes = ["h0", "h1"];
|
||||||
|
|
||||||
|
let mut h: History<i64, _, _, &'static str> = History::builder()
|
||||||
|
.mu(0.0)
|
||||||
|
.sigma(6.0)
|
||||||
|
.beta(1.0)
|
||||||
|
.score_sigma(2.0)
|
||||||
|
.drift(ConstantDrift(0.0))
|
||||||
|
.convergence(ConvergenceOptions {
|
||||||
|
max_iter: 20_000,
|
||||||
|
epsilon: 1e-12,
|
||||||
|
alpha: 1.0,
|
||||||
|
})
|
||||||
|
.build();
|
||||||
|
|
||||||
|
let mut seed = 3u64;
|
||||||
|
let mut rnd = move || {
|
||||||
|
seed ^= seed << 13;
|
||||||
|
seed ^= seed >> 7;
|
||||||
|
seed ^= seed << 17;
|
||||||
|
seed
|
||||||
|
};
|
||||||
|
// true skills, so we know what the data encodes
|
||||||
|
let truth_p = [2.0, 0.0, -2.0];
|
||||||
|
let truth_h = [1.0, -1.0];
|
||||||
|
|
||||||
|
let mut events = Vec::new();
|
||||||
|
for _ in 0..60 {
|
||||||
|
let p = (rnd() as usize) % 3;
|
||||||
|
let q = (rnd() as usize) % 2;
|
||||||
|
let noise = ((rnd() % 1000) as f64 / 1000.0 - 0.5) * 2.0;
|
||||||
|
let score = truth_p[p] + truth_h[q] + noise;
|
||||||
|
events.push(Event {
|
||||||
|
time: 1,
|
||||||
|
teams: smallvec![
|
||||||
|
Team::with_members([Member::new(players[p]), Member::new(holes[q])]),
|
||||||
|
Team::with_members([Member::new("reference")]),
|
||||||
|
],
|
||||||
|
outcome: Outcome::scores([score, 0.0]),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
h.add_events(events).unwrap();
|
||||||
|
let r = h.converge().unwrap();
|
||||||
|
assert!(r.converged, "{:?}", r.final_step);
|
||||||
|
|
||||||
|
println!("\n== marginals (what current_skill reports) ==");
|
||||||
|
for k in players.iter().chain(holes.iter()) {
|
||||||
|
let g = h.current_skill(k).unwrap();
|
||||||
|
println!(" {k}: mu {:>8.4} sigma {:>8.4}", g.mu(), g.sigma());
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("\n== the same nodes via posterior_of (exact marginal) ==");
|
||||||
|
for k in players.iter().chain(holes.iter()) {
|
||||||
|
let g = h.posterior_of(&[(k, 1.0)]).unwrap();
|
||||||
|
println!(" {k}: mu {:>8.4} sigma {:>8.4}", g.mu(), g.sigma());
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("\n== combinations the data actually pins ==");
|
||||||
|
for (label, terms) in [
|
||||||
|
("p0 + h0 (a round)", vec![(&"p0", 1.0), (&"h0", 1.0)]),
|
||||||
|
(
|
||||||
|
"p0 - p1 (rank two players)",
|
||||||
|
vec![(&"p0", 1.0), (&"p1", -1.0)],
|
||||||
|
),
|
||||||
|
("p0 - p2", vec![(&"p0", 1.0), (&"p2", -1.0)]),
|
||||||
|
(
|
||||||
|
"h0 - h1 (rank two holes)",
|
||||||
|
vec![(&"h0", 1.0), (&"h1", -1.0)],
|
||||||
|
),
|
||||||
|
] {
|
||||||
|
let joint = h.posterior_of(&terms).unwrap();
|
||||||
|
// what a consumer gets today by adding marginals
|
||||||
|
let naive: f64 = terms
|
||||||
|
.iter()
|
||||||
|
.map(|(k, c)| c * c * h.current_skill(*k).unwrap().sigma().powi(2))
|
||||||
|
.sum::<f64>()
|
||||||
|
.sqrt();
|
||||||
|
println!(
|
||||||
|
" {label:<28} exact sigma {:>7.4} adding marginals {:>7.4} {:>5.2}x over",
|
||||||
|
joint.sigma(),
|
||||||
|
naive,
|
||||||
|
naive / joint.sigma()
|
||||||
|
);
|
||||||
|
|
||||||
|
let ratio = naive / joint.sigma();
|
||||||
|
if label.contains('+') {
|
||||||
|
assert!(
|
||||||
|
ratio < 0.5,
|
||||||
|
"{label}: adding marginals should be badly OVERconfident for a \
|
||||||
|
sum, got {ratio:.3}x"
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
assert!(
|
||||||
|
(0.8..1.25).contains(&ratio),
|
||||||
|
"{label}: adding marginals happens to be close for a difference, \
|
||||||
|
got {ratio:.3}x"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A single node in an additive model is weakly identified: its exact
|
||||||
|
// posterior is far wider than message passing reports, because the level it
|
||||||
|
// shares with its partners is pinned only by the prior.
|
||||||
|
for k in players.iter().chain(holes.iter()) {
|
||||||
|
let bp = h.current_skill(k).unwrap().sigma();
|
||||||
|
let exact = h.posterior_of(&[(k, 1.0)]).unwrap().sigma();
|
||||||
|
assert!(
|
||||||
|
exact > 3.0 * bp,
|
||||||
|
"{k}: exact marginal {exact} should be much wider than the reported \
|
||||||
|
{bp} in an additive model"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
+7
-7
@@ -65,7 +65,7 @@ fn add_events_draw() {
|
|||||||
outcome: Outcome::draw(2),
|
outcome: Outcome::draw(2),
|
||||||
}];
|
}];
|
||||||
h.add_events(events).unwrap();
|
h.add_events(events).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -123,7 +123,7 @@ fn fluent_event_builder_winner_convenience() {
|
|||||||
.winner(0)
|
.winner(0)
|
||||||
.commit()
|
.commit()
|
||||||
.unwrap();
|
.unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -141,7 +141,7 @@ fn fluent_event_builder_draw() {
|
|||||||
.draw()
|
.draw()
|
||||||
.commit()
|
.commit()
|
||||||
.unwrap();
|
.unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -155,7 +155,7 @@ fn current_skill_and_learning_curve() {
|
|||||||
.build();
|
.build();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.record_winner(&"a", &"b", 2).unwrap();
|
h.record_winner(&"a", &"b", 2).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let a = h.current_skill(&"a").unwrap();
|
let a = h.current_skill(&"a").unwrap();
|
||||||
assert!(a.mu() > 25.0);
|
assert!(a.mu() > 25.0);
|
||||||
@@ -201,7 +201,7 @@ fn predict_quality_two_teams() {
|
|||||||
.p_draw(0.0)
|
.p_draw(0.0)
|
||||||
.build();
|
.build();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let q = h.predict_quality(&[&[&"a"], &[&"b"]]).unwrap();
|
let q = h.predict_quality(&[&[&"a"], &[&"b"]]).unwrap();
|
||||||
assert!(q > 0.0 && q <= 1.0);
|
assert!(q > 0.0 && q <= 1.0);
|
||||||
@@ -217,7 +217,7 @@ fn predict_outcome_two_teams_sums_to_one() {
|
|||||||
.p_draw(0.0)
|
.p_draw(0.0)
|
||||||
.build();
|
.build();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let p = h.predict_outcome(&[&[&"a"], &[&"b"]]).unwrap();
|
let p = h.predict_outcome(&[&[&"a"], &[&"b"]]).unwrap();
|
||||||
let wins = p.win_probabilities();
|
let wins = p.win_probabilities();
|
||||||
@@ -245,7 +245,7 @@ fn fluent_event_builder_scores() {
|
|||||||
.scores([12.0, 4.0])
|
.scores([12.0, 4.0])
|
||||||
.commit()
|
.commit()
|
||||||
.unwrap();
|
.unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let a = h.current_skill(&"alice").unwrap();
|
let a = h.current_skill(&"alice").unwrap();
|
||||||
let b = h.current_skill(&"bob").unwrap();
|
let b = h.current_skill(&"bob").unwrap();
|
||||||
|
|||||||
+10
-10
@@ -64,13 +64,13 @@ fn a_prior_applies_to_a_new_competitor() {
|
|||||||
let mut with = history();
|
let mut with = history();
|
||||||
with.add_events(vec![bout("a", "b", 0, Some(seeded), None)])
|
with.add_events(vec![bout("a", "b", 0, Some(seeded), None)])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
with.converge().unwrap();
|
let _ = with.converge().unwrap();
|
||||||
|
|
||||||
let mut without = history();
|
let mut without = history();
|
||||||
without
|
without
|
||||||
.add_events(vec![bout("a", "b", 0, None, None)])
|
.add_events(vec![bout("a", "b", 0, None, None)])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
without.converge().unwrap();
|
let _ = without.converge().unwrap();
|
||||||
|
|
||||||
assert!(
|
assert!(
|
||||||
(skill_of(&with, "a").mu() - skill_of(&without, "a").mu()).abs() > 1.0,
|
(skill_of(&with, "a").mu() - skill_of(&without, "a").mu()).abs() > 1.0,
|
||||||
@@ -91,7 +91,7 @@ fn a_prior_applies_to_a_competitor_the_history_already_knows() {
|
|||||||
// "a" now exists. Configuring it here used to do nothing whatsoever.
|
// "a" now exists. Configuring it here used to do nothing whatsoever.
|
||||||
late.add_events(vec![bout("a", "b", 1, Some(seeded), None)])
|
late.add_events(vec![bout("a", "b", 1, Some(seeded), None)])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
late.converge().unwrap();
|
let _ = late.converge().unwrap();
|
||||||
|
|
||||||
let mut never = history();
|
let mut never = history();
|
||||||
never
|
never
|
||||||
@@ -100,7 +100,7 @@ fn a_prior_applies_to_a_competitor_the_history_already_knows() {
|
|||||||
bout("a", "b", 1, None, None),
|
bout("a", "b", 1, None, None),
|
||||||
])
|
])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
never.converge().unwrap();
|
let _ = never.converge().unwrap();
|
||||||
|
|
||||||
assert!(
|
assert!(
|
||||||
(skill_of(&late, "a").mu() - skill_of(&never, "a").mu()).abs() > 1.0,
|
(skill_of(&late, "a").mu() - skill_of(&never, "a").mu()).abs() > 1.0,
|
||||||
@@ -122,7 +122,7 @@ fn a_prior_is_whole_history_scoped_not_per_event() {
|
|||||||
.unwrap();
|
.unwrap();
|
||||||
late.add_events(vec![bout("a", "b", 1, Some(seeded), None)])
|
late.add_events(vec![bout("a", "b", 1, Some(seeded), None)])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
late.converge().unwrap();
|
let _ = late.converge().unwrap();
|
||||||
|
|
||||||
let mut early = history();
|
let mut early = history();
|
||||||
early
|
early
|
||||||
@@ -131,7 +131,7 @@ fn a_prior_is_whole_history_scoped_not_per_event() {
|
|||||||
bout("a", "b", 1, Some(seeded), None),
|
bout("a", "b", 1, Some(seeded), None),
|
||||||
])
|
])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
early.converge().unwrap();
|
let _ = early.converge().unwrap();
|
||||||
|
|
||||||
let (l, e) = (skill_of(&late, "a"), skill_of(&early, "a"));
|
let (l, e) = (skill_of(&late, "a"), skill_of(&early, "a"));
|
||||||
assert!(
|
assert!(
|
||||||
@@ -150,7 +150,7 @@ fn repeating_the_same_prior_is_inert() {
|
|||||||
bout("a", "b", 1, None, None),
|
bout("a", "b", 1, None, None),
|
||||||
])
|
])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
once.converge().unwrap();
|
let _ = once.converge().unwrap();
|
||||||
|
|
||||||
let mut every_time = history();
|
let mut every_time = history();
|
||||||
every_time
|
every_time
|
||||||
@@ -159,7 +159,7 @@ fn repeating_the_same_prior_is_inert() {
|
|||||||
bout("a", "b", 1, Some(seeded), None),
|
bout("a", "b", 1, Some(seeded), None),
|
||||||
])
|
])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
every_time.converge().unwrap();
|
let _ = every_time.converge().unwrap();
|
||||||
|
|
||||||
let (o, e) = (skill_of(&once, "a"), skill_of(&every_time, "a"));
|
let (o, e) = (skill_of(&once, "a"), skill_of(&every_time, "a"));
|
||||||
assert!(
|
assert!(
|
||||||
@@ -203,7 +203,7 @@ fn setting_one_field_late_leaves_the_other_alone() {
|
|||||||
// Only the scale this time — the prior above must survive.
|
// Only the scale this time — the prior above must survive.
|
||||||
h.add_events(vec![bout("a", "b", 1, None, Some(0.5))])
|
h.add_events(vec![bout("a", "b", 1, None, Some(0.5))])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let mut both_upfront = history();
|
let mut both_upfront = history();
|
||||||
both_upfront
|
both_upfront
|
||||||
@@ -212,7 +212,7 @@ fn setting_one_field_late_leaves_the_other_alone() {
|
|||||||
bout("a", "b", 1, None, None),
|
bout("a", "b", 1, None, None),
|
||||||
])
|
])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
both_upfront.converge().unwrap();
|
let _ = both_upfront.converge().unwrap();
|
||||||
|
|
||||||
let (a, b) = (skill_of(&h, "a"), skill_of(&both_upfront, "a"));
|
let (a, b) = (skill_of(&h, "a"), skill_of(&both_upfront, "a"));
|
||||||
assert!(
|
assert!(
|
||||||
|
|||||||
@@ -351,7 +351,7 @@ fn zero_weight_does_not_produce_a_non_finite_posterior() {
|
|||||||
.commit()
|
.commit()
|
||||||
.expect("a zero weight is accepted today; update this test if that changes");
|
.expect("a zero weight is accepted today; update this test if that changes");
|
||||||
|
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
assert_curve_finite(&h, &["a", "b"], "zero weight");
|
assert_curve_finite(&h, &["a", "b"], "zero weight");
|
||||||
}
|
}
|
||||||
@@ -368,7 +368,7 @@ fn negative_weight_does_not_produce_a_non_finite_posterior() {
|
|||||||
.commit()
|
.commit()
|
||||||
.expect("a negative weight is accepted today; update this test if that changes");
|
.expect("a negative weight is accepted today; update this test if that changes");
|
||||||
|
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
assert_curve_finite(&h, &["a", "b"], "negative weight");
|
assert_curve_finite(&h, &["a", "b"], "negative weight");
|
||||||
}
|
}
|
||||||
@@ -389,7 +389,7 @@ fn out_of_order_timestamps_converge_to_the_same_answer() {
|
|||||||
h.record_winner(&"a", &"b", time).unwrap();
|
h.record_winner(&"a", &"b", time).unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
h
|
h
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -416,7 +416,7 @@ fn extreme_beta_and_sigma_stay_finite() {
|
|||||||
|
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.record_winner(&"a", &"b", 2).unwrap();
|
h.record_winner(&"a", &"b", 2).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
assert_curve_finite(&h, &["a", "b"], &format!("beta={beta} sigma={sigma}"));
|
assert_curve_finite(&h, &["a", "b"], &format!("beta={beta} sigma={sigma}"));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ fn build_and_converge(seed: u64) -> Vec<(i64, trueskill_tt::Gaussian)> {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
h.add_events(events).unwrap();
|
h.add_events(events).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
// Sample one competitor's curve for the comparison.
|
// Sample one competitor's curve for the comparison.
|
||||||
h.learning_curve("p0")
|
h.learning_curve("p0")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -58,7 +58,7 @@ fn fit(events: Vec<Event<i64, &'static str>>, gamma: f64) -> Fit {
|
|||||||
.build();
|
.build();
|
||||||
|
|
||||||
h.add_events(events).unwrap();
|
h.add_events(events).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
h
|
h
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -385,7 +385,7 @@ fn drift_scale_applies_when_set_after_first_appearance() {
|
|||||||
outcome: Outcome::winner(1, 2),
|
outcome: Outcome::winner(1, 2),
|
||||||
}])
|
}])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
late.converge().unwrap();
|
let _ = late.converge().unwrap();
|
||||||
|
|
||||||
let applied = curve(&late, "anchor");
|
let applied = curve(&late, "anchor");
|
||||||
let pinned_from_the_start = curve(&fit(distant_pair(Some(0.0)), 25.0 / 300.0), "anchor");
|
let pinned_from_the_start = curve(&fit(distant_pair(Some(0.0)), 25.0 / 300.0), "anchor");
|
||||||
|
|||||||
+6
-6
@@ -47,7 +47,7 @@ fn tight() -> ConvergenceOptions {
|
|||||||
fn filtered_evidence_sits_between_coin_flip_and_batch() {
|
fn filtered_evidence_sits_between_coin_flip_and_batch() {
|
||||||
let mut history = repeated_winner(5);
|
let mut history = repeated_winner(5);
|
||||||
|
|
||||||
history.converge().unwrap();
|
let _ = history.converge().unwrap();
|
||||||
|
|
||||||
let coin_flip = 5.0 * 0.5f64.ln();
|
let coin_flip = 5.0 * 0.5f64.ln();
|
||||||
let batch = history.log_evidence();
|
let batch = history.log_evidence();
|
||||||
@@ -71,7 +71,7 @@ fn filtered_evidence_sits_between_coin_flip_and_batch() {
|
|||||||
fn filtered_first_point_is_less_certain_than_smoothed() {
|
fn filtered_first_point_is_less_certain_than_smoothed() {
|
||||||
let mut history = repeated_winner(12);
|
let mut history = repeated_winner(12);
|
||||||
|
|
||||||
history.converge().unwrap();
|
let _ = history.converge().unwrap();
|
||||||
|
|
||||||
let smoothed = history.learning_curve("a");
|
let smoothed = history.learning_curve("a");
|
||||||
let filtered = history.filtered_learning_curve("a");
|
let filtered = history.filtered_learning_curve("a");
|
||||||
@@ -121,7 +121,7 @@ fn filtered_first_point_is_less_certain_than_smoothed() {
|
|||||||
fn filtered_curves_plural_agrees_with_singular() {
|
fn filtered_curves_plural_agrees_with_singular() {
|
||||||
let mut history = repeated_winner(4);
|
let mut history = repeated_winner(4);
|
||||||
|
|
||||||
history.converge().unwrap();
|
let _ = history.converge().unwrap();
|
||||||
|
|
||||||
let curves = history.filtered_learning_curves();
|
let curves = history.filtered_learning_curves();
|
||||||
|
|
||||||
@@ -180,7 +180,7 @@ fn single_slice_filtered_matches_smoothed() {
|
|||||||
])
|
])
|
||||||
.unwrap();
|
.unwrap();
|
||||||
|
|
||||||
history.converge().unwrap();
|
let _ = history.converge().unwrap();
|
||||||
|
|
||||||
let smoothed = history.learning_curve("a");
|
let smoothed = history.learning_curve("a");
|
||||||
let filtered = history.filtered_learning_curve("a");
|
let filtered = history.filtered_learning_curve("a");
|
||||||
@@ -223,13 +223,13 @@ fn filtered_curves_do_not_depend_on_ingestion_order() {
|
|||||||
|
|
||||||
let mut batched = History::builder().convergence(tight()).build();
|
let mut batched = History::builder().convergence(tight()).build();
|
||||||
batched.add_events(all.clone()).unwrap();
|
batched.add_events(all.clone()).unwrap();
|
||||||
batched.converge().unwrap();
|
let _ = batched.converge().unwrap();
|
||||||
|
|
||||||
let mut incremental = History::builder().convergence(tight()).build();
|
let mut incremental = History::builder().convergence(tight()).build();
|
||||||
for event in all {
|
for event in all {
|
||||||
incremental.add_events([event]).unwrap();
|
incremental.add_events([event]).unwrap();
|
||||||
}
|
}
|
||||||
incremental.converge().unwrap();
|
let _ = incremental.converge().unwrap();
|
||||||
|
|
||||||
let from_batched = batched.filtered_learning_curve("a");
|
let from_batched = batched.filtered_learning_curve("a");
|
||||||
let from_incremental = incremental.filtered_learning_curve("a");
|
let from_incremental = incremental.filtered_learning_curve("a");
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ fn nan_after_fit(players: usize) -> usize {
|
|||||||
let (w, l) = if rng.coin() { (a, b) } else { (b, a) };
|
let (w, l) = if rng.coin() { (a, b) } else { (b, a) };
|
||||||
h.record_winner(&ids[w], &ids[l], 0).unwrap();
|
h.record_winner(&ids[w], &ids[l], 0).unwrap();
|
||||||
}
|
}
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
ids.iter()
|
ids.iter()
|
||||||
.filter(|id| {
|
.filter(|id| {
|
||||||
|
|||||||
@@ -0,0 +1,367 @@
|
|||||||
|
//! Calibration of the crate's marginals against the EXACT posterior.
|
||||||
|
//!
|
||||||
|
//! A scored history is linear-Gaussian — `MarginFactor` encodes
|
||||||
|
//! `score_a - score_b ~ N(perf_a - perf_b, score_sigma^2)` — so the true joint
|
||||||
|
//! posterior has a closed form and the crate can be checked against ground
|
||||||
|
//! truth rather than against intuition. That is not possible for ranked
|
||||||
|
//! outcomes, whose truncation likelihood EP genuinely approximates.
|
||||||
|
//!
|
||||||
|
//! Two things are pinned here, and one is deliberately only recorded.
|
||||||
|
//!
|
||||||
|
//! **Pinned: on a tree the crate is exact**, means and variances both. Message
|
||||||
|
//! passing has no approximation to make when the factor graph has no cycles, so
|
||||||
|
//! any drift here would be a real defect.
|
||||||
|
//!
|
||||||
|
//! **Pinned: means are exact even with cycles.** This is the standard result
|
||||||
|
//! for Gaussian belief propagation (Weiss & Freeman 2001) and it is what makes
|
||||||
|
//! ratings trustworthy.
|
||||||
|
//!
|
||||||
|
//! **Recorded, not asserted: with cycles, marginal variances are too narrow.**
|
||||||
|
//! Measured on the round-robin fixture below, the crate reports sigma 1.430
|
||||||
|
//! where the exact posterior is 2.851 — a ratio of 0.502. That is the known
|
||||||
|
//! behaviour of loopy Gaussian BP, not a bug in this crate, and it is left
|
||||||
|
//! unasserted because fixing it is exactly what #46 proposes.
|
||||||
|
//!
|
||||||
|
//! Why that matters for a consumer, and why #46 cannot be implemented as "add
|
||||||
|
//! a covariance accessor": the exact correlation between two nodes here is
|
||||||
|
//! +0.857, so a consumer computing `sqrt(sa^2 + sb^2)` for a difference
|
||||||
|
//! overstates its width. But the too-narrow marginals partially cancel that,
|
||||||
|
//! leaving 1.327x rather than 2.646x. Adding true correlations to these
|
||||||
|
//! marginals without also correcting them would give 0.765 against a true
|
||||||
|
//! 1.524 — *overconfident*, which is the unsafe direction.
|
||||||
|
|
||||||
|
use smallvec::smallvec;
|
||||||
|
use trueskill_tt::{ConstantDrift, ConvergenceOptions, Event, History, Member, Outcome, Team};
|
||||||
|
|
||||||
|
const N: usize = 5;
|
||||||
|
const MU0: f64 = 0.0;
|
||||||
|
const SIGMA0: f64 = 6.0;
|
||||||
|
const BETA: f64 = 1.0;
|
||||||
|
const SCORE_SIGMA: f64 = 2.0;
|
||||||
|
|
||||||
|
/// A STAR: every event touches c0, so the node-event graph is a tree and
|
||||||
|
/// Gaussian BP is exact. Any discrepancy here is not caused by loops.
|
||||||
|
fn tree_fixture() -> Vec<(usize, usize, f64)> {
|
||||||
|
vec![(0, 1, 3.0), (0, 2, 5.0), (0, 3, 4.0), (0, 4, 6.0)]
|
||||||
|
}
|
||||||
|
|
||||||
|
/// (winner, loser, score_diff)
|
||||||
|
fn fixture() -> Vec<(usize, usize, f64)> {
|
||||||
|
vec![
|
||||||
|
(0, 1, 3.0),
|
||||||
|
(0, 2, 5.0),
|
||||||
|
(1, 2, 2.0),
|
||||||
|
(3, 4, 1.0),
|
||||||
|
(0, 3, 4.0),
|
||||||
|
(1, 4, 2.5),
|
||||||
|
(2, 3, 0.5),
|
||||||
|
(0, 4, 6.0),
|
||||||
|
(1, 3, 1.5),
|
||||||
|
(2, 4, 3.0),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Invert a small symmetric positive-definite matrix by Gauss-Jordan.
|
||||||
|
fn inverse(mut a: Vec<Vec<f64>>) -> Vec<Vec<f64>> {
|
||||||
|
let n = a.len();
|
||||||
|
let mut inv: Vec<Vec<f64>> = (0..n)
|
||||||
|
.map(|i| (0..n).map(|j| if i == j { 1.0 } else { 0.0 }).collect())
|
||||||
|
.collect();
|
||||||
|
for col in 0..n {
|
||||||
|
// partial pivot
|
||||||
|
let mut piv = col;
|
||||||
|
for r in col + 1..n {
|
||||||
|
if a[r][col].abs() > a[piv][col].abs() {
|
||||||
|
piv = r;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
a.swap(col, piv);
|
||||||
|
inv.swap(col, piv);
|
||||||
|
let d = a[col][col];
|
||||||
|
for j in 0..n {
|
||||||
|
a[col][j] /= d;
|
||||||
|
inv[col][j] /= d;
|
||||||
|
}
|
||||||
|
for r in 0..n {
|
||||||
|
if r == col {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let f = a[r][col];
|
||||||
|
for j in 0..n {
|
||||||
|
a[r][j] -= f * a[col][j];
|
||||||
|
inv[r][j] -= f * inv[col][j];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inv
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The exact posterior of a linear-Gaussian model:
|
||||||
|
/// precision = prior precision + sum of a_k a_k^T / v_k.
|
||||||
|
fn exact_for(obs: &[(usize, usize, f64)]) -> (Vec<f64>, Vec<Vec<f64>>) {
|
||||||
|
let mut lambda = vec![vec![0.0; N]; N];
|
||||||
|
let mut eta = [0.0; N];
|
||||||
|
for (i, row) in lambda.iter_mut().enumerate() {
|
||||||
|
row[i] = 1.0 / (SIGMA0 * SIGMA0);
|
||||||
|
eta[i] = MU0 / (SIGMA0 * SIGMA0);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each 1v1 observation: d ~ N(x_a - x_b, score_sigma^2 + 2 beta^2)
|
||||||
|
let v = SCORE_SIGMA * SCORE_SIGMA + 2.0 * BETA * BETA;
|
||||||
|
for &(a, b, d) in obs {
|
||||||
|
let mut vec_a = [0.0; N];
|
||||||
|
vec_a[a] = 1.0;
|
||||||
|
vec_a[b] = -1.0;
|
||||||
|
for i in 0..N {
|
||||||
|
for j in 0..N {
|
||||||
|
lambda[i][j] += vec_a[i] * vec_a[j] / v;
|
||||||
|
}
|
||||||
|
eta[i] += vec_a[i] * d / v;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let cov = inverse(lambda);
|
||||||
|
let mean: Vec<f64> = (0..N)
|
||||||
|
.map(|i| (0..N).map(|j| cov[i][j] * eta[j]).sum())
|
||||||
|
.collect();
|
||||||
|
(mean, cov)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn key(i: usize) -> &'static str {
|
||||||
|
["c0", "c1", "c2", "c3", "c4"][i]
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns (worst mean error, worst sd ratio).
|
||||||
|
fn fitted(
|
||||||
|
obs: &[(usize, usize, f64)],
|
||||||
|
) -> History<i64, ConstantDrift, trueskill_tt::NullObserver, &'static str> {
|
||||||
|
let mut h: History<i64, _, _, &'static str> = History::builder()
|
||||||
|
.mu(MU0)
|
||||||
|
.sigma(SIGMA0)
|
||||||
|
.beta(BETA)
|
||||||
|
.score_sigma(SCORE_SIGMA)
|
||||||
|
.drift(ConstantDrift(0.0))
|
||||||
|
.convergence(ConvergenceOptions {
|
||||||
|
max_iter: 20_000,
|
||||||
|
epsilon: 1e-13,
|
||||||
|
alpha: 1.0,
|
||||||
|
})
|
||||||
|
.build();
|
||||||
|
|
||||||
|
let events: Vec<Event<i64, &'static str>> = obs
|
||||||
|
.iter()
|
||||||
|
.copied()
|
||||||
|
.map(|(a, b, d)| Event {
|
||||||
|
time: 1,
|
||||||
|
teams: smallvec![
|
||||||
|
Team::with_members([Member::new(key(a))]),
|
||||||
|
Team::with_members([Member::new(key(b))]),
|
||||||
|
],
|
||||||
|
outcome: Outcome::scores([d, 0.0]),
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
h.add_events(events).unwrap();
|
||||||
|
let report = h.converge().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.converged,
|
||||||
|
"fixture must converge: {:?}",
|
||||||
|
report.final_step
|
||||||
|
);
|
||||||
|
|
||||||
|
h
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns (worst mean error, worst sd ratio gap).
|
||||||
|
fn run(name: &str, obs: Vec<(usize, usize, f64)>) -> (f64, f64) {
|
||||||
|
println!("\n########## {name} ##########");
|
||||||
|
let h = fitted(&obs);
|
||||||
|
let (mean, cov) = exact_for(&obs);
|
||||||
|
|
||||||
|
println!("\n== marginals: crate vs the exact linear-Gaussian posterior ==");
|
||||||
|
println!(
|
||||||
|
"{:>4} {:>12} {:>12} {:>12} {:>12} {:>8}",
|
||||||
|
"node", "crate mu", "exact mu", "crate sd", "exact sd", "sd ratio"
|
||||||
|
);
|
||||||
|
for i in 0..N {
|
||||||
|
let g = h.current_skill(&key(i)).unwrap();
|
||||||
|
let exact_sd = cov[i][i].sqrt();
|
||||||
|
println!(
|
||||||
|
"{:>4} {:>12.6} {:>12.6} {:>12.6} {:>12.6} {:>8.3}",
|
||||||
|
key(i),
|
||||||
|
g.mu(),
|
||||||
|
mean[i],
|
||||||
|
g.sigma(),
|
||||||
|
exact_sd,
|
||||||
|
g.sigma() / exact_sd
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut worst_mean = 0.0f64;
|
||||||
|
let mut worst_ratio_gap = 0.0f64;
|
||||||
|
for i in 0..N {
|
||||||
|
let g = h.current_skill(&key(i)).unwrap();
|
||||||
|
worst_mean = worst_mean.max((g.mu() - mean[i]).abs());
|
||||||
|
worst_ratio_gap = worst_ratio_gap.max((g.sigma() / cov[i][i].sqrt() - 1.0).abs());
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("\n== what a consumer actually computes for a DIFFERENCE ==");
|
||||||
|
println!(
|
||||||
|
"{:>8} {:>12} {:>14} {:>14} {:>12}",
|
||||||
|
"pair", "exact", "naive(exact)", "naive(crate)", "crate err"
|
||||||
|
);
|
||||||
|
for i in 0..N {
|
||||||
|
for j in i + 1..N {
|
||||||
|
if i != 0 && j != 1 {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let gi = h.current_skill(&key(i)).unwrap();
|
||||||
|
let gj = h.current_skill(&key(j)).unwrap();
|
||||||
|
let exact_sd = (cov[i][i] + cov[j][j] - 2.0 * cov[i][j]).sqrt();
|
||||||
|
let naive_exact = (cov[i][i] + cov[j][j]).sqrt();
|
||||||
|
let naive_crate = (gi.sigma().powi(2) + gj.sigma().powi(2)).sqrt();
|
||||||
|
let corr = cov[i][j] / (cov[i][i].sqrt() * cov[j][j].sqrt());
|
||||||
|
println!(
|
||||||
|
"{:>8} {:>12.6} {:>14.6} {:>14.6} {:>11.3}x (corr {corr:.4})",
|
||||||
|
format!("{}-{}", key(i), key(j)),
|
||||||
|
exact_sd,
|
||||||
|
naive_exact,
|
||||||
|
naive_crate,
|
||||||
|
naive_crate / exact_sd
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
(worst_mean, worst_ratio_gap)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// With no cycles there is nothing for message passing to approximate.
|
||||||
|
#[test]
|
||||||
|
fn on_a_tree_the_marginals_are_exact() {
|
||||||
|
let (mean_err, sd_gap) = run("TREE (star: no loops, BP is exact)", tree_fixture());
|
||||||
|
assert!(
|
||||||
|
mean_err < 1e-9,
|
||||||
|
"tree means should be exact, worst error {mean_err}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
sd_gap < 1e-9,
|
||||||
|
"tree sigmas should be exact, worst ratio gap {sd_gap}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// With cycles the means stay exact — the property ratings depend on — while
|
||||||
|
/// the variances do not. The variance gap is measured and reported rather than
|
||||||
|
/// asserted; see the module docs.
|
||||||
|
#[test]
|
||||||
|
fn with_cycles_the_means_stay_exact_but_the_variances_shrink() {
|
||||||
|
let (mean_err, sd_gap) = run("LOOPY (round robin)", fixture());
|
||||||
|
assert!(
|
||||||
|
mean_err < 1e-9,
|
||||||
|
"loopy means must still be exact, worst error {mean_err}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
sd_gap > 0.1,
|
||||||
|
"the loopy variance gap is the premise of #46; if it has closed, that \
|
||||||
|
issue and these docs need revisiting (worst ratio gap {sd_gap})"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The point of #46: `posterior_of` must reproduce the exact joint, including
|
||||||
|
/// the correlation that marginals cannot express.
|
||||||
|
#[test]
|
||||||
|
fn posterior_of_matches_the_exact_joint() {
|
||||||
|
for (name, obs) in [("tree", tree_fixture()), ("loopy", fixture())] {
|
||||||
|
let h = fitted(&obs);
|
||||||
|
let (_, cov) = exact_for(&obs);
|
||||||
|
|
||||||
|
println!("\n== posterior_of vs exact ({name}) ==");
|
||||||
|
println!(
|
||||||
|
"{:>12} {:>14} {:>14} {:>10}",
|
||||||
|
"functional", "posterior_of", "exact", "ratio"
|
||||||
|
);
|
||||||
|
|
||||||
|
for (i, j) in [(0usize, 1usize), (0, 2), (1, 3), (2, 4)] {
|
||||||
|
let got = h
|
||||||
|
.posterior_of(&[(&key(i), 1.0), (&key(j), -1.0)])
|
||||||
|
.expect("scored slice should have a joint");
|
||||||
|
let exact_sd = (cov[i][i] + cov[j][j] - 2.0 * cov[i][j]).sqrt();
|
||||||
|
println!(
|
||||||
|
"{:>12} {:>14.6} {:>14.6} {:>10.4}",
|
||||||
|
format!("{}-{}", key(i), key(j)),
|
||||||
|
got.sigma(),
|
||||||
|
exact_sd,
|
||||||
|
got.sigma() / exact_sd
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
(got.sigma() - exact_sd).abs() / exact_sd < 1e-9,
|
||||||
|
"{name} {}-{}: posterior_of gave {} where the exact joint is {exact_sd}",
|
||||||
|
key(i),
|
||||||
|
key(j),
|
||||||
|
got.sigma()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// A single competitor: this is where the loopy marginal was 2x narrow.
|
||||||
|
for (i, row) in cov.iter().enumerate() {
|
||||||
|
let got = h.posterior_of(&[(&key(i), 1.0)]).unwrap();
|
||||||
|
let exact_sd = row[i].sqrt();
|
||||||
|
assert!(
|
||||||
|
(got.sigma() - exact_sd).abs() / exact_sd < 1e-9,
|
||||||
|
"{name} {}: posterior_of gave {} where exact is {exact_sd}",
|
||||||
|
key(i),
|
||||||
|
got.sigma()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
println!(" single-competitor marginals also exact");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cost of the dense solve as the slice grows. Recorded, not asserted.
|
||||||
|
#[test]
|
||||||
|
#[ignore = "timing probe, run explicitly"]
|
||||||
|
fn cost_scaling() {
|
||||||
|
use std::time::Instant;
|
||||||
|
for n in [50usize, 100, 200, 400, 800] {
|
||||||
|
let names: Vec<String> = (0..n).map(|i| format!("c{i}")).collect();
|
||||||
|
let mut h: History<i64, _, _, String> = History::builder_with_key()
|
||||||
|
.score_sigma(2.0)
|
||||||
|
.drift(ConstantDrift(0.0))
|
||||||
|
.convergence(ConvergenceOptions {
|
||||||
|
max_iter: 200,
|
||||||
|
epsilon: 1e-8,
|
||||||
|
alpha: 1.0,
|
||||||
|
})
|
||||||
|
.build();
|
||||||
|
let mut seed = 5u64;
|
||||||
|
let mut rnd = move || {
|
||||||
|
seed ^= seed << 13;
|
||||||
|
seed ^= seed >> 7;
|
||||||
|
seed ^= seed << 17;
|
||||||
|
seed
|
||||||
|
};
|
||||||
|
let events: Vec<Event<i64, String>> = (0..n * 4)
|
||||||
|
.map(|_| {
|
||||||
|
let a = (rnd() as usize) % n;
|
||||||
|
let mut b = (rnd() as usize) % n;
|
||||||
|
if b == a {
|
||||||
|
b = (b + 1) % n;
|
||||||
|
}
|
||||||
|
Event {
|
||||||
|
time: 1,
|
||||||
|
teams: smallvec![
|
||||||
|
Team::with_members([Member::new(names[a].clone())]),
|
||||||
|
Team::with_members([Member::new(names[b].clone())]),
|
||||||
|
],
|
||||||
|
outcome: Outcome::scores([1.0, 0.0]),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
h.add_events(events).unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
|
let t = Instant::now();
|
||||||
|
let g = h
|
||||||
|
.posterior_of(&[(&names[0], 1.0), (&names[1], -1.0)])
|
||||||
|
.unwrap();
|
||||||
|
println!(" n={n:>4}: {:>10.2?} sigma {:.6}", t.elapsed(), g.sigma());
|
||||||
|
}
|
||||||
|
}
|
||||||
+8
-8
@@ -42,7 +42,7 @@ fn every_observer_callback_fires() {
|
|||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.record_winner(&"b", &"c", 2).unwrap();
|
h.record_winner(&"b", &"c", 2).unwrap();
|
||||||
h.record_winner(&"c", &"a", 3).unwrap();
|
h.record_winner(&"c", &"a", 3).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
assert!(
|
assert!(
|
||||||
!recorder.iterations.lock().unwrap().is_empty(),
|
!recorder.iterations.lock().unwrap().is_empty(),
|
||||||
@@ -65,7 +65,7 @@ fn slice_callbacks_report_the_slice_they_swept() {
|
|||||||
|
|
||||||
h.record_winner(&"a", &"b", 10).unwrap();
|
h.record_winner(&"a", &"b", 10).unwrap();
|
||||||
h.record_winner(&"a", &"b", 20).unwrap();
|
h.record_winner(&"a", &"b", 20).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let slices = recorder.slices.lock().unwrap();
|
let slices = recorder.slices.lock().unwrap();
|
||||||
|
|
||||||
@@ -93,7 +93,7 @@ fn a_single_slice_history_still_reports_its_sweep() {
|
|||||||
let mut h = History::builder().observer(Arc::clone(&recorder)).build();
|
let mut h = History::builder().observer(Arc::clone(&recorder)).build();
|
||||||
|
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let slices = recorder.slices.lock().unwrap();
|
let slices = recorder.slices.lock().unwrap();
|
||||||
assert!(
|
assert!(
|
||||||
@@ -112,7 +112,7 @@ fn a_shared_observer_reaches_the_callers_handle() {
|
|||||||
let mut h = History::builder().observer(Arc::clone(&recorder)).build();
|
let mut h = History::builder().observer(Arc::clone(&recorder)).build();
|
||||||
|
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
assert!(!recorder.iterations.lock().unwrap().is_empty());
|
assert!(!recorder.iterations.lock().unwrap().is_empty());
|
||||||
assert!(!recorder.slices.lock().unwrap().is_empty());
|
assert!(!recorder.slices.lock().unwrap().is_empty());
|
||||||
@@ -125,12 +125,12 @@ fn a_trait_object_observer_works() {
|
|||||||
let boxed: Box<dyn Observer<i64>> = Box::new(Recorder::default());
|
let boxed: Box<dyn Observer<i64>> = Box::new(Recorder::default());
|
||||||
let mut h = History::builder().observer(boxed).build();
|
let mut h = History::builder().observer(boxed).build();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let shared: Arc<dyn Observer<i64>> = Arc::new(Recorder::default());
|
let shared: Arc<dyn Observer<i64>> = Arc::new(Recorder::default());
|
||||||
let mut h = History::builder().observer(Arc::clone(&shared)).build();
|
let mut h = History::builder().observer(Arc::clone(&shared)).build();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A non-shared observer can be reclaimed after convergence instead.
|
/// A non-shared observer can be reclaimed after convergence instead.
|
||||||
@@ -138,7 +138,7 @@ fn a_trait_object_observer_works() {
|
|||||||
fn into_observer_returns_the_accumulated_state() {
|
fn into_observer_returns_the_accumulated_state() {
|
||||||
let mut h = History::builder().observer(Recorder::default()).build();
|
let mut h = History::builder().observer(Recorder::default()).build();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
// Readable in place...
|
// Readable in place...
|
||||||
assert!(!h.observer().iterations.lock().unwrap().is_empty());
|
assert!(!h.observer().iterations.lock().unwrap().is_empty());
|
||||||
@@ -155,7 +155,7 @@ fn a_borrowed_observer_works() {
|
|||||||
{
|
{
|
||||||
let mut h = History::builder().observer(&recorder).build();
|
let mut h = History::builder().observer(&recorder).build();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
}
|
}
|
||||||
assert!(!recorder.iterations.lock().unwrap().is_empty());
|
assert!(!recorder.iterations.lock().unwrap().is_empty());
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,153 @@
|
|||||||
|
//! `predict_margin`: the predictive distribution of a scored matchup.
|
||||||
|
|
||||||
|
use smallvec::smallvec;
|
||||||
|
use trueskill_tt::{
|
||||||
|
ConstantDrift, ConvergenceOptions, Event, History, InferenceError, Member, Outcome, Team,
|
||||||
|
UnknownKeys,
|
||||||
|
};
|
||||||
|
|
||||||
|
fn builder(
|
||||||
|
policy: UnknownKeys,
|
||||||
|
) -> History<i64, ConstantDrift, trueskill_tt::NullObserver, &'static str> {
|
||||||
|
History::builder()
|
||||||
|
.mu(0.0)
|
||||||
|
.sigma(6.0)
|
||||||
|
.beta(1.0)
|
||||||
|
.score_sigma(2.0)
|
||||||
|
.drift(ConstantDrift(0.0))
|
||||||
|
.unknown_keys(policy)
|
||||||
|
.convergence(ConvergenceOptions {
|
||||||
|
max_iter: 5_000,
|
||||||
|
epsilon: 1e-12,
|
||||||
|
alpha: 1.0,
|
||||||
|
})
|
||||||
|
.build()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn round(a: &'static str, b: &'static str, sa: f64, sb: f64) -> Event<i64, &'static str> {
|
||||||
|
Event {
|
||||||
|
time: 1,
|
||||||
|
teams: smallvec![
|
||||||
|
Team::with_members([Member::new(a)]),
|
||||||
|
Team::with_members([Member::new(b)]),
|
||||||
|
],
|
||||||
|
outcome: Outcome::scores([sa, sb]),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A history where "veteran" and "regular" are well observed and "novice"
|
||||||
|
/// appears once.
|
||||||
|
fn fitted(
|
||||||
|
policy: UnknownKeys,
|
||||||
|
) -> History<i64, ConstantDrift, trueskill_tt::NullObserver, &'static str> {
|
||||||
|
let mut h = builder(policy);
|
||||||
|
let mut events: Vec<_> = (0..40)
|
||||||
|
.map(|t| round("veteran", "regular", 10.0 + f64::from(t % 3), 5.0))
|
||||||
|
.collect();
|
||||||
|
events.push(round("veteran", "novice", 10.0, 6.0));
|
||||||
|
h.add_events(events).unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
h
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The property #48 exists for: the interval must widen when the model knows
|
||||||
|
/// less. Their hand-fitted noise law quoted the same sigma for a competitor
|
||||||
|
/// with forty rounds and one with none.
|
||||||
|
#[test]
|
||||||
|
fn the_interval_widens_as_the_model_knows_less() {
|
||||||
|
let h = fitted(UnknownKeys::Prior);
|
||||||
|
|
||||||
|
let well_known = h
|
||||||
|
.predict_margin(&[&[&"veteran"], &[&"regular"]])
|
||||||
|
.unwrap()
|
||||||
|
.sigma();
|
||||||
|
let thin = h
|
||||||
|
.predict_margin(&[&[&"veteran"], &[&"novice"]])
|
||||||
|
.unwrap()
|
||||||
|
.sigma();
|
||||||
|
let unseen = h
|
||||||
|
.predict_margin(&[&[&"veteran"], &[&"stranger"]])
|
||||||
|
.unwrap()
|
||||||
|
.sigma();
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
well_known < thin && thin < unseen,
|
||||||
|
"margin width should grow as evidence thins: {well_known} < {thin} < {unseen}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// #48's second requirement: an unseen competitor is a legitimate question, not
|
||||||
|
/// an error, and the answer should come from the prior rather than be faked.
|
||||||
|
#[test]
|
||||||
|
fn an_unseen_competitor_is_answered_from_the_prior() {
|
||||||
|
let h = fitted(UnknownKeys::Prior);
|
||||||
|
let g = h.predict_margin(&[&[&"nobody"], &[&"no_one"]]).unwrap();
|
||||||
|
|
||||||
|
// Two unknowns: the gap is centred on zero and carries both priors plus
|
||||||
|
// both performance noises plus the observation noise.
|
||||||
|
assert!(g.mu().abs() < 1e-9, "mu {}", g.mu());
|
||||||
|
let expected = (2.0 * 36.0 + 2.0 * 1.0 + 4.0f64).sqrt();
|
||||||
|
assert!(
|
||||||
|
(g.sigma() - expected).abs() < 1e-9,
|
||||||
|
"sigma {} vs expected {expected}",
|
||||||
|
g.sigma()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn reject_still_rejects() {
|
||||||
|
let h = fitted(UnknownKeys::Reject);
|
||||||
|
assert!(matches!(
|
||||||
|
h.predict_margin(&[&[&"veteran"], &[&"stranger"]]),
|
||||||
|
Err(InferenceError::UnknownKey { .. })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The margin is the *difference*, so it must be antisymmetric in the teams.
|
||||||
|
#[test]
|
||||||
|
fn swapping_the_teams_negates_the_margin() {
|
||||||
|
let h = fitted(UnknownKeys::Prior);
|
||||||
|
let forward = h.predict_margin(&[&[&"veteran"], &[&"regular"]]).unwrap();
|
||||||
|
let reverse = h.predict_margin(&[&[&"regular"], &[&"veteran"]]).unwrap();
|
||||||
|
|
||||||
|
assert!((forward.mu() + reverse.mu()).abs() < 1e-9);
|
||||||
|
assert!((forward.sigma() - reverse.sigma()).abs() < 1e-12);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The predictive interval must be wider than the skill gap alone: it also
|
||||||
|
/// carries per-event performance noise and the observation noise.
|
||||||
|
#[test]
|
||||||
|
fn the_predictive_interval_exceeds_the_skill_uncertainty() {
|
||||||
|
let h = fitted(UnknownKeys::Prior);
|
||||||
|
let skill_gap = h
|
||||||
|
.posterior_of(&[(&"veteran", 1.0), (&"regular", -1.0)])
|
||||||
|
.unwrap();
|
||||||
|
let predictive = h.predict_margin(&[&[&"veteran"], &[&"regular"]]).unwrap();
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
(predictive.mu() - skill_gap.mu()).abs() < 1e-12,
|
||||||
|
"means agree"
|
||||||
|
);
|
||||||
|
// beta^2 twice plus score_sigma^2 = 2 + 4.
|
||||||
|
let expected = (skill_gap.sigma().powi(2) + 6.0).sqrt();
|
||||||
|
assert!((predictive.sigma() - expected).abs() < 1e-12);
|
||||||
|
assert!(predictive.sigma() > skill_gap.sigma());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn shape_errors_are_reported() {
|
||||||
|
let h = fitted(UnknownKeys::Prior);
|
||||||
|
assert!(matches!(
|
||||||
|
h.predict_margin(&[&[&"veteran"]]),
|
||||||
|
Err(InferenceError::MismatchedShape {
|
||||||
|
expected: 2,
|
||||||
|
got: 1,
|
||||||
|
..
|
||||||
|
})
|
||||||
|
));
|
||||||
|
let empty: [&&str; 0] = [];
|
||||||
|
assert!(matches!(
|
||||||
|
h.predict_margin(&[&[&"veteran"], &empty]),
|
||||||
|
Err(InferenceError::EmptyTeam { team: 1 })
|
||||||
|
));
|
||||||
|
}
|
||||||
+133
-7
@@ -9,7 +9,7 @@ fn history_with(names: &[&'static str], p_draw: f64) -> History {
|
|||||||
for pair in names.windows(2) {
|
for pair in names.windows(2) {
|
||||||
h.record_winner(&pair[0], &pair[1], 1).unwrap();
|
h.record_winner(&pair[0], &pair[1], 1).unwrap();
|
||||||
}
|
}
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
h
|
h
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -20,7 +20,14 @@ fn unknown_keys_are_reported_not_silently_dropped() {
|
|||||||
let err = h
|
let err = h
|
||||||
.predict_outcome(&[&[&"a"], &[&"ghost"]])
|
.predict_outcome(&[&[&"a"], &[&"ghost"]])
|
||||||
.expect_err("an unknown key must not yield a confident prediction");
|
.expect_err("an unknown key must not yield a confident prediction");
|
||||||
assert_eq!(err, InferenceError::UnknownKey { team: 1, member: 0 });
|
assert_eq!(
|
||||||
|
err,
|
||||||
|
InferenceError::UnknownKey {
|
||||||
|
team: 1,
|
||||||
|
member: 0,
|
||||||
|
key: "\"ghost\"".to_owned(),
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
// Every prediction entry point, not just one.
|
// Every prediction entry point, not just one.
|
||||||
assert!(
|
assert!(
|
||||||
@@ -35,7 +42,14 @@ fn unknown_keys_are_reported_not_silently_dropped() {
|
|||||||
fn an_entirely_unknown_team_is_an_error() {
|
fn an_entirely_unknown_team_is_an_error() {
|
||||||
let h = history_with(&["a", "b"], 0.0);
|
let h = history_with(&["a", "b"], 0.0);
|
||||||
let err = h.predict_outcome(&[&[&"a"], &[&"x", &"y"]]).unwrap_err();
|
let err = h.predict_outcome(&[&[&"a"], &[&"x", &"y"]]).unwrap_err();
|
||||||
assert_eq!(err, InferenceError::UnknownKey { team: 1, member: 0 });
|
assert_eq!(
|
||||||
|
err,
|
||||||
|
InferenceError::UnknownKey {
|
||||||
|
team: 1,
|
||||||
|
member: 0,
|
||||||
|
key: "\"x\"".to_owned(),
|
||||||
|
}
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -184,7 +198,7 @@ fn the_stronger_competitor_is_favoured() {
|
|||||||
for t in 1..=10 {
|
for t in 1..=10 {
|
||||||
h.record_winner(&"strong", &"weak", t).unwrap();
|
h.record_winner(&"strong", &"weak", t).unwrap();
|
||||||
}
|
}
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let p = h.predict_outcome(&[&[&"strong"], &[&"weak"]]).unwrap();
|
let p = h.predict_outcome(&[&[&"strong"], &[&"weak"]]).unwrap();
|
||||||
let (best, _) = p.most_likely().expect("a most likely outcome");
|
let (best, _) = p.most_likely().expect("a most likely outcome");
|
||||||
@@ -206,7 +220,7 @@ fn team_size_affects_the_prediction() {
|
|||||||
.winner(0)
|
.winner(0)
|
||||||
.commit()
|
.commit()
|
||||||
.unwrap();
|
.unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let p = h.predict_outcome(&[&[&"a", &"b"], &[&"c"]]).unwrap();
|
let p = h.predict_outcome(&[&[&"a", &"b"], &[&"c"]]).unwrap();
|
||||||
assert!((p.total() - 1.0).abs() < 1e-6, "total = {}", p.total());
|
assert!((p.total() - 1.0).abs() < 1e-6, "total = {}", p.total());
|
||||||
@@ -229,7 +243,7 @@ fn information_gain_prefers_the_uncertain_pairing() {
|
|||||||
h.record_winner(&"rival", &"known", t + 100).unwrap();
|
h.record_winner(&"rival", &"known", t + 100).unwrap();
|
||||||
}
|
}
|
||||||
h.record_winner(&"known", &"newcomer", 500).unwrap();
|
h.record_winner(&"known", &"newcomer", 500).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let settled = h
|
let settled = h
|
||||||
.expected_information_gain(&[&[&"known"], &[&"rival"]])
|
.expected_information_gain(&[&[&"known"], &[&"rival"]])
|
||||||
@@ -271,7 +285,11 @@ fn information_gain_reports_unknown_keys() {
|
|||||||
assert_eq!(
|
assert_eq!(
|
||||||
h.expected_information_gain(&[&[&"a"], &[&"ghost"]])
|
h.expected_information_gain(&[&[&"a"], &[&"ghost"]])
|
||||||
.unwrap_err(),
|
.unwrap_err(),
|
||||||
InferenceError::UnknownKey { team: 1, member: 0 }
|
InferenceError::UnknownKey {
|
||||||
|
team: 1,
|
||||||
|
member: 0,
|
||||||
|
key: "\"ghost\"".to_owned(),
|
||||||
|
}
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -289,3 +307,111 @@ fn information_gain_accounts_for_draws() {
|
|||||||
let dist = with_draws.predict_outcome(&[&[&"a"], &[&"b"]]).unwrap();
|
let dist = with_draws.predict_outcome(&[&[&"a"], &[&"b"]]).unwrap();
|
||||||
assert!(dist.probability_of(&[0, 0]) > 0.0);
|
assert!(dist.probability_of(&[0, 0]) > 0.0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The defect that cost a consumer a day: `UnknownKey { team: 0, member: 0 }`
|
||||||
|
/// says nothing about *which* key is unknown, so the natural handling — log it,
|
||||||
|
/// fall back to a neutral value — converts a total miss into a plausible
|
||||||
|
/// constant. The key has to be in the error, and in its `Display`.
|
||||||
|
#[test]
|
||||||
|
fn unknown_key_names_the_key_it_could_not_find() {
|
||||||
|
let h = history_with(&["a", "b"], 0.0);
|
||||||
|
let err = h.predict_outcome(&[&[&"a"], &[&"never_seen"]]).unwrap_err();
|
||||||
|
|
||||||
|
match &err {
|
||||||
|
InferenceError::UnknownKey { key, .. } => {
|
||||||
|
assert!(
|
||||||
|
key.contains("never_seen"),
|
||||||
|
"the error should name the key, got {key}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
other => panic!("expected UnknownKey, got {other:?}"),
|
||||||
|
}
|
||||||
|
|
||||||
|
let rendered = err.to_string();
|
||||||
|
assert!(
|
||||||
|
rendered.contains("never_seen"),
|
||||||
|
"Display should name the key: {rendered}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
rendered.contains("pre-filter"),
|
||||||
|
"Display should say what to do about it: {rendered}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// UnknownKeys policy
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
fn history_with_policy(names: &[&'static str], policy: trueskill_tt::UnknownKeys) -> History {
|
||||||
|
let mut h = History::builder().unknown_keys(policy).build();
|
||||||
|
for pair in names.windows(2) {
|
||||||
|
h.record_winner(&pair[0], &pair[1], 1).unwrap();
|
||||||
|
}
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
h
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn reject_is_the_default() {
|
||||||
|
let h = history_with(&["a", "b"], 0.0);
|
||||||
|
assert!(matches!(
|
||||||
|
h.predict_outcome(&[&[&"a"], &[&"ghost"]]),
|
||||||
|
Err(InferenceError::UnknownKey { .. })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn prior_answers_instead_of_erroring() {
|
||||||
|
let h = history_with_policy(&["a", "b"], trueskill_tt::UnknownKeys::Prior);
|
||||||
|
let p = h
|
||||||
|
.predict_outcome(&[&[&"a"], &[&"ghost"]])
|
||||||
|
.expect("Prior should answer rather than reject");
|
||||||
|
assert!((p.total() - 1.0).abs() < 1e-6);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Two competitors the model has never seen are genuinely a coin flip. The
|
||||||
|
/// point is that this is now *derived* rather than a constant a caller
|
||||||
|
/// substitutes after swallowing an error.
|
||||||
|
#[test]
|
||||||
|
fn two_unknown_competitors_are_an_honest_coin_flip() {
|
||||||
|
let h = history_with_policy(&["a", "b"], trueskill_tt::UnknownKeys::Prior);
|
||||||
|
let wins = h
|
||||||
|
.predict_win_probabilities(&[&[&"nobody"], &[&"no_one"]])
|
||||||
|
.unwrap();
|
||||||
|
assert!((wins[0] - 0.5).abs() < 1e-9, "{wins:?}");
|
||||||
|
assert!((wins[1] - 0.5).abs() < 1e-9, "{wins:?}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The property that rules out a `Skip` mode: an unknown member must make a
|
||||||
|
/// team *less* certain, never more. Skipping would drop the member's variance
|
||||||
|
/// from the sum and narrow the team, which is backwards.
|
||||||
|
#[test]
|
||||||
|
fn an_unknown_member_widens_its_team_rather_than_narrowing_it() {
|
||||||
|
let h = history_with_policy(&["a", "b", "c"], trueskill_tt::UnknownKeys::Prior);
|
||||||
|
|
||||||
|
// "a" alone against "b" — then "a" plus an unknown partner against "b".
|
||||||
|
let solo = h.predict_win_probabilities(&[&[&"a"], &[&"b"]]).unwrap();
|
||||||
|
let with_unknown = h
|
||||||
|
.predict_win_probabilities(&[&[&"a", &"stranger"], &[&"b"]])
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
// Adding an unknown partner pulls the outcome toward even, because the
|
||||||
|
// team's performance spread grew.
|
||||||
|
assert!(
|
||||||
|
(with_unknown[0] - 0.5).abs() < (solo[0] - 0.5).abs(),
|
||||||
|
"an unknown partner should make the result less certain: solo {solo:?}, \
|
||||||
|
with unknown {with_unknown:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn prior_reaches_every_prediction_entry_point() {
|
||||||
|
let h = history_with_policy(&["a", "b"], trueskill_tt::UnknownKeys::Prior);
|
||||||
|
let teams: &[&[&&str]] = &[&[&"a"], &[&"ghost"]];
|
||||||
|
|
||||||
|
assert!(h.predict_quality(teams).is_ok());
|
||||||
|
assert!(h.predict_win_probabilities(teams).is_ok());
|
||||||
|
assert!(h.predict_outcome(teams).is_ok());
|
||||||
|
assert!(h.predict_ranking(teams, &[0, 1]).is_ok());
|
||||||
|
assert!(h.expected_information_gain(teams).is_ok());
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# Seeds for failure cases proptest has generated in the past. It is
|
||||||
|
# automatically read and these particular cases re-run before any
|
||||||
|
# novel cases are generated.
|
||||||
|
#
|
||||||
|
# It is recommended to check this file in to source control so that
|
||||||
|
# everyone who runs the test benefits from these saved cases.
|
||||||
|
cc 8859be600e638573980f78622b8fcd8b4553ca34a9a041746c417f7e4293f89c # shrinks to games = [(0, 1), (0, 1), (4, 6), (2, 0), (0, 1), (0, 6), (0, 1), (2, 0), (6, 4), (0, 1), (0, 2), (6, 4), (1, 0), (4, 0), (0, 2)]
|
||||||
+23
-7
@@ -26,7 +26,10 @@ const KEYS: [&str; 8] = ["a", "b", "c", "d", "e", "f", "g", "h"];
|
|||||||
fn history_from(games: &[(usize, usize)]) -> History {
|
fn history_from(games: &[(usize, usize)]) -> History {
|
||||||
let mut h = History::builder()
|
let mut h = History::builder()
|
||||||
.convergence(ConvergenceOptions {
|
.convergence(ConvergenceOptions {
|
||||||
max_iter: 200,
|
// 200 was not enough: the batched side stopped at the cap with a
|
||||||
|
// step of 3.4e-9, so this test was comparing two truncated fits and
|
||||||
|
// attributing the gap to ingestion order.
|
||||||
|
max_iter: 20_000,
|
||||||
epsilon: 1e-10,
|
epsilon: 1e-10,
|
||||||
..ConvergenceOptions::default()
|
..ConvergenceOptions::default()
|
||||||
})
|
})
|
||||||
@@ -61,7 +64,7 @@ proptest! {
|
|||||||
fn converged_posteriors_are_always_finite(games in pairs()) {
|
fn converged_posteriors_are_always_finite(games in pairs()) {
|
||||||
let mut h = history_from(&games);
|
let mut h = history_from(&games);
|
||||||
|
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
for key in KEYS {
|
for key in KEYS {
|
||||||
for (time, g) in h.learning_curve(key) {
|
for (time, g) in h.learning_curve(key) {
|
||||||
@@ -79,7 +82,7 @@ proptest! {
|
|||||||
fn log_evidence_is_a_finite_log_probability(games in pairs()) {
|
fn log_evidence_is_a_finite_log_probability(games in pairs()) {
|
||||||
let mut h = history_from(&games);
|
let mut h = history_from(&games);
|
||||||
|
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let batch = h.log_evidence();
|
let batch = h.log_evidence();
|
||||||
let filtered = h.filtered_log_evidence();
|
let filtered = h.filtered_log_evidence();
|
||||||
@@ -98,7 +101,7 @@ proptest! {
|
|||||||
|
|
||||||
let before = h.filtered_log_evidence();
|
let before = h.filtered_log_evidence();
|
||||||
|
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let after = h.filtered_log_evidence();
|
let after = h.filtered_log_evidence();
|
||||||
|
|
||||||
@@ -114,14 +117,21 @@ proptest! {
|
|||||||
fn ingestion_order_does_not_change_the_answer(games in pairs()) {
|
fn ingestion_order_does_not_change_the_answer(games in pairs()) {
|
||||||
let batched = {
|
let batched = {
|
||||||
let mut h = history_from(&games);
|
let mut h = history_from(&games);
|
||||||
h.converge().unwrap();
|
let report = h.converge().unwrap();
|
||||||
|
prop_assert!(
|
||||||
|
report.converged,
|
||||||
|
"batched side stopped at {} iterations with step {:?}; comparing \
|
||||||
|
two fits that have not converged measures truncation, not order",
|
||||||
|
report.iterations,
|
||||||
|
report.final_step
|
||||||
|
);
|
||||||
h
|
h
|
||||||
};
|
};
|
||||||
|
|
||||||
let incremental = {
|
let incremental = {
|
||||||
let mut h = History::builder()
|
let mut h = History::builder()
|
||||||
.convergence(ConvergenceOptions {
|
.convergence(ConvergenceOptions {
|
||||||
max_iter: 200,
|
max_iter: 20_000,
|
||||||
epsilon: 1e-10,
|
epsilon: 1e-10,
|
||||||
..ConvergenceOptions::default()
|
..ConvergenceOptions::default()
|
||||||
})
|
})
|
||||||
@@ -139,7 +149,13 @@ proptest! {
|
|||||||
.unwrap();
|
.unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
h.converge().unwrap();
|
let report = h.converge().unwrap();
|
||||||
|
prop_assert!(
|
||||||
|
report.converged,
|
||||||
|
"incremental side stopped at {} iterations with step {:?}",
|
||||||
|
report.iterations,
|
||||||
|
report.final_step
|
||||||
|
);
|
||||||
h
|
h
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
+48
-1
@@ -108,7 +108,7 @@ fn history_predict_quality_supports_three_teams() {
|
|||||||
let mut h = History::default();
|
let mut h = History::default();
|
||||||
h.record_winner(&"a", &"b", 1).unwrap();
|
h.record_winner(&"a", &"b", 1).unwrap();
|
||||||
h.record_winner(&"b", &"c", 2).unwrap();
|
h.record_winner(&"b", &"c", 2).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let q = h.predict_quality(&[&[&"a"], &[&"b"], &[&"c"]]).unwrap();
|
let q = h.predict_quality(&[&[&"a"], &[&"b"], &[&"c"]]).unwrap();
|
||||||
assert!(
|
assert!(
|
||||||
@@ -117,3 +117,50 @@ fn history_predict_quality_supports_three_teams() {
|
|||||||
);
|
);
|
||||||
assert!((0.0..=1.0).contains(&q), "out of range: {q}");
|
assert!((0.0..=1.0).contains(&q), "out of range: {q}");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `quality()` for N identical teams has a closed form, which pins the N-group
|
||||||
|
/// determinant path across the whole range rather than at a single golden.
|
||||||
|
///
|
||||||
|
/// For two identical single-player teams the standard result is
|
||||||
|
/// `sqrt(2b^2 / (2b^2 + s1^2 + s2^2))`. With the conventional parameters
|
||||||
|
/// (`sigma = 25/3`, `beta = 25/6`) that ratio is exactly `1/5`, and the N-group
|
||||||
|
/// generalisation is `(1/5)^((n-1)/2)` — one factor per adjacent pair.
|
||||||
|
///
|
||||||
|
/// The n=3 and n=5 values this produces (0.200 and 0.040) are also what the
|
||||||
|
/// `trueskill` Python package returns for the same configuration, so this
|
||||||
|
/// doubles as the cross-implementation check the README asked for.
|
||||||
|
#[test]
|
||||||
|
fn quality_of_identical_teams_follows_its_closed_form() {
|
||||||
|
let g = Gaussian::from_ms(25.0, 25.0 / 3.0);
|
||||||
|
let beta = 25.0 / 6.0;
|
||||||
|
|
||||||
|
for n in 2..=10usize {
|
||||||
|
let groups: Vec<Vec<Gaussian>> = (0..n).map(|_| vec![g]).collect();
|
||||||
|
let refs: Vec<&[Gaussian]> = groups.iter().map(Vec::as_slice).collect();
|
||||||
|
|
||||||
|
let got = quality(&refs, beta);
|
||||||
|
let expected = 0.2f64.powf((n - 1) as f64 / 2.0);
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
(got - expected).abs() / expected < 1e-9,
|
||||||
|
"n={n}: quality {got}, closed form {expected}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Spot-check against the two values the `trueskill` Python package is known
|
||||||
|
/// to produce for this configuration, stated as literals so a future change to
|
||||||
|
/// the closed-form reasoning above cannot quietly take these with it.
|
||||||
|
#[test]
|
||||||
|
fn quality_matches_the_reference_implementation() {
|
||||||
|
let g = Gaussian::from_ms(25.0, 25.0 / 3.0);
|
||||||
|
let beta = 25.0 / 6.0;
|
||||||
|
|
||||||
|
let three: Vec<Vec<Gaussian>> = (0..3).map(|_| vec![g]).collect();
|
||||||
|
let refs: Vec<&[Gaussian]> = three.iter().map(Vec::as_slice).collect();
|
||||||
|
assert!((quality(&refs, beta) - 0.200).abs() < 1e-9);
|
||||||
|
|
||||||
|
let five: Vec<Vec<Gaussian>> = (0..5).map(|_| vec![g]).collect();
|
||||||
|
let refs: Vec<&[Gaussian]> = five.iter().map(Vec::as_slice).collect();
|
||||||
|
assert!((quality(&refs, beta) - 0.040).abs() < 1e-9);
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,165 @@
|
|||||||
|
//! Converging, appending, and converging again must reach the same fixed point
|
||||||
|
//! as converging once over the whole event set.
|
||||||
|
//!
|
||||||
|
//! `tests/ingestion_equivalence.rs` covers a different question: it varies how
|
||||||
|
//! events are *batched* but converges only at the end. This file converges
|
||||||
|
//! between batches, which is the path a caller takes when it fits, serves for a
|
||||||
|
//! while, then ingests more.
|
||||||
|
//!
|
||||||
|
//! The property matters beyond ergonomics. It says `converge` reaches a fixed
|
||||||
|
//! point determined by the events, ratings and configuration alone — not by the
|
||||||
|
//! message state it started from. That is what makes a restored snapshot safe:
|
||||||
|
//! an inexact one cannot corrupt the answer, only cost an extra sweep. See #45.
|
||||||
|
|
||||||
|
use smallvec::smallvec;
|
||||||
|
use trueskill_tt::{ConvergenceOptions, Event, Gaussian, History, Member, Outcome, Team};
|
||||||
|
|
||||||
|
fn tight() -> ConvergenceOptions {
|
||||||
|
ConvergenceOptions {
|
||||||
|
max_iter: 5_000,
|
||||||
|
epsilon: 1e-12,
|
||||||
|
alpha: 1.0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ev(a: &str, b: &str, time: i64) -> Event<i64, String> {
|
||||||
|
Event {
|
||||||
|
time,
|
||||||
|
teams: smallvec![
|
||||||
|
Team::with_members([Member::new(a.to_string())]),
|
||||||
|
Team::with_members([Member::new(b.to_string())]),
|
||||||
|
],
|
||||||
|
outcome: Outcome::winner(0, 2),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ingest each chunk in turn, converging fully after every one.
|
||||||
|
fn fit_in_chunks(chunks: Vec<Events>) -> Vec<(String, Gaussian)> {
|
||||||
|
let mut h: History<i64, _, _, String> =
|
||||||
|
History::builder_with_key().convergence(tight()).build();
|
||||||
|
|
||||||
|
for chunk in chunks {
|
||||||
|
h.add_events(chunk).unwrap();
|
||||||
|
let report = h.converge().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.converged,
|
||||||
|
"a chunk failed to converge, so any comparison would be measuring \
|
||||||
|
truncation rather than the fixed point; final step {:?}",
|
||||||
|
report.final_step
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut skills: Vec<(String, Gaussian)> = h
|
||||||
|
.learning_curves()
|
||||||
|
.into_iter()
|
||||||
|
.map(|(k, curve)| (k, curve.last().unwrap().1))
|
||||||
|
.collect();
|
||||||
|
skills.sort_by(|a, b| a.0.cmp(&b.0));
|
||||||
|
skills
|
||||||
|
}
|
||||||
|
|
||||||
|
fn assert_same(a: &[(String, Gaussian)], b: &[(String, Gaussian)], what: &str) {
|
||||||
|
assert_eq!(a.len(), b.len(), "{what}: competitor count differs");
|
||||||
|
for ((ka, ga), (kb, gb)) in a.iter().zip(b) {
|
||||||
|
assert_eq!(ka, kb, "{what}: key order differs");
|
||||||
|
// Measured: 6.2e-13 for a later append, 8.9e-11 for an interleaved one.
|
||||||
|
// The bar is well clear of both but far under anything that would let a
|
||||||
|
// genuine divergence through.
|
||||||
|
assert!(
|
||||||
|
(ga.mu() - gb.mu()).abs() < 1e-8 && (ga.sigma() - gb.sigma()).abs() < 1e-8,
|
||||||
|
"{what}: {ka} differs — one-shot mu={} sigma={}, chunked mu={} sigma={}",
|
||||||
|
ga.mu(),
|
||||||
|
ga.sigma(),
|
||||||
|
gb.mu(),
|
||||||
|
gb.sigma()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type Events = Vec<Event<i64, String>>;
|
||||||
|
|
||||||
|
/// Two chunks of events: the first at times 0..20, the second at 100..120.
|
||||||
|
fn fixture() -> (Events, Events) {
|
||||||
|
let names = ["a", "b", "c", "d", "e"];
|
||||||
|
let mut seed = 7u64;
|
||||||
|
let mut rnd = move || {
|
||||||
|
seed ^= seed << 13;
|
||||||
|
seed ^= seed >> 7;
|
||||||
|
seed ^= seed << 17;
|
||||||
|
seed
|
||||||
|
};
|
||||||
|
|
||||||
|
let (mut early, mut late) = (Vec::new(), Vec::new());
|
||||||
|
for t in 0..40i64 {
|
||||||
|
let i = (rnd() % 5) as usize;
|
||||||
|
let mut j = (rnd() % 5) as usize;
|
||||||
|
if j == i {
|
||||||
|
j = (j + 1) % 5;
|
||||||
|
}
|
||||||
|
if t < 20 {
|
||||||
|
early.push(ev(names[i], names[j], t));
|
||||||
|
} else {
|
||||||
|
late.push(ev(names[i], names[j], 100 + t));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
(early, late)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The ordinary case: new events are strictly later than everything fitted.
|
||||||
|
#[test]
|
||||||
|
fn appending_later_events_matches_a_single_fit() {
|
||||||
|
let (early, late) = fixture();
|
||||||
|
let all: Vec<_> = early.iter().cloned().chain(late.iter().cloned()).collect();
|
||||||
|
|
||||||
|
assert_same(
|
||||||
|
&fit_in_chunks(vec![all]),
|
||||||
|
&fit_in_chunks(vec![early, late]),
|
||||||
|
"append strictly later",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The case the design question suspected might be weaker: appended events
|
||||||
|
/// interleave with slices that are already fitted, so the append legitimately
|
||||||
|
/// revises the past. It is not weaker — Through Time revises the past on every
|
||||||
|
/// converge regardless, so there is nothing special about doing it in two steps.
|
||||||
|
#[test]
|
||||||
|
fn appending_interleaved_events_matches_a_single_fit() {
|
||||||
|
let (early, late) = fixture();
|
||||||
|
let all: Vec<_> = early.iter().cloned().chain(late.iter().cloned()).collect();
|
||||||
|
|
||||||
|
// Split by parity so the second chunk is back-dated into the first's range.
|
||||||
|
let first: Vec<_> = all.iter().step_by(2).cloned().collect();
|
||||||
|
let second: Vec<_> = all.iter().skip(1).step_by(2).cloned().collect();
|
||||||
|
let together: Vec<_> = first
|
||||||
|
.iter()
|
||||||
|
.cloned()
|
||||||
|
.chain(second.iter().cloned())
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
assert_same(
|
||||||
|
&fit_in_chunks(vec![together]),
|
||||||
|
&fit_in_chunks(vec![first, second]),
|
||||||
|
"append interleaved",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Converging an already-converged history is a no-op, which is what makes a
|
||||||
|
/// restored snapshot worth having: the work is skipped rather than redone.
|
||||||
|
#[test]
|
||||||
|
fn re_converging_an_unchanged_history_costs_one_iteration() {
|
||||||
|
let (early, late) = fixture();
|
||||||
|
let all: Vec<_> = early.into_iter().chain(late).collect();
|
||||||
|
|
||||||
|
let mut h: History<i64, _, _, String> =
|
||||||
|
History::builder_with_key().convergence(tight()).build();
|
||||||
|
h.add_events(all).unwrap();
|
||||||
|
let first = h.converge().unwrap();
|
||||||
|
assert!(first.converged);
|
||||||
|
|
||||||
|
let again = h.converge().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
again.iterations, 1,
|
||||||
|
"a converged history should settle immediately, not re-grind"
|
||||||
|
);
|
||||||
|
assert!(again.converged);
|
||||||
|
}
|
||||||
@@ -15,7 +15,7 @@ fn record_winner_builds_history() {
|
|||||||
.build();
|
.build();
|
||||||
|
|
||||||
h.record_winner(&"alice", &"bob", 1).unwrap();
|
h.record_winner(&"alice", &"bob", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
let a_idx = h.lookup(&"alice").unwrap();
|
let a_idx = h.lookup(&"alice").unwrap();
|
||||||
let b_idx = h.lookup(&"bob").unwrap();
|
let b_idx = h.lookup(&"bob").unwrap();
|
||||||
@@ -48,7 +48,7 @@ fn record_draw_with_p_draw_set() {
|
|||||||
.build();
|
.build();
|
||||||
|
|
||||||
h.record_draw(&"alice", &"bob", 1).unwrap();
|
h.record_draw(&"alice", &"bob", 1).unwrap();
|
||||||
h.converge().unwrap();
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
assert!(h.lookup(&"alice").is_some());
|
assert!(h.lookup(&"alice").is_some());
|
||||||
assert!(h.lookup(&"bob").is_some());
|
assert!(h.lookup(&"bob").is_some());
|
||||||
|
|||||||
@@ -0,0 +1,296 @@
|
|||||||
|
//! The joint must span slices, because Through Time reads each competitor at
|
||||||
|
//! their own last appearance.
|
||||||
|
//!
|
||||||
|
//! The exact posterior of a multi-slice scored history is still Gaussian: the
|
||||||
|
//! prior, the drift between appearances, and the scored likelihoods are all
|
||||||
|
//! Gaussian. So it can be written out by hand and compared against, which is
|
||||||
|
//! the check a single-slice fixture cannot make.
|
||||||
|
|
||||||
|
use smallvec::smallvec;
|
||||||
|
use trueskill_tt::{
|
||||||
|
ConstantDrift, ConvergenceOptions, Event, History, Member, Outcome, Team, UnknownKeys,
|
||||||
|
};
|
||||||
|
|
||||||
|
const SIGMA0: f64 = 6.0;
|
||||||
|
const BETA: f64 = 1.0;
|
||||||
|
const SCORE_SIGMA: f64 = 2.0;
|
||||||
|
const GAMMA: f64 = 0.5;
|
||||||
|
|
||||||
|
type H = History<i64, ConstantDrift, trueskill_tt::NullObserver, &'static str>;
|
||||||
|
|
||||||
|
fn history(gamma: f64) -> H {
|
||||||
|
History::builder()
|
||||||
|
.mu(0.0)
|
||||||
|
.sigma(SIGMA0)
|
||||||
|
.beta(BETA)
|
||||||
|
.score_sigma(SCORE_SIGMA)
|
||||||
|
.drift(ConstantDrift(gamma))
|
||||||
|
.unknown_keys(UnknownKeys::Reject)
|
||||||
|
.convergence(ConvergenceOptions {
|
||||||
|
max_iter: 20_000,
|
||||||
|
epsilon: 1e-13,
|
||||||
|
alpha: 1.0,
|
||||||
|
})
|
||||||
|
.build()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn duel(a: &'static str, b: &'static str, t: i64, sa: f64, sb: f64) -> Event<i64, &'static str> {
|
||||||
|
Event {
|
||||||
|
time: t,
|
||||||
|
teams: smallvec![
|
||||||
|
Team::with_members([Member::new(a)]),
|
||||||
|
Team::with_members([Member::new(b)]),
|
||||||
|
],
|
||||||
|
outcome: Outcome::scores([sa, sb]),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn inverse(mut a: Vec<Vec<f64>>) -> Vec<Vec<f64>> {
|
||||||
|
let n = a.len();
|
||||||
|
let mut inv: Vec<Vec<f64>> = (0..n)
|
||||||
|
.map(|i| (0..n).map(|j| f64::from(u8::from(i == j))).collect())
|
||||||
|
.collect();
|
||||||
|
for col in 0..n {
|
||||||
|
let mut piv = col;
|
||||||
|
for r in col + 1..n {
|
||||||
|
if a[r][col].abs() > a[piv][col].abs() {
|
||||||
|
piv = r;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
a.swap(col, piv);
|
||||||
|
inv.swap(col, piv);
|
||||||
|
let d = a[col][col];
|
||||||
|
for j in 0..n {
|
||||||
|
a[col][j] /= d;
|
||||||
|
inv[col][j] /= d;
|
||||||
|
}
|
||||||
|
for r in 0..n {
|
||||||
|
if r == col {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let f = a[r][col];
|
||||||
|
for j in 0..n {
|
||||||
|
a[r][j] -= f * a[col][j];
|
||||||
|
inv[r][j] -= f * inv[col][j];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inv
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Two competitors, two slices ten units apart, one duel in each.
|
||||||
|
///
|
||||||
|
/// The exact precision is written out explicitly here rather than obtained
|
||||||
|
/// from the crate, so this is an independent check rather than a restatement.
|
||||||
|
/// Variables are `[a0, b0, a1, b1]`.
|
||||||
|
#[test]
|
||||||
|
fn a_two_slice_joint_matches_the_exact_posterior() {
|
||||||
|
let mut h = history(GAMMA);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("a", "b", 10, 4.0, 3.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let report = h.converge().unwrap();
|
||||||
|
assert!(report.converged, "{:?}", report.final_step);
|
||||||
|
|
||||||
|
let prior_prec = 1.0 / (SIGMA0 * SIGMA0);
|
||||||
|
let drift_prec = 1.0 / (10.0 * GAMMA * GAMMA);
|
||||||
|
let obs_prec = 1.0 / (SCORE_SIGMA * SCORE_SIGMA + 2.0 * BETA * BETA);
|
||||||
|
|
||||||
|
let mut lambda = vec![vec![0.0; 4]; 4];
|
||||||
|
// priors on the first appearances
|
||||||
|
lambda[0][0] += prior_prec;
|
||||||
|
lambda[1][1] += prior_prec;
|
||||||
|
// drift a0-a1 and b0-b1
|
||||||
|
for (p, q) in [(0usize, 2usize), (1, 3)] {
|
||||||
|
lambda[p][p] += drift_prec;
|
||||||
|
lambda[q][q] += drift_prec;
|
||||||
|
lambda[p][q] -= drift_prec;
|
||||||
|
lambda[q][p] -= drift_prec;
|
||||||
|
}
|
||||||
|
// one duel per slice: contrast (+1, -1) on that slice's variables
|
||||||
|
for (p, q) in [(0usize, 1usize), (2, 3)] {
|
||||||
|
lambda[p][p] += obs_prec;
|
||||||
|
lambda[q][q] += obs_prec;
|
||||||
|
lambda[p][q] -= obs_prec;
|
||||||
|
lambda[q][p] -= obs_prec;
|
||||||
|
}
|
||||||
|
let cov = inverse(lambda);
|
||||||
|
|
||||||
|
// The crate reads each competitor at their latest appearance: a1, b1.
|
||||||
|
let exact_gap = (cov[2][2] + cov[3][3] - 2.0 * cov[2][3]).sqrt();
|
||||||
|
let got = h.posterior_of(&[(&"a", 1.0), (&"b", -1.0)]).unwrap();
|
||||||
|
assert!(
|
||||||
|
(got.sigma() - exact_gap).abs() / exact_gap < 1e-9,
|
||||||
|
"difference: got {} exact {exact_gap}",
|
||||||
|
got.sigma()
|
||||||
|
);
|
||||||
|
|
||||||
|
let exact_single = cov[2][2].sqrt();
|
||||||
|
let got_single = h.posterior_of(&[(&"a", 1.0)]).unwrap();
|
||||||
|
assert!(
|
||||||
|
(got_single.sigma() - exact_single).abs() / exact_single < 1e-9,
|
||||||
|
"single node: got {} exact {exact_single}",
|
||||||
|
got_single.sigma()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The case that motivated this: competitors read at *different* slices, with
|
||||||
|
/// the last slice holding only one of them. Under the old latest-slice joint
|
||||||
|
/// this was `UnknownKey`.
|
||||||
|
#[test]
|
||||||
|
fn competitors_last_seen_in_different_slices_are_comparable() {
|
||||||
|
let mut h = history(GAMMA);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("a", "c", 10, 4.0, 3.0),
|
||||||
|
// the final slice holds one duel that does not involve b at all
|
||||||
|
duel("a", "c", 20, 6.0, 1.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
|
// b last appeared at time 0; a and c at time 20. All three must resolve.
|
||||||
|
for (x, y) in [("a", "b"), ("b", "c"), ("a", "c")] {
|
||||||
|
let g = h
|
||||||
|
.posterior_of(&[(&x, 1.0), (&y, -1.0)])
|
||||||
|
.unwrap_or_else(|e| panic!("{x} - {y} should resolve across slices: {e}"));
|
||||||
|
assert!(g.sigma() > 0.0 && g.sigma().is_finite());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The mean must agree with what message passing reports, which is exact even
|
||||||
|
/// with cycles. Only the second moment needs the joint.
|
||||||
|
#[test]
|
||||||
|
fn means_agree_with_the_marginals() {
|
||||||
|
let mut h = history(GAMMA);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("b", "c", 5, 3.0, 1.0),
|
||||||
|
duel("a", "c", 10, 4.0, 2.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
|
for k in ["a", "b", "c"] {
|
||||||
|
let marginal = h.current_skill(&k).unwrap().mu();
|
||||||
|
let joint = h.posterior_of(&[(&k, 1.0)]).unwrap().mu();
|
||||||
|
assert!(
|
||||||
|
(marginal - joint).abs() < 1e-9,
|
||||||
|
"{k}: marginal {marginal}, joint {joint}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// With zero drift a competitor has one latent skill however many slices it
|
||||||
|
/// appears in, so spreading the same events over time must not change the
|
||||||
|
/// answer. This exercises the appearance-merging path.
|
||||||
|
#[test]
|
||||||
|
fn zero_drift_makes_slice_layout_irrelevant() {
|
||||||
|
let spread = {
|
||||||
|
let mut h = history(0.0);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("a", "b", 10, 4.0, 3.0),
|
||||||
|
duel("a", "b", 20, 6.0, 1.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
h.posterior_of(&[(&"a", 1.0), (&"b", -1.0)]).unwrap()
|
||||||
|
};
|
||||||
|
let together = {
|
||||||
|
let mut h = history(0.0);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("a", "b", 0, 4.0, 3.0),
|
||||||
|
duel("a", "b", 0, 6.0, 1.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
h.posterior_of(&[(&"a", 1.0), (&"b", -1.0)]).unwrap()
|
||||||
|
};
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
(spread.sigma() - together.sigma()).abs() < 1e-9,
|
||||||
|
"zero drift: spread {} vs together {}",
|
||||||
|
spread.sigma(),
|
||||||
|
together.sigma()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// More drift means less is carried forward from old evidence, so a comparison
|
||||||
|
/// against a competitor last seen long ago must widen.
|
||||||
|
#[test]
|
||||||
|
fn drift_widens_a_comparison_across_time() {
|
||||||
|
let mut previous = 0.0;
|
||||||
|
for gamma in [0.0f64, 0.1, 0.5, 2.0] {
|
||||||
|
let mut h = history(gamma);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("a", "c", 100, 4.0, 3.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
|
// b was last seen at time 0; a at time 100.
|
||||||
|
let g = h.posterior_of(&[(&"a", 1.0), (&"b", -1.0)]).unwrap();
|
||||||
|
assert!(
|
||||||
|
g.sigma() > previous,
|
||||||
|
"gamma={gamma}: sigma {} did not exceed {previous}",
|
||||||
|
g.sigma()
|
||||||
|
);
|
||||||
|
previous = g.sigma();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `posterior_of_at` pins the reading to a moment, where `posterior_of` takes
|
||||||
|
/// each competitor wherever they were last seen.
|
||||||
|
#[test]
|
||||||
|
fn posterior_of_at_reads_as_of_a_time() {
|
||||||
|
let mut h = history(GAMMA);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("a", "b", 10, 4.0, 3.0),
|
||||||
|
duel("a", "b", 20, 6.0, 1.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
|
let early = h.posterior_of_at(0, &[(&"a", 1.0), (&"b", -1.0)]).unwrap();
|
||||||
|
let late = h.posterior_of_at(20, &[(&"a", 1.0), (&"b", -1.0)]).unwrap();
|
||||||
|
let latest = h.posterior_of(&[(&"a", 1.0), (&"b", -1.0)]).unwrap();
|
||||||
|
|
||||||
|
// Asking as of the final slice is the same as asking for the latest.
|
||||||
|
assert!((late.mu() - latest.mu()).abs() < 1e-9);
|
||||||
|
assert!((late.sigma() - latest.sigma()).abs() < 1e-9);
|
||||||
|
|
||||||
|
// Reading at time 0 is a different quantity, and the smoothed estimate
|
||||||
|
// there is informed by everything that came after.
|
||||||
|
assert!(
|
||||||
|
(early.mu() - late.mu()).abs() > 1e-6,
|
||||||
|
"as-of-0 and as-of-20 should differ: {} vs {}",
|
||||||
|
early.mu(),
|
||||||
|
late.mu()
|
||||||
|
);
|
||||||
|
|
||||||
|
// A time before any event has nothing to read.
|
||||||
|
assert!(h.posterior_of_at(-1, &[(&"a", 1.0)]).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Times between slices resolve to the latest appearance at or before them.
|
||||||
|
#[test]
|
||||||
|
fn a_time_between_slices_reads_the_previous_appearance() {
|
||||||
|
let mut h = history(GAMMA);
|
||||||
|
h.add_events(vec![
|
||||||
|
duel("a", "b", 0, 5.0, 2.0),
|
||||||
|
duel("a", "b", 100, 4.0, 3.0),
|
||||||
|
])
|
||||||
|
.unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
|
||||||
|
let at_zero = h.posterior_of_at(0, &[(&"a", 1.0)]).unwrap();
|
||||||
|
let between = h.posterior_of_at(50, &[(&"a", 1.0)]).unwrap();
|
||||||
|
assert!((at_zero.mu() - between.mu()).abs() < 1e-12);
|
||||||
|
assert!((at_zero.sigma() - between.sigma()).abs() < 1e-12);
|
||||||
|
}
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
//! `expected_variance_reduction`: which matchup best sharpens a given question.
|
||||||
|
|
||||||
|
use smallvec::smallvec;
|
||||||
|
use trueskill_tt::{
|
||||||
|
ConstantDrift, ConvergenceOptions, Event, History, InferenceError, Member, Outcome, Team,
|
||||||
|
UnknownKeys,
|
||||||
|
};
|
||||||
|
|
||||||
|
type H = History<i64, ConstantDrift, trueskill_tt::NullObserver, &'static str>;
|
||||||
|
|
||||||
|
fn round(a: &'static str, b: &'static str, sa: f64, sb: f64) -> Event<i64, &'static str> {
|
||||||
|
Event {
|
||||||
|
time: 1,
|
||||||
|
teams: smallvec![
|
||||||
|
Team::with_members([Member::new(a)]),
|
||||||
|
Team::with_members([Member::new(b)]),
|
||||||
|
],
|
||||||
|
outcome: Outcome::scores([sa, sb]),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn base() -> Vec<Event<i64, &'static str>> {
|
||||||
|
vec![
|
||||||
|
round("a", "b", 5.0, 2.0),
|
||||||
|
round("a", "c", 6.0, 1.0),
|
||||||
|
round("b", "c", 4.0, 3.0),
|
||||||
|
round("c", "d", 2.0, 1.0),
|
||||||
|
round("a", "d", 7.0, 2.0),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fit(extra: Option<Event<i64, &'static str>>, policy: UnknownKeys) -> H {
|
||||||
|
let mut h: History<i64, _, _, &'static str> = History::builder()
|
||||||
|
.mu(0.0)
|
||||||
|
.sigma(6.0)
|
||||||
|
.beta(1.0)
|
||||||
|
.score_sigma(2.0)
|
||||||
|
.drift(ConstantDrift(0.0))
|
||||||
|
.unknown_keys(policy)
|
||||||
|
.convergence(ConvergenceOptions {
|
||||||
|
max_iter: 20_000,
|
||||||
|
epsilon: 1e-13,
|
||||||
|
alpha: 1.0,
|
||||||
|
})
|
||||||
|
.build();
|
||||||
|
let mut ev = base();
|
||||||
|
if let Some(e) = extra {
|
||||||
|
ev.push(e);
|
||||||
|
}
|
||||||
|
h.add_events(ev).unwrap();
|
||||||
|
let _ = h.converge().unwrap();
|
||||||
|
h
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The closed form must equal what actually happens if the matchup is played.
|
||||||
|
/// This is the assertion that makes the whole call trustworthy: a wrong
|
||||||
|
/// acquisition function returns plausible numbers and quietly picks worse
|
||||||
|
/// matchups forever.
|
||||||
|
#[test]
|
||||||
|
fn the_closed_form_matches_an_actual_refit() {
|
||||||
|
let h = fit(None, UnknownKeys::Reject);
|
||||||
|
let target: Vec<(&&str, f64)> = vec![(&"a", 1.0), (&"b", -1.0)];
|
||||||
|
let before = h.posterior_of(&target).unwrap().sigma().powi(2);
|
||||||
|
|
||||||
|
for (x, y) in [("a", "b"), ("c", "d"), ("a", "c"), ("b", "d")] {
|
||||||
|
let predicted = h
|
||||||
|
.expected_variance_reduction(&[&[&x], &[&y]], &target)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let after = fit(Some(round(x, y, 3.0, 1.0)), UnknownKeys::Reject);
|
||||||
|
let actual = before - after.posterior_of(&target).unwrap().sigma().powi(2);
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
(predicted - actual).abs() / actual.abs() < 1e-9,
|
||||||
|
"{x} vs {y}: predicted {predicted}, actual {actual}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The reduction cannot depend on the score, because for a Gaussian likelihood
|
||||||
|
/// the posterior variance update is data-independent. This is why the call
|
||||||
|
/// needs no expectation despite its name.
|
||||||
|
#[test]
|
||||||
|
fn the_outcome_does_not_change_the_reduction() {
|
||||||
|
let target: Vec<(&&str, f64)> = vec![(&"a", 1.0), (&"b", -1.0)];
|
||||||
|
let h = fit(None, UnknownKeys::Reject);
|
||||||
|
let before = h.posterior_of(&target).unwrap().sigma().powi(2);
|
||||||
|
|
||||||
|
let mut seen = Vec::new();
|
||||||
|
for (sa, sb) in [(3.0, 1.0), (100.0, -50.0), (0.0, 0.0)] {
|
||||||
|
let after = fit(Some(round("c", "d", sa, sb)), UnknownKeys::Reject);
|
||||||
|
seen.push(before - after.posterior_of(&target).unwrap().sigma().powi(2));
|
||||||
|
}
|
||||||
|
for w in seen.windows(2) {
|
||||||
|
assert!(
|
||||||
|
(w[0] - w[1]).abs() < 1e-12,
|
||||||
|
"variance reduction moved with the observed score: {seen:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The point of the call: it must rank candidate matchups usefully. Playing the
|
||||||
|
/// pair you are trying to separate helps most; an unrelated pair helps least.
|
||||||
|
#[test]
|
||||||
|
fn it_ranks_candidates_by_how_much_they_answer_the_question() {
|
||||||
|
let h = fit(None, UnknownKeys::Reject);
|
||||||
|
let target: Vec<(&&str, f64)> = vec![(&"a", 1.0), (&"b", -1.0)];
|
||||||
|
|
||||||
|
let direct = h
|
||||||
|
.expected_variance_reduction(&[&[&"a"], &[&"b"]], &target)
|
||||||
|
.unwrap();
|
||||||
|
let unrelated = h
|
||||||
|
.expected_variance_reduction(&[&[&"c"], &[&"d"]], &target)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert!(direct > 0.0 && unrelated > 0.0);
|
||||||
|
assert!(
|
||||||
|
direct > 5.0 * unrelated,
|
||||||
|
"playing the target pair should dominate: {direct} vs {unrelated}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A matchup between two competitors nobody has seen still teaches something
|
||||||
|
/// about them, but nothing about a target that does not involve them.
|
||||||
|
#[test]
|
||||||
|
fn an_unrelated_unseen_matchup_teaches_nothing_about_the_target() {
|
||||||
|
let h = fit(None, UnknownKeys::Prior);
|
||||||
|
let target: Vec<(&&str, f64)> = vec![(&"a", 1.0), (&"b", -1.0)];
|
||||||
|
|
||||||
|
let reduction = h
|
||||||
|
.expected_variance_reduction(&[&[&"stranger"], &[&"nobody"]], &target)
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
reduction.abs() < 1e-12,
|
||||||
|
"an unseen pair shares nothing with the target: {reduction}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn shape_errors_are_reported() {
|
||||||
|
let h = fit(None, UnknownKeys::Reject);
|
||||||
|
let target: Vec<(&&str, f64)> = vec![(&"a", 1.0), (&"b", -1.0)];
|
||||||
|
|
||||||
|
assert!(matches!(
|
||||||
|
h.expected_variance_reduction(&[&[&"a"]], &target),
|
||||||
|
Err(InferenceError::MismatchedShape {
|
||||||
|
expected: 2,
|
||||||
|
got: 1,
|
||||||
|
..
|
||||||
|
})
|
||||||
|
));
|
||||||
|
assert!(matches!(
|
||||||
|
h.expected_variance_reduction(&[&[&"a"], &[&"ghost"]], &target),
|
||||||
|
Err(InferenceError::UnknownKey { .. })
|
||||||
|
));
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user