Three issues from two downstream consumers, all small, all sharing a theme: the crate had the information and would not hand it over. #44 — `UnknownKey { team: 0, member: 0 }` did not say which key. A consumer upgrading 0.1.2 -> 0.4.1 had every one of 5591 predictions return this error, fell back to a neutral 0.5, and lost its entire metadata model for a day. Nothing crashed and nothing logged; it was found by sweeping an unrelated parameter and noticing the output did not move. The 0.4.0 change that made unknown keys an error was right — the error was just too anonymous to act on. It now carries the key's `Debug` rendering, and its `Display` says what to do about it. The precondition is documented on every prediction entry point, which the reporter said would alone have saved the day. #43 — `cdf` was `pub(crate)`, so a consumer asking "is this competitor below the cutoff" approximated it with a `mu + z * sigma` band and had no way to say what confidence any `z` bought. Adds `Gaussian::probability_below` / `probability_above`. The second is separate on purpose: `1 - cdf` collapses to exactly zero past ~8.3 sigma, and a stopping rule is evaluated precisely there. Both route through the survival function added in 0.4.1, so this is visibility rather than new numerics. #50 — `ConvergenceReport` was not `#[must_use]`, so the one signal that a fit stopped short was trivially discarded. It now is, and that immediately found 78 sites doing exactly that — including this crate's own ATP example, which was capped at 10 sweeps when the history needs 30. The example now reads the report and says so. `ITERATIONS = 30` is documented as the floor it is, with the three measurements to hand: 400 events over 100 competitors already stops there at ~7e-3 against a 1e-6 tolerance, the ATP example needs 30 at a much looser one, and a consumer's 2000-node model needs 76 to 161. BREAKING CHANGE: `InferenceError::UnknownKey` gains a `key` field, and the prediction methods now require `K: Debug` in order to fill it. Closes #43, #50. Refs #44 — its third ask, an opt-in `UnknownKeys::Skip` mode, is a live API question and deliberately not answered here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
118 lines
4.4 KiB
Rust
118 lines
4.4 KiB
Rust
//! End-to-end History::converge benchmark.
|
||
//!
|
||
//! Workload shapes designed to expose rayon's within-slice color-group
|
||
//! parallelism. Events in the same color group are processed in parallel
|
||
//! via direct-write with disjoint index sets (no data races). Color groups
|
||
//! smaller than a threshold fall back to the sequential path to avoid
|
||
//! rayon overhead on small workloads.
|
||
//!
|
||
//! On Apple M5 Pro, the P-core count (6) is the optimal thread count.
|
||
//! The rayon thread pool is initialised to `min(P-cores, available)` to
|
||
//! avoid scheduling onto the slower E-cores.
|
||
//!
|
||
//! ## Results (Apple M5 Pro, 2026-04-24, after SmallVec revert)
|
||
//!
|
||
//! | Workload | Sequential | Parallel | Speedup |
|
||
//! |---------------------------------------------|------------:|-----------:|--------:|
|
||
//! | History::converge/500x100@10perslice | 4.03 ms | 4.24 ms | 1.0× |
|
||
//! | History::converge/2000x200@20perslice | 20.18 ms | 19.82 ms | 1.0× |
|
||
//! | History::converge/1v1-5000x50000@5000perslice| 11.88 ms | 9.10 ms | 1.3× |
|
||
//!
|
||
//! T3 acceptance gate: ≥2× speedup on at least one workload — NOT achieved after revert.
|
||
//! The SmallVec storage that enabled the 2× gate caused a +28% regression in the
|
||
//! sequential Batch::iteration benchmark and was reverted. Small workloads still fall
|
||
//! below the RAYON_THRESHOLD (64 events/color) and run sequentially with near-zero overhead.
|
||
|
||
use criterion::{BatchSize, Criterion, criterion_group, criterion_main};
|
||
use smallvec::smallvec;
|
||
use trueskill_tt::{
|
||
ConstantDrift, ConvergenceOptions, Event, History, Member, NullObserver, Outcome, Team,
|
||
};
|
||
|
||
fn build_history_1v1(
|
||
n_events: usize,
|
||
n_competitors: usize,
|
||
events_per_slice: usize,
|
||
seed: u64,
|
||
) -> History<i64, ConstantDrift, NullObserver, String> {
|
||
let mut rng = seed;
|
||
let mut next = || {
|
||
rng = rng
|
||
.wrapping_mul(6364136223846793005)
|
||
.wrapping_add(1442695040888963407);
|
||
rng
|
||
};
|
||
|
||
let mut h = History::<i64, _, _, String>::builder_with_key()
|
||
.mu(25.0)
|
||
.sigma(25.0 / 3.0)
|
||
.beta(25.0 / 6.0)
|
||
.drift(ConstantDrift(25.0 / 300.0))
|
||
.convergence(ConvergenceOptions {
|
||
max_iter: 30,
|
||
epsilon: 1e-6,
|
||
alpha: 1.0,
|
||
})
|
||
.build();
|
||
|
||
let mut events: Vec<Event<i64, String>> = Vec::with_capacity(n_events);
|
||
for ev_i in 0..n_events {
|
||
let a = (next() as usize) % n_competitors;
|
||
let mut b = (next() as usize) % n_competitors;
|
||
while b == a {
|
||
b = (next() as usize) % n_competitors;
|
||
}
|
||
events.push(Event {
|
||
time: (ev_i as i64 / events_per_slice as i64) + 1,
|
||
teams: smallvec![
|
||
Team::with_members([Member::new(format!("p{a}"))]),
|
||
Team::with_members([Member::new(format!("p{b}"))]),
|
||
],
|
||
outcome: Outcome::winner((next() % 2) as u32, 2),
|
||
});
|
||
}
|
||
h.add_events(events).unwrap();
|
||
h
|
||
}
|
||
|
||
fn bench_converge(c: &mut Criterion) {
|
||
// Two original task workloads (small per-slice event count;
|
||
// fall below RAYON_THRESHOLD so sequential path runs — near-zero overhead).
|
||
c.bench_function("History::converge/500x100@10perslice", |b| {
|
||
b.iter_batched(
|
||
|| build_history_1v1(500, 100, 10, 42),
|
||
|mut h| {
|
||
let _ = h.converge().unwrap();
|
||
},
|
||
BatchSize::SmallInput,
|
||
);
|
||
});
|
||
|
||
c.bench_function("History::converge/2000x200@20perslice", |b| {
|
||
b.iter_batched(
|
||
|| build_history_1v1(2000, 200, 20, 42),
|
||
|mut h| {
|
||
let _ = h.converge().unwrap();
|
||
},
|
||
BatchSize::SmallInput,
|
||
);
|
||
});
|
||
|
||
// Large single-slice workload: 5000 events, 50000 competitors.
|
||
// All events in one slice → color-0 gets ~4900 disjoint events, well above
|
||
// the 64-event RAYON_THRESHOLD. 30 iterations × 1 slice = 30 sweeps, each
|
||
// parallelised across P-core threads. Shows ≥2× speedup.
|
||
c.bench_function("History::converge/1v1-5000x50000@5000perslice", |b| {
|
||
b.iter_batched(
|
||
|| build_history_1v1(5000, 50000, 5000, 42),
|
||
|mut h| {
|
||
let _ = h.converge().unwrap();
|
||
},
|
||
BatchSize::SmallInput,
|
||
);
|
||
});
|
||
}
|
||
|
||
criterion_group!(benches, bench_converge);
|
||
criterion_main!(benches);
|