logaritmiskandClaude Opus 5
0801acebd1
#54 asks whether benchmark regressions can be gated. The threshold is the whole problem — too tight and CI goes red on noise, which trains the reflex to re-run until green; too loose and it never fires — and which of those is possible depends on a number nobody has measured. This adds a manually-triggered job that runs one unchanged benchmark ten times and reports min/median/max/mean and the spread. `joint_factorise_480_appearances` is the probe: ~9 ms, long enough not to be dominated by timer overhead, and the measurement this crate most wants protected — it is the dense factorisation #52 is about replacing. `benches/joint.rs` did not run at all. Its fixture asked for `epsilon: 1e-10` within `max_iter: 30` and never got there, so once `converge` stopped returning short fits silently it panicked: NotConverged { iterations: 30, final_step: (4.5e-4, 0.0), epsilon: 1e-10 } It now uses the default `ITERATIONS` cap. Measuring a factorisation on an unconverged fit would have been measuring something nobody runs. The other four benchmarks were checked and are fine. Two things in the report step were got wrong first and fixed by running them, not by reading them: - `asort` is a gawk extension and the runner's `awk` is mawk. Sorting goes through `sort -n` instead. - Criterion picks a unit per run, so a mixed batch would compare 9 ms against 9 us as though they were the same number. The job refuses to report a spread unless every run agrees on the unit. Refs #54. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
82 lines
3.0 KiB
Rust
82 lines
3.0 KiB
Rust
//! Cost of the joint posterior: factorising versus querying.
|
|
//!
|
|
//! The split is the whole point of `History::joint`. Factorising is `O(n^3)` in
|
|
//! the history's appearances and depends only on the fit; a query is `O(n^2)`
|
|
//! and depends only on the question. `posterior_of_one_shot` pays both every
|
|
//! time, `joint_query` pays only the second.
|
|
|
|
use criterion::{Criterion, criterion_group, criterion_main};
|
|
use smallvec::smallvec;
|
|
use trueskill_tt::{ConstantDrift, ConvergenceOptions, Event, History, Member, Outcome, Team};
|
|
|
|
/// 30 slices of 8 duels: 480 appearances over 100 competitors.
|
|
fn fitted() -> History<String> {
|
|
let mut h: History<String> = History::builder()
|
|
.key_type::<String>()
|
|
.mu(0.0)
|
|
.sigma(6.0)
|
|
.beta(1.0)
|
|
.score_sigma(2.0)
|
|
.drift(ConstantDrift::new(0.05))
|
|
// `max_iter: 30` was here, and this fixture needs more: `converge`
|
|
// reported `NotConverged { iterations: 30, final_step: (4.5e-4, 0.0) }`
|
|
// once it stopped returning short fits silently. The benchmark measures
|
|
// the factorisation, whose cost depends on the fit's *shape* rather
|
|
// than its exactness — but measuring it on an unconverged fit is still
|
|
// measuring something nobody would run.
|
|
.convergence(ConvergenceOptions {
|
|
max_iter: trueskill_tt::ITERATIONS,
|
|
epsilon: 1e-10,
|
|
alpha: 1.0,
|
|
})
|
|
.build();
|
|
|
|
let mut events: Vec<Event<i64, String>> = Vec::new();
|
|
let mut k = 0usize;
|
|
for t in 0..30i64 {
|
|
for _ in 0..8 {
|
|
k += 1;
|
|
events.push(Event {
|
|
time: t,
|
|
teams: smallvec![
|
|
Team::with_members([Member::new(format!("p{}", k % 100))]),
|
|
Team::with_members([Member::new(format!("p{}", (k + 37) % 100))]),
|
|
],
|
|
outcome: Outcome::scores([
|
|
(k as f64 * 0.3).sin().abs() * 20.0,
|
|
(k as f64 * 0.3).cos().abs() * 20.0,
|
|
]),
|
|
});
|
|
}
|
|
}
|
|
h.add_events(events).unwrap();
|
|
let _ = h.converge().unwrap();
|
|
h
|
|
}
|
|
|
|
fn bench_joint(c: &mut Criterion) {
|
|
let h = fitted();
|
|
let a = "p0".to_string();
|
|
let b = "p1".to_string();
|
|
let terms = [(&a, 1.0), (&b, -1.0)];
|
|
|
|
c.bench_function("joint_factorise_480_appearances", |bencher| {
|
|
bencher.iter(|| std::hint::black_box(h.joint().unwrap().variables()));
|
|
});
|
|
|
|
// Factorise-and-query, the cost the deleted `History::posterior_of`
|
|
// wrapper paid on every call. Kept as the baseline the cached query below
|
|
// is measured against.
|
|
c.bench_function("posterior_of_one_shot_480_appearances", |bencher| {
|
|
bencher.iter(|| std::hint::black_box(h.joint().unwrap().posterior_of(&terms).unwrap()));
|
|
});
|
|
|
|
let joint = h.joint().unwrap();
|
|
c.bench_function("joint_query_480_appearances", |bencher| {
|
|
bencher.iter(|| std::hint::black_box(joint.posterior_of(&terms).unwrap()));
|
|
});
|
|
}
|
|
|
|
criterion_group!(benches, bench_joint);
|
|
criterion_main!(benches);
|