test: make the determinism test exercise the parallel sweep

It proved less than it appeared to. `sweep_color_groups` takes its
`par_iter` branch only for colour groups of at least RAYON_THRESHOLD (64)
events, and a colour group is a subset of ONE slice's events. The fixture
built 20 slices of 10, so the branch was unreachable — the test named the
parallel path and ran the sequential one.

It also compared one competitor's curve out of forty, and never compared
log_evidence, final_step or iterations.

The new fixture reaches the branch by construction: within a slice every
event uses a disjoint competitor pair, so greedy colouring puts all 96 in
colour 0. Competitors recur across slices, so the fit keeps temporal
coupling and drift rather than degenerating into independent duels.

Verified by instrumenting `sweep_color_groups`: 872 sweeps, one colour
group of 96 each, parallel branch taken all 872 times.

Worth recording how that verification went, because I nearly drew the
opposite conclusion. My first two instrumented runs printed nothing and I
read that as "the branch is still unreachable" — but `cargo test` captures
stderr without `--nocapture`, so the probe was invisible, not absent. An
instrument that cannot report is indistinguishable from a negative result.

Now compares every competitor's curve plus log_evidence, final_step and
iterations, and asserts the curve count so it cannot silently go back to
measuring almost nothing. A companion test pins EVENTS_PER_SLICE against
the threshold, so shrinking the fixture fails loudly rather than quietly
returning the suite to the sequential path.

Cross-process coverage is separate, in tests/cross_process_determinism.rs
(#62) — an in-process test cannot see hasher-order effects at all, since
every sample shares one seed.

Closes #64

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
2026-09-09 18:08:06 +02:00
co-authored by Claude Opus 5
parent 7aa7fb62dd
commit c69a397d80
+150 -50
View File
@@ -1,101 +1,201 @@
//! Determinism tests: identical posteriors across RAYON_NUM_THREADS //! Determinism across `RAYON_NUM_THREADS`, on a workload that actually reaches
//! values. Only compiled with the `rayon` feature. //! the parallel path.
//!
//! This test previously proved less than it appeared to. `sweep_color_groups`
//! takes its `par_iter` branch only for colour groups of at least
//! `RAYON_THRESHOLD` (64) events, and the old fixture built 20 slices of 10
//! events — a colour group is a subset of one slice's events, so it could never
//! exceed 10. The branch was unreachable, confirmed by CPU-vs-wall time:
//! `user 0.64` on eight threads is one core.
//!
//! It also compared a single competitor's curve out of forty, and never
//! compared `log_evidence`, `final_step` or `iterations`.
//!
//! The fixture below guarantees the parallel branch **by construction**: within
//! a slice every event uses a disjoint pair of competitors, so greedy colouring
//! puts all of them in colour 0, and that group is `EVENTS_PER_SLICE` long.
//! Competitors recur across slices, so the fit still has temporal coupling and
//! drift rather than being a set of independent duels.
#![cfg(feature = "rayon")] #![cfg(feature = "rayon")]
use smallvec::smallvec; use smallvec::smallvec;
use trueskill_tt::{ConstantDrift, ConvergenceOptions, Event, History, Member, Outcome, Team}; use trueskill_tt::{
ConstantDrift, ConvergenceOptions, Event, Gaussian, History, Member, Outcome, Team,
};
/// Build a deterministic workload using a simple LCG (no external rand crate). /// Comfortably above the crate's internal `RAYON_THRESHOLD` of 64.
fn build_and_converge(seed: u64) -> Vec<(i64, trueskill_tt::Gaussian)> { const EVENTS_PER_SLICE: usize = 96;
const SLICES: i64 = 8;
/// Two per event, all disjoint within a slice.
const COMPETITORS: usize = EVENTS_PER_SLICE * 2;
/// Everything a thread count could plausibly perturb.
struct Fingerprint {
curves: Vec<(String, Vec<(i64, Gaussian)>)>,
log_evidence: f64,
final_step: (f64, f64),
iterations: usize,
}
fn build_and_converge() -> Fingerprint {
let mut h = History::<i64, _, _, String>::builder_with_key() let mut h = History::<i64, _, _, String>::builder_with_key()
.mu(25.0) .mu(25.0)
.sigma(25.0 / 3.0) .sigma(25.0 / 3.0)
.beta(25.0 / 6.0) .beta(25.0 / 6.0)
.drift(ConstantDrift(25.0 / 300.0)) .drift(ConstantDrift(25.0 / 300.0))
.convergence(ConvergenceOptions { .convergence(ConvergenceOptions {
max_iter: 30, max_iter: 20_000,
epsilon: 1e-6, epsilon: 1e-9,
alpha: 1.0, alpha: 1.0,
}) })
.build(); .build();
// LCG for deterministic pseudo-random ints. let mut events: Vec<Event<i64, String>> = Vec::new();
let mut rng = seed; for slice in 0..SLICES {
let mut next = || { for e in 0..EVENTS_PER_SLICE {
rng = rng // Disjoint within the slice: event `e` owns competitors 2e and
.wrapping_mul(6364136223846793005) // 2e+1. Rotating by the slice index makes the pairings differ
.wrapping_add(1442695040888963407); // between slices, so competitors accumulate a real history.
rng let a = (2 * e + slice as usize) % COMPETITORS;
}; let b = (2 * e + 1 + slice as usize * 3) % COMPETITORS;
if a == b {
let mut events: Vec<Event<i64, String>> = Vec::with_capacity(200); continue;
for ev_i in 0..200 {
let a = (next() % 40) as usize;
let mut b = (next() % 40) as usize;
while b == a {
b = (next() % 40) as usize;
} }
// ~10 events per slice so color groups have material parallelism.
events.push(Event { events.push(Event {
time: (ev_i as i64 / 10) + 1, time: slice + 1,
teams: smallvec![ teams: smallvec![
Team::with_members([Member::new(format!("p{a}"))]), Team::with_members([Member::new(format!("p{a}"))]),
Team::with_members([Member::new(format!("p{b}"))]), Team::with_members([Member::new(format!("p{b}"))]),
], ],
outcome: Outcome::winner((next() % 2) as u32, 2), outcome: Outcome::winner(u32::from((e + slice as usize) % 2 == 0), 2),
}); });
} }
}
h.add_events(events).unwrap(); h.add_events(events).unwrap();
let _ = h.converge().unwrap();
// Sample one competitor's curve for the comparison. let report = h.converge().expect("fixture must converge");
h.learning_curve("p0")
let mut curves: Vec<(String, Vec<(i64, Gaussian)>)> = h
.learning_curves()
.into_iter()
.map(|(k, v)| (k.clone(), v))
.collect();
curves.sort_by(|a, b| a.0.cmp(&b.0));
Fingerprint {
curves,
log_evidence: h.log_evidence(),
final_step: report.final_step,
iterations: report.iterations,
}
} }
#[test] #[test]
fn posteriors_identical_across_thread_counts() { fn posteriors_identical_across_thread_counts() {
let sizes = [1usize, 2, 4, 8]; let sizes = [1usize, 2, 4, 8];
let mut results: Vec<Vec<(i64, trueskill_tt::Gaussian)>> = Vec::new(); let mut results: Vec<Fingerprint> = Vec::new();
for &n in &sizes { for &n in &sizes {
let pool = rayon::ThreadPoolBuilder::new() let pool = rayon::ThreadPoolBuilder::new()
.num_threads(n) .num_threads(n)
.build() .build()
.expect("rayon pool build"); .expect("rayon pool build");
let curve = pool.install(|| build_and_converge(42)); results.push(pool.install(build_and_converge));
results.push(curve);
} }
let reference = &results[0]; let reference = &results[0];
for (i, curve) in results.iter().enumerate().skip(1) {
// Guard against the failure this test previously had: passing while
// measuring almost nothing.
assert!(
reference.curves.len() > 100,
"expected every competitor's curve, got {}",
reference.curves.len()
);
for (i, got) in results.iter().enumerate().skip(1) {
let n = sizes[i];
assert_eq!(
got.iterations, reference.iterations,
"iterations differ at {n} threads"
);
assert_eq!(
got.final_step.0.to_bits(),
reference.final_step.0.to_bits(),
"final_step.0 differs at {n} threads: {:?} vs {:?}",
reference.final_step,
got.final_step
);
assert_eq!(
got.final_step.1.to_bits(),
reference.final_step.1.to_bits(),
"final_step.1 differs at {n} threads"
);
assert_eq!(
got.log_evidence.to_bits(),
reference.log_evidence.to_bits(),
"log_evidence differs at {n} threads: {} vs {}",
reference.log_evidence,
got.log_evidence
);
assert_eq!(
got.curves.len(),
reference.curves.len(),
"competitor count differs at {n} threads"
);
for ((ref_key, ref_curve), (key, curve)) in reference.curves.iter().zip(got.curves.iter()) {
assert_eq!(ref_key, key, "competitor order differs at {n} threads");
assert_eq!( assert_eq!(
curve.len(), curve.len(),
reference.len(), ref_curve.len(),
"curve length differs at {n} threads", "curve length differs for {key} at {n} threads"
n = sizes[i],
);
for (j, (&(t_ref, g_ref), &(t, g))) in reference.iter().zip(curve.iter()).enumerate() {
assert_eq!(
t_ref,
t,
"time point {j} differs at {n} threads: ref={t_ref} vs got={t}",
n = sizes[i],
); );
for (&(t_ref, g_ref), &(t, g)) in ref_curve.iter().zip(curve.iter()) {
assert_eq!(t_ref, t, "time point differs for {key} at {n} threads");
assert_eq!( assert_eq!(
g_ref.mu().to_bits(), g_ref.mu().to_bits(),
g.mu().to_bits(), g.mu().to_bits(),
"mu bits differ at {n} threads, time {t}: ref={ref_mu} got={got_mu}", "mu differs for {key} at t={t}, {n} threads: {} vs {}",
n = sizes[i], g_ref.mu(),
ref_mu = g_ref.mu(), g.mu()
got_mu = g.mu(),
); );
assert_eq!( assert_eq!(
g_ref.sigma().to_bits(), g_ref.sigma().to_bits(),
g.sigma().to_bits(), g.sigma().to_bits(),
"sigma bits differ at {n} threads, time {t}: ref={ref_sigma} got={got_sigma}", "sigma differs for {key} at t={t}, {n} threads: {} vs {}",
n = sizes[i], g_ref.sigma(),
ref_sigma = g_ref.sigma(), g.sigma()
got_sigma = g.sigma(),
); );
} }
} }
} }
}
/// The fixture must keep reaching the parallel branch.
///
/// `RAYON_THRESHOLD` is private, so this pins the property that makes the
/// branch reachable rather than the branch itself: within a slice every event
/// uses a disjoint competitor pair, so greedy colouring puts all
/// `EVENTS_PER_SLICE` of them in one colour group. If someone shrinks the
/// fixture, this fails rather than the suite quietly going back to testing the
/// sequential path.
#[test]
fn the_fixture_still_exceeds_the_rayon_threshold() {
const RAYON_THRESHOLD: usize = 64;
const {
assert!(
EVENTS_PER_SLICE >= RAYON_THRESHOLD,
"a colour group holds at most EVENTS_PER_SLICE events, which must \
reach the crate's RAYON_THRESHOLD for the parallel sweep to run"
);
}
// Measured by instrumenting `sweep_color_groups`: this fixture produces
// one colour group of 96 events and takes the parallel branch on all 872
// sweeps. The old fixture's 10-event slices could not reach 64 at all.
assert_eq!(EVENTS_PER_SLICE, 96);
}