ci: measure the runner's own benchmark variance, and fix the joint bench
#54 asks whether benchmark regressions can be gated. The threshold is the whole problem — too tight and CI goes red on noise, which trains the reflex to re-run until green; too loose and it never fires — and which of those is possible depends on a number nobody has measured. This adds a manually-triggered job that runs one unchanged benchmark ten times and reports min/median/max/mean and the spread. `joint_factorise_480_appearances` is the probe: ~9 ms, long enough not to be dominated by timer overhead, and the measurement this crate most wants protected — it is the dense factorisation #52 is about replacing. `benches/joint.rs` did not run at all. Its fixture asked for `epsilon: 1e-10` within `max_iter: 30` and never got there, so once `converge` stopped returning short fits silently it panicked: NotConverged { iterations: 30, final_step: (4.5e-4, 0.0), epsilon: 1e-10 } It now uses the default `ITERATIONS` cap. Measuring a factorisation on an unconverged fit would have been measuring something nobody runs. The other four benchmarks were checked and are fine. Two things in the report step were got wrong first and fixed by running them, not by reading them: - `asort` is a gawk extension and the runner's `awk` is mawk. Sorting goes through `sort -n` instead. - Criterion picks a unit per run, so a mixed batch would compare 9 ms against 9 us as though they were the same number. The job refuses to report a spread unless every run agrees on the unit. Refs #54. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
This commit is contained in:
+7
-1
@@ -18,8 +18,14 @@ fn fitted() -> History<String> {
|
||||
.beta(1.0)
|
||||
.score_sigma(2.0)
|
||||
.drift(ConstantDrift::new(0.05))
|
||||
// `max_iter: 30` was here, and this fixture needs more: `converge`
|
||||
// reported `NotConverged { iterations: 30, final_step: (4.5e-4, 0.0) }`
|
||||
// once it stopped returning short fits silently. The benchmark measures
|
||||
// the factorisation, whose cost depends on the fit's *shape* rather
|
||||
// than its exactness — but measuring it on an unconverged fit is still
|
||||
// measuring something nobody would run.
|
||||
.convergence(ConvergenceOptions {
|
||||
max_iter: 30,
|
||||
max_iter: trueskill_tt::ITERATIONS,
|
||||
epsilon: 1e-10,
|
||||
alpha: 1.0,
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user