#54 asks whether benchmark regressions can be gated. The threshold is
the whole problem — too tight and CI goes red on noise, which trains the
reflex to re-run until green; too loose and it never fires — and which
of those is possible depends on a number nobody has measured. This adds
a manually-triggered job that runs one unchanged benchmark ten times and
reports min/median/max/mean and the spread.
`joint_factorise_480_appearances` is the probe: ~9 ms, long enough not
to be dominated by timer overhead, and the measurement this crate most
wants protected — it is the dense factorisation #52 is about replacing.
`benches/joint.rs` did not run at all. Its fixture asked for
`epsilon: 1e-10` within `max_iter: 30` and never got there, so once
`converge` stopped returning short fits silently it panicked:
NotConverged { iterations: 30, final_step: (4.5e-4, 0.0), epsilon: 1e-10 }
It now uses the default `ITERATIONS` cap. Measuring a factorisation on
an unconverged fit would have been measuring something nobody runs. The
other four benchmarks were checked and are fine.
Two things in the report step were got wrong first and fixed by running
them, not by reading them:
- `asort` is a gawk extension and the runner's `awk` is mawk. Sorting
goes through `sort -n` instead.
- Criterion picks a unit per run, so a mixed batch would compare 9 ms
against 9 us as though they were the same number. The job refuses to
report a spread unless every run agrees on the unit.
Refs #54.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
logaritmiskandClaude Opus 5
0801acebd1