0b9997354de236d4968342951434b3fafef2bfb1
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  logaritmiskandClaude Opus 5
|
695bb822ef |
perf!: sparse Cholesky with an AMD ordering for the joint
745 ms -> 1.11 ms on the fixture #52 was opened about. The joint precision matrix is 0.19% dense at scale and gets sparser as the history grows. We allocated all n^2 entries — 31 MB at n = 1976, 128 MB at ustat's ~4000 appearances — filled 99.8% of it with zeros, and ran an O(n^3) factorisation over the whole thing. Two measurements shaped the fix, and the first killed the plan #52 proposed. **Ordering alone does nothing to a dense factorisation.** Its inner loops run over every k whether the entry is zero or not, so a permutation changes which entries are zero and not how many multiplications happen. A 700x700 banded matrix at 0.43% density: 30.196 ms in band order, 29.544 ms under a scramble that destroyed the band. Identical, as the flop count says it must be. #52's step 1 — "reorder with AMD, keep our own Cholesky, and measure" — could not have worked, and measuring said so before any of it was written. **Sparsity and AMD together are worth four orders of magnitude.** Symbolic factorisation on the n = 1976 fixture, against 2.572e9 dense flops: sparse in natural order needs 5.597e7 (46x), sparse under AMD needs 8.656e4 — 29,710x. AMD is worth 646x on top of sparsity and nothing without it. Natural order fills in badly for exactly the reason #52 predicted about bandwidth: nnz(L) is 292,437 against A's 7,504, because a competitor idle from slice 0 to slice 75 links across the whole matrix. Measured end to end, factorising through `History::joint`: n = 480 215 us (bench: 9.11 ms -> 167 us, 54x) n = 1976 1.112 ms (was ~745 ms, 670x) n = 7800 4.616 ms (dense would be 1.58e11 flops) Scaling is near-linear now rather than cubic: 16x the variables costs 21x the time, where dense would cost 4096x. The factorisation is the up-looking sparse Cholesky of Davis's *Direct Methods for Sparse Linear Systems*, written here rather than taken from a crate. The scouting in #52 still holds and got one addition: `feral` itself pulls `pulp`, so it has the same runtime CPU-dispatch problem that ruled out `faer` — results could differ between an AVX-512 host and an AVX2 one, the drift the libm-over-std decision was made to avoid. `sprs-ldl` is still LGPL and `nalgebra-sparse` still disclaims fill-reduction in its own docs. Only the ordering is a dependency: `feral-amd`, two crates, both `#![forbid(unsafe_code)]`. The matrix is accumulated into a `BTreeMap`, not a hash map: the iteration order becomes the summation order, and a hash map's varies per process. `tests/cross_process_determinism.rs` exists because that has bitten before. `whiten` returns its result in the permuted order and leaves it there — a dot product does not care, as long as both operands were permuted the same way — so `bilinear` is unchanged. Correctness: the existing analytic goldens are 2x2 and 3x3, too small to permute or fill in, so they could not have caught a symbolic-pass bug. `agrees_with_a_dense_reference_on_random_sparse_systems` checks every bilinear form against a deliberately naive dense factorisation that shares no code with the thing it is checking, on chain-plus-long-range matrices up to n = 60. Closes #52. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ |
||
|
logaritmiskandClaude Opus 5
|
1bb6bb31d8 |
feat: factorise the joint once with History::joint
`posterior_of`, `posterior_of_at` and `expected_variance_reduction` each
built the joint precision matrix, factorised it, asked one question and
threw it away. The factorisation is O(n^3) in the history's appearances
and depends only on the fit, so a caller asking about every pair in a
standings table, every cell in a grid, or every candidate in an
active-learning sweep paid for the same factorisation once per question.
`History::joint()` returns a `Joint` handle that pays it once. Measured
on 1976 appearances, 90 queries: 68.4s one-shot against 745ms factorise
plus 93ms of queries — 81.6x, with bit-identical answers. Per query,
Criterion at 480 appearances: 9.0ms one-shot against 48us cached, 187x.
The handle borrows the history, which is what makes it correct with no
invalidation logic: the borrow checker forbids adding events or refitting
while it is alive, so there is no window in which the factorisation could
describe a fit that no longer exists. It also makes the lifetime of the
n^2 factor explicit rather than parking it in the history forever — at
4000 appearances that is 128MB, which is not something to cache silently.
Every question the joint answers turns out to be a bilinear form,
c^T A^-1 a = (L^-1 c) . (L^-1 a)
so no caller ever needs L^-1 c itself. Replacing the general solve with a
forward substitution drops the back substitution as wasted work, halving
a query, and removes a failure mode: a variance as `c . (A^-1 c)` is a
difference of products that can round negative, where `|L^-1 c|^2` is a
sum of squares and cannot.
The one-shot calls are unchanged in cost and now delegate to the handle,
so the two paths cannot drift apart. tests/joint_handle.rs asserts they
agree bit for bit, including at pinned times, under UnknownKeys::Prior,
and across candidate matchups.
Refs #51
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ
|
||
|
logaritmiskandClaude Opus 5
|
c52e2550af |
feat: add History::posterior_of for a linear combination of competitors
#46: every accessor returns a per-competitor marginal, and almost nothing a consumer publishes is one competitor. Combining marginals assumes independence, and competitors are correlated through every event they share. `posterior_of(&[(a, 1.0), (b, -1.0)])` returns the posterior of that combination with the correlation intact. Validated against the exact linear-Gaussian posterior on both a tree and a loopy fixture, for differences and for single competitors: agreement to 1e-9 relative in every case. The investigation that preceded this is why it is not a covariance accessor. Marginals from loopy message passing are about half the true width, and ignoring correlation overstates a difference — the two errors partially cancel, leaving 1.327x rather than 2.646x. Bolting true correlations onto the existing marginals would have given 0.765 against a true 1.524, which is overconfident: the direction the reporter specifically called unsafe. Rebuilding the joint from the factor structure fixes both at once, and a single-competitor query now returns the exact marginal rather than the narrow one. The precision matrix depends only on structure — who played whom, with what weights and what noise — not on the observed outcomes, and the means were already exact. So only the second moment is reconstructed. Known limits, all deliberate and documented on the method: - Latest slice only. A functional spanning times, such as "current versus career", needs the time-expanded joint and is not covered. - Scored events only. A ranked outcome's truncation is EP-approximated and its converged factors are not retained after inference, so ranked slices return `JointUnavailable` rather than a plausible wrong number. - Dense Cholesky, O(n^3) per query in the slice's competitor count: 38.8us at 50, 5.66ms at 400, 49.1ms at 800. Fine for the sizes this serves today; caching the factorization per slice would make repeat queries O(n^2), and sparsity is the next step after that. Refs #46, #47, #48 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hcFjNDmHXZF8URGLku5zZ |