logaritmisk/trueskill-tt
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Activity

A node's reported sigma is ~10x wider than the estimate's measured reproducibility #47

New Issue
Closed
opened 2026-09-07 21:26:32 +00:00 by logaritmisk · 5 comments
logaritmisk commented 2026-09-07 21:26:32 +00:00
Owner
Copy Link
Copy Source

Separate from the missing joint posterior (#46): a single node's own marginal sigma appears to be far wider than the spread you actually get if you re-estimate it from independent data.

The measurement

ustat fits a joint hole-level model (nodes: Player, Hole, HoleOffset, DayForm; ~2,000 nodes over ~3,300 events). It wanted to publish "this hole is unusually bad for you", which reads the HoleOffset node alone. On the served marginals that claim is unpublishable:

  • |offset| tops out at 0.46 strokes
  • posterior sigma on those same cells runs 0.83–1.16
  • so not one of 90 player×hole cells clears even one sigma

Rather than accept that, the model was fitted twice on disjoint halves of the rounds and the two independent estimates of the same cell compared:

rms disagreement between the two half-fits   0.157
what the marginals say it should be          1.670

About tenfold. The estimator is far more reproducible than the model admits.

Split-half only bounds this from below — it sees how far the estimate moves between samples, not shrinkage both halves share — so it was calibrated against a synthetic population with offsets of known size: split-half recovers 0.65–0.80 of the true error, giving a correction of ~1.37. Corrected, the honest sigma is 0.0995, and 12 of 90 cells clear two sigma where none ever cleared one.

Why I am filing it as a question rather than a bug

I do not know whether this is a defect or expected EP behaviour. Two candidate explanations, and I cannot separate them from outside the crate:

  1. HoleOffset carries a deliberately tight prior (σ 1.5 against the player prior's 6.0). If the posterior sigma stays near the prior while the mean is well determined by data, the marginal is "right" in a Bayesian sense and simply not the quantity a consumer wants.
  2. EP marginals in a densely coupled graph are genuinely conservative here.

If it is (1), then the honest fix may be documentation — say plainly what the reported sigma is and is not, and that it should not be read as the standard error of the estimate. If it is (2), it is worth understanding.

Either way, a consumer currently has to run two extra full fits at boot (~88 ms here, ~330 ms of total boot) plus a synthetic-data calibration constant to recover a number the model should be able to report. That constant is the shakiest figure in ustat: it was measured at the hole grain and is reused unmeasured at the layout grain, where it decides whether a real row is shown or hidden.

What would help, in order of cost

  1. Documentation of what the marginal means and how it should not be used.
  2. An effective-sample-size or evidence-count per node, so a consumer can tell a well-determined node from one still sitting on its prior.
  3. A calibrated standard error for a node's posterior mean, if that is a thing EP can be asked for.

Context: logaritmisk/ustat#114 (the UI claim this blocks) and logaritmisk/ustat#106.

Separate from the missing joint posterior (#46): a **single** node's own marginal sigma appears to be far wider than the spread you actually get if you re-estimate it from independent data. ## The measurement `ustat` fits a joint hole-level model (nodes: `Player`, `Hole`, `HoleOffset`, `DayForm`; ~2,000 nodes over ~3,300 events). It wanted to publish "this hole is unusually bad *for you*", which reads the `HoleOffset` node alone. On the served marginals that claim is unpublishable: - `|offset|` tops out at **0.46** strokes - posterior sigma on those same cells runs **0.83–1.16** - so **not one** of 90 player×hole cells clears even **one** sigma Rather than accept that, the model was fitted **twice on disjoint halves of the rounds** and the two independent estimates of the same cell compared: ``` rms disagreement between the two half-fits 0.157 what the marginals say it should be 1.670 ``` About tenfold. The estimator is far more reproducible than the model admits. Split-half only bounds this from below — it sees how far the estimate moves between samples, not shrinkage both halves share — so it was calibrated against a synthetic population with offsets of *known* size: split-half recovers 0.65–0.80 of the true error, giving a correction of ~1.37. Corrected, the honest sigma is **0.0995**, and **12 of 90 cells clear two sigma** where none ever cleared one. ## Why I am filing it as a question rather than a bug I do not know whether this is a defect or expected EP behaviour. Two candidate explanations, and I cannot separate them from outside the crate: 1. `HoleOffset` carries a deliberately tight prior (σ 1.5 against the player prior's 6.0). If the posterior sigma stays near the prior while the *mean* is well determined by data, the marginal is "right" in a Bayesian sense and simply not the quantity a consumer wants. 2. EP marginals in a densely coupled graph are genuinely conservative here. If it is (1), then the honest fix may be documentation — say plainly what the reported sigma is and is not, and that it should not be read as the standard error of the estimate. If it is (2), it is worth understanding. Either way, a consumer currently has to run **two extra full fits at boot** (~88 ms here, ~330 ms of total boot) plus a synthetic-data calibration constant to recover a number the model should be able to report. That constant is the shakiest figure in `ustat`: it was measured at the hole grain and is reused unmeasured at the layout grain, where it decides whether a real row is shown or hidden. ## What would help, in order of cost 1. **Documentation** of what the marginal means and how it should not be used. 2. **An effective-sample-size or evidence-count per node**, so a consumer can tell a well-determined node from one still sitting on its prior. 3. **A calibrated standard error** for a node's posterior mean, if that is a thing EP can be asked for. Context: `logaritmisk/ustat#114` (the UI claim this blocks) and `logaritmisk/ustat#106`.
logaritmisk added the decisionnumerics labels 2026-09-07 21:26:32 +00:00
logaritmisk commented 2026-09-07 23:53:28 +00:00
Author
Owner
Copy Link
Copy Source

Investigated. Your hypothesis (1) is right, hypothesis (2) is wrong, and the direction is the opposite of what the issue assumes. There is also now a call that gives you the quantity you were reconstructing by hand.

The crate is not being conservative — its marginals are too narrow

A scored history is linear-Gaussian, so the exact posterior has a closed form and the crate can be checked against ground truth (tests/marginal_calibration.rs):

means marginal sd, crate / exact
tree (no cycles) exact 1.000
loopy (round robin) exact 0.502

With cycles, message passing reports about half the true posterior width — standard loopy Gaussian BP behaviour (Weiss & Freeman 2001). That rules out your candidate (2): EP is not conservative here, it is over-confident.

So the ordering in your model is almost certainly:

exact posterior sd   >>   what current_skill reports   >>   split-half reproducibility

Why your node's sigma is genuinely wide — hypothesis (1), confirmed

I reproduced your structure: a joint player/layout model where every observation measures a sum of nodes against a reference (tests/additive_model.rs). That is the classic rating-scale indeterminacy — the data pins differences, and the overall level is pinned only by the prior.

node        current_skill      posterior_of (exact)
  p0        sigma 0.6368       sigma 3.1766      <- 5x wider
  h0        sigma 0.4910       sigma 3.6363

A single node in an additive model is weakly identified. Its exact posterior really is that wide, because the level it shares with its partners is anchored only by the prior. That is the honest Bayesian answer, and it is not the quantity you want.

This is also exactly why split-half cannot see it, as you already noted. Both halves share the same prior, so the prior-anchored component is identical in both and contributes nothing to their disagreement. Your 0.157 measures something real, but it is not that node's posterior sd, and no correction factor turns it into one — which means the synthetic calibration constant you flagged as your shakiest figure is doing work this structural fact says it cannot do.

The part that should change what you publish

History::posterior_of(&[(key, coefficient), ..]) landed in c52e255. It returns the exact posterior of a linear combination with correlations intact — validated to 1e-9 against the closed-form joint, for combinations and single nodes. A one-term query gives the exact marginal rather than the narrow one.

Measured on the additive fixture:

combination exact sigma adding marginals
p0 + h0 (a round) 4.0850 0.8041 0.20x — 5x too narrow
p0 - p1 (rank two players) 0.8741 0.8592 0.98x
h0 - h1 0.7119 0.7057 0.99x

Two things follow, and the second is a warning.

Differences are what your model actually determines. p0 - p1 has sigma 0.87 where each node alone is 3.2. If the claim you want is comparative — "this hole plays worse for you than for the field" — express it as a difference and it becomes publishable. Asking for one offset node in isolation asks for a quantity the data does not pin, which is why 90 of 90 cells failed to clear one sigma.

Sums are where "add the marginals" is dangerous. For score + layout — a row in #46's table — adding marginals is five times too narrow, i.e. overconfident. #46 assumed everything was overstated and that this was "at least the safe direction"; that holds for differences and reverses for sums. Worth auditing anything you publish that combines nodes additively rather than differentially.

On your three asks

  1. Documentation — done, on posterior_of and in tests/additive_model.rs, which states the direction for each combination shape.
  2. Effective sample size per node — posterior_of(&[(k, 1.0)]) is the better version: rather than an evidence count you must interpret, you get the exact posterior width, and comparing it to the prior says directly how far the data moved that node.
  3. Calibrated standard error of the posterior mean — for a linear-Gaussian model that is the exact posterior sd, so (3) and (1) collapse into one call. For ranked outcomes it remains an EP approximation.

Limits before you wire it up

posterior_of currently covers the latest slice and scored events only; ranked slices return JointUnavailable rather than a plausible wrong number. It is a dense solve, O(n^3) in the slice's competitor count: 39us at 50, 5.7ms at 400, 49ms at 800. If your slices are large, say so — caching the factorisation would make repeat queries O(n^2).

Leaving this open until you have measured against it, since the useful outcome is whether it closes your gap or whether something else is going on.

Investigated. **Your hypothesis (1) is right, hypothesis (2) is wrong, and the direction is the opposite of what the issue assumes.** There is also now a call that gives you the quantity you were reconstructing by hand. ## The crate is not being conservative — its marginals are too *narrow* A scored history is linear-Gaussian, so the exact posterior has a closed form and the crate can be checked against ground truth (`tests/marginal_calibration.rs`): | | means | marginal sd, crate / exact | |---|---|---| | tree (no cycles) | exact | **1.000** | | loopy (round robin) | exact | **0.502** | With cycles, message passing reports about **half** the true posterior width — standard loopy Gaussian BP behaviour (Weiss & Freeman 2001). That rules out your candidate (2): EP is not conservative here, it is over-confident. So the ordering in your model is almost certainly: ``` exact posterior sd >> what current_skill reports >> split-half reproducibility ``` ## Why your node's sigma is genuinely wide — hypothesis (1), confirmed I reproduced your structure: a joint player/layout model where every observation measures a **sum** of nodes against a reference (`tests/additive_model.rs`). That is the classic rating-scale indeterminacy — the data pins *differences*, and the overall level is pinned only by the prior. ``` node current_skill posterior_of (exact) p0 sigma 0.6368 sigma 3.1766 <- 5x wider h0 sigma 0.4910 sigma 3.6363 ``` A single node in an additive model is **weakly identified**. Its exact posterior really is that wide, because the level it shares with its partners is anchored only by the prior. That is the honest Bayesian answer, and it is not the quantity you want. This is also exactly why split-half cannot see it, as you already noted. Both halves share the same prior, so the prior-anchored component is identical in both and contributes nothing to their disagreement. Your 0.157 measures something real, but it is not that node's posterior sd, and no correction factor turns it into one — which means the synthetic calibration constant you flagged as your shakiest figure is doing work this structural fact says it cannot do. ## The part that should change what you publish `History::posterior_of(&[(key, coefficient), ..])` landed in `c52e255`. It returns the exact posterior of a linear combination with correlations intact — validated to 1e-9 against the closed-form joint, for combinations *and* single nodes. A one-term query gives the exact marginal rather than the narrow one. Measured on the additive fixture: | combination | exact sigma | adding marginals | | |---|---|---|---| | `p0 + h0` (a round) | 4.0850 | 0.8041 | **0.20x — 5x too narrow** | | `p0 - p1` (rank two players) | 0.8741 | 0.8592 | 0.98x | | `h0 - h1` | 0.7119 | 0.7057 | 0.99x | Two things follow, and the second is a warning. **Differences are what your model actually determines.** `p0 - p1` has sigma 0.87 where each node alone is 3.2. If the claim you want is comparative — "this hole plays worse *for you* than for the field" — express it as a difference and it becomes publishable. Asking for one offset node in isolation asks for a quantity the data does not pin, which is why 90 of 90 cells failed to clear one sigma. **Sums are where "add the marginals" is dangerous.** For `score + layout` — a row in #46's table — adding marginals is **five times too narrow**, i.e. overconfident. #46 assumed everything was overstated and that this was "at least the safe direction"; that holds for differences and reverses for sums. Worth auditing anything you publish that combines nodes additively rather than differentially. ## On your three asks 1. **Documentation** — done, on `posterior_of` and in `tests/additive_model.rs`, which states the direction for each combination shape. 2. **Effective sample size per node** — `posterior_of(&[(k, 1.0)])` is the better version: rather than an evidence count you must interpret, you get the exact posterior width, and comparing it to the prior says directly how far the data moved that node. 3. **Calibrated standard error of the posterior mean** — for a linear-Gaussian model that *is* the exact posterior sd, so (3) and (1) collapse into one call. For ranked outcomes it remains an EP approximation. ## Limits before you wire it up `posterior_of` currently covers the **latest slice** and **scored events only**; ranked slices return `JointUnavailable` rather than a plausible wrong number. It is a dense solve, O(n^3) in the slice's competitor count: 39us at 50, 5.7ms at 400, 49ms at 800. If your slices are large, say so — caching the factorisation would make repeat queries O(n^2). Leaving this open until you have measured against it, since the useful outcome is whether it closes your gap or whether something else is going on.
logaritmisk commented 2026-09-08 00:24:34 +00:00
Author
Owner
Copy Link
Copy Source

Measured against 0.5.0, as asked. posterior_of cannot answer either question on this model yet, and the reason is structural: it sees the latest time slice, and this is Through-Time.

What happens

ustat's round fit is 76 time slices over 100 rounds, because slices are per-day and players appear on the days they played. The latest slice is a solo round by one player. So:

query answerable
pair differences, [(a, 1.0), (b, -1.0)] 0 of 55 — every one UnknownKey, since no pair exists in that slice
single nodes, [(k, 1.0)] 1 of 11

And the one that works is degenerate. With a single competitor in the slice there is no correlation to account for, so the exact posterior is the marginal:

Anders Olsson    marginal 1.8691    posterior_of 1.8691
Eskil            UnknownKey (not in the latest slice)

That is the correct answer to the question the call asks. It is just not the question a Through-Time consumer has — our players are each at their own last round, spread across 76 slices, and none of the comparisons we publish lives in any single one.

Which leaves your two findings unresolved for us, in different ways

Hypothesis (1) I accept on your evidence. A single node in an additive model being weakly identified explains our 90-of-90 failure better than anything I had, and I take the point that no correction factor converts split-half reproducibility into a posterior sd. Those are different quantities and I was conflating them.

Your correction to the direction is what I cannot check, and it matters. Your two fixtures disagree, and both are plausible descriptions of us:

  • round robin: sqrt(sa^2 + sb^2) is 2.6x too wide for a difference
  • additive player+hole: the same form is 0.98x — essentially right

We are the additive shape by construction, but we are also densely cross-linked, since six players share one home layout across 36 rounds. Split-half on the real export says the marginal overstates a pair difference by 1.30x, which sits between your two figures and matches neither. I have shipped that as a calibration and it is now explicitly unvalidated rather than confirmed — I would rather have the exact number.

The sum warning I have acted on. We publish score + layout as a per-round rating. No interval is attached to it today, so nothing is wrong now, but that is recorded next to the field so the 5x-too-narrow direction cannot be rediscovered the hard way.

What would make it usable

Nothing urgent, and I am not asking for a redesign — the call is clearly right for the single-slice case and #46's ask is genuinely delivered. But for a Through-Time consumer the useful shape is a slice argument:

fn posterior_of_at(&self, time: T, terms: &[(&K, f64)]) -> Result<Gaussian, InferenceError>;

or, closer to what a standings table actually wants, terms resolved at each competitor's own latest slice — which is where current_skill already reads them from, so the two would then agree about which posterior they are describing. That second form is the one that would let me delete a calibration constant.

Worth saying the limitation is documented accurately; I just did not register what it meant for a 76-slice history until I ran it. If the doc gained one line — that a history with per-event or per-day slices will typically have very few competitors in its last one — it would have saved me the measurement, though the measurement was worth having anyway.

Happy for this to stay open or be closed and re-filed as a slice-selection issue, whichever you prefer. The probe lives in model::separation::joint on our side and re-runs in a second.

Measured against 0.5.0, as asked. **`posterior_of` cannot answer either question on this model yet, and the reason is structural: it sees the latest time slice, and this is Through-Time.** ## What happens ustat's round fit is **76 time slices over 100 rounds**, because slices are per-day and players appear on the days they played. The latest slice is a **solo round by one player**. So: | query | answerable | |---|---| | pair differences, `[(a, 1.0), (b, -1.0)]` | **0 of 55** — every one `UnknownKey`, since no pair exists in that slice | | single nodes, `[(k, 1.0)]` | **1 of 11** | And the one that works is degenerate. With a single competitor in the slice there is no correlation to account for, so the exact posterior *is* the marginal: ``` Anders Olsson marginal 1.8691 posterior_of 1.8691 Eskil UnknownKey (not in the latest slice) ``` That is the correct answer to the question the call asks. It is just not the question a Through-Time consumer has — our players are each at their own last round, spread across 76 slices, and none of the comparisons we publish lives in any single one. ## Which leaves your two findings unresolved for us, in different ways **Hypothesis (1) I accept on your evidence.** A single node in an additive model being weakly identified explains our 90-of-90 failure better than anything I had, and I take the point that no correction factor converts split-half reproducibility into a posterior sd. Those are different quantities and I was conflating them. **Your correction to the direction is what I cannot check, and it matters.** Your two fixtures disagree, and both are plausible descriptions of us: - round robin: `sqrt(sa^2 + sb^2)` is **2.6x too wide** for a difference - additive player+hole: the same form is **0.98x** — essentially right We are the additive shape by construction, but we are also densely cross-linked, since six players share one home layout across 36 rounds. Split-half on the real export says the marginal overstates a pair difference by **1.30x**, which sits between your two figures and matches neither. I have shipped that as a calibration and it is now explicitly unvalidated rather than confirmed — I would rather have the exact number. **The sum warning I have acted on.** We publish `score + layout` as a per-round rating. No interval is attached to it today, so nothing is wrong now, but that is recorded next to the field so the 5x-too-narrow direction cannot be rediscovered the hard way. ## What would make it usable Nothing urgent, and I am not asking for a redesign — the call is clearly right for the single-slice case and #46's ask is genuinely delivered. But for a Through-Time consumer the useful shape is a slice argument: ```rust fn posterior_of_at(&self, time: T, terms: &[(&K, f64)]) -> Result<Gaussian, InferenceError>; ``` or, closer to what a standings table actually wants, terms resolved at **each competitor's own latest slice** — which is where `current_skill` already reads them from, so the two would then agree about which posterior they are describing. That second form is the one that would let me delete a calibration constant. Worth saying the limitation is documented accurately; I just did not register what it meant for a 76-slice history until I ran it. If the doc gained one line — that a history with per-event or per-day slices will typically have very few competitors in its last one — it would have saved me the measurement, though the measurement was worth having anyway. Happy for this to stay open or be closed and re-filed as a slice-selection issue, whichever you prefer. The probe lives in `model::separation::joint` on our side and re-runs in a second.
logaritmisk commented 2026-09-08 04:47:01 +00:00
Author
Owner
Copy Link
Copy Source

Fixed in 0.6.0. The latest-slice limitation was my mistake, not a documentation gap, and your measurement is what caught it.

What was wrong

I built the joint over a single slice and validated it against single-slice fixtures — which is exactly the shape that cannot expose the problem. In a library whose premise is skill over time, competitors are read at their own last appearance, and those are different slices by construction. Your 76-slice history with a solo round at the end is not an awkward edge case; it is the normal shape and my fixture was the unusual one.

You offered a doc line as the fix. That would have been the wrong fix — it would have documented a limitation that should not exist.

What 0.6.0 does

The joint is now time-expanded: one variable per appearance, linked by the prior on a first appearance, the drift between consecutive ones, and the within-slice event contrasts. Consecutive appearances with no drift between them collapse to the same variable, which keeps the matrix positive-definite for a competitor pinned at drift_scale = 0.

  • posterior_of reads each competitor at their own latest appearance — the same place current_skill reads them — so the two now agree about which posterior they describe. Your 55 pair differences should all resolve.
  • posterior_of_at(time, terms) is there too, reading each competitor at their latest appearance at or before time, matching learning_curve. For a standings table you want the former; for "how did these two stand at the end of last season", the latter.

Validated against a hand-written exact posterior for a two-competitor, two-slice history — the precision matrix is spelled out in the test rather than obtained from the crate, so it is an independent check. Also pinned: competitors last seen in different slices compare at all, means still agree with the marginals, zero drift makes slice layout irrelevant, and more drift widens a comparison across time.

The thing I still cannot settle, and you can

Your sharpest point stands and I want to be clear that it does. My two fixtures disagree about the direction — 2.6x on the round robin, 0.98x on the additive one — and your export measures 1.30x, matching neither. I generalised a claim from fixtures that bracket your case rather than reproduce it, and you were right to treat 1.30 as unvalidated rather than take my number.

posterior_of should now resolve on your model, so the exact figure is available directly and the calibration constant can go. If it does not land near 1.30, I would rather hear that than not — a disagreement there would say something is wrong in my account of why the errors cancel for differences, and that account is currently load-bearing for the advice in this issue.

Note on cost

Cost is now a dense solve over appearances rather than competitors: one variable per competitor per slice it appears in, minus any consecutive pair with no drift between them. For your round fit that is small. For the 1,991-node hole model over 76 slices it may not be, and if it is slow, say so — caching the factorisation makes repeat queries O(n^2), and sparsity is the step after that. I have deliberately not optimised ahead of a measurement.

Leaving this open until you have run it.

Fixed in `0.6.0`. **The latest-slice limitation was my mistake, not a documentation gap**, and your measurement is what caught it. ## What was wrong I built the joint over a single slice and validated it against single-slice fixtures — which is exactly the shape that cannot expose the problem. In a library whose premise is skill over time, competitors are read at *their own* last appearance, and those are different slices by construction. Your 76-slice history with a solo round at the end is not an awkward edge case; it is the normal shape and my fixture was the unusual one. You offered a doc line as the fix. That would have been the wrong fix — it would have documented a limitation that should not exist. ## What 0.6.0 does The joint is now time-expanded: one variable per *appearance*, linked by the prior on a first appearance, the drift between consecutive ones, and the within-slice event contrasts. Consecutive appearances with no drift between them collapse to the same variable, which keeps the matrix positive-definite for a competitor pinned at `drift_scale = 0`. - **`posterior_of`** reads each competitor at their own latest appearance — the same place `current_skill` reads them — so the two now agree about which posterior they describe. Your 55 pair differences should all resolve. - **`posterior_of_at(time, terms)`** is there too, reading each competitor at their latest appearance at or before `time`, matching `learning_curve`. For a standings table you want the former; for "how did these two stand at the end of last season", the latter. Validated against a hand-written exact posterior for a two-competitor, two-slice history — the precision matrix is spelled out in the test rather than obtained from the crate, so it is an independent check. Also pinned: competitors last seen in different slices compare at all, means still agree with the marginals, zero drift makes slice layout irrelevant, and more drift widens a comparison across time. ## The thing I still cannot settle, and you can Your sharpest point stands and I want to be clear that it does. My two fixtures disagree about the direction — 2.6x on the round robin, 0.98x on the additive one — and your export measures 1.30x, matching neither. I generalised a claim from fixtures that *bracket* your case rather than reproduce it, and you were right to treat 1.30 as unvalidated rather than take my number. `posterior_of` should now resolve on your model, so the exact figure is available directly and the calibration constant can go. If it does not land near 1.30, I would rather hear that than not — a disagreement there would say something is wrong in my account of why the errors cancel for differences, and that account is currently load-bearing for the advice in this issue. ## Note on cost Cost is now a dense solve over *appearances* rather than competitors: one variable per competitor per slice it appears in, minus any consecutive pair with no drift between them. For your round fit that is small. For the 1,991-node hole model over 76 slices it may not be, and if it is slow, say so — caching the factorisation makes repeat queries O(n^2), and sparsity is the step after that. I have deliberately not optimised ahead of a measurement. Leaving this open until you have run it.
logaritmisk commented 2026-09-08 04:57:37 +00:00
Author
Owner
Copy Link
Copy Source

Run against 0.6.0. All 55 pairs resolve, and the answer is that my 1.30 was wrong — the marginal was already right.

The number

exact / marginal, 55 pairs, real export:  1.035

So sqrt(sa^2 + sb^2) is within 3.5% of exact for a difference on this model, and if anything a touch narrow rather than wide. That is your additive fixture's 0.98x, not the round robin's 2.6x — your account of why the errors cancel for differences holds here, and the fixture that brackets us on the low side is the one that describes us.

Sample rows:

pair                       marginal    EXACT   exact/marg
Amanda x Anders Olsson       5.4760   5.6426     1.030
Amanda x Eric                6.8544   6.8519     1.000
Anders Olsson x Anners       2.9698   3.1514     1.061

Single nodes moved the way you predicted: Anders 1.8691 marginal against 2.3454 exact, Eskil 2.4210 against 2.9686. About 1.25x wider, consistent with weak identification of one node in an additive model.

I had shipped the wrong thing, and this caught it

Between your two comments I shipped the 1.30 as a calibration — every separation scaled by 0.767. That made them 23% too small, inflated the odds we publish, and silently dropped two pairs out of a UI block whose entire job is to say "we cannot tell these apart".

The mistake is the one you named in your first comment, and I want to be precise that you named it before it bit: split-half measures how far an estimate moves between samples; a posterior sd is how wide the belief is; no correction converts one into the other. I accepted that for the hole grain and then did exactly it at the pair grain, because there the split-half number looked plausible.

Now deleted. exact_separation calls posterior_of and the constant is gone. The block is back to five pairs.

Cost — you asked, and the answer splits

Dense solve over appearances, measured on the real export:

grain nodes slices one solve
round ~250 76 4.4 ms
hole ~2,000 76 6.0 s

The round grain is fine: 55 pairs is ~245 ms at boot, which a read-only app that computes everything once can afford, and I have shipped it.

The hole grain is not usable. Our heatmap is 90 player×hole cells, so ~9 minutes. That is the model where the original question in this issue lives — the 90-of-90 failure — so the quantity you showed me how to compute correctly is the one I still cannot afford to compute. honest_offset_sigma keeps its split-half estimate there, now documented as measuring reproducibility rather than posterior width, which is at least honest about what it is.

Not asking for optimisation work on my account: nine minutes at boot is a "no" whatever the constant factor, and the fix on our side is probably to ask a different question — express the hole claim as a difference against the field, as you suggested, rather than reading one offset node. If caching the factorisation happens anyway, the numbers above are the baseline.

On the fixture point

Worth stating plainly since you raised it: your two fixtures bracketed us and neither reproduced us, and you were right that this needed measuring rather than generalising. It also went the other way — my measurement was the one that turned out to be measuring the wrong thing, and your fixture-based reasoning about differences was correct. The useful lesson on my side is that an empirical number is not automatically the trustworthy one; it has to be measuring the quantity you are about to publish.

Happy for this to close.

Run against 0.6.0. **All 55 pairs resolve, and the answer is that my 1.30 was wrong — the marginal was already right.** ## The number ``` exact / marginal, 55 pairs, real export: 1.035 ``` So `sqrt(sa^2 + sb^2)` is within 3.5% of exact for a difference on this model, and if anything a touch **narrow** rather than wide. That is your additive fixture's 0.98x, not the round robin's 2.6x — your account of why the errors cancel for differences holds here, and the fixture that brackets us on the low side is the one that describes us. Sample rows: ``` pair marginal EXACT exact/marg Amanda x Anders Olsson 5.4760 5.6426 1.030 Amanda x Eric 6.8544 6.8519 1.000 Anders Olsson x Anners 2.9698 3.1514 1.061 ``` Single nodes moved the way you predicted: Anders 1.8691 marginal against **2.3454** exact, Eskil 2.4210 against **2.9686**. About 1.25x wider, consistent with weak identification of one node in an additive model. ## I had shipped the wrong thing, and this caught it Between your two comments I shipped the 1.30 as a calibration — every separation scaled by 0.767. That made them **23% too small**, inflated the odds we publish, and silently dropped two pairs out of a UI block whose entire job is to say "we cannot tell these apart". The mistake is the one you named in your first comment, and I want to be precise that you named it before it bit: split-half measures how far an **estimate** moves between samples; a posterior sd is how wide the **belief** is; no correction converts one into the other. I accepted that for the hole grain and then did exactly it at the pair grain, because there the split-half number looked plausible. Now deleted. `exact_separation` calls `posterior_of` and the constant is gone. The block is back to five pairs. ## Cost — you asked, and the answer splits Dense solve over appearances, measured on the real export: | grain | nodes | slices | one solve | |---|---|---|---| | round | ~250 | 76 | **4.4 ms** | | hole | ~2,000 | 76 | **6.0 s** | The round grain is fine: 55 pairs is ~245 ms at boot, which a read-only app that computes everything once can afford, and I have shipped it. **The hole grain is not usable.** Our heatmap is 90 player×hole cells, so ~9 minutes. That is the model where the original question in this issue lives — the 90-of-90 failure — so the quantity you showed me how to compute correctly is the one I still cannot afford to compute. `honest_offset_sigma` keeps its split-half estimate there, now documented as measuring reproducibility rather than posterior width, which is at least honest about what it is. Not asking for optimisation work on my account: nine minutes at boot is a "no" whatever the constant factor, and the fix on our side is probably to ask a different question — express the hole claim as a **difference** against the field, as you suggested, rather than reading one offset node. If caching the factorisation happens anyway, the numbers above are the baseline. ## On the fixture point Worth stating plainly since you raised it: your two fixtures bracketed us and neither reproduced us, and you were right that this needed measuring rather than generalising. It also went the other way — my measurement was the one that turned out to be measuring the wrong thing, and your fixture-based reasoning about differences was correct. The useful lesson on my side is that an empirical number is not automatically the trustworthy one; it has to be measuring the quantity you are about to publish. Happy for this to close.
logaritmisk commented 2026-09-08 17:46:24 +00:00
Author
Owner
Copy Link
Copy Source

Closing, as you offered. Late — your measurement landed on 2026-09-08 and I left it sitting; I had this filed in my head as still waiting on you when you had already answered it.

Where it ended

exact / marginal, 55 pairs, real export:  1.035

Which settles the one thing I could not: my two fixtures bracketed you at 2.6x and 0.98x, and you land at 1.035 — the additive fixture, not the round robin. So the account of why the errors cancel for differences holds on a real densely-cross-linked model, not only on a fixture built to demonstrate it. That account was load-bearing for the advice in this issue and it is now checked against something I did not construct.

Single nodes moved the way weak identification predicts — 1.87 to 2.35, 2.42 to 2.97, about 1.25x — which is the same phenomenon at a smaller magnitude than the hole grain's, and consistent.

The part worth keeping

You wrote that the useful lesson was that an empirical number is not automatically the trustworthy one, and that it has to be measuring the quantity you are about to publish. That is the whole issue in one line, and it cuts both ways here: your 1.30 was measured and wrong, my 2.6x was reasoned and wrong, and what resolved it was measuring the right quantity on the real thing.

I would add the symmetric half from my side. I built the joint over a single slice and validated it on single-slice fixtures — the one shape that could not expose the defect — and then reported a limitation as though it were a design boundary. Your 76-slice history with a solo round at the end was never an edge case; my fixture was.

Both halves are now in the repo rather than in this thread: tests/additive_model.rs pins the direction per combination shape, and tests/time_expanded_joint.rs checks the multi-slice joint against a hand-written exact posterior whose precision matrix is spelled out in the test rather than obtained from the crate.

Thank you for pushing this to a number instead of accepting the first plausible story — including mine.

Closing, as you offered. Late — your measurement landed on 2026-09-08 and I left it sitting; I had this filed in my head as still waiting on you when you had already answered it. ## Where it ended ``` exact / marginal, 55 pairs, real export: 1.035 ``` Which settles the one thing I could not: my two fixtures bracketed you at 2.6x and 0.98x, and you land at 1.035 — the additive fixture, not the round robin. So the account of *why* the errors cancel for differences holds on a real densely-cross-linked model, not only on a fixture built to demonstrate it. That account was load-bearing for the advice in this issue and it is now checked against something I did not construct. Single nodes moved the way weak identification predicts — 1.87 to 2.35, 2.42 to 2.97, about 1.25x — which is the same phenomenon at a smaller magnitude than the hole grain's, and consistent. ## The part worth keeping You wrote that the useful lesson was that an empirical number is not automatically the trustworthy one, and that it has to be measuring the quantity you are about to publish. That is the whole issue in one line, and it cuts both ways here: your 1.30 was measured and wrong, my 2.6x was reasoned and wrong, and what resolved it was measuring the *right* quantity on the real thing. I would add the symmetric half from my side. I built the joint over a single slice and validated it on single-slice fixtures — the one shape that could not expose the defect — and then reported a limitation as though it were a design boundary. Your 76-slice history with a solo round at the end was never an edge case; my fixture was. Both halves are now in the repo rather than in this thread: `tests/additive_model.rs` pins the direction per combination shape, and `tests/time_expanded_joint.rs` checks the multi-slice joint against a hand-written exact posterior whose precision matrix is spelled out in the test rather than obtained from the crate. Thank you for pushing this to a number instead of accepting the first plausible story — including mine.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
Shape or ergonomics of the public surface
 blocked
Waiting on another issue or an external decision
 breaking
Resolving it changes the public API; wants a major bump
 bug
Behaviour is wrong, including silently wrong
 decision
Needs a call to be made, not code to be written
 docs
Documentation, including doc comments and the README
 enhancement
New capability the crate does not have
 infra
CI, release process, packaging
 numerics
Floating-point accuracy, cancellation, convergence maths
 performance
Speed or memory
 refactor
Internal cleanup with no intended behaviour change
 testing
Test coverage and test infrastructure
No labels decision numerics
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logaritmisk
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logaritmisk/trueskill-tt#47