Design question: can a fitted History be persisted, or must every consumer replay its whole event log? #45

New Issue

2026-09-07T21:20:47Z

logaritmisk commented

2026-09-07 21:20:47 +00:00

The situation downstream

Nothing in the crate is serializable — no serde dependency, no derives, and History's internals (time_slices, agents) are private. So the only way to obtain a fitted model is to replay every event through it.

lester-stash does exactly that. Its Trainer::Reset re-reads the entire event CSV, rebuilds the history from scratch and re-converges. That is the startup path, and until recently it was also triggered on a user action — a full-history retrain per click, which we fixed by removing the trigger rather than by making the retrain cheaper, because there was no way to make it cheaper.

Today that costs a few thousand events. It grows monotonically, and it is the kind of cost that is invisible until it isn't.

Why I am filing this as a question, not a feature request

Persistence for TrueSkill Through Time is not obviously well-defined, and I would rather ask than assume:

The premise cuts against it. TTT's whole point is that new evidence revises the past — converge() runs backward and forward passes until the whole history agrees. A snapshot is a fixed point of the events seen so far. Restore it, append new events, converge again: is that identical to converging the full log from scratch? For a single forward pass, plainly yes. For a smoothed history where a new event legitimately revises skills at earlier times, I do not think the answer is obvious, and it may depend on how far back the new events sit.

A wrong answer here is the crate's known failure mode. If restore-then-append silently diverges from full-replay, the result is not a crash — it is a slightly wrong model that looks entirely plausible. That is the same shape as the defects this crate has already fixed twice, and it argues for either a correctness proof or an explicit "this is an approximation" contract, rather than a convenient Serialize derive.

Questions

Is restore(snapshot) + add_events(new) + converge() intended to equal add_events(all) + converge()? Exactly, approximately, or not at all?
If not exactly, is there a bound — e.g. exact when new events are strictly later than every slice in the snapshot?
Would a narrower primitive be sounder than full serialization: exporting learning_curves() as a set of priors and reconstructing from those, accepting the loss of cross-slice messages?
Is this simply out of scope, and consumers should keep their event log as the source of truth? That is a perfectly good answer — the event log is small, durable and already the thing we back up.

What would make it moot

If the answer is (4), a line in the docs saying so would help. Right now the absence of serde reads as an omission rather than a decision, which invites exactly this issue from every consumer in turn.

## The situation downstream Nothing in the crate is serializable — no `serde` dependency, no derives, and `History`'s internals (`time_slices`, `agents`) are private. So the only way to obtain a fitted model is to replay every event through it. lester-stash does exactly that. Its `Trainer::Reset` re-reads the **entire** event CSV, rebuilds the history from scratch and re-converges. That is the startup path, and until recently it was also triggered on a user action — a full-history retrain per click, which we fixed by removing the trigger rather than by making the retrain cheaper, because there was no way to make it cheaper. Today that costs a few thousand events. It grows monotonically, and it is the kind of cost that is invisible until it isn't. ## Why I am filing this as a question, not a feature request Persistence for TrueSkill Through Time is not obviously well-defined, and I would rather ask than assume: **The premise cuts against it.** TTT's whole point is that new evidence revises the past — `converge()` runs backward and forward passes until the whole history agrees. A snapshot is a fixed point of *the events seen so far*. Restore it, append new events, converge again: is that identical to converging the full log from scratch? For a single forward pass, plainly yes. For a smoothed history where a new event legitimately revises skills at earlier times, I do not think the answer is obvious, and it may depend on how far back the new events sit. **A wrong answer here is the crate's known failure mode.** If restore-then-append silently diverges from full-replay, the result is not a crash — it is a slightly wrong model that looks entirely plausible. That is the same shape as the defects this crate has already fixed twice, and it argues for either a correctness proof or an explicit "this is an approximation" contract, rather than a convenient `Serialize` derive. ## Questions 1. Is `restore(snapshot) + add_events(new) + converge()` intended to equal `add_events(all) + converge()`? Exactly, approximately, or not at all? 2. If not exactly, is there a bound — e.g. exact when new events are strictly later than every slice in the snapshot? 3. Would a narrower primitive be sounder than full serialization: exporting `learning_curves()` as a set of priors and reconstructing from those, accepting the loss of cross-slice messages? 4. Is this simply out of scope, and consumers should keep their event log as the source of truth? That is a perfectly good answer — the event log is small, durable and already the thing we back up. ## What would make it moot If the answer is (4), a line in the docs saying so would help. Right now the absence of serde reads as an omission rather than a decision, which invites exactly this issue from every consumer in turn.

logaritmisk added the decision docs labels 2026-09-07 21:41:40 +00:00

logaritmisk referenced this issue from a commit

2026-09-07 23:18:33 +00:00

test: pin that re-convergence is path-independent

logaritmisk commented

2026-09-07 23:21:36 +00:00

Answering with measurements rather than opinion. Short version: your premise is right that a snapshot is delicate, but wrong about where the delicacy is — and Q1's answer is better than you feared.

Q1: is `restore + append + converge` equal to `add_events(all) + converge`?

Exactly, to the tolerance converge itself guarantees. Now pinned in tests/reconvergence_equivalence.rs:

	worst abs(delta mu)	worst abs(delta sigma)
append strictly later	6.2e-13	1.1e-16
append interleaved / back-dated	8.9e-11	5.6e-15

Both at the convergence epsilon (1e-12 in the fixture). This is a stronger property than tests/ingestion_equivalence.rs covers — that file varies batching but converges only at the end. These converge between chunks, which is your actual path.

Q2: is there a bound, e.g. exact only when new events are strictly later?

No such restriction exists. The interleaved case above back-dates the second chunk into the first's range and is equally exact.

The reason is the thing that makes the whole question tractable: converge reaches a fixed point determined by the events, ratings and configuration alone — not by the message state it started from. Through Time revises the past on every converge regardless, so doing it in two steps is not a special case. add_events calls competitor::clean(.., true) and re-primes the slices anyway, which is why the starting state cannot leak through.

Q3: would exporting `learning_curves()` as priors be sounder?

No — strictly worse, and unnecessary. It discards the cross-slice coupling, which is the mechanism the model exists for, and buys nothing: full-fidelity restore is already sound by Q1/Q2.

Q4: is it out of scope?

No, but not for the reason the issue assumes.

Because the fixed point is path-independent, a snapshot contains no information your event log does not. Persisting fitted state saves recomputation, never information. So this was never a correctness question — and the corollary is the useful part: a restored snapshot cannot corrupt the answer, only the speed. An inexact or stale one just costs an extra sweep. That is the opposite of the risk you were worried about, and it makes serialization far safer than it looked.

The benefit is real but narrower than "make appends cheap". Measured on 2000 events / 200 competitors:

cold converge                        : 365.27ms  iterations 390
append one event + re-converge       : 508.89ms  iterations 546
no-op re-converge (nothing appended) :   0.91ms  iterations 1

Restart with no new events: 400x cheaper. That is Trainer::Reset at boot, and it is exactly what a snapshot buys.
Append then re-converge: no cheaper at all, and that is correct rather than a limitation. #32 spiked this — appending one event moves the two participants ~1.2 sigma across their entire history, back to their first appearance. The previous solution is genuinely far from the new one, so re-convergence is real work, not repeated work. Cost per append is inherently O(history); an implementation that made it cheap would be computing a filtering estimate, which filtered_learning_curve already provides, rather than Through Time.

Concretely

Your (4) — keep the event log as the source of truth — is the right answer today, and it should read as a decision rather than an omission.

But the door is open and the shape is clear: a serde feature snapshotting the fitted state would be a boot cache, worth ~400x on restart and nothing on append. The path-independence result means it is low-risk — the worst a bad snapshot can do is cost you one convergence.

If the retrain-per-click is gone and boot is tolerable, I would leave it. If boot becomes the pain point as your log grows, say so and I will treat snapshot-as-cache as a real feature.

One thing this turned up

Chasing path-independence surfaced a latent bug in this crate's own property test: ingestion_order_does_not_change_the_answer was comparing two fits that had both stopped at their iteration cap and reporting the truncation as an order dependence. Fixed in 35d7512. Same failure mode as #50 — a short fit looks exactly like a real answer.

Answering with measurements rather than opinion. Short version: **your premise is right that a snapshot is delicate, but wrong about where the delicacy is** — and Q1's answer is better than you feared. ## Q1: is `restore + append + converge` equal to `add_events(all) + converge`? **Exactly**, to the tolerance `converge` itself guarantees. Now pinned in `tests/reconvergence_equivalence.rs`: | | worst abs(delta mu) | worst abs(delta sigma) | |---|---|---| | append strictly later | **6.2e-13** | 1.1e-16 | | append interleaved / back-dated | **8.9e-11** | 5.6e-15 | Both at the convergence epsilon (1e-12 in the fixture). This is a stronger property than `tests/ingestion_equivalence.rs` covers — that file varies *batching* but converges only at the end. These converge *between* chunks, which is your actual path. ## Q2: is there a bound, e.g. exact only when new events are strictly later? **No such restriction exists.** The interleaved case above back-dates the second chunk into the first's range and is equally exact. The reason is the thing that makes the whole question tractable: **`converge` reaches a fixed point determined by the events, ratings and configuration alone — not by the message state it started from.** Through Time revises the past on *every* converge regardless, so doing it in two steps is not a special case. `add_events` calls `competitor::clean(.., true)` and re-primes the slices anyway, which is why the starting state cannot leak through. ## Q3: would exporting `learning_curves()` as priors be sounder? **No — strictly worse, and unnecessary.** It discards the cross-slice coupling, which is the mechanism the model exists for, and buys nothing: full-fidelity restore is already sound by Q1/Q2. ## Q4: is it out of scope? **No, but not for the reason the issue assumes.** Because the fixed point is path-independent, **a snapshot contains no information your event log does not**. Persisting fitted state saves *recomputation*, never information. So this was never a correctness question — and the corollary is the useful part: **a restored snapshot cannot corrupt the answer, only the speed.** An inexact or stale one just costs an extra sweep. That is the opposite of the risk you were worried about, and it makes serialization far safer than it looked. The benefit is real but narrower than "make appends cheap". Measured on 2000 events / 200 competitors: ``` cold converge : 365.27ms iterations 390 append one event + re-converge : 508.89ms iterations 546 no-op re-converge (nothing appended) : 0.91ms iterations 1 ``` - **Restart with no new events: 400x cheaper.** That is `Trainer::Reset` at boot, and it is exactly what a snapshot buys. - **Append then re-converge: no cheaper at all**, and that is correct rather than a limitation. #32 spiked this — appending one event moves the two participants ~1.2 sigma across their *entire* history, back to their first appearance. The previous solution is genuinely far from the new one, so re-convergence is real work, not repeated work. Cost per append is inherently O(history); an implementation that made it cheap would be computing a filtering estimate, which `filtered_learning_curve` already provides, rather than Through Time. ## Concretely Your (4) — keep the event log as the source of truth — is the right answer today, and it should read as a decision rather than an omission. But the door is open and the shape is clear: a `serde` feature snapshotting the fitted state would be a **boot cache**, worth ~400x on restart and nothing on append. The path-independence result means it is low-risk — the worst a bad snapshot can do is cost you one convergence. If the retrain-per-click is gone and boot is tolerable, I would leave it. If boot becomes the pain point as your log grows, say so and I will treat snapshot-as-cache as a real feature. ## One thing this turned up Chasing path-independence surfaced a latent bug in this crate's own property test: `ingestion_order_does_not_change_the_answer` was comparing two fits that had both stopped at their iteration cap and reporting the truncation as an order dependence. Fixed in `35d7512`. Same failure mode as #50 — a short fit looks exactly like a real answer.

logaritmisk referenced this issue from a commit

2026-09-07 23:22:32 +00:00

docs: record that the event log is the source of truth, and why

logaritmisk closed this issue

2026-09-07 23:22:33 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logaritmisk/trueskill-tt#45

Design question: can a fitted History be persisted, or must every consumer replay its whole event log? #45