fix(history): stop reprocessing the slice that was just appended to

`add_events_with_prior` advanced `k` past the slice it had written when it
created a new one, but not when it appended to an existing one. The trailing
forward-refresh loop therefore started *on* the slice just modified and ran
`new_forward_info` over it again.

That is not merely redundant work. The loop immediately above it sets each
agent's message to `forward * likelihood` for that slice, and
`new_forward_info` then assigns `skill.forward = message.forget(drift)` —
folding the slice's own likelihood back into its own forward prior. The
skills it produced depended on how events had been batched.

Ingesting one event at a time now converges to the same fixed point as
ingesting the same events in a single call, which it previously did not:
for five events sharing a timestamp, competitor `a` converged to
mu=7.44 sigma=3.90 batched versus mu=7.99 sigma=3.10 incrementally. Both
runs had converged; the gap was not a convergence residual.

The numerical goldens never caught this because they all ingest in one call
with a distinct timestamp per event, so the append-to-existing-slice branch
is never taken. `tests/ingestion_equivalence.rs` covers it directly, and
asserts convergence before comparing so that a residual cannot be mistaken
for agreement.

Removing the redundant re-inference also removes the dominant cost of
incremental ingestion, which was quadratic in the number of events already
in the slice:

    events   before     after    speedup
       500   45.8ms     1.1ms       42x
      1000  179.5ms     2.8ms       64x
      2000  721.8ms     9.9ms       73x
      4000    2.9s     35.4ms       82x

Ingesting one at a time is now 1.8x a single batched call, down from 148x.

Two supporting changes are included:

- Color groups are rebuilt lazily rather than on every append. Nothing
  reads the partition between an append and the next full sweep, so the
  per-append rebuild was pure waste.
- `ColorGroups::groups_are_contiguous` is asserted after each rebuild and
  in `color_range`. The parallel sweep derives one `&mut` sub-slice per
  color from those ranges and relies on them being disjoint; that invariant
  was established by construction but never checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnsaJg74eNSva3PJjK2eej
This commit is contained in:
2026-08-04 21:51:02 +02:00
co-authored by Claude Opus 5
parent 0f1a1b8911
commit c088214fed
5 changed files with 233 additions and 17 deletions
+22 -1
View File
@@ -174,6 +174,14 @@ pub struct TimeSlice<T: Time = i64> {
pub(crate) convergence: crate::ConvergenceOptions,
arena: ScratchArena,
pub(crate) color_groups: ColorGroups,
/// Whether `color_groups` still reflects `events`.
///
/// Coloring is rebuilt lazily, on the first full sweep after an append,
/// rather than eagerly per append: the partition is thrown away and
/// recomputed wholesale either way, so doing it per append made ingesting
/// n events O(n^2) with no benefit — nothing reads the partition between
/// an append and the next full sweep.
color_groups_dirty: bool,
}
impl<T: Time> TimeSlice<T> {
@@ -186,6 +194,7 @@ impl<T: Time> TimeSlice<T> {
convergence,
arena: ScratchArena::new(),
color_groups: ColorGroups::new(),
color_groups_dirty: false,
}
}
@@ -198,6 +207,7 @@ impl<T: Time> TimeSlice<T> {
let n = self.events.len();
if n == 0 {
self.color_groups = ColorGroups::new();
self.color_groups_dirty = false;
return;
}
@@ -221,6 +231,12 @@ impl<T: Time> TimeSlice<T> {
self.events = reordered;
self.color_groups = ColorGroups { groups: new_groups };
self.color_groups_dirty = false;
debug_assert!(
self.color_groups.groups_are_contiguous(),
"color groups must occupy contiguous event ranges"
);
}
pub fn add_events<D: Drift<T>>(
@@ -306,8 +322,9 @@ impl<T: Time> TimeSlice<T> {
self.events.extend(events);
self.color_groups_dirty = true;
self.iteration(from, agents);
self.recompute_color_groups();
}
pub(crate) fn posteriors(&self) -> HashMap<Index, Gaussian> {
@@ -318,6 +335,10 @@ impl<T: Time> TimeSlice<T> {
}
pub fn iteration<D: Drift<T>>(&mut self, from: usize, agents: &CompetitorStore<T, D>) {
if from == 0 && self.color_groups_dirty {
self.recompute_color_groups();
}
if from > 0 || self.color_groups.is_empty() {
// Initial pass (add_events) or no color groups yet: simple sequential sweep.
for event in self.events.iter_mut().skip(from) {