posterior_of is nondeterministic across processes: float sums over a HashMap #62

New Issue

2026-09-09T14:52:50Z

logaritmisk commented

2026-09-09 14:52:50 +00:00

src/history.rs:2317-2320 and :2435-2440. ResolvedTerms::unseen (:264) is a HashMap<String, f64>, built at :1058, and three float reductions iterate it:

// distribution(), :2317
+ resolved.unseen.values().map(|c| c * c * prior_var).sum::<f64>()
// expected_variance_reduction(), :2435 and :2440
let cross_unseen: f64 = target.unseen.iter().map(...).sum();
let self_unseen:  f64 = matchup.unseen.values().map(|c| c * c * prior_var).sum();

Rust's default hasher is seeded per process, so the addition order — and therefore the sum — changes run to run. Addition is not associative.

Measured

Same binary, same input, separate processes. Reproduced independently at 40 runs (two distinct bit patterns, 24/16); the original sweep at 60 runs:

60  scored_digest bda4ea80ee78f53a       <- converged posteriors: stable
60  scored_le     c08ee607b2488c1d       <- log_evidence: stable
57  posterior_of  sigma=4051f46fa4bb18dd
 3  posterior_of  sigma=4051f46fa4bb18dc  <- 2 distinct bit patterns

25  evr 408972e942462c2f  8.14363895939086319231e2
19  evr 408972e942462c2d  8.14363895939086091857e2
13  evr 408972e942462c31  8.14363895939086546605e2
 2  evr 408972e942462c2b  8.14363895939085864484e2
 1  evr 408972e942462c32  8.14363895939086660292e2   <- 5 values, ~7 ULP

Mechanism isolated by falsification

Same map, same size, same code path — determinism returns exactly when the terms become equal and summation becomes order-independent:

unseen keys	coefficients	distinct results over 60 processes
1, 2, 3, 4, 8	distinct	1 (deterministic)
12	distinct	2
24	distinct	2
3, 8, 12	all 1.0	1 (deterministic)

That rules out any other per-process variation. Reproduced identically in debug and release, with rayon on and off.

Scope

Reachable via History::posterior_of, posterior_of_at, Joint::posterior_of, Joint::posterior_of_at, Joint::expected_variance_reduction — whenever UnknownKeys::Prior is set and three or more unknown keys with unequal coefficients are passed.

Magnitude is 1–7 ULP, so it will not change a decision. But these functions are not reproducible, and a golden test over them would flake at a low rate — the worst kind of CI failure to diagnose.

Fix

The unseen contributions are independent of everything else, so their order is arbitrary but must be fixed. Either sort by key before summing, or accumulate into a BTreeMap, or collect to a Vec and sort. Sorting by magnitude would additionally reduce the rounding error, though at 7 ULP that is not the motivation.

Same class, separate symptom

src/history.rs:1804 — for (agent, batch) in &priors iterates a HashMap<Index, CompetitorConfig> and returns on the first conflict found. Measured over 40 processes on identical input with a conflict introduced across two batches:

8x competitor: 0   6x competitor: 5   3x competitor: 6   3x competitor: 2
3x competitor: 14  3x competitor: 1   2x competitor: 9   2x competitor: 8  ...

15 different competitors named across 40 runs. The error is correctly raised every time; only which competitor it blames is random. Low severity, but it makes the error value unreproducible and the message misleading — a user fixing "competitor 0" may find the next run blames competitor 5.

Not a defect, but worth a doc note

learning_curves() and filtered_learning_curves() return HashMaps whose contents are deterministic (verified across 50 processes), but a caller who iterates one and sums floats inherits the same per-process order randomness. KeyTable::keys() (src/key_table.rs:66) likewise iterates the forward HashMap rather than the dense reverse Vec, so it yields keys in random order — no internal caller, but it is public.

Found by a floating-point audit, 2026-09-09. Verified independently.

`src/history.rs:2317-2320` and `:2435-2440`. `ResolvedTerms::unseen` (`:264`) is a `HashMap<String, f64>`, built at `:1058`, and three float reductions iterate it: ```rust // distribution(), :2317 + resolved.unseen.values().map(|c| c * c * prior_var).sum::<f64>() // expected_variance_reduction(), :2435 and :2440 let cross_unseen: f64 = target.unseen.iter().map(...).sum(); let self_unseen: f64 = matchup.unseen.values().map(|c| c * c * prior_var).sum(); ``` Rust's default hasher is seeded per process, so the addition order — and therefore the sum — changes run to run. Addition is not associative. ## Measured Same binary, same input, separate processes. Reproduced independently at 40 runs (two distinct bit patterns, 24/16); the original sweep at 60 runs: ``` 60 scored_digest bda4ea80ee78f53a <- converged posteriors: stable 60 scored_le c08ee607b2488c1d <- log_evidence: stable 57 posterior_of sigma=4051f46fa4bb18dd 3 posterior_of sigma=4051f46fa4bb18dc <- 2 distinct bit patterns 25 evr 408972e942462c2f 8.14363895939086319231e2 19 evr 408972e942462c2d 8.14363895939086091857e2 13 evr 408972e942462c31 8.14363895939086546605e2 2 evr 408972e942462c2b 8.14363895939085864484e2 1 evr 408972e942462c32 8.14363895939086660292e2 <- 5 values, ~7 ULP ``` ## Mechanism isolated by falsification Same map, same size, same code path — determinism returns exactly when the terms become **equal** and summation becomes order-independent: | unseen keys | coefficients | distinct results over 60 processes | |---|---|---| | 1, 2, 3, 4, 8 | distinct | 1 (deterministic) | | 12 | distinct | 2 | | 24 | distinct | 2 | | 3, 8, 12 | **all 1.0** | **1 (deterministic)** | That rules out any other per-process variation. Reproduced identically in debug and release, with `rayon` on and off. ## Scope Reachable via `History::posterior_of`, `posterior_of_at`, `Joint::posterior_of`, `Joint::posterior_of_at`, `Joint::expected_variance_reduction` — whenever `UnknownKeys::Prior` is set and three or more unknown keys with unequal coefficients are passed. Magnitude is 1–7 ULP, so it will not change a decision. But these functions are **not reproducible**, and a golden test over them would flake at a low rate — the worst kind of CI failure to diagnose. ## Fix The unseen contributions are independent of everything else, so their order is arbitrary but must be *fixed*. Either sort by key before summing, or accumulate into a `BTreeMap`, or collect to a `Vec` and sort. Sorting by magnitude would additionally reduce the rounding error, though at 7 ULP that is not the motivation. ## Same class, separate symptom `src/history.rs:1804` — `for (agent, batch) in &priors` iterates a `HashMap<Index, CompetitorConfig>` and returns on the **first** conflict found. Measured over 40 processes on identical input with a conflict introduced across two batches: ``` 8x competitor: 0 6x competitor: 5 3x competitor: 6 3x competitor: 2 3x competitor: 14 3x competitor: 1 2x competitor: 9 2x competitor: 8 ... ``` 15 different competitors named across 40 runs. The error is correctly *raised* every time; only which competitor it blames is random. Low severity, but it makes the error value unreproducible and the message misleading — a user fixing "competitor 0" may find the next run blames competitor 5. ## Not a defect, but worth a doc note `learning_curves()` and `filtered_learning_curves()` return `HashMap`s whose *contents* are deterministic (verified across 50 processes), but a caller who iterates one and sums floats inherits the same per-process order randomness. `KeyTable::keys()` (`src/key_table.rs:66`) likewise iterates the forward `HashMap` rather than the dense reverse `Vec`, so it yields keys in random order — no internal caller, but it is public. Found by a floating-point audit, 2026-09-09. Verified independently.

logaritmisk referenced this issue

2026-09-09 14:53:27 +00:00

`tests/determinism.rs` never exercises the parallel sweep it exists to prove #64

logaritmisk added the bug numerics labels 2026-09-09 14:54:06 +00:00

logaritmisk referenced this issue from a commit

2026-09-09 16:10:52 +00:00

fix: make posterior_of reproducible across processes

logaritmisk closed this issue

2026-09-09 16:10:52 +00:00

logaritmisk referenced this issue from a commit

2026-09-09 16:10:52 +00:00

test: make the determinism test exercise the parallel sweep