feat: add Sankoff parsimony model and directory locking

Implements a calibrated 16-state Sankoff substitution cost matrix and CLI pipeline for evolutionary distance computation, including empirical calibration via saturation-filtered SNP counts. Refactors the sibling scanning stage to use batched transforms for improved synchronization efficiency. Introduces an OS-level advisory directory lock across all index-modifying commands to prevent concurrent write corruption. Updates dependencies and exposes new Sankoff utilities in the public API.

docmd/theory/evolutionary_distances.md

@@ -379,84 +379,156 @@ per pair loses. Not yet run.
 
 ### A concrete Sankoff cost matrix for the 16-state alphabet
 
+**Why `c_gl` and `c_ctx` are the same constant, not two.** An earlier version
+of this design used two independent parameters: `c_gl` for an ordinary
+gain/loss step between two nonempty states (e.g. `{A,C}->{A}`), and `c_ctx`
+only for a state collapsing all the way to `∅`. That distinction doesn't
+survive contact with what the index actually observes. Presence/absence
+tracking never sees "the flanks" separately from "the centre" — it sees
+whole, distinct, homologous 31-mers (two 31-mers sharing >=30 bases are
+homologous by construction, though orthology vs. paralogy is undecidable
+from that alone — settled earlier, see "Locus eligibility"). Losing the `A`
+form of a family while `C` remains (`{A,C}->{C}`) and losing the last
+remaining form (`{A}->∅`) are the *same kind of event*: a specific, complete
+31-mer that used to be observed no longer is, because at least one of its
+31 positions mutated. There is no separate "gain/loss of a still-recognised
+allele" mechanism distinct from "loss of context" — both are exactly the
+event `c_ctx(p)` (below) already computes the expected cost of. So: one
+constant, `c_ctx`, used everywhere a member is gained or lost — `X -> ∅`
+included, and (see next point) not even as a special case there.
+
 **Set-edit-distance formula.** For two states `X, Y ⊆ {A,C,G,T}`, split into
 `seulement_X = X \ Y` (size `a`) and `seulement_Y = Y \ X` (size `b`).
 Elements present in both cost nothing. Pair up to `min(a,b)` of the
 remaining elements as **substitutions** (cheaper than treating them as an
-unrelated loss + gain whenever `c_sub < 2*c_gl`, which any sane parameter
+unrelated loss + gain whenever `c_sub < 2*c_ctx`, which any sane parameter
 choice satisfies); whatever is left over after pairing is a pure
 **gain/loss**:
 
 ```
-cost(X,Y) = min(a,b)*c_sub + |a-b|*c_gl
+cost(X,Y) = min(a,b)*c_sub + |a-b|*c_ctx
 ```
 
-Worked examples (`c_sub = c_gl = 1`): `{A}->{C}` = 1 (one substitution);
+Worked examples (`c_sub = c_ctx = 1`): `{A}->{C}` = 1 (one substitution);
 `{A}->{A,C}` = 1 (one gain, no substitution pair available since nothing is
 only-in-Y that matches an only-in-X element after the shared `A` is
 excluded); `{A,C}->{G,T}` = 2 (two substitution pairs, `A/C` vs `G/T`, both
 same-size sets share nothing); `{A,C,G}->{A,C,T}` = 1 (`G`/`T` is the only
 mismatched pair, `A,C` shared).
 
-**`∅` is a flat-cost special case, not a instance of the general formula.**
-Applying the formula naively to e.g. `{A,C,G} -> ∅` would charge `3*c_gl`
-(three independent losses). Reject that: total context disappearance
-(flank-breaking, or true structural loss) is plausibly **one** event, not
-`|X|` of them, so:
+**`∅` is *not* a flat-cost special case — a corrected position, reversing
+an earlier draft of this design.** An earlier version charged
+`cost(X, ∅) = c_ctx` flat, independent of `|X|`, on the grounds that every
+member of a family shares identical flanking sequence, so one mutation
+breaking that context should unrecognise all of them at once rather than
+`|X|` separate times. That argument doesn't survive comparison with how the
+rest of the matrix already works: `{A,C,T} -> {A}` (losing two members,
+one remaining) is charged `2*c_ctx` via the general formula above, with no
+equivalent "maybe it was one shared event" discount — nothing distinguishes
+that case from `{A,C} -> ∅` (losing two members, none remaining) other than
+which state happens to be the target. Singling out `∅` for special
+treatment was arbitrary, not principled: the uncertainty about "one event
+or several" is identical in both cases, and the model already resolves it
+uniformly elsewhere by simply counting elements. So `∅` is now an ordinary
+node in the state graph below like any other, with no patch applied
+afterwards:
 
 ```
-cost(X, ∅) = cost(∅, X) = c_ctx   (constant, independent of |X|)
+cost(X, ∅) = cost(∅, X) = |X| * c_ctx   (via |X| single-element graph edges)
 cost(∅, ∅) = 0
 ```
 
-`c_ctx` should not be guessed — it is exactly the value already derived for
-the `>1` bucket in "Context, detectability, and a 3-way ordinal distance per
-pair" above (`(2m*p_hat)/(1-(1-p_hat)^(2m)) + p_hat`), reused rather than
-invented.
+**Computing `c_ctx`.** Not guessed — built from the value already derived
+for the `>1` bucket in "Context, detectability, and a 3-way ordinal
+distance per pair" above:
+
+```
+c_ctx(p) = mean_sub_cost * [(2m*p) / (1 - (1-p)^(2m)) + p]
+```
+
+`p` is `p_hat`, the calibrated per-site mutation probability (see
+`calibrate_p_hat` / the "Experiment" section below); `m = (k-1)/2`, the
+flank length on each side of the centre (`k=31` → `m=15`). Read the
+bracketed term, well, term by term: `(1-p)^(2m)` is the probability that
+*none* of the `2m` flanking positions mutated, so `1-(1-p)^(2m)` is the
+probability that *at least one* did — i.e. the probability that this
+specific 31-mer stops being observable at all, which is exactly the event
+`c_ctx` prices. `2m*p` is the unconditional expected mutation count over
+those `2m` positions. Their ratio is the conditional expectation
+`E[mutations | at least one occurred]` — provable directly: for any
+nonnegative integer random variable `X`, `X · 1{X>=1} = X` always (both
+sides are `0` when `X=0`, both are `X` otherwise), so
+`E[X | X>=1] = E[X·1{X>=1}]/P(X>=1) = E[X]/P(X>=1)`, which is
+`(2m*p)/(1-(1-p)^(2m))` here. The trailing `+p` adds the (much smaller,
+first-order, not itself a conditional expectation) marginal contribution
+of the centre position's own mutation probability.
+
+**The `mean_sub_cost` multiplier — a correction, not part of the original
+derivation.** The bracketed term is a *count* of expected mutations, not a
+cost — an earlier version used it directly as `c_ctx`, implicitly pricing
+every one of those mutations at a flat `1` regardless of type. That stopped
+being defensible once substitution costs were calibrated per base-pair
+category (`substitution_costs_from_tally`): transitions are markedly more
+frequent than transversions in this project's own data (roughly 5-6x, e.g.
+130k vs. 20-28k observed instances per category in the eubacteria run
+below) and correspondingly cheaper, so "one mutation" isn't worth a flat
+unit. `mean_sub_cost` (`mean_substitution_cost`) is the empirical average
+substitution cost, weighted by each category's observed frequency — it
+converts the bracketed term's expected mutation *count* into an actual
+expected *cost*. Concretely, in the eubacteria run: the bracketed term
+alone is `~1.17` (close to `1`, i.e. context loss is *usually* attributable
+to a single mutation, since `p_hat` is small); `mean_sub_cost ~= 1.48`
+(pulled up from the cheapest transition cost of `1.0` by the substantial
+minority of transversions in the mix); final `c_ctx ~= 1.73` — sensibly
+between the cheapest transition (`~1.0`) and the transversion costs
+(`~2.5`-`2.9`), rather than coinciding almost exactly with the cheapest
+transition purely by construction accident, which is what the unmultiplied
+version did and which is what exposed this gap in the first place.
 
 **Better construction method: shortest path in a small state graph, not the
-closed-form formula directly.** Build a graph on the 16 states with two edge
-types — substitution edges between same-cardinality sets differing by one
-element (weight `c_sub`, or `c_ts`/`c_tv` if split further below), and
-gain/loss edges between sets whose cardinality differs by one (weight
-`c_gl`) — then define `cost(X,Y)` as shortest-path distance in that graph,
-precomputed once (16 nodes, trivial) into a dense 16x16 matrix before
-feeding it to Sankoff/TNT. Verified equivalent to the closed-form formula
-above in the uniform-cost case (checked by hand on `{A}->{C,G,T}`: both give
-`c_sub + 2*c_gl`). The graph construction is not just a reformulation for
+closed-form formula directly.** Build a graph on all 16 states, `∅`
+included, with two edge types — substitution edges between same-cardinality
+sets differing by one element (weight `c_sub`, or `c_ts`/`c_tv` if split
+further below), and gain/loss edges between sets whose cardinality differs
+by one (weight `c_ctx`, per the unification above; `∅` connects to each
+singleton state this way, being a strict subset of it) — then define
+`cost(X,Y)` as shortest-path distance in that graph, uniformly for every
+`X,Y` including `∅`, precomputed once (16 nodes, trivial) into a dense
+16x16 matrix before feeding it to Sankoff/TNT. Verified equivalent to the
+closed-form formula above in the
+uniform-cost case (checked by hand on `{A}->{C,G,T}`: both give
+`c_sub + 2*c_ctx`). The graph construction is not just a reformulation for
 its own sake: it is the version that generalises correctly once substitution
 costs stop being uniform (next point) — the closed-form's `min(a,b)`
 counting silently assumes *any* pairing costs the same, which breaks the
 moment `c_sub` depends on which two bases are involved.
 
-**Transition/transversion refinement.** Split `c_sub` into `c_ts` (A<->G or
-C<->T) and `c_tv` (the other four pairs) — already well-defined in canonical
-space (see "Canonical invariance" above: a transition maps to a transition,
-a transversion to a transversion, regardless of orientation). This turns
+**Substitution refinement — implemented as the full 6-category symmetric
+matrix, not just Ts/Tv.** `c_sub` was originally going to split into just
+`c_ts` (A<->G or C<->T) and `c_tv` (the other four pairs) — already
+well-defined in canonical space (see "Canonical invariance" above: a
+transition maps to a transition, a transversion to a transversion,
+regardless of orientation). A symmetric cost matrix allows finer
+resolution than that 2-category split, though: each of the 6 distinct
+undirected base pairs (AC, AG, AT, CG, CT, GT) can be costed
+independently, with no grouping at all — `SankoffWeights::sub_cost` is a
+full `[[f64;4];4]` table (`SankoffWeights::ts_tv` remains as a convenience
+constructor for the coarser 2-category case, if ever wanted). This turns
 "pick `min(a,b)` substitution pairs" into a genuine (tiny, <=4 elements per
 side, trivially enumerable) minimum-cost bipartite matching problem instead
 of a plain count — the state-graph shortest-path construction handles this
-automatically, no separate logic needed. Calibration is not a guess either:
-the "Sufficient statistic: 4x4 base-pair tally" section below already plans
-to collect the joint `(centre_i, centre_j)` distribution over resolved (`0`
-or `1`) sites — that tally directly gives the empirical Ts/Tv ratio, from
-which `c_ts`/`c_tv` follow (e.g. `cost ∝ -log(observed rate)`, the standard
-generalised-parsimony step-weighting heuristic), reusing a statistic already
-planned rather than adding a new one.
-
-**`gamma`/`mu` (i.e. `c_gl`/`c_sub`) sensitivity sweep before calibration.**
-No strong prior on whether gain/loss events should cost more or less than a
-point substitution — duplication/deletion rates are not a priori equal to
-point-mutation rates, but the direction isn't obvious, and setting it too
-high risks the parsimony search *eliminating* exactly the
-heterozygosity/paralogy signal the design is meant to tolerate (see
-"Heterozygosity, ploidy..." below). Cheap first step: run the topology at a
-handful of ratios (`0.5, 1, 2, 5`) and check whether it's stable — a robust
-topology across that range is far more trustworthy than one built on a
-single, unvalidated guess. Empirical calibration of `c_gl` from the
-family-size distribution already available (`sibling_annex_stats`) is a
-natural follow-up once the sensitivity sweep shows the topology is worth
-refining further.
+automatically, no separate logic needed. Calibration is not a guess
+either: `KmerIndex::base_pair_tally` collects the pooled `(centre_i,
+centre_j)` distribution over resolved (SNP) sites — restricted to the same
+ratio-ceiling-included genome pairs as `p_hat`'s own calibration, for the
+same saturation-exclusion reason — and `substitution_costs_from_tally`
+turns that into `cost(a,b) = -ln(rate(a,b))`, normalised so the most
+frequent category costs `1.0` (the standard generalised-parsimony
+step-weighting heuristic, generalised from 2 categories to 6). A second
+full pass over the annex is required for this (see `base_pair_tally`'s own
+docs for why it can't share `raw_snp_distance`'s single pass) — both now
+share one traversal helper, `scan_family_pairs`, rather than duplicating
+the per-family reconciliation logic.
 
 **Caveat carried over from the `D_F = min(a,b)` rejection earlier:** this
 cost matrix is only valid as **Sankoff step-cost input**, re-evaluated for
@@ -469,7 +541,7 @@ specific genomes.
 
 **Feasibility confirmed:** TNT's `costs` command accepts custom step
 matrices for multistate characters, so this whole construction (16x16
-matrix derived from the state graph, `c_ts`/`c_tv`/`c_gl`/`c_ctx` as
+matrix derived from the state graph, `c_ts`/`c_tv`/`c_ctx` as the three
 tunable parameters) is directly usable there — no new tooling required
 before testing it.
 
@@ -491,7 +563,9 @@ emits a complete TNT script (`xread` matrix + `smatrix` step-matrix + `hold`/
 `write_raw_snp_distance_csv` to emit raw counts, `c_ctx` was approximated as
 the unweighted mean of the pairwise `snp/(snp+shared)` ratios already
 present in `rawsnp.csv`, restricted to the relevant taxon subset. Used
-values: `mu = gamma = 10`, `c_ctx = 18` (ratio `c_ctx/mu = 1.8`, consistent
+values: `mu = gamma = 10`, `c_ctx = 18` (this run predates the `c_gl`/`c_ctx`
+unification above — `gamma` here is what `c_gl` was called before it turned
+out to just be `c_ctx`; ratio `c_ctx/mu = 1.8`, consistent
 with the observed pairwise ratios for genomes within Enterobacteriaceae/
 Eubacteria, which cluster around 1.5-2 and barely move as the taxon set
 widens — the context signal is stable, not sensitive to which subset is