obikmer/docmd/theory/evolutionary_distances.md

Central-position SNP distance (discussion)

Not implemented. Design discussion for a substitution-rate estimator that observes SNPs directly from paired-genome k-mer comparison, as an alternative to Mash's Poisson-Jaccard inference (see obicompactvec for the implemented Jaccard/Mash distances).

Motivation

Primary intent: restrict the comparison to what is actually comparable. Mash's Jaccard is computed over the union of both genomes' k-mer content: anything not identically shared is folded into a single undifferentiated mass, whether the cause is a point substitution, a genuinely absent homologous region (lineage-specific content, gene-family expansion, HGT, genome-size asymmetry), or a diverged paralogous copy. The model then back-infers a single mutation rate from that mass, silently attributing non-homology to mutation. The central-SNP approach instead conditions every comparison on local, positive evidence of homology: a locus only enters the statistic if its 2m flanking bases (m = (k-1)/2) are found intact in both genomes — genuinely absent or non-homologous content is excluded from the comparison entirely (neither numerator nor denominator), rather than silently counted as divergence. This is a conditioning on comparability, not just a richer summary statistic — see "Statistic and correspondence with shared" below for how it plays out against genome-size asymmetry and diverged gene families, and "Heterozygosity, ploidy, and consensus-assembly inputs" for the corresponding paralogy/heterozygosity filter.

Secondary benefit: access to the substitution's nature. Because the central base of an odd-k window is directly observable once the flanks are confirmed conserved, this also yields more than a rate — the transition/transversion split — enabling classical corrected distances (Jukes-Cantor, Kimura 2-parameter, LogDet) that a single Jaccard scalar cannot support.

Statistic and correspondence with shared

A genomic position p is covered by k overlapping k-mer windows. Requiring the substitution to sit at the window's center makes exactly one window per SNP eligible — a 1:1 correspondence between SNP and center-neighbor k-mer pair, avoiding the ~k-fold overcount of an any-position neighbor search.

A locus with a fully conserved k-window (flanks and center) is an exact-shared k-mer at that locus; a locus with conserved flanks but a substituted center is a "central SNP". Both count each locus exactly once, in matching units:

p_hat[i,j] = SNP[i,j] / (SNP[i,j] + shared[i,j])

p_hat is P(center substituted | 2m flanks conserved). shared[i,j] here is not the general-purpose shared_kmers matrix used by Jaccard/Mash (--shared-kmers, BitPartials::partial_jaccard / CountPartials::partial_threshold_jaccard) — that matrix counts raw k-mer identity with no per-genome copy-number constraint, whereas p_hat's denominator applies the eligibility rule defined below (raw or paralogy-filtered). Both SNP and shared are accumulated by the same sweep, from the same per-locus candidate set (source k-mer + 3 variants), under the same eligibility rule — see "Locus eligibility" below, and "Heterozygosity, ploidy, and consensus-assembly inputs" for why the copy-number constraint matters and what it costs.

Canonical invariance: for odd k, the central position maps to itself under reverse-complement (m -> k-1-m = m, base complemented). A transition maps to a transition, a transversion to a transversion — the transition/transversion split is well-defined in canonical space.

Locus eligibility: raw definition vs. paralogy filter

For each k-mer x observed in genome A (source, one MPHF slot; the 3 central-position variants generated as in the sweep below): check whether A's locus (flanks fixed) is resolvable in genome B under one of the 4 central forms.

Raw / no model. The locus counts in the denominator iff at least one of the 4 forms is present in B; it counts in the numerator iff the form found in B differs from A's own. No constraint on A's or B's own copy number at this locus. Open question, not resolved: what if more than one of the 4 forms is present in B simultaneously (ambiguous target — count once arbitrarily, count all, or drop)? The stringent filter below sidesteps the question by construction rather than answering it.

Stringent / paralogy-aware. The locus counts only if exactly one of the 4 forms is present in A and exactly one is present in B (count == 1 at that slot too, when a count index is available, to also exclude same-allele duplicates that presence alone cannot see). This drops the raw definition's ambiguous-B case automatically, at the cost of also dropping heterozygous sites indiscriminately alongside true duplications (see "Heterozygosity, ploidy, and consensus-assembly inputs" below).

Rejected: parsimony-based multiset pairing for multiplicity > 1. Rather than dropping ambiguous loci, pair identical alleles between A and B first (0-mutation explanation preferred), then take min(unmatched_A, unmatched_B) as inferred SNP pairs. Rejected on two grounds: (1) circularity — selecting pairs by minimal apparent divergence, then measuring divergence on those same pairs, deflates the estimate by construction, not a neutral heuristic; (2) the discriminating signal is a single base among 4 possible values, and the flanks are already guaranteed identical for every candidate by construction (that is how the locus was selected) — no information remains in a k-mer window to tell which copy in B truly corresponds to which copy in A once multiplicity > 1 on either side. Any pairing rule invents a correspondence the data cannot support. Multiplicity > 1 is treated as non-identifiable, not as a puzzle to solve with a heuristic.

Heterozygosity, ploidy, and consensus-assembly inputs

A within-genome multiplicity signal (more than one of the 4 central forms present at a locus) is produced identically by two distinct causes: paralogous duplication and diploid/polyploid heterozygosity. K-mer data alone cannot distinguish them. The one real discriminator is sequencing depth (heterozygous site: total depth of the present forms ~= the genome's single-copy average; duplication: ~2x or more) — but that signal only exists if genome "counts" are raw-read depth (FASTQ input), not occurrence counts in an assembled FASTA, where per-locus depth is not preserved.

Magnitude is taxon- and mating-system-dependent, not universal. Heterozygosity density: mammals ~1 site / 1-1.5 kb (~0.1%); highly outcrossing plants (maize, poplar) reported an order of magnitude higher (~1%); self-fertilising plants (Arabidopsis thaliana) near zero — but with a documented failure mode where segmental duplication masquerades as "pseudo-heterozygosity"; fungi split between haploid vegetative stages (non-issue) and dikaryotic Basidiomycetes, where two long-diverged haploid nuclei coexist without fusing. The estimator's target use case (closely related genomes, k=31) is exactly where the stringent filter above costs the least for low-heterozygosity taxa and the most for outcrossing/dikaryotic ones — no universal threshold; this is a scope caveat to document, not a problem to solve generically.

Why assembled-consensus inputs don't make measured distances wrong. Phylogenetic inputs are near-universally assemblies, not raw reads, and assemblers collapse heterozygous sites to one consensus allele per position — effectively an arbitrary, largely uncorrelated-between-assemblies choice at each het site. This does not inject unbounded noise: standard population genetics gives d_xy = d_a + (pi_A + pi_B)/2 — the expected pairwise difference between a random allele of population A and a random allele of population B equals the net (fixed) divergence d_a plus the average of the two populations' own within-population diversity pi. Consensus flattening realises exactly this random-allele draw, so the measured genome-to-genome distance is a d_xy-like quantity, not d_a — inflated by heterozygosity by a well-characterised additive term, not distorted unpredictably. The term is negligible when pi << d_xy (the common case for cross-species comparisons), and becomes material precisely in the two cases already flagged above: very closely related genomes (this estimator's explicit target) and highly heterozygous outcrossing organisms, where pi and d_xy are the same order of magnitude.

Caveat: this assumes the flattening is uncorrelated with the phylogenetic signal — plausible for de novo assembly, not guaranteed for reference-guided assembly biased toward one allele (e.g. the reference's) at each het site, which would turn the noise term into a systematic bias toward the reference lineage. Not evaluated here.

Forward-looking implication, not part of the current design. The multiplicity > 1 signal discarded by the stringent filter is a crude per-genome proxy for pi (under low background paralogy). If a pi_hat per genome were tallied alongside SnpTally, a d_a correction (p_hat - mean(pi_hat_i, pi_hat_j)/2, roughly) could recover an estimate closer to net divergence instead of d_xy — a possible extension, not scoped here.

Sufficient statistic: 4x4 base-pair tally

Tabulating the joint distribution of (center_i, center_j) over conserved-flank loci, per genome pair, is sufficient for every downstream correction:

Estimator	Input	Formula
Raw p-distance	total off-diagonal / total	p = SNP / (SNP + shared)
Jukes-Cantor	p	d = -3/4 * ln(1 - 4p/3)
Kimura 2-parameter	transition rate P, transversion rate Q	d = 1/2 ln(1/(1-2P-Q)) + 1/4 ln(1/(1-2Q))
LogDet/paralinear	full 4x4 + base-composition margins	d ~= -1/4 ln det(F), robust to non-stationary base composition

JC/K2P need only the total and the transition/transversion split (the diagonal collapses to a single "shared" total). LogDet needs the full 4x4, already populated at no extra cost (see Step 1/2 below).

Memory for the 4x4 tally: n^2 * 16 counters. Trivial for the project's genome-scale use case (tens to hundreds of genomes); ~13 GB at n=10^4 — outside scope but worth flagging if n grows.

Biases (properties of the estimator, not defects)

1. Conserved-flank ascertainment bias. Only SNPs with intact 2m-base flanks are visible; window-intact probability decays as (1-p)^{2m}. For k=31 (2m=30): 0.74 at p=1%, 0.21 at p=5%, 0.04 at p=10%. This estimator targets closely related genomes. Under rate heterogeneity across sites (universal in practice), conserved flanks correlate with slow centers, so p_hat underestimates the genome-wide average rate — it specifically estimates the substitution rate of conserved regions. Two distinct factors are at play here, not one: P(centre of a given window is a SNP) = p exactly, independent of k — a direct restatement of the raw per-site rate via the bijective window<->centre-position correspondence (Statistic section above), not a k-dependent quantity. (1-p)^{2m} is the separate, genuinely k-dependent ascertainment factor (are the flanks also intact). The two multiply: P(usable window showing a central SNP) = p * (1-p)^{2m} — e.g. at p=1/31 (~3.2%), k=31: p * (1-p)^30 ~= 0.0323 * 0.374 ~= 1.2%, i.e. about 1 window in 83, not 1 in 31 (which is only the centre-mutated fraction, before requiring intact flanks).
2. Bias toward isolated SNPs. Two SNPs within k of each other disqualify each other's flanks. Hypervariable regions are invisible by construction.
3. Indels are invisible. A frameshift destroys k-mer matches in a block; this channel captures substitutions only. Indel divergence shows up as lost shared k-mers (lower Jaccard/Mash), not as SNP signal.
4. k-dependent specificity. "A k-mer match implies common ancestry" is quantitative. For a 3 Gbp genome, expected random flank-30 collisions (k=31): (3e9)^2 / 4^30 ~= 8 — negligible. At k=21: (3e9)^2 / 4^20 ~= 2e6 — no longer negligible. k=31 is safe; k<=21 is marginal to unreliable for large genomes. The large k that guarantees homology is the same k that shrinks the detectable-divergence window — an inherent tension.

Implementation: avoid materializing a de Bruijn graph

A central-SNP pair is topologically a simple bubble in the colored de Bruijn graph (source/sink k-mer shared, two length-k branches differing only at the midpoint). Classical bubble-calling (Cortex/discoSNP-style) finds these, but requires the graph — nodes plus adjacency for ~10^9 colored k-mers — resident in memory. Rejected: prohibitive RAM for this project's scale.

A naive per-pair generalisation of variant lookup across n genomes (query each non-shared k-mer's 3 central variants against every counterpart genome's index) costs O(n^2 . N . 3) random lookups, with the same k-mer's 3 variants regenerated and requeried once per counterpart genome — pure redundant work. Rejected as the basis for an n-genome design.

Implementation: sequential per-partition sweep (no scratch, no graph)

KmerIndex::distance() already opens every partition's presence_store/ count_store simultaneously, memory-mapped, into one LayeredStore (distance.rs:73-77). "Querying another partition" is therefore not a new I/O pattern to design — it is the same O(1) MPHF+evidence lookup the query command already performs at scale. This lets the SNP tally be computed with no scratch files and no auxiliary graph, by sweeping partitions once each as a source:

1. For source partition p, enumerate its distinct k-mers (one per MPHF slot; each already carries its full multi-genome presence/count vector — no need to explode per (k-mer, genome) occurrence).
2. For each, generate the 3 central-substitution variants and canonicalise each independently (min(kmer, revcomp), exactly as any normal query) — this avoids the orientation edge case a masked-flank grouping would have (a substitution that flips canonical orientation is handled correctly because each variant is canonicalised on its own, not inferred from a fixed-orientation flank key).
3. Compute each variant's target partition q via its minimizer; batch/sort the partition's outgoing variant queries by q for locality.
4. Look up each variant in q's already-mmap'd MPHF+evidence; on a hit, combine the source's presence vector (base a) with the variant's presence vector (base b): for every i carrying a and j carrying b, tally[i,j][a,b] += 1.

Deduplication needs no persisted state. Sweeping partitions in a fixed order p = 0, 1, ..., P-1 and only acting on a variant when its target partition q >= p guarantees each unordered SNP pair is counted exactly once: a pair with q < p was already resolved earlier, when q was itself the source partition and p (being >= q) was a valid forward target. No cross-partition flag array is needed — the sweep order is the deduplication rule. Within the same partition (q == p), a lightweight transient tie-break suffices: either a #slots(p)-bit scratch flag reset per partition, or simply comparing the two k-mers' raw u64 encodings and only counting when kmer_source < kmer_variant — no storage at all.

This is a distinct computation stage, not a partial_* in the existing additive-by-partition sense: step 3-4 read across partition boundaries by construction, unlike the row-local partial_jaccard/partial_threshold_jaccard primitives. But it requires no new index files, permanent or scratch: unitigs.bin, mphf.bin, evidence.bin, and the presence/count columns are read as-is, and the only extra memory is the current partition's small outgoing-query batch (#kmers(p) * 3, released once p is done) plus the persistent tally accumulator (n^2 * 16 counters, see above).

Outer loop (over source partitions p) must stay sequential. Two independent reasons, not just one: (a) memory — the bounded-footprint claim above only holds with one partition's outgoing-query batch in flight; running T source partitions concurrently multiplies that batch by T, exactly the blowup the design avoids; (b) correctness — the q >= p deduplication rule requires partitions to be claimed as sources in a fixed order; running p1 < p2 concurrently gives no guarantee p1 has finished claiming its q >= p1 targets before p2 starts claiming its own, breaking the "counted exactly once" property.

Inner loop (target-partition lookups for a fixed p) parallelises safely. Each lookup is an O(1) read against an already-mmap'd structure, independent of the others, with no growing allocation — no memory blowup, no ordering dependency between different q. The only shared mutable state is tally; give each worker thread a thread-local partial tally (fixed n^2 * 16 size, independent of partition size) and merge into the global tally once p's inner loop completes — the same reduce-then-merge pattern Rayon already uses elsewhere in this codebase to open partitions in parallel. Extra memory: #threads * n^2 * 16, negligible (512 MB at n1000, 32 threads) and unrelated to partition size.

Cost: 3 * N_distinct MPHF lookups total across the whole index (each partition swept once as source) — the same order of magnitude and the same operation as running query over the index's entire k-mer content against itself, three times. This is the tool's already-optimized regime, not a new I/O profile to validate.

Cheaper: subsampling

Since the target is a ratio, restricting the source-partition sweep to a bottom-s hash sketch (only enumerate k-mers with hash < threshold as sources) divides the lookup count by the sampling factor without biasing p_hat. Mash-like tradeoff: rate estimated from a sample, not the full k-mer set.

Recommendation

Sequential per-partition sweep (Route D): reuse the already-mmap'd per-partition MPHF/evidence/presence structures for O(1) variant lookups, dedup via the fixed sweep-order rule (q >= p, plus an in-partition tie-break), no scratch files, no graph materialisation. Both the SNP (off-diagonal) and shared (diagonal) counts are accumulated by this same sweep, under the locus-eligibility rule chosen (raw or paralogy-filtered) — not reused from the general-purpose shared_kmers matrix, whose raw-identity definition does not apply the same copy-number constraint. Distances (p, JC, K2P, LogDet) as finalisations of the resulting 4x4 tally, mirroring the partial_* -> *_dist_matrix pattern used for Jaccard/Mash/Bray-Curtis/etc.

Detailed implementation plan

Grounded in the current codebase. File/type references are anchors, not prescriptions; adjust to reality when implementing.

Step 0 — new low-level primitives (obikseq)

Two helpers do not yet exist and are prerequisites:

1. Central neighbours. CanonicalKmerOf<L> already exposes left_canonical_neighbors() / right_canonical_neighbors() (obikseq/src/kmer.rs), each returning the 4 canonicalised neighbours at an end position. Add central_canonical_neighbors() returning the 4 variants at position m = (k-1)/2 (each independently canonicalised via .canonical()). The 3 that differ from the source are the query variants; skip the identity. Building on nucleotide(i) / the raw 2-bit layout keeps it O(1).
2. Lone-k-mer minimiser. Routing a synthetic variant to its partition needs its minimiser, but RollingStat (obiskbuilder/src/rolling_stat.rs) only computes minimisers incrementally along a sequence. Add a standalone minimizer(kmer) -> Minimizer that scans the k-m+1 m-mer windows (PackedSeq::mmer, obikseq/src/packed_seq.rs), canonicalises each, and takes the min by seq_hash() — the same selection RollingStat performs, evaluated once. Partition index is then (minimizer.seq_hash() & (n_partitions - 1)) as usize, exactly as QueryBatch::from_records (obikmer/src/cmd/query.rs:142); n_partitions is a power of two so the mask is valid.

Step 1 — the tally accumulator (obikindex)

A SnpTally holding, per genome pair, the 4x4 joint count of central bases: n * n * 4 * 4 u64 (or a packed lower-triangular form since it is symmetric). Provide merge(&mut self, other: &SnpTally) for the thread-local reduce, and accessors yielding, per pair (i,j): total off-diagonal (SNP), diagonal (shared, i.e. p_hat's denominator minus SNP), transition count P, transversion count Q. The diagonal is always populated — it is not an optional LogDet-only extra, since p_hat's denominator is no longer sourced from the external shared_kmers matrix (see "Locus eligibility" and "Statistic and correspondence with shared" above): the source k-mer's own presence/count vector, already in hand when it is enumerated, supplies the diagonal entry directly, at no extra lookup cost.

Step 2 — the sweep (obikindex, new snp.rs)

Mirror distance.rs: open the presence or count store per partition. But instead of a per-partition partial_*, run the sequential source sweep:

for p in 0..n_partitions:                       # OUTER — sequential
    open source partition p's layers (QueryLayer-style, obikpartitionner)
    enumerate distinct canonical k-mers of p (one per MPHF slot) with their
        presence/count vectors                   # column-major, as query stage 2
    par_iter over these source k-mers:           # INNER — rayon, thread-local tally
        apply eligibility rule to the source's own vector (raw: none;         # diagonal
            stringent: exactly one of the 4 forms present in each genome)     # gate
        for i in genomes eligible with source base a:
            for j in genomes eligible with source base a:
                thread_tally[i,j][a,a] += 1        # diagonal — no extra lookup
        for each of the 3 central variants:
            q = partition_of(variant)
            if q < p: continue                    # dedup: forward targets only
            if q == p and variant <= source.raw(): continue   # in-partition tie-break
            slot = layers[q].find_slot(variant)   # MphfLayer::find, mmap'd
            if hit:
                vb = variant presence/count vector
                apply eligibility rule to vb (as above)
                for i in eligible genomes with source base a:
                    for j in eligible genomes with variant base b:
                        thread_tally[i,j][a,b] += 1
    merge thread-local tallies into global SnpTally

The inner lookup is precisely QueryLayer::find_slot + col_value(g, slot) (obikpartitionner/src/query_layer.rs) — reuse or factor out that path rather than reimplementing MPHF access. Enumerating "all distinct k-mers of a partition with their vectors" is the dump/query stage-2 column-major scan already implemented in dump_layer.rs / query_partition_with; factor a reusable iterator if none fits.

presence_threshold applies exactly as elsewhere: a genome "carries base b" iff its count at that slot is >= presence_threshold (trivially >= 1 for presence indexes).

Open problem (unresolved, session end — not yet fully convinced)

The q >= p / tie-break dedup rule in Step 2's pseudocode above is flawed for the stringent (paralogy-filtered) eligibility rule: it only ever brings two family members into view at once (the source and one looked-up variant), never all four simultaneously, and which subset gets compared depends on partition sweep order. "Exactly one of the 4 forms present in genome A" is a whole-family property and cannot be decided correctly from a sequence of pairwise, order-dependent glimpses — the pseudocode above needs revision, not just the eligibility gate bolted onto it as written.

Direction discussed, not yet settled:

1. Every distinct source k-mer looks up all 3 variants unconditionally (drop the q < p skip entirely) so that every observed family member independently gathers all 4 vectors (its own + whichever of the 3 variants exist) at once — a whole-family, order-independent view, computed redundantly once per observed member. Same total lookup order of magnitude as already budgeted (3 * N_distinct), just organised differently (no lookup actually skipped, versus the original rule which skipped roughly half).
2. Tie-break after gathering, not before: only the member whose own canonical encoding is the smallest among the members actually observed (now known, since all were just looked up) writes to SnpTally; the others silently discard their redundant computation. Deterministic, order-independent — as a side effect this also removes the "outer loop must stay sequential" constraint from the cost/parallelism discussion above, since no step depends on partition processing order any more.
3. Proposed optimisation: precompute, once at index build time, a compact global (not per-genome) annex per MPHF slot — the count of other family members observed anywhere in the dataset (0-3). Slots with count 0 (majority under low divergence and few genomes, but see the scaling caveat below) need no cross-lookup at all: eligibility reduces to a local count == 1 check at that single slot, and only slots with count

   = 1 enter the 3-lookup sweep machinery above. Revised (see Step 2b below): minorant status is stored alongside the count after all, on 3 bits rather than 2 — it comes for free from the same lookups needed to count siblings, and storing it lets the sweep discard non-minorant slots without re-fetching anything.

Minorant/sibling-count relationship, worked out precisely. "Minorant" is a one-way implication from sibling count, not an equivalence: 0 siblings => minorant (trivially — with no other observed member, the k-mer is by definition the smallest of the observed set, itself alone), and its contrapositive not minorant => >= 1 sibling. The converse does not hold: being the minorant says nothing about sibling count — a minorant can have 0, 1, 2 or 3 siblings, all with larger encodings than itself. Consequence: this confirms, as a logical necessity rather than a heuristic, that a 0-sibling slot can always write its diagonal contribution with zero ambiguity and no lookup (it is unconditionally its own minorant) — but it gives no shortcut for the >= 1-sibling case, where minorant status still requires the actual comparison of gathered encodings; sibling count alone never determines it.

When to compute the annex, and cache invalidation. Sibling count is a property of the whole set of columns (genomes/groups) currently in the index, not of any single genome — it cannot be computed correctly at mono-genome build time (a family may gain siblings, or its minorant may change, once more genomes are merged in later). Computing it eagerly at every merge would also waste work on intermediate merged states nobody ever queries. Instead: compute it lazily, on first distance call against a given index, and persist the result alongside that index for subsequent calls — the same lazy-derived-cache pattern PersistentBitMatrix already uses for Columnar -> Packed. This requires no explicit invalidation for merge or filter (obikindex/src/merge.rs, obikmer/src/cmd/filter.rs): both only ever write to a fresh --output directory, never mutate an input index in place, so a re-merged/re-filtered index is simply a new state with no annex yet. select --in-place (select_layer.rs:139-235) is the exception: it aggregates genome columns into groups (Any/All/None/Sum/Min/ Max) by mutating the existing index's files without changing its location. It does not remove k-mer rows, but it can still change eligibility and sibling counts derived from those rows (e.g. a Sum over several single-copy genomes can read as multi-copy at the group level). Because it mutates in place, select --in-place must explicitly invalidate (delete or mark stale) any cached sibling-count annex for that index — the one operation in the current pipeline where this doesn't happen for free.

Not yet convinced this is the right shape, and Step 2's pseudocode above has not been rewritten to match — flagged for the next pass rather than resolved here.

Step 2b — sibling-count / minorant annex (consolidated plan)

Scope: only the precursor annex (sibling count 0-3 per slot, minorant decided on demand) — not the SNP tally itself, whose Step 2 sweep remains unresolved above. This piece is simpler than the sweep, because it writes to an independent per-slot value, not a shared cross-k-mer accumulator, so it needs no dedup/ownership logic at all at this stage.

1. Primitive. Reuse central_canonical_neighbors() from Step 0 unchanged — the 3 canonicalised central-substitution variants of a k-mer.

2. New annex type (obicompactvec, alongside bitmatrix.rs): a 3-bit- per-slot packed array, one per partition — same on-disk shape family as PersistentBitMatrix's Packed variant, but simpler (no per-genome columns, a single derived read-only value per slot). 3 bits, not 2: revised to also store minorant status alongside sibling count, since it comes for free from the same lookups (point 3 below) — 5 real states (not-minorant; minorant with 0/1/2/3 siblings) fit in 3 bits (8 states, 3 unused). This lets the future SNP sweep discard a non-minorant slot instantly, with no lookup at all, instead of having to regenerate and look up its siblings just to rediscover it isn't the designated writer — moving that cost into this one-time, cached pass instead of repeating it on every future sweep. The otherwise-unreachable combination "not-minorant + 0 siblings" (impossible: 0 siblings always implies minorant, see below) doubles as a free "not yet computed" sentinel — annex files for all partitions/layers can be pre-initialised to this value before the computation pass runs, distinguishing genuinely-computed 0-sibling slots from not-yet-processed ones with no extra storage.

3. Computation pass (obikindex, new siblings.rs): one obipipeline run per layer, iterated sequentially over the index's layers — settled after two false starts, worth recording both.

   • False start 1: "fully parallel over every partition/slot at once, no ordering at all". Correctness is fine with this (sibling count and minorant are order-independent, unlike the old q >= p dedup they replace), but it reintroduces, at a larger scale, exactly the memory-blowup the original Step 2 sweep's sequential-outer-loop constraint existed to prevent: scattering every source partition at once multiplies the in-flight outgoing-query volume by the number of partitions.
   • False start 2: push the layer loop itself into the pipeline (source = the index's layers, a first Flat stage expands each layer into its k-mers). obipipeline's scheduler already bounds memory on its own — it dispatches every item through a shared worker pool at each stage boundary (scheduler.rs:217-372, dispatch() into a common worker_tx queue, any free worker picks up any pending item; not "one worker owns a chunk end to end"), with a biased Select that prioritises draining items already advanced in the chain over admitting new source items (scheduler.rs:271-282: stage results outrank the source, "vider le pipeline en priorité" / "dernier recours" for new data) — so bounded channel capacity plus this drain-first bias already caps in-flight work without any external sequential discipline. Correct, but it means k-mers from several layers can be completing concurrently, so the sink would need to track several open per-layer annex-file writers at once — real, avoidable complexity.
   • Settled design: keep the layer loop external and sequential — not for memory (the pipeline's own capacity/priority mechanism already provides that, for free, regardless), but so each pipeline run's sink targets exactly one layer's annex file, no concurrent multi-writer bookkeeping. Per layer: source = that layer's distinct k-mers; a Flat (1->N) stage generates the 3 central variants of a k-mer, each tagged with its origin (local slot); a transform stage routes each variant to its target partition (unchanged per-k-mer minimiser); a transform stage performs the lookup (existence-only — find_slot hit/miss, cheaper than the SNP sweep's full column fetch); a final stage/sink folds each answer into its origin's running state (below) and, once a layer's k-mers are all resolved, flushes the completed array to that layer's annex file. Many small, single-purpose stages on purpose, to let the scheduler interleave them finely across many in-flight items — this deliberately does not mirror how obipipeline is used elsewhere today: query.rs's process_chunk lumps parse+route+query+serialise into one closure (query.rs:325,743-758), and scatter.rs only pipelines file- reading/superkmer construction, routing partitions afterwards in a plain sequential loop (KmerPartition::write_batch, partition.rs:140) — both under-use the fine-grained scheduling the mechanism offers, so they are not precedents to copy, only existing (and arguably improvable, out of scope here) usages. Cross-partition lookups (querying another layer's MPHF for a variant) remain necessary as before — only the output side is kept single-layer.
   • Reconciliation: processed at the granularity of one answer batch per destination partition, not one source k-mer at a time — this is a proper shuffle, not a per-k-mer wait. Each source partition p holds a small array of running states (minorant = true, siblings = 0), one per local slot, initialised at scatter time and persisting across however many destination-partition batches answer it (up to 3, one per variant, not necessarily all from the same q). Every scattered query carries an origin tag (source partition + local slot) so its answer can be routed back. When target partition q returns its batch (all answers for every query that named q, regardless of which source k-mer or which source partition they came from), that batch is walked once, locally, and each answer updates — via its origin tag — the matching entry in its source partition's array: a miss changes nothing; a hit does siblings += 1, and if the found sibling's own encoding is smaller than the source's, minorant = false. A given source k-mer's state is final only once every destination batch concerning it has been folded in; its partition's array is flushed to the persistent annex once complete. Commutative per entry, so the order in which destination batches arrive and get folded in doesn't matter.

   Open optimisation, not adopted yet — real tradeoff, not a free win. Since looking up sibling y from x's visit already yields everything needed to fill y's own annex entry too, one visit per family could in principle replace one visit per observed family member — cutting this pass's cost roughly by the average family size instead of paying 3 * N_distinct regardless. But it means threads processing different source k-mers can end up writing the same sibling's slot concurrently — the fully independent, ownership-free parallelism of the plan above is deliberately traded away for this gain. It stays safe only because the computed value for a given slot is deterministic regardless of who computes it, so redundant concurrent writes converge to the same value — correct as long as each write is atomic, no locking needed — but it is a real design complexity increase over "every member redoes its own 3 lookups independently," not a strict improvement to adopt by default.

4. Trigger and caching (obikindex::KmerIndex/distance.rs): compute lazily on first distance call for an SNP-family metric against a given index; check for an existing annex file first (mirrors PersistentBitMatrix::open()'s auto-detect-and-fall-back, bitmatrix.rs:264-287); if absent, run step 3 and persist; if present, mmap and reuse.

5. Invalidation. merge and filter always write to a fresh --output directory (obikindex/src/merge.rs, obikmer/src/cmd/filter.rs) so a re-merged/re-filtered index simply has no annex yet — nothing to invalidate. select --in-place (select_layer.rs:139-235) mutates columns of an existing index without changing its location, which can change sibling counts without removing rows — it must explicitly delete any cached annex for that index as part of its in-place rewrite.

6. Testing: hand-built tiny indexes with known sibling counts (0-3); order-independence (recompute twice on a static index, identical result, given the fully-parallel no-ownership design); invalidation (annex absent/correctly recomputed after select --in-place); once Step 2's sweep is fixed, a regression check that sibling_count == 0 slots are never looked up cross-partition during the sweep.

Cost: 3 * N_distinct existence-only lookups, computed once per index state and amortised over every subsequent distance call that reuses the cached annex — cheaper per-lookup than the sweep itself (hit/miss only, no column fetch).

Step 3 — finalisation (obikindex)

From the global SnpTally alone (diagonal and off-diagonal both populated by the sweep, see Step 1/2 — no dependency on the external shared_kmers matrix), derive n x n distance matrices, each a pure function of the accumulated counts (same shape as jaccard_to_mash):

• p_hat[i,j] = SNP / (SNP + shared)
• Jukes-Cantor, Kimura-2P (from P, Q), optionally LogDet (needs the diagonal + base-composition margins).

Guard the singularities (p >= 3/4 for JC, 1-2P-Q <= 0 or 1-2Q <= 0 for K2P) by clamping to a max distance, as jaccard_to_mash clamps J <= 0.

Step 4 — surfacing (obikindex + obikmer CLI)

These metrics do not fit DistanceMetric's current LayeredStore-partial dispatch (they need the cross-partition sweep and produce a different intermediate). Two options, to decide:

• (a) New DistanceMetric variants (Pdistance, JukesCantor, Kimura2P, LogDet) whose KmerIndex::distance arm calls the sweep (snp.rs) instead of the partial path, still returning DistanceOutput. Keeps one CLI surface (--metric jukes-cantor), at the cost of a branch in distance() that ignores the LayeredStore it built.
• (b) A dedicated pathway (KmerIndex::snp_distance) and a distinct CLI entry, if mixing a cross-partition sweep into the partition-local distance command is judged architecturally muddy.

Recommendation: (a) for user ergonomics (all pairwise distances under distance, all feeding NJ/UPGMA/--shared-kmers unchanged), but compute the sweep lazily only when an SNP-family metric is requested, so the existing metrics keep their partition-local fast path untouched.

Step 5 — subsampling flag

Add --snp-sample <fraction> (or a bottom-s hash threshold): restrict the source-k-mer enumeration in Step 2 to seq_hash(kmer) < threshold. Divides lookups proportionally; p_hat is unbiased. Off by default (exact).

Testing

• Primitive unit tests: central_canonical_neighbors on hand-checked k-mers incl. palindrome-boundary cases; lone-k-mer minimizer against RollingStat's incremental result on the same k-mer.
• End-to-end tiny index: two 1-genome indexes differing by a handful of known isolated SNPs (transitions and transversions placed by hand), assert exact SNP, P, Q counts and the resulting JC/K2P values.
• Dedup invariant: assert the tally is identical regardless of genome/ partition order and that no pair is double-counted (compare against a brute-force all-pairs reference on a small index).
• Subsampling: p_hat within sampling error of the exact run.

Suggested phasing

1. Step 0 primitives + their unit tests (self-contained, no distance wiring). This also unblocks the long-declared-but-unimplemented query --mismatch (obikmer/src/cmd/query.rs:676, currently a warning), which needs the same neighbour + routing machinery.
2. SnpTally + finalisation math with a brute-force (non-swept) reference backend, validated on a tiny index.
3. The real per-partition sweep (Step 2) behind the same finalisation; assert it matches the brute-force backend.
4. CLI surfacing (Step 4a) and NJ/UPGMA integration (already generic over the matrix).
5. Subsampling (Step 5).

References

The Mash mutation-rate model this discussion contrasts with: [@Mash-distances-doc; @Fan2015-mash-formula].