docs: add central-position SNP distance estimator spec
Introduces a design specification for inferring substitution rates directly from k-mers with conserved flanks. The document details a memory-efficient implementation that computes 4x4 base-pair tallies using existing MPHF structures, enabling classical corrections without de Bruijn graph materialization. Updates MkDocs navigation to include the new theory page.
This commit is contained in:
Eric Coissac
@@ -0,0 +1,329 @@
|
|||||||
|
# Central-position SNP distance (discussion)
|
||||||
|
|
||||||
|
Not implemented. Design discussion for a substitution-rate estimator that
|
||||||
|
observes SNPs directly from paired-genome k-mer comparison, as an alternative
|
||||||
|
to Mash's Poisson-Jaccard inference (see [obicompactvec](../implementation/obicompactvec.md)
|
||||||
|
for the implemented Jaccard/Mash distances).
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
Mash infers a mutation rate from a single scalar (Jaccard) via a model that
|
||||||
|
assumes independence across the k positions. For k odd, a substitution at the
|
||||||
|
**central base** of a k-mer, with the `2m` flanking bases (`m = (k-1)/2`)
|
||||||
|
otherwise conserved, is directly observable: it is a SNP. This gives access not
|
||||||
|
just to a rate but to the substitution's nature (transition/transversion),
|
||||||
|
enabling classical corrected distances.
|
||||||
|
|
||||||
|
## Statistic and correspondence with `shared`
|
||||||
|
|
||||||
|
A genomic position `p` is covered by `k` overlapping k-mer windows. Requiring
|
||||||
|
the substitution to sit at the window's **center** makes exactly one window
|
||||||
|
per SNP eligible — a 1:1 correspondence between SNP and center-neighbor k-mer
|
||||||
|
pair, avoiding the ~k-fold overcount of an any-position neighbor search.
|
||||||
|
|
||||||
|
A locus with a fully conserved `k`-window (flanks **and** center) is precisely
|
||||||
|
an exact-shared k-mer — already produced by the existing `shared_kmers` matrix
|
||||||
|
(`--shared-kmers`, `BitPartials::partial_jaccard` / `CountPartials::partial_threshold_jaccard`).
|
||||||
|
A locus with conserved flanks but a substituted center is a "central SNP". Both
|
||||||
|
count each locus exactly once, in matching units:
|
||||||
|
|
||||||
|
```
|
||||||
|
p_hat[i,j] = SNP[i,j] / (SNP[i,j] + shared[i,j])
|
||||||
|
```
|
||||||
|
|
||||||
|
`p_hat` is `P(center substituted | 2m flanks conserved)`. Only the numerator
|
||||||
|
`SNP[i,j]` needs computing; the denominator term is already available.
|
||||||
|
|
||||||
|
**Canonical invariance**: for odd k, the central position maps to itself under
|
||||||
|
reverse-complement (`m -> k-1-m = m`, base complemented). A transition maps to
|
||||||
|
a transition, a transversion to a transversion — the transition/transversion
|
||||||
|
split is well-defined in canonical space.
|
||||||
|
|
||||||
|
## Sufficient statistic: 4x4 base-pair tally
|
||||||
|
|
||||||
|
Tabulating the joint distribution of `(center_i, center_j)` over conserved-flank
|
||||||
|
loci, per genome pair, is sufficient for every downstream correction:
|
||||||
|
|
||||||
|
| Estimator | Input | Formula |
|
||||||
|
|---|---|---|
|
||||||
|
| Raw p-distance | total off-diagonal / total | `p = SNP / (SNP + shared)` |
|
||||||
|
| Jukes-Cantor | p | `d = -3/4 * ln(1 - 4p/3)` |
|
||||||
|
| Kimura 2-parameter | transition rate P, transversion rate Q | `d = 1/2 ln(1/(1-2P-Q)) + 1/4 ln(1/(1-2Q))` |
|
||||||
|
| LogDet/paralinear | full 4x4 + base-composition margins | `d ~= -1/4 ln det(F)`, robust to non-stationary base composition |
|
||||||
|
|
||||||
|
JC/K2P need only the total and the transition/transversion split. LogDet needs
|
||||||
|
the full 4x4 (adds a diagonal tally of shared k-mers' central base — cheap).
|
||||||
|
|
||||||
|
Memory for the 4x4 tally: `n^2 * 16` counters. Trivial for the project's
|
||||||
|
genome-scale use case (tens to hundreds of genomes); ~13 GB at n=10^4 — outside
|
||||||
|
scope but worth flagging if n grows.
|
||||||
|
|
||||||
|
## Biases (properties of the estimator, not defects)
|
||||||
|
|
||||||
|
1. **Conserved-flank ascertainment bias.** Only SNPs with intact `2m`-base
|
||||||
|
flanks are visible; window-intact probability decays as `(1-p)^{2m}`. For
|
||||||
|
k=31 (2m=30): 0.74 at p=1%, 0.21 at p=5%, 0.04 at p=10%. This estimator
|
||||||
|
targets **closely related genomes**. Under rate heterogeneity across sites
|
||||||
|
(universal in practice), conserved flanks correlate with slow centers, so
|
||||||
|
`p_hat` underestimates the genome-wide average rate — it specifically
|
||||||
|
estimates the substitution rate of **conserved regions**.
|
||||||
|
2. **Bias toward isolated SNPs.** Two SNPs within k of each other disqualify
|
||||||
|
each other's flanks. Hypervariable regions are invisible by construction.
|
||||||
|
3. **Indels are invisible.** A frameshift destroys k-mer matches in a block;
|
||||||
|
this channel captures substitutions only. Indel divergence shows up as lost
|
||||||
|
shared k-mers (lower Jaccard/Mash), not as SNP signal.
|
||||||
|
4. **k-dependent specificity.** "A k-mer match implies common ancestry" is
|
||||||
|
quantitative. For a 3 Gbp genome, expected random flank-30 collisions
|
||||||
|
(k=31): `(3e9)^2 / 4^30 ~= 8` — negligible. At k=21: `(3e9)^2 / 4^20 ~= 2e6`
|
||||||
|
— no longer negligible. k=31 is safe; k<=21 is marginal to unreliable for
|
||||||
|
large genomes. The large k that guarantees homology is the same k that
|
||||||
|
shrinks the detectable-divergence window — an inherent tension.
|
||||||
|
|
||||||
|
## Implementation: avoid materializing a de Bruijn graph
|
||||||
|
|
||||||
|
A central-SNP pair is topologically a simple bubble in the colored de Bruijn
|
||||||
|
graph (source/sink k-mer shared, two length-k branches differing only at the
|
||||||
|
midpoint). Classical bubble-calling (Cortex/discoSNP-style) finds these, but
|
||||||
|
requires the graph — nodes plus adjacency for ~10^9 colored k-mers — resident
|
||||||
|
in memory. **Rejected**: prohibitive RAM for this project's scale.
|
||||||
|
|
||||||
|
A naive per-pair generalisation of variant lookup across n genomes (query each
|
||||||
|
non-shared k-mer's 3 central variants against every counterpart genome's
|
||||||
|
index) costs `O(n^2 . N . 3)` random lookups, with the same k-mer's 3 variants
|
||||||
|
regenerated and requeried once per counterpart genome — pure redundant work.
|
||||||
|
**Rejected** as the basis for an n-genome design.
|
||||||
|
|
||||||
|
## Implementation: sequential per-partition sweep (no scratch, no graph)
|
||||||
|
|
||||||
|
`KmerIndex::distance()` already opens every partition's `presence_store`/
|
||||||
|
`count_store` simultaneously, memory-mapped, into one `LayeredStore`
|
||||||
|
(`distance.rs:73-77`). "Querying another partition" is therefore not a new
|
||||||
|
I/O pattern to design — it is the same O(1) MPHF+evidence lookup the `query`
|
||||||
|
command already performs at scale. This lets the SNP tally be computed with
|
||||||
|
**no scratch files and no auxiliary graph**, by sweeping partitions once each
|
||||||
|
as a source:
|
||||||
|
|
||||||
|
1. For source partition `p`, enumerate its **distinct** k-mers (one per MPHF
|
||||||
|
slot; each already carries its full multi-genome presence/count vector —
|
||||||
|
no need to explode per (k-mer, genome) occurrence).
|
||||||
|
2. For each, generate the 3 central-substitution variants and **canonicalise
|
||||||
|
each independently** (`min(kmer, revcomp)`, exactly as any normal query) —
|
||||||
|
this avoids the orientation edge case a masked-flank grouping would have
|
||||||
|
(a substitution that flips canonical orientation is handled correctly
|
||||||
|
because each variant is canonicalised on its own, not inferred from a
|
||||||
|
fixed-orientation flank key).
|
||||||
|
3. Compute each variant's target partition `q` via its minimizer; batch/sort
|
||||||
|
the partition's outgoing variant queries by `q` for locality.
|
||||||
|
4. Look up each variant in `q`'s already-mmap'd MPHF+evidence; on a hit,
|
||||||
|
combine the source's presence vector (base `a`) with the variant's
|
||||||
|
presence vector (base `b`): for every `i` carrying `a` and `j` carrying
|
||||||
|
`b`, `tally[i,j][a,b] += 1`.
|
||||||
|
|
||||||
|
**Deduplication needs no persisted state.** Sweeping partitions in a fixed
|
||||||
|
order `p = 0, 1, ..., P-1` and only acting on a variant when its target
|
||||||
|
partition `q >= p` guarantees each unordered SNP pair is counted exactly
|
||||||
|
once: a pair with `q < p` was already resolved earlier, when `q` was itself
|
||||||
|
the source partition and `p` (being `>= q`) was a valid forward target. No
|
||||||
|
cross-partition flag array is needed — the sweep order *is* the
|
||||||
|
deduplication rule. Within the same partition (`q == p`), a lightweight
|
||||||
|
transient tie-break suffices: either a `#slots(p)`-bit scratch flag reset per
|
||||||
|
partition, or simply comparing the two k-mers' raw `u64` encodings and only
|
||||||
|
counting when `kmer_source < kmer_variant` — no storage at all.
|
||||||
|
|
||||||
|
This is a distinct computation stage, not a `partial_*` in the existing
|
||||||
|
additive-by-partition sense: step 3-4 read across partition boundaries by
|
||||||
|
construction, unlike the row-local `partial_jaccard`/`partial_threshold_jaccard`
|
||||||
|
primitives. But it requires no new index files, permanent or scratch:
|
||||||
|
`unitigs.bin`, `mphf.bin`, `evidence.bin`, and the presence/count columns are
|
||||||
|
read as-is, and the only extra memory is the current partition's small
|
||||||
|
outgoing-query batch (`#kmers(p) * 3`, released once `p` is done) plus the
|
||||||
|
persistent `tally` accumulator (`n^2 * 16` counters, see above).
|
||||||
|
|
||||||
|
**Outer loop (over source partitions `p`) must stay sequential.** Two
|
||||||
|
independent reasons, not just one: (a) memory — the bounded-footprint claim
|
||||||
|
above only holds with one partition's outgoing-query batch in flight; running
|
||||||
|
`T` source partitions concurrently multiplies that batch by `T`, exactly the
|
||||||
|
blowup the design avoids; (b) correctness — the `q >= p` deduplication rule
|
||||||
|
requires partitions to be claimed as sources in a fixed order; running `p1 <
|
||||||
|
p2` concurrently gives no guarantee `p1` has finished claiming its `q >= p1`
|
||||||
|
targets before `p2` starts claiming its own, breaking the "counted exactly
|
||||||
|
once" property.
|
||||||
|
|
||||||
|
**Inner loop (target-partition lookups for a fixed `p`) parallelises safely.**
|
||||||
|
Each lookup is an O(1) read against an already-mmap'd structure, independent
|
||||||
|
of the others, with no growing allocation — no memory blowup, no ordering
|
||||||
|
dependency between different `q`. The only shared mutable state is `tally`;
|
||||||
|
give each worker thread a **thread-local partial tally** (fixed `n^2 * 16`
|
||||||
|
size, independent of partition size) and merge into the global `tally` once
|
||||||
|
`p`'s inner loop completes — the same reduce-then-merge pattern Rayon already
|
||||||
|
uses elsewhere in this codebase to open partitions in parallel. Extra memory:
|
||||||
|
`#threads * n^2 * 16`, negligible (~512 MB at n~1000, 32 threads) and
|
||||||
|
unrelated to partition size.
|
||||||
|
|
||||||
|
**Cost**: `3 * N_distinct` MPHF lookups total across the whole index (each
|
||||||
|
partition swept once as source) — the same order of magnitude and the same
|
||||||
|
operation as running `query` over the index's entire k-mer content against
|
||||||
|
itself, three times. This is the tool's already-optimized regime, not a new
|
||||||
|
I/O profile to validate.
|
||||||
|
|
||||||
|
## Cheaper: subsampling
|
||||||
|
|
||||||
|
Since the target is a ratio, restricting the source-partition sweep to a
|
||||||
|
bottom-`s` hash sketch (only enumerate k-mers with `hash < threshold` as
|
||||||
|
sources) divides the lookup count by the sampling factor without biasing
|
||||||
|
`p_hat`. Mash-like tradeoff: rate estimated from a sample, not the full
|
||||||
|
k-mer set.
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
Sequential per-partition sweep (Route D): reuse the already-mmap'd
|
||||||
|
per-partition MPHF/evidence/presence structures for O(1) variant lookups,
|
||||||
|
dedup via the fixed sweep-order rule (`q >= p`, plus an in-partition
|
||||||
|
tie-break), no scratch files, no graph materialisation. Denominator reused
|
||||||
|
from the existing `shared_kmers` matrix. Distances (p, JC, K2P, LogDet) as
|
||||||
|
finalisations of the resulting 4x4 tally, mirroring the `partial_* -> *_dist_matrix`
|
||||||
|
pattern used for Jaccard/Mash/Bray-Curtis/etc.
|
||||||
|
|
||||||
|
## Detailed implementation plan
|
||||||
|
|
||||||
|
Grounded in the current codebase. File/type references are anchors, not
|
||||||
|
prescriptions; adjust to reality when implementing.
|
||||||
|
|
||||||
|
### Step 0 — new low-level primitives (`obikseq`)
|
||||||
|
|
||||||
|
Two helpers do not yet exist and are prerequisites:
|
||||||
|
|
||||||
|
1. **Central neighbours.** `CanonicalKmerOf<L>` already exposes
|
||||||
|
`left_canonical_neighbors()` / `right_canonical_neighbors()`
|
||||||
|
(`obikseq/src/kmer.rs`), each returning the 4 canonicalised neighbours at
|
||||||
|
an end position. Add `central_canonical_neighbors()` returning the 4
|
||||||
|
variants at position `m = (k-1)/2` (each independently canonicalised via
|
||||||
|
`.canonical()`). The 3 that differ from the source are the query variants;
|
||||||
|
skip the identity. Building on `nucleotide(i)` / the raw 2-bit layout keeps
|
||||||
|
it O(1).
|
||||||
|
2. **Lone-k-mer minimiser.** Routing a *synthetic* variant to its partition
|
||||||
|
needs its minimiser, but `RollingStat` (`obiskbuilder/src/rolling_stat.rs`)
|
||||||
|
only computes minimisers incrementally along a sequence. Add a standalone
|
||||||
|
`minimizer(kmer) -> Minimizer` that scans the `k-m+1` m-mer windows
|
||||||
|
(`PackedSeq::mmer`, `obikseq/src/packed_seq.rs`), canonicalises each, and
|
||||||
|
takes the min by `seq_hash()` — the same selection `RollingStat` performs,
|
||||||
|
evaluated once. Partition index is then
|
||||||
|
`(minimizer.seq_hash() & (n_partitions - 1)) as usize`, exactly as
|
||||||
|
`QueryBatch::from_records` (`obikmer/src/cmd/query.rs:142`); `n_partitions`
|
||||||
|
is a power of two so the mask is valid.
|
||||||
|
|
||||||
|
### Step 1 — the tally accumulator (`obikindex`)
|
||||||
|
|
||||||
|
A `SnpTally` holding, per genome pair, the 4x4 joint count of central bases:
|
||||||
|
`n * n * 4 * 4` `u64` (or a packed lower-triangular form since it is
|
||||||
|
symmetric). Provide `merge(&mut self, other: &SnpTally)` for the thread-local
|
||||||
|
reduce, and accessors yielding, per pair `(i,j)`: total off-diagonal (SNP),
|
||||||
|
transition count `P`, transversion count `Q`. The diagonal need not be stored
|
||||||
|
for JC/K2P (the denominator comes from `shared_kmers`); store it only if
|
||||||
|
LogDet is wanted, by also tallying shared k-mers' central base.
|
||||||
|
|
||||||
|
### Step 2 — the sweep (`obikindex`, new `snp.rs`)
|
||||||
|
|
||||||
|
Mirror `distance.rs`: open the presence or count store per partition. But
|
||||||
|
instead of a per-partition `partial_*`, run the sequential source sweep:
|
||||||
|
|
||||||
|
```text
|
||||||
|
for p in 0..n_partitions: # OUTER — sequential
|
||||||
|
open source partition p's layers (QueryLayer-style, obikpartitionner)
|
||||||
|
enumerate distinct canonical k-mers of p (one per MPHF slot) with their
|
||||||
|
presence/count vectors # column-major, as query stage 2
|
||||||
|
par_iter over these source k-mers: # INNER — rayon, thread-local tally
|
||||||
|
for each of the 3 central variants:
|
||||||
|
q = partition_of(variant)
|
||||||
|
if q < p: continue # dedup: forward targets only
|
||||||
|
if q == p and variant <= source.raw(): continue # in-partition tie-break
|
||||||
|
slot = layers[q].find_slot(variant) # MphfLayer::find, mmap'd
|
||||||
|
if hit:
|
||||||
|
vb = variant presence/count vector
|
||||||
|
for i in genomes with source base a:
|
||||||
|
for j in genomes with variant base b:
|
||||||
|
thread_tally[i,j][a,b] += 1
|
||||||
|
merge thread-local tallies into global SnpTally
|
||||||
|
```
|
||||||
|
|
||||||
|
The inner lookup is precisely `QueryLayer::find_slot` +
|
||||||
|
`col_value(g, slot)` (`obikpartitionner/src/query_layer.rs`) — reuse or factor
|
||||||
|
out that path rather than reimplementing MPHF access. Enumerating "all distinct
|
||||||
|
k-mers of a partition with their vectors" is the `dump`/`query` stage-2
|
||||||
|
column-major scan already implemented in `dump_layer.rs` /
|
||||||
|
`query_partition_with`; factor a reusable iterator if none fits.
|
||||||
|
|
||||||
|
`presence_threshold` applies exactly as elsewhere: a genome "carries base b"
|
||||||
|
iff its count at that slot is `>= presence_threshold` (trivially `>= 1` for
|
||||||
|
presence indexes).
|
||||||
|
|
||||||
|
### Step 3 — finalisation (`obikindex`)
|
||||||
|
|
||||||
|
From the global `SnpTally` + the existing `shared_kmers` matrix, derive n x n
|
||||||
|
distance matrices, each a pure function of the accumulated counts (same shape
|
||||||
|
as `jaccard_to_mash`):
|
||||||
|
|
||||||
|
- `p_hat[i,j] = SNP / (SNP + shared)`
|
||||||
|
- Jukes-Cantor, Kimura-2P (from `P`, `Q`), optionally LogDet (needs the
|
||||||
|
diagonal + base-composition margins).
|
||||||
|
|
||||||
|
Guard the singularities (`p >= 3/4` for JC, `1-2P-Q <= 0` or `1-2Q <= 0` for
|
||||||
|
K2P) by clamping to a max distance, as `jaccard_to_mash` clamps `J <= 0`.
|
||||||
|
|
||||||
|
### Step 4 — surfacing (`obikindex` + `obikmer` CLI)
|
||||||
|
|
||||||
|
These metrics do not fit `DistanceMetric`'s current `LayeredStore`-partial
|
||||||
|
dispatch (they need the cross-partition sweep and produce a different
|
||||||
|
intermediate). Two options, to decide:
|
||||||
|
|
||||||
|
- **(a)** New `DistanceMetric` variants (`Pdistance`, `JukesCantor`,
|
||||||
|
`Kimura2P`, `LogDet`) whose `KmerIndex::distance` arm calls the sweep
|
||||||
|
(`snp.rs`) instead of the partial path, still returning `DistanceOutput`.
|
||||||
|
Keeps one CLI surface (`--metric jukes-cantor`), at the cost of a branch in
|
||||||
|
`distance()` that ignores the `LayeredStore` it built.
|
||||||
|
- **(b)** A dedicated pathway (`KmerIndex::snp_distance`) and a distinct CLI
|
||||||
|
entry, if mixing a cross-partition sweep into the partition-local `distance`
|
||||||
|
command is judged architecturally muddy.
|
||||||
|
|
||||||
|
Recommendation: (a) for user ergonomics (all pairwise distances under
|
||||||
|
`distance`, all feeding NJ/UPGMA/`--shared-kmers` unchanged), but compute the
|
||||||
|
sweep lazily only when an SNP-family metric is requested, so the existing
|
||||||
|
metrics keep their partition-local fast path untouched.
|
||||||
|
|
||||||
|
### Step 5 — subsampling flag
|
||||||
|
|
||||||
|
Add `--snp-sample <fraction>` (or a bottom-`s` hash threshold): restrict the
|
||||||
|
source-k-mer enumeration in Step 2 to `seq_hash(kmer) < threshold`. Divides
|
||||||
|
lookups proportionally; `p_hat` is unbiased. Off by default (exact).
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
- **Primitive unit tests**: `central_canonical_neighbors` on hand-checked
|
||||||
|
k-mers incl. palindrome-boundary cases; lone-k-mer `minimizer` against
|
||||||
|
`RollingStat`'s incremental result on the same k-mer.
|
||||||
|
- **End-to-end tiny index**: two 1-genome indexes differing by a handful of
|
||||||
|
known isolated SNPs (transitions and transversions placed by hand), assert
|
||||||
|
exact `SNP`, `P`, `Q` counts and the resulting JC/K2P values.
|
||||||
|
- **Dedup invariant**: assert the tally is identical regardless of genome/
|
||||||
|
partition order and that no pair is double-counted (compare against a
|
||||||
|
brute-force all-pairs reference on a small index).
|
||||||
|
- **Subsampling**: `p_hat` within sampling error of the exact run.
|
||||||
|
|
||||||
|
### Suggested phasing
|
||||||
|
|
||||||
|
1. Step 0 primitives + their unit tests (self-contained, no distance wiring).
|
||||||
|
This also unblocks the long-declared-but-unimplemented `query --mismatch`
|
||||||
|
(`obikmer/src/cmd/query.rs:676`, currently a warning), which needs the same
|
||||||
|
neighbour + routing machinery.
|
||||||
|
2. `SnpTally` + finalisation math with a brute-force (non-swept) reference
|
||||||
|
backend, validated on a tiny index.
|
||||||
|
3. The real per-partition sweep (Step 2) behind the same finalisation; assert
|
||||||
|
it matches the brute-force backend.
|
||||||
|
4. CLI surfacing (Step 4a) and NJ/UPGMA integration (already generic over the
|
||||||
|
matrix).
|
||||||
|
5. Subsampling (Step 5).
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
The Mash mutation-rate model this discussion contrasts with:
|
||||||
|
[@Mash-distances-doc; @Fan2015-mash-formula].
|
||||||
@@ -36,6 +36,7 @@ nav:
|
|||||||
- Entropy filter: theory/entropy.md
|
- Entropy filter: theory/entropy.md
|
||||||
- Minimizer selection: theory/minimizer.md
|
- Minimizer selection: theory/minimizer.md
|
||||||
- Partitioning architecture: theory/indexing.md
|
- Partitioning architecture: theory/indexing.md
|
||||||
|
- Central-position SNP distance (discussion): theory/evolutionary_distances.md
|
||||||
- Implementation:
|
- Implementation:
|
||||||
- SuperKmer: implementation/superkmer.md
|
- SuperKmer: implementation/superkmer.md
|
||||||
- Kmer: implementation/kmer.md
|
- Kmer: implementation/kmer.md
|
||||||
|
|||||||
Reference in New Issue
Block a user