Files
obikmer/DevDocMD/theory/sibling_partition_unification.md
T
Eric Coissac 0d12494f1a Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
2026-10-13 08:54:21 +02:00

6.0 KiB

Sibling partition unification and masked canonical form (discussion)

Status: open, nothing implemented. Records the state of the reflection on (a) routing the 4 members of a sibling family to one partition and (b) a canonical form that makes siblings directly comparable. Measurements are throwaway (not in the repo); figures are for k=31, m=11, 256 partitions, 20 bacterial genomes (98.4 M kmer positions), unless stated.

Problem

A sibling family = the 4 kmers differing only at the central base (position c=(k-1)/2, k odd). Routing is not a hash of all bases: a kmer goes to minimizer().seq_hash() & mask, and consecutive kmers of a superkmer share their minimizer. A family splits across partitions whenever the central base falls in (or changes) the selected m-mer.

Fraction of families whose 4 variants are not all in one partition, standard minimizer: 4m/(k+2m+1) (fixed candidates k-2m+1 versus 4m variable ones). Measured: 81.4 % at m=11 (60.2 % at m=7, 46.8 % at m=5). This is the theoretical rate assuming all 4 members present; for two present members it is ≈ 68 %. Real families are overwhelmingly of size 1, so it only becomes cost where something probes variants.

Masked canonical form (Can3)

Notation: CentreA(K) = K with its central base set to A; RC(K) its reverse complement.

  • Can1 = min(K, RC(K)) (current). Each sibling canonicalised alone, so some may be flipped relative to the others.
  • Can2 = orient by CentreA(K) vs RC(CentreA(K)). Not a valid canonical form: when the flanks are reverse complements of each other (L=\mathrm{rc}(R), frequency ≈ 4^{-c}, 1/1024 at k=11), K and \mathrm{RC}(K) get different canonicals (k=5: ACAGT and ACTGT both kept). Cause: CentreA(RC(K)) ≠ RC(CentreA(K)) (the latter has centre T).
  • Can3 (retained):
Can3(K) = K       if CentreA(K) <  CentreA(RC(K))
        = RC(K)   if CentreA(K) >  CentreA(RC(K))
        = Can1(K) otherwise              // self-revcomp family

Orientation is decided on the flanks only (centre ignored), tie broken on the full kmer. Gain: the 4 canonical siblings have identical flanks in the same orientation, so they compare directly with no reverse complement. Self-revcomp families have only 2 distinct canonical members (centres A/C; T/G fold onto them). The A↔T (C↔G) substitution in that context is undetectable by any double-strand canonical set, same as under Can1 — not a loss.

Gain is simplification only: central_canonical_neighbors already canonicalises each variant independently and is correct (see evolutionary_distances.md, per-partition sweep). The earlier "masked-flank grouping" there is the Can2 idea; Can3 is its repaired form. No performance gain by itself, and it breaks existing indexes (canonical orientation changes for some kmers → MPHF hashes change; rebuild).

Prerequisite if adopted: Minimizer = CanonicalKmerOf<MLen> shares the canonical type with index kmers; an m-mer has no sibling family and must keep Can1. Decouple with a distinct Minimizer type (6 files outside kmer.rs: routable.rs, packed_seq.rs, obiskbuilder/{rolling_stat,stream_iter,iter}.rs, obifastwrite).

Routing that excludes the centre (scheme B) — rejected

Candidates restricted to m-mers not overlapping the centre (m\le c): S(j)=\min hash over m-mers p\in[j,j+c-m], minimizer$(i)=\min(S(i),S(i+c+1))$. Family- and strand-invariant by construction (checked). But unitigs are partition-local (pipeline.md), and the hole moves with each kmer, so runs shrink:

m mean run, current mean run, B bases/kmer, current → B
5 14.9 6.3 3.02 → 5.76
11 11.0 3.0 3.72 → 10.96
15 9.0 1.0 4.32 → 30.9

Proxy: run = unitig (upper bound; real unitigs may merge). Independent of partition count. In benchmark/run/global_index_count (786 MB) unitigs.bin + idx ≈ 241 MB (31 %), so tripling them ≈ +60 % total index. Hashing the masked kmer per kmer (no minimizer) would cut every path at every kmer: worse.

Structural tension: path coherence (long unitigs) needs a key shared by consecutive kmers, family coherence needs a key independent of each kmer's own centre, which is an ordinary base for its neighbour. Decoupling sequence storage from MPHF partitioning fails because a partition can only enumerate its kmers from its own unitigs (MPHF not invertible). Redefining the family on a terminal base would cut the split rate to ≈ 17 % but is not closed under revcomp and changes the SNP semantics. Lower m reduces the split rate for free but deviates from the literature k/m\approx 2.5 optimum (balance, bucket size), unmeasured here — m=11 kept.

What the cross-partition cost actually is

  • Phylo-time: only families of size > 1 trigger remote lookups (a small share).
  • Annex construction probes the 3 variants of every kmer (size-1 families included), ≈ 2/3 of probes crossing partitions; batch bounded to #kmers(p)*3 per partition (not the whole index).
  • Real remaining benefit of unification: the outer sweep over source partitions could run in parallel (it is sequential today for memory and for the q >= p dedup rule); plus code simplification in family_scan.rs / annex.rs. Not quantified.
  • Unification would also allow an MPHF keyed on the family (one lookup, 4-bit member mask, union/intersection as OR/AND); needs the family-size histogram (1–4 members) to judge storage.

Open measurements (index not on this machine)

  1. Family-size histogram of the real index: obikmer phylo --sibling-hist (needs a built annex).
  2. Time and peak RSS of --sibling-annex on the Vaccinium/Salix index (Stage "sibling_annex" traces time; peak RSS from outside).
  3. Only if those show the sequential sweep or the probe volume is a real bottleneck: real-index test of scheme B or of a smaller m.

Decision state

No code change decided. Can3 + Minimizer decoupling is the only self-contained step and is independent of the measurements above; it forces an index rebuild, so it is worth doing only if its simplification is wanted.