60 lines
6.0 KiB
Markdown
60 lines
6.0 KiB
Markdown
# Sibling partition unification and masked canonical form (discussion)
|
|||
|
|
|
||
|
|
Status: **open, nothing implemented**. Records the state of the reflection on (a) routing the 4 members of a sibling family to one partition and (b) a canonical form that makes siblings directly comparable. Measurements are throwaway (not in the repo); figures are for $k=31$, $m=11$, 256 partitions, 20 bacterial genomes (98.4 M kmer positions), unless stated.
|
||
|
|
|
||
|
|
## Problem
|
||
|
|
|
||
|
|
A sibling family = the 4 kmers differing only at the central base (position $c=(k-1)/2$, $k$ odd). Routing is **not** a hash of all bases: a kmer goes to `minimizer().seq_hash() & mask`, and consecutive kmers of a superkmer share their minimizer. A family splits across partitions whenever the central base falls in (or changes) the selected m-mer.
|
||
|
|
|
||
|
|
Fraction of families whose 4 variants are not all in one partition, standard minimizer: $4m/(k+2m+1)$ (fixed candidates $k-2m+1$ versus $4m$ variable ones). Measured: 81.4 % at $m=11$ (60.2 % at $m=7$, 46.8 % at $m=5$). This is the *theoretical* rate assuming all 4 members present; for two present members it is ≈ 68 %. Real families are overwhelmingly of size 1, so it only becomes cost where something probes variants.
|
||
|
|
|
||
|
|
## Masked canonical form (Can3)
|
||
|
|
|
||
|
|
Notation: `CentreA(K)` = K with its central base set to A; `RC(K)` its reverse complement.
|
||
|
|
|
||
|
|
- Can1 = `min(K, RC(K))` (current). Each sibling canonicalised alone, so some may be flipped relative to the others.
|
||
|
|
- Can2 = orient by `CentreA(K)` vs `RC(CentreA(K))`. **Not a valid canonical form**: when the flanks are reverse complements of each other ($L=\mathrm{rc}(R)$, frequency ≈ $4^{-c}$, 1/1024 at $k=11$), $K$ and $\mathrm{RC}(K)$ get different canonicals (k=5: `ACAGT` and `ACTGT` both kept). Cause: `CentreA(RC(K)) ≠ RC(CentreA(K))` (the latter has centre T).
|
||
|
|
- **Can3** (retained):
|
||
|
|
|
||
|
|
```
|
||
|
|
Can3(K) = K if CentreA(K) < CentreA(RC(K))
|
||
|
|
= RC(K) if CentreA(K) > CentreA(RC(K))
|
||
|
|
= Can1(K) otherwise // self-revcomp family
|
||
|
|
```
|
||
|
|
|
||
|
|
Orientation is decided on the flanks only (centre ignored), tie broken on the full kmer. Gain: the 4 canonical siblings have identical flanks in the same orientation, so they compare directly with no reverse complement. Self-revcomp families have only 2 distinct canonical members (centres A/C; T/G fold onto them). The A↔T (C↔G) substitution in that context is undetectable by any double-strand canonical set, same as under Can1 — not a loss.
|
||
|
|
|
||
|
|
Gain is simplification only: `central_canonical_neighbors` already canonicalises each variant independently and is correct (see `evolutionary_distances.md`, per-partition sweep). The earlier "masked-flank grouping" there is the Can2 idea; Can3 is its repaired form. **No performance gain by itself**, and it breaks existing indexes (canonical orientation changes for some kmers → MPHF hashes change; rebuild).
|
||
|
|
|
||
|
|
Prerequisite if adopted: `Minimizer = CanonicalKmerOf<MLen>` shares the canonical type with index kmers; an m-mer has no sibling family and must keep Can1. Decouple with a distinct `Minimizer` type (6 files outside `kmer.rs`: `routable.rs`, `packed_seq.rs`, `obiskbuilder/{rolling_stat,stream_iter,iter}.rs`, `obifastwrite`).
|
||
|
|
|
||
|
|
## Routing that excludes the centre (scheme B) — rejected
|
||
|
|
|
||
|
|
Candidates restricted to m-mers not overlapping the centre ($m\le c$): $S(j)=\min$ hash over m-mers $p\in[j,j+c-m]$, minimizer$(i)=\min(S(i),S(i+c+1))$. Family- and strand-invariant by construction (checked). But unitigs are partition-local (`pipeline.md`), and the hole moves with each kmer, so runs shrink:
|
||
|
|
|
||
|
|
| $m$ | mean run, current | mean run, B | bases/kmer, current → B |
|
||
|
|
|---|---|---|---|
|
||
|
|
| 5 | 14.9 | 6.3 | 3.02 → 5.76 |
|
||
|
|
| 11 | 11.0 | 3.0 | 3.72 → 10.96 |
|
||
|
|
| 15 | 9.0 | 1.0 | 4.32 → 30.9 |
|
||
|
|
|
||
|
|
Proxy: run = unitig (upper bound; real unitigs may merge). Independent of partition count. In `benchmark/run/global_index_count` (786 MB) `unitigs.bin` + idx ≈ 241 MB (31 %), so tripling them ≈ +60 % total index. Hashing the masked kmer per kmer (no minimizer) would cut every path at every kmer: worse.
|
||
|
|
|
||
|
|
Structural tension: path coherence (long unitigs) needs a key shared by consecutive kmers, family coherence needs a key independent of each kmer's own centre, which is an ordinary base for its neighbour. Decoupling sequence storage from MPHF partitioning fails because a partition can only enumerate its kmers from its own unitigs (MPHF not invertible). Redefining the family on a terminal base would cut the split rate to ≈ 17 % but is not closed under revcomp and changes the SNP semantics. Lower $m$ reduces the split rate for free but deviates from the literature $k/m\approx 2.5$ optimum (balance, bucket size), unmeasured here — $m=11$ kept.
|
||
|
|
|
||
|
|
## What the cross-partition cost actually is
|
||
|
|
|
||
|
|
- Phylo-time: only families of size > 1 trigger remote lookups (a small share).
|
||
|
|
- Annex construction probes the 3 variants of **every** kmer (size-1 families included), ≈ 2/3 of probes crossing partitions; batch bounded to `#kmers(p)*3` per partition (not the whole index).
|
||
|
|
- Real remaining benefit of unification: the outer sweep over source partitions could run in parallel (it is sequential today for memory and for the `q >= p` dedup rule); plus code simplification in `family_scan.rs` / `annex.rs`. Not quantified.
|
||
|
|
- Unification would also allow an MPHF keyed on the family (one lookup, 4-bit member mask, union/intersection as OR/AND); needs the family-size histogram (1–4 members) to judge storage.
|
||
|
|
|
||
|
|
## Open measurements (index not on this machine)
|
||
|
|
|
||
|
|
1. Family-size histogram of the real index: `obikmer phylo --sibling-hist` (needs a built annex).
|
||
|
|
2. Time and peak RSS of `--sibling-annex` on the Vaccinium/Salix index (`Stage` "sibling_annex" traces time; peak RSS from outside).
|
||
|
|
3. Only if those show the sequential sweep or the probe volume is a real bottleneck: real-index test of scheme B or of a smaller $m$.
|
||
|
|
|
||
|
|
## Decision state
|
||
|
|
|
||
|
|
No code change decided. Can3 + `Minimizer` decoupling is the only self-contained step and is independent of the measurements above; it forces an index rebuild, so it is worth doing only if its simplification is wanted.
|