Refactor k-mer sibling tracking to compact bitmask and on-demand counts

Replaces the explicit `SiblingInfo` struct and 3-bit minorant flags with a derived 4-bit presence mask (`FamilyMask`) that tracks observed bases per family. This eliminates redundant file I/O overhead by introducing a `PartitionCache` for batch lookups, simplifies serialization, and updates all downstream builders, stats computation, and tests to operate on the new bitmask representation. Adjusts CLI output to report deduplicated family sizes instead of histograms, ignores generated CSV files, and updates documentation to reflect the fixed canonical reference and new theory.

docmd/theory/evolutionary_distances.md

@@ -65,6 +65,50 @@ reverse-complement (`m -> k-1-m = m`, base complemented). A transition maps to
 a transition, a transversion to a transversion — the transition/transversion
 split is well-defined in canonical space.
 
+### Definitions: family, and the canonical form of a family
+
+**Family.** The family of a k-mer `x` is the set of (up to) 4 k-mers sharing
+`x`'s `2m` flanking bases, differing only at the central base `m`. Membership
+is a property of the flank pattern, not of `x` itself: any of the 4 possible
+central substitutions belongs to the same family.
+
+**`central_canonical_neighbors()`** (`obikseq`, `CanonicalKmerOf::central_canonical_neighbors`)
+generates all 4 members from any one of them (observed or not), each
+independently canonicalised (`.canonical()`, i.e. `min(kmer, revcomp(kmer))`).
+This independent canonicalisation is necessary because a central substitution
+can flip which orientation is lexicographically smaller — two members of the
+same family can end up canonicalised in *different* orientations. Despite
+that, the **set** of 4 resulting canonical k-mers is invariant: calling
+`central_canonical_neighbors()` on any member of a family — present in the
+index or not — yields the same 4 values. This is relied upon throughout the
+rest of this document.
+
+**Canonical form of a family.** Because orientation can differ member to
+member, "which of the 4 is the reference" cannot be defined relative to
+*whichever member happened to be visited first*, nor relative to the
+minorant (see below) — both are data-dependent (they depend on what is
+actually observed), so using either as the reference would make the
+reference itself vary depending on what happens to be present in a given
+index. Instead: **the canonical form of a family is, by definition, the
+member whose own central base — read in its own already-canonical
+orientation — is `A`.** This is well-defined for every family, computed
+purely from the flank pattern, whether or not that specific member (or any
+member at all) is actually observed anywhere in the index. Concretely: call
+`central_canonical_neighbors()` on any member (observed or not) to get the
+family's 4 canonical forms; the one among them whose own centre nucleotide is
+`A` is the family's canonical form. The other 3 (`C`, `G`, `T`) are labelled
+relative to *that* fixed reference, not relative to the calling member's own
+orientation.
+
+**Consequence for the minorant.** With this fixed A-referenced labelling,
+`minorant` (the smallest raw encoding among the family's *observed* members,
+introduced further below) becomes directly computable rather than needing to
+be tracked as extra state: regenerate the family's 4 canonical forms from
+any member's own k-mer (cheap, no lookup), compare the raw encodings of
+whichever are marked present, and take the smallest. No separate stored bit
+is required — see Step 2b below, where this replaces the earlier
+minorant-bit design.
+
 ## Locus eligibility: raw definition vs. paralogy filter
 
 For each k-mer `x` observed in genome A (source, one MPHF slot; the 3
@@ -482,31 +526,58 @@ here.
 
 ### Step 2b — sibling-count / minorant annex (consolidated plan)
 
-Scope: only the precursor annex (sibling count 0-3 per slot, minorant
-decided on demand) — not the SNP tally itself, whose Step 2 sweep remains
-unresolved above. This piece is simpler than the sweep, because it writes to
-an independent per-slot value, not a shared cross-k-mer accumulator, so it
-needs no dedup/ownership logic at all at this stage.
+Scope: only the precursor annex — not the SNP tally itself, whose Step 2
+sweep remains unresolved above. This piece is simpler than the sweep,
+because it writes to an independent per-slot value, not a shared
+cross-k-mer accumulator, so it needs no dedup/ownership logic at all at this
+stage.
+
+**Revised annex encoding — 4-bit presence mask, not 3-bit (minorant +
+count).** Superseded after settling the "canonical form of a family"
+definition above. The 3-bit design (1 minorant bit + 2-bit sibling count,
+§ below, kept for the historical record) had two problems: it discards
+*which* variants are present (only how many), so any future consumer
+(the SNP sweep, or a stats pass — see below) that needs to know which bases
+exist still has to regenerate and blindly re-query all 3 candidates; and
+the minorant bit's meaning was tied to whichever member was visited, not to
+a fixed reference. Storing instead a **4-bit mask** — one bit per base
+(A/C/G/T), set iff that member of the family (labelled relative to the
+family's fixed canonical form, i.e. the member with `A` at the centre — see
+above) is observed anywhere in the index — fixes both:
+- **Sibling count is derived, not stored**: `siblings = popcount(mask) - 1`.
+- **Minorant is derived, not stored**: regenerate the family's 4 canonical
+  forms from the slot's own k-mer (cheap, no lookup — see above), compare
+  the raw encodings of whichever bits are set in the mask, take the
+  smallest.
+- **A future consumer knows exactly which variants to (re-)query** —
+  `popcount(mask) - 1` lookups instead of always 3, and it knows *which*
+  3 (or fewer) to issue, not just how many hits to expect.
+- The all-zero value (no base present at all) is still logically
+  unreachable as a real result — the slot's *own* base is always present in
+  its own family — so it remains available as a free "not yet computed"
+  sentinel, exactly as before.
 
 1. **Primitive.** Reuse `central_canonical_neighbors()` from Step 0
-   unchanged — the 3 canonicalised central-substitution variants of a k-mer.
-2. **New annex type** (`obicompactvec`, alongside `bitmatrix.rs`): a 3-bit-
-   per-slot packed array, one per partition — same on-disk shape family as
-   `PersistentBitMatrix`'s `Packed` variant, but simpler (no per-genome
-   columns, a single derived read-only value per slot). 3 bits, not 2:
-   revised to also store minorant status alongside sibling count, since it
-   comes for free from the same lookups (point 3 below) — 5 real states
+   unchanged — the 3 canonicalised central-substitution variants of a k-mer
+   (plus the identity, i.e. all 4 members of the family — see "Definitions"
+   above).
+2. **New annex type** (`obicompactvec`, alongside `bitmatrix.rs`): a 4-bit-
+   per-slot packed array (the presence mask above), one per partition — same
+   on-disk shape family as `PersistentBitMatrix`'s `Packed` variant, but
+   simpler (no per-genome columns, a single derived read-only value per
+   slot).
+   <details><summary>Superseded 3-bit design (historical)</summary>
+   3 bits, storing minorant status alongside sibling count directly, since
+   it came for free from the same lookups (point 3 below) — 5 real states
    (not-minorant; minorant with 0/1/2/3 siblings) fit in 3 bits (8 states,
-   3 unused). This lets the future SNP sweep discard a non-minorant slot
-   **instantly**, with no lookup at all, instead of having to regenerate and
-   look up its siblings just to rediscover it isn't the designated writer —
-   moving that cost into this one-time, cached pass instead of repeating it
-   on every future sweep. The otherwise-unreachable combination
-   "not-minorant + 0 siblings" (impossible: 0 siblings always implies
-   minorant, see below) doubles as a free **"not yet computed" sentinel** —
-   annex files for all partitions/layers can be pre-initialised to this
-   value before the computation pass runs, distinguishing genuinely-computed
-   0-sibling slots from not-yet-processed ones with no extra storage.
+   3 unused). This let the future SNP sweep discard a non-minorant slot
+   instantly, with no lookup at all. The otherwise-unreachable combination
+   "not-minorant + 0 siblings" (0 siblings always implies minorant) doubled
+   as the "not yet computed" sentinel. Replaced by the 4-bit mask above,
+   which subsumes this benefit (minorant still derivable, now for free at
+   read time rather than stored) while also fixing the "which variant"
+   blindness.
+   </details>
 3. **Computation pass** (`obikindex`, new `siblings.rs`): **one
    `obipipeline` run per layer, iterated sequentially over the index's
    layers** — settled after two false starts, worth recording both.