docs: clarify MPHF indexing, storage layout, and distance traits

Formalize the two-phase MPHF indexing architecture and update Phase 6 to use `evidence.bin` for direct kmer extraction. Simplify the evidence and unitig storage layouts to flat packed formats enabling O(1) random access. Introduce aggregation traits (`ColumnWeights`, `CountPartials`, `BitPartials`) to support additive distance metric decomposition across partitions. Narrow the documented scope from metagenomic to individual genome datasets, and replace speculative open questions with concrete implementation specifications.
This commit is contained in:
Eric Coissac
2026-05-17 10:20:22 +08:00
parent cf693f17f2
commit f36b095ce2
17 changed files with 916 additions and 1031 deletions
@@ -258,3 +258,51 @@ impl LayerData for PersistentCompactIntMatrix {
fn read(&self, slot: usize) -> Box<[u32]> { self.row(slot) }
}
```
---
## Aggregation traits — `obicompactvec::traits`
`PersistentCompactIntMatrix` implements two aggregation traits used by `LayeredStore<S>` for cross-layer and cross-partition distance computations.
### ColumnWeights
```rust
impl ColumnWeights for PersistentCompactIntMatrix {
fn col_weights(&self) -> Array1<u64> // = self.sum()
}
```
`col_weights()[c]` = sum of all values in column `c` across all slots.
### CountPartials
```rust
impl CountPartials for PersistentCompactIntMatrix {
// Self-contained partials (additive across layers, no external parameter)
fn partial_bray(&self) -> Array2<u64>
fn partial_euclidean(&self) -> Array2<f64>
fn partial_threshold_jaccard(&self, threshold: u32) -> (Array2<u64>, Array2<u64>)
// Normalised partials (require global col_weights across all layers/partitions)
fn partial_relfreq_bray(&self, global: &Array1<u64>) -> Array2<f64>
fn partial_relfreq_euclidean(&self, global: &Array1<u64>) -> Array2<f64>
fn partial_hellinger(&self, global: &Array1<u64>) -> Array2<f64>
// Provided finalisations (default implementations on the trait)
fn bray_dist_matrix(&self) -> Array2<f64>
fn euclidean_dist_matrix(&self) -> Array2<f64>
fn threshold_jaccard_dist_matrix(&self, threshold: u32) -> Array2<f64>
fn relfreq_bray_dist_matrix(&self) -> Array2<f64>
fn relfreq_euclidean_dist_matrix(&self) -> Array2<f64>
fn hellinger_dist_matrix(&self) -> Array2<f64>
}
```
**Self-contained partials** are additively decomposable: summing `partial_bray()` across all `(partition, layer)` pairs and finalising gives the same result as computing on the combined data.
**Normalised partials** require the global column weights (sum across all layers and all partitions). The `global` parameter must reflect the complete index, not a per-layer sum. The provided `relfreq_bray_dist_matrix()` etc. call `col_weights()` first (pass 1) then the normalised partial (pass 2); when called on a `LayeredStore<LayeredStore<…>>` these two-pass calls cascade automatically through the blanket impls.
**`partial_bray` returns `Array2<u64>`** (sum_min only, not a tuple). The denominator is always reconstructible as `col_weights()[i] + col_weights()[j]`.
**`partial_threshold_jaccard` returns `(inter, union)`** as a pair because `union[i,j]` is not reconstructible from per-column statistics — it depends on both columns simultaneously.