OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document k-mer index architecture and refactor distance metrics

Browse Source 
Add comprehensive documentation for the `obilayeredmap` crate, `PersistentCompactIntVec`, `PersistentBitVec`, and the hierarchical k-mer index architecture, including sidebar navigation updates across all documentation pages. Refactor the Bray-Curtis distance computation in `obicompactvec` to decouple numerator and denominator calculations, replacing direct pairwise calls with explicit loops over precomputed sums. Update tests to verify column sum accuracy and align with the simplified API.
This commit is contained in:
Eric Coissac
2026-05-15 21:07:23 +08:00
parent 8409c852ef
commit 45d49ed501
25 changed files with 8842 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/404.html
+112
View File
@@ -632,6 +632,90 @@
<li class="md-nav__item">
<a href="/implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -714,6 +798,34 @@
<li class="md-nav__item">
<a href="/architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/architecture/index_architecture/index.html
+1816
View File
File diff suppressed because it is too large Load Diff
doc/architecture/sequences/invariant/index.html
+115 -1
View File
@@ -9,9 +9,11 @@
<link rel="prev" href="../../../implementation/unitig_evidence/">
<link rel="prev" href="../../../implementation/persistent_bit_vec/">
<link rel="next" href="../../index_architecture/">
@@ -639,6 +641,90 @@
<li class="md-nav__item">
<a href="../../../implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -733,6 +819,34 @@
<li class="md-nav__item">
<a href="../../index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/implementation/chunkreader/index.html
+48
View File
@@ -347,6 +347,42 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../obilayeredmap/">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../persistent_compact_int_vec/">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../persistent_bit_vec/">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
@@ -385,6 +421,18 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../../architecture/index_architecture/">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
doc/implementation/kmer/index.html
+112
View File
@@ -745,6 +745,90 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -827,6 +911,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/implementation/mphf/index.html
+316 -3
View File
@@ -745,6 +745,89 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#multilayer-index-architecture" class="md-nav__link">
<span class="md-ellipsis">
Multilayer index architecture
</span>
</a>
<nav class="md-nav" aria-label="Multilayer index architecture">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#motivation" class="md-nav__link">
<span class="md-ellipsis">
Motivation
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#layer-structure" class="md-nav__link">
<span class="md-ellipsis">
Layer structure
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#membership-verification" class="md-nav__link">
<span class="md-ellipsis">
Membership verification
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#query-algorithm" class="md-nav__link">
<span class="md-ellipsis">
Query algorithm
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#layer-count-and-probe-cost" class="md-nav__link">
<span class="md-ellipsis">
Layer count and probe cost
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#merging-layers" class="md-nav__link">
<span class="md-ellipsis">
Merging layers
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
@@ -795,6 +878,90 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -877,6 +1044,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
@@ -1002,6 +1197,89 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#multilayer-index-architecture" class="md-nav__link">
<span class="md-ellipsis">
Multilayer index architecture
</span>
</a>
<nav class="md-nav" aria-label="Multilayer index architecture">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#motivation" class="md-nav__link">
<span class="md-ellipsis">
Motivation
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#layer-structure" class="md-nav__link">
<span class="md-ellipsis">
Layer structure
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#membership-verification" class="md-nav__link">
<span class="md-ellipsis">
Membership verification
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#query-algorithm" class="md-nav__link">
<span class="md-ellipsis">
Query algorithm
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#layer-count-and-probe-cost" class="md-nav__link">
<span class="md-ellipsis">
Layer count and probe cost
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#merging-layers" class="md-nav__link">
<span class="md-ellipsis">
Merging layers
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
@@ -1071,7 +1349,7 @@
</ul>
<h2 id="mphf-choice-per-phase">MPHF choice per phase</h2>
<p><strong>Phase 1</strong> (provisional, discarded after spectrum computation): FMPHGO. Tolerates overestimated capacity, compact, no need to optimise for query speed on a temporary structure.</p>
<p><strong>Phase 2</strong> (persistent, queried repeatedly): open between FMPHGO and ptr_hash. Exact key count is available, so both operate optimally. ptr_hash's query speed advantage (2.1–3.3×) is meaningful for the persistent index but carries the risk of a very young crate. FMPHGO is the conservative default; ptr_hash is worth revisiting once it has broader production use.</p>
<p><strong>Phase 2</strong> (persistent, queried repeatedly): <strong>ptr_hash</strong>. Exact key count is available at phase 2, so ptr_hash operates optimally. Its query speed (≥2.1× over FMPHGO) and construction speed (≥3.1×) are meaningful for the persistent index; the space overhead at 2.4 bits/key is acceptable. The crate's youth (Feb 2025) was previously a concern; it is now accepted given the performance profile and the fact that each layer MPHF is independently rebuildable from its unitig file if needed.</p>
<p>boomphf is effectively eliminated: its space overhead is the largest and its streaming-construction advantage does not apply here.</p>
<hr />
<h2 id="space-at-scale">Space at scale</h2>
@@ -1106,12 +1384,47 @@
<h2 id="on-disk-and-mmap-considerations">On-disk and mmap considerations</h2>
<p>All three are in-memory structures. Their internal representation is flat bit arrays (no heap pointers), making them serialisable as contiguous byte blobs and mmappable per partition. True zero-copy access would require rkyv integration; the <code>ph</code> crate currently uses serde, so loading involves a copy. Given per-partition MPHF sizes of 1–8 MB, the OS page cache handles this transparently — strict zero-copy is a refinement, not a blocker.</p>
<p>No established Rust crate provides a natively on-disk MPHF. <strong>SSHash</strong> (Sparse and Skew Hash) is a complete kmer dictionary designed for disk access and is order-preserving (overlapping kmers receive consecutive indices → cache-friendly count access), but it is C++-only and covers more than just the MPHF layer.</p>
<hr />
<h2 id="multilayer-index-architecture">Multilayer index architecture</h2>
<h3 id="motivation">Motivation</h3>
<p>An index built from a single dataset A can be extended with a new dataset B without rebuilding. This supports incremental construction (adding species, samples, or sequencing runs) and enables set operations across heterogeneous sources.</p>
<h3 id="layer-structure">Layer structure</h3>
<p>Each layer is a self-contained unit:</p>
<div class="highlight"><pre><span></span><code>layer_i/
unitigs.bin — packed 2-bit nucleotide sequences
mphf.bin — ptr_hash index (phase-2, exact key count)
evidence.bin — [(unitig_id, rank)] per MPHF slot (see unitig_evidence.md)
counts.bin — [u32] per MPHF slot
</code></pre></div>
<p>Layers are <strong>disjoint</strong>: a canonical kmer belongs to exactly one layer. Layer 0 is built from dataset A. Adding dataset B proceeds as follows:</p>
<ol>
<li>For each kmer in B: query layer 0 — if found, accumulate count into <code>counts_0[MPHF_0(kmer)]</code>.</li>
<li>Collect all kmers of B not present in any existing layer → set <code>B \ A</code>.</li>
<li>Build layer 1 from <code>B \ A</code> using the standard two-phase pipeline (spectrum, filter, ptr_hash).</li>
</ol>
<p>Adding a third dataset C repeats the process: probe layer 0, then layer 1, then build layer 2 from <code>C \ A \ B</code>.</p>
<h3 id="membership-verification">Membership verification</h3>
<p>ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry: decode the kmer from <code>(unitig_id, rank)</code> and compare to the query. A mismatch means the kmer is absent from this layer; probe the next layer.</p>
<p>This makes the evidence layer load-bearing for correctness, not only for locality.</p>
<h3 id="query-algorithm">Query algorithm</h3>
<div class="highlight"><pre><span></span><code>fn query(kmer) → Option&lt;count&gt;:
for layer in layers:
slot = layer.mphf.query(kmer)
if layer.evidence.decode(slot) == kmer:
return Some(layer.counts[slot])
return None
</code></pre></div>
<p>Expected probe depth: 1 for kmers present in layer 0, increasing for rare kmers added in later layers. In practice, the dominant dataset (largest A) should be layer 0 to minimise average probe depth.</p>
<h3 id="layer-count-and-probe-cost">Layer count and probe cost</h3>
<p>Each probe is a ptr_hash lookup (~10 ns) plus one evidence decode (two array accesses). For L layers the worst case is L probes + 1 None. In practice L is small (2–5 for typical multi-species databases). No global data structure is needed to route queries; the layer chain is traversed in order.</p>
<h3 id="merging-layers">Merging layers</h3>
<p>Two layer chains can be merged by re-indexing their union through the standard pipeline. This is expensive (full rebuild) but produces an optimal single-layer index. Merge is a maintenance operation, not a query-path requirement.</p>
<h2 id="open-questions">Open questions</h2>
<ul>
<li>Confirm actual partition sizes and overestimation factor on representative metagenomic datasets.</li>
<li>Revisit ptr_hash for phase 2 once the crate has broader production track record.</li>
<li>Assess rkyv integration cost for FMPHGO if true zero-copy mmap becomes necessary for the persistent index.</li>
<li><strong>rkyv integration</strong>: all flat arrays in a layer (evidence, counts, presence/absence matrix) map trivially to <code>rkyv::Archive</code> — fixed-size element types, no heap indirection. The presence/absence matrix is the strongest case: at 10 M kmers × 1 000 samples ≈ 1.25 GB per partition, zero-copy mmap via rkyv avoids loading the entire matrix at open time, letting the OS page cache serve only accessed pages. ptr_hash itself is internally a flat bit array and is structurally compatible with rkyv, but requires either native crate support or a wrapper. Assess the wrapper cost and whether ptr_hash is willing to adopt rkyv upstream.</li>
<li>Keep SSHash in mind if the indexing architecture is reconsidered at a higher level.</li>
<li>Determine optimal layer ordering heuristic (by kmer count? by query frequency?) for multi-species databases.</li>
</ul>
doc/implementation/obilayeredmap/index.html
+1611
View File
File diff suppressed because it is too large Load Diff
doc/implementation/obipipeline/index.html
+112
View File
@@ -795,6 +795,90 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -877,6 +961,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/implementation/persistent_bit_vec/index.html
+1581
View File
File diff suppressed because it is too large Load Diff
doc/implementation/persistent_compact_int_vec/index.html
+1596
View File
File diff suppressed because it is too large Load Diff
doc/implementation/pipeline/index.html
+112
View File
@@ -767,6 +767,90 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -849,6 +933,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/implementation/storage/index.html
+112
View File
@@ -723,6 +723,90 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -805,6 +889,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/implementation/superkmer/index.html
+112
View File
@@ -745,6 +745,90 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -827,6 +911,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/implementation/unitig_evidence/index.html
+150 -2
View File
@@ -6,7 +6,7 @@
<meta charset="utf-8"/>
<meta content="width=device-width,initial-scale=1" name="viewport"/>
<link href="../mphf/" rel="prev"/>
<link href="../../architecture/sequences/invariant/" rel="next"/>
<link href="../obilayeredmap/" rel="next"/>
<link href="../../assets/images/favicon.png" rel="icon"/>
<meta content="mkdocs-1.6.1, mkdocs-material-9.7.6" name="generator"/>
<title>Unitig evidence encoding - obikmer</title>
@@ -467,6 +467,37 @@
</nav>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#non-determinism-of-the-unitig-decomposition">
<span class="md-ellipsis">
Non-determinism of the unitig decomposition
</span>
</a>
<nav aria-label="Non-determinism of the unitig decomposition" class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a class="md-nav__link" href="#source-of-non-determinism">
<span class="md-ellipsis">
Source of non-determinism
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#consequence-for-mphf-construction">
<span class="md-ellipsis">
Consequence for MPHF construction
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#open-questions">
<span class="md-ellipsis">
@@ -478,6 +509,42 @@
</ul>
</nav>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../obilayeredmap/">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../persistent_compact_int_vec/">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../persistent_bit_vec/">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
</li>
@@ -513,6 +580,18 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../../architecture/index_architecture/">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
@@ -698,6 +777,37 @@
</nav>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#non-determinism-of-the-unitig-decomposition">
<span class="md-ellipsis">
Non-determinism of the unitig decomposition
</span>
</a>
<nav aria-label="Non-determinism of the unitig decomposition" class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a class="md-nav__link" href="#source-of-non-determinism">
<span class="md-ellipsis">
Source of non-determinism
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#consequence-for-mphf-construction">
<span class="md-ellipsis">
Consequence for MPHF construction
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#open-questions">
<span class="md-ellipsis">
@@ -774,7 +884,8 @@ b_B = \left\lceil \log_2 U \right\rceil + \left\lceil \log_2 L_{max} \right\rcei
</tr>
</tbody>
</table>
<p>On <em>Betula nana</em> (k=31, 256 partitions), m_u ≈ 37.9 kmers/unitig on average; no unitig length distribution data measured yet. The <code>rank</code> field (kmer index within the unitig) fits in a <code>u8</code> as long as no unitig exceeds 255 kmers — guaranteed by the split strategy below.</p>
<p><strong>Structural maximum from superkmer construction.</strong> For k=31 and m=11, the maximum number of consecutive kmers sharing the same minimiser is k − m + 1 = <strong>21 kmers</strong> (the minimiser traverses from position k−m to 0 as the window slides). A unitig that is a single full superkmer therefore has exactly 21 kmers. This is confirmed by a bimodal distribution in empirical data: a sharp peak at 21 kmers appears in all partitions, including the anomalous partition 145. The observed maximum is ~46 kmers (unitigs spanning more than one superkmer), well within u8 range.</p>
<p>On <em>Betula nana</em> (k=31, 256 partitions), m_u ≈ 37.9 kmers/unitig on average. The <code>rank</code> field (kmer index within the unitig) fits in a <code>u8</code> as long as no unitig exceeds 255 kmers — guaranteed by the split strategy below and amply satisfied by empirical maximums (~46 kmers observed).</p>
<h3 id="split-strategy-for-long-unitigs">Split strategy for long unitigs</h3>
<p>For the rare cases where a unitig exceeds 255 kmers, the unitig is split into chunks of at most 255 kmers, with a <strong>k−1 nucleotide overlap</strong> at each junction — identical to the way super-kmers are delimited at partition boundaries. Each chunk is self-contained and independently decodable.</p>
<div class="highlight"><pre><span></span><code>original unitig: kmer_0 … kmer_254 | kmer_255 … kmer_N
@@ -1026,6 +1137,43 @@ kmer = nucleotides(unitig_id)[rank .. rank + k] // 2-bit packed slice
<h3 id="forward-vs-reverse-complement">Forward vs reverse complement</h3>
<p>The De Bruijn graph stores only canonical kmers. The evidence encodes the canonical orientation. Callers that need the strand of the original kmer must compare the retrieved kmer with its revcomp at query time; this is a single 64-bit comparison.</p>
<hr/>
<h2 id="non-determinism-of-the-unitig-decomposition">Non-determinism of the unitig decomposition</h2>
<p>The unitig extraction is <strong>not deterministic</strong>: two runs on identical input can produce a different number of unitigs with different sequences, while covering exactly the same canonical k-mer set.</p>
<h3 id="source-of-non-determinism">Source of non-determinism</h3>
<p>The graph nodes are stored in a hash map whose iteration order depends on the hash seed (random per run with <code>ahash::RandomState::new()</code>). The <code>start_iter</code> first pass emits every node whose <code>can_extend_left</code> flag is false — which includes not only true dead-end nodes but also <strong>branch points</strong> (nodes with 2 or more left neighbours, for which <code>unique_neighbor</code> returns <code>None</code>).</p>
<p>When a branch point is encountered before its upstream neighbours, it claims the downstream chain and those neighbours later produce length-k degenerate unitigs. When upstream neighbours are encountered first, they extend through the branch point and consume it.</p>
<p><strong>Example</strong> — fork topology (k = 31):</p>
<div class="highlight"><pre><span></span><code>A → B ← C
↓
D
</code></pre></div>
<p>All four nodes are in the graph. B has two left neighbours (A and C), so <code>can_extend_left = false</code>; B also has one right neighbour D, so <code>can_extend_right = true</code>.</p>
<table>
<thead>
<tr>
<th>iteration order</th>
<th>unitigs produced</th>
<th>count</th>
</tr>
</thead>
<tbody>
<tr>
<td>A first, then B, C</td>
<td>ABD · C</td>
<td>2</td>
</tr>
<tr>
<td>B first, then A, C</td>
<td>BD · A · C</td>
<td>3</td>
</tr>
</tbody>
</table>
<p>Both tilings cover the same 4 canonical k-mers.</p>
<p>Pure cycles (all nodes have both extensions present) are unaffected by this: they are never emitted in the first pass and each cycle produces exactly one unitig regardless of which node the second pass starts from. Only the cycle cut point (and therefore the sequence content) varies.</p>
<h3 id="consequence-for-mphf-construction">Consequence for MPHF construction</h3>
<p>The MPHF is built from the <strong>k-mer set</strong>, not from the unitig sequences themselves. Because both tilings contain the same canonical k-mers, the resulting MPHF is identical. The non-determinism is benign for this use case.</p>
<hr/>
<h2 id="open-questions">Open questions</h2>
<ul>
<li><strong>Rank field width</strong>: u8 covers 255 kmers; storing lengths and ranks in kmer units (not nucleotides) buys k−1 extra units of headroom at no cost. On <em>B. nana</em> (k=31), m_u ≈ 38 — well within u8 range on average, but the maximum unitig length has not been measured yet. For genomes with very long unitigs, u16 may be needed; the header could record the actual width if portability is required.</li>
doc/index.html
+112
View File
@@ -708,6 +708,90 @@
<li class="md-nav__item">
<a href="implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -790,6 +874,34 @@
<li class="md-nav__item">
<a href="architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/kmers/index.html
+112
View File
@@ -740,6 +740,90 @@
<li class="md-nav__item">
<a href="../implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -822,6 +906,34 @@
<li class="md-nav__item">
<a href="../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/sitemap.xml.gz
BIN
View File
Binary file not shown.
doc/theory/encoding/index.html
+112
View File
@@ -712,6 +712,90 @@
<li class="md-nav__item">
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -794,6 +878,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/theory/entropy/index.html
+112
View File
@@ -767,6 +767,90 @@
<li class="md-nav__item">
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -849,6 +933,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/theory/indexing/index.html
+112
View File
@@ -712,6 +712,90 @@
<li class="md-nav__item">
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -794,6 +878,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>
doc/theory/minimizer/index.html
+112
View File
@@ -756,6 +756,90 @@
<li class="md-nav__item">
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../implementation/persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
</ul>
</nav>
@@ -838,6 +922,34 @@
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
</ul>
</nav>