add some doc about optimisation for query

This commit is contained in:
Eric Coissac
2026-08-20 13:59:12 +02:00
parent 89ea077456
commit a4eb20e67e
60 changed files with 3521 additions and 13 deletions
+34 -11
View File
@@ -412,6 +412,18 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../benchmark_query_testing/">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
@@ -1144,29 +1156,40 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
</code></pre></div>
<hr/>
<h2 id="matrix-types">Matrix types</h2>
<p>Four matrix types, two encodings × two formats:</p>
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has two variants (<code>Columnar</code>, <code>Packed</code>). <code>PersistentBitMatrix</code> has four:</p>
<table>
<thead>
<tr>
<th></th>
<th>Columnar format</th>
<th>Packed format</th>
<th>Variant</th>
<th>Storage</th>
<th>When</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Bit</strong></td>
<td><code>PersistentBitMatrix</code> (Columnar variant)</td>
<td><code>PersistentBitMatrix</code> (Packed variant)</td>
<td><code>Columnar</code></td>
<td>one <code>.pbiv</code>/<code>.pciv</code> file per column + <code>meta.json</code></td>
<td>build-time default (<code>*Builder::new</code>)</td>
</tr>
<tr>
<td><strong>Int</strong></td>
<td><code>PersistentCompactIntMatrix</code> (Columnar variant)</td>
<td><code>PersistentCompactIntMatrix</code> (Packed variant)</td>
<td><code>Packed</code></td>
<td>single <code>matrix.pbmx</code> mmap file</td>
<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
</tr>
<tr>
<td><code>Sparse</code> (bit only)</td>
<td><code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major</td>
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
</tr>
<tr>
<td><code>Implicit</code> (bit only)</td>
<td>no file at all</td>
<td>mono-genome presence layers — <code>n_cols</code> is always reported as <code>1</code>, every value is <code>true</code></td>
</tr>
</tbody>
</table>
<p>Both matrix types are enums (<code>Columnar</code> / <code>Packed</code> / <code>Implicit</code> for bit) behind a transparent API. <code>col_view(c)</code> returns the appropriate view directly:</p>
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o">&lt;'</span><span class="nb">_</span><span class="o">&gt;</span>