From a4eb20e67e34672f4bac2672e469a10140039e6d Mon Sep 17 00:00:00 2001 From: Eric Coissac Date: Thu, 20 Aug 2026 03:32:37 +0200 Subject: [PATCH] add some doc about optimisation for query --- DevDoc/404.html | 28 + .../index_architecture.refs/index.html | 28 + .../index_architecture/index.html | 28 + .../numa_partition_runner/index.html | 28 + .../architecture/numa_worker_pools/index.html | 28 + DevDoc/architecture/query.refs/index.html | 28 + DevDoc/architecture/query/index.html | 28 + DevDoc/architecture/rebuild_filter/index.html | 28 + .../sequences/invariant.refs/index.html | 28 + .../sequences/invariant/index.html | 30 +- DevDoc/architecture/siblings/index.html | 127 ++ .../benchmark_query_testing/index.html | 1582 +++++++++++++++++ .../chunkreader.refs/index.html | 28 + DevDoc/implementation/chunkreader/index.html | 12 + .../evidence_elimination.refs/index.html | 28 + .../evidence_elimination/index.html | 28 + DevDoc/implementation/filtering/index.html | 28 + DevDoc/implementation/kmer.refs/index.html | 28 + DevDoc/implementation/kmer/index.html | 28 + DevDoc/implementation/merge.refs/index.html | 28 + DevDoc/implementation/merge/index.html | 28 + .../merge_parallelism/index.html | 28 + DevDoc/implementation/mphf.refs/index.html | 28 + DevDoc/implementation/mphf/index.html | 28 + .../implementation/obicompactvec/index.html | 45 +- .../obilayeredmap.refs/index.html | 28 + .../implementation/obilayeredmap/index.html | 182 ++ .../obipipeline.refs/index.html | 28 + DevDoc/implementation/obipipeline/index.html | 28 + DevDoc/implementation/obitaxonomy/index.html | 30 +- .../persistent_bit_vec.refs/index.html | 28 + .../persistent_bit_vec/index.html | 28 + .../index.html | 28 + .../persistent_compact_int_vec/index.html | 28 + .../implementation/pipeline.refs/index.html | 28 + DevDoc/implementation/pipeline/index.html | 28 + DevDoc/implementation/select/index.html | 28 + DevDoc/implementation/storage.refs/index.html | 28 + DevDoc/implementation/storage/index.html | 28 + .../implementation/superkmer.refs/index.html | 28 + DevDoc/implementation/superkmer/index.html | 28 + .../unitig_evidence.refs/index.html | 28 + .../implementation/unitig_evidence/index.html | 28 + DevDoc/index.html | 28 + DevDoc/index.refs/index.html | 28 + DevDoc/installation/index.html | 28 + DevDoc/kmers.refs/index.html | 28 + DevDoc/kmers/index.html | 28 + DevDoc/sitemap.xml.gz | Bin 127 -> 127 bytes DevDoc/theory/encoding.refs/index.html | 28 + DevDoc/theory/encoding/index.html | 28 + DevDoc/theory/entropy.refs/index.html | 28 + DevDoc/theory/entropy/index.html | 28 + .../theory/evolutionary_distances/index.html | 28 + DevDoc/theory/indexing.refs/index.html | 28 + DevDoc/theory/indexing/index.html | 28 + DevDoc/theory/minimizer.refs/index.html | 28 + DevDoc/theory/minimizer/index.html | 28 + DevDocMD/architecture/siblings.md | 126 ++ doc/sitemap.xml.gz | Bin 127 -> 127 bytes 60 files changed, 3521 insertions(+), 13 deletions(-) create mode 100644 DevDoc/implementation/benchmark_query_testing/index.html diff --git a/DevDoc/404.html b/DevDoc/404.html index 54445983..43ca7c0d 100644 --- a/DevDoc/404.html +++ b/DevDoc/404.html @@ -939,6 +939,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/index_architecture.refs/index.html b/DevDoc/architecture/index_architecture.refs/index.html index 06376b2b..5b17230a 100644 --- a/DevDoc/architecture/index_architecture.refs/index.html +++ b/DevDoc/architecture/index_architecture.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/index_architecture/index.html b/DevDoc/architecture/index_architecture/index.html index 2b2e1c03..e0663969 100644 --- a/DevDoc/architecture/index_architecture/index.html +++ b/DevDoc/architecture/index_architecture/index.html @@ -948,6 +948,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/numa_partition_runner/index.html b/DevDoc/architecture/numa_partition_runner/index.html index b6fa936f..27b9cc37 100644 --- a/DevDoc/architecture/numa_partition_runner/index.html +++ b/DevDoc/architecture/numa_partition_runner/index.html @@ -946,6 +946,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/numa_worker_pools/index.html b/DevDoc/architecture/numa_worker_pools/index.html index ef50f61c..043289da 100644 --- a/DevDoc/architecture/numa_worker_pools/index.html +++ b/DevDoc/architecture/numa_worker_pools/index.html @@ -948,6 +948,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/query.refs/index.html b/DevDoc/architecture/query.refs/index.html index 7c458324..b0fc23d0 100644 --- a/DevDoc/architecture/query.refs/index.html +++ b/DevDoc/architecture/query.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/query/index.html b/DevDoc/architecture/query/index.html index 00ca5e89..9cbb8a79 100644 --- a/DevDoc/architecture/query/index.html +++ b/DevDoc/architecture/query/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/rebuild_filter/index.html b/DevDoc/architecture/rebuild_filter/index.html index 9e8cbce5..36c8d27b 100644 --- a/DevDoc/architecture/rebuild_filter/index.html +++ b/DevDoc/architecture/rebuild_filter/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/sequences/invariant.refs/index.html b/DevDoc/architecture/sequences/invariant.refs/index.html index b6c9dfc3..1baaad00 100644 --- a/DevDoc/architecture/sequences/invariant.refs/index.html +++ b/DevDoc/architecture/sequences/invariant.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/sequences/invariant/index.html b/DevDoc/architecture/sequences/invariant/index.html index 72751107..cba1d01b 100644 --- a/DevDoc/architecture/sequences/invariant/index.html +++ b/DevDoc/architecture/sequences/invariant/index.html @@ -9,7 +9,7 @@ - + @@ -948,6 +948,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/architecture/siblings/index.html b/DevDoc/architecture/siblings/index.html index 8062296e..02bcdb46 100644 --- a/DevDoc/architecture/siblings/index.html +++ b/DevDoc/architecture/siblings/index.html @@ -948,6 +948,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + @@ -1289,6 +1317,17 @@ + + +
  • + + + + query never benefits from sparse row-major access (found 2026-08-19, not implemented) + + + +
    • @@ -1552,6 +1591,17 @@ + + +
  • + + + + query never benefits from sparse row-major access (found 2026-08-19, not implemented) + + + +
    • @@ -2319,6 +2369,83 @@ scan the full unsampled index — never threaded --subsample/ out of scope here since the reported problem was specifically about the --sankoff/--tnt pipeline's redundant/inconsistent scans, not these two standalone flags.

    +

    query never benefits from sparse row-major access (found 2026-08-19, not implemented)

    +

    Benchmarked obikmer query against global_index_presence (dense-packed) +vs. global_index_presence_sparse (pack --sparse), 100k simulated reads +× 2 specimens (benchmark/, see +benchmark_query_testing.md). +Correctness: 0 mismatches — sparse and dense return bit-identical query +results. Performance: sparse consistently slower than dense (~30-50%, +reproducible across two runs with warm disk cache), the opposite of +pack --sparse's stated intent ("faster for single-row access... like +query").

    +

    Root cause, read from source, not measured in isolation: +KmerPartition::query_partition_with (obikpartitionner/src/query_layer.rs:155-220) +is architecturally column-major: stage 2 walks for g in 0..n_cols { for +slot in hit_slots { layer.col_value(g, slot) } }, documented (correctly) +as the right locality strategy for the packed/columnar formats, where +col_value → PersistentBitMatrix::get is a genuine O(1) mmap'd column +read (persistent.rs:110-113).

    +

    For Self::Sparse, that same get(c, slot) (persistent.rs:114-118) +allocates a full n_cols-wide buffer and calls fill_row — materializing +the entire row — just to return one cell. Called from inside the +column-major double loop, this reconstructs the same row once per genome +column touched: O(hits × n_cols) full-row rebuilds instead of O(hits). +PersistentSparseBitMatrix's own native row-major decode +(for_each_genome_in_row, sparse.rs:164-177, used correctly by its own +row/fill_row/fill_sub_matrix) is never reached from the query path +at all.

    +

    fill_sub_matrix (the existing BinaryMatrix trait primitive, +traits.rs:13-37) is not the right replacement for query either, even +once its own dispatch bug is fixed (see next section) — its output shape +is inherently column-dense: out[col] gets an entry for every column, +including columns with zero hits among the requested slots. On real +sparse data (a hit typically touching a handful of genomes out of dozens) +that's still O(n_cols) output regardless of true sparsity. What query +actually wants is the sparse triple stream (slot, col, value) it already +consumes as QueryHit::Value — not a materialized sub-matrix.

    +

    Proposed primitive (design only, not implemented — explicit ask: keep +count matrices not excluded, even though effort right now is +presence/absence only):

    +
    /// Visit every nonzero cell among `slots`. Order unspecified.
+fn for_each_nonzero(&self, slots: &[usize], f: impl FnMut(usize /*idx into slots*/, usize /*col*/, u32 /*value*/));
+
    + +

    Would let query_partition_with's stage 2 collapse to one +layer.matrix().for_each_nonzero(&hit_slots, |i, g, v| on_event(...)) +call per layer, format-agnostic, with each backend's existing (or +default-derived) implementation deciding the actual access pattern.

    +

    Separately, an existing bug in the generic path (found while tracing +this, itself not yet fixed): PersistentBitMatrix::fill_sub_matrix +(persistent.rs:190-215, the enum wrapper backing BinaryMatrix's +default trait impl) does not delegate to +PersistentSparseBitMatrix::fill_sub_matrix for Self::Sparse — it +reimplements the same naive per-(column, slot) fill_row_bool loop +instead, bypassing the efficient native method one file over +(sparse.rs:249-258). obikphylo::siblings::cache::Mat +(cache.rs:138-145) independently built its own parallel enum wrapper +that dispatches correctly — a sign this was worked around rather than +fixed at the source. Any future for_each_nonzero work should fix this +dispatch too (or route through it), rather than adding a third +independently-dispatching wrapper.

    diff --git a/DevDoc/implementation/benchmark_query_testing/index.html b/DevDoc/implementation/benchmark_query_testing/index.html new file mode 100644 index 00000000..1ac8d0ae --- /dev/null +++ b/DevDoc/implementation/benchmark_query_testing/index.html @@ -0,0 +1,1582 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + Benchmark: query-path testing - obikmer + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + Skip to content + + +
    +
    + +
    + + + + + + +
    + + +
    + +
    + + + + + + +
    +
    + + + +
    +
    +
    + + + + + +
    +
    +
    + + + +
    +
    +
    + + + +
    +
    +
    + + + +
    + +
    + + + + + +

    Benchmark: query-path testing

    +

    benchmark/Makefile exercises indexing, merge, and phylo distance +reconstruction against simulated bacterial genomes. It now also covers +obikmer query — the read-matching path — and the sparse packed +presence-matrix format (obikmer pack --sparse), previously untested by +this pipeline.

    +

    Motivation

    +
      +
    • query had no end-to-end coverage. A regression there would not be caught + by verify_presence/verify_merge_presence, which only check index + content against the .npz truth, never the query API.
      • +
    • pack --sparse produces a presence-matrix format documented (see + siblings.md) as faster for single-row + access (query) and slower for column-oriented access (phylo --metric). + global_index_presence/ built by merge_presence.sh is always packed + dense (packing is a stage inside merge, not a separate pack + invocation) — there was no dense/sparse regression check.
      • +
    +

    Query read source

    +

    Query reads are independent of simulated_data/ (which is folded into the +index being queried): reusing those reads would test against the exact +error draw the index was built from. query_data/<species>/<strain>/ holds +a second, independent iss generate run against the same reference +genome, via simulate_query_one.sh — unseeded, so a second draw picks up +different sequencing errors than simulate_one.sh's draw for the same +genome. Fixed at 100,000 read pairs per genome (not coverage-proportional +like the 15x used for simulated_data/), so wall/RSS numbers stay +comparable across genomes of very different sizes.

    +

    Two query-source specimens, hardcoded as QUERY_SPECIMENS in +make_deps.py: Escherichia_coli--K-12_MG1655 (common, well-represented +bacterium) and Saccharolobus_islandicus--M.16.4 (the only archaeon in +SPECIES — distant lineage, stresses the query path differently from a +close-relative match). Two is enough to catch a dense/sparse regression +without duplicating the exhaustive per-specimen coverage +verify_merge_presence already provides across all SPECIMENS.

    +

    Sparse global index

    +

    global_index_presence_sparse/ is built by pack_sparse.sh: copy +global_index_presence/ wholesale, then obikmer pack --sparse in place. +This works directly because merge's pack stage (merge.rs:252, +pack_matrices(false)) keeps the per-genome column files on disk after +dense-packing — pack_sparse_bit_matrix (obicompactvec/src/bitmatrix/sparse.rs:447) +reads those, is idempotent, and removes matrix.pbmx once the sparse form +is written, so Persistent::open falls through to the sparse format +afterward. No separate merge run needed.

    +

    Query runs

    +

    query_one.sh dense|sparse SPECIMEN runs obikmer query --count-missing +against global_index_presence or global_index_presence_sparse, output +gzipped to query_{dense,sparse}/SPECIMEN.fasta.gz, Reporter wall/RSS +captured to stats/query_{dense,sparse}/SPECIMEN.stats (same +stderr-parsing convention as merge_presence.sh).

    +

    Flags: --count-missing only. --mismatch is a no-op today +(query/mod.rs:212-213, prints "not yet implemented, ignored") — left off +rather than tested for a feature that doesn't exist yet.

    +

    Dense/sparse regression

    +

    verify_query.py compares the two query outputs per specimen, matched by +read id (not stream position — the query pipeline chunks input across +worker threads and doesn't guarantee output order). Compares kmer_count, +kmer_missing, and the full kmer_strict_matches map per read. Any +mismatch is a real regression: dense and sparse must be content-identical, +only I/O access pattern differs. .stats → stats/verify_query/, +aggregated by aggregate_stats.sh query|verify_query-style cases +(query_dense, query_sparse, verify_query).

    +

    Performance comparison

    +

    No dedicated script: the wall/RSS columns from the query_dense and +query_sparse aggregated .stats CSVs are the dense-vs-sparse performance +comparison — the expected win for query on sparse, per the pack --sparse +help text.

    +

    Scope

    +

    count track excluded from the sparse branch: pack --sparse targets +presence matrices only (per CLI help); pack_matrices leaves count +matrices untouched regardless of the sparse flag +(obikindex/src/index.rs:308).

    +

    New Makefile targets

    +

    simulate_query, pack_sparse, query_dense, query_sparse, +aggregate_query_dense, aggregate_query_sparse, verify_query, +aggregate_verify_query — the last three folded into all.

    + + + + + + + + + + + + + +
    +
    + + + +
    + +
    + + + +
    +
    +
    +
    + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/DevDoc/implementation/chunkreader.refs/index.html b/DevDoc/implementation/chunkreader.refs/index.html index 70a9ae3a..3be4d1b2 100644 --- a/DevDoc/implementation/chunkreader.refs/index.html +++ b/DevDoc/implementation/chunkreader.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/chunkreader/index.html b/DevDoc/implementation/chunkreader/index.html index 69852bca..0fea7d09 100644 --- a/DevDoc/implementation/chunkreader/index.html +++ b/DevDoc/implementation/chunkreader/index.html @@ -497,6 +497,18 @@ + + + +
  • + + + + + Benchmark: query-path testing + + +
    • diff --git a/DevDoc/implementation/evidence_elimination.refs/index.html b/DevDoc/implementation/evidence_elimination.refs/index.html index 8215d476..aaa4b067 100644 --- a/DevDoc/implementation/evidence_elimination.refs/index.html +++ b/DevDoc/implementation/evidence_elimination.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/evidence_elimination/index.html b/DevDoc/implementation/evidence_elimination/index.html index 17a240d5..7fa83734 100644 --- a/DevDoc/implementation/evidence_elimination/index.html +++ b/DevDoc/implementation/evidence_elimination/index.html @@ -1085,6 +1085,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/filtering/index.html b/DevDoc/implementation/filtering/index.html index 283f5b8d..8ca925a4 100644 --- a/DevDoc/implementation/filtering/index.html +++ b/DevDoc/implementation/filtering/index.html @@ -1208,6 +1208,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/kmer.refs/index.html b/DevDoc/implementation/kmer.refs/index.html index 5fb4ced9..b7c1c9e3 100644 --- a/DevDoc/implementation/kmer.refs/index.html +++ b/DevDoc/implementation/kmer.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/kmer/index.html b/DevDoc/implementation/kmer/index.html index 060b0c84..af80cf0b 100644 --- a/DevDoc/implementation/kmer/index.html +++ b/DevDoc/implementation/kmer/index.html @@ -1085,6 +1085,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/merge.refs/index.html b/DevDoc/implementation/merge.refs/index.html index 81792f6b..ea0f4334 100644 --- a/DevDoc/implementation/merge.refs/index.html +++ b/DevDoc/implementation/merge.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/merge/index.html b/DevDoc/implementation/merge/index.html index d8edd17a..0a65b288 100644 --- a/DevDoc/implementation/merge/index.html +++ b/DevDoc/implementation/merge/index.html @@ -1157,6 +1157,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/merge_parallelism/index.html b/DevDoc/implementation/merge_parallelism/index.html index 2a11b80e..41b5bd0f 100644 --- a/DevDoc/implementation/merge_parallelism/index.html +++ b/DevDoc/implementation/merge_parallelism/index.html @@ -1152,6 +1152,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/mphf.refs/index.html b/DevDoc/implementation/mphf.refs/index.html index 522d221a..8c39c997 100644 --- a/DevDoc/implementation/mphf.refs/index.html +++ b/DevDoc/implementation/mphf.refs/index.html @@ -944,6 +944,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/mphf/index.html b/DevDoc/implementation/mphf/index.html index a6d6ce9a..726d9ba0 100644 --- a/DevDoc/implementation/mphf/index.html +++ b/DevDoc/implementation/mphf/index.html @@ -1163,6 +1163,34 @@ + + + + + + +
  • + + + + + + + + Benchmark: query-path testing + + + + + + + + +
    • + + + + diff --git a/DevDoc/implementation/obicompactvec/index.html b/DevDoc/implementation/obicompactvec/index.html index e498ee61..bfff68f1 100644 --- a/DevDoc/implementation/obicompactvec/index.html +++ b/DevDoc/implementation/obicompactvec/index.html @@ -412,6 +412,18 @@ + + + +
  • + + + + + Benchmark: query-path testing + + +
    • @@ -1144,29 +1156,40 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)

    Matrix types

    -

    Four matrix types, two encodings × two formats:

    +

    Both matrix types are enums behind a transparent API — the caller never matches on the variant. PersistentCompactIntMatrix has two variants (Columnar, Packed). PersistentBitMatrix has four:

    - - - + + + - - - + + + - - - + + + + + + + + + + + + +
    Columnar formatPacked formatVariantStorageWhen
    BitPersistentBitMatrix (Columnar variant)PersistentBitMatrix (Packed variant)Columnarone .pbiv/.pciv file per column + meta.jsonbuild-time default (*Builder::new)
    IntPersistentCompactIntMatrix (Columnar variant)PersistentCompactIntMatrix (Packed variant)Packedsingle matrix.pbmx mmap filequery-optimised, produced by pack_bit_matrix/pack_compact_int_matrix
    Sparse (bit only)sparse_meta.json + PFIV/Elias-Fano component files, row-majorpack --sparse; see siblings.md for the sparse-vs-dense access-pattern trade-off
    Implicit (bit only)no file at allmono-genome presence layers — n_cols is always reported as 1, every value is true
    -

    Both matrix types are enums (Columnar / Packed / Implicit for bit) behind a transparent API. col_view(c) returns the appropriate view directly:

    +

    PersistentBitMatrix::open(layer_dir) auto-detects the variant, in order: matrix.pbmx → Packed, presence/meta.json → Columnar, presence/sparse_meta.json → Sparse, layer_meta.json (no presence dir at all) → Implicit. col_view/col/sub_matrix panic on Sparse/Implicit where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through row/fill_row.

    +

    col_view(c) returns the appropriate view directly:

    // PersistentBitMatrix
 pub fn col_view(&self, c: usize) -> BitSliceView<'_>
 
diff --git a/DevDoc/implementation/obilayeredmap.refs/index.html b/DevDoc/implementation/obilayeredmap.refs/index.html
index 0b716f14..49fa3c13 100644
--- a/DevDoc/implementation/obilayeredmap.refs/index.html
+++ b/DevDoc/implementation/obilayeredmap.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/obilayeredmap/index.html b/DevDoc/implementation/obilayeredmap/index.html
index 8ff0ac9a..2439b8a6 100644
--- a/DevDoc/implementation/obilayeredmap/index.html
+++ b/DevDoc/implementation/obilayeredmap/index.html
@@ -940,6 +940,67 @@
       
     
   
+
+      
+        
  • 
+  
+    
+      
+        Layer\<D> — raw mapping, iteration, and batch access
+      
+    
+  
+  
+    
+  
 
    • 
       
         
  • 
@@ -1263,6 +1324,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
@@ -1638,6 +1727,67 @@
       
     
   
+
+      
+        
  • 
+  
+    
+      
+        Layer\<D> — raw mapping, iteration, and batch access
+      
+    
+  
+  
+    
+  
 
    • 
       
         
  • 
@@ -2000,6 +2150,38 @@

    • Mode 3 (PersistentBitMatrix) has no push_layer on LayeredMap; callers build directly via Layer<PersistentBitMatrix>::build_presence.

    +

    Layer\<D> — raw mapping, iteration, and batch access

    +

    Beyond query/find (membership-checked), Layer<D> exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.

    +

    Raw kmer → slot mapping

    +
    pub fn index(&self, kmer: CanonicalKmer) -> usize
+pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize>
+
    +

    Pure MPHF mapping, no evidence/fingerprint check — equivalent to MphfOnly::index. Only meaningful when the caller already knows kmer belongs to the layer; on an absent kmer the MPHF still returns some slot (undefined, not None).

    +

    Kmer iteration

    +

    Four iterators, all built from unitigs.bin (physical layout order, not correlated with MPHF slot numbers):

    +
    pub fn iter_kmers(&self) -> KmerIter<'_>
+pub fn enumerate_kmers(&self) -> Enumerate<KmerIter<'_>>              // (order_index, kmer)
+pub fn iter_kmers_batch(&self, n: usize) -> KmerBatchIter<'_>          // Vec<CanonicalKmer> of size ≤ n
+pub fn enumerate_kmers_batch(&self, n: usize) -> impl Iterator<Item = (usize, Vec<CanonicalKmer>)> + Send + 'static
+
    +

    KmerIter/KmerBatchIter own a clone of the underlying Arc<UnitigFileReader> rather than borrowing self — Send + 'static, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. enumerate_kmers_batch's index is the batch's starting offset in iteration order (a multiple of n except for the final, possibly shorter, batch).

    +

    Batch lookup on payload vectors/views

    +

    PersistentCompactIntVec, PersistentBitVec, IntSliceView, BitSliceView all expose:

    +
    fn get_batch(&self, slots: &[usize]) -> Vec<T>
+fn fill_batch(&self, slots: &[usize], out: &mut [T])
+
    +

    Both sort slots internally for sequential mmap access, then reorder results back to the caller's original order. fill_batch fills a caller-provided buffer, avoiding the Vec allocation.

    +

    sub_matrix / fill_sub_matrix

    +
    // Layer<PersistentCompactIntMatrix>
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<u32>>            // column-first
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<u32>])
+
+// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
+
    +

    Column-first to match the on-disk column-major layout. fill_sub_matrix sorts slots once, then calls each column's fill_batch in turn — no redundant per-column sort. On PersistentSparseBitMatrix (k-mer-major, no column method) this degrades to a row-by-row decode; see siblings.md.

    +

    LayeredStore\<S> and aggregation traits

    LayeredStore<S> is a generic aggregation wrapper over Vec<S>. It propagates three traits from obicompactvec::traits up the hierarchy via blanket impls:

    pub struct LayeredStore<S>(pub Vec<S>);
diff --git a/DevDoc/implementation/obipipeline.refs/index.html b/DevDoc/implementation/obipipeline.refs/index.html
index 79fa7072..885e8109 100644
--- a/DevDoc/implementation/obipipeline.refs/index.html
+++ b/DevDoc/implementation/obipipeline.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/obipipeline/index.html b/DevDoc/implementation/obipipeline/index.html
index 9c6babe6..08a6fdf8 100644
--- a/DevDoc/implementation/obipipeline/index.html
+++ b/DevDoc/implementation/obipipeline/index.html
@@ -1113,6 +1113,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/obitaxonomy/index.html b/DevDoc/implementation/obitaxonomy/index.html
index d477e6ab..ce0218b7 100644
--- a/DevDoc/implementation/obitaxonomy/index.html
+++ b/DevDoc/implementation/obitaxonomy/index.html
@@ -12,7 +12,7 @@
         
       
       
-        
+        
       
       
         
@@ -1125,6 +1125,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/persistent_bit_vec.refs/index.html b/DevDoc/implementation/persistent_bit_vec.refs/index.html
index ed24d31a..d4d77cc0 100644
--- a/DevDoc/implementation/persistent_bit_vec.refs/index.html
+++ b/DevDoc/implementation/persistent_bit_vec.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/persistent_bit_vec/index.html b/DevDoc/implementation/persistent_bit_vec/index.html
index ff48156b..17aaf5e9 100644
--- a/DevDoc/implementation/persistent_bit_vec/index.html
+++ b/DevDoc/implementation/persistent_bit_vec/index.html
@@ -1225,6 +1225,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
index 031ff570..1eb1fdc4 100644
--- a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
+++ b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/persistent_compact_int_vec/index.html b/DevDoc/implementation/persistent_compact_int_vec/index.html
index 274ec9bf..150a73c4 100644
--- a/DevDoc/implementation/persistent_compact_int_vec/index.html
+++ b/DevDoc/implementation/persistent_compact_int_vec/index.html
@@ -1208,6 +1208,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/pipeline.refs/index.html b/DevDoc/implementation/pipeline.refs/index.html
index fec369f9..c2dc4f46 100644
--- a/DevDoc/implementation/pipeline.refs/index.html
+++ b/DevDoc/implementation/pipeline.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/pipeline/index.html b/DevDoc/implementation/pipeline/index.html
index 211b01f0..c4d8a225 100644
--- a/DevDoc/implementation/pipeline/index.html
+++ b/DevDoc/implementation/pipeline/index.html
@@ -1074,6 +1074,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/select/index.html b/DevDoc/implementation/select/index.html
index d1351204..3199a4e0 100644
--- a/DevDoc/implementation/select/index.html
+++ b/DevDoc/implementation/select/index.html
@@ -1196,6 +1196,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/storage.refs/index.html b/DevDoc/implementation/storage.refs/index.html
index be053b27..d5387f03 100644
--- a/DevDoc/implementation/storage.refs/index.html
+++ b/DevDoc/implementation/storage.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/storage/index.html b/DevDoc/implementation/storage/index.html
index c463fba5..e526bad4 100644
--- a/DevDoc/implementation/storage/index.html
+++ b/DevDoc/implementation/storage/index.html
@@ -1146,6 +1146,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/superkmer.refs/index.html b/DevDoc/implementation/superkmer.refs/index.html
index 62a1ac84..9ea4dbbc 100644
--- a/DevDoc/implementation/superkmer.refs/index.html
+++ b/DevDoc/implementation/superkmer.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/superkmer/index.html b/DevDoc/implementation/superkmer/index.html
index f7a43cec..4695129f 100644
--- a/DevDoc/implementation/superkmer/index.html
+++ b/DevDoc/implementation/superkmer/index.html
@@ -1052,6 +1052,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/unitig_evidence.refs/index.html b/DevDoc/implementation/unitig_evidence.refs/index.html
index eb9e7a42..d1a1e6ff 100644
--- a/DevDoc/implementation/unitig_evidence.refs/index.html
+++ b/DevDoc/implementation/unitig_evidence.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/implementation/unitig_evidence/index.html b/DevDoc/implementation/unitig_evidence/index.html
index 17064faf..a59911ec 100644
--- a/DevDoc/implementation/unitig_evidence/index.html
+++ b/DevDoc/implementation/unitig_evidence/index.html
@@ -1174,6 +1174,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/index.html b/DevDoc/index.html
index 6b85aa94..2dd207f6 100644
--- a/DevDoc/index.html
+++ b/DevDoc/index.html
@@ -1048,6 +1048,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/index.refs/index.html b/DevDoc/index.refs/index.html
index 9eae697e..6a96f80c 100644
--- a/DevDoc/index.refs/index.html
+++ b/DevDoc/index.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/installation/index.html b/DevDoc/installation/index.html
index 768aaa36..4ce8a4a6 100644
--- a/DevDoc/installation/index.html
+++ b/DevDoc/installation/index.html
@@ -1134,6 +1134,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/kmers.refs/index.html b/DevDoc/kmers.refs/index.html
index 7ccf378e..847d8c23 100644
--- a/DevDoc/kmers.refs/index.html
+++ b/DevDoc/kmers.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/kmers/index.html b/DevDoc/kmers/index.html
index f0f09bbe..5336e366 100644
--- a/DevDoc/kmers/index.html
+++ b/DevDoc/kmers/index.html
@@ -1047,6 +1047,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/sitemap.xml.gz b/DevDoc/sitemap.xml.gz
index d240d53a38bf83f4e1976065c5477c6a68e9751f..425040cd954e2b1b059a7ef7e70cb82e7f9be68e 100644
GIT binary patch
delta 13
Ucmb=gXP58h;An7eo5)@P02}fI82|tP

delta 13
Ucmb=gXP58h;9yu&Kasrx02<>1`v3p{

diff --git a/DevDoc/theory/encoding.refs/index.html b/DevDoc/theory/encoding.refs/index.html
index 26b92a90..2d5effdd 100644
--- a/DevDoc/theory/encoding.refs/index.html
+++ b/DevDoc/theory/encoding.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/encoding/index.html b/DevDoc/theory/encoding/index.html
index fdb70802..dd295cb4 100644
--- a/DevDoc/theory/encoding/index.html
+++ b/DevDoc/theory/encoding/index.html
@@ -1019,6 +1019,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/entropy.refs/index.html b/DevDoc/theory/entropy.refs/index.html
index c046f97e..e0f18901 100644
--- a/DevDoc/theory/entropy.refs/index.html
+++ b/DevDoc/theory/entropy.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/entropy/index.html b/DevDoc/theory/entropy/index.html
index 0f66af11..b0ae350f 100644
--- a/DevDoc/theory/entropy/index.html
+++ b/DevDoc/theory/entropy/index.html
@@ -1085,6 +1085,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/evolutionary_distances/index.html b/DevDoc/theory/evolutionary_distances/index.html
index e7627d12..87fadedf 100644
--- a/DevDoc/theory/evolutionary_distances/index.html
+++ b/DevDoc/theory/evolutionary_distances/index.html
@@ -1461,6 +1461,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/indexing.refs/index.html b/DevDoc/theory/indexing.refs/index.html
index ee81d98c..17bc0289 100644
--- a/DevDoc/theory/indexing.refs/index.html
+++ b/DevDoc/theory/indexing.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/indexing/index.html b/DevDoc/theory/indexing/index.html
index 3de3a3e9..87e40d5c 100644
--- a/DevDoc/theory/indexing/index.html
+++ b/DevDoc/theory/indexing/index.html
@@ -1019,6 +1019,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/minimizer.refs/index.html b/DevDoc/theory/minimizer.refs/index.html
index 046fa357..4756a4c2 100644
--- a/DevDoc/theory/minimizer.refs/index.html
+++ b/DevDoc/theory/minimizer.refs/index.html
@@ -944,6 +944,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDoc/theory/minimizer/index.html b/DevDoc/theory/minimizer/index.html
index cf959bac..0392b8be 100644
--- a/DevDoc/theory/minimizer/index.html
+++ b/DevDoc/theory/minimizer/index.html
@@ -1063,6 +1063,34 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    
  • 
+      
+        
+  
+  
+  
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  
+  
+  
+
+      
+    
    • 
+  
+
+              
+            
           
         
       
diff --git a/DevDocMD/architecture/siblings.md b/DevDocMD/architecture/siblings.md
index 1247cbdb..56396eb7 100644
--- a/DevDocMD/architecture/siblings.md
+++ b/DevDocMD/architecture/siblings.md
@@ -756,3 +756,129 @@ scan the full unsampled index — never threaded `--subsample`/`--entropy`,
 out of scope here since the reported problem was specifically about the
 `--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these
 two standalone flags.
+
+## `query` never benefits from sparse row-major access (found 2026-08-19, not implemented)
+
+Benchmarked `obikmer query` against `global_index_presence` (dense-packed)
+vs. `global_index_presence_sparse` (`pack --sparse`), 100k simulated reads
+× 2 specimens (`benchmark/`, see
+[benchmark_query_testing.md](../implementation/benchmark_query_testing.md)).
+Correctness: 0 mismatches — sparse and dense return bit-identical query
+results. Performance: sparse consistently *slower* than dense (~30-50%,
+reproducible across two runs with warm disk cache), the opposite of
+`pack --sparse`'s stated intent ("faster for single-row access... like
+query").
+
+**Root cause, read from source, not measured in isolation:**
+`KmerPartition::query_partition_with` (`obikpartitionner/src/query_layer.rs:155-220`)
+is architecturally column-major: stage 2 walks `for g in 0..n_cols { for
+slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly)
+as the right locality strategy for the packed/columnar formats, where
+`col_value` → `PersistentBitMatrix::get` is a genuine O(1) mmap'd column
+read (`persistent.rs:110-113`).
+
+For `Self::Sparse`, that same `get(c, slot)` (`persistent.rs:114-118`)
+allocates a full `n_cols`-wide buffer and calls `fill_row` — materializing
+the *entire row* — just to return one cell. Called from inside the
+column-major double loop, this reconstructs the same row once per genome
+column touched: O(hits × n_cols) full-row rebuilds instead of O(hits).
+`PersistentSparseBitMatrix`'s own native row-major decode
+(`for_each_genome_in_row`, `sparse.rs:164-177`, used correctly by its own
+`row`/`fill_row`/`fill_sub_matrix`) is never reached from the query path
+at all.
+
+**`fill_sub_matrix` (the existing `BinaryMatrix` trait primitive,
+`traits.rs:13-37`) is not the right replacement for `query` either**, even
+once its own dispatch bug is fixed (see next section) — its output shape
+is inherently column-dense: `out[col]` gets an entry for every column,
+including columns with zero hits among the requested slots. On real
+sparse data (a hit typically touching a handful of genomes out of dozens)
+that's still O(n_cols) output regardless of true sparsity. What `query`
+actually wants is the sparse triple stream `(slot, col, value)` it already
+consumes as `QueryHit::Value` — not a materialized sub-matrix.
+
+**Proposed primitive** (design only, not implemented — explicit ask: keep
+count matrices *not excluded*, even though effort right now is
+presence/absence only):
+
+Not a closure-driven `for_each` — a real `Iterator`, one concrete struct
+per matrix format, so the traversal state (current position in the sorted
+slot list, current column, permutation, sparse-row decode cursor…) lives
+in named struct fields instead of being threaded implicitly through
+recursion or a captured closure. RPITIT (stable since 1.75, and this
+workspace is edition 2024) means the trait method can return it without
+naming or boxing the concrete type:
+
+```rust
+/// Yields every nonzero cell among `slots`, in implementation-defined order.
+fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator + 'a;
+// item: (idx into `slots`, col, value)
+```
+
+This is the one foundational primitive per format — both `fill_sub_matrix`
+and a `for_each`-style callback become trivial, free consumers of it
+(`.for_each(f)` is already `Iterator::for_each` from std; `fill_sub_matrix`
+becomes "drain the iterator, scatter into `out[][]`"), instead of two
+independently-maintained traversals that can silently diverge (see the bug
+below — this is exactly how it happened).
+
+- **On `PersistentSparseBitMatrix`**: the struct is nearly free to write —
+  it wraps the existing (currently private) `for_each_genome_in_row`
+  per-row decode, advancing to the next `slots` entry on exhaustion. O(Σ
+  row nnz), zero `n_cols`-wide allocation.
+- **On `PersistentBitMatrix::{Packed,Columnar}`**: revised — cheaper than
+  first thought, by reusing the same split already used for
+  `fill_matrix`'s own implementation instead of hand-writing a resumable
+  state machine at the matrix level. The base-vector layer
+  (`BitSliceView`, `views.rs`) already separates the two concerns:
+  `fill_batch_sorted` (`views.rs:55-60`, sorted-slot batch lookup) and a
+  genuine per-bit `Iterator` (`BitSliceIter`, `views.rs:94+`) sit side by
+  side there, one level below the matrix. Adding a
+  "positions among `sorted_slots` where the bit is set" iterator at that
+  same vector level is a `filter` over the existing `get()` — no new
+  state machine, since `std::iter::Filter` already *is* one, generated by
+  the compiler:
+  ```rust
+  fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator + 's {
+      sorted_slots.iter().copied().filter(move |&slot| self.get(slot))
+  }
+  ```
+  The matrix-level `nonzero_iter` then composes these per column with
+  `flat_map` over `0..n_cols` (each column's hits, tagged with `c`,
+  slot mapped back through the sort permutation `fill_batch`/
+  `fill_batch_sorted` already carry) — again a combinator chain, not a
+  hand-rolled struct. Same algorithm, same mmap/sort locality as today's
+  `fill_sub_matrix`; just assembled from `std` iterator adaptors instead
+  of a loop body writing into a buffer, mirroring the vector/matrix split
+  the codebase already uses for `fill_batch_sorted` rather than
+  introducing a new shape.
+- **`Implicit`**: trivial (`slots.iter().map(|&i| (i, 0, 1))`, one column,
+  always present).
+- **On `PersistentCompactIntMatrix` (counts)**: same treatment as
+  `Packed`/`Columnar` — no sparse count format exists yet ("Explicitly
+  deferred" per `traits.rs:9-12`), so no native low-effort case the way
+  `Sparse` has one, but not excluded either: the iterator's `Item` is
+  already `(usize, usize, u32)`, not `bool`, specifically so presence
+  (`0`/`1`) and counts (arbitrary `u32`) share one primitive instead of a
+  bool/u32 split forcing counts out of the design. Ready for a native
+  sparse-count struct later without a signature change.
+
+Would let `query_partition_with`'s stage 2 collapse to one
+`for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) }`
+per layer, format-agnostic, each backend's struct deciding the actual
+traversal.
+
+**This also closes the existing dispatch bug for free, by construction**:
+`PersistentBitMatrix::fill_sub_matrix` (`persistent.rs:190-215`, the enum
+wrapper backing `BinaryMatrix`'s trait impl) today does *not* delegate to
+`PersistentSparseBitMatrix::fill_sub_matrix` for `Self::Sparse` — it
+reimplements the same naive per-(column, slot) `fill_row_bool` loop
+instead, bypassing the efficient native method one file over
+(`sparse.rs:249-258`). `obikphylo::siblings::cache::Mat`
+(`cache.rs:138-145`) independently built its own parallel enum wrapper
+that dispatches correctly — evidence this was worked around rather than
+fixed at the source: two hand-written traversals for the same format,
+free to drift apart, and they did. If `fill_sub_matrix` itself is
+rewritten as "drain `nonzero_iter`, scatter into `out[][]`", there is only
+one traversal per format left to get right — the bug class doesn't just
+get fixed once, it stops being possible to reintroduce.
diff --git a/doc/sitemap.xml.gz b/doc/sitemap.xml.gz
index d240d53a38bf83f4e1976065c5477c6a68e9751f..425040cd954e2b1b059a7ef7e70cb82e7f9be68e 100644
GIT binary patch
delta 13
Ucmb=gXP58h;An7eo5)@P02}fI82|tP

delta 13
Ucmb=gXP58h;9yu&Kasrx02<>1`v3p{