refactor(benchmark): consolidate generated artifacts under run/ directory

Restructure the benchmark pipeline to direct all simulated data, indices, statistics, and query outputs into a unified `run/` directory. Update Makefile targets, shell scripts, and Python utilities to resolve paths relative to this new base. Adjust documentation and dependency tracking to match the revised layout, and remove outdated temporary artifacts.

.gitignore

@@ -1,32 +1,36 @@
 .venv/
+.DS_Store
+.claude/
+.kilo/
+.serena/
+.zed/
+.ast-cache/
+CLAUDE.md
+memory/
+sandbox/
 src/target
 data-stress
 *.fasta
 *.fasta.gz
+*.fastq
+*.fastq.gz
+*.vcf
 *.zst
 *.zst.meta
 *.pb
-./**/*.json
+**/*.json
 *.bin
 *.log
 *.csv
+*.meta
+*.pfiv
+*.done
+*.efh
+*.efl
+*.bin.idx
+*.prsb
 Betula_exilis--IGA-24-33
-benchmark/genomes
-benchmark/genomes_orig
-benchmark/simulated_data
-benchmark/specimen_index_presence
-benchmark/specimen_index_count
-benchmark/global_index_presence
-benchmark/global_index_presence_orig
-benchmark/global_index_presence_sav
-benchmark/all_specific
-benchmark/global_index_count
-benchmark/stats
-benchmark/reference_index
-benchmark/reference_dist
-benchmark/obikmer_dist
-benchmark/specific_index_count
-benchmark/specific_index_presence
+benchmark/run/
 TNT
 phyg
 biblio

.serena/project.yml

@@ -1,39 +1,6 @@
-# the name by which the project can be referenced within Serena
+# the name by which the project can be referenced within Serena/when chatting with the LLM.
 project_name: "obikmer"
 
-
-# list of languages for which language servers are started; choose from:
-#   al                  angular             ansible             bash                clojure
-#   cpp                 cpp_ccls            crystal             csharp              csharp_omnisharp
-#   dart                elixir              elm                 erlang              fortran
-#   fsharp              go                  groovy              haskell             haxe
-#   hlsl                html                java                json                julia
-#   kotlin              lean4               lua                 luau                markdown
-#   matlab              msl                 nix                 ocaml               pascal
-#   perl                php                 php_phpactor        powershell          python
-#   python_jedi         python_ty           r                   rego                ruby
-#   ruby_solargraph     rust                scala               scss                solidity
-#   svelte              swift               systemverilog       terraform           toml
-#   typescript          typescript_vts      vue                 yaml                zig
-#   (This list may be outdated. For the current list, see values of Language enum here:
-#   https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
-#   For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
-# Note:
-#   - For C, use cpp
-#   - For JavaScript, use typescript
-#   - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
-#   - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
-#   - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
-#   - For Free Pascal/Lazarus, use pascal
-# Special requirements:
-#   Some languages require additional setup/installations.
-#   See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
-# When using multiple languages, the first language server that supports a given file will be used for that file.
-# The first language is the default language and the respective language server will be used as a fallback.
-# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
-languages:
-- rust
-
 # the encoding used by text files in the project
 # For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
 encoding: "utf-8"
@@ -55,23 +22,19 @@ ignore_all_files_in_gitignore: true
 
 # advanced configuration option allowing to configure language server-specific options.
 # Maps the language key to the options.
-# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
-# No documentation on options means no options are available.
+# The settings are considered only if the project is trusted (see global configuration to define trusted projects).
+# See https://oraios.github.io/serena/02-usage/050_configuration.html#language-server-specific-settings
 ls_specific_settings: {}
 
-# list of additional workspace folder paths for cross-package reference support (e.g. in monorepos).
-# Paths can be absolute or relative to the project root.
-# Each folder is registered as an LSP workspace folder, enabling language servers to discover
-# symbols and references across package boundaries.
-# Currently supported for: TypeScript.
-# Example:
-#   additional_workspace_folders:
-#     - ../sibling-package
-#     - ../shared-lib
-additional_workspace_folders: []
-
 # list of additional paths to ignore in this project.
 # Same syntax as gitignore, so you can use * and **.
+# Important: quote patterns that start with `*`, otherwise YAML treats them as aliases.
+# Example:
+#   ignored_paths:
+#     - "examples/**"
+#     - ".worktrees/**"
+#     - "**/bin/**"
+#     - "**/obj/**"
 # Note: global ignored_paths from serena_config.yml are also applied additively.
 ignored_paths: []
 
@@ -131,3 +94,76 @@ read_only_memory_patterns: []
 # Extends the list from the global configuration, merging the two lists.
 # Example: ["_archive/.*", "_episodes/.*"]
 ignored_memory_patterns: []
+
+# list of additional workspace folder paths for cross-package reference support.
+# Paths can be absolute or relative to the project root.
+# Each folder is registered as an LSP workspace folder, enabling language servers to discover
+# symbols and references across package boundaries, but these folders are not indexed by Serena,
+# i.e. the respective symbols will not be found using Serena's symbol search tools.
+# Example:
+#   additional_workspace_folders:
+#     - ../sibling-package
+#     - ../shared-lib
+ls_additional_workspace_folders: []
+
+# list of language servers to start when using the LSP backend; choose from:
+#   ada                    al                     angular                ansible                bash
+#   bsl                    clojure                cpp                    cpp_ccls               crystal
+#   csharp                 csharp_omnisharp       cue                    dart                   deno
+#   elixir                 elm                    erlang                 fortran                fsharp
+#   gdscript               gleam                  go                     groovy                 haskell
+#   haxe                   hlsl                   html                   java                   json
+#   julia                  kotlin                 latex                  lean4                  lua
+#   luau                   markdown               matlab                 msl                    nextflow
+#   nix                    ocaml                  pascal                 perl                   php
+#   php_phpactor           php_phpantom           powershell             python                 python_basedpyright
+#   python_jedi            python_pyrefly         python_ty              qml                    r
+#   rego                   ruby                   ruby_solargraph        rust                   scala
+#   scss                   solidity               svelte                 swift                  systemverilog
+#   terraform              toml                   typescript             typescript_vts         vue
+#   wolfram                yaml                   zig
+#   (This list may be outdated; generated with scripts/print_language_list.py;
+#   For the current list, see values of the LanguageServerId enum here:
+#   https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
+# For some languages, there are several alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
+# Note:
+#   - For C, use cpp
+#   - For JavaScript, use typescript
+#   - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
+#   - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
+#   - For Deno projects, use deno (serves the same .ts/.js files as typescript; requires the deno CLI on PATH)
+#   - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
+#   - For Free Pascal/Lazarus, use pascal
+# Special requirements:
+#   Some language servers require additional setup/installations.
+#   See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
+# When using multiple language servers, the first language server that supports a given file will be used for that file.
+# The first language server is the default language and the respective language server will be used as a fallback.
+# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
+language_servers:
+- rust
+
+# list of workspace folder paths (LSP backend only).
+# These folders will be used to build up Serena's symbol index.
+# Paths must be within the project root and should thus be relative to the project root.
+# Furthermore, the paths should not be filtered by ignore settings.
+# Default setting: The entire project root folder (".") is considered.
+# In (large) monorepos, this can be used to index only subfolders of the project root, e.g.
+#   ls_workspace_folders:
+#     - "./subproject1"
+#     - "./subproject2"
+ls_workspace_folders:
+- .
+
+# optional shell command to run before the language backend (LSP or JetBrains) is initialised.
+# the command runs in the project root directory and is only executed if the project is trusted
+# (see trusted_project_path_patterns in the global configuration).
+# serena waits for the command to exit: a non-zero exit code is logged as an error but does not
+# abort activation. a per-project timeout (activation_command_timeout, default 180s) is the safety
+# backstop for non-terminating commands; on expiry the process is killed and activation continues.
+# example: activation_command: "npx nx run-many -t build"
+activation_command:
+
+# maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
+# must be a positive number.
+activation_command_timeout: 180.0

CLAUDE.md

@@ -1,103 +0,0 @@
-**PROMPT**
-
-Tu es ma base de connaissance et mon bloc-notes intelligent sur le projet **obikmer**. Tu ne proposes pas, tu ne codes pas spontanément — tu réponds à mes questions et tu structures mes idées au fur et à mesure que je les exprime.
-
-**Règle absolue : une question appelle une réponse, pas une action.**
-Ne modifier aucun fichier à moins d'une demande explicite de modification. En particulier : observer un bug ou une incohérence dans le code montré ne constitue pas un mandat pour le corriger. Le code montré peut refléter une intention en cours — modifier sans mandat risque d'introduire un vrai bug là où tu croyais corriger.
-
-**Règle absolue : ne jamais substituer une dépendance ou une bibliothèque sans validation explicite.**
-Si une dépendance demandée pose problème (erreur de compilation, bug, API manquante), exposer le problème et proposer des alternatives — ne jamais switcher silencieusement vers une autre bibliothèque. Le choix des dépendances est une décision d'architecture qui appartient au développeur.
-
-**Règle absolue : le code existant est une hypothèse, pas une vérité.**
-Quand une nouvelle construction (type, itérateur, abstraction) rend du code historique injustifié, le signaler immédiatement et proposer de le supprimer — ne pas conserver les deux en parallèle par inertie. Le développeur demande explicitement de remettre en cause le code base : ne pas attendre qu'il insiste.
-
-Tu maintiens en **anglais**, dense et sans remplissage, les documents suivants :
-- `DevDocMD/index.md` — document de discussion de base, enrichi progressivement au fil de nos échanges ; il reflète l'état courant de la réflexion sur le projet
-- les autres fichiers Markdown dans `DevDocMD/` selon leur thème respectif
-
-Les snippets de code y sont courts et illustrent uniquement des principes architecturaux. Nos échanges se font en **français**.
-
----
-
-## Contexte du projet
-
-`obikmer` est un outil Rust de manipulation, comptage, indexation et opérations ensemblistes sur des séquences ADN représentées comme des ensembles de kmers.
-
-**Contraintes fondamentales**
-- Efficacité maximale en calcul, mémoire et disque
-- Données métagénomiques : plusieurs dizaines de Gbases, milliards de kmers
-- k impair, k ∈ [11, 31], fixé à l'exécution
-- Formats d'entrée : FASTA, FASTQ, gzip, streaming stdin
-
-**Opérations prioritaires**
-- Comptage de kmers (fréquences)
-- Recherche / requête rapide
-- Opérations ensemblistes (union, intersection, diff)
-
-**Ce qui a déjà été discuté**
-- Encodage 2 bits/base → kmer tient dans un `u64`
-- Forme canonique : `min(kmer, revcomp)` pour réduire l'espace de moitié
-
----
-
-## Infrastructure de documentation
-
-La documentation est gérée via **MkDocs + thème Material**, avec publication sur **GitHub Pages**. Deux arbres de documentation indépendants, deux configurations à la racine du dépôt (voir aussi `UserDocMD/` ci-dessous pour la doc utilisateur) :
-
-**Structure des répertoires**
-
-```
-DevDocMD/       ← sources Markdown, doc développeur (discussion, historique, rationale)
-UserDocMD/      ← sources Markdown, doc utilisateur (état factuel courant, sans code Rust)
-mkdocs.yml      ← config doc développeur : docs_dir DevDocMD, site_dir DevDoc
-mkdocs-user.yml ← config doc utilisateur : docs_dir UserDocMD, site_dir doc
-DevDoc/         ← site HTML généré (doc développeur)
-doc/            ← site HTML généré (doc utilisateur — publié par GitHub Pages, répertoire par défaut)
-.venv/          ← environnement Python (ignoré par git)
-```
-
-**Commandes Makefile**
-
-| Commande | Effet |
-|---|---|
-| `make doc` | Construit la doc développeur dans `DevDoc/` |
-| `make doc-serve` | Serveur local (doc développeur), rechargement automatique |
-| `make clean-doc` | Supprime `DevDoc/` |
-| `make doc-user` | Construit la doc utilisateur dans `doc/` |
-| `make doc-user-serve` | Serveur local (doc utilisateur), rechargement automatique |
-| `make clean-doc-user` | Supprime `doc/` |
-| `make clean` | Supprime `DevDoc/`, `doc/` et `.venv/` |
-
-Le `.venv/` est dans `.gitignore`. `DevDoc/` et `doc/` (sorties HTML) sont versionnés — `doc/` spécifiquement parce que c'est le répertoire par défaut que GitHub Pages sert.
-
-Lors de l'ajout de nouveaux fichiers Markdown dans `DevDocMD/`, mettre à jour la section `nav:` de `mkdocs.yml` ; dans `UserDocMD/`, mettre à jour `mkdocs-user.yml`.
-
----
-
-Je continue à poser mes questions et à guider la discussion.
-
----
-
-## MCP Tools
-
-**Règle absolue : avant tout travail de code, appeler `mcp__serena__initial_instructions` pour charger les instructions Serena.**
-
-### Hiérarchie des outils pour ce projet Rust
-
-**Navigation et édition de code → serena en priorité**
-- Trouver un symbole, une déclaration, les implémentations d'un trait : `mcp__serena__find_symbol`, `mcp__serena__find_declaration`, `mcp__serena__find_implementations`
-- Trouver les usages d'un symbole : `mcp__serena__find_referencing_symbols`
-- Diagnostics LSP (erreurs de compilation) : `mcp__serena__get_diagnostics_for_file`
-- Vue d'ensemble d'un fichier : `mcp__serena__get_symbols_overview`
-- Modifier le corps d'une fonction/impl : `mcp__serena__replace_symbol_body`
-- Ne pas utiliser `cclsp` quand serena couvre le besoin
-
-**Analyse architecturale → jcodemunch**
-- Hotspots, couplage, dead code, dépendances entre modules
-- Utiliser avant de refactorer une zone critique
-
-**Raisonnement complexe → sequential-thinking**
-- Décisions d'architecture, choix d'algorithme, trade-offs non triviaux
-
-**Documentation de crates → context7**
-- Toujours consulter avant d'utiliser une API de bibliothèque externe

DevDoc/404.html

@@ -722,14 +722,14 @@
   
   
     <li class="md-nav__item">
-      <a href="/implementation/obilayeredmap/" class="md-nav__link">
+      <a href="/implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -939,6 +939,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="/implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="/implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/index_architecture.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,10 +1270,10 @@
 <h1 id="coverage-architectureindex_architecturemd">Coverage: architecture/index_architecture.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix</li>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum</li>
-<li><code>obilayeredmap/src/map.rs</code> — LayeredMap<D></li>
-<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta, PartitionMeta</li>
+<li><code>obikindex/src/layer/typed_layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix</li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum</li>
+<li><code>obikindex/src/layer/map.rs</code> — LayeredMap<D></li>
+<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta, PartitionMeta</li>
 <li><code>obikindex/src/meta.rs</code> — IndexConfig (kmer_size, n_bits, with_counts, evidence, block_bits), IndexMeta</li>
 <li><code>obikindex/src/index.rs</code> — KmerIndex, build_layers</li>
 <li><code>obicompactvec/src/</code> — PersistentCompactIntMatrix, PersistentBitMatrix (DataStore implementations)</li>

DevDoc/architecture/index_architecture/index.html

@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/numa_partition_runner/index.html

@@ -729,14 +729,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -946,6 +946,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1311,6 +1367,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1497,6 +1564,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1780,11 +1858,27 @@ dormant workers have been woken and grows every node by the same amount per
 step, capped by that node's remaining dormant workers and by the run's total
 budget (<code>n_total</code>) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.</p>
+<h2 id="panic-propagation-fixed-2026-08-28">Panic propagation (fixed, 2026-08-28)</h2>
+<p>A worker whose <code>f(i)</code> panicked used to never send its <code>WorkerEvent::Completed</code>
+— the controller's <code>while completed &lt; n_total</code> loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept <code>event_rx</code> open. Discovered via <code>obikselect</code>
+panicking on an unimplemented <code>Sparse</code> matrix case (see <code>implementation/select.md</code>):
+the process hung indefinitely instead of erroring out.</p>
+<p>Fixed: each worker wraps <code>f(i)</code> in <code>std::panic::catch_unwind</code> and sends a new
+<code>WorkerEvent::Panicked(i, payload)</code> instead of silently dropping the
+partition. The controller counts it toward <code>completed</code> (unblocking the loop)
+and keeps the first payload seen; once <code>run</code> returns, <code>panic::resume_unwind</code>
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain <code>Err</code> (a panic means a bug, not a normal typed failure).</p>
 <h2 id="open-questions">Open questions</h2>
 <ul>
 <li>
 <p><strong>Error handling</strong>: <code>run</code> currently returns the first error; remaining errors
-  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics.</p>
+  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain <code>Err</code> results.)</p>
 </li>
 <li>
 <p><strong><code>INITIAL_DIVISOR</code> / <code>GROWTH_DIVISOR</code> tuning</strong>: currently <code>4</code> and <code>8</code>

DevDoc/architecture/numa_worker_pools/index.html

@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1504,7 +1560,7 @@
 <td>Replace <code>available_parallelism()</code> with per-NUMA core count for spawn criterion</td>
 </tr>
 <tr>
-<td><code>obikpartitionner/src/merge_layer.rs</code></td>
+<td><code>obikindex/src/partition/merge_layer.rs</code></td>
 <td>No change — <code>merge_partition</code> already works inside any Rayon context</td>
 </tr>
 <tr>
@@ -1512,7 +1568,7 @@
 <td>No change — <code>par_iter</code> and <code>current_num_threads</code> are pool-context-aware</td>
 </tr>
 <tr>
-<td><code>obikpartitionner/src/partition.rs</code></td>
+<td><code>obikindex/src/partition/partition.rs</code></td>
 <td>No change — same reason</td>
 </tr>
 </tbody>

DevDoc/architecture/query.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1215,7 +1271,7 @@
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
 <li><code>obikmer/src/cmd/query.rs</code> — commande query, format de sortie</li>
-<li><code>obikpartitionner/src/query_layer.rs</code> — routage de la requête à travers les partitions</li>
+<li><code>obikindex/src/partition/query_layer.rs</code> — routage de la requête à travers les partitions</li>
 <li><code>obiread/src/lib.rs</code> — lecture des séquences d'entrée pour la requête</li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/architecture/query/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1499,7 +1555,7 @@
 </ul>
 <hr />
 <h2 id="algorithm">Algorithm</h2>
-<p>The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside <code>process_chunk</code> (<code>query.rs</code>); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (<code>obikpartitionner::query_partition_with</code>), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.</p>
+<p>The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside <code>process_chunk</code> (<code>query.rs</code>); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (<code>obikindex::partition::query_partition_with</code>), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.</p>
 <div class="highlight"><pre><span></span><code>for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
     build QueryBatch (QueryBatch::from_records):
         decompose all sequences into superkmers (SuperKmerIter) — construction only,
@@ -1574,9 +1630,9 @@
 <p>The <code>-z</code> CLI option overrides the index metadata value. A higher z increases stringency (lower FP, some true positives may be discarded at sequence ends); a lower z increases sensitivity.</p>
 <hr />
 <h2 id="layer-lookup-mphflayerfind">Layer lookup: <code>MphfLayer::find</code></h2>
-<p><code>MphfLayer::open(dir, mode: &amp;IndexMode)</code> receives the mode from <code>PartitionMeta</code> — no per-layer file is read. The caller (<code>QueryLayer</code>) never chooses the dispatch path: it is fixed at open time by <code>LayerEvidence</code>. See <a href="../../implementation/obilayeredmap/">obilayeredmap</a> for the full <code>find</code> / <code>find_strict</code> API.</p>
+<p><code>MphfLayer::open(dir, mode: &amp;IndexMode)</code> receives the mode from <code>PartitionMeta</code> — no per-layer file is read. The caller (<code>QueryLayer</code>) never chooses the dispatch path: it is fixed at open time by <code>LayerEvidence</code>. See <a href="../../implementation/layer_tier/">obikindex::layer</a> for the full <code>find</code> / <code>find_strict</code> API.</p>
 <h3 id="querylayer-variant-selection"><code>QueryLayer</code> variant selection</h3>
-<p><code>QueryLayer::open</code> (<code>obikpartitionner/src/query_layer.rs:28-45</code>) only ever returns two variants — <code>Presence</code> or <code>Count</code>, checked in this order:</p>
+<p><code>QueryLayer::open</code> (<code>obikindex/src/partition/query_layer.rs:28-45</code>) only ever returns two variants — <code>Presence</code> or <code>Count</code>, checked in this order:</p>
 <table>
 <thead>
 <tr>
@@ -1746,7 +1802,7 @@ Fix direction: restructure <code>query</code>'s pipe with an initial <code>Flat<
 <p><strong>3. Chunk-size memory formula ignores <code>n_genomes</code>.</strong>
 <code>chunk_bytes = available_memory_bytes() / (n_workers * 16)</code> (<code>query.rs:407-414</code>) assumes a fixed ~8–16× overhead per raw input byte. But <code>KmerResults::new</code> (<code>query.rs:165-179</code>) allocates <code>data: Vec&lt;u32&gt;</code> sized <code>total_kmers_in_chunk × n_genomes</code> — dense, <strong>for every k-mer position in the chunk, hit or not</strong> — plus <code>win_min</code> and (with <code>--detail</code>) <code>cov</code>, same scaling. Real per-chunk memory is <code>O(n_genomes)</code>, not constant; the formula doesn't know <code>n_genomes</code> at all. This is the direct cause of the OOM kill on indexes with many reference genomes.</p>
 <p><strong>4. MPHF lookup and matrix-row fetch are fused, not staged.</strong>
-<code>QueryLayer::find_into</code> (<code>obikpartitionner/src/query_layer.rs:48-67</code>) does the MPHF <code>find</code> <em>and</em> the <code>fill_row</code> matrix read in one call per k-mer, inside a single-threaded loop (<code>query_partition_with</code>). There is no separation between "is this k-mer indexed" (cheap, <code>O(1)</code>, independent of <code>n_genomes</code>) and "what are its per-genome values" (the expensive, <code>n_genomes</code>-scaling part).</p>
+<code>QueryLayer::find_into</code> (<code>obikindex/src/partition/query_layer.rs:48-67</code>) does the MPHF <code>find</code> <em>and</em> the <code>fill_row</code> matrix read in one call per k-mer, inside a single-threaded loop (<code>query_partition_with</code>). There is no separation between "is this k-mer indexed" (cheap, <code>O(1)</code>, independent of <code>n_genomes</code>) and "what are its per-genome values" (the expensive, <code>n_genomes</code>-scaling part).</p>
 <p><strong>5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.</strong>
 <code>QueryBatch::from_records</code> currently dereplicates at the <em>superkmer</em> level (<code>HashMap&lt;RoutableSuperKmer, Vec&lt;SKDesc&gt;&gt;</code>, <code>query.rs:112</code>). This misses redundancy between k-mers shared by <em>different</em> superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer <em>construction</em> (<code>SuperKmerIter</code>) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on <code>CanonicalKmer</code>, in the same pass: <code>HashMap&lt;CanonicalKmer, Vec&lt;(seq_idx, pos)&gt;&gt;</code>. This also means the MPHF <code>find</code> itself runs once per <strong>distinct</strong> k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.</p>
 <p><strong>6. Stage 1 output: bucket confirmed hits by layer, keyed by MPHF slot.</strong>
@@ -1778,12 +1834,12 @@ Stage 2's output should be <code>HashMap&lt;genome_idx, Vec&lt;(seq_idx, positio
 <p>None of this is implemented yet — parked here as a coherent roadmap while the design is discussed further. Suggested dependency order: (1) I/O parallelism → (3) genome-aware chunk sizing → (4)–(9) staged/k-mer-deduped/NUMA-aware-partition-and-column-major/sparse query engine (larger refactor, biggest structural payoff — reuses <code>PartitionRunner</code> rather than inventing a new parallelism mechanism) → (2) parallel gzip (separate, orthogonal, tracked in chunkreader.md) → secondary diagnostics patterns.</p>
 <hr />
 <h2 id="implementation-plan">Implementation plan</h2>
-<p>Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current <code>query</code> output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning <code>obikmer</code>, <code>obikpartitionner</code>, and <code>obicompactvec</code> — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense <code>KmerResults</code>) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if <code>rapidgzip-rs</code> is validated (see <a href="../../implementation/chunkreader/#future-work--parallel-gzip-decompression-in-xopen">chunkreader.md</a>).</p>
+<p>Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current <code>query</code> output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning <code>obikmer</code>, <code>obikindex::partition</code>, and <code>obicompactvec</code> — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense <code>KmerResults</code>) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if <code>rapidgzip-rs</code> is validated (see <a href="../../implementation/chunkreader/#future-work--parallel-gzip-decompression-in-xopen">chunkreader.md</a>).</p>
 <p>Instrumentation is deliberately sequenced <em>before</em> the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.</p>
 <p>Performance measurement on the reference 192-core/8-NUMA machine is done by the project owner, not from this development environment (macOS, 16 cores — <code>PartitionRunner</code>'s NUMA pinning is Linux-only, so even phase 4's mechanism can't be functionally exercised for its actual purpose here). Each phase below is therefore written to be <em>self-measuring</em>: the debug-level logging it adds must be enough, on its own, to judge whether that phase's algorithmic choice paid off from a cluster run's logs, without needing to attach a profiler.</p>
 <h3 id="conventions-applied-to-every-phase-below">Conventions applied to every phase below</h3>
 <p><strong>Debug logging.</strong> Every phase that changes an algorithmic choice (not phase 0, which <em>is</em> the logging) adds <code>tracing::debug!</code>/<code>trace!</code> at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF <code>find</code> calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured <code>debug!</code> per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.</p>
-<p><strong>Unit tests.</strong> This project's convention (<code>obiread</code>, <code>obikseq</code>, <code>obidebruinj</code>, <code>obicompactvec</code>, <code>obilayeredmap</code>, <code>obiskio</code>, <code>obifastwrite</code>) is <code>#[cfg(test)] #[path = "tests/&lt;name&gt;.rs"] mod tests;</code> at the bottom of the source file, with the actual test code in a sibling <code>src/tests/&lt;name&gt;.rs</code>. Neither <code>obikmer</code> nor <code>obikpartitionner</code> (the two crates phases 3 and 5 touch most) currently have a <code>src/tests/</code> directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.</p>
+<p><strong>Unit tests.</strong> This project's convention (<code>obiread</code>, <code>obikseq</code>, <code>obidebruinj</code>, <code>obicompactvec</code>, <code>obikindex::layer</code>, <code>obiskio</code>, <code>obifastwrite</code>) is <code>#[cfg(test)] #[path = "tests/&lt;name&gt;.rs"] mod tests;</code> at the bottom of the source file, with the actual test code in a sibling <code>src/tests/&lt;name&gt;.rs</code>. Neither <code>obikmer</code> nor <code>obikindex::partition</code> (the two crates phases 3 and 5 touch most) currently have a <code>src/tests/</code> directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.</p>
 <p><strong>Workflow (<code>jj</code>).</strong> Work happens in a fresh <code>jj</code> commit, easy to abandon. <code>jj new</code> between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then <code>query_layer.rs</code> split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.</p>
 <h3 id="phase-0-instrumentation-prerequisite-for-measuring-every-later-phase">Phase 0 — Instrumentation (prerequisite for measuring every later phase)</h3>
 <p><strong>Goal</strong>: make core utilization, throughput, and per-stage timing visible on a real run, so phases 1–5 can be justified with numbers instead of assumption.</p>
@@ -1826,26 +1882,26 @@ Stage 2's output should be <code>HashMap&lt;genome_idx, Vec&lt;(seq_idx, positio
 <li><code>obikmer/src/cmd/query.rs</code>:</li>
 <li>Replace <code>QueryBatch::from_records</code>'s dedup map (<code>HashMap&lt;RoutableSuperKmer, Vec&lt;SKDesc&gt;&gt;</code>, current <code>query.rs:112</code>) with a per-partition <code>HashMap&lt;CanonicalKmer, Vec&lt;(seq_idx: u32, pos: u32)&gt;&gt;</code>, built in the same <code>SuperKmerIter</code> pass: superkmer construction and partition routing (<code>part_idx</code> from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each <code>CanonicalKmer</code> within a superkmer is inserted individually instead of the whole superkmer being the dedup key.</li>
 <li><strong>Verified</strong>: <code>CanonicalKmer</code> (<code>obikseq/src/kmer.rs:390</code>, <code>pub type CanonicalKmer = CanonicalKmerOf&lt;KLen&gt;</code>) — the underlying <code>CanonicalKmerOf&lt;L&gt;</code> derives <code>Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash</code> (<code>kmer.rs:269</code>). Usable as a <code>HashMap</code>/<code>HashSet</code> key as-is, no change needed.</li>
-<li><code>obikpartitionner/src/query_layer.rs</code>:</li>
+<li><code>obikindex/src/partition/query_layer.rs</code>:</li>
 <li>Split <code>QueryLayer::find_into</code> (<code>query_layer.rs:48-67</code>) into two methods: <code>find_slot(&amp;self, kmer: CanonicalKmer) -&gt; Option&lt;usize&gt;</code> (MPHF only, no matrix touch) and keep <code>fill_row</code> as-is for phase 4 to call later.</li>
 <li>Replace <code>query_partition_with</code>'s inner loop (<code>query_layer.rs:103-113</code>) with a version that, for each unique <code>CanonicalKmer</code>, calls <code>find_slot</code> across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records <code>(layer_idx, slot)</code>.</li>
 <li>New return shape for the partition-level query, replacing today's <code>on_hit(sk_idx, kmer_idx, row)</code> callback: <code>HashMap&lt;layer_idx, HashMap&lt;slot, Vec&lt;(seq_idx, pos)&gt;&gt;&gt;</code> (roadmap point 6) — built directly from the k-mer dedup map's <code>Vec&lt;(seq_idx,pos)&gt;</code> values, keyed by the resolved slot instead of the k-mer.</li>
 <li><strong>This phase alone has no throughput benefit yet</strong> (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.</li>
 <li>Log, per chunk: total k-mer occurrences vs. unique <code>CanonicalKmer</code> count (the dedup ratio — the entire justification for this phase) and the resulting MPHF <code>find</code> call count. If the dedup ratio is close to <code>1.0</code> on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.</li>
-<li><strong>Unit tests</strong>: create <code>obikmer/src/cmd/tests/query.rs</code> (new <code>src/tests/</code> dir for this crate, following the project's <code>#[cfg(test)] #[path = "tests/query.rs"] mod tests;</code> convention) and <code>obikpartitionner/src/tests/query_layer.rs</code> (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the <code>find_slot</code>/bucket-by-layer-and-slot construction against a small hand-built <code>QueryLayer</code> fixture, asserting the <code>(layer_idx, slot, seq_idx, pos)</code> tuples match what the old per-occurrence loop would have produced.</li>
+<li><strong>Unit tests</strong>: create <code>obikmer/src/cmd/tests/query.rs</code> (new <code>src/tests/</code> dir for this crate, following the project's <code>#[cfg(test)] #[path = "tests/query.rs"] mod tests;</code> convention) and <code>obikindex/src/partition/tests/query_layer.rs</code> (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the <code>find_slot</code>/bucket-by-layer-and-slot construction against a small hand-built <code>QueryLayer</code> fixture, asserting the <code>(layer_idx, slot, seq_idx, pos)</code> tuples match what the old per-occurrence loop would have produced.</li>
 </ul>
 <h3 id="phase-4-column-major-matrix-fetch-roadmap-points-78-implemented-numa-parallelism-deferred">Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred</h3>
 <p><strong>Goal (revised during implementation)</strong>: replace <code>fill_row</code>-per-hit (row-major, worst-case mmap locality) with a column-major scan. <code>PartitionRunner</code> turned out to be the wrong mechanism for this at this call granularity — see below; the column-major fetch itself is implemented and validated, without it.</p>
 <p><strong>What shipped:</strong>
 - <code>obicompactvec</code>: the per-column accessors this phase needed <strong>already existed</strong> — <code>PersistentCompactIntMatrix::col_view(c)</code> and <code>PersistentBitMatrix::col_view(c)</code> are public, and <code>IntSliceView::get(slot)</code>/<code>BitSliceView::get(slot)</code> are public — the original plan underestimated how much of this plumbing the pairwise-distance code (<code>dump</code>/<code>select</code>/<code>stats</code>) had already required. The one real gap: <code>PersistentBitMatrix::col_view()</code> panics on the <code>Implicit</code> variant (the documented mono-genome fast path, <code>bitmatrix.rs</code>). Added <code>PersistentBitMatrix::get(c, slot) -&gt; u32</code> (<code>bitmatrix.rs</code>), a non-panicking column-major point lookup that returns <code>1</code> for <code>Implicit</code> regardless of <code>c</code> — the smallest surface needed, not a new <code>col_get</code> API from scratch.
-- <code>obikpartitionner/src/query_layer.rs</code>: <code>query_partition_with</code> is now two explicit stages, matching roadmap points 6–8: <strong>stage 1</strong> (MPHF-only, per unique k-mer, bucket hits by <code>(layer_idx, slot)</code>, emits <code>QueryHit::Found</code>) then <strong>stage 2</strong> (per layer with ≥1 hit, column-major: for each genome column <code>g</code> in <code>0..layer.n_cols().min(n_genomes)</code>, scan that layer's bucketed slots and call <code>col_value(g, slot)</code>, emitting <code>QueryHit::Value(descs, g, value)</code> on nonzero). <code>QueryHit</code> is a single enum delivered through one <code>FnMut(QueryHit)</code> callback — an earlier two-closure design (<code>on_found</code> + <code>on_value</code>) didn't borrow-check, since the caller's single mutable accumulator (<code>KmerResults</code>) can't be captured by two separate <code>FnMut</code> closures passed to the same call.
+- <code>obikindex/src/partition/query_layer.rs</code>: <code>query_partition_with</code> is now two explicit stages, matching roadmap points 6–8: <strong>stage 1</strong> (MPHF-only, per unique k-mer, bucket hits by <code>(layer_idx, slot)</code>, emits <code>QueryHit::Found</code>) then <strong>stage 2</strong> (per layer with ≥1 hit, column-major: for each genome column <code>g</code> in <code>0..layer.n_cols().min(n_genomes)</code>, scan that layer's bucketed slots and call <code>col_value(g, slot)</code>, emitting <code>QueryHit::Value(descs, g, value)</code> on nonzero). <code>QueryHit</code> is a single enum delivered through one <code>FnMut(QueryHit)</code> callback — an earlier two-closure design (<code>on_found</code> + <code>on_value</code>) didn't borrow-check, since the caller's single mutable accumulator (<code>KmerResults</code>) can't be captured by two separate <code>FnMut</code> closures passed to the same call.
 - <code>obikmer/src/cmd/query.rs</code>: <code>KmerResults::set</code> (row-major, whole-row-at-once) replaced by <code>mark_found</code> (stage 1: flag a position as indexed, independent of any genome's value) and <code>set_one</code> (stage 2: write one genome's value at one position). <code>QueryStats</code> extended with <code>n_columns_scanned</code>/<code>n_col_get_calls</code>, logged per chunk.
 - Total <code>get()</code>-equivalent calls are unchanged from the row-major version (<code>n_hits × n_cols</code> in the worst case, confirmed by <code>n_col_get_calls</code> in the debug log) — the win is locality (sequential access within one layer's column at a time, across <code>mmap</code>'d regions, instead of jumping across all columns per hit), exactly as predicted.</p>
 <p><strong>What did not ship, and why — <code>PartitionRunner</code> is architecturally the wrong tool here:</strong>
 Reading <code>obikindex/src/numa.rs</code>'s actual <code>run()</code> body (not just its doc comments) shows every call spawns a timer thread <strong>plus one OS thread per worker slot on every NUMA node</strong> (<code>std::thread::scope</code> + one <code>s.spawn()</code> per node per <code>max_workers</code>) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned <strong>per call</strong>. This is fine for its actual, established usage in this codebase (<code>merge.rs</code>, <code>index.rs</code>'s <code>build_layers</code>): one <code>PartitionRunner::new()</code> + one <code>run()</code> call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for <code>query</code>'s call pattern: <code>query_partition_with</code> runs once per <code>(chunk, partition)</code>, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
 The column-major loop in stage 2 is therefore a <strong>plain sequential loop</strong> for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) <code>rayon</code>'s already-warm global pool (<code>into_par_iter()</code>) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern <code>numa_worker_pools.md</code> warns about for a <em>different</em> workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one <code>mmap</code>'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so <code>PartitionRunner</code> is invoked once per whole <code>query</code> run (or per large batch of chunks) rather than per <code>(chunk, partition)</code>, amortising its spawn cost the way <code>merge</code>/<code>build_layers</code> do — a bigger structural change than this phase's scope.
 - Log (implemented): <code>QueryStats::n_columns_scanned</code>/<code>n_col_get_calls</code>, folded into the existing per-chunk <code>debug!("k-mer dedup + column-major fetch", ...)</code> line (<code>query.rs</code>) alongside phase 3's dedup counters.
-- <strong>Unit tests</strong>: extended <code>obikpartitionner/src/tests/query_layer.rs</code> (phase 3's file) — <code>query_partition_with</code>'s empty/missing-index paths updated for the new <code>QueryStats</code> fields and single-callback signature.
+- <strong>Unit tests</strong>: extended <code>obikindex/src/partition/tests/query_layer.rs</code> (phase 3's file) — <code>query_partition_with</code>'s empty/missing-index paths updated for the new <code>QueryStats</code> fields and single-callback signature.
 - <strong>Validation performed</strong>: full workspace build + <code>cargo test --workspace</code>, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same <code>kmer_count</code>/<code>kmer_strict_matches</code> on every record); (2) the existing 20-genome <code>benchmark/global_index_presence</code> index — runs correctly, <code>n_hits=0</code> for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the <code>Implicit</code>/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) <strong>the critical correctness case</strong>: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from <code>genomeA</code> matched <strong>only</strong> <code>genomeA</code> (<code>kmer_count</code> identical to the pre-dedup occurrence count, zero leakage into <code>genomeB</code>'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in <code>n_cols</code>, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
 - <strong>Not yet done</strong>: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-<code>n_genomes</code> layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.</p>
 <h3 id="phase-5-sparse-findere-rework-roadmap-point-9">Phase 5 — Sparse Findere rework (roadmap point 9)</h3>

DevDoc/architecture/rebuild_filter/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1389,9 +1445,9 @@ IO pattern: sequential read of each source column → random write into new colu
 </table>
 <h2 id="files-to-modify">Files to modify</h2>
 <ul>
-<li><code>src/obikpartitionner/src/rebuild_layer.rs</code> — <code>rebuild_partition</code> and <code>iter_src_layers</code></li>
+<li><code>src/obikindex/src/partition/rebuild_layer.rs</code> — <code>rebuild_partition</code> and <code>iter_src_layers</code></li>
 <li>Possibly <code>src/obicompactvec/</code> — add column iterator API if not already present</li>
-<li><code>src/obilayeredmap/</code> — check if per-column sequential access is exposed on <code>SrcLayerData</code></li>
+<li><code>src/obikindex/src/layer/</code> — check if per-column sequential access is exposed on <code>SrcLayerData</code></li>
 </ul>
 <h2 id="open-questions">Open questions</h2>
 <ul>

DevDoc/architecture/sequences/invariant.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/sequences/invariant/index.html

@@ -9,7 +9,7 @@
       
       
       
-        <link rel="prev" href="../../../implementation/obitaxonomy/">
+        <link rel="prev" href="../../../implementation/partition_layer_cache/">
       
       
         <link rel="next" href="../../index_architecture/">
@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/siblings/index.html

@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1289,6 +1345,39 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#query-never-benefits-from-sparse-row-major-access-found-2026-08-19-not-implemented" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        query never benefits from sparse row-major access (found 2026-08-19, not implemented)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#implemented-2026-08-20" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Implemented (2026-08-20)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1552,6 +1641,39 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#query-never-benefits-from-sparse-row-major-access-found-2026-08-19-not-implemented" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        query never benefits from sparse row-major access (found 2026-08-19, not implemented)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#implemented-2026-08-20" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Implemented (2026-08-20)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1587,7 +1709,7 @@ one-way function, not an invertible bijection with a stored inverse. Any
 method that reconstructs a kmer from a bare slot number is wrong by
 construction, regardless of the mechanism used (MPHF re-hash, or evidence
 decode + direct unitig read). See <code>MphfLayer::kmer_at</code>
-(<code>obilayeredmap/src/mphf_layer.rs</code>) — flagged for removal, currently called
+(<code>obikindex/src/layer/mphf_layer.rs</code>) — flagged for removal, currently called
 from <code>obikphylo/siblings/build.rs</code> and <code>family_scan.rs</code> (since removed — see
 "Pending work" status below).</p>
 <h2 id="two-pipelines-never-mixed">Two pipelines, never mixed</h2>
@@ -1650,7 +1772,7 @@ partition is unknown) and must keep going through
 <code>index()</code>.</p>
 <h2 id="pending-work-done">Pending work — done</h2>
 <p>The plan above shipped: <code>obikphylo</code> (a new crate — phylo-domain extension
-traits over <code>obikindex::KmerIndex</code>/<code>obilayeredmap::Layer&lt;D&gt;</code>, replacing the
+traits over <code>obikindex::KmerIndex</code>/<code>obikindex::layer::Layer&lt;D&gt;</code>, replacing the
 old <code>obikindex::siblings</code> module) builds and reads the annex purely in
 iteration order (<code>SiblingLayerExt::iter_siblings</code>/<code>iter_minorants</code>, both with
 batch variants, mirroring <code>Layer&lt;D&gt;</code>'s own <code>KmerIter</code>/<code>KmerBatchIter</code>
@@ -2218,13 +2340,13 @@ BRWT-style column-correlation exploitation.</p>
 <p><code>PersistentSparseBitMatrix</code> went from a validated but unused type to a
 real, selectable on-disk format:</p>
 <ul>
-<li><strong>Generic <code>Layer&lt;D&gt;</code></strong>: <code>obilayeredmap::Layer&lt;D&gt;</code>'s presence-only methods
+<li><strong>Generic <code>Layer&lt;D&gt;</code></strong>: <code>obikindex::layer::Layer&lt;D&gt;</code>'s presence-only methods
   (<code>n_cols</code>, <code>sub_matrix</code>, <code>fill_sub_matrix</code>) are generic over any
   <code>D: LayerData&lt;Item = Box&lt;[bool]&gt;&gt; + BinaryMatrix</code>, not hardcoded to
   <code>PersistentBitMatrix</code> — <code>PersistentSparseBitMatrix</code> implements
   <code>LayerData</code> (<code>open</code>/<code>read</code>) the same way. <code>find_slot</code>/<code>index_batch</code> were
   already generic over any <code>D: LayerData</code>, so they needed no change.
-  Verified by <code>obilayeredmap</code>'s
+  Verified by <code>obikindex::layer</code>'s
   <code>presence_layer_generic_over_sparse_matches_dense</code> test: build a dense
   presence layer, convert it to sparse via <code>build_from_dense</code>, open both
   as <code>Layer&lt;PersistentBitMatrix&gt;</code>/<code>Layer&lt;PersistentSparseBitMatrix&gt;</code> on
@@ -2234,7 +2356,7 @@ real, selectable on-disk format:</p>
   <code>AtomicUsize</code>s in test builds, not thread-local, so a test using a
   different <code>k</code> races every other test in the same crate binary; a k=11
   version of this test passed alone but failed under the full
-  <code>obilayeredmap</code> suite for exactly that reason before being fixed.)</li>
+  <code>obikindex::layer</code> suite for exactly that reason before being fixed.)</li>
 <li><strong><code>obikphylo::siblings::cache::Mat</code></strong> gained a third variant,
   <code>SparsePresence(Layer&lt;PersistentSparseBitMatrix&gt;)</code>, alongside <code>Count</code>
   and <code>Presence</code> — every method (<code>find_slot</code>, <code>index_batch</code>,
@@ -2264,7 +2386,7 @@ real, selectable on-disk format:</p>
   dense-path test (<code>sibling_annex_one_sibling_each</code>) exactly — proves the
   sparse format round-trips through the real build pipeline
   (<code>PartitionCache</code> sparse-detection included), not just the
-  <code>obicompactvec</code>/<code>obilayeredmap</code> unit layers below it.</li>
+  <code>obicompactvec</code>/<code>obikindex::layer</code> unit layers below it.</li>
 </ul>
 <p>Full workspace <code>cargo test</code> (all crates, unit + doc tests) green after
 this change.</p>
@@ -2310,7 +2432,7 @@ this change.</p>
   (activation: either given; defaults <code>1.0</code>/<code>0.5</code> for whichever is unset).</li>
 </ul>
 <p>Full workspace <code>cargo test</code> green after this change (167 unit tests in
-<code>obicompactvec</code>+<code>obilayeredmap</code>+<code>obikphylo</code> alone, plus every other
+<code>obicompactvec</code>+<code>obikindex::layer</code>+<code>obikphylo</code> alone, plus every other
 crate's suite, no regressions).</p>
 <p><strong>Still open, not part of this change</strong> (per "Correction to the 'single
 pass' framing" above): <code>--raw-snp-distance</code>/<code>--raw-snp-counts</code> (the
@@ -2319,6 +2441,261 @@ scan the full unsampled index — never threaded <code>--subsample</code>/<code>
 out of scope here since the reported problem was specifically about the
 <code>--sankoff</code>/<code>--tnt</code> pipeline's redundant/inconsistent scans, not these
 two standalone flags.</p>
+<h2 id="query-never-benefits-from-sparse-row-major-access-found-2026-08-19-not-implemented"><code>query</code> never benefits from sparse row-major access (found 2026-08-19, not implemented)</h2>
+<p>Benchmarked <code>obikmer query</code> against <code>global_index_presence</code> (dense-packed)
+vs. <code>global_index_presence_sparse</code> (<code>pack --sparse</code>), 100k simulated reads
+× 2 specimens (<code>benchmark/</code>, see
+<a href="../../implementation/benchmark_query_testing/">benchmark_query_testing.md</a>).
+Correctness: 0 mismatches — sparse and dense return bit-identical query
+results. Performance: sparse consistently <em>slower</em> than dense (~30-50%,
+reproducible across two runs with warm disk cache), the opposite of
+<code>pack --sparse</code>'s stated intent ("faster for single-row access... like
+query").</p>
+<p><strong>Root cause, read from source, not measured in isolation:</strong>
+<code>KmerPartition::query_partition_with</code> (<code>obikindex/src/partition/query_layer.rs:155-220</code>)
+is architecturally column-major: stage 2 walks <code>for g in 0..n_cols { for
+slot in hit_slots { layer.col_value(g, slot) } }</code>, documented (correctly)
+as the right locality strategy for the packed/columnar formats, where
+<code>col_value</code> → <code>PersistentBitMatrix::get</code> is a genuine O(1) mmap'd column
+read (<code>persistent.rs:110-113</code>).</p>
+<p>For <code>Self::Sparse</code>, that same <code>get(c, slot)</code> (<code>persistent.rs:114-118</code>)
+allocates a full <code>n_cols</code>-wide buffer and calls <code>fill_row</code> — materializing
+the <em>entire row</em> — just to return one cell. Called from inside the
+column-major double loop, this reconstructs the same row once per genome
+column touched: O(hits × n_cols) full-row rebuilds instead of O(hits).
+<code>PersistentSparseBitMatrix</code>'s own native row-major decode
+(<code>for_each_genome_in_row</code>, <code>sparse.rs:164-177</code>, used correctly by its own
+<code>row</code>/<code>fill_row</code>/<code>fill_sub_matrix</code>) is never reached from the query path
+at all.</p>
+<p><strong><code>fill_sub_matrix</code> (the existing <code>BinaryMatrix</code> trait primitive,
+<code>traits.rs:13-37</code>) is not the right replacement for <code>query</code> either</strong>, even
+once its own dispatch bug is fixed (see next section) — its output shape
+is inherently column-dense: <code>out[col]</code> gets an entry for every column,
+including columns with zero hits among the requested slots. On real
+sparse data (a hit typically touching a handful of genomes out of dozens)
+that's still O(n_cols) output regardless of true sparsity. What <code>query</code>
+actually wants is the sparse triple stream <code>(slot, col, value)</code> it already
+consumes as <code>QueryHit::Value</code> — not a materialized sub-matrix.</p>
+<p><strong>Proposed primitive</strong> (design only, not implemented — explicit ask: keep
+count matrices <em>not excluded</em>, even though effort right now is
+presence/absence only):</p>
+<p>Not a closure-driven <code>for_each</code> — a real <code>Iterator</code>, one concrete struct
+per matrix format, so the traversal state (current position in the sorted
+slot list, current column, permutation, sparse-row decode cursor…) lives
+in named struct fields instead of being threaded implicitly through
+recursion or a captured closure. RPITIT (stable since 1.75, and this
+workspace is edition 2024) means the trait method can return it without
+naming or boxing the concrete type:</p>
+<div class="highlight"><pre><span></span><code><span class="sd">/// Yields every nonzero cell among `slots`, in implementation-defined order.</span>
+<span class="k">fn</span><span class="w"> </span><span class="nf">nonzero_iter</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="p">(</span><span class="o">&amp;&#39;</span><span class="na">a</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">a</span><span class="w"> </span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="o">&lt;</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="o">&gt;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">&#39;</span><span class="na">a</span><span class="p">;</span>
+<span class="c1">// item: (idx into `slots`, col, value)</span>
+</code></pre></div>
+<p>This is the one foundational primitive per format — both <code>fill_sub_matrix</code>
+and a <code>for_each</code>-style callback become trivial, free consumers of it
+(<code>.for_each(f)</code> is already <code>Iterator::for_each</code> from std; <code>fill_sub_matrix</code>
+becomes "drain the iterator, scatter into <code>out[][]</code>"), instead of two
+independently-maintained traversals that can silently diverge (see the bug
+below — this is exactly how it happened).</p>
+<ul>
+<li><strong>On <code>PersistentSparseBitMatrix</code></strong>: the struct is nearly free to write —
+  it wraps the existing (currently private) <code>for_each_genome_in_row</code>
+  per-row decode, advancing to the next <code>slots</code> entry on exhaustion. O(Σ
+  row nnz), zero <code>n_cols</code>-wide allocation.</li>
+<li><strong>On <code>PersistentBitMatrix::{Packed,Columnar}</code></strong>: revised — cheaper than
+  first thought, by reusing the same split already used for
+  <code>fill_matrix</code>'s own implementation instead of hand-writing a resumable
+  state machine at the matrix level. The base-vector layer
+  (<code>BitSliceView</code>, <code>views.rs</code>) already separates the two concerns:
+  <code>fill_batch_sorted</code> (<code>views.rs:55-60</code>, sorted-slot batch lookup) and a
+  genuine per-bit <code>Iterator</code> (<code>BitSliceIter</code>, <code>views.rs:94+</code>) sit side by
+  side there, one level below the matrix. Adding a
+  "positions among <code>sorted_slots</code> where the bit is set" iterator at that
+  same vector level is a <code>filter</code> over the existing <code>get()</code> — no new
+  state machine, since <code>std::iter::Filter</code> already <em>is</em> one, generated by
+  the compiler:
+  <div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">nonzero_among_sorted</span><span class="o">&lt;&#39;</span><span class="na">s</span><span class="o">&gt;</span><span class="p">(</span><span class="o">&amp;&#39;</span><span class="na">s</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">sorted_slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">s</span><span class="w"> </span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="o">&lt;</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">usize</span><span class="o">&gt;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">&#39;</span><span class="na">s</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="n">sorted_slots</span><span class="p">.</span><span class="n">iter</span><span class="p">().</span><span class="n">copied</span><span class="p">().</span><span class="n">filter</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">|&amp;</span><span class="n">slot</span><span class="o">|</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">slot</span><span class="p">))</span>
+<span class="p">}</span>
+</code></pre></div>
+  The matrix-level <code>nonzero_iter</code> then composes these per column with
+  <code>flat_map</code> over <code>0..n_cols</code> (each column's hits, tagged with <code>c</code>,
+  slot mapped back through the sort permutation <code>fill_batch</code>/
+  <code>fill_batch_sorted</code> already carry) — again a combinator chain, not a
+  hand-rolled struct. Same algorithm, same mmap/sort locality as today's
+  <code>fill_sub_matrix</code>; just assembled from <code>std</code> iterator adaptors instead
+  of a loop body writing into a buffer, mirroring the vector/matrix split
+  the codebase already uses for <code>fill_batch_sorted</code> rather than
+  introducing a new shape.</li>
+<li><strong><code>Implicit</code></strong>: trivial (<code>slots.iter().map(|&amp;i| (i, 0, 1))</code>, one column,
+  always present).</li>
+<li><strong>On <code>PersistentCompactIntMatrix</code> (counts)</strong>: same treatment as
+  <code>Packed</code>/<code>Columnar</code> — no sparse count format exists yet ("Explicitly
+  deferred" per <code>traits.rs:9-12</code>), so no native low-effort case the way
+  <code>Sparse</code> has one, but not excluded either: the iterator's <code>Item</code> is
+  already <code>(usize, usize, u32)</code>, not <code>bool</code>, specifically so presence
+  (<code>0</code>/<code>1</code>) and counts (arbitrary <code>u32</code>) share one primitive instead of a
+  bool/u32 split forcing counts out of the design. Ready for a native
+  sparse-count struct later without a signature change.</li>
+</ul>
+<p>Would let <code>query_partition_with</code>'s stage 2 collapse to one
+<code>for (i, g, v) in layer.matrix().nonzero_iter(&amp;hit_slots) { on_event(...) }</code>
+per layer, format-agnostic, each backend's struct deciding the actual
+traversal.</p>
+<p><strong>This also closes the existing dispatch bug for free, by construction</strong>:
+<code>PersistentBitMatrix::fill_sub_matrix</code> (<code>persistent.rs:190-215</code>, the enum
+wrapper backing <code>BinaryMatrix</code>'s trait impl) today does <em>not</em> delegate to
+<code>PersistentSparseBitMatrix::fill_sub_matrix</code> for <code>Self::Sparse</code> — it
+reimplements the same naive per-(column, slot) <code>fill_row_bool</code> loop
+instead, bypassing the efficient native method one file over
+(<code>sparse.rs:249-258</code>). <code>obikphylo::siblings::cache::Mat</code>
+(<code>cache.rs:138-145</code>) independently built its own parallel enum wrapper
+that dispatches correctly — evidence this was worked around rather than
+fixed at the source: two hand-written traversals for the same format,
+free to drift apart, and they did. If <code>fill_sub_matrix</code> itself is
+rewritten as "drain <code>nonzero_iter</code>, scatter into <code>out[][]</code>", there is only
+one traversal per format left to get right — the bug class doesn't just
+get fixed once, it stops being possible to reintroduce.</p>
+<h2 id="implemented-2026-08-20">Implemented (2026-08-20)</h2>
+<p>Built as designed above, with one deviation from the original sketch:
+<code>nonzero_iter</code> ended up <code>Box&lt;dyn Iterator&lt;...&gt;&gt;</code>, not a bare <code>impl
+Iterator</code>, because <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are genuinely
+different concrete types and this method isn't on a trait (kept off
+<code>BinaryMatrix</code> deliberately — that trait is used as <code>dyn BinaryMatrix</code> in
+<code>tests/sparse.rs</code>, and RPITIT methods aren't dyn-compatible). One <code>Box</code>
+per <code>nonzero_iter</code> call, not per cell — negligible next to what it
+replaces.</p>
+<ul>
+<li><code>BitSliceView::nonzero_among_sorted</code> / <code>IntSliceView::nonzero_among_sorted</code>
+  (<code>obicompactvec/src/views.rs</code>): the vector-level <code>filter</code>/<code>filter_map</code>
+  primitive, exactly as sketched — no new state machine, <code>std</code>'s own.</li>
+<li><code>PersistentSparseBitMatrix::nonzero_iter</code> (<code>bitmatrix/sparse.rs</code>): native,
+  <code>std::iter::from_fn</code> over one buffered row at a time via the existing
+  <code>for_each_genome_in_row</code> — no <code>n_cols</code>-wide allocation, ever.</li>
+<li><code>PersistentBitMatrix::nonzero_iter</code> (<code>bitmatrix/persistent.rs</code>): dispatches
+  to the above for <code>Sparse</code>; for <code>Columnar</code>/<code>Packed</code>, loops columns,
+  collects each column's <code>nonzero_among_sorted</code> hits via <code>.extend()</code> (not
+  <code>flat_map</code> — a <code>flat_map</code> closure can't lazily return something
+  borrowing its own captured sort permutation across separate calls
+  without either boxing per-column or fighting the borrow checker; eager
+  collection into one <code>Vec</code> sidesteps it, at zero cost since
+  <code>fill_sub_matrix</code> already fully materialized anyway). <code>Implicit</code> trivial.</li>
+<li><code>PersistentBitMatrix::fill_sub_matrix</code> and <code>sub_matrix</code> rewritten to
+  drain <code>nonzero_iter</code> — the dispatch bug is gone because there is now
+  only one traversal per format, not because the old one was patched.
+  <code>PersistentCompactIntMatrix::nonzero_iter</code> added the same way (counts
+  not excluded, per the earlier ask) — no native low-effort case, since no
+  sparse count format exists, but on the same primitive, ready for one.</li>
+<li><code>KmerPartition::query_partition_with</code> (<code>obikindex/src/partition/query_layer.rs</code>):
+  stage 2's column-major <code>for g { for slot { col_value } }</code> replaced by one
+  <code>layer.nonzero_iter(&amp;slot_list)</code> call per layer, format-agnostic.</li>
+<li>Tests: <code>nonzero_iter_matches_dense</code>, <code>nonzero_iter_matches_row</code>, and —
+  the one that actually targets the dispatch bug rather than each type's
+  own correctness — <code>enum_wrapper_dispatches_to_native_sparse</code> (builds
+  <code>PersistentBitMatrix::Sparse(...)</code> directly, not through <code>open</code>, since
+  <code>open</code> only auto-detects <code>Sparse</code> from a <code>presence/</code> dir layout).
+  <code>cargo test --workspace</code>: green, no regressions.</li>
+</ul>
+<p><strong>Measured</strong>: re-ran the <code>benchmark/</code> query branch (100k reads × 2
+specimens, same setup as the original finding). Correctness still 0
+mismatches. The dense/sparse performance gap is gone — previously sparse
+~30-50% slower than dense, reproducibly; now within ~1-3% either way
+(7.42s dense vs 7.60s sparse for <code>Escherichia_coli--K-12_MG1655</code>; 5.25s vs
+5.30s for <code>Saccharolobus_islandicus--M.16.4</code>) — noise-level, not a
+systematic gap. <code>pack --sparse</code>'s claimed query win isn't confirmed
+outright by this (sparse should arguably now <em>beat</em> dense on truly sparse
+real data, not just tie), but the pathological regression is fixed.</p>
+<h2 id="persistentcompactintmatrixsparse-implemented-2026-08-26"><code>PersistentCompactIntMatrix::Sparse</code> — implemented (2026-08-26)</h2>
+<p>Closes the gap flagged throughout this document ("no sparse count format
+exists yet", <code>traits.rs:9-12</code>'s "Explicitly deferred"): <code>obicompactvec</code>
+already had <code>PersistentSparseCompactIntMatrix</code> (row-major, built on top of
+<code>PersistentSparseBitMatrix</code> as its "which columns are non-zero" support,
+values <em>not</em> deduplicated — see that struct's own doc comment), but it was
+never wired into <code>PersistentCompactIntMatrix</code>, the dense-dispatching enum
+every real consumer (<code>TypedLayer&lt;PersistentCompactIntMatrix&gt;</code>,
+<code>KmerLayer::Count</code>) actually holds. Concretely: <code>kmer_index.rs::
+pack_matrices(sparse=true)</code> already called <code>pack_sparse_compact_int_matrix</code>
+on every layer's <code>counts/</code> — but <code>PersistentCompactIntMatrix::open</code> had no
+code path back to what that just wrote, so a <code>Count</code> layer became
+unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
+anyone ran <code>pack --sparse</code> on an index with count layers. Root cause, not a
+workaround: add the missing <code>Sparse</code> variant.</p>
+<ul>
+<li>
+<p><strong>Enum + dispatch</strong> (<code>intmatrix.rs</code>): <code>PersistentCompactIntMatrix::Sparse
+  (PersistentSparseCompactIntMatrix)</code>, detected in <code>open</code>/<code>detect_storage</code>
+  via a <code>singleton_values.pciv</code> marker (mirrors <code>PersistentBitMatrix</code>'s own
+  <code>sparse_meta.json</code> check), reported via <code>storage_kind()</code>. <code>col</code>/
+  <code>col_view</code>/<code>col_persist</code> panic/<code>Unsupported</code> on <code>Sparse</code>, same convention
+  as the bit side. <code>sub_matrix</code>/<code>fill_sub_matrix</code> and <code>nonzero_iter</code>
+  unified the same way <code>PersistentBitMatrix</code>'s already are (drain
+  <code>nonzero_iter</code>, one traversal per format — see "Implemented
+  (2026-08-20)" above); <code>nonzero_iter</code> had to become <code>Box&lt;dyn Iterator&lt;...&gt;&gt;</code>
+  for the same reason (<code>Columnar</code>/<code>Packed</code>/<code>Sparse</code> are different concrete
+  types). No change needed in <code>obikindex</code> at all — <code>KmerLayer::Count</code>
+  already only ever holds <code>TypedLayer&lt;PersistentCompactIntMatrix&gt;</code>, so the
+  enum absorbing <code>Sparse</code> fixes the unreadable-layer bug for free, same as
+  <code>PersistentBitMatrix::Sparse</code> already did on the presence side.</p>
+</li>
+<li>
+<p><strong><code>CountPartials</code>, non-naive</strong> (<code>sparse_intmatrix.rs</code>): unlike
+  <code>PersistentSparseBitMatrix</code>'s dict-driven <code>col_weights_and_pair_counts</code>,
+  values here aren't deduplicated (two rows can share the same non-zero
+  column set via the same <code>dict_id</code> while carrying different counts), so
+  the "weight by how many rows share a dict entry" shortcut doesn't carry
+  over. What does: a single row-major pass (<code>row_major_pairwise</code>, decodes
+  each row once via <code>for_each_cell_in_row</code>, nests over that row's own
+  co-present columns) — <code>O(Σ k̄²)</code> over populated rows instead of the naive
+  <code>O(n_cols² × n)</code> column-pair rescan, same complexity class as the bit
+  side minus the dict multiplicity discount. Kernels used: <code>min(a,b)</code>
+  (bray, relfreq-bray — both vanish when either side is absent, so no
+  correction needed), <code>a·b</code> and <code>√(a·b)</code> (euclidean/relfreq-euclidean and
+  hellinger — these <em>do</em> need a correction, reconstructed from per-column
+  marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>, since <code>(a-0)² = a² ≠ 0</code> unlike
+  the <code>min</code>-based formulas). <code>threshold_jaccard(1)</code> shortcuts straight to
+  <code>support</code>'s own <code>BitPartials::partial_jaccard</code> (threshold 1 is exactly
+  presence); <code>threshold_jaccard(0)</code> is closed-form (every <code>u32</code> is <code>≥ 0</code>).</p>
+</li>
+<li>
+<p><strong>Two pre-existing bugs found and fixed while wiring the <code>threshold==1</code>
+  shortcut</strong> (<code>bitmatrix/sparse.rs</code>, <code>BitPartials for
+  PersistentSparseBitMatrix</code>, present since the 2026-08-15 implementation
+  above, never caught because no test compared <code>Sparse</code>'s raw <code>partial_*</code>
+  output against dense on real data — only the diagonal-blind
+  <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested):</p>
+</li>
+<li><code>partial_jaccard</code>'s diagonal was <code>(0, 2×col_weights[i])</code> instead of a
+     genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
+     <code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with
+     itself by construction.</li>
+<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: <code>total -
+     union</code> (count of rows where <em>neither</em> column is present) instead of
+     the actual Hamming distance <code>col_weights[i] + col_weights[j] -
+     2×inter[i,j]</code> (symmetric-difference size). Only coincides with the
+     correct value when <code>col_weights[i] + col_weights[j] == total</code>, so
+     small/synthetic test data could easily have hidden it.</li>
+</ul>
+<p>Neither surfaced through <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code>
+  (both explicitly zero their own diagonal at finalisation, and the
+  off-diagonal <code>partial_hamming</code> bug had gone untested against dense
+  entirely) — only visible to a caller of the raw <code>partial_*</code> methods
+  directly, which is exactly what <code>partial_threshold_jaccard(1)</code>'s new
+  shortcut became. Fixed at the source, not patched around at the call
+  site; regression test added:
+  <code>tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal</code>.</p>
+<ul>
+<li><strong>Tests</strong>: <code>tests::intmatrix::sparse_roundtrip_matches_columnar</code>/
+  <code>sparse_roundtrip_from_packed</code> (the <code>open</code>-dispatch fix, both build
+  paths); <code>tests::intmatrix::sparse_count_partials_match_dense</code> (all six
+  <code>CountPartials</code> formulas, thresholds 0/1/2/3, against <code>Columnar</code> on
+  asymmetric-presence data — this is what caught the diagonal gap in the
+  int side's own new code before it shipped, the same way it exposed the
+  two pre-existing bit-side bugs above); <code>obikindex</code>'s
+  <code>count_layer_transparently_reads_sparse_after_pack</code> — the actual
+  end-to-end regression test for the original "layer unreadable after
+  <code>pack --sparse</code>" bug, built → packed sparse → reopened, compared against
+  the pre-pack dense read. <code>cargo test -p obicompactvec -p obikindex</code>:
+  green, no regressions (180 + 12 tests).</li>
+</ul>
 
 
 

DevDoc/implementation/chunkreader.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/chunkreader/index.html

@@ -405,11 +405,11 @@
 </a>
 </li>
 <li class="md-nav__item">
-<a class="md-nav__link" href="../obilayeredmap/">
+<a class="md-nav__link" href="../layer_tier/">
 <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -497,6 +497,30 @@
   
 
     
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../benchmark_query_testing/">
+<span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../partition_layer_cache/">
+<span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
   </span>
 </a>
 </li>

DevDoc/implementation/evidence_elimination.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,9 +1270,9 @@
 <h1 id="coverage-implementationevidence_eliminationmd">Coverage: implementation/evidence_elimination.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()</li>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — build_approx_evidence(dir, b, z), find_approx()</li>
-<li><code>obilayeredmap/src/meta.rs</code> — EvidenceKind::Approx { b, z }, LayerMeta</li>
+<li><code>obikindex/src/layer/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()</li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — build_approx_evidence(dir, b, z), find_approx()</li>
+<li><code>obikindex/src/layer/meta.rs</code> — EvidenceKind::Approx { b, z }, LayerMeta</li>
 <li><code>obikindex/src/reindex.rs</code> — KmerIndex::reindex(), conversion exact↔approx en place</li>
 <li><code>obikmer/src/cmd/reindex.rs</code> — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size</li>
 <li><code>obikmer/src/cmd/index.rs</code> — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp</li>

DevDoc/implementation/evidence_elimination/index.html

@@ -12,7 +12,7 @@
         <link rel="prev" href="../unitig_evidence/">
       
       
-        <link rel="next" href="../obilayeredmap/">
+        <link rel="next" href="../layer_tier/">
       
       
         
@@ -868,14 +868,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1085,6 +1085,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/filtering/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1208,6 +1208,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -2130,7 +2186,7 @@ obikmer<span class="w"> </span>phylo<span class="w"> </span>myindex<span class="
 <h2 id="implementation">Implementation</h2>
 <ul>
 <li>
-<p><strong><code>obikpartitionner::filter::GroupQuorumFilter</code></strong> — implements <code>KmerFilter</code>
+<p><strong><code>obikindex::partition::filter::GroupQuorumFilter</code></strong> — implements <code>KmerFilter</code>
   using pre-computed ingroup and outgroup index vectors. The heavy logic
   (predicate parsing, three-value evaluation, genome classification) happens
   once before any iteration; each k-mer row evaluation is a simple index
@@ -2143,7 +2199,7 @@ obikmer<span class="w"> </span>phylo<span class="w"> </span>myindex<span class="
   list.</p>
 </li>
 <li>
-<p><strong><code>obikpartitionner::KmerPartition::iter_partition_kmers</code></strong> — accepts
+<p><strong><code>obikindex::partition::KmerPartition::iter_partition_kmers</code></strong> — accepts
   <code>filters: &amp;[Box&lt;dyn KmerFilter&gt;]</code> and applies them per-kmer before invoking
   the callback. <code>filter</code>, <code>dump</code>, and <code>unitig</code> all go through this single
   entry point.</p>

DevDoc/implementation/kmer.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/kmer/index.html

@@ -868,14 +868,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1085,6 +1085,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/layer_tier.refs/index.html

@@ -19,7 +19,7 @@
     
     
       
-        <title>Obilayeredmap.refs - obikmer</title>
+        <title>Layer tier.refs - obikmer</title>
       
     
     
@@ -60,7 +60,7 @@
     <div data-md-component="skip">
       
         
-        <a href="#coverage-implementationobilayeredmapmd" class="md-skip">
+        <a href="#coverage-implementationlayer_tiermd" class="md-skip">
           Skip to content
         </a>
       
@@ -96,7 +96,7 @@
         <div class="md-header__topic" data-md-component="header-topic">
           <span class="md-ellipsis">
             
-              Obilayeredmap.refs
+              Layer tier.refs
             
           </span>
         </div>
@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1211,15 +1267,15 @@
 
 
 <!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
-<h1 id="coverage-implementationobilayeredmapmd">Coverage: implementation/obilayeredmap.md</h1>
+<h1 id="coverage-implementationlayer_tiermd">Coverage: implementation/layer_tier.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()</li>
-<li><code>obilayeredmap/src/layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()</li>
-<li><code>obilayeredmap/src/map.rs</code> — LayeredMap<D>, push_layer(), query()</li>
-<li><code>obilayeredmap/src/evidence.rs</code> — Evidence, EvidenceWriter, encodage chunk_id:rank</li>
-<li><code>obilayeredmap/src/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, matches()</li>
-<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta, EvidenceKind (Exact / Approx { b, z })</li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()</li>
+<li><code>obikindex/src/layer/typed_layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()</li>
+<li><code>obikindex/src/layer/map.rs</code> — LayeredMap<D>, push_layer(), query()</li>
+<li><code>obikindex/src/layer/evidence.rs</code> — Evidence, EvidenceWriter, encodage chunk_id:rank</li>
+<li><code>obikindex/src/layer/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, matches()</li>
+<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta, EvidenceKind (Exact / Approx { b, z })</li>
 </ul>
 <h2 id="notes">Notes</h2>
 <p>FORT RISQUE DE DÉRIVE. C'est le fichier le plus affecté par les changements récents :

DevDoc/implementation/layer_tier/index.html

@@ -23,7 +23,7 @@
     
     
       
-        <title>obilayeredmap crate - obikmer</title>
+        <title>obikindex layer tier - obikmer</title>
       
     
     
@@ -64,7 +64,7 @@
     <div data-md-component="skip">
       
         
-        <a href="#obilayeredmap-layered-kmer-index-crate" class="md-skip">
+        <a href="#obikindexlayer-the-layer-tier" class="md-skip">
           Skip to content
         </a>
       
@@ -100,7 +100,7 @@
         <div class="md-header__topic" data-md-component="header-topic">
           <span class="md-ellipsis">
             
-              obilayeredmap crate
+              obikindex layer tier
             
           </span>
         </div>
@@ -749,7 +749,7 @@
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -767,7 +767,7 @@
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -940,6 +940,67 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#layerd-raw-mapping-iteration-and-batch-access" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Layer\&lt;D> — raw mapping, iteration, and batch access
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="Layer\&lt;D&gt; — raw mapping, iteration, and batch access">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#raw-kmer-slot-mapping" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Raw kmer → slot mapping
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#kmer-iteration" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Kmer iteration
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#batch-lookup-on-payload-vectorsviews" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Batch lookup on payload vectors/views
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#sub_matrix-fill_sub_matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        sub_matrix / fill_sub_matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1263,6 +1324,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1638,6 +1755,67 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#layerd-raw-mapping-iteration-and-batch-access" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Layer\&lt;D> — raw mapping, iteration, and batch access
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="Layer\&lt;D&gt; — raw mapping, iteration, and batch access">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#raw-kmer-slot-mapping" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Raw kmer → slot mapping
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#kmer-iteration" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Kmer iteration
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#batch-lookup-on-payload-vectorsviews" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Batch lookup on payload vectors/views
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#sub_matrix-fill_sub_matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        sub_matrix / fill_sub_matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1773,9 +1951,9 @@
 
 
 
-<h1 id="obilayeredmap-layered-kmer-index-crate">obilayeredmap — layered kmer index crate</h1>
+<h1 id="obikindexlayer-the-layer-tier">obikindex::layer — the Layer tier</h1>
 <h2 id="purpose">Purpose</h2>
-<p><code>obilayeredmap</code> implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a <code>ptr_hash</code> MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.</p>
+<p><code>obikindex::layer</code> (the <code>layer/</code> submodule of the <code>obikindex</code> crate — a standalone <code>obilayeredmap</code> crate until 2026-08-21, folded back in alongside <code>obikpartition</code> as part of a broader <code>Index { Partition { Layer } }</code> submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a <code>ptr_hash</code> MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.</p>
 <hr />
 <h2 id="three-usage-modes">Three usage modes</h2>
 <p>The MPHF + evidence infrastructure is the same for all modes. The <strong>payload</strong> varies.</p>
@@ -2000,6 +2178,38 @@
 </code></pre></div>
 <p>Mode 3 (<code>PersistentBitMatrix</code>) has no <code>push_layer</code> on <code>LayeredMap</code>; callers build directly via <code>Layer&lt;PersistentBitMatrix&gt;::build_presence</code>.</p>
 <hr />
+<h2 id="layerd-raw-mapping-iteration-and-batch-access">Layer\&lt;D&gt; — raw mapping, iteration, and batch access</h2>
+<p>Beyond <code>query</code>/<code>find</code> (membership-checked), <code>Layer&lt;D&gt;</code> exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.</p>
+<h3 id="raw-kmer-slot-mapping">Raw kmer → slot mapping</h3>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">index</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kmer</span><span class="p">:</span><span class="w"> </span><span class="nc">CanonicalKmer</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">usize</span>
+<span class="nc">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">index_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kmers</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="n">CanonicalKmer</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">usize</span><span class="o">&gt;</span>
+</code></pre></div>
+<p>Pure MPHF mapping, no evidence/fingerprint check — equivalent to <code>MphfOnly::index</code>. Only meaningful when the caller already knows <code>kmer</code> belongs to the layer; on an absent kmer the MPHF still returns <em>some</em> slot (undefined, not <code>None</code>).</p>
+<h3 id="kmer-iteration">Kmer iteration</h3>
+<p>Four iterators, all built from <code>unitigs.bin</code> (physical layout order, <strong>not</strong> correlated with MPHF slot numbers):</p>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">iter_kmers</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">KmerIter</span><span class="o">&lt;&#39;</span><span class="nb">_</span><span class="o">&gt;</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">enumerate_kmers</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Enumerate</span><span class="o">&lt;</span><span class="n">KmerIter</span><span class="o">&lt;&#39;</span><span class="nb">_</span><span class="o">&gt;&gt;</span><span class="w">              </span><span class="c1">// (order_index, kmer)</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">iter_kmers_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">KmerBatchIter</span><span class="o">&lt;&#39;</span><span class="nb">_</span><span class="o">&gt;</span><span class="w">          </span><span class="c1">// Vec&lt;CanonicalKmer&gt; of size ≤ n</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">enumerate_kmers_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="o">&lt;</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="n">CanonicalKmer</span><span class="o">&gt;</span><span class="p">)</span><span class="o">&gt;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">&#39;</span><span class="nb">static</span>
+</code></pre></div>
+<p><code>KmerIter</code>/<code>KmerBatchIter</code> own a clone of the underlying <code>Arc&lt;UnitigFileReader&gt;</code> rather than borrowing <code>self</code> — <code>Send + 'static</code>, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. <code>enumerate_kmers_batch</code>'s index is the batch's starting offset in iteration order (a multiple of <code>n</code> except for the final, possibly shorter, batch).</p>
+<h3 id="batch-lookup-on-payload-vectorsviews">Batch lookup on payload vectors/views</h3>
+<p><code>PersistentCompactIntVec</code>, <code>PersistentBitVec</code>, <code>IntSliceView</code>, <code>BitSliceView</code> all expose:</p>
+<div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">get_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span>
+<span class="k">fn</span><span class="w"> </span><span class="nf">fill_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="n">T</span><span class="p">])</span>
+</code></pre></div>
+<p>Both sort <code>slots</code> internally for sequential mmap access, then reorder results back to the caller's original order. <code>fill_batch</code> fills a caller-provided buffer, avoiding the <code>Vec</code> allocation.</p>
+<h3 id="sub_matrix-fill_sub_matrix">sub_matrix / fill_sub_matrix</h3>
+<div class="highlight"><pre><span></span><code><span class="c1">// Layer&lt;PersistentCompactIntMatrix&gt;</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;&gt;</span><span class="w">            </span><span class="c1">// column-first</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;</span><span class="p">])</span>
+
+<span class="c1">// Layer&lt;PersistentBitMatrix&gt; (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;&gt;</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;</span><span class="p">])</span>
+</code></pre></div>
+<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this degrades to a row-by-row decode; see <a href="../../architecture/siblings/">siblings.md</a>.</p>
+<hr />
 <h2 id="layeredstores-and-aggregation-traits">LayeredStore\&lt;S&gt; and aggregation traits</h2>
 <p><code>LayeredStore&lt;S&gt;</code> is a generic aggregation wrapper over <code>Vec&lt;S&gt;</code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
 <div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayeredStore</span><span class="o">&lt;</span><span class="n">S</span><span class="o">&gt;</span><span class="p">(</span><span class="k">pub</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="n">S</span><span class="o">&gt;</span><span class="p">);</span>

DevDoc/implementation/merge.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1215,9 +1271,9 @@
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
 <li><code>obikindex/src/merge.rs</code> — <code>KmerIndex::merge()</code>, validation de compatibilité d'évidence, <code>validate_evidence_compat()</code></li>
-<li><code>obikpartitionner/src/merge_layer.rs</code> — <code>merge_partition()</code>, construction de la nouvelle layer, paramètre <code>block_bits</code></li>
-<li><code>obikpartitionner/src/rebuild_layer.rs</code> — <code>rebuild_partition()</code>, paramètre <code>block_bits</code></li>
-<li><code>obilayeredmap/src/layer.rs</code> — <code>Layer::append_genome_column()</code> (PersistentCompactIntMatrix et PersistentBitMatrix)</li>
+<li><code>obikindex/src/partition/merge_layer.rs</code> — <code>merge_partition()</code>, construction de la nouvelle layer, paramètre <code>block_bits</code></li>
+<li><code>obikindex/src/partition/rebuild_layer.rs</code> — <code>rebuild_partition()</code>, paramètre <code>block_bits</code></li>
+<li><code>obikindex/src/layer/typed_layer.rs</code> — <code>Layer::append_genome_column()</code> (PersistentCompactIntMatrix et PersistentBitMatrix)</li>
 <li><code>obicompactvec/src/intmatrix.rs</code> — <code>append_column</code> pour PersistentCompactIntMatrix</li>
 <li><code>obicompactvec/src/bitmatrix.rs</code> — <code>append_column</code> pour PersistentBitMatrix</li>
 </ul>

DevDoc/implementation/merge/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1034,6 +1034,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1157,6 +1168,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1542,6 +1609,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1739,6 +1817,14 @@ spectrums/
   &lt;label&gt;.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 </code></pre></div>
+<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
+<h2 id="known-issue-not-yet-fixed-2026-08-28">Known issue (not yet fixed, 2026-08-28)</h2>
+<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank &lt; self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
+<ul>
+<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
+<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
+</ul>
+<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
 
 
 

DevDoc/implementation/merge_parallelism/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1152,6 +1152,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/mphf.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,8 +1270,8 @@
 <h1 id="coverage-implementationmphfmd">Coverage: implementation/mphf.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, <code>build()</code>, <code>build_exact_evidence()</code>, <code>build_approx_evidence()</code>, <code>build_evidence()</code></li>
-<li><code>obikpartitionner/src/index_layer.rs</code> — <code>build_index_layer()</code> avec passage de <code>block_bits</code></li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, <code>build()</code>, <code>build_exact_evidence()</code>, <code>build_approx_evidence()</code>, <code>build_evidence()</code></li>
+<li><code>obikindex/src/partition/index_layer.rs</code> — <code>build_index_layer()</code> avec passage de <code>block_bits</code></li>
 </ul>
 <h2 id="notes">Notes</h2>
 <p>FORT RISQUE DE DÉRIVE. Changements récents :

DevDoc/implementation/mphf/index.html

@@ -946,14 +946,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1163,6 +1163,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1577,7 +1633,7 @@
 <h2 id="why-two-phases-are-needed">Why two phases are needed</h2>
 <p>Kmer indexing per partition proceeds in two phases. The separation is necessary because the exact number of surviving unique kmers is not known until after counting and filtering low-abundance kmers.</p>
 <h3 id="phase-1-provisional-mphf-kmer-spectrum">Phase 1 — provisional MPHF + kmer spectrum</h3>
-<p>Implemented in <code>obikpartitionner::KmerPartition::count_kmer()</code> → <code>count_partition()</code>.</p>
+<p>Implemented in <code>obikindex::partition::KmerPartition::count_kmer()</code> → <code>count_partition()</code>.</p>
 <ol>
 <li><strong>External sort</strong>: read the dereplicated superkmer file; extract the raw <code>u64</code> canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: <code>sorted_unique.bin</code> — a flat array of f0 distinct sorted <code>u64</code> values. Exact kmer count f0 is known at this point.</li>
 <li><strong>Build provisional MPHF</strong> (ptr_hash, same configuration as phase 2) over <code>sorted_unique.bin</code> using <code>new_from_par_iter</code>. Delete <code>sorted_unique.bin</code> immediately after. Persist to <code>mphf1.bin</code>.</li>
@@ -1667,7 +1723,7 @@
 <hr />
 <h2 id="multilayer-index-architecture">Multilayer index architecture</h2>
 <h3 id="layer-structure">Layer structure</h3>
-<p>Each layer is a self-contained unit. See <a href="../obilayeredmap/">obilayeredmap</a> for the full on-disk layout. The MPHF-relevant files are:</p>
+<p>Each layer is a self-contained unit. See <a href="../layer_tier/">obikindex::layer</a> for the full on-disk layout. The MPHF-relevant files are:</p>
 <div class="highlight"><pre><span></span><code>layer_i/
   unitigs.bin      — packed 2-bit nucleotide sequences (kmer evidence source)
   unitigs.bin.idx  — random-access block index (block_bits controls granularity)
@@ -1702,7 +1758,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
     Uses open_sequential(); never writes .idx
 </code></pre></div>
 <p>There is no <code>build_evidence</code> dispatch wrapper. Callers choose the appropriate post-hoc build directly.</p>
-<p>In <code>obikpartitionner</code>, <code>build_index_layer</code> receives <code>block_bits: u8</code> from <code>IndexConfig::block_bits</code> and forwards it directly to <code>Layer::build</code> and <code>Layer::build_approx_evidence</code>.</p>
+<p>In <code>obikindex::partition</code>, <code>build_index_layer</code> receives <code>block_bits: u8</code> from <code>IndexConfig::block_bits</code> and forwards it directly to <code>Layer::build</code> and <code>Layer::build_approx_evidence</code>.</p>
 <h3 id="membership-verification">Membership verification</h3>
 <p>ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry:</p>
 <ul>

DevDoc/implementation/obicompactvec/index.html

@@ -320,11 +320,11 @@
 </a>
 </li>
 <li class="md-nav__item">
-<a class="md-nav__link" href="../obilayeredmap/">
+<a class="md-nav__link" href="../layer_tier/">
 <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -412,6 +412,30 @@
   
 
     
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../benchmark_query_testing/">
+<span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../partition_layer_cache/">
+<span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
   </span>
 </a>
 </li>
@@ -1144,29 +1168,41 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
 </code></pre></div>
 <hr/>
 <h2 id="matrix-types">Matrix types</h2>
-<p>Four matrix types, two encodings × two formats:</p>
+<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
 <table>
 <thead>
 <tr>
-<th></th>
-<th>Columnar format</th>
-<th>Packed format</th>
+<th>Variant</th>
+<th>Storage</th>
+<th>When</th>
 </tr>
 </thead>
 <tbody>
 <tr>
-<td><strong>Bit</strong></td>
-<td><code>PersistentBitMatrix</code> (Columnar variant)</td>
-<td><code>PersistentBitMatrix</code> (Packed variant)</td>
+<td><code>Columnar</code></td>
+<td>one <code>.pbiv</code>/<code>.pciv</code> file per column + <code>meta.json</code></td>
+<td>build-time default (<code>*Builder::new</code>)</td>
 </tr>
 <tr>
-<td><strong>Int</strong></td>
-<td><code>PersistentCompactIntMatrix</code> (Columnar variant)</td>
-<td><code>PersistentCompactIntMatrix</code> (Packed variant)</td>
+<td><code>Packed</code></td>
+<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
+<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
+</tr>
+<tr>
+<td><code>Sparse</code></td>
+<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
+<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
+</tr>
+<tr>
+<td><code>Implicit</code> (bit only)</td>
+<td>no file at all</td>
+<td>mono-genome presence layers — <code>n_cols</code> is always reported as <code>1</code>, every value is <code>true</code></td>
 </tr>
 </tbody>
 </table>
-<p>Both matrix types are enums (<code>Columnar</code> / <code>Packed</code> / <code>Implicit</code> for bit) behind a transparent API. <code>col_view(c)</code> returns the appropriate view directly:</p>
+<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
+<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
+<p><code>col_view(c)</code> returns the appropriate view directly:</p>
 <div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
 <span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o">&lt;'</span><span class="nb">_</span><span class="o">&gt;</span>
 

DevDoc/implementation/obipipeline.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1219,7 +1275,7 @@
 </ul>
 <h2 id="notes">Notes</h2>
 <p>Document stable (librairie générique, peu de risque de dérive).
-Vérifier si <code>obipipeline</code> est toujours utilisé dans la phase scatter de <code>obikpartitionner</code>
+Vérifier si <code>obipipeline</code> est toujours utilisé dans la phase scatter de <code>obikindex::partition</code>
 ou s'il a été remplacé par Rayon dans certains chemins.</p>
 
 

DevDoc/implementation/obipipeline/index.html

@@ -896,14 +896,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1113,6 +1113,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/obitaxonomy/index.html

@@ -12,7 +12,7 @@
         <link rel="prev" href="../select/">
       
       
-        <link rel="next" href="../../architecture/sequences/invariant/">
+        <link rel="next" href="../benchmark_query_testing/">
       
       
         
@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1125,6 +1125,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_bit_vec.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_bit_vec/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1225,6 +1225,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_compact_int_vec.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_compact_int_vec/index.html

@@ -9,7 +9,7 @@
       
       
       
-        <link rel="prev" href="../obilayeredmap/">
+        <link rel="prev" href="../layer_tier/">
       
       
         <link rel="next" href="../persistent_bit_vec/">
@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1208,6 +1208,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/pipeline.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,12 +1270,12 @@
 <h1 id="coverage-implementationpipelinemd">Coverage: implementation/pipeline.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obikpartitionner/src/partition.rs</code> — estimation des paramètres (phase 0)</li>
+<li><code>obikindex/src/partition/partition.rs</code> — estimation des paramètres (phase 0)</li>
 <li><code>obiskbuilder/src/iter.rs</code> — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)</li>
-<li><code>obikpartitionner/src/filter.rs</code> — déduplication bucket-sort (phase 2)</li>
-<li><code>obikpartitionner/src/kmer_sort.rs</code> — tri externe + agrégation de comptages (phase 3)</li>
+<li><code>obikindex/src/partition/filter.rs</code> — déduplication bucket-sort (phase 2)</li>
+<li><code>obikindex/src/partition/kmer_sort.rs</code> — tri externe + agrégation de comptages (phase 3)</li>
 <li><code>obidebruinj/src/debruijn.rs</code> — graphe De Bruijn, extraction des unitigs (phase 5)</li>
-<li><code>obikpartitionner/src/index_layer.rs</code> — construction MPHF + évidence (phase 6), paramètre <code>block_bits</code></li>
+<li><code>obikindex/src/partition/index_layer.rs</code> — construction MPHF + évidence (phase 6), paramètre <code>block_bits</code></li>
 <li><code>obikindex/src/index.rs</code> — <code>build_layers()</code>, <code>dereplicate_and_count()</code></li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/implementation/pipeline/index.html

@@ -857,14 +857,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1074,6 +1074,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1546,7 +1602,7 @@ branching / dead-end → unitig start or end
       counts/                ← PersistentCompactIntMatrix (with_counts only)
 </code></pre></div>
 <p><strong>Cleanup:</strong> unless <code>--keep-intermediate</code> is set, <code>remove_build_artifacts</code> deletes <code>dereplicated.skmer.zst</code>, <code>mphf1.bin</code>, and <code>counts1.bin</code> after all partitions are indexed.</p>
-<p>See <a href="../obilayeredmap/">obilayeredmap</a> and <a href="../mphf/">MPHF selection</a> for data structure details.</p>
+<p>See <a href="../layer_tier/">obikindex::layer</a> and <a href="../mphf/">MPHF selection</a> for data structure details.</p>
 <p><strong>Query path (exact evidence):</strong></p>
 <div class="highlight"><pre><span></span><code>query kmer q
   → canonical_minimizer(q) → hash → PART → part_XXXXX/

DevDoc/implementation/select/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1119,17 +1119,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1157,6 +1146,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1196,6 +1202,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1582,17 +1644,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1620,6 +1671,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1672,23 +1740,32 @@ are preserved unchanged; only the data matrices are rewritten.</p>
 <hr />
 <h2 id="synopsis">Synopsis</h2>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>&lt;input-index&gt;
-<span class="w">        </span><span class="o">{</span><span class="w"> </span>--output<span class="w"> </span>&lt;dir&gt;<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="w"> </span><span class="o">}</span>
+<span class="w">        </span>--output<span class="w"> </span>&lt;dir&gt;
 <span class="w">        </span><span class="o">[</span>--group<span class="w">    </span>&lt;name&gt;:&lt;pred&gt;<span class="w">  </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--group-op<span class="w"> </span>&lt;name&gt;:&lt;op&gt;<span class="w">    </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-by<span class="w"> </span>&lt;key&gt;<span class="w">          </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-op<span class="w"> </span>&lt;op&gt;<span class="w">           </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--select<span class="w">   </span>&lt;col1,col2,...&gt;<span class="w">    </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--presence-threshold<span class="w"> </span>&lt;N&gt;<span class="w">      </span><span class="o">]</span>
+<span class="w">        </span><span class="o">[</span>--force-copy<span class="w">                  </span><span class="o">]</span>
 </code></pre></div>
 <hr />
 <h2 id="output-destination">Output destination</h2>
-<p>Exactly one of <code>--output</code> or <code>--in-place</code> must be specified.</p>
-<p><strong><code>--output &lt;dir&gt;</code></strong> — writes a new index to <code>&lt;dir&gt;</code>. The source index is
-unchanged. The MPHF and unitig files are copied; only the data matrices are
-rewritten with the new column layout.</p>
-<p><strong><code>--in-place</code></strong> — rewrites the data matrices of the source index directly.
-Removed or replaced columns are lost. The operation writes to temporary files
-first, then renames atomically, so an interrupted run leaves the index intact.</p>
+<p><code>--output &lt;dir&gt;</code> is required — <code>select</code> always writes a new index; there is no
+<code>--in-place</code> mode (2026-08-28: never implemented, removed from the design). The
+source index is unchanged.</p>
+<p>Each layer's kmer-identity files (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/
+<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code>) are never rewritten by a
+column projection/aggregation, so they are hard-linked into the output rather
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); <code>--force-copy</code>
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+<code>select</code> itself would affect the other). Only the <code>presence</code>/<code>counts</code>
+subdirectory is ever a genuinely new, independent file.</p>
+<p>To replace an index with a selected version of itself, select to a temporary
+directory and swap it in (<code>rm -rf INDEX &amp;&amp; mv INDEX.tmp INDEX</code>) — the case
+<code>--in-place</code> used to cover.</p>
 <hr />
 <h2 id="defining-output-columns">Defining output columns</h2>
 <h3 id="named-groups-group">Named groups — <code>--group</code></h3>
@@ -1852,9 +1929,6 @@ is &gt; N.</p>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>out<span class="w"> </span><span class="se">\</span>
 <span class="w">  </span>--select<span class="w"> </span><span class="s2">&quot;Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01&quot;</span>
 </code></pre></div>
-<h3 id="in-place-keep-only-group-a">In-place: keep only group A</h3>
-<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--in-place<span class="w"> </span>--group<span class="w"> </span><span class="s2">&quot;A:group=A&quot;</span><span class="w"> </span>--select<span class="w"> </span><span class="s2">&quot;A&quot;</span>
-</code></pre></div>
 <h3 id="compose-with-filter">Compose with filter</h3>
 <div class="highlight"><pre><span></span><code><span class="c1"># Step 1: keep only B. nana-specific k-mers</span>
 obikmer<span class="w"> </span>filter<span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>filtered<span class="w"> </span><span class="se">\</span>
@@ -1865,32 +1939,46 @@ obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </s
 </code></pre></div>
 <hr />
 <h2 id="implementation-notes">Implementation notes</h2>
-<p><code>select</code> does not rebuild the MPHF. The 256 partitions are processed in parallel
-(rayon), each writing its output independently; results require no synchronisation
-because every partition owns a distinct set of files.</p>
-<p>For each layer in each partition:</p>
+<p><code>select</code> does not rebuild the MPHF. Every partition is processed independently
+(<code>PartitionRunner</code>), each writing its own output layers; no cross-partition
+synchronisation is needed.</p>
+<p>For each layer in each partition (<code>obikselect::select_layer::select_partition</code>):</p>
 <ol>
-<li>The slot count <code>n</code> is read by opening the source data matrix.</li>
-<li>A new data matrix is built with M columns (M = number of output columns).</li>
-<li>For each slot <code>s</code> in <code>0..n</code>:</li>
-<li><code>old_row = matrix.fill_row(s)</code> — reads the original <code>N</code>-column row without allocating.</li>
-<li>For each output column <code>j</code>:<ul>
-<li><code>new_row[j] = aggregate(op, old_row[group_indices])</code>.</li>
-<li>Pass-through columns are represented as single-element groups with the
-   default operator (<code>any</code> for presence, <code>sum</code> for count) — same code path.</li>
-</ul>
-</li>
-<li>The new row is written slot by slot into each column builder.</li>
-<li>All plain files in the source layer directory (<code>mphf.bin</code>, <code>unitigs.bin</code>,
-   evidence files, <code>layer_meta.json</code>) are copied verbatim; only the <code>presence/</code>
-   or <code>counts/</code> subdirectory is rewritten.</li>
+<li><code>copy_layer_files</code> hard-links the source layer's kmer-identity files
+   (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/
+   <code>fingerprint.bin</code>/<code>layer_meta.json</code>) into the destination — never a real
+   copy unless linking fails or <code>--force-copy</code> is given.</li>
+<li>A new data matrix is built with M columns (M = number of output columns),
+   under a fresh <code>presence/</code>/<code>counts/</code> subdirectory (never touching the
+   source's own).</li>
+<li><strong>Presence source (2026-08-28: <code>batch_presence_counts</code>)</strong>: one shared pass
+   over the source bit matrix computes every output group's presence count at
+   once — row-major native for a <code>Sparse</code> source (<code>for_each_genome_in_row</code>,
+   which has no column representation to read a <code>col_view</code> from at all — the
+   reason this replaced the old per-group loop, not just an optimisation of
+   it), deduplicated column-major (one <code>col_view</code> per <em>distinct referenced
+   column</em>, not per group) for <code>Columnar</code>/<code>Packed</code>. Every <code>AggOp</code> for a bit
+   matrix is then a cheap derivation of that one count vector (<code>sum</code> = the
+   count itself, <code>any</code>/<code>max</code> = <code>count ≥ 1</code>, <code>all</code>/<code>min</code> = <code>count == group
+   size</code>, <code>none</code> = <code>count == 0</code>) — see
+   <code>obikselect::select_layer::agg_result_from_count</code>.</li>
+<li><strong>Count source</strong>: unchanged, one <code>col_view</code>-driven pass per output column
+   via <code>MatrixGroupOps</code> — <code>sum</code>/<code>min</code>/<code>max</code> are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.</li>
 <li><code>index.meta</code> is rewritten with the new genome list and updated <code>with_counts</code>.</li>
 </ol>
-<p><strong><code>--in-place</code> write strategy:</strong> new data is written to a temporary sibling
-directory (<code>presence_new/</code> or <code>counts_new/</code>); on success the old directory is
-removed and the temporary one is renamed into place. An interrupted run leaves
-at most one stale <code>*_new/</code> directory; the original data is intact until the
-rename step.</p>
+<h3 id="known-gap-not-yet-fixed-2026-08-28">Known gap (not yet fixed, 2026-08-28)</h3>
+<p>Step 4 above still panics (<code>col_view() not available on Sparse
+PersistentCompactIntMatrix</code>) if the source is a <strong>count</strong> index packed
+sparse — <code>batch_presence_counts</code>' row-major treatment was only ported to the
+bit-matrix (<code>Presence</code>) case, since that was the one actually blocking a real
+benchmark run. <code>select</code>/<code>filter</code> on a sparse-packed count index still hits
+this; the fix would follow the same shape (a <code>PersistentSparseCompactIntMatrix</code>
+row-major decode, analogous to <code>for_each_genome_in_row</code>), just not done. Since
+<code>obisys::numa::runner::PartitionRunner</code>'s panic-propagation fix (see
+<code>architecture/numa_partition_runner.md</code>), this at least fails fast (process
+panic, exit 101) instead of hanging.</p>
 
 
 

DevDoc/implementation/storage.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1216,7 +1272,7 @@
 <ul>
 <li><code>obikindex/src/meta.rs</code> — IndexMeta, IndexConfig (version, config, genomes)</li>
 <li><code>obikindex/src/index.rs</code> — layout sur disque : partitions/, index.meta</li>
-<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta (evidence kind), PartitionMeta (n_layers)</li>
+<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta (evidence kind), PartitionMeta (n_layers)</li>
 <li><code>obiskio/src/unitig_index.rs</code> — fichiers unitigs.bin + unitigs.bin.idx</li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/implementation/storage/index.html

@@ -929,14 +929,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1146,6 +1146,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/superkmer.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/superkmer/index.html

@@ -835,14 +835,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1052,6 +1052,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/unitig_evidence.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1215,7 +1271,7 @@
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
 <li><code>obiskio/src/unitig_index.rs</code> — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()</li>
-<li><code>obilayeredmap/src/evidence.rs</code> — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter</li>
+<li><code>obikindex/src/layer/evidence.rs</code> — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter</li>
 <li><code>obidebruinj/src/debruijn.rs</code> — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK</li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/implementation/unitig_evidence/index.html

@@ -957,14 +957,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1174,6 +1174,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/index.html

@@ -831,14 +831,14 @@
   
   
     <li class="md-nav__item">
-      <a href="implementation/obilayeredmap/" class="md-nav__link">
+      <a href="implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1048,6 +1048,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/index.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/installation/index.html

@@ -917,14 +917,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1134,6 +1134,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/kmers.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/kmers/index.html

@@ -830,14 +830,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1047,6 +1047,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/encoding.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/encoding/index.html

@@ -802,14 +802,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1019,6 +1019,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/entropy.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/entropy/index.html

@@ -868,14 +868,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1085,6 +1085,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/evolutionary_distances/index.html

@@ -909,6 +909,56 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        --distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#snp-distance-catalog" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        snp-* distance catalog
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Exact formulas (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Output format: PHYLIP-relaxed by default for the distance matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1244,14 +1294,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1461,6 +1511,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -2139,6 +2245,56 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        --distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#snp-distance-catalog" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        snp-* distance catalog
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Exact formulas (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Output format: PHYLIP-relaxed by default for the distance matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -3554,7 +3710,7 @@ diagonal entry directly, at no extra lookup cost.</p>
 <p>Mirror <code>distance.rs</code>: open the presence or count store per partition. But
 instead of a per-partition <code>partial_*</code>, run the sequential source sweep:</p>
 <div class="highlight"><pre><span></span><code>for p in 0..n_partitions:                       # OUTER — sequential
-    open source partition p&#39;s layers (QueryLayer-style, obikpartitionner)
+    open source partition p&#39;s layers (QueryLayer-style, obikindex::partition)
     enumerate distinct canonical k-mers of p (one per MPHF slot) with their
         presence/count vectors                   # column-major, as query stage 2
     par_iter over these source k-mers:           # INNER — rayon, thread-local tally
@@ -3577,7 +3733,7 @@ instead of a per-partition <code>partial_*</code>, run the sequential source swe
     merge thread-local tallies into global SnpTally
 </code></pre></div>
 <p>The inner lookup is precisely <code>QueryLayer::find_slot</code> +
-<code>col_value(g, slot)</code> (<code>obikpartitionner/src/query_layer.rs</code>) — reuse or factor
+<code>col_value(g, slot)</code> (<code>obikindex/src/partition/query_layer.rs</code>) — reuse or factor
 out that path rather than reimplementing MPHF access. Enumerating "all distinct
 k-mers of a partition with their vectors" is the <code>dump</code>/<code>query</code> stage-2
 column-major scan already implemented in <code>dump_layer.rs</code> /
@@ -4205,6 +4361,249 @@ among the survivors) — a single extra pass is sufficient.</p>
 <code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
 from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
 workspace <code>cargo test</code> green.</p>
+<h2 id="-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28"><code>--distance</code> unification: SNP corrections as first-class metrics (implemented, 2026-08-28)</h2>
+<p><strong>Implemented.</strong> <code>--metric</code> (renamed <code>--distance</code> — several of
+its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
+<code>--metric</code> was a misnomer) gains a family of <code>snp-*</code> values computed from the
+central-position SNP pipeline, routed internally to the sibling-annex
+machinery (<code>PairwiseTally</code>, <code>obikphylo::siblings::algorithms::pairwise</code>)
+instead of <code>cache.distance(...)</code>'s existing per-layer traversal — a different
+code path behind the same CLI surface, not just another branch of one
+formula function.</p>
+<p><strong>Why unify at the CLI level despite the implementation split</strong>: phylogenetically
+a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
+to how the matrix was produced, so exposing it as a special-cased subcommand
+instead of a <code>--distance</code> value would misrepresent its role. The
+implementation divergence (sibling-annex-based vs. plain index scan) is real
+but belongs at the routing layer, invisible to the CLI's own vocabulary.</p>
+<p><strong><code>--subsample</code> becomes optional for <code>snp-*</code> distances</strong> (it stays mandatory
+for <code>--sankoff</code>/<code>--pseudo-alignment</code>, unrelated commands): absent means
+exhaustive, achieved for free by reusing <code>sample_index</code>'s existing
+proportional-per-layer-quota mechanism with <code>n</code> set to the index-wide total
+non-monomorphic-minorant count (already available from the sibling-annex
+stats) — every layer's quota then equals its own full count, giving Bernoulli
+<code>p = 1</code> everywhere, i.e. every eligible family is drawn. No second,
+exhaustive-only driver needed. Present means sampled, exactly as <code>--sankoff</code>
+already behaves.</p>
+<p><strong>One shared tally, many derived formulas.</strong> <code>PairwiseTally</code>'s <code>subst[4][4]</code>
+per-pair substitution counts (plus marginal base frequencies derived from it)
+are the sufficient statistic for every closed-form correction below — each
+is a small pure function <code>PairwiseTally -&gt; Array2&lt;f64&gt;</code>, at the same level as
+the already-implemented <code>raw_snp_distance</code>/<code>base_pair_tally</code>/
+<code>cardinality_tally</code>. No new full scan per formula, whether the tally itself
+was built exhaustively or from a subsample.</p>
+<p><strong><code>--raw-snp-counts</code> stays a separate, unrelated flag</strong> — same underlying
+tally, but a diagnostic (<code>n_snp</code>/<code>n_shared</code>/<code>n_eligible</code> per genome pair, one
+row per pair) rather than a distance value, and its long-table shape doesn't
+fold into a single N×N matrix the way a distance does. No change to its
+existing CSV format.</p>
+<h3 id="snp-distance-catalog"><code>snp-*</code> distance catalog</h3>
+<p>All closed-form (method-of-moments / direct formula), none requiring
+per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
+<em>tree</em>-ML usage but not its <em>pairwise</em> estimator, which is closed-form like
+F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
+<table>
+<thead>
+<tr>
+<th>value</th>
+<th>corrects for</th>
+<th>inputs beyond raw counts</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>snp-raw</code></td>
+<td>nothing (uncorrected p-distance)</td>
+<td>—</td>
+</tr>
+<tr>
+<td><code>snp-jc</code> (Jukes-Cantor, JC69)</td>
+<td>multiple substitutions per site</td>
+<td>—</td>
+</tr>
+<tr>
+<td><code>snp-k2p</code> (Kimura 2-parameter, K80)</td>
+<td>+ transition/transversion rate bias</td>
+<td>ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-k81</code> (Kimura 3-parameter, K3ST)</td>
+<td>+ splits transversions into 2 categories</td>
+<td>ts/tv split, by category</td>
+</tr>
+<tr>
+<td><code>snp-f81</code> (Felsenstein 81)</td>
+<td>+ unequal base frequencies (no ts/tv split)</td>
+<td>empirical base freqs</td>
+</tr>
+<tr>
+<td><code>snp-tajima-nei</code> (Tajima-Nei 1984)</td>
+<td>same goal as F81 (equal-input model), different formula, better small-sample behavior</td>
+<td>empirical base freqs</td>
+</tr>
+<tr>
+<td><code>snp-t92</code> (Tamura 3-parameter)</td>
+<td>K2P + GC-content bias</td>
+<td>ts/tv split, GC content</td>
+</tr>
+<tr>
+<td><code>snp-f84</code> (Felsenstein 84)</td>
+<td>full empirical base freqs + single ts/tv rate</td>
+<td>empirical base freqs, ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-hky85</code> (Hasegawa-Kishino-Yano, pairwise estimator)</td>
+<td>same inputs as F84, different formula</td>
+<td>empirical base freqs, ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-tn93</code> (Tamura-Nei)</td>
+<td>full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate</td>
+<td>empirical base freqs, purine-ts/pyrimidine-ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-logdet</code> (LogDet / paralinear)</td>
+<td>no shared-model or stationarity assumption at all — general divergence-matrix determinant</td>
+<td>full empirical 4×4 divergence matrix (already <code>subst[4][4]</code>)</td>
+</tr>
+<tr>
+<td><code>snp-tv</code> (transversions-only p-distance)</td>
+<td>diagnostic/deep-divergence variant — drops transitions entirely (they saturate first)</td>
+<td>tv-only counts</td>
+</tr>
+</tbody>
+</table>
+<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
+<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
+nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
+formula as the base correction, weighted by a shape parameter <code>α</code> supplied
+by the user (<code>--gamma-shape &lt;alpha&gt;</code>), not estimated by ML. A modifier on
+existing values, not a separate enum arm per distance.</p>
+<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
+<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
+"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
+<code>snp-logdet</code> are catalogued above but <strong>not implemented</strong>: <code>snp-logdet</code>
+needs the true <em>directional</em> per-pair base co-occurrence matrix
+(<code>PairwiseTally</code> only keeps the symmetrised substitution counts
+<code>BasePairTally</code> itself wants — see <code>snp_distance.rs</code>'s own module docs for
+why that loses exactly the compositional-asymmetry information LogDet
+exists to detect), <code>snp-tajima-nei</code> needs each genome's <em>own</em> base
+composition (not the pair-pooled estimate the formulas below use), and
+<code>snp-f84</code>/<code>snp-hky85</code> had no formula independently verified against a
+primary source at implementation time (unlike every formula below, checked
+line-by-line against <a href="https://github.com/emmanuelparadis/ape">ape</a>'s own
+<code>src/dist_dna.c</code>, not re-derived from memory). Adding any of these later is
+a new function in <code>obikphylo::siblings::algorithms::snp_distance</code>, plus for
+<code>snp-logdet</code>/<code>snp-tajima-nei</code> a new field on <code>PairStats</code>/a per-genome
+accumulator — not an architecture change.</p>
+<h3 id="exact-formulas-implemented-2026-08-28">Exact formulas (implemented, 2026-08-28)</h3>
+<p>Sufficient statistic, per genome pair <code>(i, j)</code>, from
+<code>PairwiseTally::categories</code>/<code>PairwiseTally::base_freq</code> (base order always
+<code>0=A, 1=C, 2=G, 3=T</code>, matching <code>FamilyMask</code>/<code>STATE_SYMBOL</code>):</p>
+<ul>
+<li><span class="arithmatex">\(n_{ts1}\)</span>: A↔G substitutions (purine transitions), <span class="arithmatex">\(n_{ts2}\)</span>: C↔T
+  (pyrimidine transitions)</li>
+<li><span class="arithmatex">\(n_{tv1}\)</span>: A↔C and G↔T substitutions, <span class="arithmatex">\(n_{tv2}\)</span>: A↔T and C↔G
+  (Kimura's two transversion categories)</li>
+<li><span class="arithmatex">\(n_{shared}\)</span>: loci where both genomes agree</li>
+<li><span class="arithmatex">\(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\)</span> (total eligible
+  loci for the pair)</li>
+<li><span class="arithmatex">\(\pi_A, \pi_C, \pi_G, \pi_T\)</span>: pair-pooled base frequencies,
+  <span class="arithmatex">\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)</span>
+  (both genomes' calls at this pair's eligible loci, pooled — Nei &amp; Kumar's
+  standard pairwise estimator, not a whole-index average)</li>
+</ul>
+<p>Derived proportions used below:</p>
+<div class="arithmatex">\[
+p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
+P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
+Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
+Q_1 = \frac{n_{tv1}}{L}, \quad
+Q_2 = \frac{n_{tv2}}{L}, \quad
+P_1 = \frac{n_{ts1}}{L}, \quad
+P_2 = \frac{n_{ts2}}{L}
+\]</div>
+<p>Every formula below was checked term-by-term against <code>ape</code>'s own
+<code>src/dist_dna.c</code> (not re-derived from memory) before being ported to
+<code>obikphylo::siblings::algorithms::snp_distance</code>.</p>
+<p><strong><code>snp-raw</code></strong> — uncorrected p-distance:</p>
+<div class="arithmatex">\[
+d_{raw} = p
+\]</div>
+<p><strong><code>snp-tv</code></strong> — transversions-only p-distance (deliberately uncorrected —
+dropping transitions, which saturate first, <em>is</em> the correction):</p>
+<div class="arithmatex">\[
+d_{tv} = Q
+\]</div>
+<p><strong><code>snp-jc</code></strong> (Jukes-Cantor, JC69):</p>
+<div class="arithmatex">\[
+d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
+\]</div>
+<p><strong><code>snp-k2p</code></strong> (Kimura 2-parameter, K80), with <span class="arithmatex">\(a_1 = 1-2P-Q\)</span>, <span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
+<div class="arithmatex">\[
+d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
+\]</div>
+<p><strong><code>snp-k81</code></strong> (Kimura 3-parameter, K3ST), with <span class="arithmatex">\(a_1 = 1-2P-2Q_1\)</span>,
+<span class="arithmatex">\(a_2 = 1-2P-2Q_2\)</span>, <span class="arithmatex">\(a_3 = 1-2Q_1-2Q_2\)</span>:</p>
+<div class="arithmatex">\[
+d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
+\]</div>
+<p><strong><code>snp-f81</code></strong> (Felsenstein 81), with <span class="arithmatex">\(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\)</span>:</p>
+<div class="arithmatex">\[
+d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
+\]</div>
+<p><strong><code>snp-t92</code></strong> (Tamura 3-parameter), with GC content
+<span class="arithmatex">\(g = \pi_C+\pi_G\)</span>, <span class="arithmatex">\(w = 2g(1-g)\)</span>, <span class="arithmatex">\(a_1 = 1 - \dfrac{P}{w} - Q\)</span>,
+<span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
+<div class="arithmatex">\[
+d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
+\]</div>
+<p><strong><code>snp-tn93</code></strong> (Tamura-Nei), with purine/pyrimidine pooled frequencies
+<span class="arithmatex">\(g_R = \pi_A+\pi_G\)</span>, <span class="arithmatex">\(g_Y = \pi_C+\pi_T\)</span>, and</p>
+<div class="arithmatex">\[
+k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
+k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
+k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
+\]</div>
+<div class="arithmatex">\[
+w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
+w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
+w_3 = 1 - \frac{Q}{2g_R g_Y}
+\]</div>
+<div class="arithmatex">\[
+d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
+\]</div>
+<p><strong><code>+Γ</code> gamma correction</strong> (Jin &amp; Nei 1990): every formula above is a
+weighted sum of <span class="arithmatex">\(-\ln(x)\)</span> terms; the gamma-corrected version replaces
+each such term with the same weight applied to
+<span class="arithmatex">\(\alpha\left(x^{-1/\alpha} - 1\right)\)</span> instead — the standard mechanical
+substitution (as <span class="arithmatex">\(\alpha \to \infty\)</span>, this expression → <span class="arithmatex">\(-\ln(x)\)</span>,
+recovering the uncorrected formula exactly). E.g. for JC:</p>
+<div class="arithmatex">\[
+d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
+\]</div>
+<p>Verified term-by-term against <code>ape</code>'s own gamma branches for JC69/K80/F81
+(including K80's two-term form — algebraically identical to the generic
+substitution applied to <code>snp-k2p</code>'s own <span class="arithmatex">\(a_1\)</span>/<span class="arithmatex">\(a_2\)</span> terms above, checked
+both symbolically and numerically before simplifying the implementation to
+share one <code>corrected_log</code> helper across every model rather than
+special-casing K80). K81/T92/TN93's gamma branches follow the same
+mechanical substitution but weren't independently checked against an
+<code>ape</code>-equivalent reference for those three specifically — flagged here, not
+silently assumed correct.</p>
+<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
+<p><strong>Implemented.</strong> The primary distance-matrix output
+(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
+default</strong> (widely read by external NJ tools — PHYLIP <code>neighbor</code>, FastME,
+T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
+label truncation, since genome labels here routinely exceed it), a <code>--csv</code>
+flag opts back into the current CSV format, PHYLIP-strict is a possible
+future addition (not now). This changes the <em>default</em> output of every
+existing <code>--distance</code> value (jaccard, hamming, bray-curtis, ...), not just
+the new <code>snp-*</code> ones — accepted explicitly (pre-release, single developer
+user, no external consumers to break). Scoped to the distance matrix only:
+<code>--shared-kmers</code> and <code>--raw-snp-counts</code> are counts, not distances, and keep
+their existing CSV-only format.</p>
 <h2 id="references">References</h2>
 <p>The Mash mutation-rate model this discussion contrasts with:
 (Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>

DevDoc/theory/indexing.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,8 +1270,8 @@
 <h1 id="coverage-theoryindexingmd">Coverage: theory/indexing.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obikpartitionner/src/partition.rs</code> — routage par hash de minimiseur, choix des paramètres</li>
-<li><code>obikpartitionner/src/lib.rs</code> — structure KmerPartition, nombre de partitions</li>
+<li><code>obikindex/src/partition/partition.rs</code> — routage par hash de minimiseur, choix des paramètres</li>
+<li><code>obikindex/src/partition/mod.rs</code> — structure KmerPartition, nombre de partitions</li>
 </ul>
 <h2 id="notes">Notes</h2>
 <p>Vérifier que la doc mentionne bien que le nombre de partitions est une puissance de 2

DevDoc/theory/indexing/index.html

@@ -802,14 +802,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1019,6 +1019,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/minimizer.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/minimizer/index.html

@@ -846,14 +846,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1063,6 +1063,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDocMD/architecture/index_architecture.refs.md

@@ -3,10 +3,10 @@
 
 ## Code couvert
 
-- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix
-- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum
-- `obilayeredmap/src/map.rs` — LayeredMap<D>
-- `obilayeredmap/src/meta.rs` — LayerMeta, PartitionMeta
+- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix
+- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum
+- `obikindex/src/layer/map.rs` — LayeredMap<D>
+- `obikindex/src/layer/meta.rs` — LayerMeta, PartitionMeta
 - `obikindex/src/meta.rs` — IndexConfig (kmer_size, n_bits, with_counts, evidence, block_bits), IndexMeta
 - `obikindex/src/index.rs` — KmerIndex, build_layers
 - `obicompactvec/src/` — PersistentCompactIntMatrix, PersistentBitMatrix (DataStore implementations)

DevDocMD/architecture/numa_partition_runner.md

@@ -304,10 +304,29 @@ step, capped by that node's remaining dormant workers and by the run's total
 budget (`n_total`) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.
 
+## Panic propagation (fixed, 2026-08-28)
+
+A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
+— the controller's `while completed < n_total` loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept `event_rx` open. Discovered via `obikselect`
+panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
+the process hung indefinitely instead of erroring out.
+
+Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
+`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
+partition. The controller counts it toward `completed` (unblocking the loop)
+and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain `Err` (a panic means a bug, not a normal typed failure).
+
 ## Open questions
 
 - **Error handling**: `run` currently returns the first error; remaining errors
-  are dropped.  A `Vec<E>` return would give complete diagnostics.
+  are dropped.  A `Vec<E>` return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain `Err` results.)
 
 - **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
   (start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an

DevDocMD/architecture/numa_worker_pools.md

@@ -72,9 +72,9 @@ Each NUMA group runs its own independent adaptive pool. Workers are distributed
 |------|--------|
 | `obikindex/src/merge.rs` | Detect NUMA topology; build N `ThreadPool`s with pinned threads; assign each pre-spawned worker to a pool; wrap `merge_partition` in `pool.install()` |
 | `obikindex/src/merge.rs` | Replace `available_parallelism()` with per-NUMA core count for spawn criterion |
-| `obikpartitionner/src/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
+| `obikindex/src/partition/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
 | `obidebruinj/src/debruijn.rs` | No change — `par_iter` and `current_num_threads` are pool-context-aware |
-| `obikpartitionner/src/partition.rs` | No change — same reason |
+| `obikindex/src/partition/partition.rs` | No change — same reason |
 
 ## Platform guard
 

DevDocMD/architecture/query.md

@@ -16,7 +16,7 @@ Given a set of query sequences, determine for each sequence how many of its k-me
 
 ## Algorithm
 
-The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikpartitionner::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
+The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikindex::partition::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
 
 ```
 for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
@@ -118,11 +118,11 @@ The `-z` CLI option overrides the index metadata value. A higher z increases str
 
 ## Layer lookup: `MphfLayer::find`
 
-`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obilayeredmap](../implementation/obilayeredmap.md) for the full `find` / `find_strict` API.
+`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obikindex::layer](../implementation/layer_tier.md) for the full `find` / `find_strict` API.
 
 ### `QueryLayer` variant selection
 
-`QueryLayer::open` (`obikpartitionner/src/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
+`QueryLayer::open` (`obikindex/src/partition/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
 
 | Order | Condition | Variant | Data returned per k-mer |
 |---|---|---|---|
@@ -244,7 +244,7 @@ Fix direction: restructure `query`'s pipe with an initial `Flat` stage analogous
 `chunk_bytes = available_memory_bytes() / (n_workers * 16)` (`query.rs:407-414`) assumes a fixed ~8–16× overhead per raw input byte. But `KmerResults::new` (`query.rs:165-179`) allocates `data: Vec<u32>` sized `total_kmers_in_chunk × n_genomes` — dense, **for every k-mer position in the chunk, hit or not** — plus `win_min` and (with `--detail`) `cov`, same scaling. Real per-chunk memory is `O(n_genomes)`, not constant; the formula doesn't know `n_genomes` at all. This is the direct cause of the OOM kill on indexes with many reference genomes.
 
 **4. MPHF lookup and matrix-row fetch are fused, not staged.**
-`QueryLayer::find_into` (`obikpartitionner/src/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
+`QueryLayer::find_into` (`obikindex/src/partition/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
 
 **5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.**
 `QueryBatch::from_records` currently dereplicates at the *superkmer* level (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, `query.rs:112`). This misses redundancy between k-mers shared by *different* superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer *construction* (`SuperKmerIter`) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on `CanonicalKmer`, in the same pass: `HashMap<CanonicalKmer, Vec<(seq_idx, pos)>>`. This also means the MPHF `find` itself runs once per **distinct** k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.
@@ -285,7 +285,7 @@ None of this is implemented yet — parked here as a coherent roadmap while the
 
 ## Implementation plan
 
-Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikpartitionner`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
+Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikindex::partition`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
 
 Instrumentation is deliberately sequenced *before* the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.
 
@@ -295,7 +295,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 
 **Debug logging.** Every phase that changes an algorithmic choice (not phase 0, which *is* the logging) adds `tracing::debug!`/`trace!` at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF `find` calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured `debug!` per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.
 
-**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obilayeredmap`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikpartitionner` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
+**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obikindex::layer`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikindex::partition` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
 
 **Workflow (`jj`).** Work happens in a fresh `jj` commit, easy to abandon. `jj new` between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then `query_layer.rs` split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.
 
@@ -345,13 +345,13 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 - `obikmer/src/cmd/query.rs`:
   - Replace `QueryBatch::from_records`'s dedup map (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, current `query.rs:112`) with a per-partition `HashMap<CanonicalKmer, Vec<(seq_idx: u32, pos: u32)>>`, built in the same `SuperKmerIter` pass: superkmer construction and partition routing (`part_idx` from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each `CanonicalKmer` within a superkmer is inserted individually instead of the whole superkmer being the dedup key.
   - **Verified**: `CanonicalKmer` (`obikseq/src/kmer.rs:390`, `pub type CanonicalKmer = CanonicalKmerOf<KLen>`) — the underlying `CanonicalKmerOf<L>` derives `Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash` (`kmer.rs:269`). Usable as a `HashMap`/`HashSet` key as-is, no change needed.
-- `obikpartitionner/src/query_layer.rs`:
+- `obikindex/src/partition/query_layer.rs`:
   - Split `QueryLayer::find_into` (`query_layer.rs:48-67`) into two methods: `find_slot(&self, kmer: CanonicalKmer) -> Option<usize>` (MPHF only, no matrix touch) and keep `fill_row` as-is for phase 4 to call later.
   - Replace `query_partition_with`'s inner loop (`query_layer.rs:103-113`) with a version that, for each unique `CanonicalKmer`, calls `find_slot` across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records `(layer_idx, slot)`.
   - New return shape for the partition-level query, replacing today's `on_hit(sk_idx, kmer_idx, row)` callback: `HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>` (roadmap point 6) — built directly from the k-mer dedup map's `Vec<(seq_idx,pos)>` values, keyed by the resolved slot instead of the k-mer.
 - **This phase alone has no throughput benefit yet** (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.
 - Log, per chunk: total k-mer occurrences vs. unique `CanonicalKmer` count (the dedup ratio — the entire justification for this phase) and the resulting MPHF `find` call count. If the dedup ratio is close to `1.0` on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.
-- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikpartitionner/src/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
+- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikindex/src/partition/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
 
 ### Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred
 
@@ -359,7 +359,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 
 **What shipped:**
 - `obicompactvec`: the per-column accessors this phase needed **already existed** — `PersistentCompactIntMatrix::col_view(c)` and `PersistentBitMatrix::col_view(c)` are public, and `IntSliceView::get(slot)`/`BitSliceView::get(slot)` are public — the original plan underestimated how much of this plumbing the pairwise-distance code (`dump`/`select`/`stats`) had already required. The one real gap: `PersistentBitMatrix::col_view()` panics on the `Implicit` variant (the documented mono-genome fast path, `bitmatrix.rs`). Added `PersistentBitMatrix::get(c, slot) -> u32` (`bitmatrix.rs`), a non-panicking column-major point lookup that returns `1` for `Implicit` regardless of `c` — the smallest surface needed, not a new `col_get` API from scratch.
-- `obikpartitionner/src/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
+- `obikindex/src/partition/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
 - `obikmer/src/cmd/query.rs`: `KmerResults::set` (row-major, whole-row-at-once) replaced by `mark_found` (stage 1: flag a position as indexed, independent of any genome's value) and `set_one` (stage 2: write one genome's value at one position). `QueryStats` extended with `n_columns_scanned`/`n_col_get_calls`, logged per chunk.
 - Total `get()`-equivalent calls are unchanged from the row-major version (`n_hits × n_cols` in the worst case, confirmed by `n_col_get_calls` in the debug log) — the win is locality (sequential access within one layer's column at a time, across `mmap`'d regions, instead of jumping across all columns per hit), exactly as predicted.
 
@@ -367,7 +367,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 Reading `obikindex/src/numa.rs`'s actual `run()` body (not just its doc comments) shows every call spawns a timer thread **plus one OS thread per worker slot on every NUMA node** (`std::thread::scope` + one `s.spawn()` per node per `max_workers`) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned **per call**. This is fine for its actual, established usage in this codebase (`merge.rs`, `index.rs`'s `build_layers`): one `PartitionRunner::new()` + one `run()` call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for `query`'s call pattern: `query_partition_with` runs once per `(chunk, partition)`, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
 The column-major loop in stage 2 is therefore a **plain sequential loop** for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) `rayon`'s already-warm global pool (`into_par_iter()`) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern `numa_worker_pools.md` warns about for a *different* workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one `mmap`'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so `PartitionRunner` is invoked once per whole `query` run (or per large batch of chunks) rather than per `(chunk, partition)`, amortising its spawn cost the way `merge`/`build_layers` do — a bigger structural change than this phase's scope.
 - Log (implemented): `QueryStats::n_columns_scanned`/`n_col_get_calls`, folded into the existing per-chunk `debug!("k-mer dedup + column-major fetch", ...)` line (`query.rs`) alongside phase 3's dedup counters.
-- **Unit tests**: extended `obikpartitionner/src/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
+- **Unit tests**: extended `obikindex/src/partition/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
 - **Validation performed**: full workspace build + `cargo test --workspace`, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same `kmer_count`/`kmer_strict_matches` on every record); (2) the existing 20-genome `benchmark/global_index_presence` index — runs correctly, `n_hits=0` for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the `Implicit`/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) **the critical correctness case**: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from `genomeA` matched **only** `genomeA` (`kmer_count` identical to the pre-dedup occurrence count, zero leakage into `genomeB`'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in `n_cols`, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
 - **Not yet done**: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-`n_genomes` layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.
 

DevDocMD/architecture/query.refs.md

@@ -4,7 +4,7 @@
 ## Code couvert
 
 - `obikmer/src/cmd/query.rs` — commande query, format de sortie
-- `obikpartitionner/src/query_layer.rs` — routage de la requête à travers les partitions
+- `obikindex/src/partition/query_layer.rs` — routage de la requête à travers les partitions
 - `obiread/src/lib.rs` — lecture des séquences d'entrée pour la requête
 
 ## Notes

DevDocMD/architecture/rebuild_filter.md

@@ -94,9 +94,9 @@ Option B avoids storing kmer values and works uniformly regardless of filter sel
 
 ## Files to modify
 
-- `src/obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
+- `src/obikindex/src/partition/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
 - Possibly `src/obicompactvec/` — add column iterator API if not already present
-- `src/obilayeredmap/` — check if per-column sequential access is exposed on `SrcLayerData`
+- `src/obikindex/src/layer/` — check if per-column sequential access is exposed on `SrcLayerData`
 
 ## Open questions
 

DevDocMD/architecture/siblings.md

@@ -17,7 +17,7 @@ one-way function, not an invertible bijection with a stored inverse. Any
 method that reconstructs a kmer from a bare slot number is wrong by
 construction, regardless of the mechanism used (MPHF re-hash, or evidence
 decode + direct unitig read). See `MphfLayer::kmer_at`
-(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
+(`obikindex/src/layer/mphf_layer.rs`) — flagged for removal, currently called
 from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
 "Pending work" status below).
 
@@ -70,7 +70,7 @@ partition is unknown) and must keep going through
 ## Pending work — done
 
 The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
-traits over `obikindex::KmerIndex`/`obilayeredmap::Layer<D>`, replacing the
+traits over `obikindex::KmerIndex`/`obikindex::layer::Layer<D>`, replacing the
 old `obikindex::siblings` module) builds and reads the annex purely in
 iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
 batch variants, mirroring `Layer<D>`'s own `KmerIter`/`KmerBatchIter`
@@ -652,13 +652,13 @@ BRWT-style column-correlation exploitation.
 `PersistentSparseBitMatrix` went from a validated but unused type to a
 real, selectable on-disk format:
 
-- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
+- **Generic `Layer<D>`**: `obikindex::layer::Layer<D>`'s presence-only methods
   (`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
   `D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
   `PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
   `LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
   already generic over any `D: LayerData`, so they needed no change.
-  Verified by `obilayeredmap`'s
+  Verified by `obikindex::layer`'s
   `presence_layer_generic_over_sparse_matches_dense` test: build a dense
   presence layer, convert it to sparse via `build_from_dense`, open both
   as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
@@ -668,7 +668,7 @@ real, selectable on-disk format:
   `AtomicUsize`s in test builds, not thread-local, so a test using a
   different `k` races every other test in the same crate binary; a k=11
   version of this test passed alone but failed under the full
-  `obilayeredmap` suite for exactly that reason before being fixed.)
+  `obikindex::layer` suite for exactly that reason before being fixed.)
 - **`obikphylo::siblings::cache::Mat`** gained a third variant,
   `SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
   and `Presence` — every method (`find_slot`, `index_batch`,
@@ -698,7 +698,7 @@ real, selectable on-disk format:
   dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
   sparse format round-trips through the real build pipeline
   (`PartitionCache` sparse-detection included), not just the
-  `obicompactvec`/`obilayeredmap` unit layers below it.
+  `obicompactvec`/`obikindex::layer` unit layers below it.
 
 Full workspace `cargo test` (all crates, unit + doc tests) green after
 this change.
@@ -746,7 +746,7 @@ Replaces the "four independent scans" problem above and implements
   (activation: either given; defaults `1.0`/`0.5` for whichever is unset).
 
 Full workspace `cargo test` green after this change (167 unit tests in
-`obicompactvec`+`obilayeredmap`+`obikphylo` alone, plus every other
+`obicompactvec`+`obikindex::layer`+`obikphylo` alone, plus every other
 crate's suite, no regressions).
 
 **Still open, not part of this change** (per "Correction to the 'single
@@ -756,3 +756,269 @@ scan the full unsampled index — never threaded `--subsample`/`--entropy`,
 out of scope here since the reported problem was specifically about the
 `--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these
 two standalone flags.
+
+## `query` never benefits from sparse row-major access (found 2026-08-19, not implemented)
+
+Benchmarked `obikmer query` against `global_index_presence` (dense-packed)
+vs. `global_index_presence_sparse` (`pack --sparse`), 100k simulated reads
+× 2 specimens (`benchmark/`, see
+[benchmark_query_testing.md](../implementation/benchmark_query_testing.md)).
+Correctness: 0 mismatches — sparse and dense return bit-identical query
+results. Performance: sparse consistently *slower* than dense (~30-50%,
+reproducible across two runs with warm disk cache), the opposite of
+`pack --sparse`'s stated intent ("faster for single-row access... like
+query").
+
+**Root cause, read from source, not measured in isolation:**
+`KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs:155-220`)
+is architecturally column-major: stage 2 walks `for g in 0..n_cols { for
+slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly)
+as the right locality strategy for the packed/columnar formats, where
+`col_value` → `PersistentBitMatrix::get` is a genuine O(1) mmap'd column
+read (`persistent.rs:110-113`).
+
+For `Self::Sparse`, that same `get(c, slot)` (`persistent.rs:114-118`)
+allocates a full `n_cols`-wide buffer and calls `fill_row` — materializing
+the *entire row* — just to return one cell. Called from inside the
+column-major double loop, this reconstructs the same row once per genome
+column touched: O(hits × n_cols) full-row rebuilds instead of O(hits).
+`PersistentSparseBitMatrix`'s own native row-major decode
+(`for_each_genome_in_row`, `sparse.rs:164-177`, used correctly by its own
+`row`/`fill_row`/`fill_sub_matrix`) is never reached from the query path
+at all.
+
+**`fill_sub_matrix` (the existing `BinaryMatrix` trait primitive,
+`traits.rs:13-37`) is not the right replacement for `query` either**, even
+once its own dispatch bug is fixed (see next section) — its output shape
+is inherently column-dense: `out[col]` gets an entry for every column,
+including columns with zero hits among the requested slots. On real
+sparse data (a hit typically touching a handful of genomes out of dozens)
+that's still O(n_cols) output regardless of true sparsity. What `query`
+actually wants is the sparse triple stream `(slot, col, value)` it already
+consumes as `QueryHit::Value` — not a materialized sub-matrix.
+
+**Proposed primitive** (design only, not implemented — explicit ask: keep
+count matrices *not excluded*, even though effort right now is
+presence/absence only):
+
+Not a closure-driven `for_each` — a real `Iterator`, one concrete struct
+per matrix format, so the traversal state (current position in the sorted
+slot list, current column, permutation, sparse-row decode cursor…) lives
+in named struct fields instead of being threaded implicitly through
+recursion or a captured closure. RPITIT (stable since 1.75, and this
+workspace is edition 2024) means the trait method can return it without
+naming or boxing the concrete type:
+
+```rust
+/// Yields every nonzero cell among `slots`, in implementation-defined order.
+fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator<Item = (usize, usize, u32)> + 'a;
+// item: (idx into `slots`, col, value)
+```
+
+This is the one foundational primitive per format — both `fill_sub_matrix`
+and a `for_each`-style callback become trivial, free consumers of it
+(`.for_each(f)` is already `Iterator::for_each` from std; `fill_sub_matrix`
+becomes "drain the iterator, scatter into `out[][]`"), instead of two
+independently-maintained traversals that can silently diverge (see the bug
+below — this is exactly how it happened).
+
+- **On `PersistentSparseBitMatrix`**: the struct is nearly free to write —
+  it wraps the existing (currently private) `for_each_genome_in_row`
+  per-row decode, advancing to the next `slots` entry on exhaustion. O(Σ
+  row nnz), zero `n_cols`-wide allocation.
+- **On `PersistentBitMatrix::{Packed,Columnar}`**: revised — cheaper than
+  first thought, by reusing the same split already used for
+  `fill_matrix`'s own implementation instead of hand-writing a resumable
+  state machine at the matrix level. The base-vector layer
+  (`BitSliceView`, `views.rs`) already separates the two concerns:
+  `fill_batch_sorted` (`views.rs:55-60`, sorted-slot batch lookup) and a
+  genuine per-bit `Iterator` (`BitSliceIter`, `views.rs:94+`) sit side by
+  side there, one level below the matrix. Adding a
+  "positions among `sorted_slots` where the bit is set" iterator at that
+  same vector level is a `filter` over the existing `get()` — no new
+  state machine, since `std::iter::Filter` already *is* one, generated by
+  the compiler:
+  ```rust
+  fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator<Item = usize> + 's {
+      sorted_slots.iter().copied().filter(move |&slot| self.get(slot))
+  }
+  ```
+  The matrix-level `nonzero_iter` then composes these per column with
+  `flat_map` over `0..n_cols` (each column's hits, tagged with `c`,
+  slot mapped back through the sort permutation `fill_batch`/
+  `fill_batch_sorted` already carry) — again a combinator chain, not a
+  hand-rolled struct. Same algorithm, same mmap/sort locality as today's
+  `fill_sub_matrix`; just assembled from `std` iterator adaptors instead
+  of a loop body writing into a buffer, mirroring the vector/matrix split
+  the codebase already uses for `fill_batch_sorted` rather than
+  introducing a new shape.
+- **`Implicit`**: trivial (`slots.iter().map(|&i| (i, 0, 1))`, one column,
+  always present).
+- **On `PersistentCompactIntMatrix` (counts)**: same treatment as
+  `Packed`/`Columnar` — no sparse count format exists yet ("Explicitly
+  deferred" per `traits.rs:9-12`), so no native low-effort case the way
+  `Sparse` has one, but not excluded either: the iterator's `Item` is
+  already `(usize, usize, u32)`, not `bool`, specifically so presence
+  (`0`/`1`) and counts (arbitrary `u32`) share one primitive instead of a
+  bool/u32 split forcing counts out of the design. Ready for a native
+  sparse-count struct later without a signature change.
+
+Would let `query_partition_with`'s stage 2 collapse to one
+`for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) }`
+per layer, format-agnostic, each backend's struct deciding the actual
+traversal.
+
+**This also closes the existing dispatch bug for free, by construction**:
+`PersistentBitMatrix::fill_sub_matrix` (`persistent.rs:190-215`, the enum
+wrapper backing `BinaryMatrix`'s trait impl) today does *not* delegate to
+`PersistentSparseBitMatrix::fill_sub_matrix` for `Self::Sparse` — it
+reimplements the same naive per-(column, slot) `fill_row_bool` loop
+instead, bypassing the efficient native method one file over
+(`sparse.rs:249-258`). `obikphylo::siblings::cache::Mat`
+(`cache.rs:138-145`) independently built its own parallel enum wrapper
+that dispatches correctly — evidence this was worked around rather than
+fixed at the source: two hand-written traversals for the same format,
+free to drift apart, and they did. If `fill_sub_matrix` itself is
+rewritten as "drain `nonzero_iter`, scatter into `out[][]`", there is only
+one traversal per format left to get right — the bug class doesn't just
+get fixed once, it stops being possible to reintroduce.
+
+## Implemented (2026-08-20)
+
+Built as designed above, with one deviation from the original sketch:
+`nonzero_iter` ended up `Box<dyn Iterator<...>>`, not a bare `impl
+Iterator`, because `Columnar`/`Packed`/`Sparse`/`Implicit` are genuinely
+different concrete types and this method isn't on a trait (kept off
+`BinaryMatrix` deliberately — that trait is used as `dyn BinaryMatrix` in
+`tests/sparse.rs`, and RPITIT methods aren't dyn-compatible). One `Box`
+per `nonzero_iter` call, not per cell — negligible next to what it
+replaces.
+
+- `BitSliceView::nonzero_among_sorted` / `IntSliceView::nonzero_among_sorted`
+  (`obicompactvec/src/views.rs`): the vector-level `filter`/`filter_map`
+  primitive, exactly as sketched — no new state machine, `std`'s own.
+- `PersistentSparseBitMatrix::nonzero_iter` (`bitmatrix/sparse.rs`): native,
+  `std::iter::from_fn` over one buffered row at a time via the existing
+  `for_each_genome_in_row` — no `n_cols`-wide allocation, ever.
+- `PersistentBitMatrix::nonzero_iter` (`bitmatrix/persistent.rs`): dispatches
+  to the above for `Sparse`; for `Columnar`/`Packed`, loops columns,
+  collects each column's `nonzero_among_sorted` hits via `.extend()` (not
+  `flat_map` — a `flat_map` closure can't lazily return something
+  borrowing its own captured sort permutation across separate calls
+  without either boxing per-column or fighting the borrow checker; eager
+  collection into one `Vec` sidesteps it, at zero cost since
+  `fill_sub_matrix` already fully materialized anyway). `Implicit` trivial.
+- `PersistentBitMatrix::fill_sub_matrix` and `sub_matrix` rewritten to
+  drain `nonzero_iter` — the dispatch bug is gone because there is now
+  only one traversal per format, not because the old one was patched.
+  `PersistentCompactIntMatrix::nonzero_iter` added the same way (counts
+  not excluded, per the earlier ask) — no native low-effort case, since no
+  sparse count format exists, but on the same primitive, ready for one.
+- `KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs`):
+  stage 2's column-major `for g { for slot { col_value } }` replaced by one
+  `layer.nonzero_iter(&slot_list)` call per layer, format-agnostic.
+- Tests: `nonzero_iter_matches_dense`, `nonzero_iter_matches_row`, and —
+  the one that actually targets the dispatch bug rather than each type's
+  own correctness — `enum_wrapper_dispatches_to_native_sparse` (builds
+  `PersistentBitMatrix::Sparse(...)` directly, not through `open`, since
+  `open` only auto-detects `Sparse` from a `presence/` dir layout).
+  `cargo test --workspace`: green, no regressions.
+
+**Measured**: re-ran the `benchmark/` query branch (100k reads × 2
+specimens, same setup as the original finding). Correctness still 0
+mismatches. The dense/sparse performance gap is gone — previously sparse
+~30-50% slower than dense, reproducibly; now within ~1-3% either way
+(7.42s dense vs 7.60s sparse for `Escherichia_coli--K-12_MG1655`; 5.25s vs
+5.30s for `Saccharolobus_islandicus--M.16.4`) — noise-level, not a
+systematic gap. `pack --sparse`'s claimed query win isn't confirmed
+outright by this (sparse should arguably now *beat* dense on truly sparse
+real data, not just tie), but the pathological regression is fixed.
+
+## `PersistentCompactIntMatrix::Sparse` — implemented (2026-08-26)
+
+Closes the gap flagged throughout this document ("no sparse count format
+exists yet", `traits.rs:9-12`'s "Explicitly deferred"): `obicompactvec`
+already had `PersistentSparseCompactIntMatrix` (row-major, built on top of
+`PersistentSparseBitMatrix` as its "which columns are non-zero" support,
+values *not* deduplicated — see that struct's own doc comment), but it was
+never wired into `PersistentCompactIntMatrix`, the dense-dispatching enum
+every real consumer (`TypedLayer<PersistentCompactIntMatrix>`,
+`KmerLayer::Count`) actually holds. Concretely: `kmer_index.rs::
+pack_matrices(sparse=true)` already called `pack_sparse_compact_int_matrix`
+on every layer's `counts/` — but `PersistentCompactIntMatrix::open` had no
+code path back to what that just wrote, so a `Count` layer became
+unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
+anyone ran `pack --sparse` on an index with count layers. Root cause, not a
+workaround: add the missing `Sparse` variant.
+
+- **Enum + dispatch** (`intmatrix.rs`): `PersistentCompactIntMatrix::Sparse
+  (PersistentSparseCompactIntMatrix)`, detected in `open`/`detect_storage`
+  via a `singleton_values.pciv` marker (mirrors `PersistentBitMatrix`'s own
+  `sparse_meta.json` check), reported via `storage_kind()`. `col`/
+  `col_view`/`col_persist` panic/`Unsupported` on `Sparse`, same convention
+  as the bit side. `sub_matrix`/`fill_sub_matrix` and `nonzero_iter`
+  unified the same way `PersistentBitMatrix`'s already are (drain
+  `nonzero_iter`, one traversal per format — see "Implemented
+  (2026-08-20)" above); `nonzero_iter` had to become `Box<dyn Iterator<...>>`
+  for the same reason (`Columnar`/`Packed`/`Sparse` are different concrete
+  types). No change needed in `obikindex` at all — `KmerLayer::Count`
+  already only ever holds `TypedLayer<PersistentCompactIntMatrix>`, so the
+  enum absorbing `Sparse` fixes the unreadable-layer bug for free, same as
+  `PersistentBitMatrix::Sparse` already did on the presence side.
+
+- **`CountPartials`, non-naive** (`sparse_intmatrix.rs`): unlike
+  `PersistentSparseBitMatrix`'s dict-driven `col_weights_and_pair_counts`,
+  values here aren't deduplicated (two rows can share the same non-zero
+  column set via the same `dict_id` while carrying different counts), so
+  the "weight by how many rows share a dict entry" shortcut doesn't carry
+  over. What does: a single row-major pass (`row_major_pairwise`, decodes
+  each row once via `for_each_cell_in_row`, nests over that row's own
+  co-present columns) — `O(Σ k̄²)` over populated rows instead of the naive
+  `O(n_cols² × n)` column-pair rescan, same complexity class as the bit
+  side minus the dict multiplicity discount. Kernels used: `min(a,b)`
+  (bray, relfreq-bray — both vanish when either side is absent, so no
+  correction needed), `a·b` and `√(a·b)` (euclidean/relfreq-euclidean and
+  hellinger — these *do* need a correction, reconstructed from per-column
+  marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`, since `(a-0)² = a² ≠ 0` unlike
+  the `min`-based formulas). `threshold_jaccard(1)` shortcuts straight to
+  `support`'s own `BitPartials::partial_jaccard` (threshold 1 is exactly
+  presence); `threshold_jaccard(0)` is closed-form (every `u32` is `≥ 0`).
+
+- **Two pre-existing bugs found and fixed while wiring the `threshold==1`
+  shortcut** (`bitmatrix/sparse.rs`, `BitPartials for
+  PersistentSparseBitMatrix`, present since the 2026-08-15 implementation
+  above, never caught because no test compared `Sparse`'s raw `partial_*`
+  output against dense on real data — only the diagonal-blind
+  `jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested):
+  1. `partial_jaccard`'s diagonal was `(0, 2×col_weights[i])` instead of a
+     genuine self-comparison `(col_weights[i], col_weights[i])` —
+     `col_weights_and_pair_counts`'s `inter` never pairs a column with
+     itself by construction.
+  2. `partial_hamming`'s off-diagonal formula itself was wrong: `total -
+     union` (count of rows where *neither* column is present) instead of
+     the actual Hamming distance `col_weights[i] + col_weights[j] -
+     2×inter[i,j]` (symmetric-difference size). Only coincides with the
+     correct value when `col_weights[i] + col_weights[j] == total`, so
+     small/synthetic test data could easily have hidden it.
+
+  Neither surfaced through `jaccard_dist_matrix`/`hamming_dist_matrix`
+  (both explicitly zero their own diagonal at finalisation, and the
+  off-diagonal `partial_hamming` bug had gone untested against dense
+  entirely) — only visible to a caller of the raw `partial_*` methods
+  directly, which is exactly what `partial_threshold_jaccard(1)`'s new
+  shortcut became. Fixed at the source, not patched around at the call
+  site; regression test added:
+  `tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal`.
+
+- **Tests**: `tests::intmatrix::sparse_roundtrip_matches_columnar`/
+  `sparse_roundtrip_from_packed` (the `open`-dispatch fix, both build
+  paths); `tests::intmatrix::sparse_count_partials_match_dense` (all six
+  `CountPartials` formulas, thresholds 0/1/2/3, against `Columnar` on
+  asymmetric-presence data — this is what caught the diagonal gap in the
+  int side's own new code before it shipped, the same way it exposed the
+  two pre-existing bit-side bugs above); `obikindex`'s
+  `count_layer_transparently_reads_sparse_after_pack` — the actual
+  end-to-end regression test for the original "layer unreadable after
+  `pack --sparse`" bug, built → packed sparse → reopened, compared against
+  the pre-pack dense read. `cargo test -p obicompactvec -p obikindex`:
+  green, no regressions (180 + 12 tests).

DevDocMD/implementation/benchmark_query_testing.md

@@ -0,0 +1,93 @@
+# Benchmark: query-path testing
+
+`benchmark/Makefile` exercises indexing, merge, and phylo distance
+reconstruction against simulated bacterial genomes. It now also covers
+`obikmer query` — the read-matching path — and the sparse packed
+presence-matrix format (`obikmer pack --sparse`), previously untested by
+this pipeline.
+
+## Motivation
+
+- `query` had no end-to-end coverage. A regression there would not be caught
+  by `verify_presence`/`verify_merge_presence`, which only check index
+  *content* against the `.npz` truth, never the query API.
+- `pack --sparse` produces a presence-matrix format documented (see
+  [siblings.md](../architecture/siblings.md)) as faster for single-row
+  access (query) and slower for column-oriented access (phylo `--metric`).
+  `global_index_presence/` built by `merge_presence.sh` is always packed
+  dense (packing is a stage inside `merge`, not a separate `pack`
+  invocation) — there was no dense/sparse regression check.
+
+## Query read source
+
+Query reads are independent of `simulated_data/` (which is folded into the
+index being queried): reusing those reads would test against the exact
+error draw the index was built from. `query_data/<species>/<strain>/` holds
+a *second*, independent `iss generate` run against the same reference
+genome, via `simulate_query_one.sh` — unseeded, so a second draw picks up
+different sequencing errors than `simulate_one.sh`'s draw for the same
+genome. Fixed at 100,000 read pairs per genome (not coverage-proportional
+like the 15x used for `simulated_data/`), so wall/RSS numbers stay
+comparable across genomes of very different sizes.
+
+Two query-source specimens, hardcoded as `QUERY_SPECIMENS` in
+`make_deps.py`: `Escherichia_coli--K-12_MG1655` (common, well-represented
+bacterium) and `Saccharolobus_islandicus--M.16.4` (the only archaeon in
+`SPECIES` — distant lineage, stresses the query path differently from a
+close-relative match). Two is enough to catch a dense/sparse regression
+without duplicating the exhaustive per-specimen coverage
+`verify_merge_presence` already provides across all `SPECIMENS`.
+
+## Sparse global index
+
+`global_index_presence_sparse/` is built by `pack_sparse.sh`: copy
+`global_index_presence/` wholesale, then `obikmer pack --sparse` in place.
+This works directly because `merge`'s pack stage (`merge.rs:252`,
+`pack_matrices(false)`) keeps the per-genome column files on disk after
+dense-packing — `pack_sparse_bit_matrix` (`obicompactvec/src/bitmatrix/sparse.rs:447`)
+reads those, is idempotent, and removes `matrix.pbmx` once the sparse form
+is written, so `Persistent::open` falls through to the sparse format
+afterward. No separate merge run needed.
+
+## Query runs
+
+`query_one.sh dense|sparse SPECIMEN` runs `obikmer query --count-missing`
+against `global_index_presence` or `global_index_presence_sparse`, output
+gzipped to `query_{dense,sparse}/SPECIMEN.fasta.gz`, Reporter wall/RSS
+captured to `stats/query_{dense,sparse}/SPECIMEN.stats` (same
+stderr-parsing convention as `merge_presence.sh`).
+
+Flags: `--count-missing` only. `--mismatch` is a no-op today
+(`query/mod.rs:212-213`, prints "not yet implemented, ignored") — left off
+rather than tested for a feature that doesn't exist yet.
+
+## Dense/sparse regression
+
+`verify_query.py` compares the two query outputs per specimen, matched by
+read id (not stream position — the query pipeline chunks input across
+worker threads and doesn't guarantee output order). Compares `kmer_count`,
+`kmer_missing`, and the full `kmer_strict_matches` map per read. Any
+mismatch is a real regression: dense and sparse must be content-identical,
+only I/O access pattern differs. `.stats` → `stats/verify_query/`,
+aggregated by `aggregate_stats.sh query|verify_query`-style cases
+(`query_dense`, `query_sparse`, `verify_query`).
+
+## Performance comparison
+
+No dedicated script: the wall/RSS columns from the `query_dense` and
+`query_sparse` aggregated `.stats` CSVs are the dense-vs-sparse performance
+comparison — the expected win for query on sparse, per the `pack --sparse`
+help text.
+
+## Scope
+
+`count` track excluded from the sparse branch: `pack --sparse` targets
+presence matrices only (per CLI help); `pack_matrices` leaves count
+matrices untouched regardless of the `sparse` flag
+(`obikindex/src/index.rs:308`).
+
+## New Makefile targets
+
+`simulate_query`, `pack_sparse`, `query_dense`, `query_sparse`,
+`aggregate_query_dense`, `aggregate_query_sparse`, `verify_query`,
+`aggregate_verify_query` — the last three folded into `all`.

DevDocMD/implementation/evidence_elimination.refs.md

@@ -3,9 +3,9 @@
 
 ## Code couvert
 
-- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
-- `obilayeredmap/src/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
-- `obilayeredmap/src/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
+- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
+- `obikindex/src/layer/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
+- `obikindex/src/layer/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
 - `obikindex/src/reindex.rs` — KmerIndex::reindex(), conversion exact↔approx en place
 - `obikmer/src/cmd/reindex.rs` — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size
 - `obikmer/src/cmd/index.rs` — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp

DevDocMD/implementation/filtering.md

@@ -303,7 +303,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
 
 ## Implementation
 
-- **`obikpartitionner::filter::GroupQuorumFilter`** — implements `KmerFilter`
+- **`obikindex::partition::filter::GroupQuorumFilter`** — implements `KmerFilter`
   using pre-computed ingroup and outgroup index vectors. The heavy logic
   (predicate parsing, three-value evaluation, genome classification) happens
   once before any iteration; each k-mer row evaluation is a simple index
@@ -314,7 +314,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
   `UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
   list.
 
-- **`obikpartitionner::KmerPartition::iter_partition_kmers`** — accepts
+- **`obikindex::partition::KmerPartition::iter_partition_kmers`** — accepts
   `filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
   the callback. `filter`, `dump`, and `unitig` all go through this single
   entry point.

DevDocMD/implementation/layer_tier.md

@@ -1,8 +1,8 @@
-# obilayeredmap — layered kmer index crate
+# obikindex::layer — the Layer tier
 
 ## Purpose
 
-`obilayeredmap` implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
+`obikindex::layer` (the `layer/` submodule of the `obikindex` crate — a standalone `obilayeredmap` crate until 2026-08-21, folded back in alongside `obikpartition` as part of a broader `Index { Partition { Layer } }` submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
 
 ---
 
@@ -250,6 +250,59 @@ Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers buil
 
 ---
 
+## Layer\<D\> — raw mapping, iteration, and batch access
+
+Beyond `query`/`find` (membership-checked), `Layer<D>` exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.
+
+### Raw kmer → slot mapping
+
+```rust
+pub fn index(&self, kmer: CanonicalKmer) -> usize
+pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize>
+```
+
+Pure MPHF mapping, no evidence/fingerprint check — equivalent to `MphfOnly::index`. Only meaningful when the caller already knows `kmer` belongs to the layer; on an absent kmer the MPHF still returns *some* slot (undefined, not `None`).
+
+### Kmer iteration
+
+Four iterators, all built from `unitigs.bin` (physical layout order, **not** correlated with MPHF slot numbers):
+
+```rust
+pub fn iter_kmers(&self) -> KmerIter<'_>
+pub fn enumerate_kmers(&self) -> Enumerate<KmerIter<'_>>              // (order_index, kmer)
+pub fn iter_kmers_batch(&self, n: usize) -> KmerBatchIter<'_>          // Vec<CanonicalKmer> of size ≤ n
+pub fn enumerate_kmers_batch(&self, n: usize) -> impl Iterator<Item = (usize, Vec<CanonicalKmer>)> + Send + 'static
+```
+
+`KmerIter`/`KmerBatchIter` own a clone of the underlying `Arc<UnitigFileReader>` rather than borrowing `self` — `Send + 'static`, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. `enumerate_kmers_batch`'s index is the batch's starting offset in iteration order (a multiple of `n` except for the final, possibly shorter, batch).
+
+### Batch lookup on payload vectors/views
+
+`PersistentCompactIntVec`, `PersistentBitVec`, `IntSliceView`, `BitSliceView` all expose:
+
+```rust
+fn get_batch(&self, slots: &[usize]) -> Vec<T>
+fn fill_batch(&self, slots: &[usize], out: &mut [T])
+```
+
+Both sort `slots` internally for sequential mmap access, then reorder results back to the caller's original order. `fill_batch` fills a caller-provided buffer, avoiding the `Vec` allocation.
+
+### sub_matrix / fill_sub_matrix
+
+```rust
+// Layer<PersistentCompactIntMatrix>
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<u32>>            // column-first
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<u32>])
+
+// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
+```
+
+Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
+
+---
+
 ## LayeredStore\<S\> and aggregation traits
 
 `LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls:

DevDocMD/implementation/layer_tier.refs.md

@@ -1,14 +1,14 @@
 <!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
-# Coverage: implementation/obilayeredmap.md
+# Coverage: implementation/layer_tier.md
 
 ## Code couvert
 
-- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
-- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
-- `obilayeredmap/src/map.rs` — LayeredMap<D>, push_layer(), query()
-- `obilayeredmap/src/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
-- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
-- `obilayeredmap/src/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
+- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
+- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
+- `obikindex/src/layer/map.rs` — LayeredMap<D>, push_layer(), query()
+- `obikindex/src/layer/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
+- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
+- `obikindex/src/layer/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
 
 ## Notes
 

DevDocMD/implementation/merge.md

@@ -194,3 +194,14 @@ spectrums/
   <label>.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 ```
+
+`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json` marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — `merge_partition` never rewrites them for pre-existing layers, only the `presence`/`counts` subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
+
+## Known issue (not yet fixed, 2026-08-28)
+
+Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. `obikmer merge -o out IDX IDX --rename-duplicates`) — panics deep in the MPHF's rank-select structure (`common_traits::select_in_word`, `assertion failed: rank < self.count_ones()`), inside `MphfLayer::find` called from `merge_partition`'s "is this source kmer already in dst" check against the bootstrap-copied `dst_layers`. Root cause not identified; ruled out so far:
+
+- **Not an empty-new-layer issue**: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
+- **Not caused by the hard-link change above**: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
+
+Only reproduces when `sources[0]` and `sources[1]` are the exact same on-disk path opened as two independent `KmerIndex` handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since `PartitionRunner`'s panic propagation fix (see `architecture/numa_partition_runner.md`) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.

DevDocMD/implementation/merge.refs.md

@@ -4,9 +4,9 @@
 ## Code couvert
 
 - `obikindex/src/merge.rs` — `KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
-- `obikpartitionner/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
-- `obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
-- `obilayeredmap/src/layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
+- `obikindex/src/partition/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
+- `obikindex/src/partition/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
+- `obikindex/src/layer/typed_layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
 - `obicompactvec/src/intmatrix.rs` — `append_column` pour PersistentCompactIntMatrix
 - `obicompactvec/src/bitmatrix.rs` — `append_column` pour PersistentBitMatrix
 

DevDocMD/implementation/mphf.md

@@ -6,7 +6,7 @@ Kmer indexing per partition proceeds in two phases. The separation is necessary
 
 ### Phase 1 — provisional MPHF + kmer spectrum
 
-Implemented in `obikpartitionner::KmerPartition::count_kmer()` → `count_partition()`.
+Implemented in `obikindex::partition::KmerPartition::count_kmer()` → `count_partition()`.
 
 1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
 2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
@@ -101,7 +101,7 @@ type Mphf = PtrHash<
 
 ### Layer structure
 
-Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
+Each layer is a self-contained unit. See [obikindex::layer](layer_tier.md) for the full on-disk layout. The MPHF-relevant files are:
 
 ```
 layer_i/
@@ -148,7 +148,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
 
 There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
 
-In `obikpartitionner`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
+In `obikindex::partition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
 
 ### Membership verification
 

DevDocMD/implementation/mphf.refs.md

@@ -3,8 +3,8 @@
 
 ## Code couvert
 
-- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
-- `obikpartitionner/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
+- `obikindex/src/layer/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
+- `obikindex/src/partition/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
 
 ## Notes
 

DevDocMD/implementation/obicompactvec.md

@@ -292,14 +292,20 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
 
 ## Matrix types
 
-Four matrix types, two encodings × two formats:
+Both matrix types are enums behind a transparent API — the caller never matches on the variant. `PersistentCompactIntMatrix` has three variants (`Columnar`, `Packed`, `Sparse`). `PersistentBitMatrix` has four:
 
-| | Columnar format | Packed format |
+| Variant | Storage | When |
 |---|---|---|
-| **Bit** | `PersistentBitMatrix` (Columnar variant) | `PersistentBitMatrix` (Packed variant) |
-| **Int** | `PersistentCompactIntMatrix` (Columnar variant) | `PersistentCompactIntMatrix` (Packed variant) |
+| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
+| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
+| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off |
+| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
 
-Both matrix types are enums (`Columnar` / `Packed` / `Implicit` for bit) behind a transparent API. `col_view(c)` returns the appropriate view directly:
+`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
+
+Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation.
+
+`col_view(c)` returns the appropriate view directly:
 
 ```rust
 // PersistentBitMatrix

DevDocMD/implementation/obipipeline.refs.md

@@ -9,5 +9,5 @@
 ## Notes
 
 Document stable (librairie générique, peu de risque de dérive).
-Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartitionner`
+Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikindex::partition`
 ou s'il a été remplacé par Rayon dans certains chemins.

DevDocMD/implementation/pipeline.md

@@ -203,7 +203,7 @@ part_XXXXX/
 
 **Cleanup:** unless `--keep-intermediate` is set, `remove_build_artifacts` deletes `dereplicated.skmer.zst`, `mphf1.bin`, and `counts1.bin` after all partitions are indexed.
 
-See [obilayeredmap](obilayeredmap.md) and [MPHF selection](mphf.md) for data structure details.
+See [obikindex::layer](layer_tier.md) and [MPHF selection](mphf.md) for data structure details.
 
 **Query path (exact evidence):**
 

DevDocMD/implementation/pipeline.refs.md

@@ -3,12 +3,12 @@
 
 ## Code couvert
 
-- `obikpartitionner/src/partition.rs` — estimation des paramètres (phase 0)
+- `obikindex/src/partition/partition.rs` — estimation des paramètres (phase 0)
 - `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
-- `obikpartitionner/src/filter.rs` — déduplication bucket-sort (phase 2)
-- `obikpartitionner/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
+- `obikindex/src/partition/filter.rs` — déduplication bucket-sort (phase 2)
+- `obikindex/src/partition/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
 - `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
-- `obikpartitionner/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
+- `obikindex/src/partition/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
 - `obikindex/src/index.rs` — `build_layers()`, `dereplicate_and_count()`
 
 ## Notes

DevDocMD/implementation/select.md

@@ -23,28 +23,37 @@ are preserved unchanged; only the data matrices are rewritten.
 
 ```sh
 obikmer select <input-index>
-        { --output <dir> | --in-place }
+        --output <dir>
         [--group    <name>:<pred>  ...]
         [--group-op <name>:<op>    ...]
         [--aggregate-by <key>          ]
         [--aggregate-op <op>           ]
         [--select   <col1,col2,...>    ]
         [--presence-threshold <N>      ]
+        [--force-copy                  ]
 ```
 
 ---
 
 ## Output destination
 
-Exactly one of `--output` or `--in-place` must be specified.
+`--output <dir>` is required — `select` always writes a new index; there is no
+`--in-place` mode (2026-08-28: never implemented, removed from the design). The
+source index is unchanged.
 
-**`--output <dir>`** — writes a new index to `<dir>`. The source index is
-unchanged. The MPHF and unitig files are copied; only the data matrices are
-rewritten with the new column layout.
+Each layer's kmer-identity files (`mphf.bin`/`unitigs.bin`/`evidence.bin`/
+`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json`) are never rewritten by a
+column projection/aggregation, so they are hard-linked into the output rather
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); `--force-copy`
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+`select` itself would affect the other). Only the `presence`/`counts`
+subdirectory is ever a genuinely new, independent file.
 
-**`--in-place`** — rewrites the data matrices of the source index directly.
-Removed or replaced columns are lost. The operation writes to temporary files
-first, then renames atomically, so an interrupted run leaves the index intact.
+To replace an index with a selected version of itself, select to a temporary
+directory and swap it in (`rm -rf INDEX && mv INDEX.tmp INDEX`) — the case
+`--in-place` used to cover.
 
 ---
 
@@ -186,12 +195,6 @@ obikmer select myindex --output out \
   --select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
 ```
 
-### In-place: keep only group A
-
-```sh
-obikmer select myindex --in-place --group "A:group=A" --select "A"
-```
-
 ### Compose with filter
 
 ```sh
@@ -207,28 +210,45 @@ obikmer select filtered --output final --aggregate-by site
 
 ## Implementation notes
 
-`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
-(rayon), each writing its output independently; results require no synchronisation
-because every partition owns a distinct set of files.
+`select` does not rebuild the MPHF. Every partition is processed independently
+(`PartitionRunner`), each writing its own output layers; no cross-partition
+synchronisation is needed.
 
-For each layer in each partition:
+For each layer in each partition (`obikselect::select_layer::select_partition`):
 
-1. The slot count `n` is read by opening the source data matrix.
-2. A new data matrix is built with M columns (M = number of output columns).
-3. For each slot `s` in `0..n`:
-   - `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
-   - For each output column `j`:
-     - `new_row[j] = aggregate(op, old_row[group_indices])`.
-     - Pass-through columns are represented as single-element groups with the
-       default operator (`any` for presence, `sum` for count) — same code path.
-   - The new row is written slot by slot into each column builder.
-4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
-   evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
-   or `counts/` subdirectory is rewritten.
+1. `copy_layer_files` hard-links the source layer's kmer-identity files
+   (`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/
+   `fingerprint.bin`/`layer_meta.json`) into the destination — never a real
+   copy unless linking fails or `--force-copy` is given.
+2. A new data matrix is built with M columns (M = number of output columns),
+   under a fresh `presence/`/`counts/` subdirectory (never touching the
+   source's own).
+3. **Presence source (2026-08-28: `batch_presence_counts`)**: one shared pass
+   over the source bit matrix computes every output group's presence count at
+   once — row-major native for a `Sparse` source (`for_each_genome_in_row`,
+   which has no column representation to read a `col_view` from at all — the
+   reason this replaced the old per-group loop, not just an optimisation of
+   it), deduplicated column-major (one `col_view` per *distinct referenced
+   column*, not per group) for `Columnar`/`Packed`. Every `AggOp` for a bit
+   matrix is then a cheap derivation of that one count vector (`sum` = the
+   count itself, `any`/`max` = `count ≥ 1`, `all`/`min` = `count == group
+   size`, `none` = `count == 0`) — see
+   `obikselect::select_layer::agg_result_from_count`.
+4. **Count source**: unchanged, one `col_view`-driven pass per output column
+   via `MatrixGroupOps` — `sum`/`min`/`max` are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.
 5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
 
-**`--in-place` write strategy:** new data is written to a temporary sibling
-directory (`presence_new/` or `counts_new/`); on success the old directory is
-removed and the temporary one is renamed into place. An interrupted run leaves
-at most one stale `*_new/` directory; the original data is intact until the
-rename step.
+### Known gap (not yet fixed, 2026-08-28)
+
+Step 4 above still panics (`col_view() not available on Sparse
+PersistentCompactIntMatrix`) if the source is a **count** index packed
+sparse — `batch_presence_counts`' row-major treatment was only ported to the
+bit-matrix (`Presence`) case, since that was the one actually blocking a real
+benchmark run. `select`/`filter` on a sparse-packed count index still hits
+this; the fix would follow the same shape (a `PersistentSparseCompactIntMatrix`
+row-major decode, analogous to `for_each_genome_in_row`), just not done. Since
+`obisys::numa::runner::PartitionRunner`'s panic-propagation fix (see
+`architecture/numa_partition_runner.md`), this at least fails fast (process
+panic, exit 101) instead of hanging.

DevDocMD/implementation/storage.refs.md

@@ -5,7 +5,7 @@
 
 - `obikindex/src/meta.rs` — IndexMeta, IndexConfig (version, config, genomes)
 - `obikindex/src/index.rs` — layout sur disque : partitions/, index.meta
-- `obilayeredmap/src/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
+- `obikindex/src/layer/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
 - `obiskio/src/unitig_index.rs` — fichiers unitigs.bin + unitigs.bin.idx
 
 ## Notes

DevDocMD/implementation/unitig_evidence.refs.md

@@ -4,7 +4,7 @@
 ## Code couvert
 
 - `obiskio/src/unitig_index.rs` — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()
-- `obilayeredmap/src/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
+- `obikindex/src/layer/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
 - `obidebruinj/src/debruijn.rs` — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK
 
 ## Notes

DevDocMD/theory/evolutionary_distances.md

@@ -1489,7 +1489,7 @@ instead of a per-partition `partial_*`, run the sequential source sweep:
 
 ```text
 for p in 0..n_partitions:                       # OUTER — sequential
-    open source partition p's layers (QueryLayer-style, obikpartitionner)
+    open source partition p's layers (QueryLayer-style, obikindex::partition)
     enumerate distinct canonical k-mers of p (one per MPHF slot) with their
         presence/count vectors                   # column-major, as query stage 2
     par_iter over these source k-mers:           # INNER — rayon, thread-local tally
@@ -1513,7 +1513,7 @@ for p in 0..n_partitions:                       # OUTER — sequential
 ```
 
 The inner lookup is precisely `QueryLayer::find_slot` +
-`col_value(g, slot)` (`obikpartitionner/src/query_layer.rs`) — reuse or factor
+`col_value(g, slot)` (`obikindex/src/partition/query_layer.rs`) — reuse or factor
 out that path rather than reimplementing MPHF access. Enumerating "all distinct
 k-mers of a partition with their vectors" is the `dump`/`query` stage-2
 column-major scan already implemented in `dump_layer.rs` /
@@ -2182,6 +2182,230 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
 from the written `_iqtree_states.csv` and `A`/`C` still present). Full
 workspace `cargo test` green.
 
+## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+
+**Implemented.** `--metric` (renamed `--distance` — several of
+its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
+`--metric` was a misnomer) gains a family of `snp-*` values computed from the
+central-position SNP pipeline, routed internally to the sibling-annex
+machinery (`PairwiseTally`, `obikphylo::siblings::algorithms::pairwise`)
+instead of `cache.distance(...)`'s existing per-layer traversal — a different
+code path behind the same CLI surface, not just another branch of one
+formula function.
+
+**Why unify at the CLI level despite the implementation split**: phylogenetically
+a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
+to how the matrix was produced, so exposing it as a special-cased subcommand
+instead of a `--distance` value would misrepresent its role. The
+implementation divergence (sibling-annex-based vs. plain index scan) is real
+but belongs at the routing layer, invisible to the CLI's own vocabulary.
+
+**`--subsample` becomes optional for `snp-*` distances** (it stays mandatory
+for `--sankoff`/`--pseudo-alignment`, unrelated commands): absent means
+exhaustive, achieved for free by reusing `sample_index`'s existing
+proportional-per-layer-quota mechanism with `n` set to the index-wide total
+non-monomorphic-minorant count (already available from the sibling-annex
+stats) — every layer's quota then equals its own full count, giving Bernoulli
+`p = 1` everywhere, i.e. every eligible family is drawn. No second,
+exhaustive-only driver needed. Present means sampled, exactly as `--sankoff`
+already behaves.
+
+**One shared tally, many derived formulas.** `PairwiseTally`'s `subst[4][4]`
+per-pair substitution counts (plus marginal base frequencies derived from it)
+are the sufficient statistic for every closed-form correction below — each
+is a small pure function `PairwiseTally -> Array2<f64>`, at the same level as
+the already-implemented `raw_snp_distance`/`base_pair_tally`/
+`cardinality_tally`. No new full scan per formula, whether the tally itself
+was built exhaustively or from a subsample.
+
+**`--raw-snp-counts` stays a separate, unrelated flag** — same underlying
+tally, but a diagnostic (`n_snp`/`n_shared`/`n_eligible` per genome pair, one
+row per pair) rather than a distance value, and its long-table shape doesn't
+fold into a single N×N matrix the way a distance does. No change to its
+existing CSV format.
+
+### `snp-*` distance catalog
+
+All closed-form (method-of-moments / direct formula), none requiring
+per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
+*tree*-ML usage but not its *pairwise* estimator, which is closed-form like
+F84/TN93 and is included below. `snp-` prefix on every CLI value.
+
+| value | corrects for | inputs beyond raw counts |
+|---|---|---|
+| `snp-raw` | nothing (uncorrected p-distance) | — |
+| `snp-jc` (Jukes-Cantor, JC69) | multiple substitutions per site | — |
+| `snp-k2p` (Kimura 2-parameter, K80) | + transition/transversion rate bias | ts/tv split |
+| `snp-k81` (Kimura 3-parameter, K3ST) | + splits transversions into 2 categories | ts/tv split, by category |
+| `snp-f81` (Felsenstein 81) | + unequal base frequencies (no ts/tv split) | empirical base freqs |
+| `snp-tajima-nei` (Tajima-Nei 1984) | same goal as F81 (equal-input model), different formula, better small-sample behavior | empirical base freqs |
+| `snp-t92` (Tamura 3-parameter) | K2P + GC-content bias | ts/tv split, GC content |
+| `snp-f84` (Felsenstein 84) | full empirical base freqs + single ts/tv rate | empirical base freqs, ts/tv split |
+| `snp-hky85` (Hasegawa-Kishino-Yano, pairwise estimator) | same inputs as F84, different formula | empirical base freqs, ts/tv split |
+| `snp-tn93` (Tamura-Nei) | full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate | empirical base freqs, purine-ts/pyrimidine-ts/tv split |
+| `snp-logdet` (LogDet / paralinear) | no shared-model or stationarity assumption at all — general divergence-matrix determinant | full empirical 4×4 divergence matrix (already `subst[4][4]`) |
+| `snp-tv` (transversions-only p-distance) | diagnostic/deep-divergence variant — drops transitions entirely (they saturate first) | tv-only counts |
+
+**`+Γ` rate-heterogeneity modifier, applicable to `snp-jc`, `snp-k2p`,
+`snp-k81`, `snp-t92`, `snp-f84`, `snp-hky85`, `snp-tn93`** (not `snp-raw`,
+nothing to correct; not `snp-logdet`, no standard gamma formulation) — same
+formula as the base correction, weighted by a shape parameter `α` supplied
+by the user (`--gamma-shape <alpha>`), not estimated by ML. A modifier on
+existing values, not a separate enum arm per distance.
+
+**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
+`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
+"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
+`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
+needs the true *directional* per-pair base co-occurrence matrix
+(`PairwiseTally` only keeps the symmetrised substitution counts
+`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
+why that loses exactly the compositional-asymmetry information LogDet
+exists to detect), `snp-tajima-nei` needs each genome's *own* base
+composition (not the pair-pooled estimate the formulas below use), and
+`snp-f84`/`snp-hky85` had no formula independently verified against a
+primary source at implementation time (unlike every formula below, checked
+line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
+`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
+a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
+`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
+accumulator — not an architecture change.
+
+### Exact formulas (implemented, 2026-08-28)
+
+Sufficient statistic, per genome pair `(i, j)`, from
+`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
+`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
+
+- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
+  (pyrimidine transitions)
+- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
+  (Kimura's two transversion categories)
+- \(n_{shared}\): loci where both genomes agree
+- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
+  loci for the pair)
+- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
+  \(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
+  (both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
+  standard pairwise estimator, not a whole-index average)
+
+Derived proportions used below:
+
+\[
+p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
+P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
+Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
+Q_1 = \frac{n_{tv1}}{L}, \quad
+Q_2 = \frac{n_{tv2}}{L}, \quad
+P_1 = \frac{n_{ts1}}{L}, \quad
+P_2 = \frac{n_{ts2}}{L}
+\]
+
+Every formula below was checked term-by-term against `ape`'s own
+`src/dist_dna.c` (not re-derived from memory) before being ported to
+`obikphylo::siblings::algorithms::snp_distance`.
+
+**`snp-raw`** — uncorrected p-distance:
+
+\[
+d_{raw} = p
+\]
+
+**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
+dropping transitions, which saturate first, *is* the correction):
+
+\[
+d_{tv} = Q
+\]
+
+**`snp-jc`** (Jukes-Cantor, JC69):
+
+\[
+d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
+\]
+
+**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
+
+\[
+d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
+\]
+
+**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
+\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
+
+\[
+d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
+\]
+
+**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
+
+\[
+d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
+\]
+
+**`snp-t92`** (Tamura 3-parameter), with GC content
+\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
+\(a_2 = 1-2Q\):
+
+\[
+d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
+\]
+
+**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
+\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
+
+\[
+k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
+k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
+k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
+\]
+
+\[
+w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
+w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
+w_3 = 1 - \frac{Q}{2g_R g_Y}
+\]
+
+\[
+d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
+\]
+
+**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
+weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
+each such term with the same weight applied to
+\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
+substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
+recovering the uncorrected formula exactly). E.g. for JC:
+
+\[
+d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
+\]
+
+Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
+(including K80's two-term form — algebraically identical to the generic
+substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
+both symbolically and numerically before simplifying the implementation to
+share one `corrected_log` helper across every model rather than
+special-casing K80). K81/T92/TN93's gamma branches follow the same
+mechanical substitution but weren't independently checked against an
+`ape`-equivalent reference for those three specifically — flagged here, not
+silently assumed correct.
+
+### Output format: PHYLIP-relaxed by default for the distance matrix
+
+**Implemented.** The primary distance-matrix output
+(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
+default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
+T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
+label truncation, since genome labels here routinely exceed it), a `--csv`
+flag opts back into the current CSV format, PHYLIP-strict is a possible
+future addition (not now). This changes the *default* output of every
+existing `--distance` value (jaccard, hamming, bray-curtis, ...), not just
+the new `snp-*` ones — accepted explicitly (pre-release, single developer
+user, no external consumers to break). Scoped to the distance matrix only:
+`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
+their existing CSV-only format.
+
 ## References
 
 The Mash mutation-rate model this discussion contrasts with:

DevDocMD/theory/indexing.refs.md

@@ -3,8 +3,8 @@
 
 ## Code couvert
 
-- `obikpartitionner/src/partition.rs` — routage par hash de minimiseur, choix des paramètres
-- `obikpartitionner/src/lib.rs` — structure KmerPartition, nombre de partitions
+- `obikindex/src/partition/partition.rs` — routage par hash de minimiseur, choix des paramètres
+- `obikindex/src/partition/mod.rs` — structure KmerPartition, nombre de partitions
 
 ## Notes
 

Le_bug_des_A.md

@@ -1,43 +0,0 @@
-Voici la version corrigée :
-
----
-
-**Bug** : dans `base_pair_tally`, toutes les transitions/comptes depuis/vers A valent 0 dans `_sankoff_params.yaml`, alors que C/G/T sont corrects.
-
-**Contexte** : obikmer, pipeline phylogénétique `--sankoff`. L’index est construit sur 20 génomes bactériens. Même symptôme sur un jeu de 100 génomes de plantes : A est toujours à 0.
-
-**Fichier clé** : `src/obikphylo/src/siblings/sankoff_bundle.rs` (Pass A + Pass B).
-
-**Ce qui a été vérifié** :
-- Le fichier de sortie `_sankoff_params.yaml` montre bien `composition_transitions` avec A à 0 partout.
-- L’index contient bien des familles avec A (`mask.has(0) == true`), et même des familles où A co-existe avec d’autres bases (`mask == 0b0011` par ex.).
-- Un k-mer propriétaire de famille avec `mask == 0b0001` (A seul) a été identifié : forward `GAACAAGAGATCTCGATCTTGTCTACAAGGA`, revcomp `TCCTTGTAGACAAGATCGAGATCTCTTGTTC`.
-- Le diagnostic CLI sur l’index réel donne :
-  - Pass A : `a_pairs=623342 a_snp=623342 a_shared=0 a_both_a=0`
-  - Pass B : `families_with_a=22965521 a_single_form_genomes=22913238 a_included_pairs=0 a_same_incremented=0 bp_same=[0, 96389, 222720, 277909] bp_counts[0]=[0, 0, 0, 0]`
-
-**Interprétation** : A est fréquemment en `single_form` (mask == 1) chez certains génomes, mais **jamais simultanément** chez deux génomes différents dans la même famille. Donc toutes les paires “avec A” sont 100% SNP → ratio = 1.0 > `ratio_ceiling=0.5` → toutes exclues par le filtre `included`. C’est pourquoi `bp_same[0]` et `bp_counts[0][*]` restent à 0.
-
-**Point crucial** : le bug n’apparaît **que sur l’index compacté sparse**. Sur le même index avant compaction (matrice dense `matrix.pbmx`), `--sankoff` produit des tallies corrects pour A. Dès qu’on compacte avec `pack --sparse`, A disparaît.
-
-**Vérifications supplémentaires (diagnostic sparse)** :
-- La compaction `pack --sparse` produit une matrice `PersistentSparseBitMatrix` dont le contenu est **strictement identique** à la matrice dense d'origine : vérification exhaustive coordonnée par coordonnée sur **1 804 774 880 cellules** (512 partitions × 2 layers), **zéro différence**.
-- `fill_row` et `fill_sub_matrix` (les deux chemins de lecture utilisés par le pipeline phylogénétique) restituent les mêmes bits sur dense et sparse.
-- **Conclusion** : le bug n'est **pas** dans la compaction sparse elle-même, ni dans les chemins de lecture individuels. La structure stocke correctement A, C, G, T.
-
-**Conséquence logique** :
-Si les matrices sont identiques mais que le résultat final diffère, le bug se situe dans l'**intersection** des informations — c'est-à-dire dans le code qui **combine** les lectures des deux matrices (ou qui transforme les résultats bruts en tallies). Deux endroits possibles :
-1. **Le scan `sankoff_bundle`** (`family_scan.rs` + `sankoff_bundle.rs`) : la boucle qui lit les matrices, construit `genome_mask`, et accumule `bp_counts` / `same`. C'est l'étape d'intersection proprement dite.
-2. **La conversion des tallies en YAML** (`obikmer/src/cmd/phylo/sankoff.rs`) : moins probable, mais possible si quelque chose sélectionne/filtre les transitions avant écriture.
-
-**Hypothèse la plus probable** : bug dans la résolution cross-partition lors de la construction de l'annex sibling (`build_sibling_annex`). A (bit 0) serait systématiquement manquant ou mal résolu quand on interroge les variants d'une famille depuis une partition différente. À vérifier dans `src/obikphylo/src/siblings/build.rs` et `src/obikphylo/src/siblings/cache.rs` (`PartitionCache::find` / `find_presence_batch`).
-
-**Prochaine étape logique** :
-1. Inspecter `build_sibling_annex` pour voir si les variants avec base A sont bien générés et bien recherchés dans `cache.find`.
-2. Vérifier `PartitionCache::find` et `resolve_layer_hits` pour un éventuel biais contre le bit 0.
-3. Si besoin, ajouter un diagnostic ciblé (compteurs par base) **uniquement** dans `cache.rs` ou `build.rs`, pas dans `sankoff_bundle.rs` qui est déjà propre.
-
-**Contraintes** :
-- Ne pas modifier `sankoff_bundle.rs` davantage.
-- Ne pas toucher à git.
-- Faire des diagnostics minimaux et ciblés.

UserDocMD/architecture.md

@@ -20,7 +20,7 @@ An index directory is organized as `KmerIndex → partitions → layers`, with a
 
 ## Parallel execution and NUMA awareness
 
-Partition-level work (index construction, `merge`, `filter`, `reindex`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
+Partition-level work (index construction, `merge`, `filter`, `convert`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
 
 - On a multi-socket / multi-NUMA-node machine, one thread pool is pinned per NUMA node, and each partition is processed entirely by threads pinned to one node — keeping the memory a partition touches local to that node's DRAM. This matters because touching kmer data across NUMA nodes without pinning can degrade throughput by an order of magnitude or more on large multi-socket machines.
 - On a single-socket machine, Apple Silicon, or if hwloc cannot report NUMA topology, all cores are treated as one node with no pinning and negligible overhead — this is the default behavior on macOS.

UserDocMD/formats/index_layout.md

@@ -19,7 +19,7 @@ Each partition's surviving kmers are mapped to a dense range of integer slots by
 
 ## Evidence: exact vs. approximate
 
-Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`reindex`](../usage/reindex.md)):
+Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)):
 
 - **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
 - **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
@@ -51,6 +51,6 @@ Two verification modes are available, selected at build time (`index --approx`)
 
 A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
 
-Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`reindex`](../usage/reindex.md) one of the sources first if needed.
+Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`convert`](../usage/convert.md) one of the sources first if needed.
 
 `obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time.

UserDocMD/index.md

@@ -24,11 +24,10 @@ All functionality is exposed through a single binary, `obikmer`, organized as su
 | [`query`](usage/query.md) | Query an index with sequences and annotate matches |
 | [`dump`](usage/dump.md) | Dump indexed kmers as CSV |
 | [`annotate`](usage/annotate.md) | Add, update, or dump genome metadata |
-| [`phylo`](usage/phylo.md) | Compute pairwise evolutionary-distance proxies, trees, and phylogenetic exports |
-| [`name-tree`](usage/name-tree.md) | Translate a TNT/PhyG numeric-label tree export back to real taxon names |
+| [`phylo`](usage/phylo.md) | Compute pairwise genome distances, trees, and phylogenetic exports |
 | [`unitig`](usage/unitig.md) | Dump the unitigs of an index as FASTA |
 | [`estimate`](usage/estimate.md) | Estimate approximate-index parameters before indexing |
-| [`reindex`](usage/reindex.md) | Convert an index's evidence representation (exact ↔ approximate) |
+| [`convert`](usage/convert.md) | Convert an index's evidence representation (exact/approximate/hybrid), in place |
 | [`utils`](usage/utils.md) | Miscellaneous index maintenance and inspection utilities |
 | [`pack`](usage/pack.md) | Pack per-column matrix files into a single-file format |
 

UserDocMD/usage/convert.md

@@ -0,0 +1,29 @@
+# convert
+
+Convert an existing index's evidence representation in place, between exact, approximate, and hybrid.
+
+```bash
+obikmer convert INDEX (--exact-evidence | --approx-evidence BITS | --hybrid-evidence) [OPTIONS]
+```
+
+## Arguments
+
+| Argument | Description |
+|---|---|
+| `INDEX` | Index directory to convert (modified in place) |
+
+## Options
+
+Exactly one of the first three is required:
+
+| Option | Description |
+|---|---|
+| `--exact-evidence` | Convert to exact evidence (zero false positives) |
+| `--approx-evidence BITS` | Convert to approximate (fingerprint-only) evidence; `BITS` = fingerprint bits per slot (b) |
+| `--hybrid-evidence` | Convert to hybrid evidence (both exact and approximate bundles kept) |
+| `--evidence-bits BITS` | Fingerprint bits per slot (b) — required with `--hybrid-evidence` when the source index is currently exact; rejected otherwise (the source already fixes `b`) |
+| `-z, --findere-z Z` | Findere z parameter: number of consecutive stored kmers that must all match to confirm a hit. This does not shorten the indexed kmer length (fixed forever at `index` build time) — it extends the effective match window: on a k=31 index, `z=2` requires 32 consecutive matching bases, not 30 |
+| `--fp FP` | Target false-positive rate per z-window (e.g. `0.01`); derives `b` or `z` when one of them isn't given directly |
+| `--block-size N` | Block size for exact evidence's on-disk index (unitigs per block). Ignored when converting to pure approximate evidence. Default `1` |
+
+See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.

UserDocMD/usage/estimate.md

@@ -15,4 +15,4 @@ obikmer estimate [OPTIONS]
 | `--evidence-bits` | none | Fingerprint bits per slot (b) |
 | `--fp` | none | Target false-positive rate per z-window |
 
-Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `reindex --approx` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
+Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.

UserDocMD/usage/index_command.md

@@ -45,6 +45,6 @@ With `--approx`, evidence is stored as a compact **fingerprint** instead, tradin
 
 $$FP = \frac{1}{2^{b \cdot z}}$$
 
-where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`reindex`](reindex.md) to convert an existing index between the two representations.
+where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`convert`](convert.md) to change an existing index's representation afterwards.
 
 `z` must be strictly less than k: the effective indexed kmer length under approximate evidence is k−z+1.

UserDocMD/usage/name-tree.md

@@ -1,21 +0,0 @@
-# name-tree
-
-Translate a numerically-labelled tree export (TNT, PhyG, or any plain Newick file with bare `1`, `2`, `3`, … leaf labels) back to real taxon names, reading the label order from the FASTA that produced it.
-
-```bash
-obikmer name-tree TREE --fasta FASTA -o OUTPUT
-```
-
-## Arguments
-
-| Argument | Description |
-|---|---|
-| `TREE` | Tree file to translate — a TNT-style NEXUS export (`tree NAME = [&U] ...;`) or a plain Newick file |
-| `--fasta` | FASTA file whose record order gives the numeric taxon labels (1-based) — typically the `_sankoff.fasta`/`_snp.fasta` used to produce `TREE` |
-| `-o, --output` | Output NEXUS file path |
-
-## Output
-
-A NEXUS file with a `taxa` block, a `translate` table (numeric label → taxon name, from `--fasta`'s header order), and every tree found in `TREE`, topology unchanged — readable directly in FigTree, PearTree, `ape` (R), etc.
-
-`--tnt`'s and `--phyg`'s exports (see [phylo](phylo.md)) both number taxa `1..N` in the same order as the pseudo-alignment FASTA they were built from (`<prefix>_sankoff.fasta`), so pass that same file as `--fasta` here.

UserDocMD/usage/pack.md

@@ -16,14 +16,14 @@ obikmer pack INDEX [--sparse]
 
 | Option | Default | Description |
 |---|---|---|
-| `--sparse` | off | Pack presence/absence matrices into a sparse, deduplicated format instead of the dense one |
+| `--sparse` | off | Pack presence/absence and count matrices into a sparse, deduplicated format instead of the dense one |
 
 The index directory is locked for exclusive access while packing.
 
 ## `--sparse`
 
-Presence/absence data (which genomes carry each kmer) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one bit per genome for every kmer regardless of how many genomes actually carry it; `--sparse` instead stores each kmer's genome list directly, and deduplicates identical lists shared by many kmers (common in real data, since kmers from the same conserved region tend to be carried by the same genomes).
+Matrix data (which genomes carry each kmer, or with what count) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one entry per genome for every kmer regardless of how many genomes actually carry it; `--sparse` instead stores each kmer's genome list directly. For presence/absence matrices, identical genome lists shared by many kmers are also deduplicated (common in real data, since kmers from the same conserved region tend to be carried by the same genomes); for count matrices, the genome list is deduplicated the same way but each kmer's actual counts are kept per-kmer, since two kmers sharing the same genome list rarely carry the same counts.
 
-On real genome collections this has measured at roughly 7x smaller on disk than the dense format, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--metric` distance-matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s plain `--metric` distance matrices.
+On real genome collections this has measured at roughly 7x smaller on disk than the dense format for presence/absence, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--distance` matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s `--distance` matrices.
 
-Count matrices (`--metric` on a count index) are not affected by `--sparse` — only presence/absence matrices are.
+`--sparse` applies to both presence/absence and count matrices — a count index (`--distance` matrix computations included) is packed sparse the same as a presence index.

UserDocMD/usage/phylo.md

@@ -1,6 +1,6 @@
 # phylo
 
-Compute pairwise evolutionary-distance proxies between the genomes stored in an index — a plain distance matrix, optionally trees (NJ/UPGMA), and optionally a central-position SNP model with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
+Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
 
 ```bash
 obikmer phylo INDEX [OPTIONS]
@@ -12,133 +12,217 @@ obikmer phylo INDEX [OPTIONS]
 |---|---|
 | `INDEX` | Index directory |
 
-## Distance matrix
+## Distance matrix (`--distance`)
 
 | Option | Default | Description |
 |---|---|---|
-| `--metric` | `jaccard` | One of `jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` |
-| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for Jaccard/Mash on a count index |
-| `--shared-kmers` | off | Also write the shared-kmer count matrix |
+| `--distance` | `jaccard` | See the two tables below for the full list of accepted values |
+| `--gamma-shape ALPHA` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see below). No effect on the other values; rejected if given together with a value that doesn't support it |
+| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
+| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
+| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index metric, not a `snp-*` value |
 | `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
 | `--upgma` | off | Compute and write a UPGMA tree (Newick) |
-| `-o, --output` | none (stdout) | Output file prefix; without it, the distance matrix is printed to stdout as CSV |
+| `-o, --output` | none (stdout) | Output file prefix |
 
-`hamming` requires a presence/absence index. All other metrics work on either index type; on a presence index, `jaccard`/`mash`/`hamming` are the only ones available.
+Every value routes to one of two independent computations:
 
-### Metric definitions
+### Whole-index metrics
 
-- **jaccard**: $D = 1 - \dfrac{|A \cap B|}{|A \cup B|}$ over the sets of kmers present in each genome.
-- **mash**: derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$.
-- **hamming**: number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$.
-- **bray-curtis**: $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts.
-- **relfreq-bray-curtis**: the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts.
-- **euclidean**: $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts.
-- **relfreq-euclidean**: the same formula on relative frequencies.
-- **hellinger**: $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$.
-- **hellinger-euclidean**: the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$.
+| Value | Definition |
+|---|---|
+| `jaccard` | $D = 1 - \dfrac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}$ over the sets of kmers present in each genome |
+| `mash` | derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$ |
+| `hamming` | number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$ |
+| `bray-curtis` | $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts |
+| `relfreq-bray-curtis` | the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts |
+| `euclidean` | $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts |
+| `relfreq-euclidean` | the same formula on relative frequencies |
+| `hellinger` | $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$ |
+| `hellinger-euclidean` | the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$ |
+
+`hamming` requires a presence/absence index; the others work on either index type.
+
+### `snp-*` corrections
+
+Computed from the central-position SNP model (see "Central-position SNP model" below): a family is the set of up to 4 kmers sharing identical flanking sequence and differing only at the central base. These values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
+
+For a genome pair, let $L$ be its total number of eligible loci (both genomes single-copy at that family), $p$ the raw proportion of substitutions among those loci, $P$/$Q$ the transition/transversion proportions, $Q_1$/$Q_2$ Kimura's two transversion categories (A↔C & G↔T vs. A↔T & C↔G), $P_1$/$P_2$ the purine (A↔G) / pyrimidine (C↔T) transition proportions, and $\pi_A,\pi_C,\pi_G,\pi_T$ the pair's pooled base frequencies.
+
+**`snp-raw`**
+
+$$d = p$$
+
+**`snp-jc`**
+
+$$d = -\frac{3}{4}\ln\!\left(1-\frac{4p}{3}\right)$$
+
+**`snp-k2p`**
+
+$$
+\begin{aligned}
+a_1 &= 1-2P-Q \\
+a_2 &= 1-2Q \\
+d   &= -\frac{1}{2}\ln a_1-\frac{1}{4}\ln a_2
+\end{aligned}
+$$
+
+**`snp-k81`**
+
+$$
+\begin{aligned}
+a_1 &= 1-2P-2Q_1 \\
+a_2 &= 1-2P-2Q_2 \\
+a_3 &= 1-2Q_1-2Q_2 \\
+d   &= -\frac{1}{4}\left(\ln a_1+\ln a_2+\ln a_3\right)
+\end{aligned}
+$$
+
+**`snp-f81`**
+
+$$
+\begin{aligned}
+E &= 1-\left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right) \\
+d &= -E\ln\!\left(1-\frac{p}{E}\right)
+\end{aligned}
+$$
+
+**`snp-t92`**
+
+$$
+\begin{aligned}
+g   &= \pi_C+\pi_G \\
+w   &= 2g(1-g) \\
+a_1 &= 1-\frac{P}{w}-Q \\
+a_2 &= 1-2Q \\
+d   &= -w\ln a_1-\frac{1}{2}(1-w)\ln a_2
+\end{aligned}
+$$
+
+**`snp-tn93`**
+
+$$
+\begin{aligned}
+g_R &= \pi_A+\pi_G \\
+g_Y &= \pi_C+\pi_T \\
+k_1 &= \frac{2\pi_A\pi_G}{g_R} \\
+k_2 &= \frac{2\pi_C\pi_T}{g_Y} \\
+k_3 &= 2\left(g_Rg_Y-\frac{\pi_A\pi_G\,g_Y}{g_R}-\frac{\pi_C\pi_T\,g_R}{g_Y}\right) \\
+w_1 &= 1-\frac{P_1}{k_1}-\frac{Q}{2g_R} \\
+w_2 &= 1-\frac{P_2}{k_2}-\frac{Q}{2g_Y} \\
+w_3 &= 1-\frac{Q}{2g_Rg_Y} \\
+d   &= -k_1\ln w_1-k_2\ln w_2-k_3\ln w_3
+\end{aligned}
+$$
+
+**`snp-tv`** — transversions only, deliberately uncorrected:
+
+$$d = Q$$
+
+`--gamma-shape ALPHA` applies to every value above except `snp-raw` and `snp-tv`: each $-\ln(x)$ term in the formulas above is replaced by $\alpha\left(x^{-1/\alpha}-1\right)$ (the same weight, same $x$).
+
+### Output
+
+Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
+
+## `--exclude-genome`, `--min-shared-family`
+
+| Option | Description |
+|---|---|
+| `--exclude-genome LABEL` | Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by `--pseudo-alignment`/`--sankoff`/a `snp-*` `--distance` value. Does not change the value computed for any remaining pair |
+| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` — never to the whole-index metrics or their matrix/NJ/UPGMA output |
+
+Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
 
 ## Central-position SNP model
 
-This is a separate operation from the distance-matrix computation above: if any option below is used, no `--metric` matrix is computed in the same invocation.
+Requires the sibling annex, built once per index:
 
-A **family** is the set of up to 4 kmers that share identical flanking sequence and differ only at the exact central base. Because $k$ is odd, the central position is well defined and maps to itself under reverse complementation. All computations below first require building the **sibling annex**, an index-wide record of which of the 4 possible central bases are observed at each family, across every genome.
+| Option | Description |
+|---|---|
+| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value |
+| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
+| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
+| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
+| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
+| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--subsample N` |
+
+A family is eligible for a genome pair $(i,j)$ only if both genomes carry exactly one of its observed forms (single-copy, unambiguous).
+
+### `--sibling-stats`
+
+`<prefix>_siblings.csv` — family size = number of distinct central bases observed at a family (1-4).
+
+| Column | Meaning |
+|---|---|
+| `genome` | genome label, or the literal `global` for the last row |
+| `1`, `2`, `3`, `4` | for a genome row: number of families of that size where the genome carries ≥ 1 member. For the `global` row: the actual deduplicated family-size histogram — not the sum of the rows above |
+
+### Family Overlap
+
+`--family-overlap` writes `<prefix>_family_overlap.csv`: header `genome,<label1>,<label2>,...`, one row per genome, cell `[i][j]` = number of variable families (family size ≥ 2) where both genome `i` and genome `j` carry a call. The diagonal is always `0`. Every genome is written, unfiltered by `--exclude-genome`/`--min-shared-family`.
+
+`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
+
+### `--pseudo-alignment`
+
+`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
 
 | Option | Default | Description |
 |---|---|---|
-| `--sibling-annex` | off | Build the sibling presence-mask annex (prerequisite for every option below) |
-| `--exclude-genome LABEL` | none | Exclude a genome (repeatable) from every SNP/Sankoff/export computation below |
-| `--min-shared-family N` | none | Auto-exclude any genome whose mean shared-family count against every other genome (see `--family-overlap`) falls below `N` — same exclusion as `--exclude-genome`, applied on top of it |
-| `--sibling-stats` | off | Write the family-size (sibling count) distribution, per genome and globally |
-| `--raw-snp-distance` | off | Write the single-copy central-SNP p-distance matrix |
-| `--raw-snp-counts` | off | Write per-pair diagnostic counts (n_snp, n_shared, n_eligible) instead of a matrix |
-| `--snp` | off | Write a SNP-only pseudo-alignment in FASTA, IUPAC-coded |
-| `--family-overlap` | off | Write an NxN matrix of, for each genome pair, how many variable families both genomes actually carry a call for; the diagonal holds each genome's own total family count |
-
-### Locus eligibility
-
-A family is eligible for a genome pair $(i, j)$ only if genome $i$ carries exactly one of the family's observed forms (single-copy, unambiguous) and genome $j$ also carries exactly one. A genome carrying more than one form at a locus makes that locus ineligible for any pair involving it.
-
-`--raw-snp-distance` tallies, over every eligible locus of every genome pair, $n_{\text{snp}}$ (the two genomes' single forms differ) versus $n_{\text{shared}}$ (they agree — this includes invariant families). The output ratio is $\hat{p} = \dfrac{n_{\text{snp}}}{n_{\text{snp}} + n_{\text{shared}}}$.
-
-`--snp` restricts itself to *variable* families (family size $\ge 2$) and writes one FASTA record per genome, one column per family, IUPAC-coded from each genome's presence mask at that family (a single form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`).
-
-`--exclude-genome` removes a genome from these computations, re-checking column variability among the remaining genomes so that a column made monomorphic by the exclusion is dropped rather than kept artificially. It does not affect the `--metric` distance-matrix path.
-
-### Family overlap and low-coverage genomes
-
-`--family-overlap` writes, for every genome pair, how many variable families both genomes actually carry a call for (neither is absent) — a direct measure of how much informative content two genomes actually share. On genome-skim or otherwise incomplete-coverage collections, a genome with very little overlap with everything else has almost nothing left to constrain its position in a tree, and tends to end up placed unstably (near-zero branch length, grafted inside an unrelated clade) by `--tnt`/`--iqtree`.
-
-`--min-shared-family N` automates the fix: it excludes, before any computation, every genome whose mean shared-family count against all other genomes (the same statistic, averaged per row of the `--family-overlap` matrix) falls below `N`. There is no universal value for `N` — it depends on how divergent and how completely covered the genome collection is; inspect `--family-overlap`'s own output to find where the real gap sits before choosing a threshold.
-
-## Sampling at scale: `--subsample`, `--shannon`, `--entropy`
-
-On a large index (billions of families), building a full pseudo-alignment or fully calibrating the Sankoff model is not just slow — it may not fit in the time you have. `--subsample` bounds the work to a fixed number of families; `--shannon` reports how informative each family is; `--entropy`/`--entropy-sd` bias which families get kept toward the informative ones instead of choosing uniformly at random.
-
-| Option | Default | Description |
-|---|---|---|
-| `--subsample N` | none (keep everything) | Cap the number of variable families (family size ≥ 2) retained, to approximately `N` |
-| `--shannon` | off | Write `<prefix>_shannon.csv`: per-family Shannon entropy, one row per family |
+| `--subsample N` | none (mandatory here) | Target number of families to sample |
+| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
+| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
 | `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
 | `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
 
-`--subsample`/`--entropy`/`--entropy-sd` affect every option that scans variable families: `--snp`, `--family-overlap`, `--shannon`, and the whole Sankoff pipeline (`--sankoff`/`--tnt`/`--phyg`/`--iqtree`, next section) — all of them draw from the *same* selection of families in one invocation, so the Sankoff calibration and the pseudo-alignment it calibrates always describe the same sites, and `--family-overlap`'s counts stay consistent with `--snp`'s columns. `--raw-snp-distance`/`--raw-snp-counts` are not affected — they always scan every family, since their p-distance estimate is a whole-index statistic, not something that benefits from being restricted to a sample.
+## Sampling at scale: `--subsample`, `--shannon`, `--entropy`
 
-### `--subsample N`
+`--subsample`, `--free-loss`, `--no-ambiguity`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--subsample` is mandatory for `--pseudo-alignment`/`--sankoff`; for a `snp-*` `--distance` value it is optional (omitted means every non-monomorphic family in the index, not an approximation).
 
-Without `--subsample`, every variable family (family size ≥ 2, i.e. every family where at least one genome differs from the rest) is used. With `--subsample N`, roughly `N` families are kept instead, chosen at random but in proportion to how many candidate families each part of the index actually holds — so the sample stays representative of the whole index, not skewed toward whichever part happens to be scanned first. If the index has fewer than `N` candidate families in the first place, `--subsample` has no effect: everything is kept.
-
-`--subsample` trades completeness for speed: `--snp`'s alignment gets fewer columns, `--sankoff`'s calibration is based on fewer observations, but the resolution work (the expensive part of a `phylo` run on a large index) scales with `N` instead of with the index's true size. Pick `N` as large as your time budget allows — a few hundred thousand to a few million families is usually enough for the transition-probability estimates in `--sankoff`'s calibration to stabilize; a smaller `N` speeds up exploratory runs.
+Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead, drawn in proportion to how many candidate families each part of the index actually holds, so the sample stays representative of the whole index. If the index has fewer than `N` candidate families, `--subsample` has no effect.
 
 ### `--shannon`: measuring how informative a family is
 
-Not every variable family is equally useful for a tree: a family that differs in only one genome out of a thousand carries very little signal, and one where the pattern looks essentially random across genomes may be too saturated (multiple substitutions have overwritten the original signal) to carry real information either. `--shannon` quantifies this with the Shannon entropy (in bits) of each family's states across the genomes that carry it — low entropy means "almost everyone agrees" (an invariant or near-invariant family, phylogenetically shallow), while entropy near the ceiling for a 4-state character means "close to a random draw between the possible bases" (saturated).
-
-`<prefix>_shannon.csv` (or `shannon.csv` without `-o`) has one row per family visited:
+`<prefix>_entropy.csv` has one row per family visited:
 
 | Column | Meaning |
 |---|---|
 | `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
 | `family_idx` | the family's position within that layer |
-| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}` — the same alphabet `--sankoff`'s 16-state model uses), genomes absent from the family excluded from the count |
-| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison — a genome carrying more than one base at once counts once per base, so this can differ from `entropy15` |
-| `family_size` | number of distinct central bases observed anywhere in the index for this family (2–4, since monomorphic families aren't visited) |
+| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
+| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison |
+| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
 | `n_genomes_present` | how many genomes the entropy was computed over |
 
-Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful to inspect the entropy distribution and decide reasonable `--entropy`/`--entropy-sd` values (see below) before committing to a full run.
+Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful for choosing `--entropy`/`--entropy-sd` values before a full run.
 
 ### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
 
-By default, `--subsample` draws families uniformly — every candidate family has the same chance of being kept, regardless of how informative it actually is. `--entropy`/`--entropy-sd` change that: instead of a uniform draw, each family's chance of being kept is weighted by how close its own entropy (the `entropy15` value `--shannon` reports) is to `MU`, using a bell-shaped (Gaussian) curve of width `SIGMA` — a family with entropy exactly `MU` is the most likely to be kept, and the chance falls off smoothly the further its entropy is from `MU`, with no hard cutoff (a few families outside the target band can still get in, just less often).
+By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff. The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
 
-The filter activates as soon as *either* `--entropy` or `--entropy-sd` is given; whichever one you don't set defaults to `1.0`/`0.5`. It can be combined with `--subsample N` (the target count is still approximately `N`, now biased toward the entropy band instead of uniform — expect somewhat *fewer* than `N` families in practice, since low-weight families are dropped rather than replaced) or used alone (`--entropy` without `--subsample`: a soft entropy filter over the whole index, no size target).
-
-The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy has to be computed once, up front, and is then saved alongside the index). Every following run — even with different `MU`/`SIGMA` values — reuses that saved data and stays fast.
+The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
 
 ## Sankoff calibration and phylogenetic exports
 
 | Option | Default | Description |
 |---|---|---|
-| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment |
-| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the calibration |
-| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, in `--sankoff`'s pseudo-alignment and every export built from it |
+| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment. Requires `--subsample N` |
+| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the base-composition part of the calibration |
+| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, throughout `--sankoff` and every export built from it |
 | `--tnt` | off | Also write a TNT script (implies `--sankoff`) |
 | `--phyg` | off | Also write PhyG input files (implies `--sankoff`) |
 | `--iqtree` | off | Also write an IQ-TREE custom model and alignment (implies `--sankoff`) |
 | `--iqtree-min-freq` | `0.001` | With `--iqtree --free-loss`: also treat as missing any state rarer than this in the alignment |
-| `--sankoff-cost-scale` | `100` | Integer scaling factor applied to costs before rounding (required by TNT/PhyG's integer-only cost commands) |
+| `--sankoff-cost-scale` | `100` | Integer scaling factor applied to costs before rounding, for TNT/PhyG's integer-only cost commands |
 
 ### The 16-state model
 
-Each family is treated as a character with 16 possible states: one per subset of the 4 possible central bases actually observed (including the empty subset). Calibration combines two tallies, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`:
+Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). Calibration combines a $5 \times 5$ transition matrix over family cardinality (0-4 observed forms) and a $4 \times 4$ base-substitution matrix from unambiguous single-copy loci, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`, into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
 
-- a $5 \times 5$ transition matrix over family cardinality (0–4 observed forms) between paired genomes, and
-- a $4 \times 4$ base-substitution transition matrix from unambiguous single-copy loci,
+`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character). It does not run any external tool.
 
-which are combined into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
-
-`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character, to avoid ambiguity with external tools' own gap semantics). It does not run any external tool.
-
-With `--free-loss`, the empty state is recoded to `?` instead — TNT/PhyG/IQ-TREE's own missing-data symbol — rather than an ordinary, costed 16th state. This matters for genome-skim or otherwise incomplete-coverage collections, where non-detection of a family is dominated by sampling failure rather than true evolutionary loss: scoring it as a real state risks grouping genomes by shared undersampling instead of shared ancestry. `?` rather than `-` because `-` still carries gap/indel semantics in these tools, and a non-detected family is not an observed deletion. `--free-loss` also drops the cardinality-transition cost between any two states, not just to/from the empty one: whether a genome shows 1 vs. 2 (etc.) detected members of a family it does carry is exactly as vulnerable to sampling failure as whether the family was detected at all, so gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family. Combine with `--min-shared-family`/`--family-overlap` above: `--free-loss` removes the false signal from non-detection, but a genome left with too little real overlap with everything else will still be placed unstably — excluding it is the other half of the fix.
+With `--free-loss`, the empty state is recoded to `?` (TNT/PhyG/IQ-TREE's own missing-data symbol) instead of an ordinary, costed 16th state — `?` rather than `-`, since `-` still carries gap/indel semantics in these tools. `--free-loss` also zeroes the cardinality-transition cost between any two states, not just to/from the empty one: gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family.
 
 ### Exports
 
@@ -146,58 +230,34 @@ All three exports reuse the `--sankoff` calibrated matrix and pseudo-alignment,
 
 - **`--tnt`**: a self-contained TNT script (alignment recoded to TNT's fixed 16-symbol alphabet, integer-scaled cost matrix re-closed to a metric, a default search block).
 - **`--phyg`**: a custom cost-matrix file plus a PhyG script reusing the `--sankoff` alignment directly.
-- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths (unlike the parsimony step-counts from TNT/PhyG). Only states actually occurring in the alignment are kept and compactly renumbered.
+- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths. Only states actually occurring in the alignment are kept and compactly renumbered.
 
-TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, …, in the same order as `<prefix>_sankoff.fasta`). Use [`name-tree`](name-tree.md) on the tool's own tree output plus that same FASTA to get a NEXUS file with real taxon names.
+TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in the order the genomes appear in `<prefix>_sankoff.fasta`).
 
 ## Output files
 
-With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the plain `--metric` distance matrix is produced, on stdout. All matrices use genome labels (from the index metadata) as row/column headers, in index order; all CSVs are comma-separated with a header row.
+With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
 
 ### Distance matrix
 
 | File | Written by | Format | Content |
 |---|---|---|---|
-| `<prefix>_dist.csv` | always | CSV matrix | the `--metric` distance, 6 decimals, symmetric, diagonal 0 |
+| `<prefix>_dist.phy` | always, unless `--csv` | relaxed PHYLIP | the `--distance` matrix |
+| `<prefix>_dist.csv` | `--csv` | CSV matrix | the `--distance` matrix, 6 decimals |
 | `<prefix>_shared.csv` | `--shared-kmers` | CSV matrix | shared-kmer count per genome pair (integers) |
-| `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree, branch lengths from the `--metric` matrix |
-| `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree, same matrix |
+| `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree |
+| `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree |
 
-Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
+CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
 
 ### Central-position SNP model
 
 | File | Written by | Format | Content |
 |---|---|---|---|
 | `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
-| `<prefix>_rawsnp.csv` | `--raw-snp-distance` | CSV matrix | single-copy central-SNP p-distance ($\hat p$), or `NA` |
-| `<prefix>_rawsnp_counts.csv` | `--raw-snp-counts` | CSV table | per-pair diagnostic counts behind `_rawsnp.csv` |
-| `<prefix>_snp.fasta` | `--snp` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
 | `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
-| `<prefix>_shannon.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
-
-**`_siblings.csv`** — family size = number of distinct central bases observed at a family (1–4), not "sibling count" (0–3).
-
-| Column | Meaning |
-|---|---|
-| `genome` | genome label, or the literal `global` for the last row |
-| `1`, `2`, `3`, `4` | for a genome row: number of families of that size where the genome carries ≥ 1 member. For the `global` row: the actual deduplicated family-size histogram — **not** the sum of the rows above (a family shared by several genomes would otherwise be counted once per genome) |
-
-**`_rawsnp.csv`** — same matrix layout as `_dist.csv`; each cell is $\hat p = n_{\text{snp}}/(n_{\text{snp}}+n_{\text{shared}})$, 6 decimals, or `NA` when the pair has zero eligible loci (distinguishes "identical everywhere eligible" from "nothing eligible at all").
-
-**`_rawsnp_counts.csv`** — one row per unordered genome pair (not a matrix), the counts `_rawsnp.csv`'s ratio is computed from:
-
-| Column | Meaning |
-|---|---|
-| `genome_a`, `genome_b` | the pair |
-| `n_snp` | eligible loci where the two genomes' single forms differ |
-| `n_shared` | eligible loci where they agree (includes invariant families) |
-| `n_eligible` | `n_snp + n_shared` |
-| `ratio` | $\hat p$ = `n_snp / n_eligible`, or `NA` if `n_eligible = 0` |
-
-**`_snp.fasta`** — one record per non-excluded genome, one column per variable family (family size ≥ 2), header carries an `n_sites` annotation. Each site is IUPAC-coded from the genome's presence mask at that family: single observed form → plain base; several forms → matching IUPAC ambiguity code; no form → `-`.
-
-**`_family_overlap.csv`** — same matrix layout as `_dist.csv`; cell `[i][j]` = number of `_snp.fasta` columns where both genome `i` and `j` carry a call (neither is `-`). Diagonal `[i][i]` is kept (not skipped): it holds genome `i`'s own total variable-family count.
+| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
+| `<prefix>_alignment.fasta` | `--pseudo-alignment` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
 
 ### Sankoff calibration and exports
 
@@ -205,7 +265,7 @@ Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below):
 |---|---|---|---|
 | `<prefix>_sankoff_matrix.csv` | `--sankoff`/`--tnt`/`--phyg`/`--iqtree` | CSV matrix | calibrated 16×16 cost matrix |
 | `<prefix>_sankoff_params.yaml` | same flags | YAML | calibration report (raw tallies + derived probabilities) |
-| `<prefix>_sankoff.fasta` | same flags | FASTA | Sankoff-recoded pseudo-alignment |
+| `<prefix>_sankoff.fasta` | same flags | FASTA | Sankoff-recoded pseudo-alignment, header carries an `n_sites` annotation |
 | `<prefix>_sankoff.tnt` | `--tnt` | TNT script | ready-to-run parsimony search |
 | `<prefix>_sankoff.tcm` | `--phyg` | PhyG TCM | cost matrix in PhyG's own format |
 | `<prefix>_sankoff.pg` | `--phyg` | PhyG script | ready-to-run parsimony search |
@@ -213,33 +273,33 @@ Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below):
 | `<prefix>_iqtree.fasta` | `--iqtree` | FASTA | alignment recoded for that model |
 | `<prefix>_iqtree_states.csv` | `--iqtree` | CSV table | maps `_iqtree.model`/`_iqtree.fasta`'s compact state symbols back to `_sankoff_matrix.csv`'s alphabet |
 
-**`_sankoff_matrix.csv`** — header `state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N`: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), `0` standing for the empty/absent state (not `-`, to avoid colliding with external tools' own gap syntax). One row per source state, one value per destination state, cost $-\ln P(a,b)$, 4 decimals.
+**`_sankoff_matrix.csv`** — header `state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N`: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), `0` standing for the empty/absent state. One row per source state, one value per destination state, cost $-\ln P(a,b)$, 4 decimals.
 
-**`_sankoff_params.yaml`** — everything the calibration estimated, structured so it can be reloaded rather than re-parsed:
+**`_sankoff_params.yaml`**:
 
 | Key | Meaning |
 |---|---|
 | `ratio_ceiling` | the `--sankoff-ratio-ceiling` value used |
-| `cardinality_transitions` | 5×5 list of `{from, to, count, probability}`, family cardinality (0–4 observed forms) |
+| `cardinality_transitions` | 5×5 list of `{from, to, count, probability}`, family cardinality (0-4 observed forms) |
 | `composition_transitions` | 4×4 list of `{from, to, count, probability}`, base letters `A/C/G/T`, single-copy substitutions |
 
-**`_sankoff.fasta`** — same sites as `_snp.fasta`, recoded to match `_sankoff_matrix.csv`'s alphabet: absent state is `0` (or `?` under `--free-loss`). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.
+**`_sankoff.fasta`** — recoded to match `_sankoff_matrix.csv`'s alphabet: absent state is `0` (or `?` under `--free-loss`). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.
 
-**`_sankoff.tnt`** (`--tnt`) — self-contained TNT script: `xread` block (alignment recoded to TNT's fixed `0-9A-F` alphabet), an integer-scaled (`--sankoff-cost-scale`) and metric-closed `smatrix`, a default `hold 20; mult; export` search. Run with `printf 'proc <path>;\nquit;\n' | tnt`. Produces `<prefix>_sankoff.tre` (bare numeric leaf labels, order matching `_sankoff.fasta`) — feed both into [`name-tree`](name-tree.md) to recover taxon names.
+**`_sankoff.tnt`** (`--tnt`) — `xread` block (alignment recoded to TNT's fixed `0-9A-F` alphabet), an integer-scaled (`--sankoff-cost-scale`) and metric-closed `smatrix`, a default `hold 20; mult; export` search. Run with `printf 'proc <path>;\nquit;\n' | tnt`. Produces `<prefix>_sankoff.tre` (bare numeric leaf labels, order matching `_sankoff.fasta`).
 
-**`_sankoff.tcm`** (`--phyg`) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state `0`, since it's never actually triggered).
+**`_sankoff.tcm`** (`--phyg`) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state `0`).
 
-**`_sankoff.pg`** (`--phyg`) — script: `read(prefasta:..., tcm:...)` against `_sankoff.fasta`/`_sankoff.tcm`, a default 300s/4-instance `search`, `report(...)` writing `<prefix>_sankoff.tre` (bare numeric labels, as for `--tnt`). Run with `phyg` from the output directory (the script uses relative file names). Feed the tree plus `_sankoff.fasta` into [`name-tree`](name-tree.md) for taxon names.
+**`_sankoff.pg`** (`--phyg`) — script: `read(prefasta:..., tcm:...)` against `_sankoff.fasta`/`_sankoff.tcm`, a default 300s/4-instance `search`, `report(...)` writing `<prefix>_sankoff.tre`. Run with `phyg` from the output directory (the script uses relative file names).
 
 **`_iqtree.model`** (`--iqtree`) — lower-triangular exchangeability matrix $R(a,b) = e^{-\text{cost}(a,b)}$ (one row of increasing length per state, whitespace-separated, PAML order), followed by one line of empirical state frequencies. Only states actually occurring in the alignment are kept, compactly renumbered `0..k-1`.
 
-**`_iqtree.fasta`** (`--iqtree`) — alignment recoded to that same compact `0..k-1` alphabet (symbols `0-9A-F`). Under `--free-loss`, non-detection becomes `?` and columns left non-informative once missing calls are ignored are dropped first (required for `+ASC`); with `--iqtree-min-freq` also set (the default), any state rarer than that threshold is folded into the same `?` treatment, and non-informative columns are re-checked and dropped again after that. Run with:
+**`_iqtree.fasta`** (`--iqtree`) — alignment recoded to that same compact `0..k-1` alphabet (symbols `0-9A-F`). Under `--free-loss`, non-detection becomes `?` and columns left non-informative once missing calls are ignored are dropped first (required for `+ASC`); with `--iqtree-min-freq` also set (the default), any state rarer than that threshold is folded into the same `?` treatment, and non-informative columns are re-checked and dropped again. Run with:
 ```
 iqtree3 -s <prefix>_iqtree.fasta --seqtype MORPH -m <prefix>_iqtree.model+ASC --prefix <prefix>_iqtree -T AUTO
 ```
 
-**`_iqtree_states.csv`** (`--iqtree`) — one row per state actually kept in `_iqtree.model`/`_iqtree.fasta` (header `iqtree_symbol,canonical_symbol,frequency`): `iqtree_symbol` is the compact `0-9A-F` symbol as written in those two files, `canonical_symbol` is the matching `_sankoff_matrix.csv` state, `frequency` is that state's empirical frequency at full precision (`_iqtree.model`'s own frequency line is rounded to 6 decimals). Under `--free-loss`, absent (`0`/`?`) is never a kept state, so it never appears here — nor does any state `--iqtree-min-freq` folded away for being too rare. Use this file to identify which real state a given row/column of `_iqtree.model`'s matrix corresponds to — e.g. to check whether a state showing zero exchangeability with everything else is expected (a state combination that never co-occurs with anything else in this data) or worth investigating further.
+**`_iqtree_states.csv`** (`--iqtree`) — one row per state actually kept in `_iqtree.model`/`_iqtree.fasta` (header `iqtree_symbol,canonical_symbol,frequency`): `iqtree_symbol` is the compact `0-9A-F` symbol as written in those two files, `canonical_symbol` is the matching `_sankoff_matrix.csv` state, `frequency` is that state's empirical frequency at full precision. Under `--free-loss`, absent (`0`/`?`) is never a kept state, so it never appears here — nor does any state `--iqtree-min-freq` folded away for being too rare.
 
 ### Rare states and `--iqtree-min-freq`
 
-States that combine 3 or 4 central bases at once (IUPAC `V`/`H`/`K`.../`N`) are inherently rare — and, on real data, rare enough that they can make `iqtree3` itself numerically unstable ("Numerical underflow for lh-derivative" warnings, near-degenerate likelihood optimization). They're also more likely to be assembly/detection noise than genuine, widely-shared multi-way polymorphism, the same "sampling failure, not true signal" reasoning `--free-loss` already applies to non-detection. With `--free-loss` set, `--iqtree-min-freq` (default `0.001`, i.e. one in a thousand) extends that same missing-data treatment to any state below this frequency, not just absence. Check `_iqtree_states.csv` to see exactly which states survived and at what frequency; set `--iqtree-min-freq 0` to disable this and keep every state that occurs at all (the old behavior). Has no effect without `--free-loss` — there is no missing-data symbol to fold rare states into otherwise.
+States that combine 3 or 4 central bases at once (IUPAC `V`/`H`/`K`.../`N`) are inherently rare, and can make `iqtree3` itself numerically unstable ("Numerical underflow for lh-derivative" warnings). With `--free-loss` set, `--iqtree-min-freq` (default `0.001`, one in a thousand) extends the missing-data treatment to any state below this frequency, not just absence. Check `_iqtree_states.csv` to see exactly which states survived and at what frequency; set `--iqtree-min-freq 0` to keep every state that occurs at all. Has no effect without `--free-loss`.

UserDocMD/usage/reindex.md

@@ -1,25 +0,0 @@
-# reindex
-
-Convert an existing index's evidence representation in place, between exact and approximate.
-
-```bash
-obikmer reindex INDEX [OPTIONS]
-```
-
-## Arguments
-
-| Argument | Description |
-|---|---|
-| `INDEX` | Index directory to convert (modified in place) |
-
-## Options
-
-| Option | Default | Description |
-|---|---|---|
-| `--approx` | off | Convert to approximate evidence (default direction is approximate → exact); requires `-z`/`--evidence-bits`/`--fp` |
-| `-z, --findere-z` | none | Findere z parameter (≥ 1) |
-| `--evidence-bits` | none | Fingerprint bits per slot (b) |
-| `--fp` | none | Target false-positive rate per z-window |
-| `--block-size` | `1` | Block size for the exact on-disk index (ignored when converting to approximate) |
-
-See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.