Merge pull request 'Push twnyxtspltux' (#76) from push-twnyxtspltux into main

Reviewed-on: #76

.gitignore

@@ -1,32 +1,36 @@
 .venv/
+.DS_Store
+.claude/
+.kilo/
+.serena/
+.zed/
+.ast-cache/
+CLAUDE.md
+memory/
+sandbox/
 src/target
 data-stress
 *.fasta
 *.fasta.gz
+*.fastq
+*.fastq.gz
+*.vcf
 *.zst
 *.zst.meta
 *.pb
-./**/*.json
+**/*.json
 *.bin
 *.log
 *.csv
+*.meta
+*.pfiv
+*.done
+*.efh
+*.efl
+*.bin.idx
+*.prsb
 Betula_exilis--IGA-24-33
-benchmark/genomes
-benchmark/genomes_orig
-benchmark/simulated_data
-benchmark/specimen_index_presence
-benchmark/specimen_index_count
-benchmark/global_index_presence
-benchmark/global_index_presence_orig
-benchmark/global_index_presence_sav
-benchmark/all_specific
-benchmark/global_index_count
-benchmark/stats
-benchmark/reference_index
-benchmark/reference_dist
-benchmark/obikmer_dist
-benchmark/specific_index_count
-benchmark/specific_index_presence
+benchmark/run/
 TNT
 phyg
 biblio

.kilo/plans/1778319532254-eager-panda.md

@@ -1,87 +0,0 @@
-# Plan d'amélioration technique - obiskio
-
-## 1. Contexte et objectifs
-- **Objectif** : Renforcer la robustesse, la maintenabilité et les performances de la crate `obiskio`.
-- **Priorités** : 
-  1. Gestion des erreurs
-  2. Optimisation de la mémoire du pool
-  3. Robustesse concurrente
-  4. Couverture de tests
-  5. Documentation
-
----
-
-## 2. Axes d'amélioration détaillés
-
-### 2.1 Gestion des erreurs
-- **Problème** : `SKError` ne couvre pas tous les cas (format invalide, taille maximale, CRC)
-- **Actions** :
-  - Ajouter variante `ParseError(String)` dans `src/error.rs`
-  - Valider les tailles de SuperKmer avant parsing
-  - Remplacer `expect()` par `unwrap_or_else` avec messages explicites
-  - Documenter chaque variante d’erreur dans le README
-
-### 2.2 Optimisation du pool de fichiers
-- **Problème** : `SKFilePool` utilise un `Vec<WriteEntry>` non contraint et n’effectue pas de nettoyage en cas d’erreur
-- **Actions** :
-  - Implémenter un `LimitedVec` avec limite stricte à `MAX_POOL_SIZE`
-  - Créer `clear_memory()` qui supprime les entrées orphelines
-  - Ajouter `evict_lru_threshold()` pour éviction proactive
-  - Introduire un `RwLock` pour les opérations de lecture massives
-
-### 2.3 Robustesse concurrente
-- **Problème** : Risque de deadlocks dans `SKFileWriter::write_batch()` et `SKFileReader::reopen_and_seek()`
-- **Actions** :
-  - Remplacer `Mutex` par `RwLock` pour les accès en lecture
-  - Ajouter un compteur de blocage et logs de timeout
-  - Utiliser `std::thread::park_timeout` pour débloquer
-  - Insérer `debug_assert!` sur les états invariants
-
-### 2.4 Couverture de tests
-- **Problème** : Absence de benchmarks, de tests de migration, de résilience de fichiers corrompus
-- **Actions** :
-  - Benchmarks I/O sur 10k+ SuperKmer avec `criterion`
-  - Tests de migration de version de fichier `.meta` → `.v2.meta`
-  - Tests de corruption volontaire (truncature, inversion de bits)
-  - Tests de stress sur pool saturation (100 threads)
-
-### 2.5 Documentation & exemples
-- **Actions** :
-  - Ajouter des examples dans chaque module (`# Examples`)
-  - Documenter la logique LRU avec diagrammes Mermaid
-  - Créer un guide « How to recover from eviction »
-  - Mettre à jour le `README.md` avec tableau des variantes d’erreur
-
----
-
-## 3. Plan d'exécution (Roadmap)
-
-| Sprint | Durée | Livrables clés |
-|--------|-------|----------------|
-| **S1** | 2 jours | Refactorisation `SKError`, ajout de tests unitaires |
-| **S2** | 3 jours | Implémentation `clear_memory()` + `LimitedVec` |
-| **S3** | 2 jours | Passage à `RwLock`, ajout de compteurs de blocage |
-| **S4** | 2 jours | Benchmarks + tests de migration |
-| **S5** | 1 jour | Documentation finale & mise à jour du README |
-
----
-
-## 4. Dépendances externes
-- Mettre à jour `niffler` vers la version 2.0 (performance compression)
-- Évaluer `bincode` vs `serde_json` pour les métas (I/O)
-- Ajouter dépendance `criterion` (dev‑dependencies)
-
----
-
-## 5. KPI de suivi
-- **Couverture de tests** : ≥85 % des chemins critiques
-- **Latence moyenne d’écriture** : ↓15 % après optimisation du pool
-- **Taux d’erreurs résolues** : 100 % des nouvelles variantes couvertes
-- **Temps de build CI** : ≤5 min pour l’ensemble des benchmarks
-
----
-
-## 6. Validation finale
-- Revue de code avec `cargo clippy -- -D warnings`
-- Analyse de toxicité avec `cargo deny open-source-licenses`
-- Vérification de la conformité aux standards de naming du projet

.serena/.gitignore

@@ -1,2 +0,0 @@
-/cache
-/project.local.yml

.serena/project.yml

@@ -1,133 +0,0 @@
-# the name by which the project can be referenced within Serena
-project_name: "obikmer"
-
-
-# list of languages for which language servers are started; choose from:
-#   al                  angular             ansible             bash                clojure
-#   cpp                 cpp_ccls            crystal             csharp              csharp_omnisharp
-#   dart                elixir              elm                 erlang              fortran
-#   fsharp              go                  groovy              haskell             haxe
-#   hlsl                html                java                json                julia
-#   kotlin              lean4               lua                 luau                markdown
-#   matlab              msl                 nix                 ocaml               pascal
-#   perl                php                 php_phpactor        powershell          python
-#   python_jedi         python_ty           r                   rego                ruby
-#   ruby_solargraph     rust                scala               scss                solidity
-#   svelte              swift               systemverilog       terraform           toml
-#   typescript          typescript_vts      vue                 yaml                zig
-#   (This list may be outdated. For the current list, see values of Language enum here:
-#   https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
-#   For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
-# Note:
-#   - For C, use cpp
-#   - For JavaScript, use typescript
-#   - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
-#   - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
-#   - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
-#   - For Free Pascal/Lazarus, use pascal
-# Special requirements:
-#   Some languages require additional setup/installations.
-#   See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
-# When using multiple languages, the first language server that supports a given file will be used for that file.
-# The first language is the default language and the respective language server will be used as a fallback.
-# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
-languages:
-- rust
-
-# the encoding used by text files in the project
-# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
-encoding: "utf-8"
-
-# line ending convention to use when writing source files.
-# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
-# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
-line_ending:
-
-# The language backend to use for this project.
-# If not set, the global setting from serena_config.yml is used.
-# Valid values: LSP, JetBrains
-# Note: the backend is fixed at startup. If a project with a different backend
-# is activated post-init, an error will be returned.
-language_backend:
-
-# whether to use project's .gitignore files to ignore files
-ignore_all_files_in_gitignore: true
-
-# advanced configuration option allowing to configure language server-specific options.
-# Maps the language key to the options.
-# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
-# No documentation on options means no options are available.
-ls_specific_settings: {}
-
-# list of additional workspace folder paths for cross-package reference support (e.g. in monorepos).
-# Paths can be absolute or relative to the project root.
-# Each folder is registered as an LSP workspace folder, enabling language servers to discover
-# symbols and references across package boundaries.
-# Currently supported for: TypeScript.
-# Example:
-#   additional_workspace_folders:
-#     - ../sibling-package
-#     - ../shared-lib
-additional_workspace_folders: []
-
-# list of additional paths to ignore in this project.
-# Same syntax as gitignore, so you can use * and **.
-# Note: global ignored_paths from serena_config.yml are also applied additively.
-ignored_paths: []
-
-# whether the project is in read-only mode
-# If set to true, all editing tools will be disabled and attempts to use them will result in an error
-# Added on 2025-04-18
-read_only: false
-
-# list of tool names to exclude.
-# This extends the existing exclusions (e.g. from the global configuration)
-# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
-excluded_tools: []
-
-# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
-# This extends the existing inclusions (e.g. from the global configuration).
-# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
-included_optional_tools: []
-
-# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
-# This cannot be combined with non-empty excluded_tools or included_optional_tools.
-# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
-fixed_tools: []
-
-# list of mode names that are to be activated by default, overriding the setting in the global configuration.
-# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
-# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
-# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
-# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
-# for this project.
-# This setting can, in turn, be overridden by CLI parameters (--mode).
-# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
-default_modes:
-
-# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
-# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
-# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
-added_modes:
-
-# initial prompt for the project. It will always be given to the LLM upon activating the project
-# (contrary to the memories, which are loaded on demand).
-initial_prompt: ""
-
-# time budget (seconds) per tool call for the retrieval of additional symbol information
-# such as docstrings or parameter information.
-# This overrides the corresponding setting in the global configuration; see the documentation there.
-# If null or missing, use the setting from the global configuration.
-symbol_info_budget:
-
-# list of regex patterns which, when matched, mark a memory entry as read‑only.
-# Extends the list from the global configuration, merging the two lists.
-read_only_memory_patterns: []
-
-# list of regex patterns for memories to completely ignore.
-# Matching memories will not appear in list_memories or activate_project output
-# and cannot be accessed via read_memory or write_memory.
-# To access ignored memory files, use the read_file tool on the raw file path.
-# Extends the list from the global configuration, merging the two lists.
-# Example: ["_archive/.*", "_episodes/.*"]
-ignored_memory_patterns: []

.zed/tasks.json

@@ -1,59 +0,0 @@
-// Project tasks configuration. See https://zed.dev/docs/tasks for documentation.
-//
-// Example:
-[
-  {
-    "label": "Example task",
-    "command": "for i in {1..5}; do echo \"Hello $i/5\"; sleep 1; done",
-    //"args": [],
-    // Env overrides for the command, will be appended to the terminal's environment from the settings.
-    "env": { "foo": "bar" },
-    // Current working directory to spawn the command into, defaults to current project root.
-    //"cwd": "/path/to/working/directory",
-    // Whether to use a new terminal tab or reuse the existing one to spawn the process, defaults to `false`.
-    "use_new_terminal": false,
-    // Whether to allow multiple instances of the same task to be run, or rather wait for the existing ones to finish, defaults to `false`.
-    "allow_concurrent_runs": false,
-    // What to do with the terminal pane and tab, after the command was started:
-    // * `always` — always show the task's pane, and focus the corresponding tab in it (default)
-    // * `no_focus` — always show the task's pane, add the task's tab in it, but don't focus it
-    // * `never` — do not alter focus, but still add/reuse the task's tab in its pane
-    "reveal": "always",
-    // Where to place the task's terminal item after starting the task:
-    // * `dock` — in the terminal dock, "regular" terminal items' place (default)
-    // * `center` — in the central pane group, "main" editor area
-    "reveal_target": "dock",
-    // What to do with the terminal pane and tab, after the command had finished:
-    // * `never` — Do nothing when the command finishes (default)
-    // * `always` — always hide the terminal tab, hide the pane also if it was the last tab in it
-    // * `on_success` — hide the terminal tab on task success only, otherwise behaves similar to `always`
-    "hide": "never",
-    // Which shell to use when running a task inside the terminal.
-    // May take 3 values:
-    // 1. (default) Use the system's default terminal configuration in /etc/passwd
-    //      "shell": "system"
-    // 2. A program:
-    //      "shell": {
-    //        "program": "sh"
-    //      }
-    // 3. A program with arguments:
-    //     "shell": {
-    //         "with_arguments": {
-    //           "program": "/bin/bash",
-    //           "args": ["--login"]
-    //         }
-    //     }
-    "shell": "system",
-    // Whether to show the task line in the output of the spawned task, defaults to `true`.
-    "show_summary": true,
-    // Whether to show the command line in the output of the spawned task, defaults to `true`.
-    "show_command": true,
-    // Which edited buffers to save before running the task:
-    // * `all` — save all edited buffers
-    // * `current` — save currently active buffer only
-    // * `none` — don't save any buffers
-    "save": "none",
-    // Represents the tags for inline runnable indicators, or spawning multiple tasks at once.
-    // "tags": []
-  },
-]

CLAUDE.md

@@ -1,103 +0,0 @@
-**PROMPT**
-
-Tu es ma base de connaissance et mon bloc-notes intelligent sur le projet **obikmer**. Tu ne proposes pas, tu ne codes pas spontanément — tu réponds à mes questions et tu structures mes idées au fur et à mesure que je les exprime.
-
-**Règle absolue : une question appelle une réponse, pas une action.**
-Ne modifier aucun fichier à moins d'une demande explicite de modification. En particulier : observer un bug ou une incohérence dans le code montré ne constitue pas un mandat pour le corriger. Le code montré peut refléter une intention en cours — modifier sans mandat risque d'introduire un vrai bug là où tu croyais corriger.
-
-**Règle absolue : ne jamais substituer une dépendance ou une bibliothèque sans validation explicite.**
-Si une dépendance demandée pose problème (erreur de compilation, bug, API manquante), exposer le problème et proposer des alternatives — ne jamais switcher silencieusement vers une autre bibliothèque. Le choix des dépendances est une décision d'architecture qui appartient au développeur.
-
-**Règle absolue : le code existant est une hypothèse, pas une vérité.**
-Quand une nouvelle construction (type, itérateur, abstraction) rend du code historique injustifié, le signaler immédiatement et proposer de le supprimer — ne pas conserver les deux en parallèle par inertie. Le développeur demande explicitement de remettre en cause le code base : ne pas attendre qu'il insiste.
-
-Tu maintiens en **anglais**, dense et sans remplissage, les documents suivants :
-- `DevDocMD/index.md` — document de discussion de base, enrichi progressivement au fil de nos échanges ; il reflète l'état courant de la réflexion sur le projet
-- les autres fichiers Markdown dans `DevDocMD/` selon leur thème respectif
-
-Les snippets de code y sont courts et illustrent uniquement des principes architecturaux. Nos échanges se font en **français**.
-
----
-
-## Contexte du projet
-
-`obikmer` est un outil Rust de manipulation, comptage, indexation et opérations ensemblistes sur des séquences ADN représentées comme des ensembles de kmers.
-
-**Contraintes fondamentales**
-- Efficacité maximale en calcul, mémoire et disque
-- Données métagénomiques : plusieurs dizaines de Gbases, milliards de kmers
-- k impair, k ∈ [11, 31], fixé à l'exécution
-- Formats d'entrée : FASTA, FASTQ, gzip, streaming stdin
-
-**Opérations prioritaires**
-- Comptage de kmers (fréquences)
-- Recherche / requête rapide
-- Opérations ensemblistes (union, intersection, diff)
-
-**Ce qui a déjà été discuté**
-- Encodage 2 bits/base → kmer tient dans un `u64`
-- Forme canonique : `min(kmer, revcomp)` pour réduire l'espace de moitié
-
----
-
-## Infrastructure de documentation
-
-La documentation est gérée via **MkDocs + thème Material**, avec publication sur **GitHub Pages**. Deux arbres de documentation indépendants, deux configurations à la racine du dépôt (voir aussi `UserDocMD/` ci-dessous pour la doc utilisateur) :
-
-**Structure des répertoires**
-
-```
-DevDocMD/       ← sources Markdown, doc développeur (discussion, historique, rationale)
-UserDocMD/      ← sources Markdown, doc utilisateur (état factuel courant, sans code Rust)
-mkdocs.yml      ← config doc développeur : docs_dir DevDocMD, site_dir DevDoc
-mkdocs-user.yml ← config doc utilisateur : docs_dir UserDocMD, site_dir doc
-DevDoc/         ← site HTML généré (doc développeur)
-doc/            ← site HTML généré (doc utilisateur — publié par GitHub Pages, répertoire par défaut)
-.venv/          ← environnement Python (ignoré par git)
-```
-
-**Commandes Makefile**
-
-| Commande | Effet |
-|---|---|
-| `make doc` | Construit la doc développeur dans `DevDoc/` |
-| `make doc-serve` | Serveur local (doc développeur), rechargement automatique |
-| `make clean-doc` | Supprime `DevDoc/` |
-| `make doc-user` | Construit la doc utilisateur dans `doc/` |
-| `make doc-user-serve` | Serveur local (doc utilisateur), rechargement automatique |
-| `make clean-doc-user` | Supprime `doc/` |
-| `make clean` | Supprime `DevDoc/`, `doc/` et `.venv/` |
-
-Le `.venv/` est dans `.gitignore`. `DevDoc/` et `doc/` (sorties HTML) sont versionnés — `doc/` spécifiquement parce que c'est le répertoire par défaut que GitHub Pages sert.
-
-Lors de l'ajout de nouveaux fichiers Markdown dans `DevDocMD/`, mettre à jour la section `nav:` de `mkdocs.yml` ; dans `UserDocMD/`, mettre à jour `mkdocs-user.yml`.
-
----
-
-Je continue à poser mes questions et à guider la discussion.
-
----
-
-## MCP Tools
-
-**Règle absolue : avant tout travail de code, appeler `mcp__serena__initial_instructions` pour charger les instructions Serena.**
-
-### Hiérarchie des outils pour ce projet Rust
-
-**Navigation et édition de code → serena en priorité**
-- Trouver un symbole, une déclaration, les implémentations d'un trait : `mcp__serena__find_symbol`, `mcp__serena__find_declaration`, `mcp__serena__find_implementations`
-- Trouver les usages d'un symbole : `mcp__serena__find_referencing_symbols`
-- Diagnostics LSP (erreurs de compilation) : `mcp__serena__get_diagnostics_for_file`
-- Vue d'ensemble d'un fichier : `mcp__serena__get_symbols_overview`
-- Modifier le corps d'une fonction/impl : `mcp__serena__replace_symbol_body`
-- Ne pas utiliser `cclsp` quand serena couvre le besoin
-
-**Analyse architecturale → jcodemunch**
-- Hotspots, couplage, dead code, dépendances entre modules
-- Utiliser avant de refactorer une zone critique
-
-**Raisonnement complexe → sequential-thinking**
-- Décisions d'architecture, choix d'algorithme, trade-offs non triviaux
-
-**Documentation de crates → context7**
-- Toujours consulter avant d'utiliser une API de bibliothèque externe

DevDoc/404.html

@@ -722,14 +722,14 @@
   
   
     <li class="md-nav__item">
-      <a href="/implementation/obilayeredmap/" class="md-nav__link">
+      <a href="/implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -939,6 +939,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="/implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="/implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/index_architecture.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,10 +1270,10 @@
 <h1 id="coverage-architectureindex_architecturemd">Coverage: architecture/index_architecture.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix</li>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum</li>
-<li><code>obilayeredmap/src/map.rs</code> — LayeredMap<D></li>
-<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta, PartitionMeta</li>
+<li><code>obikindex/src/layer/typed_layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix</li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum</li>
+<li><code>obikindex/src/layer/map.rs</code> — LayeredMap<D></li>
+<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta, PartitionMeta</li>
 <li><code>obikindex/src/meta.rs</code> — IndexConfig (kmer_size, n_bits, with_counts, evidence, block_bits), IndexMeta</li>
 <li><code>obikindex/src/index.rs</code> — KmerIndex, build_layers</li>
 <li><code>obicompactvec/src/</code> — PersistentCompactIntMatrix, PersistentBitMatrix (DataStore implementations)</li>

DevDoc/architecture/index_architecture/index.html

@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/numa_partition_runner/index.html

@@ -729,14 +729,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -946,6 +946,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1311,6 +1367,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1497,6 +1564,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1780,11 +1858,27 @@ dormant workers have been woken and grows every node by the same amount per
 step, capped by that node's remaining dormant workers and by the run's total
 budget (<code>n_total</code>) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.</p>
+<h2 id="panic-propagation-fixed-2026-08-28">Panic propagation (fixed, 2026-08-28)</h2>
+<p>A worker whose <code>f(i)</code> panicked used to never send its <code>WorkerEvent::Completed</code>
+— the controller's <code>while completed &lt; n_total</code> loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept <code>event_rx</code> open. Discovered via <code>obikselect</code>
+panicking on an unimplemented <code>Sparse</code> matrix case (see <code>implementation/select.md</code>):
+the process hung indefinitely instead of erroring out.</p>
+<p>Fixed: each worker wraps <code>f(i)</code> in <code>std::panic::catch_unwind</code> and sends a new
+<code>WorkerEvent::Panicked(i, payload)</code> instead of silently dropping the
+partition. The controller counts it toward <code>completed</code> (unblocking the loop)
+and keeps the first payload seen; once <code>run</code> returns, <code>panic::resume_unwind</code>
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain <code>Err</code> (a panic means a bug, not a normal typed failure).</p>
 <h2 id="open-questions">Open questions</h2>
 <ul>
 <li>
 <p><strong>Error handling</strong>: <code>run</code> currently returns the first error; remaining errors
-  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics.</p>
+  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain <code>Err</code> results.)</p>
 </li>
 <li>
 <p><strong><code>INITIAL_DIVISOR</code> / <code>GROWTH_DIVISOR</code> tuning</strong>: currently <code>4</code> and <code>8</code>

DevDoc/architecture/numa_worker_pools/index.html

@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1504,7 +1560,7 @@
 <td>Replace <code>available_parallelism()</code> with per-NUMA core count for spawn criterion</td>
 </tr>
 <tr>
-<td><code>obikpartitionner/src/merge_layer.rs</code></td>
+<td><code>obikindex/src/partition/merge_layer.rs</code></td>
 <td>No change — <code>merge_partition</code> already works inside any Rayon context</td>
 </tr>
 <tr>
@@ -1512,7 +1568,7 @@
 <td>No change — <code>par_iter</code> and <code>current_num_threads</code> are pool-context-aware</td>
 </tr>
 <tr>
-<td><code>obikpartitionner/src/partition.rs</code></td>
+<td><code>obikindex/src/partition/partition.rs</code></td>
 <td>No change — same reason</td>
 </tr>
 </tbody>

DevDoc/architecture/query.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1215,7 +1271,7 @@
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
 <li><code>obikmer/src/cmd/query.rs</code> — commande query, format de sortie</li>
-<li><code>obikpartitionner/src/query_layer.rs</code> — routage de la requête à travers les partitions</li>
+<li><code>obikindex/src/partition/query_layer.rs</code> — routage de la requête à travers les partitions</li>
 <li><code>obiread/src/lib.rs</code> — lecture des séquences d'entrée pour la requête</li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/architecture/query/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1499,7 +1555,7 @@
 </ul>
 <hr />
 <h2 id="algorithm">Algorithm</h2>
-<p>The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside <code>process_chunk</code> (<code>query.rs</code>); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (<code>obikpartitionner::query_partition_with</code>), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.</p>
+<p>The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside <code>process_chunk</code> (<code>query.rs</code>); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (<code>obikindex::partition::query_partition_with</code>), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.</p>
 <div class="highlight"><pre><span></span><code>for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
     build QueryBatch (QueryBatch::from_records):
         decompose all sequences into superkmers (SuperKmerIter) — construction only,
@@ -1574,9 +1630,9 @@
 <p>The <code>-z</code> CLI option overrides the index metadata value. A higher z increases stringency (lower FP, some true positives may be discarded at sequence ends); a lower z increases sensitivity.</p>
 <hr />
 <h2 id="layer-lookup-mphflayerfind">Layer lookup: <code>MphfLayer::find</code></h2>
-<p><code>MphfLayer::open(dir, mode: &amp;IndexMode)</code> receives the mode from <code>PartitionMeta</code> — no per-layer file is read. The caller (<code>QueryLayer</code>) never chooses the dispatch path: it is fixed at open time by <code>LayerEvidence</code>. See <a href="../../implementation/obilayeredmap/">obilayeredmap</a> for the full <code>find</code> / <code>find_strict</code> API.</p>
+<p><code>MphfLayer::open(dir, mode: &amp;IndexMode)</code> receives the mode from <code>PartitionMeta</code> — no per-layer file is read. The caller (<code>QueryLayer</code>) never chooses the dispatch path: it is fixed at open time by <code>LayerEvidence</code>. See <a href="../../implementation/layer_tier/">obikindex::layer</a> for the full <code>find</code> / <code>find_strict</code> API.</p>
 <h3 id="querylayer-variant-selection"><code>QueryLayer</code> variant selection</h3>
-<p><code>QueryLayer::open</code> (<code>obikpartitionner/src/query_layer.rs:28-45</code>) only ever returns two variants — <code>Presence</code> or <code>Count</code>, checked in this order:</p>
+<p><code>QueryLayer::open</code> (<code>obikindex/src/partition/query_layer.rs:28-45</code>) only ever returns two variants — <code>Presence</code> or <code>Count</code>, checked in this order:</p>
 <table>
 <thead>
 <tr>
@@ -1746,7 +1802,7 @@ Fix direction: restructure <code>query</code>'s pipe with an initial <code>Flat<
 <p><strong>3. Chunk-size memory formula ignores <code>n_genomes</code>.</strong>
 <code>chunk_bytes = available_memory_bytes() / (n_workers * 16)</code> (<code>query.rs:407-414</code>) assumes a fixed ~8–16× overhead per raw input byte. But <code>KmerResults::new</code> (<code>query.rs:165-179</code>) allocates <code>data: Vec&lt;u32&gt;</code> sized <code>total_kmers_in_chunk × n_genomes</code> — dense, <strong>for every k-mer position in the chunk, hit or not</strong> — plus <code>win_min</code> and (with <code>--detail</code>) <code>cov</code>, same scaling. Real per-chunk memory is <code>O(n_genomes)</code>, not constant; the formula doesn't know <code>n_genomes</code> at all. This is the direct cause of the OOM kill on indexes with many reference genomes.</p>
 <p><strong>4. MPHF lookup and matrix-row fetch are fused, not staged.</strong>
-<code>QueryLayer::find_into</code> (<code>obikpartitionner/src/query_layer.rs:48-67</code>) does the MPHF <code>find</code> <em>and</em> the <code>fill_row</code> matrix read in one call per k-mer, inside a single-threaded loop (<code>query_partition_with</code>). There is no separation between "is this k-mer indexed" (cheap, <code>O(1)</code>, independent of <code>n_genomes</code>) and "what are its per-genome values" (the expensive, <code>n_genomes</code>-scaling part).</p>
+<code>QueryLayer::find_into</code> (<code>obikindex/src/partition/query_layer.rs:48-67</code>) does the MPHF <code>find</code> <em>and</em> the <code>fill_row</code> matrix read in one call per k-mer, inside a single-threaded loop (<code>query_partition_with</code>). There is no separation between "is this k-mer indexed" (cheap, <code>O(1)</code>, independent of <code>n_genomes</code>) and "what are its per-genome values" (the expensive, <code>n_genomes</code>-scaling part).</p>
 <p><strong>5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.</strong>
 <code>QueryBatch::from_records</code> currently dereplicates at the <em>superkmer</em> level (<code>HashMap&lt;RoutableSuperKmer, Vec&lt;SKDesc&gt;&gt;</code>, <code>query.rs:112</code>). This misses redundancy between k-mers shared by <em>different</em> superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer <em>construction</em> (<code>SuperKmerIter</code>) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on <code>CanonicalKmer</code>, in the same pass: <code>HashMap&lt;CanonicalKmer, Vec&lt;(seq_idx, pos)&gt;&gt;</code>. This also means the MPHF <code>find</code> itself runs once per <strong>distinct</strong> k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.</p>
 <p><strong>6. Stage 1 output: bucket confirmed hits by layer, keyed by MPHF slot.</strong>
@@ -1778,12 +1834,12 @@ Stage 2's output should be <code>HashMap&lt;genome_idx, Vec&lt;(seq_idx, positio
 <p>None of this is implemented yet — parked here as a coherent roadmap while the design is discussed further. Suggested dependency order: (1) I/O parallelism → (3) genome-aware chunk sizing → (4)–(9) staged/k-mer-deduped/NUMA-aware-partition-and-column-major/sparse query engine (larger refactor, biggest structural payoff — reuses <code>PartitionRunner</code> rather than inventing a new parallelism mechanism) → (2) parallel gzip (separate, orthogonal, tracked in chunkreader.md) → secondary diagnostics patterns.</p>
 <hr />
 <h2 id="implementation-plan">Implementation plan</h2>
-<p>Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current <code>query</code> output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning <code>obikmer</code>, <code>obikpartitionner</code>, and <code>obicompactvec</code> — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense <code>KmerResults</code>) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if <code>rapidgzip-rs</code> is validated (see <a href="../../implementation/chunkreader/#future-work--parallel-gzip-decompression-in-xopen">chunkreader.md</a>).</p>
+<p>Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current <code>query</code> output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning <code>obikmer</code>, <code>obikindex::partition</code>, and <code>obicompactvec</code> — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense <code>KmerResults</code>) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if <code>rapidgzip-rs</code> is validated (see <a href="../../implementation/chunkreader/#future-work--parallel-gzip-decompression-in-xopen">chunkreader.md</a>).</p>
 <p>Instrumentation is deliberately sequenced <em>before</em> the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.</p>
 <p>Performance measurement on the reference 192-core/8-NUMA machine is done by the project owner, not from this development environment (macOS, 16 cores — <code>PartitionRunner</code>'s NUMA pinning is Linux-only, so even phase 4's mechanism can't be functionally exercised for its actual purpose here). Each phase below is therefore written to be <em>self-measuring</em>: the debug-level logging it adds must be enough, on its own, to judge whether that phase's algorithmic choice paid off from a cluster run's logs, without needing to attach a profiler.</p>
 <h3 id="conventions-applied-to-every-phase-below">Conventions applied to every phase below</h3>
 <p><strong>Debug logging.</strong> Every phase that changes an algorithmic choice (not phase 0, which <em>is</em> the logging) adds <code>tracing::debug!</code>/<code>trace!</code> at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF <code>find</code> calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured <code>debug!</code> per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.</p>
-<p><strong>Unit tests.</strong> This project's convention (<code>obiread</code>, <code>obikseq</code>, <code>obidebruinj</code>, <code>obicompactvec</code>, <code>obilayeredmap</code>, <code>obiskio</code>, <code>obifastwrite</code>) is <code>#[cfg(test)] #[path = "tests/&lt;name&gt;.rs"] mod tests;</code> at the bottom of the source file, with the actual test code in a sibling <code>src/tests/&lt;name&gt;.rs</code>. Neither <code>obikmer</code> nor <code>obikpartitionner</code> (the two crates phases 3 and 5 touch most) currently have a <code>src/tests/</code> directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.</p>
+<p><strong>Unit tests.</strong> This project's convention (<code>obiread</code>, <code>obikseq</code>, <code>obidebruinj</code>, <code>obicompactvec</code>, <code>obikindex::layer</code>, <code>obiskio</code>, <code>obifastwrite</code>) is <code>#[cfg(test)] #[path = "tests/&lt;name&gt;.rs"] mod tests;</code> at the bottom of the source file, with the actual test code in a sibling <code>src/tests/&lt;name&gt;.rs</code>. Neither <code>obikmer</code> nor <code>obikindex::partition</code> (the two crates phases 3 and 5 touch most) currently have a <code>src/tests/</code> directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.</p>
 <p><strong>Workflow (<code>jj</code>).</strong> Work happens in a fresh <code>jj</code> commit, easy to abandon. <code>jj new</code> between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then <code>query_layer.rs</code> split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.</p>
 <h3 id="phase-0-instrumentation-prerequisite-for-measuring-every-later-phase">Phase 0 — Instrumentation (prerequisite for measuring every later phase)</h3>
 <p><strong>Goal</strong>: make core utilization, throughput, and per-stage timing visible on a real run, so phases 1–5 can be justified with numbers instead of assumption.</p>
@@ -1826,26 +1882,26 @@ Stage 2's output should be <code>HashMap&lt;genome_idx, Vec&lt;(seq_idx, positio
 <li><code>obikmer/src/cmd/query.rs</code>:</li>
 <li>Replace <code>QueryBatch::from_records</code>'s dedup map (<code>HashMap&lt;RoutableSuperKmer, Vec&lt;SKDesc&gt;&gt;</code>, current <code>query.rs:112</code>) with a per-partition <code>HashMap&lt;CanonicalKmer, Vec&lt;(seq_idx: u32, pos: u32)&gt;&gt;</code>, built in the same <code>SuperKmerIter</code> pass: superkmer construction and partition routing (<code>part_idx</code> from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each <code>CanonicalKmer</code> within a superkmer is inserted individually instead of the whole superkmer being the dedup key.</li>
 <li><strong>Verified</strong>: <code>CanonicalKmer</code> (<code>obikseq/src/kmer.rs:390</code>, <code>pub type CanonicalKmer = CanonicalKmerOf&lt;KLen&gt;</code>) — the underlying <code>CanonicalKmerOf&lt;L&gt;</code> derives <code>Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash</code> (<code>kmer.rs:269</code>). Usable as a <code>HashMap</code>/<code>HashSet</code> key as-is, no change needed.</li>
-<li><code>obikpartitionner/src/query_layer.rs</code>:</li>
+<li><code>obikindex/src/partition/query_layer.rs</code>:</li>
 <li>Split <code>QueryLayer::find_into</code> (<code>query_layer.rs:48-67</code>) into two methods: <code>find_slot(&amp;self, kmer: CanonicalKmer) -&gt; Option&lt;usize&gt;</code> (MPHF only, no matrix touch) and keep <code>fill_row</code> as-is for phase 4 to call later.</li>
 <li>Replace <code>query_partition_with</code>'s inner loop (<code>query_layer.rs:103-113</code>) with a version that, for each unique <code>CanonicalKmer</code>, calls <code>find_slot</code> across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records <code>(layer_idx, slot)</code>.</li>
 <li>New return shape for the partition-level query, replacing today's <code>on_hit(sk_idx, kmer_idx, row)</code> callback: <code>HashMap&lt;layer_idx, HashMap&lt;slot, Vec&lt;(seq_idx, pos)&gt;&gt;&gt;</code> (roadmap point 6) — built directly from the k-mer dedup map's <code>Vec&lt;(seq_idx,pos)&gt;</code> values, keyed by the resolved slot instead of the k-mer.</li>
 <li><strong>This phase alone has no throughput benefit yet</strong> (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.</li>
 <li>Log, per chunk: total k-mer occurrences vs. unique <code>CanonicalKmer</code> count (the dedup ratio — the entire justification for this phase) and the resulting MPHF <code>find</code> call count. If the dedup ratio is close to <code>1.0</code> on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.</li>
-<li><strong>Unit tests</strong>: create <code>obikmer/src/cmd/tests/query.rs</code> (new <code>src/tests/</code> dir for this crate, following the project's <code>#[cfg(test)] #[path = "tests/query.rs"] mod tests;</code> convention) and <code>obikpartitionner/src/tests/query_layer.rs</code> (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the <code>find_slot</code>/bucket-by-layer-and-slot construction against a small hand-built <code>QueryLayer</code> fixture, asserting the <code>(layer_idx, slot, seq_idx, pos)</code> tuples match what the old per-occurrence loop would have produced.</li>
+<li><strong>Unit tests</strong>: create <code>obikmer/src/cmd/tests/query.rs</code> (new <code>src/tests/</code> dir for this crate, following the project's <code>#[cfg(test)] #[path = "tests/query.rs"] mod tests;</code> convention) and <code>obikindex/src/partition/tests/query_layer.rs</code> (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the <code>find_slot</code>/bucket-by-layer-and-slot construction against a small hand-built <code>QueryLayer</code> fixture, asserting the <code>(layer_idx, slot, seq_idx, pos)</code> tuples match what the old per-occurrence loop would have produced.</li>
 </ul>
 <h3 id="phase-4-column-major-matrix-fetch-roadmap-points-78-implemented-numa-parallelism-deferred">Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred</h3>
 <p><strong>Goal (revised during implementation)</strong>: replace <code>fill_row</code>-per-hit (row-major, worst-case mmap locality) with a column-major scan. <code>PartitionRunner</code> turned out to be the wrong mechanism for this at this call granularity — see below; the column-major fetch itself is implemented and validated, without it.</p>
 <p><strong>What shipped:</strong>
 - <code>obicompactvec</code>: the per-column accessors this phase needed <strong>already existed</strong> — <code>PersistentCompactIntMatrix::col_view(c)</code> and <code>PersistentBitMatrix::col_view(c)</code> are public, and <code>IntSliceView::get(slot)</code>/<code>BitSliceView::get(slot)</code> are public — the original plan underestimated how much of this plumbing the pairwise-distance code (<code>dump</code>/<code>select</code>/<code>stats</code>) had already required. The one real gap: <code>PersistentBitMatrix::col_view()</code> panics on the <code>Implicit</code> variant (the documented mono-genome fast path, <code>bitmatrix.rs</code>). Added <code>PersistentBitMatrix::get(c, slot) -&gt; u32</code> (<code>bitmatrix.rs</code>), a non-panicking column-major point lookup that returns <code>1</code> for <code>Implicit</code> regardless of <code>c</code> — the smallest surface needed, not a new <code>col_get</code> API from scratch.
-- <code>obikpartitionner/src/query_layer.rs</code>: <code>query_partition_with</code> is now two explicit stages, matching roadmap points 6–8: <strong>stage 1</strong> (MPHF-only, per unique k-mer, bucket hits by <code>(layer_idx, slot)</code>, emits <code>QueryHit::Found</code>) then <strong>stage 2</strong> (per layer with ≥1 hit, column-major: for each genome column <code>g</code> in <code>0..layer.n_cols().min(n_genomes)</code>, scan that layer's bucketed slots and call <code>col_value(g, slot)</code>, emitting <code>QueryHit::Value(descs, g, value)</code> on nonzero). <code>QueryHit</code> is a single enum delivered through one <code>FnMut(QueryHit)</code> callback — an earlier two-closure design (<code>on_found</code> + <code>on_value</code>) didn't borrow-check, since the caller's single mutable accumulator (<code>KmerResults</code>) can't be captured by two separate <code>FnMut</code> closures passed to the same call.
+- <code>obikindex/src/partition/query_layer.rs</code>: <code>query_partition_with</code> is now two explicit stages, matching roadmap points 6–8: <strong>stage 1</strong> (MPHF-only, per unique k-mer, bucket hits by <code>(layer_idx, slot)</code>, emits <code>QueryHit::Found</code>) then <strong>stage 2</strong> (per layer with ≥1 hit, column-major: for each genome column <code>g</code> in <code>0..layer.n_cols().min(n_genomes)</code>, scan that layer's bucketed slots and call <code>col_value(g, slot)</code>, emitting <code>QueryHit::Value(descs, g, value)</code> on nonzero). <code>QueryHit</code> is a single enum delivered through one <code>FnMut(QueryHit)</code> callback — an earlier two-closure design (<code>on_found</code> + <code>on_value</code>) didn't borrow-check, since the caller's single mutable accumulator (<code>KmerResults</code>) can't be captured by two separate <code>FnMut</code> closures passed to the same call.
 - <code>obikmer/src/cmd/query.rs</code>: <code>KmerResults::set</code> (row-major, whole-row-at-once) replaced by <code>mark_found</code> (stage 1: flag a position as indexed, independent of any genome's value) and <code>set_one</code> (stage 2: write one genome's value at one position). <code>QueryStats</code> extended with <code>n_columns_scanned</code>/<code>n_col_get_calls</code>, logged per chunk.
 - Total <code>get()</code>-equivalent calls are unchanged from the row-major version (<code>n_hits × n_cols</code> in the worst case, confirmed by <code>n_col_get_calls</code> in the debug log) — the win is locality (sequential access within one layer's column at a time, across <code>mmap</code>'d regions, instead of jumping across all columns per hit), exactly as predicted.</p>
 <p><strong>What did not ship, and why — <code>PartitionRunner</code> is architecturally the wrong tool here:</strong>
 Reading <code>obikindex/src/numa.rs</code>'s actual <code>run()</code> body (not just its doc comments) shows every call spawns a timer thread <strong>plus one OS thread per worker slot on every NUMA node</strong> (<code>std::thread::scope</code> + one <code>s.spawn()</code> per node per <code>max_workers</code>) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned <strong>per call</strong>. This is fine for its actual, established usage in this codebase (<code>merge.rs</code>, <code>index.rs</code>'s <code>build_layers</code>): one <code>PartitionRunner::new()</code> + one <code>run()</code> call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for <code>query</code>'s call pattern: <code>query_partition_with</code> runs once per <code>(chunk, partition)</code>, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
 The column-major loop in stage 2 is therefore a <strong>plain sequential loop</strong> for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) <code>rayon</code>'s already-warm global pool (<code>into_par_iter()</code>) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern <code>numa_worker_pools.md</code> warns about for a <em>different</em> workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one <code>mmap</code>'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so <code>PartitionRunner</code> is invoked once per whole <code>query</code> run (or per large batch of chunks) rather than per <code>(chunk, partition)</code>, amortising its spawn cost the way <code>merge</code>/<code>build_layers</code> do — a bigger structural change than this phase's scope.
 - Log (implemented): <code>QueryStats::n_columns_scanned</code>/<code>n_col_get_calls</code>, folded into the existing per-chunk <code>debug!("k-mer dedup + column-major fetch", ...)</code> line (<code>query.rs</code>) alongside phase 3's dedup counters.
-- <strong>Unit tests</strong>: extended <code>obikpartitionner/src/tests/query_layer.rs</code> (phase 3's file) — <code>query_partition_with</code>'s empty/missing-index paths updated for the new <code>QueryStats</code> fields and single-callback signature.
+- <strong>Unit tests</strong>: extended <code>obikindex/src/partition/tests/query_layer.rs</code> (phase 3's file) — <code>query_partition_with</code>'s empty/missing-index paths updated for the new <code>QueryStats</code> fields and single-callback signature.
 - <strong>Validation performed</strong>: full workspace build + <code>cargo test --workspace</code>, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same <code>kmer_count</code>/<code>kmer_strict_matches</code> on every record); (2) the existing 20-genome <code>benchmark/global_index_presence</code> index — runs correctly, <code>n_hits=0</code> for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the <code>Implicit</code>/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) <strong>the critical correctness case</strong>: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from <code>genomeA</code> matched <strong>only</strong> <code>genomeA</code> (<code>kmer_count</code> identical to the pre-dedup occurrence count, zero leakage into <code>genomeB</code>'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in <code>n_cols</code>, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
 - <strong>Not yet done</strong>: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-<code>n_genomes</code> layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.</p>
 <h3 id="phase-5-sparse-findere-rework-roadmap-point-9">Phase 5 — Sparse Findere rework (roadmap point 9)</h3>

DevDoc/architecture/rebuild_filter/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1389,9 +1445,9 @@ IO pattern: sequential read of each source column → random write into new colu
 </table>
 <h2 id="files-to-modify">Files to modify</h2>
 <ul>
-<li><code>src/obikpartitionner/src/rebuild_layer.rs</code> — <code>rebuild_partition</code> and <code>iter_src_layers</code></li>
+<li><code>src/obikindex/src/partition/rebuild_layer.rs</code> — <code>rebuild_partition</code> and <code>iter_src_layers</code></li>
 <li>Possibly <code>src/obicompactvec/</code> — add column iterator API if not already present</li>
-<li><code>src/obilayeredmap/</code> — check if per-column sequential access is exposed on <code>SrcLayerData</code></li>
+<li><code>src/obikindex/src/layer/</code> — check if per-column sequential access is exposed on <code>SrcLayerData</code></li>
 </ul>
 <h2 id="open-questions">Open questions</h2>
 <ul>

DevDoc/architecture/sequences/invariant.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/sequences/invariant/index.html

@@ -9,7 +9,7 @@
       
       
       
-        <link rel="prev" href="../../../implementation/obitaxonomy/">
+        <link rel="prev" href="../../../implementation/partition_layer_cache/">
       
       
         <link rel="next" href="../../index_architecture/">
@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/architecture/siblings/index.html

@@ -731,14 +731,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -948,6 +948,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1289,6 +1345,39 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#query-never-benefits-from-sparse-row-major-access-found-2026-08-19-not-implemented" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        query never benefits from sparse row-major access (found 2026-08-19, not implemented)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#implemented-2026-08-20" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Implemented (2026-08-20)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1552,6 +1641,39 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#query-never-benefits-from-sparse-row-major-access-found-2026-08-19-not-implemented" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        query never benefits from sparse row-major access (found 2026-08-19, not implemented)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#implemented-2026-08-20" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Implemented (2026-08-20)
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1587,7 +1709,7 @@ one-way function, not an invertible bijection with a stored inverse. Any
 method that reconstructs a kmer from a bare slot number is wrong by
 construction, regardless of the mechanism used (MPHF re-hash, or evidence
 decode + direct unitig read). See <code>MphfLayer::kmer_at</code>
-(<code>obilayeredmap/src/mphf_layer.rs</code>) — flagged for removal, currently called
+(<code>obikindex/src/layer/mphf_layer.rs</code>) — flagged for removal, currently called
 from <code>obikphylo/siblings/build.rs</code> and <code>family_scan.rs</code> (since removed — see
 "Pending work" status below).</p>
 <h2 id="two-pipelines-never-mixed">Two pipelines, never mixed</h2>
@@ -1650,7 +1772,7 @@ partition is unknown) and must keep going through
 <code>index()</code>.</p>
 <h2 id="pending-work-done">Pending work — done</h2>
 <p>The plan above shipped: <code>obikphylo</code> (a new crate — phylo-domain extension
-traits over <code>obikindex::KmerIndex</code>/<code>obilayeredmap::Layer&lt;D&gt;</code>, replacing the
+traits over <code>obikindex::KmerIndex</code>/<code>obikindex::layer::Layer&lt;D&gt;</code>, replacing the
 old <code>obikindex::siblings</code> module) builds and reads the annex purely in
 iteration order (<code>SiblingLayerExt::iter_siblings</code>/<code>iter_minorants</code>, both with
 batch variants, mirroring <code>Layer&lt;D&gt;</code>'s own <code>KmerIter</code>/<code>KmerBatchIter</code>
@@ -2218,13 +2340,13 @@ BRWT-style column-correlation exploitation.</p>
 <p><code>PersistentSparseBitMatrix</code> went from a validated but unused type to a
 real, selectable on-disk format:</p>
 <ul>
-<li><strong>Generic <code>Layer&lt;D&gt;</code></strong>: <code>obilayeredmap::Layer&lt;D&gt;</code>'s presence-only methods
+<li><strong>Generic <code>Layer&lt;D&gt;</code></strong>: <code>obikindex::layer::Layer&lt;D&gt;</code>'s presence-only methods
   (<code>n_cols</code>, <code>sub_matrix</code>, <code>fill_sub_matrix</code>) are generic over any
   <code>D: LayerData&lt;Item = Box&lt;[bool]&gt;&gt; + BinaryMatrix</code>, not hardcoded to
   <code>PersistentBitMatrix</code> — <code>PersistentSparseBitMatrix</code> implements
   <code>LayerData</code> (<code>open</code>/<code>read</code>) the same way. <code>find_slot</code>/<code>index_batch</code> were
   already generic over any <code>D: LayerData</code>, so they needed no change.
-  Verified by <code>obilayeredmap</code>'s
+  Verified by <code>obikindex::layer</code>'s
   <code>presence_layer_generic_over_sparse_matches_dense</code> test: build a dense
   presence layer, convert it to sparse via <code>build_from_dense</code>, open both
   as <code>Layer&lt;PersistentBitMatrix&gt;</code>/<code>Layer&lt;PersistentSparseBitMatrix&gt;</code> on
@@ -2234,7 +2356,7 @@ real, selectable on-disk format:</p>
   <code>AtomicUsize</code>s in test builds, not thread-local, so a test using a
   different <code>k</code> races every other test in the same crate binary; a k=11
   version of this test passed alone but failed under the full
-  <code>obilayeredmap</code> suite for exactly that reason before being fixed.)</li>
+  <code>obikindex::layer</code> suite for exactly that reason before being fixed.)</li>
 <li><strong><code>obikphylo::siblings::cache::Mat</code></strong> gained a third variant,
   <code>SparsePresence(Layer&lt;PersistentSparseBitMatrix&gt;)</code>, alongside <code>Count</code>
   and <code>Presence</code> — every method (<code>find_slot</code>, <code>index_batch</code>,
@@ -2264,7 +2386,7 @@ real, selectable on-disk format:</p>
   dense-path test (<code>sibling_annex_one_sibling_each</code>) exactly — proves the
   sparse format round-trips through the real build pipeline
   (<code>PartitionCache</code> sparse-detection included), not just the
-  <code>obicompactvec</code>/<code>obilayeredmap</code> unit layers below it.</li>
+  <code>obicompactvec</code>/<code>obikindex::layer</code> unit layers below it.</li>
 </ul>
 <p>Full workspace <code>cargo test</code> (all crates, unit + doc tests) green after
 this change.</p>
@@ -2310,7 +2432,7 @@ this change.</p>
   (activation: either given; defaults <code>1.0</code>/<code>0.5</code> for whichever is unset).</li>
 </ul>
 <p>Full workspace <code>cargo test</code> green after this change (167 unit tests in
-<code>obicompactvec</code>+<code>obilayeredmap</code>+<code>obikphylo</code> alone, plus every other
+<code>obicompactvec</code>+<code>obikindex::layer</code>+<code>obikphylo</code> alone, plus every other
 crate's suite, no regressions).</p>
 <p><strong>Still open, not part of this change</strong> (per "Correction to the 'single
 pass' framing" above): <code>--raw-snp-distance</code>/<code>--raw-snp-counts</code> (the
@@ -2319,6 +2441,261 @@ scan the full unsampled index — never threaded <code>--subsample</code>/<code>
 out of scope here since the reported problem was specifically about the
 <code>--sankoff</code>/<code>--tnt</code> pipeline's redundant/inconsistent scans, not these
 two standalone flags.</p>
+<h2 id="query-never-benefits-from-sparse-row-major-access-found-2026-08-19-not-implemented"><code>query</code> never benefits from sparse row-major access (found 2026-08-19, not implemented)</h2>
+<p>Benchmarked <code>obikmer query</code> against <code>global_index_presence</code> (dense-packed)
+vs. <code>global_index_presence_sparse</code> (<code>pack --sparse</code>), 100k simulated reads
+× 2 specimens (<code>benchmark/</code>, see
+<a href="../../implementation/benchmark_query_testing/">benchmark_query_testing.md</a>).
+Correctness: 0 mismatches — sparse and dense return bit-identical query
+results. Performance: sparse consistently <em>slower</em> than dense (~30-50%,
+reproducible across two runs with warm disk cache), the opposite of
+<code>pack --sparse</code>'s stated intent ("faster for single-row access... like
+query").</p>
+<p><strong>Root cause, read from source, not measured in isolation:</strong>
+<code>KmerPartition::query_partition_with</code> (<code>obikindex/src/partition/query_layer.rs:155-220</code>)
+is architecturally column-major: stage 2 walks <code>for g in 0..n_cols { for
+slot in hit_slots { layer.col_value(g, slot) } }</code>, documented (correctly)
+as the right locality strategy for the packed/columnar formats, where
+<code>col_value</code> → <code>PersistentBitMatrix::get</code> is a genuine O(1) mmap'd column
+read (<code>persistent.rs:110-113</code>).</p>
+<p>For <code>Self::Sparse</code>, that same <code>get(c, slot)</code> (<code>persistent.rs:114-118</code>)
+allocates a full <code>n_cols</code>-wide buffer and calls <code>fill_row</code> — materializing
+the <em>entire row</em> — just to return one cell. Called from inside the
+column-major double loop, this reconstructs the same row once per genome
+column touched: O(hits × n_cols) full-row rebuilds instead of O(hits).
+<code>PersistentSparseBitMatrix</code>'s own native row-major decode
+(<code>for_each_genome_in_row</code>, <code>sparse.rs:164-177</code>, used correctly by its own
+<code>row</code>/<code>fill_row</code>/<code>fill_sub_matrix</code>) is never reached from the query path
+at all.</p>
+<p><strong><code>fill_sub_matrix</code> (the existing <code>BinaryMatrix</code> trait primitive,
+<code>traits.rs:13-37</code>) is not the right replacement for <code>query</code> either</strong>, even
+once its own dispatch bug is fixed (see next section) — its output shape
+is inherently column-dense: <code>out[col]</code> gets an entry for every column,
+including columns with zero hits among the requested slots. On real
+sparse data (a hit typically touching a handful of genomes out of dozens)
+that's still O(n_cols) output regardless of true sparsity. What <code>query</code>
+actually wants is the sparse triple stream <code>(slot, col, value)</code> it already
+consumes as <code>QueryHit::Value</code> — not a materialized sub-matrix.</p>
+<p><strong>Proposed primitive</strong> (design only, not implemented — explicit ask: keep
+count matrices <em>not excluded</em>, even though effort right now is
+presence/absence only):</p>
+<p>Not a closure-driven <code>for_each</code> — a real <code>Iterator</code>, one concrete struct
+per matrix format, so the traversal state (current position in the sorted
+slot list, current column, permutation, sparse-row decode cursor…) lives
+in named struct fields instead of being threaded implicitly through
+recursion or a captured closure. RPITIT (stable since 1.75, and this
+workspace is edition 2024) means the trait method can return it without
+naming or boxing the concrete type:</p>
+<div class="highlight"><pre><span></span><code><span class="sd">/// Yields every nonzero cell among `slots`, in implementation-defined order.</span>
+<span class="k">fn</span><span class="w"> </span><span class="nf">nonzero_iter</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="p">(</span><span class="o">&amp;&#39;</span><span class="na">a</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">a</span><span class="w"> </span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="o">&lt;</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="o">&gt;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">&#39;</span><span class="na">a</span><span class="p">;</span>
+<span class="c1">// item: (idx into `slots`, col, value)</span>
+</code></pre></div>
+<p>This is the one foundational primitive per format — both <code>fill_sub_matrix</code>
+and a <code>for_each</code>-style callback become trivial, free consumers of it
+(<code>.for_each(f)</code> is already <code>Iterator::for_each</code> from std; <code>fill_sub_matrix</code>
+becomes "drain the iterator, scatter into <code>out[][]</code>"), instead of two
+independently-maintained traversals that can silently diverge (see the bug
+below — this is exactly how it happened).</p>
+<ul>
+<li><strong>On <code>PersistentSparseBitMatrix</code></strong>: the struct is nearly free to write —
+  it wraps the existing (currently private) <code>for_each_genome_in_row</code>
+  per-row decode, advancing to the next <code>slots</code> entry on exhaustion. O(Σ
+  row nnz), zero <code>n_cols</code>-wide allocation.</li>
+<li><strong>On <code>PersistentBitMatrix::{Packed,Columnar}</code></strong>: revised — cheaper than
+  first thought, by reusing the same split already used for
+  <code>fill_matrix</code>'s own implementation instead of hand-writing a resumable
+  state machine at the matrix level. The base-vector layer
+  (<code>BitSliceView</code>, <code>views.rs</code>) already separates the two concerns:
+  <code>fill_batch_sorted</code> (<code>views.rs:55-60</code>, sorted-slot batch lookup) and a
+  genuine per-bit <code>Iterator</code> (<code>BitSliceIter</code>, <code>views.rs:94+</code>) sit side by
+  side there, one level below the matrix. Adding a
+  "positions among <code>sorted_slots</code> where the bit is set" iterator at that
+  same vector level is a <code>filter</code> over the existing <code>get()</code> — no new
+  state machine, since <code>std::iter::Filter</code> already <em>is</em> one, generated by
+  the compiler:
+  <div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">nonzero_among_sorted</span><span class="o">&lt;&#39;</span><span class="na">s</span><span class="o">&gt;</span><span class="p">(</span><span class="o">&amp;&#39;</span><span class="na">s</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">sorted_slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">s</span><span class="w"> </span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="o">&lt;</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">usize</span><span class="o">&gt;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">&#39;</span><span class="na">s</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="n">sorted_slots</span><span class="p">.</span><span class="n">iter</span><span class="p">().</span><span class="n">copied</span><span class="p">().</span><span class="n">filter</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">|&amp;</span><span class="n">slot</span><span class="o">|</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">slot</span><span class="p">))</span>
+<span class="p">}</span>
+</code></pre></div>
+  The matrix-level <code>nonzero_iter</code> then composes these per column with
+  <code>flat_map</code> over <code>0..n_cols</code> (each column's hits, tagged with <code>c</code>,
+  slot mapped back through the sort permutation <code>fill_batch</code>/
+  <code>fill_batch_sorted</code> already carry) — again a combinator chain, not a
+  hand-rolled struct. Same algorithm, same mmap/sort locality as today's
+  <code>fill_sub_matrix</code>; just assembled from <code>std</code> iterator adaptors instead
+  of a loop body writing into a buffer, mirroring the vector/matrix split
+  the codebase already uses for <code>fill_batch_sorted</code> rather than
+  introducing a new shape.</li>
+<li><strong><code>Implicit</code></strong>: trivial (<code>slots.iter().map(|&amp;i| (i, 0, 1))</code>, one column,
+  always present).</li>
+<li><strong>On <code>PersistentCompactIntMatrix</code> (counts)</strong>: same treatment as
+  <code>Packed</code>/<code>Columnar</code> — no sparse count format exists yet ("Explicitly
+  deferred" per <code>traits.rs:9-12</code>), so no native low-effort case the way
+  <code>Sparse</code> has one, but not excluded either: the iterator's <code>Item</code> is
+  already <code>(usize, usize, u32)</code>, not <code>bool</code>, specifically so presence
+  (<code>0</code>/<code>1</code>) and counts (arbitrary <code>u32</code>) share one primitive instead of a
+  bool/u32 split forcing counts out of the design. Ready for a native
+  sparse-count struct later without a signature change.</li>
+</ul>
+<p>Would let <code>query_partition_with</code>'s stage 2 collapse to one
+<code>for (i, g, v) in layer.matrix().nonzero_iter(&amp;hit_slots) { on_event(...) }</code>
+per layer, format-agnostic, each backend's struct deciding the actual
+traversal.</p>
+<p><strong>This also closes the existing dispatch bug for free, by construction</strong>:
+<code>PersistentBitMatrix::fill_sub_matrix</code> (<code>persistent.rs:190-215</code>, the enum
+wrapper backing <code>BinaryMatrix</code>'s trait impl) today does <em>not</em> delegate to
+<code>PersistentSparseBitMatrix::fill_sub_matrix</code> for <code>Self::Sparse</code> — it
+reimplements the same naive per-(column, slot) <code>fill_row_bool</code> loop
+instead, bypassing the efficient native method one file over
+(<code>sparse.rs:249-258</code>). <code>obikphylo::siblings::cache::Mat</code>
+(<code>cache.rs:138-145</code>) independently built its own parallel enum wrapper
+that dispatches correctly — evidence this was worked around rather than
+fixed at the source: two hand-written traversals for the same format,
+free to drift apart, and they did. If <code>fill_sub_matrix</code> itself is
+rewritten as "drain <code>nonzero_iter</code>, scatter into <code>out[][]</code>", there is only
+one traversal per format left to get right — the bug class doesn't just
+get fixed once, it stops being possible to reintroduce.</p>
+<h2 id="implemented-2026-08-20">Implemented (2026-08-20)</h2>
+<p>Built as designed above, with one deviation from the original sketch:
+<code>nonzero_iter</code> ended up <code>Box&lt;dyn Iterator&lt;...&gt;&gt;</code>, not a bare <code>impl
+Iterator</code>, because <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are genuinely
+different concrete types and this method isn't on a trait (kept off
+<code>BinaryMatrix</code> deliberately — that trait is used as <code>dyn BinaryMatrix</code> in
+<code>tests/sparse.rs</code>, and RPITIT methods aren't dyn-compatible). One <code>Box</code>
+per <code>nonzero_iter</code> call, not per cell — negligible next to what it
+replaces.</p>
+<ul>
+<li><code>BitSliceView::nonzero_among_sorted</code> / <code>IntSliceView::nonzero_among_sorted</code>
+  (<code>obicompactvec/src/views.rs</code>): the vector-level <code>filter</code>/<code>filter_map</code>
+  primitive, exactly as sketched — no new state machine, <code>std</code>'s own.</li>
+<li><code>PersistentSparseBitMatrix::nonzero_iter</code> (<code>bitmatrix/sparse.rs</code>): native,
+  <code>std::iter::from_fn</code> over one buffered row at a time via the existing
+  <code>for_each_genome_in_row</code> — no <code>n_cols</code>-wide allocation, ever.</li>
+<li><code>PersistentBitMatrix::nonzero_iter</code> (<code>bitmatrix/persistent.rs</code>): dispatches
+  to the above for <code>Sparse</code>; for <code>Columnar</code>/<code>Packed</code>, loops columns,
+  collects each column's <code>nonzero_among_sorted</code> hits via <code>.extend()</code> (not
+  <code>flat_map</code> — a <code>flat_map</code> closure can't lazily return something
+  borrowing its own captured sort permutation across separate calls
+  without either boxing per-column or fighting the borrow checker; eager
+  collection into one <code>Vec</code> sidesteps it, at zero cost since
+  <code>fill_sub_matrix</code> already fully materialized anyway). <code>Implicit</code> trivial.</li>
+<li><code>PersistentBitMatrix::fill_sub_matrix</code> and <code>sub_matrix</code> rewritten to
+  drain <code>nonzero_iter</code> — the dispatch bug is gone because there is now
+  only one traversal per format, not because the old one was patched.
+  <code>PersistentCompactIntMatrix::nonzero_iter</code> added the same way (counts
+  not excluded, per the earlier ask) — no native low-effort case, since no
+  sparse count format exists, but on the same primitive, ready for one.</li>
+<li><code>KmerPartition::query_partition_with</code> (<code>obikindex/src/partition/query_layer.rs</code>):
+  stage 2's column-major <code>for g { for slot { col_value } }</code> replaced by one
+  <code>layer.nonzero_iter(&amp;slot_list)</code> call per layer, format-agnostic.</li>
+<li>Tests: <code>nonzero_iter_matches_dense</code>, <code>nonzero_iter_matches_row</code>, and —
+  the one that actually targets the dispatch bug rather than each type's
+  own correctness — <code>enum_wrapper_dispatches_to_native_sparse</code> (builds
+  <code>PersistentBitMatrix::Sparse(...)</code> directly, not through <code>open</code>, since
+  <code>open</code> only auto-detects <code>Sparse</code> from a <code>presence/</code> dir layout).
+  <code>cargo test --workspace</code>: green, no regressions.</li>
+</ul>
+<p><strong>Measured</strong>: re-ran the <code>benchmark/</code> query branch (100k reads × 2
+specimens, same setup as the original finding). Correctness still 0
+mismatches. The dense/sparse performance gap is gone — previously sparse
+~30-50% slower than dense, reproducibly; now within ~1-3% either way
+(7.42s dense vs 7.60s sparse for <code>Escherichia_coli--K-12_MG1655</code>; 5.25s vs
+5.30s for <code>Saccharolobus_islandicus--M.16.4</code>) — noise-level, not a
+systematic gap. <code>pack --sparse</code>'s claimed query win isn't confirmed
+outright by this (sparse should arguably now <em>beat</em> dense on truly sparse
+real data, not just tie), but the pathological regression is fixed.</p>
+<h2 id="persistentcompactintmatrixsparse-implemented-2026-08-26"><code>PersistentCompactIntMatrix::Sparse</code> — implemented (2026-08-26)</h2>
+<p>Closes the gap flagged throughout this document ("no sparse count format
+exists yet", <code>traits.rs:9-12</code>'s "Explicitly deferred"): <code>obicompactvec</code>
+already had <code>PersistentSparseCompactIntMatrix</code> (row-major, built on top of
+<code>PersistentSparseBitMatrix</code> as its "which columns are non-zero" support,
+values <em>not</em> deduplicated — see that struct's own doc comment), but it was
+never wired into <code>PersistentCompactIntMatrix</code>, the dense-dispatching enum
+every real consumer (<code>TypedLayer&lt;PersistentCompactIntMatrix&gt;</code>,
+<code>KmerLayer::Count</code>) actually holds. Concretely: <code>kmer_index.rs::
+pack_matrices(sparse=true)</code> already called <code>pack_sparse_compact_int_matrix</code>
+on every layer's <code>counts/</code> — but <code>PersistentCompactIntMatrix::open</code> had no
+code path back to what that just wrote, so a <code>Count</code> layer became
+unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
+anyone ran <code>pack --sparse</code> on an index with count layers. Root cause, not a
+workaround: add the missing <code>Sparse</code> variant.</p>
+<ul>
+<li>
+<p><strong>Enum + dispatch</strong> (<code>intmatrix.rs</code>): <code>PersistentCompactIntMatrix::Sparse
+  (PersistentSparseCompactIntMatrix)</code>, detected in <code>open</code>/<code>detect_storage</code>
+  via a <code>singleton_values.pciv</code> marker (mirrors <code>PersistentBitMatrix</code>'s own
+  <code>sparse_meta.json</code> check), reported via <code>storage_kind()</code>. <code>col</code>/
+  <code>col_view</code>/<code>col_persist</code> panic/<code>Unsupported</code> on <code>Sparse</code>, same convention
+  as the bit side. <code>sub_matrix</code>/<code>fill_sub_matrix</code> and <code>nonzero_iter</code>
+  unified the same way <code>PersistentBitMatrix</code>'s already are (drain
+  <code>nonzero_iter</code>, one traversal per format — see "Implemented
+  (2026-08-20)" above); <code>nonzero_iter</code> had to become <code>Box&lt;dyn Iterator&lt;...&gt;&gt;</code>
+  for the same reason (<code>Columnar</code>/<code>Packed</code>/<code>Sparse</code> are different concrete
+  types). No change needed in <code>obikindex</code> at all — <code>KmerLayer::Count</code>
+  already only ever holds <code>TypedLayer&lt;PersistentCompactIntMatrix&gt;</code>, so the
+  enum absorbing <code>Sparse</code> fixes the unreadable-layer bug for free, same as
+  <code>PersistentBitMatrix::Sparse</code> already did on the presence side.</p>
+</li>
+<li>
+<p><strong><code>CountPartials</code>, non-naive</strong> (<code>sparse_intmatrix.rs</code>): unlike
+  <code>PersistentSparseBitMatrix</code>'s dict-driven <code>col_weights_and_pair_counts</code>,
+  values here aren't deduplicated (two rows can share the same non-zero
+  column set via the same <code>dict_id</code> while carrying different counts), so
+  the "weight by how many rows share a dict entry" shortcut doesn't carry
+  over. What does: a single row-major pass (<code>row_major_pairwise</code>, decodes
+  each row once via <code>for_each_cell_in_row</code>, nests over that row's own
+  co-present columns) — <code>O(Σ k̄²)</code> over populated rows instead of the naive
+  <code>O(n_cols² × n)</code> column-pair rescan, same complexity class as the bit
+  side minus the dict multiplicity discount. Kernels used: <code>min(a,b)</code>
+  (bray, relfreq-bray — both vanish when either side is absent, so no
+  correction needed), <code>a·b</code> and <code>√(a·b)</code> (euclidean/relfreq-euclidean and
+  hellinger — these <em>do</em> need a correction, reconstructed from per-column
+  marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>, since <code>(a-0)² = a² ≠ 0</code> unlike
+  the <code>min</code>-based formulas). <code>threshold_jaccard(1)</code> shortcuts straight to
+  <code>support</code>'s own <code>BitPartials::partial_jaccard</code> (threshold 1 is exactly
+  presence); <code>threshold_jaccard(0)</code> is closed-form (every <code>u32</code> is <code>≥ 0</code>).</p>
+</li>
+<li>
+<p><strong>Two pre-existing bugs found and fixed while wiring the <code>threshold==1</code>
+  shortcut</strong> (<code>bitmatrix/sparse.rs</code>, <code>BitPartials for
+  PersistentSparseBitMatrix</code>, present since the 2026-08-15 implementation
+  above, never caught because no test compared <code>Sparse</code>'s raw <code>partial_*</code>
+  output against dense on real data — only the diagonal-blind
+  <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested):</p>
+</li>
+<li><code>partial_jaccard</code>'s diagonal was <code>(0, 2×col_weights[i])</code> instead of a
+     genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
+     <code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with
+     itself by construction.</li>
+<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: <code>total -
+     union</code> (count of rows where <em>neither</em> column is present) instead of
+     the actual Hamming distance <code>col_weights[i] + col_weights[j] -
+     2×inter[i,j]</code> (symmetric-difference size). Only coincides with the
+     correct value when <code>col_weights[i] + col_weights[j] == total</code>, so
+     small/synthetic test data could easily have hidden it.</li>
+</ul>
+<p>Neither surfaced through <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code>
+  (both explicitly zero their own diagonal at finalisation, and the
+  off-diagonal <code>partial_hamming</code> bug had gone untested against dense
+  entirely) — only visible to a caller of the raw <code>partial_*</code> methods
+  directly, which is exactly what <code>partial_threshold_jaccard(1)</code>'s new
+  shortcut became. Fixed at the source, not patched around at the call
+  site; regression test added:
+  <code>tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal</code>.</p>
+<ul>
+<li><strong>Tests</strong>: <code>tests::intmatrix::sparse_roundtrip_matches_columnar</code>/
+  <code>sparse_roundtrip_from_packed</code> (the <code>open</code>-dispatch fix, both build
+  paths); <code>tests::intmatrix::sparse_count_partials_match_dense</code> (all six
+  <code>CountPartials</code> formulas, thresholds 0/1/2/3, against <code>Columnar</code> on
+  asymmetric-presence data — this is what caught the diagonal gap in the
+  int side's own new code before it shipped, the same way it exposed the
+  two pre-existing bit-side bugs above); <code>obikindex</code>'s
+  <code>count_layer_transparently_reads_sparse_after_pack</code> — the actual
+  end-to-end regression test for the original "layer unreadable after
+  <code>pack --sparse</code>" bug, built → packed sparse → reopened, compared against
+  the pre-pack dense read. <code>cargo test -p obicompactvec -p obikindex</code>:
+  green, no regressions (180 + 12 tests).</li>
+</ul>
 
 
 

DevDoc/implementation/chunkreader.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/chunkreader/index.html

@@ -405,11 +405,11 @@
 </a>
 </li>
 <li class="md-nav__item">
-<a class="md-nav__link" href="../obilayeredmap/">
+<a class="md-nav__link" href="../layer_tier/">
 <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -497,6 +497,30 @@
   
 
     
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../benchmark_query_testing/">
+<span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../partition_layer_cache/">
+<span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
   </span>
 </a>
 </li>

DevDoc/implementation/evidence_elimination.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,9 +1270,9 @@
 <h1 id="coverage-implementationevidence_eliminationmd">Coverage: implementation/evidence_elimination.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()</li>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — build_approx_evidence(dir, b, z), find_approx()</li>
-<li><code>obilayeredmap/src/meta.rs</code> — EvidenceKind::Approx { b, z }, LayerMeta</li>
+<li><code>obikindex/src/layer/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()</li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — build_approx_evidence(dir, b, z), find_approx()</li>
+<li><code>obikindex/src/layer/meta.rs</code> — EvidenceKind::Approx { b, z }, LayerMeta</li>
 <li><code>obikindex/src/reindex.rs</code> — KmerIndex::reindex(), conversion exact↔approx en place</li>
 <li><code>obikmer/src/cmd/reindex.rs</code> — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size</li>
 <li><code>obikmer/src/cmd/index.rs</code> — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp</li>

DevDoc/implementation/evidence_elimination/index.html

@@ -12,7 +12,7 @@
         <link rel="prev" href="../unitig_evidence/">
       
       
-        <link rel="next" href="../obilayeredmap/">
+        <link rel="next" href="../layer_tier/">
       
       
         
@@ -868,14 +868,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1085,6 +1085,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/filtering/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1208,6 +1208,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -2130,7 +2186,7 @@ obikmer<span class="w"> </span>phylo<span class="w"> </span>myindex<span class="
 <h2 id="implementation">Implementation</h2>
 <ul>
 <li>
-<p><strong><code>obikpartitionner::filter::GroupQuorumFilter</code></strong> — implements <code>KmerFilter</code>
+<p><strong><code>obikindex::partition::filter::GroupQuorumFilter</code></strong> — implements <code>KmerFilter</code>
   using pre-computed ingroup and outgroup index vectors. The heavy logic
   (predicate parsing, three-value evaluation, genome classification) happens
   once before any iteration; each k-mer row evaluation is a simple index
@@ -2143,7 +2199,7 @@ obikmer<span class="w"> </span>phylo<span class="w"> </span>myindex<span class="
   list.</p>
 </li>
 <li>
-<p><strong><code>obikpartitionner::KmerPartition::iter_partition_kmers</code></strong> — accepts
+<p><strong><code>obikindex::partition::KmerPartition::iter_partition_kmers</code></strong> — accepts
   <code>filters: &amp;[Box&lt;dyn KmerFilter&gt;]</code> and applies them per-kmer before invoking
   the callback. <code>filter</code>, <code>dump</code>, and <code>unitig</code> all go through this single
   entry point.</p>

DevDoc/implementation/kmer.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/kmer/index.html

@@ -868,14 +868,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1085,6 +1085,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/layer_tier.refs/index.html

@@ -19,7 +19,7 @@
     
     
       
-        <title>Obilayeredmap.refs - obikmer</title>
+        <title>Layer tier.refs - obikmer</title>
       
     
     
@@ -60,7 +60,7 @@
     <div data-md-component="skip">
       
         
-        <a href="#coverage-implementationobilayeredmapmd" class="md-skip">
+        <a href="#coverage-implementationlayer_tiermd" class="md-skip">
           Skip to content
         </a>
       
@@ -96,7 +96,7 @@
         <div class="md-header__topic" data-md-component="header-topic">
           <span class="md-ellipsis">
             
-              Obilayeredmap.refs
+              Layer tier.refs
             
           </span>
         </div>
@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1211,15 +1267,15 @@
 
 
 <!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
-<h1 id="coverage-implementationobilayeredmapmd">Coverage: implementation/obilayeredmap.md</h1>
+<h1 id="coverage-implementationlayer_tiermd">Coverage: implementation/layer_tier.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()</li>
-<li><code>obilayeredmap/src/layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()</li>
-<li><code>obilayeredmap/src/map.rs</code> — LayeredMap<D>, push_layer(), query()</li>
-<li><code>obilayeredmap/src/evidence.rs</code> — Evidence, EvidenceWriter, encodage chunk_id:rank</li>
-<li><code>obilayeredmap/src/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, matches()</li>
-<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta, EvidenceKind (Exact / Approx { b, z })</li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()</li>
+<li><code>obikindex/src/layer/typed_layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()</li>
+<li><code>obikindex/src/layer/map.rs</code> — LayeredMap<D>, push_layer(), query()</li>
+<li><code>obikindex/src/layer/evidence.rs</code> — Evidence, EvidenceWriter, encodage chunk_id:rank</li>
+<li><code>obikindex/src/layer/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, matches()</li>
+<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta, EvidenceKind (Exact / Approx { b, z })</li>
 </ul>
 <h2 id="notes">Notes</h2>
 <p>FORT RISQUE DE DÉRIVE. C'est le fichier le plus affecté par les changements récents :

DevDoc/implementation/layer_tier/index.html

@@ -23,7 +23,7 @@
     
     
       
-        <title>obilayeredmap crate - obikmer</title>
+        <title>obikindex layer tier - obikmer</title>
       
     
     
@@ -64,7 +64,7 @@
     <div data-md-component="skip">
       
         
-        <a href="#obilayeredmap-layered-kmer-index-crate" class="md-skip">
+        <a href="#obikindexlayer-the-layer-tier" class="md-skip">
           Skip to content
         </a>
       
@@ -100,7 +100,7 @@
         <div class="md-header__topic" data-md-component="header-topic">
           <span class="md-ellipsis">
             
-              obilayeredmap crate
+              obikindex layer tier
             
           </span>
         </div>
@@ -749,7 +749,7 @@
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -767,7 +767,7 @@
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -940,6 +940,67 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#layerd-raw-mapping-iteration-and-batch-access" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Layer\&lt;D> — raw mapping, iteration, and batch access
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="Layer\&lt;D&gt; — raw mapping, iteration, and batch access">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#raw-kmer-slot-mapping" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Raw kmer → slot mapping
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#kmer-iteration" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Kmer iteration
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#batch-lookup-on-payload-vectorsviews" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Batch lookup on payload vectors/views
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#sub_matrix-fill_sub_matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        sub_matrix / fill_sub_matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1263,6 +1324,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1638,6 +1755,67 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#layerd-raw-mapping-iteration-and-batch-access" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Layer\&lt;D> — raw mapping, iteration, and batch access
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="Layer\&lt;D&gt; — raw mapping, iteration, and batch access">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#raw-kmer-slot-mapping" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Raw kmer → slot mapping
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#kmer-iteration" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Kmer iteration
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#batch-lookup-on-payload-vectorsviews" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Batch lookup on payload vectors/views
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#sub_matrix-fill_sub_matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        sub_matrix / fill_sub_matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1773,9 +1951,9 @@
 
 
 
-<h1 id="obilayeredmap-layered-kmer-index-crate">obilayeredmap — layered kmer index crate</h1>
+<h1 id="obikindexlayer-the-layer-tier">obikindex::layer — the Layer tier</h1>
 <h2 id="purpose">Purpose</h2>
-<p><code>obilayeredmap</code> implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a <code>ptr_hash</code> MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.</p>
+<p><code>obikindex::layer</code> (the <code>layer/</code> submodule of the <code>obikindex</code> crate — a standalone <code>obilayeredmap</code> crate until 2026-08-21, folded back in alongside <code>obikpartition</code> as part of a broader <code>Index { Partition { Layer } }</code> submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a <code>ptr_hash</code> MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.</p>
 <hr />
 <h2 id="three-usage-modes">Three usage modes</h2>
 <p>The MPHF + evidence infrastructure is the same for all modes. The <strong>payload</strong> varies.</p>
@@ -2000,6 +2178,38 @@
 </code></pre></div>
 <p>Mode 3 (<code>PersistentBitMatrix</code>) has no <code>push_layer</code> on <code>LayeredMap</code>; callers build directly via <code>Layer&lt;PersistentBitMatrix&gt;::build_presence</code>.</p>
 <hr />
+<h2 id="layerd-raw-mapping-iteration-and-batch-access">Layer\&lt;D&gt; — raw mapping, iteration, and batch access</h2>
+<p>Beyond <code>query</code>/<code>find</code> (membership-checked), <code>Layer&lt;D&gt;</code> exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.</p>
+<h3 id="raw-kmer-slot-mapping">Raw kmer → slot mapping</h3>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">index</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kmer</span><span class="p">:</span><span class="w"> </span><span class="nc">CanonicalKmer</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">usize</span>
+<span class="nc">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">index_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kmers</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="n">CanonicalKmer</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">usize</span><span class="o">&gt;</span>
+</code></pre></div>
+<p>Pure MPHF mapping, no evidence/fingerprint check — equivalent to <code>MphfOnly::index</code>. Only meaningful when the caller already knows <code>kmer</code> belongs to the layer; on an absent kmer the MPHF still returns <em>some</em> slot (undefined, not <code>None</code>).</p>
+<h3 id="kmer-iteration">Kmer iteration</h3>
+<p>Four iterators, all built from <code>unitigs.bin</code> (physical layout order, <strong>not</strong> correlated with MPHF slot numbers):</p>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">iter_kmers</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">KmerIter</span><span class="o">&lt;&#39;</span><span class="nb">_</span><span class="o">&gt;</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">enumerate_kmers</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Enumerate</span><span class="o">&lt;</span><span class="n">KmerIter</span><span class="o">&lt;&#39;</span><span class="nb">_</span><span class="o">&gt;&gt;</span><span class="w">              </span><span class="c1">// (order_index, kmer)</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">iter_kmers_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">KmerBatchIter</span><span class="o">&lt;&#39;</span><span class="nb">_</span><span class="o">&gt;</span><span class="w">          </span><span class="c1">// Vec&lt;CanonicalKmer&gt; of size ≤ n</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">enumerate_kmers_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="o">&lt;</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="n">CanonicalKmer</span><span class="o">&gt;</span><span class="p">)</span><span class="o">&gt;</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">&#39;</span><span class="nb">static</span>
+</code></pre></div>
+<p><code>KmerIter</code>/<code>KmerBatchIter</code> own a clone of the underlying <code>Arc&lt;UnitigFileReader&gt;</code> rather than borrowing <code>self</code> — <code>Send + 'static</code>, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. <code>enumerate_kmers_batch</code>'s index is the batch's starting offset in iteration order (a multiple of <code>n</code> except for the final, possibly shorter, batch).</p>
+<h3 id="batch-lookup-on-payload-vectorsviews">Batch lookup on payload vectors/views</h3>
+<p><code>PersistentCompactIntVec</code>, <code>PersistentBitVec</code>, <code>IntSliceView</code>, <code>BitSliceView</code> all expose:</p>
+<div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">get_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span>
+<span class="k">fn</span><span class="w"> </span><span class="nf">fill_batch</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="n">T</span><span class="p">])</span>
+</code></pre></div>
+<p>Both sort <code>slots</code> internally for sequential mmap access, then reorder results back to the caller's original order. <code>fill_batch</code> fills a caller-provided buffer, avoiding the <code>Vec</code> allocation.</p>
+<h3 id="sub_matrix-fill_sub_matrix">sub_matrix / fill_sub_matrix</h3>
+<div class="highlight"><pre><span></span><code><span class="c1">// Layer&lt;PersistentCompactIntMatrix&gt;</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;&gt;</span><span class="w">            </span><span class="c1">// column-first</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;</span><span class="p">])</span>
+
+<span class="c1">// Layer&lt;PersistentBitMatrix&gt; (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;&gt;</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;</span><span class="p">])</span>
+</code></pre></div>
+<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this degrades to a row-by-row decode; see <a href="../../architecture/siblings/">siblings.md</a>.</p>
+<hr />
 <h2 id="layeredstores-and-aggregation-traits">LayeredStore\&lt;S&gt; and aggregation traits</h2>
 <p><code>LayeredStore&lt;S&gt;</code> is a generic aggregation wrapper over <code>Vec&lt;S&gt;</code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
 <div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayeredStore</span><span class="o">&lt;</span><span class="n">S</span><span class="o">&gt;</span><span class="p">(</span><span class="k">pub</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="n">S</span><span class="o">&gt;</span><span class="p">);</span>

DevDoc/implementation/merge.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1215,9 +1271,9 @@
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
 <li><code>obikindex/src/merge.rs</code> — <code>KmerIndex::merge()</code>, validation de compatibilité d'évidence, <code>validate_evidence_compat()</code></li>
-<li><code>obikpartitionner/src/merge_layer.rs</code> — <code>merge_partition()</code>, construction de la nouvelle layer, paramètre <code>block_bits</code></li>
-<li><code>obikpartitionner/src/rebuild_layer.rs</code> — <code>rebuild_partition()</code>, paramètre <code>block_bits</code></li>
-<li><code>obilayeredmap/src/layer.rs</code> — <code>Layer::append_genome_column()</code> (PersistentCompactIntMatrix et PersistentBitMatrix)</li>
+<li><code>obikindex/src/partition/merge_layer.rs</code> — <code>merge_partition()</code>, construction de la nouvelle layer, paramètre <code>block_bits</code></li>
+<li><code>obikindex/src/partition/rebuild_layer.rs</code> — <code>rebuild_partition()</code>, paramètre <code>block_bits</code></li>
+<li><code>obikindex/src/layer/typed_layer.rs</code> — <code>Layer::append_genome_column()</code> (PersistentCompactIntMatrix et PersistentBitMatrix)</li>
 <li><code>obicompactvec/src/intmatrix.rs</code> — <code>append_column</code> pour PersistentCompactIntMatrix</li>
 <li><code>obicompactvec/src/bitmatrix.rs</code> — <code>append_column</code> pour PersistentBitMatrix</li>
 </ul>

DevDoc/implementation/merge/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1034,6 +1034,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1157,6 +1168,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1542,6 +1609,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1739,6 +1817,14 @@ spectrums/
   &lt;label&gt;.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 </code></pre></div>
+<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
+<h2 id="known-issue-not-yet-fixed-2026-08-28">Known issue (not yet fixed, 2026-08-28)</h2>
+<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank &lt; self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
+<ul>
+<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
+<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
+</ul>
+<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
 
 
 

DevDoc/implementation/merge_parallelism/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1152,6 +1152,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/mphf.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,8 +1270,8 @@
 <h1 id="coverage-implementationmphfmd">Coverage: implementation/mphf.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obilayeredmap/src/mphf_layer.rs</code> — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, <code>build()</code>, <code>build_exact_evidence()</code>, <code>build_approx_evidence()</code>, <code>build_evidence()</code></li>
-<li><code>obikpartitionner/src/index_layer.rs</code> — <code>build_index_layer()</code> avec passage de <code>block_bits</code></li>
+<li><code>obikindex/src/layer/mphf_layer.rs</code> — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, <code>build()</code>, <code>build_exact_evidence()</code>, <code>build_approx_evidence()</code>, <code>build_evidence()</code></li>
+<li><code>obikindex/src/partition/index_layer.rs</code> — <code>build_index_layer()</code> avec passage de <code>block_bits</code></li>
 </ul>
 <h2 id="notes">Notes</h2>
 <p>FORT RISQUE DE DÉRIVE. Changements récents :

DevDoc/implementation/mphf/index.html

@@ -946,14 +946,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1163,6 +1163,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1577,7 +1633,7 @@
 <h2 id="why-two-phases-are-needed">Why two phases are needed</h2>
 <p>Kmer indexing per partition proceeds in two phases. The separation is necessary because the exact number of surviving unique kmers is not known until after counting and filtering low-abundance kmers.</p>
 <h3 id="phase-1-provisional-mphf-kmer-spectrum">Phase 1 — provisional MPHF + kmer spectrum</h3>
-<p>Implemented in <code>obikpartitionner::KmerPartition::count_kmer()</code> → <code>count_partition()</code>.</p>
+<p>Implemented in <code>obikindex::partition::KmerPartition::count_kmer()</code> → <code>count_partition()</code>.</p>
 <ol>
 <li><strong>External sort</strong>: read the dereplicated superkmer file; extract the raw <code>u64</code> canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: <code>sorted_unique.bin</code> — a flat array of f0 distinct sorted <code>u64</code> values. Exact kmer count f0 is known at this point.</li>
 <li><strong>Build provisional MPHF</strong> (ptr_hash, same configuration as phase 2) over <code>sorted_unique.bin</code> using <code>new_from_par_iter</code>. Delete <code>sorted_unique.bin</code> immediately after. Persist to <code>mphf1.bin</code>.</li>
@@ -1667,7 +1723,7 @@
 <hr />
 <h2 id="multilayer-index-architecture">Multilayer index architecture</h2>
 <h3 id="layer-structure">Layer structure</h3>
-<p>Each layer is a self-contained unit. See <a href="../obilayeredmap/">obilayeredmap</a> for the full on-disk layout. The MPHF-relevant files are:</p>
+<p>Each layer is a self-contained unit. See <a href="../layer_tier/">obikindex::layer</a> for the full on-disk layout. The MPHF-relevant files are:</p>
 <div class="highlight"><pre><span></span><code>layer_i/
   unitigs.bin      — packed 2-bit nucleotide sequences (kmer evidence source)
   unitigs.bin.idx  — random-access block index (block_bits controls granularity)
@@ -1702,7 +1758,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
     Uses open_sequential(); never writes .idx
 </code></pre></div>
 <p>There is no <code>build_evidence</code> dispatch wrapper. Callers choose the appropriate post-hoc build directly.</p>
-<p>In <code>obikpartitionner</code>, <code>build_index_layer</code> receives <code>block_bits: u8</code> from <code>IndexConfig::block_bits</code> and forwards it directly to <code>Layer::build</code> and <code>Layer::build_approx_evidence</code>.</p>
+<p>In <code>obikindex::partition</code>, <code>build_index_layer</code> receives <code>block_bits: u8</code> from <code>IndexConfig::block_bits</code> and forwards it directly to <code>Layer::build</code> and <code>Layer::build_approx_evidence</code>.</p>
 <h3 id="membership-verification">Membership verification</h3>
 <p>ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry:</p>
 <ul>

DevDoc/implementation/obicompactvec/index.html

@@ -320,11 +320,11 @@
 </a>
 </li>
 <li class="md-nav__item">
-<a class="md-nav__link" href="../obilayeredmap/">
+<a class="md-nav__link" href="../layer_tier/">
 <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -412,6 +412,30 @@
   
 
     
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../benchmark_query_testing/">
+<span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+</a>
+</li>
+<li class="md-nav__item">
+<a class="md-nav__link" href="../partition_layer_cache/">
+<span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
   </span>
 </a>
 </li>
@@ -1144,29 +1168,41 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
 </code></pre></div>
 <hr/>
 <h2 id="matrix-types">Matrix types</h2>
-<p>Four matrix types, two encodings × two formats:</p>
+<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
 <table>
 <thead>
 <tr>
-<th></th>
-<th>Columnar format</th>
-<th>Packed format</th>
+<th>Variant</th>
+<th>Storage</th>
+<th>When</th>
 </tr>
 </thead>
 <tbody>
 <tr>
-<td><strong>Bit</strong></td>
-<td><code>PersistentBitMatrix</code> (Columnar variant)</td>
-<td><code>PersistentBitMatrix</code> (Packed variant)</td>
+<td><code>Columnar</code></td>
+<td>one <code>.pbiv</code>/<code>.pciv</code> file per column + <code>meta.json</code></td>
+<td>build-time default (<code>*Builder::new</code>)</td>
 </tr>
 <tr>
-<td><strong>Int</strong></td>
-<td><code>PersistentCompactIntMatrix</code> (Columnar variant)</td>
-<td><code>PersistentCompactIntMatrix</code> (Packed variant)</td>
+<td><code>Packed</code></td>
+<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
+<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
+</tr>
+<tr>
+<td><code>Sparse</code></td>
+<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
+<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
+</tr>
+<tr>
+<td><code>Implicit</code> (bit only)</td>
+<td>no file at all</td>
+<td>mono-genome presence layers — <code>n_cols</code> is always reported as <code>1</code>, every value is <code>true</code></td>
 </tr>
 </tbody>
 </table>
-<p>Both matrix types are enums (<code>Columnar</code> / <code>Packed</code> / <code>Implicit</code> for bit) behind a transparent API. <code>col_view(c)</code> returns the appropriate view directly:</p>
+<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
+<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
+<p><code>col_view(c)</code> returns the appropriate view directly:</p>
 <div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
 <span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o">&lt;'</span><span class="nb">_</span><span class="o">&gt;</span>
 

DevDoc/implementation/obipipeline.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1219,7 +1275,7 @@
 </ul>
 <h2 id="notes">Notes</h2>
 <p>Document stable (librairie générique, peu de risque de dérive).
-Vérifier si <code>obipipeline</code> est toujours utilisé dans la phase scatter de <code>obikpartitionner</code>
+Vérifier si <code>obipipeline</code> est toujours utilisé dans la phase scatter de <code>obikindex::partition</code>
 ou s'il a été remplacé par Rayon dans certains chemins.</p>
 
 

DevDoc/implementation/obipipeline/index.html

@@ -896,14 +896,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1113,6 +1113,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/obitaxonomy/index.html

@@ -12,7 +12,7 @@
         <link rel="prev" href="../select/">
       
       
-        <link rel="next" href="../../architecture/sequences/invariant/">
+        <link rel="next" href="../benchmark_query_testing/">
       
       
         
@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1125,6 +1125,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_bit_vec.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_bit_vec/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1225,6 +1225,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_compact_int_vec.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/persistent_compact_int_vec/index.html

@@ -9,7 +9,7 @@
       
       
       
-        <link rel="prev" href="../obilayeredmap/">
+        <link rel="prev" href="../layer_tier/">
       
       
         <link rel="next" href="../persistent_bit_vec/">
@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1208,6 +1208,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/pipeline.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,12 +1270,12 @@
 <h1 id="coverage-implementationpipelinemd">Coverage: implementation/pipeline.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obikpartitionner/src/partition.rs</code> — estimation des paramètres (phase 0)</li>
+<li><code>obikindex/src/partition/partition.rs</code> — estimation des paramètres (phase 0)</li>
 <li><code>obiskbuilder/src/iter.rs</code> — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)</li>
-<li><code>obikpartitionner/src/filter.rs</code> — déduplication bucket-sort (phase 2)</li>
-<li><code>obikpartitionner/src/kmer_sort.rs</code> — tri externe + agrégation de comptages (phase 3)</li>
+<li><code>obikindex/src/partition/filter.rs</code> — déduplication bucket-sort (phase 2)</li>
+<li><code>obikindex/src/partition/kmer_sort.rs</code> — tri externe + agrégation de comptages (phase 3)</li>
 <li><code>obidebruinj/src/debruijn.rs</code> — graphe De Bruijn, extraction des unitigs (phase 5)</li>
-<li><code>obikpartitionner/src/index_layer.rs</code> — construction MPHF + évidence (phase 6), paramètre <code>block_bits</code></li>
+<li><code>obikindex/src/partition/index_layer.rs</code> — construction MPHF + évidence (phase 6), paramètre <code>block_bits</code></li>
 <li><code>obikindex/src/index.rs</code> — <code>build_layers()</code>, <code>dereplicate_and_count()</code></li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/implementation/pipeline/index.html

@@ -857,14 +857,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1074,6 +1074,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1546,7 +1602,7 @@ branching / dead-end → unitig start or end
       counts/                ← PersistentCompactIntMatrix (with_counts only)
 </code></pre></div>
 <p><strong>Cleanup:</strong> unless <code>--keep-intermediate</code> is set, <code>remove_build_artifacts</code> deletes <code>dereplicated.skmer.zst</code>, <code>mphf1.bin</code>, and <code>counts1.bin</code> after all partitions are indexed.</p>
-<p>See <a href="../obilayeredmap/">obilayeredmap</a> and <a href="../mphf/">MPHF selection</a> for data structure details.</p>
+<p>See <a href="../layer_tier/">obikindex::layer</a> and <a href="../mphf/">MPHF selection</a> for data structure details.</p>
 <p><strong>Query path (exact evidence):</strong></p>
 <div class="highlight"><pre><span></span><code>query kmer q
   → canonical_minimizer(q) → hash → PART → part_XXXXX/

DevDoc/implementation/select/index.html

@@ -733,14 +733,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1119,17 +1119,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1157,6 +1146,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1196,6 +1202,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1582,17 +1644,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1620,6 +1671,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1672,23 +1740,32 @@ are preserved unchanged; only the data matrices are rewritten.</p>
 <hr />
 <h2 id="synopsis">Synopsis</h2>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>&lt;input-index&gt;
-<span class="w">        </span><span class="o">{</span><span class="w"> </span>--output<span class="w"> </span>&lt;dir&gt;<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="w"> </span><span class="o">}</span>
+<span class="w">        </span>--output<span class="w"> </span>&lt;dir&gt;
 <span class="w">        </span><span class="o">[</span>--group<span class="w">    </span>&lt;name&gt;:&lt;pred&gt;<span class="w">  </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--group-op<span class="w"> </span>&lt;name&gt;:&lt;op&gt;<span class="w">    </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-by<span class="w"> </span>&lt;key&gt;<span class="w">          </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-op<span class="w"> </span>&lt;op&gt;<span class="w">           </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--select<span class="w">   </span>&lt;col1,col2,...&gt;<span class="w">    </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--presence-threshold<span class="w"> </span>&lt;N&gt;<span class="w">      </span><span class="o">]</span>
+<span class="w">        </span><span class="o">[</span>--force-copy<span class="w">                  </span><span class="o">]</span>
 </code></pre></div>
 <hr />
 <h2 id="output-destination">Output destination</h2>
-<p>Exactly one of <code>--output</code> or <code>--in-place</code> must be specified.</p>
-<p><strong><code>--output &lt;dir&gt;</code></strong> — writes a new index to <code>&lt;dir&gt;</code>. The source index is
-unchanged. The MPHF and unitig files are copied; only the data matrices are
-rewritten with the new column layout.</p>
-<p><strong><code>--in-place</code></strong> — rewrites the data matrices of the source index directly.
-Removed or replaced columns are lost. The operation writes to temporary files
-first, then renames atomically, so an interrupted run leaves the index intact.</p>
+<p><code>--output &lt;dir&gt;</code> is required — <code>select</code> always writes a new index; there is no
+<code>--in-place</code> mode (2026-08-28: never implemented, removed from the design). The
+source index is unchanged.</p>
+<p>Each layer's kmer-identity files (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/
+<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code>) are never rewritten by a
+column projection/aggregation, so they are hard-linked into the output rather
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); <code>--force-copy</code>
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+<code>select</code> itself would affect the other). Only the <code>presence</code>/<code>counts</code>
+subdirectory is ever a genuinely new, independent file.</p>
+<p>To replace an index with a selected version of itself, select to a temporary
+directory and swap it in (<code>rm -rf INDEX &amp;&amp; mv INDEX.tmp INDEX</code>) — the case
+<code>--in-place</code> used to cover.</p>
 <hr />
 <h2 id="defining-output-columns">Defining output columns</h2>
 <h3 id="named-groups-group">Named groups — <code>--group</code></h3>
@@ -1852,9 +1929,6 @@ is &gt; N.</p>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>out<span class="w"> </span><span class="se">\</span>
 <span class="w">  </span>--select<span class="w"> </span><span class="s2">&quot;Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01&quot;</span>
 </code></pre></div>
-<h3 id="in-place-keep-only-group-a">In-place: keep only group A</h3>
-<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--in-place<span class="w"> </span>--group<span class="w"> </span><span class="s2">&quot;A:group=A&quot;</span><span class="w"> </span>--select<span class="w"> </span><span class="s2">&quot;A&quot;</span>
-</code></pre></div>
 <h3 id="compose-with-filter">Compose with filter</h3>
 <div class="highlight"><pre><span></span><code><span class="c1"># Step 1: keep only B. nana-specific k-mers</span>
 obikmer<span class="w"> </span>filter<span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>filtered<span class="w"> </span><span class="se">\</span>
@@ -1865,32 +1939,46 @@ obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </s
 </code></pre></div>
 <hr />
 <h2 id="implementation-notes">Implementation notes</h2>
-<p><code>select</code> does not rebuild the MPHF. The 256 partitions are processed in parallel
-(rayon), each writing its output independently; results require no synchronisation
-because every partition owns a distinct set of files.</p>
-<p>For each layer in each partition:</p>
+<p><code>select</code> does not rebuild the MPHF. Every partition is processed independently
+(<code>PartitionRunner</code>), each writing its own output layers; no cross-partition
+synchronisation is needed.</p>
+<p>For each layer in each partition (<code>obikselect::select_layer::select_partition</code>):</p>
 <ol>
-<li>The slot count <code>n</code> is read by opening the source data matrix.</li>
-<li>A new data matrix is built with M columns (M = number of output columns).</li>
-<li>For each slot <code>s</code> in <code>0..n</code>:</li>
-<li><code>old_row = matrix.fill_row(s)</code> — reads the original <code>N</code>-column row without allocating.</li>
-<li>For each output column <code>j</code>:<ul>
-<li><code>new_row[j] = aggregate(op, old_row[group_indices])</code>.</li>
-<li>Pass-through columns are represented as single-element groups with the
-   default operator (<code>any</code> for presence, <code>sum</code> for count) — same code path.</li>
-</ul>
-</li>
-<li>The new row is written slot by slot into each column builder.</li>
-<li>All plain files in the source layer directory (<code>mphf.bin</code>, <code>unitigs.bin</code>,
-   evidence files, <code>layer_meta.json</code>) are copied verbatim; only the <code>presence/</code>
-   or <code>counts/</code> subdirectory is rewritten.</li>
+<li><code>copy_layer_files</code> hard-links the source layer's kmer-identity files
+   (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/
+   <code>fingerprint.bin</code>/<code>layer_meta.json</code>) into the destination — never a real
+   copy unless linking fails or <code>--force-copy</code> is given.</li>
+<li>A new data matrix is built with M columns (M = number of output columns),
+   under a fresh <code>presence/</code>/<code>counts/</code> subdirectory (never touching the
+   source's own).</li>
+<li><strong>Presence source (2026-08-28: <code>batch_presence_counts</code>)</strong>: one shared pass
+   over the source bit matrix computes every output group's presence count at
+   once — row-major native for a <code>Sparse</code> source (<code>for_each_genome_in_row</code>,
+   which has no column representation to read a <code>col_view</code> from at all — the
+   reason this replaced the old per-group loop, not just an optimisation of
+   it), deduplicated column-major (one <code>col_view</code> per <em>distinct referenced
+   column</em>, not per group) for <code>Columnar</code>/<code>Packed</code>. Every <code>AggOp</code> for a bit
+   matrix is then a cheap derivation of that one count vector (<code>sum</code> = the
+   count itself, <code>any</code>/<code>max</code> = <code>count ≥ 1</code>, <code>all</code>/<code>min</code> = <code>count == group
+   size</code>, <code>none</code> = <code>count == 0</code>) — see
+   <code>obikselect::select_layer::agg_result_from_count</code>.</li>
+<li><strong>Count source</strong>: unchanged, one <code>col_view</code>-driven pass per output column
+   via <code>MatrixGroupOps</code> — <code>sum</code>/<code>min</code>/<code>max</code> are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.</li>
 <li><code>index.meta</code> is rewritten with the new genome list and updated <code>with_counts</code>.</li>
 </ol>
-<p><strong><code>--in-place</code> write strategy:</strong> new data is written to a temporary sibling
-directory (<code>presence_new/</code> or <code>counts_new/</code>); on success the old directory is
-removed and the temporary one is renamed into place. An interrupted run leaves
-at most one stale <code>*_new/</code> directory; the original data is intact until the
-rename step.</p>
+<h3 id="known-gap-not-yet-fixed-2026-08-28">Known gap (not yet fixed, 2026-08-28)</h3>
+<p>Step 4 above still panics (<code>col_view() not available on Sparse
+PersistentCompactIntMatrix</code>) if the source is a <strong>count</strong> index packed
+sparse — <code>batch_presence_counts</code>' row-major treatment was only ported to the
+bit-matrix (<code>Presence</code>) case, since that was the one actually blocking a real
+benchmark run. <code>select</code>/<code>filter</code> on a sparse-packed count index still hits
+this; the fix would follow the same shape (a <code>PersistentSparseCompactIntMatrix</code>
+row-major decode, analogous to <code>for_each_genome_in_row</code>), just not done. Since
+<code>obisys::numa::runner::PartitionRunner</code>'s panic-propagation fix (see
+<code>architecture/numa_partition_runner.md</code>), this at least fails fast (process
+panic, exit 101) instead of hanging.</p>
 
 
 

DevDoc/implementation/storage.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1216,7 +1272,7 @@
 <ul>
 <li><code>obikindex/src/meta.rs</code> — IndexMeta, IndexConfig (version, config, genomes)</li>
 <li><code>obikindex/src/index.rs</code> — layout sur disque : partitions/, index.meta</li>
-<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta (evidence kind), PartitionMeta (n_layers)</li>
+<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta (evidence kind), PartitionMeta (n_layers)</li>
 <li><code>obiskio/src/unitig_index.rs</code> — fichiers unitigs.bin + unitigs.bin.idx</li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/implementation/storage/index.html

@@ -929,14 +929,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1146,6 +1146,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/superkmer.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/superkmer/index.html

@@ -835,14 +835,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1052,6 +1052,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/implementation/unitig_evidence.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1215,7 +1271,7 @@
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
 <li><code>obiskio/src/unitig_index.rs</code> — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()</li>
-<li><code>obilayeredmap/src/evidence.rs</code> — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter</li>
+<li><code>obikindex/src/layer/evidence.rs</code> — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter</li>
 <li><code>obidebruinj/src/debruijn.rs</code> — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK</li>
 </ul>
 <h2 id="notes">Notes</h2>

DevDoc/implementation/unitig_evidence/index.html

@@ -957,14 +957,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../obilayeredmap/" class="md-nav__link">
+      <a href="../layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1174,6 +1174,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/index.html

@@ -831,14 +831,14 @@
   
   
     <li class="md-nav__item">
-      <a href="implementation/obilayeredmap/" class="md-nav__link">
+      <a href="implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1048,6 +1048,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/index.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/installation/index.html

@@ -917,14 +917,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1134,6 +1134,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/kmers.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/kmers/index.html

@@ -830,14 +830,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1047,6 +1047,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/encoding.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/encoding/index.html

@@ -802,14 +802,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1019,6 +1019,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/entropy.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/entropy/index.html

@@ -868,14 +868,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1085,6 +1085,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/evolutionary_distances/index.html

@@ -909,6 +909,56 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        --distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#snp-distance-catalog" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        snp-* distance catalog
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Exact formulas (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Output format: PHYLIP-relaxed by default for the distance matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1244,14 +1294,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1461,6 +1511,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -2139,6 +2245,56 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        --distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#snp-distance-catalog" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        snp-* distance catalog
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Exact formulas (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Output format: PHYLIP-relaxed by default for the distance matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -3554,7 +3710,7 @@ diagonal entry directly, at no extra lookup cost.</p>
 <p>Mirror <code>distance.rs</code>: open the presence or count store per partition. But
 instead of a per-partition <code>partial_*</code>, run the sequential source sweep:</p>
 <div class="highlight"><pre><span></span><code>for p in 0..n_partitions:                       # OUTER — sequential
-    open source partition p&#39;s layers (QueryLayer-style, obikpartitionner)
+    open source partition p&#39;s layers (QueryLayer-style, obikindex::partition)
     enumerate distinct canonical k-mers of p (one per MPHF slot) with their
         presence/count vectors                   # column-major, as query stage 2
     par_iter over these source k-mers:           # INNER — rayon, thread-local tally
@@ -3577,7 +3733,7 @@ instead of a per-partition <code>partial_*</code>, run the sequential source swe
     merge thread-local tallies into global SnpTally
 </code></pre></div>
 <p>The inner lookup is precisely <code>QueryLayer::find_slot</code> +
-<code>col_value(g, slot)</code> (<code>obikpartitionner/src/query_layer.rs</code>) — reuse or factor
+<code>col_value(g, slot)</code> (<code>obikindex/src/partition/query_layer.rs</code>) — reuse or factor
 out that path rather than reimplementing MPHF access. Enumerating "all distinct
 k-mers of a partition with their vectors" is the <code>dump</code>/<code>query</code> stage-2
 column-major scan already implemented in <code>dump_layer.rs</code> /
@@ -4205,6 +4361,249 @@ among the survivors) — a single extra pass is sufficient.</p>
 <code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
 from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
 workspace <code>cargo test</code> green.</p>
+<h2 id="-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28"><code>--distance</code> unification: SNP corrections as first-class metrics (implemented, 2026-08-28)</h2>
+<p><strong>Implemented.</strong> <code>--metric</code> (renamed <code>--distance</code> — several of
+its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
+<code>--metric</code> was a misnomer) gains a family of <code>snp-*</code> values computed from the
+central-position SNP pipeline, routed internally to the sibling-annex
+machinery (<code>PairwiseTally</code>, <code>obikphylo::siblings::algorithms::pairwise</code>)
+instead of <code>cache.distance(...)</code>'s existing per-layer traversal — a different
+code path behind the same CLI surface, not just another branch of one
+formula function.</p>
+<p><strong>Why unify at the CLI level despite the implementation split</strong>: phylogenetically
+a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
+to how the matrix was produced, so exposing it as a special-cased subcommand
+instead of a <code>--distance</code> value would misrepresent its role. The
+implementation divergence (sibling-annex-based vs. plain index scan) is real
+but belongs at the routing layer, invisible to the CLI's own vocabulary.</p>
+<p><strong><code>--subsample</code> becomes optional for <code>snp-*</code> distances</strong> (it stays mandatory
+for <code>--sankoff</code>/<code>--pseudo-alignment</code>, unrelated commands): absent means
+exhaustive, achieved for free by reusing <code>sample_index</code>'s existing
+proportional-per-layer-quota mechanism with <code>n</code> set to the index-wide total
+non-monomorphic-minorant count (already available from the sibling-annex
+stats) — every layer's quota then equals its own full count, giving Bernoulli
+<code>p = 1</code> everywhere, i.e. every eligible family is drawn. No second,
+exhaustive-only driver needed. Present means sampled, exactly as <code>--sankoff</code>
+already behaves.</p>
+<p><strong>One shared tally, many derived formulas.</strong> <code>PairwiseTally</code>'s <code>subst[4][4]</code>
+per-pair substitution counts (plus marginal base frequencies derived from it)
+are the sufficient statistic for every closed-form correction below — each
+is a small pure function <code>PairwiseTally -&gt; Array2&lt;f64&gt;</code>, at the same level as
+the already-implemented <code>raw_snp_distance</code>/<code>base_pair_tally</code>/
+<code>cardinality_tally</code>. No new full scan per formula, whether the tally itself
+was built exhaustively or from a subsample.</p>
+<p><strong><code>--raw-snp-counts</code> stays a separate, unrelated flag</strong> — same underlying
+tally, but a diagnostic (<code>n_snp</code>/<code>n_shared</code>/<code>n_eligible</code> per genome pair, one
+row per pair) rather than a distance value, and its long-table shape doesn't
+fold into a single N×N matrix the way a distance does. No change to its
+existing CSV format.</p>
+<h3 id="snp-distance-catalog"><code>snp-*</code> distance catalog</h3>
+<p>All closed-form (method-of-moments / direct formula), none requiring
+per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
+<em>tree</em>-ML usage but not its <em>pairwise</em> estimator, which is closed-form like
+F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
+<table>
+<thead>
+<tr>
+<th>value</th>
+<th>corrects for</th>
+<th>inputs beyond raw counts</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>snp-raw</code></td>
+<td>nothing (uncorrected p-distance)</td>
+<td>—</td>
+</tr>
+<tr>
+<td><code>snp-jc</code> (Jukes-Cantor, JC69)</td>
+<td>multiple substitutions per site</td>
+<td>—</td>
+</tr>
+<tr>
+<td><code>snp-k2p</code> (Kimura 2-parameter, K80)</td>
+<td>+ transition/transversion rate bias</td>
+<td>ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-k81</code> (Kimura 3-parameter, K3ST)</td>
+<td>+ splits transversions into 2 categories</td>
+<td>ts/tv split, by category</td>
+</tr>
+<tr>
+<td><code>snp-f81</code> (Felsenstein 81)</td>
+<td>+ unequal base frequencies (no ts/tv split)</td>
+<td>empirical base freqs</td>
+</tr>
+<tr>
+<td><code>snp-tajima-nei</code> (Tajima-Nei 1984)</td>
+<td>same goal as F81 (equal-input model), different formula, better small-sample behavior</td>
+<td>empirical base freqs</td>
+</tr>
+<tr>
+<td><code>snp-t92</code> (Tamura 3-parameter)</td>
+<td>K2P + GC-content bias</td>
+<td>ts/tv split, GC content</td>
+</tr>
+<tr>
+<td><code>snp-f84</code> (Felsenstein 84)</td>
+<td>full empirical base freqs + single ts/tv rate</td>
+<td>empirical base freqs, ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-hky85</code> (Hasegawa-Kishino-Yano, pairwise estimator)</td>
+<td>same inputs as F84, different formula</td>
+<td>empirical base freqs, ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-tn93</code> (Tamura-Nei)</td>
+<td>full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate</td>
+<td>empirical base freqs, purine-ts/pyrimidine-ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-logdet</code> (LogDet / paralinear)</td>
+<td>no shared-model or stationarity assumption at all — general divergence-matrix determinant</td>
+<td>full empirical 4×4 divergence matrix (already <code>subst[4][4]</code>)</td>
+</tr>
+<tr>
+<td><code>snp-tv</code> (transversions-only p-distance)</td>
+<td>diagnostic/deep-divergence variant — drops transitions entirely (they saturate first)</td>
+<td>tv-only counts</td>
+</tr>
+</tbody>
+</table>
+<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
+<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
+nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
+formula as the base correction, weighted by a shape parameter <code>α</code> supplied
+by the user (<code>--gamma-shape &lt;alpha&gt;</code>), not estimated by ML. A modifier on
+existing values, not a separate enum arm per distance.</p>
+<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
+<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
+"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
+<code>snp-logdet</code> are catalogued above but <strong>not implemented</strong>: <code>snp-logdet</code>
+needs the true <em>directional</em> per-pair base co-occurrence matrix
+(<code>PairwiseTally</code> only keeps the symmetrised substitution counts
+<code>BasePairTally</code> itself wants — see <code>snp_distance.rs</code>'s own module docs for
+why that loses exactly the compositional-asymmetry information LogDet
+exists to detect), <code>snp-tajima-nei</code> needs each genome's <em>own</em> base
+composition (not the pair-pooled estimate the formulas below use), and
+<code>snp-f84</code>/<code>snp-hky85</code> had no formula independently verified against a
+primary source at implementation time (unlike every formula below, checked
+line-by-line against <a href="https://github.com/emmanuelparadis/ape">ape</a>'s own
+<code>src/dist_dna.c</code>, not re-derived from memory). Adding any of these later is
+a new function in <code>obikphylo::siblings::algorithms::snp_distance</code>, plus for
+<code>snp-logdet</code>/<code>snp-tajima-nei</code> a new field on <code>PairStats</code>/a per-genome
+accumulator — not an architecture change.</p>
+<h3 id="exact-formulas-implemented-2026-08-28">Exact formulas (implemented, 2026-08-28)</h3>
+<p>Sufficient statistic, per genome pair <code>(i, j)</code>, from
+<code>PairwiseTally::categories</code>/<code>PairwiseTally::base_freq</code> (base order always
+<code>0=A, 1=C, 2=G, 3=T</code>, matching <code>FamilyMask</code>/<code>STATE_SYMBOL</code>):</p>
+<ul>
+<li><span class="arithmatex">\(n_{ts1}\)</span>: A↔G substitutions (purine transitions), <span class="arithmatex">\(n_{ts2}\)</span>: C↔T
+  (pyrimidine transitions)</li>
+<li><span class="arithmatex">\(n_{tv1}\)</span>: A↔C and G↔T substitutions, <span class="arithmatex">\(n_{tv2}\)</span>: A↔T and C↔G
+  (Kimura's two transversion categories)</li>
+<li><span class="arithmatex">\(n_{shared}\)</span>: loci where both genomes agree</li>
+<li><span class="arithmatex">\(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\)</span> (total eligible
+  loci for the pair)</li>
+<li><span class="arithmatex">\(\pi_A, \pi_C, \pi_G, \pi_T\)</span>: pair-pooled base frequencies,
+  <span class="arithmatex">\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)</span>
+  (both genomes' calls at this pair's eligible loci, pooled — Nei &amp; Kumar's
+  standard pairwise estimator, not a whole-index average)</li>
+</ul>
+<p>Derived proportions used below:</p>
+<div class="arithmatex">\[
+p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
+P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
+Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
+Q_1 = \frac{n_{tv1}}{L}, \quad
+Q_2 = \frac{n_{tv2}}{L}, \quad
+P_1 = \frac{n_{ts1}}{L}, \quad
+P_2 = \frac{n_{ts2}}{L}
+\]</div>
+<p>Every formula below was checked term-by-term against <code>ape</code>'s own
+<code>src/dist_dna.c</code> (not re-derived from memory) before being ported to
+<code>obikphylo::siblings::algorithms::snp_distance</code>.</p>
+<p><strong><code>snp-raw</code></strong> — uncorrected p-distance:</p>
+<div class="arithmatex">\[
+d_{raw} = p
+\]</div>
+<p><strong><code>snp-tv</code></strong> — transversions-only p-distance (deliberately uncorrected —
+dropping transitions, which saturate first, <em>is</em> the correction):</p>
+<div class="arithmatex">\[
+d_{tv} = Q
+\]</div>
+<p><strong><code>snp-jc</code></strong> (Jukes-Cantor, JC69):</p>
+<div class="arithmatex">\[
+d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
+\]</div>
+<p><strong><code>snp-k2p</code></strong> (Kimura 2-parameter, K80), with <span class="arithmatex">\(a_1 = 1-2P-Q\)</span>, <span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
+<div class="arithmatex">\[
+d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
+\]</div>
+<p><strong><code>snp-k81</code></strong> (Kimura 3-parameter, K3ST), with <span class="arithmatex">\(a_1 = 1-2P-2Q_1\)</span>,
+<span class="arithmatex">\(a_2 = 1-2P-2Q_2\)</span>, <span class="arithmatex">\(a_3 = 1-2Q_1-2Q_2\)</span>:</p>
+<div class="arithmatex">\[
+d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
+\]</div>
+<p><strong><code>snp-f81</code></strong> (Felsenstein 81), with <span class="arithmatex">\(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\)</span>:</p>
+<div class="arithmatex">\[
+d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
+\]</div>
+<p><strong><code>snp-t92</code></strong> (Tamura 3-parameter), with GC content
+<span class="arithmatex">\(g = \pi_C+\pi_G\)</span>, <span class="arithmatex">\(w = 2g(1-g)\)</span>, <span class="arithmatex">\(a_1 = 1 - \dfrac{P}{w} - Q\)</span>,
+<span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
+<div class="arithmatex">\[
+d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
+\]</div>
+<p><strong><code>snp-tn93</code></strong> (Tamura-Nei), with purine/pyrimidine pooled frequencies
+<span class="arithmatex">\(g_R = \pi_A+\pi_G\)</span>, <span class="arithmatex">\(g_Y = \pi_C+\pi_T\)</span>, and</p>
+<div class="arithmatex">\[
+k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
+k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
+k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
+\]</div>
+<div class="arithmatex">\[
+w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
+w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
+w_3 = 1 - \frac{Q}{2g_R g_Y}
+\]</div>
+<div class="arithmatex">\[
+d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
+\]</div>
+<p><strong><code>+Γ</code> gamma correction</strong> (Jin &amp; Nei 1990): every formula above is a
+weighted sum of <span class="arithmatex">\(-\ln(x)\)</span> terms; the gamma-corrected version replaces
+each such term with the same weight applied to
+<span class="arithmatex">\(\alpha\left(x^{-1/\alpha} - 1\right)\)</span> instead — the standard mechanical
+substitution (as <span class="arithmatex">\(\alpha \to \infty\)</span>, this expression → <span class="arithmatex">\(-\ln(x)\)</span>,
+recovering the uncorrected formula exactly). E.g. for JC:</p>
+<div class="arithmatex">\[
+d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
+\]</div>
+<p>Verified term-by-term against <code>ape</code>'s own gamma branches for JC69/K80/F81
+(including K80's two-term form — algebraically identical to the generic
+substitution applied to <code>snp-k2p</code>'s own <span class="arithmatex">\(a_1\)</span>/<span class="arithmatex">\(a_2\)</span> terms above, checked
+both symbolically and numerically before simplifying the implementation to
+share one <code>corrected_log</code> helper across every model rather than
+special-casing K80). K81/T92/TN93's gamma branches follow the same
+mechanical substitution but weren't independently checked against an
+<code>ape</code>-equivalent reference for those three specifically — flagged here, not
+silently assumed correct.</p>
+<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
+<p><strong>Implemented.</strong> The primary distance-matrix output
+(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
+default</strong> (widely read by external NJ tools — PHYLIP <code>neighbor</code>, FastME,
+T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
+label truncation, since genome labels here routinely exceed it), a <code>--csv</code>
+flag opts back into the current CSV format, PHYLIP-strict is a possible
+future addition (not now). This changes the <em>default</em> output of every
+existing <code>--distance</code> value (jaccard, hamming, bray-curtis, ...), not just
+the new <code>snp-*</code> ones — accepted explicitly (pre-release, single developer
+user, no external consumers to break). Scoped to the distance matrix only:
+<code>--shared-kmers</code> and <code>--raw-snp-counts</code> are counts, not distances, and keep
+their existing CSV-only format.</p>
 <h2 id="references">References</h2>
 <p>The Mash mutation-rate model this discussion contrasts with:
 (Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>

DevDoc/theory/indexing.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       
@@ -1214,8 +1270,8 @@
 <h1 id="coverage-theoryindexingmd">Coverage: theory/indexing.md</h1>
 <h2 id="code-couvert">Code couvert</h2>
 <ul>
-<li><code>obikpartitionner/src/partition.rs</code> — routage par hash de minimiseur, choix des paramètres</li>
-<li><code>obikpartitionner/src/lib.rs</code> — structure KmerPartition, nombre de partitions</li>
+<li><code>obikindex/src/partition/partition.rs</code> — routage par hash de minimiseur, choix des paramètres</li>
+<li><code>obikindex/src/partition/mod.rs</code> — structure KmerPartition, nombre de partitions</li>
 </ul>
 <h2 id="notes">Notes</h2>
 <p>Vérifier que la doc mentionne bien que le nombre de partitions est une puissance de 2

DevDoc/theory/indexing/index.html

@@ -802,14 +802,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1019,6 +1019,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/minimizer.refs/index.html

@@ -727,14 +727,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -944,6 +944,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDoc/theory/minimizer/index.html

@@ -846,14 +846,14 @@
   
   
     <li class="md-nav__item">
-      <a href="../../implementation/obilayeredmap/" class="md-nav__link">
+      <a href="../../implementation/layer_tier/" class="md-nav__link">
         
   
   
   <span class="md-ellipsis">
     
   
-    obilayeredmap crate
+    obikindex layer tier
   
 
     
@@ -1063,6 +1063,62 @@
 
               
             
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Benchmark: query-path testing
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
+              
+                
+  
+  
+  
+  
+    <li class="md-nav__item">
+      <a href="../../implementation/partition_layer_cache/" class="md-nav__link">
+        
+  
+  
+  <span class="md-ellipsis">
+    
+  
+    Partition and layer caching (discussion)
+  
+
+    
+  </span>
+  
+  
+
+      </a>
+    </li>
+  
+
+              
+            
           </ul>
         </nav>
       

DevDocMD/architecture/index_architecture.refs.md

@@ -3,10 +3,10 @@
 
 ## Code couvert
 
-- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix
-- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum
-- `obilayeredmap/src/map.rs` — LayeredMap<D>
-- `obilayeredmap/src/meta.rs` — LayerMeta, PartitionMeta
+- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix
+- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum
+- `obikindex/src/layer/map.rs` — LayeredMap<D>
+- `obikindex/src/layer/meta.rs` — LayerMeta, PartitionMeta
 - `obikindex/src/meta.rs` — IndexConfig (kmer_size, n_bits, with_counts, evidence, block_bits), IndexMeta
 - `obikindex/src/index.rs` — KmerIndex, build_layers
 - `obicompactvec/src/` — PersistentCompactIntMatrix, PersistentBitMatrix (DataStore implementations)

DevDocMD/architecture/numa_partition_runner.md

@@ -304,10 +304,29 @@ step, capped by that node's remaining dormant workers and by the run's total
 budget (`n_total`) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.
 
+## Panic propagation (fixed, 2026-08-28)
+
+A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
+— the controller's `while completed < n_total` loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept `event_rx` open. Discovered via `obikselect`
+panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
+the process hung indefinitely instead of erroring out.
+
+Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
+`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
+partition. The controller counts it toward `completed` (unblocking the loop)
+and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain `Err` (a panic means a bug, not a normal typed failure).
+
 ## Open questions
 
 - **Error handling**: `run` currently returns the first error; remaining errors
-  are dropped.  A `Vec<E>` return would give complete diagnostics.
+  are dropped.  A `Vec<E>` return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain `Err` results.)
 
 - **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
   (start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an

DevDocMD/architecture/numa_worker_pools.md

@@ -72,9 +72,9 @@ Each NUMA group runs its own independent adaptive pool. Workers are distributed
 |------|--------|
 | `obikindex/src/merge.rs` | Detect NUMA topology; build N `ThreadPool`s with pinned threads; assign each pre-spawned worker to a pool; wrap `merge_partition` in `pool.install()` |
 | `obikindex/src/merge.rs` | Replace `available_parallelism()` with per-NUMA core count for spawn criterion |
-| `obikpartitionner/src/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
+| `obikindex/src/partition/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
 | `obidebruinj/src/debruijn.rs` | No change — `par_iter` and `current_num_threads` are pool-context-aware |
-| `obikpartitionner/src/partition.rs` | No change — same reason |
+| `obikindex/src/partition/partition.rs` | No change — same reason |
 
 ## Platform guard
 

DevDocMD/architecture/query.md

@@ -16,7 +16,7 @@ Given a set of query sequences, determine for each sequence how many of its k-me
 
 ## Algorithm
 
-The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikpartitionner::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
+The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikindex::partition::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
 
 ```
 for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
@@ -118,11 +118,11 @@ The `-z` CLI option overrides the index metadata value. A higher z increases str
 
 ## Layer lookup: `MphfLayer::find`
 
-`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obilayeredmap](../implementation/obilayeredmap.md) for the full `find` / `find_strict` API.
+`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obikindex::layer](../implementation/layer_tier.md) for the full `find` / `find_strict` API.
 
 ### `QueryLayer` variant selection
 
-`QueryLayer::open` (`obikpartitionner/src/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
+`QueryLayer::open` (`obikindex/src/partition/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
 
 | Order | Condition | Variant | Data returned per k-mer |
 |---|---|---|---|
@@ -244,7 +244,7 @@ Fix direction: restructure `query`'s pipe with an initial `Flat` stage analogous
 `chunk_bytes = available_memory_bytes() / (n_workers * 16)` (`query.rs:407-414`) assumes a fixed ~8–16× overhead per raw input byte. But `KmerResults::new` (`query.rs:165-179`) allocates `data: Vec<u32>` sized `total_kmers_in_chunk × n_genomes` — dense, **for every k-mer position in the chunk, hit or not** — plus `win_min` and (with `--detail`) `cov`, same scaling. Real per-chunk memory is `O(n_genomes)`, not constant; the formula doesn't know `n_genomes` at all. This is the direct cause of the OOM kill on indexes with many reference genomes.
 
 **4. MPHF lookup and matrix-row fetch are fused, not staged.**
-`QueryLayer::find_into` (`obikpartitionner/src/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
+`QueryLayer::find_into` (`obikindex/src/partition/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
 
 **5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.**
 `QueryBatch::from_records` currently dereplicates at the *superkmer* level (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, `query.rs:112`). This misses redundancy between k-mers shared by *different* superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer *construction* (`SuperKmerIter`) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on `CanonicalKmer`, in the same pass: `HashMap<CanonicalKmer, Vec<(seq_idx, pos)>>`. This also means the MPHF `find` itself runs once per **distinct** k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.
@@ -285,7 +285,7 @@ None of this is implemented yet — parked here as a coherent roadmap while the
 
 ## Implementation plan
 
-Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikpartitionner`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
+Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikindex::partition`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
 
 Instrumentation is deliberately sequenced *before* the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.
 
@@ -295,7 +295,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 
 **Debug logging.** Every phase that changes an algorithmic choice (not phase 0, which *is* the logging) adds `tracing::debug!`/`trace!` at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF `find` calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured `debug!` per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.
 
-**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obilayeredmap`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikpartitionner` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
+**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obikindex::layer`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikindex::partition` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
 
 **Workflow (`jj`).** Work happens in a fresh `jj` commit, easy to abandon. `jj new` between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then `query_layer.rs` split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.
 
@@ -345,13 +345,13 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 - `obikmer/src/cmd/query.rs`:
   - Replace `QueryBatch::from_records`'s dedup map (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, current `query.rs:112`) with a per-partition `HashMap<CanonicalKmer, Vec<(seq_idx: u32, pos: u32)>>`, built in the same `SuperKmerIter` pass: superkmer construction and partition routing (`part_idx` from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each `CanonicalKmer` within a superkmer is inserted individually instead of the whole superkmer being the dedup key.
   - **Verified**: `CanonicalKmer` (`obikseq/src/kmer.rs:390`, `pub type CanonicalKmer = CanonicalKmerOf<KLen>`) — the underlying `CanonicalKmerOf<L>` derives `Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash` (`kmer.rs:269`). Usable as a `HashMap`/`HashSet` key as-is, no change needed.
-- `obikpartitionner/src/query_layer.rs`:
+- `obikindex/src/partition/query_layer.rs`:
   - Split `QueryLayer::find_into` (`query_layer.rs:48-67`) into two methods: `find_slot(&self, kmer: CanonicalKmer) -> Option<usize>` (MPHF only, no matrix touch) and keep `fill_row` as-is for phase 4 to call later.
   - Replace `query_partition_with`'s inner loop (`query_layer.rs:103-113`) with a version that, for each unique `CanonicalKmer`, calls `find_slot` across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records `(layer_idx, slot)`.
   - New return shape for the partition-level query, replacing today's `on_hit(sk_idx, kmer_idx, row)` callback: `HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>` (roadmap point 6) — built directly from the k-mer dedup map's `Vec<(seq_idx,pos)>` values, keyed by the resolved slot instead of the k-mer.
 - **This phase alone has no throughput benefit yet** (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.
 - Log, per chunk: total k-mer occurrences vs. unique `CanonicalKmer` count (the dedup ratio — the entire justification for this phase) and the resulting MPHF `find` call count. If the dedup ratio is close to `1.0` on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.
-- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikpartitionner/src/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
+- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikindex/src/partition/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
 
 ### Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred
 
@@ -359,7 +359,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 
 **What shipped:**
 - `obicompactvec`: the per-column accessors this phase needed **already existed** — `PersistentCompactIntMatrix::col_view(c)` and `PersistentBitMatrix::col_view(c)` are public, and `IntSliceView::get(slot)`/`BitSliceView::get(slot)` are public — the original plan underestimated how much of this plumbing the pairwise-distance code (`dump`/`select`/`stats`) had already required. The one real gap: `PersistentBitMatrix::col_view()` panics on the `Implicit` variant (the documented mono-genome fast path, `bitmatrix.rs`). Added `PersistentBitMatrix::get(c, slot) -> u32` (`bitmatrix.rs`), a non-panicking column-major point lookup that returns `1` for `Implicit` regardless of `c` — the smallest surface needed, not a new `col_get` API from scratch.
-- `obikpartitionner/src/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
+- `obikindex/src/partition/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
 - `obikmer/src/cmd/query.rs`: `KmerResults::set` (row-major, whole-row-at-once) replaced by `mark_found` (stage 1: flag a position as indexed, independent of any genome's value) and `set_one` (stage 2: write one genome's value at one position). `QueryStats` extended with `n_columns_scanned`/`n_col_get_calls`, logged per chunk.
 - Total `get()`-equivalent calls are unchanged from the row-major version (`n_hits × n_cols` in the worst case, confirmed by `n_col_get_calls` in the debug log) — the win is locality (sequential access within one layer's column at a time, across `mmap`'d regions, instead of jumping across all columns per hit), exactly as predicted.
 
@@ -367,7 +367,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
 Reading `obikindex/src/numa.rs`'s actual `run()` body (not just its doc comments) shows every call spawns a timer thread **plus one OS thread per worker slot on every NUMA node** (`std::thread::scope` + one `s.spawn()` per node per `max_workers`) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned **per call**. This is fine for its actual, established usage in this codebase (`merge.rs`, `index.rs`'s `build_layers`): one `PartitionRunner::new()` + one `run()` call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for `query`'s call pattern: `query_partition_with` runs once per `(chunk, partition)`, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
 The column-major loop in stage 2 is therefore a **plain sequential loop** for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) `rayon`'s already-warm global pool (`into_par_iter()`) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern `numa_worker_pools.md` warns about for a *different* workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one `mmap`'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so `PartitionRunner` is invoked once per whole `query` run (or per large batch of chunks) rather than per `(chunk, partition)`, amortising its spawn cost the way `merge`/`build_layers` do — a bigger structural change than this phase's scope.
 - Log (implemented): `QueryStats::n_columns_scanned`/`n_col_get_calls`, folded into the existing per-chunk `debug!("k-mer dedup + column-major fetch", ...)` line (`query.rs`) alongside phase 3's dedup counters.
-- **Unit tests**: extended `obikpartitionner/src/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
+- **Unit tests**: extended `obikindex/src/partition/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
 - **Validation performed**: full workspace build + `cargo test --workspace`, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same `kmer_count`/`kmer_strict_matches` on every record); (2) the existing 20-genome `benchmark/global_index_presence` index — runs correctly, `n_hits=0` for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the `Implicit`/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) **the critical correctness case**: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from `genomeA` matched **only** `genomeA` (`kmer_count` identical to the pre-dedup occurrence count, zero leakage into `genomeB`'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in `n_cols`, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
 - **Not yet done**: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-`n_genomes` layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.
 

DevDocMD/architecture/query.refs.md

@@ -4,7 +4,7 @@
 ## Code couvert
 
 - `obikmer/src/cmd/query.rs` — commande query, format de sortie
-- `obikpartitionner/src/query_layer.rs` — routage de la requête à travers les partitions
+- `obikindex/src/partition/query_layer.rs` — routage de la requête à travers les partitions
 - `obiread/src/lib.rs` — lecture des séquences d'entrée pour la requête
 
 ## Notes

DevDocMD/architecture/rebuild_filter.md

@@ -94,9 +94,9 @@ Option B avoids storing kmer values and works uniformly regardless of filter sel
 
 ## Files to modify
 
-- `src/obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
+- `src/obikindex/src/partition/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
 - Possibly `src/obicompactvec/` — add column iterator API if not already present
-- `src/obilayeredmap/` — check if per-column sequential access is exposed on `SrcLayerData`
+- `src/obikindex/src/layer/` — check if per-column sequential access is exposed on `SrcLayerData`
 
 ## Open questions
 

DevDocMD/architecture/siblings.md

@@ -17,7 +17,7 @@ one-way function, not an invertible bijection with a stored inverse. Any
 method that reconstructs a kmer from a bare slot number is wrong by
 construction, regardless of the mechanism used (MPHF re-hash, or evidence
 decode + direct unitig read). See `MphfLayer::kmer_at`
-(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
+(`obikindex/src/layer/mphf_layer.rs`) — flagged for removal, currently called
 from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
 "Pending work" status below).
 
@@ -70,7 +70,7 @@ partition is unknown) and must keep going through
 ## Pending work — done
 
 The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
-traits over `obikindex::KmerIndex`/`obilayeredmap::Layer<D>`, replacing the
+traits over `obikindex::KmerIndex`/`obikindex::layer::Layer<D>`, replacing the
 old `obikindex::siblings` module) builds and reads the annex purely in
 iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
 batch variants, mirroring `Layer<D>`'s own `KmerIter`/`KmerBatchIter`
@@ -652,13 +652,13 @@ BRWT-style column-correlation exploitation.
 `PersistentSparseBitMatrix` went from a validated but unused type to a
 real, selectable on-disk format:
 
-- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
+- **Generic `Layer<D>`**: `obikindex::layer::Layer<D>`'s presence-only methods
   (`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
   `D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
   `PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
   `LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
   already generic over any `D: LayerData`, so they needed no change.
-  Verified by `obilayeredmap`'s
+  Verified by `obikindex::layer`'s
   `presence_layer_generic_over_sparse_matches_dense` test: build a dense
   presence layer, convert it to sparse via `build_from_dense`, open both
   as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
@@ -668,7 +668,7 @@ real, selectable on-disk format:
   `AtomicUsize`s in test builds, not thread-local, so a test using a
   different `k` races every other test in the same crate binary; a k=11
   version of this test passed alone but failed under the full
-  `obilayeredmap` suite for exactly that reason before being fixed.)
+  `obikindex::layer` suite for exactly that reason before being fixed.)
 - **`obikphylo::siblings::cache::Mat`** gained a third variant,
   `SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
   and `Presence` — every method (`find_slot`, `index_batch`,
@@ -698,7 +698,7 @@ real, selectable on-disk format:
   dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
   sparse format round-trips through the real build pipeline
   (`PartitionCache` sparse-detection included), not just the
-  `obicompactvec`/`obilayeredmap` unit layers below it.
+  `obicompactvec`/`obikindex::layer` unit layers below it.
 
 Full workspace `cargo test` (all crates, unit + doc tests) green after
 this change.
@@ -746,7 +746,7 @@ Replaces the "four independent scans" problem above and implements
   (activation: either given; defaults `1.0`/`0.5` for whichever is unset).
 
 Full workspace `cargo test` green after this change (167 unit tests in
-`obicompactvec`+`obilayeredmap`+`obikphylo` alone, plus every other
+`obicompactvec`+`obikindex::layer`+`obikphylo` alone, plus every other
 crate's suite, no regressions).
 
 **Still open, not part of this change** (per "Correction to the 'single
@@ -756,3 +756,269 @@ scan the full unsampled index — never threaded `--subsample`/`--entropy`,
 out of scope here since the reported problem was specifically about the
 `--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these
 two standalone flags.
+
+## `query` never benefits from sparse row-major access (found 2026-08-19, not implemented)
+
+Benchmarked `obikmer query` against `global_index_presence` (dense-packed)
+vs. `global_index_presence_sparse` (`pack --sparse`), 100k simulated reads
+× 2 specimens (`benchmark/`, see
+[benchmark_query_testing.md](../implementation/benchmark_query_testing.md)).
+Correctness: 0 mismatches — sparse and dense return bit-identical query
+results. Performance: sparse consistently *slower* than dense (~30-50%,
+reproducible across two runs with warm disk cache), the opposite of
+`pack --sparse`'s stated intent ("faster for single-row access... like
+query").
+
+**Root cause, read from source, not measured in isolation:**
+`KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs:155-220`)
+is architecturally column-major: stage 2 walks `for g in 0..n_cols { for
+slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly)
+as the right locality strategy for the packed/columnar formats, where
+`col_value` → `PersistentBitMatrix::get` is a genuine O(1) mmap'd column
+read (`persistent.rs:110-113`).
+
+For `Self::Sparse`, that same `get(c, slot)` (`persistent.rs:114-118`)
+allocates a full `n_cols`-wide buffer and calls `fill_row` — materializing
+the *entire row* — just to return one cell. Called from inside the
+column-major double loop, this reconstructs the same row once per genome
+column touched: O(hits × n_cols) full-row rebuilds instead of O(hits).
+`PersistentSparseBitMatrix`'s own native row-major decode
+(`for_each_genome_in_row`, `sparse.rs:164-177`, used correctly by its own
+`row`/`fill_row`/`fill_sub_matrix`) is never reached from the query path
+at all.
+
+**`fill_sub_matrix` (the existing `BinaryMatrix` trait primitive,
+`traits.rs:13-37`) is not the right replacement for `query` either**, even
+once its own dispatch bug is fixed (see next section) — its output shape
+is inherently column-dense: `out[col]` gets an entry for every column,
+including columns with zero hits among the requested slots. On real
+sparse data (a hit typically touching a handful of genomes out of dozens)
+that's still O(n_cols) output regardless of true sparsity. What `query`
+actually wants is the sparse triple stream `(slot, col, value)` it already
+consumes as `QueryHit::Value` — not a materialized sub-matrix.
+
+**Proposed primitive** (design only, not implemented — explicit ask: keep
+count matrices *not excluded*, even though effort right now is
+presence/absence only):
+
+Not a closure-driven `for_each` — a real `Iterator`, one concrete struct
+per matrix format, so the traversal state (current position in the sorted
+slot list, current column, permutation, sparse-row decode cursor…) lives
+in named struct fields instead of being threaded implicitly through
+recursion or a captured closure. RPITIT (stable since 1.75, and this
+workspace is edition 2024) means the trait method can return it without
+naming or boxing the concrete type:
+
+```rust
+/// Yields every nonzero cell among `slots`, in implementation-defined order.
+fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator<Item = (usize, usize, u32)> + 'a;
+// item: (idx into `slots`, col, value)
+```
+
+This is the one foundational primitive per format — both `fill_sub_matrix`
+and a `for_each`-style callback become trivial, free consumers of it
+(`.for_each(f)` is already `Iterator::for_each` from std; `fill_sub_matrix`
+becomes "drain the iterator, scatter into `out[][]`"), instead of two
+independently-maintained traversals that can silently diverge (see the bug
+below — this is exactly how it happened).
+
+- **On `PersistentSparseBitMatrix`**: the struct is nearly free to write —
+  it wraps the existing (currently private) `for_each_genome_in_row`
+  per-row decode, advancing to the next `slots` entry on exhaustion. O(Σ
+  row nnz), zero `n_cols`-wide allocation.
+- **On `PersistentBitMatrix::{Packed,Columnar}`**: revised — cheaper than
+  first thought, by reusing the same split already used for
+  `fill_matrix`'s own implementation instead of hand-writing a resumable
+  state machine at the matrix level. The base-vector layer
+  (`BitSliceView`, `views.rs`) already separates the two concerns:
+  `fill_batch_sorted` (`views.rs:55-60`, sorted-slot batch lookup) and a
+  genuine per-bit `Iterator` (`BitSliceIter`, `views.rs:94+`) sit side by
+  side there, one level below the matrix. Adding a
+  "positions among `sorted_slots` where the bit is set" iterator at that
+  same vector level is a `filter` over the existing `get()` — no new
+  state machine, since `std::iter::Filter` already *is* one, generated by
+  the compiler:
+  ```rust
+  fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator<Item = usize> + 's {
+      sorted_slots.iter().copied().filter(move |&slot| self.get(slot))
+  }
+  ```
+  The matrix-level `nonzero_iter` then composes these per column with
+  `flat_map` over `0..n_cols` (each column's hits, tagged with `c`,
+  slot mapped back through the sort permutation `fill_batch`/
+  `fill_batch_sorted` already carry) — again a combinator chain, not a
+  hand-rolled struct. Same algorithm, same mmap/sort locality as today's
+  `fill_sub_matrix`; just assembled from `std` iterator adaptors instead
+  of a loop body writing into a buffer, mirroring the vector/matrix split
+  the codebase already uses for `fill_batch_sorted` rather than
+  introducing a new shape.
+- **`Implicit`**: trivial (`slots.iter().map(|&i| (i, 0, 1))`, one column,
+  always present).
+- **On `PersistentCompactIntMatrix` (counts)**: same treatment as
+  `Packed`/`Columnar` — no sparse count format exists yet ("Explicitly
+  deferred" per `traits.rs:9-12`), so no native low-effort case the way
+  `Sparse` has one, but not excluded either: the iterator's `Item` is
+  already `(usize, usize, u32)`, not `bool`, specifically so presence
+  (`0`/`1`) and counts (arbitrary `u32`) share one primitive instead of a
+  bool/u32 split forcing counts out of the design. Ready for a native
+  sparse-count struct later without a signature change.
+
+Would let `query_partition_with`'s stage 2 collapse to one
+`for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) }`
+per layer, format-agnostic, each backend's struct deciding the actual
+traversal.
+
+**This also closes the existing dispatch bug for free, by construction**:
+`PersistentBitMatrix::fill_sub_matrix` (`persistent.rs:190-215`, the enum
+wrapper backing `BinaryMatrix`'s trait impl) today does *not* delegate to
+`PersistentSparseBitMatrix::fill_sub_matrix` for `Self::Sparse` — it
+reimplements the same naive per-(column, slot) `fill_row_bool` loop
+instead, bypassing the efficient native method one file over
+(`sparse.rs:249-258`). `obikphylo::siblings::cache::Mat`
+(`cache.rs:138-145`) independently built its own parallel enum wrapper
+that dispatches correctly — evidence this was worked around rather than
+fixed at the source: two hand-written traversals for the same format,
+free to drift apart, and they did. If `fill_sub_matrix` itself is
+rewritten as "drain `nonzero_iter`, scatter into `out[][]`", there is only
+one traversal per format left to get right — the bug class doesn't just
+get fixed once, it stops being possible to reintroduce.
+
+## Implemented (2026-08-20)
+
+Built as designed above, with one deviation from the original sketch:
+`nonzero_iter` ended up `Box<dyn Iterator<...>>`, not a bare `impl
+Iterator`, because `Columnar`/`Packed`/`Sparse`/`Implicit` are genuinely
+different concrete types and this method isn't on a trait (kept off
+`BinaryMatrix` deliberately — that trait is used as `dyn BinaryMatrix` in
+`tests/sparse.rs`, and RPITIT methods aren't dyn-compatible). One `Box`
+per `nonzero_iter` call, not per cell — negligible next to what it
+replaces.
+
+- `BitSliceView::nonzero_among_sorted` / `IntSliceView::nonzero_among_sorted`
+  (`obicompactvec/src/views.rs`): the vector-level `filter`/`filter_map`
+  primitive, exactly as sketched — no new state machine, `std`'s own.
+- `PersistentSparseBitMatrix::nonzero_iter` (`bitmatrix/sparse.rs`): native,
+  `std::iter::from_fn` over one buffered row at a time via the existing
+  `for_each_genome_in_row` — no `n_cols`-wide allocation, ever.
+- `PersistentBitMatrix::nonzero_iter` (`bitmatrix/persistent.rs`): dispatches
+  to the above for `Sparse`; for `Columnar`/`Packed`, loops columns,
+  collects each column's `nonzero_among_sorted` hits via `.extend()` (not
+  `flat_map` — a `flat_map` closure can't lazily return something
+  borrowing its own captured sort permutation across separate calls
+  without either boxing per-column or fighting the borrow checker; eager
+  collection into one `Vec` sidesteps it, at zero cost since
+  `fill_sub_matrix` already fully materialized anyway). `Implicit` trivial.
+- `PersistentBitMatrix::fill_sub_matrix` and `sub_matrix` rewritten to
+  drain `nonzero_iter` — the dispatch bug is gone because there is now
+  only one traversal per format, not because the old one was patched.
+  `PersistentCompactIntMatrix::nonzero_iter` added the same way (counts
+  not excluded, per the earlier ask) — no native low-effort case, since no
+  sparse count format exists, but on the same primitive, ready for one.
+- `KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs`):
+  stage 2's column-major `for g { for slot { col_value } }` replaced by one
+  `layer.nonzero_iter(&slot_list)` call per layer, format-agnostic.
+- Tests: `nonzero_iter_matches_dense`, `nonzero_iter_matches_row`, and —
+  the one that actually targets the dispatch bug rather than each type's
+  own correctness — `enum_wrapper_dispatches_to_native_sparse` (builds
+  `PersistentBitMatrix::Sparse(...)` directly, not through `open`, since
+  `open` only auto-detects `Sparse` from a `presence/` dir layout).
+  `cargo test --workspace`: green, no regressions.
+
+**Measured**: re-ran the `benchmark/` query branch (100k reads × 2
+specimens, same setup as the original finding). Correctness still 0
+mismatches. The dense/sparse performance gap is gone — previously sparse
+~30-50% slower than dense, reproducibly; now within ~1-3% either way
+(7.42s dense vs 7.60s sparse for `Escherichia_coli--K-12_MG1655`; 5.25s vs
+5.30s for `Saccharolobus_islandicus--M.16.4`) — noise-level, not a
+systematic gap. `pack --sparse`'s claimed query win isn't confirmed
+outright by this (sparse should arguably now *beat* dense on truly sparse
+real data, not just tie), but the pathological regression is fixed.
+
+## `PersistentCompactIntMatrix::Sparse` — implemented (2026-08-26)
+
+Closes the gap flagged throughout this document ("no sparse count format
+exists yet", `traits.rs:9-12`'s "Explicitly deferred"): `obicompactvec`
+already had `PersistentSparseCompactIntMatrix` (row-major, built on top of
+`PersistentSparseBitMatrix` as its "which columns are non-zero" support,
+values *not* deduplicated — see that struct's own doc comment), but it was
+never wired into `PersistentCompactIntMatrix`, the dense-dispatching enum
+every real consumer (`TypedLayer<PersistentCompactIntMatrix>`,
+`KmerLayer::Count`) actually holds. Concretely: `kmer_index.rs::
+pack_matrices(sparse=true)` already called `pack_sparse_compact_int_matrix`
+on every layer's `counts/` — but `PersistentCompactIntMatrix::open` had no
+code path back to what that just wrote, so a `Count` layer became
+unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
+anyone ran `pack --sparse` on an index with count layers. Root cause, not a
+workaround: add the missing `Sparse` variant.
+
+- **Enum + dispatch** (`intmatrix.rs`): `PersistentCompactIntMatrix::Sparse
+  (PersistentSparseCompactIntMatrix)`, detected in `open`/`detect_storage`
+  via a `singleton_values.pciv` marker (mirrors `PersistentBitMatrix`'s own
+  `sparse_meta.json` check), reported via `storage_kind()`. `col`/
+  `col_view`/`col_persist` panic/`Unsupported` on `Sparse`, same convention
+  as the bit side. `sub_matrix`/`fill_sub_matrix` and `nonzero_iter`
+  unified the same way `PersistentBitMatrix`'s already are (drain
+  `nonzero_iter`, one traversal per format — see "Implemented
+  (2026-08-20)" above); `nonzero_iter` had to become `Box<dyn Iterator<...>>`
+  for the same reason (`Columnar`/`Packed`/`Sparse` are different concrete
+  types). No change needed in `obikindex` at all — `KmerLayer::Count`
+  already only ever holds `TypedLayer<PersistentCompactIntMatrix>`, so the
+  enum absorbing `Sparse` fixes the unreadable-layer bug for free, same as
+  `PersistentBitMatrix::Sparse` already did on the presence side.
+
+- **`CountPartials`, non-naive** (`sparse_intmatrix.rs`): unlike
+  `PersistentSparseBitMatrix`'s dict-driven `col_weights_and_pair_counts`,
+  values here aren't deduplicated (two rows can share the same non-zero
+  column set via the same `dict_id` while carrying different counts), so
+  the "weight by how many rows share a dict entry" shortcut doesn't carry
+  over. What does: a single row-major pass (`row_major_pairwise`, decodes
+  each row once via `for_each_cell_in_row`, nests over that row's own
+  co-present columns) — `O(Σ k̄²)` over populated rows instead of the naive
+  `O(n_cols² × n)` column-pair rescan, same complexity class as the bit
+  side minus the dict multiplicity discount. Kernels used: `min(a,b)`
+  (bray, relfreq-bray — both vanish when either side is absent, so no
+  correction needed), `a·b` and `√(a·b)` (euclidean/relfreq-euclidean and
+  hellinger — these *do* need a correction, reconstructed from per-column
+  marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`, since `(a-0)² = a² ≠ 0` unlike
+  the `min`-based formulas). `threshold_jaccard(1)` shortcuts straight to
+  `support`'s own `BitPartials::partial_jaccard` (threshold 1 is exactly
+  presence); `threshold_jaccard(0)` is closed-form (every `u32` is `≥ 0`).
+
+- **Two pre-existing bugs found and fixed while wiring the `threshold==1`
+  shortcut** (`bitmatrix/sparse.rs`, `BitPartials for
+  PersistentSparseBitMatrix`, present since the 2026-08-15 implementation
+  above, never caught because no test compared `Sparse`'s raw `partial_*`
+  output against dense on real data — only the diagonal-blind
+  `jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested):
+  1. `partial_jaccard`'s diagonal was `(0, 2×col_weights[i])` instead of a
+     genuine self-comparison `(col_weights[i], col_weights[i])` —
+     `col_weights_and_pair_counts`'s `inter` never pairs a column with
+     itself by construction.
+  2. `partial_hamming`'s off-diagonal formula itself was wrong: `total -
+     union` (count of rows where *neither* column is present) instead of
+     the actual Hamming distance `col_weights[i] + col_weights[j] -
+     2×inter[i,j]` (symmetric-difference size). Only coincides with the
+     correct value when `col_weights[i] + col_weights[j] == total`, so
+     small/synthetic test data could easily have hidden it.
+
+  Neither surfaced through `jaccard_dist_matrix`/`hamming_dist_matrix`
+  (both explicitly zero their own diagonal at finalisation, and the
+  off-diagonal `partial_hamming` bug had gone untested against dense
+  entirely) — only visible to a caller of the raw `partial_*` methods
+  directly, which is exactly what `partial_threshold_jaccard(1)`'s new
+  shortcut became. Fixed at the source, not patched around at the call
+  site; regression test added:
+  `tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal`.
+
+- **Tests**: `tests::intmatrix::sparse_roundtrip_matches_columnar`/
+  `sparse_roundtrip_from_packed` (the `open`-dispatch fix, both build
+  paths); `tests::intmatrix::sparse_count_partials_match_dense` (all six
+  `CountPartials` formulas, thresholds 0/1/2/3, against `Columnar` on
+  asymmetric-presence data — this is what caught the diagonal gap in the
+  int side's own new code before it shipped, the same way it exposed the
+  two pre-existing bit-side bugs above); `obikindex`'s
+  `count_layer_transparently_reads_sparse_after_pack` — the actual
+  end-to-end regression test for the original "layer unreadable after
+  `pack --sparse`" bug, built → packed sparse → reopened, compared against
+  the pre-pack dense read. `cargo test -p obicompactvec -p obikindex`:
+  green, no regressions (180 + 12 tests).

DevDocMD/implementation/benchmark_query_testing.md

@@ -0,0 +1,93 @@
+# Benchmark: query-path testing
+
+`benchmark/Makefile` exercises indexing, merge, and phylo distance
+reconstruction against simulated bacterial genomes. It now also covers
+`obikmer query` — the read-matching path — and the sparse packed
+presence-matrix format (`obikmer pack --sparse`), previously untested by
+this pipeline.
+
+## Motivation
+
+- `query` had no end-to-end coverage. A regression there would not be caught
+  by `verify_presence`/`verify_merge_presence`, which only check index
+  *content* against the `.npz` truth, never the query API.
+- `pack --sparse` produces a presence-matrix format documented (see
+  [siblings.md](../architecture/siblings.md)) as faster for single-row
+  access (query) and slower for column-oriented access (phylo `--metric`).
+  `global_index_presence/` built by `merge_presence.sh` is always packed
+  dense (packing is a stage inside `merge`, not a separate `pack`
+  invocation) — there was no dense/sparse regression check.
+
+## Query read source
+
+Query reads are independent of `simulated_data/` (which is folded into the
+index being queried): reusing those reads would test against the exact
+error draw the index was built from. `query_data/<species>/<strain>/` holds
+a *second*, independent `iss generate` run against the same reference
+genome, via `simulate_query_one.sh` — unseeded, so a second draw picks up
+different sequencing errors than `simulate_one.sh`'s draw for the same
+genome. Fixed at 100,000 read pairs per genome (not coverage-proportional
+like the 15x used for `simulated_data/`), so wall/RSS numbers stay
+comparable across genomes of very different sizes.
+
+Two query-source specimens, hardcoded as `QUERY_SPECIMENS` in
+`make_deps.py`: `Escherichia_coli--K-12_MG1655` (common, well-represented
+bacterium) and `Saccharolobus_islandicus--M.16.4` (the only archaeon in
+`SPECIES` — distant lineage, stresses the query path differently from a
+close-relative match). Two is enough to catch a dense/sparse regression
+without duplicating the exhaustive per-specimen coverage
+`verify_merge_presence` already provides across all `SPECIMENS`.
+
+## Sparse global index
+
+`global_index_presence_sparse/` is built by `pack_sparse.sh`: copy
+`global_index_presence/` wholesale, then `obikmer pack --sparse` in place.
+This works directly because `merge`'s pack stage (`merge.rs:252`,
+`pack_matrices(false)`) keeps the per-genome column files on disk after
+dense-packing — `pack_sparse_bit_matrix` (`obicompactvec/src/bitmatrix/sparse.rs:447`)
+reads those, is idempotent, and removes `matrix.pbmx` once the sparse form
+is written, so `Persistent::open` falls through to the sparse format
+afterward. No separate merge run needed.
+
+## Query runs
+
+`query_one.sh dense|sparse SPECIMEN` runs `obikmer query --count-missing`
+against `global_index_presence` or `global_index_presence_sparse`, output
+gzipped to `query_{dense,sparse}/SPECIMEN.fasta.gz`, Reporter wall/RSS
+captured to `stats/query_{dense,sparse}/SPECIMEN.stats` (same
+stderr-parsing convention as `merge_presence.sh`).
+
+Flags: `--count-missing` only. `--mismatch` is a no-op today
+(`query/mod.rs:212-213`, prints "not yet implemented, ignored") — left off
+rather than tested for a feature that doesn't exist yet.
+
+## Dense/sparse regression
+
+`verify_query.py` compares the two query outputs per specimen, matched by
+read id (not stream position — the query pipeline chunks input across
+worker threads and doesn't guarantee output order). Compares `kmer_count`,
+`kmer_missing`, and the full `kmer_strict_matches` map per read. Any
+mismatch is a real regression: dense and sparse must be content-identical,
+only I/O access pattern differs. `.stats` → `stats/verify_query/`,
+aggregated by `aggregate_stats.sh query|verify_query`-style cases
+(`query_dense`, `query_sparse`, `verify_query`).
+
+## Performance comparison
+
+No dedicated script: the wall/RSS columns from the `query_dense` and
+`query_sparse` aggregated `.stats` CSVs are the dense-vs-sparse performance
+comparison — the expected win for query on sparse, per the `pack --sparse`
+help text.
+
+## Scope
+
+`count` track excluded from the sparse branch: `pack --sparse` targets
+presence matrices only (per CLI help); `pack_matrices` leaves count
+matrices untouched regardless of the `sparse` flag
+(`obikindex/src/index.rs:308`).
+
+## New Makefile targets
+
+`simulate_query`, `pack_sparse`, `query_dense`, `query_sparse`,
+`aggregate_query_dense`, `aggregate_query_sparse`, `verify_query`,
+`aggregate_verify_query` — the last three folded into `all`.

DevDocMD/implementation/evidence_elimination.refs.md

@@ -3,9 +3,9 @@
 
 ## Code couvert
 
-- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
-- `obilayeredmap/src/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
-- `obilayeredmap/src/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
+- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
+- `obikindex/src/layer/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
+- `obikindex/src/layer/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
 - `obikindex/src/reindex.rs` — KmerIndex::reindex(), conversion exact↔approx en place
 - `obikmer/src/cmd/reindex.rs` — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size
 - `obikmer/src/cmd/index.rs` — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp

DevDocMD/implementation/filtering.md

@@ -303,7 +303,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
 
 ## Implementation
 
-- **`obikpartitionner::filter::GroupQuorumFilter`** — implements `KmerFilter`
+- **`obikindex::partition::filter::GroupQuorumFilter`** — implements `KmerFilter`
   using pre-computed ingroup and outgroup index vectors. The heavy logic
   (predicate parsing, three-value evaluation, genome classification) happens
   once before any iteration; each k-mer row evaluation is a simple index
@@ -314,7 +314,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
   `UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
   list.
 
-- **`obikpartitionner::KmerPartition::iter_partition_kmers`** — accepts
+- **`obikindex::partition::KmerPartition::iter_partition_kmers`** — accepts
   `filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
   the callback. `filter`, `dump`, and `unitig` all go through this single
   entry point.

DevDocMD/implementation/layer_tier.md

@@ -1,8 +1,8 @@
-# obilayeredmap — layered kmer index crate
+# obikindex::layer — the Layer tier
 
 ## Purpose
 
-`obilayeredmap` implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
+`obikindex::layer` (the `layer/` submodule of the `obikindex` crate — a standalone `obilayeredmap` crate until 2026-08-21, folded back in alongside `obikpartition` as part of a broader `Index { Partition { Layer } }` submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
 
 ---
 
@@ -250,6 +250,59 @@ Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers buil
 
 ---
 
+## Layer\<D\> — raw mapping, iteration, and batch access
+
+Beyond `query`/`find` (membership-checked), `Layer<D>` exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.
+
+### Raw kmer → slot mapping
+
+```rust
+pub fn index(&self, kmer: CanonicalKmer) -> usize
+pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize>
+```
+
+Pure MPHF mapping, no evidence/fingerprint check — equivalent to `MphfOnly::index`. Only meaningful when the caller already knows `kmer` belongs to the layer; on an absent kmer the MPHF still returns *some* slot (undefined, not `None`).
+
+### Kmer iteration
+
+Four iterators, all built from `unitigs.bin` (physical layout order, **not** correlated with MPHF slot numbers):
+
+```rust
+pub fn iter_kmers(&self) -> KmerIter<'_>
+pub fn enumerate_kmers(&self) -> Enumerate<KmerIter<'_>>              // (order_index, kmer)
+pub fn iter_kmers_batch(&self, n: usize) -> KmerBatchIter<'_>          // Vec<CanonicalKmer> of size ≤ n
+pub fn enumerate_kmers_batch(&self, n: usize) -> impl Iterator<Item = (usize, Vec<CanonicalKmer>)> + Send + 'static
+```
+
+`KmerIter`/`KmerBatchIter` own a clone of the underlying `Arc<UnitigFileReader>` rather than borrowing `self` — `Send + 'static`, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. `enumerate_kmers_batch`'s index is the batch's starting offset in iteration order (a multiple of `n` except for the final, possibly shorter, batch).
+
+### Batch lookup on payload vectors/views
+
+`PersistentCompactIntVec`, `PersistentBitVec`, `IntSliceView`, `BitSliceView` all expose:
+
+```rust
+fn get_batch(&self, slots: &[usize]) -> Vec<T>
+fn fill_batch(&self, slots: &[usize], out: &mut [T])
+```
+
+Both sort `slots` internally for sequential mmap access, then reorder results back to the caller's original order. `fill_batch` fills a caller-provided buffer, avoiding the `Vec` allocation.
+
+### sub_matrix / fill_sub_matrix
+
+```rust
+// Layer<PersistentCompactIntMatrix>
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<u32>>            // column-first
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<u32>])
+
+// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
+```
+
+Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
+
+---
+
 ## LayeredStore\<S\> and aggregation traits
 
 `LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls:

DevDocMD/implementation/layer_tier.refs.md

@@ -1,14 +1,14 @@
 <!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
-# Coverage: implementation/obilayeredmap.md
+# Coverage: implementation/layer_tier.md
 
 ## Code couvert
 
-- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
-- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
-- `obilayeredmap/src/map.rs` — LayeredMap<D>, push_layer(), query()
-- `obilayeredmap/src/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
-- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
-- `obilayeredmap/src/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
+- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
+- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
+- `obikindex/src/layer/map.rs` — LayeredMap<D>, push_layer(), query()
+- `obikindex/src/layer/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
+- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
+- `obikindex/src/layer/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
 
 ## Notes
 

DevDocMD/implementation/merge.md

@@ -194,3 +194,14 @@ spectrums/
   <label>.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 ```
+
+`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json` marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — `merge_partition` never rewrites them for pre-existing layers, only the `presence`/`counts` subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
+
+## Known issue (not yet fixed, 2026-08-28)
+
+Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. `obikmer merge -o out IDX IDX --rename-duplicates`) — panics deep in the MPHF's rank-select structure (`common_traits::select_in_word`, `assertion failed: rank < self.count_ones()`), inside `MphfLayer::find` called from `merge_partition`'s "is this source kmer already in dst" check against the bootstrap-copied `dst_layers`. Root cause not identified; ruled out so far:
+
+- **Not an empty-new-layer issue**: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
+- **Not caused by the hard-link change above**: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
+
+Only reproduces when `sources[0]` and `sources[1]` are the exact same on-disk path opened as two independent `KmerIndex` handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since `PartitionRunner`'s panic propagation fix (see `architecture/numa_partition_runner.md`) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.

DevDocMD/implementation/merge.refs.md

@@ -4,9 +4,9 @@
 ## Code couvert
 
 - `obikindex/src/merge.rs` — `KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
-- `obikpartitionner/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
-- `obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
-- `obilayeredmap/src/layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
+- `obikindex/src/partition/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
+- `obikindex/src/partition/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
+- `obikindex/src/layer/typed_layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
 - `obicompactvec/src/intmatrix.rs` — `append_column` pour PersistentCompactIntMatrix
 - `obicompactvec/src/bitmatrix.rs` — `append_column` pour PersistentBitMatrix
 

DevDocMD/implementation/mphf.md

@@ -6,7 +6,7 @@ Kmer indexing per partition proceeds in two phases. The separation is necessary
 
 ### Phase 1 — provisional MPHF + kmer spectrum
 
-Implemented in `obikpartitionner::KmerPartition::count_kmer()` → `count_partition()`.
+Implemented in `obikindex::partition::KmerPartition::count_kmer()` → `count_partition()`.
 
 1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
 2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
@@ -101,7 +101,7 @@ type Mphf = PtrHash<
 
 ### Layer structure
 
-Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
+Each layer is a self-contained unit. See [obikindex::layer](layer_tier.md) for the full on-disk layout. The MPHF-relevant files are:
 
 ```
 layer_i/
@@ -148,7 +148,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
 
 There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
 
-In `obikpartitionner`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
+In `obikindex::partition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
 
 ### Membership verification
 

DevDocMD/implementation/mphf.refs.md

@@ -3,8 +3,8 @@
 
 ## Code couvert
 
-- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
-- `obikpartitionner/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
+- `obikindex/src/layer/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
+- `obikindex/src/partition/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
 
 ## Notes
 

DevDocMD/implementation/obicompactvec.md

@@ -292,14 +292,20 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
 
 ## Matrix types
 
-Four matrix types, two encodings × two formats:
+Both matrix types are enums behind a transparent API — the caller never matches on the variant. `PersistentCompactIntMatrix` has three variants (`Columnar`, `Packed`, `Sparse`). `PersistentBitMatrix` has four:
 
-| | Columnar format | Packed format |
+| Variant | Storage | When |
 |---|---|---|
-| **Bit** | `PersistentBitMatrix` (Columnar variant) | `PersistentBitMatrix` (Packed variant) |
-| **Int** | `PersistentCompactIntMatrix` (Columnar variant) | `PersistentCompactIntMatrix` (Packed variant) |
+| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
+| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
+| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off |
+| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
 
-Both matrix types are enums (`Columnar` / `Packed` / `Implicit` for bit) behind a transparent API. `col_view(c)` returns the appropriate view directly:
+`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
+
+Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation.
+
+`col_view(c)` returns the appropriate view directly:
 
 ```rust
 // PersistentBitMatrix

DevDocMD/implementation/obipipeline.refs.md

@@ -9,5 +9,5 @@
 ## Notes
 
 Document stable (librairie générique, peu de risque de dérive).
-Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartitionner`
+Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikindex::partition`
 ou s'il a été remplacé par Rayon dans certains chemins.

DevDocMD/implementation/pipeline.md

@@ -203,7 +203,7 @@ part_XXXXX/
 
 **Cleanup:** unless `--keep-intermediate` is set, `remove_build_artifacts` deletes `dereplicated.skmer.zst`, `mphf1.bin`, and `counts1.bin` after all partitions are indexed.
 
-See [obilayeredmap](obilayeredmap.md) and [MPHF selection](mphf.md) for data structure details.
+See [obikindex::layer](layer_tier.md) and [MPHF selection](mphf.md) for data structure details.
 
 **Query path (exact evidence):**
 

DevDocMD/implementation/pipeline.refs.md

@@ -3,12 +3,12 @@
 
 ## Code couvert
 
-- `obikpartitionner/src/partition.rs` — estimation des paramètres (phase 0)
+- `obikindex/src/partition/partition.rs` — estimation des paramètres (phase 0)
 - `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
-- `obikpartitionner/src/filter.rs` — déduplication bucket-sort (phase 2)
-- `obikpartitionner/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
+- `obikindex/src/partition/filter.rs` — déduplication bucket-sort (phase 2)
+- `obikindex/src/partition/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
 - `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
-- `obikpartitionner/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
+- `obikindex/src/partition/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
 - `obikindex/src/index.rs` — `build_layers()`, `dereplicate_and_count()`
 
 ## Notes

DevDocMD/implementation/select.md

@@ -23,28 +23,37 @@ are preserved unchanged; only the data matrices are rewritten.
 
 ```sh
 obikmer select <input-index>
-        { --output <dir> | --in-place }
+        --output <dir>
         [--group    <name>:<pred>  ...]
         [--group-op <name>:<op>    ...]
         [--aggregate-by <key>          ]
         [--aggregate-op <op>           ]
         [--select   <col1,col2,...>    ]
         [--presence-threshold <N>      ]
+        [--force-copy                  ]
 ```
 
 ---
 
 ## Output destination
 
-Exactly one of `--output` or `--in-place` must be specified.
+`--output <dir>` is required — `select` always writes a new index; there is no
+`--in-place` mode (2026-08-28: never implemented, removed from the design). The
+source index is unchanged.
 
-**`--output <dir>`** — writes a new index to `<dir>`. The source index is
-unchanged. The MPHF and unitig files are copied; only the data matrices are
-rewritten with the new column layout.
+Each layer's kmer-identity files (`mphf.bin`/`unitigs.bin`/`evidence.bin`/
+`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json`) are never rewritten by a
+column projection/aggregation, so they are hard-linked into the output rather
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); `--force-copy`
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+`select` itself would affect the other). Only the `presence`/`counts`
+subdirectory is ever a genuinely new, independent file.
 
-**`--in-place`** — rewrites the data matrices of the source index directly.
-Removed or replaced columns are lost. The operation writes to temporary files
-first, then renames atomically, so an interrupted run leaves the index intact.
+To replace an index with a selected version of itself, select to a temporary
+directory and swap it in (`rm -rf INDEX && mv INDEX.tmp INDEX`) — the case
+`--in-place` used to cover.
 
 ---
 
@@ -186,12 +195,6 @@ obikmer select myindex --output out \
   --select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
 ```
 
-### In-place: keep only group A
-
-```sh
-obikmer select myindex --in-place --group "A:group=A" --select "A"
-```
-
 ### Compose with filter
 
 ```sh
@@ -207,28 +210,45 @@ obikmer select filtered --output final --aggregate-by site
 
 ## Implementation notes
 
-`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
-(rayon), each writing its output independently; results require no synchronisation
-because every partition owns a distinct set of files.
+`select` does not rebuild the MPHF. Every partition is processed independently
+(`PartitionRunner`), each writing its own output layers; no cross-partition
+synchronisation is needed.
 
-For each layer in each partition:
+For each layer in each partition (`obikselect::select_layer::select_partition`):
 
-1. The slot count `n` is read by opening the source data matrix.
-2. A new data matrix is built with M columns (M = number of output columns).
-3. For each slot `s` in `0..n`:
-   - `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
-   - For each output column `j`:
-     - `new_row[j] = aggregate(op, old_row[group_indices])`.
-     - Pass-through columns are represented as single-element groups with the
-       default operator (`any` for presence, `sum` for count) — same code path.
-   - The new row is written slot by slot into each column builder.
-4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
-   evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
-   or `counts/` subdirectory is rewritten.
+1. `copy_layer_files` hard-links the source layer's kmer-identity files
+   (`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/
+   `fingerprint.bin`/`layer_meta.json`) into the destination — never a real
+   copy unless linking fails or `--force-copy` is given.
+2. A new data matrix is built with M columns (M = number of output columns),
+   under a fresh `presence/`/`counts/` subdirectory (never touching the
+   source's own).
+3. **Presence source (2026-08-28: `batch_presence_counts`)**: one shared pass
+   over the source bit matrix computes every output group's presence count at
+   once — row-major native for a `Sparse` source (`for_each_genome_in_row`,
+   which has no column representation to read a `col_view` from at all — the
+   reason this replaced the old per-group loop, not just an optimisation of
+   it), deduplicated column-major (one `col_view` per *distinct referenced
+   column*, not per group) for `Columnar`/`Packed`. Every `AggOp` for a bit
+   matrix is then a cheap derivation of that one count vector (`sum` = the
+   count itself, `any`/`max` = `count ≥ 1`, `all`/`min` = `count == group
+   size`, `none` = `count == 0`) — see
+   `obikselect::select_layer::agg_result_from_count`.
+4. **Count source**: unchanged, one `col_view`-driven pass per output column
+   via `MatrixGroupOps` — `sum`/`min`/`max` are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.
 5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
 
-**`--in-place` write strategy:** new data is written to a temporary sibling
-directory (`presence_new/` or `counts_new/`); on success the old directory is
-removed and the temporary one is renamed into place. An interrupted run leaves
-at most one stale `*_new/` directory; the original data is intact until the
-rename step.
+### Known gap (not yet fixed, 2026-08-28)
+
+Step 4 above still panics (`col_view() not available on Sparse
+PersistentCompactIntMatrix`) if the source is a **count** index packed
+sparse — `batch_presence_counts`' row-major treatment was only ported to the
+bit-matrix (`Presence`) case, since that was the one actually blocking a real
+benchmark run. `select`/`filter` on a sparse-packed count index still hits
+this; the fix would follow the same shape (a `PersistentSparseCompactIntMatrix`
+row-major decode, analogous to `for_each_genome_in_row`), just not done. Since
+`obisys::numa::runner::PartitionRunner`'s panic-propagation fix (see
+`architecture/numa_partition_runner.md`), this at least fails fast (process
+panic, exit 101) instead of hanging.

DevDocMD/implementation/storage.refs.md

@@ -5,7 +5,7 @@
 
 - `obikindex/src/meta.rs` — IndexMeta, IndexConfig (version, config, genomes)
 - `obikindex/src/index.rs` — layout sur disque : partitions/, index.meta
-- `obilayeredmap/src/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
+- `obikindex/src/layer/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
 - `obiskio/src/unitig_index.rs` — fichiers unitigs.bin + unitigs.bin.idx
 
 ## Notes

DevDocMD/implementation/unitig_evidence.refs.md

@@ -4,7 +4,7 @@
 ## Code couvert
 
 - `obiskio/src/unitig_index.rs` — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()
-- `obilayeredmap/src/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
+- `obikindex/src/layer/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
 - `obidebruinj/src/debruijn.rs` — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK
 
 ## Notes

DevDocMD/theory/evolutionary_distances.md

@@ -1489,7 +1489,7 @@ instead of a per-partition `partial_*`, run the sequential source sweep:
 
 ```text
 for p in 0..n_partitions:                       # OUTER — sequential
-    open source partition p's layers (QueryLayer-style, obikpartitionner)
+    open source partition p's layers (QueryLayer-style, obikindex::partition)
     enumerate distinct canonical k-mers of p (one per MPHF slot) with their
         presence/count vectors                   # column-major, as query stage 2
     par_iter over these source k-mers:           # INNER — rayon, thread-local tally
@@ -1513,7 +1513,7 @@ for p in 0..n_partitions:                       # OUTER — sequential
 ```
 
 The inner lookup is precisely `QueryLayer::find_slot` +
-`col_value(g, slot)` (`obikpartitionner/src/query_layer.rs`) — reuse or factor
+`col_value(g, slot)` (`obikindex/src/partition/query_layer.rs`) — reuse or factor
 out that path rather than reimplementing MPHF access. Enumerating "all distinct
 k-mers of a partition with their vectors" is the `dump`/`query` stage-2
 column-major scan already implemented in `dump_layer.rs` /
@@ -2182,6 +2182,697 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
 from the written `_iqtree_states.csv` and `A`/`C` still present). Full
 workspace `cargo test` green.
 
+## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+
+**Implemented.** `--metric` (renamed `--distance` — several of
+its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
+`--metric` was a misnomer) gains a family of `snp-*` values computed from the
+central-position SNP pipeline, routed internally to the sibling-annex
+machinery (`PairwiseTally`, `obikphylo::siblings::algorithms::pairwise`)
+instead of `cache.distance(...)`'s existing per-layer traversal — a different
+code path behind the same CLI surface, not just another branch of one
+formula function.
+
+**Why unify at the CLI level despite the implementation split**: phylogenetically
+a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
+to how the matrix was produced, so exposing it as a special-cased subcommand
+instead of a `--distance` value would misrepresent its role. The
+implementation divergence (sibling-annex-based vs. plain index scan) is real
+but belongs at the routing layer, invisible to the CLI's own vocabulary.
+
+**`--subsample` becomes optional for `snp-*` distances** (it stays mandatory
+for `--sankoff`/`--pseudo-alignment`, unrelated commands): absent means
+exhaustive, achieved for free by reusing `sample_index`'s existing
+proportional-per-layer-quota mechanism with `n` set to the index-wide total
+non-monomorphic-minorant count (already available from the sibling-annex
+stats) — every layer's quota then equals its own full count, giving Bernoulli
+`p = 1` everywhere, i.e. every eligible family is drawn. No second,
+exhaustive-only driver needed. Present means sampled, exactly as `--sankoff`
+already behaves.
+
+**One shared tally, many derived formulas.** `PairwiseTally`'s `subst[4][4]`
+per-pair substitution counts (plus marginal base frequencies derived from it)
+are the sufficient statistic for every closed-form correction below — each
+is a small pure function `PairwiseTally -> Array2<f64>`, at the same level as
+the already-implemented `raw_snp_distance`/`base_pair_tally`/
+`cardinality_tally`. No new full scan per formula, whether the tally itself
+was built exhaustively or from a subsample.
+
+**`--raw-snp-counts` stays a separate, unrelated flag** — same underlying
+tally, but a diagnostic (`n_snp`/`n_shared`/`n_eligible` per genome pair, one
+row per pair) rather than a distance value, and its long-table shape doesn't
+fold into a single N×N matrix the way a distance does. No change to its
+existing CSV format.
+
+### `snp-*` distance catalog
+
+All closed-form (method-of-moments / direct formula), none requiring
+per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
+*tree*-ML usage but not its *pairwise* estimator, which is closed-form like
+F84/TN93 and is included below. `snp-` prefix on every CLI value.
+
+| value | corrects for | inputs beyond raw counts |
+|---|---|---|
+| `snp-raw` | nothing (uncorrected p-distance) | — |
+| `snp-jc` (Jukes-Cantor, JC69) | multiple substitutions per site | — |
+| `snp-k2p` (Kimura 2-parameter, K80) | + transition/transversion rate bias | ts/tv split |
+| `snp-k81` (Kimura 3-parameter, K3ST) | + splits transversions into 2 categories | ts/tv split, by category |
+| `snp-f81` (Felsenstein 81) | + unequal base frequencies (no ts/tv split) | empirical base freqs |
+| `snp-tajima-nei` (Tajima-Nei 1984) | same goal as F81 (equal-input model), different formula, better small-sample behavior | empirical base freqs |
+| `snp-t92` (Tamura 3-parameter) | K2P + GC-content bias | ts/tv split, GC content |
+| `snp-f84` (Felsenstein 84) | full empirical base freqs + single ts/tv rate | empirical base freqs, ts/tv split |
+| `snp-hky85` (Hasegawa-Kishino-Yano, pairwise estimator) | same inputs as F84, different formula | empirical base freqs, ts/tv split |
+| `snp-tn93` (Tamura-Nei) | full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate | empirical base freqs, purine-ts/pyrimidine-ts/tv split |
+| `snp-logdet` (LogDet / paralinear) | no shared-model or stationarity assumption at all — general divergence-matrix determinant | full empirical 4×4 divergence matrix (already `subst[4][4]`) |
+| `snp-tv` (transversions-only p-distance) | diagnostic/deep-divergence variant — drops transitions entirely (they saturate first) | tv-only counts |
+
+**`+Γ` rate-heterogeneity modifier, applicable to `snp-jc`, `snp-k2p`,
+`snp-k81`, `snp-t92`, `snp-f84`, `snp-hky85`, `snp-tn93`** (not `snp-raw`,
+nothing to correct; not `snp-logdet`, no standard gamma formulation) — same
+formula as the base correction, weighted by a shape parameter `α` either
+supplied by the user (`--gamma-shape <alpha>`) or estimated from the data
+(`--gamma-shape auto`/`estimate`, method-of-moments — not ML; see
+"Automatic α estimation" below). A modifier on existing values, not a
+separate enum arm per distance.
+
+### Automatic α estimation (`--gamma-shape auto`)
+
+**Correction (verified against the primary source, 2026-09-11):** Jin &
+Nei (1990) itself (*"Limitations of the Evolutionary Parsimony Method of
+Phylogenetic Analysis"*, Mol. Biol. Evol. 7(2):82–102 — the paper this
+whole `+Γ` correction is cited from, confirmed algebraically to match this
+codebase's `corrected_log`/`k2p` exactly against their eq. A4, general, and
+A8, the `a = 1` case) contains **no** data-driven α-estimation procedure.
+Their own recommendation (p. 98) is a fixed default: *"we suggest that the
+gamma distance with a = 1 [eq. A8] be used. However, one may choose a
+different gamma distance, estimating a from data. Wilson et al. (1989)
+recently used a distance with a = 1/2 for restriction-site data of
+mitochondrial DNA in hominoids."* — i.e. Jin & Nei explicitly punt
+data-driven estimation to a *different* paper (Wilson et al. 1989), not
+read/verified here. The estimator below is therefore **not** "Jin & Nei's
+method" under any framing — that attribution (present in an earlier
+revision of this section) was wrong, not just under-cited.
+
+**Implemented** (`PartitionDispersion`, `obikphylo/src/siblings/algorithms/pairwise.rs`)
+as an independent method-of-moments estimator, unrelated to any specific
+published procedure: pools substitution counts by **partition** rather
+than by genome pair, during the same `reduce_pairwise` pass that builds
+`PairwiseTally` (no second scan).
+
+For partition `i`: `n_i` = substitutions pooled over every genome pair,
+`L_i` = eligible loci pooled over every genome pair, `R_i = n_i / L_i`.
+Modeling among-site rate heterogeneity the same way as the `+Γ` correction
+itself (a `Gamma(α, α)`-distributed, mean-1, multiplicative rate shared by
+every locus in a partition — the classical Poisson-Gamma/negative-binomial
+mixture, the general identity behind gamma-rate-heterogeneity corrections,
+also behind Uzzell & Corbin's (1971) original observation that substitution
+counts across sites/regions are over-dispersed relative to Poisson):
+
+\[
+\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
+\]
+
+Weighting each partition's squared deviation by its own `L_i` cancels the
+Poisson term before attributing what's left to `α`:
+
+\[
+\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
+V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
+\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
+\]
+
+where `\bar L` is the mean partition size. Returns "no estimate" (falls
+back to the uncorrected formula, warns) when fewer than 2 partitions have
+data, `\hat\mu \le 0`, or `V` doesn't exceed the Poisson floor
+`\hat\mu/\bar L` — no detectable over-dispersion, `α` would be unbounded.
+
+**Caveat, stated explicitly rather than left implicit**: unlike every
+closed-form correction in `snp_distance.rs` (each verified line-by-line
+against `ape`'s `dist_dna.c`, and now also against Jin & Nei 1990 directly
+for the base `+Γ` formula), this estimator is derived from first
+principles (the general Poisson-Gamma/negative-binomial identity) with no
+primary-source procedure behind it at all — not Jin & Nei's (confirmed
+above), and Wilson et al. (1989), the paper they point to instead, hasn't
+been read/verified either. Mathematically self-consistent (re-derived and
+checked, not guessed), but a from-scratch method, not a literature
+implementation. If `--gamma-shape` needs a value with a literature
+pedigree rather than an estimated one, Jin & Nei's own stated default,
+`α = 1` (`--gamma-shape 1`), is the better-supported choice today.
+
+Deliberately **not** gated by `--sankoff-ratio-ceiling` the way
+`base_pair_tally` is (same precedent as `cardinality_tally` — see its own
+doc comment): that filter excludes individual saturated *pairs* from a
+composition estimate computed once at the very end, from the complete
+`PairwiseTally`; the partition axis needed here only exists transiently,
+one partition at a time, while `PairwiseTally` is still being built — long
+before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
+is known. `--exclude-genome` isn't applied either, matching
+`reduce_pairwise`'s own raw per-pair fold.
+
+**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
+`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
+"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
+`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
+needs the true *directional* per-pair base co-occurrence matrix
+(`PairwiseTally` only keeps the symmetrised substitution counts
+`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
+why that loses exactly the compositional-asymmetry information LogDet
+exists to detect), `snp-tajima-nei` needs each genome's *own* base
+composition (not the pair-pooled estimate the formulas below use), and
+`snp-f84`/`snp-hky85` had no formula independently verified against a
+primary source at implementation time (unlike every formula below, checked
+line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
+`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
+a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
+`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
+accumulator — not an architecture change.
+
+### Exact formulas (implemented, 2026-08-28)
+
+Sufficient statistic, per genome pair `(i, j)`, from
+`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
+`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
+
+- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
+  (pyrimidine transitions)
+- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
+  (Kimura's two transversion categories)
+- \(n_{shared}\): loci where both genomes agree
+- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
+  loci for the pair)
+- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
+  \(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
+  (both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
+  standard pairwise estimator, not a whole-index average)
+
+Derived proportions used below:
+
+\[
+p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
+P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
+Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
+Q_1 = \frac{n_{tv1}}{L}, \quad
+Q_2 = \frac{n_{tv2}}{L}, \quad
+P_1 = \frac{n_{ts1}}{L}, \quad
+P_2 = \frac{n_{ts2}}{L}
+\]
+
+Every formula below was checked term-by-term against `ape`'s own
+`src/dist_dna.c` (not re-derived from memory) before being ported to
+`obikphylo::siblings::algorithms::snp_distance`.
+
+**`snp-raw`** — uncorrected p-distance:
+
+\[
+d_{raw} = p
+\]
+
+**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
+dropping transitions, which saturate first, *is* the correction):
+
+\[
+d_{tv} = Q
+\]
+
+**`snp-jc`** (Jukes-Cantor, JC69):
+
+\[
+d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
+\]
+
+**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
+
+\[
+d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
+\]
+
+**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
+\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
+
+\[
+d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
+\]
+
+**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
+
+\[
+d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
+\]
+
+**`snp-t92`** (Tamura 3-parameter), with GC content
+\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
+\(a_2 = 1-2Q\):
+
+\[
+d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
+\]
+
+**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
+\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
+
+\[
+k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
+k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
+k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
+\]
+
+\[
+w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
+w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
+w_3 = 1 - \frac{Q}{2g_R g_Y}
+\]
+
+\[
+d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
+\]
+
+**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
+weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
+each such term with the same weight applied to
+\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
+substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
+recovering the uncorrected formula exactly). E.g. for JC:
+
+\[
+d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
+\]
+
+Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
+(including K80's two-term form — algebraically identical to the generic
+substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
+both symbolically and numerically before simplifying the implementation to
+share one `corrected_log` helper across every model rather than
+special-casing K80). K81/T92/TN93's gamma branches follow the same
+mechanical substitution but weren't independently checked against an
+`ape`-equivalent reference for those three specifically — flagged here, not
+silently assumed correct.
+
+### α-estimation literature survey and design discussion (2026-09-11)
+
+Follow-up discussion after implementing `PartitionDispersion::estimate_alpha`
+(previous section), triggered by the user pointing out that the literature
+conflates two distinct problems: *estimating α from an alignment* versus
+*using a given α in a Jin-Nei-style corrected distance*. Jin & Nei (1990)
+itself is squarely in the second camp — α is a user-supplied parameter
+there, not something their paper estimates (confirmed by reading the paper
+directly, see previous section). This section catalogs the actual
+α-estimation literature and records which approaches were considered and
+why each was or wasn't adopted.
+
+**Bibliography** (methodological papers, not exhaustive users of `+Γ`):
+
+| Method | Reference | Needs a tree? | Estimates "true" α or a task-specific one? | Cost |
+|---|---|---|---|---|
+| ML, continuous Γ (foundational) | **Gu, Fu & Li (1995)**, MBE 12:546–557 — read in full | yes, topology fixed a priori | true α (+ θ jointly, invariant+Γ) | very high, ≤5–6 taxa in practice |
+| ML, discrete Γ | Yang (1993/94), MBE 10:1396; JME 39:306 | yes | true α | high |
+| Parsimony-based ML on inferred changes | Yang & Kumar (1996), MBE 13:650 | yes, equal branch lengths assumed | true α, biased when branches unequal | low |
+| Corrected substitutions/site → NB fit | **Gu & Zhang (1997)**, MBE 14:1106 — read in full | yes, + ancestral reconstruction | true α, close to ML accuracy | low (given a tree) |
+| Topology-optimizing grid search | **Guindon & Gascuel (2002)**, MBE 19:534 — read in full | yes, but built *from* the candidate distances themselves (BIONJ) | **not** true α — deliberately biased *upward* for topological accuracy | low–moderate (grid × tree build) |
+| Free (non-parametric) rate distribution | Susko et al. (2003), Syst. Biol. 52:594 | yes | tests the Γ assumption itself | high |
+| Parsimony counts/site → NB fit | **Wakeley (1993)**, JME 37:613 — read in full | yes, NJ tree + Fitch parsimony ancestral reconstruction | true α, biased *upward* by parsimony (conservative) | low (given a tree) |
+| Poisson-Gamma/negative-binomial, foundational | Uzzell & Corbin (1971), Science 172:1089 — **not read**, only seen via citations | unknown — likely also count-per-site/tree-based, unverified | true α (indirect, via NB overdispersion) | unknown |
+| Bayesian posterior over α | BPP/MrBayes/BEAST2/RevBayes | yes | posterior, not point estimate | very high |
+
+**Correction (2026-09-11, after reading Wakeley in full):** an earlier
+revision of this table filed Wakeley (1993) under "no tree needed" —
+wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
+1987, on the real mtDNA data; coalescent-simulated for the simulation
+study), reconstruct ancestral states at internal nodes by Fitch (1971)
+parsimony, count the *minimum* number of changes per site this implies,
+then fit either a two-rate Poisson mixture or a gamma-distributed-rates
+model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
+Johnson & Kotz 1969) to the resulting per-site count distribution — the
+same general "tree + ancestral reconstruction + per-site counts + NB fit"
+shape as Gu & Zhang (1997), just using raw parsimony counts directly
+rather than Gu & Zhang's multiple-hit-corrected `k̂`. Confirmed by
+simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
+mean *and* (more severely) the variance of per-site change counts, which
+biases `α` **upward** (toward apparent uniformity) — consistent with Gu &
+Zhang's own citation of Wakeley for exactly this bias, and with the
+general "parsimony overestimates α" pattern noted throughout this
+literature. Real hypervariable-region-1 mtDNA data gave `α̂ ≈ 0.44–0.60`
+per data set, `0.47` combined — despite the conservative bias, still
+solidly in the "strong heterogeneity" range, which is the paper's own
+point: the method is biased but usefully conservative, not useless.
+
+**Where `PartitionDispersion::estimate_alpha` actually sits**: nowhere in
+this table's tree-dependent rows. It shares only the *abstract
+mathematical identity* (Poisson-Gamma mixture ⟹ negative binomial
+over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
+genuinely different, tree-free and ancestral-reconstruction-free route:
+pairwise genome-to-genome substitution counts pooled by *partition*
+instead of per-site counts pooled across an alignment's columns after
+ancestral state reconstruction. As far as this survey has established
+(Uzzell & Corbin not read, so not fully ruled out), **no tree and no
+ancestral reconstruction of any kind** appears to be a genuinely
+distinguishing property of `PartitionDispersion`, not something it
+inherits from prior art — consistent with it being an independent
+derivation rather than a literature implementation (see previous
+section's caveat, now on firmer footing).
+
+**Why a tree-free estimate is even valid — the identifiability question.**
+Raised directly by the user after reading Gu, Fu & Li (1995), which states
+explicitly: *"one cannot estimate the rate heterogeneity when the number
+of sequences is <3."* This is a real non-identifiability result, not a
+practical inconvenience: with only 2 sequences, a site's history is
+summarized by one binary observation (differ / don't differ), collapsed
+across all sites into two aggregate numbers (transition and transversion
+proportions) — divergence time and among-site rate variance are
+confounded in that pair of numbers, with no way to separate them. Every
+method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
+sequences *sharing one evolutionary history* (a tree) specifically to get
+multiple independent looks at the *same* site's rate across different
+lineages, which is what identifies its variance.
+
+`PartitionDispersion` doesn't violate this, because it isn't estimating
+the same quantity. The literature above estimates rate variation **across
+sites, within one shared tree**. `PartitionDispersion` estimates rate
+variation **across partitions, pooled over every genome pair in the
+index** — each genome pair stands in for one independent draw of a
+partition's relative rate, the same identifying role multiple lineages
+play in the tree-based methods, just substituting "genome pair" for
+"lineage" and "partition" for "site." With `P` genome pairs (potentially
+in the hundreds here) all contributing to every partition's pooled count,
+the ≥3-samples identifiability requirement is met by the pair count, not
+by tree depth — closer in spirit to Jin & Nei's own original motivating
+scenario (comparing several genomic *regions* across one fixed panel of
+taxa) than to single-tree site-rate estimation.
+
+This substitution carries its own assumption, which must be named rather
+than left implicit: that a partition's *relative* rate (fast/slow) is
+reasonably stable **across genome pairs**, not just across sites within
+one pair — the direct analogue of the "no lineage-specific rate variation,
+only site-specific" assumption every tree-based `+Γ` method already makes,
+just moved from lineage→pair and site→partition. If this breaks — e.g. an
+index mixing very closely related and very divergent genome pairs, where a
+partition's saturation behavior differs qualitatively between the two —
+the estimate could be misled in a way a tree-based method would at least
+have the topology to detect and a tree-free, pool-everything method
+cannot. Not yet tested against real data with strongly heterogeneous
+pairwise divergence; worth keeping in mind as the main open validity
+question for `--gamma-shape auto`, not the citation question (now settled)
+this whole discussion started from.
+
+**Gu, Fu & Li (1995), read in full — the foundational ML method, not a
+candidate to implement.** Models rate variation as invariant+Γ (`θ` =
+proportion of invariant sites, `α` = gamma shape among the variable sites),
+and derives the exact joint likelihood over nucleotide *configurations*
+across all `n` sequences on a **fixed, a-priori-known tree topology** —
+for 3 sequences this is closed-form (Jukes-Cantor), for `n` sequences it's
+a sum over `4^n` configurations weighted by coefficients tied to the
+specific topology's branching structure (their eq. 32–34), with branch
+lengths, `α`, and `θ` jointly optimized by Newton-Raphson (Hessian over all
+free parameters). The site-rate integral has the same closed form
+`E = θ + (1-θ)(1+D/α)^{-α}` that underlies `+Γ` distance corrections
+generally (the same mechanism as `corrected_log`, not a coincidence — Jin &
+Nei's own formula is the `θ=0` special case applied to a pairwise `D`
+rather than a whole-tree one). Explicitly stated by the authors as
+infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
+candidate for obikmer's typical genome counts. Notable contribution worth
+keeping in mind regardless: `ρ = (1+θα)/(1+α)` is shown to be a more robust
+summary of rate heterogeneity than `α` alone whenever `θ` isn't negligible
+— `α` and `θ` are confounded (nearly-invariant sites can come from either a
+small `α`'s left tail or from a nonzero `θ`), so `α` alone can be very
+unstable while `ρ` stays well-behaved. `PartitionDispersion` has no `θ`
+term at all (no invariant-site component), so this confound doesn't arise
+for it the same way, but it's worth remembering if an invariant+Γ variant
+is ever considered.
+
+**Gu & Zhang (1997), read in full — rejected for now.** Their procedure:
+(1) a tree with least-squares branch lengths must already exist; (2)
+ancestral states are reconstructed at every internal node (they use Zhang &
+Nei 1997's likelihood method, but note plain parsimony would work, just
+less accurately); (3) per site, branches are split into "changed"/"unchanged"
+given the ancestral reconstruction, and the *expected* substitution count
+`k̂` (corrected for multiple hits) is obtained by solving their eq. (7)
+(or eq. 12 for the generalized model) numerically — a root-find per site,
+not a closed form; (4) the site-level `k̂` values (real-valued, not
+integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
+get α — again no closed form, numerical optimization. This is a real
+per-site pipeline requiring a tree *before* distances/α can be computed —
+the reverse of obikmer's current dependency order (`snp_distance` computes
+distances, which `--nj`/`--upgma` only turn into a tree *afterward*). Their
+own answer to this chicken-and-egg problem is an iterative
+distance→tree→α→distance loop, which they themselves flag as expensive.
+Adopting this would mean a new ancestral-reconstruction subsystem (even a
+parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
+ML fit — a much bigger addition than `PartitionDispersion`, not something
+to build without a concrete need beyond "closer to Gu&Zhang's own
+simulated accuracy than our moment estimator."
+
+**Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.**
+Their **Q criterion**: for a candidate α, build a tree (they use BIONJ) from
+the α-corrected distance matrix; for every internal branch, group taxa into
+the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
+distances \(\bar\delta_{AB}\), \(\bar\delta_{CD}\), etc., and let `S ≤ M ≤ L`
+be the three pairings' sums (`{AB,CD}`, `{AC,BD}`, `{AD,BC}`); the branch's
+reliability score is `Q_branch = L − M` (zero when the four points are
+perfectly tree-additive). The whole-tree `Q` is the mean over internal
+branches (negative/zero branches excluded). `α* = argmin_α Q(α)` over a
+grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
+sensitivity is highest). Complexity: `O(n²l)` once for the base
+frequencies, then `O(n²r)` for the `r` candidate distance matrices, then
+`O(n³r)` worst case for building `r` trees and evaluating `Q` on each
+(same order as the tree-building step itself, so "free" relative to it).
+
+Central finding, **the opposite direction from an initial misreading in
+this discussion**: `α_opt` (Guindon-Gascuel's topology-optimal value) is
+**always ≥ the true α**, not the reverse — underestimating rate
+heterogeneity (i.e., picking a larger α than reality) reduces the variance
+of distance estimates and thereby improves NJ/BIONJ topological accuracy,
+especially when the molecular clock roughly holds (where `α_opt → ∞`, i.e.
+no correction at all is topologically best). This is an *empirical
+regularity* observed across their tested conditions (η ∈ {0.5, 2.0}, true
+α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
+guarantee would be another unverified-claim mistake of the kind this whole
+discussion has been correcting.
+
+**Proposed integration (not yet implemented)**: since `α_true ≤ α_opt`
+empirically, `PartitionDispersion::estimate_alpha()`'s output could seed
+the *lower bound* of Guindon-Gascuel's grid (search `[α̂, ∞)` instead of
+`[0.1, 5000]`), narrowing the grid substantially and reusing the existing
+estimator rather than needing an independent "true α" source — this also
+sidesteps Gu & Zhang's tree-first dependency problem entirely, since
+Guindon-Gascuel's own grid already builds a tree per candidate. Still
+requires: (a) a **BIONJ** implementation (see below), (b) the Q-criterion
+subtree-grouping logic, (c) a grid-search driver. Not started.
+
+**BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.**
+Searched crates.io directly (API query for "bionj": 0 results) and read the
+docs.rs pages of the most plausible candidates: `phylotree` (builds/reads/
+manipulates trees but has no distance-matrix reconstruction — no NJ, no
+BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
+after checking the actual docs), `speedytree` (NJ only — Canonical/RapidNJ/
+Hybrid variants, explicitly no BIONJ), `nj` (plain NJ only), `phylo` (no
+distance-based reconstruction at all). `obikphylo::neighbor_joining`
+(`tree.rs`) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
+how branch lengths are weighted during agglomeration (accounts for
+distance-estimate variance, not just the sum), a modest delta over an
+existing NJ implementation rather than a rewrite, but would need to be
+written in-house if the Guindon-Gascuel integration above is ever pursued
+(their own paper notes other tree-building methods gave similar results in
+their simulations, so plain NJ may be an acceptable substitute if BIONJ
+proper is judged not worth the effort).
+
+### Sharing one sample across algorithms (`--sankoff`/`--tnt`/`--phyg`/`--iqtree` + `snp-*` `--distance`) — implemented 2026-09-11
+
+**Correctness bug, not just a performance one.** `--sankoff`/`--tnt`/`--phyg`/
+`--iqtree` (via `SiblingExt::sankoff_bundle`) and a `snp-*` `--distance`
+(via `SiblingExt::snp_distance`) always consume the *exact same* selection
+parameters when requested together in one `obikmer phylo` invocation
+(`n`/`free_loss`/`no_ambiguity`/excluded-set/`entropy_bias` — the CLI has
+no way to give them different values in one run, see `cmd/phylo/args.rs`).
+Before this fix, each independently called `sample_index`, and
+`sample_layer`'s `rand::rng()` (`subsample.rs:235`) is a **thread-local
+generator that advances across calls, not reseeded each time** — so the
+second call silently drew a *different* random sample of sites than the
+first, even with identical parameters. Verified empirically (throwaway
+8-genome index): running `snp-jc` alone vs. combined with `--sankoff`
+produced visibly different distance matrices from the same index/params.
+This defeated the actual point of combining these flags — running several
+algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a `snp-*` distance) on
+one *identical* site selection for direct comparison.
+
+**Fix**: `SankoffBundle` (`sankoff.rs`) now retains its `PairwiseTally`/
+`PartitionDispersion` internally (previously consumed into `raw`/
+`base_pair_tally`/`cardinality_tally` and dropped) and exposes
+`SankoffBundle::snp_distance(kind, gamma_shape)`, which computes the
+matrix from that *same* already-sampled tally — no second `sample_index`
+call. The shared post-sampling logic (the `--gamma-shape` support check,
+`alpha` resolution, final matrix build — previously all inline in the
+`snp_distance()` free function) was factored into
+`pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)` in
+`snp_distance.rs`, called by both the standalone `snp_distance()` (after
+its own fresh sampling) and `SankoffBundle::snp_distance` (reusing the
+bundle's). `cmd/phylo/mod.rs` keeps the `Option<SankoffBundle>` alive past
+the Sankoff-family `if` block and, when a `snp-*` `--distance` is also
+requested, calls `bundle.snp_distance(...)` instead of
+`cache.snp_distance(...)` whenever a bundle was built — since both branches
+are driven by the same `args.*` fields, the parameters trivially always
+match when both fire; no runtime "do the params match" check needed.
+Verified end-to-end: the reused path logs `(reusing the Sankoff bundle's
+sample)` and the `snp_distance` stage timer reads `0ms` (`formula(kind)`
+is `O(n²)` post-processing, no I/O), vs. ~150-230ms for a fresh sample on
+the same tiny test index.
+
+#### Explicit `--session` (implemented 2026-09-12)
+
+The in-process fix above only covered reuse *within one command*.
+`--session DIR` (`cmd/phylo/args.rs`) names a directory (outside the
+index) that persists the `snp-*` `--distance` sample — the
+`PairwiseTally`/`PartitionDispersion` pair `sample_index` would otherwise
+rebuild from scratch — across *separate* `obikmer phylo` invocations.
+
+**Crate placement, as planned**: a new **`obiksession`** crate
+(`src/obiksession/`), domain-agnostic — no `PairwiseTally`/"site" concept
+anywhere in it. `Session::open(dir, params: &[u8], force: bool)` handles
+directory lifecycle and the params-conflict check against an opaque byte
+blob; `Session::store(name, bytes)`/`Session::restore(name) -> Option<Mmap>`
+handle a checksummed artifact cache. `obikphylo`
+(`siblings/algorithms/session_cache.rs`, shared by `snp_distance.rs` and
+`sankoff.rs` — see below) is the consumer: it serializes its own
+`PairwiseTally`/`PartitionDispersion`/`SnpAlignment` to bytes (via `rkyv`)
+and hands them to `obiksession`, which never sees their type.
+
+**Serialization**: `rkyv` 0.8.18, added as planned. One deviation from the
+original zero-copy pitch, scoped down deliberately: artifacts are restored
+via `rkyv::from_bytes` (full owned deserialize) rather than `rkyv::access`
+(zero-copy over the `mmap`). True zero-copy would require every
+`PairwiseTally` query method (`categories`, `base_freq`, `pair`, ...) to
+work generically over `Archived<PairwiseTally>` as well as the owned type
+— a separate, larger change not needed to get the actual win (skipping
+`sample_index`'s expensive re-scan of the sibling annex; deserializing an
+already-in-memory-sized count array is comparatively cheap CPU, not I/O).
+`#[derive(Archive, Serialize, Deserialize)]` was added to `PairStats`,
+`PairwiseTally`, and `PartitionDispersion` (`pairwise.rs`) for this.
+
+**Startup behavior, as designed**: `Session::open` creates `DIR` if
+missing (fresh, params recorded); if `DIR` already holds different saved
+params, `cmd/phylo` reports a hard error and exits rather than silently
+preferring one side, unless `--session-force` is given (which discards the
+directory's cached artifacts and starts over under the new params) —
+implements exactly the "explicit escape hatch, not silent override"
+decision from the design discussion. `obiksession` itself doesn't decode
+either side's bytes to produce a field-by-field diff (true to "opaque
+blob," left to whoever needs it); `cmd/phylo`'s conflict message is
+currently a single generic sentence listing which flags could be the
+cause, not a computed diff — an honest v1 simplification, not a limitation
+of `obiksession` itself.
+
+**Per-artifact caching, as designed**: only `PairwiseTally`/
+`PartitionDispersion` are cached (`snp_distance.rs`'s `restore_tally`/
+`store_tally`) — `ratio_ceiling`/`gamma_shape` remain pure post-processing
+over whatever tally is in hand, cached or fresh, exactly as planned.
+
+**Locking, reconsidered and kept as the whole-session `DirLock`.** First
+written up as a corner cut ("per-entry locking was sketched, not
+implemented"). On actually working through the concrete race, whole-session
+locking turned out to be the *correct* choice, not a simplification:
+`--session-force` resets a session by deleting its whole `artifacts/`
+directory. If the directory lock were only held briefly (during
+`Session::open`, then released so a long-running `Session` doesn't block
+other processes from touching *different* artifacts), a second process
+calling `open(..., force: true)` could acquire the lock, see mismatched
+params, and wipe `artifacts/` **while a first process's already-open
+`Session` is mid-checkpoint** — genuine corruption risk, not just wasted
+work. Holding the lock for the `Session`'s entire lifetime makes that
+sequence impossible: a `force` reset must wait for every currently-open
+`Session` on that directory to be dropped first. The concurrency this
+gives up (two *different* processes reading/writing *different* artifacts
+of the same session at once) isn't exercised by anything today — only one
+code path (`snp_distance`/`sankoff_bundle`, sequentially per invocation)
+uses sessions at all.
+
+**Per-layer chunked/resumable dumps — implemented.** `subsample::sample_index`
+gained a `skip_layers: usize` parameter (skip the first N (partition,
+layer) pairs, in its own deterministic iteration order, entirely — no
+I/O, no `on_layer` call) and its `on_layer` callback gained a `raw_index`
+argument (that pair's 0-based position in the same order), so a caller can
+tell exactly how far a run has gotten. `snp_distance`/`sankoff_bundle`
+checkpoint their tally/dispersion (and, for `sankoff_bundle`, the
+in-progress `SnpAlignment`) to the session every
+`session_cache::CHECKPOINT_INTERVAL_LAYERS` (8) layers or
+`CHECKPOINT_INTERVAL` (30s) of wall time, whichever comes first — bounding
+how much completed scanning a crash can lose, without paying a full
+tally-rewrite on every one of what can be thousands of layers. A `progress`
+artifact records how many layers are reflected in the checkpoint, with a
+`PROGRESS_COMPLETE` (`u64::MAX`) sentinel once every layer has been
+processed (distinct from any real layer count, so "finished" is never
+confused with "stopped after N layers where N happens to equal the total").
+
+Verified with a real kill: built a 10-genome/64-partition index, ran
+`--distance snp-jc --session DIR` under `timeout 0.4`, confirmed a
+checkpoint at `progress=8` (exactly `CHECKPOINT_INTERVAL_LAYERS`) was on
+disk, then re-ran the same command and confirmed it resumed from layer 8
+and completed successfully — 0-effort verification that the mechanism
+itself is real, not just plausible-sounding.
+
+**Important correction from that same test — resume is *not* bit-reproducible,
+and shouldn't be.** The output matrix from the resumed run differed from
+an uninterrupted control run against the same index/params. Root cause:
+`sample_layer`'s random draws come from `rand::rng()`, a per-process,
+OS-seeded generator with no continuity across a process boundary — the
+layers processed before a kill and the layers processed after a resume
+come from two independent random streams. An earlier revision of this doc
+(and of the code comments) claimed resume "reproduces the exact same
+sample an uninterrupted run would have" — that's wrong, and was corrected
+after the user pointed out the actual reasoning error: true randomness has
+no notion of continuity in the first place, so there was never a
+correctness requirement for a resumed run to match an uninterrupted one
+bit-for-bit. The sample produced after a resume is exactly as legitimate a
+random draw as one from an uninterrupted run — just a *different* one,
+same as running the command twice without `--session` already gives two
+different samples today. That variance is explicitly wanted (see below),
+not a defect to fix. Comments in `subsample.rs`/`session_cache.rs` were
+corrected to state this plainly instead of the false reproducibility claim.
+
+**Explicitly declined: deterministic/seeded sampling.** Raised as a
+possible fix for the above (seed `sample_layer`'s RNG from a per-session
+value plus `(partition, layer)`, making a resume reproduce bit-identical
+results, and, if applied everywhere, making *any* two runs with identical
+parameters produce identical samples). The user explicitly declined this
+for now: repeated runs are relied on to measure sampling variance, and a
+`--seed` flag would need its own explicit design if wanted later — this is
+not an accidental gap, it's a stated preference to revisit only if asked.
+
+**`SankoffBundle` integration — implemented.** `sankoff_bundle` accepts
+the same `session: Option<&Session>` and uses the *same* artifact names
+(`session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}`)
+as `snp_distance`, plus its own `ALIGNMENT_ARTIFACT` (`sankoff.rs`) folded
+in lockstep with the tally at every checkpoint (same `on_layer` call
+produces both, so they can never drift out of sync). Consequence verified
+directly: a tally checkpointed by `--sankoff --session DIR` is restored by
+a *later, separate* `--distance snp-k2p --session DIR` invocation with no
+`--sankoff` at all, and vice versa — cross-consumer reuse, not just
+same-command reuse, exactly the synergy this design aimed for.
+
+**Per-artifact caching, as designed**: only the tally/dispersion (plus,
+for `sankoff_bundle`, the alignment) are cached — `ratio_ceiling`/
+`gamma_shape` remain pure post-processing over whatever tally is in hand,
+cached or fresh, exactly as planned.
+
+**Still not done — genuine follow-up, not a justified tradeoff:**
+**exhaustive-sample caching for `--subsample`-varying exploration.** The
+design discussion's suggestion — cache the exhaustive (`n = None`) sample
+once and subsample *from* it on each run instead of one session per exact
+`n` — wasn't implemented; a session today is still scoped to one exact
+parameter tuple, `--subsample` included.
+
+### Output format: PHYLIP-relaxed by default for the distance matrix
+
+**Implemented.** The primary distance-matrix output
+(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
+default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
+T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
+label truncation, since genome labels here routinely exceed it), a `--csv`
+flag opts back into the current CSV format, PHYLIP-strict is a possible
+future addition (not now). This changes the *default* output of every
+existing `--distance` value (jaccard, hamming, bray-curtis, ...), not just
+the new `snp-*` ones — accepted explicitly (pre-release, single developer
+user, no external consumers to break). Scoped to the distance matrix only:
+`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
+their existing CSV-only format.
+
 ## References
 
 The Mash mutation-rate model this discussion contrasts with:

DevDocMD/theory/indexing.refs.md

@@ -3,8 +3,8 @@
 
 ## Code couvert
 
-- `obikpartitionner/src/partition.rs` — routage par hash de minimiseur, choix des paramètres
-- `obikpartitionner/src/lib.rs` — structure KmerPartition, nombre de partitions
+- `obikindex/src/partition/partition.rs` — routage par hash de minimiseur, choix des paramètres
+- `obikindex/src/partition/mod.rs` — structure KmerPartition, nombre de partitions
 
 ## Notes
 

Le_bug_des_A.md

@@ -1,43 +0,0 @@
-Voici la version corrigée :
-
----
-
-**Bug** : dans `base_pair_tally`, toutes les transitions/comptes depuis/vers A valent 0 dans `_sankoff_params.yaml`, alors que C/G/T sont corrects.
-
-**Contexte** : obikmer, pipeline phylogénétique `--sankoff`. L’index est construit sur 20 génomes bactériens. Même symptôme sur un jeu de 100 génomes de plantes : A est toujours à 0.
-
-**Fichier clé** : `src/obikphylo/src/siblings/sankoff_bundle.rs` (Pass A + Pass B).
-
-**Ce qui a été vérifié** :
-- Le fichier de sortie `_sankoff_params.yaml` montre bien `composition_transitions` avec A à 0 partout.
-- L’index contient bien des familles avec A (`mask.has(0) == true`), et même des familles où A co-existe avec d’autres bases (`mask == 0b0011` par ex.).
-- Un k-mer propriétaire de famille avec `mask == 0b0001` (A seul) a été identifié : forward `GAACAAGAGATCTCGATCTTGTCTACAAGGA`, revcomp `TCCTTGTAGACAAGATCGAGATCTCTTGTTC`.
-- Le diagnostic CLI sur l’index réel donne :
-  - Pass A : `a_pairs=623342 a_snp=623342 a_shared=0 a_both_a=0`
-  - Pass B : `families_with_a=22965521 a_single_form_genomes=22913238 a_included_pairs=0 a_same_incremented=0 bp_same=[0, 96389, 222720, 277909] bp_counts[0]=[0, 0, 0, 0]`
-
-**Interprétation** : A est fréquemment en `single_form` (mask == 1) chez certains génomes, mais **jamais simultanément** chez deux génomes différents dans la même famille. Donc toutes les paires “avec A” sont 100% SNP → ratio = 1.0 > `ratio_ceiling=0.5` → toutes exclues par le filtre `included`. C’est pourquoi `bp_same[0]` et `bp_counts[0][*]` restent à 0.
-
-**Point crucial** : le bug n’apparaît **que sur l’index compacté sparse**. Sur le même index avant compaction (matrice dense `matrix.pbmx`), `--sankoff` produit des tallies corrects pour A. Dès qu’on compacte avec `pack --sparse`, A disparaît.
-
-**Vérifications supplémentaires (diagnostic sparse)** :
-- La compaction `pack --sparse` produit une matrice `PersistentSparseBitMatrix` dont le contenu est **strictement identique** à la matrice dense d'origine : vérification exhaustive coordonnée par coordonnée sur **1 804 774 880 cellules** (512 partitions × 2 layers), **zéro différence**.
-- `fill_row` et `fill_sub_matrix` (les deux chemins de lecture utilisés par le pipeline phylogénétique) restituent les mêmes bits sur dense et sparse.
-- **Conclusion** : le bug n'est **pas** dans la compaction sparse elle-même, ni dans les chemins de lecture individuels. La structure stocke correctement A, C, G, T.
-
-**Conséquence logique** :
-Si les matrices sont identiques mais que le résultat final diffère, le bug se situe dans l'**intersection** des informations — c'est-à-dire dans le code qui **combine** les lectures des deux matrices (ou qui transforme les résultats bruts en tallies). Deux endroits possibles :
-1. **Le scan `sankoff_bundle`** (`family_scan.rs` + `sankoff_bundle.rs`) : la boucle qui lit les matrices, construit `genome_mask`, et accumule `bp_counts` / `same`. C'est l'étape d'intersection proprement dite.
-2. **La conversion des tallies en YAML** (`obikmer/src/cmd/phylo/sankoff.rs`) : moins probable, mais possible si quelque chose sélectionne/filtre les transitions avant écriture.
-
-**Hypothèse la plus probable** : bug dans la résolution cross-partition lors de la construction de l'annex sibling (`build_sibling_annex`). A (bit 0) serait systématiquement manquant ou mal résolu quand on interroge les variants d'une famille depuis une partition différente. À vérifier dans `src/obikphylo/src/siblings/build.rs` et `src/obikphylo/src/siblings/cache.rs` (`PartitionCache::find` / `find_presence_batch`).
-
-**Prochaine étape logique** :
-1. Inspecter `build_sibling_annex` pour voir si les variants avec base A sont bien générés et bien recherchés dans `cache.find`.
-2. Vérifier `PartitionCache::find` et `resolve_layer_hits` pour un éventuel biais contre le bit 0.
-3. Si besoin, ajouter un diagnostic ciblé (compteurs par base) **uniquement** dans `cache.rs` ou `build.rs`, pas dans `sankoff_bundle.rs` qui est déjà propre.
-
-**Contraintes** :
-- Ne pas modifier `sankoff_bundle.rs` davantage.
-- Ne pas toucher à git.
-- Faire des diagnostics minimaux et ciblés.

UserDocMD/architecture.md

@@ -20,7 +20,7 @@ An index directory is organized as `KmerIndex → partitions → layers`, with a
 
 ## Parallel execution and NUMA awareness
 
-Partition-level work (index construction, `merge`, `filter`, `reindex`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
+Partition-level work (index construction, `merge`, `filter`, `convert`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
 
 - On a multi-socket / multi-NUMA-node machine, one thread pool is pinned per NUMA node, and each partition is processed entirely by threads pinned to one node — keeping the memory a partition touches local to that node's DRAM. This matters because touching kmer data across NUMA nodes without pinning can degrade throughput by an order of magnitude or more on large multi-socket machines.
 - On a single-socket machine, Apple Silicon, or if hwloc cannot report NUMA topology, all cores are treated as one node with no pinning and negligible overhead — this is the default behavior on macOS.

UserDocMD/formats/index_layout.md

@@ -19,7 +19,7 @@ Each partition's surviving kmers are mapped to a dense range of integer slots by
 
 ## Evidence: exact vs. approximate
 
-Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`reindex`](../usage/reindex.md)):
+Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)):
 
 - **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
 - **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
@@ -51,6 +51,6 @@ Two verification modes are available, selected at build time (`index --approx`)
 
 A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
 
-Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`reindex`](../usage/reindex.md) one of the sources first if needed.
+Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`convert`](../usage/convert.md) one of the sources first if needed.
 
 `obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time.

UserDocMD/index.md

@@ -24,11 +24,10 @@ All functionality is exposed through a single binary, `obikmer`, organized as su
 | [`query`](usage/query.md) | Query an index with sequences and annotate matches |
 | [`dump`](usage/dump.md) | Dump indexed kmers as CSV |
 | [`annotate`](usage/annotate.md) | Add, update, or dump genome metadata |
-| [`phylo`](usage/phylo.md) | Compute pairwise evolutionary-distance proxies, trees, and phylogenetic exports |
-| [`name-tree`](usage/name-tree.md) | Translate a TNT/PhyG numeric-label tree export back to real taxon names |
+| [`phylo`](usage/phylo.md) | Compute pairwise genome distances, trees, and phylogenetic exports |
 | [`unitig`](usage/unitig.md) | Dump the unitigs of an index as FASTA |
 | [`estimate`](usage/estimate.md) | Estimate approximate-index parameters before indexing |
-| [`reindex`](usage/reindex.md) | Convert an index's evidence representation (exact ↔ approximate) |
+| [`convert`](usage/convert.md) | Convert an index's evidence representation (exact/approximate/hybrid), in place |
 | [`utils`](usage/utils.md) | Miscellaneous index maintenance and inspection utilities |
 | [`pack`](usage/pack.md) | Pack per-column matrix files into a single-file format |
 

UserDocMD/usage/convert.md

@@ -0,0 +1,29 @@
+# convert
+
+Convert an existing index's evidence representation in place, between exact, approximate, and hybrid.
+
+```bash
+obikmer convert INDEX (--exact-evidence | --approx-evidence BITS | --hybrid-evidence) [OPTIONS]
+```
+
+## Arguments
+
+| Argument | Description |
+|---|---|
+| `INDEX` | Index directory to convert (modified in place) |
+
+## Options
+
+Exactly one of the first three is required:
+
+| Option | Description |
+|---|---|
+| `--exact-evidence` | Convert to exact evidence (zero false positives) |
+| `--approx-evidence BITS` | Convert to approximate (fingerprint-only) evidence; `BITS` = fingerprint bits per slot (b) |
+| `--hybrid-evidence` | Convert to hybrid evidence (both exact and approximate bundles kept) |
+| `--evidence-bits BITS` | Fingerprint bits per slot (b) — required with `--hybrid-evidence` when the source index is currently exact; rejected otherwise (the source already fixes `b`) |
+| `-z, --findere-z Z` | Findere z parameter: number of consecutive stored kmers that must all match to confirm a hit. This does not shorten the indexed kmer length (fixed forever at `index` build time) — it extends the effective match window: on a k=31 index, `z=2` requires 32 consecutive matching bases, not 30 |
+| `--fp FP` | Target false-positive rate per z-window (e.g. `0.01`); derives `b` or `z` when one of them isn't given directly |
+| `--block-size N` | Block size for exact evidence's on-disk index (unitigs per block). Ignored when converting to pure approximate evidence. Default `1` |
+
+See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.

UserDocMD/usage/estimate.md

@@ -15,4 +15,4 @@ obikmer estimate [OPTIONS]
 | `--evidence-bits` | none | Fingerprint bits per slot (b) |
 | `--fp` | none | Target false-positive rate per z-window |
 
-Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `reindex --approx` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
+Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.

UserDocMD/usage/index_command.md

@@ -45,6 +45,6 @@ With `--approx`, evidence is stored as a compact **fingerprint** instead, tradin
 
 $$FP = \frac{1}{2^{b \cdot z}}$$
 
-where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`reindex`](reindex.md) to convert an existing index between the two representations.
+where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`convert`](convert.md) to change an existing index's representation afterwards.
 
 `z` must be strictly less than k: the effective indexed kmer length under approximate evidence is k−z+1.

UserDocMD/usage/name-tree.md

@@ -1,21 +0,0 @@
-# name-tree
-
-Translate a numerically-labelled tree export (TNT, PhyG, or any plain Newick file with bare `1`, `2`, `3`, … leaf labels) back to real taxon names, reading the label order from the FASTA that produced it.
-
-```bash
-obikmer name-tree TREE --fasta FASTA -o OUTPUT
-```
-
-## Arguments
-
-| Argument | Description |
-|---|---|
-| `TREE` | Tree file to translate — a TNT-style NEXUS export (`tree NAME = [&U] ...;`) or a plain Newick file |
-| `--fasta` | FASTA file whose record order gives the numeric taxon labels (1-based) — typically the `_sankoff.fasta`/`_snp.fasta` used to produce `TREE` |
-| `-o, --output` | Output NEXUS file path |
-
-## Output
-
-A NEXUS file with a `taxa` block, a `translate` table (numeric label → taxon name, from `--fasta`'s header order), and every tree found in `TREE`, topology unchanged — readable directly in FigTree, PearTree, `ape` (R), etc.
-
-`--tnt`'s and `--phyg`'s exports (see [phylo](phylo.md)) both number taxa `1..N` in the same order as the pseudo-alignment FASTA they were built from (`<prefix>_sankoff.fasta`), so pass that same file as `--fasta` here.