Implement session persistence and checkpointing

Introduces session persistence via the `--session` option, allowing sampled sets to be restored. Implements chunked dumps and shared checkpointing for alignment and tally artifacts, ensuring state restoration upon interruption. Refines sampling logic and introduces serialization mechanisms for state management.

.gitignore

@@ -30,23 +30,7 @@ data-stress
 *.bin.idx
 *.prsb
 Betula_exilis--IGA-24-33
-benchmark/genomes
+benchmark/run/
-benchmark/genomes_orig
-benchmark/simulated_data
-benchmark/specimen_index_presence
-benchmark/specimen_index_count
-benchmark/global_index_presence
-benchmark/global_index_presence_orig
-benchmark/global_index_presence_sav
-benchmark/all_specific
-benchmark/global_index_count
-benchmark/stats
-benchmark/reference_index
-benchmark/reference_dist
-benchmark/obikmer_dist
-benchmark/specific_index_count
-benchmark/specific_index_presence
-benchmark/query_data
 TNT
 phyg
 biblio

.kilo/plans/1778319532254-eager-panda.md

@@ -1,87 +0,0 @@
-# Plan d'amélioration technique - obiskio
-
-## 1. Contexte et objectifs
-- **Objectif** : Renforcer la robustesse, la maintenabilité et les performances de la crate `obiskio`.
-- **Priorités** : 
-  1. Gestion des erreurs
-  2. Optimisation de la mémoire du pool
-  3. Robustesse concurrente
-  4. Couverture de tests
-  5. Documentation
-
----
-
-## 2. Axes d'amélioration détaillés
-
-### 2.1 Gestion des erreurs
-- **Problème** : `SKError` ne couvre pas tous les cas (format invalide, taille maximale, CRC)
-- **Actions** :
-  - Ajouter variante `ParseError(String)` dans `src/error.rs`
-  - Valider les tailles de SuperKmer avant parsing
-  - Remplacer `expect()` par `unwrap_or_else` avec messages explicites
-  - Documenter chaque variante d’erreur dans le README
-
-### 2.2 Optimisation du pool de fichiers
-- **Problème** : `SKFilePool` utilise un `Vec<WriteEntry>` non contraint et n’effectue pas de nettoyage en cas d’erreur
-- **Actions** :
-  - Implémenter un `LimitedVec` avec limite stricte à `MAX_POOL_SIZE`
-  - Créer `clear_memory()` qui supprime les entrées orphelines
-  - Ajouter `evict_lru_threshold()` pour éviction proactive
-  - Introduire un `RwLock` pour les opérations de lecture massives
-
-### 2.3 Robustesse concurrente
-- **Problème** : Risque de deadlocks dans `SKFileWriter::write_batch()` et `SKFileReader::reopen_and_seek()`
-- **Actions** :
-  - Remplacer `Mutex` par `RwLock` pour les accès en lecture
-  - Ajouter un compteur de blocage et logs de timeout
-  - Utiliser `std::thread::park_timeout` pour débloquer
-  - Insérer `debug_assert!` sur les états invariants
-
-### 2.4 Couverture de tests
-- **Problème** : Absence de benchmarks, de tests de migration, de résilience de fichiers corrompus
-- **Actions** :
-  - Benchmarks I/O sur 10k+ SuperKmer avec `criterion`
-  - Tests de migration de version de fichier `.meta` → `.v2.meta`
-  - Tests de corruption volontaire (truncature, inversion de bits)
-  - Tests de stress sur pool saturation (100 threads)
-
-### 2.5 Documentation & exemples
-- **Actions** :
-  - Ajouter des examples dans chaque module (`# Examples`)
-  - Documenter la logique LRU avec diagrammes Mermaid
-  - Créer un guide « How to recover from eviction »
-  - Mettre à jour le `README.md` avec tableau des variantes d’erreur
-
----
-
-## 3. Plan d'exécution (Roadmap)
-
-| Sprint | Durée | Livrables clés |
-|--------|-------|----------------|
-| **S1** | 2 jours | Refactorisation `SKError`, ajout de tests unitaires |
-| **S2** | 3 jours | Implémentation `clear_memory()` + `LimitedVec` |
-| **S3** | 2 jours | Passage à `RwLock`, ajout de compteurs de blocage |
-| **S4** | 2 jours | Benchmarks + tests de migration |
-| **S5** | 1 jour | Documentation finale & mise à jour du README |
-
----
-
-## 4. Dépendances externes
-- Mettre à jour `niffler` vers la version 2.0 (performance compression)
-- Évaluer `bincode` vs `serde_json` pour les métas (I/O)
-- Ajouter dépendance `criterion` (dev‑dependencies)
-
----
-
-## 5. KPI de suivi
-- **Couverture de tests** : ≥85 % des chemins critiques
-- **Latence moyenne d’écriture** : ↓15 % après optimisation du pool
-- **Taux d’erreurs résolues** : 100 % des nouvelles variantes couvertes
-- **Temps de build CI** : ≤5 min pour l’ensemble des benchmarks
-
----
-
-## 6. Validation finale
-- Revue de code avec `cargo clippy -- -D warnings`
-- Analyse de toxicité avec `cargo deny open-source-licenses`
-- Vérification de la conformité aux standards de naming du projet

.serena/.gitignore

@@ -1,2 +0,0 @@
-/cache
-/project.local.yml

.serena/project.yml

@@ -1,169 +0,0 @@
-# the name by which the project can be referenced within Serena/when chatting with the LLM.
-project_name: "obikmer"
-
-# the encoding used by text files in the project
-# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
-encoding: "utf-8"
-
-# line ending convention to use when writing source files.
-# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
-# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
-line_ending:
-
-# The language backend to use for this project.
-# If not set, the global setting from serena_config.yml is used.
-# Valid values: LSP, JetBrains
-# Note: the backend is fixed at startup. If a project with a different backend
-# is activated post-init, an error will be returned.
-language_backend:
-
-# whether to use project's .gitignore files to ignore files
-ignore_all_files_in_gitignore: true
-
-# advanced configuration option allowing to configure language server-specific options.
-# Maps the language key to the options.
-# The settings are considered only if the project is trusted (see global configuration to define trusted projects).
-# See https://oraios.github.io/serena/02-usage/050_configuration.html#language-server-specific-settings
-ls_specific_settings: {}
-
-# list of additional paths to ignore in this project.
-# Same syntax as gitignore, so you can use * and **.
-# Important: quote patterns that start with `*`, otherwise YAML treats them as aliases.
-# Example:
-#   ignored_paths:
-#     - "examples/**"
-#     - ".worktrees/**"
-#     - "**/bin/**"
-#     - "**/obj/**"
-# Note: global ignored_paths from serena_config.yml are also applied additively.
-ignored_paths: []
-
-# whether the project is in read-only mode
-# If set to true, all editing tools will be disabled and attempts to use them will result in an error
-# Added on 2025-04-18
-read_only: false
-
-# list of tool names to exclude.
-# This extends the existing exclusions (e.g. from the global configuration)
-# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
-excluded_tools: []
-
-# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
-# This extends the existing inclusions (e.g. from the global configuration).
-# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
-included_optional_tools: []
-
-# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
-# This cannot be combined with non-empty excluded_tools or included_optional_tools.
-# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
-fixed_tools: []
-
-# list of mode names that are to be activated by default, overriding the setting in the global configuration.
-# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
-# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
-# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
-# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
-# for this project.
-# This setting can, in turn, be overridden by CLI parameters (--mode).
-# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
-default_modes:
-
-# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
-# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
-# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
-added_modes:
-
-# initial prompt for the project. It will always be given to the LLM upon activating the project
-# (contrary to the memories, which are loaded on demand).
-initial_prompt: ""
-
-# time budget (seconds) per tool call for the retrieval of additional symbol information
-# such as docstrings or parameter information.
-# This overrides the corresponding setting in the global configuration; see the documentation there.
-# If null or missing, use the setting from the global configuration.
-symbol_info_budget:
-
-# list of regex patterns which, when matched, mark a memory entry as read‑only.
-# Extends the list from the global configuration, merging the two lists.
-read_only_memory_patterns: []
-
-# list of regex patterns for memories to completely ignore.
-# Matching memories will not appear in list_memories or activate_project output
-# and cannot be accessed via read_memory or write_memory.
-# To access ignored memory files, use the read_file tool on the raw file path.
-# Extends the list from the global configuration, merging the two lists.
-# Example: ["_archive/.*", "_episodes/.*"]
-ignored_memory_patterns: []
-
-# list of additional workspace folder paths for cross-package reference support.
-# Paths can be absolute or relative to the project root.
-# Each folder is registered as an LSP workspace folder, enabling language servers to discover
-# symbols and references across package boundaries, but these folders are not indexed by Serena,
-# i.e. the respective symbols will not be found using Serena's symbol search tools.
-# Example:
-#   additional_workspace_folders:
-#     - ../sibling-package
-#     - ../shared-lib
-ls_additional_workspace_folders: []
-
-# list of language servers to start when using the LSP backend; choose from:
-#   ada                    al                     angular                ansible                bash
-#   bsl                    clojure                cpp                    cpp_ccls               crystal
-#   csharp                 csharp_omnisharp       cue                    dart                   deno
-#   elixir                 elm                    erlang                 fortran                fsharp
-#   gdscript               gleam                  go                     groovy                 haskell
-#   haxe                   hlsl                   html                   java                   json
-#   julia                  kotlin                 latex                  lean4                  lua
-#   luau                   markdown               matlab                 msl                    nextflow
-#   nix                    ocaml                  pascal                 perl                   php
-#   php_phpactor           php_phpantom           powershell             python                 python_basedpyright
-#   python_jedi            python_pyrefly         python_ty              qml                    r
-#   rego                   ruby                   ruby_solargraph        rust                   scala
-#   scss                   solidity               svelte                 swift                  systemverilog
-#   terraform              toml                   typescript             typescript_vts         vue
-#   wolfram                yaml                   zig
-#   (This list may be outdated; generated with scripts/print_language_list.py;
-#   For the current list, see values of the LanguageServerId enum here:
-#   https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
-# For some languages, there are several alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
-# Note:
-#   - For C, use cpp
-#   - For JavaScript, use typescript
-#   - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
-#   - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
-#   - For Deno projects, use deno (serves the same .ts/.js files as typescript; requires the deno CLI on PATH)
-#   - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
-#   - For Free Pascal/Lazarus, use pascal
-# Special requirements:
-#   Some language servers require additional setup/installations.
-#   See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
-# When using multiple language servers, the first language server that supports a given file will be used for that file.
-# The first language server is the default language and the respective language server will be used as a fallback.
-# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
-language_servers:
-- rust
-
-# list of workspace folder paths (LSP backend only).
-# These folders will be used to build up Serena's symbol index.
-# Paths must be within the project root and should thus be relative to the project root.
-# Furthermore, the paths should not be filtered by ignore settings.
-# Default setting: The entire project root folder (".") is considered.
-# In (large) monorepos, this can be used to index only subfolders of the project root, e.g.
-#   ls_workspace_folders:
-#     - "./subproject1"
-#     - "./subproject2"
-ls_workspace_folders:
-- .
-
-# optional shell command to run before the language backend (LSP or JetBrains) is initialised.
-# the command runs in the project root directory and is only executed if the project is trusted
-# (see trusted_project_path_patterns in the global configuration).
-# serena waits for the command to exit: a non-zero exit code is logged as an error but does not
-# abort activation. a per-project timeout (activation_command_timeout, default 180s) is the safety
-# backstop for non-terminating commands; on expiry the process is killed and activation continues.
-# example: activation_command: "npx nx run-many -t build"
-activation_command:
-
-# maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
-# must be a positive number.
-activation_command_timeout: 180.0

.zed/tasks.json

@@ -1,59 +0,0 @@
-// Project tasks configuration. See https://zed.dev/docs/tasks for documentation.
-//
-// Example:
-[
-  {
-    "label": "Example task",
-    "command": "for i in {1..5}; do echo \"Hello $i/5\"; sleep 1; done",
-    //"args": [],
-    // Env overrides for the command, will be appended to the terminal's environment from the settings.
-    "env": { "foo": "bar" },
-    // Current working directory to spawn the command into, defaults to current project root.
-    //"cwd": "/path/to/working/directory",
-    // Whether to use a new terminal tab or reuse the existing one to spawn the process, defaults to `false`.
-    "use_new_terminal": false,
-    // Whether to allow multiple instances of the same task to be run, or rather wait for the existing ones to finish, defaults to `false`.
-    "allow_concurrent_runs": false,
-    // What to do with the terminal pane and tab, after the command was started:
-    // * `always` — always show the task's pane, and focus the corresponding tab in it (default)
-    // * `no_focus` — always show the task's pane, add the task's tab in it, but don't focus it
-    // * `never` — do not alter focus, but still add/reuse the task's tab in its pane
-    "reveal": "always",
-    // Where to place the task's terminal item after starting the task:
-    // * `dock` — in the terminal dock, "regular" terminal items' place (default)
-    // * `center` — in the central pane group, "main" editor area
-    "reveal_target": "dock",
-    // What to do with the terminal pane and tab, after the command had finished:
-    // * `never` — Do nothing when the command finishes (default)
-    // * `always` — always hide the terminal tab, hide the pane also if it was the last tab in it
-    // * `on_success` — hide the terminal tab on task success only, otherwise behaves similar to `always`
-    "hide": "never",
-    // Which shell to use when running a task inside the terminal.
-    // May take 3 values:
-    // 1. (default) Use the system's default terminal configuration in /etc/passwd
-    //      "shell": "system"
-    // 2. A program:
-    //      "shell": {
-    //        "program": "sh"
-    //      }
-    // 3. A program with arguments:
-    //     "shell": {
-    //         "with_arguments": {
-    //           "program": "/bin/bash",
-    //           "args": ["--login"]
-    //         }
-    //     }
-    "shell": "system",
-    // Whether to show the task line in the output of the spawned task, defaults to `true`.
-    "show_summary": true,
-    // Whether to show the command line in the output of the spawned task, defaults to `true`.
-    "show_command": true,
-    // Which edited buffers to save before running the task:
-    // * `all` — save all edited buffers
-    // * `current` — save currently active buffer only
-    // * `none` — don't save any buffers
-    "save": "none",
-    // Represents the tags for inline runnable indicators, or spawning multiple tasks at once.
-    // "tags": []
-  },
-]

DevDoc/architecture/numa_partition_runner/index.html

@@ -1367,6 +1367,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1553,6 +1564,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1836,11 +1858,27 @@ dormant workers have been woken and grows every node by the same amount per
 step, capped by that node's remaining dormant workers and by the run's total
 budget (<code>n_total</code>) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.</p>
+<h2 id="panic-propagation-fixed-2026-08-28">Panic propagation (fixed, 2026-08-28)</h2>
+<p>A worker whose <code>f(i)</code> panicked used to never send its <code>WorkerEvent::Completed</code>
+— the controller's <code>while completed &lt; n_total</code> loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept <code>event_rx</code> open. Discovered via <code>obikselect</code>
+panicking on an unimplemented <code>Sparse</code> matrix case (see <code>implementation/select.md</code>):
+the process hung indefinitely instead of erroring out.</p>
+<p>Fixed: each worker wraps <code>f(i)</code> in <code>std::panic::catch_unwind</code> and sends a new
+<code>WorkerEvent::Panicked(i, payload)</code> instead of silently dropping the
+partition. The controller counts it toward <code>completed</code> (unblocking the loop)
+and keeps the first payload seen; once <code>run</code> returns, <code>panic::resume_unwind</code>
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain <code>Err</code> (a panic means a bug, not a normal typed failure).</p>
 <h2 id="open-questions">Open questions</h2>
 <ul>
 <li>
 <p><strong>Error handling</strong>: <code>run</code> currently returns the first error; remaining errors
-  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics.</p>
+  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain <code>Err</code> results.)</p>
 </li>
 <li>
 <p><strong><code>INITIAL_DIVISOR</code> / <code>GROWTH_DIVISOR</code> tuning</strong>: currently <code>4</code> and <code>8</code>

DevDoc/architecture/siblings/index.html

@@ -1367,6 +1367,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1652,6 +1663,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -2582,6 +2604,98 @@ mismatches. The dense/sparse performance gap is gone — previously sparse
 systematic gap. <code>pack --sparse</code>'s claimed query win isn't confirmed
 outright by this (sparse should arguably now <em>beat</em> dense on truly sparse
 real data, not just tie), but the pathological regression is fixed.</p>
+<h2 id="persistentcompactintmatrixsparse-implemented-2026-08-26"><code>PersistentCompactIntMatrix::Sparse</code> — implemented (2026-08-26)</h2>
+<p>Closes the gap flagged throughout this document ("no sparse count format
+exists yet", <code>traits.rs:9-12</code>'s "Explicitly deferred"): <code>obicompactvec</code>
+already had <code>PersistentSparseCompactIntMatrix</code> (row-major, built on top of
+<code>PersistentSparseBitMatrix</code> as its "which columns are non-zero" support,
+values <em>not</em> deduplicated — see that struct's own doc comment), but it was
+never wired into <code>PersistentCompactIntMatrix</code>, the dense-dispatching enum
+every real consumer (<code>TypedLayer&lt;PersistentCompactIntMatrix&gt;</code>,
+<code>KmerLayer::Count</code>) actually holds. Concretely: <code>kmer_index.rs::
+pack_matrices(sparse=true)</code> already called <code>pack_sparse_compact_int_matrix</code>
+on every layer's <code>counts/</code> — but <code>PersistentCompactIntMatrix::open</code> had no
+code path back to what that just wrote, so a <code>Count</code> layer became
+unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
+anyone ran <code>pack --sparse</code> on an index with count layers. Root cause, not a
+workaround: add the missing <code>Sparse</code> variant.</p>
+<ul>
+<li>
+<p><strong>Enum + dispatch</strong> (<code>intmatrix.rs</code>): <code>PersistentCompactIntMatrix::Sparse
+  (PersistentSparseCompactIntMatrix)</code>, detected in <code>open</code>/<code>detect_storage</code>
+  via a <code>singleton_values.pciv</code> marker (mirrors <code>PersistentBitMatrix</code>'s own
+  <code>sparse_meta.json</code> check), reported via <code>storage_kind()</code>. <code>col</code>/
+  <code>col_view</code>/<code>col_persist</code> panic/<code>Unsupported</code> on <code>Sparse</code>, same convention
+  as the bit side. <code>sub_matrix</code>/<code>fill_sub_matrix</code> and <code>nonzero_iter</code>
+  unified the same way <code>PersistentBitMatrix</code>'s already are (drain
+  <code>nonzero_iter</code>, one traversal per format — see "Implemented
+  (2026-08-20)" above); <code>nonzero_iter</code> had to become <code>Box&lt;dyn Iterator&lt;...&gt;&gt;</code>
+  for the same reason (<code>Columnar</code>/<code>Packed</code>/<code>Sparse</code> are different concrete
+  types). No change needed in <code>obikindex</code> at all — <code>KmerLayer::Count</code>
+  already only ever holds <code>TypedLayer&lt;PersistentCompactIntMatrix&gt;</code>, so the
+  enum absorbing <code>Sparse</code> fixes the unreadable-layer bug for free, same as
+  <code>PersistentBitMatrix::Sparse</code> already did on the presence side.</p>
+</li>
+<li>
+<p><strong><code>CountPartials</code>, non-naive</strong> (<code>sparse_intmatrix.rs</code>): unlike
+  <code>PersistentSparseBitMatrix</code>'s dict-driven <code>col_weights_and_pair_counts</code>,
+  values here aren't deduplicated (two rows can share the same non-zero
+  column set via the same <code>dict_id</code> while carrying different counts), so
+  the "weight by how many rows share a dict entry" shortcut doesn't carry
+  over. What does: a single row-major pass (<code>row_major_pairwise</code>, decodes
+  each row once via <code>for_each_cell_in_row</code>, nests over that row's own
+  co-present columns) — <code>O(Σ k̄²)</code> over populated rows instead of the naive
+  <code>O(n_cols² × n)</code> column-pair rescan, same complexity class as the bit
+  side minus the dict multiplicity discount. Kernels used: <code>min(a,b)</code>
+  (bray, relfreq-bray — both vanish when either side is absent, so no
+  correction needed), <code>a·b</code> and <code>√(a·b)</code> (euclidean/relfreq-euclidean and
+  hellinger — these <em>do</em> need a correction, reconstructed from per-column
+  marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>, since <code>(a-0)² = a² ≠ 0</code> unlike
+  the <code>min</code>-based formulas). <code>threshold_jaccard(1)</code> shortcuts straight to
+  <code>support</code>'s own <code>BitPartials::partial_jaccard</code> (threshold 1 is exactly
+  presence); <code>threshold_jaccard(0)</code> is closed-form (every <code>u32</code> is <code>≥ 0</code>).</p>
+</li>
+<li>
+<p><strong>Two pre-existing bugs found and fixed while wiring the <code>threshold==1</code>
+  shortcut</strong> (<code>bitmatrix/sparse.rs</code>, <code>BitPartials for
+  PersistentSparseBitMatrix</code>, present since the 2026-08-15 implementation
+  above, never caught because no test compared <code>Sparse</code>'s raw <code>partial_*</code>
+  output against dense on real data — only the diagonal-blind
+  <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested):</p>
+</li>
+<li><code>partial_jaccard</code>'s diagonal was <code>(0, 2×col_weights[i])</code> instead of a
+     genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
+     <code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with
+     itself by construction.</li>
+<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: <code>total -
+     union</code> (count of rows where <em>neither</em> column is present) instead of
+     the actual Hamming distance <code>col_weights[i] + col_weights[j] -
+     2×inter[i,j]</code> (symmetric-difference size). Only coincides with the
+     correct value when <code>col_weights[i] + col_weights[j] == total</code>, so
+     small/synthetic test data could easily have hidden it.</li>
+</ul>
+<p>Neither surfaced through <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code>
+  (both explicitly zero their own diagonal at finalisation, and the
+  off-diagonal <code>partial_hamming</code> bug had gone untested against dense
+  entirely) — only visible to a caller of the raw <code>partial_*</code> methods
+  directly, which is exactly what <code>partial_threshold_jaccard(1)</code>'s new
+  shortcut became. Fixed at the source, not patched around at the call
+  site; regression test added:
+  <code>tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal</code>.</p>
+<ul>
+<li><strong>Tests</strong>: <code>tests::intmatrix::sparse_roundtrip_matches_columnar</code>/
+  <code>sparse_roundtrip_from_packed</code> (the <code>open</code>-dispatch fix, both build
+  paths); <code>tests::intmatrix::sparse_count_partials_match_dense</code> (all six
+  <code>CountPartials</code> formulas, thresholds 0/1/2/3, against <code>Columnar</code> on
+  asymmetric-presence data — this is what caught the diagonal gap in the
+  int side's own new code before it shipped, the same way it exposed the
+  two pre-existing bit-side bugs above); <code>obikindex</code>'s
+  <code>count_layer_transparently_reads_sparse_after_pack</code> — the actual
+  end-to-end regression test for the original "layer unreadable after
+  <code>pack --sparse</code>" bug, built → packed sparse → reopened, compared against
+  the pre-pack dense read. <code>cargo test -p obicompactvec -p obikindex</code>:
+  green, no regressions (180 + 12 tests).</li>
+</ul>
 
 
 

DevDoc/implementation/merge/index.html

@@ -1034,6 +1034,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1598,6 +1609,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1795,6 +1817,14 @@ spectrums/
   &lt;label&gt;.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 </code></pre></div>
+<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
+<h2 id="known-issue-not-yet-fixed-2026-08-28">Known issue (not yet fixed, 2026-08-28)</h2>
+<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank &lt; self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
+<ul>
+<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
+<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
+</ul>
+<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
 
 
 

DevDoc/implementation/obicompactvec/index.html

@@ -1168,7 +1168,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
 </code></pre></div>
 <hr/>
 <h2 id="matrix-types">Matrix types</h2>
-<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has two variants (<code>Columnar</code>, <code>Packed</code>). <code>PersistentBitMatrix</code> has four:</p>
+<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
 <table>
 <thead>
 <tr>
@@ -1185,12 +1185,12 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
 </tr>
 <tr>
 <td><code>Packed</code></td>
-<td>single <code>matrix.pbmx</code> mmap file</td>
+<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
 <td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
 </tr>
 <tr>
-<td><code>Sparse</code> (bit only)</td>
+<td><code>Sparse</code></td>
-<td><code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major</td>
+<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
 <td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
 </tr>
 <tr>
@@ -1200,7 +1200,8 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
 </tr>
 </tbody>
 </table>
-<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
+<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
+<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
 <p><code>col_view(c)</code> returns the appropriate view directly:</p>
 <div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
 <span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o">&lt;'</span><span class="nb">_</span><span class="o">&gt;</span>

DevDoc/implementation/partition_layer_cache/index.html

@@ -1213,6 +1213,122 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#why-this-came-up" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Why this came up
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        The general pattern (not obikindexer-specific)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Concretely, next to implement (two items, in order)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1719,6 +1835,122 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#why-this-came-up" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Why this came up
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        The general pattern (not obikindexer-specific)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Concretely, next to implement (two items, in order)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
+      
+    </span>
+  </a>
+  
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        (12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1853,6 +2085,21 @@ The underlying module boundary and its rationale (Layer tier / Partition
 tier / Index tier, each depending only downward) are unchanged; only the
 crate-vs-module packaging changed. See <a href="../layer_tier/">obikindex::layer</a>
 for the current module doc.</p>
+<p><strong>Superseded, second event, same day (2026-08-21):</strong> <code>obikpartitionner</code>
+and <code>obikderep</code>, the two algorithm crates, are also gone — but unlike
+<code>obikpartition</code>/<code>obilayeredmap</code> above, they were <strong>not</strong> folded into
+<code>obikindex</code>. They were first (mistakenly) merged into <code>obikindex</code> as an
+<code>algorithms</code> submodule, then corrected into a new sibling crate,
+<strong><code>obikindexer</code></strong>, holding <code>obikindexer::algorithms::{partitionner,
+dereplicator}</code> and depending on <code>obikindex</code> — never the reverse, same
+dependency direction <code>obikpartitionner</code>/<code>obikderep</code> already had. Read
+<code>obikpartitionner::X</code> as <code>obikindexer::algorithms::partitionner::X</code> and
+<code>obikderep::X</code> as <code>obikindexer::algorithms::dereplicator::X</code> throughout
+what follows. The distinction the mistake surfaced, worth keeping: data
+crates (<code>obikindex</code>, holding the <code>index</code>/<code>partition</code>/<code>layer</code> model) merge
+naturally into one crate as submodules; algorithm crates that operate on
+that model from outside stay separate, so the dependency only ever runs
+one way.</p>
 <p>Status (2026-08-20, latest pass): (1) done — <code>obilayeredmap::Layer</code>
 exists, <code>Mat</code> is gone. (1b) done — <code>Layer::Empty</code>, the first non-ready
 state, added (panics on every read method). (2a) done — the
@@ -1872,7 +2119,24 @@ turned out to be bigger than <code>KmerPartition</code> alone: <code>Layer</code
 constructors don't self-name either. Full redesign of both, agreed in
 detail, session ended (budget) before implementation — see "(5) design
 agreed" below; <strong>read it before touching <code>KmerPartition</code>/<code>Layer</code>
-signatures</strong>, the shape is fully specified. Earlier mix-up, for
+signatures</strong>, the shape is fully specified. (6) done — <code>Counter</code>, a third
+algorithm, extracted from <code>PartitionRouter</code> the same way <code>Dereplicator</code>
+was in (4). (7) done — <code>LayerBuilder</code>, the fourth and last pipeline
+algorithm; the indexing pipeline is now fully decomposed into
+<code>obikindexer::algorithms::{partitionner, dereplicator, counter,
+layer_builder}</code>. (8) design agreed, item 1 done in (9) —
+<code>obikindexer::extensions::PrivateBuilder</code>, private, six construction-only
+<code>KmerIndex</code> methods moved out. (10) done, same session — <code>obikindex::
+IndexBuilder</code>, public, the four maintenance methods
+(<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>)
+shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>. Item 2 from (8)
+(<code>obikalgorithm::Algorithm</code>) done in (12) — new crate, <code>type Output</code> +
+<code>fn run(&amp;mut self) -&gt; SKResult&lt;Self::Output&gt;</code>, <code>on_progress</code> moved off
+<code>run()</code>'s signature entirely into a per-algorithm <code>.on_progress(...)</code>
+setter. Note: <code>Layer</code> renamed
+<code>KmerLayer</code> (2026-08-21, outside this conversation). (5) itself still not
+implemented, still first on the "order of remaining work" list. Earlier
+mix-up, for
 context: an earlier
 version of this doc used the name <code>KmerPartition</code> (singular) for what was
 actually the <em>collection</em> type (later renamed <code>KmerPartitions</code>, later
@@ -2515,6 +2779,622 @@ itself or a new crate.</p>
    examples were judged not enough to be sure of the shape (<code>Fn+Sync</code> vs
    <code>FnMut</code> callback bound already diverged between the two that exist).</li>
 </ol>
+<h2 id="6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator">(6) done (2026-08-21): <code>Counter</code> — third algorithm, extracted the same way as <code>Dereplicator</code></h2>
+<p>Between (5) and this, the user did a session of their own crate
+restructuring (see the two "Superseded" notes at the top of this file):
+<code>obikpartition</code>/<code>obilayeredmap</code> folded into <code>obikindex</code> as submodules
+(<code>obikindex::partition</code>, <code>obikindex::layer</code>), and <code>obikpartitionner</code>/
+<code>obikderep</code> merged into one sibling crate, <code>obikindexer</code>, holding
+<code>obikindexer::algorithms::{partitionner, dereplicator}</code>. (5)'s design
+(<code>Layer</code>/<code>KmerPartition</code> self-naming by number, <code>PartitionRouter</code>'s
+<code>&amp;mut</code> → <code>&amp;</code> fix) was <strong>not</strong> part of that — pure crate/module packaging,
+confirmed by reading the actual code (<code>Layer::open</code>/<code>create</code> still take an
+external <code>dir: &amp;Path</code>, <code>KmerPartition</code> still eagerly opens all layers,
+<code>PartitionRouter</code> still holds <code>&amp;mut KmerIndex</code>). (5) remains exactly as
+specified, not yet implemented.</p>
+<p>This step: <code>count_kmer</code> (still living on <code>PartitionRouter</code>, per (4)'s own
+"still not done" note) extracted into <code>obikindexer::algorithms::counter::
+Counter</code>, mirroring <code>Dereplicator</code> exactly — third data point for the
+eventual <code>obikalgorithm</code> trait, still not extracted (still only 3 examples
+with 2 different callback bounds; holding off per (5)'s "order of
+remaining work").</p>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">Counter</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
+<span class="w">    </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
+<span class="w">    </span><span class="n">keep_partial</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
+<span class="p">}</span>
+
+<span class="k">impl</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="w"> </span><span class="n">Counter</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_partial</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span><span class="w">   </span><span class="c1">// setter, mirrors PartitionRouter&#39;s style; defaults to false</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o">&lt;</span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">&gt;</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">SKResult</span><span class="o">&lt;</span><span class="n">KmerSpectrum</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="p">}</span>
+</code></pre></div>
+<p>Same shape as <code>Dereplicator</code> throughout: <code>Fn(Progress) + Sync</code> (not
+<code>FnMut</code>) since counting is also a parallel <code>par_iter</code> over partitions, an
+<code>AtomicU64</code> position counter incremented from inside the parallel closure
+so progress reports arrive in real time rather than bursting at the end
+once <code>.collect()</code> finishes, <code>total: Some(n_partitions)</code> (known up front).
+<code>KmerSpectrum</code> (the <code>{f0, f1, counts}</code> aggregate) moved from
+<code>partitionner::router</code> to <code>counter</code>, since it's <code>Counter::run</code>'s return
+value now, not <code>PartitionRouter</code>'s. <code>count.rs</code>/<code>kmer_sort.rs</code> moved
+verbatim from <code>partitionner/</code> to <code>counter/</code> (unchanged bodies — only
+<code>count_kmer</code> itself, <code>KmerSpectrum</code>, and the imports they pulled in were
+removed from <code>router.rs</code>).</p>
+<p>One divergence from <code>Dereplicator</code>: a <code>keep_partial</code> setter exists (no
+equivalent on <code>Dereplicator</code>, which has no setters at all) — a real,
+already-present parameter (<code>keep_intermediate</code> at the CLI), not a
+speculative addition.</p>
+<p><code>count_kmer</code>'s three former callers (<code>obikmer::cmd::index</code>, <code>obikphylo</code>'s
+test harness, <code>obikindexer::algorithms::partitionner</code>'s own
+<code>pipeline_counts</code> test helper) all updated to <code>Counter::new(&amp;idx).
+run(...)</code> — the last one simplified further: it used to read back
+<code>kmer_spectrum_raw.json</code> from disk after calling <code>count_partition</code>
+directly (white-box), now it just uses the <code>KmerSpectrum</code> <code>Counter::run</code>
+already returns.</p>
+<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
+<code>cargo test --workspace</code>, exit code 0), plus an end-to-end CLI smoke test
+against real FASTA data (scatter → dereplicate → count → index-build →
+query) — required every time per (3)'s lesson, and it earned its keep
+again: the very first smoke-test query returned zero matches, which
+looked like a regression until traced to the query sequence itself being
+low-complexity ("GGCCCCCCACG", six same-base runs) and rejected by
+<em>query's own</em> default entropy threshold — nothing to do with this change.
+Re-tested with a different substring, confirmed working (kmer found,
+count matched the index).</p>
+<p>Still not done: (5) (<code>Layer</code>/<code>KmerPartition</code> redesign, <code>PartitionRouter</code>'s
+<code>&amp;mut</code>→<code>&amp;</code>), the future cache crate, <code>build_layers</code> (still a <code>KmerIndex</code>
+inherent method, not an algorithm), and <code>obikalgorithm</code> itself.</p>
+<h2 id="7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm">(7) done (2026-08-21): <code>LayerBuilder</code> — fourth and last pipeline algorithm</h2>
+<p>Closes out the indexing pipeline: <code>build_layers</code>/<code>build_index_layer</code>
+(the last stage still living as <code>KmerIndex</code> inherent methods, flagged as
+inconsistent since (6)) extracted into <code>obikindexer::algorithms::
+layer_builder::LayerBuilder</code>, same two-phase shape as the other three.</p>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayerBuilder</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
+<span class="w">    </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
+<span class="w">    </span><span class="n">min_abundance</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span>
+<span class="w">    </span><span class="n">max_abundance</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="w">    </span><span class="n">keep_intermediate</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
+<span class="p">}</span>
+
+<span class="k">impl</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="w"> </span><span class="n">LayerBuilder</span><span class="o">&lt;&#39;</span><span class="na">a</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="o">&#39;</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">min_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">max_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_intermediate</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
+<span class="w">    </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o">&lt;</span><span class="k">impl</span><span class="w"> </span><span class="nb">FnMut</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="o">&gt;</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">SKResult</span><span class="o">&lt;</span><span class="kt">usize</span><span class="o">&gt;</span><span class="p">;</span><span class="w">  </span><span class="c1">// returns total kmers built</span>
+<span class="p">}</span>
+</code></pre></div>
+<p><strong>Different from all three prior extractions in one respect, deliberately
+not "fixed" to match them</strong>: the actual per-partition construction logic
+(De Bruijn graph from dereplicated superkmers + provisional counts →
+unitigs → MPHF → matrix) stayed put as <code>KmerIndex::build_index_layer</code>/
+<code>remove_build_artifacts</code> (both already <code>pub</code>) — not moved into
+<code>obikindexer</code>. Checked first: unlike <code>dereplicate_partition</code>/
+<code>count_partition</code> (which only ever had one caller), <code>build_index_layer</code>
+depends on several <code>obikindex</code>-internal helpers (<code>graph_pipeline::
+{write_graph_as_unitigs, materialize_layer}</code>, <code>common::olm_to_sk</code>) that
+are <code>pub(crate)</code> and shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>'s own
+layer-construction paths — moving <code>build_index_layer</code> out would have
+meant either exporting that internal surface just for this one algorithm
+or duplicating it. Neither was needed: <code>build_index_layer</code>/
+<code>remove_build_artifacts</code> were <em>already</em> public <code>KmerIndex</code> methods, so
+<code>LayerBuilder</code>'s job is purely the orchestration around them (scheduling,
+config, progress) — the exact same "algorithm calls already-public
+<code>KmerIndex</code> primitives" shape <code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>
+already have, just at a coarser grain for this one stage. This is the
+"is the producer's API actually deficient?" check from
+[[feedback_no_spaghetti_petits_pois]] applied and answered "no" — not
+skipped.</p>
+<p><strong>Two more real divergences, both forced by <code>PartitionRunner</code>, not
+arbitrary:</strong>
+- Uses <code>obikindex::PartitionRunner</code> (NUMA-aware scheduler, already
+  <code>pub use</code>d from <code>obikindex</code>) instead of plain <code>rayon::into_par_iter</code>
+  like <code>Dereplicator</code>/<code>Counter</code> — matches what <code>build_layers</code> already used
+  before extraction; this stage is more CPU/memory-intensive per partition
+  (graph construction) than scatter/dereplicate/count.
+- Callback bound is <code>FnMut(Progress) + Send</code> — a third variant, not
+  matching either prior shape. <code>PartitionRunner::run</code>'s <code>on_done</code> is
+  invoked from its own single controller thread (never concurrently, so
+  no <code>Sync</code> needed, unlike <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn + Sync</code>), but
+  that controller thread is itself <code>std::thread::scope</code>-spawned, so the
+  closure still has to be <code>Send</code> to cross into it — caught immediately by
+  the compiler (<code>cannot be sent between threads safely</code>) when <code>Send</code> was
+  first omitted, not a design guess. <code>obikalgorithm</code>'s eventual shared
+  trait now has three real callback-bound data points to reconcile
+  (<code>FnMut</code> alone for <code>PartitionRouter::run</code>'s sequential loop, <code>FnMut +
+  Send</code> here, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code>'s <code>rayon</code>
+  <code>par_iter</code>), not two.</p>
+<p><code>KmerIndex::build_layers</code> deleted outright (<code>KmerIndex</code> stays a pure data
+structure — no compute orchestration methods, consistent with <code>dereplicate</code>/
+<code>count_kmer</code>'s removal in (4)/(6)). New <code>KmerIndex::mark_indexed()</code> added,
+symmetric to <code>mark_scattered</code>/<code>mark_counted</code>, replacing the inline
+<code>touch(SENTINEL_INDEXED)</code> that used to live inside <code>build_layers</code>.
+<code>Stage::start("index")</code>/<code>rep.push(...)</code> and the <code>progress_bar</code>/
+<code>"{n} total kmers indexed"</code> log line both moved to <code>cmd/index/mod.rs</code>,
+same pattern as (3)/(4)/(6) — <code>LayerBuilder</code> renders nothing itself, just
+reports <code>Progress</code>.</p>
+<p>All callers updated: <code>cmd/index/mod.rs</code> (Stage 3), <code>obikphylo</code>'s test
+harness (also gained a <code>mark_indexed()</code> call it was missing — harmless
+before since nothing checked <code>IndexState::Indexed</code> in that test, but now
+correct).</p>
+<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
+<code>cargo test --workspace</code>, exit code 0), plus the end-to-end CLI smoke test
+(<code>scripts/smoke_test_index.sh</code>, built earlier specifically so this
+verification step is a one-liner from now on) — 870 kmers indexed, query
+round-trip confirmed, same numbers as (6).</p>
+<p><strong>The indexing pipeline is now fully decomposed</strong>: <code>obikindexer::
+algorithms::{partitionner, dereplicator, counter, layer_builder}</code>, each a
+<code>new</code>/(setters)/<code>run</code> algorithm operating on a <code>&amp;KmerIndex</code> (or <code>&amp;mut</code> for
+<code>PartitionRouter</code>, not yet fixed — see (5)), <code>KmerIndex</code> itself holding no
+pipeline-orchestration logic anymore. Still not done: (5), the future
+cache crate, <code>obikalgorithm</code> (now unblocked — three real callback-bound
+variants observed, worth revisiting whether a single trait can express
+all three or whether that's itself the answer: it can't, and the trait
+should not force it).</p>
+<h2 id="8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename">(8) design agreed, not yet implemented (2026-08-21): <code>obikalgorithm::Algorithm</code> trait + <code>obikindexer::extensions</code> — private/public extension-trait split, <code>KmerLayer</code> rename</h2>
+<p>Session note: <code>Layer</code> was renamed <code>KmerLayer</code> (user, outside this
+conversation, alongside other naming homogenisation with <code>KmerIndex</code>/
+<code>KmerPartition</code>) — every reference to <code>Layer</code> in this doc from before
+2026-08-21 means today's <code>obikindex::layer::KmerLayer</code>.</p>
+<h3 id="why-this-came-up">Why this came up</h3>
+<p>Verifying "does <code>cmd/index</code> now rest entirely on the algorithm structs"
+(it doesn't quite — see below) led to sorting <code>KmerIndex</code>'s own methods by
+a criterion the user was explicit is <strong>semantic, not mechanical</strong>: "les
+méthodes qui, sémantiquement, n'ont pas d'intérêt hors de la construction
+de l'index" (methods that have no semantic interest outside index
+construction) — not "methods only called from <code>cmd/index</code> today," which
+a grep could answer but would miss methods construction-adjacent code
+elsewhere (<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>) also depends on for the
+same reason.</p>
+<p><strong>Checked, not assumed</strong> (grepped every call site before classifying):</p>
+<ul>
+<li><strong>Construction-only, real candidates for a private extension trait</strong>:
+  <code>KmerIndex::{mark_scattered, mark_counted, mark_indexed, write_spectrum,
+  build_index_layer, remove_build_artifacts, clear_output_for_create,
+  create_skeleton, finalize_indexed, state}</code>. The last four are called
+  from <code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> too (as
+  precondition checks — "is my source <code>Indexed</code>?" — or shared
+  skeleton/finalize machinery), not just from the 4-stage pipeline — so
+  this extension trait's scope is "construction of any kind," not
+  narrowly "the initial build pipeline."</li>
+<li><strong>Looked construction-only by name, checked, and kept on <code>KmerIndex</code></strong>:
+  <code>layer_unitigs_path</code> (unitigs are the only way to recover a built
+  index's kmer sequences — read by <code>rebuild_layer.rs</code> and others, well
+  beyond construction — see [[project_unitigs_always_kept]]),
+  <code>pack_matrices</code> (re-runnable maintenance on an already-finished index
+  via <code>obikmer pack</code>, not just a pipeline step), <code>upgrade_layer_meta</code>
+  (migration, runnable on any existing index at any time).</li>
+</ul>
+<h3 id="the-general-pattern-not-obikindexer-specific">The general pattern (not obikindexer-specific)</h3>
+<p><code>KmerIndex</code>/<code>KmerPartition</code>/<code>KmerLayer</code> stay generic, in <code>obikindex</code> —
+every domain-specific consumer crate gets to attach its own extension
+trait(s), of two kinds:</p>
+<ul>
+<li><strong>Private</strong> (<code>pub(crate)</code>, invisible outside the defining crate) — for
+  plumbing only that crate's own algorithms need. <code>obikindexer</code> gets
+  exactly one of these (see below); no public counterpart makes sense for
+  it — "l'index est tellement central que le second trait n'a pas
+  vraiment d'intérêt" for construction specifically: nothing external
+  should ever want to call <code>mark_scattered</code> or <code>build_index_layer</code>.</li>
+<li><strong>Public</strong> — for a genuinely reusable domain extension. The user's own
+  example, found while discussing this, not hypothetical: <code>obikindex/src/
+  index/distance.rs</code> (phylogenetic distance metrics) is currently an
+  <code>impl KmerIndex</code> block <strong>inside <code>obikindex</code> itself</strong> — under this
+  principle it should be a public extension trait owned by <code>obikphylo</code>
+  instead (distance metrics are a phylo concept, <code>obikindex</code> has no more
+  business defining them than <code>obikindex::layer</code> has defining
+  "family"/"minorant", the reasoning <code>SiblingLayerExt</code> already followed
+  for <code>KmerLayer</code> — see <code>obikphylo/src/siblings/iter.rs</code>). <strong>Explicitly
+  deferred</strong> — noted here so it isn't lost, not part of this round.</li>
+<li>The future cache-manager crate (still blocked on (5), see above) will
+  add its own <strong>public</strong> extension trait mirroring part of <code>KmerIndex</code>'s/
+  <code>KmerPartition</code>'s own read API in cached form (e.g. a cached
+  <code>.partition(i)</code> that doesn't re-touch disk) — same pattern, third data
+  point once built.</li>
+</ul>
+<h3 id="concretely-next-to-implement-two-items-in-order">Concretely, next to implement (two items, in order)</h3>
+<ol>
+<li><strong><code>obikindexer::extensions</code></strong> — a private (<code>pub(crate)</code>) extension
+   trait, most likely named something like <code>IndexBuildExt</code> (final name
+   not yet chosen), implemented for <code>KmerIndex</code>, carrying the ten methods
+   listed above, moved out of <code>obikindex::index::{kmer_index,
+   index_layer}</code>. Every algorithm in <code>obikindexer::algorithms::*</code> that
+   currently calls <code>idx.mark_scattered()</code>/etc. keeps the same call syntax
+   (extension trait methods are called the same way as inherent ones,
+   just need the trait in scope) — <code>cmd/index/mod.rs</code> itself would need
+   <code>use obikindexer::extensions::IndexBuildExt;</code> (or the module re-exports
+   it) to keep compiling, since it's the one place outside <code>obikindexer</code>'s
+   own algorithms that currently calls <code>mark_scattered</code>/<code>write_spectrum</code>/
+   <code>mark_counted</code>/<code>mark_indexed</code> directly. <strong>Not yet decided</strong>: exact
+   trait name, whether it's one trait or split further (e.g. sentinel
+   marking vs. skeleton/finalize machinery), and whether <code>merge</code>/<code>select</code>/
+   <code>rebuild</code>/<code>reindex</code> (not yet extracted into algorithms themselves) move
+   onto it now too or keep calling the soon-to-be-inherent-no-longer
+   methods some other way in the meantime — <strong>ask before implementing</strong>,
+   this changes the blast radius significantly (4 more <code>obikindex</code>
+   internal files depend on <code>clear_output_for_create</code>/<code>create_skeleton</code>/
+   <code>finalize_indexed</code>/<code>state</code>).</li>
+<li><strong><code>obikalgorithm::Algorithm</code> trait</strong> — new crate, the shared trait
+   <code>obikpartitionner</code>→<code>obikindexer</code> merge (session start of 2026-08-21)
+   and (6)/(7) were deliberately building toward, now with four real
+   <code>new</code>/(setters)/<code>run</code> examples and three distinct callback-bound
+   shapes to reconcile (plain <code>FnMut</code> for <code>PartitionRouter</code>, <code>FnMut +
+   Send</code> for <code>LayerBuilder</code>, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code> —
+   see (7)). Exact shape not yet drafted in this doc — do that as its own
+   design pass before coding, same discipline as everything above.</li>
+</ol>
+<p>Both items: <strong>design only, nothing implemented yet</strong> — this section is
+the record to resume from, not a plan already executed.</p>
+<h2 id="9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented">(9) done (2026-08-21): <code>obikindexer::extensions::PrivateBuilder</code> — item 1 above, implemented</h2>
+<p>Renamed from <code>IndexBuilder</code> to <code>PrivateBuilder</code> immediately after (same
+session), freeing the name <code>IndexBuilder</code> for (10)'s public trait — read
+<code>IndexBuilder</code> below as <code>PrivateBuilder</code> throughout this section.</p>
+<p>Scoped down from (8)'s six-method list to the concrete set that's
+genuinely movable without further ripple — checked, not assumed, before
+writing anything:</p>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_scattered</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">OKIResult</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_counted</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">OKIResult</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_indexed</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">OKIResult</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">write_spectrum</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">f0</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">f1</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">counts</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">BTreeMap</span><span class="o">&lt;</span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="kt">u64</span><span class="o">&gt;</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">OKIResult</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">build_index_layer</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">min_ab</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="n">max_ab</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o">&lt;</span><span class="kt">u32</span><span class="o">&gt;</span><span class="p">,</span><span class="w"> </span><span class="n">with_counts</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="n">mode</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">IndexMode</span><span class="p">,</span><span class="w"> </span><span class="n">block_bits</span><span class="p">:</span><span class="w"> </span><span class="kt">u8</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Result</span><span class="o">&lt;</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">SKError</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">remove_build_artifacts</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">);</span>
+<span class="p">}</span>
+<span class="k">impl</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
+</code></pre></div>
+<p>All six moved bodily out of <code>obikindex::index::{kmer_index, index_layer}</code>
+into <code>obikindexer::extensions</code> (new module, <code>pub(crate)</code>) — <code>index_layer.rs</code>
+is now empty and deleted outright.
+<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>
+stayed inherent on <code>KmerIndex</code>, per (8)'s reasoning: <code>merge</code>/<code>select</code>/
+<code>rebuild</code>/<code>reindex</code> — living <em>inside</em> <code>obikindex</code> itself — call them too,
+and <code>obikindex</code> can never depend on <code>obikindexer</code> to reach a trait defined
+there. Moving those four is real future work (extract
+merge/select/rebuild/reindex into algorithms first), not part of this
+step.</p>
+<p><strong>One new, small, deliberate API widening in <code>obikindex</code></strong>: <code>build_index_layer</code>
+depends on three helpers that were <code>pub(crate)</code> to <code>obikindex</code>
+(<code>graph_pipeline::{write_graph_as_unitigs, materialize_layer}</code>,
+<code>common::olm_to_sk</code>) — widened to <code>pub</code> (re-exported from <code>obikindex</code>'s
+crate root) so <code>obikindexer</code> could reach them. This is exactly the
+"enrich shared/lower-level APIs instead of ad hoc local code" call the
+project's own rules ask for, made explicitly rather than routed around:
+three functions, already generically written (no rewrite needed), now
+serve a second caller instead of being duplicated.</p>
+<p><strong>Why the trait had to be defined in <code>obikindexer</code>, not <code>obikindex</code></strong>:
+Rust's orphan rule — implementing a trait for a foreign type requires
+either the trait or the type to be local to the current crate. <code>KmerIndex</code>
+is foreign to <code>obikindexer</code>, so the trait must be the local half; if it
+were defined in <code>obikindex</code> instead, <code>pub(crate)</code> there would make it
+invisible to <code>obikindexer</code> too (crate-private means private to <em>that</em>
+crate, not "private except to one named dependent") — the opposite of
+what was wanted.</p>
+<p><strong>A real design decision made while wiring callers up, not a mechanical
+rename</strong>: <code>PrivateBuilder</code> being genuinely <code>pub(crate)</code> to <code>obikindexer</code>
+means <code>obikmer::cmd::index</code> (a different crate) can no longer call
+<code>mark_scattered</code>/<code>mark_counted</code>/<code>mark_indexed</code>/<code>write_spectrum</code> directly —
+it never could have, once privacy was real rather than aspirational. Each
+algorithm now marks its own completion as part of <code>run()</code>/<code>close()</code>
+instead of leaving it to the caller:
+- <code>PartitionRouter::close()</code> (not <code>run()</code>) calls <code>mark_scattered()</code> —
+  <code>close()</code>, not <code>run()</code>, is the actual shared completion point between
+  the file-driven <code>run()</code> path and the manual <code>write</code>/<code>write_batch</code>+
+  <code>close()</code> path low-level callers (tests) use; putting it in <code>run()</code>
+  alone would have silently skipped marking for every caller that never
+  calls <code>run()</code>. <code>run()</code> already calls <code>self.close()</code> at its own end, so
+  this covers both paths through one line, not two.
+- <code>Counter::run</code> calls <code>write_spectrum</code> then <code>mark_counted</code> before
+  returning.
+- <code>LayerBuilder::run</code> calls <code>mark_indexed</code> before returning.</p>
+<p><code>cmd/index/mod.rs</code> lost all four direct calls (<code>mark_scattered</code>/
+<code>write_spectrum</code>/<code>mark_counted</code>/<code>mark_indexed</code>) — each stage's <code>if
+idx.state() &lt; IndexState::X { ... }</code> block is now purely "run the
+algorithm," no separate bookkeeping call after it. Confirms, precisely
+this time (checked by re-reading the whole file, not assumed): <code>cmd/index</code>
+now rests on the four algorithms for every read/write of pipeline state
+except <code>KmerIndex::{exists, create, state, n_partitions}</code>, which are
+genuinely index-identity concerns, not construction bookkeeping — the
+original question this whole design pass started from.</p>
+<p>Same fix applied to <code>obikphylo</code>'s test harness (its four explicit
+<code>mark_*</code>/<code>write_spectrum</code> calls removed, relying on the algorithms now
+doing it themselves) — <code>obikindexer::algorithms::partitionner</code>'s own
+<code>pipeline_counts</code> test needed no change (never called <code>mark_*</code> directly).</p>
+<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
+<code>cargo test --workspace</code>, exit code 0), plus the CLI smoke test — 870
+kmers, same as (6)/(7).</p>
+<p>Still not done at the time of writing: item 2 from (8) (<code>obikalgorithm::
+Algorithm</code>), (5), the future cache crate, the <code>distance.rs</code> →
+<code>obikphylo</code> relocation (noted in (8), explicitly deferred), and
+extracting <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code> into algorithms.</p>
+<h2 id="10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session">(10) done (2026-08-21): <code>obikindex::IndexBuilder</code> — the public counterpart, same session</h2>
+<p>Immediate correction to (9): the private trait built there was renamed
+<code>PrivateBuilder</code> (freeing the name), and the four methods (9) had left
+inherent on <code>KmerIndex</code> — <code>clear_output_for_create</code>/<code>create_skeleton</code>/
+<code>finalize_indexed</code>/<code>state</code> — got their own trait after all: <strong><code>IndexBuilder</code></strong>,
+public, defined in <code>obikindex</code> itself (not <code>obikindexer</code>):</p>
+<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="p">:</span><span class="w"> </span><span class="nb">Sized</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">clear_output_for_create</span><span class="o">&lt;</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o">&lt;</span><span class="n">Path</span><span class="o">&gt;&gt;</span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">force</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">OKIResult</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">create_skeleton</span><span class="o">&lt;</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o">&lt;</span><span class="n">Path</span><span class="o">&gt;&gt;</span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">meta</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">IndexMeta</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">OKIResult</span><span class="o">&lt;</span><span class="bp">Self</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">finalize_indexed</span><span class="o">&lt;</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o">&lt;</span><span class="n">Path</span><span class="o">&gt;&gt;</span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">rep</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">Reporter</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">OKIResult</span><span class="o">&lt;</span><span class="bp">Self</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">state</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">IndexState</span><span class="p">;</span>
+<span class="p">}</span>
+<span class="k">impl</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
+</code></pre></div>
+<p>User's framing: these four are "maintenance", not "scientific computation
+on an index" — a different kind of non-generic-ness than (9)'s six
+(<code>mark_*</code>/<code>write_spectrum</code>/<code>build_index_layer</code>/<code>remove_build_artifacts</code>,
+exclusive to the 4-stage pipeline). Maintenance is used more broadly
+(<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>), so it gets a real, public trait —
+not folded back into <code>KmerIndex</code>'s inherent surface, and not private
+either.</p>
+<p><strong>Where it lives, and why that's not arbitrary</strong>: (9) needed the orphan
+rule to force its trait into <code>obikindexer</code>, to achieve genuine
+crate-private visibility. Here the requirement is the opposite:
+<code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> — the trait's own
+heaviest users — live <em>inside</em> <code>obikindex</code>. A trait they need to reach
+must be local to <code>obikindex</code> (or a crate <code>obikindex</code> itself depends on,
+which doesn't exist for this). So <code>IndexBuilder</code> lives in a new
+<code>obikindex/src/index/builder.rs</code>, <code>pub trait</code> (no orphan-rule tension at
+all here — both trait and type are local to the same crate), re-exported
+from <code>obikindex</code>'s crate root alongside <code>PrivateBuilder</code>'s sibling
+<code>obikindexer::extensions::PrivateBuilder</code> staying where it is. Two
+traits, two crates, two different reasons, not a contradiction.</p>
+<p><strong>Blast radius, all inside <code>obikindex</code> plus one external crate</strong>: every
+internal caller of these four methods needs the trait imported now that
+they're no longer inherent — <code>merge.rs</code>, <code>select.rs</code>, <code>rebuild.rs</code>,
+<code>reindex.rs</code> (<code>use crate::index::builder::IndexBuilder;</code>) and, externally,
+<code>obikmer::cmd::index::mod</code> (<code>use obikindex::IndexBuilder;</code>, for the three
+<code>idx.state() &lt; IndexState::X</code> resumability checks). Call syntax at every
+site is unchanged (<code>KmerIndex::create_skeleton(...)</code>,
+<code>self.state()</code>) — only trait-in-scope requirements are new, which is
+exactly the point: same ergonomics, less surface baked into <code>KmerIndex</code>
+itself.</p>
+<p>Verification went one step further than (9): beyond
+<code>cargo check --workspace --all-targets</code> + <code>cargo test --workspace</code> +
+<code>scripts/smoke_test_index.sh</code> (all green, 870 kmers again), ran
+<code>obikmer merge</code> end to end on two freshly built indexes (exercises
+<code>clear_output_for_create</code>/<code>finalize_indexed</code> directly, the two methods
+<code>scripts/smoke_test_index.sh</code> itself never touches) — exit 0, <code>pack</code>
+stage completed. Test suite alone would not have caught a regression
+here: no existing test builds two real indexes and merges them through
+the CLI.</p>
+<p><code>KmerIndex</code> itself now carries only: identity/config accessors
+(<code>root_path</code>/<code>meta</code>/<code>kmer_size</code>/<code>n_bits</code>/<code>evidence_mode</code>/<code>genomes</code>/...),
+path resolution (<code>partition_dir</code>/<code>index_dir</code>/<code>layer_dir</code>/
+<code>partition_meta</code>/<code>n_layers</code>), and a few index-maintenance operations not
+yet sorted into either trait (<code>layer_unitigs_path</code>, <code>pack_matrices</code>,
+<code>upgrade_layer_meta</code> — see (8)'s "tested and discarded" list; still
+correctly inherent, not construction-only by the semantic criterion) —
+<code>create</code>/<code>open</code>/<code>exists</code> (identity, can't be trait methods needing <code>Self</code>
+before one exists) round that out.</p>
+<h2 id="11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files">(11) done (2026-08-21): <code>KmerIndex</code>/<code>IndexMeta</code> made fully stateless, <code>IndexState</code> moved off sentinel files</h2>
+<p>Triggered mid-discussion of <code>obikalgorithm::Algorithm</code> (still not started —
+see "Still not done" below): user asked why <code>PartitionRouter::new</code> still
+took <code>&amp;mut KmerIndex</code> at all, and questioned whether <code>mark_scattered</code>
+belonged in the algorithm or in <code>cmd/index</code>. Investigation found <code>&amp;mut</code>
+had become <em>newly</em> necessary since (9) — <code>mark_scattered</code> was mutating
+<code>self.meta.genomes</code> in memory so <code>Counter</code>'s later <code>write_spectrum</code> call
+(same <code>idx</code> instance) would see the derived label. User's resolution: the
+"disk is truth, stateless" principle already agreed for <code>KmerPartition</code>/
+<code>Layer</code> in (5) (still unimplemented for those two) should extend to
+<code>KmerIndex</code> itself — move <code>IndexState</code> (<code>Empty</code>/<code>Scattered</code>/<code>Counted</code>/
+<code>Indexed</code>) off the three sentinel files (<code>scatter.done</code>/<code>count.done</code>/
+<code>index.done</code>, detected by existence) and into a field of <code>index.meta</code>'s
+own JSON, so the <code>mark_*</code> calls become plain disk writes an algorithm can
+legitimately make on <code>&amp;self</code> — no in-memory mutation left to protect.</p>
+<p><strong>Shape of <code>IndexMeta</code>, per the user's explicit spec</strong>: one JSON file per
+index (<code>index.meta</code>), one <code>IndexMeta</code> instance per index, held and
+returned as <code>Arc&lt;IndexMeta&gt;</code> (not <code>&amp;IndexMeta</code>) by <code>KmerIndex::meta()</code>.
+<code>config</code> (<code>kmer_size</code>/<code>minimizer_size</code>/<code>n_bits</code>/<code>with_counts</code>/<code>evidence</code>/
+<code>block_bits</code>) is fixed at construction, cached as a <code>pub</code> field (getter
+kept alongside, for symmetry) — "les champs constants restent des champs
+de la structure", read once, never re-read from disk. <code>genomes</code> and
+<code>state</code> are the opposite: no in-memory cache at all, every accessor
+(<code>genomes()</code>, <code>state()</code>) re-reads <code>index.meta</code> from disk, every mutator
+(<code>push_genome</code>/<code>rename_genome</code>/<code>set_genomes</code>/<code>set_state</code>/<code>mark_scattered</code>/
+<code>mark_counted</code>/<code>mark_indexed</code>) does a full read-modify-write of the same
+file. An internal <code>std::sync::RwLock&lt;()&gt;</code> is held across each
+read-modify-write sequence (not just the write) so two callers sharing
+the same <code>Arc&lt;IndexMeta&gt;</code> can't lose an update to each other — this is
+<em>not</em> a cross-process lock (that's <code>obisys::DirLock</code>, already held by
+<code>cmd/index</code> for the whole build); it only serialises access through one
+shared in-process instance.</p>
+<p><strong>Construction, chain-of-responsibility style, matching (5)'s pattern</strong>:
+<code>IndexMeta::create(&amp;KmerIndex, config, genomes)</code> / <code>IndexMeta::open(&amp;KmerIndex)</code>
+ask the index for its own root path rather than taking one directly. Since
+<code>KmerIndex::create</code> doesn't have a complete <code>KmerIndex</code> yet to hand in
+(it's what's being built), added lower-level <code>pub(crate)</code> path-based
+primitives <code>create_at(&amp;Path, ...)</code> / <code>open_at(&amp;Path)</code> that <code>KmerIndex::create</code>/
+<code>open</code> and <code>builder.rs</code>'s <code>create_skeleton</code> call directly, bypassing the
+convenience wrappers for that one bootstrap case.</p>
+<p><strong>The one deliberate exception</strong>: <code>select_in_place</code> and <code>reindex</code>
+genuinely rewrite <code>config</code> after an index already exists (output
+type/evidence mode changes in place) — contradicting "config never
+changes" for the general case. Resolved with a separate, explicitly
+rare-labelled <code>IndexMeta::rewrite_config(config, genomes)</code> (preserves
+<code>state</code>, overwrites everything else); callers refresh their own cached
+<code>Arc&lt;IndexMeta&gt;</code> afterward (<code>self.meta = Arc::new(IndexMeta::open(self)?)</code>)
+since <code>IndexMeta</code> has no way to reach back into whichever <code>KmerIndex</code>
+holds it.</p>
+<p><strong>Consequence confirmed, not just hoped for</strong>: with <code>mark_scattered</code> no
+longer touching anything in memory, <code>PartitionRouter</code> genuinely never
+needs <code>&amp;mut KmerIndex</code> — <code>PartitionRouter&lt;'a&gt; { index: &amp;'a KmerIndex }</code>,
+<code>new(&amp;'a KmerIndex)</code>. This is effectively the <code>PartitionRouter</code> half of
+(5)'s "order of remaining work" item done as a side effect; <code>KmerPartition</code>/
+<code>Layer</code> themselves are still unimplemented for (5).</p>
+<p><strong>Blast radius — much larger than (9)/(10), touched nearly every crate</strong>:
+every <code>.meta().genomes</code>/<code>.meta.genomes</code> field access became a fallible
+<code>.genomes()?</code> method call (<code>genomes</code> reads <code>io::Result&lt;Vec&lt;GenomeInfo&gt;&gt;</code>
+now, not a field), and <code>.meta_mut()</code> was removed outright (no more direct
+field mutation from outside <code>IndexMeta</code>). Fixed across:
+- <code>obikindex</code> internals: <code>meta.rs</code>/<code>state.rs</code>/<code>kmer_index.rs</code>/<code>builder.rs</code>
+  (full rewrites), <code>reindex.rs</code>/<code>select.rs</code> (switched to <code>rewrite_config</code>),
+  <code>merge.rs</code> (heaviest single file — genome counts precomputed once per
+  source into a <code>Vec&lt;Vec&lt;GenomeInfo&gt;&gt;</code> up front rather than re-reading
+  <code>index.meta</code> from disk repeatedly through the function, sentinel write
+  replaced with <code>dst2.meta.mark_indexed()</code>), <code>stats.rs</code>, <code>distance.rs</code>,
+  <code>dump.rs</code>, <code>predicate.rs</code> (its <code>IndexMeta</code>-inherent <code>matching_genome_indices</code>/
+  <code>build_group_filter</code> now read genomes fresh internally), <code>mod.rs</code>/<code>lib.rs</code>
+  (sentinel constant re-exports removed — <code>IndexState</code> no longer has
+  <code>SENTINEL_*</code>/<code>detect()</code> at all).
+- <code>obikindexer::extensions::PrivateBuilder</code>: <code>mark_scattered</code> signature
+  dropped <code>&amp;mut self</code> → <code>&amp;self</code>; the four <code>mark_*</code>/<code>write_spectrum</code> bodies
+  became one-line delegations to <code>self.meta().mark_*()</code>.
+- <code>obikphylo::siblings</code>: <code>alignment.rs</code>/<code>cardinality.rs</code>/<code>distance.rs</code>/
+  <code>entropy.rs</code>/<code>sankoff_bundle.rs</code>/<code>stats.rs</code>/<code>tests.rs</code> — all had
+  <code>self.meta().genomes.len()</code>-shaped reads, mechanically fixed to
+  <code>.genomes().map_err(OKIError::Io)?.len()</code> (tests: <code>.unwrap()</code>).
+- <code>obikmer::cmd::*</code>: <code>annotate</code> (rewrote its rename path to load genomes
+  once, mutate the in-memory <code>Vec</code>, then <code>idx.meta().set_genomes(...)</code>
+  instead of <code>meta_mut()</code>), <code>filter</code>/<code>pack</code>/<code>dump</code>/<code>unitig</code>/<code>merge</code>/<code>select</code>/
+  <code>phylo</code> (fetch-once-then-use pattern for genome counts/labels),
+  <code>utils/maintenance.rs</code> (<code>run_rename</code> now calls the pre-existing
+  <code>IndexMeta::rename_genome</code>, dropping its own hand-rolled field mutation
+  entirely), <code>index/mod.rs</code> (three <code>idx.state() &lt; IndexState::X</code>
+  resumability checks needed a fallible read — factored into a small
+  <code>current_state(&amp;KmerIndex) -&gt; IndexState</code> helper rather than repeating
+  the same <code>unwrap_or_else</code> three times), <code>query/*</code> (<code>emit_batch</code>'s
+  signature changed from <code>&amp;IndexMeta</code> to <code>&amp;[GenomeInfo]</code>, and <code>genomes</code> is
+  now fetched once in <code>run()</code> and threaded down through <code>process_chunk</code>
+  as <code>Arc&lt;Vec&lt;GenomeInfo&gt;&gt;</code> rather than re-reading <code>index.meta</code> from disk
+  on every chunk — a deliberate deviation from the "always re-read"
+  default, justified because this is a genuine per-chunk hot path, unlike
+  every other call site touched in this pass).
+- One <code>&amp;IndexMeta</code>-vs-<code>Arc&lt;IndexMeta&gt;</code> argument-type mismatch pattern
+  recurred at several CLI call sites (<code>build_filters</code>/<code>build_specs</code>/
+  <code>emit_batch</code>'s original signature) — resolved via <code>Arc</code>'s deref
+  coercion (<code>&amp;idx.meta()</code> coerces to <code>&amp;IndexMeta</code>) rather than changing
+  every downstream signature to accept <code>Arc&lt;IndexMeta&gt;</code>.</p>
+<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and
+<code>cargo test --workspace</code> both green (0 failures) after the full
+propagation, <code>scripts/smoke_test_index.sh</code> green (870 kmers, same as every
+prior round), plus a manual CLI run of <code>index</code> (×2) → <code>merge</code> → <code>select</code>
+→ <code>reindex</code> → <code>utils --new-label</code> (rename) → <code>utils --stats</code>, all exit 0,
+confirming the four most-affected commands (the ones (10)'s verification
+already flagged as under-covered by the automated test suite) still work
+end to end against the new <code>Arc&lt;IndexMeta&gt;</code>/on-disk-<code>IndexState</code> shape.</p>
+<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign itself
+(only the <code>PartitionRouter</code>-<code>&amp;mut</code>-removal piece landed, as a side
+effect); the <code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly
+deferred); the future cache-manager crate. (8)'s <code>obikalgorithm::
+Algorithm</code> trait, resumed and closed in (12) below.</p>
+<h2 id="12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session">(12) done (2026-08-21): <code>obikalgorithm::Algorithm</code> — the shared trait, resumed and closed in one session</h2>
+<p>Resumed (8)'s point 2 through a point-by-point discussion of what's
+actually common across the four pipeline algorithms, now that (11) made
+<code>KmerIndex</code> itself immutable everywhere. Four sub-points, each closed
+before moving to the next:</p>
+<p><strong>1. Receiver (<code>&amp;self</code> vs <code>&amp;mut self</code>)</strong> — investigated whether (11)'s
+removal of <code>&amp;mut KmerIndex</code> also removed the need for <code>PartitionRouter::
+run</code> to take <code>&amp;mut self</code>. It didn't: <code>PartitionRouter</code> holds real
+per-run state of its own (<code>writers: Vec&lt;Option&lt;SKFileWriter&gt;&gt;</code>, open file
+handles, purely in-process RAM — confirmed by checking where <code>writers</code> is
+stored, nothing to do with <code>KmerIndex</code>/disk truth), unrelated to the
+index. First proposal (wrap <code>writers</code> in <code>RefCell</code> so all four could
+share a uniform <code>&amp;self</code>) was retracted on pushback: manufacturing
+interior mutability with runtime borrow checks to satisfy a cosmetic
+uniformity that Rust doesn't even require is over-engineering — a trait
+method's receiver must match exactly across implementors, but nothing
+stops that shared receiver from being <code>&amp;mut self</code> with three of the four
+implementations simply not using the mutability. Settled: trait declares
+<code>&amp;mut self</code>; <code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> (previously <code>&amp;self</code>)
+now also take <code>&amp;mut self</code>, unused.</p>
+<p><strong>2. <code>path_source</code> as a <code>PartitionRouter</code> setter, not a <code>run()</code> param</strong> —
+added a <code>files: Option&lt;Box&lt;dyn Iterator&lt;Item = PathBuf&gt; + Send&gt;&gt;</code> field +
+<code>.files(impl Iterator&lt;Item = PathBuf&gt; + Send + 'static) -&gt; Self</code> setter
+(boxed rather than a generic type parameter on <code>PartitionRouter&lt;'a&gt;</code>:
+negligible cost — one <code>PathBuf</code> per input <em>file</em>, not per k-mer — for a
+much more usable type when passing the builder around). <code>run</code> now does
+<code>self.files.take().ok_or_else(...)</code>, erroring if <code>.files(...)</code> was never
+called, instead of taking <code>path_source</code> as a parameter.</p>
+<p><strong>3. Unifying the three progress-callback bound shapes</strong> — reopened, then
+resolved differently than (8) originally framed it. First proposal
+(force everything to <code>FnMut(Progress) + Send</code>) was rejected on the same
+principle as point 1: <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn(Progress) + Sync</code>
+isn't arbitrary — their callback is invoked concurrently from multiple
+rayon worker threads, and <code>FnMut</code> requires exclusive access, so forcing
+it would mean wrapping the callback in a <code>Mutex</code> for zero benefit at the
+one real call site (<code>pb.inc(1)</code>, already thread-safe). The actual
+resolution: move <code>on_progress</code> off <code>run()</code>'s signature entirely, onto a
+per-algorithm <code>.on_progress(...)</code> setter — same treatment as point 2's
+<code>path_source</code> — so each algorithm keeps its own bound (<code>PartitionRouter</code>:
+<code>FnMut(Progress) + 'a</code>, sequential, no <code>Send</code> needed; <code>LayerBuilder</code>:
+<code>FnMut(Progress) + Send + 'a</code>, crosses into <code>PartitionRunner</code>'s
+<code>thread::scope</code>-spawned controller thread once; <code>Dereplicator</code>/<code>Counter</code>:
+<code>Fn(Progress) + Sync + 'a</code>, invoked concurrently from rayon workers).
+This <em>also</em> dissolves the original problem <code>run()</code> had: once the
+callback isn't part of <code>run</code>'s signature at all, there's nothing left to
+unify there, and point 4 (below) becomes trivial.</p>
+<p><strong>4. <code>Output</code> as an associated type, <code>Error</code> fixed</strong> — trivial once (3)
+moved the callback out: <code>Error</code> was already uniform (all four return
+<code>obiskio::SKResult&lt;T&gt;</code> = <code>Result&lt;T, SKError&gt;</code>), only <code>Output</code> varied
+(<code>()</code>/<code>()</code>/<code>KmerSpectrum</code>/<code>usize</code>). First cut reused <code>obiskio::SKResult</code>
+directly as the trait's return type — <strong>caught and corrected the same
+session</strong>: <code>SKError</code> enumerates I/O-specific cases (<code>BadMagic</code>/
+<code>Truncated</code>/<code>Compression</code>/...), meaningless at the level of a generic
+"algorithm" abstraction, and borrowing it made <code>obikalgorithm</code> — meant to
+be minimal and neutral — depend on a low-level I/O crate purely to reuse
+its error type. Textbook instance of the "petits pois" failure mode
+(patch around a convenient existing type instead of asking what this
+crate should actually own). Fixed to a genuinely generic, boxed error
+type owned by <code>obikalgorithm</code> itself:</p>
+<div class="highlight"><pre><span></span><code><span class="c1">// obikalgorithm — no dependency on obiskio or any other crate</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span><span class="o">&lt;</span><span class="k">dyn</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">error</span><span class="p">::</span><span class="n">Error</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nb">Result</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">result</span><span class="p">::</span><span class="nb">Result</span><span class="o">&lt;</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">Error</span><span class="o">&gt;</span><span class="p">;</span>
+
+<span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Algorithm</span><span class="w"> </span><span class="p">{</span>
+<span class="w">    </span><span class="k">type</span><span class="w"> </span><span class="nc">Output</span><span class="p">;</span>
+<span class="w">    </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Result</span><span class="o">&lt;</span><span class="bp">Self</span><span class="p">::</span><span class="n">Output</span><span class="o">&gt;</span><span class="p">;</span>
+<span class="p">}</span>
+</code></pre></div>
+<p>Any concrete error (<code>SKError</code>, <code>std::io::Error</code>, ...) converts
+automatically via <code>?</code>, through <code>std</code>'s own blanket <code>impl&lt;E: Error + Send
++ Sync&gt; From&lt;E&gt; for Box&lt;dyn Error + Send + Sync&gt;</code> — no custom <code>From</code> impl
+needed, no dependency on the crate that defines the concrete error type.
+The four algorithms' <code>run</code> bodies needed no change beyond the signature's
+return type (every existing <code>?</code> on an <code>SKError</code>-returning subcall keeps
+compiling, converting through the same blanket impl at the boundary).</p>
+<p><code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> each <code>impl
+Algorithm for X&lt;'_&gt; { type Output = ...; fn run(&amp;mut self) -&gt; obikalgorithm::Result&lt;...&gt; { ... } }</code>
+— the old inherent <code>run</code> methods were removed outright (not kept as
+duplicates), so callers now <code>use obikalgorithm::Algorithm;</code> to call
+<code>.run()</code>. Every field-lifetime-bound boxed callback (<code>Box&lt;dyn
+FnMut(Progress) + 'a&gt;</code> etc.) is tied to the algorithm's own <code>'a</code> (the
+<code>&amp;'a KmerIndex</code> lifetime already on the struct), not <code>'static</code> — avoids
+forcing callers' progress closures to <code>move</code>-capture (and therefore clone
+or <code>Arc</code>-wrap) local state like <code>TracedBar</code>/EMA-rate accumulators that
+they'd otherwise want to keep using by reference after <code>run()</code> returns.</p>
+<p><strong>Why a new crate, not a submodule of <code>obikindexer</code></strong>: <code>obikmer::cmd::
+index::mod</code> and <code>obikphylo</code>'s own test helpers both need to call <code>.run()</code>
+on these algorithms, so the trait has to be reachable from outside
+<code>obikindexer</code> — putting it in <code>obikindexer</code> itself would work file-wise
+but conflates "the trait every algorithm implements" with "one crate's
+particular four implementations of it", the same reasoning that already
+separated <code>obikindex</code> (data model) from <code>obikindexer</code> (algorithms
+operating on it). <code>obikalgorithm</code> has <strong>no dependencies at all</strong> (see
+above); <code>obikindexer</code>, <code>obikmer</code>, and <code>obikphylo</code> (dev-dependency, for
+its test helper) all depend on it.</p>
+<p><strong>Blast radius</strong>: <code>obikindexer</code>'s four algorithm modules (struct field +
+setter + trait impl each); <code>obikmer::cmd::index::mod</code> (three call sites:
+<code>.on_progress(cb)</code> before <code>.run()</code>, unqualified now that the trait is in
+scope); <code>obikindexer::algorithms::partitionner::tests</code> and <code>obikphylo::
+siblings::tests</code> (both had direct <code>.run(None::&lt;fn(Progress)&gt;)</code>-shaped
+calls needing the same treatment). New <code>obikalgorithm</code> crate registered
+in the workspace <code>Cargo.toml</code>, depended on by <code>obikindexer</code>/<code>obikmer</code>
+(regular) and <code>obikphylo</code> (dev).</p>
+<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and <code>cargo test
+--workspace</code> both green (0 failures), <code>scripts/smoke_test_index.sh</code> green
+(870 kmers, same as every prior round) — this round didn't repeat the
+manual <code>merge</code>/<code>select</code>/<code>reindex</code> CLI exercise from (11), since nothing
+in this pass touched those commands' code paths (only the four pipeline
+algorithms and <code>cmd::index</code>, already covered by the smoke test). Reverified
+after the <code>obiskio</code>-dependency fix above (same three checks, still green,
+<code>obikalgorithm/Cargo.toml</code> now has zero <code>[dependencies]</code>).</p>
+<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign; the
+<code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly deferred); the
+future cache-manager crate (mentioned in (8) as a later, mirrored
+extension-trait exercise, not started).</p>
 <h2 id="the-problem">The problem</h2>
 <p>Reading a layer's data (MPHF + matrix) is not free: <code>MphfLayer::open</code> mmaps
 <code>mphf.bin</code> plus (<code>evidence.bin</code>/<code>fingerprint.bin</code> + <code>unitigs.bin</code>), and the

DevDoc/implementation/select/index.html

@@ -1119,17 +1119,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1157,6 +1146,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1638,17 +1644,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1676,6 +1671,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1728,23 +1740,32 @@ are preserved unchanged; only the data matrices are rewritten.</p>
 <hr />
 <h2 id="synopsis">Synopsis</h2>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>&lt;input-index&gt;
-<span class="w">        </span><span class="o">{</span><span class="w"> </span>--output<span class="w"> </span>&lt;dir&gt;<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="w"> </span><span class="o">}</span>
+<span class="w">        </span>--output<span class="w"> </span>&lt;dir&gt;
 <span class="w">        </span><span class="o">[</span>--group<span class="w">    </span>&lt;name&gt;:&lt;pred&gt;<span class="w">  </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--group-op<span class="w"> </span>&lt;name&gt;:&lt;op&gt;<span class="w">    </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-by<span class="w"> </span>&lt;key&gt;<span class="w">          </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-op<span class="w"> </span>&lt;op&gt;<span class="w">           </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--select<span class="w">   </span>&lt;col1,col2,...&gt;<span class="w">    </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--presence-threshold<span class="w"> </span>&lt;N&gt;<span class="w">      </span><span class="o">]</span>
+<span class="w">        </span><span class="o">[</span>--force-copy<span class="w">                  </span><span class="o">]</span>
 </code></pre></div>
 <hr />
 <h2 id="output-destination">Output destination</h2>
-<p>Exactly one of <code>--output</code> or <code>--in-place</code> must be specified.</p>
+<p><code>--output &lt;dir&gt;</code> is required — <code>select</code> always writes a new index; there is no
-<p><strong><code>--output &lt;dir&gt;</code></strong> — writes a new index to <code>&lt;dir&gt;</code>. The source index is
+<code>--in-place</code> mode (2026-08-28: never implemented, removed from the design). The
-unchanged. The MPHF and unitig files are copied; only the data matrices are
+source index is unchanged.</p>
-rewritten with the new column layout.</p>
+<p>Each layer's kmer-identity files (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/
-<p><strong><code>--in-place</code></strong> — rewrites the data matrices of the source index directly.
+<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code>) are never rewritten by a
-Removed or replaced columns are lost. The operation writes to temporary files
+column projection/aggregation, so they are hard-linked into the output rather
-first, then renames atomically, so an interrupted run leaves the index intact.</p>
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); <code>--force-copy</code>
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+<code>select</code> itself would affect the other). Only the <code>presence</code>/<code>counts</code>
+subdirectory is ever a genuinely new, independent file.</p>
+<p>To replace an index with a selected version of itself, select to a temporary
+directory and swap it in (<code>rm -rf INDEX &amp;&amp; mv INDEX.tmp INDEX</code>) — the case
+<code>--in-place</code> used to cover.</p>
 <hr />
 <h2 id="defining-output-columns">Defining output columns</h2>
 <h3 id="named-groups-group">Named groups — <code>--group</code></h3>
@@ -1908,9 +1929,6 @@ is &gt; N.</p>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>out<span class="w"> </span><span class="se">\</span>
 <span class="w">  </span>--select<span class="w"> </span><span class="s2">&quot;Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01&quot;</span>
 </code></pre></div>
-<h3 id="in-place-keep-only-group-a">In-place: keep only group A</h3>
-<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--in-place<span class="w"> </span>--group<span class="w"> </span><span class="s2">&quot;A:group=A&quot;</span><span class="w"> </span>--select<span class="w"> </span><span class="s2">&quot;A&quot;</span>
-</code></pre></div>
 <h3 id="compose-with-filter">Compose with filter</h3>
 <div class="highlight"><pre><span></span><code><span class="c1"># Step 1: keep only B. nana-specific k-mers</span>
 obikmer<span class="w"> </span>filter<span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>filtered<span class="w"> </span><span class="se">\</span>
@@ -1921,32 +1939,46 @@ obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </s
 </code></pre></div>
 <hr />
 <h2 id="implementation-notes">Implementation notes</h2>
-<p><code>select</code> does not rebuild the MPHF. The 256 partitions are processed in parallel
+<p><code>select</code> does not rebuild the MPHF. Every partition is processed independently
-(rayon), each writing its output independently; results require no synchronisation
+(<code>PartitionRunner</code>), each writing its own output layers; no cross-partition
-because every partition owns a distinct set of files.</p>
+synchronisation is needed.</p>
-<p>For each layer in each partition:</p>
+<p>For each layer in each partition (<code>obikselect::select_layer::select_partition</code>):</p>
 <ol>
-<li>The slot count <code>n</code> is read by opening the source data matrix.</li>
+<li><code>copy_layer_files</code> hard-links the source layer's kmer-identity files
-<li>A new data matrix is built with M columns (M = number of output columns).</li>
+   (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/
-<li>For each slot <code>s</code> in <code>0..n</code>:</li>
+   <code>fingerprint.bin</code>/<code>layer_meta.json</code>) into the destination — never a real
-<li><code>old_row = matrix.fill_row(s)</code> — reads the original <code>N</code>-column row without allocating.</li>
+   copy unless linking fails or <code>--force-copy</code> is given.</li>
-<li>For each output column <code>j</code>:<ul>
+<li>A new data matrix is built with M columns (M = number of output columns),
-<li><code>new_row[j] = aggregate(op, old_row[group_indices])</code>.</li>
+   under a fresh <code>presence/</code>/<code>counts/</code> subdirectory (never touching the
-<li>Pass-through columns are represented as single-element groups with the
+   source's own).</li>
-   default operator (<code>any</code> for presence, <code>sum</code> for count) — same code path.</li>
+<li><strong>Presence source (2026-08-28: <code>batch_presence_counts</code>)</strong>: one shared pass
-</ul>
+   over the source bit matrix computes every output group's presence count at
-</li>
+   once — row-major native for a <code>Sparse</code> source (<code>for_each_genome_in_row</code>,
-<li>The new row is written slot by slot into each column builder.</li>
+   which has no column representation to read a <code>col_view</code> from at all — the
-<li>All plain files in the source layer directory (<code>mphf.bin</code>, <code>unitigs.bin</code>,
+   reason this replaced the old per-group loop, not just an optimisation of
-   evidence files, <code>layer_meta.json</code>) are copied verbatim; only the <code>presence/</code>
+   it), deduplicated column-major (one <code>col_view</code> per <em>distinct referenced
-   or <code>counts/</code> subdirectory is rewritten.</li>
+   column</em>, not per group) for <code>Columnar</code>/<code>Packed</code>. Every <code>AggOp</code> for a bit
+   matrix is then a cheap derivation of that one count vector (<code>sum</code> = the
+   count itself, <code>any</code>/<code>max</code> = <code>count ≥ 1</code>, <code>all</code>/<code>min</code> = <code>count == group
+   size</code>, <code>none</code> = <code>count == 0</code>) — see
+   <code>obikselect::select_layer::agg_result_from_count</code>.</li>
+<li><strong>Count source</strong>: unchanged, one <code>col_view</code>-driven pass per output column
+   via <code>MatrixGroupOps</code> — <code>sum</code>/<code>min</code>/<code>max</code> are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.</li>
 <li><code>index.meta</code> is rewritten with the new genome list and updated <code>with_counts</code>.</li>
 </ol>
-<p><strong><code>--in-place</code> write strategy:</strong> new data is written to a temporary sibling
+<h3 id="known-gap-not-yet-fixed-2026-08-28">Known gap (not yet fixed, 2026-08-28)</h3>
-directory (<code>presence_new/</code> or <code>counts_new/</code>); on success the old directory is
+<p>Step 4 above still panics (<code>col_view() not available on Sparse
-removed and the temporary one is renamed into place. An interrupted run leaves
+PersistentCompactIntMatrix</code>) if the source is a <strong>count</strong> index packed
-at most one stale <code>*_new/</code> directory; the original data is intact until the
+sparse — <code>batch_presence_counts</code>' row-major treatment was only ported to the
-rename step.</p>
+bit-matrix (<code>Presence</code>) case, since that was the one actually blocking a real
+benchmark run. <code>select</code>/<code>filter</code> on a sparse-packed count index still hits
+this; the fix would follow the same shape (a <code>PersistentSparseCompactIntMatrix</code>
+row-major decode, analogous to <code>for_each_genome_in_row</code>), just not done. Since
+<code>obisys::numa::runner::PartitionRunner</code>'s panic-propagation fix (see
+<code>architecture/numa_partition_runner.md</code>), this at least fails fast (process
+panic, exit 101) instead of hanging.</p>
 
 
 

DevDoc/theory/evolutionary_distances/index.html

@@ -909,6 +909,56 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        --distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#snp-distance-catalog" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        snp-* distance catalog
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Exact formulas (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Output format: PHYLIP-relaxed by default for the distance matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -2195,6 +2245,56 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        --distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+    <nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#snp-distance-catalog" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        snp-* distance catalog
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Exact formulas (implemented, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Output format: PHYLIP-relaxed by default for the distance matrix
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
         <li class="md-nav__item">
@@ -4261,6 +4361,249 @@ among the survivors) — a single extra pass is sufficient.</p>
 <code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
 from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
 workspace <code>cargo test</code> green.</p>
+<h2 id="-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28"><code>--distance</code> unification: SNP corrections as first-class metrics (implemented, 2026-08-28)</h2>
+<p><strong>Implemented.</strong> <code>--metric</code> (renamed <code>--distance</code> — several of
+its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
+<code>--metric</code> was a misnomer) gains a family of <code>snp-*</code> values computed from the
+central-position SNP pipeline, routed internally to the sibling-annex
+machinery (<code>PairwiseTally</code>, <code>obikphylo::siblings::algorithms::pairwise</code>)
+instead of <code>cache.distance(...)</code>'s existing per-layer traversal — a different
+code path behind the same CLI surface, not just another branch of one
+formula function.</p>
+<p><strong>Why unify at the CLI level despite the implementation split</strong>: phylogenetically
+a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
+to how the matrix was produced, so exposing it as a special-cased subcommand
+instead of a <code>--distance</code> value would misrepresent its role. The
+implementation divergence (sibling-annex-based vs. plain index scan) is real
+but belongs at the routing layer, invisible to the CLI's own vocabulary.</p>
+<p><strong><code>--subsample</code> becomes optional for <code>snp-*</code> distances</strong> (it stays mandatory
+for <code>--sankoff</code>/<code>--pseudo-alignment</code>, unrelated commands): absent means
+exhaustive, achieved for free by reusing <code>sample_index</code>'s existing
+proportional-per-layer-quota mechanism with <code>n</code> set to the index-wide total
+non-monomorphic-minorant count (already available from the sibling-annex
+stats) — every layer's quota then equals its own full count, giving Bernoulli
+<code>p = 1</code> everywhere, i.e. every eligible family is drawn. No second,
+exhaustive-only driver needed. Present means sampled, exactly as <code>--sankoff</code>
+already behaves.</p>
+<p><strong>One shared tally, many derived formulas.</strong> <code>PairwiseTally</code>'s <code>subst[4][4]</code>
+per-pair substitution counts (plus marginal base frequencies derived from it)
+are the sufficient statistic for every closed-form correction below — each
+is a small pure function <code>PairwiseTally -&gt; Array2&lt;f64&gt;</code>, at the same level as
+the already-implemented <code>raw_snp_distance</code>/<code>base_pair_tally</code>/
+<code>cardinality_tally</code>. No new full scan per formula, whether the tally itself
+was built exhaustively or from a subsample.</p>
+<p><strong><code>--raw-snp-counts</code> stays a separate, unrelated flag</strong> — same underlying
+tally, but a diagnostic (<code>n_snp</code>/<code>n_shared</code>/<code>n_eligible</code> per genome pair, one
+row per pair) rather than a distance value, and its long-table shape doesn't
+fold into a single N×N matrix the way a distance does. No change to its
+existing CSV format.</p>
+<h3 id="snp-distance-catalog"><code>snp-*</code> distance catalog</h3>
+<p>All closed-form (method-of-moments / direct formula), none requiring
+per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
+<em>tree</em>-ML usage but not its <em>pairwise</em> estimator, which is closed-form like
+F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
+<table>
+<thead>
+<tr>
+<th>value</th>
+<th>corrects for</th>
+<th>inputs beyond raw counts</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>snp-raw</code></td>
+<td>nothing (uncorrected p-distance)</td>
+<td>—</td>
+</tr>
+<tr>
+<td><code>snp-jc</code> (Jukes-Cantor, JC69)</td>
+<td>multiple substitutions per site</td>
+<td>—</td>
+</tr>
+<tr>
+<td><code>snp-k2p</code> (Kimura 2-parameter, K80)</td>
+<td>+ transition/transversion rate bias</td>
+<td>ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-k81</code> (Kimura 3-parameter, K3ST)</td>
+<td>+ splits transversions into 2 categories</td>
+<td>ts/tv split, by category</td>
+</tr>
+<tr>
+<td><code>snp-f81</code> (Felsenstein 81)</td>
+<td>+ unequal base frequencies (no ts/tv split)</td>
+<td>empirical base freqs</td>
+</tr>
+<tr>
+<td><code>snp-tajima-nei</code> (Tajima-Nei 1984)</td>
+<td>same goal as F81 (equal-input model), different formula, better small-sample behavior</td>
+<td>empirical base freqs</td>
+</tr>
+<tr>
+<td><code>snp-t92</code> (Tamura 3-parameter)</td>
+<td>K2P + GC-content bias</td>
+<td>ts/tv split, GC content</td>
+</tr>
+<tr>
+<td><code>snp-f84</code> (Felsenstein 84)</td>
+<td>full empirical base freqs + single ts/tv rate</td>
+<td>empirical base freqs, ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-hky85</code> (Hasegawa-Kishino-Yano, pairwise estimator)</td>
+<td>same inputs as F84, different formula</td>
+<td>empirical base freqs, ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-tn93</code> (Tamura-Nei)</td>
+<td>full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate</td>
+<td>empirical base freqs, purine-ts/pyrimidine-ts/tv split</td>
+</tr>
+<tr>
+<td><code>snp-logdet</code> (LogDet / paralinear)</td>
+<td>no shared-model or stationarity assumption at all — general divergence-matrix determinant</td>
+<td>full empirical 4×4 divergence matrix (already <code>subst[4][4]</code>)</td>
+</tr>
+<tr>
+<td><code>snp-tv</code> (transversions-only p-distance)</td>
+<td>diagnostic/deep-divergence variant — drops transitions entirely (they saturate first)</td>
+<td>tv-only counts</td>
+</tr>
+</tbody>
+</table>
+<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
+<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
+nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
+formula as the base correction, weighted by a shape parameter <code>α</code> supplied
+by the user (<code>--gamma-shape &lt;alpha&gt;</code>), not estimated by ML. A modifier on
+existing values, not a separate enum arm per distance.</p>
+<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
+<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
+"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
+<code>snp-logdet</code> are catalogued above but <strong>not implemented</strong>: <code>snp-logdet</code>
+needs the true <em>directional</em> per-pair base co-occurrence matrix
+(<code>PairwiseTally</code> only keeps the symmetrised substitution counts
+<code>BasePairTally</code> itself wants — see <code>snp_distance.rs</code>'s own module docs for
+why that loses exactly the compositional-asymmetry information LogDet
+exists to detect), <code>snp-tajima-nei</code> needs each genome's <em>own</em> base
+composition (not the pair-pooled estimate the formulas below use), and
+<code>snp-f84</code>/<code>snp-hky85</code> had no formula independently verified against a
+primary source at implementation time (unlike every formula below, checked
+line-by-line against <a href="https://github.com/emmanuelparadis/ape">ape</a>'s own
+<code>src/dist_dna.c</code>, not re-derived from memory). Adding any of these later is
+a new function in <code>obikphylo::siblings::algorithms::snp_distance</code>, plus for
+<code>snp-logdet</code>/<code>snp-tajima-nei</code> a new field on <code>PairStats</code>/a per-genome
+accumulator — not an architecture change.</p>
+<h3 id="exact-formulas-implemented-2026-08-28">Exact formulas (implemented, 2026-08-28)</h3>
+<p>Sufficient statistic, per genome pair <code>(i, j)</code>, from
+<code>PairwiseTally::categories</code>/<code>PairwiseTally::base_freq</code> (base order always
+<code>0=A, 1=C, 2=G, 3=T</code>, matching <code>FamilyMask</code>/<code>STATE_SYMBOL</code>):</p>
+<ul>
+<li><span class="arithmatex">\(n_{ts1}\)</span>: A↔G substitutions (purine transitions), <span class="arithmatex">\(n_{ts2}\)</span>: C↔T
+  (pyrimidine transitions)</li>
+<li><span class="arithmatex">\(n_{tv1}\)</span>: A↔C and G↔T substitutions, <span class="arithmatex">\(n_{tv2}\)</span>: A↔T and C↔G
+  (Kimura's two transversion categories)</li>
+<li><span class="arithmatex">\(n_{shared}\)</span>: loci where both genomes agree</li>
+<li><span class="arithmatex">\(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\)</span> (total eligible
+  loci for the pair)</li>
+<li><span class="arithmatex">\(\pi_A, \pi_C, \pi_G, \pi_T\)</span>: pair-pooled base frequencies,
+  <span class="arithmatex">\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)</span>
+  (both genomes' calls at this pair's eligible loci, pooled — Nei &amp; Kumar's
+  standard pairwise estimator, not a whole-index average)</li>
+</ul>
+<p>Derived proportions used below:</p>
+<div class="arithmatex">\[
+p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
+P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
+Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
+Q_1 = \frac{n_{tv1}}{L}, \quad
+Q_2 = \frac{n_{tv2}}{L}, \quad
+P_1 = \frac{n_{ts1}}{L}, \quad
+P_2 = \frac{n_{ts2}}{L}
+\]</div>
+<p>Every formula below was checked term-by-term against <code>ape</code>'s own
+<code>src/dist_dna.c</code> (not re-derived from memory) before being ported to
+<code>obikphylo::siblings::algorithms::snp_distance</code>.</p>
+<p><strong><code>snp-raw</code></strong> — uncorrected p-distance:</p>
+<div class="arithmatex">\[
+d_{raw} = p
+\]</div>
+<p><strong><code>snp-tv</code></strong> — transversions-only p-distance (deliberately uncorrected —
+dropping transitions, which saturate first, <em>is</em> the correction):</p>
+<div class="arithmatex">\[
+d_{tv} = Q
+\]</div>
+<p><strong><code>snp-jc</code></strong> (Jukes-Cantor, JC69):</p>
+<div class="arithmatex">\[
+d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
+\]</div>
+<p><strong><code>snp-k2p</code></strong> (Kimura 2-parameter, K80), with <span class="arithmatex">\(a_1 = 1-2P-Q\)</span>, <span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
+<div class="arithmatex">\[
+d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
+\]</div>
+<p><strong><code>snp-k81</code></strong> (Kimura 3-parameter, K3ST), with <span class="arithmatex">\(a_1 = 1-2P-2Q_1\)</span>,
+<span class="arithmatex">\(a_2 = 1-2P-2Q_2\)</span>, <span class="arithmatex">\(a_3 = 1-2Q_1-2Q_2\)</span>:</p>
+<div class="arithmatex">\[
+d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
+\]</div>
+<p><strong><code>snp-f81</code></strong> (Felsenstein 81), with <span class="arithmatex">\(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\)</span>:</p>
+<div class="arithmatex">\[
+d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
+\]</div>
+<p><strong><code>snp-t92</code></strong> (Tamura 3-parameter), with GC content
+<span class="arithmatex">\(g = \pi_C+\pi_G\)</span>, <span class="arithmatex">\(w = 2g(1-g)\)</span>, <span class="arithmatex">\(a_1 = 1 - \dfrac{P}{w} - Q\)</span>,
+<span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
+<div class="arithmatex">\[
+d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
+\]</div>
+<p><strong><code>snp-tn93</code></strong> (Tamura-Nei), with purine/pyrimidine pooled frequencies
+<span class="arithmatex">\(g_R = \pi_A+\pi_G\)</span>, <span class="arithmatex">\(g_Y = \pi_C+\pi_T\)</span>, and</p>
+<div class="arithmatex">\[
+k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
+k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
+k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
+\]</div>
+<div class="arithmatex">\[
+w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
+w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
+w_3 = 1 - \frac{Q}{2g_R g_Y}
+\]</div>
+<div class="arithmatex">\[
+d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
+\]</div>
+<p><strong><code>+Γ</code> gamma correction</strong> (Jin &amp; Nei 1990): every formula above is a
+weighted sum of <span class="arithmatex">\(-\ln(x)\)</span> terms; the gamma-corrected version replaces
+each such term with the same weight applied to
+<span class="arithmatex">\(\alpha\left(x^{-1/\alpha} - 1\right)\)</span> instead — the standard mechanical
+substitution (as <span class="arithmatex">\(\alpha \to \infty\)</span>, this expression → <span class="arithmatex">\(-\ln(x)\)</span>,
+recovering the uncorrected formula exactly). E.g. for JC:</p>
+<div class="arithmatex">\[
+d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
+\]</div>
+<p>Verified term-by-term against <code>ape</code>'s own gamma branches for JC69/K80/F81
+(including K80's two-term form — algebraically identical to the generic
+substitution applied to <code>snp-k2p</code>'s own <span class="arithmatex">\(a_1\)</span>/<span class="arithmatex">\(a_2\)</span> terms above, checked
+both symbolically and numerically before simplifying the implementation to
+share one <code>corrected_log</code> helper across every model rather than
+special-casing K80). K81/T92/TN93's gamma branches follow the same
+mechanical substitution but weren't independently checked against an
+<code>ape</code>-equivalent reference for those three specifically — flagged here, not
+silently assumed correct.</p>
+<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
+<p><strong>Implemented.</strong> The primary distance-matrix output
+(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
+default</strong> (widely read by external NJ tools — PHYLIP <code>neighbor</code>, FastME,
+T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
+label truncation, since genome labels here routinely exceed it), a <code>--csv</code>
+flag opts back into the current CSV format, PHYLIP-strict is a possible
+future addition (not now). This changes the <em>default</em> output of every
+existing <code>--distance</code> value (jaccard, hamming, bray-curtis, ...), not just
+the new <code>snp-*</code> ones — accepted explicitly (pre-release, single developer
+user, no external consumers to break). Scoped to the distance matrix only:
+<code>--shared-kmers</code> and <code>--raw-snp-counts</code> are counts, not distances, and keep
+their existing CSV-only format.</p>
 <h2 id="references">References</h2>
 <p>The Mash mutation-rate model this discussion contrasts with:
 (Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>

DevDocMD/architecture/numa_partition_runner.md

@@ -304,10 +304,29 @@ step, capped by that node's remaining dormant workers and by the run's total
 budget (`n_total`) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.
 
+## Panic propagation (fixed, 2026-08-28)
+
+A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
+— the controller's `while completed < n_total` loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept `event_rx` open. Discovered via `obikselect`
+panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
+the process hung indefinitely instead of erroring out.
+
+Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
+`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
+partition. The controller counts it toward `completed` (unblocking the loop)
+and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain `Err` (a panic means a bug, not a normal typed failure).
+
 ## Open questions
 
 - **Error handling**: `run` currently returns the first error; remaining errors
-  are dropped.  A `Vec<E>` return would give complete diagnostics.
+  are dropped.  A `Vec<E>` return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain `Err` results.)
 
 - **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
   (start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an

DevDocMD/implementation/merge.md

@@ -194,3 +194,14 @@ spectrums/
   <label>.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 ```
+
+`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json` marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — `merge_partition` never rewrites them for pre-existing layers, only the `presence`/`counts` subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
+
+## Known issue (not yet fixed, 2026-08-28)
+
+Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. `obikmer merge -o out IDX IDX --rename-duplicates`) — panics deep in the MPHF's rank-select structure (`common_traits::select_in_word`, `assertion failed: rank < self.count_ones()`), inside `MphfLayer::find` called from `merge_partition`'s "is this source kmer already in dst" check against the bootstrap-copied `dst_layers`. Root cause not identified; ruled out so far:
+
+- **Not an empty-new-layer issue**: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
+- **Not caused by the hard-link change above**: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
+
+Only reproduces when `sources[0]` and `sources[1]` are the exact same on-disk path opened as two independent `KmerIndex` handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since `PartitionRunner`'s panic propagation fix (see `architecture/numa_partition_runner.md`) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.

DevDocMD/implementation/select.md

@@ -23,28 +23,37 @@ are preserved unchanged; only the data matrices are rewritten.
 
 ```sh
 obikmer select <input-index>
-        { --output <dir> | --in-place }
+        --output <dir>
         [--group    <name>:<pred>  ...]
         [--group-op <name>:<op>    ...]
         [--aggregate-by <key>          ]
         [--aggregate-op <op>           ]
         [--select   <col1,col2,...>    ]
         [--presence-threshold <N>      ]
+        [--force-copy                  ]
 ```
 
 ---
 
 ## Output destination
 
-Exactly one of `--output` or `--in-place` must be specified.
+`--output <dir>` is required — `select` always writes a new index; there is no
+`--in-place` mode (2026-08-28: never implemented, removed from the design). The
+source index is unchanged.
 
-**`--output <dir>`** — writes a new index to `<dir>`. The source index is
+Each layer's kmer-identity files (`mphf.bin`/`unitigs.bin`/`evidence.bin`/
-unchanged. The MPHF and unitig files are copied; only the data matrices are
+`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json`) are never rewritten by a
-rewritten with the new column layout.
+column projection/aggregation, so they are hard-linked into the output rather
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); `--force-copy`
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+`select` itself would affect the other). Only the `presence`/`counts`
+subdirectory is ever a genuinely new, independent file.
 
-**`--in-place`** — rewrites the data matrices of the source index directly.
+To replace an index with a selected version of itself, select to a temporary
-Removed or replaced columns are lost. The operation writes to temporary files
+directory and swap it in (`rm -rf INDEX && mv INDEX.tmp INDEX`) — the case
-first, then renames atomically, so an interrupted run leaves the index intact.
+`--in-place` used to cover.
 
 ---
 
@@ -186,12 +195,6 @@ obikmer select myindex --output out \
   --select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
 ```
 
-### In-place: keep only group A
-
-```sh
-obikmer select myindex --in-place --group "A:group=A" --select "A"
-```
-
 ### Compose with filter
 
 ```sh
@@ -207,28 +210,45 @@ obikmer select filtered --output final --aggregate-by site
 
 ## Implementation notes
 
-`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
+`select` does not rebuild the MPHF. Every partition is processed independently
-(rayon), each writing its output independently; results require no synchronisation
+(`PartitionRunner`), each writing its own output layers; no cross-partition
-because every partition owns a distinct set of files.
+synchronisation is needed.
 
-For each layer in each partition:
+For each layer in each partition (`obikselect::select_layer::select_partition`):
 
-1. The slot count `n` is read by opening the source data matrix.
+1. `copy_layer_files` hard-links the source layer's kmer-identity files
-2. A new data matrix is built with M columns (M = number of output columns).
+   (`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/
-3. For each slot `s` in `0..n`:
+   `fingerprint.bin`/`layer_meta.json`) into the destination — never a real
-   - `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
+   copy unless linking fails or `--force-copy` is given.
-   - For each output column `j`:
+2. A new data matrix is built with M columns (M = number of output columns),
-     - `new_row[j] = aggregate(op, old_row[group_indices])`.
+   under a fresh `presence/`/`counts/` subdirectory (never touching the
-     - Pass-through columns are represented as single-element groups with the
+   source's own).
-       default operator (`any` for presence, `sum` for count) — same code path.
+3. **Presence source (2026-08-28: `batch_presence_counts`)**: one shared pass
-   - The new row is written slot by slot into each column builder.
+   over the source bit matrix computes every output group's presence count at
-4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
+   once — row-major native for a `Sparse` source (`for_each_genome_in_row`,
-   evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
+   which has no column representation to read a `col_view` from at all — the
-   or `counts/` subdirectory is rewritten.
+   reason this replaced the old per-group loop, not just an optimisation of
+   it), deduplicated column-major (one `col_view` per *distinct referenced
+   column*, not per group) for `Columnar`/`Packed`. Every `AggOp` for a bit
+   matrix is then a cheap derivation of that one count vector (`sum` = the
+   count itself, `any`/`max` = `count ≥ 1`, `all`/`min` = `count == group
+   size`, `none` = `count == 0`) — see
+   `obikselect::select_layer::agg_result_from_count`.
+4. **Count source**: unchanged, one `col_view`-driven pass per output column
+   via `MatrixGroupOps` — `sum`/`min`/`max` are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.
 5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
 
-**`--in-place` write strategy:** new data is written to a temporary sibling
+### Known gap (not yet fixed, 2026-08-28)
-directory (`presence_new/` or `counts_new/`); on success the old directory is
+
-removed and the temporary one is renamed into place. An interrupted run leaves
+Step 4 above still panics (`col_view() not available on Sparse
-at most one stale `*_new/` directory; the original data is intact until the
+PersistentCompactIntMatrix`) if the source is a **count** index packed
-rename step.
+sparse — `batch_presence_counts`' row-major treatment was only ported to the
+bit-matrix (`Presence`) case, since that was the one actually blocking a real
+benchmark run. `select`/`filter` on a sparse-packed count index still hits
+this; the fix would follow the same shape (a `PersistentSparseCompactIntMatrix`
+row-major decode, analogous to `for_each_genome_in_row`), just not done. Since
+`obisys::numa::runner::PartitionRunner`'s panic-propagation fix (see
+`architecture/numa_partition_runner.md`), this at least fails fast (process
+panic, exit 101) instead of hanging.

DevDocMD/theory/evolutionary_distances.md

@@ -2182,9 +2182,9 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
 from the written `_iqtree_states.csv` and `A`/`C` still present). Full
 workspace `cargo test` green.
 
-## `--distance` unification: SNP corrections as first-class metrics (discussion, 2026-08-28)
+## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
 
-**Decided, not yet implemented.** `--metric` (renamed `--distance` — several of
+**Implemented.** `--metric` (renamed `--distance` — several of
 its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
 `--metric` was a misnomer) gains a family of `snp-*` values computed from the
 central-position SNP pipeline, routed internally to the sibling-annex
@@ -2249,13 +2249,618 @@ F84/TN93 and is included below. `snp-` prefix on every CLI value.
 **`+Γ` rate-heterogeneity modifier, applicable to `snp-jc`, `snp-k2p`,
 `snp-k81`, `snp-t92`, `snp-f84`, `snp-hky85`, `snp-tn93`** (not `snp-raw`,
 nothing to correct; not `snp-logdet`, no standard gamma formulation) — same
-formula as the base correction, weighted by a shape parameter `α` supplied
+formula as the base correction, weighted by a shape parameter `α` either
-by the user (`--gamma-shape <alpha>`), not estimated by ML. A modifier on
+supplied by the user (`--gamma-shape <alpha>`) or estimated from the data
-existing values, not a separate enum arm per distance.
+(`--gamma-shape auto`/`estimate`, method-of-moments — not ML; see
+"Automatic α estimation" below). A modifier on existing values, not a
+separate enum arm per distance.
+
+### Automatic α estimation (`--gamma-shape auto`)
+
+**Correction (verified against the primary source, 2026-09-11):** Jin &
+Nei (1990) itself (*"Limitations of the Evolutionary Parsimony Method of
+Phylogenetic Analysis"*, Mol. Biol. Evol. 7(2):82–102 — the paper this
+whole `+Γ` correction is cited from, confirmed algebraically to match this
+codebase's `corrected_log`/`k2p` exactly against their eq. A4, general, and
+A8, the `a = 1` case) contains **no** data-driven α-estimation procedure.
+Their own recommendation (p. 98) is a fixed default: *"we suggest that the
+gamma distance with a = 1 [eq. A8] be used. However, one may choose a
+different gamma distance, estimating a from data. Wilson et al. (1989)
+recently used a distance with a = 1/2 for restriction-site data of
+mitochondrial DNA in hominoids."* — i.e. Jin & Nei explicitly punt
+data-driven estimation to a *different* paper (Wilson et al. 1989), not
+read/verified here. The estimator below is therefore **not** "Jin & Nei's
+method" under any framing — that attribution (present in an earlier
+revision of this section) was wrong, not just under-cited.
+
+**Implemented** (`PartitionDispersion`, `obikphylo/src/siblings/algorithms/pairwise.rs`)
+as an independent method-of-moments estimator, unrelated to any specific
+published procedure: pools substitution counts by **partition** rather
+than by genome pair, during the same `reduce_pairwise` pass that builds
+`PairwiseTally` (no second scan).
+
+For partition `i`: `n_i` = substitutions pooled over every genome pair,
+`L_i` = eligible loci pooled over every genome pair, `R_i = n_i / L_i`.
+Modeling among-site rate heterogeneity the same way as the `+Γ` correction
+itself (a `Gamma(α, α)`-distributed, mean-1, multiplicative rate shared by
+every locus in a partition — the classical Poisson-Gamma/negative-binomial
+mixture, the general identity behind gamma-rate-heterogeneity corrections,
+also behind Uzzell & Corbin's (1971) original observation that substitution
+counts across sites/regions are over-dispersed relative to Poisson):
+
+\[
+\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
+\]
+
+Weighting each partition's squared deviation by its own `L_i` cancels the
+Poisson term before attributing what's left to `α`:
+
+\[
+\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
+V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
+\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
+\]
+
+where `\bar L` is the mean partition size. Returns "no estimate" (falls
+back to the uncorrected formula, warns) when fewer than 2 partitions have
+data, `\hat\mu \le 0`, or `V` doesn't exceed the Poisson floor
+`\hat\mu/\bar L` — no detectable over-dispersion, `α` would be unbounded.
+
+**Caveat, stated explicitly rather than left implicit**: unlike every
+closed-form correction in `snp_distance.rs` (each verified line-by-line
+against `ape`'s `dist_dna.c`, and now also against Jin & Nei 1990 directly
+for the base `+Γ` formula), this estimator is derived from first
+principles (the general Poisson-Gamma/negative-binomial identity) with no
+primary-source procedure behind it at all — not Jin & Nei's (confirmed
+above), and Wilson et al. (1989), the paper they point to instead, hasn't
+been read/verified either. Mathematically self-consistent (re-derived and
+checked, not guessed), but a from-scratch method, not a literature
+implementation. If `--gamma-shape` needs a value with a literature
+pedigree rather than an estimated one, Jin & Nei's own stated default,
+`α = 1` (`--gamma-shape 1`), is the better-supported choice today.
+
+Deliberately **not** gated by `--sankoff-ratio-ceiling` the way
+`base_pair_tally` is (same precedent as `cardinality_tally` — see its own
+doc comment): that filter excludes individual saturated *pairs* from a
+composition estimate computed once at the very end, from the complete
+`PairwiseTally`; the partition axis needed here only exists transiently,
+one partition at a time, while `PairwiseTally` is still being built — long
+before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
+is known. `--exclude-genome` isn't applied either, matching
+`reduce_pairwise`'s own raw per-pair fold.
+
+**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
+`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
+"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
+`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
+needs the true *directional* per-pair base co-occurrence matrix
+(`PairwiseTally` only keeps the symmetrised substitution counts
+`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
+why that loses exactly the compositional-asymmetry information LogDet
+exists to detect), `snp-tajima-nei` needs each genome's *own* base
+composition (not the pair-pooled estimate the formulas below use), and
+`snp-f84`/`snp-hky85` had no formula independently verified against a
+primary source at implementation time (unlike every formula below, checked
+line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
+`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
+a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
+`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
+accumulator — not an architecture change.
+
+### Exact formulas (implemented, 2026-08-28)
+
+Sufficient statistic, per genome pair `(i, j)`, from
+`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
+`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
+
+- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
+  (pyrimidine transitions)
+- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
+  (Kimura's two transversion categories)
+- \(n_{shared}\): loci where both genomes agree
+- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
+  loci for the pair)
+- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
+  \(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
+  (both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
+  standard pairwise estimator, not a whole-index average)
+
+Derived proportions used below:
+
+\[
+p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
+P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
+Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
+Q_1 = \frac{n_{tv1}}{L}, \quad
+Q_2 = \frac{n_{tv2}}{L}, \quad
+P_1 = \frac{n_{ts1}}{L}, \quad
+P_2 = \frac{n_{ts2}}{L}
+\]
+
+Every formula below was checked term-by-term against `ape`'s own
+`src/dist_dna.c` (not re-derived from memory) before being ported to
+`obikphylo::siblings::algorithms::snp_distance`.
+
+**`snp-raw`** — uncorrected p-distance:
+
+\[
+d_{raw} = p
+\]
+
+**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
+dropping transitions, which saturate first, *is* the correction):
+
+\[
+d_{tv} = Q
+\]
+
+**`snp-jc`** (Jukes-Cantor, JC69):
+
+\[
+d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
+\]
+
+**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
+
+\[
+d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
+\]
+
+**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
+\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
+
+\[
+d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
+\]
+
+**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
+
+\[
+d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
+\]
+
+**`snp-t92`** (Tamura 3-parameter), with GC content
+\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
+\(a_2 = 1-2Q\):
+
+\[
+d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
+\]
+
+**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
+\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
+
+\[
+k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
+k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
+k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
+\]
+
+\[
+w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
+w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
+w_3 = 1 - \frac{Q}{2g_R g_Y}
+\]
+
+\[
+d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
+\]
+
+**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
+weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
+each such term with the same weight applied to
+\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
+substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
+recovering the uncorrected formula exactly). E.g. for JC:
+
+\[
+d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
+\]
+
+Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
+(including K80's two-term form — algebraically identical to the generic
+substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
+both symbolically and numerically before simplifying the implementation to
+share one `corrected_log` helper across every model rather than
+special-casing K80). K81/T92/TN93's gamma branches follow the same
+mechanical substitution but weren't independently checked against an
+`ape`-equivalent reference for those three specifically — flagged here, not
+silently assumed correct.
+
+### α-estimation literature survey and design discussion (2026-09-11)
+
+Follow-up discussion after implementing `PartitionDispersion::estimate_alpha`
+(previous section), triggered by the user pointing out that the literature
+conflates two distinct problems: *estimating α from an alignment* versus
+*using a given α in a Jin-Nei-style corrected distance*. Jin & Nei (1990)
+itself is squarely in the second camp — α is a user-supplied parameter
+there, not something their paper estimates (confirmed by reading the paper
+directly, see previous section). This section catalogs the actual
+α-estimation literature and records which approaches were considered and
+why each was or wasn't adopted.
+
+**Bibliography** (methodological papers, not exhaustive users of `+Γ`):
+
+| Method | Reference | Needs a tree? | Estimates "true" α or a task-specific one? | Cost |
+|---|---|---|---|---|
+| ML, continuous Γ (foundational) | **Gu, Fu & Li (1995)**, MBE 12:546–557 — read in full | yes, topology fixed a priori | true α (+ θ jointly, invariant+Γ) | very high, ≤5–6 taxa in practice |
+| ML, discrete Γ | Yang (1993/94), MBE 10:1396; JME 39:306 | yes | true α | high |
+| Parsimony-based ML on inferred changes | Yang & Kumar (1996), MBE 13:650 | yes, equal branch lengths assumed | true α, biased when branches unequal | low |
+| Corrected substitutions/site → NB fit | **Gu & Zhang (1997)**, MBE 14:1106 — read in full | yes, + ancestral reconstruction | true α, close to ML accuracy | low (given a tree) |
+| Topology-optimizing grid search | **Guindon & Gascuel (2002)**, MBE 19:534 — read in full | yes, but built *from* the candidate distances themselves (BIONJ) | **not** true α — deliberately biased *upward* for topological accuracy | low–moderate (grid × tree build) |
+| Free (non-parametric) rate distribution | Susko et al. (2003), Syst. Biol. 52:594 | yes | tests the Γ assumption itself | high |
+| Parsimony counts/site → NB fit | **Wakeley (1993)**, JME 37:613 — read in full | yes, NJ tree + Fitch parsimony ancestral reconstruction | true α, biased *upward* by parsimony (conservative) | low (given a tree) |
+| Poisson-Gamma/negative-binomial, foundational | Uzzell & Corbin (1971), Science 172:1089 — **not read**, only seen via citations | unknown — likely also count-per-site/tree-based, unverified | true α (indirect, via NB overdispersion) | unknown |
+| Bayesian posterior over α | BPP/MrBayes/BEAST2/RevBayes | yes | posterior, not point estimate | very high |
+
+**Correction (2026-09-11, after reading Wakeley in full):** an earlier
+revision of this table filed Wakeley (1993) under "no tree needed" —
+wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
+1987, on the real mtDNA data; coalescent-simulated for the simulation
+study), reconstruct ancestral states at internal nodes by Fitch (1971)
+parsimony, count the *minimum* number of changes per site this implies,
+then fit either a two-rate Poisson mixture or a gamma-distributed-rates
+model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
+Johnson & Kotz 1969) to the resulting per-site count distribution — the
+same general "tree + ancestral reconstruction + per-site counts + NB fit"
+shape as Gu & Zhang (1997), just using raw parsimony counts directly
+rather than Gu & Zhang's multiple-hit-corrected `k̂`. Confirmed by
+simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
+mean *and* (more severely) the variance of per-site change counts, which
+biases `α` **upward** (toward apparent uniformity) — consistent with Gu &
+Zhang's own citation of Wakeley for exactly this bias, and with the
+general "parsimony overestimates α" pattern noted throughout this
+literature. Real hypervariable-region-1 mtDNA data gave `α̂ ≈ 0.44–0.60`
+per data set, `0.47` combined — despite the conservative bias, still
+solidly in the "strong heterogeneity" range, which is the paper's own
+point: the method is biased but usefully conservative, not useless.
+
+**Where `PartitionDispersion::estimate_alpha` actually sits**: nowhere in
+this table's tree-dependent rows. It shares only the *abstract
+mathematical identity* (Poisson-Gamma mixture ⟹ negative binomial
+over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
+genuinely different, tree-free and ancestral-reconstruction-free route:
+pairwise genome-to-genome substitution counts pooled by *partition*
+instead of per-site counts pooled across an alignment's columns after
+ancestral state reconstruction. As far as this survey has established
+(Uzzell & Corbin not read, so not fully ruled out), **no tree and no
+ancestral reconstruction of any kind** appears to be a genuinely
+distinguishing property of `PartitionDispersion`, not something it
+inherits from prior art — consistent with it being an independent
+derivation rather than a literature implementation (see previous
+section's caveat, now on firmer footing).
+
+**Why a tree-free estimate is even valid — the identifiability question.**
+Raised directly by the user after reading Gu, Fu & Li (1995), which states
+explicitly: *"one cannot estimate the rate heterogeneity when the number
+of sequences is <3."* This is a real non-identifiability result, not a
+practical inconvenience: with only 2 sequences, a site's history is
+summarized by one binary observation (differ / don't differ), collapsed
+across all sites into two aggregate numbers (transition and transversion
+proportions) — divergence time and among-site rate variance are
+confounded in that pair of numbers, with no way to separate them. Every
+method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
+sequences *sharing one evolutionary history* (a tree) specifically to get
+multiple independent looks at the *same* site's rate across different
+lineages, which is what identifies its variance.
+
+`PartitionDispersion` doesn't violate this, because it isn't estimating
+the same quantity. The literature above estimates rate variation **across
+sites, within one shared tree**. `PartitionDispersion` estimates rate
+variation **across partitions, pooled over every genome pair in the
+index** — each genome pair stands in for one independent draw of a
+partition's relative rate, the same identifying role multiple lineages
+play in the tree-based methods, just substituting "genome pair" for
+"lineage" and "partition" for "site." With `P` genome pairs (potentially
+in the hundreds here) all contributing to every partition's pooled count,
+the ≥3-samples identifiability requirement is met by the pair count, not
+by tree depth — closer in spirit to Jin & Nei's own original motivating
+scenario (comparing several genomic *regions* across one fixed panel of
+taxa) than to single-tree site-rate estimation.
+
+This substitution carries its own assumption, which must be named rather
+than left implicit: that a partition's *relative* rate (fast/slow) is
+reasonably stable **across genome pairs**, not just across sites within
+one pair — the direct analogue of the "no lineage-specific rate variation,
+only site-specific" assumption every tree-based `+Γ` method already makes,
+just moved from lineage→pair and site→partition. If this breaks — e.g. an
+index mixing very closely related and very divergent genome pairs, where a
+partition's saturation behavior differs qualitatively between the two —
+the estimate could be misled in a way a tree-based method would at least
+have the topology to detect and a tree-free, pool-everything method
+cannot. Not yet tested against real data with strongly heterogeneous
+pairwise divergence; worth keeping in mind as the main open validity
+question for `--gamma-shape auto`, not the citation question (now settled)
+this whole discussion started from.
+
+**Gu, Fu & Li (1995), read in full — the foundational ML method, not a
+candidate to implement.** Models rate variation as invariant+Γ (`θ` =
+proportion of invariant sites, `α` = gamma shape among the variable sites),
+and derives the exact joint likelihood over nucleotide *configurations*
+across all `n` sequences on a **fixed, a-priori-known tree topology** —
+for 3 sequences this is closed-form (Jukes-Cantor), for `n` sequences it's
+a sum over `4^n` configurations weighted by coefficients tied to the
+specific topology's branching structure (their eq. 32–34), with branch
+lengths, `α`, and `θ` jointly optimized by Newton-Raphson (Hessian over all
+free parameters). The site-rate integral has the same closed form
+`E = θ + (1-θ)(1+D/α)^{-α}` that underlies `+Γ` distance corrections
+generally (the same mechanism as `corrected_log`, not a coincidence — Jin &
+Nei's own formula is the `θ=0` special case applied to a pairwise `D`
+rather than a whole-tree one). Explicitly stated by the authors as
+infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
+candidate for obikmer's typical genome counts. Notable contribution worth
+keeping in mind regardless: `ρ = (1+θα)/(1+α)` is shown to be a more robust
+summary of rate heterogeneity than `α` alone whenever `θ` isn't negligible
+— `α` and `θ` are confounded (nearly-invariant sites can come from either a
+small `α`'s left tail or from a nonzero `θ`), so `α` alone can be very
+unstable while `ρ` stays well-behaved. `PartitionDispersion` has no `θ`
+term at all (no invariant-site component), so this confound doesn't arise
+for it the same way, but it's worth remembering if an invariant+Γ variant
+is ever considered.
+
+**Gu & Zhang (1997), read in full — rejected for now.** Their procedure:
+(1) a tree with least-squares branch lengths must already exist; (2)
+ancestral states are reconstructed at every internal node (they use Zhang &
+Nei 1997's likelihood method, but note plain parsimony would work, just
+less accurately); (3) per site, branches are split into "changed"/"unchanged"
+given the ancestral reconstruction, and the *expected* substitution count
+`k̂` (corrected for multiple hits) is obtained by solving their eq. (7)
+(or eq. 12 for the generalized model) numerically — a root-find per site,
+not a closed form; (4) the site-level `k̂` values (real-valued, not
+integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
+get α — again no closed form, numerical optimization. This is a real
+per-site pipeline requiring a tree *before* distances/α can be computed —
+the reverse of obikmer's current dependency order (`snp_distance` computes
+distances, which `--nj`/`--upgma` only turn into a tree *afterward*). Their
+own answer to this chicken-and-egg problem is an iterative
+distance→tree→α→distance loop, which they themselves flag as expensive.
+Adopting this would mean a new ancestral-reconstruction subsystem (even a
+parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
+ML fit — a much bigger addition than `PartitionDispersion`, not something
+to build without a concrete need beyond "closer to Gu&Zhang's own
+simulated accuracy than our moment estimator."
+
+**Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.**
+Their **Q criterion**: for a candidate α, build a tree (they use BIONJ) from
+the α-corrected distance matrix; for every internal branch, group taxa into
+the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
+distances \(\bar\delta_{AB}\), \(\bar\delta_{CD}\), etc., and let `S ≤ M ≤ L`
+be the three pairings' sums (`{AB,CD}`, `{AC,BD}`, `{AD,BC}`); the branch's
+reliability score is `Q_branch = L − M` (zero when the four points are
+perfectly tree-additive). The whole-tree `Q` is the mean over internal
+branches (negative/zero branches excluded). `α* = argmin_α Q(α)` over a
+grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
+sensitivity is highest). Complexity: `O(n²l)` once for the base
+frequencies, then `O(n²r)` for the `r` candidate distance matrices, then
+`O(n³r)` worst case for building `r` trees and evaluating `Q` on each
+(same order as the tree-building step itself, so "free" relative to it).
+
+Central finding, **the opposite direction from an initial misreading in
+this discussion**: `α_opt` (Guindon-Gascuel's topology-optimal value) is
+**always ≥ the true α**, not the reverse — underestimating rate
+heterogeneity (i.e., picking a larger α than reality) reduces the variance
+of distance estimates and thereby improves NJ/BIONJ topological accuracy,
+especially when the molecular clock roughly holds (where `α_opt → ∞`, i.e.
+no correction at all is topologically best). This is an *empirical
+regularity* observed across their tested conditions (η ∈ {0.5, 2.0}, true
+α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
+guarantee would be another unverified-claim mistake of the kind this whole
+discussion has been correcting.
+
+**Proposed integration (not yet implemented)**: since `α_true ≤ α_opt`
+empirically, `PartitionDispersion::estimate_alpha()`'s output could seed
+the *lower bound* of Guindon-Gascuel's grid (search `[α̂, ∞)` instead of
+`[0.1, 5000]`), narrowing the grid substantially and reusing the existing
+estimator rather than needing an independent "true α" source — this also
+sidesteps Gu & Zhang's tree-first dependency problem entirely, since
+Guindon-Gascuel's own grid already builds a tree per candidate. Still
+requires: (a) a **BIONJ** implementation (see below), (b) the Q-criterion
+subtree-grouping logic, (c) a grid-search driver. Not started.
+
+**BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.**
+Searched crates.io directly (API query for "bionj": 0 results) and read the
+docs.rs pages of the most plausible candidates: `phylotree` (builds/reads/
+manipulates trees but has no distance-matrix reconstruction — no NJ, no
+BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
+after checking the actual docs), `speedytree` (NJ only — Canonical/RapidNJ/
+Hybrid variants, explicitly no BIONJ), `nj` (plain NJ only), `phylo` (no
+distance-based reconstruction at all). `obikphylo::neighbor_joining`
+(`tree.rs`) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
+how branch lengths are weighted during agglomeration (accounts for
+distance-estimate variance, not just the sum), a modest delta over an
+existing NJ implementation rather than a rewrite, but would need to be
+written in-house if the Guindon-Gascuel integration above is ever pursued
+(their own paper notes other tree-building methods gave similar results in
+their simulations, so plain NJ may be an acceptable substitute if BIONJ
+proper is judged not worth the effort).
+
+### Sharing one sample across algorithms (`--sankoff`/`--tnt`/`--phyg`/`--iqtree` + `snp-*` `--distance`) — implemented 2026-09-11
+
+**Correctness bug, not just a performance one.** `--sankoff`/`--tnt`/`--phyg`/
+`--iqtree` (via `SiblingExt::sankoff_bundle`) and a `snp-*` `--distance`
+(via `SiblingExt::snp_distance`) always consume the *exact same* selection
+parameters when requested together in one `obikmer phylo` invocation
+(`n`/`free_loss`/`no_ambiguity`/excluded-set/`entropy_bias` — the CLI has
+no way to give them different values in one run, see `cmd/phylo/args.rs`).
+Before this fix, each independently called `sample_index`, and
+`sample_layer`'s `rand::rng()` (`subsample.rs:235`) is a **thread-local
+generator that advances across calls, not reseeded each time** — so the
+second call silently drew a *different* random sample of sites than the
+first, even with identical parameters. Verified empirically (throwaway
+8-genome index): running `snp-jc` alone vs. combined with `--sankoff`
+produced visibly different distance matrices from the same index/params.
+This defeated the actual point of combining these flags — running several
+algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a `snp-*` distance) on
+one *identical* site selection for direct comparison.
+
+**Fix**: `SankoffBundle` (`sankoff.rs`) now retains its `PairwiseTally`/
+`PartitionDispersion` internally (previously consumed into `raw`/
+`base_pair_tally`/`cardinality_tally` and dropped) and exposes
+`SankoffBundle::snp_distance(kind, gamma_shape)`, which computes the
+matrix from that *same* already-sampled tally — no second `sample_index`
+call. The shared post-sampling logic (the `--gamma-shape` support check,
+`alpha` resolution, final matrix build — previously all inline in the
+`snp_distance()` free function) was factored into
+`pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)` in
+`snp_distance.rs`, called by both the standalone `snp_distance()` (after
+its own fresh sampling) and `SankoffBundle::snp_distance` (reusing the
+bundle's). `cmd/phylo/mod.rs` keeps the `Option<SankoffBundle>` alive past
+the Sankoff-family `if` block and, when a `snp-*` `--distance` is also
+requested, calls `bundle.snp_distance(...)` instead of
+`cache.snp_distance(...)` whenever a bundle was built — since both branches
+are driven by the same `args.*` fields, the parameters trivially always
+match when both fire; no runtime "do the params match" check needed.
+Verified end-to-end: the reused path logs `(reusing the Sankoff bundle's
+sample)` and the `snp_distance` stage timer reads `0ms` (`formula(kind)`
+is `O(n²)` post-processing, no I/O), vs. ~150-230ms for a fresh sample on
+the same tiny test index.
+
+#### Explicit `--session` (implemented 2026-09-12)
+
+The in-process fix above only covered reuse *within one command*.
+`--session DIR` (`cmd/phylo/args.rs`) names a directory (outside the
+index) that persists the `snp-*` `--distance` sample — the
+`PairwiseTally`/`PartitionDispersion` pair `sample_index` would otherwise
+rebuild from scratch — across *separate* `obikmer phylo` invocations.
+
+**Crate placement, as planned**: a new **`obiksession`** crate
+(`src/obiksession/`), domain-agnostic — no `PairwiseTally`/"site" concept
+anywhere in it. `Session::open(dir, params: &[u8], force: bool)` handles
+directory lifecycle and the params-conflict check against an opaque byte
+blob; `Session::store(name, bytes)`/`Session::restore(name) -> Option<Mmap>`
+handle a checksummed artifact cache. `obikphylo`
+(`siblings/algorithms/session_cache.rs`, shared by `snp_distance.rs` and
+`sankoff.rs` — see below) is the consumer: it serializes its own
+`PairwiseTally`/`PartitionDispersion`/`SnpAlignment` to bytes (via `rkyv`)
+and hands them to `obiksession`, which never sees their type.
+
+**Serialization**: `rkyv` 0.8.18, added as planned. One deviation from the
+original zero-copy pitch, scoped down deliberately: artifacts are restored
+via `rkyv::from_bytes` (full owned deserialize) rather than `rkyv::access`
+(zero-copy over the `mmap`). True zero-copy would require every
+`PairwiseTally` query method (`categories`, `base_freq`, `pair`, ...) to
+work generically over `Archived<PairwiseTally>` as well as the owned type
+— a separate, larger change not needed to get the actual win (skipping
+`sample_index`'s expensive re-scan of the sibling annex; deserializing an
+already-in-memory-sized count array is comparatively cheap CPU, not I/O).
+`#[derive(Archive, Serialize, Deserialize)]` was added to `PairStats`,
+`PairwiseTally`, and `PartitionDispersion` (`pairwise.rs`) for this.
+
+**Startup behavior, as designed**: `Session::open` creates `DIR` if
+missing (fresh, params recorded); if `DIR` already holds different saved
+params, `cmd/phylo` reports a hard error and exits rather than silently
+preferring one side, unless `--session-force` is given (which discards the
+directory's cached artifacts and starts over under the new params) —
+implements exactly the "explicit escape hatch, not silent override"
+decision from the design discussion. `obiksession` itself doesn't decode
+either side's bytes to produce a field-by-field diff (true to "opaque
+blob," left to whoever needs it); `cmd/phylo`'s conflict message is
+currently a single generic sentence listing which flags could be the
+cause, not a computed diff — an honest v1 simplification, not a limitation
+of `obiksession` itself.
+
+**Per-artifact caching, as designed**: only `PairwiseTally`/
+`PartitionDispersion` are cached (`snp_distance.rs`'s `restore_tally`/
+`store_tally`) — `ratio_ceiling`/`gamma_shape` remain pure post-processing
+over whatever tally is in hand, cached or fresh, exactly as planned.
+
+**Locking, reconsidered and kept as the whole-session `DirLock`.** First
+written up as a corner cut ("per-entry locking was sketched, not
+implemented"). On actually working through the concrete race, whole-session
+locking turned out to be the *correct* choice, not a simplification:
+`--session-force` resets a session by deleting its whole `artifacts/`
+directory. If the directory lock were only held briefly (during
+`Session::open`, then released so a long-running `Session` doesn't block
+other processes from touching *different* artifacts), a second process
+calling `open(..., force: true)` could acquire the lock, see mismatched
+params, and wipe `artifacts/` **while a first process's already-open
+`Session` is mid-checkpoint** — genuine corruption risk, not just wasted
+work. Holding the lock for the `Session`'s entire lifetime makes that
+sequence impossible: a `force` reset must wait for every currently-open
+`Session` on that directory to be dropped first. The concurrency this
+gives up (two *different* processes reading/writing *different* artifacts
+of the same session at once) isn't exercised by anything today — only one
+code path (`snp_distance`/`sankoff_bundle`, sequentially per invocation)
+uses sessions at all.
+
+**Per-layer chunked/resumable dumps — implemented.** `subsample::sample_index`
+gained a `skip_layers: usize` parameter (skip the first N (partition,
+layer) pairs, in its own deterministic iteration order, entirely — no
+I/O, no `on_layer` call) and its `on_layer` callback gained a `raw_index`
+argument (that pair's 0-based position in the same order), so a caller can
+tell exactly how far a run has gotten. `snp_distance`/`sankoff_bundle`
+checkpoint their tally/dispersion (and, for `sankoff_bundle`, the
+in-progress `SnpAlignment`) to the session every
+`session_cache::CHECKPOINT_INTERVAL_LAYERS` (8) layers or
+`CHECKPOINT_INTERVAL` (30s) of wall time, whichever comes first — bounding
+how much completed scanning a crash can lose, without paying a full
+tally-rewrite on every one of what can be thousands of layers. A `progress`
+artifact records how many layers are reflected in the checkpoint, with a
+`PROGRESS_COMPLETE` (`u64::MAX`) sentinel once every layer has been
+processed (distinct from any real layer count, so "finished" is never
+confused with "stopped after N layers where N happens to equal the total").
+
+Verified with a real kill: built a 10-genome/64-partition index, ran
+`--distance snp-jc --session DIR` under `timeout 0.4`, confirmed a
+checkpoint at `progress=8` (exactly `CHECKPOINT_INTERVAL_LAYERS`) was on
+disk, then re-ran the same command and confirmed it resumed from layer 8
+and completed successfully — 0-effort verification that the mechanism
+itself is real, not just plausible-sounding.
+
+**Important correction from that same test — resume is *not* bit-reproducible,
+and shouldn't be.** The output matrix from the resumed run differed from
+an uninterrupted control run against the same index/params. Root cause:
+`sample_layer`'s random draws come from `rand::rng()`, a per-process,
+OS-seeded generator with no continuity across a process boundary — the
+layers processed before a kill and the layers processed after a resume
+come from two independent random streams. An earlier revision of this doc
+(and of the code comments) claimed resume "reproduces the exact same
+sample an uninterrupted run would have" — that's wrong, and was corrected
+after the user pointed out the actual reasoning error: true randomness has
+no notion of continuity in the first place, so there was never a
+correctness requirement for a resumed run to match an uninterrupted one
+bit-for-bit. The sample produced after a resume is exactly as legitimate a
+random draw as one from an uninterrupted run — just a *different* one,
+same as running the command twice without `--session` already gives two
+different samples today. That variance is explicitly wanted (see below),
+not a defect to fix. Comments in `subsample.rs`/`session_cache.rs` were
+corrected to state this plainly instead of the false reproducibility claim.
+
+**Explicitly declined: deterministic/seeded sampling.** Raised as a
+possible fix for the above (seed `sample_layer`'s RNG from a per-session
+value plus `(partition, layer)`, making a resume reproduce bit-identical
+results, and, if applied everywhere, making *any* two runs with identical
+parameters produce identical samples). The user explicitly declined this
+for now: repeated runs are relied on to measure sampling variance, and a
+`--seed` flag would need its own explicit design if wanted later — this is
+not an accidental gap, it's a stated preference to revisit only if asked.
+
+**`SankoffBundle` integration — implemented.** `sankoff_bundle` accepts
+the same `session: Option<&Session>` and uses the *same* artifact names
+(`session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}`)
+as `snp_distance`, plus its own `ALIGNMENT_ARTIFACT` (`sankoff.rs`) folded
+in lockstep with the tally at every checkpoint (same `on_layer` call
+produces both, so they can never drift out of sync). Consequence verified
+directly: a tally checkpointed by `--sankoff --session DIR` is restored by
+a *later, separate* `--distance snp-k2p --session DIR` invocation with no
+`--sankoff` at all, and vice versa — cross-consumer reuse, not just
+same-command reuse, exactly the synergy this design aimed for.
+
+**Per-artifact caching, as designed**: only the tally/dispersion (plus,
+for `sankoff_bundle`, the alignment) are cached — `ratio_ceiling`/
+`gamma_shape` remain pure post-processing over whatever tally is in hand,
+cached or fresh, exactly as planned.
+
+**Still not done — genuine follow-up, not a justified tradeoff:**
+**exhaustive-sample caching for `--subsample`-varying exploration.** The
+design discussion's suggestion — cache the exhaustive (`n = None`) sample
+once and subsample *from* it on each run instead of one session per exact
+`n` — wasn't implemented; a session today is still scoped to one exact
+parameter tuple, `--subsample` included.
 
 ### Output format: PHYLIP-relaxed by default for the distance matrix
 
-**Decided, not yet implemented.** The primary distance-matrix output
+**Implemented.** The primary distance-matrix output
 (`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
 default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
 T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character

UserDocMD/architecture.md

@@ -20,7 +20,7 @@ An index directory is organized as `KmerIndex → partitions → layers`, with a
 
 ## Parallel execution and NUMA awareness
 
-Partition-level work (index construction, `merge`, `filter`, `reindex`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
+Partition-level work (index construction, `merge`, `filter`, `convert`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
 
 - On a multi-socket / multi-NUMA-node machine, one thread pool is pinned per NUMA node, and each partition is processed entirely by threads pinned to one node — keeping the memory a partition touches local to that node's DRAM. This matters because touching kmer data across NUMA nodes without pinning can degrade throughput by an order of magnitude or more on large multi-socket machines.
 - On a single-socket machine, Apple Silicon, or if hwloc cannot report NUMA topology, all cores are treated as one node with no pinning and negligible overhead — this is the default behavior on macOS.

UserDocMD/formats/index_layout.md

@@ -19,7 +19,7 @@ Each partition's surviving kmers are mapped to a dense range of integer slots by
 
 ## Evidence: exact vs. approximate
 
-Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`reindex`](../usage/reindex.md)):
+Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)):
 
 - **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
 - **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
@@ -51,6 +51,6 @@ Two verification modes are available, selected at build time (`index --approx`)
 
 A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
 
-Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`reindex`](../usage/reindex.md) one of the sources first if needed.
+Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`convert`](../usage/convert.md) one of the sources first if needed.
 
 `obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time.

UserDocMD/index.md

@@ -24,11 +24,10 @@ All functionality is exposed through a single binary, `obikmer`, organized as su
 | [`query`](usage/query.md) | Query an index with sequences and annotate matches |
 | [`dump`](usage/dump.md) | Dump indexed kmers as CSV |
 | [`annotate`](usage/annotate.md) | Add, update, or dump genome metadata |
-| [`phylo`](usage/phylo.md) | Compute pairwise evolutionary-distance proxies, trees, and phylogenetic exports |
+| [`phylo`](usage/phylo.md) | Compute pairwise genome distances, trees, and phylogenetic exports |
-| [`name-tree`](usage/name-tree.md) | Translate a TNT/PhyG numeric-label tree export back to real taxon names |
 | [`unitig`](usage/unitig.md) | Dump the unitigs of an index as FASTA |
 | [`estimate`](usage/estimate.md) | Estimate approximate-index parameters before indexing |
-| [`reindex`](usage/reindex.md) | Convert an index's evidence representation (exact ↔ approximate) |
+| [`convert`](usage/convert.md) | Convert an index's evidence representation (exact/approximate/hybrid), in place |
 | [`utils`](usage/utils.md) | Miscellaneous index maintenance and inspection utilities |
 | [`pack`](usage/pack.md) | Pack per-column matrix files into a single-file format |
 

UserDocMD/usage/convert.md

@@ -0,0 +1,29 @@
+# convert
+
+Convert an existing index's evidence representation in place, between exact, approximate, and hybrid.
+
+```bash
+obikmer convert INDEX (--exact-evidence | --approx-evidence BITS | --hybrid-evidence) [OPTIONS]
+```
+
+## Arguments
+
+| Argument | Description |
+|---|---|
+| `INDEX` | Index directory to convert (modified in place) |
+
+## Options
+
+Exactly one of the first three is required:
+
+| Option | Description |
+|---|---|
+| `--exact-evidence` | Convert to exact evidence (zero false positives) |
+| `--approx-evidence BITS` | Convert to approximate (fingerprint-only) evidence; `BITS` = fingerprint bits per slot (b) |
+| `--hybrid-evidence` | Convert to hybrid evidence (both exact and approximate bundles kept) |
+| `--evidence-bits BITS` | Fingerprint bits per slot (b) — required with `--hybrid-evidence` when the source index is currently exact; rejected otherwise (the source already fixes `b`) |
+| `-z, --findere-z Z` | Findere z parameter: number of consecutive stored kmers that must all match to confirm a hit. This does not shorten the indexed kmer length (fixed forever at `index` build time) — it extends the effective match window: on a k=31 index, `z=2` requires 32 consecutive matching bases, not 30 |
+| `--fp FP` | Target false-positive rate per z-window (e.g. `0.01`); derives `b` or `z` when one of them isn't given directly |
+| `--block-size N` | Block size for exact evidence's on-disk index (unitigs per block). Ignored when converting to pure approximate evidence. Default `1` |
+
+See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.

UserDocMD/usage/estimate.md

@@ -15,4 +15,4 @@ obikmer estimate [OPTIONS]
 | `--evidence-bits` | none | Fingerprint bits per slot (b) |
 | `--fp` | none | Target false-positive rate per z-window |
 
-Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `reindex --approx` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
+Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.

UserDocMD/usage/index_command.md

@@ -45,6 +45,6 @@ With `--approx`, evidence is stored as a compact **fingerprint** instead, tradin
 
 $$FP = \frac{1}{2^{b \cdot z}}$$
 
-where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`reindex`](reindex.md) to convert an existing index between the two representations.
+where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`convert`](convert.md) to change an existing index's representation afterwards.
 
 `z` must be strictly less than k: the effective indexed kmer length under approximate evidence is k−z+1.

UserDocMD/usage/name-tree.md

@@ -1,21 +0,0 @@
-# name-tree
-
-Translate a numerically-labelled tree export (TNT, PhyG, or any plain Newick file with bare `1`, `2`, `3`, … leaf labels) back to real taxon names, reading the label order from the FASTA that produced it.
-
-```bash
-obikmer name-tree TREE --fasta FASTA -o OUTPUT
-```
-
-## Arguments
-
-| Argument | Description |
-|---|---|
-| `TREE` | Tree file to translate — a TNT-style NEXUS export (`tree NAME = [&U] ...;`) or a plain Newick file |
-| `--fasta` | FASTA file whose record order gives the numeric taxon labels (1-based) — typically the `_sankoff.fasta`/`_snp.fasta` used to produce `TREE` |
-| `-o, --output` | Output NEXUS file path |
-
-## Output
-
-A NEXUS file with a `taxa` block, a `translate` table (numeric label → taxon name, from `--fasta`'s header order), and every tree found in `TREE`, topology unchanged — readable directly in FigTree, PearTree, `ape` (R), etc.
-
-`--tnt`'s and `--phyg`'s exports (see [phylo](phylo.md)) both number taxa `1..N` in the same order as the pseudo-alignment FASTA they were built from (`<prefix>_sankoff.fasta`), so pass that same file as `--fasta` here.

UserDocMD/usage/pack.md

@@ -24,6 +24,6 @@ The index directory is locked for exclusive access while packing.
 
 Matrix data (which genomes carry each kmer, or with what count) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one entry per genome for every kmer regardless of how many genomes actually carry it; `--sparse` instead stores each kmer's genome list directly. For presence/absence matrices, identical genome lists shared by many kmers are also deduplicated (common in real data, since kmers from the same conserved region tend to be carried by the same genomes); for count matrices, the genome list is deduplicated the same way but each kmer's actual counts are kept per-kmer, since two kmers sharing the same genome list rarely carry the same counts.
 
-On real genome collections this has measured at roughly 7x smaller on disk than the dense format for presence/absence, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--metric` distance-matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s plain `--metric` distance matrices.
+On real genome collections this has measured at roughly 7x smaller on disk than the dense format for presence/absence, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--distance` matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s `--distance` matrices.
 
-`--sparse` applies to both presence/absence and count matrices — a count index (`--metric` distance matrices included) is packed sparse the same as a presence index.
+`--sparse` applies to both presence/absence and count matrices — a count index (`--distance` matrix computations included) is packed sparse the same as a presence index.

UserDocMD/usage/phylo.md

@@ -1,6 +1,6 @@
 # phylo
 
-Compute pairwise evolutionary-distance proxies between the genomes stored in an index — a plain distance matrix, optionally trees (NJ/UPGMA), and optionally a central-position SNP model with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
+Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
 
 ```bash
 obikmer phylo INDEX [OPTIONS]
@@ -12,133 +12,256 @@ obikmer phylo INDEX [OPTIONS]
 |---|---|
 | `INDEX` | Index directory |
 
-## Distance matrix
+## Distance matrix (`--distance`)
 
 | Option | Default | Description |
 |---|---|---|
-| `--metric` | `jaccard` | One of `jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` |
+| `--distance` | `jaccard` | See the two tables below for the full list of accepted values |
-| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for Jaccard/Mash on a count index |
+| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see below). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data (see "Automatic $\alpha$ estimation" below). No effect on the other values; rejected if given together with a value that doesn't support it |
-| `--shared-kmers` | off | Also write the shared-kmer count matrix |
+| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
+| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
+| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index metric, not a `snp-*` value |
 | `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
 | `--upgma` | off | Compute and write a UPGMA tree (Newick) |
-| `-o, --output` | none (stdout) | Output file prefix; without it, the distance matrix is printed to stdout as CSV |
+| `-o, --output` | none (stdout) | Output file prefix |
 
-`hamming` requires a presence/absence index. All other metrics work on either index type; on a presence index, `jaccard`/`mash`/`hamming` are the only ones available.
+Every value routes to one of two independent computations:
 
-### Metric definitions
+### Whole-index metrics
 
-- **jaccard**: $D = 1 - \dfrac{|A \cap B|}{|A \cup B|}$ over the sets of kmers present in each genome.
+| Value | Definition |
-- **mash**: derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$.
+|---|---|
-- **hamming**: number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$.
+| `jaccard` | $D = 1 - \dfrac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}$ over the sets of kmers present in each genome |
-- **bray-curtis**: $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts.
+| `mash` | derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$ |
-- **relfreq-bray-curtis**: the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts.
+| `hamming` | number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$ |
-- **euclidean**: $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts.
+| `bray-curtis` | $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts |
-- **relfreq-euclidean**: the same formula on relative frequencies.
+| `relfreq-bray-curtis` | the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts |
-- **hellinger**: $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$.
+| `euclidean` | $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts |
-- **hellinger-euclidean**: the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$.
+| `relfreq-euclidean` | the same formula on relative frequencies |
+| `hellinger` | $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$ |
+| `hellinger-euclidean` | the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$ |
+
+`hamming` requires a presence/absence index; the others work on either index type.
+
+### `snp-*` corrections
+
+Computed from the central-position SNP model (see "Central-position SNP model" below): a family is the set of up to 4 kmers sharing identical flanking sequence and differing only at the central base. These values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
+
+For a genome pair, let $L$ be its total number of eligible loci (both genomes single-copy at that family), $p$ the raw proportion of substitutions among those loci, $P$/$Q$ the transition/transversion proportions, $Q_1$/$Q_2$ Kimura's two transversion categories (A↔C & G↔T vs. A↔T & C↔G), $P_1$/$P_2$ the purine (A↔G) / pyrimidine (C↔T) transition proportions, and $\pi_A,\pi_C,\pi_G,\pi_T$ the pair's pooled base frequencies.
+
+**`snp-raw`**
+
+$$d = p$$
+
+**`snp-jc`**
+
+$$d = -\frac{3}{4}\ln\!\left(1-\frac{4p}{3}\right)$$
+
+**`snp-k2p`**
+
+$$
+\begin{aligned}
+a_1 &= 1-2P-Q \\
+a_2 &= 1-2Q \\
+d   &= -\frac{1}{2}\ln a_1-\frac{1}{4}\ln a_2
+\end{aligned}
+$$
+
+**`snp-k81`**
+
+$$
+\begin{aligned}
+a_1 &= 1-2P-2Q_1 \\
+a_2 &= 1-2P-2Q_2 \\
+a_3 &= 1-2Q_1-2Q_2 \\
+d   &= -\frac{1}{4}\left(\ln a_1+\ln a_2+\ln a_3\right)
+\end{aligned}
+$$
+
+**`snp-f81`**
+
+$$
+\begin{aligned}
+E &= 1-\left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right) \\
+d &= -E\ln\!\left(1-\frac{p}{E}\right)
+\end{aligned}
+$$
+
+**`snp-t92`**
+
+$$
+\begin{aligned}
+g   &= \pi_C+\pi_G \\
+w   &= 2g(1-g) \\
+a_1 &= 1-\frac{P}{w}-Q \\
+a_2 &= 1-2Q \\
+d   &= -w\ln a_1-\frac{1}{2}(1-w)\ln a_2
+\end{aligned}
+$$
+
+**`snp-tn93`**
+
+$$
+\begin{aligned}
+g_R &= \pi_A+\pi_G \\
+g_Y &= \pi_C+\pi_T \\
+k_1 &= \frac{2\pi_A\pi_G}{g_R} \\
+k_2 &= \frac{2\pi_C\pi_T}{g_Y} \\
+k_3 &= 2\left(g_Rg_Y-\frac{\pi_A\pi_G\,g_Y}{g_R}-\frac{\pi_C\pi_T\,g_R}{g_Y}\right) \\
+w_1 &= 1-\frac{P_1}{k_1}-\frac{Q}{2g_R} \\
+w_2 &= 1-\frac{P_2}{k_2}-\frac{Q}{2g_Y} \\
+w_3 &= 1-\frac{Q}{2g_Rg_Y} \\
+d   &= -k_1\ln w_1-k_2\ln w_2-k_3\ln w_3
+\end{aligned}
+$$
+
+**`snp-tv`** — transversions only, deliberately uncorrected:
+
+$$d = Q$$
+
+`--gamma-shape ALPHA` applies to every value above except `snp-raw` and `snp-tv`: each $-\ln(x)$ term in the formulas above is replaced by $\alpha\left(x^{-1/\alpha}-1\right)$ (the same weight, same $x$).
+
+### Automatic $\alpha$ estimation (`--gamma-shape auto`)
+
+`--gamma-shape auto` (or the equivalent `--gamma-shape estimate`) fits $\alpha$ from the index itself instead of requiring a user-supplied value, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.
+
+The estimator pools substitution counts by **partition** rather than by genome pair: for partition $i$, let $n_i$ be the total number of substitutions observed across every genome pair, and $L_i$ the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:
+
+$$R_i = \frac{n_i}{L_i}$$
+
+Under a single shared substitution rate with no among-site heterogeneity, each $R_i$ would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as elsewhere in this correction, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:
+
+$$\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}$$
+
+where $\mu$ is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own $L_i$ removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:
+
+$$\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}}$$
+
+$$\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}$$
+
+If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), $\alpha$ is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged. When an estimate is produced, it's logged at the `info` level before the distance matrix is computed.
+
+Note: this is a method-of-moments estimator derived from the standard Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition — it is not part of Jin & Nei's (1990) original publication, which only defines the `+Γ` distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ (`--gamma-shape 1`) rather than proposing a way to estimate it from data. `alpha < 1` indicates strong among-site rate heterogeneity (many near-invariant loci, a few fast ones); `alpha` growing large makes the correction converge to the uncorrected formula.
+
+### Output
+
+Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
+
+## `--exclude-genome`, `--min-shared-family`
+
+| Option | Description |
+|---|---|
+| `--exclude-genome LABEL` | Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by `--pseudo-alignment`/`--sankoff`/a `snp-*` `--distance` value. Does not change the value computed for any remaining pair |
+| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` — never to the whole-index metrics or their matrix/NJ/UPGMA output |
+
+Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
 
 ## Central-position SNP model
 
-This is a separate operation from the distance-matrix computation above: if any option below is used, no `--metric` matrix is computed in the same invocation.
+Requires the sibling annex, built once per index:
 
-A **family** is the set of up to 4 kmers that share identical flanking sequence and differ only at the exact central base. Because $k$ is odd, the central position is well defined and maps to itself under reverse complementation. All computations below first require building the **sibling annex**, an index-wide record of which of the 4 possible central bases are observed at each family, across every genome.
+| Option | Description |
+|---|---|
+| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value |
+| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
+| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
+| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
+| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
+| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--subsample N` |
+
+A family is eligible for a genome pair $(i,j)$ only if both genomes carry exactly one of its observed forms (single-copy, unambiguous).
+
+### `--sibling-stats`
+
+`<prefix>_siblings.csv` — family size = number of distinct central bases observed at a family (1-4).
+
+| Column | Meaning |
+|---|---|
+| `genome` | genome label, or the literal `global` for the last row |
+| `1`, `2`, `3`, `4` | for a genome row: number of families of that size where the genome carries ≥ 1 member. For the `global` row: the actual deduplicated family-size histogram — not the sum of the rows above |
+
+### Family Overlap
+
+`--family-overlap` writes `<prefix>_family_overlap.csv`: header `genome,<label1>,<label2>,...`, one row per genome, cell `[i][j]` = number of variable families (family size ≥ 2) where both genome `i` and genome `j` carry a call. The diagonal is always `0`. Every genome is written, unfiltered by `--exclude-genome`/`--min-shared-family`.
+
+`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
+
+### `--pseudo-alignment`
+
+`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
 
 | Option | Default | Description |
 |---|---|---|
-| `--sibling-annex` | off | Build the sibling presence-mask annex (prerequisite for every option below) |
+| `--subsample N` | none (mandatory here) | Target number of families to sample |
-| `--exclude-genome LABEL` | none | Exclude a genome (repeatable) from every SNP/Sankoff/export computation below |
+| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
-| `--min-shared-family N` | none | Auto-exclude any genome whose mean shared-family count against every other genome (see `--family-overlap`) falls below `N` — same exclusion as `--exclude-genome`, applied on top of it |
+| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
-| `--sibling-stats` | off | Write the family-size (sibling count) distribution, per genome and globally |
-| `--raw-snp-distance` | off | Write the single-copy central-SNP p-distance matrix |
-| `--raw-snp-counts` | off | Write per-pair diagnostic counts (n_snp, n_shared, n_eligible) instead of a matrix |
-| `--snp` | off | Write a SNP-only pseudo-alignment in FASTA, IUPAC-coded |
-| `--family-overlap` | off | Write an NxN matrix of, for each genome pair, how many variable families both genomes actually carry a call for; the diagonal holds each genome's own total family count |
-
-### Locus eligibility
-
-A family is eligible for a genome pair $(i, j)$ only if genome $i$ carries exactly one of the family's observed forms (single-copy, unambiguous) and genome $j$ also carries exactly one. A genome carrying more than one form at a locus makes that locus ineligible for any pair involving it.
-
-`--raw-snp-distance` tallies, over every eligible locus of every genome pair, $n_{\text{snp}}$ (the two genomes' single forms differ) versus $n_{\text{shared}}$ (they agree — this includes invariant families). The output ratio is $\hat{p} = \dfrac{n_{\text{snp}}}{n_{\text{snp}} + n_{\text{shared}}}$.
-
-`--snp` restricts itself to *variable* families (family size $\ge 2$) and writes one FASTA record per genome, one column per family, IUPAC-coded from each genome's presence mask at that family (a single form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`).
-
-`--exclude-genome` removes a genome from these computations, re-checking column variability among the remaining genomes so that a column made monomorphic by the exclusion is dropped rather than kept artificially. It does not affect the `--metric` distance-matrix path.
-
-### Family overlap and low-coverage genomes
-
-`--family-overlap` writes, for every genome pair, how many variable families both genomes actually carry a call for (neither is absent) — a direct measure of how much informative content two genomes actually share. On genome-skim or otherwise incomplete-coverage collections, a genome with very little overlap with everything else has almost nothing left to constrain its position in a tree, and tends to end up placed unstably (near-zero branch length, grafted inside an unrelated clade) by `--tnt`/`--iqtree`.
-
-`--min-shared-family N` automates the fix: it excludes, before any computation, every genome whose mean shared-family count against all other genomes (the same statistic, averaged per row of the `--family-overlap` matrix) falls below `N`. There is no universal value for `N` — it depends on how divergent and how completely covered the genome collection is; inspect `--family-overlap`'s own output to find where the real gap sits before choosing a threshold.
-
-## Sampling at scale: `--subsample`, `--shannon`, `--entropy`
-
-On a large index (billions of families), building a full pseudo-alignment or fully calibrating the Sankoff model is not just slow — it may not fit in the time you have. `--subsample` bounds the work to a fixed number of families; `--shannon` reports how informative each family is; `--entropy`/`--entropy-sd` bias which families get kept toward the informative ones instead of choosing uniformly at random.
-
-| Option | Default | Description |
-|---|---|---|
-| `--subsample N` | none (keep everything) | Cap the number of variable families (family size ≥ 2) retained, to approximately `N` |
-| `--shannon` | off | Write `<prefix>_shannon.csv`: per-family Shannon entropy, one row per family |
 | `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
 | `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
 
-`--subsample`/`--entropy`/`--entropy-sd` affect every option that scans variable families: `--snp`, `--family-overlap`, `--shannon`, and the whole Sankoff pipeline (`--sankoff`/`--tnt`/`--phyg`/`--iqtree`, next section) — all of them draw from the *same* selection of families in one invocation, so the Sankoff calibration and the pseudo-alignment it calibrates always describe the same sites, and `--family-overlap`'s counts stay consistent with `--snp`'s columns. `--raw-snp-distance`/`--raw-snp-counts` are not affected — they always scan every family, since their p-distance estimate is a whole-index statistic, not something that benefits from being restricted to a sample.
+## Sampling at scale: `--subsample`, `--shannon`, `--entropy`
 
-### `--subsample N`
+`--subsample`, `--free-loss`, `--no-ambiguity`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--subsample` is mandatory for `--pseudo-alignment`/`--sankoff`; for a `snp-*` `--distance` value it is optional (omitted means every non-monomorphic family in the index, not an approximation).
 
-Without `--subsample`, every variable family (family size ≥ 2, i.e. every family where at least one genome differs from the rest) is used. With `--subsample N`, roughly `N` families are kept instead, chosen at random but in proportion to how many candidate families each part of the index actually holds — so the sample stays representative of the whole index, not skewed toward whichever part happens to be scanned first. If the index has fewer than `N` candidate families in the first place, `--subsample` has no effect: everything is kept.
+Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
 
-`--subsample` trades completeness for speed: `--snp`'s alignment gets fewer columns, `--sankoff`'s calibration is based on fewer observations, but the resolution work (the expensive part of a `phylo` run on a large index) scales with `N` instead of with the index's true size. Pick `N` as large as your time budget allows — a few hundred thousand to a few million families is usually enough for the transition-probability estimates in `--sankoff`'s calibration to stabilize; a smaller `N` speeds up exploratory runs.
+Every invocation, with or without `--session`, draws its own fresh random sample by default — running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately. `--session` does not change this: it makes a *specific* sample reusable on request, it does not make sampling itself reproducible from one independent run to the next.
+
+### `--session`: reusing a sample across separate commands
+
+| Option | Default | Description |
+|---|---|---|
+| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
+| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
+
+`DIR` is created if it doesn't exist. If it already holds a sample built with different `--subsample`/`--free-loss`/`--no-ambiguity`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
+
+A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
+
+If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
+
+Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead, drawn in proportion to how many candidate families each part of the index actually holds, so the sample stays representative of the whole index. If the index has fewer than `N` candidate families, `--subsample` has no effect.
 
 ### `--shannon`: measuring how informative a family is
 
-Not every variable family is equally useful for a tree: a family that differs in only one genome out of a thousand carries very little signal, and one where the pattern looks essentially random across genomes may be too saturated (multiple substitutions have overwritten the original signal) to carry real information either. `--shannon` quantifies this with the Shannon entropy (in bits) of each family's states across the genomes that carry it — low entropy means "almost everyone agrees" (an invariant or near-invariant family, phylogenetically shallow), while entropy near the ceiling for a 4-state character means "close to a random draw between the possible bases" (saturated).
+`<prefix>_entropy.csv` has one row per family visited:
-
-`<prefix>_shannon.csv` (or `shannon.csv` without `-o`) has one row per family visited:
 
 | Column | Meaning |
 |---|---|
 | `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
 | `family_idx` | the family's position within that layer |
-| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}` — the same alphabet `--sankoff`'s 16-state model uses), genomes absent from the family excluded from the count |
+| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
-| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison — a genome carrying more than one base at once counts once per base, so this can differ from `entropy15` |
+| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison |
-| `family_size` | number of distinct central bases observed anywhere in the index for this family (2–4, since monomorphic families aren't visited) |
+| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
 | `n_genomes_present` | how many genomes the entropy was computed over |
 
-Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful to inspect the entropy distribution and decide reasonable `--entropy`/`--entropy-sd` values (see below) before committing to a full run.
+Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful for choosing `--entropy`/`--entropy-sd` values before a full run.
 
 ### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
 
-By default, `--subsample` draws families uniformly — every candidate family has the same chance of being kept, regardless of how informative it actually is. `--entropy`/`--entropy-sd` change that: instead of a uniform draw, each family's chance of being kept is weighted by how close its own entropy (the `entropy15` value `--shannon` reports) is to `MU`, using a bell-shaped (Gaussian) curve of width `SIGMA` — a family with entropy exactly `MU` is the most likely to be kept, and the chance falls off smoothly the further its entropy is from `MU`, with no hard cutoff (a few families outside the target band can still get in, just less often).
+By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff. The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
 
-The filter activates as soon as *either* `--entropy` or `--entropy-sd` is given; whichever one you don't set defaults to `1.0`/`0.5`. It can be combined with `--subsample N` (the target count is still approximately `N`, now biased toward the entropy band instead of uniform — expect somewhat *fewer* than `N` families in practice, since low-weight families are dropped rather than replaced) or used alone (`--entropy` without `--subsample`: a soft entropy filter over the whole index, no size target).
+The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
-
-The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy has to be computed once, up front, and is then saved alongside the index). Every following run — even with different `MU`/`SIGMA` values — reuses that saved data and stays fast.
 
 ## Sankoff calibration and phylogenetic exports
 
 | Option | Default | Description |
 |---|---|---|
-| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment |
+| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment. Requires `--subsample N` |
-| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the calibration |
+| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the base-composition part of the calibration |
-| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, in `--sankoff`'s pseudo-alignment and every export built from it |
+| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, throughout `--sankoff` and every export built from it |
 | `--tnt` | off | Also write a TNT script (implies `--sankoff`) |
 | `--phyg` | off | Also write PhyG input files (implies `--sankoff`) |
 | `--iqtree` | off | Also write an IQ-TREE custom model and alignment (implies `--sankoff`) |
 | `--iqtree-min-freq` | `0.001` | With `--iqtree --free-loss`: also treat as missing any state rarer than this in the alignment |
-| `--sankoff-cost-scale` | `100` | Integer scaling factor applied to costs before rounding (required by TNT/PhyG's integer-only cost commands) |
+| `--sankoff-cost-scale` | `100` | Integer scaling factor applied to costs before rounding, for TNT/PhyG's integer-only cost commands |
 
 ### The 16-state model
 
-Each family is treated as a character with 16 possible states: one per subset of the 4 possible central bases actually observed (including the empty subset). Calibration combines two tallies, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`:
+Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). Calibration combines a $5 \times 5$ transition matrix over family cardinality (0-4 observed forms) and a $4 \times 4$ base-substitution matrix from unambiguous single-copy loci, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`, into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
 
-- a $5 \times 5$ transition matrix over family cardinality (0–4 observed forms) between paired genomes, and
+`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character). It does not run any external tool.
-- a $4 \times 4$ base-substitution transition matrix from unambiguous single-copy loci,
 
-which are combined into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
+With `--free-loss`, the empty state is recoded to `?` (TNT/PhyG/IQ-TREE's own missing-data symbol) instead of an ordinary, costed 16th state — `?` rather than `-`, since `-` still carries gap/indel semantics in these tools. `--free-loss` also zeroes the cardinality-transition cost between any two states, not just to/from the empty one: gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family.
-
-`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character, to avoid ambiguity with external tools' own gap semantics). It does not run any external tool.
-
-With `--free-loss`, the empty state is recoded to `?` instead — TNT/PhyG/IQ-TREE's own missing-data symbol — rather than an ordinary, costed 16th state. This matters for genome-skim or otherwise incomplete-coverage collections, where non-detection of a family is dominated by sampling failure rather than true evolutionary loss: scoring it as a real state risks grouping genomes by shared undersampling instead of shared ancestry. `?` rather than `-` because `-` still carries gap/indel semantics in these tools, and a non-detected family is not an observed deletion. `--free-loss` also drops the cardinality-transition cost between any two states, not just to/from the empty one: whether a genome shows 1 vs. 2 (etc.) detected members of a family it does carry is exactly as vulnerable to sampling failure as whether the family was detected at all, so gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family. Combine with `--min-shared-family`/`--family-overlap` above: `--free-loss` removes the false signal from non-detection, but a genome left with too little real overlap with everything else will still be placed unstably — excluding it is the other half of the fix.
 
 ### Exports
 
@@ -146,58 +269,34 @@ All three exports reuse the `--sankoff` calibrated matrix and pseudo-alignment,
 
 - **`--tnt`**: a self-contained TNT script (alignment recoded to TNT's fixed 16-symbol alphabet, integer-scaled cost matrix re-closed to a metric, a default search block).
 - **`--phyg`**: a custom cost-matrix file plus a PhyG script reusing the `--sankoff` alignment directly.
-- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths (unlike the parsimony step-counts from TNT/PhyG). Only states actually occurring in the alignment are kept and compactly renumbered.
+- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths. Only states actually occurring in the alignment are kept and compactly renumbered.
 
-TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, …, in the same order as `<prefix>_sankoff.fasta`). Use [`name-tree`](name-tree.md) on the tool's own tree output plus that same FASTA to get a NEXUS file with real taxon names.
+TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in the order the genomes appear in `<prefix>_sankoff.fasta`).
 
 ## Output files
 
-With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the plain `--metric` distance matrix is produced, on stdout. All matrices use genome labels (from the index metadata) as row/column headers, in index order; all CSVs are comma-separated with a header row.
+With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
 
 ### Distance matrix
 
 | File | Written by | Format | Content |
 |---|---|---|---|
-| `<prefix>_dist.csv` | always | CSV matrix | the `--metric` distance, 6 decimals, symmetric, diagonal 0 |
+| `<prefix>_dist.phy` | always, unless `--csv` | relaxed PHYLIP | the `--distance` matrix |
+| `<prefix>_dist.csv` | `--csv` | CSV matrix | the `--distance` matrix, 6 decimals |
 | `<prefix>_shared.csv` | `--shared-kmers` | CSV matrix | shared-kmer count per genome pair (integers) |
-| `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree, branch lengths from the `--metric` matrix |
+| `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree |
-| `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree, same matrix |
+| `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree |
 
-Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
+CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
 
 ### Central-position SNP model
 
 | File | Written by | Format | Content |
 |---|---|---|---|
 | `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
-| `<prefix>_rawsnp.csv` | `--raw-snp-distance` | CSV matrix | single-copy central-SNP p-distance ($\hat p$), or `NA` |
-| `<prefix>_rawsnp_counts.csv` | `--raw-snp-counts` | CSV table | per-pair diagnostic counts behind `_rawsnp.csv` |
-| `<prefix>_snp.fasta` | `--snp` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
 | `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
-| `<prefix>_shannon.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
+| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
-
+| `<prefix>_alignment.fasta` | `--pseudo-alignment` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
-**`_siblings.csv`** — family size = number of distinct central bases observed at a family (1–4), not "sibling count" (0–3).
-
-| Column | Meaning |
-|---|---|
-| `genome` | genome label, or the literal `global` for the last row |
-| `1`, `2`, `3`, `4` | for a genome row: number of families of that size where the genome carries ≥ 1 member. For the `global` row: the actual deduplicated family-size histogram — **not** the sum of the rows above (a family shared by several genomes would otherwise be counted once per genome) |
-
-**`_rawsnp.csv`** — same matrix layout as `_dist.csv`; each cell is $\hat p = n_{\text{snp}}/(n_{\text{snp}}+n_{\text{shared}})$, 6 decimals, or `NA` when the pair has zero eligible loci (distinguishes "identical everywhere eligible" from "nothing eligible at all").
-
-**`_rawsnp_counts.csv`** — one row per unordered genome pair (not a matrix), the counts `_rawsnp.csv`'s ratio is computed from:
-
-| Column | Meaning |
-|---|---|
-| `genome_a`, `genome_b` | the pair |
-| `n_snp` | eligible loci where the two genomes' single forms differ |
-| `n_shared` | eligible loci where they agree (includes invariant families) |
-| `n_eligible` | `n_snp + n_shared` |
-| `ratio` | $\hat p$ = `n_snp / n_eligible`, or `NA` if `n_eligible = 0` |
-
-**`_snp.fasta`** — one record per non-excluded genome, one column per variable family (family size ≥ 2), header carries an `n_sites` annotation. Each site is IUPAC-coded from the genome's presence mask at that family: single observed form → plain base; several forms → matching IUPAC ambiguity code; no form → `-`.
-
-**`_family_overlap.csv`** — same matrix layout as `_dist.csv`; cell `[i][j]` = number of `_snp.fasta` columns where both genome `i` and `j` carry a call (neither is `-`). Diagonal `[i][i]` is kept (not skipped): it holds genome `i`'s own total variable-family count.
 
 ### Sankoff calibration and exports
 
@@ -205,7 +304,7 @@ Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below):
 |---|---|---|---|
 | `<prefix>_sankoff_matrix.csv` | `--sankoff`/`--tnt`/`--phyg`/`--iqtree` | CSV matrix | calibrated 16×16 cost matrix |
 | `<prefix>_sankoff_params.yaml` | same flags | YAML | calibration report (raw tallies + derived probabilities) |
-| `<prefix>_sankoff.fasta` | same flags | FASTA | Sankoff-recoded pseudo-alignment |
+| `<prefix>_sankoff.fasta` | same flags | FASTA | Sankoff-recoded pseudo-alignment, header carries an `n_sites` annotation |
 | `<prefix>_sankoff.tnt` | `--tnt` | TNT script | ready-to-run parsimony search |
 | `<prefix>_sankoff.tcm` | `--phyg` | PhyG TCM | cost matrix in PhyG's own format |
 | `<prefix>_sankoff.pg` | `--phyg` | PhyG script | ready-to-run parsimony search |
@@ -213,33 +312,33 @@ Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below):
 | `<prefix>_iqtree.fasta` | `--iqtree` | FASTA | alignment recoded for that model |
 | `<prefix>_iqtree_states.csv` | `--iqtree` | CSV table | maps `_iqtree.model`/`_iqtree.fasta`'s compact state symbols back to `_sankoff_matrix.csv`'s alphabet |
 
-**`_sankoff_matrix.csv`** — header `state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N`: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), `0` standing for the empty/absent state (not `-`, to avoid colliding with external tools' own gap syntax). One row per source state, one value per destination state, cost $-\ln P(a,b)$, 4 decimals.
+**`_sankoff_matrix.csv`** — header `state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N`: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), `0` standing for the empty/absent state. One row per source state, one value per destination state, cost $-\ln P(a,b)$, 4 decimals.
 
-**`_sankoff_params.yaml`** — everything the calibration estimated, structured so it can be reloaded rather than re-parsed:
+**`_sankoff_params.yaml`**:
 
 | Key | Meaning |
 |---|---|
 | `ratio_ceiling` | the `--sankoff-ratio-ceiling` value used |
-| `cardinality_transitions` | 5×5 list of `{from, to, count, probability}`, family cardinality (0–4 observed forms) |
+| `cardinality_transitions` | 5×5 list of `{from, to, count, probability}`, family cardinality (0-4 observed forms) |
 | `composition_transitions` | 4×4 list of `{from, to, count, probability}`, base letters `A/C/G/T`, single-copy substitutions |
 
-**`_sankoff.fasta`** — same sites as `_snp.fasta`, recoded to match `_sankoff_matrix.csv`'s alphabet: absent state is `0` (or `?` under `--free-loss`). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.
+**`_sankoff.fasta`** — recoded to match `_sankoff_matrix.csv`'s alphabet: absent state is `0` (or `?` under `--free-loss`). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.
 
-**`_sankoff.tnt`** (`--tnt`) — self-contained TNT script: `xread` block (alignment recoded to TNT's fixed `0-9A-F` alphabet), an integer-scaled (`--sankoff-cost-scale`) and metric-closed `smatrix`, a default `hold 20; mult; export` search. Run with `printf 'proc <path>;\nquit;\n' | tnt`. Produces `<prefix>_sankoff.tre` (bare numeric leaf labels, order matching `_sankoff.fasta`) — feed both into [`name-tree`](name-tree.md) to recover taxon names.
+**`_sankoff.tnt`** (`--tnt`) — `xread` block (alignment recoded to TNT's fixed `0-9A-F` alphabet), an integer-scaled (`--sankoff-cost-scale`) and metric-closed `smatrix`, a default `hold 20; mult; export` search. Run with `printf 'proc <path>;\nquit;\n' | tnt`. Produces `<prefix>_sankoff.tre` (bare numeric leaf labels, order matching `_sankoff.fasta`).
 
-**`_sankoff.tcm`** (`--phyg`) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state `0`, since it's never actually triggered).
+**`_sankoff.tcm`** (`--phyg`) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state `0`).
 
-**`_sankoff.pg`** (`--phyg`) — script: `read(prefasta:..., tcm:...)` against `_sankoff.fasta`/`_sankoff.tcm`, a default 300s/4-instance `search`, `report(...)` writing `<prefix>_sankoff.tre` (bare numeric labels, as for `--tnt`). Run with `phyg` from the output directory (the script uses relative file names). Feed the tree plus `_sankoff.fasta` into [`name-tree`](name-tree.md) for taxon names.
+**`_sankoff.pg`** (`--phyg`) — script: `read(prefasta:..., tcm:...)` against `_sankoff.fasta`/`_sankoff.tcm`, a default 300s/4-instance `search`, `report(...)` writing `<prefix>_sankoff.tre`. Run with `phyg` from the output directory (the script uses relative file names).
 
 **`_iqtree.model`** (`--iqtree`) — lower-triangular exchangeability matrix $R(a,b) = e^{-\text{cost}(a,b)}$ (one row of increasing length per state, whitespace-separated, PAML order), followed by one line of empirical state frequencies. Only states actually occurring in the alignment are kept, compactly renumbered `0..k-1`.
 
-**`_iqtree.fasta`** (`--iqtree`) — alignment recoded to that same compact `0..k-1` alphabet (symbols `0-9A-F`). Under `--free-loss`, non-detection becomes `?` and columns left non-informative once missing calls are ignored are dropped first (required for `+ASC`); with `--iqtree-min-freq` also set (the default), any state rarer than that threshold is folded into the same `?` treatment, and non-informative columns are re-checked and dropped again after that. Run with:
+**`_iqtree.fasta`** (`--iqtree`) — alignment recoded to that same compact `0..k-1` alphabet (symbols `0-9A-F`). Under `--free-loss`, non-detection becomes `?` and columns left non-informative once missing calls are ignored are dropped first (required for `+ASC`); with `--iqtree-min-freq` also set (the default), any state rarer than that threshold is folded into the same `?` treatment, and non-informative columns are re-checked and dropped again. Run with:
 ```
 iqtree3 -s <prefix>_iqtree.fasta --seqtype MORPH -m <prefix>_iqtree.model+ASC --prefix <prefix>_iqtree -T AUTO
 ```
 
-**`_iqtree_states.csv`** (`--iqtree`) — one row per state actually kept in `_iqtree.model`/`_iqtree.fasta` (header `iqtree_symbol,canonical_symbol,frequency`): `iqtree_symbol` is the compact `0-9A-F` symbol as written in those two files, `canonical_symbol` is the matching `_sankoff_matrix.csv` state, `frequency` is that state's empirical frequency at full precision (`_iqtree.model`'s own frequency line is rounded to 6 decimals). Under `--free-loss`, absent (`0`/`?`) is never a kept state, so it never appears here — nor does any state `--iqtree-min-freq` folded away for being too rare. Use this file to identify which real state a given row/column of `_iqtree.model`'s matrix corresponds to — e.g. to check whether a state showing zero exchangeability with everything else is expected (a state combination that never co-occurs with anything else in this data) or worth investigating further.
+**`_iqtree_states.csv`** (`--iqtree`) — one row per state actually kept in `_iqtree.model`/`_iqtree.fasta` (header `iqtree_symbol,canonical_symbol,frequency`): `iqtree_symbol` is the compact `0-9A-F` symbol as written in those two files, `canonical_symbol` is the matching `_sankoff_matrix.csv` state, `frequency` is that state's empirical frequency at full precision. Under `--free-loss`, absent (`0`/`?`) is never a kept state, so it never appears here — nor does any state `--iqtree-min-freq` folded away for being too rare.
 
 ### Rare states and `--iqtree-min-freq`
 
-States that combine 3 or 4 central bases at once (IUPAC `V`/`H`/`K`.../`N`) are inherently rare — and, on real data, rare enough that they can make `iqtree3` itself numerically unstable ("Numerical underflow for lh-derivative" warnings, near-degenerate likelihood optimization). They're also more likely to be assembly/detection noise than genuine, widely-shared multi-way polymorphism, the same "sampling failure, not true signal" reasoning `--free-loss` already applies to non-detection. With `--free-loss` set, `--iqtree-min-freq` (default `0.001`, i.e. one in a thousand) extends that same missing-data treatment to any state below this frequency, not just absence. Check `_iqtree_states.csv` to see exactly which states survived and at what frequency; set `--iqtree-min-freq 0` to disable this and keep every state that occurs at all (the old behavior). Has no effect without `--free-loss` — there is no missing-data symbol to fold rare states into otherwise.
+States that combine 3 or 4 central bases at once (IUPAC `V`/`H`/`K`.../`N`) are inherently rare, and can make `iqtree3` itself numerically unstable ("Numerical underflow for lh-derivative" warnings). With `--free-loss` set, `--iqtree-min-freq` (default `0.001`, one in a thousand) extends the missing-data treatment to any state below this frequency, not just absence. Check `_iqtree_states.csv` to see exactly which states survived and at what frequency; set `--iqtree-min-freq 0` to keep every state that occurs at all. Has no effect without `--free-loss`.

UserDocMD/usage/reindex.md

@@ -1,25 +0,0 @@
-# reindex
-
-Convert an existing index's evidence representation in place, between exact and approximate.
-
-```bash
-obikmer reindex INDEX [OPTIONS]
-```
-
-## Arguments
-
-| Argument | Description |
-|---|---|
-| `INDEX` | Index directory to convert (modified in place) |
-
-## Options
-
-| Option | Default | Description |
-|---|---|---|
-| `--approx` | off | Convert to approximate evidence (default direction is approximate → exact); requires `-z`/`--evidence-bits`/`--fp` |
-| `-z, --findere-z` | none | Findere z parameter (≥ 1) |
-| `--evidence-bits` | none | Fingerprint bits per slot (b) |
-| `--fp` | none | Target false-positive rate per z-window |
-| `--block-size` | `1` | Block size for the exact on-disk index (ignored when converting to approximate) |
-
-See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.

UserDocMD/usage/select.md

@@ -1,9 +1,9 @@
 # select
 
-Project and/or aggregate the genome columns of an index into a new (or in-place) index. Where [`filter`](filter.md) selects rows (kmers), `select` operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.
+Project and/or aggregate the genome columns of an index into a new index. Where [`filter`](filter.md) selects rows (kmers), `select` operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.
 
 ```bash
-obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
+obikmer select SOURCE --output OUTPUT [OPTIONS]
 ```
 
 ## Arguments
@@ -16,8 +16,7 @@ obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
 
 | Option | Default | Description |
 |---|---|---|
-| `--output` | — | Output index directory (mutually exclusive with `--in-place`) |
+| `-o, --output` | — | Output index directory (required) |
-| `--in-place` | off | Rewrite the source index in place (mutually exclusive with `--output`) |
 | `-f, --force` | off | Overwrite an existing output directory |
 | `--group NAME:PRED` | none | Define a named group of genomes by predicate (repeatable; mutually exclusive with `--aggregate-by`) |
 | `--group-op NAME:OP` | none | Aggregation operator for a named group |
@@ -25,6 +24,8 @@ obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
 | `--aggregate-op OP` | none | Aggregation operator applied to every auto-generated group |
 | `--select COL,...` | all columns | Output columns, in order (group names or genome labels) |
 | `--presence-threshold` | `0` | Minimum count for a genome to be considered a carrier (logical operators only) |
+| `--dense` | off | Pack the output's presence matrices in the dense format instead of the default sparse one |
+| `--force-copy` | off | Copy each layer's unchanged kmer-identity files (mphf/unitigs/evidence/fingerprint) instead of hard-linking them |
 
 ## Aggregation operators
 
@@ -32,4 +33,16 @@ obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
 
 A `select` never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in `--select`) is a cheap copy.
 
-At least one of `--output`/`--in-place` is required, and at least one output column must be defined; every name listed in `--select` must resolve to either a defined group or an existing genome label. See [Genome predicates and taxonomy paths](predicates.md) for the predicate syntax used by `--group`.
+At least one output column must be defined; every name listed in `--select` must resolve to either a defined group or an existing genome label. See [Genome predicates and taxonomy paths](predicates.md) for the predicate syntax used by `--group`.
+
+## Disk usage
+
+`select` always writes to a new output directory — there is no in-place mode. Each layer's kmer-identity files (MPHF, unitigs, evidence, fingerprint) never change under a column projection/aggregation, so they are hard-linked into the output rather than copied: no extra disk is used for them, even on a very large index. Linking falls back to a real copy automatically if it fails (e.g. `SOURCE`/`OUTPUT` on different filesystems). Use `--force-copy` to always copy instead — needed when the output must be able to survive independently of the source on disk (a hard link shares the same underlying data, so overwriting one path outside `select` itself would affect the other).
+
+To replace an index with a selected version of itself, select to a temporary directory and swap it in:
+
+```bash
+obikmer select INDEX --output INDEX.tmp --group ... --group-op ... --select ...
+rm -rf INDEX
+mv INDEX.tmp INDEX
+```

benchmark/Makefile

@@ -2,23 +2,27 @@
 BINARY  := ../src/target/release/obikmer
 VENV_PY := ../.venv/bin/python3
 
-GENOMES := $(wildcard genomes/*.fna.gz)
+# All generated/downloaded artifacts live under RUN/ so the whole tree can be
+# gitignored with a single entry (benchmark/run/) — see benchmark/README.md.
+RUN := run
+
+GENOMES := $(wildcard $(RUN)/genomes/*.fna.gz)
 
 # SPECIMENS, SPECIES, and the full dependency graph are generated by
 # make_deps.py from the genome FASTA headers — like .d files in C.
 # Make rebuilds deps.mk whenever genomes/ changes and restarts.
 -include deps.mk
 
-REF_NPZS              := $(SPECIMENS:%=reference_index/%.npz)
+REF_NPZS              := $(SPECIMENS:%=$(RUN)/reference_index/%.npz)
-REF_DIST_CSVS         := $(addprefix reference_dist/, \
+REF_DIST_CSVS         := $(addprefix $(RUN)/reference_dist/, \
 	shared_kmers.csv hamming_dist.csv jaccard_dist.csv \
 	bray_curtis_dist.csv relfreq_bray_curtis_dist.csv \
 	euclidean_dist.csv relfreq_euclidean_dist.csv \
 	hellinger_dist.csv hellinger_euclidean_dist.csv)
-OBIKMER_PRESENCE_DIST := $(addprefix obikmer_dist/presence/, \
+OBIKMER_PRESENCE_DIST := $(addprefix $(RUN)/obikmer_dist/presence/, \
 	jaccard_dist.csv jaccard_shared.csv jaccard_nj.nwk \
 	hamming_dist.csv hamming_nj.nwk)
-OBIKMER_COUNT_DIST    := $(addprefix obikmer_dist/count/, \
+OBIKMER_COUNT_DIST    := $(addprefix $(RUN)/obikmer_dist/count/, \
 	jaccard_dist.csv jaccard_shared.csv jaccard_nj.nwk \
 	bray_curtis_dist.csv bray_curtis_nj.nwk \
 	relfreq_bray_curtis_dist.csv relfreq_bray_curtis_nj.nwk \
@@ -26,24 +30,28 @@ OBIKMER_COUNT_DIST    := $(addprefix obikmer_dist/count/, \
 	relfreq_euclidean_dist.csv relfreq_euclidean_nj.nwk \
 	hellinger_dist.csv hellinger_nj.nwk \
 	hellinger_euclidean_dist.csv hellinger_euclidean_nj.nwk)
-DIST_COMPARISON       := stats/dist_comparison/summary.csv
+DIST_COMPARISON       := $(RUN)/stats/dist_comparison/summary.csv
-PRESENCE_DONE         := $(SPECIMENS:%=specimen_index_presence/%/index.done)
+PRESENCE_DONE         := $(SPECIMENS:%=$(RUN)/specimen_index_presence/%/index.done)
-PRESENCE_STATS        := $(SPECIMENS:%=stats/indexing_presence/%.stats)
+PRESENCE_STATS        := $(SPECIMENS:%=$(RUN)/stats/indexing_presence/%.stats)
-COUNT_DONE            := $(SPECIMENS:%=specimen_index_count/%/index.done)
+COUNT_DONE            := $(SPECIMENS:%=$(RUN)/specimen_index_count/%/index.done)
-COUNT_STATS           := $(SPECIMENS:%=stats/indexing_count/%.stats)
+COUNT_STATS           := $(SPECIMENS:%=$(RUN)/stats/indexing_count/%.stats)
-VERIFY_PRESENCE_STATS := $(SPECIMENS:%=stats/verify_presence/%.stats)
+VERIFY_PRESENCE_STATS := $(SPECIMENS:%=$(RUN)/stats/verify_presence/%.stats)
-VERIFY_COUNT_STATS    := $(SPECIMENS:%=stats/verify_count/%.stats)
+VERIFY_COUNT_STATS    := $(SPECIMENS:%=$(RUN)/stats/verify_count/%.stats)
-SPECIFIC_PRESENCE_DONE  := $(SPECIES:%=specific_index_presence/%/index.done)
+SPECIFIC_PRESENCE_DONE  := $(SPECIES:%=$(RUN)/specific_index_presence/%/index.done)
-SPECIFIC_PRESENCE_STATS := $(SPECIES:%=stats/specific_kmer_presence/%.stats)
+SPECIFIC_PRESENCE_STATS := $(SPECIES:%=$(RUN)/stats/specific_kmer_presence/%.stats)
-SPECIFIC_COUNT_DONE     := $(SPECIES:%=specific_index_count/%/index.done)
+SPECIFIC_COUNT_DONE     := $(SPECIES:%=$(RUN)/specific_index_count/%/index.done)
-SPECIFIC_COUNT_STATS    := $(SPECIES:%=stats/specific_kmer_count/%.stats)
+SPECIFIC_COUNT_STATS    := $(SPECIES:%=$(RUN)/stats/specific_kmer_count/%.stats)
-SIMULATED_READS := $(foreach s,$(SPECIMENS),simulated_data/$(subst --,/,$s)/reads_R1.fastq.gz)
+SIMULATED_READS := $(foreach s,$(SPECIMENS),$(RUN)/simulated_data/$(subst --,/,$s)/reads_R1.fastq.gz)
-QUERY_READS      := $(foreach s,$(QUERY_SPECIMENS),query_data/$(subst --,/,$s)/reads_R1.fastq.gz)
+QUERY_READS      := $(foreach s,$(QUERY_SPECIMENS),$(RUN)/query_data/$(subst --,/,$s)/reads_R1.fastq.gz)
-QUERY_DENSE_DONE  := $(QUERY_SPECIMENS:%=query_dense/%.fasta.gz)
+QUERY_PRESENCE_DENSE_DONE   := $(QUERY_SPECIMENS:%=$(RUN)/query_presence_dense/%.fasta.gz)
-QUERY_DENSE_STATS := $(QUERY_SPECIMENS:%=stats/query_dense/%.stats)
+QUERY_PRESENCE_DENSE_STATS  := $(QUERY_SPECIMENS:%=$(RUN)/stats/query_presence_dense/%.stats)
-QUERY_SPARSE_DONE  := $(QUERY_SPECIMENS:%=query_sparse/%.fasta.gz)
+QUERY_PRESENCE_SPARSE_DONE  := $(QUERY_SPECIMENS:%=$(RUN)/query_presence_sparse/%.fasta.gz)
-QUERY_SPARSE_STATS := $(QUERY_SPECIMENS:%=stats/query_sparse/%.stats)
+QUERY_PRESENCE_SPARSE_STATS := $(QUERY_SPECIMENS:%=$(RUN)/stats/query_presence_sparse/%.stats)
-VERIFY_QUERY_STATS := $(QUERY_SPECIMENS:%=stats/verify_query/%.stats)
+QUERY_COUNT_DENSE_DONE      := $(QUERY_SPECIMENS:%=$(RUN)/query_count_dense/%.fasta.gz)
+QUERY_COUNT_DENSE_STATS     := $(QUERY_SPECIMENS:%=$(RUN)/stats/query_count_dense/%.stats)
+QUERY_COUNT_SPARSE_DONE     := $(QUERY_SPECIMENS:%=$(RUN)/query_count_sparse/%.fasta.gz)
+QUERY_COUNT_SPARSE_STATS    := $(QUERY_SPECIMENS:%=$(RUN)/stats/query_count_sparse/%.stats)
+VERIFY_QUERY_STATS := $(QUERY_SPECIMENS:%=$(RUN)/stats/verify_query/%.stats)
 
 .NOTPARALLEL:
 
@@ -58,19 +66,23 @@ VERIFY_QUERY_STATS := $(QUERY_SPECIMENS:%=stats/verify_query/%.stats)
         verify_merge_presence verify_merge_count \
         filter_presence filter_count \
         aggregate_filter_presence aggregate_filter_count \
-        pack_sparse simulate_query \
+        pack_dense_presence pack_dense_count simulate_query \
-        query_dense query_sparse \
+        query_presence_dense query_presence_sparse \
-        aggregate_query_dense aggregate_query_sparse \
+        query_count_dense query_count_sparse \
+        aggregate_query_presence_dense aggregate_query_presence_sparse \
+        aggregate_query_count_dense aggregate_query_count_sparse \
         verify_query aggregate_verify_query
 
-verify_merge_presence: stats/verify_merge_presence/current.csv
+verify_merge_presence: $(RUN)/stats/verify_merge_presence/current.csv
-verify_merge_count:    stats/verify_merge_count/current.csv
+verify_merge_count:    $(RUN)/stats/verify_merge_count/current.csv
 
 all: aggregate_verify_presence aggregate_verify_count \
      verify_merge_presence verify_merge_count \
      aggregate_filter_presence aggregate_filter_count \
      dist_comparison \
-     aggregate_query_dense aggregate_query_sparse aggregate_verify_query
+     aggregate_query_presence_dense aggregate_query_presence_sparse \
+     aggregate_query_count_dense aggregate_query_count_sparse \
+     aggregate_verify_query
 
 # ── dependency file ───────────────────────────────────────────────────────────
 
@@ -96,7 +108,7 @@ simulate_query: $(QUERY_READS)
 # ── reference kmer sets ───────────────────────────────────────────────────────
 # Prerequisites (reads → npz) are in deps.mk.
 
-reference_index/%.npz:
+$(RUN)/reference_index/%.npz:
 	bash build_reference.sh $*
 
 reference: $(REF_NPZS)
@@ -104,57 +116,58 @@ reference: $(REF_NPZS)
 # ── reference distance matrices ───────────────────────────────────────────────
 
 $(REF_DIST_CSVS) &: $(REF_NPZS) build_reference_dist.py
-	$(VENV_PY) build_reference_dist.py
+	$(VENV_PY) build_reference_dist.py \
+		--ref-dir $(RUN)/reference_index --out-dir $(RUN)/reference_dist
 
 reference_dist: $(REF_DIST_CSVS)
 
 # ── obikmer phylo (presence index) ──────────────────────────────────────────
 
-$(OBIKMER_PRESENCE_DIST) &: global_index_presence/index.done $(BINARY)
+$(OBIKMER_PRESENCE_DIST) &: $(RUN)/global_index_presence/index.done $(BINARY)
-	mkdir -p obikmer_dist/presence
+	mkdir -p $(RUN)/obikmer_dist/presence
 	$(BINARY) phylo \
-		--output obikmer_dist/presence/jaccard \
+		--output $(RUN)/obikmer_dist/presence/jaccard \
-		--metric jaccard --shared-kmers --nj \
+		--distance jaccard --csv --shared-kmers --nj \
-		global_index_presence
+		$(RUN)/global_index_presence
 	$(BINARY) phylo \
-		--output obikmer_dist/presence/hamming \
+		--output $(RUN)/obikmer_dist/presence/hamming \
-		--metric hamming --nj \
+		--distance hamming --csv --nj \
-		global_index_presence
+		$(RUN)/global_index_presence
 
 obikmer_dist_presence: $(OBIKMER_PRESENCE_DIST)
 
 # ── obikmer phylo (count index) ─────────────────────────────────────────────
 
-$(OBIKMER_COUNT_DIST) &: global_index_count/index.done $(BINARY)
+$(OBIKMER_COUNT_DIST) &: $(RUN)/global_index_count/index.done $(BINARY)
-	mkdir -p obikmer_dist/count
+	mkdir -p $(RUN)/obikmer_dist/count
 	$(BINARY) phylo \
-		--output obikmer_dist/count/jaccard \
+		--output $(RUN)/obikmer_dist/count/jaccard \
-		--metric jaccard --shared-kmers --nj \
+		--distance jaccard --csv --shared-kmers --nj \
-		global_index_count
+		$(RUN)/global_index_count
 	$(BINARY) phylo \
-		--output obikmer_dist/count/bray_curtis \
+		--output $(RUN)/obikmer_dist/count/bray_curtis \
-		--metric bray-curtis --nj \
+		--distance bray-curtis --csv --nj \
-		global_index_count
+		$(RUN)/global_index_count
 	$(BINARY) phylo \
-		--output obikmer_dist/count/relfreq_bray_curtis \
+		--output $(RUN)/obikmer_dist/count/relfreq_bray_curtis \
-		--metric relfreq-bray-curtis --nj \
+		--distance relfreq-bray-curtis --csv --nj \
-		global_index_count
+		$(RUN)/global_index_count
 	$(BINARY) phylo \
-		--output obikmer_dist/count/euclidean \
+		--output $(RUN)/obikmer_dist/count/euclidean \
-		--metric euclidean --nj \
+		--distance euclidean --csv --nj \
-		global_index_count
+		$(RUN)/global_index_count
 	$(BINARY) phylo \
-		--output obikmer_dist/count/relfreq_euclidean \
+		--output $(RUN)/obikmer_dist/count/relfreq_euclidean \
-		--metric relfreq-euclidean --nj \
+		--distance relfreq-euclidean --csv --nj \
-		global_index_count
+		$(RUN)/global_index_count
 	$(BINARY) phylo \
-		--output obikmer_dist/count/hellinger \
+		--output $(RUN)/obikmer_dist/count/hellinger \
-		--metric hellinger --nj \
+		--distance hellinger --csv --nj \
-		global_index_count
+		$(RUN)/global_index_count
 	$(BINARY) phylo \
-		--output obikmer_dist/count/hellinger_euclidean \
+		--output $(RUN)/obikmer_dist/count/hellinger_euclidean \
-		--metric hellinger-euclidean --nj \
+		--distance hellinger-euclidean --csv --nj \
-		global_index_count
+		$(RUN)/global_index_count
 
 obikmer_dist_count: $(OBIKMER_COUNT_DIST)
 
@@ -163,19 +176,19 @@ obikmer_dist: obikmer_dist_presence obikmer_dist_count
 # ── distance comparison ───────────────────────────────────────────────────────
 
 $(DIST_COMPARISON): $(REF_DIST_CSVS) $(OBIKMER_PRESENCE_DIST) $(OBIKMER_COUNT_DIST) compare_all_dist.py
-	$(VENV_PY) compare_all_dist.py --out $(DIST_COMPARISON)
+	$(VENV_PY) compare_all_dist.py --run-dir $(RUN) --out $(DIST_COMPARISON)
 
 dist_comparison: $(DIST_COMPARISON)
 
 # ── per-specimen indexing ─────────────────────────────────────────────────────
 # Prerequisites (reads → index.done + .stats) are in deps.mk.
 
-specimen_index_presence/%/index.done \
+$(RUN)/specimen_index_presence/%/index.done \
-stats/indexing_presence/%.stats &: $(BINARY)
+$(RUN)/stats/indexing_presence/%.stats &: $(BINARY)
 	bash index_one_presence.sh $*
 
-specimen_index_count/%/index.done \
+$(RUN)/specimen_index_count/%/index.done \
-stats/indexing_count/%.stats &: $(BINARY)
+$(RUN)/stats/indexing_count/%.stats &: $(BINARY)
 	bash index_one_count.sh $*
 
 index_presence: $(PRESENCE_DONE)
@@ -191,22 +204,22 @@ aggregate_index_count: $(COUNT_STATS)
 
 # ── global merge ──────────────────────────────────────────────────────────────
 
-global_index_presence/index.done: $(PRESENCE_DONE) $(BINARY)
+$(RUN)/global_index_presence/index.done: $(PRESENCE_DONE) $(BINARY)
 	bash merge_presence.sh
 
-global_index_count/index.done: $(COUNT_DONE) $(BINARY)
+$(RUN)/global_index_count/index.done: $(COUNT_DONE) $(BINARY)
 	bash merge_count.sh
 
-merge_presence: global_index_presence/index.done
+merge_presence: $(RUN)/global_index_presence/index.done
-merge_count:    global_index_count/index.done
+merge_count:    $(RUN)/global_index_count/index.done
 
 # ── per-specimen verification ─────────────────────────────────────────────────
 # Prerequisites (index.done + npz → .stats) are in deps.mk.
 
-stats/verify_presence/%.stats:
+$(RUN)/stats/verify_presence/%.stats:
 	bash verify_one_presence.sh $*
 
-stats/verify_count/%.stats:
+$(RUN)/stats/verify_count/%.stats:
 	bash verify_one_count.sh $*
 
 verify_presence: $(VERIFY_PRESENCE_STATS)
@@ -223,12 +236,12 @@ aggregate_verify_count: $(VERIFY_COUNT_STATS)
 # ── species-specific indexes ──────────────────────────────────────────────────
 # Prerequisites (global index → specific index) are in deps.mk.
 
-specific_index_presence/%/index.done \
+$(RUN)/specific_index_presence/%/index.done \
-stats/specific_kmer_presence/%.stats &: $(BINARY)
+$(RUN)/stats/specific_kmer_presence/%.stats &: $(BINARY)
 	bash filter_one_presence.sh $*
 
-specific_index_count/%/index.done \
+$(RUN)/specific_index_count/%/index.done \
-stats/specific_kmer_count/%.stats &: $(BINARY)
+$(RUN)/stats/specific_kmer_count/%.stats &: $(BINARY)
 	bash filter_one_count.sh $*
 
 filter_presence: $(SPECIFIC_PRESENCE_DONE)
@@ -242,42 +255,68 @@ aggregate_filter_count: $(SPECIFIC_COUNT_STATS)
 
 # ── merged index verification ─────────────────────────────────────────────────
 
-stats/verify_merge_presence/current.csv: $(REF_NPZS) global_index_presence/index.done
+$(RUN)/stats/verify_merge_presence/current.csv: $(REF_NPZS) $(RUN)/global_index_presence/index.done
 	bash verify_merge_presence.sh
 
-stats/verify_merge_count/current.csv: $(REF_NPZS) global_index_count/index.done
+$(RUN)/stats/verify_merge_count/current.csv: $(REF_NPZS) $(RUN)/global_index_count/index.done
 	bash verify_merge_count.sh
 
-# ── sparse presence index (query benchmark) ─────────────────────────────────────
+# ── dense variants (query benchmark) ────────────────────────────────────────────
+# `merge` packs sparse by default (2026-08-28) — global_index_presence/
+# global_index_count *are* the sparse variants already; the dense ones are
+# built explicitly here, from a hard-link-based copy (see
+# copy_index_hardlink.sh) rather than a full `cp -r`.
 
-global_index_presence_sparse/index.done: global_index_presence/index.done $(BINARY)
+$(RUN)/global_index_presence_dense/index.done: $(RUN)/global_index_presence/index.done $(BINARY)
-	bash pack_sparse.sh
+	bash pack_dense.sh presence
 
-pack_sparse: global_index_presence_sparse/index.done
+# Rebuilt from the per-specimen count sources directly (via `merge --dense`),
+# not repacked from global_index_count — see pack_dense.sh's own comment.
+$(RUN)/global_index_count_dense/index.done: $(COUNT_DONE) $(BINARY)
+	bash pack_dense.sh count
 
-# ── query: dense vs sparse ───────────────────────────────────────────────────────
+pack_dense_presence: $(RUN)/global_index_presence_dense/index.done
+pack_dense_count:    $(RUN)/global_index_count_dense/index.done
+
+# ── query: dense vs sparse, presence and count ──────────────────────────────────
 # Prerequisites (reads + index → output + .stats) are in deps.mk.
 
-query_dense/%.fasta.gz \
+$(RUN)/query_presence_dense/%.fasta.gz \
-stats/query_dense/%.stats &: $(BINARY)
+$(RUN)/stats/query_presence_dense/%.stats &: $(BINARY) $(RUN)/global_index_presence_dense/index.done
-	bash query_one.sh dense $*
+	bash query_one.sh presence dense $*
 
-query_sparse/%.fasta.gz \
+$(RUN)/query_presence_sparse/%.fasta.gz \
-stats/query_sparse/%.stats &: $(BINARY)
+$(RUN)/stats/query_presence_sparse/%.stats &: $(BINARY) $(RUN)/global_index_presence/index.done
-	bash query_one.sh sparse $*
+	bash query_one.sh presence sparse $*
 
-query_dense:  $(QUERY_DENSE_DONE)
+$(RUN)/query_count_dense/%.fasta.gz \
-query_sparse: $(QUERY_SPARSE_DONE)
+$(RUN)/stats/query_count_dense/%.stats &: $(BINARY) $(RUN)/global_index_count_dense/index.done
+	bash query_one.sh count dense $*
 
-aggregate_query_dense: $(QUERY_DENSE_STATS)
+$(RUN)/query_count_sparse/%.fasta.gz \
-	bash aggregate_stats.sh query_dense
+$(RUN)/stats/query_count_sparse/%.stats &: $(BINARY) $(RUN)/global_index_count/index.done
+	bash query_one.sh count sparse $*
 
-aggregate_query_sparse: $(QUERY_SPARSE_STATS)
+query_presence_dense:  $(QUERY_PRESENCE_DENSE_DONE)
-	bash aggregate_stats.sh query_sparse
+query_presence_sparse: $(QUERY_PRESENCE_SPARSE_DONE)
+query_count_dense:     $(QUERY_COUNT_DENSE_DONE)
+query_count_sparse:    $(QUERY_COUNT_SPARSE_DONE)
+
+aggregate_query_presence_dense: $(QUERY_PRESENCE_DENSE_STATS)
+	bash aggregate_stats.sh query_presence_dense
+
+aggregate_query_presence_sparse: $(QUERY_PRESENCE_SPARSE_STATS)
+	bash aggregate_stats.sh query_presence_sparse
+
+aggregate_query_count_dense: $(QUERY_COUNT_DENSE_STATS)
+	bash aggregate_stats.sh query_count_dense
+
+aggregate_query_count_sparse: $(QUERY_COUNT_SPARSE_STATS)
+	bash aggregate_stats.sh query_count_sparse
 
 # ── query: dense/sparse regression ──────────────────────────────────────────────
 
-stats/verify_query/%.stats:
+$(RUN)/stats/verify_query/%.stats:
 	bash verify_query_one.sh $*
 
 verify_query: $(VERIFY_QUERY_STATS)

benchmark/README.md

@@ -8,98 +8,112 @@ gmake simulate     # simulation only
 gmake reference    # reference kmer sets only
 ```
 
+All generated and downloaded artifacts live under `run/` (see
+[Directory layout](#directory-layout)), so the whole tree is gitignored with
+a single `benchmark/run/` entry.
+
 ## Pipeline overview
 
 ```mermaid
 flowchart TD
-    GENOMES["genomes/*.fna.gz"]
+    GENOMES["run/genomes/*.fna.gz"]
     BIN["obikmer binary"]
 
     GENOMES --> simulate
-    simulate --> simdata[("simulated_data/")]
+    simulate --> simdata[("run/simulated_data/")]
 
     simdata --> reference
-    reference --> refnpz[("reference_index/*.npz")]
+    reference --> refnpz[("run/reference_index/*.npz")]
 
     subgraph presence ["Presence track"]
         simdata  --> index_presence
         BIN      --> index_presence
-        index_presence --> pres_done[("specimen_index_presence/")]
+        index_presence --> pres_done[("run/specimen_index_presence/")]
-        index_presence --> pres_istats[("stats/indexing_presence/")]
+        index_presence --> pres_istats[("run/stats/indexing_presence/")]
         pres_istats --> aggregate_index_presence
 
         pres_done --> merge_presence
         BIN       --> merge_presence
-        merge_presence --> gpres[("global_index_presence/")]
+        merge_presence --> gpres[("run/global_index_presence/")]
 
         refnpz    --> verify_presence
         pres_done --> verify_presence
-        verify_presence --> vpres_stats[("stats/verify_presence/")]
+        verify_presence --> vpres_stats[("run/stats/verify_presence/")]
         vpres_stats --> aggregate_verify_presence
 
         gpres --> filter_presence
         BIN   --> filter_presence
-        filter_presence --> spec_pres[("specific_index_presence/")]
+        filter_presence --> spec_pres[("run/specific_index_presence/")]
-        filter_presence --> spec_pres_stats[("stats/specific_kmer_presence/")]
+        filter_presence --> spec_pres_stats[("run/stats/specific_kmer_presence/")]
         spec_pres_stats --> aggregate_filter_presence
 
         refnpz --> verify_merge_presence
         gpres  --> verify_merge_presence
-        verify_merge_presence --> vmp[("stats/verify_merge_presence/")]
+        verify_merge_presence --> vmp[("run/stats/verify_merge_presence/")]
     end
 
     subgraph count ["Count track"]
         simdata --> index_count
         BIN     --> index_count
-        index_count --> count_done[("specimen_index_count/")]
+        index_count --> count_done[("run/specimen_index_count/")]
-        index_count --> count_istats[("stats/indexing_count/")]
+        index_count --> count_istats[("run/stats/indexing_count/")]
         count_istats --> aggregate_index_count
 
         count_done --> merge_count
         BIN        --> merge_count
-        merge_count --> gcount[("global_index_count/")]
+        merge_count --> gcount[("run/global_index_count/")]
 
         refnpz     --> verify_count
         count_done --> verify_count
-        verify_count --> vcount_stats[("stats/verify_count/")]
+        verify_count --> vcount_stats[("run/stats/verify_count/")]
         vcount_stats --> aggregate_verify_count
 
         gcount --> filter_count
         BIN    --> filter_count
-        filter_count --> spec_count[("specific_index_count/")]
+        filter_count --> spec_count[("run/specific_index_count/")]
-        filter_count --> spec_count_stats[("stats/specific_kmer_count/")]
+        filter_count --> spec_count_stats[("run/stats/specific_kmer_count/")]
         spec_count_stats --> aggregate_filter_count
 
         refnpz --> verify_merge_count
         gcount --> verify_merge_count
-        verify_merge_count --> vmc[("stats/verify_merge_count/")]
+        verify_merge_count --> vmc[("run/stats/verify_merge_count/")]
     end
 
     subgraph query ["Query track (2 specimens: E. coli + archaeon)"]
         GENOMES --> simulate_query
-        simulate_query --> qdata[("query_data/")]
+        simulate_query --> qdata[("run/query_data/")]
 
-        gpres --> pack_sparse
+        gpres --> pack_dense_presence
-        BIN   --> pack_sparse
+        BIN   --> pack_dense_presence
-        pack_sparse --> gsparse[("global_index_presence_sparse/")]
+        pack_dense_presence --> gpresd[("run/global_index_presence_dense/")]
 
-        qdata  --> query_dense
+        count_done --> pack_dense_count
-        gpres  --> query_dense
+        BIN        --> pack_dense_count
-        BIN    --> query_dense
+        pack_dense_count --> gcountd[("run/global_index_count_dense/")]
-        query_dense --> qd[("query_dense/")]
-        query_dense --> qd_stats[("stats/query_dense/")]
-        qd_stats --> aggregate_query_dense
 
-        qdata   --> query_sparse
+        qdata  --> query_presence_dense
-        gsparse --> query_sparse
+        gpresd --> query_presence_dense
-        BIN     --> query_sparse
+        query_presence_dense --> qpd[("run/query_presence_dense/")]
-        query_sparse --> qs[("query_sparse/")]
+        qpd --> aggregate_query_presence_dense
-        query_sparse --> qs_stats[("stats/query_sparse/")]
-        qs_stats --> aggregate_query_sparse
 
-        qd --> verify_query
+        qdata  --> query_presence_sparse
-        qs --> verify_query
+        gpres  --> query_presence_sparse
-        verify_query --> vq_stats[("stats/verify_query/")]
+        query_presence_sparse --> qps[("run/query_presence_sparse/")]
+        qps --> aggregate_query_presence_sparse
+
+        qdata   --> query_count_dense
+        gcountd --> query_count_dense
+        query_count_dense --> qcd[("run/query_count_dense/")]
+        qcd --> aggregate_query_count_dense
+
+        qdata  --> query_count_sparse
+        gcount --> query_count_sparse
+        query_count_sparse --> qcs[("run/query_count_sparse/")]
+        qcs --> aggregate_query_count_sparse
+
+        qpd --> verify_query
+        qps --> verify_query
+        verify_query --> vq_stats[("run/stats/verify_query/")]
         vq_stats --> aggregate_verify_query
     end
 
@@ -107,8 +121,10 @@ flowchart TD
     aggregate_verify_count            --> all
     vmp                               --> all
     vmc                               --> all
-    aggregate_query_dense      --> all
+    aggregate_query_presence_dense    --> all
-    aggregate_query_sparse     --> all
+    aggregate_query_presence_sparse   --> all
+    aggregate_query_count_dense       --> all
+    aggregate_query_count_sparse      --> all
     aggregate_verify_query            --> all
     all -. "$(MAKE) re-eval" .-> aggregate_filter_presence
     all -. "$(MAKE) re-eval" .-> aggregate_filter_count
@@ -137,34 +153,45 @@ flowchart TD
 | `verify_merge_presence` | `verify_merge_presence.sh` | Verify global presence index against all reference sets |
 | `verify_merge_count` | `verify_merge_count.sh` | Verify global count index against all reference sets |
 | `simulate_query` | `simulate_query_one.sh` | Simulate a fixed-size (100k pairs) read set per query specimen |
-| `pack_sparse` | `pack_sparse.sh` | Build `global_index_presence_sparse/` from `global_index_presence/` |
+| `pack_dense_presence` | `pack_dense.sh presence` | Build `global_index_presence_dense/` from `global_index_presence/` |
-| `query_dense` | `query_one.sh dense` | Query each query specimen's reads against the dense global index |
+| `pack_dense_count` | `pack_dense.sh count` | Build `global_index_count_dense/` from the per-specimen count sources |
-| `query_sparse` | `query_one.sh sparse` | Query each query specimen's reads against the sparse global index |
+| `query_presence_dense` | `query_one.sh presence dense` | Query against the dense presence global index |
-| `aggregate_query_dense` | `aggregate_stats.sh` | Aggregate dense query wall/RSS stats |
+| `query_presence_sparse` | `query_one.sh presence sparse` | Query against the sparse (as-merged) presence global index |
-| `aggregate_query_sparse` | `aggregate_stats.sh` | Aggregate sparse query wall/RSS stats |
+| `query_count_dense` | `query_one.sh count dense` | Query against the dense count global index |
-| `verify_query` | `verify_query_one.sh` | Diff dense vs sparse query output per specimen (regression check) |
+| `query_count_sparse` | `query_one.sh count sparse` | Query against the sparse (as-merged) count global index |
+| `aggregate_query_presence_dense` | `aggregate_stats.sh` | Aggregate dense presence-query wall/RSS stats |
+| `aggregate_query_presence_sparse` | `aggregate_stats.sh` | Aggregate sparse presence-query wall/RSS stats |
+| `aggregate_query_count_dense` | `aggregate_stats.sh` | Aggregate dense count-query wall/RSS stats |
+| `aggregate_query_count_sparse` | `aggregate_stats.sh` | Aggregate sparse count-query wall/RSS stats |
+| `verify_query` | `verify_query_one.sh` | Diff dense vs sparse presence-query output per specimen (regression check) |
 | `aggregate_verify_query` | `aggregate_stats.sh` | Aggregate dense/sparse query regression stats |
 
 ## Directory layout
 
 ```
 benchmark/
-├── genomes/                        # input reference genomes (.fna.gz)
+└── run/                                 # everything generated/downloaded — gitignored as a whole
-├── simulated_data/                 # generated by simulate
+    ├── genomes/                         # input reference genomes (.fna.gz), downloaded by downloads.sh
-│   └── <species>/<specimen>/
+    ├── simulated_data/                  # generated by simulate
-├── query_data/                     # generated by simulate_query (2 specimens, fixed 100k pairs)
+    │   └── <species>/<specimen>/
-│   └── <species>/<specimen>/
+    ├── query_data/                      # generated by simulate_query (2 specimens, fixed 100k pairs)
-├── reference_index/                # reference kmer sets (.npz)
+    │   └── <species>/<specimen>/
-├── specimen_index_presence/        # per-specimen presence indexes
+    ├── reference_index/                 # reference kmer sets (.npz)
-├── specimen_index_count/           # per-specimen count indexes
+    ├── reference_dist/                  # reference pairwise distance matrices
-├── global_index_presence/          # merged global presence index (dense-packed)
+    ├── obikmer_dist/                    # obikmer phylo distance matrices (presence/, count/)
-├── global_index_presence_sparse/   # global presence index, sparse-packed (query benchmark)
+    ├── specimen_index_presence/         # per-specimen presence indexes
-├── global_index_count/             # merged global count index
+    ├── specimen_index_count/            # per-specimen count indexes
-├── specific_index_presence/        # species-specific presence indexes
+    ├── global_index_presence/           # merged global presence index (sparse, as merged)
-├── specific_index_count/           # species-specific count indexes
+    ├── global_index_presence_dense/     # global presence index, dense-repacked (query benchmark)
-├── query_dense/                    # query output against global_index_presence
+    ├── global_index_count/              # merged global count index (sparse, as merged)
-├── query_sparse/                   # query output against global_index_presence_sparse
+    ├── global_index_count_dense/        # global count index, dense-repacked (query benchmark)
-└── stats/                          # all benchmark statistics
+    ├── specific_index_presence/         # species-specific presence indexes
+    ├── specific_index_count/            # species-specific count indexes
+    ├── query_presence_dense/            # query output against global_index_presence_dense
+    ├── query_presence_sparse/           # query output against global_index_presence
+    ├── query_count_dense/               # query output against global_index_count_dense
+    ├── query_count_sparse/              # query output against global_index_count
+    └── stats/                           # all benchmark statistics
         ├── indexing_presence/
         ├── indexing_count/
         ├── verify_presence/
@@ -173,8 +200,12 @@ benchmark/
         ├── specific_kmer_count/
         ├── verify_merge_presence/
         ├── verify_merge_count/
-    ├── pack_sparse/
+        ├── pack_dense_presence/
-    ├── query_dense/
+        ├── pack_dense_count/
-    ├── query_sparse/
+        ├── query_presence_dense/
+        ├── query_presence_sparse/
+        ├── query_count_dense/
+        ├── query_count_sparse/
+        ├── dist_comparison/
         └── verify_query/
 ```

benchmark/aggregate_stats.sh

@@ -1,6 +1,8 @@
 #!/usr/bin/env bash
 # Usage: aggregate_stats.sh TYPE
-# TYPE = indexing_presence | indexing_count | verify_presence | verify_count
+# TYPE = indexing_presence | indexing_count | verify_presence | verify_count |
+#        query_presence_dense | query_presence_sparse |
+#        query_count_dense | query_count_sparse
 #
 # Reads all stats/TYPE/*.stats files (one CSV data row each, no header).
 # Creates a new stats/TYPE/run_NNN.csv only if any .stats file is newer than
@@ -9,7 +11,7 @@ set -euo pipefail
 
 TYPE="$1"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-STATS_DIR="${SCRIPT_DIR}/stats/${TYPE}"
+STATS_DIR="${SCRIPT_DIR}/run/stats/${TYPE}"
 
 case "${TYPE}" in
     indexing_presence|indexing_count)
@@ -24,7 +26,7 @@ case "${TYPE}" in
     specific_kmer_presence|specific_kmer_count)
         HEADER="run,species,rebuild_wall_s,rebuild_rss_b,pack_wall_s,pack_rss_b,filter_total_wall_s,filter_total_rss_b,select_wall_s,select_rss_b,select_total_wall_s,select_total_rss_b"
         ;;
-    query_dense|query_sparse)
+    query_presence_dense|query_presence_sparse|query_count_dense|query_count_sparse)
         HEADER="run,species,strain,query_wall_s,query_rss_b,total_wall_s,total_rss_b"
         ;;
     verify_query)

benchmark/build_reference.sh

@@ -2,8 +2,9 @@
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-SIMDATA_DIR="${SCRIPT_DIR}/simulated_data"
+RUN_DIR="${SCRIPT_DIR}/run"
-REF_DIR="${SCRIPT_DIR}/reference_index"
+SIMDATA_DIR="${RUN_DIR}/simulated_data"
+REF_DIR="${RUN_DIR}/reference_index"
 PYTHON="${SCRIPT_DIR}/../.venv/bin/python3"
 BUILD_PY="${SCRIPT_DIR}/build_reference.py"
 

benchmark/build_reference_dist.py

@@ -176,10 +176,10 @@ def write_csv(path: Path, labels: list[str], mat: np.ndarray, fmt: str) -> None:
 def main() -> None:
     ap = argparse.ArgumentParser(description=__doc__,
                                  formatter_class=argparse.RawDescriptionHelpFormatter)
-    ap.add_argument('--ref-dir', default='reference_index',
+    ap.add_argument('--ref-dir', default='run/reference_index',
-                    help='Directory with per-specimen .npz files (default: reference_index)')
+                    help='Directory with per-specimen .npz files (default: run/reference_index)')
-    ap.add_argument('--out-dir', default='reference_dist',
+    ap.add_argument('--out-dir', default='run/reference_dist',
-                    help='Output directory for CSV files (default: reference_dist)')
+                    help='Output directory for CSV files (default: run/reference_dist)')
     args = ap.parse_args()
 
     ref_dir = Path(args.ref_dir)

benchmark/compare_all_dist.py

@@ -150,17 +150,20 @@ def main() -> None:
                                  formatter_class=argparse.RawDescriptionHelpFormatter)
     ap.add_argument('--tol', type=float, default=1e-4,
                     help='Max abs diff threshold for PASS/FAIL (default 1e-4)')
-    ap.add_argument('--out', default='stats/dist_comparison/summary.csv',
+    ap.add_argument('--run-dir', default='run',
+                    help='Root directory holding reference_dist/ and obikmer_dist/ (default: run)')
+    ap.add_argument('--out', default='run/stats/dist_comparison/summary.csv',
                     help='Output summary CSV path')
     args = ap.parse_args()
 
     out_path = Path(args.out)
     out_path.parent.mkdir(parents=True, exist_ok=True)
 
+    run_dir = Path(args.run_dir)
     print(f'Comparing {len(COMPARISONS)} matrix pairs…', file=sys.stderr)
     rows = []
     for label, ref, obi in COMPARISONS:
-        rows.append(compare(label, Path(ref), Path(obi), tol=args.tol))
+        rows.append(compare(label, run_dir / ref, run_dir / obi, tol=args.tol))
 
     fields = ['comparison', 'max_abs', 'mean_abs', 'rmse', 'n_pairs', 'status']
     with out_path.open('w', newline='') as fh:

benchmark/deps.mk

@@ -2,212 +2,212 @@ SPECIMENS := Escherichia_coli--K-12_MG1655 Escherichia_coli--EDL933 Salmonella_e
 SPECIES   := Escherichia_coli Salmonella_enterica Bacillus_subtilis Shouchella_clausii Klebsiella_pneumoniae Opitutus_terrae Saccharolobus_islandicus Acidobacterium_capsulatum Proteus_mirabilis Wolbachia_endosymbiont Yersinia_ruckeri Candidozyma_auris
 
 # Escherichia_coli--K-12_MG1655
-simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz: genomes/GCF_000005845.2_ASM584v2_genomic.fna.gz
+run/simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz: run/genomes/GCF_000005845.2_ASM584v2_genomic.fna.gz
-reference_index/Escherichia_coli--K-12_MG1655.npz: simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz
+run/reference_index/Escherichia_coli--K-12_MG1655.npz: run/simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz
-specimen_index_presence/Escherichia_coli--K-12_MG1655/index.done stats/indexing_presence/Escherichia_coli--K-12_MG1655.stats: simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz
+run/specimen_index_presence/Escherichia_coli--K-12_MG1655/index.done run/stats/indexing_presence/Escherichia_coli--K-12_MG1655.stats: run/simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz
-specimen_index_count/Escherichia_coli--K-12_MG1655/index.done stats/indexing_count/Escherichia_coli--K-12_MG1655.stats: simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz
+run/specimen_index_count/Escherichia_coli--K-12_MG1655/index.done run/stats/indexing_count/Escherichia_coli--K-12_MG1655.stats: run/simulated_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz
-stats/verify_presence/Escherichia_coli--K-12_MG1655.stats: reference_index/Escherichia_coli--K-12_MG1655.npz specimen_index_presence/Escherichia_coli--K-12_MG1655/index.done
+run/stats/verify_presence/Escherichia_coli--K-12_MG1655.stats: run/reference_index/Escherichia_coli--K-12_MG1655.npz run/specimen_index_presence/Escherichia_coli--K-12_MG1655/index.done
-stats/verify_count/Escherichia_coli--K-12_MG1655.stats: reference_index/Escherichia_coli--K-12_MG1655.npz specimen_index_count/Escherichia_coli--K-12_MG1655/index.done
+run/stats/verify_count/Escherichia_coli--K-12_MG1655.stats: run/reference_index/Escherichia_coli--K-12_MG1655.npz run/specimen_index_count/Escherichia_coli--K-12_MG1655/index.done
 
 # Escherichia_coli--EDL933
-simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz: genomes/GCF_000006665.1_ASM666v1_genomic.fna.gz
+run/simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz: run/genomes/GCF_000006665.1_ASM666v1_genomic.fna.gz
-reference_index/Escherichia_coli--EDL933.npz: simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz
+run/reference_index/Escherichia_coli--EDL933.npz: run/simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz
-specimen_index_presence/Escherichia_coli--EDL933/index.done stats/indexing_presence/Escherichia_coli--EDL933.stats: simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz
+run/specimen_index_presence/Escherichia_coli--EDL933/index.done run/stats/indexing_presence/Escherichia_coli--EDL933.stats: run/simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz
-specimen_index_count/Escherichia_coli--EDL933/index.done stats/indexing_count/Escherichia_coli--EDL933.stats: simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz
+run/specimen_index_count/Escherichia_coli--EDL933/index.done run/stats/indexing_count/Escherichia_coli--EDL933.stats: run/simulated_data/Escherichia_coli/EDL933/reads_R1.fastq.gz
-stats/verify_presence/Escherichia_coli--EDL933.stats: reference_index/Escherichia_coli--EDL933.npz specimen_index_presence/Escherichia_coli--EDL933/index.done
+run/stats/verify_presence/Escherichia_coli--EDL933.stats: run/reference_index/Escherichia_coli--EDL933.npz run/specimen_index_presence/Escherichia_coli--EDL933/index.done
-stats/verify_count/Escherichia_coli--EDL933.stats: reference_index/Escherichia_coli--EDL933.npz specimen_index_count/Escherichia_coli--EDL933/index.done
+run/stats/verify_count/Escherichia_coli--EDL933.stats: run/reference_index/Escherichia_coli--EDL933.npz run/specimen_index_count/Escherichia_coli--EDL933/index.done
 
 # Salmonella_enterica--LT2
-simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz: genomes/GCF_000006945.2_ASM694v2_genomic.fna.gz
+run/simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz: run/genomes/GCF_000006945.2_ASM694v2_genomic.fna.gz
-reference_index/Salmonella_enterica--LT2.npz: simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz
+run/reference_index/Salmonella_enterica--LT2.npz: run/simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz
-specimen_index_presence/Salmonella_enterica--LT2/index.done stats/indexing_presence/Salmonella_enterica--LT2.stats: simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz
+run/specimen_index_presence/Salmonella_enterica--LT2/index.done run/stats/indexing_presence/Salmonella_enterica--LT2.stats: run/simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz
-specimen_index_count/Salmonella_enterica--LT2/index.done stats/indexing_count/Salmonella_enterica--LT2.stats: simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz
+run/specimen_index_count/Salmonella_enterica--LT2/index.done run/stats/indexing_count/Salmonella_enterica--LT2.stats: run/simulated_data/Salmonella_enterica/LT2/reads_R1.fastq.gz
-stats/verify_presence/Salmonella_enterica--LT2.stats: reference_index/Salmonella_enterica--LT2.npz specimen_index_presence/Salmonella_enterica--LT2/index.done
+run/stats/verify_presence/Salmonella_enterica--LT2.stats: run/reference_index/Salmonella_enterica--LT2.npz run/specimen_index_presence/Salmonella_enterica--LT2/index.done
-stats/verify_count/Salmonella_enterica--LT2.stats: reference_index/Salmonella_enterica--LT2.npz specimen_index_count/Salmonella_enterica--LT2/index.done
+run/stats/verify_count/Salmonella_enterica--LT2.stats: run/reference_index/Salmonella_enterica--LT2.npz run/specimen_index_count/Salmonella_enterica--LT2/index.done
 
 # Escherichia_coli--CFT073
-simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz: genomes/GCF_000007445.1_ASM744v1_genomic.fna.gz
+run/simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz: run/genomes/GCF_000007445.1_ASM744v1_genomic.fna.gz
-reference_index/Escherichia_coli--CFT073.npz: simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz
+run/reference_index/Escherichia_coli--CFT073.npz: run/simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz
-specimen_index_presence/Escherichia_coli--CFT073/index.done stats/indexing_presence/Escherichia_coli--CFT073.stats: simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz
+run/specimen_index_presence/Escherichia_coli--CFT073/index.done run/stats/indexing_presence/Escherichia_coli--CFT073.stats: run/simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz
-specimen_index_count/Escherichia_coli--CFT073/index.done stats/indexing_count/Escherichia_coli--CFT073.stats: simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz
+run/specimen_index_count/Escherichia_coli--CFT073/index.done run/stats/indexing_count/Escherichia_coli--CFT073.stats: run/simulated_data/Escherichia_coli/CFT073/reads_R1.fastq.gz
-stats/verify_presence/Escherichia_coli--CFT073.stats: reference_index/Escherichia_coli--CFT073.npz specimen_index_presence/Escherichia_coli--CFT073/index.done
+run/stats/verify_presence/Escherichia_coli--CFT073.stats: run/reference_index/Escherichia_coli--CFT073.npz run/specimen_index_presence/Escherichia_coli--CFT073/index.done
-stats/verify_count/Escherichia_coli--CFT073.stats: reference_index/Escherichia_coli--CFT073.npz specimen_index_count/Escherichia_coli--CFT073/index.done
+run/stats/verify_count/Escherichia_coli--CFT073.stats: run/reference_index/Escherichia_coli--CFT073.npz run/specimen_index_count/Escherichia_coli--CFT073/index.done
 
 # Bacillus_subtilis--168
-simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz: genomes/GCF_000009045.1_ASM904v1_genomic.fna.gz
+run/simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz: run/genomes/GCF_000009045.1_ASM904v1_genomic.fna.gz
-reference_index/Bacillus_subtilis--168.npz: simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz
+run/reference_index/Bacillus_subtilis--168.npz: run/simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz
-specimen_index_presence/Bacillus_subtilis--168/index.done stats/indexing_presence/Bacillus_subtilis--168.stats: simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz
+run/specimen_index_presence/Bacillus_subtilis--168/index.done run/stats/indexing_presence/Bacillus_subtilis--168.stats: run/simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz
-specimen_index_count/Bacillus_subtilis--168/index.done stats/indexing_count/Bacillus_subtilis--168.stats: simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz
+run/specimen_index_count/Bacillus_subtilis--168/index.done run/stats/indexing_count/Bacillus_subtilis--168.stats: run/simulated_data/Bacillus_subtilis/168/reads_R1.fastq.gz
-stats/verify_presence/Bacillus_subtilis--168.stats: reference_index/Bacillus_subtilis--168.npz specimen_index_presence/Bacillus_subtilis--168/index.done
+run/stats/verify_presence/Bacillus_subtilis--168.stats: run/reference_index/Bacillus_subtilis--168.npz run/specimen_index_presence/Bacillus_subtilis--168/index.done
-stats/verify_count/Bacillus_subtilis--168.stats: reference_index/Bacillus_subtilis--168.npz specimen_index_count/Bacillus_subtilis--168/index.done
+run/stats/verify_count/Bacillus_subtilis--168.stats: run/reference_index/Bacillus_subtilis--168.npz run/specimen_index_count/Bacillus_subtilis--168/index.done
 
 # Salmonella_enterica--P125109
-simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz: genomes/GCF_000009505.1_ASM950v1_genomic.fna.gz
+run/simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz: run/genomes/GCF_000009505.1_ASM950v1_genomic.fna.gz
-reference_index/Salmonella_enterica--P125109.npz: simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz
+run/reference_index/Salmonella_enterica--P125109.npz: run/simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz
-specimen_index_presence/Salmonella_enterica--P125109/index.done stats/indexing_presence/Salmonella_enterica--P125109.stats: simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz
+run/specimen_index_presence/Salmonella_enterica--P125109/index.done run/stats/indexing_presence/Salmonella_enterica--P125109.stats: run/simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz
-specimen_index_count/Salmonella_enterica--P125109/index.done stats/indexing_count/Salmonella_enterica--P125109.stats: simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz
+run/specimen_index_count/Salmonella_enterica--P125109/index.done run/stats/indexing_count/Salmonella_enterica--P125109.stats: run/simulated_data/Salmonella_enterica/P125109/reads_R1.fastq.gz
-stats/verify_presence/Salmonella_enterica--P125109.stats: reference_index/Salmonella_enterica--P125109.npz specimen_index_presence/Salmonella_enterica--P125109/index.done
+run/stats/verify_presence/Salmonella_enterica--P125109.stats: run/reference_index/Salmonella_enterica--P125109.npz run/specimen_index_presence/Salmonella_enterica--P125109/index.done
-stats/verify_count/Salmonella_enterica--P125109.stats: reference_index/Salmonella_enterica--P125109.npz specimen_index_count/Salmonella_enterica--P125109/index.done
+run/stats/verify_count/Salmonella_enterica--P125109.stats: run/reference_index/Salmonella_enterica--P125109.npz run/specimen_index_count/Salmonella_enterica--P125109/index.done
 
 # Shouchella_clausii--KSM-K16
-simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz: genomes/GCF_000009825.1_ASM982v1_genomic.fna.gz
+run/simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz: run/genomes/GCF_000009825.1_ASM982v1_genomic.fna.gz
-reference_index/Shouchella_clausii--KSM-K16.npz: simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz
+run/reference_index/Shouchella_clausii--KSM-K16.npz: run/simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz
-specimen_index_presence/Shouchella_clausii--KSM-K16/index.done stats/indexing_presence/Shouchella_clausii--KSM-K16.stats: simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz
+run/specimen_index_presence/Shouchella_clausii--KSM-K16/index.done run/stats/indexing_presence/Shouchella_clausii--KSM-K16.stats: run/simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz
-specimen_index_count/Shouchella_clausii--KSM-K16/index.done stats/indexing_count/Shouchella_clausii--KSM-K16.stats: simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz
+run/specimen_index_count/Shouchella_clausii--KSM-K16/index.done run/stats/indexing_count/Shouchella_clausii--KSM-K16.stats: run/simulated_data/Shouchella_clausii/KSM-K16/reads_R1.fastq.gz
-stats/verify_presence/Shouchella_clausii--KSM-K16.stats: reference_index/Shouchella_clausii--KSM-K16.npz specimen_index_presence/Shouchella_clausii--KSM-K16/index.done
+run/stats/verify_presence/Shouchella_clausii--KSM-K16.stats: run/reference_index/Shouchella_clausii--KSM-K16.npz run/specimen_index_presence/Shouchella_clausii--KSM-K16/index.done
-stats/verify_count/Shouchella_clausii--KSM-K16.stats: reference_index/Shouchella_clausii--KSM-K16.npz specimen_index_count/Shouchella_clausii--KSM-K16/index.done
+run/stats/verify_count/Shouchella_clausii--KSM-K16.stats: run/reference_index/Shouchella_clausii--KSM-K16.npz run/specimen_index_count/Shouchella_clausii--KSM-K16/index.done
 
 # Escherichia_coli--K-12_W3110
-simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz: genomes/GCF_000010245.2_ASM1024v1_genomic.fna.gz
+run/simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz: run/genomes/GCF_000010245.2_ASM1024v1_genomic.fna.gz
-reference_index/Escherichia_coli--K-12_W3110.npz: simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz
+run/reference_index/Escherichia_coli--K-12_W3110.npz: run/simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz
-specimen_index_presence/Escherichia_coli--K-12_W3110/index.done stats/indexing_presence/Escherichia_coli--K-12_W3110.stats: simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz
+run/specimen_index_presence/Escherichia_coli--K-12_W3110/index.done run/stats/indexing_presence/Escherichia_coli--K-12_W3110.stats: run/simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz
-specimen_index_count/Escherichia_coli--K-12_W3110/index.done stats/indexing_count/Escherichia_coli--K-12_W3110.stats: simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz
+run/specimen_index_count/Escherichia_coli--K-12_W3110/index.done run/stats/indexing_count/Escherichia_coli--K-12_W3110.stats: run/simulated_data/Escherichia_coli/K-12_W3110/reads_R1.fastq.gz
-stats/verify_presence/Escherichia_coli--K-12_W3110.stats: reference_index/Escherichia_coli--K-12_W3110.npz specimen_index_presence/Escherichia_coli--K-12_W3110/index.done
+run/stats/verify_presence/Escherichia_coli--K-12_W3110.stats: run/reference_index/Escherichia_coli--K-12_W3110.npz run/specimen_index_presence/Escherichia_coli--K-12_W3110/index.done
-stats/verify_count/Escherichia_coli--K-12_W3110.stats: reference_index/Escherichia_coli--K-12_W3110.npz specimen_index_count/Escherichia_coli--K-12_W3110/index.done
+run/stats/verify_count/Escherichia_coli--K-12_W3110.stats: run/reference_index/Escherichia_coli--K-12_W3110.npz run/specimen_index_count/Escherichia_coli--K-12_W3110/index.done
 
 # Klebsiella_pneumoniae--MGH_78578
-simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz: genomes/GCF_000016305.1_ASM1630v1_genomic.fna.gz
+run/simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz: run/genomes/GCF_000016305.1_ASM1630v1_genomic.fna.gz
-reference_index/Klebsiella_pneumoniae--MGH_78578.npz: simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz
+run/reference_index/Klebsiella_pneumoniae--MGH_78578.npz: run/simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz
-specimen_index_presence/Klebsiella_pneumoniae--MGH_78578/index.done stats/indexing_presence/Klebsiella_pneumoniae--MGH_78578.stats: simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz
+run/specimen_index_presence/Klebsiella_pneumoniae--MGH_78578/index.done run/stats/indexing_presence/Klebsiella_pneumoniae--MGH_78578.stats: run/simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz
-specimen_index_count/Klebsiella_pneumoniae--MGH_78578/index.done stats/indexing_count/Klebsiella_pneumoniae--MGH_78578.stats: simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz
+run/specimen_index_count/Klebsiella_pneumoniae--MGH_78578/index.done run/stats/indexing_count/Klebsiella_pneumoniae--MGH_78578.stats: run/simulated_data/Klebsiella_pneumoniae/MGH_78578/reads_R1.fastq.gz
-stats/verify_presence/Klebsiella_pneumoniae--MGH_78578.stats: reference_index/Klebsiella_pneumoniae--MGH_78578.npz specimen_index_presence/Klebsiella_pneumoniae--MGH_78578/index.done
+run/stats/verify_presence/Klebsiella_pneumoniae--MGH_78578.stats: run/reference_index/Klebsiella_pneumoniae--MGH_78578.npz run/specimen_index_presence/Klebsiella_pneumoniae--MGH_78578/index.done
-stats/verify_count/Klebsiella_pneumoniae--MGH_78578.stats: reference_index/Klebsiella_pneumoniae--MGH_78578.npz specimen_index_count/Klebsiella_pneumoniae--MGH_78578/index.done
+run/stats/verify_count/Klebsiella_pneumoniae--MGH_78578.stats: run/reference_index/Klebsiella_pneumoniae--MGH_78578.npz run/specimen_index_count/Klebsiella_pneumoniae--MGH_78578/index.done
 
 # Opitutus_terrae--PB90-1
-simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz: genomes/GCF_000019965.1_ASM1996v1_genomic.fna.gz
+run/simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz: run/genomes/GCF_000019965.1_ASM1996v1_genomic.fna.gz
-reference_index/Opitutus_terrae--PB90-1.npz: simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz
+run/reference_index/Opitutus_terrae--PB90-1.npz: run/simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz
-specimen_index_presence/Opitutus_terrae--PB90-1/index.done stats/indexing_presence/Opitutus_terrae--PB90-1.stats: simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz
+run/specimen_index_presence/Opitutus_terrae--PB90-1/index.done run/stats/indexing_presence/Opitutus_terrae--PB90-1.stats: run/simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz
-specimen_index_count/Opitutus_terrae--PB90-1/index.done stats/indexing_count/Opitutus_terrae--PB90-1.stats: simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz
+run/specimen_index_count/Opitutus_terrae--PB90-1/index.done run/stats/indexing_count/Opitutus_terrae--PB90-1.stats: run/simulated_data/Opitutus_terrae/PB90-1/reads_R1.fastq.gz
-stats/verify_presence/Opitutus_terrae--PB90-1.stats: reference_index/Opitutus_terrae--PB90-1.npz specimen_index_presence/Opitutus_terrae--PB90-1/index.done
+run/stats/verify_presence/Opitutus_terrae--PB90-1.stats: run/reference_index/Opitutus_terrae--PB90-1.npz run/specimen_index_presence/Opitutus_terrae--PB90-1/index.done
-stats/verify_count/Opitutus_terrae--PB90-1.stats: reference_index/Opitutus_terrae--PB90-1.npz specimen_index_count/Opitutus_terrae--PB90-1/index.done
+run/stats/verify_count/Opitutus_terrae--PB90-1.stats: run/reference_index/Opitutus_terrae--PB90-1.npz run/specimen_index_count/Opitutus_terrae--PB90-1/index.done
 
 # Saccharolobus_islandicus--M.16.4
-simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz: genomes/GCF_000022445.1_ASM2244v1_genomic.fna.gz
+run/simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz: run/genomes/GCF_000022445.1_ASM2244v1_genomic.fna.gz
-reference_index/Saccharolobus_islandicus--M.16.4.npz: simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz
+run/reference_index/Saccharolobus_islandicus--M.16.4.npz: run/simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz
-specimen_index_presence/Saccharolobus_islandicus--M.16.4/index.done stats/indexing_presence/Saccharolobus_islandicus--M.16.4.stats: simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz
+run/specimen_index_presence/Saccharolobus_islandicus--M.16.4/index.done run/stats/indexing_presence/Saccharolobus_islandicus--M.16.4.stats: run/simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz
-specimen_index_count/Saccharolobus_islandicus--M.16.4/index.done stats/indexing_count/Saccharolobus_islandicus--M.16.4.stats: simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz
+run/specimen_index_count/Saccharolobus_islandicus--M.16.4/index.done run/stats/indexing_count/Saccharolobus_islandicus--M.16.4.stats: run/simulated_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz
-stats/verify_presence/Saccharolobus_islandicus--M.16.4.stats: reference_index/Saccharolobus_islandicus--M.16.4.npz specimen_index_presence/Saccharolobus_islandicus--M.16.4/index.done
+run/stats/verify_presence/Saccharolobus_islandicus--M.16.4.stats: run/reference_index/Saccharolobus_islandicus--M.16.4.npz run/specimen_index_presence/Saccharolobus_islandicus--M.16.4/index.done
-stats/verify_count/Saccharolobus_islandicus--M.16.4.stats: reference_index/Saccharolobus_islandicus--M.16.4.npz specimen_index_count/Saccharolobus_islandicus--M.16.4/index.done
+run/stats/verify_count/Saccharolobus_islandicus--M.16.4.stats: run/reference_index/Saccharolobus_islandicus--M.16.4.npz run/specimen_index_count/Saccharolobus_islandicus--M.16.4/index.done
 
 # Acidobacterium_capsulatum--ATCC_51196
-simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz: genomes/GCF_000022565.1_ASM2256v1_genomic.fna.gz
+run/simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz: run/genomes/GCF_000022565.1_ASM2256v1_genomic.fna.gz
-reference_index/Acidobacterium_capsulatum--ATCC_51196.npz: simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz
+run/reference_index/Acidobacterium_capsulatum--ATCC_51196.npz: run/simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz
-specimen_index_presence/Acidobacterium_capsulatum--ATCC_51196/index.done stats/indexing_presence/Acidobacterium_capsulatum--ATCC_51196.stats: simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz
+run/specimen_index_presence/Acidobacterium_capsulatum--ATCC_51196/index.done run/stats/indexing_presence/Acidobacterium_capsulatum--ATCC_51196.stats: run/simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz
-specimen_index_count/Acidobacterium_capsulatum--ATCC_51196/index.done stats/indexing_count/Acidobacterium_capsulatum--ATCC_51196.stats: simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz
+run/specimen_index_count/Acidobacterium_capsulatum--ATCC_51196/index.done run/stats/indexing_count/Acidobacterium_capsulatum--ATCC_51196.stats: run/simulated_data/Acidobacterium_capsulatum/ATCC_51196/reads_R1.fastq.gz
-stats/verify_presence/Acidobacterium_capsulatum--ATCC_51196.stats: reference_index/Acidobacterium_capsulatum--ATCC_51196.npz specimen_index_presence/Acidobacterium_capsulatum--ATCC_51196/index.done
+run/stats/verify_presence/Acidobacterium_capsulatum--ATCC_51196.stats: run/reference_index/Acidobacterium_capsulatum--ATCC_51196.npz run/specimen_index_presence/Acidobacterium_capsulatum--ATCC_51196/index.done
-stats/verify_count/Acidobacterium_capsulatum--ATCC_51196.stats: reference_index/Acidobacterium_capsulatum--ATCC_51196.npz specimen_index_count/Acidobacterium_capsulatum--ATCC_51196/index.done
+run/stats/verify_count/Acidobacterium_capsulatum--ATCC_51196.stats: run/reference_index/Acidobacterium_capsulatum--ATCC_51196.npz run/specimen_index_count/Acidobacterium_capsulatum--ATCC_51196/index.done
 
 # Salmonella_enterica--AKU_12601
-simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz: genomes/GCF_000026565.1_ASM2656v1_genomic.fna.gz
+run/simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz: run/genomes/GCF_000026565.1_ASM2656v1_genomic.fna.gz
-reference_index/Salmonella_enterica--AKU_12601.npz: simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz
+run/reference_index/Salmonella_enterica--AKU_12601.npz: run/simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz
-specimen_index_presence/Salmonella_enterica--AKU_12601/index.done stats/indexing_presence/Salmonella_enterica--AKU_12601.stats: simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz
+run/specimen_index_presence/Salmonella_enterica--AKU_12601/index.done run/stats/indexing_presence/Salmonella_enterica--AKU_12601.stats: run/simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz
-specimen_index_count/Salmonella_enterica--AKU_12601/index.done stats/indexing_count/Salmonella_enterica--AKU_12601.stats: simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz
+run/specimen_index_count/Salmonella_enterica--AKU_12601/index.done run/stats/indexing_count/Salmonella_enterica--AKU_12601.stats: run/simulated_data/Salmonella_enterica/AKU_12601/reads_R1.fastq.gz
-stats/verify_presence/Salmonella_enterica--AKU_12601.stats: reference_index/Salmonella_enterica--AKU_12601.npz specimen_index_presence/Salmonella_enterica--AKU_12601/index.done
+run/stats/verify_presence/Salmonella_enterica--AKU_12601.stats: run/reference_index/Salmonella_enterica--AKU_12601.npz run/specimen_index_presence/Salmonella_enterica--AKU_12601/index.done
-stats/verify_count/Salmonella_enterica--AKU_12601.stats: reference_index/Salmonella_enterica--AKU_12601.npz specimen_index_count/Salmonella_enterica--AKU_12601/index.done
+run/stats/verify_count/Salmonella_enterica--AKU_12601.stats: run/reference_index/Salmonella_enterica--AKU_12601.npz run/specimen_index_count/Salmonella_enterica--AKU_12601/index.done
 
 # Proteus_mirabilis--HI4320
-simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz: genomes/GCF_000069965.1_ASM6996v1_genomic.fna.gz
+run/simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz: run/genomes/GCF_000069965.1_ASM6996v1_genomic.fna.gz
-reference_index/Proteus_mirabilis--HI4320.npz: simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz
+run/reference_index/Proteus_mirabilis--HI4320.npz: run/simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz
-specimen_index_presence/Proteus_mirabilis--HI4320/index.done stats/indexing_presence/Proteus_mirabilis--HI4320.stats: simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz
+run/specimen_index_presence/Proteus_mirabilis--HI4320/index.done run/stats/indexing_presence/Proteus_mirabilis--HI4320.stats: run/simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz
-specimen_index_count/Proteus_mirabilis--HI4320/index.done stats/indexing_count/Proteus_mirabilis--HI4320.stats: simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz
+run/specimen_index_count/Proteus_mirabilis--HI4320/index.done run/stats/indexing_count/Proteus_mirabilis--HI4320.stats: run/simulated_data/Proteus_mirabilis/HI4320/reads_R1.fastq.gz
-stats/verify_presence/Proteus_mirabilis--HI4320.stats: reference_index/Proteus_mirabilis--HI4320.npz specimen_index_presence/Proteus_mirabilis--HI4320/index.done
+run/stats/verify_presence/Proteus_mirabilis--HI4320.stats: run/reference_index/Proteus_mirabilis--HI4320.npz run/specimen_index_presence/Proteus_mirabilis--HI4320/index.done
-stats/verify_count/Proteus_mirabilis--HI4320.stats: reference_index/Proteus_mirabilis--HI4320.npz specimen_index_count/Proteus_mirabilis--HI4320/index.done
+run/stats/verify_count/Proteus_mirabilis--HI4320.stats: run/reference_index/Proteus_mirabilis--HI4320.npz run/specimen_index_count/Proteus_mirabilis--HI4320/index.done
 
 # Salmonella_enterica--CT18
-simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz: genomes/GCF_000195995.1_ASM19599v1_genomic.fna.gz
+run/simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz: run/genomes/GCF_000195995.1_ASM19599v1_genomic.fna.gz
-reference_index/Salmonella_enterica--CT18.npz: simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz
+run/reference_index/Salmonella_enterica--CT18.npz: run/simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz
-specimen_index_presence/Salmonella_enterica--CT18/index.done stats/indexing_presence/Salmonella_enterica--CT18.stats: simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz
+run/specimen_index_presence/Salmonella_enterica--CT18/index.done run/stats/indexing_presence/Salmonella_enterica--CT18.stats: run/simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz
-specimen_index_count/Salmonella_enterica--CT18/index.done stats/indexing_count/Salmonella_enterica--CT18.stats: simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz
+run/specimen_index_count/Salmonella_enterica--CT18/index.done run/stats/indexing_count/Salmonella_enterica--CT18.stats: run/simulated_data/Salmonella_enterica/CT18/reads_R1.fastq.gz
-stats/verify_presence/Salmonella_enterica--CT18.stats: reference_index/Salmonella_enterica--CT18.npz specimen_index_presence/Salmonella_enterica--CT18/index.done
+run/stats/verify_presence/Salmonella_enterica--CT18.stats: run/reference_index/Salmonella_enterica--CT18.npz run/specimen_index_presence/Salmonella_enterica--CT18/index.done
-stats/verify_count/Salmonella_enterica--CT18.stats: reference_index/Salmonella_enterica--CT18.npz specimen_index_count/Salmonella_enterica--CT18/index.done
+run/stats/verify_count/Salmonella_enterica--CT18.stats: run/reference_index/Salmonella_enterica--CT18.npz run/specimen_index_count/Salmonella_enterica--CT18/index.done
 
 # Klebsiella_pneumoniae--HS11286
-simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz: genomes/GCF_000240185.1_ASM24018v2_genomic.fna.gz
+run/simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz: run/genomes/GCF_000240185.1_ASM24018v2_genomic.fna.gz
-reference_index/Klebsiella_pneumoniae--HS11286.npz: simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz
+run/reference_index/Klebsiella_pneumoniae--HS11286.npz: run/simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz
-specimen_index_presence/Klebsiella_pneumoniae--HS11286/index.done stats/indexing_presence/Klebsiella_pneumoniae--HS11286.stats: simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz
+run/specimen_index_presence/Klebsiella_pneumoniae--HS11286/index.done run/stats/indexing_presence/Klebsiella_pneumoniae--HS11286.stats: run/simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz
-specimen_index_count/Klebsiella_pneumoniae--HS11286/index.done stats/indexing_count/Klebsiella_pneumoniae--HS11286.stats: simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz
+run/specimen_index_count/Klebsiella_pneumoniae--HS11286/index.done run/stats/indexing_count/Klebsiella_pneumoniae--HS11286.stats: run/simulated_data/Klebsiella_pneumoniae/HS11286/reads_R1.fastq.gz
-stats/verify_presence/Klebsiella_pneumoniae--HS11286.stats: reference_index/Klebsiella_pneumoniae--HS11286.npz specimen_index_presence/Klebsiella_pneumoniae--HS11286/index.done
+run/stats/verify_presence/Klebsiella_pneumoniae--HS11286.stats: run/reference_index/Klebsiella_pneumoniae--HS11286.npz run/specimen_index_presence/Klebsiella_pneumoniae--HS11286/index.done
-stats/verify_count/Klebsiella_pneumoniae--HS11286.stats: reference_index/Klebsiella_pneumoniae--HS11286.npz specimen_index_count/Klebsiella_pneumoniae--HS11286/index.done
+run/stats/verify_count/Klebsiella_pneumoniae--HS11286.stats: run/reference_index/Klebsiella_pneumoniae--HS11286.npz run/specimen_index_count/Klebsiella_pneumoniae--HS11286/index.done
 
 # Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1
-simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz: genomes/GCF_000306885.1_ASM30688v1_genomic.fna.gz
+run/simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz: run/genomes/GCF_000306885.1_ASM30688v1_genomic.fna.gz
-reference_index/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.npz: simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz
+run/reference_index/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.npz: run/simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz
-specimen_index_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done stats/indexing_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz
+run/specimen_index_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done run/stats/indexing_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: run/simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz
-specimen_index_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done stats/indexing_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz
+run/specimen_index_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done run/stats/indexing_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: run/simulated_data/Wolbachia_endosymbiont/GCF_000306885.1_ASM30688v1/reads_R1.fastq.gz
-stats/verify_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: reference_index/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.npz specimen_index_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done
+run/stats/verify_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: run/reference_index/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.npz run/specimen_index_presence/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done
-stats/verify_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: reference_index/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.npz specimen_index_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done
+run/stats/verify_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.stats: run/reference_index/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1.npz run/specimen_index_count/Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1/index.done
 
 # Klebsiella_pneumoniae--ATCC_13883
-simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz: genomes/GCF_000742135.1_ASM74213v1_genomic.fna.gz
+run/simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz: run/genomes/GCF_000742135.1_ASM74213v1_genomic.fna.gz
-reference_index/Klebsiella_pneumoniae--ATCC_13883.npz: simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz
+run/reference_index/Klebsiella_pneumoniae--ATCC_13883.npz: run/simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz
-specimen_index_presence/Klebsiella_pneumoniae--ATCC_13883/index.done stats/indexing_presence/Klebsiella_pneumoniae--ATCC_13883.stats: simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz
+run/specimen_index_presence/Klebsiella_pneumoniae--ATCC_13883/index.done run/stats/indexing_presence/Klebsiella_pneumoniae--ATCC_13883.stats: run/simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz
-specimen_index_count/Klebsiella_pneumoniae--ATCC_13883/index.done stats/indexing_count/Klebsiella_pneumoniae--ATCC_13883.stats: simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz
+run/specimen_index_count/Klebsiella_pneumoniae--ATCC_13883/index.done run/stats/indexing_count/Klebsiella_pneumoniae--ATCC_13883.stats: run/simulated_data/Klebsiella_pneumoniae/ATCC_13883/reads_R1.fastq.gz
-stats/verify_presence/Klebsiella_pneumoniae--ATCC_13883.stats: reference_index/Klebsiella_pneumoniae--ATCC_13883.npz specimen_index_presence/Klebsiella_pneumoniae--ATCC_13883/index.done
+run/stats/verify_presence/Klebsiella_pneumoniae--ATCC_13883.stats: run/reference_index/Klebsiella_pneumoniae--ATCC_13883.npz run/specimen_index_presence/Klebsiella_pneumoniae--ATCC_13883/index.done
-stats/verify_count/Klebsiella_pneumoniae--ATCC_13883.stats: reference_index/Klebsiella_pneumoniae--ATCC_13883.npz specimen_index_count/Klebsiella_pneumoniae--ATCC_13883/index.done
+run/stats/verify_count/Klebsiella_pneumoniae--ATCC_13883.stats: run/reference_index/Klebsiella_pneumoniae--ATCC_13883.npz run/specimen_index_count/Klebsiella_pneumoniae--ATCC_13883/index.done
 
 # Yersinia_ruckeri--YRB
-simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz: genomes/GCF_000834255.1_ASM83425v1_genomic.fna.gz
+run/simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz: run/genomes/GCF_000834255.1_ASM83425v1_genomic.fna.gz
-reference_index/Yersinia_ruckeri--YRB.npz: simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz
+run/reference_index/Yersinia_ruckeri--YRB.npz: run/simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz
-specimen_index_presence/Yersinia_ruckeri--YRB/index.done stats/indexing_presence/Yersinia_ruckeri--YRB.stats: simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz
+run/specimen_index_presence/Yersinia_ruckeri--YRB/index.done run/stats/indexing_presence/Yersinia_ruckeri--YRB.stats: run/simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz
-specimen_index_count/Yersinia_ruckeri--YRB/index.done stats/indexing_count/Yersinia_ruckeri--YRB.stats: simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz
+run/specimen_index_count/Yersinia_ruckeri--YRB/index.done run/stats/indexing_count/Yersinia_ruckeri--YRB.stats: run/simulated_data/Yersinia_ruckeri/YRB/reads_R1.fastq.gz
-stats/verify_presence/Yersinia_ruckeri--YRB.stats: reference_index/Yersinia_ruckeri--YRB.npz specimen_index_presence/Yersinia_ruckeri--YRB/index.done
+run/stats/verify_presence/Yersinia_ruckeri--YRB.stats: run/reference_index/Yersinia_ruckeri--YRB.npz run/specimen_index_presence/Yersinia_ruckeri--YRB/index.done
-stats/verify_count/Yersinia_ruckeri--YRB.stats: reference_index/Yersinia_ruckeri--YRB.npz specimen_index_count/Yersinia_ruckeri--YRB/index.done
+run/stats/verify_count/Yersinia_ruckeri--YRB.stats: run/reference_index/Yersinia_ruckeri--YRB.npz run/specimen_index_count/Yersinia_ruckeri--YRB/index.done
 
 # Candidozyma_auris--GCF_003013715.1_ASM301371v2
-simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz: genomes/GCF_003013715.1_ASM301371v2_genomic.fna.gz
+run/simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz: run/genomes/GCF_003013715.1_ASM301371v2_genomic.fna.gz
-reference_index/Candidozyma_auris--GCF_003013715.1_ASM301371v2.npz: simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz
+run/reference_index/Candidozyma_auris--GCF_003013715.1_ASM301371v2.npz: run/simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz
-specimen_index_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done stats/indexing_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz
+run/specimen_index_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done run/stats/indexing_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: run/simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz
-specimen_index_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done stats/indexing_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz
+run/specimen_index_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done run/stats/indexing_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: run/simulated_data/Candidozyma_auris/GCF_003013715.1_ASM301371v2/reads_R1.fastq.gz
-stats/verify_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: reference_index/Candidozyma_auris--GCF_003013715.1_ASM301371v2.npz specimen_index_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done
+run/stats/verify_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: run/reference_index/Candidozyma_auris--GCF_003013715.1_ASM301371v2.npz run/specimen_index_presence/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done
-stats/verify_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: reference_index/Candidozyma_auris--GCF_003013715.1_ASM301371v2.npz specimen_index_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done
+run/stats/verify_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2.stats: run/reference_index/Candidozyma_auris--GCF_003013715.1_ASM301371v2.npz run/specimen_index_count/Candidozyma_auris--GCF_003013715.1_ASM301371v2/index.done
 
 # Escherichia_coli
-specific_index_presence/Escherichia_coli/index.done stats/specific_kmer_presence/Escherichia_coli.stats: global_index_presence/index.done
+run/specific_index_presence/Escherichia_coli/index.done run/stats/specific_kmer_presence/Escherichia_coli.stats: run/global_index_presence/index.done
-specific_index_count/Escherichia_coli/index.done stats/specific_kmer_count/Escherichia_coli.stats: global_index_count/index.done
+run/specific_index_count/Escherichia_coli/index.done run/stats/specific_kmer_count/Escherichia_coli.stats: run/global_index_count/index.done
 # Salmonella_enterica
-specific_index_presence/Salmonella_enterica/index.done stats/specific_kmer_presence/Salmonella_enterica.stats: global_index_presence/index.done
+run/specific_index_presence/Salmonella_enterica/index.done run/stats/specific_kmer_presence/Salmonella_enterica.stats: run/global_index_presence/index.done
-specific_index_count/Salmonella_enterica/index.done stats/specific_kmer_count/Salmonella_enterica.stats: global_index_count/index.done
+run/specific_index_count/Salmonella_enterica/index.done run/stats/specific_kmer_count/Salmonella_enterica.stats: run/global_index_count/index.done
 # Bacillus_subtilis
-specific_index_presence/Bacillus_subtilis/index.done stats/specific_kmer_presence/Bacillus_subtilis.stats: global_index_presence/index.done
+run/specific_index_presence/Bacillus_subtilis/index.done run/stats/specific_kmer_presence/Bacillus_subtilis.stats: run/global_index_presence/index.done
-specific_index_count/Bacillus_subtilis/index.done stats/specific_kmer_count/Bacillus_subtilis.stats: global_index_count/index.done
+run/specific_index_count/Bacillus_subtilis/index.done run/stats/specific_kmer_count/Bacillus_subtilis.stats: run/global_index_count/index.done
 # Shouchella_clausii
-specific_index_presence/Shouchella_clausii/index.done stats/specific_kmer_presence/Shouchella_clausii.stats: global_index_presence/index.done
+run/specific_index_presence/Shouchella_clausii/index.done run/stats/specific_kmer_presence/Shouchella_clausii.stats: run/global_index_presence/index.done
-specific_index_count/Shouchella_clausii/index.done stats/specific_kmer_count/Shouchella_clausii.stats: global_index_count/index.done
+run/specific_index_count/Shouchella_clausii/index.done run/stats/specific_kmer_count/Shouchella_clausii.stats: run/global_index_count/index.done
 # Klebsiella_pneumoniae
-specific_index_presence/Klebsiella_pneumoniae/index.done stats/specific_kmer_presence/Klebsiella_pneumoniae.stats: global_index_presence/index.done
+run/specific_index_presence/Klebsiella_pneumoniae/index.done run/stats/specific_kmer_presence/Klebsiella_pneumoniae.stats: run/global_index_presence/index.done
-specific_index_count/Klebsiella_pneumoniae/index.done stats/specific_kmer_count/Klebsiella_pneumoniae.stats: global_index_count/index.done
+run/specific_index_count/Klebsiella_pneumoniae/index.done run/stats/specific_kmer_count/Klebsiella_pneumoniae.stats: run/global_index_count/index.done
 # Opitutus_terrae
-specific_index_presence/Opitutus_terrae/index.done stats/specific_kmer_presence/Opitutus_terrae.stats: global_index_presence/index.done
+run/specific_index_presence/Opitutus_terrae/index.done run/stats/specific_kmer_presence/Opitutus_terrae.stats: run/global_index_presence/index.done
-specific_index_count/Opitutus_terrae/index.done stats/specific_kmer_count/Opitutus_terrae.stats: global_index_count/index.done
+run/specific_index_count/Opitutus_terrae/index.done run/stats/specific_kmer_count/Opitutus_terrae.stats: run/global_index_count/index.done
 # Saccharolobus_islandicus
-specific_index_presence/Saccharolobus_islandicus/index.done stats/specific_kmer_presence/Saccharolobus_islandicus.stats: global_index_presence/index.done
+run/specific_index_presence/Saccharolobus_islandicus/index.done run/stats/specific_kmer_presence/Saccharolobus_islandicus.stats: run/global_index_presence/index.done
-specific_index_count/Saccharolobus_islandicus/index.done stats/specific_kmer_count/Saccharolobus_islandicus.stats: global_index_count/index.done
+run/specific_index_count/Saccharolobus_islandicus/index.done run/stats/specific_kmer_count/Saccharolobus_islandicus.stats: run/global_index_count/index.done
 # Acidobacterium_capsulatum
-specific_index_presence/Acidobacterium_capsulatum/index.done stats/specific_kmer_presence/Acidobacterium_capsulatum.stats: global_index_presence/index.done
+run/specific_index_presence/Acidobacterium_capsulatum/index.done run/stats/specific_kmer_presence/Acidobacterium_capsulatum.stats: run/global_index_presence/index.done
-specific_index_count/Acidobacterium_capsulatum/index.done stats/specific_kmer_count/Acidobacterium_capsulatum.stats: global_index_count/index.done
+run/specific_index_count/Acidobacterium_capsulatum/index.done run/stats/specific_kmer_count/Acidobacterium_capsulatum.stats: run/global_index_count/index.done
 # Proteus_mirabilis
-specific_index_presence/Proteus_mirabilis/index.done stats/specific_kmer_presence/Proteus_mirabilis.stats: global_index_presence/index.done
+run/specific_index_presence/Proteus_mirabilis/index.done run/stats/specific_kmer_presence/Proteus_mirabilis.stats: run/global_index_presence/index.done
-specific_index_count/Proteus_mirabilis/index.done stats/specific_kmer_count/Proteus_mirabilis.stats: global_index_count/index.done
+run/specific_index_count/Proteus_mirabilis/index.done run/stats/specific_kmer_count/Proteus_mirabilis.stats: run/global_index_count/index.done
 # Wolbachia_endosymbiont
-specific_index_presence/Wolbachia_endosymbiont/index.done stats/specific_kmer_presence/Wolbachia_endosymbiont.stats: global_index_presence/index.done
+run/specific_index_presence/Wolbachia_endosymbiont/index.done run/stats/specific_kmer_presence/Wolbachia_endosymbiont.stats: run/global_index_presence/index.done
-specific_index_count/Wolbachia_endosymbiont/index.done stats/specific_kmer_count/Wolbachia_endosymbiont.stats: global_index_count/index.done
+run/specific_index_count/Wolbachia_endosymbiont/index.done run/stats/specific_kmer_count/Wolbachia_endosymbiont.stats: run/global_index_count/index.done
 # Yersinia_ruckeri
-specific_index_presence/Yersinia_ruckeri/index.done stats/specific_kmer_presence/Yersinia_ruckeri.stats: global_index_presence/index.done
+run/specific_index_presence/Yersinia_ruckeri/index.done run/stats/specific_kmer_presence/Yersinia_ruckeri.stats: run/global_index_presence/index.done
-specific_index_count/Yersinia_ruckeri/index.done stats/specific_kmer_count/Yersinia_ruckeri.stats: global_index_count/index.done
+run/specific_index_count/Yersinia_ruckeri/index.done run/stats/specific_kmer_count/Yersinia_ruckeri.stats: run/global_index_count/index.done
 # Candidozyma_auris
-specific_index_presence/Candidozyma_auris/index.done stats/specific_kmer_presence/Candidozyma_auris.stats: global_index_presence/index.done
+run/specific_index_presence/Candidozyma_auris/index.done run/stats/specific_kmer_presence/Candidozyma_auris.stats: run/global_index_presence/index.done
-specific_index_count/Candidozyma_auris/index.done stats/specific_kmer_count/Candidozyma_auris.stats: global_index_count/index.done
+run/specific_index_count/Candidozyma_auris/index.done run/stats/specific_kmer_count/Candidozyma_auris.stats: run/global_index_count/index.done
 
 QUERY_SPECIMENS := Escherichia_coli--K-12_MG1655 Saccharolobus_islandicus--M.16.4
 
 # query: Escherichia_coli--K-12_MG1655
-query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz: genomes/GCF_000005845.2_ASM584v2_genomic.fna.gz
+run/query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz: run/genomes/GCF_000005845.2_ASM584v2_genomic.fna.gz
-query_dense/Escherichia_coli--K-12_MG1655.fasta.gz stats/query_dense/Escherichia_coli--K-12_MG1655.stats: query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz global_index_presence/index.done
+run/query_presence_dense/Escherichia_coli--K-12_MG1655.fasta.gz run/stats/query_presence_dense/Escherichia_coli--K-12_MG1655.stats: run/query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz run/global_index_presence_dense/index.done
-query_sparse/Escherichia_coli--K-12_MG1655.fasta.gz stats/query_sparse/Escherichia_coli--K-12_MG1655.stats: query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz global_index_presence_sparse/index.done
+run/query_presence_sparse/Escherichia_coli--K-12_MG1655.fasta.gz run/stats/query_presence_sparse/Escherichia_coli--K-12_MG1655.stats: run/query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz run/global_index_presence/index.done
-stats/verify_query/Escherichia_coli--K-12_MG1655.stats: query_dense/Escherichia_coli--K-12_MG1655.fasta.gz query_sparse/Escherichia_coli--K-12_MG1655.fasta.gz
+run/stats/verify_query/Escherichia_coli--K-12_MG1655.stats: run/query_presence_dense/Escherichia_coli--K-12_MG1655.fasta.gz run/query_presence_sparse/Escherichia_coli--K-12_MG1655.fasta.gz
 
 # query: Saccharolobus_islandicus--M.16.4
-query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz: genomes/GCF_000022445.1_ASM2244v1_genomic.fna.gz
+run/query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz: run/genomes/GCF_000022445.1_ASM2244v1_genomic.fna.gz
-query_dense/Saccharolobus_islandicus--M.16.4.fasta.gz stats/query_dense/Saccharolobus_islandicus--M.16.4.stats: query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz global_index_presence/index.done
+run/query_presence_dense/Saccharolobus_islandicus--M.16.4.fasta.gz run/stats/query_presence_dense/Saccharolobus_islandicus--M.16.4.stats: run/query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz run/global_index_presence_dense/index.done
-query_sparse/Saccharolobus_islandicus--M.16.4.fasta.gz stats/query_sparse/Saccharolobus_islandicus--M.16.4.stats: query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz global_index_presence_sparse/index.done
+run/query_presence_sparse/Saccharolobus_islandicus--M.16.4.fasta.gz run/stats/query_presence_sparse/Saccharolobus_islandicus--M.16.4.stats: run/query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz run/global_index_presence/index.done
-stats/verify_query/Saccharolobus_islandicus--M.16.4.stats: query_dense/Saccharolobus_islandicus--M.16.4.fasta.gz query_sparse/Saccharolobus_islandicus--M.16.4.fasta.gz
+run/stats/verify_query/Saccharolobus_islandicus--M.16.4.stats: run/query_presence_dense/Saccharolobus_islandicus--M.16.4.fasta.gz run/query_presence_sparse/Saccharolobus_islandicus--M.16.4.fasta.gz

benchmark/downloads.sh

@@ -29,7 +29,10 @@ assemblies=(
     GCF_000834255.1
 )
 
-mkdir -p genomes
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+GENOMES_DIR="${SCRIPT_DIR}/run/genomes"
+
+mkdir -p "${GENOMES_DIR}"
 
 for acc in "${assemblies[@]}"; do
     echo "Downloading ${acc}"
@@ -41,7 +44,7 @@ for acc in "${assemblies[@]}"; do
     unzip -q "${acc}.zip" -d "${acc}"
     find "${acc}" -name "*.fna" |
         while read file; do
-            obiconvert -Z ${file} >genomes/$(basename ${file}).gz
+            obiconvert -Z ${file} >"${GENOMES_DIR}/$(basename ${file}).gz"
         done
 
     rm -rf "${acc}" "${acc}.zip"

benchmark/filter_one_count.sh

@@ -1,7 +1,10 @@
 #!/usr/bin/env bash
 # Usage: filter_one_count.sh SPECIES
 # Filters global_index_count to keep only kmers specific to SPECIES,
-# then selects the SPECIES column in-place.
+# then selects the SPECIES column (to a temp dir, swapped over the
+# original — obikmer select has no --in-place mode, but its output hard-
+# links each layer's unchanged kmer-identity files rather than copying
+# them, so this costs no extra disk for those).
 # Outputs:
 #   specific_index_count/SPECIES/index.done  (written by obikmer select)
 #   stats/specific_kmer_count/SPECIES.stats  (one CSV data row, no header)
@@ -9,11 +12,12 @@ set -euo pipefail
 
 SPECIES="$1"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
 
-SOURCE="${SCRIPT_DIR}/global_index_count"
+SOURCE="${RUN_DIR}/global_index_count"
-OUTPUT="${SCRIPT_DIR}/specific_index_count/${SPECIES}"
+OUTPUT="${RUN_DIR}/specific_index_count/${SPECIES}"
-STATS_DIR="${SCRIPT_DIR}/stats/specific_kmer_count"
+STATS_DIR="${RUN_DIR}/stats/specific_kmer_count"
 STATS_FILE="${STATS_DIR}/${SPECIES}.stats"
 
 mkdir -p "${STATS_DIR}"
@@ -37,8 +41,11 @@ trap 'rm -f "${LOG_FILTER}" "${LOG_SELECT}"' EXIT
 
 cat "${LOG_FILTER}" >&2
 
+SELECT_TMP="${OUTPUT}.select_tmp"
+rm -rf "${SELECT_TMP}"
+
 "${BINARY}" select \
-    --in-place \
+    --output "${SELECT_TMP}" \
     --group "${SPECIES}:species=${SPECIES}" \
     --group-op "${SPECIES}:any" \
     --select "${SPECIES}" \
@@ -47,6 +54,9 @@ cat "${LOG_FILTER}" >&2
 
 cat "${LOG_SELECT}" >&2
 
+rm -rf "${OUTPUT}"
+mv "${SELECT_TMP}" "${OUTPUT}"
+
 python3 - "${SPECIES}" "${LOG_FILTER}" "${LOG_SELECT}" <<'PYEOF' >"${STATS_FILE}"
 import sys, re
 

benchmark/filter_one_presence.sh

@@ -1,7 +1,10 @@
 #!/usr/bin/env bash
 # Usage: filter_one_presence.sh SPECIES
 # Filters global_index_presence to keep only kmers specific to SPECIES,
-# then selects the SPECIES column in-place.
+# then selects the SPECIES column (to a temp dir, swapped over the
+# original — obikmer select has no --in-place mode, but its output hard-
+# links each layer's unchanged kmer-identity files rather than copying
+# them, so this costs no extra disk for those).
 # Outputs:
 #   specific_index_presence/SPECIES/index.done  (written by obikmer select)
 #   stats/specific_kmer_presence/SPECIES.stats  (one CSV data row, no header)
@@ -9,11 +12,12 @@ set -euo pipefail
 
 SPECIES="$1"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
 
-SOURCE="${SCRIPT_DIR}/global_index_presence"
+SOURCE="${RUN_DIR}/global_index_presence"
-OUTPUT="${SCRIPT_DIR}/specific_index_presence/${SPECIES}"
+OUTPUT="${RUN_DIR}/specific_index_presence/${SPECIES}"
-STATS_DIR="${SCRIPT_DIR}/stats/specific_kmer_presence"
+STATS_DIR="${RUN_DIR}/stats/specific_kmer_presence"
 STATS_FILE="${STATS_DIR}/${SPECIES}.stats"
 
 mkdir -p "${STATS_DIR}"
@@ -37,8 +41,11 @@ trap 'rm -f "${LOG_FILTER}" "${LOG_SELECT}"' EXIT
 
 cat "${LOG_FILTER}" >&2
 
+SELECT_TMP="${OUTPUT}.select_tmp"
+rm -rf "${SELECT_TMP}"
+
 "${BINARY}" select \
-    --in-place \
+    --output "${SELECT_TMP}" \
     --group "${SPECIES}:species=${SPECIES}" \
     --group-op "${SPECIES}:any" \
     --select "${SPECIES}" \
@@ -47,6 +54,9 @@ cat "${LOG_FILTER}" >&2
 
 cat "${LOG_SELECT}" >&2
 
+rm -rf "${OUTPUT}"
+mv "${SELECT_TMP}" "${OUTPUT}"
+
 python3 - "${SPECIES}" "${LOG_FILTER}" "${LOG_SELECT}" <<'PYEOF' >"${STATS_FILE}"
 import sys, re
 

benchmark/index_one_count.sh

@@ -8,14 +8,15 @@ set -euo pipefail
 
 SPECIMEN="$1"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
 
 species="${SPECIMEN%%--*}"
 strain="${SPECIMEN#*--}"
 
-READS_DIR="${SCRIPT_DIR}/simulated_data/${species}/${strain}"
+READS_DIR="${RUN_DIR}/simulated_data/${species}/${strain}"
-INDEX_PATH="${SCRIPT_DIR}/specimen_index_count/${SPECIMEN}"
+INDEX_PATH="${RUN_DIR}/specimen_index_count/${SPECIMEN}"
-STATS_DIR="${SCRIPT_DIR}/stats/indexing_count"
+STATS_DIR="${RUN_DIR}/stats/indexing_count"
 STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
 
 mkdir -p "${STATS_DIR}"

benchmark/index_one_presence.sh

@@ -8,14 +8,15 @@ set -euo pipefail
 
 SPECIMEN="$1"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
 
 species="${SPECIMEN%%--*}"
 strain="${SPECIMEN#*--}"
 
-READS_DIR="${SCRIPT_DIR}/simulated_data/${species}/${strain}"
+READS_DIR="${RUN_DIR}/simulated_data/${species}/${strain}"
-INDEX_PATH="${SCRIPT_DIR}/specimen_index_presence/${SPECIMEN}"
+INDEX_PATH="${RUN_DIR}/specimen_index_presence/${SPECIMEN}"
-STATS_DIR="${SCRIPT_DIR}/stats/indexing_presence"
+STATS_DIR="${RUN_DIR}/stats/indexing_presence"
 STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
 
 mkdir -p "${STATS_DIR}"

benchmark/make_deps.py

@@ -13,6 +13,10 @@ STOP_WORDS    = {'complete', 'chromosome', 'whole', 'sequence', 'genome',
                  'endosymbiont', 'of'}
 STOP_PREFIXES = ('scaffold', 'contig', 'plasmid')
 
+# All generated paths live under RUN/ (see Makefile) so the whole tree can be
+# gitignored with a single entry.
+RUN = 'run'
+
 # Specimens used as read sources for the query benchmark (see
 # DevDocMD/implementation/benchmark_query_testing.md): one common bacterium,
 # one distant lineage (the only archaeon in SPECIES).
@@ -80,7 +84,7 @@ def main():
         defn    = first_definition(path)
         sp, st  = parse_organism(defn, gcf_id)
         specimen = f'{sp}--{st}'
-        sim_dir  = f'simulated_data/{sp}/{st}'
+        sim_dir  = f'{RUN}/simulated_data/{sp}/{st}'
         entries.append((specimen, sp, sim_dir, path))
         if sp not in species_seen:
             species_seen.append(sp)
@@ -91,13 +95,13 @@ def main():
 
     for specimen, species, sim_dir, genome in entries:
         reads = f'{sim_dir}/reads_R1.fastq.gz'
-        p_done  = f'specimen_index_presence/{specimen}/index.done'
+        p_done  = f'{RUN}/specimen_index_presence/{specimen}/index.done'
-        p_stats = f'stats/indexing_presence/{specimen}.stats'
+        p_stats = f'{RUN}/stats/indexing_presence/{specimen}.stats'
-        c_done  = f'specimen_index_count/{specimen}/index.done'
+        c_done  = f'{RUN}/specimen_index_count/{specimen}/index.done'
-        c_stats = f'stats/indexing_count/{specimen}.stats'
+        c_stats = f'{RUN}/stats/indexing_count/{specimen}.stats'
-        ref     = f'reference_index/{specimen}.npz'
+        ref     = f'{RUN}/reference_index/{specimen}.npz'
-        vp      = f'stats/verify_presence/{specimen}.stats'
+        vp      = f'{RUN}/stats/verify_presence/{specimen}.stats'
-        vc      = f'stats/verify_count/{specimen}.stats'
+        vc      = f'{RUN}/stats/verify_count/{specimen}.stats'
 
         print()
         print(f'# {specimen}')
@@ -110,13 +114,13 @@ def main():
 
     print()
     for sp in species_seen:
-        sp_done  = f'specific_index_presence/{sp}/index.done'
+        sp_done  = f'{RUN}/specific_index_presence/{sp}/index.done'
-        sp_stats = f'stats/specific_kmer_presence/{sp}.stats'
+        sp_stats = f'{RUN}/stats/specific_kmer_presence/{sp}.stats'
-        sc_done  = f'specific_index_count/{sp}/index.done'
+        sc_done  = f'{RUN}/specific_index_count/{sp}/index.done'
-        sc_stats = f'stats/specific_kmer_count/{sp}.stats'
+        sc_stats = f'{RUN}/stats/specific_kmer_count/{sp}.stats'
         print(f'# {sp}')
-        print(f'{sp_done} {sp_stats}: global_index_presence/index.done')
+        print(f'{sp_done} {sp_stats}: {RUN}/global_index_presence/index.done')
-        print(f'{sc_done} {sc_stats}: global_index_count/index.done')
+        print(f'{sc_done} {sc_stats}: {RUN}/global_index_count/index.done')
 
     print()
     print('QUERY_SPECIMENS :=', ' '.join(QUERY_SPECIMENS))
@@ -126,17 +130,17 @@ def main():
         _, species, sim_dir, genome = by_specimen[specimen]
         query_dir = sim_dir.replace('simulated_data/', 'query_data/', 1)
         reads      = f'{query_dir}/reads_R1.fastq.gz'
-        dense_out  = f'query_dense/{specimen}.fasta.gz'
+        dense_out  = f'{RUN}/query_presence_dense/{specimen}.fasta.gz'
-        dense_stat = f'stats/query_dense/{specimen}.stats'
+        dense_stat = f'{RUN}/stats/query_presence_dense/{specimen}.stats'
-        sparse_out  = f'query_sparse/{specimen}.fasta.gz'
+        sparse_out  = f'{RUN}/query_presence_sparse/{specimen}.fasta.gz'
-        sparse_stat = f'stats/query_sparse/{specimen}.stats'
+        sparse_stat = f'{RUN}/stats/query_presence_sparse/{specimen}.stats'
-        vq_stat     = f'stats/verify_query/{specimen}.stats'
+        vq_stat     = f'{RUN}/stats/verify_query/{specimen}.stats'
 
         print()
         print(f'# query: {specimen}')
         print(f'{reads}: {genome}')
-        print(f'{dense_out} {dense_stat}: {reads} global_index_presence/index.done')
+        print(f'{dense_out} {dense_stat}: {reads} {RUN}/global_index_presence_dense/index.done')
-        print(f'{sparse_out} {sparse_stat}: {reads} global_index_presence_sparse/index.done')
+        print(f'{sparse_out} {sparse_stat}: {reads} {RUN}/global_index_presence/index.done')
         print(f'{vq_stat}: {dense_out} {sparse_out}')
 
 

benchmark/merge_count.sh

@@ -2,10 +2,11 @@
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
-IDX_DIR="${SCRIPT_DIR}/specimen_index_count"
+IDX_DIR="${RUN_DIR}/specimen_index_count"
-OUTPUT="${SCRIPT_DIR}/global_index_count"
+OUTPUT="${RUN_DIR}/global_index_count"
-STATS_DIR="${SCRIPT_DIR}/stats/merge_count"
+STATS_DIR="${RUN_DIR}/stats/merge_count"
 
 mkdir -p "${STATS_DIR}"
 

benchmark/merge_presence.sh

@@ -2,10 +2,11 @@
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
-IDX_DIR="${SCRIPT_DIR}/specimen_index_presence"
+IDX_DIR="${RUN_DIR}/specimen_index_presence"
-OUTPUT="${SCRIPT_DIR}/global_index_presence"
+OUTPUT="${RUN_DIR}/global_index_presence"
-STATS_DIR="${SCRIPT_DIR}/stats/merge_presence"
+STATS_DIR="${RUN_DIR}/stats/merge_presence"
 
 mkdir -p "${STATS_DIR}"
 

benchmark/pack_dense.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Usage: pack_dense.sh KIND   (KIND = presence | count)
+#
+# Builds global_index_KIND_dense/ from global_index_KIND/ — `merge` packs
+# sparse by default (2026-08-28), so the dense comparison arm needs an
+# explicit rebuild.
+#
+# `obikmer pack --dense` cannot do this: it only converts the raw, freshly-
+# built columnar (per-genome-file, unpacked) matrix into a packed format —
+# `finalize_indexed` already packs (sparse by default) as the last step of
+# every index-building command, so there is no columnar leftover for a
+# second `pack` invocation to work from; it fails ("No such file or
+# directory", `obicompactvec::bitmatrix::packed::pack_bit_matrix` looking
+# for a `meta.json` that packing already cleaned up).
+#
+# `obikmer select` doesn't have that limitation — it always rebuilds its
+# output from scratch via `MatrixBuilder`, reading the source through the
+# format-agnostic `PersistentBitMatrix`/`PersistentIntMatrix` (Sparse
+# included, both content kinds — see `obicompactvec::batch_presence_counts`/
+# `batch_int_group_stats`) — so a full, unaggregated passthrough (`--select`
+# naming every genome, no `--group`) with `--dense` genuinely repacks
+# Sparse → Dense for either kind, and gets the layer-identity hard-linking
+# already implemented in `obikselect::select_layer::copy_layer_files` for
+# free.
+#
+# Outputs:
+#   global_index_KIND_dense/index.done   (rebuilt via `select`, dense-packed)
+#   stats/pack_dense_KIND/current.stats  (one CSV data row, no header)
+set -euo pipefail
+
+KIND="$1"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
+BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
+SOURCE="${RUN_DIR}/global_index_${KIND}"
+OUTPUT="${RUN_DIR}/global_index_${KIND}_dense"
+STATS_DIR="${RUN_DIR}/stats/pack_dense_${KIND}"
+STATS_FILE="${STATS_DIR}/current.stats"
+
+mkdir -p "${STATS_DIR}"
+
+echo "[pack_dense_${KIND}] ${SOURCE} → ${OUTPUT}"
+
+LABELS=$("${BINARY}" annotate "${SOURCE}" --dump | tail -n +2 | python3 -c "
+import sys, csv
+r = csv.reader(sys.stdin)
+print(','.join(row[0] for row in r if row))
+")
+
+STDERR_LOG=$(mktemp)
+trap 'rm -f "${STDERR_LOG}"' EXIT
+
+"${BINARY}" select \
+    --output "${OUTPUT}" \
+    --force \
+    --dense \
+    --select "${LABELS}" \
+    "${SOURCE}" \
+    2>"${STDERR_LOG}"
+
+cat "${STDERR_LOG}" >&2
+
+python3 - "${STDERR_LOG}" <<'PYEOF' >"${STATS_FILE}"
+import sys, re
+
+logfile = sys.argv[1]
+
+def strip_ansi(s):
+    return re.sub(r'\x1b\[[\x30-\x3f]*[\x20-\x2f]*[\x40-\x7e]', '', s)
+
+def parse_wall(s):
+    s = s.strip()
+    if s.endswith('ms'): return float(s[:-2]) / 1000.0
+    if s.endswith('s'):  return float(s[:-1])
+    return 0.0
+
+def parse_rss(s):
+    m = re.match(r'([\d.]+)\s*(GB|MB|KB|B)', s.strip())
+    if not m: return 0
+    return int(float(m.group(1)) * {'GB': 1<<30, 'MB': 1<<20, 'KB': 1024, 'B': 1}[m.group(2)])
+
+def is_sep(s):
+    return bool(s) and not re.search(r'[A-Za-z0-9]', s)
+
+stats = {}
+state = 'scan'
+with open(logfile, errors='replace') as fh:
+    for raw in fh:
+        line = strip_ansi(raw.rstrip('\n'))
+        s    = line.strip()
+        if state == 'scan':
+            if re.search(r'\bstage\b.*\bwall\b', line):
+                state = 'in_header'
+        elif state == 'in_header':
+            if is_sep(s): state = 'rows'
+        elif state == 'rows':
+            if is_sep(s): state = 'total'
+            elif s:
+                parts = re.split(r'  +', s)
+                if len(parts) >= 4:
+                    stats[parts[0]] = (parse_wall(parts[1]), parse_rss(parts[3]))
+        elif state == 'total':
+            if s:
+                parts = re.split(r'  +', s)
+                if len(parts) >= 3:
+                    stats['TOTAL'] = (parse_wall(parts[1]),
+                                      parse_rss(parts[3]) if len(parts) > 3 else 0)
+            break
+
+w, r = stats.get('select', ('', ''))
+tw, tr = stats.get('TOTAL', ('', ''))
+row = [f'{w:.3f}' if isinstance(w, float) else '', str(r),
+       f'{tw:.3f}' if isinstance(tw, float) else '', str(tr)]
+print(','.join(row))
+PYEOF
+
+echo "Done → ${OUTPUT}"

benchmark/pack_sparse.sh

@@ -1,86 +0,0 @@
-#!/usr/bin/env bash
-# Builds global_index_presence_sparse/ from global_index_presence/ by
-# copying the index (column files are kept in place after merge's dense
-# pack — see obikindex::KmerIndex::pack_matrices) and repacking in place
-# with --sparse.
-# Outputs:
-#   global_index_presence_sparse/index.done  (copied from source)
-#   stats/pack_sparse/current.stats          (one CSV data row, no header)
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
-SOURCE="${SCRIPT_DIR}/global_index_presence"
-OUTPUT="${SCRIPT_DIR}/global_index_presence_sparse"
-STATS_DIR="${SCRIPT_DIR}/stats/pack_sparse"
-STATS_FILE="${STATS_DIR}/current.stats"
-
-mkdir -p "${STATS_DIR}"
-
-echo "[pack_sparse] ${SOURCE} → ${OUTPUT}"
-
-rm -rf "${OUTPUT}"
-cp -r "${SOURCE}" "${OUTPUT}"
-
-STDERR_LOG=$(mktemp)
-trap 'rm -f "${STDERR_LOG}"' EXIT
-
-"${BINARY}" pack --sparse "${OUTPUT}" 2>"${STDERR_LOG}"
-
-cat "${STDERR_LOG}" >&2
-
-python3 - "${STDERR_LOG}" <<'PYEOF' >"${STATS_FILE}"
-import sys, re
-
-logfile = sys.argv[1]
-
-def strip_ansi(s):
-    return re.sub(r'\x1b\[[\x30-\x3f]*[\x20-\x2f]*[\x40-\x7e]', '', s)
-
-def parse_wall(s):
-    s = s.strip()
-    if s.endswith('ms'): return float(s[:-2]) / 1000.0
-    if s.endswith('s'):  return float(s[:-1])
-    return 0.0
-
-def parse_rss(s):
-    m = re.match(r'([\d.]+)\s*(GB|MB|KB|B)', s.strip())
-    if not m: return 0
-    return int(float(m.group(1)) * {'GB': 1<<30, 'MB': 1<<20, 'KB': 1024, 'B': 1}[m.group(2)])
-
-def is_sep(s):
-    return bool(s) and not re.search(r'[A-Za-z0-9]', s)
-
-stats = {}
-state = 'scan'
-with open(logfile, errors='replace') as fh:
-    for raw in fh:
-        line = strip_ansi(raw.rstrip('\n'))
-        s    = line.strip()
-        if state == 'scan':
-            if re.search(r'\bstage\b.*\bwall\b', line):
-                state = 'in_header'
-        elif state == 'in_header':
-            if is_sep(s): state = 'rows'
-        elif state == 'rows':
-            if is_sep(s): state = 'total'
-            elif s:
-                parts = re.split(r'  +', s)
-                if len(parts) >= 4:
-                    stats[parts[0]] = (parse_wall(parts[1]), parse_rss(parts[3]))
-        elif state == 'total':
-            if s:
-                parts = re.split(r'  +', s)
-                if len(parts) >= 3:
-                    stats['TOTAL'] = (parse_wall(parts[1]),
-                                      parse_rss(parts[3]) if len(parts) > 3 else 0)
-            break
-
-w, r = stats.get('pack', ('', ''))
-tw, tr = stats.get('TOTAL', ('', ''))
-row = [f'{w:.3f}' if isinstance(w, float) else '', str(r),
-       f'{tw:.3f}' if isinstance(tw, float) else '', str(tr)]
-print(','.join(row))
-PYEOF
-
-echo "Done → ${OUTPUT}"

benchmark/query_one.sh

@@ -1,29 +1,39 @@
 #!/usr/bin/env bash
-# Usage: query_one.sh MODE SPECIMEN
+# Usage: query_one.sh KIND MODE SPECIMEN
-# MODE = dense | sparse
+# KIND = presence | count
+# MODE = dense | sparse — `merge` packs sparse by default (2026-08-28), so
+#        "sparse" is global_index_KIND itself, unmodified; "dense" is the
+#        explicitly repacked global_index_KIND_dense (see pack_dense.sh).
 # SPECIMEN = "species--strain" (Make pattern stem), reads from query_data/
 # Outputs:
-#   query_MODE/SPECIMEN.fasta.gz     (obikmer query output, --count-missing)
+#   query_KIND_MODE/SPECIMEN.fasta.gz     (obikmer query output, --count-missing)
-#   stats/query_MODE/SPECIMEN.stats  (one CSV data row, no header)
+#   stats/query_KIND_MODE/SPECIMEN.stats  (one CSV data row, no header)
 set -euo pipefail
 
-MODE="$1"
+KIND="$1"
-SPECIMEN="$2"
+MODE="$2"
+SPECIMEN="$3"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
 
+case "${KIND}" in
+    presence|count) ;;
+    *) echo "ERROR: unknown kind '${KIND}' (expected presence|count)" >&2; exit 1 ;;
+esac
+
 case "${MODE}" in
-    dense)  INDEX="${SCRIPT_DIR}/global_index_presence" ;;
+    sparse) INDEX="${RUN_DIR}/global_index_${KIND}" ;;
-    sparse) INDEX="${SCRIPT_DIR}/global_index_presence_sparse" ;;
+    dense)  INDEX="${RUN_DIR}/global_index_${KIND}_dense" ;;
     *) echo "ERROR: unknown mode '${MODE}' (expected dense|sparse)" >&2; exit 1 ;;
 esac
 
 species="${SPECIMEN%%--*}"
 strain="${SPECIMEN#*--}"
 
-READS_DIR="${SCRIPT_DIR}/query_data/${species}/${strain}"
+READS_DIR="${RUN_DIR}/query_data/${species}/${strain}"
-OUT_DIR="${SCRIPT_DIR}/query_${MODE}"
+OUT_DIR="${RUN_DIR}/query_${KIND}_${MODE}"
-STATS_DIR="${SCRIPT_DIR}/stats/query_${MODE}"
+STATS_DIR="${RUN_DIR}/stats/query_${KIND}_${MODE}"
 OUT_FILE="${OUT_DIR}/${SPECIMEN}.fasta.gz"
 STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
 
@@ -36,7 +46,7 @@ if [[ ! -f "${r1}" || ! -f "${r2}" ]]; then
     exit 1
 fi
 
-echo "[${SPECIMEN}] query (${MODE}) → ${OUT_FILE}"
+echo "[${SPECIMEN}] query (${KIND}/${MODE}) → ${OUT_FILE}"
 
 STDERR_LOG=$(mktemp)
 trap 'rm -f "${STDERR_LOG}"' EXIT

benchmark/simulate.sh

@@ -4,8 +4,9 @@
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${SCRIPT_DIR}/run"
 
-for genome_file in "${SCRIPT_DIR}"/genomes/*.fna.gz; do
+for genome_file in "${RUN_DIR}"/genomes/*.fna.gz; do
     out_dir=$("${SCRIPT_DIR}/../.venv/bin/python3" "${SCRIPT_DIR}/make_deps.py" \
         --dir-for "${genome_file}")
     bash "${SCRIPT_DIR}/simulate_one.sh" "${genome_file}" "${out_dir}"

benchmark/test_dist.csv

@@ -1,21 +0,0 @@
-genome,Candidozyma_auris--GCF_003013715.1_ASM301371v2,Acidobacterium_capsulatum--ATCC_51196,Bacillus_subtilis--168,Escherichia_coli--CFT073,Escherichia_coli--EDL933,Escherichia_coli--K-12_MG1655,Escherichia_coli--K-12_W3110,Klebsiella_pneumoniae--ATCC_13883,Klebsiella_pneumoniae--HS11286,Klebsiella_pneumoniae--MGH_78578,Opitutus_terrae--PB90-1,Proteus_mirabilis--HI4320,Saccharolobus_islandicus--M.16.4,Salmonella_enterica--AKU_12601,Salmonella_enterica--CT18,Salmonella_enterica--LT2,Salmonella_enterica--P125109,Shouchella_clausii--KSM-K16,Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1,Yersinia_ruckeri--YRB
-Candidozyma_auris--GCF_003013715.1_ASM301371v2,0.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000
-Acidobacterium_capsulatum--ATCC_51196,1.000000,0.000000,0.999981,0.999990,0.999989,0.999987,0.999987,0.999990,0.999988,0.999988,0.999994,0.999989,1.000000,0.999988,0.999987,0.999987,0.999988,0.999989,0.999991,0.999987
-Bacillus_subtilis--168,1.000000,0.999981,0.000000,0.999990,0.999989,0.999989,0.999989,0.999989,0.999988,0.999986,0.999995,0.999985,0.999999,0.999988,0.999987,0.999989,0.999988,0.999778,0.999993,0.999987
-Escherichia_coli--CFT073,1.000000,0.999990,0.999990,0.000000,0.825741,0.807495,0.807218,0.991156,0.996855,0.997849,0.999996,0.999633,1.000000,0.993885,0.996736,0.994148,0.993821,0.999991,0.999984,0.999291
-Escherichia_coli--EDL933,1.000000,0.999989,0.999989,0.825741,0.000000,0.735107,0.734775,0.996126,0.998058,0.997908,0.999997,0.999640,1.000000,0.993993,0.997126,0.994390,0.994059,0.999991,0.999986,0.999292
-Escherichia_coli--K-12_MG1655,1.000000,0.999987,0.999989,0.807495,0.735107,0.000000,0.382567,0.996190,0.997747,0.997455,0.999996,0.999604,1.000000,0.993444,0.996645,0.993773,0.993431,0.999989,0.999984,0.999174
-Escherichia_coli--K-12_W3110,1.000000,0.999987,0.999989,0.807218,0.734775,0.382567,0.000000,0.996220,0.997761,0.997467,0.999995,0.999604,1.000000,0.993445,0.996669,0.993769,0.993443,0.999990,0.999985,0.999165
-Klebsiella_pneumoniae--ATCC_13883,1.000000,0.999990,0.999989,0.991156,0.996126,0.996190,0.996220,0.000000,0.845220,0.840545,0.999997,0.999648,1.000000,0.996177,0.998128,0.996268,0.996052,0.999990,0.999987,0.999325
-Klebsiella_pneumoniae--HS11286,1.000000,0.999988,0.999988,0.996855,0.998058,0.997747,0.997761,0.845220,0.000000,0.906475,0.999996,0.999683,1.000000,0.997724,0.995697,0.997776,0.997769,0.999989,0.999979,0.999463
-Klebsiella_pneumoniae--MGH_78578,1.000000,0.999988,0.999986,0.997849,0.997908,0.997455,0.997467,0.840545,0.906475,0.000000,0.999996,0.999704,1.000000,0.997928,0.995054,0.997844,0.997868,0.999990,0.999980,0.999479
-Opitutus_terrae--PB90-1,1.000000,0.999994,0.999995,0.999996,0.999997,0.999996,0.999995,0.999997,0.999996,0.999996,0.000000,0.999997,0.999998,0.999996,0.999996,0.999996,0.999995,0.999997,0.999993,0.999996
-Proteus_mirabilis--HI4320,1.000000,0.999989,0.999985,0.999633,0.999640,0.999604,0.999604,0.999648,0.999683,0.999704,0.999997,0.000000,1.000000,0.999604,0.999699,0.999622,0.999613,0.999987,0.999983,0.999505
-Saccharolobus_islandicus--M.16.4,1.000000,1.000000,0.999999,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,0.999998,1.000000,0.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000,1.000000
-Salmonella_enterica--AKU_12601,1.000000,0.999988,0.999988,0.993885,0.993993,0.993444,0.993445,0.996177,0.997724,0.997928,0.999996,0.999604,1.000000,0.000000,0.869238,0.682277,0.663383,0.999990,0.999985,0.999260
-Salmonella_enterica--CT18,1.000000,0.999987,0.999987,0.996736,0.997126,0.996645,0.996669,0.998128,0.995697,0.995054,0.999996,0.999699,1.000000,0.869238,0.000000,0.890872,0.886148,0.999988,0.999976,0.999524
-Salmonella_enterica--LT2,1.000000,0.999987,0.999989,0.994148,0.994390,0.993773,0.993769,0.996268,0.997776,0.997844,0.999996,0.999622,1.000000,0.682277,0.890872,0.000000,0.622606,0.999989,0.999985,0.999296
-Salmonella_enterica--P125109,1.000000,0.999988,0.999988,0.993821,0.994059,0.993431,0.993443,0.996052,0.997769,0.997868,0.999995,0.999613,1.000000,0.663383,0.886148,0.622606,0.000000,0.999988,0.999983,0.999270
-Shouchella_clausii--KSM-K16,1.000000,0.999989,0.999778,0.999991,0.999991,0.999989,0.999990,0.999990,0.999989,0.999990,0.999997,0.999987,1.000000,0.999990,0.999988,0.999989,0.999988,0.000000,0.999991,0.999988
-Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1,1.000000,0.999991,0.999993,0.999984,0.999986,0.999984,0.999985,0.999987,0.999979,0.999980,0.999993,0.999983,1.000000,0.999985,0.999976,0.999985,0.999983,0.999991,0.000000,0.999983
-Yersinia_ruckeri--YRB,1.000000,0.999987,0.999987,0.999291,0.999292,0.999174,0.999165,0.999325,0.999463,0.999479,0.999996,0.999505,1.000000,0.999260,0.999524,0.999296,0.999270,0.999988,0.999983,0.000000

benchmark/test_nj.nwk

@@ -1 +0,0 @@
-(((((((((((Candidozyma_auris--GCF_003013715.1_ASM301371v2:0.5000001881725941,Saccharolobus_islandicus--M.16.4:0.4999993211600824):0.0000023411501775538747,Opitutus_terrae--PB90-1:0.499997075187947):0.0000029791191795691675,(Acidobacterium_capsulatum--ATCC_51196:0.49999227771334689,(Bacillus_subtilis--168:0.49988797935621456,Shouchella_clausii--KSM-K16:0.49988984146059159):0.0001037210285571577):0.0000023959836053522034):0.0000034093646568700288,Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1:0.4999920159222422):0.000199555100890203,Proteus_mirabilis--HI4320:0.49979129185300427):0.00010103619067070024,Yersinia_ruckeri--YRB:0.4996806650749249):0.0013719139155004,(Klebsiella_pneumoniae--HS11286:0.43798845051648258,(Klebsiella_pneumoniae--ATCC_13883:0.41780293826821265,Klebsiella_pneumoniae--MGH_78578:0.42274184870836559):0.017586732339732737):0.0604124197073832):0.0006482538063555254,(Salmonella_enterica--CT18:0.43952894448143017,(Salmonella_enterica--AKU_12601:0.3357977326267918,(Salmonella_enterica--LT2:0.31203395843666389,Salmonella_enterica--P125109:0.31057217324861216):0.025729515856701136):0.10292985918524672):0.05825411485542886):0.08937928015651564,Escherichia_coli--CFT073:0.40806501650701029):0.0410131211869626,Escherichia_coli--EDL933:0.3681464750911808):0.1755112579711463,Escherichia_coli--K-12_MG1655:0.19129818036662728,Escherichia_coli--K-12_W3110:0.19126872019906239);

benchmark/test_shared.csv

@@ -1,21 +0,0 @@
-genome,Candidozyma_auris--GCF_003013715.1_ASM301371v2,Acidobacterium_capsulatum--ATCC_51196,Bacillus_subtilis--168,Escherichia_coli--CFT073,Escherichia_coli--EDL933,Escherichia_coli--K-12_MG1655,Escherichia_coli--K-12_W3110,Klebsiella_pneumoniae--ATCC_13883,Klebsiella_pneumoniae--HS11286,Klebsiella_pneumoniae--MGH_78578,Opitutus_terrae--PB90-1,Proteus_mirabilis--HI4320,Saccharolobus_islandicus--M.16.4,Salmonella_enterica--AKU_12601,Salmonella_enterica--CT18,Salmonella_enterica--LT2,Salmonella_enterica--P125109,Shouchella_clausii--KSM-K16,Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1,Yersinia_ruckeri--YRB
-Candidozyma_auris--GCF_003013715.1_ASM301371v2,0,0,0,0,0,0,0,0,0,0,0,0,8,0,1,0,0,0,0,3
-Acidobacterium_capsulatum--ATCC_51196,0,0,203,119,128,141,140,116,109,111,78,112,0,136,109,147,134,117,55,129
-Bacillus_subtilis--168,0,203,0,124,132,128,123,133,109,130,66,158,6,131,112,124,135,2393,46,124
-Escherichia_coli--CFT073,0,119,124,0,1966777,1998059,1999094,117743,32029,22312,63,4225,0,74946,31918,73311,76585,113,128,7854
-Escherichia_coli--EDL933,0,128,132,1966777,0,2627885,2628700,52488,20134,22064,48,4202,0,74655,28602,71244,74665,112,108,7963
-Escherichia_coli--K-12_MG1655,0,141,128,1998059,2627885,0,4452541,48302,21382,24602,47,4277,0,75729,30449,73622,76778,119,111,8566
-Escherichia_coli--K-12_W3110,0,140,123,1999094,2628700,4452541,0,47894,21226,24470,68,4278,0,75658,30207,73614,76583,112,108,8660
-Klebsiella_pneumoniae--ATCC_13883,0,116,133,117743,52488,48302,47894,0,1416091,1477759,42,4172,0,48296,18988,48144,50416,120,106,7712
-Klebsiella_pneumoniae--HS11286,0,109,109,32029,20134,21382,21226,1416091,0,644063,42,2738,0,21498,29758,21606,21376,99,102,4417
-Klebsiella_pneumoniae--MGH_78578,0,111,130,22312,22064,24602,24470,1477759,644063,0,42,2614,0,19948,35067,21330,20813,97,102,4374
-Opitutus_terrae--PB90-1,0,78,66,63,48,47,68,42,42,42,0,43,18,57,42,53,66,39,58,43
-Proteus_mirabilis--HI4320,0,112,158,4225,4202,4277,4278,4172,2738,2614,43,0,0,4254,2481,4166,4215,131,103,4704
-Saccharolobus_islandicus--M.16.4,8,0,6,0,0,0,0,0,0,0,18,0,0,0,0,0,0,0,0,0
-Salmonella_enterica--AKU_12601,0,136,131,74946,74655,75729,75658,48296,21498,19948,57,4254,0,0,1047731,2857146,2951421,117,108,7643
-Salmonella_enterica--CT18,1,109,112,31918,28602,30449,30207,18988,29758,35067,42,2481,0,1047731,0,917948,940297,106,106,3716
-Salmonella_enterica--LT2,0,147,124,73311,71244,73622,73614,48144,21606,21330,53,4166,0,2857146,917948,0,3284800,122,108,7460
-Salmonella_enterica--P125109,0,134,135,76585,74665,76778,76583,50416,21376,20813,66,4215,0,2951421,940297,3284800,0,134,124,7645
-Shouchella_clausii--KSM-K16,0,117,2393,113,112,119,112,120,99,97,39,131,0,117,106,122,134,0,58,124
-Wolbachia_endosymbiont--GCF_000306885.1_ASM30688v1,0,55,46,128,108,111,108,106,102,102,58,103,0,108,106,108,124,58,0,96
-Yersinia_ruckeri--YRB,3,129,124,7854,7963,8566,8660,7712,4417,4374,43,4704,0,7643,3716,7460,7645,124,96,0