Compare commits
38
Commits
0b40d2d0da
..
v1.3.6
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f29dcbcfb8 | ||
|
|
0d12494f1a | ||
|
|
95bbc0b2ae | ||
|
|
fdfc1eb3dd | ||
|
|
996ebaf977 | ||
|
|
cd06fa2ce3 | ||
|
|
1702addbf9 | ||
|
|
0d1ecac233 | ||
|
|
e715528d34 | ||
|
|
9da0876ee6 | ||
|
|
755cf85334 | ||
|
|
2693a6af09 | ||
|
|
7861e886e7 | ||
|
|
a8bcf5ffac | ||
|
|
be838da256 | ||
|
|
020b391636 | ||
|
|
e846d35adb | ||
|
|
c0e1968654 | ||
|
|
e280b6d0a3 | ||
|
|
98dba1802d | ||
|
|
dd4285b269 | ||
|
|
b66a488277 | ||
|
|
b954032faa | ||
|
|
43d89d8f02 | ||
|
|
8bb1d52514 | ||
|
|
77d4429645 | ||
|
|
4d55258ada | ||
|
|
7e28ca4853 | ||
|
|
52f26809af | ||
|
|
54e4600120 | ||
|
|
579cfd8752 | ||
|
|
6b0c0867cc | ||
|
|
93fe838f97 | ||
|
|
7363370065 | ||
|
|
caf59b5658 | ||
|
|
95fa0c93b2 | ||
|
|
e101f629e6 | ||
|
|
4f34a646c5 |
No files matched your search
+2
-17
@@ -30,23 +30,7 @@ data-stress
|
||||
*.bin.idx
|
||||
*.prsb
|
||||
Betula_exilis--IGA-24-33
|
||||
benchmark/genomes
|
||||
benchmark/genomes_orig
|
||||
benchmark/simulated_data
|
||||
benchmark/specimen_index_presence
|
||||
benchmark/specimen_index_count
|
||||
benchmark/global_index_presence
|
||||
benchmark/global_index_presence_orig
|
||||
benchmark/global_index_presence_sav
|
||||
benchmark/all_specific
|
||||
benchmark/global_index_count
|
||||
benchmark/stats
|
||||
benchmark/reference_index
|
||||
benchmark/reference_dist
|
||||
benchmark/obikmer_dist
|
||||
benchmark/specific_index_count
|
||||
benchmark/specific_index_presence
|
||||
benchmark/query_data
|
||||
benchmark/run/
|
||||
TNT
|
||||
phyg
|
||||
biblio
|
||||
@@ -60,3 +44,4 @@ biblio
|
||||
*.parstree
|
||||
*.ckp.gz
|
||||
*.model
|
||||
scripts/__pycache__/mk_wiki.cpython-314.pyc
|
||||
@@ -1,87 +0,0 @@
|
||||
# Plan d'amélioration technique - obiskio
|
||||
|
||||
## 1. Contexte et objectifs
|
||||
- **Objectif** : Renforcer la robustesse, la maintenabilité et les performances de la crate `obiskio`.
|
||||
- **Priorités** :
|
||||
1. Gestion des erreurs
|
||||
2. Optimisation de la mémoire du pool
|
||||
3. Robustesse concurrente
|
||||
4. Couverture de tests
|
||||
5. Documentation
|
||||
|
||||
---
|
||||
|
||||
## 2. Axes d'amélioration détaillés
|
||||
|
||||
### 2.1 Gestion des erreurs
|
||||
- **Problème** : `SKError` ne couvre pas tous les cas (format invalide, taille maximale, CRC)
|
||||
- **Actions** :
|
||||
- Ajouter variante `ParseError(String)` dans `src/error.rs`
|
||||
- Valider les tailles de SuperKmer avant parsing
|
||||
- Remplacer `expect()` par `unwrap_or_else` avec messages explicites
|
||||
- Documenter chaque variante d’erreur dans le README
|
||||
|
||||
### 2.2 Optimisation du pool de fichiers
|
||||
- **Problème** : `SKFilePool` utilise un `Vec<WriteEntry>` non contraint et n’effectue pas de nettoyage en cas d’erreur
|
||||
- **Actions** :
|
||||
- Implémenter un `LimitedVec` avec limite stricte à `MAX_POOL_SIZE`
|
||||
- Créer `clear_memory()` qui supprime les entrées orphelines
|
||||
- Ajouter `evict_lru_threshold()` pour éviction proactive
|
||||
- Introduire un `RwLock` pour les opérations de lecture massives
|
||||
|
||||
### 2.3 Robustesse concurrente
|
||||
- **Problème** : Risque de deadlocks dans `SKFileWriter::write_batch()` et `SKFileReader::reopen_and_seek()`
|
||||
- **Actions** :
|
||||
- Remplacer `Mutex` par `RwLock` pour les accès en lecture
|
||||
- Ajouter un compteur de blocage et logs de timeout
|
||||
- Utiliser `std::thread::park_timeout` pour débloquer
|
||||
- Insérer `debug_assert!` sur les états invariants
|
||||
|
||||
### 2.4 Couverture de tests
|
||||
- **Problème** : Absence de benchmarks, de tests de migration, de résilience de fichiers corrompus
|
||||
- **Actions** :
|
||||
- Benchmarks I/O sur 10k+ SuperKmer avec `criterion`
|
||||
- Tests de migration de version de fichier `.meta` → `.v2.meta`
|
||||
- Tests de corruption volontaire (truncature, inversion de bits)
|
||||
- Tests de stress sur pool saturation (100 threads)
|
||||
|
||||
### 2.5 Documentation & exemples
|
||||
- **Actions** :
|
||||
- Ajouter des examples dans chaque module (`# Examples`)
|
||||
- Documenter la logique LRU avec diagrammes Mermaid
|
||||
- Créer un guide « How to recover from eviction »
|
||||
- Mettre à jour le `README.md` avec tableau des variantes d’erreur
|
||||
|
||||
---
|
||||
|
||||
## 3. Plan d'exécution (Roadmap)
|
||||
|
||||
| Sprint | Durée | Livrables clés |
|
||||
|--------|-------|----------------|
|
||||
| **S1** | 2 jours | Refactorisation `SKError`, ajout de tests unitaires |
|
||||
| **S2** | 3 jours | Implémentation `clear_memory()` + `LimitedVec` |
|
||||
| **S3** | 2 jours | Passage à `RwLock`, ajout de compteurs de blocage |
|
||||
| **S4** | 2 jours | Benchmarks + tests de migration |
|
||||
| **S5** | 1 jour | Documentation finale & mise à jour du README |
|
||||
|
||||
---
|
||||
|
||||
## 4. Dépendances externes
|
||||
- Mettre à jour `niffler` vers la version 2.0 (performance compression)
|
||||
- Évaluer `bincode` vs `serde_json` pour les métas (I/O)
|
||||
- Ajouter dépendance `criterion` (dev‑dependencies)
|
||||
|
||||
---
|
||||
|
||||
## 5. KPI de suivi
|
||||
- **Couverture de tests** : ≥85 % des chemins critiques
|
||||
- **Latence moyenne d’écriture** : ↓15 % après optimisation du pool
|
||||
- **Taux d’erreurs résolues** : 100 % des nouvelles variantes couvertes
|
||||
- **Temps de build CI** : ≤5 min pour l’ensemble des benchmarks
|
||||
|
||||
---
|
||||
|
||||
## 6. Validation finale
|
||||
- Revue de code avec `cargo clippy -- -D warnings`
|
||||
- Analyse de toxicité avec `cargo deny open-source-licenses`
|
||||
- Vérification de la conformité aux standards de naming du projet
|
||||
@@ -1,2 +0,0 @@
|
||||
/cache
|
||||
/project.local.yml
|
||||
@@ -1,169 +0,0 @@
|
||||
# the name by which the project can be referenced within Serena/when chatting with the LLM.
|
||||
project_name: "obikmer"
|
||||
|
||||
# the encoding used by text files in the project
|
||||
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
|
||||
encoding: "utf-8"
|
||||
|
||||
# line ending convention to use when writing source files.
|
||||
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
|
||||
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
|
||||
line_ending:
|
||||
|
||||
# The language backend to use for this project.
|
||||
# If not set, the global setting from serena_config.yml is used.
|
||||
# Valid values: LSP, JetBrains
|
||||
# Note: the backend is fixed at startup. If a project with a different backend
|
||||
# is activated post-init, an error will be returned.
|
||||
language_backend:
|
||||
|
||||
# whether to use project's .gitignore files to ignore files
|
||||
ignore_all_files_in_gitignore: true
|
||||
|
||||
# advanced configuration option allowing to configure language server-specific options.
|
||||
# Maps the language key to the options.
|
||||
# The settings are considered only if the project is trusted (see global configuration to define trusted projects).
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#language-server-specific-settings
|
||||
ls_specific_settings: {}
|
||||
|
||||
# list of additional paths to ignore in this project.
|
||||
# Same syntax as gitignore, so you can use * and **.
|
||||
# Important: quote patterns that start with `*`, otherwise YAML treats them as aliases.
|
||||
# Example:
|
||||
# ignored_paths:
|
||||
# - "examples/**"
|
||||
# - ".worktrees/**"
|
||||
# - "**/bin/**"
|
||||
# - "**/obj/**"
|
||||
# Note: global ignored_paths from serena_config.yml are also applied additively.
|
||||
ignored_paths: []
|
||||
|
||||
# whether the project is in read-only mode
|
||||
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
|
||||
# Added on 2025-04-18
|
||||
read_only: false
|
||||
|
||||
# list of tool names to exclude.
|
||||
# This extends the existing exclusions (e.g. from the global configuration)
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
excluded_tools: []
|
||||
|
||||
# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
|
||||
# This extends the existing inclusions (e.g. from the global configuration).
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
included_optional_tools: []
|
||||
|
||||
# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
|
||||
# This cannot be combined with non-empty excluded_tools or included_optional_tools.
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
fixed_tools: []
|
||||
|
||||
# list of mode names that are to be activated by default, overriding the setting in the global configuration.
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
|
||||
# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
|
||||
# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
|
||||
# for this project.
|
||||
# This setting can, in turn, be overridden by CLI parameters (--mode).
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
default_modes:
|
||||
|
||||
# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
added_modes:
|
||||
|
||||
# initial prompt for the project. It will always be given to the LLM upon activating the project
|
||||
# (contrary to the memories, which are loaded on demand).
|
||||
initial_prompt: ""
|
||||
|
||||
# time budget (seconds) per tool call for the retrieval of additional symbol information
|
||||
# such as docstrings or parameter information.
|
||||
# This overrides the corresponding setting in the global configuration; see the documentation there.
|
||||
# If null or missing, use the setting from the global configuration.
|
||||
symbol_info_budget:
|
||||
|
||||
# list of regex patterns which, when matched, mark a memory entry as read‑only.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
read_only_memory_patterns: []
|
||||
|
||||
# list of regex patterns for memories to completely ignore.
|
||||
# Matching memories will not appear in list_memories or activate_project output
|
||||
# and cannot be accessed via read_memory or write_memory.
|
||||
# To access ignored memory files, use the read_file tool on the raw file path.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
# Example: ["_archive/.*", "_episodes/.*"]
|
||||
ignored_memory_patterns: []
|
||||
|
||||
# list of additional workspace folder paths for cross-package reference support.
|
||||
# Paths can be absolute or relative to the project root.
|
||||
# Each folder is registered as an LSP workspace folder, enabling language servers to discover
|
||||
# symbols and references across package boundaries, but these folders are not indexed by Serena,
|
||||
# i.e. the respective symbols will not be found using Serena's symbol search tools.
|
||||
# Example:
|
||||
# additional_workspace_folders:
|
||||
# - ../sibling-package
|
||||
# - ../shared-lib
|
||||
ls_additional_workspace_folders: []
|
||||
|
||||
# list of language servers to start when using the LSP backend; choose from:
|
||||
# ada al angular ansible bash
|
||||
# bsl clojure cpp cpp_ccls crystal
|
||||
# csharp csharp_omnisharp cue dart deno
|
||||
# elixir elm erlang fortran fsharp
|
||||
# gdscript gleam go groovy haskell
|
||||
# haxe hlsl html java json
|
||||
# julia kotlin latex lean4 lua
|
||||
# luau markdown matlab msl nextflow
|
||||
# nix ocaml pascal perl php
|
||||
# php_phpactor php_phpantom powershell python python_basedpyright
|
||||
# python_jedi python_pyrefly python_ty qml r
|
||||
# rego ruby ruby_solargraph rust scala
|
||||
# scss solidity svelte swift systemverilog
|
||||
# terraform toml typescript typescript_vts vue
|
||||
# wolfram yaml zig
|
||||
# (This list may be outdated; generated with scripts/print_language_list.py;
|
||||
# For the current list, see values of the LanguageServerId enum here:
|
||||
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
|
||||
# For some languages, there are several alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
|
||||
# Note:
|
||||
# - For C, use cpp
|
||||
# - For JavaScript, use typescript
|
||||
# - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
|
||||
# - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
|
||||
# - For Deno projects, use deno (serves the same .ts/.js files as typescript; requires the deno CLI on PATH)
|
||||
# - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
|
||||
# - For Free Pascal/Lazarus, use pascal
|
||||
# Special requirements:
|
||||
# Some language servers require additional setup/installations.
|
||||
# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
|
||||
# When using multiple language servers, the first language server that supports a given file will be used for that file.
|
||||
# The first language server is the default language and the respective language server will be used as a fallback.
|
||||
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
|
||||
language_servers:
|
||||
- rust
|
||||
|
||||
# list of workspace folder paths (LSP backend only).
|
||||
# These folders will be used to build up Serena's symbol index.
|
||||
# Paths must be within the project root and should thus be relative to the project root.
|
||||
# Furthermore, the paths should not be filtered by ignore settings.
|
||||
# Default setting: The entire project root folder (".") is considered.
|
||||
# In (large) monorepos, this can be used to index only subfolders of the project root, e.g.
|
||||
# ls_workspace_folders:
|
||||
# - "./subproject1"
|
||||
# - "./subproject2"
|
||||
ls_workspace_folders:
|
||||
- .
|
||||
|
||||
# optional shell command to run before the language backend (LSP or JetBrains) is initialised.
|
||||
# the command runs in the project root directory and is only executed if the project is trusted
|
||||
# (see trusted_project_path_patterns in the global configuration).
|
||||
# serena waits for the command to exit: a non-zero exit code is logged as an error but does not
|
||||
# abort activation. a per-project timeout (activation_command_timeout, default 180s) is the safety
|
||||
# backstop for non-terminating commands; on expiry the process is killed and activation continues.
|
||||
# example: activation_command: "npx nx run-many -t build"
|
||||
activation_command:
|
||||
|
||||
# maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
|
||||
# must be a positive number.
|
||||
activation_command_timeout: 180.0
|
||||
@@ -1,59 +0,0 @@
|
||||
// Project tasks configuration. See https://zed.dev/docs/tasks for documentation.
|
||||
//
|
||||
// Example:
|
||||
[
|
||||
{
|
||||
"label": "Example task",
|
||||
"command": "for i in {1..5}; do echo \"Hello $i/5\"; sleep 1; done",
|
||||
//"args": [],
|
||||
// Env overrides for the command, will be appended to the terminal's environment from the settings.
|
||||
"env": { "foo": "bar" },
|
||||
// Current working directory to spawn the command into, defaults to current project root.
|
||||
//"cwd": "/path/to/working/directory",
|
||||
// Whether to use a new terminal tab or reuse the existing one to spawn the process, defaults to `false`.
|
||||
"use_new_terminal": false,
|
||||
// Whether to allow multiple instances of the same task to be run, or rather wait for the existing ones to finish, defaults to `false`.
|
||||
"allow_concurrent_runs": false,
|
||||
// What to do with the terminal pane and tab, after the command was started:
|
||||
// * `always` — always show the task's pane, and focus the corresponding tab in it (default)
|
||||
// * `no_focus` — always show the task's pane, add the task's tab in it, but don't focus it
|
||||
// * `never` — do not alter focus, but still add/reuse the task's tab in its pane
|
||||
"reveal": "always",
|
||||
// Where to place the task's terminal item after starting the task:
|
||||
// * `dock` — in the terminal dock, "regular" terminal items' place (default)
|
||||
// * `center` — in the central pane group, "main" editor area
|
||||
"reveal_target": "dock",
|
||||
// What to do with the terminal pane and tab, after the command had finished:
|
||||
// * `never` — Do nothing when the command finishes (default)
|
||||
// * `always` — always hide the terminal tab, hide the pane also if it was the last tab in it
|
||||
// * `on_success` — hide the terminal tab on task success only, otherwise behaves similar to `always`
|
||||
"hide": "never",
|
||||
// Which shell to use when running a task inside the terminal.
|
||||
// May take 3 values:
|
||||
// 1. (default) Use the system's default terminal configuration in /etc/passwd
|
||||
// "shell": "system"
|
||||
// 2. A program:
|
||||
// "shell": {
|
||||
// "program": "sh"
|
||||
// }
|
||||
// 3. A program with arguments:
|
||||
// "shell": {
|
||||
// "with_arguments": {
|
||||
// "program": "/bin/bash",
|
||||
// "args": ["--login"]
|
||||
// }
|
||||
// }
|
||||
"shell": "system",
|
||||
// Whether to show the task line in the output of the spawned task, defaults to `true`.
|
||||
"show_summary": true,
|
||||
// Whether to show the command line in the output of the spawned task, defaults to `true`.
|
||||
"show_command": true,
|
||||
// Which edited buffers to save before running the task:
|
||||
// * `all` — save all edited buffers
|
||||
// * `current` — save currently active buffer only
|
||||
// * `none` — don't save any buffers
|
||||
"save": "none",
|
||||
// Represents the tags for inline runnable indicators, or spawning multiple tasks at once.
|
||||
// "tags": []
|
||||
},
|
||||
]
|
||||
+1
-1
@@ -1119,7 +1119,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1370,7 +1370,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1128,7 +1128,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1367,6 +1367,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Panic propagation (fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1553,6 +1564,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Panic propagation (fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1836,11 +1858,27 @@ dormant workers have been woken and grows every node by the same amount per
|
||||
step, capped by that node's remaining dormant workers and by the run's total
|
||||
budget (<code>n_total</code>) — balance across nodes is now guaranteed by construction,
|
||||
not incidental to channel implementation details.</p>
|
||||
<h2 id="panic-propagation-fixed-2026-08-28">Panic propagation (fixed, 2026-08-28)</h2>
|
||||
<p>A worker whose <code>f(i)</code> panicked used to never send its <code>WorkerEvent::Completed</code>
|
||||
— the controller's <code>while completed < n_total</code> loop then waited forever for
|
||||
an event that partition could no longer produce, since other live
|
||||
workers/timer threads kept <code>event_rx</code> open. Discovered via <code>obikselect</code>
|
||||
panicking on an unimplemented <code>Sparse</code> matrix case (see <code>implementation/select.md</code>):
|
||||
the process hung indefinitely instead of erroring out.</p>
|
||||
<p>Fixed: each worker wraps <code>f(i)</code> in <code>std::panic::catch_unwind</code> and sends a new
|
||||
<code>WorkerEvent::Panicked(i, payload)</code> instead of silently dropping the
|
||||
partition. The controller counts it toward <code>completed</code> (unblocking the loop)
|
||||
and keeps the first payload seen; once <code>run</code> returns, <code>panic::resume_unwind</code>
|
||||
re-raises it on the caller's thread — the original message/backtrace still
|
||||
surfaces, from the right place, instead of a silent deadlock. Takes priority
|
||||
over a plain <code>Err</code> (a panic means a bug, not a normal typed failure).</p>
|
||||
<h2 id="open-questions">Open questions</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Error handling</strong>: <code>run</code> currently returns the first error; remaining errors
|
||||
are dropped. A <code>Vec<E></code> return would give complete diagnostics.</p>
|
||||
are dropped. A <code>Vec<E></code> return would give complete diagnostics. (Panics are
|
||||
now caught and re-raised individually — see above — this only concerns
|
||||
plain <code>Err</code> results.)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong><code>INITIAL_DIVISOR</code> / <code>GROWTH_DIVISOR</code> tuning</strong>: currently <code>4</code> and <code>8</code>
|
||||
|
||||
@@ -1130,7 +1130,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1140,7 +1140,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+179
-1120
File diff suppressed because it is too large.
Load diff
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1485,8 +1485,9 @@ this pipeline.</p>
|
||||
by <code>verify_presence</code>/<code>verify_merge_presence</code>, which only check index
|
||||
<em>content</em> against the <code>.npz</code> truth, never the query API.</li>
|
||||
<li><code>pack --sparse</code> produces a presence-matrix format documented (see
|
||||
<a href="../../architecture/siblings/">siblings.md</a>) as faster for single-row
|
||||
access (query) and slower for column-oriented access (phylo <code>--metric</code>).
|
||||
<a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats") as faster
|
||||
for single-row access (query) and slower for column-oriented access
|
||||
(phylo <code>--metric</code>).
|
||||
<code>global_index_presence/</code> built by <code>merge_presence.sh</code> is always packed
|
||||
dense (packing is a stage inside <code>merge</code>, not a separate <code>pack</code>
|
||||
invocation) — there was no dense/sparse regression check.</li>
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -579,7 +579,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1388,7 +1388,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1504,7 +1504,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -2208,7 +2208,7 @@
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">>></span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">></span><span class="p">])</span>
|
||||
</code></pre></div>
|
||||
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this degrades to a row-by-row decode; see <a href="../../architecture/siblings/">siblings.md</a>.</p>
|
||||
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this goes through <code>nonzero_iter</code> instead, decoding one row at a time; see <a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats".</p>
|
||||
<hr />
|
||||
<h2 id="layeredstores-and-aggregation-traits">LayeredStore\<S> and aggregation traits</h2>
|
||||
<p><code>LayeredStore<S></code> is a generic aggregation wrapper over <code>Vec<S></code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1034,6 +1034,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known issue (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1337,7 +1348,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1598,6 +1609,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known issue (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1795,6 +1817,14 @@ spectrums/
|
||||
<label>.json ← one file per genome, rebuilt from all sources
|
||||
index.meta ← complete genome list + evidence kind written at bootstrap
|
||||
</code></pre></div>
|
||||
<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
|
||||
<h2 id="known-issue-not-yet-fixed-2026-08-28">Known issue (not yet fixed, 2026-08-28)</h2>
|
||||
<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank < self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
|
||||
<ul>
|
||||
<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
|
||||
<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
|
||||
</ul>
|
||||
<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1332,7 +1332,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1343,7 +1343,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -494,7 +494,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -632,6 +632,46 @@
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#sparse-matrix-formats">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sparse matrix formats
|
||||
|
||||
</span>
|
||||
</a>
|
||||
<nav aria-label="Sparse matrix formats" class="md-nav">
|
||||
<ul class="md-nav__list">
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#persistentsparsebitmatrix">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentSparseBitMatrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#nonzero_iter-the-one-traversal-per-format">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
nonzero_iter — the one traversal per format
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#persistentsparsecompactintmatrix">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentSparseCompactIntMatrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#aggregation-traits-matrix-level">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
@@ -1168,7 +1208,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</code></pre></div>
|
||||
<hr/>
|
||||
<h2 id="matrix-types">Matrix types</h2>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has two variants (<code>Columnar</code>, <code>Packed</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -1185,13 +1225,13 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Packed</code></td>
|
||||
<td>single <code>matrix.pbmx</code> mmap file</td>
|
||||
<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
|
||||
<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Sparse</code> (bit only)</td>
|
||||
<td><code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major</td>
|
||||
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
|
||||
<td><code>Sparse</code></td>
|
||||
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
||||
<td><code>pack --sparse</code>; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Implicit</code> (bit only)</td>
|
||||
@@ -1200,7 +1240,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||
@@ -1211,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
|
||||
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
|
||||
<hr/>
|
||||
<h2 id="sparse-matrix-formats">Sparse matrix formats</h2>
|
||||
<p>Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||
k-mers — present in only a handful of genome columns out of dozens to
|
||||
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||
formats above, built once per layer like the rest of the pipeline (never the
|
||||
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||
Structures for Collections of Sets" (2025,
|
||||
<code>biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf</code>) — a plain dedup dictionary, not their full subset-containment
|
||||
hierarchy.</p>
|
||||
<h3 id="persistentsparsebitmatrix"><code>PersistentSparseBitMatrix</code></h3>
|
||||
<p>Four mmap-backed on-disk components: an <code>is_multi</code> rank-capable flag per row
|
||||
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||
(genome index directly, <code>ceil(log2(n_cols))</code> bits), a separate fixed-width
|
||||
array for multi-genome rows (<code>dict_id</code>, <code>ceil(log2(n_distinct_multi_sets))</code>
|
||||
bits — kept apart from the singleton array because <code>n_distinct_multi_sets</code>
|
||||
can be large in absolute terms even when multi-genome rows are a small
|
||||
<em>fraction</em> of all rows, and a shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||
blob). Supporting primitives: <code>PersistentFixedIntVec</code> (arbitrary,
|
||||
runtime-parameterised bit width, width <code>0</code> included — needed once an
|
||||
<code>EliasFano</code> reopen bug surfaced, see below), <code>PersistentRankSelectBitVec</code>
|
||||
(rank1/rank0/select1 on top of the crate's existing <code>count_ones</code>, via
|
||||
<code>common_traits::SelectInWord</code>), <code>EliasFano</code> (composes the two). <code>BinaryMatrix</code>
|
||||
(<code>n</code>, <code>n_cols</code>, <code>row</code>/<code>fill_row</code>, <code>fill_sub_matrix</code>, <code>count_ones</code>) is the
|
||||
trait unifying dense and sparse at the call sites that need both
|
||||
interchangeably; column-oriented methods (<code>col</code>, <code>col_view</code>, the
|
||||
<code>partial_*_dist_matrix</code> family) stay dense-only — <code>Sparse</code>/<code>Implicit</code> panic
|
||||
there (see the <code>PersistentBitMatrix::open</code> note above).</p>
|
||||
<p>Measured on real data (one layer, 30.2M rows, 91 genomes):</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th></th>
|
||||
<th>dense</th>
|
||||
<th>sparse</th>
|
||||
<th>ratio</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>on-disk size</td>
|
||||
<td>328.1MB</td>
|
||||
<td>43.7MB</td>
|
||||
<td><strong>7.5x</strong> smaller</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>build time / peak RSS</td>
|
||||
<td>—</td>
|
||||
<td>4.26s / 628MB</td>
|
||||
<td>per-layer, in-memory construction</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>row access, sequential (2M reads)</td>
|
||||
<td>43ns/row</td>
|
||||
<td>32ns/row</td>
|
||||
<td>sparse faster (smaller structure, better cache fit)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>row access, random (2M reads)</td>
|
||||
<td>409ns/row</td>
|
||||
<td>85ns/row</td>
|
||||
<td>sparse <strong>~4.8x faster</strong> — the <code>--shannon</code>/family-lookup access shape</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>column access, one full column (30.2M rows)</td>
|
||||
<td>11.5ms</td>
|
||||
<td>993ms</td>
|
||||
<td>sparse <strong>86x slower</strong> — no native column method, every read decodes a full row to keep one bit</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>The row-access wins (compactness was the stated design goal, not speed) come
|
||||
from dense's genome-major layout scattering a single row read across a much
|
||||
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||
is this much smaller. Column access is the flip side of the same layout
|
||||
choice; <code>nonzero_iter</code> (below) is what removes it from the hot path that
|
||||
actually matters (<code>query</code>).</p>
|
||||
<p>Two reopen-correctness bugs, now fixed at the source: (1) <code>EliasFano::open</code>
|
||||
re-derived its low-bits width from the persisted low-vector's own width
|
||||
byte; the width-<code>0</code> case was built with a dummy 1-bit placeholder (the
|
||||
builder used to reject true width <code>0</code>), silently doubling every reopened
|
||||
value. Fixed by making <code>PersistentFixedIntVec</code> genuinely support width <code>0</code>
|
||||
(no storage, <code>get</code> always <code>0</code>). (2) An empty row (cardinality 0) used to be
|
||||
recorded as a singleton at genome <code>0</code>, indistinguishable on read-back from a
|
||||
<em>real</em> singleton at genome <code>0</code>. Fixed by routing cardinality-0 rows through
|
||||
the dictionary path (a genuine empty entry).</p>
|
||||
<p>Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||
exact-duplicate case is built), a sparse <code>PersistentCompactIntMatrix</code>
|
||||
variant beyond what's described below, BRWT-style column-correlation
|
||||
exploitation, and a row-major rewrite of
|
||||
<code>partial_jaccard_dist_matrix</code>/<code>partial_hamming_dist_matrix</code>/etc. as a
|
||||
per-row co-occurrence accumulation (<code>O(Σ_rows k̄²)</code>) so <code>obikindex</code>'s
|
||||
<code>--metric</code> distance-matrix path stops paying the column-access penalty on
|
||||
<code>Sparse</code> — needs its own design pass (how it plugs into
|
||||
<code>BitPartials</code>/<code>ColumnWeights</code> so both matrix types keep serving <code>--metric</code>),
|
||||
not just "port the loop".</p>
|
||||
<h3 id="nonzero_iter-the-one-traversal-per-format"><code>nonzero_iter</code> — the one traversal per format</h3>
|
||||
<p><code>fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a></code>
|
||||
(item: index into <code>slots</code>, column, value) is the single foundational
|
||||
per-format traversal primitive every consumer builds on — <code>fill_sub_matrix</code>
|
||||
and a <code>for_each</code>-style callback are both trivial, free consumers of it
|
||||
(<code>.for_each(f)</code> is <code>Iterator::for_each</code>; <code>fill_sub_matrix</code> drains the
|
||||
iterator and scatters into <code>out[][]</code>), instead of two independently
|
||||
hand-maintained traversals that can silently diverge. <code>Box<dyn Iterator></code>,
|
||||
not a bare <code>impl Iterator</code>: <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are
|
||||
genuinely different concrete types and the method isn't on a
|
||||
dyn-compatible trait (<code>BinaryMatrix</code> is used as <code>dyn BinaryMatrix</code>
|
||||
elsewhere, and RPITIT methods aren't dyn-compatible) — one <code>Box</code> per
|
||||
<code>nonzero_iter</code> call, not per cell.</p>
|
||||
<p>Per-format implementation: <code>PersistentSparseBitMatrix</code> decodes natively,
|
||||
buffered one row at a time (<code>std::iter::from_fn</code> over the existing
|
||||
<code>for_each_genome_in_row</code>, no <code>n_cols</code>-wide allocation ever). <code>Columnar</code>/
|
||||
<code>Packed</code> loop columns, collecting each column's hits via
|
||||
<code>BitSliceView::nonzero_among_sorted</code>/<code>IntSliceView::nonzero_among_sorted</code>
|
||||
(a <code>filter</code>/<code>filter_map</code> over the vector-level <code>get()</code>, no new state
|
||||
machine — <code>std</code>'s own <code>Filter</code> iterator already is one) into one <code>Vec</code> via
|
||||
<code>.extend()</code> (not <code>flat_map</code>: a <code>flat_map</code> closure can't lazily return
|
||||
something borrowing its own captured sort permutation across separate calls
|
||||
without either boxing per-column or fighting the borrow checker — eager
|
||||
collection sidesteps it at zero extra cost, since <code>fill_sub_matrix</code> already
|
||||
fully materialises anyway). <code>Implicit</code> is trivial (<code>slots.iter().map(|&i| (i, 0, 1))</code>).</p>
|
||||
<p>This closed a real dispatch bug by construction, not by patching it:
|
||||
<code>PersistentBitMatrix::fill_sub_matrix</code>/<code>sub_matrix</code> used to reimplement a
|
||||
naive per-<code>(column, slot)</code> loop instead of delegating to
|
||||
<code>PersistentSparseBitMatrix</code>'s own efficient native method for <code>Self::Sparse</code>
|
||||
— two hand-written traversals for the same format, free to drift apart, and
|
||||
they had (<code>obikphylo</code>'s own cache layer had independently built a parallel
|
||||
enum wrapper that <em>did</em> dispatch correctly, evidence the bug was worked
|
||||
around rather than fixed at the source). With <code>fill_sub_matrix</code>/<code>sub_matrix</code>
|
||||
rewritten to drain <code>nonzero_iter</code>, there is only one traversal per format
|
||||
left to get right. <code>KmerPartition::query_partition_with</code>
|
||||
(<code>obikindex/src/partition/query_layer.rs</code>) — previously column-major
|
||||
(<code>for g { for slot { col_value } }</code>, which for <code>Sparse</code> rebuilt a full row
|
||||
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||
motivation for this primitive) — now issues one <code>layer.nonzero_iter(&slot_list)</code>
|
||||
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||
reproducibly; now within ~1-3% either way — noise level).</p>
|
||||
<h3 id="persistentsparsecompactintmatrix"><code>PersistentSparseCompactIntMatrix</code></h3>
|
||||
<p>The count-matrix analogue, built on <code>PersistentSparseBitMatrix</code> as its
|
||||
"which columns are non-zero" support, values <strong>not</strong> deduplicated (unlike
|
||||
the bit side, two rows can share the same non-zero column set — same
|
||||
<code>dict_id</code> — while carrying different counts, so the dict-multiplicity
|
||||
shortcut <code>BitPartials for PersistentSparseBitMatrix</code> uses doesn't carry
|
||||
over). <code>CountPartials</code> here: a single row-major pass (<code>row_major_pairwise</code>
|
||||
in <code>sparse_intmatrix.rs</code>, decodes each row once via
|
||||
<code>for_each_cell_in_row</code>, nests over that row's own co-present columns) —
|
||||
<code>O(Σ k̄²)</code> over populated rows, avoiding the naive <code>O(n_cols² × n)</code>
|
||||
column-pair rescan. <code>min(a,b)</code>-based kernels (<code>bray</code>, <code>relfreq_bray</code>) need
|
||||
no correction — they vanish when either side is absent. <code>a·b</code> and <code>√(a·b)</code>
|
||||
kernels (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) do need one, since
|
||||
<code>(a-0)² = a² ≠ 0</code> unlike the <code>min</code>-based formulas — reconstructed from
|
||||
per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>.
|
||||
<code>threshold_jaccard(1)</code> shortcuts to <code>BitPartials::partial_jaccard</code>
|
||||
(threshold 1 is exactly presence); <code>threshold_jaccard(0)</code> is closed-form
|
||||
(every <code>u32</code> is <code>≥ 0</code>).</p>
|
||||
<p>Wiring the <code>threshold==1</code> shortcut surfaced two pre-existing formula bugs in
|
||||
<code>BitPartials for PersistentSparseBitMatrix</code> (present since that type's own
|
||||
introduction, never caught because no test had compared <code>Sparse</code>'s raw
|
||||
<code>partial_*</code> output against dense on real data — only the diagonal-blind
|
||||
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested),
|
||||
fixed at the source:</p>
|
||||
<ol>
|
||||
<li><code>partial_jaccard</code>'s diagonal used to be <code>(0, 2×col_weights[i])</code> instead
|
||||
of a genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
|
||||
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with itself
|
||||
by construction, so the diagonal needs its own case.</li>
|
||||
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: it computed
|
||||
<code>total - union</code> (rows where <em>neither</em> column is present) instead of the
|
||||
actual Hamming distance <code>col_weights[i] + col_weights[j] - 2×inter[i,j]</code>
|
||||
(symmetric-difference size) — the two only coincide when
|
||||
<code>col_weights[i] + col_weights[j] == total</code>, which small/synthetic test
|
||||
data can easily satisfy by accident.</li>
|
||||
</ol>
|
||||
<h2 id="aggregation-traits-matrix-level">Aggregation traits (matrix level)</h2>
|
||||
<h3 id="columnweights">ColumnWeights</h3>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">trait</span><span class="w"> </span><span class="n">ColumnWeights</span><span class="p">:</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="p">{</span>
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1293,7 +1293,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1305,7 +1305,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1213,6 +1213,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1447,7 +1563,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1719,6 +1835,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1853,6 +2085,21 @@ The underlying module boundary and its rationale (Layer tier / Partition
|
||||
tier / Index tier, each depending only downward) are unchanged; only the
|
||||
crate-vs-module packaging changed. See <a href="../layer_tier/">obikindex::layer</a>
|
||||
for the current module doc.</p>
|
||||
<p><strong>Superseded, second event, same day (2026-08-21):</strong> <code>obikpartitionner</code>
|
||||
and <code>obikderep</code>, the two algorithm crates, are also gone — but unlike
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> above, they were <strong>not</strong> folded into
|
||||
<code>obikindex</code>. They were first (mistakenly) merged into <code>obikindex</code> as an
|
||||
<code>algorithms</code> submodule, then corrected into a new sibling crate,
|
||||
<strong><code>obikindexer</code></strong>, holding <code>obikindexer::algorithms::{partitionner,
|
||||
dereplicator}</code> and depending on <code>obikindex</code> — never the reverse, same
|
||||
dependency direction <code>obikpartitionner</code>/<code>obikderep</code> already had. Read
|
||||
<code>obikpartitionner::X</code> as <code>obikindexer::algorithms::partitionner::X</code> and
|
||||
<code>obikderep::X</code> as <code>obikindexer::algorithms::dereplicator::X</code> throughout
|
||||
what follows. The distinction the mistake surfaced, worth keeping: data
|
||||
crates (<code>obikindex</code>, holding the <code>index</code>/<code>partition</code>/<code>layer</code> model) merge
|
||||
naturally into one crate as submodules; algorithm crates that operate on
|
||||
that model from outside stay separate, so the dependency only ever runs
|
||||
one way.</p>
|
||||
<p>Status (2026-08-20, latest pass): (1) done — <code>obilayeredmap::Layer</code>
|
||||
exists, <code>Mat</code> is gone. (1b) done — <code>Layer::Empty</code>, the first non-ready
|
||||
state, added (panics on every read method). (2a) done — the
|
||||
@@ -1872,7 +2119,24 @@ turned out to be bigger than <code>KmerPartition</code> alone: <code>Layer</code
|
||||
constructors don't self-name either. Full redesign of both, agreed in
|
||||
detail, session ended (budget) before implementation — see "(5) design
|
||||
agreed" below; <strong>read it before touching <code>KmerPartition</code>/<code>Layer</code>
|
||||
signatures</strong>, the shape is fully specified. Earlier mix-up, for
|
||||
signatures</strong>, the shape is fully specified. (6) done — <code>Counter</code>, a third
|
||||
algorithm, extracted from <code>PartitionRouter</code> the same way <code>Dereplicator</code>
|
||||
was in (4). (7) done — <code>LayerBuilder</code>, the fourth and last pipeline
|
||||
algorithm; the indexing pipeline is now fully decomposed into
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator, counter,
|
||||
layer_builder}</code>. (8) design agreed, item 1 done in (9) —
|
||||
<code>obikindexer::extensions::PrivateBuilder</code>, private, six construction-only
|
||||
<code>KmerIndex</code> methods moved out. (10) done, same session — <code>obikindex::
|
||||
IndexBuilder</code>, public, the four maintenance methods
|
||||
(<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>)
|
||||
shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>. Item 2 from (8)
|
||||
(<code>obikalgorithm::Algorithm</code>) done in (12) — new crate, <code>type Output</code> +
|
||||
<code>fn run(&mut self) -> SKResult<Self::Output></code>, <code>on_progress</code> moved off
|
||||
<code>run()</code>'s signature entirely into a per-algorithm <code>.on_progress(...)</code>
|
||||
setter. Note: <code>Layer</code> renamed
|
||||
<code>KmerLayer</code> (2026-08-21, outside this conversation). (5) itself still not
|
||||
implemented, still first on the "order of remaining work" list. Earlier
|
||||
mix-up, for
|
||||
context: an earlier
|
||||
version of this doc used the name <code>KmerPartition</code> (singular) for what was
|
||||
actually the <em>collection</em> type (later renamed <code>KmerPartitions</code>, later
|
||||
@@ -2515,6 +2779,622 @@ itself or a new crate.</p>
|
||||
examples were judged not enough to be sure of the shape (<code>Fn+Sync</code> vs
|
||||
<code>FnMut</code> callback bound already diverged between the two that exist).</li>
|
||||
</ol>
|
||||
<h2 id="6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator">(6) done (2026-08-21): <code>Counter</code> — third algorithm, extracted the same way as <code>Dereplicator</code></h2>
|
||||
<p>Between (5) and this, the user did a session of their own crate
|
||||
restructuring (see the two "Superseded" notes at the top of this file):
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> folded into <code>obikindex</code> as submodules
|
||||
(<code>obikindex::partition</code>, <code>obikindex::layer</code>), and <code>obikpartitionner</code>/
|
||||
<code>obikderep</code> merged into one sibling crate, <code>obikindexer</code>, holding
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator}</code>. (5)'s design
|
||||
(<code>Layer</code>/<code>KmerPartition</code> self-naming by number, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code> → <code>&</code> fix) was <strong>not</strong> part of that — pure crate/module packaging,
|
||||
confirmed by reading the actual code (<code>Layer::open</code>/<code>create</code> still take an
|
||||
external <code>dir: &Path</code>, <code>KmerPartition</code> still eagerly opens all layers,
|
||||
<code>PartitionRouter</code> still holds <code>&mut KmerIndex</code>). (5) remains exactly as
|
||||
specified, not yet implemented.</p>
|
||||
<p>This step: <code>count_kmer</code> (still living on <code>PartitionRouter</code>, per (4)'s own
|
||||
"still not done" note) extracted into <code>obikindexer::algorithms::counter::
|
||||
Counter</code>, mirroring <code>Dereplicator</code> exactly — third data point for the
|
||||
eventual <code>obikalgorithm</code> trait, still not extracted (still only 3 examples
|
||||
with 2 different callback bounds; holding off per (5)'s "order of
|
||||
remaining work").</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_partial</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_partial</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span><span class="w"> </span><span class="c1">// setter, mirrors PartitionRouter's style; defaults to false</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="n">KmerSpectrum</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Same shape as <code>Dereplicator</code> throughout: <code>Fn(Progress) + Sync</code> (not
|
||||
<code>FnMut</code>) since counting is also a parallel <code>par_iter</code> over partitions, an
|
||||
<code>AtomicU64</code> position counter incremented from inside the parallel closure
|
||||
so progress reports arrive in real time rather than bursting at the end
|
||||
once <code>.collect()</code> finishes, <code>total: Some(n_partitions)</code> (known up front).
|
||||
<code>KmerSpectrum</code> (the <code>{f0, f1, counts}</code> aggregate) moved from
|
||||
<code>partitionner::router</code> to <code>counter</code>, since it's <code>Counter::run</code>'s return
|
||||
value now, not <code>PartitionRouter</code>'s. <code>count.rs</code>/<code>kmer_sort.rs</code> moved
|
||||
verbatim from <code>partitionner/</code> to <code>counter/</code> (unchanged bodies — only
|
||||
<code>count_kmer</code> itself, <code>KmerSpectrum</code>, and the imports they pulled in were
|
||||
removed from <code>router.rs</code>).</p>
|
||||
<p>One divergence from <code>Dereplicator</code>: a <code>keep_partial</code> setter exists (no
|
||||
equivalent on <code>Dereplicator</code>, which has no setters at all) — a real,
|
||||
already-present parameter (<code>keep_intermediate</code> at the CLI), not a
|
||||
speculative addition.</p>
|
||||
<p><code>count_kmer</code>'s three former callers (<code>obikmer::cmd::index</code>, <code>obikphylo</code>'s
|
||||
test harness, <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test helper) all updated to <code>Counter::new(&idx).
|
||||
run(...)</code> — the last one simplified further: it used to read back
|
||||
<code>kmer_spectrum_raw.json</code> from disk after calling <code>count_partition</code>
|
||||
directly (white-box), now it just uses the <code>KmerSpectrum</code> <code>Counter::run</code>
|
||||
already returns.</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus an end-to-end CLI smoke test
|
||||
against real FASTA data (scatter → dereplicate → count → index-build →
|
||||
query) — required every time per (3)'s lesson, and it earned its keep
|
||||
again: the very first smoke-test query returned zero matches, which
|
||||
looked like a regression until traced to the query sequence itself being
|
||||
low-complexity ("GGCCCCCCACG", six same-base runs) and rejected by
|
||||
<em>query's own</em> default entropy threshold — nothing to do with this change.
|
||||
Re-tested with a different substring, confirmed working (kmer found,
|
||||
count matched the index).</p>
|
||||
<p>Still not done: (5) (<code>Layer</code>/<code>KmerPartition</code> redesign, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code>→<code>&</code>), the future cache crate, <code>build_layers</code> (still a <code>KmerIndex</code>
|
||||
inherent method, not an algorithm), and <code>obikalgorithm</code> itself.</p>
|
||||
<h2 id="7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm">(7) done (2026-08-21): <code>LayerBuilder</code> — fourth and last pipeline algorithm</h2>
|
||||
<p>Closes out the indexing pipeline: <code>build_layers</code>/<code>build_index_layer</code>
|
||||
(the last stage still living as <code>KmerIndex</code> inherent methods, flagged as
|
||||
inconsistent since (6)) extracted into <code>obikindexer::algorithms::
|
||||
layer_builder::LayerBuilder</code>, same two-phase shape as the other three.</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">min_abundance</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">max_abundance</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_intermediate</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">min_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">max_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_intermediate</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">FnMut</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="p">;</span><span class="w"> </span><span class="c1">// returns total kmers built</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p><strong>Different from all three prior extractions in one respect, deliberately
|
||||
not "fixed" to match them</strong>: the actual per-partition construction logic
|
||||
(De Bruijn graph from dereplicated superkmers + provisional counts →
|
||||
unitigs → MPHF → matrix) stayed put as <code>KmerIndex::build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> (both already <code>pub</code>) — not moved into
|
||||
<code>obikindexer</code>. Checked first: unlike <code>dereplicate_partition</code>/
|
||||
<code>count_partition</code> (which only ever had one caller), <code>build_index_layer</code>
|
||||
depends on several <code>obikindex</code>-internal helpers (<code>graph_pipeline::
|
||||
{write_graph_as_unitigs, materialize_layer}</code>, <code>common::olm_to_sk</code>) that
|
||||
are <code>pub(crate)</code> and shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>'s own
|
||||
layer-construction paths — moving <code>build_index_layer</code> out would have
|
||||
meant either exporting that internal surface just for this one algorithm
|
||||
or duplicating it. Neither was needed: <code>build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> were <em>already</em> public <code>KmerIndex</code> methods, so
|
||||
<code>LayerBuilder</code>'s job is purely the orchestration around them (scheduling,
|
||||
config, progress) — the exact same "algorithm calls already-public
|
||||
<code>KmerIndex</code> primitives" shape <code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>
|
||||
already have, just at a coarser grain for this one stage. This is the
|
||||
"is the producer's API actually deficient?" check from
|
||||
[[feedback_no_spaghetti_petits_pois]] applied and answered "no" — not
|
||||
skipped.</p>
|
||||
<p><strong>Two more real divergences, both forced by <code>PartitionRunner</code>, not
|
||||
arbitrary:</strong>
|
||||
- Uses <code>obikindex::PartitionRunner</code> (NUMA-aware scheduler, already
|
||||
<code>pub use</code>d from <code>obikindex</code>) instead of plain <code>rayon::into_par_iter</code>
|
||||
like <code>Dereplicator</code>/<code>Counter</code> — matches what <code>build_layers</code> already used
|
||||
before extraction; this stage is more CPU/memory-intensive per partition
|
||||
(graph construction) than scatter/dereplicate/count.
|
||||
- Callback bound is <code>FnMut(Progress) + Send</code> — a third variant, not
|
||||
matching either prior shape. <code>PartitionRunner::run</code>'s <code>on_done</code> is
|
||||
invoked from its own single controller thread (never concurrently, so
|
||||
no <code>Sync</code> needed, unlike <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn + Sync</code>), but
|
||||
that controller thread is itself <code>std::thread::scope</code>-spawned, so the
|
||||
closure still has to be <code>Send</code> to cross into it — caught immediately by
|
||||
the compiler (<code>cannot be sent between threads safely</code>) when <code>Send</code> was
|
||||
first omitted, not a design guess. <code>obikalgorithm</code>'s eventual shared
|
||||
trait now has three real callback-bound data points to reconcile
|
||||
(<code>FnMut</code> alone for <code>PartitionRouter::run</code>'s sequential loop, <code>FnMut +
|
||||
Send</code> here, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code>'s <code>rayon</code>
|
||||
<code>par_iter</code>), not two.</p>
|
||||
<p><code>KmerIndex::build_layers</code> deleted outright (<code>KmerIndex</code> stays a pure data
|
||||
structure — no compute orchestration methods, consistent with <code>dereplicate</code>/
|
||||
<code>count_kmer</code>'s removal in (4)/(6)). New <code>KmerIndex::mark_indexed()</code> added,
|
||||
symmetric to <code>mark_scattered</code>/<code>mark_counted</code>, replacing the inline
|
||||
<code>touch(SENTINEL_INDEXED)</code> that used to live inside <code>build_layers</code>.
|
||||
<code>Stage::start("index")</code>/<code>rep.push(...)</code> and the <code>progress_bar</code>/
|
||||
<code>"{n} total kmers indexed"</code> log line both moved to <code>cmd/index/mod.rs</code>,
|
||||
same pattern as (3)/(4)/(6) — <code>LayerBuilder</code> renders nothing itself, just
|
||||
reports <code>Progress</code>.</p>
|
||||
<p>All callers updated: <code>cmd/index/mod.rs</code> (Stage 3), <code>obikphylo</code>'s test
|
||||
harness (also gained a <code>mark_indexed()</code> call it was missing — harmless
|
||||
before since nothing checked <code>IndexState::Indexed</code> in that test, but now
|
||||
correct).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the end-to-end CLI smoke test
|
||||
(<code>scripts/smoke_test_index.sh</code>, built earlier specifically so this
|
||||
verification step is a one-liner from now on) — 870 kmers indexed, query
|
||||
round-trip confirmed, same numbers as (6).</p>
|
||||
<p><strong>The indexing pipeline is now fully decomposed</strong>: <code>obikindexer::
|
||||
algorithms::{partitionner, dereplicator, counter, layer_builder}</code>, each a
|
||||
<code>new</code>/(setters)/<code>run</code> algorithm operating on a <code>&KmerIndex</code> (or <code>&mut</code> for
|
||||
<code>PartitionRouter</code>, not yet fixed — see (5)), <code>KmerIndex</code> itself holding no
|
||||
pipeline-orchestration logic anymore. Still not done: (5), the future
|
||||
cache crate, <code>obikalgorithm</code> (now unblocked — three real callback-bound
|
||||
variants observed, worth revisiting whether a single trait can express
|
||||
all three or whether that's itself the answer: it can't, and the trait
|
||||
should not force it).</p>
|
||||
<h2 id="8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename">(8) design agreed, not yet implemented (2026-08-21): <code>obikalgorithm::Algorithm</code> trait + <code>obikindexer::extensions</code> — private/public extension-trait split, <code>KmerLayer</code> rename</h2>
|
||||
<p>Session note: <code>Layer</code> was renamed <code>KmerLayer</code> (user, outside this
|
||||
conversation, alongside other naming homogenisation with <code>KmerIndex</code>/
|
||||
<code>KmerPartition</code>) — every reference to <code>Layer</code> in this doc from before
|
||||
2026-08-21 means today's <code>obikindex::layer::KmerLayer</code>.</p>
|
||||
<h3 id="why-this-came-up">Why this came up</h3>
|
||||
<p>Verifying "does <code>cmd/index</code> now rest entirely on the algorithm structs"
|
||||
(it doesn't quite — see below) led to sorting <code>KmerIndex</code>'s own methods by
|
||||
a criterion the user was explicit is <strong>semantic, not mechanical</strong>: "les
|
||||
méthodes qui, sémantiquement, n'ont pas d'intérêt hors de la construction
|
||||
de l'index" (methods that have no semantic interest outside index
|
||||
construction) — not "methods only called from <code>cmd/index</code> today," which
|
||||
a grep could answer but would miss methods construction-adjacent code
|
||||
elsewhere (<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>) also depends on for the
|
||||
same reason.</p>
|
||||
<p><strong>Checked, not assumed</strong> (grepped every call site before classifying):</p>
|
||||
<ul>
|
||||
<li><strong>Construction-only, real candidates for a private extension trait</strong>:
|
||||
<code>KmerIndex::{mark_scattered, mark_counted, mark_indexed, write_spectrum,
|
||||
build_index_layer, remove_build_artifacts, clear_output_for_create,
|
||||
create_skeleton, finalize_indexed, state}</code>. The last four are called
|
||||
from <code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> too (as
|
||||
precondition checks — "is my source <code>Indexed</code>?" — or shared
|
||||
skeleton/finalize machinery), not just from the 4-stage pipeline — so
|
||||
this extension trait's scope is "construction of any kind," not
|
||||
narrowly "the initial build pipeline."</li>
|
||||
<li><strong>Looked construction-only by name, checked, and kept on <code>KmerIndex</code></strong>:
|
||||
<code>layer_unitigs_path</code> (unitigs are the only way to recover a built
|
||||
index's kmer sequences — read by <code>rebuild_layer.rs</code> and others, well
|
||||
beyond construction — see [[project_unitigs_always_kept]]),
|
||||
<code>pack_matrices</code> (re-runnable maintenance on an already-finished index
|
||||
via <code>obikmer pack</code>, not just a pipeline step), <code>upgrade_layer_meta</code>
|
||||
(migration, runnable on any existing index at any time).</li>
|
||||
</ul>
|
||||
<h3 id="the-general-pattern-not-obikindexer-specific">The general pattern (not obikindexer-specific)</h3>
|
||||
<p><code>KmerIndex</code>/<code>KmerPartition</code>/<code>KmerLayer</code> stay generic, in <code>obikindex</code> —
|
||||
every domain-specific consumer crate gets to attach its own extension
|
||||
trait(s), of two kinds:</p>
|
||||
<ul>
|
||||
<li><strong>Private</strong> (<code>pub(crate)</code>, invisible outside the defining crate) — for
|
||||
plumbing only that crate's own algorithms need. <code>obikindexer</code> gets
|
||||
exactly one of these (see below); no public counterpart makes sense for
|
||||
it — "l'index est tellement central que le second trait n'a pas
|
||||
vraiment d'intérêt" for construction specifically: nothing external
|
||||
should ever want to call <code>mark_scattered</code> or <code>build_index_layer</code>.</li>
|
||||
<li><strong>Public</strong> — for a genuinely reusable domain extension. The user's own
|
||||
example, found while discussing this, not hypothetical: <code>obikindex/src/
|
||||
index/distance.rs</code> (phylogenetic distance metrics) is currently an
|
||||
<code>impl KmerIndex</code> block <strong>inside <code>obikindex</code> itself</strong> — under this
|
||||
principle it should be a public extension trait owned by <code>obikphylo</code>
|
||||
instead (distance metrics are a phylo concept, <code>obikindex</code> has no more
|
||||
business defining them than <code>obikindex::layer</code> has defining
|
||||
"family"/"minorant", the reasoning <code>SiblingLayerExt</code> already followed
|
||||
for <code>KmerLayer</code> — see <code>obikphylo/src/siblings/iter.rs</code>). <strong>Explicitly
|
||||
deferred</strong> — noted here so it isn't lost, not part of this round.</li>
|
||||
<li>The future cache-manager crate (still blocked on (5), see above) will
|
||||
add its own <strong>public</strong> extension trait mirroring part of <code>KmerIndex</code>'s/
|
||||
<code>KmerPartition</code>'s own read API in cached form (e.g. a cached
|
||||
<code>.partition(i)</code> that doesn't re-touch disk) — same pattern, third data
|
||||
point once built.</li>
|
||||
</ul>
|
||||
<h3 id="concretely-next-to-implement-two-items-in-order">Concretely, next to implement (two items, in order)</h3>
|
||||
<ol>
|
||||
<li><strong><code>obikindexer::extensions</code></strong> — a private (<code>pub(crate)</code>) extension
|
||||
trait, most likely named something like <code>IndexBuildExt</code> (final name
|
||||
not yet chosen), implemented for <code>KmerIndex</code>, carrying the ten methods
|
||||
listed above, moved out of <code>obikindex::index::{kmer_index,
|
||||
index_layer}</code>. Every algorithm in <code>obikindexer::algorithms::*</code> that
|
||||
currently calls <code>idx.mark_scattered()</code>/etc. keeps the same call syntax
|
||||
(extension trait methods are called the same way as inherent ones,
|
||||
just need the trait in scope) — <code>cmd/index/mod.rs</code> itself would need
|
||||
<code>use obikindexer::extensions::IndexBuildExt;</code> (or the module re-exports
|
||||
it) to keep compiling, since it's the one place outside <code>obikindexer</code>'s
|
||||
own algorithms that currently calls <code>mark_scattered</code>/<code>write_spectrum</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code> directly. <strong>Not yet decided</strong>: exact
|
||||
trait name, whether it's one trait or split further (e.g. sentinel
|
||||
marking vs. skeleton/finalize machinery), and whether <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> (not yet extracted into algorithms themselves) move
|
||||
onto it now too or keep calling the soon-to-be-inherent-no-longer
|
||||
methods some other way in the meantime — <strong>ask before implementing</strong>,
|
||||
this changes the blast radius significantly (4 more <code>obikindex</code>
|
||||
internal files depend on <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code>).</li>
|
||||
<li><strong><code>obikalgorithm::Algorithm</code> trait</strong> — new crate, the shared trait
|
||||
<code>obikpartitionner</code>→<code>obikindexer</code> merge (session start of 2026-08-21)
|
||||
and (6)/(7) were deliberately building toward, now with four real
|
||||
<code>new</code>/(setters)/<code>run</code> examples and three distinct callback-bound
|
||||
shapes to reconcile (plain <code>FnMut</code> for <code>PartitionRouter</code>, <code>FnMut +
|
||||
Send</code> for <code>LayerBuilder</code>, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code> —
|
||||
see (7)). Exact shape not yet drafted in this doc — do that as its own
|
||||
design pass before coding, same discipline as everything above.</li>
|
||||
</ol>
|
||||
<p>Both items: <strong>design only, nothing implemented yet</strong> — this section is
|
||||
the record to resume from, not a plan already executed.</p>
|
||||
<h2 id="9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented">(9) done (2026-08-21): <code>obikindexer::extensions::PrivateBuilder</code> — item 1 above, implemented</h2>
|
||||
<p>Renamed from <code>IndexBuilder</code> to <code>PrivateBuilder</code> immediately after (same
|
||||
session), freeing the name <code>IndexBuilder</code> for (10)'s public trait — read
|
||||
<code>IndexBuilder</code> below as <code>PrivateBuilder</code> throughout this section.</p>
|
||||
<p>Scoped down from (8)'s six-method list to the concrete set that's
|
||||
genuinely movable without further ripple — checked, not assumed, before
|
||||
writing anything:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_scattered</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_counted</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_indexed</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">write_spectrum</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">f0</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">f1</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">counts</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">BTreeMap</span><span class="o"><</span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="kt">u64</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">build_index_layer</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">min_ab</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="n">max_ab</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">with_counts</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="n">mode</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMode</span><span class="p">,</span><span class="w"> </span><span class="n">block_bits</span><span class="p">:</span><span class="w"> </span><span class="kt">u8</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">SKError</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">remove_build_artifacts</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">);</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>All six moved bodily out of <code>obikindex::index::{kmer_index, index_layer}</code>
|
||||
into <code>obikindexer::extensions</code> (new module, <code>pub(crate)</code>) — <code>index_layer.rs</code>
|
||||
is now empty and deleted outright.
|
||||
<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>
|
||||
stayed inherent on <code>KmerIndex</code>, per (8)'s reasoning: <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> — living <em>inside</em> <code>obikindex</code> itself — call them too,
|
||||
and <code>obikindex</code> can never depend on <code>obikindexer</code> to reach a trait defined
|
||||
there. Moving those four is real future work (extract
|
||||
merge/select/rebuild/reindex into algorithms first), not part of this
|
||||
step.</p>
|
||||
<p><strong>One new, small, deliberate API widening in <code>obikindex</code></strong>: <code>build_index_layer</code>
|
||||
depends on three helpers that were <code>pub(crate)</code> to <code>obikindex</code>
|
||||
(<code>graph_pipeline::{write_graph_as_unitigs, materialize_layer}</code>,
|
||||
<code>common::olm_to_sk</code>) — widened to <code>pub</code> (re-exported from <code>obikindex</code>'s
|
||||
crate root) so <code>obikindexer</code> could reach them. This is exactly the
|
||||
"enrich shared/lower-level APIs instead of ad hoc local code" call the
|
||||
project's own rules ask for, made explicitly rather than routed around:
|
||||
three functions, already generically written (no rewrite needed), now
|
||||
serve a second caller instead of being duplicated.</p>
|
||||
<p><strong>Why the trait had to be defined in <code>obikindexer</code>, not <code>obikindex</code></strong>:
|
||||
Rust's orphan rule — implementing a trait for a foreign type requires
|
||||
either the trait or the type to be local to the current crate. <code>KmerIndex</code>
|
||||
is foreign to <code>obikindexer</code>, so the trait must be the local half; if it
|
||||
were defined in <code>obikindex</code> instead, <code>pub(crate)</code> there would make it
|
||||
invisible to <code>obikindexer</code> too (crate-private means private to <em>that</em>
|
||||
crate, not "private except to one named dependent") — the opposite of
|
||||
what was wanted.</p>
|
||||
<p><strong>A real design decision made while wiring callers up, not a mechanical
|
||||
rename</strong>: <code>PrivateBuilder</code> being genuinely <code>pub(crate)</code> to <code>obikindexer</code>
|
||||
means <code>obikmer::cmd::index</code> (a different crate) can no longer call
|
||||
<code>mark_scattered</code>/<code>mark_counted</code>/<code>mark_indexed</code>/<code>write_spectrum</code> directly —
|
||||
it never could have, once privacy was real rather than aspirational. Each
|
||||
algorithm now marks its own completion as part of <code>run()</code>/<code>close()</code>
|
||||
instead of leaving it to the caller:
|
||||
- <code>PartitionRouter::close()</code> (not <code>run()</code>) calls <code>mark_scattered()</code> —
|
||||
<code>close()</code>, not <code>run()</code>, is the actual shared completion point between
|
||||
the file-driven <code>run()</code> path and the manual <code>write</code>/<code>write_batch</code>+
|
||||
<code>close()</code> path low-level callers (tests) use; putting it in <code>run()</code>
|
||||
alone would have silently skipped marking for every caller that never
|
||||
calls <code>run()</code>. <code>run()</code> already calls <code>self.close()</code> at its own end, so
|
||||
this covers both paths through one line, not two.
|
||||
- <code>Counter::run</code> calls <code>write_spectrum</code> then <code>mark_counted</code> before
|
||||
returning.
|
||||
- <code>LayerBuilder::run</code> calls <code>mark_indexed</code> before returning.</p>
|
||||
<p><code>cmd/index/mod.rs</code> lost all four direct calls (<code>mark_scattered</code>/
|
||||
<code>write_spectrum</code>/<code>mark_counted</code>/<code>mark_indexed</code>) — each stage's <code>if
|
||||
idx.state() < IndexState::X { ... }</code> block is now purely "run the
|
||||
algorithm," no separate bookkeeping call after it. Confirms, precisely
|
||||
this time (checked by re-reading the whole file, not assumed): <code>cmd/index</code>
|
||||
now rests on the four algorithms for every read/write of pipeline state
|
||||
except <code>KmerIndex::{exists, create, state, n_partitions}</code>, which are
|
||||
genuinely index-identity concerns, not construction bookkeeping — the
|
||||
original question this whole design pass started from.</p>
|
||||
<p>Same fix applied to <code>obikphylo</code>'s test harness (its four explicit
|
||||
<code>mark_*</code>/<code>write_spectrum</code> calls removed, relying on the algorithms now
|
||||
doing it themselves) — <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test needed no change (never called <code>mark_*</code> directly).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the CLI smoke test — 870
|
||||
kmers, same as (6)/(7).</p>
|
||||
<p>Still not done at the time of writing: item 2 from (8) (<code>obikalgorithm::
|
||||
Algorithm</code>), (5), the future cache crate, the <code>distance.rs</code> →
|
||||
<code>obikphylo</code> relocation (noted in (8), explicitly deferred), and
|
||||
extracting <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code> into algorithms.</p>
|
||||
<h2 id="10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session">(10) done (2026-08-21): <code>obikindex::IndexBuilder</code> — the public counterpart, same session</h2>
|
||||
<p>Immediate correction to (9): the private trait built there was renamed
|
||||
<code>PrivateBuilder</code> (freeing the name), and the four methods (9) had left
|
||||
inherent on <code>KmerIndex</code> — <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code> — got their own trait after all: <strong><code>IndexBuilder</code></strong>,
|
||||
public, defined in <code>obikindex</code> itself (not <code>obikindexer</code>):</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="p">:</span><span class="w"> </span><span class="nb">Sized</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">clear_output_for_create</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">force</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">create_skeleton</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">meta</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMeta</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">finalize_indexed</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">rep</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">Reporter</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">state</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">IndexState</span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>User's framing: these four are "maintenance", not "scientific computation
|
||||
on an index" — a different kind of non-generic-ness than (9)'s six
|
||||
(<code>mark_*</code>/<code>write_spectrum</code>/<code>build_index_layer</code>/<code>remove_build_artifacts</code>,
|
||||
exclusive to the 4-stage pipeline). Maintenance is used more broadly
|
||||
(<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>), so it gets a real, public trait —
|
||||
not folded back into <code>KmerIndex</code>'s inherent surface, and not private
|
||||
either.</p>
|
||||
<p><strong>Where it lives, and why that's not arbitrary</strong>: (9) needed the orphan
|
||||
rule to force its trait into <code>obikindexer</code>, to achieve genuine
|
||||
crate-private visibility. Here the requirement is the opposite:
|
||||
<code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> — the trait's own
|
||||
heaviest users — live <em>inside</em> <code>obikindex</code>. A trait they need to reach
|
||||
must be local to <code>obikindex</code> (or a crate <code>obikindex</code> itself depends on,
|
||||
which doesn't exist for this). So <code>IndexBuilder</code> lives in a new
|
||||
<code>obikindex/src/index/builder.rs</code>, <code>pub trait</code> (no orphan-rule tension at
|
||||
all here — both trait and type are local to the same crate), re-exported
|
||||
from <code>obikindex</code>'s crate root alongside <code>PrivateBuilder</code>'s sibling
|
||||
<code>obikindexer::extensions::PrivateBuilder</code> staying where it is. Two
|
||||
traits, two crates, two different reasons, not a contradiction.</p>
|
||||
<p><strong>Blast radius, all inside <code>obikindex</code> plus one external crate</strong>: every
|
||||
internal caller of these four methods needs the trait imported now that
|
||||
they're no longer inherent — <code>merge.rs</code>, <code>select.rs</code>, <code>rebuild.rs</code>,
|
||||
<code>reindex.rs</code> (<code>use crate::index::builder::IndexBuilder;</code>) and, externally,
|
||||
<code>obikmer::cmd::index::mod</code> (<code>use obikindex::IndexBuilder;</code>, for the three
|
||||
<code>idx.state() < IndexState::X</code> resumability checks). Call syntax at every
|
||||
site is unchanged (<code>KmerIndex::create_skeleton(...)</code>,
|
||||
<code>self.state()</code>) — only trait-in-scope requirements are new, which is
|
||||
exactly the point: same ergonomics, less surface baked into <code>KmerIndex</code>
|
||||
itself.</p>
|
||||
<p>Verification went one step further than (9): beyond
|
||||
<code>cargo check --workspace --all-targets</code> + <code>cargo test --workspace</code> +
|
||||
<code>scripts/smoke_test_index.sh</code> (all green, 870 kmers again), ran
|
||||
<code>obikmer merge</code> end to end on two freshly built indexes (exercises
|
||||
<code>clear_output_for_create</code>/<code>finalize_indexed</code> directly, the two methods
|
||||
<code>scripts/smoke_test_index.sh</code> itself never touches) — exit 0, <code>pack</code>
|
||||
stage completed. Test suite alone would not have caught a regression
|
||||
here: no existing test builds two real indexes and merges them through
|
||||
the CLI.</p>
|
||||
<p><code>KmerIndex</code> itself now carries only: identity/config accessors
|
||||
(<code>root_path</code>/<code>meta</code>/<code>kmer_size</code>/<code>n_bits</code>/<code>evidence_mode</code>/<code>genomes</code>/...),
|
||||
path resolution (<code>partition_dir</code>/<code>index_dir</code>/<code>layer_dir</code>/
|
||||
<code>partition_meta</code>/<code>n_layers</code>), and a few index-maintenance operations not
|
||||
yet sorted into either trait (<code>layer_unitigs_path</code>, <code>pack_matrices</code>,
|
||||
<code>upgrade_layer_meta</code> — see (8)'s "tested and discarded" list; still
|
||||
correctly inherent, not construction-only by the semantic criterion) —
|
||||
<code>create</code>/<code>open</code>/<code>exists</code> (identity, can't be trait methods needing <code>Self</code>
|
||||
before one exists) round that out.</p>
|
||||
<h2 id="11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files">(11) done (2026-08-21): <code>KmerIndex</code>/<code>IndexMeta</code> made fully stateless, <code>IndexState</code> moved off sentinel files</h2>
|
||||
<p>Triggered mid-discussion of <code>obikalgorithm::Algorithm</code> (still not started —
|
||||
see "Still not done" below): user asked why <code>PartitionRouter::new</code> still
|
||||
took <code>&mut KmerIndex</code> at all, and questioned whether <code>mark_scattered</code>
|
||||
belonged in the algorithm or in <code>cmd/index</code>. Investigation found <code>&mut</code>
|
||||
had become <em>newly</em> necessary since (9) — <code>mark_scattered</code> was mutating
|
||||
<code>self.meta.genomes</code> in memory so <code>Counter</code>'s later <code>write_spectrum</code> call
|
||||
(same <code>idx</code> instance) would see the derived label. User's resolution: the
|
||||
"disk is truth, stateless" principle already agreed for <code>KmerPartition</code>/
|
||||
<code>Layer</code> in (5) (still unimplemented for those two) should extend to
|
||||
<code>KmerIndex</code> itself — move <code>IndexState</code> (<code>Empty</code>/<code>Scattered</code>/<code>Counted</code>/
|
||||
<code>Indexed</code>) off the three sentinel files (<code>scatter.done</code>/<code>count.done</code>/
|
||||
<code>index.done</code>, detected by existence) and into a field of <code>index.meta</code>'s
|
||||
own JSON, so the <code>mark_*</code> calls become plain disk writes an algorithm can
|
||||
legitimately make on <code>&self</code> — no in-memory mutation left to protect.</p>
|
||||
<p><strong>Shape of <code>IndexMeta</code>, per the user's explicit spec</strong>: one JSON file per
|
||||
index (<code>index.meta</code>), one <code>IndexMeta</code> instance per index, held and
|
||||
returned as <code>Arc<IndexMeta></code> (not <code>&IndexMeta</code>) by <code>KmerIndex::meta()</code>.
|
||||
<code>config</code> (<code>kmer_size</code>/<code>minimizer_size</code>/<code>n_bits</code>/<code>with_counts</code>/<code>evidence</code>/
|
||||
<code>block_bits</code>) is fixed at construction, cached as a <code>pub</code> field (getter
|
||||
kept alongside, for symmetry) — "les champs constants restent des champs
|
||||
de la structure", read once, never re-read from disk. <code>genomes</code> and
|
||||
<code>state</code> are the opposite: no in-memory cache at all, every accessor
|
||||
(<code>genomes()</code>, <code>state()</code>) re-reads <code>index.meta</code> from disk, every mutator
|
||||
(<code>push_genome</code>/<code>rename_genome</code>/<code>set_genomes</code>/<code>set_state</code>/<code>mark_scattered</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code>) does a full read-modify-write of the same
|
||||
file. An internal <code>std::sync::RwLock<()></code> is held across each
|
||||
read-modify-write sequence (not just the write) so two callers sharing
|
||||
the same <code>Arc<IndexMeta></code> can't lose an update to each other — this is
|
||||
<em>not</em> a cross-process lock (that's <code>obisys::DirLock</code>, already held by
|
||||
<code>cmd/index</code> for the whole build); it only serialises access through one
|
||||
shared in-process instance.</p>
|
||||
<p><strong>Construction, chain-of-responsibility style, matching (5)'s pattern</strong>:
|
||||
<code>IndexMeta::create(&KmerIndex, config, genomes)</code> / <code>IndexMeta::open(&KmerIndex)</code>
|
||||
ask the index for its own root path rather than taking one directly. Since
|
||||
<code>KmerIndex::create</code> doesn't have a complete <code>KmerIndex</code> yet to hand in
|
||||
(it's what's being built), added lower-level <code>pub(crate)</code> path-based
|
||||
primitives <code>create_at(&Path, ...)</code> / <code>open_at(&Path)</code> that <code>KmerIndex::create</code>/
|
||||
<code>open</code> and <code>builder.rs</code>'s <code>create_skeleton</code> call directly, bypassing the
|
||||
convenience wrappers for that one bootstrap case.</p>
|
||||
<p><strong>The one deliberate exception</strong>: <code>select_in_place</code> and <code>reindex</code>
|
||||
genuinely rewrite <code>config</code> after an index already exists (output
|
||||
type/evidence mode changes in place) — contradicting "config never
|
||||
changes" for the general case. Resolved with a separate, explicitly
|
||||
rare-labelled <code>IndexMeta::rewrite_config(config, genomes)</code> (preserves
|
||||
<code>state</code>, overwrites everything else); callers refresh their own cached
|
||||
<code>Arc<IndexMeta></code> afterward (<code>self.meta = Arc::new(IndexMeta::open(self)?)</code>)
|
||||
since <code>IndexMeta</code> has no way to reach back into whichever <code>KmerIndex</code>
|
||||
holds it.</p>
|
||||
<p><strong>Consequence confirmed, not just hoped for</strong>: with <code>mark_scattered</code> no
|
||||
longer touching anything in memory, <code>PartitionRouter</code> genuinely never
|
||||
needs <code>&mut KmerIndex</code> — <code>PartitionRouter<'a> { index: &'a KmerIndex }</code>,
|
||||
<code>new(&'a KmerIndex)</code>. This is effectively the <code>PartitionRouter</code> half of
|
||||
(5)'s "order of remaining work" item done as a side effect; <code>KmerPartition</code>/
|
||||
<code>Layer</code> themselves are still unimplemented for (5).</p>
|
||||
<p><strong>Blast radius — much larger than (9)/(10), touched nearly every crate</strong>:
|
||||
every <code>.meta().genomes</code>/<code>.meta.genomes</code> field access became a fallible
|
||||
<code>.genomes()?</code> method call (<code>genomes</code> reads <code>io::Result<Vec<GenomeInfo>></code>
|
||||
now, not a field), and <code>.meta_mut()</code> was removed outright (no more direct
|
||||
field mutation from outside <code>IndexMeta</code>). Fixed across:
|
||||
- <code>obikindex</code> internals: <code>meta.rs</code>/<code>state.rs</code>/<code>kmer_index.rs</code>/<code>builder.rs</code>
|
||||
(full rewrites), <code>reindex.rs</code>/<code>select.rs</code> (switched to <code>rewrite_config</code>),
|
||||
<code>merge.rs</code> (heaviest single file — genome counts precomputed once per
|
||||
source into a <code>Vec<Vec<GenomeInfo>></code> up front rather than re-reading
|
||||
<code>index.meta</code> from disk repeatedly through the function, sentinel write
|
||||
replaced with <code>dst2.meta.mark_indexed()</code>), <code>stats.rs</code>, <code>distance.rs</code>,
|
||||
<code>dump.rs</code>, <code>predicate.rs</code> (its <code>IndexMeta</code>-inherent <code>matching_genome_indices</code>/
|
||||
<code>build_group_filter</code> now read genomes fresh internally), <code>mod.rs</code>/<code>lib.rs</code>
|
||||
(sentinel constant re-exports removed — <code>IndexState</code> no longer has
|
||||
<code>SENTINEL_*</code>/<code>detect()</code> at all).
|
||||
- <code>obikindexer::extensions::PrivateBuilder</code>: <code>mark_scattered</code> signature
|
||||
dropped <code>&mut self</code> → <code>&self</code>; the four <code>mark_*</code>/<code>write_spectrum</code> bodies
|
||||
became one-line delegations to <code>self.meta().mark_*()</code>.
|
||||
- <code>obikphylo::siblings</code>: <code>alignment.rs</code>/<code>cardinality.rs</code>/<code>distance.rs</code>/
|
||||
<code>entropy.rs</code>/<code>sankoff_bundle.rs</code>/<code>stats.rs</code>/<code>tests.rs</code> — all had
|
||||
<code>self.meta().genomes.len()</code>-shaped reads, mechanically fixed to
|
||||
<code>.genomes().map_err(OKIError::Io)?.len()</code> (tests: <code>.unwrap()</code>).
|
||||
- <code>obikmer::cmd::*</code>: <code>annotate</code> (rewrote its rename path to load genomes
|
||||
once, mutate the in-memory <code>Vec</code>, then <code>idx.meta().set_genomes(...)</code>
|
||||
instead of <code>meta_mut()</code>), <code>filter</code>/<code>pack</code>/<code>dump</code>/<code>unitig</code>/<code>merge</code>/<code>select</code>/
|
||||
<code>phylo</code> (fetch-once-then-use pattern for genome counts/labels),
|
||||
<code>utils/maintenance.rs</code> (<code>run_rename</code> now calls the pre-existing
|
||||
<code>IndexMeta::rename_genome</code>, dropping its own hand-rolled field mutation
|
||||
entirely), <code>index/mod.rs</code> (three <code>idx.state() < IndexState::X</code>
|
||||
resumability checks needed a fallible read — factored into a small
|
||||
<code>current_state(&KmerIndex) -> IndexState</code> helper rather than repeating
|
||||
the same <code>unwrap_or_else</code> three times), <code>query/*</code> (<code>emit_batch</code>'s
|
||||
signature changed from <code>&IndexMeta</code> to <code>&[GenomeInfo]</code>, and <code>genomes</code> is
|
||||
now fetched once in <code>run()</code> and threaded down through <code>process_chunk</code>
|
||||
as <code>Arc<Vec<GenomeInfo>></code> rather than re-reading <code>index.meta</code> from disk
|
||||
on every chunk — a deliberate deviation from the "always re-read"
|
||||
default, justified because this is a genuine per-chunk hot path, unlike
|
||||
every other call site touched in this pass).
|
||||
- One <code>&IndexMeta</code>-vs-<code>Arc<IndexMeta></code> argument-type mismatch pattern
|
||||
recurred at several CLI call sites (<code>build_filters</code>/<code>build_specs</code>/
|
||||
<code>emit_batch</code>'s original signature) — resolved via <code>Arc</code>'s deref
|
||||
coercion (<code>&idx.meta()</code> coerces to <code>&IndexMeta</code>) rather than changing
|
||||
every downstream signature to accept <code>Arc<IndexMeta></code>.</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and
|
||||
<code>cargo test --workspace</code> both green (0 failures) after the full
|
||||
propagation, <code>scripts/smoke_test_index.sh</code> green (870 kmers, same as every
|
||||
prior round), plus a manual CLI run of <code>index</code> (×2) → <code>merge</code> → <code>select</code>
|
||||
→ <code>reindex</code> → <code>utils --new-label</code> (rename) → <code>utils --stats</code>, all exit 0,
|
||||
confirming the four most-affected commands (the ones (10)'s verification
|
||||
already flagged as under-covered by the automated test suite) still work
|
||||
end to end against the new <code>Arc<IndexMeta></code>/on-disk-<code>IndexState</code> shape.</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign itself
|
||||
(only the <code>PartitionRouter</code>-<code>&mut</code>-removal piece landed, as a side
|
||||
effect); the <code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly
|
||||
deferred); the future cache-manager crate. (8)'s <code>obikalgorithm::
|
||||
Algorithm</code> trait, resumed and closed in (12) below.</p>
|
||||
<h2 id="12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session">(12) done (2026-08-21): <code>obikalgorithm::Algorithm</code> — the shared trait, resumed and closed in one session</h2>
|
||||
<p>Resumed (8)'s point 2 through a point-by-point discussion of what's
|
||||
actually common across the four pipeline algorithms, now that (11) made
|
||||
<code>KmerIndex</code> itself immutable everywhere. Four sub-points, each closed
|
||||
before moving to the next:</p>
|
||||
<p><strong>1. Receiver (<code>&self</code> vs <code>&mut self</code>)</strong> — investigated whether (11)'s
|
||||
removal of <code>&mut KmerIndex</code> also removed the need for <code>PartitionRouter::
|
||||
run</code> to take <code>&mut self</code>. It didn't: <code>PartitionRouter</code> holds real
|
||||
per-run state of its own (<code>writers: Vec<Option<SKFileWriter>></code>, open file
|
||||
handles, purely in-process RAM — confirmed by checking where <code>writers</code> is
|
||||
stored, nothing to do with <code>KmerIndex</code>/disk truth), unrelated to the
|
||||
index. First proposal (wrap <code>writers</code> in <code>RefCell</code> so all four could
|
||||
share a uniform <code>&self</code>) was retracted on pushback: manufacturing
|
||||
interior mutability with runtime borrow checks to satisfy a cosmetic
|
||||
uniformity that Rust doesn't even require is over-engineering — a trait
|
||||
method's receiver must match exactly across implementors, but nothing
|
||||
stops that shared receiver from being <code>&mut self</code> with three of the four
|
||||
implementations simply not using the mutability. Settled: trait declares
|
||||
<code>&mut self</code>; <code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> (previously <code>&self</code>)
|
||||
now also take <code>&mut self</code>, unused.</p>
|
||||
<p><strong>2. <code>path_source</code> as a <code>PartitionRouter</code> setter, not a <code>run()</code> param</strong> —
|
||||
added a <code>files: Option<Box<dyn Iterator<Item = PathBuf> + Send>></code> field +
|
||||
<code>.files(impl Iterator<Item = PathBuf> + Send + 'static) -> Self</code> setter
|
||||
(boxed rather than a generic type parameter on <code>PartitionRouter<'a></code>:
|
||||
negligible cost — one <code>PathBuf</code> per input <em>file</em>, not per k-mer — for a
|
||||
much more usable type when passing the builder around). <code>run</code> now does
|
||||
<code>self.files.take().ok_or_else(...)</code>, erroring if <code>.files(...)</code> was never
|
||||
called, instead of taking <code>path_source</code> as a parameter.</p>
|
||||
<p><strong>3. Unifying the three progress-callback bound shapes</strong> — reopened, then
|
||||
resolved differently than (8) originally framed it. First proposal
|
||||
(force everything to <code>FnMut(Progress) + Send</code>) was rejected on the same
|
||||
principle as point 1: <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn(Progress) + Sync</code>
|
||||
isn't arbitrary — their callback is invoked concurrently from multiple
|
||||
rayon worker threads, and <code>FnMut</code> requires exclusive access, so forcing
|
||||
it would mean wrapping the callback in a <code>Mutex</code> for zero benefit at the
|
||||
one real call site (<code>pb.inc(1)</code>, already thread-safe). The actual
|
||||
resolution: move <code>on_progress</code> off <code>run()</code>'s signature entirely, onto a
|
||||
per-algorithm <code>.on_progress(...)</code> setter — same treatment as point 2's
|
||||
<code>path_source</code> — so each algorithm keeps its own bound (<code>PartitionRouter</code>:
|
||||
<code>FnMut(Progress) + 'a</code>, sequential, no <code>Send</code> needed; <code>LayerBuilder</code>:
|
||||
<code>FnMut(Progress) + Send + 'a</code>, crosses into <code>PartitionRunner</code>'s
|
||||
<code>thread::scope</code>-spawned controller thread once; <code>Dereplicator</code>/<code>Counter</code>:
|
||||
<code>Fn(Progress) + Sync + 'a</code>, invoked concurrently from rayon workers).
|
||||
This <em>also</em> dissolves the original problem <code>run()</code> had: once the
|
||||
callback isn't part of <code>run</code>'s signature at all, there's nothing left to
|
||||
unify there, and point 4 (below) becomes trivial.</p>
|
||||
<p><strong>4. <code>Output</code> as an associated type, <code>Error</code> fixed</strong> — trivial once (3)
|
||||
moved the callback out: <code>Error</code> was already uniform (all four return
|
||||
<code>obiskio::SKResult<T></code> = <code>Result<T, SKError></code>), only <code>Output</code> varied
|
||||
(<code>()</code>/<code>()</code>/<code>KmerSpectrum</code>/<code>usize</code>). First cut reused <code>obiskio::SKResult</code>
|
||||
directly as the trait's return type — <strong>caught and corrected the same
|
||||
session</strong>: <code>SKError</code> enumerates I/O-specific cases (<code>BadMagic</code>/
|
||||
<code>Truncated</code>/<code>Compression</code>/...), meaningless at the level of a generic
|
||||
"algorithm" abstraction, and borrowing it made <code>obikalgorithm</code> — meant to
|
||||
be minimal and neutral — depend on a low-level I/O crate purely to reuse
|
||||
its error type. Textbook instance of the "petits pois" failure mode
|
||||
(patch around a convenient existing type instead of asking what this
|
||||
crate should actually own). Fixed to a genuinely generic, boxed error
|
||||
type owned by <code>obikalgorithm</code> itself:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// obikalgorithm — no dependency on obiskio or any other crate</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">error</span><span class="p">::</span><span class="n">Error</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">result</span><span class="p">::</span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">Error</span><span class="o">></span><span class="p">;</span>
|
||||
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Algorithm</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Output</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="bp">Self</span><span class="p">::</span><span class="n">Output</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Any concrete error (<code>SKError</code>, <code>std::io::Error</code>, ...) converts
|
||||
automatically via <code>?</code>, through <code>std</code>'s own blanket <code>impl<E: Error + Send
|
||||
+ Sync> From<E> for Box<dyn Error + Send + Sync></code> — no custom <code>From</code> impl
|
||||
needed, no dependency on the crate that defines the concrete error type.
|
||||
The four algorithms' <code>run</code> bodies needed no change beyond the signature's
|
||||
return type (every existing <code>?</code> on an <code>SKError</code>-returning subcall keeps
|
||||
compiling, converting through the same blanket impl at the boundary).</p>
|
||||
<p><code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> each <code>impl
|
||||
Algorithm for X<'_> { type Output = ...; fn run(&mut self) -> obikalgorithm::Result<...> { ... } }</code>
|
||||
— the old inherent <code>run</code> methods were removed outright (not kept as
|
||||
duplicates), so callers now <code>use obikalgorithm::Algorithm;</code> to call
|
||||
<code>.run()</code>. Every field-lifetime-bound boxed callback (<code>Box<dyn
|
||||
FnMut(Progress) + 'a></code> etc.) is tied to the algorithm's own <code>'a</code> (the
|
||||
<code>&'a KmerIndex</code> lifetime already on the struct), not <code>'static</code> — avoids
|
||||
forcing callers' progress closures to <code>move</code>-capture (and therefore clone
|
||||
or <code>Arc</code>-wrap) local state like <code>TracedBar</code>/EMA-rate accumulators that
|
||||
they'd otherwise want to keep using by reference after <code>run()</code> returns.</p>
|
||||
<p><strong>Why a new crate, not a submodule of <code>obikindexer</code></strong>: <code>obikmer::cmd::
|
||||
index::mod</code> and <code>obikphylo</code>'s own test helpers both need to call <code>.run()</code>
|
||||
on these algorithms, so the trait has to be reachable from outside
|
||||
<code>obikindexer</code> — putting it in <code>obikindexer</code> itself would work file-wise
|
||||
but conflates "the trait every algorithm implements" with "one crate's
|
||||
particular four implementations of it", the same reasoning that already
|
||||
separated <code>obikindex</code> (data model) from <code>obikindexer</code> (algorithms
|
||||
operating on it). <code>obikalgorithm</code> has <strong>no dependencies at all</strong> (see
|
||||
above); <code>obikindexer</code>, <code>obikmer</code>, and <code>obikphylo</code> (dev-dependency, for
|
||||
its test helper) all depend on it.</p>
|
||||
<p><strong>Blast radius</strong>: <code>obikindexer</code>'s four algorithm modules (struct field +
|
||||
setter + trait impl each); <code>obikmer::cmd::index::mod</code> (three call sites:
|
||||
<code>.on_progress(cb)</code> before <code>.run()</code>, unqualified now that the trait is in
|
||||
scope); <code>obikindexer::algorithms::partitionner::tests</code> and <code>obikphylo::
|
||||
siblings::tests</code> (both had direct <code>.run(None::<fn(Progress)>)</code>-shaped
|
||||
calls needing the same treatment). New <code>obikalgorithm</code> crate registered
|
||||
in the workspace <code>Cargo.toml</code>, depended on by <code>obikindexer</code>/<code>obikmer</code>
|
||||
(regular) and <code>obikphylo</code> (dev).</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and <code>cargo test
|
||||
--workspace</code> both green (0 failures), <code>scripts/smoke_test_index.sh</code> green
|
||||
(870 kmers, same as every prior round) — this round didn't repeat the
|
||||
manual <code>merge</code>/<code>select</code>/<code>reindex</code> CLI exercise from (11), since nothing
|
||||
in this pass touched those commands' code paths (only the four pipeline
|
||||
algorithms and <code>cmd::index</code>, already covered by the smoke test). Reverified
|
||||
after the <code>obiskio</code>-dependency fix above (same three checks, still green,
|
||||
<code>obikalgorithm/Cargo.toml</code> now has zero <code>[dependencies]</code>).</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign; the
|
||||
<code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly deferred); the
|
||||
future cache-manager crate (mentioned in (8) as a later, mirrored
|
||||
extension-trait exercise, not started).</p>
|
||||
<h2 id="the-problem">The problem</h2>
|
||||
<p>Reading a layer's data (MPHF + matrix) is not free: <code>MphfLayer::open</code> mmaps
|
||||
<code>mphf.bin</code> plus (<code>evidence.bin</code>/<code>fingerprint.bin</code> + <code>unitigs.bin</code>), and the
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1405,7 +1405,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1388,7 +1388,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1254,7 +1254,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1119,17 +1119,6 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#in-place-keep-only-group-a" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
In-place: keep only group A
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1157,6 +1146,23 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Implementation notes">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known gap (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1376,7 +1382,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1638,17 +1644,6 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#in-place-keep-only-group-a" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
In-place: keep only group A
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1676,6 +1671,23 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Implementation notes">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known gap (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1728,23 +1740,32 @@ are preserved unchanged; only the data matrices are rewritten.</p>
|
||||
<hr />
|
||||
<h2 id="synopsis">Synopsis</h2>
|
||||
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span><input-index>
|
||||
<span class="w"> </span><span class="o">{</span><span class="w"> </span>--output<span class="w"> </span><dir><span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="w"> </span><span class="o">}</span>
|
||||
<span class="w"> </span>--output<span class="w"> </span><dir>
|
||||
<span class="w"> </span><span class="o">[</span>--group<span class="w"> </span><name>:<pred><span class="w"> </span>...<span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--group-op<span class="w"> </span><name>:<op><span class="w"> </span>...<span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--aggregate-by<span class="w"> </span><key><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--aggregate-op<span class="w"> </span><op><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--select<span class="w"> </span><col1,col2,...><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--presence-threshold<span class="w"> </span><N><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--force-copy<span class="w"> </span><span class="o">]</span>
|
||||
</code></pre></div>
|
||||
<hr />
|
||||
<h2 id="output-destination">Output destination</h2>
|
||||
<p>Exactly one of <code>--output</code> or <code>--in-place</code> must be specified.</p>
|
||||
<p><strong><code>--output <dir></code></strong> — writes a new index to <code><dir></code>. The source index is
|
||||
unchanged. The MPHF and unitig files are copied; only the data matrices are
|
||||
rewritten with the new column layout.</p>
|
||||
<p><strong><code>--in-place</code></strong> — rewrites the data matrices of the source index directly.
|
||||
Removed or replaced columns are lost. The operation writes to temporary files
|
||||
first, then renames atomically, so an interrupted run leaves the index intact.</p>
|
||||
<p><code>--output <dir></code> is required — <code>select</code> always writes a new index; there is no
|
||||
<code>--in-place</code> mode (2026-08-28: never implemented, removed from the design). The
|
||||
source index is unchanged.</p>
|
||||
<p>Each layer's kmer-identity files (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/
|
||||
<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code>) are never rewritten by a
|
||||
column projection/aggregation, so they are hard-linked into the output rather
|
||||
than copied — no extra disk for them even on a large index. Falls back to a
|
||||
real copy automatically if linking fails (different filesystems); <code>--force-copy</code>
|
||||
forces a real copy always, for an output that must survive independently of the
|
||||
source on disk (a hard link shares the same inode — rewriting one path outside
|
||||
<code>select</code> itself would affect the other). Only the <code>presence</code>/<code>counts</code>
|
||||
subdirectory is ever a genuinely new, independent file.</p>
|
||||
<p>To replace an index with a selected version of itself, select to a temporary
|
||||
directory and swap it in (<code>rm -rf INDEX && mv INDEX.tmp INDEX</code>) — the case
|
||||
<code>--in-place</code> used to cover.</p>
|
||||
<hr />
|
||||
<h2 id="defining-output-columns">Defining output columns</h2>
|
||||
<h3 id="named-groups-group">Named groups — <code>--group</code></h3>
|
||||
@@ -1908,9 +1929,6 @@ is > N.</p>
|
||||
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>out<span class="w"> </span><span class="se">\</span>
|
||||
<span class="w"> </span>--select<span class="w"> </span><span class="s2">"Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"</span>
|
||||
</code></pre></div>
|
||||
<h3 id="in-place-keep-only-group-a">In-place: keep only group A</h3>
|
||||
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--in-place<span class="w"> </span>--group<span class="w"> </span><span class="s2">"A:group=A"</span><span class="w"> </span>--select<span class="w"> </span><span class="s2">"A"</span>
|
||||
</code></pre></div>
|
||||
<h3 id="compose-with-filter">Compose with filter</h3>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1"># Step 1: keep only B. nana-specific k-mers</span>
|
||||
obikmer<span class="w"> </span>filter<span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>filtered<span class="w"> </span><span class="se">\</span>
|
||||
@@ -1921,32 +1939,46 @@ obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </s
|
||||
</code></pre></div>
|
||||
<hr />
|
||||
<h2 id="implementation-notes">Implementation notes</h2>
|
||||
<p><code>select</code> does not rebuild the MPHF. The 256 partitions are processed in parallel
|
||||
(rayon), each writing its output independently; results require no synchronisation
|
||||
because every partition owns a distinct set of files.</p>
|
||||
<p>For each layer in each partition:</p>
|
||||
<p><code>select</code> does not rebuild the MPHF. Every partition is processed independently
|
||||
(<code>PartitionRunner</code>), each writing its own output layers; no cross-partition
|
||||
synchronisation is needed.</p>
|
||||
<p>For each layer in each partition (<code>obikselect::select_layer::select_partition</code>):</p>
|
||||
<ol>
|
||||
<li>The slot count <code>n</code> is read by opening the source data matrix.</li>
|
||||
<li>A new data matrix is built with M columns (M = number of output columns).</li>
|
||||
<li>For each slot <code>s</code> in <code>0..n</code>:</li>
|
||||
<li><code>old_row = matrix.fill_row(s)</code> — reads the original <code>N</code>-column row without allocating.</li>
|
||||
<li>For each output column <code>j</code>:<ul>
|
||||
<li><code>new_row[j] = aggregate(op, old_row[group_indices])</code>.</li>
|
||||
<li>Pass-through columns are represented as single-element groups with the
|
||||
default operator (<code>any</code> for presence, <code>sum</code> for count) — same code path.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>The new row is written slot by slot into each column builder.</li>
|
||||
<li>All plain files in the source layer directory (<code>mphf.bin</code>, <code>unitigs.bin</code>,
|
||||
evidence files, <code>layer_meta.json</code>) are copied verbatim; only the <code>presence/</code>
|
||||
or <code>counts/</code> subdirectory is rewritten.</li>
|
||||
<li><code>copy_layer_files</code> hard-links the source layer's kmer-identity files
|
||||
(<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/
|
||||
<code>fingerprint.bin</code>/<code>layer_meta.json</code>) into the destination — never a real
|
||||
copy unless linking fails or <code>--force-copy</code> is given.</li>
|
||||
<li>A new data matrix is built with M columns (M = number of output columns),
|
||||
under a fresh <code>presence/</code>/<code>counts/</code> subdirectory (never touching the
|
||||
source's own).</li>
|
||||
<li><strong>Presence source (2026-08-28: <code>batch_presence_counts</code>)</strong>: one shared pass
|
||||
over the source bit matrix computes every output group's presence count at
|
||||
once — row-major native for a <code>Sparse</code> source (<code>for_each_genome_in_row</code>,
|
||||
which has no column representation to read a <code>col_view</code> from at all — the
|
||||
reason this replaced the old per-group loop, not just an optimisation of
|
||||
it), deduplicated column-major (one <code>col_view</code> per <em>distinct referenced
|
||||
column</em>, not per group) for <code>Columnar</code>/<code>Packed</code>. Every <code>AggOp</code> for a bit
|
||||
matrix is then a cheap derivation of that one count vector (<code>sum</code> = the
|
||||
count itself, <code>any</code>/<code>max</code> = <code>count ≥ 1</code>, <code>all</code>/<code>min</code> = <code>count == group
|
||||
size</code>, <code>none</code> = <code>count == 0</code>) — see
|
||||
<code>obikselect::select_layer::agg_result_from_count</code>.</li>
|
||||
<li><strong>Count source</strong>: unchanged, one <code>col_view</code>-driven pass per output column
|
||||
via <code>MatrixGroupOps</code> — <code>sum</code>/<code>min</code>/<code>max</code> are genuine per-value reductions
|
||||
for a count matrix, not derivable from a single presence count the way
|
||||
they are for a bit matrix.</li>
|
||||
<li><code>index.meta</code> is rewritten with the new genome list and updated <code>with_counts</code>.</li>
|
||||
</ol>
|
||||
<p><strong><code>--in-place</code> write strategy:</strong> new data is written to a temporary sibling
|
||||
directory (<code>presence_new/</code> or <code>counts_new/</code>); on success the old directory is
|
||||
removed and the temporary one is renamed into place. An interrupted run leaves
|
||||
at most one stale <code>*_new/</code> directory; the original data is intact until the
|
||||
rename step.</p>
|
||||
<h3 id="known-gap-not-yet-fixed-2026-08-28">Known gap (not yet fixed, 2026-08-28)</h3>
|
||||
<p>Step 4 above still panics (<code>col_view() not available on Sparse
|
||||
PersistentCompactIntMatrix</code>) if the source is a <strong>count</strong> index packed
|
||||
sparse — <code>batch_presence_counts</code>' row-major treatment was only ported to the
|
||||
bit-matrix (<code>Presence</code>) case, since that was the one actually blocking a real
|
||||
benchmark run. <code>select</code>/<code>filter</code> on a sparse-packed count index still hits
|
||||
this; the fix would follow the same shape (a <code>PersistentSparseCompactIntMatrix</code>
|
||||
row-major decode, analogous to <code>for_each_genome_in_row</code>), just not done. Since
|
||||
<code>obisys::numa::runner::PartitionRunner</code>'s panic-propagation fix (see
|
||||
<code>architecture/numa_partition_runner.md</code>), this at least fails fast (process
|
||||
panic, exit 101) instead of hanging.</p>
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1326,7 +1326,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1232,7 +1232,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1354,7 +1354,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -1228,7 +1228,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1314,7 +1314,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1227,7 +1227,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1199,7 +1199,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -909,6 +909,106 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Automatic α estimation (--gamma-shape auto)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Explicit --session (implemented 2026-09-12)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1641,7 +1741,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -2195,6 +2295,106 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Automatic α estimation (--gamma-shape auto)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Explicit --session (implemented 2026-09-12)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -4261,6 +4461,739 @@ among the survivors) — a single extra pass is sufficient.</p>
|
||||
<code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
|
||||
from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
|
||||
workspace <code>cargo test</code> green.</p>
|
||||
<h2 id="-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28"><code>--distance</code> unification: SNP corrections as first-class metrics (implemented, 2026-08-28)</h2>
|
||||
<p><strong>Implemented.</strong> <code>--metric</code> (renamed <code>--distance</code> — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
<code>--metric</code> was a misnomer) gains a family of <code>snp-*</code> values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
machinery (<code>PairwiseTally</code>, <code>obikphylo::siblings::algorithms::pairwise</code>)
|
||||
instead of <code>cache.distance(...)</code>'s existing per-layer traversal — a different
|
||||
code path behind the same CLI surface, not just another branch of one
|
||||
formula function.</p>
|
||||
<p><strong>Why unify at the CLI level despite the implementation split</strong>: phylogenetically
|
||||
a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
|
||||
to how the matrix was produced, so exposing it as a special-cased subcommand
|
||||
instead of a <code>--distance</code> value would misrepresent its role. The
|
||||
implementation divergence (sibling-annex-based vs. plain index scan) is real
|
||||
but belongs at the routing layer, invisible to the CLI's own vocabulary.</p>
|
||||
<p><strong><code>--subsample</code> becomes optional for <code>snp-*</code> distances</strong> (it stays mandatory
|
||||
for <code>--sankoff</code>/<code>--pseudo-alignment</code>, unrelated commands): absent means
|
||||
exhaustive, achieved for free by reusing <code>sample_index</code>'s existing
|
||||
proportional-per-layer-quota mechanism with <code>n</code> set to the index-wide total
|
||||
non-monomorphic-minorant count (already available from the sibling-annex
|
||||
stats) — every layer's quota then equals its own full count, giving Bernoulli
|
||||
<code>p = 1</code> everywhere, i.e. every eligible family is drawn. No second,
|
||||
exhaustive-only driver needed. Present means sampled, exactly as <code>--sankoff</code>
|
||||
already behaves.</p>
|
||||
<p><strong>One shared tally, many derived formulas.</strong> <code>PairwiseTally</code>'s <code>subst[4][4]</code>
|
||||
per-pair substitution counts (plus marginal base frequencies derived from it)
|
||||
are the sufficient statistic for every closed-form correction below — each
|
||||
is a small pure function <code>PairwiseTally -> Array2<f64></code>, at the same level as
|
||||
the already-implemented <code>raw_snp_distance</code>/<code>base_pair_tally</code>/
|
||||
<code>cardinality_tally</code>. No new full scan per formula, whether the tally itself
|
||||
was built exhaustively or from a subsample.</p>
|
||||
<p><strong><code>--raw-snp-counts</code> stays a separate, unrelated flag</strong> — same underlying
|
||||
tally, but a diagnostic (<code>n_snp</code>/<code>n_shared</code>/<code>n_eligible</code> per genome pair, one
|
||||
row per pair) rather than a distance value, and its long-table shape doesn't
|
||||
fold into a single N×N matrix the way a distance does. No change to its
|
||||
existing CSV format.</p>
|
||||
<h3 id="snp-distance-catalog"><code>snp-*</code> distance catalog</h3>
|
||||
<p>All closed-form (method-of-moments / direct formula), none requiring
|
||||
per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
|
||||
<em>tree</em>-ML usage but not its <em>pairwise</em> estimator, which is closed-form like
|
||||
F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>value</th>
|
||||
<th>corrects for</th>
|
||||
<th>inputs beyond raw counts</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>snp-raw</code></td>
|
||||
<td>nothing (uncorrected p-distance)</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-jc</code> (Jukes-Cantor, JC69)</td>
|
||||
<td>multiple substitutions per site</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k2p</code> (Kimura 2-parameter, K80)</td>
|
||||
<td>+ transition/transversion rate bias</td>
|
||||
<td>ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k81</code> (Kimura 3-parameter, K3ST)</td>
|
||||
<td>+ splits transversions into 2 categories</td>
|
||||
<td>ts/tv split, by category</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f81</code> (Felsenstein 81)</td>
|
||||
<td>+ unequal base frequencies (no ts/tv split)</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tajima-nei</code> (Tajima-Nei 1984)</td>
|
||||
<td>same goal as F81 (equal-input model), different formula, better small-sample behavior</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-t92</code> (Tamura 3-parameter)</td>
|
||||
<td>K2P + GC-content bias</td>
|
||||
<td>ts/tv split, GC content</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f84</code> (Felsenstein 84)</td>
|
||||
<td>full empirical base freqs + single ts/tv rate</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-hky85</code> (Hasegawa-Kishino-Yano, pairwise estimator)</td>
|
||||
<td>same inputs as F84, different formula</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tn93</code> (Tamura-Nei)</td>
|
||||
<td>full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate</td>
|
||||
<td>empirical base freqs, purine-ts/pyrimidine-ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-logdet</code> (LogDet / paralinear)</td>
|
||||
<td>no shared-model or stationarity assumption at all — general divergence-matrix determinant</td>
|
||||
<td>full empirical 4×4 divergence matrix (already <code>subst[4][4]</code>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tv</code> (transversions-only p-distance)</td>
|
||||
<td>diagnostic/deep-divergence variant — drops transitions entirely (they saturate first)</td>
|
||||
<td>tv-only counts</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
||||
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
||||
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> either
|
||||
supplied by the user (<code>--gamma-shape <alpha></code>) or estimated from the data
|
||||
(<code>--gamma-shape auto</code>/<code>estimate</code>, method-of-moments — not ML; see
|
||||
"Automatic α estimation" below). A modifier on existing values, not a
|
||||
separate enum arm per distance.</p>
|
||||
<h3 id="automatic-estimation-gamma-shape-auto">Automatic α estimation (<code>--gamma-shape auto</code>)</h3>
|
||||
<p><strong>Correction (verified against the primary source, 2026-09-11):</strong> Jin &
|
||||
Nei (1990) itself (<em>"Limitations of the Evolutionary Parsimony Method of
|
||||
Phylogenetic Analysis"</em>, Mol. Biol. Evol. 7(2):82–102 — the paper this
|
||||
whole <code>+Γ</code> correction is cited from, confirmed algebraically to match this
|
||||
codebase's <code>corrected_log</code>/<code>k2p</code> exactly against their eq. A4, general, and
|
||||
A8, the <code>a = 1</code> case) contains <strong>no</strong> data-driven α-estimation procedure.
|
||||
Their own recommendation (p. 98) is a fixed default: <em>"we suggest that the
|
||||
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
|
||||
different gamma distance, estimating a from data. Wilson et al. (1989)
|
||||
recently used a distance with a = 1/2 for restriction-site data of
|
||||
mitochondrial DNA in hominoids."</em> — i.e. Jin & Nei explicitly punt
|
||||
data-driven estimation to a <em>different</em> paper (Wilson et al. 1989), not
|
||||
read/verified here. The estimator below is therefore <strong>not</strong> "Jin & Nei's
|
||||
method" under any framing — that attribution (present in an earlier
|
||||
revision of this section) was wrong, not just under-cited.</p>
|
||||
<p><strong>Implemented</strong> (<code>PartitionDispersion</code>, <code>obikphylo/src/siblings/algorithms/pairwise.rs</code>)
|
||||
as an independent method-of-moments estimator, unrelated to any specific
|
||||
published procedure: pools substitution counts by <strong>partition</strong> rather
|
||||
than by genome pair, during the same <code>reduce_pairwise</code> pass that builds
|
||||
<code>PairwiseTally</code> (no second scan).</p>
|
||||
<p>For partition <code>i</code>: <code>n_i</code> = substitutions pooled over every genome pair,
|
||||
<code>L_i</code> = eligible loci pooled over every genome pair, <code>R_i = n_i / L_i</code>.
|
||||
Modeling among-site rate heterogeneity the same way as the <code>+Γ</code> correction
|
||||
itself (a <code>Gamma(α, α)</code>-distributed, mean-1, multiplicative rate shared by
|
||||
every locus in a partition — the classical Poisson-Gamma/negative-binomial
|
||||
mixture, the general identity behind gamma-rate-heterogeneity corrections,
|
||||
also behind Uzzell & Corbin's (1971) original observation that substitution
|
||||
counts across sites/regions are over-dispersed relative to Poisson):</p>
|
||||
<div class="arithmatex">\[
|
||||
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
|
||||
\]</div>
|
||||
<p>Weighting each partition's squared deviation by its own <code>L_i</code> cancels the
|
||||
Poisson term before attributing what's left to <code>α</code>:</p>
|
||||
<div class="arithmatex">\[
|
||||
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
|
||||
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
|
||||
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
|
||||
\]</div>
|
||||
<p>where <code>\bar L</code> is the mean partition size. Returns "no estimate" (falls
|
||||
back to the uncorrected formula, warns) when fewer than 2 partitions have
|
||||
data, <code>\hat\mu \le 0</code>, or <code>V</code> doesn't exceed the Poisson floor
|
||||
<code>\hat\mu/\bar L</code> — no detectable over-dispersion, <code>α</code> would be unbounded.</p>
|
||||
<p><strong>Caveat, stated explicitly rather than left implicit</strong>: unlike every
|
||||
closed-form correction in <code>snp_distance.rs</code> (each verified line-by-line
|
||||
against <code>ape</code>'s <code>dist_dna.c</code>, and now also against Jin & Nei 1990 directly
|
||||
for the base <code>+Γ</code> formula), this estimator is derived from first
|
||||
principles (the general Poisson-Gamma/negative-binomial identity) with no
|
||||
primary-source procedure behind it at all — not Jin & Nei's (confirmed
|
||||
above), and Wilson et al. (1989), the paper they point to instead, hasn't
|
||||
been read/verified either. Mathematically self-consistent (re-derived and
|
||||
checked, not guessed), but a from-scratch method, not a literature
|
||||
implementation. If <code>--gamma-shape</code> needs a value with a literature
|
||||
pedigree rather than an estimated one, Jin & Nei's own stated default,
|
||||
<code>α = 1</code> (<code>--gamma-shape 1</code>), is the better-supported choice today.</p>
|
||||
<p>Deliberately <strong>not</strong> gated by <code>--sankoff-ratio-ceiling</code> the way
|
||||
<code>base_pair_tally</code> is (same precedent as <code>cardinality_tally</code> — see its own
|
||||
doc comment): that filter excludes individual saturated <em>pairs</em> from a
|
||||
composition estimate computed once at the very end, from the complete
|
||||
<code>PairwiseTally</code>; the partition axis needed here only exists transiently,
|
||||
one partition at a time, while <code>PairwiseTally</code> is still being built — long
|
||||
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
|
||||
is known. <code>--exclude-genome</code> isn't applied either, matching
|
||||
<code>reduce_pairwise</code>'s own raw per-pair fold.</p>
|
||||
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
||||
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
||||
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
||||
<code>snp-logdet</code> are catalogued above but <strong>not implemented</strong>: <code>snp-logdet</code>
|
||||
needs the true <em>directional</em> per-pair base co-occurrence matrix
|
||||
(<code>PairwiseTally</code> only keeps the symmetrised substitution counts
|
||||
<code>BasePairTally</code> itself wants — see <code>snp_distance.rs</code>'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), <code>snp-tajima-nei</code> needs each genome's <em>own</em> base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
<code>snp-f84</code>/<code>snp-hky85</code> had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against <a href="https://github.com/emmanuelparadis/ape">ape</a>'s own
|
||||
<code>src/dist_dna.c</code>, not re-derived from memory). Adding any of these later is
|
||||
a new function in <code>obikphylo::siblings::algorithms::snp_distance</code>, plus for
|
||||
<code>snp-logdet</code>/<code>snp-tajima-nei</code> a new field on <code>PairStats</code>/a per-genome
|
||||
accumulator — not an architecture change.</p>
|
||||
<h3 id="exact-formulas-implemented-2026-08-28">Exact formulas (implemented, 2026-08-28)</h3>
|
||||
<p>Sufficient statistic, per genome pair <code>(i, j)</code>, from
|
||||
<code>PairwiseTally::categories</code>/<code>PairwiseTally::base_freq</code> (base order always
|
||||
<code>0=A, 1=C, 2=G, 3=T</code>, matching <code>FamilyMask</code>/<code>STATE_SYMBOL</code>):</p>
|
||||
<ul>
|
||||
<li><span class="arithmatex">\(n_{ts1}\)</span>: A↔G substitutions (purine transitions), <span class="arithmatex">\(n_{ts2}\)</span>: C↔T
|
||||
(pyrimidine transitions)</li>
|
||||
<li><span class="arithmatex">\(n_{tv1}\)</span>: A↔C and G↔T substitutions, <span class="arithmatex">\(n_{tv2}\)</span>: A↔T and C↔G
|
||||
(Kimura's two transversion categories)</li>
|
||||
<li><span class="arithmatex">\(n_{shared}\)</span>: loci where both genomes agree</li>
|
||||
<li><span class="arithmatex">\(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\)</span> (total eligible
|
||||
loci for the pair)</li>
|
||||
<li><span class="arithmatex">\(\pi_A, \pi_C, \pi_G, \pi_T\)</span>: pair-pooled base frequencies,
|
||||
<span class="arithmatex">\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)</span>
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)</li>
|
||||
</ul>
|
||||
<p>Derived proportions used below:</p>
|
||||
<div class="arithmatex">\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]</div>
|
||||
<p>Every formula below was checked term-by-term against <code>ape</code>'s own
|
||||
<code>src/dist_dna.c</code> (not re-derived from memory) before being ported to
|
||||
<code>obikphylo::siblings::algorithms::snp_distance</code>.</p>
|
||||
<p><strong><code>snp-raw</code></strong> — uncorrected p-distance:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{raw} = p
|
||||
\]</div>
|
||||
<p><strong><code>snp-tv</code></strong> — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, <em>is</em> the correction):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{tv} = Q
|
||||
\]</div>
|
||||
<p><strong><code>snp-jc</code></strong> (Jukes-Cantor, JC69):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-k2p</code></strong> (Kimura 2-parameter, K80), with <span class="arithmatex">\(a_1 = 1-2P-Q\)</span>, <span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-k81</code></strong> (Kimura 3-parameter, K3ST), with <span class="arithmatex">\(a_1 = 1-2P-2Q_1\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2P-2Q_2\)</span>, <span class="arithmatex">\(a_3 = 1-2Q_1-2Q_2\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-f81</code></strong> (Felsenstein 81), with <span class="arithmatex">\(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-t92</code></strong> (Tamura 3-parameter), with GC content
|
||||
<span class="arithmatex">\(g = \pi_C+\pi_G\)</span>, <span class="arithmatex">\(w = 2g(1-g)\)</span>, <span class="arithmatex">\(a_1 = 1 - \dfrac{P}{w} - Q\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-tn93</code></strong> (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
<span class="arithmatex">\(g_R = \pi_A+\pi_G\)</span>, <span class="arithmatex">\(g_Y = \pi_C+\pi_T\)</span>, and</p>
|
||||
<div class="arithmatex">\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]</div>
|
||||
<p><strong><code>+Γ</code> gamma correction</strong> (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of <span class="arithmatex">\(-\ln(x)\)</span> terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
<span class="arithmatex">\(\alpha\left(x^{-1/\alpha} - 1\right)\)</span> instead — the standard mechanical
|
||||
substitution (as <span class="arithmatex">\(\alpha \to \infty\)</span>, this expression → <span class="arithmatex">\(-\ln(x)\)</span>,
|
||||
recovering the uncorrected formula exactly). E.g. for JC:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]</div>
|
||||
<p>Verified term-by-term against <code>ape</code>'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to <code>snp-k2p</code>'s own <span class="arithmatex">\(a_1\)</span>/<span class="arithmatex">\(a_2\)</span> terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one <code>corrected_log</code> helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.</p>
|
||||
<h3 id="-estimation-literature-survey-and-design-discussion-2026-09-11">α-estimation literature survey and design discussion (2026-09-11)</h3>
|
||||
<p>Follow-up discussion after implementing <code>PartitionDispersion::estimate_alpha</code>
|
||||
(previous section), triggered by the user pointing out that the literature
|
||||
conflates two distinct problems: <em>estimating α from an alignment</em> versus
|
||||
<em>using a given α in a Jin-Nei-style corrected distance</em>. Jin & Nei (1990)
|
||||
itself is squarely in the second camp — α is a user-supplied parameter
|
||||
there, not something their paper estimates (confirmed by reading the paper
|
||||
directly, see previous section). This section catalogs the actual
|
||||
α-estimation literature and records which approaches were considered and
|
||||
why each was or wasn't adopted.</p>
|
||||
<p><strong>Bibliography</strong> (methodological papers, not exhaustive users of <code>+Γ</code>):</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Method</th>
|
||||
<th>Reference</th>
|
||||
<th>Needs a tree?</th>
|
||||
<th>Estimates "true" α or a task-specific one?</th>
|
||||
<th>Cost</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>ML, continuous Γ (foundational)</td>
|
||||
<td><strong>Gu, Fu & Li (1995)</strong>, MBE 12:546–557 — read in full</td>
|
||||
<td>yes, topology fixed a priori</td>
|
||||
<td>true α (+ θ jointly, invariant+Γ)</td>
|
||||
<td>very high, ≤5–6 taxa in practice</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ML, discrete Γ</td>
|
||||
<td>Yang (1993/94), MBE 10:1396; JME 39:306</td>
|
||||
<td>yes</td>
|
||||
<td>true α</td>
|
||||
<td>high</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Parsimony-based ML on inferred changes</td>
|
||||
<td>Yang & Kumar (1996), MBE 13:650</td>
|
||||
<td>yes, equal branch lengths assumed</td>
|
||||
<td>true α, biased when branches unequal</td>
|
||||
<td>low</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Corrected substitutions/site → NB fit</td>
|
||||
<td><strong>Gu & Zhang (1997)</strong>, MBE 14:1106 — read in full</td>
|
||||
<td>yes, + ancestral reconstruction</td>
|
||||
<td>true α, close to ML accuracy</td>
|
||||
<td>low (given a tree)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Topology-optimizing grid search</td>
|
||||
<td><strong>Guindon & Gascuel (2002)</strong>, MBE 19:534 — read in full</td>
|
||||
<td>yes, but built <em>from</em> the candidate distances themselves (BIONJ)</td>
|
||||
<td><strong>not</strong> true α — deliberately biased <em>upward</em> for topological accuracy</td>
|
||||
<td>low–moderate (grid × tree build)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Free (non-parametric) rate distribution</td>
|
||||
<td>Susko et al. (2003), Syst. Biol. 52:594</td>
|
||||
<td>yes</td>
|
||||
<td>tests the Γ assumption itself</td>
|
||||
<td>high</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Parsimony counts/site → NB fit</td>
|
||||
<td><strong>Wakeley (1993)</strong>, JME 37:613 — read in full</td>
|
||||
<td>yes, NJ tree + Fitch parsimony ancestral reconstruction</td>
|
||||
<td>true α, biased <em>upward</em> by parsimony (conservative)</td>
|
||||
<td>low (given a tree)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Poisson-Gamma/negative-binomial, foundational</td>
|
||||
<td>Uzzell & Corbin (1971), Science 172:1089 — <strong>not read</strong>, only seen via citations</td>
|
||||
<td>unknown — likely also count-per-site/tree-based, unverified</td>
|
||||
<td>true α (indirect, via NB overdispersion)</td>
|
||||
<td>unknown</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Bayesian posterior over α</td>
|
||||
<td>BPP/MrBayes/BEAST2/RevBayes</td>
|
||||
<td>yes</td>
|
||||
<td>posterior, not point estimate</td>
|
||||
<td>very high</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong>Correction (2026-09-11, after reading Wakeley in full):</strong> an earlier
|
||||
revision of this table filed Wakeley (1993) under "no tree needed" —
|
||||
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
|
||||
1987, on the real mtDNA data; coalescent-simulated for the simulation
|
||||
study), reconstruct ancestral states at internal nodes by Fitch (1971)
|
||||
parsimony, count the <em>minimum</em> number of changes per site this implies,
|
||||
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
|
||||
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
|
||||
Johnson & Kotz 1969) to the resulting per-site count distribution — the
|
||||
same general "tree + ancestral reconstruction + per-site counts + NB fit"
|
||||
shape as Gu & Zhang (1997), just using raw parsimony counts directly
|
||||
rather than Gu & Zhang's multiple-hit-corrected <code>k̂</code>. Confirmed by
|
||||
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
|
||||
mean <em>and</em> (more severely) the variance of per-site change counts, which
|
||||
biases <code>α</code> <strong>upward</strong> (toward apparent uniformity) — consistent with Gu &
|
||||
Zhang's own citation of Wakeley for exactly this bias, and with the
|
||||
general "parsimony overestimates α" pattern noted throughout this
|
||||
literature. Real hypervariable-region-1 mtDNA data gave <code>α̂ ≈ 0.44–0.60</code>
|
||||
per data set, <code>0.47</code> combined — despite the conservative bias, still
|
||||
solidly in the "strong heterogeneity" range, which is the paper's own
|
||||
point: the method is biased but usefully conservative, not useless.</p>
|
||||
<p><strong>Where <code>PartitionDispersion::estimate_alpha</code> actually sits</strong>: nowhere in
|
||||
this table's tree-dependent rows. It shares only the <em>abstract
|
||||
mathematical identity</em> (Poisson-Gamma mixture ⟹ negative binomial
|
||||
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
|
||||
genuinely different, tree-free and ancestral-reconstruction-free route:
|
||||
pairwise genome-to-genome substitution counts pooled by <em>partition</em>
|
||||
instead of per-site counts pooled across an alignment's columns after
|
||||
ancestral state reconstruction. As far as this survey has established
|
||||
(Uzzell & Corbin not read, so not fully ruled out), <strong>no tree and no
|
||||
ancestral reconstruction of any kind</strong> appears to be a genuinely
|
||||
distinguishing property of <code>PartitionDispersion</code>, not something it
|
||||
inherits from prior art — consistent with it being an independent
|
||||
derivation rather than a literature implementation (see previous
|
||||
section's caveat, now on firmer footing).</p>
|
||||
<p><strong>Why a tree-free estimate is even valid — the identifiability question.</strong>
|
||||
Raised directly by the user after reading Gu, Fu & Li (1995), which states
|
||||
explicitly: <em>"one cannot estimate the rate heterogeneity when the number
|
||||
of sequences is <3."</em> This is a real non-identifiability result, not a
|
||||
practical inconvenience: with only 2 sequences, a site's history is
|
||||
summarized by one binary observation (differ / don't differ), collapsed
|
||||
across all sites into two aggregate numbers (transition and transversion
|
||||
proportions) — divergence time and among-site rate variance are
|
||||
confounded in that pair of numbers, with no way to separate them. Every
|
||||
method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
|
||||
sequences <em>sharing one evolutionary history</em> (a tree) specifically to get
|
||||
multiple independent looks at the <em>same</em> site's rate across different
|
||||
lineages, which is what identifies its variance.</p>
|
||||
<p><code>PartitionDispersion</code> doesn't violate this, because it isn't estimating
|
||||
the same quantity. The literature above estimates rate variation <strong>across
|
||||
sites, within one shared tree</strong>. <code>PartitionDispersion</code> estimates rate
|
||||
variation <strong>across partitions, pooled over every genome pair in the
|
||||
index</strong> — each genome pair stands in for one independent draw of a
|
||||
partition's relative rate, the same identifying role multiple lineages
|
||||
play in the tree-based methods, just substituting "genome pair" for
|
||||
"lineage" and "partition" for "site." With <code>P</code> genome pairs (potentially
|
||||
in the hundreds here) all contributing to every partition's pooled count,
|
||||
the ≥3-samples identifiability requirement is met by the pair count, not
|
||||
by tree depth — closer in spirit to Jin & Nei's own original motivating
|
||||
scenario (comparing several genomic <em>regions</em> across one fixed panel of
|
||||
taxa) than to single-tree site-rate estimation.</p>
|
||||
<p>This substitution carries its own assumption, which must be named rather
|
||||
than left implicit: that a partition's <em>relative</em> rate (fast/slow) is
|
||||
reasonably stable <strong>across genome pairs</strong>, not just across sites within
|
||||
one pair — the direct analogue of the "no lineage-specific rate variation,
|
||||
only site-specific" assumption every tree-based <code>+Γ</code> method already makes,
|
||||
just moved from lineage→pair and site→partition. If this breaks — e.g. an
|
||||
index mixing very closely related and very divergent genome pairs, where a
|
||||
partition's saturation behavior differs qualitatively between the two —
|
||||
the estimate could be misled in a way a tree-based method would at least
|
||||
have the topology to detect and a tree-free, pool-everything method
|
||||
cannot. Not yet tested against real data with strongly heterogeneous
|
||||
pairwise divergence; worth keeping in mind as the main open validity
|
||||
question for <code>--gamma-shape auto</code>, not the citation question (now settled)
|
||||
this whole discussion started from.</p>
|
||||
<p><strong>Gu, Fu & Li (1995), read in full — the foundational ML method, not a
|
||||
candidate to implement.</strong> Models rate variation as invariant+Γ (<code>θ</code> =
|
||||
proportion of invariant sites, <code>α</code> = gamma shape among the variable sites),
|
||||
and derives the exact joint likelihood over nucleotide <em>configurations</em>
|
||||
across all <code>n</code> sequences on a <strong>fixed, a-priori-known tree topology</strong> —
|
||||
for 3 sequences this is closed-form (Jukes-Cantor), for <code>n</code> sequences it's
|
||||
a sum over <code>4^n</code> configurations weighted by coefficients tied to the
|
||||
specific topology's branching structure (their eq. 32–34), with branch
|
||||
lengths, <code>α</code>, and <code>θ</code> jointly optimized by Newton-Raphson (Hessian over all
|
||||
free parameters). The site-rate integral has the same closed form
|
||||
<code>E = θ + (1-θ)(1+D/α)^{-α}</code> that underlies <code>+Γ</code> distance corrections
|
||||
generally (the same mechanism as <code>corrected_log</code>, not a coincidence — Jin &
|
||||
Nei's own formula is the <code>θ=0</code> special case applied to a pairwise <code>D</code>
|
||||
rather than a whole-tree one). Explicitly stated by the authors as
|
||||
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
|
||||
candidate for obikmer's typical genome counts. Notable contribution worth
|
||||
keeping in mind regardless: <code>ρ = (1+θα)/(1+α)</code> is shown to be a more robust
|
||||
summary of rate heterogeneity than <code>α</code> alone whenever <code>θ</code> isn't negligible
|
||||
— <code>α</code> and <code>θ</code> are confounded (nearly-invariant sites can come from either a
|
||||
small <code>α</code>'s left tail or from a nonzero <code>θ</code>), so <code>α</code> alone can be very
|
||||
unstable while <code>ρ</code> stays well-behaved. <code>PartitionDispersion</code> has no <code>θ</code>
|
||||
term at all (no invariant-site component), so this confound doesn't arise
|
||||
for it the same way, but it's worth remembering if an invariant+Γ variant
|
||||
is ever considered.</p>
|
||||
<p><strong>Gu & Zhang (1997), read in full — rejected for now.</strong> Their procedure:
|
||||
(1) a tree with least-squares branch lengths must already exist; (2)
|
||||
ancestral states are reconstructed at every internal node (they use Zhang &
|
||||
Nei 1997's likelihood method, but note plain parsimony would work, just
|
||||
less accurately); (3) per site, branches are split into "changed"/"unchanged"
|
||||
given the ancestral reconstruction, and the <em>expected</em> substitution count
|
||||
<code>k̂</code> (corrected for multiple hits) is obtained by solving their eq. (7)
|
||||
(or eq. 12 for the generalized model) numerically — a root-find per site,
|
||||
not a closed form; (4) the site-level <code>k̂</code> values (real-valued, not
|
||||
integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
|
||||
get α — again no closed form, numerical optimization. This is a real
|
||||
per-site pipeline requiring a tree <em>before</em> distances/α can be computed —
|
||||
the reverse of obikmer's current dependency order (<code>snp_distance</code> computes
|
||||
distances, which <code>--nj</code>/<code>--upgma</code> only turn into a tree <em>afterward</em>). Their
|
||||
own answer to this chicken-and-egg problem is an iterative
|
||||
distance→tree→α→distance loop, which they themselves flag as expensive.
|
||||
Adopting this would mean a new ancestral-reconstruction subsystem (even a
|
||||
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
|
||||
ML fit — a much bigger addition than <code>PartitionDispersion</code>, not something
|
||||
to build without a concrete need beyond "closer to Gu&Zhang's own
|
||||
simulated accuracy than our moment estimator."</p>
|
||||
<p><strong>Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.</strong>
|
||||
Their <strong>Q criterion</strong>: for a candidate α, build a tree (they use BIONJ) from
|
||||
the α-corrected distance matrix; for every internal branch, group taxa into
|
||||
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
|
||||
distances <span class="arithmatex">\(\bar\delta_{AB}\)</span>, <span class="arithmatex">\(\bar\delta_{CD}\)</span>, etc., and let <code>S ≤ M ≤ L</code>
|
||||
be the three pairings' sums (<code>{AB,CD}</code>, <code>{AC,BD}</code>, <code>{AD,BC}</code>); the branch's
|
||||
reliability score is <code>Q_branch = L − M</code> (zero when the four points are
|
||||
perfectly tree-additive). The whole-tree <code>Q</code> is the mean over internal
|
||||
branches (negative/zero branches excluded). <code>α* = argmin_α Q(α)</code> over a
|
||||
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
|
||||
sensitivity is highest). Complexity: <code>O(n²l)</code> once for the base
|
||||
frequencies, then <code>O(n²r)</code> for the <code>r</code> candidate distance matrices, then
|
||||
<code>O(n³r)</code> worst case for building <code>r</code> trees and evaluating <code>Q</code> on each
|
||||
(same order as the tree-building step itself, so "free" relative to it).</p>
|
||||
<p>Central finding, <strong>the opposite direction from an initial misreading in
|
||||
this discussion</strong>: <code>α_opt</code> (Guindon-Gascuel's topology-optimal value) is
|
||||
<strong>always ≥ the true α</strong>, not the reverse — underestimating rate
|
||||
heterogeneity (i.e., picking a larger α than reality) reduces the variance
|
||||
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
|
||||
especially when the molecular clock roughly holds (where <code>α_opt → ∞</code>, i.e.
|
||||
no correction at all is topologically best). This is an <em>empirical
|
||||
regularity</em> observed across their tested conditions (η ∈ {0.5, 2.0}, true
|
||||
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
|
||||
guarantee would be another unverified-claim mistake of the kind this whole
|
||||
discussion has been correcting.</p>
|
||||
<p><strong>Proposed integration (not yet implemented)</strong>: since <code>α_true ≤ α_opt</code>
|
||||
empirically, <code>PartitionDispersion::estimate_alpha()</code>'s output could seed
|
||||
the <em>lower bound</em> of Guindon-Gascuel's grid (search <code>[α̂, ∞)</code> instead of
|
||||
<code>[0.1, 5000]</code>), narrowing the grid substantially and reusing the existing
|
||||
estimator rather than needing an independent "true α" source — this also
|
||||
sidesteps Gu & Zhang's tree-first dependency problem entirely, since
|
||||
Guindon-Gascuel's own grid already builds a tree per candidate. Still
|
||||
requires: (a) a <strong>BIONJ</strong> implementation (see below), (b) the Q-criterion
|
||||
subtree-grouping logic, (c) a grid-search driver. Not started.</p>
|
||||
<p><strong>BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.</strong>
|
||||
Searched crates.io directly (API query for "bionj": 0 results) and read the
|
||||
docs.rs pages of the most plausible candidates: <code>phylotree</code> (builds/reads/
|
||||
manipulates trees but has no distance-matrix reconstruction — no NJ, no
|
||||
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
|
||||
after checking the actual docs), <code>speedytree</code> (NJ only — Canonical/RapidNJ/
|
||||
Hybrid variants, explicitly no BIONJ), <code>nj</code> (plain NJ only), <code>phylo</code> (no
|
||||
distance-based reconstruction at all). <code>obikphylo::neighbor_joining</code>
|
||||
(<code>tree.rs</code>) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
|
||||
how branch lengths are weighted during agglomeration (accounts for
|
||||
distance-estimate variance, not just the sum), a modest delta over an
|
||||
existing NJ implementation rather than a rewrite, but would need to be
|
||||
written in-house if the Guindon-Gascuel integration above is ever pursued
|
||||
(their own paper notes other tree-building methods gave similar results in
|
||||
their simulations, so plain NJ may be an acceptable substitute if BIONJ
|
||||
proper is judged not worth the effort).</p>
|
||||
<h3 id="sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11">Sharing one sample across algorithms (<code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/<code>--iqtree</code> + <code>snp-*</code> <code>--distance</code>) — implemented 2026-09-11</h3>
|
||||
<p><strong>Correctness bug, not just a performance one.</strong> <code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/
|
||||
<code>--iqtree</code> (via <code>SiblingExt::sankoff_bundle</code>) and a <code>snp-*</code> <code>--distance</code>
|
||||
(via <code>SiblingExt::snp_distance</code>) always consume the <em>exact same</em> selection
|
||||
parameters when requested together in one <code>obikmer phylo</code> invocation
|
||||
(<code>n</code>/<code>free_loss</code>/<code>no_ambiguity</code>/excluded-set/<code>entropy_bias</code> — the CLI has
|
||||
no way to give them different values in one run, see <code>cmd/phylo/args.rs</code>).
|
||||
Before this fix, each independently called <code>sample_index</code>, and
|
||||
<code>sample_layer</code>'s <code>rand::rng()</code> (<code>subsample.rs:235</code>) is a <strong>thread-local
|
||||
generator that advances across calls, not reseeded each time</strong> — so the
|
||||
second call silently drew a <em>different</em> random sample of sites than the
|
||||
first, even with identical parameters. Verified empirically (throwaway
|
||||
8-genome index): running <code>snp-jc</code> alone vs. combined with <code>--sankoff</code>
|
||||
produced visibly different distance matrices from the same index/params.
|
||||
This defeated the actual point of combining these flags — running several
|
||||
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a <code>snp-*</code> distance) on
|
||||
one <em>identical</em> site selection for direct comparison.</p>
|
||||
<p><strong>Fix</strong>: <code>SankoffBundle</code> (<code>sankoff.rs</code>) now retains its <code>PairwiseTally</code>/
|
||||
<code>PartitionDispersion</code> internally (previously consumed into <code>raw</code>/
|
||||
<code>base_pair_tally</code>/<code>cardinality_tally</code> and dropped) and exposes
|
||||
<code>SankoffBundle::snp_distance(kind, gamma_shape)</code>, which computes the
|
||||
matrix from that <em>same</em> already-sampled tally — no second <code>sample_index</code>
|
||||
call. The shared post-sampling logic (the <code>--gamma-shape</code> support check,
|
||||
<code>alpha</code> resolution, final matrix build — previously all inline in the
|
||||
<code>snp_distance()</code> free function) was factored into
|
||||
<code>pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)</code> in
|
||||
<code>snp_distance.rs</code>, called by both the standalone <code>snp_distance()</code> (after
|
||||
its own fresh sampling) and <code>SankoffBundle::snp_distance</code> (reusing the
|
||||
bundle's). <code>cmd/phylo/mod.rs</code> keeps the <code>Option<SankoffBundle></code> alive past
|
||||
the Sankoff-family <code>if</code> block and, when a <code>snp-*</code> <code>--distance</code> is also
|
||||
requested, calls <code>bundle.snp_distance(...)</code> instead of
|
||||
<code>cache.snp_distance(...)</code> whenever a bundle was built — since both branches
|
||||
are driven by the same <code>args.*</code> fields, the parameters trivially always
|
||||
match when both fire; no runtime "do the params match" check needed.
|
||||
Verified end-to-end: the reused path logs <code>(reusing the Sankoff bundle's
|
||||
sample)</code> and the <code>snp_distance</code> stage timer reads <code>0ms</code> (<code>formula(kind)</code>
|
||||
is <code>O(n²)</code> post-processing, no I/O), vs. ~150-230ms for a fresh sample on
|
||||
the same tiny test index.</p>
|
||||
<h4 id="explicit-session-implemented-2026-09-12">Explicit <code>--session</code> (implemented 2026-09-12)</h4>
|
||||
<p>The in-process fix above only covered reuse <em>within one command</em>.
|
||||
<code>--session DIR</code> (<code>cmd/phylo/args.rs</code>) names a directory (outside the
|
||||
index) that persists the <code>snp-*</code> <code>--distance</code> sample — the
|
||||
<code>PairwiseTally</code>/<code>PartitionDispersion</code> pair <code>sample_index</code> would otherwise
|
||||
rebuild from scratch — across <em>separate</em> <code>obikmer phylo</code> invocations.</p>
|
||||
<p><strong>Crate placement, as planned</strong>: a new <strong><code>obiksession</code></strong> crate
|
||||
(<code>src/obiksession/</code>), domain-agnostic — no <code>PairwiseTally</code>/"site" concept
|
||||
anywhere in it. <code>Session::open(dir, params: &[u8], force: bool)</code> handles
|
||||
directory lifecycle and the params-conflict check against an opaque byte
|
||||
blob; <code>Session::store(name, bytes)</code>/<code>Session::restore(name) -> Option<Mmap></code>
|
||||
handle a checksummed artifact cache. <code>obikphylo</code>
|
||||
(<code>siblings/algorithms/session_cache.rs</code>, shared by <code>snp_distance.rs</code> and
|
||||
<code>sankoff.rs</code> — see below) is the consumer: it serializes its own
|
||||
<code>PairwiseTally</code>/<code>PartitionDispersion</code>/<code>SnpAlignment</code> to bytes (via <code>rkyv</code>)
|
||||
and hands them to <code>obiksession</code>, which never sees their type.</p>
|
||||
<p><strong>Serialization</strong>: <code>rkyv</code> 0.8.18, added as planned. One deviation from the
|
||||
original zero-copy pitch, scoped down deliberately: artifacts are restored
|
||||
via <code>rkyv::from_bytes</code> (full owned deserialize) rather than <code>rkyv::access</code>
|
||||
(zero-copy over the <code>mmap</code>). True zero-copy would require every
|
||||
<code>PairwiseTally</code> query method (<code>categories</code>, <code>base_freq</code>, <code>pair</code>, ...) to
|
||||
work generically over <code>Archived<PairwiseTally></code> as well as the owned type
|
||||
— a separate, larger change not needed to get the actual win (skipping
|
||||
<code>sample_index</code>'s expensive re-scan of the sibling annex; deserializing an
|
||||
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
|
||||
<code>#[derive(Archive, Serialize, Deserialize)]</code> was added to <code>PairStats</code>,
|
||||
<code>PairwiseTally</code>, and <code>PartitionDispersion</code> (<code>pairwise.rs</code>) for this.</p>
|
||||
<p><strong>Startup behavior, as designed</strong>: <code>Session::open</code> creates <code>DIR</code> if
|
||||
missing (fresh, params recorded); if <code>DIR</code> already holds different saved
|
||||
params, <code>cmd/phylo</code> reports a hard error and exits rather than silently
|
||||
preferring one side, unless <code>--session-force</code> is given (which discards the
|
||||
directory's cached artifacts and starts over under the new params) —
|
||||
implements exactly the "explicit escape hatch, not silent override"
|
||||
decision from the design discussion. <code>obiksession</code> itself doesn't decode
|
||||
either side's bytes to produce a field-by-field diff (true to "opaque
|
||||
blob," left to whoever needs it); <code>cmd/phylo</code>'s conflict message is
|
||||
currently a single generic sentence listing which flags could be the
|
||||
cause, not a computed diff — an honest v1 simplification, not a limitation
|
||||
of <code>obiksession</code> itself.</p>
|
||||
<p><strong>Per-artifact caching, as designed</strong>: only <code>PairwiseTally</code>/
|
||||
<code>PartitionDispersion</code> are cached (<code>snp_distance.rs</code>'s <code>restore_tally</code>/
|
||||
<code>store_tally</code>) — <code>ratio_ceiling</code>/<code>gamma_shape</code> remain pure post-processing
|
||||
over whatever tally is in hand, cached or fresh, exactly as planned.</p>
|
||||
<p><strong>Locking, reconsidered and kept as the whole-session <code>DirLock</code>.</strong> First
|
||||
written up as a corner cut ("per-entry locking was sketched, not
|
||||
implemented"). On actually working through the concrete race, whole-session
|
||||
locking turned out to be the <em>correct</em> choice, not a simplification:
|
||||
<code>--session-force</code> resets a session by deleting its whole <code>artifacts/</code>
|
||||
directory. If the directory lock were only held briefly (during
|
||||
<code>Session::open</code>, then released so a long-running <code>Session</code> doesn't block
|
||||
other processes from touching <em>different</em> artifacts), a second process
|
||||
calling <code>open(..., force: true)</code> could acquire the lock, see mismatched
|
||||
params, and wipe <code>artifacts/</code> <strong>while a first process's already-open
|
||||
<code>Session</code> is mid-checkpoint</strong> — genuine corruption risk, not just wasted
|
||||
work. Holding the lock for the <code>Session</code>'s entire lifetime makes that
|
||||
sequence impossible: a <code>force</code> reset must wait for every currently-open
|
||||
<code>Session</code> on that directory to be dropped first. The concurrency this
|
||||
gives up (two <em>different</em> processes reading/writing <em>different</em> artifacts
|
||||
of the same session at once) isn't exercised by anything today — only one
|
||||
code path (<code>snp_distance</code>/<code>sankoff_bundle</code>, sequentially per invocation)
|
||||
uses sessions at all.</p>
|
||||
<p><strong>Per-layer chunked/resumable dumps — implemented.</strong> <code>subsample::sample_index</code>
|
||||
gained a <code>skip_layers: usize</code> parameter (skip the first N (partition,
|
||||
layer) pairs, in its own deterministic iteration order, entirely — no
|
||||
I/O, no <code>on_layer</code> call) and its <code>on_layer</code> callback gained a <code>raw_index</code>
|
||||
argument (that pair's 0-based position in the same order), so a caller can
|
||||
tell exactly how far a run has gotten. <code>snp_distance</code>/<code>sankoff_bundle</code>
|
||||
checkpoint their tally/dispersion (and, for <code>sankoff_bundle</code>, the
|
||||
in-progress <code>SnpAlignment</code>) to the session every
|
||||
<code>session_cache::CHECKPOINT_INTERVAL_LAYERS</code> (8) layers or
|
||||
<code>CHECKPOINT_INTERVAL</code> (30s) of wall time, whichever comes first — bounding
|
||||
how much completed scanning a crash can lose, without paying a full
|
||||
tally-rewrite on every one of what can be thousands of layers. A <code>progress</code>
|
||||
artifact records how many layers are reflected in the checkpoint, with a
|
||||
<code>PROGRESS_COMPLETE</code> (<code>u64::MAX</code>) sentinel once every layer has been
|
||||
processed (distinct from any real layer count, so "finished" is never
|
||||
confused with "stopped after N layers where N happens to equal the total").</p>
|
||||
<p>Verified with a real kill: built a 10-genome/64-partition index, ran
|
||||
<code>--distance snp-jc --session DIR</code> under <code>timeout 0.4</code>, confirmed a
|
||||
checkpoint at <code>progress=8</code> (exactly <code>CHECKPOINT_INTERVAL_LAYERS</code>) was on
|
||||
disk, then re-ran the same command and confirmed it resumed from layer 8
|
||||
and completed successfully — 0-effort verification that the mechanism
|
||||
itself is real, not just plausible-sounding.</p>
|
||||
<p><strong>Important correction from that same test — resume is <em>not</em> bit-reproducible,
|
||||
and shouldn't be.</strong> The output matrix from the resumed run differed from
|
||||
an uninterrupted control run against the same index/params. Root cause:
|
||||
<code>sample_layer</code>'s random draws come from <code>rand::rng()</code>, a per-process,
|
||||
OS-seeded generator with no continuity across a process boundary — the
|
||||
layers processed before a kill and the layers processed after a resume
|
||||
come from two independent random streams. An earlier revision of this doc
|
||||
(and of the code comments) claimed resume "reproduces the exact same
|
||||
sample an uninterrupted run would have" — that's wrong, and was corrected
|
||||
after the user pointed out the actual reasoning error: true randomness has
|
||||
no notion of continuity in the first place, so there was never a
|
||||
correctness requirement for a resumed run to match an uninterrupted one
|
||||
bit-for-bit. The sample produced after a resume is exactly as legitimate a
|
||||
random draw as one from an uninterrupted run — just a <em>different</em> one,
|
||||
same as running the command twice without <code>--session</code> already gives two
|
||||
different samples today. That variance is explicitly wanted (see below),
|
||||
not a defect to fix. Comments in <code>subsample.rs</code>/<code>session_cache.rs</code> were
|
||||
corrected to state this plainly instead of the false reproducibility claim.</p>
|
||||
<p><strong>Explicitly declined: deterministic/seeded sampling.</strong> Raised as a
|
||||
possible fix for the above (seed <code>sample_layer</code>'s RNG from a per-session
|
||||
value plus <code>(partition, layer)</code>, making a resume reproduce bit-identical
|
||||
results, and, if applied everywhere, making <em>any</em> two runs with identical
|
||||
parameters produce identical samples). The user explicitly declined this
|
||||
for now: repeated runs are relied on to measure sampling variance, and a
|
||||
<code>--seed</code> flag would need its own explicit design if wanted later — this is
|
||||
not an accidental gap, it's a stated preference to revisit only if asked.</p>
|
||||
<p><strong><code>SankoffBundle</code> integration — implemented.</strong> <code>sankoff_bundle</code> accepts
|
||||
the same <code>session: Option<&Session></code> and uses the <em>same</em> artifact names
|
||||
(<code>session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}</code>)
|
||||
as <code>snp_distance</code>, plus its own <code>ALIGNMENT_ARTIFACT</code> (<code>sankoff.rs</code>) folded
|
||||
in lockstep with the tally at every checkpoint (same <code>on_layer</code> call
|
||||
produces both, so they can never drift out of sync). Consequence verified
|
||||
directly: a tally checkpointed by <code>--sankoff --session DIR</code> is restored by
|
||||
a <em>later, separate</em> <code>--distance snp-k2p --session DIR</code> invocation with no
|
||||
<code>--sankoff</code> at all, and vice versa — cross-consumer reuse, not just
|
||||
same-command reuse, exactly the synergy this design aimed for.</p>
|
||||
<p><strong>Per-artifact caching, as designed</strong>: only the tally/dispersion (plus,
|
||||
for <code>sankoff_bundle</code>, the alignment) are cached — <code>ratio_ceiling</code>/
|
||||
<code>gamma_shape</code> remain pure post-processing over whatever tally is in hand,
|
||||
cached or fresh, exactly as planned.</p>
|
||||
<p><strong>Still not done — genuine follow-up, not a justified tradeoff:</strong>
|
||||
<strong>exhaustive-sample caching for <code>--subsample</code>-varying exploration.</strong> The
|
||||
design discussion's suggestion — cache the exhaustive (<code>n = None</code>) sample
|
||||
once and subsample <em>from</em> it on each run instead of one session per exact
|
||||
<code>n</code> — wasn't implemented; a session today is still scoped to one exact
|
||||
parameter tuple, <code>--subsample</code> included.</p>
|
||||
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
||||
<p><strong>Implemented.</strong> The primary distance-matrix output
|
||||
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
||||
default</strong> (widely read by external NJ tools — PHYLIP <code>neighbor</code>, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
label truncation, since genome labels here routinely exceed it), a <code>--csv</code>
|
||||
flag opts back into the current CSV format, PHYLIP-strict is a possible
|
||||
future addition (not now). This changes the <em>default</em> output of every
|
||||
existing <code>--distance</code> value (jaccard, hamming, bray-curtis, ...), not just
|
||||
the new <code>snp-*</code> ones — accepted explicitly (pre-release, single developer
|
||||
user, no external consumers to break). Scoped to the distance matrix only:
|
||||
<code>--shared-kmers</code> and <code>--raw-snp-counts</code> are counts, not distances, and keep
|
||||
their existing CSV-only format.</p>
|
||||
<h2 id="references">References</h2>
|
||||
<p>The Mash mutation-rate model this discussion contrasts with:
|
||||
(Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1199,7 +1199,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1243,7 +1243,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -304,10 +304,29 @@ step, capped by that node's remaining dormant workers and by the run's total
|
||||
budget (`n_total`) — balance across nodes is now guaranteed by construction,
|
||||
not incidental to channel implementation details.
|
||||
|
||||
## Panic propagation (fixed, 2026-08-28)
|
||||
|
||||
A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
|
||||
— the controller's `while completed < n_total` loop then waited forever for
|
||||
an event that partition could no longer produce, since other live
|
||||
workers/timer threads kept `event_rx` open. Discovered via `obikselect`
|
||||
panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
|
||||
the process hung indefinitely instead of erroring out.
|
||||
|
||||
Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
|
||||
`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
|
||||
partition. The controller counts it toward `completed` (unblocking the loop)
|
||||
and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
|
||||
re-raises it on the caller's thread — the original message/backtrace still
|
||||
surfaces, from the right place, instead of a silent deadlock. Takes priority
|
||||
over a plain `Err` (a panic means a bug, not a normal typed failure).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Error handling**: `run` currently returns the first error; remaining errors
|
||||
are dropped. A `Vec<E>` return would give complete diagnostics.
|
||||
are dropped. A `Vec<E>` return would give complete diagnostics. (Panics are
|
||||
now caught and re-raised individually — see above — this only concerns
|
||||
plain `Err` results.)
|
||||
|
||||
- **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
|
||||
(start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an
|
||||
|
||||
+160
-1024
File diff suppressed because it is too large.
Load diff
@@ -12,8 +12,9 @@ this pipeline.
|
||||
by `verify_presence`/`verify_merge_presence`, which only check index
|
||||
*content* against the `.npz` truth, never the query API.
|
||||
- `pack --sparse` produces a presence-matrix format documented (see
|
||||
[siblings.md](../architecture/siblings.md)) as faster for single-row
|
||||
access (query) and slower for column-oriented access (phylo `--metric`).
|
||||
[obicompactvec.md](obicompactvec.md), "Sparse matrix formats") as faster
|
||||
for single-row access (query) and slower for column-oriented access
|
||||
(phylo `--metric`).
|
||||
`global_index_presence/` built by `merge_presence.sh` is always packed
|
||||
dense (packing is a stage inside `merge`, not a separate `pack`
|
||||
invocation) — there was no dense/sparse regression check.
|
||||
|
||||
@@ -299,7 +299,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
|
||||
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
|
||||
```
|
||||
|
||||
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
|
||||
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this goes through `nonzero_iter` instead, decoding one row at a time; see [obicompactvec.md](obicompactvec.md), "Sparse matrix formats".
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -194,3 +194,14 @@ spectrums/
|
||||
<label>.json ← one file per genome, rebuilt from all sources
|
||||
index.meta ← complete genome list + evidence kind written at bootstrap
|
||||
```
|
||||
|
||||
`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json` marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — `merge_partition` never rewrites them for pre-existing layers, only the `presence`/`counts` subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
|
||||
|
||||
## Known issue (not yet fixed, 2026-08-28)
|
||||
|
||||
Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. `obikmer merge -o out IDX IDX --rename-duplicates`) — panics deep in the MPHF's rank-select structure (`common_traits::select_in_word`, `assertion failed: rank < self.count_ones()`), inside `MphfLayer::find` called from `merge_partition`'s "is this source kmer already in dst" check against the bootstrap-copied `dst_layers`. Root cause not identified; ruled out so far:
|
||||
|
||||
- **Not an empty-new-layer issue**: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
|
||||
- **Not caused by the hard-link change above**: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
|
||||
|
||||
Only reproduces when `sources[0]` and `sources[1]` are the exact same on-disk path opened as two independent `KmerIndex` handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since `PartitionRunner`'s panic propagation fix (see `architecture/numa_partition_runner.md`) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.
|
||||
@@ -298,13 +298,11 @@ Both matrix types are enums behind a transparent API — the caller never matche
|
||||
|---|---|---|
|
||||
| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
|
||||
| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
|
||||
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off |
|
||||
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off |
|
||||
| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
|
||||
|
||||
`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
|
||||
|
||||
Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation.
|
||||
|
||||
`col_view(c)` returns the appropriate view directly:
|
||||
|
||||
```rust
|
||||
@@ -321,6 +319,163 @@ No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view
|
||||
|
||||
---
|
||||
|
||||
## Sparse matrix formats
|
||||
|
||||
Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||
k-mers — present in only a handful of genome columns out of dozens to
|
||||
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||
formats above, built once per layer like the rest of the pipeline (never the
|
||||
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||
Structures for Collections of Sets" (2025,
|
||||
`biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf`) — a plain dedup dictionary, not their full subset-containment
|
||||
hierarchy.
|
||||
|
||||
### `PersistentSparseBitMatrix`
|
||||
|
||||
Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row
|
||||
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||
(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width
|
||||
array for multi-genome rows (`dict_id`, `ceil(log2(n_distinct_multi_sets))`
|
||||
bits — kept apart from the singleton array because `n_distinct_multi_sets`
|
||||
can be large in absolute terms even when multi-genome rows are a small
|
||||
*fraction* of all rows, and a shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||
blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary,
|
||||
runtime-parameterised bit width, width `0` included — needed once an
|
||||
`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec`
|
||||
(rank1/rank0/select1 on top of the crate's existing `count_ones`, via
|
||||
`common_traits::SelectInWord`), `EliasFano` (composes the two). `BinaryMatrix`
|
||||
(`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, `count_ones`) is the
|
||||
trait unifying dense and sparse at the call sites that need both
|
||||
interchangeably; column-oriented methods (`col`, `col_view`, the
|
||||
`partial_*_dist_matrix` family) stay dense-only — `Sparse`/`Implicit` panic
|
||||
there (see the `PersistentBitMatrix::open` note above).
|
||||
|
||||
Measured on real data (one layer, 30.2M rows, 91 genomes):
|
||||
|
||||
| | dense | sparse | ratio |
|
||||
|---|---|---|---|
|
||||
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
|
||||
| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction |
|
||||
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) |
|
||||
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** — the `--shannon`/family-lookup access shape |
|
||||
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit |
|
||||
|
||||
The row-access wins (compactness was the stated design goal, not speed) come
|
||||
from dense's genome-major layout scattering a single row read across a much
|
||||
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||
is this much smaller. Column access is the flip side of the same layout
|
||||
choice; `nonzero_iter` (below) is what removes it from the hot path that
|
||||
actually matters (`query`).
|
||||
|
||||
Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open`
|
||||
re-derived its low-bits width from the persisted low-vector's own width
|
||||
byte; the width-`0` case was built with a dummy 1-bit placeholder (the
|
||||
builder used to reject true width `0`), silently doubling every reopened
|
||||
value. Fixed by making `PersistentFixedIntVec` genuinely support width `0`
|
||||
(no storage, `get` always `0`). (2) An empty row (cardinality 0) used to be
|
||||
recorded as a singleton at genome `0`, indistinguishable on read-back from a
|
||||
*real* singleton at genome `0`. Fixed by routing cardinality-0 rows through
|
||||
the dictionary path (a genuine empty entry).
|
||||
|
||||
Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||
exact-duplicate case is built), a sparse `PersistentCompactIntMatrix`
|
||||
variant beyond what's described below, BRWT-style column-correlation
|
||||
exploitation, and a row-major rewrite of
|
||||
`partial_jaccard_dist_matrix`/`partial_hamming_dist_matrix`/etc. as a
|
||||
per-row co-occurrence accumulation (`O(Σ_rows k̄²)`) so `obikindex`'s
|
||||
`--metric` distance-matrix path stops paying the column-access penalty on
|
||||
`Sparse` — needs its own design pass (how it plugs into
|
||||
`BitPartials`/`ColumnWeights` so both matrix types keep serving `--metric`),
|
||||
not just "port the loop".
|
||||
|
||||
### `nonzero_iter` — the one traversal per format
|
||||
|
||||
`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a>`
|
||||
(item: index into `slots`, column, value) is the single foundational
|
||||
per-format traversal primitive every consumer builds on — `fill_sub_matrix`
|
||||
and a `for_each`-style callback are both trivial, free consumers of it
|
||||
(`.for_each(f)` is `Iterator::for_each`; `fill_sub_matrix` drains the
|
||||
iterator and scatters into `out[][]`), instead of two independently
|
||||
hand-maintained traversals that can silently diverge. `Box<dyn Iterator>`,
|
||||
not a bare `impl Iterator`: `Columnar`/`Packed`/`Sparse`/`Implicit` are
|
||||
genuinely different concrete types and the method isn't on a
|
||||
dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix`
|
||||
elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per
|
||||
`nonzero_iter` call, not per cell.
|
||||
|
||||
Per-format implementation: `PersistentSparseBitMatrix` decodes natively,
|
||||
buffered one row at a time (`std::iter::from_fn` over the existing
|
||||
`for_each_genome_in_row`, no `n_cols`-wide allocation ever). `Columnar`/
|
||||
`Packed` loop columns, collecting each column's hits via
|
||||
`BitSliceView::nonzero_among_sorted`/`IntSliceView::nonzero_among_sorted`
|
||||
(a `filter`/`filter_map` over the vector-level `get()`, no new state
|
||||
machine — `std`'s own `Filter` iterator already is one) into one `Vec` via
|
||||
`.extend()` (not `flat_map`: a `flat_map` closure can't lazily return
|
||||
something borrowing its own captured sort permutation across separate calls
|
||||
without either boxing per-column or fighting the borrow checker — eager
|
||||
collection sidesteps it at zero extra cost, since `fill_sub_matrix` already
|
||||
fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))`).
|
||||
|
||||
This closed a real dispatch bug by construction, not by patching it:
|
||||
`PersistentBitMatrix::fill_sub_matrix`/`sub_matrix` used to reimplement a
|
||||
naive per-`(column, slot)` loop instead of delegating to
|
||||
`PersistentSparseBitMatrix`'s own efficient native method for `Self::Sparse`
|
||||
— two hand-written traversals for the same format, free to drift apart, and
|
||||
they had (`obikphylo`'s own cache layer had independently built a parallel
|
||||
enum wrapper that *did* dispatch correctly, evidence the bug was worked
|
||||
around rather than fixed at the source). With `fill_sub_matrix`/`sub_matrix`
|
||||
rewritten to drain `nonzero_iter`, there is only one traversal per format
|
||||
left to get right. `KmerPartition::query_partition_with`
|
||||
(`obikindex/src/partition/query_layer.rs`) — previously column-major
|
||||
(`for g { for slot { col_value } }`, which for `Sparse` rebuilt a full row
|
||||
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||
motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)`
|
||||
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||
reproducibly; now within ~1-3% either way — noise level).
|
||||
|
||||
### `PersistentSparseCompactIntMatrix`
|
||||
|
||||
The count-matrix analogue, built on `PersistentSparseBitMatrix` as its
|
||||
"which columns are non-zero" support, values **not** deduplicated (unlike
|
||||
the bit side, two rows can share the same non-zero column set — same
|
||||
`dict_id` — while carrying different counts, so the dict-multiplicity
|
||||
shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry
|
||||
over). `CountPartials` here: a single row-major pass (`row_major_pairwise`
|
||||
in `sparse_intmatrix.rs`, decodes each row once via
|
||||
`for_each_cell_in_row`, nests over that row's own co-present columns) —
|
||||
`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)`
|
||||
column-pair rescan. `min(a,b)`-based kernels (`bray`, `relfreq_bray`) need
|
||||
no correction — they vanish when either side is absent. `a·b` and `√(a·b)`
|
||||
kernels (`euclidean`/`relfreq_euclidean`/`hellinger`) do need one, since
|
||||
`(a-0)² = a² ≠ 0` unlike the `min`-based formulas — reconstructed from
|
||||
per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`.
|
||||
`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard`
|
||||
(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form
|
||||
(every `u32` is `≥ 0`).
|
||||
|
||||
Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in
|
||||
`BitPartials for PersistentSparseBitMatrix` (present since that type's own
|
||||
introduction, never caught because no test had compared `Sparse`'s raw
|
||||
`partial_*` output against dense on real data — only the diagonal-blind
|
||||
`jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested),
|
||||
fixed at the source:
|
||||
|
||||
1. `partial_jaccard`'s diagonal used to be `(0, 2×col_weights[i])` instead
|
||||
of a genuine self-comparison `(col_weights[i], col_weights[i])` —
|
||||
`col_weights_and_pair_counts`'s `inter` never pairs a column with itself
|
||||
by construction, so the diagonal needs its own case.
|
||||
2. `partial_hamming`'s off-diagonal formula itself was wrong: it computed
|
||||
`total - union` (rows where *neither* column is present) instead of the
|
||||
actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]`
|
||||
(symmetric-difference size) — the two only coincide when
|
||||
`col_weights[i] + col_weights[j] == total`, which small/synthetic test
|
||||
data can easily satisfy by accident.
|
||||
|
||||
## Aggregation traits (matrix level)
|
||||
|
||||
### ColumnWeights
|
||||
|
||||
@@ -23,28 +23,37 @@ are preserved unchanged; only the data matrices are rewritten.
|
||||
|
||||
```sh
|
||||
obikmer select <input-index>
|
||||
{ --output <dir> | --in-place }
|
||||
--output <dir>
|
||||
[--group <name>:<pred> ...]
|
||||
[--group-op <name>:<op> ...]
|
||||
[--aggregate-by <key> ]
|
||||
[--aggregate-op <op> ]
|
||||
[--select <col1,col2,...> ]
|
||||
[--presence-threshold <N> ]
|
||||
[--force-copy ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output destination
|
||||
|
||||
Exactly one of `--output` or `--in-place` must be specified.
|
||||
`--output <dir>` is required — `select` always writes a new index; there is no
|
||||
`--in-place` mode (2026-08-28: never implemented, removed from the design). The
|
||||
source index is unchanged.
|
||||
|
||||
**`--output <dir>`** — writes a new index to `<dir>`. The source index is
|
||||
unchanged. The MPHF and unitig files are copied; only the data matrices are
|
||||
rewritten with the new column layout.
|
||||
Each layer's kmer-identity files (`mphf.bin`/`unitigs.bin`/`evidence.bin`/
|
||||
`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json`) are never rewritten by a
|
||||
column projection/aggregation, so they are hard-linked into the output rather
|
||||
than copied — no extra disk for them even on a large index. Falls back to a
|
||||
real copy automatically if linking fails (different filesystems); `--force-copy`
|
||||
forces a real copy always, for an output that must survive independently of the
|
||||
source on disk (a hard link shares the same inode — rewriting one path outside
|
||||
`select` itself would affect the other). Only the `presence`/`counts`
|
||||
subdirectory is ever a genuinely new, independent file.
|
||||
|
||||
**`--in-place`** — rewrites the data matrices of the source index directly.
|
||||
Removed or replaced columns are lost. The operation writes to temporary files
|
||||
first, then renames atomically, so an interrupted run leaves the index intact.
|
||||
To replace an index with a selected version of itself, select to a temporary
|
||||
directory and swap it in (`rm -rf INDEX && mv INDEX.tmp INDEX`) — the case
|
||||
`--in-place` used to cover.
|
||||
|
||||
---
|
||||
|
||||
@@ -186,12 +195,6 @@ obikmer select myindex --output out \
|
||||
--select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
|
||||
```
|
||||
|
||||
### In-place: keep only group A
|
||||
|
||||
```sh
|
||||
obikmer select myindex --in-place --group "A:group=A" --select "A"
|
||||
```
|
||||
|
||||
### Compose with filter
|
||||
|
||||
```sh
|
||||
@@ -207,28 +210,45 @@ obikmer select filtered --output final --aggregate-by site
|
||||
|
||||
## Implementation notes
|
||||
|
||||
`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
|
||||
(rayon), each writing its output independently; results require no synchronisation
|
||||
because every partition owns a distinct set of files.
|
||||
`select` does not rebuild the MPHF. Every partition is processed independently
|
||||
(`PartitionRunner`), each writing its own output layers; no cross-partition
|
||||
synchronisation is needed.
|
||||
|
||||
For each layer in each partition:
|
||||
For each layer in each partition (`obikselect::select_layer::select_partition`):
|
||||
|
||||
1. The slot count `n` is read by opening the source data matrix.
|
||||
2. A new data matrix is built with M columns (M = number of output columns).
|
||||
3. For each slot `s` in `0..n`:
|
||||
- `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
|
||||
- For each output column `j`:
|
||||
- `new_row[j] = aggregate(op, old_row[group_indices])`.
|
||||
- Pass-through columns are represented as single-element groups with the
|
||||
default operator (`any` for presence, `sum` for count) — same code path.
|
||||
- The new row is written slot by slot into each column builder.
|
||||
4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
|
||||
evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
|
||||
or `counts/` subdirectory is rewritten.
|
||||
1. `copy_layer_files` hard-links the source layer's kmer-identity files
|
||||
(`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/
|
||||
`fingerprint.bin`/`layer_meta.json`) into the destination — never a real
|
||||
copy unless linking fails or `--force-copy` is given.
|
||||
2. A new data matrix is built with M columns (M = number of output columns),
|
||||
under a fresh `presence/`/`counts/` subdirectory (never touching the
|
||||
source's own).
|
||||
3. **Presence source (2026-08-28: `batch_presence_counts`)**: one shared pass
|
||||
over the source bit matrix computes every output group's presence count at
|
||||
once — row-major native for a `Sparse` source (`for_each_genome_in_row`,
|
||||
which has no column representation to read a `col_view` from at all — the
|
||||
reason this replaced the old per-group loop, not just an optimisation of
|
||||
it), deduplicated column-major (one `col_view` per *distinct referenced
|
||||
column*, not per group) for `Columnar`/`Packed`. Every `AggOp` for a bit
|
||||
matrix is then a cheap derivation of that one count vector (`sum` = the
|
||||
count itself, `any`/`max` = `count ≥ 1`, `all`/`min` = `count == group
|
||||
size`, `none` = `count == 0`) — see
|
||||
`obikselect::select_layer::agg_result_from_count`.
|
||||
4. **Count source**: unchanged, one `col_view`-driven pass per output column
|
||||
via `MatrixGroupOps` — `sum`/`min`/`max` are genuine per-value reductions
|
||||
for a count matrix, not derivable from a single presence count the way
|
||||
they are for a bit matrix.
|
||||
5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
|
||||
|
||||
**`--in-place` write strategy:** new data is written to a temporary sibling
|
||||
directory (`presence_new/` or `counts_new/`); on success the old directory is
|
||||
removed and the temporary one is renamed into place. An interrupted run leaves
|
||||
at most one stale `*_new/` directory; the original data is intact until the
|
||||
rename step.
|
||||
### Known gap (not yet fixed, 2026-08-28)
|
||||
|
||||
Step 4 above still panics (`col_view() not available on Sparse
|
||||
PersistentCompactIntMatrix`) if the source is a **count** index packed
|
||||
sparse — `batch_presence_counts`' row-major treatment was only ported to the
|
||||
bit-matrix (`Presence`) case, since that was the one actually blocking a real
|
||||
benchmark run. `select`/`filter` on a sparse-packed count index still hits
|
||||
this; the fix would follow the same shape (a `PersistentSparseCompactIntMatrix`
|
||||
row-major decode, analogous to `for_each_genome_in_row`), just not done. Since
|
||||
`obisys::numa::runner::PartitionRunner`'s panic-propagation fix (see
|
||||
`architecture/numa_partition_runner.md`), this at least fails fast (process
|
||||
panic, exit 101) instead of hanging.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Chance sharing of a kmer across genomes (discussion)
|
||||
|
||||
Null model for "a kmer is present in $r$ of $S$ genomes": how likely is this by chance alone? Extends the single-pair estimate $P_{\mathrm{chance}}\simeq G/4^k$ (see user doc, *K-mer specificity*).
|
||||
|
||||
## Homogeneous case: binomial tail
|
||||
|
||||
If every genome has the same per-genome chance probability $p$, the number $R$ of genomes containing a given kmer is $\mathrm{Bin}(S,p)$:
|
||||
|
||||
$$P(R\geq r)=\sum_{j=r}^{S}\binom{S}{j}p^j(1-p)^{S-j}=I_p(r,\,S-r+1)$$
|
||||
|
||||
where $I_p$ is the regularized incomplete beta function. No elementary closed form; use $I_p$ directly.
|
||||
|
||||
### Numerical evaluation of the sum
|
||||
|
||||
- Term recurrence, one `lgamma` start: $t_{j+1}=t_j\cdot\frac{S-j}{j+1}\cdot\frac{p}{1-p}$.
|
||||
- Sum the tail **not containing the mode** ($Sp$, not $S/2$). If $r>Sp$, sum $\sum_{j\ge r}$ directly and stop when terms are negligible. The complement $1-\sum_{j=0}^{r-1}$ (note: from $j=0$) suffers catastrophic cancellation when the p-value is tiny.
|
||||
- Large $S$: normal approximation with continuity correction, or Poisson($Sp$) when $p$ is small. Unreliable in extreme tails.
|
||||
|
||||
## Heterogeneous case: $p_i = f(G_i)$
|
||||
|
||||
The $G_i$ (number of kmers of genome $i$) are known and differ. With $p_i\simeq G_i/4^k$, the indicators "kmer present in genome $i$ by chance" are independent Bernoulli($p_i$), so $R$ follows a **Poisson-binomial** law, not a binomial:
|
||||
|
||||
$$P(R=j)=\sum_{|A|=j}\prod_{i\in A}p_i\prod_{i\notin A}(1-p_i).$$
|
||||
|
||||
Everything lives in canonical space: $G_i$ is the number of distinct canonical kmers of genome $i$, and the kmer space has size
|
||||
|
||||
$$N=\frac{4^k}{2}$$
|
||||
|
||||
exactly, since $k$ is odd (no kmer equals its own reverse complement, so canonicalization pairs the $4^k$ kmers perfectly). Exact per-genome probability: $p_i=1-(1-1/N)^{G_i}\simeq 2G_i/4^k$. This is twice the $G/4^k$ of the user doc, where $G$ is a number of positions on one strand.
|
||||
|
||||
### Exact computation
|
||||
|
||||
Dynamic programming over genomes, $d_j$ = probability of $j$ hits so far:
|
||||
|
||||
$$d'_j=d_j(1-p_i)+d_{j-1}p_i.$$
|
||||
|
||||
For the upper tail only $d_0\ldots d_{r-1}$ are needed: $P(R\ge r)=1-\sum_{j<r}d_j$, cost $O(S\,r)$, per distinct $(p_1..p_S)$ vector — independent of the kmer, so computed once per index (table of $P(R\ge r)$ for $r=1..S$), not per kmer. Cancellation caveat as above: for tiny tails compute $\Pr(R\ge r)$ by the DP on the upper side (track $d_j$ for $j\ge r$ only from a top-down recurrence) or in log space. Alternative: DFT of the characteristic function (Hong 2013), $O(S^2)$ exact.
|
||||
|
||||
### Approximations
|
||||
|
||||
- **Poisson**: $R\approx\mathrm{Poisson}(\lambda=\sum_i p_i)$ with total variation error $\le\sum_i p_i^2$ (Le Cam). Excellent when all $p_i\ll1$, i.e. $4^k\gg G_i$ (large $k$).
|
||||
- **Binomial with mean $\bar p=\frac1S\sum_i p_i$**: Hoeffding (1956) shows the Poisson-binomial is more concentrated than $\mathrm{Bin}(S,\bar p)$. Hence for $r\ge S\bar p+1$, $P_{\mathrm{PB}}(R\ge r)\le P_{\mathrm{Bin}}(R\ge r)$: using the mean $\bar G$ overestimates the chance probability, i.e. is **conservative** in the upper tail. Wrong direction for $r\le S\bar p$ (lower tail), irrelevant for significance testing.
|
||||
|
||||
Consequence: assuming a common $G$ is safe (conservative) if $G=\bar G$ (arithmetic mean), but loses power when genome sizes span orders of magnitude; the exact DP is cheap enough (once per index) to make the approximation unnecessary. Small genomes with $k=11$ saturate ($p_i\to1$) and break the Poisson approximation, not the DP.
|
||||
|
||||
## Open points
|
||||
|
||||
- Repeats are not a problem. The model is presence/absence per genome, so copy number within a genome is irrelevant, and $G_i$ is the number of **distinct** kmers of genome $i$. A repeat shared between genomes is paralogy, i.e. homology: a true positive, not a false one. Low-complexity kmers (which would share by composition, not descent) are removed upstream by the [entropy filter](entropy.md).
|
||||
- Residual violation of the equiprobability behind $p_i$: nucleotide composition bias (see Karlin–Ost in the user doc).
|
||||
- Test dependence between overlapping kmers (see FDR section).
|
||||
|
||||
## Goal: an FDR of homology
|
||||
|
||||
$H_0$: the $S$ genomes are unrelated (independent random sequences of sizes $G_i$). Rejecting $H_0$ for a kmer, i.e. sharing that chance cannot explain, is the evidence of homology. The quantity wanted is the false discovery rate of the call "shared kmer $\Rightarrow$ homologous".
|
||||
|
||||
### Empirical FDR from the count histogram
|
||||
|
||||
Let $N$ be the size of the kmer space and $n_{\ge r}$ the observed number of distinct kmers present in at least $r$ genomes (a by-product of the index). Under $H_0$ the expected number of kmers with $R\ge r$ over the whole space is exactly $N\,P(R\ge r)$, with $P$ the Poisson-binomial tail above. Hence
|
||||
|
||||
$$\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P(R\ge r)}{n_{\ge r}}\right).$$
|
||||
|
||||
- No per-kmer multiple-testing correction: the number of tests ($N$) is already in the numerator.
|
||||
- Conservative: the observed $n_{\ge r}$ mixes true homology and chance, and the true-null proportion $\pi_0\le1$ is taken as $1$ (Storey-type upper bound).
|
||||
- Take the smallest $r$ with $\widehat{\mathrm{FDR}}(r)\le\alpha$ as the sharing threshold.
|
||||
- Equivalent per-kmer view: BH on $p=P(R\ge r)$ with $N$ tests. The zero-truncation matters if one tests only kmers present in the index ($R\ge1$ by selection): use $P(R\ge r)/P(R\ge1)$, not $P(R\ge r)$, otherwise the test is too conservative by a factor $\sim1/\lambda$.
|
||||
|
||||
### Dependence
|
||||
|
||||
Kmers of one shared region overlap and are not independent tests. The FDR above holds per kmer; for a per-region FDR, count regions (unitigs / super-kmers, with the appropriate clumping of $N$ and $n_{\ge r}$) rather than kmers. Not settled.
|
||||
@@ -2182,9 +2182,9 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
|
||||
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
|
||||
workspace `cargo test` green.
|
||||
|
||||
## `--distance` unification: SNP corrections as first-class metrics (discussion, 2026-08-28)
|
||||
## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
**Decided, not yet implemented.** `--metric` (renamed `--distance` — several of
|
||||
**Implemented.** `--metric` (renamed `--distance` — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
`--metric` was a misnomer) gains a family of `snp-*` values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
@@ -2249,13 +2249,618 @@ F84/TN93 and is included below. `snp-` prefix on every CLI value.
|
||||
**`+Γ` rate-heterogeneity modifier, applicable to `snp-jc`, `snp-k2p`,
|
||||
`snp-k81`, `snp-t92`, `snp-f84`, `snp-hky85`, `snp-tn93`** (not `snp-raw`,
|
||||
nothing to correct; not `snp-logdet`, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter `α` supplied
|
||||
by the user (`--gamma-shape <alpha>`), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.
|
||||
formula as the base correction, weighted by a shape parameter `α` either
|
||||
supplied by the user (`--gamma-shape <alpha>`) or estimated from the data
|
||||
(`--gamma-shape auto`/`estimate`, method-of-moments — not ML; see
|
||||
"Automatic α estimation" below). A modifier on existing values, not a
|
||||
separate enum arm per distance.
|
||||
|
||||
### Automatic α estimation (`--gamma-shape auto`)
|
||||
|
||||
**Correction (verified against the primary source, 2026-09-11):** Jin &
|
||||
Nei (1990) itself (*"Limitations of the Evolutionary Parsimony Method of
|
||||
Phylogenetic Analysis"*, Mol. Biol. Evol. 7(2):82–102 — the paper this
|
||||
whole `+Γ` correction is cited from, confirmed algebraically to match this
|
||||
codebase's `corrected_log`/`k2p` exactly against their eq. A4, general, and
|
||||
A8, the `a = 1` case) contains **no** data-driven α-estimation procedure.
|
||||
Their own recommendation (p. 98) is a fixed default: *"we suggest that the
|
||||
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
|
||||
different gamma distance, estimating a from data. Wilson et al. (1989)
|
||||
recently used a distance with a = 1/2 for restriction-site data of
|
||||
mitochondrial DNA in hominoids."* — i.e. Jin & Nei explicitly punt
|
||||
data-driven estimation to a *different* paper (Wilson et al. 1989), not
|
||||
read/verified here. The estimator below is therefore **not** "Jin & Nei's
|
||||
method" under any framing — that attribution (present in an earlier
|
||||
revision of this section) was wrong, not just under-cited.
|
||||
|
||||
**Implemented** (`PartitionDispersion`, `obikphylo/src/siblings/algorithms/pairwise.rs`)
|
||||
as an independent method-of-moments estimator, unrelated to any specific
|
||||
published procedure: pools substitution counts by **partition** rather
|
||||
than by genome pair, during the same `reduce_pairwise` pass that builds
|
||||
`PairwiseTally` (no second scan).
|
||||
|
||||
For partition `i`: `n_i` = substitutions pooled over every genome pair,
|
||||
`L_i` = eligible loci pooled over every genome pair, `R_i = n_i / L_i`.
|
||||
Modeling among-site rate heterogeneity the same way as the `+Γ` correction
|
||||
itself (a `Gamma(α, α)`-distributed, mean-1, multiplicative rate shared by
|
||||
every locus in a partition — the classical Poisson-Gamma/negative-binomial
|
||||
mixture, the general identity behind gamma-rate-heterogeneity corrections,
|
||||
also behind Uzzell & Corbin's (1971) original observation that substitution
|
||||
counts across sites/regions are over-dispersed relative to Poisson):
|
||||
|
||||
\[
|
||||
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
|
||||
\]
|
||||
|
||||
Weighting each partition's squared deviation by its own `L_i` cancels the
|
||||
Poisson term before attributing what's left to `α`:
|
||||
|
||||
\[
|
||||
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
|
||||
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
|
||||
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
|
||||
\]
|
||||
|
||||
where `\bar L` is the mean partition size. Returns "no estimate" (falls
|
||||
back to the uncorrected formula, warns) when fewer than 2 partitions have
|
||||
data, `\hat\mu \le 0`, or `V` doesn't exceed the Poisson floor
|
||||
`\hat\mu/\bar L` — no detectable over-dispersion, `α` would be unbounded.
|
||||
|
||||
**Caveat, stated explicitly rather than left implicit**: unlike every
|
||||
closed-form correction in `snp_distance.rs` (each verified line-by-line
|
||||
against `ape`'s `dist_dna.c`, and now also against Jin & Nei 1990 directly
|
||||
for the base `+Γ` formula), this estimator is derived from first
|
||||
principles (the general Poisson-Gamma/negative-binomial identity) with no
|
||||
primary-source procedure behind it at all — not Jin & Nei's (confirmed
|
||||
above), and Wilson et al. (1989), the paper they point to instead, hasn't
|
||||
been read/verified either. Mathematically self-consistent (re-derived and
|
||||
checked, not guessed), but a from-scratch method, not a literature
|
||||
implementation. If `--gamma-shape` needs a value with a literature
|
||||
pedigree rather than an estimated one, Jin & Nei's own stated default,
|
||||
`α = 1` (`--gamma-shape 1`), is the better-supported choice today.
|
||||
|
||||
Deliberately **not** gated by `--sankoff-ratio-ceiling` the way
|
||||
`base_pair_tally` is (same precedent as `cardinality_tally` — see its own
|
||||
doc comment): that filter excludes individual saturated *pairs* from a
|
||||
composition estimate computed once at the very end, from the complete
|
||||
`PairwiseTally`; the partition axis needed here only exists transiently,
|
||||
one partition at a time, while `PairwiseTally` is still being built — long
|
||||
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
|
||||
is known. `--exclude-genome` isn't applied either, matching
|
||||
`reduce_pairwise`'s own raw per-pair fold.
|
||||
|
||||
**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
|
||||
`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
|
||||
"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
|
||||
`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
|
||||
needs the true *directional* per-pair base co-occurrence matrix
|
||||
(`PairwiseTally` only keeps the symmetrised substitution counts
|
||||
`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), `snp-tajima-nei` needs each genome's *own* base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
`snp-f84`/`snp-hky85` had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
|
||||
`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
|
||||
a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
|
||||
`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
|
||||
accumulator — not an architecture change.
|
||||
|
||||
### Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
Sufficient statistic, per genome pair `(i, j)`, from
|
||||
`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
|
||||
`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
|
||||
|
||||
- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
|
||||
(pyrimidine transitions)
|
||||
- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
|
||||
(Kimura's two transversion categories)
|
||||
- \(n_{shared}\): loci where both genomes agree
|
||||
- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
|
||||
loci for the pair)
|
||||
- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
|
||||
\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)
|
||||
|
||||
Derived proportions used below:
|
||||
|
||||
\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]
|
||||
|
||||
Every formula below was checked term-by-term against `ape`'s own
|
||||
`src/dist_dna.c` (not re-derived from memory) before being ported to
|
||||
`obikphylo::siblings::algorithms::snp_distance`.
|
||||
|
||||
**`snp-raw`** — uncorrected p-distance:
|
||||
|
||||
\[
|
||||
d_{raw} = p
|
||||
\]
|
||||
|
||||
**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, *is* the correction):
|
||||
|
||||
\[
|
||||
d_{tv} = Q
|
||||
\]
|
||||
|
||||
**`snp-jc`** (Jukes-Cantor, JC69):
|
||||
|
||||
\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]
|
||||
|
||||
**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
|
||||
\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
|
||||
|
||||
\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]
|
||||
|
||||
**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
|
||||
|
||||
\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]
|
||||
|
||||
**`snp-t92`** (Tamura 3-parameter), with GC content
|
||||
\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
|
||||
\(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
|
||||
|
||||
\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]
|
||||
|
||||
\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]
|
||||
|
||||
\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]
|
||||
|
||||
**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
|
||||
substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
|
||||
recovering the uncorrected formula exactly). E.g. for JC:
|
||||
|
||||
\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]
|
||||
|
||||
Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one `corrected_log` helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
`ape`-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.
|
||||
|
||||
### α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
Follow-up discussion after implementing `PartitionDispersion::estimate_alpha`
|
||||
(previous section), triggered by the user pointing out that the literature
|
||||
conflates two distinct problems: *estimating α from an alignment* versus
|
||||
*using a given α in a Jin-Nei-style corrected distance*. Jin & Nei (1990)
|
||||
itself is squarely in the second camp — α is a user-supplied parameter
|
||||
there, not something their paper estimates (confirmed by reading the paper
|
||||
directly, see previous section). This section catalogs the actual
|
||||
α-estimation literature and records which approaches were considered and
|
||||
why each was or wasn't adopted.
|
||||
|
||||
**Bibliography** (methodological papers, not exhaustive users of `+Γ`):
|
||||
|
||||
| Method | Reference | Needs a tree? | Estimates "true" α or a task-specific one? | Cost |
|
||||
|---|---|---|---|---|
|
||||
| ML, continuous Γ (foundational) | **Gu, Fu & Li (1995)**, MBE 12:546–557 — read in full | yes, topology fixed a priori | true α (+ θ jointly, invariant+Γ) | very high, ≤5–6 taxa in practice |
|
||||
| ML, discrete Γ | Yang (1993/94), MBE 10:1396; JME 39:306 | yes | true α | high |
|
||||
| Parsimony-based ML on inferred changes | Yang & Kumar (1996), MBE 13:650 | yes, equal branch lengths assumed | true α, biased when branches unequal | low |
|
||||
| Corrected substitutions/site → NB fit | **Gu & Zhang (1997)**, MBE 14:1106 — read in full | yes, + ancestral reconstruction | true α, close to ML accuracy | low (given a tree) |
|
||||
| Topology-optimizing grid search | **Guindon & Gascuel (2002)**, MBE 19:534 — read in full | yes, but built *from* the candidate distances themselves (BIONJ) | **not** true α — deliberately biased *upward* for topological accuracy | low–moderate (grid × tree build) |
|
||||
| Free (non-parametric) rate distribution | Susko et al. (2003), Syst. Biol. 52:594 | yes | tests the Γ assumption itself | high |
|
||||
| Parsimony counts/site → NB fit | **Wakeley (1993)**, JME 37:613 — read in full | yes, NJ tree + Fitch parsimony ancestral reconstruction | true α, biased *upward* by parsimony (conservative) | low (given a tree) |
|
||||
| Poisson-Gamma/negative-binomial, foundational | Uzzell & Corbin (1971), Science 172:1089 — **not read**, only seen via citations | unknown — likely also count-per-site/tree-based, unverified | true α (indirect, via NB overdispersion) | unknown |
|
||||
| Bayesian posterior over α | BPP/MrBayes/BEAST2/RevBayes | yes | posterior, not point estimate | very high |
|
||||
|
||||
**Correction (2026-09-11, after reading Wakeley in full):** an earlier
|
||||
revision of this table filed Wakeley (1993) under "no tree needed" —
|
||||
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
|
||||
1987, on the real mtDNA data; coalescent-simulated for the simulation
|
||||
study), reconstruct ancestral states at internal nodes by Fitch (1971)
|
||||
parsimony, count the *minimum* number of changes per site this implies,
|
||||
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
|
||||
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
|
||||
Johnson & Kotz 1969) to the resulting per-site count distribution — the
|
||||
same general "tree + ancestral reconstruction + per-site counts + NB fit"
|
||||
shape as Gu & Zhang (1997), just using raw parsimony counts directly
|
||||
rather than Gu & Zhang's multiple-hit-corrected `k̂`. Confirmed by
|
||||
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
|
||||
mean *and* (more severely) the variance of per-site change counts, which
|
||||
biases `α` **upward** (toward apparent uniformity) — consistent with Gu &
|
||||
Zhang's own citation of Wakeley for exactly this bias, and with the
|
||||
general "parsimony overestimates α" pattern noted throughout this
|
||||
literature. Real hypervariable-region-1 mtDNA data gave `α̂ ≈ 0.44–0.60`
|
||||
per data set, `0.47` combined — despite the conservative bias, still
|
||||
solidly in the "strong heterogeneity" range, which is the paper's own
|
||||
point: the method is biased but usefully conservative, not useless.
|
||||
|
||||
**Where `PartitionDispersion::estimate_alpha` actually sits**: nowhere in
|
||||
this table's tree-dependent rows. It shares only the *abstract
|
||||
mathematical identity* (Poisson-Gamma mixture ⟹ negative binomial
|
||||
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
|
||||
genuinely different, tree-free and ancestral-reconstruction-free route:
|
||||
pairwise genome-to-genome substitution counts pooled by *partition*
|
||||
instead of per-site counts pooled across an alignment's columns after
|
||||
ancestral state reconstruction. As far as this survey has established
|
||||
(Uzzell & Corbin not read, so not fully ruled out), **no tree and no
|
||||
ancestral reconstruction of any kind** appears to be a genuinely
|
||||
distinguishing property of `PartitionDispersion`, not something it
|
||||
inherits from prior art — consistent with it being an independent
|
||||
derivation rather than a literature implementation (see previous
|
||||
section's caveat, now on firmer footing).
|
||||
|
||||
**Why a tree-free estimate is even valid — the identifiability question.**
|
||||
Raised directly by the user after reading Gu, Fu & Li (1995), which states
|
||||
explicitly: *"one cannot estimate the rate heterogeneity when the number
|
||||
of sequences is <3."* This is a real non-identifiability result, not a
|
||||
practical inconvenience: with only 2 sequences, a site's history is
|
||||
summarized by one binary observation (differ / don't differ), collapsed
|
||||
across all sites into two aggregate numbers (transition and transversion
|
||||
proportions) — divergence time and among-site rate variance are
|
||||
confounded in that pair of numbers, with no way to separate them. Every
|
||||
method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
|
||||
sequences *sharing one evolutionary history* (a tree) specifically to get
|
||||
multiple independent looks at the *same* site's rate across different
|
||||
lineages, which is what identifies its variance.
|
||||
|
||||
`PartitionDispersion` doesn't violate this, because it isn't estimating
|
||||
the same quantity. The literature above estimates rate variation **across
|
||||
sites, within one shared tree**. `PartitionDispersion` estimates rate
|
||||
variation **across partitions, pooled over every genome pair in the
|
||||
index** — each genome pair stands in for one independent draw of a
|
||||
partition's relative rate, the same identifying role multiple lineages
|
||||
play in the tree-based methods, just substituting "genome pair" for
|
||||
"lineage" and "partition" for "site." With `P` genome pairs (potentially
|
||||
in the hundreds here) all contributing to every partition's pooled count,
|
||||
the ≥3-samples identifiability requirement is met by the pair count, not
|
||||
by tree depth — closer in spirit to Jin & Nei's own original motivating
|
||||
scenario (comparing several genomic *regions* across one fixed panel of
|
||||
taxa) than to single-tree site-rate estimation.
|
||||
|
||||
This substitution carries its own assumption, which must be named rather
|
||||
than left implicit: that a partition's *relative* rate (fast/slow) is
|
||||
reasonably stable **across genome pairs**, not just across sites within
|
||||
one pair — the direct analogue of the "no lineage-specific rate variation,
|
||||
only site-specific" assumption every tree-based `+Γ` method already makes,
|
||||
just moved from lineage→pair and site→partition. If this breaks — e.g. an
|
||||
index mixing very closely related and very divergent genome pairs, where a
|
||||
partition's saturation behavior differs qualitatively between the two —
|
||||
the estimate could be misled in a way a tree-based method would at least
|
||||
have the topology to detect and a tree-free, pool-everything method
|
||||
cannot. Not yet tested against real data with strongly heterogeneous
|
||||
pairwise divergence; worth keeping in mind as the main open validity
|
||||
question for `--gamma-shape auto`, not the citation question (now settled)
|
||||
this whole discussion started from.
|
||||
|
||||
**Gu, Fu & Li (1995), read in full — the foundational ML method, not a
|
||||
candidate to implement.** Models rate variation as invariant+Γ (`θ` =
|
||||
proportion of invariant sites, `α` = gamma shape among the variable sites),
|
||||
and derives the exact joint likelihood over nucleotide *configurations*
|
||||
across all `n` sequences on a **fixed, a-priori-known tree topology** —
|
||||
for 3 sequences this is closed-form (Jukes-Cantor), for `n` sequences it's
|
||||
a sum over `4^n` configurations weighted by coefficients tied to the
|
||||
specific topology's branching structure (their eq. 32–34), with branch
|
||||
lengths, `α`, and `θ` jointly optimized by Newton-Raphson (Hessian over all
|
||||
free parameters). The site-rate integral has the same closed form
|
||||
`E = θ + (1-θ)(1+D/α)^{-α}` that underlies `+Γ` distance corrections
|
||||
generally (the same mechanism as `corrected_log`, not a coincidence — Jin &
|
||||
Nei's own formula is the `θ=0` special case applied to a pairwise `D`
|
||||
rather than a whole-tree one). Explicitly stated by the authors as
|
||||
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
|
||||
candidate for obikmer's typical genome counts. Notable contribution worth
|
||||
keeping in mind regardless: `ρ = (1+θα)/(1+α)` is shown to be a more robust
|
||||
summary of rate heterogeneity than `α` alone whenever `θ` isn't negligible
|
||||
— `α` and `θ` are confounded (nearly-invariant sites can come from either a
|
||||
small `α`'s left tail or from a nonzero `θ`), so `α` alone can be very
|
||||
unstable while `ρ` stays well-behaved. `PartitionDispersion` has no `θ`
|
||||
term at all (no invariant-site component), so this confound doesn't arise
|
||||
for it the same way, but it's worth remembering if an invariant+Γ variant
|
||||
is ever considered.
|
||||
|
||||
**Gu & Zhang (1997), read in full — rejected for now.** Their procedure:
|
||||
(1) a tree with least-squares branch lengths must already exist; (2)
|
||||
ancestral states are reconstructed at every internal node (they use Zhang &
|
||||
Nei 1997's likelihood method, but note plain parsimony would work, just
|
||||
less accurately); (3) per site, branches are split into "changed"/"unchanged"
|
||||
given the ancestral reconstruction, and the *expected* substitution count
|
||||
`k̂` (corrected for multiple hits) is obtained by solving their eq. (7)
|
||||
(or eq. 12 for the generalized model) numerically — a root-find per site,
|
||||
not a closed form; (4) the site-level `k̂` values (real-valued, not
|
||||
integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
|
||||
get α — again no closed form, numerical optimization. This is a real
|
||||
per-site pipeline requiring a tree *before* distances/α can be computed —
|
||||
the reverse of obikmer's current dependency order (`snp_distance` computes
|
||||
distances, which `--nj`/`--upgma` only turn into a tree *afterward*). Their
|
||||
own answer to this chicken-and-egg problem is an iterative
|
||||
distance→tree→α→distance loop, which they themselves flag as expensive.
|
||||
Adopting this would mean a new ancestral-reconstruction subsystem (even a
|
||||
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
|
||||
ML fit — a much bigger addition than `PartitionDispersion`, not something
|
||||
to build without a concrete need beyond "closer to Gu&Zhang's own
|
||||
simulated accuracy than our moment estimator."
|
||||
|
||||
**Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.**
|
||||
Their **Q criterion**: for a candidate α, build a tree (they use BIONJ) from
|
||||
the α-corrected distance matrix; for every internal branch, group taxa into
|
||||
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
|
||||
distances \(\bar\delta_{AB}\), \(\bar\delta_{CD}\), etc., and let `S ≤ M ≤ L`
|
||||
be the three pairings' sums (`{AB,CD}`, `{AC,BD}`, `{AD,BC}`); the branch's
|
||||
reliability score is `Q_branch = L − M` (zero when the four points are
|
||||
perfectly tree-additive). The whole-tree `Q` is the mean over internal
|
||||
branches (negative/zero branches excluded). `α* = argmin_α Q(α)` over a
|
||||
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
|
||||
sensitivity is highest). Complexity: `O(n²l)` once for the base
|
||||
frequencies, then `O(n²r)` for the `r` candidate distance matrices, then
|
||||
`O(n³r)` worst case for building `r` trees and evaluating `Q` on each
|
||||
(same order as the tree-building step itself, so "free" relative to it).
|
||||
|
||||
Central finding, **the opposite direction from an initial misreading in
|
||||
this discussion**: `α_opt` (Guindon-Gascuel's topology-optimal value) is
|
||||
**always ≥ the true α**, not the reverse — underestimating rate
|
||||
heterogeneity (i.e., picking a larger α than reality) reduces the variance
|
||||
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
|
||||
especially when the molecular clock roughly holds (where `α_opt → ∞`, i.e.
|
||||
no correction at all is topologically best). This is an *empirical
|
||||
regularity* observed across their tested conditions (η ∈ {0.5, 2.0}, true
|
||||
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
|
||||
guarantee would be another unverified-claim mistake of the kind this whole
|
||||
discussion has been correcting.
|
||||
|
||||
**Proposed integration (not yet implemented)**: since `α_true ≤ α_opt`
|
||||
empirically, `PartitionDispersion::estimate_alpha()`'s output could seed
|
||||
the *lower bound* of Guindon-Gascuel's grid (search `[α̂, ∞)` instead of
|
||||
`[0.1, 5000]`), narrowing the grid substantially and reusing the existing
|
||||
estimator rather than needing an independent "true α" source — this also
|
||||
sidesteps Gu & Zhang's tree-first dependency problem entirely, since
|
||||
Guindon-Gascuel's own grid already builds a tree per candidate. Still
|
||||
requires: (a) a **BIONJ** implementation (see below), (b) the Q-criterion
|
||||
subtree-grouping logic, (c) a grid-search driver. Not started.
|
||||
|
||||
**BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.**
|
||||
Searched crates.io directly (API query for "bionj": 0 results) and read the
|
||||
docs.rs pages of the most plausible candidates: `phylotree` (builds/reads/
|
||||
manipulates trees but has no distance-matrix reconstruction — no NJ, no
|
||||
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
|
||||
after checking the actual docs), `speedytree` (NJ only — Canonical/RapidNJ/
|
||||
Hybrid variants, explicitly no BIONJ), `nj` (plain NJ only), `phylo` (no
|
||||
distance-based reconstruction at all). `obikphylo::neighbor_joining`
|
||||
(`tree.rs`) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
|
||||
how branch lengths are weighted during agglomeration (accounts for
|
||||
distance-estimate variance, not just the sum), a modest delta over an
|
||||
existing NJ implementation rather than a rewrite, but would need to be
|
||||
written in-house if the Guindon-Gascuel integration above is ever pursued
|
||||
(their own paper notes other tree-building methods gave similar results in
|
||||
their simulations, so plain NJ may be an acceptable substitute if BIONJ
|
||||
proper is judged not worth the effort).
|
||||
|
||||
### Sharing one sample across algorithms (`--sankoff`/`--tnt`/`--phyg`/`--iqtree` + `snp-*` `--distance`) — implemented 2026-09-11
|
||||
|
||||
**Correctness bug, not just a performance one.** `--sankoff`/`--tnt`/`--phyg`/
|
||||
`--iqtree` (via `SiblingExt::sankoff_bundle`) and a `snp-*` `--distance`
|
||||
(via `SiblingExt::snp_distance`) always consume the *exact same* selection
|
||||
parameters when requested together in one `obikmer phylo` invocation
|
||||
(`n`/`free_loss`/`no_ambiguity`/excluded-set/`entropy_bias` — the CLI has
|
||||
no way to give them different values in one run, see `cmd/phylo/args.rs`).
|
||||
Before this fix, each independently called `sample_index`, and
|
||||
`sample_layer`'s `rand::rng()` (`subsample.rs:235`) is a **thread-local
|
||||
generator that advances across calls, not reseeded each time** — so the
|
||||
second call silently drew a *different* random sample of sites than the
|
||||
first, even with identical parameters. Verified empirically (throwaway
|
||||
8-genome index): running `snp-jc` alone vs. combined with `--sankoff`
|
||||
produced visibly different distance matrices from the same index/params.
|
||||
This defeated the actual point of combining these flags — running several
|
||||
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a `snp-*` distance) on
|
||||
one *identical* site selection for direct comparison.
|
||||
|
||||
**Fix**: `SankoffBundle` (`sankoff.rs`) now retains its `PairwiseTally`/
|
||||
`PartitionDispersion` internally (previously consumed into `raw`/
|
||||
`base_pair_tally`/`cardinality_tally` and dropped) and exposes
|
||||
`SankoffBundle::snp_distance(kind, gamma_shape)`, which computes the
|
||||
matrix from that *same* already-sampled tally — no second `sample_index`
|
||||
call. The shared post-sampling logic (the `--gamma-shape` support check,
|
||||
`alpha` resolution, final matrix build — previously all inline in the
|
||||
`snp_distance()` free function) was factored into
|
||||
`pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)` in
|
||||
`snp_distance.rs`, called by both the standalone `snp_distance()` (after
|
||||
its own fresh sampling) and `SankoffBundle::snp_distance` (reusing the
|
||||
bundle's). `cmd/phylo/mod.rs` keeps the `Option<SankoffBundle>` alive past
|
||||
the Sankoff-family `if` block and, when a `snp-*` `--distance` is also
|
||||
requested, calls `bundle.snp_distance(...)` instead of
|
||||
`cache.snp_distance(...)` whenever a bundle was built — since both branches
|
||||
are driven by the same `args.*` fields, the parameters trivially always
|
||||
match when both fire; no runtime "do the params match" check needed.
|
||||
Verified end-to-end: the reused path logs `(reusing the Sankoff bundle's
|
||||
sample)` and the `snp_distance` stage timer reads `0ms` (`formula(kind)`
|
||||
is `O(n²)` post-processing, no I/O), vs. ~150-230ms for a fresh sample on
|
||||
the same tiny test index.
|
||||
|
||||
#### Explicit `--session` (implemented 2026-09-12)
|
||||
|
||||
The in-process fix above only covered reuse *within one command*.
|
||||
`--session DIR` (`cmd/phylo/args.rs`) names a directory (outside the
|
||||
index) that persists the `snp-*` `--distance` sample — the
|
||||
`PairwiseTally`/`PartitionDispersion` pair `sample_index` would otherwise
|
||||
rebuild from scratch — across *separate* `obikmer phylo` invocations.
|
||||
|
||||
**Crate placement, as planned**: a new **`obiksession`** crate
|
||||
(`src/obiksession/`), domain-agnostic — no `PairwiseTally`/"site" concept
|
||||
anywhere in it. `Session::open(dir, params: &[u8], force: bool)` handles
|
||||
directory lifecycle and the params-conflict check against an opaque byte
|
||||
blob; `Session::store(name, bytes)`/`Session::restore(name) -> Option<Mmap>`
|
||||
handle a checksummed artifact cache. `obikphylo`
|
||||
(`siblings/algorithms/session_cache.rs`, shared by `snp_distance.rs` and
|
||||
`sankoff.rs` — see below) is the consumer: it serializes its own
|
||||
`PairwiseTally`/`PartitionDispersion`/`SnpAlignment` to bytes (via `rkyv`)
|
||||
and hands them to `obiksession`, which never sees their type.
|
||||
|
||||
**Serialization**: `rkyv` 0.8.18, added as planned. One deviation from the
|
||||
original zero-copy pitch, scoped down deliberately: artifacts are restored
|
||||
via `rkyv::from_bytes` (full owned deserialize) rather than `rkyv::access`
|
||||
(zero-copy over the `mmap`). True zero-copy would require every
|
||||
`PairwiseTally` query method (`categories`, `base_freq`, `pair`, ...) to
|
||||
work generically over `Archived<PairwiseTally>` as well as the owned type
|
||||
— a separate, larger change not needed to get the actual win (skipping
|
||||
`sample_index`'s expensive re-scan of the sibling annex; deserializing an
|
||||
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
|
||||
`#[derive(Archive, Serialize, Deserialize)]` was added to `PairStats`,
|
||||
`PairwiseTally`, and `PartitionDispersion` (`pairwise.rs`) for this.
|
||||
|
||||
**Startup behavior, as designed**: `Session::open` creates `DIR` if
|
||||
missing (fresh, params recorded); if `DIR` already holds different saved
|
||||
params, `cmd/phylo` reports a hard error and exits rather than silently
|
||||
preferring one side, unless `--session-force` is given (which discards the
|
||||
directory's cached artifacts and starts over under the new params) —
|
||||
implements exactly the "explicit escape hatch, not silent override"
|
||||
decision from the design discussion. `obiksession` itself doesn't decode
|
||||
either side's bytes to produce a field-by-field diff (true to "opaque
|
||||
blob," left to whoever needs it); `cmd/phylo`'s conflict message is
|
||||
currently a single generic sentence listing which flags could be the
|
||||
cause, not a computed diff — an honest v1 simplification, not a limitation
|
||||
of `obiksession` itself.
|
||||
|
||||
**Per-artifact caching, as designed**: only `PairwiseTally`/
|
||||
`PartitionDispersion` are cached (`snp_distance.rs`'s `restore_tally`/
|
||||
`store_tally`) — `ratio_ceiling`/`gamma_shape` remain pure post-processing
|
||||
over whatever tally is in hand, cached or fresh, exactly as planned.
|
||||
|
||||
**Locking, reconsidered and kept as the whole-session `DirLock`.** First
|
||||
written up as a corner cut ("per-entry locking was sketched, not
|
||||
implemented"). On actually working through the concrete race, whole-session
|
||||
locking turned out to be the *correct* choice, not a simplification:
|
||||
`--session-force` resets a session by deleting its whole `artifacts/`
|
||||
directory. If the directory lock were only held briefly (during
|
||||
`Session::open`, then released so a long-running `Session` doesn't block
|
||||
other processes from touching *different* artifacts), a second process
|
||||
calling `open(..., force: true)` could acquire the lock, see mismatched
|
||||
params, and wipe `artifacts/` **while a first process's already-open
|
||||
`Session` is mid-checkpoint** — genuine corruption risk, not just wasted
|
||||
work. Holding the lock for the `Session`'s entire lifetime makes that
|
||||
sequence impossible: a `force` reset must wait for every currently-open
|
||||
`Session` on that directory to be dropped first. The concurrency this
|
||||
gives up (two *different* processes reading/writing *different* artifacts
|
||||
of the same session at once) isn't exercised by anything today — only one
|
||||
code path (`snp_distance`/`sankoff_bundle`, sequentially per invocation)
|
||||
uses sessions at all.
|
||||
|
||||
**Per-layer chunked/resumable dumps — implemented.** `subsample::sample_index`
|
||||
gained a `skip_layers: usize` parameter (skip the first N (partition,
|
||||
layer) pairs, in its own deterministic iteration order, entirely — no
|
||||
I/O, no `on_layer` call) and its `on_layer` callback gained a `raw_index`
|
||||
argument (that pair's 0-based position in the same order), so a caller can
|
||||
tell exactly how far a run has gotten. `snp_distance`/`sankoff_bundle`
|
||||
checkpoint their tally/dispersion (and, for `sankoff_bundle`, the
|
||||
in-progress `SnpAlignment`) to the session every
|
||||
`session_cache::CHECKPOINT_INTERVAL_LAYERS` (8) layers or
|
||||
`CHECKPOINT_INTERVAL` (30s) of wall time, whichever comes first — bounding
|
||||
how much completed scanning a crash can lose, without paying a full
|
||||
tally-rewrite on every one of what can be thousands of layers. A `progress`
|
||||
artifact records how many layers are reflected in the checkpoint, with a
|
||||
`PROGRESS_COMPLETE` (`u64::MAX`) sentinel once every layer has been
|
||||
processed (distinct from any real layer count, so "finished" is never
|
||||
confused with "stopped after N layers where N happens to equal the total").
|
||||
|
||||
Verified with a real kill: built a 10-genome/64-partition index, ran
|
||||
`--distance snp-jc --session DIR` under `timeout 0.4`, confirmed a
|
||||
checkpoint at `progress=8` (exactly `CHECKPOINT_INTERVAL_LAYERS`) was on
|
||||
disk, then re-ran the same command and confirmed it resumed from layer 8
|
||||
and completed successfully — 0-effort verification that the mechanism
|
||||
itself is real, not just plausible-sounding.
|
||||
|
||||
**Important correction from that same test — resume is *not* bit-reproducible,
|
||||
and shouldn't be.** The output matrix from the resumed run differed from
|
||||
an uninterrupted control run against the same index/params. Root cause:
|
||||
`sample_layer`'s random draws come from `rand::rng()`, a per-process,
|
||||
OS-seeded generator with no continuity across a process boundary — the
|
||||
layers processed before a kill and the layers processed after a resume
|
||||
come from two independent random streams. An earlier revision of this doc
|
||||
(and of the code comments) claimed resume "reproduces the exact same
|
||||
sample an uninterrupted run would have" — that's wrong, and was corrected
|
||||
after the user pointed out the actual reasoning error: true randomness has
|
||||
no notion of continuity in the first place, so there was never a
|
||||
correctness requirement for a resumed run to match an uninterrupted one
|
||||
bit-for-bit. The sample produced after a resume is exactly as legitimate a
|
||||
random draw as one from an uninterrupted run — just a *different* one,
|
||||
same as running the command twice without `--session` already gives two
|
||||
different samples today. That variance is explicitly wanted (see below),
|
||||
not a defect to fix. Comments in `subsample.rs`/`session_cache.rs` were
|
||||
corrected to state this plainly instead of the false reproducibility claim.
|
||||
|
||||
**Explicitly declined: deterministic/seeded sampling.** Raised as a
|
||||
possible fix for the above (seed `sample_layer`'s RNG from a per-session
|
||||
value plus `(partition, layer)`, making a resume reproduce bit-identical
|
||||
results, and, if applied everywhere, making *any* two runs with identical
|
||||
parameters produce identical samples). The user explicitly declined this
|
||||
for now: repeated runs are relied on to measure sampling variance, and a
|
||||
`--seed` flag would need its own explicit design if wanted later — this is
|
||||
not an accidental gap, it's a stated preference to revisit only if asked.
|
||||
|
||||
**`SankoffBundle` integration — implemented.** `sankoff_bundle` accepts
|
||||
the same `session: Option<&Session>` and uses the *same* artifact names
|
||||
(`session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}`)
|
||||
as `snp_distance`, plus its own `ALIGNMENT_ARTIFACT` (`sankoff.rs`) folded
|
||||
in lockstep with the tally at every checkpoint (same `on_layer` call
|
||||
produces both, so they can never drift out of sync). Consequence verified
|
||||
directly: a tally checkpointed by `--sankoff --session DIR` is restored by
|
||||
a *later, separate* `--distance snp-k2p --session DIR` invocation with no
|
||||
`--sankoff` at all, and vice versa — cross-consumer reuse, not just
|
||||
same-command reuse, exactly the synergy this design aimed for.
|
||||
|
||||
**Per-artifact caching, as designed**: only the tally/dispersion (plus,
|
||||
for `sankoff_bundle`, the alignment) are cached — `ratio_ceiling`/
|
||||
`gamma_shape` remain pure post-processing over whatever tally is in hand,
|
||||
cached or fresh, exactly as planned.
|
||||
|
||||
**Still not done — genuine follow-up, not a justified tradeoff:**
|
||||
**exhaustive-sample caching for `--subsample`-varying exploration.** The
|
||||
design discussion's suggestion — cache the exhaustive (`n = None`) sample
|
||||
once and subsample *from* it on each run instead of one session per exact
|
||||
`n` — wasn't implemented; a session today is still scoped to one exact
|
||||
parameter tuple, `--subsample` included.
|
||||
|
||||
### Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
**Decided, not yet implemented.** The primary distance-matrix output
|
||||
**Implemented.** The primary distance-matrix output
|
||||
(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
|
||||
default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
|
||||
@@ -0,0 +1,286 @@
|
||||
# Pair-coverage-biased family sampling (discussion, 2026-10-07)
|
||||
|
||||
Idea: bias the family sample used by `--pseudo-alignment`/`--sankoff`/`snp-*` so that the `family_overlap` matrix of the *sample* is as uniform as possible, compensating the scarcity of shared families between distant species. Complements, does not replace, the existing proportional-by-partition and entropy-biased draws (see user doc, *Sampling design*).
|
||||
|
||||
## Motivation: observed on Primulaceae (180 genomes)
|
||||
|
||||
Scatter of patristic distance (IQ-TREE on the pseudo-alignment) vs. normalized `family_overlap` shows three pair classes:
|
||||
|
||||
- close intra-genus pairs: high overlap, well resolved;
|
||||
- inter-genus plus distant intra-genus pairs: overlap collapses by ~2 orders of magnitude, one large noisy cloud;
|
||||
- a small low-overlap cluster at the largest distances, plausibly a few poorly sequenced genomes (user hypothesis, unverified; the tree shows one tip, `Androsace_saussurei--PHA010690`, with a 2.9 terminal branch).
|
||||
|
||||
Uniform family draw reproduces this imbalance in the pseudo-alignment: distant pairs share few informative columns, close pairs many. The tree quality for deep splits is then limited by sampled-column count on those pairs, not by the total number of families.
|
||||
|
||||
## What defines a target
|
||||
|
||||
A family $f$ is carried by a genome set $S_f$ and contributes to every pair in $S_f \times S_f$. For a sample $\mathcal{F}$ with inclusion probabilities $\pi_f$, the expected sample overlap of pair $(i,j)$ is
|
||||
|
||||
$$\mathbb{E}[O^{\mathcal{F}}_{ij}] = \sum_{f:\, i,j\in S_f} \pi_f$$
|
||||
|
||||
The goal is a $\pi$ making this approximately constant over pairs. The full-index $O_{ij}$ (already computed by `--family-overlap`) is the pool size per pair; $\min_{ij} O_{ij}$ is a hard ceiling on the flat level reachable.
|
||||
|
||||
## Nesting hypothesis (user, unverified)
|
||||
|
||||
Families covering the poorest pairs also cover the richest ones: absent genome-skimming sampling bias (locus dropout), a family conserved across distant species is not expected to be missing between closer ones. If it holds, $S_f$ is approximately a clade closure and the family pool is nested by pair distance, which changes the design:
|
||||
|
||||
- no sequential/stateful draw needed: a family's weight depends only on its own reach (farthest pair covered, or equivalently its coverage of the poorest pairs), computable in one pass;
|
||||
- flattening reduces to a monotone reweighting by reach: keep (nearly) all long-reach families, thin the short-reach ones, until the sample overlap of close pairs drops to the level of distant ones.
|
||||
|
||||
My earlier remark that the weight is stateful (depends on what other families already covered) holds only without nesting.
|
||||
|
||||
## Measuring nesting: `scripts/pseudo_alignment_nesting.py`
|
||||
|
||||
Works on a pseudo-alignment FASTA plus the IQ-TREE treefile (columns = families; a call is any character other than `-`/`?`). Outputs the sample's `family_overlap` matrix and, per clade-size stratum, the *completeness* $|S_f|/|C(S_f)|$ where $C(S_f)$ is the smallest clade containing $S_f$. Tree rooted as written in the Newick unless `--outgroup`. Uses the already-existing families × genomes presence structure only; no `S_f` is stored (clade of a column = LCA of its first and last carrier in DFS leaf order, precomputed 180×180 matrix).
|
||||
|
||||
### Result, Primulaceae pseudo-alignment (180 genomes, 5.0 M columns, uniform draw, root as written)
|
||||
|
||||
| Clade size | Families | Mean completeness | Clade-closed |
|
||||
|---|---|---|---|
|
||||
| 2 | 145 047 | 1.00 | 100 % |
|
||||
| 3-4 | 272 918 | 0.71 | 20 % |
|
||||
| 5-8 | 503 114 | 0.45 | 0.6 % |
|
||||
| 9-16 | 637 896 | 0.35 | 0.1 % |
|
||||
| 17-32 | 1 656 229 | 0.21 | 0 % |
|
||||
| 33-64 | 299 632 | 0.18 | 0 % |
|
||||
| 65-128 | 181 126 | 0.08 | 0 % |
|
||||
| 129-180 | 1 307 452 | 0.12 | 0 % |
|
||||
|
||||
Overall 4.1 % clade-closed, mean completeness 0.275; off-diagonal overlap min 4, median 1 665, max 387 057, CV 1.57. **Strict nesting does not hold on this sample.** Caveats: unrooted tree, so the 129-180 stratum is inflated by families spanning the arbitrary root (rerun with `--outgroup` pending); this is the sample fed to IQ-TREE, not the full index; absence symbols `-`/`?` both count as absent.
|
||||
|
||||
### Lift test (`--lift-test`): margin-preserving null
|
||||
|
||||
Question: do families covering a rare pair $(A,B)$ also cover an abundant pair $(A,C)$ more often than margins imply? For each $A$, $B$ = its 10 % rarest partners, $C$ = its 10 % most abundant (ranked on pair overlap). Null: curveball swap randomisation of the genome × family matrix (exact row and column sums), re-selecting $B$/$C$ in every replicate with the same procedure. 200 754 sampled columns, 20 null replicates, 5 000 trades between replicates.
|
||||
|
||||
| Statistic | Observed | Null |
|
||||
|---|---|---|
|
||||
| $T=P(C\mid A,B)=O_{ABC}/O_{AB}$ | 0.559 | 0.498 ± 0.003 (no replicate ≥ observed, p at floor 1/21) |
|
||||
| $L=\sum n_A O_{ABC}/\sum O_{AB}O_{AC}$ | 1.18 | 1.694 ± 0.004 |
|
||||
|
||||
The two statistics disagree, not contradictorily: $T$ above null supports the "positive association beyond margins" statement; $L$ below null says that, relative to $P(C\mid A)$, knowing $B$ adds less than under the null — the observed abundant partners of $A$ are far more abundant than null ones (phylogenetic structure inflates the $L$ denominator). Unverified: mixing of the chain (is 5 000 trades enough? the tiny null sd may reflect correlation), sensitivity to the 10 % threshold and to the 200 k-column subsample.
|
||||
|
||||
Combined reading: association exists but is patchy; a reach-only weight is not enough, weighting must be per presence pattern $S_f$.
|
||||
|
||||
## Candidate weighting strategy: pair raking (proposal, not implemented)
|
||||
|
||||
Goal: inclusion probabilities $\pi_f$ with $\sum_{f\ni i,j}\pi_f\approx c$ for all pairs.
|
||||
|
||||
1. First-order weight: $w_f=\sum_{i,j\in S_f}1/O_{ij}$ (draw a pair uniformly, then a family uniformly among those covering it). With the presence matrix $P$ and $R=1/O$, $w=\tfrac12\,\mathrm{rowsum}(P\odot RP)$, one pass in the existing chunked loop, no $S_f$ storage.
|
||||
2. Iterate (raking/IPF with one multiplier per pair): update $R_{ij}$ from the overlap the current $\pi$ yields, recompute $w_f$; corrects the fact that large families are drawn for many pairs at once. $\pi_f=\min(1,c\,w_f)$, $c$ set to hit the target sample size; independent Poisson draw per family.
|
||||
3. Control: recompute the sample's `family_overlap` and its CV with the script above.
|
||||
|
||||
Pre-conditions: exclude anecdotal genomes first (e.g. `Androsace_saussurei--PHA010690`, terminal branch 2.9), else their tiny $O_{ij}$ dominate $R$. Overlap uniformity is only a proxy: information for a deep split is informative sites across it, not mere co-presence; to be checked on node support of the resulting tree. Unknown: whether long-reach families are rare because conserved-single-copy or because repeats/misassemblies (the latter would be amplified by up-weighting).
|
||||
|
||||
## Alternative / complement: site classes with their own models
|
||||
|
||||
Analogy with multigene supermatrices: loci differ in rate *and* in taxon occupancy. Families covering rare pairs (long reach, conserved, slow) vs. families covering close pairs only could be two (or a few) site classes, each with its own substitution model / branch-length scale (IQ-TREE partitions; edge-proportional vs edge-linked options to be checked in the manual). Independent of the weighting: sampling fixes scarcity, partitioning fixes rate heterogeneity; with class-specific scales, oversampling a class would not distort that class's branch lengths, removing the need for $1/\pi_f$ reweighting (IQ-TREE support for per-site weights unverified).
|
||||
|
||||
Caveats: reach is a continuum, two classes is a simplification and there is no criterion for the cut; class membership is defined from presence patterns, hence correlated with the data the model sees (assignment bias, worst with `?` missing data; magnitude not assessed); partitioning does not fix repeats/misassembly, it only gives them their own rate.
|
||||
|
||||
The pseudo-alignment already keeps only variable families, so branch lengths are not substitutions/site anyway (ascertainment bias, handled by `+ASC`, see the constant-column fix section in [evolutionary_distances.md](evolutionary_distances.md)). With partitions the ascertainment correction must be per class: each class excludes a different number of constant sites. Whether the existing fix applies per partition is unchecked.
|
||||
|
||||
Literature: see the next section.
|
||||
|
||||
## Literature on informative-site selection (2026-10-07; read from search extracts, not full papers)
|
||||
|
||||
- **Phylogenetic informativeness.** Townsend 2007 (*Syst. Biol.* 56:222-231) links each site's rate of change to its power to resolve splits at a given depth: invariant sites are useless, saturated sites misleading, with an optimum in between that depends on the age of the split. Implemented online as PhyDesign (requires an ultrametric tree). Relevance: supports an entropy centre $\mu$ that depends on depth (pair distance) rather than a single value; cf. the $\mu(n)$ discussion.
|
||||
- **Entropy-based trimming.** BMGE (Criscuolo & Gribaldo 2010, *BMC Evol. Biol.* 10:210) computes a per-character entropy-like score weighted by a similarity matrix and removes regions above a threshold. Uses entropy to *discard* over-variable characters, the opposite direction of the Gaussian band here.
|
||||
- **Saturation / fast sites.** Classical removal methods: Slow-Fast, LQP, TIGER, Observed Variability. Counter-evidence (Tree-of-Life protein datasets, PMC10609509): even the fastest sites carry reliable signal, trimming them can lower accuracy, slow sites recover fewer true bipartitions than the fastest. Argument against penalizing high entropy ($H\ge2$) without checking that those families do not carry the deep signal; domain (conserved proteins) differs from central-position SNPs, transfer unverified.
|
||||
- **Taxon-coverage theory.** Sanderson, McMahon & Steel 2010 (*BMC Evol. Biol.*, "Phylogenomics with incomplete taxon coverage: the limits to inference") characterise phylogenetically decisive collections of taxon sets via the four-way partition property: when the coverage pattern suffices to determine the tree uniquely irrespective of signal. Formal frame for pair-coverage sampling; applicability at 180 genomes not assessed.
|
||||
|
||||
Takeaways to check before weighting: (i) whether high-entropy families are the deep-signal ones (entropy vs carrier count measured, see below; the cross with reach is still open); (ii) depth-dependent entropy centre; (iii) whether a decisiveness criterion can serve as a sufficiency test for the sample's pair coverage.
|
||||
|
||||
## Simpler weight: poorest pair of the family (user preference, 2026-10-07)
|
||||
|
||||
$w_f=1/\min_{i,j\in S_f}O_{ij}$, $\pi_f=\min(1,c\,w_f)$. Equivalent in spirit to a reach weight but measured on actual overlaps (no tree needed), same $O(|S_f|^2)$ cost per family as the sum form. Limit: it ignores that large families also feed already-abundant pairs; flattening from above then relies only on dropping short-reach families, which is exact only under nesting, and nesting holds partially (see measurements). Untested.
|
||||
|
||||
### Estimating $O$ locally (per partition / per layer)
|
||||
|
||||
- **Per layer: not representative.** A layer is an increment of kmer content added to a partition (typically one `merge`), not a batch of genomes; all genome columns exist in every layer (`UserDocMD/formats/index_layout.md`). Layer content is dominated by kmers novel at merge time, so a layer's $O_{ij}$ reflects merge order, not the index-wide distribution; not an unbiased estimator of the global $O_{ij}$.
|
||||
- **Per partition: valid only if partitions are exchangeable samples of families** (hash-based; not verified, minimizer distribution may be skewed). Even then, the pairs that matter are the rare ones, with near-zero count per partition; a `min` over noisy estimates is biased downward (winner's curse), weights would go to pairs that were unlucky in that partition rather than globally poorest.
|
||||
- **Recommended:** compute $O$ globally once (the `--family-overlap` scan already exists), apply the weight locally per partition, keeping partitions independent. Without an extra pass: pilot estimate on a random subset of partitions.
|
||||
- **Sequential variant:** update the $O$ estimate as partitions are processed in random order, boosting still-undersampled pairs. Self-correcting but breaks partition independence and makes the result order-dependent.
|
||||
- **Quota consequence:** the current per-partition quota is proportional to candidate-family count; with weights it must be proportional to the partition's weight mass $\sum_f w_f$, otherwise partitions rich in high-weight families are undersampled.
|
||||
|
||||
## Does the entropy bias distort pair representation? (measured 2026-10-07)
|
||||
|
||||
Sample drawn with `--entropy 1 --entropy-sd 0.5` (μ=1 explicit, σ default 0.5): family weight $\exp(-2(H-1)^2)$, $H$ = `entropy15` in bits; weight 1 at $H=1$, 0.135 at $H=0$ or $2$, $\approx 3\!\times\!10^{-4}$ at $H=3$.
|
||||
|
||||
Comparison, per genome pair, of the full-index `family_overlap` $O^{\text{full}}$ (index scan) to the sample's overlap $O^{\text{samp}}$ (5.0 M columns), sampling fraction $r_{ij}=O^{\text{samp}}_{ij}/O^{\text{full}}_{ij}$ (uniform draw would give constant $r$):
|
||||
|
||||
| Quantity | Value |
|
||||
|---|---|
|
||||
| Spearman $O^{\text{full}}$ vs $O^{\text{samp}}$ | 0.994 (Pearson on logs 0.998): ranking preserved |
|
||||
| global fraction $\sum O^{\text{samp}}/\sum O^{\text{full}}$ | 0.52 % |
|
||||
| $r$ quantiles 5 / 50 / 95 % | 0.0025 / 0.0037 / 0.0055 |
|
||||
| slope $\log r$ vs $\log O^{\text{full}}$ | +0.12 |
|
||||
| Spearman($r$, patristic distance) | −0.71 |
|
||||
|
||||
Per decile of $O^{\text{full}}$, median $r$ rises from 0.0027 (poorest decile) to 0.0053 (three richest, plateau): the poorest pairs are represented about half as much as the richest, so the draw amplifies the imbalance. $r$ climbs slowly over deciles 1-6 (0.0027 to 0.0037), jumps at deciles 7-8 (0.0045-0.0052), where the $O^{\text{full}}$ distribution has a gap (689 k to 11.6 M, presumably close intra-clade pairs, unchecked).
|
||||
|
||||
Not established: (a) causality — a dependence of $r$ on pair richness is compatible with the entropy filter but no uniform control draw of the same size exists; (b) noise — the poorest sample pairs have tiny counts (min 4).
|
||||
|
||||
**Candidate mechanism: first hypothesis refuted, see the next section.** I first suggested that long-reach families, carried by many divergent genomes, spread over several states ($H\approx2\text{-}3$) and are downweighted from above by the Gaussian. The full entropy table (next section) shows the opposite: families with many carriers are mostly *low*-entropy, and are downweighted from below.
|
||||
|
||||
**Consequence for the pair-weight proposal:** entropy and reach are likely correlated, not independent criteria. Since the entropy weight of each family is known, the pair weight can compensate it exactly (divide by it) rather than compose with it; order of the two filters must then be defined.
|
||||
|
||||
## Entropy vs number of carrier genomes, full index (measured 2026-10-09)
|
||||
|
||||
Source: `phylo --shannon`, which is **always a full, unsampled scan** of every non-monomorphic minorant of every layer (`SiblingExt::shannon_entropy_csv(path)` takes no `subsample`/`entropy_bias` argument; `--subsample`/`--entropy` apply only to `--pseudo-alignment`/`--sankoff`/`snp-*`). Here: 19.7 GB CSV, 535 257 857 families, read in 11 s with DuckDB (no in-memory load). `UserDocMD/usage/phylo.md` line 121 ("`--subsample N --shannon` gives a bounded diagnostic sample") is stale and contradicts line 55 and the code; to fix.
|
||||
|
||||
| Carriers $n$ | Families | mean $H$ | median $H$ | $P(H<0.5)$ | $P(H\ge1.5)$ | mean weight ($\mu=1,\sigma=0.5$) |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 2 | 163.3 M | 0.66 | 1.00 | 34.5 % | 0 % | 0.70 |
|
||||
| 3-4 | 104.6 M | 0.94 | 0.92 | 0.5 % | 7.9 % | 0.93 |
|
||||
| 5-8 | 91.8 M | 0.83 | 0.72 | 0.1 % | 4.6 % | 0.80 |
|
||||
| 9-16 | 62.7 M | 0.70 | 0.50 | 43.6 % | 4.9 % | 0.66 |
|
||||
| 17-32 | 45.4 M | 0.59 | 0.35 | 58.9 % | 5.9 % | 0.53 |
|
||||
| 33-64 | 48.1 M | 0.34 | 0.19 | 79.7 % | 1.1 % | 0.40 |
|
||||
| 65-128 | 19.1 M | 0.37 | 0.18 | 77.4 % | 3.8 % | 0.37 |
|
||||
| 129-180 | 0.19 M | 0.83 | 0.47 | 53.0 % | 21.4 % | 0.46 |
|
||||
|
||||
Findings:
|
||||
|
||||
- Families with many carriers are mostly one dominant state plus rare variants ($H<0.5$ for ~77-80 % at $n\ge33$); only 1-4 % have $H\ge1.5$. The Gaussian at $\mu=1$ therefore penalizes them from the **low** side (weight 0.135 at $H=0$), not the high side.
|
||||
- Relative to a uniform draw, expected inclusion is about ×1.32 at 3-4 carriers and ×0.53-0.56 at 33-128 carriers: a factor ~2.4 between small and large $n$. Same order as the ~2× pair-level undersampling of poor pairs measured above; compatible, but a family-level ratio is not a pair-level one, so causality remains unproven (no uniform control draw).
|
||||
- The filter does what it is designed for (drop near-autapomorphic variants), but those low-entropy high-$n$ families are also the ones covering distant pairs, hence the side effect.
|
||||
- Informative high-$n$ families are scarce: at $n>32$ only ~19 % have $H\in[0.5,1.5)$, and the 129-180 class is 0.04 % of all families. Flattening deep-pair coverage works on little informative material whatever the weights.
|
||||
- Consequence for a $\mu(n)$ that grows with $n$: it would not recover these families (80 % sit below 0.5 bit); what matters at large $n$ is the share of families in the band around 1 bit, not the position of the centre. Conditional weight $w_H(f)/E[w_H\mid n_f]$ remains an option to make the entropy filter neutral to $n$; $E[w_H\mid n]$ is now available from this table.
|
||||
|
||||
Limit: the table has `n_genomes_present` but not the reach of the family, so the link to specific pairs is still not measured.
|
||||
|
||||
## Site concordance factors on the Primulaceae tree (measured 2026-10-12)
|
||||
|
||||
Command (IQ-TREE 3.1.3): `iqtree3 -t <treefile> -s <alignment> -m GTR+G4+ASC --scfl 100 --cf-verbose -T 5 --prefix concord`. With `-t` plus `-m`, IQ-TREE did **not** just annotate the reference tree: it ran a full ML search started from it (102 iterations, ~59 h wall-clock). `concord.treefile` differs from the input treefile by one bipartition (Robinson-Foulds 2), so sCF refer to the re-inferred tree. Without `-m`, ModelFinder is launched and required 157 GB (first attempt failed). The log also warns that all 180 sequences contain more than 50 % gaps/ambiguity. `sN` below = informative sites averaged over the 100 sampled quartets of a branch (alignment: 5 003 414 columns).
|
||||
|
||||
**Deep Primula/Androsace interface: too few informative sites (data scarcity).**
|
||||
|
||||
| Node | Tips (Primula / Androsace) | sCF | sDF1 | sDF2 | `sN` |
|
||||
|---|---|---|---|---|---|
|
||||
| 209 | 67 (43 / 24) | 27 | 47 | 26 | 17 |
|
||||
| 210 | 66 (43 / 23) | 37 | 33 | 30 | 43 |
|
||||
| 208 | 74 (50 / 24) | 54 | 16 | 29 | 190 |
|
||||
| 211 | 24 (2 / 22) | 64 | 18 | 18 | 226 |
|
||||
| 212 | 21 (2 / 19) | 30 | 38 | 32 | 184 |
|
||||
| 216 | 10 (2 / 8) | 21 | 47 | 32 | 7 |
|
||||
| 217 | 9 (1 / 8) | 43 | 20 | 36 | 7 |
|
||||
|
||||
Quartets crossing these nodes have tens to a few hundred informative sites out of 5 M columns, the quantitative counterpart of the pair-overlap scarcity. Primula tips are spread over several nodes mixed with Androsace (e.g. 43 + 24 at node 209); no monophyletic Primula in this tree.
|
||||
|
||||
**Within Androsace: conflict, not scarcity.** Nodes 283 (72 Androsace, `sN` 17 127), 292 (49 tips, 10 335), 305 (22, 11 980), 342 (11, 6 566) have thousands of informative sites but sCF 33-37 % with both alternatives nearly as supported: sites split almost evenly among the three quartet topologies. More sites would not fix this; cause not established (a poor fit of a tree model to intra-species divergence is one hypothesis, untested). Node 185 (23 Androsace): sDF1 53.6 % > sCF 30.6 %, a majority of the 893 sites prefer an alternative (real conflict).
|
||||
|
||||
**Well-supported nodes have long branches:** node 296 (45 tips, length 0.23, sCF 90 %), 341 (0.21, 91 %), 330 (0.09, 85.5 %), 181 (0.099, 89.5 %); most branches of length 0.01-0.05 have sCF 33-45 %. Degenerate: node 186 (length 0.001, `sN` 0.37) and 187 (`sN` 2.2) carry no informative site.
|
||||
|
||||
Median sCF shows no gradient with clade size (about 39-48 % in every class), so the clade-size class hides the effect; the scarcity is specific to the interface nodes above.
|
||||
|
||||
**Implication for sampling.** The part a sampling design can address is the interface scarcity: select families carried by genomes from the four groups around each target node (a quartet-targeted selection) rather than flattening the whole pair matrix. Not assessed: how many such families the index holds (the `sN` of nodes 209 and 216 suggests few). For the intra-Androsace conflict no sampling strategy is suggested by these measurements alone.
|
||||
|
||||
## Quartet coverage (measured 2026-10-12, `--quartet-test`)
|
||||
|
||||
`scripts/pseudo_alignment_nesting.py --quartet-test --quartets N` draws random genome quartets and counts exactly the columns where all four carry a call, stratified by the size of the smallest tree clade containing the four. Cost: 20 000 quartets over 5.0 M columns in about 35 s total. Exact per-family quartet weighting is out of reach ($\binom{180}{4}\approx 42.3$ M quartets; one family carried by 100 genomes covers $\binom{100}{4}\approx 3.9$ M), so quartets serve as a *diagnostic* of a sample, not as a weight.
|
||||
|
||||
Result on the current sample: 4-way count quantiles 0 / 1 / 10 / 50 / 90 / 99 / 100 % = 0, 3, 68, 200, 2 329, 66 504, 194 460 (CV 3.5, 0.1 % uncovered). By depth: median 131-209 for quartets inside clades ≥ 65 genomes versus 11 580-57 387 for clades 17-64. The ratio of the 4-way count to the smallest of the six pair overlaps has median 0.20-0.29 for deep quartets (0.53 for clades 33-64; 1 would be perfect nesting): pair overlap is only a loose upper bound of 4-way coverage, so flattening pairs does not by itself flatten quartets. 88 % of random quartets fall in the deepest class (root as written; strata with < 5 quartets are noise). Co-presence is not informativeness (a column must also differ among the four). Next measurement: repeat on a sample drawn with the pair weight and check whether the ratio moves toward 1.
|
||||
|
||||
## Two kinds of family loss: sequencing dropout vs divergence (2026-10-12)
|
||||
|
||||
- **Sequencing dropout (genome skimming, ~1× coverage).** About $e^{-1}\approx 37\%$ of each genome is uncovered, independently per family. Predicted for a family truly present in a clade of $m$ genomes: completeness ≈ 0.63-0.70 and clade-closed fraction $c^m$, independent of clade depth. Consistent with small clades (3-4 tips: completeness 0.71, 20 % closed vs $0.65^3=27\%$, $0.65^4=18\%$). Does not predict the observed decay at larger clades (0.45 at 5-8 tips, 0.21 at 17-32, 0.08-0.12 beyond).
|
||||
- **Loss by divergence (user observation).** A family is a kmer whose 30 flanking positions (15 each side for $k=31$) are identical, with a variable central base; with per-site divergence $d$ the flanks survive with probability ≈ $(1-d)^{30}\approx e^{-30d}$ (0.74, 0.21, 0.042, 0.0012 for $d$ = 0.01, 0.05, 0.10, 0.20). Phylogeny-dependent and tied to local conservation, unlike sequencing dropout. Explains the loss of completeness with clade size, and sets a floor: no weighting can sample families that no longer exist for pairs whose $e^{-30d}$ is negligible. Surviving deep families are selected for conserved flanks, hence plausibly slow-evolving (fewer variable sites for deep splits; hypothesis, untested).
|
||||
- The patristic distances of the IQ-TREE tree (1 to 3.5) come from an alignment with variable sites only, hence strongly inflated; the true $d$ is unknown and these numbers cannot be plugged into $e^{-30d}$.
|
||||
- **Test (user's plan), done 2026-10-12: see the next section.**
|
||||
|
||||
## Overlap vs Mash distance on the full index (measured 2026-10-12)
|
||||
|
||||
Inputs: `phylo --distance mash --csv --shared-kmers --nj` on the 180-genome index (distance matrix, shared-kmer matrix, NJ tree) against the full-index `family_overlap` $O_{ij}$. Mash $D$: range 0.011-0.337, median 0.20.
|
||||
|
||||
| Regression | Slope | $R^2$ |
|
||||
|---|---|---|
|
||||
| $\log O_{ij}$ vs $D$, all pairs | -26.95 | 0.979 |
|
||||
| $\log$(shared kmers) vs $D$ | -31.87 | 0.984 |
|
||||
| $\log O_{ij}$ vs $D$ + additive genome effect | -25.3 | 0.9999 |
|
||||
|
||||
Spearman($D$, $O$) = -0.95; Spearman($D$, shared kmers) = -0.95; Spearman($D$, IQ-TREE patristic) = 0.81.
|
||||
|
||||
Reading:
|
||||
|
||||
- Overlap decays log-linearly with distance, slope about -27 versus -30 predicted by flank survival (previous section). **Not an independent confirmation of the flank model**: Mash $D$ is derived from kmer sharing and $O$ counts families defined by kmers, so both measure the same shared content. The kmer slope of -32 is $-k$ by construction of Mash (implementation check only). The ratio of the two slopes, 27/32 ≈ 0.85, is below the 30/31 ≈ 0.97 expected from flank survival alone; cause unknown, the gap is modest.
|
||||
- Local slopes vary: -21 for $D\in[0.011,0.07]$ ($R^2$ 0.45), -28 for [0.07, 0.20] (0.96), -35 for $D>0.23$ (0.83). Half the pairs lie in $D\in[0.20,0.23]$, where the fit explains almost nothing ($R^2$ 0.26 and 0.07): a plateau where Mash has little resolution.
|
||||
- Genome effects (log scale, sd 0.21): lowest `Androsace_saussurei--PHA010690` (factor ≈ 0.39), `Androsace_triflora--PHA000558` (≈ 0.62), `Androsace_vesulensis--PHA010698` and `Primula_stricta_var_obesior--TROM_V_149128` (≈ 0.65); highest `Androsace_pubescens--PHA010697` (≈ 2.8). Three of the four lowest are also among the genomes with the fewest calls in the pseudo-alignment (saussurei, triflora, stricta), pubescens has the most; consistent with a coverage effect at the extremes, correlation over all genomes not computed.
|
||||
- NJ on Mash vs the IQ-TREE tree: Robinson-Foulds 182 of 354 (0.51 normalized); the Mash plateau near 0.2 probably limits NJ resolution for most pairs.
|
||||
- Consequence for normalization: the residual $\log O_{ij}-(a+bD_{ij})$ measures coverage deviation at a given distance, a normalization of $O$ that needs no per-genome call counts. Untested on a weighted sample.
|
||||
|
||||
## Data-poor genomes (measured 2026-10-12)
|
||||
|
||||
Per-genome share of the 5 003 414 sampled columns with a call: median 5.2 %, max 14.3 %; 8 genomes below 1 % and 23 below 2 %. Fewest calls: `Androsace_triflora--PHA000558` 1 936, `Primula_stricta_var_obesior--TROM_V_149128` 2 403, `Primula_carniolica--PHA007185` 4 937, `Androsace_saussurei--PHA010690` 6 986, `Primula_pumila--TROM_V_135924` 17 393, `Androsace_maxima--PHA000534` 17 939, `Primula_auricula_subsp_serratifolia--PHA007184` 30 621, `Androsace_pyrenaica--PHA000543` 37 725. Three of them (saussurei, serratifolia, pyrenaica PHA000543) are among the tips whose placement looked aberrant, previously attributed to possible mislabelling or contamination. A simpler alternative is too little data to place them reliably; testable by removing them (`--exclude-genome`) and watching deep-node support.
|
||||
|
||||
Consequence for flattening: raw $O_{ij}$ mixes evolutionary distance and per-genome coverage; flattening on it chases pairs involving data-poor genomes, for which no family exists. Normalizing, $O_{ij}/(c_ic_j)$ with $c_i$ the number of families carried by genome $i$ (per-genome counts from `--sibling-stats`), targets distance instead of coverage; the best normalization (smoothing of $c_i$, etc.) is not established.
|
||||
|
||||
## Implemented: entropy annex parametrised by `--no-ambiguity` (2026-10-13)
|
||||
|
||||
Design decision (user): the way genomes are counted depends on `--no-ambiguity`, so that flag, not an "alphabet" option, is the annex parameter. Over single-member genomes the 15-state entropy only has the four single-base states, hence `entropy4` (redefined: 4 bases over single-member genomes) equals `entropy15` under `--no-ambiguity`; a unit test checks it.
|
||||
|
||||
- `entropy_annex.rs`: header `flags` (the former padding), bit 0 = `no_ambiguity`; old files have 0 and stay valid. One file per convention (`entropy.pent`, `entropy.noamb.pent`), so switching the flag builds the missing one without discarding the other. `EntropyAnnex::no_ambiguity()`; `iter_full_entropy` rejects a header that disagrees with the requested convention.
|
||||
- `ensure_layer_entropy_annex` takes the list of conventions to build; one scan feeds every missing annex and the overlap accumulator. `sample_index` builds and reads the convention matching its `no_ambiguity` argument, removing the mismatch between the draw (previously always raw masks) and the alignment (masked).
|
||||
- `--sibling-annex` builds both conventions and invalidates both on rebuild. `--shannon` no longer builds any annex as a prerequisite (it recomputes everything from its own scan).
|
||||
- Checked on the 1x benchmark index: a `--no-ambiguity` run creates `entropy.noamb.pent` beside the existing `entropy.pent` in all 512 layers; workspace tests pass.
|
||||
- Still to do, in this order: `--quartet` (per-family bit annex, same parameter), `--pair-balance` (mean of 1/shared-families over the carrier pairs; second overlap matrix for the single-member convention, annex header carrying a fingerprint of the matrix it came from).
|
||||
|
||||
## Implemented: `--quartet` (2026-10-13)
|
||||
|
||||
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Absence never counts as a state; with `--no-ambiguity`, multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; the `--no-ambiguity` convention is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`) because that header has no room for a flag. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
|
||||
|
||||
Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list.
|
||||
|
||||
First test on the 1x benchmark testbed (3 seeds, exhaustive alignment, `GTR+G4+ASC`, species monophyly out of 3 per seed): without filter 1 / 0 / 0 correct monophylies; carriers >= 4 only: 3 / 1 / 2; `--quartet`: 2 / 2 / 3 (370, 148, 369 columns, all informative by construction). *Klebsiella* stays fragile (wrong in 2 seeds out of 3); two seeds still show a branch at the IQ-TREE ceiling of 10 with so few columns. Small testbed, three seeds: a direction, not a measurement.
|
||||
|
||||
## Implemented: `--subsample` renamed `--n-sites N|all` (2026-10-13)
|
||||
|
||||
`--subsample` read as "draw a subset", while `--pseudo-alignment`/`--sankoff` made it mandatory and accepted no way to say "everything" (the workaround was a number larger than the family count, e.g. `--subsample 100000000`). The option is now `--n-sites <N|all>`: `all` is an explicit value, parsed into its own type (`NSitesArg::{All, N}`, like `GammaShapeArg`) rather than an in-band sentinel (negative number, `0`); `all` gives the exhaustive scan the `snp-*` path already ran when the option was omitted. Still mandatory for `--pseudo-alignment`/`--sankoff`; still optional for `snp-*` `--distance`. No alias is kept for the old name (the software is in active development). The benchmark commands and measurements recorded in the earlier sections of this file were run with the old name.
|
||||
|
||||
## Implemented: `--pair-balance` (2026-10-13)
|
||||
|
||||
Weight of a family = **mean** over the pairs of its carriers of 1/O (O = index-wide shared variable families of the pair, same `--no-ambiguity` convention): the simulation on the *Primulaceae* sample (above) had the mean flatten best (expected-overlap range 23 000× to ~600×, slope 0.37) without chasing high-carrier families, unlike the sum (65 % of the draw at >= 33 carriers).
|
||||
|
||||
- `algorithms/pair_weight.rs`: `inverse_overlap_matrix` (dense n×n of 1/O from the persisted overlap annex), `pair_balance_weight` (mean over carrier pairs; 0 below two carriers). Carriers are defined once, `helpers::is_carrier(mask, no_ambiguity)`, shared with the overlap accumulator and the quartet test.
|
||||
- Storage: no new format. The entropy annex is generalised to `FamilyScalarAnnex` with a `ScalarKind` (`Entropy` magic `PENT`, `PairWeight` magic `PPWT`, one file per kind and convention, header flag `no_ambiguity`); `entropy_annex.rs` became `family_scalar_annex.rs`. Family overlap is now one file per convention too (`family_overlap.povl`, `family_overlap.noamb.povl`, header flag); `--family-overlap` and `--min-shared-family` follow `--no-ambiguity`.
|
||||
- Build order: sweep 1 builds entropy, quartet and the missing overlap annexes together; sweep 2 builds the pair weights, which need the finished matrix (`ensure_layer_pair_weight_annexes`). `AnnexRequest` now lists `entropy`, `quartet`, `overlap`, `pair_weight`; a pair weight implies its overlap. A rebuilt sibling annex removes every annex derived from it (entropy, quartet, pair weight, overlap), so a weight can never outlive the matrix it came from; there is no fingerprint, only that single invalidation point.
|
||||
- Sampler: `LayerPlan` (eligible bits, minorant count, mass) replaces the positional tuple; a layer's quota is `n × mass / Σ mass` (mass = eligible count without the option, Σ weights with it); a family is accepted with probability `quota/mass × weight × entropy kernel`, capped at 1 (the cap, and kernels below 1, undershoot the quota, absorbed by the existing top-up rounds).
|
||||
- Cost note: weights need O(carriers²) per family at build time; not measured on a large index.
|
||||
|
||||
First test, 1x benchmark testbed (20 genomes, 46 706 variable families, `--n-sites 5000 --free-loss --no-ambiguity`, 8 replicates each; sample overlap computed from the alignment, compared with the index `--family-overlap --no-ambiguity`):
|
||||
|
||||
| | slope of log(sample overlap) on log(index overlap) | pairs (index overlap > 0) covered by >= 1 column | columns with >= 10 carriers |
|
||||
|---|---|---|---|
|
||||
| uniform | 1.02 ± 0.14 | 64 % | 0.18 % |
|
||||
| `--pair-balance` | 0.84 ± 0.001 | 97 % | 1.1 % |
|
||||
|
||||
The weight flattens the representation (slope toward 0) and, above all, covers most of the pairs a uniform draw misses. Not measured: effect on the tree (sCF / monophyly), interaction with `--entropy`, behaviour on the *Primulaceae* index. 20 genomes only.
|
||||
|
||||
## Quota rounding fix, and the combined test on the 1x testbed (2026-10-13)
|
||||
|
||||
**Fix.** `sample_index` rounded each layer's quota down on its own, losing up to one site per layer: with 512 layers a budget of 100 sites gave 0 to 3 columns. Quotas now use cumulative rounding (`floor(n × cumulative mass / total)` minus what earlier layers got), so they sum to exactly `n`; a 100-site budget now yields 50-80 usable columns. Not a result of the options below, a defect the small testbed exposed.
|
||||
|
||||
**Test.** 1x testbed, 3 seeds × 10 replicates per variant, 11 genomes (the 9 genomes with fewest calls excluded, so no all-missing sequence), `--free-loss --no-ambiguity`, IQ-TREE `GTR+G4+ASC` on each sampled alignment; score = number of the three species (*E. coli*, *Salmonella*, *Klebsiella*) recovered monophyletic (0-3), mean over the 30 runs; "informative" = columns with two bases each carried by >= 2 genomes.
|
||||
|
||||
| `--n-sites` | variant | trees obtained | columns | informative columns | mean species monophyletic (of 3) | all 3 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 100 | uniform | 77 % | 49 | 0.5 | 0.04 | 0 % |
|
||||
| 100 | `--quartet` | 100 % | 82 | 82 | 2.30 | 50 % |
|
||||
| 100 | `--pair-balance` | 100 % | 52 | 2.1 | 0.20 | 0 % |
|
||||
| 100 | `--quartet --pair-balance` | 100 % | 67 | 67 | 2.53 | 53 % |
|
||||
| 100 | `... --entropy 1` | 100 % | 67 | 67 | 2.17 | 17 % |
|
||||
| 300 | uniform | 93 % | 163 | 2.1 | 0.04 | 0 % |
|
||||
| 300 | `--quartet` | 100 % | 239 | 239 | 2.33 | 33 % |
|
||||
| 300 | `--pair-balance` | 100 % | 179 | 6.2 | 0.57 | 0 % |
|
||||
| 300 | `--quartet --pair-balance` | 100 % | 197 | 197 | 2.67 | 67 % |
|
||||
| 300 | `... --entropy 1` | 100 % | 196 | 196 | 2.67 | 67 % |
|
||||
|
||||
(Mean over all runs counting a failed tree as 0 is identical to the "mean species" column wherever trees were obtained; the uniform rows lose runs because IQ-TREE refuses alignments where a genome has no data.)
|
||||
|
||||
Reading: `--quartet` is the decisive option here — a uniform draw of 100-300 sites contains almost no informative column (0.5-2), the restricted draw contains only informative ones. `--pair-balance` alone barely helps (it favours rare pairs but does not remove the uninformative columns); added to `--quartet` it raises the share of replicates recovering all three species (50→53 % and 33→67 %). The entropy kernel (centre 1 bit, default width) adds nothing at 300 sites and costs at 100. Limits: 11 genomes, three simulated 1x datasets, small budgets, a single criterion (species monophyly, not deep branches), and the differences between the `--quartet` variants (a few runs out of 30) are small relative to the replicate noise — a direction, not a ranking.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
|
||||
- Combination with entropy bias: not independent criteria if entropy correlates with reach (see section above); options: compensate the known entropy weight inside the pair weight, or apply the Gaussian inside reach strata.
|
||||
- Pairs with near-zero $O_{ij}$ (suspected badly sequenced genomes) set the ceiling for everyone; `--min-shared-family`/`--exclude-genome` should be applied first so the floor is not dictated by anecdotal genomes.
|
||||
- Effect on the SNP-distance estimators: a non-uniform draw over pairs biases per-pair eligible-family counts, but each pair's $p$-distance is a ratio over its own eligible families, so unbiasedness per pair is expected to hold. To confirm before relying on it.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Sibling partition unification and masked canonical form (discussion)
|
||||
|
||||
Status: **open, nothing implemented**. Records the state of the reflection on (a) routing the 4 members of a sibling family to one partition and (b) a canonical form that makes siblings directly comparable. Measurements are throwaway (not in the repo); figures are for $k=31$, $m=11$, 256 partitions, 20 bacterial genomes (98.4 M kmer positions), unless stated.
|
||||
|
||||
## Problem
|
||||
|
||||
A sibling family = the 4 kmers differing only at the central base (position $c=(k-1)/2$, $k$ odd). Routing is **not** a hash of all bases: a kmer goes to `minimizer().seq_hash() & mask`, and consecutive kmers of a superkmer share their minimizer. A family splits across partitions whenever the central base falls in (or changes) the selected m-mer.
|
||||
|
||||
Fraction of families whose 4 variants are not all in one partition, standard minimizer: $4m/(k+2m+1)$ (fixed candidates $k-2m+1$ versus $4m$ variable ones). Measured: 81.4 % at $m=11$ (60.2 % at $m=7$, 46.8 % at $m=5$). This is the *theoretical* rate assuming all 4 members present; for two present members it is ≈ 68 %. Real families are overwhelmingly of size 1, so it only becomes cost where something probes variants.
|
||||
|
||||
## Masked canonical form (Can3)
|
||||
|
||||
Notation: `CentreA(K)` = K with its central base set to A; `RC(K)` its reverse complement.
|
||||
|
||||
- Can1 = `min(K, RC(K))` (current). Each sibling canonicalised alone, so some may be flipped relative to the others.
|
||||
- Can2 = orient by `CentreA(K)` vs `RC(CentreA(K))`. **Not a valid canonical form**: when the flanks are reverse complements of each other ($L=\mathrm{rc}(R)$, frequency ≈ $4^{-c}$, 1/1024 at $k=11$), $K$ and $\mathrm{RC}(K)$ get different canonicals (k=5: `ACAGT` and `ACTGT` both kept). Cause: `CentreA(RC(K)) ≠ RC(CentreA(K))` (the latter has centre T).
|
||||
- **Can3** (retained):
|
||||
|
||||
```
|
||||
Can3(K) = K if CentreA(K) < CentreA(RC(K))
|
||||
= RC(K) if CentreA(K) > CentreA(RC(K))
|
||||
= Can1(K) otherwise // self-revcomp family
|
||||
```
|
||||
|
||||
Orientation is decided on the flanks only (centre ignored), tie broken on the full kmer. Gain: the 4 canonical siblings have identical flanks in the same orientation, so they compare directly with no reverse complement. Self-revcomp families have only 2 distinct canonical members (centres A/C; T/G fold onto them). The A↔T (C↔G) substitution in that context is undetectable by any double-strand canonical set, same as under Can1 — not a loss.
|
||||
|
||||
Gain is simplification only: `central_canonical_neighbors` already canonicalises each variant independently and is correct (see `evolutionary_distances.md`, per-partition sweep). The earlier "masked-flank grouping" there is the Can2 idea; Can3 is its repaired form. **No performance gain by itself**, and it breaks existing indexes (canonical orientation changes for some kmers → MPHF hashes change; rebuild).
|
||||
|
||||
Prerequisite if adopted: `Minimizer = CanonicalKmerOf<MLen>` shares the canonical type with index kmers; an m-mer has no sibling family and must keep Can1. Decouple with a distinct `Minimizer` type (6 files outside `kmer.rs`: `routable.rs`, `packed_seq.rs`, `obiskbuilder/{rolling_stat,stream_iter,iter}.rs`, `obifastwrite`).
|
||||
|
||||
## Routing that excludes the centre (scheme B) — rejected
|
||||
|
||||
Candidates restricted to m-mers not overlapping the centre ($m\le c$): $S(j)=\min$ hash over m-mers $p\in[j,j+c-m]$, minimizer$(i)=\min(S(i),S(i+c+1))$. Family- and strand-invariant by construction (checked). But unitigs are partition-local (`pipeline.md`), and the hole moves with each kmer, so runs shrink:
|
||||
|
||||
| $m$ | mean run, current | mean run, B | bases/kmer, current → B |
|
||||
|---|---|---|---|
|
||||
| 5 | 14.9 | 6.3 | 3.02 → 5.76 |
|
||||
| 11 | 11.0 | 3.0 | 3.72 → 10.96 |
|
||||
| 15 | 9.0 | 1.0 | 4.32 → 30.9 |
|
||||
|
||||
Proxy: run = unitig (upper bound; real unitigs may merge). Independent of partition count. In `benchmark/run/global_index_count` (786 MB) `unitigs.bin` + idx ≈ 241 MB (31 %), so tripling them ≈ +60 % total index. Hashing the masked kmer per kmer (no minimizer) would cut every path at every kmer: worse.
|
||||
|
||||
Structural tension: path coherence (long unitigs) needs a key shared by consecutive kmers, family coherence needs a key independent of each kmer's own centre, which is an ordinary base for its neighbour. Decoupling sequence storage from MPHF partitioning fails because a partition can only enumerate its kmers from its own unitigs (MPHF not invertible). Redefining the family on a terminal base would cut the split rate to ≈ 17 % but is not closed under revcomp and changes the SNP semantics. Lower $m$ reduces the split rate for free but deviates from the literature $k/m\approx 2.5$ optimum (balance, bucket size), unmeasured here — $m=11$ kept.
|
||||
|
||||
## What the cross-partition cost actually is
|
||||
|
||||
- Phylo-time: only families of size > 1 trigger remote lookups (a small share).
|
||||
- Annex construction probes the 3 variants of **every** kmer (size-1 families included), ≈ 2/3 of probes crossing partitions; batch bounded to `#kmers(p)*3` per partition (not the whole index).
|
||||
- Real remaining benefit of unification: the outer sweep over source partitions could run in parallel (it is sequential today for memory and for the `q >= p` dedup rule); plus code simplification in `family_scan.rs` / `annex.rs`. Not quantified.
|
||||
- Unification would also allow an MPHF keyed on the family (one lookup, 4-bit member mask, union/intersection as OR/AND); needs the family-size histogram (1–4 members) to judge storage.
|
||||
|
||||
## Open measurements (index not on this machine)
|
||||
|
||||
1. Family-size histogram of the real index: `obikmer phylo --sibling-hist` (needs a built annex).
|
||||
2. Time and peak RSS of `--sibling-annex` on the Vaccinium/Salix index (`Stage` "sibling_annex" traces time; peak RSS from outside).
|
||||
3. Only if those show the sequential sweep or the probe volume is a real bottleneck: real-index test of scheme B or of a smaller $m$.
|
||||
|
||||
## Decision state
|
||||
|
||||
No code change decided. Can3 + `Minimizer` decoupling is the only self-contained step and is independent of the measurements above; it forces an index rebuild, so it is worth doing only if its simplification is wanted.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Architecture notes for advanced use
|
||||
|
||||
This page describes execution-level behavior relevant to sizing and running `obikmer` on large datasets or multi-socket machines. It complements the [index format](formats/index_layout.md) and [theory](theory/indexing_architecture.md) pages.
|
||||
This page describes execution-level behavior relevant to sizing and running `obikmer` on large datasets or multi-socket machines. It complements the [index format](formats/index_layout.md) and [theory](theory/kmer_indexing/indexing_architecture.md) pages.
|
||||
|
||||
## Sequence invariant
|
||||
|
||||
@@ -8,7 +8,7 @@ Every input sequence is treated purely as a compact representation of a set of o
|
||||
|
||||
- Only the `A`/`C`/`G`/`T` alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.
|
||||
- Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.
|
||||
- Kmers are always handled in canonical form (see [DNA encoding](theory/encoding.md)), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.
|
||||
- Kmers are always handled in canonical form (see [DNA encoding](theory/kmer_indexing/encoding.md)), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.
|
||||
|
||||
## Index dimensioning
|
||||
|
||||
@@ -20,7 +20,7 @@ An index directory is organized as `KmerIndex → partitions → layers`, with a
|
||||
|
||||
## Parallel execution and NUMA awareness
|
||||
|
||||
Partition-level work (index construction, `merge`, `filter`, `reindex`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
|
||||
Partition-level work (index construction, `merge`, `filter`, `convert`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
|
||||
|
||||
- On a multi-socket / multi-NUMA-node machine, one thread pool is pinned per NUMA node, and each partition is processed entirely by threads pinned to one node — keeping the memory a partition touches local to that node's DRAM. This matters because touching kmer data across NUMA nodes without pinning can degrade throughput by an order of magnitude or more on large multi-socket machines.
|
||||
- On a single-socket machine, Apple Silicon, or if hwloc cannot report NUMA topology, all cores are treated as one node with no pinning and negligible overhead — this is the default behavior on macOS.
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Construction pipeline
|
||||
|
||||
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/indexing_architecture.md)):
|
||||
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)):
|
||||
|
||||
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition.
|
||||
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition.
|
||||
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
|
||||
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.
|
||||
4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set.
|
||||
@@ -19,7 +19,7 @@ Each partition's surviving kmers are mapped to a dense range of integer slots by
|
||||
|
||||
## Evidence: exact vs. approximate
|
||||
|
||||
Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`reindex`](../usage/reindex.md)):
|
||||
Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)):
|
||||
|
||||
- **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
|
||||
- **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
|
||||
@@ -51,6 +51,6 @@ Two verification modes are available, selected at build time (`index --approx`)
|
||||
|
||||
A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
|
||||
|
||||
Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`reindex`](../usage/reindex.md) one of the sources first if needed.
|
||||
Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`convert`](../usage/convert.md) one of the sources first if needed.
|
||||
|
||||
`obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time.
|
||||
+6
-7
@@ -6,11 +6,11 @@ All functionality is exposed through a single binary, `obikmer`, organized as su
|
||||
|
||||
## Core principles
|
||||
|
||||
- Kmers are of fixed, odd length $k$, chosen at index-construction time in the range $[11, 31]$ (see [Kmers and super-kmers](theory/kmers_and_superkmers.md)).
|
||||
- Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see [DNA encoding](theory/encoding.md)).
|
||||
- Kmers are of fixed, odd length $k$, chosen at index-construction time in the range $[11, 31]$ (see [Kmers](theory/kmer_indexing/kmers.md)).
|
||||
- Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see [DNA encoding](theory/kmer_indexing/encoding.md)).
|
||||
- Kmers are handled in **canonical form** ($\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))$), making counting strand-independent.
|
||||
- Sequences are decomposed into **super-kmers** before storage, anchored on a hash-selected **minimizer** (see [Minimizer selection](theory/minimizer_selection.md)), then routed to one of several **partitions** for parallel, memory-bounded processing (see [Partitioning and indexing architecture](theory/indexing_architecture.md)).
|
||||
- Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see [Low-complexity kmer filter](theory/entropy_filter.md)).
|
||||
- Sequences are decomposed into **super-kmers** before storage, anchored on a hash-selected **minimizer** (see [Minimizer selection](theory/kmer_indexing/minimizer_selection.md)), then routed to one of several **partitions** for parallel, memory-bounded processing (see [Partitioning and indexing architecture](theory/kmer_indexing/indexing_architecture.md)).
|
||||
- Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see [Low-complexity kmer filter](theory/kmer_indexing/entropy_filter.md)).
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -24,11 +24,10 @@ All functionality is exposed through a single binary, `obikmer`, organized as su
|
||||
| [`query`](usage/query.md) | Query an index with sequences and annotate matches |
|
||||
| [`dump`](usage/dump.md) | Dump indexed kmers as CSV |
|
||||
| [`annotate`](usage/annotate.md) | Add, update, or dump genome metadata |
|
||||
| [`phylo`](usage/phylo.md) | Compute pairwise evolutionary-distance proxies, trees, and phylogenetic exports |
|
||||
| [`name-tree`](usage/name-tree.md) | Translate a TNT/PhyG numeric-label tree export back to real taxon names |
|
||||
| [`phylo`](usage/phylo.md) | Compute pairwise genome distances, trees, and phylogenetic exports |
|
||||
| [`unitig`](usage/unitig.md) | Dump the unitigs of an index as FASTA |
|
||||
| [`estimate`](usage/estimate.md) | Estimate approximate-index parameters before indexing |
|
||||
| [`reindex`](usage/reindex.md) | Convert an index's evidence representation (exact ↔ approximate) |
|
||||
| [`convert`](usage/convert.md) | Convert an index's evidence representation (exact/approximate/hybrid), in place |
|
||||
| [`utils`](usage/utils.md) | Miscellaneous index maintenance and inspection utilities |
|
||||
| [`pack`](usage/pack.md) | Pack per-column matrix files into a single-file format |
|
||||
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" default-locale="en-US" demote-non-dropping-particle="sort-only">
|
||||
<info>
|
||||
<title>Molecular Ecology Modified</title>
|
||||
<id>http://www.zotero.org/styles/molecular-ecology</id>
|
||||
<link href="http://www.zotero.org/styles/molecular-ecology" rel="self"/>
|
||||
<link href="http://www.zotero.org/styles/ecology-letters" rel="template"/>
|
||||
<link href="http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291365-294X/homepage/ForAuthors.html" rel="documentation"/>
|
||||
<author>
|
||||
<name>Xiaodong Dang</name>
|
||||
<email>dangxdong@gmail.com</email>
|
||||
</author>
|
||||
<contributor>
|
||||
<name>Johan Asplund</name>
|
||||
<email>asplundj@gmail.com</email>
|
||||
</contributor>
|
||||
<contributor>
|
||||
<name>David Kaplan</name>
|
||||
<email>david.kaplan@ird.fr</email>
|
||||
</contributor>
|
||||
<category citation-format="author-date"/>
|
||||
<category field="biology"/>
|
||||
<issn>0962-1083</issn>
|
||||
<eissn>1365-294X</eissn>
|
||||
<updated>2012-09-09T22:24:43+00:00</updated>
|
||||
<rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
|
||||
</info>
|
||||
<macro name="editor-translator">
|
||||
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
|
||||
<name initialize-with="" delimiter=", "/>
|
||||
<label form="short" prefix=", " text-case="capitalize-first" suffix="," strip-periods="true"/>
|
||||
</names>
|
||||
</macro>
|
||||
<macro name="author">
|
||||
<names variable="author">
|
||||
<name name-as-sort-order="all" sort-separator=" " initialize-with="" delimiter=", " delimiter-precedes-et-al="never"/>
|
||||
<label form="short" prefix=" (" suffix=")" text-case="capitalize-first"/>
|
||||
<et-al font-style="italic"/>
|
||||
<substitute>
|
||||
<names variable="editor"/>
|
||||
<names variable="translator"/>
|
||||
<text macro="title"/>
|
||||
</substitute>
|
||||
</names>
|
||||
</macro>
|
||||
<macro name="author-short">
|
||||
<names variable="author">
|
||||
<name form="short" and="symbol" delimiter=", " initialize-with=". "/>
|
||||
<et-al font-style="italic"/>
|
||||
<substitute>
|
||||
<names variable="editor"/>
|
||||
<names variable="translator"/>
|
||||
<choose>
|
||||
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||
<text variable="title" form="short" font-style="italic"/>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="title" form="short" quotes="true"/>
|
||||
</else>
|
||||
</choose>
|
||||
</substitute>
|
||||
</names>
|
||||
</macro>
|
||||
<macro name="title">
|
||||
<choose>
|
||||
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||
<text variable="title" font-style="italic"/>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="title"/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="access-detail">
|
||||
<group delimiter=" ">
|
||||
<text variable="URL" prefix="Available at: "/>
|
||||
<group prefix="(" suffix=")" delimiter=" ">
|
||||
<text term="accessed"/>
|
||||
<date variable="accessed">
|
||||
<date-part name="day" suffix=" "/>
|
||||
<date-part name="month" form="long" suffix=" "/>
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</group>
|
||||
</group>
|
||||
</macro>
|
||||
<macro name="access">
|
||||
<choose>
|
||||
<if type="webpage post post-weblog entry entry-encyclopedia dataset software report manuscript document" match="any">
|
||||
<text macro="access-detail"/>
|
||||
</if>
|
||||
<else-if variable="accessed">
|
||||
<text macro="access-detail"/>
|
||||
</else-if>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="access-fallback">
|
||||
<group delimiter=" ">
|
||||
<choose>
|
||||
<if variable="URL">
|
||||
<text macro="access-detail" suffix="."/>
|
||||
</if>
|
||||
<else-if variable="publisher">
|
||||
<text variable="publisher" prefix="Available at: " suffix="."/>
|
||||
</else-if>
|
||||
</choose>
|
||||
<text variable="note" suffix="."/>
|
||||
</group>
|
||||
</macro>
|
||||
<macro name="issued-year">
|
||||
<choose>
|
||||
<if variable="issued">
|
||||
<date variable="issued">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
<else>
|
||||
<text term="no date" form="short"/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="publisher">
|
||||
<group delimiter=", ">
|
||||
<text variable="publisher"/>
|
||||
<text variable="publisher-place"/>
|
||||
</group>
|
||||
</macro>
|
||||
<citation collapse="year-suffix" et-al-min="3" et-al-use-first="1" disambiguate-add-year-suffix="true">
|
||||
<sort>
|
||||
<key variable="issued"/>
|
||||
<key variable="author"/>
|
||||
</sort>
|
||||
<layout prefix="(" suffix=")" delimiter="; ">
|
||||
<group delimiter=" ">
|
||||
<text macro="author-short"/>
|
||||
<text macro="issued-year"/>
|
||||
<group>
|
||||
<label variable="locator" form="short"/>
|
||||
<text variable="locator" prefix=" "/>
|
||||
</group>
|
||||
</group>
|
||||
</layout>
|
||||
</citation>
|
||||
<bibliography hanging-indent="true" et-al-min="6" et-al-use-first="1000" entry-spacing="0.5" line-spacing="1">
|
||||
<sort>
|
||||
<key macro="author-short"/>
|
||||
<key variable="issued"/>
|
||||
</sort>
|
||||
<layout>
|
||||
<text macro="author" suffix=" "/>
|
||||
<text macro="issued-year" prefix="(" suffix=")"/>
|
||||
<choose>
|
||||
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||
<group suffix=".">
|
||||
<text macro="title" prefix=" "/>
|
||||
<text macro="editor-translator" prefix=" "/>
|
||||
</group>
|
||||
<text prefix=" " suffix="." macro="publisher"/>
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</if>
|
||||
<else-if type="thesis" match="any">
|
||||
<group suffix=".">
|
||||
<text macro="title" prefix=" " suffix="."/>
|
||||
<text variable="genre" prefix=" " suffix=" Thesis."/>
|
||||
<text prefix=" " macro="publisher"/>
|
||||
</group>
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</else-if>
|
||||
<else-if type="chapter paper-conference" match="any">
|
||||
<text macro="title" prefix=" "/>
|
||||
<group prefix=".">
|
||||
<group prefix=" In:">
|
||||
<text variable="container-title" font-style="italic" prefix=" "/>
|
||||
<text variable="collection-title" prefix=" " suffix="."/>
|
||||
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
|
||||
<label form="short" suffix=" " strip-periods="true"/>
|
||||
<name name-as-sort-order="all" sort-separator=" " initialize-with="" delimiter=", " delimiter-precedes-last="never"/>
|
||||
</names>
|
||||
<group suffix=".">
|
||||
<label variable="page" form="short" prefix=", " suffix=" "/>
|
||||
<text variable="page"/>
|
||||
<text macro="publisher" prefix=". " suffix="."/>
|
||||
</group>
|
||||
</group>
|
||||
</group>
|
||||
</else-if>
|
||||
<else-if type="webpage post post-weblog entry entry-encyclopedia dataset software" match="any">
|
||||
<group suffix=".">
|
||||
<text macro="title" prefix=" "/>
|
||||
<text macro="editor-translator" prefix=" "/>
|
||||
</group>
|
||||
<group prefix=" " suffix=".">
|
||||
<text variable="container-title" text-case="title" font-style="italic"/>
|
||||
<text variable="genre" prefix=", "/>
|
||||
<text macro="publisher" prefix=". "/>
|
||||
</group>
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</else-if>
|
||||
<else>
|
||||
<group suffix=".">
|
||||
<text macro="title" prefix=" "/>
|
||||
<text macro="editor-translator" prefix=" "/>
|
||||
</group>
|
||||
<group prefix=" " suffix=".">
|
||||
<text variable="container-title" text-case="title" font-style="italic"/>
|
||||
<group prefix=", ">
|
||||
<text variable="volume" font-weight="bold"/>
|
||||
</group>
|
||||
<text variable="page" prefix=", "/>
|
||||
</group>
|
||||
<choose>
|
||||
<if variable="container-title">
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</if>
|
||||
<else>
|
||||
<text macro="access-fallback" prefix=" "/>
|
||||
</else>
|
||||
</choose>
|
||||
</else>
|
||||
</choose>
|
||||
</layout>
|
||||
</bibliography>
|
||||
</style>
|
||||
@@ -259,3 +259,601 @@
|
||||
url = {https://doi.org/10.1186/s12864-015-1647-5},
|
||||
volume = 16,
|
||||
year = 2015}
|
||||
|
||||
@article{Roberts2004-rz,
|
||||
title = "{Reducing storage requirements for biological sequence
|
||||
comparison}",
|
||||
author = "Roberts, Michael and Hayes, Wayne and Hunt, Brian R and Mount,
|
||||
Stephen M and Yorke, James A",
|
||||
journal = "Bioinformatics (Oxford, England)",
|
||||
publisher = "Oxford University Press (OUP)",
|
||||
volume = 20,
|
||||
number = 18,
|
||||
pages = "3363--3369",
|
||||
abstract = "MOTIVATION: Comparison of nucleic acid and protein sequences is a
|
||||
fundamental tool of modern bioinformatics. A dominant method of
|
||||
such string matching is the 'seed-and-extend' approach, in which
|
||||
occurrences of short subsequences called 'seeds' are used to
|
||||
search for potentially longer matches in a large database of
|
||||
sequences. Each such potential match is then checked to see if it
|
||||
extends beyond the seed. To be effective, the seed-and-extend
|
||||
approach needs to catalogue seeds from virtually every substring
|
||||
in the database of search strings. Projects such as mammalian
|
||||
genome assemblies and large-scale protein matching, however, have
|
||||
such large sequence databases that the resulting list of seeds
|
||||
cannot be stored in RAM on a single computer. This significantly
|
||||
slows the matching process. RESULTS: We present a simple and
|
||||
elegant method in which only a small fraction of seeds, called
|
||||
'minimizers', needs to be stored. Using minimizers can speed up
|
||||
string-matching computations by a large factor while missing only
|
||||
a small fraction of the matches found using all seeds.",
|
||||
month = dec,
|
||||
year = 2004,
|
||||
url = "https://dx.doi.org/10.1093/bioinformatics/bth408",
|
||||
doi = "10.1093/bioinformatics/bth408",
|
||||
pmid = 15256412,
|
||||
issn = "1367-4803,1367-4811",
|
||||
language = "en"
|
||||
}
|
||||
|
||||
@INPROCEEDINGS{Steele2014-cz,
|
||||
title = "{Fast splittable pseudorandom number generators}",
|
||||
author = "Steele, Jr, Guy L and Lea, Doug and Flood, Christine H",
|
||||
booktitle = "{Proceedings of the 2014 ACM International Conference on Object
|
||||
Oriented Programming Systems Languages \& Applications}",
|
||||
publisher = "ACM",
|
||||
address = "New York, NY, USA",
|
||||
abstract = "We describe a new algorithm SplitMix for an object-oriented and
|
||||
splittable pseudorandom number generator (PRNG) that is quite
|
||||
fast: 9 64-bit arithmetic/logical operations per 64 bits
|
||||
generated. A conventional linear PRNG object provides a generate
|
||||
method that returns one pseudorandom value and updates the state
|
||||
of the PRNG, but a splittable PRNG object also has a second
|
||||
operation, split, that replaces the original PRNG object with two
|
||||
(seemingly) independent PRNG objects, by creating and returning a
|
||||
new such object and updating the state of the original object.
|
||||
Splittable PRNG objects make it easy to organize the use of
|
||||
pseudorandom numbers in multithreaded programs structured using
|
||||
fork-join parallelism. No locking or synchronization is required
|
||||
(other than the usual memory fence immediately after object
|
||||
creation). Because the generate method has no loops or
|
||||
conditionals, it is suitable for SIMD or GPU implementation. We
|
||||
derive SplitMix from the DotMix algorithm of Leiserson, Schardl,
|
||||
and Sukha by making a series of program transformations and
|
||||
engineering improvements. The end result is an object-oriented
|
||||
version of the purely functional API used in the Haskell library
|
||||
for over a decade, but SplitMix is faster and produces
|
||||
pseudorandom sequences of higher quality; it is also far superior
|
||||
in quality and speed to java.util.Random, and has been included
|
||||
in Java JDK8 as the class java.util.SplittableRandom. We have
|
||||
tested the pseudorandom sequences produced by SplitMix using two
|
||||
standard statistical test suites (DieHarder and TestU01) and they
|
||||
appear to be adequate for ``everyday'' use, such as in Monte
|
||||
Carlo algorithms and randomized data structures where speed is
|
||||
important.",
|
||||
month = oct,
|
||||
year = 2014,
|
||||
url = "http://dx.doi.org/10.1145/2660193.2660195",
|
||||
doi = "10.1145/2660193.2660195",
|
||||
isbn = 9781450325851
|
||||
}
|
||||
|
||||
@online{Stafford2011,
|
||||
author = {Stafford, David},
|
||||
title = {Better Bit Mixing - Improving on MurmurHash3's 64-bit Finalizer},
|
||||
date = {2011-09-28},
|
||||
url = {https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html},
|
||||
urldate = {2026-09-12}
|
||||
}
|
||||
|
||||
@ARTICLE{Hamming1950-cw,
|
||||
title = "{Error detecting and error correcting codes}",
|
||||
author = "Hamming, R W",
|
||||
journal = "The Bell System Technical Journal",
|
||||
publisher = "Institute of Electrical and Electronics Engineers (IEEE)",
|
||||
volume = 29,
|
||||
number = 2,
|
||||
pages = "147--160",
|
||||
abstract = "The author was led to the study given in this paper from a
|
||||
consideration of large scale computing machines in which a large
|
||||
number of operations must be performed without a single error in
|
||||
the end result. This problem of “doing things right” on a large
|
||||
scale is not essentially new; in a telephone central office, for
|
||||
example, a very large number of operations are performed while
|
||||
the errors leading to wrong numbers are kept well under control,
|
||||
though they have not been completely eliminated. This has been
|
||||
achieved, in part, through the use of self-checking circuits. The
|
||||
occasional failure that escapes routine checking is still
|
||||
detected by the customer and will, if it persists, result in
|
||||
customer complaint, while if it is transient it will produce only
|
||||
occasional wrong numbers. At the same time the rest of the
|
||||
central office functions satisfactorily. In a digital computer,
|
||||
on the other hand, a single failure usually means the complete
|
||||
failure, in the sense that if it is detected no more computing
|
||||
can be done until the failure is located and corrected, while if
|
||||
it escapes detection then it invalidates all subsequent
|
||||
operations of the machine. Put in other words, in a telephone
|
||||
central office there are a number of parallel paths which are
|
||||
more or less independent of each other; in a digital machine
|
||||
there is usually a single long path which passes through the same
|
||||
piece of equipment many, many times before the answer is
|
||||
obtained.",
|
||||
month = apr,
|
||||
year = 1950,
|
||||
url = "http://dx.doi.org/10.1002/j.1538-7305.1950.tb00463.x",
|
||||
doi = "10.1002/j.1538-7305.1950.tb00463.x",
|
||||
issn = "0005-8580"
|
||||
}
|
||||
|
||||
@ARTICLE{Ondov2016-dq,
|
||||
title = "{Mash: fast genome and metagenome distance estimation using
|
||||
{MinHash}}",
|
||||
author = "Ondov, Brian D and Treangen, Todd J and Melsted, Páll and
|
||||
Mallonee, Adam B and Bergman, Nicholas H and Koren, Sergey and
|
||||
Phillippy, Adam M",
|
||||
journal = "Genome biology",
|
||||
volume = 17,
|
||||
number = 1,
|
||||
pages = 132,
|
||||
abstract = "Mash extends the MinHash dimensionality-reduction technique to
|
||||
include a pairwise mutation distance and P value significance
|
||||
test, enabling the efficient clustering and search of massive
|
||||
sequence collections. Mash reduces large sequences and sequence
|
||||
sets to small, representative sketches, from which global mutation
|
||||
distances can be rapidly estimated. We demonstrate several use
|
||||
cases, including the clustering of all 54,118 NCBI RefSeq genomes
|
||||
in 33 CPU h; real-time database search using assembled or
|
||||
unassembled Illumina, Pacific Biosciences, and Oxford Nanopore
|
||||
data; and the scalable clustering of hundreds of metagenomic
|
||||
samples by composition. Mash is freely released under a BSD
|
||||
license ( https://github.com/marbl/mash ).",
|
||||
month = jun,
|
||||
year = 2016,
|
||||
url = "http://dx.doi.org/10.1186/s13059-016-0997-x",
|
||||
keywords = "Alignment; Comparative genomics; Genomic distance; Metagenomics;
|
||||
Nanopore; Sequencing",
|
||||
doi = "10.1186/s13059-016-0997-x",
|
||||
pmc = "PMC4915045",
|
||||
pmid = 27323842,
|
||||
issn = "1465-6906",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Bray1957-vv,
|
||||
title = "{An ordination of the upland forest communities of southern
|
||||
Wisconsin}",
|
||||
author = "Bray, J Roger and Curtis, J T",
|
||||
journal = "Ecological Monographs",
|
||||
publisher = "Wiley",
|
||||
volume = 27,
|
||||
number = 4,
|
||||
pages = "325--349",
|
||||
month = oct,
|
||||
year = 1957,
|
||||
url = "http://dx.doi.org/10.2307/1942268",
|
||||
doi = "10.2307/1942268",
|
||||
issn = "0012-9615,1557-7015",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Gower1986-ri,
|
||||
title = "{Metric and Euclidean properties of dissimilarity coefficients}",
|
||||
author = "Gower, J C and Legendre, P",
|
||||
journal = "Journal of Classification",
|
||||
publisher = "Springer Science and Business Media LLC",
|
||||
volume = 3,
|
||||
number = 1,
|
||||
pages = "5--48",
|
||||
abstract = "We assemble here properties of certain dissimilarity coefficients
|
||||
and are specially concerned with their metric and Euclidean
|
||||
status. No attempt is made to be exhaustive as far as
|
||||
coefficients are concerned, but certain mathematical results that
|
||||
we have found useful are presented and should help establish
|
||||
similar properties for other coefficients. The response to
|
||||
different types of data is investigated, leading to guidance on
|
||||
the choice of an appropriate coefficient.",
|
||||
month = mar,
|
||||
year = 1986,
|
||||
url = "http://dx.doi.org/10.1007/bf01896809",
|
||||
doi = "10.1007/bf01896809",
|
||||
issn = "0176-4268,1432-1343",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Legendre2001-cq,
|
||||
title = "{Ecologically meaningful transformations for ordination of species
|
||||
data}",
|
||||
author = "Legendre, Pierre and Gallagher, Eugene D",
|
||||
journal = "Oecologia",
|
||||
volume = 129,
|
||||
number = 2,
|
||||
pages = "271--280",
|
||||
abstract = "This paper examines how to obtain species biplots in unconstrained
|
||||
or constrained ordination without resorting to the Euclidean
|
||||
distance [used in principal-component analysis (PCA) and
|
||||
redundancy analysis (RDA)] or the chi-square distance [preserved
|
||||
in correspondence analysis (CA) and canonical correspondence
|
||||
analysis (CCA)] which are not always appropriate for the analysis
|
||||
of community composition data. To achieve this goal,
|
||||
transformations are proposed for species data tables. They allow
|
||||
ecologists to use ordination methods such as PCA and RDA, which
|
||||
are Euclidean-based, for the analysis of community data, while
|
||||
circumventing the problems associated with the Euclidean distance,
|
||||
and avoiding CA and CCA which present problems of their own in
|
||||
some cases. This allows the use of the original (transformed)
|
||||
species data in RDA carried out to test for relationships with
|
||||
explanatory variables (i.e. environmental variables, or factors of
|
||||
a multifactorial analysis-of-variance model); ecologists can then
|
||||
draw biplots displaying the relationships of the species to the
|
||||
explanatory variables. Another application allows the use of
|
||||
species data in other methods of multivariate data analysis which
|
||||
optimize a least-squares loss function; an example is K-means
|
||||
partitioning.",
|
||||
month = oct,
|
||||
year = 2001,
|
||||
url = "http://dx.doi.org/10.1007/s004420100716",
|
||||
keywords = "Biplot diagram; Canonical correspondence analysis; Correspondence
|
||||
analysis; Principal-component analysis; Redundancy analysis",
|
||||
doi = "10.1007/s004420100716",
|
||||
pmid = 28547606,
|
||||
issn = "0029-8549,1432-1939",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Anderson2011-rq,
|
||||
title = "{Navigating the multiple meanings of β diversity: a roadmap for
|
||||
the practicing ecologist}",
|
||||
author = "Anderson, Marti J and Crist, Thomas O and Chase, Jonathan M and
|
||||
Vellend, Mark and Inouye, Brian D and Freestone, Amy L and
|
||||
Sanders, Nathan J and Cornell, Howard V and Comita, Liza S and
|
||||
Davies, Kendi F and Harrison, Susan P and Kraft, Nathan J B and
|
||||
Stegen, James C and Swenson, Nathan G",
|
||||
journal = "Ecology Letters",
|
||||
publisher = "Wiley",
|
||||
volume = 14,
|
||||
number = 1,
|
||||
pages = "19--28",
|
||||
abstract = "A recent increase in studies of β diversity has yielded a
|
||||
confusing array of concepts, measures and methods. Here, we
|
||||
provide a roadmap of the most widely used and ecologically
|
||||
relevant approaches for analysis through a series of mission
|
||||
statements. We distinguish two types of β diversity: directional
|
||||
turnover along a gradient vs. non-directional variation.
|
||||
Different measures emphasize different properties of ecological
|
||||
data. Such properties include the degree of emphasis on
|
||||
presence/absence vs. relative abundance information and the
|
||||
inclusion vs. exclusion of joint absences. Judicious use of
|
||||
multiple measures in concert can uncover the underlying nature of
|
||||
patterns in β diversity for a given dataset. A case study of
|
||||
Indonesian coral assemblages shows the utility of a multi-faceted
|
||||
approach. We advocate careful consideration of relevant
|
||||
questions, matched by appropriate analyses. The rigorous
|
||||
application of null models will also help to reveal potential
|
||||
processes driving observed patterns in β diversity.",
|
||||
month = jan,
|
||||
year = 2011,
|
||||
url = "http://dx.doi.org/10.1111/j.1461-0248.2010.01552.x",
|
||||
doi = "10.1111/j.1461-0248.2010.01552.x",
|
||||
pmid = 21070562,
|
||||
issn = "1461-023X,1461-0248",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Jaccard1901-mx,
|
||||
title = "{Étude comparative de la distribution florale dans une portion
|
||||
des Alpes et du Jura}",
|
||||
author = "Jaccard, Paul",
|
||||
journal = "Bulletin de la Société vaudoise des sciences naturelles",
|
||||
publisher = "Imprimerie Corbaz \& Comp.",
|
||||
volume = 37,
|
||||
number = 142,
|
||||
pages = "547--579",
|
||||
abstract = "Schweizer Zeitschriften online - Ein Service der ETH-Bibliothek.
|
||||
Revues suisses en ligne - Un service de ETH Library. Swiss
|
||||
journals online - A service by ETH Library.",
|
||||
year = 1901,
|
||||
url = "http://dx.doi.org/10.5169/seals-266450",
|
||||
doi = "10.5169/seals-266450",
|
||||
issn = "0037-9603",
|
||||
language = "de"
|
||||
}
|
||||
@ARTICLE{Kosub2019-xt,
|
||||
title = "{A note on the triangle inequality for the Jaccard distance}",
|
||||
author = "Kosub, Sven",
|
||||
journal = "Pattern Recognition Letters",
|
||||
publisher = "Elsevier BV",
|
||||
volume = 120,
|
||||
pages = "36--38",
|
||||
month = apr,
|
||||
year = 2019,
|
||||
url = "http://dx.doi.org/10.1016/j.patrec.2018.12.007",
|
||||
doi = "10.1016/j.patrec.2018.12.007",
|
||||
issn = "0167-8655,1872-7344",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Fleischmann2014-ed,
|
||||
title = {Evolution of genome size and chromosome number in the
|
||||
carnivorous plant genus Genlisea (Lentibulariaceae), with a new
|
||||
estimate of the minimum genome size in angiosperms},
|
||||
author = "Fleischmann, Andreas and Michael, Todd P and Rivadavia, Fernando
|
||||
and Sousa, Aretuza and Wang, Wenqin and Temsch, Eva M and
|
||||
Greilhuber, Johann and Müller, Kai F and Heubl, Günther",
|
||||
journal = "Annals of Botany",
|
||||
publisher = "Oxford University Press (OUP)",
|
||||
volume = 114,
|
||||
number = 8,
|
||||
pages = "1651--1663",
|
||||
abstract = "BACKGROUND AND AIMS: Some species of Genlisea possess ultrasmall
|
||||
nuclear genomes, the smallest known among angiosperms, and some
|
||||
have been found to have chromosomes of diminutive size, which may
|
||||
explain why chromosome numbers and karyotypes are not known for
|
||||
the majority of species of the genus. However, other members of
|
||||
the genus do not possess ultrasmall genomes, nor do most taxa
|
||||
studied in related genera of the family or order. This study
|
||||
therefore examined the evolution of genome sizes and chromosome
|
||||
numbers in Genlisea in a phylogenetic context. The correlations
|
||||
of genome size with chromosome number and size, with the
|
||||
phylogeny of the group and with growth forms and habitats were
|
||||
also examined. METHODS: Nuclear genome sizes were measured from
|
||||
cultivated plant material for a comprehensive sampling of taxa,
|
||||
including nearly half of all species of Genlisea and representing
|
||||
all major lineages. Flow cytometric measurements were conducted
|
||||
in parallel in two laboratories in order to compare the
|
||||
consistency of different methods and controls. Chromosome counts
|
||||
were performed for the majority of taxa, comparing different
|
||||
staining techniques for the ultrasmall chromosomes. KEY RESULTS:
|
||||
Genome sizes of 15 taxa of Genlisea are presented and interpreted
|
||||
in a phylogenetic context. A high degree of congruence was found
|
||||
between genome size distribution and the major phylogenetic
|
||||
lineages. Ultrasmall genomes with 1C values of <100 Mbp were
|
||||
almost exclusively found in a derived lineage of South American
|
||||
species. The ancestral haploid chromosome number was inferred to
|
||||
be n = 8. Chromosome numbers in Genlisea ranged from 2n = 2x = 16
|
||||
to 2n = 4x = 32. Ascendant dysploid series (2n = 36, 38) are
|
||||
documented for three derived taxa. The different ploidy levels
|
||||
corresponded to the two subgenera, but were not directly
|
||||
correlated to differences in genome size; the three different
|
||||
karyotype ranges mirrored the different sections of the genus.
|
||||
The smallest known plant genomes were not found in G. margaretae,
|
||||
as previously reported, but in G. tuberosa (1C ≈ 61 Mbp) and some
|
||||
strains of G. aurea (1C ≈ 64 Mbp). CONCLUSIONS: Genlisea is an
|
||||
ideal candidate model organism for the understanding of genome
|
||||
reduction as the genus includes species with both relatively
|
||||
large (∼1700 Mbp) and ultrasmall (∼61 Mbp) genomes. This
|
||||
comparative, phylogeny-based analysis of genome sizes and
|
||||
karyotypes in Genlisea provides essential data for selection of
|
||||
suitable species for comparative whole-genome analyses, as well
|
||||
as for further studies on both the molecular and cytogenetic
|
||||
basis of genome reduction in plants.",
|
||||
month = dec,
|
||||
year = 2014,
|
||||
url = "http://dx.doi.org/10.1093/aob/mcu189",
|
||||
keywords = "Bladderwort; Genlisea; Lamiales; Lentibulariaceae; carnivorous
|
||||
plant; chromosome number; flow cytometry; genome miniaturization;
|
||||
genome size",
|
||||
doi = "10.1093/aob/mcu189",
|
||||
pmc = "PMC4649684",
|
||||
pmid = 25274549,
|
||||
issn = "0305-7364,1095-8290",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Fernandez2024-kd,
|
||||
title = "A 160 Gbp fork fern genome shatters size record for eukaryotes",
|
||||
author = "Fernández, Pol and Amice, Rémy and Bruy, David and Christenhusz,
|
||||
Maarten J M and Leitch, Ilia J and Leitch, Andrew L and Pokorny,
|
||||
Lisa and Hidalgo, Oriane and Pellicer, Jaume",
|
||||
journal = "iScience",
|
||||
publisher = "Elsevier BV",
|
||||
volume = 27,
|
||||
number = 6,
|
||||
pages = 109889,
|
||||
abstract = "Vascular plants are exceptional among eukaryotes due to their
|
||||
outstanding genome size diversity which ranges ∼2,400-fold,
|
||||
including the largest genome so far recorded in the angiosperm
|
||||
Paris japonica (148.89 Gbp/1C). Despite available data showing
|
||||
that giant genomes are restricted across the Tree of Life, the
|
||||
biological limits to genome size expansion remain to be
|
||||
established. Here, we report the discovery of an even larger
|
||||
eukaryotic genome in Tmesipteris oblanceolata, a New Caledonian
|
||||
fork fern. At 160.45 Gbp/1C, this record-breaking genome
|
||||
challenges current understanding and opens new avenues to explore
|
||||
the evolutionary dynamics of genomic gigantism.",
|
||||
month = jun,
|
||||
year = 2024,
|
||||
url = "http://dx.doi.org/10.1016/j.isci.2024.109889",
|
||||
keywords = "Plant evolution; Plant genetics",
|
||||
doi = "10.1016/j.isci.2024.109889",
|
||||
pmc = "PMC11270024",
|
||||
pmid = 39055604,
|
||||
issn = "2589-0042",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Fraser1995-jc,
|
||||
title = "The minimal gene complement of Mycoplasma genitalium",
|
||||
author = "Fraser, C M and Gocayne, J D and White, O and Adams, M D and
|
||||
Clayton, R A and Fleischmann, R D and Bult, C J and Kerlavage, A
|
||||
R and Sutton, G and Kelley, J M and Fritchman, R D and Weidman, J
|
||||
F and Small, K V and Sandusky, M and Fuhrmann, J and Nguyen, D
|
||||
and Utterback, T R and Saudek, D M and Phillips, C A and Merrick,
|
||||
J M and Tomb, J F and Dougherty, B A and Bott, K F and Hu, P C
|
||||
and Lucier, T S and Peterson, S N and Smith, H O and Hutchison,
|
||||
3rd, C A and Venter, J C",
|
||||
journal = "Science (New York, N.Y.)",
|
||||
publisher = "American Association for the Advancement of Science (AAAS)",
|
||||
volume = 270,
|
||||
number = 5235,
|
||||
pages = "397--403",
|
||||
abstract = "The complete nucleotide sequence (580,070 base pairs) of the
|
||||
Mycoplasma genitalium genome, the smallest known genome of any
|
||||
free-living organism, has been determined by whole-genome random
|
||||
sequencing and assembly. A total of only 470 predicted coding
|
||||
regions were identified that include genes required for DNA
|
||||
replication, transcription and translation, DNA repair, cellular
|
||||
transport, and energy metabolism. Comparison of this genome to
|
||||
that of Haemophilus influenzae suggests that differences in
|
||||
genome content are reflected as profound differences in
|
||||
physiology and metabolic capacity between these two organisms.",
|
||||
month = oct,
|
||||
year = 1995,
|
||||
url = "http://dx.doi.org/10.1126/science.270.5235.397",
|
||||
doi = "10.1126/science.270.5235.397",
|
||||
pmid = 7569993,
|
||||
issn = "0036-8075,1095-9203",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Schneiker2007-gs,
|
||||
title = "Complete genome sequence of the myxobacterium Sorangium
|
||||
cellulosum",
|
||||
author = "Schneiker, Susanne and Perlova, Olena and Kaiser, Olaf and Gerth,
|
||||
Klaus and Alici, Aysel and Altmeyer, Matthias O and Bartels,
|
||||
Daniela and Bekel, Thomas and Beyer, Stefan and Bode, Edna and
|
||||
Bode, Helge B and Bolten, Christoph J and Choudhuri, Jomuna V and
|
||||
Doss, Sabrina and Elnakady, Yasser A and Frank, Bettina and
|
||||
Gaigalat, Lars and Goesmann, Alexander and Groeger, Carolin and
|
||||
Gross, Frank and Jelsbak, Lars and Jelsbak, Lotte and Kalinowski,
|
||||
Jörn and Kegler, Carsten and Knauber, Tina and Konietzny,
|
||||
Sebastian and Kopp, Maren and Krause, Lutz and Krug, Daniel and
|
||||
Linke, Bukhard and Mahmud, Taifo and Martinez-Arias, Rosa and
|
||||
McHardy, Alice C and Merai, Michelle and Meyer, Folker and
|
||||
Mormann, Sascha and Muñoz-Dorado, Jose and Perez, Juana and
|
||||
Pradella, Silke and Rachid, Shwan and Raddatz, Günter and
|
||||
Rosenau, Frank and Rückert, Christian and Sasse, Florenz and
|
||||
Scharfe, Maren and Schuster, Stephan C and Suen, Garret and
|
||||
Treuner-Lange, Anke and Velicer, Gregory J and Vorhölter,
|
||||
Frank-Jörg and Weissman, Kira J and Welch, Roy D and Wenzel,
|
||||
Silke C and Whitworth, David E and Wilhelm, Susanne and Wittmann,
|
||||
Christoph and Blöcker, Helmut and Pühler, Alfred and Müller, Rolf",
|
||||
journal = "Nature Biotechnology",
|
||||
publisher = "Springer Science and Business Media LLC",
|
||||
volume = 25,
|
||||
number = 11,
|
||||
pages = "1281--1289",
|
||||
abstract = "The genus Sorangium synthesizes approximately half of the
|
||||
secondary metabolites isolated from myxobacteria, including the
|
||||
anti-cancer metabolite epothilone. We report the complete genome
|
||||
sequence of the model Sorangium strain S. cellulosum So ce56,
|
||||
which produces several natural products and has morphological and
|
||||
physiological properties typical of the genus. The circular
|
||||
genome, comprising 13,033,779 base pairs, is the largest
|
||||
bacterial genome sequenced to date. No global synteny with the
|
||||
genome of Myxococcus xanthus is apparent, revealing an
|
||||
unanticipated level of divergence between these myxobacteria. A
|
||||
large percentage of the genome is devoted to regulation,
|
||||
particularly post-translational phosphorylation, which probably
|
||||
supports the strain's complex, social lifestyle. This regulatory
|
||||
network includes the highest number of eukaryotic protein
|
||||
kinase-like kinases discovered in any organism. Seventeen
|
||||
secondary metabolite loci are encoded in the genome, as well as
|
||||
many enzymes with potential utility in industry.",
|
||||
month = nov,
|
||||
year = 2007,
|
||||
url = "http://dx.doi.org/10.1038/nbt1354",
|
||||
doi = "10.1038/nbt1354",
|
||||
pmid = 17965706,
|
||||
issn = "1087-0156,1546-1696",
|
||||
language = "en"
|
||||
}
|
||||
@incollection{Karlin1985,
|
||||
author = {Karlin, Samuel and Ost, Fr{\'e}d{\'e}ric},
|
||||
title = {Maximal segmental match length among random sequences from a finite alphabet},
|
||||
booktitle = {Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer},
|
||||
editor = {Le Cam, Lucien M. and Olshen, Richard A.},
|
||||
volume = {1},
|
||||
pages = {225--243},
|
||||
year = {1985},
|
||||
publisher = {Wadsworth},
|
||||
address = {Monterey, CA}
|
||||
}
|
||||
@article{Karlin1988,
|
||||
author = {Karlin, Samuel and Ost, Fr{\'e}d{\'e}ric},
|
||||
title = {Maximal length of common words among random letter sequences},
|
||||
journal = {The Annals of Probability},
|
||||
volume = {16},
|
||||
number = {2},
|
||||
pages = {535--563},
|
||||
year = {1988}
|
||||
}
|
||||
@ARTICLE{Le-Cam1960-pe,
|
||||
title = "{An approximation theorem for the Poisson binomial distribution}",
|
||||
author = "Le Cam, Lucien",
|
||||
journal = "Pacific Journal of Mathematics",
|
||||
publisher = "Mathematical Sciences Publishers",
|
||||
volume = 10,
|
||||
number = 4,
|
||||
pages = "1181--1197",
|
||||
month = dec,
|
||||
year = 1960,
|
||||
url = "http://dx.doi.org/10.2140/pjm.1960.10.1181",
|
||||
doi = "10.2140/pjm.1960.10.1181",
|
||||
issn = "0030-8730,1945-5844"
|
||||
}
|
||||
|
||||
@ARTICLE{Hoeffding1956-bx,
|
||||
title = "{On the distribution of the number of successes in independent
|
||||
trials}",
|
||||
author = "Hoeffding, Wassily",
|
||||
journal = "The Annals of Mathematical Statistics",
|
||||
publisher = "Institute of Mathematical Statistics",
|
||||
volume = 27,
|
||||
number = 3,
|
||||
pages = "713--721",
|
||||
abstract = "Let $S$ be the number of successes in $n$ independent trials, and
|
||||
let $p_j$ denote the probability of success in the $j$th trial,
|
||||
$j = 1, 2, \cdots, n$ (Poisson trials). We consider the problem
|
||||
of finding the maximum and the minimum of $Eg(S),$ the expected
|
||||
value of a given real-valued function of $S,$ when $ES = np$ is
|
||||
fixed. It is well known that the maximum of the variance of $S$
|
||||
is attained when $p_1 = p_2 = \cdots = p_n = p.$ This can be
|
||||
interpreted as showing that the variability in the number of
|
||||
successes is highest when the successes are equally probable
|
||||
(Bernoulli trials). This interpretation is further supported by
|
||||
the following two theorems, proved in this paper. If $b$ and $c$
|
||||
are two integers, $0 \leqq b \leqq np \leqq c \leqq n,$ the
|
||||
probability $P(b \leqq S \leqq c)$ attains its minimum if and
|
||||
only if $p_1 = p_2 = \cdots = p_n = p,$ unless $b = 0$ and $c =
|
||||
n$ (Theorem 5, a corollary of Theorem 4, which gives the maximum
|
||||
and the minimum of $P(S \leqq c)$). If $g$ is a strictly convex
|
||||
function, $Eg(S)$ attains its maximum if and only if $p_1 = p_2 =
|
||||
\cdots = p_n = p$ (Theorem 3). These results are obtained with
|
||||
the help of two theorems concerning the extrema of the expected
|
||||
value of an arbitrary function $g(S)$ under the condition $ES =
|
||||
np.$ Theorem 1 gives necessary conditions for the maximum and the
|
||||
minimum of $Eg(S).$ Theorem 2 gives a partial characterization of
|
||||
the set of points at which an extremum is attained. Corollary 2.1
|
||||
states that the maximum and the minimum are attained when $p_1,
|
||||
p_2, \cdots, p_n$ take on, at most, three different values, only
|
||||
one of which is distinct from 0 and 1. Applications of Theorems 3
|
||||
and 5 to problems of estimation and testing are pointed out in
|
||||
Section 5.",
|
||||
month = sep,
|
||||
year = 1956,
|
||||
url = "http://dx.doi.org/10.1214/aoms/1177728178",
|
||||
doi = "10.1214/aoms/1177728178",
|
||||
issn = "0003-4851,2168-8990",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Gleser1975-yu,
|
||||
title = "{On the distribution of the number of successes in independent
|
||||
trials}",
|
||||
author = "Gleser, Leon Jay",
|
||||
journal = "Annals of Probability",
|
||||
publisher = "Institute of Mathematical Statistics",
|
||||
volume = 3,
|
||||
number = 1,
|
||||
pages = "182--188",
|
||||
abstract = "Let $S$ be the number of successes in $n$ independent Bernoulli
|
||||
trials, where $p_j$ is the probability of success on the $j$th
|
||||
trial. Let $\mathbf{p} = (p_1, p_2, \cdots, p_n)$, and for any
|
||||
integer $c, 0 \leqq c \leqq n$, let $H(c \mid \mathbf{p}) = P\{S
|
||||
\leqq c\}$. Let $\mathbf{p}^{(1)}$ be one possible choice of
|
||||
$\mathbf{p}$ for which $E(S) = \lambda$. For any $n \times n$
|
||||
doubly stochastic matrix $\Pi$, let $\mathbf{p}^{(2)} =
|
||||
\mathbf{p}^{(1)}\Pi$. Then in the present paper it is shown that
|
||||
$H(c \mid \mathbf{p}^{(1)}) \leqq H(c \mid \mathbf{p}^{(2)})$ for
|
||||
$0 \leqq c \leqq \lbrack\lambda - 2\rbrack$, and $H(c \mid
|
||||
\mathbf{p}^{(1)}) \geqq H(c \mid \mathbf{p}^{(2)})$ for
|
||||
$\lbrack\lambda + 2\rbrack \leqq c \leqq n$. These results
|
||||
provide a refinement of inequalities for $H(c \mid \mathbf{p})$
|
||||
obtained by Hoeffding [3]. Their derivation is achieved by
|
||||
applying consequences of the partial ordering of majorization.",
|
||||
month = feb,
|
||||
year = 1975,
|
||||
url = "http://dx.doi.org/10.2307/2959277",
|
||||
doi = "10.1214/aop/1176996461",
|
||||
issn = "0091-1798,2168-894X"
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
# Partitioning and indexing architecture
|
||||
|
||||
An index is split into a fixed number of **partitions**, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
|
||||
|
||||
## Routing
|
||||
|
||||
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition:
|
||||
|
||||
```
|
||||
canonical minimizer → hash(minimizer) → p-bit value → partition index
|
||||
```
|
||||
|
||||
The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.
|
||||
|
||||
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
|
||||
|
||||
## Why hashing is necessary
|
||||
|
||||
A canonical minimizer is an m-mer ($m \in \{9, 11, 13, 15\}$), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.
|
||||
|
||||
Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits $p$ is chosen well below the number of bits available in the minimizer ($2m$): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
|
||||
|
||||
## Parameter guidance
|
||||
|
||||
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|
||||
|----|-----------|-----------|------------|
|
||||
| 9 | 18 | 6–8 | 64–256 |
|
||||
| 11 | 22 | 8–10 | 256–1 024 |
|
||||
| 13 | 26 | 10–12 | 1 024–4 096|
|
||||
| 15 | 30 | 10–14 | 1 024–16 384|
|
||||
|
||||
The number of partitions must satisfy $p \le 2m$, and in practice $p$ is chosen well below that bound to leave a comfortable entropy margin. For $k=31$, $m=13$, $p=10$ (1024 partitions), partition load is well balanced on real genomic data.
|
||||
@@ -21,8 +21,8 @@ Reverse complement is computed by bit manipulation directly on the packed word,
|
||||
|
||||
## Canonical form
|
||||
|
||||
The canonical form of a kmer is the lexicographic minimum of the kmer and its reverse complement:
|
||||
The canonical form of a packed 2-bit sequence is the lexicographic minimum of the sequence and its reverse complement:
|
||||
|
||||
$$\text{canonical}(kmer) = \min\big(kmer,\ \text{revcomp}(kmer)\big)$$
|
||||
$$\text{canonical}(x) = \min\big(x,\ \text{revcomp}(x)\big)$$
|
||||
|
||||
Using the canonical form halves the kmer space and makes counting strand-independent: a kmer and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.
|
||||
This is a single operation on the packed representation, so it applies identically regardless of what `x` represents — a kmer (see [Kmers](kmers.md)), an m-mer (see [Minimizer selection](minimizer_selection.md)), or a super-kmer (see [Super-kmers](superkmers.md)). Using the canonical form halves the relevant space and makes counting/selection strand-independent: a sequence and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.
|
||||
File renamed without changes.
@@ -0,0 +1,28 @@
|
||||
# Partitioning and indexing architecture
|
||||
|
||||
An index is split into a fixed number of **partitions**, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
|
||||
|
||||
## Routing
|
||||
|
||||
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition — the low $p$ bits of the minimizer's hash:
|
||||
|
||||
```
|
||||
canonical minimizer → hash(minimizer) → p-bit value → partition index
|
||||
```
|
||||
|
||||
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
|
||||
|
||||
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
|
||||
|
||||
## Parameter guidance
|
||||
|
||||
Even though $H$ already makes minimizer values well-distributed (see [Minimizer selection](minimizer_selection.md)), choosing $p$ well below the number of bits available in the minimizer ($2m$) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
|
||||
|
||||
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|
||||
|----|-----------|-----------|------------|
|
||||
| 9 | 18 | 6–8 | 64–256 |
|
||||
| 11 | 22 | 8–10 | 256–1 024 |
|
||||
| 13 | 26 | 10–12 | 1 024–4 096|
|
||||
| 15 | 30 | 10–14 | 1 024–16 384|
|
||||
|
||||
The number of partitions must satisfy $p \le 2m$, and in practice $p$ is chosen well below that bound to leave a comfortable entropy margin. For $k=31$, $m=13$, $p=10$ (1024 partitions), partition load is well balanced on real genomic data.
|
||||
@@ -0,0 +1,416 @@
|
||||
# K-mer size
|
||||
|
||||
A k-mer is a DNA sequence of length $k$. OBIKmer is designed to index large k-mers. Here, *large* refers to a k-mer length for which the probability of observing the same k-mer in several genomes by chance alone is sufficiently small. The k-mer length must therefore be large enough that, when a k-mer is observed in several genomes of the dataset, the probability that this shared occurrence is due to chance alone is sufficiently low for the match to be considered evidence of homology.
|
||||
|
||||
OBIKmer restricts $k$ to odd values in the range
|
||||
|
||||
$$
|
||||
11 \leq k \leq 31.
|
||||
$$
|
||||
|
||||
These limits result from two different considerations. The upper bound is imposed by the representation used by OBIKmer, whereas the lower bound defines a practical range of k-mer lengths for genome-scale sequence comparisons. This range covers most practical applications: the lower limit is well below the k-mer size that can be considered large, even for the smallest bacterial genomes, whereas the upper limit provides sufficient specificity for most applications, including those involving very large genomes.
|
||||
|
||||
## Representation limit
|
||||
|
||||
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer requires $2k$ bits. To ensure computational efficiency, OBIKmer limits the representation of k-mers to a single 64-bit word, thereby restricting their length.
|
||||
|
||||
$$
|
||||
2k \leq 64,
|
||||
$$
|
||||
|
||||
This gives the hard representation limit: $k \leq 32.$ Furthermore, OBIKmer only considers odd values of $k$, meaning the largest supported k-mer size is $31$.
|
||||
|
||||
### K-mer specificity
|
||||
|
||||
One simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are $4^k$ possible k-mers, so a particular k-mer is expected to occur once every $4^k$ positions on average. In a sequence of length $G$, the expected number of occurrences of a particular k-mer is approximately $G \cdot 4^{-k}$. When k is large enough to ensure $4^k >> G$, the chance of a second occurrence of the same k-mer becomes close to zero. Consequently, observing the same k-mer twice can only be explained by a homology relationship, except for low-complexity k-mers, which are overrepresented in genomes.
|
||||
|
||||
### Estimating False discovery rate on homology inference
|
||||
|
||||
OBIKmer uses k-mers long enough that their occurrence in multiple indexed genomes provides evidence of homology. The corresponding false discovery rate (FDR) can be expressed in terms of the probability of observing a k-mer in at least $R$ of the $S$ indexed genomes under a purely random model.
|
||||
|
||||
#### The random model
|
||||
|
||||
The random model, or null hypothesis, assumes that the $S$ genomes are unrelated and that each genome is an independent random sequence in which each nucleotide occurs with equal probability. Rejecting this hypothesis for a given k-mer indicates that its presence in multiple genomes is unlikely to be explained by chance alone, providing evidence for homology.
|
||||
|
||||
For a genome of size $G$, the number of occurrences of a given canonical k-mer $K$ is approximated by a Poisson distribution with parameter
|
||||
|
||||
$$ \lambda = 2\frac{G}{4^k}, $$
|
||||
|
||||
which is also the expected number of occurrences under the random model. The factor of two accounts for k-mer canonicalisation, which halves the k-mer space when $k$ is odd.
|
||||
|
||||
The probability of observing at least one occurrence of this k-mer is therefore
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
P(n_K > 0) &= 1-e^{-\lambda} \\
|
||||
&= 1-e^{-2G/4^k}.
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
|
||||
|
||||
|
||||
$$
|
||||
P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right){S-j}.
|
||||
$$
|
||||
|
||||
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
|
||||
|
||||
$$
|
||||
p = 1-e^{-2G/4^k}.
|
||||
$$
|
||||
|
||||
Different genome sizes
|
||||
|
||||
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
|
||||
|
||||
$$ \bar G = \frac{1}{S}\sum_{i=1}^{S}G_i. $$
|
||||
|
||||
This gives a common probability
|
||||
|
||||
$$ \bar p = 1-e^{-2\bar G/4^k}, $$
|
||||
|
||||
and the number of genomes containing the k-mer can then be approximated by a binomial distribution:
|
||||
|
||||
$$ P(X\geq r)=\sum_{j=r}^{S}\binom{S}{j}\bar p^j(1-\bar p)^{S-j}. $$
|
||||
|
||||
This approximation is conservative for the upper tail. Hoeffding [-@Hoeffding1956-bx, p. 717, Eq. 26] shows that, for a fixed mean, the binomial upper tail is at least as large as the corresponding Poisson-binomial tail for
|
||||
|
||||
$$ r\geq S\bar p+1. $$
|
||||
|
||||
Using the mean genome size therefore provides a conservative estimate of the FDR.
|
||||
|
||||
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
|
||||
|
||||
### Exact tail probability
|
||||
|
||||
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
|
||||
|
||||
Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
|
||||
|
||||
$$
|
||||
\left \{
|
||||
\begin{aligned}
|
||||
d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\
|
||||
d_0&=1
|
||||
\end{aligned}
|
||||
\right .
|
||||
$$
|
||||
|
||||
before processing the first genome.
|
||||
|
||||
The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then
|
||||
|
||||
$$
|
||||
P(X\geq r)=1-\sum_{j=0}^{r-1}d_j.
|
||||
$$
|
||||
|
||||
Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$.
|
||||
|
||||
When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean
|
||||
|
||||
$$
|
||||
\lambda=\sum_{i=1}^{S}p_i.
|
||||
$$
|
||||
|
||||
The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe].
|
||||
|
||||
|
||||
*******
|
||||
|
||||
This probability decreases by a factor of four for every additional nucleotide in the k-mer. Conversely, it increases linearly with the amount of sequence searched.
|
||||
|
||||
This simple model already shows why very short k-mers provide limited specificity for genome-scale analyses. With $k=11$,
|
||||
|
||||
$$
|
||||
4^{11}\simeq4.2\times10^6,
|
||||
$$
|
||||
|
||||
which is a small combinatorial space even compared with the size of very small bacterial genomes. In contrast,
|
||||
|
||||
$$
|
||||
4^{31}\simeq4.6\times10^{18},
|
||||
$$
|
||||
|
||||
which is vastly larger than the number of k-mer positions even in very large genomes. Thus, increasing $k$ from 11 to 31 provides an enormous increase in sequence specificity.
|
||||
|
||||
The estimate above is intentionally simplistic. Real genomes do not consist of independent, equiprobable nucleotides: nucleotide composition and sequence dependencies affect the probability of observing particular words. More accurate estimates of the probability of observing matches by chance therefore take sequence length and nucleotide composition into account.
|
||||
|
||||
Karlin and Ost provide an analytical framework for this problem based on the distribution of the longest exact match expected between random sequences. If $L_{\max}$ denotes the length of the longest seed observed in random sequences of given size and nucleotide composition, the corresponding seed-level probability is
|
||||
|
||||
$$
|
||||
P_{\mathrm{seeds}}(L)=P(L_{\max}\geq L).
|
||||
$$
|
||||
|
||||
Conversely, for a chosen false-positive probability $P$, one can determine the minimum seed length $L_{\min}$ such that matches of length $L_{\min}$ or greater are not expected to occur by chance. This is the approach used by Repseek to determine statistically significant seed lengths.
|
||||
|
||||
The simple $G/4^k$ estimate and the Karlin–Ost statistics therefore describe the same fundamental effect at different levels of approximation: the probability of a chance match decreases rapidly with k-mer length, while the amount of sequence being searched increases the opportunity for such matches.
|
||||
|
||||
Consequently, the appropriate value of $k$ depends on the total amount of sequence being compared and its nucleotide composition. Choosing $k$ appropriately allows the false-positive rate of homology inference to be controlled.
|
||||
|
||||
|
||||
|
||||
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Mycoplasma genitalium* having a size $G=580\,Mb$ [@Fraser1995-jc], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
P(C\geq1) &= 1- e^{-\frac{580.000^2}{2 \cdot 4^{11}}} \\
|
||||
&\simeq 1
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
, whereas 21-mers already provide a space of several trillion possible sequences.
|
||||
|
||||
Plant genomes illustrate the opposite situation. Plant genome sizes span several orders of magnitude, from the smallest angiosperm genomes present in the carnivorous plans genus [@Fleischmann2014-ed] about 60 Mb to the largest one 160 Gb in Tmesipteris oblanceolata, a New Caledonian fork fern [@Fernandez2024-kd], and large genomes often contain substantial amounts of repeated sequence. Even at $k=31$, however, the theoretical k-mer space remains vastly larger than any individual genome. At these lengths, the limiting factor is therefore not exhaustion of the theoretical k-mer space but the biological repetitiveness and non-random composition of the genome.
|
||||
|
||||
For example, the number of possible k-mers grows as follows:
|
||||
|
||||
| $k$ | $4^k$ | $4^k/2$ after canonicalization |
|
||||
| ----: | --------------------: | -------------------------------: |
|
||||
| 11 | $4.19\times10^6$ | $2.10\times10^6$ |
|
||||
| 15 | $1.07\times10^9$ | $5.37\times10^8$ |
|
||||
| 21 | $4.40\times10^{12}$ | $2.20\times10^{12}$ |
|
||||
| 25 | $1.13\times10^{15}$ | $5.63\times10^{14}$ |
|
||||
| 31 | $4.61\times10^{18}$ | $2.31\times10^{18}$ |
|
||||
|
||||
Canonicalization approximately halves the number of possible representations because a k-mer and its reverse complement are represented by the same canonical form.
|
||||
|
||||
|
||||
|
||||
The distinction between the simple $4^k$ model and the Karlin--Ost framework is important. The former describes the size of the combinatorial k-mer space; the latter describes the probability of observing long matches in sequences whose composition may depart from the uniform random model. Consequently, the value of $k$ cannot be selected from genome size alone.
|
||||
|
||||
|
||||
These scales have very different implications for different genome sizes.
|
||||
|
||||
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995; @Karlin1997]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
|
||||
|
||||
This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.
|
||||
|
||||
The supported range $11\leq k\leq31$ should therefore be understood as an engineering compromise rather than as a universal biological optimum. Smaller values increase the probability that unrelated genomic regions share the same k-mers, while larger values increase specificity but also make the representation more sensitive to substitutions, indels, and incomplete homology. The appropriate value within this range consequently depends on the biological scale and comparison task.
|
||||
|
||||
## Odd k-mer lengths
|
||||
|
||||
OBIKmer additionally requires $k$ to be odd. This constraint is independent of the resolution argument above.
|
||||
|
||||
For an odd-length DNA sequence, a k-mer cannot be identical to its own reverse complement. Consequently, every k-mer has two distinct orientations, which can be unambiguously collapsed into a canonical representation. For even $k$, self-reverse-complementary sequences exist, so a k-mer can coincide with its own reverse complement.
|
||||
|
||||
Odd $k$ is therefore **not required for canonicalization itself**: the canonical form
|
||||
|
||||
$$
|
||||
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
|
||||
$$
|
||||
|
||||
remains well defined for even $k$. The odd-$k$ restriction instead guarantees that the two orientations are always distinct, which simplifies the representation and orientation handling used by OBIKmer.
|
||||
|
||||
## Karlin--Ost (1988) formula
|
||||
|
||||
[@Karlin1988] considers $s$ independent stationary letter sequences over a finite alphabet and studies $K_{r,s}(N)$, the length of the longest word common to $r$ of the $s$ sequences, each of length $N$. For i.i.d. equiprobable letters over an alphabet of size $m$ (here $m=4$ for DNA), the characteristic local match parameter reduces to
|
||||
|
||||
$$
|
||||
\lambda^{[r]} = m^{1-r},
|
||||
$$
|
||||
|
||||
so for $r=2$ (a word shared by two occurrences), $\lambda = m^{-1} = 1/4$.
|
||||
|
||||
The expected number of common words of length $k$ among the $\binom{s}{r}$ relevant groups of sequences is
|
||||
|
||||
$$
|
||||
S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1)),
|
||||
$$
|
||||
|
||||
with $\gamma=1$ exactly in the i.i.d. equiprobable case. Theorem 2.2 gives the limiting distribution of $K_{r,s}(N)$, with $k$ set on the scale $k=\left[\dfrac{\log N^{r}}{-\log\lambda^{*}}+x+1\right]$:
|
||||
|
||||
$$
|
||||
\lim_{N\to\infty}\left[\Pr\{K_{r,s}(N)<k\} - \exp\{-(1-\lambda^{*})\,S_1(N,k)\}\right] = 0,
|
||||
$$
|
||||
|
||||
equivalently
|
||||
|
||||
$$
|
||||
\frac{-\log\Pr\{K_{r,s}(N)<k\}}{(\lambda^{*})^{\rho(N,x)}} \xrightarrow[N\to\infty]{} (1-\lambda^{*})\,\gamma^{*}(\lambda^{*})^{x}.
|
||||
$$
|
||||
|
||||
The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985].
|
||||
|
||||
|
||||
$o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ».
|
||||
|
||||
Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ :
|
||||
|
||||
- $N^{r}\binom{s}{r}\gamma\,\lambda^{k}$ est la valeur **exacte à la limite**, celle qu'on utilise en pratique (c'est elle qui redonne $E[C]$ de notre discussion).
|
||||
- $(1+o(1))$ dit : « la vraie valeur de $S_1(N,k)$ n'est pas *exactement* égale à ce produit pour $N$ fini, mais le rapport entre les deux tend vers 1 quand $N\to\infty$ ». Autrement dit $S_1(N,k) / \left[N^{r}\binom{s}{r}\gamma\lambda^{k}\right] \to 1$.
|
||||
|
||||
Pourquoi ce terme existe : la formule $F_\sigma(k)/\lambda^k \to \gamma$ (éq. 2.8 du papier) est elle-même une limite quand $k\to\infty$ — pour $k$ fini, il y a un petit écart entre $F_\sigma(k)$ et $\gamma\lambda^k$. C'est cet écart-là qui devient le $o(1)$.
|
||||
|
||||
Ce que ça implique pour l'usage pratique : dans le cas iid équiprobable que tu utilises (ADN, $\gamma=1$ exactement, pas seulement à la limite — je l'ai vérifié directement sur la dérivation : $F_\sigma(k)=m^{-k}=\lambda^k$ pour tout $k$, pas juste asymptotiquement), le $o(1)$ est en réalité **nul** dans ce cas précis, pas juste négligeable. La formule $S_1(N,k)=N^r\binom{s}{r}\lambda^k$ est donc exacte (à l'asymptotique en $N$ près, qui elle vient d'ailleurs — de l'approximation de Poisson elle-même, pas de ce $\gamma$).
|
||||
|
||||
|
||||
Les deux approches sont légitimes, mais elles répondent à des questions différentes — et il y a une relation algébrique exacte entre les deux qui vaut la peine d'être explicitée.
|
||||
|
||||
**Ce que calcule l'approche naïve.** Pour **un k-mer $w$ déjà observé et fixé**, $p_i\simeq1-\exp(-G_i/4^k)$ est la probabilité que *ce mot précis* apparaisse quelque part dans le génome $i$ par hasard, et $P(R\geq r)$ (Poisson-binomiale ou binomiale) est la probabilité que *ce mot précis* soit partagé par au moins $r$ des $S$ génomes par hasard. C'est une **p-value par observation** : « j'ai vu ce k-mer particulier dans 5 génomes sur 10, est-ce surprenant pour lui ? »
|
||||
|
||||
**Ce que calcule Karlin-Ost.** $S_1(k)$ et $P(K_{r,s}\geq k)$ répondent à : « parmi **tous les $4^k$ mots possibles**, quelle est la probabilité qu'au moins un seul d'entre eux atteigne ce seuil de partage par hasard, n'importe où dans les génomes ? » C'est une question **globale sur tout l'espace des k-mers**, pas sur un mot déjà identifié.
|
||||
|
||||
**La relation exacte entre les deux.** Reprenons le cas $r=s=2$, génomes égaux de taille $G$ :
|
||||
|
||||
- Naïf (mot $w$ fixé) : $P(R=2\mid w)\simeq p^2=(G\cdot4^{-k})^2=G^2\cdot4^{-2k}$
|
||||
- Karlin-Ost : $S_1=G^2\cdot4^{-k}$
|
||||
|
||||
Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nombre de mots possibles. Ce n'est pas une coïncidence : $S_1=N^r\binom{s}{r}\lambda^k$ vient de $\lambda^k=\sum_w \Pr\{w\}^r$, une somme explicite sur les $4^k$ mots, alors que le calcul naïf tel que présenté ne somme sur rien — il traite $w$ comme donné.
|
||||
|
||||
**Donc l'approche naïve, formellement, c'est Karlin-Ost auquel il manque le facteur « nombre de k-mers testés ».** Pour la rendre comparable à la question « dois-je m'attendre à *une* collision fortuite n'importe où dans mon jeu de données », il faudrait une correction à la Bonferroni : multiplier $P(R\geq r\mid w)$ par le nombre effectif de k-mers canoniques testables ($\approx4^k/2$). Karlin-Ost fait cette correction analytiquement et correctement (avec en prime le facteur de déclumping $(1-\lambda)$ qu'on a discuté, qui corrige le sur-comptage des matches chevauchants — une union bound brute comme le Bonferroni décrit ci-dessus le surestimerait légèrement).
|
||||
|
||||
**Ce qui veut dire, en pratique, que les deux outils ne sont pas concurrents mais complémentaires :**
|
||||
- **Choix a priori de $k$ / FPR global du dataset** (la question de `kmers.md` depuis le début de cette conversation) → Karlin-Ost, parce qu'il intègre nativement la multiplicité sur tous les mots possibles.
|
||||
- **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant.
|
||||
|
||||
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
|
||||
|
||||
|
||||
## Chance sharing across genomes and the FDR of homology
|
||||
|
||||
The formulas above ask how likely a match is between two sequences. The question OBIKmer actually has to answer is different: a k-mer is found in $r$ of the $S$ indexed genomes, and one wants to know whether this sharing can be explained by chance or whether it is evidence of homology. This section builds a null model for that question and turns it into a false discovery rate (FDR) for the call "a k-mer shared by at least $r$ genomes is homologous".
|
||||
|
||||
### The null model
|
||||
|
||||
The null hypothesis is that the $S$ genomes are unrelated, that is, that each one is an independent random sequence. Rejecting it for a k-mer means that chance alone cannot explain why the k-mer is shared, which is the signal of homology one is looking for.
|
||||
|
||||
The model only records whether a k-mer is present in a genome, not how many times. Two consequences follow. First, repeats inside a genome are irrelevant, and the size $G_i$ of genome $i$ is its number of **distinct canonical k-mers**. Second, a repeat family shared between genomes is paralogy, which is a form of homology; it is a true positive and not a false one. The only k-mers that could be shared by composition rather than by descent are low-complexity ones, which OBIKmer removes beforehand with its entropy filter.
|
||||
|
||||
Everything takes place in canonical space. Because $k$ is odd, no k-mer equals its own reverse complement, so the $4^k$ k-mers pair up perfectly and the space contains exactly
|
||||
|
||||
$$
|
||||
N=\frac{4^k}{2}
|
||||
$$
|
||||
|
||||
canonical k-mers. A given canonical k-mer is present in genome $i$ by chance with probability
|
||||
|
||||
$$
|
||||
p_i = 1-\left(1-\frac1N\right)^{G_i}\simeq \frac{G_i}{N}=\frac{2\,G_i}{4^k}.
|
||||
$$
|
||||
|
||||
This is twice the $G/4^k$ used earlier in this page, because there $G$ is a number of positions on one strand, whereas $G_i$ here counts canonical k-mers, for which both strands are already merged.
|
||||
|
||||
Let $R$ be the number of genomes, out of $S$, that contain a given canonical k-mer by chance. Genomes are independent under the null, so $R$ is a sum of independent presence indicators, each with its own probability $p_i$.
|
||||
|
||||
### Exact tail probability
|
||||
|
||||
When genome sizes differ, the $p_i$ differ, and $R$ follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be hopeless, since there are up to $2^S$ subsets. The distribution is instead obtained by adding the genomes one at a time. Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
|
||||
|
||||
$$
|
||||
d'_j = d_j\,(1-p_i) + d_{j-1}\,p_i, \qquad d_0=1 \text{ before the first genome.}
|
||||
$$
|
||||
|
||||
The table $d$ only needs to remember how many genomes were hit, not which ones, and this brings the cost down to $O(S^2)$. The result does not depend on the order in which the genomes are processed. It does not depend on the k-mer either, so it is computed once per index. The upper tail is then
|
||||
|
||||
$$
|
||||
P_0(R\geq r) = 1-\sum_{j=0}^{r-1} d_j .
|
||||
$$
|
||||
|
||||
Only $d_0,\dots,d_{r-1}$ are needed, so the cost is $O(S\,r)$.
|
||||
|
||||
### Approximation with equal genome sizes
|
||||
|
||||
If all genomes are given the same size, taken as the mean
|
||||
|
||||
$$
|
||||
\bar G=\frac1S\sum_{i=1}^{S}G_i, \qquad \bar p=\frac{2\bar G}{4^k},
|
||||
$$
|
||||
|
||||
then $R$ follows a plain binomial distribution and the tail is
|
||||
|
||||
$$
|
||||
P_0(R\geq r)\simeq\sum_{j=r}^{S}\binom{S}{j}\,\bar p^{\,j}(1-\bar p)^{S-j}.
|
||||
$$
|
||||
|
||||
This sum is the regularized incomplete beta function $I_{\bar p}(r,\,S-r+1)$, available in standard statistical libraries. If it is summed by hand, the terms are obtained one from the next by $t_{j+1}=t_j\,\frac{S-j}{j+1}\,\frac{\bar p}{1-\bar p}$.
|
||||
|
||||
A theorem of Hoeffding [@Hoeffding1956-bx] shows that this approximation errs on the safe side. For a fixed mean, the Poisson-binomial distribution is more concentrated than the binomial one, so for $r\geq S\bar p+1$ the binomial tail is never smaller than the exact tail. Using the mean genome size therefore overestimates the chance of sharing and can only make the FDR more pessimistic. Gleser [@Gleser1975-yu] refines this result: replacing the $p_i$ by any partial average of themselves, not only by their full mean, also moves the upper tail towards the binomial one, provided $r-1\geq S\bar p+2$. Grouping genomes of similar size is therefore safe in the same sense. What it loses is power when genome sizes differ by orders of magnitude, since one large genome then weighs as much as several small ones in $\bar p$.
|
||||
|
||||
When all $p_i$ are small, which is the case for large $k$, the number of hits is also well approximated by a Poisson distribution of mean $\lambda=\sum_i p_i$, with an error bounded by $\sum_i p_i^2$ [@Le-Cam1960-pe].
|
||||
|
||||
For the numerical evaluation, the tail that does not contain the mode $S\bar p$ should be summed directly. When $r>S\bar p$, computing the sum from $j=r$ upwards is accurate even for very small probabilities, whereas computing $1$ minus the sum of the lower terms loses all precision once the tail falls below the machine epsilon.
|
||||
|
||||
### False discovery rate
|
||||
|
||||
The FDR of the call "present in at least $r$ genomes" is the fraction of retained k-mers expected to be pure chance:
|
||||
|
||||
$$
|
||||
\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P_0(R\geq r)}{n_{\geq r}}\right).
|
||||
$$
|
||||
|
||||
The numerator is the number of k-mers expected to reach $r$ genomes by chance if the genomes were unrelated. Each of the $N$ canonical k-mers has probability $P_0(R\geq r)$ of doing so, and no further correction for multiple testing is needed because the number of tests is already in the product. The denominator $n_{\geq r}$ is the number of distinct k-mers actually observed in at least $r$ genomes, which is read directly from the index. The observed count mixes chance and true homology, so the ratio is an upper bound on the true FDR: it implicitly assumes that all k-mers are null, and it never underestimates the noise. The minimal $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls below the accepted level, for example 5 %, is the sharing threshold to use.
|
||||
|
||||
As an illustration, take $k=21$ (so $N=2.2\times10^{12}$), $S=10$ genomes and $G_i=10^9$ distinct canonical k-mers each, hence $p=4.5\times10^{-4}$. The number of k-mers expected to be shared by chance is:
|
||||
|
||||
| $r$ | $N\,P_0(R\geq r)$ |
|
||||
| ---: | ---: |
|
||||
| 2 | $2\times10^{7}$ |
|
||||
| 3 | $2.5\times10^{4}$ |
|
||||
| 4 | $\approx 20$ |
|
||||
| 5 | $\approx 0.01$ |
|
||||
|
||||
Chance alone produces millions of pairwise shared 21-mers at this genome size, a few tens of thousands shared by three genomes, and almost none shared by five or more. Whether a given $r$ is acceptable then depends on how many k-mers the index actually reports at that level.
|
||||
|
||||
### Limits of the model
|
||||
|
||||
- The FDR is computed per k-mer. Neighbouring k-mers of a shared region overlap and are not independent, so a single homologous region contributes many correlated k-mers. A per-region FDR would count regions instead of k-mers and is not covered here.
|
||||
- The probability $p_i$ assumes equiprobable, independent nucleotides. The composition biases discussed earlier in this page violate this assumption, and the resulting error is not accounted for.
|
||||
|
||||
---
|
||||
|
||||
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
|
||||
|
||||
## 1. FDR d'un ensemble de k-mers sélectionnés
|
||||
|
||||
**Estimation de l'espérance du nombre de faux positifs.** Soit $M$ le nombre de k-mers canoniques testés, $M = 4^k/2$ pour k impair. Soit $q(r)=P(X\ge r)$ la queue du modèle nul. Le nombre attendu de k-mers présents par hasard dans au moins $r$ génomes vaut
|
||||
|
||||
$$
|
||||
E[\mathrm{FP}(r)] = M\,q(r).
|
||||
$$
|
||||
|
||||
Cette égalité est exacte par linéarité de l'espérance, même si les k-mers chevauchants sont dépendants. La dépendance n'affecte que la variance, pas l'espérance. Seule l'approximation de Poisson par génome reste une approximation.
|
||||
|
||||
**FDR.** Soit $N(r)=\#\{i : r_i\ge r\}$ le nombre observé de k-mers couvrant au moins $r$ génomes. Alors
|
||||
|
||||
$$
|
||||
\mathrm{FDR}(r)=\min\!\Big(1,\ \frac{M\,q(r)}{N(r)}\Big).
|
||||
$$
|
||||
|
||||
Cette estimation est légèrement conservatrice. Elle suppose $\pi_0=1$, c'est-à-dire que presque tout l'espace $M$ est nul, ce qui est vrai pour k grand.
|
||||
|
||||
**Ensemble hétérogène en $r_i$.** Si le jeu n'est pas défini par un seuil unique, on utilise le local fdr par classe de couverture :
|
||||
|
||||
$$
|
||||
\mathrm{fdr}(r)=\frac{M\,P(X=r)}{N_r},\qquad N_r=\#\{i : r_i=r\}.
|
||||
$$
|
||||
|
||||
Le FDR de l'ensemble est alors la moyenne $\frac1N\sum_i \mathrm{fdr}(r_i)$. Pour un ensemble défini par seuil $r_{\min}$, cela redonne exactement la formule précédente.
|
||||
|
||||
**Remarque.** $M\,p\approx G$, d'où $E[\mathrm{FP}(r)]\approx\binom{S}{r}\,G^r/M^{r-1}$ en régime de petits $p$. Cela donne une lecture directe de la quantité.
|
||||
|
||||
## 2. Markov d'ordre 0 avec fréquences inégales
|
||||
|
||||
Soit $\pi_b$ la fréquence du nucléotide $b$. Pour un k-mer $w$ de composition $n=(n_A,n_C,n_G,n_T)$ avec $\sum n_b=k$, on a $P(w)=\prod_b \pi_b^{n_b}$.
|
||||
|
||||
**Attendu par génome.** Le k-mer canonique $K=\{w,\overline w\}$ est vu quand le brin direct porte $w$ ou $\overline w$, donc
|
||||
|
||||
$$
|
||||
\lambda_K = G\,\big(P(w)+P(\overline w)\big)=G\Big(\prod_b\pi_b^{n_b}+\pi_A^{n_T}\pi_T^{n_A}\pi_C^{n_G}\pi_G^{n_C}\Big).
|
||||
$$
|
||||
|
||||
Cela ne dépend que de la composition. Si $\pi_A=\pi_T$ et $\pi_C=\pi_G$, on retrouve $2P(w)$.
|
||||
|
||||
**Faux positifs.** Le test n'est plus homogène : chaque composition a sa propre $p(n)=1-e^{-\lambda(n)}$. Il faut donc sommer sur les classes de composition (un k-mer et son complément inverse partagent le même $\lambda$, d'où le facteur ½) :
|
||||
|
||||
$$
|
||||
E[\mathrm{FP}(r)]=\frac12\sum_{n}\binom{k}{n_A,n_C,n_G,n_T}\,P\big(X\ge r\mid p(n)\big).
|
||||
$$
|
||||
|
||||
Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Elle est donc légère avec la binomiale de la moyenne, ou avec une Poisson-binomiale par composition si les $G_i$ diffèrent. Si les $\pi_b$ varient d'un génome à l'autre (GC différent), les $p_i(n)$ se construisent par génome.
|
||||
|
||||
**Effet attendu.** La moyenne de $\lambda_w$ sur tous les k-mers reste $2G/4^k$, car $\sum_w P(w)=1$. Dans le régime de petits $p$, le terme dominant $\binom{S}{r}p^r$ est convexe en $\lambda$. Par Jensen, le modèle équiprobable est donc le minimum de $E[\mathrm{FP}]$, et la déviation de composition ne peut qu'augmenter le FDR. Les k-mers riches en AT ou GC dominent la somme, ce qui rejoint ton remarque sur la faible complexité. C'est un résultat sur le terme dominant, pas une preuve pour la queue exacte.
|
||||
|
||||
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
|
||||
|
||||
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
|
||||
@@ -0,0 +1,82 @@
|
||||
# Minimizer selection
|
||||
|
||||
## Definition
|
||||
|
||||
The **minimizer** of a k-mer is the canonical form of the m-mer that is smallest according to a chosen ordering among the $k-m+1$ overlapping m-mers contained in the k-mer, with $m<k$ [@Roberts2004-rz]. Canonical form is the same operation defined for a full kmer (see [DNA encoding](encoding.md)), applied here to an m-mer $s$ instead: $s^c = \min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)$.
|
||||
|
||||
In **OBIkmer**, canonical m-mers are ordered according to a deterministic hash function $H$ applied to their canonical form:
|
||||
|
||||
$$
|
||||
s_1^c <_h s_2^c
|
||||
\quad\Longleftrightarrow\quad
|
||||
H(s_1^c) < H(s_2^c).
|
||||
$$
|
||||
|
||||
Thus, for a k-mer $K$, let
|
||||
|
||||
$$
|
||||
M(K)=(s_1,\ldots,s_{k-m+1})
|
||||
$$
|
||||
|
||||
be its sequence of overlapping m-mers. The minimizer is
|
||||
|
||||
$$
|
||||
\operatorname{minimizer}(K) = s_j^c,
|
||||
\qquad
|
||||
j=\underset{i}{\operatorname{argmin}}\;H(s_i^c).
|
||||
$$
|
||||
|
||||
In other words, the hash function defines the ordering of canonical m-mers, while the minimizer itself is the canonical form of the m-mer selected by that ordering.
|
||||
|
||||
Applying the minimizer to consecutive k-mers partitions a sequence into super-kmers (see [Super-kmers](superkmers.md)).
|
||||
|
||||
## Hash-based minimizer ordering
|
||||
|
||||
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf].
|
||||
|
||||
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijective mixing function with good avalanche properties [@Steele2014-cz], its output behaves approximately as a composition-independent pseudorandom ordering of distinct m-mers. Under the usual random-hash assumption, each distinct m-mer in a window has approximately the same probability of being the minimum, independently of nucleotide composition.
|
||||
|
||||
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
|
||||
|
||||
### Hash function
|
||||
|
||||
The hash function \(H\) applies Stafford's 64-bit mixing function, variant 13 [@Stafford2011], to the m-mer encoding XORed with a seed $\sigma$:
|
||||
|
||||
|
||||
```text
|
||||
H(x):
|
||||
x ← x ⊕ σ
|
||||
x ← x ⊕ (x >> 30)
|
||||
x ← x × 0xbf58476d1ce4e5b9
|
||||
x ← x ⊕ (x >> 27)
|
||||
x ← x × 0x94d049bb133111eb
|
||||
return x ⊕ (x >> 31)
|
||||
```
|
||||
|
||||
The hash uses a fixed non-zero seed
|
||||
|
||||
$$ \sigma=\texttt{0x9e3779b97f4a7c15}. $$
|
||||
|
||||
The choice of $\sigma$ is not arbitrary. Low-complexity m-mers (homopolymers and short tandem repeats) are disproportionately abundant in real genomes. If one of them happened to be the global argmin of $H$, it could therefore occur as the minimizer in far more windows than expected from the composition-uniform behavior described above. This is not a bias of the mixing function itself, but a consequence of the repeated occurrence of the same input in real sequence data.
|
||||
|
||||
In particular, with $\sigma=0$, the all-A m-mer is encoded as $0$, and
|
||||
|
||||
$$ \operatorname{Mix13}(0)=0, $$
|
||||
|
||||
This makes it the global argmin of the hash function. The non-zero seed eliminates the particular pathological correspondence between the all-A m-mer and the zero output of the mixer.
|
||||
|
||||
Exhaustive enumeration of all $4^m$ m-mers confirms that, with $\sigma=\texttt{0x9e3779b97f4a7c15}$, the global argmin is neither a homopolymer nor a periodic repeat for any tested \(m\):
|
||||
|
||||
| $m$ | argmin (canonical) | decoded sequence | minimal period |
|
||||
|---|---|---|---|
|
||||
| 3 | 16 | `CAA` | 3 |
|
||||
| 5 | 78 | `ACATG` | 5 |
|
||||
| 7 | 5512 | `CCCGAGA` | 7 |
|
||||
| 9 | 108760 | `CGGGATCGA` | 9 |
|
||||
| 11 | 179014 | `AAGGTGTCACG` | 11 |
|
||||
| 13 | 33759044 | `GAAATACTTCACA` | 13 |
|
||||
| 15 | 29869313 | `AACTACTTACCAAAC` | 15 |
|
||||
|
||||
When the minimal period of the sequence is $m$, as observed in the table above, the sequence is primitive: it cannot be represented as repetitions of a shorter sequence.
|
||||
|
||||
See [Partitioning and indexing architecture](indexing_architecture.md) for how this hash routes a super-kmer to a partition.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Super-kmers
|
||||
|
||||
A **super-kmer** is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see [Minimizer selection](minimizer_selection.md)). Each kmer in the run overlaps the next by $k-1$ nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
|
||||
|
||||
For a random minimizer of length $m$ over kmers of length $k$, the expected length of a super-kmer is approximately [@Zheng2020-ji; @Golan2025-xf]:
|
||||
|
||||
$$L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1$$
|
||||
|
||||
For $k=31$, $m=13$ this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.
|
||||
|
||||
## Canonical super-kmers
|
||||
|
||||
A **canonical super-kmer** is the canonical form of a super-kmer (see [DNA encoding](encoding.md)): the lexicographic minimum of the super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.
|
||||
|
||||
Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.
|
||||
@@ -1,24 +0,0 @@
|
||||
# Kmers and super-kmers
|
||||
|
||||
## Kmers
|
||||
|
||||
A **kmer** is a DNA subsequence of fixed length $k$. Two constraints apply to $k$, both enforced when a command starts (an invalid value exits immediately with an error):
|
||||
|
||||
- $k \in [11, 31]$: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base ($k \le 32$ is the hard limit; $k < 11$ gives insufficient specificity).
|
||||
- $k$ **is odd**: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see [DNA encoding](encoding.md)) to be well defined.
|
||||
|
||||
## Super-kmers
|
||||
|
||||
A **super-kmer** is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see [Minimizer selection](minimizer_selection.md)). Each kmer in the run overlaps the next by $k-1$ nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
|
||||
|
||||
For a random minimizer of length $m$ over kmers of length $k$, the expected length of a super-kmer is approximately [@Zheng2020-ji; @Golan2025-xf]:
|
||||
|
||||
$$L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1$$
|
||||
|
||||
For $k=31$, $m=13$ this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.
|
||||
|
||||
### Canonical super-kmers
|
||||
|
||||
A **canonical super-kmer** is the lexicographic minimum of a super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.
|
||||
|
||||
Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.
|
||||
@@ -1,42 +0,0 @@
|
||||
# Minimizer selection
|
||||
|
||||
## Definition
|
||||
|
||||
A **minimizer** of a kmer window is the m-mer ($m < k$) that is smallest, among all $k - m + 1$ overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent.
|
||||
|
||||
The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
|
||||
|
||||
## Hash-based ("random") minimizer
|
||||
|
||||
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions.
|
||||
|
||||
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition.
|
||||
|
||||
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
|
||||
|
||||
### Hash function
|
||||
|
||||
The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed:
|
||||
|
||||
$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
|
||||
|
||||
```
|
||||
H(x):
|
||||
x ← x ⊕ 0x9e3779b97f4a7c15
|
||||
x ← x ⊕ (x >> 30)
|
||||
x ← x × 0xbf58476d1ce4e5b9
|
||||
x ← x ⊕ (x >> 27)
|
||||
x ← x × 0x94d049bb133111eb
|
||||
return x ⊕ (x >> 31)
|
||||
```
|
||||
|
||||
The XOR seed avoids the finalizer's fixed point at 0 ($\text{mix64}(0) = 0$), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison.
|
||||
|
||||
## Partition routing is independent of minimizer selection
|
||||
|
||||
The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately:
|
||||
|
||||
- **Selection** uses $H$ applied to every candidate m-mer in the window, keeping the minimum.
|
||||
- **Partition routing** recomputes $H$ on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.
|
||||
|
||||
See [Partitioning and indexing architecture](indexing_architecture.md) for how the routing value is turned into a partition index.
|
||||
@@ -0,0 +1,140 @@
|
||||
# Whole k-mer set distances
|
||||
|
||||
The k-mer composition of a genome provides a representation of its sequence content that can be compared directly between genomes. The distances considered here quantify dissimilarity between genomes from their complete k-mer composition, independently of any particular representation, storage scheme, or indexing strategy.
|
||||
|
||||
Two complementary classes of comparison arise according to the information retained about k-mer occurrences [@Anderson2011-rq]. **Presence/absence-based distances** consider only whether each distinct k-mer occurs in a genome, whereas **abundance-based distances** also incorporate its observed frequency. Abundance-based measures may be computed either from raw k-mer counts or from the corresponding relative frequencies.
|
||||
|
||||
Let $A$ and $B$ denote the k-mer sets of two genomes. For each distinct k-mer $i$, let $c_i^A$ and $c_i^B$ denote its counts in the two genomes, and let
|
||||
|
||||
$$
|
||||
p_i^A=\frac{c_i^A}{\sum_j c_j^A},
|
||||
\qquad
|
||||
p_i^B=\frac{c_i^B}{\sum_j c_j^B}
|
||||
$$
|
||||
|
||||
denote the corresponding relative frequencies.
|
||||
|
||||
The resulting measures can be classified along two independent dimensions: the **information retained about k-mer occurrences** (presence/absence versus abundance) and the **metric properties** of the resulting dissimilarity, in particular whether the triangle inequality is satisfied.
|
||||
|
||||
## Presence/absence-based distances
|
||||
|
||||
When only occurrence is retained, each genome is represented by its incidence vector over the k-mer space. The comparison therefore depends exclusively on the presence or absence of each distinct k-mer and is insensitive to its multiplicity.
|
||||
|
||||
The **Jaccard distance** compares two k-mer sets from the cardinalities of their intersection and union [@Jaccard1901-mx]:
|
||||
|
||||
$$
|
||||
D = 1 - \frac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}.
|
||||
$$
|
||||
|
||||
Jaccard distance is a metric and therefore satisfies the triangle inequality [@Kosub2019-xt].
|
||||
|
||||
The **Hamming distance** is the $L^1$ distance between the corresponding binary incidence vectors:
|
||||
|
||||
$$
|
||||
D = \sum_i \mathbb{1}[a_i \ne b_i],
|
||||
$$
|
||||
|
||||
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded.
|
||||
|
||||
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
|
||||
|
||||
$$
|
||||
J = 1-D_{\mathrm{Jaccard}},
|
||||
$$
|
||||
|
||||
the Mash distance is defined as
|
||||
|
||||
$$
|
||||
D = -\frac{1}{k}\ln\left(\frac{2J}{1+J}\right)
|
||||
= -\frac{1}{k}\ln\left(\frac{2(1-J_{\mathrm{Jaccard}})}
|
||||
{2-J_{\mathrm{Jaccard}}}\right),
|
||||
$$
|
||||
|
||||
with the result clamped to $1.0$ when the Jaccard similarity reaches zero. This transformation is monotone increasing with Jaccard distance and therefore preserves the ordering of pairwise dissimilarities. Monotonicity alone, however, does not guarantee preservation of the triangle inequality. Mash should consequently be regarded as an evolutionary distance estimator derived from k-mer similarity rather than assumed to be a metric.
|
||||
|
||||
## Abundance-based distances
|
||||
|
||||
When k-mer abundances are retained, genomes are represented by non-negative vectors of k-mer counts or frequencies. Distances can then distinguish genomes that contain the same k-mers but differ in their relative abundances.
|
||||
|
||||
The **Euclidean distance** is the ordinary $L^2$ distance between the corresponding vectors. On raw counts it is
|
||||
|
||||
$$
|
||||
D = \sqrt{\sum_i (c_i^A-c_i^B)^2},
|
||||
$$
|
||||
|
||||
and the same definition applied to relative frequencies gives `relfreq-euclidean`. In both cases, Euclidean distance is a metric.
|
||||
|
||||
The **Hellinger distance** applies the Euclidean distance to the square-root-transformed relative frequencies [@Legendre2001-cq]:
|
||||
|
||||
$$
|
||||
D =
|
||||
\frac{1}{\sqrt{2}}\sqrt{\sum_i\left(\sqrt{p_i^A}-\sqrt{p_i^B}\right)^2}.
|
||||
$$
|
||||
|
||||
It is bounded between 0 and 1 and is a metric because it is a positive rescaling of the Euclidean distance between the transformed vectors $\sqrt{p^A}$ and $\sqrt{p^B}$. Its unnormalized form, `hellinger-euclidean`,
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
D &=\sqrt{\sum_i\left(\sqrt{p_i^A}-\sqrt{p_i^B}\right)^2} \\
|
||||
&=\sqrt{2}\,D_{\mathrm{Hellinger}},
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
is likewise a metric, since multiplication by a positive constant preserves the triangle inequality.
|
||||
|
||||
The **Bray–Curtis dissimilarity** [@Bray1957-vv] is defined from the amount of abundance shared between two vectors:
|
||||
|
||||
$$
|
||||
D = 1-\frac{2\sum_i\min(c_i^A,c_i^B)}{\sum_i c_i^A+\sum_i c_i^B}.
|
||||
$$
|
||||
|
||||
Using
|
||||
|
||||
$$
|
||||
a+b-2\min(a,b)=|a-b|,
|
||||
$$
|
||||
|
||||
this can equivalently be written as
|
||||
|
||||
$$
|
||||
D =
|
||||
\frac{\sum_i|c_i^A-c_i^B|}
|
||||
{\sum_i c_i^A+\sum_i c_i^B}.
|
||||
$$
|
||||
|
||||
For arbitrary count vectors, **Bray–Curtis** is not a metric [@Gower1986-ri]: the pair-dependent normalization by the combined abundance can lead to violations of the triangle inequality. This distinguishes it from the $L^1$-based distances above, whose normalization factor is fixed independently of the pair being compared.
|
||||
|
||||
The metric property is recovered when all vectors under comparison have the same total abundance $S$. In that case,
|
||||
|
||||
$$
|
||||
\sum_i c_i^A=\sum_i c_i^B=S
|
||||
$$
|
||||
|
||||
for every pair, and **Bray–Curtis** reduces to
|
||||
|
||||
$$
|
||||
D =
|
||||
\frac{1}{2S}
|
||||
\sum_i|c_i^A-c_i^B|.
|
||||
$$
|
||||
|
||||
It is therefore a positive rescaling of the $L^1$ distance and is a metric on this fixed-total space.
|
||||
|
||||
Relative frequencies satisfy this condition by construction, since
|
||||
|
||||
$$
|
||||
\sum_i p_i^A=\sum_i p_i^B=1.
|
||||
$$
|
||||
|
||||
Consequently, `relfreq-bray-curtis` reduces to
|
||||
|
||||
$$
|
||||
D =
|
||||
\frac12\sum_i|p_i^A-p_i^B|,
|
||||
$$
|
||||
|
||||
which is the **total variation distance** between the two frequency distributions. It is therefore a metric.
|
||||
|
||||
Thus, `bray-curtis` and `relfreq-bray-curtis` have the same algebraic form but operate on different domains: the former compares count vectors whose total abundance may vary between genomes, whereas the latter compares normalized frequency vectors with a fixed total. This difference in normalization is sufficient to change the metric properties of the measure.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md) for how to select a distance and the resulting output formats.
|
||||
@@ -0,0 +1,85 @@
|
||||
# Central-position SNP model
|
||||
|
||||
## Family
|
||||
|
||||
A **family** is the set of up to 4 kmers sharing identical flanking sequence and differing only at the central base — the odd kmer length guarantees a single, well-defined central position (see [Kmers](../../kmer_indexing/kmers.md)). A family is **eligible** for a genome pair $(i,j)$ only if both genomes carry exactly one of its observed forms (single-copy, unambiguous) — this excludes multi-copy and absent loci from the comparison rather than folding them into an undifferentiated "not identical" bucket the way a whole-index distance would.
|
||||
|
||||
Conditioning on eligibility this way restricts every comparison to loci that are directly, positively confirmed comparable in both genomes: a locus never enters the statistic because of a genuinely absent homologous region, a diverged paralogous copy, or a genome-size asymmetry — only because a real single-copy substitution (or lack of one) was observed at flanks confirmed intact in both genomes.
|
||||
|
||||
## Substitution corrections (`snp-*`)
|
||||
|
||||
For a genome pair, let $L$ be its total number of eligible loci, $p$ the raw proportion of substitutions among those loci, $P$/$Q$ the transition/transversion proportions, $Q_1$/$Q_2$ Kimura's two transversion categories (A↔C & G↔T vs. A↔T & C↔G), $P_1$/$P_2$ the purine (A↔G) / pyrimidine (C↔T) transition proportions, and $\pi_A,\pi_C,\pi_G,\pi_T$ the pair's pooled base frequencies.
|
||||
|
||||
**`snp-raw`**
|
||||
|
||||
$$d = p$$
|
||||
|
||||
**`snp-jc`**
|
||||
|
||||
$$d = -\frac{3}{4}\ln\!\left(1-\frac{4p}{3}\right)$$
|
||||
|
||||
**`snp-k2p`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
a_1 &= 1-2P-Q \\
|
||||
a_2 &= 1-2Q \\
|
||||
d &= -\frac{1}{2}\ln a_1-\frac{1}{4}\ln a_2
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-k81`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
a_1 &= 1-2P-2Q_1 \\
|
||||
a_2 &= 1-2P-2Q_2 \\
|
||||
a_3 &= 1-2Q_1-2Q_2 \\
|
||||
d &= -\frac{1}{4}\left(\ln a_1+\ln a_2+\ln a_3\right)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-f81`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
E &= 1-\left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right) \\
|
||||
d &= -E\ln\!\left(1-\frac{p}{E}\right)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-t92`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
g &= \pi_C+\pi_G \\
|
||||
w &= 2g(1-g) \\
|
||||
a_1 &= 1-\frac{P}{w}-Q \\
|
||||
a_2 &= 1-2Q \\
|
||||
d &= -w\ln a_1-\frac{1}{2}(1-w)\ln a_2
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-tn93`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
g_R &= \pi_A+\pi_G \\
|
||||
g_Y &= \pi_C+\pi_T \\
|
||||
k_1 &= \frac{2\pi_A\pi_G}{g_R} \\
|
||||
k_2 &= \frac{2\pi_C\pi_T}{g_Y} \\
|
||||
k_3 &= 2\left(g_Rg_Y-\frac{\pi_A\pi_G\,g_Y}{g_R}-\frac{\pi_C\pi_T\,g_R}{g_Y}\right) \\
|
||||
w_1 &= 1-\frac{P_1}{k_1}-\frac{Q}{2g_R} \\
|
||||
w_2 &= 1-\frac{P_2}{k_2}-\frac{Q}{2g_Y} \\
|
||||
w_3 &= 1-\frac{Q}{2g_Rg_Y} \\
|
||||
d &= -k_1\ln w_1-k_2\ln w_2-k_3\ln w_3
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-tv`** — transversions only, deliberately uncorrected:
|
||||
|
||||
$$d = Q$$
|
||||
|
||||
A rate-heterogeneity correction applies to every formula above except `snp-raw` and `snp-tv`: each $-\ln(x)$ term is replaced by $\alpha\left(x^{-1/\alpha}-1\right)$ (same weight, same $x$) — see [Gamma rate heterogeneity](gamma_rate_heterogeneity.md) for the correction itself and how $\alpha$ is estimated.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md) for how to select a `snp-*` value and the prerequisite index annex it requires.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Gamma rate heterogeneity
|
||||
|
||||
Real substitution rates vary across sites rather than being uniform, which a plain substitution correction (see [Central-position SNP model](central_snp_model.md)) ignores. The standard $+\Gamma$ correction models this by replacing each $-\ln(x)$ term in a correction formula with $\alpha\left(x^{-1/\alpha}-1\right)$ — the same $x$, the same weight, under the assumption that the per-site rate is $\mathrm{Gamma}(\alpha,\alpha)$-distributed (mean 1) rather than fixed. A small $\alpha$ indicates strong among-site rate heterogeneity (many near-invariant sites, a few fast ones); as $\alpha$ grows large the correction converges to the uncorrected formula.
|
||||
|
||||
## Automatic $\alpha$ estimation
|
||||
|
||||
$\alpha$ can be estimated automatically from the index itself, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.
|
||||
|
||||
The estimator pools substitution counts by **partition** rather than by genome pair: for partition $i$, let $n_i$ be the total number of substitutions observed across every genome pair, and $L_i$ the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:
|
||||
|
||||
$$R_i = \frac{n_i}{L_i}$$
|
||||
|
||||
Under a single shared substitution rate with no among-site heterogeneity, each $R_i$ would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as in the correction itself, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:
|
||||
|
||||
$$\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}$$
|
||||
|
||||
where $\mu$ is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own $L_i$ removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:
|
||||
|
||||
$$\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}}$$
|
||||
|
||||
$$\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}$$
|
||||
|
||||
If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), $\alpha$ is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged.
|
||||
|
||||
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Sampling design
|
||||
|
||||
Exhaustive computation over every non-monomorphic family in an index (see [Central-position SNP model](central_snp_model.md)) is the default, but can be bounded to approximately `N` families instead. Sampling is designed around two properties: staying representative of the whole index, and letting the user favor families that actually carry signal.
|
||||
|
||||
## Proportional sampling
|
||||
|
||||
Families are drawn in proportion to how many candidate families each part of the index actually holds, rather than uniformly across index partitions — this keeps the sample's composition representative of the whole index regardless of how unevenly candidate families happen to be distributed across partitions.
|
||||
|
||||
## Per-family informativeness: Shannon entropy
|
||||
|
||||
A family's informativeness is measured directly rather than assumed uniform. For a family, entropy is computed over its observed states across genomes:
|
||||
|
||||
| Quantity | Definition |
|
||||
|---|---|
|
||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||
| `entropy4` | the same over the 4 plain bases, restricted to the genomes carrying exactly one member of the family (those that stay informative when multi-member genomes are treated as missing data) |
|
||||
|
||||
With `--no-ambiguity` the entropy is `entropy4`, computed over the genomes carrying exactly one member of the family, so that the weighting sees the same genomes as the final alignment; without it, `entropy15` is used. Each is stored in its own annex.
|
||||
|
||||
A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).
|
||||
|
||||
## Quartet restriction
|
||||
|
||||
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts called states only, so absence never forms one of the two pairs.
|
||||
|
||||
## Pair balance
|
||||
|
||||
How well a pair of genomes is represented in a sample depends on how many families the pair shares: pairs of distant genomes share few, so a uniform draw gives them few sites, and the deep branches of a tree — which rest on exactly those pairs — are the least supported. `--pair-balance` weights each family by the mean, over the pairs of genomes carrying it, of 1 / (families shared by the pair), computed on the whole index. Averaging rather than summing keeps the weight from growing with the number of carriers, which would favour the families carried by most genomes regardless of which pairs they inform. Layers are allotted their share of the draw in proportion to the total weight of their families.
|
||||
|
||||
## Entropy-biased sampling
|
||||
|
||||
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
||||
@@ -0,0 +1,22 @@
|
||||
# The 16-state Sankoff model
|
||||
|
||||
Each family (see [Central-position SNP model](central_snp_model.md)) is treated as a character with 16 possible states: one per subset of the 4 possible central bases, including the empty subset (no observed form). A genome pair's calibration reduces to two separately-estimated first-order Markov models, both restricted to genome pairs below a configurable SNP-ratio ceiling (excluding saturated pairs from skewing the calibration):
|
||||
|
||||
- **Cardinality transitions** ($5 \times 5$, row-stochastic): the probability that a family observed with $i$ forms in one genome ($i \in \{0,\dots,4\}$) is observed with $j$ forms in the other.
|
||||
- **Composition transitions** ($4 \times 4$, row-stochastic): the probability that an unambiguous, single-copy base observed as one of A/C/G/T in one genome is observed as another base in the other genome.
|
||||
|
||||
## Combining into a 16×16 cost matrix
|
||||
|
||||
The two matrices are *not* combined as a plain tensor product. For a pair of states $(A, B)$ — each a subset of $\{A,C,G,T\}$ — let `shared` $= A \cap B$, `lost` $= A \setminus B$, `gained` $= B \setminus A$. The log-probability of the transition $A \to B$ is:
|
||||
|
||||
$$\log P(A,B) = \underbrace{\log P_{\text{card}}(|A|,|B|)}_{\text{cardinality term, omitted when loss events are treated as free}} \;+\; \underbrace{\sum_{i \,\in\, \text{shared}} \log P_{\text{comp}}(i,i)}_{\text{bases conserved on both sides}} \;-\; \underbrace{\text{best\_pairing\_cost}(\text{lost}, \text{gained})}_{\text{substitutions among the differing bases}}$$
|
||||
|
||||
`best_pairing_cost` enumerates every injection pairing elements of `lost` with elements of `gained` and keeps the one minimizing $-\sum \log P_{\text{comp}}$ over the pairs — the discrete analogue of "prefer a substitution over an independent loss and gain": if a base is lost from one side and a different base is gained on the other, that is scored as a single substitution between them (cheaper, under any sane calibration, than treating them as two unrelated cardinality-changing events) whenever such a pairing is possible.
|
||||
|
||||
The resulting $16 \times 16$ log-probability matrix is row-normalized in log-space (log-sum-exp, not a direct `exp()`/divide, for numerical stability), giving a genuine row-stochastic transition matrix $P$. The cost matrix is then $\text{cost}(A,B) = -\ln P(A,B)$, symmetrized by simple averaging:
|
||||
|
||||
$$\text{sym}(A,B) = \frac{\text{cost}(A,B) + \text{cost}(B,A)}{2}$$
|
||||
|
||||
This calibrated cost matrix is what every downstream export (for TNT, PhyG, IQ-TREE) reuses, recoded into each tool's own format.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md#sankoff-calibration-and-phylogenetic-exports) for how to run the calibration and produce these exports.
|
||||
@@ -0,0 +1,29 @@
|
||||
# convert
|
||||
|
||||
Convert an existing index's evidence representation in place, between exact, approximate, and hybrid.
|
||||
|
||||
```bash
|
||||
obikmer convert INDEX (--exact-evidence | --approx-evidence BITS | --hybrid-evidence) [OPTIONS]
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `INDEX` | Index directory to convert (modified in place) |
|
||||
|
||||
## Options
|
||||
|
||||
Exactly one of the first three is required:
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--exact-evidence` | Convert to exact evidence (zero false positives) |
|
||||
| `--approx-evidence BITS` | Convert to approximate (fingerprint-only) evidence; `BITS` = fingerprint bits per slot (b) |
|
||||
| `--hybrid-evidence` | Convert to hybrid evidence (both exact and approximate bundles kept) |
|
||||
| `--evidence-bits BITS` | Fingerprint bits per slot (b) — required with `--hybrid-evidence` when the source index is currently exact; rejected otherwise (the source already fixes `b`) |
|
||||
| `-z, --findere-z Z` | Findere z parameter: number of consecutive stored kmers that must all match to confirm a hit. This does not shorten the indexed kmer length (fixed forever at `index` build time) — it extends the effective match window: on a k=31 index, `z=2` requires 32 consecutive matching bases, not 30 |
|
||||
| `--fp FP` | Target false-positive rate per z-window (e.g. `0.01`); derives `b` or `z` when one of them isn't given directly |
|
||||
| `--block-size N` | Block size for exact evidence's on-disk index (unitigs per block). Ignored when converting to pure approximate evidence. Default `1` |
|
||||
|
||||
See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.
|
||||
@@ -15,4 +15,4 @@ obikmer estimate [OPTIONS]
|
||||
| `--evidence-bits` | none | Fingerprint bits per slot (b) |
|
||||
| `--fp` | none | Target false-positive rate per z-window |
|
||||
|
||||
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `reindex --approx` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
|
||||
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
|
||||
@@ -45,6 +45,6 @@ With `--approx`, evidence is stored as a compact **fingerprint** instead, tradin
|
||||
|
||||
$$FP = \frac{1}{2^{b \cdot z}}$$
|
||||
|
||||
where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`reindex`](reindex.md) to convert an existing index between the two representations.
|
||||
where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`convert`](convert.md) to change an existing index's representation afterwards.
|
||||
|
||||
`z` must be strictly less than k: the effective indexed kmer length under approximate evidence is k−z+1.
|
||||
@@ -1,21 +0,0 @@
|
||||
# name-tree
|
||||
|
||||
Translate a numerically-labelled tree export (TNT, PhyG, or any plain Newick file with bare `1`, `2`, `3`, … leaf labels) back to real taxon names, reading the label order from the FASTA that produced it.
|
||||
|
||||
```bash
|
||||
obikmer name-tree TREE --fasta FASTA -o OUTPUT
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `TREE` | Tree file to translate — a TNT-style NEXUS export (`tree NAME = [&U] ...;`) or a plain Newick file |
|
||||
| `--fasta` | FASTA file whose record order gives the numeric taxon labels (1-based) — typically the `_sankoff.fasta`/`_snp.fasta` used to produce `TREE` |
|
||||
| `-o, --output` | Output NEXUS file path |
|
||||
|
||||
## Output
|
||||
|
||||
A NEXUS file with a `taxa` block, a `translate` table (numeric label → taxon name, from `--fasta`'s header order), and every tree found in `TREE`, topology unchanged — readable directly in FigTree, PearTree, `ape` (R), etc.
|
||||
|
||||
`--tnt`'s and `--phyg`'s exports (see [phylo](phylo.md)) both number taxa `1..N` in the same order as the pseudo-alignment FASTA they were built from (`<prefix>_sankoff.fasta`), so pass that same file as `--fasta` here.
|
||||
@@ -24,6 +24,6 @@ The index directory is locked for exclusive access while packing.
|
||||
|
||||
Matrix data (which genomes carry each kmer, or with what count) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one entry per genome for every kmer regardless of how many genomes actually carry it; `--sparse` instead stores each kmer's genome list directly. For presence/absence matrices, identical genome lists shared by many kmers are also deduplicated (common in real data, since kmers from the same conserved region tend to be carried by the same genomes); for count matrices, the genome list is deduplicated the same way but each kmer's actual counts are kept per-kmer, since two kmers sharing the same genome list rarely carry the same counts.
|
||||
|
||||
On real genome collections this has measured at roughly 7x smaller on disk than the dense format for presence/absence, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--metric` distance-matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s plain `--metric` distance matrices.
|
||||
On real genome collections this has measured at roughly 7x smaller on disk than the dense format for presence/absence, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--distance` matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s `--distance` matrices.
|
||||
|
||||
`--sparse` applies to both presence/absence and count matrices — a count index (`--metric` distance matrices included) is packed sparse the same as a presence index.
|
||||
`--sparse` applies to both presence/absence and count matrices — a count index (`--distance` matrix computations included) is packed sparse the same as a presence index.
|
||||
+135
-134
@@ -1,6 +1,6 @@
|
||||
# phylo
|
||||
|
||||
Compute pairwise evolutionary-distance proxies between the genomes stored in an index — a plain distance matrix, optionally trees (NJ/UPGMA), and optionally a central-position SNP model with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
|
||||
Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
|
||||
|
||||
```bash
|
||||
obikmer phylo INDEX [OPTIONS]
|
||||
@@ -12,133 +12,158 @@ obikmer phylo INDEX [OPTIONS]
|
||||
|---|---|
|
||||
| `INDEX` | Index directory |
|
||||
|
||||
## Distance matrix
|
||||
## Distance matrix (`--distance`)
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--metric` | `jaccard` | One of `jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` |
|
||||
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for Jaccard/Mash on a count index |
|
||||
| `--shared-kmers` | off | Also write the shared-kmer count matrix |
|
||||
| `--distance` | `jaccard` | See the two tables below for the full list of accepted values |
|
||||
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
|
||||
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
|
||||
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
|
||||
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index distance, not a `snp-*` value |
|
||||
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
|
||||
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
|
||||
| `-o, --output` | none (stdout) | Output file prefix; without it, the distance matrix is printed to stdout as CSV |
|
||||
| `-o, --output` | none (stdout) | Output file prefix |
|
||||
|
||||
`hamming` requires a presence/absence index. All other metrics work on either index type; on a presence index, `jaccard`/`mash`/`hamming` are the only ones available.
|
||||
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
||||
|
||||
### Metric definitions
|
||||
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||
|
||||
- **jaccard**: $D = 1 - \dfrac{|A \cap B|}{|A \cup B|}$ over the sets of kmers present in each genome.
|
||||
- **mash**: derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$.
|
||||
- **hamming**: number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$.
|
||||
- **bray-curtis**: $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts.
|
||||
- **relfreq-bray-curtis**: the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts.
|
||||
- **euclidean**: $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts.
|
||||
- **relfreq-euclidean**: the same formula on relative frequencies.
|
||||
- **hellinger**: $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$.
|
||||
- **hellinger-euclidean**: the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$.
|
||||
### Output
|
||||
|
||||
Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
|
||||
|
||||
## `--exclude-genome`, `--min-shared-family`
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--exclude-genome LABEL` | Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by `--pseudo-alignment`/`--sankoff`/a `snp-*` `--distance` value. Does not change the value computed for any remaining pair |
|
||||
| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` — never to the whole-index distances or their matrix/NJ/UPGMA output |
|
||||
|
||||
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
|
||||
|
||||
## Central-position SNP model
|
||||
|
||||
This is a separate operation from the distance-matrix computation above: if any option below is used, no `--metric` matrix is computed in the same invocation.
|
||||
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index:
|
||||
|
||||
A **family** is the set of up to 4 kmers that share identical flanking sequence and differ only at the exact central base. Because $k$ is odd, the central position is well defined and maps to itself under reverse complementation. All computations below first require building the **sibling annex**, an index-wide record of which of the 4 possible central bases are observed at each family, across every genome.
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value |
|
||||
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
|
||||
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
||||
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
||||
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
||||
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--sibling-annex` | off | Build the sibling presence-mask annex (prerequisite for every option below) |
|
||||
| `--exclude-genome LABEL` | none | Exclude a genome (repeatable) from every SNP/Sankoff/export computation below |
|
||||
| `--min-shared-family N` | none | Auto-exclude any genome whose mean shared-family count against every other genome (see `--family-overlap`) falls below `N` — same exclusion as `--exclude-genome`, applied on top of it |
|
||||
| `--sibling-stats` | off | Write the family-size (sibling count) distribution, per genome and globally |
|
||||
| `--raw-snp-distance` | off | Write the single-copy central-SNP p-distance matrix |
|
||||
| `--raw-snp-counts` | off | Write per-pair diagnostic counts (n_snp, n_shared, n_eligible) instead of a matrix |
|
||||
| `--snp` | off | Write a SNP-only pseudo-alignment in FASTA, IUPAC-coded |
|
||||
| `--family-overlap` | off | Write an NxN matrix of, for each genome pair, how many variable families both genomes actually carry a call for; the diagonal holds each genome's own total family count |
|
||||
### `--sibling-stats`
|
||||
|
||||
### Locus eligibility
|
||||
|
||||
A family is eligible for a genome pair $(i, j)$ only if genome $i$ carries exactly one of the family's observed forms (single-copy, unambiguous) and genome $j$ also carries exactly one. A genome carrying more than one form at a locus makes that locus ineligible for any pair involving it.
|
||||
|
||||
`--raw-snp-distance` tallies, over every eligible locus of every genome pair, $n_{\text{snp}}$ (the two genomes' single forms differ) versus $n_{\text{shared}}$ (they agree — this includes invariant families). The output ratio is $\hat{p} = \dfrac{n_{\text{snp}}}{n_{\text{snp}} + n_{\text{shared}}}$.
|
||||
|
||||
`--snp` restricts itself to *variable* families (family size $\ge 2$) and writes one FASTA record per genome, one column per family, IUPAC-coded from each genome's presence mask at that family (a single form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`).
|
||||
|
||||
`--exclude-genome` removes a genome from these computations, re-checking column variability among the remaining genomes so that a column made monomorphic by the exclusion is dropped rather than kept artificially. It does not affect the `--metric` distance-matrix path.
|
||||
|
||||
### Family overlap and low-coverage genomes
|
||||
|
||||
`--family-overlap` writes, for every genome pair, how many variable families both genomes actually carry a call for (neither is absent) — a direct measure of how much informative content two genomes actually share. On genome-skim or otherwise incomplete-coverage collections, a genome with very little overlap with everything else has almost nothing left to constrain its position in a tree, and tends to end up placed unstably (near-zero branch length, grafted inside an unrelated clade) by `--tnt`/`--iqtree`.
|
||||
|
||||
`--min-shared-family N` automates the fix: it excludes, before any computation, every genome whose mean shared-family count against all other genomes (the same statistic, averaged per row of the `--family-overlap` matrix) falls below `N`. There is no universal value for `N` — it depends on how divergent and how completely covered the genome collection is; inspect `--family-overlap`'s own output to find where the real gap sits before choosing a threshold.
|
||||
|
||||
## Sampling at scale: `--subsample`, `--shannon`, `--entropy`
|
||||
|
||||
On a large index (billions of families), building a full pseudo-alignment or fully calibrating the Sankoff model is not just slow — it may not fit in the time you have. `--subsample` bounds the work to a fixed number of families; `--shannon` reports how informative each family is; `--entropy`/`--entropy-sd` bias which families get kept toward the informative ones instead of choosing uniformly at random.
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--subsample N` | none (keep everything) | Cap the number of variable families (family size ≥ 2) retained, to approximately `N` |
|
||||
| `--shannon` | off | Write `<prefix>_shannon.csv`: per-family Shannon entropy, one row per family |
|
||||
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
||||
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
||||
|
||||
`--subsample`/`--entropy`/`--entropy-sd` affect every option that scans variable families: `--snp`, `--family-overlap`, `--shannon`, and the whole Sankoff pipeline (`--sankoff`/`--tnt`/`--phyg`/`--iqtree`, next section) — all of them draw from the *same* selection of families in one invocation, so the Sankoff calibration and the pseudo-alignment it calibrates always describe the same sites, and `--family-overlap`'s counts stay consistent with `--snp`'s columns. `--raw-snp-distance`/`--raw-snp-counts` are not affected — they always scan every family, since their p-distance estimate is a whole-index statistic, not something that benefits from being restricted to a sample.
|
||||
|
||||
### `--subsample N`
|
||||
|
||||
Without `--subsample`, every variable family (family size ≥ 2, i.e. every family where at least one genome differs from the rest) is used. With `--subsample N`, roughly `N` families are kept instead, chosen at random but in proportion to how many candidate families each part of the index actually holds — so the sample stays representative of the whole index, not skewed toward whichever part happens to be scanned first. If the index has fewer than `N` candidate families in the first place, `--subsample` has no effect: everything is kept.
|
||||
|
||||
`--subsample` trades completeness for speed: `--snp`'s alignment gets fewer columns, `--sankoff`'s calibration is based on fewer observations, but the resolution work (the expensive part of a `phylo` run on a large index) scales with `N` instead of with the index's true size. Pick `N` as large as your time budget allows — a few hundred thousand to a few million families is usually enough for the transition-probability estimates in `--sankoff`'s calibration to stabilize; a smaller `N` speeds up exploratory runs.
|
||||
|
||||
### `--shannon`: measuring how informative a family is
|
||||
|
||||
Not every variable family is equally useful for a tree: a family that differs in only one genome out of a thousand carries very little signal, and one where the pattern looks essentially random across genomes may be too saturated (multiple substitutions have overwritten the original signal) to carry real information either. `--shannon` quantifies this with the Shannon entropy (in bits) of each family's states across the genomes that carry it — low entropy means "almost everyone agrees" (an invariant or near-invariant family, phylogenetically shallow), while entropy near the ceiling for a 4-state character means "close to a random draw between the possible bases" (saturated).
|
||||
|
||||
`<prefix>_shannon.csv` (or `shannon.csv` without `-o`) has one row per family visited:
|
||||
`<prefix>_siblings.csv` — family size = number of distinct central bases observed at a family (1-4).
|
||||
|
||||
| Column | Meaning |
|
||||
|---|---|
|
||||
| `genome` | genome label, or the literal `global` for the last row |
|
||||
| `1`, `2`, `3`, `4` | for a genome row: number of families of that size where the genome carries ≥ 1 member. For the `global` row: the actual deduplicated family-size histogram — not the sum of the rows above |
|
||||
|
||||
### Family Overlap
|
||||
|
||||
`--family-overlap` writes `<prefix>_family_overlap.csv`: header `genome,<label1>,<label2>,...`, one row per genome, cell `[i][j]` = number of variable families (family size ≥ 2) where both genome `i` and genome `j` carry a call. The diagonal is always `0`. Every genome is written, unfiltered by `--exclude-genome`/`--min-shared-family`. With `--no-ambiguity`, a genome carrying several members of a family is not counted as carrying it, as in the alignment; without it, any genome with a call counts. The matrix is stored once per convention, so switching `--no-ambiguity` builds the other one without discarding the first (`--min-shared-family` follows the same convention).
|
||||
|
||||
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
|
||||
|
||||
### `--pseudo-alignment`
|
||||
|
||||
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
|
||||
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
||||
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
||||
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct called states, each carried by at least two genomes. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
||||
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
||||
|
||||
## Sampling at scale: `--n-sites`, `--shannon`, `--entropy`
|
||||
|
||||
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
|
||||
|
||||
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
|
||||
|
||||
Every invocation, with or without `--session`, draws its own fresh random sample by default — running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately. `--session` does not change this: it makes a *specific* sample reusable on request, it does not make sampling itself reproducible from one independent run to the next.
|
||||
|
||||
### `--session`: reusing a sample across separate commands
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
|
||||
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
|
||||
|
||||
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
|
||||
|
||||
A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
|
||||
|
||||
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
||||
|
||||
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
|
||||
|
||||
### `--shannon`: measuring how informative a family is
|
||||
|
||||
`<prefix>_entropy.csv` has one row per family visited:
|
||||
|
||||
| Column | Meaning |
|
||||
|---|---|
|
||||
| `partition` | the index partition that holds the family |
|
||||
| `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
|
||||
| `family_idx` | the family's position within that layer |
|
||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}` — the same alphabet `--sankoff`'s 16-state model uses), genomes absent from the family excluded from the count |
|
||||
| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison — a genome carrying more than one base at once counts once per base, so this can differ from `entropy15` |
|
||||
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2–4, since monomorphic families aren't visited) |
|
||||
| `n_genomes_present` | how many genomes the entropy was computed over |
|
||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||
| `entropy4` | Shannon entropy (bits) over the 4 plain bases, computed only over the genomes that carry exactly one member of the family — the genomes that remain informative under `--no-ambiguity`, since a genome carrying several members of a family (paralogous copies) is treated as missing data there. `0` when `n_genomes_unambiguous` is `0` |
|
||||
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
|
||||
| `n_genomes_present` | how many genomes the family is present in; `entropy15` is computed over these |
|
||||
| `n_genomes_unambiguous` | how many of those genomes carry exactly one member of the family; `entropy4` is computed over these |
|
||||
|
||||
Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful to inspect the entropy distribution and decide reasonable `--entropy`/`--entropy-sd` values (see below) before committing to a full run.
|
||||
`--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
||||
|
||||
### `--quartet`: keeping only families that can resolve a tree
|
||||
|
||||
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. Absence is never a state here: a genome that does not carry the family does not count, with or without `--free-loss`. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
||||
|
||||
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` convention, in one pass shared with the other annexes, and built for both conventions by `--sibling-annex`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
||||
|
||||
### `--pair-balance`: representing every pair of genomes
|
||||
|
||||
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
|
||||
|
||||
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` convention, after the family-overlap matrix it depends on, and by `--sibling-annex` for both conventions). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
|
||||
|
||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||
|
||||
By default, `--subsample` draws families uniformly — every candidate family has the same chance of being kept, regardless of how informative it actually is. `--entropy`/`--entropy-sd` change that: instead of a uniform draw, each family's chance of being kept is weighted by how close its own entropy (the `entropy15` value `--shannon` reports) is to `MU`, using a bell-shaped (Gaussian) curve of width `SIGMA` — a family with entropy exactly `MU` is the most likely to be kept, and the chance falls off smoothly the further its entropy is from `MU`, with no hard cutoff (a few families outside the target band can still get in, just less often).
|
||||
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||
|
||||
The filter activates as soon as *either* `--entropy` or `--entropy-sd` is given; whichever one you don't set defaults to `1.0`/`0.5`. It can be combined with `--subsample N` (the target count is still approximately `N`, now biased toward the entropy band instead of uniform — expect somewhat *fewer* than `N` families in practice, since low-weight families are dropped rather than replaced) or used alone (`--entropy` without `--subsample`: a soft entropy filter over the whole index, no size target).
|
||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
||||
|
||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy has to be computed once, up front, and is then saved alongside the index). Every following run — even with different `MU`/`SIGMA` values — reuses that saved data and stays fast.
|
||||
The entropy that weights the draw follows `--no-ambiguity`, so that it describes the same genomes as the alignment. Without the flag, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With the flag, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). The two entropies are saved separately: switching `--no-ambiguity` builds the other one once, without discarding the first, and `--sibling-annex` builds both.
|
||||
|
||||
## Sankoff calibration and phylogenetic exports
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment |
|
||||
| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the calibration |
|
||||
| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, in `--sankoff`'s pseudo-alignment and every export built from it |
|
||||
| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment. Requires `--n-sites N\|all` |
|
||||
| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the base-composition part of the calibration |
|
||||
| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, throughout `--sankoff` and every export built from it |
|
||||
| `--tnt` | off | Also write a TNT script (implies `--sankoff`) |
|
||||
| `--phyg` | off | Also write PhyG input files (implies `--sankoff`) |
|
||||
| `--iqtree` | off | Also write an IQ-TREE custom model and alignment (implies `--sankoff`) |
|
||||
| `--iqtree-min-freq` | `0.001` | With `--iqtree --free-loss`: also treat as missing any state rarer than this in the alignment |
|
||||
| `--sankoff-cost-scale` | `100` | Integer scaling factor applied to costs before rounding (required by TNT/PhyG's integer-only cost commands) |
|
||||
| `--sankoff-cost-scale` | `100` | Integer scaling factor applied to costs before rounding, for TNT/PhyG's integer-only cost commands |
|
||||
|
||||
### The 16-state model
|
||||
|
||||
Each family is treated as a character with 16 possible states: one per subset of the 4 possible central bases actually observed (including the empty subset). Calibration combines two tallies, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`:
|
||||
Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). See [The 16-state Sankoff model](../theory/phylogeny/snp_based/sankoff_model.md) for how the calibrated cost matrix is built from cardinality and base-composition transitions.
|
||||
|
||||
- a $5 \times 5$ transition matrix over family cardinality (0–4 observed forms) between paired genomes, and
|
||||
- a $4 \times 4$ base-substitution transition matrix from unambiguous single-copy loci,
|
||||
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character). It does not run any external tool.
|
||||
|
||||
which are combined into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
|
||||
|
||||
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character, to avoid ambiguity with external tools' own gap semantics). It does not run any external tool.
|
||||
|
||||
With `--free-loss`, the empty state is recoded to `?` instead — TNT/PhyG/IQ-TREE's own missing-data symbol — rather than an ordinary, costed 16th state. This matters for genome-skim or otherwise incomplete-coverage collections, where non-detection of a family is dominated by sampling failure rather than true evolutionary loss: scoring it as a real state risks grouping genomes by shared undersampling instead of shared ancestry. `?` rather than `-` because `-` still carries gap/indel semantics in these tools, and a non-detected family is not an observed deletion. `--free-loss` also drops the cardinality-transition cost between any two states, not just to/from the empty one: whether a genome shows 1 vs. 2 (etc.) detected members of a family it does carry is exactly as vulnerable to sampling failure as whether the family was detected at all, so gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family. Combine with `--min-shared-family`/`--family-overlap` above: `--free-loss` removes the false signal from non-detection, but a genome left with too little real overlap with everything else will still be placed unstably — excluding it is the other half of the fix.
|
||||
With `--free-loss`, the empty state is recoded to `?` (TNT/PhyG/IQ-TREE's own missing-data symbol) instead of an ordinary, costed 16th state — `?` rather than `-`, since `-` still carries gap/indel semantics in these tools. `--free-loss` also zeroes the cardinality-transition cost between any two states, not just to/from the empty one: gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family.
|
||||
|
||||
### Exports
|
||||
|
||||
@@ -146,58 +171,34 @@ All three exports reuse the `--sankoff` calibrated matrix and pseudo-alignment,
|
||||
|
||||
- **`--tnt`**: a self-contained TNT script (alignment recoded to TNT's fixed 16-symbol alphabet, integer-scaled cost matrix re-closed to a metric, a default search block).
|
||||
- **`--phyg`**: a custom cost-matrix file plus a PhyG script reusing the `--sankoff` alignment directly.
|
||||
- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths (unlike the parsimony step-counts from TNT/PhyG). Only states actually occurring in the alignment are kept and compactly renumbered.
|
||||
- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths. Only states actually occurring in the alignment are kept and compactly renumbered.
|
||||
|
||||
TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, …, in the same order as `<prefix>_sankoff.fasta`). Use [`name-tree`](name-tree.md) on the tool's own tree output plus that same FASTA to get a NEXUS file with real taxon names.
|
||||
TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in the order the genomes appear in `<prefix>_sankoff.fasta`).
|
||||
|
||||
## Output files
|
||||
|
||||
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the plain `--metric` distance matrix is produced, on stdout. All matrices use genome labels (from the index metadata) as row/column headers, in index order; all CSVs are comma-separated with a header row.
|
||||
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
|
||||
|
||||
### Distance matrix
|
||||
|
||||
| File | Written by | Format | Content |
|
||||
|---|---|---|---|
|
||||
| `<prefix>_dist.csv` | always | CSV matrix | the `--metric` distance, 6 decimals, symmetric, diagonal 0 |
|
||||
| `<prefix>_dist.phy` | always, unless `--csv` | relaxed PHYLIP | the `--distance` matrix |
|
||||
| `<prefix>_dist.csv` | `--csv` | CSV matrix | the `--distance` matrix, 6 decimals |
|
||||
| `<prefix>_shared.csv` | `--shared-kmers` | CSV matrix | shared-kmer count per genome pair (integers) |
|
||||
| `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree, branch lengths from the `--metric` matrix |
|
||||
| `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree, same matrix |
|
||||
| `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree |
|
||||
| `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree |
|
||||
|
||||
Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
|
||||
CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
|
||||
|
||||
### Central-position SNP model
|
||||
|
||||
| File | Written by | Format | Content |
|
||||
|---|---|---|---|
|
||||
| `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
|
||||
| `<prefix>_rawsnp.csv` | `--raw-snp-distance` | CSV matrix | single-copy central-SNP p-distance ($\hat p$), or `NA` |
|
||||
| `<prefix>_rawsnp_counts.csv` | `--raw-snp-counts` | CSV table | per-pair diagnostic counts behind `_rawsnp.csv` |
|
||||
| `<prefix>_snp.fasta` | `--snp` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
|
||||
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
|
||||
| `<prefix>_shannon.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
|
||||
|
||||
**`_siblings.csv`** — family size = number of distinct central bases observed at a family (1–4), not "sibling count" (0–3).
|
||||
|
||||
| Column | Meaning |
|
||||
|---|---|
|
||||
| `genome` | genome label, or the literal `global` for the last row |
|
||||
| `1`, `2`, `3`, `4` | for a genome row: number of families of that size where the genome carries ≥ 1 member. For the `global` row: the actual deduplicated family-size histogram — **not** the sum of the rows above (a family shared by several genomes would otherwise be counted once per genome) |
|
||||
|
||||
**`_rawsnp.csv`** — same matrix layout as `_dist.csv`; each cell is $\hat p = n_{\text{snp}}/(n_{\text{snp}}+n_{\text{shared}})$, 6 decimals, or `NA` when the pair has zero eligible loci (distinguishes "identical everywhere eligible" from "nothing eligible at all").
|
||||
|
||||
**`_rawsnp_counts.csv`** — one row per unordered genome pair (not a matrix), the counts `_rawsnp.csv`'s ratio is computed from:
|
||||
|
||||
| Column | Meaning |
|
||||
|---|---|
|
||||
| `genome_a`, `genome_b` | the pair |
|
||||
| `n_snp` | eligible loci where the two genomes' single forms differ |
|
||||
| `n_shared` | eligible loci where they agree (includes invariant families) |
|
||||
| `n_eligible` | `n_snp + n_shared` |
|
||||
| `ratio` | $\hat p$ = `n_snp / n_eligible`, or `NA` if `n_eligible = 0` |
|
||||
|
||||
**`_snp.fasta`** — one record per non-excluded genome, one column per variable family (family size ≥ 2), header carries an `n_sites` annotation. Each site is IUPAC-coded from the genome's presence mask at that family: single observed form → plain base; several forms → matching IUPAC ambiguity code; no form → `-`.
|
||||
|
||||
**`_family_overlap.csv`** — same matrix layout as `_dist.csv`; cell `[i][j]` = number of `_snp.fasta` columns where both genome `i` and `j` carry a call (neither is `-`). Diagonal `[i][i]` is kept (not skipped): it holds genome `i`'s own total variable-family count.
|
||||
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
|
||||
| `<prefix>_alignment.fasta` | `--pseudo-alignment` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
|
||||
|
||||
### Sankoff calibration and exports
|
||||
|
||||
@@ -205,7 +206,7 @@ Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below):
|
||||
|---|---|---|---|
|
||||
| `<prefix>_sankoff_matrix.csv` | `--sankoff`/`--tnt`/`--phyg`/`--iqtree` | CSV matrix | calibrated 16×16 cost matrix |
|
||||
| `<prefix>_sankoff_params.yaml` | same flags | YAML | calibration report (raw tallies + derived probabilities) |
|
||||
| `<prefix>_sankoff.fasta` | same flags | FASTA | Sankoff-recoded pseudo-alignment |
|
||||
| `<prefix>_sankoff.fasta` | same flags | FASTA | Sankoff-recoded pseudo-alignment, header carries an `n_sites` annotation |
|
||||
| `<prefix>_sankoff.tnt` | `--tnt` | TNT script | ready-to-run parsimony search |
|
||||
| `<prefix>_sankoff.tcm` | `--phyg` | PhyG TCM | cost matrix in PhyG's own format |
|
||||
| `<prefix>_sankoff.pg` | `--phyg` | PhyG script | ready-to-run parsimony search |
|
||||
@@ -213,33 +214,33 @@ Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below):
|
||||
| `<prefix>_iqtree.fasta` | `--iqtree` | FASTA | alignment recoded for that model |
|
||||
| `<prefix>_iqtree_states.csv` | `--iqtree` | CSV table | maps `_iqtree.model`/`_iqtree.fasta`'s compact state symbols back to `_sankoff_matrix.csv`'s alphabet |
|
||||
|
||||
**`_sankoff_matrix.csv`** — header `state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N`: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), `0` standing for the empty/absent state (not `-`, to avoid colliding with external tools' own gap syntax). One row per source state, one value per destination state, cost $-\ln P(a,b)$, 4 decimals.
|
||||
**`_sankoff_matrix.csv`** — header `state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N`: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), `0` standing for the empty/absent state. One row per source state, one value per destination state, cost $-\ln P(a,b)$, 4 decimals.
|
||||
|
||||
**`_sankoff_params.yaml`** — everything the calibration estimated, structured so it can be reloaded rather than re-parsed:
|
||||
**`_sankoff_params.yaml`**:
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `ratio_ceiling` | the `--sankoff-ratio-ceiling` value used |
|
||||
| `cardinality_transitions` | 5×5 list of `{from, to, count, probability}`, family cardinality (0–4 observed forms) |
|
||||
| `cardinality_transitions` | 5×5 list of `{from, to, count, probability}`, family cardinality (0-4 observed forms) |
|
||||
| `composition_transitions` | 4×4 list of `{from, to, count, probability}`, base letters `A/C/G/T`, single-copy substitutions |
|
||||
|
||||
**`_sankoff.fasta`** — same sites as `_snp.fasta`, recoded to match `_sankoff_matrix.csv`'s alphabet: absent state is `0` (or `?` under `--free-loss`). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.
|
||||
**`_sankoff.fasta`** — recoded to match `_sankoff_matrix.csv`'s alphabet: absent state is `0` (or `?` under `--free-loss`). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.
|
||||
|
||||
**`_sankoff.tnt`** (`--tnt`) — self-contained TNT script: `xread` block (alignment recoded to TNT's fixed `0-9A-F` alphabet), an integer-scaled (`--sankoff-cost-scale`) and metric-closed `smatrix`, a default `hold 20; mult; export` search. Run with `printf 'proc <path>;\nquit;\n' | tnt`. Produces `<prefix>_sankoff.tre` (bare numeric leaf labels, order matching `_sankoff.fasta`) — feed both into [`name-tree`](name-tree.md) to recover taxon names.
|
||||
**`_sankoff.tnt`** (`--tnt`) — `xread` block (alignment recoded to TNT's fixed `0-9A-F` alphabet), an integer-scaled (`--sankoff-cost-scale`) and metric-closed `smatrix`, a default `hold 20; mult; export` search. Run with `printf 'proc <path>;\nquit;\n' | tnt`. Produces `<prefix>_sankoff.tre` (bare numeric leaf labels, order matching `_sankoff.fasta`).
|
||||
|
||||
**`_sankoff.tcm`** (`--phyg`) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state `0`, since it's never actually triggered).
|
||||
**`_sankoff.tcm`** (`--phyg`) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state `0`).
|
||||
|
||||
**`_sankoff.pg`** (`--phyg`) — script: `read(prefasta:..., tcm:...)` against `_sankoff.fasta`/`_sankoff.tcm`, a default 300s/4-instance `search`, `report(...)` writing `<prefix>_sankoff.tre` (bare numeric labels, as for `--tnt`). Run with `phyg` from the output directory (the script uses relative file names). Feed the tree plus `_sankoff.fasta` into [`name-tree`](name-tree.md) for taxon names.
|
||||
**`_sankoff.pg`** (`--phyg`) — script: `read(prefasta:..., tcm:...)` against `_sankoff.fasta`/`_sankoff.tcm`, a default 300s/4-instance `search`, `report(...)` writing `<prefix>_sankoff.tre`. Run with `phyg` from the output directory (the script uses relative file names).
|
||||
|
||||
**`_iqtree.model`** (`--iqtree`) — lower-triangular exchangeability matrix $R(a,b) = e^{-\text{cost}(a,b)}$ (one row of increasing length per state, whitespace-separated, PAML order), followed by one line of empirical state frequencies. Only states actually occurring in the alignment are kept, compactly renumbered `0..k-1`.
|
||||
|
||||
**`_iqtree.fasta`** (`--iqtree`) — alignment recoded to that same compact `0..k-1` alphabet (symbols `0-9A-F`). Under `--free-loss`, non-detection becomes `?` and columns left non-informative once missing calls are ignored are dropped first (required for `+ASC`); with `--iqtree-min-freq` also set (the default), any state rarer than that threshold is folded into the same `?` treatment, and non-informative columns are re-checked and dropped again after that. Run with:
|
||||
**`_iqtree.fasta`** (`--iqtree`) — alignment recoded to that same compact `0..k-1` alphabet (symbols `0-9A-F`). Under `--free-loss`, non-detection becomes `?` and columns left non-informative once missing calls are ignored are dropped first (required for `+ASC`); with `--iqtree-min-freq` also set (the default), any state rarer than that threshold is folded into the same `?` treatment, and non-informative columns are re-checked and dropped again. Run with:
|
||||
```
|
||||
iqtree3 -s <prefix>_iqtree.fasta --seqtype MORPH -m <prefix>_iqtree.model+ASC --prefix <prefix>_iqtree -T AUTO
|
||||
```
|
||||
|
||||
**`_iqtree_states.csv`** (`--iqtree`) — one row per state actually kept in `_iqtree.model`/`_iqtree.fasta` (header `iqtree_symbol,canonical_symbol,frequency`): `iqtree_symbol` is the compact `0-9A-F` symbol as written in those two files, `canonical_symbol` is the matching `_sankoff_matrix.csv` state, `frequency` is that state's empirical frequency at full precision (`_iqtree.model`'s own frequency line is rounded to 6 decimals). Under `--free-loss`, absent (`0`/`?`) is never a kept state, so it never appears here — nor does any state `--iqtree-min-freq` folded away for being too rare. Use this file to identify which real state a given row/column of `_iqtree.model`'s matrix corresponds to — e.g. to check whether a state showing zero exchangeability with everything else is expected (a state combination that never co-occurs with anything else in this data) or worth investigating further.
|
||||
**`_iqtree_states.csv`** (`--iqtree`) — one row per state actually kept in `_iqtree.model`/`_iqtree.fasta` (header `iqtree_symbol,canonical_symbol,frequency`): `iqtree_symbol` is the compact `0-9A-F` symbol as written in those two files, `canonical_symbol` is the matching `_sankoff_matrix.csv` state, `frequency` is that state's empirical frequency at full precision. Under `--free-loss`, absent (`0`/`?`) is never a kept state, so it never appears here — nor does any state `--iqtree-min-freq` folded away for being too rare.
|
||||
|
||||
### Rare states and `--iqtree-min-freq`
|
||||
|
||||
States that combine 3 or 4 central bases at once (IUPAC `V`/`H`/`K`.../`N`) are inherently rare — and, on real data, rare enough that they can make `iqtree3` itself numerically unstable ("Numerical underflow for lh-derivative" warnings, near-degenerate likelihood optimization). They're also more likely to be assembly/detection noise than genuine, widely-shared multi-way polymorphism, the same "sampling failure, not true signal" reasoning `--free-loss` already applies to non-detection. With `--free-loss` set, `--iqtree-min-freq` (default `0.001`, i.e. one in a thousand) extends that same missing-data treatment to any state below this frequency, not just absence. Check `_iqtree_states.csv` to see exactly which states survived and at what frequency; set `--iqtree-min-freq 0` to disable this and keep every state that occurs at all (the old behavior). Has no effect without `--free-loss` — there is no missing-data symbol to fold rare states into otherwise.
|
||||
States that combine 3 or 4 central bases at once (IUPAC `V`/`H`/`K`.../`N`) are inherently rare, and can make `iqtree3` itself numerically unstable ("Numerical underflow for lh-derivative" warnings). With `--free-loss` set, `--iqtree-min-freq` (default `0.001`, one in a thousand) extends the missing-data treatment to any state below this frequency, not just absence. Check `_iqtree_states.csv` to see exactly which states survived and at what frequency; set `--iqtree-min-freq 0` to keep every state that occurs at all. Has no effect without `--free-loss`.
|
||||
Loaded 100 of 1329 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user