Compare commits
143
Commits
v1.2.2
..
ee62b131f9
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ee62b131f9 | ||
|
|
85296a2ef8 | ||
|
|
9ac3a37529 | ||
|
|
7c7d7615cb | ||
|
|
c6226dd40c | ||
|
|
8296e3544c | ||
|
|
12afbe819c | ||
|
|
f9edefd135 | ||
|
|
025c40c0db | ||
|
|
8468319a63 | ||
|
|
d32d86193f | ||
|
|
093d81c359 | ||
|
|
09c80468b2 | ||
|
|
fe38424682 | ||
|
|
8b87078478 | ||
|
|
d686d39506 | ||
|
|
3bd7fd4f41 | ||
|
|
c226744405 | ||
|
|
0e40e83fd1 | ||
|
|
cb5fc202b6 | ||
|
|
2232628e58 | ||
|
|
f29dcbcfb8 | ||
|
|
0d12494f1a | ||
|
|
95bbc0b2ae | ||
|
|
fdfc1eb3dd | ||
|
|
996ebaf977 | ||
|
|
cd06fa2ce3 | ||
|
|
1702addbf9 | ||
|
|
0d1ecac233 | ||
|
|
e715528d34 | ||
|
|
9da0876ee6 | ||
|
|
755cf85334 | ||
|
|
2693a6af09 | ||
|
|
7861e886e7 | ||
|
|
a8bcf5ffac | ||
|
|
be838da256 | ||
|
|
020b391636 | ||
|
|
e846d35adb | ||
|
|
c0e1968654 | ||
|
|
e280b6d0a3 | ||
|
|
98dba1802d | ||
|
|
dd4285b269 | ||
|
|
b66a488277 | ||
|
|
b954032faa | ||
|
|
43d89d8f02 | ||
|
|
8bb1d52514 | ||
|
|
77d4429645 | ||
|
|
4d55258ada | ||
|
|
7e28ca4853 | ||
|
|
52f26809af | ||
|
|
54e4600120 | ||
|
|
579cfd8752 | ||
|
|
6b0c0867cc | ||
|
|
93fe838f97 | ||
|
|
7363370065 | ||
|
|
caf59b5658 | ||
|
|
95fa0c93b2 | ||
|
|
e101f629e6 | ||
|
|
4f34a646c5 | ||
|
|
0b40d2d0da | ||
|
|
1536217ac1 | ||
|
|
9043868228 | ||
|
|
e015362ce6 | ||
|
|
89629e118d | ||
|
|
beb2951c20 | ||
|
|
5a0b71d105 | ||
|
|
bbb58a698f | ||
|
|
920120d20e | ||
|
|
d084396aba | ||
|
|
9dee6dcd08 | ||
|
|
635fc830d1 | ||
|
|
4b6005962e | ||
|
|
6bdc9354d3 | ||
|
|
4ea3cd32ba | ||
|
|
bd7729b095 | ||
|
|
10a0909687 | ||
|
|
b0890d1781 | ||
|
|
904d85f33b | ||
|
|
96dfb5300b | ||
|
|
768fa0183d | ||
|
|
737966899f | ||
|
|
354e6f9bf1 | ||
|
|
b7a8b5e6cf | ||
|
|
294f132a0a | ||
|
|
16ade823d6 | ||
|
|
881b1532b5 | ||
|
|
ad0b7173a3 | ||
|
|
5dc16b0127 | ||
|
|
cbf6893f38 | ||
|
|
bb380d0c7d | ||
|
|
23812d1af8 | ||
|
|
fba9c65b1a | ||
|
|
fb31a35c76 | ||
|
|
5048f43eea | ||
|
|
1261aeba86 | ||
|
|
7183e3adb4 | ||
|
|
2419a6c21d | ||
|
|
fc4464a0ef | ||
|
|
c9d10d55c7 | ||
|
|
00ba968628 | ||
|
|
8d6ba6546b | ||
|
|
02dbdd11aa | ||
|
|
da3aa5a2cb | ||
|
|
31bb324752 | ||
|
|
4b7b3c3c1a | ||
|
|
346095b9eb | ||
|
|
0299b608e2 | ||
|
|
878b63566f | ||
|
|
96b6517541 | ||
|
|
5c1584967f | ||
|
|
abc51c2add | ||
|
|
616cb76af3 | ||
|
|
164e879585 | ||
|
|
c4b69e1af5 | ||
|
|
1c54e60c9a | ||
|
|
6c860f120f | ||
|
|
b5ec0122d0 | ||
|
|
f64ac14f0e | ||
|
|
9379bbaad8 | ||
|
|
f9ef6b8391 | ||
|
|
76cbd3a886 | ||
|
|
f7ebc7a1ab | ||
|
|
7eaa8c2016 | ||
|
|
5a91817488 | ||
|
|
640f29725b | ||
|
|
c2e0533fa9 | ||
|
|
9abee87af3 | ||
|
|
19f9954050 | ||
|
|
82374deca5 | ||
|
|
ac38aa759b | ||
|
|
82ddeaddcd | ||
|
|
a4eb20e67e | ||
|
|
89ea077456 | ||
|
|
5a9d903e51 | ||
|
|
32bcbd1465 | ||
|
|
0da725ffe9 | ||
|
|
3b65319529 | ||
|
|
69747dcb53 | ||
|
|
2dba217482 | ||
|
|
dbd8af376c | ||
|
|
dc3d82f8db | ||
|
|
308d2b9f92 | ||
|
|
0f389b37f2 |
No files matched your search
@@ -62,7 +62,8 @@ jobs:
|
||||
~/.cargo/registry
|
||||
~/.cargo/git
|
||||
src/target
|
||||
key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock') }}
|
||||
tools/migrate-mphf/target
|
||||
key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock', 'tools/migrate-mphf/Cargo.lock') }}
|
||||
restore-keys: linux-musl-cargo-
|
||||
|
||||
- name: Build static binary
|
||||
@@ -83,6 +84,28 @@ jobs:
|
||||
-H "Authorization: token $GITEA_TOKEN" \
|
||||
-F "attachment=@/tmp/dist/obikmer-linux-x86_64"
|
||||
|
||||
# One-shot converter for indexes built with ptr_hash 1.x. A standalone
|
||||
# crate (own Cargo.lock/target, outside src/'s workspace), built after
|
||||
# obikmer's own asset is already uploaded: it never blocks that one.
|
||||
- name: Build migrate-mphf (static)
|
||||
working-directory: tools/migrate-mphf
|
||||
env:
|
||||
PKG_CONFIG_ALLOW_CROSS: "1"
|
||||
run: cargo zigbuild --release --target x86_64-unknown-linux-musl
|
||||
|
||||
- name: Upload migrate-mphf artifact
|
||||
working-directory: tools/migrate-mphf
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEATOKEN }}
|
||||
RELEASE_ID: ${{ needs.create-release.outputs.release_id }}
|
||||
run: |
|
||||
cp target/x86_64-unknown-linux-musl/release/migrate-mphf /tmp/dist/migrate-mphf-linux-x86_64
|
||||
strip /tmp/dist/migrate-mphf-linux-x86_64
|
||||
curl -s -X POST \
|
||||
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases/$RELEASE_ID/assets" \
|
||||
-H "Authorization: token $GITEA_TOKEN" \
|
||||
-F "attachment=@/tmp/dist/migrate-mphf-linux-x86_64"
|
||||
|
||||
build-macos-arm64:
|
||||
needs: create-release
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
+23
-17
@@ -1,32 +1,37 @@
|
||||
.venv/
|
||||
.DS_Store
|
||||
.claude/
|
||||
.kilo/
|
||||
.serena/
|
||||
.zed/
|
||||
.ast-cache/
|
||||
CLAUDE.md
|
||||
memory/
|
||||
sandbox/
|
||||
src/target
|
||||
tools/migrate-mphf/target
|
||||
data-stress
|
||||
*.fasta
|
||||
*.fasta.gz
|
||||
*.fastq
|
||||
*.fastq.gz
|
||||
*.vcf
|
||||
*.zst
|
||||
*.zst.meta
|
||||
*.pb
|
||||
./**/*.json
|
||||
**/*.json
|
||||
*.bin
|
||||
*.log
|
||||
*.csv
|
||||
*.meta
|
||||
*.pfiv
|
||||
*.done
|
||||
*.efh
|
||||
*.efl
|
||||
*.bin.idx
|
||||
*.prsb
|
||||
Betula_exilis--IGA-24-33
|
||||
benchmark/genomes
|
||||
benchmark/genomes_orig
|
||||
benchmark/simulated_data
|
||||
benchmark/specimen_index_presence
|
||||
benchmark/specimen_index_count
|
||||
benchmark/global_index_presence
|
||||
benchmark/global_index_presence_orig
|
||||
benchmark/global_index_presence_sav
|
||||
benchmark/all_specific
|
||||
benchmark/global_index_count
|
||||
benchmark/stats
|
||||
benchmark/reference_index
|
||||
benchmark/reference_dist
|
||||
benchmark/obikmer_dist
|
||||
benchmark/specific_index_count
|
||||
benchmark/specific_index_presence
|
||||
benchmark/run/
|
||||
TNT
|
||||
phyg
|
||||
biblio
|
||||
@@ -40,3 +45,4 @@ biblio
|
||||
*.parstree
|
||||
*.ckp.gz
|
||||
*.model
|
||||
scripts/__pycache__/mk_wiki.cpython-314.pyc
|
||||
@@ -1,87 +0,0 @@
|
||||
# Plan d'amélioration technique - obiskio
|
||||
|
||||
## 1. Contexte et objectifs
|
||||
- **Objectif** : Renforcer la robustesse, la maintenabilité et les performances de la crate `obiskio`.
|
||||
- **Priorités** :
|
||||
1. Gestion des erreurs
|
||||
2. Optimisation de la mémoire du pool
|
||||
3. Robustesse concurrente
|
||||
4. Couverture de tests
|
||||
5. Documentation
|
||||
|
||||
---
|
||||
|
||||
## 2. Axes d'amélioration détaillés
|
||||
|
||||
### 2.1 Gestion des erreurs
|
||||
- **Problème** : `SKError` ne couvre pas tous les cas (format invalide, taille maximale, CRC)
|
||||
- **Actions** :
|
||||
- Ajouter variante `ParseError(String)` dans `src/error.rs`
|
||||
- Valider les tailles de SuperKmer avant parsing
|
||||
- Remplacer `expect()` par `unwrap_or_else` avec messages explicites
|
||||
- Documenter chaque variante d’erreur dans le README
|
||||
|
||||
### 2.2 Optimisation du pool de fichiers
|
||||
- **Problème** : `SKFilePool` utilise un `Vec<WriteEntry>` non contraint et n’effectue pas de nettoyage en cas d’erreur
|
||||
- **Actions** :
|
||||
- Implémenter un `LimitedVec` avec limite stricte à `MAX_POOL_SIZE`
|
||||
- Créer `clear_memory()` qui supprime les entrées orphelines
|
||||
- Ajouter `evict_lru_threshold()` pour éviction proactive
|
||||
- Introduire un `RwLock` pour les opérations de lecture massives
|
||||
|
||||
### 2.3 Robustesse concurrente
|
||||
- **Problème** : Risque de deadlocks dans `SKFileWriter::write_batch()` et `SKFileReader::reopen_and_seek()`
|
||||
- **Actions** :
|
||||
- Remplacer `Mutex` par `RwLock` pour les accès en lecture
|
||||
- Ajouter un compteur de blocage et logs de timeout
|
||||
- Utiliser `std::thread::park_timeout` pour débloquer
|
||||
- Insérer `debug_assert!` sur les états invariants
|
||||
|
||||
### 2.4 Couverture de tests
|
||||
- **Problème** : Absence de benchmarks, de tests de migration, de résilience de fichiers corrompus
|
||||
- **Actions** :
|
||||
- Benchmarks I/O sur 10k+ SuperKmer avec `criterion`
|
||||
- Tests de migration de version de fichier `.meta` → `.v2.meta`
|
||||
- Tests de corruption volontaire (truncature, inversion de bits)
|
||||
- Tests de stress sur pool saturation (100 threads)
|
||||
|
||||
### 2.5 Documentation & exemples
|
||||
- **Actions** :
|
||||
- Ajouter des examples dans chaque module (`# Examples`)
|
||||
- Documenter la logique LRU avec diagrammes Mermaid
|
||||
- Créer un guide « How to recover from eviction »
|
||||
- Mettre à jour le `README.md` avec tableau des variantes d’erreur
|
||||
|
||||
---
|
||||
|
||||
## 3. Plan d'exécution (Roadmap)
|
||||
|
||||
| Sprint | Durée | Livrables clés |
|
||||
|--------|-------|----------------|
|
||||
| **S1** | 2 jours | Refactorisation `SKError`, ajout de tests unitaires |
|
||||
| **S2** | 3 jours | Implémentation `clear_memory()` + `LimitedVec` |
|
||||
| **S3** | 2 jours | Passage à `RwLock`, ajout de compteurs de blocage |
|
||||
| **S4** | 2 jours | Benchmarks + tests de migration |
|
||||
| **S5** | 1 jour | Documentation finale & mise à jour du README |
|
||||
|
||||
---
|
||||
|
||||
## 4. Dépendances externes
|
||||
- Mettre à jour `niffler` vers la version 2.0 (performance compression)
|
||||
- Évaluer `bincode` vs `serde_json` pour les métas (I/O)
|
||||
- Ajouter dépendance `criterion` (dev‑dependencies)
|
||||
|
||||
---
|
||||
|
||||
## 5. KPI de suivi
|
||||
- **Couverture de tests** : ≥85 % des chemins critiques
|
||||
- **Latence moyenne d’écriture** : ↓15 % après optimisation du pool
|
||||
- **Taux d’erreurs résolues** : 100 % des nouvelles variantes couvertes
|
||||
- **Temps de build CI** : ≤5 min pour l’ensemble des benchmarks
|
||||
|
||||
---
|
||||
|
||||
## 6. Validation finale
|
||||
- Revue de code avec `cargo clippy -- -D warnings`
|
||||
- Analyse de toxicité avec `cargo deny open-source-licenses`
|
||||
- Vérification de la conformité aux standards de naming du projet
|
||||
@@ -1,2 +0,0 @@
|
||||
/cache
|
||||
/project.local.yml
|
||||
@@ -1,133 +0,0 @@
|
||||
# the name by which the project can be referenced within Serena
|
||||
project_name: "obikmer"
|
||||
|
||||
|
||||
# list of languages for which language servers are started; choose from:
|
||||
# al angular ansible bash clojure
|
||||
# cpp cpp_ccls crystal csharp csharp_omnisharp
|
||||
# dart elixir elm erlang fortran
|
||||
# fsharp go groovy haskell haxe
|
||||
# hlsl html java json julia
|
||||
# kotlin lean4 lua luau markdown
|
||||
# matlab msl nix ocaml pascal
|
||||
# perl php php_phpactor powershell python
|
||||
# python_jedi python_ty r rego ruby
|
||||
# ruby_solargraph rust scala scss solidity
|
||||
# svelte swift systemverilog terraform toml
|
||||
# typescript typescript_vts vue yaml zig
|
||||
# (This list may be outdated. For the current list, see values of Language enum here:
|
||||
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
|
||||
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
|
||||
# Note:
|
||||
# - For C, use cpp
|
||||
# - For JavaScript, use typescript
|
||||
# - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
|
||||
# - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
|
||||
# - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
|
||||
# - For Free Pascal/Lazarus, use pascal
|
||||
# Special requirements:
|
||||
# Some languages require additional setup/installations.
|
||||
# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
|
||||
# When using multiple languages, the first language server that supports a given file will be used for that file.
|
||||
# The first language is the default language and the respective language server will be used as a fallback.
|
||||
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
|
||||
languages:
|
||||
- rust
|
||||
|
||||
# the encoding used by text files in the project
|
||||
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
|
||||
encoding: "utf-8"
|
||||
|
||||
# line ending convention to use when writing source files.
|
||||
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
|
||||
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
|
||||
line_ending:
|
||||
|
||||
# The language backend to use for this project.
|
||||
# If not set, the global setting from serena_config.yml is used.
|
||||
# Valid values: LSP, JetBrains
|
||||
# Note: the backend is fixed at startup. If a project with a different backend
|
||||
# is activated post-init, an error will be returned.
|
||||
language_backend:
|
||||
|
||||
# whether to use project's .gitignore files to ignore files
|
||||
ignore_all_files_in_gitignore: true
|
||||
|
||||
# advanced configuration option allowing to configure language server-specific options.
|
||||
# Maps the language key to the options.
|
||||
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
|
||||
# No documentation on options means no options are available.
|
||||
ls_specific_settings: {}
|
||||
|
||||
# list of additional workspace folder paths for cross-package reference support (e.g. in monorepos).
|
||||
# Paths can be absolute or relative to the project root.
|
||||
# Each folder is registered as an LSP workspace folder, enabling language servers to discover
|
||||
# symbols and references across package boundaries.
|
||||
# Currently supported for: TypeScript.
|
||||
# Example:
|
||||
# additional_workspace_folders:
|
||||
# - ../sibling-package
|
||||
# - ../shared-lib
|
||||
additional_workspace_folders: []
|
||||
|
||||
# list of additional paths to ignore in this project.
|
||||
# Same syntax as gitignore, so you can use * and **.
|
||||
# Note: global ignored_paths from serena_config.yml are also applied additively.
|
||||
ignored_paths: []
|
||||
|
||||
# whether the project is in read-only mode
|
||||
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
|
||||
# Added on 2025-04-18
|
||||
read_only: false
|
||||
|
||||
# list of tool names to exclude.
|
||||
# This extends the existing exclusions (e.g. from the global configuration)
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
excluded_tools: []
|
||||
|
||||
# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
|
||||
# This extends the existing inclusions (e.g. from the global configuration).
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
included_optional_tools: []
|
||||
|
||||
# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
|
||||
# This cannot be combined with non-empty excluded_tools or included_optional_tools.
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
fixed_tools: []
|
||||
|
||||
# list of mode names that are to be activated by default, overriding the setting in the global configuration.
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
|
||||
# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
|
||||
# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
|
||||
# for this project.
|
||||
# This setting can, in turn, be overridden by CLI parameters (--mode).
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
default_modes:
|
||||
|
||||
# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
added_modes:
|
||||
|
||||
# initial prompt for the project. It will always be given to the LLM upon activating the project
|
||||
# (contrary to the memories, which are loaded on demand).
|
||||
initial_prompt: ""
|
||||
|
||||
# time budget (seconds) per tool call for the retrieval of additional symbol information
|
||||
# such as docstrings or parameter information.
|
||||
# This overrides the corresponding setting in the global configuration; see the documentation there.
|
||||
# If null or missing, use the setting from the global configuration.
|
||||
symbol_info_budget:
|
||||
|
||||
# list of regex patterns which, when matched, mark a memory entry as read‑only.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
read_only_memory_patterns: []
|
||||
|
||||
# list of regex patterns for memories to completely ignore.
|
||||
# Matching memories will not appear in list_memories or activate_project output
|
||||
# and cannot be accessed via read_memory or write_memory.
|
||||
# To access ignored memory files, use the read_file tool on the raw file path.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
# Example: ["_archive/.*", "_episodes/.*"]
|
||||
ignored_memory_patterns: []
|
||||
@@ -1,59 +0,0 @@
|
||||
// Project tasks configuration. See https://zed.dev/docs/tasks for documentation.
|
||||
//
|
||||
// Example:
|
||||
[
|
||||
{
|
||||
"label": "Example task",
|
||||
"command": "for i in {1..5}; do echo \"Hello $i/5\"; sleep 1; done",
|
||||
//"args": [],
|
||||
// Env overrides for the command, will be appended to the terminal's environment from the settings.
|
||||
"env": { "foo": "bar" },
|
||||
// Current working directory to spawn the command into, defaults to current project root.
|
||||
//"cwd": "/path/to/working/directory",
|
||||
// Whether to use a new terminal tab or reuse the existing one to spawn the process, defaults to `false`.
|
||||
"use_new_terminal": false,
|
||||
// Whether to allow multiple instances of the same task to be run, or rather wait for the existing ones to finish, defaults to `false`.
|
||||
"allow_concurrent_runs": false,
|
||||
// What to do with the terminal pane and tab, after the command was started:
|
||||
// * `always` — always show the task's pane, and focus the corresponding tab in it (default)
|
||||
// * `no_focus` — always show the task's pane, add the task's tab in it, but don't focus it
|
||||
// * `never` — do not alter focus, but still add/reuse the task's tab in its pane
|
||||
"reveal": "always",
|
||||
// Where to place the task's terminal item after starting the task:
|
||||
// * `dock` — in the terminal dock, "regular" terminal items' place (default)
|
||||
// * `center` — in the central pane group, "main" editor area
|
||||
"reveal_target": "dock",
|
||||
// What to do with the terminal pane and tab, after the command had finished:
|
||||
// * `never` — Do nothing when the command finishes (default)
|
||||
// * `always` — always hide the terminal tab, hide the pane also if it was the last tab in it
|
||||
// * `on_success` — hide the terminal tab on task success only, otherwise behaves similar to `always`
|
||||
"hide": "never",
|
||||
// Which shell to use when running a task inside the terminal.
|
||||
// May take 3 values:
|
||||
// 1. (default) Use the system's default terminal configuration in /etc/passwd
|
||||
// "shell": "system"
|
||||
// 2. A program:
|
||||
// "shell": {
|
||||
// "program": "sh"
|
||||
// }
|
||||
// 3. A program with arguments:
|
||||
// "shell": {
|
||||
// "with_arguments": {
|
||||
// "program": "/bin/bash",
|
||||
// "args": ["--login"]
|
||||
// }
|
||||
// }
|
||||
"shell": "system",
|
||||
// Whether to show the task line in the output of the spawned task, defaults to `true`.
|
||||
"show_summary": true,
|
||||
// Whether to show the command line in the output of the spawned task, defaults to `true`.
|
||||
"show_command": true,
|
||||
// Which edited buffers to save before running the task:
|
||||
// * `all` — save all edited buffers
|
||||
// * `current` — save currently active buffer only
|
||||
// * `none` — don't save any buffers
|
||||
"save": "none",
|
||||
// Represents the tags for inline runnable indicators, or spawning multiple tasks at once.
|
||||
// "tags": []
|
||||
},
|
||||
]
|
||||
@@ -1,103 +0,0 @@
|
||||
**PROMPT**
|
||||
|
||||
Tu es ma base de connaissance et mon bloc-notes intelligent sur le projet **obikmer**. Tu ne proposes pas, tu ne codes pas spontanément — tu réponds à mes questions et tu structures mes idées au fur et à mesure que je les exprime.
|
||||
|
||||
**Règle absolue : une question appelle une réponse, pas une action.**
|
||||
Ne modifier aucun fichier à moins d'une demande explicite de modification. En particulier : observer un bug ou une incohérence dans le code montré ne constitue pas un mandat pour le corriger. Le code montré peut refléter une intention en cours — modifier sans mandat risque d'introduire un vrai bug là où tu croyais corriger.
|
||||
|
||||
**Règle absolue : ne jamais substituer une dépendance ou une bibliothèque sans validation explicite.**
|
||||
Si une dépendance demandée pose problème (erreur de compilation, bug, API manquante), exposer le problème et proposer des alternatives — ne jamais switcher silencieusement vers une autre bibliothèque. Le choix des dépendances est une décision d'architecture qui appartient au développeur.
|
||||
|
||||
**Règle absolue : le code existant est une hypothèse, pas une vérité.**
|
||||
Quand une nouvelle construction (type, itérateur, abstraction) rend du code historique injustifié, le signaler immédiatement et proposer de le supprimer — ne pas conserver les deux en parallèle par inertie. Le développeur demande explicitement de remettre en cause le code base : ne pas attendre qu'il insiste.
|
||||
|
||||
Tu maintiens en **anglais**, dense et sans remplissage, les documents suivants :
|
||||
- `DevDocMD/index.md` — document de discussion de base, enrichi progressivement au fil de nos échanges ; il reflète l'état courant de la réflexion sur le projet
|
||||
- les autres fichiers Markdown dans `DevDocMD/` selon leur thème respectif
|
||||
|
||||
Les snippets de code y sont courts et illustrent uniquement des principes architecturaux. Nos échanges se font en **français**.
|
||||
|
||||
---
|
||||
|
||||
## Contexte du projet
|
||||
|
||||
`obikmer` est un outil Rust de manipulation, comptage, indexation et opérations ensemblistes sur des séquences ADN représentées comme des ensembles de kmers.
|
||||
|
||||
**Contraintes fondamentales**
|
||||
- Efficacité maximale en calcul, mémoire et disque
|
||||
- Données métagénomiques : plusieurs dizaines de Gbases, milliards de kmers
|
||||
- k impair, k ∈ [11, 31], fixé à l'exécution
|
||||
- Formats d'entrée : FASTA, FASTQ, gzip, streaming stdin
|
||||
|
||||
**Opérations prioritaires**
|
||||
- Comptage de kmers (fréquences)
|
||||
- Recherche / requête rapide
|
||||
- Opérations ensemblistes (union, intersection, diff)
|
||||
|
||||
**Ce qui a déjà été discuté**
|
||||
- Encodage 2 bits/base → kmer tient dans un `u64`
|
||||
- Forme canonique : `min(kmer, revcomp)` pour réduire l'espace de moitié
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure de documentation
|
||||
|
||||
La documentation est gérée via **MkDocs + thème Material**, avec publication sur **GitHub Pages**. Deux arbres de documentation indépendants, deux configurations à la racine du dépôt (voir aussi `UserDocMD/` ci-dessous pour la doc utilisateur) :
|
||||
|
||||
**Structure des répertoires**
|
||||
|
||||
```
|
||||
DevDocMD/ ← sources Markdown, doc développeur (discussion, historique, rationale)
|
||||
UserDocMD/ ← sources Markdown, doc utilisateur (état factuel courant, sans code Rust)
|
||||
mkdocs.yml ← config doc développeur : docs_dir DevDocMD, site_dir DevDoc
|
||||
mkdocs-user.yml ← config doc utilisateur : docs_dir UserDocMD, site_dir doc
|
||||
DevDoc/ ← site HTML généré (doc développeur)
|
||||
doc/ ← site HTML généré (doc utilisateur — publié par GitHub Pages, répertoire par défaut)
|
||||
.venv/ ← environnement Python (ignoré par git)
|
||||
```
|
||||
|
||||
**Commandes Makefile**
|
||||
|
||||
| Commande | Effet |
|
||||
|---|---|
|
||||
| `make doc` | Construit la doc développeur dans `DevDoc/` |
|
||||
| `make doc-serve` | Serveur local (doc développeur), rechargement automatique |
|
||||
| `make clean-doc` | Supprime `DevDoc/` |
|
||||
| `make doc-user` | Construit la doc utilisateur dans `doc/` |
|
||||
| `make doc-user-serve` | Serveur local (doc utilisateur), rechargement automatique |
|
||||
| `make clean-doc-user` | Supprime `doc/` |
|
||||
| `make clean` | Supprime `DevDoc/`, `doc/` et `.venv/` |
|
||||
|
||||
Le `.venv/` est dans `.gitignore`. `DevDoc/` et `doc/` (sorties HTML) sont versionnés — `doc/` spécifiquement parce que c'est le répertoire par défaut que GitHub Pages sert.
|
||||
|
||||
Lors de l'ajout de nouveaux fichiers Markdown dans `DevDocMD/`, mettre à jour la section `nav:` de `mkdocs.yml` ; dans `UserDocMD/`, mettre à jour `mkdocs-user.yml`.
|
||||
|
||||
---
|
||||
|
||||
Je continue à poser mes questions et à guider la discussion.
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools
|
||||
|
||||
**Règle absolue : avant tout travail de code, appeler `mcp__serena__initial_instructions` pour charger les instructions Serena.**
|
||||
|
||||
### Hiérarchie des outils pour ce projet Rust
|
||||
|
||||
**Navigation et édition de code → serena en priorité**
|
||||
- Trouver un symbole, une déclaration, les implémentations d'un trait : `mcp__serena__find_symbol`, `mcp__serena__find_declaration`, `mcp__serena__find_implementations`
|
||||
- Trouver les usages d'un symbole : `mcp__serena__find_referencing_symbols`
|
||||
- Diagnostics LSP (erreurs de compilation) : `mcp__serena__get_diagnostics_for_file`
|
||||
- Vue d'ensemble d'un fichier : `mcp__serena__get_symbols_overview`
|
||||
- Modifier le corps d'une fonction/impl : `mcp__serena__replace_symbol_body`
|
||||
- Ne pas utiliser `cclsp` quand serena couvre le besoin
|
||||
|
||||
**Analyse architecturale → jcodemunch**
|
||||
- Hotspots, couplage, dead code, dépendances entre modules
|
||||
- Utiliser avant de refactorer une zone critique
|
||||
|
||||
**Raisonnement complexe → sequential-thinking**
|
||||
- Décisions d'architecture, choix d'algorithme, trade-offs non triviaux
|
||||
|
||||
**Documentation de crates → context7**
|
||||
- Toujours consulter avant d'utiliser une API de bibliothèque externe
|
||||
+59
-3
@@ -722,14 +722,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="/implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="/implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -939,6 +939,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="/implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="/implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1063,7 +1119,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1214,10 +1270,10 @@
|
||||
<h1 id="coverage-architectureindex_architecturemd">Coverage: architecture/index_architecture.md</h1>
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obilayeredmap/src/layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix</li>
|
||||
<li><code>obilayeredmap/src/mphf_layer.rs</code> — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum</li>
|
||||
<li><code>obilayeredmap/src/map.rs</code> — LayeredMap<D></li>
|
||||
<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta, PartitionMeta</li>
|
||||
<li><code>obikindex/src/layer/typed_layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix</li>
|
||||
<li><code>obikindex/src/layer/mphf_layer.rs</code> — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum</li>
|
||||
<li><code>obikindex/src/layer/map.rs</code> — LayeredMap<D></li>
|
||||
<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta, PartitionMeta</li>
|
||||
<li><code>obikindex/src/meta.rs</code> — IndexConfig (kmer_size, n_bits, with_counts, evidence, block_bits), IndexMeta</li>
|
||||
<li><code>obikindex/src/index.rs</code> — KmerIndex, build_layers</li>
|
||||
<li><code>obicompactvec/src/</code> — PersistentCompactIntMatrix, PersistentBitMatrix (DataStore implementations)</li>
|
||||
|
||||
@@ -731,14 +731,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -948,6 +948,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1314,7 +1370,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -729,14 +729,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -946,6 +946,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1072,7 +1128,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1311,6 +1367,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Panic propagation (fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1497,6 +1564,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Panic propagation (fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1780,11 +1858,27 @@ dormant workers have been woken and grows every node by the same amount per
|
||||
step, capped by that node's remaining dormant workers and by the run's total
|
||||
budget (<code>n_total</code>) — balance across nodes is now guaranteed by construction,
|
||||
not incidental to channel implementation details.</p>
|
||||
<h2 id="panic-propagation-fixed-2026-08-28">Panic propagation (fixed, 2026-08-28)</h2>
|
||||
<p>A worker whose <code>f(i)</code> panicked used to never send its <code>WorkerEvent::Completed</code>
|
||||
— the controller's <code>while completed < n_total</code> loop then waited forever for
|
||||
an event that partition could no longer produce, since other live
|
||||
workers/timer threads kept <code>event_rx</code> open. Discovered via <code>obikselect</code>
|
||||
panicking on an unimplemented <code>Sparse</code> matrix case (see <code>implementation/select.md</code>):
|
||||
the process hung indefinitely instead of erroring out.</p>
|
||||
<p>Fixed: each worker wraps <code>f(i)</code> in <code>std::panic::catch_unwind</code> and sends a new
|
||||
<code>WorkerEvent::Panicked(i, payload)</code> instead of silently dropping the
|
||||
partition. The controller counts it toward <code>completed</code> (unblocking the loop)
|
||||
and keeps the first payload seen; once <code>run</code> returns, <code>panic::resume_unwind</code>
|
||||
re-raises it on the caller's thread — the original message/backtrace still
|
||||
surfaces, from the right place, instead of a silent deadlock. Takes priority
|
||||
over a plain <code>Err</code> (a panic means a bug, not a normal typed failure).</p>
|
||||
<h2 id="open-questions">Open questions</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Error handling</strong>: <code>run</code> currently returns the first error; remaining errors
|
||||
are dropped. A <code>Vec<E></code> return would give complete diagnostics.</p>
|
||||
are dropped. A <code>Vec<E></code> return would give complete diagnostics. (Panics are
|
||||
now caught and re-raised individually — see above — this only concerns
|
||||
plain <code>Err</code> results.)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong><code>INITIAL_DIVISOR</code> / <code>GROWTH_DIVISOR</code> tuning</strong>: currently <code>4</code> and <code>8</code>
|
||||
|
||||
@@ -731,14 +731,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -948,6 +948,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1074,7 +1130,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1504,7 +1560,7 @@
|
||||
<td>Replace <code>available_parallelism()</code> with per-NUMA core count for spawn criterion</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>obikpartitionner/src/merge_layer.rs</code></td>
|
||||
<td><code>obikindex/src/partition/merge_layer.rs</code></td>
|
||||
<td>No change — <code>merge_partition</code> already works inside any Rayon context</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -1512,7 +1568,7 @@
|
||||
<td>No change — <code>par_iter</code> and <code>current_num_threads</code> are pool-context-aware</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>obikpartitionner/src/partition.rs</code></td>
|
||||
<td><code>obikindex/src/partition/partition.rs</code></td>
|
||||
<td>No change — same reason</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1215,7 +1271,7 @@
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obikmer/src/cmd/query.rs</code> — commande query, format de sortie</li>
|
||||
<li><code>obikpartitionner/src/query_layer.rs</code> — routage de la requête à travers les partitions</li>
|
||||
<li><code>obikindex/src/partition/query_layer.rs</code> — routage de la requête à travers les partitions</li>
|
||||
<li><code>obiread/src/lib.rs</code> — lecture des séquences d'entrée pour la requête</li>
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1499,7 +1555,7 @@
|
||||
</ul>
|
||||
<hr />
|
||||
<h2 id="algorithm">Algorithm</h2>
|
||||
<p>The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside <code>process_chunk</code> (<code>query.rs</code>); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (<code>obikpartitionner::query_partition_with</code>), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.</p>
|
||||
<p>The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside <code>process_chunk</code> (<code>query.rs</code>); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (<code>obikindex::partition::query_partition_with</code>), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.</p>
|
||||
<div class="highlight"><pre><span></span><code>for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
|
||||
build QueryBatch (QueryBatch::from_records):
|
||||
decompose all sequences into superkmers (SuperKmerIter) — construction only,
|
||||
@@ -1574,9 +1630,9 @@
|
||||
<p>The <code>-z</code> CLI option overrides the index metadata value. A higher z increases stringency (lower FP, some true positives may be discarded at sequence ends); a lower z increases sensitivity.</p>
|
||||
<hr />
|
||||
<h2 id="layer-lookup-mphflayerfind">Layer lookup: <code>MphfLayer::find</code></h2>
|
||||
<p><code>MphfLayer::open(dir, mode: &IndexMode)</code> receives the mode from <code>PartitionMeta</code> — no per-layer file is read. The caller (<code>QueryLayer</code>) never chooses the dispatch path: it is fixed at open time by <code>LayerEvidence</code>. See <a href="../../implementation/obilayeredmap/">obilayeredmap</a> for the full <code>find</code> / <code>find_strict</code> API.</p>
|
||||
<p><code>MphfLayer::open(dir, mode: &IndexMode)</code> receives the mode from <code>PartitionMeta</code> — no per-layer file is read. The caller (<code>QueryLayer</code>) never chooses the dispatch path: it is fixed at open time by <code>LayerEvidence</code>. See <a href="../../implementation/layer_tier/">obikindex::layer</a> for the full <code>find</code> / <code>find_strict</code> API.</p>
|
||||
<h3 id="querylayer-variant-selection"><code>QueryLayer</code> variant selection</h3>
|
||||
<p><code>QueryLayer::open</code> (<code>obikpartitionner/src/query_layer.rs:28-45</code>) only ever returns two variants — <code>Presence</code> or <code>Count</code>, checked in this order:</p>
|
||||
<p><code>QueryLayer::open</code> (<code>obikindex/src/partition/query_layer.rs:28-45</code>) only ever returns two variants — <code>Presence</code> or <code>Count</code>, checked in this order:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -1746,7 +1802,7 @@ Fix direction: restructure <code>query</code>'s pipe with an initial <code>Flat<
|
||||
<p><strong>3. Chunk-size memory formula ignores <code>n_genomes</code>.</strong>
|
||||
<code>chunk_bytes = available_memory_bytes() / (n_workers * 16)</code> (<code>query.rs:407-414</code>) assumes a fixed ~8–16× overhead per raw input byte. But <code>KmerResults::new</code> (<code>query.rs:165-179</code>) allocates <code>data: Vec<u32></code> sized <code>total_kmers_in_chunk × n_genomes</code> — dense, <strong>for every k-mer position in the chunk, hit or not</strong> — plus <code>win_min</code> and (with <code>--detail</code>) <code>cov</code>, same scaling. Real per-chunk memory is <code>O(n_genomes)</code>, not constant; the formula doesn't know <code>n_genomes</code> at all. This is the direct cause of the OOM kill on indexes with many reference genomes.</p>
|
||||
<p><strong>4. MPHF lookup and matrix-row fetch are fused, not staged.</strong>
|
||||
<code>QueryLayer::find_into</code> (<code>obikpartitionner/src/query_layer.rs:48-67</code>) does the MPHF <code>find</code> <em>and</em> the <code>fill_row</code> matrix read in one call per k-mer, inside a single-threaded loop (<code>query_partition_with</code>). There is no separation between "is this k-mer indexed" (cheap, <code>O(1)</code>, independent of <code>n_genomes</code>) and "what are its per-genome values" (the expensive, <code>n_genomes</code>-scaling part).</p>
|
||||
<code>QueryLayer::find_into</code> (<code>obikindex/src/partition/query_layer.rs:48-67</code>) does the MPHF <code>find</code> <em>and</em> the <code>fill_row</code> matrix read in one call per k-mer, inside a single-threaded loop (<code>query_partition_with</code>). There is no separation between "is this k-mer indexed" (cheap, <code>O(1)</code>, independent of <code>n_genomes</code>) and "what are its per-genome values" (the expensive, <code>n_genomes</code>-scaling part).</p>
|
||||
<p><strong>5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.</strong>
|
||||
<code>QueryBatch::from_records</code> currently dereplicates at the <em>superkmer</em> level (<code>HashMap<RoutableSuperKmer, Vec<SKDesc>></code>, <code>query.rs:112</code>). This misses redundancy between k-mers shared by <em>different</em> superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer <em>construction</em> (<code>SuperKmerIter</code>) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on <code>CanonicalKmer</code>, in the same pass: <code>HashMap<CanonicalKmer, Vec<(seq_idx, pos)>></code>. This also means the MPHF <code>find</code> itself runs once per <strong>distinct</strong> k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.</p>
|
||||
<p><strong>6. Stage 1 output: bucket confirmed hits by layer, keyed by MPHF slot.</strong>
|
||||
@@ -1778,12 +1834,12 @@ Stage 2's output should be <code>HashMap<genome_idx, Vec<(seq_idx, positio
|
||||
<p>None of this is implemented yet — parked here as a coherent roadmap while the design is discussed further. Suggested dependency order: (1) I/O parallelism → (3) genome-aware chunk sizing → (4)–(9) staged/k-mer-deduped/NUMA-aware-partition-and-column-major/sparse query engine (larger refactor, biggest structural payoff — reuses <code>PartitionRunner</code> rather than inventing a new parallelism mechanism) → (2) parallel gzip (separate, orthogonal, tracked in chunkreader.md) → secondary diagnostics patterns.</p>
|
||||
<hr />
|
||||
<h2 id="implementation-plan">Implementation plan</h2>
|
||||
<p>Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current <code>query</code> output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning <code>obikmer</code>, <code>obikpartitionner</code>, and <code>obicompactvec</code> — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense <code>KmerResults</code>) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if <code>rapidgzip-rs</code> is validated (see <a href="../../implementation/chunkreader/#future-work--parallel-gzip-decompression-in-xopen">chunkreader.md</a>).</p>
|
||||
<p>Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current <code>query</code> output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning <code>obikmer</code>, <code>obikindex::partition</code>, and <code>obicompactvec</code> — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense <code>KmerResults</code>) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if <code>rapidgzip-rs</code> is validated (see <a href="../../implementation/chunkreader/#future-work--parallel-gzip-decompression-in-xopen">chunkreader.md</a>).</p>
|
||||
<p>Instrumentation is deliberately sequenced <em>before</em> the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.</p>
|
||||
<p>Performance measurement on the reference 192-core/8-NUMA machine is done by the project owner, not from this development environment (macOS, 16 cores — <code>PartitionRunner</code>'s NUMA pinning is Linux-only, so even phase 4's mechanism can't be functionally exercised for its actual purpose here). Each phase below is therefore written to be <em>self-measuring</em>: the debug-level logging it adds must be enough, on its own, to judge whether that phase's algorithmic choice paid off from a cluster run's logs, without needing to attach a profiler.</p>
|
||||
<h3 id="conventions-applied-to-every-phase-below">Conventions applied to every phase below</h3>
|
||||
<p><strong>Debug logging.</strong> Every phase that changes an algorithmic choice (not phase 0, which <em>is</em> the logging) adds <code>tracing::debug!</code>/<code>trace!</code> at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF <code>find</code> calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured <code>debug!</code> per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.</p>
|
||||
<p><strong>Unit tests.</strong> This project's convention (<code>obiread</code>, <code>obikseq</code>, <code>obidebruinj</code>, <code>obicompactvec</code>, <code>obilayeredmap</code>, <code>obiskio</code>, <code>obifastwrite</code>) is <code>#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;</code> at the bottom of the source file, with the actual test code in a sibling <code>src/tests/<name>.rs</code>. Neither <code>obikmer</code> nor <code>obikpartitionner</code> (the two crates phases 3 and 5 touch most) currently have a <code>src/tests/</code> directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.</p>
|
||||
<p><strong>Unit tests.</strong> This project's convention (<code>obiread</code>, <code>obikseq</code>, <code>obidebruinj</code>, <code>obicompactvec</code>, <code>obikindex::layer</code>, <code>obiskio</code>, <code>obifastwrite</code>) is <code>#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;</code> at the bottom of the source file, with the actual test code in a sibling <code>src/tests/<name>.rs</code>. Neither <code>obikmer</code> nor <code>obikindex::partition</code> (the two crates phases 3 and 5 touch most) currently have a <code>src/tests/</code> directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.</p>
|
||||
<p><strong>Workflow (<code>jj</code>).</strong> Work happens in a fresh <code>jj</code> commit, easy to abandon. <code>jj new</code> between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then <code>query_layer.rs</code> split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.</p>
|
||||
<h3 id="phase-0-instrumentation-prerequisite-for-measuring-every-later-phase">Phase 0 — Instrumentation (prerequisite for measuring every later phase)</h3>
|
||||
<p><strong>Goal</strong>: make core utilization, throughput, and per-stage timing visible on a real run, so phases 1–5 can be justified with numbers instead of assumption.</p>
|
||||
@@ -1826,26 +1882,26 @@ Stage 2's output should be <code>HashMap<genome_idx, Vec<(seq_idx, positio
|
||||
<li><code>obikmer/src/cmd/query.rs</code>:</li>
|
||||
<li>Replace <code>QueryBatch::from_records</code>'s dedup map (<code>HashMap<RoutableSuperKmer, Vec<SKDesc>></code>, current <code>query.rs:112</code>) with a per-partition <code>HashMap<CanonicalKmer, Vec<(seq_idx: u32, pos: u32)>></code>, built in the same <code>SuperKmerIter</code> pass: superkmer construction and partition routing (<code>part_idx</code> from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each <code>CanonicalKmer</code> within a superkmer is inserted individually instead of the whole superkmer being the dedup key.</li>
|
||||
<li><strong>Verified</strong>: <code>CanonicalKmer</code> (<code>obikseq/src/kmer.rs:390</code>, <code>pub type CanonicalKmer = CanonicalKmerOf<KLen></code>) — the underlying <code>CanonicalKmerOf<L></code> derives <code>Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash</code> (<code>kmer.rs:269</code>). Usable as a <code>HashMap</code>/<code>HashSet</code> key as-is, no change needed.</li>
|
||||
<li><code>obikpartitionner/src/query_layer.rs</code>:</li>
|
||||
<li><code>obikindex/src/partition/query_layer.rs</code>:</li>
|
||||
<li>Split <code>QueryLayer::find_into</code> (<code>query_layer.rs:48-67</code>) into two methods: <code>find_slot(&self, kmer: CanonicalKmer) -> Option<usize></code> (MPHF only, no matrix touch) and keep <code>fill_row</code> as-is for phase 4 to call later.</li>
|
||||
<li>Replace <code>query_partition_with</code>'s inner loop (<code>query_layer.rs:103-113</code>) with a version that, for each unique <code>CanonicalKmer</code>, calls <code>find_slot</code> across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records <code>(layer_idx, slot)</code>.</li>
|
||||
<li>New return shape for the partition-level query, replacing today's <code>on_hit(sk_idx, kmer_idx, row)</code> callback: <code>HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>></code> (roadmap point 6) — built directly from the k-mer dedup map's <code>Vec<(seq_idx,pos)></code> values, keyed by the resolved slot instead of the k-mer.</li>
|
||||
<li><strong>This phase alone has no throughput benefit yet</strong> (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.</li>
|
||||
<li>Log, per chunk: total k-mer occurrences vs. unique <code>CanonicalKmer</code> count (the dedup ratio — the entire justification for this phase) and the resulting MPHF <code>find</code> call count. If the dedup ratio is close to <code>1.0</code> on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.</li>
|
||||
<li><strong>Unit tests</strong>: create <code>obikmer/src/cmd/tests/query.rs</code> (new <code>src/tests/</code> dir for this crate, following the project's <code>#[cfg(test)] #[path = "tests/query.rs"] mod tests;</code> convention) and <code>obikpartitionner/src/tests/query_layer.rs</code> (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the <code>find_slot</code>/bucket-by-layer-and-slot construction against a small hand-built <code>QueryLayer</code> fixture, asserting the <code>(layer_idx, slot, seq_idx, pos)</code> tuples match what the old per-occurrence loop would have produced.</li>
|
||||
<li><strong>Unit tests</strong>: create <code>obikmer/src/cmd/tests/query.rs</code> (new <code>src/tests/</code> dir for this crate, following the project's <code>#[cfg(test)] #[path = "tests/query.rs"] mod tests;</code> convention) and <code>obikindex/src/partition/tests/query_layer.rs</code> (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the <code>find_slot</code>/bucket-by-layer-and-slot construction against a small hand-built <code>QueryLayer</code> fixture, asserting the <code>(layer_idx, slot, seq_idx, pos)</code> tuples match what the old per-occurrence loop would have produced.</li>
|
||||
</ul>
|
||||
<h3 id="phase-4-column-major-matrix-fetch-roadmap-points-78-implemented-numa-parallelism-deferred">Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred</h3>
|
||||
<p><strong>Goal (revised during implementation)</strong>: replace <code>fill_row</code>-per-hit (row-major, worst-case mmap locality) with a column-major scan. <code>PartitionRunner</code> turned out to be the wrong mechanism for this at this call granularity — see below; the column-major fetch itself is implemented and validated, without it.</p>
|
||||
<p><strong>What shipped:</strong>
|
||||
- <code>obicompactvec</code>: the per-column accessors this phase needed <strong>already existed</strong> — <code>PersistentCompactIntMatrix::col_view(c)</code> and <code>PersistentBitMatrix::col_view(c)</code> are public, and <code>IntSliceView::get(slot)</code>/<code>BitSliceView::get(slot)</code> are public — the original plan underestimated how much of this plumbing the pairwise-distance code (<code>dump</code>/<code>select</code>/<code>stats</code>) had already required. The one real gap: <code>PersistentBitMatrix::col_view()</code> panics on the <code>Implicit</code> variant (the documented mono-genome fast path, <code>bitmatrix.rs</code>). Added <code>PersistentBitMatrix::get(c, slot) -> u32</code> (<code>bitmatrix.rs</code>), a non-panicking column-major point lookup that returns <code>1</code> for <code>Implicit</code> regardless of <code>c</code> — the smallest surface needed, not a new <code>col_get</code> API from scratch.
|
||||
- <code>obikpartitionner/src/query_layer.rs</code>: <code>query_partition_with</code> is now two explicit stages, matching roadmap points 6–8: <strong>stage 1</strong> (MPHF-only, per unique k-mer, bucket hits by <code>(layer_idx, slot)</code>, emits <code>QueryHit::Found</code>) then <strong>stage 2</strong> (per layer with ≥1 hit, column-major: for each genome column <code>g</code> in <code>0..layer.n_cols().min(n_genomes)</code>, scan that layer's bucketed slots and call <code>col_value(g, slot)</code>, emitting <code>QueryHit::Value(descs, g, value)</code> on nonzero). <code>QueryHit</code> is a single enum delivered through one <code>FnMut(QueryHit)</code> callback — an earlier two-closure design (<code>on_found</code> + <code>on_value</code>) didn't borrow-check, since the caller's single mutable accumulator (<code>KmerResults</code>) can't be captured by two separate <code>FnMut</code> closures passed to the same call.
|
||||
- <code>obikindex/src/partition/query_layer.rs</code>: <code>query_partition_with</code> is now two explicit stages, matching roadmap points 6–8: <strong>stage 1</strong> (MPHF-only, per unique k-mer, bucket hits by <code>(layer_idx, slot)</code>, emits <code>QueryHit::Found</code>) then <strong>stage 2</strong> (per layer with ≥1 hit, column-major: for each genome column <code>g</code> in <code>0..layer.n_cols().min(n_genomes)</code>, scan that layer's bucketed slots and call <code>col_value(g, slot)</code>, emitting <code>QueryHit::Value(descs, g, value)</code> on nonzero). <code>QueryHit</code> is a single enum delivered through one <code>FnMut(QueryHit)</code> callback — an earlier two-closure design (<code>on_found</code> + <code>on_value</code>) didn't borrow-check, since the caller's single mutable accumulator (<code>KmerResults</code>) can't be captured by two separate <code>FnMut</code> closures passed to the same call.
|
||||
- <code>obikmer/src/cmd/query.rs</code>: <code>KmerResults::set</code> (row-major, whole-row-at-once) replaced by <code>mark_found</code> (stage 1: flag a position as indexed, independent of any genome's value) and <code>set_one</code> (stage 2: write one genome's value at one position). <code>QueryStats</code> extended with <code>n_columns_scanned</code>/<code>n_col_get_calls</code>, logged per chunk.
|
||||
- Total <code>get()</code>-equivalent calls are unchanged from the row-major version (<code>n_hits × n_cols</code> in the worst case, confirmed by <code>n_col_get_calls</code> in the debug log) — the win is locality (sequential access within one layer's column at a time, across <code>mmap</code>'d regions, instead of jumping across all columns per hit), exactly as predicted.</p>
|
||||
<p><strong>What did not ship, and why — <code>PartitionRunner</code> is architecturally the wrong tool here:</strong>
|
||||
Reading <code>obikindex/src/numa.rs</code>'s actual <code>run()</code> body (not just its doc comments) shows every call spawns a timer thread <strong>plus one OS thread per worker slot on every NUMA node</strong> (<code>std::thread::scope</code> + one <code>s.spawn()</code> per node per <code>max_workers</code>) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned <strong>per call</strong>. This is fine for its actual, established usage in this codebase (<code>merge.rs</code>, <code>index.rs</code>'s <code>build_layers</code>): one <code>PartitionRunner::new()</code> + one <code>run()</code> call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for <code>query</code>'s call pattern: <code>query_partition_with</code> runs once per <code>(chunk, partition)</code>, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
|
||||
The column-major loop in stage 2 is therefore a <strong>plain sequential loop</strong> for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) <code>rayon</code>'s already-warm global pool (<code>into_par_iter()</code>) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern <code>numa_worker_pools.md</code> warns about for a <em>different</em> workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one <code>mmap</code>'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so <code>PartitionRunner</code> is invoked once per whole <code>query</code> run (or per large batch of chunks) rather than per <code>(chunk, partition)</code>, amortising its spawn cost the way <code>merge</code>/<code>build_layers</code> do — a bigger structural change than this phase's scope.
|
||||
- Log (implemented): <code>QueryStats::n_columns_scanned</code>/<code>n_col_get_calls</code>, folded into the existing per-chunk <code>debug!("k-mer dedup + column-major fetch", ...)</code> line (<code>query.rs</code>) alongside phase 3's dedup counters.
|
||||
- <strong>Unit tests</strong>: extended <code>obikpartitionner/src/tests/query_layer.rs</code> (phase 3's file) — <code>query_partition_with</code>'s empty/missing-index paths updated for the new <code>QueryStats</code> fields and single-callback signature.
|
||||
- <strong>Unit tests</strong>: extended <code>obikindex/src/partition/tests/query_layer.rs</code> (phase 3's file) — <code>query_partition_with</code>'s empty/missing-index paths updated for the new <code>QueryStats</code> fields and single-callback signature.
|
||||
- <strong>Validation performed</strong>: full workspace build + <code>cargo test --workspace</code>, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same <code>kmer_count</code>/<code>kmer_strict_matches</code> on every record); (2) the existing 20-genome <code>benchmark/global_index_presence</code> index — runs correctly, <code>n_hits=0</code> for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the <code>Implicit</code>/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) <strong>the critical correctness case</strong>: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from <code>genomeA</code> matched <strong>only</strong> <code>genomeA</code> (<code>kmer_count</code> identical to the pre-dedup occurrence count, zero leakage into <code>genomeB</code>'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in <code>n_cols</code>, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
|
||||
- <strong>Not yet done</strong>: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-<code>n_genomes</code> layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.</p>
|
||||
<h3 id="phase-5-sparse-findere-rework-roadmap-point-9">Phase 5 — Sparse Findere rework (roadmap point 9)</h3>
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1389,9 +1445,9 @@ IO pattern: sequential read of each source column → random write into new colu
|
||||
</table>
|
||||
<h2 id="files-to-modify">Files to modify</h2>
|
||||
<ul>
|
||||
<li><code>src/obikpartitionner/src/rebuild_layer.rs</code> — <code>rebuild_partition</code> and <code>iter_src_layers</code></li>
|
||||
<li><code>src/obikindex/src/partition/rebuild_layer.rs</code> — <code>rebuild_partition</code> and <code>iter_src_layers</code></li>
|
||||
<li>Possibly <code>src/obicompactvec/</code> — add column iterator API if not already present</li>
|
||||
<li><code>src/obilayeredmap/</code> — check if per-column sequential access is exposed on <code>SrcLayerData</code></li>
|
||||
<li><code>src/obikindex/src/layer/</code> — check if per-column sequential access is exposed on <code>SrcLayerData</code></li>
|
||||
</ul>
|
||||
<h2 id="open-questions">Open questions</h2>
|
||||
<ul>
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
|
||||
|
||||
|
||||
<link rel="prev" href="../../../implementation/obitaxonomy/">
|
||||
<link rel="prev" href="../../../implementation/partition_layer_cache/">
|
||||
|
||||
|
||||
<link rel="next" href="../../index_architecture/">
|
||||
@@ -731,14 +731,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -948,6 +948,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1084,7 +1140,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+699
-323
File diff suppressed because it is too large.
Load diff
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -405,11 +405,11 @@
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="../obilayeredmap/">
|
||||
<a class="md-nav__link" href="../layer_tier/">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -497,6 +497,30 @@
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="../benchmark_query_testing/">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="../partition_layer_cache/">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
@@ -555,7 +579,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1214,9 +1270,9 @@
|
||||
<h1 id="coverage-implementationevidence_eliminationmd">Coverage: implementation/evidence_elimination.md</h1>
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obilayeredmap/src/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()</li>
|
||||
<li><code>obilayeredmap/src/mphf_layer.rs</code> — build_approx_evidence(dir, b, z), find_approx()</li>
|
||||
<li><code>obilayeredmap/src/meta.rs</code> — EvidenceKind::Approx { b, z }, LayerMeta</li>
|
||||
<li><code>obikindex/src/layer/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()</li>
|
||||
<li><code>obikindex/src/layer/mphf_layer.rs</code> — build_approx_evidence(dir, b, z), find_approx()</li>
|
||||
<li><code>obikindex/src/layer/meta.rs</code> — EvidenceKind::Approx { b, z }, LayerMeta</li>
|
||||
<li><code>obikindex/src/reindex.rs</code> — KmerIndex::reindex(), conversion exact↔approx en place</li>
|
||||
<li><code>obikmer/src/cmd/reindex.rs</code> — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size</li>
|
||||
<li><code>obikmer/src/cmd/index.rs</code> — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp</li>
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
<link rel="prev" href="../unitig_evidence/">
|
||||
|
||||
|
||||
<link rel="next" href="../obilayeredmap/">
|
||||
<link rel="next" href="../layer_tier/">
|
||||
|
||||
|
||||
|
||||
@@ -868,14 +868,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1085,6 +1085,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1209,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -733,14 +733,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1208,6 +1208,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1332,7 +1388,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -2130,7 +2186,7 @@ obikmer<span class="w"> </span>phylo<span class="w"> </span>myindex<span class="
|
||||
<h2 id="implementation">Implementation</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong><code>obikpartitionner::filter::GroupQuorumFilter</code></strong> — implements <code>KmerFilter</code>
|
||||
<p><strong><code>obikindex::partition::filter::GroupQuorumFilter</code></strong> — implements <code>KmerFilter</code>
|
||||
using pre-computed ingroup and outgroup index vectors. The heavy logic
|
||||
(predicate parsing, three-value evaluation, genome classification) happens
|
||||
once before any iteration; each k-mer row evaluation is a simple index
|
||||
@@ -2143,7 +2199,7 @@ obikmer<span class="w"> </span>phylo<span class="w"> </span>myindex<span class="
|
||||
list.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong><code>obikpartitionner::KmerPartition::iter_partition_kmers</code></strong> — accepts
|
||||
<p><strong><code>obikindex::partition::KmerPartition::iter_partition_kmers</code></strong> — accepts
|
||||
<code>filters: &[Box<dyn KmerFilter>]</code> and applies them per-kmer before invoking
|
||||
the callback. <code>filter</code>, <code>dump</code>, and <code>unitig</code> all go through this single
|
||||
entry point.</p>
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -868,14 +868,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1085,6 +1085,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1209,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+69
-13
@@ -19,7 +19,7 @@
|
||||
|
||||
|
||||
|
||||
<title>Obilayeredmap.refs - obikmer</title>
|
||||
<title>Layer tier.refs - obikmer</title>
|
||||
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@
|
||||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#coverage-implementationobilayeredmapmd" class="md-skip">
|
||||
<a href="#coverage-implementationlayer_tiermd" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
@@ -96,7 +96,7 @@
|
||||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Obilayeredmap.refs
|
||||
Layer tier.refs
|
||||
|
||||
</span>
|
||||
</div>
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1211,15 +1267,15 @@
|
||||
|
||||
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
<h1 id="coverage-implementationobilayeredmapmd">Coverage: implementation/obilayeredmap.md</h1>
|
||||
<h1 id="coverage-implementationlayer_tiermd">Coverage: implementation/layer_tier.md</h1>
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obilayeredmap/src/mphf_layer.rs</code> — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()</li>
|
||||
<li><code>obilayeredmap/src/layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()</li>
|
||||
<li><code>obilayeredmap/src/map.rs</code> — LayeredMap<D>, push_layer(), query()</li>
|
||||
<li><code>obilayeredmap/src/evidence.rs</code> — Evidence, EvidenceWriter, encodage chunk_id:rank</li>
|
||||
<li><code>obilayeredmap/src/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, matches()</li>
|
||||
<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta, EvidenceKind (Exact / Approx { b, z })</li>
|
||||
<li><code>obikindex/src/layer/mphf_layer.rs</code> — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()</li>
|
||||
<li><code>obikindex/src/layer/typed_layer.rs</code> — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()</li>
|
||||
<li><code>obikindex/src/layer/map.rs</code> — LayeredMap<D>, push_layer(), query()</li>
|
||||
<li><code>obikindex/src/layer/evidence.rs</code> — Evidence, EvidenceWriter, encodage chunk_id:rank</li>
|
||||
<li><code>obikindex/src/layer/fingerprint.rs</code> — FingerprintVec, FingerprintVecWriter, matches()</li>
|
||||
<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta, EvidenceKind (Exact / Approx { b, z })</li>
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
<p>FORT RISQUE DE DÉRIVE. C'est le fichier le plus affecté par les changements récents :
|
||||
+218
-8
@@ -23,7 +23,7 @@
|
||||
|
||||
|
||||
|
||||
<title>obilayeredmap crate - obikmer</title>
|
||||
<title>obikindex layer tier - obikmer</title>
|
||||
|
||||
|
||||
|
||||
@@ -64,7 +64,7 @@
|
||||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#obilayeredmap-layered-kmer-index-crate" class="md-skip">
|
||||
<a href="#obikindexlayer-the-layer-tier" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
@@ -100,7 +100,7 @@
|
||||
<div class="md-header__topic" data-md-component="header-topic">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
</span>
|
||||
</div>
|
||||
@@ -749,7 +749,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -767,7 +767,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -940,6 +940,67 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#layerd-raw-mapping-iteration-and-batch-access" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Layer\<D> — raw mapping, iteration, and batch access
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Layer\<D> — raw mapping, iteration, and batch access">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#raw-kmer-slot-mapping" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Raw kmer → slot mapping
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#kmer-iteration" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Kmer iteration
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#batch-lookup-on-payload-vectorsviews" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Batch lookup on payload vectors/views
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sub_matrix-fill_sub_matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
sub_matrix / fill_sub_matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1263,6 +1324,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1387,7 +1504,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1638,6 +1755,67 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#layerd-raw-mapping-iteration-and-batch-access" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Layer\<D> — raw mapping, iteration, and batch access
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Layer\<D> — raw mapping, iteration, and batch access">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#raw-kmer-slot-mapping" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Raw kmer → slot mapping
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#kmer-iteration" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Kmer iteration
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#batch-lookup-on-payload-vectorsviews" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Batch lookup on payload vectors/views
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sub_matrix-fill_sub_matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
sub_matrix / fill_sub_matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1773,9 +1951,9 @@
|
||||
|
||||
|
||||
|
||||
<h1 id="obilayeredmap-layered-kmer-index-crate">obilayeredmap — layered kmer index crate</h1>
|
||||
<h1 id="obikindexlayer-the-layer-tier">obikindex::layer — the Layer tier</h1>
|
||||
<h2 id="purpose">Purpose</h2>
|
||||
<p><code>obilayeredmap</code> implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a <code>ptr_hash</code> MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.</p>
|
||||
<p><code>obikindex::layer</code> (the <code>layer/</code> submodule of the <code>obikindex</code> crate — a standalone <code>obilayeredmap</code> crate until 2026-08-21, folded back in alongside <code>obikpartition</code> as part of a broader <code>Index { Partition { Layer } }</code> submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a <code>ptr_hash</code> MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.</p>
|
||||
<hr />
|
||||
<h2 id="three-usage-modes">Three usage modes</h2>
|
||||
<p>The MPHF + evidence infrastructure is the same for all modes. The <strong>payload</strong> varies.</p>
|
||||
@@ -2000,6 +2178,38 @@
|
||||
</code></pre></div>
|
||||
<p>Mode 3 (<code>PersistentBitMatrix</code>) has no <code>push_layer</code> on <code>LayeredMap</code>; callers build directly via <code>Layer<PersistentBitMatrix>::build_presence</code>.</p>
|
||||
<hr />
|
||||
<h2 id="layerd-raw-mapping-iteration-and-batch-access">Layer\<D> — raw mapping, iteration, and batch access</h2>
|
||||
<p>Beyond <code>query</code>/<code>find</code> (membership-checked), <code>Layer<D></code> exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.</p>
|
||||
<h3 id="raw-kmer-slot-mapping">Raw kmer → slot mapping</h3>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">index</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kmer</span><span class="p">:</span><span class="w"> </span><span class="nc">CanonicalKmer</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="kt">usize</span>
|
||||
<span class="nc">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">index_batch</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kmers</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="n">CanonicalKmer</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span>
|
||||
</code></pre></div>
|
||||
<p>Pure MPHF mapping, no evidence/fingerprint check — equivalent to <code>MphfOnly::index</code>. Only meaningful when the caller already knows <code>kmer</code> belongs to the layer; on an absent kmer the MPHF still returns <em>some</em> slot (undefined, not <code>None</code>).</p>
|
||||
<h3 id="kmer-iteration">Kmer iteration</h3>
|
||||
<p>Four iterators, all built from <code>unitigs.bin</code> (physical layout order, <strong>not</strong> correlated with MPHF slot numbers):</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">iter_kmers</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">KmerIter</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">enumerate_kmers</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Enumerate</span><span class="o"><</span><span class="n">KmerIter</span><span class="o"><'</span><span class="nb">_</span><span class="o">>></span><span class="w"> </span><span class="c1">// (order_index, kmer)</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">iter_kmers_batch</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">KmerBatchIter</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span><span class="w"> </span><span class="c1">// Vec<CanonicalKmer> of size ≤ n</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">enumerate_kmers_batch</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">impl</span><span class="w"> </span><span class="nb">Iterator</span><span class="o"><</span><span class="n">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="n">CanonicalKmer</span><span class="o">></span><span class="p">)</span><span class="o">></span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">'</span><span class="nb">static</span>
|
||||
</code></pre></div>
|
||||
<p><code>KmerIter</code>/<code>KmerBatchIter</code> own a clone of the underlying <code>Arc<UnitigFileReader></code> rather than borrowing <code>self</code> — <code>Send + 'static</code>, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. <code>enumerate_kmers_batch</code>'s index is the batch's starting offset in iteration order (a multiple of <code>n</code> except for the final, possibly shorter, batch).</p>
|
||||
<h3 id="batch-lookup-on-payload-vectorsviews">Batch lookup on payload vectors/views</h3>
|
||||
<p><code>PersistentCompactIntVec</code>, <code>PersistentBitVec</code>, <code>IntSliceView</code>, <code>BitSliceView</code> all expose:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">get_batch</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="n">T</span><span class="o">></span>
|
||||
<span class="k">fn</span><span class="w"> </span><span class="nf">fill_batch</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="n">T</span><span class="p">])</span>
|
||||
</code></pre></div>
|
||||
<p>Both sort <code>slots</code> internally for sequential mmap access, then reorder results back to the caller's original order. <code>fill_batch</code> fills a caller-provided buffer, avoiding the <code>Vec</code> allocation.</p>
|
||||
<h3 id="sub_matrix-fill_sub_matrix">sub_matrix / fill_sub_matrix</h3>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// Layer<PersistentCompactIntMatrix></span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">u32</span><span class="o">>></span><span class="w"> </span><span class="c1">// column-first</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">])</span>
|
||||
|
||||
<span class="c1">// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">>></span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">></span><span class="p">])</span>
|
||||
</code></pre></div>
|
||||
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this goes through <code>nonzero_iter</code> instead, decoding one row at a time; see <a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats".</p>
|
||||
<hr />
|
||||
<h2 id="layeredstores-and-aggregation-traits">LayeredStore\<S> and aggregation traits</h2>
|
||||
<p><code>LayeredStore<S></code> is a generic aggregation wrapper over <code>Vec<S></code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayeredStore</span><span class="o"><</span><span class="n">S</span><span class="o">></span><span class="p">(</span><span class="k">pub</span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="n">S</span><span class="o">></span><span class="p">);</span>
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1215,9 +1271,9 @@
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obikindex/src/merge.rs</code> — <code>KmerIndex::merge()</code>, validation de compatibilité d'évidence, <code>validate_evidence_compat()</code></li>
|
||||
<li><code>obikpartitionner/src/merge_layer.rs</code> — <code>merge_partition()</code>, construction de la nouvelle layer, paramètre <code>block_bits</code></li>
|
||||
<li><code>obikpartitionner/src/rebuild_layer.rs</code> — <code>rebuild_partition()</code>, paramètre <code>block_bits</code></li>
|
||||
<li><code>obilayeredmap/src/layer.rs</code> — <code>Layer::append_genome_column()</code> (PersistentCompactIntMatrix et PersistentBitMatrix)</li>
|
||||
<li><code>obikindex/src/partition/merge_layer.rs</code> — <code>merge_partition()</code>, construction de la nouvelle layer, paramètre <code>block_bits</code></li>
|
||||
<li><code>obikindex/src/partition/rebuild_layer.rs</code> — <code>rebuild_partition()</code>, paramètre <code>block_bits</code></li>
|
||||
<li><code>obikindex/src/layer/typed_layer.rs</code> — <code>Layer::append_genome_column()</code> (PersistentCompactIntMatrix et PersistentBitMatrix)</li>
|
||||
<li><code>obicompactvec/src/intmatrix.rs</code> — <code>append_column</code> pour PersistentCompactIntMatrix</li>
|
||||
<li><code>obicompactvec/src/bitmatrix.rs</code> — <code>append_column</code> pour PersistentBitMatrix</li>
|
||||
</ul>
|
||||
|
||||
@@ -733,14 +733,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1034,6 +1034,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known issue (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1157,6 +1168,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1281,7 +1348,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1542,6 +1609,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known issue (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1739,6 +1817,14 @@ spectrums/
|
||||
<label>.json ← one file per genome, rebuilt from all sources
|
||||
index.meta ← complete genome list + evidence kind written at bootstrap
|
||||
</code></pre></div>
|
||||
<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
|
||||
<h2 id="known-issue-not-yet-fixed-2026-08-28">Known issue (not yet fixed, 2026-08-28)</h2>
|
||||
<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank < self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
|
||||
<ul>
|
||||
<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
|
||||
<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
|
||||
</ul>
|
||||
<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -733,14 +733,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1152,6 +1152,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1276,7 +1332,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1214,8 +1270,8 @@
|
||||
<h1 id="coverage-implementationmphfmd">Coverage: implementation/mphf.md</h1>
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obilayeredmap/src/mphf_layer.rs</code> — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, <code>build()</code>, <code>build_exact_evidence()</code>, <code>build_approx_evidence()</code>, <code>build_evidence()</code></li>
|
||||
<li><code>obikpartitionner/src/index_layer.rs</code> — <code>build_index_layer()</code> avec passage de <code>block_bits</code></li>
|
||||
<li><code>obikindex/src/layer/mphf_layer.rs</code> — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, <code>build()</code>, <code>build_exact_evidence()</code>, <code>build_approx_evidence()</code>, <code>build_evidence()</code></li>
|
||||
<li><code>obikindex/src/partition/index_layer.rs</code> — <code>build_index_layer()</code> avec passage de <code>block_bits</code></li>
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
<p>FORT RISQUE DE DÉRIVE. Changements récents :
|
||||
|
||||
@@ -946,14 +946,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1163,6 +1163,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1287,7 +1343,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1577,7 +1633,7 @@
|
||||
<h2 id="why-two-phases-are-needed">Why two phases are needed</h2>
|
||||
<p>Kmer indexing per partition proceeds in two phases. The separation is necessary because the exact number of surviving unique kmers is not known until after counting and filtering low-abundance kmers.</p>
|
||||
<h3 id="phase-1-provisional-mphf-kmer-spectrum">Phase 1 — provisional MPHF + kmer spectrum</h3>
|
||||
<p>Implemented in <code>obikpartitionner::KmerPartition::count_kmer()</code> → <code>count_partition()</code>.</p>
|
||||
<p>Implemented in <code>obikindex::partition::KmerPartition::count_kmer()</code> → <code>count_partition()</code>.</p>
|
||||
<ol>
|
||||
<li><strong>External sort</strong>: read the dereplicated superkmer file; extract the raw <code>u64</code> canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: <code>sorted_unique.bin</code> — a flat array of f0 distinct sorted <code>u64</code> values. Exact kmer count f0 is known at this point.</li>
|
||||
<li><strong>Build provisional MPHF</strong> (ptr_hash, same configuration as phase 2) over <code>sorted_unique.bin</code> using <code>new_from_par_iter</code>. Delete <code>sorted_unique.bin</code> immediately after. Persist to <code>mphf1.bin</code>.</li>
|
||||
@@ -1667,7 +1723,7 @@
|
||||
<hr />
|
||||
<h2 id="multilayer-index-architecture">Multilayer index architecture</h2>
|
||||
<h3 id="layer-structure">Layer structure</h3>
|
||||
<p>Each layer is a self-contained unit. See <a href="../obilayeredmap/">obilayeredmap</a> for the full on-disk layout. The MPHF-relevant files are:</p>
|
||||
<p>Each layer is a self-contained unit. See <a href="../layer_tier/">obikindex::layer</a> for the full on-disk layout. The MPHF-relevant files are:</p>
|
||||
<div class="highlight"><pre><span></span><code>layer_i/
|
||||
unitigs.bin — packed 2-bit nucleotide sequences (kmer evidence source)
|
||||
unitigs.bin.idx — random-access block index (block_bits controls granularity)
|
||||
@@ -1702,7 +1758,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
|
||||
Uses open_sequential(); never writes .idx
|
||||
</code></pre></div>
|
||||
<p>There is no <code>build_evidence</code> dispatch wrapper. Callers choose the appropriate post-hoc build directly.</p>
|
||||
<p>In <code>obikpartitionner</code>, <code>build_index_layer</code> receives <code>block_bits: u8</code> from <code>IndexConfig::block_bits</code> and forwards it directly to <code>Layer::build</code> and <code>Layer::build_approx_evidence</code>.</p>
|
||||
<p>In <code>obikindex::partition</code>, <code>build_index_layer</code> receives <code>block_bits: u8</code> from <code>IndexConfig::block_bits</code> and forwards it directly to <code>Layer::build</code> and <code>Layer::build_approx_evidence</code>.</p>
|
||||
<h3 id="membership-verification">Membership verification</h3>
|
||||
<p>ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry:</p>
|
||||
<ul>
|
||||
|
||||
@@ -320,11 +320,11 @@
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="../obilayeredmap/">
|
||||
<a class="md-nav__link" href="../layer_tier/">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -412,6 +412,30 @@
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="../benchmark_query_testing/">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="../partition_layer_cache/">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
@@ -470,7 +494,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -608,6 +632,46 @@
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#sparse-matrix-formats">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sparse matrix formats
|
||||
|
||||
</span>
|
||||
</a>
|
||||
<nav aria-label="Sparse matrix formats" class="md-nav">
|
||||
<ul class="md-nav__list">
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#persistentsparsebitmatrix">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentSparseBitMatrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#nonzero_iter-the-one-traversal-per-format">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
nonzero_iter — the one traversal per format
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#persistentsparsecompactintmatrix">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentSparseCompactIntMatrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#aggregation-traits-matrix-level">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
@@ -1144,29 +1208,40 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</code></pre></div>
|
||||
<hr/>
|
||||
<h2 id="matrix-types">Matrix types</h2>
|
||||
<p>Four matrix types, two encodings × two formats:</p>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th></th>
|
||||
<th>Columnar format</th>
|
||||
<th>Packed format</th>
|
||||
<th>Variant</th>
|
||||
<th>Storage</th>
|
||||
<th>When</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>Bit</strong></td>
|
||||
<td><code>PersistentBitMatrix</code> (Columnar variant)</td>
|
||||
<td><code>PersistentBitMatrix</code> (Packed variant)</td>
|
||||
<td><code>Columnar</code></td>
|
||||
<td>one <code>.pbiv</code>/<code>.pciv</code> file per column + <code>meta.json</code></td>
|
||||
<td>build-time default (<code>*Builder::new</code>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Int</strong></td>
|
||||
<td><code>PersistentCompactIntMatrix</code> (Columnar variant)</td>
|
||||
<td><code>PersistentCompactIntMatrix</code> (Packed variant)</td>
|
||||
<td><code>Packed</code></td>
|
||||
<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
|
||||
<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Sparse</code></td>
|
||||
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
||||
<td><code>pack --sparse</code>; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Implicit</code> (bit only)</td>
|
||||
<td>no file at all</td>
|
||||
<td>mono-genome presence layers — <code>n_cols</code> is always reported as <code>1</code>, every value is <code>true</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>Both matrix types are enums (<code>Columnar</code> / <code>Packed</code> / <code>Implicit</code> for bit) behind a transparent API. <code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||
|
||||
@@ -1176,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
|
||||
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
|
||||
<hr/>
|
||||
<h2 id="sparse-matrix-formats">Sparse matrix formats</h2>
|
||||
<p>Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||
k-mers — present in only a handful of genome columns out of dozens to
|
||||
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||
formats above, built once per layer like the rest of the pipeline (never the
|
||||
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||
Structures for Collections of Sets" (2025,
|
||||
<code>biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf</code>) — a plain dedup dictionary, not their full subset-containment
|
||||
hierarchy.</p>
|
||||
<h3 id="persistentsparsebitmatrix"><code>PersistentSparseBitMatrix</code></h3>
|
||||
<p>Four mmap-backed on-disk components: an <code>is_multi</code> rank-capable flag per row
|
||||
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||
(genome index directly, <code>ceil(log2(n_cols))</code> bits), a separate fixed-width
|
||||
array for multi-genome rows (<code>dict_id</code>, <code>ceil(log2(n_distinct_multi_sets))</code>
|
||||
bits — kept apart from the singleton array because <code>n_distinct_multi_sets</code>
|
||||
can be large in absolute terms even when multi-genome rows are a small
|
||||
<em>fraction</em> of all rows, and a shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||
blob). Supporting primitives: <code>PersistentFixedIntVec</code> (arbitrary,
|
||||
runtime-parameterised bit width, width <code>0</code> included — needed once an
|
||||
<code>EliasFano</code> reopen bug surfaced, see below), <code>PersistentRankSelectBitVec</code>
|
||||
(rank1/rank0/select1 on top of the crate's existing <code>count_ones</code>, via
|
||||
<code>common_traits::SelectInWord</code>), <code>EliasFano</code> (composes the two). <code>BinaryMatrix</code>
|
||||
(<code>n</code>, <code>n_cols</code>, <code>row</code>/<code>fill_row</code>, <code>fill_sub_matrix</code>, <code>count_ones</code>) is the
|
||||
trait unifying dense and sparse at the call sites that need both
|
||||
interchangeably; column-oriented methods (<code>col</code>, <code>col_view</code>, the
|
||||
<code>partial_*_dist_matrix</code> family) stay dense-only — <code>Sparse</code>/<code>Implicit</code> panic
|
||||
there (see the <code>PersistentBitMatrix::open</code> note above).</p>
|
||||
<p>Measured on real data (one layer, 30.2M rows, 91 genomes):</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th></th>
|
||||
<th>dense</th>
|
||||
<th>sparse</th>
|
||||
<th>ratio</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>on-disk size</td>
|
||||
<td>328.1MB</td>
|
||||
<td>43.7MB</td>
|
||||
<td><strong>7.5x</strong> smaller</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>build time / peak RSS</td>
|
||||
<td>—</td>
|
||||
<td>4.26s / 628MB</td>
|
||||
<td>per-layer, in-memory construction</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>row access, sequential (2M reads)</td>
|
||||
<td>43ns/row</td>
|
||||
<td>32ns/row</td>
|
||||
<td>sparse faster (smaller structure, better cache fit)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>row access, random (2M reads)</td>
|
||||
<td>409ns/row</td>
|
||||
<td>85ns/row</td>
|
||||
<td>sparse <strong>~4.8x faster</strong> — the <code>--shannon</code>/family-lookup access shape</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>column access, one full column (30.2M rows)</td>
|
||||
<td>11.5ms</td>
|
||||
<td>993ms</td>
|
||||
<td>sparse <strong>86x slower</strong> — no native column method, every read decodes a full row to keep one bit</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>The row-access wins (compactness was the stated design goal, not speed) come
|
||||
from dense's genome-major layout scattering a single row read across a much
|
||||
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||
is this much smaller. Column access is the flip side of the same layout
|
||||
choice; <code>nonzero_iter</code> (below) is what removes it from the hot path that
|
||||
actually matters (<code>query</code>).</p>
|
||||
<p>Two reopen-correctness bugs, now fixed at the source: (1) <code>EliasFano::open</code>
|
||||
re-derived its low-bits width from the persisted low-vector's own width
|
||||
byte; the width-<code>0</code> case was built with a dummy 1-bit placeholder (the
|
||||
builder used to reject true width <code>0</code>), silently doubling every reopened
|
||||
value. Fixed by making <code>PersistentFixedIntVec</code> genuinely support width <code>0</code>
|
||||
(no storage, <code>get</code> always <code>0</code>). (2) An empty row (cardinality 0) used to be
|
||||
recorded as a singleton at genome <code>0</code>, indistinguishable on read-back from a
|
||||
<em>real</em> singleton at genome <code>0</code>. Fixed by routing cardinality-0 rows through
|
||||
the dictionary path (a genuine empty entry).</p>
|
||||
<p>Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||
exact-duplicate case is built), a sparse <code>PersistentCompactIntMatrix</code>
|
||||
variant beyond what's described below, BRWT-style column-correlation
|
||||
exploitation, and a row-major rewrite of
|
||||
<code>partial_jaccard_dist_matrix</code>/<code>partial_hamming_dist_matrix</code>/etc. as a
|
||||
per-row co-occurrence accumulation (<code>O(Σ_rows k̄²)</code>) so <code>obikindex</code>'s
|
||||
<code>--metric</code> distance-matrix path stops paying the column-access penalty on
|
||||
<code>Sparse</code> — needs its own design pass (how it plugs into
|
||||
<code>BitPartials</code>/<code>ColumnWeights</code> so both matrix types keep serving <code>--metric</code>),
|
||||
not just "port the loop".</p>
|
||||
<h3 id="nonzero_iter-the-one-traversal-per-format"><code>nonzero_iter</code> — the one traversal per format</h3>
|
||||
<p><code>fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a></code>
|
||||
(item: index into <code>slots</code>, column, value) is the single foundational
|
||||
per-format traversal primitive every consumer builds on — <code>fill_sub_matrix</code>
|
||||
and a <code>for_each</code>-style callback are both trivial, free consumers of it
|
||||
(<code>.for_each(f)</code> is <code>Iterator::for_each</code>; <code>fill_sub_matrix</code> drains the
|
||||
iterator and scatters into <code>out[][]</code>), instead of two independently
|
||||
hand-maintained traversals that can silently diverge. <code>Box<dyn Iterator></code>,
|
||||
not a bare <code>impl Iterator</code>: <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are
|
||||
genuinely different concrete types and the method isn't on a
|
||||
dyn-compatible trait (<code>BinaryMatrix</code> is used as <code>dyn BinaryMatrix</code>
|
||||
elsewhere, and RPITIT methods aren't dyn-compatible) — one <code>Box</code> per
|
||||
<code>nonzero_iter</code> call, not per cell.</p>
|
||||
<p>Per-format implementation: <code>PersistentSparseBitMatrix</code> decodes natively,
|
||||
buffered one row at a time (<code>std::iter::from_fn</code> over the existing
|
||||
<code>for_each_genome_in_row</code>, no <code>n_cols</code>-wide allocation ever). <code>Columnar</code>/
|
||||
<code>Packed</code> loop columns, collecting each column's hits via
|
||||
<code>BitSliceView::nonzero_among_sorted</code>/<code>IntSliceView::nonzero_among_sorted</code>
|
||||
(a <code>filter</code>/<code>filter_map</code> over the vector-level <code>get()</code>, no new state
|
||||
machine — <code>std</code>'s own <code>Filter</code> iterator already is one) into one <code>Vec</code> via
|
||||
<code>.extend()</code> (not <code>flat_map</code>: a <code>flat_map</code> closure can't lazily return
|
||||
something borrowing its own captured sort permutation across separate calls
|
||||
without either boxing per-column or fighting the borrow checker — eager
|
||||
collection sidesteps it at zero extra cost, since <code>fill_sub_matrix</code> already
|
||||
fully materialises anyway). <code>Implicit</code> is trivial (<code>slots.iter().map(|&i| (i, 0, 1))</code>).</p>
|
||||
<p>This closed a real dispatch bug by construction, not by patching it:
|
||||
<code>PersistentBitMatrix::fill_sub_matrix</code>/<code>sub_matrix</code> used to reimplement a
|
||||
naive per-<code>(column, slot)</code> loop instead of delegating to
|
||||
<code>PersistentSparseBitMatrix</code>'s own efficient native method for <code>Self::Sparse</code>
|
||||
— two hand-written traversals for the same format, free to drift apart, and
|
||||
they had (<code>obikphylo</code>'s own cache layer had independently built a parallel
|
||||
enum wrapper that <em>did</em> dispatch correctly, evidence the bug was worked
|
||||
around rather than fixed at the source). With <code>fill_sub_matrix</code>/<code>sub_matrix</code>
|
||||
rewritten to drain <code>nonzero_iter</code>, there is only one traversal per format
|
||||
left to get right. <code>KmerPartition::query_partition_with</code>
|
||||
(<code>obikindex/src/partition/query_layer.rs</code>) — previously column-major
|
||||
(<code>for g { for slot { col_value } }</code>, which for <code>Sparse</code> rebuilt a full row
|
||||
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||
motivation for this primitive) — now issues one <code>layer.nonzero_iter(&slot_list)</code>
|
||||
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||
reproducibly; now within ~1-3% either way — noise level).</p>
|
||||
<h3 id="persistentsparsecompactintmatrix"><code>PersistentSparseCompactIntMatrix</code></h3>
|
||||
<p>The count-matrix analogue, built on <code>PersistentSparseBitMatrix</code> as its
|
||||
"which columns are non-zero" support, values <strong>not</strong> deduplicated (unlike
|
||||
the bit side, two rows can share the same non-zero column set — same
|
||||
<code>dict_id</code> — while carrying different counts, so the dict-multiplicity
|
||||
shortcut <code>BitPartials for PersistentSparseBitMatrix</code> uses doesn't carry
|
||||
over). <code>CountPartials</code> here: a single row-major pass (<code>row_major_pairwise</code>
|
||||
in <code>sparse_intmatrix.rs</code>, decodes each row once via
|
||||
<code>for_each_cell_in_row</code>, nests over that row's own co-present columns) —
|
||||
<code>O(Σ k̄²)</code> over populated rows, avoiding the naive <code>O(n_cols² × n)</code>
|
||||
column-pair rescan. <code>min(a,b)</code>-based kernels (<code>bray</code>, <code>relfreq_bray</code>) need
|
||||
no correction — they vanish when either side is absent. <code>a·b</code> and <code>√(a·b)</code>
|
||||
kernels (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) do need one, since
|
||||
<code>(a-0)² = a² ≠ 0</code> unlike the <code>min</code>-based formulas — reconstructed from
|
||||
per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>.
|
||||
<code>threshold_jaccard(1)</code> shortcuts to <code>BitPartials::partial_jaccard</code>
|
||||
(threshold 1 is exactly presence); <code>threshold_jaccard(0)</code> is closed-form
|
||||
(every <code>u32</code> is <code>≥ 0</code>).</p>
|
||||
<p>Wiring the <code>threshold==1</code> shortcut surfaced two pre-existing formula bugs in
|
||||
<code>BitPartials for PersistentSparseBitMatrix</code> (present since that type's own
|
||||
introduction, never caught because no test had compared <code>Sparse</code>'s raw
|
||||
<code>partial_*</code> output against dense on real data — only the diagonal-blind
|
||||
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested),
|
||||
fixed at the source:</p>
|
||||
<ol>
|
||||
<li><code>partial_jaccard</code>'s diagonal used to be <code>(0, 2×col_weights[i])</code> instead
|
||||
of a genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
|
||||
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with itself
|
||||
by construction, so the diagonal needs its own case.</li>
|
||||
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: it computed
|
||||
<code>total - union</code> (rows where <em>neither</em> column is present) instead of the
|
||||
actual Hamming distance <code>col_weights[i] + col_weights[j] - 2×inter[i,j]</code>
|
||||
(symmetric-difference size) — the two only coincide when
|
||||
<code>col_weights[i] + col_weights[j] == total</code>, which small/synthetic test
|
||||
data can easily satisfy by accident.</li>
|
||||
</ol>
|
||||
<h2 id="aggregation-traits-matrix-level">Aggregation traits (matrix level)</h2>
|
||||
<h3 id="columnweights">ColumnWeights</h3>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">trait</span><span class="w"> </span><span class="n">ColumnWeights</span><span class="p">:</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="p">{</span>
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1219,7 +1275,7 @@
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
<p>Document stable (librairie générique, peu de risque de dérive).
|
||||
Vérifier si <code>obipipeline</code> est toujours utilisé dans la phase scatter de <code>obikpartitionner</code>
|
||||
Vérifier si <code>obipipeline</code> est toujours utilisé dans la phase scatter de <code>obikindex::partition</code>
|
||||
ou s'il a été remplacé par Rayon dans certains chemins.</p>
|
||||
|
||||
|
||||
|
||||
@@ -896,14 +896,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1113,6 +1113,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1237,7 +1293,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
<link rel="prev" href="../select/">
|
||||
|
||||
|
||||
<link rel="next" href="../../architecture/sequences/invariant/">
|
||||
<link rel="next" href="../benchmark_query_testing/">
|
||||
|
||||
|
||||
|
||||
@@ -733,14 +733,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1125,6 +1125,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1249,7 +1305,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -733,14 +733,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1225,6 +1225,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1349,7 +1405,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
|
||||
|
||||
|
||||
<link rel="prev" href="../obilayeredmap/">
|
||||
<link rel="prev" href="../layer_tier/">
|
||||
|
||||
|
||||
<link rel="next" href="../persistent_bit_vec/">
|
||||
@@ -733,14 +733,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1208,6 +1208,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1332,7 +1388,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1214,12 +1270,12 @@
|
||||
<h1 id="coverage-implementationpipelinemd">Coverage: implementation/pipeline.md</h1>
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obikpartitionner/src/partition.rs</code> — estimation des paramètres (phase 0)</li>
|
||||
<li><code>obikindex/src/partition/partition.rs</code> — estimation des paramètres (phase 0)</li>
|
||||
<li><code>obiskbuilder/src/iter.rs</code> — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)</li>
|
||||
<li><code>obikpartitionner/src/filter.rs</code> — déduplication bucket-sort (phase 2)</li>
|
||||
<li><code>obikpartitionner/src/kmer_sort.rs</code> — tri externe + agrégation de comptages (phase 3)</li>
|
||||
<li><code>obikindex/src/partition/filter.rs</code> — déduplication bucket-sort (phase 2)</li>
|
||||
<li><code>obikindex/src/partition/kmer_sort.rs</code> — tri externe + agrégation de comptages (phase 3)</li>
|
||||
<li><code>obidebruinj/src/debruijn.rs</code> — graphe De Bruijn, extraction des unitigs (phase 5)</li>
|
||||
<li><code>obikpartitionner/src/index_layer.rs</code> — construction MPHF + évidence (phase 6), paramètre <code>block_bits</code></li>
|
||||
<li><code>obikindex/src/partition/index_layer.rs</code> — construction MPHF + évidence (phase 6), paramètre <code>block_bits</code></li>
|
||||
<li><code>obikindex/src/index.rs</code> — <code>build_layers()</code>, <code>dereplicate_and_count()</code></li>
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
|
||||
@@ -857,14 +857,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1074,6 +1074,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1198,7 +1254,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1546,7 +1602,7 @@ branching / dead-end → unitig start or end
|
||||
counts/ ← PersistentCompactIntMatrix (with_counts only)
|
||||
</code></pre></div>
|
||||
<p><strong>Cleanup:</strong> unless <code>--keep-intermediate</code> is set, <code>remove_build_artifacts</code> deletes <code>dereplicated.skmer.zst</code>, <code>mphf1.bin</code>, and <code>counts1.bin</code> after all partitions are indexed.</p>
|
||||
<p>See <a href="../obilayeredmap/">obilayeredmap</a> and <a href="../mphf/">MPHF selection</a> for data structure details.</p>
|
||||
<p>See <a href="../layer_tier/">obikindex::layer</a> and <a href="../mphf/">MPHF selection</a> for data structure details.</p>
|
||||
<p><strong>Query path (exact evidence):</strong></p>
|
||||
<div class="highlight"><pre><span></span><code>query kmer q
|
||||
→ canonical_minimizer(q) → hash → PART → part_XXXXX/
|
||||
|
||||
@@ -733,14 +733,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1119,17 +1119,6 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#in-place-keep-only-group-a" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
In-place: keep only group A
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1157,6 +1146,23 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Implementation notes">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known gap (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1196,6 +1202,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1320,7 +1382,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1582,17 +1644,6 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#in-place-keep-only-group-a" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
In-place: keep only group A
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1620,6 +1671,23 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Implementation notes">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Known gap (not yet fixed, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1672,23 +1740,32 @@ are preserved unchanged; only the data matrices are rewritten.</p>
|
||||
<hr />
|
||||
<h2 id="synopsis">Synopsis</h2>
|
||||
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span><input-index>
|
||||
<span class="w"> </span><span class="o">{</span><span class="w"> </span>--output<span class="w"> </span><dir><span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="w"> </span><span class="o">}</span>
|
||||
<span class="w"> </span>--output<span class="w"> </span><dir>
|
||||
<span class="w"> </span><span class="o">[</span>--group<span class="w"> </span><name>:<pred><span class="w"> </span>...<span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--group-op<span class="w"> </span><name>:<op><span class="w"> </span>...<span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--aggregate-by<span class="w"> </span><key><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--aggregate-op<span class="w"> </span><op><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--select<span class="w"> </span><col1,col2,...><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--presence-threshold<span class="w"> </span><N><span class="w"> </span><span class="o">]</span>
|
||||
<span class="w"> </span><span class="o">[</span>--force-copy<span class="w"> </span><span class="o">]</span>
|
||||
</code></pre></div>
|
||||
<hr />
|
||||
<h2 id="output-destination">Output destination</h2>
|
||||
<p>Exactly one of <code>--output</code> or <code>--in-place</code> must be specified.</p>
|
||||
<p><strong><code>--output <dir></code></strong> — writes a new index to <code><dir></code>. The source index is
|
||||
unchanged. The MPHF and unitig files are copied; only the data matrices are
|
||||
rewritten with the new column layout.</p>
|
||||
<p><strong><code>--in-place</code></strong> — rewrites the data matrices of the source index directly.
|
||||
Removed or replaced columns are lost. The operation writes to temporary files
|
||||
first, then renames atomically, so an interrupted run leaves the index intact.</p>
|
||||
<p><code>--output <dir></code> is required — <code>select</code> always writes a new index; there is no
|
||||
<code>--in-place</code> mode (2026-08-28: never implemented, removed from the design). The
|
||||
source index is unchanged.</p>
|
||||
<p>Each layer's kmer-identity files (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/
|
||||
<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code>) are never rewritten by a
|
||||
column projection/aggregation, so they are hard-linked into the output rather
|
||||
than copied — no extra disk for them even on a large index. Falls back to a
|
||||
real copy automatically if linking fails (different filesystems); <code>--force-copy</code>
|
||||
forces a real copy always, for an output that must survive independently of the
|
||||
source on disk (a hard link shares the same inode — rewriting one path outside
|
||||
<code>select</code> itself would affect the other). Only the <code>presence</code>/<code>counts</code>
|
||||
subdirectory is ever a genuinely new, independent file.</p>
|
||||
<p>To replace an index with a selected version of itself, select to a temporary
|
||||
directory and swap it in (<code>rm -rf INDEX && mv INDEX.tmp INDEX</code>) — the case
|
||||
<code>--in-place</code> used to cover.</p>
|
||||
<hr />
|
||||
<h2 id="defining-output-columns">Defining output columns</h2>
|
||||
<h3 id="named-groups-group">Named groups — <code>--group</code></h3>
|
||||
@@ -1852,9 +1929,6 @@ is > N.</p>
|
||||
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>out<span class="w"> </span><span class="se">\</span>
|
||||
<span class="w"> </span>--select<span class="w"> </span><span class="s2">"Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"</span>
|
||||
</code></pre></div>
|
||||
<h3 id="in-place-keep-only-group-a">In-place: keep only group A</h3>
|
||||
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--in-place<span class="w"> </span>--group<span class="w"> </span><span class="s2">"A:group=A"</span><span class="w"> </span>--select<span class="w"> </span><span class="s2">"A"</span>
|
||||
</code></pre></div>
|
||||
<h3 id="compose-with-filter">Compose with filter</h3>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1"># Step 1: keep only B. nana-specific k-mers</span>
|
||||
obikmer<span class="w"> </span>filter<span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>filtered<span class="w"> </span><span class="se">\</span>
|
||||
@@ -1865,32 +1939,46 @@ obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </s
|
||||
</code></pre></div>
|
||||
<hr />
|
||||
<h2 id="implementation-notes">Implementation notes</h2>
|
||||
<p><code>select</code> does not rebuild the MPHF. The 256 partitions are processed in parallel
|
||||
(rayon), each writing its output independently; results require no synchronisation
|
||||
because every partition owns a distinct set of files.</p>
|
||||
<p>For each layer in each partition:</p>
|
||||
<p><code>select</code> does not rebuild the MPHF. Every partition is processed independently
|
||||
(<code>PartitionRunner</code>), each writing its own output layers; no cross-partition
|
||||
synchronisation is needed.</p>
|
||||
<p>For each layer in each partition (<code>obikselect::select_layer::select_partition</code>):</p>
|
||||
<ol>
|
||||
<li>The slot count <code>n</code> is read by opening the source data matrix.</li>
|
||||
<li>A new data matrix is built with M columns (M = number of output columns).</li>
|
||||
<li>For each slot <code>s</code> in <code>0..n</code>:</li>
|
||||
<li><code>old_row = matrix.fill_row(s)</code> — reads the original <code>N</code>-column row without allocating.</li>
|
||||
<li>For each output column <code>j</code>:<ul>
|
||||
<li><code>new_row[j] = aggregate(op, old_row[group_indices])</code>.</li>
|
||||
<li>Pass-through columns are represented as single-element groups with the
|
||||
default operator (<code>any</code> for presence, <code>sum</code> for count) — same code path.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>The new row is written slot by slot into each column builder.</li>
|
||||
<li>All plain files in the source layer directory (<code>mphf.bin</code>, <code>unitigs.bin</code>,
|
||||
evidence files, <code>layer_meta.json</code>) are copied verbatim; only the <code>presence/</code>
|
||||
or <code>counts/</code> subdirectory is rewritten.</li>
|
||||
<li><code>copy_layer_files</code> hard-links the source layer's kmer-identity files
|
||||
(<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/
|
||||
<code>fingerprint.bin</code>/<code>layer_meta.json</code>) into the destination — never a real
|
||||
copy unless linking fails or <code>--force-copy</code> is given.</li>
|
||||
<li>A new data matrix is built with M columns (M = number of output columns),
|
||||
under a fresh <code>presence/</code>/<code>counts/</code> subdirectory (never touching the
|
||||
source's own).</li>
|
||||
<li><strong>Presence source (2026-08-28: <code>batch_presence_counts</code>)</strong>: one shared pass
|
||||
over the source bit matrix computes every output group's presence count at
|
||||
once — row-major native for a <code>Sparse</code> source (<code>for_each_genome_in_row</code>,
|
||||
which has no column representation to read a <code>col_view</code> from at all — the
|
||||
reason this replaced the old per-group loop, not just an optimisation of
|
||||
it), deduplicated column-major (one <code>col_view</code> per <em>distinct referenced
|
||||
column</em>, not per group) for <code>Columnar</code>/<code>Packed</code>. Every <code>AggOp</code> for a bit
|
||||
matrix is then a cheap derivation of that one count vector (<code>sum</code> = the
|
||||
count itself, <code>any</code>/<code>max</code> = <code>count ≥ 1</code>, <code>all</code>/<code>min</code> = <code>count == group
|
||||
size</code>, <code>none</code> = <code>count == 0</code>) — see
|
||||
<code>obikselect::select_layer::agg_result_from_count</code>.</li>
|
||||
<li><strong>Count source</strong>: unchanged, one <code>col_view</code>-driven pass per output column
|
||||
via <code>MatrixGroupOps</code> — <code>sum</code>/<code>min</code>/<code>max</code> are genuine per-value reductions
|
||||
for a count matrix, not derivable from a single presence count the way
|
||||
they are for a bit matrix.</li>
|
||||
<li><code>index.meta</code> is rewritten with the new genome list and updated <code>with_counts</code>.</li>
|
||||
</ol>
|
||||
<p><strong><code>--in-place</code> write strategy:</strong> new data is written to a temporary sibling
|
||||
directory (<code>presence_new/</code> or <code>counts_new/</code>); on success the old directory is
|
||||
removed and the temporary one is renamed into place. An interrupted run leaves
|
||||
at most one stale <code>*_new/</code> directory; the original data is intact until the
|
||||
rename step.</p>
|
||||
<h3 id="known-gap-not-yet-fixed-2026-08-28">Known gap (not yet fixed, 2026-08-28)</h3>
|
||||
<p>Step 4 above still panics (<code>col_view() not available on Sparse
|
||||
PersistentCompactIntMatrix</code>) if the source is a <strong>count</strong> index packed
|
||||
sparse — <code>batch_presence_counts</code>' row-major treatment was only ported to the
|
||||
bit-matrix (<code>Presence</code>) case, since that was the one actually blocking a real
|
||||
benchmark run. <code>select</code>/<code>filter</code> on a sparse-packed count index still hits
|
||||
this; the fix would follow the same shape (a <code>PersistentSparseCompactIntMatrix</code>
|
||||
row-major decode, analogous to <code>for_each_genome_in_row</code>), just not done. Since
|
||||
<code>obisys::numa::runner::PartitionRunner</code>'s panic-propagation fix (see
|
||||
<code>architecture/numa_partition_runner.md</code>), this at least fails fast (process
|
||||
panic, exit 101) instead of hanging.</p>
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1216,7 +1272,7 @@
|
||||
<ul>
|
||||
<li><code>obikindex/src/meta.rs</code> — IndexMeta, IndexConfig (version, config, genomes)</li>
|
||||
<li><code>obikindex/src/index.rs</code> — layout sur disque : partitions/, index.meta</li>
|
||||
<li><code>obilayeredmap/src/meta.rs</code> — LayerMeta (evidence kind), PartitionMeta (n_layers)</li>
|
||||
<li><code>obikindex/src/layer/meta.rs</code> — LayerMeta (evidence kind), PartitionMeta (n_layers)</li>
|
||||
<li><code>obiskio/src/unitig_index.rs</code> — fichiers unitigs.bin + unitigs.bin.idx</li>
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
|
||||
@@ -929,14 +929,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1146,6 +1146,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1270,7 +1326,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -835,14 +835,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1052,6 +1052,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1176,7 +1232,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1215,7 +1271,7 @@
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obiskio/src/unitig_index.rs</code> — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()</li>
|
||||
<li><code>obilayeredmap/src/evidence.rs</code> — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter</li>
|
||||
<li><code>obikindex/src/layer/evidence.rs</code> — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter</li>
|
||||
<li><code>obidebruinj/src/debruijn.rs</code> — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK</li>
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
|
||||
@@ -957,14 +957,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../obilayeredmap/" class="md-nav__link">
|
||||
<a href="../layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1174,6 +1174,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1298,7 +1354,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+59
-3
@@ -831,14 +831,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1048,6 +1048,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1172,7 +1228,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -917,14 +917,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1134,6 +1134,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1258,7 +1314,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+59
-3
@@ -830,14 +830,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1047,6 +1047,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1171,7 +1227,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -802,14 +802,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1019,6 +1019,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1143,7 +1199,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -868,14 +868,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1085,6 +1085,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1209,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1214,8 +1270,8 @@
|
||||
<h1 id="coverage-theoryindexingmd">Coverage: theory/indexing.md</h1>
|
||||
<h2 id="code-couvert">Code couvert</h2>
|
||||
<ul>
|
||||
<li><code>obikpartitionner/src/partition.rs</code> — routage par hash de minimiseur, choix des paramètres</li>
|
||||
<li><code>obikpartitionner/src/lib.rs</code> — structure KmerPartition, nombre de partitions</li>
|
||||
<li><code>obikindex/src/partition/partition.rs</code> — routage par hash de minimiseur, choix des paramètres</li>
|
||||
<li><code>obikindex/src/partition/mod.rs</code> — structure KmerPartition, nombre de partitions</li>
|
||||
</ul>
|
||||
<h2 id="notes">Notes</h2>
|
||||
<p>Vérifier que la doc mentionne bien que le nombre de partitions est une puissance de 2
|
||||
|
||||
@@ -802,14 +802,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1019,6 +1019,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1143,7 +1199,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -727,14 +727,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -944,6 +944,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1068,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -846,14 +846,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/obilayeredmap/" class="md-nav__link">
|
||||
<a href="../../implementation/layer_tier/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
obilayeredmap crate
|
||||
obikindex layer tier
|
||||
|
||||
|
||||
|
||||
@@ -1063,6 +1063,62 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/benchmark_query_testing/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Benchmark: query-path testing
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../implementation/partition_layer_cache/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Partition and layer caching (discussion)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1187,7 +1243,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Hot resume and memoization of long computations (discussion, 2026-10-14)
|
||||
|
||||
## Goal (user)
|
||||
|
||||
1. **Hot resume**: a run killed midway (crash, `Ctrl-C`, scheduler walltime) restarts from the last finished unit of work, not from zero.
|
||||
2. **No recomputation**: a long function called again with the same parameters (same command rerun, or another command needing the same intermediate) reuses its stored result.
|
||||
|
||||
Not a goal: bit-identical resumed samples. Sampling uses an OS-seeded RNG on purpose, repeated runs must stay independent draws to measure variance; a resumed run being a *different* valid draw is not a defect.
|
||||
|
||||
## Why it matters: one real run
|
||||
|
||||
`phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance --family-overlap --sibling-stats` on the 180-genome Primulaceae index, from the stderr of the OAR job (2026-10-13), 256 partitions:
|
||||
|
||||
| Stage | Wall |
|
||||
|---|---|
|
||||
| `sibling_stats` (full scan, `Selection::All`) | 8 235 s |
|
||||
| `family_overlap` (sweep 1: overlap annex) | 5 582 s |
|
||||
| `sampled_scan`: sweep 1 again (entropy + quartet annexes) | ≈ 5 347 s |
|
||||
| `sampled_scan`: sweep 2 (pair-weight annex) | ≈ 5 296 s |
|
||||
| `sampled_scan`: the draw itself | ≈ 9 573 s |
|
||||
| `distance` (Jaccard, not asked for; older binary) | 53 s |
|
||||
| Total | 34 085 s (9 h 28) |
|
||||
|
||||
The three annex sweeps cost about the same although their per-family work differs, so the cost is the layer traversal. Part of it was addressed by restricting the matrix pass to selected families (`family_scan.rs`, same session); the sweeps were not fused. Nothing in this run could be resumed or reused: every stage restarts from its beginning.
|
||||
|
||||
## What exists today (verified in the code, 2026-10-14)
|
||||
|
||||
**Persisted per-layer annexes.** Entropy (`entropy*.pent`), quartet (`quartet*.pbiv`), pair weight, sibling annex: one file per layer, built by `ensure_layer_*` and skipped when `path.exists()` (`family_annexes.rs`). A rerun with the same conventions skips every sweep. This is already memoization, keyed by *file name only* (convention in the name, no fingerprint of the inputs; a rebuilt sibling annex deletes everything derived from it, see `pair_coverage_sampling.md`).
|
||||
|
||||
**Not persisted.** The family-overlap accumulator (`FamilyOverlapAccumulator`) is written once at the end of sweep 1 (`sibling_builder.rs`). The draw (`sample_index`) and the `sibling_stats` scan keep their state in memory only.
|
||||
|
||||
**Defect found, then fixed (2026-10-14): a killed annex build left a file that looked finished.** `create_mmap_file` (`obicompactvec/src/mmap_file.rs`) created the file at its *final path* with magic, header and the full length (`set_len`) before any data was written. A kill during a layer left a valid-looking file with unwritten data; the next run saw `path.exists()` and treated the annex as built, silently.
|
||||
|
||||
Reproduced on the 1x testbed (20 genomes, 512 layers): derived annexes deleted, `phylo --family-overlap --quartet --pair-balance --entropy 1 …` killed by `timeout 6`, then rerun. Old binary: 141 finals after the kill, 2 differing from a reference build; the rerun exited 0 and kept both. Fixed binary: kill at 6 s left 499 finals, all identical to the reference, plus 2 `.building` files; the rerun completed 1 537 files identical to the reference, no `.building` left. (Kill at 3 s: nothing published, nothing broken.)
|
||||
|
||||
**Fix.** `obicompactvec::building_path` / `publish_built`: a file is built under `<name>.building` and moved to its final name (same-directory `rename`, atomic) by the builder's `close()`, after the flush. Applied to the builders that go through `create_mmap_file` or write their own mmap on the annex path: `PersistentBitVecBuilder` (all constructors; quartet annex), `PersistentFixedIntVecBuilder`, `PersistentRankSelectBitVecBuilder`, `FamilyScalarAnnexBuilder` (entropy, pair weight), `SiblingAnnexBuilder` (`siblings.psib`) and `FamilyOverlapAccumulator::write` (overlap matrix). A leftover `.building` is truncated by the next build of that file. A build that is dropped without `close()` no longer produces the final file (no caller did; workspace tests pass). Also reordered in `--sibling-annex`: derived annexes are now removed *before* the new sibling annex is built and published, so a kill between the two leaves the old annex with no derived files, never a new annex next to files computed against the old numbering.
|
||||
|
||||
**Not covered.** The builders outside that path still write at their final name: `PersistentCompactIntVecBuilder`, the int/sparse matrix builders, `EliasFanoBuilder`, `TempBitVecBuilder`/`TempCompactIntVecBuilder` (temporary by nature) and the index-construction files that use them. The same reasoning applies to them as soon as a stage built on them has to be resumed; not audited. The in-memory overlap accumulator is still lost on a kill (a sweep restarts), now without leaving a corrupt file.
|
||||
|
||||
**`obiksession` crate** (`src/obiksession/`): `Session` (named directory, opaque parameter blob checked on open, whole-lifetime `DirLock`, artifacts checksummed with xxh3 and mmap-restored) and `Memoize` (`get_or_compute[_with_key]`, `invalidate`, `cached_keys`, `MemoKey` from `Debug` + xxh3, the `session_memoize!` macro). Compiles, 15 tests pass, **no caller**: `obikmer` lists the dependency without using it. `phylo --session` exits with an error (`cmd/phylo/mod.rs:271`) since `session_cache.rs` (the consumer tied to `PairwiseTally`/`PartitionDispersion`/`SnpAlignment`) was removed. `sample_index(skip_layers)` remains as a seam, never called with a non-zero value.
|
||||
|
||||
**Previous design and why it missed the goal** (`theory/evolutionary_distances.md`, section "Explicit `--session`"): one session per *exact* parameter tuple (changing `--n-sites` made another session), artifacts specific to the sampling result, resume only inside the draw (checkpoint every 8 layers or 30 s). Reuse across commands existed only for the sample. An exhaustive-sample cache from which smaller samples would be drawn was proposed and never done. User judgement: this version did not answer the objective.
|
||||
|
||||
## Proposed direction (not implemented; to be validated)
|
||||
|
||||
Start from the goal, not from the existing crate.
|
||||
|
||||
1. **Unit of work = one `(partition, layer)` step that already produces files.** The annex builds, the draw per layer and the scans all iterate that way (`for part … for layer …`). Resume means "skip the units whose output is complete", exactly what `exists()` approximates today.
|
||||
2. **Complete-or-absent outputs.** Done for the annex path (see the fix above); the remaining builders are to be covered as their stages become resumable. Prerequisite for everything below. The mechanism lives in `obicompactvec` (shared lower-level API), not in each consumer.
|
||||
3. **Key = function identity + parameters + fingerprint of the inputs.** A result is reusable iff the function, every parameter that influences it and the identity of its inputs (index content, upstream artifacts) match. Today the key is a file name plus an invalidation convention. The fingerprint requirement is the cost: what identifies an index version cheaply is to be decided (meta file, rebuild counter, hash of annex headers).
|
||||
4. **The cache holds results of long functions, not domain objects.** The crate must not know `PairwiseTally`. Callers serialize their own values (`rkyv` was already chosen for that). Whether `Memoize`'s API fits is unproven: it has no caller, no documentation and was written against the earlier design.
|
||||
5. **Granularity matters more than the API.** A cached 9 h result helps only the exact rerun; a per-layer cache also helps a changed parameter that leaves earlier layers untouched. The overlap accumulator (end-of-sweep write) would need a per-layer partial (a sum of per-layer matrices, mergeable) to be resumable.
|
||||
6. **Fuse before caching.** Three sweeps over the same layers where two suffice (overlap first, then entropy + quartet + pair weight together, since the pair weight needs only the finished overlap matrix) is a cost no cache removes on a first run; to do independently.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Fingerprint of an index version: which cheap, stable identity?
|
||||
- Overlap accumulator resume: per-layer partial matrices on disk (size n² × layers) or a checkpoint every N layers?
|
||||
- `Memoize`/`Session`: keep (adapt to the points above), rewrite, or delete the crate until a caller exists (the "existing code is a hypothesis" rule applies; today it has no user).
|
||||
- Scope of the next step: atomic writes are done for the annex path; extend to the other builders, or go to the cache.
|
||||
- Draw resume: a layer's draw is independent of the others except through the quotas, which depend on every layer's mass; quotas can be recomputed from annexes on resume (cheap), so only the per-layer survivors need storing.
|
||||
@@ -3,10 +3,10 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix
|
||||
- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum
|
||||
- `obilayeredmap/src/map.rs` — LayeredMap<D>
|
||||
- `obilayeredmap/src/meta.rs` — LayerMeta, PartitionMeta
|
||||
- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix
|
||||
- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum
|
||||
- `obikindex/src/layer/map.rs` — LayeredMap<D>
|
||||
- `obikindex/src/layer/meta.rs` — LayerMeta, PartitionMeta
|
||||
- `obikindex/src/meta.rs` — IndexConfig (kmer_size, n_bits, with_counts, evidence, block_bits), IndexMeta
|
||||
- `obikindex/src/index.rs` — KmerIndex, build_layers
|
||||
- `obicompactvec/src/` — PersistentCompactIntMatrix, PersistentBitMatrix (DataStore implementations)
|
||||
|
||||
@@ -304,10 +304,29 @@ step, capped by that node's remaining dormant workers and by the run's total
|
||||
budget (`n_total`) — balance across nodes is now guaranteed by construction,
|
||||
not incidental to channel implementation details.
|
||||
|
||||
## Panic propagation (fixed, 2026-08-28)
|
||||
|
||||
A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
|
||||
— the controller's `while completed < n_total` loop then waited forever for
|
||||
an event that partition could no longer produce, since other live
|
||||
workers/timer threads kept `event_rx` open. Discovered via `obikselect`
|
||||
panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
|
||||
the process hung indefinitely instead of erroring out.
|
||||
|
||||
Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
|
||||
`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
|
||||
partition. The controller counts it toward `completed` (unblocking the loop)
|
||||
and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
|
||||
re-raises it on the caller's thread — the original message/backtrace still
|
||||
surfaces, from the right place, instead of a silent deadlock. Takes priority
|
||||
over a plain `Err` (a panic means a bug, not a normal typed failure).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Error handling**: `run` currently returns the first error; remaining errors
|
||||
are dropped. A `Vec<E>` return would give complete diagnostics.
|
||||
are dropped. A `Vec<E>` return would give complete diagnostics. (Panics are
|
||||
now caught and re-raised individually — see above — this only concerns
|
||||
plain `Err` results.)
|
||||
|
||||
- **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
|
||||
(start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an
|
||||
|
||||
@@ -72,9 +72,9 @@ Each NUMA group runs its own independent adaptive pool. Workers are distributed
|
||||
|------|--------|
|
||||
| `obikindex/src/merge.rs` | Detect NUMA topology; build N `ThreadPool`s with pinned threads; assign each pre-spawned worker to a pool; wrap `merge_partition` in `pool.install()` |
|
||||
| `obikindex/src/merge.rs` | Replace `available_parallelism()` with per-NUMA core count for spawn criterion |
|
||||
| `obikpartitionner/src/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
|
||||
| `obikindex/src/partition/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
|
||||
| `obidebruinj/src/debruijn.rs` | No change — `par_iter` and `current_num_threads` are pool-context-aware |
|
||||
| `obikpartitionner/src/partition.rs` | No change — same reason |
|
||||
| `obikindex/src/partition/partition.rs` | No change — same reason |
|
||||
|
||||
## Platform guard
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Given a set of query sequences, determine for each sequence how many of its k-me
|
||||
|
||||
## Algorithm
|
||||
|
||||
The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikpartitionner::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
|
||||
The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikindex::partition::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
|
||||
|
||||
```
|
||||
for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
|
||||
@@ -24,7 +24,7 @@ for each chunk of sequences (parallel workers via obipipeline, one call to proce
|
||||
decompose all sequences into superkmers (SuperKmerIter) — construction only,
|
||||
not the dedup key
|
||||
deduplicate at k-mer granularity, split by partition in the same pass:
|
||||
by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>> ← KmerDesc = (seq_idx, pos)
|
||||
by_partition: Vec<PartitionKmers> ← sorted unique k-mers + flat KmerDesc = (seq_idx, pos) slices (2026-10-15, was a HashMap — see "Query memory" below)
|
||||
allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers —
|
||||
NOT multiplied by n_genomes
|
||||
allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome —
|
||||
@@ -118,11 +118,11 @@ The `-z` CLI option overrides the index metadata value. A higher z increases str
|
||||
|
||||
## Layer lookup: `MphfLayer::find`
|
||||
|
||||
`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obilayeredmap](../implementation/obilayeredmap.md) for the full `find` / `find_strict` API.
|
||||
`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obikindex::layer](../implementation/layer_tier.md) for the full `find` / `find_strict` API.
|
||||
|
||||
### `QueryLayer` variant selection
|
||||
|
||||
`QueryLayer::open` (`obikpartitionner/src/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
|
||||
`QueryLayer::open` (`obikindex/src/partition/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
|
||||
|
||||
| Order | Condition | Variant | Data returned per k-mer |
|
||||
|---|---|---|---|
|
||||
@@ -244,7 +244,7 @@ Fix direction: restructure `query`'s pipe with an initial `Flat` stage analogous
|
||||
`chunk_bytes = available_memory_bytes() / (n_workers * 16)` (`query.rs:407-414`) assumes a fixed ~8–16× overhead per raw input byte. But `KmerResults::new` (`query.rs:165-179`) allocates `data: Vec<u32>` sized `total_kmers_in_chunk × n_genomes` — dense, **for every k-mer position in the chunk, hit or not** — plus `win_min` and (with `--detail`) `cov`, same scaling. Real per-chunk memory is `O(n_genomes)`, not constant; the formula doesn't know `n_genomes` at all. This is the direct cause of the OOM kill on indexes with many reference genomes.
|
||||
|
||||
**4. MPHF lookup and matrix-row fetch are fused, not staged.**
|
||||
`QueryLayer::find_into` (`obikpartitionner/src/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
|
||||
`QueryLayer::find_into` (`obikindex/src/partition/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
|
||||
|
||||
**5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.**
|
||||
`QueryBatch::from_records` currently dereplicates at the *superkmer* level (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, `query.rs:112`). This misses redundancy between k-mers shared by *different* superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer *construction* (`SuperKmerIter`) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on `CanonicalKmer`, in the same pass: `HashMap<CanonicalKmer, Vec<(seq_idx, pos)>>`. This also means the MPHF `find` itself runs once per **distinct** k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.
|
||||
@@ -285,7 +285,7 @@ None of this is implemented yet — parked here as a coherent roadmap while the
|
||||
|
||||
## Implementation plan
|
||||
|
||||
Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikpartitionner`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
|
||||
Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikindex::partition`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
|
||||
|
||||
Instrumentation is deliberately sequenced *before* the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.
|
||||
|
||||
@@ -295,7 +295,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
|
||||
**Debug logging.** Every phase that changes an algorithmic choice (not phase 0, which *is* the logging) adds `tracing::debug!`/`trace!` at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF `find` calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured `debug!` per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.
|
||||
|
||||
**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obilayeredmap`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikpartitionner` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
|
||||
**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obikindex::layer`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikindex::partition` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
|
||||
|
||||
**Workflow (`jj`).** Work happens in a fresh `jj` commit, easy to abandon. `jj new` between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then `query_layer.rs` split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.
|
||||
|
||||
@@ -345,13 +345,13 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
- `obikmer/src/cmd/query.rs`:
|
||||
- Replace `QueryBatch::from_records`'s dedup map (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, current `query.rs:112`) with a per-partition `HashMap<CanonicalKmer, Vec<(seq_idx: u32, pos: u32)>>`, built in the same `SuperKmerIter` pass: superkmer construction and partition routing (`part_idx` from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each `CanonicalKmer` within a superkmer is inserted individually instead of the whole superkmer being the dedup key.
|
||||
- **Verified**: `CanonicalKmer` (`obikseq/src/kmer.rs:390`, `pub type CanonicalKmer = CanonicalKmerOf<KLen>`) — the underlying `CanonicalKmerOf<L>` derives `Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash` (`kmer.rs:269`). Usable as a `HashMap`/`HashSet` key as-is, no change needed.
|
||||
- `obikpartitionner/src/query_layer.rs`:
|
||||
- `obikindex/src/partition/query_layer.rs`:
|
||||
- Split `QueryLayer::find_into` (`query_layer.rs:48-67`) into two methods: `find_slot(&self, kmer: CanonicalKmer) -> Option<usize>` (MPHF only, no matrix touch) and keep `fill_row` as-is for phase 4 to call later.
|
||||
- Replace `query_partition_with`'s inner loop (`query_layer.rs:103-113`) with a version that, for each unique `CanonicalKmer`, calls `find_slot` across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records `(layer_idx, slot)`.
|
||||
- New return shape for the partition-level query, replacing today's `on_hit(sk_idx, kmer_idx, row)` callback: `HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>` (roadmap point 6) — built directly from the k-mer dedup map's `Vec<(seq_idx,pos)>` values, keyed by the resolved slot instead of the k-mer.
|
||||
- **This phase alone has no throughput benefit yet** (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.
|
||||
- Log, per chunk: total k-mer occurrences vs. unique `CanonicalKmer` count (the dedup ratio — the entire justification for this phase) and the resulting MPHF `find` call count. If the dedup ratio is close to `1.0` on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.
|
||||
- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikpartitionner/src/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
|
||||
- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikindex/src/partition/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
|
||||
|
||||
### Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred
|
||||
|
||||
@@ -359,7 +359,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
|
||||
**What shipped:**
|
||||
- `obicompactvec`: the per-column accessors this phase needed **already existed** — `PersistentCompactIntMatrix::col_view(c)` and `PersistentBitMatrix::col_view(c)` are public, and `IntSliceView::get(slot)`/`BitSliceView::get(slot)` are public — the original plan underestimated how much of this plumbing the pairwise-distance code (`dump`/`select`/`stats`) had already required. The one real gap: `PersistentBitMatrix::col_view()` panics on the `Implicit` variant (the documented mono-genome fast path, `bitmatrix.rs`). Added `PersistentBitMatrix::get(c, slot) -> u32` (`bitmatrix.rs`), a non-panicking column-major point lookup that returns `1` for `Implicit` regardless of `c` — the smallest surface needed, not a new `col_get` API from scratch.
|
||||
- `obikpartitionner/src/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
|
||||
- `obikindex/src/partition/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
|
||||
- `obikmer/src/cmd/query.rs`: `KmerResults::set` (row-major, whole-row-at-once) replaced by `mark_found` (stage 1: flag a position as indexed, independent of any genome's value) and `set_one` (stage 2: write one genome's value at one position). `QueryStats` extended with `n_columns_scanned`/`n_col_get_calls`, logged per chunk.
|
||||
- Total `get()`-equivalent calls are unchanged from the row-major version (`n_hits × n_cols` in the worst case, confirmed by `n_col_get_calls` in the debug log) — the win is locality (sequential access within one layer's column at a time, across `mmap`'d regions, instead of jumping across all columns per hit), exactly as predicted.
|
||||
|
||||
@@ -367,7 +367,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
Reading `obikindex/src/numa.rs`'s actual `run()` body (not just its doc comments) shows every call spawns a timer thread **plus one OS thread per worker slot on every NUMA node** (`std::thread::scope` + one `s.spawn()` per node per `max_workers`) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned **per call**. This is fine for its actual, established usage in this codebase (`merge.rs`, `index.rs`'s `build_layers`): one `PartitionRunner::new()` + one `run()` call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for `query`'s call pattern: `query_partition_with` runs once per `(chunk, partition)`, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
|
||||
The column-major loop in stage 2 is therefore a **plain sequential loop** for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) `rayon`'s already-warm global pool (`into_par_iter()`) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern `numa_worker_pools.md` warns about for a *different* workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one `mmap`'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so `PartitionRunner` is invoked once per whole `query` run (or per large batch of chunks) rather than per `(chunk, partition)`, amortising its spawn cost the way `merge`/`build_layers` do — a bigger structural change than this phase's scope.
|
||||
- Log (implemented): `QueryStats::n_columns_scanned`/`n_col_get_calls`, folded into the existing per-chunk `debug!("k-mer dedup + column-major fetch", ...)` line (`query.rs`) alongside phase 3's dedup counters.
|
||||
- **Unit tests**: extended `obikpartitionner/src/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
|
||||
- **Unit tests**: extended `obikindex/src/partition/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
|
||||
- **Validation performed**: full workspace build + `cargo test --workspace`, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same `kmer_count`/`kmer_strict_matches` on every record); (2) the existing 20-genome `benchmark/global_index_presence` index — runs correctly, `n_hits=0` for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the `Implicit`/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) **the critical correctness case**: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from `genomeA` matched **only** `genomeA` (`kmer_count` identical to the pre-dedup occurrence count, zero leakage into `genomeB`'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in `n_cols`, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
|
||||
- **Not yet done**: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-`n_genomes` layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.
|
||||
|
||||
@@ -404,3 +404,45 @@ Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-w
|
||||
- **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone.
|
||||
- **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition.
|
||||
- **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action.
|
||||
|
||||
|
||||
## Query memory: `PartitionKmers` replaces the per-partition `HashMap` (2026-10-15)
|
||||
|
||||
**Problem.** A full-size query (two ~450 MB fastq.gz of a skim against a 354 M-kmer index) reached **74–81 GB RSS** with the default 256 MiB chunk. Cause, read in `batch.rs`/`chunk.rs` and measured with the `by_partition memory retained` debug log: `QueryBatch::by_partition` was a `HashMap<CanonicalKmer, Vec<KmerDesc>>` per partition. On genomic reads almost every k-mer is unique (12.5 M unique for 13.5 M occurrences in a 48 MB chunk, a dedup of 8 %), so each k-mer paid a table slot, growth slack and its own heap `Vec`: **889 MB per 48 MB chunk, ~71 bytes per unique k-mer, 18× the chunk size** — the dominant term of the per-chunk memory, several chunks being in flight at once.
|
||||
|
||||
**Change.** `obikquery::PartitionKmers`: occurrences collected as `(kmer, KmerDesc)` per partition, sorted by `(kmer, seq_idx, pos)` and stored as three flat arrays (unique k-mers, u32 offsets, descs). `query_partition_with` takes `&PartitionKmers`; `QueryHit` already carried slices. Occurrence order per k-mer is unchanged (query order); no per-k-mer allocation.
|
||||
|
||||
**Measured** (same inputs, release build, identical sorted output digests before and after):
|
||||
|
||||
| | `by_partition` (48 MB chunk) | RSS, 2×48 MB chunks | RSS, full run (≈ 1 GB reads) | wall, full run |
|
||||
|---|---|---|---|---|
|
||||
| before | 889 MB | 5.6 GB | 73.7–80.7 GB | 135–172 s (one 312 s outlier) |
|
||||
| after | 259 MB | 4.1 GB | **28.2 GB** | **103 s** |
|
||||
|
||||
With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the touched pages of the index itself, so after the change the remaining per-chunk anonymous memory is ≈ 6× the chunk size, below the formula's factor 12 (`BASE_OVERHEAD + 8·n_genomes`, still a conservative bound — left unchanged). The sort replaces hashing in `from_records` and was not slower (17.3 s vs 21.2 s on the small test).
|
||||
|
||||
**Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here.
|
||||
|
||||
**Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed.
|
||||
|
||||
### Chunk size sweep on a small input (2026-10-15)
|
||||
|
||||
Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer single-genome index (4 partitions), 16 cores on AC power, 3 runs per size, sorted outputs identical (same md5) for every size:
|
||||
|
||||
| `--chunk-size` (MiB) | chunks | wall (s) | user CPU (s) | RSS (GB) |
|
||||
|---|---|---|---|---|
|
||||
| 2 | ~50 | 2.5–2.6 | 25.0–25.3 | 3.2 |
|
||||
| 4 | ~25 | 2.5–2.7 | 24.3–25.5 | 3.5 |
|
||||
| 8 | ~12 | 2.5–2.7 | 23.6–26.1 | 4.1 |
|
||||
| 16 | ~6 | 4.4 | 23.1–23.7 | 4.1 |
|
||||
| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 |
|
||||
| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 |
|
||||
|
||||
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs — **implemented, see below**. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
|
||||
|
||||
|
||||
### Automatic chunk size now follows the input size (2026-10-15)
|
||||
|
||||
`cmd/query/mod.rs`: when `--chunk-size` is not given, the memory-derived size (unchanged: `available RAM / (workers × overhead × 2)`, clamped to 4–256 MiB) is further capped by `estimated input bytes / (workers × 2)`, never below 4 MiB. The estimate is `obiread::estimated_decompressed_len`: the file length, ×3 when the magic bytes say gzip (≈ 2.8 measured on FASTQ), ×4 for bzip2/xz/zstd — a deliberately rough figure, detected from the magic bytes, not the extension. For stdin, URLs or unreadable files the size is unknown and the memory-derived size stands. An explicit `--chunk-size` still wins.
|
||||
|
||||
Measured (2 × 48 MB reads, 16 cores, AC power, outputs identical): single-genome index, 354 M kmers — default chunk 256 MiB → 4 MiB: **11.4 s → 2.4 s** (−79 %), CPU 22.3 → 26 s (+17 %), peak footprint 1.44 → 0.75 GB. On the 91-genome index the memory-derived size was already 4 MiB (overhead factor 4 + 8 × 91), so nothing changes there (5.1–5.7 s either way). Not measured: a large input (≈ 1 GB gz, whose estimate gives ≈ 95 MiB chunks instead of 256 MiB: more, smaller chunks, expected lower memory).
|
||||
@@ -4,7 +4,7 @@
|
||||
## Code couvert
|
||||
|
||||
- `obikmer/src/cmd/query.rs` — commande query, format de sortie
|
||||
- `obikpartitionner/src/query_layer.rs` — routage de la requête à travers les partitions
|
||||
- `obikindex/src/partition/query_layer.rs` — routage de la requête à travers les partitions
|
||||
- `obiread/src/lib.rs` — lecture des séquences d'entrée pour la requête
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -94,9 +94,9 @@ Option B avoids storing kmer values and works uniformly regardless of filter sel
|
||||
|
||||
## Files to modify
|
||||
|
||||
- `src/obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
|
||||
- `src/obikindex/src/partition/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
|
||||
- Possibly `src/obicompactvec/` — add column iterator API if not already present
|
||||
- `src/obilayeredmap/` — check if per-column sequential access is exposed on `SrcLayerData`
|
||||
- `src/obikindex/src/layer/` — check if per-column sequential access is exposed on `SrcLayerData`
|
||||
|
||||
## Open questions
|
||||
|
||||
|
||||
+160
-758
@@ -1,758 +1,160 @@
|
||||
# Sibling annex — architecture (discussion)
|
||||
|
||||
Status: architecture decided (2026-08-14). Implementation not yet mandated.
|
||||
|
||||
## Two index spaces, uncorrelated
|
||||
|
||||
Every kmer stored in a `Layer` lives in two independent index spaces:
|
||||
|
||||
- **Iteration order**: its position when enumerating `unitigs.bin` (the
|
||||
superkmer file), deterministic but arbitrary with respect to slot.
|
||||
- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns.
|
||||
|
||||
The two are not correlated by any formula. Converting from one to the other
|
||||
requires either recomputing the MPHF (kmer → slot) or scanning the iteration
|
||||
stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a
|
||||
one-way function, not an invertible bijection with a stored inverse. Any
|
||||
method that reconstructs a kmer from a bare slot number is wrong by
|
||||
construction, regardless of the mechanism used (MPHF re-hash, or evidence
|
||||
decode + direct unitig read). See `MphfLayer::kmer_at`
|
||||
(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
|
||||
from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
|
||||
"Pending work" status below).
|
||||
|
||||
## Two pipelines, never mixed
|
||||
|
||||
| | origin of the kmer | membership known? | correct mapping |
|
||||
|---|---|---|---|
|
||||
| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check |
|
||||
| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence |
|
||||
|
||||
Evidence exists solely to answer "is this external kmer a member of the
|
||||
layer" for the query pipeline. Using it (or the MPHF) to go the other way —
|
||||
recover a kmer from a slot, or re-verify a kmer that was just produced by
|
||||
iterating the layer — is a conceptual error: evidence can be probabilistic
|
||||
(`Approx` mode), so any slot→kmer attempt is unsound in general, and
|
||||
pointless even in `Exact`/`Hybrid` mode since the kmer was already known.
|
||||
|
||||
## Sibling annex: an iteration-pipeline artifact only
|
||||
|
||||
The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`,
|
||||
`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a
|
||||
family minorant and which family members are present in the index. Its only
|
||||
consumers (`obikphylo/siblings/stats.rs`, `family_scan.rs`) enumerate it
|
||||
exhaustively (`0..annex.len()`); no query-pipeline code path touches it.
|
||||
|
||||
**Decision**: the annex must be persisted in iteration order, not slot
|
||||
order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex
|
||||
file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
|
||||
no `kmer_at`. It also enables specialized iterators building on this zip:
|
||||
minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.
|
||||
|
||||
Today the annex is built and stored in **slot** order
|
||||
(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated
|
||||
via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is
|
||||
threaded through the whole cross-partition reconciliation pipeline (variant
|
||||
generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This
|
||||
must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading
|
||||
the **iteration index** instead of the slot end to end — eliminating
|
||||
`kmer_at` from the build path entirely, not just the read path. No
|
||||
slot-indexed intermediate is needed even during construction; the
|
||||
iteration-order id is sufficient throughout.
|
||||
|
||||
The cross-partition side of the same pipeline is unaffected: checking
|
||||
whether a generated family-variant kmer exists in another partition is a
|
||||
genuine query-pipeline operation (the variant's membership in the *target*
|
||||
partition is unknown) and must keep going through
|
||||
`KmerPartition::query_partition_with` (MPHF + evidence), never a raw
|
||||
`index()`.
|
||||
|
||||
## Pending work — done
|
||||
|
||||
The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
|
||||
traits over `obikindex::KmerIndex`/`obilayeredmap::Layer<D>`, replacing the
|
||||
old `obikindex::siblings` module) builds and reads the annex purely in
|
||||
iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
|
||||
batch variants, mirroring `Layer<D>`'s own `KmerIter`/`KmerBatchIter`
|
||||
shape). `MphfLayer::kmer_at` has no remaining callers.
|
||||
|
||||
A separate, unrelated bug surfaced during this work and was fixed
|
||||
(2026-08-14): `MphfLayer::enumerate_kmers_batch` computed its
|
||||
`batch_start_index` via the stdlib `.enumerate()` adapter, which counts
|
||||
*batches* (0, 1, 2…), not the cumulative k-mer offset the annex is actually
|
||||
keyed on — every batch past the first wrote its mask/annex entries at the
|
||||
wrong iteration-order position. Fixed by tracking a running offset instead;
|
||||
regression tests added (`sibling_annex_no_empty_masks_after_build`,
|
||||
`sibling_histogram_does_not_panic_on_partial_last_batch`).
|
||||
|
||||
## Performance: `build_sibling_annex` parallelism (2026-08-14)
|
||||
|
||||
Investigated on a real multi-genome run (`phyloskims_sal_vac`, k=31/m=11).
|
||||
Baseline: mostly one active core, with short multi-core bursts — average
|
||||
~3 cores.
|
||||
|
||||
**Fixes that helped, kept:**
|
||||
|
||||
- `CanonicalKmerOf::minimizer()` (`obikseq/src/kmer.rs`) — a direct O(k)
|
||||
bit-arithmetic minimiser for a single isolated k-mer, replacing a
|
||||
`RollingStat` instance fed byte-by-byte through an ASCII round-trip (used
|
||||
by `helpers::partition_of`, called for every generated family variant).
|
||||
~3x wall-clock improvement on its own, confirmed by sampling
|
||||
(`obiskbuilder::rolling_stat`/`obikentropy` frames disappeared from the
|
||||
hot path). `CanonicalKmerOf::partition()` added alongside it (wraps
|
||||
`minimizer().seq_hash() & mask`, the same routing rule
|
||||
`KmerPartition`/`RoutableSuperKmer` use).
|
||||
- Cross-partition resolution (`outgoing.par_iter()` in
|
||||
`build_layer_sibling_annex`) parallelised at the *partition* level — one
|
||||
Rayon task per non-empty `outgoing[dest]` bucket. For k=31/m=11, a
|
||||
central-base substitution changes the winning minimiser (and thus the
|
||||
destination partition) only when that window overlaps the central base:
|
||||
~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants
|
||||
route right back to the partition already being built. That self bucket
|
||||
ends up far larger than any other, so the per-partition split pinned one
|
||||
thread to it alone while the rest of the pool finished instantly —
|
||||
confirmed by sampling: one thread solid in `MphfLayer::find`, everyone
|
||||
else idle. Fixed by splitting each non-empty bucket into
|
||||
`total_queries / n_workers` (capped 4096) chunks *before* `par_iter()`,
|
||||
preserving per-partition mmap locality (each chunk stays contiguous
|
||||
within one partition) while letting Rayon spread an oversized bucket
|
||||
across several threads. Net effect of both fixes together: ~3 cores
|
||||
average → ~10-13 cores average on the same run, and a projected total
|
||||
build time of ~1h15 down to ~30min on the real `phyloskims_sal_vac` run
|
||||
this was measured against.
|
||||
- `TracedBar`'s ETA (`obisys/src/progress.rs`) was silently starved: the
|
||||
custom progress message and the self-computed ETA text used to share one
|
||||
`pb.set_message()` slot, with the ETA holding off for 2s after any custom
|
||||
message — fine when custom messages are rare, broken once
|
||||
`build_sibling_annex`'s per-partition callback fires more often than
|
||||
that. Fixed by keeping the two texts in separate fields, composed
|
||||
together on every render instead of one overwriting the other.
|
||||
|
||||
**Tried and reverted — do not repeat blindly:**
|
||||
|
||||
- Parallelising the *outer* partition loop in `build_sibling_annex` with
|
||||
`obikindex::PartitionRunner` (already used by `merge`/`build_layers`),
|
||||
splitting a fixed core budget between outer (partition) and inner
|
||||
(pipeline + resolution) concurrency so their product wouldn't exceed the
|
||||
budget. Measured *worse*: throughput dropped over time (26
|
||||
partitions/5min → 38/11-12min) and peak resolution concurrency fell from
|
||||
~11-12 cores to ~7-8. Cause: this capped the resolution burst — which
|
||||
scales very well on its own — to make room for outer concurrency, and
|
||||
running several partitions' resolution at once scatters access across
|
||||
multiple partitions' mmap regions at once, working against the
|
||||
locality `outgoing`'s per-partition grouping exists for. `PartitionRunner`
|
||||
stayed exported from `obikindex` (`new_capped` too) since it's
|
||||
general-purpose, but nothing in `obikphylo` calls it.
|
||||
- Splitting resolution chunks even finer (`/(n_workers*8)`, cap 1024,
|
||||
instead of `/n_workers`, cap 4096) to smooth the residual sawtooth.
|
||||
Measured ~10% *slower*, wider dips, not narrower. Reverted to the
|
||||
original chunk sizing.
|
||||
|
||||
**Known remaining limitation, not yet worth fixing:** within one layer, the
|
||||
four stages (sequential `unitigs.bin` read → parallel generation →
|
||||
parallel resolution → sequential annex write) never overlap — confirmed by
|
||||
1s-interval sampling: generation alone occupies ~17 threads evenly, but the
|
||||
next layer's read/generation never starts until the current layer's
|
||||
resolution and write are both done. This produces a real, periodic (~layer
|
||||
duration) alternation between "many cores" and "few cores" that neither of
|
||||
the fixes above touches, since both operate *within* one layer's resolution
|
||||
step. The only remaining lever is overlapping consecutive layers (e.g. a
|
||||
depth-2 pipeline: start layer N+1's read/generation while layer N's
|
||||
resolution/write is still running) — a real restructuring, not a parameter
|
||||
tweak, and explicitly *not* to be combined with the reverted
|
||||
budget-capping idea above (let each phase use however many cores it
|
||||
naturally wants; only the *scheduling* needs to overlap). Deferred, not
|
||||
started.
|
||||
|
||||
## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)
|
||||
|
||||
`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`,
|
||||
`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already
|
||||
implements most of a dispatch/accumulate/resolve pipeline: generation
|
||||
(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and
|
||||
buckets cross-partition queries) runs on an `obipipeline::throttle` +
|
||||
`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter`
|
||||
*across partitions*, one generated batch resolved at a time, never several
|
||||
concurrently — this ordering is deliberate, see the module's own docs on a
|
||||
reverted concurrent-batch-resolution attempt that scattered mmap access).
|
||||
The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`)
|
||||
already exists: `n_layers <= 7` (checked once from the first non-empty
|
||||
partition's `PartitionMeta::n_layers`, documented as identical across every
|
||||
partition of an index — a structural, build-time property, never a
|
||||
per-partition state) decides whether `FamilyMask`'s recorded `layer_value`
|
||||
can be trusted to skip straight to the right layer
|
||||
(`find_presence_batch_fast`) or must fall back to scanning every layer of
|
||||
the destination partition (`find_presence_batch`).
|
||||
|
||||
**Real gap, confirmed not implemented**: resolution is triggered by the
|
||||
*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the
|
||||
scanned layer), not by an *output* accumulator filling up. Since most
|
||||
central-base variants of a family route back to the same partition being
|
||||
scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s
|
||||
`outgoing[dest]` is large for the local/self partition and thin for the
|
||||
other ~255 (or however many) destination partitions — each of those gets
|
||||
resolved at low query density every batch instead of being accumulated
|
||||
across several source batches until resolving it is worthwhile. This is
|
||||
distinct from, and not fixed by, the fast/slow layer gate above.
|
||||
|
||||
Redesign sketched (not built): per-destination accumulators decoupled from
|
||||
`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch
|
||||
completion — a "hot" accumulator for the partition being scanned (sharded
|
||||
one-per-generation-worker, no lock, since all `n_workers` pipeline workers
|
||||
write to it concurrently — this differs from an earlier, simpler mental
|
||||
model of "one thread owns one layer's local collector," which doesn't hold
|
||||
here since `n_workers` threads cooperate on scanning *one* layer at a time,
|
||||
not one thread per layer) and "cold" mutex-per-partition accumulators for
|
||||
the rest, low contention expected since traffic to any single cold
|
||||
destination is a small fraction of total.
|
||||
|
||||
This breaks the current strict-iteration-order delivery of `on_family`
|
||||
(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH`
|
||||
resolves atomically). With cross-batch accumulation, a family only becomes
|
||||
complete once *every* accumulator holding one of its outgoing queries has
|
||||
flushed, at unpredictable, independent times — no longer streamable
|
||||
strictly in order without a large, unbounded pending buffer. Resolution
|
||||
sketched: replace order-dependent consumers with coordinate-addressed
|
||||
writes instead of order-dependent appends (see `PseudoAlignment` idea
|
||||
below) wherever possible, since `sibling_annex_stats`'s reduction (plain
|
||||
counts) is already order-independent and needs nothing here.
|
||||
|
||||
**Superseded 2026-08-15** by the `--subsample`/`--shannon` design below,
|
||||
which sidesteps the accumulator redesign for now: bounding the number of
|
||||
families actually resolved per layer (via sampling) keeps per-layer
|
||||
resolution volume small enough that the batch-density problem above stops
|
||||
mattering in practice for these two consumers. The accumulator redesign
|
||||
remains relevant for a future *unsampled, full-index* run, but is not
|
||||
required to ship `--subsample`/`--shannon`.
|
||||
|
||||
## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15)
|
||||
|
||||
The reference run (`phyloskims_sal_vac`-scale bacterial test set,
|
||||
`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites
|
||||
(4.8 MB) — trivially small. The in-progress plant index is expected to
|
||||
carry on the order of 9 billion minorant families; a dense byte-per-cell
|
||||
alignment at that column count is unbuildable regardless of genome count
|
||||
(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000–
|
||||
8,000 genomes on a large machine, which makes the per-cell cost dominant in
|
||||
the other dimension too. Pruning the retained family set before
|
||||
materializing anything is mandatory, not an optimization.
|
||||
|
||||
**Already free**: `family_size() < 2` (no sibling variant registered at
|
||||
all) is a zero-cost structural filter, read directly off `FamilyMask` bits,
|
||||
already applied in `snp_pseudo_alignment`. Insufficient alone — per the
|
||||
~80-85% mono-family estimate from earlier discussion, this only brings 9
|
||||
billion down to roughly 1.3-1.8 billion, still unusable.
|
||||
|
||||
**Entropy definition — settled 2026-08-15, correcting an earlier wrong
|
||||
turn.** The project does **not** encode families as IUPAC ambiguity codes
|
||||
interpreted the classical way (Fitch-parsimony subset-compatibility, or
|
||||
ML's "one true state, uncertain which"); see
|
||||
`docmd/theory/evolutionary_distances.md` ("Why the IUPAC/DNA encoding used
|
||||
for the first `--snp` test was wrong") and the Sankoff resolution that
|
||||
followed it. The real model is a genuine 16-state alphabet (the powerset of
|
||||
`{A,C,G,T}`, `∅` included as a real state) scored with a *calibrated
|
||||
pairwise cost matrix* (`obikphylo::cardcomp::pairwise_cost_matrix`,
|
||||
`cmd/phylo/sankoff.rs`), not a compatibility/subset relation between
|
||||
states. Under that model, each of the 16 states — including multi-bit ones
|
||||
like `AC` — is a first-class, independently-costed state, not an
|
||||
uncertainty encoding of a single true base. So: **entropy over the 15
|
||||
non-empty states (`∅` excluded, matching the earlier decision to exclude
|
||||
genomes where the family is absent) is the correct informativeness
|
||||
measure** for this project — not a 4-symbol reduction, which would discard
|
||||
exactly the cardinality/composition information the calibrated cost matrix
|
||||
is built to exploit.
|
||||
|
||||
## `--subsample` / `--shannon` — sampling strategy (decided 2026-08-15)
|
||||
|
||||
Goal: make both the pseudo-alignment (`--snp`) and a Shannon-entropy
|
||||
diagnostic usable at any index scale, from the 13-genome bacterial
|
||||
reference run up to the 9-billion-family plant index, without requiring the
|
||||
batched-accumulator redesign above.
|
||||
|
||||
**`--subsample N`** (integer, families to retain): bounds the pseudo-alignment
|
||||
to `N` minorant families, sampled **proportionally per layer** among
|
||||
non-monomorphic minorants (`family_size >= 2`) — this sidesteps the need
|
||||
for a true global reservoir merge across layers while still approximating a
|
||||
uniform sample over the whole index, and directly answers the earlier open
|
||||
question of global-vs-per-layer selection scope.
|
||||
|
||||
Three passes, in order:
|
||||
|
||||
1. **Global count** (cheap, structural, parallel across layers — same shape
|
||||
as the existing `sibling_family_size_histogram`, extended to report a
|
||||
**per-layer** breakdown rather than one index-wide aggregate): for each
|
||||
layer, `count_layer` = number of non-monomorphic minorants. Gives
|
||||
`total_count = Σ count_layer`.
|
||||
2. **Per-layer proportional reservoir sampling** (cheap, structural, one
|
||||
pass per layer, no cross-partition resolution): `N_layer = round(N ×
|
||||
count_layer / total_count)`. Since `N_layer` is a proportion of
|
||||
`count_layer`, it can never exceed it as long as `N <= total_count` — the
|
||||
one edge case is `total_count <= N`, in which case sampling is skipped
|
||||
entirely and *every* non-monomorphic minorant of every layer is kept
|
||||
(no reservoir needed, `N` was never a real constraint). Otherwise:
|
||||
Algorithm-R reservoir sampling over the layer's non-monomorphic minorant
|
||||
indices, producing `N_layer` iteration-order indices directly, no
|
||||
intermediate full list ever materialized.
|
||||
3. **Filtered resolution** (the expensive step, the existing
|
||||
`scan_layer_families` engine, unchanged): re-scan the layer, generating
|
||||
and resolving cross-partition queries **only** for the indices selected
|
||||
in step 2 (cheap membership test against a small per-layer index set) —
|
||||
this is what keeps `--subsample` cheap even on an unsampled-scale index,
|
||||
since the cross-partition resolution volume is bounded by `N`, not by
|
||||
the layer's true size.
|
||||
|
||||
Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir
|
||||
sampling would waste step-3's expensive resolution work on candidates later
|
||||
evicted by the reservoir. Step 1 must fully complete (every layer) before
|
||||
step 2 can start for any layer, since `total_count` is a global quantity.
|
||||
|
||||
**`--shannon`** (no argument): emits a CSV of per-family Shannon entropy
|
||||
(15 non-empty states, `∅`/absent genomes excluded from the denominator, per
|
||||
the settled definition above). Independent of `--subsample` — entropy is
|
||||
computed and written per family as soon as its `genome_mask` resolves,
|
||||
O(1) memory per family, so it streams fine even unsampled at full index
|
||||
scale (a time cost, not a memory one). Combined with `--subsample N`, it
|
||||
delivers the original exploratory diagnostic (e.g. `--subsample 1000000
|
||||
--shannon`) directly from this general machinery, rather than a
|
||||
purpose-built one-off script.
|
||||
|
||||
Validated end-to-end (2026-08-15) against real data: `--sibling-hist` on
|
||||
`phyloskims_sal_vac` (91 real genomes, k=31/m=11, 256 partitions × 2
|
||||
layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238
|
||||
total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier
|
||||
guess, corrected here). `--subsample`/`--shannon` on the smaller 20-genome
|
||||
bacterial reference (`benchmark/global_index_presence`) produced a sample
|
||||
size within rounding of the request (99,742/100,000) and a
|
||||
[0.8,1.2)-bucket share (40.7%) matching the full unsampled population
|
||||
(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a
|
||||
real bug in `reservoir_sample_layer` (see next section), now fixed.
|
||||
|
||||
**Bug found and fixed (2026-08-15): `family_idx` numbering mismatch.**
|
||||
`scan_layer_families`'s `family_idx` counts *every minorant* of a layer
|
||||
(monomorphic ones included, since `iter_minorants_batch` filters only on
|
||||
`is_minorant()`), not the raw annex slot (`SiblingAnnex::get(slot)` spans
|
||||
every k-mer, minorant or not) and not a counter over non-monomorphic
|
||||
minorants alone. `subsample.rs`'s `reservoir_sample_layer` originally
|
||||
stored raw slot numbers in its `HashSet<usize>` selection, which drifts
|
||||
away from `family_idx` as soon as *any* monomorphic minorant is seen —
|
||||
i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by
|
||||
tracking two separate counters: `family_idx` (every minorant, matching
|
||||
`scan_layer_families`) and `seen` (non-monomorphic minorants only, what
|
||||
Algorithm R actually samples over) — only `family_idx` values are ever
|
||||
stored in the selection set. The existing unit test never exercised this
|
||||
(its fixture has exactly one non-monomorphic family, always hitting the
|
||||
"keep everything" shortcut) — a stronger fixture with several interleaved
|
||||
monomorphic/non-monomorphic families would be needed to catch a regression
|
||||
here automatically; not yet written.
|
||||
|
||||
## Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15)
|
||||
|
||||
Motivation: on real data (bacterial reference, full unsampled run), only
|
||||
~5‰ of non-monomorphic minorants fall in the `[0.5, 1.5]` bit band judged
|
||||
phylogenetically interesting (entropy too low = uninformative near-invariant
|
||||
site; too high = saturated/noisy, see `family_entropy`'s 15-state
|
||||
discussion) — roughly 1 in 10,000 minorants overall. Computing exact
|
||||
entropy for every candidate just to discard 99.99% of them is wasteful at
|
||||
the full 9-billion scale.
|
||||
|
||||
**The idea**: `PersistentBitMatrix::col_view(c)` gives a genome's whole
|
||||
presence column as a `BitSliceView` (sequential, no MPHF, no cross-partition
|
||||
routing — purely local to one layer's own matrix). Accumulating
|
||||
`TempCompactIntVecBuilder::inc_present(col)` (already exists in
|
||||
`obicompactvec/src/builder.rs:121`, along with `add`/`min`/`max`/`diff` on
|
||||
`IntSliceView` — no new low-level API needed) over every column of a layer
|
||||
produces `coverage[slot]`: how many genomes carry each exact k-mer, in one
|
||||
sequential per-layer pass, entirely decoupled from family/sibling
|
||||
structure. Persisted once per layer, a family's members' coverage could
|
||||
then be looked up via a plain local MPHF `index()` (cheap) instead of a
|
||||
full cross-partition presence resolution (`find_presence_batch`) — the
|
||||
expensive part today is specifically the cross-partition/cross-layer
|
||||
routing to a sibling's own matrix, not the bit-reading itself, and
|
||||
`coverage[slot]` sidesteps that routing entirely by moving the cost into a
|
||||
one-time, purely local, embarrassingly-parallel build step.
|
||||
|
||||
**Why not implemented**: `coverage[slot]` is a per-member marginal —
|
||||
summing members' coverages to approximate a family's entropy silently
|
||||
assumes no genome carries more than one member at once. It cannot
|
||||
represent or detect joint co-occurrence (a genome carrying both `A` and
|
||||
`C` at once, i.e. a combined 15-symbol state) at all, which is exactly the
|
||||
phenomenon `family_entropy`'s 15-state definition exists to capture (see
|
||||
`CardinalityTally`/`cardinality_transition_probs`, the project's own
|
||||
existing machinery for this same co-occurrence structure, built for the
|
||||
Sankoff matrix calibration). A family that is in truth uniformly `AC`
|
||||
across every carrying genome would look like a well-balanced 2-state split
|
||||
under the marginal approximation (entropy ≈ 1) while its true 15-state
|
||||
entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the
|
||||
"saturated, uninformative" tail this pre-filter would need to catch,
|
||||
undermining the point. It stays plausible as a coarse filter for the
|
||||
*low* tail only (a dominant single member's marginal share reliably
|
||||
predicts low true entropy too), but not as a stand-in for the high tail —
|
||||
not pursued further for now.
|
||||
|
||||
## `--free-loss`/`--tnt` pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below)
|
||||
|
||||
Measured on `phyloskims_sal_vac` (91 genomes): `obikmer phylo --subsample 500000
|
||||
--free-loss --tnt` logs four sequential stages —
|
||||
`raw_snp_distance` (1413s), `base_pair_tally` (1456s),
|
||||
`cardinality_tally` (2078s), `snp_pseudo_alignment` (143s). Reading the
|
||||
code (`obikmer/src/cmd/phylo/mod.rs:210-242`,
|
||||
`obikphylo/src/siblings/distance.rs`, `cardinality.rs`, `alignment.rs`)
|
||||
surfaced two compounding problems, not one:
|
||||
|
||||
1. **Four separate full scans of the annex**, each opening its own
|
||||
`KmerPartition`/`PartitionCache` and calling `scan_family_pairs`/
|
||||
`scan_layer_families` independently — nothing computed in one stage is
|
||||
reused by another. `base_pair_tally` is explicitly documented
|
||||
(`distance.rs:138-143`) as a second full pass over the same
|
||||
traversal `raw_snp_distance` already did, needed only because
|
||||
`raw_snp_distance` doesn't keep the resolved bases, only aggregate
|
||||
counts. `cardinality_tally` and `snp_pseudo_alignment` are each a
|
||||
third and fourth independent full pass. Per the module's own earlier
|
||||
profiling note (`family_scan.rs:26-28`, cited already above), this
|
||||
traversal is page-fault/mmap-bound, not compute-bound — the ~10-12%
|
||||
CPU efficiency ("contention" status) observed on these three slow
|
||||
stages is consistent with I/O stalls scaled by repeated full scans,
|
||||
not lock contention (there are no `Mutex`/`RwLock` anywhere in
|
||||
`siblings/*.rs`; shared writes use per-slot `AtomicU8::fetch_or`).
|
||||
2. **Worse: `raw_snp_distance` and `cardinality_tally` don't honor
|
||||
`--subsample` at all** — both call `scan_layer_families` with
|
||||
`Selection::All` hardcoded, and `args.subsample` isn't even threaded
|
||||
into their function signatures (`mod.rs:212`, `mod.rs:226`). Only
|
||||
`snp_pseudo_alignment(args.subsample)` builds a real reservoir-sampled
|
||||
`Selection::Some(set)` (via `compute_selections`, `alignment.rs:89`).
|
||||
So today, `--subsample 500000` only bounds the pseudo-alignment step —
|
||||
the SNP-distance matrix, the Sankoff base-pair calibration, and the
|
||||
cardinality histogram are always computed over the **full, unsampled**
|
||||
index regardless of the flag. This is not merely "different subsamples
|
||||
per stage" (which would already be a problem worth fixing) — it's that
|
||||
three of the four stages never subsample, which explains most of the
|
||||
~10x runtime gap against `snp_pseudo_alignment` on its own.
|
||||
|
||||
**Decided requirement**: all four stages consume **one shared selection**,
|
||||
computed once, not each stage either scanning everything or drawing its
|
||||
own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair
|
||||
calibration, the cardinality histogram, and the pseudo-alignment all
|
||||
describe the same set of families — the calibration and the alignment it
|
||||
calibrates are now guaranteed to agree on which sites exist.
|
||||
|
||||
**Correction to the "single pass" framing above**: `base_pair_tally`/
|
||||
`cardinality_tally` both need `raw_snp_distance`'s *complete* aggregate
|
||||
SNP/shared counts before they can derive `included[i,j]` (the
|
||||
`ratio_ceiling` filter) — a genuine sequential dependency (`included`
|
||||
can't be known until every pair's aggregate count is final), not an
|
||||
artifact of the old code's structure. So the fix is **two** passes over
|
||||
the shared selection, not one: pass A computes the aggregate counts (and
|
||||
derives `included`); pass B fuses `base_pair_tally` + `cardinality_tally`
|
||||
+ the pseudo-alignment (mutually independent once `included` is known)
|
||||
into a single scan. Still a 4→2 reduction, and — per the clarification
|
||||
that settled this — pass A itself now runs over the *same shared
|
||||
selection* pass B uses (not the full unsampled index): "les ratios, on
|
||||
les fait sur les sites sélectionnés, c'est tout, les autres sites
|
||||
n'existent pas" — once a selection is chosen, both passes are bounded by
|
||||
it, so on a real `--subsample`/`--entropy` run pass A is cheap too, not
|
||||
just pass B.
|
||||
|
||||
**Entropy-biased selection's own resolution to the "forward-looking
|
||||
complication"** (entropy must be known before selection, but selection
|
||||
happens during the same scan that would resolve it): see "Entropy-biased
|
||||
selection" below — resolved via a persisted per-layer entropy annex, not
|
||||
by restructuring the scan into an inline pre-pass.
|
||||
|
||||
## Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15)
|
||||
|
||||
Refines the "forward-looking complication" above with a concrete
|
||||
mechanism. Instead of a hard `[low, high]` entropy band (or any other
|
||||
exact cutoff) deciding which non-monomorphic minorant families are
|
||||
eligible, selection is weighted by an **unnormalized Gaussian kernel**
|
||||
centered on a target entropy: `w(entropy) = exp(-(entropy - μ)² / (2σ²))`
|
||||
— deliberately not the normalized Gaussian density (which would peak
|
||||
below 1 and complicate the "probability" reading) — this kernel form
|
||||
equals 1 exactly at `entropy = μ` and decays smoothly to 0 away from it,
|
||||
so it reads directly as an acceptance weight: the further a family's
|
||||
entropy from the target, the less likely it is picked, with no hard
|
||||
in/out boundary — a few "bad" sites can still get in, by design. `μ`
|
||||
(default ~1.0) and `σ` (default ~0.5) are meant to be user-tunable.
|
||||
|
||||
**Mechanism: joint probability, not a weighted reservoir**. Not
|
||||
Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex
|
||||
proposal, superseded before implementation) — instead, a single
|
||||
independent accept/reject draw per qualifying candidate: draw
|
||||
`u ~ Uniform(0,1)`, accept iff `u < p₀ · w(entropy)`. `p₀` is a single
|
||||
**index-wide** rate, `N / total_count` (`total_count` = the sum of
|
||||
`non_monomorphic_counts` across every layer, `N` = `--subsample`'s
|
||||
target), applied identically at every layer — this alone already gives
|
||||
each layer its proportional share (the same effect the old uniform
|
||||
reservoir's explicit per-layer `n_layer = N · count_layer / total_count`
|
||||
computation achieved, but without needing to compute it: applying one
|
||||
rate uniformly is mathematically the same as proportioning per layer).
|
||||
`p₀ = 1.0` when there is no `--subsample` at all — `--entropy` alone is a
|
||||
pure soft entropy filter over the whole index, no size target. Properties:
|
||||
(1) **strictly generalizes the existing uniform sampler** — with `σ` large
|
||||
enough that `w ≈ 1` everywhere, this reduces to the old uniform `N/total_count`
|
||||
draw; (2) yields "approximately N", not exactly N — expected accepted
|
||||
count is `N · mean(w)`, always `≤ N` — an intentional relaxation, matching
|
||||
"sous-échantillonnage à environ n" rather than the old reservoir's exact-N
|
||||
guarantee; (3) one streaming pass, one random draw per candidate, no
|
||||
reservoir state.
|
||||
|
||||
**Resolving "entropy must be known before selection, but selection
|
||||
happens during the same resolving scan"**: solved with a **persisted,
|
||||
per-layer entropy annex** (`obikphylo/src/siblings/entropy_annex.rs`,
|
||||
`EntropyAnnex`/`EntropyAnnexBuilder`), not by restructuring the scan.
|
||||
Mirrors `SiblingAnnex`'s mmap-backed, read-only-after-build convention,
|
||||
but indexed by `family_idx` (every minorant of the layer, monomorphic
|
||||
included — the same numbering `Selection`/`scan_layer_families` already
|
||||
use), one `f32` entropy15 value per entry, `-1.0` sentinel for monomorphic/
|
||||
not-yet-computed. First use of `--entropy`/`--entropy-sd` on an index
|
||||
pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`) — every
|
||||
later run (any `μ`/`σ`, any command) reads the file positionally, no
|
||||
re-scan, restoring the usual `Selection::Some` "skip resolving excluded
|
||||
families" speedup that a naive "weigh during the resolving scan" design
|
||||
would have permanently forfeited.
|
||||
|
||||
**Bug found and fixed (2026-08-15): `ensure_entropy_annexes` scanned with
|
||||
`Selection::All` instead of bounding to non-monomorphic minorants.**
|
||||
Monomorphism (`family_size() < 2`) is knowable directly from the annex
|
||||
bits alone, no per-genome resolution needed — but the original
|
||||
implementation called `scan_layer_families` with `Selection::All`
|
||||
anyway, so `fill_sub_matrix_carries` (the expensive per-genome
|
||||
resolution) ran for *every* minorant, ~98% of which are monomorphic
|
||||
(measured elsewhere in this doc) and had their `genome_mask` immediately
|
||||
discarded once the callback checked `family_size() < 2`. Fixed by adding
|
||||
[`subsample::non_monomorphic_selection_layer`] — a cheap, annex-only,
|
||||
non-sampling pass (same shape as `reservoir_sample_layer`, but keeping
|
||||
every non-monomorphic minorant's `family_idx` instead of a bounded
|
||||
reservoir) — and passing `Selection::Some(&eligible)` instead of
|
||||
`Selection::All`, so the expensive resolution now runs only for the ~2%
|
||||
of minorants that can actually produce a real entropy value. A
|
||||
`debug_assert!(mask.family_size() >= 2, ...)` inside the
|
||||
`scan_layer_families` callback guards the invariant.
|
||||
|
||||
**Resolved**: the existing hard "non-monomorphic minorant" eligibility
|
||||
filter stays a hard gate upstream of the Gaussian weighting — only
|
||||
qualifying families ever get a stored entropy value or a weighted draw.
|
||||
|
||||
**CLI, implemented**: two `phylo` options, `--entropy <μ>` and
|
||||
`--entropy-sd <σ>` (`obikmer/src/cmd/phylo/args.rs`). The entropic filter
|
||||
activates as soon as *either* is given (`mod.rs`, computed once into an
|
||||
`Option<EntropyBias>` threaded through `--snp`/`--family-overlap`/
|
||||
`--shannon`/the fused sankoff pipeline below). If active but one or both
|
||||
are unset, defaults are `μ = 1.0`, `σ = 0.5`.
|
||||
|
||||
## Two entropy definitions kept side by side, for comparison (2026-08-15)
|
||||
|
||||
`--shannon`'s CSV carries both `entropy15` (`family_entropy` — the settled
|
||||
15-non-empty-state definition, see above) and `entropy4` (`family_entropy_4`
|
||||
— plain nucleotide reduction), computed from the same already-resolved
|
||||
`genome_mask`, not from the marginal approximation above. A genome carrying
|
||||
several bases at once contributes to *each* base's count (counted once per
|
||||
base present, not fractionally split, not folded into one combined state)
|
||||
— a genome polymorphic for the family is present at more than one base by
|
||||
construction, so it is expected to count more than once; the denominator is
|
||||
the total base-occurrence count, not the genome count (the two coincide
|
||||
only when no genome carries more than one base). Kept side by side
|
||||
specifically to measure, on real data, how much the two diverge — not yet
|
||||
analyzed.
|
||||
|
||||
## `PersistentSparseBitMatrix` — implemented and measured (2026-08-15)
|
||||
|
||||
A row-major (k-mer-major), deduplicated sparse alternative to
|
||||
`obicompactvec::PersistentBitMatrix`, motivated by the same sparsity that
|
||||
drove `--subsample`/`--shannon` above, but pursued as a foundational
|
||||
storage-layer change rather than an index-level workaround. Full design
|
||||
history, rationale, and rejected alternatives (external Elias-Fano crates,
|
||||
`cacheline-ef`, a single unsplit `dict_id` array) are in the dedicated
|
||||
implementation plan (`vivid-mapping-tiger.md` at the time of writing — the
|
||||
content below is the durable summary, not a pointer to a session-scoped
|
||||
file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi,
|
||||
"Compact Data Structures for Collections of Sets" (2025,
|
||||
`biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf`) — this design implements only their exact-duplicate special
|
||||
case (a plain dedup dictionary), not their full subset-containment
|
||||
hierarchy.
|
||||
|
||||
**Design**: four on-disk components, each mmap-backed, built once per
|
||||
layer (matching how the rest of the build pipeline already works — never
|
||||
the whole multi-billion-row index at once): an `is_multi` rank-capable
|
||||
flag per row (singleton vs. multi-genome), a fixed-bit-width array for
|
||||
singleton rows (genome index directly, `ceil(log2(n_cols))` bits), a
|
||||
separate fixed-bit-width array for multi-genome rows (`dict_id`,
|
||||
`ceil(log2(n_distinct_multi_sets))` bits — kept apart from the singleton
|
||||
array specifically because `n_distinct_multi_sets` can be large in
|
||||
absolute terms even when multi-genome rows are a small *fraction* of all
|
||||
rows, and a single shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of
|
||||
distinct multi-genome sets (Elias-Fano-encoded byte offsets + a
|
||||
varint-encoded values blob). New low-level primitives added to
|
||||
`obicompactvec` to build this: `PersistentFixedIntVec` (arbitrary,
|
||||
runtime-parameterized bit width, width 0 included — needed once a real
|
||||
bug surfaced, see below), `PersistentRankSelectBitVec` (rank1/rank0/select1
|
||||
on top of the crate's existing `count_ones`, using
|
||||
`common_traits::SelectInWord`), `EliasFano` (composes the two). A new
|
||||
`BinaryMatrix` trait (`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`,
|
||||
`count_ones`) unifies dense and sparse at the one call site that needs
|
||||
both interchangeably (`obikphylo::siblings::cache::Mat`) — column-oriented
|
||||
methods (`col`, `col_view`, the `partial_*_dist_matrix` family) stay
|
||||
dense-only.
|
||||
|
||||
**Two real bugs caught by tests, not by inspection**: (1) `EliasFano::open`
|
||||
re-derived its low-bits width from the persisted low-vector file's own
|
||||
width byte; the zero-width case was built with a dummy 1-bit placeholder
|
||||
(the builder rejected true width 0), so every reopened value silently
|
||||
doubled. Fixed by making `PersistentFixedIntVec` genuinely support width 0
|
||||
(no storage, `get` always 0) instead of working around the limitation in
|
||||
`EliasFano`. (2) An empty row (cardinality 0 — not expected on a real
|
||||
built index, but not guarded against either) was recorded as a singleton
|
||||
at genome 0, indistinguishable on read-back from a *real* singleton at
|
||||
genome 0. Fixed by routing cardinality-0 rows through the dictionary path
|
||||
(a genuine empty entry) instead of the singleton shortcut. Both caught by
|
||||
`obicompactvec`'s test suite (142 tests, including disk-reopen round-trips
|
||||
that drop every builder/mmap before reopening fresh), not by manual
|
||||
review — worth remembering next time a "this edge case can't happen in
|
||||
practice" shortcut is tempting.
|
||||
|
||||
**Measured on real data** (`layer_1` of `phyloskims_sal_vac`'s
|
||||
`part_00018`, 30,246,774 rows, 91 genomes — `#[ignore]`d benchmarks in
|
||||
`obikphylo/src/siblings/tests.rs`):
|
||||
|
||||
| | dense | sparse | ratio |
|
||||
|---|---|---|---|
|
||||
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
|
||||
| build time / peak RSS | — | 4.26s / 628MB | (per-layer, in-memory construction — comfortable) |
|
||||
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse **faster** (smaller structure, better cache fit) |
|
||||
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** (the real `--shannon`/family-lookup shape) |
|
||||
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** (no native column method — every read decodes a full row to keep one bit) |
|
||||
|
||||
The row-access wins (both directions) weren't the design's stated goal —
|
||||
compactness was — but turn out real: dense's genome-major layout scatters
|
||||
a single row read across a much bigger file, which costs more than
|
||||
sparse's rank/select/varint decode once the file is this much smaller.
|
||||
The column-access cost is the flip side of the same layout choice, and is
|
||||
exactly what the next item below exists to fix.
|
||||
|
||||
**Next, not yet planned**: rewrite `partial_jaccard_dist_matrix`/
|
||||
`partial_hamming_dist_matrix`/etc. (`obicompactvec/src/bitmatrix/pairwise.rs`)
|
||||
as a row-major co-occurrence accumulation (`O(Σ_rows k²)`, per-row
|
||||
increments into an `NxN` genome-pair counter — the known alternative to
|
||||
today's column-fold, plausibly cheaper on data this sparse, not just a
|
||||
fallback) so `obikindex`'s `--metric`/distance-matrix path can use the
|
||||
sparse type without the measured 86x column-access penalty. Needs its own
|
||||
design pass (in particular how it plugs into the `BitPartials`/
|
||||
`ColumnWeights` traits so both matrix types keep serving `--metric`)
|
||||
before implementation — not just "port the loop", a genuinely different
|
||||
algorithm.
|
||||
|
||||
Also still deferred, unchanged from the implementation plan: full Alanko
|
||||
et al. subset-hierarchy compression (only the exact-duplicate special case
|
||||
is built), a sparse `PersistentCompactIntMatrix` (count matrices), and
|
||||
BRWT-style column-correlation exploitation.
|
||||
|
||||
## Wired into `pack` and the sibling-annex build path (2026-08-15)
|
||||
|
||||
`PersistentSparseBitMatrix` went from a validated but unused type to a
|
||||
real, selectable on-disk format:
|
||||
|
||||
- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
|
||||
(`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
|
||||
`D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
|
||||
`PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
|
||||
`LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
|
||||
already generic over any `D: LayerData`, so they needed no change.
|
||||
Verified by `obilayeredmap`'s
|
||||
`presence_layer_generic_over_sparse_matches_dense` test: build a dense
|
||||
presence layer, convert it to sparse via `build_from_dense`, open both
|
||||
as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
|
||||
the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree.
|
||||
(This test must stay at `k=4` with mutually non-colliding canonical
|
||||
4-mers across its input sequences — `K`/`M` are process-wide
|
||||
`AtomicUsize`s in test builds, not thread-local, so a test using a
|
||||
different `k` races every other test in the same crate binary; a k=11
|
||||
version of this test passed alone but failed under the full
|
||||
`obilayeredmap` suite for exactly that reason before being fixed.)
|
||||
- **`obikphylo::siblings::cache::Mat`** gained a third variant,
|
||||
`SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
|
||||
and `Presence` — every method (`find_slot`, `index_batch`,
|
||||
`iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches
|
||||
to it identically to `Presence`, since both go through the same generic
|
||||
`Layer<D>` code. `PartitionCache::build` picks the variant per layer by
|
||||
checking for `presence/is_multi.prsb` (the sparse format's own marker
|
||||
file, see the design section above) before falling back to the dense
|
||||
open path.
|
||||
- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`):
|
||||
`pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb`
|
||||
first); packs to dense `matrix.pbmx` first if that hasn't happened yet
|
||||
(the dense→sparse transpose needs random row access, which only the
|
||||
packed/columnar dense forms give), then `build_from_dense`s the sparse
|
||||
form into the same directory and deletes `matrix.pbmx` — old-format
|
||||
files are removed only after the new format is fully written, mirroring
|
||||
`pack_bit_matrix`'s own crash-safety convention.
|
||||
- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through
|
||||
`KmerIndex::pack_matrices` (all other call sites — `select`, `merge`,
|
||||
`finalize_indexed` — pass `false`, unchanged dense behaviour). Count
|
||||
matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix`
|
||||
— see "still deferred" above).
|
||||
- **End-to-end coverage**: `obikphylo::siblings::tests::
|
||||
sibling_annex_works_after_pack_sparse` builds a two-genome index, packs
|
||||
it `--sparse`, asserts `is_multi.prsb` exists, then runs
|
||||
`build_sibling_annex` and checks the resulting `FamilyMask`s match the
|
||||
dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
|
||||
sparse format round-trips through the real build pipeline
|
||||
(`PartitionCache` sparse-detection included), not just the
|
||||
`obicompactvec`/`obilayeredmap` unit layers below it.
|
||||
|
||||
Full workspace `cargo test` (all crates, unit + doc tests) green after
|
||||
this change.
|
||||
|
||||
## Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15)
|
||||
|
||||
Replaces the "four independent scans" problem above and implements
|
||||
"Entropy-biased selection" above, end to end:
|
||||
|
||||
- **`SankoffBundleExt::sankoff_bundle`** (new,
|
||||
`obikphylo/src/siblings/sankoff_bundle.rs`) — the `--sankoff`/`--tnt`/
|
||||
`--phyg`/`--iqtree` block in `obikmer/src/cmd/phylo/mod.rs` now calls
|
||||
this once instead of three separate `raw_snp_distance`/
|
||||
`base_pair_tally`/`cardinality_tally` calls. One `PartitionCache`, one
|
||||
shared (possibly subsampled/entropy-biased) selection computed once via
|
||||
`compute_selections`, two scans over it: pass A (aggregate SNP/shared
|
||||
counts, `--exclude-genome` zeroing, then `included[i,j]`), pass B
|
||||
(`base_pair_tally` + `cardinality_tally` + the pseudo-alignment, fused
|
||||
into one `scan_layer_families` call per layer, all three read off the
|
||||
same resolved `genome_mask`). `snp_pseudo_alignment`/
|
||||
`shannon_entropy_csv` (still used standalone by `--snp`/
|
||||
`--family-overlap`/`--shannon`) both gained an `entropy_bias` parameter
|
||||
too, so entropy-biased selection isn't sankoff-specific.
|
||||
- **Regression proof**: `sankoff_bundle_matches_old_separate_calls`
|
||||
(`obikphylo/src/siblings/tests.rs`) asserts `sankoff_bundle`'s four
|
||||
outputs are bit-identical to calling the old, separate
|
||||
`raw_snp_distance`/`base_pair_tally`/`cardinality_tally`/
|
||||
`snp_pseudo_alignment` on the same fixture with no subsample — the
|
||||
fusion is a performance change, not a behavior change.
|
||||
- **`EntropyAnnex`/`EntropyAnnexBuilder`** (new,
|
||||
`obikphylo/src/siblings/entropy_annex.rs`) and **`ensure_entropy_annexes`**
|
||||
(`entropy.rs`) implement the persisted-entropy mechanism from
|
||||
"Entropy-biased selection" above. `entropy_annex_builds_on_demand_and_biases_selection`
|
||||
(`tests.rs`) proves, on a fixture with one known-entropy family: the
|
||||
annex file doesn't exist before any entropy-biased call; `compute_selections`
|
||||
builds it on first use; `μ` set to the family's exact entropy with
|
||||
`p₀ = 1.0` selects it deterministically (`u < 1.0` always, for
|
||||
`u ∈ [0,1)`); `μ` set far away with tiny `σ` deterministically excludes
|
||||
it (`w` underflows to exactly `0.0`); the persisted value matches
|
||||
`--shannon`'s own `family_entropy` computation to `1e-6`.
|
||||
- **`EntropyBias`** (`pub`, `obikphylo::siblings::EntropyBias { mu, sigma }`)
|
||||
is the one new public type threading `--entropy`/`--entropy-sd` through
|
||||
every `Option<EntropyBias>`-accepting method — resolved once in
|
||||
`obikmer/src/cmd/phylo/mod.rs` from `args.entropy`/`args.entropy_sd`
|
||||
(activation: either given; defaults `1.0`/`0.5` for whichever is unset).
|
||||
|
||||
Full workspace `cargo test` green after this change (167 unit tests in
|
||||
`obicompactvec`+`obilayeredmap`+`obikphylo` alone, plus every other
|
||||
crate's suite, no regressions).
|
||||
|
||||
**Still open, not part of this change** (per "Correction to the 'single
|
||||
pass' framing" above): `--raw-snp-distance`/`--raw-snp-counts` (the
|
||||
standalone diagnostic flags, not the `--sankoff` pipeline) still always
|
||||
scan the full unsampled index — never threaded `--subsample`/`--entropy`,
|
||||
out of scope here since the reported problem was specifically about the
|
||||
`--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these
|
||||
two standalone flags.
|
||||
# Sibling annex & SNP-family pipeline — architecture
|
||||
|
||||
## Data model
|
||||
|
||||
A **family** is the up-to-4 k-mers sharing all flanks and differing only at
|
||||
the central base (well-defined for odd `k`). `FamilyMask` (`siblingannex.rs`)
|
||||
is a 4-bit presence mask (bit `b` set iff the family member whose own
|
||||
canonical central base is `b` is observed anywhere in the index) plus a
|
||||
minorant flag; `family_size()`/`siblings()` derive member count from it.
|
||||
`SiblingAnnex`/`SiblingAnnexBuilder` persist one `FamilyMask` per k-mer,
|
||||
**keyed by iteration order** (the physical layout of `unitigs.bin`), never by
|
||||
MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream
|
||||
against the annex file directly, no MPHF/slot indirection.
|
||||
|
||||
## Construction
|
||||
|
||||
`algorithms::annex::build_layer_sibling_annex` builds one layer's annex:
|
||||
seeds each k-mer's own central base, generates its up-to-3 central-base
|
||||
variants, resolves each cross-partition (`IndexCache::find_in_partition`,
|
||||
grouped and chunked per destination partition for mmap locality), writes the
|
||||
final mask (with the minorant flag) in a second streamed pass.
|
||||
`SiblingExt::build_sibling_annex` drives this for every layer of every
|
||||
partition, then invalidates and rebuilds the entropy and family-overlap
|
||||
annexes (their `family_idx` numbering depends on the sibling annex).
|
||||
|
||||
## Shared traversal
|
||||
|
||||
`algorithms::family_scan::scan_layer_families` is the **only** place that
|
||||
resolves a family's full per-genome presence (`genome_mask: &[u8]`,
|
||||
cross-partition) — every other consumer in this module is built on it, never
|
||||
reimplements the resolution. Two decoupled concurrency mechanisms: batch
|
||||
generation (CPU-only) runs on an `obipipeline` throttle/pipe; per-batch
|
||||
resolution (I/O-bound, mmap page faults) runs `rayon`-parallel across
|
||||
destination partitions, one generated batch at a time (never several
|
||||
concurrently, to keep one partition's pages hot instead of scattering
|
||||
access). `fast_mode` (`is_fast_mode`, `<= 7` layers per partition) picks
|
||||
between trusting a family member's recorded layer index outright or probing
|
||||
every layer of the destination partition.
|
||||
|
||||
`Selection<'a>` bounds which families of a layer are resolved:
|
||||
|
||||
- `All` — every minorant, monomorphic included (`SiblingExt::shannon_entropy_csv`,
|
||||
`sibling_annex_stats`).
|
||||
- `Some(&HashSet<usize>)` — a caller-picked subset, skipping the expensive
|
||||
cross-partition resolution entirely for excluded families
|
||||
(`subsample::sample_layer`'s accepted-candidates batch;
|
||||
`entropy::ensure_layer_entropy_annex`'s non-monomorphic-only restriction).
|
||||
|
||||
`minorant_selection::eligible_bitset_layer`/`non_monomorphic_reindex_layer`
|
||||
provide the cheap, annex-bits-only (no cross-partition resolution) filters
|
||||
these restricted scans are built from: monomorphism (`family_size() < 2`) is
|
||||
knowable straight from the annex, so nothing pays `scan_layer_families`'s
|
||||
resolution cost just to discard a monomorphic result.
|
||||
|
||||
## Masking — computed once, shared everywhere
|
||||
|
||||
`algorithms::masking::masked_state(m, free_loss, no_ambiguity)` is the single
|
||||
function that turns a raw per-genome 4-bit state into the masked view every
|
||||
consumer needs (`None` for absence under `--free-loss` or ambiguity under
|
||||
`--no-ambiguity`). It is called **exactly once per (family, genome)**, in
|
||||
`subsample::sample_layer`, immediately after `scan_layer_families` resolves
|
||||
the raw state — never recomputed downstream. `survives_masking` (does the
|
||||
family still carry ≥2 distinct masked states among included genomes?) and
|
||||
`SurvivingFamily::masked: Vec<Option<u8>>` both consume this same
|
||||
precomputed array, which is also what `AlignmentReducer`/`PairwiseReducer`
|
||||
read (see below) — this is what guarantees the alignment and the pairwise
|
||||
tally can never disagree about a genome's state at a given site.
|
||||
|
||||
## Precomputed annexes (entropy, family overlap)
|
||||
|
||||
`algorithms::entropy::ensure_layer_entropy_annex` builds `EntropyAnnex`
|
||||
(`entropy_annex.rs`, one `f32` Shannon-entropy value per non-monomorphic
|
||||
minorant, mmap-backed) and/or feeds `FamilyOverlapAccumulator`
|
||||
(`family_overlap.rs`) from one shared `Selection::Some` scan — both need the
|
||||
same `genome_mask` resolution, so a missing one never pays for it twice.
|
||||
`family_entropy` (15 non-empty states of `{A,C,G,T}`'s powerset, `∅`
|
||||
excluded from the denominator — the project's settled informativeness
|
||||
definition, matching the calibrated 16-state Sankoff cost matrix) and
|
||||
`family_entropy_4` (plain 4-symbol reduction, kept only for comparison) are
|
||||
pure functions of `genome_mask`. `extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes`
|
||||
drives this for every cached layer, lazily (only what's missing).
|
||||
|
||||
## Sampling and the shared scan trunk
|
||||
|
||||
`algorithms::subsample::sample_index` draws up to `n` non-monomorphic
|
||||
families index-wide, proportionally per layer: for each layer, a target
|
||||
quota sized from that layer's share of the index-wide eligible count, filled
|
||||
by a circular walk with a Bernoulli(`p`) draw per candidate — `p` optionally
|
||||
weighted by an unnormalized Gaussian kernel over the candidate's entropy
|
||||
(`EntropyBias { mu, sigma }`, read from the entropy annex). Accepted
|
||||
candidates are resolved in batches (`scan_layer_families`) and checked
|
||||
against `survives_masking`; rejected-after-masking candidates don't count
|
||||
toward quota, triggering further "topup" rounds (bounded, `MAX_TOPUP_ROUNDS
|
||||
= 5`) rather than silently under-filling. Every `SurvivingFamily` reaching a
|
||||
caller has already survived both the draw and post-masking informativeness —
|
||||
no second filter pass needed downstream.
|
||||
|
||||
`algorithms::scan::sampled_scan` is the single entry point every sampled
|
||||
computation goes through: one `sample_index` pass, fanned out to whichever
|
||||
`LayerReducer` implementations the caller actually needs, each receiving the
|
||||
exact same per-layer `SurvivingFamily` batch (an `Arc`, cloned not copied).
|
||||
This is what guarantees several outputs requested together
|
||||
(`--pseudo-alignment` and `--sankoff` and a `snp-*` `--distance` in one CLI
|
||||
invocation) come from **one** random sample, not independently-drawn ones —
|
||||
there is no other driver of `sample_index` anywhere in this crate.
|
||||
|
||||
Two `LayerReducer` implementations exist today:
|
||||
|
||||
- `AlignmentReducer` (`alignment.rs`) — accumulates `SnpAlignment` (one
|
||||
IUPAC-coded row per included genome), reading `family.masked` directly.
|
||||
- `PairwiseReducer` (`pairwise.rs`) — accumulates `PairwiseTally`/
|
||||
`PartitionDispersion`, also reading `family.masked` directly (substitution/
|
||||
agreement counts and cardinality both derive from the same masked state
|
||||
the alignment uses).
|
||||
|
||||
## From a tally to distances and calibration
|
||||
|
||||
`PairwiseTally` stores per-genome-pair substitution/agreement/cardinality
|
||||
counts, unfiltered, from which every downstream output is cheap `O(n²)`
|
||||
post-processing (never a second scan):
|
||||
|
||||
- `algorithms::snp_distance::distance_matrix` — closed-form `snp-*`
|
||||
corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions-
|
||||
only), optionally gamma-corrected (`GammaShape`, `PartitionDispersion::estimate_alpha`
|
||||
for `Auto`).
|
||||
- `algorithms::sankoff::SankoffBundle::finish` — assembles the
|
||||
`--sankoff`/`--tnt`/`--phyg`/`--iqtree` pipeline's inputs
|
||||
(`RawSnpDistanceOutput`, `BasePairTally`, `CardinalityTally`) from an
|
||||
already-scanned `AlignmentReducer`/`PairwiseReducer` pair, plus
|
||||
`SankoffBundle::snp_distance` to reuse that exact sample for a `snp-*`
|
||||
`--distance` in the same invocation without a second scan.
|
||||
- `algorithms::cardcomp` — pure functions turning `CardinalityTally`/
|
||||
`BasePairTally` into row-stochastic transition probability matrices, then
|
||||
the calibrated 16-state Sankoff cost matrix (`pairwise_cost_matrix`).
|
||||
|
||||
## Public API surface
|
||||
|
||||
`extensions::SiblingExt` (CLI-facing trait on `IndexCache`) covers only the
|
||||
operations with no cross-invocation sharing concern: `build_sibling_annex`,
|
||||
`shannon_entropy_csv`, `sibling_family_size_histogram`/`sibling_annex_stats`,
|
||||
`family_overlap`. The sampled SNP path (`sampled_scan`, `AlignmentReducer`,
|
||||
`PairwiseReducer`, `distance_matrix`, `SankoffBundle`) is **not** wrapped in
|
||||
trait methods — the CLI (`obikmer::cmd::phylo`) composes them directly,
|
||||
because it alone knows the union of reducers a given invocation needs
|
||||
(`--pseudo-alignment`/Sankoff group/`snp-*` `--distance` can combine in one
|
||||
call). A per-output trait method would have to either duplicate that
|
||||
composition logic or reintroduce the "two independent scans" problem this
|
||||
design replaced.
|
||||
|
||||
## Known gaps
|
||||
|
||||
- `sample_index`'s `skip_layers` parameter (checkpointed-resume support) is
|
||||
never called with a non-zero value by any current caller — `--session`
|
||||
persistence was removed pending a redesign on top of `sampled_scan`
|
||||
(single shared trunk, not per-driver checkpoint plumbing). The parameter
|
||||
stays as the seam a future resume mechanism would use.
|
||||
- No cheap entropy pre-filter exists yet for the very high (saturated) tail —
|
||||
a row-marginal coverage approximation was considered and rejected (cannot
|
||||
represent joint multi-base co-occurrence, which is exactly what the
|
||||
15-state entropy definition exists to capture).
|
||||
@@ -0,0 +1,94 @@
|
||||
# Benchmark: query-path testing
|
||||
|
||||
`benchmark/Makefile` exercises indexing, merge, and phylo distance
|
||||
reconstruction against simulated bacterial genomes. It now also covers
|
||||
`obikmer query` — the read-matching path — and the sparse packed
|
||||
presence-matrix format (`obikmer pack --sparse`), previously untested by
|
||||
this pipeline.
|
||||
|
||||
## Motivation
|
||||
|
||||
- `query` had no end-to-end coverage. A regression there would not be caught
|
||||
by `verify_presence`/`verify_merge_presence`, which only check index
|
||||
*content* against the `.npz` truth, never the query API.
|
||||
- `pack --sparse` produces a presence-matrix format documented (see
|
||||
[obicompactvec.md](obicompactvec.md), "Sparse matrix formats") as faster
|
||||
for single-row access (query) and slower for column-oriented access
|
||||
(phylo `--metric`).
|
||||
`global_index_presence/` built by `merge_presence.sh` is always packed
|
||||
dense (packing is a stage inside `merge`, not a separate `pack`
|
||||
invocation) — there was no dense/sparse regression check.
|
||||
|
||||
## Query read source
|
||||
|
||||
Query reads are independent of `simulated_data/` (which is folded into the
|
||||
index being queried): reusing those reads would test against the exact
|
||||
error draw the index was built from. `query_data/<species>/<strain>/` holds
|
||||
a *second*, independent `iss generate` run against the same reference
|
||||
genome, via `simulate_query_one.sh` — unseeded, so a second draw picks up
|
||||
different sequencing errors than `simulate_one.sh`'s draw for the same
|
||||
genome. Fixed at 100,000 read pairs per genome (not coverage-proportional
|
||||
like the 15x used for `simulated_data/`), so wall/RSS numbers stay
|
||||
comparable across genomes of very different sizes.
|
||||
|
||||
Two query-source specimens, hardcoded as `QUERY_SPECIMENS` in
|
||||
`make_deps.py`: `Escherichia_coli--K-12_MG1655` (common, well-represented
|
||||
bacterium) and `Saccharolobus_islandicus--M.16.4` (the only archaeon in
|
||||
`SPECIES` — distant lineage, stresses the query path differently from a
|
||||
close-relative match). Two is enough to catch a dense/sparse regression
|
||||
without duplicating the exhaustive per-specimen coverage
|
||||
`verify_merge_presence` already provides across all `SPECIMENS`.
|
||||
|
||||
## Sparse global index
|
||||
|
||||
`global_index_presence_sparse/` is built by `pack_sparse.sh`: copy
|
||||
`global_index_presence/` wholesale, then `obikmer pack --sparse` in place.
|
||||
This works directly because `merge`'s pack stage (`merge.rs:252`,
|
||||
`pack_matrices(false)`) keeps the per-genome column files on disk after
|
||||
dense-packing — `pack_sparse_bit_matrix` (`obicompactvec/src/bitmatrix/sparse.rs:447`)
|
||||
reads those, is idempotent, and removes `matrix.pbmx` once the sparse form
|
||||
is written, so `Persistent::open` falls through to the sparse format
|
||||
afterward. No separate merge run needed.
|
||||
|
||||
## Query runs
|
||||
|
||||
`query_one.sh dense|sparse SPECIMEN` runs `obikmer query --count-missing`
|
||||
against `global_index_presence` or `global_index_presence_sparse`, output
|
||||
gzipped to `query_{dense,sparse}/SPECIMEN.fasta.gz`, Reporter wall/RSS
|
||||
captured to `stats/query_{dense,sparse}/SPECIMEN.stats` (same
|
||||
stderr-parsing convention as `merge_presence.sh`).
|
||||
|
||||
Flags: `--count-missing` only. `--mismatch` is a no-op today
|
||||
(`query/mod.rs:212-213`, prints "not yet implemented, ignored") — left off
|
||||
rather than tested for a feature that doesn't exist yet.
|
||||
|
||||
## Dense/sparse regression
|
||||
|
||||
`verify_query.py` compares the two query outputs per specimen, matched by
|
||||
read id (not stream position — the query pipeline chunks input across
|
||||
worker threads and doesn't guarantee output order). Compares `kmer_count`,
|
||||
`kmer_missing`, and the full `kmer_strict_matches` map per read. Any
|
||||
mismatch is a real regression: dense and sparse must be content-identical,
|
||||
only I/O access pattern differs. `.stats` → `stats/verify_query/`,
|
||||
aggregated by `aggregate_stats.sh query|verify_query`-style cases
|
||||
(`query_dense`, `query_sparse`, `verify_query`).
|
||||
|
||||
## Performance comparison
|
||||
|
||||
No dedicated script: the wall/RSS columns from the `query_dense` and
|
||||
`query_sparse` aggregated `.stats` CSVs are the dense-vs-sparse performance
|
||||
comparison — the expected win for query on sparse, per the `pack --sparse`
|
||||
help text.
|
||||
|
||||
## Scope
|
||||
|
||||
`count` track excluded from the sparse branch: `pack --sparse` targets
|
||||
presence matrices only (per CLI help); `pack_matrices` leaves count
|
||||
matrices untouched regardless of the `sparse` flag
|
||||
(`obikindex/src/index.rs:308`).
|
||||
|
||||
## New Makefile targets
|
||||
|
||||
`simulate_query`, `pack_sparse`, `query_dense`, `query_sparse`,
|
||||
`aggregate_query_dense`, `aggregate_query_sparse`, `verify_query`,
|
||||
`aggregate_verify_query` — the last three folded into `all`.
|
||||
@@ -3,9 +3,9 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
|
||||
- `obilayeredmap/src/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
|
||||
- `obilayeredmap/src/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
|
||||
- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
|
||||
- `obikindex/src/layer/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
|
||||
- `obikindex/src/layer/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
|
||||
- `obikindex/src/reindex.rs` — KmerIndex::reindex(), conversion exact↔approx en place
|
||||
- `obikmer/src/cmd/reindex.rs` — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size
|
||||
- `obikmer/src/cmd/index.rs` — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp
|
||||
|
||||
@@ -303,7 +303,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
|
||||
|
||||
## Implementation
|
||||
|
||||
- **`obikpartitionner::filter::GroupQuorumFilter`** — implements `KmerFilter`
|
||||
- **`obikindex::partition::filter::GroupQuorumFilter`** — implements `KmerFilter`
|
||||
using pre-computed ingroup and outgroup index vectors. The heavy logic
|
||||
(predicate parsing, three-value evaluation, genome classification) happens
|
||||
once before any iteration; each k-mer row evaluation is a simple index
|
||||
@@ -314,7 +314,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
|
||||
`UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
|
||||
list.
|
||||
|
||||
- **`obikpartitionner::KmerPartition::iter_partition_kmers`** — accepts
|
||||
- **`obikindex::partition::KmerPartition::iter_partition_kmers`** — accepts
|
||||
`filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
|
||||
the callback. `filter`, `dump`, and `unitig` all go through this single
|
||||
entry point.
|
||||
@@ -1,8 +1,8 @@
|
||||
# obilayeredmap — layered kmer index crate
|
||||
# obikindex::layer — the Layer tier
|
||||
|
||||
## Purpose
|
||||
|
||||
`obilayeredmap` implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
|
||||
`obikindex::layer` (the `layer/` submodule of the `obikindex` crate — a standalone `obilayeredmap` crate until 2026-08-21, folded back in alongside `obikpartition` as part of a broader `Index { Partition { Layer } }` submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
|
||||
|
||||
---
|
||||
|
||||
@@ -250,6 +250,59 @@ Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers buil
|
||||
|
||||
---
|
||||
|
||||
## Layer\<D\> — raw mapping, iteration, and batch access
|
||||
|
||||
Beyond `query`/`find` (membership-checked), `Layer<D>` exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.
|
||||
|
||||
### Raw kmer → slot mapping
|
||||
|
||||
```rust
|
||||
pub fn index(&self, kmer: CanonicalKmer) -> usize
|
||||
pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize>
|
||||
```
|
||||
|
||||
Pure MPHF mapping, no evidence/fingerprint check — equivalent to `MphfOnly::index`. Only meaningful when the caller already knows `kmer` belongs to the layer; on an absent kmer the MPHF still returns *some* slot (undefined, not `None`).
|
||||
|
||||
### Kmer iteration
|
||||
|
||||
Four iterators, all built from `unitigs.bin` (physical layout order, **not** correlated with MPHF slot numbers):
|
||||
|
||||
```rust
|
||||
pub fn iter_kmers(&self) -> KmerIter<'_>
|
||||
pub fn enumerate_kmers(&self) -> Enumerate<KmerIter<'_>> // (order_index, kmer)
|
||||
pub fn iter_kmers_batch(&self, n: usize) -> KmerBatchIter<'_> // Vec<CanonicalKmer> of size ≤ n
|
||||
pub fn enumerate_kmers_batch(&self, n: usize) -> impl Iterator<Item = (usize, Vec<CanonicalKmer>)> + Send + 'static
|
||||
```
|
||||
|
||||
`KmerIter`/`KmerBatchIter` own a clone of the underlying `Arc<UnitigFileReader>` rather than borrowing `self` — `Send + 'static`, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. `enumerate_kmers_batch`'s index is the batch's starting offset in iteration order (a multiple of `n` except for the final, possibly shorter, batch).
|
||||
|
||||
### Batch lookup on payload vectors/views
|
||||
|
||||
`PersistentCompactIntVec`, `PersistentBitVec`, `IntSliceView`, `BitSliceView` all expose:
|
||||
|
||||
```rust
|
||||
fn get_batch(&self, slots: &[usize]) -> Vec<T>
|
||||
fn fill_batch(&self, slots: &[usize], out: &mut [T])
|
||||
```
|
||||
|
||||
Both sort `slots` internally for sequential mmap access, then reorder results back to the caller's original order. `fill_batch` fills a caller-provided buffer, avoiding the `Vec` allocation.
|
||||
|
||||
### sub_matrix / fill_sub_matrix
|
||||
|
||||
```rust
|
||||
// Layer<PersistentCompactIntMatrix>
|
||||
pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<u32>> // column-first
|
||||
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<u32>])
|
||||
|
||||
// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)
|
||||
pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
|
||||
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
|
||||
```
|
||||
|
||||
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this goes through `nonzero_iter` instead, decoding one row at a time; see [obicompactvec.md](obicompactvec.md), "Sparse matrix formats".
|
||||
|
||||
---
|
||||
|
||||
## LayeredStore\<S\> and aggregation traits
|
||||
|
||||
`LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls:
|
||||
+7
-7
@@ -1,14 +1,14 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/obilayeredmap.md
|
||||
# Coverage: implementation/layer_tier.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
|
||||
- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
|
||||
- `obilayeredmap/src/map.rs` — LayeredMap<D>, push_layer(), query()
|
||||
- `obilayeredmap/src/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
|
||||
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
|
||||
- `obilayeredmap/src/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
|
||||
- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
|
||||
- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
|
||||
- `obikindex/src/layer/map.rs` — LayeredMap<D>, push_layer(), query()
|
||||
- `obikindex/src/layer/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
|
||||
- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
|
||||
- `obikindex/src/layer/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -194,3 +194,14 @@ spectrums/
|
||||
<label>.json ← one file per genome, rebuilt from all sources
|
||||
index.meta ← complete genome list + evidence kind written at bootstrap
|
||||
```
|
||||
|
||||
`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json` marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — `merge_partition` never rewrites them for pre-existing layers, only the `presence`/`counts` subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
|
||||
|
||||
## Known issue (not yet fixed, 2026-08-28)
|
||||
|
||||
Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. `obikmer merge -o out IDX IDX --rename-duplicates`) — panics deep in the MPHF's rank-select structure (`common_traits::select_in_word`, `assertion failed: rank < self.count_ones()`), inside `MphfLayer::find` called from `merge_partition`'s "is this source kmer already in dst" check against the bootstrap-copied `dst_layers`. Root cause not identified; ruled out so far:
|
||||
|
||||
- **Not an empty-new-layer issue**: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
|
||||
- **Not caused by the hard-link change above**: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
|
||||
|
||||
Only reproduces when `sources[0]` and `sources[1]` are the exact same on-disk path opened as two independent `KmerIndex` handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since `PartitionRunner`'s panic propagation fix (see `architecture/numa_partition_runner.md`) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.
|
||||
@@ -4,9 +4,9 @@
|
||||
## Code couvert
|
||||
|
||||
- `obikindex/src/merge.rs` — `KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
|
||||
- `obikpartitionner/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
|
||||
- `obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
|
||||
- `obilayeredmap/src/layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
|
||||
- `obikindex/src/partition/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
|
||||
- `obikindex/src/partition/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
|
||||
- `obikindex/src/layer/typed_layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
|
||||
- `obicompactvec/src/intmatrix.rs` — `append_column` pour PersistentCompactIntMatrix
|
||||
- `obicompactvec/src/bitmatrix.rs` — `append_column` pour PersistentBitMatrix
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ Kmer indexing per partition proceeds in two phases. The separation is necessary
|
||||
|
||||
### Phase 1 — provisional MPHF + kmer spectrum
|
||||
|
||||
Implemented in `obikpartitionner::KmerPartition::count_kmer()` → `count_partition()`.
|
||||
Implemented in `obikindex::partition::KmerPartition::count_kmer()` → `count_partition()`.
|
||||
|
||||
1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
|
||||
2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
|
||||
@@ -101,7 +101,7 @@ type Mphf = PtrHash<
|
||||
|
||||
### Layer structure
|
||||
|
||||
Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
|
||||
Each layer is a self-contained unit. See [obikindex::layer](layer_tier.md) for the full on-disk layout. The MPHF-relevant files are:
|
||||
|
||||
```
|
||||
layer_i/
|
||||
@@ -148,7 +148,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
|
||||
|
||||
There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
|
||||
|
||||
In `obikpartitionner`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
|
||||
In `obikindex::partition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
|
||||
|
||||
### Membership verification
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
|
||||
- `obikpartitionner/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
|
||||
- `obikindex/src/layer/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
|
||||
- `obikindex/src/partition/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -0,0 +1,369 @@
|
||||
# MPHF batch queries — site inventory and expected gains
|
||||
|
||||
Investigation of 2026-10-14 (`ptr_hash` 2.1.2, `epserde` 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked **[read]** (verified in source), **[computed]** (derived from stated sizes) or **[estimate]** (reasoning, not measured). Measurements, once made, are in the last section.
|
||||
|
||||
## 1. What `ptr_hash` 2.1.2 actually offers [read]
|
||||
|
||||
`Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>`.
|
||||
|
||||
- **`V` is the pilot array** (one byte per bucket), **`F` is only the remap storage**. A layer's `mphf.bin` (~2.4 bits/key **[computed]**: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetch `pilots[bucket]` — the right line. The remap (read only when `slot >= n`, a small fraction of keys) is **not** prefetched.
|
||||
- `index_batch<K>(xs: [Q; K]) -> [usize; K]` — stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (`len % K`) is the caller's job.
|
||||
- `index_stream<B>(keys: IntoIterator) -> impl Iterator` — ring buffer of B keys; handles head and tail itself; accepts any iterator. **Its `next()` is `unimplemented!`**: only fold-based consumption works (`for_each`, `fold`, `sum`). `collect()`, `zip`, `for x in it` panic. Consume with `for_each` into a pre-sized `Vec`.
|
||||
- `index_batch_exact` needs the nightly `unstable` feature; `index_batch_exact2` is `#[doc(hidden)]`. Neither is usable.
|
||||
- Prefetch is `prefetch-index 0.2.1`: `_mm_prefetch(T0)` on x86_64, asm on aarch64, **no-op elsewhere**.
|
||||
- **`SINGLE_PART`**: neither `index_stream` nor `index_batch` branches on it — both always run the multi-part `bucket()`/`slot()`. Only `index()` has the faster single-part path. So with batched queries the `SINGLE_PART = true` advantage disappears, and `false` (our choice) costs nothing while keeping parallel construction (`true` also panics with sharding).
|
||||
- Old `ptr_hash` 1.1.0 (used by `tools/migrate-mphf`) has the same family with an extra `MINIMAL` const: `index_stream<B, MINIMAL, Q>`, `index_batch<K, MINIMAL, Q>`. No new dependency needed there.
|
||||
|
||||
```rust
|
||||
// consuming index_stream: fold-based only
|
||||
let mut out = Vec::with_capacity(kmers.len());
|
||||
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
|
||||
.for_each(|slot| out.push(slot));
|
||||
```
|
||||
|
||||
## 2. Existing "batch" APIs that are not batched [read]
|
||||
|
||||
`MphfLayer::hash_batch` and `MphfOnly::index_batch` loop `.index()` one kmer at a time. Consumers: `obikfilter/partition_iter.rs` (`iter_layer_kmers`), `obikmerge/partition_merge.rs` (pass 2: `hash_batch`, `index_batch`), `obikphylo/siblings/algorithms/family_scan.rs` (two sites).
|
||||
|
||||
## 3. Cost chain of a membership query (`MphfLayer::find`) [read]
|
||||
|
||||
1. `pilots[bucket]` — the only access `ptr_hash` prefetches.
|
||||
2. Exact/Hybrid-exact: `evidence.decode(slot)` — 4 B/slot, ~520 MB per 130 M-slot layer **[computed]**, random mmap read.
|
||||
3. `verify_canonical_kmer` → `block_offsets[...]` (O(1) with the default `--block-size 1`; a forward scan of up to `2^block_bits` records otherwise).
|
||||
4. `extract_kmer_raw` — random read in the `unitigs.bin` mmap.
|
||||
|
||||
Approx/Hybrid-fast: pilots then `fingerprint.get(slot)` — one extra random access.
|
||||
Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. `ptr_hash` only helps with stage 1; stages 2–4 need prefetch support in `obikindex`/`obiskio` (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.
|
||||
|
||||
Layer sizes: `layer_0` MPHF 2–3 MB (cache-resident), `layer_1` ~40 MB (beyond one CCD's L3 on EPYC-class machines — **hypothesis**, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on `layer_1`. For branch-free independent loops (`hash_batch`) out-of-order execution already overlaps several misses, so the explicit prefetch gains less **[estimate]** than for branchy loops (`find` with early `break`).
|
||||
|
||||
## 4. Call-site inventory
|
||||
|
||||
(a) already batched; (b) batch buildable; (c) hard/pointless.
|
||||
|
||||
| Site | What | Class |
|
||||
|---|---|---|
|
||||
| `mphf_layer.rs` `hash_batch`, `MphfOnly::index_batch` | loop of `.index()` | (a) |
|
||||
| `obikfilter/partition_iter.rs:106` | `hash_batch`, 32768 kmers | (a) |
|
||||
| `obikmerge/partition_merge.rs:489, 521` | `hash_batch`, `index_batch`, 4096 | (a) |
|
||||
| `obikphylo/.../family_scan.rs:285, 397` | `hash_batch` | (a) |
|
||||
| `obikquery/query_layer.rs` stage 1 | `find_in_layer` per kmer, layer by layer, `break` on hit | (b) |
|
||||
| `partition_merge.rs` `find_batch_grouped` | `find_slot` per kmer over dst layers | (b) |
|
||||
| `partition_merge.rs` (`build_graph` filter) | `all(find_slot.is_none())`, every layer, per kmer; `build_graph` takes `Fn(CanonicalKmer)->bool` | (b), API change |
|
||||
| `family_scan.rs` slow path; `annex.rs` | `find_slot` / `find_in_partition` per variant | (b) |
|
||||
| `mphf_layer.rs` `build`, `build_exact_evidence`, `build_approx_evidence` | `mphf.index` then random writes (evidence, fingerprint, `seen`) | (b), chunked |
|
||||
| `obikindexer/counter/count.rs:47` | `mphf.index` then `counts.get/set` | (b), hard (needs (kmer, count) pairs) |
|
||||
| `obikindexer/extensions/private_builder.rs:177` | abundance filter | (b), conditional |
|
||||
| `private_builder.rs:196`, `verify.rs`, `IndexCache::find_in_*` | per-kmer by nature / sampled | (c) |
|
||||
| `tools/migrate-mphf/plan.rs:41-42` | `old.index` + `new.index` per kmer, random writes | (b) |
|
||||
|
||||
Dead or suspect code found along the way (candidates for removal, not removed):
|
||||
|
||||
- `MphfLayer::find_strict` — no caller.
|
||||
- the single-kmer `hash` chain (`MphfLayer::hash` → `TypedLayer::hash` → `KmerLayer::hash` → `IndexCache::hash`) — no caller outside the chain.
|
||||
- `IndexCache::find` — no caller.
|
||||
- `obikmerge/src/merge_layer/` — not declared in `lib.rs` (`mod merge; mod partition_merge;` only); still imports the `LayeredMap` removed in 2026-08. To be confirmed by the compiler.
|
||||
|
||||
## 5. Interactions
|
||||
|
||||
- **Verification chain**: the right shape is one `find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>)` on `MphfLayer`, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops).
|
||||
- **Memory**: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one `Vec<usize>` per call. `index_stream` itself only allocates a B-element ring on the stack.
|
||||
- **Parallelism**: no new parallelism. Batches already come out of `obipipeline` stages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. In `query_partition_with` the unique-kmer `HashMap` is already materialised; iterate it chunk-wise.
|
||||
|
||||
## 6. Benchmark
|
||||
|
||||
`benchmark/bench_mphf.sh LABEL` times, on the 20-genome benchmark data, `index` (2 genomes), `merge` (20 presence indexes), `query` (presence/count × 2 genomes), `filter`, `convert` (approx/exact round trip) and `phylo` (`--sibling-annex`, `snp-jc`), `REPS` times each, and records order-independent output digests that must not change across labels. See the results section below.
|
||||
|
||||
**Limit of this data set**: 20 small genomes → per-partition `layer_1` MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it **cannot** measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).
|
||||
|
||||
### Throwaway prod-scale benchmark (not written)
|
||||
|
||||
Open one real partition's `layer_1` (~130 M keys) and `layer_0` (~8 M) with `MphfOnly`/`MphfLayer`; key sets: present (sample of `iter_kmers`, random order and sequential order) and absent (random, checked absent); measure kmers/s for `index`, `index_stream::<B>` (B ∈ 8…128), `index_batch::<K>` (K ∈ 8…64), and the full `find` vs staged `find_batch`; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of `mphf.bin`, `evidence.bin`, `unitigs.bin(.idx)` reported; first check `index_stream` slots == `index` slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in `mphf.md`) is a paper claim, not tied to our sizes or machines.
|
||||
|
||||
## 7. Not verified
|
||||
|
||||
No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and `--block-size` used in prod; whether `migrate-mphf` already ran; share of total index/merge/query time spent in these loops; `count.rs` f0 per partition at prod scale; whether `merge_layer/` is dead (only the head of `lib.rs` read).
|
||||
|
||||
## 8. First improvement and measurements (2026-10-15)
|
||||
|
||||
**Implemented** (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):
|
||||
|
||||
- `MphfLayer` now holds a `MphfOnly` instead of its own `MemCase<Mphf>` (one mmap-open path, one place for the stream call).
|
||||
- `MphfOnly::for_each_slot(kmers, f)` wraps `index_stream::<32>`; `index_batch`/`hash_batch` now really use it (they previously looped `.index()`). Callback-shaped because `index_stream`'s `next()` is `unimplemented!`.
|
||||
- `MphfLayer::find_batch` / `TypedLayer::find_slot_batch` / `KmerLayer::find_slot_batch`; the mode-dependent half of `find` is factored into `MphfLayer::confirm`, shared by both.
|
||||
- `query_partition_with` stage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins, `n_mphf_calls` unchanged).
|
||||
- Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive; `prefetch-index` is already in the tree via `ptr_hash`, but adding it to `obikindex` is a dependency decision), and the other (b) sites.
|
||||
|
||||
**Measurements** (`benchmark/bench_mphf.sh`, `benchmark/summarize_bench.py`; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:
|
||||
|
||||
| | wall (s) | user CPU (s) |
|
||||
|---|---|---|
|
||||
| before | 5.94–6.02 (median 5.96) | 10.95–11.07 |
|
||||
| after | 5.63–6.00 (median 5.67) | 10.33–10.66 |
|
||||
|
||||
About −5 % wall and −5.5 % CPU, ranges non-overlapping **[measured]**. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.
|
||||
|
||||
**Reading the number.** Small by construction: this data set's `layer_1` MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains **unmeasured** and needs a prod-scale layer (§6).
|
||||
|
||||
**Noise caveat.** With a load average of 16–22 from unrelated processes, single-run walls of `index`, `merge` and `filter` moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The `index`/`merge`/`filter`/`convert`/`phylo` rows of the first comparison are noise, not effects (`index` pass 2 was not changed at all; a later interleaved `filter` A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).
|
||||
|
||||
**Stale benchmark indexes.** The indexes already under `benchmark/run/` (`specimen_index_*`, `global_index_*`, built 2026-08-28) carry `ptr_hash` 1.x MPHFs and fail with "Major version mismatch" on current binaries (`tools/migrate-mphf` converts them). `bench_mphf.sh` therefore builds its own copies under `run/bench_mphf/data/` on first use.
|
||||
|
||||
## 9. Benchmark procedure
|
||||
|
||||
Prerequisites: the simulated reads (`benchmark/run/simulated_data/`) and query reads (`benchmark/run/query_data/`) produced by `gmake simulate simulate_query`; macOS BSD `time -l` and `md5 -q` (the script uses both), Python 3.
|
||||
|
||||
1. Build the binary to measure and **copy it** under a name that identifies it (cargo overwrites `target/release/obikmer`): `cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>`.
|
||||
2. `BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label>` (from `benchmark/`; `REPS=3` by default, `CPU_MAX=n` to restrict cores). The first call builds the shared data set in `run/bench_mphf/data/` with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise delete `data/` and rebuild.
|
||||
3. Output: `run/bench_mphf/<label>/results.csv` (wall and peak RSS per step and repetition, plus one row `step:stage` per Reporter stage) and `checksums.txt`.
|
||||
4. Compare: `python3 benchmark/summarize_bench.py <ref_label> <new_label>` (medians and ratio) and `diff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt` — **any difference is a bug**, not a performance matter.
|
||||
5. A ratio is only a candidate effect. Check the spread by running the **same** binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from `/usr/bin/time -l`).
|
||||
6. Rule of thumb for this data set: only `query` (≈ 5 s, stable to ±1 %) resolves effects of a few percent; `index`, `merge`, `filter`, `phylo` need interleaved runs or a quiet machine.
|
||||
|
||||
Step → code exercised:
|
||||
|
||||
| Step | Code on the MPHF path |
|
||||
|---|---|
|
||||
| `index_*` (`:index` stage) | `MphfLayer::build` pass 2 |
|
||||
| `merge_20_presence` | `find_batch_grouped`, `build_graph` filter, `hash_batch`, `index_batch` |
|
||||
| `query_*` | `query_partition_with` stage 1 |
|
||||
| `filter_species` | `iter_layer_kmers` → `hash_batch` |
|
||||
| `convert_to_*` | `build_exact_evidence`, `build_approx_evidence` |
|
||||
| `phylo_sibling_annex` | `annex.rs` `find_in_partition` per variant |
|
||||
| `phylo_snp_distance` | `family_scan.rs` `hash_batch`, `find_slot` |
|
||||
|
||||
## 10. Second step — construction pass 2 (2026-10-15)
|
||||
|
||||
**Implemented**: `layer/slotted.rs`, `SlottedKmers`, an iterator adapter yielding `(slot, kmer, chunk_id, rank)` for the wrapped `(kmer, chunk_id, rank)` stream, computing slots 4096 kmers at a time through `index_stream::<32>` (bounded memory, never the whole layer). The five loops of `MphfLayer::build` (Exact, Approx, Hybrid), `build_exact_evidence` and `build_approx_evidence` now iterate it instead of calling `mphf.index` per kmer. Output digests unchanged.
|
||||
|
||||
**Measured** (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):
|
||||
|
||||
| Command | before | after |
|
||||
|---|---|---|
|
||||
| `convert --exact-evidence` (20-genome index) | 0.56–0.68 s / 2.22–2.67 s | 0.59–0.74 s / 2.25–2.52 s |
|
||||
| `index` E. coli reads | 2.45–2.63 s / 13.1–14.3 s | 2.48–2.75 s / 13.2–14.0 s |
|
||||
|
||||
**No measurable effect** on this data set (differences within the run-to-run spread; if anything `convert` is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and `index`/`convert` time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/`seen` writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random *writes* to `evidence.bin`/`seen`/`fingerprint.bin` (not prefetched by `ptr_hash`) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.
|
||||
|
||||
## 11. Why the two changes gain so little (profiling, 2026-10-15)
|
||||
|
||||
Measured with macOS `sample` (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.
|
||||
|
||||
**Query.** In a 2 s window on the `query` stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):
|
||||
|
||||
- `query_partition_with` ≈ 260 samples (8 % of active CPU), of which `find_batch` 217, of which **`verify_canonical_kmer` 185** and everything else in `find_batch` (the `ptr_hash` stream and `confirm`) **32**.
|
||||
- `verify_canonical_kmer` is the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.
|
||||
- Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/`unitigs.bin`/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side.
|
||||
- The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (`rolling_stat`, entropy tracker, string formatting) and by sequential parts, not by lookups.
|
||||
|
||||
**What produced the −5 %.** Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer `index` (stream disabled) 6.0 s; restructuring + `index_stream` 5.85 s. **Most of the gain comes from the restructuring, not from the prefetch** (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called `IndexCache::get_layer` — an `Arc<KmerLayer>` clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.
|
||||
|
||||
**Index (pass 2).** In a 1 s window late in the `index` stage, `SlottedKmers` appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (`obidebruinj`) and ≈ 35 for `ptr_hash::build` (pass 1, MPHF *construction*). Pass 2 is a few percent of `index` and `convert` here: nothing to win even if its lookups were free. `convert` is `build_*_evidence` on 2 MB-scale layers, same conclusion.
|
||||
|
||||
**Consequences for priorities.**
|
||||
|
||||
1. On this data the next lever is the verification (`verify_canonical_kmer`, evidence decode), not the MPHF — consistent with §3, now observed.
|
||||
2. Hot `Arc` clones in per-kmer paths (`get_layer`, `IndexCache::find_in_layer`, `find_slot` loops in merge and phylo) are worth auditing independently of MPHF batching.
|
||||
3. The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).
|
||||
|
||||
## 12. Calibration: a single skim at fewer partitions (2026-10-15)
|
||||
|
||||
`Vaccinium myrtillus` skim (2 × ~440 MB fastq.gz, `theta 0`), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (`--partitions` is a power of two; MPHF size ≈ 2.4 bits/key **[measured]**):
|
||||
|
||||
| `-p` | keys / partition | `mphf.bin` / partition | index wall | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| 256 (default) | ≈ 1.4 M **[computed from p=16]** | ≈ 0.4 MB **[computed]** | – | – |
|
||||
| 16 | 22.1 M | 6.65 MB | 120 s | 12.3 GB |
|
||||
| 4 | 86–90 M | 25.9–27.0 MB | 171 s | 19.5 GB |
|
||||
|
||||
Total ≈ 354 M distinct kmers for one skim. `-p 4` is already in the 30–45 MB range of the prod `layer_1` MPHFs; `-p 2` would give ≈ 53 MB. Indexes kept in `benchmark/run/bench_mphf/skim_p16` and `skim_p4` (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.
|
||||
|
||||
## 13. Query on a real-scale index (2026-10-15)
|
||||
|
||||
**Index.** Working index `phyloskims_sal_vac_sav` (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with `tools/migrate-mphf` **on an APFS clone** (`cp -c`, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at `/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`; its sibling annex is in `stale_annexes_pre_ptrhash2/` and would need rebuilding to be used. Per partition: `layer_1` ≈ 31 M keys / 9.3 MB MPHF, `layer_0` ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB).
|
||||
|
||||
**Query** (2 × 48 MB of *V. oxycoccos* reads, `--chunk-size 8`, AC power, 3 interleaved runs per binary, identical sorted output for all three):
|
||||
|
||||
| binary | wall (s) | user CPU (s) | `query` stage (s) |
|
||||
|---|---|---|---|
|
||||
| original | 28.2–30.5 | 70.9–74.4 | 14.6–17.1 |
|
||||
| + batched query stage 1 | 26.9–28.0 | 48.1–48.6 | 13.6–14.4 |
|
||||
| + `PartitionKmers` memory fix, build-side `SlottedKmers` | 26.4–26.7 | 41.9–42.5 | 12.9–13.3 |
|
||||
|
||||
The `query` stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. **The CPU drop is much larger than the wall drop** and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer `Arc` clones of `get_layer`, not isolated); the original binary was spending CPU in contention that wall time hid.
|
||||
|
||||
**Index-open cost (not related to the changes above).** A query of a single 64-base sequence on this index takes **15 s and a 27 GB peak footprint**. `vmmap` during the open: 24.5 GB of `MALLOC_LARGE` in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by `IndexCache::new` for every layer of every partition whatever the query. Known contributors: `UnitigFileReader` reads each `.idx` into a `Vec` (≈ 30 MB for `layer_1`, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied.
|
||||
|
||||
## 14. Index-open memory: `.idx` offsets and `dict_values` mapped, not copied (2026-10-15)
|
||||
|
||||
**Cause** (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, **18.2 GB** were `unitigs.bin.idx` read into a `Vec<u32>` **twice** per exact-evidence layer (`MphfLayer::open` builds two `UnitigFileReader`s of the same file — one for the evidence check, one for `iter_kmers`/`find_strict`; both go through `open` → `open_direct_access` → `read_idx` → `fs::read` + copy) and **7.7 GB** were `dict_values.bin` read with `fs::read` in `PersistentSparseBitMatrix::open`. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (`PersistentRankSelectBitVec`, `PersistentFixedIntVec`, `EliasFano`) was already mmapped.
|
||||
|
||||
**Change.**
|
||||
- `obiskio::unitig_index::reader`: `block_offsets: Vec<u32>` → `BlockOffsets`, a view into the memory-mapped `.idx` (header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages.
|
||||
- `obicompactvec::bitmatrix::sparse`: `dict_values: Vec<u8>` → `Option<Mmap>` (`None` for an empty file, which cannot be mapped), read through `dict_bytes()`.
|
||||
|
||||
**Measured** (same session, same inputs; sorted query outputs identical, md5 `e9e7c35b`):
|
||||
|
||||
| | open only (1-sequence query) | 2×48 MB reads, `--chunk-size 8` |
|
||||
|---|---|---|
|
||||
| before | 10.6–15.7 s, **27.4 GB** footprint | 27.7 s (51 s cold), 30.9 GB, `query` stage 13.8 s |
|
||||
| after | **0.26–0.81 s, 0.04 GB** | **10.7 s, 3.6 GB**, `query` stage 9.95 s |
|
||||
|
||||
Tests: `cargo test --workspace` green; `scripts/smoke_test_index.sh` PASS; `bench_mphf.sh` (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a `.idx` map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect.
|
||||
|
||||
## 15. Unbuffered reads: `index.meta` and superkmer files (2026-10-15)
|
||||
|
||||
Follow-up of §14 (search for the same family of problems: file content handled inefficiently at open). Static search over `src/` for `fs::read`, `read_to_end`, `load_full`, `from_reader`, bare `File` readers and mmap `to_vec`: every persistent structure of `obicompactvec`, `obikindex` and the `obikphylo` annexes is already mmapped; the two real findings were *unbuffered* readers.
|
||||
|
||||
**1. `IndexMeta::read_full` (`obikindex/src/index/meta.rs`)** passed a bare `File` to `serde_json::from_reader`, which issues **one `read` syscall per byte**. `index.meta` of the 91-genome index is 20 KB and is read afresh by every `state()`/`genomes()` call (stateless design, §obikindex audit), so each call cost ≈ 0.35 s here (macOS). Profile of `obikmer dump --head 10` (`sample`): ≈ 20 000 samples in `read`, called from `Bytes<File>::next` ← `serde_json` ← `IndexMeta`. Fix: `fs::read` then `serde_json::from_slice`. **`dump --head 10`: 9.0–10.0 s (91–102 s of system CPU) → 0.16–0.84 s**; the same change applies to `SKFileMeta::read` (`obiskio/src/meta.rs`, per-superkmer-file sidecars).
|
||||
|
||||
**2. `SKFileReader::open` (`obiskio/src/reader.rs`)**: for an uncompressed file `niffler::get_reader` returns the sniffed bytes chained to the bare `File`, and `read_superkmer` reads each field with its own small `read_exact`. Fix: wrap the file in a 64 KiB `BufReader` before `niffler`. Effect on `index` (E. coli, 4 interleaved runs each): `count_kmer` 0.51–0.56 s → 0.42–0.50 s (≈ −12 %), total wall unchanged (≈ 1.95 s). Small, low risk, kept.
|
||||
|
||||
Tests: `cargo test --workspace` 549 passed / 0 failed; `scripts/smoke_test_index.sh` PASS; query output on the 91-genome index identical (md5 `e9e7c35b`).
|
||||
|
||||
**Probes of other commands on the 91-genome index (180 GB, 256 partitions)**: `phylo --distance jaccard` peaks at 155–190 MB of footprint while running and finishes in < 30 s; `utils --stats` 130–150 MB. `filter` (and by the same check `select`, `merge`) refused the index — "source index is not fully built" — with the unmodified baseline binary too: this index's `index.meta` predates the `state` field, so they could not be probed.
|
||||
|
||||
**Observations not investigated**: `index` on E. coli shows ≈ 9 s of *system* CPU in 2 s of wall time (both before and after these changes), consistent with the `swtch_pri` (yield) spinning seen in the `index` profile of §11 rather than with I/O; and wall times of the same query on the same binary varied 4.8 s ↔ 10.7 s ↔ 43 s depending on the state of the page cache, so only same-session comparisons are meaningful.
|
||||
|
||||
## 16. Profile of `index` on E. coli reads (2026-10-15)
|
||||
|
||||
`sample` (1 ms stacks) of `obikmer index` on the E. coli K-12 simulated reads (2 × 15x, 16 threads, one run; the `index` stage lasted 2.4 s under the sampler vs ≈ 0.9 s normally, so proportions are approximate). Stage walls: scatter 0.40 s, dereplicate 0.14, count_kmer 0.60, index 2.36.
|
||||
|
||||
**Threads are mostly idle.** Of ≈ 30 800 thread-samples: 24 900 blocked or yielding, 5 900 active. The ≈ 9 s of *system* CPU per run (§15) is `swtch_pri` (`sched_yield`) — 4 550 samples: ≈ 70 % rayon workers spinning for work (`WorkerThread` wait loop), ≈ 28 % `crossbeam` bounded-channel send/recv spinning, the rest in `obidebruinj` thread-locals. It is waiting, not I/O: no `read` samples remain. It costs no wall time on an otherwise idle machine, but burns CPU (visible in `user + sys` accounting and on a shared 192-core machine).
|
||||
|
||||
**Where the active CPU goes** (share of the 5 900 active samples):
|
||||
|
||||
| | share |
|
||||
|---|---|
|
||||
| xxh3 hashing of `CanonicalKmer` for the de Bruijn graph's `HashMap` (callers: `count_neighbors` 791, `hash_one` 358, `walk` 285, `unitig_iter` 252, `push` 164, `unitig_nucleotides` 75, rehash 68; plus `xxh3_stateful_update` 300) | ≈ 34 % |
|
||||
| `debruijn::count_neighbors` itself (inclusive of its hashing ≈ 2 300 samples: `compute_degrees_and_mark_starts`) | 12.5 % own |
|
||||
| `ptr_hash::build` (MPHF construction) | 4.8 % |
|
||||
| crossbeam channel send | 4.8 % |
|
||||
| `open` / `close` / `write` (many small files over 256 partitions) | ≈ 9 % |
|
||||
| debruijn `unitig_iter` / `walk` own time | 8.3 % |
|
||||
|
||||
So at this scale the cost of `index` is the **de Bruijn graph's hash table** (several lookups of neighbouring k-mers per k-mer, each hashed with xxh3 and probably missing the cache), not the MPHF passes this document started from (`SlottedKmers` ≈ 2 %, §11). Not measured: what fraction of those lookups miss the cache, and the same profile on a prod-size partition.
|
||||
|
||||
## 17. Profile of `index` on a skim: *Vaccinium vitis-idaea*, 2.1 GB, `--partitions 16` (2026-10-15)
|
||||
|
||||
One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slows the process; walls below include it): scatter 19.7 s, dereplicate 8.0 s, count_kmer 25.2 s, **index 139.1 s**; total 192 s real, 1830 s user, **225 s system** (12 % of user, against ≈ 90 % on E. coli, §16), peak footprint 17.8 GB.
|
||||
|
||||
**Index stage** (samples taken fully inside it, 91 000 active vs 187 000 idle/yielding thread-samples):
|
||||
|
||||
| share of active CPU | |
|
||||
|---|---|
|
||||
| xxh3 hashing of kmers (de Bruijn `HashMap`) | 32.7 % |
|
||||
| `debruijn::walk` | 12.4 % |
|
||||
| `debruijn::count_neighbors` | 10.7 % |
|
||||
| `GraphDeBruijn::push` | 6.9 % |
|
||||
| `crossbeam` channel | 6.6 % |
|
||||
| `hashbrown` table operations | 5.0 % |
|
||||
| `debruijn::unitig_iter` | 4.8 % |
|
||||
| memcpy/memset, malloc/free, file I/O, SipHash | ≈ 7 % |
|
||||
| other | 13.7 % |
|
||||
|
||||
- **The de Bruijn graph is ≈ 72 % of the active CPU of `index`; the MPHF construction does not appear (< 1 %)**, and `SlottedKmers`/evidence filling is negligible. The conclusion of §16 holds at 100× the E. coli scale.
|
||||
- **The stage is poorly parallel**: on average ≈ 5 of 16 threads are active (33 % of thread-samples); `swtch_pri` (spin/yield) is 7 % of all thread-samples. With 16 partitions of unequal size the tail partitions run alone. The 12 % system CPU is the same spinning as §16, at a smaller relative weight.
|
||||
- **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.
|
||||
|
||||
Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture.
|
||||
|
||||
## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)
|
||||
|
||||
Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench <unitigs.bin> <k> [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`.
|
||||
|
||||
| | push | degrees | unitigs | unitigs produced | nucleotides |
|
||||
|---|---|---|---|---|---|
|
||||
| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M |
|
||||
| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M |
|
||||
| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** |
|
||||
|
||||
Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).
|
||||
|
||||
**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision).
|
||||
|
||||
**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).
|
||||
|
||||
**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig).
|
||||
|
||||
Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).
|
||||
|
||||
**End-to-end** (current binary vs previous one, same inputs):
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) |
|
||||
| … `index` stage | 114.6 s | **59.2 s** (−48 %) |
|
||||
| … user CPU | 1 870 s | 1 304 s |
|
||||
| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB |
|
||||
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
|
||||
|
||||
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
|
||||
|
||||
## 19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15)
|
||||
|
||||
`Evidence` packs `(chunk_id, rank)` into a `u32` with a 7-bit rank (`encode` masks `& 0x7F`), but `MAX_KMERS_PER_CHUNK` was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: **`--partitions 256`** → max chunk 49–50 k-mers, results correct with the old and the new binary; **`--partitions 1`** → max chunk 256, query output wrong with both the old (md5 `bde0f886a0`) and the post-graph-fix (`8f1c94ef32`) binary versus the correct `f99e971087`. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts.
|
||||
|
||||
**Change** (`obikseq::MAX_KMERS_PER_CHUNK`: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (`SuperKmer::write_to_binary`): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test `chunk_ranks_fit_the_7_bit_evidence_field`.
|
||||
|
||||
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
|
||||
|
||||
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
|
||||
|
||||
## 20. Summary of what was gained (2026-10-15)
|
||||
|
||||
Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated.
|
||||
|
||||
### Speed
|
||||
|
||||
| Operation | Original | Now | Where |
|
||||
|---|---|---|---|
|
||||
| Open an index (query of one 64-base sequence), 91 genomes, 180 GB | 10.4–13.3 s | **0.18–0.19 s** | §14, same-session A/B |
|
||||
| Query, 2 × 48 MB reads on that index (`--chunk-size 8`) | 30.7–32.0 s (61.8 s cold), 70–76 s CPU | **11.6–13.2 s**, 36–39 s CPU (−62 % wall, −50 % CPU) | §14, same-session A/B |
|
||||
| `dump --head 10` on that index | 9.0–10.0 s (91–102 s system CPU) | **0.16–0.84 s** | §15 |
|
||||
| `index` of a 2.1 GB skim (*V. vitis-idaea*, `--partitions 16`) | 163.8 s (stage `index` 114.6 s) | **116.5 s** (stage `index` 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU | §18 |
|
||||
| `index` of E. coli reads | 1.97 s (stage 0.91 s), 10.2 s CPU | 1.63 s (stage 0.59 s), 7.3 s CPU | §18 |
|
||||
| De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) | 4.1 s / 2.0 s / 3.9 s | 1.0 s / 0.8 s / 1.0 s (×3.5 overall) | §18 |
|
||||
| `index` stage `count_kmer` (E. coli) | 0.51–0.56 s | 0.42–0.50 s | §15 |
|
||||
| Query on a small input (2 × 48 MB reads), single-genome index | 11.4 s (2 chunks = 2 busy cores) | **2.4 s** (−79 %) — automatic chunk size now follows the input size (`architecture/query.md`); on the 91-genome index the size was already 4 MiB | §20 follow-up |
|
||||
|
||||
### Memory (peak RSS / footprint)
|
||||
|
||||
| Operation | Original | Now |
|
||||
|---|---|---|
|
||||
| Opening the 91-genome index (any command that builds an `IndexCache`) | **27.4 GB** | **0.04 GB** (§14: `.idx` offsets and `dict_values.bin` mapped instead of copied; they were read twice and once) |
|
||||
| Query, 2 × 48 MB reads on that index | 32.2–32.5 GB | **3.5–3.6 GB** |
|
||||
| Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size | **73.7–80.7 GB** | **28.2 GB** (§12 of `architecture/query.md`: `PartitionKmers` instead of a `HashMap<kmer, Vec>` per partition: 71 → ~20 bytes per unique kmer) |
|
||||
| `index` of the 2.1 GB skim | 19.7 GB | 18.3 GB |
|
||||
|
||||
### Index size (newly built indexes; existing ones are unchanged)
|
||||
|
||||
| | Original | Now |
|
||||
|---|---|---|
|
||||
| `unitigs.bin` / `unitigs.bin.idx` (2.1 GB skim, 521.6 M kmers, 16 partitions) | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18) |
|
||||
| Whole index of that skim | 3.5 GB | **2.9 GB** (−17 %) |
|
||||
| `evidence.bin` | 4 bytes per kmer | unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented) |
|
||||
|
||||
### Correctness fixes found on the way
|
||||
|
||||
- Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18).
|
||||
- Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, `scripts/check_unitig_chunks.py` to audit an index).
|
||||
- `index.meta` and super-kmer files were parsed byte by byte (§15).
|
||||
|
||||
### What is **not** established
|
||||
|
||||
- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
|
||||
- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone).
|
||||
- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).
|
||||
|
||||
## 21. Dead code removed (2026-10-15)
|
||||
|
||||
The items listed as unused in §4 were removed after Serena's `safe_delete_symbol` and the compiler confirmed no reference: `MphfLayer::find_strict`; the single-kmer `hash` chain (`MphfLayer::hash`, `TypedLayer::hash`, `KmerLayer::hash`, `IndexCache::hash`); `IndexCache::find`; the uncompiled `obikmerge/src/merge_layer/` directory (689 lines; `lib.rs` only declares `merge` and `partition_merge`); and `xxhash-rust` from `obidebruinj/Cargo.toml` (it still serves `obikseq`, `obifastwrite` and `obiksession`).
|
||||
|
||||
Side effect: `find_strict` was the only reader of the `unitigs` handle that `Hybrid` layers opened, so that field (and the extra `.idx` map and mmap per Hybrid layer) was dropped too. `find` on a Hybrid layer was already a fingerprint check; the exact evidence stays on disk (it gives the slot count and is what `convert` works from) but no API consults it any more. Restoring an exact lookup for Hybrid layers means re-adding `find_strict` and that field.
|
||||
|
||||
`ahash` was not referenced by any source file of `obidebruinj` either and was removed from its `Cargo.toml` too.
|
||||
|
||||
Checks after the removal: `cargo build --release` without warnings, `cargo test --workspace` 555 passed / 0 failed, `scripts/smoke_test_index.sh` PASS.
|
||||
|
||||
Scratch artifacts of this work were deleted: the intermediate binaries, the two skim indexes (`skim_p4`, `skim_p16`) and the shared data set `benchmark/run/bench_mphf/data` (rebuilt automatically, ≈ 2 min, by the next `bench_mphf.sh`). Kept in `benchmark/run/bench_mphf/`: the original binary `obikmer_baseline` (reference for A/B runs) and the per-label results and checksums (18 MB in all). The 91-genome index converted to the current MPHF format (`/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`, with its `index.meta` patched, original `index.meta.before_state_fix` beside it) was left in place.
|
||||
@@ -292,14 +292,18 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
|
||||
## Matrix types
|
||||
|
||||
Four matrix types, two encodings × two formats:
|
||||
Both matrix types are enums behind a transparent API — the caller never matches on the variant. `PersistentCompactIntMatrix` has three variants (`Columnar`, `Packed`, `Sparse`). `PersistentBitMatrix` has four:
|
||||
|
||||
| | Columnar format | Packed format |
|
||||
| Variant | Storage | When |
|
||||
|---|---|---|
|
||||
| **Bit** | `PersistentBitMatrix` (Columnar variant) | `PersistentBitMatrix` (Packed variant) |
|
||||
| **Int** | `PersistentCompactIntMatrix` (Columnar variant) | `PersistentCompactIntMatrix` (Packed variant) |
|
||||
| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
|
||||
| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
|
||||
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off |
|
||||
| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
|
||||
|
||||
Both matrix types are enums (`Columnar` / `Packed` / `Implicit` for bit) behind a transparent API. `col_view(c)` returns the appropriate view directly:
|
||||
`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
|
||||
|
||||
`col_view(c)` returns the appropriate view directly:
|
||||
|
||||
```rust
|
||||
// PersistentBitMatrix
|
||||
@@ -315,6 +319,163 @@ No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view
|
||||
|
||||
---
|
||||
|
||||
## Sparse matrix formats
|
||||
|
||||
Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||
k-mers — present in only a handful of genome columns out of dozens to
|
||||
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||
formats above, built once per layer like the rest of the pipeline (never the
|
||||
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||
Structures for Collections of Sets" (2025,
|
||||
`biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf`) — a plain dedup dictionary, not their full subset-containment
|
||||
hierarchy.
|
||||
|
||||
### `PersistentSparseBitMatrix`
|
||||
|
||||
Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row
|
||||
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||
(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width
|
||||
array for multi-genome rows (`dict_id`, `ceil(log2(n_distinct_multi_sets))`
|
||||
bits — kept apart from the singleton array because `n_distinct_multi_sets`
|
||||
can be large in absolute terms even when multi-genome rows are a small
|
||||
*fraction* of all rows, and a shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||
blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary,
|
||||
runtime-parameterised bit width, width `0` included — needed once an
|
||||
`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec`
|
||||
(rank1/rank0/select1 on top of the crate's existing `count_ones`, via
|
||||
`common_traits::SelectInWord`), `EliasFano` (composes the two). `BinaryMatrix`
|
||||
(`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, `count_ones`) is the
|
||||
trait unifying dense and sparse at the call sites that need both
|
||||
interchangeably; column-oriented methods (`col`, `col_view`, the
|
||||
`partial_*_dist_matrix` family) stay dense-only — `Sparse`/`Implicit` panic
|
||||
there (see the `PersistentBitMatrix::open` note above).
|
||||
|
||||
Measured on real data (one layer, 30.2M rows, 91 genomes):
|
||||
|
||||
| | dense | sparse | ratio |
|
||||
|---|---|---|---|
|
||||
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
|
||||
| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction |
|
||||
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) |
|
||||
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** — the `--shannon`/family-lookup access shape |
|
||||
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit |
|
||||
|
||||
The row-access wins (compactness was the stated design goal, not speed) come
|
||||
from dense's genome-major layout scattering a single row read across a much
|
||||
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||
is this much smaller. Column access is the flip side of the same layout
|
||||
choice; `nonzero_iter` (below) is what removes it from the hot path that
|
||||
actually matters (`query`).
|
||||
|
||||
Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open`
|
||||
re-derived its low-bits width from the persisted low-vector's own width
|
||||
byte; the width-`0` case was built with a dummy 1-bit placeholder (the
|
||||
builder used to reject true width `0`), silently doubling every reopened
|
||||
value. Fixed by making `PersistentFixedIntVec` genuinely support width `0`
|
||||
(no storage, `get` always `0`). (2) An empty row (cardinality 0) used to be
|
||||
recorded as a singleton at genome `0`, indistinguishable on read-back from a
|
||||
*real* singleton at genome `0`. Fixed by routing cardinality-0 rows through
|
||||
the dictionary path (a genuine empty entry).
|
||||
|
||||
Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||
exact-duplicate case is built), a sparse `PersistentCompactIntMatrix`
|
||||
variant beyond what's described below, BRWT-style column-correlation
|
||||
exploitation, and a row-major rewrite of
|
||||
`partial_jaccard_dist_matrix`/`partial_hamming_dist_matrix`/etc. as a
|
||||
per-row co-occurrence accumulation (`O(Σ_rows k̄²)`) so `obikindex`'s
|
||||
`--metric` distance-matrix path stops paying the column-access penalty on
|
||||
`Sparse` — needs its own design pass (how it plugs into
|
||||
`BitPartials`/`ColumnWeights` so both matrix types keep serving `--metric`),
|
||||
not just "port the loop".
|
||||
|
||||
### `nonzero_iter` — the one traversal per format
|
||||
|
||||
`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a>`
|
||||
(item: index into `slots`, column, value) is the single foundational
|
||||
per-format traversal primitive every consumer builds on — `fill_sub_matrix`
|
||||
and a `for_each`-style callback are both trivial, free consumers of it
|
||||
(`.for_each(f)` is `Iterator::for_each`; `fill_sub_matrix` drains the
|
||||
iterator and scatters into `out[][]`), instead of two independently
|
||||
hand-maintained traversals that can silently diverge. `Box<dyn Iterator>`,
|
||||
not a bare `impl Iterator`: `Columnar`/`Packed`/`Sparse`/`Implicit` are
|
||||
genuinely different concrete types and the method isn't on a
|
||||
dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix`
|
||||
elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per
|
||||
`nonzero_iter` call, not per cell.
|
||||
|
||||
Per-format implementation: `PersistentSparseBitMatrix` decodes natively,
|
||||
buffered one row at a time (`std::iter::from_fn` over the existing
|
||||
`for_each_genome_in_row`, no `n_cols`-wide allocation ever). `Columnar`/
|
||||
`Packed` loop columns, collecting each column's hits via
|
||||
`BitSliceView::nonzero_among_sorted`/`IntSliceView::nonzero_among_sorted`
|
||||
(a `filter`/`filter_map` over the vector-level `get()`, no new state
|
||||
machine — `std`'s own `Filter` iterator already is one) into one `Vec` via
|
||||
`.extend()` (not `flat_map`: a `flat_map` closure can't lazily return
|
||||
something borrowing its own captured sort permutation across separate calls
|
||||
without either boxing per-column or fighting the borrow checker — eager
|
||||
collection sidesteps it at zero extra cost, since `fill_sub_matrix` already
|
||||
fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))`).
|
||||
|
||||
This closed a real dispatch bug by construction, not by patching it:
|
||||
`PersistentBitMatrix::fill_sub_matrix`/`sub_matrix` used to reimplement a
|
||||
naive per-`(column, slot)` loop instead of delegating to
|
||||
`PersistentSparseBitMatrix`'s own efficient native method for `Self::Sparse`
|
||||
— two hand-written traversals for the same format, free to drift apart, and
|
||||
they had (`obikphylo`'s own cache layer had independently built a parallel
|
||||
enum wrapper that *did* dispatch correctly, evidence the bug was worked
|
||||
around rather than fixed at the source). With `fill_sub_matrix`/`sub_matrix`
|
||||
rewritten to drain `nonzero_iter`, there is only one traversal per format
|
||||
left to get right. `KmerPartition::query_partition_with`
|
||||
(`obikindex/src/partition/query_layer.rs`) — previously column-major
|
||||
(`for g { for slot { col_value } }`, which for `Sparse` rebuilt a full row
|
||||
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||
motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)`
|
||||
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||
reproducibly; now within ~1-3% either way — noise level).
|
||||
|
||||
### `PersistentSparseCompactIntMatrix`
|
||||
|
||||
The count-matrix analogue, built on `PersistentSparseBitMatrix` as its
|
||||
"which columns are non-zero" support, values **not** deduplicated (unlike
|
||||
the bit side, two rows can share the same non-zero column set — same
|
||||
`dict_id` — while carrying different counts, so the dict-multiplicity
|
||||
shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry
|
||||
over). `CountPartials` here: a single row-major pass (`row_major_pairwise`
|
||||
in `sparse_intmatrix.rs`, decodes each row once via
|
||||
`for_each_cell_in_row`, nests over that row's own co-present columns) —
|
||||
`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)`
|
||||
column-pair rescan. `min(a,b)`-based kernels (`bray`, `relfreq_bray`) need
|
||||
no correction — they vanish when either side is absent. `a·b` and `√(a·b)`
|
||||
kernels (`euclidean`/`relfreq_euclidean`/`hellinger`) do need one, since
|
||||
`(a-0)² = a² ≠ 0` unlike the `min`-based formulas — reconstructed from
|
||||
per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`.
|
||||
`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard`
|
||||
(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form
|
||||
(every `u32` is `≥ 0`).
|
||||
|
||||
Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in
|
||||
`BitPartials for PersistentSparseBitMatrix` (present since that type's own
|
||||
introduction, never caught because no test had compared `Sparse`'s raw
|
||||
`partial_*` output against dense on real data — only the diagonal-blind
|
||||
`jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested),
|
||||
fixed at the source:
|
||||
|
||||
1. `partial_jaccard`'s diagonal used to be `(0, 2×col_weights[i])` instead
|
||||
of a genuine self-comparison `(col_weights[i], col_weights[i])` —
|
||||
`col_weights_and_pair_counts`'s `inter` never pairs a column with itself
|
||||
by construction, so the diagonal needs its own case.
|
||||
2. `partial_hamming`'s off-diagonal formula itself was wrong: it computed
|
||||
`total - union` (rows where *neither* column is present) instead of the
|
||||
actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]`
|
||||
(symmetric-difference size) — the two only coincide when
|
||||
`col_weights[i] + col_weights[j] == total`, which small/synthetic test
|
||||
data can easily satisfy by accident.
|
||||
|
||||
## Aggregation traits (matrix level)
|
||||
|
||||
### ColumnWeights
|
||||
|
||||
@@ -9,5 +9,5 @@
|
||||
## Notes
|
||||
|
||||
Document stable (librairie générique, peu de risque de dérive).
|
||||
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartitionner`
|
||||
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikindex::partition`
|
||||
ou s'il a été remplacé par Rayon dans certains chemins.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -203,7 +203,7 @@ part_XXXXX/
|
||||
|
||||
**Cleanup:** unless `--keep-intermediate` is set, `remove_build_artifacts` deletes `dereplicated.skmer.zst`, `mphf1.bin`, and `counts1.bin` after all partitions are indexed.
|
||||
|
||||
See [obilayeredmap](obilayeredmap.md) and [MPHF selection](mphf.md) for data structure details.
|
||||
See [obikindex::layer](layer_tier.md) and [MPHF selection](mphf.md) for data structure details.
|
||||
|
||||
**Query path (exact evidence):**
|
||||
|
||||
|
||||
@@ -3,12 +3,12 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikpartitionner/src/partition.rs` — estimation des paramètres (phase 0)
|
||||
- `obikindex/src/partition/partition.rs` — estimation des paramètres (phase 0)
|
||||
- `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
|
||||
- `obikpartitionner/src/filter.rs` — déduplication bucket-sort (phase 2)
|
||||
- `obikpartitionner/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
|
||||
- `obikindex/src/partition/filter.rs` — déduplication bucket-sort (phase 2)
|
||||
- `obikindex/src/partition/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
|
||||
- `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
|
||||
- `obikpartitionner/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
|
||||
- `obikindex/src/partition/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
|
||||
- `obikindex/src/index.rs` — `build_layers()`, `dereplicate_and_count()`
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -23,28 +23,37 @@ are preserved unchanged; only the data matrices are rewritten.
|
||||
|
||||
```sh
|
||||
obikmer select <input-index>
|
||||
{ --output <dir> | --in-place }
|
||||
--output <dir>
|
||||
[--group <name>:<pred> ...]
|
||||
[--group-op <name>:<op> ...]
|
||||
[--aggregate-by <key> ]
|
||||
[--aggregate-op <op> ]
|
||||
[--select <col1,col2,...> ]
|
||||
[--presence-threshold <N> ]
|
||||
[--force-copy ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output destination
|
||||
|
||||
Exactly one of `--output` or `--in-place` must be specified.
|
||||
`--output <dir>` is required — `select` always writes a new index; there is no
|
||||
`--in-place` mode (2026-08-28: never implemented, removed from the design). The
|
||||
source index is unchanged.
|
||||
|
||||
**`--output <dir>`** — writes a new index to `<dir>`. The source index is
|
||||
unchanged. The MPHF and unitig files are copied; only the data matrices are
|
||||
rewritten with the new column layout.
|
||||
Each layer's kmer-identity files (`mphf.bin`/`unitigs.bin`/`evidence.bin`/
|
||||
`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json`) are never rewritten by a
|
||||
column projection/aggregation, so they are hard-linked into the output rather
|
||||
than copied — no extra disk for them even on a large index. Falls back to a
|
||||
real copy automatically if linking fails (different filesystems); `--force-copy`
|
||||
forces a real copy always, for an output that must survive independently of the
|
||||
source on disk (a hard link shares the same inode — rewriting one path outside
|
||||
`select` itself would affect the other). Only the `presence`/`counts`
|
||||
subdirectory is ever a genuinely new, independent file.
|
||||
|
||||
**`--in-place`** — rewrites the data matrices of the source index directly.
|
||||
Removed or replaced columns are lost. The operation writes to temporary files
|
||||
first, then renames atomically, so an interrupted run leaves the index intact.
|
||||
To replace an index with a selected version of itself, select to a temporary
|
||||
directory and swap it in (`rm -rf INDEX && mv INDEX.tmp INDEX`) — the case
|
||||
`--in-place` used to cover.
|
||||
|
||||
---
|
||||
|
||||
@@ -186,12 +195,6 @@ obikmer select myindex --output out \
|
||||
--select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
|
||||
```
|
||||
|
||||
### In-place: keep only group A
|
||||
|
||||
```sh
|
||||
obikmer select myindex --in-place --group "A:group=A" --select "A"
|
||||
```
|
||||
|
||||
### Compose with filter
|
||||
|
||||
```sh
|
||||
@@ -207,28 +210,45 @@ obikmer select filtered --output final --aggregate-by site
|
||||
|
||||
## Implementation notes
|
||||
|
||||
`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
|
||||
(rayon), each writing its output independently; results require no synchronisation
|
||||
because every partition owns a distinct set of files.
|
||||
`select` does not rebuild the MPHF. Every partition is processed independently
|
||||
(`PartitionRunner`), each writing its own output layers; no cross-partition
|
||||
synchronisation is needed.
|
||||
|
||||
For each layer in each partition:
|
||||
For each layer in each partition (`obikselect::select_layer::select_partition`):
|
||||
|
||||
1. The slot count `n` is read by opening the source data matrix.
|
||||
2. A new data matrix is built with M columns (M = number of output columns).
|
||||
3. For each slot `s` in `0..n`:
|
||||
- `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
|
||||
- For each output column `j`:
|
||||
- `new_row[j] = aggregate(op, old_row[group_indices])`.
|
||||
- Pass-through columns are represented as single-element groups with the
|
||||
default operator (`any` for presence, `sum` for count) — same code path.
|
||||
- The new row is written slot by slot into each column builder.
|
||||
4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
|
||||
evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
|
||||
or `counts/` subdirectory is rewritten.
|
||||
1. `copy_layer_files` hard-links the source layer's kmer-identity files
|
||||
(`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/
|
||||
`fingerprint.bin`/`layer_meta.json`) into the destination — never a real
|
||||
copy unless linking fails or `--force-copy` is given.
|
||||
2. A new data matrix is built with M columns (M = number of output columns),
|
||||
under a fresh `presence/`/`counts/` subdirectory (never touching the
|
||||
source's own).
|
||||
3. **Presence source (2026-08-28: `batch_presence_counts`)**: one shared pass
|
||||
over the source bit matrix computes every output group's presence count at
|
||||
once — row-major native for a `Sparse` source (`for_each_genome_in_row`,
|
||||
which has no column representation to read a `col_view` from at all — the
|
||||
reason this replaced the old per-group loop, not just an optimisation of
|
||||
it), deduplicated column-major (one `col_view` per *distinct referenced
|
||||
column*, not per group) for `Columnar`/`Packed`. Every `AggOp` for a bit
|
||||
matrix is then a cheap derivation of that one count vector (`sum` = the
|
||||
count itself, `any`/`max` = `count ≥ 1`, `all`/`min` = `count == group
|
||||
size`, `none` = `count == 0`) — see
|
||||
`obikselect::select_layer::agg_result_from_count`.
|
||||
4. **Count source**: unchanged, one `col_view`-driven pass per output column
|
||||
via `MatrixGroupOps` — `sum`/`min`/`max` are genuine per-value reductions
|
||||
for a count matrix, not derivable from a single presence count the way
|
||||
they are for a bit matrix.
|
||||
5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
|
||||
|
||||
**`--in-place` write strategy:** new data is written to a temporary sibling
|
||||
directory (`presence_new/` or `counts_new/`); on success the old directory is
|
||||
removed and the temporary one is renamed into place. An interrupted run leaves
|
||||
at most one stale `*_new/` directory; the original data is intact until the
|
||||
rename step.
|
||||
### Known gap (not yet fixed, 2026-08-28)
|
||||
|
||||
Step 4 above still panics (`col_view() not available on Sparse
|
||||
PersistentCompactIntMatrix`) if the source is a **count** index packed
|
||||
sparse — `batch_presence_counts`' row-major treatment was only ported to the
|
||||
bit-matrix (`Presence`) case, since that was the one actually blocking a real
|
||||
benchmark run. `select`/`filter` on a sparse-packed count index still hits
|
||||
this; the fix would follow the same shape (a `PersistentSparseCompactIntMatrix`
|
||||
row-major decode, analogous to `for_each_genome_in_row`), just not done. Since
|
||||
`obisys::numa::runner::PartitionRunner`'s panic-propagation fix (see
|
||||
`architecture/numa_partition_runner.md`), this at least fails fast (process
|
||||
panic, exit 101) instead of hanging.
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
- `obikindex/src/meta.rs` — IndexMeta, IndexConfig (version, config, genomes)
|
||||
- `obikindex/src/index.rs` — layout sur disque : partitions/, index.meta
|
||||
- `obilayeredmap/src/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
|
||||
- `obikindex/src/layer/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
|
||||
- `obiskio/src/unitig_index.rs` — fichiers unitigs.bin + unitigs.bin.idx
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -22,10 +22,10 @@ A sequence of binary records. Each record:
|
||||
- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
|
||||
- Byte count for packed sequence: `ceil(seql / 4)`.
|
||||
|
||||
Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
|
||||
Unitigs with more than `MAX_KMERS_PER_CHUNK = 128` k-mers are transparently split into overlapping chunks. Each chunk has at most 128 k-mers (= `seql − k + 1 ≤ 128`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
|
||||
|
||||
```
|
||||
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (256 kmers)
|
||||
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (128 kmers)
|
||||
chunk 2: nucleotides [256, end] (remaining kmers)
|
||||
overlap: k−1 nucleotides shared between the two chunks
|
||||
```
|
||||
@@ -95,11 +95,11 @@ Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extract
|
||||
|
||||
| field | bits | range | capacity check (*B. nana*, 256 partitions) |
|
||||
|------------|------|---------------|---------------------------------------------|
|
||||
| `seql − k` | 8 | 0–255 | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` |
|
||||
| `seql − k` | 8 | 0–127 used | max `n_kmers` per chunk = 128 = `MAX_KMERS_PER_CHUNK` (the field could carry 256; capped so that every rank fits 7 bits) |
|
||||
| `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
|
||||
| `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition |
|
||||
|
||||
The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total.
|
||||
The rank field is 7 bits (max 127) and the rank of a k-mer is its index among the chunk's k-mers, so a chunk must hold at most 128 of them: that is what `MAX_KMERS_PER_CHUNK = 128` guarantees. (Until 2026-10-15 the cap was 256 and the earlier justification here — that rank counts only canonical k-mers, at most half the total — was wrong: `iter_indexed_canonical_kmers` yields one entry per position. Chunks of 129–256 k-mers occur with few partitions — 6 of them in a `--partitions 4` index of a 2 GB skim, none with 256 partitions where the observed maximum is ≈ 50 — and their k-mers beyond rank 127 were silently stored with a truncated rank. `scripts/check_unitig_chunks.py` finds such chunks in an existing index.)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
## Code couvert
|
||||
|
||||
- `obiskio/src/unitig_index.rs` — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()
|
||||
- `obilayeredmap/src/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
|
||||
- `obikindex/src/layer/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
|
||||
- `obidebruinj/src/debruijn.rs` — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# Chance sharing of a kmer across genomes (discussion)
|
||||
|
||||
Null model for "a kmer is present in $r$ of $S$ genomes": how likely is this by chance alone? Extends the single-pair estimate $P_{\mathrm{chance}}\simeq G/4^k$ (see user doc, *K-mer specificity*).
|
||||
|
||||
## Homogeneous case: binomial tail
|
||||
|
||||
If every genome has the same per-genome chance probability $p$, the number $R$ of genomes containing a given kmer is $\mathrm{Bin}(S,p)$:
|
||||
|
||||
$$P(R\geq r)=\sum_{j=r}^{S}\binom{S}{j}p^j(1-p)^{S-j}=I_p(r,\,S-r+1)$$
|
||||
|
||||
where $I_p$ is the regularized incomplete beta function. No elementary closed form; use $I_p$ directly.
|
||||
|
||||
### Numerical evaluation of the sum
|
||||
|
||||
- Term recurrence, one `lgamma` start: $t_{j+1}=t_j\cdot\frac{S-j}{j+1}\cdot\frac{p}{1-p}$.
|
||||
- Sum the tail **not containing the mode** ($Sp$, not $S/2$). If $r>Sp$, sum $\sum_{j\ge r}$ directly and stop when terms are negligible. The complement $1-\sum_{j=0}^{r-1}$ (note: from $j=0$) suffers catastrophic cancellation when the p-value is tiny.
|
||||
- Large $S$: normal approximation with continuity correction, or Poisson($Sp$) when $p$ is small. Unreliable in extreme tails.
|
||||
|
||||
## Heterogeneous case: $p_i = f(G_i)$
|
||||
|
||||
The $G_i$ (number of kmers of genome $i$) are known and differ. With $p_i\simeq G_i/4^k$, the indicators "kmer present in genome $i$ by chance" are independent Bernoulli($p_i$), so $R$ follows a **Poisson-binomial** law, not a binomial:
|
||||
|
||||
$$P(R=j)=\sum_{|A|=j}\prod_{i\in A}p_i\prod_{i\notin A}(1-p_i).$$
|
||||
|
||||
Everything lives in canonical space: $G_i$ is the number of distinct canonical kmers of genome $i$, and the kmer space has size
|
||||
|
||||
$$N=\frac{4^k}{2}$$
|
||||
|
||||
exactly, since $k$ is odd (no kmer equals its own reverse complement, so canonicalization pairs the $4^k$ kmers perfectly). Exact per-genome probability: $p_i=1-(1-1/N)^{G_i}\simeq 2G_i/4^k$. This is twice the $G/4^k$ of the user doc, where $G$ is a number of positions on one strand.
|
||||
|
||||
### Exact computation
|
||||
|
||||
Dynamic programming over genomes, $d_j$ = probability of $j$ hits so far:
|
||||
|
||||
$$d'_j=d_j(1-p_i)+d_{j-1}p_i.$$
|
||||
|
||||
For the upper tail only $d_0\ldots d_{r-1}$ are needed: $P(R\ge r)=1-\sum_{j<r}d_j$, cost $O(S\,r)$, per distinct $(p_1..p_S)$ vector — independent of the kmer, so computed once per index (table of $P(R\ge r)$ for $r=1..S$), not per kmer. Cancellation caveat as above: for tiny tails compute $\Pr(R\ge r)$ by the DP on the upper side (track $d_j$ for $j\ge r$ only from a top-down recurrence) or in log space. Alternative: DFT of the characteristic function (Hong 2013), $O(S^2)$ exact.
|
||||
|
||||
### Approximations
|
||||
|
||||
- **Poisson**: $R\approx\mathrm{Poisson}(\lambda=\sum_i p_i)$ with total variation error $\le\sum_i p_i^2$ (Le Cam). Excellent when all $p_i\ll1$, i.e. $4^k\gg G_i$ (large $k$).
|
||||
- **Binomial with mean $\bar p=\frac1S\sum_i p_i$**: Hoeffding (1956) shows the Poisson-binomial is more concentrated than $\mathrm{Bin}(S,\bar p)$. Hence for $r\ge S\bar p+1$, $P_{\mathrm{PB}}(R\ge r)\le P_{\mathrm{Bin}}(R\ge r)$: using the mean $\bar G$ overestimates the chance probability, i.e. is **conservative** in the upper tail. Wrong direction for $r\le S\bar p$ (lower tail), irrelevant for significance testing.
|
||||
|
||||
Consequence: assuming a common $G$ is safe (conservative) if $G=\bar G$ (arithmetic mean), but loses power when genome sizes span orders of magnitude; the exact DP is cheap enough (once per index) to make the approximation unnecessary. Small genomes with $k=11$ saturate ($p_i\to1$) and break the Poisson approximation, not the DP.
|
||||
|
||||
## Open points
|
||||
|
||||
- Repeats are not a problem. The model is presence/absence per genome, so copy number within a genome is irrelevant, and $G_i$ is the number of **distinct** kmers of genome $i$. A repeat shared between genomes is paralogy, i.e. homology: a true positive, not a false one. Low-complexity kmers (which would share by composition, not descent) are removed upstream by the [entropy filter](entropy.md).
|
||||
- Residual violation of the equiprobability behind $p_i$: nucleotide composition bias (see Karlin–Ost in the user doc).
|
||||
- Test dependence between overlapping kmers (see FDR section).
|
||||
|
||||
## Goal: an FDR of homology
|
||||
|
||||
$H_0$: the $S$ genomes are unrelated (independent random sequences of sizes $G_i$). Rejecting $H_0$ for a kmer, i.e. sharing that chance cannot explain, is the evidence of homology. The quantity wanted is the false discovery rate of the call "shared kmer $\Rightarrow$ homologous".
|
||||
|
||||
### Empirical FDR from the count histogram
|
||||
|
||||
Let $N$ be the size of the kmer space and $n_{\ge r}$ the observed number of distinct kmers present in at least $r$ genomes (a by-product of the index). Under $H_0$ the expected number of kmers with $R\ge r$ over the whole space is exactly $N\,P(R\ge r)$, with $P$ the Poisson-binomial tail above. Hence
|
||||
|
||||
$$\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P(R\ge r)}{n_{\ge r}}\right).$$
|
||||
|
||||
- No per-kmer multiple-testing correction: the number of tests ($N$) is already in the numerator.
|
||||
- Conservative: the observed $n_{\ge r}$ mixes true homology and chance, and the true-null proportion $\pi_0\le1$ is taken as $1$ (Storey-type upper bound).
|
||||
- Take the smallest $r$ with $\widehat{\mathrm{FDR}}(r)\le\alpha$ as the sharing threshold.
|
||||
- Equivalent per-kmer view: BH on $p=P(R\ge r)$ with $N$ tests. The zero-truncation matters if one tests only kmers present in the index ($R\ge1$ by selection): use $P(R\ge r)/P(R\ge1)$, not $P(R\ge r)$, otherwise the test is too conservative by a factor $\sim1/\lambda$.
|
||||
|
||||
### Dependence
|
||||
|
||||
Kmers of one shared region overlap and are not independent tests. The FDR above holds per kmer; for a per-region FDR, count regions (unitigs / super-kmers, with the appropriate clumping of $N$ and $n_{\ge r}$) rather than kmers. Not settled.
|
||||
@@ -1489,7 +1489,7 @@ instead of a per-partition `partial_*`, run the sequential source sweep:
|
||||
|
||||
```text
|
||||
for p in 0..n_partitions: # OUTER — sequential
|
||||
open source partition p's layers (QueryLayer-style, obikpartitionner)
|
||||
open source partition p's layers (QueryLayer-style, obikindex::partition)
|
||||
enumerate distinct canonical k-mers of p (one per MPHF slot) with their
|
||||
presence/count vectors # column-major, as query stage 2
|
||||
par_iter over these source k-mers: # INNER — rayon, thread-local tally
|
||||
@@ -1513,7 +1513,7 @@ for p in 0..n_partitions: # OUTER — sequential
|
||||
```
|
||||
|
||||
The inner lookup is precisely `QueryLayer::find_slot` +
|
||||
`col_value(g, slot)` (`obikpartitionner/src/query_layer.rs`) — reuse or factor
|
||||
`col_value(g, slot)` (`obikindex/src/partition/query_layer.rs`) — reuse or factor
|
||||
out that path rather than reimplementing MPHF access. Enumerating "all distinct
|
||||
k-mers of a partition with their vectors" is the `dump`/`query` stage-2
|
||||
column-major scan already implemented in `dump_layer.rs` /
|
||||
@@ -2182,6 +2182,697 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
|
||||
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
|
||||
workspace `cargo test` green.
|
||||
|
||||
## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
**Implemented.** `--metric` (renamed `--distance` — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
`--metric` was a misnomer) gains a family of `snp-*` values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
machinery (`PairwiseTally`, `obikphylo::siblings::algorithms::pairwise`)
|
||||
instead of `cache.distance(...)`'s existing per-layer traversal — a different
|
||||
code path behind the same CLI surface, not just another branch of one
|
||||
formula function.
|
||||
|
||||
**Why unify at the CLI level despite the implementation split**: phylogenetically
|
||||
a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
|
||||
to how the matrix was produced, so exposing it as a special-cased subcommand
|
||||
instead of a `--distance` value would misrepresent its role. The
|
||||
implementation divergence (sibling-annex-based vs. plain index scan) is real
|
||||
but belongs at the routing layer, invisible to the CLI's own vocabulary.
|
||||
|
||||
**`--subsample` becomes optional for `snp-*` distances** (it stays mandatory
|
||||
for `--sankoff`/`--pseudo-alignment`, unrelated commands): absent means
|
||||
exhaustive, achieved for free by reusing `sample_index`'s existing
|
||||
proportional-per-layer-quota mechanism with `n` set to the index-wide total
|
||||
non-monomorphic-minorant count (already available from the sibling-annex
|
||||
stats) — every layer's quota then equals its own full count, giving Bernoulli
|
||||
`p = 1` everywhere, i.e. every eligible family is drawn. No second,
|
||||
exhaustive-only driver needed. Present means sampled, exactly as `--sankoff`
|
||||
already behaves.
|
||||
|
||||
**One shared tally, many derived formulas.** `PairwiseTally`'s `subst[4][4]`
|
||||
per-pair substitution counts (plus marginal base frequencies derived from it)
|
||||
are the sufficient statistic for every closed-form correction below — each
|
||||
is a small pure function `PairwiseTally -> Array2<f64>`, at the same level as
|
||||
the already-implemented `raw_snp_distance`/`base_pair_tally`/
|
||||
`cardinality_tally`. No new full scan per formula, whether the tally itself
|
||||
was built exhaustively or from a subsample.
|
||||
|
||||
**`--raw-snp-counts` stays a separate, unrelated flag** — same underlying
|
||||
tally, but a diagnostic (`n_snp`/`n_shared`/`n_eligible` per genome pair, one
|
||||
row per pair) rather than a distance value, and its long-table shape doesn't
|
||||
fold into a single N×N matrix the way a distance does. No change to its
|
||||
existing CSV format.
|
||||
|
||||
### `snp-*` distance catalog
|
||||
|
||||
All closed-form (method-of-moments / direct formula), none requiring
|
||||
per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
|
||||
*tree*-ML usage but not its *pairwise* estimator, which is closed-form like
|
||||
F84/TN93 and is included below. `snp-` prefix on every CLI value.
|
||||
|
||||
| value | corrects for | inputs beyond raw counts |
|
||||
|---|---|---|
|
||||
| `snp-raw` | nothing (uncorrected p-distance) | — |
|
||||
| `snp-jc` (Jukes-Cantor, JC69) | multiple substitutions per site | — |
|
||||
| `snp-k2p` (Kimura 2-parameter, K80) | + transition/transversion rate bias | ts/tv split |
|
||||
| `snp-k81` (Kimura 3-parameter, K3ST) | + splits transversions into 2 categories | ts/tv split, by category |
|
||||
| `snp-f81` (Felsenstein 81) | + unequal base frequencies (no ts/tv split) | empirical base freqs |
|
||||
| `snp-tajima-nei` (Tajima-Nei 1984) | same goal as F81 (equal-input model), different formula, better small-sample behavior | empirical base freqs |
|
||||
| `snp-t92` (Tamura 3-parameter) | K2P + GC-content bias | ts/tv split, GC content |
|
||||
| `snp-f84` (Felsenstein 84) | full empirical base freqs + single ts/tv rate | empirical base freqs, ts/tv split |
|
||||
| `snp-hky85` (Hasegawa-Kishino-Yano, pairwise estimator) | same inputs as F84, different formula | empirical base freqs, ts/tv split |
|
||||
| `snp-tn93` (Tamura-Nei) | full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate | empirical base freqs, purine-ts/pyrimidine-ts/tv split |
|
||||
| `snp-logdet` (LogDet / paralinear) | no shared-model or stationarity assumption at all — general divergence-matrix determinant | full empirical 4×4 divergence matrix (already `subst[4][4]`) |
|
||||
| `snp-tv` (transversions-only p-distance) | diagnostic/deep-divergence variant — drops transitions entirely (they saturate first) | tv-only counts |
|
||||
|
||||
**`+Γ` rate-heterogeneity modifier, applicable to `snp-jc`, `snp-k2p`,
|
||||
`snp-k81`, `snp-t92`, `snp-f84`, `snp-hky85`, `snp-tn93`** (not `snp-raw`,
|
||||
nothing to correct; not `snp-logdet`, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter `α` either
|
||||
supplied by the user (`--gamma-shape <alpha>`) or estimated from the data
|
||||
(`--gamma-shape auto`/`estimate`, method-of-moments — not ML; see
|
||||
"Automatic α estimation" below). A modifier on existing values, not a
|
||||
separate enum arm per distance.
|
||||
|
||||
### Automatic α estimation (`--gamma-shape auto`)
|
||||
|
||||
**Correction (verified against the primary source, 2026-09-11):** Jin &
|
||||
Nei (1990) itself (*"Limitations of the Evolutionary Parsimony Method of
|
||||
Phylogenetic Analysis"*, Mol. Biol. Evol. 7(2):82–102 — the paper this
|
||||
whole `+Γ` correction is cited from, confirmed algebraically to match this
|
||||
codebase's `corrected_log`/`k2p` exactly against their eq. A4, general, and
|
||||
A8, the `a = 1` case) contains **no** data-driven α-estimation procedure.
|
||||
Their own recommendation (p. 98) is a fixed default: *"we suggest that the
|
||||
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
|
||||
different gamma distance, estimating a from data. Wilson et al. (1989)
|
||||
recently used a distance with a = 1/2 for restriction-site data of
|
||||
mitochondrial DNA in hominoids."* — i.e. Jin & Nei explicitly punt
|
||||
data-driven estimation to a *different* paper (Wilson et al. 1989), not
|
||||
read/verified here. The estimator below is therefore **not** "Jin & Nei's
|
||||
method" under any framing — that attribution (present in an earlier
|
||||
revision of this section) was wrong, not just under-cited.
|
||||
|
||||
**Implemented** (`PartitionDispersion`, `obikphylo/src/siblings/algorithms/pairwise.rs`)
|
||||
as an independent method-of-moments estimator, unrelated to any specific
|
||||
published procedure: pools substitution counts by **partition** rather
|
||||
than by genome pair, during the same `reduce_pairwise` pass that builds
|
||||
`PairwiseTally` (no second scan).
|
||||
|
||||
For partition `i`: `n_i` = substitutions pooled over every genome pair,
|
||||
`L_i` = eligible loci pooled over every genome pair, `R_i = n_i / L_i`.
|
||||
Modeling among-site rate heterogeneity the same way as the `+Γ` correction
|
||||
itself (a `Gamma(α, α)`-distributed, mean-1, multiplicative rate shared by
|
||||
every locus in a partition — the classical Poisson-Gamma/negative-binomial
|
||||
mixture, the general identity behind gamma-rate-heterogeneity corrections,
|
||||
also behind Uzzell & Corbin's (1971) original observation that substitution
|
||||
counts across sites/regions are over-dispersed relative to Poisson):
|
||||
|
||||
\[
|
||||
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
|
||||
\]
|
||||
|
||||
Weighting each partition's squared deviation by its own `L_i` cancels the
|
||||
Poisson term before attributing what's left to `α`:
|
||||
|
||||
\[
|
||||
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
|
||||
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
|
||||
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
|
||||
\]
|
||||
|
||||
where `\bar L` is the mean partition size. Returns "no estimate" (falls
|
||||
back to the uncorrected formula, warns) when fewer than 2 partitions have
|
||||
data, `\hat\mu \le 0`, or `V` doesn't exceed the Poisson floor
|
||||
`\hat\mu/\bar L` — no detectable over-dispersion, `α` would be unbounded.
|
||||
|
||||
**Caveat, stated explicitly rather than left implicit**: unlike every
|
||||
closed-form correction in `snp_distance.rs` (each verified line-by-line
|
||||
against `ape`'s `dist_dna.c`, and now also against Jin & Nei 1990 directly
|
||||
for the base `+Γ` formula), this estimator is derived from first
|
||||
principles (the general Poisson-Gamma/negative-binomial identity) with no
|
||||
primary-source procedure behind it at all — not Jin & Nei's (confirmed
|
||||
above), and Wilson et al. (1989), the paper they point to instead, hasn't
|
||||
been read/verified either. Mathematically self-consistent (re-derived and
|
||||
checked, not guessed), but a from-scratch method, not a literature
|
||||
implementation. If `--gamma-shape` needs a value with a literature
|
||||
pedigree rather than an estimated one, Jin & Nei's own stated default,
|
||||
`α = 1` (`--gamma-shape 1`), is the better-supported choice today.
|
||||
|
||||
Deliberately **not** gated by `--sankoff-ratio-ceiling` the way
|
||||
`base_pair_tally` is (same precedent as `cardinality_tally` — see its own
|
||||
doc comment): that filter excludes individual saturated *pairs* from a
|
||||
composition estimate computed once at the very end, from the complete
|
||||
`PairwiseTally`; the partition axis needed here only exists transiently,
|
||||
one partition at a time, while `PairwiseTally` is still being built — long
|
||||
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
|
||||
is known. `--exclude-genome` isn't applied either, matching
|
||||
`reduce_pairwise`'s own raw per-pair fold.
|
||||
|
||||
**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
|
||||
`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
|
||||
"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
|
||||
`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
|
||||
needs the true *directional* per-pair base co-occurrence matrix
|
||||
(`PairwiseTally` only keeps the symmetrised substitution counts
|
||||
`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), `snp-tajima-nei` needs each genome's *own* base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
`snp-f84`/`snp-hky85` had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
|
||||
`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
|
||||
a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
|
||||
`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
|
||||
accumulator — not an architecture change.
|
||||
|
||||
### Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
Sufficient statistic, per genome pair `(i, j)`, from
|
||||
`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
|
||||
`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
|
||||
|
||||
- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
|
||||
(pyrimidine transitions)
|
||||
- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
|
||||
(Kimura's two transversion categories)
|
||||
- \(n_{shared}\): loci where both genomes agree
|
||||
- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
|
||||
loci for the pair)
|
||||
- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
|
||||
\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)
|
||||
|
||||
Derived proportions used below:
|
||||
|
||||
\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]
|
||||
|
||||
Every formula below was checked term-by-term against `ape`'s own
|
||||
`src/dist_dna.c` (not re-derived from memory) before being ported to
|
||||
`obikphylo::siblings::algorithms::snp_distance`.
|
||||
|
||||
**`snp-raw`** — uncorrected p-distance:
|
||||
|
||||
\[
|
||||
d_{raw} = p
|
||||
\]
|
||||
|
||||
**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, *is* the correction):
|
||||
|
||||
\[
|
||||
d_{tv} = Q
|
||||
\]
|
||||
|
||||
**`snp-jc`** (Jukes-Cantor, JC69):
|
||||
|
||||
\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]
|
||||
|
||||
**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
|
||||
\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
|
||||
|
||||
\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]
|
||||
|
||||
**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
|
||||
|
||||
\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]
|
||||
|
||||
**`snp-t92`** (Tamura 3-parameter), with GC content
|
||||
\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
|
||||
\(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
|
||||
|
||||
\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]
|
||||
|
||||
\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]
|
||||
|
||||
\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]
|
||||
|
||||
**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
|
||||
substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
|
||||
recovering the uncorrected formula exactly). E.g. for JC:
|
||||
|
||||
\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]
|
||||
|
||||
Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one `corrected_log` helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
`ape`-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.
|
||||
|
||||
### α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
Follow-up discussion after implementing `PartitionDispersion::estimate_alpha`
|
||||
(previous section), triggered by the user pointing out that the literature
|
||||
conflates two distinct problems: *estimating α from an alignment* versus
|
||||
*using a given α in a Jin-Nei-style corrected distance*. Jin & Nei (1990)
|
||||
itself is squarely in the second camp — α is a user-supplied parameter
|
||||
there, not something their paper estimates (confirmed by reading the paper
|
||||
directly, see previous section). This section catalogs the actual
|
||||
α-estimation literature and records which approaches were considered and
|
||||
why each was or wasn't adopted.
|
||||
|
||||
**Bibliography** (methodological papers, not exhaustive users of `+Γ`):
|
||||
|
||||
| Method | Reference | Needs a tree? | Estimates "true" α or a task-specific one? | Cost |
|
||||
|---|---|---|---|---|
|
||||
| ML, continuous Γ (foundational) | **Gu, Fu & Li (1995)**, MBE 12:546–557 — read in full | yes, topology fixed a priori | true α (+ θ jointly, invariant+Γ) | very high, ≤5–6 taxa in practice |
|
||||
| ML, discrete Γ | Yang (1993/94), MBE 10:1396; JME 39:306 | yes | true α | high |
|
||||
| Parsimony-based ML on inferred changes | Yang & Kumar (1996), MBE 13:650 | yes, equal branch lengths assumed | true α, biased when branches unequal | low |
|
||||
| Corrected substitutions/site → NB fit | **Gu & Zhang (1997)**, MBE 14:1106 — read in full | yes, + ancestral reconstruction | true α, close to ML accuracy | low (given a tree) |
|
||||
| Topology-optimizing grid search | **Guindon & Gascuel (2002)**, MBE 19:534 — read in full | yes, but built *from* the candidate distances themselves (BIONJ) | **not** true α — deliberately biased *upward* for topological accuracy | low–moderate (grid × tree build) |
|
||||
| Free (non-parametric) rate distribution | Susko et al. (2003), Syst. Biol. 52:594 | yes | tests the Γ assumption itself | high |
|
||||
| Parsimony counts/site → NB fit | **Wakeley (1993)**, JME 37:613 — read in full | yes, NJ tree + Fitch parsimony ancestral reconstruction | true α, biased *upward* by parsimony (conservative) | low (given a tree) |
|
||||
| Poisson-Gamma/negative-binomial, foundational | Uzzell & Corbin (1971), Science 172:1089 — **not read**, only seen via citations | unknown — likely also count-per-site/tree-based, unverified | true α (indirect, via NB overdispersion) | unknown |
|
||||
| Bayesian posterior over α | BPP/MrBayes/BEAST2/RevBayes | yes | posterior, not point estimate | very high |
|
||||
|
||||
**Correction (2026-09-11, after reading Wakeley in full):** an earlier
|
||||
revision of this table filed Wakeley (1993) under "no tree needed" —
|
||||
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
|
||||
1987, on the real mtDNA data; coalescent-simulated for the simulation
|
||||
study), reconstruct ancestral states at internal nodes by Fitch (1971)
|
||||
parsimony, count the *minimum* number of changes per site this implies,
|
||||
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
|
||||
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
|
||||
Johnson & Kotz 1969) to the resulting per-site count distribution — the
|
||||
same general "tree + ancestral reconstruction + per-site counts + NB fit"
|
||||
shape as Gu & Zhang (1997), just using raw parsimony counts directly
|
||||
rather than Gu & Zhang's multiple-hit-corrected `k̂`. Confirmed by
|
||||
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
|
||||
mean *and* (more severely) the variance of per-site change counts, which
|
||||
biases `α` **upward** (toward apparent uniformity) — consistent with Gu &
|
||||
Zhang's own citation of Wakeley for exactly this bias, and with the
|
||||
general "parsimony overestimates α" pattern noted throughout this
|
||||
literature. Real hypervariable-region-1 mtDNA data gave `α̂ ≈ 0.44–0.60`
|
||||
per data set, `0.47` combined — despite the conservative bias, still
|
||||
solidly in the "strong heterogeneity" range, which is the paper's own
|
||||
point: the method is biased but usefully conservative, not useless.
|
||||
|
||||
**Where `PartitionDispersion::estimate_alpha` actually sits**: nowhere in
|
||||
this table's tree-dependent rows. It shares only the *abstract
|
||||
mathematical identity* (Poisson-Gamma mixture ⟹ negative binomial
|
||||
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
|
||||
genuinely different, tree-free and ancestral-reconstruction-free route:
|
||||
pairwise genome-to-genome substitution counts pooled by *partition*
|
||||
instead of per-site counts pooled across an alignment's columns after
|
||||
ancestral state reconstruction. As far as this survey has established
|
||||
(Uzzell & Corbin not read, so not fully ruled out), **no tree and no
|
||||
ancestral reconstruction of any kind** appears to be a genuinely
|
||||
distinguishing property of `PartitionDispersion`, not something it
|
||||
inherits from prior art — consistent with it being an independent
|
||||
derivation rather than a literature implementation (see previous
|
||||
section's caveat, now on firmer footing).
|
||||
|
||||
**Why a tree-free estimate is even valid — the identifiability question.**
|
||||
Raised directly by the user after reading Gu, Fu & Li (1995), which states
|
||||
explicitly: *"one cannot estimate the rate heterogeneity when the number
|
||||
of sequences is <3."* This is a real non-identifiability result, not a
|
||||
practical inconvenience: with only 2 sequences, a site's history is
|
||||
summarized by one binary observation (differ / don't differ), collapsed
|
||||
across all sites into two aggregate numbers (transition and transversion
|
||||
proportions) — divergence time and among-site rate variance are
|
||||
confounded in that pair of numbers, with no way to separate them. Every
|
||||
method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
|
||||
sequences *sharing one evolutionary history* (a tree) specifically to get
|
||||
multiple independent looks at the *same* site's rate across different
|
||||
lineages, which is what identifies its variance.
|
||||
|
||||
`PartitionDispersion` doesn't violate this, because it isn't estimating
|
||||
the same quantity. The literature above estimates rate variation **across
|
||||
sites, within one shared tree**. `PartitionDispersion` estimates rate
|
||||
variation **across partitions, pooled over every genome pair in the
|
||||
index** — each genome pair stands in for one independent draw of a
|
||||
partition's relative rate, the same identifying role multiple lineages
|
||||
play in the tree-based methods, just substituting "genome pair" for
|
||||
"lineage" and "partition" for "site." With `P` genome pairs (potentially
|
||||
in the hundreds here) all contributing to every partition's pooled count,
|
||||
the ≥3-samples identifiability requirement is met by the pair count, not
|
||||
by tree depth — closer in spirit to Jin & Nei's own original motivating
|
||||
scenario (comparing several genomic *regions* across one fixed panel of
|
||||
taxa) than to single-tree site-rate estimation.
|
||||
|
||||
This substitution carries its own assumption, which must be named rather
|
||||
than left implicit: that a partition's *relative* rate (fast/slow) is
|
||||
reasonably stable **across genome pairs**, not just across sites within
|
||||
one pair — the direct analogue of the "no lineage-specific rate variation,
|
||||
only site-specific" assumption every tree-based `+Γ` method already makes,
|
||||
just moved from lineage→pair and site→partition. If this breaks — e.g. an
|
||||
index mixing very closely related and very divergent genome pairs, where a
|
||||
partition's saturation behavior differs qualitatively between the two —
|
||||
the estimate could be misled in a way a tree-based method would at least
|
||||
have the topology to detect and a tree-free, pool-everything method
|
||||
cannot. Not yet tested against real data with strongly heterogeneous
|
||||
pairwise divergence; worth keeping in mind as the main open validity
|
||||
question for `--gamma-shape auto`, not the citation question (now settled)
|
||||
this whole discussion started from.
|
||||
|
||||
**Gu, Fu & Li (1995), read in full — the foundational ML method, not a
|
||||
candidate to implement.** Models rate variation as invariant+Γ (`θ` =
|
||||
proportion of invariant sites, `α` = gamma shape among the variable sites),
|
||||
and derives the exact joint likelihood over nucleotide *configurations*
|
||||
across all `n` sequences on a **fixed, a-priori-known tree topology** —
|
||||
for 3 sequences this is closed-form (Jukes-Cantor), for `n` sequences it's
|
||||
a sum over `4^n` configurations weighted by coefficients tied to the
|
||||
specific topology's branching structure (their eq. 32–34), with branch
|
||||
lengths, `α`, and `θ` jointly optimized by Newton-Raphson (Hessian over all
|
||||
free parameters). The site-rate integral has the same closed form
|
||||
`E = θ + (1-θ)(1+D/α)^{-α}` that underlies `+Γ` distance corrections
|
||||
generally (the same mechanism as `corrected_log`, not a coincidence — Jin &
|
||||
Nei's own formula is the `θ=0` special case applied to a pairwise `D`
|
||||
rather than a whole-tree one). Explicitly stated by the authors as
|
||||
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
|
||||
candidate for obikmer's typical genome counts. Notable contribution worth
|
||||
keeping in mind regardless: `ρ = (1+θα)/(1+α)` is shown to be a more robust
|
||||
summary of rate heterogeneity than `α` alone whenever `θ` isn't negligible
|
||||
— `α` and `θ` are confounded (nearly-invariant sites can come from either a
|
||||
small `α`'s left tail or from a nonzero `θ`), so `α` alone can be very
|
||||
unstable while `ρ` stays well-behaved. `PartitionDispersion` has no `θ`
|
||||
term at all (no invariant-site component), so this confound doesn't arise
|
||||
for it the same way, but it's worth remembering if an invariant+Γ variant
|
||||
is ever considered.
|
||||
|
||||
**Gu & Zhang (1997), read in full — rejected for now.** Their procedure:
|
||||
(1) a tree with least-squares branch lengths must already exist; (2)
|
||||
ancestral states are reconstructed at every internal node (they use Zhang &
|
||||
Nei 1997's likelihood method, but note plain parsimony would work, just
|
||||
less accurately); (3) per site, branches are split into "changed"/"unchanged"
|
||||
given the ancestral reconstruction, and the *expected* substitution count
|
||||
`k̂` (corrected for multiple hits) is obtained by solving their eq. (7)
|
||||
(or eq. 12 for the generalized model) numerically — a root-find per site,
|
||||
not a closed form; (4) the site-level `k̂` values (real-valued, not
|
||||
integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
|
||||
get α — again no closed form, numerical optimization. This is a real
|
||||
per-site pipeline requiring a tree *before* distances/α can be computed —
|
||||
the reverse of obikmer's current dependency order (`snp_distance` computes
|
||||
distances, which `--nj`/`--upgma` only turn into a tree *afterward*). Their
|
||||
own answer to this chicken-and-egg problem is an iterative
|
||||
distance→tree→α→distance loop, which they themselves flag as expensive.
|
||||
Adopting this would mean a new ancestral-reconstruction subsystem (even a
|
||||
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
|
||||
ML fit — a much bigger addition than `PartitionDispersion`, not something
|
||||
to build without a concrete need beyond "closer to Gu&Zhang's own
|
||||
simulated accuracy than our moment estimator."
|
||||
|
||||
**Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.**
|
||||
Their **Q criterion**: for a candidate α, build a tree (they use BIONJ) from
|
||||
the α-corrected distance matrix; for every internal branch, group taxa into
|
||||
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
|
||||
distances \(\bar\delta_{AB}\), \(\bar\delta_{CD}\), etc., and let `S ≤ M ≤ L`
|
||||
be the three pairings' sums (`{AB,CD}`, `{AC,BD}`, `{AD,BC}`); the branch's
|
||||
reliability score is `Q_branch = L − M` (zero when the four points are
|
||||
perfectly tree-additive). The whole-tree `Q` is the mean over internal
|
||||
branches (negative/zero branches excluded). `α* = argmin_α Q(α)` over a
|
||||
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
|
||||
sensitivity is highest). Complexity: `O(n²l)` once for the base
|
||||
frequencies, then `O(n²r)` for the `r` candidate distance matrices, then
|
||||
`O(n³r)` worst case for building `r` trees and evaluating `Q` on each
|
||||
(same order as the tree-building step itself, so "free" relative to it).
|
||||
|
||||
Central finding, **the opposite direction from an initial misreading in
|
||||
this discussion**: `α_opt` (Guindon-Gascuel's topology-optimal value) is
|
||||
**always ≥ the true α**, not the reverse — underestimating rate
|
||||
heterogeneity (i.e., picking a larger α than reality) reduces the variance
|
||||
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
|
||||
especially when the molecular clock roughly holds (where `α_opt → ∞`, i.e.
|
||||
no correction at all is topologically best). This is an *empirical
|
||||
regularity* observed across their tested conditions (η ∈ {0.5, 2.0}, true
|
||||
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
|
||||
guarantee would be another unverified-claim mistake of the kind this whole
|
||||
discussion has been correcting.
|
||||
|
||||
**Proposed integration (not yet implemented)**: since `α_true ≤ α_opt`
|
||||
empirically, `PartitionDispersion::estimate_alpha()`'s output could seed
|
||||
the *lower bound* of Guindon-Gascuel's grid (search `[α̂, ∞)` instead of
|
||||
`[0.1, 5000]`), narrowing the grid substantially and reusing the existing
|
||||
estimator rather than needing an independent "true α" source — this also
|
||||
sidesteps Gu & Zhang's tree-first dependency problem entirely, since
|
||||
Guindon-Gascuel's own grid already builds a tree per candidate. Still
|
||||
requires: (a) a **BIONJ** implementation (see below), (b) the Q-criterion
|
||||
subtree-grouping logic, (c) a grid-search driver. Not started.
|
||||
|
||||
**BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.**
|
||||
Searched crates.io directly (API query for "bionj": 0 results) and read the
|
||||
docs.rs pages of the most plausible candidates: `phylotree` (builds/reads/
|
||||
manipulates trees but has no distance-matrix reconstruction — no NJ, no
|
||||
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
|
||||
after checking the actual docs), `speedytree` (NJ only — Canonical/RapidNJ/
|
||||
Hybrid variants, explicitly no BIONJ), `nj` (plain NJ only), `phylo` (no
|
||||
distance-based reconstruction at all). `obikphylo::neighbor_joining`
|
||||
(`tree.rs`) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
|
||||
how branch lengths are weighted during agglomeration (accounts for
|
||||
distance-estimate variance, not just the sum), a modest delta over an
|
||||
existing NJ implementation rather than a rewrite, but would need to be
|
||||
written in-house if the Guindon-Gascuel integration above is ever pursued
|
||||
(their own paper notes other tree-building methods gave similar results in
|
||||
their simulations, so plain NJ may be an acceptable substitute if BIONJ
|
||||
proper is judged not worth the effort).
|
||||
|
||||
### Sharing one sample across algorithms (`--sankoff`/`--tnt`/`--phyg`/`--iqtree` + `snp-*` `--distance`) — implemented 2026-09-11
|
||||
|
||||
**Correctness bug, not just a performance one.** `--sankoff`/`--tnt`/`--phyg`/
|
||||
`--iqtree` (via `SiblingExt::sankoff_bundle`) and a `snp-*` `--distance`
|
||||
(via `SiblingExt::snp_distance`) always consume the *exact same* selection
|
||||
parameters when requested together in one `obikmer phylo` invocation
|
||||
(`n`/`free_loss`/`no_ambiguity`/excluded-set/`entropy_bias` — the CLI has
|
||||
no way to give them different values in one run, see `cmd/phylo/args.rs`).
|
||||
Before this fix, each independently called `sample_index`, and
|
||||
`sample_layer`'s `rand::rng()` (`subsample.rs:235`) is a **thread-local
|
||||
generator that advances across calls, not reseeded each time** — so the
|
||||
second call silently drew a *different* random sample of sites than the
|
||||
first, even with identical parameters. Verified empirically (throwaway
|
||||
8-genome index): running `snp-jc` alone vs. combined with `--sankoff`
|
||||
produced visibly different distance matrices from the same index/params.
|
||||
This defeated the actual point of combining these flags — running several
|
||||
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a `snp-*` distance) on
|
||||
one *identical* site selection for direct comparison.
|
||||
|
||||
**Fix**: `SankoffBundle` (`sankoff.rs`) now retains its `PairwiseTally`/
|
||||
`PartitionDispersion` internally (previously consumed into `raw`/
|
||||
`base_pair_tally`/`cardinality_tally` and dropped) and exposes
|
||||
`SankoffBundle::snp_distance(kind, gamma_shape)`, which computes the
|
||||
matrix from that *same* already-sampled tally — no second `sample_index`
|
||||
call. The shared post-sampling logic (the `--gamma-shape` support check,
|
||||
`alpha` resolution, final matrix build — previously all inline in the
|
||||
`snp_distance()` free function) was factored into
|
||||
`pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)` in
|
||||
`snp_distance.rs`, called by both the standalone `snp_distance()` (after
|
||||
its own fresh sampling) and `SankoffBundle::snp_distance` (reusing the
|
||||
bundle's). `cmd/phylo/mod.rs` keeps the `Option<SankoffBundle>` alive past
|
||||
the Sankoff-family `if` block and, when a `snp-*` `--distance` is also
|
||||
requested, calls `bundle.snp_distance(...)` instead of
|
||||
`cache.snp_distance(...)` whenever a bundle was built — since both branches
|
||||
are driven by the same `args.*` fields, the parameters trivially always
|
||||
match when both fire; no runtime "do the params match" check needed.
|
||||
Verified end-to-end: the reused path logs `(reusing the Sankoff bundle's
|
||||
sample)` and the `snp_distance` stage timer reads `0ms` (`formula(kind)`
|
||||
is `O(n²)` post-processing, no I/O), vs. ~150-230ms for a fresh sample on
|
||||
the same tiny test index.
|
||||
|
||||
#### Explicit `--session` (implemented 2026-09-12)
|
||||
|
||||
The in-process fix above only covered reuse *within one command*.
|
||||
`--session DIR` (`cmd/phylo/args.rs`) names a directory (outside the
|
||||
index) that persists the `snp-*` `--distance` sample — the
|
||||
`PairwiseTally`/`PartitionDispersion` pair `sample_index` would otherwise
|
||||
rebuild from scratch — across *separate* `obikmer phylo` invocations.
|
||||
|
||||
**Crate placement, as planned**: a new **`obiksession`** crate
|
||||
(`src/obiksession/`), domain-agnostic — no `PairwiseTally`/"site" concept
|
||||
anywhere in it. `Session::open(dir, params: &[u8], force: bool)` handles
|
||||
directory lifecycle and the params-conflict check against an opaque byte
|
||||
blob; `Session::store(name, bytes)`/`Session::restore(name) -> Option<Mmap>`
|
||||
handle a checksummed artifact cache. `obikphylo`
|
||||
(`siblings/algorithms/session_cache.rs`, shared by `snp_distance.rs` and
|
||||
`sankoff.rs` — see below) is the consumer: it serializes its own
|
||||
`PairwiseTally`/`PartitionDispersion`/`SnpAlignment` to bytes (via `rkyv`)
|
||||
and hands them to `obiksession`, which never sees their type.
|
||||
|
||||
**Serialization**: `rkyv` 0.8.18, added as planned. One deviation from the
|
||||
original zero-copy pitch, scoped down deliberately: artifacts are restored
|
||||
via `rkyv::from_bytes` (full owned deserialize) rather than `rkyv::access`
|
||||
(zero-copy over the `mmap`). True zero-copy would require every
|
||||
`PairwiseTally` query method (`categories`, `base_freq`, `pair`, ...) to
|
||||
work generically over `Archived<PairwiseTally>` as well as the owned type
|
||||
— a separate, larger change not needed to get the actual win (skipping
|
||||
`sample_index`'s expensive re-scan of the sibling annex; deserializing an
|
||||
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
|
||||
`#[derive(Archive, Serialize, Deserialize)]` was added to `PairStats`,
|
||||
`PairwiseTally`, and `PartitionDispersion` (`pairwise.rs`) for this.
|
||||
|
||||
**Startup behavior, as designed**: `Session::open` creates `DIR` if
|
||||
missing (fresh, params recorded); if `DIR` already holds different saved
|
||||
params, `cmd/phylo` reports a hard error and exits rather than silently
|
||||
preferring one side, unless `--session-force` is given (which discards the
|
||||
directory's cached artifacts and starts over under the new params) —
|
||||
implements exactly the "explicit escape hatch, not silent override"
|
||||
decision from the design discussion. `obiksession` itself doesn't decode
|
||||
either side's bytes to produce a field-by-field diff (true to "opaque
|
||||
blob," left to whoever needs it); `cmd/phylo`'s conflict message is
|
||||
currently a single generic sentence listing which flags could be the
|
||||
cause, not a computed diff — an honest v1 simplification, not a limitation
|
||||
of `obiksession` itself.
|
||||
|
||||
**Per-artifact caching, as designed**: only `PairwiseTally`/
|
||||
`PartitionDispersion` are cached (`snp_distance.rs`'s `restore_tally`/
|
||||
`store_tally`) — `ratio_ceiling`/`gamma_shape` remain pure post-processing
|
||||
over whatever tally is in hand, cached or fresh, exactly as planned.
|
||||
|
||||
**Locking, reconsidered and kept as the whole-session `DirLock`.** First
|
||||
written up as a corner cut ("per-entry locking was sketched, not
|
||||
implemented"). On actually working through the concrete race, whole-session
|
||||
locking turned out to be the *correct* choice, not a simplification:
|
||||
`--session-force` resets a session by deleting its whole `artifacts/`
|
||||
directory. If the directory lock were only held briefly (during
|
||||
`Session::open`, then released so a long-running `Session` doesn't block
|
||||
other processes from touching *different* artifacts), a second process
|
||||
calling `open(..., force: true)` could acquire the lock, see mismatched
|
||||
params, and wipe `artifacts/` **while a first process's already-open
|
||||
`Session` is mid-checkpoint** — genuine corruption risk, not just wasted
|
||||
work. Holding the lock for the `Session`'s entire lifetime makes that
|
||||
sequence impossible: a `force` reset must wait for every currently-open
|
||||
`Session` on that directory to be dropped first. The concurrency this
|
||||
gives up (two *different* processes reading/writing *different* artifacts
|
||||
of the same session at once) isn't exercised by anything today — only one
|
||||
code path (`snp_distance`/`sankoff_bundle`, sequentially per invocation)
|
||||
uses sessions at all.
|
||||
|
||||
**Per-layer chunked/resumable dumps — implemented.** `subsample::sample_index`
|
||||
gained a `skip_layers: usize` parameter (skip the first N (partition,
|
||||
layer) pairs, in its own deterministic iteration order, entirely — no
|
||||
I/O, no `on_layer` call) and its `on_layer` callback gained a `raw_index`
|
||||
argument (that pair's 0-based position in the same order), so a caller can
|
||||
tell exactly how far a run has gotten. `snp_distance`/`sankoff_bundle`
|
||||
checkpoint their tally/dispersion (and, for `sankoff_bundle`, the
|
||||
in-progress `SnpAlignment`) to the session every
|
||||
`session_cache::CHECKPOINT_INTERVAL_LAYERS` (8) layers or
|
||||
`CHECKPOINT_INTERVAL` (30s) of wall time, whichever comes first — bounding
|
||||
how much completed scanning a crash can lose, without paying a full
|
||||
tally-rewrite on every one of what can be thousands of layers. A `progress`
|
||||
artifact records how many layers are reflected in the checkpoint, with a
|
||||
`PROGRESS_COMPLETE` (`u64::MAX`) sentinel once every layer has been
|
||||
processed (distinct from any real layer count, so "finished" is never
|
||||
confused with "stopped after N layers where N happens to equal the total").
|
||||
|
||||
Verified with a real kill: built a 10-genome/64-partition index, ran
|
||||
`--distance snp-jc --session DIR` under `timeout 0.4`, confirmed a
|
||||
checkpoint at `progress=8` (exactly `CHECKPOINT_INTERVAL_LAYERS`) was on
|
||||
disk, then re-ran the same command and confirmed it resumed from layer 8
|
||||
and completed successfully — 0-effort verification that the mechanism
|
||||
itself is real, not just plausible-sounding.
|
||||
|
||||
**Important correction from that same test — resume is *not* bit-reproducible,
|
||||
and shouldn't be.** The output matrix from the resumed run differed from
|
||||
an uninterrupted control run against the same index/params. Root cause:
|
||||
`sample_layer`'s random draws come from `rand::rng()`, a per-process,
|
||||
OS-seeded generator with no continuity across a process boundary — the
|
||||
layers processed before a kill and the layers processed after a resume
|
||||
come from two independent random streams. An earlier revision of this doc
|
||||
(and of the code comments) claimed resume "reproduces the exact same
|
||||
sample an uninterrupted run would have" — that's wrong, and was corrected
|
||||
after the user pointed out the actual reasoning error: true randomness has
|
||||
no notion of continuity in the first place, so there was never a
|
||||
correctness requirement for a resumed run to match an uninterrupted one
|
||||
bit-for-bit. The sample produced after a resume is exactly as legitimate a
|
||||
random draw as one from an uninterrupted run — just a *different* one,
|
||||
same as running the command twice without `--session` already gives two
|
||||
different samples today. That variance is explicitly wanted (see below),
|
||||
not a defect to fix. Comments in `subsample.rs`/`session_cache.rs` were
|
||||
corrected to state this plainly instead of the false reproducibility claim.
|
||||
|
||||
**Explicitly declined: deterministic/seeded sampling.** Raised as a
|
||||
possible fix for the above (seed `sample_layer`'s RNG from a per-session
|
||||
value plus `(partition, layer)`, making a resume reproduce bit-identical
|
||||
results, and, if applied everywhere, making *any* two runs with identical
|
||||
parameters produce identical samples). The user explicitly declined this
|
||||
for now: repeated runs are relied on to measure sampling variance, and a
|
||||
`--seed` flag would need its own explicit design if wanted later — this is
|
||||
not an accidental gap, it's a stated preference to revisit only if asked.
|
||||
|
||||
**`SankoffBundle` integration — implemented.** `sankoff_bundle` accepts
|
||||
the same `session: Option<&Session>` and uses the *same* artifact names
|
||||
(`session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}`)
|
||||
as `snp_distance`, plus its own `ALIGNMENT_ARTIFACT` (`sankoff.rs`) folded
|
||||
in lockstep with the tally at every checkpoint (same `on_layer` call
|
||||
produces both, so they can never drift out of sync). Consequence verified
|
||||
directly: a tally checkpointed by `--sankoff --session DIR` is restored by
|
||||
a *later, separate* `--distance snp-k2p --session DIR` invocation with no
|
||||
`--sankoff` at all, and vice versa — cross-consumer reuse, not just
|
||||
same-command reuse, exactly the synergy this design aimed for.
|
||||
|
||||
**Per-artifact caching, as designed**: only the tally/dispersion (plus,
|
||||
for `sankoff_bundle`, the alignment) are cached — `ratio_ceiling`/
|
||||
`gamma_shape` remain pure post-processing over whatever tally is in hand,
|
||||
cached or fresh, exactly as planned.
|
||||
|
||||
**Still not done — genuine follow-up, not a justified tradeoff:**
|
||||
**exhaustive-sample caching for `--subsample`-varying exploration.** The
|
||||
design discussion's suggestion — cache the exhaustive (`n = None`) sample
|
||||
once and subsample *from* it on each run instead of one session per exact
|
||||
`n` — wasn't implemented; a session today is still scoped to one exact
|
||||
parameter tuple, `--subsample` included.
|
||||
|
||||
### Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
**Implemented.** The primary distance-matrix output
|
||||
(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
|
||||
default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
label truncation, since genome labels here routinely exceed it), a `--csv`
|
||||
flag opts back into the current CSV format, PHYLIP-strict is a possible
|
||||
future addition (not now). This changes the *default* output of every
|
||||
existing `--distance` value (jaccard, hamming, bray-curtis, ...), not just
|
||||
the new `snp-*` ones — accepted explicitly (pre-release, single developer
|
||||
user, no external consumers to break). Scoped to the distance matrix only:
|
||||
`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
|
||||
their existing CSV-only format.
|
||||
|
||||
## References
|
||||
|
||||
The Mash mutation-rate model this discussion contrasts with:
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikpartitionner/src/partition.rs` — routage par hash de minimiseur, choix des paramètres
|
||||
- `obikpartitionner/src/lib.rs` — structure KmerPartition, nombre de partitions
|
||||
- `obikindex/src/partition/partition.rs` — routage par hash de minimiseur, choix des paramètres
|
||||
- `obikindex/src/partition/mod.rs` — structure KmerPartition, nombre de partitions
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -0,0 +1,366 @@
|
||||
&@")--# Pair-coverage-biased family sampling (discussion, 2026-10-07)
|
||||
|
||||
Idea: bias the family sample used by `--pseudo-alignment`/`--sankoff`/`snp-*` so that the `family_overlap` matrix of the *sample* is as uniform as possible, compensating the scarcity of shared families between distant species. Complements, does not replace, the existing proportional-by-partition and entropy-biased draws (see user doc, *Sampling design*).
|
||||
|
||||
## Motivation: observed on Primulaceae (180 genomes)
|
||||
|
||||
Scatter of patristic distance (IQ-TREE on the pseudo-alignment) vs. normalized `family_overlap` shows three pair classes:
|
||||
|
||||
- close intra-genus pairs: high overlap, well resolved;
|
||||
- inter-genus plus distant intra-genus pairs: overlap collapses by ~2 orders of magnitude, one large noisy cloud;
|
||||
- a small low-overlap cluster at the largest distances, plausibly a few poorly sequenced genomes (user hypothesis, unverified; the tree shows one tip, `Androsace_saussurei--PHA010690`, with a 2.9 terminal branch).
|
||||
|
||||
Uniform family draw reproduces this imbalance in the pseudo-alignment: distant pairs share few informative columns, close pairs many. The tree quality for deep splits is then limited by sampled-column count on those pairs, not by the total number of families.
|
||||
|
||||
## What defines a target
|
||||
|
||||
A family $f$ is carried by a genome set $S_f$ and contributes to every pair in $S_f \times S_f$. For a sample $\mathcal{F}$ with inclusion probabilities $\pi_f$, the expected sample overlap of pair $(i,j)$ is
|
||||
|
||||
$$\mathbb{E}[O^{\mathcal{F}}_{ij}] = \sum_{f:\, i,j\in S_f} \pi_f$$
|
||||
|
||||
The goal is a $\pi$ making this approximately constant over pairs. The full-index $O_{ij}$ (already computed by `--family-overlap`) is the pool size per pair; $\min_{ij} O_{ij}$ is a hard ceiling on the flat level reachable.
|
||||
|
||||
## Nesting hypothesis (user, unverified)
|
||||
|
||||
Families covering the poorest pairs also cover the richest ones: absent genome-skimming sampling bias (locus dropout), a family conserved across distant species is not expected to be missing between closer ones. If it holds, $S_f$ is approximately a clade closure and the family pool is nested by pair distance, which changes the design:
|
||||
|
||||
- no sequential/stateful draw needed: a family's weight depends only on its own reach (farthest pair covered, or equivalently its coverage of the poorest pairs), computable in one pass;
|
||||
- flattening reduces to a monotone reweighting by reach: keep (nearly) all long-reach families, thin the short-reach ones, until the sample overlap of close pairs drops to the level of distant ones.
|
||||
|
||||
My earlier remark that the weight is stateful (depends on what other families already covered) holds only without nesting.
|
||||
|
||||
## Measuring nesting: `scripts/pseudo_alignment_nesting.py`
|
||||
|
||||
Works on a pseudo-alignment FASTA plus the IQ-TREE treefile (columns = families; a call is any character other than `-`/`?`). Outputs the sample's `family_overlap` matrix and, per clade-size stratum, the *completeness* $|S_f|/|C(S_f)|$ where $C(S_f)$ is the smallest clade containing $S_f$. Tree rooted as written in the Newick unless `--outgroup`. Uses the already-existing families × genomes presence structure only; no `S_f` is stored (clade of a column = LCA of its first and last carrier in DFS leaf order, precomputed 180×180 matrix).
|
||||
|
||||
### Result, Primulaceae pseudo-alignment (180 genomes, 5.0 M columns, uniform draw, root as written)
|
||||
|
||||
| Clade size | Families | Mean completeness | Clade-closed |
|
||||
|---|---|---|---|
|
||||
| 2 | 145 047 | 1.00 | 100 % |
|
||||
| 3-4 | 272 918 | 0.71 | 20 % |
|
||||
| 5-8 | 503 114 | 0.45 | 0.6 % |
|
||||
| 9-16 | 637 896 | 0.35 | 0.1 % |
|
||||
| 17-32 | 1 656 229 | 0.21 | 0 % |
|
||||
| 33-64 | 299 632 | 0.18 | 0 % |
|
||||
| 65-128 | 181 126 | 0.08 | 0 % |
|
||||
| 129-180 | 1 307 452 | 0.12 | 0 % |
|
||||
|
||||
Overall 4.1 % clade-closed, mean completeness 0.275; off-diagonal overlap min 4, median 1 665, max 387 057, CV 1.57. **Strict nesting does not hold on this sample.** Caveats: unrooted tree, so the 129-180 stratum is inflated by families spanning the arbitrary root (rerun with `--outgroup` pending); this is the sample fed to IQ-TREE, not the full index; absence symbols `-`/`?` both count as absent.
|
||||
|
||||
### Lift test (`--lift-test`): margin-preserving null
|
||||
|
||||
Question: do families covering a rare pair $(A,B)$ also cover an abundant pair $(A,C)$ more often than margins imply? For each $A$, $B$ = its 10 % rarest partners, $C$ = its 10 % most abundant (ranked on pair overlap). Null: curveball swap randomisation of the genome × family matrix (exact row and column sums), re-selecting $B$/$C$ in every replicate with the same procedure. 200 754 sampled columns, 20 null replicates, 5 000 trades between replicates.
|
||||
|
||||
| Statistic | Observed | Null |
|
||||
|---|---|---|
|
||||
| $T=P(C\mid A,B)=O_{ABC}/O_{AB}$ | 0.559 | 0.498 ± 0.003 (no replicate ≥ observed, p at floor 1/21) |
|
||||
| $L=\sum n_A O_{ABC}/\sum O_{AB}O_{AC}$ | 1.18 | 1.694 ± 0.004 |
|
||||
|
||||
The two statistics disagree, not contradictorily: $T$ above null supports the "positive association beyond margins" statement; $L$ below null says that, relative to $P(C\mid A)$, knowing $B$ adds less than under the null — the observed abundant partners of $A$ are far more abundant than null ones (phylogenetic structure inflates the $L$ denominator). Unverified: mixing of the chain (is 5 000 trades enough? the tiny null sd may reflect correlation), sensitivity to the 10 % threshold and to the 200 k-column subsample.
|
||||
|
||||
Combined reading: association exists but is patchy; a reach-only weight is not enough, weighting must be per presence pattern $S_f$.
|
||||
|
||||
## Candidate weighting strategy: pair raking (proposal, not implemented)
|
||||
|
||||
Goal: inclusion probabilities $\pi_f$ with $\sum_{f\ni i,j}\pi_f\approx c$ for all pairs.
|
||||
|
||||
1. First-order weight: $w_f=\sum_{i,j\in S_f}1/O_{ij}$ (draw a pair uniformly, then a family uniformly among those covering it). With the presence matrix $P$ and $R=1/O$, $w=\tfrac12\,\mathrm{rowsum}(P\odot RP)$, one pass in the existing chunked loop, no $S_f$ storage.
|
||||
2. Iterate (raking/IPF with one multiplier per pair): update $R_{ij}$ from the overlap the current $\pi$ yields, recompute $w_f$; corrects the fact that large families are drawn for many pairs at once. $\pi_f=\min(1,c\,w_f)$, $c$ set to hit the target sample size; independent Poisson draw per family.
|
||||
3. Control: recompute the sample's `family_overlap` and its CV with the script above.
|
||||
|
||||
Pre-conditions: exclude anecdotal genomes first (e.g. `Androsace_saussurei--PHA010690`, terminal branch 2.9), else their tiny $O_{ij}$ dominate $R$. Overlap uniformity is only a proxy: information for a deep split is informative sites across it, not mere co-presence; to be checked on node support of the resulting tree. Unknown: whether long-reach families are rare because conserved-single-copy or because repeats/misassemblies (the latter would be amplified by up-weighting).
|
||||
|
||||
## Alternative / complement: site classes with their own models
|
||||
|
||||
Analogy with multigene supermatrices: loci differ in rate *and* in taxon occupancy. Families covering rare pairs (long reach, conserved, slow) vs. families covering close pairs only could be two (or a few) site classes, each with its own substitution model / branch-length scale (IQ-TREE partitions; edge-proportional vs edge-linked options to be checked in the manual). Independent of the weighting: sampling fixes scarcity, partitioning fixes rate heterogeneity; with class-specific scales, oversampling a class would not distort that class's branch lengths, removing the need for $1/\pi_f$ reweighting (IQ-TREE support for per-site weights unverified).
|
||||
|
||||
Caveats: reach is a continuum, two classes is a simplification and there is no criterion for the cut; class membership is defined from presence patterns, hence correlated with the data the model sees (assignment bias, worst with `?` missing data; magnitude not assessed); partitioning does not fix repeats/misassembly, it only gives them their own rate.
|
||||
|
||||
The pseudo-alignment already keeps only variable families, so branch lengths are not substitutions/site anyway (ascertainment bias, handled by `+ASC`, see the constant-column fix section in [evolutionary_distances.md](evolutionary_distances.md)). With partitions the ascertainment correction must be per class: each class excludes a different number of constant sites. Whether the existing fix applies per partition is unchecked.
|
||||
|
||||
Literature: see the next section.
|
||||
|
||||
## Literature on informative-site selection (2026-10-07; read from search extracts, not full papers)
|
||||
|
||||
- **Phylogenetic informativeness.** Townsend 2007 (*Syst. Biol.* 56:222-231) links each site's rate of change to its power to resolve splits at a given depth: invariant sites are useless, saturated sites misleading, with an optimum in between that depends on the age of the split. Implemented online as PhyDesign (requires an ultrametric tree). Relevance: supports an entropy centre $\mu$ that depends on depth (pair distance) rather than a single value; cf. the $\mu(n)$ discussion.
|
||||
- **Entropy-based trimming.** BMGE (Criscuolo & Gribaldo 2010, *BMC Evol. Biol.* 10:210) computes a per-character entropy-like score weighted by a similarity matrix and removes regions above a threshold. Uses entropy to *discard* over-variable characters, the opposite direction of the Gaussian band here.
|
||||
- **Saturation / fast sites.** Classical removal methods: Slow-Fast, LQP, TIGER, Observed Variability. Counter-evidence (Tree-of-Life protein datasets, PMC10609509): even the fastest sites carry reliable signal, trimming them can lower accuracy, slow sites recover fewer true bipartitions than the fastest. Argument against penalizing high entropy ($H\ge2$) without checking that those families do not carry the deep signal; domain (conserved proteins) differs from central-position SNPs, transfer unverified.
|
||||
- **Taxon-coverage theory.** Sanderson, McMahon & Steel 2010 (*BMC Evol. Biol.*, "Phylogenomics with incomplete taxon coverage: the limits to inference") characterise phylogenetically decisive collections of taxon sets via the four-way partition property: when the coverage pattern suffices to determine the tree uniquely irrespective of signal. Formal frame for pair-coverage sampling; applicability at 180 genomes not assessed.
|
||||
|
||||
Takeaways to check before weighting: (i) whether high-entropy families are the deep-signal ones (entropy vs carrier count measured, see below; the cross with reach is still open); (ii) depth-dependent entropy centre; (iii) whether a decisiveness criterion can serve as a sufficiency test for the sample's pair coverage.
|
||||
|
||||
## Simpler weight: poorest pair of the family (user preference, 2026-10-07)
|
||||
|
||||
$w_f=1/\min_{i,j\in S_f}O_{ij}$, $\pi_f=\min(1,c\,w_f)$. Equivalent in spirit to a reach weight but measured on actual overlaps (no tree needed), same $O(|S_f|^2)$ cost per family as the sum form. Limit: it ignores that large families also feed already-abundant pairs; flattening from above then relies only on dropping short-reach families, which is exact only under nesting, and nesting holds partially (see measurements). Untested.
|
||||
|
||||
### Estimating $O$ locally (per partition / per layer)
|
||||
|
||||
- **Per layer: not representative.** A layer is an increment of kmer content added to a partition (typically one `merge`), not a batch of genomes; all genome columns exist in every layer (`UserDocMD/formats/index_layout.md`). Layer content is dominated by kmers novel at merge time, so a layer's $O_{ij}$ reflects merge order, not the index-wide distribution; not an unbiased estimator of the global $O_{ij}$.
|
||||
- **Per partition: valid only if partitions are exchangeable samples of families** (hash-based; not verified, minimizer distribution may be skewed). Even then, the pairs that matter are the rare ones, with near-zero count per partition; a `min` over noisy estimates is biased downward (winner's curse), weights would go to pairs that were unlucky in that partition rather than globally poorest.
|
||||
- **Recommended:** compute $O$ globally once (the `--family-overlap` scan already exists), apply the weight locally per partition, keeping partitions independent. Without an extra pass: pilot estimate on a random subset of partitions.
|
||||
- **Sequential variant:** update the $O$ estimate as partitions are processed in random order, boosting still-undersampled pairs. Self-correcting but breaks partition independence and makes the result order-dependent.
|
||||
- **Quota consequence:** the current per-partition quota is proportional to candidate-family count; with weights it must be proportional to the partition's weight mass $\sum_f w_f$, otherwise partitions rich in high-weight families are undersampled.
|
||||
|
||||
## Does the entropy bias distort pair representation? (measured 2026-10-07)
|
||||
|
||||
Sample drawn with `--entropy 1 --entropy-sd 0.5` (μ=1 explicit, σ default 0.5): family weight $\exp(-2(H-1)^2)$, $H$ = `entropy15` in bits; weight 1 at $H=1$, 0.135 at $H=0$ or $2$, $\approx 3\!\times\!10^{-4}$ at $H=3$.
|
||||
|
||||
Comparison, per genome pair, of the full-index `family_overlap` $O^{\text{full}}$ (index scan) to the sample's overlap $O^{\text{samp}}$ (5.0 M columns), sampling fraction $r_{ij}=O^{\text{samp}}_{ij}/O^{\text{full}}_{ij}$ (uniform draw would give constant $r$):
|
||||
|
||||
| Quantity | Value |
|
||||
|---|---|
|
||||
| Spearman $O^{\text{full}}$ vs $O^{\text{samp}}$ | 0.994 (Pearson on logs 0.998): ranking preserved |
|
||||
| global fraction $\sum O^{\text{samp}}/\sum O^{\text{full}}$ | 0.52 % |
|
||||
| $r$ quantiles 5 / 50 / 95 % | 0.0025 / 0.0037 / 0.0055 |
|
||||
| slope $\log r$ vs $\log O^{\text{full}}$ | +0.12 |
|
||||
| Spearman($r$, patristic distance) | −0.71 |
|
||||
|
||||
Per decile of $O^{\text{full}}$, median $r$ rises from 0.0027 (poorest decile) to 0.0053 (three richest, plateau): the poorest pairs are represented about half as much as the richest, so the draw amplifies the imbalance. $r$ climbs slowly over deciles 1-6 (0.0027 to 0.0037), jumps at deciles 7-8 (0.0045-0.0052), where the $O^{\text{full}}$ distribution has a gap (689 k to 11.6 M, presumably close intra-clade pairs, unchecked).
|
||||
|
||||
Not established: (a) causality — a dependence of $r$ on pair richness is compatible with the entropy filter but no uniform control draw of the same size exists; (b) noise — the poorest sample pairs have tiny counts (min 4).
|
||||
|
||||
**Candidate mechanism: first hypothesis refuted, see the next section.** I first suggested that long-reach families, carried by many divergent genomes, spread over several states ($H\approx2\text{-}3$) and are downweighted from above by the Gaussian. The full entropy table (next section) shows the opposite: families with many carriers are mostly *low*-entropy, and are downweighted from below.
|
||||
|
||||
**Consequence for the pair-weight proposal:** entropy and reach are likely correlated, not independent criteria. Since the entropy weight of each family is known, the pair weight can compensate it exactly (divide by it) rather than compose with it; order of the two filters must then be defined.
|
||||
|
||||
## Entropy vs number of carrier genomes, full index (measured 2026-10-09)
|
||||
|
||||
Source: `phylo --shannon`, which is **always a full, unsampled scan** of every non-monomorphic minorant of every layer (`SiblingExt::shannon_entropy_csv(path)` takes no `subsample`/`entropy_bias` argument; `--subsample`/`--entropy` apply only to `--pseudo-alignment`/`--sankoff`/`snp-*`). Here: 19.7 GB CSV, 535 257 857 families, read in 11 s with DuckDB (no in-memory load). `UserDocMD/usage/phylo.md` line 121 ("`--subsample N --shannon` gives a bounded diagnostic sample") is stale and contradicts line 55 and the code; to fix.
|
||||
|
||||
| Carriers $n$ | Families | mean $H$ | median $H$ | $P(H<0.5)$ | $P(H\ge1.5)$ | mean weight ($\mu=1,\sigma=0.5$) |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 2 | 163.3 M | 0.66 | 1.00 | 34.5 % | 0 % | 0.70 |
|
||||
| 3-4 | 104.6 M | 0.94 | 0.92 | 0.5 % | 7.9 % | 0.93 |
|
||||
| 5-8 | 91.8 M | 0.83 | 0.72 | 0.1 % | 4.6 % | 0.80 |
|
||||
| 9-16 | 62.7 M | 0.70 | 0.50 | 43.6 % | 4.9 % | 0.66 |
|
||||
| 17-32 | 45.4 M | 0.59 | 0.35 | 58.9 % | 5.9 % | 0.53 |
|
||||
| 33-64 | 48.1 M | 0.34 | 0.19 | 79.7 % | 1.1 % | 0.40 |
|
||||
| 65-128 | 19.1 M | 0.37 | 0.18 | 77.4 % | 3.8 % | 0.37 |
|
||||
| 129-180 | 0.19 M | 0.83 | 0.47 | 53.0 % | 21.4 % | 0.46 |
|
||||
|
||||
Findings:
|
||||
|
||||
- Families with many carriers are mostly one dominant state plus rare variants ($H<0.5$ for ~77-80 % at $n\ge33$); only 1-4 % have $H\ge1.5$. The Gaussian at $\mu=1$ therefore penalizes them from the **low** side (weight 0.135 at $H=0$), not the high side.
|
||||
- Relative to a uniform draw, expected inclusion is about ×1.32 at 3-4 carriers and ×0.53-0.56 at 33-128 carriers: a factor ~2.4 between small and large $n$. Same order as the ~2× pair-level undersampling of poor pairs measured above; compatible, but a family-level ratio is not a pair-level one, so causality remains unproven (no uniform control draw).
|
||||
- The filter does what it is designed for (drop near-autapomorphic variants), but those low-entropy high-$n$ families are also the ones covering distant pairs, hence the side effect.
|
||||
- Informative high-$n$ families are scarce: at $n>32$ only ~19 % have $H\in[0.5,1.5)$, and the 129-180 class is 0.04 % of all families. Flattening deep-pair coverage works on little informative material whatever the weights.
|
||||
- Consequence for a $\mu(n)$ that grows with $n$: it would not recover these families (80 % sit below 0.5 bit); what matters at large $n$ is the share of families in the band around 1 bit, not the position of the centre. Conditional weight $w_H(f)/E[w_H\mid n_f]$ remains an option to make the entropy filter neutral to $n$; $E[w_H\mid n]$ is now available from this table.
|
||||
|
||||
Limit: the table has `n_genomes_present` but not the reach of the family, so the link to specific pairs is still not measured.
|
||||
|
||||
## Site concordance factors on the Primulaceae tree (measured 2026-10-12)
|
||||
|
||||
Command (IQ-TREE 3.1.3): `iqtree3 -t <treefile> -s <alignment> -m GTR+G4+ASC --scfl 100 --cf-verbose -T 5 --prefix concord`. With `-t` plus `-m`, IQ-TREE did **not** just annotate the reference tree: it ran a full ML search started from it (102 iterations, ~59 h wall-clock). `concord.treefile` differs from the input treefile by one bipartition (Robinson-Foulds 2), so sCF refer to the re-inferred tree. Without `-m`, ModelFinder is launched and required 157 GB (first attempt failed). The log also warns that all 180 sequences contain more than 50 % gaps/ambiguity. `sN` below = informative sites averaged over the 100 sampled quartets of a branch (alignment: 5 003 414 columns).
|
||||
|
||||
**Deep Primula/Androsace interface: too few informative sites (data scarcity).**
|
||||
|
||||
| Node | Tips (Primula / Androsace) | sCF | sDF1 | sDF2 | `sN` |
|
||||
|---|---|---|---|---|---|
|
||||
| 209 | 67 (43 / 24) | 27 | 47 | 26 | 17 |
|
||||
| 210 | 66 (43 / 23) | 37 | 33 | 30 | 43 |
|
||||
| 208 | 74 (50 / 24) | 54 | 16 | 29 | 190 |
|
||||
| 211 | 24 (2 / 22) | 64 | 18 | 18 | 226 |
|
||||
| 212 | 21 (2 / 19) | 30 | 38 | 32 | 184 |
|
||||
| 216 | 10 (2 / 8) | 21 | 47 | 32 | 7 |
|
||||
| 217 | 9 (1 / 8) | 43 | 20 | 36 | 7 |
|
||||
|
||||
Quartets crossing these nodes have tens to a few hundred informative sites out of 5 M columns, the quantitative counterpart of the pair-overlap scarcity. Primula tips are spread over several nodes mixed with Androsace (e.g. 43 + 24 at node 209); no monophyletic Primula in this tree.
|
||||
|
||||
**Within Androsace: conflict, not scarcity.** Nodes 283 (72 Androsace, `sN` 17 127), 292 (49 tips, 10 335), 305 (22, 11 980), 342 (11, 6 566) have thousands of informative sites but sCF 33-37 % with both alternatives nearly as supported: sites split almost evenly among the three quartet topologies. More sites would not fix this; cause not established (a poor fit of a tree model to intra-species divergence is one hypothesis, untested). Node 185 (23 Androsace): sDF1 53.6 % > sCF 30.6 %, a majority of the 893 sites prefer an alternative (real conflict).
|
||||
|
||||
**Well-supported nodes have long branches:** node 296 (45 tips, length 0.23, sCF 90 %), 341 (0.21, 91 %), 330 (0.09, 85.5 %), 181 (0.099, 89.5 %); most branches of length 0.01-0.05 have sCF 33-45 %. Degenerate: node 186 (length 0.001, `sN` 0.37) and 187 (`sN` 2.2) carry no informative site.
|
||||
|
||||
Median sCF shows no gradient with clade size (about 39-48 % in every class), so the clade-size class hides the effect; the scarcity is specific to the interface nodes above.
|
||||
|
||||
**Implication for sampling.** The part a sampling design can address is the interface scarcity: select families carried by genomes from the four groups around each target node (a quartet-targeted selection) rather than flattening the whole pair matrix. Not assessed: how many such families the index holds (the `sN` of nodes 209 and 216 suggests few). For the intra-Androsace conflict no sampling strategy is suggested by these measurements alone.
|
||||
|
||||
## Quartet coverage (measured 2026-10-12, `--quartet-test`)
|
||||
|
||||
`scripts/pseudo_alignment_nesting.py --quartet-test --quartets N` draws random genome quartets and counts exactly the columns where all four carry a call, stratified by the size of the smallest tree clade containing the four. Cost: 20 000 quartets over 5.0 M columns in about 35 s total. Exact per-family quartet weighting is out of reach ($\binom{180}{4}\approx 42.3$ M quartets; one family carried by 100 genomes covers $\binom{100}{4}\approx 3.9$ M), so quartets serve as a *diagnostic* of a sample, not as a weight.
|
||||
|
||||
Result on the current sample: 4-way count quantiles 0 / 1 / 10 / 50 / 90 / 99 / 100 % = 0, 3, 68, 200, 2 329, 66 504, 194 460 (CV 3.5, 0.1 % uncovered). By depth: median 131-209 for quartets inside clades ≥ 65 genomes versus 11 580-57 387 for clades 17-64. The ratio of the 4-way count to the smallest of the six pair overlaps has median 0.20-0.29 for deep quartets (0.53 for clades 33-64; 1 would be perfect nesting): pair overlap is only a loose upper bound of 4-way coverage, so flattening pairs does not by itself flatten quartets. 88 % of random quartets fall in the deepest class (root as written; strata with < 5 quartets are noise). Co-presence is not informativeness (a column must also differ among the four). Next measurement: repeat on a sample drawn with the pair weight and check whether the ratio moves toward 1.
|
||||
|
||||
## Two kinds of family loss: sequencing dropout vs divergence (2026-10-12)
|
||||
|
||||
- **Sequencing dropout (genome skimming, ~1× coverage).** About $e^{-1}\approx 37\%$ of each genome is uncovered, independently per family. Predicted for a family truly present in a clade of $m$ genomes: completeness ≈ 0.63-0.70 and clade-closed fraction $c^m$, independent of clade depth. Consistent with small clades (3-4 tips: completeness 0.71, 20 % closed vs $0.65^3=27\%$, $0.65^4=18\%$). Does not predict the observed decay at larger clades (0.45 at 5-8 tips, 0.21 at 17-32, 0.08-0.12 beyond).
|
||||
- **Loss by divergence (user observation).** A family is a kmer whose 30 flanking positions (15 each side for $k=31$) are identical, with a variable central base; with per-site divergence $d$ the flanks survive with probability ≈ $(1-d)^{30}\approx e^{-30d}$ (0.74, 0.21, 0.042, 0.0012 for $d$ = 0.01, 0.05, 0.10, 0.20). Phylogeny-dependent and tied to local conservation, unlike sequencing dropout. Explains the loss of completeness with clade size, and sets a floor: no weighting can sample families that no longer exist for pairs whose $e^{-30d}$ is negligible. Surviving deep families are selected for conserved flanks, hence plausibly slow-evolving (fewer variable sites for deep splits; hypothesis, untested).
|
||||
- The patristic distances of the IQ-TREE tree (1 to 3.5) come from an alignment with variable sites only, hence strongly inflated; the true $d$ is unknown and these numbers cannot be plugged into $e^{-30d}$.
|
||||
- **Test (user's plan), done 2026-10-12: see the next section.**
|
||||
|
||||
## Overlap vs Mash distance on the full index (measured 2026-10-12)
|
||||
|
||||
Inputs: `phylo --distance mash --csv --shared-kmers --nj` on the 180-genome index (distance matrix, shared-kmer matrix, NJ tree) against the full-index `family_overlap` $O_{ij}$. Mash $D$: range 0.011-0.337, median 0.20.
|
||||
|
||||
| Regression | Slope | $R^2$ |
|
||||
|---|---|---|
|
||||
| $\log O_{ij}$ vs $D$, all pairs | -26.95 | 0.979 |
|
||||
| $\log$(shared kmers) vs $D$ | -31.87 | 0.984 |
|
||||
| $\log O_{ij}$ vs $D$ + additive genome effect | -25.3 | 0.9999 |
|
||||
|
||||
Spearman($D$, $O$) = -0.95; Spearman($D$, shared kmers) = -0.95; Spearman($D$, IQ-TREE patristic) = 0.81.
|
||||
|
||||
Reading:
|
||||
|
||||
- Overlap decays log-linearly with distance, slope about -27 versus -30 predicted by flank survival (previous section). **Not an independent confirmation of the flank model**: Mash $D$ is derived from kmer sharing and $O$ counts families defined by kmers, so both measure the same shared content. The kmer slope of -32 is $-k$ by construction of Mash (implementation check only). The ratio of the two slopes, 27/32 ≈ 0.85, is below the 30/31 ≈ 0.97 expected from flank survival alone; cause unknown, the gap is modest.
|
||||
- Local slopes vary: -21 for $D\in[0.011,0.07]$ ($R^2$ 0.45), -28 for [0.07, 0.20] (0.96), -35 for $D>0.23$ (0.83). Half the pairs lie in $D\in[0.20,0.23]$, where the fit explains almost nothing ($R^2$ 0.26 and 0.07): a plateau where Mash has little resolution.
|
||||
- Genome effects (log scale, sd 0.21): lowest `Androsace_saussurei--PHA010690` (factor ≈ 0.39), `Androsace_triflora--PHA000558` (≈ 0.62), `Androsace_vesulensis--PHA010698` and `Primula_stricta_var_obesior--TROM_V_149128` (≈ 0.65); highest `Androsace_pubescens--PHA010697` (≈ 2.8). Three of the four lowest are also among the genomes with the fewest calls in the pseudo-alignment (saussurei, triflora, stricta), pubescens has the most; consistent with a coverage effect at the extremes, correlation over all genomes not computed.
|
||||
- NJ on Mash vs the IQ-TREE tree: Robinson-Foulds 182 of 354 (0.51 normalized); the Mash plateau near 0.2 probably limits NJ resolution for most pairs.
|
||||
- Consequence for normalization: the residual $\log O_{ij}-(a+bD_{ij})$ measures coverage deviation at a given distance, a normalization of $O$ that needs no per-genome call counts. Untested on a weighted sample.
|
||||
|
||||
## Data-poor genomes (measured 2026-10-12)
|
||||
|
||||
Per-genome share of the 5 003 414 sampled columns with a call: median 5.2 %, max 14.3 %; 8 genomes below 1 % and 23 below 2 %. Fewest calls: `Androsace_triflora--PHA000558` 1 936, `Primula_stricta_var_obesior--TROM_V_149128` 2 403, `Primula_carniolica--PHA007185` 4 937, `Androsace_saussurei--PHA010690` 6 986, `Primula_pumila--TROM_V_135924` 17 393, `Androsace_maxima--PHA000534` 17 939, `Primula_auricula_subsp_serratifolia--PHA007184` 30 621, `Androsace_pyrenaica--PHA000543` 37 725. Three of them (saussurei, serratifolia, pyrenaica PHA000543) are among the tips whose placement looked aberrant, previously attributed to possible mislabelling or contamination. A simpler alternative is too little data to place them reliably; testable by removing them (`--exclude-genome`) and watching deep-node support.
|
||||
|
||||
Consequence for flattening: raw $O_{ij}$ mixes evolutionary distance and per-genome coverage; flattening on it chases pairs involving data-poor genomes, for which no family exists. Normalizing, $O_{ij}/(c_ic_j)$ with $c_i$ the number of families carried by genome $i$ (per-genome counts from `--sibling-stats`), targets distance instead of coverage; the best normalization (smoothing of $c_i$, etc.) is not established.
|
||||
|
||||
## Implemented: entropy annex parametrised by `--no-ambiguity` (2026-10-13)
|
||||
|
||||
Design decision (user): the way genomes are counted depends on `--no-ambiguity`, so that flag, not an "alphabet" option, is the annex parameter. Over single-member genomes the 15-state entropy only has the four single-base states, hence `entropy4` (redefined: 4 bases over single-member genomes) equals `entropy15` under `--no-ambiguity`; a unit test checks it.
|
||||
|
||||
- `entropy_annex.rs`: header `flags` (the former padding), bit 0 = `no_ambiguity`; old files have 0 and stay valid. One file per convention (`entropy.pent`, `entropy.noamb.pent`), so switching the flag builds the missing one without discarding the other. `EntropyAnnex::no_ambiguity()`; `iter_full_entropy` rejects a header that disagrees with the requested convention.
|
||||
- `ensure_layer_entropy_annex` takes the list of conventions to build; one scan feeds every missing annex and the overlap accumulator. `sample_index` builds and reads the convention matching its `no_ambiguity` argument, removing the mismatch between the draw (previously always raw masks) and the alignment (masked).
|
||||
- `--sibling-annex` builds both conventions and invalidates both on rebuild. `--shannon` no longer builds any annex as a prerequisite (it recomputes everything from its own scan).
|
||||
- Checked on the 1x benchmark index: a `--no-ambiguity` run creates `entropy.noamb.pent` beside the existing `entropy.pent` in all 512 layers; workspace tests pass.
|
||||
- Still to do, in this order: `--quartet` (per-family bit annex, same parameter), `--pair-balance` (mean of 1/shared-families over the carrier pairs; second overlap matrix for the single-member convention, annex header carrying a fingerprint of the matrix it came from).
|
||||
|
||||
## Implemented: `--quartet` (2026-10-13)
|
||||
|
||||
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Absence never counts as a state; with `--no-ambiguity`, multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; the `--no-ambiguity` convention is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`) because that header has no room for a flag. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
|
||||
|
||||
Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list.
|
||||
|
||||
First test on the 1x benchmark testbed (3 seeds, exhaustive alignment, `GTR+G4+ASC`, species monophyly out of 3 per seed): without filter 1 / 0 / 0 correct monophylies; carriers >= 4 only: 3 / 1 / 2; `--quartet`: 2 / 2 / 3 (370, 148, 369 columns, all informative by construction). *Klebsiella* stays fragile (wrong in 2 seeds out of 3); two seeds still show a branch at the IQ-TREE ceiling of 10 with so few columns. Small testbed, three seeds: a direction, not a measurement.
|
||||
|
||||
## Implemented: `--subsample` renamed `--n-sites N|all` (2026-10-13)
|
||||
|
||||
`--subsample` read as "draw a subset", while `--pseudo-alignment`/`--sankoff` made it mandatory and accepted no way to say "everything" (the workaround was a number larger than the family count, e.g. `--subsample 100000000`). The option is now `--n-sites <N|all>`: `all` is an explicit value, parsed into its own type (`NSitesArg::{All, N}`, like `GammaShapeArg`) rather than an in-band sentinel (negative number, `0`); `all` gives the exhaustive scan the `snp-*` path already ran when the option was omitted. Still mandatory for `--pseudo-alignment`/`--sankoff`; still optional for `snp-*` `--distance`. No alias is kept for the old name (the software is in active development). The benchmark commands and measurements recorded in the earlier sections of this file were run with the old name.
|
||||
|
||||
## Implemented: `--pair-balance` (2026-10-13)
|
||||
|
||||
Weight of a family = **mean** over the pairs of its carriers of 1/O (O = index-wide shared variable families of the pair, same `--no-ambiguity` convention): the simulation on the *Primulaceae* sample (above) had the mean flatten best (expected-overlap range 23 000× to ~600×, slope 0.37) without chasing high-carrier families, unlike the sum (65 % of the draw at >= 33 carriers).
|
||||
|
||||
- `algorithms/pair_weight.rs`: `inverse_overlap_matrix` (dense n×n of 1/O from the persisted overlap annex), `pair_balance_weight` (mean over carrier pairs; 0 below two carriers). Carriers are defined once, `helpers::is_carrier(mask, no_ambiguity)`, shared with the overlap accumulator and the quartet test.
|
||||
- Storage: no new format. The entropy annex is generalised to `FamilyScalarAnnex` with a `ScalarKind` (`Entropy` magic `PENT`, `PairWeight` magic `PPWT`, one file per kind and convention, header flag `no_ambiguity`); `entropy_annex.rs` became `family_scalar_annex.rs`. Family overlap is now one file per convention too (`family_overlap.povl`, `family_overlap.noamb.povl`, header flag); `--family-overlap` and `--min-shared-family` follow `--no-ambiguity`.
|
||||
- Build order: sweep 1 builds entropy, quartet and the missing overlap annexes together; sweep 2 builds the pair weights, which need the finished matrix (`ensure_layer_pair_weight_annexes`). `AnnexRequest` now lists `entropy`, `quartet`, `overlap`, `pair_weight`; a pair weight implies its overlap. A rebuilt sibling annex removes every annex derived from it (entropy, quartet, pair weight, overlap), so a weight can never outlive the matrix it came from; there is no fingerprint, only that single invalidation point.
|
||||
- Sampler: `LayerPlan` (eligible bits, minorant count, mass) replaces the positional tuple; a layer's quota is `n × mass / Σ mass` (mass = eligible count without the option, Σ weights with it); a family is accepted with probability `quota/mass × weight × entropy kernel`, capped at 1 (the cap, and kernels below 1, undershoot the quota, absorbed by the existing top-up rounds).
|
||||
- Cost note: weights need O(carriers²) per family at build time; not measured on a large index.
|
||||
|
||||
First test, 1x benchmark testbed (20 genomes, 46 706 variable families, `--n-sites 5000 --free-loss --no-ambiguity`, 8 replicates each; sample overlap computed from the alignment, compared with the index `--family-overlap --no-ambiguity`):
|
||||
|
||||
| | slope of log(sample overlap) on log(index overlap) | pairs (index overlap > 0) covered by >= 1 column | columns with >= 10 carriers |
|
||||
|---|---|---|---|
|
||||
| uniform | 1.02 ± 0.14 | 64 % | 0.18 % |
|
||||
| `--pair-balance` | 0.84 ± 0.001 | 97 % | 1.1 % |
|
||||
|
||||
The weight flattens the representation (slope toward 0) and, above all, covers most of the pairs a uniform draw misses. Not measured: effect on the tree (sCF / monophyly), interaction with `--entropy`, behaviour on the *Primulaceae* index. 20 genomes only.
|
||||
|
||||
## Quota rounding fix, and the combined test on the 1x testbed (2026-10-13)
|
||||
|
||||
**Fix.** `sample_index` rounded each layer's quota down on its own, losing up to one site per layer: with 512 layers a budget of 100 sites gave 0 to 3 columns. Quotas now use cumulative rounding (`floor(n × cumulative mass / total)` minus what earlier layers got), so they sum to exactly `n`; a 100-site budget now yields 50-80 usable columns. Not a result of the options below, a defect the small testbed exposed.
|
||||
|
||||
**Test.** 1x testbed, 3 seeds × 10 replicates per variant, 11 genomes (the 9 genomes with fewest calls excluded, so no all-missing sequence), `--free-loss --no-ambiguity`, IQ-TREE `GTR+G4+ASC` on each sampled alignment; score = number of the three species (*E. coli*, *Salmonella*, *Klebsiella*) recovered monophyletic (0-3), mean over the 30 runs; "informative" = columns with two bases each carried by >= 2 genomes.
|
||||
|
||||
| `--n-sites` | variant | trees obtained | columns | informative columns | mean species monophyletic (of 3) | all 3 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 100 | uniform | 77 % | 49 | 0.5 | 0.04 | 0 % |
|
||||
| 100 | `--quartet` | 100 % | 82 | 82 | 2.30 | 50 % |
|
||||
| 100 | `--pair-balance` | 100 % | 52 | 2.1 | 0.20 | 0 % |
|
||||
| 100 | `--quartet --pair-balance` | 100 % | 67 | 67 | 2.53 | 53 % |
|
||||
| 100 | `... --entropy 1` | 100 % | 67 | 67 | 2.17 | 17 % |
|
||||
| 300 | uniform | 93 % | 163 | 2.1 | 0.04 | 0 % |
|
||||
| 300 | `--quartet` | 100 % | 239 | 239 | 2.33 | 33 % |
|
||||
| 300 | `--pair-balance` | 100 % | 179 | 6.2 | 0.57 | 0 % |
|
||||
| 300 | `--quartet --pair-balance` | 100 % | 197 | 197 | 2.67 | 67 % |
|
||||
| 300 | `... --entropy 1` | 100 % | 196 | 196 | 2.67 | 67 % |
|
||||
|
||||
(Mean over all runs counting a failed tree as 0 is identical to the "mean species" column wherever trees were obtained; the uniform rows lose runs because IQ-TREE refuses alignments where a genome has no data.)
|
||||
|
||||
Reading: `--quartet` is the decisive option here — a uniform draw of 100-300 sites contains almost no informative column (0.5-2), the restricted draw contains only informative ones. `--pair-balance` alone barely helps (it favours rare pairs but does not remove the uninformative columns); added to `--quartet` it raises the share of replicates recovering all three species (50→53 % and 33→67 %). The entropy kernel (centre 1 bit, default width) adds nothing at 300 sites and costs at 100. Limits: 11 genomes, three simulated 1x datasets, small budgets, a single criterion (species monophyly, not deep branches), and the differences between the `--quartet` variants (a few runs out of 30) are small relative to the replicate noise — a direction, not a ranking.
|
||||
|
||||
## Implemented: no distance matrix unless one is wanted (2026-10-13)
|
||||
|
||||
`--distance` had a default (`jaccard`) and `phylo` always computed and wrote a matrix, even for `--sibling-annex`, `--family-overlap` or `--pseudo-alignment` alone. `--distance` is now optional: a matrix is computed only if `--distance` is given, or `--nj`/`--upgma`/`--shared-kmers` needs one (then `jaccard`, `args::distance_to_compute`, unit-tested); `--csv` alone changes nothing. When nothing is wanted the command stops before the distance step. Checked on the 1x testbed: `--family-overlap` and `--pseudo-alignment` runs write no `_dist` file and run no distance stage; `--nj` writes `_dist.phy` and `_nj.nwk`; `--distance mash --csv` writes `_dist.csv`. Cost saved on a large index: not measured.
|
||||
|
||||
## Implemented: diagnostic outputs (2026-10-13)
|
||||
|
||||
Added so the measurements this file reports no longer need ad hoc scripts, and so a draw can be diagnosed from the files of the run that produced it:
|
||||
|
||||
- `--sample-stats`: a `SampleStatsReducer` (`algorithms/sample_stats.rs`), a `LayerReducer` run beside the alignment over the same surviving families; writes `_sample_genomes.csv`, `_sample_carriers.csv`, `_sample_overlap.csv`. A call is a masked state other than `0`; informative = `has_bipartition` (the quartet rule, now shared). Checked on the 1x testbed: per-genome calls and the overlap matrix equal those recomputed from the alignment.
|
||||
- `--sample-plan`: `sampled_scan` now returns a `SamplePlan` (one `LayerReport` per layer: eligible, mass, quota, resolved, kept); `_sample_plan.csv` has the layer lines only (quotas sum to the target by cumulative rounding; on the testbed 3000 asked, 2695 kept).
|
||||
- `--shannon-summary`: `--shannon` and the summary share one traversal (`scan_family_entropies`, which replaces the loop that lived in `shannon_entropy_csv`); the summary keeps counts per (entropy kind, genomes present, genomes without ambiguity, 0.05-bit bin) instead of one line per family: 156 lines for the 46 706 families of the testbed (a few hundred thousand lines at most on any index, by construction: the bins are bounded). Checked: its `entropy15` counts sum to the number of variable families.
|
||||
- Writers are in `cmd/phylo/stats.rs`; the library computes, the command layer formats and writes.
|
||||
|
||||
## Combined draw on Primulaceae: `--quartet --pair-balance --entropy 1` (measured 2026-10-13)
|
||||
|
||||
Command: `phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance -o primulaceae_quartet_balance --family-overlap --sibling-stats primulaceae`. One draw, 180 genomes, 5 000 000 columns requested and obtained (no shortfall; no `--sample-plan`, so per-layer quotas are not inspectable). The three options are on together: **effects of entropy, quartet and pair balance are not separable here**; the testbed result that the entropy kernel adds nothing at small budgets was not rechecked.
|
||||
|
||||
**Baseline.** The same statistics were recomputed with one script on the alignment fed to IQ-TREE (`primulaceae-gamma_alignment.fasta`, 5 003 414 columns). It reproduces the figures of the earlier sections (sample overlap min 4 / median 1 665 / max 387 057 / CV 1.57; sampling fraction $r$ quantiles 0.0025 / 0.0037 / 0.0055; 5.2 % median calls per genome), hence **is the entropy-drawn sample** (`--entropy 1 --entropy-sd 0.5`, section on entropy bias), not a uniform draw. The "uniform draw" label of the nesting section is inconsistent with this (to settle). So the comparison below is *entropy-only vs entropy + quartet + pair balance*, never against a uniform control. A call is any character other than `?`/`-`. With `--free-loss` absence is written `?` (`masking::masked_state`); `-` only occurs without it. Neither file contains IUPAC characters.
|
||||
|
||||
| | Baseline (entropy only) | Combined |
|
||||
|---|---|---|
|
||||
| Informative columns (≥ 2 bases, each in ≥ 2 genomes) | 21.4 % | 100 % |
|
||||
| Carriers per column: median / share ≤ 4 / share ≥ 33 | 4 / 58.1 % / 8.0 % | 8 / 12.4 % / 11.3 % |
|
||||
| Calls per genome, median; genomes < 1 % / < 2 % | 5.2 %; 8 / 23 | 8.1 %; 4 / 6 |
|
||||
| Sample pair overlap min / median / max | 4 / 1 665 / 387 057 | 282 / 27 380 / 491 038 |
|
||||
| Sample overlap CV; max/min | 1.57; 96 764 | 1.10; 1 741 |
|
||||
| 4-way count (20 000 random quartets, 2 seeds): median | 200-202 | 3 738-3 797 |
|
||||
| 4-way p1 / p10 | 3 / 66-67 | 93-95 / 1 092-1 104 |
|
||||
| 4-way p90 / p99 | 3 000-3 644 / 65 700-67 300 | 14 500-15 600 / 82 900-84 600 |
|
||||
| 4-way CV; share of quartets with 0 | 3.4-3.5; 0.11-0.14 % | 1.83-1.87; 0 % |
|
||||
|
||||
(p90/p99 move by 20 % between the two seeds: read them as orders of magnitude only.) The gain is at the bottom of the 4-way distribution (median ×19, p1 ×30); the top is unchanged. 4-way coverage is not stratified by clade depth here (needs the tree). "Informative" is measured over the 180 genomes: a column informative overall is not necessarily informative among the four genomes of a given quartet.
|
||||
|
||||
**Pair representation.** Index-wide overlap used for the comparison: baseline, the raw matrix (`primulaceae_phylo_family_overlap.csv`); combined, the `--no-ambiguity` matrix of the same run. Different conventions, so the two slopes are indicative.
|
||||
|
||||
| | Baseline | Combined |
|
||||
|---|---|---|
|
||||
| Slope of log(sample overlap) on log(index overlap) | 1.12 ($R^2$ 0.997) | 0.54 ($R^2$ 0.91) |
|
||||
| Pairs with index overlap > 0 and no sample column | 0 | 0 |
|
||||
| Median $r$ = sample/index, poorest decile of index overlap | 0.0027 | 0.059 |
|
||||
| Median $r$, richest decile | 0.0053 | 0.0067 |
|
||||
| Median $r$ by decile 1 → 10 | 0.0027 … 0.0053, rising | 0.059, 0.065, 0.067, 0.067, 0.069, 0.074, 0.023, 0.0082, 0.0077, 0.0067 |
|
||||
|
||||
The weight does not merely flatten: the poorest pairs are now about 9× better represented (relative to their pool) than the richest, where the baseline gave them about half. Deciles 1-6 are flat; the drop sits at deciles 7-10, around the gap of the index-overlap distribution already noted in the entropy-bias section (cause of that gap still unchecked). **Caveat on $r$:** the numerator only holds quartet-informative families while the denominator counts every variable family, and the matrices differ in convention, so $r$ is no longer a sampling fraction of one pool; the decile pattern is a measurement of the sample, not of an inclusion probability.
|
||||
|
||||
**Per-genome richness.** `--sibling-stats` now gives the variable families (size ≥ 2) per genome: 0.72 M to 93.9 M, median 34.5 M. Spearman between a genome's calls in the sample and its variable-family count: **0.974 in the baseline, 0.435 in the combined draw**: the baseline mirrored genome richness, the combined draw largely decouples from it (on the same genomes the correlation with *all* families is 0.66 and 0.32). Genomes poorest in variable families: *A. saussurei* 0.72 M (0.30 % of its families), *A. triflora* 1.26 M (0.72 %), *P. stricta var. obesior* 2.39 M (1.33 %), then *P. pumila* 5.2 M, *P. auricula* subsp. *serratifolia* 5.9 M (1.92 %), *P. carniolica* 6.8 M (1.77 %), *A. pyrenaica* PHA000543 8.4 M (1.46 %). Their calls in the sample went from 6 986 / 1 936 / 2 403 (saussurei, triflora, stricta) to 15 725 / 22 988 / 25 392, and carniolica from 4 937 to 52 310; *saussurei* stays the poorest genome in both. These genomes are data-poor, not empty: their families are mostly of size 1. This is the per-genome $c_i$ the normalisation of the data-poor section calls for.
|
||||
|
||||
Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 / 2 / 3 / 4 = 33 512 358 231 (98.43 %) / 479 910 992 (1.41 %) / 45 621 663 (0.13 %) / 9 725 201 (0.03 %). Sizes ≥ 2 sum to 535 257 856, matching the 535 257 857 of the entropy table within 1.
|
||||
|
||||
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
|
||||
|
||||
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment had not been given to IQ-TREE (done 2026-10-16: see the last measured section).
|
||||
|
||||
## Tree-level effect of the combined draw: sCF on three trees (measured 2026-10-16)
|
||||
|
||||
Closes the "not measured" item of the previous section (the combined alignment given to IQ-TREE). Three trees, all sCF by `--scfl 100 --cf-verbose`, model `GTR+G4+ASC`:
|
||||
|
||||
- **A** = `concord` (12/10): entropy-only draw, 180 genomes, 5 003 414 columns; `-t` + `-m` re-ran a full ML search (59 h).
|
||||
- **B** = `concord_quartet_balance`: combined draw (`--entropy 1 --quartet --pair-balance`), 180 genomes, 5 000 000 columns. Its log stops during the tree search (14/10), so the command that produced its `cf.*` (16/10) is not in the files; presumably `-te` on the 15/10 treefile as for G (unverified).
|
||||
- **G** = `scf_good`: same draw as B without the 3 data-poor genomes (*A. pyrenaica* PHA000543, *A. saussurei*, *A. triflora*), 177 genomes. 72 sites became invariant and were removed to satisfy ASC; the search (`concord_quartet_balance_good`, 107 iterations, 6 h 47 on 120 threads) crashed in the sCF step on the FASTA (`Assertion all_prob_const < 1.0`; cause untested, ASC with constant sites is a suspect). sCF were recomputed with `-te <treefile> -s <varsites.phy> -m GTR+G4+ASC --scfl 100`: **no search** (0 iterations, lnL −41 053 676.53 vs −41 053 676.41 from the search), 4 999 729 columns. A used `-t`, B/G a fixed tree: the protocols differ.
|
||||
|
||||
| | A | B | G |
|
||||
|---|---|---|---|
|
||||
| Branches | 177 | 177 | 174 |
|
||||
| Mean `sN` | 3 219 | 13 163 | 13 396 |
|
||||
| Branches with `sN` < 100 / < 1 000 | 13 / 50 | 0 / 5 | 0 / 2 |
|
||||
| Median sCF | 44.2 | 42.9 | 42.8 |
|
||||
| Branches with sCF < 40 / ≥ 70 | 74 / 28 | 67 / 26 | 69 / 25 |
|
||||
| *Primula* and *Androsace* monophyletic | no / no | no / no | yes / yes |
|
||||
|
||||
**Sampling effect (A → B, same 180 genomes).** `sN` ×4.5 and the scarcity is gone. Sites do not become more concordant: on the 113 shared bipartitions (of 177) median sCF 48.0 → 47.3 (44 up, 69 down). Robinson-Foulds 128: the draw rearranged the tree deeply. Consistent with the earlier reading that the intra-*Androsace* conflict is not a matter of site count; the number of branches with sCF < 40 % stays around 70.
|
||||
|
||||
**Interface nodes.** Of the old nodes 209/211/212/216/217 (`sN` 17 / 226 / 184 / 7 / 7) none exists in G. Node 208 survives with `sN` 190 → 5 218 (sCF 54.5 → 37.5). Six mixed clades remain in G (83 to 72 tips, 50 *Primula* + 22-33 *Androsace*), all with `sN` > 3 000.
|
||||
|
||||
**Effect of the 3 genomes (B → G, same draw).** RF 16 (166 of 174 shared), median sCF 43.2 → 43.6, median `sN` unchanged. In B, *A. pyrenaica* and *A. triflora* sit inside *Primula* (a clade of 52 tips = 50 *Primula* + these 2; each has a *Primula* sister, sCF 92 % and 52 %, `sN` 556 and 153); *A. saussurei* stays with *Androsace*. Removing them restores both monophylies and changes nothing else measurably: support for "too little data to place them" (data-poor section), not for contamination, though the latter is not excluded.
|
||||
|
||||
**Where A and G disagree (2026-10-16).** On the 177 common tips: 124 shared bipartitions, 50 only in A, 50 only in G. Differing bipartitions are weakly supported in both trees (median sCF 35.7 in A-only, 37.7 in G-only, versus 47.5-48.1 on the shared ones), so the disagreement sits where support was already low. Differences are mostly local: the closest bipartition of the other tree is 1 tip away for 25 of 50 A-only and 21 of 50 G-only splits (≤ 2 tips: 37 and 30); only 8 and 9 are 5+ tips apart. Greedy removal of the tip that lowers RF most: the first (*P. auricula* subsp. *serratifolia*) takes RF from 100 to 86, then about 2 per tip (*A. maxima*, *P. stricta* var. *obesior*, then ordinary tips); the top three are data-poor genomes. A bipartition separates a data-poor genome (the 5 left after the 3 removals) in 1 of 124 shared splits but in 23 of 50 A-only and 10 of 50 G-only splits; removing those 5 genomes gives RF 72, against 96.6 (range 90-100, 40 draws) for 5 random tips. So about 28 of the 100 differences trace to data-poor genomes, and the remaining 72 do not. Backbone: A has a clade of 50 *Primula* + 25 *Androsace* (sCF 76, `sN` 453), G one of 50 + 26 (sCF 73, `sN` 6 677): one tip apart. Genuine conflict: a clade of 72 *Androsace* exists in both trees (A: sCF 34, `sN` 17 127; G: sCF 38, `sN` 49 473) with 16 different tips, i.e. a topological disagreement carried by tens of thousands of sites in each, not a scarcity effect; consistent with the intra-*Androsace* conflict above.
|
||||
|
||||
**Not established.** Separate contribution of entropy / quartet / pair balance (no ablation); stability of the tree between two independent draws (one draw per setting; A → B shows the topology moves with the draw); whether the quartet-informative restriction biases branch lengths (ASC corrects for variable-only ascertainment, not for requiring two states each in two genomes; tree length 22.5 is not interpretable as substitutions/site).
|
||||
|
||||
## Open questions
|
||||
|
||||
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
|
||||
- Combination with entropy bias: not independent criteria if entropy correlates with reach (see section above); options: compensate the known entropy weight inside the pair weight, or apply the Gaussian inside reach strata.
|
||||
- Pairs with near-zero $O_{ij}$ (suspected badly sequenced genomes) set the ceiling for everyone; `--min-shared-family`/`--exclude-genome` should be applied first so the floor is not dictated by anecdotal genomes.
|
||||
- Effect on the SNP-distance estimators: a non-uniform draw over pairs biases per-pair eligible-family counts, but each pair's $p$-distance is a ratio over its own eligible families, so unbiasedness per pair is expected to hold. To confirm before relying on it.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Sibling partition unification and masked canonical form (discussion)
|
||||
|
||||
Status: **open, nothing implemented**. Records the state of the reflection on (a) routing the 4 members of a sibling family to one partition and (b) a canonical form that makes siblings directly comparable. Measurements are throwaway (not in the repo); figures are for $k=31$, $m=11$, 256 partitions, 20 bacterial genomes (98.4 M kmer positions), unless stated.
|
||||
|
||||
## Problem
|
||||
|
||||
A sibling family = the 4 kmers differing only at the central base (position $c=(k-1)/2$, $k$ odd). Routing is **not** a hash of all bases: a kmer goes to `minimizer().seq_hash() & mask`, and consecutive kmers of a superkmer share their minimizer. A family splits across partitions whenever the central base falls in (or changes) the selected m-mer.
|
||||
|
||||
Fraction of families whose 4 variants are not all in one partition, standard minimizer: $4m/(k+2m+1)$ (fixed candidates $k-2m+1$ versus $4m$ variable ones). Measured: 81.4 % at $m=11$ (60.2 % at $m=7$, 46.8 % at $m=5$). This is the *theoretical* rate assuming all 4 members present; for two present members it is ≈ 68 %. Real families are overwhelmingly of size 1, so it only becomes cost where something probes variants.
|
||||
|
||||
## Masked canonical form (Can3)
|
||||
|
||||
Notation: `CentreA(K)` = K with its central base set to A; `RC(K)` its reverse complement.
|
||||
|
||||
- Can1 = `min(K, RC(K))` (current). Each sibling canonicalised alone, so some may be flipped relative to the others.
|
||||
- Can2 = orient by `CentreA(K)` vs `RC(CentreA(K))`. **Not a valid canonical form**: when the flanks are reverse complements of each other ($L=\mathrm{rc}(R)$, frequency ≈ $4^{-c}$, 1/1024 at $k=11$), $K$ and $\mathrm{RC}(K)$ get different canonicals (k=5: `ACAGT` and `ACTGT` both kept). Cause: `CentreA(RC(K)) ≠ RC(CentreA(K))` (the latter has centre T).
|
||||
- **Can3** (retained):
|
||||
|
||||
```
|
||||
Can3(K) = K if CentreA(K) < CentreA(RC(K))
|
||||
= RC(K) if CentreA(K) > CentreA(RC(K))
|
||||
= Can1(K) otherwise // self-revcomp family
|
||||
```
|
||||
|
||||
Orientation is decided on the flanks only (centre ignored), tie broken on the full kmer. Gain: the 4 canonical siblings have identical flanks in the same orientation, so they compare directly with no reverse complement. Self-revcomp families have only 2 distinct canonical members (centres A/C; T/G fold onto them). The A↔T (C↔G) substitution in that context is undetectable by any double-strand canonical set, same as under Can1 — not a loss.
|
||||
|
||||
Gain is simplification only: `central_canonical_neighbors` already canonicalises each variant independently and is correct (see `evolutionary_distances.md`, per-partition sweep). The earlier "masked-flank grouping" there is the Can2 idea; Can3 is its repaired form. **No performance gain by itself**, and it breaks existing indexes (canonical orientation changes for some kmers → MPHF hashes change; rebuild).
|
||||
|
||||
Prerequisite if adopted: `Minimizer = CanonicalKmerOf<MLen>` shares the canonical type with index kmers; an m-mer has no sibling family and must keep Can1. Decouple with a distinct `Minimizer` type (6 files outside `kmer.rs`: `routable.rs`, `packed_seq.rs`, `obiskbuilder/{rolling_stat,stream_iter,iter}.rs`, `obifastwrite`).
|
||||
|
||||
## Routing that excludes the centre (scheme B) — rejected
|
||||
|
||||
Candidates restricted to m-mers not overlapping the centre ($m\le c$): $S(j)=\min$ hash over m-mers $p\in[j,j+c-m]$, minimizer$(i)=\min(S(i),S(i+c+1))$. Family- and strand-invariant by construction (checked). But unitigs are partition-local (`pipeline.md`), and the hole moves with each kmer, so runs shrink:
|
||||
|
||||
| $m$ | mean run, current | mean run, B | bases/kmer, current → B |
|
||||
|---|---|---|---|
|
||||
| 5 | 14.9 | 6.3 | 3.02 → 5.76 |
|
||||
| 11 | 11.0 | 3.0 | 3.72 → 10.96 |
|
||||
| 15 | 9.0 | 1.0 | 4.32 → 30.9 |
|
||||
|
||||
Proxy: run = unitig (upper bound; real unitigs may merge). Independent of partition count. In `benchmark/run/global_index_count` (786 MB) `unitigs.bin` + idx ≈ 241 MB (31 %), so tripling them ≈ +60 % total index. Hashing the masked kmer per kmer (no minimizer) would cut every path at every kmer: worse.
|
||||
|
||||
Structural tension: path coherence (long unitigs) needs a key shared by consecutive kmers, family coherence needs a key independent of each kmer's own centre, which is an ordinary base for its neighbour. Decoupling sequence storage from MPHF partitioning fails because a partition can only enumerate its kmers from its own unitigs (MPHF not invertible). Redefining the family on a terminal base would cut the split rate to ≈ 17 % but is not closed under revcomp and changes the SNP semantics. Lower $m$ reduces the split rate for free but deviates from the literature $k/m\approx 2.5$ optimum (balance, bucket size), unmeasured here — $m=11$ kept.
|
||||
|
||||
## What the cross-partition cost actually is
|
||||
|
||||
- Phylo-time: only families of size > 1 trigger remote lookups (a small share).
|
||||
- Annex construction probes the 3 variants of **every** kmer (size-1 families included), ≈ 2/3 of probes crossing partitions; batch bounded to `#kmers(p)*3` per partition (not the whole index).
|
||||
- Real remaining benefit of unification: the outer sweep over source partitions could run in parallel (it is sequential today for memory and for the `q >= p` dedup rule); plus code simplification in `family_scan.rs` / `annex.rs`. Not quantified.
|
||||
- Unification would also allow an MPHF keyed on the family (one lookup, 4-bit member mask, union/intersection as OR/AND); needs the family-size histogram (1–4 members) to judge storage.
|
||||
|
||||
## Open measurements (index not on this machine)
|
||||
|
||||
1. Family-size histogram of the real index: `obikmer phylo --sibling-hist` (needs a built annex).
|
||||
2. Time and peak RSS of `--sibling-annex` on the Vaccinium/Salix index (`Stage` "sibling_annex" traces time; peak RSS from outside).
|
||||
3. Only if those show the sequential sweep or the probe volume is a real bottleneck: real-index test of scheme B or of a smaller $m$.
|
||||
|
||||
## Decision state
|
||||
|
||||
No code change decided. Can3 + `Minimizer` decoupling is the only self-contained step and is independent of the measurements above; it forces an index rebuild, so it is worth doing only if its simplification is wanted.
|
||||
@@ -1,43 +0,0 @@
|
||||
Voici la version corrigée :
|
||||
|
||||
---
|
||||
|
||||
**Bug** : dans `base_pair_tally`, toutes les transitions/comptes depuis/vers A valent 0 dans `_sankoff_params.yaml`, alors que C/G/T sont corrects.
|
||||
|
||||
**Contexte** : obikmer, pipeline phylogénétique `--sankoff`. L’index est construit sur 20 génomes bactériens. Même symptôme sur un jeu de 100 génomes de plantes : A est toujours à 0.
|
||||
|
||||
**Fichier clé** : `src/obikphylo/src/siblings/sankoff_bundle.rs` (Pass A + Pass B).
|
||||
|
||||
**Ce qui a été vérifié** :
|
||||
- Le fichier de sortie `_sankoff_params.yaml` montre bien `composition_transitions` avec A à 0 partout.
|
||||
- L’index contient bien des familles avec A (`mask.has(0) == true`), et même des familles où A co-existe avec d’autres bases (`mask == 0b0011` par ex.).
|
||||
- Un k-mer propriétaire de famille avec `mask == 0b0001` (A seul) a été identifié : forward `GAACAAGAGATCTCGATCTTGTCTACAAGGA`, revcomp `TCCTTGTAGACAAGATCGAGATCTCTTGTTC`.
|
||||
- Le diagnostic CLI sur l’index réel donne :
|
||||
- Pass A : `a_pairs=623342 a_snp=623342 a_shared=0 a_both_a=0`
|
||||
- Pass B : `families_with_a=22965521 a_single_form_genomes=22913238 a_included_pairs=0 a_same_incremented=0 bp_same=[0, 96389, 222720, 277909] bp_counts[0]=[0, 0, 0, 0]`
|
||||
|
||||
**Interprétation** : A est fréquemment en `single_form` (mask == 1) chez certains génomes, mais **jamais simultanément** chez deux génomes différents dans la même famille. Donc toutes les paires “avec A” sont 100% SNP → ratio = 1.0 > `ratio_ceiling=0.5` → toutes exclues par le filtre `included`. C’est pourquoi `bp_same[0]` et `bp_counts[0][*]` restent à 0.
|
||||
|
||||
**Point crucial** : le bug n’apparaît **que sur l’index compacté sparse**. Sur le même index avant compaction (matrice dense `matrix.pbmx`), `--sankoff` produit des tallies corrects pour A. Dès qu’on compacte avec `pack --sparse`, A disparaît.
|
||||
|
||||
**Vérifications supplémentaires (diagnostic sparse)** :
|
||||
- La compaction `pack --sparse` produit une matrice `PersistentSparseBitMatrix` dont le contenu est **strictement identique** à la matrice dense d'origine : vérification exhaustive coordonnée par coordonnée sur **1 804 774 880 cellules** (512 partitions × 2 layers), **zéro différence**.
|
||||
- `fill_row` et `fill_sub_matrix` (les deux chemins de lecture utilisés par le pipeline phylogénétique) restituent les mêmes bits sur dense et sparse.
|
||||
- **Conclusion** : le bug n'est **pas** dans la compaction sparse elle-même, ni dans les chemins de lecture individuels. La structure stocke correctement A, C, G, T.
|
||||
|
||||
**Conséquence logique** :
|
||||
Si les matrices sont identiques mais que le résultat final diffère, le bug se situe dans l'**intersection** des informations — c'est-à-dire dans le code qui **combine** les lectures des deux matrices (ou qui transforme les résultats bruts en tallies). Deux endroits possibles :
|
||||
1. **Le scan `sankoff_bundle`** (`family_scan.rs` + `sankoff_bundle.rs`) : la boucle qui lit les matrices, construit `genome_mask`, et accumule `bp_counts` / `same`. C'est l'étape d'intersection proprement dite.
|
||||
2. **La conversion des tallies en YAML** (`obikmer/src/cmd/phylo/sankoff.rs`) : moins probable, mais possible si quelque chose sélectionne/filtre les transitions avant écriture.
|
||||
|
||||
**Hypothèse la plus probable** : bug dans la résolution cross-partition lors de la construction de l'annex sibling (`build_sibling_annex`). A (bit 0) serait systématiquement manquant ou mal résolu quand on interroge les variants d'une famille depuis une partition différente. À vérifier dans `src/obikphylo/src/siblings/build.rs` et `src/obikphylo/src/siblings/cache.rs` (`PartitionCache::find` / `find_presence_batch`).
|
||||
|
||||
**Prochaine étape logique** :
|
||||
1. Inspecter `build_sibling_annex` pour voir si les variants avec base A sont bien générés et bien recherchés dans `cache.find`.
|
||||
2. Vérifier `PartitionCache::find` et `resolve_layer_hits` pour un éventuel biais contre le bit 0.
|
||||
3. Si besoin, ajouter un diagnostic ciblé (compteurs par base) **uniquement** dans `cache.rs` ou `build.rs`, pas dans `sankoff_bundle.rs` qui est déjà propre.
|
||||
|
||||
**Contraintes** :
|
||||
- Ne pas modifier `sankoff_bundle.rs` davantage.
|
||||
- Ne pas toucher à git.
|
||||
- Faire des diagnostics minimaux et ciblés.
|
||||
Loaded 100 of 507 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user