Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Eric Coissac
|
1f0233d033 |
@@ -1,8 +1,9 @@
|
||||
pname: CI
|
||||
name: CI
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches: ['main']
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
build:
|
||||
@@ -25,8 +26,8 @@ jobs:
|
||||
~/.cargo/registry
|
||||
~/.cargo/git
|
||||
src/target
|
||||
key: ${{ runner.os }}-cargo-v2-${{ hashFiles('src/Cargo.lock') }}
|
||||
restore-keys: ${{ runner.os }}-cargo-v2-
|
||||
key: ${{ runner.os }}-cargo-${{ hashFiles('src/Cargo.lock') }}
|
||||
restore-keys: ${{ runner.os }}-cargo-
|
||||
|
||||
- name: Build
|
||||
run: cargo build --release
|
||||
|
||||
@@ -11,10 +11,6 @@ jobs:
|
||||
outputs:
|
||||
release_id: ${{ steps.create.outputs.release_id }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Create Gitea release
|
||||
id: create
|
||||
env:
|
||||
@@ -22,12 +18,11 @@ jobs:
|
||||
TAG: ${{ github.ref_name }}
|
||||
run: |
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq jq
|
||||
body=$(git for-each-ref --format='%(contents)' "refs/tags/$TAG")
|
||||
release_id=$(curl -s -X POST \
|
||||
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases" \
|
||||
-H "Authorization: token $GITEA_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"tag_name\":\"$TAG\",\"name\":\"$TAG\",\"body\":$(echo "$body" | jq -Rs .)}" | jq -r '.id')
|
||||
-d "{\"tag_name\":\"$TAG\",\"name\":\"$TAG\"}" | jq -r '.id')
|
||||
echo "release_id=$release_id" >> $GITHUB_OUTPUT
|
||||
|
||||
build-linux-x86_64:
|
||||
@@ -86,11 +81,20 @@ jobs:
|
||||
build-macos-arm64:
|
||||
needs: create-release
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: src
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Login to registry
|
||||
run: echo "${{ secrets.REGISTRYTOKEN }}" | docker login registry.metabarcoding.org -u ${{ secrets.REGISTRYUSER }} --password-stdin
|
||||
- name: Install Rust + zigbuild
|
||||
run: |
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
|
||||
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq jq
|
||||
pip install ziglang --quiet --break-system-packages
|
||||
$HOME/.cargo/bin/cargo install cargo-zigbuild
|
||||
$HOME/.cargo/bin/rustup target add aarch64-apple-darwin
|
||||
|
||||
- name: Cache cargo registry
|
||||
uses: actions/cache@v4
|
||||
@@ -103,24 +107,15 @@ jobs:
|
||||
restore-keys: macos-arm64-cargo-
|
||||
|
||||
- name: Build macOS binary
|
||||
run: |
|
||||
CID=$(docker create \
|
||||
-w /src/src \
|
||||
registry.metabarcoding.org/cibuilder/rustcrossosx:latest \
|
||||
cargo build --release --target aarch64-apple-darwin --no-default-features)
|
||||
docker cp . "$CID:/src"
|
||||
docker start -a "$CID"
|
||||
STATUS=$(docker wait "$CID")
|
||||
mkdir -p /tmp/dist
|
||||
docker cp "$CID:/src/src/target/aarch64-apple-darwin/release/obikmer" /tmp/dist/obikmer-macos-arm64
|
||||
docker rm "$CID" > /dev/null
|
||||
[ "$STATUS" -eq 0 ]
|
||||
run: cargo zigbuild --release --target aarch64-apple-darwin --no-default-features
|
||||
|
||||
- name: Prepare and upload artifact
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEATOKEN }}
|
||||
RELEASE_ID: ${{ needs.create-release.outputs.release_id }}
|
||||
run: |
|
||||
mkdir -p /tmp/dist
|
||||
cp target/aarch64-apple-darwin/release/obikmer /tmp/dist/obikmer-macos-arm64
|
||||
curl -s -X POST \
|
||||
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases/$RELEASE_ID/assets" \
|
||||
-H "Authorization: token $GITEA_TOKEN" \
|
||||
|
||||
+13
-36
@@ -1,46 +1,23 @@
|
||||
.venv/
|
||||
.DS_Store
|
||||
.claude/
|
||||
.kilo/
|
||||
.serena/
|
||||
.zed/
|
||||
.ast-cache/
|
||||
CLAUDE.md
|
||||
memory/
|
||||
sandbox/
|
||||
src/target
|
||||
data-stress
|
||||
*.fasta
|
||||
*.fasta.gz
|
||||
*.fastq
|
||||
*.fastq.gz
|
||||
*.vcf
|
||||
*.zst
|
||||
*.zst.meta
|
||||
*.pb
|
||||
**/*.json
|
||||
./**/*.json
|
||||
*.bin
|
||||
*.log
|
||||
*.csv
|
||||
*.meta
|
||||
*.pfiv
|
||||
*.done
|
||||
*.efh
|
||||
*.efl
|
||||
*.bin.idx
|
||||
*.prsb
|
||||
Betula_exilis--IGA-24-33
|
||||
benchmark/run/
|
||||
TNT
|
||||
phyg
|
||||
biblio
|
||||
*.tnt
|
||||
*.tre
|
||||
*.phy
|
||||
*.treefile
|
||||
*.bionj
|
||||
*.iqtree
|
||||
*.mldist
|
||||
*.parstree
|
||||
*.ckp.gz
|
||||
*.model
|
||||
benchmark/genomes
|
||||
benchmark/simulated_data
|
||||
benchmark/specimen_index_presence
|
||||
benchmark/specimen_index_count
|
||||
benchmark/global_index_presence
|
||||
benchmark/global_index_count
|
||||
benchmark/stats
|
||||
benchmark/reference_index
|
||||
benchmark/reference_dist
|
||||
benchmark/obikmer_dist
|
||||
benchmark/specific_index_count
|
||||
benchmark/specific_index_presence
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# Plan d'amélioration technique - obiskio
|
||||
|
||||
## 1. Contexte et objectifs
|
||||
- **Objectif** : Renforcer la robustesse, la maintenabilité et les performances de la crate `obiskio`.
|
||||
- **Priorités** :
|
||||
1. Gestion des erreurs
|
||||
2. Optimisation de la mémoire du pool
|
||||
3. Robustesse concurrente
|
||||
4. Couverture de tests
|
||||
5. Documentation
|
||||
|
||||
---
|
||||
|
||||
## 2. Axes d'amélioration détaillés
|
||||
|
||||
### 2.1 Gestion des erreurs
|
||||
- **Problème** : `SKError` ne couvre pas tous les cas (format invalide, taille maximale, CRC)
|
||||
- **Actions** :
|
||||
- Ajouter variante `ParseError(String)` dans `src/error.rs`
|
||||
- Valider les tailles de SuperKmer avant parsing
|
||||
- Remplacer `expect()` par `unwrap_or_else` avec messages explicites
|
||||
- Documenter chaque variante d’erreur dans le README
|
||||
|
||||
### 2.2 Optimisation du pool de fichiers
|
||||
- **Problème** : `SKFilePool` utilise un `Vec<WriteEntry>` non contraint et n’effectue pas de nettoyage en cas d’erreur
|
||||
- **Actions** :
|
||||
- Implémenter un `LimitedVec` avec limite stricte à `MAX_POOL_SIZE`
|
||||
- Créer `clear_memory()` qui supprime les entrées orphelines
|
||||
- Ajouter `evict_lru_threshold()` pour éviction proactive
|
||||
- Introduire un `RwLock` pour les opérations de lecture massives
|
||||
|
||||
### 2.3 Robustesse concurrente
|
||||
- **Problème** : Risque de deadlocks dans `SKFileWriter::write_batch()` et `SKFileReader::reopen_and_seek()`
|
||||
- **Actions** :
|
||||
- Remplacer `Mutex` par `RwLock` pour les accès en lecture
|
||||
- Ajouter un compteur de blocage et logs de timeout
|
||||
- Utiliser `std::thread::park_timeout` pour débloquer
|
||||
- Insérer `debug_assert!` sur les états invariants
|
||||
|
||||
### 2.4 Couverture de tests
|
||||
- **Problème** : Absence de benchmarks, de tests de migration, de résilience de fichiers corrompus
|
||||
- **Actions** :
|
||||
- Benchmarks I/O sur 10k+ SuperKmer avec `criterion`
|
||||
- Tests de migration de version de fichier `.meta` → `.v2.meta`
|
||||
- Tests de corruption volontaire (truncature, inversion de bits)
|
||||
- Tests de stress sur pool saturation (100 threads)
|
||||
|
||||
### 2.5 Documentation & exemples
|
||||
- **Actions** :
|
||||
- Ajouter des examples dans chaque module (`# Examples`)
|
||||
- Documenter la logique LRU avec diagrammes Mermaid
|
||||
- Créer un guide « How to recover from eviction »
|
||||
- Mettre à jour le `README.md` avec tableau des variantes d’erreur
|
||||
|
||||
---
|
||||
|
||||
## 3. Plan d'exécution (Roadmap)
|
||||
|
||||
| Sprint | Durée | Livrables clés |
|
||||
|--------|-------|----------------|
|
||||
| **S1** | 2 jours | Refactorisation `SKError`, ajout de tests unitaires |
|
||||
| **S2** | 3 jours | Implémentation `clear_memory()` + `LimitedVec` |
|
||||
| **S3** | 2 jours | Passage à `RwLock`, ajout de compteurs de blocage |
|
||||
| **S4** | 2 jours | Benchmarks + tests de migration |
|
||||
| **S5** | 1 jour | Documentation finale & mise à jour du README |
|
||||
|
||||
---
|
||||
|
||||
## 4. Dépendances externes
|
||||
- Mettre à jour `niffler` vers la version 2.0 (performance compression)
|
||||
- Évaluer `bincode` vs `serde_json` pour les métas (I/O)
|
||||
- Ajouter dépendance `criterion` (dev‑dependencies)
|
||||
|
||||
---
|
||||
|
||||
## 5. KPI de suivi
|
||||
- **Couverture de tests** : ≥85 % des chemins critiques
|
||||
- **Latence moyenne d’écriture** : ↓15 % après optimisation du pool
|
||||
- **Taux d’erreurs résolues** : 100 % des nouvelles variantes couvertes
|
||||
- **Temps de build CI** : ≤5 min pour l’ensemble des benchmarks
|
||||
|
||||
---
|
||||
|
||||
## 6. Validation finale
|
||||
- Revue de code avec `cargo clippy -- -D warnings`
|
||||
- Analyse de toxicité avec `cargo deny open-source-licenses`
|
||||
- Vérification de la conformité aux standards de naming du projet
|
||||
@@ -0,0 +1,2 @@
|
||||
/cache
|
||||
/project.local.yml
|
||||
@@ -0,0 +1,133 @@
|
||||
# the name by which the project can be referenced within Serena
|
||||
project_name: "obikmer"
|
||||
|
||||
|
||||
# list of languages for which language servers are started; choose from:
|
||||
# al angular ansible bash clojure
|
||||
# cpp cpp_ccls crystal csharp csharp_omnisharp
|
||||
# dart elixir elm erlang fortran
|
||||
# fsharp go groovy haskell haxe
|
||||
# hlsl html java json julia
|
||||
# kotlin lean4 lua luau markdown
|
||||
# matlab msl nix ocaml pascal
|
||||
# perl php php_phpactor powershell python
|
||||
# python_jedi python_ty r rego ruby
|
||||
# ruby_solargraph rust scala scss solidity
|
||||
# svelte swift systemverilog terraform toml
|
||||
# typescript typescript_vts vue yaml zig
|
||||
# (This list may be outdated. For the current list, see values of Language enum here:
|
||||
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
|
||||
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
|
||||
# Note:
|
||||
# - For C, use cpp
|
||||
# - For JavaScript, use typescript
|
||||
# - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
|
||||
# - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
|
||||
# - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
|
||||
# - For Free Pascal/Lazarus, use pascal
|
||||
# Special requirements:
|
||||
# Some languages require additional setup/installations.
|
||||
# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
|
||||
# When using multiple languages, the first language server that supports a given file will be used for that file.
|
||||
# The first language is the default language and the respective language server will be used as a fallback.
|
||||
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
|
||||
languages:
|
||||
- rust
|
||||
|
||||
# the encoding used by text files in the project
|
||||
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
|
||||
encoding: "utf-8"
|
||||
|
||||
# line ending convention to use when writing source files.
|
||||
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
|
||||
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
|
||||
line_ending:
|
||||
|
||||
# The language backend to use for this project.
|
||||
# If not set, the global setting from serena_config.yml is used.
|
||||
# Valid values: LSP, JetBrains
|
||||
# Note: the backend is fixed at startup. If a project with a different backend
|
||||
# is activated post-init, an error will be returned.
|
||||
language_backend:
|
||||
|
||||
# whether to use project's .gitignore files to ignore files
|
||||
ignore_all_files_in_gitignore: true
|
||||
|
||||
# advanced configuration option allowing to configure language server-specific options.
|
||||
# Maps the language key to the options.
|
||||
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
|
||||
# No documentation on options means no options are available.
|
||||
ls_specific_settings: {}
|
||||
|
||||
# list of additional workspace folder paths for cross-package reference support (e.g. in monorepos).
|
||||
# Paths can be absolute or relative to the project root.
|
||||
# Each folder is registered as an LSP workspace folder, enabling language servers to discover
|
||||
# symbols and references across package boundaries.
|
||||
# Currently supported for: TypeScript.
|
||||
# Example:
|
||||
# additional_workspace_folders:
|
||||
# - ../sibling-package
|
||||
# - ../shared-lib
|
||||
additional_workspace_folders: []
|
||||
|
||||
# list of additional paths to ignore in this project.
|
||||
# Same syntax as gitignore, so you can use * and **.
|
||||
# Note: global ignored_paths from serena_config.yml are also applied additively.
|
||||
ignored_paths: []
|
||||
|
||||
# whether the project is in read-only mode
|
||||
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
|
||||
# Added on 2025-04-18
|
||||
read_only: false
|
||||
|
||||
# list of tool names to exclude.
|
||||
# This extends the existing exclusions (e.g. from the global configuration)
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
excluded_tools: []
|
||||
|
||||
# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
|
||||
# This extends the existing inclusions (e.g. from the global configuration).
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
included_optional_tools: []
|
||||
|
||||
# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
|
||||
# This cannot be combined with non-empty excluded_tools or included_optional_tools.
|
||||
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
|
||||
fixed_tools: []
|
||||
|
||||
# list of mode names that are to be activated by default, overriding the setting in the global configuration.
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
|
||||
# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
|
||||
# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
|
||||
# for this project.
|
||||
# This setting can, in turn, be overridden by CLI parameters (--mode).
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
default_modes:
|
||||
|
||||
# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
|
||||
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
|
||||
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
|
||||
added_modes:
|
||||
|
||||
# initial prompt for the project. It will always be given to the LLM upon activating the project
|
||||
# (contrary to the memories, which are loaded on demand).
|
||||
initial_prompt: ""
|
||||
|
||||
# time budget (seconds) per tool call for the retrieval of additional symbol information
|
||||
# such as docstrings or parameter information.
|
||||
# This overrides the corresponding setting in the global configuration; see the documentation there.
|
||||
# If null or missing, use the setting from the global configuration.
|
||||
symbol_info_budget:
|
||||
|
||||
# list of regex patterns which, when matched, mark a memory entry as read‑only.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
read_only_memory_patterns: []
|
||||
|
||||
# list of regex patterns for memories to completely ignore.
|
||||
# Matching memories will not appear in list_memories or activate_project output
|
||||
# and cannot be accessed via read_memory or write_memory.
|
||||
# To access ignored memory files, use the read_file tool on the raw file path.
|
||||
# Extends the list from the global configuration, merging the two lists.
|
||||
# Example: ["_archive/.*", "_episodes/.*"]
|
||||
ignored_memory_patterns: []
|
||||
@@ -0,0 +1,59 @@
|
||||
// Project tasks configuration. See https://zed.dev/docs/tasks for documentation.
|
||||
//
|
||||
// Example:
|
||||
[
|
||||
{
|
||||
"label": "Example task",
|
||||
"command": "for i in {1..5}; do echo \"Hello $i/5\"; sleep 1; done",
|
||||
//"args": [],
|
||||
// Env overrides for the command, will be appended to the terminal's environment from the settings.
|
||||
"env": { "foo": "bar" },
|
||||
// Current working directory to spawn the command into, defaults to current project root.
|
||||
//"cwd": "/path/to/working/directory",
|
||||
// Whether to use a new terminal tab or reuse the existing one to spawn the process, defaults to `false`.
|
||||
"use_new_terminal": false,
|
||||
// Whether to allow multiple instances of the same task to be run, or rather wait for the existing ones to finish, defaults to `false`.
|
||||
"allow_concurrent_runs": false,
|
||||
// What to do with the terminal pane and tab, after the command was started:
|
||||
// * `always` — always show the task's pane, and focus the corresponding tab in it (default)
|
||||
// * `no_focus` — always show the task's pane, add the task's tab in it, but don't focus it
|
||||
// * `never` — do not alter focus, but still add/reuse the task's tab in its pane
|
||||
"reveal": "always",
|
||||
// Where to place the task's terminal item after starting the task:
|
||||
// * `dock` — in the terminal dock, "regular" terminal items' place (default)
|
||||
// * `center` — in the central pane group, "main" editor area
|
||||
"reveal_target": "dock",
|
||||
// What to do with the terminal pane and tab, after the command had finished:
|
||||
// * `never` — Do nothing when the command finishes (default)
|
||||
// * `always` — always hide the terminal tab, hide the pane also if it was the last tab in it
|
||||
// * `on_success` — hide the terminal tab on task success only, otherwise behaves similar to `always`
|
||||
"hide": "never",
|
||||
// Which shell to use when running a task inside the terminal.
|
||||
// May take 3 values:
|
||||
// 1. (default) Use the system's default terminal configuration in /etc/passwd
|
||||
// "shell": "system"
|
||||
// 2. A program:
|
||||
// "shell": {
|
||||
// "program": "sh"
|
||||
// }
|
||||
// 3. A program with arguments:
|
||||
// "shell": {
|
||||
// "with_arguments": {
|
||||
// "program": "/bin/bash",
|
||||
// "args": ["--login"]
|
||||
// }
|
||||
// }
|
||||
"shell": "system",
|
||||
// Whether to show the task line in the output of the spawned task, defaults to `true`.
|
||||
"show_summary": true,
|
||||
// Whether to show the command line in the output of the spawned task, defaults to `true`.
|
||||
"show_command": true,
|
||||
// Which edited buffers to save before running the task:
|
||||
// * `all` — save all edited buffers
|
||||
// * `current` — save currently active buffer only
|
||||
// * `none` — don't save any buffers
|
||||
"save": "none",
|
||||
// Represents the tags for inline runnable indicators, or spawning multiple tasks at once.
|
||||
// "tags": []
|
||||
},
|
||||
]
|
||||
@@ -0,0 +1,101 @@
|
||||
**PROMPT**
|
||||
|
||||
Tu es ma base de connaissance et mon bloc-notes intelligent sur le projet **obikmer**. Tu ne proposes pas, tu ne codes pas spontanément — tu réponds à mes questions et tu structures mes idées au fur et à mesure que je les exprime.
|
||||
|
||||
**Règle absolue : une question appelle une réponse, pas une action.**
|
||||
Ne modifier aucun fichier à moins d'une demande explicite de modification. En particulier : observer un bug ou une incohérence dans le code montré ne constitue pas un mandat pour le corriger. Le code montré peut refléter une intention en cours — modifier sans mandat risque d'introduire un vrai bug là où tu croyais corriger.
|
||||
|
||||
**Règle absolue : ne jamais substituer une dépendance ou une bibliothèque sans validation explicite.**
|
||||
Si une dépendance demandée pose problème (erreur de compilation, bug, API manquante), exposer le problème et proposer des alternatives — ne jamais switcher silencieusement vers une autre bibliothèque. Le choix des dépendances est une décision d'architecture qui appartient au développeur.
|
||||
|
||||
**Règle absolue : le code existant est une hypothèse, pas une vérité.**
|
||||
Quand une nouvelle construction (type, itérateur, abstraction) rend du code historique injustifié, le signaler immédiatement et proposer de le supprimer — ne pas conserver les deux en parallèle par inertie. Le développeur demande explicitement de remettre en cause le code base : ne pas attendre qu'il insiste.
|
||||
|
||||
Tu maintiens en **anglais**, dense et sans remplissage, les documents suivants :
|
||||
- `docmd/index.md` — document de discussion de base, enrichi progressivement au fil de nos échanges ; il reflète l'état courant de la réflexion sur le projet
|
||||
- les autres fichiers Markdown dans `docmd/` selon leur thème respectif
|
||||
|
||||
Les snippets de code y sont courts et illustrent uniquement des principes architecturaux. Nos échanges se font en **français**.
|
||||
|
||||
---
|
||||
|
||||
## Contexte du projet
|
||||
|
||||
`obikmer` est un outil Rust de manipulation, comptage, indexation et opérations ensemblistes sur des séquences ADN représentées comme des ensembles de kmers.
|
||||
|
||||
**Contraintes fondamentales**
|
||||
- Efficacité maximale en calcul, mémoire et disque
|
||||
- Données métagénomiques : plusieurs dizaines de Gbases, milliards de kmers
|
||||
- k impair, k ∈ [11, 31], fixé à l'exécution
|
||||
- Formats d'entrée : FASTA, FASTQ, gzip, streaming stdin
|
||||
|
||||
**Opérations prioritaires**
|
||||
- Comptage de kmers (fréquences)
|
||||
- Recherche / requête rapide
|
||||
- Opérations ensemblistes (union, intersection, diff)
|
||||
|
||||
**Ce qui a déjà été discuté**
|
||||
- Encodage 2 bits/base → kmer tient dans un `u64`
|
||||
- Forme canonique : `min(kmer, revcomp)` pour réduire l'espace de moitié
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure de documentation
|
||||
|
||||
La documentation est gérée via **MkDocs + thème Material**, avec publication sur **GitHub Pages**.
|
||||
|
||||
**Structure des répertoires**
|
||||
|
||||
```
|
||||
docmd/ ← sources Markdown + mkdocs.yml
|
||||
docmd/mkdocs.yml
|
||||
doc/ ← site HTML généré (servi par GitHub Pages)
|
||||
.venv/ ← environnement Python (ignoré par git)
|
||||
```
|
||||
|
||||
**Configuration `docmd/mkdocs.yml`**
|
||||
- `docs_dir: .` (sources = `docmd/` lui-même)
|
||||
- `site_dir: ../doc` (sortie = `doc/`)
|
||||
|
||||
**Commandes Makefile**
|
||||
|
||||
| Commande | Effet |
|
||||
|---|---|
|
||||
| `make doc` | Construit le HTML dans `doc/` |
|
||||
| `make doc-serve` | Serveur local avec rechargement automatique |
|
||||
| `make clean-doc` | Supprime `doc/` |
|
||||
| `make clean` | Supprime `doc/` et `.venv/` |
|
||||
|
||||
Le `.venv/` est dans `.gitignore`. Le répertoire `doc/` (sortie HTML) est versionné pour GitHub Pages.
|
||||
|
||||
Lors de l'ajout de nouveaux fichiers Markdown dans `docmd/`, mettre à jour la section `nav:` de `docmd/mkdocs.yml`.
|
||||
|
||||
---
|
||||
|
||||
Je continue à poser mes questions et à guider la discussion.
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools
|
||||
|
||||
**Règle absolue : avant tout travail de code, appeler `mcp__serena__initial_instructions` pour charger les instructions Serena.**
|
||||
|
||||
### Hiérarchie des outils pour ce projet Rust
|
||||
|
||||
**Navigation et édition de code → serena en priorité**
|
||||
- Trouver un symbole, une déclaration, les implémentations d'un trait : `mcp__serena__find_symbol`, `mcp__serena__find_declaration`, `mcp__serena__find_implementations`
|
||||
- Trouver les usages d'un symbole : `mcp__serena__find_referencing_symbols`
|
||||
- Diagnostics LSP (erreurs de compilation) : `mcp__serena__get_diagnostics_for_file`
|
||||
- Vue d'ensemble d'un fichier : `mcp__serena__get_symbols_overview`
|
||||
- Modifier le corps d'une fonction/impl : `mcp__serena__replace_symbol_body`
|
||||
- Ne pas utiliser `cclsp` quand serena couvre le besoin
|
||||
|
||||
**Analyse architecturale → jcodemunch**
|
||||
- Hotspots, couplage, dead code, dépendances entre modules
|
||||
- Utiliser avant de refactorer une zone critique
|
||||
|
||||
**Raisonnement complexe → sequential-thinking**
|
||||
- Décisions d'architecture, choix d'algorithme, trade-offs non triviaux
|
||||
|
||||
**Documentation de crates → context7**
|
||||
- Toujours consulter avant d'utiliser une API de bibliothèque externe
|
||||
-1275
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 1.8 KiB |
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,18 +0,0 @@
|
||||
/*!
|
||||
* Lunr languages, `Danish` language
|
||||
* https://github.com/MihaiValentin/lunr-languages
|
||||
*
|
||||
* Copyright 2014, Mihai Valentin
|
||||
* http://www.mozilla.org/MPL/
|
||||
*/
|
||||
/*!
|
||||
* based on
|
||||
* Snowball JavaScript Library v0.3
|
||||
* http://code.google.com/p/urim/
|
||||
* http://snowball.tartarus.org/
|
||||
*
|
||||
* Copyright 2010, Oleg Mazko
|
||||
* http://www.mozilla.org/MPL/
|
||||
*/
|
||||
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.da=function(){this.pipeline.reset(),this.pipeline.add(e.da.trimmer,e.da.stopWordFilter,e.da.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.da.stemmer))},e.da.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.da.trimmer=e.trimmerSupport.generateTrimmer(e.da.wordCharacters),e.Pipeline.registerFunction(e.da.trimmer,"trimmer-da"),e.da.stemmer=function(){var r=e.stemmerSupport.Among,i=e.stemmerSupport.SnowballProgram,n=new function(){function e(){var e,r=f.cursor+3;if(d=f.limit,0<=r&&r<=f.limit){for(a=r;;){if(e=f.cursor,f.in_grouping(w,97,248)){f.cursor=e;break}if(f.cursor=e,e>=f.limit)return;f.cursor++}for(;!f.out_grouping(w,97,248);){if(f.cursor>=f.limit)return;f.cursor++}d=f.cursor,d<a&&(d=a)}}function n(){var e,r;if(f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(c,32),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del();break;case 2:f.in_grouping_b(p,97,229)&&f.slice_del()}}function t(){var e,r=f.limit-f.cursor;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.find_among_b(l,4)?(f.bra=f.cursor,f.limit_backward=e,f.cursor=f.limit-r,f.cursor>f.limit_backward&&(f.cursor--,f.bra=f.cursor,f.slice_del())):f.limit_backward=e)}function s(){var e,r,i,n=f.limit-f.cursor;if(f.ket=f.cursor,f.eq_s_b(2,"st")&&(f.bra=f.cursor,f.eq_s_b(2,"ig")&&f.slice_del()),f.cursor=f.limit-n,f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(m,5),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del(),i=f.limit-f.cursor,t(),f.cursor=f.limit-i;break;case 2:f.slice_from("løs")}}function o(){var e;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.out_grouping_b(w,97,248)?(f.bra=f.cursor,u=f.slice_to(u),f.limit_backward=e,f.eq_v_b(u)&&f.slice_del()):f.limit_backward=e)}var a,d,u,c=[new r("hed",-1,1),new r("ethed",0,1),new r("ered",-1,1),new r("e",-1,1),new r("erede",3,1),new r("ende",3,1),new r("erende",5,1),new r("ene",3,1),new r("erne",3,1),new r("ere",3,1),new r("en",-1,1),new r("heden",10,1),new r("eren",10,1),new r("er",-1,1),new r("heder",13,1),new r("erer",13,1),new r("s",-1,2),new r("heds",16,1),new r("es",16,1),new r("endes",18,1),new r("erendes",19,1),new r("enes",18,1),new r("ernes",18,1),new r("eres",18,1),new r("ens",16,1),new r("hedens",24,1),new r("erens",24,1),new r("ers",16,1),new r("ets",16,1),new r("erets",28,1),new r("et",-1,1),new r("eret",30,1)],l=[new r("gd",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("elig",1,1),new r("els",-1,1),new r("løst",-1,2)],w=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],p=[239,254,42,3,0,0,0,0,0,0,0,0,0,0,0,0,16],f=new i;this.setCurrent=function(e){f.setCurrent(e)},this.getCurrent=function(){return f.getCurrent()},this.stem=function(){var r=f.cursor;return e(),f.limit_backward=r,f.cursor=f.limit,n(),f.cursor=f.limit,t(),f.cursor=f.limit,s(),f.cursor=f.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return n.setCurrent(e),n.stem(),n.getCurrent()}):(n.setCurrent(e),n.stem(),n.getCurrent())}}(),e.Pipeline.registerFunction(e.da.stemmer,"stemmer-da"),e.da.stopWordFilter=e.generateStopWordFilter("ad af alle alt anden at blev blive bliver da de dem den denne der deres det dette dig din disse dog du efter eller en end er et for fra ham han hans har havde have hende hendes her hos hun hvad hvis hvor i ikke ind jeg jer jo kunne man mange med meget men mig min mine mit mod ned noget nogle nu når og også om op os over på selv sig sin sine sit skal skulle som sådan thi til ud under var vi vil ville vor være været".split(" ")),e.Pipeline.registerFunction(e.da.stopWordFilter,"stopWordFilter-da")}});
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hi=function(){this.pipeline.reset(),this.pipeline.add(e.hi.trimmer,e.hi.stopWordFilter,e.hi.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.hi.stemmer))},e.hi.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿa-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.hi.trimmer=e.trimmerSupport.generateTrimmer(e.hi.wordCharacters),e.Pipeline.registerFunction(e.hi.trimmer,"trimmer-hi"),e.hi.stopWordFilter=e.generateStopWordFilter("अत अपना अपनी अपने अभी अंदर आदि आप इत्यादि इन इनका इन्हीं इन्हें इन्हों इस इसका इसकी इसके इसमें इसी इसे उन उनका उनकी उनके उनको उन्हीं उन्हें उन्हों उस उसके उसी उसे एक एवं एस ऐसे और कई कर करता करते करना करने करें कहते कहा का काफ़ी कि कितना किन्हें किन्हों किया किर किस किसी किसे की कुछ कुल के को कोई कौन कौनसा गया घर जब जहाँ जा जितना जिन जिन्हें जिन्हों जिस जिसे जीधर जैसा जैसे जो तक तब तरह तिन तिन्हें तिन्हों तिस तिसे तो था थी थे दबारा दिया दुसरा दूसरे दो द्वारा न नके नहीं ना निहायत नीचे ने पर पहले पूरा पे फिर बनी बही बहुत बाद बाला बिलकुल भी भीतर मगर मानो मे में यदि यह यहाँ यही या यिह ये रखें रहा रहे ऱ्वासा लिए लिये लेकिन व वग़ैरह वर्ग वह वहाँ वहीं वाले वुह वे वो सकता सकते सबसे सभी साथ साबुत साभ सारा से सो संग ही हुआ हुई हुए है हैं हो होता होती होते होना होने".split(" ")),e.hi.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.hi.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var t=i.toString().toLowerCase().replace(/^\s+/,"");return r.cut(t).split("|")},e.Pipeline.registerFunction(e.hi.stemmer,"stemmer-hi"),e.Pipeline.registerFunction(e.hi.stopWordFilter,"stopWordFilter-hi")}});
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hy=function(){this.pipeline.reset(),this.pipeline.add(e.hy.trimmer,e.hy.stopWordFilter)},e.hy.wordCharacters="[A-Za-z-֏ff-ﭏ]",e.hy.trimmer=e.trimmerSupport.generateTrimmer(e.hy.wordCharacters),e.Pipeline.registerFunction(e.hy.trimmer,"trimmer-hy"),e.hy.stopWordFilter=e.generateStopWordFilter("դու և եք էիր էիք հետո նաև նրանք որը վրա է որ պիտի են այս մեջ ն իր ու ի այդ որոնք այն կամ էր մի ես համար այլ իսկ էին ենք հետ ին թ էինք մենք նրա նա դուք եմ էի ըստ որպես ում".split(" ")),e.Pipeline.registerFunction(e.hy.stopWordFilter,"stopWordFilter-hy"),e.hy.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}(),e.Pipeline.registerFunction(e.hy.stemmer,"stemmer-hy")}});
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.ja=function(){this.pipeline.reset(),this.pipeline.add(e.ja.trimmer,e.ja.stopWordFilter,e.ja.stemmer),r?this.tokenizer=e.ja.tokenizer:(e.tokenizer&&(e.tokenizer=e.ja.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.ja.tokenizer))};var t=new e.TinySegmenter;e.ja.tokenizer=function(i){var n,o,s,p,a,u,m,l,c,f;if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t.toLowerCase()):t.toLowerCase()});for(o=i.toString().toLowerCase().replace(/^\s+/,""),n=o.length-1;n>=0;n--)if(/\S/.test(o.charAt(n))){o=o.substring(0,n+1);break}for(a=[],s=o.length,c=0,l=0;c<=s;c++)if(u=o.charAt(c),m=c-l,u.match(/\s/)||c==s){if(m>0)for(p=t.segment(o.slice(l,c)).filter(function(e){return!!e}),f=l,n=0;n<p.length;n++)r?a.push(new e.Token(p[n],{position:[f,p[n].length],index:a.length})):a.push(p[n]),f+=p[n].length;l=c+1}return a},e.ja.stemmer=function(){return function(e){return e}}(),e.Pipeline.registerFunction(e.ja.stemmer,"stemmer-ja"),e.ja.wordCharacters="一二三四五六七八九十百千万億兆一-龠々〆ヵヶぁ-んァ-ヴーｱ-ﾝﾞa-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.ja.trimmer=e.trimmerSupport.generateTrimmer(e.ja.wordCharacters),e.Pipeline.registerFunction(e.ja.trimmer,"trimmer-ja"),e.ja.stopWordFilter=e.generateStopWordFilter("これ それ あれ この その あの ここ そこ あそこ こちら どこ だれ なに なん 何 私 貴方 貴方方 我々 私達 あの人 あのかた 彼女 彼 です あります おります います は が の に を で え から まで より も どの と し それで しかし".split(" ")),e.Pipeline.registerFunction(e.ja.stopWordFilter,"stopWordFilter-ja"),e.jp=e.ja,e.Pipeline.registerFunction(e.jp.stemmer,"stemmer-jp"),e.Pipeline.registerFunction(e.jp.trimmer,"trimmer-jp"),e.Pipeline.registerFunction(e.jp.stopWordFilter,"stopWordFilter-jp")}});
|
||||
@@ -1 +0,0 @@
|
||||
module.exports=require("./lunr.ja");
|
||||
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.kn=function(){this.pipeline.reset(),this.pipeline.add(e.kn.trimmer,e.kn.stopWordFilter,e.kn.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.kn.stemmer))},e.kn.wordCharacters="ಀ-಄ಅ-ಔಕ-ಹಾ-ೌ಼-ಽೕ-ೖೝ-ೞೠ-ೡೢ-ೣ೦-೯ೱ-ೳ",e.kn.trimmer=e.trimmerSupport.generateTrimmer(e.kn.wordCharacters),e.Pipeline.registerFunction(e.kn.trimmer,"trimmer-kn"),e.kn.stopWordFilter=e.generateStopWordFilter("ಮತ್ತು ಈ ಒಂದು ರಲ್ಲಿ ಹಾಗೂ ಎಂದು ಅಥವಾ ಇದು ರ ಅವರು ಎಂಬ ಮೇಲೆ ಅವರ ತನ್ನ ಆದರೆ ತಮ್ಮ ನಂತರ ಮೂಲಕ ಹೆಚ್ಚು ನ ಆ ಕೆಲವು ಅನೇಕ ಎರಡು ಹಾಗು ಪ್ರಮುಖ ಇದನ್ನು ಇದರ ಸುಮಾರು ಅದರ ಅದು ಮೊದಲ ಬಗ್ಗೆ ನಲ್ಲಿ ರಂದು ಇತರ ಅತ್ಯಂತ ಹೆಚ್ಚಿನ ಸಹ ಸಾಮಾನ್ಯವಾಗಿ ನೇ ಹಲವಾರು ಹೊಸ ದಿ ಕಡಿಮೆ ಯಾವುದೇ ಹೊಂದಿದೆ ದೊಡ್ಡ ಅನ್ನು ಇವರು ಪ್ರಕಾರ ಇದೆ ಮಾತ್ರ ಕೂಡ ಇಲ್ಲಿ ಎಲ್ಲಾ ವಿವಿಧ ಅದನ್ನು ಹಲವು ರಿಂದ ಕೇವಲ ದ ದಕ್ಷಿಣ ಗೆ ಅವನ ಅತಿ ನೆಯ ಬಹಳ ಕೆಲಸ ಎಲ್ಲ ಪ್ರತಿ ಇತ್ಯಾದಿ ಇವು ಬೇರೆ ಹೀಗೆ ನಡುವೆ ಇದಕ್ಕೆ ಎಸ್ ಇವರ ಮೊದಲು ಶ್ರೀ ಮಾಡುವ ಇದರಲ್ಲಿ ರೀತಿಯ ಮಾಡಿದ ಕಾಲ ಅಲ್ಲಿ ಮಾಡಲು ಅದೇ ಈಗ ಅವು ಗಳು ಎ ಎಂಬುದು ಅವನು ಅಂದರೆ ಅವರಿಗೆ ಇರುವ ವಿಶೇಷ ಮುಂದೆ ಅವುಗಳ ಮುಂತಾದ ಮೂಲ ಬಿ ಮೀ ಒಂದೇ ಇನ್ನೂ ಹೆಚ್ಚಾಗಿ ಮಾಡಿ ಅವರನ್ನು ಇದೇ ಯ ರೀತಿಯಲ್ಲಿ ಜೊತೆ ಅದರಲ್ಲಿ ಮಾಡಿದರು ನಡೆದ ಆಗ ಮತ್ತೆ ಪೂರ್ವ ಆತ ಬಂದ ಯಾವ ಒಟ್ಟು ಇತರೆ ಹಿಂದೆ ಪ್ರಮಾಣದ ಗಳನ್ನು ಕುರಿತು ಯು ಆದ್ದರಿಂದ ಅಲ್ಲದೆ ನಗರದ ಮೇಲಿನ ಏಕೆಂದರೆ ರಷ್ಟು ಎಂಬುದನ್ನು ಬಾರಿ ಎಂದರೆ ಹಿಂದಿನ ಆದರೂ ಆದ ಸಂಬಂಧಿಸಿದ ಮತ್ತೊಂದು ಸಿ ಆತನ ".split(" ")),e.kn.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.kn.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var n=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(n).split("|")},e.Pipeline.registerFunction(e.kn.stemmer,"stemmer-kn"),e.Pipeline.registerFunction(e.kn.stopWordFilter,"stopWordFilter-kn")}});
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){e.multiLanguage=function(){for(var t=Array.prototype.slice.call(arguments),i=t.join("-"),r="",n=[],s=[],p=0;p<t.length;++p)"en"==t[p]?(r+="\\w",n.unshift(e.stopWordFilter),n.push(e.stemmer),s.push(e.stemmer)):(r+=e[t[p]].wordCharacters,e[t[p]].stopWordFilter&&n.unshift(e[t[p]].stopWordFilter),e[t[p]].stemmer&&(n.push(e[t[p]].stemmer),s.push(e[t[p]].stemmer)));var o=e.trimmerSupport.generateTrimmer(r);return e.Pipeline.registerFunction(o,"lunr-multi-trimmer-"+i),n.unshift(o),function(){this.pipeline.reset(),this.pipeline.add.apply(this.pipeline,n),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add.apply(this.searchPipeline,s))}}}});
|
||||
File diff suppressed because one or more lines are too long
@@ -1,18 +0,0 @@
|
||||
/*!
|
||||
* Lunr languages, `Norwegian` language
|
||||
* https://github.com/MihaiValentin/lunr-languages
|
||||
*
|
||||
* Copyright 2014, Mihai Valentin
|
||||
* http://www.mozilla.org/MPL/
|
||||
*/
|
||||
/*!
|
||||
* based on
|
||||
* Snowball JavaScript Library v0.3
|
||||
* http://code.google.com/p/urim/
|
||||
* http://snowball.tartarus.org/
|
||||
*
|
||||
* Copyright 2010, Oleg Mazko
|
||||
* http://www.mozilla.org/MPL/
|
||||
*/
|
||||
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.no=function(){this.pipeline.reset(),this.pipeline.add(e.no.trimmer,e.no.stopWordFilter,e.no.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.no.stemmer))},e.no.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.no.trimmer=e.trimmerSupport.generateTrimmer(e.no.wordCharacters),e.Pipeline.registerFunction(e.no.trimmer,"trimmer-no"),e.no.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,i=new function(){function e(){var e,r=w.cursor+3;if(a=w.limit,0<=r||r<=w.limit){for(s=r;;){if(e=w.cursor,w.in_grouping(d,97,248)){w.cursor=e;break}if(e>=w.limit)return;w.cursor=e+1}for(;!w.out_grouping(d,97,248);){if(w.cursor>=w.limit)return;w.cursor++}a=w.cursor,a<s&&(a=s)}}function i(){var e,r,n;if(w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(m,29),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:n=w.limit-w.cursor,w.in_grouping_b(c,98,122)?w.slice_del():(w.cursor=w.limit-n,w.eq_s_b(1,"k")&&w.out_grouping_b(d,97,248)&&w.slice_del());break;case 3:w.slice_from("er")}}function t(){var e,r=w.limit-w.cursor;w.cursor>=a&&(e=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,w.find_among_b(u,2)?(w.bra=w.cursor,w.limit_backward=e,w.cursor=w.limit-r,w.cursor>w.limit_backward&&(w.cursor--,w.bra=w.cursor,w.slice_del())):w.limit_backward=e)}function o(){var e,r;w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(l,11),e?(w.bra=w.cursor,w.limit_backward=r,1==e&&w.slice_del()):w.limit_backward=r)}var s,a,m=[new r("a",-1,1),new r("e",-1,1),new r("ede",1,1),new r("ande",1,1),new r("ende",1,1),new r("ane",1,1),new r("ene",1,1),new r("hetene",6,1),new r("erte",1,3),new r("en",-1,1),new r("heten",9,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",12,1),new r("s",-1,2),new r("as",14,1),new r("es",14,1),new r("edes",16,1),new r("endes",16,1),new r("enes",16,1),new r("hetenes",19,1),new r("ens",14,1),new r("hetens",21,1),new r("ers",14,1),new r("ets",14,1),new r("et",-1,1),new r("het",25,1),new r("ert",-1,3),new r("ast",-1,1)],u=[new r("dt",-1,-1),new r("vt",-1,-1)],l=[new r("leg",-1,1),new r("eleg",0,1),new r("ig",-1,1),new r("eig",2,1),new r("lig",2,1),new r("elig",4,1),new r("els",-1,1),new r("lov",-1,1),new r("elov",7,1),new r("slov",7,1),new r("hetslov",9,1)],d=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],c=[119,125,149,1],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,i(),w.cursor=w.limit,t(),w.cursor=w.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return i.setCurrent(e),i.stem(),i.getCurrent()}):(i.setCurrent(e),i.stem(),i.getCurrent())}}(),e.Pipeline.registerFunction(e.no.stemmer,"stemmer-no"),e.no.stopWordFilter=e.generateStopWordFilter("alle at av bare begge ble blei bli blir blitt både båe da de deg dei deim deira deires dem den denne der dere deres det dette di din disse ditt du dykk dykkar då eg ein eit eitt eller elles en enn er et ett etter for fordi fra før ha hadde han hans har hennar henne hennes her hjå ho hoe honom hoss hossen hun hva hvem hver hvilke hvilken hvis hvor hvordan hvorfor i ikke ikkje ikkje ingen ingi inkje inn inni ja jeg kan kom korleis korso kun kunne kva kvar kvarhelst kven kvi kvifor man mange me med medan meg meget mellom men mi min mine mitt mot mykje ned no noe noen noka noko nokon nokor nokre nå når og også om opp oss over på samme seg selv si si sia sidan siden sin sine sitt sjøl skal skulle slik so som som somme somt så sånn til um upp ut uten var vart varte ved vere verte vi vil ville vore vors vort vår være være vært å".split(" ")),e.Pipeline.registerFunction(e.no.stopWordFilter,"stopWordFilter-no")}});
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sa=function(){this.pipeline.reset(),this.pipeline.add(e.sa.trimmer,e.sa.stopWordFilter,e.sa.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sa.stemmer))},e.sa.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿ꣠-꣱ꣲ-ꣷ꣸-ꣻ꣼-ꣽꣾ-ꣿᆰ0-ᆰ9",e.sa.trimmer=e.trimmerSupport.generateTrimmer(e.sa.wordCharacters),e.Pipeline.registerFunction(e.sa.trimmer,"trimmer-sa"),e.sa.stopWordFilter=e.generateStopWordFilter('तथा अयम् एकम् इत्यस्मिन् तथा तत् वा अयम् इत्यस्य ते आहूत उपरि तेषाम् किन्तु तेषाम् तदा इत्यनेन अधिकः इत्यस्य तत् केचन बहवः द्वि तथा महत्वपूर्णः अयम् अस्य विषये अयं अस्ति तत् प्रथमः विषये इत्युपरि इत्युपरि इतर अधिकतमः अधिकः अपि सामान्यतया ठ इतरेतर नूतनम् द न्यूनम् कश्चित् वा विशालः द सः अस्ति तदनुसारम् तत्र अस्ति केवलम् अपि अत्र सर्वे विविधाः तत् बहवः यतः इदानीम् द दक्षिण इत्यस्मै तस्य उपरि नथ अतीव कार्यम् सर्वे एकैकम् इत्यादि। एते सन्ति उत इत्थम् मध्ये एतदर्थं . स कस्य प्रथमः श्री. करोति अस्मिन् प्रकारः निर्मिता कालः तत्र कर्तुं समान अधुना ते सन्ति स एकः अस्ति सः अर्थात् तेषां कृते . स्थितम् विशेषः अग्रिम तेषाम् समान स्रोतः ख म समान इदानीमपि अधिकतया करोतु ते समान इत्यस्य वीथी सह यस्मिन् कृतवान् धृतः तदा पुनः पूर्वं सः आगतः किम् कुल इतर पुरा मात्रा स विषये उ अतएव अपि नगरस्य उपरि यतः प्रतिशतं कतरः कालः साधनानि भूत तथापि जात सम्बन्धि अन्यत् ग अतः अस्माकं स्वकीयाः अस्माकं इदानीं अन्तः इत्यादयः भवन्तः इत्यादयः एते एताः तस्य अस्य इदम् एते तेषां तेषां तेषां तान् तेषां तेषां तेषां समानः सः एकः च तादृशाः बहवः अन्ये च वदन्ति यत् कियत् कस्मै कस्मै यस्मै यस्मै यस्मै यस्मै न अतिनीचः किन्तु प्रथमं सम्पूर्णतया ततः चिरकालानन्तरं पुस्तकं सम्पूर्णतया अन्तः किन्तु अत्र वा इह इव श्रद्धाय अवशिष्यते परन्तु अन्ये वर्गाः सन्ति ते सन्ति शक्नुवन्ति सर्वे मिलित्वा सर्वे एकत्र"'.split(" ")),e.sa.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.sa.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var i=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(i).split("|")},e.Pipeline.registerFunction(e.sa.stemmer,"stemmer-sa"),e.Pipeline.registerFunction(e.sa.stopWordFilter,"stopWordFilter-sa")}});
|
||||
@@ -1 +0,0 @@
|
||||
!function(r,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(r.lunr)}(this,function(){return function(r){r.stemmerSupport={Among:function(r,t,i,s){if(this.toCharArray=function(r){for(var t=r.length,i=new Array(t),s=0;s<t;s++)i[s]=r.charCodeAt(s);return i},!r&&""!=r||!t&&0!=t||!i)throw"Bad Among initialisation: s:"+r+", substring_i: "+t+", result: "+i;this.s_size=r.length,this.s=this.toCharArray(r),this.substring_i=t,this.result=i,this.method=s},SnowballProgram:function(){var r;return{bra:0,ket:0,limit:0,cursor:0,limit_backward:0,setCurrent:function(t){r=t,this.cursor=0,this.limit=t.length,this.limit_backward=0,this.bra=this.cursor,this.ket=this.limit},getCurrent:function(){var t=r;return r=null,t},in_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},in_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},out_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e>s||e<i)return this.cursor++,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},out_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e>s||e<i)return this.cursor--,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},eq_s:function(t,i){if(this.limit-this.cursor<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor+s)!=i.charCodeAt(s))return!1;return this.cursor+=t,!0},eq_s_b:function(t,i){if(this.cursor-this.limit_backward<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor-t+s)!=i.charCodeAt(s))return!1;return this.cursor-=t,!0},find_among:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=l;m<_.s_size;m++){if(n+l==u){f=-1;break}if(f=r.charCodeAt(n+l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n+_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n+_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},find_among_b:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit_backward,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=_.s_size-1-l;m>=0;m--){if(n-l==u){f=-1;break}if(f=r.charCodeAt(n-1-l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n-_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n-_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},replace_s:function(t,i,s){var e=s.length-(i-t),n=r.substring(0,t),u=r.substring(i);return r=n+s+u,this.limit+=e,this.cursor>=i?this.cursor+=e:this.cursor>t&&(this.cursor=t),e},slice_check:function(){if(this.bra<0||this.bra>this.ket||this.ket>this.limit||this.limit>r.length)throw"faulty slice operation"},slice_from:function(r){this.slice_check(),this.replace_s(this.bra,this.ket,r)},slice_del:function(){this.slice_from("")},insert:function(r,t,i){var s=this.replace_s(r,t,i);r<=this.bra&&(this.bra+=s),r<=this.ket&&(this.ket+=s)},slice_to:function(){return this.slice_check(),r.substring(this.bra,this.ket)},eq_v_b:function(r){return this.eq_s_b(r.length,r)}}}},r.trimmerSupport={generateTrimmer:function(r){var t=new RegExp("^[^"+r+"]+"),i=new RegExp("[^"+r+"]+$");return function(r){return"function"==typeof r.update?r.update(function(r){return r.replace(t,"").replace(i,"")}):r.replace(t,"").replace(i,"")}}}}});
|
||||
@@ -1,18 +0,0 @@
|
||||
/*!
|
||||
* Lunr languages, `Swedish` language
|
||||
* https://github.com/MihaiValentin/lunr-languages
|
||||
*
|
||||
* Copyright 2014, Mihai Valentin
|
||||
* http://www.mozilla.org/MPL/
|
||||
*/
|
||||
/*!
|
||||
* based on
|
||||
* Snowball JavaScript Library v0.3
|
||||
* http://code.google.com/p/urim/
|
||||
* http://snowball.tartarus.org/
|
||||
*
|
||||
* Copyright 2010, Oleg Mazko
|
||||
* http://www.mozilla.org/MPL/
|
||||
*/
|
||||
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sv=function(){this.pipeline.reset(),this.pipeline.add(e.sv.trimmer,e.sv.stopWordFilter,e.sv.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sv.stemmer))},e.sv.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.sv.trimmer=e.trimmerSupport.generateTrimmer(e.sv.wordCharacters),e.Pipeline.registerFunction(e.sv.trimmer,"trimmer-sv"),e.sv.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,t=new function(){function e(){var e,r=w.cursor+3;if(o=w.limit,0<=r||r<=w.limit){for(a=r;;){if(e=w.cursor,w.in_grouping(l,97,246)){w.cursor=e;break}if(w.cursor=e,w.cursor>=w.limit)return;w.cursor++}for(;!w.out_grouping(l,97,246);){if(w.cursor>=w.limit)return;w.cursor++}o=w.cursor,o<a&&(o=a)}}function t(){var e,r=w.limit_backward;if(w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(u,37),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.in_grouping_b(d,98,121)&&w.slice_del()}}function i(){var e=w.limit_backward;w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.find_among_b(c,7)&&(w.cursor=w.limit,w.ket=w.cursor,w.cursor>w.limit_backward&&(w.bra=--w.cursor,w.slice_del())),w.limit_backward=e)}function s(){var e,r;if(w.cursor>=o){if(r=w.limit_backward,w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(m,5))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.slice_from("lös");break;case 3:w.slice_from("full")}w.limit_backward=r}}var a,o,u=[new r("a",-1,1),new r("arna",0,1),new r("erna",0,1),new r("heterna",2,1),new r("orna",0,1),new r("ad",-1,1),new r("e",-1,1),new r("ade",6,1),new r("ande",6,1),new r("arne",6,1),new r("are",6,1),new r("aste",6,1),new r("en",-1,1),new r("anden",12,1),new r("aren",12,1),new r("heten",12,1),new r("ern",-1,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",18,1),new r("or",-1,1),new r("s",-1,2),new r("as",21,1),new r("arnas",22,1),new r("ernas",22,1),new r("ornas",22,1),new r("es",21,1),new r("ades",26,1),new r("andes",26,1),new r("ens",21,1),new r("arens",29,1),new r("hetens",29,1),new r("erns",21,1),new r("at",-1,1),new r("andet",-1,1),new r("het",-1,1),new r("ast",-1,1)],c=[new r("dd",-1,-1),new r("gd",-1,-1),new r("nn",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1),new r("tt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("els",-1,1),new r("fullt",-1,3),new r("löst",-1,2)],l=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,24,0,32],d=[119,127,149],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,t(),w.cursor=w.limit,i(),w.cursor=w.limit,s(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return t.setCurrent(e),t.stem(),t.getCurrent()}):(t.setCurrent(e),t.stem(),t.getCurrent())}}(),e.Pipeline.registerFunction(e.sv.stemmer,"stemmer-sv"),e.sv.stopWordFilter=e.generateStopWordFilter("alla allt att av blev bli blir blivit de dem den denna deras dess dessa det detta dig din dina ditt du där då efter ej eller en er era ert ett från för ha hade han hans har henne hennes hon honom hur här i icke ingen inom inte jag ju kan kunde man med mellan men mig min mina mitt mot mycket ni nu när någon något några och om oss på samma sedan sig sin sina sitta själv skulle som så sådan sådana sådant till under upp ut utan vad var vara varför varit varje vars vart vem vi vid vilka vilkas vilken vilket vår våra vårt än är åt över".split(" ")),e.Pipeline.registerFunction(e.sv.stopWordFilter,"stopWordFilter-sv")}});
|
||||
@@ -1 +0,0 @@
|
||||
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.ta=function(){this.pipeline.reset(),this.pipeline.add(e.ta.trimmer,e.ta.stopWordFilter,e.ta.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.ta.stemmer))},e.ta.wordCharacters="-உஊ-ஏஐ-ஙச-ட-னப-யர-ஹ-ிீ-ொ-ௐ---௩௪-௯௰-௹௺-a-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.ta.trimmer=e.trimmerSupport.generateTrimmer(e.ta.wordCharacters),e.Pipeline.registerFunction(e.ta.trimmer,"trimmer-ta"),e.ta.stopWordFilter=e.generateStopWordFilter("அங்கு அங்கே அது அதை அந்த அவர் அவர்கள் அவள் அவன் அவை ஆக ஆகவே ஆகையால் ஆதலால் ஆதலினால் ஆனாலும் ஆனால் இங்கு இங்கே இது இதை இந்த இப்படி இவர் இவர்கள் இவள் இவன் இவை இவ்வளவு உனக்கு உனது உன் உன்னால் எங்கு எங்கே எது எதை எந்த எப்படி எவர் எவர்கள் எவள் எவன் எவை எவ்வளவு எனக்கு எனது எனவே என் என்ன என்னால் ஏது ஏன் தனது தன்னால் தானே தான் நாங்கள் நாம் நான் நீ நீங்கள்".split(" ")),e.ta.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.ta.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.ta.stemmer,"stemmer-ta"),e.Pipeline.registerFunction(e.ta.stopWordFilter,"stopWordFilter-ta")}});
|
||||
@@ -1 +0,0 @@
|
||||
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.te=function(){this.pipeline.reset(),this.pipeline.add(e.te.trimmer,e.te.stopWordFilter,e.te.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.te.stemmer))},e.te.wordCharacters="ఀ-ఄఅ-ఔక-హా-ౌౕ-ౖౘ-ౚౠ-ౡౢ-ౣ౦-౯౸-౿఼ఽ్ౝ౷",e.te.trimmer=e.trimmerSupport.generateTrimmer(e.te.wordCharacters),e.Pipeline.registerFunction(e.te.trimmer,"trimmer-te"),e.te.stopWordFilter=e.generateStopWordFilter("అందరూ అందుబాటులో అడగండి అడగడం అడ్డంగా అనుగుణంగా అనుమతించు అనుమతిస్తుంది అయితే ఇప్పటికే ఉన్నారు ఎక్కడైనా ఎప్పుడు ఎవరైనా ఎవరో ఏ ఏదైనా ఏమైనప్పటికి ఒక ఒకరు కనిపిస్తాయి కాదు కూడా గా గురించి చుట్టూ చేయగలిగింది తగిన తర్వాత దాదాపు దూరంగా నిజంగా పై ప్రకారం ప్రక్కన మధ్య మరియు మరొక మళ్ళీ మాత్రమే మెచ్చుకో వద్ద వెంట వేరుగా వ్యతిరేకంగా సంబంధం".split(" ")),e.te.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.te.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.te.stemmer,"stemmer-te"),e.Pipeline.registerFunction(e.te.stopWordFilter,"stopWordFilter-te")}});
|
||||
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.th=function(){this.pipeline.reset(),this.pipeline.add(e.th.trimmer),r?this.tokenizer=e.th.tokenizer:(e.tokenizer&&(e.tokenizer=e.th.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.th.tokenizer))},e.th.wordCharacters="[-]",e.th.trimmer=e.trimmerSupport.generateTrimmer(e.th.wordCharacters),e.Pipeline.registerFunction(e.th.trimmer,"trimmer-th");var t=e.wordcut;t.init(),e.th.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t):t});var n=i.toString().replace(/^\s+/,"");return t.cut(n).split("|")}}});
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.vi=function(){this.pipeline.reset(),this.pipeline.add(e.vi.stopWordFilter,e.vi.trimmer)},e.vi.wordCharacters="[A-Za-ẓ̀͐́͑̉̃̓ÂâÊêÔôĂ-ăĐ-đƠ-ơƯ-ư]",e.vi.trimmer=e.trimmerSupport.generateTrimmer(e.vi.wordCharacters),e.Pipeline.registerFunction(e.vi.trimmer,"trimmer-vi"),e.vi.stopWordFilter=e.generateStopWordFilter("là cái nhưng mà".split(" "))}});
|
||||
@@ -1 +0,0 @@
|
||||
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r(require("@node-rs/jieba")):r()(e.lunr)}(this,function(e){return function(r,t){if(void 0===r)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===r.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var i="2"==r.version[0];r.zh=function(){this.pipeline.reset(),this.pipeline.add(r.zh.trimmer,r.zh.stopWordFilter,r.zh.stemmer),i?this.tokenizer=r.zh.tokenizer:(r.tokenizer&&(r.tokenizer=r.zh.tokenizer),this.tokenizerFn&&(this.tokenizerFn=r.zh.tokenizer))},r.zh.tokenizer=function(n){if(!arguments.length||null==n||void 0==n)return[];if(Array.isArray(n))return n.map(function(e){return i?new r.Token(e.toLowerCase()):e.toLowerCase()});t&&e.load(t);var o=n.toString().trim().toLowerCase(),s=[];e.cut(o,!0).forEach(function(e){s=s.concat(e.split(" "))}),s=s.filter(function(e){return!!e});var u=0;return s.map(function(e,t){if(i){var n=o.indexOf(e,u),s={};return s.position=[n,e.length],s.index=t,u=n,new r.Token(e,s)}return e})},r.zh.wordCharacters="\\w一-龥",r.zh.trimmer=r.trimmerSupport.generateTrimmer(r.zh.wordCharacters),r.Pipeline.registerFunction(r.zh.trimmer,"trimmer-zh"),r.zh.stemmer=function(){return function(e){return e}}(),r.Pipeline.registerFunction(r.zh.stemmer,"stemmer-zh"),r.zh.stopWordFilter=r.generateStopWordFilter("的 一 不 在 人 有 是 为 為 以 于 於 上 他 而 后 後 之 来 來 及 了 因 下 可 到 由 这 這 与 與 也 此 但 并 並 个 個 其 已 无 無 小 我 们 們 起 最 再 今 去 好 只 又 或 很 亦 某 把 那 你 乃 它 吧 被 比 别 趁 当 當 从 從 得 打 凡 儿 兒 尔 爾 该 該 各 给 給 跟 和 何 还 還 即 几 幾 既 看 据 據 距 靠 啦 另 么 麽 每 嘛 拿 哪 您 凭 憑 且 却 卻 让 讓 仍 啥 如 若 使 谁 誰 虽 雖 随 隨 同 所 她 哇 嗡 往 些 向 沿 哟 喲 用 咱 则 則 怎 曾 至 致 着 著 诸 諸 自".split(" ")),r.Pipeline.registerFunction(r.zh.stopWordFilter,"stopWordFilter-zh")}});
|
||||
@@ -1,206 +0,0 @@
|
||||
/**
|
||||
* export the module via AMD, CommonJS or as a browser global
|
||||
* Export code from https://github.com/umdjs/umd/blob/master/returnExports.js
|
||||
*/
|
||||
;(function (root, factory) {
|
||||
if (typeof define === 'function' && define.amd) {
|
||||
// AMD. Register as an anonymous module.
|
||||
define(factory)
|
||||
} else if (typeof exports === 'object') {
|
||||
/**
|
||||
* Node. Does not work with strict CommonJS, but
|
||||
* only CommonJS-like environments that support module.exports,
|
||||
* like Node.
|
||||
*/
|
||||
module.exports = factory()
|
||||
} else {
|
||||
// Browser globals (root is window)
|
||||
factory()(root.lunr);
|
||||
}
|
||||
}(this, function () {
|
||||
/**
|
||||
* Just return a value to define the module export.
|
||||
* This example returns an object, but the module
|
||||
* can return a function as the exported value.
|
||||
*/
|
||||
|
||||
return function(lunr) {
|
||||
// TinySegmenter 0.1 -- Super compact Japanese tokenizer in Javascript
|
||||
// (c) 2008 Taku Kudo <taku@chasen.org>
|
||||
// TinySegmenter is freely distributable under the terms of a new BSD licence.
|
||||
// For details, see http://chasen.org/~taku/software/TinySegmenter/LICENCE.txt
|
||||
|
||||
function TinySegmenter() {
|
||||
var patterns = {
|
||||
"[一二三四五六七八九十百千万億兆]":"M",
|
||||
"[一-龠々〆ヵヶ]":"H",
|
||||
"[ぁ-ん]":"I",
|
||||
"[ァ-ヴーｱ-ﾝﾞｰ]":"K",
|
||||
"[a-zA-Zａ-ｚＡ-Ｚ]":"A",
|
||||
"[0-9０-９]":"N"
|
||||
}
|
||||
this.chartype_ = [];
|
||||
for (var i in patterns) {
|
||||
var regexp = new RegExp(i);
|
||||
this.chartype_.push([regexp, patterns[i]]);
|
||||
}
|
||||
|
||||
this.BIAS__ = -332
|
||||
this.BC1__ = {"HH":6,"II":2461,"KH":406,"OH":-1378};
|
||||
this.BC2__ = {"AA":-3267,"AI":2744,"AN":-878,"HH":-4070,"HM":-1711,"HN":4012,"HO":3761,"IA":1327,"IH":-1184,"II":-1332,"IK":1721,"IO":5492,"KI":3831,"KK":-8741,"MH":-3132,"MK":3334,"OO":-2920};
|
||||
this.BC3__ = {"HH":996,"HI":626,"HK":-721,"HN":-1307,"HO":-836,"IH":-301,"KK":2762,"MK":1079,"MM":4034,"OA":-1652,"OH":266};
|
||||
this.BP1__ = {"BB":295,"OB":304,"OO":-125,"UB":352};
|
||||
this.BP2__ = {"BO":60,"OO":-1762};
|
||||
this.BQ1__ = {"BHH":1150,"BHM":1521,"BII":-1158,"BIM":886,"BMH":1208,"BNH":449,"BOH":-91,"BOO":-2597,"OHI":451,"OIH":-296,"OKA":1851,"OKH":-1020,"OKK":904,"OOO":2965};
|
||||
this.BQ2__ = {"BHH":118,"BHI":-1159,"BHM":466,"BIH":-919,"BKK":-1720,"BKO":864,"OHH":-1139,"OHM":-181,"OIH":153,"UHI":-1146};
|
||||
this.BQ3__ = {"BHH":-792,"BHI":2664,"BII":-299,"BKI":419,"BMH":937,"BMM":8335,"BNN":998,"BOH":775,"OHH":2174,"OHM":439,"OII":280,"OKH":1798,"OKI":-793,"OKO":-2242,"OMH":-2402,"OOO":11699};
|
||||
this.BQ4__ = {"BHH":-3895,"BIH":3761,"BII":-4654,"BIK":1348,"BKK":-1806,"BMI":-3385,"BOO":-12396,"OAH":926,"OHH":266,"OHK":-2036,"ONN":-973};
|
||||
this.BW1__ = {",と":660,",同":727,"B1あ":1404,"B1同":542,"、と":660,"、同":727,"」と":1682,"あっ":1505,"いう":1743,"いっ":-2055,"いる":672,"うし":-4817,"うん":665,"から":3472,"がら":600,"こう":-790,"こと":2083,"こん":-1262,"さら":-4143,"さん":4573,"した":2641,"して":1104,"すで":-3399,"そこ":1977,"それ":-871,"たち":1122,"ため":601,"った":3463,"つい":-802,"てい":805,"てき":1249,"でき":1127,"です":3445,"では":844,"とい":-4915,"とみ":1922,"どこ":3887,"ない":5713,"なっ":3015,"など":7379,"なん":-1113,"にし":2468,"には":1498,"にも":1671,"に対":-912,"の一":-501,"の中":741,"ませ":2448,"まで":1711,"まま":2600,"まる":-2155,"やむ":-1947,"よっ":-2565,"れた":2369,"れで":-913,"をし":1860,"を見":731,"亡く":-1886,"京都":2558,"取り":-2784,"大き":-2604,"大阪":1497,"平方":-2314,"引き":-1336,"日本":-195,"本当":-2423,"毎日":-2113,"目指":-724,"Ｂ１あ":1404,"Ｂ１同":542,"｣と":1682};
|
||||
this.BW2__ = {"..":-11822,"11":-669,"――":-5730,"−−":-13175,"いう":-1609,"うか":2490,"かし":-1350,"かも":-602,"から":-7194,"かれ":4612,"がい":853,"がら":-3198,"きた":1941,"くな":-1597,"こと":-8392,"この":-4193,"させ":4533,"され":13168,"さん":-3977,"しい":-1819,"しか":-545,"した":5078,"して":972,"しな":939,"その":-3744,"たい":-1253,"たた":-662,"ただ":-3857,"たち":-786,"たと":1224,"たは":-939,"った":4589,"って":1647,"っと":-2094,"てい":6144,"てき":3640,"てく":2551,"ては":-3110,"ても":-3065,"でい":2666,"でき":-1528,"でし":-3828,"です":-4761,"でも":-4203,"とい":1890,"とこ":-1746,"とと":-2279,"との":720,"とみ":5168,"とも":-3941,"ない":-2488,"なが":-1313,"など":-6509,"なの":2614,"なん":3099,"にお":-1615,"にし":2748,"にな":2454,"によ":-7236,"に対":-14943,"に従":-4688,"に関":-11388,"のか":2093,"ので":-7059,"のに":-6041,"のの":-6125,"はい":1073,"はが":-1033,"はず":-2532,"ばれ":1813,"まし":-1316,"まで":-6621,"まれ":5409,"めて":-3153,"もい":2230,"もの":-10713,"らか":-944,"らし":-1611,"らに":-1897,"りし":651,"りま":1620,"れた":4270,"れて":849,"れば":4114,"ろう":6067,"われ":7901,"を通":-11877,"んだ":728,"んな":-4115,"一人":602,"一方":-1375,"一日":970,"一部":-1051,"上が":-4479,"会社":-1116,"出て":2163,"分の":-7758,"同党":970,"同日":-913,"大阪":-2471,"委員":-1250,"少な":-1050,"年度":-8669,"年間":-1626,"府県":-2363,"手権":-1982,"新聞":-4066,"日新":-722,"日本":-7068,"日米":3372,"曜日":-601,"朝鮮":-2355,"本人":-2697,"東京":-1543,"然と":-1384,"社会":-1276,"立て":-990,"第に":-1612,"米国":-4268,"１１":-669};
|
||||
this.BW3__ = {"あた":-2194,"あり":719,"ある":3846,"い.":-1185,"い。":-1185,"いい":5308,"いえ":2079,"いく":3029,"いた":2056,"いっ":1883,"いる":5600,"いわ":1527,"うち":1117,"うと":4798,"えと":1454,"か.":2857,"か。":2857,"かけ":-743,"かっ":-4098,"かに":-669,"から":6520,"かり":-2670,"が,":1816,"が、":1816,"がき":-4855,"がけ":-1127,"がっ":-913,"がら":-4977,"がり":-2064,"きた":1645,"けど":1374,"こと":7397,"この":1542,"ころ":-2757,"さい":-714,"さを":976,"し,":1557,"し、":1557,"しい":-3714,"した":3562,"して":1449,"しな":2608,"しま":1200,"す.":-1310,"す。":-1310,"する":6521,"ず,":3426,"ず、":3426,"ずに":841,"そう":428,"た.":8875,"た。":8875,"たい":-594,"たの":812,"たり":-1183,"たる":-853,"だ.":4098,"だ。":4098,"だっ":1004,"った":-4748,"って":300,"てい":6240,"てお":855,"ても":302,"です":1437,"でに":-1482,"では":2295,"とう":-1387,"とし":2266,"との":541,"とも":-3543,"どう":4664,"ない":1796,"なく":-903,"など":2135,"に,":-1021,"に、":-1021,"にし":1771,"にな":1906,"には":2644,"の,":-724,"の、":-724,"の子":-1000,"は,":1337,"は、":1337,"べき":2181,"まし":1113,"ます":6943,"まっ":-1549,"まで":6154,"まれ":-793,"らし":1479,"られ":6820,"るる":3818,"れ,":854,"れ、":854,"れた":1850,"れて":1375,"れば":-3246,"れる":1091,"われ":-605,"んだ":606,"んで":798,"カ月":990,"会議":860,"入り":1232,"大会":2217,"始め":1681,"市":965,"新聞":-5055,"日,":974,"日、":974,"社会":2024,"ｶ月":990};
|
||||
this.TC1__ = {"AAA":1093,"HHH":1029,"HHM":580,"HII":998,"HOH":-390,"HOM":-331,"IHI":1169,"IOH":-142,"IOI":-1015,"IOM":467,"MMH":187,"OOI":-1832};
|
||||
this.TC2__ = {"HHO":2088,"HII":-1023,"HMM":-1154,"IHI":-1965,"KKH":703,"OII":-2649};
|
||||
this.TC3__ = {"AAA":-294,"HHH":346,"HHI":-341,"HII":-1088,"HIK":731,"HOH":-1486,"IHH":128,"IHI":-3041,"IHO":-1935,"IIH":-825,"IIM":-1035,"IOI":-542,"KHH":-1216,"KKA":491,"KKH":-1217,"KOK":-1009,"MHH":-2694,"MHM":-457,"MHO":123,"MMH":-471,"NNH":-1689,"NNO":662,"OHO":-3393};
|
||||
this.TC4__ = {"HHH":-203,"HHI":1344,"HHK":365,"HHM":-122,"HHN":182,"HHO":669,"HIH":804,"HII":679,"HOH":446,"IHH":695,"IHO":-2324,"IIH":321,"III":1497,"IIO":656,"IOO":54,"KAK":4845,"KKA":3386,"KKK":3065,"MHH":-405,"MHI":201,"MMH":-241,"MMM":661,"MOM":841};
|
||||
this.TQ1__ = {"BHHH":-227,"BHHI":316,"BHIH":-132,"BIHH":60,"BIII":1595,"BNHH":-744,"BOHH":225,"BOOO":-908,"OAKK":482,"OHHH":281,"OHIH":249,"OIHI":200,"OIIH":-68};
|
||||
this.TQ2__ = {"BIHH":-1401,"BIII":-1033,"BKAK":-543,"BOOO":-5591};
|
||||
this.TQ3__ = {"BHHH":478,"BHHM":-1073,"BHIH":222,"BHII":-504,"BIIH":-116,"BIII":-105,"BMHI":-863,"BMHM":-464,"BOMH":620,"OHHH":346,"OHHI":1729,"OHII":997,"OHMH":481,"OIHH":623,"OIIH":1344,"OKAK":2792,"OKHH":587,"OKKA":679,"OOHH":110,"OOII":-685};
|
||||
this.TQ4__ = {"BHHH":-721,"BHHM":-3604,"BHII":-966,"BIIH":-607,"BIII":-2181,"OAAA":-2763,"OAKK":180,"OHHH":-294,"OHHI":2446,"OHHO":480,"OHIH":-1573,"OIHH":1935,"OIHI":-493,"OIIH":626,"OIII":-4007,"OKAK":-8156};
|
||||
this.TW1__ = {"につい":-4681,"東京都":2026};
|
||||
this.TW2__ = {"ある程":-2049,"いった":-1256,"ころが":-2434,"しょう":3873,"その後":-4430,"だって":-1049,"ていた":1833,"として":-4657,"ともに":-4517,"もので":1882,"一気に":-792,"初めて":-1512,"同時に":-8097,"大きな":-1255,"対して":-2721,"社会党":-3216};
|
||||
this.TW3__ = {"いただ":-1734,"してい":1314,"として":-4314,"につい":-5483,"にとっ":-5989,"に当た":-6247,"ので,":-727,"ので、":-727,"のもの":-600,"れから":-3752,"十二月":-2287};
|
||||
this.TW4__ = {"いう.":8576,"いう。":8576,"からな":-2348,"してい":2958,"たが,":1516,"たが、":1516,"ている":1538,"という":1349,"ました":5543,"ません":1097,"ようと":-4258,"よると":5865};
|
||||
this.UC1__ = {"A":484,"K":93,"M":645,"O":-505};
|
||||
this.UC2__ = {"A":819,"H":1059,"I":409,"M":3987,"N":5775,"O":646};
|
||||
this.UC3__ = {"A":-1370,"I":2311};
|
||||
this.UC4__ = {"A":-2643,"H":1809,"I":-1032,"K":-3450,"M":3565,"N":3876,"O":6646};
|
||||
this.UC5__ = {"H":313,"I":-1238,"K":-799,"M":539,"O":-831};
|
||||
this.UC6__ = {"H":-506,"I":-253,"K":87,"M":247,"O":-387};
|
||||
this.UP1__ = {"O":-214};
|
||||
this.UP2__ = {"B":69,"O":935};
|
||||
this.UP3__ = {"B":189};
|
||||
this.UQ1__ = {"BH":21,"BI":-12,"BK":-99,"BN":142,"BO":-56,"OH":-95,"OI":477,"OK":410,"OO":-2422};
|
||||
this.UQ2__ = {"BH":216,"BI":113,"OK":1759};
|
||||
this.UQ3__ = {"BA":-479,"BH":42,"BI":1913,"BK":-7198,"BM":3160,"BN":6427,"BO":14761,"OI":-827,"ON":-3212};
|
||||
this.UW1__ = {",":156,"、":156,"「":-463,"あ":-941,"う":-127,"が":-553,"き":121,"こ":505,"で":-201,"と":-547,"ど":-123,"に":-789,"の":-185,"は":-847,"も":-466,"や":-470,"よ":182,"ら":-292,"り":208,"れ":169,"を":-446,"ん":-137,"・":-135,"主":-402,"京":-268,"区":-912,"午":871,"国":-460,"大":561,"委":729,"市":-411,"日":-141,"理":361,"生":-408,"県":-386,"都":-718,"｢":-463,"･":-135};
|
||||
this.UW2__ = {",":-829,"、":-829,"〇":892,"「":-645,"」":3145,"あ":-538,"い":505,"う":134,"お":-502,"か":1454,"が":-856,"く":-412,"こ":1141,"さ":878,"ざ":540,"し":1529,"す":-675,"せ":300,"そ":-1011,"た":188,"だ":1837,"つ":-949,"て":-291,"で":-268,"と":-981,"ど":1273,"な":1063,"に":-1764,"の":130,"は":-409,"ひ":-1273,"べ":1261,"ま":600,"も":-1263,"や":-402,"よ":1639,"り":-579,"る":-694,"れ":571,"を":-2516,"ん":2095,"ア":-587,"カ":306,"キ":568,"ッ":831,"三":-758,"不":-2150,"世":-302,"中":-968,"主":-861,"事":492,"人":-123,"会":978,"保":362,"入":548,"初":-3025,"副":-1566,"北":-3414,"区":-422,"大":-1769,"天":-865,"太":-483,"子":-1519,"学":760,"実":1023,"小":-2009,"市":-813,"年":-1060,"強":1067,"手":-1519,"揺":-1033,"政":1522,"文":-1355,"新":-1682,"日":-1815,"明":-1462,"最":-630,"朝":-1843,"本":-1650,"東":-931,"果":-665,"次":-2378,"民":-180,"気":-1740,"理":752,"発":529,"目":-1584,"相":-242,"県":-1165,"立":-763,"第":810,"米":509,"自":-1353,"行":838,"西":-744,"見":-3874,"調":1010,"議":1198,"込":3041,"開":1758,"間":-1257,"｢":-645,"｣":3145,"ｯ":831,"ｱ":-587,"ｶ":306,"ｷ":568};
|
||||
this.UW3__ = {",":4889,"1":-800,"−":-1723,"、":4889,"々":-2311,"〇":5827,"」":2670,"〓":-3573,"あ":-2696,"い":1006,"う":2342,"え":1983,"お":-4864,"か":-1163,"が":3271,"く":1004,"け":388,"げ":401,"こ":-3552,"ご":-3116,"さ":-1058,"し":-395,"す":584,"せ":3685,"そ":-5228,"た":842,"ち":-521,"っ":-1444,"つ":-1081,"て":6167,"で":2318,"と":1691,"ど":-899,"な":-2788,"に":2745,"の":4056,"は":4555,"ひ":-2171,"ふ":-1798,"へ":1199,"ほ":-5516,"ま":-4384,"み":-120,"め":1205,"も":2323,"や":-788,"よ":-202,"ら":727,"り":649,"る":5905,"れ":2773,"わ":-1207,"を":6620,"ん":-518,"ア":551,"グ":1319,"ス":874,"ッ":-1350,"ト":521,"ム":1109,"ル":1591,"ロ":2201,"ン":278,"・":-3794,"一":-1619,"下":-1759,"世":-2087,"両":3815,"中":653,"主":-758,"予":-1193,"二":974,"人":2742,"今":792,"他":1889,"以":-1368,"低":811,"何":4265,"作":-361,"保":-2439,"元":4858,"党":3593,"全":1574,"公":-3030,"六":755,"共":-1880,"円":5807,"再":3095,"分":457,"初":2475,"別":1129,"前":2286,"副":4437,"力":365,"動":-949,"務":-1872,"化":1327,"北":-1038,"区":4646,"千":-2309,"午":-783,"協":-1006,"口":483,"右":1233,"各":3588,"合":-241,"同":3906,"和":-837,"員":4513,"国":642,"型":1389,"場":1219,"外":-241,"妻":2016,"学":-1356,"安":-423,"実":-1008,"家":1078,"小":-513,"少":-3102,"州":1155,"市":3197,"平":-1804,"年":2416,"広":-1030,"府":1605,"度":1452,"建":-2352,"当":-3885,"得":1905,"思":-1291,"性":1822,"戸":-488,"指":-3973,"政":-2013,"教":-1479,"数":3222,"文":-1489,"新":1764,"日":2099,"旧":5792,"昨":-661,"時":-1248,"曜":-951,"最":-937,"月":4125,"期":360,"李":3094,"村":364,"東":-805,"核":5156,"森":2438,"業":484,"氏":2613,"民":-1694,"決":-1073,"法":1868,"海":-495,"無":979,"物":461,"特":-3850,"生":-273,"用":914,"町":1215,"的":7313,"直":-1835,"省":792,"県":6293,"知":-1528,"私":4231,"税":401,"立":-960,"第":1201,"米":7767,"系":3066,"約":3663,"級":1384,"統":-4229,"総":1163,"線":1255,"者":6457,"能":725,"自":-2869,"英":785,"見":1044,"調":-562,"財":-733,"費":1777,"車":1835,"軍":1375,"込":-1504,"通":-1136,"選":-681,"郎":1026,"郡":4404,"部":1200,"金":2163,"長":421,"開":-1432,"間":1302,"関":-1282,"雨":2009,"電":-1045,"非":2066,"駅":1620,"１":-800,"｣":2670,"･":-3794,"ｯ":-1350,"ｱ":551,"ｸﾞ":1319,"ｽ":874,"ﾄ":521,"ﾑ":1109,"ﾙ":1591,"ﾛ":2201,"ﾝ":278};
|
||||
this.UW4__ = {",":3930,".":3508,"―":-4841,"、":3930,"。":3508,"〇":4999,"「":1895,"」":3798,"〓":-5156,"あ":4752,"い":-3435,"う":-640,"え":-2514,"お":2405,"か":530,"が":6006,"き":-4482,"ぎ":-3821,"く":-3788,"け":-4376,"げ":-4734,"こ":2255,"ご":1979,"さ":2864,"し":-843,"じ":-2506,"す":-731,"ず":1251,"せ":181,"そ":4091,"た":5034,"だ":5408,"ち":-3654,"っ":-5882,"つ":-1659,"て":3994,"で":7410,"と":4547,"な":5433,"に":6499,"ぬ":1853,"ね":1413,"の":7396,"は":8578,"ば":1940,"ひ":4249,"び":-4134,"ふ":1345,"へ":6665,"べ":-744,"ほ":1464,"ま":1051,"み":-2082,"む":-882,"め":-5046,"も":4169,"ゃ":-2666,"や":2795,"ょ":-1544,"よ":3351,"ら":-2922,"り":-9726,"る":-14896,"れ":-2613,"ろ":-4570,"わ":-1783,"を":13150,"ん":-2352,"カ":2145,"コ":1789,"セ":1287,"ッ":-724,"ト":-403,"メ":-1635,"ラ":-881,"リ":-541,"ル":-856,"ン":-3637,"・":-4371,"ー":-11870,"一":-2069,"中":2210,"予":782,"事":-190,"井":-1768,"人":1036,"以":544,"会":950,"体":-1286,"作":530,"側":4292,"先":601,"党":-2006,"共":-1212,"内":584,"円":788,"初":1347,"前":1623,"副":3879,"力":-302,"動":-740,"務":-2715,"化":776,"区":4517,"協":1013,"参":1555,"合":-1834,"和":-681,"員":-910,"器":-851,"回":1500,"国":-619,"園":-1200,"地":866,"場":-1410,"塁":-2094,"士":-1413,"多":1067,"大":571,"子":-4802,"学":-1397,"定":-1057,"寺":-809,"小":1910,"屋":-1328,"山":-1500,"島":-2056,"川":-2667,"市":2771,"年":374,"庁":-4556,"後":456,"性":553,"感":916,"所":-1566,"支":856,"改":787,"政":2182,"教":704,"文":522,"方":-856,"日":1798,"時":1829,"最":845,"月":-9066,"木":-485,"来":-442,"校":-360,"業":-1043,"氏":5388,"民":-2716,"気":-910,"沢":-939,"済":-543,"物":-735,"率":672,"球":-1267,"生":-1286,"産":-1101,"田":-2900,"町":1826,"的":2586,"目":922,"省":-3485,"県":2997,"空":-867,"立":-2112,"第":788,"米":2937,"系":786,"約":2171,"経":1146,"統":-1169,"総":940,"線":-994,"署":749,"者":2145,"能":-730,"般":-852,"行":-792,"規":792,"警":-1184,"議":-244,"谷":-1000,"賞":730,"車":-1481,"軍":1158,"輪":-1433,"込":-3370,"近":929,"道":-1291,"選":2596,"郎":-4866,"都":1192,"野":-1100,"銀":-2213,"長":357,"間":-2344,"院":-2297,"際":-2604,"電":-878,"領":-1659,"題":-792,"館":-1984,"首":1749,"高":2120,"｢":1895,"｣":3798,"･":-4371,"ｯ":-724,"ｰ":-11870,"ｶ":2145,"ｺ":1789,"ｾ":1287,"ﾄ":-403,"ﾒ":-1635,"ﾗ":-881,"ﾘ":-541,"ﾙ":-856,"ﾝ":-3637};
|
||||
this.UW5__ = {",":465,".":-299,"1":-514,"E2":-32768,"]":-2762,"、":465,"。":-299,"「":363,"あ":1655,"い":331,"う":-503,"え":1199,"お":527,"か":647,"が":-421,"き":1624,"ぎ":1971,"く":312,"げ":-983,"さ":-1537,"し":-1371,"す":-852,"だ":-1186,"ち":1093,"っ":52,"つ":921,"て":-18,"で":-850,"と":-127,"ど":1682,"な":-787,"に":-1224,"の":-635,"は":-578,"べ":1001,"み":502,"め":865,"ゃ":3350,"ょ":854,"り":-208,"る":429,"れ":504,"わ":419,"を":-1264,"ん":327,"イ":241,"ル":451,"ン":-343,"中":-871,"京":722,"会":-1153,"党":-654,"務":3519,"区":-901,"告":848,"員":2104,"大":-1296,"学":-548,"定":1785,"嵐":-1304,"市":-2991,"席":921,"年":1763,"思":872,"所":-814,"挙":1618,"新":-1682,"日":218,"月":-4353,"査":932,"格":1356,"機":-1508,"氏":-1347,"田":240,"町":-3912,"的":-3149,"相":1319,"省":-1052,"県":-4003,"研":-997,"社":-278,"空":-813,"統":1955,"者":-2233,"表":663,"語":-1073,"議":1219,"選":-1018,"郎":-368,"長":786,"間":1191,"題":2368,"館":-689,"１":-514,"Ｅ２":-32768,"｢":363,"ｲ":241,"ﾙ":451,"ﾝ":-343};
|
||||
this.UW6__ = {",":227,".":808,"1":-270,"E1":306,"、":227,"。":808,"あ":-307,"う":189,"か":241,"が":-73,"く":-121,"こ":-200,"じ":1782,"す":383,"た":-428,"っ":573,"て":-1014,"で":101,"と":-105,"な":-253,"に":-149,"の":-417,"は":-236,"も":-206,"り":187,"る":-135,"を":195,"ル":-673,"ン":-496,"一":-277,"中":201,"件":-800,"会":624,"前":302,"区":1792,"員":-1212,"委":798,"学":-960,"市":887,"広":-695,"後":535,"業":-697,"相":753,"社":-507,"福":974,"空":-822,"者":1811,"連":463,"郎":1082,"１":-270,"Ｅ１":306,"ﾙ":-673,"ﾝ":-496};
|
||||
|
||||
return this;
|
||||
}
|
||||
TinySegmenter.prototype.ctype_ = function(str) {
|
||||
for (var i in this.chartype_) {
|
||||
if (str.match(this.chartype_[i][0])) {
|
||||
return this.chartype_[i][1];
|
||||
}
|
||||
}
|
||||
return "O";
|
||||
}
|
||||
|
||||
TinySegmenter.prototype.ts_ = function(v) {
|
||||
if (v) { return v; }
|
||||
return 0;
|
||||
}
|
||||
|
||||
TinySegmenter.prototype.segment = function(input) {
|
||||
if (input == null || input == undefined || input == "") {
|
||||
return [];
|
||||
}
|
||||
var result = [];
|
||||
var seg = ["B3","B2","B1"];
|
||||
var ctype = ["O","O","O"];
|
||||
var o = input.split("");
|
||||
for (i = 0; i < o.length; ++i) {
|
||||
seg.push(o[i]);
|
||||
ctype.push(this.ctype_(o[i]))
|
||||
}
|
||||
seg.push("E1");
|
||||
seg.push("E2");
|
||||
seg.push("E3");
|
||||
ctype.push("O");
|
||||
ctype.push("O");
|
||||
ctype.push("O");
|
||||
var word = seg[3];
|
||||
var p1 = "U";
|
||||
var p2 = "U";
|
||||
var p3 = "U";
|
||||
for (var i = 4; i < seg.length - 3; ++i) {
|
||||
var score = this.BIAS__;
|
||||
var w1 = seg[i-3];
|
||||
var w2 = seg[i-2];
|
||||
var w3 = seg[i-1];
|
||||
var w4 = seg[i];
|
||||
var w5 = seg[i+1];
|
||||
var w6 = seg[i+2];
|
||||
var c1 = ctype[i-3];
|
||||
var c2 = ctype[i-2];
|
||||
var c3 = ctype[i-1];
|
||||
var c4 = ctype[i];
|
||||
var c5 = ctype[i+1];
|
||||
var c6 = ctype[i+2];
|
||||
score += this.ts_(this.UP1__[p1]);
|
||||
score += this.ts_(this.UP2__[p2]);
|
||||
score += this.ts_(this.UP3__[p3]);
|
||||
score += this.ts_(this.BP1__[p1 + p2]);
|
||||
score += this.ts_(this.BP2__[p2 + p3]);
|
||||
score += this.ts_(this.UW1__[w1]);
|
||||
score += this.ts_(this.UW2__[w2]);
|
||||
score += this.ts_(this.UW3__[w3]);
|
||||
score += this.ts_(this.UW4__[w4]);
|
||||
score += this.ts_(this.UW5__[w5]);
|
||||
score += this.ts_(this.UW6__[w6]);
|
||||
score += this.ts_(this.BW1__[w2 + w3]);
|
||||
score += this.ts_(this.BW2__[w3 + w4]);
|
||||
score += this.ts_(this.BW3__[w4 + w5]);
|
||||
score += this.ts_(this.TW1__[w1 + w2 + w3]);
|
||||
score += this.ts_(this.TW2__[w2 + w3 + w4]);
|
||||
score += this.ts_(this.TW3__[w3 + w4 + w5]);
|
||||
score += this.ts_(this.TW4__[w4 + w5 + w6]);
|
||||
score += this.ts_(this.UC1__[c1]);
|
||||
score += this.ts_(this.UC2__[c2]);
|
||||
score += this.ts_(this.UC3__[c3]);
|
||||
score += this.ts_(this.UC4__[c4]);
|
||||
score += this.ts_(this.UC5__[c5]);
|
||||
score += this.ts_(this.UC6__[c6]);
|
||||
score += this.ts_(this.BC1__[c2 + c3]);
|
||||
score += this.ts_(this.BC2__[c3 + c4]);
|
||||
score += this.ts_(this.BC3__[c4 + c5]);
|
||||
score += this.ts_(this.TC1__[c1 + c2 + c3]);
|
||||
score += this.ts_(this.TC2__[c2 + c3 + c4]);
|
||||
score += this.ts_(this.TC3__[c3 + c4 + c5]);
|
||||
score += this.ts_(this.TC4__[c4 + c5 + c6]);
|
||||
// score += this.ts_(this.TC5__[c4 + c5 + c6]);
|
||||
score += this.ts_(this.UQ1__[p1 + c1]);
|
||||
score += this.ts_(this.UQ2__[p2 + c2]);
|
||||
score += this.ts_(this.UQ3__[p3 + c3]);
|
||||
score += this.ts_(this.BQ1__[p2 + c2 + c3]);
|
||||
score += this.ts_(this.BQ2__[p2 + c3 + c4]);
|
||||
score += this.ts_(this.BQ3__[p3 + c2 + c3]);
|
||||
score += this.ts_(this.BQ4__[p3 + c3 + c4]);
|
||||
score += this.ts_(this.TQ1__[p2 + c1 + c2 + c3]);
|
||||
score += this.ts_(this.TQ2__[p2 + c2 + c3 + c4]);
|
||||
score += this.ts_(this.TQ3__[p3 + c1 + c2 + c3]);
|
||||
score += this.ts_(this.TQ4__[p3 + c2 + c3 + c4]);
|
||||
var p = "O";
|
||||
if (score > 0) {
|
||||
result.push(word);
|
||||
word = "";
|
||||
p = "B";
|
||||
}
|
||||
p1 = p2;
|
||||
p2 = p3;
|
||||
p3 = p;
|
||||
word += seg[i];
|
||||
}
|
||||
result.push(word);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
lunr.TinySegmenter = TinySegmenter;
|
||||
};
|
||||
|
||||
}));
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1 +0,0 @@
|
||||
{"version":3,"sources":["src/templates/assets/stylesheets/palette/_scheme.scss","../../../../src/templates/assets/stylesheets/palette.scss","src/templates/assets/stylesheets/palette/_accent.scss","src/templates/assets/stylesheets/palette/_primary.scss","src/templates/assets/stylesheets/utilities/_break.scss"],"names":[],"mappings":"AA2BA,cAGE,6BAME,sDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CACA,mDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CAGA,mDAAA,CACA,gDAAA,CACA,yDAAA,CACA,4DAAA,CAGA,0BAAA,CACA,mCAAA,CAGA,iCAAA,CACA,kCAAA,CACA,mCAAA,CACA,mCAAA,CACA,kCAAA,CACA,iCAAA,CACA,+CAAA,CACA,6DAAA,CACA,gEAAA,CACA,4DAAA,CACA,4DAAA,CACA,6DAAA,CAGA,6CAAA,CAGA,+CAAA,CAGA,uDAAA,CACA,6DAAA,CACA,2DAAA,CAGA,iCAAA,CAGA,yDAAA,CACA,iEAAA,CAGA,mDAAA,CACA,mDAAA,CAGA,qDAAA,CACA,uDAAA,CAGA,8DAAA,CAKA,8DAAA,CAKA,0DAAA,CAzEA,iBCiBF,CD6DE,kHAEE,YC3DJ,CDkFE,yDACE,4BChFJ,CD+EE,2DACE,4BC7EJ,CD4EE,gEACE,4BC1EJ,CDyEE,2DACE,4BCvEJ,CDsEE,yDACE,4BCpEJ,CDmEE,0DACE,4BCjEJ,CDgEE,gEACE,4BC9DJ,CD6DE,0DACE,4BC3DJ,CD0DE,2OACE,4BC/CJ,CDsDA,+FAGE,iCCpDF,CACF,CCjDE,2BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD6CN,CCvDE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDoDN,CC9DE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD2DN,CCrEE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDkEN,CC5EE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDyEN,CCnFE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDgFN,CC1FE,kCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDuFN,CCjGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD8FN,CCxGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDqGN,CC/GE,6BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD4GN,CCtHE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDmHN,CC7HE,4BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD6HN,CCpIE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDoIN,CC3IE,6BACE,yBAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD2IN,CClJE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDkJN,CCzJE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDsJN,CE3JE,4BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwJN,CEnKE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgKN,CE3KE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwKN,CEnLE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgLN,CE3LE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwLN,CEnME,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgMN,CE3ME,mCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwMN,CEnNE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgNN,CE3NE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwNN,CEnOE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgON,CE3OE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwON,CEnPE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmPN,CE3PE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2PN,CEnQE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmQN,CE3QE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2QN,CEnRE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgRN,CE3RE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwRN,CEnSE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BF4RN,CE5SE,kCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BFqSN,CEtRE,sEACE,4BFyRJ,CE1RE,+DACE,4BF6RJ,CE9RE,iEACE,4BFiSJ,CElSE,gEACE,4BFqSJ,CEtSE,iEACE,4BFySJ,CEhSA,8BACE,mDAAA,CACA,4DAAA,CACA,0DAAA,CACA,oDAAA,CACA,2DAAA,CAGA,4BFiSF,CE9RE,yCACE,+BFgSJ,CE7RI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCFiSN,CG7MI,mCD1EA,+CACE,8CF0RJ,CEvRI,qDACE,8CFyRN,CEpRE,iEACE,mCFsRJ,CACF,CGxNI,sCDvDA,uCACE,oCFkRJ,CACF,CEzQA,8BACE,kDAAA,CACA,4DAAA,CACA,wDAAA,CACA,oDAAA,CACA,6DAAA,CAGA,4BF0QF,CEvQE,yCACE,+BFyQJ,CEtQI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCF0QN,CEnQE,yCACE,6CFqQJ,CG9NI,0CDhCA,8CACE,gDFiQJ,CACF,CGnOI,0CDvBA,iFACE,6CF6PJ,CACF,CG3PI,sCDKA,uCACE,6CFyPJ,CACF","file":"palette.css"}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
-1625
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,261 +0,0 @@
|
||||
%% This BibTeX bibliography file was created using BibDesk.
|
||||
%% https://bibdesk.sourceforge.io/
|
||||
|
||||
%% Created for Eric Coissac at 2026-04-18 08:19:36 +0200
|
||||
|
||||
|
||||
%% Saved with string encoding Unicode (UTF-8)
|
||||
|
||||
|
||||
|
||||
@article{Zheng2020-ji,
|
||||
abstract = {MOTIVATION: Minimizers are methods to sample k-mers from a
|
||||
string, with the guarantee that similar set of k-mers will be
|
||||
chosen on similar strings. It is parameterized by the k-mer
|
||||
length k, a window length w and an order on the k-mers.
|
||||
Minimizers are used in a large number of softwares and pipelines
|
||||
to improve computation efficiency and decrease memory usage.
|
||||
Despite the method's popularity, many theoretical questions
|
||||
regarding its performance remain open. The core metric for
|
||||
measuring performance of a minimizer is the density, which
|
||||
measures the sparsity of sampled k-mers. The theoretical optimal
|
||||
density for a minimizer is 1/w, provably not achievable in
|
||||
general. For given k and w, little is known about asymptotically
|
||||
optimal minimizers, that is minimizers with density O(1/w).
|
||||
RESULTS: We derive a necessary and sufficient condition for
|
||||
existence of asymptotically optimal minimizers. We also provide a
|
||||
randomized algorithm, called the Miniception, to design
|
||||
minimizers with the best theoretical guarantee to date on density
|
||||
in practical scenarios. Constructing and using the Miniception is
|
||||
as easy as constructing and using a random minimizer, which
|
||||
allows the design of efficient minimizers that scale to the
|
||||
values of k and w used in current bioinformatics software
|
||||
programs. AVAILABILITY AND IMPLEMENTATION: Reference
|
||||
implementation of the Miniception and the codes for analysis can
|
||||
be found at https://github.com/kingsford-group/miniception.
|
||||
SUPPLEMENTARY INFORMATION: Supplementary data are available at
|
||||
Bioinformatics online.},
|
||||
author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
|
||||
doi = {10.1093/bioinformatics/btaa472},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = jul,
|
||||
number = {Suppl_1},
|
||||
pages = {i119--i127},
|
||||
pmc = {PMC8248892},
|
||||
pmid = 32657376,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {Improved design and analysis of practical minimizers},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btaa472},
|
||||
volume = 36,
|
||||
year = 2020,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btaa472}}
|
||||
|
||||
@article{Zheng2021-cc,
|
||||
abstract = {MOTIVATION: Minimizers are efficient methods to sample k-mers
|
||||
from genomic sequences that unconditionally preserve sufficiently
|
||||
long matches between sequences. Well-established methods to
|
||||
construct efficient minimizers focus on sampling fewer k-mers on
|
||||
a random sequence and use universal hitting sets (sets of k-mers
|
||||
that appear frequently enough) to upper bound the sketch size. In
|
||||
contrast, the problem of sequence-specific minimizers, which is
|
||||
to construct efficient minimizers to sample fewer k-mers on a
|
||||
specific sequence such as the reference genome, is less studied.
|
||||
Currently, the theoretical understanding of this problem is
|
||||
lacking, and existing methods do not specialize well to sketch
|
||||
specific sequences. RESULTS: We propose the concept of polar
|
||||
sets, complementary to the existing idea of universal hitting
|
||||
sets. Polar sets are k-mer sets that are spread out enough on the
|
||||
reference, and provably specialize well to specific sequences.
|
||||
Link energy measures how well spread out a polar set is, and with
|
||||
it, the sketch size can be bounded from above and below in a
|
||||
theoretically sound way. This allows for direct optimization of
|
||||
sketch size. We propose efficient heuristics to construct polar
|
||||
sets, and via experiments on the human reference genome, show
|
||||
their practical superiority in designing efficient
|
||||
sequence-specific minimizers. AVAILABILITY AND IMPLEMENTATION: A
|
||||
reference implementation and code for analyses under an
|
||||
open-source license are at
|
||||
https://github.com/kingsford-group/polarset. SUPPLEMENTARY
|
||||
INFORMATION: Supplementary data are available at Bioinformatics
|
||||
online.},
|
||||
author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
|
||||
doi = {10.1093/bioinformatics/btab313},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = jul,
|
||||
number = {Suppl\_1},
|
||||
pages = {i187--i195},
|
||||
pmc = {PMC8686682},
|
||||
pmid = 34252928,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {Sequence-specific minimizers via polar sets},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btab313},
|
||||
volume = 37,
|
||||
year = 2021,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btab313}}
|
||||
|
||||
@article{Pan2024-hb,
|
||||
abstract = {MOTIVATION: The minimizer concept is a data structure for
|
||||
sequence sketching. The standard canonical minimizer selects a
|
||||
subset of k-mers from the given DNA sequence by comparing the
|
||||
forward and reverse k-mers in a window simultaneously according
|
||||
to a predefined selection scheme. It is widely employed by
|
||||
sequence analysis such as read mapping and assembly. k-mer
|
||||
density, k-mer repetitiveness (e.g. k-mer bias), and
|
||||
computational efficiency are three critical measurements for
|
||||
minimizer selection schemes. However, there exist trade-offs
|
||||
between kinds of minimizer variants. Generic, effective, and
|
||||
efficient are always the requirements for high-performance
|
||||
minimizer algorithms. RESULTS: We propose a simple minimizer
|
||||
operator as a refinement of the standard canonical minimizer. It
|
||||
takes only a few operations to compute. However, it can improve
|
||||
the k-mer repetitiveness, especially for the lexicographic order.
|
||||
It applies to other selection schemes of total orders (e.g.
|
||||
random orders). Moreover, it is computationally efficient and the
|
||||
density is close to that of the standard minimizer. The refined
|
||||
minimizer may benefit high-performance applications like binning
|
||||
and read mapping. AVAILABILITY AND IMPLEMENTATION: The source
|
||||
code of the benchmark in this work is available at the github
|
||||
repository https://github.com/xp3i4/mini\_benchmark.},
|
||||
author = {Pan, Chenxu and Reinert, Knut},
|
||||
doi = {10.1093/bioinformatics/btae045},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = feb,
|
||||
number = 2,
|
||||
pmc = {PMC10868324},
|
||||
pmid = 38269626,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {A simple refined DNA minimizer operator enables 2-fold faster computation},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btae045},
|
||||
volume = 40,
|
||||
year = 2024,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btae045}}
|
||||
|
||||
@article{Kille2023-px,
|
||||
abstract = {MOTIVATION: The Jaccard similarity on k-mer sets has shown to be
|
||||
a convenient proxy for sequence identity. By avoiding expensive
|
||||
base-level alignments and comparing reduced sequence
|
||||
representations, tools such as MashMap can scale to massive
|
||||
numbers of pairwise comparisons while still providing useful
|
||||
similarity estimates. However, due to their reliance on minimizer
|
||||
winnowing, previous versions of MashMap were shown to be biased
|
||||
and inconsistent estimators of Jaccard similarity. This directly
|
||||
impacts downstream tools that rely on the accuracy of these
|
||||
estimates. RESULTS: To address this, we propose the minmer
|
||||
winnowing scheme, which generalizes the minimizer scheme by use
|
||||
of a rolling minhash with multiple sampled k-mers per window. We
|
||||
show both theoretically and empirically that minmers yield an
|
||||
unbiased estimator of local Jaccard similarity, and we implement
|
||||
this scheme in an updated version of MashMap. The minmer-based
|
||||
implementation is over 10 times faster than the minimizer-based
|
||||
version under the default ANI threshold, making it well-suited
|
||||
for large-scale comparative genomics applications. AVAILABILITY
|
||||
AND IMPLEMENTATION: MashMap3 is available at
|
||||
https://github.com/marbl/MashMap.},
|
||||
author = {Kille, Bryce and Garrison, Erik and Treangen, Todd J and Phillippy, Adam M},
|
||||
doi = {10.1093/bioinformatics/btad512},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = sep,
|
||||
number = 9,
|
||||
pmc = {PMC10505501},
|
||||
pmid = 37603771,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {Minmers are a generalization of minimizers that enable unbiased local Jaccard estimation},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btad512},
|
||||
volume = 39,
|
||||
year = 2023,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btad512}}
|
||||
|
||||
@incollection{Golan2025-xf,
|
||||
address = {Cham},
|
||||
author = {Golan, Shay and Shur, Arseny M},
|
||||
booktitle = {Lecture Notes in Computer Science},
|
||||
doi = {10.1007/978-3-031-82670-2\_25},
|
||||
isbn = {9783031826696,9783031826702},
|
||||
issn = {0302-9743,1611-3349},
|
||||
language = {en},
|
||||
pages = {347--360},
|
||||
publisher = {Springer Nature Switzerland},
|
||||
series = {Lecture Notes in Computer Science},
|
||||
title = {Expected density of random minimizers},
|
||||
url = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
|
||||
year = 2025,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
|
||||
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-031-82670-2%5C_25}}
|
||||
|
||||
@article{Mohamadi2017-ok,
|
||||
abstract = {Motivation: Many bioinformatics algorithms are designed for the
|
||||
analysis of sequences of some uniform length, conventionally
|
||||
referred to as k -mers. These include de Bruijn graph assembly
|
||||
methods and sequence alignment tools. An efficient algorithm to
|
||||
enumerate the number of unique k -mers, or even better, to build
|
||||
a histogram of k -mer frequencies would be desirable for these
|
||||
tools and their downstream analysis pipelines. Among other
|
||||
applications, estimated frequencies can be used to predict genome
|
||||
sizes, measure sequencing error rates, and tune runtime
|
||||
parameters for analysis tools. However, calculating a k -mer
|
||||
histogram from large volumes of sequencing data is a challenging
|
||||
task. Results: Here, we present ntCard, a streaming algorithm for
|
||||
estimating the frequencies of k -mers in genomics datasets. At
|
||||
its core, ntCard uses the ntHash algorithm to efficiently compute
|
||||
hash values for streamed sequences. It then samples the
|
||||
calculated hash values to build a reduced representation
|
||||
multiplicity table describing the sample distribution. Finally,
|
||||
it uses a statistical model to reconstruct the population
|
||||
distribution from the sample distribution. We have compared the
|
||||
performance of ntCard and other cardinality estimation
|
||||
algorithms. We used three datasets of 480 GB, 500 GB and 2.4 TB
|
||||
in size, where the first two representing whole genome shotgun
|
||||
sequencing experiments on the human genome and the last one on
|
||||
the white spruce genome. Results show ntCard estimates k -mer
|
||||
coverage frequencies >15× faster than the state-of-the-art
|
||||
algorithms, using similar amount of memory, and with higher
|
||||
accuracy rates. Thus, our benchmarks demonstrate ntCard as a
|
||||
potentially enabling technology for large-scale genomics
|
||||
applications. Availability and Implementation: ntCard is written
|
||||
in C ++ and is released under the GPL license. It is freely
|
||||
available at https://github.com/bcgsc/ntCard. Contact:
|
||||
hmohamadi@bcgsc.ca or ibirol@bcgsc.ca. Supplementary information:
|
||||
Supplementary data are available at Bioinformatics online.},
|
||||
author = {Mohamadi, Hamid and Khan, Hamza and Birol, Inanc},
|
||||
date-modified = {2026-04-18 08:19:36 +0200},
|
||||
doi = {10.1093/bioinformatics/btw832},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = may,
|
||||
number = 9,
|
||||
pages = {1324--1330},
|
||||
pmc = {PMC5408799},
|
||||
pmid = 28453674,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {ntCard: a streaming algorithm for cardinality estimation in genomics data},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btw832},
|
||||
volume = 33,
|
||||
year = 2017,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btw832}}
|
||||
|
||||
@misc{Mash-distances-doc,
|
||||
author = {{Marbl Lab}},
|
||||
howpublished = {Mash documentation},
|
||||
title = {Mash Distance},
|
||||
url = {https://mash.readthedocs.io/en/latest/distances.html},
|
||||
urldate = {2026-07-09},
|
||||
year = 2026}
|
||||
|
||||
@article{Fan2015-mash-formula,
|
||||
author = {Fan, Huan and Ives, Anthony R and Surget-Groba, Yann and Cannon, Charles H},
|
||||
doi = {10.1186/s12864-015-1647-5},
|
||||
journal = {BMC Genomics},
|
||||
number = 1,
|
||||
title = {An assembly and alignment-free method of phylogeny reconstruction from next-generation sequencing data},
|
||||
url = {https://doi.org/10.1186/s12864-015-1647-5},
|
||||
volume = 16,
|
||||
year = 2015}
|
||||
@@ -1,3 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
</urlset>
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,342 +0,0 @@
|
||||
# NUMA-aware partition runner
|
||||
|
||||
## Problem
|
||||
|
||||
All partition-level parallel loops in obikindex currently fall into two
|
||||
categories:
|
||||
|
||||
**Naive Rayon** — used in `build_layers`, `pack_matrices`, `dump`, `select`,
|
||||
`stats`, `rebuild`, `reindex`:
|
||||
|
||||
```rust
|
||||
(0..n).into_par_iter().for_each(|i| work(i));
|
||||
```
|
||||
|
||||
Threads come from the global Rayon pool with no NUMA awareness. On
|
||||
multi-socket machines this produces cross-socket memory traffic and degrades
|
||||
performance super-linearly (see [NUMA-aware worker pools](numa_worker_pools.md)).
|
||||
|
||||
**Ad-hoc adaptive pool** — used in `merge`:
|
||||
|
||||
A bespoke implementation with pre-spawned workers, channel-based dispatch, and
|
||||
activation control. It handles NUMA correctly but is not reusable.
|
||||
|
||||
Both cases should be replaced by a single generic mechanism.
|
||||
|
||||
## Unified model
|
||||
|
||||
The key insight is that **UMA is just the NUMA case with a single node**. The
|
||||
runner always works the same way: one controller thread per node, each
|
||||
independently managing its own workers with the same adaptive logic. The only
|
||||
difference between UMA and NUMA is the number of nodes and whether workers are
|
||||
pinned.
|
||||
|
||||
```
|
||||
NUMA (k nodes) UMA (1 node)
|
||||
|
||||
controller-0 controller-1 … controller-0
|
||||
│ │ │
|
||||
workers[0] workers[1] workers[0]
|
||||
(pinned) (pinned) (global pool)
|
||||
└───────────────┴──────────────────┘
|
||||
shared work queue
|
||||
```
|
||||
|
||||
On each node, the Rayon `ThreadPool` is pinned to that node's CPUs.
|
||||
`pool.install()` ensures all internal Rayon calls (inside the work function)
|
||||
use the node-local pool. Linux first-touch then places heap allocations in
|
||||
local DRAM automatically.
|
||||
|
||||
On UMA the global Rayon pool is used directly — no pinning, no overhead.
|
||||
|
||||
## Adaptive mechanism
|
||||
|
||||
Each controller follows the same logic regardless of node count:
|
||||
|
||||
1. Pre-spawn `workers_per_node` dormant worker threads (blocked on `activate_rx`).
|
||||
2. Activate the first worker immediately.
|
||||
3. Loop on result channel with a `SPAWN_POLL` timeout:
|
||||
- On result: call `on_done`; check whether to activate the next worker.
|
||||
- On timeout: same check.
|
||||
- Activation criterion: `should_spawn_worker(active, global_efficiency, prev_efficiency)`.
|
||||
4. Drop `activate_tx` when done — dormant workers exit cleanly.
|
||||
|
||||
**Global CPU efficiency** (`CpuSample`, reads `/proc/stat` on Linux) is used by
|
||||
all controllers — no per-node measurement needed. The signal is coarser than
|
||||
per-node efficiency but correct in practice: if any node saturates memory
|
||||
bandwidth, the global efficiency drops and all controllers stop activating
|
||||
workers. Using a standard portable primitive avoids platform-specific CPU
|
||||
accounting and keeps the implementation clean.
|
||||
|
||||
## Proposed API
|
||||
|
||||
```rust
|
||||
pub struct PartitionRunner {
|
||||
// One entry per NUMA node; one entry total on UMA.
|
||||
nodes: Vec<NodeConfig>,
|
||||
}
|
||||
|
||||
struct NodeConfig {
|
||||
pool: Option<Arc<rayon::ThreadPool>>, // None = global Rayon pool (UMA)
|
||||
cpu_ids: Vec<usize>, // empty = no pinning (UMA)
|
||||
max_workers: usize,
|
||||
}
|
||||
|
||||
impl PartitionRunner {
|
||||
/// Detect topology and build the runner.
|
||||
/// Returns a single-node runner on UMA / macOS / hwloc failure.
|
||||
pub fn new() -> Self;
|
||||
|
||||
/// Run `f(i)` for every index in `order`, collecting results.
|
||||
///
|
||||
/// `on_done(i, result, elapsed)` is called under an internal mutex as
|
||||
/// each partition completes — use it for progress bars and aggregation.
|
||||
/// The runner serialises all calls to `on_done` via an internal
|
||||
/// `Arc<Mutex<C>>`, so no `Sync` bound is required on the callback.
|
||||
/// `Send` is required because the Arc clone crosses thread boundaries.
|
||||
///
|
||||
/// Serialisation is free in practice: a partition takes seconds to
|
||||
/// minutes; the callback takes microseconds. Contention is negligible.
|
||||
///
|
||||
/// Returns the first error from `f`, if any.
|
||||
pub fn run<F, R, E, C>(
|
||||
&self,
|
||||
order: &[usize],
|
||||
f: F,
|
||||
on_done: C,
|
||||
) -> Result<(), E>
|
||||
where
|
||||
F: Fn(usize) -> Result<R, E> + Send + Sync,
|
||||
R: Send,
|
||||
E: Send,
|
||||
C: FnMut(usize, R, Duration) + Send; // Send required, Sync is not
|
||||
}
|
||||
```
|
||||
|
||||
`order` is caller-supplied so each command chooses its scheduling strategy:
|
||||
largest-first for `merge`, sequential for `build_layers`, etc.
|
||||
|
||||
## Migration examples
|
||||
|
||||
### merge.rs (before: ~180 lines of bespoke machinery)
|
||||
|
||||
```rust
|
||||
let runner = PartitionRunner::new();
|
||||
runner.run(
|
||||
&order,
|
||||
|i| dst_partition.merge_partition(i, srcs, mode, n_dst_genomes, block_bits, evidence)
|
||||
.map_err(OKIError::Partition),
|
||||
|i, g_len, dur| {
|
||||
pb.inc(1);
|
||||
debug!("partition {i}: done in {:.1}s — {g_len} new kmers", dur.as_secs_f64());
|
||||
part_stats.push(PartStat { id: i, unitig_bytes: partition_sizes[i], g_len });
|
||||
},
|
||||
)?;
|
||||
```
|
||||
|
||||
### index.rs build_layers (before: naive into_par_iter)
|
||||
|
||||
```rust
|
||||
let order: Vec<usize> = (0..n).collect();
|
||||
let runner = PartitionRunner::new();
|
||||
runner.run(
|
||||
&order,
|
||||
|i| self.partition.build_index_layer(i, min_ab, max_ab, with_counts, &evidence, block_bits)
|
||||
.map_err(OKIError::Partition),
|
||||
|_, n_kmers, _| {
|
||||
total_kmers.fetch_add(n_kmers, Ordering::Relaxed);
|
||||
pb.inc(1);
|
||||
},
|
||||
)?;
|
||||
```
|
||||
|
||||
All other sites (`pack_matrices`, `dump`, `select`, etc.) follow the same
|
||||
pattern.
|
||||
|
||||
## Placement
|
||||
|
||||
`PartitionRunner` lives in `obikindex/src/numa.rs` alongside `NumaSetup`.
|
||||
It depends only on standard library primitives and Rayon — no new dependencies.
|
||||
|
||||
A single `PartitionRunner` instance can be built once per command invocation
|
||||
and reused across multiple `run()` calls (e.g. `merge` runs
|
||||
`merge_partitions` then `pack_matrices`).
|
||||
|
||||
## Known issue: CPU-only activation signal stalls on I/O-bound stages
|
||||
|
||||
Observed on a real `filter` run (109 genomes, 256 partitions, 8×24-core NUMA):
|
||||
`rebuild` (CPU-bound — k-mer construction) scales cleanly from 9 to 43 active
|
||||
workers as `CpuSample::do_i_activate` (`obisys::lib.rs`) sees efficiency climb.
|
||||
`pack_matrices` (I/O-bound — reopens and recomposes per-genome column files
|
||||
into `.pbmx`/`.pcmx`) activates one extra worker then flatlines at 10/192 for
|
||||
the rest of the stage, even though 256 partitions keep completing over several
|
||||
minutes. This matches the documented intent (§ Adaptive mechanism — "avoids
|
||||
over-provisioning ... I/O-bound ... workloads") but conflates two different
|
||||
things: *"CPU is not the bottleneck"* and *"more workers would not help"*. On
|
||||
storage with real queue depth (NVMe, RAID, parallel FS) the second stage could
|
||||
still benefit from more concurrent workers even with flat CPU usage — a signal
|
||||
the current mechanism cannot see.
|
||||
|
||||
A one-off artefact was also found in the same log: right after a stage
|
||||
transition, `do_i_activate` produced a physically impossible spike (efficiency
|
||||
~94 cores on a 192-core box) because it has no minimum-window guard — unlike
|
||||
its sibling `cpu_efficiency`, which returns `0.0` if `wall < 0.1s`
|
||||
(`obisys::lib.rs:260`). `do_i_activate` unconditionally overwrites
|
||||
`self.wall`/`self.user_secs`/`self.sys_secs` even when the elapsed window is
|
||||
too short to be meaningful, so a burst of rapid completions right after
|
||||
activating a worker can divide a real CPU delta by a near-zero wall delta.
|
||||
|
||||
### Implemented: I/O signal + shared debounce guard
|
||||
|
||||
`IoSample` (`obisys::lib.rs`, alongside `CpuSample`) is fed by
|
||||
`read_bytes`/`write_bytes` from `/proc/self/io` on Linux (actual bytes
|
||||
submitted to the block layer — not `rchar`/`wchar`, which also count
|
||||
page-cache hits, and not `ru_inblock`/`ru_oublock`, unreliable on macOS), with
|
||||
a `proc_pid_rusage(RUSAGE_INFO_V4)` fallback on macOS
|
||||
(`ri_diskio_bytesread`/`ri_diskio_byteswritten`, FFI only via `libc`, no new
|
||||
dependency — same pattern as the existing `getrusage` bindings). Any other
|
||||
target degrades gracefully to a signal that never triggers (falls back to
|
||||
CPU-only activation), same pattern as `cgroup_v2_available`.
|
||||
|
||||
`maybe_activate` (`numa.rs`) activates a worker if *either* signal still shows
|
||||
headroom, making `PartitionRunner` adapt to whichever resource is actually the
|
||||
bottleneck without per-call configuration. Both samplers are called
|
||||
unconditionally — no `||` short-circuit — so neither window starves behind
|
||||
whichever signal fires first:
|
||||
|
||||
```rust
|
||||
let cpu_threshold = CPU_SPAWN_THRESHOLD * activation.last_step() as f64;
|
||||
let cpu_wants_more = cpu_sample.do_i_activate(cpu_threshold);
|
||||
let io_wants_more = io_sample.do_i_activate(IO_SPAWN_THRESHOLD);
|
||||
if cpu_wants_more || io_wants_more {
|
||||
activation.grow(GROWTH_DIVISOR, n_total);
|
||||
}
|
||||
```
|
||||
|
||||
The CPU threshold is *not* the flat absolute delta it started as: it scales
|
||||
with `activation.last_step()` — the number of workers activated in the last
|
||||
growth step, tracked by `NodeActivation` (`numa.rs`) and updated every time
|
||||
`grow()` actually grows something. Growing by 8 workers should add ~8 cores of
|
||||
efficiency if the workload is truly CPU-bound; requiring only
|
||||
`CPU_SPAWN_THRESHOLD` (20 %) of that expected gain confirms the growth was
|
||||
useful without demanding perfect linear scaling. Scaling by the *last step's
|
||||
size* rather than the cumulative total keeps the bar equally meaningful
|
||||
whether it's the 2nd growth step or the 20th — a flat absolute threshold
|
||||
(0.2 core) is a strong signal at 8 active workers but pure noise at 150; a
|
||||
threshold scaled by the *cumulative* total instead (considered and rejected)
|
||||
would have made the bar essentially impossible to clear late in the ramp,
|
||||
strangling exactly the CPU-bound saturation the mechanism exists to allow.
|
||||
|
||||
Unlike the CPU signal (an absolute delta in cores — a bounded, portable unit),
|
||||
raw I/O throughput has no natural scale across devices, so `IoSample` uses a
|
||||
**relative** growth threshold instead of an absolute one:
|
||||
|
||||
```rust
|
||||
pub fn do_i_activate(&mut self, threshold: f64) -> bool {
|
||||
let elapsed = self.wall.elapsed().as_secs_f64();
|
||||
if elapsed < 0.1 { return false; } // state untouched — window keeps accumulating
|
||||
|
||||
let n = Self::read_bytes();
|
||||
let rate = n.saturating_sub(self.bytes) as f64 / elapsed;
|
||||
let activate = if self.previous_rate == 0.0 {
|
||||
rate > 0.0 // bootstrap: any measured throughput is signal
|
||||
} else {
|
||||
(rate - self.previous_rate) / self.previous_rate >= threshold
|
||||
};
|
||||
|
||||
self.bytes = n;
|
||||
self.wall = Instant::now(); // reset only on a real sample
|
||||
activate
|
||||
}
|
||||
```
|
||||
|
||||
The `elapsed < 0.1s → return false without mutating state` guard was also
|
||||
back-ported into `CpuSample::do_i_activate` (previously missing — source of
|
||||
the ~94-core artefact above) — one fix for both problems, and it removes the
|
||||
need for any arbitrary I/O-rate floor: a short/noisy window is rejected
|
||||
outright rather than papered over with a hardware-dependent constant.
|
||||
|
||||
Both spawn thresholds (`CPU_SPAWN_THRESHOLD`, `IO_SPAWN_THRESHOLD`, module-level
|
||||
`const` in `numa.rs`, both `0.2`) are a starting point, not a derived value:
|
||||
`0.2` (20 % relative growth) for `IoSample` was chosen to match the CPU
|
||||
threshold's *implicit* relative sensitivity (in the observed log, an 8→9
|
||||
worker step raised efficiency by ~12 %) — but I/O throughput is lumpier than
|
||||
CPU time (buffered writes flush in bursts), so it needs empirical validation
|
||||
against a real `pack` run before being considered final.
|
||||
|
||||
## Known issue: ramp-up too slow, and confused with node count
|
||||
|
||||
The original design started `n_nodes` workers (one per node) and grew one
|
||||
worker at a time. On a real `filter` run this took ~10 minutes to climb from
|
||||
9 to ~40 active workers even on the CPU-bound `rebuild` stage — most of a
|
||||
35-minute stage spent under-provisioned while waiting for evidence to
|
||||
accumulate one worker at a time. There is no scale-down mechanism (`n_active`
|
||||
only grows), so the original caution was deliberate — but a quarter of
|
||||
available cores is still far from saturation, and the real risk zone (over-provisioning
|
||||
a memory-bandwidth-bound stage) only shows up much later in the ramp, near
|
||||
full occupancy — not at 25 %.
|
||||
|
||||
The fix decouples ramp speed from node *count*: both the initial size and the
|
||||
growth step are a fraction of `workers_per_node` (node *size*), applied
|
||||
identically on every node. A single-NUMA-node (UMA) machine ramps exactly as
|
||||
fast as an 8-node one — growing by `n_nodes` per step, as first considered,
|
||||
would have degenerated to "grow by 1" on UMA, reproducing the original
|
||||
problem for exactly the machines that need the fix most.
|
||||
|
||||
```rust
|
||||
// NodeActivation::grow — called both at startup (activate_initial) and on
|
||||
// every CPU/IO-triggered growth step, with a different divisor each time.
|
||||
let wanted = (self.caps[idx] / divisor).max(1); // INITIAL_DIVISOR=4 at startup, GROWTH_DIVISOR=8 per step
|
||||
let room = self.caps[idx].saturating_sub(self.active[idx]);
|
||||
let grow = wanted.min(room).min(n_total.saturating_sub(self.total));
|
||||
```
|
||||
|
||||
This also fixed a latent correctness gap: the original single shared
|
||||
`activate_tx`/`activate_rx` pair had *no* per-node addressing — sending one
|
||||
activation signal woke up whichever dormant worker (from any node) happened
|
||||
to win the race on that channel. `crossbeam_channel` gives no fairness
|
||||
guarantee across competing receivers, so "round-robin across nodes" was an
|
||||
assumption the code never actually enforced. `PartitionRunner::run` now opens
|
||||
one activation channel per node (`activate_txs`/`activate_rxs`, one pair per
|
||||
`NodeConfig`); `NodeActivation` (`numa.rs`) tracks how many of each node's
|
||||
dormant workers have been woken and grows every node by the same amount per
|
||||
step, capped by that node's remaining dormant workers and by the run's total
|
||||
budget (`n_total`) — balance across nodes is now guaranteed by construction,
|
||||
not incidental to channel implementation details.
|
||||
|
||||
## Panic propagation (fixed, 2026-08-28)
|
||||
|
||||
A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
|
||||
— the controller's `while completed < n_total` loop then waited forever for
|
||||
an event that partition could no longer produce, since other live
|
||||
workers/timer threads kept `event_rx` open. Discovered via `obikselect`
|
||||
panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
|
||||
the process hung indefinitely instead of erroring out.
|
||||
|
||||
Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
|
||||
`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
|
||||
partition. The controller counts it toward `completed` (unblocking the loop)
|
||||
and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
|
||||
re-raises it on the caller's thread — the original message/backtrace still
|
||||
surfaces, from the right place, instead of a silent deadlock. Takes priority
|
||||
over a plain `Err` (a panic means a bug, not a normal typed failure).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Error handling**: `run` currently returns the first error; remaining errors
|
||||
are dropped. A `Vec<E>` return would give complete diagnostics. (Panics are
|
||||
now caught and re-raised individually — see above — this only concerns
|
||||
plain `Err` results.)
|
||||
|
||||
- **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
|
||||
(start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an
|
||||
observed too-slow ramp — not yet validated against a real `pack` (I/O-bound)
|
||||
run, where over-provisioning risk is different from the CPU-bound `rebuild`
|
||||
case this was tuned against.
|
||||
|
||||
- **`on_done` ordering**: the runner serialises calls to `on_done` via an
|
||||
internal `Arc<Mutex<C>>`. `Send` is required (the Arc clone crosses thread
|
||||
boundaries); `Sync` is not (only one thread holds the lock at a time).
|
||||
Contention is negligible because a partition takes seconds while the callback
|
||||
takes microseconds. The callback is therefore simple to write (plain
|
||||
`Vec::push`, plain `FnMut`) with no measurable performance cost.
|
||||
@@ -1,406 +0,0 @@
|
||||
# Query system
|
||||
|
||||
## Goal
|
||||
|
||||
Given a set of query sequences, determine for each sequence how many of its k-mers are found in the index and, for each indexed genome, how many k-mers match. The query system is the foundation for read classification and sequence-to-genome mapping.
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
- Query sequences in FASTA or FASTQ format (gzip supported, streaming stdin supported). GenBank flat files are not supported at query time (only at index time).
|
||||
- Sequences shorter than k bases are silently skipped.
|
||||
- Non-ACGT characters are handled by the superkmer decomposition layer: they act as hard breaks, producing shorter superkmers (identical to the behaviour at indexing time).
|
||||
|
||||
---
|
||||
|
||||
## Algorithm
|
||||
|
||||
The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikindex::partition::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
|
||||
|
||||
```
|
||||
for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
|
||||
build QueryBatch (QueryBatch::from_records):
|
||||
decompose all sequences into superkmers (SuperKmerIter) — construction only,
|
||||
not the dedup key
|
||||
deduplicate at k-mer granularity, split by partition in the same pass:
|
||||
by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>> ← KmerDesc = (seq_idx, pos)
|
||||
allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers —
|
||||
NOT multiplied by n_genomes
|
||||
allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome —
|
||||
stays empty (zero cost) for every genome this chunk never matches
|
||||
for each partition p:
|
||||
query_partition_with(p, kmers_for_p, on_event):
|
||||
stage 1 (MPHF-only): for each unique k-mer, try each layer's MphfLayer::find
|
||||
in turn, stop at the first hit; bucket confirmed hits by (layer, slot);
|
||||
emit QueryHit::Found(descs) once per hit k-mer
|
||||
stage 2 (column-major fetch): for each layer with ≥1 hit, for each genome
|
||||
column g in 0..layer.n_cols(): scan that layer's bucketed slots, look up
|
||||
col_value(g, slot); emit QueryHit::Value(descs, g, value) on nonzero
|
||||
on_event dispatches: Found → SmerIndex::mark_found for every desc;
|
||||
Value → push (seq_idx, pos, value) into by_genome[g]
|
||||
for each genome g with ≥1 hit (sparse_findere_for_genome):
|
||||
sort by_genome[g] by (seq_idx, pos); detect maximal runs of consecutive pos
|
||||
within one seq_idx; monotone-deque window-minimum scoped to each run →
|
||||
confirmed_by_genome[g]: Vec<(seq_idx, pos_out, value)>
|
||||
accumulate genome_totals per sequence from confirmed_by_genome (per genome, direct)
|
||||
accumulate kmer_count / kmer_missing per (sequence, output position), O(1) each,
|
||||
using only the confirmed-any bitmap and SmerIndex — independent of n_genomes
|
||||
if --detail: densify confirmed_by_genome into per-(seq, genome) coverage arrays
|
||||
emit annotated sequences (emit_batch)
|
||||
```
|
||||
|
||||
Superkmers that appear more than once in the batch (same sequence or across sequences), or different superkmers that happen to share a k-mer (read overlaps, repeats, a SNP splitting an otherwise-identical run), are deduplicated at k-mer granularity: each unique `CanonicalKmer` triggers at most one MPHF lookup and, on hit, one matrix fetch, broadcast to every `KmerDesc` occurrence referencing it.
|
||||
|
||||
**Findere requires full-sequence aggregation.** The sliding window (now per-run, not per-sequence — see [Findere z-window filter](#findere-z-window-filter)) only ever runs after all partitions have contributed their hits to `by_genome`. Applying it per superkmer would produce false negatives at superkmer boundaries, where the z-window spans two superkmers.
|
||||
|
||||
Batches are processed in parallel via `obipipeline` workers; the `--threads` flag controls the number of worker threads.
|
||||
|
||||
---
|
||||
|
||||
## Findere z-window filter
|
||||
|
||||
For approximate index modes, the index physically stores s-mers of size `s = k_user − z + 1`; `idx.kmer_size()` (bound to `k` in `process_chunk`) is this physically-indexed s-mer size, so decomposing the query at `k` naturally produces s-mer results.
|
||||
|
||||
The z-window aggregation is **sparse**, per genome, implemented in `sparse_findere_for_genome` (`query.rs`) — a run-detection pass followed by a monotone-deque sliding-window minimum scoped to each run, not a dense scan over every s-mer position of every sequence:
|
||||
|
||||
```
|
||||
sparse_findere_for_genome(hits, z, presence, threshold):
|
||||
// hits: raw (seq_idx, pos_smer, value) triples for this genome, as delivered
|
||||
// by query_partition_with's QueryHit::Value — only ever nonzero entries;
|
||||
// a position with no hit for this genome simply has no entry at all.
|
||||
sort hits by (seq_idx, pos_smer)
|
||||
|
||||
for each maximal run of consecutive pos_smer values within the same seq_idx:
|
||||
dq: VecDeque<(run-relative index, value)>
|
||||
for k, (_, pos, value) in enumerate(run):
|
||||
maintain dq monotone non-decreasing (pop back while back.value >= value)
|
||||
push (k, value)
|
||||
evict dq entries with run-relative index <= k - z
|
||||
if k + 1 >= z:
|
||||
win_min = dq.front().value
|
||||
if win_min > 0:
|
||||
pos_out = pos + 1 - z
|
||||
confirmed.push((seq_idx, pos_out, adjust(win_min)))
|
||||
return confirmed
|
||||
```
|
||||
|
||||
A window can only be confirmed (`win_min > 0`) when all `z` s-mers in it are present *and* nonzero for this genome — which, by construction, can only happen strictly inside one contiguous run of hits (any gap — an absent or zero-valued s-mer — forces `win_min = 0` for every window spanning it, exactly matching the old dense scan's "not in index counts as 0" rule, just never materialising the zero). The deque logic is otherwise identical to the pre-sparsification version; it's scoped to run-relative indices instead of the whole sequence.
|
||||
|
||||
This runs once per genome that has at least one hit in the chunk (`process_chunk` iterates `by_genome`, one `Vec<(seq_idx, pos_smer, value)>` per genome, built from `QueryHit::Value` during the partition loop — genomes with zero hits in this chunk have an empty `Vec` and cost nothing beyond the iteration itself). Total work is `O(hits log hits)` per genome (the sort) rather than `O(n_smers)` per genome regardless of hit count — a genuine complexity win on top of the memory one, for the common case where most `(chunk, genome)` pairs have no or few hits.
|
||||
|
||||
Output position `pos_out` is confirmed for genome `g` iff its run produced a nonzero `win_min` — equivalent to "all `z` consecutive s-mer values in the window are nonzero for `g`", same semantics as before.
|
||||
|
||||
**The value reported per confirmed position is the window minimum, not the leftmost s-mer's raw value** — unchanged from the dense version. For presence indexes (0/1 values) this is equivalent to a logical AND either way. For count indexes it is not: the accumulated count for genome `g` at position `pos_out` is the minimum across the window, the weakest link — not the leftmost s-mer's own count. The presence/count adjustment (`u32::from(win_min >= threshold)` vs. raw `win_min`) is applied once, inside `sparse_findere_for_genome`, rather than later during accumulation.
|
||||
|
||||
**`kmer_missing` bookkeeping is independent of the per-genome sparse structures**, by design (see roadmap point 9): a lightweight dense `SmerIndex` (`in_index: Vec<bool>`, sized `total_smers` — **not** multiplied by `n_genomes`) is populated from `QueryHit::Found` during the partition loop, one entry per hit k-mer regardless of which genome(s) it matched. A position with no genome confirmed counts as `kmer_missing` iff the leftmost s-mer of that window is absent from `SmerIndex` entirely (see [`kmer_missing` semantics](#kmer_missing-semantics)).
|
||||
|
||||
**Coverage (`--detail`)** is built by re-scanning each genome's confirmed-hit list (already computed, no extra pass over raw data) and densifying into the `[u32; n_kmers_out]` arrays the JSON output format requires — but only when `--detail` is actually requested; the sparse structures cost nothing extra when it isn't.
|
||||
|
||||
**Short sequences**: when a sequence's s-mer count is less than `z`, its run(s) — if any hits exist at all — can never reach length `z`, so no window is ever confirmed for it; no k_user-mer is emitted, same outcome as the dense version's `n_kmers_out == 0` early-skip, reached here as a natural consequence rather than a separate check.
|
||||
|
||||
**Exact indexes**: `z = 1`, every single-hit "run" of length 1 immediately satisfies `k + 1 >= z`, so every hit is its own confirmed window with `win_min` equal to its own value — a passthrough, as before.
|
||||
|
||||
### Effective z at query time
|
||||
|
||||
`effective_z` is resolved at the start of `run()`:
|
||||
|
||||
```rust
|
||||
let effective_z = args.findere_z.unwrap_or_else(|| match idx.meta().config.evidence {
|
||||
IndexMode::Approx { z, .. } | IndexMode::Hybrid { z, .. } => z as usize,
|
||||
IndexMode::Exact => 1,
|
||||
});
|
||||
```
|
||||
|
||||
The `-z` CLI option overrides the index metadata value. A higher z increases stringency (lower FP, some true positives may be discarded at sequence ends); a lower z increases sensitivity.
|
||||
|
||||
---
|
||||
|
||||
## Layer lookup: `MphfLayer::find`
|
||||
|
||||
`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obikindex::layer](../implementation/layer_tier.md) for the full `find` / `find_strict` API.
|
||||
|
||||
### `QueryLayer` variant selection
|
||||
|
||||
`QueryLayer::open` (`obikindex/src/partition/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
|
||||
|
||||
| Order | Condition | Variant | Data returned per k-mer |
|
||||
|---|---|---|---|
|
||||
| 1 | `with_counts=true` and `counts/` exists | `Count` | raw count per genome |
|
||||
| 2 | (else) `presence/` exists, or `counts/` doesn't exist at all | `Presence` | see below |
|
||||
| 3 | (else — `counts/` exists, `presence/` doesn't, `with_counts=false`) | `Count` | counts used as-is |
|
||||
|
||||
There is no `QueryLayer::SetOnly` variant. The "no on-disk matrix at all" case is handled one level down: `Presence` wraps `PersistentBitMatrix`, whose own `open()` (`obicompactvec/src/bitmatrix.rs:260-288`) auto-detects among **three** internal representations — `Packed` (`presence/matrix.pbmx`), `Columnar` (`presence/meta.json`), or `Implicit { n_rows, n_cols }` when neither file exists (built from `layer_meta.json`, `fill_row` returning all-`1`s without touching disk). This is where "1 for every genome" actually happens — not at the `QueryLayer` level.
|
||||
|
||||
**Worth double-checking, not confirmed as a bug**: `PersistentBitMatrix::open`'s `Implicit` branch constructs `Implicit { n_rows: meta.n, n_cols: 1 }` — `n_cols` is hardcoded to `1`, not to the layer's actual `n_genomes`. `fill_row` for `Implicit` only writes `buf[..1]`, leaving the rest of a longer `n_genomes`-sized buffer untouched (zeroed by the caller beforehand). If this path is ever reached for a layer covering more than one genome, only genome index 0 would read as present. Whether that's reachable in practice (layers might always be single-genome when they fall back to `Implicit`) wasn't verified here — flagging for follow-up, not fixing.
|
||||
|
||||
---
|
||||
|
||||
## Presence / count mode at query time
|
||||
|
||||
The `--force-presence` flag and `--presence-threshold` control how per-genome values are accumulated, independently of what the index stores:
|
||||
|
||||
```
|
||||
genome_totals[g] += if presence { u32::from(v >= threshold) } else { v }
|
||||
```
|
||||
|
||||
`presence` is true when `--force-presence` is set or when the index has no counts (`!with_counts`). The default `presence_threshold` is 1, so any nonzero count counts as a match.
|
||||
|
||||
---
|
||||
|
||||
## Coverage vectors (`--detail`)
|
||||
|
||||
When `--detail` is requested, a 3-D accumulator `cov[seq_idx][genome][kmer_pos]` is allocated after all partitions are queried, with dimensions derived from `n_kmers_out = n_smers − z + 1` (k_user-mer positions, not s-mer positions):
|
||||
|
||||
```
|
||||
cov[seq_idx][g][pos] += contribution
|
||||
where pos is the k_user-mer index in the filtered (post-Findere) vector
|
||||
```
|
||||
|
||||
Coverage reflects confirmed k_user-mers only. The vectors are emitted in the JSON annotation under the key `"coverage"`.
|
||||
|
||||
---
|
||||
|
||||
## `kmer_missing` semantics
|
||||
|
||||
`kmer_missing` counts k_user-mer positions where the leftmost s-mer of the window (`smer_index.is_in_index(seq_idx, pos)`, `SmerIndex`) is `false` — i.e. absent from the index entirely. K-mers where the z-window fails because a later s-mer is absent or zero (but the leftmost one is present) are not counted as missing — the leftmost s-mer being present is used as proxy for index membership.
|
||||
|
||||
---
|
||||
|
||||
## Output format
|
||||
|
||||
Output sequences are written in **OBITools4 format**: the original sequence with a JSON annotation map in the title line.
|
||||
|
||||
```
|
||||
>read_id {"kmer_count":59,"kmer_strict_matches":{"genome_a":42,"genome_b":7}}
|
||||
ATCGATCG...
|
||||
```
|
||||
|
||||
With `--detail`:
|
||||
|
||||
```
|
||||
>read_id {"kmer_count":59,"kmer_strict_matches":{...},"coverage":{"genome_a":[0,1,2,...],...}}
|
||||
ATCGATCG...
|
||||
```
|
||||
|
||||
Genome keys follow the iteration order of `meta.genomes`.
|
||||
|
||||
---
|
||||
|
||||
## Annotation schema
|
||||
|
||||
| Key | Type | Condition | Semantics |
|
||||
|---|---|---|---|
|
||||
| `kmer_count` | int | always | k-mers confirmed (post-Findere) with at least one genome match |
|
||||
| `kmer_missing` | int | `--count-missing` | k-mers absent from the index entirely (leftmost s-mer of the window not found) |
|
||||
| `kmer_strict_matches` | object | always | per-genome accumulated value, non-zero entries only (label → count or 0/1) |
|
||||
| `coverage` | object | `--detail` | per-genome array of per-position contributions (label → [u32]) |
|
||||
|
||||
`kmer_count + kmer_missing` ≤ total k_user-mers in the sequence. The gap corresponds to k_user-mers whose z-window was not fully confirmed (at least one s-mer absent or zero for all genomes) but whose first s-mer was present in the index.
|
||||
|
||||
---
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
obikmer query <index> [--detail] [--mismatch] [--count-missing]
|
||||
[--force-presence] [--presence-threshold <n>]
|
||||
[-z <z>] [-T <threads>] [--chunk-size <MiB>]
|
||||
<query.fa> [<query2.fa> ...]
|
||||
```
|
||||
|
||||
| Option | Default | Semantics |
|
||||
|---|---|---|
|
||||
| `-z` / `--findere-z` | from index metadata | Override Findere z parameter |
|
||||
| `--detail` | off | Emit per-position coverage vectors in JSON |
|
||||
| `--count-missing` | off | Add `kmer_missing` field to JSON |
|
||||
| `--force-presence` | off | Report 0/1 per genome regardless of index counts |
|
||||
| `--presence-threshold` | 1 | Minimum count to declare genome present |
|
||||
| `-T` / `--threads` | all CPUs | Worker threads |
|
||||
| `--chunk-size` | auto (from available RAM and thread count) | I/O chunk size in MiB — see [Future work, point 3](#throughput--parallelism--identified-potential-not-yet-implemented) for why the auto-sizing formula currently under-estimates memory on indexes with many genomes |
|
||||
|
||||
`--mismatch` is accepted but currently ignored with a warning on stderr.
|
||||
|
||||
---
|
||||
|
||||
## Future work
|
||||
|
||||
- **`--mismatch`**: 1-mismatch approximate matching — generate `3·k` single-substitution variants per k-mer, look each up independently.
|
||||
- **Read classification** (`--classify`): assign each read to the genome with the highest match score.
|
||||
- **Whitelist / blacklist filtering**: threshold-based accept/reject on per-genome match scores.
|
||||
|
||||
### Throughput & parallelism — identified potential (not yet implemented)
|
||||
|
||||
Observed on a 192-core (8×24 NUMA) machine: `query` uses ~10 cores or fewer, and the default chunk size gets the process OOM-killed. Root causes and candidate fixes, in dependency order:
|
||||
|
||||
**1. Single-threaded I/O source (main core-utilization bottleneck).**
|
||||
`run()` builds `all_chunks` via `paths.into_iter().flat_map(read_sequence_chunks_sized(...))` and passes it directly as the `input` iterator to `pipe.apply()`. In `obipipeline::Pipe::apply` (`scheduler.rs`), `input.next()` is called exclusively from the dedicated source thread — so file opening, decompression, and FASTA/FASTQ chunk-boundary parsing for *all* input files run serially in one thread, regardless of `--threads`. Compare with `steps::scatter` (used by `index`) and `cmd/superkmer.rs`: there, file opening + streaming is itself a `Flat` pipeline stage (`||?`), executed across the `n_workers` pool, with `obipipeline::throttle(paths, max_open)` bounding concurrently-open files in the source thread. That pattern parallelises I/O across files (and NUMA nodes); `query.rs` cannot.
|
||||
Fix direction: restructure `query`'s pipe with an initial `Flat` stage analogous to `scatter`'s, opening/chunking files across workers instead of in `flat_map`.
|
||||
|
||||
**2. Gzip decompression is inherently single-threaded per file.**
|
||||
`niffler`/`flate2` (used by `xopen`) do standard DEFLATE, which has no parallel-decodable structure for an arbitrary stream. Fix (1) parallelises *across* files but not *within* one large gzip file. Parking a possible fix (`rapidgzip-rs`) is tracked in [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen).
|
||||
|
||||
**3. Chunk-size memory formula ignores `n_genomes`.**
|
||||
`chunk_bytes = available_memory_bytes() / (n_workers * 16)` (`query.rs:407-414`) assumes a fixed ~8–16× overhead per raw input byte. But `KmerResults::new` (`query.rs:165-179`) allocates `data: Vec<u32>` sized `total_kmers_in_chunk × n_genomes` — dense, **for every k-mer position in the chunk, hit or not** — plus `win_min` and (with `--detail`) `cov`, same scaling. Real per-chunk memory is `O(n_genomes)`, not constant; the formula doesn't know `n_genomes` at all. This is the direct cause of the OOM kill on indexes with many reference genomes.
|
||||
|
||||
**4. MPHF lookup and matrix-row fetch are fused, not staged.**
|
||||
`QueryLayer::find_into` (`obikindex/src/partition/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
|
||||
|
||||
**5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.**
|
||||
`QueryBatch::from_records` currently dereplicates at the *superkmer* level (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, `query.rs:112`). This misses redundancy between k-mers shared by *different* superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer *construction* (`SuperKmerIter`) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on `CanonicalKmer`, in the same pass: `HashMap<CanonicalKmer, Vec<(seq_idx, pos)>>`. This also means the MPHF `find` itself runs once per **distinct** k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.
|
||||
|
||||
**6. Stage 1 output: bucket confirmed hits by layer, keyed by MPHF slot.**
|
||||
For each unique canonical k-mer, MPHF lookup across a partition's layers stops at the first match (`query_partition_with:105-111`) — a k-mer belongs to at most one layer. So stage 1's output can be reshaped directly into:
|
||||
```
|
||||
HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>
|
||||
```
|
||||
replacing the `CanonicalKmer` key by the resolved `slot` (compact integer, and exactly what stage 2 needs to address the matrix). K-mers matching no layer simply have no entry here (they still count toward `in_index`/`kmer_missing` bookkeeping, which stays `O(1)` per position, independent of `n_genomes`).
|
||||
|
||||
**7. Partition-level parallelism is currently absent — and a NUMA-aware mechanism for exactly this already exists, unused, in `obikindex`.**
|
||||
`process_chunk`'s partition loop (`query.rs:250-278`, `for (part_idx, part_sks) in by_part.iter().enumerate()`) processes every partition of a chunk sequentially on the single worker thread that owns that chunk. This is a parallelism axis on its own, independent of the column question below.
|
||||
More importantly: `docmd/architecture/numa_partition_runner.md` and `numa_worker_pools.md` document `PartitionRunner` (`obikindex/src/numa.rs`), **already implemented** and already used by `merge.rs`, `index.rs` (`build_layers`), `select.rs`, `reindex.rs`, `rebuild.rs` — one controller thread per NUMA node, a Rayon pool pinned to that node's CPUs (`hwlocality`, `numa` feature, default-on in `obikindex/Cargo.toml`), adaptive worker activation driven by *both* a CPU-efficiency signal and an I/O-throughput signal (`CpuSample`/`IoSample`, `/proc/self/io` on Linux). It exists precisely because a naive `into_par_iter()` on the global Rayon pool measurably degrades ×60 on this codebase's own 192-core/8-NUMA reference machine (`numa_worker_pools.md`, § Problem) once workers contend for cross-socket memory bandwidth on shared mmap'd/hashed structures — exactly the shape of the matrix-column scan in point 8 below.
|
||||
`obikmer` already depends on `obikindex` (`obikmer/Cargo.toml`, for `KmerIndex`), so `PartitionRunner` is directly reachable from `cmd/query.rs` — no new dependency. Both the partition-level loop and (see point 8) the genome-column scan should be driven through it rather than through ad-hoc `rayon::into_par_iter()`, to avoid reproducing the already-measured-and-fixed contention problem. Also relevant: the "CPU-only signal stalls on I/O-bound stages" issue documented for `pack_matrices` (mmap-heavy, page-fault-bound) applies just as much to a column-major mmap scan over persistent matrices — reuse the existing dual CPU/IO activation signal rather than re-deriving one.
|
||||
>
|
||||
> **Correction from implementation (Phase 4 below)**: this turned out not to be viable as described. `PartitionRunner::run()`'s actual body spawns roughly one OS thread per worker slot across every NUMA node **on every call** (confirmed by reading `numa.rs`, not just its doc comments) — fine for the one-call-per-command-invocation batch usage in `merge`/`build_layers`, but `query_partition_with` runs once per `(chunk, partition)`, far too frequently to absorb that spawn cost. Partition-level parallelism via `PartitionRunner` is deferred, not implemented. See Phase 4's "What did not ship, and why" for the detail.
|
||||
|
||||
**8. Stage 2: column-major matrix fetch, parallel across genome columns — via `PartitionRunner`, not naive `rayon`.**
|
||||
Both persistent matrix formats are column-oriented on disk: `ColumnarCompactIntMatrix`/`ColumnarBitMatrix` (`obicompactvec/src/{intmatrix,bitmatrix}.rs`) mmap one file per genome column; `PackedCompactIntMatrix`/`PackedBitMatrix` mmap one region-offset per column in a single file. `fill_row(slot, buf)` as used today (`query.rs:262-272` via `on_hit`) reads **one slot across all `n_genomes` columns** per hit — the worst possible access pattern for this layout (up to `n_genomes` scattered mmap regions touched per single k-mer).
|
||||
Better: for each layer, walk the matrix **column by column** (genome by genome): for each genome, scan the `slot` keys collected in step 6 for that layer and call `col.get(slot)`, keeping only nonzero results, and broadcast to the associated `(seq_idx, pos)` list. Total `get()` calls are unchanged (`n_hits × n_genomes` in the worst case) — the win is locality (sequential access within one mmap'd column at a time, not scattered across all columns per hit), not fewer operations.
|
||||
Columns are independent (read-only, disjoint mmap regions) → embarrassingly parallel across genomes, *but* — per point 7 — `obicompactvec`'s existing `into_par_iter()` over `0..n_cols` (`sum()`, `count_nonzero()`, pairwise distance matrices) is the **naive, unpinned** pattern the rest of the codebase is actively migrating away from, not a model to copy here. Route this through `PartitionRunner` (or the same NUMA-pool machinery) instead. Two things to settle when this is designed: how the partition axis (point 7), the column axis, and the existing chunk-level `n_workers` `obipipeline` pool compose without oversubscribing the machine (three different concurrency mechanisms — raw-thread pipe workers, `PartitionRunner`'s pinned Rayon pools, and whatever drives the column scan — need a single reconciled thread budget, not three independent ones); and the threshold below which per-column dispatch overhead outweighs the gain (small `n_genomes` or small per-layer hit counts) — to be measured, not assumed.
|
||||
>
|
||||
> **Correction from implementation (Phase 4 below)**: column-major fetch is implemented — but as a plain sequential loop, not parallelised via `PartitionRunner`. Same reason as point 7's correction above. The column-major *locality* win (the actual claim of this point) does not depend on adding parallelism on top of it, and is validated independently. Column-level parallelism is deferred pending a mechanism that fits this call frequency (candidates noted in Phase 4).
|
||||
|
||||
**9. Sparse per-genome representation, fed directly to Findere.**
|
||||
Stage 2's output should be `HashMap<genome_idx, Vec<(seq_idx, position, count)>>`, **sorted by `(seq_idx, position)`** once collected, instead of a dense `KmerResults`-style matrix — the key must carry `seq_idx`, not just `genome_idx`, because a chunk batches many sequences and `position` is only meaningful within one; a plain `Vec<(position, count)>` per genome would silently mix positions from different sequences and corrupt the sliding-window scan. This bounds retained memory by actual nonzero hits on both axes (position sparsity from non-matching k-mers, genome sparsity from a matched k-mer typically belonging to only a handful of genomes out of possibly many). The Findere sliding-window (`process_chunk`, the `win_min`/deque loop) would need reworking to run per `(sequence, genome)` over its sparse, sorted `(position, count)` list — detect runs of ≥`z` consecutive positions, window-min within each run — instead of today's dense `O(total_kmers × n_genomes)` scan. This is also a genuine complexity win (`O(hits log hits)` per genome vs. dense scan), not just memory.
|
||||
**Not covered by this sparsification**: `--detail`'s `cov` accumulator (`query.rs:304-308`) has the identical `n_genomes`-dense scaling problem and wasn't folded into points above. It doesn't need to be retained densely throughout processing, though — only the final JSON serialization (`emit_batch`) requires a dense `[u32]` per `(seq, genome)`, and only for the sequences actually being output with `--detail`. Densification can stay a late, output-time-only step, reconstructed from the sparse per-genome lists.
|
||||
|
||||
**Secondary patterns available from `scatter.rs`/`superkmer.rs`, not yet in `query.rs`:**
|
||||
- `throttle()` + `CommonArgs::effective_max_open()` to bound concurrently-open input files (query.rs defines its own `QueryArgs`, doesn't reuse this).
|
||||
- Progress bar with EMA throughput + live active-worker gauges (`obisys::spinner`, `flat_active`/`transform_active` counters) — diagnostic value for locating the bottleneck.
|
||||
- `obisys::Reporter`/`Stage::start`/`stop` timing per phase (used by `index`, `filter`; absent from `query`).
|
||||
|
||||
None of this is implemented yet — parked here as a coherent roadmap while the design is discussed further. Suggested dependency order: (1) I/O parallelism → (3) genome-aware chunk sizing → (4)–(9) staged/k-mer-deduped/NUMA-aware-partition-and-column-major/sparse query engine (larger refactor, biggest structural payoff — reuses `PartitionRunner` rather than inventing a new parallelism mechanism) → (2) parallel gzip (separate, orthogonal, tracked in chunkreader.md) → secondary diagnostics patterns.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan
|
||||
|
||||
Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikindex::partition`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
|
||||
|
||||
Instrumentation is deliberately sequenced *before* the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.
|
||||
|
||||
Performance measurement on the reference 192-core/8-NUMA machine is done by the project owner, not from this development environment (macOS, 16 cores — `PartitionRunner`'s NUMA pinning is Linux-only, so even phase 4's mechanism can't be functionally exercised for its actual purpose here). Each phase below is therefore written to be *self-measuring*: the debug-level logging it adds must be enough, on its own, to judge whether that phase's algorithmic choice paid off from a cluster run's logs, without needing to attach a profiler.
|
||||
|
||||
### Conventions applied to every phase below
|
||||
|
||||
**Debug logging.** Every phase that changes an algorithmic choice (not phase 0, which *is* the logging) adds `tracing::debug!`/`trace!` at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF `find` calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured `debug!` per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.
|
||||
|
||||
**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obikindex::layer`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikindex::partition` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
|
||||
|
||||
**Workflow (`jj`).** Work happens in a fresh `jj` commit, easy to abandon. `jj new` between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then `query_layer.rs` split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.
|
||||
|
||||
### Phase 0 — Instrumentation (prerequisite for measuring every later phase)
|
||||
|
||||
**Goal**: make core utilization, throughput, and per-stage timing visible on a real run, so phases 1–5 can be justified with numbers instead of assumption.
|
||||
|
||||
- `obikmer/src/cmd/query.rs`: wrap `run()`'s main loop with `obisys::Reporter`/`Stage::start("query")`/`.stop()`, printed at the end via `rep.print()` — same pattern as `index.rs`/`filter.rs`.
|
||||
- Add an `obisys::spinner("query")` progress bar around the `pipe.apply(...)` loop, with an EMA throughput readout (bases/s or k-mers/s, mirroring `steps::scatter`'s `ema_rate` computation, `scatter.rs:88-118`) and live gauges for "chunks in flight" / "workers busy" — reuse the `AtomicU32` counter pattern from `scatter.rs` (`flat_active`, `transform_active`) rather than inventing a new one.
|
||||
- Add `max_open_files: Option<usize>` to `QueryArgs` and a `effective_max_open()` method mirroring `CommonArgs::effective_max_open()` (`obikmer/src/cli.rs:90-94`) — needed by phase 1's `throttle()` call. (`QueryArgs` can't just embed `CommonArgs` — it doesn't take `kmer_size`/`minimizer_size`/`partitions`/`level_max`/`theta` from the CLI, those come from the index metadata — so this is a small standalone addition, not a flatten.)
|
||||
- Add one structured `debug!` per `process_chunk` call: chunk byte size, sequence count, s-mer count, wall time, and (once later phases exist) the fields they add — this single log line is the baseline every later phase's own logging gets compared against.
|
||||
- **Validation**: none needed beyond "the numbers appear and look sane" — this phase changes no query logic or output.
|
||||
- **Deliverable used by every phase below**: a before/after throughput and core-utilization measurement on the reference machine.
|
||||
|
||||
### Phase 1 — Parallel per-file I/O (fixes root cause of low core utilization)
|
||||
|
||||
**Goal**: file opening, decompression, and chunk-boundary parsing run across the `n_workers` pool instead of serially in the pipe's dedicated source thread.
|
||||
|
||||
- `obikmer/src/cmd/query.rs`:
|
||||
- Replace the `paths.into_iter().flat_map(read_sequence_chunks_sized(...))` construction (current `run()`, building `all_chunks`) with `obipipeline::throttle(paths.into_iter(), args.effective_max_open())`, passed as the pipe's `input`.
|
||||
- Add a new `QueryData::Path(PathBuf)` variant (alongside `Chunk`/`Output`) to carry the throttled path through the pipe's type-erasure mechanism.
|
||||
- Add a new **first** pipe stage, `Flat`/fallible (`||?`), modeled on `scatter.rs:60-86` and `superkmer.rs:54-65`: given a `Throttled<PathBuf>`, call `read_sequence_chunks_sized(path, chunk_bytes)` and yield each `Rope` chunk, keeping `pw.guard` alive until the file's iterator is exhausted (reuse or adapt `scatter.rs`'s `GuardedIter` wrapper — same lifetime problem, same fix).
|
||||
- The existing `process_chunk` transform stage becomes the pipe's **second** stage, unchanged in its own logic — it still receives one `Rope` chunk at a time, just no longer all coming from one serial source.
|
||||
- `make_pipe!` invocation grows from one stage (`Chunk => Output`) to two (`Path => Chunk => Output`).
|
||||
- Log, per file: time spent waiting on the `throttle()` slot (queueing due to `max_open`), and time spent opening/decompressing/producing the first chunk — this is what directly proves (or disproves) that I/O is now spread across workers instead of serialized.
|
||||
- **Validation**: run `query` on a small multi-file input, diff output against the pre-change version — content must be identical; **record order across files is not guaranteed to be preserved** even before this change (chunk-level dispatch across `n_workers` already reorders completions), so the diff must be order-insensitive (sort by read id, or compare as sets) if it wasn't already.
|
||||
- **Measure**: core utilization on the reference machine with several large input files, compare against phase 0's baseline.
|
||||
|
||||
### Phase 2 — Genome-aware chunk-size formula (fixes OOM)
|
||||
|
||||
**Goal**: `chunk_bytes` reflects actual per-chunk memory (`O(n_genomes)`), not a fixed multiplier.
|
||||
|
||||
- `obikmer/src/cmd/query.rs`, `run()`: `n_genomes` and `args.detail` are already computed above the `chunk_bytes` calculation (`n_genomes` at the top of `run()`, before line 407 in the current file) — reorder if needed, then replace:
|
||||
```rust
|
||||
let computed = avail / (n_workers as u64 * 16);
|
||||
```
|
||||
with a formula that scales the divisor by `n_genomes` (and roughly doubles it when `--detail` is set, since `cov` duplicates the per-genome accumulation): e.g. `per_chunk_multiplier = base_overhead + n_genomes as u64 * BYTES_PER_KMER_PER_GENOME * if detail { 2 } else { 1 }`, replacing the flat `16`. `BYTES_PER_KMER_PER_GENOME` should be derived from `KmerResults`'s actual layout (`4` bytes per `u32` entry in `data`, plus the `bool` in `in_index`, plus `win_min`'s equal-sized buffer) rather than guessed.
|
||||
- `args.chunk_size` (manual `--chunk-size` override) keeps taking priority, unchanged.
|
||||
- Log the resolved `chunk_bytes`, `n_genomes`, and the estimated peak per-chunk memory (`chunk_bytes` × the same multiplier used to derive it) once at startup — lets a cluster run confirm the estimate was actually respected, not just that the process didn't get OOM-killed (which could also happen to be true for the wrong reason).
|
||||
- **Validation**: build a test index with a large `n_genomes` (e.g. hundreds), run `query` with default chunk sizing under a memory limit (`ulimit -v` or a cgroup), confirm it no longer gets OOM-killed and that memory scales as predicted when `n_genomes` grows.
|
||||
- **Note**: this phase is superseded once phase 5 lands (sparse retained memory no longer scales with `n_genomes × total_kmers` at all) — but it's needed immediately regardless, since phases 3–5 are a bigger, riskier change and users need a working `query` in the meantime.
|
||||
|
||||
### Phase 3 — K-mer-level dereplication, staged MPHF/matrix lookup
|
||||
|
||||
**Goal**: replace superkmer-level dedup with k-mer-level dedup (roadmap point 5), and split the fused MPHF-find/matrix-fetch (point 4) so stage 1's output is bucketed by layer and MPHF slot (point 6).
|
||||
|
||||
- `obikmer/src/cmd/query.rs`:
|
||||
- Replace `QueryBatch::from_records`'s dedup map (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, current `query.rs:112`) with a per-partition `HashMap<CanonicalKmer, Vec<(seq_idx: u32, pos: u32)>>`, built in the same `SuperKmerIter` pass: superkmer construction and partition routing (`part_idx` from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each `CanonicalKmer` within a superkmer is inserted individually instead of the whole superkmer being the dedup key.
|
||||
- **Verified**: `CanonicalKmer` (`obikseq/src/kmer.rs:390`, `pub type CanonicalKmer = CanonicalKmerOf<KLen>`) — the underlying `CanonicalKmerOf<L>` derives `Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash` (`kmer.rs:269`). Usable as a `HashMap`/`HashSet` key as-is, no change needed.
|
||||
- `obikindex/src/partition/query_layer.rs`:
|
||||
- Split `QueryLayer::find_into` (`query_layer.rs:48-67`) into two methods: `find_slot(&self, kmer: CanonicalKmer) -> Option<usize>` (MPHF only, no matrix touch) and keep `fill_row` as-is for phase 4 to call later.
|
||||
- Replace `query_partition_with`'s inner loop (`query_layer.rs:103-113`) with a version that, for each unique `CanonicalKmer`, calls `find_slot` across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records `(layer_idx, slot)`.
|
||||
- New return shape for the partition-level query, replacing today's `on_hit(sk_idx, kmer_idx, row)` callback: `HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>` (roadmap point 6) — built directly from the k-mer dedup map's `Vec<(seq_idx,pos)>` values, keyed by the resolved slot instead of the k-mer.
|
||||
- **This phase alone has no throughput benefit yet** (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.
|
||||
- Log, per chunk: total k-mer occurrences vs. unique `CanonicalKmer` count (the dedup ratio — the entire justification for this phase) and the resulting MPHF `find` call count. If the dedup ratio is close to `1.0` on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.
|
||||
- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikindex/src/partition/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
|
||||
|
||||
### Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred
|
||||
|
||||
**Goal (revised during implementation)**: replace `fill_row`-per-hit (row-major, worst-case mmap locality) with a column-major scan. `PartitionRunner` turned out to be the wrong mechanism for this at this call granularity — see below; the column-major fetch itself is implemented and validated, without it.
|
||||
|
||||
**What shipped:**
|
||||
- `obicompactvec`: the per-column accessors this phase needed **already existed** — `PersistentCompactIntMatrix::col_view(c)` and `PersistentBitMatrix::col_view(c)` are public, and `IntSliceView::get(slot)`/`BitSliceView::get(slot)` are public — the original plan underestimated how much of this plumbing the pairwise-distance code (`dump`/`select`/`stats`) had already required. The one real gap: `PersistentBitMatrix::col_view()` panics on the `Implicit` variant (the documented mono-genome fast path, `bitmatrix.rs`). Added `PersistentBitMatrix::get(c, slot) -> u32` (`bitmatrix.rs`), a non-panicking column-major point lookup that returns `1` for `Implicit` regardless of `c` — the smallest surface needed, not a new `col_get` API from scratch.
|
||||
- `obikindex/src/partition/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
|
||||
- `obikmer/src/cmd/query.rs`: `KmerResults::set` (row-major, whole-row-at-once) replaced by `mark_found` (stage 1: flag a position as indexed, independent of any genome's value) and `set_one` (stage 2: write one genome's value at one position). `QueryStats` extended with `n_columns_scanned`/`n_col_get_calls`, logged per chunk.
|
||||
- Total `get()`-equivalent calls are unchanged from the row-major version (`n_hits × n_cols` in the worst case, confirmed by `n_col_get_calls` in the debug log) — the win is locality (sequential access within one layer's column at a time, across `mmap`'d regions, instead of jumping across all columns per hit), exactly as predicted.
|
||||
|
||||
**What did not ship, and why — `PartitionRunner` is architecturally the wrong tool here:**
|
||||
Reading `obikindex/src/numa.rs`'s actual `run()` body (not just its doc comments) shows every call spawns a timer thread **plus one OS thread per worker slot on every NUMA node** (`std::thread::scope` + one `s.spawn()` per node per `max_workers`) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned **per call**. This is fine for its actual, established usage in this codebase (`merge.rs`, `index.rs`'s `build_layers`): one `PartitionRunner::new()` + one `run()` call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for `query`'s call pattern: `query_partition_with` runs once per `(chunk, partition)`, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
|
||||
The column-major loop in stage 2 is therefore a **plain sequential loop** for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) `rayon`'s already-warm global pool (`into_par_iter()`) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern `numa_worker_pools.md` warns about for a *different* workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one `mmap`'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so `PartitionRunner` is invoked once per whole `query` run (or per large batch of chunks) rather than per `(chunk, partition)`, amortising its spawn cost the way `merge`/`build_layers` do — a bigger structural change than this phase's scope.
|
||||
- Log (implemented): `QueryStats::n_columns_scanned`/`n_col_get_calls`, folded into the existing per-chunk `debug!("k-mer dedup + column-major fetch", ...)` line (`query.rs`) alongside phase 3's dedup counters.
|
||||
- **Unit tests**: extended `obikindex/src/partition/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
|
||||
- **Validation performed**: full workspace build + `cargo test --workspace`, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same `kmer_count`/`kmer_strict_matches` on every record); (2) the existing 20-genome `benchmark/global_index_presence` index — runs correctly, `n_hits=0` for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the `Implicit`/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) **the critical correctness case**: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from `genomeA` matched **only** `genomeA` (`kmer_count` identical to the pre-dedup occurrence count, zero leakage into `genomeB`'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in `n_cols`, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
|
||||
- **Not yet done**: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-`n_genomes` layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.
|
||||
|
||||
### Phase 5 — Sparse Findere rework (roadmap point 9)
|
||||
|
||||
**Goal**: replace the dense `KmerResults`/`win_min` sliding-window scan with one operating on phase 4's sparse per-genome output.
|
||||
|
||||
- `obikmer/src/cmd/query.rs`, `process_chunk`:
|
||||
- Remove `KmerResults` (`query.rs:157-202`) and the dense `win_min` allocation (`query.rs:290-291`, sized `max_n_kmers × n_genomes`).
|
||||
- Keep a lightweight dense `in_index: Vec<bool>` per chunk (sized `total_kmers`, independent of `n_genomes`) from phase 3's stage 1 — still needed for `kmer_missing` bookkeeping (leftmost-s-mer-of-window membership test), which phase 4's sparse structure doesn't carry (a k-mer with no genome hit has no entry there at all).
|
||||
- New per-`(seq_idx, genome)` scan: for each genome's `Vec<(seq_idx, pos, count)>` (sorted, per phase 4), group by `seq_idx` (contiguous after sort), then within each sequence's positions detect runs of `pos, pos+1, pos+2, ...` of length ≥ `z`; within each run, the existing monotone-deque window-minimum logic (`query.rs`'s current `dq` loop, conceptually unchanged) applies — but the deque now only scans real entries in the run, never zero-filled gaps.
|
||||
- Update `SeqAcc` accumulation and `emit_batch` to consume this per-genome sparse iteration instead of `results.val`/`results.is_in_index`.
|
||||
- `--detail`/`cov`: build sparsely during the same scan (only positions with a confirmed contribution get an entry), densify into the `[u32]` JSON array only in `emit_batch`, only for genomes/sequences actually being serialized (per roadmap point 9's note, `query.rs:304-308`'s current dense allocation goes away).
|
||||
- Log, per chunk: total sparse entries retained vs. what the old dense `KmerResults` would have allocated (`total_smers × n_genomes`) — the sparsity ratio is this phase's entire reason for existing, so it must be directly visible in the logs, not inferred from process RSS. Also log the run-detection stats (number of runs found, average run length) — a low average run length relative to `z` would mean most positions still fail to form a full window, worth knowing.
|
||||
- **Unit tests**: `obikmer/src/cmd/tests/query.rs` (extended from phase 3) — the property test described below is the primary deliverable here, not an afterthought; write it as an actual `#[test]` (or a small internal fuzz/property-style loop over randomized fixtures if a property-testing crate isn't already a dependency — check before adding one, per this project's dependency-approval rule) rather than a one-off manual comparison.
|
||||
- **Validation — this is the correctness-critical phase**: property-test comparing old (dense, pre-phase-3) and new (sparse) implementations on the same randomized input/index fixtures, asserting identical `kmer_count`, `kmer_missing`, `kmer_strict_matches`, and (with `--detail`) `coverage` for every sequence. Keep both implementations compiled side by side (behind a debug-only flag or a temporary parallel code path) only for the duration of this validation; delete the dense path once parity is confirmed — per this project's own convention, superseded code is not kept "just in case."
|
||||
- **Update `docmd/architecture/query.md` itself**: once this phase lands, the "Findere z-window filter" section (which currently — correctly — describes the dense deque-over-`0..n_smers` scan) needs another pass to describe the sparse run-detection algorithm instead, as already flagged when this phase was discussed.
|
||||
|
||||
**Implemented as planned, no deviations discovered this time.** What shipped:
|
||||
- `KmerResults` removed entirely, replaced by `SmerIndex` (`in_index: Vec<bool>` + `offsets`, unchanged size/purpose, renamed since it's no longer "results" — just the O(1)-per-position "was this k-mer found at all" bookkeeping) and `by_genome: Vec<Vec<(seq_idx, pos, value)>>` (one empty `Vec` per genome until a hit arrives — genomes with zero hits in a chunk cost nothing beyond the outer `Vec`'s own allocation).
|
||||
- New `sparse_findere_for_genome(hits, z, presence, threshold) -> (Vec<ConfirmedHit>, n_runs, total_run_len)` (`query.rs`): sorts one genome's raw hits by `(seq_idx, pos)`, detects maximal runs of consecutive `pos` within one sequence, runs the same monotone-deque window-minimum as before but scoped to each run (run-relative indices for eviction, absolute `pos` for computing `pos_out`). Presence/count adjustment (`u32::from(win_min >= threshold)` vs. raw) is applied inside this function, once per confirmed hit, rather than later during accumulation.
|
||||
- `process_chunk` restructured into three passes after the partition loop: (1) run `sparse_findere_for_genome` per genome, collecting `confirmed_by_genome` and run-detection stats; (2) accumulate `genome_totals` directly from `confirmed_by_genome` and mark a `confirmed_any: Vec<bool>` (sized `total_kmers_out`, not `× n_genomes`); (3) a position-only pass (`O(total_kmers_out)`, no genome factor) computing `kmer_count`/`kmer_missing` from `confirmed_any` + `SmerIndex`. `cov` (`--detail`) is populated by re-scanning `confirmed_by_genome` — only when `--detail` is actually set, otherwise skipped entirely.
|
||||
- Debug log added (`"sparse Findere"`): `n_dense_would_be` (`n_occurrences × n_genomes` — what the deleted dense path would have allocated), `n_sparse_entries` (what's actually retained), `n_runs`/`avg_run_len` (per the plan's ask, to see whether hits mostly fail to form complete windows).
|
||||
- **Unit tests**: `sparse_findere_matches_dense_reference_on_random_inputs` (`obikmer/src/cmd/tests/query.rs`) — 200 randomized cases (sequence count/length, `z`, presence/count mode, threshold, hit density from sparse to fully-dense) comparing `sparse_findere_for_genome` against `dense_reference_findere`, a faithful reimplementation of the deleted dense algorithm kept only as a test-local correctness oracle (no property-testing crate added — checked first, none was a workspace dependency; a small `std`-only xorshift64 PRNG stands in for one, deterministic and dependency-free). All 200 cases pass.
|
||||
- **Functional validation performed**: full workspace build + `cargo test --workspace`, zero failures. End-to-end against real indexes: baseline output (no flags) unchanged from pre-phase-5 recorded values on the same fixtures; `--count-missing` correct (`kmer_missing: 0` on a self-match); `--detail` correct — coverage array length matches `kmer_count`, and critically, re-ran the two-genome cross-contamination check from phase 4 with `--detail --count-missing`: `genomeA` reads show coverage sum `106` for `genomeA` and `0` for `genomeB` (and vice versa) — confirms the sparse-to-dense `cov` reconstruction doesn't leak across genomes either, not just the scalar `kmer_strict_matches` path.
|
||||
- This phase's roadmap item ("update the Findere z-window filter section") — done, see above; the "Algorithm" section's pseudocode was also updated, since it still named `KmerResults`/`SKDesc` from before phases 3–4.
|
||||
|
||||
### Phase 6 — Parallel gzip decompression (independent, optional)
|
||||
|
||||
Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen); parked pending validation of `rapidgzip-rs` on real data. Not a dependency of, or a dependency for, phases 0–5 — `xopen` is shared infrastructure (`obiread`), phase 1 benefits from it but doesn't require it (phase 1 parallelises *across* files; this phase would additionally parallelise *within* one large file).
|
||||
|
||||
### Cross-cutting risks
|
||||
|
||||
- **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone.
|
||||
- **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition.
|
||||
- **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,230 +0,0 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" demote-non-dropping-particle="sort-only" default-locale="en-US">
|
||||
<info>
|
||||
<title>Ecology Letters</title>
|
||||
<id>http://www.zotero.org/styles/ecology-letters</id>
|
||||
<link href="http://www.zotero.org/styles/ecology-letters" rel="self"/>
|
||||
<link href="http://www.zotero.org/styles/apa" rel="template"/>
|
||||
<link href="http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291461-0248/homepage/ForAuthors.html" rel="documentation"/>
|
||||
<author>
|
||||
<name>David Kaplan</name>
|
||||
<email>david.kaplan@ird.fr</email>
|
||||
</author>
|
||||
<contributor>
|
||||
<name>Sebastian Karcher</name>
|
||||
</contributor>
|
||||
<category citation-format="author-date"/>
|
||||
<category field="biology"/>
|
||||
<issn>1461-023X</issn>
|
||||
<eissn>1461-0248</eissn>
|
||||
<updated>2023-10-11T10:45:32+00:00</updated>
|
||||
<rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
|
||||
</info>
|
||||
<macro name="container">
|
||||
<choose>
|
||||
<if type="chapter paper-conference" match="any">
|
||||
<text term="in" text-case="capitalize-first" suffix=": "/>
|
||||
<text variable="container-title" font-style="italic"/>
|
||||
<text variable="collection-title" prefix=", "/>
|
||||
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
|
||||
<label form="short" suffix=" "/>
|
||||
<name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
|
||||
</names>
|
||||
</if>
|
||||
<else>
|
||||
<group delimiter=", ">
|
||||
<text variable="container-title" font-style="italic" form="short"/>
|
||||
<text variable="collection-title"/>
|
||||
</group>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="author">
|
||||
<names variable="author">
|
||||
<name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
|
||||
<label form="short" prefix=" (" suffix=")" text-case="capitalize-first"/>
|
||||
<et-al font-style="italic"/>
|
||||
<substitute>
|
||||
<names variable="editor"/>
|
||||
<names variable="translator"/>
|
||||
<text macro="title"/>
|
||||
</substitute>
|
||||
</names>
|
||||
</macro>
|
||||
<macro name="author-short">
|
||||
<names variable="author">
|
||||
<name form="short" and="symbol" delimiter=", " initialize-with=". "/>
|
||||
<et-al font-style="italic"/>
|
||||
<substitute>
|
||||
<names variable="editor"/>
|
||||
<names variable="translator"/>
|
||||
<choose>
|
||||
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||
<text variable="title" form="short" font-style="italic"/>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="title" form="short" quotes="true"/>
|
||||
</else>
|
||||
</choose>
|
||||
</substitute>
|
||||
</names>
|
||||
</macro>
|
||||
<macro name="access">
|
||||
<choose>
|
||||
<if type="webpage">
|
||||
<group>
|
||||
<text term="available at" text-case="capitalize-first" suffix=": "/>
|
||||
<text variable="URL" suffix="."/>
|
||||
</group>
|
||||
<text value="Last accessed" prefix=" " suffix=" "/>
|
||||
<date variable="accessed">
|
||||
<date-part name="day" suffix=" "/>
|
||||
<date-part name="month" suffix=" "/>
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="title">
|
||||
<choose>
|
||||
<if type="report" match="any">
|
||||
<text variable="title" font-style="italic"/>
|
||||
<group prefix=" (" suffix=")">
|
||||
<text variable="genre"/>
|
||||
<text variable="number" prefix=" No. "/>
|
||||
</group>
|
||||
</if>
|
||||
<else-if type="bill book graphic legal_case legislation motion_picture report song speech" match="any">
|
||||
<text variable="title" font-style="italic"/>
|
||||
</else-if>
|
||||
<else-if type="webpage">
|
||||
<text variable="title" font-style="italic"/>
|
||||
</else-if>
|
||||
<else>
|
||||
<text variable="title"/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="publisher">
|
||||
<choose>
|
||||
<if type="report" match="any">
|
||||
<group delimiter=", ">
|
||||
<text variable="publisher"/>
|
||||
<text variable="publisher-place"/>
|
||||
</group>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="genre" suffix=". "/>
|
||||
<group delimiter=", ">
|
||||
<text variable="publisher"/>
|
||||
<text variable="publisher-place"/>
|
||||
</group>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="event">
|
||||
<choose>
|
||||
<if variable="event">
|
||||
<text term="presented at" text-case="capitalize-first" suffix=" "/>
|
||||
<text variable="event"/>
|
||||
</if>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="issued">
|
||||
<choose>
|
||||
<if variable="issued">
|
||||
<date variable="issued">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
<else-if variable="accessed">
|
||||
<choose>
|
||||
<if type="webpage">
|
||||
<date variable="accessed">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
<else>
|
||||
<text term="no date" form="short"/>
|
||||
</else>
|
||||
</choose>
|
||||
</else-if>
|
||||
<else>
|
||||
<text term="no date" form="short"/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="edition">
|
||||
<choose>
|
||||
<if is-numeric="edition">
|
||||
<group delimiter=" ">
|
||||
<number variable="edition" form="ordinal"/>
|
||||
<text value="edn"/>
|
||||
</group>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="edition" suffix="."/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="locators">
|
||||
<choose>
|
||||
<if type="article-journal article-magazine article-newspaper" match="any">
|
||||
<group prefix=", " delimiter=", ">
|
||||
<group>
|
||||
<text variable="volume"/>
|
||||
</group>
|
||||
<text variable="page"/>
|
||||
</group>
|
||||
</if>
|
||||
<else-if type="bill book graphic legal_case legislation motion_picture report song thesis" match="any">
|
||||
<group delimiter=". " prefix=". ">
|
||||
<text macro="edition"/>
|
||||
<text macro="event"/>
|
||||
<text macro="publisher"/>
|
||||
</group>
|
||||
</else-if>
|
||||
<else-if type="chapter paper-conference" match="any">
|
||||
<group delimiter=", " prefix=". ">
|
||||
<text macro="event"/>
|
||||
<text macro="publisher"/>
|
||||
<group>
|
||||
<label variable="page" form="short" suffix=" "/>
|
||||
<text variable="page"/>
|
||||
</group>
|
||||
</group>
|
||||
</else-if>
|
||||
</choose>
|
||||
</macro>
|
||||
<citation et-al-min="3" et-al-use-first="1" disambiguate-add-year-suffix="true" collapse="year-suffix" year-suffix-delimiter=", ">
|
||||
<sort>
|
||||
<key macro="author"/>
|
||||
<key macro="issued"/>
|
||||
</sort>
|
||||
<layout prefix="(" suffix=")" delimiter="; ">
|
||||
<group delimiter=" ">
|
||||
<text macro="author-short"/>
|
||||
<text macro="issued"/>
|
||||
</group>
|
||||
</layout>
|
||||
</citation>
|
||||
<bibliography et-al-min="7" et-al-use-first="6" entry-spacing="0" hanging-indent="true">
|
||||
<sort>
|
||||
<key macro="author"/>
|
||||
<key macro="issued" sort="ascending"/>
|
||||
<key macro="title"/>
|
||||
</sort>
|
||||
<layout>
|
||||
<group suffix=".">
|
||||
<text macro="author" suffix="."/>
|
||||
<text macro="issued" prefix=" (" suffix="). "/>
|
||||
<group delimiter=". ">
|
||||
<text macro="title"/>
|
||||
<text macro="container"/>
|
||||
</group>
|
||||
<text macro="locators"/>
|
||||
<text macro="access" prefix=". "/>
|
||||
</group>
|
||||
</layout>
|
||||
</bibliography>
|
||||
</style>
|
||||
@@ -1,93 +0,0 @@
|
||||
# Benchmark: query-path testing
|
||||
|
||||
`benchmark/Makefile` exercises indexing, merge, and phylo distance
|
||||
reconstruction against simulated bacterial genomes. It now also covers
|
||||
`obikmer query` — the read-matching path — and the sparse packed
|
||||
presence-matrix format (`obikmer pack --sparse`), previously untested by
|
||||
this pipeline.
|
||||
|
||||
## Motivation
|
||||
|
||||
- `query` had no end-to-end coverage. A regression there would not be caught
|
||||
by `verify_presence`/`verify_merge_presence`, which only check index
|
||||
*content* against the `.npz` truth, never the query API.
|
||||
- `pack --sparse` produces a presence-matrix format documented (see
|
||||
[siblings.md](../architecture/siblings.md)) as faster for single-row
|
||||
access (query) and slower for column-oriented access (phylo `--metric`).
|
||||
`global_index_presence/` built by `merge_presence.sh` is always packed
|
||||
dense (packing is a stage inside `merge`, not a separate `pack`
|
||||
invocation) — there was no dense/sparse regression check.
|
||||
|
||||
## Query read source
|
||||
|
||||
Query reads are independent of `simulated_data/` (which is folded into the
|
||||
index being queried): reusing those reads would test against the exact
|
||||
error draw the index was built from. `query_data/<species>/<strain>/` holds
|
||||
a *second*, independent `iss generate` run against the same reference
|
||||
genome, via `simulate_query_one.sh` — unseeded, so a second draw picks up
|
||||
different sequencing errors than `simulate_one.sh`'s draw for the same
|
||||
genome. Fixed at 100,000 read pairs per genome (not coverage-proportional
|
||||
like the 15x used for `simulated_data/`), so wall/RSS numbers stay
|
||||
comparable across genomes of very different sizes.
|
||||
|
||||
Two query-source specimens, hardcoded as `QUERY_SPECIMENS` in
|
||||
`make_deps.py`: `Escherichia_coli--K-12_MG1655` (common, well-represented
|
||||
bacterium) and `Saccharolobus_islandicus--M.16.4` (the only archaeon in
|
||||
`SPECIES` — distant lineage, stresses the query path differently from a
|
||||
close-relative match). Two is enough to catch a dense/sparse regression
|
||||
without duplicating the exhaustive per-specimen coverage
|
||||
`verify_merge_presence` already provides across all `SPECIMENS`.
|
||||
|
||||
## Sparse global index
|
||||
|
||||
`global_index_presence_sparse/` is built by `pack_sparse.sh`: copy
|
||||
`global_index_presence/` wholesale, then `obikmer pack --sparse` in place.
|
||||
This works directly because `merge`'s pack stage (`merge.rs:252`,
|
||||
`pack_matrices(false)`) keeps the per-genome column files on disk after
|
||||
dense-packing — `pack_sparse_bit_matrix` (`obicompactvec/src/bitmatrix/sparse.rs:447`)
|
||||
reads those, is idempotent, and removes `matrix.pbmx` once the sparse form
|
||||
is written, so `Persistent::open` falls through to the sparse format
|
||||
afterward. No separate merge run needed.
|
||||
|
||||
## Query runs
|
||||
|
||||
`query_one.sh dense|sparse SPECIMEN` runs `obikmer query --count-missing`
|
||||
against `global_index_presence` or `global_index_presence_sparse`, output
|
||||
gzipped to `query_{dense,sparse}/SPECIMEN.fasta.gz`, Reporter wall/RSS
|
||||
captured to `stats/query_{dense,sparse}/SPECIMEN.stats` (same
|
||||
stderr-parsing convention as `merge_presence.sh`).
|
||||
|
||||
Flags: `--count-missing` only. `--mismatch` is a no-op today
|
||||
(`query/mod.rs:212-213`, prints "not yet implemented, ignored") — left off
|
||||
rather than tested for a feature that doesn't exist yet.
|
||||
|
||||
## Dense/sparse regression
|
||||
|
||||
`verify_query.py` compares the two query outputs per specimen, matched by
|
||||
read id (not stream position — the query pipeline chunks input across
|
||||
worker threads and doesn't guarantee output order). Compares `kmer_count`,
|
||||
`kmer_missing`, and the full `kmer_strict_matches` map per read. Any
|
||||
mismatch is a real regression: dense and sparse must be content-identical,
|
||||
only I/O access pattern differs. `.stats` → `stats/verify_query/`,
|
||||
aggregated by `aggregate_stats.sh query|verify_query`-style cases
|
||||
(`query_dense`, `query_sparse`, `verify_query`).
|
||||
|
||||
## Performance comparison
|
||||
|
||||
No dedicated script: the wall/RSS columns from the `query_dense` and
|
||||
`query_sparse` aggregated `.stats` CSVs are the dense-vs-sparse performance
|
||||
comparison — the expected win for query on sparse, per the `pack --sparse`
|
||||
help text.
|
||||
|
||||
## Scope
|
||||
|
||||
`count` track excluded from the sparse branch: `pack --sparse` targets
|
||||
presence matrices only (per CLI help); `pack_matrices` leaves count
|
||||
matrices untouched regardless of the `sparse` flag
|
||||
(`obikindex/src/index.rs:308`).
|
||||
|
||||
## New Makefile targets
|
||||
|
||||
`simulate_query`, `pack_sparse`, `query_dense`, `query_sparse`,
|
||||
`aggregate_query_dense`, `aggregate_query_sparse`, `verify_query`,
|
||||
`aggregate_verify_query` — the last three folded into `all`.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,84 +0,0 @@
|
||||
# Kmer entropy filter
|
||||
|
||||
Low-complexity kmers (polyA, polyT, tandem repeats) are detected and excluded during phase 1. The filter computes a **normalized Shannon entropy** over sub-words of multiple sizes, corrected for one source of bias: the small number of observations within a single kmer relative to the number of possible sub-words.
|
||||
|
||||
## Sub-word frequencies
|
||||
|
||||
For a kmer of length k and a sub-word size ws (1 ≤ ws ≤ ws_max, typically ws_max = 6), extract the $n_{\text{words}} = k - ws + 1$ overlapping sub-words by sliding a window of length ws:
|
||||
|
||||
$$w_i = \text{kmer}[i \mathinner{..} i+ws-1], \quad i = 0, \ldots, n_{\text{words}}-1$$
|
||||
|
||||
Each sub-word is tallied under its own raw 2-bit-packed value — **no canonicalization**. Let $f_j$ be the count of raw word $j$ among the $n_{\text{words}}$ sub-words ($\sum_j f_j = n_{\text{words}}$), over the $4^{ws}$ possible raw words.
|
||||
|
||||
An earlier version of this filter first folded each sub-word into a circular+reverse-complement equivalence class, then "unfolded" the observed class frequency back onto its members to correct for unequal class sizes. That machinery bought nothing it was claimed for — see *Why no equivalence classes* below — while measurably weakening detection of the very sequences the filter exists to catch, so it was removed.
|
||||
|
||||
## Corrected Shannon entropy
|
||||
|
||||
$$H_{\text{corr}} = \log(n_{\text{words}}) - \frac{1}{n_{\text{words}}} \sum_j f_j \log f_j$$
|
||||
|
||||
This is a plain Shannon entropy over the observed raw-word frequencies.
|
||||
|
||||
## Maximum entropy correction for small samples
|
||||
|
||||
With only $n_{\text{words}}$ observations over $4^{ws}$ possible raw words, the achievable maximum entropy is bounded by the most uniform integer distribution over $4^{ws}$ categories.
|
||||
|
||||
Let $c = \lfloor n_{\text{words}} / 4^{ws} \rfloor$ and $r = n_{\text{words}} \bmod 4^{ws}$. The most uniform integer distribution assigns frequency $c+1$ to $r$ categories and $c$ to the remaining $4^{ws} - r$, with the convention $0 \log 0 = 0$:
|
||||
|
||||
$$H_{\max} = -\left[(4^{ws} - r)\,\frac{c}{n_{\text{words}}}\log\frac{c}{n_{\text{words}}} + r\,\frac{c+1}{n_{\text{words}}}\log\frac{c+1}{n_{\text{words}}}\right]$$
|
||||
|
||||
When $n_{\text{words}} < 4^{ws}$: $c=0$, $r=n_{\text{words}}$, and the formula reduces to $H_{\max} = \log(n_{\text{words}})$ — a single unified expression covers both regimes. A truly random sequence achieves $H_{\text{corr}} \approx H_{\max}$.
|
||||
|
||||
## Normalized entropy
|
||||
|
||||
$$\hat{H}(ws) = \frac{H_{\text{corr}}}{H_{\max}} \in [0, 1]$$
|
||||
|
||||
## Final score
|
||||
|
||||
The filter computes $\hat{H}(ws)$ for each word size ws from 1 to ws_max and returns the **minimum**:
|
||||
|
||||
$$\text{entropy}(kmer) = \min_{ws=1}^{ws_{\max}} \hat{H}(ws)$$
|
||||
|
||||
A value near 0 indicates low complexity (e.g. AAAA…); near 1 indicates high complexity. A kmer is rejected if $\text{entropy}(kmer) < \theta$, where $\theta$ is a collection parameter (default 0.7). The minimum across word sizes ensures that any scale of repetition is detected independently: polyA is caught at ws=1, dinucleotide repeats at ws=2, etc.
|
||||
|
||||
## Why no equivalence classes
|
||||
|
||||
A prior design folded each sub-word into the canonical form of its circular-rotation + reverse-complement equivalence class before tallying, on the reasoning that (a) it guarantees $\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))$, and (b) collapsing phase-shifted repeats (e.g. `ATG` ≡ `TGA` ≡ `GAT`) into one class better reflects that they are "the same" low-complexity pattern.
|
||||
|
||||
Both properties already hold for the raw, unfolded entropy above, without any class machinery:
|
||||
|
||||
- **Reverse complement**: for any K of length n, window $j$ of $\text{revcomp}(K)$ equals $\text{revcomp}$ of window $(n{-}ws{-}j)$ of K. This is a bijection between the window sets under which each window maps to its own revcomp — and revcomp is itself a bijection (involution) on the space of raw ws-mers. So the multiset of raw-word frequencies for $\text{revcomp}(K)$ is exactly a relabeling of the multiset for K, and Shannon entropy — a function of the frequency multiset alone — is exactly invariant. No folding required, for any K.
|
||||
- **Tandem repeats**: a period-p repeat sampled by a stride-1 sliding window naturally cycles through its own rotations as raw tokens (e.g. `ATGATGATG…` yields the raw words `ATG`, `TGA`, `GAT` in rotation as the window slides). The low diversity this represents (few distinct raw words out of $4^{ws}$ possible) is already visible in the raw frequency distribution — no folding needed to detect it.
|
||||
|
||||
What the fold-then-unfold step actually did was credit each observed class with the frequency of equivalence-class members that were **never observed on the read strand**, inflating $H_{\text{corr}}$ for genuine repeats. Worked example: k=31, ws=3, kmer = `ATG` repeated ($n_{\text{words}}=29$, all 29 windows fall into one class of size 6 under the old scheme — 3 rotations × forward/revcomp):
|
||||
|
||||
| | $H_{\text{corr}}$ | normalized |
|
||||
|---|---|---|
|
||||
| old (folded, class size 6) | $\log 6 \approx 1.79$ | $\approx 0.53$ |
|
||||
| current (raw, unfolded) | $\log 3 \approx 1.10$ | $\approx 0.33$ |
|
||||
|
||||
The gap is not a rounding artifact: per sub-word order, the folded score for this same repeat swings from 0.53 (ws=3, aligned with the period) up to **1.03** (ws=5, misaligned with the period) — i.e. a period-3 repeat could score *above* the theoretical maximum for a random sequence, depending on which ws happens to divide the repeat's period. The raw formula stays flat at ≈0.33–0.40 across ws=2..6 regardless of alignment, which is the robustness the "minimum across ws" design was meant to provide in the first place.
|
||||
|
||||
## Interpretation as an effective number of classes
|
||||
|
||||
$H_{\text{corr}}$ is a standard Shannon entropy over raw words, so the classical perplexity interpretation holds directly: $N_{\text{eff}} = e^{H_{\text{corr}}}$ is the number of equiprobable raw words that would yield the same entropy.
|
||||
|
||||
For the normalised score $\hat{H}$, dividing by $H_{\max}$ changes the logarithm base:
|
||||
|
||||
$$\hat{H} = \frac{\log N_{\text{eff}}}{\log N_{\max}} = \log_{N_{\max}} N_{\text{eff}} \quad \Longleftrightarrow \quad N_{\text{eff}} = N_{\max}^{\,\hat{H}}$$
|
||||
|
||||
The property is preserved: $\hat{H}$ is the logarithm (in base $N_{\max}$) of the effective number of equi-represented raw words.
|
||||
|
||||
In the large-sample limit ($n_{\text{words}} \gg 4^{ws}$), $N_{\max} \approx 4^{ws}$, giving:
|
||||
|
||||
$$N_{\text{eff}} \approx 4^{ws \cdot \hat{H}}$$
|
||||
|
||||
This has a clean interpretation: $ws \cdot \hat{H}$ is the **effective word length** (in bases) of a perfectly uniform distribution that would produce the same entropy. At $\hat{H} = 1$ the full space of $4^{ws}$ words is used; at $\hat{H} = 0.5$ with ws=2, only $4^1 = 4$ effective words out of 16 are occupied.
|
||||
|
||||
In our actual regime, $n_{\text{words}}$ is small and $4^{ws}$ can exceed $n_{\text{words}}$, so $H_{\max} < \log(4^{ws})$ due to the small-sample correction. The exact effective count is $N_{\max}^{\hat{H}}$, not $4^{ws \cdot \hat{H}}$.
|
||||
|
||||
## Properties
|
||||
|
||||
The entropy score is a function of the kmer sequence alone — it does not depend on the surrounding context or on the position within any genome. Two consequences:
|
||||
|
||||
- **Orientation invariance**: $\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))$ — see *Why no equivalence classes* above for why this holds without any explicit strand-folding step.
|
||||
- **Context independence**: the same kmer is always rejected or always kept, regardless of which genome it occurs in, where in that genome it appears, or which strand is considered. The filter defines a fixed partition of the kmer space into low-complexity and valid kmers.
|
||||
@@ -1,16 +0,0 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: theory/entropy.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikentropy/src/table.rs`, `obikentropy/src/tracker.rs` — formule d'entropie et tables de correction petits effectifs
|
||||
- `obikentropy/src/kmer_entropy.rs` — entropie d'un kmer isolé (`KmerEntropy`)
|
||||
- `obiskbuilder/src/rolling_stat.rs` — composition de `obikentropy::EntropyTracker` dans le suivi streaming (sélection de minimiseur + entropie)
|
||||
- `obiskbuilder/src/iter.rs`, `obiskbuilder/src/stream_iter.rs` — application du filtre lors du scatter (phase 1)
|
||||
|
||||
## Notes
|
||||
|
||||
Le repli en classes d'équivalence circulaires + brin inverse (décrit dans une version antérieure de ce document) a été supprimé : voir la section « Why no equivalence classes » de `entropy.md` pour la justification théorique et numérique.
|
||||
|
||||
Vérifier que les paramètres `theta` et `level_max` dans le CLI
|
||||
(`obikmer/src/cli.rs` → `CommonArgs`) correspondent bien à ce qui est décrit.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -5,16 +5,11 @@ MKDOCS := $(VENV)/bin/mkdocs
|
||||
|
||||
CARGO_DIR := src
|
||||
|
||||
DOC_DIR := DevDocMD
|
||||
DOC_DIR := docmd
|
||||
DOC_FILE := mkdocs.yml
|
||||
DOC_SITE := DevDoc
|
||||
DOC_SITE := doc
|
||||
DOC_PORT := 8001
|
||||
|
||||
DOC_USER_DIR := UserDocMD
|
||||
DOC_USER_FILE := mkdocs-user.yml
|
||||
DOC_USER_SITE := doc
|
||||
DOC_USER_PORT := 8002
|
||||
|
||||
# ── virtualenv ────────────────────────────────────────────────────────────────
|
||||
|
||||
$(VENV)/bin/activate:
|
||||
@@ -65,22 +60,8 @@ doc-serve: $(MKDOCS)
|
||||
clean-doc:
|
||||
rm -rf $(DOC_SITE)/
|
||||
|
||||
.PHONY: doc-user
|
||||
doc-user: $(MKDOCS)
|
||||
$(MKDOCS) build -f $(DOC_USER_FILE)
|
||||
|
||||
.PHONY: doc-user-serve
|
||||
doc-user-serve: $(MKDOCS)
|
||||
$(MKDOCS) serve -f $(DOC_USER_FILE) \
|
||||
--dev-addr=127.0.0.1:$(DOC_USER_PORT) \
|
||||
--livereload
|
||||
|
||||
.PHONY: clean-doc-user
|
||||
clean-doc-user:
|
||||
rm -rf $(DOC_USER_SITE)/
|
||||
|
||||
.PHONY: clean
|
||||
clean: clean-doc clean-doc-user
|
||||
clean: clean-doc
|
||||
rm -rf $(VENV)
|
||||
|
||||
# ── release ───────────────────────────────────────────────────────────────────
|
||||
@@ -105,13 +86,9 @@ bump-version:
|
||||
|
||||
.PHONY: release
|
||||
release: bump-version
|
||||
@jj auto-doc
|
||||
@jj auto-describe
|
||||
@jj git push --change @
|
||||
@new_version=$$(grep '^version = ' $(CARGO_TOML) | head -n 1 | sed 's/version = "\(.*\)"/\1/'); \
|
||||
git_hash=$$(jj log -r @ --no-graph -T 'commit_id'); \
|
||||
commits=$$(jj log -r 'latest(tags())..@' --no-graph -T 'description ++ "\n"' 2>/dev/null || \
|
||||
jj log --no-graph -T 'description ++ "\n"' --limit 30); \
|
||||
notes=$$(printf 'Write concise markdown release notes for obikmer (a Rust kmer genomics tool). Be technical and direct. Base them strictly on these commit messages:\n\n%s' "$$commits" | aichat 2>/dev/null); \
|
||||
tag_msg="$${notes:-Release v$$new_version}"; \
|
||||
git tag -a "v$$new_version" -m "$$tag_msg" "$$git_hash" && \
|
||||
git tag "v$$new_version" "$$git_hash" && \
|
||||
git push origin "v$$new_version"
|
||||
|
||||
@@ -66,11 +66,9 @@ Non-ACGT characters act as hard breaks between k-mer segments in all formats.
|
||||
Annotates each sequence with per-genome k-mer match counts
|
||||
and optional per-position coverage vectors (--detail).
|
||||
Parallel over sequence chunks.
|
||||
phylo Compute pairwise evolutionary-distance proxies between all
|
||||
indexed genomes (Bray-Curtis, Jaccard, etc.), optionally
|
||||
a Newick NJ/UPGMA tree, and optionally a central-position
|
||||
SNP/Sankoff calibration with exports for external
|
||||
phylogenetic tools (TNT, PhyG, IQ-TREE).
|
||||
distance Compute a pairwise Bray-Curtis or Jaccard distance matrix
|
||||
between all indexed genomes.
|
||||
Optionally outputs a Newick NJ or UPGMA tree.
|
||||
annotate Add or update genome metadata (taxonomy, etc.) from a CSV
|
||||
file; or dump the current metadata as CSV.
|
||||
estimate Dry-run: resolve and print approximate-index parameters
|
||||
@@ -108,7 +106,7 @@ obikmer reindex --approx -z 5 --evidence-bits 8 index/
|
||||
obikmer query index/ reads.fq.gz > annotated.fa
|
||||
|
||||
# Pairwise distances
|
||||
obikmer phylo index/ > distances.tsv
|
||||
obikmer distance index/ > distances.tsv
|
||||
```
|
||||
|
||||
## Parameter constraints
|
||||
@@ -121,6 +119,5 @@ obikmer phylo index/ > distances.tsv
|
||||
|
||||
## Documentation
|
||||
|
||||
Extended architecture and implementation notes are in `DevDocMD/`. Build with
|
||||
`make doc` (requires Python + MkDocs Material). User-facing documentation is
|
||||
in `UserDocMD/`, built with `make doc-user`.
|
||||
Extended architecture and implementation notes are in `docmd/`. Build with
|
||||
`make doc` (requires Python + MkDocs Material).
|
||||
|
||||
@@ -1,33 +0,0 @@
|
||||
# Architecture notes for advanced use
|
||||
|
||||
This page describes execution-level behavior relevant to sizing and running `obikmer` on large datasets or multi-socket machines. It complements the [index format](formats/index_layout.md) and [theory](theory/indexing_architecture.md) pages.
|
||||
|
||||
## Sequence invariant
|
||||
|
||||
Every input sequence is treated purely as a compact representation of a set of overlapping kmers:
|
||||
|
||||
- Only the `A`/`C`/`G`/`T` alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.
|
||||
- Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.
|
||||
- Kmers are always handled in canonical form (see [DNA encoding](theory/encoding.md)), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.
|
||||
|
||||
## Index dimensioning
|
||||
|
||||
An index directory is organized as `KmerIndex → partitions → layers`, with a canonical kmer belonging to exactly one (partition, layer) pair. This is what makes set operations (merge, filter, distance) parallel and coordination-free across partitions.
|
||||
|
||||
- **Partition count** (`-p`/`--partitions`, rounded up to a power of 2) is the main dimensioning knob: more partitions means more independent parallel units and a smaller working set per partition, at the cost of more open files during construction.
|
||||
- **Layers** accumulate as an index grows through successive merges; per-partition query cost grows with the number of layers (worst case linear, expected constant since most kmer lookups resolve in the first layer they could plausibly be in).
|
||||
- Genome columns (count or presence data) are kept at a consistent width across every layer and partition after a merge, which is what allows whole-index aggregate distances (Jaccard, Bray-Curtis, Euclidean, Hellinger, …) to be computed as a two-pass cascade (local partial sums per partition, then a global combination) with no double counting.
|
||||
|
||||
## Parallel execution and NUMA awareness
|
||||
|
||||
Partition-level work (index construction, `merge`, `filter`, `convert`, `select`, `phylo`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
|
||||
|
||||
- On a multi-socket / multi-NUMA-node machine, one thread pool is pinned per NUMA node, and each partition is processed entirely by threads pinned to one node — keeping the memory a partition touches local to that node's DRAM. This matters because touching kmer data across NUMA nodes without pinning can degrade throughput by an order of magnitude or more on large multi-socket machines.
|
||||
- On a single-socket machine, Apple Silicon, or if hwloc cannot report NUMA topology, all cores are treated as one node with no pinning and negligible overhead — this is the default behavior on macOS.
|
||||
- Within a node, the number of active worker threads ramps up progressively rather than being fixed up front: it starts conservatively and grows in steps, but only as long as measured CPU efficiency or disk I/O throughput keeps improving. If neither improves after a growth step, the runner stops adding workers — avoiding oversubscription on stages that are memory-bandwidth-bound rather than CPU- or I/O-bound. Ramp speed scales with the number of cores per node, so a single-node machine ramps just as fast as a large multi-node one.
|
||||
|
||||
No CLI flag controls this directly; it is fully automatic at runtime. NUMA-aware pinning can be compiled out (Cargo feature `numa`, on by default), in which case a plain global thread pool is used instead.
|
||||
|
||||
## Kmer filtering (`filter`)
|
||||
|
||||
[`filter`](usage/filter.md) evaluates predicates against the genome metadata matrix directly whenever every active filter can be expressed as a column-level test (e.g. "any outgroup column non-zero"), producing a per-slot keep/drop decision without touching kmer sequence data at all. If any active filter cannot be expressed this way, evaluation falls back to a per-kmer, row-level check. Either way, the result is always written as a single, freshly compacted layer (`unitigs.bin` and the MPHF are rebuilt from the surviving kmers), never as an additional layer on top of the source index.
|
||||
@@ -1,230 +0,0 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" demote-non-dropping-particle="sort-only" default-locale="en-US">
|
||||
<info>
|
||||
<title>Ecology Letters</title>
|
||||
<id>http://www.zotero.org/styles/ecology-letters</id>
|
||||
<link href="http://www.zotero.org/styles/ecology-letters" rel="self"/>
|
||||
<link href="http://www.zotero.org/styles/apa" rel="template"/>
|
||||
<link href="http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291461-0248/homepage/ForAuthors.html" rel="documentation"/>
|
||||
<author>
|
||||
<name>David Kaplan</name>
|
||||
<email>david.kaplan@ird.fr</email>
|
||||
</author>
|
||||
<contributor>
|
||||
<name>Sebastian Karcher</name>
|
||||
</contributor>
|
||||
<category citation-format="author-date"/>
|
||||
<category field="biology"/>
|
||||
<issn>1461-023X</issn>
|
||||
<eissn>1461-0248</eissn>
|
||||
<updated>2023-10-11T10:45:32+00:00</updated>
|
||||
<rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
|
||||
</info>
|
||||
<macro name="container">
|
||||
<choose>
|
||||
<if type="chapter paper-conference" match="any">
|
||||
<text term="in" text-case="capitalize-first" suffix=": "/>
|
||||
<text variable="container-title" font-style="italic"/>
|
||||
<text variable="collection-title" prefix=", "/>
|
||||
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
|
||||
<label form="short" suffix=" "/>
|
||||
<name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
|
||||
</names>
|
||||
</if>
|
||||
<else>
|
||||
<group delimiter=", ">
|
||||
<text variable="container-title" font-style="italic" form="short"/>
|
||||
<text variable="collection-title"/>
|
||||
</group>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="author">
|
||||
<names variable="author">
|
||||
<name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
|
||||
<label form="short" prefix=" (" suffix=")" text-case="capitalize-first"/>
|
||||
<et-al font-style="italic"/>
|
||||
<substitute>
|
||||
<names variable="editor"/>
|
||||
<names variable="translator"/>
|
||||
<text macro="title"/>
|
||||
</substitute>
|
||||
</names>
|
||||
</macro>
|
||||
<macro name="author-short">
|
||||
<names variable="author">
|
||||
<name form="short" and="symbol" delimiter=", " initialize-with=". "/>
|
||||
<et-al font-style="italic"/>
|
||||
<substitute>
|
||||
<names variable="editor"/>
|
||||
<names variable="translator"/>
|
||||
<choose>
|
||||
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||
<text variable="title" form="short" font-style="italic"/>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="title" form="short" quotes="true"/>
|
||||
</else>
|
||||
</choose>
|
||||
</substitute>
|
||||
</names>
|
||||
</macro>
|
||||
<macro name="access">
|
||||
<choose>
|
||||
<if type="webpage">
|
||||
<group>
|
||||
<text term="available at" text-case="capitalize-first" suffix=": "/>
|
||||
<text variable="URL" suffix="."/>
|
||||
</group>
|
||||
<text value="Last accessed" prefix=" " suffix=" "/>
|
||||
<date variable="accessed">
|
||||
<date-part name="day" suffix=" "/>
|
||||
<date-part name="month" suffix=" "/>
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="title">
|
||||
<choose>
|
||||
<if type="report" match="any">
|
||||
<text variable="title" font-style="italic"/>
|
||||
<group prefix=" (" suffix=")">
|
||||
<text variable="genre"/>
|
||||
<text variable="number" prefix=" No. "/>
|
||||
</group>
|
||||
</if>
|
||||
<else-if type="bill book graphic legal_case legislation motion_picture report song speech" match="any">
|
||||
<text variable="title" font-style="italic"/>
|
||||
</else-if>
|
||||
<else-if type="webpage">
|
||||
<text variable="title" font-style="italic"/>
|
||||
</else-if>
|
||||
<else>
|
||||
<text variable="title"/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="publisher">
|
||||
<choose>
|
||||
<if type="report" match="any">
|
||||
<group delimiter=", ">
|
||||
<text variable="publisher"/>
|
||||
<text variable="publisher-place"/>
|
||||
</group>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="genre" suffix=". "/>
|
||||
<group delimiter=", ">
|
||||
<text variable="publisher"/>
|
||||
<text variable="publisher-place"/>
|
||||
</group>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="event">
|
||||
<choose>
|
||||
<if variable="event">
|
||||
<text term="presented at" text-case="capitalize-first" suffix=" "/>
|
||||
<text variable="event"/>
|
||||
</if>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="issued">
|
||||
<choose>
|
||||
<if variable="issued">
|
||||
<date variable="issued">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
<else-if variable="accessed">
|
||||
<choose>
|
||||
<if type="webpage">
|
||||
<date variable="accessed">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
<else>
|
||||
<text term="no date" form="short"/>
|
||||
</else>
|
||||
</choose>
|
||||
</else-if>
|
||||
<else>
|
||||
<text term="no date" form="short"/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="edition">
|
||||
<choose>
|
||||
<if is-numeric="edition">
|
||||
<group delimiter=" ">
|
||||
<number variable="edition" form="ordinal"/>
|
||||
<text value="edn"/>
|
||||
</group>
|
||||
</if>
|
||||
<else>
|
||||
<text variable="edition" suffix="."/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="locators">
|
||||
<choose>
|
||||
<if type="article-journal article-magazine article-newspaper" match="any">
|
||||
<group prefix=", " delimiter=", ">
|
||||
<group>
|
||||
<text variable="volume"/>
|
||||
</group>
|
||||
<text variable="page"/>
|
||||
</group>
|
||||
</if>
|
||||
<else-if type="bill book graphic legal_case legislation motion_picture report song thesis" match="any">
|
||||
<group delimiter=". " prefix=". ">
|
||||
<text macro="edition"/>
|
||||
<text macro="event"/>
|
||||
<text macro="publisher"/>
|
||||
</group>
|
||||
</else-if>
|
||||
<else-if type="chapter paper-conference" match="any">
|
||||
<group delimiter=", " prefix=". ">
|
||||
<text macro="event"/>
|
||||
<text macro="publisher"/>
|
||||
<group>
|
||||
<label variable="page" form="short" suffix=" "/>
|
||||
<text variable="page"/>
|
||||
</group>
|
||||
</group>
|
||||
</else-if>
|
||||
</choose>
|
||||
</macro>
|
||||
<citation et-al-min="3" et-al-use-first="1" disambiguate-add-year-suffix="true" collapse="year-suffix" year-suffix-delimiter=", ">
|
||||
<sort>
|
||||
<key macro="author"/>
|
||||
<key macro="issued"/>
|
||||
</sort>
|
||||
<layout prefix="(" suffix=")" delimiter="; ">
|
||||
<group delimiter=" ">
|
||||
<text macro="author-short"/>
|
||||
<text macro="issued"/>
|
||||
</group>
|
||||
</layout>
|
||||
</citation>
|
||||
<bibliography et-al-min="7" et-al-use-first="6" entry-spacing="0" hanging-indent="true">
|
||||
<sort>
|
||||
<key macro="author"/>
|
||||
<key macro="issued" sort="ascending"/>
|
||||
<key macro="title"/>
|
||||
</sort>
|
||||
<layout>
|
||||
<group suffix=".">
|
||||
<text macro="author" suffix="."/>
|
||||
<text macro="issued" prefix=" (" suffix="). "/>
|
||||
<group delimiter=". ">
|
||||
<text macro="title"/>
|
||||
<text macro="container"/>
|
||||
</group>
|
||||
<text macro="locators"/>
|
||||
<text macro="access" prefix=". "/>
|
||||
</group>
|
||||
</layout>
|
||||
</bibliography>
|
||||
</style>
|
||||
@@ -1,56 +0,0 @@
|
||||
# Index construction and on-disk layout
|
||||
|
||||
## Construction pipeline
|
||||
|
||||
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/indexing_architecture.md)):
|
||||
|
||||
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition.
|
||||
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
|
||||
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.
|
||||
4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set.
|
||||
5. **Local assembly.** The surviving kmers of each partition are assembled into unitigs — maximal non-branching runs of a local de Bruijn graph — such that every kmer appears exactly once, at one (unitig, offset) location.
|
||||
6. **MPHF and evidence construction.** A minimal perfect hash function is built over the canonical kmers of each partition, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested.
|
||||
|
||||
Phases 1–5 are independent per partition and run in parallel; phase 6 finalizes each partition once its kmer set is fixed.
|
||||
|
||||
## Minimal perfect hash function (MPHF)
|
||||
|
||||
Each partition's surviving kmers are mapped to a dense range of integer slots by a minimal perfect hash function: no collisions, near-optimal space (a few bits per key), O(1) lookup. Because an MPHF maps *any* input to some slot — including kmers that were never indexed — a lookup alone cannot distinguish a genuinely indexed kmer from an arbitrary one; every lookup is followed by an evidence check.
|
||||
|
||||
## Evidence: exact vs. approximate
|
||||
|
||||
Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)):
|
||||
|
||||
- **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
|
||||
- **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
|
||||
|
||||
## On-disk layout
|
||||
|
||||
```
|
||||
<index_root>/
|
||||
index.meta global configuration (k, minimizer size, partition count,
|
||||
evidence mode, whether counts are stored) and genome list/metadata
|
||||
scatter.done / count.done / index.done build-progress sentinels
|
||||
spectrums/<label>.json per-genome kmer frequency histogram
|
||||
partitions/
|
||||
part_00000/ ... part_NNNNN/
|
||||
index/
|
||||
meta.json number of layers in this partition
|
||||
layer_0/
|
||||
unitigs.bin reconstructible kmer sequence data — always kept
|
||||
unitigs.bin.idx random-access index into unitigs.bin (exact evidence only)
|
||||
mphf.bin the minimal perfect hash function
|
||||
evidence.bin exact evidence (exact mode only)
|
||||
fingerprint.bin approximate evidence (approximate mode only)
|
||||
counts/ per-genome kmer counts (if counts were requested)
|
||||
presence/ per-genome presence/absence bits
|
||||
layer_1/, layer_2/, ... added by later merges, same internal structure
|
||||
```
|
||||
|
||||
`unitigs.bin` is the only file from which the indexed kmer content can be fully recovered; it is always retained. Every other file (MPHF, evidence, counts) is derived from it.
|
||||
|
||||
A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
|
||||
|
||||
Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`convert`](../usage/convert.md) one of the sources first if needed.
|
||||
|
||||
`obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time.
|
||||
@@ -1,68 +0,0 @@
|
||||
# obikmer
|
||||
|
||||
`obikmer` is a command-line tool for counting, indexing, querying and comparing DNA sequences represented as kmer sets. It targets individual genome datasets of tens of gigabases, with an emphasis on computational, memory, and disk efficiency.
|
||||
|
||||
All functionality is exposed through a single binary, `obikmer`, organized as subcommands.
|
||||
|
||||
## Core principles
|
||||
|
||||
- Kmers are of fixed, odd length $k$, chosen at index-construction time in the range $[11, 31]$ (see [Kmers and super-kmers](theory/kmers_and_superkmers.md)).
|
||||
- Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see [DNA encoding](theory/encoding.md)).
|
||||
- Kmers are handled in **canonical form** ($\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))$), making counting strand-independent.
|
||||
- Sequences are decomposed into **super-kmers** before storage, anchored on a hash-selected **minimizer** (see [Minimizer selection](theory/minimizer_selection.md)), then routed to one of several **partitions** for parallel, memory-bounded processing (see [Partitioning and indexing architecture](theory/indexing_architecture.md)).
|
||||
- Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see [Low-complexity kmer filter](theory/entropy_filter.md)).
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| [`superkmer`](usage/superkmer.md) | Extract super-kmers from a sequence file and write them to stdout |
|
||||
| [`index`](usage/index_command.md) | Build a genome index |
|
||||
| [`merge`](usage/merge.md) | Merge multiple indexes into one |
|
||||
| [`filter`](usage/filter.md) | Retain only kmers matching ingroup/outgroup predicates |
|
||||
| [`select`](usage/select.md) | Project and/or aggregate genome columns of an index |
|
||||
| [`query`](usage/query.md) | Query an index with sequences and annotate matches |
|
||||
| [`dump`](usage/dump.md) | Dump indexed kmers as CSV |
|
||||
| [`annotate`](usage/annotate.md) | Add, update, or dump genome metadata |
|
||||
| [`phylo`](usage/phylo.md) | Compute pairwise genome distances, trees, and phylogenetic exports |
|
||||
| [`unitig`](usage/unitig.md) | Dump the unitigs of an index as FASTA |
|
||||
| [`estimate`](usage/estimate.md) | Estimate approximate-index parameters before indexing |
|
||||
| [`convert`](usage/convert.md) | Convert an index's evidence representation (exact/approximate/hybrid), in place |
|
||||
| [`utils`](usage/utils.md) | Miscellaneous index maintenance and inspection utilities |
|
||||
| [`pack`](usage/pack.md) | Pack per-column matrix files into a single-file format |
|
||||
|
||||
See [Genome predicates and taxonomy paths](usage/predicates.md) for the selection language shared by `filter`, `select`, `dump`, and `unitig`.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Index construction and on-disk layout](formats/index_layout.md)
|
||||
- [Architecture notes for advanced use](architecture.md) — parallel execution, NUMA awareness, index dimensioning
|
||||
|
||||
## Input formats
|
||||
|
||||
- `superkmer` and `index`: FASTA (`.fa`, `.fasta`), FASTQ (`.fq`, `.fastq`), GenBank flat file (`.gb`, `.gbk`, `.gbff`), all optionally gzip-compressed; directories are expanded recursively; streaming stdin via `-` or when no input path is given.
|
||||
- `query`: FASTA or FASTQ, optionally gzip-compressed; streaming stdin the same way.
|
||||
|
||||
## Parameter constraints
|
||||
|
||||
These constraints are checked at startup; an invalid value exits immediately with an error.
|
||||
|
||||
| Parameter | Constraint | Reason |
|
||||
|---|---|---|
|
||||
| $k$ (`--kmer-size`) | odd, $k \in [11, 31]$ | odd length guarantees the canonical form is always well defined; the range keeps a kmer within a 64-bit word while retaining specificity |
|
||||
| $m$ (`--minimizer-size`) | odd, $3 \le m \le k-1$ | same palindrome argument as $k$; must be strictly shorter than the kmer |
|
||||
| $z$ (`-z`, approximate evidence only) | $z \le k-1$ | the effective indexed kmer size is $k-z+1$ |
|
||||
|
||||
## Genome label constraints
|
||||
|
||||
Genome labels are arbitrary Unicode strings, with the following restrictions:
|
||||
|
||||
| Character | Forbidden | Reason |
|
||||
|---|---|---|
|
||||
| `/` | yes | filesystem path separator |
|
||||
| `=` | yes | separator used by `--new-label` |
|
||||
| `\0` | yes | null byte |
|
||||
| `\n`, `\r`, `\t` | yes | would break CSV output |
|
||||
| spaces | allowed | quote in the shell, e.g. `--new-label 'new label=old label'` |
|
||||
|
||||
Empty labels are rejected. A label derived automatically from the input file name (when `--label` is omitted) is not validated, since it is already filesystem-safe.
|
||||
@@ -1,84 +0,0 @@
|
||||
# Installation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Rust toolchain
|
||||
|
||||
`obikmer` requires **Rust 1.85 or later** (edition 2024). Install or update via [rustup](https://rustup.rs):
|
||||
|
||||
```bash
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
rustup update stable
|
||||
```
|
||||
|
||||
### C build environment (required for hwloc)
|
||||
|
||||
`obikmer` embeds [hwloc](https://www.open-mpi.org/projects/hwloc/) (Hardware Locality) for NUMA-aware thread placement on multi-socket machines. hwloc is built from source at compile time, which requires a standard C build environment.
|
||||
|
||||
#### Linux (Debian/Ubuntu)
|
||||
|
||||
```bash
|
||||
apt install build-essential automake libtool autoconf pkg-config
|
||||
```
|
||||
|
||||
#### Linux (RHEL/Rocky/AlmaLinux)
|
||||
|
||||
```bash
|
||||
dnf install gcc make automake libtool autoconf pkgconfig
|
||||
```
|
||||
|
||||
#### HPC clusters
|
||||
|
||||
Most HPC clusters provide these tools via the module system:
|
||||
|
||||
```bash
|
||||
module load gcc automake libtool autoconf
|
||||
```
|
||||
|
||||
If in doubt, check that `autoreconf --version` and `libtool --version` return successfully.
|
||||
|
||||
#### macOS
|
||||
|
||||
```bash
|
||||
brew install automake libtool autoconf pkg-config
|
||||
```
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd obikmer/src
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
The compiled binary is at `target/release/obikmer`.
|
||||
|
||||
### Building on HPC clusters (network filesystems)
|
||||
|
||||
HPC home directories are typically on a network filesystem (Lustre, NFS) optimized for large sequential reads, not for the many small file operations Cargo generates during compilation. Building directly on such a filesystem can be extremely slow.
|
||||
|
||||
Redirect the build directory to a local scratch disk:
|
||||
|
||||
```bash
|
||||
CARGO_TARGET_DIR=/scratch/$USER/cargo-target cargo build --release
|
||||
```
|
||||
|
||||
Adapt the path to the scratch space available on your cluster (`/var/tmp`, `/tmp`, `/scratch/local`, etc.). Once built, copy the binary to a permanent location:
|
||||
|
||||
```bash
|
||||
cp /scratch/$USER/cargo-target/release/obikmer ~/bin/
|
||||
```
|
||||
|
||||
## NUMA support
|
||||
|
||||
NUMA-aware thread placement is active automatically on multi-socket Linux machines, detected at runtime via hwloc. No build flag is required — it falls back gracefully to a single-pool strategy on:
|
||||
|
||||
- macOS (Apple Silicon, unified memory)
|
||||
- single-socket Linux machines
|
||||
- any system where hwloc reports only one NUMA node
|
||||
|
||||
## Verifying the installation
|
||||
|
||||
```bash
|
||||
obikmer --help
|
||||
```
|
||||
@@ -1,261 +0,0 @@
|
||||
%% This BibTeX bibliography file was created using BibDesk.
|
||||
%% https://bibdesk.sourceforge.io/
|
||||
|
||||
%% Created for Eric Coissac at 2026-04-18 08:19:36 +0200
|
||||
|
||||
|
||||
%% Saved with string encoding Unicode (UTF-8)
|
||||
|
||||
|
||||
|
||||
@article{Zheng2020-ji,
|
||||
abstract = {MOTIVATION: Minimizers are methods to sample k-mers from a
|
||||
string, with the guarantee that similar set of k-mers will be
|
||||
chosen on similar strings. It is parameterized by the k-mer
|
||||
length k, a window length w and an order on the k-mers.
|
||||
Minimizers are used in a large number of softwares and pipelines
|
||||
to improve computation efficiency and decrease memory usage.
|
||||
Despite the method's popularity, many theoretical questions
|
||||
regarding its performance remain open. The core metric for
|
||||
measuring performance of a minimizer is the density, which
|
||||
measures the sparsity of sampled k-mers. The theoretical optimal
|
||||
density for a minimizer is 1/w, provably not achievable in
|
||||
general. For given k and w, little is known about asymptotically
|
||||
optimal minimizers, that is minimizers with density O(1/w).
|
||||
RESULTS: We derive a necessary and sufficient condition for
|
||||
existence of asymptotically optimal minimizers. We also provide a
|
||||
randomized algorithm, called the Miniception, to design
|
||||
minimizers with the best theoretical guarantee to date on density
|
||||
in practical scenarios. Constructing and using the Miniception is
|
||||
as easy as constructing and using a random minimizer, which
|
||||
allows the design of efficient minimizers that scale to the
|
||||
values of k and w used in current bioinformatics software
|
||||
programs. AVAILABILITY AND IMPLEMENTATION: Reference
|
||||
implementation of the Miniception and the codes for analysis can
|
||||
be found at https://github.com/kingsford-group/miniception.
|
||||
SUPPLEMENTARY INFORMATION: Supplementary data are available at
|
||||
Bioinformatics online.},
|
||||
author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
|
||||
doi = {10.1093/bioinformatics/btaa472},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = jul,
|
||||
number = {Suppl_1},
|
||||
pages = {i119--i127},
|
||||
pmc = {PMC8248892},
|
||||
pmid = 32657376,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {Improved design and analysis of practical minimizers},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btaa472},
|
||||
volume = 36,
|
||||
year = 2020,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btaa472}}
|
||||
|
||||
@article{Zheng2021-cc,
|
||||
abstract = {MOTIVATION: Minimizers are efficient methods to sample k-mers
|
||||
from genomic sequences that unconditionally preserve sufficiently
|
||||
long matches between sequences. Well-established methods to
|
||||
construct efficient minimizers focus on sampling fewer k-mers on
|
||||
a random sequence and use universal hitting sets (sets of k-mers
|
||||
that appear frequently enough) to upper bound the sketch size. In
|
||||
contrast, the problem of sequence-specific minimizers, which is
|
||||
to construct efficient minimizers to sample fewer k-mers on a
|
||||
specific sequence such as the reference genome, is less studied.
|
||||
Currently, the theoretical understanding of this problem is
|
||||
lacking, and existing methods do not specialize well to sketch
|
||||
specific sequences. RESULTS: We propose the concept of polar
|
||||
sets, complementary to the existing idea of universal hitting
|
||||
sets. Polar sets are k-mer sets that are spread out enough on the
|
||||
reference, and provably specialize well to specific sequences.
|
||||
Link energy measures how well spread out a polar set is, and with
|
||||
it, the sketch size can be bounded from above and below in a
|
||||
theoretically sound way. This allows for direct optimization of
|
||||
sketch size. We propose efficient heuristics to construct polar
|
||||
sets, and via experiments on the human reference genome, show
|
||||
their practical superiority in designing efficient
|
||||
sequence-specific minimizers. AVAILABILITY AND IMPLEMENTATION: A
|
||||
reference implementation and code for analyses under an
|
||||
open-source license are at
|
||||
https://github.com/kingsford-group/polarset. SUPPLEMENTARY
|
||||
INFORMATION: Supplementary data are available at Bioinformatics
|
||||
online.},
|
||||
author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
|
||||
doi = {10.1093/bioinformatics/btab313},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = jul,
|
||||
number = {Suppl\_1},
|
||||
pages = {i187--i195},
|
||||
pmc = {PMC8686682},
|
||||
pmid = 34252928,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {Sequence-specific minimizers via polar sets},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btab313},
|
||||
volume = 37,
|
||||
year = 2021,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btab313}}
|
||||
|
||||
@article{Pan2024-hb,
|
||||
abstract = {MOTIVATION: The minimizer concept is a data structure for
|
||||
sequence sketching. The standard canonical minimizer selects a
|
||||
subset of k-mers from the given DNA sequence by comparing the
|
||||
forward and reverse k-mers in a window simultaneously according
|
||||
to a predefined selection scheme. It is widely employed by
|
||||
sequence analysis such as read mapping and assembly. k-mer
|
||||
density, k-mer repetitiveness (e.g. k-mer bias), and
|
||||
computational efficiency are three critical measurements for
|
||||
minimizer selection schemes. However, there exist trade-offs
|
||||
between kinds of minimizer variants. Generic, effective, and
|
||||
efficient are always the requirements for high-performance
|
||||
minimizer algorithms. RESULTS: We propose a simple minimizer
|
||||
operator as a refinement of the standard canonical minimizer. It
|
||||
takes only a few operations to compute. However, it can improve
|
||||
the k-mer repetitiveness, especially for the lexicographic order.
|
||||
It applies to other selection schemes of total orders (e.g.
|
||||
random orders). Moreover, it is computationally efficient and the
|
||||
density is close to that of the standard minimizer. The refined
|
||||
minimizer may benefit high-performance applications like binning
|
||||
and read mapping. AVAILABILITY AND IMPLEMENTATION: The source
|
||||
code of the benchmark in this work is available at the github
|
||||
repository https://github.com/xp3i4/mini\_benchmark.},
|
||||
author = {Pan, Chenxu and Reinert, Knut},
|
||||
doi = {10.1093/bioinformatics/btae045},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = feb,
|
||||
number = 2,
|
||||
pmc = {PMC10868324},
|
||||
pmid = 38269626,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {A simple refined DNA minimizer operator enables 2-fold faster computation},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btae045},
|
||||
volume = 40,
|
||||
year = 2024,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btae045}}
|
||||
|
||||
@article{Kille2023-px,
|
||||
abstract = {MOTIVATION: The Jaccard similarity on k-mer sets has shown to be
|
||||
a convenient proxy for sequence identity. By avoiding expensive
|
||||
base-level alignments and comparing reduced sequence
|
||||
representations, tools such as MashMap can scale to massive
|
||||
numbers of pairwise comparisons while still providing useful
|
||||
similarity estimates. However, due to their reliance on minimizer
|
||||
winnowing, previous versions of MashMap were shown to be biased
|
||||
and inconsistent estimators of Jaccard similarity. This directly
|
||||
impacts downstream tools that rely on the accuracy of these
|
||||
estimates. RESULTS: To address this, we propose the minmer
|
||||
winnowing scheme, which generalizes the minimizer scheme by use
|
||||
of a rolling minhash with multiple sampled k-mers per window. We
|
||||
show both theoretically and empirically that minmers yield an
|
||||
unbiased estimator of local Jaccard similarity, and we implement
|
||||
this scheme in an updated version of MashMap. The minmer-based
|
||||
implementation is over 10 times faster than the minimizer-based
|
||||
version under the default ANI threshold, making it well-suited
|
||||
for large-scale comparative genomics applications. AVAILABILITY
|
||||
AND IMPLEMENTATION: MashMap3 is available at
|
||||
https://github.com/marbl/MashMap.},
|
||||
author = {Kille, Bryce and Garrison, Erik and Treangen, Todd J and Phillippy, Adam M},
|
||||
doi = {10.1093/bioinformatics/btad512},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = sep,
|
||||
number = 9,
|
||||
pmc = {PMC10505501},
|
||||
pmid = 37603771,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {Minmers are a generalization of minimizers that enable unbiased local Jaccard estimation},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btad512},
|
||||
volume = 39,
|
||||
year = 2023,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btad512}}
|
||||
|
||||
@incollection{Golan2025-xf,
|
||||
address = {Cham},
|
||||
author = {Golan, Shay and Shur, Arseny M},
|
||||
booktitle = {Lecture Notes in Computer Science},
|
||||
doi = {10.1007/978-3-031-82670-2\_25},
|
||||
isbn = {9783031826696,9783031826702},
|
||||
issn = {0302-9743,1611-3349},
|
||||
language = {en},
|
||||
pages = {347--360},
|
||||
publisher = {Springer Nature Switzerland},
|
||||
series = {Lecture Notes in Computer Science},
|
||||
title = {Expected density of random minimizers},
|
||||
url = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
|
||||
year = 2025,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
|
||||
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-031-82670-2%5C_25}}
|
||||
|
||||
@article{Mohamadi2017-ok,
|
||||
abstract = {Motivation: Many bioinformatics algorithms are designed for the
|
||||
analysis of sequences of some uniform length, conventionally
|
||||
referred to as k -mers. These include de Bruijn graph assembly
|
||||
methods and sequence alignment tools. An efficient algorithm to
|
||||
enumerate the number of unique k -mers, or even better, to build
|
||||
a histogram of k -mer frequencies would be desirable for these
|
||||
tools and their downstream analysis pipelines. Among other
|
||||
applications, estimated frequencies can be used to predict genome
|
||||
sizes, measure sequencing error rates, and tune runtime
|
||||
parameters for analysis tools. However, calculating a k -mer
|
||||
histogram from large volumes of sequencing data is a challenging
|
||||
task. Results: Here, we present ntCard, a streaming algorithm for
|
||||
estimating the frequencies of k -mers in genomics datasets. At
|
||||
its core, ntCard uses the ntHash algorithm to efficiently compute
|
||||
hash values for streamed sequences. It then samples the
|
||||
calculated hash values to build a reduced representation
|
||||
multiplicity table describing the sample distribution. Finally,
|
||||
it uses a statistical model to reconstruct the population
|
||||
distribution from the sample distribution. We have compared the
|
||||
performance of ntCard and other cardinality estimation
|
||||
algorithms. We used three datasets of 480 GB, 500 GB and 2.4 TB
|
||||
in size, where the first two representing whole genome shotgun
|
||||
sequencing experiments on the human genome and the last one on
|
||||
the white spruce genome. Results show ntCard estimates k -mer
|
||||
coverage frequencies >15× faster than the state-of-the-art
|
||||
algorithms, using similar amount of memory, and with higher
|
||||
accuracy rates. Thus, our benchmarks demonstrate ntCard as a
|
||||
potentially enabling technology for large-scale genomics
|
||||
applications. Availability and Implementation: ntCard is written
|
||||
in C ++ and is released under the GPL license. It is freely
|
||||
available at https://github.com/bcgsc/ntCard. Contact:
|
||||
hmohamadi@bcgsc.ca or ibirol@bcgsc.ca. Supplementary information:
|
||||
Supplementary data are available at Bioinformatics online.},
|
||||
author = {Mohamadi, Hamid and Khan, Hamza and Birol, Inanc},
|
||||
date-modified = {2026-04-18 08:19:36 +0200},
|
||||
doi = {10.1093/bioinformatics/btw832},
|
||||
issn = {1367-4803,1367-4811},
|
||||
journal = {Bioinformatics (Oxford, England)},
|
||||
language = {en},
|
||||
month = may,
|
||||
number = 9,
|
||||
pages = {1324--1330},
|
||||
pmc = {PMC5408799},
|
||||
pmid = 28453674,
|
||||
publisher = {Oxford University Press (OUP)},
|
||||
title = {ntCard: a streaming algorithm for cardinality estimation in genomics data},
|
||||
url = {http://dx.doi.org/10.1093/bioinformatics/btw832},
|
||||
volume = 33,
|
||||
year = 2017,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btw832}}
|
||||
|
||||
@misc{Mash-distances-doc,
|
||||
author = {{Marbl Lab}},
|
||||
howpublished = {Mash documentation},
|
||||
title = {Mash Distance},
|
||||
url = {https://mash.readthedocs.io/en/latest/distances.html},
|
||||
urldate = {2026-07-09},
|
||||
year = 2026}
|
||||
|
||||
@article{Fan2015-mash-formula,
|
||||
author = {Fan, Huan and Ives, Anthony R and Surget-Groba, Yann and Cannon, Charles H},
|
||||
doi = {10.1186/s12864-015-1647-5},
|
||||
journal = {BMC Genomics},
|
||||
number = 1,
|
||||
title = {An assembly and alignment-free method of phylogeny reconstruction from next-generation sequencing data},
|
||||
url = {https://doi.org/10.1186/s12864-015-1647-5},
|
||||
volume = 16,
|
||||
year = 2015}
|
||||
@@ -1,28 +0,0 @@
|
||||
# DNA encoding
|
||||
|
||||
## 2-bit nucleotide encoding
|
||||
|
||||
Every nucleotide is encoded on 2 bits, most-significant-bit first within each word:
|
||||
|
||||
| Base | Encoding |
|
||||
|------|----------|
|
||||
| A | `00` |
|
||||
| C | `01` |
|
||||
| G | `10` |
|
||||
| T | `11` |
|
||||
|
||||
The Watson-Crick complement of a base is its bitwise NOT on 2 bits: $\text{complement}(base) = \lnot base \mathbin{\&} \texttt{0b11}$.
|
||||
|
||||
## Kmer encoding
|
||||
|
||||
A kmer of length $k$ ($k \le 31$) fits in a single 64-bit word. The first nucleotide occupies the two most significant bits, each following nucleotide occupies the next two bits, and unused low-order bits are zero. Extracting nucleotide i (0-indexed from the 5′ end) is a shift-and-mask operation.
|
||||
|
||||
Reverse complement is computed by bit manipulation directly on the packed word, without any lookup table: complement every base, reverse the byte order, then reverse the order of 2-bit groups within each byte in two more passes, and finally realign the result to the most-significant bits.
|
||||
|
||||
## Canonical form
|
||||
|
||||
The canonical form of a kmer is the lexicographic minimum of the kmer and its reverse complement:
|
||||
|
||||
$$\text{canonical}(kmer) = \min\big(kmer,\ \text{revcomp}(kmer)\big)$$
|
||||
|
||||
Using the canonical form halves the kmer space and makes counting strand-independent: a kmer and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.
|
||||
@@ -1,36 +0,0 @@
|
||||
# Low-complexity kmer filter
|
||||
|
||||
Low-complexity kmers (homopolymer runs, tandem repeats) can dominate an index without carrying useful information. `obikmer` detects and excludes them during index construction using a normalized Shannon entropy score.
|
||||
|
||||
## Sub-word frequencies
|
||||
|
||||
For a kmer of length $k$ and a sub-word size $ws$ ($1 \le ws \le ws_{\max}$, default $ws_{\max} = 6$), the kmer is decomposed into its $k - ws + 1$ overlapping sub-words of length $ws$ by sliding a window across it. Each sub-word is tallied under its raw 2-bit-packed value, with no canonicalization.
|
||||
|
||||
## Corrected Shannon entropy
|
||||
|
||||
Let $f_j$ be the observed count of raw sub-word $j$, and $n_{\text{words}} = k - ws + 1$ the total number of sub-words. The entropy is:
|
||||
|
||||
$$H_{\text{corr}} = \log(n_{\text{words}}) - \frac{1}{n_{\text{words}}} \sum_j f_j \log f_j$$
|
||||
|
||||
## Small-sample correction
|
||||
|
||||
Because only $n_{\text{words}}$ sub-words are observed among up to $4^{ws}$ possible values, the achievable maximum entropy $H_{\max}$ is bounded below $\log(4^{ws})$ for small samples. $H_{\max}$ is computed from the most uniform integer distribution achievable with $n_{\text{words}}$ observations over $4^{ws}$ categories. The normalized entropy is:
|
||||
|
||||
$$\hat{H}(ws) = \frac{H_{\text{corr}}}{H_{\max}} \in [0, 1]$$
|
||||
|
||||
A value near 0 indicates low complexity (e.g. a homopolymer run); near 1 indicates high complexity, characteristic of a random sequence.
|
||||
|
||||
## Final score
|
||||
|
||||
The filter evaluates $\hat{H}(ws)$ for every word size from 1 to ws_max and keeps the minimum:
|
||||
|
||||
$$\text{entropy}(kmer) = \min_{ws=1}^{ws_{\max}} \hat{H}(ws)$$
|
||||
|
||||
Taking the minimum across word sizes ensures that repetition at any scale is detected: a homopolymer is caught at $ws=1$, a dinucleotide repeat at $ws=2$, and so on. A kmer is rejected if its entropy score falls below a threshold $\theta$ (default 0.7), a configurable collection parameter.
|
||||
|
||||
## Properties
|
||||
|
||||
The entropy score depends only on the kmer sequence itself, not on where or how many times it occurs:
|
||||
|
||||
- **Orientation invariance**: a kmer and its reverse complement always receive the same score.
|
||||
- **Context independence**: a given kmer is always accepted or always rejected, regardless of which genome or read it appears in. The filter defines a fixed partition of the kmer space into low-complexity and valid kmers.
|
||||
@@ -1,32 +0,0 @@
|
||||
# Partitioning and indexing architecture
|
||||
|
||||
An index is split into a fixed number of **partitions**, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
|
||||
|
||||
## Routing
|
||||
|
||||
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition:
|
||||
|
||||
```
|
||||
canonical minimizer → hash(minimizer) → p-bit value → partition index
|
||||
```
|
||||
|
||||
The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.
|
||||
|
||||
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
|
||||
|
||||
## Why hashing is necessary
|
||||
|
||||
A canonical minimizer is an m-mer ($m \in \{9, 11, 13, 15\}$), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.
|
||||
|
||||
Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits $p$ is chosen well below the number of bits available in the minimizer ($2m$): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
|
||||
|
||||
## Parameter guidance
|
||||
|
||||
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|
||||
|----|-----------|-----------|------------|
|
||||
| 9 | 18 | 6–8 | 64–256 |
|
||||
| 11 | 22 | 8–10 | 256–1 024 |
|
||||
| 13 | 26 | 10–12 | 1 024–4 096|
|
||||
| 15 | 30 | 10–14 | 1 024–16 384|
|
||||
|
||||
The number of partitions must satisfy $p \le 2m$, and in practice $p$ is chosen well below that bound to leave a comfortable entropy margin. For $k=31$, $m=13$, $p=10$ (1024 partitions), partition load is well balanced on real genomic data.
|
||||
@@ -1,24 +0,0 @@
|
||||
# Kmers and super-kmers
|
||||
|
||||
## Kmers
|
||||
|
||||
A **kmer** is a DNA subsequence of fixed length $k$. Two constraints apply to $k$, both enforced when a command starts (an invalid value exits immediately with an error):
|
||||
|
||||
- $k \in [11, 31]$: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base ($k \le 32$ is the hard limit; $k < 11$ gives insufficient specificity).
|
||||
- $k$ **is odd**: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see [DNA encoding](encoding.md)) to be well defined.
|
||||
|
||||
## Super-kmers
|
||||
|
||||
A **super-kmer** is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see [Minimizer selection](minimizer_selection.md)). Each kmer in the run overlaps the next by $k-1$ nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
|
||||
|
||||
For a random minimizer of length $m$ over kmers of length $k$, the expected length of a super-kmer is approximately [@Zheng2020-ji; @Golan2025-xf]:
|
||||
|
||||
$$L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1$$
|
||||
|
||||
For $k=31$, $m=13$ this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.
|
||||
|
||||
### Canonical super-kmers
|
||||
|
||||
A **canonical super-kmer** is the lexicographic minimum of a super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.
|
||||
|
||||
Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.
|
||||
@@ -1,42 +0,0 @@
|
||||
# Minimizer selection
|
||||
|
||||
## Definition
|
||||
|
||||
A **minimizer** of a kmer window is the m-mer ($m < k$) that is smallest, among all $k - m + 1$ overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent.
|
||||
|
||||
The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
|
||||
|
||||
## Hash-based ("random") minimizer
|
||||
|
||||
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions.
|
||||
|
||||
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition.
|
||||
|
||||
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
|
||||
|
||||
### Hash function
|
||||
|
||||
The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed:
|
||||
|
||||
$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
|
||||
|
||||
```
|
||||
H(x):
|
||||
x ← x ⊕ 0x9e3779b97f4a7c15
|
||||
x ← x ⊕ (x >> 30)
|
||||
x ← x × 0xbf58476d1ce4e5b9
|
||||
x ← x ⊕ (x >> 27)
|
||||
x ← x × 0x94d049bb133111eb
|
||||
return x ⊕ (x >> 31)
|
||||
```
|
||||
|
||||
The XOR seed avoids the finalizer's fixed point at 0 ($\text{mix64}(0) = 0$), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison.
|
||||
|
||||
## Partition routing is independent of minimizer selection
|
||||
|
||||
The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately:
|
||||
|
||||
- **Selection** uses $H$ applied to every candidate m-mer in the window, keeping the minimum.
|
||||
- **Partition routing** recomputes $H$ on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.
|
||||
|
||||
See [Partitioning and indexing architecture](indexing_architecture.md) for how the routing value is turned into a partition index.
|
||||
@@ -1,25 +0,0 @@
|
||||
# annotate
|
||||
|
||||
Add or update genome metadata of an index from a CSV file, or dump the current metadata as CSV.
|
||||
|
||||
```bash
|
||||
obikmer annotate INDEX --csv FILE [OPTIONS]
|
||||
obikmer annotate INDEX --dump
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `INDEX` | Index directory to annotate (modified in place) |
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--csv` | — | CSV file of metadata to apply (must contain an id column); required unless `--dump` is used |
|
||||
| `--sep` | `,` | CSV field separator |
|
||||
| `--id-col` | `id` | Name of the column containing genome labels |
|
||||
| `--na-value` | `NA` | Value meaning "remove this field" (deletes the existing key if present) |
|
||||
| `--no-overwrite` | off | Do not overwrite existing metadata keys |
|
||||
| `--dump` | off | Print all genome metadata as CSV to stdout instead of applying a file |
|
||||
@@ -1,29 +0,0 @@
|
||||
# convert
|
||||
|
||||
Convert an existing index's evidence representation in place, between exact, approximate, and hybrid.
|
||||
|
||||
```bash
|
||||
obikmer convert INDEX (--exact-evidence | --approx-evidence BITS | --hybrid-evidence) [OPTIONS]
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `INDEX` | Index directory to convert (modified in place) |
|
||||
|
||||
## Options
|
||||
|
||||
Exactly one of the first three is required:
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--exact-evidence` | Convert to exact evidence (zero false positives) |
|
||||
| `--approx-evidence BITS` | Convert to approximate (fingerprint-only) evidence; `BITS` = fingerprint bits per slot (b) |
|
||||
| `--hybrid-evidence` | Convert to hybrid evidence (both exact and approximate bundles kept) |
|
||||
| `--evidence-bits BITS` | Fingerprint bits per slot (b) — required with `--hybrid-evidence` when the source index is currently exact; rejected otherwise (the source already fixes `b`) |
|
||||
| `-z, --findere-z Z` | Findere z parameter: number of consecutive stored kmers that must all match to confirm a hit. This does not shorten the indexed kmer length (fixed forever at `index` build time) — it extends the effective match window: on a k=31 index, `z=2` requires 32 consecutive matching bases, not 30 |
|
||||
| `--fp FP` | Target false-positive rate per z-window (e.g. `0.01`); derives `b` or `z` when one of them isn't given directly |
|
||||
| `--block-size N` | Block size for exact evidence's on-disk index (unitigs per block). Ignored when converting to pure approximate evidence. Default `1` |
|
||||
|
||||
See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.
|
||||
@@ -1,25 +0,0 @@
|
||||
# dump
|
||||
|
||||
Dump all kmers of an index as CSV, one row per kmer, with per-genome counts or presence.
|
||||
|
||||
```bash
|
||||
obikmer dump INDEX [OPTIONS]
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `INDEX` | Index directory to dump |
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--force-presence` | off | Output presence/absence (0/1) even if the index stores counts |
|
||||
| `--debug` | off | Prefix each row with the partition and layer columns |
|
||||
| `--head N` | none | Limit output to the first N kmers |
|
||||
|
||||
`dump` also accepts the shared [predicate options](filter.md#predicate-options) (`--ingroup`, `--outgroup`, `--min-count`, etc.) to restrict which kmers are dumped.
|
||||
|
||||
Output is CSV on stdout.
|
||||
@@ -1,18 +0,0 @@
|
||||
# estimate
|
||||
|
||||
Estimate approximate-index parameters (z, evidence bits, false-positive rate) before building an index with `--approx`, without touching any files.
|
||||
|
||||
```bash
|
||||
obikmer estimate [OPTIONS]
|
||||
```
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `-k, --kmer-size` | `31` | Kmer size used at query time (matches `index`'s `--kmer-size`) |
|
||||
| `-z, --findere-z` | none | Findere z parameter |
|
||||
| `--evidence-bits` | none | Fingerprint bits per slot (b) |
|
||||
| `--fp` | none | Target false-positive rate per z-window |
|
||||
|
||||
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
|
||||
@@ -1,47 +0,0 @@
|
||||
# filter
|
||||
|
||||
Apply row-level selection to an index: retain only kmers matching ingroup/outgroup predicates over genome membership, plus optional total-count and complexity thresholds. The output is a new, single-layer index.
|
||||
|
||||
```bash
|
||||
obikmer filter SOURCE -o OUTPUT [OPTIONS]
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `SOURCE` | Source index directory |
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `-o, --output` | — (required) | Output index directory |
|
||||
| `-f, --force` | off | Overwrite an existing output directory |
|
||||
| `--presence` | off | Output presence/absence instead of counts |
|
||||
| `--min-total-count` | none | Minimum total count across all genomes (count index only) |
|
||||
| `--max-total-count` | none | Maximum total count across all genomes |
|
||||
| `--min-complexity` | none | Minimum normalized entropy (same score as `--theta` at index build time), recomputed from the stored unitig sequences |
|
||||
| `--complexity-level-max` | `6` | Maximum sub-word size for the complexity score (used only with `--min-complexity`) |
|
||||
|
||||
## Predicate options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--ingroup` | none | Ingroup predicate (repeatable; each occurrence is ANDed) |
|
||||
| `--outgroup` | none | Outgroup predicate (repeatable; each occurrence is ORed) |
|
||||
| `--min-count` | 0, or group size + N if negative | Minimum number of ingroup genomes carrying the kmer |
|
||||
| `--max-count` | ingroup group size | Maximum number of ingroup genomes carrying the kmer |
|
||||
| `--min-frac` | `1.0` if `--ingroup` given without an explicit quorum, else `0.0` | Minimum fraction of ingroup genomes |
|
||||
| `--max-frac` | `1.0` | Maximum fraction of ingroup genomes |
|
||||
| `--min-outgroup-count` | `0` | Minimum number of outgroup genomes carrying the kmer |
|
||||
| `--max-outgroup-count` | `0` if `--outgroup` given without an explicit quorum, else outgroup group size | Maximum number of outgroup genomes |
|
||||
| `--min-outgroup-frac` | `0.0` | Minimum fraction of outgroup genomes |
|
||||
| `--max-outgroup-frac` | `1.0` | Maximum fraction of outgroup genomes |
|
||||
| `--presence-threshold` | `0` | Minimum count for a genome to be considered a carrier of a kmer |
|
||||
|
||||
See [Genome predicates and taxonomy paths](predicates.md) for the predicate syntax used by `--ingroup`/`--outgroup`.
|
||||
|
||||
A negative `--min-count`/`--max-count` is interpreted as an offset from the group size — e.g. `--min-count=-1` means "all but one".
|
||||
|
||||
Declaring `--ingroup` with no explicit ingroup quorum flag implicitly sets `--min-frac 1.0` (present in every ingroup genome). Declaring `--outgroup` with no explicit outgroup quorum flag implicitly sets `--max-outgroup-count 0` (absent from every outgroup genome). Any explicit quorum flag for a group disables that group's implicit default.
|
||||
@@ -1,50 +0,0 @@
|
||||
# index
|
||||
|
||||
Build a genome index from one or more sequence files. Construction proceeds in phases (scatter → dereplicate → count → layered MPHF), described in [On-disk storage](../formats/index_layout.md).
|
||||
|
||||
```bash
|
||||
obikmer index -o OUTPUT [OPTIONS] [INPUTS...]
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. |
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `-o, --output` | — (required) | Output index directory |
|
||||
| `--force` | off | Overwrite an existing output directory |
|
||||
| `--label` | input file name without extension | Genome label stored in the index |
|
||||
| `--meta KEY=VALUE` | none | Attach a categorical metadata field to the genome (repeatable) |
|
||||
| `-k, --kmer-size` | `31` | Kmer size (odd, in [11, 31]) |
|
||||
| `-m, --minimizer-size` | `11` | Minimizer size (odd, in $[3, k-1]$) |
|
||||
| `--theta` | `0.7` | Entropy threshold for the low-complexity filter |
|
||||
| `--level-max` | `6` | Maximum sub-word size for the entropy score |
|
||||
| `-p, --partitions` | `256` | Number of partitions (rounded up to a power of 2) |
|
||||
| `-T, --threads` | detected core count | Number of worker threads |
|
||||
| `--max-open-files` | `threads / 4` (min 1) | Maximum number of input files open simultaneously |
|
||||
| `--min-abundance` | `1` | Minimum abundance (inclusive) for a kmer to be retained |
|
||||
| `--max-abundance` | none | Maximum abundance (inclusive) |
|
||||
| `--with-counts` | off | Store per-kmer counts; otherwise only presence/absence is stored |
|
||||
| `--keep-intermediate` | off | Keep intermediate build files instead of deleting them after construction |
|
||||
| `--approx` | off | Use approximate evidence (Findere fingerprint) instead of exact evidence |
|
||||
| `-z, --findere-z` | see below | Findere z parameter: number of consecutive kmers that must all match (approximate evidence only) |
|
||||
| `--evidence-bits` | see below | Fingerprint bits per slot (b), approximate evidence only |
|
||||
| `--fp` | see below | Target false-positive rate per z-window, approximate evidence only |
|
||||
| `--block-size` | `1` | Block size, in unitigs, for the exact on-disk index (rounded up to a power of 2) |
|
||||
|
||||
## Exact vs. approximate evidence
|
||||
|
||||
By default, an index stores **exact** evidence: a kmer is either present or absent (or has an exact count with `--with-counts`), with no false positives.
|
||||
|
||||
With `--approx`, evidence is stored as a compact **fingerprint** instead, trading a small, tunable false-positive rate for reduced memory/disk usage. The false-positive model is:
|
||||
|
||||
$$FP = \frac{1}{2^{b \cdot z}}$$
|
||||
|
||||
where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`convert`](convert.md) to change an existing index's representation afterwards.
|
||||
|
||||
`z` must be strictly less than k: the effective indexed kmer length under approximate evidence is k−z+1.
|
||||
@@ -1,29 +0,0 @@
|
||||
# merge
|
||||
|
||||
Merge multiple built indexes into a single index.
|
||||
|
||||
```bash
|
||||
obikmer merge -o OUTPUT SOURCE... [OPTIONS]
|
||||
```
|
||||
|
||||
## Arguments
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `SOURCE...` | Index directories to merge (at least one required) |
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `-o, --output` | — (required) | Output index directory |
|
||||
| `--force` | off | Overwrite an existing output directory |
|
||||
| `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts |
|
||||
| `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing |
|
||||
| `--budget-fraction` | `0.5` | Fraction of available RAM reserved as the memory budget for parallel partition merging |
|
||||
|
||||
## Behaviour
|
||||
|
||||
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources.
|
||||
|
||||
By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user