Compare commits
11
Commits
cbc344e4b4
...
v1.4.5
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5db7b440da | ||
|
|
c93616b432 | ||
|
|
c4eacb3f72 | ||
|
|
14433e80a5 | ||
|
|
7f829ea557 | ||
|
|
07c0a54e04 | ||
|
|
af525cd268 | ||
|
|
d0668eb167 | ||
|
|
4e98de4b77 | ||
|
|
03eaa4544a | ||
|
|
24291a4484 |
No files matched your search
@@ -83,8 +83,14 @@ B would take the 8 down to ~2.
|
||||
|
||||
**Follow-up — sparse pack (same day).** `pack_sparse_bit_matrix`/`pack_sparse_compact_int_matrix` used to open the columnar matrix directly, which maps every column file of a layer at once: `n_genomes` maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single `matrix.pbmx`/`.pcmx` mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: `pack` went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale `matrix.pbmx` that predates newer columnar data is now re-packed instead of being used as is (`pack_bit_matrix` compares `n_cols`). Both oracles pass.
|
||||
|
||||
## Known issue (found while building the oracle, not fixed)
|
||||
## Re-merging a merge output (resolved 2026-11-18)
|
||||
|
||||
Re-merging a merge output fails with `I/O error: No such file or directory` in `merge_partitions`, in every mode (presence sparse, presence `--dense`, counts). Cause, verified: `finalize_indexed` packs the output matrices (`sparse_meta.json` + dict/PFIV files, or `matrix.pbmx`), so `presence/`/`counts/` no longer holds `meta.json` + `col_*`. When that index is the merge base, bootstrap copies the packed layout and `MatrixBuilder::resume` → `MatrixMeta::load` finds no `meta.json`. Pre-existing, independent of this refactor; deferred to a separate fix (decision 2026-11-17). The synthetic oracle (`scripts/smoke_test_merge.sh`) therefore omits the "merge AB, then AB C D" scenario.
|
||||
Symptom: `I/O error: No such file or directory` in `merge_partitions` when a merge output was one of the sources. Cause (verified): `finalize_indexed` packs the output matrices (`sparse_meta.json` + dict/PFIV files, or `matrix.pbmx`), so `presence/`/`counts/` no longer holds `meta.json` + `col_*`. `choose_base` picks the largest source as destination, i.e. the merged one; bootstrap copies the packed layout and `MatrixBuilder::resume` → `MatrixMeta::load` finds no `meta.json`.
|
||||
|
||||
Fix: a packed index can only be a *source* (pass 2 reads it through `PersistentBitMatrix`/`PersistentIntMatrix::open`, which handle every layout). `choose_base` now filters candidates with `can_be_destination` (`obikmerge/src/merge.rs`): one layer is probed with `detect_storage`; `Columnar` qualifies, `Implicit` only for a single genome (materialized on the fly), `Packed`/`Sparse` never. With a single source there is nothing to append and the filter is skipped. When several sources are given and none qualifies, `run` returns an explicit error before any work. Consequence: the result's column order follows the chosen base first, so it can differ from the command-line order.
|
||||
|
||||
Not done: in-place unpack of a packed index to allow it as destination, or a destination-side append on the packed format — the sparse incremental strategy is tracked in `merge_scaling.md`.
|
||||
|
||||
Oracle: `scripts/smoke_test_merge.sh` scenario 2 (A B → AB, then AB C D) is compared against the flat merge by genome label (not against the golden dumps, whose layer placement differs), and merging only packed indexes is checked to be refused.
|
||||
|
||||
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to `mmap`.
|
||||
@@ -509,6 +509,7 @@ Provided finalisations:
|
||||
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
|
||||
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
|
||||
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
|
||||
| `threshold_hamming_dist_matrix(t)` | `union[i,j] − inter[i,j]` (k-mers present in exactly one genome) — no separate partial |
|
||||
|
||||
### BitPartials
|
||||
|
||||
|
||||
+2
-2
@@ -13,11 +13,11 @@
|
||||
| `query` | Query an index with sequences and annotate matches |
|
||||
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
|
||||
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
|
||||
| `phylo` | Compute pairwise evolutionary-distance proxies between genomes (`--metric jaccard\|mash\|hamming\|bray-curtis\|relfreq-bray-curtis\|euclidean\|relfreq-euclidean\|hellinger\|hellinger-euclidean`); optionally build NJ/UPGMA trees; `--presence-threshold N` sets the minimum count to consider a k-mer present when computing Jaccard/Mash on count indexes (default 1); optionally a central-position SNP/Sankoff calibration with exports for TNT/PhyG/IQ-TREE (see [evolutionary distances](theory/evolutionary_distances.md)) |
|
||||
| `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash/Hamming on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
|
||||
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
|
||||
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
|
||||
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
|
||||
| `reindex` | Convert an index's evidence in-place: exact ↔ approx |
|
||||
| `convert` | Convert an index's evidence representation (exact / approximate / hybrid), in place |
|
||||
| `utils` | Miscellaneous index utilities: `--new-label NEW=OLD` renames a genome label; `--upgrade-index` adds missing `layer_meta.json` to old indexes |
|
||||
| `pack` | Pack per-column matrix files into single-file format to reduce query I/O |
|
||||
|
||||
|
||||
@@ -226,7 +226,9 @@ Design decision (user): the way genomes are counted depends on `--no-ambiguity`,
|
||||
|
||||
## Implemented: `--quartet` (2026-10-13)
|
||||
|
||||
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Absence never counts as a state; with `--no-ambiguity`, multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; the `--no-ambiguity` convention is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`) because that header has no room for a flag. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
|
||||
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Genomes are read through `masked_state`, exactly as in the alignment: absence is a state of its own unless `--free-loss` (then it is missing and counts for nothing), and with `--no-ambiguity` multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; its counting convention (`QuartetConvention`: ambiguity policy × `--free-loss`) is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`, `quartet.diploid.pbiv` for `--free-loss`; `quartet.abs.*.pbiv` for absence-as-a-state) because that header has no room for a flag.
|
||||
|
||||
*Correction (2026-11-19).* The first version never counted absence, whatever `--free-loss` said, which contradicted the alignment: without `--free-loss` a genome lacking the family has the state `∅`, so `A A ∅ ∅` separates two pairs yet was rejected. Fixed in `has_bipartition` (index 0 is now counted, and is only ever populated when absence is a state), in the annex (one file per `QuartetConvention`) and in `--sample-stats` (informative sites follow the same rule). The old file names keep the `--free-loss` semantics, which is exactly the old rule, so annexes already built with `--free-loss` stay valid; the sampling measured below used `--free-loss` and is unaffected. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
|
||||
|
||||
Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list.
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ A family where every genome carries the same single state has entropy 0 (uninfor
|
||||
|
||||
## Quartet restriction
|
||||
|
||||
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts called states only, so absence never forms one of the two pairs.
|
||||
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts the states of the alignment: without `--free-loss`, absence is a state like any other and can form one of the two pairs; with `--free-loss` it is missing data and never does.
|
||||
|
||||
## Pair balance
|
||||
|
||||
|
||||
@@ -12,6 +12,14 @@ obikmer index -o OUTPUT [OPTIONS] [INPUTS...]
|
||||
|---|---|
|
||||
| `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. |
|
||||
|
||||
## Which files are read
|
||||
|
||||
A file named explicitly on the command line is always read, whatever its extension: the format (FASTA, FASTQ or GenBank) and the compression (gzip, bzip2, xz or zstd) are detected from the content. If the content is not a sequence file, the command reports an error rather than skipping the file.
|
||||
|
||||
When an input is a directory, the files inside it are selected by extension: `.fasta`, `.fa`, `.fna` (the extension of NCBI genome downloads), `.ffn`, `.fas`, `.fsa`, `.fastq`, `.fq`, `.gbff`, `.gbk` and `.gb`, each optionally followed by `.gz`.
|
||||
|
||||
A small input spread over many partitions leaves most partitions empty. This is valid: the index is built normally and empty partitions simply hold no k-mers.
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|
||||
@@ -36,6 +36,10 @@ The merge works partition by partition, and processes several partitions in para
|
||||
|
||||
Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.
|
||||
|
||||
## Current limitation
|
||||
## Merging an index that was already merged
|
||||
|
||||
An index produced by `merge` cannot itself be used as a source of a later merge yet: the command stops with a "No such file or directory" error. To add genomes to a collection, merge the original single-genome indexes again, together with the new ones.
|
||||
An index produced by `merge` can be used as one of the sources of a later merge, which is how a collection is extended without starting again from the original genomes. There is one rule: among the sources, at least one must not have been through a merge. A merged index is stored in a compact packed format (see [pack](pack.md)) that is read very efficiently but cannot be extended in place. The merge therefore takes a freshly built index as its starting point and brings the packed index in as a source. A typical case is a merged index of many genomes plus one or a few newly built single-genome indexes, and this works.
|
||||
|
||||
If every source has already been merged or packed, the command stops before doing any work and says so. In that case, merge the original single-genome indexes together instead.
|
||||
|
||||
The genome columns of the result follow the order of the starting index first, then the other sources, so they may not come out in the order the sources were listed. The genome labels always identify each column unambiguously.
|
||||
@@ -87,7 +87,7 @@ The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta`
|
||||
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
||||
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
||||
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
|
||||
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct called states, each carried by at least two genomes. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
|
||||
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
|
||||
@@ -136,9 +136,9 @@ With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`)
|
||||
|
||||
### `--quartet`: keeping only families that can resolve a tree
|
||||
|
||||
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. Absence is never a state here: a genome that does not carry the family does not count, with or without `--free-loss`. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
||||
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
||||
|
||||
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
||||
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
||||
|
||||
### Describing a draw: `--sample-stats`, `--sample-plan`
|
||||
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Prompt — où optimiser l'étape `pack` du merge (diagnostic d'abord)
|
||||
|
||||
> À coller tel quel au début d'une nouvelle session Claude Code ouverte à la racine du dépôt obikmer.
|
||||
> Cette session est un **diagnostic** : elle produit une analyse chiffrée et des pistes classées, pas du code. Toute modification du code attend mon feu vert explicite, piste par piste.
|
||||
|
||||
---
|
||||
|
||||
## Contexte
|
||||
|
||||
`obikmer merge` se termine par une étape `finalize`, dont l'essentiel est `pack` : les matrices de présence/comptage, écrites en colonnes pendant le merge (`MatrixBuilder`), sont transposées en un format compact par partition (sparse par défaut). Le point d'entrée est `KmerIndex::finalize_indexed` (`src/obikindex/src/index/builder.rs`, étape « pack » vers la ligne 109), qui appelle `KmerIndex::pack_matrices(sparse)` (`src/obikindex/src/index/kmer_index.rs`). Les primitives sont dans `src/obicompactvec/src/` : `pack_bit_matrix` (`bitmatrix/packed.rs`), `pack_compact_int_matrix` (`intmatrix.rs`), `pack_sparse_compact_int_matrix` (`sparse_intmatrix.rs`), et le côté sparse des bits dans `bitmatrix/`.
|
||||
|
||||
Lis d'abord `DevDocMD/implementation/merge_scaling.md` (état du pack sparse : passage par une matrice dense séquentielle puis sparse, un seul mapping par partition) et `DevDocMD/implementation/merge_mmap_audit.md`.
|
||||
|
||||
## Mesure de référence (production, 2026-11-18)
|
||||
|
||||
Merge de 186 sources (`Cedrus`), 256 partitions, binaire glibc compilé localement, **192 CPU non contraints**, 17,5 milliards de k-mers nouveaux au total. Mode **présence**, compaction par défaut, donc **sparse** (pas de `--dense`) :
|
||||
|
||||
| Étape | Durée | Efficacité CPU | RSS max |
|
||||
|---|---|---|---|
|
||||
| `merge_partitions` | 748 s | 141×/192 (73 %) | 354 Go |
|
||||
| `pack` | 231 s | 65×/192 (34 %) | **503,7 Go** |
|
||||
|
||||
La RAM de la machine est de 754 Go. Le RSS max est celui du process : il inclut les pages de fichiers mappés, pas seulement la mémoire anonyme. **Je n'ai pas encore séparé les deux.**
|
||||
|
||||
Progression de `pack` (partitions terminées par tranche de 10 %, début à 04:08:09) :
|
||||
`+53 s` (10 %) · `+35 s` (20 %) · `+13 s` (30 %) · `+48 s` (40 %) · `+24 s` (50 %) · `+9 s` · `+6 s` · `+6 s` · `+6 s` · `+11 s` (100 %), puis `+20 s` avant la fin de l'étape. Les partitions les plus grosses passent en premier (tri décroissant dans `merge.rs`), mais je ne sais pas si `pack` les traite dans le même ordre ni si l'ordre est le bon.
|
||||
|
||||
## Ce que je veux savoir
|
||||
|
||||
1. **Où part le temps** de `pack` ? Lecture des colonnes, transposition, écriture, `sync`/`fsync`, attente d'un verrou, déséquilibre entre partitions ? Efficacité de 34 % sur 192 workers : qu'est-ce qui les bloque ?
|
||||
2. **Où part la mémoire** (504 Go) ? Combien est de la mémoire anonyme, combien des pages de fichiers mappés ? Est-elle proportionnelle au nombre de workers ? Quelle est la mémoire vivante par partition en cours de `pack` ?
|
||||
3. **Quel est le parallélisme réel** : combien de partitions `pack` traite-t-il en même temps, et qu'est-ce qui le fixe (`PartitionRunner`, `--cpu-max`, NUMA) ?
|
||||
4. Y a-t-il du **travail inutile** (relectures, copies intermédiaires, colonnes vides ou toutes-absentes ré-écrites, étape dense qui pourrait être évitée) ?
|
||||
|
||||
## Méthode (impérative)
|
||||
|
||||
- **Lire et mesurer avant de proposer.** Pas de récit plausible sans preuve : tout ce qui n'est pas vérifié par lecture du code ou par mesure est marqué « hypothèse ».
|
||||
- Pour mesurer sans la production : `scripts/docker_debug/` fournit un conteneur Linux (build glibc ou musl) qui lance un merge de N sources synthétiques ou dérivées de génomes réels (`run.sh`, variables `GEN`, `REAL`, `GENOME_LEN`, `TARGET`, `TOOL=heaptrack` pour glibc) avec un moniteur CSV de mappings et de RSS. Les génomes de 2 Mb donnent des partitions assez grosses pour que `pack` ait quelque chose à faire. `heaptrack` fonctionne sur le build glibc (pas sur le build musl statique). `gdb` ne fonctionne pas sous Rosetta.
|
||||
- Pour séparer mémoire anonyme et fichiers mappés, utilise `/proc/PID/status` (`RssAnon`, `RssFile`, `RssShmem`), **jamais `smaps` en entier** sur un process à beaucoup de mappings (il verrouille `mmap_lock` et ralentit le process).
|
||||
- Les estimations de coût (octets, secondes) sont marquées **estimées** ; les chiffres observés sont marqués **mesurés**.
|
||||
|
||||
## Règles du projet (non négociables)
|
||||
|
||||
- Avant tout travail de code : `mcp__serena__initial_instructions`, puis Serena (`find_symbol`, `get_symbols_overview`, `find_referencing_symbols`) pour naviguer. `rust-analyzer-mcp` et `qmd` quand ils servent. Consulte `context7` avant d'utiliser une API de bibliothèque externe.
|
||||
- **Aucune commande VCS mutante** (git/jj : commit, add, mv, rm, describe, push…). Lecture seule autorisée.
|
||||
- **Aucune nouvelle dépendance, aucun changement de version** sans mon accord explicite. Si un problème de bibliothèque se pose, expose-le et propose des alternatives.
|
||||
- **Une question appelle une réponse, pas une action.** Ne modifie aucun fichier de `src/` sans demande explicite. Cette session est en lecture seule sur le code.
|
||||
- Le code existant est une hypothèse, pas une vérité : signale tout ce qui est devenu injustifié plutôt que de le garder par inertie.
|
||||
- Si tu proposes d'ajouter une opération à une structure partagée, mets-la comme méthode générique sur cette structure (crate `obicompactvec`), pas comme fonction libre dans le consommateur.
|
||||
- Ne jamais matérialiser en mémoire une structure volumineuse qui peut être streamée.
|
||||
- Échanges en **français**, documentation en **anglais**.
|
||||
|
||||
## Livrable
|
||||
|
||||
Un rapport (dans la conversation, pas dans un fichier, sauf si je le demande) avec :
|
||||
1. Le chemin de code de `pack`, de `finalize_indexed` jusqu'aux écritures disque, résumé en quelques lignes par étape.
|
||||
2. Pour chaque question ci-dessus : ce qui est **mesuré**, ce qui est **hypothèse**, et comment vérifier l'hypothèse.
|
||||
3. Une liste de **pistes d'optimisation classées** par gain attendu/coût/risque. Pour chaque piste : ce qu'elle changerait, ce qu'elle risque, et comment on verrait qu'elle marche. Aucune implémentation.
|
||||
4. Les vérifications de non-régression à prévoir : le résultat sur disque doit rester **identique octet pour octet** (voir les oracles `scripts/smoke_test_merge.sh` et `scripts/bench_merge_oracle.sh`).
|
||||
|
||||
Termine par la question : « Quelle piste veux-tu que j'implémente ? » et attends ma réponse.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Linux debug environment for the merge mapping/memory-growth hunt.
|
||||
# Same toolchain as .gitea/workflows/release.yml for the musl target
|
||||
# (cargo-zigbuild + zig cc), plus a glibc toolchain and diagnostic tools.
|
||||
FROM rust:1-bookworm
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential cmake pkg-config autoconf automake libtool \
|
||||
heaptrack valgrind gdb strace procps python3 python3-pip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN pip install ziglang --quiet --break-system-packages \
|
||||
&& cargo install cargo-zigbuild \
|
||||
&& rustup target add x86_64-unknown-linux-musl
|
||||
RUN ZIG=$(python3 -c "import ziglang, os; print(os.path.join(os.path.dirname(ziglang.__file__), 'zig'))") \
|
||||
&& printf '#!/bin/sh\nexec "%s" cc -target x86_64-linux-musl "$@"\n' "$ZIG" > /usr/local/bin/x86_64-linux-musl-gcc \
|
||||
&& printf '#!/bin/sh\nexec "%s" c++ -target x86_64-linux-musl "$@"\n' "$ZIG" > /usr/local/bin/x86_64-linux-musl-g++ \
|
||||
&& chmod +x /usr/local/bin/x86_64-linux-musl-gcc /usr/local/bin/x86_64-linux-musl-g++
|
||||
WORKDIR /work
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Builds the debug image. PLATFORM defaults to linux/amd64 (Rosetta on Apple Silicon).
|
||||
set -euo pipefail
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
PLATFORM="${PLATFORM:-linux/amd64}"
|
||||
docker build --platform "$PLATFORM" -t "obikmer-debug:${PLATFORM//\//-}" .
|
||||
Executable
+82
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env bash
|
||||
# Index compatibility test between two obikmer binaries that differ only by
|
||||
# their ptr_hash version (A = baseline, B = candidate).
|
||||
#
|
||||
# Usage: compat_ptrhash.sh VOL_A VOL_B OUT
|
||||
# VOL_A / VOL_B docker volumes holding a musl release build (CARGO_TARGET_DIR)
|
||||
# OUT host directory receiving report.txt
|
||||
#
|
||||
# What it checks (everything compared by md5 of dump/query output):
|
||||
# T0 on-disk bytes of an index built by A vs by B (informational)
|
||||
# T1 A-built single-genome indexes read by A and by B
|
||||
# T2 merges: A merging A-built sources (reference), B merging A-built
|
||||
# sources, B merging B-built sources; each result dumped by A and by B
|
||||
# T3 queries (hits and misses) on indexes read by A and by B
|
||||
# T4 incremental merge into an existing A-built merged index, by A and B
|
||||
set -u
|
||||
VOL_A="$1"; VOL_B="$2"; OUT="$3"
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd "$HERE/../.." && pwd)"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
docker run --rm --platform linux/amd64 \
|
||||
-v "$VOL_A:/va:ro" -v "$VOL_B:/vb:ro" \
|
||||
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" -v "$OUT:/out" \
|
||||
obikmer-debug:linux-amd64 bash -c '
|
||||
set -u
|
||||
R=x86_64-unknown-linux-musl/release/obikmer
|
||||
mkdir -p /tmp/bin && cp /va/$R /tmp/bin/A && cp /vb/$R /tmp/bin/B
|
||||
A=/tmp/bin/A; B=/tmp/bin/B
|
||||
W=/tmp/w; rm -rf $W; mkdir -p $W/ia $W/ib
|
||||
exec > >(tee /out/report.txt) 2>&1
|
||||
echo "A sha: $(sha256sum $A | cut -c1-16) B sha: $(sha256sum $B | cut -c1-16)"
|
||||
mapfile -t G < <(ls -S /genomes_ro/*.fna.gz | sed -n 8,12p)
|
||||
echo "genomes: ${#G[@]}"; printf " %s\n" "${G[@]##*/}"
|
||||
mkdir -p $W/fa; n=0; for gz in "${G[@]}"; do n=$((n+1)); g=$W/fa/L$n.fasta; zcat "$gz" > "$g"
|
||||
$A index -k 21 -m 9 --theta 0 -p 256 --label L$n -o $W/ia/L$n.idx "$g" 2>$W/ia/L$n.err >/dev/null || { echo "A index failed L$n"; tail -2 $W/ia/L$n.err; }
|
||||
$B index -k 21 -m 9 --theta 0 -p 256 --label L$n -o $W/ib/L$n.idx "$g" 2>$W/ib/L$n.err >/dev/null || { echo "B index failed L$n"; tail -2 $W/ib/L$n.err; }
|
||||
done
|
||||
|
||||
echo; echo "== T0 on-disk bytes, index built by A vs by B"
|
||||
nd=$(diff -rq $W/ia $W/ib | wc -l); nt=$(find $W/ia -type f | wc -l)
|
||||
echo "files differing: $nd / $nt"
|
||||
diff -rq $W/ia $W/ib | sed "s#$W/##g" | awk "{print \$2}" | sed -E "s#/partitions/.*/#/…/#" | sort | uniq -c | sort -rn | head -8
|
||||
|
||||
h() { md5sum | cut -c1-12; }
|
||||
echo; echo "== T1 A-built single-genome indexes: dump by A vs by B"
|
||||
for i in 1 2 3 4 5; do
|
||||
a=$($A dump $W/ia/L$i.idx 2>/dev/null | h); b=$($B dump $W/ia/L$i.idx 2>/dev/null | h)
|
||||
echo "L$i A:$a B:$b $([ "$a" = "$b" ] && echo OK || echo DIFF)"
|
||||
done
|
||||
|
||||
cp_src() { rm -rf $2; mkdir -p $2; for i in $1; do cp -r $3/L$i.idx $2/; done; }
|
||||
echo; echo "== T2 merges of 4 genomes (presence)"
|
||||
cp_src "1 2 3 4" $W/s_aa $W/ia; cp_src "1 2 3 4" $W/s_ba $W/ia; cp_src "1 2 3 4" $W/s_bb $W/ib
|
||||
$A merge --output $W/M_AA --force --force-presence $W/s_aa/L*.idx 2>$W/m_aa.err; echo "M_AA rc=$?"
|
||||
$B merge --output $W/M_BA --force --force-presence $W/s_ba/L*.idx 2>$W/m_ba.err; echo "M_BA rc=$? (B merging A-built sources)"
|
||||
$B merge --output $W/M_BB --force --force-presence $W/s_bb/L*.idx 2>$W/m_bb.err; echo "M_BB rc=$? (B merging B-built sources)"
|
||||
for X in M_AA M_BA M_BB; do for T in A B; do
|
||||
eval "BIN=\$$T"; $BIN dump $W/$X 2>/dev/null > $W/d_${X}_$T.csv
|
||||
echo "dump $X by $T: $(wc -l < $W/d_${X}_$T.csv) lines raw:$(h < $W/d_${X}_$T.csv) sorted:$(LC_ALL=C sort -S 2G $W/d_${X}_$T.csv | h)"
|
||||
done; done
|
||||
|
||||
echo; echo "== T3 queries"
|
||||
head -c 60000 $W/fa/L1.fasta > $W/q.fa
|
||||
python3 - <<PY >> $W/q.fa
|
||||
import random
|
||||
random.seed(1)
|
||||
print(">rand")
|
||||
print("".join(random.choice("ACGT") for _ in range(5000)))
|
||||
PY
|
||||
for X in ia/L1.idx M_AA M_BA M_BB; do for T in A B; do
|
||||
eval "BIN=\$$T"; echo "query $X by $T: $($BIN query $W/$X $W/q.fa 2>/dev/null | h)"
|
||||
done; done
|
||||
|
||||
echo; echo "== T4 incremental merge (existing A-built merged index + a 5th genome)"
|
||||
for T in A B; do
|
||||
eval "BIN=\$$T"; rm -rf $W/inc_$T $W/dst_$T; cp -r $W/M_AA $W/dst_$T; cp -r $W/ia/L5.idx $W/l5_$T.idx
|
||||
$BIN merge --output $W/inc_$T --force --force-presence $W/dst_$T $W/l5_$T.idx 2>$W/inc_$T.err; echo "inc by $T rc=$?"
|
||||
$BIN dump $W/inc_$T 2>/dev/null | h
|
||||
done
|
||||
echo; echo done
|
||||
'
|
||||
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Follow-up to compat_ptrhash.sh: determinism control (A built twice) and the
|
||||
# incremental merge with its stderr shown. Usage: compat_ptrhash2.sh VOL_A VOL_B OUT
|
||||
set -u
|
||||
VOL_A="$1"; VOL_B="$2"; OUT="$3"
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; REPO="$(cd "$HERE/../.." && pwd)"
|
||||
mkdir -p "$OUT"
|
||||
docker run --rm --platform linux/amd64 -v "$VOL_A:/va:ro" -v "$VOL_B:/vb:ro" \
|
||||
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" -v "$OUT:/out" obikmer-debug:linux-amd64 bash -c '
|
||||
set -u
|
||||
R=x86_64-unknown-linux-musl/release/obikmer
|
||||
mkdir -p /tmp/bin && cp /va/$R /tmp/bin/A && cp /vb/$R /tmp/bin/B; A=/tmp/bin/A; B=/tmp/bin/B
|
||||
W=/tmp/w; rm -rf $W; mkdir -p $W/i1 $W/i2 $W/ib $W/fa
|
||||
exec > >(tee /out/report2.txt) 2>&1
|
||||
mapfile -t G < <(ls -S /genomes_ro/*.fna.gz | sed -n 8,12p)
|
||||
n=0; for gz in "${G[@]}"; do n=$((n+1)); zcat "$gz" > $W/fa/L$n.fasta; done
|
||||
h() { md5sum | cut -c1-12; }
|
||||
echo "== C1 determinism control: same binary A, same input, built twice"
|
||||
$A index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/i1/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
|
||||
$A index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/i2/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
|
||||
$B index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/ib/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
|
||||
echo "A vs A: $(diff -rq $W/i1 $W/i2 | wc -l) / $(find $W/i1 -type f | wc -l) files differ"
|
||||
echo "A vs B: $(diff -rq $W/i1 $W/ib | wc -l) / $(find $W/i1 -type f | wc -l) files differ"
|
||||
for f in mphf.bin unitigs.bin evidence.bin unitigs.bin.idx; do
|
||||
p=$(find $W/i1 -name $f | head -1); rel=${p#$W/i1/}
|
||||
echo " $f AvsA: $(find $W/i1 -name $f | while read q; do cmp -s $q $W/i2/${q#$W/i1/} || echo x; done | wc -l)/256 differ AvsB: $(find $W/i1 -name $f | while read q; do cmp -s $q $W/ib/${q#$W/i1/} || echo x; done | wc -l)/256 differ"
|
||||
done
|
||||
echo; echo "== C2 incremental merge, stderr shown"
|
||||
for i in 2 3 4 5; do $A index -k 21 -m 9 --theta 0 -p 256 --label L$i -o $W/i1/L$i.idx $W/fa/L$i.fasta >/dev/null 2>&1; done
|
||||
mkdir -p $W/s; for i in 1 2 3; do cp -r $W/i1/L$i.idx $W/s/; done
|
||||
$A merge --output $W/M3 --force --force-presence $W/s/L*.idx 2>/dev/null; echo "M3 rc=$?"
|
||||
for T in A B; do
|
||||
eval "BIN=\$$T"; rm -rf $W/inc_$T $W/dst_$T $W/l4_$T.idx; cp -r $W/M3 $W/dst_$T; cp -r $W/i1/L4.idx $W/l4_$T.idx
|
||||
$BIN merge --output $W/inc_$T --force --force-presence $W/dst_$T $W/l4_$T.idx 2>$W/inc_$T.err; rc=$?
|
||||
echo "inc by $T rc=$rc"; sed "s/\x1b\[[0-9;]*m//g" $W/inc_$T.err | tail -3
|
||||
[ $rc = 0 ] && echo " dump sorted: $($BIN dump $W/inc_$T 2>/dev/null | LC_ALL=C sort -S 2G | h)"
|
||||
done
|
||||
echo done
|
||||
'
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gen_fasta.py N OUTDIR [GENOME_LEN] — N deterministic synthetic genomes.
|
||||
|
||||
Each genome = unique random sequence + a random subset of a shared fragment
|
||||
pool (some fragments lightly mutated), so a merge both hits k-mers already
|
||||
in the destination and creates new ones, like related genomes would.
|
||||
"""
|
||||
import random, sys
|
||||
|
||||
n, out = int(sys.argv[1]), sys.argv[2]
|
||||
glen = int(sys.argv[3]) if len(sys.argv) > 3 else 150_000
|
||||
seed = int(sys.argv[4]) if len(sys.argv) > 4 else 20261118
|
||||
rng = random.Random(seed)
|
||||
|
||||
def rnd(m):
|
||||
return "".join(rng.choices("ACGT", k=m))
|
||||
|
||||
def mutate(s, rate=0.01):
|
||||
s = list(s)
|
||||
for i in range(len(s)):
|
||||
if rng.random() < rate:
|
||||
s[i] = rng.choice("ACGT")
|
||||
return "".join(s)
|
||||
|
||||
pool = [rnd(2000) for _ in range(300)] # 600 kb shared pool
|
||||
for g in range(n):
|
||||
parts = [rnd(glen)]
|
||||
for f in rng.sample(range(len(pool)), 60):
|
||||
parts.append(mutate(pool[f]) if rng.random() < 0.3 else pool[f])
|
||||
rng.shuffle(parts)
|
||||
with open(f"{out}/G{g:03d}.fasta", "w") as fh:
|
||||
for j, p in enumerate(parts):
|
||||
fh.write(f">G{g:03d}_{j}\n{p}\n")
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gen_fasta_real.py N OUTDIR GENOMES_DIR [WINDOW] [SEED] [MUT]
|
||||
|
||||
N derived genomes built from *real* genomes (GENOMES_DIR/*.fna.gz): each one is
|
||||
a random window of WINDOW bp of a randomly chosen real genome, point-mutated at
|
||||
rate MUT (default 3 %, which alters ~47 % of 21-mers), so a merge both finds
|
||||
k-mers already present in the destination and creates new ones, with realistic
|
||||
sequence statistics (no dedicated degenerate structure).
|
||||
"""
|
||||
import glob, gzip, random, sys
|
||||
|
||||
n, out, gdir = int(sys.argv[1]), sys.argv[2], sys.argv[3]
|
||||
window = int(sys.argv[4]) if len(sys.argv) > 4 else 300_000
|
||||
seed = int(sys.argv[5]) if len(sys.argv) > 5 else 20261118
|
||||
mut = float(sys.argv[6]) if len(sys.argv) > 6 else 0.03
|
||||
rng = random.Random(seed)
|
||||
|
||||
def read_fna(path):
|
||||
seq = []
|
||||
with gzip.open(path, "rt") as fh:
|
||||
for line in fh:
|
||||
if not line.startswith(">"):
|
||||
seq.append(line.strip().upper())
|
||||
return "".join(seq)
|
||||
|
||||
genomes = [read_fna(p) for p in sorted(glob.glob(f"{gdir}/*.fna.gz"))]
|
||||
genomes = [g for g in genomes if len(g) > window]
|
||||
for i in range(n):
|
||||
g = rng.choice(genomes)
|
||||
start = rng.randrange(0, len(g) - window)
|
||||
s = list(g[start:start + window])
|
||||
for j in range(len(s)):
|
||||
if s[j] in "ACGT" and rng.random() < mut:
|
||||
s[j] = rng.choice("ACGT")
|
||||
seq = "".join(c if c in "ACGT" else "A" for c in s)
|
||||
with open(f"{out}/R{i:03d}.fasta", "w") as fh:
|
||||
fh.write(f">R{i:03d}\n")
|
||||
for k in range(0, len(seq), 80):
|
||||
fh.write(seq[k:k + 80] + "\n")
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# monitor.sh PID OUT [INTERVAL_S] — samples mappings / VM / commit figures until PID exits.
|
||||
PID=$1; OUT=$2; DT=${3:-5}
|
||||
printf 'time,maps,vmsize_kb,vmdata_kb,vmrss_kb,threads,committed_as_kb,commit_limit_kb\n' >"$OUT"
|
||||
while kill -0 "$PID" 2>/dev/null; do
|
||||
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null) || break
|
||||
st=$(grep -E '^(VmSize|VmData|VmRSS|Threads):' "/proc/$PID/status" | awk '{printf "%s,", $2}')
|
||||
mi=$(grep -E '^(Committed_AS|CommitLimit):' /proc/meminfo | awk '{printf "%s,", $2}')
|
||||
echo "$(date +%T),$maps,${st}${mi%,}" >>"$OUT"
|
||||
sleep "$DT"
|
||||
done
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
# run.sh — builds obikmer (release, debug symbols) and merges the benchmark presence
|
||||
# indexes in the container while sampling mappings/RSS/commit in parallel.
|
||||
#
|
||||
# Env:
|
||||
# PLATFORM linux/amd64 (default, Rosetta) | linux/arm64
|
||||
# GEN if set: build GEN synthetic indexes (gen_fasta.py) instead of using
|
||||
# the benchmark ones; GENOME_LEN (default 150000), K (21), M (9), PART (256)
|
||||
# REAL if set (with GEN): derive genomes from benchmark/run/genomes (gen_fasta_real.py)
|
||||
# SRC_DIR alternative source tree (default: <repo>/src); VARIANT: volume suffix;
|
||||
# LOCKED=0 to allow Cargo.lock changes (needed when a dep is added)
|
||||
# NSRC number of source indexes to merge (default: all 20)
|
||||
# TARGET gnu (default, glibc) | musl (static, same chain as the release CI)
|
||||
# TOOL none (default) | heaptrack (heaptrack output goes to OUT/)
|
||||
# MAXMAP if set, vm.max_map_count is lowered to this inside the VM for the
|
||||
# run and restored afterwards (needs --privileged; affects the whole
|
||||
# OrbStack VM while it runs)
|
||||
# MALLOC_ARENA_MAX, MALLOC_MMAP_THRESHOLD_, MALLOC_TRIM_THRESHOLD_ passed through
|
||||
# OUT host output directory (default: ./out/<timestamp>)
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd "$HERE/../.." && pwd)"
|
||||
PLATFORM="${PLATFORM:-linux/amd64}"
|
||||
NSRC="${NSRC:-20}"
|
||||
TOOL="${TOOL:-none}"
|
||||
TARGET="${TARGET:-gnu}"
|
||||
OUT="${OUT:-$HERE/out/$(date +%Y%m%d-%H%M%S)}"
|
||||
mkdir -p "$OUT"
|
||||
IMG="obikmer-debug:${PLATFORM//\//-}"
|
||||
VOL="obikmer-debug-target-${PLATFORM//\//-}-${TARGET}${VARIANT:+-$VARIANT}"
|
||||
|
||||
docker run --rm --privileged --platform "$PLATFORM" \
|
||||
-v "${SRC_DIR:-$REPO/src}:/work/src:$([ -n "${SRC_DIR:-}" ] && echo rw || echo ro)" \
|
||||
-v "$REPO/benchmark/run/specimen_index_presence:/idx_ro:ro" \
|
||||
-v "$VOL:/target" \
|
||||
-v "$OUT:/out" \
|
||||
-v "$HERE/monitor.sh:/monitor.sh:ro" \
|
||||
-v "$HERE/gen_fasta.py:/gen_fasta.py:ro" \
|
||||
-v "$HERE/gen_fasta_real.py:/gen_fasta_real.py:ro" \
|
||||
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" \
|
||||
-e CARGO_TARGET_DIR=/target -e CARGO_PROFILE_RELEASE_DEBUG=1 \
|
||||
-e GEN="${GEN:-}" -e LOCKED_FLAG="$([ "${LOCKED:-1}" = 0 ] || echo --locked)" -e SEED="${SEED:-20261118}" -e REAL="${REAL:-}" -e GENOME_LEN="${GENOME_LEN:-150000}" -e K="${K:-21}" -e M="${M:-9}" -e PART="${PART:-256}" -e RUST_LOG="${RUST_LOG:-}" -e NSRC="$NSRC" -e TARGET="$TARGET" -e TOOL="$TOOL" -e MAXMAP="${MAXMAP:-}" \
|
||||
-e MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-}" \
|
||||
-e MALLOC_MMAP_THRESHOLD_="${MALLOC_MMAP_THRESHOLD_:-}" \
|
||||
-e MALLOC_TRIM_THRESHOLD_="${MALLOC_TRIM_THRESHOLD_:-}" \
|
||||
"$IMG" bash -c '
|
||||
set -euo pipefail
|
||||
for v in MALLOC_ARENA_MAX MALLOC_MMAP_THRESHOLD_ MALLOC_TRIM_THRESHOLD_ RUST_LOG; do
|
||||
[ -n "${!v}" ] || unset $v
|
||||
done
|
||||
if [ "$TARGET" = musl ]; then
|
||||
PKG_CONFIG_ALLOW_CROSS=1 cargo zigbuild --release $LOCKED_FLAG --target x86_64-unknown-linux-musl \
|
||||
--manifest-path /work/src/Cargo.toml --bin obikmer 2>&1 | tail -3
|
||||
BIN=/target/x86_64-unknown-linux-musl/release/obikmer
|
||||
else
|
||||
cargo build --release $LOCKED_FLAG --manifest-path /work/src/Cargo.toml --bin obikmer 2>&1 | tail -3
|
||||
BIN=/target/release/obikmer
|
||||
fi
|
||||
file "$BIN" 2>/dev/null | cut -c1-150 || true
|
||||
# merge writes into its sources (lock files…): work on a copy, never on benchmark/
|
||||
rm -rf /tmp/idx && mkdir -p /tmp/idx
|
||||
if [ -n "$GEN" ]; then
|
||||
mkdir -p /tmp/fa
|
||||
if [ -n "$REAL" ]; then python3 /gen_fasta_real.py "$GEN" /tmp/fa /genomes_ro "$GENOME_LEN" "$SEED"
|
||||
else python3 /gen_fasta.py "$GEN" /tmp/fa "$GENOME_LEN" "$SEED"; fi
|
||||
NSRC=$GEN
|
||||
echo "indexing $GEN synthetic genomes (k=$K m=$M p=$PART)…"
|
||||
ls /tmp/fa/*.fasta | xargs -P 8 -I{} sh -c \
|
||||
"b=\$(basename {} .fasta); $BIN index -k $K -m $M --theta 0 -p $PART --label \$b -o /tmp/idx/\$b.idx {} >/dev/null 2>&1 || echo index failed: \$b"
|
||||
else
|
||||
cp -r /idx_ro/. /tmp/idx/
|
||||
fi
|
||||
mapfile -t SRC < <(find /tmp/idx -mindepth 1 -maxdepth 1 -type d | sort | head -n "$NSRC")
|
||||
echo "sources: ${#SRC[@]} target: $TARGET tool: $TOOL max_map_count: $(cat /proc/sys/vm/max_map_count)"
|
||||
if [ -n "$MAXMAP" ]; then
|
||||
OLD=$(cat /proc/sys/vm/max_map_count); sysctl -w vm.max_map_count="$MAXMAP" >/dev/null
|
||||
trap "sysctl -w vm.max_map_count=$OLD >/dev/null" EXIT
|
||||
fi
|
||||
rm -rf /tmp/merged
|
||||
CMD=("$BIN" merge --output /tmp/merged --force --force-presence "${SRC[@]}")
|
||||
if [ "$TOOL" = heaptrack ]; then CMD=(heaptrack -o /out/heaptrack "${CMD[@]}"); fi
|
||||
"${CMD[@]}" 2>/out/merge.stderr &
|
||||
PID=$!
|
||||
/monitor.sh "$PID" /out/monitor.csv 1 &
|
||||
MON=$!
|
||||
set +e; wait "$PID"; RC=$?; set -e
|
||||
wait "$MON" || true
|
||||
echo "merge exit code: $RC" | tee /out/exit_code.txt
|
||||
tail -5 /out/merge.stderr
|
||||
'
|
||||
echo "outputs: $OUT"
|
||||
Executable
+102
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env bash
|
||||
# monitor_pack.sh — 1 Hz sampler for I/O- and memory-bound stages (pack, merge_partitions).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/monitor_pack.sh --pid PID OUT_DIR [OUTPUT_INDEX_DIR] # attach to a running process
|
||||
# scripts/monitor_pack.sh --cmd OUT_DIR [OUTPUT_INDEX_DIR] -- CMD... # start CMD and sample it
|
||||
# env: DT=1 (seconds between samples)
|
||||
#
|
||||
# Writes OUT_DIR/samples.csv (one row per sample) and OUT_DIR/env.txt (context
|
||||
# needed to read it). Linux only. Never reads smaps (it takes mmap_lock and
|
||||
# slows a process with many mappings); only /proc/PID/{status,io,task/*/stat}
|
||||
# and /proc/meminfo.
|
||||
#
|
||||
# Columns of samples.csv
|
||||
# t epoch seconds
|
||||
# rss_anon_kb, rss_file_kb, rss_shmem_kb from /proc/PID/status
|
||||
# threads thread count
|
||||
# n_R n_D n_S threads running / in uninterruptible sleep (I/O wait) / sleeping
|
||||
# (futex, channel, sleep: NOT disk). n_D is the direct count of
|
||||
# workers blocked on I/O.
|
||||
# maps number of mappings (wc -l /proc/PID/maps)
|
||||
# read_bytes write_bytes /proc/PID/io (cumulative, bytes submitted to the
|
||||
# block layer; buffered writes counted at page dirtying)
|
||||
# rchar wchar /proc/PID/io (cumulative, bytes through read()/write() syscalls)
|
||||
# dirty_kb writeback_kb cached_kb mem_avail_kb /proc/meminfo
|
||||
#
|
||||
# Pair it with the process's own log (RUST_LOG=info, plus obisys=debug for the
|
||||
# "activated N worker(s)" lines and obikindex=debug for per-partition timings)
|
||||
# to align stages on wall-clock time: the log timestamps are UTC ISO-8601, `t`
|
||||
# is epoch seconds.
|
||||
set -euo pipefail
|
||||
|
||||
MODE=${1:-}; shift || true
|
||||
case "$MODE" in
|
||||
--pid) PID=${1:?PID}; shift ;;
|
||||
--cmd) ;;
|
||||
*) sed -n 2,12p "$0"; exit 2 ;;
|
||||
esac
|
||||
OUT=${1:?OUT_DIR}; shift
|
||||
IDX=""
|
||||
if [ "${1:-}" != "" ] && [ "${1:-}" != "--" ]; then IDX=$1; shift; fi
|
||||
mkdir -p "$OUT"
|
||||
|
||||
if [ "$MODE" = "--cmd" ]; then
|
||||
[ "${1:-}" = "--" ] || { echo "--cmd needs: -- CMD..." >&2; exit 2; }
|
||||
shift
|
||||
"$@" 2>"$OUT/cmd.stderr" &
|
||||
PID=$!
|
||||
fi
|
||||
|
||||
{
|
||||
echo "date_utc: $(date -u +%FT%TZ)"
|
||||
echo "pid: $PID"
|
||||
echo "nproc: $(nproc)"
|
||||
echo "kernel: $(uname -r)"
|
||||
grep -E 'MemTotal|SwapTotal' /proc/meminfo
|
||||
for k in vm.dirty_ratio vm.dirty_background_ratio vm.dirty_bytes vm.dirty_background_bytes \
|
||||
vm.dirty_expire_centisecs vm.dirty_writeback_centisecs vm.max_map_count; do
|
||||
echo "$k: $(sysctl -n $k 2>/dev/null || echo '?')"
|
||||
done
|
||||
echo "ulimit -n: $(ulimit -n)"
|
||||
echo "cmdline: $(tr '\0' ' ' </proc/$PID/cmdline 2>/dev/null)"
|
||||
if [ -n "$IDX" ]; then
|
||||
echo "output_dir: $IDX"
|
||||
echo "fs_type: $(stat -f -c %T "$IDX" 2>/dev/null || echo '?')"
|
||||
echo "mount: $(df -P "$IDX" 2>/dev/null | tail -1)"
|
||||
command -v beegfs-ctl >/dev/null && echo "beegfs: $(beegfs-ctl --getentryinfo "$IDX" 2>&1 | head -8 | tr '\n' ' ')"
|
||||
fi
|
||||
env | grep -E '^(MALLOC_|RUST_LOG|RAYON_)' || true
|
||||
} >"$OUT/env.txt"
|
||||
|
||||
DT=${DT:-1}
|
||||
printf 't,rss_anon_kb,rss_file_kb,rss_shmem_kb,threads,n_R,n_D,n_S,maps,read_bytes,write_bytes,rchar,wchar,dirty_kb,writeback_kb,cached_kb,mem_avail_kb\n' >"$OUT/samples.csv"
|
||||
|
||||
while [ -d "/proc/$PID" ]; do
|
||||
t=$(date +%s)
|
||||
st=$(awk '/^RssAnon:/{a=$2} /^RssFile:/{f=$2} /^RssShmem:/{s=$2} /^Threads:/{th=$2} END{printf "%s,%s,%s,%s",a,f,s,th}' "/proc/$PID/status" 2>/dev/null) || break
|
||||
# thread states: field 3 of /proc/PID/task/TID/stat, after the ")" closing comm
|
||||
states=$(cat /proc/$PID/task/*/stat 2>/dev/null | sed -E 's/^.*\) (.).*/\1/' | sort | uniq -c \
|
||||
| awk '{c[$2]=$1} END{printf "%d,%d,%d", c["R"]+0, c["D"]+0, c["S"]+0}')
|
||||
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null || echo 0)
|
||||
io=$(awk '/^rchar:/{rc=$2} /^wchar:/{wc=$2} /^read_bytes:/{r=$2} /^write_bytes:/{w=$2} END{printf "%s,%s,%s,%s",r,w,rc,wc}' "/proc/$PID/io" 2>/dev/null || echo ",,,")
|
||||
mi=$(awk '/^Dirty:/{d=$2} /^Writeback:/{w=$2} /^Cached:/{c=$2} /^MemAvailable:/{m=$2} END{printf "%s,%s,%s,%s",d,w,c,m}' /proc/meminfo)
|
||||
echo "$t,$st,$states,$maps,$io,$mi" >>"$OUT/samples.csv"
|
||||
sleep "$DT"
|
||||
done
|
||||
|
||||
if [ "$MODE" = "--cmd" ]; then wait "$PID" || echo "command exit code: $?" >>"$OUT/env.txt"; fi
|
||||
|
||||
# ── summary ──────────────────────────────────────────────────────────────────
|
||||
awk -F, 'NR==1{next}
|
||||
{ n++; if(NR==2){t0=$1; r0=$10; w0=$11}
|
||||
if($2>ma)ma=$2; if($3>mf)mf=$3; if($4>ms)ms=$4; if($5>mt)mt=$5
|
||||
if($6>mR)mR=$6; if($7>mD)mD=$7; if($8>mS)mS=$8; sR+=$6; sD+=$7; sS+=$8
|
||||
if($14>md)md=$14; if($15>mw)mw=$15; if($9>mm)mm=$9
|
||||
t1=$1; r1=$10; w1=$11 }
|
||||
END{ if(!n)exit
|
||||
printf "samples: %d over %ds\n", n, t1-t0
|
||||
printf "peak RssAnon %.1f GB | RssFile %.1f GB | RssShmem %.1f GB | threads %d | maps %d\n", ma/1e6, mf/1e6, ms/1e6, mt, mm
|
||||
printf "threads by state (mean / max): R %.1f / %d | D(I/O wait) %.1f / %d | S %.1f / %d\n", sR/n, mR, sD/n, mD, sS/n, mS
|
||||
printf "peak Dirty %.1f GB | Writeback %.1f GB\n", md/1e6, mw/1e6
|
||||
printf "block I/O over the run: read %.1f GB, write %.1f GB\n", (r1-r0)/1e9, (w1-w0)/1e9 }' "$OUT/samples.csv" | tee "$OUT/summary.txt"
|
||||
@@ -125,4 +125,26 @@ if [ "$FOUND" -ne 1 ]; then
|
||||
fail "no candidate k-mer round-tripped (tried ${#CANDIDATES[@]}) — likely a real regression, not a low-complexity fixture"
|
||||
fi
|
||||
|
||||
echo "PASS: index+query round-trip OK — $N_KMERS kmers indexed, query k-mer '$QUERY_KMER' found"
|
||||
# ── 5. regressions: (a) many partitions against a tiny input leaves most of
|
||||
# them empty (no layer 0 on disk) — must still build; (b) an NCBI-style
|
||||
# `.fna.gz` name must be read, not silently skipped (it used to be
|
||||
# taken for a directory and ignored, yielding an empty index).
|
||||
count_kmers() { # $1 = output index dir, rest = obikmer index args
|
||||
local out="$1"; shift
|
||||
local log="$WORK/regress.log"
|
||||
( cd "$REPO_ROOT/src" && cargo run -q -p obikmer --bin obikmer -- \
|
||||
index -k "$K" -m "$M" --theta 0 -o "$out" "$@" ) > "$log" 2>&1 \
|
||||
|| { cat "$log" >&2; fail "obikmer index $* exited non-zero"; }
|
||||
grep -o '[0-9]* total kmers indexed' "$log" | grep -o '^[0-9]*' || echo 0
|
||||
}
|
||||
|
||||
N_EMPTY="$(count_kmers "$WORK/empty_parts.idx" -p 256 "$WORK/test.fasta")"
|
||||
[ "$N_EMPTY" = "$N_KMERS" ] \
|
||||
|| fail "-p 256 (mostly empty partitions) indexed $N_EMPTY kmers, expected $N_KMERS"
|
||||
|
||||
gzip -c "$WORK/test.fasta" > "$WORK/test.fna.gz"
|
||||
N_GZ="$(count_kmers "$WORK/gz.idx" -p "$PARTITIONS" "$WORK/test.fna.gz")"
|
||||
[ "$N_GZ" = "$N_KMERS" ] \
|
||||
|| fail ".fna.gz input indexed $N_GZ kmers, expected $N_KMERS (input skipped?)"
|
||||
|
||||
echo "PASS: index+query round-trip OK — $N_KMERS kmers indexed, query k-mer '$QUERY_KMER' found; empty-partition and .fna.gz regressions OK"
|
||||
@@ -18,12 +18,14 @@
|
||||
# 4. runs the merge scenarios:
|
||||
# s1_presence / s1_count merge A B C D
|
||||
# s3_dense merge A B C D --dense (presence)
|
||||
# Scenario 2 (merge A B -> AB, then merge AB C D: multi-genome source,
|
||||
# multi-layer destination) is NOT included: the current code cannot
|
||||
# re-merge a merge output — its matrices are packed (sparse or dense)
|
||||
# at finalize, and `MatrixBuilder::resume` expects the columnar
|
||||
# `meta.json` layout (ENOENT). Known bug, to be fixed separately; see
|
||||
# DevDocMD/implementation/merge_mmap_audit.md, "Known issue".
|
||||
# Scenario 2 (merge A B -> AB, then merge AB C D: a packed merge output
|
||||
# re-merged as a *source*) is checked relatively, not against the
|
||||
# golden dumps: its table (k-mer x genome label) must equal s1_presence's,
|
||||
# whatever the column order and layer placement. A packed index cannot
|
||||
# be a merge *destination* (its matrices are no longer column-by-column,
|
||||
# see `can_be_destination` in obikmerge); `choose_base` therefore picks
|
||||
# an unpacked source, and merging only packed indexes is refused with a
|
||||
# clear error, which this script also checks.
|
||||
# 5. dumps every result with `obikmer dump --debug` (partition + layer
|
||||
# columns included, so a k-mer moving between layers is caught), keeps
|
||||
# the CSV header first and sorts the rows (LC_ALL=C)
|
||||
@@ -147,6 +149,35 @@ run s3_dense merge --dense -o "$WORK/s3_dense.idx" \
|
||||
"$WORK/A.pres.idx" "$WORK/B.pres.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
|
||||
dump s3_dense "$WORK/s3_dense.idx"
|
||||
|
||||
# scenario 2: re-merge a packed merge output as a source (checked against s1)
|
||||
run s2_ab merge -o "$WORK/s2_ab.idx" "$WORK/A.pres.idx" "$WORK/B.pres.idx"
|
||||
run s2 merge -o "$WORK/s2.idx" "$WORK/s2_ab.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
|
||||
for N in s1_pres s2; do
|
||||
"$BIN" dump "$WORK/$N.idx" > "$WORK/plain_$N.csv" 2> "$WORK/dump_plain_$N.log" \
|
||||
|| { cat "$WORK/dump_plain_$N.log" >&2; fail "obikmer dump $N exited non-zero"; }
|
||||
done
|
||||
cat > "$WORK/compare_tables.py" <<'PY'
|
||||
import csv, sys
|
||||
def table(path):
|
||||
rows = list(csv.reader(open(path)))
|
||||
head = rows[0]
|
||||
return {r[0]: {head[i]: r[i] for i in range(1, len(head))} for r in rows[1:]}
|
||||
a, b = table(sys.argv[1]), table(sys.argv[2])
|
||||
if a != b:
|
||||
print(f"k-mers: {len(a)} vs {len(b)}; only in flat: {len(set(a) - set(b))}; "
|
||||
f"only in re-merge: {len(set(b) - set(a))}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
PY
|
||||
python3 "$WORK/compare_tables.py" "$WORK/plain_s1_pres.csv" "$WORK/plain_s2.csv" \
|
||||
|| fail "scenario 2 (re-merge of a packed output) differs from the flat merge"
|
||||
echo "ok s2 (re-merge of a packed output) equals the flat merge"
|
||||
if "$BIN" merge -o "$WORK/s2_bad.idx" "$WORK/s2_ab.idx" "$WORK/s1_pres.idx" > "$WORK/s2_bad.log" 2>&1; then
|
||||
fail "merging only packed indexes should be refused"
|
||||
fi
|
||||
grep -q "none of the sources can be the destination" "$WORK/s2_bad.log" \
|
||||
|| { cat "$WORK/s2_bad.log" >&2; fail "refusal of an all-packed merge lacks the expected message"; }
|
||||
echo "ok merge of only packed indexes is refused with a clear message"
|
||||
|
||||
# sanity: no dump may be empty (header + at least one row)
|
||||
for f in "$OUT"/*.csv; do
|
||||
[ "$(wc -l < "$f")" -gt 1 ] || fail "empty dump: $(basename "$f")"
|
||||
|
||||
Executable
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env bash
|
||||
# watch_merge.sh PID OUT.csv — samples a running process and kills it past a threshold.
|
||||
#
|
||||
# Every INTERVAL seconds appends one CSV row: number of memory mappings
|
||||
# (lines of /proc/PID/maps), VmSize, VmData, VmRSS, threads, and the
|
||||
# system-wide Committed_AS / CommitLimit. If the mapping count exceeds
|
||||
# MAPS_MAX, VmRSS exceeds RSS_MAX_GB, or VmSize exceeds VMSIZE_MAX_GB (stay
|
||||
# below `ulimit -v`: past it, allocations fail instead of the guard firing), the process receives SIGTERM (then
|
||||
# SIGKILL after 30 s) and the reason is written to OUT.csv.reason — a guard
|
||||
# so a runaway merge ends with a clean message instead of a failed
|
||||
# allocation (or the OOM killer on a shared node).
|
||||
#
|
||||
# Reading /proc/PID/maps is itself slow when the process has hundreds of
|
||||
# thousands of mappings (it held the kernel's mmap lock in earlier
|
||||
# measurements): keep INTERVAL >= 5. It never reads /proc/PID/smaps.
|
||||
#
|
||||
# Usage: scripts/watch_merge.sh PID OUT.csv
|
||||
# env: INTERVAL=5 MAPS_MAX=500000 RSS_MAX_GB=400 VMSIZE_MAX_GB=440
|
||||
#
|
||||
# Typical use (Linux only):
|
||||
# nohup scripts/watch_merge.sh "$(pgrep -f 'obikmer merge' | head -1)" /tmp/watch.csv &
|
||||
set -u
|
||||
PID=${1:?usage: watch_merge.sh PID OUT.csv}
|
||||
OUT=${2:?usage: watch_merge.sh PID OUT.csv}
|
||||
INTERVAL=${INTERVAL:-5}
|
||||
MAPS_MAX=${MAPS_MAX:-500000}
|
||||
RSS_MAX_KB=$(( ${RSS_MAX_GB:-400} * 1024 * 1024 ))
|
||||
VMSIZE_MAX_KB=$(( ${VMSIZE_MAX_GB:-440} * 1024 * 1024 ))
|
||||
|
||||
[ -r "/proc/$PID/maps" ] || { echo "cannot read /proc/$PID/maps" >&2; exit 1; }
|
||||
printf 'time,maps,vmsize_kb,vmdata_kb,vmrss_kb,threads,committed_as_kb,commit_limit_kb\n' >"$OUT"
|
||||
|
||||
while kill -0 "$PID" 2>/dev/null; do
|
||||
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null) || break
|
||||
st=$(grep -E '^(VmSize|VmData|VmRSS|Threads):' "/proc/$PID/status" 2>/dev/null | awk '{printf "%s,", $2}')
|
||||
rss=$(grep '^VmRSS:' "/proc/$PID/status" 2>/dev/null | awk '{print $2}')
|
||||
vsz=$(grep '^VmSize:' "/proc/$PID/status" 2>/dev/null | awk '{print $2}')
|
||||
mi=$(grep -E '^(Committed_AS|CommitLimit):' /proc/meminfo | awk '{printf "%s,", $2}')
|
||||
echo "$(date +%T),$maps,${st}${mi%,}" >>"$OUT"
|
||||
|
||||
reason=""
|
||||
[ "$maps" -gt "$MAPS_MAX" ] && reason="mappings $maps > MAPS_MAX $MAPS_MAX"
|
||||
[ -n "${rss:-}" ] && [ "$rss" -gt "$RSS_MAX_KB" ] && reason="VmRSS ${rss} kB > RSS_MAX_GB"
|
||||
[ -n "${vsz:-}" ] && [ "$vsz" -gt "$VMSIZE_MAX_KB" ] && reason="VmSize ${vsz} kB > VMSIZE_MAX_GB"
|
||||
if [ -n "$reason" ]; then
|
||||
echo "$(date +%T) GUARD: $reason — sending SIGTERM to $PID" | tee "$OUT.reason" >&2
|
||||
kill -TERM "$PID" 2>/dev/null
|
||||
for _ in $(seq 30); do kill -0 "$PID" 2>/dev/null || exit 0; sleep 1; done
|
||||
kill -KILL "$PID" 2>/dev/null
|
||||
exit 0
|
||||
fi
|
||||
sleep "$INTERVAL"
|
||||
done
|
||||
Generated
+10
-19
@@ -422,7 +422,7 @@ version = "3.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
|
||||
dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -758,7 +758,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"windows-sys 0.61.2",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1040,7 +1040,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
|
||||
dependencies = [
|
||||
"hermit-abi",
|
||||
"libc",
|
||||
"windows-sys 0.61.2",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1408,10 +1408,8 @@ version = "0.1.0"
|
||||
dependencies = [
|
||||
"crossbeam-channel",
|
||||
"hashbrown 0.17.1",
|
||||
"obifastwrite",
|
||||
"obikseq",
|
||||
"rayon",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1492,14 +1490,10 @@ dependencies = [
|
||||
"ndarray",
|
||||
"niffler",
|
||||
"obicompactvec",
|
||||
"obidebruinj",
|
||||
"obikentropy",
|
||||
"obikseq",
|
||||
"obipipeline",
|
||||
"obiread",
|
||||
"obiskio",
|
||||
"obisys",
|
||||
"obitaxonomy",
|
||||
"ptr_hash",
|
||||
"rayon",
|
||||
"serde",
|
||||
@@ -1537,7 +1531,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "obikmer"
|
||||
version = "1.4.4"
|
||||
version = "1.4.5"
|
||||
dependencies = [
|
||||
"clap",
|
||||
"csv",
|
||||
@@ -1556,7 +1550,6 @@ dependencies = [
|
||||
"obikrope",
|
||||
"obikselect",
|
||||
"obikseq",
|
||||
"obiksession",
|
||||
"obikstats",
|
||||
"obipipeline",
|
||||
"obiread",
|
||||
@@ -1565,6 +1558,7 @@ dependencies = [
|
||||
"rayon",
|
||||
"serde",
|
||||
"serde_yaml",
|
||||
"tikv-jemallocator",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
]
|
||||
@@ -1600,7 +1594,6 @@ dependencies = [
|
||||
"obikseq",
|
||||
"obipipeline",
|
||||
"obiread",
|
||||
"obiskbuilder",
|
||||
"obiskio",
|
||||
"obisys",
|
||||
"petgraph",
|
||||
@@ -1646,7 +1639,6 @@ dependencies = [
|
||||
name = "obikrope"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"criterion2",
|
||||
]
|
||||
|
||||
@@ -1724,7 +1716,6 @@ name = "obiskbuilder"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"criterion2",
|
||||
"lazy_static",
|
||||
"obikentropy",
|
||||
"obikrope",
|
||||
"obikseq",
|
||||
@@ -1984,9 +1975,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "ptr_hash"
|
||||
version = "2.1.2"
|
||||
version = "2.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b22cf1be774892680e056d3387db0dcaa4fd46964b270671ef661e127186b969"
|
||||
checksum = "0d2381547e2e36c72a455c1cc418d5c60ecd6e94f4819d7560456108e99a4efa"
|
||||
dependencies = [
|
||||
"bitvec",
|
||||
"cacheline-ef",
|
||||
@@ -2278,7 +2269,7 @@ dependencies = [
|
||||
"errno",
|
||||
"libc",
|
||||
"linux-raw-sys",
|
||||
"windows-sys 0.61.2",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2610,7 +2601,7 @@ dependencies = [
|
||||
"getrandom 0.4.3",
|
||||
"once_cell",
|
||||
"rustix",
|
||||
"windows-sys 0.61.2",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3015,7 +3006,7 @@ version = "0.1.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
|
||||
dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
@@ -18,6 +18,7 @@ mod packed;
|
||||
mod pairwise;
|
||||
mod persistent;
|
||||
mod sparse;
|
||||
mod sparse_columnar;
|
||||
|
||||
pub use builder::PersistentBitMatrixBuilder;
|
||||
pub use group_ops::batch_presence_counts;
|
||||
|
||||
@@ -105,7 +105,7 @@ impl PackedBitMatrix {
|
||||
/// Reads just the `n_cols` field from an existing packed matrix's header,
|
||||
/// without mapping the file. Used by `pack_bit_matrix` to tell a genuinely
|
||||
/// complete pack from a stale one that predates a later column-widening.
|
||||
fn packed_bit_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
||||
pub(super) fn packed_bit_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
||||
let mut f = File::open(path)?;
|
||||
let mut header = [0u8; PBMX_HEADER];
|
||||
f.read_exact(&mut header)?;
|
||||
|
||||
@@ -31,12 +31,12 @@ use ndarray::{Array1, Array2};
|
||||
use crate::traits::{BitPartials, ColumnWeights};
|
||||
use crate::eliasfano::{EliasFano, EliasFanoBuilder};
|
||||
use crate::fixedintvec::{PersistentFixedIntVec, PersistentFixedIntVecBuilder, bit_width_for_range};
|
||||
use crate::meta::field;
|
||||
use crate::meta::{MatrixMeta, field};
|
||||
use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
||||
|
||||
use super::PersistentBitMatrix;
|
||||
use super::packed::{PackedBitMatrix, pack_bit_matrix};
|
||||
use crate::views::nonzero_triples;
|
||||
use super::col_path;
|
||||
use super::packed::{PackedBitMatrix, pack_bit_matrix, packed_bit_matrix_n_cols};
|
||||
|
||||
fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") }
|
||||
fn singleton_path(dir: &Path) -> PathBuf { dir.join("singleton.pfiv") }
|
||||
@@ -499,12 +499,12 @@ impl PersistentSparseBitMatrixBuilder {
|
||||
/// existing index.
|
||||
///
|
||||
/// Processes rows in batches of [`BUILD_FROM_DENSE_BATCH`] instead of
|
||||
/// one row at a time: `nonzero_triples(..., sorted_slots=true)` walks
|
||||
/// each column *once per batch*, sequentially — matching how
|
||||
/// `PersistentBitMatrix::nonzero_iter` already batches slots for
|
||||
/// queries — rather than `fill_row`'s row-major loop, which re-reads
|
||||
/// every column with one `get()` per cell and jumps between all
|
||||
/// `n_cols` column files on every single row.
|
||||
/// one row at a time: [`BitSliceView::for_each_set_in`] walks each
|
||||
/// column *once per batch*, word by word, visiting only the set bits —
|
||||
/// rather than `fill_row`'s row-major loop, which re-reads every column
|
||||
/// with one `get()` per cell and jumps between all `n_cols` columns on
|
||||
/// every single row. Columns are visited in ascending order, so each
|
||||
/// row's genome list comes out sorted, as `push_row` requires.
|
||||
pub fn build_from_dense(dense: &PersistentBitMatrix, dir: &Path) -> io::Result<Self> {
|
||||
let n = dense.n();
|
||||
let n_cols = dense.n_cols();
|
||||
@@ -513,9 +513,10 @@ impl PersistentSparseBitMatrixBuilder {
|
||||
let mut start = 0;
|
||||
while start < n {
|
||||
let end = (start + BUILD_FROM_DENSE_BATCH).min(n);
|
||||
let slots: Vec<usize> = (start..end).collect();
|
||||
for (row, c, _) in nonzero_triples(&slots, n_cols, |c| dense.col_view(c), true) {
|
||||
row_buffers[row].push(c as u32);
|
||||
for c in 0..n_cols {
|
||||
dense.col_view(c).for_each_set_in(start..end, |slot| {
|
||||
row_buffers[slot - start].push(c as u32);
|
||||
});
|
||||
}
|
||||
for buf in row_buffers.iter_mut().take(end - start) {
|
||||
builder.push_row(buf);
|
||||
@@ -620,24 +621,38 @@ impl BitPartials for PersistentSparseBitMatrix {
|
||||
/// mid-conversion leaves a previous, still valid format. Idempotent: does
|
||||
/// nothing if `is_multi.prsb` already exists.
|
||||
///
|
||||
/// Always goes through the dense packed form, even right after a build.
|
||||
/// That costs one extra write and read of the dense matrix, but keeps the
|
||||
/// number of mappings at one instead of one per genome column.
|
||||
/// Columnar data is transposed straight into the sparse format, a window of
|
||||
/// rows at a time, one column file open at a time
|
||||
/// ([`PersistentSparseBitMatrixBuilder::build_from_columnar`]): no dense
|
||||
/// intermediate is written, and a worker holds no mapping of the columns
|
||||
/// whatever their number. A dense `matrix.pbmx` is only read when it is the
|
||||
/// current form of the layer: no columnar data pending, or a `matrix.pbmx`
|
||||
/// that already has as many columns as the columnar metadata says.
|
||||
pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> {
|
||||
if is_multi_path(dir).exists() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Columnar → dense packed first, then dense packed → sparse. Opening the
|
||||
// columnar matrix directly would map every column file at once (one
|
||||
// mapping per genome, unbounded); `pack_bit_matrix` instead
|
||||
// concatenates the column files one at a time, unmapped, and leaves a
|
||||
// single `matrix.pbmx` to read from (one mapping). It is a no-op when
|
||||
// the layer is already dense-packed, and re-packs a stale `matrix.pbmx`
|
||||
// that predates newer columnar data. Every intermediate state is a
|
||||
// valid format, so a crash leaves something this function resumes.
|
||||
pack_bit_matrix(dir)?;
|
||||
let packed_path = dir.join("matrix.pbmx");
|
||||
|
||||
if let Ok(meta) = MatrixMeta::load(dir) {
|
||||
if packed_bit_matrix_n_cols(&packed_path).ok() != Some(meta.n_cols) {
|
||||
// Columnar data is the freshest form (any `matrix.pbmx` here is
|
||||
// stale, e.g. copied from a merge base before the layer was
|
||||
// widened): transpose it directly.
|
||||
PersistentSparseBitMatrixBuilder::build_from_columnar(dir, &meta)?.close()?;
|
||||
for c in 0..meta.n_cols {
|
||||
fs::remove_file(col_path(dir, c))?;
|
||||
}
|
||||
fs::remove_file(dir.join("meta.json"))?;
|
||||
let _ = fs::remove_file(&packed_path);
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
|
||||
// Dense packed form is the current one. `pack_bit_matrix` is a no-op when
|
||||
// there is no columnar data left, and otherwise only drops the
|
||||
// now-redundant column files.
|
||||
pack_bit_matrix(dir)?;
|
||||
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?);
|
||||
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
//! Building the sparse presence matrix straight from the columnar
|
||||
//! `col_*.pbiv` files, with no dense intermediate and no mapping.
|
||||
//!
|
||||
//! Each column is opened, a window of it is read with `pread`, and it is
|
||||
//! closed again: a worker holds at most one column file at a time,
|
||||
//! whatever the number of genomes. See [`crate::transpose`].
|
||||
|
||||
use std::fs::File;
|
||||
use std::io;
|
||||
use std::os::unix::fs::FileExt;
|
||||
use std::path::Path;
|
||||
|
||||
use crate::bitvec::MAGIC;
|
||||
use crate::meta::MatrixMeta;
|
||||
use crate::transpose::{DEFAULT_MAX_HITS, DEFAULT_WINDOW_ROWS, Hit, transpose_windows};
|
||||
use crate::views::BitSliceView;
|
||||
|
||||
use super::PersistentSparseBitMatrixBuilder;
|
||||
use super::col_path;
|
||||
|
||||
/// `PBIV` header: magic(4) + pad(4) + n(8).
|
||||
const PBIV_HEADER: usize = 16;
|
||||
|
||||
/// Reads the bits of `path` for rows `start..end` (`start` a multiple of 64)
|
||||
/// into `words`, after checking the header against `n`.
|
||||
fn read_bit_window(
|
||||
path: &Path,
|
||||
n: usize,
|
||||
start: usize,
|
||||
end: usize,
|
||||
bytes: &mut Vec<u8>,
|
||||
words: &mut Vec<u64>,
|
||||
) -> io::Result<()> {
|
||||
debug_assert_eq!(start % 64, 0);
|
||||
let file = File::open(path)?;
|
||||
let mut header = [0u8; PBIV_HEADER];
|
||||
file.read_exact_at(&mut header, 0)?;
|
||||
if header[..4] != MAGIC || u64::from_le_bytes(header[8..16].try_into().unwrap()) as usize != n {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
format!("{}: not a PBIV column of {n} rows", path.display()),
|
||||
));
|
||||
}
|
||||
let nw = (end - start).div_ceil(64);
|
||||
bytes.clear();
|
||||
bytes.resize(nw * 8, 0);
|
||||
file.read_exact_at(bytes, (PBIV_HEADER + start / 8) as u64)?;
|
||||
words.clear();
|
||||
words.extend(bytes.chunks_exact(8).map(|b| u64::from_le_bytes(b.try_into().unwrap())));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl PersistentSparseBitMatrixBuilder {
|
||||
/// Builds the sparse matrix of the columnar layer in `dir` (`meta.json`
|
||||
/// plus `col_*.pbiv`), writing the sparse files into `dir`. Same rows,
|
||||
/// same genome order and so same bytes as going through a dense
|
||||
/// `matrix.pbmx` (`build_from_dense`), without ever writing it.
|
||||
pub fn build_from_columnar(dir: &Path, meta: &MatrixMeta) -> io::Result<Self> {
|
||||
Self::build_from_columnar_with(dir, meta, DEFAULT_WINDOW_ROWS, DEFAULT_MAX_HITS)
|
||||
}
|
||||
|
||||
/// [`build_from_columnar`](Self::build_from_columnar) with explicit window
|
||||
/// size (rows) and non-zero-cell cap; see [`crate::transpose`].
|
||||
pub fn build_from_columnar_with(
|
||||
dir: &Path,
|
||||
meta: &MatrixMeta,
|
||||
window_rows: usize,
|
||||
max_hits: usize,
|
||||
) -> io::Result<Self> {
|
||||
let mut builder = Self::new(meta.n, meta.n_cols, dir)?;
|
||||
let mut bytes: Vec<u8> = Vec::new();
|
||||
let mut words: Vec<u64> = Vec::new();
|
||||
transpose_windows(
|
||||
meta.n,
|
||||
meta.n_cols,
|
||||
window_rows,
|
||||
max_hits,
|
||||
(),
|
||||
|c, start, end, hits: &mut Vec<Hit<()>>| {
|
||||
read_bit_window(&col_path(dir, c), meta.n, start, end, &mut bytes, &mut words)?;
|
||||
BitSliceView::new(&words, end - start).for_each_set_in(0..end - start, |r| {
|
||||
hits.push((r as u32, c as u32, ()));
|
||||
});
|
||||
Ok(())
|
||||
},
|
||||
|_, batch| {
|
||||
for i in 0..batch.len() {
|
||||
builder.push_row(batch.cols(i));
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
)?;
|
||||
Ok(builder)
|
||||
}
|
||||
}
|
||||
@@ -298,7 +298,7 @@ impl PackedCompactIntMatrix {
|
||||
/// without mapping the file. Used by `pack_compact_int_matrix` to tell a
|
||||
/// genuinely complete pack from a stale one that predates a later
|
||||
/// column-widening.
|
||||
fn packed_int_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
||||
pub(crate) fn packed_int_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
||||
let mut f = File::open(path)?;
|
||||
let mut header = [0u8; PCMX_HEADER];
|
||||
f.read_exact(&mut header)?;
|
||||
|
||||
@@ -15,11 +15,14 @@ mod mmap_file;
|
||||
mod rankselect;
|
||||
mod reader;
|
||||
mod sparse_intmatrix;
|
||||
mod sparse_intmatrix_columnar;
|
||||
mod storage_kind;
|
||||
mod tempbitvec;
|
||||
mod tempintvec;
|
||||
mod transpose;
|
||||
pub mod traits;
|
||||
mod views;
|
||||
mod views_scan;
|
||||
|
||||
pub use bitmatrix::{
|
||||
PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentSparseBitMatrix,
|
||||
|
||||
@@ -33,6 +33,7 @@ use ndarray::{Array1, Array2};
|
||||
use crate::bitmatrix::RowRank;
|
||||
use crate::builder::PersistentCompactIntVecBuilder;
|
||||
use crate::eliasfano::{EliasFano, EliasFanoBuilder};
|
||||
use crate::meta::MatrixMeta;
|
||||
use crate::reader::PersistentCompactIntVec;
|
||||
use crate::traits::{BitPartials, ColumnWeights, CountPartials};
|
||||
use crate::{PersistentIntMatrix, PersistentSparseBitMatrix, PersistentSparseBitMatrixBuilder};
|
||||
@@ -618,22 +619,34 @@ impl PersistentSparseCompactIntMatrixBuilder {
|
||||
// ── pack_sparse_compact_int_matrix ──────────────────────────────────────────
|
||||
|
||||
/// Converts a layer's count matrix (columnar `col_*.pciv` or dense
|
||||
/// `matrix.pcmx`) to the sparse on-disk format in place, through the dense
|
||||
/// packed form, exactly like [`crate::pack_sparse_bit_matrix`]. Old-format
|
||||
/// files are removed only once the new format is fully written.
|
||||
/// Idempotent: a no-op if the sparse format is already present.
|
||||
/// `matrix.pcmx`) to the sparse on-disk format in place, exactly like
|
||||
/// [`crate::pack_sparse_bit_matrix`]: columnar data is transposed straight
|
||||
/// into the sparse format, window by window, one column file open at a time
|
||||
/// (no dense intermediate, no mapping); a dense `matrix.pcmx` is read only
|
||||
/// when it is the current form. Old-format files are removed only once the
|
||||
/// new format is fully written. Idempotent: a no-op if the sparse format is
|
||||
/// already present.
|
||||
pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> {
|
||||
use crate::intmatrix::{PackedCompactIntMatrix, pack_compact_int_matrix};
|
||||
use crate::intmatrix::{PackedCompactIntMatrix, pack_compact_int_matrix, packed_int_matrix_n_cols};
|
||||
|
||||
if is_present(dir) {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Columnar → dense packed first, then dense packed → sparse: one
|
||||
// mapping (`matrix.pcmx`) instead of one per column file. See
|
||||
// `pack_sparse_bit_matrix` for the rationale and crash-safety argument.
|
||||
pack_compact_int_matrix(dir)?;
|
||||
let packed_path = dir.join("matrix.pcmx");
|
||||
|
||||
if let Ok(meta) = MatrixMeta::load(dir) {
|
||||
if packed_int_matrix_n_cols(&packed_path).ok() != Some(meta.n_cols) {
|
||||
PersistentSparseCompactIntMatrixBuilder::build_from_columnar(dir, &meta)?.close()?;
|
||||
for c in 0..meta.n_cols {
|
||||
fs::remove_file(crate::intmatrix::col_path(dir, c))?;
|
||||
}
|
||||
fs::remove_file(dir.join("meta.json"))?;
|
||||
let _ = fs::remove_file(&packed_path);
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
|
||||
pack_compact_int_matrix(dir)?;
|
||||
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?);
|
||||
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
//! Building the sparse count matrix straight from the columnar
|
||||
//! `col_*.pciv` files, with no dense intermediate and no mapping. Counterpart
|
||||
//! of `bitmatrix::sparse_columnar`; see [`crate::transpose`].
|
||||
//!
|
||||
//! A `PCIV` column is `header(40) | primary(n bytes) | overflow(n_overflow
|
||||
//! × 12 B, sorted by slot) | index`. A primary byte `255` is a sentinel: the
|
||||
//! true value (≥ 255) is the overflow entry of that slot. A window therefore
|
||||
//! reads its primary bytes, then looks up the overflow entries of its
|
||||
//! sentinels (binary search for the first one, one contiguous read for the
|
||||
//! rest).
|
||||
|
||||
use std::fs::File;
|
||||
use std::io;
|
||||
use std::os::unix::fs::FileExt;
|
||||
use std::path::Path;
|
||||
|
||||
use crate::format::{HEADER_SIZE, MAGIC, OVERFLOW_ENTRY_SIZE, parse_overflow_entry};
|
||||
use crate::intmatrix::col_path;
|
||||
use crate::meta::MatrixMeta;
|
||||
use crate::sparse_intmatrix::PersistentSparseCompactIntMatrixBuilder;
|
||||
use crate::transpose::{DEFAULT_MAX_HITS, DEFAULT_WINDOW_ROWS, Hit, transpose_windows};
|
||||
|
||||
fn invalid(path: &Path, what: &str) -> io::Error {
|
||||
io::Error::new(io::ErrorKind::InvalidData, format!("{}: {what}", path.display()))
|
||||
}
|
||||
|
||||
/// Appends the non-zero cells of column `c` (file `path`) for rows
|
||||
/// `start..end` to `hits`, as `(row - start, c, value)`.
|
||||
fn read_int_window(
|
||||
path: &Path,
|
||||
n: usize,
|
||||
c: usize,
|
||||
start: usize,
|
||||
end: usize,
|
||||
primary: &mut Vec<u8>,
|
||||
hits: &mut Vec<Hit<u32>>,
|
||||
) -> io::Result<()> {
|
||||
let file = File::open(path)?;
|
||||
let mut header = [0u8; HEADER_SIZE];
|
||||
file.read_exact_at(&mut header, 0)?;
|
||||
if header[..4] != MAGIC || u64::from_le_bytes(header[8..16].try_into().unwrap()) as usize != n {
|
||||
return Err(invalid(path, &format!("not a PCIV column of {n} rows")));
|
||||
}
|
||||
let n_overflow = u64::from_le_bytes(header[16..24].try_into().unwrap()) as usize;
|
||||
|
||||
primary.clear();
|
||||
primary.resize(end - start, 0);
|
||||
file.read_exact_at(primary, (HEADER_SIZE + start) as u64)?;
|
||||
|
||||
// Slots (relative to `start`) whose primary byte is the 255 sentinel,
|
||||
// pushed to `hits` with a placeholder and patched below.
|
||||
let first_hit = hits.len();
|
||||
let mut n_sentinels = 0usize;
|
||||
let mut chunks = primary.chunks_exact(8);
|
||||
let mut base = 0usize;
|
||||
for chunk in &mut chunks {
|
||||
if u64::from_le_bytes(chunk.try_into().unwrap()) != 0 {
|
||||
for (j, &b) in chunk.iter().enumerate() {
|
||||
if b != 0 {
|
||||
n_sentinels += (b == 255) as usize;
|
||||
hits.push(((base + j) as u32, c as u32, b as u32));
|
||||
}
|
||||
}
|
||||
}
|
||||
base += 8;
|
||||
}
|
||||
for (j, &b) in chunks.remainder().iter().enumerate() {
|
||||
if b != 0 {
|
||||
n_sentinels += (b == 255) as usize;
|
||||
hits.push(((base + j) as u32, c as u32, b as u32));
|
||||
}
|
||||
}
|
||||
if n_sentinels == 0 {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Overflow entries of this window: first entry with slot >= start, found
|
||||
// by binary search over the on-disk section, then one contiguous read.
|
||||
let ov_base = (HEADER_SIZE + n) as u64;
|
||||
let slot_at = |i: usize| -> io::Result<usize> {
|
||||
let mut b = [0u8; 8];
|
||||
file.read_exact_at(&mut b, ov_base + (i * OVERFLOW_ENTRY_SIZE) as u64)?;
|
||||
Ok(u64::from_le_bytes(b) as usize)
|
||||
};
|
||||
let (mut lo, mut hi) = (0usize, n_overflow);
|
||||
while lo < hi {
|
||||
let mid = lo + (hi - lo) / 2;
|
||||
if slot_at(mid)? < start {
|
||||
lo = mid + 1;
|
||||
} else {
|
||||
hi = mid;
|
||||
}
|
||||
}
|
||||
if lo + n_sentinels > n_overflow {
|
||||
return Err(invalid(path, "fewer overflow entries than sentinel bytes"));
|
||||
}
|
||||
let mut raw = vec![0u8; n_sentinels * OVERFLOW_ENTRY_SIZE];
|
||||
file.read_exact_at(&mut raw, ov_base + (lo * OVERFLOW_ENTRY_SIZE) as u64)?;
|
||||
let mut k = 0usize;
|
||||
for h in &mut hits[first_hit..] {
|
||||
if h.2 == 255 {
|
||||
let (slot, value) = parse_overflow_entry(&raw, 0, k);
|
||||
if slot != start + h.0 as usize {
|
||||
return Err(invalid(path, "overflow entry does not match its sentinel slot"));
|
||||
}
|
||||
h.2 = value;
|
||||
k += 1;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl PersistentSparseCompactIntMatrixBuilder {
|
||||
/// Builds the sparse count matrix of the columnar layer in `dir`
|
||||
/// (`meta.json` plus `col_*.pciv`), writing the sparse files into `dir`.
|
||||
/// Same rows, same cells and so same bytes as going through a dense
|
||||
/// `matrix.pcmx` (`build_from_dense`), without ever writing it.
|
||||
pub fn build_from_columnar(dir: &Path, meta: &MatrixMeta) -> io::Result<Self> {
|
||||
Self::build_from_columnar_with(dir, meta, DEFAULT_WINDOW_ROWS, DEFAULT_MAX_HITS)
|
||||
}
|
||||
|
||||
/// [`build_from_columnar`](Self::build_from_columnar) with explicit window
|
||||
/// size (rows) and non-zero-cell cap; see [`crate::transpose`].
|
||||
pub fn build_from_columnar_with(
|
||||
dir: &Path,
|
||||
meta: &MatrixMeta,
|
||||
window_rows: usize,
|
||||
max_hits: usize,
|
||||
) -> io::Result<Self> {
|
||||
let mut builder = Self::new(meta.n, meta.n_cols, dir)?;
|
||||
let mut primary: Vec<u8> = Vec::new();
|
||||
transpose_windows(
|
||||
meta.n,
|
||||
meta.n_cols,
|
||||
window_rows,
|
||||
max_hits,
|
||||
0u32,
|
||||
|c, start, end, hits| read_int_window(&col_path(dir, c), meta.n, c, start, end, &mut primary, hits),
|
||||
|_, batch| {
|
||||
for i in 0..batch.len() {
|
||||
builder.push_row(batch.cols(i), batch.vals(i));
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
)?;
|
||||
Ok(builder)
|
||||
}
|
||||
}
|
||||
@@ -249,6 +249,20 @@ fn partial_threshold_jaccard_consistent() {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn threshold_hamming_counts_kmers_present_in_one_genome_only() {
|
||||
// At threshold 2 the genomes (columns) hold the k-mers (rows):
|
||||
// c0 = {r0, r2}, c1 = {r1}, c2 = {r1}.
|
||||
let (_d, m) = make_matrix(&[&[3, 0, 2], &[1, 2, 0], &[0, 3, 1]]);
|
||||
let h = m.threshold_hamming_dist_matrix(2);
|
||||
let expected = [[0, 3, 3], [3, 0, 0], [3, 0, 0]];
|
||||
for i in 0..3 {
|
||||
for j in 0..3 {
|
||||
assert_eq!(h[[i, j]], expected[i][j], "[{i},{j}]");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn single_col_distance_matrix_is_zero() {
|
||||
let (_d, m) = make_matrix(&[&[1, 2, 3]]);
|
||||
|
||||
@@ -10,6 +10,8 @@ mod mem_builder;
|
||||
mod rankselect;
|
||||
mod sparse;
|
||||
mod sparse_intmatrix;
|
||||
mod transpose_columnar;
|
||||
mod views_scan;
|
||||
|
||||
use tempfile::tempdir;
|
||||
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
//! The windowed columnar → sparse transposition must produce the very bytes
|
||||
//! the dense route (`pack` to `matrix.pbmx`/`.pcmx`, then `build_from_dense`)
|
||||
//! produces, whatever the window size and non-zero-cell cap — including the
|
||||
//! retry-with-a-smaller-window path.
|
||||
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
use tempfile::tempdir;
|
||||
|
||||
use crate::meta::MatrixMeta;
|
||||
use crate::{
|
||||
PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentCompactIntMatrixBuilder,
|
||||
PersistentIntMatrix, PersistentSparseBitMatrixBuilder, PersistentSparseCompactIntMatrixBuilder,
|
||||
pack_bit_matrix, pack_compact_int_matrix,
|
||||
};
|
||||
|
||||
const N: usize = 5_000 + 37; // not a multiple of 64
|
||||
const N_COLS: usize = 7;
|
||||
|
||||
fn xorshift(s: &mut u64) -> u64 {
|
||||
*s ^= *s << 13;
|
||||
*s ^= *s >> 7;
|
||||
*s ^= *s << 17;
|
||||
*s
|
||||
}
|
||||
|
||||
/// Column `c`: all zero, very sparse, sparse, dense, ... so that rows have
|
||||
/// from 0 to `N_COLS` cells and whole words are zero in some columns.
|
||||
fn cell(c: usize, s: &mut u64) -> Option<u32> {
|
||||
let r = xorshift(s);
|
||||
let keep = match c % 4 {
|
||||
0 => false,
|
||||
1 => r % 97 == 0,
|
||||
2 => r % 7 == 0,
|
||||
_ => r % 2 == 0,
|
||||
};
|
||||
keep.then(|| match (r >> 20) % 11 {
|
||||
0 => 255,
|
||||
1 => 256,
|
||||
2 => 70_000,
|
||||
v => (v as u32) + 1,
|
||||
})
|
||||
}
|
||||
|
||||
fn build_bits(root: &Path) {
|
||||
let mut b = PersistentBitMatrixBuilder::new(N, &root.join("presence")).unwrap();
|
||||
let mut s = 0x9E37_79B9_7F4A_7C15u64;
|
||||
for c in 0..N_COLS {
|
||||
let mut cb = b.add_col().unwrap();
|
||||
for slot in 0..N {
|
||||
cb.set(slot, cell(c, &mut s).is_some());
|
||||
}
|
||||
cb.close().unwrap();
|
||||
}
|
||||
b.close().unwrap();
|
||||
}
|
||||
|
||||
fn build_ints(root: &Path) {
|
||||
let mut b = PersistentCompactIntMatrixBuilder::new(N, &root.join("counts")).unwrap();
|
||||
let mut s = 0x9E37_79B9_7F4A_7C15u64;
|
||||
for c in 0..N_COLS {
|
||||
let mut cb = b.add_col().unwrap();
|
||||
for slot in 0..N {
|
||||
if let Some(v) = cell(c, &mut s) {
|
||||
cb.set(slot, v);
|
||||
}
|
||||
}
|
||||
cb.close().unwrap();
|
||||
}
|
||||
b.close().unwrap();
|
||||
}
|
||||
|
||||
/// Sparse-format files of `dir` (columnar leftovers and dense files excluded),
|
||||
/// as `(name, bytes)` sorted by name.
|
||||
fn sparse_files(dir: &Path) -> Vec<(String, Vec<u8>)> {
|
||||
let mut v: Vec<(String, Vec<u8>)> = fs::read_dir(dir)
|
||||
.unwrap()
|
||||
.map(|e| e.unwrap().path())
|
||||
.filter(|p| {
|
||||
let n = p.file_name().unwrap().to_string_lossy().into_owned();
|
||||
!(n.starts_with("col_") || n == "meta.json" || n.starts_with("matrix."))
|
||||
})
|
||||
.map(|p| (p.file_name().unwrap().to_string_lossy().into_owned(), fs::read(&p).unwrap()))
|
||||
.collect();
|
||||
v.sort();
|
||||
v
|
||||
}
|
||||
|
||||
const WINDOWS: &[(usize, usize)] = &[
|
||||
(64, usize::MAX / 4), // smallest window, never too dense
|
||||
(192, 50), // forces the retry-with-half-window path
|
||||
(4096, 10_000), // a few windows
|
||||
(crate::transpose::DEFAULT_WINDOW_ROWS, crate::transpose::DEFAULT_MAX_HITS),
|
||||
];
|
||||
|
||||
#[test]
|
||||
fn bit_columnar_transpose_is_byte_identical_to_the_dense_route() {
|
||||
let ref_root = tempdir().unwrap();
|
||||
build_bits(ref_root.path());
|
||||
let ref_dir = ref_root.path().join("presence");
|
||||
pack_bit_matrix(&ref_dir).unwrap();
|
||||
let dense = PersistentBitMatrix::open(ref_root.path()).unwrap();
|
||||
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, &ref_dir).unwrap().close().unwrap();
|
||||
drop(dense);
|
||||
let expected = sparse_files(&ref_dir);
|
||||
assert!(!expected.is_empty());
|
||||
|
||||
for &(window, max_hits) in WINDOWS {
|
||||
let root = tempdir().unwrap();
|
||||
build_bits(root.path());
|
||||
let dir = root.path().join("presence");
|
||||
let meta = MatrixMeta::load(&dir).unwrap();
|
||||
PersistentSparseBitMatrixBuilder::build_from_columnar_with(&dir, &meta, window, max_hits)
|
||||
.unwrap()
|
||||
.close()
|
||||
.unwrap();
|
||||
assert_eq!(sparse_files(&dir), expected, "window={window} max_hits={max_hits}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn int_columnar_transpose_is_byte_identical_to_the_dense_route() {
|
||||
let ref_root = tempdir().unwrap();
|
||||
build_ints(ref_root.path());
|
||||
let ref_dir = ref_root.path().join("counts");
|
||||
pack_compact_int_matrix(&ref_dir).unwrap();
|
||||
let dense = PersistentIntMatrix::open(ref_root.path()).unwrap();
|
||||
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, &ref_dir).unwrap().close().unwrap();
|
||||
drop(dense);
|
||||
let expected = sparse_files(&ref_dir);
|
||||
assert!(!expected.is_empty());
|
||||
|
||||
for &(window, max_hits) in WINDOWS {
|
||||
let root = tempdir().unwrap();
|
||||
build_ints(root.path());
|
||||
let dir = root.path().join("counts");
|
||||
let meta = MatrixMeta::load(&dir).unwrap();
|
||||
PersistentSparseCompactIntMatrixBuilder::build_from_columnar_with(&dir, &meta, window, max_hits)
|
||||
.unwrap()
|
||||
.close()
|
||||
.unwrap();
|
||||
assert_eq!(sparse_files(&dir), expected, "window={window} max_hits={max_hits}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn columnar_transpose_rejects_a_column_of_another_length() {
|
||||
let root = tempdir().unwrap();
|
||||
build_bits(root.path());
|
||||
let dir = root.path().join("presence");
|
||||
let mut meta = MatrixMeta::load(&dir).unwrap();
|
||||
meta.n += 64; // metadata claims more rows than the column files hold
|
||||
let err = PersistentSparseBitMatrixBuilder::build_from_columnar(&dir, &meta).err().expect("must fail");
|
||||
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
use crate::views::BitSliceView;
|
||||
|
||||
/// Deterministic pseudo-random words (xorshift), thinned so that many words
|
||||
/// are zero: the case the word-wise scan has to skip correctly.
|
||||
fn words(n_words: usize, seed: u64, thin: u32) -> Vec<u64> {
|
||||
let mut s = seed | 1;
|
||||
(0..n_words)
|
||||
.map(|_| {
|
||||
let mut w = u64::MAX;
|
||||
for _ in 0..thin {
|
||||
s ^= s << 13;
|
||||
s ^= s >> 7;
|
||||
s ^= s << 17;
|
||||
w &= s;
|
||||
}
|
||||
w
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn naive(v: &BitSliceView<'_>, r: std::ops::Range<usize>) -> Vec<usize> {
|
||||
r.filter(|&s| v.get(s)).collect()
|
||||
}
|
||||
|
||||
fn scanned(v: &BitSliceView<'_>, r: std::ops::Range<usize>) -> Vec<usize> {
|
||||
let mut out = Vec::new();
|
||||
v.for_each_set_in(r, |s| out.push(s));
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn for_each_set_in_matches_get_loop_on_all_boundaries() {
|
||||
// Lengths around word boundaries; padding bits of the last word are zero.
|
||||
for n in [0usize, 1, 63, 64, 65, 127, 128, 129, 200, 4096, 4097] {
|
||||
for (seed, thin) in [(1u64, 0u32), (7, 1), (11, 3), (13, 6)] {
|
||||
let mut w = words(n.div_ceil(64), seed, thin);
|
||||
if n % 64 != 0 {
|
||||
if let Some(last) = w.last_mut() {
|
||||
*last &= (1u64 << (n % 64)) - 1;
|
||||
}
|
||||
}
|
||||
let v = BitSliceView::new(&w, n);
|
||||
let cuts = [0, 1, 63, 64, 65, 100, 127, 128, 129, n / 2, n.saturating_sub(1), n];
|
||||
for &a in &cuts {
|
||||
for &b in &cuts {
|
||||
if a <= n && b <= n {
|
||||
assert_eq!(scanned(&v, a..b), naive(&v, a..b), "n={n} seed={seed} thin={thin} range={a}..{b}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn for_each_set_in_empty_and_reversed_ranges_do_nothing() {
|
||||
let w = [u64::MAX; 2];
|
||||
let v = BitSliceView::new(&w, 128);
|
||||
assert!(scanned(&v, 5..5).is_empty());
|
||||
#[allow(clippy::reversed_empty_ranges)]
|
||||
let rev = 9..3;
|
||||
assert!(scanned(&v, rev).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[should_panic]
|
||||
fn for_each_set_in_rejects_range_past_the_view() {
|
||||
let w = [u64::MAX; 2];
|
||||
let v = BitSliceView::new(&w, 100);
|
||||
scanned(&v, 0..101);
|
||||
}
|
||||
@@ -129,6 +129,16 @@ pub trait CountPartials: ColumnWeights {
|
||||
jaccard_to_mash(&self.threshold_jaccard_dist_matrix(threshold), k)
|
||||
}
|
||||
|
||||
/// Hamming distance on presence/absence: the number of k-mers present
|
||||
/// (count ≥ `threshold`) in exactly one of the two genomes, i.e. the union
|
||||
/// minus the intersection that [`partial_threshold_jaccard`] already gives.
|
||||
///
|
||||
/// [`partial_threshold_jaccard`]: Self::partial_threshold_jaccard
|
||||
fn threshold_hamming_dist_matrix(&self, threshold: u32) -> Array2<u64> {
|
||||
let (inter, union) = self.partial_threshold_jaccard(threshold);
|
||||
union - inter
|
||||
}
|
||||
|
||||
fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
|
||||
let global = self.col_weights();
|
||||
let mut m = self.partial_relfreq_bray(&global).mapv(|v| 1.0 - v);
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
//! Windowed column → row transposition, shared by the presence and count
|
||||
//! packers.
|
||||
//!
|
||||
//! A columnar matrix stores one file per genome; the sparse formats are
|
||||
//! built row by row, each row needing the cells of *every* column. Holding
|
||||
//! every column mapped (or open) at once costs one mapping (or descriptor)
|
||||
//! per column and per worker. This module never holds more than one column:
|
||||
//! it cuts the rows into windows and, for each window, visits the columns
|
||||
//! one after the other through a caller-supplied reader, collecting the
|
||||
//! non-zero cells; the cells are then regrouped by row and handed to the
|
||||
//! caller. No intermediate dense matrix is written.
|
||||
//!
|
||||
//! Memory is bounded by `max_hits`, not by the partition size: when a window
|
||||
//! turns out denser than expected it is abandoned and retried with half as
|
||||
//! many rows.
|
||||
|
||||
use std::io;
|
||||
|
||||
/// Default rows per window. A multiple of 64, so that a bit column's window
|
||||
/// starts on a word boundary. 2 M rows = 256 KiB per bit column and read.
|
||||
pub const DEFAULT_WINDOW_ROWS: usize = 1 << 21;
|
||||
|
||||
/// Default cap on the non-zero cells held for one window.
|
||||
pub const DEFAULT_MAX_HITS: usize = 4 << 20;
|
||||
|
||||
/// Smallest window the retry logic shrinks to.
|
||||
const MIN_WINDOW_ROWS: usize = 64;
|
||||
|
||||
/// One non-zero cell of a window: row relative to the window start, column,
|
||||
/// value. `V = ()` for presence matrices.
|
||||
pub(crate) type Hit<V> = (u32, u32, V);
|
||||
|
||||
/// The rows of one window, regrouped by row, columns ascending within a row.
|
||||
pub(crate) struct RowBatch<V> {
|
||||
offsets: Vec<u32>,
|
||||
cols: Vec<u32>,
|
||||
vals: Vec<V>,
|
||||
}
|
||||
|
||||
impl<V: Copy> RowBatch<V> {
|
||||
fn new() -> Self {
|
||||
Self { offsets: Vec::new(), cols: Vec::new(), vals: Vec::new() }
|
||||
}
|
||||
|
||||
/// Number of rows in the window.
|
||||
pub(crate) fn len(&self) -> usize {
|
||||
self.offsets.len().saturating_sub(1)
|
||||
}
|
||||
|
||||
/// Columns of row `i`, ascending.
|
||||
pub(crate) fn cols(&self, i: usize) -> &[u32] {
|
||||
&self.cols[self.offsets[i] as usize..self.offsets[i + 1] as usize]
|
||||
}
|
||||
|
||||
/// Values of row `i`, in the order of [`cols`](Self::cols).
|
||||
pub(crate) fn vals(&self, i: usize) -> &[V] {
|
||||
&self.vals[self.offsets[i] as usize..self.offsets[i + 1] as usize]
|
||||
}
|
||||
|
||||
/// Stable counting sort of `hits` by row. `hits` was produced column by
|
||||
/// column in ascending column order, so stability leaves each row's
|
||||
/// columns ascending, as `push_row` requires, without a per-row sort.
|
||||
fn fill(&mut self, rows: usize, hits: &[Hit<V>], zero: V) {
|
||||
self.offsets.clear();
|
||||
self.offsets.resize(rows + 1, 0);
|
||||
for &(r, _, _) in hits {
|
||||
self.offsets[r as usize + 1] += 1;
|
||||
}
|
||||
for i in 0..rows {
|
||||
self.offsets[i + 1] += self.offsets[i];
|
||||
}
|
||||
self.cols.clear();
|
||||
self.cols.resize(hits.len(), 0);
|
||||
self.vals.clear();
|
||||
self.vals.resize(hits.len(), zero);
|
||||
// `next[r]` = write position of the next cell of row `r`; reuse the
|
||||
// tail of `offsets` would alias, so keep a small separate cursor.
|
||||
let mut next: Vec<u32> = self.offsets[..rows].to_vec();
|
||||
for &(r, c, v) in hits {
|
||||
let p = next[r as usize] as usize;
|
||||
self.cols[p] = c;
|
||||
self.vals[p] = v;
|
||||
next[r as usize] += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Transposes an `n`-row, `n_cols`-column matrix window by window.
|
||||
///
|
||||
/// `read_col(c, start, end, hits)` must append to `hits` every non-zero cell
|
||||
/// of column `c` in rows `start..end`, as `(row - start, c, value)`.
|
||||
/// `on_batch(start, batch)` receives each completed window, in ascending row
|
||||
/// order; `batch` row `i` is matrix row `start + i`.
|
||||
///
|
||||
/// `window_rows` is rounded down to a multiple of 64 (minimum 64), and `start`
|
||||
/// is always a multiple of 64.
|
||||
pub(crate) fn transpose_windows<V: Copy>(
|
||||
n: usize,
|
||||
n_cols: usize,
|
||||
window_rows: usize,
|
||||
max_hits: usize,
|
||||
zero: V,
|
||||
mut read_col: impl FnMut(usize, usize, usize, &mut Vec<Hit<V>>) -> io::Result<()>,
|
||||
mut on_batch: impl FnMut(usize, &RowBatch<V>) -> io::Result<()>,
|
||||
) -> io::Result<()> {
|
||||
let window_max = (window_rows / 64 * 64).clamp(MIN_WINDOW_ROWS, u32::MAX as usize / 2);
|
||||
// `Hit` rows are u32 and offsets count hits in u32.
|
||||
let max_hits = max_hits.min(u32::MAX as usize / 2);
|
||||
let mut w = window_max;
|
||||
let mut hits: Vec<Hit<V>> = Vec::new();
|
||||
let mut batch = RowBatch::new();
|
||||
let mut start = 0usize;
|
||||
while start < n {
|
||||
let end = (start + w).min(n);
|
||||
hits.clear();
|
||||
let mut too_dense = false;
|
||||
for c in 0..n_cols {
|
||||
read_col(c, start, end, &mut hits)?;
|
||||
if hits.len() > max_hits && w > MIN_WINDOW_ROWS {
|
||||
too_dense = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if too_dense {
|
||||
w = (w / 2 / 64 * 64).max(MIN_WINDOW_ROWS);
|
||||
continue;
|
||||
}
|
||||
batch.fill(end - start, &hits, zero);
|
||||
on_batch(start, &batch)?;
|
||||
start = end;
|
||||
if hits.len() < max_hits / 4 && w < window_max {
|
||||
w = (w * 2).min(window_max);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
//! Word-wise scans over a [`BitSliceView`]. Kept apart from `views.rs`, which
|
||||
//! is already past the size budget; these are inherent methods of the same type.
|
||||
|
||||
use std::ops::Range;
|
||||
|
||||
use crate::views::BitSliceView;
|
||||
|
||||
impl BitSliceView<'_> {
|
||||
/// Calls `f(slot)` for every set bit with `slot` in `range`, in ascending
|
||||
/// order.
|
||||
///
|
||||
/// Walks the underlying `u64` words and jumps from set bit to set bit
|
||||
/// with `trailing_zeros`, so the cost is proportional to the number of
|
||||
/// words plus the number of set bits — not to the number of slots, as a
|
||||
/// `get(slot)` loop would be. Words that are entirely zero cost one load
|
||||
/// and one test, which is what makes this the right traversal for the
|
||||
/// sparse presence columns `build_from_dense` reads.
|
||||
///
|
||||
/// Panics if `range.end > self.len()`.
|
||||
#[inline]
|
||||
pub fn for_each_set_in(&self, range: Range<usize>, mut f: impl FnMut(usize)) {
|
||||
assert!(range.end <= self.n, "range end {} past view length {}", range.end, self.n);
|
||||
if range.start >= range.end {
|
||||
return;
|
||||
}
|
||||
let first = range.start >> 6;
|
||||
let last = (range.end - 1) >> 6;
|
||||
for w in first..=last {
|
||||
let mut word = self.words[w];
|
||||
if word == 0 {
|
||||
continue;
|
||||
}
|
||||
if w == first {
|
||||
word &= !0u64 << (range.start & 63);
|
||||
}
|
||||
if w == last {
|
||||
let tail = range.end & 63;
|
||||
if tail != 0 {
|
||||
word &= (1u64 << tail) - 1;
|
||||
}
|
||||
}
|
||||
while word != 0 {
|
||||
f((w << 6) | word.trailing_zeros() as usize);
|
||||
word &= word - 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,11 +5,9 @@ edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
obikseq = { path = "../obikseq" }
|
||||
obifastwrite = { path = "../obifastwrite" }
|
||||
hashbrown = { version = "0.17", features = ["rayon"] }
|
||||
rayon = "1"
|
||||
crossbeam-channel = "0.5"
|
||||
tracing = "0.1"
|
||||
|
||||
[features]
|
||||
test-utils = []
|
||||
|
||||
@@ -81,7 +81,7 @@ impl IndexDump for Arc<KmerIndex> {
|
||||
if remaining.load(Ordering::Relaxed) == 0 { return Ok(vec![]); }
|
||||
let mut buf = Vec::<u8>::new();
|
||||
let try_write = |buf: &mut Vec<u8>, row: &[u32], prefix: &str| -> bool {
|
||||
match remaining.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |cur| {
|
||||
match remaining.try_update(Ordering::SeqCst, Ordering::SeqCst, |cur| {
|
||||
if cur > 0 { Some(cur - 1) } else { None }
|
||||
}) {
|
||||
Err(_) => false,
|
||||
|
||||
@@ -5,13 +5,9 @@ edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
obikseq = { path = "../obikseq" }
|
||||
obitaxonomy = { path = "../obitaxonomy" }
|
||||
obiskio = { path = "../obiskio" }
|
||||
obisys = { path = "../obisys", default-features = false }
|
||||
obicompactvec = { path = "../obicompactvec" }
|
||||
obidebruinj = { path = "../obidebruinj" }
|
||||
obipipeline = { path = "../obipipeline" }
|
||||
obikentropy = { path = "../obikentropy" }
|
||||
cacheline-ef = "2.0"
|
||||
epserde = "0.13"
|
||||
ptr_hash = { version = "2.1", features = ["epserde"] }
|
||||
|
||||
@@ -280,7 +280,7 @@ impl KmerIndex {
|
||||
/// on-disk formats (`PersistentSparseBitMatrix` for presence,
|
||||
/// `PersistentSparseCompactIntMatrix` for counts) — smaller and faster
|
||||
/// for single-row access on real, sparse data; column-oriented access
|
||||
/// (`--metric` distance matrices) is much slower on the sparse format.
|
||||
/// (`phylo --distance` matrices) is much slower on the sparse format.
|
||||
///
|
||||
/// Public — and not moved out of this crate, despite covering the same
|
||||
/// "maintenance operation on an already-built index" territory as
|
||||
@@ -296,7 +296,14 @@ impl KmerIndex {
|
||||
};
|
||||
|
||||
let n = self.n_partitions();
|
||||
let order: Vec<usize> = (0..n).collect();
|
||||
// Largest partitions first (LFD), so the long jobs do not end up as
|
||||
// stragglers at the tail. Sizes are `stat()` only; the sort is stable,
|
||||
// ties keep partition-number order. Partitions are independent, so
|
||||
// the order has no effect on what is written.
|
||||
let sizes: Vec<u64> = (0..n).map(|i| self.pending_matrix_bytes(i)).collect();
|
||||
let mut order: Vec<usize> = (0..n).collect();
|
||||
order.sort_by_key(|&i| std::cmp::Reverse(sizes[i]));
|
||||
let mut timings: Vec<(usize, u64, std::time::Duration)> = Vec::with_capacity(n);
|
||||
let pb = progress_bar("pack", n as u64, "partitions");
|
||||
obisys::PartitionRunner::new().run(
|
||||
&order,
|
||||
@@ -327,14 +334,50 @@ impl KmerIndex {
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
|_, _, _| {
|
||||
|i, _, elapsed| {
|
||||
tracing::debug!(
|
||||
partition = i,
|
||||
matrix_bytes = sizes[i],
|
||||
secs = elapsed.as_secs_f64(),
|
||||
"pack: partition done"
|
||||
);
|
||||
timings.push((i, sizes[i], elapsed));
|
||||
pb.inc(1);
|
||||
},
|
||||
)?;
|
||||
pb.finish_and_clear();
|
||||
report_pack_timings(&timings);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Bytes of matrix data (`presence/` and `counts/` of every layer) still
|
||||
/// present in partition `i`, from `stat()` only. Before packing this is
|
||||
/// the columnar data, i.e. a proxy for the work `pack_matrices` has to do
|
||||
/// on that partition. Missing partition or directories count as 0.
|
||||
pub fn pending_matrix_bytes(&self, i: usize) -> u64 {
|
||||
let Ok(n_layers) = self.n_layers(i) else {
|
||||
return 0;
|
||||
};
|
||||
let mut total = 0u64;
|
||||
for l in 0..n_layers {
|
||||
let Ok(layer_dir) = self.layer_dir(i, l) else {
|
||||
continue;
|
||||
};
|
||||
for sub in ["presence", "counts"] {
|
||||
let Ok(entries) = fs::read_dir(layer_dir.join(sub)) else {
|
||||
continue;
|
||||
};
|
||||
total += entries
|
||||
.flatten()
|
||||
.filter_map(|e| e.metadata().ok())
|
||||
.filter(|m| m.is_file())
|
||||
.map(|m| m.len())
|
||||
.sum::<u64>();
|
||||
}
|
||||
}
|
||||
total
|
||||
}
|
||||
|
||||
// /// Write a `layer_meta.json` in any layer directory that is missing one.
|
||||
// ///
|
||||
// /// Old indexes were built before this file was required. The number of
|
||||
@@ -394,3 +437,34 @@ impl KmerIndex {
|
||||
// Ok(())
|
||||
// }
|
||||
}
|
||||
|
||||
/// One `info!` summary of `pack_matrices` per-partition timings: the sum of
|
||||
/// partition times (divide by the stage wall time to get the mean number of
|
||||
/// concurrently busy workers), the median and maximum, and the slowest
|
||||
/// partitions with their pre-pack matrix size. `timings` holds
|
||||
/// `(partition, matrix_bytes, elapsed)`.
|
||||
fn report_pack_timings(timings: &[(usize, u64, std::time::Duration)]) {
|
||||
if timings.is_empty() {
|
||||
return;
|
||||
}
|
||||
let mut by_time: Vec<&(usize, u64, std::time::Duration)> = timings.iter().collect();
|
||||
by_time.sort_by_key(|t| std::cmp::Reverse(t.2));
|
||||
let sum: f64 = timings.iter().map(|t| t.2.as_secs_f64()).sum();
|
||||
let bytes: u64 = timings.iter().map(|t| t.1).sum();
|
||||
let median = by_time[by_time.len() / 2].2.as_secs_f64();
|
||||
tracing::info!(
|
||||
"pack: {} partition(s), sum of partition times {:.0}s, median {:.1}s, max {:.1}s, {:.1} GB of matrix data",
|
||||
timings.len(),
|
||||
sum,
|
||||
median,
|
||||
by_time[0].2.as_secs_f64(),
|
||||
bytes as f64 / 1e9,
|
||||
);
|
||||
for (i, b, d) in by_time.iter().take(5).map(|t| (t.0, t.1, t.2)) {
|
||||
tracing::info!(
|
||||
"pack: slowest partition {i}: {:.1}s, {:.2} GB of matrix data",
|
||||
d.as_secs_f64(),
|
||||
b as f64 / 1e9,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -135,6 +135,12 @@ impl PrivateBuilder for KmerIndex {
|
||||
mode: &IndexMode,
|
||||
block_bits: u8,
|
||||
) -> Result<usize, SKError> {
|
||||
// A partition that received no k-mer has no layer 0 on disk at all
|
||||
// (small inputs against many partitions): nothing to build, not an
|
||||
// error.
|
||||
if self.partition(i).map_err(|e| io::Error::other(e.to_string()))?.n_layers() == 0 {
|
||||
return Ok(0);
|
||||
}
|
||||
let layer0 = self
|
||||
.layer0(i)
|
||||
.map_err(|e| io::Error::other(e.to_string()))?;
|
||||
@@ -208,6 +214,10 @@ impl PrivateBuilder for KmerIndex {
|
||||
}
|
||||
|
||||
fn remove_build_artifacts(&self, i: usize) {
|
||||
// No layer 0 = empty partition: no build artifacts to remove.
|
||||
if self.partition(i).map_or(false, |p| p.n_layers() == 0) {
|
||||
return;
|
||||
}
|
||||
let layer0 = match self.layer0(i) {
|
||||
Ok(layer) => layer,
|
||||
Err(e) => {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "obikmer"
|
||||
version = "1.4.4"
|
||||
version = "1.4.5"
|
||||
edition = "2024"
|
||||
|
||||
[[bin]]
|
||||
@@ -24,7 +24,6 @@ obikstats = { path = "../obikstats" }
|
||||
obikquery = { path = "../obikquery" }
|
||||
obikidxcache = { path = "../obikidxcache" }
|
||||
obikphylo = { path = "../obikphylo" }
|
||||
obiksession = { path = "../obiksession" }
|
||||
obikrope = { path = "../obikrope" }
|
||||
obifastwrite = { path = "../obifastwrite" }
|
||||
obiskbuilder = { path = "../obiskbuilder" }
|
||||
@@ -37,6 +36,13 @@ serde_yaml = "0.9"
|
||||
tracing = "0.1.44"
|
||||
tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
|
||||
|
||||
# musl's built-in allocator fragments badly under the merge's multi-threaded
|
||||
# allocation pattern (map count and RSS drift, see DevDocMD/implementation/
|
||||
# merge_mmap_audit.md). Static musl builds use jemalloc instead; glibc and
|
||||
# macOS builds keep the system allocator.
|
||||
[target.'cfg(target_env = "musl")'.dependencies]
|
||||
tikv-jemallocator = "0.5.4"
|
||||
|
||||
[features]
|
||||
default = ["numa"]
|
||||
numa = ["obisys/numa"]
|
||||
@@ -19,7 +19,7 @@ pub struct PackArgs {
|
||||
|
||||
/// Pack presence and count matrices into the dense on-disk format instead
|
||||
/// of the default sparse, deduplicated one. Dense is faster for
|
||||
/// column-oriented access (`--metric` distance matrices); sparse is
|
||||
/// column-oriented access (`phylo --distance` matrices); sparse is
|
||||
/// smaller and faster for single-row access on real, sparse data.
|
||||
#[arg(long, default_value_t = false)]
|
||||
pub dense: bool,
|
||||
|
||||
@@ -29,7 +29,7 @@ pub(super) fn write_alignment_outputs(
|
||||
let n_sites = alignment.sequences.first().map_or(0, Vec::len);
|
||||
let n_genomes = alignment.genome_indices.len();
|
||||
|
||||
if args.fasta || !args.vcf {
|
||||
if args.align.fasta || !args.align.vcf {
|
||||
let path = path_for("fasta");
|
||||
let mut f = create(&path);
|
||||
for (&g, seq) in alignment.genome_indices.iter().zip(alignment.sequences.iter()) {
|
||||
@@ -38,10 +38,10 @@ pub(super) fn write_alignment_outputs(
|
||||
info!("pseudo-alignment ({n_sites} site(s), {n_genomes} genome(s)) → {path}");
|
||||
}
|
||||
|
||||
if args.vcf {
|
||||
if args.align.vcf {
|
||||
let path = path_for("vcf");
|
||||
let mut f = create(&path);
|
||||
let written = alignment.write_vcf(labels, !args.no_ambiguity, &mut f).unwrap_or_else(|e| {
|
||||
let written = alignment.write_vcf(labels, !args.sample.no_ambiguity, &mut f).unwrap_or_else(|e| {
|
||||
eprintln!("error writing {path}: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
|
||||
@@ -1,558 +0,0 @@
|
||||
use std::path::PathBuf;
|
||||
|
||||
use clap::Args;
|
||||
use obikphylo::siblings::{AmbiguityPolicy, AnnexSelection};
|
||||
use obikphylo::{DistanceMetric, GammaShape, SnpDistanceKind};
|
||||
|
||||
/// `--gamma-shape` CLI value: a fixed `alpha`, or `auto`/`estimate` to fit
|
||||
/// it from the data (see [`GammaShape::Auto`]). Kept as its own tiny type
|
||||
/// (rather than parsing straight to `GammaShape`) so the CLI layer stays
|
||||
/// free of `GammaShape::Disabled`, which the `Option<GammaShapeArg>` around
|
||||
/// this already expresses on its own (`None` = flag not given at all).
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub enum GammaShapeArg {
|
||||
Fixed(f64),
|
||||
Auto,
|
||||
}
|
||||
|
||||
impl From<GammaShapeArg> for GammaShape {
|
||||
fn from(arg: GammaShapeArg) -> Self {
|
||||
match arg {
|
||||
GammaShapeArg::Fixed(a) => GammaShape::Fixed(a),
|
||||
GammaShapeArg::Auto => GammaShape::Auto,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_gamma_shape(s: &str) -> Result<GammaShapeArg, String> {
|
||||
if s.eq_ignore_ascii_case("auto") || s.eq_ignore_ascii_case("estimate") {
|
||||
return Ok(GammaShapeArg::Auto);
|
||||
}
|
||||
s.parse::<f64>()
|
||||
.map(GammaShapeArg::Fixed)
|
||||
.map_err(|_| format!("invalid --gamma-shape value {s:?}: expected a positive number or \"auto\"/\"estimate\""))
|
||||
}
|
||||
|
||||
/// `--n-sites` CLI value: a number of sites to draw, or `all` for every site
|
||||
/// — the draw then keeps every variable family that passes the other filters,
|
||||
/// with no sampling involved. Its own tiny type (like [`GammaShapeArg`]) so
|
||||
/// the two cases stay distinct up to the call that needs the target, instead
|
||||
/// of an in-band sentinel (a negative number, `0`, ...) that a typo could hit.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum NSitesArg {
|
||||
All,
|
||||
N(usize),
|
||||
}
|
||||
|
||||
impl NSitesArg {
|
||||
/// The target to hand to the sampler: `None` = exhaustive.
|
||||
pub fn target(self) -> Option<usize> {
|
||||
match self {
|
||||
NSitesArg::All => None,
|
||||
NSitesArg::N(n) => Some(n),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_n_sites(s: &str) -> Result<NSitesArg, String> {
|
||||
if s.eq_ignore_ascii_case("all") {
|
||||
return Ok(NSitesArg::All);
|
||||
}
|
||||
s.parse::<usize>()
|
||||
.map(NSitesArg::N)
|
||||
.map_err(|_| format!("invalid --n-sites value {s:?}: expected a non-negative integer or \"all\""))
|
||||
}
|
||||
|
||||
/// `--distance` value — either one of `obikphylo::DistanceMetric`'s
|
||||
/// whole-index metrics (routed to `IndexCache::distance`) or one of
|
||||
/// `obikphylo::SnpDistanceKind`'s `snp-*` corrections (routed to
|
||||
/// `obikphylo::siblings::distance_matrix`, over a tally built by
|
||||
/// `sampled_scan`/`PairwiseReducer`, the sibling-annex pipeline) — two
|
||||
/// genuinely different code paths behind one CLI vocabulary, see
|
||||
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance` unification".
|
||||
#[derive(clap::ValueEnum, Clone, Copy, Debug)]
|
||||
pub enum DistanceArg {
|
||||
Jaccard,
|
||||
Mash,
|
||||
Hamming,
|
||||
BrayCurtis,
|
||||
#[value(name = "relfreq-bray-curtis")]
|
||||
RelfreqBrayCurtis,
|
||||
Euclidean,
|
||||
#[value(name = "relfreq-euclidean")]
|
||||
RelfreqEuclidean,
|
||||
Hellinger,
|
||||
#[value(name = "hellinger-euclidean")]
|
||||
HellingerEuclidean,
|
||||
#[value(name = "snp-raw")]
|
||||
SnpRaw,
|
||||
#[value(name = "snp-jc")]
|
||||
SnpJc,
|
||||
#[value(name = "snp-k2p")]
|
||||
SnpK2p,
|
||||
#[value(name = "snp-k81")]
|
||||
SnpK81,
|
||||
#[value(name = "snp-f81")]
|
||||
SnpF81,
|
||||
#[value(name = "snp-t92")]
|
||||
SnpT92,
|
||||
#[value(name = "snp-tn93")]
|
||||
SnpTn93,
|
||||
#[value(name = "snp-tv")]
|
||||
SnpTv,
|
||||
}
|
||||
|
||||
/// The distance matrix this invocation must compute, if any: the one asked for
|
||||
/// with `--distance`, else `jaccard` when `--nj`, `--upgma` or
|
||||
/// `--shared-kmers` needs a matrix to work from, else none — a matrix is never
|
||||
/// computed (nor written) just because nothing said otherwise.
|
||||
pub fn distance_to_compute(
|
||||
distance: Option<DistanceArg>,
|
||||
nj: bool,
|
||||
upgma: bool,
|
||||
shared_kmers: bool,
|
||||
) -> Option<DistanceArg> {
|
||||
distance.or_else(|| (nj || upgma || shared_kmers).then_some(DistanceArg::Jaccard))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod distance_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn nothing_asked_nothing_computed() {
|
||||
assert!(distance_to_compute(None, false, false, false).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tree_and_shared_kmer_outputs_imply_jaccard() {
|
||||
for (nj, upgma, shared) in [(true, false, false), (false, true, false), (false, false, true)] {
|
||||
assert!(matches!(
|
||||
distance_to_compute(None, nj, upgma, shared),
|
||||
Some(DistanceArg::Jaccard)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_explicit_distance_wins() {
|
||||
assert!(matches!(
|
||||
distance_to_compute(Some(DistanceArg::Mash), true, true, true),
|
||||
Some(DistanceArg::Mash)
|
||||
));
|
||||
assert!(matches!(
|
||||
distance_to_compute(Some(DistanceArg::SnpJc), false, false, false),
|
||||
Some(DistanceArg::SnpJc)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
impl DistanceArg {
|
||||
/// `Some` for the whole-index metrics, `None` for `snp-*` values.
|
||||
pub fn as_classic(self) -> Option<DistanceMetric> {
|
||||
Some(match self {
|
||||
DistanceArg::Jaccard => DistanceMetric::Jaccard,
|
||||
DistanceArg::Mash => DistanceMetric::Mash,
|
||||
DistanceArg::Hamming => DistanceMetric::Hamming,
|
||||
DistanceArg::BrayCurtis => DistanceMetric::BrayCurtis,
|
||||
DistanceArg::RelfreqBrayCurtis => DistanceMetric::RelfreqBrayCurtis,
|
||||
DistanceArg::Euclidean => DistanceMetric::Euclidean,
|
||||
DistanceArg::RelfreqEuclidean => DistanceMetric::RelfreqEuclidean,
|
||||
DistanceArg::Hellinger => DistanceMetric::Hellinger,
|
||||
DistanceArg::HellingerEuclidean => DistanceMetric::HellingerEuclidean,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
/// `Some` for the `snp-*` values, `None` for the whole-index metrics.
|
||||
pub fn as_snp(self) -> Option<SnpDistanceKind> {
|
||||
Some(match self {
|
||||
DistanceArg::SnpRaw => SnpDistanceKind::Raw,
|
||||
DistanceArg::SnpJc => SnpDistanceKind::Jc,
|
||||
DistanceArg::SnpK2p => SnpDistanceKind::K2p,
|
||||
DistanceArg::SnpK81 => SnpDistanceKind::K81,
|
||||
DistanceArg::SnpF81 => SnpDistanceKind::F81,
|
||||
DistanceArg::SnpT92 => SnpDistanceKind::T92,
|
||||
DistanceArg::SnpTn93 => SnpDistanceKind::Tn93,
|
||||
DistanceArg::SnpTv => SnpDistanceKind::Tv,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Genome-vs-genome distance computation: the whole-index `--distance` path
|
||||
/// (classic metrics + `snp-*` corrections/NJ/UPGMA), annex construction
|
||||
/// (`--sibling-annex`), annex diagnostics (`--sibling-stats`,
|
||||
/// `--sibling-hist`), entropy reporting (`--shannon`), SNP pseudo-alignment
|
||||
/// sampling (`--pseudo-alignment`, `--n-sites`, `--free-loss`,
|
||||
/// `--no-ambiguity`, `--entropy`/`--entropy-sd`), Sankoff cost-matrix
|
||||
/// calibration (`--sankoff`, `--sankoff-ratio-ceiling`) and its TNT/PhyG/
|
||||
/// IQ-TREE exports (`--tnt`, `--phyg`, `--iqtree`/`--iqtree-min-freq`,
|
||||
/// `--sankoff-cost-scale`), and Family Overlap (`--family-overlap`,
|
||||
/// `--min-shared-family`).
|
||||
#[derive(Args)]
|
||||
pub struct PhyloArgs {
|
||||
/// Index directory
|
||||
pub index: PathBuf,
|
||||
|
||||
/// Exclude a genome (by its exact label) — from `--pseudo-alignment`'s
|
||||
/// sampling (a family whose only polymorphism lived in an excluded
|
||||
/// genome is discarded during sampling, not filtered afterward — see
|
||||
/// `obikphylo::siblings::sampled_scan`'s own docs) and from the
|
||||
/// distance matrix / shared-kmer matrix CSV output (row and column
|
||||
/// both dropped; the underlying computation itself is unaffected).
|
||||
/// Repeatable.
|
||||
#[arg(long = "exclude-genome", value_name = "LABEL")]
|
||||
pub exclude_genome: Vec<String>,
|
||||
|
||||
/// Auto-exclude any genome whose mean shared-variable-family count
|
||||
/// against every other genome (`FamilyOverlap::mean_row` — the same
|
||||
/// per-row statistic `--family-overlap`'s own matrix shows) falls below
|
||||
/// this threshold — same exclusion machinery as `--exclude-genome`,
|
||||
/// applied on top of it rather than instead of it. Applies to the
|
||||
/// `snp-*` `--distance`/`--pseudo-alignment`/`--sankoff` computations
|
||||
/// below (all sibling-annex-based); does *not* affect the whole-index
|
||||
/// `--distance` metrics (jaccard, hamming, bray-curtis, ...) or their
|
||||
/// matrix/NJ/UPGMA output — a genome with too little SNP-family
|
||||
/// coverage to trust is a different concern from one whose plain k-mer
|
||||
/// profile is simply divergent. Requires the Family Overlap annex
|
||||
/// (built on demand if missing, same as every other annex here — see
|
||||
/// `obikphylo::siblings::extensions::SiblingExt::family_overlap`'s own
|
||||
/// docs).
|
||||
#[arg(long, value_name = "N")]
|
||||
pub min_shared_family: Option<f64>,
|
||||
|
||||
/// Build (or rebuild) the sibling-count/minorant annex — independent of
|
||||
/// the distance metric below, meant to be run routinely, ahead of any
|
||||
/// SNP-family distance computation that will later consume it. Rebuilding
|
||||
/// it invalidates the annexes derived from it; those are rebuilt here
|
||||
/// only for the options given alongside (`--entropy` the entropy annex,
|
||||
/// `--quartet` the quartet one, `--pair-balance` the pair-weight and
|
||||
/// family-overlap ones, `--family-overlap`/`--min-shared-family` the
|
||||
/// family-overlap one), each under the `--no-ambiguity`/`--only-diploid`
|
||||
/// convention given — anything not asked for is built later, on demand.
|
||||
#[arg(long)]
|
||||
pub sibling_annex: bool,
|
||||
|
||||
/// Tally the sibling-count distribution (CSV) of an already-built annex
|
||||
/// (run with `--sibling-annex` first, in this invocation or an earlier
|
||||
/// one). A separate, occasional diagnostic pass — not run every time the
|
||||
/// annex itself is (re)built.
|
||||
#[arg(long)]
|
||||
pub sibling_stats: bool,
|
||||
|
||||
/// Print just the global family-size histogram (1-4 members) of an
|
||||
/// already-built annex — the `global` row `--sibling-stats` also
|
||||
/// writes, but without the per-genome breakdown, so it skips
|
||||
/// `--sibling-stats`'s cross-partition resolution entirely (annex bits
|
||||
/// only).
|
||||
#[arg(long)]
|
||||
pub sibling_hist: bool,
|
||||
|
||||
/// Write the Family Overlap matrix (CSV) — number of shared *variable*
|
||||
/// families per genome pair, index-wide (`obikphylo::siblings::FamilyOverlap`).
|
||||
/// Built on demand if missing (same as every other annex here); no
|
||||
/// `--sibling-annex` prerequisite beyond that. Every genome is written,
|
||||
/// unfiltered by `--exclude-genome`/`--min-shared-family` — a raw
|
||||
/// coverage diagnostic, not a computation those exclusions are meant to
|
||||
/// protect.
|
||||
#[arg(long)]
|
||||
pub family_overlap: bool,
|
||||
|
||||
/// Write a per-family Shannon entropy report (CSV) — requires an
|
||||
/// already-built sibling annex (`--sibling-annex` first, in this
|
||||
/// invocation or an earlier one). Always a full, unsampled scan of
|
||||
/// every family (`--n-sites`/`--entropy`/`--entropy-sd` below only
|
||||
/// apply to `--pseudo-alignment`, not this).
|
||||
#[arg(long)]
|
||||
pub shannon: bool,
|
||||
|
||||
/// Write the same entropy scan as `--shannon` as a distribution
|
||||
/// (`<prefix>_shannon_summary.csv`): the number of families per entropy
|
||||
/// kind, number of genomes carrying the family, number of those without
|
||||
/// ambiguity and 0.05-bit entropy bin — a few hundred thousand lines at
|
||||
/// most, whatever the size of the index, instead of one line per family.
|
||||
/// Requires an already-built sibling annex, like `--shannon`.
|
||||
#[arg(long)]
|
||||
pub shannon_summary: bool,
|
||||
|
||||
/// Write a SNP-only pseudo-alignment (FASTA) — requires
|
||||
/// `--n-sites <N|all>` and an already-built sibling annex
|
||||
/// (`--sibling-annex` first, in this invocation or an earlier one).
|
||||
#[arg(long)]
|
||||
pub pseudo_alignment: bool,
|
||||
|
||||
/// Write the pseudo-alignment as FASTA (`<prefix>_alignment.fasta`) —
|
||||
/// the default format when neither `--fasta` nor `--vcf` is given. Implies
|
||||
/// `--pseudo-alignment`; combinable with `--vcf`.
|
||||
#[arg(long)]
|
||||
pub fasta: bool,
|
||||
|
||||
/// Write the pseudo-alignment as VCF (`<prefix>_alignment.vcf`): one
|
||||
/// contig `obikmer_<partition>_<layer>` per scanned layer, `POS` the
|
||||
/// family's index within it, `ID` the family's masked canonical k-mer.
|
||||
/// Implies `--pseudo-alignment`; combinable with `--fasta`.
|
||||
#[arg(long)]
|
||||
pub vcf: bool,
|
||||
|
||||
/// Number of variable sites to draw index-wide, or `all` for every site
|
||||
/// that passes the other filters (no sampling then). Mandatory for
|
||||
/// `--pseudo-alignment`/`--sankoff`; optional for a `snp-*`
|
||||
/// `--distance` value, where omitting it also means every site. A
|
||||
/// number is a target, not a guarantee (proportional per-layer
|
||||
/// sampling; see `obikphylo::siblings::sampled_scan`'s own docs).
|
||||
#[arg(long = "n-sites", value_name = "N|all", value_parser = parse_n_sites)]
|
||||
pub n_sites: Option<NSitesArg>,
|
||||
|
||||
/// In `--pseudo-alignment`, treat a genome carrying none of a family's
|
||||
/// observed members (`∅`) as missing data (`?`) rather than a real
|
||||
/// character state.
|
||||
#[arg(long)]
|
||||
pub free_loss: bool,
|
||||
|
||||
/// In `--pseudo-alignment`, treat a genome carrying more than one
|
||||
/// member of a family (ambiguous) as missing data (`?`) rather than an
|
||||
/// IUPAC ambiguity code.
|
||||
#[arg(long)]
|
||||
pub no_ambiguity: bool,
|
||||
|
||||
/// In `--pseudo-alignment`, keep only the genomes carrying one or two
|
||||
/// members of a family (a base, or a two-base IUPAC code) and treat the
|
||||
/// others (three or four members) as missing data (`?`). Milder than
|
||||
/// `--no-ambiguity`, which it excludes.
|
||||
#[arg(long, conflicts_with = "no_ambiguity")]
|
||||
pub only_diploid: bool,
|
||||
|
||||
/// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, keep only
|
||||
/// the families that can separate two pairs of genomes from each other
|
||||
/// (a bipartition): at least two distinct called states, each carried
|
||||
/// by at least two genomes. A singleton variant, or a family carried by
|
||||
/// fewer than four genomes, fits every topology and tells a tree
|
||||
/// nothing. Which genomes count as carriers follows `--no-ambiguity`.
|
||||
#[arg(long)]
|
||||
pub quartet: bool,
|
||||
|
||||
/// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, favour the
|
||||
/// families that cover the pairs of genomes sharing the fewest families
|
||||
/// across the index: each family is weighted by the mean, over the pairs
|
||||
/// of genomes carrying it, of 1 / (number of families the pair shares).
|
||||
/// Evens out how well each pair of genomes is represented in the sample.
|
||||
/// Which genomes count as carriers follows `--no-ambiguity`.
|
||||
#[arg(long)]
|
||||
pub pair_balance: bool,
|
||||
|
||||
/// Write what the draw contains, from the very sites it kept
|
||||
/// (requires `--pseudo-alignment`, `--sankoff` or a `snp-*`
|
||||
/// `--distance`): `<prefix>_sample_genomes.csv` (per genome, the sites it
|
||||
/// has a call in and the informative sites it takes part in),
|
||||
/// `<prefix>_sample_carriers.csv` (sites and informative sites per number
|
||||
/// of carriers) and `<prefix>_sample_overlap.csv` (the sample's own
|
||||
/// genome × genome overlap matrix, to compare with `--family-overlap`).
|
||||
#[arg(long)]
|
||||
pub sample_stats: bool,
|
||||
|
||||
/// Write how the draw was spread over the layers of the index
|
||||
/// (requires `--pseudo-alignment`, `--sankoff` or a `snp-*`
|
||||
/// `--distance`): `<prefix>_sample_plan.csv`, one line per layer with the
|
||||
/// families eligible, their total weight, the layer's quota, the families
|
||||
/// drawn and the ones kept after masking.
|
||||
#[arg(long)]
|
||||
pub sample_plan: bool,
|
||||
|
||||
/// Entropy-biased sampling target (Gaussian kernel mean) for
|
||||
/// `--pseudo-alignment` — activates biasing as soon as this or
|
||||
/// `--entropy-sd` is given; the other defaults to 1.0/0.5.
|
||||
#[arg(long)]
|
||||
pub entropy: Option<f64>,
|
||||
|
||||
/// Entropy-biased sampling kernel width (Gaussian standard deviation)
|
||||
/// for `--pseudo-alignment` — see `--entropy`.
|
||||
#[arg(long)]
|
||||
pub entropy_sd: Option<f64>,
|
||||
|
||||
/// Persist the `snp-*` `--distance` sample (the pairwise substitution
|
||||
/// tally) in `DIR` across separate `obikmer phylo` invocations: a later
|
||||
/// run against the same index with the exact same
|
||||
/// `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/
|
||||
/// `--min-shared-family`/`--entropy`/`--entropy-sd` restores it instead
|
||||
/// of resampling. `DIR` is created if it doesn't exist. If it already
|
||||
/// holds a sample built under *different* selection parameters, this
|
||||
/// run is rejected with an error listing what changed, unless
|
||||
/// `--session-force` is also given (which discards the old sample and
|
||||
/// starts fresh under this run's parameters). Only covers a `snp-*`
|
||||
/// `--distance` today — `--sankoff`/`--tnt`/`--phyg`/`--iqtree` don't
|
||||
/// read or write a `--session` yet.
|
||||
#[arg(long, value_name = "DIR")]
|
||||
pub session: Option<PathBuf>,
|
||||
|
||||
/// With `--session DIR`: overwrite its saved selection parameters (and
|
||||
/// discard its cached sample) instead of erroring out when this run's
|
||||
/// parameters don't match. No effect without `--session`.
|
||||
#[arg(long, requires = "session")]
|
||||
pub session_force: bool,
|
||||
|
||||
/// Calibrate a 16-state Sankoff cost matrix (and its matching
|
||||
/// pseudo-alignment) from an already-built sibling annex — requires
|
||||
/// `--n-sites <N|all>`, and shares `--free-loss`/`--no-ambiguity`/`--quartet`/
|
||||
/// `--entropy`/`--entropy-sd` with `--pseudo-alignment` (one draw, same
|
||||
/// selection feeds both the alignment and every calibration tally).
|
||||
#[arg(long)]
|
||||
pub sankoff: bool,
|
||||
|
||||
/// Exclude genome pairs whose raw SNP ratio exceeds this value from the
|
||||
/// base-pair (composition) calibration `--sankoff` pools — a pair this
|
||||
/// close to substitution saturation carries no information about the
|
||||
/// true substitution spectrum. Does *not* gate the cardinality
|
||||
/// calibration (see `obikphylo::siblings::CardinalityTally`'s own
|
||||
/// docs for why).
|
||||
#[arg(long, default_value = "0.5")]
|
||||
pub sankoff_ratio_ceiling: f64,
|
||||
|
||||
/// Also write <prefix>_sankoff.tnt, a ready-to-run TNT script (`proc
|
||||
/// <file>;`) for the same matrix/alignment `--sankoff` computes —
|
||||
/// recoded to TNT's default xread alphabet (0-9A-F only; TNT rejects
|
||||
/// the wider IUPAC set `--sankoff`'s own output uses unless `nstates
|
||||
/// dna` is set, which imposes TNT's own incompatible DNA encoding
|
||||
/// instead) with integer-scaled costs (TNT's smatrix/cost commands
|
||||
/// reject decimals). Implies `--sankoff`.
|
||||
#[arg(long)]
|
||||
pub tnt: bool,
|
||||
|
||||
/// Also write <prefix>_sankoff.tcm and <prefix>_sankoff.pg, a
|
||||
/// custom-alphabet cost matrix and a ready-to-run PhyG script (`read`/
|
||||
/// `search`/`report`) for the same matrix/alignment `--sankoff`
|
||||
/// computes. Reuses `--sankoff`'s own `_sankoff.fasta` directly — PhyG's
|
||||
/// `tcm:` alphabet is read from the matrix file itself, so the IUPAC+`0`
|
||||
/// alphabet needs no recoding here, unlike `--tnt`. Implies `--sankoff`.
|
||||
#[arg(long)]
|
||||
pub phyg: bool,
|
||||
|
||||
/// Also write <prefix>_iqtree.model and <prefix>_iqtree.fasta, a
|
||||
/// custom-model file and a matching recoded alignment for genuine
|
||||
/// maximum-likelihood inference with IQ-TREE (`iqtree3 -s ...
|
||||
/// --seqtype MORPH -m ...+ASC`) — real branch lengths, unlike
|
||||
/// `--tnt`/`--phyg`'s parsimony step counts. The model is the
|
||||
/// reversible `Q(i,j) = R(i,j)·π_j` construction: `R` (exchangeability,
|
||||
/// symmetric) recovered from the same calibrated cost matrix
|
||||
/// `--sankoff` computes, `π` the real empirical state frequencies
|
||||
/// counted from the alignment. Only the states that actually occur in
|
||||
/// this alignment are kept, compactly renumbered (IQ-TREE infers its
|
||||
/// state count from the alignment itself, and a gap in the numbering
|
||||
/// would silently misalign the model file). Implies `--sankoff`.
|
||||
#[arg(long)]
|
||||
pub iqtree: bool,
|
||||
|
||||
/// Under `--iqtree --free-loss`, also recode to `?` (the same
|
||||
/// missing-data treatment as `-`) any state whose empirical frequency
|
||||
/// in the alignment falls below this threshold — not just genuinely
|
||||
/// absent calls. States encoding 3 or 4 simultaneously-observed central
|
||||
/// bases (IUPAC `V`/`H`/`K`.../`N` for 3, `N` for 4) are rare by
|
||||
/// construction and often land in exactly this low-frequency range —
|
||||
/// more likely assembly/detection noise than a genuine, widely-preserved
|
||||
/// multi-way polymorphism, the same "sampling failure, not true signal"
|
||||
/// reasoning `--free-loss` already applies to absence. No effect
|
||||
/// without `--free-loss` (there is no missing-data symbol to recode to
|
||||
/// otherwise). `<prefix>_iqtree_states.csv` reports the frequency
|
||||
/// actually used to decide.
|
||||
#[arg(long, default_value = "0.001")]
|
||||
pub iqtree_min_freq: f64,
|
||||
|
||||
/// Scale factor applied before rounding real-valued costs to the
|
||||
/// integers both `--tnt`'s smatrix/cost commands and `--phyg`'s `tcm:`
|
||||
/// matrix require. Keep this small: the total tree score is this scale
|
||||
/// times the sum of per-character costs across every character, and
|
||||
/// there are hints in TNT's own manual that at least some of its
|
||||
/// internal accumulators are 32-bit — a large scale risks a silent
|
||||
/// integer overflow (undetectable, not just a crash) far more costly
|
||||
/// than the resolution a bigger factor would buy. Shared between `--tnt`
|
||||
/// and `--phyg` rather than split into two flags: both scale the same
|
||||
/// calibrated matrix for the same reason (integer-only cost commands).
|
||||
#[arg(long, default_value = "100")]
|
||||
pub sankoff_cost_scale: f64,
|
||||
|
||||
/// Distance matrix to compute and write — none is computed unless this
|
||||
/// is given, or `--nj`/`--upgma`/`--shared-kmers` is (they need a matrix
|
||||
/// and use `jaccard` then, see [`distance_to_compute`]). Either a
|
||||
/// whole-index metric (`jaccard`,
|
||||
/// `mash`, `hamming`, `bray-curtis`, ...) or a `snp-*` correction over
|
||||
/// the central-position SNP substitution spectrum (`snp-raw`, `snp-jc`,
|
||||
/// `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`) —
|
||||
/// the latter route to a different computation entirely
|
||||
/// (`obikphylo::siblings::sampled_scan` + `distance_matrix`, requires
|
||||
/// `--sibling-annex` first; see
|
||||
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance`
|
||||
/// unification" for the full catalog and why LogDet/Tajima-Nei/F84/
|
||||
/// HKY85 aren't offered yet).
|
||||
#[arg(long, value_enum)]
|
||||
pub distance: Option<DistanceArg>,
|
||||
|
||||
/// Rate-heterogeneity correction (Jin-Nei gamma shape parameter `α`)
|
||||
/// for `snp-*` `--distance` values that support it
|
||||
/// (`obikphylo::SnpDistanceKind::supports_gamma`: every one except
|
||||
/// `snp-raw`/`snp-tv`, which have nothing to correct/are deliberately
|
||||
/// uncorrected). Has no effect on the whole-index metrics. Rejected at
|
||||
/// runtime if given alongside an unsupported `--distance` value.
|
||||
/// Either a positive number (fixed `alpha`) or `auto`/`estimate` to fit
|
||||
/// `alpha` from the data itself — method-of-moments over the
|
||||
/// over-dispersion of substitution rate across partitions.
|
||||
#[arg(long, value_name = "ALPHA|auto", value_parser = parse_gamma_shape)]
|
||||
pub gamma_shape: Option<GammaShapeArg>,
|
||||
|
||||
/// Minimum count to consider a kmer present when computing Jaccard on count indexes
|
||||
#[arg(long, default_value = "1")]
|
||||
pub presence_threshold: u32,
|
||||
|
||||
/// Write the primary distance matrix as plain CSV instead of the
|
||||
/// default relaxed-PHYLIP format (`n` on the first line, then one
|
||||
/// `label<TAB>value...` row per genome — no 10-character label
|
||||
/// truncation, unlike strict PHYLIP, not yet offered here). PHYLIP is
|
||||
/// the default because it's what external NJ tools (PHYLIP `neighbor`,
|
||||
/// FastME, T-REX, SplitsTree) actually read; CSV stays available for
|
||||
/// scripting/inspection. Only affects the primary distance matrix —
|
||||
/// `--shared-kmers` keeps its own CSV-only format regardless of this
|
||||
/// flag.
|
||||
#[arg(long)]
|
||||
pub csv: bool,
|
||||
|
||||
/// Also output the shared-kmer count matrix (CSV)
|
||||
#[arg(long)]
|
||||
pub shared_kmers: bool,
|
||||
|
||||
/// Compute and write a Neighbor-Joining tree (Newick)
|
||||
#[arg(long)]
|
||||
pub nj: bool,
|
||||
|
||||
/// Compute and write a UPGMA tree (Newick)
|
||||
#[arg(long)]
|
||||
pub upgma: bool,
|
||||
|
||||
/// Output prefix: <prefix>_dist.csv, <prefix>_shared.csv, <prefix>_nj.nwk,
|
||||
/// <prefix>_upgma.nwk. If omitted, the distance matrix is written to stdout.
|
||||
#[arg(short, long)]
|
||||
pub output: Option<PathBuf>,
|
||||
}
|
||||
|
||||
impl PhyloArgs {
|
||||
/// How genomes carrying several members of a family are treated.
|
||||
pub fn ambiguity(&self) -> AmbiguityPolicy {
|
||||
AmbiguityPolicy::from_flags(self.no_ambiguity, self.only_diploid)
|
||||
}
|
||||
|
||||
/// The derived annexes the options of this invocation read, all under
|
||||
/// [`ambiguity`](Self::ambiguity).
|
||||
pub fn annex_selection(&self) -> AnnexSelection {
|
||||
AnnexSelection {
|
||||
policy: self.ambiguity(),
|
||||
entropy: self.entropy.is_some(),
|
||||
quartet: self.quartet,
|
||||
overlap: self.family_overlap || self.min_shared_family.is_some() || self.pair_balance,
|
||||
pair_weight: self.pair_balance,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a pseudo-alignment must be drawn and written (`--pseudo-alignment`,
|
||||
/// or any of its output-format flags).
|
||||
pub fn alignment_output(&self) -> bool {
|
||||
self.pseudo_alignment || self.fasta || self.vcf
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
use std::path::PathBuf;
|
||||
|
||||
use clap::Args;
|
||||
|
||||
use super::super::values::{DistanceArg, GammaShapeArg, parse_gamma_shape};
|
||||
|
||||
/// Distance estimation and tree reconstruction.
|
||||
#[derive(Args)]
|
||||
#[command(next_help_heading = "Distance estimation and tree reconstruction")]
|
||||
pub struct DistanceArgs {
|
||||
/// Distance used to compare genomes
|
||||
///
|
||||
/// Computes a genome-by-genome distance matrix. Two families of
|
||||
/// distances are available.
|
||||
///
|
||||
/// Whole-genome distances, based on the k-mer content of each genome.
|
||||
/// The first three compare which k-mers are present or absent (on an
|
||||
/// index that stores counts, a k-mer is present when its count reaches
|
||||
/// --presence-threshold):
|
||||
/// jaccard Jaccard distance
|
||||
/// mash Mash distance, a divergence estimate derived
|
||||
/// from the Jaccard index
|
||||
/// hamming number of k-mers present in one genome and
|
||||
/// absent from the other
|
||||
/// The others compare k-mer counts, and need an index that stores them:
|
||||
/// bray-curtis Bray-Curtis dissimilarity on k-mer counts
|
||||
/// relfreq-bray-curtis Bray-Curtis dissimilarity on counts divided
|
||||
/// by each genome's total count
|
||||
/// euclidean Euclidean distance on k-mer counts
|
||||
/// relfreq-euclidean Euclidean distance on relative frequencies
|
||||
/// hellinger Hellinger distance on k-mer counts
|
||||
/// hellinger-euclidean Euclidean distance on square roots of
|
||||
/// relative frequencies (unnormalised Hellinger)
|
||||
///
|
||||
/// SNP-based distances, calculated from the nucleotide differences at
|
||||
/// the central position of related k-mers:
|
||||
/// snp-raw proportion of differing sites, no correction
|
||||
/// snp-jc Jukes-Cantor correction
|
||||
/// snp-k2p Kimura two-parameter correction
|
||||
/// snp-k81 Kimura three-parameter correction
|
||||
/// snp-f81 Felsenstein 1981 correction
|
||||
/// snp-t92 Tamura 1992 correction
|
||||
/// snp-tn93 Tamura-Nei 1993 correction
|
||||
/// snp-tv proportion of transversions only, no correction
|
||||
/// They use every eligible variable site, or a sample drawn with
|
||||
/// --n-sites.
|
||||
#[arg(
|
||||
long,
|
||||
value_enum,
|
||||
verbatim_doc_comment,
|
||||
hide_possible_values = true,
|
||||
value_name = "DISTANCE"
|
||||
)]
|
||||
pub distance: Option<DistanceArg>,
|
||||
|
||||
/// Minimum k-mer count to count as present
|
||||
///
|
||||
/// Converts counts into presence/absence for the Jaccard, Mash and
|
||||
/// Hamming distances and for the shared k-mer counts, when the index
|
||||
/// stores counts: a k-mer is present in a genome only when its count reaches
|
||||
/// this threshold. It has no effect on indexes without counts, nor on
|
||||
/// the distances that use the counts themselves.
|
||||
#[arg(long, default_value = "1")]
|
||||
pub presence_threshold: u32,
|
||||
|
||||
/// Account for different evolution rates among sites
|
||||
///
|
||||
/// Applies a gamma rate-heterogeneity correction to SNP-based
|
||||
/// distances. A positive number is the gamma shape parameter α. `auto`
|
||||
/// (or `estimate`) estimates α from the data, using the variation of
|
||||
/// the substitution rate across the partitions of the index.
|
||||
///
|
||||
/// Available for every snp-* distance except snp-raw and snp-tv, which
|
||||
/// have no evolutionary correction to refine. It has no effect on
|
||||
/// whole-genome distances, and combining it with a distance that does
|
||||
/// not support it is an error.
|
||||
#[arg(long, value_name = "ALPHA|auto", value_parser = parse_gamma_shape)]
|
||||
pub gamma_shape: Option<GammaShapeArg>,
|
||||
|
||||
/// Save and reuse the sample of a SNP-based distance
|
||||
///
|
||||
/// Saves, in this directory, the sample's tally of pairwise differences
|
||||
/// between genomes, so that a later run on the same index can reuse it
|
||||
/// instead of drawing again. The directory is created if needed.
|
||||
///
|
||||
/// The saved sample is reused only if these options are identical:
|
||||
/// --n-sites, --free-loss, --no-ambiguity, --quartet, --exclude-genome,
|
||||
/// --min-shared-family, --entropy and --entropy-sd. Otherwise the run
|
||||
/// stops and lists the differences, unless --session-force is given.
|
||||
///
|
||||
/// For now only SNP-based distances use sessions; Sankoff analyses and
|
||||
/// the TNT, PhyG and IQ-TREE exports neither read nor write them.
|
||||
#[arg(long, value_name = "DIR")]
|
||||
pub session: Option<PathBuf>,
|
||||
|
||||
/// Replace a saved sample made with other parameters
|
||||
///
|
||||
/// With --session, discards the saved sample when its parameters differ
|
||||
/// from this run's, and draws a new one. Without this option, a
|
||||
/// difference is an error.
|
||||
#[arg(long, requires = "session")]
|
||||
pub session_force: bool,
|
||||
|
||||
/// Write the distance matrix as CSV instead of PHYLIP
|
||||
///
|
||||
/// By default the main matrix is written in relaxed PHYLIP format: the
|
||||
/// number of genomes on the first line, then one row per genome made of
|
||||
/// its label (never truncated) and its distances to the others. CSV is
|
||||
/// easier to load in scripts or spreadsheets. Only the main matrix is
|
||||
/// affected: the --shared-kmers matrix is always CSV.
|
||||
#[arg(long)]
|
||||
pub csv: bool,
|
||||
|
||||
/// Also write the number of k-mers shared by each pair of genomes
|
||||
///
|
||||
/// Writes a genome-by-genome CSV matrix of shared k-mer counts, to
|
||||
/// `PREFIX_shared.csv`.
|
||||
#[arg(long)]
|
||||
pub shared_kmers: bool,
|
||||
|
||||
/// Build a Neighbor-Joining tree
|
||||
///
|
||||
/// Builds the tree from the distance matrix and writes it in Newick
|
||||
/// format to `PREFIX_nj.nwk`. If --distance is not given, the Jaccard
|
||||
/// distance is used.
|
||||
#[arg(long)]
|
||||
pub nj: bool,
|
||||
|
||||
/// Build a UPGMA tree
|
||||
///
|
||||
/// Builds a tree with the Unweighted Pair Group Method with Arithmetic
|
||||
/// Mean and writes it in Newick format to `PREFIX_upgma.nwk`. If
|
||||
/// --distance is not given, the Jaccard distance is used.
|
||||
#[arg(long)]
|
||||
pub upgma: bool,
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
//! Command-line arguments of `obikmer phylo`.
|
||||
//!
|
||||
//! The options are split by theme into flattened groups, each carrying its own
|
||||
//! help heading and its own help texts (short for `-h`, long for `--help`).
|
||||
//! Those texts are written for biologists: no internal names, no design
|
||||
//! rationale — that belongs in the documentation.
|
||||
|
||||
mod distance;
|
||||
mod outputs;
|
||||
mod sampling;
|
||||
mod select;
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use clap::Args;
|
||||
use obikphylo::siblings::{AmbiguityPolicy, AnnexSelection};
|
||||
|
||||
pub use distance::DistanceArgs;
|
||||
pub use outputs::{AlignArgs, CostsArgs};
|
||||
pub use sampling::SamplingArgs;
|
||||
pub use select::{AnnexArgs, SelectArgs};
|
||||
|
||||
/// Text shown at the end of `obikmer phylo --help`.
|
||||
pub const WORKFLOW_HELP: &str = "\
|
||||
Typical workflows
|
||||
|
||||
Compare genome-wide k-mer content and infer a Neighbor-Joining tree:
|
||||
obikmer phylo INDEX --distance jaccard --nj -o out
|
||||
|
||||
Compute SNP-based distances and infer a Neighbor-Joining tree:
|
||||
obikmer phylo INDEX --distance snp-k2p --n-sites 200000 --nj -o out
|
||||
|
||||
Generate a SNP pseudo-alignment:
|
||||
obikmer phylo INDEX --pseudo-alignment --n-sites 200000 -o out
|
||||
|
||||
Prepare a maximum-likelihood analysis for IQ-TREE:
|
||||
obikmer phylo INDEX --iqtree --n-sites 200000 -o out
|
||||
|
||||
Option dependencies
|
||||
|
||||
--fasta, --vcf imply --pseudo-alignment
|
||||
--tnt, --phyg, --iqtree imply --sankoff
|
||||
--nj, --upgma use the Jaccard distance when --distance
|
||||
is not given
|
||||
--pseudo-alignment, --sankoff require --n-sites
|
||||
--session-force requires --session";
|
||||
|
||||
/// Genome comparison and phylogenetic analysis
|
||||
///
|
||||
/// Compares the genomes stored in an index and performs phylogenetic
|
||||
/// analyses. Two complementary approaches are available:
|
||||
///
|
||||
/// - Whole-genome distances compare the k-mer content of the genomes,
|
||||
/// without any alignment.
|
||||
/// - SNP-based distances and analyses compare nucleotide variants at the
|
||||
/// central position of related k-mers.
|
||||
///
|
||||
/// Distance matrices can be used to build Neighbor-Joining or UPGMA trees.
|
||||
/// SNP-based analyses can also produce pseudo-alignments and calibrated
|
||||
/// substitution costs, for phylogenetic inference with TNT, PhyG or
|
||||
/// IQ-TREE.
|
||||
///
|
||||
/// SNP families
|
||||
///
|
||||
/// SNP-based analyses use groups of k-mers that differ only at their
|
||||
/// central nucleotide. These groups are called families. A variable family
|
||||
/// is one character of the analysis, and the variants a genome carries
|
||||
/// give its state for that character.
|
||||
#[derive(Args)]
|
||||
#[command(verbatim_doc_comment)]
|
||||
pub struct PhyloArgs {
|
||||
/// Index directory
|
||||
pub index: PathBuf,
|
||||
|
||||
/// Prefix of the output files
|
||||
///
|
||||
/// Output files are named after this prefix: for example
|
||||
/// `PREFIX_dist.phy` (or `PREFIX_dist.csv` with --csv), `PREFIX_nj.nwk`
|
||||
/// and `PREFIX_upgma.nwk`, plus the files described with the other
|
||||
/// options.
|
||||
///
|
||||
/// Without this option, the main distance matrix is written to standard
|
||||
/// output, and the other files are written in the current directory
|
||||
/// under their name without prefix (for example `nj.nwk`).
|
||||
#[arg(short, long)]
|
||||
pub output: Option<PathBuf>,
|
||||
|
||||
#[command(flatten)]
|
||||
pub dist: DistanceArgs,
|
||||
|
||||
#[command(flatten)]
|
||||
pub select: SelectArgs,
|
||||
|
||||
#[command(flatten)]
|
||||
pub sample: SamplingArgs,
|
||||
|
||||
#[command(flatten)]
|
||||
pub annex: AnnexArgs,
|
||||
|
||||
#[command(flatten)]
|
||||
pub align: AlignArgs,
|
||||
|
||||
#[command(flatten)]
|
||||
pub costs: CostsArgs,
|
||||
}
|
||||
|
||||
impl PhyloArgs {
|
||||
/// How genomes carrying several members of a family are treated.
|
||||
pub fn ambiguity(&self) -> AmbiguityPolicy {
|
||||
AmbiguityPolicy::from_flags(self.sample.no_ambiguity, self.sample.only_diploid)
|
||||
}
|
||||
|
||||
/// The derived annexes the options of this invocation read, all under
|
||||
/// [`ambiguity`](Self::ambiguity).
|
||||
pub fn annex_selection(&self) -> AnnexSelection {
|
||||
AnnexSelection {
|
||||
policy: self.ambiguity(),
|
||||
free_loss: self.sample.free_loss,
|
||||
entropy: self.sample.entropy.is_some(),
|
||||
quartet: self.sample.quartet,
|
||||
overlap: self.annex.family_overlap
|
||||
|| self.select.min_shared_family.is_some()
|
||||
|| self.sample.pair_balance,
|
||||
pair_weight: self.sample.pair_balance,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a pseudo-alignment must be drawn and written (`--pseudo-alignment`,
|
||||
/// or any of its output-format flags).
|
||||
pub fn alignment_output(&self) -> bool {
|
||||
self.align.pseudo_alignment || self.align.fasta || self.align.vcf
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
use clap::Args;
|
||||
|
||||
/// SNP pseudo-alignment.
|
||||
///
|
||||
/// A pseudo-alignment has one column per variable family and one state per
|
||||
/// genome. Unlike a conventional sequence alignment, its columns come
|
||||
/// directly from groups of related k-mers of the index.
|
||||
#[derive(Args)]
|
||||
#[command(next_help_heading = "SNP pseudo-alignment")]
|
||||
pub struct AlignArgs {
|
||||
/// Write a SNP pseudo-alignment
|
||||
///
|
||||
/// Writes an alignment with one column per selected variable family;
|
||||
/// the states are the central nucleotides of the k-mers of the family.
|
||||
/// The format is FASTA unless --vcf is given.
|
||||
///
|
||||
/// The number of sites is set with --n-sites (required). The options of
|
||||
/// the "SNP character selection and sampling" group control which
|
||||
/// families are included and how ambiguous states are written.
|
||||
#[arg(long)]
|
||||
pub pseudo_alignment: bool,
|
||||
|
||||
/// Write the pseudo-alignment in FASTA format
|
||||
///
|
||||
/// Writes `PREFIX_alignment.fasta`. This is the default format. Implies
|
||||
/// --pseudo-alignment; can be combined with --vcf to get both formats.
|
||||
#[arg(long)]
|
||||
pub fasta: bool,
|
||||
|
||||
/// Write the pseudo-alignment in VCF format
|
||||
///
|
||||
/// Writes `PREFIX_alignment.vcf`. Each scanned index layer is a contig
|
||||
/// named `obikmer_<partition>_<layer>`; `POS` is the position of the
|
||||
/// family within the layer and `ID` is the masked canonical k-mer of the
|
||||
/// family. Implies --pseudo-alignment; can be combined with --fasta to
|
||||
/// get both formats.
|
||||
#[arg(long)]
|
||||
pub vcf: bool,
|
||||
}
|
||||
|
||||
/// Sankoff cost calibration and exports for phylogenetic software.
|
||||
///
|
||||
/// Sankoff parsimony lets different changes between states have different
|
||||
/// costs. These costs are estimated here from the observed variation, and
|
||||
/// exported for external phylogenetic software.
|
||||
#[derive(Args)]
|
||||
#[command(next_help_heading = "Sankoff cost calibration and phylogenetic software exports")]
|
||||
pub struct CostsArgs {
|
||||
/// Estimate a Sankoff cost matrix and its alignment
|
||||
///
|
||||
/// Estimates, from the selected variable families, the cost of changing
|
||||
/// from each of the 16 possible states to each other state, for weighted
|
||||
/// parsimony. Writes `PREFIX_sankoff_matrix.csv` (costs),
|
||||
/// `PREFIX_sankoff_params.yaml` (parameters) and
|
||||
/// `PREFIX_sankoff.fasta` (alignment).
|
||||
///
|
||||
/// Needs --n-sites. The selection and sampling options are the same as
|
||||
/// for pseudo-alignments, and a single sample of families serves both
|
||||
/// the alignment and the estimation of the costs.
|
||||
#[arg(long)]
|
||||
pub sankoff: bool,
|
||||
|
||||
/// Leave very divergent genome pairs out of the calibration
|
||||
///
|
||||
/// Genome pairs whose raw SNP ratio exceeds this value are not used to
|
||||
/// estimate the costs between nucleotides: at such divergence, repeated
|
||||
/// substitutions at the same site blur the original substitution
|
||||
/// pattern. The costs related to the number of variants are not
|
||||
/// affected.
|
||||
#[arg(long, default_value = "0.5")]
|
||||
pub sankoff_ratio_ceiling: f64,
|
||||
|
||||
/// Export the Sankoff analysis for TNT
|
||||
///
|
||||
/// Writes `PREFIX_sankoff.tnt`, a script ready to run in TNT with
|
||||
/// `proc FILE;`. States are recoded with TNT's default alphabet (0-9 and
|
||||
/// A-F) and costs are turned into integers with --sankoff-cost-scale,
|
||||
/// as TNT requires. Implies --sankoff.
|
||||
#[arg(long)]
|
||||
pub tnt: bool,
|
||||
|
||||
/// Export the Sankoff analysis for PhyG
|
||||
///
|
||||
/// Writes:
|
||||
/// - `PREFIX_sankoff.tcm`: the cost matrix;
|
||||
/// - `PREFIX_sankoff.pg`: the PhyG script.
|
||||
/// The alignment is the `PREFIX_sankoff.fasta` written by --sankoff,
|
||||
/// used without recoding. Costs are turned into integers with
|
||||
/// --sankoff-cost-scale. Implies --sankoff.
|
||||
#[arg(long, verbatim_doc_comment)]
|
||||
pub phyg: bool,
|
||||
|
||||
/// Export a model and alignment for IQ-TREE maximum likelihood
|
||||
///
|
||||
/// Writes:
|
||||
/// - `PREFIX_iqtree.model`: a substitution model derived from the
|
||||
/// calibrated Sankoff costs;
|
||||
/// - `PREFIX_iqtree.fasta`: the alignment with recoded states.
|
||||
/// The model combines the calibrated costs with the frequencies of the
|
||||
/// states observed in the alignment. Only the states that actually
|
||||
/// occur are kept, renumbered consecutively.
|
||||
///
|
||||
/// Run IQ-TREE with:
|
||||
/// iqtree3 -s PREFIX_iqtree.fasta --seqtype MORPH -m ...+ASC
|
||||
/// giving the model file to -m. Unlike the parsimony analyses of TNT
|
||||
/// and PhyG, maximum likelihood estimates branch lengths.
|
||||
///
|
||||
/// Implies --sankoff.
|
||||
#[arg(long, verbatim_doc_comment)]
|
||||
pub iqtree: bool,
|
||||
|
||||
/// Treat rare states as missing in the IQ-TREE export
|
||||
///
|
||||
/// With --iqtree and --free-loss, the states whose frequency in the
|
||||
/// alignment is below this value are recoded as missing data (`?`).
|
||||
/// States combining three or four nucleotides are typically that rare,
|
||||
/// and are more likely noise than a real polymorphism.
|
||||
///
|
||||
/// No effect without --free-loss. The frequency actually applied is
|
||||
/// reported in `PREFIX_iqtree_states.csv`.
|
||||
#[arg(long, default_value = "0.001")]
|
||||
pub iqtree_min_freq: f64,
|
||||
|
||||
/// Factor turning costs into integers for TNT and PhyG
|
||||
///
|
||||
/// TNT and PhyG only accept integer costs: the calibrated costs are
|
||||
/// multiplied by this factor, then rounded. Keep it small. The tree
|
||||
/// score is multiplied by the same factor, and a very large value can
|
||||
/// overflow integers in TNT without any error.
|
||||
#[arg(long, default_value = "100")]
|
||||
pub sankoff_cost_scale: f64,
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
use clap::Args;
|
||||
|
||||
use super::super::values::{NSitesArg, parse_n_sites};
|
||||
|
||||
/// SNP character selection and sampling.
|
||||
///
|
||||
/// SNP-based analyses can use every eligible variable family, or a subset
|
||||
/// drawn from the index. These options control which characters are
|
||||
/// selected, which genomes and states are kept, and how the sample is spread.
|
||||
#[derive(Args)]
|
||||
#[command(next_help_heading = "SNP character selection and sampling")]
|
||||
pub struct SamplingArgs {
|
||||
/// Number of variable sites to select
|
||||
///
|
||||
/// Target number of variable families to draw from the index. Either a
|
||||
/// positive integer, or `all` to keep every family that passes the other
|
||||
/// filters, with no sampling.
|
||||
///
|
||||
/// Required for pseudo-alignments and Sankoff analyses. Optional for
|
||||
/// SNP-based distances: when omitted, every eligible site is used.
|
||||
///
|
||||
/// Sites are drawn in proportion across the layers of the index, so the
|
||||
/// number actually selected can differ slightly from the target.
|
||||
#[arg(long = "n-sites", value_name = "N|all", value_parser = parse_n_sites)]
|
||||
pub n_sites: Option<NSitesArg>,
|
||||
|
||||
/// Treat genomes lacking a family as missing data (?)
|
||||
///
|
||||
/// When a genome carries none of the variants found in a family, its
|
||||
/// state is coded as missing data (`?`) instead of a real state.
|
||||
///
|
||||
/// Applies to every SNP-based analysis: distances, pseudo-alignments and
|
||||
/// the Sankoff calibration with its exports.
|
||||
#[arg(long)]
|
||||
pub free_loss: bool,
|
||||
|
||||
/// Treat genomes with several variants as missing data (?)
|
||||
///
|
||||
/// A genome carrying more than one nucleotide variant of a family is
|
||||
/// coded as missing data (`?`) instead of an IUPAC ambiguity code.
|
||||
///
|
||||
/// This also decides which genomes count as carriers for --quartet and
|
||||
/// --pair-balance.
|
||||
#[arg(long)]
|
||||
pub no_ambiguity: bool,
|
||||
|
||||
/// Keep genomes with one or two variants per family
|
||||
///
|
||||
/// A genome with one variant is written as a single nucleotide, a genome
|
||||
/// with two variants as the matching two-base IUPAC code. Genomes with
|
||||
/// three or four variants are coded as missing data (`?`).
|
||||
///
|
||||
/// A milder alternative to --no-ambiguity; the two cannot be combined.
|
||||
#[arg(long, conflicts_with = "no_ambiguity")]
|
||||
pub only_diploid: bool,
|
||||
|
||||
/// Keep families that can tell tree topologies apart
|
||||
///
|
||||
/// A family is kept only when at least two different states are each
|
||||
/// found in at least two genomes. A variant seen in a single genome, or
|
||||
/// a family carried by fewer than four genomes, fits every arrangement
|
||||
/// of the genomes and teaches nothing about the tree.
|
||||
///
|
||||
/// The states are those of the alignment. Without --free-loss, absence
|
||||
/// is a state of its own, so two genomes with a base and two genomes
|
||||
/// lacking the family form two pairs. With --free-loss, absence is
|
||||
/// missing data and never forms a pair. Which genomes count as carriers
|
||||
/// also follows --no-ambiguity. Applies to pseudo-alignments, Sankoff
|
||||
/// analyses and SNP-based distances.
|
||||
#[arg(long)]
|
||||
pub quartet: bool,
|
||||
|
||||
/// Favour families covering poorly represented genome pairs
|
||||
///
|
||||
/// Each family gets a weight: the mean, over the pairs of genomes
|
||||
/// carrying it, of 1 / (number of families the pair shares). Families
|
||||
/// covering pairs that share little information are therefore drawn
|
||||
/// more often, which balances how well each pair is represented in the
|
||||
/// sample.
|
||||
///
|
||||
/// Which genomes count as carriers follows --no-ambiguity. Applies to
|
||||
/// pseudo-alignments, Sankoff analyses and SNP-based distances.
|
||||
#[arg(long)]
|
||||
pub pair_balance: bool,
|
||||
|
||||
/// Favour families of a given Shannon entropy
|
||||
///
|
||||
/// Families are drawn with a weight following a bell-shaped curve
|
||||
/// centred on this entropy. Giving this option, or --entropy-sd, turns
|
||||
/// the weighting on; the other one then takes its default (centre 1.0
|
||||
/// bit, standard deviation 0.5 bit).
|
||||
#[arg(long)]
|
||||
pub entropy: Option<f64>,
|
||||
|
||||
/// Width of the entropy weighting
|
||||
///
|
||||
/// Standard deviation of the bell-shaped curve used by --entropy.
|
||||
/// Giving this option also turns the weighting on.
|
||||
#[arg(long)]
|
||||
pub entropy_sd: Option<f64>,
|
||||
|
||||
/// Write statistics about the selected sample
|
||||
///
|
||||
/// Writes three CSV files describing the sites actually drawn:
|
||||
/// - `PREFIX_sample_genomes.csv`: for each genome, the number of sites
|
||||
/// where it has a called state and of informative sites;
|
||||
/// - `PREFIX_sample_carriers.csv`: the number of sites and of
|
||||
/// informative sites by number of carrier genomes;
|
||||
/// - `PREFIX_sample_overlap.csv`: the genome-by-genome matrix of
|
||||
/// families shared within the sample.
|
||||
/// The overlap matrix can be compared with the one written by
|
||||
/// --family-overlap, to check how well the sample represents the index.
|
||||
///
|
||||
/// Needs --pseudo-alignment, --sankoff or a SNP-based distance.
|
||||
#[arg(long, verbatim_doc_comment)]
|
||||
pub sample_stats: bool,
|
||||
|
||||
/// Write how the sites were spread over the index layers
|
||||
///
|
||||
/// Writes `PREFIX_sample_plan.csv`, one line per layer: number of
|
||||
/// eligible families, their total weight, the layer's quota, the number
|
||||
/// of families drawn and the number kept after masking.
|
||||
///
|
||||
/// Needs --pseudo-alignment, --sankoff or a SNP-based distance.
|
||||
#[arg(long)]
|
||||
pub sample_plan: bool,
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
use clap::Args;
|
||||
|
||||
/// Genome selection and filtering.
|
||||
#[derive(Args)]
|
||||
#[command(next_help_heading = "Genome selection and filtering")]
|
||||
pub struct SelectArgs {
|
||||
/// Exclude a genome from the analysis (repeatable)
|
||||
///
|
||||
/// Removes the genome identified by its exact label. Repeat the option
|
||||
/// to exclude several genomes.
|
||||
///
|
||||
/// For SNP-based analyses, excluded genomes are removed before the
|
||||
/// variable sites are selected: a family that was variable only because
|
||||
/// of an excluded genome is no longer considered.
|
||||
///
|
||||
/// Excluded genomes are also removed from the distance and shared-k-mer
|
||||
/// matrices; the distances between the remaining genomes are not
|
||||
/// otherwise modified.
|
||||
#[arg(long = "exclude-genome", value_name = "LABEL")]
|
||||
pub exclude_genome: Vec<String>,
|
||||
|
||||
/// Exclude genomes sharing too few variable families
|
||||
///
|
||||
/// Removes the genomes whose average number of variable families shared
|
||||
/// with the other genomes is below N. The average is the per-genome mean
|
||||
/// of the matrix written by --family-overlap. This comes on top of
|
||||
/// --exclude-genome.
|
||||
///
|
||||
/// The filter removes genomes that carry too little information for
|
||||
/// SNP-based comparisons. It applies to SNP-based distances,
|
||||
/// pseudo-alignments and Sankoff analyses, not to whole-genome distances
|
||||
/// such as Jaccard: a genome with little SNP information is not
|
||||
/// necessarily a genome whose overall k-mer content is divergent.
|
||||
///
|
||||
/// The family-overlap matrix is computed automatically if needed.
|
||||
#[arg(long, value_name = "N")]
|
||||
pub min_shared_family: Option<f64>,
|
||||
}
|
||||
|
||||
/// SNP family preparation and diagnostics.
|
||||
#[derive(Args)]
|
||||
#[command(next_help_heading = "SNP family preparation and diagnostics")]
|
||||
pub struct AnnexArgs {
|
||||
/// Prepare in advance the tables used by SNP analyses
|
||||
///
|
||||
/// SNP-based distances, pseudo-alignments, Sankoff analyses and entropy
|
||||
/// calculations rely on precomputed tables describing the families of
|
||||
/// k-mers that differ only at their central nucleotide. They are
|
||||
/// prepared automatically when first needed, so this option is never
|
||||
/// required. It lets you prepare them beforehand, for example before a
|
||||
/// series of analyses on the same index, or force them to be rebuilt.
|
||||
///
|
||||
/// Rebuilding the main table invalidates the tables derived from it.
|
||||
/// Among those, only the ones requested in the same command are rebuilt
|
||||
/// immediately (--entropy, --quartet, --pair-balance, --family-overlap
|
||||
/// and --min-shared-family), following the treatment of ambiguous
|
||||
/// states given by --no-ambiguity or --only-diploid. The others are
|
||||
/// rebuilt when next needed.
|
||||
#[arg(long)]
|
||||
pub sibling_annex: bool,
|
||||
|
||||
/// Write the distribution of family sizes, by genome
|
||||
///
|
||||
/// Writes `PREFIX_siblings.csv`. For each genome, it gives the number of
|
||||
/// families of each size (1 to 4 members) in which the genome carries at
|
||||
/// least one member. A final `global` row gives the number of families
|
||||
/// of each size over the whole index. Mostly a diagnostic.
|
||||
#[arg(long)]
|
||||
pub sibling_stats: bool,
|
||||
|
||||
/// Write the global distribution of family sizes
|
||||
///
|
||||
/// Writes `PREFIX_sibling_hist.csv`, the number of families having 1 to
|
||||
/// 4 members over the whole index. These are the figures of the `global`
|
||||
/// row of --sibling-stats, without the per-genome breakdown, and they
|
||||
/// are faster to obtain.
|
||||
#[arg(long)]
|
||||
pub sibling_hist: bool,
|
||||
|
||||
/// Write the number of variable families shared by each genome pair
|
||||
///
|
||||
/// Writes `PREFIX_family_overlap.csv`, a genome-by-genome matrix giving,
|
||||
/// for each pair of genomes, the number of variable families they share
|
||||
/// over the whole index. It tells how much SNP information is available
|
||||
/// to compare each pair, and helps spot genomes that are poorly covered.
|
||||
///
|
||||
/// Every genome is included, whatever --exclude-genome and
|
||||
/// --min-shared-family say.
|
||||
#[arg(long)]
|
||||
pub family_overlap: bool,
|
||||
|
||||
/// Write the Shannon entropy of every family
|
||||
///
|
||||
/// Writes `PREFIX_entropy.csv`, one line per family. All families are
|
||||
/// scanned, with no sampling: --n-sites, --entropy and --entropy-sd do
|
||||
/// not apply.
|
||||
#[arg(long)]
|
||||
pub shannon: bool,
|
||||
|
||||
/// Summarise the distribution of family entropy
|
||||
///
|
||||
/// Same complete scan as --shannon, but writes a compact summary,
|
||||
/// `PREFIX_shannon_summary.csv`, instead of one line per family. It
|
||||
/// counts the families according to:
|
||||
/// - entropy category;
|
||||
/// - number of genomes carrying the family;
|
||||
/// - number of those carriers without ambiguity;
|
||||
/// - entropy bins of 0.05 bits.
|
||||
/// It has a few hundred thousand lines at most, whatever the size of
|
||||
/// the index.
|
||||
#[arg(long, verbatim_doc_comment)]
|
||||
pub shannon_summary: bool,
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
mod alignment_out;
|
||||
mod args;
|
||||
mod shared;
|
||||
mod values;
|
||||
mod iqtree;
|
||||
mod phyg;
|
||||
mod phylip;
|
||||
@@ -21,7 +23,8 @@ use obisys::{Reporter, Stage};
|
||||
use tracing::info;
|
||||
|
||||
use alignment_out::write_alignment_outputs;
|
||||
use args::{DistanceArg, NSitesArg, distance_to_compute};
|
||||
use shared::write_shared_kmers;
|
||||
use values::{DistanceArg, NSitesArg, distance_to_compute};
|
||||
use iqtree::write_iqtree;
|
||||
use phyg::write_sankoff_phyg;
|
||||
use stats::{write_sample_plan, write_sample_stats};
|
||||
@@ -29,7 +32,7 @@ use phylip::write_phylip_relaxed;
|
||||
use sankoff::{write_sankoff_alignment_fasta, write_sankoff_matrix_csv, write_sankoff_params};
|
||||
use tnt::write_sankoff_tnt;
|
||||
|
||||
pub use args::PhyloArgs;
|
||||
pub use args::{PhyloArgs, WORKFLOW_HELP};
|
||||
|
||||
pub fn run(args: PhyloArgs) {
|
||||
let idx = Arc::new(KmerIndex::open(&args.index).unwrap_or_else(|e| {
|
||||
@@ -56,7 +59,7 @@ pub fn run(args: PhyloArgs) {
|
||||
// `cache.distance(...)` computation is unaffected either way.
|
||||
let exclude_mask: Vec<bool> = {
|
||||
let mut mask = vec![false; n];
|
||||
for label in &args.exclude_genome {
|
||||
for label in &args.select.exclude_genome {
|
||||
match labels.iter().position(|l| l == label) {
|
||||
Some(i) => mask[i] = true,
|
||||
None => {
|
||||
@@ -79,7 +82,7 @@ pub fn run(args: PhyloArgs) {
|
||||
// ── Sibling-count/minorant annex (independent of the distance metric) ──
|
||||
// Meant to be (re)built routinely, ahead of any SNP-family distance
|
||||
// computation that will later consume it.
|
||||
if args.sibling_annex {
|
||||
if args.annex.sibling_annex {
|
||||
// Writes into the index directory; `build_sibling_annex` holds the
|
||||
// index's exclusive lock for the duration (`KmerIndex::with_lock`),
|
||||
// so a concurrent run on the same index can't corrupt these writes.
|
||||
@@ -93,7 +96,7 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
|
||||
// ── Sibling-count distribution (`--sibling-stats`) ──────────────────────────
|
||||
if args.sibling_stats {
|
||||
if args.annex.sibling_stats {
|
||||
let t = Stage::start("sibling_stats");
|
||||
let stats = cache.sibling_annex_stats().unwrap_or_else(|e| {
|
||||
eprintln!("error computing sibling-annex stats: {e}");
|
||||
@@ -126,7 +129,7 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
|
||||
// ── Family-size histogram (`--sibling-hist`) ────────────────────────────────
|
||||
if args.sibling_hist {
|
||||
if args.annex.sibling_hist {
|
||||
let t = Stage::start("sibling_hist");
|
||||
let counts = cache.sibling_family_size_histogram().unwrap_or_else(|e| {
|
||||
eprintln!("error computing sibling family-size histogram: {e}");
|
||||
@@ -153,7 +156,7 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
|
||||
// ── Family Overlap matrix (`--family-overlap`) ──────────────────────────────
|
||||
if args.family_overlap {
|
||||
if args.annex.family_overlap {
|
||||
let t = Stage::start("family_overlap");
|
||||
let overlap = cache.family_overlap(args.ambiguity()).unwrap_or_else(|e| {
|
||||
eprintln!("error computing family overlap: {e}");
|
||||
@@ -180,7 +183,7 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
|
||||
// ── Shannon entropy report (`--shannon`) ────────────────────────────────────
|
||||
if args.shannon {
|
||||
if args.annex.shannon {
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_entropy.csv", p.display()))
|
||||
.unwrap_or_else(|| "entropy.csv".into());
|
||||
@@ -195,7 +198,7 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
|
||||
// ── Shannon entropy distribution (`--shannon-summary`) ──────────────────────
|
||||
if args.shannon_summary {
|
||||
if args.annex.shannon_summary {
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_shannon_summary.csv", p.display()))
|
||||
.unwrap_or_else(|| "shannon_summary.csv".into());
|
||||
@@ -215,7 +218,7 @@ pub fn run(args: PhyloArgs) {
|
||||
// `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` only, never the
|
||||
// whole-index metrics' `kept`/`--shared-kmers` output (see
|
||||
// `args::PhyloArgs::min_shared_family`'s own docs on why).
|
||||
let snp_exclude_mask: Vec<bool> = match args.min_shared_family {
|
||||
let snp_exclude_mask: Vec<bool> = match args.select.min_shared_family {
|
||||
Some(threshold) => {
|
||||
let overlap = cache.family_overlap(args.ambiguity()).unwrap_or_else(|e| {
|
||||
eprintln!("error computing family overlap: {e}");
|
||||
@@ -243,19 +246,19 @@ pub fn run(args: PhyloArgs) {
|
||||
// Shared by `--pseudo-alignment` and `--sankoff` — same activation rule:
|
||||
// either flag given activates entropy-biased sampling, the other
|
||||
// defaults to 1.0/0.5.
|
||||
let entropy_bias = if args.entropy.is_some() || args.entropy_sd.is_some() {
|
||||
let entropy_bias = if args.sample.entropy.is_some() || args.sample.entropy_sd.is_some() {
|
||||
Some(EntropyBias {
|
||||
mu: args.entropy.unwrap_or(1.0),
|
||||
sigma: args.entropy_sd.unwrap_or(0.5),
|
||||
mu: args.sample.entropy.unwrap_or(1.0),
|
||||
sigma: args.sample.entropy_sd.unwrap_or(0.5),
|
||||
})
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let sampling = SamplingSpec {
|
||||
free_loss: args.free_loss,
|
||||
free_loss: args.sample.free_loss,
|
||||
ambiguity: args.ambiguity(),
|
||||
quartet: args.quartet,
|
||||
pair_balance: args.pair_balance,
|
||||
quartet: args.sample.quartet,
|
||||
pair_balance: args.sample.pair_balance,
|
||||
entropy_bias,
|
||||
};
|
||||
|
||||
@@ -266,7 +269,7 @@ pub fn run(args: PhyloArgs) {
|
||||
// `algorithms::scan`'s own module docs) — persistence needs to be
|
||||
// rebuilt on top of the new single `sampled_scan` trunk instead of
|
||||
// silently becoming a no-op.
|
||||
if args.session.is_some() {
|
||||
if args.dist.session.is_some() {
|
||||
eprintln!(
|
||||
"error: --session is temporarily unavailable (being rebuilt on top of the unified \
|
||||
sampling trunk) — rerun without --session/--session-force"
|
||||
@@ -284,23 +287,23 @@ pub fn run(args: PhyloArgs) {
|
||||
// instead of each running its own scan (which used to silently draw a
|
||||
// different sample per flag — see `obikphylo::siblings::LayerReducer`'s
|
||||
// own module docs).
|
||||
let sankoff_requested = args.sankoff || args.tnt || args.phyg || args.iqtree;
|
||||
let distance = distance_to_compute(args.distance, args.nj, args.upgma, args.shared_kmers);
|
||||
let sankoff_requested = args.costs.sankoff || args.costs.tnt || args.costs.phyg || args.costs.iqtree;
|
||||
let distance = distance_to_compute(args.dist.distance, args.dist.nj, args.dist.upgma);
|
||||
let snp_kind = distance.and_then(DistanceArg::as_snp);
|
||||
let need_alignment = args.alignment_output() || sankoff_requested;
|
||||
let need_pairwise = sankoff_requested || snp_kind.is_some();
|
||||
// `None` = every site (`--n-sites all`, or `--n-sites` omitted where it is optional)
|
||||
let target = args.n_sites.and_then(NSitesArg::target);
|
||||
let target = args.sample.n_sites.and_then(NSitesArg::target);
|
||||
|
||||
if args.alignment_output() && args.n_sites.is_none() {
|
||||
if args.alignment_output() && args.sample.n_sites.is_none() {
|
||||
eprintln!("error: --pseudo-alignment/--fasta/--vcf require --n-sites <N|all>");
|
||||
std::process::exit(1);
|
||||
}
|
||||
if sankoff_requested && args.n_sites.is_none() {
|
||||
if sankoff_requested && args.sample.n_sites.is_none() {
|
||||
eprintln!("error: --sankoff requires --n-sites <N|all>");
|
||||
std::process::exit(1);
|
||||
}
|
||||
if (args.sample_stats || args.sample_plan) && !(need_alignment || need_pairwise) {
|
||||
if (args.sample.sample_stats || args.sample.sample_plan) && !(need_alignment || need_pairwise) {
|
||||
eprintln!(
|
||||
"error: --sample-stats/--sample-plan describe a draw: they need --pseudo-alignment, \
|
||||
--sankoff or a snp-* --distance"
|
||||
@@ -312,7 +315,7 @@ pub fn run(args: PhyloArgs) {
|
||||
need_alignment.then(|| obikphylo::siblings::AlignmentReducer::new(&cache, &snp_exclude_mask));
|
||||
let mut pairwise_reducer = need_pairwise.then(|| obikphylo::siblings::PairwiseReducer::new(n));
|
||||
let mut stats_reducer =
|
||||
args.sample_stats.then(|| obikphylo::siblings::SampleStatsReducer::new(n, &snp_exclude_mask));
|
||||
args.sample.sample_stats.then(|| obikphylo::siblings::SampleStatsReducer::new(n, &snp_exclude_mask));
|
||||
let mut sample_plan = None;
|
||||
|
||||
if need_alignment || need_pairwise {
|
||||
@@ -349,7 +352,7 @@ pub fn run(args: PhyloArgs) {
|
||||
if let Some(reducer) = stats_reducer {
|
||||
write_sample_stats(&reducer.finish(), &labels, &snp_exclude_mask, &args.output);
|
||||
}
|
||||
if args.sample_plan {
|
||||
if args.sample.sample_plan {
|
||||
write_sample_plan(&sample_plan.expect("scanned above: need_alignment || need_pairwise"), &args.output);
|
||||
}
|
||||
|
||||
@@ -377,13 +380,13 @@ pub fn run(args: PhyloArgs) {
|
||||
let b = obikphylo::siblings::SankoffBundle::finish(
|
||||
alignment,
|
||||
pairwise,
|
||||
args.sankoff_ratio_ceiling,
|
||||
args.costs.sankoff_ratio_ceiling,
|
||||
&snp_exclude_mask,
|
||||
);
|
||||
|
||||
let p_card = cardinality_transition_probs(&b.cardinality_tally);
|
||||
let p_comp = composition_transition_probs(&b.base_pair_tally);
|
||||
let matrix = pairwise_cost_matrix(&p_card, &p_comp, args.free_loss);
|
||||
let matrix = pairwise_cost_matrix(&p_card, &p_comp, args.sample.free_loss);
|
||||
|
||||
write_sankoff_matrix_csv(&matrix, &args.output);
|
||||
write_sankoff_params(
|
||||
@@ -391,43 +394,54 @@ pub fn run(args: PhyloArgs) {
|
||||
&p_card,
|
||||
&b.base_pair_tally,
|
||||
&p_comp,
|
||||
args.sankoff_ratio_ceiling,
|
||||
args.costs.sankoff_ratio_ceiling,
|
||||
&args.output,
|
||||
);
|
||||
write_sankoff_alignment_fasta(&b.alignment, &labels, &args.output, args.free_loss);
|
||||
write_sankoff_alignment_fasta(&b.alignment, &labels, &args.output, args.sample.free_loss);
|
||||
|
||||
if args.tnt {
|
||||
if args.costs.tnt {
|
||||
write_sankoff_tnt(
|
||||
&matrix,
|
||||
&b.alignment,
|
||||
&labels,
|
||||
&args.output,
|
||||
args.sankoff_cost_scale,
|
||||
args.free_loss,
|
||||
args.costs.sankoff_cost_scale,
|
||||
args.sample.free_loss,
|
||||
);
|
||||
}
|
||||
if args.phyg {
|
||||
write_sankoff_phyg(&matrix, &args.output, args.sankoff_cost_scale);
|
||||
if args.costs.phyg {
|
||||
write_sankoff_phyg(&matrix, &args.output, args.costs.sankoff_cost_scale);
|
||||
}
|
||||
if args.iqtree {
|
||||
if args.costs.iqtree {
|
||||
write_iqtree(
|
||||
&matrix,
|
||||
&b.alignment,
|
||||
&labels,
|
||||
&args.output,
|
||||
args.free_loss,
|
||||
args.iqtree_min_freq,
|
||||
args.sample.free_loss,
|
||||
args.costs.iqtree_min_freq,
|
||||
);
|
||||
}
|
||||
bundle = Some(b);
|
||||
}
|
||||
|
||||
// ── Distance computation: classic whole-index metric vs. `snp-*` ───────────
|
||||
// Nothing below (matrix, shared-kmer matrix, NJ/UPGMA trees) is wanted
|
||||
// unless a distance was asked for (`--distance`) or implied
|
||||
// (`--nj`/`--upgma`/`--shared-kmers`): stop here rather than compute and
|
||||
// write a matrix nobody requested.
|
||||
// Nothing below (distance matrix, NJ/UPGMA trees) is wanted unless a
|
||||
// distance was asked for (`--distance`) or implied (`--nj`/`--upgma`): stop
|
||||
// here rather than compute and write a matrix nobody requested.
|
||||
// `--shared-kmers` alone only asks for its own matrix, not for a distance.
|
||||
let Some(distance) = distance else {
|
||||
if args.dist.shared_kmers {
|
||||
info!("computing shared k-mer counts for {n} genome(s)");
|
||||
let t = Stage::start("shared_kmers");
|
||||
let shared = cache.shared_kmers(args.dist.presence_threshold).unwrap_or_else(|e| {
|
||||
eprintln!("error computing shared k-mer counts: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
rep.push(t.stop());
|
||||
let kept: Vec<usize> = (0..n).filter(|&i| !exclude_mask[i]).collect();
|
||||
write_shared_kmers(&shared, &labels, &kept, &args.output);
|
||||
}
|
||||
rep.print();
|
||||
return;
|
||||
};
|
||||
@@ -436,10 +450,10 @@ pub fn run(args: PhyloArgs) {
|
||||
let (matrix, shared_kmers) = match distance.as_classic() {
|
||||
Some(metric) => {
|
||||
info!("computing {metric:?} distances for {n} genome(s)");
|
||||
let need_shared = args.shared_kmers || args.nj || args.upgma;
|
||||
let need_shared = args.dist.shared_kmers;
|
||||
let t = Stage::start("distance");
|
||||
let result = cache
|
||||
.distance(metric, need_shared, args.presence_threshold)
|
||||
.distance(metric, need_shared, args.dist.presence_threshold)
|
||||
.unwrap_or_else(|e| {
|
||||
eprintln!("error computing distances: {e}");
|
||||
std::process::exit(1);
|
||||
@@ -448,12 +462,12 @@ pub fn run(args: PhyloArgs) {
|
||||
(result.matrix, result.shared_kmers)
|
||||
}
|
||||
None => {
|
||||
if args.shared_kmers {
|
||||
if args.dist.shared_kmers {
|
||||
eprintln!("error: --shared-kmers has no meaning for a snp-* --distance value");
|
||||
std::process::exit(1);
|
||||
}
|
||||
let kind = snp_kind.expect("DistanceArg is always classic or snp");
|
||||
let gamma_shape = args.gamma_shape.map_or(GammaShape::Disabled, Into::into);
|
||||
let gamma_shape = args.dist.gamma_shape.map_or(GammaShape::Disabled, Into::into);
|
||||
let t = Stage::start("snp_distance");
|
||||
let matrix = match &bundle {
|
||||
Some(b) => {
|
||||
@@ -485,7 +499,7 @@ pub fn run(args: PhyloArgs) {
|
||||
|
||||
// ── Distance matrix → relaxed PHYLIP (default) or CSV (`--csv`) ────────────
|
||||
let write_dist = |w: &mut dyn Write| {
|
||||
if args.csv {
|
||||
if args.dist.csv {
|
||||
write!(w, "genome").unwrap();
|
||||
for &j in &kept { write!(w, ",{}", labels[j]).unwrap(); }
|
||||
writeln!(w).unwrap();
|
||||
@@ -503,7 +517,7 @@ pub fn run(args: PhyloArgs) {
|
||||
|
||||
match &args.output {
|
||||
Some(prefix) => {
|
||||
let suffix = if args.csv { "_dist.csv" } else { "_dist.phy" };
|
||||
let suffix = if args.dist.csv { "_dist.csv" } else { "_dist.phy" };
|
||||
let path = format!("{}{suffix}", prefix.display());
|
||||
let mut f = BufWriter::new(std::fs::File::create(&path).unwrap_or_else(|e| {
|
||||
eprintln!("error creating {path}: {e}");
|
||||
@@ -520,29 +534,12 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
|
||||
// ── Shared-kmer matrix → CSV ──────────────────────────────────────────────
|
||||
if args.shared_kmers {
|
||||
if let Some(shared) = &shared_kmers {
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_shared.csv", p.display()))
|
||||
.unwrap_or_else(|| "shared.csv".into());
|
||||
let mut f = BufWriter::new(std::fs::File::create(&path).unwrap_or_else(|e| {
|
||||
eprintln!("error creating {path}: {e}");
|
||||
std::process::exit(1);
|
||||
}));
|
||||
write!(f, "genome").unwrap();
|
||||
for &j in &kept { write!(f, ",{}", labels[j]).unwrap(); }
|
||||
writeln!(f).unwrap();
|
||||
for &i in &kept {
|
||||
write!(f, "{}", labels[i]).unwrap();
|
||||
for &j in &kept { write!(f, ",{}", shared[[i, j]]).unwrap(); }
|
||||
writeln!(f).unwrap();
|
||||
}
|
||||
info!("shared-kmer matrix → {path}");
|
||||
}
|
||||
if let Some(shared) = shared_kmers.as_ref().filter(|_| args.dist.shared_kmers) {
|
||||
write_shared_kmers(shared, &labels, &kept, &args.output);
|
||||
}
|
||||
|
||||
// ── NJ tree ────────────────────────────────────────────────────────────────
|
||||
if args.nj {
|
||||
if args.dist.nj {
|
||||
let tree = neighbor_joining(&matrix, &labels).unwrap_or_else(|e| {
|
||||
eprintln!("error computing NJ tree: {e}");
|
||||
std::process::exit(1);
|
||||
@@ -559,7 +556,7 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
|
||||
// ── UPGMA tree ───────────────────────────────────────────────────────────────
|
||||
if args.upgma {
|
||||
if args.dist.upgma {
|
||||
let newick = upgma(&matrix, &labels).to_newick();
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_upgma.nwk", p.display()))
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
//! `--shared-kmers`: the genome-by-genome matrix of shared k-mer counts.
|
||||
|
||||
use std::fs::File;
|
||||
use std::io::{BufWriter, Write};
|
||||
use std::path::PathBuf;
|
||||
|
||||
use ndarray::Array2;
|
||||
use tracing::info;
|
||||
|
||||
/// Write `shared` (rows and columns of the genomes in `kept` only) as CSV to
|
||||
/// `PREFIX_shared.csv`, or `shared.csv` without a prefix.
|
||||
pub(super) fn write_shared_kmers(
|
||||
shared: &Array2<u64>,
|
||||
labels: &[String],
|
||||
kept: &[usize],
|
||||
output: &Option<PathBuf>,
|
||||
) {
|
||||
let path = output
|
||||
.as_ref()
|
||||
.map(|p| format!("{}_shared.csv", p.display()))
|
||||
.unwrap_or_else(|| "shared.csv".into());
|
||||
let mut f = BufWriter::new(File::create(&path).unwrap_or_else(|e| {
|
||||
eprintln!("error creating {path}: {e}");
|
||||
std::process::exit(1);
|
||||
}));
|
||||
write!(f, "genome").unwrap();
|
||||
for &j in kept {
|
||||
write!(f, ",{}", labels[j]).unwrap();
|
||||
}
|
||||
writeln!(f).unwrap();
|
||||
for &i in kept {
|
||||
write!(f, "{}", labels[i]).unwrap();
|
||||
for &j in kept {
|
||||
write!(f, ",{}", shared[[i, j]]).unwrap();
|
||||
}
|
||||
writeln!(f).unwrap();
|
||||
}
|
||||
info!("shared-kmer matrix → {path}");
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
use obikphylo::{DistanceMetric, GammaShape, SnpDistanceKind};
|
||||
|
||||
/// `--gamma-shape` CLI value: a fixed `alpha`, or `auto`/`estimate` to fit
|
||||
/// it from the data (see [`GammaShape::Auto`]). Kept as its own tiny type
|
||||
/// (rather than parsing straight to `GammaShape`) so the CLI layer stays
|
||||
/// free of `GammaShape::Disabled`, which the `Option<GammaShapeArg>` around
|
||||
/// this already expresses on its own (`None` = flag not given at all).
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub enum GammaShapeArg {
|
||||
Fixed(f64),
|
||||
Auto,
|
||||
}
|
||||
|
||||
impl From<GammaShapeArg> for GammaShape {
|
||||
fn from(arg: GammaShapeArg) -> Self {
|
||||
match arg {
|
||||
GammaShapeArg::Fixed(a) => GammaShape::Fixed(a),
|
||||
GammaShapeArg::Auto => GammaShape::Auto,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_gamma_shape(s: &str) -> Result<GammaShapeArg, String> {
|
||||
if s.eq_ignore_ascii_case("auto") || s.eq_ignore_ascii_case("estimate") {
|
||||
return Ok(GammaShapeArg::Auto);
|
||||
}
|
||||
s.parse::<f64>()
|
||||
.map(GammaShapeArg::Fixed)
|
||||
.map_err(|_| format!("invalid --gamma-shape value {s:?}: expected a positive number or \"auto\"/\"estimate\""))
|
||||
}
|
||||
|
||||
/// `--n-sites` CLI value: a number of sites to draw, or `all` for every site
|
||||
/// — the draw then keeps every variable family that passes the other filters,
|
||||
/// with no sampling involved. Its own tiny type (like [`GammaShapeArg`]) so
|
||||
/// the two cases stay distinct up to the call that needs the target, instead
|
||||
/// of an in-band sentinel (a negative number, `0`, ...) that a typo could hit.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum NSitesArg {
|
||||
All,
|
||||
N(usize),
|
||||
}
|
||||
|
||||
impl NSitesArg {
|
||||
/// The target to hand to the sampler: `None` = exhaustive.
|
||||
pub fn target(self) -> Option<usize> {
|
||||
match self {
|
||||
NSitesArg::All => None,
|
||||
NSitesArg::N(n) => Some(n),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_n_sites(s: &str) -> Result<NSitesArg, String> {
|
||||
if s.eq_ignore_ascii_case("all") {
|
||||
return Ok(NSitesArg::All);
|
||||
}
|
||||
s.parse::<usize>()
|
||||
.map(NSitesArg::N)
|
||||
.map_err(|_| format!("invalid --n-sites value {s:?}: expected a non-negative integer or \"all\""))
|
||||
}
|
||||
|
||||
/// `--distance` value — either one of `obikphylo::DistanceMetric`'s
|
||||
/// whole-index metrics (routed to `IndexCache::distance`) or one of
|
||||
/// `obikphylo::SnpDistanceKind`'s `snp-*` corrections (routed to
|
||||
/// `obikphylo::siblings::distance_matrix`, over a tally built by
|
||||
/// `sampled_scan`/`PairwiseReducer`, the sibling-annex pipeline) — two
|
||||
/// genuinely different code paths behind one CLI vocabulary, see
|
||||
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance` unification".
|
||||
#[derive(clap::ValueEnum, Clone, Copy, Debug)]
|
||||
pub enum DistanceArg {
|
||||
Jaccard,
|
||||
Mash,
|
||||
Hamming,
|
||||
BrayCurtis,
|
||||
#[value(name = "relfreq-bray-curtis")]
|
||||
RelfreqBrayCurtis,
|
||||
Euclidean,
|
||||
#[value(name = "relfreq-euclidean")]
|
||||
RelfreqEuclidean,
|
||||
Hellinger,
|
||||
#[value(name = "hellinger-euclidean")]
|
||||
HellingerEuclidean,
|
||||
#[value(name = "snp-raw")]
|
||||
SnpRaw,
|
||||
#[value(name = "snp-jc")]
|
||||
SnpJc,
|
||||
#[value(name = "snp-k2p")]
|
||||
SnpK2p,
|
||||
#[value(name = "snp-k81")]
|
||||
SnpK81,
|
||||
#[value(name = "snp-f81")]
|
||||
SnpF81,
|
||||
#[value(name = "snp-t92")]
|
||||
SnpT92,
|
||||
#[value(name = "snp-tn93")]
|
||||
SnpTn93,
|
||||
#[value(name = "snp-tv")]
|
||||
SnpTv,
|
||||
}
|
||||
|
||||
/// The distance matrix this invocation must compute, if any: the one asked for
|
||||
/// with `--distance`, else `jaccard` when `--nj` or `--upgma` needs a matrix to
|
||||
/// work from, else none — a matrix is never computed (nor written) just
|
||||
/// because nothing said otherwise. `--shared-kmers` asks for its own matrix,
|
||||
/// not for a distance.
|
||||
pub fn distance_to_compute(distance: Option<DistanceArg>, nj: bool, upgma: bool) -> Option<DistanceArg> {
|
||||
distance.or_else(|| (nj || upgma).then_some(DistanceArg::Jaccard))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod distance_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn nothing_asked_nothing_computed() {
|
||||
assert!(distance_to_compute(None, false, false).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tree_outputs_imply_jaccard() {
|
||||
for (nj, upgma) in [(true, false), (false, true)] {
|
||||
assert!(matches!(distance_to_compute(None, nj, upgma), Some(DistanceArg::Jaccard)));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_explicit_distance_wins() {
|
||||
assert!(matches!(
|
||||
distance_to_compute(Some(DistanceArg::Mash), true, true),
|
||||
Some(DistanceArg::Mash)
|
||||
));
|
||||
assert!(matches!(
|
||||
distance_to_compute(Some(DistanceArg::SnpJc), false, false),
|
||||
Some(DistanceArg::SnpJc)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
impl DistanceArg {
|
||||
/// `Some` for the whole-index metrics, `None` for `snp-*` values.
|
||||
pub fn as_classic(self) -> Option<DistanceMetric> {
|
||||
Some(match self {
|
||||
DistanceArg::Jaccard => DistanceMetric::Jaccard,
|
||||
DistanceArg::Mash => DistanceMetric::Mash,
|
||||
DistanceArg::Hamming => DistanceMetric::Hamming,
|
||||
DistanceArg::BrayCurtis => DistanceMetric::BrayCurtis,
|
||||
DistanceArg::RelfreqBrayCurtis => DistanceMetric::RelfreqBrayCurtis,
|
||||
DistanceArg::Euclidean => DistanceMetric::Euclidean,
|
||||
DistanceArg::RelfreqEuclidean => DistanceMetric::RelfreqEuclidean,
|
||||
DistanceArg::Hellinger => DistanceMetric::Hellinger,
|
||||
DistanceArg::HellingerEuclidean => DistanceMetric::HellingerEuclidean,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
/// `Some` for the `snp-*` values, `None` for the whole-index metrics.
|
||||
pub fn as_snp(self) -> Option<SnpDistanceKind> {
|
||||
Some(match self {
|
||||
DistanceArg::SnpRaw => SnpDistanceKind::Raw,
|
||||
DistanceArg::SnpJc => SnpDistanceKind::Jc,
|
||||
DistanceArg::SnpK2p => SnpDistanceKind::K2p,
|
||||
DistanceArg::SnpK81 => SnpDistanceKind::K81,
|
||||
DistanceArg::SnpF81 => SnpDistanceKind::F81,
|
||||
DistanceArg::SnpT92 => SnpDistanceKind::T92,
|
||||
DistanceArg::SnpTn93 => SnpDistanceKind::Tn93,
|
||||
DistanceArg::SnpTv => SnpDistanceKind::Tv,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
}
|
||||
+11
-6
@@ -1,3 +1,8 @@
|
||||
// Static musl builds: replace musl's allocator (see obikmer/Cargo.toml).
|
||||
#[cfg(target_env = "musl")]
|
||||
#[global_allocator]
|
||||
static GLOBAL: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
|
||||
|
||||
mod cli;
|
||||
mod cmd;
|
||||
|
||||
@@ -8,10 +13,11 @@ use tracing_subscriber::{EnvFilter, fmt};
|
||||
#[derive(Parser)]
|
||||
#[command(name = "obikmer2", about = "DNA k-mer tools", version)]
|
||||
struct Cli {
|
||||
/// Hard ceiling on the number of CPU cores the process may use — bounds
|
||||
/// both the command's worker pool and every internal rayon pool.
|
||||
/// Can only lower the budget, never raise it above the cores actually
|
||||
/// available to the process. Defaults to that available count.
|
||||
/// Maximum number of CPU cores to use
|
||||
///
|
||||
/// Caps every thread pool of the process. It can only lower the number of
|
||||
/// cores, never raise it above those available. By default all available
|
||||
/// cores may be used.
|
||||
#[arg(long, global = true, value_name = "N")]
|
||||
cpu_max: Option<usize>,
|
||||
|
||||
@@ -47,8 +53,7 @@ enum Commands {
|
||||
Utils(cmd::utils::UtilsArgs),
|
||||
/// Convert an index's evidence representation (exact/approximate/hybrid), in place
|
||||
Convert(cmd::convert::ConvertArgs),
|
||||
/// Genome-vs-genome distance matrix (+ optional NJ/UPGMA tree, sibling-annex-based
|
||||
/// SNP corrections, Sankoff/TNT/PhyG/IQ-TREE exports)
|
||||
#[command(after_long_help = cmd::phylo::WORKFLOW_HELP)]
|
||||
Phylo(cmd::phylo::PhyloArgs),
|
||||
}
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ use std::fs;
|
||||
use std::io;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use obicompactvec::{PersistentBitMatrix, PersistentIntMatrix, StorageKind};
|
||||
use obikalgorithm::Algorithm;
|
||||
use obikindex::layer::IndexMode;
|
||||
use obikindex::partition::PARTITIONS_SUBDIR;
|
||||
@@ -176,7 +177,17 @@ impl Algorithm for Merge<'_> {
|
||||
);
|
||||
}
|
||||
|
||||
let base_idx = choose_base(sources, &src_genomes, self.mode);
|
||||
let Some(base_idx) = choose_base(sources, &src_genomes, self.mode) else {
|
||||
return Err(
|
||||
"merge: none of the sources can be the destination. Every candidate has its \
|
||||
matrices packed (finalized index): a packed index can be merged in as a \
|
||||
source, but at least one source must be unpacked — typically a freshly \
|
||||
built mono-genome index. To extend a packed index, merge it with new \
|
||||
mono-genome indexes; to combine several packed indexes, merge their \
|
||||
original sources instead."
|
||||
.into(),
|
||||
);
|
||||
};
|
||||
let needs_approx = sources.iter().enumerate().any(|(i, src)| {
|
||||
!is_trivial(&src_genomes[i], self.mode)
|
||||
&& matches!(
|
||||
@@ -490,7 +501,15 @@ fn index_unitig_size(src: &KmerIndex) -> u64 {
|
||||
(0..n).map(|i| partition_unitig_bytes(src, i)).sum()
|
||||
}
|
||||
|
||||
fn choose_base(sources: &[&KmerIndex], src_genomes: &[Vec<GenomeInfo>], mode: MergeMode) -> usize {
|
||||
/// The source whose partition data is copied as the merge destination: the
|
||||
/// largest one (least data to merge in). `None` when several sources are
|
||||
/// given and none of them can be resumed as a destination (see
|
||||
/// [`can_be_destination`]).
|
||||
fn choose_base(
|
||||
sources: &[&KmerIndex],
|
||||
src_genomes: &[Vec<GenomeInfo>],
|
||||
mode: MergeMode,
|
||||
) -> Option<usize> {
|
||||
let needs_approx = sources.iter().enumerate().any(|(i, src)| {
|
||||
!is_trivial(&src_genomes[i], mode)
|
||||
&& matches!(
|
||||
@@ -509,9 +528,42 @@ fn choose_base(sources: &[&KmerIndex], src_genomes: &[Vec<GenomeInfo>], mode: Me
|
||||
IndexMode::Approx { .. } | IndexMode::Hybrid { .. }
|
||||
)
|
||||
})
|
||||
// With a single source there is nothing to append: the bootstrap
|
||||
// copy is the whole merge, whatever its matrix format.
|
||||
.filter(|(i, src)| sources.len() == 1 || can_be_destination(src, &src_genomes[*i], mode))
|
||||
.max_by_key(|(_, src)| index_unitig_size(src))
|
||||
.map(|(i, _)| i)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
/// Whether `index` can be the destination of a merge: pass 2 appends the
|
||||
/// other sources' columns to its matrices (`MatrixBuilder::resume`), which
|
||||
/// needs them in the column-by-column format. A finalized index has its
|
||||
/// matrices packed (sparse by default, or dense with `--dense`) into
|
||||
/// single-file formats that cannot be extended — such an index can only
|
||||
/// come in as a *source*. An implicit matrix (mono-genome presence) is
|
||||
/// materialized on the fly, so it qualifies only for a single genome.
|
||||
///
|
||||
/// One layer is probed: `pack` rewrites every partition the same way. An
|
||||
/// index with no k-mer at all has nothing to resume and qualifies.
|
||||
fn can_be_destination(index: &KmerIndex, genomes: &[GenomeInfo], mode: MergeMode) -> bool {
|
||||
let layer_dir = (0..index.n_partitions()).find_map(|i| {
|
||||
let partition = index.partition(i).ok()?;
|
||||
(partition.n_layers() > 0)
|
||||
.then(|| partition.layer_dir(0).ok())
|
||||
.flatten()
|
||||
});
|
||||
let Some(layer_dir) = layer_dir else {
|
||||
return true;
|
||||
};
|
||||
let kind = match mode {
|
||||
MergeMode::Presence => PersistentBitMatrix::detect_storage(&layer_dir),
|
||||
MergeMode::Count => PersistentIntMatrix::detect_storage(&layer_dir),
|
||||
};
|
||||
match kind {
|
||||
Ok(StorageKind::Columnar) => true,
|
||||
Ok(StorageKind::Implicit) => genomes.len() == 1,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Copies `src` (the base source's whole `partitions/` tree) into `dst` at
|
||||
|
||||
@@ -10,7 +10,6 @@ obiskio = { path = "../obiskio" }
|
||||
obisys = { path = "../obisys", default-features = false }
|
||||
obicompactvec = { path = "../obicompactvec" }
|
||||
obikidxcache = { path = "../obikidxcache" }
|
||||
obiskbuilder = { path = "../obiskbuilder" }
|
||||
obipipeline = { path = "../obipipeline" }
|
||||
kodama = "0.3.0"
|
||||
memmap2 = "0.9"
|
||||
|
||||
@@ -51,20 +51,6 @@ pub struct DistanceOutput {
|
||||
pub shared_kmers: Option<Array2<u64>>,
|
||||
}
|
||||
|
||||
impl DistanceMetric {
|
||||
pub fn requires_counts(self) -> bool {
|
||||
matches!(
|
||||
self,
|
||||
DistanceMetric::BrayCurtis
|
||||
| DistanceMetric::RelfreqBrayCurtis
|
||||
| DistanceMetric::Euclidean
|
||||
| DistanceMetric::RelfreqEuclidean
|
||||
| DistanceMetric::Hellinger
|
||||
| DistanceMetric::HellingerEuclidean
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Aggregators — bounded parallel reduction of every cached layer's own
|
||||
// partial (of the wanted content kind) into one global partial, then handed
|
||||
// off to `CountPartials`/`BitPartials`'s own provided finalisation methods.
|
||||
@@ -188,10 +174,33 @@ impl BitPartials for AggregatedBit<'_> {
|
||||
/// extension trait in this codebase.
|
||||
pub trait Metrics {
|
||||
fn distance(&self, metric: DistanceMetric, shared_kmers: bool, presence_threshold: u32) -> OKIResult<DistanceOutput>;
|
||||
|
||||
/// Genome-by-genome matrix of the number of k-mers shared by each pair of
|
||||
/// genomes — on a count index a k-mer is present when its count reaches
|
||||
/// `presence_threshold`. Independent of any distance: nothing else is
|
||||
/// computed.
|
||||
fn shared_kmers(&self, presence_threshold: u32) -> OKIResult<Array2<u64>>;
|
||||
}
|
||||
|
||||
impl Metrics for IndexCache {
|
||||
fn shared_kmers(&self, presence_threshold: u32) -> OKIResult<Array2<u64>> {
|
||||
let n_genomes = self.meta().genomes().len();
|
||||
if n_genomes < 2 {
|
||||
return Err(OKIError::InvalidInput(
|
||||
"shared k-mer counts require at least 2 genomes in the index".into(),
|
||||
));
|
||||
}
|
||||
let (inter, _union) = if self.meta().config().with_counts {
|
||||
AggregatedCount { cache: self, n_genomes }.partial_threshold_jaccard(presence_threshold)
|
||||
} else {
|
||||
AggregatedBit { cache: self, n_genomes }.partial_jaccard()
|
||||
};
|
||||
Ok(inter)
|
||||
}
|
||||
|
||||
fn distance(&self, metric: DistanceMetric, shared_kmers: bool, presence_threshold: u32) -> OKIResult<DistanceOutput> {
|
||||
use DistanceMetric::*;
|
||||
|
||||
let n_genomes = self.meta().genomes().len();
|
||||
if n_genomes < 2 {
|
||||
return Err(OKIError::InvalidInput(
|
||||
@@ -199,64 +208,40 @@ impl Metrics for IndexCache {
|
||||
));
|
||||
}
|
||||
|
||||
let use_counts = self.meta().config().with_counts;
|
||||
if metric.requires_counts() && !use_counts {
|
||||
return Err(OKIError::InvalidInput(format!(
|
||||
"{metric:?} requires a count index (with_counts = true)"
|
||||
)));
|
||||
}
|
||||
|
||||
let kmer_size = self.meta().config().kmer_size;
|
||||
let use_counts = self.meta().config().with_counts;
|
||||
let count = AggregatedCount { cache: self, n_genomes };
|
||||
let bit = AggregatedBit { cache: self, n_genomes };
|
||||
|
||||
if use_counts {
|
||||
let agg = AggregatedCount { cache: self, n_genomes };
|
||||
// One construction decides both what the index can serve and how:
|
||||
// the six count-based distances need a count index; Jaccard, Mash and
|
||||
// Hamming work on either (on counts, through the presence threshold).
|
||||
let matrix = match (use_counts, metric) {
|
||||
(true, BrayCurtis) => count.bray_dist_matrix(),
|
||||
(true, RelfreqBrayCurtis) => count.relfreq_bray_dist_matrix(),
|
||||
(true, Euclidean) => count.euclidean_dist_matrix(),
|
||||
(true, RelfreqEuclidean) => count.relfreq_euclidean_dist_matrix(),
|
||||
(true, Hellinger) => count.hellinger_dist_matrix(),
|
||||
(true, HellingerEuclidean) => count.hellinger_euclidean_dist_matrix(),
|
||||
(true, Jaccard) => count.threshold_jaccard_dist_matrix(presence_threshold),
|
||||
(true, Mash) => count.threshold_mash_dist_matrix(kmer_size, presence_threshold),
|
||||
(true, Hamming) => count.threshold_hamming_dist_matrix(presence_threshold).mapv(|v| v as f64),
|
||||
(false, Jaccard) => bit.jaccard_dist_matrix(),
|
||||
(false, Mash) => bit.mash_dist_matrix(kmer_size),
|
||||
(false, Hamming) => bit.hamming_dist_matrix().mapv(|v| v as f64),
|
||||
(
|
||||
false,
|
||||
BrayCurtis | RelfreqBrayCurtis | Euclidean | RelfreqEuclidean | Hellinger | HellingerEuclidean,
|
||||
) => {
|
||||
return Err(OKIError::InvalidInput(format!(
|
||||
"{metric:?} requires a count index (with_counts = true); \
|
||||
on an index without counts use --distance jaccard, mash or hamming"
|
||||
)));
|
||||
}
|
||||
};
|
||||
|
||||
let matrix = match metric {
|
||||
DistanceMetric::BrayCurtis => agg.bray_dist_matrix(),
|
||||
DistanceMetric::RelfreqBrayCurtis => agg.relfreq_bray_dist_matrix(),
|
||||
DistanceMetric::Euclidean => agg.euclidean_dist_matrix(),
|
||||
DistanceMetric::RelfreqEuclidean => agg.relfreq_euclidean_dist_matrix(),
|
||||
DistanceMetric::Hellinger => agg.hellinger_dist_matrix(),
|
||||
DistanceMetric::HellingerEuclidean => agg.hellinger_euclidean_dist_matrix(),
|
||||
DistanceMetric::Jaccard => agg.threshold_jaccard_dist_matrix(presence_threshold),
|
||||
DistanceMetric::Mash => agg.threshold_mash_dist_matrix(kmer_size, presence_threshold),
|
||||
DistanceMetric::Hamming => {
|
||||
return Err(OKIError::InvalidInput(
|
||||
"Hamming is only available for presence/absence indexes".into(),
|
||||
));
|
||||
}
|
||||
};
|
||||
let shared = shared_kmers.then(|| self.shared_kmers(presence_threshold)).transpose()?;
|
||||
|
||||
let shared = if shared_kmers {
|
||||
let (inter, _) = agg.partial_threshold_jaccard(presence_threshold);
|
||||
Some(inter)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
Ok(DistanceOutput { matrix, shared_kmers: shared })
|
||||
} else {
|
||||
let agg = AggregatedBit { cache: self, n_genomes };
|
||||
|
||||
let matrix = match metric {
|
||||
DistanceMetric::Jaccard => agg.jaccard_dist_matrix(),
|
||||
DistanceMetric::Mash => agg.mash_dist_matrix(kmer_size),
|
||||
DistanceMetric::Hamming => agg.hamming_dist_matrix().mapv(|v| v as f64),
|
||||
other => {
|
||||
return Err(OKIError::InvalidInput(format!(
|
||||
"{other:?} requires a count index; use --metric jaccard or --metric hamming"
|
||||
)));
|
||||
}
|
||||
};
|
||||
|
||||
let shared = if shared_kmers {
|
||||
let (inter, _) = agg.partial_jaccard();
|
||||
Some(inter)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
Ok(DistanceOutput { matrix, shared_kmers: shared })
|
||||
}
|
||||
Ok(DistanceOutput { matrix, shared_kmers: shared })
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::io;
|
||||
use std::path::PathBuf;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use obicompactvec::PersistentBitVecBuilder;
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
@@ -22,19 +22,20 @@ use super::masking::is_quartet_informative;
|
||||
use super::minorant_selection::non_monomorphic_reindex_layer;
|
||||
use super::pair_weight::pair_balance_weight;
|
||||
use crate::siblings::AmbiguityPolicy;
|
||||
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
||||
use crate::siblings::quartet_annex::{QuartetConvention, quartet_annex_file_name};
|
||||
use crate::siblings::{
|
||||
FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind, family_overlap_file_name,
|
||||
};
|
||||
|
||||
/// Which annexes a caller needs, each listed once per ambiguity policy (how
|
||||
/// genomes carrying several members of a family are counted) — an empty list
|
||||
/// asks for none. A pair weight is computed from the family-overlap annex of
|
||||
/// asks for none (the quartet annex also depends on `--free-loss`, so its
|
||||
/// convention is a [`QuartetConvention`]). A pair weight is computed from the family-overlap annex of
|
||||
/// its own policy, so asking for one asks for the other.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub(crate) struct AnnexRequest {
|
||||
pub entropy: Vec<AmbiguityPolicy>,
|
||||
pub quartet: Vec<AmbiguityPolicy>,
|
||||
pub quartet: Vec<QuartetConvention>,
|
||||
pub overlap: Vec<AmbiguityPolicy>,
|
||||
pub pair_weight: Vec<AmbiguityPolicy>,
|
||||
}
|
||||
@@ -46,6 +47,9 @@ pub(crate) struct AnnexRequest {
|
||||
#[derive(Clone, Copy, Debug, Default)]
|
||||
pub struct AnnexSelection {
|
||||
pub policy: AmbiguityPolicy,
|
||||
/// `--free-loss`: part of the quartet annex's convention (whether absence
|
||||
/// counts as a state), irrelevant to the other annexes.
|
||||
pub free_loss: bool,
|
||||
pub entropy: bool,
|
||||
pub quartet: bool,
|
||||
pub overlap: bool,
|
||||
@@ -57,7 +61,11 @@ impl AnnexSelection {
|
||||
let one = |wanted: bool| if wanted { vec![self.policy] } else { Vec::new() };
|
||||
AnnexRequest {
|
||||
entropy: one(self.entropy),
|
||||
quartet: one(self.quartet),
|
||||
quartet: if self.quartet {
|
||||
vec![QuartetConvention { policy: self.policy, free_loss: self.free_loss }]
|
||||
} else {
|
||||
Vec::new()
|
||||
},
|
||||
overlap: one(self.overlap),
|
||||
pair_weight: one(self.pair_weight),
|
||||
}
|
||||
@@ -100,8 +108,22 @@ impl AnnexRequest {
|
||||
}
|
||||
}
|
||||
|
||||
fn dedup(variants: impl IntoIterator<Item = AmbiguityPolicy>) -> Vec<AmbiguityPolicy> {
|
||||
let mut v: Vec<AmbiguityPolicy> = variants.into_iter().collect();
|
||||
/// The annexes among `conventions` (each once) whose file does not exist yet
|
||||
/// in `dir`, with their path.
|
||||
fn missing_files<T: Ord + Copy>(
|
||||
dir: &Path,
|
||||
name: impl Fn(T) -> &'static str,
|
||||
conventions: &[T],
|
||||
) -> Vec<(T, PathBuf)> {
|
||||
dedup(conventions.iter().copied())
|
||||
.into_iter()
|
||||
.map(|convention| (convention, dir.join(name(convention))))
|
||||
.filter(|(_, path)| !path.exists())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn dedup<T: Ord>(variants: impl IntoIterator<Item = T>) -> Vec<T> {
|
||||
let mut v: Vec<T> = variants.into_iter().collect();
|
||||
v.sort_unstable();
|
||||
v.dedup();
|
||||
v
|
||||
@@ -131,17 +153,9 @@ pub(crate) fn ensure_layer_family_annexes(
|
||||
let layer = cache
|
||||
.get_layer(partition, layer_idx)
|
||||
.expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)");
|
||||
let missing = |name: &dyn Fn(AmbiguityPolicy) -> &'static str,
|
||||
variants: &[AmbiguityPolicy]|
|
||||
-> Vec<(AmbiguityPolicy, PathBuf)> {
|
||||
dedup(variants.iter().copied())
|
||||
.into_iter()
|
||||
.map(|policy| (policy, layer.dir().join(name(policy))))
|
||||
.filter(|(_, path)| !path.exists())
|
||||
.collect()
|
||||
};
|
||||
let missing_entropy = missing(&|p| ScalarKind::Entropy.file_name(p), &request.entropy);
|
||||
let missing_quartet = missing(&quartet_annex_file_name, &request.quartet);
|
||||
let missing_entropy =
|
||||
missing_files(layer.dir(), |p| ScalarKind::Entropy.file_name(p), &request.entropy);
|
||||
let missing_quartet = missing_files(layer.dir(), quartet_annex_file_name, &request.quartet);
|
||||
if missing_entropy.is_empty() && missing_quartet.is_empty() && overlap_accs.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
@@ -182,8 +196,11 @@ pub(crate) fn ensure_layer_family_annexes(
|
||||
builder.set(compact, h as f32);
|
||||
}
|
||||
}
|
||||
for (policy, builder) in quartet_builders.iter_mut() {
|
||||
builder.set(compact, is_quartet_informative(genome_mask, *policy));
|
||||
for (convention, builder) in quartet_builders.iter_mut() {
|
||||
builder.set(
|
||||
compact,
|
||||
is_quartet_informative(genome_mask, convention.policy, convention.free_loss),
|
||||
);
|
||||
}
|
||||
for (_, acc) in overlap_accs.iter_mut() {
|
||||
acc.add_family(genome_mask);
|
||||
|
||||
@@ -18,7 +18,6 @@
|
||||
//! data is dropped.
|
||||
|
||||
use crate::siblings::AmbiguityPolicy;
|
||||
use crate::siblings::helpers::is_carrier;
|
||||
|
||||
/// This genome's masked character state, or `None` for "?" (missing). `m`
|
||||
/// is one `genome_mask` entry (bit `b` set iff this genome carries the
|
||||
@@ -69,39 +68,43 @@ pub(crate) fn survives_masking(masked: &[Option<u8>], excluded: &[bool]) -> bool
|
||||
}
|
||||
|
||||
/// Whether `counts` — the number of genomes per *called* state, indexed by
|
||||
/// the `0..=15` state value, index `0` (absence) never counted — shows a
|
||||
/// bipartition: at least two distinct states each carried by at least two
|
||||
/// genomes. That is the pattern that can separate two pairs of genomes from
|
||||
/// each other (`AB|CD`); a singleton variant, or a family carried by fewer
|
||||
/// than four genomes, fits every topology and informs none.
|
||||
/// the `0..=15` state value — shows a bipartition: at least two distinct
|
||||
/// states each carried by at least two genomes. That is the pattern that can
|
||||
/// separate two pairs of genomes from each other (`AB|CD`); a singleton
|
||||
/// variant, or a family carried by fewer than four genomes, fits every
|
||||
/// topology and informs none.
|
||||
///
|
||||
/// Index `0` is absence (`∅`). It holds genomes only when absence is a real
|
||||
/// state of the alignment (no `--free-loss`), and is then a state like any
|
||||
/// other; under `--free-loss` absent genomes are missing data, are never
|
||||
/// counted, and the index stays empty.
|
||||
pub(crate) fn has_bipartition(counts: &[u32; 16]) -> bool {
|
||||
counts[1..].iter().filter(|&&c| c >= 2).count() >= 2
|
||||
counts.iter().filter(|&&c| c >= 2).count() >= 2
|
||||
}
|
||||
|
||||
/// Quartet-informativeness of a family from its raw per-genome masks alone,
|
||||
/// under the `--no-ambiguity` counting convention: absent genomes (`0`)
|
||||
/// never count, and with `no_ambiguity` a genome carrying several members
|
||||
/// doesn't either — otherwise its combination of bases is a state of its
|
||||
/// own. Ignores `--exclude-genome` and `--free-loss` on purpose: absence is
|
||||
/// never a called state here, and removing genomes can only destroy a
|
||||
/// bipartition, never create one — so `false` is final, and `true` is only
|
||||
/// a necessary condition, re-checked exactly by [`survives_quartet`] once
|
||||
/// the family is resolved. This is what the persisted quartet annex stores.
|
||||
pub(crate) fn is_quartet_informative(genome_mask: &[u8], policy: AmbiguityPolicy) -> bool {
|
||||
/// under one counting convention: each genome is read through
|
||||
/// [`masked_state`] exactly as in the alignment — absence is a state of its
|
||||
/// own unless `free_loss`, and `policy` makes the genomes carrying too many
|
||||
/// members missing. Ignores `--exclude-genome` on purpose: removing genomes
|
||||
/// can only destroy a bipartition, never create one — so `false` is final, and
|
||||
/// `true` is only a necessary condition, re-checked exactly by
|
||||
/// [`survives_quartet`] once the family is resolved. This is what the
|
||||
/// persisted quartet annex stores.
|
||||
pub(crate) fn is_quartet_informative(genome_mask: &[u8], policy: AmbiguityPolicy, free_loss: bool) -> bool {
|
||||
let mut counts = [0u32; 16];
|
||||
for &m in genome_mask {
|
||||
if !is_carrier(m, policy) {
|
||||
continue;
|
||||
if let Some(state) = masked_state(m, free_loss, policy) {
|
||||
counts[(state & 0b1111) as usize] += 1;
|
||||
}
|
||||
counts[(m & 0b1111) as usize] += 1;
|
||||
}
|
||||
has_bipartition(&counts)
|
||||
}
|
||||
|
||||
/// Exact quartet check on an already-[`masked_state`]d family, over the
|
||||
/// *included* genomes (`excluded` as in [`survives_masking`]). Only called
|
||||
/// states count: a genome whose masked state is `0` (absence kept as a real
|
||||
/// state, i.e. no `--free-loss`) is not a carrier of anything.
|
||||
/// *included* genomes (`excluded` as in [`survives_masking`]). Missing
|
||||
/// genomes (`None`) count for nothing; absence kept as a real state (no
|
||||
/// `--free-loss`, masked state `0`) is a state like any other.
|
||||
pub(crate) fn survives_quartet(masked: &[Option<u8>], excluded: &[bool]) -> bool {
|
||||
let mut counts = [0u32; 16];
|
||||
for (g, &state) in masked.iter().enumerate() {
|
||||
@@ -119,33 +122,48 @@ pub(crate) fn survives_quartet(masked: &[Option<u8>], excluded: &[bool]) -> bool
|
||||
mod quartet_tests {
|
||||
use super::*;
|
||||
|
||||
const KEEP: AmbiguityPolicy = AmbiguityPolicy::Keep;
|
||||
const FORBID: AmbiguityPolicy = AmbiguityPolicy::Forbid;
|
||||
|
||||
#[test]
|
||||
fn two_pairs_of_distinct_states_is_informative() {
|
||||
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], AmbiguityPolicy::Keep));
|
||||
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], AmbiguityPolicy::Forbid));
|
||||
for free_loss in [true, false] {
|
||||
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], KEEP, free_loss));
|
||||
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], FORBID, free_loss));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn singleton_variant_is_not_informative() {
|
||||
// three genomes in one state plus one lone variant: fits any topology
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0001, 0b0100], AmbiguityPolicy::Keep));
|
||||
// fewer than four carriers can never be informative
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0100, 0, 0], AmbiguityPolicy::Keep));
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0100], AmbiguityPolicy::Keep));
|
||||
for free_loss in [true, false] {
|
||||
// three genomes in one state plus one lone variant: fits any topology
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0001, 0b0100], KEEP, free_loss));
|
||||
// fewer than four genomes with a state can never be informative
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0100, 0, 0], KEEP, free_loss));
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0100], KEEP, free_loss));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absent_genomes_are_not_a_state() {
|
||||
// two carriers of one base, two absences: absence must not form the 2nd pair
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0, 0], AmbiguityPolicy::Keep));
|
||||
fn under_free_loss_absence_is_missing_and_forms_no_pair() {
|
||||
// two carriers of one base, two absences: absence is missing data
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0, 0], KEEP, true));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn without_free_loss_absence_is_a_state_that_can_form_a_pair() {
|
||||
// A A ∅ ∅ separates two pairs of genomes once ∅ is a character state
|
||||
assert!(is_quartet_informative(&[0b0001, 0b0001, 0, 0], KEEP, false));
|
||||
// ... but a single absent genome is a lone state, like any singleton
|
||||
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0001, 0], KEEP, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_ambiguity_drops_multi_member_genomes_from_the_count() {
|
||||
// the two-member genomes form a pair of their own state without the flag
|
||||
let masks = [0b0001u8, 0b0001, 0b0011, 0b0011];
|
||||
assert!(is_quartet_informative(&masks, AmbiguityPolicy::Keep));
|
||||
assert!(!is_quartet_informative(&masks, AmbiguityPolicy::Forbid));
|
||||
assert!(is_quartet_informative(&masks, KEEP, true));
|
||||
assert!(!is_quartet_informative(&masks, FORBID, true));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -154,8 +172,19 @@ mod quartet_tests {
|
||||
assert!(survives_quartet(&masked, &[]));
|
||||
// excluding one genome of the second pair breaks the bipartition
|
||||
assert!(!survives_quartet(&masked, &[false, false, true]));
|
||||
// an absence kept as state 0 (no --free-loss) is not a called state
|
||||
assert!(!survives_quartet(&[Some(1), Some(1), Some(0), Some(0)], &[]));
|
||||
// an absence kept as state 0 (no --free-loss) is a state: ∅ ∅ is a pair
|
||||
assert!(survives_quartet(&[Some(1), Some(1), Some(0), Some(0)], &[]));
|
||||
// the same genomes under --free-loss are missing: no second pair
|
||||
assert!(!survives_quartet(&[Some(1), Some(1), None, None], &[]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_annex_rule_and_the_exact_check_agree_on_the_same_family() {
|
||||
let raw = [0b0001u8, 0b0001, 0, 0];
|
||||
for free_loss in [true, false] {
|
||||
let masked: Vec<_> = raw.iter().map(|&m| masked_state(m, free_loss, KEEP)).collect();
|
||||
assert_eq!(is_quartet_informative(&raw, KEEP, free_loss), survives_quartet(&masked, &[]));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -7,12 +7,12 @@ use obicompactvec::{PersistentBitVec, TempBitVecBuilder};
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
|
||||
use super::minorant_selection::is_non_monomorphic_minorant;
|
||||
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
||||
use crate::siblings::{AmbiguityPolicy, SiblingAnnex};
|
||||
use crate::siblings::quartet_annex::{QuartetConvention, quartet_annex_file_name};
|
||||
use crate::siblings::SiblingAnnex;
|
||||
|
||||
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
|
||||
/// included), every family the quartet annex flags as not quartet-informative
|
||||
/// under the `policy` counting convention. The annex is indexed by the
|
||||
/// under the `convention` counting convention. The annex is indexed by the
|
||||
/// compact non-monomorphic ordinal, so the sibling annex is walked in the same
|
||||
/// order to translate one numbering into the other — annex-only, no
|
||||
/// cross-partition resolution. The annex must already exist
|
||||
@@ -20,10 +20,10 @@ use crate::siblings::{AmbiguityPolicy, SiblingAnnex};
|
||||
pub(crate) fn restrict_to_quartet(
|
||||
eligible: &mut TempBitVecBuilder,
|
||||
layer_dir: &Path,
|
||||
policy: AmbiguityPolicy,
|
||||
convention: QuartetConvention,
|
||||
) -> OKIResult<()> {
|
||||
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(policy)))
|
||||
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(convention)))
|
||||
.map_err(OKIError::Io)?;
|
||||
|
||||
let mut family_idx = 0usize;
|
||||
|
||||
@@ -84,8 +84,12 @@ impl SampleStatsReducer {
|
||||
let mut counts = [0u32; 16];
|
||||
let mut carriers: Vec<usize> = Vec::new();
|
||||
for (g, &state) in masked.iter().enumerate() {
|
||||
if let Some(state) = state.filter(|&s| s != 0 && !self.excluded[g]) {
|
||||
counts[(state & 0b1111) as usize] += 1;
|
||||
let Some(state) = state.filter(|_| !self.excluded[g]) else { continue };
|
||||
// Absence (state 0, only present without `--free-loss`) is a state
|
||||
// of the alignment, so it takes part in the bipartition — but its
|
||||
// genome carries none of the family's members.
|
||||
counts[(state & 0b1111) as usize] += 1;
|
||||
if state != 0 {
|
||||
carriers.push(g);
|
||||
}
|
||||
}
|
||||
@@ -140,6 +144,19 @@ mod tests {
|
||||
assert_eq!(s.overlap(3, 0), s.overlap(0, 3));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absence_as_a_state_makes_a_site_informative() {
|
||||
let mut r = SampleStatsReducer::new(4, &[]);
|
||||
// A A ∅ ∅ (no --free-loss): two pairs of states, though only two carriers
|
||||
r.add_site(&[Some(1), Some(1), Some(0), Some(0)]);
|
||||
// the same genomes under --free-loss: the absences are missing
|
||||
r.add_site(&[Some(1), Some(1), None, None]);
|
||||
let s = r.finish();
|
||||
assert_eq!(s.sites_by_carriers[2], 2);
|
||||
assert_eq!(s.informative_by_carriers[2], 1);
|
||||
assert_eq!(s.calls, vec![2, 2, 0, 0]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn excluded_genomes_count_nowhere() {
|
||||
let mut r = SampleStatsReducer::new(4, &[false, false, false, true]);
|
||||
|
||||
@@ -57,7 +57,7 @@ use super::family_scan::{Selection, scan_layer_families};
|
||||
use super::masking::{masked_state, survives_masking, survives_quartet};
|
||||
use super::minorant_selection::{eligible_bitset_layer, iter_full_scalar};
|
||||
use super::{AnnexRequest, is_fast_mode, restrict_to_quartet};
|
||||
use crate::siblings::{AmbiguityPolicy, FamilyMask, ScalarKind};
|
||||
use crate::siblings::{AmbiguityPolicy, FamilyMask, QuartetConvention, ScalarKind};
|
||||
use crate::siblings::extensions::SiblingBuilder;
|
||||
|
||||
/// Resolution rounds a single layer's sampling may take before giving up on
|
||||
@@ -122,6 +122,15 @@ pub struct SamplingSpec {
|
||||
pub entropy_bias: Option<EntropyBias>,
|
||||
}
|
||||
|
||||
impl SamplingSpec {
|
||||
/// The counting convention of the quartet annex this draw reads: how
|
||||
/// genomes with several members count, and whether absence is a state —
|
||||
/// the same two choices the post-hoc masking makes.
|
||||
pub(crate) fn quartet_convention(&self) -> QuartetConvention {
|
||||
QuartetConvention { policy: self.ambiguity, free_loss: self.free_loss }
|
||||
}
|
||||
}
|
||||
|
||||
/// What one layer contributed to a draw — one row of `--sample-plan`.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct LayerReport {
|
||||
@@ -260,7 +269,7 @@ pub(crate) fn sample_index(
|
||||
// convention) — build any missing one now, once, rather than mid-walk.
|
||||
let request = AnnexRequest {
|
||||
entropy: spec.entropy_bias.map(|_| vec![spec.ambiguity]).unwrap_or_default(),
|
||||
quartet: if spec.quartet { vec![spec.ambiguity] } else { Vec::new() },
|
||||
quartet: if spec.quartet { vec![spec.quartet_convention()] } else { Vec::new() },
|
||||
overlap: Vec::new(),
|
||||
pair_weight: if spec.pair_balance { vec![spec.ambiguity] } else { Vec::new() },
|
||||
};
|
||||
@@ -282,7 +291,7 @@ pub(crate) fn sample_index(
|
||||
.expect("cache-consistent: n_layer(partition) already bounds layer_idx");
|
||||
let mut eligible = eligible_bitset_layer(layer.dir())?;
|
||||
if spec.quartet {
|
||||
restrict_to_quartet(&mut eligible, layer.dir(), spec.ambiguity)?;
|
||||
restrict_to_quartet(&mut eligible, layer.dir(), spec.quartet_convention())?;
|
||||
}
|
||||
let n_minorants = eligible.view().len();
|
||||
let eligible_count = eligible.view().count_ones();
|
||||
|
||||
@@ -40,8 +40,9 @@ pub(super) fn triangle_index(n: usize, i: usize, j: usize) -> usize {
|
||||
/// the `--no-ambiguity` counting convention: any non-empty state does, except
|
||||
/// — with `no_ambiguity` — a genome carrying several members, which is
|
||||
/// missing data in the alignment and so shares nothing there. One definition
|
||||
/// for every annex that counts carriers (family overlap, pair weight,
|
||||
/// quartet), so they can never disagree.
|
||||
/// for every annex that counts carriers (family overlap, pair weight), so they
|
||||
/// can never disagree. (The quartet annex does not use it: absence can be a
|
||||
/// state there, so it reads genomes through `masking::masked_state`.)
|
||||
#[inline]
|
||||
pub(crate) fn is_carrier(mask: u8, policy: AmbiguityPolicy) -> bool {
|
||||
mask != 0 && !policy.drops(mask)
|
||||
|
||||
@@ -35,7 +35,7 @@ pub(crate) use family_overlap::{
|
||||
pub(crate) use family_scalar_annex::ScalarKind;
|
||||
pub use family_scalar_annex::{FamilyScalarAnnex, FamilyScalarAnnexBuilder};
|
||||
pub use family_overlap::FamilyOverlap;
|
||||
pub(crate) use quartet_annex::QUARTET_ANNEX_FILE_NAMES;
|
||||
pub(crate) use quartet_annex::{QUARTET_ANNEX_FILE_NAMES, QuartetConvention};
|
||||
pub use ambiguity::AmbiguityPolicy;
|
||||
pub use iter::{
|
||||
MinorantBatchIter, MinorantIter, SiblingBatchIter, SiblingEntry, SiblingIter, SiblingLayerExt,
|
||||
|
||||
@@ -12,10 +12,15 @@
|
||||
//! fewer than four genomes — fits every topology and tells a tree nothing.
|
||||
//! See `algorithms::masking::is_quartet_informative` for the exact rule.
|
||||
//!
|
||||
//! Which genomes count as carriers depends on the ambiguity policy
|
||||
//! (`--no-ambiguity`, `--only-diploid`), so each policy has its own file, and **the file name is the parameter**: a
|
||||
//! `PersistentBitVec` header has no room for a flag, and changing it would
|
||||
//! change a shared structure for the sake of one annex.
|
||||
//! Absence counts as a state exactly when it does in the alignment: without
|
||||
//! `--free-loss` a genome carrying none of the family's members has the state
|
||||
//! `∅`, which can form a pair like any base; with `--free-loss` it is missing
|
||||
//! data and counts for nothing. Which genomes count as carriers also depends on
|
||||
//! the ambiguity policy (`--no-ambiguity`, `--only-diploid`). Each
|
||||
//! [`QuartetConvention`] (ambiguity policy × `--free-loss`) has its own file,
|
||||
//! and **the file name is the parameter**: a `PersistentBitVec` header has no
|
||||
//! room for a flag, and changing it would change a shared structure for the
|
||||
//! sake of one annex.
|
||||
//!
|
||||
//! The flag is computed over every genome of the index. A later
|
||||
//! `--exclude-genome` can only remove carriers, never create a bipartition,
|
||||
@@ -26,29 +31,59 @@
|
||||
|
||||
use crate::siblings::AmbiguityPolicy;
|
||||
|
||||
/// How a quartet annex counts the genomes of a family: one annex per value.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub(crate) struct QuartetConvention {
|
||||
/// How genomes carrying several members of the family are counted.
|
||||
pub policy: AmbiguityPolicy,
|
||||
/// `--free-loss`: absence is missing data instead of a state of its own.
|
||||
pub free_loss: bool,
|
||||
}
|
||||
|
||||
impl QuartetConvention {
|
||||
/// Every convention, in a fixed order.
|
||||
#[cfg(test)]
|
||||
pub(crate) const ALL: [QuartetConvention; 6] = [
|
||||
QuartetConvention { policy: AmbiguityPolicy::Keep, free_loss: true },
|
||||
QuartetConvention { policy: AmbiguityPolicy::Forbid, free_loss: true },
|
||||
QuartetConvention { policy: AmbiguityPolicy::OnlyDiploid, free_loss: true },
|
||||
QuartetConvention { policy: AmbiguityPolicy::Keep, free_loss: false },
|
||||
QuartetConvention { policy: AmbiguityPolicy::Forbid, free_loss: false },
|
||||
QuartetConvention { policy: AmbiguityPolicy::OnlyDiploid, free_loss: false },
|
||||
];
|
||||
}
|
||||
|
||||
// The three `--free-loss` files keep the names the annex had when absence was
|
||||
// never counted: that rule is exactly the `--free-loss` one, so annexes already
|
||||
// on disk stay valid under it. The files for absence-as-a-state are new names,
|
||||
// so an annex built under the old rule can never be read as one of them.
|
||||
pub(crate) const QUARTET_ANNEX_FILE_NAME: &str = "quartet.pbiv";
|
||||
|
||||
/// Same annex, built under [`AmbiguityPolicy::Forbid`].
|
||||
pub(crate) const QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME: &str = "quartet.noamb.pbiv";
|
||||
|
||||
/// Same annex, built under [`AmbiguityPolicy::OnlyDiploid`].
|
||||
pub(crate) const QUARTET_ANNEX_ONLY_DIPLOID_FILE_NAME: &str = "quartet.diploid.pbiv";
|
||||
pub(crate) const QUARTET_ANNEX_ABSENCE_FILE_NAME: &str = "quartet.abs.pbiv";
|
||||
pub(crate) const QUARTET_ANNEX_ABSENCE_NO_AMBIGUITY_FILE_NAME: &str = "quartet.abs.noamb.pbiv";
|
||||
pub(crate) const QUARTET_ANNEX_ABSENCE_ONLY_DIPLOID_FILE_NAME: &str = "quartet.abs.diploid.pbiv";
|
||||
|
||||
/// Every quartet-annex file name, whatever its counting convention — for
|
||||
/// invalidating them all together when the sibling annex they were numbered
|
||||
/// against is rebuilt.
|
||||
pub(crate) const QUARTET_ANNEX_FILE_NAMES: [&str; 3] = [
|
||||
/// Every quartet-annex file name, whatever its convention — for invalidating
|
||||
/// them all when the sibling annex they are numbered against is rebuilt.
|
||||
pub(crate) const QUARTET_ANNEX_FILE_NAMES: [&str; 6] = [
|
||||
QUARTET_ANNEX_FILE_NAME,
|
||||
QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME,
|
||||
QUARTET_ANNEX_ONLY_DIPLOID_FILE_NAME,
|
||||
QUARTET_ANNEX_ABSENCE_FILE_NAME,
|
||||
QUARTET_ANNEX_ABSENCE_NO_AMBIGUITY_FILE_NAME,
|
||||
QUARTET_ANNEX_ABSENCE_ONLY_DIPLOID_FILE_NAME,
|
||||
];
|
||||
|
||||
/// File name of the quartet annex built under the given ambiguity policy.
|
||||
pub(crate) fn quartet_annex_file_name(policy: AmbiguityPolicy) -> &'static str {
|
||||
match policy {
|
||||
AmbiguityPolicy::Keep => QUARTET_ANNEX_FILE_NAME,
|
||||
AmbiguityPolicy::Forbid => QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME,
|
||||
AmbiguityPolicy::OnlyDiploid => QUARTET_ANNEX_ONLY_DIPLOID_FILE_NAME,
|
||||
/// File name of the quartet annex built under the given convention.
|
||||
pub(crate) fn quartet_annex_file_name(convention: QuartetConvention) -> &'static str {
|
||||
match (convention.free_loss, convention.policy) {
|
||||
(true, AmbiguityPolicy::Keep) => QUARTET_ANNEX_FILE_NAME,
|
||||
(true, AmbiguityPolicy::Forbid) => QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME,
|
||||
(true, AmbiguityPolicy::OnlyDiploid) => QUARTET_ANNEX_ONLY_DIPLOID_FILE_NAME,
|
||||
(false, AmbiguityPolicy::Keep) => QUARTET_ANNEX_ABSENCE_FILE_NAME,
|
||||
(false, AmbiguityPolicy::Forbid) => QUARTET_ANNEX_ABSENCE_NO_AMBIGUITY_FILE_NAME,
|
||||
(false, AmbiguityPolicy::OnlyDiploid) => QUARTET_ANNEX_ABSENCE_ONLY_DIPLOID_FILE_NAME,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,11 +92,13 @@ mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_policies_never_share_a_file() {
|
||||
let names = AmbiguityPolicy::ALL.map(quartet_annex_file_name);
|
||||
assert!(names[0] != names[1] && names[1] != names[2] && names[0] != names[2]);
|
||||
for name in names {
|
||||
assert!(QUARTET_ANNEX_FILE_NAMES.contains(&name));
|
||||
fn the_conventions_never_share_a_file() {
|
||||
let mut names: Vec<&str> = QuartetConvention::ALL.map(quartet_annex_file_name).to_vec();
|
||||
for name in &names {
|
||||
assert!(QUARTET_ANNEX_FILE_NAMES.contains(name));
|
||||
}
|
||||
names.sort_unstable();
|
||||
names.dedup();
|
||||
assert_eq!(names.len(), QuartetConvention::ALL.len());
|
||||
}
|
||||
}
|
||||
@@ -7,4 +7,3 @@ edition = "2024"
|
||||
criterion2 = { version = "3", features = ["cargo_bench_support"] }
|
||||
|
||||
[dependencies]
|
||||
bytes = "1.11.1"
|
||||
@@ -342,6 +342,9 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn memo_key_from_debug() {
|
||||
// Fields are only read through `Debug`, which is what the memo key
|
||||
// is derived from — and what dead-code analysis ignores.
|
||||
#[allow(dead_code)]
|
||||
#[derive(Debug)]
|
||||
struct Args {
|
||||
x: u64,
|
||||
|
||||
@@ -13,7 +13,10 @@ niffler = { version = "3", default-features = false, features = ["gz", "bz2",
|
||||
bzip2 = "0.6"
|
||||
liblzma = { version = "0.4", features = ["static"] }
|
||||
ureq = "3"
|
||||
tracing = "0.1.44"
|
||||
tracing-subscriber = { version = "0.3.23", features = ["fmt", "env-filter"] }
|
||||
infer = "0.22.0"
|
||||
regex = "1"
|
||||
|
||||
[dev-dependencies]
|
||||
# Used by examples/expand_path.rs only.
|
||||
tracing = "0.1.44"
|
||||
tracing-subscriber = { version = "0.3.23", features = ["fmt", "env-filter"] }
|
||||
@@ -1,3 +1,4 @@
|
||||
use std::collections::HashSet;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
@@ -5,6 +6,10 @@ use std::path::{Path, PathBuf};
|
||||
pub struct PathIter {
|
||||
dir_stack: Vec<PathBuf>, // dossiers qu'il reste à explorer
|
||||
file_buffer: Vec<PathBuf>, // fichiers déjà listés (à distribuer)
|
||||
/// Paths given by the caller whose extension is not a recognised sequence
|
||||
/// one. They are tried as directories first; when that fails they are
|
||||
/// handed back as files (see [`PathIter::next`]) rather than dropped.
|
||||
explicit: HashSet<PathBuf>,
|
||||
}
|
||||
|
||||
impl PathIter {
|
||||
@@ -12,11 +17,18 @@ impl PathIter {
|
||||
///
|
||||
/// For non-directory paths only the file extension is checked (no stat syscall).
|
||||
/// Directories are left on the stack and expanded lazily in `next()`.
|
||||
/// Invalid or missing paths are silently skipped when `open_chunks` fails later.
|
||||
///
|
||||
/// A path without a recognised extension is tried as a directory; if it
|
||||
/// is not one, it is yielded as a file so the reader reports the real
|
||||
/// problem (missing file, not a FASTA/FASTQ stream) instead of the path
|
||||
/// being skipped silently. The extension filter only matters when
|
||||
/// expanding directories: format and compression are detected from the
|
||||
/// content.
|
||||
pub fn new(paths: Vec<PathBuf>) -> Self {
|
||||
let mut iter = PathIter {
|
||||
dir_stack: Vec::new(),
|
||||
file_buffer: Vec::new(),
|
||||
explicit: HashSet::new(),
|
||||
};
|
||||
for path in paths {
|
||||
// "-" is the stdin sentinel — pass it through without any extension
|
||||
@@ -28,6 +40,7 @@ impl PathIter {
|
||||
// where metadata operations can be 100s of milliseconds each.
|
||||
iter.file_buffer.push(path);
|
||||
} else {
|
||||
iter.explicit.insert(path.clone());
|
||||
iter.dir_stack.push(path);
|
||||
}
|
||||
}
|
||||
@@ -46,8 +59,19 @@ impl Iterator for PathIter {
|
||||
// Lire le dossier
|
||||
let entries = match fs::read_dir(&dir) {
|
||||
Ok(entries) => entries,
|
||||
Err(_) => continue, // si erreur (perm, etc.), on ignore ce dossier
|
||||
Err(_) => {
|
||||
// Not a readable directory. A path the caller named
|
||||
// explicitly is handed back as a file: the reader will
|
||||
// then fail with a precise error (no such file, not a
|
||||
// FASTA/FASTQ stream). Unreadable sub-directories met
|
||||
// during expansion are still skipped.
|
||||
if self.explicit.remove(&dir) {
|
||||
self.file_buffer.push(dir);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
self.explicit.remove(&dir);
|
||||
for entry in entries.flatten() {
|
||||
let path = entry.path();
|
||||
// Prefer file_type() over path.is_dir()/is_file() — on most filesystems
|
||||
@@ -72,20 +96,18 @@ pub fn path_iter(paths: &[String]) -> PathIter {
|
||||
}
|
||||
|
||||
/// Returns true if the path ends with a recognised sequence file extension.
|
||||
///
|
||||
/// `.fna`/`.ffn` (NCBI genomes and genes) and `.fas`/`.fsa` are FASTA
|
||||
/// extensions in common use, plain or gzip-compressed.
|
||||
fn is_fasta_or_fastq(path: &Path) -> bool {
|
||||
const SEQ_EXTENSIONS: [&str; 10] = [
|
||||
".fasta", ".fa", ".fna", ".ffn", ".fas", ".fsa", ".fastq", ".fq", ".gbff", ".gbk",
|
||||
];
|
||||
let name = path.file_name().and_then(|n| n.to_str()).unwrap_or("");
|
||||
name.ends_with(".fasta")
|
||||
|| name.ends_with(".fa")
|
||||
|| name.ends_with(".fastq")
|
||||
|| name.ends_with(".fq")
|
||||
|| name.ends_with(".fasta.gz")
|
||||
|| name.ends_with(".fa.gz")
|
||||
|| name.ends_with(".fastq.gz")
|
||||
|| name.ends_with(".fq.gz")
|
||||
|| name.ends_with(".gbff")
|
||||
|| name.ends_with(".gbk")
|
||||
|| name.ends_with(".gb")
|
||||
|| name.ends_with(".gbff.gz")
|
||||
|| name.ends_with(".gbk.gz")
|
||||
|| name.ends_with(".gb.gz")
|
||||
let name = name.strip_suffix(".gz").unwrap_or(name);
|
||||
SEQ_EXTENSIONS.iter().any(|ext| name.ends_with(ext)) || name.ends_with(".gb")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "tests/path_iterator.rs"]
|
||||
mod tests;
|
||||
@@ -0,0 +1,69 @@
|
||||
use super::*;
|
||||
use std::fs;
|
||||
|
||||
fn scratch_dir(tag: &str) -> PathBuf {
|
||||
let dir = std::env::temp_dir().join(format!("obiread_pathiter_{}_{tag}", std::process::id()));
|
||||
let _ = fs::remove_dir_all(&dir);
|
||||
fs::create_dir_all(&dir).unwrap();
|
||||
dir
|
||||
}
|
||||
|
||||
fn names(it: PathIter) -> Vec<String> {
|
||||
let mut v: Vec<String> = it
|
||||
.map(|p| p.file_name().unwrap().to_string_lossy().into_owned())
|
||||
.collect();
|
||||
v.sort();
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ncbi_and_common_fasta_extensions_are_recognised() {
|
||||
for name in [
|
||||
"g.fna", "g.fna.gz", "g.ffn", "g.fas", "g.fsa.gz", "g.fasta", "g.fa.gz", "g.fq", "g.fastq.gz",
|
||||
"g.gbff", "g.gbk.gz", "g.gb",
|
||||
] {
|
||||
assert!(is_fasta_or_fastq(Path::new(name)), "{name} should be recognised");
|
||||
}
|
||||
for name in ["g.txt", "g.gz", "g.fna.txt", "g", "g.bam"] {
|
||||
assert!(!is_fasta_or_fastq(Path::new(name)), "{name} should not be recognised");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_file_with_unknown_extension_is_not_dropped() {
|
||||
// The extension filter only applies when expanding directories: a file
|
||||
// named on the command line reaches the reader, which detects the format
|
||||
// from its content (and reports a precise error if it is not a sequence).
|
||||
let dir = scratch_dir("explicit");
|
||||
let f = dir.join("genome.unknownext");
|
||||
fs::write(&f, b">s\nACGT\n").unwrap();
|
||||
assert_eq!(names(PathIter::new(vec![f])), vec!["genome.unknownext"]);
|
||||
fs::remove_dir_all(dir).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_missing_path_is_handed_to_the_reader() {
|
||||
// Not silently skipped: opening it later yields the real I/O error.
|
||||
let missing = std::env::temp_dir().join(format!("obiread_pathiter_{}_missing.xyz", std::process::id()));
|
||||
assert_eq!(PathIter::new(vec![missing.clone()]).collect::<Vec<_>>(), vec![missing]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn directory_expansion_still_filters_by_extension() {
|
||||
let dir = scratch_dir("dir");
|
||||
for n in ["a.fna.gz", "b.fa", "c.txt", "d.bam"] {
|
||||
fs::write(dir.join(n), b"x").unwrap();
|
||||
}
|
||||
fs::create_dir(dir.join("sub")).unwrap();
|
||||
fs::write(dir.join("sub").join("e.fastq"), b"x").unwrap();
|
||||
assert_eq!(
|
||||
names(PathIter::new(vec![dir.clone()])),
|
||||
vec!["a.fna.gz", "b.fa", "e.fastq"]
|
||||
);
|
||||
fs::remove_dir_all(dir).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stdin_sentinel_passes_through() {
|
||||
assert_eq!(PathIter::new(vec![PathBuf::from("-")]).collect::<Vec<_>>(), vec![PathBuf::from("-")]);
|
||||
}
|
||||
@@ -8,7 +8,6 @@ obikseq = { path = "../obikseq" }
|
||||
obikrope = { path = "../obikrope" }
|
||||
obiread = { path = "../obiread" }
|
||||
obikentropy = { path = "../obikentropy" }
|
||||
lazy_static = "1.5.0"
|
||||
|
||||
[dev-dependencies]
|
||||
obikseq = { path = "../obikseq", features = ["test-utils"] }
|
||||
|
||||
@@ -49,6 +49,14 @@ impl Stage {
|
||||
wall_secs = format_args!("{:.3}", stats.wall_secs),
|
||||
rss = %fmt_bytes(stats.max_rss_bytes),
|
||||
swaps = stats.swaps,
|
||||
user_secs = format_args!("{:.1}", stats.user_secs),
|
||||
sys_secs = format_args!("{:.1}", stats.sys_secs),
|
||||
minor_faults = stats.minor_faults,
|
||||
major_faults = stats.major_faults,
|
||||
vol_ctx = stats.vol_ctx,
|
||||
invol_ctx = stats.invol_ctx,
|
||||
in_blocks = stats.in_blocks,
|
||||
out_blocks = stats.out_blocks,
|
||||
"done"
|
||||
);
|
||||
if stats.swaps > 0 {
|
||||
|
||||
Reference in new issue
Block a user