Compare commits

..
12 Commits
Author SHA1 Message Date
Eric Coissac 07bc5fc634 docs: update sparse packing strategy and benchmark scripts
ci.yml / build (pull_request) Successful in 4m53s
Release / create-release (push) Successful in 28s
Release / build-linux-x86_64 (push) Successful in 6m46s
Release / build-macos-arm64 (push) Successful in 4m8s
Update documentation to reflect that sparse packing now maps all columns of a layer at once and packs layers sequentially, reversing the previous choice to avoid materializing a packed copy. This change introduces an extra write/read cycle for the dense matrix, increasing pack benchmark time from 1.11s to 1.60s.

- Modify `pack_sparse_bit_matrix` in `src/obicompactvec/src/bitmatrix/sparse.rs` to always convert via the dense packed format, simplifying logic by removing conditional handling of existing files and ensuring a single mapping.
- Update `pack_sparse_compact_int_matrix` in `src/obicompactvec/src/sparse_intmatrix.rs` to exclusively handle conversion from the dense packed format (`matrix.pcmx`) to sparse, removing columnar format handling and simplifying file cleanup.
- Adjust `scripts/bench_merge_oracle.sh` to extract `PACK_WALL` metrics via `stage_field` for merge operations, updating grep patterns and output messages to report pack wall time.
- Revise `DevDocMD/implementation/merge_mmap_audit.md` and `merge_scaling.md` to document the new sequential packing strategy, noting that `N_part` scaling applies only to incremental merges and that `obisys::MemoryBudget` is no longer used by merge.
- Bump package version in `src/obikmer/Cargo.toml` from 1.4.3 to 1.4.4.
2026-11-17 15:35:41 +01:00
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00
coissac b35127e4e3 Merge pull request 'Push smsmsoyssmnr' (#80) from push-smsmsoyssmnr into main
Reviewed-on: #80
2026-11-09 10:01:21 +00:00
Eric Coissac 0134c69cf4 docs: clarify VCF pseudo-alignment format and update obikmer dependency
ci.yml / build (pull_request) Successful in 3m40s
Release / create-release (push) Successful in 1m7s
Release / build-macos-arm64 (push) Successful in 1m52s
Release / build-linux-x86_64 (push) Successful in 6m41s
Update the VCF export documentation to reflect that `-` represents an overlapping deletion (extra allele `*`) and introduce the `<ABS>` symbolic allele for absent families. Add a new dedicated documentation file for the VCF pseudo-alignment format, remove obsolete descriptions from phylo.md, and fix benchmark scripts to handle errors and directory creation correctly.

- Update `obikmer` dependency from 1.4.1 to 1.4.2 in Cargo.toml and Cargo.lock
- Add `UserDocMD/formats/vcf.md` describing VCF pseudo-alignment structure, alleles, and genotypes
- Remove obsolete VCF descriptions from `UserDocMD/usage/phylo.md` and update ploidy/free-loss notes
- Update `DevDocMD/theory/evolutionary_distances.md` to clarify `-` as overlapping deletion and `<ABS>` for others
- Fix `benchmark/index_one_count.sh` to create index directory and check read command errors
- Fix `benchmark/index_one_presence.sh` to create index directory and check simulation errors
- Update `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` to use `<ABS>` for absent families instead of `*`
2026-11-09 20:59:28 +11:00
Eric Coissac 5368c98875 refactor: replace no_ambiguity bool with AmbiguityPolicy enum
Replace the binary `no_ambiguity` flag with an `AmbiguityPolicy` enum (`Keep`, `Forbid`, `OnlyDiploid`) to support multiple ambiguity conventions. This change updates CLI arguments, internal logic, and persistent storage formats to distinguish between keeping all genomes, forbidding ambiguous ones, or restricting analysis to diploid sites only.

- Add `src/obikphylo/src/siblings/ambiguity.rs` defining the `AmbiguityPolicy` enum with variants `Keep`, `Forbid`, and `OnlyDiploid`, including methods for header flags and determining if a genome is dropped.
- Update `PhyloArgs` in `args.rs` to derive `AmbiquityPolicy` from CLI flags and construct `AnnexSelection` for sibling annexes.
- Modify `SamplingSpec` in `subsample.rs` to store `ambiguity: AmbiguityPolicy` instead of `no_ambiguity`, updating masking and scalar value logic accordingly.
- Update `SiblingExt` in `sibling_ext.rs` so `build_sibling_annex` accepts an `AnnexSelection` to build only requested derived annexes, and `family_overlap` uses the specific policy.
- Refactor `FamilyOverlap` in `family_overlap.rs` to store and use `AmbiguityPolicy`, replacing boolean checks with policy-based carrier determination.
- Update `FamilyScalarAnnex` in `family_scalar_annex.rs` to store `AmbiguityPolicy` in headers, enabling distinct file names for each policy variant.
- Modify `quartet_annex.rs` to generate distinct file names based on `AmbiguityPolicy`, ensuring no overlap between Keep, Forbid, and OnlyDiploid annexes.
- Update `family_annexes.rs` to use `AnnexSelection` for building entropy, quartet, and pair weight annexes under specific policies instead of a single boolean convention.
- Refactor algorithm modules (`entropy.rs`, `masking.rs`, `pair_weight.rs`, `quartet.rs`, `snp_distance.rs`) to accept `AmbiguityPolicy` arguments, replacing boolean checks with policy-specific logic (e.g., `OnlyDiploid` masking).
- Update documentation in `phylo.md` and `evolutionary_distances.md` to describe the new `--only-diploid` option, the behavior of `--sibling-annex` with specific policies, and the separate storage of entropies per convention.
2026-11-09 20:59:09 +11:00
Eric Coissac b55123a376 feat: add --only-diploid flag to filter for diploid genotypes only
Introduce a new `--only-diploid` option that restricts analysis to genomes with one or two family members (base or two-base IUPAC codes), treating others as missing data. This complements the existing `--no-ambiguity` behavior by providing a stricter filter for diploid-compatible sites, while ensuring conflict detection between the two flags.

- Add `only_diploid` boolean argument to `PhyloArgs` in `src/obikmer/src/cmd/phylo/args.rs`, documenting its effect on pseudo-alignment and its conflict with `--no-ambiguity`.
- Propagate `only_diploid` through `SamplingSpec` in `src/obikmer/src/cmd/phylo/mod.rs` and update the `run` function signature to accept the new argument.
- Implement masking logic in `src/obikphylo/src/siblings/algorithms/masking.rs`: modify `masked_state` to accept `only_diploid`, add `only_diploid_keeps_up_to_two_members` helper, and update `genome_mask` documentation.
- Update `sample_layer` in `src/obikphylo/src/siblings/algorithms/subsample.rs` to use `spec.only_diploid` for masking decisions and update `SamplingSpec` documentation.
- Adjust `tally_2genomes` in `src/obikphylo/src/siblings/algorithms/snp_distance.rs` to pass the new `only_diploid` parameter to `masked_state`.
- Update `DevDocMD/theory/evolutionary_distances.md` to describe the new `--only-diploid` option, the `masked_state` rule (`count_ones() > 2`), and clarify toy index behavior.
- Update `UserDocMD/usage/phylo.md` to document `--only-diploid`, `--quartet`, and `--pair-balance` behaviors, clarifying that diploid choice is a convention and explaining how three/four-allele genomes are treated as missing.
2026-11-09 20:58:21 +11:00
Eric Coissac b132f0847f feat: add VCF export and refine alignment output infrastructure
Introduce `--vcf` and `--fasta` options for phylo commands, allowing users to
export pseudo-alignment data as VCF or FASTA. Implement comprehensive VCF logic
including stable virtual coordinates, canonical ID generation, and flexible
genotype representation based on ambiguity settings. Refactor internal alignment
structures to track k-mer families and site loci, ensuring consistent data flow
from scanning through to output generation.

- Add `src/obikmer/src/cmd/phylo/alignment_out.rs` with `create` and
  `write_alignment_outputs` to handle file creation and dispatching of FASTA/VCF
  formats based on `PhyloArgs`.
- Add `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` implementing
  VCF-specific logic: `write_vcf`, `alignment`, `cell_alleles`, `majority_base`,
  and site construction via `site` using fixed $K=11$. Includes helpers for
  IUPAC state inversion, genotype ordering, and stable coordinate generation.
- Update `src/obikmer/src/cmd/phylo/args.rs` to add `fasta` and `vcf` flags
  and a new `alignment_output()` method that returns true if any output format
  is requested. Add `pseudo_alignment` field documentation.
- Modify `src/obikmer/src/cmd/phylo/mod.rs` to use `alignment_output()` for
  determining alignment needs, update error messages for `--n-sites`, and import
  `write_alignment_outputs` from the new module.
- Refactor `src/obikphylo/src/siblings/algorithms/alignment.rs` by adding
  `SiteLocus` struct (partition, layer, family_idx, family_id) and updating
  `SnpAlignment` and `AlignmentReducer` to track sites. Update `new`, `finish`,
  and `reduce` to populate site information from `SurvivingFamily`.
- Update `src/obikphylo/src/siblings/algorithms/iqtree.rs`: `alignment` now
  returns `SnpAlignment` with empty sites; update `drop_ascertainment_noninformative`
  and `recode_symbols_as_absent` to handle the new `sites` field.
- Update `src/obikphylo/src/siblings/algorithms/family_scan.rs`: `GeneratedBatch`
  now includes a `kmers` field; `scan_layer_families` accepts `CanonicalKmer`
  and passes it to closures instead of masks.
- Update `src/obikphylo/src/siblings/algorithms/subsample.rs`: `SurvivingFamily`
  gains a `kmer` field; `sample_layer` closure accepts and uses this k-mer.
- Update `src/obikphylo/src/siblings/algorithms/snp_distance.rs`: `tally_2genomes`
  initializes `SurvivingFamily` with explicit `CanonicalKmer` values.
- Update `src/obikphylo/src/siblings/algorithms/stats.rs`: `sibling_annex_stats`
  signature adds `_kmer` parameter.
- Update `src/obikphylo/src/siblings/algorithms/family_annexes.rs`: closures in
  `ensure_layer_family_annexes` and `ensure_layer_pair_weight_annexes` now accept
  `_kmer`.
- Update `src/obikphylo/src/siblings/algorithms/pairwise.rs`: `ambiguous_vs_single`
  initializes k-mer using `CanonicalKmer::from_raw_unchecked(0)`.
- Add `iupac_state` function to `src/obikphylo/src/siblings/algorithms/masking.rs`
  mapping DNA characters to binary codes.
- Update `src/obikphylo/src/siblings/algorithms/mod.rs` to export `SiteLocus`
  and include new `alignment_vcf` module.
- Add `family_id` and `write_family_id_ascii` methods to
  `src/obikseq/src/kmer.rs` for stable k-mer family identification.
- Add tests `family_id_partitions_kmers_into_families` and
  `family_id_ascii_masks_centre` to `src/obikseq/src/tests/kmer.rs`.
- Update `src/obikphylo/src/siblings/mod.rs` imports to include new algorithm
  exports.
2026-11-09 20:58:05 +11:00
coissac ee62b131f9 Merge pull request 'Push ltpzonnnkpqs' (#79) from push-ltpzonnnkpqs into main
Reviewed-on: #79
2026-10-16 14:43:50 +00:00
Eric Coissac 85296a2ef8 Implement directory locking with shared and exclusive modes
Release / create-release (push) Successful in 2m41s
Release / build-linux-x86_64 (push) Successful in 9m17s
Release / build-macos-arm64 (push) Successful in 2m2s
ci.yml / build (pull_request) Successful in 3m48s
Introduces a robust, OS-level directory locking mechanism using LockSlot for managing shared and exclusive access, supporting lock transitions and non-blocking probes. Also sets up the index structure with new builder and index modules.
2026-10-16 16:02:00 +02:00
Eric Coissac 9ac3a37529 Implement explicit directory locking for index operations
Introduces explicit directory locking mechanisms (`DirLock`) to ensure exclusive access during index operations across command stages. This includes modifying index opening logic to acquire locks before proceeding, handling lock acquisition failures by exiting, and implementing safe conditional building of sibling annexes using locking to prevent concurrent corruption.
2026-10-16 16:00:49 +02:00
Eric Coissac 7c7d7615cb Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
2026-10-16 15:59:40 +02:00
coissac c6226dd40c Merge pull request 'Push wpvskkyozppn' (#78) from push-wpvskkyozppn into main
Reviewed-on: #78
2026-10-15 10:38:58 +00:00
74 changed files with 3650 additions and 1255 deletions

No files matched your search

+18 -32
View File
@@ -62,52 +62,38 @@ fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
Check all sources against the constraints above. Abort on any mismatch. Check all sources against the constraints above. Abort on any mismatch.
### 2. Bootstrap output from first source ### 2. Bootstrap output from the base source
Recursive file copy of `sources[0]` → `output`. Immediately after the copy: The base is chosen by `choose_base` (`merge.rs`): the source with the largest total `unitigs.bin` size (restricted to approximate-evidence sources when the output must be approximate). It is not necessarily `sources[0]`, and the genome columns follow the base first. Because `obikmer index` is not byte-reproducible (unitig segmentation varies), two near-equal sources can swap roles between runs. Its directory is copied to `output` (hard links for files `merge_partition` never rewrites, see below). Immediately after the copy:
- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind. - `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
- In `Presence` mode, any `counts/` directories inherited from source_0 are removed. - In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels. - `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination. This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The base source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
### 3. For each subsequent source (parallel across partitions) ### 3. For each subsequent source (parallel across partitions)
`KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)` is called for each partition index `i`. `block_bits` is taken from `dst.meta.config.block_bits`. `obikmerge::partition_merge::merge_partition(dst, i, sources, mode, n_dst_genomes, block_bits, evidence)` runs once per partition index `i` (scheduled largest first by `PartitionRunner`). Each entry of `sources` is `(&KmerIndex, n_genomes)`, where `n_genomes` is the number of columns that source contributes (> 1 for an already-merged source). Code: `partition_merge/mod.rs` (orchestration, pass 1) and `partition_merge/pass2.rs` (per-group fill pipeline).
Each entry in `sources` is `(&KmerPartition, n_genomes)` where `n_genomes` is the column count that source contributes (> 1 when the source is itself a merged index). **First merge, Presence mode**: when `n_dst_genomes == 1`, `TypedLayer::<()>::init_presence_matrix` is called on every existing dst layer first. It materialises the implicit single-genome matrix (`presence/col_000000.pbiv`, all true) so that source columns can be appended to it.
**First merge, Presence mode**: when `n_dst_genomes == 1`, `Layer::<()>::init_presence_matrix` is called on every existing destination layer before any source column is appended. This creates `presence/col_000000.pbiv` set all-true (genome 0 is present in every slot). The dst layers are opened once per partition and shared by both passes.
**Pass 1 — classify kmers** **Pass 1 — new k-mers into a de Bruijn graph.** Only the sources' `unitigs.bin` *paths* are needed (`KmerLayer::new(..).unitigs_path()`); no source layer is opened. The files are streamed (at most `max_open = 2` open at once), and every k-mer absent from all dst layers is pushed into one `GraphDeBruijn`. All new k-mers of all sources go into a **single** new layer per merge, built by `materialize_layer` (unitigs + MPHF + evidence).
Iterate all kmers from all source partitions (via `UnitigFileReader` + canonical kmer iteration). For each kmer, probe the destination `LayeredMap<()>`: **New-layer matrix.** If a new layer exists, its matrix is created with `n_dst_genomes` absent columns (all zero/false) and closed immediately. This keeps the column-count invariant, and every source group then appends to it like to the existing layers.
- **Hit**: kmer already in the destination; record for Pass 2. **Pass 2 — by consecutive groups of `SRC_GROUP = 2` sources, in source order.** Per group:
- **Miss**: push kmer into a `GraphDeBruijn` accumulator.
**New layer construction** 1. open the group's source layers for this partition;
2. `MatrixBuilder::resume` every dst matrix (existing layers + new layer), check `n_cols == n_dst_genomes + col_base`, append the group's `group_n` columns;
3. run the pipeline: stream each source layer's k-mers, `hash_batch` + `fill_sub_matrix` on the source, locate each k-mer in the dst layers (`find_batch_grouped`) or else in the new layer's MPHF, and write the values column-major into the group's builders;
4. close the columns, then the matrices; drop the group's layers; `col_base += group_n`.
If the accumulator is non-empty, compute de Bruijn unitigs and call `Layer::<()>::build(&new_layer_dir, block_bits)`. All kmers absent from the destination — across **all** sources — accumulate into a **single** graph, producing one new layer per merge operation (not one per source). Only one group's source layers and destination columns are alive at a time, so the number of live mappings per partition does not depend on the number of sources. Columns are appended in source order, so every genome lands in the same absolute column as in a single-session merge.
**Pass 2 — fill column builders** **In-memory columns.** The group's columns are built in RAM (`MatrixBuilder::add_mem_col` → `MemBitVecBuilder` / `MemCompactIntVecBuilder`) and written once, sequentially, at `close` (`.building` file, then `rename`). The files are byte-identical to the former mmap-backed builders' output. There is no sparse file and no `MmapMut` on this path, so a block the FS cannot allocate is an `io::Error` instead of a SIGBUS. Files are not `fsync`ed: one sync per column file made merge ~8× slower. RAM per partition is at most `SRC_GROUP × src_n × N_part × (1/8 | 1)` bytes (presence | counts). History and figures: `merge_mmap_audit.md`; scaling limits: `merge_scaling.md`.
For each source and each of its layers, re-iterate unitigs and look up stored values via `SrcLayerData::lookup(kmer, src_n)`:
- `SrcLayerData::SetMembership` — no data directory exists; every kmer returns `vec![1; n_genomes]`
- `SrcLayerData::Presence` — reads `PersistentBitMatrix` from `presence/`
- `SrcLayerData::Count` — reads `PersistentCompactIntMatrix` from `counts/`
Hits are routed to `exist_builders[dst_layer][src_col]`; misses are routed to `new_src_builders[src_col]`.
**Column prepending for new layers**
Before source columns are written to the new layer, `n_dst_genomes` absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.
**Close and update metadata**
Close all builders; update `presence/meta.json` or `counts/meta.json` with `{"n": N, "n_cols": n_dst_genomes + n_src_total}`; increment `PartitionMeta::n_layers` if a new layer was added.
### 4. Update index metadata ### 4. Update index metadata
@@ -129,7 +115,7 @@ impl Layer<PersistentBitMatrix> {
} }
``` ```
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written directly via `PersistentBitVecBuilder` / `PersistentCompactIntVecBuilder` rather than through these helpers, but the invariant they enforce is the same. Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written through `MatrixBuilder::add_mem_col` (in-memory builders) rather than through these helpers, but the invariant they enforce is the same.
--- ---
@@ -139,7 +125,7 @@ After any merge, **every layer in every partition has exactly `n_genomes` column
Maintained by three mechanisms: Maintained by three mechanisms:
1. **Existing layers**: `n_src_total` columns appended (one per source genome). 1. **Existing layers**: `n_src_total` columns appended (one per source genome), group by group.
2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns. 2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0. 3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
@@ -161,7 +147,7 @@ The invariant is a precondition of `LayeredStore` aggregation traits: `col_weigh
## On-disk impact ## On-disk impact
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest): After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest), *before* `finalize` packs each matrix (sparse by default, `matrix.pbmx` with `--dense`):
``` ```
partitions/ partitions/
@@ -0,0 +1,90 @@
# Merge: mmap audit (2026-11-17)
Audit of the memory-mapping behaviour of `obikmerge::partition_merge::merge_partition`, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS `/hoyt`). Status: **A + C implemented (2026-11-17)**, B deferred. See [Implementation](#implementation-2026-11-17). Implementation prompt: `prompts/merge_mmap_refactor.md` (repository root, outside the doc tree).
## Observed failures
| Run | Flags | Failure | When |
|---|---|---|---|
| 1 | none | `Bus error` (SIGBUS) | seconds after `merge_partitions` starts |
| 2 | `--cpu-max 32` | `memory allocation of 32768 bytes failed` → abort, many threads at once | after ~1 h, 128/256 partitions done |
Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, `merge.rs:289`), so at 50 % every remaining partition is smaller than one already merged. Small `malloc` failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from `malloc` means `mmap`/`brk` returned `ENOMEM`: `vm.max_map_count` (default 65 530), `RLIMIT_AS`, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.
## Inventory: mappings held by one `merge_partition` call
Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).
| Resource | Where | Maps | Lifetime | Needed? |
|---|---|---|---|---|
| Source layers, fully opened | `partition_merge.rs:219-227` → `KmerLayer::open` → `MphfLayer::open` (`mphf_layer.rs:171`) | 4 per layer: MPHF (`MphfOnly`), `unitigs.bin` (direct access), `evidence.bin`, `unitigs.bin` again (`LayerEvidence::open`, `mphf_layer.rs:100`). ≈ **740** | whole partition | Only the MPHF. Pass 2 calls `hash_batch` (MPHF only, `mphf_layer.rs:258`) and `fill_sub_matrix` (matrix: `Implicit` for mono-genome, zero maps). Pass 1 only needs the `unitigs.bin` *path*. |
| Writable columns, created up front | `partition_merge.rs:327`, `:349` → `PersistentBitVecBuilder::new` / `PersistentCompactIntVecBuilder::new` | `set_len` (sparse) + `MmapMut`, `n_src_total × (L_dst + 1)` ≈ **370** | whole pass 2 | Only the columns of the ≤ 2 sources being streamed (`max_open = 2`). |
| dst layers, new-layer MPHF, sequential readers | various | ≈ 10 | variable | yes |
≈ **1 100 maps per partition**, > 95 % idle at any instant. × ~32 concurrent partitions (`--cpu-max 32`) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, **not measured**. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is **not established**.
## Weaknesses
1. **Source layers opened in full** although merge needs MPHF + matrix only. Evidence and both `unitigs.bin` maps are dead weight.
2. **All sources opened up front, for the whole partition.** This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
3. **All destination columns created up front**, the same mismatch on the write side.
4. **Sparse files written through `MmapMut`.** A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing.
5. **No bound on live mappings.** `obipipeline::throttle` only bounds open read streams.
6. *(minor)* `obicompactvec::mmap_file::open_mmap_file` validates magic and header length, not file length against `n`.
Side finding: `merge_parallelism.md` documents a `MemoryBudget` semaphore and `--budget-fraction`. `obisys::MemoryBudget` still exists but `merge.rs` no longer uses it (scheduling is `PartitionRunner`). That page is stale on this point.
## Plan
**A. Source-group streaming in pass 2** (`obikmerge`). Process sources in consecutive groups of `G` (G = 2, matching `max_open`). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its `src_n` columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through `KmerLayer::new(..).unitigs_path()` (valid on `Empty`). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.
**C. In-memory column builders** (`obicompactvec`). Under A only `G × src_n × (L_dst+1)` columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with `write_all` to the `.building` path, then publish. No sparse file and no `MmapMut` on the merge write path, so `ENOSPC`/quota becomes an `io::Error`. The on-disk format must be byte-identical to the current builders' output. Exposed through `MatrixBuilder`/`ColBuilder` so `obikselect`/`obikfilter` can adopt it later. Fixes 4.
**B. Slots-only layer opening** (`obikindex`, the authority crate). **Deferred to a separate work item** (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same `hash_batch` + `fill_sub_matrix` pattern without `find` appears in three crates: `obikmerge` (`partition_merge.rs:489`), `obikfilter` (`partition_iter.rs:106`), `obikphylo` (`family_scan.rs:289`). `family_scan.rs:402` hashes `variants`; whether those are guaranteed members is unchecked. Fixes 1.
- **Rejected: lazy evidence inside `MphfLayer`.** `find`/`confirm` return `Option<usize>`, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (`find`, needs evidence) vs in-layer iteration (`index`/`index_batch`, no evidence).
- **Rejected: a standalone parallel type** next to `KmerLayer`/`TypedLayer`. It would duplicate count/presence detection and matrix access, and two opening paths would drift.
- **Retained: composition.** A slots layer = `MphfOnly` + matrix, carrying `hash_batch`, `fill_sub_matrix`, `n_cols`. The full layer = slots layer + evidence, keeping `find`/`confirm`. `KmerLayer` exposes both openings (`open()` / e.g. `open_slots()`) over a single count/presence detection. Open point: `MphfLayer::n` currently comes from the evidence (`evidence.len()` / `fingerprint.n()`). A slots layer must take it from the matrix (`n_rows`; `layer_meta.json` for `Implicit`), so the two must be checked to always agree.
Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the `obikindex` query path and needs its own session with `query` performance checks.
Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.
## Implementation (2026-11-17)
**A — source-group streaming** (`obikmerge/src/partition_merge/`, split into `mod.rs` + `pass2.rs`). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates `sources.chunks(SRC_GROUP = 2)`, resuming each dst matrix, checking `n_cols == n_dst_genomes + col_base` (`InvalidData` otherwise), and appending the group's columns. Prerequisite: `MatrixBuilder::n_cols` (`obicompactvec`).
**C — in-memory columns** (`obicompactvec`). New `mem_builder.rs` (`MemBitVecBuilder`, `MemCompactIntVecBuilder`), `add_mem_col` on both matrix builders and on `MatrixBuilder`, and `ColBuilder::MemBit`/`MemInt`. Format writing is factored rather than duplicated: `format::write_pciv_header`/`write_pciv_tail` (also used by `finalize_pciv`), `mmap_file::write_std_header`/`write_built`. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses `add_mem_col` for source columns; `add_absent_col` is unchanged (never written through its mapping).
Deviations from the plan:
- **No `sync_all` in `close`.** With one forced sync per column file, `merge_partitions` went from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. The `rename` still prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface at `close(2)`, which `File`'s drop ignores.
- `add_mem_col` returns `ColBuilder`, not `io::Result<ColBuilder>`: it does no I/O.
- The mmap count builder (`PersistentCompactIntVecBuilder`) writes in place, without `.building`; the in-memory one goes through `.building` + `rename`. Same final bytes.
- `obikmerge`'s `matrix_builder_tests` were removed: they only exercised `obicompactvec::MatrixBuilder` and are covered by `obicompactvec/src/tests/{matrix_builder,mem_builder}.rs`. `tempfile` dev-dependency of `obikmerge` removed with them.
**Verification.** Two golden-output oracles, recorded on the unmodified code:
- `scripts/smoke_test_merge.sh`: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/`--dense`; frozen source indexes;
- `scripts/bench_merge_oracle.sh`: 20 real specimen presence indexes, sorted `dump --debug` hash, 85 681 700 k-mers.
Both pass after every step. The existing `global_index_presence` (October binary) gives the same hash.
`merge_partitions` wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.
**Live mappings per partition (estimate, not measured).** Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.
| | Source layers | Dst columns | Other | Total |
|---|---|---|---|---|
| Before | 186 × 4 ≈ 744 | 186 × 2 ≈ 372 (`MmapMut`) | ≈ 10 | **≈ 1 100** |
| After A + C | 2 × 4 = 8 | 0 (in RAM) | ≈ 10 | **≈ 20**, independent of the source count |
B would take the 8 down to ~2.
**Follow-up — sparse pack (same day).** `pack_sparse_bit_matrix`/`pack_sparse_compact_int_matrix` used to open the columnar matrix directly, which maps every column file of a layer at once: `n_genomes` maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single `matrix.pbmx`/`.pcmx` mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: `pack` went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale `matrix.pbmx` that predates newer columnar data is now re-packed instead of being used as is (`pack_bit_matrix` compares `n_cols`). Both oracles pass.
## Known issue (found while building the oracle, not fixed)
Re-merging a merge output fails with `I/O error: No such file or directory` in `merge_partitions`, in every mode (presence sparse, presence `--dense`, counts). Cause, verified: `finalize_indexed` packs the output matrices (`sparse_meta.json` + dict/PFIV files, or `matrix.pbmx`), so `presence/`/`counts/` no longer holds `meta.json` + `col_*`. When that index is the merge base, bootstrap copies the packed layout and `MatrixBuilder::resume` → `MatrixMeta::load` finds no `meta.json`. Pre-existing, independent of this refactor; deferred to a separate fix (decision 2026-11-17). The synthetic oracle (`scripts/smoke_test_merge.sh`) therefore omits the "merge AB, then AB C D" scenario.
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to `mmap`.
+41
View File
@@ -0,0 +1,41 @@
# Merge: scaling to thousands of genomes (open, 2026-11-17)
Open design question raised during the mmap refactor (`merge_mmap_audit.md`). Target: merges of ~6 000 genomes of a few Gb each. Nothing here is implemented or measured. Figures are estimates, labelled as such.
## What scales with what (after the mmap refactor)
| Cost | Depends on | Status |
|---|---|---|
| Live source layers + dst columns per partition (merge pass 2) | `SRC_GROUP × src_n`, not the source count | fixed by plan A |
| Pass-2 column RAM | `SRC_GROUP × src_n × N_part × (1/8 \| 1)` B | bounded, see below |
| Pass-1 de Bruijn graph | ≥ ~19 B per *new* k-mer (`HashMap<u64, AtomicU8>`, verified in `obidebruinj/src/debruijn/graph.rs`) | unchanged |
| Columnar intermediate files | `n_genomes × n_layers × n_partitions` (6 000 × 2 × 256 ≈ 3 M files, estimate) | **does not scale** |
| Sparse `pack` | used to map **all** columns of a layer at once (`ColumnarBitMatrix::open`); layers are packed one after another, so `n_genomes` maps per partition (186 × 192 concurrent ≈ 36 k, estimate) | **fixed 2026-11-17**: now goes columnar → dense `matrix.pbmx` (sequential concatenation, unmapped) → sparse, one mapping per partition |
`N_part`: k-mers of one dst partition over all layers. Pass-2 column RAM stays below pass-1 graph RAM for one-shot merges. It dominates only for **incremental** merges, where few genomes are added to a large index: the new layer is small, the existing layers are large. Example (estimate): `N_total = 1e12`, counts, 192 concurrent partitions → ~1.5 TB. A memory-budgeted concurrency cap is needed there whatever the column strategy. `obisys::MemoryBudget` exists but is no longer used by merge.
## Why a columnar intermediate at all
The sparse format (`bitmatrix/sparse.rs`) is row-major and deduplicated: per slot, either one genome (`singleton.pfiv`) or a `dict_id` (`multi.pfiv`) into a dictionary of distinct genome sets (`dict_offsets` + `dict_values.bin`). Its builder only offers `push_row(genomes)`, i.e. the **complete** genome set of a row, rows in **slot order**.
Merge discovers values **column by column** (source by source), in **random slot order** (MPHF order). The columnar files are a transposition buffer: they absorb random per-column writes, and `pack` transposes them row by row.
## Options to build the final format directly
1. **All sources at once.** That is exactly what plan A removed. No.
2. **Whole matrix in RAM.** `N_part × n_genomes / 8` B per partition: ~3 TB at 6 000 genomes and `N_part = 4e9` (estimate). No.
3. **External sort of `(slot, genome)` pairs.** Volume ∝ non-zeros ≈ Σ genome k-mers (~6 000 × 3e9). That is huge, while the final format is small precisely because genome sets repeat. No.
4. **Incremental sparse update per source group** (most promising). Per group:
- fill the group's columns in RAM (as now);
- stream the partition's existing sparse matrix **sequentially in slot order**;
- for each row: new set = old set ∪ group bits. Through the dictionary this is a transition `(old dict_id, group bit pattern) → new dict_id`, which can be memoised in a table;
- write the new sparse matrix sequentially, then publish.
No columnar files, no `pack`, no per-genome mapping, sequential I/O only. It also removes the re-merge bug: it starts from an existing sparse matrix instead of requiring a columnar one.
The cost is **one full pass over the partition's sparse matrix per group**. With `SRC_GROUP = 2` and 6 000 genomes that is 3 000 rewrites, far too many. Groups must therefore be large (64–256 sources), and group column RAM grows with them: `group × N_part / 8` B, e.g. 32 GB per partition for 64 sources at `N_part = 4e9` (estimate). One parameter trades passes against RAM, and it should be sized from a memory budget. The count matrix (`sparse_intmatrix.rs`) has its own format and needs the same analysis.
## Related findings
- The module header of `bitmatrix/sparse.rs` says "not used by any production code path yet". This is stale: `finalize_indexed` packs to sparse by default.
- The new layer of a merge is not byte-reproducible run to run: unitig segmentation varies, so the MPHF, evidence and matrix row order vary with it (measured 2026-11-17, `layer_1/*` differs in all 256 partitions; `layer_0` and metadata are identical). `obikmer index` shows the same behaviour. Merge oracles therefore compare sorted dumps, not file hashes. Where the non-determinism originates (parallel graph fill or unitig walk) is not established.
+84
View File
@@ -2873,6 +2873,90 @@ user, no external consumers to break). Scoped to the distance matrix only:
`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep `--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
their existing CSV-only format. their existing CSV-only format.
## VCF export of the pseudo-alignment
`phylo --vcf` writes the drawn sites of `--pseudo-alignment` as a VCF
(`--fasta`/`--vcf` choose the formats, FASTA is the default, both can be
combined). A family has no genomic coordinate, so coordinates are virtual and
derived from where the scan finds the family: `CHROM = obikmer_<partition>_<layer>`,
`POS = family_idx + 1` (`scan_layer_families`' iteration-order index inside
that layer). Stable for a given index + annex, sorted by construction because
`on_family` is replayed in `family_idx` order. A global dense rank or a bit
packing of (layer, partition, idx) into one `POS` was rejected: it needs
per-layer minorant counts and can exceed VCF's 2^31-1 bound.
`ID` is `CanonicalKmer::family_id()`: centre forced to `A` on the
k-mer and on its revcomp, each canonicalised, minimum of the two (centre
printed as `N`). Index-independent, hence the key for merging VCFs from
different indexes. No `INFO/LOCUS` duplicate (dropped: same string as `ID`, costly
at 5M loci); `QUAL`/`FILTER`/`INFO` are all `.`. An alternative defined as the minimum of
`central_canonical_neighbors()` was dropped: it canonicalises 4 k-mers where
this one needs 2, with no other benefit. `family_id` idempotence and
family-wide equality are tested exhaustively for k = 3, 5, 7.
`SurvivingFamily` now carries the minorant k-mer (the scan callback passes it),
and `SnpAlignment` a `SiteLocus` per column. Alleles are the unambiguous cells
of the IUPAC matrix, in the FASTA's convention (central base of the member's
canonical form, not necessarily the `ID` k-mer's strand).
Absence handling. `masked_state` keeps absence (`m == 0`, `-`) as a real state
unless `--free-loss`, where it becomes missing (`?`). A first version mapped
`-` and `?` both to `.`: on a toy 4-genome index half of the lines then had
`ALT = .` (presence/absence-only sites, selected by `survives_masking`), i.e.
polymorphic sites rendered as monomorphic. Absence became an extra allele,
first written `*`; that was a misuse: in VCF 4.5 (section on the ALT column)
`*` is "allele missing due to overlapping deletion", and `<*>` is the gVCF
unspecified allele, `DEL`/`INS`/`DUP`/`INV`/`CNV` are structural-variant
types. Now the symbolic allele `<ABS>` (free-form ID, declared by a
`##ALT=<ID=ABS,...>` line, always written in the header, listed after the
bases); `?` and IUPAC ambiguity codes stay `.` (haploid GT cannot express a
genome carrying several forms; superseded below). A symbolic ALT formally
asks for a padding base before `POS`; moot here, positions being virtual.
Checked with `bcftools view`/`stats` and `tabix`: no warning, the sites with
`<ABS>` are classed "others" by `bcftools stats`, and
`-e 'ALT="<ABS>"'` drops exactly them. The writer needs no knowledge of
`--free-loss`: `-` simply never occurs with it. A reference-genome option
(`--vcf-ref`) was implemented then dropped: `REF` is always the per-site
majority base (alphabetical on ties).
Ploidy. VCF allows any ploidy in `GT` (number of alleles written, variable
per sample/site), but an obikmer cell only says *which* forms a genome
carries, not their multiplicity (`A/A/G` and `A/G/G` are both `R`), nor
whether several forms mean heterozygosity or paralogues. Compromise chosen:
`--no-ambiguity` => haploid (ambiguous cells are already `?`); otherwise
diploid, single base `X` -> `X/X`, two-base code -> `X/Y`, `-` -> `<ABS>/<ABS>`, and
three/four-base codes or `?` -> `./.`. `REF` is the majority allele counting
each copy once (so a het contributes 1+1, a homozygote 2). Counting copies
for a real dosage (polyploids) would need k-mer counts (`--with-counts`),
not used by the pseudo-alignment scan today. `--only-diploid` (selection option, `conflicts_with --no-ambiguity`) adds a
third `masked_state` rule: `count_ones() > 2` -> missing. It lives in
`SamplingSpec`/`masked_state`, i.e. the post-hoc masking and
`survives_masking`, so FASTA, Sankoff and `snp-*` see it too. Unit-tested
(`masked_state`); the toy index has no 3/4-member cells, so the end-to-end
run is identical to the default mode.
First version of this flag read the persisted annexes under the default
convention (a third variant was judged not worth it). Rejected: the draw
would then describe other genomes than the alignment, and the pair-balance
weights and entropy are computed from the carrier set. Replaced by
`AmbiguityPolicy { Keep, Forbid, OnlyDiploid }` (one type instead of
`free_loss`/`no_ambiguity`/`only_diploid` booleans in `masked_state`,
`is_carrier`, `SamplingSpec` and every annex key). Annex headers store
`AmbiguityPolicy::header_flags` (0 = Keep, 1 = Forbid, as the old bit, 2 =
OnlyDiploid), so annexes written before the change stay valid; each policy
has its own file (`*.diploid.*` for the new one). `--sibling-annex` no longer
precomputes everything (`AnnexRequest::everything` removed, a rebuild only
costs the base annex): it builds the derived annexes named by the other
options of the command (`AnnexSelection`: `--entropy`, `--quartet`,
`--pair-balance`, `--family-overlap`/`--min-shared-family`, under the one
policy given); the rest is built on demand as before. A rebuild still drops
every derived annex (their `family_idx` numbering is tied to the base one).
Checked on the toy index: `--sibling-annex` alone leaves only
`siblings.psib`; `--quartet --only-diploid` adds `quartet.diploid.pbiv` only.
Toy index: default mode 187
lines, no `ALT = .`; `--no-ambiguity` 93 lines (the sites whose polymorphism
was only ambiguity codes disappear, since those cells became `?`).
## References ## References
The Mash mutation-rate model this discussion contrasts with: The Mash mutation-rate model this discussion contrasts with:
+28 -1
View File
@@ -329,7 +329,34 @@ Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 /
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above. **`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment has not been given to IQ-TREE. **Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment had not been given to IQ-TREE (done 2026-10-16: see the last measured section).
## Tree-level effect of the combined draw: sCF on three trees (measured 2026-10-16)
Closes the "not measured" item of the previous section (the combined alignment given to IQ-TREE). Three trees, all sCF by `--scfl 100 --cf-verbose`, model `GTR+G4+ASC`:
- **A** = `concord` (12/10): entropy-only draw, 180 genomes, 5 003 414 columns; `-t` + `-m` re-ran a full ML search (59 h).
- **B** = `concord_quartet_balance`: combined draw (`--entropy 1 --quartet --pair-balance`), 180 genomes, 5 000 000 columns. Its log stops during the tree search (14/10), so the command that produced its `cf.*` (16/10) is not in the files; presumably `-te` on the 15/10 treefile as for G (unverified).
- **G** = `scf_good`: same draw as B without the 3 data-poor genomes (*A. pyrenaica* PHA000543, *A. saussurei*, *A. triflora*), 177 genomes. 72 sites became invariant and were removed to satisfy ASC; the search (`concord_quartet_balance_good`, 107 iterations, 6 h 47 on 120 threads) crashed in the sCF step on the FASTA (`Assertion all_prob_const < 1.0`; cause untested, ASC with constant sites is a suspect). sCF were recomputed with `-te <treefile> -s <varsites.phy> -m GTR+G4+ASC --scfl 100`: **no search** (0 iterations, lnL −41 053 676.53 vs −41 053 676.41 from the search), 4 999 729 columns. A used `-t`, B/G a fixed tree: the protocols differ.
| | A | B | G |
|---|---|---|---|
| Branches | 177 | 177 | 174 |
| Mean `sN` | 3 219 | 13 163 | 13 396 |
| Branches with `sN` < 100 / < 1 000 | 13 / 50 | 0 / 5 | 0 / 2 |
| Median sCF | 44.2 | 42.9 | 42.8 |
| Branches with sCF < 40 / ≥ 70 | 74 / 28 | 67 / 26 | 69 / 25 |
| *Primula* and *Androsace* monophyletic | no / no | no / no | yes / yes |
**Sampling effect (A → B, same 180 genomes).** `sN` ×4.5 and the scarcity is gone. Sites do not become more concordant: on the 113 shared bipartitions (of 177) median sCF 48.0 → 47.3 (44 up, 69 down). Robinson-Foulds 128: the draw rearranged the tree deeply. Consistent with the earlier reading that the intra-*Androsace* conflict is not a matter of site count; the number of branches with sCF < 40 % stays around 70.
**Interface nodes.** Of the old nodes 209/211/212/216/217 (`sN` 17 / 226 / 184 / 7 / 7) none exists in G. Node 208 survives with `sN` 190 → 5 218 (sCF 54.5 → 37.5). Six mixed clades remain in G (83 to 72 tips, 50 *Primula* + 22-33 *Androsace*), all with `sN` > 3 000.
**Effect of the 3 genomes (B → G, same draw).** RF 16 (166 of 174 shared), median sCF 43.2 → 43.6, median `sN` unchanged. In B, *A. pyrenaica* and *A. triflora* sit inside *Primula* (a clade of 52 tips = 50 *Primula* + these 2; each has a *Primula* sister, sCF 92 % and 52 %, `sN` 556 and 153); *A. saussurei* stays with *Androsace*. Removing them restores both monophylies and changes nothing else measurably: support for "too little data to place them" (data-poor section), not for contamination, though the latter is not excluded.
**Where A and G disagree (2026-10-16).** On the 177 common tips: 124 shared bipartitions, 50 only in A, 50 only in G. Differing bipartitions are weakly supported in both trees (median sCF 35.7 in A-only, 37.7 in G-only, versus 47.5-48.1 on the shared ones), so the disagreement sits where support was already low. Differences are mostly local: the closest bipartition of the other tree is 1 tip away for 25 of 50 A-only and 21 of 50 G-only splits (≤ 2 tips: 37 and 30); only 8 and 9 are 5+ tips apart. Greedy removal of the tip that lowers RF most: the first (*P. auricula* subsp. *serratifolia*) takes RF from 100 to 86, then about 2 per tip (*A. maxima*, *P. stricta* var. *obesior*, then ordinary tips); the top three are data-poor genomes. A bipartition separates a data-poor genome (the 5 left after the 3 removals) in 1 of 124 shared splits but in 23 of 50 A-only and 10 of 50 G-only splits; removing those 5 genomes gives RF 72, against 96.6 (range 90-100, 40 draws) for 5 random tips. So about 28 of the 100 differences trace to data-poor genomes, and the remaining 72 do not. Backbone: A has a clade of 50 *Primula* + 25 *Androsace* (sCF 76, `sN` 453), G one of 50 + 26 (sCF 73, `sN` 6 677): one tip apart. Genuine conflict: a clade of 72 *Androsace* exists in both trees (A: sCF 34, `sN` 17 127; G: sCF 38, `sN` 49 473) with 16 different tips, i.e. a topological disagreement carried by tens of thousands of sites in each, not a scarcity effect; consistent with the intra-*Androsace* conflict above.
**Not established.** Separate contribution of entropy / quartet / pair balance (no ablation); stability of the tree between two independent draws (one draw per setting; A → B shows the topology moves with the draw); whether the quartet-informative restriction biases branch lengths (ASC corrects for variable-only ascertainment, not for requiring two states each in two genomes; tree length 22.5 is not interpretable as substitutions/site).
## Open questions ## Open questions
+107
View File
@@ -0,0 +1,107 @@
# VCF pseudo-alignment
[`phylo --vcf`](../usage/phylo.md) writes the sites of a SNP pseudo-alignment as a VCF file, `<prefix>_alignment.vcf`. This page describes exactly what that file contains, because the k-mer families it is built from are not ordinary genomic variants and a few conventions follow from that.
## Why a VCF, and what a site is
The pseudo-alignment (see [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md)) has one column per *family*: the up to four k-mers that share all their bases except the central one. A genome that carries one of these k-mers shows a particular base at the centre, and the columns of the alignment are those central bases, genome by genome. The FASTA output keeps this as a matrix of letters. The VCF output lists the same columns as sites with alleles and genotypes, which is the form that variant-oriented tools (`bcftools`, population-genetics software, genotype viewers) expect.
Nothing here comes from a reference genome. The file has no real coordinates, and the sites are not located on a chromosome: the positions below are labels that make the file valid and reproducible, not map positions.
## Producing the file
```
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
```
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, together with an already-built sibling annex (see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
## Layout
An illustrative excerpt (columns are tab-separated in the file):
```
##fileformat=VCFv4.5
##source=obikmer
##contig=<ID=obikmer_0_0>
##contig=<ID=obikmer_0_1>
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype (diploid)">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT idx0 idx1 idx2 idx3
obikmer_0_0 440 CGATTNGTTAA A G . . . GT 0/0 1/1 0/0 0/0
obikmer_0_0 1511 CACGANTGTAG A G,T . . . GT 0/0 0/0 1/1 2/2
obikmer_0_1 9 TTACGNAGCTA C * . . . GT 0/0 0/0 1/1 0/0
```
There is one sample column per genome that takes part in the draw, named after the genome label, in the index's own order. A genome removed with `--exclude-genome` or `--min-shared-family` has no column at all. `QUAL`, `FILTER` and `INFO` are always `.`: obikmer applies no variant filter and attaches no annotation, and leaving them empty keeps a file of millions of sites compact.
## Coordinates: `CHROM` and `POS`
An index is organised in partitions, and each partition holds one or more layers of k-mers (see [Index construction and on-disk layout](index_layout.md)). A family always belongs to exactly one layer of one partition, and the families of a layer are visited in a fixed order. The VCF uses that structure as coordinates:
- `CHROM` is `obikmer_<partition>_<layer>`, one virtual contig per layer of each partition in which at least one site was drawn;
- `POS` is the rank of the family inside that layer, starting at 1.
For a given index and a given sibling annex, a family therefore always receives the same `CHROM` and `POS`, whatever the draw, the options or the number of sites requested. Two runs on the same index can be compared position by position. This stability is lost if the sibling annex is rebuilt, or if the index is rebuilt, merged or filtered, because the families are then enumerated again. Positions are never comparable between two different indexes; use `ID` for that. Even rebuilding an index from the very same genomes may enumerate the families in another order: on a small test index built twice from identical input, the family `CGATTNGTTAA` had `POS` 440 the first time and 371 the second, with the same `ID`.
The file is sorted by partition, then layer, then position, which is also the order of the `##contig` lines, so tools that require a sorted file accept it directly. Positions are sparse: only the drawn families are written.
## Identifier: `ID`
The `ID` column holds a label for the family that does not depend on the index. It is built from one of the family's k-mers by replacing its central base with `N` (for example `CGATTNGTTAA` for k = 11), after choosing, among the family's possible orientations, the one that sorts first. All the members of a family, whichever strand they were read on, give the same label.
This is what makes two files comparable even when they come from different indexes: the same family has the same `ID` in both, while its `CHROM` and `POS` will generally differ. It is also unique within a file, since two different families never share a label.
## Alleles: `REF` and `ALT`
The alleles are the bases carried at the centre of the family's k-mers, as in the FASTA. Two points are worth knowing.
First, the base is the central base of the member's *canonical* form (the smaller of the k-mer and its reverse complement). That is not necessarily the strand of the `ID` label, so the letters should not be matched to the `ID` sequence by position; they are labels for the allele, not a reading of that sequence.
Second, `REF` carries no meaning of reference: it is simply the most frequent allele among the genomes of the draw, each allele copy counted once, with the alphabetical order A < C < G < T deciding ties. `ALT` lists the remaining observed bases in the order A, C, G, T, separated by commas. There is no option to choose a reference genome.
### Absence: the `<ABS>` allele
A genome may carry none of the family's k-mers. Without `--free-loss` this absence is treated as a real state (the family was lost, or is absent from that genome), which the FASTA shows as `-`. In the VCF it becomes one more allele, the symbolic allele `<ABS>`, listed last in `ALT` and declared in the header by `##ALT=<ID=ABS,Description="Family absent from the genome">`. A genome without the family receives the genotype of `<ABS>`; for a site with `ALT` equal to `G,T,<ABS>` that is `3/3`. A site that differs between genomes only by presence versus absence has `ALT` equal to `<ABS>`.
The VCF specification gives the plain character `*` a meaning of its own (an allele missing because of an overlapping deletion described by another record), which is not the situation here, so it is deliberately not used. `<ABS>` is a free-form symbolic identifier, outside the reserved structural-variant names (`DEL`, `INS`, `DUP`, `INV`, `CNV`) and outside `<*>`, which belongs to reference blocks. Tools that only look at sequence alleles treat the site as having an alternative that is not a base.
With `--free-loss`, absence is declared missing data instead: `<ABS>` never appears, and the genomes concerned get a missing genotype.
## Genotypes
Every genome gets one `GT` value per site. The ploidy of these values is a convention chosen from the options, and the header says which (`Genotype (diploid)` or `Genotype (haploid)`).
| Cell in the alignment | Diploid (default, `--only-diploid`) | Haploid (`--no-ambiguity`) |
|---|---|---|
| one base `X` | `X/X`, for example `0/0` or `1/1` | `X`, for example `0` |
| two bases (IUPAC `R Y S W K M`) | the heterozygote, for example `0/1` | does not occur |
| three or four bases (`B D H V N`) | `./.` (not expressible) | does not occur |
| absent family (`<ABS>`) | `<ABS>/<ABS>`, for example `3/3` | the index of `<ABS>` |
| missing (`?`) | `./.` | `.` |
The two allele indices of a diploid genotype are always written in increasing order (`0/1`, never `1/0`), and the separator is `/` because no phasing information exists.
Why diploid by default? A genome can carry several forms of the same family, which the alignment records as an IUPAC ambiguity code. A diploid genotype can express up to two forms, and `--only-diploid` removes the genomes that would need more (three or four forms) before the sites are selected, so that no `./.` comes from them. `--no-ambiguity` goes further and treats every such genome as missing, which makes all genotypes haploid.
This is a convention, not a measurement. The file records which forms a genome carries, not how many copies of each: `A/A/G` and `A/G/G` both give the code `R` and the genotype `0/1`. Likewise, a genome with two forms may be heterozygous at one locus, or it may carry two similar regions of its genome (paralogues, repeats); the file cannot tell these apart. Do not read the genotypes as calls from sequencing reads.
## Which sites are written
Only the families that survive the draw and its masking appear. Within those:
- a site needs at least one genome with a base or (in diploid mode) a two-base code, because `REF` is a base; a site where every genome is missing, absent or has an ambiguity code too wide for the mode is left out of the VCF while remaining in the FASTA;
- the log line at the end of the run gives the number of sites written out of the number drawn, so the difference is visible.
A site is selected because its genomes show at least two distinct states, and absence counts as a state. A site that differs between genomes only by presence versus absence is therefore a legitimate site, written with `ALT` equal to `<ABS>`.
## Using the file with other tools
- The contigs are virtual and have no declared length, and the file refers to no reference. Tools that need one (annotation, liftover) cannot use it; tools that only read sites and genotypes (`bcftools view`, `query`, `stats`, population-genetics software, PLINK-style converters) can.
- If a tool rejects multiallelic sites or the `<ABS>` allele, `bcftools view -m2 -M2` keeps biallelic sites only, and `bcftools view -e 'ALT="<ABS>"'` removes the sites involving absence. The file as written was read by `bcftools` and indexed by `tabix` without complaint on a small test index, in both diploid and haploid modes. `bcftools stats` counts the sites with `<ABS>` among the "others" (symbolic alleles) rather than as SNPs.
- If a tool expects haploid data, run `phylo` with `--no-ambiguity` rather than converting the genotypes afterwards.
## See also
- [`phylo`](../usage/phylo.md): all the options that shape the draw.
- [Sampling design](../theory/phylogeny/snp_based/sampling_design.md): how the sites are drawn.
- [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md): what a family is.
+13 -1
View File
@@ -20,10 +20,22 @@ obikmer merge -o OUTPUT SOURCE... [OPTIONS]
| `--force` | off | Overwrite an existing output directory | | `--force` | off | Overwrite an existing output directory |
| `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts | | `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts |
| `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing | | `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing |
| `--budget-fraction` | `0.5` | Fraction of available RAM reserved as the memory budget for parallel partition merging | | `--dense` | off | Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see [pack](pack.md)) |
## Behaviour ## Behaviour
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources. The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources.
By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies. By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies.
The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; `--dense` keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The [pack](pack.md) page explains the trade-off between the two formats.
## Resource usage
The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.
Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.
## Current limitation
An index produced by `merge` cannot itself be used as a source of a later merge yet: the command stops with a "No such file or directory" error. To add genomes to a collection, merge the original single-genome indexes again, together with the new ones.
+13 -5
View File
@@ -48,13 +48,15 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
| Option | Description | | Option | Description |
|---|---| |---|---|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value | | `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the `--no-ambiguity` / `--only-diploid` convention given: `--entropy` the entropy annex, `--quartet` the quartet annex, `--pair-balance` the pair-weight and family-overlap annexes, `--family-overlap` or `--min-shared-family` the family-overlap annex. For example `--sibling-annex --quartet --only-diploid` prepares exactly the quartet annex of that convention, and `--sibling-annex` alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally | | `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only | | `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for | | `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan | | `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
| `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index | | `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index |
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` | | `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
### `--sibling-stats` ### `--sibling-stats`
@@ -75,11 +77,16 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`. `<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
#### VCF output (`--vcf`)
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
| Option | Default | Description | | Option | Default | Description |
|---|---|---| |---|---|---|
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. | | `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` | | `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code | | `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct called states, each carried by at least two genomes. Which genomes count as carriers follows `--no-ambiguity` | | `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct called states, each carried by at least two genomes. Which genomes count as carriers follows `--no-ambiguity` |
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` | | `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) | | `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
@@ -131,7 +138,7 @@ With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`)
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. Absence is never a state here: a genome that does not carry the family does not count, with or without `--free-loss`. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment. A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. Absence is never a state here: a genome that does not carry the family does not count, with or without `--free-loss`. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` convention, in one pass shared with the other annexes, and built for both conventions by `--sibling-annex`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families. The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
### Describing a draw: `--sample-stats`, `--sample-plan` ### Describing a draw: `--sample-stats`, `--sample-plan`
@@ -155,7 +162,7 @@ A call is any state other than the empty one: absence is never a call, with or w
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs. Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` convention, after the family-overlap matrix it depends on, and by `--sibling-annex` for both conventions). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first. Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families ### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
@@ -163,7 +170,7 @@ By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data. The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
The entropy that weights the draw follows `--no-ambiguity`, so that it describes the same genomes as the alignment. Without the flag, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With the flag, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). The two entropies are saved separately: switching `--no-ambiguity` builds the other one once, without discarding the first, and `--sibling-annex` builds both. The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
## Sankoff calibration and phylogenetic exports ## Sankoff calibration and phylogenetic exports
@@ -222,7 +229,8 @@ CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `g
| `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers | | `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers |
| `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes | | `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes |
| `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept | | `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept |
| `<prefix>_alignment.fasta` | `--pseudo-alignment` | FASTA | SNP-only pseudo-alignment, IUPAC-coded | | `<prefix>_alignment.fasta` | `--pseudo-alignment` or `--fasta` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
| `<prefix>_alignment.vcf` | `--vcf` | VCF | Same sites as the FASTA, with stable virtual coordinates and a k-mer-based `ID` |
### Sankoff calibration and exports ### Sankoff calibration and exports
+2 -2
View File
@@ -19,7 +19,7 @@ INDEX_PATH="${RUN_DIR}/specimen_index_count/${SPECIMEN}"
STATS_DIR="${RUN_DIR}/stats/indexing_count" STATS_DIR="${RUN_DIR}/stats/indexing_count"
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats" STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
mkdir -p "${STATS_DIR}" mkdir -p "${STATS_DIR}" "$(dirname "${INDEX_PATH}")"
r1="${READS_DIR}/reads_R1.fastq.gz" r1="${READS_DIR}/reads_R1.fastq.gz"
r2="${READS_DIR}/reads_R2.fastq.gz" r2="${READS_DIR}/reads_R2.fastq.gz"
@@ -41,7 +41,7 @@ trap 'rm -f "${STDERR_LOG}"' EXIT
--label "${SPECIMEN}" \ --label "${SPECIMEN}" \
--meta "species=${species}" \ --meta "species=${species}" \
"${r1}" "${r2}" \ "${r1}" "${r2}" \
2>"${STDERR_LOG}" 2>"${STDERR_LOG}" || { cat "${STDERR_LOG}" >&2; exit 1; }
cat "${STDERR_LOG}" >&2 cat "${STDERR_LOG}" >&2
+2 -2
View File
@@ -19,7 +19,7 @@ INDEX_PATH="${RUN_DIR}/specimen_index_presence/${SPECIMEN}"
STATS_DIR="${RUN_DIR}/stats/indexing_presence" STATS_DIR="${RUN_DIR}/stats/indexing_presence"
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats" STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
mkdir -p "${STATS_DIR}" mkdir -p "${STATS_DIR}" "$(dirname "${INDEX_PATH}")"
r1="${READS_DIR}/reads_R1.fastq.gz" r1="${READS_DIR}/reads_R1.fastq.gz"
r2="${READS_DIR}/reads_R2.fastq.gz" r2="${READS_DIR}/reads_R2.fastq.gz"
@@ -40,7 +40,7 @@ trap 'rm -f "${STDERR_LOG}"' EXIT
--label "${SPECIMEN}" \ --label "${SPECIMEN}" \
--meta "species=${species}" \ --meta "species=${species}" \
"${r1}" "${r2}" \ "${r1}" "${r2}" \
2>"${STDERR_LOG}" 2>"${STDERR_LOG}" || { cat "${STDERR_LOG}" >&2; exit 1; }
cat "${STDERR_LOG}" >&2 cat "${STDERR_LOG}" >&2
+1
View File
@@ -64,6 +64,7 @@ nav:
- Predicates and taxonomy paths: usage/predicates.md - Predicates and taxonomy paths: usage/predicates.md
- Formats: - Formats:
- Index construction and on-disk layout: formats/index_layout.md - Index construction and on-disk layout: formats/index_layout.md
- VCF pseudo-alignment: formats/vcf.md
- Architecture notes: architecture.md - Architecture notes: architecture.md
watch: watch:
+2
View File
@@ -56,6 +56,8 @@ nav:
- PersistentBitVec: implementation/persistent_bit_vec.md - PersistentBitVec: implementation/persistent_bit_vec.md
- Merge command: implementation/merge.md - Merge command: implementation/merge.md
- Merge parallelism & memory: implementation/merge_parallelism.md - Merge parallelism & memory: implementation/merge_parallelism.md
- "Merge mmap audit (2026-11)": implementation/merge_mmap_audit.md
- "Merge scaling (open)": implementation/merge_scaling.md
- Kmer filtering: implementation/filtering.md - Kmer filtering: implementation/filtering.md
- Select command: implementation/select.md - Select command: implementation/select.md
- obitaxonomy crate: implementation/obitaxonomy.md - obitaxonomy crate: implementation/obitaxonomy.md
+150
View File
@@ -0,0 +1,150 @@
# Prompt — refonte mmap de `obikmerge` (plans A puis C)
> À coller tel quel au début d'une nouvelle session Claude Code ouverte à la racine du dépôt obikmer.
> Conçu pour être suivi par **Sonnet** : toutes les décisions d'architecture sont fixées ci-dessous. Le plan B de l'audit (ouverture « slots only » dans `obikindex`) est **hors périmètre** de cette session (voir étape 5).
---
## Contexte
Lis d'abord `DevDocMD/implementation/merge_mmap_audit.md`. C'est le diagnostic complet, et ce prompt en implémente le plan.
En résumé : `obikmerge::partition_merge::merge_partition` ouvre, pour **chaque** partition, **toutes** les couches de **toutes** les sources (4 mmaps chacune). Elle crée aussi **toutes** les colonnes destination d'avance, en fichiers creux mappés en écriture (`set_len` + `MmapMut`). Avec 186 sources, ça fait environ 1 100 mappings par partition, ce qui épuise `vm.max_map_count` en production (pas d'accès root pour le relever). L'écriture via `MmapMut` dans des fichiers creux transforme aussi tout refus d'allocation disque en SIGBUS.
Objectif : le nombre de mappings vivants par partition ne doit plus dépendre du nombre de sources, et l'écriture des colonnes du merge ne doit plus passer par `MmapMut`. **Le résultat sur disque doit rester strictement identique.**
## Règles (non négociables)
- Appelle `mcp__serena__initial_instructions` avant tout travail de code. Utilise Serena (`find_symbol`, `get_symbols_overview`, `find_referencing_symbols`, `replace_symbol_body`, `replace_content`) pour naviguer et éditer le Rust.
- **Aucune commande VCS mutante** (git/jj : commit, add, mv, rm, describe, push…). Lecture seule autorisée.
- **Aucune nouvelle dépendance**, ni changement de version.
- `obikindex` fait autorité : cette session n'y modifie et n'y ajoute rien.
- Pas de fonctions libres ad hoc dans `obikmerge` pour combler un manque d'API. Si une opération générique manque sur un type de `obicompactvec`, ajoute-la comme méthode sur ce type.
- Fichiers source sous ~500 lignes. `partition_merge.rs` en fait déjà 605 : il **doit** être découpé (étape 6).
- **Ne jamais `.collect()` un flux de k-mers ou de unitigs** pour satisfaire une contrainte de trait.
- Échanges en **français**, documentation en **anglais**.
- **Point d'arrêt obligatoire** à la fin de chaque étape. Résume ce qui a été fait et les résultats de tests (sortie réelle), puis **attends mon feu vert** avant l'étape suivante.
- Si un test ou l'oracle échoue et que tu ne trouves pas la cause en une tentative, **arrête-toi et expose le problème**, sans contournement.
## Étape 1 — Oracle de non-régression (avant toute modification de code)
### 1a. Oracle synthétique (rapide, couvre comptage et merge en deux temps)
1. Lis `scripts/smoke_test_index.sh` (modèle de style, options de `obikmer index`) et les `--help` de `obikmer index`, `merge` et `dump`.
2. Écris `scripts/smoke_test_merge.sh`, sur le même modèle :
- génère 4 FASTA aléatoires déterministes A, B, C, D, petits, avec **des fragments partagés** entre eux. Le merge doit à la fois retrouver des k-mers déjà présents dans la destination (chemin « couche existante ») et en créer de nouveaux (chemin « nouvelle couche ») ;
- indexe chacun deux fois : présence (défaut) et comptage (`--with-counts`). Plusieurs partitions (`-p 4`) ;
- scénario 1 : `merge A B C D` (présence), puis la même chose en comptage ;
- scénario 2 : `merge A B` → AB, puis `merge AB C D`. Ce scénario couvre une source multi-génome et une destination à plusieurs couches ;
- scénario 3 : comme le scénario 1, avec `--dense` ;
- pour chaque sortie : `obikmer dump` → `sort` → un fichier texte ;
- deux modes : `--record DIR` écrit les sorties triées dans `DIR`, `--check DIR` les régénère et fait un `diff` avec `DIR`. Affiche PASS/FAIL clairement, avec un code de sortie cohérent.
3. Lance `--record benchmark/run/merge_oracle/synthetic` **avec le code actuel, non modifié**. Vérifie que les fichiers ne sont pas vides et que les colonnes de génomes ont du sens.
4. Lance aussitôt `--check` sur le même code pour vérifier que le script est déterministe.
### 1b. Oracle sur données réelles : `benchmark/`
`benchmark/run/specimen_index_presence/` contient 20 index présence construits à partir de vrais génomes (bactéries + une levure, lectures simulées). Lis `benchmark/README.md` et `benchmark/merge_presence.sh` (le merge de référence y utilise `--force --force-presence`).
État constaté au 2026-11-17, à revérifier :
- `specimen_index_count/` est **vide** : pas de piste comptage sans relancer `gmake index_count` (GNU make ≥ 4.3 : `gmake` sur macOS). **Demande-moi** avant de le lancer, car ça prend du temps ;
- `reference_index/` (`.npz`) est **absent** : `verify_merge_*.py` est inutilisable, ne t'appuie pas dessus ;
- `global_index_presence/` a été produit par un binaire antérieur au code actuel (release du 2026-10-22) : **ce n'est pas la référence**.
Procédure :
1. Recompile le binaire release à partir du **code actuel non modifié** (`cargo build --release` dans `src/`). Ne touche pas aux artefacts existants de `benchmark/run/`.
2. Lance le merge des 20 index avec les mêmes options que `merge_presence.sh`, mais vers un répertoire **dédié**, par exemple `benchmark/run/merge_oracle/bench/merged`, jamais `benchmark/run/global_index_presence`. Garde le log stderr : le rapport de stages (wall et RSS de `merge_partitions`) sert de référence de performance.
3. `obikmer dump` du résultat → `sort` → compresse (`zstd`) + `sha256sum`, dans `benchmark/run/merge_oracle/bench/`. La sortie peut être volumineuse : compare d'abord les hash, et ne fais un `diff` que s'ils diffèrent.
4. Contrôle croisé informatif : fais la même chose (dump trié + hash) sur `benchmark/run/global_index_presence`. Si les hash diffèrent, **signale-le sans conclure** : l'écart peut venir d'évolutions légitimes du code depuis octobre.
5. Écris ces étapes dans un script `scripts/bench_merge_oracle.sh` avec deux modes : `--record DIR` et `--check DIR`. En mode `--check`, il affiche aussi le temps de `merge_partitions` avant/après.
Après chaque étape de code (3, 4, 6), les **deux** oracles doivent passer : `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check`. Si `merge_partitions` est plus de 20 % plus lent qu'avant sur le benchmark, signale-le au point d'arrêt.
**Point d'arrêt.**
## Étape 2 — `MatrixBuilder::n_cols` (petit prérequis, `obicompactvec`)
Ajoute `pub fn n_cols(&self) -> usize` à `obicompactvec::MatrixBuilder` (`src/obicompactvec/src/matrix_builder.rs`), en déléguant au builder Bit ou Int. Si `PersistentCompactIntMatrixBuilder` n'expose pas `n_cols()`, ajoute-le sur ce type. Ajoute un test unitaire. `cargo test -p obicompactvec`.
Pas de point d'arrêt séparé : enchaîne sur l'étape 3.
## Étape 3 — Plan A : passe 2 par groupes de sources (`obikmerge`)
Fichier : `src/obikmerge/src/partition_merge.rs`, fonction `merge_partition`.
### 3a. Passe 1 sans ouvrir les couches sources
Construis la liste des chemins `unitigs.bin` sans `open()`. Pour chaque source et chaque `l in 0..partition.n_layers()`, prends `KmerLayer::new(&partition, l).unitigs_path()` (ce chemin est valide sur la variante `Empty`). Garde le filtre `exists()`. Ne touche pas au reste de la passe 1 (`build_graph`, `materialize_layer`, ouverture de `new_mphf`).
### 3b. Nouvelle couche : matrice initialisée seule
Si `any_new`, crée la matrice de la nouvelle couche avec `MatrixBuilder::new(presence, n_new, data_dir)`, ajoute les `n_dst_genomes` colonnes absentes, puis **ferme**. Les colonnes sources seront ajoutées par `resume`, groupe par groupe.
### 3c. Boucle sur les groupes
- `const SRC_GROUP: usize = 2;` (aligné sur `max_open`), avec un commentaire qui explique pourquoi.
- Itère sur `sources.chunks(SRC_GROUP)`, **dans l'ordre**, en tenant un `col_base` = nombre de colonnes sources déjà ajoutées avant ce groupe.
- Pour chaque groupe :
1. ouvre les couches de ses sources pour cette partition (`partition.layer(l)?.open()` puis `Arc::new`), comme aujourd'hui mais **seulement pour ce groupe** ;
2. `group_n` = somme des `src_n` du groupe ;
3. pour chaque couche dst existante, puis pour la nouvelle couche si `any_new` : `MatrixBuilder::resume`, vérifie `n_cols() == n_dst_genomes + col_base`, puis `add_col()` × `group_n` ;
4. construis les `pass2_items` du groupe. Le `col_offset` est **local au groupe** (0..group_n) ;
5. lance le pipeline de passe 2 existant, inchangé dans sa logique (Flat → Transform → Sink), sur ces items et ces builders. Les vecteurs `exist_sink`/`new_sink` ne contiennent que les colonnes du groupe ;
6. vérifie l'erreur de passe 2, ferme toutes les `ColBuilder` puis tous les `MatrixBuilder` du groupe ;
7. **drop** les couches du groupe avant de passer au suivant ;
8. `col_base += group_n`.
- L'initialisation `init_presence_matrix` (premier merge en présence) reste **avant** la boucle, inchangée.
- `dst_layers` (ouverts une seule fois) et `new_mphf` restent partagés par tous les groupes.
Invariant à documenter dans un commentaire : les colonnes sont ajoutées dans l'ordre des sources, donc la colonne absolue `n_dst_genomes + col_base + offset_local + g` est la même qu'avant la refonte.
### 3d. Vérification
`cargo test -p obikmerge`, `cargo clippy -p obikmerge`, puis `scripts/smoke_test_merge.sh --check benchmark/run/merge_oracle/synthetic` et `scripts/bench_merge_oracle.sh --check benchmark/run/merge_oracle/bench` (recompile le release avant). Les deux **doivent** passer sans aucun diff.
**Point d'arrêt.**
## Étape 4 — Plan C : colonnes construites en mémoire (`obicompactvec`)
But : plus aucun `MmapMut` ni fichier creux pour les colonnes **remplies** par le merge.
1. Lis `PersistentBitVecBuilder` (`bitvec.rs` : `new`, `set`, `close`), `PersistentCompactIntVecBuilder` (`builder.rs` : `new`, `set`, `close`), `format.rs` (`finalize_pciv`, `HEADER_SIZE`) et `mmap_file.rs` (`building_path`, `publish_built`). Note **exactement** les octets écrits à la fermeture : en-tête, données, overflow, `.building` puis `rename`.
2. Ajoute deux types :
- `MemBitVecBuilder` (dans `bitvec.rs`, ou un nouveau `mem_builder.rs` si `bitvec.rs` dépasse 500 lignes) : `words: Vec<u64>`, `n`, `path` ; `new(n, path)`, `set(slot, bool)`, `close() -> io::Result<()>` ;
- `MemCompactIntVecBuilder` : `primary: Vec<u8>`, `overflow: HashMap<usize, u32>`, `n`, `path` ; `new`, `set(slot, u32)` (même sémantique sentinelle 255 que le builder persistant), `close()`.
`close()` écrit avec un `BufWriter` dans `building_path(path)`, produit le **même fichier, octet pour octet**, que le builder persistant, fait `flush` + `sync_all`, puis `publish_built(path)`. Réutilise les fonctions de format existantes plutôt que de dupliquer la mise en page de l'en-tête. S'il faut factoriser l'écriture de l'en-tête ou de l'overflow pour la partager, fais-le dans `format.rs` / `mmap_file.rs`.
3. Dans `matrix_builder.rs` : ajoute les variantes `ColBuilder::MemBit(MemBitVecBuilder)` et `ColBuilder::MemInt(MemCompactIntVecBuilder)` (`set_val` et `close` les gèrent), plus `MatrixBuilder::add_mem_col() -> io::Result<ColBuilder>`. Ce dernier incrémente `n_cols` exactement comme `add_col`. Si les builders de matrice sous-jacents n'exposent ni le chemin de la prochaine colonne ni l'incrément, ajoute-leur une méthode `add_mem_col` symétrique de `add_col`. Le nommage des colonnes reste privé à la crate.
4. Tests unitaires (`src/obicompactvec/src/tests/`) : pour n ∈ {0, 1, 63, 64, 65, 10 000} et des valeurs pseudo-aléatoires (incluant des comptes ≥ 255), comparer **octet par octet** le fichier produit par le builder mémoire et celui du builder persistant. Vérifier aussi qu'une matrice construite avec `add_mem_col` se relit via `PersistentBitMatrix::open` / `PersistentIntMatrix::open`.
5. Dans `merge_partition`, remplace `add_col()` par `add_mem_col()` pour les colonnes sources (couches existantes et nouvelle couche). `add_absent_col` reste tel quel : on n'écrit jamais dans son mapping, donc aucun risque de SIGBUS.
6. Ajoute un commentaire sur le coût mémoire : `SRC_GROUP × src_n × (L_dst + 1) × (n/8 ou n)` octets au plus par partition.
7. `cargo test -p obicompactvec -p obikmerge`, clippy, puis `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check` (release recompilé).
`obikselect` et `obikfilter` ne sont **pas** migrés dans cette session.
**Point d'arrêt.**
## Étape 5 — Plan B : hors périmètre
**Ne pas implémenter.** Le plan B (ouverture « slots only » des couches dans `obikindex`) a été tranché par le développeur : ce sera un chantier séparé (forme retenue : composition, voir la section « Plan B » de `merge_mmap_audit.md`). Après l'étape 3, son gain en mmaps pour le merge est négligeable (~8 → ~2 par partition). Il touche en revanche le chemin de requête d'`obikindex` et concerne trois crates.
Les couches sources de la passe 2 continuent donc d'être ouvertes par `partition.layer(l)?.open()`, mais seulement groupe par groupe (étape 3). Ne crée aucun type ni méthode « lookup only » dans `obikindex` ou `obikmerge`.
## Étape 6 — Découpage, doc, finition
1. Découpe `partition_merge.rs` en module `partition_merge/` (par exemple `mod.rs` pour l'orchestration, `pass1.rs`, `pass2.rs`, `tests.rs`), chaque fichier sous 500 lignes, sans changement de comportement.
2. Supprime tout code devenu mort : `find_referencing_symbols` sur ce que la refonte a remplacé. Signale tout ce qui est devenu injustifié plutôt que de le garder par inertie.
3. `cargo test --workspace`, `cargo clippy --workspace`, `cargo check --workspace --features numa` si la feature existe, `scripts/smoke_test_index.sh`, `scripts/smoke_test_merge.sh --check` et `scripts/bench_merge_oracle.sh --check` (release recompilé ; reporte le temps de `merge_partitions` avant/après).
4. Documentation :
- `DevDocMD/implementation/merge_mmap_audit.md` : passe le statut à « implemented », en indiquant ce qui a été fait et les écarts éventuels au plan ;
- `DevDocMD/implementation/merge.md`, section « Algorithm » : décris le traitement par groupes de sources et les colonnes en mémoire ;
- vérifie si `UserDocMD/` décrit le comportement mémoire du merge, et le cas échéant mets-le à jour, en prose, sans code Rust ;
- mets à jour la `nav:` des `mkdocs*.yml` si un fichier a été ajouté, puis lance `qmd embed`.
5. Rapport final : liste des fichiers modifiés, résultats de tests, et estimation du nombre de mmaps par partition avant/après (calculée, marquée comme estimée). Termine par la commande de mesure à lancer en production, qui ne demande pas root :
```bash
PID=$(pgrep -f 'obikmer merge'); while sleep 60; do
echo "$(date +%T) maps=$(wc -l < /proc/$PID/maps) threads=$(ls /proc/$PID/task | wc -l)"; done
```
Ne commite rien : c'est moi qui le ferai.
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env bash
# bench_merge_oracle.sh — golden-output + timing oracle for `obikmer merge`
# on the real-data benchmark (benchmark/run/specimen_index_presence, 20
# presence indexes built from real genomes).
#
# Companion of smoke_test_merge.sh for refactors of obikmerge (see
# DevDocMD/implementation/merge_mmap_audit.md). Same merge options as
# benchmark/merge_presence.sh (--force --force-presence), but the output goes
# to a dedicated directory, never to benchmark/run/global_index_presence.
#
# What it does:
# 1. builds the release `obikmer`
# 2. merges every specimen presence index (sorted, as merge_presence.sh)
# and keeps the stderr log (stage report: wall/RSS of merge_partitions)
# 3. `obikmer dump --debug` of the result → header first, rows sorted
# (LC_ALL=C) → sha256 of that sorted text, plus a zstd copy
# 4. --record DIR: writes DIR/merged (the merged index), DIR/merge.stderr.log, DIR/dump.sha256,
# DIR/dump.sorted.csv.zst, and (informative cross-check)
# DIR/global_index_presence.sha256 = same dump hash computed on the
# existing benchmark/run/global_index_presence (older binary — a
# difference there is reported, not judged)
# --check DIR: merges into a scratch directory, compares the hash with
# DIR/dump.sha256 (full diff against the zstd copy only on mismatch),
# and prints merge_partitions wall/RSS and pack wall before → after
#
# Usage:
# scripts/bench_merge_oracle.sh (--record DIR | --check DIR) [--keep]
set -euo pipefail
KEEP=0
MODE=""
DIR=""
while [ $# -gt 0 ]; do
case "$1" in
--record) MODE=record; DIR="$2"; shift 2 ;;
--check) MODE=check; DIR="$2"; shift 2 ;;
--keep) KEEP=1; shift ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
done
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [--keep]" >&2; exit 2; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
RUN_DIR="$REPO_ROOT/benchmark/run"
IDX_DIR="$RUN_DIR/specimen_index_presence"
GLOBAL_OLD="$RUN_DIR/global_index_presence"
BIN="$REPO_ROOT/src/target/release/obikmer"
mkdir -p "$DIR" "$RUN_DIR/merge_oracle"
# Scratch space on the same file system as the benchmark data (merge's
# bootstrap hard-links from its base source; sort needs room for temp files).
WORK="$(mktemp -d "$RUN_DIR/merge_oracle/.work.XXXXXX")"
cleanup() {
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
}
trap cleanup EXIT
fail() {
echo "FAIL: $1" >&2
exit 1
}
# ── 1. build ────────────────────────────────────────────────────────────────
if ! (cd "$REPO_ROOT/src" && cargo build --release -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
cat "$WORK/build.log" >&2
fail "cargo build --release failed"
fi
sorted_dump() { # sorted_dump INDEX OUT_TXT — header first, rows sorted
local raw="$WORK/dump.raw.csv"
"$BIN" dump --debug "$1" > "$raw" 2> "$WORK/dump.log" \
|| { cat "$WORK/dump.log" >&2; fail "obikmer dump $1 exited non-zero"; }
{ head -n 1 "$raw"
tail -n +2 "$raw" | LC_ALL=C sort -S 25% --parallel=8 -T "$WORK"
} > "$2"
rm -f "$raw"
[ "$(head -n 2 "$2" | wc -l)" -eq 2 ] || fail "empty dump of $1"
}
hash_of() { sha256sum "$1" | cut -d' ' -f1; }
stage_field() { # stage_field LOG FIELD [STAGE] — FIELD of the 'done stage=STAGE' line (default merge_partitions)
perl -pe 's/\e\[[0-9;]*m//g' "$1" \
| grep "done stage=${3:-merge_partitions} " \
| sed -n "s/.*$2=\([^ ]*\( [KMG]*B\)\{0,1\}\).*/\1/p" | tail -n 1
}
# ── 2. merge ────────────────────────────────────────────────────────────────
mapfile -t SOURCES < <(find "$IDX_DIR" -mindepth 1 -maxdepth 1 -type d | sort)
[ "${#SOURCES[@]}" -gt 0 ] || fail "no indexes found in $IDX_DIR"
if [ "$MODE" = record ]; then
MERGED="$DIR/merged"
LOG="$DIR/merge.stderr.log"
else
MERGED="$WORK/merged"
LOG="$WORK/merge.stderr.log"
fi
echo "merging ${#SOURCES[@]} presence indexes → $MERGED"
if ! "$BIN" merge --output "$MERGED" --force --force-presence "${SOURCES[@]}" \
> /dev/null 2> "$LOG"; then
tail -n 40 "$LOG" >&2
fail "obikmer merge exited non-zero"
fi
WALL="$(stage_field "$LOG" wall_secs)"
RSS="$(stage_field "$LOG" rss)"
PACK_WALL="$(stage_field "$LOG" wall_secs pack)"
echo "merge_partitions: wall=${WALL}s rss=${RSS} pack: wall=${PACK_WALL}s"
# ── 3. sorted dump + hash ───────────────────────────────────────────────────
DUMP="$WORK/dump.sorted.csv"
sorted_dump "$MERGED" "$DUMP"
H="$(hash_of "$DUMP")"
echo "dump: $(($(wc -l < "$DUMP") - 1)) k-mers, sha256 $H"
# ── 4. record or check ──────────────────────────────────────────────────────
if [ "$MODE" = record ]; then
echo "$H" > "$DIR/dump.sha256"
zstd -q -T0 -f "$DUMP" -o "$DIR/dump.sorted.csv.zst"
if [ -d "$GLOBAL_OLD" ]; then
OLD_DUMP="$WORK/old.sorted.csv"
sorted_dump "$GLOBAL_OLD" "$OLD_DUMP"
HO="$(hash_of "$OLD_DUMP")"
echo "$HO" > "$DIR/global_index_presence.sha256"
if [ "$HO" = "$H" ]; then
echo "cross-check: global_index_presence (older binary) dump is identical"
else
echo "cross-check: global_index_presence (older binary) dump DIFFERS" \
"($(($(wc -l < "$OLD_DUMP") - 1)) k-mers, sha256 $HO) — informative only"
fi
fi
echo "RECORDED → $DIR"
exit 0
fi
[ -f "$DIR/dump.sha256" ] || fail "no reference in $DIR (run --record first)"
REF_H="$(cat "$DIR/dump.sha256")"
REF_WALL="$(stage_field "$DIR/merge.stderr.log" wall_secs)"
REF_RSS="$(stage_field "$DIR/merge.stderr.log" rss)"
REF_PACK="$(stage_field "$DIR/merge.stderr.log" wall_secs pack)"
echo "merge_partitions wall: ${REF_WALL}s → ${WALL}s rss: ${REF_RSS} → ${RSS}"
echo "pack wall: ${REF_PACK}s → ${PACK_WALL}s"
python3 -c "import sys; r,n=float(sys.argv[1]),float(sys.argv[2]); print(f'merge_partitions wall ratio: {n/r:.2f}' + (' (> 1.20: SLOWER THAN 20 %)' if n > 1.2*r else ''))" \
"$REF_WALL" "$WALL" || true
if [ "$H" = "$REF_H" ]; then
echo "PASS: dump identical to reference ($H)"
exit 0
fi
echo "FAIL: dump hash differs (ref $REF_H, now $H)" >&2
zstd -q -d -c "$DIR/dump.sorted.csv.zst" | { diff - "$DUMP" || true; } | head -n 40 >&2
exit 1
+186
View File
@@ -0,0 +1,186 @@
#!/usr/bin/env bash
# smoke_test_merge.sh — end-to-end golden-output oracle for `obikmer merge`
#
# Non-regression oracle for refactors of obikmerge (see
# DevDocMD/implementation/merge_mmap_audit.md). Like smoke_test_index.sh it
# drives the real CLI, never the library APIs.
#
# What it does:
# 1. builds `obikmer` (debug)
# 2. generates (on --record) 4 small deterministic FASTA files A, B, C, D sharing
# fragments, so that merge both hits k-mers already present in the
# destination ("existing layer" path) and creates new ones ("new layer"
# path). D carries a short motif repeated > 255 times so the count track
# exercises the compact-int overflow (counts >= 255).
# 3. on --record, indexes each genome twice (presence, --with-counts) and
# freezes the indexes in DIR/sources; --check reuses them (see below),
# with -p PARTITIONS partitions
# 4. runs the merge scenarios:
# s1_presence / s1_count merge A B C D
# s3_dense merge A B C D --dense (presence)
# Scenario 2 (merge A B -> AB, then merge AB C D: multi-genome source,
# multi-layer destination) is NOT included: the current code cannot
# re-merge a merge output — its matrices are packed (sparse or dense)
# at finalize, and `MatrixBuilder::resume` expects the columnar
# `meta.json` layout (ENOENT). Known bug, to be fixed separately; see
# DevDocMD/implementation/merge_mmap_audit.md, "Known issue".
# 5. dumps every result with `obikmer dump --debug` (partition + layer
# columns included, so a k-mer moving between layers is caught), keeps
# the CSV header first and sorts the rows (LC_ALL=C)
# 6. --record DIR writes the sorted dumps to DIR;
# --check DIR regenerates them and diffs against DIR
#
# Usage:
# scripts/smoke_test_merge.sh (--record DIR | --check DIR) [-p PARTITIONS] [--keep]
set -euo pipefail
K=21
M=9
PARTITIONS=4
KEEP=0
MODE=""
DIR=""
while [ $# -gt 0 ]; do
case "$1" in
--record) MODE=record; DIR="$2"; shift 2 ;;
--check) MODE=check; DIR="$2"; shift 2 ;;
-p) PARTITIONS="$2"; shift 2 ;;
--keep) KEEP=1; shift ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
done
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [-p N] [--keep]" >&2; exit 2; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
WORK="$(mktemp -d -t obikmer_merge_smoke.XXXXXX)"
cleanup() {
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
}
trap cleanup EXIT
fail() {
echo "FAIL: $1" >&2
exit 1
}
# ── 1. build ────────────────────────────────────────────────────────────────
if ! (cd "$REPO_ROOT/src" && cargo build -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
cat "$WORK/build.log" >&2
fail "cargo build failed"
fi
BIN="$REPO_ROOT/src/target/debug/obikmer"
run() { # run LOGNAME CMD... — runs obikmer, dumps its log on failure
local log="$WORK/$1.log"; shift
if ! "$BIN" "$@" > "$log" 2>&1; then
cat "$log" >&2
fail "obikmer $1 exited non-zero"
fi
}
# ── 2. deterministic genomes with shared fragments ──────────────────────────
python3 - "$WORK" <<'EOF'
import random, sys
out = sys.argv[1]
rng = random.Random(20261117)
def rnd(n): return "".join(rng.choice("ACGT") for _ in range(n))
frag = {i: rnd(600) for i in range(5)} # shared fragments F0..F4
motif = rnd(40) # tandem motif (counts >= 255)
genomes = {
"A": [rnd(1500), frag[0], rnd(300), frag[1]],
"B": [frag[1], rnd(1200), frag[2], motif * 3],
"C": [frag[0], rnd(800), frag[2], rnd(400), frag[3]],
"D": [frag[3], rnd(1000), motif * 300, frag[4], rnd(200), frag[0][:250]],
}
for name, parts in genomes.items():
with open(f"{out}/{name}.fasta", "w") as f:
for j, p in enumerate(parts):
f.write(f">{name}_{j}\n{p}\n")
EOF
# ── 3. source indexes: built on --record, frozen in DIR/sources, reused on
# --check. `obikmer index` is not byte-stable run to run (unitig
# segmentation, hence unitigs.bin size, varies — measured 2026-11-17),
# and the merge base is the source with the largest unitigs.bin
# (`choose_base`), so re-indexing on every check would make the merge
# base — and the column order — flip between A and C. Freezing the
# sources makes this an oracle of `merge` alone. Copied (not linked)
# into WORK, since merge's bootstrap hard-links from its base source.
if [ "$MODE" = record ]; then
for G in A B C D; do
run "index_${G}_pres" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
--label "$G" -o "$WORK/$G.pres.idx" "$WORK/$G.fasta"
run "index_${G}_cnt" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
--with-counts --label "$G" -o "$WORK/$G.cnt.idx" "$WORK/$G.fasta"
done
rm -rf "$DIR/sources"
mkdir -p "$DIR/sources"
cp -R "$WORK"/*.idx "$DIR/sources/"
else
[ -d "$DIR/sources" ] || fail "no frozen source indexes in $DIR/sources (run --record first)"
cp -R "$DIR/sources"/*.idx "$WORK/"
fi
# ── 4. merge scenarios ──────────────────────────────────────────────────────
OUT="$WORK/out"
mkdir -p "$OUT"
dump() { # dump NAME INDEX — header first, rows sorted
local raw="$WORK/$1.dump.csv"
if ! "$BIN" dump --debug "$2" > "$raw" 2> "$WORK/dump_$1.log"; then
cat "$WORK/dump_$1.log" >&2
fail "obikmer dump $1 exited non-zero"
fi
{ head -n 1 "$raw"; tail -n +2 "$raw" | LC_ALL=C sort; } > "$OUT/$1.csv"
}
for T in pres cnt; do
run "s1_$T" merge -o "$WORK/s1_$T.idx" \
"$WORK/A.$T.idx" "$WORK/B.$T.idx" "$WORK/C.$T.idx" "$WORK/D.$T.idx"
dump "s1_$T" "$WORK/s1_$T.idx"
done
run s3_dense merge --dense -o "$WORK/s3_dense.idx" \
"$WORK/A.pres.idx" "$WORK/B.pres.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
dump s3_dense "$WORK/s3_dense.idx"
# sanity: no dump may be empty (header + at least one row)
for f in "$OUT"/*.csv; do
[ "$(wc -l < "$f")" -gt 1 ] || fail "empty dump: $(basename "$f")"
done
# ── 5. record or check ──────────────────────────────────────────────────────
if [ "$MODE" = record ]; then
mkdir -p "$DIR"
rm -f "$DIR"/*.csv
cp "$OUT"/*.csv "$DIR"/
echo "RECORDED: $(ls "$OUT" | wc -l | tr -d ' ') dumps → $DIR"
wc -l "$DIR"/*.csv
exit 0
fi
[ -d "$DIR" ] || fail "reference directory not found: $DIR"
STATUS=0
for ref in "$DIR"/*.csv; do
name="$(basename "$ref")"
if [ ! -f "$OUT/$name" ]; then
echo "FAIL $name (not regenerated)"; STATUS=1
elif diff -q "$ref" "$OUT/$name" > /dev/null; then
echo "ok $name"
else
echo "FAIL $name"; { diff "$ref" "$OUT/$name" || true; } | head -20; STATUS=1
fi
done
for new in "$OUT"/*.csv; do
[ -f "$DIR/$(basename "$new")" ] || { echo "FAIL $(basename "$new") (no reference)"; STATUS=1; }
done
if [ "$STATUS" -eq 0 ]; then
echo "PASS: all merge dumps identical to $DIR"
else
echo "FAIL: merge dumps differ from $DIR" >&2
fi
exit "$STATUS"
+1 -2
View File
@@ -1537,7 +1537,7 @@ dependencies = [
[[package]] [[package]]
name = "obikmer" name = "obikmer"
version = "1.3.6" version = "1.4.4"
dependencies = [ dependencies = [
"clap", "clap",
"csv", "csv",
@@ -1582,7 +1582,6 @@ dependencies = [
"obiskio", "obiskio",
"obisys", "obisys",
"rayon", "rayon",
"tempfile",
"tracing", "tracing",
] ]
@@ -3,6 +3,7 @@ use std::io;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use crate::bitvec::PersistentBitVecBuilder; use crate::bitvec::PersistentBitVecBuilder;
use crate::mem_builder::MemBitVecBuilder;
use crate::meta::MatrixMeta; use crate::meta::MatrixMeta;
use crate::tempbitvec::TempBitVec; use crate::tempbitvec::TempBitVec;
use crate::tempintvec::TempCompactIntVec; use crate::tempintvec::TempCompactIntVec;
@@ -43,6 +44,14 @@ impl PersistentBitMatrixBuilder {
PersistentBitVecBuilder::new(self.n, &path) PersistentBitVecBuilder::new(self.n, &path)
} }
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
/// the column is built in RAM and written once at `close`.
pub fn add_mem_col(&mut self) -> MemBitVecBuilder {
let path = col_path(&self.dir, self.n_cols);
self.n_cols += 1;
MemBitVecBuilder::new(self.n, &path)
}
pub fn add_col_ones(&mut self) -> io::Result<PersistentBitVecBuilder> { pub fn add_col_ones(&mut self) -> io::Result<PersistentBitVecBuilder> {
let path = col_path(&self.dir, self.n_cols); let path = col_path(&self.dir, self.n_cols);
self.n_cols += 1; self.n_cols += 1;
+22 -28
View File
@@ -35,9 +35,7 @@ use crate::meta::field;
use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder}; use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
use super::PersistentBitMatrix; use super::PersistentBitMatrix;
use super::col_path; use super::packed::{PackedBitMatrix, pack_bit_matrix};
use super::columnar::ColumnarBitMatrix;
use super::packed::PackedBitMatrix;
use crate::views::nonzero_triples; use crate::views::nonzero_triples;
fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") } fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") }
@@ -615,37 +613,33 @@ impl BitPartials for PersistentSparseBitMatrix {
m m
} }
} }
/// sparse on-disk format in place, mirroring [`super::pack_bit_matrix`]'s
/// convention (old-format files removed only after the new format is /// Converts a layer's presence matrix (columnar `col_*.pbiv` or dense
/// fully written, so a crash mid-conversion leaves the previous, still /// `matrix.pbmx`) to the sparse on-disk format in place. Old-format files
/// valid format rather than a half-written one). Idempotent — does /// are removed only after the new format is fully written, so a crash
/// mid-conversion leaves a previous, still valid format. Idempotent: does
/// nothing if `is_multi.prsb` already exists. /// nothing if `is_multi.prsb` already exists.
/// ///
/// The dense→sparse transpose (`build_from_dense`) only needs `fill_row`, /// Always goes through the dense packed form, even right after a build.
/// which costs the same `O(n_cols)` per row on `Columnar` as on `Packed` /// That costs one extra write and read of the dense matrix, but keeps the
/// (see `ColumnarBitMatrix::fill_row`) — so this reads straight from /// number of mappings at one instead of one per genome column.
/// whichever dense form is already on disk. In particular, when this runs
/// right after a build (the common case: `presence/` is still `Columnar`,
/// `matrix.pbmx` doesn't exist yet), it never materialises a full packed
/// copy of the presence matrix just to read it back once and delete it.
pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> { pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> {
if is_multi_path(dir).exists() { if is_multi_path(dir).exists() {
return Ok(()); return Ok(());
} }
// Columnar → dense packed first, then dense packed → sparse. Opening the
// columnar matrix directly would map every column file at once (one
// mapping per genome, unbounded); `pack_bit_matrix` instead
// concatenates the column files one at a time, unmapped, and leaves a
// single `matrix.pbmx` to read from (one mapping). It is a no-op when
// the layer is already dense-packed, and re-packs a stale `matrix.pbmx`
// that predates newer columnar data. Every intermediate state is a
// valid format, so a crash leaves something this function resumes.
pack_bit_matrix(dir)?;
let packed_path = dir.join("matrix.pbmx"); let packed_path = dir.join("matrix.pbmx");
if packed_path.exists() { let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?);
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?); PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?; drop(dense);
drop(dense); fs::remove_file(&packed_path)
fs::remove_file(&packed_path)?;
} else {
let dense = PersistentBitMatrix::Columnar(ColumnarBitMatrix::open(dir)?);
let n_cols = dense.n_cols();
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
drop(dense);
for c in 0..n_cols { let _ = fs::remove_file(col_path(dir, c)); }
let _ = fs::remove_file(dir.join("meta.json"));
}
Ok(())
} }
+1 -1
View File
@@ -8,7 +8,7 @@ use crate::mmap_file::{building_path, create_mmap_file, open_mmap_file, publish_
use crate::reader::PersistentCompactIntVec; use crate::reader::PersistentCompactIntVec;
use crate::views::{BitSliceIter, BitSliceView, IntSliceView}; use crate::views::{BitSliceIter, BitSliceView, IntSliceView};
const MAGIC: [u8; 4] = *b"PBIV"; pub(crate) const MAGIC: [u8; 4] = *b"PBIV";
// Header: magic(4) + _pad(4) + n(8) = 16 bytes. // Header: magic(4) + _pad(4) + n(8) = 16 bytes.
// Data starts at offset 16, u64-aligned (mmap base is page-aligned, 16 % 8 == 0). // Data starts at offset 16, u64-aligned (mmap base is page-aligned, 16 % 8 == 0).
+48 -38
View File
@@ -1,5 +1,5 @@
use std::fs::OpenOptions; use std::fs::OpenOptions;
use std::io::{self, BufWriter, Seek, SeekFrom, Write as _}; use std::io::{self, BufWriter, Seek, SeekFrom, Write};
use std::path::Path; use std::path::Path;
pub const MAGIC: [u8; 4] = *b"PCIV"; pub const MAGIC: [u8; 4] = *b"PCIV";
@@ -54,53 +54,63 @@ pub fn parse_index_entry(data: &[u8], base: usize, i: usize) -> (usize, usize) {
// Sparse index target: ≤ 32 KB in L1 cache (16 B per entry → 2048 entries). // Sparse index target: ≤ 32 KB in L1 cache (16 B per entry → 2048 entries).
pub const L1_INDEX_ENTRIES: usize = 2048; pub const L1_INDEX_ENTRIES: usize = 2048;
/// Sparse-index `step` and entry count for `n_overflow` overflow entries:
/// no index (`step == 0`) while the overflow section fits the L1 target.
fn pciv_index_layout(n_overflow: usize) -> (usize, usize) {
if n_overflow <= L1_INDEX_ENTRIES {
(0, 0)
} else {
let step = n_overflow.div_ceil(L1_INDEX_ENTRIES);
(step, n_overflow.div_ceil(step))
}
}
/// Write the 40-byte PCIV header for a vector of `n` slots with
/// `n_overflow` overflow entries (index layout derived from it).
pub(crate) fn write_pciv_header<W: Write>(w: &mut W, n: usize, n_overflow: usize) -> io::Result<()> {
let (step, n_index) = pciv_index_layout(n_overflow);
w.write_all(&MAGIC)?;
w.write_all(&[0u8; 4])?;
w.write_all(&(n as u64).to_le_bytes())?;
w.write_all(&(n_overflow as u64).to_le_bytes())?;
w.write_all(&(n_index as u64).to_le_bytes())?;
w.write_all(&(step as u64).to_le_bytes())
}
/// Write the sections that follow the primary bytes:
/// overflow n_overflow × 12 B (slot: u64, value: u32), `entries` sorted by slot
/// index n_index × 16 B (slot: u64, pos: u64) sparse index
pub(crate) fn write_pciv_tail<W: Write>(w: &mut W, entries: &[(usize, u32)]) -> io::Result<()> {
for &(slot, value) in entries {
w.write_all(&(slot as u64).to_le_bytes())?;
w.write_all(&value.to_le_bytes())?;
}
let (step, _) = pciv_index_layout(entries.len());
if step > 0 {
for (block, chunk) in entries.chunks(step).enumerate() {
w.write_all(&(chunk[0].0 as u64).to_le_bytes())?;
w.write_all(&((block * step) as u64).to_le_bytes())?;
}
}
Ok(())
}
/// Finalise a PCIV file whose placeholder header and primary section are already on disk. /// Finalise a PCIV file whose placeholder header and primary section are already on disk.
/// ///
/// Truncates the file to `HEADER_SIZE + n`, then appends: /// Truncates the file to `HEADER_SIZE + n`, appends the overflow and index
/// overflow n_overflow × 12 B (slot: u64, value: u32) sorted by slot /// sections ([`write_pciv_tail`]) and overwrites the placeholder header at
/// index n_index × 16 B (slot: u64, pos: u64) sparse index /// offset 0 ([`write_pciv_header`]).
/// and overwrites the placeholder header at offset 0.
pub fn finalize_pciv(path: &Path, n: usize, entries: &[(usize, u32)]) -> io::Result<()> { pub fn finalize_pciv(path: &Path, n: usize, entries: &[(usize, u32)]) -> io::Result<()> {
let n_overflow = entries.len();
let file = OpenOptions::new().read(true).write(true).open(path)?; let file = OpenOptions::new().read(true).write(true).open(path)?;
file.set_len((HEADER_SIZE + n) as u64)?; file.set_len((HEADER_SIZE + n) as u64)?;
let mut w = BufWriter::new(file); let mut w = BufWriter::new(file);
w.seek(SeekFrom::End(0))?; w.seek(SeekFrom::End(0))?;
write_pciv_tail(&mut w, entries)?;
for &(slot, value) in entries {
w.write_all(&(slot as u64).to_le_bytes())?;
w.write_all(&value.to_le_bytes())?;
}
let step: usize = if n_overflow <= L1_INDEX_ENTRIES {
0
} else {
n_overflow.div_ceil(L1_INDEX_ENTRIES)
};
let n_index: usize = if step > 0 {
let count = n_overflow.div_ceil(step);
for (block, chunk) in entries.chunks(step).enumerate() {
let slot = chunk[0].0 as u64;
let pos = (block * step) as u64;
w.write_all(&slot.to_le_bytes())?;
w.write_all(&pos.to_le_bytes())?;
}
count
} else {
0
};
w.flush()?; w.flush()?;
let mut file = w.into_inner().map_err(|e| e.into_error())?; let mut file = w.into_inner().map_err(|e| e.into_error())?;
file.seek(SeekFrom::Start(0))?; file.seek(SeekFrom::Start(0))?;
file.write_all(&MAGIC)?; write_pciv_header(&mut file, n, entries.len())?;
file.write_all(&[0u8; 4])?;
file.write_all(&(n as u64).to_le_bytes())?;
file.write_all(&(n_overflow as u64).to_le_bytes())?;
file.write_all(&(n_index as u64).to_le_bytes())?;
file.write_all(&(step as u64).to_le_bytes())?;
file.flush() file.flush()
} }
+9
View File
@@ -11,6 +11,7 @@ use crate::bitmatrix::{
use crate::builder::PersistentCompactIntVecBuilder; use crate::builder::PersistentCompactIntVecBuilder;
use crate::colgroup::{ColGroup, MatrixGroupOps, chunked_presence_count}; use crate::colgroup::{ColGroup, MatrixGroupOps, chunked_presence_count};
use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE}; use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE};
use crate::mem_builder::MemCompactIntVecBuilder;
use crate::meta::MatrixMeta; use crate::meta::MatrixMeta;
use crate::reader::PersistentCompactIntVec; use crate::reader::PersistentCompactIntVec;
use crate::sparse_intmatrix::{PersistentSparseCompactIntMatrix, is_present as sparse_is_present}; use crate::sparse_intmatrix::{PersistentSparseCompactIntMatrix, is_present as sparse_is_present};
@@ -717,6 +718,14 @@ impl PersistentCompactIntMatrixBuilder {
PersistentCompactIntVecBuilder::new(self.n, &path) PersistentCompactIntVecBuilder::new(self.n, &path)
} }
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
/// the column is built in RAM and written once at `close`.
pub fn add_mem_col(&mut self) -> MemCompactIntVecBuilder {
let path = col_path(&self.dir, self.n_cols);
self.n_cols += 1;
MemCompactIntVecBuilder::new(self.n, &path)
}
pub fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()> { pub fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()> {
src.make_persistent(&col_path(&self.dir, self.n_cols))?; src.make_persistent(&col_path(&self.dir, self.n_cols))?;
self.n_cols += 1; self.n_cols += 1;
+2
View File
@@ -9,6 +9,7 @@ mod int_group_ops;
mod intmatrix; mod intmatrix;
mod layer_meta; mod layer_meta;
mod matrix_builder; mod matrix_builder;
mod mem_builder;
mod meta; mod meta;
mod mmap_file; mod mmap_file;
mod rankselect; mod rankselect;
@@ -35,6 +36,7 @@ pub use intmatrix::{
}; };
pub use layer_meta::LayerMeta; pub use layer_meta::LayerMeta;
pub use mem_builder::{MemBitVecBuilder, MemCompactIntVecBuilder};
pub use mmap_file::{building_path, publish_built}; pub use mmap_file::{building_path, publish_built};
pub use matrix_builder::{ColBuilder, MatrixBuilder}; pub use matrix_builder::{ColBuilder, MatrixBuilder};
pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder}; pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
+31 -1
View File
@@ -10,13 +10,19 @@ use std::path::Path;
use crate::{PersistentBitMatrixBuilder as BitMB, PersistentBitVecBuilder}; use crate::{PersistentBitMatrixBuilder as BitMB, PersistentBitVecBuilder};
use crate::{PersistentCompactIntMatrixBuilder as IntMB, PersistentCompactIntVecBuilder}; use crate::{PersistentCompactIntMatrixBuilder as IntMB, PersistentCompactIntVecBuilder};
use crate::{TempBitVec, TempCompactIntVec}; use crate::{MemBitVecBuilder, MemCompactIntVecBuilder, TempBitVec, TempCompactIntVec};
// ── ColBuilder ──────────────────────────────────────────────────────────────── // ── ColBuilder ────────────────────────────────────────────────────────────────
/// One column being built. `Bit`/`Int` write through a mmap of a sparse
/// file as values are set; `MemBit`/`MemInt` keep the column in RAM and
/// write it once at `close` (same file, byte for byte — see
/// [`MemBitVecBuilder`]).
pub enum ColBuilder { pub enum ColBuilder {
Bit(PersistentBitVecBuilder), Bit(PersistentBitVecBuilder),
Int(PersistentCompactIntVecBuilder), Int(PersistentCompactIntVecBuilder),
MemBit(MemBitVecBuilder),
MemInt(MemCompactIntVecBuilder),
} }
impl ColBuilder { impl ColBuilder {
@@ -24,6 +30,8 @@ impl ColBuilder {
match self { match self {
ColBuilder::Bit(b) => b.set(slot, value > 0), ColBuilder::Bit(b) => b.set(slot, value > 0),
ColBuilder::Int(b) => b.set(slot, value), ColBuilder::Int(b) => b.set(slot, value),
ColBuilder::MemBit(b) => b.set(slot, value > 0),
ColBuilder::MemInt(b) => b.set(slot, value),
} }
} }
@@ -31,6 +39,8 @@ impl ColBuilder {
match self { match self {
ColBuilder::Bit(b) => b.close(), ColBuilder::Bit(b) => b.close(),
ColBuilder::Int(b) => b.close(), ColBuilder::Int(b) => b.close(),
ColBuilder::MemBit(b) => b.close(),
ColBuilder::MemInt(b) => b.close(),
} }
} }
} }
@@ -64,6 +74,15 @@ impl MatrixBuilder {
}) })
} }
/// Number of columns created so far (absent + data columns, including
/// those inherited from a resumed directory).
pub fn n_cols(&self) -> usize {
match self {
MatrixBuilder::Bit(b) => b.n_cols(),
MatrixBuilder::Int(b) => b.n_cols(),
}
}
/// Add a column with no data written (all-zero/false) — for columns /// Add a column with no data written (all-zero/false) — for columns
/// absent from a given source (e.g. destination-only genomes not /// absent from a given source (e.g. destination-only genomes not
/// contributed by the source being merged in). /// contributed by the source being merged in).
@@ -81,6 +100,17 @@ impl MatrixBuilder {
}) })
} }
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
/// the column is built in RAM and written once, sequentially, at
/// `close`: no sparse file, no writable mapping. Costs `n / 8` bytes
/// (presence) or `n` bytes + overflow (counts) of RAM until closed.
pub fn add_mem_col(&mut self) -> ColBuilder {
match self {
MatrixBuilder::Bit(b) => ColBuilder::MemBit(b.add_mem_col()),
MatrixBuilder::Int(b) => ColBuilder::MemInt(b.add_mem_col()),
}
}
/// Add a column already computed as a bit vector. /// Add a column already computed as a bit vector.
pub fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> { pub fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> {
match self { match self {
+122
View File
@@ -0,0 +1,122 @@
//! In-memory column builders: the content of a column is built in RAM and
//! written once, sequentially, when the builder is closed.
//!
//! They produce exactly the same file, byte for byte, as
//! [`PersistentBitVecBuilder`](crate::PersistentBitVecBuilder) /
//! [`PersistentCompactIntVecBuilder`](crate::PersistentCompactIntVecBuilder),
//! but never create a sparse file nor a writable mapping: a block the file
//! system cannot allocate (full disk, quota, network FS) is reported as an
//! `io::Error` by `close`, instead of a SIGBUS on the page fault that first
//! touches it. The price is that the whole column lives in RAM until
//! `close`: `n / 8` bytes for bits, `n` bytes (+ overflow map) for counts.
use std::collections::HashMap;
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use crate::bitvec::{n_words, MAGIC as PBIV_MAGIC};
use crate::format::{write_pciv_header, write_pciv_tail};
use crate::mmap_file::{write_built, write_std_header};
// ── Bits (PBIV) ───────────────────────────────────────────────────────────────
/// In-memory counterpart of `PersistentBitVecBuilder` (`set` + `close` only).
pub struct MemBitVecBuilder {
words: Vec<u64>,
n: usize,
path: PathBuf,
}
impl MemBitVecBuilder {
/// All-false vector of `n` bits, to be written at `path` by `close`.
pub fn new(n: usize, path: &Path) -> Self {
Self { words: vec![0u64; n_words(n)], n, path: path.to_path_buf() }
}
#[inline]
pub fn len(&self) -> usize {
self.n
}
#[inline]
pub fn is_empty(&self) -> bool {
self.n == 0
}
#[inline]
pub fn set(&mut self, slot: usize, value: bool) {
debug_assert!(slot < self.n);
let bit = 1u64 << (slot & 63);
if value {
self.words[slot >> 6] |= bit;
} else {
self.words[slot >> 6] &= !bit;
}
}
/// Writes the PBIV file — header, then the words in native byte order,
/// exactly as the mmap-backed builder lays them out — and publishes it.
pub fn close(self) -> io::Result<()> {
let Self { words, n, path } = self;
write_built(&path, |w| {
write_std_header(w, PBIV_MAGIC, [0u8; 4], n)?;
for word in &words {
w.write_all(&word.to_ne_bytes())?;
}
Ok(())
})
}
}
// ── Compact ints (PCIV) ───────────────────────────────────────────────────────
/// In-memory counterpart of `PersistentCompactIntVecBuilder` (`set` +
/// `close` only). Same encoding: one byte per slot, values ≥ 255 stored as
/// the sentinel 255 plus an overflow entry.
pub struct MemCompactIntVecBuilder {
primary: Vec<u8>,
overflow: HashMap<usize, u32>,
n: usize,
path: PathBuf,
}
impl MemCompactIntVecBuilder {
/// All-zero vector of `n` slots, to be written at `path` by `close`.
pub fn new(n: usize, path: &Path) -> Self {
Self { primary: vec![0u8; n], overflow: HashMap::new(), n, path: path.to_path_buf() }
}
#[inline]
pub fn len(&self) -> usize {
self.n
}
#[inline]
pub fn is_empty(&self) -> bool {
self.n == 0
}
#[inline]
pub fn set(&mut self, slot: usize, value: u32) {
if value < 255 {
self.primary[slot] = value as u8;
self.overflow.remove(&slot);
} else {
self.primary[slot] = 255;
self.overflow.insert(slot, value);
}
}
/// Writes the PCIV file — header, primary bytes, overflow entries sorted
/// by slot, sparse index — and publishes it.
pub fn close(self) -> io::Result<()> {
let Self { primary, overflow, n, path } = self;
let mut entries: Vec<(usize, u32)> = overflow.into_iter().collect();
entries.sort_unstable_by_key(|&(slot, _)| slot);
write_built(&path, |w| {
write_pciv_header(w, n, entries.len())?;
w.write_all(&primary)?;
write_pciv_tail(w, &entries)
})
}
}
+33 -4
View File
@@ -11,7 +11,7 @@
//! between formats and aren't forced into a shared shape. //! between formats and aren't forced into a shared shape.
use std::fs::{self, File, OpenOptions}; use std::fs::{self, File, OpenOptions};
use std::io::{self, Seek, SeekFrom, Write as _}; use std::io::{self, BufWriter, Seek, SeekFrom, Write};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut}; use memmap2::{Mmap, MmapMut};
@@ -37,6 +37,37 @@ pub fn publish_built(path: &Path) -> io::Result<()> {
fs::rename(building_path(path), path) fs::rename(building_path(path), path)
} }
/// Writes the crate-wide standard header: `magic` + `pad` + `n as u64` (LE).
pub(crate) fn write_std_header<W: Write>(w: &mut W, magic: [u8; 4], pad: [u8; 4], n: usize) -> io::Result<()> {
w.write_all(&magic)?;
w.write_all(&pad)?;
w.write_all(&(n as u64).to_le_bytes())
}
/// Writes a whole file sequentially through `write` into
/// [`building_path`]`(path)`, flushes it, then publishes it at `path`. No
/// sparse file and no writable mapping: an allocation failure on the file
/// system (full disk, quota) surfaces as an `io::Error` from a `write`,
/// never as a SIGBUS on a page fault.
///
/// Deliberately no `sync_all`: one forced sync per file made the merge
/// ~8× slower (thousands of column files; measured 2026-11-17 on macOS),
/// and the mmap builders it replaces never synced either. Durability is the
/// OS's, like the rest of an index; the `rename` still guarantees that no
/// partially written file is ever published. Caveat: on a write-back
/// network FS an allocation error may only be reported at `close`, which
/// `File`'s drop ignores.
pub(crate) fn write_built(
path: &Path,
write: impl FnOnce(&mut BufWriter<File>) -> io::Result<()>,
) -> io::Result<()> {
let mut w = BufWriter::new(File::create(building_path(path))?);
write(&mut w)?;
w.flush()?;
drop(w.into_inner().map_err(|e| e.into_error())?);
publish_built(path)
}
/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it /// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it
/// with [`publish_built`] once complete — writes the standard header — `magic` + /// with [`publish_built`] once complete — writes the standard header — `magic` +
/// `pad` (4 bytes right after `magic`; usually all-zero, but /// `pad` (4 bytes right after `magic`; usually all-zero, but
@@ -60,9 +91,7 @@ pub(crate) fn create_mmap_file(
.create(true) .create(true)
.truncate(true) .truncate(true)
.open(building_path(path))?; .open(building_path(path))?;
file.write_all(&magic)?; write_std_header(&mut file, magic, pad, n)?;
file.write_all(&pad)?;
file.write_all(&(n as u64).to_le_bytes())?;
if extra_header_zeros > 0 { if extra_header_zeros > 0 {
file.write_all(&vec![0u8; extra_header_zeros])?; file.write_all(&vec![0u8; extra_header_zeros])?;
} }
+14 -26
View File
@@ -617,37 +617,25 @@ impl PersistentSparseCompactIntMatrixBuilder {
// ── pack_sparse_compact_int_matrix ────────────────────────────────────────── // ── pack_sparse_compact_int_matrix ──────────────────────────────────────────
/// Converts whichever dense count-matrix format is on disk at `dir` /// Converts a layer's count matrix (columnar `col_*.pciv` or dense
/// (`Packed`, i.e. `matrix.pcmx`, or `Columnar`, i.e. `col_*.pciv` + /// `matrix.pcmx`) to the sparse on-disk format in place, through the dense
/// `meta.json`) into the sparse on-disk format in place — generic over the /// packed form, exactly like [`crate::pack_sparse_bit_matrix`]. Old-format
/// dense source the same way [`super::bitmatrix::pack_sparse_bit_matrix`] /// files are removed only once the new format is fully written.
/// is: detects the format, transposes straight from it, and only removes /// Idempotent: a no-op if the sparse format is already present.
/// the old dense files once the new format is fully written (a crash
/// mid-conversion leaves the previous, still-valid format rather than a
/// half-written one). Idempotent — a no-op if `singleton_values.pciv`
/// already exists.
pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> { pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> {
use crate::intmatrix::{ColumnarCompactIntMatrix, PackedCompactIntMatrix, col_path}; use crate::intmatrix::{PackedCompactIntMatrix, pack_compact_int_matrix};
if is_present(dir) { if is_present(dir) {
return Ok(()); return Ok(());
} }
// Columnar → dense packed first, then dense packed → sparse: one
// mapping (`matrix.pcmx`) instead of one per column file. See
// `pack_sparse_bit_matrix` for the rationale and crash-safety argument.
pack_compact_int_matrix(dir)?;
let packed_path = dir.join("matrix.pcmx"); let packed_path = dir.join("matrix.pcmx");
if packed_path.exists() { let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?);
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?); PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?; drop(dense);
drop(dense); fs::remove_file(&packed_path)
fs::remove_file(&packed_path)?;
} else {
let dense = PersistentIntMatrix::Columnar(ColumnarCompactIntMatrix::open(dir)?);
let n_cols = dense.n_cols();
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
drop(dense);
for c in 0..n_cols {
let _ = fs::remove_file(col_path(dir, c));
}
let _ = fs::remove_file(dir.join("meta.json"));
}
Ok(())
} }
@@ -0,0 +1,27 @@
use tempfile::tempdir;
use crate::MatrixBuilder;
#[test]
fn n_cols_counts_absent_data_and_resumed_columns() {
for presence in [true, false] {
let dir = tempdir().unwrap();
let data = dir.path().join("m");
let mut mb = MatrixBuilder::new(presence, 5, &data).unwrap();
assert_eq!(mb.n_cols(), 0);
mb.add_absent_col().unwrap();
assert_eq!(mb.n_cols(), 1);
let mut c = mb.add_col().unwrap();
c.set_val(2, 7);
c.close().unwrap();
assert_eq!(mb.n_cols(), 2);
mb.close().unwrap();
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
assert_eq!(mb.n_cols(), 2, "presence={presence}");
mb.add_absent_col().unwrap();
assert_eq!(mb.n_cols(), 3);
mb.close().unwrap();
}
}
+117
View File
@@ -0,0 +1,117 @@
use std::fs;
use tempfile::tempdir;
use crate::{
MatrixBuilder, MemBitVecBuilder, MemCompactIntVecBuilder, PersistentBitMatrix,
PersistentBitVecBuilder, PersistentCompactIntVecBuilder, PersistentIntMatrix,
};
const SIZES: [usize; 6] = [0, 1, 63, 64, 65, 10_000];
/// Deterministic xorshift64* — no RNG dependency needed for test data.
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
self.0 ^= self.0 >> 12;
self.0 ^= self.0 << 25;
self.0 ^= self.0 >> 27;
self.0.wrapping_mul(0x2545_F491_4F6C_DD1D)
}
}
/// `(slot, value)` writes in random order, with overwrites; for counts,
/// about a third of the values are ≥ 255 (overflow), enough at n = 10 000
/// to exceed `L1_INDEX_ENTRIES` and exercise the sparse index section.
fn writes(n: usize, seed: u64, max_small: u64) -> Vec<(usize, u32)> {
let mut rng = Rng(seed);
(0..2 * n)
.map(|_| {
let slot = (rng.next() % n as u64) as usize;
let r = rng.next();
let value = match r % 3 {
0 => 255 + (r >> 8) as u32 % 100_000, // overflow
1 => 0, // also clears a previous overflow
_ => (r >> 8) as u32 % max_small as u32,
};
(slot, value)
})
.collect()
}
#[test]
fn mem_bitvec_is_byte_identical_to_persistent() {
for n in SIZES {
let dir = tempdir().unwrap();
let (pp, mp) = (dir.path().join("p.pbiv"), dir.path().join("m.pbiv"));
let mut p = PersistentBitVecBuilder::new(n, &pp).unwrap();
let mut m = MemBitVecBuilder::new(n, &mp);
for (slot, v) in writes(n, 0x9E37_79B9 + n as u64, 2) {
p.set(slot, v > 0);
m.set(slot, v > 0);
}
p.close().unwrap();
m.close().unwrap();
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
}
}
#[test]
fn mem_compact_int_is_byte_identical_to_persistent() {
for n in SIZES {
let dir = tempdir().unwrap();
let (pp, mp) = (dir.path().join("p.pciv"), dir.path().join("m.pciv"));
let mut p = PersistentCompactIntVecBuilder::new(n, &pp).unwrap();
let mut m = MemCompactIntVecBuilder::new(n, &mp);
for (slot, v) in writes(n, 0xC2B2_AE35 + n as u64, 255) {
p.set(slot, v);
m.set(slot, v);
}
p.close().unwrap();
m.close().unwrap();
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
}
}
/// `add_mem_col` columns, interleaved with absent columns and continued
/// across a `resume`, read back through the regular matrix readers.
#[test]
fn add_mem_col_matrix_reads_back() {
for presence in [true, false] {
let n = 100;
let dir = tempdir().unwrap();
let data = dir.path().join(if presence { "presence" } else { "counts" });
let value = |c: usize, slot: usize| -> u32 {
if (slot + c).is_multiple_of(3) { (slot * 7 + c) as u32 % 400 + 1 } else { 0 }
};
let mut mb = MatrixBuilder::new(presence, n, &data).unwrap();
mb.add_absent_col().unwrap();
let mut c1 = mb.add_mem_col();
(0..n).for_each(|s| c1.set_val(s, value(1, s)));
c1.close().unwrap();
mb.close().unwrap();
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
assert_eq!(mb.n_cols(), 2);
let mut c2 = mb.add_mem_col();
(0..n).for_each(|s| c2.set_val(s, value(2, s)));
c2.close().unwrap();
assert_eq!(mb.n_cols(), 3);
mb.close().unwrap();
for s in 0..n {
let expect = [0, value(1, s), value(2, s)];
if presence {
let m = PersistentBitMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
let want: Vec<bool> = expect.iter().map(|&v| v > 0).collect();
assert_eq!(&*m.row(s), want.as_slice(), "slot {s}");
} else {
let m = PersistentIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
assert_eq!(&*m.row(s), &expect, "slot {s}");
}
}
}
}
+2
View File
@@ -5,6 +5,8 @@ mod eliasfano;
mod fixedintvec; mod fixedintvec;
mod int_group_ops; mod int_group_ops;
mod intmatrix; mod intmatrix;
mod matrix_builder;
mod mem_builder;
mod rankselect; mod rankselect;
mod sparse; mod sparse;
mod sparse_intmatrix; mod sparse_intmatrix;
+4 -2
View File
@@ -90,7 +90,7 @@ impl IndexBuilder for KmerIndex {
let idx = KmerIndex { let idx = KmerIndex {
root_path: output.to_owned(), root_path: output.to_owned(),
meta: Arc::new(meta), meta: Arc::new(meta),
lock: std::sync::Mutex::new(None), lock: obisys::LockSlot::empty(output),
}; };
if !idx.acquire_lock() { if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other( return Err(OKIError::Io(std::io::Error::other(
@@ -102,7 +102,9 @@ impl IndexBuilder for KmerIndex {
fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> { fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> {
let output = output.as_ref(); let output = output.as_ref();
let idx = KmerIndex::open(output)?; // `output` is held exclusively by the builder that wrote it: reopen
// without a lock, a locking `open` would wait for that builder for ever.
let idx = KmerIndex::open_unlocked(output)?;
idx.meta.mark_indexed().map_err(OKIError::Io)?; idx.meta.mark_indexed().map_err(OKIError::Io)?;
let t_pack = Stage::start("pack"); let t_pack = Stage::start("pack");
idx.pack_matrices(sparse)?; idx.pack_matrices(sparse)?;
+78 -56
View File
@@ -1,6 +1,6 @@
use std::fs; use std::fs;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex}; use std::sync::Arc;
use crate::layer::KmerLayer; use crate::layer::KmerLayer;
use crate::partition::KmerPartition; use crate::partition::KmerPartition;
@@ -15,7 +15,7 @@ use crate::index::meta::{GenomeInfo, IndexConfig, IndexMeta};
pub struct KmerIndex { pub struct KmerIndex {
pub(crate) root_path: PathBuf, pub(crate) root_path: PathBuf,
pub(crate) meta: Arc<IndexMeta>, pub(crate) meta: Arc<IndexMeta>,
pub(crate) lock: Mutex<Option<obisys::DirLock>>, pub(crate) lock: obisys::LockSlot,
} }
impl KmerIndex { impl KmerIndex {
@@ -37,11 +37,8 @@ impl KmerIndex {
set_m(config.minimizer_size); set_m(config.minimizer_size);
let genomes = genome_info.into_iter().collect(); let genomes = genome_info.into_iter().collect();
let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?; let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?;
let idx = Self { let lock = obisys::LockSlot::empty(&root_path);
root_path, let idx = Self { root_path, meta: Arc::new(meta), lock };
meta: Arc::new(meta),
lock: Mutex::new(None),
};
if !idx.acquire_lock() { if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other( return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock right after create", "failed to acquire index lock right after create",
@@ -50,67 +47,92 @@ impl KmerIndex {
Ok(idx) Ok(idx)
} }
/// Open an existing index for reading — doesn't touch the directory /// Open an existing index for reading, holding a **shared** directory
/// lock at all: concurrent readers (`query`, `dump`, a `merge` source, /// lock for as long as the returned index lives: any number of readers
/// ...) must never block on each other. A caller that needs exclusive /// coexist, but a writer ([`open_lock`](Self::open_lock),
/// access (resuming an interrupted build) uses /// [`with_lock`](Self::with_lock)) waits for them all, and readers wait
/// [`open_lock`](Self::open_lock) instead; either way, `self.lock` /// for a writer at work — a reader never sees files change under it.
/// starts empty — [`test_lock`](Self::test_lock)/ /// The lock is taken before the metadata is read.
/// [`acquire_lock`](Self::acquire_lock) are opt-in from here. ///
/// Never open, from a process that holds the write lock on a directory
/// through another handle, that same directory again with `open`: the
/// new handle would wait for the first for ever.
pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> { pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let root_path = path.as_ref().to_owned(); Self::open_with(path.as_ref(), obisys::LockMode::Shared)
let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
set_k(meta.config.kmer_size);
set_m(meta.config.minimizer_size);
Ok(Self {
root_path,
meta: Arc::new(meta),
lock: Mutex::new(None),
})
} }
/// `open`, then hold the directory lock exclusively — for a caller /// `open`, holding the directory lock **exclusively** for as long as the
/// that resumes a write pipeline (e.g. `obikmer index` re-run on an /// returned index lives — **the** way for a command that modifies an
/// existing, not-yet-`Indexed` directory) rather than reading /// existing index in place (`convert`, `pack`, `annotate`, `utils
/// concurrently with others. /// rename`, resuming an interrupted build, ...) to claim it. The lock is
/// taken *before* the metadata is read, so what the caller reads is what
/// it will modify.
///
/// A command that only reads, or that writes a *new* index elsewhere
/// (`merge`, `filter`, `select`: exclusive on their destination, shared
/// on their sources), uses [`open`](Self::open). Library code that has to
/// write into an index it was handed (annexes built on demand) wraps that
/// write in [`with_lock`](Self::with_lock).
pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> { pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let idx = Self::open(path)?; Self::open_with(path.as_ref(), obisys::LockMode::Exclusive)
if !idx.acquire_lock() { }
return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock", fn open_with(path: &Path, mode: obisys::LockMode) -> OKIResult<Self> {
))); // A directory that is not an index is reported by `open_unlocked`,
// without a lock file being created in it.
if !Self::is_an_index(path) {
return Self::open_unlocked(path);
} }
let lock = match mode {
obisys::LockMode::Shared => obisys::LockSlot::shared(path),
obisys::LockMode::Exclusive => obisys::LockSlot::exclusive(path),
}
.map_err(OKIError::Io)?;
let mut idx = Self::open_unlocked(path)?;
idx.lock = lock;
Ok(idx) Ok(idx)
} }
/// Block until this index's directory lock is acquired, then hold it /// `open` without taking any lock — for a caller that already holds the
/// (released on [`release_lock`](Self::release_lock) or when `self` is /// directory through another handle (`finalize_indexed`, reopening the
/// dropped). Returns `false` only on a genuine I/O failure acquiring /// index its own builder holds exclusively).
/// the lock file itself — contention blocks rather than failing. pub(crate) fn open_unlocked(path: &Path) -> OKIResult<Self> {
pub fn acquire_lock(&self) -> bool { let root_path = path.to_owned();
match obisys::DirLock::acquire(&self.root_path) { let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
Ok(lock) => { set_k(meta.config.kmer_size);
*self.lock.lock().unwrap() = Some(lock); set_m(meta.config.minimizer_size);
true let lock = obisys::LockSlot::empty(&root_path);
} Ok(Self { root_path, meta: Arc::new(meta), lock })
Err(_) => false, }
}
/// Take the exclusive directory lock and hold it until `self` is dropped
/// — for an index just created, which is about to be written to.
/// Idempotent. Returns `false` only on a genuine I/O failure.
pub(crate) fn acquire_lock(&self) -> bool {
self.lock.acquire_exclusive().is_ok()
}
/// Whether this index object currently holds its directory lock
/// exclusively (a shared lock is not "write access").
pub fn holds_write_lock(&self) -> bool {
self.lock.holds_write_lock()
}
/// Run `f` with the directory lock held **exclusively** — promoting the
/// shared lock a handle from [`open`](Self::open) holds, and restoring it
/// afterwards; see [`obisys::LockSlot::with_exclusive`], including what
/// `f` must re-check. Library code that writes into an index it was
/// handed (annexes built on demand) wraps that write in this.
pub fn with_lock<T>(&self, f: impl FnOnce() -> OKIResult<T>) -> OKIResult<T> {
self.lock.with_exclusive(f)
} }
/// Non-blocking probe of this index's directory lock — doesn't retain /// Non-blocking probe of this index's directory lock — doesn't retain
/// it either way: `true` if the lock was free (briefly acquired, then /// anything: `true` if no process currently holds it exclusively, `false`
/// immediately released), `false` if another process currently holds /// if one does (this process included). For a caller that wants to warn
/// it. For a read-only caller that wants to warn about a build in /// about a write in progress without waiting for it.
/// progress without itself becoming exclusive; use
/// [`acquire_lock`](Self::acquire_lock)/[`open_lock`](Self::open_lock)
/// to actually hold it.
pub fn test_lock(&self) -> bool { pub fn test_lock(&self) -> bool {
matches!(obisys::DirLock::try_acquire(&self.root_path), Ok(Some(_))) self.lock.is_free_of_writer()
}
/// Release this index's directory lock, if held. A no-op if it isn't.
pub fn release_lock(&self) {
*self.lock.lock().unwrap() = None;
} }
/// Return `true` if `path` points to a valid obikmer index directory. /// Return `true` if `path` points to a valid obikmer index directory.
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "obikmer" name = "obikmer"
version = "1.4.0" version = "1.4.4"
edition = "2024" edition = "2024"
[[bin]] [[bin]]
+5 -1
View File
@@ -93,7 +93,11 @@ fn run_annotate(args: &AnnotateArgs) {
} }
}; };
let idx = open_index(&args.index); // Rewrites the index metadata in place: exclusive for the whole command.
let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}");
std::process::exit(1);
});
let mut genomes = idx.meta().genomes().unwrap_or_else(|e| { let mut genomes = idx.meta().genomes().unwrap_or_else(|e| {
eprintln!("error reading index metadata: {e}"); eprintln!("error reading index metadata: {e}");
+2 -8
View File
@@ -70,14 +70,8 @@ struct ModeArgs {
} }
pub fn run(args: ConvertArgs) { pub fn run(args: ConvertArgs) {
// Modifies the index in place; acquired before opening so a concurrent // Modifies the index in place: exclusive for the whole command.
// writer can't slip in between the open and the convert below. let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = KmerIndex::open(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
}); });
+2 -8
View File
@@ -26,14 +26,8 @@ pub struct PackArgs {
} }
pub fn run(args: PackArgs) { pub fn run(args: PackArgs) {
// Modifies the index in place; acquired before opening so a concurrent // Modifies the index in place: exclusive for the whole command.
// writer can't slip in between the open and the pack below. let idx = Arc::new(KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = Arc::new(KmerIndex::open(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
})); }));
@@ -0,0 +1,51 @@
//! `--pseudo-alignment` / `--fasta` / `--vcf` file output.
use std::io::{BufWriter, Write};
use obikphylo::siblings::SnpAlignment;
use tracing::info;
use super::args::PhyloArgs;
fn create(path: &str) -> BufWriter<std::fs::File> {
BufWriter::new(std::fs::File::create(path).unwrap_or_else(|e| {
eprintln!("error creating {path}: {e}");
std::process::exit(1);
}))
}
/// Writes the drawn alignment as FASTA (default, or `--fasta`) and/or VCF
/// (`--vcf`) next to `-o <prefix>`.
pub(super) fn write_alignment_outputs(
alignment: &SnpAlignment,
labels: &[String],
args: &PhyloArgs,
) {
let prefix = args.output.as_ref().map(|p| p.display().to_string());
let path_for = |suffix: &str| match &prefix {
Some(p) => format!("{p}_alignment.{suffix}"),
None => format!("alignment.{suffix}"),
};
let n_sites = alignment.sequences.first().map_or(0, Vec::len);
let n_genomes = alignment.genome_indices.len();
if args.fasta || !args.vcf {
let path = path_for("fasta");
let mut f = create(&path);
for (&g, seq) in alignment.genome_indices.iter().zip(alignment.sequences.iter()) {
obifastwrite::write_plain_record(seq, &labels[g], &mut f).unwrap();
}
info!("pseudo-alignment ({n_sites} site(s), {n_genomes} genome(s)) → {path}");
}
if args.vcf {
let path = path_for("vcf");
let mut f = create(&path);
let written = alignment.write_vcf(labels, !args.no_ambiguity, &mut f).unwrap_or_else(|e| {
eprintln!("error writing {path}: {e}");
std::process::exit(1);
});
f.flush().unwrap();
info!("pseudo-alignment VCF ({written}/{n_sites} site(s), {n_genomes} genome(s)) → {path}");
}
}
+53 -1
View File
@@ -1,6 +1,7 @@
use std::path::PathBuf; use std::path::PathBuf;
use clap::Args; use clap::Args;
use obikphylo::siblings::{AmbiguityPolicy, AnnexSelection};
use obikphylo::{DistanceMetric, GammaShape, SnpDistanceKind}; use obikphylo::{DistanceMetric, GammaShape, SnpDistanceKind};
/// `--gamma-shape` CLI value: a fixed `alpha`, or `auto`/`estimate` to fit /// `--gamma-shape` CLI value: a fixed `alpha`, or `auto`/`estimate` to fit
@@ -223,7 +224,13 @@ pub struct PhyloArgs {
/// Build (or rebuild) the sibling-count/minorant annex — independent of /// Build (or rebuild) the sibling-count/minorant annex — independent of
/// the distance metric below, meant to be run routinely, ahead of any /// the distance metric below, meant to be run routinely, ahead of any
/// SNP-family distance computation that will later consume it. /// SNP-family distance computation that will later consume it. Rebuilding
/// it invalidates the annexes derived from it; those are rebuilt here
/// only for the options given alongside (`--entropy` the entropy annex,
/// `--quartet` the quartet one, `--pair-balance` the pair-weight and
/// family-overlap ones, `--family-overlap`/`--min-shared-family` the
/// family-overlap one), each under the `--no-ambiguity`/`--only-diploid`
/// convention given — anything not asked for is built later, on demand.
#[arg(long)] #[arg(long)]
pub sibling_annex: bool, pub sibling_annex: bool,
@@ -275,6 +282,19 @@ pub struct PhyloArgs {
#[arg(long)] #[arg(long)]
pub pseudo_alignment: bool, pub pseudo_alignment: bool,
/// Write the pseudo-alignment as FASTA (`<prefix>_alignment.fasta`) —
/// the default format when neither `--fasta` nor `--vcf` is given. Implies
/// `--pseudo-alignment`; combinable with `--vcf`.
#[arg(long)]
pub fasta: bool,
/// Write the pseudo-alignment as VCF (`<prefix>_alignment.vcf`): one
/// contig `obikmer_<partition>_<layer>` per scanned layer, `POS` the
/// family's index within it, `ID` the family's masked canonical k-mer.
/// Implies `--pseudo-alignment`; combinable with `--fasta`.
#[arg(long)]
pub vcf: bool,
/// Number of variable sites to draw index-wide, or `all` for every site /// Number of variable sites to draw index-wide, or `all` for every site
/// that passes the other filters (no sampling then). Mandatory for /// that passes the other filters (no sampling then). Mandatory for
/// `--pseudo-alignment`/`--sankoff`; optional for a `snp-*` /// `--pseudo-alignment`/`--sankoff`; optional for a `snp-*`
@@ -296,6 +316,13 @@ pub struct PhyloArgs {
#[arg(long)] #[arg(long)]
pub no_ambiguity: bool, pub no_ambiguity: bool,
/// In `--pseudo-alignment`, keep only the genomes carrying one or two
/// members of a family (a base, or a two-base IUPAC code) and treat the
/// others (three or four members) as missing data (`?`). Milder than
/// `--no-ambiguity`, which it excludes.
#[arg(long, conflicts_with = "no_ambiguity")]
pub only_diploid: bool,
/// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, keep only /// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, keep only
/// the families that can separate two pairs of genomes from each other /// the families that can separate two pairs of genomes from each other
/// (a bipartition): at least two distinct called states, each carried /// (a bipartition): at least two distinct called states, each carried
@@ -504,3 +531,28 @@ pub struct PhyloArgs {
#[arg(short, long)] #[arg(short, long)]
pub output: Option<PathBuf>, pub output: Option<PathBuf>,
} }
impl PhyloArgs {
/// How genomes carrying several members of a family are treated.
pub fn ambiguity(&self) -> AmbiguityPolicy {
AmbiguityPolicy::from_flags(self.no_ambiguity, self.only_diploid)
}
/// The derived annexes the options of this invocation read, all under
/// [`ambiguity`](Self::ambiguity).
pub fn annex_selection(&self) -> AnnexSelection {
AnnexSelection {
policy: self.ambiguity(),
entropy: self.entropy.is_some(),
quartet: self.quartet,
overlap: self.family_overlap || self.min_shared_family.is_some() || self.pair_balance,
pair_weight: self.pair_balance,
}
}
/// Whether a pseudo-alignment must be drawn and written (`--pseudo-alignment`,
/// or any of its output-format flags).
pub fn alignment_output(&self) -> bool {
self.pseudo_alignment || self.fasta || self.vcf
}
}
+10 -3
View File
@@ -100,7 +100,14 @@ fn drop_ascertainment_noninformative(alignment: &SnpAlignment) -> SnpAlignment {
.collect() .collect()
}) })
.collect(); .collect();
SnpAlignment { sequences, genome_indices: alignment.genome_indices.clone() } let sites = alignment
.sites
.iter()
.zip(keep.iter())
.filter(|&(_, &k)| k)
.map(|(&site, _)| site)
.collect();
SnpAlignment { sequences, genome_indices: alignment.genome_indices.clone(), sites }
} }
/// Recode every occurrence of a byte in `symbols` to `-` — the same /// Recode every occurrence of a byte in `symbols` to `-` — the same
@@ -119,7 +126,7 @@ fn recode_symbols_as_absent(alignment: &SnpAlignment, symbols: &[u8]) -> SnpAlig
.collect() .collect()
}) })
.collect(); .collect();
SnpAlignment { sequences, genome_indices: alignment.genome_indices.clone() } SnpAlignment { sequences, genome_indices: alignment.genome_indices.clone(), sites: alignment.sites.clone() }
} }
fn compact_alphabet(alignment: &SnpAlignment, free_loss: bool) -> CompactAlphabet { fn compact_alphabet(alignment: &SnpAlignment, free_loss: bool) -> CompactAlphabet {
@@ -384,7 +391,7 @@ mod tests {
fn alignment(sequences: Vec<Vec<u8>>) -> SnpAlignment { fn alignment(sequences: Vec<Vec<u8>>) -> SnpAlignment {
let genome_indices = (0..sequences.len()).collect(); let genome_indices = (0..sequences.len()).collect();
SnpAlignment { sequences, genome_indices } SnpAlignment { sequences, genome_indices, sites: Vec::new() }
} }
#[test] #[test]
+14 -28
View File
@@ -1,3 +1,4 @@
mod alignment_out;
mod args; mod args;
mod iqtree; mod iqtree;
mod phyg; mod phyg;
@@ -19,6 +20,7 @@ use obikphylo::{Metrics, neighbor_joining, upgma};
use obisys::{Reporter, Stage}; use obisys::{Reporter, Stage};
use tracing::info; use tracing::info;
use alignment_out::write_alignment_outputs;
use args::{DistanceArg, NSitesArg, distance_to_compute}; use args::{DistanceArg, NSitesArg, distance_to_compute};
use iqtree::write_iqtree; use iqtree::write_iqtree;
use phyg::write_sankoff_phyg; use phyg::write_sankoff_phyg;
@@ -78,16 +80,12 @@ pub fn run(args: PhyloArgs) {
// Meant to be (re)built routinely, ahead of any SNP-family distance // Meant to be (re)built routinely, ahead of any SNP-family distance
// computation that will later consume it. // computation that will later consume it.
if args.sibling_annex { if args.sibling_annex {
// Writes into the index directory — hold an exclusive lock for the // Writes into the index directory; `build_sibling_annex` holds the
// duration so a second, concurrent `--sibling-annex` run on the // index's exclusive lock for the duration (`KmerIndex::with_lock`),
// same index can't corrupt these writes (see obisys::DirLock). // so a concurrent run on the same index can't corrupt these writes.
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
info!("building sibling-count/minorant annex"); info!("building sibling-count/minorant annex");
let t = Stage::start("sibling_annex"); let t = Stage::start("sibling_annex");
cache.build_sibling_annex().unwrap_or_else(|e| { cache.build_sibling_annex(&args.annex_selection()).unwrap_or_else(|e| {
eprintln!("error building sibling annex: {e}"); eprintln!("error building sibling annex: {e}");
std::process::exit(1); std::process::exit(1);
}); });
@@ -157,7 +155,7 @@ pub fn run(args: PhyloArgs) {
// ── Family Overlap matrix (`--family-overlap`) ────────────────────────────── // ── Family Overlap matrix (`--family-overlap`) ──────────────────────────────
if args.family_overlap { if args.family_overlap {
let t = Stage::start("family_overlap"); let t = Stage::start("family_overlap");
let overlap = cache.family_overlap(args.no_ambiguity).unwrap_or_else(|e| { let overlap = cache.family_overlap(args.ambiguity()).unwrap_or_else(|e| {
eprintln!("error computing family overlap: {e}"); eprintln!("error computing family overlap: {e}");
std::process::exit(1); std::process::exit(1);
}); });
@@ -219,7 +217,7 @@ pub fn run(args: PhyloArgs) {
// `args::PhyloArgs::min_shared_family`'s own docs on why). // `args::PhyloArgs::min_shared_family`'s own docs on why).
let snp_exclude_mask: Vec<bool> = match args.min_shared_family { let snp_exclude_mask: Vec<bool> = match args.min_shared_family {
Some(threshold) => { Some(threshold) => {
let overlap = cache.family_overlap(args.no_ambiguity).unwrap_or_else(|e| { let overlap = cache.family_overlap(args.ambiguity()).unwrap_or_else(|e| {
eprintln!("error computing family overlap: {e}"); eprintln!("error computing family overlap: {e}");
std::process::exit(1); std::process::exit(1);
}); });
@@ -255,7 +253,7 @@ pub fn run(args: PhyloArgs) {
}; };
let sampling = SamplingSpec { let sampling = SamplingSpec {
free_loss: args.free_loss, free_loss: args.free_loss,
no_ambiguity: args.no_ambiguity, ambiguity: args.ambiguity(),
quartet: args.quartet, quartet: args.quartet,
pair_balance: args.pair_balance, pair_balance: args.pair_balance,
entropy_bias, entropy_bias,
@@ -289,13 +287,13 @@ pub fn run(args: PhyloArgs) {
let sankoff_requested = args.sankoff || args.tnt || args.phyg || args.iqtree; let sankoff_requested = args.sankoff || args.tnt || args.phyg || args.iqtree;
let distance = distance_to_compute(args.distance, args.nj, args.upgma, args.shared_kmers); let distance = distance_to_compute(args.distance, args.nj, args.upgma, args.shared_kmers);
let snp_kind = distance.and_then(DistanceArg::as_snp); let snp_kind = distance.and_then(DistanceArg::as_snp);
let need_alignment = args.pseudo_alignment || sankoff_requested; let need_alignment = args.alignment_output() || sankoff_requested;
let need_pairwise = sankoff_requested || snp_kind.is_some(); let need_pairwise = sankoff_requested || snp_kind.is_some();
// `None` = every site (`--n-sites all`, or `--n-sites` omitted where it is optional) // `None` = every site (`--n-sites all`, or `--n-sites` omitted where it is optional)
let target = args.n_sites.and_then(NSitesArg::target); let target = args.n_sites.and_then(NSitesArg::target);
if args.pseudo_alignment && args.n_sites.is_none() { if args.alignment_output() && args.n_sites.is_none() {
eprintln!("error: --pseudo-alignment requires --n-sites <N|all>"); eprintln!("error: --pseudo-alignment/--fasta/--vcf require --n-sites <N|all>");
std::process::exit(1); std::process::exit(1);
} }
if sankoff_requested && args.n_sites.is_none() { if sankoff_requested && args.n_sites.is_none() {
@@ -361,21 +359,9 @@ pub fn run(args: PhyloArgs) {
let alignment = alignment_reducer.map(obikphylo::siblings::AlignmentReducer::finish); let alignment = alignment_reducer.map(obikphylo::siblings::AlignmentReducer::finish);
// ── SNP pseudo-alignment (`--pseudo-alignment`) ───────────────────────────── // ── SNP pseudo-alignment (`--pseudo-alignment`) ─────────────────────────────
if args.pseudo_alignment { if args.alignment_output() {
let alignment = alignment.as_ref().expect("scanned above: need_alignment"); let alignment = alignment.as_ref().expect("scanned above: need_alignment");
write_alignment_outputs(alignment, &labels, &args);
let path = args.output.as_ref()
.map(|p| format!("{}_alignment.fasta", p.display()))
.unwrap_or_else(|| "alignment.fasta".into());
let mut f = BufWriter::new(std::fs::File::create(&path).unwrap_or_else(|e| {
eprintln!("error creating {path}: {e}");
std::process::exit(1);
}));
let n_sites = alignment.sequences.first().map_or(0, Vec::len);
for (&g, seq) in alignment.genome_indices.iter().zip(alignment.sequences.iter()) {
obifastwrite::write_plain_record(seq, &labels[g], &mut f).unwrap();
}
info!("pseudo-alignment ({n_sites} site(s), {} genome(s)) → {path}", alignment.genome_indices.len());
} }
// ── Sankoff cost-matrix calibration (`--sankoff`, `--tnt`, `--phyg`, `--iqtree`) ── // ── Sankoff cost-matrix calibration (`--sankoff`, `--tnt`, `--phyg`, `--iqtree`) ──
+3 -1
View File
@@ -53,7 +53,9 @@ pub(super) fn run_bits_per_kmer(index_path: &PathBuf) {
pub(super) fn run_rename(index_path: &PathBuf, spec: &str) { pub(super) fn run_rename(index_path: &PathBuf, spec: &str) {
let (old_label, new_label) = parse_rename_spec(spec); let (old_label, new_label) = parse_rename_spec(spec);
let idx = KmerIndex::open(index_path).unwrap_or_else(|e| { // Rewrites the index metadata and renames a spectrum file in place:
// exclusive for the whole command.
let idx = KmerIndex::open_lock(index_path).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
}); });
-2
View File
@@ -15,5 +15,3 @@ obisys = { path = "../obisys", default-features = false }
rayon = "1" rayon = "1"
tracing = "0.1.44" tracing = "0.1.44"
[dev-dependencies]
tempfile = "3"
-605
View File
@@ -1,605 +0,0 @@
//! Merging a source partition's new layer into a destination partition's
//! de Bruijn graph union (pass 1) then column fill (pass 2). Free function,
//! not a `KmerIndex` method — `KmerIndex` stays the stateless
//! `Index { Partition { Layer } }` data model; this is one of the
//! algorithms that reads/writes through it, living here in `obikmerge`
//! instead.
use std::fs;
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use obipipeline::{
Pipeline, PipelineError, PipelineSender, SharedFlatFn, Stage, ThrottleGuard, WorkerPool,
make_sink, make_source, make_transform, throttle,
};
use tracing::debug;
use obicompactvec::{ColBuilder, MatrixBuilder};
use obikindex::layer::{IndexMode, KmerLayer, TypedLayer};
use obikindex::{KmerIndex, OKIError, OKIResult};
use obikindexer::{build_graph, materialize_layer};
use obikseq::CanonicalKmer;
use obiskio::UnitigFileReader;
use crate::MergeMode;
// `MatrixBuilder::new`/`resume` take a `presence: bool` — `MergeMode` maps
// to it directly (`Presence` → `true`, `Count` → `false`).
fn presence(mode: MergeMode) -> bool {
mode == MergeMode::Presence
}
#[cfg(test)]
mod matrix_builder_tests {
use tempfile::tempdir;
use obicompactvec::{PersistentBitMatrix, PersistentIntMatrix};
use super::{ColBuilder, MatrixBuilder};
/// Mirrors `merge_partition`'s "new layer" setup: absent (dst-genome)
/// columns get no data, then source columns are filled — all through one
/// continuous `MatrixBuilder` session, closed once at the end.
#[test]
fn new_layer_absent_then_source_columns_presence() {
let dir = tempdir().unwrap();
let data_dir = dir.path().join("presence");
let mut mb = MatrixBuilder::new(true, 3, &data_dir).unwrap();
// Two absent (dst-genome) columns.
mb.add_absent_col().unwrap();
mb.add_absent_col().unwrap();
// One source column, filled like pass 2 would.
let mut col = mb.add_col().unwrap();
match &mut col {
ColBuilder::Bit(b) => {
b.set(0, true);
b.set(1, false);
b.set(2, true);
}
ColBuilder::Int(_) => unreachable!(),
}
col.close().unwrap();
mb.close().unwrap();
let m = PersistentBitMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
assert_eq!(&*m.row(0), &[false, false, true]);
assert_eq!(&*m.row(1), &[false, false, false]);
assert_eq!(&*m.row(2), &[false, false, true]);
}
#[test]
fn new_layer_absent_then_source_columns_count() {
let dir = tempdir().unwrap();
let data_dir = dir.path().join("counts");
let mut mb = MatrixBuilder::new(false, 2, &data_dir).unwrap();
mb.add_absent_col().unwrap();
let mut col = mb.add_col().unwrap();
match &mut col {
ColBuilder::Int(b) => {
b.set(0, 7);
b.set(1, 42);
}
ColBuilder::Bit(_) => unreachable!(),
}
col.close().unwrap();
mb.close().unwrap();
let m = PersistentIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 2);
assert_eq!(&*m.row(0), &[0u32, 7]);
assert_eq!(&*m.row(1), &[0u32, 42]);
}
/// Mirrors `merge_partition`'s "existing dst layer" setup: an
/// already-closed matrix (from a previous merge) gets more columns
/// appended via `resume`, without callers ever seeing `col_path`/
/// `MatrixMeta` — `n` itself is read back from the matrix directory.
#[test]
fn resume_appends_source_columns_to_existing_layer() {
let dir = tempdir().unwrap();
let data_dir = dir.path().join("presence");
// Previous merge: one dst-genome column already on disk.
let mut mb0 = MatrixBuilder::new(true, 3, &data_dir).unwrap();
mb0.add_absent_col().unwrap();
mb0.close().unwrap();
// This merge: resume and append two more source columns.
let mut mb = MatrixBuilder::resume(true, &data_dir).unwrap();
for vals in [[true, false, true], [false, true, false]] {
let mut col = mb.add_col().unwrap();
match &mut col {
ColBuilder::Bit(b) => {
for (slot, v) in vals.into_iter().enumerate() {
b.set(slot, v);
}
}
ColBuilder::Int(_) => unreachable!(),
}
col.close().unwrap();
}
mb.close().unwrap();
let m = PersistentBitMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
assert_eq!(&*m.row(0), &[false, true, false]);
assert_eq!(&*m.row(1), &[false, false, true]);
assert_eq!(&*m.row(2), &[false, true, false]);
}
}
// ── dst cross-layer lookup ──────────────────────────────────────────────────
/// Batch membership lookup across `dst_layers` — first layer that carries
/// each kmer wins. Groups hits by layer index (as `(kmer batch index,
/// slot)` pairs) rather than returning them in query order, so pass 2 can
/// follow with one column-major write per `(layer, column)` instead of
/// interleaving across layers kmer by kmer. The kmers with no hit in any
/// layer come back as a separate index list, for the new-layer MPHF batch
/// lookup on a miss.
fn find_batch_grouped(
layers: &[KmerLayer],
kmers: &[CanonicalKmer],
) -> (Vec<Vec<(usize, usize)>>, Vec<usize>) {
let mut by_layer: Vec<Vec<(usize, usize)>> = vec![Vec::new(); layers.len()];
let mut not_found: Vec<usize> = Vec::new();
for (idx, &kmer) in kmers.iter().enumerate() {
match layers
.iter()
.enumerate()
.find_map(|(li, l)| l.find_slot(kmer).map(|slot| (li, slot)))
{
Some((li, slot)) => by_layer[li].push((idx, slot)),
None => not_found.push(idx),
}
}
(by_layer, not_found)
}
// ── merge_partition ────────────────────────────────────────────────────────
/// Merge `sources` into destination partition `i`.
///
/// Each entry in `sources` is `(index, n_genomes)` where `n_genomes` is the
/// number of genome columns that source contributes. A merged index
/// contributes more than one. The total new columns added to the
/// destination is `sum(n_genomes)`.
///
/// `n_dst_genomes` is the number of genome columns already in the
/// destination matrices (copied from the bootstrap source before this
/// call).
pub(crate) fn merge_partition(
dst: &KmerIndex,
i: usize,
sources: &[(&KmerIndex, usize)],
mode: MergeMode,
n_dst_genomes: usize,
block_bits: u8,
evidence: &IndexMode,
) -> OKIResult<usize> {
let dst_partition = dst.partition(i)?;
let n_dst_layers = dst_partition.n_layers();
let n_src_total: usize = sources.iter().map(|(_, n)| *n).sum();
// First merge in presence mode: materialize implicit single-genome
// presence matrices on existing layers (all slots true — every kmer in
// those layers belongs to genome_0) so a resumed builder can append
// source columns onto them below.
if n_dst_genomes == 1 && mode == MergeMode::Presence {
for l in 0..n_dst_layers {
let layer = dst_partition.layer(l)?.open()?;
TypedLayer::<()>::init_presence_matrix(layer.dir(), layer.n())?;
}
}
// dst's `0..n_dst_layers` layers and every source's own layers, for
// this one partition — all opened the same direct way, not via
// `IndexCache`. Every layer read here (dst's pre-existing ones, and
// every source's) is just as stable for the duration of this call: dst
// only ever gets a brand new layer *appended* below, never one of these
// read from, so there's no "dst isn't stable" distinction between the
// two loops — `IndexCache` still doesn't fit either one, for a single,
// uniform reason: `obikindexer::build_graph`/`obipipeline::WorkerPool`
// (pass 1/2 below) are plain-OS-thread pipelines requiring `Send + Sync
// + 'static`, and `IndexCache<'_>` always borrows `&KmerIndex` — no
// amount of logical stability changes that a borrowed cache can't cross
// that boundary, only owned `KmerLayer`s can (each owns its own mmap'd
// files, no reference back to the index it came from).
let dst_layers: Vec<KmerLayer> = (0..n_dst_layers)
.map(|l| dst_partition.layer(l)?.open())
.collect::<OKIResult<_>>()?;
let dst_layers = Arc::new(dst_layers);
let src_layers: Vec<Vec<Arc<KmerLayer>>> = sources
.iter()
.map(|(src, _)| -> OKIResult<Vec<Arc<KmerLayer>>> {
let partition = src.partition(i)?;
(0..partition.n_layers())
.map(|l| partition.layer(l)?.open().map(Arc::new))
.collect()
})
.collect::<OKIResult<_>>()?;
// ── Pass 1: pipeline — parallel file read + dst filter + graph fill ────
//
// Source : list of unitigs.bin paths (one per source × layer)
// Flat : open file, emit Vec<CanonicalKmer> batches (BeeGFS parallel I/O)
// Transform: filter via dst_layers — Arc<[KmerLayer]>: Sync
// Sink : push new kmers into GraphDeBruijn (single thread, no locks needed)
let mut unitig_paths: Vec<PathBuf> = Vec::new();
for layers in &src_layers {
for layer in layers {
let p = layer.unitigs_path();
if p.exists() {
unitig_paths.push(p);
}
}
}
let n_src_layers = unitig_paths.len();
debug!("partition {i}: de Bruijn graph build start — {n_src_layers} source layer(s)");
const BATCH: usize = 4096;
let n_workers = rayon::current_num_threads().min(16).max(4);
// At most 2 files open simultaneously: keeps n_workers-2 workers free
// for the Transform stage. Each open file monopolises one worker for the
// full duration of its read, so this must stay well below n_workers.
let max_open = 2;
let dst_filter = Arc::clone(&dst_layers);
let g = build_graph(
unitig_paths.into_iter(),
move |path: PathBuf, emit: &mut dyn FnMut(Vec<CanonicalKmer>)| -> OKIResult<()> {
let reader = UnitigFileReader::open_sequential(&path)?;
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
batch.push(kmer);
if batch.len() == BATCH {
emit(std::mem::replace(&mut batch, Vec::with_capacity(BATCH)));
}
}
if !batch.is_empty() {
emit(batch);
}
Ok(())
},
move |kmer| dst_filter.iter().all(|l| l.find_slot(kmer).is_none()),
n_workers,
max_open,
)?;
let any_new = g.len() > 0;
debug!(
"partition {i}: de Bruijn graph done — {} new kmers",
g.len()
);
// Build new layer from de Bruijn graph if there are new kmers.
let new_layer_idx = n_dst_layers;
let new_layer_dir = KmerLayer::new(&dst_partition, new_layer_idx)
.dir()
.to_path_buf();
let n_new = if any_new {
debug!("partition {i}: unitig traversal start — {} nodes", g.len());
let n_nodes = materialize_layer(g, &new_layer_dir, block_bits, evidence)?;
debug!("partition {i}: MPHF build done");
n_nodes
} else {
drop(g);
0
};
let t_open = std::time::Instant::now();
let new_mphf: Option<Arc<obikindex::layer::MphfOnly>> = if any_new {
Some(Arc::new(obikindex::layer::MphfOnly::open(&new_layer_dir)?))
} else {
None
};
debug!(
"partition {i}: MPHF open in {:.3}s",
t_open.elapsed().as_secs_f64()
);
// ── Prepare matrix directories for the new layer ──────────────────────
// Absent columns (dst genomes) get an all-zero/false column. Source-genome
// columns are created as mutable builders for pass 2. `new_mb` is kept
// open (not `close`d) until pass 2 has filled every source column, so its
// `n_cols` bookkeeping stays in sync with the columns actually created.
let (new_src_builders, new_mb): (Vec<ColBuilder>, Option<MatrixBuilder>) = if any_new {
let data_dir = match mode {
MergeMode::Presence => new_layer_dir.join("presence"),
MergeMode::Count => new_layer_dir.join("counts"),
};
fs::create_dir_all(&data_dir)?;
let mut mb = MatrixBuilder::new(presence(mode), n_new, &data_dir).map_err(OKIError::Io)?;
for _ in 0..n_dst_genomes {
mb.add_absent_col().map_err(OKIError::Io)?;
}
let cols = (0..n_src_total)
.map(|_| mb.add_col().map_err(OKIError::Io))
.collect::<OKIResult<Vec<_>>>()?;
(cols, Some(mb))
} else {
(vec![], None)
};
let t_builders = std::time::Instant::now();
// Builders for existing layers: n_src_total per layer, resumed from
// each layer's own matrix directory (already holding n_dst_genomes
// columns from a previous merge). Columns land at
// n_dst_genomes .. n_dst_genomes + n_src_total - 1.
let mut exist_mbs: Vec<MatrixBuilder> = Vec::with_capacity(n_dst_layers);
let mut exist_builders: Vec<Vec<ColBuilder>> = Vec::with_capacity(n_dst_layers);
for l in 0..n_dst_layers {
let layer_dir = dst_layers[l].dir().to_path_buf();
let data_dir = match mode {
MergeMode::Presence => layer_dir.join("presence"),
MergeMode::Count => layer_dir.join("counts"),
};
let mut mb = MatrixBuilder::resume(presence(mode), &data_dir).map_err(OKIError::Io)?;
let cols = (0..n_src_total)
.map(|_| mb.add_col().map_err(OKIError::Io))
.collect::<OKIResult<Vec<_>>>()?;
exist_mbs.push(mb);
exist_builders.push(cols);
}
debug!(
"partition {i}: builders ready in {:.3}s",
t_builders.elapsed().as_secs_f64()
);
// ── Pass 2: fill builders (pipeline) ─────────────────────────────────
let t_pass2 = std::time::Instant::now();
// Collect source items before the pipeline so errors propagate via ?
// before any worker thread is spawned.
let mut pass2_items: Vec<(usize, usize, PathBuf, Arc<KmerLayer>)> = Vec::new();
{
let mut col_offset = 0usize;
for ((_, src_n), layers) in sources.iter().zip(src_layers.iter()) {
for layer in layers {
let unitigs_path = layer.unitigs_path();
if unitigs_path.exists() {
pass2_items.push((col_offset, *src_n, unitigs_path, Arc::clone(layer)));
}
}
col_offset += src_n;
}
}
enum Pass2Data {
SrcLayer((usize, usize, PathBuf, Arc<KmerLayer>, ThrottleGuard)),
RawBatch((usize, usize, Arc<KmerLayer>, Vec<CanonicalKmer>)),
/// One entry per `(dst layer, absolute column)` touched by this
/// batch, each carrying every `(slot, value)` pair for that column
/// together — column-major, so the sink below locks that column's
/// `ColBuilder` once per batch and writes the whole run, instead of
/// once per `(kmer, genome)` pair.
WriteBatch(Vec<(Option<usize>, usize, Vec<(usize, u32)>)>),
}
let exist_locked: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_builders
.into_iter()
.map(|layer| layer.into_iter().map(|b| Arc::new(Mutex::new(b))).collect())
.collect();
let new_locked: Vec<Arc<Mutex<ColBuilder>>> = new_src_builders
.into_iter()
.map(|b| Arc::new(Mutex::new(b)))
.collect();
let exist_sink: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_locked
.iter()
.map(|layer| layer.iter().map(Arc::clone).collect())
.collect();
let new_sink: Vec<Arc<Mutex<ColBuilder>>> = new_locked.iter().map(Arc::clone).collect();
let dst_layers_t2 = Arc::clone(&dst_layers);
let new_mphf_t2 = new_mphf.clone();
let pass2_err: Arc<Mutex<Option<String>>> = Arc::new(Mutex::new(None));
let err_cap2 = Arc::clone(&pass2_err);
let capacity = 2;
let throttled_pass2 = throttle(pass2_items.into_iter(), max_open);
let pipeline2 = Pipeline::new(
make_source!(
Pass2Data,
throttled_pass2.map(|t| {
let (col_offset, src_n, unitigs_path, src_layer) = t.item;
(col_offset, src_n, unitigs_path, src_layer, t.guard)
}),
SrcLayer
),
vec![
Stage::Flat(Arc::new(
move |data: Pass2Data,
push: &PipelineSender<Result<Pass2Data, PipelineError>>,
delta: &PipelineSender<isize>| {
if let Pass2Data::SrcLayer((
col_offset,
src_n,
unitigs_path,
src_layer,
_guard,
)) = data
{
// _guard dropped at end of block, releasing the slot.
// `src_layer` (MPHF + matrix) was already opened up
// front — only the unitigs.bin stream itself is
// opened (and throttled) here.
let reader = match UnitigFileReader::open_sequential(&unitigs_path) {
Ok(r) => r,
Err(e) => {
*err_cap2.lock().unwrap() = Some(e.to_string());
delta.send(-1).ok();
return;
}
};
const BATCH: usize = 4096;
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
let mut count: isize = 0;
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
batch.push(kmer);
if batch.len() == BATCH {
let b = std::mem::replace(&mut batch, Vec::with_capacity(BATCH));
push.send(Ok(Pass2Data::RawBatch((
col_offset,
src_n,
Arc::clone(&src_layer),
b,
))))
.ok();
count += 1;
}
}
if !batch.is_empty() {
push.send(Ok(Pass2Data::RawBatch((
col_offset, src_n, src_layer, batch,
))))
.ok();
count += 1;
}
delta.send(count - 1).ok();
}
},
) as SharedFlatFn<Pass2Data>),
make_transform!(
Pass2Data,
{
move |(col_offset, src_n, src_layer, kmers): (
usize,
usize,
Arc<KmerLayer>,
Vec<CanonicalKmer>,
)|
-> Vec<(Option<usize>, usize, Vec<(usize, u32)>)> {
// Column-first, one pass for the whole batch: a single
// MPHF hash sweep (`hash_batch`) instead of one hash
// per kmer, then one read pass per source genome
// column (`fill_sub_matrix`) instead of one row read
// per kmer — matches the underlying matrix's own
// column-major layout.
let slots = src_layer.hash_batch(&kmers);
let mut cols: Vec<Vec<u32>> = (0..src_n)
.map(|_| Vec::with_capacity(kmers.len()))
.collect();
src_layer.fill_sub_matrix(&slots, &mut cols);
// Membership against dst, grouped by dst layer
// instead of interleaved kmer by kmer — same
// grouping `PartitionCache::find_presence_batch`
// does in `obikphylo`, via the shared primitive.
let (by_dst_layer, not_found) = find_batch_grouped(&dst_layers_t2, &kmers);
// Genome-major within each layer group (not kmer-major)
// so every `(layer, col)` pair's values land together —
// one entry per column, not one per `(kmer, genome)`.
let mut ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)> =
Vec::with_capacity(src_n * (by_dst_layer.len() + 1));
for (dst_layer, hits) in by_dst_layer.into_iter().enumerate() {
if hits.is_empty() {
continue;
}
for g in 0..src_n {
let col_vals: Vec<(usize, u32)> = hits
.iter()
.map(|&(idx, slot)| (slot, cols[g][idx]))
.collect();
ops.push((Some(dst_layer), col_offset + g, col_vals));
}
}
if let (Some(mphf), false) = (&new_mphf_t2, not_found.is_empty()) {
let not_found_kmers: Vec<CanonicalKmer> =
not_found.iter().map(|&idx| kmers[idx]).collect();
let new_slots = mphf.index_batch(&not_found_kmers);
for g in 0..src_n {
let col_vals: Vec<(usize, u32)> = not_found
.iter()
.zip(new_slots.iter())
.map(|(&idx, &slot)| (slot, cols[g][idx]))
.collect();
ops.push((None, col_offset + g, col_vals));
}
}
ops
}
},
RawBatch,
WriteBatch
),
],
make_sink!(
Pass2Data,
{
move |ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)>| {
for (layer_opt, col, col_vals) in ops {
let builder = match layer_opt {
Some(l) => &exist_sink[l][col],
None => &new_sink[col],
};
let mut b = builder.lock().unwrap();
for (slot, val) in col_vals {
b.set_val(slot, val);
}
}
}
},
WriteBatch
),
);
WorkerPool::new(pipeline2, n_workers, capacity).run();
debug!(
"partition {i}: pass2 pipeline done in {:.3}s",
t_pass2.elapsed().as_secs_f64()
);
if let Some(msg) = Arc::try_unwrap(pass2_err)
.unwrap_or_else(|_| panic!("pass2: pass2_err not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
{
return Err(OKIError::InvalidData {
context: "merge pass2",
detail: msg,
});
}
let t_close = std::time::Instant::now();
// ── Close builders ──────────────────────────────────────────────────
for (mb, builders) in exist_mbs.into_iter().zip(exist_locked.into_iter()) {
for b in builders {
Arc::try_unwrap(b)
.unwrap_or_else(|_| panic!("pass2: exist_builder not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
.close()?;
}
mb.close().map_err(OKIError::Io)?;
}
for b in new_locked {
Arc::try_unwrap(b)
.unwrap_or_else(|_| panic!("pass2: new_builder not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
.close()?;
}
if let Some(mb) = new_mb {
mb.close().map_err(OKIError::Io)?;
}
debug!(
"partition {i}: builders closed in {:.3}s",
t_close.elapsed().as_secs_f64()
);
Ok(n_new)
}
+304
View File
@@ -0,0 +1,304 @@
//! Merging a source partition's new layer into a destination partition's
//! de Bruijn graph union (pass 1) then column fill (pass 2). Free function,
//! not a `KmerIndex` method — `KmerIndex` stays the stateless
//! `Index { Partition { Layer } }` data model; this is one of the
//! algorithms that reads/writes through it, living here in `obikmerge`
//! instead.
//!
//! Split: this file orchestrates one partition (pass 1, new-layer setup,
//! loop over source groups); [`pass2`] holds the per-group column-fill
//! pipeline.
mod pass2;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use tracing::debug;
use obicompactvec::{ColBuilder, MatrixBuilder};
use obikindex::layer::{IndexMode, KmerLayer, TypedLayer};
use obikindex::{KmerIndex, OKIError, OKIResult};
use obikindexer::{build_graph, materialize_layer};
use obikseq::CanonicalKmer;
use obiskio::UnitigFileReader;
use crate::MergeMode;
use pass2::run_pass2;
// `MatrixBuilder::new`/`resume` take a `presence: bool` — `MergeMode` maps
// to it directly (`Presence` → `true`, `Count` → `false`).
fn presence(mode: MergeMode) -> bool {
mode == MergeMode::Presence
}
// ── merge_partition ────────────────────────────────────────────────────────
/// Merge `sources` into destination partition `i`.
///
/// Each entry in `sources` is `(index, n_genomes)` where `n_genomes` is the
/// number of genome columns that source contributes. A merged index
/// contributes more than one. The total new columns added to the
/// destination is `sum(n_genomes)`.
///
/// `n_dst_genomes` is the number of genome columns already in the
/// destination matrices (copied from the bootstrap source before this
/// call).
pub(crate) fn merge_partition(
dst: &KmerIndex,
i: usize,
sources: &[(&KmerIndex, usize)],
mode: MergeMode,
n_dst_genomes: usize,
block_bits: u8,
evidence: &IndexMode,
) -> OKIResult<usize> {
let dst_partition = dst.partition(i)?;
let n_dst_layers = dst_partition.n_layers();
// First merge in presence mode: materialize implicit single-genome
// presence matrices on existing layers (all slots true — every kmer in
// those layers belongs to genome_0) so a resumed builder can append
// source columns onto them below.
if n_dst_genomes == 1 && mode == MergeMode::Presence {
for l in 0..n_dst_layers {
let layer = dst_partition.layer(l)?.open()?;
TypedLayer::<()>::init_presence_matrix(layer.dir(), layer.n())?;
}
}
// dst's `0..n_dst_layers` layers, opened once for the whole partition
// and shared by pass 1 (filter) and every pass-2 group (membership).
// Opened directly, not via `IndexCache`: `obikindexer::build_graph`/
// `obipipeline::WorkerPool` are plain-OS-thread pipelines requiring
// `Send + Sync + 'static`, and `IndexCache<'_>` always borrows
// `&KmerIndex` — only owned `KmerLayer`s can cross that boundary.
let dst_layers: Vec<KmerLayer> = (0..n_dst_layers)
.map(|l| dst_partition.layer(l)?.open())
.collect::<OKIResult<_>>()?;
let dst_layers = Arc::new(dst_layers);
// ── Pass 1: pipeline — parallel file read + dst filter + graph fill ────
//
// Source : list of unitigs.bin paths (one per source × layer)
// Flat : open file, emit Vec<CanonicalKmer> batches (BeeGFS parallel I/O)
// Transform: filter via dst_layers — Arc<[KmerLayer]>: Sync
// Sink : push new kmers into GraphDeBruijn (single thread, no locks needed)
//
// Only the unitigs.bin *paths* are needed here: no source layer is
// opened (`KmerLayer::new` is the unopened `Empty` variant, whose
// `unitigs_path` is valid), so pass 1 holds no source mapping.
let mut unitig_paths: Vec<PathBuf> = Vec::new();
for (src, _) in sources {
let partition = src.partition(i)?;
for l in 0..partition.n_layers() {
let p = KmerLayer::new(&partition, l).unitigs_path();
if p.exists() {
unitig_paths.push(p);
}
}
}
let n_src_layers = unitig_paths.len();
debug!("partition {i}: de Bruijn graph build start — {n_src_layers} source layer(s)");
const BATCH: usize = 4096;
let n_workers = rayon::current_num_threads().min(16).max(4);
// At most 2 files open simultaneously: keeps n_workers-2 workers free
// for the Transform stage. Each open file monopolises one worker for the
// full duration of its read, so this must stay well below n_workers.
let max_open = 2;
let dst_filter = Arc::clone(&dst_layers);
let g = build_graph(
unitig_paths.into_iter(),
move |path: PathBuf, emit: &mut dyn FnMut(Vec<CanonicalKmer>)| -> OKIResult<()> {
let reader = UnitigFileReader::open_sequential(&path)?;
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
batch.push(kmer);
if batch.len() == BATCH {
emit(std::mem::replace(&mut batch, Vec::with_capacity(BATCH)));
}
}
if !batch.is_empty() {
emit(batch);
}
Ok(())
},
move |kmer| dst_filter.iter().all(|l| l.find_slot(kmer).is_none()),
n_workers,
max_open,
)?;
let any_new = g.len() > 0;
debug!(
"partition {i}: de Bruijn graph done — {} new kmers",
g.len()
);
// Build new layer from de Bruijn graph if there are new kmers.
let new_layer_idx = n_dst_layers;
let new_layer_dir = KmerLayer::new(&dst_partition, new_layer_idx)
.dir()
.to_path_buf();
let n_new = if any_new {
debug!("partition {i}: unitig traversal start — {} nodes", g.len());
let n_nodes = materialize_layer(g, &new_layer_dir, block_bits, evidence)?;
debug!("partition {i}: MPHF build done");
n_nodes
} else {
drop(g);
0
};
let t_open = std::time::Instant::now();
let new_mphf: Option<Arc<obikindex::layer::MphfOnly>> = if any_new {
Some(Arc::new(obikindex::layer::MphfOnly::open(&new_layer_dir)?))
} else {
None
};
debug!(
"partition {i}: MPHF open in {:.3}s",
t_open.elapsed().as_secs_f64()
);
let matrix_dir = |layer_dir: &Path| match mode {
MergeMode::Presence => layer_dir.join("presence"),
MergeMode::Count => layer_dir.join("counts"),
};
// ── New layer matrix: absent (dst-genome) columns only ────────────────
// Created and closed here; every source group below resumes it and
// appends its own columns, exactly like the existing dst layers.
let new_data_dir: Option<PathBuf> = if any_new {
let data_dir = matrix_dir(&new_layer_dir);
fs::create_dir_all(&data_dir)?;
let mut mb = MatrixBuilder::new(presence(mode), n_new, &data_dir).map_err(OKIError::Io)?;
for _ in 0..n_dst_genomes {
mb.add_absent_col().map_err(OKIError::Io)?;
}
mb.close().map_err(OKIError::Io)?;
Some(data_dir)
} else {
None
};
let exist_data_dirs: Vec<PathBuf> = dst_layers.iter().map(|l| matrix_dir(l.dir())).collect();
// ── Pass 2, by consecutive groups of sources ──────────────────────────
//
// Only one group's source layers and destination columns are alive at a
// time, so the number of live mappings per partition no longer grows
// with the number of sources. SRC_GROUP matches `max_open`: pass 2
// never streams more than `max_open` unitigs.bin files at once, so a
// larger group would only keep more layers/columns open without adding
// read parallelism.
//
// Invariant: groups are visited in source order and each appends its
// `group_n` columns in source order, so the absolute column of genome
// `g` of a source is `n_dst_genomes + col_base + offset_local + g` —
// the same column as when every source was processed in one session.
//
// Memory: the group's columns are built in RAM (`add_mem_col`), at most
// `SRC_GROUP × src_n × (L_dst + 1)` live columns per partition, each of
// `n_layer / 8` bytes (presence) or `n_layer` bytes + overflow (counts):
// about `SRC_GROUP × src_n × N_part × (1/8 | 1)` bytes, `N_part` being
// the partition's k-mer count over all dst layers. For mono-genome
// sources this stays below pass 1's de Bruijn graph (≥ ~19 B per new
// k-mer) unless the new layer is tiny next to the existing ones
// (incremental merge into a large index).
const SRC_GROUP: usize = 2;
let t_pass2 = std::time::Instant::now();
let mut col_base = 0usize;
for group in sources.chunks(SRC_GROUP) {
let group_n: usize = group.iter().map(|(_, n)| *n).sum();
// This group's source layers, for this partition only. Opened
// before any builder so errors propagate via `?` before any worker
// thread is spawned. `col_offset` is local to the group.
let mut items: Vec<(usize, usize, PathBuf, Arc<KmerLayer>)> = Vec::new();
let mut col_offset = 0usize;
for (src, src_n) in group {
let partition = src.partition(i)?;
for l in 0..partition.n_layers() {
let layer = Arc::new(partition.layer(l)?.open()?);
let unitigs_path = layer.unitigs_path();
if unitigs_path.exists() {
items.push((col_offset, *src_n, unitigs_path, layer));
}
}
col_offset += src_n;
}
let expected_cols = n_dst_genomes + col_base;
let open_group = |data_dir: &Path| -> OKIResult<(MatrixBuilder, Vec<ColBuilder>)> {
let mut mb = MatrixBuilder::resume(presence(mode), data_dir).map_err(OKIError::Io)?;
if mb.n_cols() != expected_cols {
return Err(OKIError::InvalidData {
context: "merge pass2",
detail: format!(
"{}: {} columns, expected {expected_cols}",
data_dir.display(),
mb.n_cols()
),
});
}
// In-memory columns: no sparse file / `MmapMut` on the write
// path, so a block the FS cannot allocate is an `io::Error` at
// `close`, not a SIGBUS (see `MatrixBuilder::add_mem_col`).
let cols = (0..group_n).map(|_| mb.add_mem_col()).collect();
Ok((mb, cols))
};
let (exist_mbs, exist_cols): (Vec<MatrixBuilder>, Vec<Vec<ColBuilder>>) = exist_data_dirs
.iter()
.map(|d| open_group(d))
.collect::<OKIResult<Vec<_>>>()?
.into_iter()
.unzip();
let (new_mb, new_cols) = match &new_data_dir {
Some(d) => {
let (mb, cols) = open_group(d)?;
(Some(mb), cols)
}
None => (None, vec![]),
};
// `items` (and with it this group's source layers) is consumed by
// the pipeline and dropped when it returns.
let (exist_cols, new_cols) = run_pass2(
items,
&dst_layers,
&new_mphf,
exist_cols,
new_cols,
n_workers,
max_open,
)?;
for (mb, cols) in exist_mbs.into_iter().zip(exist_cols) {
for c in cols {
c.close()?;
}
mb.close().map_err(OKIError::Io)?;
}
for c in new_cols {
c.close()?;
}
if let Some(mb) = new_mb {
mb.close().map_err(OKIError::Io)?;
}
col_base += group_n;
}
debug!(
"partition {i}: pass2 done in {:.3}s",
t_pass2.elapsed().as_secs_f64()
);
Ok(n_new)
}
+271
View File
@@ -0,0 +1,271 @@
//! Pass 2 of a partition merge, for one group of sources: stream each
//! source layer's k-mers, read their values from the source matrix, locate
//! them in dst (existing layers, else the new layer's MPHF) and write them
//! into the group's destination columns.
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use obipipeline::{
Pipeline, PipelineError, PipelineSender, SharedFlatFn, Stage, ThrottleGuard, WorkerPool,
make_sink, make_source, make_transform, throttle,
};
use obicompactvec::ColBuilder;
use obikindex::layer::KmerLayer;
use obikindex::{OKIError, OKIResult};
use obikseq::CanonicalKmer;
use obiskio::UnitigFileReader;
// ── dst cross-layer lookup ──────────────────────────────────────────────────
/// Batch membership lookup across `dst_layers` — first layer that carries
/// each kmer wins. Groups hits by layer index (as `(kmer batch index,
/// slot)` pairs) rather than returning them in query order, so pass 2 can
/// follow with one column-major write per `(layer, column)` instead of
/// interleaving across layers kmer by kmer. The kmers with no hit in any
/// layer come back as a separate index list, for the new-layer MPHF batch
/// lookup on a miss.
fn find_batch_grouped(
layers: &[KmerLayer],
kmers: &[CanonicalKmer],
) -> (Vec<Vec<(usize, usize)>>, Vec<usize>) {
let mut by_layer: Vec<Vec<(usize, usize)>> = vec![Vec::new(); layers.len()];
let mut not_found: Vec<usize> = Vec::new();
for (idx, &kmer) in kmers.iter().enumerate() {
match layers
.iter()
.enumerate()
.find_map(|(li, l)| l.find_slot(kmer).map(|slot| (li, slot)))
{
Some((li, slot)) => by_layer[li].push((idx, slot)),
None => not_found.push(idx),
}
}
(by_layer, not_found)
}
// ── per-group pipeline ─────────────────────────────────────────────────────
/// Pass 2 for one group of sources: stream each source layer's unitigs,
/// look its k-mers up in the source matrix and in dst (existing layers,
/// then the new layer's MPHF), and write the values into the group's
/// column builders. `items` carry group-local column offsets; the builders
/// are handed back (still open) for the caller to close.
#[allow(clippy::type_complexity)]
pub(super) fn run_pass2(
items: Vec<(usize, usize, PathBuf, Arc<KmerLayer>)>,
dst_layers: &Arc<Vec<KmerLayer>>,
new_mphf: &Option<Arc<obikindex::layer::MphfOnly>>,
exist_builders: Vec<Vec<ColBuilder>>,
new_builders: Vec<ColBuilder>,
n_workers: usize,
max_open: usize,
) -> OKIResult<(Vec<Vec<ColBuilder>>, Vec<ColBuilder>)> {
enum Pass2Data {
SrcLayer((usize, usize, PathBuf, Arc<KmerLayer>, ThrottleGuard)),
RawBatch((usize, usize, Arc<KmerLayer>, Vec<CanonicalKmer>)),
/// One entry per `(dst layer, column)` touched by this batch, each
/// carrying every `(slot, value)` pair for that column together —
/// column-major, so the sink below locks that column's `ColBuilder`
/// once per batch and writes the whole run, instead of once per
/// `(kmer, genome)` pair.
WriteBatch(Vec<(Option<usize>, usize, Vec<(usize, u32)>)>),
}
let exist_locked: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_builders
.into_iter()
.map(|layer| layer.into_iter().map(|b| Arc::new(Mutex::new(b))).collect())
.collect();
let new_locked: Vec<Arc<Mutex<ColBuilder>>> = new_builders
.into_iter()
.map(|b| Arc::new(Mutex::new(b)))
.collect();
let exist_sink: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_locked
.iter()
.map(|layer| layer.iter().map(Arc::clone).collect())
.collect();
let new_sink: Vec<Arc<Mutex<ColBuilder>>> = new_locked.iter().map(Arc::clone).collect();
let dst_layers_t2 = Arc::clone(dst_layers);
let new_mphf_t2 = new_mphf.clone();
let pass2_err: Arc<Mutex<Option<String>>> = Arc::new(Mutex::new(None));
let err_cap2 = Arc::clone(&pass2_err);
let capacity = 2;
let throttled_pass2 = throttle(items.into_iter(), max_open);
let pipeline2 = Pipeline::new(
make_source!(
Pass2Data,
throttled_pass2.map(|t| {
let (col_offset, src_n, unitigs_path, src_layer) = t.item;
(col_offset, src_n, unitigs_path, src_layer, t.guard)
}),
SrcLayer
),
vec![
Stage::Flat(Arc::new(
move |data: Pass2Data,
push: &PipelineSender<Result<Pass2Data, PipelineError>>,
delta: &PipelineSender<isize>| {
if let Pass2Data::SrcLayer((
col_offset,
src_n,
unitigs_path,
src_layer,
_guard,
)) = data
{
// _guard dropped at end of block, releasing the slot.
// `src_layer` (MPHF + matrix) was already opened by
// the caller — only the unitigs.bin stream itself is
// opened (and throttled) here.
let reader = match UnitigFileReader::open_sequential(&unitigs_path) {
Ok(r) => r,
Err(e) => {
*err_cap2.lock().unwrap() = Some(e.to_string());
delta.send(-1).ok();
return;
}
};
const BATCH: usize = 4096;
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
let mut count: isize = 0;
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
batch.push(kmer);
if batch.len() == BATCH {
let b = std::mem::replace(&mut batch, Vec::with_capacity(BATCH));
push.send(Ok(Pass2Data::RawBatch((
col_offset,
src_n,
Arc::clone(&src_layer),
b,
))))
.ok();
count += 1;
}
}
if !batch.is_empty() {
push.send(Ok(Pass2Data::RawBatch((
col_offset, src_n, src_layer, batch,
))))
.ok();
count += 1;
}
delta.send(count - 1).ok();
}
},
) as SharedFlatFn<Pass2Data>),
make_transform!(
Pass2Data,
{
move |(col_offset, src_n, src_layer, kmers): (
usize,
usize,
Arc<KmerLayer>,
Vec<CanonicalKmer>,
)|
-> Vec<(Option<usize>, usize, Vec<(usize, u32)>)> {
// Column-first, one pass for the whole batch: a single
// MPHF hash sweep (`hash_batch`) instead of one hash
// per kmer, then one read pass per source genome
// column (`fill_sub_matrix`) instead of one row read
// per kmer — matches the underlying matrix's own
// column-major layout.
let slots = src_layer.hash_batch(&kmers);
let mut cols: Vec<Vec<u32>> = (0..src_n)
.map(|_| Vec::with_capacity(kmers.len()))
.collect();
src_layer.fill_sub_matrix(&slots, &mut cols);
// Membership against dst, grouped by dst layer
// instead of interleaved kmer by kmer — same
// grouping `PartitionCache::find_presence_batch`
// does in `obikphylo`, via the shared primitive.
let (by_dst_layer, not_found) = find_batch_grouped(&dst_layers_t2, &kmers);
// Genome-major within each layer group (not kmer-major)
// so every `(layer, col)` pair's values land together —
// one entry per column, not one per `(kmer, genome)`.
let mut ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)> =
Vec::with_capacity(src_n * (by_dst_layer.len() + 1));
for (dst_layer, hits) in by_dst_layer.into_iter().enumerate() {
if hits.is_empty() {
continue;
}
for g in 0..src_n {
let col_vals: Vec<(usize, u32)> = hits
.iter()
.map(|&(idx, slot)| (slot, cols[g][idx]))
.collect();
ops.push((Some(dst_layer), col_offset + g, col_vals));
}
}
if let (Some(mphf), false) = (&new_mphf_t2, not_found.is_empty()) {
let not_found_kmers: Vec<CanonicalKmer> =
not_found.iter().map(|&idx| kmers[idx]).collect();
let new_slots = mphf.index_batch(&not_found_kmers);
for g in 0..src_n {
let col_vals: Vec<(usize, u32)> = not_found
.iter()
.zip(new_slots.iter())
.map(|(&idx, &slot)| (slot, cols[g][idx]))
.collect();
ops.push((None, col_offset + g, col_vals));
}
}
ops
}
},
RawBatch,
WriteBatch
),
],
make_sink!(
Pass2Data,
{
move |ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)>| {
for (layer_opt, col, col_vals) in ops {
let builder = match layer_opt {
Some(l) => &exist_sink[l][col],
None => &new_sink[col],
};
let mut b = builder.lock().unwrap();
for (slot, val) in col_vals {
b.set_val(slot, val);
}
}
}
},
WriteBatch
),
);
WorkerPool::new(pipeline2, n_workers, capacity).run();
if let Some(msg) = Arc::try_unwrap(pass2_err)
.unwrap_or_else(|_| panic!("pass2: pass2_err not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
{
return Err(OKIError::InvalidData {
context: "merge pass2",
detail: msg,
});
}
let unlock = |b: Arc<Mutex<ColBuilder>>| -> ColBuilder {
Arc::try_unwrap(b)
.unwrap_or_else(|_| panic!("pass2: column builder not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
};
let exist = exist_locked
.into_iter()
.map(|layer| layer.into_iter().map(unlock).collect())
.collect();
let new = new_locked.into_iter().map(unlock).collect();
Ok((exist, new))
}
@@ -5,12 +5,26 @@
//! masking ([`super::masking::survives_masking`]). //! masking ([`super::masking::survives_masking`]).
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikseq::RawKmer;
use rkyv::{Archive, Deserialize, Serialize}; use rkyv::{Archive, Deserialize, Serialize};
use super::masking::iupac_code; use super::masking::iupac_code;
use super::scan::LayerReducer; use super::scan::LayerReducer;
use super::subsample::SurvivingFamily; use super::subsample::SurvivingFamily;
/// Where an alignment column comes from: enough to give the same family the
/// same virtual coordinates every time for a given index (VCF `CHROM`/`POS`)
/// and an index-independent label (`family_id`).
#[derive(Clone, Copy, Archive, Serialize, Deserialize)]
pub struct SiteLocus {
pub partition: usize,
pub layer: usize,
/// `scan_layer_families`'s iteration-order index within `(partition, layer)`.
pub family_idx: usize,
/// Raw encoding of [`obikseq::CanonicalKmer::family_id`].
pub family_id: RawKmer,
}
/// `sequences[i]` is `genome_indices[i]`'s IUPAC-coded row — every row the /// `sequences[i]` is `genome_indices[i]`'s IUPAC-coded row — every row the
/// same length (one byte per sampled family, in the order /// same length (one byte per sampled family, in the order
/// [`super::scan::sampled_scan`] produced them: layer by layer, /// [`super::scan::sampled_scan`] produced them: layer by layer,
@@ -27,6 +41,8 @@ pub struct SnpAlignment {
/// order — the caller's authoritative mapping back to genome labels, so /// order — the caller's authoritative mapping back to genome labels, so
/// it never has to re-derive which genomes were excluded itself. /// it never has to re-derive which genomes were excluded itself.
pub genome_indices: Vec<usize>, pub genome_indices: Vec<usize>,
/// `sites[j]` describes column `j` of every row.
pub sites: Vec<SiteLocus>,
} }
/// Accumulates a [`SnpAlignment`] over a [`super::scan::sampled_scan`] /// Accumulates a [`SnpAlignment`] over a [`super::scan::sampled_scan`]
@@ -40,6 +56,7 @@ pub struct SnpAlignment {
pub struct AlignmentReducer { pub struct AlignmentReducer {
genome_indices: Vec<usize>, genome_indices: Vec<usize>,
sequences: Vec<Vec<u8>>, sequences: Vec<Vec<u8>>,
sites: Vec<SiteLocus>,
} }
impl AlignmentReducer { impl AlignmentReducer {
@@ -49,16 +66,22 @@ impl AlignmentReducer {
.filter(|&g| !excluded.get(g).copied().unwrap_or(false)) .filter(|&g| !excluded.get(g).copied().unwrap_or(false))
.collect(); .collect();
let sequences = vec![Vec::new(); genome_indices.len()]; let sequences = vec![Vec::new(); genome_indices.len()];
Self { genome_indices, sequences } Self { genome_indices, sequences, sites: Vec::new() }
} }
pub fn finish(self) -> SnpAlignment { pub fn finish(self) -> SnpAlignment {
SnpAlignment { sequences: self.sequences, genome_indices: self.genome_indices } SnpAlignment { sequences: self.sequences, genome_indices: self.genome_indices, sites: self.sites }
} }
} }
impl LayerReducer for AlignmentReducer { impl LayerReducer for AlignmentReducer {
fn reduce(&mut self, _partition: usize, _layer: usize, survivors: &[SurvivingFamily]) { fn reduce(&mut self, partition: usize, layer: usize, survivors: &[SurvivingFamily]) {
self.sites.extend(survivors.iter().map(|f| SiteLocus {
partition,
layer,
family_idx: f.family_idx,
family_id: f.kmer.family_id().raw(),
}));
reduce_alignment(survivors, &self.genome_indices, &mut self.sequences); reduce_alignment(survivors, &self.genome_indices, &mut self.sequences);
} }
} }
@@ -0,0 +1,270 @@
//! VCF rendering of a [`SnpAlignment`].
//!
//! One contig per `(partition, layer)` (`obikmer_<partition>_<layer>`), `POS`
//! = the family's `family_idx + 1` inside it: for a given index and sibling
//! annex the same family always gets the same virtual coordinates, and
//! sorting by `(partition, layer, family_idx)` is what makes the file a
//! well-formed (sorted) VCF. `ID` is the index-independent
//! [`obikseq::CanonicalKmer::family_id`] label (centre base masked as `N`).
//!
//! `REF`/`ALT` are the unambiguous bases (`A`/`C`/`G`/`T` cells of the
//! alignment) observed at the site, in the alignment's own convention: the
//! central base of the family member's *canonical* form, which is not
//! necessarily the strand of the `ID` k-mer. A `-` cell (family absent from
//! the genome — a real state, unlike `?`, which `--free-loss` produces and
//! which means missing) is the symbolic allele `<ABS>`, declared by a `##ALT`
//! line and listed after the bases. (`*` is not used: in VCF it means an
//! allele missing because of an overlapping deletion.)
//!
//! Ploidy of the genotypes is a caller choice. Haploid: one allele per
//! genome; an ambiguous cell is missing. Diploid: a single base `X` is
//! `X/X`, a two-base code is the heterozygote, `<ABS>` is `<ABS>/<ABS>`; codes with
//! three or four bases (and `?`) are missing (`./.`). In both modes a
//! genome gets the allele indices in ascending order.
use std::io::{self, Write};
use obikseq::CanonicalKmer;
use super::alignment::SnpAlignment;
use super::masking::iupac_state;
const BASES: [u8; 4] = *b"ACGT";
/// Internal marker of the absence allele (the alignment's own `-`), written
/// as [`ABSENT_ALT`].
const ABSENT: u8 = b'-';
/// How the absence allele appears in the `ALT` column.
const ABSENT_ALT: &[u8] = b"<ABS>";
fn base_index(cell: u8) -> Option<usize> {
BASES.iter().position(|&b| b == cell)
}
/// One cell as alleles: `None` = missing; else its (one or, diploid, two)
/// allele characters — a base, or [`ABSENT`] for absence. A cell with more forms
/// than the ploidy allows, or `?`, is missing.
fn cell_alleles(cell: u8, diploid: bool) -> Option<[u8; 2]> {
let state = iupac_state(cell)?;
let allele = |bit: u8| if bit == 0 { ABSENT } else { BASES[bit.trailing_zeros() as usize] };
match state.count_ones() {
0 => Some([ABSENT, ABSENT]),
1 => Some([allele(state), allele(state)]),
2 if diploid => {
let low = state & state.wrapping_neg();
Some([allele(low), allele(state ^ low)])
}
_ => None,
}
}
/// `REF` base (index into `ACGT`) of one site: the most frequent unambiguous
/// base, lexicographic minimum on ties. `None` when no cell carries one.
fn majority_base(counts: &[usize; 4]) -> Option<usize> {
(0..4)
.filter(|&b| counts[b] > 0)
.max_by_key(|&b| (counts[b], std::cmp::Reverse(b)))
}
impl SnpAlignment {
/// Writes this alignment as a VCF (haploid `GT` per genome). `labels` is
/// indexed by the index's own genome numbering ([`Self::genome_indices`]
/// values). `diploid` selects the genotype ploidy (see the module docs).
/// `REF` is the per-site majority allele (each allele copy counted once).
/// Returns the number of sites written: a site with no base among its
/// calls has no `REF` to report and is skipped.
pub fn write_vcf<W: Write>(&self, labels: &[String], diploid: bool, out: &mut W) -> io::Result<usize> {
let n_sites = self.sequences.first().map_or(0, Vec::len);
if self.sites.len() != n_sites {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
format!("alignment has {n_sites} column(s) but {} site locus record(s)", self.sites.len()),
));
}
let mut order: Vec<usize> = (0..n_sites).collect();
order.sort_unstable_by_key(|&j| {
let s = &self.sites[j];
(s.partition, s.layer, s.family_idx)
});
writeln!(out, "##fileformat=VCFv4.5")?;
writeln!(out, "##source=obikmer")?;
let mut last = None;
for &j in &order {
let s = &self.sites[j];
if last != Some((s.partition, s.layer)) {
writeln!(out, "##contig=<ID=obikmer_{}_{}>", s.partition, s.layer)?;
last = Some((s.partition, s.layer));
}
}
writeln!(out, "##ALT=<ID=ABS,Description=\"Family absent from the genome\">")?;
writeln!(out, "##FORMAT=<ID=GT,Number=1,Type=String,Description=\"Genotype ({})\">", if diploid { "diploid" } else { "haploid" })?;
write!(out, "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT")?;
for &g in &self.genome_indices {
write!(out, "\t{}", labels[g])?;
}
writeln!(out)?;
let mut written = 0usize;
let mut locus = Vec::new();
let mut calls: Vec<Option<[u8; 2]>> = Vec::with_capacity(self.sequences.len());
let copies = if diploid { 2 } else { 1 };
for &j in &order {
calls.clear();
calls.extend(self.sequences.iter().map(|seq| cell_alleles(seq[j], diploid)));
let mut counts = [0usize; 4];
let mut absent = false;
for call in calls.iter().flatten() {
for &a in &call[..copies] {
match base_index(a) {
Some(b) => counts[b] += 1,
None => absent = true,
}
}
}
let Some(ref_base) = majority_base(&counts) else {
continue;
};
let mut alleles = vec![BASES[ref_base]];
alleles.extend((0..4).filter(|&b| b != ref_base && counts[b] > 0).map(|b| BASES[b]));
if absent {
alleles.push(ABSENT);
}
let s = &self.sites[j];
locus.clear();
CanonicalKmer::from_raw_unchecked(s.family_id).write_family_id_ascii(&mut locus)?;
write!(out, "obikmer_{}_{}\t{}\t", s.partition, s.layer, s.family_idx + 1)?;
out.write_all(&locus)?;
write!(out, "\t{}\t", alleles[0] as char)?;
if alleles.len() == 1 {
out.write_all(b".")?;
} else {
for (i, &a) in alleles[1..].iter().enumerate() {
if i > 0 {
out.write_all(b",")?;
}
if a == ABSENT {
out.write_all(ABSENT_ALT)?;
} else {
out.write_all(&[a])?;
}
}
}
out.write_all(b"\t.\t.\t.\tGT")?;
for call in &calls {
let idx = |a: u8| alleles.iter().position(|&x| x == a);
match call.map(|c| (idx(c[0]), idx(c[1]))) {
Some((Some(x), Some(y))) if diploid => write!(out, "\t{}/{}", x.min(y), x.max(y))?,
Some((Some(x), _)) => write!(out, "\t{x}")?,
_ => out.write_all(if diploid { b"\t./." } else { b"\t." })?,
}
}
out.write_all(b"\n")?;
written += 1;
}
Ok(written)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::siblings::SiteLocus;
fn alignment(rows: &[&[u8]], sites: Vec<SiteLocus>) -> SnpAlignment {
SnpAlignment {
sequences: rows.iter().map(|r| r.to_vec()).collect(),
genome_indices: (0..rows.len()).collect(),
sites,
}
}
fn site(partition: usize, layer: usize, family_idx: usize) -> SiteLocus {
// Same global k as the crate's other tests (`siblingannex.rs`): the
// global is process-wide here, so it must not conflict.
obikseq::params::set_k(11);
SiteLocus { partition, layer, family_idx, family_id: 0 }
}
fn first_record(a: &SnpAlignment) -> Vec<String> {
first_record_ploidy(a, false)
}
fn first_record_ploidy(a: &SnpAlignment, diploid: bool) -> Vec<String> {
let labels: Vec<String> = (0..a.sequences.len()).map(|i| format!("g{i}")).collect();
let mut out = Vec::new();
a.write_vcf(&labels, diploid, &mut out).unwrap();
let text = String::from_utf8(out).unwrap();
let line = text.lines().find(|l| !l.starts_with('#')).unwrap();
line.split('\t').map(String::from).collect()
}
#[test]
fn reference_is_majority_then_lexicographic() {
assert_eq!(majority_base(&[1, 3, 0, 0]), Some(1));
assert_eq!(majority_base(&[2, 2, 0, 0]), Some(0));
assert_eq!(majority_base(&[0, 0, 0, 0]), None);
}
#[test]
fn genotypes_follow_ref_then_alt_order() {
// Column 0: A,G,G,R,? -> REF=G (majority), ALT=A; Column 1: all ambiguous -> skipped.
let a = alignment(
&[b"AY", b"GY", b"GR", b"RR", b"?R"],
vec![site(0, 0, 4), site(0, 0, 9)],
);
let f = first_record(&a);
assert_eq!(&f[..2], ["obikmer_0_0", "5"]);
assert_eq!(&f[3..5], ["G", "A"]);
assert_eq!(&f[9..], ["1", "0", "0", ".", "."]);
}
#[test]
fn absence_is_the_abs_allele_after_the_bases() {
// A, T, - (family absent), ? (missing): REF=A, ALT=T,<ABS>.
let a = alignment(&[b"A", b"T", b"-", b"?"], vec![site(0, 0, 0)]);
let f = first_record(&a);
assert_eq!(&f[3..5], ["A", "T,<ABS>"]);
assert_eq!(&f[9..], ["0", "1", "2", "."]);
// Only presence/absence differs: REF=A, ALT=<ABS>.
let a = alignment(&[b"A", b"-"], vec![site(0, 0, 0)]);
let f = first_record(&a);
assert_eq!(&f[3..5], ["A", "<ABS>"]);
assert_eq!(&f[9..], ["0", "1"]);
}
#[test]
fn iupac_state_inverts_iupac_code() {
for state in 0u8..16 {
assert_eq!(iupac_state(super::super::masking::iupac_code(state)), Some(state));
}
assert_eq!(iupac_state(b'?'), None);
}
#[test]
fn diploid_genotypes() {
// A, R (A/G), G, N (4 forms -> missing), - (absent), ? (missing).
let a = alignment(&[b"A", b"R", b"G", b"N", b"-", b"?"], vec![site(0, 0, 0)]);
let f = first_record_ploidy(&a, true);
// majority: A = 2 + 1 = 3, G = 1 + 2 = 3 -> tie -> A; ALT = G,<ABS>
assert_eq!(&f[3..5], ["A", "G,<ABS>"]);
assert_eq!(&f[9..], ["0/0", "0/1", "1/1", "./.", "2/2", "./."]);
// Same column, haploid: the ambiguous cells are missing.
let f = first_record_ploidy(&a, false);
assert_eq!(&f[9..], ["0", ".", "1", ".", "2", "."]);
}
#[test]
fn sites_sorted_by_contig_then_position() {
let a = alignment(&[b"AA", b"GG"], vec![site(1, 0, 0), site(0, 2, 7)]);
let labels = vec!["x".to_string(), "y".to_string()];
let mut out = Vec::new();
a.write_vcf(&labels, false, &mut out).unwrap();
let text = String::from_utf8(out).unwrap();
let body: Vec<&str> = text.lines().filter(|l| !l.starts_with('#')).collect();
assert!(body[0].starts_with("obikmer_0_2\t8\t"));
assert!(body[1].starts_with("obikmer_1_0\t1\t"));
}
}
@@ -16,10 +16,16 @@
/// real minorant, guarded against rather than dividing by zero). Second /// real minorant, guarded against rather than dividing by zero). Second
/// value: the denominator (`n_genomes_present`) used, for traceability. /// value: the denominator (`n_genomes_present`) used, for traceability.
pub(crate) fn family_entropy(genome_mask: &[u8]) -> Option<(f64, usize)> { pub(crate) fn family_entropy(genome_mask: &[u8]) -> Option<(f64, usize)> {
family_entropy_with(genome_mask, AmbiguityPolicy::Keep)
}
/// [`family_entropy`] over the genomes `policy` keeps as carriers
/// ([`is_carrier`]).
pub(crate) fn family_entropy_with(genome_mask: &[u8], policy: AmbiguityPolicy) -> Option<(f64, usize)> {
let mut freq = [0u32; 16]; // index 0 (∅) never incremented, kept for direct `m as usize` indexing let mut freq = [0u32; 16]; // index 0 (∅) never incremented, kept for direct `m as usize` indexing
let mut present = 0u32; let mut present = 0u32;
for &m in genome_mask { for &m in genome_mask {
if m != 0 { if is_carrier(m, policy) {
freq[m as usize] += 1; freq[m as usize] += 1;
present += 1; present += 1;
} }
@@ -70,6 +76,9 @@ pub(crate) fn family_entropy_4(genome_mask: &[u8]) -> Option<(f64, usize)> {
Some((h, n_single as usize)) Some((h, n_single as usize))
} }
use crate::siblings::AmbiguityPolicy;
use crate::siblings::helpers::is_carrier;
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -104,10 +113,16 @@ mod tests {
#[test] #[test]
fn entropy_under_selects_the_counting_convention() { fn entropy_under_selects_the_counting_convention() {
let masks = [0b0001u8, 0b0001, 0b0011]; // the last genome carries two members let masks = [0b0001u8, 0b0001, 0b0011]; // the last genome carries two members
assert_eq!(family_entropy_under(&masks, true).unwrap().1, 2); use AmbiguityPolicy::{Forbid, Keep, OnlyDiploid};
assert_eq!(family_entropy_under(&masks, false).unwrap().1, 3); assert_eq!(family_entropy_under(&masks, Forbid).unwrap().1, 2);
assert_eq!(family_entropy_under(&masks, true).unwrap().0, 0.0); assert_eq!(family_entropy_under(&masks, Keep).unwrap().1, 3);
assert!(family_entropy_under(&masks, false).unwrap().0 > 0.0); assert_eq!(family_entropy_under(&masks, OnlyDiploid).unwrap().1, 3);
assert_eq!(family_entropy_under(&masks, Forbid).unwrap().0, 0.0);
assert!(family_entropy_under(&masks, Keep).unwrap().0 > 0.0);
// three members: kept by Keep, dropped by OnlyDiploid
let masks = [0b0001u8, 0b0001, 0b0111];
assert_eq!(family_entropy_under(&masks, Keep).unwrap().1, 3);
assert_eq!(family_entropy_under(&masks, OnlyDiploid).unwrap().1, 2);
} }
#[test] #[test]
@@ -117,21 +132,24 @@ mod tests {
} }
} }
/// Shannon entropy (bits) of one family under the counting convention /// Shannon entropy (bits) of one family under the counting convention an
/// `--no-ambiguity` selects: /// [`AmbiguityPolicy`] selects:
/// - `false`: [`family_entropy`] — every genome carrying the family counts, /// - `Keep`: [`family_entropy`] — every genome carrying the family counts,
/// a genome with several members being a state of its own; /// a genome with several members being a state of its own;
/// - `true`: [`family_entropy_4`] — only the genomes carrying exactly one /// - `Forbid` (`--no-ambiguity`): [`family_entropy_4`] — only the genomes
/// member count, the others being missing data, exactly as the alignment /// carrying exactly one member count, the others being missing data,
/// itself will see them. Over those genomes the 15-state entropy can only /// exactly as the alignment itself will see them. Over those genomes the
/// ever take the four single-base states, so the two definitions coincide. /// 15-state entropy can only ever take the four single-base states, so the
/// two definitions coincide;
/// - `OnlyDiploid`: the 15-state entropy over the genomes with one or two
/// members (ten states), the others being missing data.
/// ///
/// Second value: the number of genomes the entropy was computed over. /// Second value: the number of genomes the entropy was computed over.
pub(crate) fn family_entropy_under(genome_mask: &[u8], no_ambiguity: bool) -> Option<(f64, usize)> { pub(crate) fn family_entropy_under(genome_mask: &[u8], policy: AmbiguityPolicy) -> Option<(f64, usize)> {
if no_ambiguity { match policy {
family_entropy_4(genome_mask) AmbiguityPolicy::Keep => family_entropy(genome_mask),
} else { AmbiguityPolicy::Forbid => family_entropy_4(genome_mask),
family_entropy(genome_mask) AmbiguityPolicy::OnlyDiploid => family_entropy_with(genome_mask, policy),
} }
} }
@@ -17,6 +17,7 @@ use super::entropy::{family_entropy, family_entropy_4};
use super::family_scan::{Selection, scan_layer_families}; use super::family_scan::{Selection, scan_layer_families};
use super::is_fast_mode; use super::is_fast_mode;
use super::minorant_selection::non_monomorphic_reindex_layer; use super::minorant_selection::non_monomorphic_reindex_layer;
use crate::siblings::extensions::SiblingBuilder;
/// One variable family, as `--shannon` reports it. /// One variable family, as `--shannon` reports it.
pub(crate) struct FamilyEntropyRow { pub(crate) struct FamilyEntropyRow {
@@ -45,6 +46,7 @@ pub(crate) fn scan_family_entropies(
cache: &IndexCache, cache: &IndexCache,
mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>, mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>,
) -> OKIResult<()> { ) -> OKIResult<()> {
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len(); let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache); let fast_mode = is_fast_mode(cache);
let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions"); let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions");
@@ -65,7 +67,7 @@ pub(crate) fn scan_family_entropies(
n_genomes, n_genomes,
fast_mode, fast_mode,
&Selection::Some(&eligible), &Selection::Some(&eligible),
|family_idx, mask, genome_mask| { |family_idx, _kmer, mask, genome_mask| {
if failure.is_some() { if failure.is_some() {
return; return;
} }
@@ -21,43 +21,87 @@ use super::family_scan::{Selection, scan_layer_families};
use super::masking::is_quartet_informative; use super::masking::is_quartet_informative;
use super::minorant_selection::non_monomorphic_reindex_layer; use super::minorant_selection::non_monomorphic_reindex_layer;
use super::pair_weight::pair_balance_weight; use super::pair_weight::pair_balance_weight;
use crate::siblings::AmbiguityPolicy;
use crate::siblings::quartet_annex::quartet_annex_file_name; use crate::siblings::quartet_annex::quartet_annex_file_name;
use crate::siblings::{FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind}; use crate::siblings::{
FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind, family_overlap_file_name,
};
/// Which annexes a caller needs, each listed once per `--no-ambiguity` /// Which annexes a caller needs, each listed once per ambiguity policy (how
/// counting convention (`false` = genomes carrying several members count as a /// genomes carrying several members of a family are counted) — an empty list
/// state of their own, `true` = they are left out) — an empty list asks for /// asks for none. A pair weight is computed from the family-overlap annex of
/// none. A pair weight is computed from the family-overlap annex of its own /// its own policy, so asking for one asks for the other.
/// convention, so asking for one asks for the other.
#[derive(Clone, Debug, Default)] #[derive(Clone, Debug, Default)]
pub(crate) struct AnnexRequest { pub(crate) struct AnnexRequest {
pub entropy: Vec<bool>, pub entropy: Vec<AmbiguityPolicy>,
pub quartet: Vec<bool>, pub quartet: Vec<AmbiguityPolicy>,
pub overlap: Vec<bool>, pub overlap: Vec<AmbiguityPolicy>,
pub pair_weight: Vec<bool>, pub pair_weight: Vec<AmbiguityPolicy>,
}
/// The derived annexes a run asks to have ready, all under the one ambiguity
/// `policy` of that run (`--sibling-annex` together with the options that
/// say which annexes, and under which convention, the following computations
/// will read). Nothing is built that is not asked for.
#[derive(Clone, Copy, Debug, Default)]
pub struct AnnexSelection {
pub policy: AmbiguityPolicy,
pub entropy: bool,
pub quartet: bool,
pub overlap: bool,
pub pair_weight: bool,
}
impl AnnexSelection {
pub(crate) fn request(&self) -> AnnexRequest {
let one = |wanted: bool| if wanted { vec![self.policy] } else { Vec::new() };
AnnexRequest {
entropy: one(self.entropy),
quartet: one(self.quartet),
overlap: one(self.overlap),
pair_weight: one(self.pair_weight),
}
}
/// Whether any derived annex is asked for.
pub fn is_empty(&self) -> bool {
!(self.entropy || self.quartet || self.overlap || self.pair_weight)
}
} }
impl AnnexRequest { impl AnnexRequest {
/// Everything under both conventions — what `--sibling-annex` precomputes.
pub(crate) fn everything() -> Self {
let both = vec![false, true];
Self { entropy: both.clone(), quartet: both.clone(), overlap: both.clone(), pair_weight: both }
}
/// The overlap conventions to ensure: the requested ones plus every one a /// The overlap conventions to ensure: the requested ones plus every one a
/// requested pair weight is computed from, each once. /// requested pair weight is computed from, each once.
pub(crate) fn overlap_conventions(&self) -> Vec<bool> { pub(crate) fn overlap_conventions(&self) -> Vec<AmbiguityPolicy> {
dedup(self.overlap.iter().chain(&self.pair_weight).copied()) dedup(self.overlap.iter().chain(&self.pair_weight).copied())
} }
/// Whether every annex this request asks for already exists — the same
/// per-file existence test the builders apply, so `true` means they would
/// find nothing to build and no lock is needed. Cheap (no file is opened).
pub(crate) fn is_satisfied(&self, cache: &IndexCache) -> bool {
let overlap = self
.overlap_conventions()
.into_iter()
.all(|p| cache.index().dir().join(family_overlap_file_name(p)).exists());
let pair_weight = self.pair_weight_conventions();
overlap
&& cache.iter_indexed().all(|(_, _, layer)| {
let has = |name: &str| layer.dir().join(name).exists();
self.entropy.iter().all(|&p| has(ScalarKind::Entropy.file_name(p)))
&& self.quartet.iter().all(|&p| has(quartet_annex_file_name(p)))
&& pair_weight.iter().all(|&p| has(ScalarKind::PairWeight.file_name(p)))
})
}
/// The pair-weight conventions requested, each once. /// The pair-weight conventions requested, each once.
pub(crate) fn pair_weight_conventions(&self) -> Vec<bool> { pub(crate) fn pair_weight_conventions(&self) -> Vec<AmbiguityPolicy> {
dedup(self.pair_weight.iter().copied()) dedup(self.pair_weight.iter().copied())
} }
} }
fn dedup(variants: impl IntoIterator<Item = bool>) -> Vec<bool> { fn dedup(variants: impl IntoIterator<Item = AmbiguityPolicy>) -> Vec<AmbiguityPolicy> {
let mut v: Vec<bool> = variants.into_iter().collect(); let mut v: Vec<AmbiguityPolicy> = variants.into_iter().collect();
v.sort_unstable(); v.sort_unstable();
v.dedup(); v.dedup();
v v
@@ -82,19 +126,21 @@ pub(crate) fn ensure_layer_family_annexes(
n_genomes: usize, n_genomes: usize,
fast_mode: bool, fast_mode: bool,
request: &AnnexRequest, request: &AnnexRequest,
overlap_accs: &mut [(bool, FamilyOverlapAccumulator)], overlap_accs: &mut [(AmbiguityPolicy, FamilyOverlapAccumulator)],
) -> OKIResult<()> { ) -> OKIResult<()> {
let layer = cache let layer = cache
.get_layer(partition, layer_idx) .get_layer(partition, layer_idx)
.expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)"); .expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)");
let missing = |name: &dyn Fn(bool) -> &'static str, variants: &[bool]| -> Vec<(bool, PathBuf)> { let missing = |name: &dyn Fn(AmbiguityPolicy) -> &'static str,
variants: &[AmbiguityPolicy]|
-> Vec<(AmbiguityPolicy, PathBuf)> {
dedup(variants.iter().copied()) dedup(variants.iter().copied())
.into_iter() .into_iter()
.map(|no_ambiguity| (no_ambiguity, layer.dir().join(name(no_ambiguity)))) .map(|policy| (policy, layer.dir().join(name(policy))))
.filter(|(_, path)| !path.exists()) .filter(|(_, path)| !path.exists())
.collect() .collect()
}; };
let missing_entropy = missing(&|nb| ScalarKind::Entropy.file_name(nb), &request.entropy); let missing_entropy = missing(&|p| ScalarKind::Entropy.file_name(p), &request.entropy);
let missing_quartet = missing(&quartet_annex_file_name, &request.quartet); let missing_quartet = missing(&quartet_annex_file_name, &request.quartet);
if missing_entropy.is_empty() && missing_quartet.is_empty() && overlap_accs.is_empty() { if missing_entropy.is_empty() && missing_quartet.is_empty() && overlap_accs.is_empty() {
return Ok(()); return Ok(());
@@ -104,16 +150,16 @@ pub(crate) fn ensure_layer_family_annexes(
let eligible: HashSet<usize> = reindex.keys().copied().collect(); let eligible: HashSet<usize> = reindex.keys().copied().collect();
let mut entropy_builders = missing_entropy let mut entropy_builders = missing_entropy
.iter() .iter()
.map(|(no_ambiguity, path)| { .map(|(policy, path)| {
FamilyScalarAnnexBuilder::new(reindex.len(), path, ScalarKind::Entropy, *no_ambiguity) FamilyScalarAnnexBuilder::new(reindex.len(), path, ScalarKind::Entropy, *policy)
.map(|b| (*no_ambiguity, b)) .map(|b| (*policy, b))
}) })
.collect::<io::Result<Vec<_>>>() .collect::<io::Result<Vec<_>>>()
.map_err(OKIError::Io)?; .map_err(OKIError::Io)?;
let mut quartet_builders = missing_quartet let mut quartet_builders = missing_quartet
.iter() .iter()
.map(|(no_ambiguity, path)| { .map(|(policy, path)| {
PersistentBitVecBuilder::new(reindex.len(), path).map(|b| (*no_ambiguity, b)) PersistentBitVecBuilder::new(reindex.len(), path).map(|b| (*policy, b))
}) })
.collect::<io::Result<Vec<_>>>() .collect::<io::Result<Vec<_>>>()
.map_err(OKIError::Io)?; .map_err(OKIError::Io)?;
@@ -125,19 +171,19 @@ pub(crate) fn ensure_layer_family_annexes(
n_genomes, n_genomes,
fast_mode, fast_mode,
&Selection::Some(&eligible), &Selection::Some(&eligible),
|family_idx, mask, genome_mask| { |family_idx, _kmer, mask, genome_mask| {
debug_assert!( debug_assert!(
mask.family_size() >= 2, mask.family_size() >= 2,
"Selection::Some(eligible) must only visit non-monomorphic minorants" "Selection::Some(eligible) must only visit non-monomorphic minorants"
); );
let compact = reindex[&family_idx]; let compact = reindex[&family_idx];
for (no_ambiguity, builder) in entropy_builders.iter_mut() { for (policy, builder) in entropy_builders.iter_mut() {
if let Some((h, _)) = family_entropy_under(genome_mask, *no_ambiguity) { if let Some((h, _)) = family_entropy_under(genome_mask, *policy) {
builder.set(compact, h as f32); builder.set(compact, h as f32);
} }
} }
for (no_ambiguity, builder) in quartet_builders.iter_mut() { for (policy, builder) in quartet_builders.iter_mut() {
builder.set(compact, is_quartet_informative(genome_mask, *no_ambiguity)); builder.set(compact, is_quartet_informative(genome_mask, *policy));
} }
for (_, acc) in overlap_accs.iter_mut() { for (_, acc) in overlap_accs.iter_mut() {
acc.add_family(genome_mask); acc.add_family(genome_mask);
@@ -155,7 +201,7 @@ pub(crate) fn ensure_layer_family_annexes(
} }
/// Builds the pair-weight annexes of one layer that don't exist yet, one per /// Builds the pair-weight annexes of one layer that don't exist yet, one per
/// `(no_ambiguity, inverse overlap matrix)` in `matrices` (see /// `(policy, inverse overlap matrix)` in `matrices` (see
/// [`super::pair_weight::inverse_overlap_matrix`]), in one scan. A no-op (no /// [`super::pair_weight::inverse_overlap_matrix`]), in one scan. A no-op (no
/// scan) when they all exist. Needs the family-overlap annex of each /// scan) when they all exist. Needs the family-overlap annex of each
/// convention to be complete: run after the sweep that builds it. /// convention to be complete: run after the sweep that builds it.
@@ -165,14 +211,14 @@ pub(crate) fn ensure_layer_pair_weight_annexes(
layer_idx: usize, layer_idx: usize,
n_genomes: usize, n_genomes: usize,
fast_mode: bool, fast_mode: bool,
matrices: &[(bool, Vec<f32>)], matrices: &[(AmbiguityPolicy, Vec<f32>)],
) -> OKIResult<()> { ) -> OKIResult<()> {
let layer = cache let layer = cache
.get_layer(partition, layer_idx) .get_layer(partition, layer_idx)
.expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)"); .expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)");
let pending: Vec<&(bool, Vec<f32>)> = matrices let pending: Vec<&(AmbiguityPolicy, Vec<f32>)> = matrices
.iter() .iter()
.filter(|(no_ambiguity, _)| !layer.dir().join(ScalarKind::PairWeight.file_name(*no_ambiguity)).exists()) .filter(|(policy, _)| !layer.dir().join(ScalarKind::PairWeight.file_name(*policy)).exists())
.collect(); .collect();
if pending.is_empty() { if pending.is_empty() {
return Ok(()); return Ok(());
@@ -182,10 +228,10 @@ pub(crate) fn ensure_layer_pair_weight_annexes(
let eligible: HashSet<usize> = reindex.keys().copied().collect(); let eligible: HashSet<usize> = reindex.keys().copied().collect();
let mut builders = pending let mut builders = pending
.iter() .iter()
.map(|(no_ambiguity, inverse)| { .map(|(policy, inverse)| {
let path = layer.dir().join(ScalarKind::PairWeight.file_name(*no_ambiguity)); let path = layer.dir().join(ScalarKind::PairWeight.file_name(*policy));
FamilyScalarAnnexBuilder::new(reindex.len(), &path, ScalarKind::PairWeight, *no_ambiguity) FamilyScalarAnnexBuilder::new(reindex.len(), &path, ScalarKind::PairWeight, *policy)
.map(|b| (*no_ambiguity, inverse, b)) .map(|b| (*policy, inverse, b))
}) })
.collect::<io::Result<Vec<_>>>() .collect::<io::Result<Vec<_>>>()
.map_err(OKIError::Io)?; .map_err(OKIError::Io)?;
@@ -197,10 +243,10 @@ pub(crate) fn ensure_layer_pair_weight_annexes(
n_genomes, n_genomes,
fast_mode, fast_mode,
&Selection::Some(&eligible), &Selection::Some(&eligible),
|family_idx, _mask, genome_mask| { |family_idx, _kmer, _mask, genome_mask| {
let compact = reindex[&family_idx]; let compact = reindex[&family_idx];
for (no_ambiguity, inverse, builder) in builders.iter_mut() { for (policy, inverse, builder) in builders.iter_mut() {
builder.set(compact, pair_balance_weight(genome_mask, *no_ambiguity, inverse)); builder.set(compact, pair_balance_weight(genome_mask, *policy, inverse));
} }
}, },
)?; )?;
@@ -54,7 +54,7 @@ use obipipeline::ThrottleGuard;
use crate::siblings::helpers::central_base; use crate::siblings::helpers::central_base;
use crate::siblings::iter::{SiblingEntry, SiblingLayerExt}; use crate::siblings::iter::{SiblingEntry, SiblingLayerExt};
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, SiblingAnnex}; use crate::siblings::{FamilyMask, SiblingAnnex};
/// Target memory budget for one batch's `genome_mask` allocation /// Target memory budget for one batch's `genome_mask` allocation
/// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding /// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding
@@ -125,6 +125,8 @@ struct SourceBatch {
struct GeneratedBatch { struct GeneratedBatch {
start_family_idx: usize, start_family_idx: usize,
masks: Vec<FamilyMask>, masks: Vec<FamilyMask>,
/// Each family's minorant k-mer, same order as `masks`.
kmers: Vec<CanonicalKmer>,
/// Flat `slots.len() * n_genomes` — `genome_mask[i * n_genomes + g]`. /// Flat `slots.len() * n_genomes` — `genome_mask[i * n_genomes + g]`.
genome_mask: Vec<u8>, genome_mask: Vec<u8>,
/// `outgoing[dest_partition]` = `(variant, family_idx_in_batch, base, /// `outgoing[dest_partition]` = `(variant, family_idx_in_batch, base,
@@ -146,8 +148,8 @@ enum FamData {
/// generation and resolution use different concurrency. `on_family` is /// generation and resolution use different concurrency. `on_family` is
/// called once per /// called once per
/// family, in iteration order, with that family's iteration-order index /// family, in iteration order, with that family's iteration-order index
/// (the same numbering [`Selection`] indices are drawn from), its own annex /// (the same numbering [`Selection`] indices are drawn from), its minorant
/// mask, and its per-genome base-presence (`genome_mask[g]`: bit `b` set /// k-mer, its own annex mask, and its per-genome base-presence (`genome_mask[g]`: bit `b` set
/// iff genome `g` carries the member whose own canonical central base is /// iff genome `g` carries the member whose own canonical central base is
/// `b`), backed by a scratch buffer reused across every call — callers that /// `b`), backed by a scratch buffer reused across every call — callers that
/// need to keep data past the call must copy it themselves. /// need to keep data past the call must copy it themselves.
@@ -164,7 +166,7 @@ pub(crate) fn scan_layer_families(
n_genomes: usize, n_genomes: usize,
fast_mode: bool, fast_mode: bool,
selection: &Selection, selection: &Selection,
mut on_family: impl FnMut(usize, FamilyMask, &[u8]), mut on_family: impl FnMut(usize, CanonicalKmer, FamilyMask, &[u8]),
) -> OKIResult<()> { ) -> OKIResult<()> {
let n_parts = cache.index().n_partitions(); let n_parts = cache.index().n_partitions();
let k = cache.index().kmer_size(); let k = cache.index().kmer_size();
@@ -180,7 +182,7 @@ pub(crate) fn scan_layer_families(
})?; })?;
let n_cols = mat.n_cols().min(n_genomes); let n_cols = mat.n_cols().min(n_genomes);
let layer_dir: &Path = mat.dir(); let layer_dir: &Path = mat.dir();
let annex = Arc::new(SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?); let annex = Arc::new(SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?);
let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k }); let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k });
@@ -225,6 +227,7 @@ pub(crate) fn scan_layer_families(
let ctx = &worker_ctx; let ctx = &worker_ctx;
let n = batch.entries.len(); let n = batch.entries.len();
let mut masks = Vec::with_capacity(n); let mut masks = Vec::with_capacity(n);
let mut kmers = Vec::with_capacity(n);
let mut bases = Vec::with_capacity(n); let mut bases = Vec::with_capacity(n);
let mut genome_mask = vec![0u8; n * ctx.n_genomes]; let mut genome_mask = vec![0u8; n * ctx.n_genomes];
let mut outgoing: Vec<Vec<(CanonicalKmer, usize, u8, u8)>> = let mut outgoing: Vec<Vec<(CanonicalKmer, usize, u8, u8)>> =
@@ -248,6 +251,7 @@ pub(crate) fn scan_layer_families(
for (i, entry) in batch.entries.iter().enumerate() { for (i, entry) in batch.entries.iter().enumerate() {
let (kmer, mask) = (entry.kmer, entry.mask); let (kmer, mask) = (entry.kmer, entry.mask);
masks.push(mask); masks.push(mask);
kmers.push(kmer);
let base = central_base(kmer, ctx.k); let base = central_base(kmer, ctx.k);
bases.push(base); bases.push(base);
let family_idx = batch.start_family_idx + i; let family_idx = batch.start_family_idx + i;
@@ -297,6 +301,7 @@ pub(crate) fn scan_layer_families(
GeneratedBatch { GeneratedBatch {
start_family_idx: batch.start_family_idx, start_family_idx: batch.start_family_idx,
masks, masks,
kmers,
genome_mask, genome_mask,
outgoing, outgoing,
_permit: batch._permit, _permit: batch._permit,
@@ -344,7 +349,7 @@ pub(crate) fn scan_layer_families(
for (g, dst) in scratch.iter_mut().enumerate() { for (g, dst) in scratch.iter_mut().enumerate() {
*dst = genome_mask[i * n_genomes + g].load(Ordering::Relaxed); *dst = genome_mask[i * n_genomes + g].load(Ordering::Relaxed);
} }
on_family(family_idx, batch.masks[i], &scratch); on_family(family_idx, batch.kmers[i], batch.masks[i], &scratch);
} }
next_expected += n; next_expected += n;
} }
@@ -17,20 +17,22 @@
//! waste sample quota on sites that die anyway once the excluded genome's //! waste sample quota on sites that die anyway once the excluded genome's
//! data is dropped. //! data is dropped.
use crate::siblings::AmbiguityPolicy;
use crate::siblings::helpers::is_carrier; use crate::siblings::helpers::is_carrier;
/// This genome's masked character state, or `None` for "?" (missing). `m` /// This genome's masked character state, or `None` for "?" (missing). `m`
/// is one `genome_mask` entry (bit `b` set iff this genome carries the /// is one `genome_mask` entry (bit `b` set iff this genome carries the
/// family member whose own canonical central base is `b`; `0` = no member /// family member whose own canonical central base is `b`; `0` = no member
/// present in this genome, `∅` — a real state on its own unless /// present in this genome, `∅` — a real state on its own unless
/// `free_loss` says otherwise). /// `free_loss` says otherwise). `policy` makes missing the genomes carrying
/// too many members ([`AmbiguityPolicy::drops`]).
#[inline] #[inline]
pub(crate) fn masked_state(m: u8, free_loss: bool, no_ambiguity: bool) -> Option<u8> { pub(crate) fn masked_state(m: u8, free_loss: bool, policy: AmbiguityPolicy) -> Option<u8> {
if free_loss && m == 0 { if free_loss && m == 0 {
return None; // absence -> missing, not a real state return None; // absence -> missing, not a real state
} }
if no_ambiguity && m.count_ones() > 1 { if policy.drops(m) {
return None; // ambiguous (>1 member in this genome) -> missing return None; // ambiguous beyond what the policy keeps -> missing
} }
Some(m) Some(m)
} }
@@ -85,10 +87,10 @@ pub(crate) fn has_bipartition(counts: &[u32; 16]) -> bool {
/// bipartition, never create one — so `false` is final, and `true` is only /// bipartition, never create one — so `false` is final, and `true` is only
/// a necessary condition, re-checked exactly by [`survives_quartet`] once /// a necessary condition, re-checked exactly by [`survives_quartet`] once
/// the family is resolved. This is what the persisted quartet annex stores. /// the family is resolved. This is what the persisted quartet annex stores.
pub(crate) fn is_quartet_informative(genome_mask: &[u8], no_ambiguity: bool) -> bool { pub(crate) fn is_quartet_informative(genome_mask: &[u8], policy: AmbiguityPolicy) -> bool {
let mut counts = [0u32; 16]; let mut counts = [0u32; 16];
for &m in genome_mask { for &m in genome_mask {
if !is_carrier(m, no_ambiguity) { if !is_carrier(m, policy) {
continue; continue;
} }
counts[(m & 0b1111) as usize] += 1; counts[(m & 0b1111) as usize] += 1;
@@ -119,31 +121,31 @@ mod quartet_tests {
#[test] #[test]
fn two_pairs_of_distinct_states_is_informative() { fn two_pairs_of_distinct_states_is_informative() {
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], false)); assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], AmbiguityPolicy::Keep));
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], true)); assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], AmbiguityPolicy::Forbid));
} }
#[test] #[test]
fn singleton_variant_is_not_informative() { fn singleton_variant_is_not_informative() {
// three genomes in one state plus one lone variant: fits any topology // three genomes in one state plus one lone variant: fits any topology
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0001, 0b0100], false)); assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0001, 0b0100], AmbiguityPolicy::Keep));
// fewer than four carriers can never be informative // fewer than four carriers can never be informative
assert!(!is_quartet_informative(&[0b0001, 0b0100, 0, 0], false)); assert!(!is_quartet_informative(&[0b0001, 0b0100, 0, 0], AmbiguityPolicy::Keep));
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0100], false)); assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0100], AmbiguityPolicy::Keep));
} }
#[test] #[test]
fn absent_genomes_are_not_a_state() { fn absent_genomes_are_not_a_state() {
// two carriers of one base, two absences: absence must not form the 2nd pair // two carriers of one base, two absences: absence must not form the 2nd pair
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0, 0], false)); assert!(!is_quartet_informative(&[0b0001, 0b0001, 0, 0], AmbiguityPolicy::Keep));
} }
#[test] #[test]
fn no_ambiguity_drops_multi_member_genomes_from_the_count() { fn no_ambiguity_drops_multi_member_genomes_from_the_count() {
// the two-member genomes form a pair of their own state without the flag // the two-member genomes form a pair of their own state without the flag
let masks = [0b0001u8, 0b0001, 0b0011, 0b0011]; let masks = [0b0001u8, 0b0001, 0b0011, 0b0011];
assert!(is_quartet_informative(&masks, false)); assert!(is_quartet_informative(&masks, AmbiguityPolicy::Keep));
assert!(!is_quartet_informative(&masks, true)); assert!(!is_quartet_informative(&masks, AmbiguityPolicy::Forbid));
} }
#[test] #[test]
@@ -188,6 +190,32 @@ pub(crate) fn iupac_code(state: u8) -> u8 {
} }
} }
/// Inverse of [`iupac_code`]: the `0..=15` state a cell character encodes
/// (`-` -> 0, a base -> its single bit, an ambiguity code -> its bits), or
/// `None` for anything else — notably `?`, a missing cell.
#[inline]
pub(crate) fn iupac_state(cell: u8) -> Option<u8> {
Some(match cell {
b'-' => 0b0000,
b'A' => 0b0001,
b'C' => 0b0010,
b'G' => 0b0100,
b'T' => 0b1000,
b'R' => 0b0101,
b'Y' => 0b1010,
b'S' => 0b0110,
b'W' => 0b1001,
b'K' => 0b1100,
b'M' => 0b0011,
b'B' => 0b1110,
b'D' => 0b1101,
b'H' => 0b1011,
b'V' => 0b0111,
b'N' => 0b1111,
_ => return None,
})
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -196,7 +224,20 @@ mod tests {
/// `subsample::sample_layer` does once, up front, before calling /// `subsample::sample_layer` does once, up front, before calling
/// [`survives_masking`]. /// [`survives_masking`].
fn masked(raw: &[u8], free_loss: bool, no_ambiguity: bool) -> Vec<Option<u8>> { fn masked(raw: &[u8], free_loss: bool, no_ambiguity: bool) -> Vec<Option<u8>> {
raw.iter().map(|&m| masked_state(m, free_loss, no_ambiguity)).collect() let policy = AmbiguityPolicy::from_flags(no_ambiguity, false);
raw.iter().map(|&m| masked_state(m, free_loss, policy)).collect()
}
#[test]
fn only_diploid_keeps_up_to_two_members() {
let m = |raw: u8| masked_state(raw, false, AmbiguityPolicy::OnlyDiploid);
assert_eq!(m(0b0000), Some(0)); // absence stays a state (no --free-loss)
assert_eq!(m(0b0100), Some(0b0100));
assert_eq!(m(0b0101), Some(0b0101)); // R
assert_eq!(m(0b0111), None); // V
assert_eq!(m(0b1111), None); // N
// `Forbid` is stricter: a heterozygote is missing there
assert_eq!(masked_state(0b0101, false, AmbiguityPolicy::Forbid), None);
} }
#[test] #[test]
@@ -11,7 +11,7 @@ use std::path::Path;
use obicompactvec::TempBitVecBuilder; use obicompactvec::TempBitVecBuilder;
use obikindex::{OKIError, OKIResult}; use obikindex::{OKIError, OKIResult};
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex}; use crate::siblings::{AmbiguityPolicy, FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool { pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
mask.is_minorant() && mask.siblings() >= 1 mask.is_minorant() && mask.siblings() >= 1
@@ -31,7 +31,7 @@ pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
/// the `Selection` a caller needs to bound `scan_layer_families` to just /// the `Selection` a caller needs to bound `scan_layer_families` to just
/// these families. /// these families.
pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> { pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let mut reindex = HashMap::new(); let mut reindex = HashMap::new();
let mut family_idx = 0usize; let mut family_idx = 0usize;
for slot in 0..annex.len() { for slot in 0..annex.len() {
@@ -62,7 +62,7 @@ pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashM
/// first counts this layer's total minorants (needed up front to size the /// first counts this layer's total minorants (needed up front to size the
/// bitset), the second sets the eligible bits. /// bitset), the second sets the eligible bits.
pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> { pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let n_minorants = (0..annex.len()) let n_minorants = (0..annex.len())
.filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant())) .filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant()))
@@ -92,7 +92,7 @@ pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBui
/// at each site. /// at each site.
/// ///
/// Reads the `kind` annex (entropy, pair weight) built under the /// Reads the `kind` annex (entropy, pair weight) built under the
/// `no_ambiguity` counting convention; it must already exist /// `policy` counting convention; it must already exist
/// (`ensure_layer_family_annexes` / `ensure_layer_pair_weight_annexes` /// (`ensure_layer_family_annexes` / `ensure_layer_pair_weight_annexes`
/// first) — read positionally (`compact` advances only on non-monomorphic /// first) — read positionally (`compact` advances only on non-monomorphic
/// minorants, mirroring exactly how the annex was written), never /// minorants, mirroring exactly how the annex was written), never
@@ -101,15 +101,15 @@ pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBui
pub(crate) fn iter_full_scalar( pub(crate) fn iter_full_scalar(
layer_dir: &Path, layer_dir: &Path,
kind: ScalarKind, kind: ScalarKind,
no_ambiguity: bool, policy: AmbiguityPolicy,
) -> OKIResult<impl Iterator<Item = f32> + use<>> { ) -> OKIResult<impl Iterator<Item = f32> + use<>> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let scalars = let scalars =
FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?; FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(policy)), kind).map_err(OKIError::Io)?;
if scalars.no_ambiguity() != no_ambiguity { if scalars.policy() != policy {
return Err(OKIError::Io(std::io::Error::new( return Err(OKIError::Io(std::io::Error::new(
std::io::ErrorKind::InvalidData, std::io::ErrorKind::InvalidData,
format!("{kind:?} annex was built under the other --no-ambiguity convention"), format!("{kind:?} annex was built under {:?}, not {policy:?}", scalars.policy()),
))); )));
} }
+3 -1
View File
@@ -14,6 +14,7 @@
//! encoding. //! encoding.
mod alignment; mod alignment;
mod alignment_vcf;
mod annex; mod annex;
mod cardcomp; mod cardcomp;
mod entropy; mod entropy;
@@ -34,7 +35,7 @@ mod subsample;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
pub use alignment::{AlignmentReducer, SnpAlignment}; pub use alignment::{AlignmentReducer, SiteLocus, SnpAlignment};
pub use cardcomp::{ pub use cardcomp::{
cardinality_transition_probs, composition_transition_probs, pairwise_cost_matrix, cardinality_transition_probs, composition_transition_probs, pairwise_cost_matrix,
}; };
@@ -48,6 +49,7 @@ pub use subsample::{EntropyBias, LayerReport, SamplePlan, SamplingSpec, Survivin
pub(crate) use annex::build_layer_sibling_annex; pub(crate) use annex::build_layer_sibling_annex;
pub(crate) use entropy_scan::{EntropySummary, scan_family_entropies}; pub(crate) use entropy_scan::{EntropySummary, scan_family_entropies};
pub use family_annexes::AnnexSelection;
pub(crate) use family_annexes::{ pub(crate) use family_annexes::{
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes, AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes,
}; };
@@ -13,7 +13,7 @@
//! Values are only meaningful relative to each other; the sampler rescales //! Values are only meaningful relative to each other; the sampler rescales
//! them per layer (`SamplingSpec::pair_balance`). //! them per layer (`SamplingSpec::pair_balance`).
use crate::siblings::FamilyOverlap; use crate::siblings::{AmbiguityPolicy, FamilyOverlap};
use crate::siblings::helpers::is_carrier; use crate::siblings::helpers::is_carrier;
/// `1 / O` for every pair of genomes, as a dense symmetric `n × n` matrix /// `1 / O` for every pair of genomes, as a dense symmetric `n × n` matrix
@@ -38,16 +38,16 @@ pub(crate) fn inverse_overlap_matrix(overlap: &FamilyOverlap) -> Vec<f32> {
/// The pair-balance weight of one family from its raw per-genome masks: /// The pair-balance weight of one family from its raw per-genome masks:
/// mean of `inv_overlap` over the pairs of carriers (carriers per /// mean of `inv_overlap` over the pairs of carriers (carriers per
/// [`is_carrier`], under `no_ambiguity`). `0.0` for a family with fewer than /// [`is_carrier`], under `policy`). `0.0` for a family with fewer than
/// two carriers — it covers no pair, so it can balance none. /// two carriers — it covers no pair, so it can balance none.
pub(crate) fn pair_balance_weight( pub(crate) fn pair_balance_weight(
genome_mask: &[u8], genome_mask: &[u8],
no_ambiguity: bool, policy: AmbiguityPolicy,
inv_overlap: &[f32], inv_overlap: &[f32],
) -> f32 { ) -> f32 {
let n = genome_mask.len(); let n = genome_mask.len();
let carriers: Vec<usize> = let carriers: Vec<usize> =
(0..n).filter(|&g| is_carrier(genome_mask[g], no_ambiguity)).collect(); (0..n).filter(|&g| is_carrier(genome_mask[g], policy)).collect();
let k = carriers.len(); let k = carriers.len();
if k < 2 { if k < 2 {
return 0.0; return 0.0;
@@ -83,30 +83,30 @@ mod tests {
#[test] #[test]
fn single_pair_weight_is_the_inverse_overlap() { fn single_pair_weight_is_the_inverse_overlap() {
let w = pair_balance_weight(&[1, 1, 0, 0], false, &matrix()); let w = pair_balance_weight(&[1, 1, 0, 0], AmbiguityPolicy::Keep, &matrix());
assert!((w - 0.5).abs() < 1e-6); assert!((w - 0.5).abs() < 1e-6);
} }
#[test] #[test]
fn weight_is_the_mean_not_the_sum_over_pairs() { fn weight_is_the_mean_not_the_sum_over_pairs() {
// carriers 0, 1, 2: pairs (0,1)=1/2, (0,2)=1/4, (1,2)=1 -> mean 7/12 // carriers 0, 1, 2: pairs (0,1)=1/2, (0,2)=1/4, (1,2)=1 -> mean 7/12
let w = pair_balance_weight(&[1, 1, 1, 0], false, &matrix()); let w = pair_balance_weight(&[1, 1, 1, 0], AmbiguityPolicy::Keep, &matrix());
assert!((w - 7.0 / 12.0).abs() < 1e-6); assert!((w - 7.0 / 12.0).abs() < 1e-6);
} }
#[test] #[test]
fn fewer_than_two_carriers_weigh_nothing() { fn fewer_than_two_carriers_weigh_nothing() {
assert_eq!(pair_balance_weight(&[1, 0, 0, 0], false, &matrix()), 0.0); assert_eq!(pair_balance_weight(&[1, 0, 0, 0], AmbiguityPolicy::Keep, &matrix()), 0.0);
assert_eq!(pair_balance_weight(&[0, 0, 0, 0], false, &matrix()), 0.0); assert_eq!(pair_balance_weight(&[0, 0, 0, 0], AmbiguityPolicy::Keep, &matrix()), 0.0);
} }
#[test] #[test]
fn no_ambiguity_leaves_multi_member_genomes_out_of_the_pairs() { fn no_ambiguity_leaves_multi_member_genomes_out_of_the_pairs() {
let masks = [1u8, 1, 0b0011, 0]; let masks = [1u8, 1, 0b0011, 0];
// without the flag genome 2 carries the family: pairs (0,1), (0,2), (1,2) // without the flag genome 2 carries the family: pairs (0,1), (0,2), (1,2)
let with = pair_balance_weight(&masks, false, &matrix()); let with = pair_balance_weight(&masks, AmbiguityPolicy::Keep, &matrix());
// with it, only the pair (0,1) remains // with it, only the pair (0,1) remains
let without = pair_balance_weight(&masks, true, &matrix()); let without = pair_balance_weight(&masks, AmbiguityPolicy::Forbid, &matrix());
assert!((with - 7.0 / 12.0).abs() < 1e-6); assert!((with - 7.0 / 12.0).abs() < 1e-6);
assert!((without - 0.5).abs() < 1e-6); assert!((without - 0.5).abs() < 1e-6);
} }
@@ -114,8 +114,8 @@ mod tests {
#[test] #[test]
fn a_rare_pair_outweighs_a_common_one() { fn a_rare_pair_outweighs_a_common_one() {
// genomes (0,2) share many families, (1,3) few // genomes (0,2) share many families, (1,3) few
let common = pair_balance_weight(&[1, 0, 1, 0], false, &matrix()); let common = pair_balance_weight(&[1, 0, 1, 0], AmbiguityPolicy::Keep, &matrix());
let rare = pair_balance_weight(&[0, 1, 0, 1], false, &matrix()); let rare = pair_balance_weight(&[0, 1, 0, 1], AmbiguityPolicy::Keep, &matrix());
assert!(rare > common); assert!(rare > common);
} }
} }
@@ -510,7 +510,7 @@ mod cardinality_masking_tests {
// `SurvivingFamily::masked`'s own docs) — the masking decision is baked // `SurvivingFamily::masked`'s own docs) — the masking decision is baked
// in here, not passed to `reduce_pairwise` itself anymore. // in here, not passed to `reduce_pairwise` itself anymore.
fn ambiguous_vs_single(masked: Vec<Option<u8>>) -> Vec<SurvivingFamily> { fn ambiguous_vs_single(masked: Vec<Option<u8>>) -> Vec<SurvivingFamily> {
vec![SurvivingFamily { family_idx: 0, mask: FamilyMask::EMPTY.with(0).with(1), masked }] vec![SurvivingFamily { family_idx: 0, kmer: obikseq::CanonicalKmer::from_raw_unchecked(0), mask: FamilyMask::EMPTY.with(0).with(1), masked }]
} }
#[test] #[test]
@@ -8,11 +8,11 @@ use obikindex::{OKIError, OKIResult};
use super::minorant_selection::is_non_monomorphic_minorant; use super::minorant_selection::is_non_monomorphic_minorant;
use crate::siblings::quartet_annex::quartet_annex_file_name; use crate::siblings::quartet_annex::quartet_annex_file_name;
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex}; use crate::siblings::{AmbiguityPolicy, SiblingAnnex};
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants /// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
/// included), every family the quartet annex flags as not quartet-informative /// included), every family the quartet annex flags as not quartet-informative
/// under the `no_ambiguity` counting convention. The annex is indexed by the /// under the `policy` counting convention. The annex is indexed by the
/// compact non-monomorphic ordinal, so the sibling annex is walked in the same /// compact non-monomorphic ordinal, so the sibling annex is walked in the same
/// order to translate one numbering into the other — annex-only, no /// order to translate one numbering into the other — annex-only, no
/// cross-partition resolution. The annex must already exist /// cross-partition resolution. The annex must already exist
@@ -20,10 +20,10 @@ use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
pub(crate) fn restrict_to_quartet( pub(crate) fn restrict_to_quartet(
eligible: &mut TempBitVecBuilder, eligible: &mut TempBitVecBuilder,
layer_dir: &Path, layer_dir: &Path,
no_ambiguity: bool, policy: AmbiguityPolicy,
) -> OKIResult<()> { ) -> OKIResult<()> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity))) let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(policy)))
.map_err(OKIError::Io)?; .map_err(OKIError::Io)?;
let mut family_idx = 0usize; let mut family_idx = 0usize;
@@ -279,13 +279,14 @@ mod tests {
/// something these tests need to exercise). /// something these tests need to exercise).
fn tally_2genomes(pairs: &[(u8, u8)], n_shared: usize) -> PairwiseTally { fn tally_2genomes(pairs: &[(u8, u8)], n_shared: usize) -> PairwiseTally {
// No masking requested (`free_loss = no_ambiguity = false`) — // No masking requested (`free_loss = no_ambiguity = false`) —
// `masked_state(m, false, false)` is always `Some(m)`, so the raw // `masked_state(m, false, AmbiguityPolicy::Keep)` is always `Some(m)`, so the raw
// mask is exactly what `subsample::sample_layer` would have // mask is exactly what `subsample::sample_layer` would have
// computed here. // computed here.
let mut families = Vec::new(); let mut families = Vec::new();
for &(a, b) in pairs { for &(a, b) in pairs {
families.push(SurvivingFamily { families.push(SurvivingFamily {
family_idx: families.len(), family_idx: families.len(),
kmer: obikseq::CanonicalKmer::from_raw_unchecked(0),
mask: FamilyMask::EMPTY.with(a.trailing_zeros() as u8).with(b.trailing_zeros() as u8), mask: FamilyMask::EMPTY.with(a.trailing_zeros() as u8).with(b.trailing_zeros() as u8),
masked: vec![Some(a), Some(b)], masked: vec![Some(a), Some(b)],
}); });
@@ -296,6 +297,7 @@ mod tests {
let other = BASES[(i + 1) % 4]; let other = BASES[(i + 1) % 4];
families.push(SurvivingFamily { families.push(SurvivingFamily {
family_idx: families.len(), family_idx: families.len(),
kmer: obikseq::CanonicalKmer::from_raw_unchecked(0),
mask: FamilyMask::EMPTY.with(b.trailing_zeros() as u8).with(other.trailing_zeros() as u8), mask: FamilyMask::EMPTY.with(b.trailing_zeros() as u8).with(other.trailing_zeros() as u8),
masked: vec![Some(b), Some(b)], masked: vec![Some(b), Some(b)],
}); });
+10 -17
View File
@@ -9,11 +9,12 @@
//! already stored — no re-derivation, no lookup). //! already stored — no re-derivation, no lookup).
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikindex::{OKIError, OKIResult}; use obikindex::OKIResult;
use super::family_scan::{Selection, scan_layer_families}; use super::family_scan::{Selection, scan_layer_families};
use super::is_fast_mode; use super::is_fast_mode;
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex}; use crate::siblings::SiblingAnnex;
use crate::siblings::extensions::SiblingBuilder;
/// Layers scanned at once per [`sibling_family_size_histogram`]'s /// Layers scanned at once per [`sibling_family_size_histogram`]'s
/// `par_map_reduce` call — same bounding reasoning as /// `par_map_reduce` call — same bounding reasoning as
@@ -45,26 +46,17 @@ pub struct SiblingAnnexStats {
/// on top of that would fight it for cache/mmap locality) — this function /// on top of that would fight it for cache/mmap locality) — this function
/// has no such inner parallelism to protect. /// has no such inner parallelism to protect.
/// ///
/// Existence of every cached layer's annex file is checked up front, before /// A missing annex is built up front (`ensure_sibling_annex`), before the
/// the parallel map runs, so a missing one is reported as an error rather /// parallel map runs, so none is silently read back as zero.
/// than silently read back as zero.
pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> { pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> {
for (part, l, layer) in cache.iter_indexed() { cache.ensure_sibling_annex()?;
let path = layer.dir().join(ANNEX_FILE_NAME);
if !path.exists() {
return Err(OKIError::InvalidInput(format!(
"no sibling annex at {} (partition {part}, layer {l}) — run --sibling-annex first",
path.display()
)));
}
}
Ok(cache.par_map_reduce( Ok(cache.par_map_reduce(
MAX_CONCURRENT_LAYERS, MAX_CONCURRENT_LAYERS,
|| [0u64; 4], || [0u64; 4],
|layer| { |layer| {
let annex = SiblingAnnex::open(&layer.dir().join(ANNEX_FILE_NAME)) let annex = SiblingAnnex::open_in(layer.dir())
.expect("existence already checked before this parallel map ran"); .expect("ensure_sibling_annex already made every layer's annex exist");
let mut counts = [0u64; 4]; let mut counts = [0u64; 4];
for slot in 0..annex.len() { for slot in 0..annex.len() {
let Some(mask) = annex.get(slot) else { continue }; let Some(mask) = annex.get(slot) else { continue };
@@ -95,6 +87,7 @@ pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u6
/// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/ /// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/
/// `algorithms::scan::sampled_scan` already follow). /// `algorithms::scan::sampled_scan` already follow).
pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> { pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> {
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len(); let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache); let fast_mode = is_fast_mode(cache);
@@ -113,7 +106,7 @@ pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexS
n_genomes, n_genomes,
fast_mode, fast_mode,
&Selection::All, &Selection::All,
|_family_idx, mask, genome_mask| { |_family_idx, _kmer, mask, genome_mask| {
// "Genome g represents this family" means g carries // "Genome g represents this family" means g carries
// *any* of its members, not just the minorant's own — // *any* of its members, not just the minorant's own —
// `genome_mask[g] != 0` is exactly that. // `genome_mask[g] != 0` is exactly that.
@@ -49,6 +49,7 @@ use std::sync::Arc;
use obicompactvec::TempBitVecBuilder; use obicompactvec::TempBitVecBuilder;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikindex::OKIResult; use obikindex::OKIResult;
use obikseq::CanonicalKmer;
use obisys::progress_bar; use obisys::progress_bar;
use rand::RngExt; use rand::RngExt;
@@ -56,7 +57,7 @@ use super::family_scan::{Selection, scan_layer_families};
use super::masking::{masked_state, survives_masking, survives_quartet}; use super::masking::{masked_state, survives_masking, survives_quartet};
use super::minorant_selection::{eligible_bitset_layer, iter_full_scalar}; use super::minorant_selection::{eligible_bitset_layer, iter_full_scalar};
use super::{AnnexRequest, is_fast_mode, restrict_to_quartet}; use super::{AnnexRequest, is_fast_mode, restrict_to_quartet};
use crate::siblings::{FamilyMask, ScalarKind}; use crate::siblings::{AmbiguityPolicy, FamilyMask, ScalarKind};
use crate::siblings::extensions::SiblingBuilder; use crate::siblings::extensions::SiblingBuilder;
/// Resolution rounds a single layer's sampling may take before giving up on /// Resolution rounds a single layer's sampling may take before giving up on
@@ -82,6 +83,8 @@ const MAX_TOPUP_ROUNDS: usize = 5;
/// batch, well after that callback returns. /// batch, well after that callback returns.
pub struct SurvivingFamily { pub struct SurvivingFamily {
pub family_idx: usize, pub family_idx: usize,
/// The family's minorant k-mer (feeds `CanonicalKmer::family_id`).
pub kmer: CanonicalKmer,
pub mask: FamilyMask, pub mask: FamilyMask,
pub masked: Vec<Option<u8>>, pub masked: Vec<Option<u8>>,
} }
@@ -106,7 +109,10 @@ pub struct EntropyBias {
#[derive(Clone, Copy)] #[derive(Clone, Copy)]
pub struct SamplingSpec { pub struct SamplingSpec {
pub free_loss: bool, pub free_loss: bool,
pub no_ambiguity: bool, /// How genomes carrying several members of a family are treated
/// (`--no-ambiguity` / `--only-diploid`): by the post-hoc masking, and as
/// the counting convention of the persisted annexes the draw reads.
pub ambiguity: AmbiguityPolicy,
pub quartet: bool, pub quartet: bool,
/// `--pair-balance`: weight each family by the mean, over the pairs of /// `--pair-balance`: weight each family by the mean, over the pairs of
/// genomes carrying it, of 1 / (number of families the pair shares), so /// genomes carrying it, of 1 / (number of families the pair shares), so
@@ -156,17 +162,17 @@ impl SamplePlan {
/// [`sample_layer`]'s own circular index sequence walks, so the streams can be /// [`sample_layer`]'s own circular index sequence walks, so the streams can be
/// `.zip()`-ed directly. Two more streamed passes over the layer's (small) /// `.zip()`-ed directly. Two more streamed passes over the layer's (small)
/// annex files, not a materialised `Vec`/`HashMap` — `iter_full_scalar` /// annex files, not a materialised `Vec`/`HashMap` — `iter_full_scalar`
/// itself never allocates more than its own iterator state. `no_ambiguity` /// itself never allocates more than its own iterator state. `policy`
/// selects the annex built under the same counting convention as the masking /// selects the annex built under the same counting convention as the masking
/// the survivors go through afterwards. /// the survivors go through afterwards.
fn circular_scalar_values( fn circular_scalar_values(
layer_dir: &Path, layer_dir: &Path,
kind: ScalarKind, kind: ScalarKind,
start: usize, start: usize,
no_ambiguity: bool, policy: AmbiguityPolicy,
) -> OKIResult<impl Iterator<Item = f32> + use<>> { ) -> OKIResult<impl Iterator<Item = f32> + use<>> {
let head = iter_full_scalar(layer_dir, kind, no_ambiguity)?.skip(start); let head = iter_full_scalar(layer_dir, kind, policy)?.skip(start);
let tail = iter_full_scalar(layer_dir, kind, no_ambiguity)?.take(start); let tail = iter_full_scalar(layer_dir, kind, policy)?.take(start);
Ok(head.chain(tail)) Ok(head.chain(tail))
} }
@@ -192,7 +198,7 @@ struct LayerPlan {
/// per layer, calling `on_layer(partition, layer, survivors)` once per /// per layer, calling `on_layer(partition, layer, survivors)` once per
/// cached layer that contributed anything, with every family from that /// cached layer that contributed anything, with every family from that
/// layer that was both drawn *and* still carries real polymorphism after /// layer that was both drawn *and* still carries real polymorphism after
/// `free_loss`/`no_ambiguity`/`excluded` masking /// `free_loss`/`ambiguity`/`excluded` masking
/// (`algorithms::masking::survives_masking`) — i.e. every /// (`algorithms::masking::survives_masking`) — i.e. every
/// [`SurvivingFamily`] this produces is already a usable alignment site, no /// [`SurvivingFamily`] this produces is already a usable alignment site, no
/// second resolution pass needed by the caller. `excluded[g]` (see /// second resolution pass needed by the caller. `excluded[g]` (see
@@ -241,6 +247,9 @@ pub(crate) fn sample_index(
skip_layers: usize, skip_layers: usize,
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>), mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
) -> OKIResult<SamplePlan> { ) -> OKIResult<SamplePlan> {
// Every selection below reads the sibling annex (`eligible_bitset_layer`),
// whichever derived annexes the options ask for.
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len(); let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache); let fast_mode = is_fast_mode(cache);
@@ -250,10 +259,10 @@ pub(crate) fn sample_index(
// layer, below, each under this run's `--no-ambiguity` counting // layer, below, each under this run's `--no-ambiguity` counting
// convention) — build any missing one now, once, rather than mid-walk. // convention) — build any missing one now, once, rather than mid-walk.
let request = AnnexRequest { let request = AnnexRequest {
entropy: spec.entropy_bias.map(|_| vec![spec.no_ambiguity]).unwrap_or_default(), entropy: spec.entropy_bias.map(|_| vec![spec.ambiguity]).unwrap_or_default(),
quartet: if spec.quartet { vec![spec.no_ambiguity] } else { Vec::new() }, quartet: if spec.quartet { vec![spec.ambiguity] } else { Vec::new() },
overlap: Vec::new(), overlap: Vec::new(),
pair_weight: if spec.pair_balance { vec![spec.no_ambiguity] } else { Vec::new() }, pair_weight: if spec.pair_balance { vec![spec.ambiguity] } else { Vec::new() },
}; };
if !request.entropy.is_empty() || !request.quartet.is_empty() || !request.pair_weight.is_empty() { if !request.entropy.is_empty() || !request.quartet.is_empty() || !request.pair_weight.is_empty() {
cache.ensure_family_annexes(&request)?; cache.ensure_family_annexes(&request)?;
@@ -273,12 +282,12 @@ pub(crate) fn sample_index(
.expect("cache-consistent: n_layer(partition) already bounds layer_idx"); .expect("cache-consistent: n_layer(partition) already bounds layer_idx");
let mut eligible = eligible_bitset_layer(layer.dir())?; let mut eligible = eligible_bitset_layer(layer.dir())?;
if spec.quartet { if spec.quartet {
restrict_to_quartet(&mut eligible, layer.dir(), spec.no_ambiguity)?; restrict_to_quartet(&mut eligible, layer.dir(), spec.ambiguity)?;
} }
let n_minorants = eligible.view().len(); let n_minorants = eligible.view().len();
let eligible_count = eligible.view().count_ones(); let eligible_count = eligible.view().count_ones();
let mass = if spec.pair_balance { let mass = if spec.pair_balance {
iter_full_scalar(layer.dir(), ScalarKind::PairWeight, spec.no_ambiguity)? iter_full_scalar(layer.dir(), ScalarKind::PairWeight, spec.ambiguity)?
.enumerate() .enumerate()
.filter(|&(family_idx, _)| eligible.view().get(family_idx)) .filter(|&(family_idx, _)| eligible.view().get(family_idx))
.map(|(_, weight)| f64::from(weight)) .map(|(_, weight)| f64::from(weight))
@@ -423,7 +432,7 @@ fn sample_layer(
&layer_dir, &layer_dir,
ScalarKind::Entropy, ScalarKind::Entropy,
round_start, round_start,
spec.no_ambiguity, spec.ambiguity,
)?), )?),
None => Box::new(std::iter::repeat(0.0f32)), // never read: kernel forced to 1.0 below None => Box::new(std::iter::repeat(0.0f32)), // never read: kernel forced to 1.0 below
}; };
@@ -432,7 +441,7 @@ fn sample_layer(
&layer_dir, &layer_dir,
ScalarKind::PairWeight, ScalarKind::PairWeight,
round_start, round_start,
spec.no_ambiguity, spec.ambiguity,
)?) )?)
} else { } else {
Box::new(std::iter::repeat(1.0f32)) Box::new(std::iter::repeat(1.0f32))
@@ -475,19 +484,19 @@ fn sample_layer(
n_genomes, n_genomes,
fast_mode, fast_mode,
&Selection::Some(&accepted), &Selection::Some(&accepted),
|family_idx, mask, genome_mask| { |family_idx, kmer, mask, genome_mask| {
debug_assert!( debug_assert!(
mask.family_size() >= 2, mask.family_size() >= 2,
"eligible bitset must only accept non-monomorphic minorants" "eligible bitset must only accept non-monomorphic minorants"
); );
let masked: Vec<Option<u8>> = genome_mask let masked: Vec<Option<u8>> = genome_mask
.iter() .iter()
.map(|&m| masked_state(m, spec.free_loss, spec.no_ambiguity)) .map(|&m| masked_state(m, spec.free_loss, spec.ambiguity))
.collect(); .collect();
if survives_masking(&masked, excluded) if survives_masking(&masked, excluded)
&& (!spec.quartet || survives_quartet(&masked, excluded)) && (!spec.quartet || survives_quartet(&masked, excluded))
{ {
survivors.push(SurvivingFamily { family_idx, mask, masked }); survivors.push(SurvivingFamily { family_idx, kmer, mask, masked });
kept += 1; kept += 1;
} }
}, },
+96
View File
@@ -0,0 +1,96 @@
//! How a genome carrying several members of a family is treated — the one
//! parameter behind `--no-ambiguity` / `--only-diploid`, and the key of every
//! persisted annex whose values depend on which genomes count as carriers
//! (entropy, quartet, pair weight, family overlap).
/// What becomes of a genome carrying more than one member of a family.
///
/// - [`Keep`](Self::Keep): nothing — its set of members is a state of its
/// own (an IUPAC ambiguity code in the alignment).
/// - [`OnlyDiploid`](Self::OnlyDiploid): kept up to two members (a
/// heterozygote); three or four members make it missing data.
/// - [`Forbid`](Self::Forbid) (`--no-ambiguity`): any genome with several
/// members is missing data.
///
/// The order is the on-disk header code (see [`header_flags`](Self::header_flags)),
/// kept so that annexes written when this was a `bool` stay valid.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum AmbiguityPolicy {
#[default]
Keep,
Forbid,
OnlyDiploid,
}
impl AmbiguityPolicy {
/// Every policy, in header-code order.
pub const ALL: [AmbiguityPolicy; 3] = [Self::Keep, Self::Forbid, Self::OnlyDiploid];
/// From the two (mutually exclusive) CLI flags.
pub fn from_flags(no_ambiguity: bool, only_diploid: bool) -> Self {
debug_assert!(!(no_ambiguity && only_diploid), "--no-ambiguity and --only-diploid are exclusive");
if no_ambiguity {
Self::Forbid
} else if only_diploid {
Self::OnlyDiploid
} else {
Self::Keep
}
}
/// Whether a genome whose family presence mask is `mask` becomes missing
/// data under this policy.
#[inline]
pub fn drops(self, mask: u8) -> bool {
match self {
Self::Keep => false,
Self::OnlyDiploid => mask.count_ones() > 2,
Self::Forbid => mask.count_ones() > 1,
}
}
/// The `flags` value persisted in an annex header: `0` for `Keep` and `1`
/// for `Forbid` are what the `no_ambiguity` bit used to be.
pub(crate) fn header_flags(self) -> u32 {
match self {
Self::Keep => 0,
Self::Forbid => 1,
Self::OnlyDiploid => 2,
}
}
/// Inverse of [`header_flags`](Self::header_flags); unknown codes read as
/// `Keep`, as a file from before the flag existed.
pub(crate) fn from_header_flags(flags: u32) -> Self {
match flags {
1 => Self::Forbid,
2 => Self::OnlyDiploid,
_ => Self::Keep,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn drops_follows_the_member_count() {
for (mask, keep, diploid, forbid) in
[(0b0000u8, false, false, false), (0b0100, false, false, false), (0b0101, false, false, true), (0b0111, false, true, true), (0b1111, false, true, true)]
{
assert_eq!(AmbiguityPolicy::Keep.drops(mask), keep);
assert_eq!(AmbiguityPolicy::OnlyDiploid.drops(mask), diploid);
assert_eq!(AmbiguityPolicy::Forbid.drops(mask), forbid);
}
}
#[test]
fn header_flags_roundtrip_and_stay_compatible_with_the_old_bool() {
for p in AmbiguityPolicy::ALL {
assert_eq!(AmbiguityPolicy::from_header_flags(p.header_flags()), p);
}
assert_eq!(AmbiguityPolicy::Keep.header_flags(), 0);
assert_eq!(AmbiguityPolicy::Forbid.header_flags(), 1);
}
}
@@ -12,10 +12,13 @@ use obikindex::{OKIError, OKIResult};
use obisys::progress_bar; use obisys::progress_bar;
use crate::siblings::algorithms::{ use crate::siblings::algorithms::{
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes, AnnexRequest, build_layer_sibling_annex, ensure_layer_family_annexes,
inverse_overlap_matrix, is_fast_mode, ensure_layer_pair_weight_annexes, inverse_overlap_matrix, is_fast_mode,
};
use crate::siblings::{
ANNEX_FILE_NAME, AmbiguityPolicy, FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, FamilyOverlapAccumulator,
QUARTET_ANNEX_FILE_NAMES, ScalarKind, family_overlap_file_name,
}; };
use crate::siblings::{FamilyOverlap, FamilyOverlapAccumulator, family_overlap_file_name};
pub(crate) trait SiblingBuilder { pub(crate) trait SiblingBuilder {
/// Build, for every cached layer, the annexes `request` asks for that /// Build, for every cached layer, the annexes `request` asks for that
@@ -26,23 +29,139 @@ pub(crate) trait SiblingBuilder {
/// a missing entropy, quartet or overlap annex triggers one scan that /// a missing entropy, quartet or overlap annex triggers one scan that
/// feeds every missing one, per `ensure_layer_family_annexes`'s own docs, /// feeds every missing one, per `ensure_layer_family_annexes`'s own docs,
/// then a second scan builds the missing pair weights, which need the /// then a second scan builds the missing pair weights, which need the
/// finished overlap matrix. A no-op, cheap existence-check pass, once /// finished overlap matrix. Makes sure the sibling annex exists first
/// everything requested already exists. /// ([`ensure_sibling_annex`](Self::ensure_sibling_annex)). A no-op, cheap
/// existence-check pass that takes no lock, once everything requested
/// already exists; otherwise the build runs under the index's exclusive
/// lock ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock)).
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>; fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
/// [`ensure_family_annexes`](Self::ensure_family_annexes)'s build step,
/// without the existence pre-check and without the lock: the caller
/// holds it.
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
/// Build the sibling annex (`siblings.psib`, every annex above is derived
/// from it) if any cached layer lacks it — the state of an index right
/// after `migrate-mphf`, which sets every annex aside. The build writes
/// into every layer, so it runs under the index's exclusive lock
/// ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock), a no-op
/// when the caller already holds it); a process that waited for it finds
/// the annex built by the one that held it, and does nothing. A no-op,
/// cheap existence-check pass that takes no lock, once every layer has
/// its annex.
fn ensure_sibling_annex(&self) -> OKIResult<()>;
/// (Re)build the sibling annex of every cached layer, unconditionally,
/// removing the annexes derived from it (entropy, quartet, pair weight,
/// family overlap), which its renumbering makes stale. Takes no lock:
/// the caller holds it ([`SiblingExt::build_sibling_annex`], or
/// [`ensure_sibling_annex`](Self::ensure_sibling_annex)).
fn build_base_sibling_annex(&self) -> OKIResult<()>;
} }
impl SiblingBuilder for IndexCache { impl SiblingBuilder for IndexCache {
fn ensure_sibling_annex(&self) -> OKIResult<()> {
let missing = || self.iter_indexed().any(|(_, _, layer)| !layer.dir().join(ANNEX_FILE_NAME).exists());
if !missing() {
return Ok(());
}
self.index().with_lock(|| {
if !missing() {
return Ok(()); // built by the process we waited for
}
tracing::info!("sibling annex missing — building it");
self.build_base_sibling_annex()
})
}
fn build_base_sibling_annex(&self) -> OKIResult<()> {
let n_parts = self.index().n_partitions();
let fast_mode = is_fast_mode(self);
if !fast_mode {
tracing::warn!(
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
disabled; consider compacting this index"
);
}
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
let mut total_slots: u64 = 0;
for part in self.partitions() {
let n_layer = self.n_layer(part).unwrap_or(0);
let mut part_slots: u64 = 0;
for l in 0..n_layer {
let layer = self
.get_layer(part, l)
.expect("cache-consistent: n_layer(part) already bounds l");
// Invalidate this layer's entropy, quartet and pair-weight
// annexes, if any — they were built against the sibling
// annex's *previous* `family_idx` numbering
// (`non_monomorphic_reindex_layer` reads the sibling
// annex), and are stale once the new one exists.
// `ensure_family_annexes` only builds what's *missing*, so
// the old files must be removed here for them to be rebuilt
// rather than silently left stale. Removed *before* the new
// annex is built and published (atomically, at its end): a
// build killed in between leaves the previous annex with no
// derived files — still consistent, rebuilt on demand —
// never a new annex next to derived files computed against
// the old numbering.
for name in ScalarKind::Entropy
.file_names()
.into_iter()
.chain(ScalarKind::PairWeight.file_names())
.chain(QUARTET_ANNEX_FILE_NAMES)
{
let derived_path = layer.dir().join(name);
if derived_path.exists() {
std::fs::remove_file(&derived_path).map_err(OKIError::Io)?;
}
}
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
}
total_slots += part_slots;
pb.inc(1);
pb.set_message(format!(
"partition {part}: {part_slots} kmers ({total_slots} total)"
));
}
pb.finish_and_clear();
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
// The family-overlap annexes are index-level, not per-layer (see
// their own module docs) — any layer's sibling annex changing makes
// them all stale (and, with them, every pair weight computed from
// them, removed per layer above), so they are invalidated once here
// rather than per-layer above.
for name in FAMILY_OVERLAP_FILE_NAMES {
let overlap_path = self.index().dir().join(name);
if overlap_path.exists() {
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
}
}
Ok(())
}
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> { fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
self.ensure_sibling_annex()?;
if request.is_satisfied(self) {
return Ok(());
}
self.index().with_lock(|| self.build_family_annexes(request))
}
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
let n_genomes = self.meta().genomes().len(); let n_genomes = self.meta().genomes().len();
let fast_mode = is_fast_mode(self); let fast_mode = is_fast_mode(self);
let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity)); let overlap_path = |policy: AmbiguityPolicy| self.index().dir().join(family_overlap_file_name(policy));
// Sweep 1: entropy, quartet, and every family-overlap annex still missing. // Sweep 1: entropy, quartet, and every family-overlap annex still missing.
let mut overlap_accs: Vec<(bool, FamilyOverlapAccumulator)> = request let mut overlap_accs: Vec<(AmbiguityPolicy, FamilyOverlapAccumulator)> = request
.overlap_conventions() .overlap_conventions()
.into_iter() .into_iter()
.filter(|&no_ambiguity| !overlap_path(no_ambiguity).exists()) .filter(|&policy| !overlap_path(policy).exists())
.map(|no_ambiguity| (no_ambiguity, FamilyOverlapAccumulator::new(n_genomes, no_ambiguity))) .map(|policy| (policy, FamilyOverlapAccumulator::new(n_genomes, policy)))
.collect(); .collect();
let pb = progress_bar("family_annexes", self.n_partition() as u64, "partitions"); let pb = progress_bar("family_annexes", self.n_partition() as u64, "partitions");
for part in self.partitions() { for part in self.partitions() {
@@ -53,8 +172,8 @@ impl SiblingBuilder for IndexCache {
pb.inc(1); pb.inc(1);
} }
pb.finish_and_clear(); pb.finish_and_clear();
for (no_ambiguity, acc) in overlap_accs { for (policy, acc) in overlap_accs {
acc.write(&overlap_path(no_ambiguity)).map_err(OKIError::Io)?; acc.write(&overlap_path(policy)).map_err(OKIError::Io)?;
} }
// Sweep 2: pair weights, from the now-complete overlap annexes. // Sweep 2: pair weights, from the now-complete overlap annexes.
@@ -64,9 +183,9 @@ impl SiblingBuilder for IndexCache {
let matrices = request let matrices = request
.pair_weight_conventions() .pair_weight_conventions()
.into_iter() .into_iter()
.map(|no_ambiguity| { .map(|policy| {
FamilyOverlap::open(&overlap_path(no_ambiguity)) FamilyOverlap::open(&overlap_path(policy))
.map(|overlap| (no_ambiguity, inverse_overlap_matrix(&overlap))) .map(|overlap| (policy, inverse_overlap_matrix(&overlap)))
.map_err(OKIError::Io) .map_err(OKIError::Io)
}) })
.collect::<OKIResult<Vec<_>>>()?; .collect::<OKIResult<Vec<_>>>()?;
@@ -11,17 +11,12 @@ use std::path::Path;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikindex::{OKIError, OKIResult}; use obikindex::{OKIError, OKIResult};
use obisys::progress_bar;
use crate::siblings::algorithms::{ use crate::siblings::algorithms::{
AnnexRequest, EntropySummary, SiblingAnnexStats, build_layer_sibling_annex, is_fast_mode, AnnexRequest, AnnexSelection, EntropySummary, SiblingAnnexStats, scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
}; };
use crate::siblings::extensions::SiblingBuilder; use crate::siblings::extensions::SiblingBuilder;
use crate::siblings::{ use crate::siblings::{AmbiguityPolicy, FamilyOverlap, family_overlap_file_name};
FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, QUARTET_ANNEX_FILE_NAMES, ScalarKind,
family_overlap_file_name,
};
pub trait SiblingExt { pub trait SiblingExt {
/// Build the sibling-count/minorant annex for every layer of every /// Build the sibling-count/minorant annex for every layer of every
@@ -44,7 +39,14 @@ pub trait SiblingExt {
/// Cross-partition/cross-layer lookups are required (a k-mer's siblings /// Cross-partition/cross-layer lookups are required (a k-mer's siblings
/// can live in any partition), but the layer loop itself — and thus the /// can live in any partition), but the layer loop itself — and thus the
/// annex file this produces — stays local to one layer at a time. /// annex file this produces — stays local to one layer at a time.
fn build_sibling_annex(&self) -> OKIResult<()>; ///
/// `derived` names the annexes derived from the sibling annex (entropy,
/// quartet, pair weight, family overlap) to have ready afterwards, all
/// under its one ambiguity policy — only those, so a routine rebuild
/// pays for nothing the following computations will not read. Derived
/// annexes that already existed are invalidated by the rebuild and are
/// rebuilt on demand, by the computation that reads them, if not listed.
fn build_sibling_annex(&self, derived: &AnnexSelection) -> OKIResult<()>;
/// Write a CSV (`partition,layer,family_idx,entropy15,entropy4, /// Write a CSV (`partition,layer,family_idx,entropy15,entropy4,
/// family_size,n_genomes_present,n_genomes_unambiguous`) of per-family /// family_size,n_genomes_present,n_genomes_unambiguous`) of per-family
@@ -96,9 +98,9 @@ pub trait SiblingExt {
/// The Family Overlap annex — number of shared *variable* families per /// The Family Overlap annex — number of shared *variable* families per
/// genome pair, index-wide (see [`FamilyOverlap`]'s own module docs), /// genome pair, index-wide (see [`FamilyOverlap`]'s own module docs),
/// under the `--no-ambiguity` counting convention `no_ambiguity` selects /// under the counting convention `policy` selects (which genomes
/// (genomes carrying several members of a family do not count as /// carrying several members of a family still count as carrying it, see
/// carrying it when `true`). Built if missing, same as every other /// [`AmbiguityPolicy`]). Built if missing, same as every other
/// annex-backed method here. Feeds `--family-overlap`'s own matrix /// annex-backed method here. Feeds `--family-overlap`'s own matrix
/// output directly, and `--min-shared-family`'s auto-exclusion /// output directly, and `--min-shared-family`'s auto-exclusion
/// (`FamilyOverlap::mean_row(g) < threshold` — same statistic /// (`FamilyOverlap::mean_row(g) < threshold` — same statistic
@@ -106,87 +108,23 @@ pub trait SiblingExt {
/// triggers adding `g` to the excluded-genome set consumed by /// triggers adding `g` to the excluded-genome set consumed by
/// `algorithms::scan::sampled_scan`'s callers; this needs no /// `algorithms::scan::sampled_scan`'s callers; this needs no
/// computation of its own beyond that check). /// computation of its own beyond that check).
fn family_overlap(&self, no_ambiguity: bool) -> OKIResult<FamilyOverlap>; fn family_overlap(&self, policy: AmbiguityPolicy) -> OKIResult<FamilyOverlap>;
} }
impl SiblingExt for IndexCache { impl SiblingExt for IndexCache {
fn build_sibling_annex(&self) -> OKIResult<()> { fn build_sibling_annex(&self, derived: &AnnexSelection) -> OKIResult<()> {
let n_parts = self.index().n_partitions(); self.index().with_lock(|| {
let fast_mode = is_fast_mode(self); self.build_base_sibling_annex()?;
if !fast_mode { // Then exactly the derived annexes asked for, now that every
tracing::warn!( // layer's fresh sibling annex is on disk
"index has more than 7 layers per partition — sibling-annex fast layer lookup \ // (`non_monomorphic_reindex_layer`, which they depend on, reads
disabled; consider compacting this index" // it).
); if derived.is_empty() {
} return Ok(());
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
let mut total_slots: u64 = 0;
for part in self.partitions() {
let n_layer = self.n_layer(part).unwrap_or(0);
let mut part_slots: u64 = 0;
for l in 0..n_layer {
let layer = self
.get_layer(part, l)
.expect("cache-consistent: n_layer(part) already bounds l");
// Invalidate this layer's entropy and quartet annexes, if
// any — they were built against the sibling annex's
// *previous* `family_idx` numbering
// (`non_monomorphic_reindex_layer` reads the sibling
// annex), and are stale once the new one exists.
// `ensure_family_annexes` below only builds what's
// *missing*, so the old files must be removed here for
// them to be rebuilt rather than silently left stale.
// Removed *before* the new annex is built and published
// (atomically, at its end): a build killed in between
// leaves the previous annex with no derived files — still
// consistent, rebuilt on demand — never a new annex next to
// derived files computed against the old numbering.
for name in ScalarKind::Entropy
.file_names()
.into_iter()
.chain(ScalarKind::PairWeight.file_names())
.chain(QUARTET_ANNEX_FILE_NAMES)
{
let entropy_path = layer.dir().join(name);
if entropy_path.exists() {
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
}
}
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
} }
total_slots += part_slots; tracing::info!("building the requested derived annexes ({:?})", derived.policy);
pb.inc(1); self.ensure_family_annexes(&derived.request())
pb.set_message(format!( })
"partition {part}: {part_slots} kmers ({total_slots} total)"
));
}
pb.finish_and_clear();
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
// The family-overlap annexes are index-level, not per-layer (see
// their own module docs) — any layer's sibling annex changing makes
// them all stale (and, with them, every pair weight computed from
// them, removed per layer above), so they are invalidated once here
// rather than per-layer above.
for name in FAMILY_OVERLAP_FILE_NAMES {
let overlap_path = self.index().dir().join(name);
if overlap_path.exists() {
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
}
}
// Precompute the entropy and family-overlap annexes too, now that
// every layer's fresh sibling annex is on disk
// (`non_monomorphic_reindex_layer`, which both depend on, reads
// it) — same routine-rebuild footing as the sibling annex itself,
// rather than leaving them to be built lazily on first
// `--shannon`/`--entropy-bias`/`--min-shared-family`/
// `--family-overlap` use.
tracing::info!("building entropy and quartet (both --no-ambiguity conventions) and family-overlap annexes");
self.ensure_family_annexes(&AnnexRequest::everything())?;
Ok(())
} }
fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> { fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> {
@@ -229,9 +167,9 @@ impl SiblingExt for IndexCache {
sibling_annex_stats(self) sibling_annex_stats(self)
} }
fn family_overlap(&self, no_ambiguity: bool) -> OKIResult<FamilyOverlap> { fn family_overlap(&self, policy: AmbiguityPolicy) -> OKIResult<FamilyOverlap> {
self.ensure_family_annexes(&AnnexRequest { overlap: vec![no_ambiguity], ..Default::default() })?; self.ensure_family_annexes(&AnnexRequest { overlap: vec![policy], ..Default::default() })?;
let path = self.index().dir().join(family_overlap_file_name(no_ambiguity)); let path = self.index().dir().join(family_overlap_file_name(policy));
FamilyOverlap::open(&path).map_err(OKIError::Io) FamilyOverlap::open(&path).map_err(OKIError::Io)
} }
} }
+49 -41
View File
@@ -25,35 +25,41 @@ use std::path::Path;
use memmap2::Mmap; use memmap2::Mmap;
use obicompactvec::{building_path, publish_built}; use obicompactvec::{building_path, publish_built};
use crate::siblings::AmbiguityPolicy;
use crate::siblings::helpers::{is_carrier, triangle_index}; use crate::siblings::helpers::{is_carrier, triangle_index};
const MAGIC: [u8; 4] = *b"POVL"; const MAGIC: [u8; 4] = *b"POVL";
// Header: magic(4) + flags(4) + n_genomes(8) = 16 bytes. Data (8 bytes/pair, // Header: magic(4) + flags(4) + n_genomes(8) = 16 bytes. Data (8 bytes/pair,
// upper triangle) follows. `flags` bit 0 = `no_ambiguity` (see // upper triangle) follows. `flags` = `AmbiguityPolicy::header_flags` (see
// `FamilyOverlapAccumulator::new`); older files have `flags == 0`, i.e. // `FamilyOverlapAccumulator::new`); older files have `flags == 0`, i.e.
// `no_ambiguity == false`, which is what they were computed under. // `Keep`, which is what they were computed under.
const HEADER_SIZE: usize = 16; const HEADER_SIZE: usize = 16;
const FLAG_NO_AMBIGUITY: u32 = 1;
pub(crate) const FAMILY_OVERLAP_FILE_NAME: &str = "family_overlap.povl"; pub(crate) const FAMILY_OVERLAP_FILE_NAME: &str = "family_overlap.povl";
/// Same annex, built with `no_ambiguity == true`. /// Same annex, built under [`AmbiguityPolicy::Forbid`].
pub(crate) const FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME: &str = "family_overlap.noamb.povl"; pub(crate) const FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME: &str = "family_overlap.noamb.povl";
/// Same annex, built under [`AmbiguityPolicy::OnlyDiploid`].
pub(crate) const FAMILY_OVERLAP_ONLY_DIPLOID_FILE_NAME: &str = "family_overlap.diploid.povl";
/// Every family-overlap file name, whatever its counting convention — for /// Every family-overlap file name, whatever its counting convention — for
/// invalidating them all together when the sibling annex they derive from is /// invalidating them all together when the sibling annex they derive from is
/// rebuilt. /// rebuilt.
pub(crate) const FAMILY_OVERLAP_FILE_NAMES: [&str; 2] = pub(crate) const FAMILY_OVERLAP_FILE_NAMES: [&str; 3] = [
[FAMILY_OVERLAP_FILE_NAME, FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME]; FAMILY_OVERLAP_FILE_NAME,
FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME,
FAMILY_OVERLAP_ONLY_DIPLOID_FILE_NAME,
];
/// File name of the family-overlap annex built under the given /// File name of the family-overlap annex built under the given ambiguity
/// `--no-ambiguity` counting convention. The two never share a file. /// policy. The policies never share a file.
pub(crate) fn family_overlap_file_name(no_ambiguity: bool) -> &'static str { pub(crate) fn family_overlap_file_name(policy: AmbiguityPolicy) -> &'static str {
if no_ambiguity { match policy {
FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME AmbiguityPolicy::Keep => FAMILY_OVERLAP_FILE_NAME,
} else { AmbiguityPolicy::Forbid => FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME,
FAMILY_OVERLAP_FILE_NAME AmbiguityPolicy::OnlyDiploid => FAMILY_OVERLAP_ONLY_DIPLOID_FILE_NAME,
} }
} }
@@ -61,7 +67,7 @@ pub(crate) fn family_overlap_file_name(no_ambiguity: bool) -> &'static str {
pub struct FamilyOverlap { pub struct FamilyOverlap {
mmap: Mmap, mmap: Mmap,
n_genomes: usize, n_genomes: usize,
no_ambiguity: bool, policy: AmbiguityPolicy,
} }
impl FamilyOverlap { impl FamilyOverlap {
@@ -79,19 +85,17 @@ impl FamilyOverlap {
return Err(io::Error::new(io::ErrorKind::InvalidData, "POVL file truncated")); return Err(io::Error::new(io::ErrorKind::InvalidData, "POVL file truncated"));
} }
let flags = u32::from_le_bytes(mmap[4..8].try_into().unwrap()); let flags = u32::from_le_bytes(mmap[4..8].try_into().unwrap());
Ok(Self { mmap, n_genomes, no_ambiguity: flags & FLAG_NO_AMBIGUITY != 0 }) Ok(Self { mmap, n_genomes, policy: AmbiguityPolicy::from_header_flags(flags) })
} }
pub fn n_genomes(&self) -> usize { pub fn n_genomes(&self) -> usize {
self.n_genomes self.n_genomes
} }
/// The counting convention these counts were computed under: `true` /// The counting convention these counts were computed under (see
/// when a genome carrying several members of a family was *not* counted /// [`AmbiguityPolicy`]): which genomes count as carrying a family.
/// as carrying it (`--no-ambiguity`), `false` when any non-empty pub fn policy(&self) -> AmbiguityPolicy {
/// state counts as carrying it. self.policy
pub fn no_ambiguity(&self) -> bool {
self.no_ambiguity
} }
/// Number of variable families shared by genomes `i` and `j` — `0` for /// Number of variable families shared by genomes `i` and `j` — `0` for
@@ -128,18 +132,17 @@ impl FamilyOverlap {
/// `EntropyAnnexBuilder` do for their own, much larger, per-k-mer data. /// `EntropyAnnexBuilder` do for their own, much larger, per-k-mer data.
pub(crate) struct FamilyOverlapAccumulator { pub(crate) struct FamilyOverlapAccumulator {
n_genomes: usize, n_genomes: usize,
no_ambiguity: bool, policy: AmbiguityPolicy,
counts: Vec<u64>, counts: Vec<u64>,
} }
impl FamilyOverlapAccumulator { impl FamilyOverlapAccumulator {
/// `no_ambiguity` is the counting convention: `false` counts every /// `policy` is the counting convention: a genome counts as carrying the
/// genome with a non-empty state as carrying the family, `true` only the /// family unless the policy makes it missing data in the alignment, in
/// genomes carrying exactly one member of it — the others are missing /// which case it shares nothing there (see [`AmbiguityPolicy::drops`]).
/// data in the alignment (`--no-ambiguity`), so they share nothing there. pub(crate) fn new(n_genomes: usize, policy: AmbiguityPolicy) -> Self {
pub(crate) fn new(n_genomes: usize, no_ambiguity: bool) -> Self {
let n_pairs = n_genomes.saturating_sub(1) * n_genomes / 2; let n_pairs = n_genomes.saturating_sub(1) * n_genomes / 2;
Self { n_genomes, no_ambiguity, counts: vec![0u64; n_pairs] } Self { n_genomes, policy, counts: vec![0u64; n_pairs] }
} }
/// One variable family's contribution: every pair of genomes both /// One variable family's contribution: every pair of genomes both
@@ -147,11 +150,11 @@ impl FamilyOverlapAccumulator {
/// once. /// once.
pub(crate) fn add_family(&mut self, genome_mask: &[u8]) { pub(crate) fn add_family(&mut self, genome_mask: &[u8]) {
for i in 0..self.n_genomes { for i in 0..self.n_genomes {
if !is_carrier(genome_mask[i], self.no_ambiguity) { if !is_carrier(genome_mask[i], self.policy) {
continue; continue;
} }
for j in (i + 1)..self.n_genomes { for j in (i + 1)..self.n_genomes {
if is_carrier(genome_mask[j], self.no_ambiguity) { if is_carrier(genome_mask[j], self.policy) {
self.counts[triangle_index(self.n_genomes, i, j)] += 1; self.counts[triangle_index(self.n_genomes, i, j)] += 1;
} }
} }
@@ -161,7 +164,7 @@ impl FamilyOverlapAccumulator {
pub(crate) fn write(&self, path: &Path) -> io::Result<()> { pub(crate) fn write(&self, path: &Path) -> io::Result<()> {
let mut buf = Vec::with_capacity(HEADER_SIZE + self.counts.len() * 8); let mut buf = Vec::with_capacity(HEADER_SIZE + self.counts.len() * 8);
buf.extend_from_slice(&MAGIC); buf.extend_from_slice(&MAGIC);
let flags = if self.no_ambiguity { FLAG_NO_AMBIGUITY } else { 0 }; let flags = self.policy.header_flags();
buf.extend_from_slice(&flags.to_le_bytes()); buf.extend_from_slice(&flags.to_le_bytes());
buf.extend_from_slice(&(self.n_genomes as u64).to_le_bytes()); buf.extend_from_slice(&(self.n_genomes as u64).to_le_bytes());
for &c in &self.counts { for &c in &self.counts {
@@ -181,7 +184,7 @@ mod tests {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("test.povl"); let path = dir.path().join("test.povl");
let mut acc = FamilyOverlapAccumulator::new(3, false); let mut acc = FamilyOverlapAccumulator::new(3, AmbiguityPolicy::Keep);
acc.add_family(&[1, 1, 0]); // genomes 0,1 share this family acc.add_family(&[1, 1, 0]); // genomes 0,1 share this family
acc.add_family(&[1, 1, 1]); // all three share this one acc.add_family(&[1, 1, 1]); // all three share this one
acc.add_family(&[0, 1, 1]); // genomes 1,2 share this family acc.add_family(&[0, 1, 1]); // genomes 1,2 share this family
@@ -201,26 +204,31 @@ mod tests {
fn single_genome_mean_row_is_zero() { fn single_genome_mean_row_is_zero() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("test.povl"); let path = dir.path().join("test.povl");
FamilyOverlapAccumulator::new(1, false).write(&path).unwrap(); FamilyOverlapAccumulator::new(1, AmbiguityPolicy::Keep).write(&path).unwrap();
let overlap = FamilyOverlap::open(&path).unwrap(); let overlap = FamilyOverlap::open(&path).unwrap();
assert_eq!(overlap.mean_row(0), 0.0); assert_eq!(overlap.mean_row(0), 0.0);
} }
#[test] #[test]
fn no_ambiguity_does_not_count_multi_member_genomes_as_carriers() { fn the_policy_decides_which_multi_member_genomes_are_carriers() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let masks: [&[u8]; 2] = [&[1, 1, 0b0011], &[1, 0b0100, 0b0100]]; // genome 2: two members in the first family, three in the second
for no_ambiguity in [false, true] { let masks: [&[u8]; 2] = [&[1, 1, 0b0011], &[1, 0b0100, 0b0111]];
let path = dir.path().join(family_overlap_file_name(no_ambiguity)); for policy in AmbiguityPolicy::ALL {
let mut acc = FamilyOverlapAccumulator::new(3, no_ambiguity); let path = dir.path().join(family_overlap_file_name(policy));
let mut acc = FamilyOverlapAccumulator::new(3, policy);
for m in masks { for m in masks {
acc.add_family(m); acc.add_family(m);
} }
acc.write(&path).unwrap(); acc.write(&path).unwrap();
let overlap = FamilyOverlap::open(&path).unwrap(); let overlap = FamilyOverlap::open(&path).unwrap();
assert_eq!(overlap.no_ambiguity(), no_ambiguity); assert_eq!(overlap.policy(), policy);
// genome 2 carries two members in the first family: only counted without the flag let shared_with_2 = match policy {
assert_eq!(overlap.get(0, 2), if no_ambiguity { 1 } else { 2 }); AmbiguityPolicy::Keep => 2,
AmbiguityPolicy::OnlyDiploid => 1,
AmbiguityPolicy::Forbid => 0,
};
assert_eq!(overlap.get(0, 2), shared_with_2);
assert_eq!(overlap.get(0, 1), 1 + 1); assert_eq!(overlap.get(0, 1), 1 + 1);
} }
} }
@@ -25,12 +25,13 @@ use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut}; use memmap2::{Mmap, MmapMut};
use obicompactvec::{building_path, publish_built}; use obicompactvec::{building_path, publish_built};
use crate::siblings::AmbiguityPolicy;
// Header: magic(4) + flags(4) + n(8) = 16 bytes. Data (4 bytes/entry) follows. // Header: magic(4) + flags(4) + n(8) = 16 bytes. Data (4 bytes/entry) follows.
// `flags` bit 0 = `no_ambiguity` (the counting convention the values were // `flags` = `AmbiguityPolicy::header_flags` (the counting convention the
// computed under, see `FamilyScalarAnnexBuilder::new`); every file written // values were computed under, see `FamilyScalarAnnexBuilder::new`); every file
// before the flag existed has `flags == 0`, i.e. `no_ambiguity == false`, // written before the flag existed has `flags == 0`, i.e. `Keep`, which is what
// which is what it was computed under. // it was computed under.
const FLAG_NO_AMBIGUITY: u32 = 1;
const HEADER_SIZE: usize = 16; const HEADER_SIZE: usize = 16;
/// Sentinel: not yet computed. Every stored value is `>= 0.0`, so any /// Sentinel: not yet computed. Every stored value is `>= 0.0`, so any
@@ -56,31 +57,32 @@ impl ScalarKind {
} }
} }
/// File name of this kind's annex built under the given `--no-ambiguity` /// File name of this kind's annex built under the given ambiguity
/// counting convention. The two conventions never share a file: /// policy. The policies never share a file: switching between them builds
/// switching between them builds the missing annex without discarding the /// the missing annex without discarding the others.
/// other. pub(crate) fn file_name(self, policy: AmbiguityPolicy) -> &'static str {
pub(crate) fn file_name(self, no_ambiguity: bool) -> &'static str { match (self, policy) {
match (self, no_ambiguity) { (ScalarKind::Entropy, AmbiguityPolicy::Keep) => "entropy.pent",
(ScalarKind::Entropy, false) => "entropy.pent", (ScalarKind::Entropy, AmbiguityPolicy::Forbid) => "entropy.noamb.pent",
(ScalarKind::Entropy, true) => "entropy.noamb.pent", (ScalarKind::Entropy, AmbiguityPolicy::OnlyDiploid) => "entropy.diploid.pent",
(ScalarKind::PairWeight, false) => "pairweight.ppwt", (ScalarKind::PairWeight, AmbiguityPolicy::Keep) => "pairweight.ppwt",
(ScalarKind::PairWeight, true) => "pairweight.noamb.ppwt", (ScalarKind::PairWeight, AmbiguityPolicy::Forbid) => "pairweight.noamb.ppwt",
(ScalarKind::PairWeight, AmbiguityPolicy::OnlyDiploid) => "pairweight.diploid.ppwt",
} }
} }
/// Every file name of this kind, whatever its counting convention — for /// Every file name of this kind, whatever its counting convention — for
/// invalidating them all together (e.g. when the sibling annex they were /// invalidating them all together (e.g. when the sibling annex they were
/// numbered against is rebuilt). /// numbered against is rebuilt).
pub(crate) fn file_names(self) -> [&'static str; 2] { pub(crate) fn file_names(self) -> [&'static str; 3] {
[self.file_name(false), self.file_name(true)] AmbiguityPolicy::ALL.map(|p| self.file_name(p))
} }
} }
pub struct FamilyScalarAnnex { pub struct FamilyScalarAnnex {
mmap: Mmap, mmap: Mmap,
n: usize, n: usize,
no_ambiguity: bool, policy: AmbiguityPolicy,
} }
impl FamilyScalarAnnex { impl FamilyScalarAnnex {
@@ -100,19 +102,16 @@ impl FamilyScalarAnnex {
return Err(io::Error::new(io::ErrorKind::InvalidData, "scalar annex file truncated")); return Err(io::Error::new(io::ErrorKind::InvalidData, "scalar annex file truncated"));
} }
let flags = u32::from_le_bytes(mmap[4..8].try_into().unwrap()); let flags = u32::from_le_bytes(mmap[4..8].try_into().unwrap());
Ok(Self { mmap, n, no_ambiguity: flags & FLAG_NO_AMBIGUITY != 0 }) Ok(Self { mmap, n, policy: AmbiguityPolicy::from_header_flags(flags) })
} }
pub fn len(&self) -> usize { pub fn len(&self) -> usize {
self.n self.n
} }
/// The counting convention these values were computed under: `true` /// The counting convention these values were computed under.
/// when genomes carrying several members of a family were left out of pub fn policy(&self) -> AmbiguityPolicy {
/// the count (`--no-ambiguity`), `false` when they were counted as a self.policy
/// state of their own.
pub fn no_ambiguity(&self) -> bool {
self.no_ambiguity
} }
/// `None` for the sentinel (not yet computed — shouldn't happen against /// `None` for the sentinel (not yet computed — shouldn't happen against
@@ -132,9 +131,9 @@ pub struct FamilyScalarAnnexBuilder {
impl FamilyScalarAnnexBuilder { impl FamilyScalarAnnexBuilder {
/// Create a new annex of `n` entries (a layer's non-monomorphic /// Create a new annex of `n` entries (a layer's non-monomorphic
/// minorant count) at `path`, pre-initialised to the sentinel. /// minorant count) at `path`, pre-initialised to the sentinel.
/// `no_ambiguity` records, in the header, the counting convention the /// `policy` records, in the header, the counting convention the values
/// values will be computed under (see [`FamilyScalarAnnex::no_ambiguity`]). /// will be computed under (see [`FamilyScalarAnnex::policy`]).
pub(crate) fn new(n: usize, path: &Path, kind: ScalarKind, no_ambiguity: bool) -> io::Result<Self> { pub(crate) fn new(n: usize, path: &Path, kind: ScalarKind, policy: AmbiguityPolicy) -> io::Result<Self> {
let file_size = HEADER_SIZE + n * 4; let file_size = HEADER_SIZE + n * 4;
let file = OpenOptions::new() let file = OpenOptions::new()
.read(true) .read(true)
@@ -145,7 +144,7 @@ impl FamilyScalarAnnexBuilder {
file.set_len(file_size as u64)?; file.set_len(file_size as u64)?;
let mut mmap = unsafe { MmapMut::map_mut(&file)? }; let mut mmap = unsafe { MmapMut::map_mut(&file)? };
mmap[0..4].copy_from_slice(&kind.magic()); mmap[0..4].copy_from_slice(&kind.magic());
let flags = if no_ambiguity { FLAG_NO_AMBIGUITY } else { 0 }; let flags = policy.header_flags();
mmap[4..8].copy_from_slice(&flags.to_le_bytes()); mmap[4..8].copy_from_slice(&flags.to_le_bytes());
mmap[8..16].copy_from_slice(&(n as u64).to_le_bytes()); mmap[8..16].copy_from_slice(&(n as u64).to_le_bytes());
for i in 0..n { for i in 0..n {
@@ -181,7 +180,7 @@ mod tests {
fn sentinel_is_negative_and_unset_entries_read_as_none() { fn sentinel_is_negative_and_unset_entries_read_as_none() {
let dir = tempdir().unwrap(); let dir = tempdir().unwrap();
let path = dir.path().join("test.pent"); let path = dir.path().join("test.pent");
FamilyScalarAnnexBuilder::new(4, &path, E, false).unwrap().close().unwrap(); FamilyScalarAnnexBuilder::new(4, &path, E, AmbiguityPolicy::Keep).unwrap().close().unwrap();
let annex = FamilyScalarAnnex::open(&path, E).unwrap(); let annex = FamilyScalarAnnex::open(&path, E).unwrap();
for i in 0..4 { for i in 0..4 {
assert_eq!(annex.get(i), None); assert_eq!(annex.get(i), None);
@@ -192,7 +191,7 @@ mod tests {
fn roundtrip_including_zero_value() { fn roundtrip_including_zero_value() {
let dir = tempdir().unwrap(); let dir = tempdir().unwrap();
let path = dir.path().join("test.pent"); let path = dir.path().join("test.pent");
let mut builder = FamilyScalarAnnexBuilder::new(4, &path, E, false).unwrap(); let mut builder = FamilyScalarAnnexBuilder::new(4, &path, E, AmbiguityPolicy::Keep).unwrap();
let values = [0.0f32, 1.5, 3.9, 0.0001]; let values = [0.0f32, 1.5, 3.9, 0.0001];
for (i, &v) in values.iter().enumerate() { for (i, &v) in values.iter().enumerate() {
builder.set(i, v); builder.set(i, v);
@@ -208,7 +207,7 @@ mod tests {
fn mixed_sentinel_and_real_values() { fn mixed_sentinel_and_real_values() {
let dir = tempdir().unwrap(); let dir = tempdir().unwrap();
let path = dir.path().join("test.pent"); let path = dir.path().join("test.pent");
let mut builder = FamilyScalarAnnexBuilder::new(3, &path, E, false).unwrap(); let mut builder = FamilyScalarAnnexBuilder::new(3, &path, E, AmbiguityPolicy::Keep).unwrap();
builder.set(1, 2.0); builder.set(1, 2.0);
builder.close().unwrap(); builder.close().unwrap();
let annex = FamilyScalarAnnex::open(&path, E).unwrap(); let annex = FamilyScalarAnnex::open(&path, E).unwrap();
@@ -222,12 +221,13 @@ mod tests {
fn counting_convention_roundtrips_through_the_header() { fn counting_convention_roundtrips_through_the_header() {
let dir = tempdir().unwrap(); let dir = tempdir().unwrap();
for kind in [ScalarKind::Entropy, ScalarKind::PairWeight] { for kind in [ScalarKind::Entropy, ScalarKind::PairWeight] {
for no_ambiguity in [false, true] { for policy in AmbiguityPolicy::ALL {
let path = dir.path().join(kind.file_name(no_ambiguity)); let path = dir.path().join(kind.file_name(policy));
FamilyScalarAnnexBuilder::new(2, &path, kind, no_ambiguity).unwrap().close().unwrap(); FamilyScalarAnnexBuilder::new(2, &path, kind, policy).unwrap().close().unwrap();
assert_eq!(FamilyScalarAnnex::open(&path, kind).unwrap().no_ambiguity(), no_ambiguity); assert_eq!(FamilyScalarAnnex::open(&path, kind).unwrap().policy(), policy);
} }
assert_ne!(kind.file_name(false), kind.file_name(true)); let names = kind.file_names();
assert!(names[0] != names[1] && names[1] != names[2] && names[0] != names[2]);
} }
} }
@@ -235,7 +235,7 @@ mod tests {
fn one_kind_cannot_be_read_as_the_other() { fn one_kind_cannot_be_read_as_the_other() {
let dir = tempdir().unwrap(); let dir = tempdir().unwrap();
let path = dir.path().join("w.ppwt"); let path = dir.path().join("w.ppwt");
FamilyScalarAnnexBuilder::new(1, &path, ScalarKind::PairWeight, false).unwrap().close().unwrap(); FamilyScalarAnnexBuilder::new(1, &path, ScalarKind::PairWeight, AmbiguityPolicy::Keep).unwrap().close().unwrap();
assert!(FamilyScalarAnnex::open(&path, ScalarKind::Entropy).is_err()); assert!(FamilyScalarAnnex::open(&path, ScalarKind::Entropy).is_err());
assert!(FamilyScalarAnnex::open(&path, ScalarKind::PairWeight).is_ok()); assert!(FamilyScalarAnnex::open(&path, ScalarKind::PairWeight).is_ok());
} }
@@ -245,9 +245,9 @@ mod tests {
// files written before the flag existed carried four zero bytes there // files written before the flag existed carried four zero bytes there
let dir = tempdir().unwrap(); let dir = tempdir().unwrap();
let path = dir.path().join("old.pent"); let path = dir.path().join("old.pent");
FamilyScalarAnnexBuilder::new(1, &path, E, false).unwrap().close().unwrap(); FamilyScalarAnnexBuilder::new(1, &path, E, AmbiguityPolicy::Keep).unwrap().close().unwrap();
let bytes = std::fs::read(&path).unwrap(); let bytes = std::fs::read(&path).unwrap();
assert_eq!(&bytes[4..8], &[0u8; 4]); assert_eq!(&bytes[4..8], &[0u8; 4]);
assert!(!FamilyScalarAnnex::open(&path, E).unwrap().no_ambiguity()); assert_eq!(FamilyScalarAnnex::open(&path, E).unwrap().policy(), AmbiguityPolicy::Keep);
} }
} }
+3 -3
View File
@@ -1,6 +1,6 @@
use obikseq::CanonicalKmer; use obikseq::CanonicalKmer;
use super::FamilyMask; use super::{AmbiguityPolicy, FamilyMask};
/// Central-position base of a canonical k-mer, in the fixed 0=A/1=C/2=G/3=T /// Central-position base of a canonical k-mer, in the fixed 0=A/1=C/2=G/3=T
/// encoding — the mask's bit index. `k` must be odd (project invariant). /// encoding — the mask's bit index. `k` must be odd (project invariant).
@@ -43,6 +43,6 @@ pub(super) fn triangle_index(n: usize, i: usize, j: usize) -> usize {
/// for every annex that counts carriers (family overlap, pair weight, /// for every annex that counts carriers (family overlap, pair weight,
/// quartet), so they can never disagree. /// quartet), so they can never disagree.
#[inline] #[inline]
pub(crate) fn is_carrier(mask: u8, no_ambiguity: bool) -> bool { pub(crate) fn is_carrier(mask: u8, policy: AmbiguityPolicy) -> bool {
mask != 0 && !(no_ambiguity && mask.count_ones() > 1) mask != 0 && !policy.drops(mask)
} }
+4 -2
View File
@@ -20,6 +20,7 @@
//! reverse. //! reverse.
pub mod algorithms; pub mod algorithms;
mod ambiguity;
pub mod extensions; pub mod extensions;
mod family_overlap; mod family_overlap;
mod family_scalar_annex; mod family_scalar_annex;
@@ -35,15 +36,16 @@ pub(crate) use family_scalar_annex::ScalarKind;
pub use family_scalar_annex::{FamilyScalarAnnex, FamilyScalarAnnexBuilder}; pub use family_scalar_annex::{FamilyScalarAnnex, FamilyScalarAnnexBuilder};
pub use family_overlap::FamilyOverlap; pub use family_overlap::FamilyOverlap;
pub(crate) use quartet_annex::QUARTET_ANNEX_FILE_NAMES; pub(crate) use quartet_annex::QUARTET_ANNEX_FILE_NAMES;
pub use ambiguity::AmbiguityPolicy;
pub use iter::{ pub use iter::{
MinorantBatchIter, MinorantIter, SiblingBatchIter, SiblingEntry, SiblingIter, SiblingLayerExt, MinorantBatchIter, MinorantIter, SiblingBatchIter, SiblingEntry, SiblingIter, SiblingLayerExt,
}; };
pub use siblingannex::{FamilyMask, SiblingAnnex, SiblingAnnexBuilder}; pub use siblingannex::{FamilyMask, SiblingAnnex, SiblingAnnexBuilder};
pub use algorithms::{ pub use algorithms::{
AlignmentReducer, BasePairTally, CardinalityTally, EntropyBias, GammaShape, LayerReducer, AlignmentReducer, AnnexSelection, BasePairTally, CardinalityTally, EntropyBias, GammaShape, LayerReducer,
LayerReport, PairwiseReducer, RawSnpDistanceOutput, SamplePlan, SampleStats, LayerReport, PairwiseReducer, RawSnpDistanceOutput, SamplePlan, SampleStats,
SampleStatsReducer, SamplingSpec, SankoffBundle, SiblingAnnexStats, SnpAlignment, SampleStatsReducer, SamplingSpec, SankoffBundle, SiblingAnnexStats, SiteLocus, SnpAlignment,
SnpDistanceKind, cardinality_transition_probs, composition_transition_probs, distance_matrix, SnpDistanceKind, cardinality_transition_probs, composition_transition_probs, distance_matrix,
pairwise_cost_matrix, sampled_scan, pairwise_cost_matrix, sampled_scan,
}; };
+24 -16
View File
@@ -12,8 +12,8 @@
//! fewer than four genomes — fits every topology and tells a tree nothing. //! fewer than four genomes — fits every topology and tells a tree nothing.
//! See `algorithms::masking::is_quartet_informative` for the exact rule. //! See `algorithms::masking::is_quartet_informative` for the exact rule.
//! //!
//! Which genomes count as carriers depends on `--no-ambiguity`, so each //! Which genomes count as carriers depends on the ambiguity policy
//! convention has its own file, and **the file name is the parameter**: a //! (`--no-ambiguity`, `--only-diploid`), so each policy has its own file, and **the file name is the parameter**: a
//! `PersistentBitVec` header has no room for a flag, and changing it would //! `PersistentBitVec` header has no room for a flag, and changing it would
//! change a shared structure for the sake of one annex. //! change a shared structure for the sake of one annex.
//! //!
@@ -24,24 +24,31 @@
//! (`algorithms::masking::survives_quartet`) still runs on the resolved //! (`algorithms::masking::survives_quartet`) still runs on the resolved
//! family. //! family.
use crate::siblings::AmbiguityPolicy;
pub(crate) const QUARTET_ANNEX_FILE_NAME: &str = "quartet.pbiv"; pub(crate) const QUARTET_ANNEX_FILE_NAME: &str = "quartet.pbiv";
/// Same annex, built with `no_ambiguity == true`. /// Same annex, built under [`AmbiguityPolicy::Forbid`].
pub(crate) const QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME: &str = "quartet.noamb.pbiv"; pub(crate) const QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME: &str = "quartet.noamb.pbiv";
/// Same annex, built under [`AmbiguityPolicy::OnlyDiploid`].
pub(crate) const QUARTET_ANNEX_ONLY_DIPLOID_FILE_NAME: &str = "quartet.diploid.pbiv";
/// Every quartet-annex file name, whatever its counting convention — for /// Every quartet-annex file name, whatever its counting convention — for
/// invalidating them all together when the sibling annex they were numbered /// invalidating them all together when the sibling annex they were numbered
/// against is rebuilt. /// against is rebuilt.
pub(crate) const QUARTET_ANNEX_FILE_NAMES: [&str; 2] = pub(crate) const QUARTET_ANNEX_FILE_NAMES: [&str; 3] = [
[QUARTET_ANNEX_FILE_NAME, QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME]; QUARTET_ANNEX_FILE_NAME,
QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME,
QUARTET_ANNEX_ONLY_DIPLOID_FILE_NAME,
];
/// File name of the quartet annex built under the given `--no-ambiguity` /// File name of the quartet annex built under the given ambiguity policy.
/// convention. pub(crate) fn quartet_annex_file_name(policy: AmbiguityPolicy) -> &'static str {
pub(crate) fn quartet_annex_file_name(no_ambiguity: bool) -> &'static str { match policy {
if no_ambiguity { AmbiguityPolicy::Keep => QUARTET_ANNEX_FILE_NAME,
QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME AmbiguityPolicy::Forbid => QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME,
} else { AmbiguityPolicy::OnlyDiploid => QUARTET_ANNEX_ONLY_DIPLOID_FILE_NAME,
QUARTET_ANNEX_FILE_NAME
} }
} }
@@ -50,10 +57,11 @@ mod tests {
use super::*; use super::*;
#[test] #[test]
fn the_two_conventions_never_share_a_file() { fn the_policies_never_share_a_file() {
assert_ne!(quartet_annex_file_name(false), quartet_annex_file_name(true)); let names = AmbiguityPolicy::ALL.map(quartet_annex_file_name);
for no_ambiguity in [false, true] { assert!(names[0] != names[1] && names[1] != names[2] && names[0] != names[2]);
assert!(QUARTET_ANNEX_FILE_NAMES.contains(&quartet_annex_file_name(no_ambiguity))); for name in names {
assert!(QUARTET_ANNEX_FILE_NAMES.contains(&name));
} }
} }
} }
@@ -269,6 +269,21 @@ impl SiblingAnnex {
Ok(Self { mmap, n, path: path.to_path_buf() }) Ok(Self { mmap, n, path: path.to_path_buf() })
} }
/// The annex of the layer directory `layer_dir`. Same as
/// [`open`](Self::open), except that an absent file is reported with its
/// path and the way to build it, rather than a bare `NotFound` — the
/// state an index is in right after `migrate-mphf` set its annexes aside.
pub fn open_in(layer_dir: &Path) -> io::Result<Self> {
let path = layer_dir.join(super::ANNEX_FILE_NAME);
Self::open(&path).map_err(|e| match e.kind() {
io::ErrorKind::NotFound => io::Error::new(
io::ErrorKind::NotFound,
format!("no sibling annex at {} — build it with `phylo --sibling-annex`", path.display()),
),
_ => e,
})
}
pub fn path(&self) -> &Path { &self.path } pub fn path(&self) -> &Path { &self.path }
pub fn len(&self) -> usize { self.n } pub fn len(&self) -> usize { self.n }
pub fn is_empty(&self) -> bool { self.n == 0 } pub fn is_empty(&self) -> bool { self.n == 0 }
+22
View File
@@ -362,6 +362,28 @@ impl<L: KmerLength> CanonicalKmerOf<L> {
] ]
} }
/// Stable identifier of this k-mer's central-base family: the centre
/// forced to `A` on the k-mer and on its reverse complement, each
/// canonicalised, then the lexicographic minimum of the two. Identical
/// for the 4 members of a family whatever the orientation they are seen
/// in, and independent of any index. Well-defined for odd `L::len()`.
pub fn family_id(&self) -> Self {
let m = (L::len() - 1) / 2;
let shift = KMER_BITS - 2 - 2 * m;
let clear = |raw: RawKmer| raw & !((0b11 as RawKmer) << shift);
let fwd = KmerOf::<L>(clear(self.0), PhantomData).canonical();
let rev = KmerOf::<L>(clear(self.revcomp().raw()), PhantomData).canonical();
if fwd.0 <= rev.0 { fwd } else { rev }
}
/// Writes [`family_id`](Self::family_id) as ASCII with the centre base
/// replaced by `N` (the family's textual locus label).
pub fn write_family_id_ascii<W: Write>(&self, writer: &mut W) -> io::Result<()> {
let mut ascii = self.family_id().to_ascii();
ascii[(L::len() - 1) / 2] = b'N';
writer.write_all(&ascii)
}
/// Return the inner value as a raw [`KmerOf<L>`]. /// Return the inner value as a raw [`KmerOf<L>`].
#[inline] #[inline]
pub fn into_kmer(self) -> KmerOf<L> { pub fn into_kmer(self) -> KmerOf<L> {
+48
View File
@@ -252,4 +252,52 @@ mod tests {
check!(5); check!(5);
check!(31); check!(31);
} }
// ── family_id ───────────────────────────────────────────────────────────
/// Exhaustive: `family_id` is shared by every member of a family (and by
/// each member's reverse complement) and separates distinct families —
/// the latter checked against `central_canonical_neighbors` as ground truth.
#[test]
fn family_id_partitions_kmers_into_families() {
macro_rules! check {
($n:expr) => {{
type K = KmerOf<ConstLen<$n>>;
let mut families: std::collections::HashMap<_, Vec<_>> = Default::default();
for code in 0..4usize.pow($n) {
let ascii: Vec<u8> = (0..$n).map(|i| b"ACGT"[(code >> (2 * i)) & 3]).collect();
let ck = K::from_ascii(&ascii).unwrap().canonical();
let id = ck.family_id();
assert_eq!(id, ck.revcomp().canonical().family_id());
let mut members: Vec<_> =
ck.central_canonical_neighbors().iter().map(|n| n.raw()).collect();
members.sort_unstable();
for n in ck.central_canonical_neighbors() {
assert_eq!(n.family_id(), id, "k={}: members disagree", $n);
}
assert_eq!(id, id.family_id(), "k={}: family_id not idempotent", $n);
families.entry(id).or_default().push(members);
}
// One id <=> one family: all members seen under an id share one member set.
for sets in families.values() {
assert!(sets.windows(2).all(|w| w[0] == w[1]), "k={}: id merges families", $n);
}
}};
}
check!(3);
check!(5);
check!(7);
}
#[test]
fn family_id_ascii_masks_centre() {
let ck = KmerOf::<ConstLen<5>>::from_ascii(b"ACGTA").unwrap().canonical();
let mut out = Vec::new();
ck.write_family_id_ascii(&mut out).unwrap();
assert_eq!(out.len(), 5);
assert_eq!(out[2], b'N');
// flanks of the id are the family's two flank pairs, in either orientation
let id_ascii = ck.family_id().to_ascii();
assert_eq!([out[0], out[1], out[3], out[4]], [id_ascii[0], id_ascii[1], id_ascii[3], id_ascii[4]]);
}
} }
+5 -1
View File
@@ -3,6 +3,9 @@
mod budget; mod budget;
mod lock; mod lock;
mod lock_slot;
#[cfg(test)]
mod lock_tests;
mod numa; mod numa;
mod profiler; mod profiler;
mod progress; mod progress;
@@ -10,7 +13,8 @@ mod resources;
mod stage; mod stage;
pub use budget::MemoryBudget; pub use budget::MemoryBudget;
pub use lock::DirLock; pub use lock::{DirLock, LockMode};
pub use lock_slot::LockSlot;
pub use numa::PartitionRunner; pub use numa::PartitionRunner;
pub use profiler::start as start_profiler; pub use profiler::start as start_profiler;
pub use progress::{Progress, TracedBar, progress_bar, spinner}; pub use progress::{Progress, TracedBar, progress_bar, spinner};
+188 -38
View File
@@ -1,56 +1,206 @@
use std::fs::File;
use std::io;
use std::path::{Path, PathBuf};
use std::time::Duration;
use tracing::info; use tracing::info;
/// Exclusive advisory lock on an index directory, held for the duration of /// Advisory readers/writer lock on an index directory, shared between
/// any command that writes into an already-existing index (building the /// processes (and between independent handles of one process).
/// sibling annex, merging into a destination, filtering/selecting in place,
/// ...). Two such commands racing on the same directory can otherwise
/// corrupt each other's writes with no error from either side.
/// ///
/// Only the directory actually being *written to* needs a lock — a command /// - **Shared** ([`acquire_shared`](DirLock::acquire_shared)): held by
/// like `merge` that reads several source indexes to build one destination /// anything that reads the index (`query`, `dump`, a `merge` source, a
/// only needs to lock the destination. /// `phylo` run, ...). Any number of shared holders coexist.
/// - **Exclusive** ([`acquire`](DirLock::acquire)): held by anything that
/// writes into an already-existing index (building annexes, `convert`,
/// `pack`, `annotate`, ...). It waits for every shared holder to leave and
/// keeps others out while held. Two writers racing on one directory can
/// otherwise corrupt each other's files with no error from either side,
/// and a reader would see files vanish under it.
/// ///
/// Uses the OS's advisory file lock (`flock` on Unix, `LockFileEx` on /// Only the directory actually being *written to* needs an exclusive lock —
/// Windows) via `std::fs::File::lock`/`try_lock`, not a hand-rolled PID /// `merge` holds shared locks on its sources and an exclusive one on its
/// file: the OS releases it automatically on process exit, including a /// destination.
/// crash — no stale-lock cleanup logic needed. ///
/// # How it is built
///
/// Each mode is one OS advisory file lock (`flock` on Unix, `LockFileEx` on
/// Windows) on `.obikmer.lock`, so a crash releases it with no stale-lock
/// cleanup. On top of that, **every transition** (acquire, release into
/// another mode, promote, demote) happens while holding a *gate* — a second
/// lock file, `.obikmer.gate`, taken for the length of one non-blocking
/// attempt and never while waiting. A blocked request therefore releases the
/// gate, sleeps, and tries again (polling, 20 ms growing to 500 ms), instead
/// of sleeping in the kernel while holding it. Because nobody else can touch
/// the lock state while the gate is held, a shared → exclusive
/// [promotion](DirLock::promote) cannot be overtaken by another writer, which
/// the OS's own `flock` conversion does not guarantee.
///
/// There is no writer preference: a steady stream of readers can delay an
/// exclusive request.
#[derive(Debug)] #[derive(Debug)]
pub struct DirLock { pub struct DirLock {
_file: std::fs::File, file: File,
dir: PathBuf,
mode: LockMode,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LockMode {
Shared,
Exclusive,
}
const LOCK_FILE: &str = ".obikmer.lock";
const GATE_FILE: &str = ".obikmer.gate";
const CLAIM_FILE: &str = ".obikmer.promote";
const POLL_MIN: Duration = Duration::from_millis(20);
const POLL_MAX: Duration = Duration::from_millis(500);
fn open_file(dir: &Path, name: &str) -> io::Result<File> {
std::fs::create_dir_all(dir)?;
std::fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(dir.join(name))
}
/// The gate: blocks, but its holders only ever run instantaneous,
/// non-blocking operations, so the wait is negligible. Released on drop.
fn gate(dir: &Path) -> io::Result<File> {
let f = open_file(dir, GATE_FILE)?;
f.lock()?;
Ok(f)
}
/// Non-blocking attempt in `mode`; `Ok(false)` means "held by someone else".
fn try_lock(file: &File, mode: LockMode) -> io::Result<bool> {
let r = match mode {
LockMode::Exclusive => file.try_lock(),
LockMode::Shared => file.try_lock_shared(),
};
match r {
Ok(()) => Ok(true),
Err(std::fs::TryLockError::WouldBlock) => Ok(false),
Err(std::fs::TryLockError::Error(e)) => Err(e),
}
}
/// Sleeps between two attempts and logs once, at the first one.
struct Backoff {
delay: Duration,
logged: bool,
}
impl Backoff {
fn new() -> Self {
Self { delay: POLL_MIN, logged: false }
}
fn wait(&mut self, dir: &Path, why: &str) {
if !self.logged {
info!(dir = %dir.display(), "{why}");
self.logged = true;
}
std::thread::sleep(self.delay);
self.delay = (self.delay * 2).min(POLL_MAX);
}
} }
impl DirLock { impl DirLock {
/// Block until the exclusive lock on `dir` is acquired (creating `dir` /// Block until the exclusive lock on `dir` is acquired (creating `dir`
/// and the lock file within it if needed). Logs once if the wait is /// and the lock files within it if needed). Logs once if the wait is
/// non-trivial, so a blocked command doesn't look silently hung. /// non-trivial, so a blocked command doesn't look silently hung.
pub fn acquire(dir: &std::path::Path) -> std::io::Result<Self> { pub fn acquire(dir: &Path) -> io::Result<Self> {
let file = Self::open_lock_file(dir)?; Self::acquire_mode(dir, LockMode::Exclusive, "waiting for other obikmer processes to release this index")
if file.try_lock().is_err() {
info!(dir = %dir.display(), "waiting for another obikmer process to release this index");
file.lock()?;
}
Ok(Self { _file: file })
} }
/// Non-blocking probe: `Ok(Some(lock))` if `dir`'s lock was free and is /// Block until a shared lock on `dir` is acquired — until no exclusive
/// now held by the returned `DirLock`, `Ok(None)` if another process /// holder remains. Logs once if the wait is non-trivial.
/// currently holds it (not an error — contention is the expected, pub fn acquire_shared(dir: &Path) -> io::Result<Self> {
/// common case this exists to distinguish from a genuine I/O failure). Self::acquire_mode(dir, LockMode::Shared, "waiting for an obikmer process writing this index")
pub fn try_acquire(dir: &std::path::Path) -> std::io::Result<Option<Self>> { }
let file = Self::open_lock_file(dir)?;
match file.try_lock() { fn acquire_mode(dir: &Path, mode: LockMode, why: &str) -> io::Result<Self> {
Ok(()) => Ok(Some(Self { _file: file })), let file = open_file(dir, LOCK_FILE)?;
Err(_) => Ok(None), let mut backoff = Backoff::new();
loop {
{
let _gate = gate(dir)?;
if try_lock(&file, mode)? {
return Ok(Self { file, dir: dir.to_owned(), mode });
}
}
backoff.wait(dir, why);
} }
} }
fn open_lock_file(dir: &std::path::Path) -> std::io::Result<std::fs::File> { pub fn mode(&self) -> LockMode {
std::fs::create_dir_all(dir)?; self.mode
let lock_path = dir.join(".obikmer.lock"); }
std::fs::OpenOptions::new()
.create(true) /// Non-blocking probe: `Ok(true)` if no exclusive holder exists right
.truncate(false) /// now (a shared lock could be taken), `Ok(false)` if one does. Retains
.write(true) /// nothing either way.
.open(&lock_path) pub fn is_free_of_writer(dir: &Path) -> io::Result<bool> {
let file = open_file(dir, LOCK_FILE)?;
let _gate = gate(dir)?;
try_lock(&file, LockMode::Shared)
}
/// Shared → exclusive. Returns the exclusive lock and whether the
/// promotion was **gapless**: `true` means the lock was never released,
/// so nothing in the directory can have changed since the shared lock
/// was held and whatever the caller read stays valid; `false` means the
/// shared lock had to be given up while waiting, so a writer may have
/// run in between and the caller must re-check what it relied on.
///
/// Gapless promotion is available to one promoter at a time (it holds
/// `.obikmer.promote` while it waits for the other readers to leave).
/// A second concurrent promoter releases its shared lock instead — two
/// promoters each keeping theirs while waiting for the other's would
/// wait for ever. An exclusive lock is returned unchanged.
pub fn promote(self) -> io::Result<(Self, bool)> {
if self.mode == LockMode::Exclusive {
return Ok((self, true));
}
let claim = open_file(&self.dir, CLAIM_FILE)?;
if !try_lock(&claim, LockMode::Exclusive)? {
let dir = self.dir.clone();
drop(self); // gives the shared lock up
return Ok((Self::acquire(&dir)?, false));
}
let mut backoff = Backoff::new();
loop {
{
let _gate = gate(&self.dir)?;
self.file.unlock()?;
if try_lock(&self.file, LockMode::Exclusive)? {
return Ok((Self { mode: LockMode::Exclusive, ..self }, true));
}
// Other readers remain. Nobody can hold the lock exclusively
// without the gate, so taking our shared lock back succeeds.
if !try_lock(&self.file, LockMode::Shared)? {
return Err(io::Error::other("lost the shared index lock during promotion"));
}
}
backoff.wait(&self.dir, "waiting for other readers of this index before writing to it");
}
}
/// Exclusive → shared, without ever leaving the lock free in between.
/// A shared lock is returned unchanged.
pub fn demote(self) -> io::Result<Self> {
if self.mode == LockMode::Shared {
return Ok(self);
}
let held_gate = gate(&self.dir)?;
self.file.unlock()?;
if !try_lock(&self.file, LockMode::Shared)? {
return Err(io::Error::other("could not take the shared index lock back after writing"));
}
drop(held_gate);
Ok(Self { mode: LockMode::Shared, ..self })
} }
} }
+96
View File
@@ -0,0 +1,96 @@
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use crate::lock::{DirLock, LockMode};
/// The directory lock of one handle on an index: nothing, shared or
/// exclusive, and the transitions between them that a handle needs. What
/// `obikindex::KmerIndex` delegates its locking to, kept apart so that it can
/// be tested without building an index.
///
/// Interior mutability (`&self` methods): a handle is shared behind `Arc`.
/// Not for concurrent use from several threads of one handle — the first
/// section to finish would restore the lock under the others.
#[derive(Debug)]
pub struct LockSlot {
dir: PathBuf,
held: Mutex<Option<DirLock>>,
}
impl LockSlot {
/// A slot holding nothing (a handle that took no lock, or one that
/// reopens a directory its own builder already holds).
pub fn empty(dir: &Path) -> Self {
Self { dir: dir.to_owned(), held: Mutex::new(None) }
}
/// A slot holding a shared lock on `dir`; blocks while a writer is at work.
pub fn shared(dir: &Path) -> io::Result<Self> {
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire_shared(dir)?)) })
}
/// A slot holding the exclusive lock on `dir`; blocks while anyone else holds it.
pub fn exclusive(dir: &Path) -> io::Result<Self> {
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire(dir)?)) })
}
/// Whether the slot holds the exclusive lock (a shared one is not write access).
pub fn holds_write_lock(&self) -> bool {
matches!(&*self.held.lock().unwrap(), Some(l) if l.mode() == LockMode::Exclusive)
}
/// Hold the exclusive lock from now on: a no-op if already held that way,
/// a promotion of a shared lock, a plain acquisition of an empty slot.
pub fn acquire_exclusive(&self) -> io::Result<()> {
let held = self.held.lock().unwrap().take();
let lock = match held {
Some(l) => l.promote()?.0,
None => DirLock::acquire(&self.dir)?,
};
*self.held.lock().unwrap() = Some(lock);
Ok(())
}
/// Run `f` with the exclusive lock held:
/// - already exclusive: `f` runs as is (so sections nest);
/// - shared: promoted for the duration of `f`, then demoted back, so the
/// slot keeps the shared lock it came with. The promotion is gapless
/// when no other process promotes at the same time; otherwise other
/// writers may run between the shared lock being given up and the
/// exclusive one being taken. Either way **`f` must re-check, before
/// building, what it is about to build**, and the caller must not hold
/// anything read earlier that a writer could have replaced;
/// - empty: the lock is taken for `f` only.
///
/// Restored also if `f` fails; an I/O error taking or restoring the lock
/// is converted to `E`.
pub fn with_exclusive<T, E: From<io::Error>>(&self, f: impl FnOnce() -> Result<T, E>) -> Result<T, E> {
enum Before {
Exclusive,
Shared,
Nothing,
}
let held = self.held.lock().unwrap().take();
let (lock, before) = match held {
Some(l) if l.mode() == LockMode::Exclusive => (l, Before::Exclusive),
Some(l) => (l.promote()?.0, Before::Shared),
None => (DirLock::acquire(&self.dir)?, Before::Nothing),
};
*self.held.lock().unwrap() = Some(lock);
let out = f();
let lock = self.held.lock().unwrap().take();
match (before, lock) {
(Before::Exclusive, lock) => *self.held.lock().unwrap() = lock,
(Before::Shared, Some(l)) => *self.held.lock().unwrap() = Some(l.demote()?),
_ => {} // taken for `f` only: released here
}
out
}
/// Non-blocking probe: `true` if no process holds the directory
/// exclusively right now (this slot included), `false` if one does.
pub fn is_free_of_writer(&self) -> bool {
DirLock::is_free_of_writer(&self.dir).unwrap_or(false)
}
}
+208
View File
@@ -0,0 +1,208 @@
//! The directory lock is a readers/writer lock: `shared` slots coexist, an
//! `exclusive` one waits for them all and keeps new ones out, and a shared
//! slot can be promoted for a section (`with_exclusive`) and demoted back.
//! No index is involved, and therefore none of the process-wide k-mer
//! parameters other crates' tests share.
use std::io;
use std::path::PathBuf;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::mpsc;
use std::sync::{Arc, Barrier};
use std::thread;
use std::time::Duration;
use crate::lock::{DirLock, LockMode};
use crate::lock_slot::LockSlot;
const SETTLE: Duration = Duration::from_millis(400);
const PATIENCE: Duration = Duration::from_secs(10);
/// A scratch directory removed on drop (no `tempfile` dependency here).
struct Scratch(PathBuf);
impl Scratch {
fn new() -> Self {
static N: AtomicUsize = AtomicUsize::new(0);
let dir = std::env::temp_dir().join(format!(
"obisys-lock-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&dir).unwrap();
Self(dir)
}
}
impl Drop for Scratch {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
#[test]
fn readers_coexist() {
let d = Scratch::new();
let a = LockSlot::shared(&d.0).unwrap();
let b = LockSlot::shared(&d.0).unwrap();
assert!(!a.holds_write_lock() && !b.holds_write_lock());
assert!(a.is_free_of_writer(), "readers are not writers");
}
#[test]
fn exclusive_is_exclusive_and_released_on_drop() {
let d = Scratch::new();
let w = LockSlot::exclusive(&d.0).unwrap();
assert!(w.holds_write_lock());
assert!(!w.is_free_of_writer());
drop(w);
assert!(LockSlot::empty(&d.0).is_free_of_writer());
}
#[test]
fn a_writer_waits_for_the_readers() {
let d = Scratch::new();
let reader = LockSlot::shared(&d.0).unwrap();
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let writer = thread::spawn(move || {
drop(LockSlot::exclusive(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the writer must wait for the reader");
drop(reader);
rx.recv_timeout(PATIENCE).expect("the writer proceeds once the reader is gone");
writer.join().unwrap();
}
#[test]
fn a_reader_waits_for_the_writer() {
let d = Scratch::new();
let writer = LockSlot::exclusive(&d.0).unwrap();
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let reader = thread::spawn(move || {
drop(LockSlot::shared(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the reader must wait for the writer");
drop(writer);
rx.recv_timeout(PATIENCE).expect("the reader proceeds once the writer is gone");
reader.join().unwrap();
}
#[test]
fn with_exclusive_on_a_reader_promotes_then_restores_the_shared_lock() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
let out: io::Result<i32> = slot.with_exclusive(|| {
assert!(slot.holds_write_lock());
assert!(!slot.is_free_of_writer());
Ok(7)
});
assert_eq!(out.unwrap(), 7);
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
// Still a reader: a writer elsewhere waits for it.
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let writer = thread::spawn(move || {
drop(LockSlot::exclusive(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the slot must be a reader again");
drop(slot);
rx.recv_timeout(PATIENCE).unwrap();
writer.join().unwrap();
}
#[test]
fn with_exclusive_on_an_empty_slot_holds_the_lock_for_the_section_only() {
let d = Scratch::new();
let slot = LockSlot::empty(&d.0);
slot.with_exclusive::<(), io::Error>(|| {
assert!(slot.holds_write_lock());
Ok(())
})
.unwrap();
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
}
#[test]
fn with_exclusive_restores_on_error() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
let r: io::Result<()> = slot.with_exclusive(|| Err(io::Error::other("boom")));
assert!(r.is_err());
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
}
#[test]
fn with_exclusive_nests_and_keeps_a_lock_taken_at_open() {
let d = Scratch::new();
let slot = LockSlot::exclusive(&d.0).unwrap();
slot.with_exclusive::<(), io::Error>(|| {
slot.with_exclusive::<(), io::Error>(|| Ok(()))?;
assert!(slot.holds_write_lock(), "an inner section must not release the outer lock");
Ok(())
})
.unwrap();
assert!(slot.holds_write_lock(), "a slot opened exclusive keeps it");
}
#[test]
fn acquire_exclusive_promotes_a_shared_slot_and_is_idempotent() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
slot.acquire_exclusive().unwrap();
slot.acquire_exclusive().unwrap();
assert!(slot.holds_write_lock());
}
/// The case a naive upgrade deadlocks on: two readers, each holding a shared
/// lock, both ask for the exclusive one at the same time.
#[test]
fn two_readers_promoting_at_once_both_finish() {
let d = Scratch::new();
let barrier = Arc::new(Barrier::new(2));
let (tx, rx) = mpsc::channel();
let handles: Vec<_> = (0..2)
.map(|_| {
let (dir, b, tx) = (d.0.clone(), Arc::clone(&barrier), tx.clone());
thread::spawn(move || {
let slot = LockSlot::shared(&dir).unwrap();
b.wait(); // both hold a shared lock
slot.with_exclusive::<(), io::Error>(|| {
assert!(slot.holds_write_lock());
thread::sleep(Duration::from_millis(50));
Ok(())
})
.unwrap();
tx.send(()).unwrap();
})
})
.collect();
for _ in 0..2 {
rx.recv_timeout(PATIENCE).expect("a promotion deadlocked");
}
for h in handles {
h.join().unwrap();
}
}
#[test]
fn a_lone_promotion_is_gapless_and_a_demotion_keeps_the_lock() {
let d = Scratch::new();
let shared = DirLock::acquire_shared(&d.0).unwrap();
let (excl, gapless) = shared.promote().unwrap();
assert_eq!(excl.mode(), LockMode::Exclusive);
assert!(gapless, "nobody else was promoting: the lock was never released");
let shared = excl.demote().unwrap();
assert_eq!(shared.mode(), LockMode::Shared);
// Shared again: other readers get in, a writer does not.
let _other = DirLock::acquire_shared(&d.0).unwrap();
assert!(DirLock::is_free_of_writer(&d.0).unwrap());
}