Compare commits

..
40 Commits
Author SHA1 Message Date
Eric Coissac 5db7b440da refactor: treat absence as state unless --free-loss and update quartet annexes
ci.yml / build (pull_request) Successful in 4m33s
Release / create-release (push) Successful in 1m56s
Release / build-linux-x86_64 (push) Successful in 11m18s
Release / build-macos-arm64 (push) Successful in 2m27s
Refactor the handling of genome absence to be a distinct state unless `--free-loss` is specified. This change introduces `QuartetConvention` to pair ambiguity policies with the free-loss flag, updating file naming conventions and logic across sampling, masking, and stats modules.

- Update `--exclude-genome` description in `pair_coverage_sampling.md` to clarify absence as a state unless `--free-loss` is specified.
- Move multi-member genome handling convention from `pair_coverage_sampling.md` to file name conventions.
- Clarify persisted annex structure in `pair_coverage_sampling.md` as `PersistentBitVec` per non-monomorphic family with counting convention carried by file name.
- Fix initial version's absence handling in `has_bipartition`, the annex, and `--sample-stats` to correctly count index 0 when absence is a state.
- Replace entropy-only builder with `AnnexRequest` and `ensure_layer_family_annexes`.
- Consolidate sampling parameters into a single `SamplingSpec`.
- Update family constraint description in `sampling_design.md`.
- Modify uniform draws and `--quartet` removal text in `sampling_design.md`.
- Clarify state counting in tests for `--free-loss` option in `sampling_design.md`.
- State that absence forms a pair without `--free-loss` but never with it in `sampling_design.md`.
- Update `--quartet` option to require at least two distinct states, each carried by at least two genomes.
- Define absence as a state unless `--free-loss` is given for family carrier definition.
- Clarify `--pair-balance` favors families covering pairs with fewest shared families.
- Modify logic for families fitting every tree topology equally well in `phylo.md`.
- Change `--quartet` restriction logic to apply after pre-checks involving `--no-ambiguity` and `--exclude-genome`.
- Update `obikmer` dependency version from 1.4.4 to 1.4.5 in `Cargo.lock` and `Cargo.toml`.
- Modify `annex_selection` in `args/mod.rs` to derive `free_loss`, `entropy`, `quartet`, and `overlap` from `SamplingSpec`.
- Update `SamplingArgs::quartet` documentation in `sampling.rs` to describe state determination based on `--free-loss`.
- Add `free_loss` field to `AnnexSelection` struct in `family_annexes.rs`.
- Change `AnnexRequest` to use `Vec<QuartetConvention>` instead of `Vec<AmbiguityPolicy>`.
- Update `request` logic in `family_annexes.rs` to conditionally create `QuartetConvention` based on `self.quartet`.
- Modify `dedup` signature in `family_annexes.rs` to accept iterator of type `T: Ord`.
- Update `ensure_layer_family_annexes` in `family_annexes.rs` to use `missing_files` and `QuartetConvention`.
- Add `missing_files` definition to `family_annexes.rs`.
- Change `has_bipartition` in `masking.rs` to filter counts where count is `>= 2`.
- Update `is_quartet_informative` in `masking.rs` to accept `free_loss` boolean parameter.
- Modify `two_pairs_of_distinct_states_is_informative` and `singleton_variant_is_not_informative` in `masking.rs` to test both `free_loss` policies.
- Add `under_free_loss_absence_is_missing_and_forms_no_pair` and `without_free_loss_absence_is_a_state_that_can_form_a_pair` definitions to `masking.rs`.
- Update `no_ambiguity_drops_multi_member_genomes_from_the_count` in `masking.rs` to check `is_quartet_informative` under `free_loss=true`.
- Add `the_annex_rule_and_the_exact_check_agree_on_the_same_family` definition to `masking.rs`.
- Remove `absent_genomes_are_not_a_state` definition from `masking.rs`.
- Update `restrict_to_quartet` in `quartet.rs` to accept `QuartetConvention` instead of `AmbiquityPolicy`.
- Modify `add_site` in `sample_stats.rs` to handle state 0 as a state participating in bipartition.
- Add `absence_as_a_state_makes_a_site_informative` definition to `sample_stats.rs`.
- Add `quartet_convention` function to `SamplingSpec` in `subsample.rs`.
- Update `sample_index` in `subsample.rs` to use `spec.quartet_convention()` for `AnnexRequest`.
- Update `restrict_to_quartet` call in `subsample.rs` to use `spec.quartet_convention()`.
- Add comment regarding `--no-ambiguity` counting convention in `helpers.rs`.
- Import `QUARTET_ANNEX_FILE_NAMES` and `QuartetConvention` from `quartet_annex` in `mod.rs`.
- Add `QuartetConvention` struct with `policy` and `free_loss` fields to `quartet_annex.rs`.
- Modify `quartet_annex_file_name` in `quartet_annex.rs` to take `QuartetConvention`.
- Add `the_conventions_never_share_a_file` function to `quartet_annex.rs`.
- Remove `the_policies_never_share_a_file` definition from `quartet_annex.rs`.
2026-11-19 07:12:36 +01:00
Eric Coissac c93616b432 feat: add threshold-based Hamming distance and refine phylo documentation
Add `threshold_hamming_dist_matrix` to calculate k-mers present in exactly one genome using union minus intersection. Update `DistanceArgs` and `distance.rs` to support this metric on count indexes via `--presence-threshold`, removing the previous restriction that hamming was only for presence/absence indexes.

- Add `threshold_hamming_dist_matrix` method to `CountPartials` in `src/obikmer/src/traits.rs`
- Implement new test `threshold_hamming_counts_kmers_present_in_one_genome_only` in `src/obicompactvec/src/tests/intmatrix.rs`
- Update `DistanceArgs` in `src/obikmer/src/cmd/phylo/args/distance.rs` to clarify threshold effects on Jaccard/Mash/Hamming
- Modify `distance` function in `src/obikphylo/src/distance.rs` to support Hamming on count indexes and use pre-computed bit structures
- Update `obicompactvec.md` documentation with new function details and partial result explanations
- Refine `index.md` descriptions for `phylo`, `unitig`, and `select` options
2026-11-19 07:11:57 +01:00
Eric Coissac c4eacb3f72 feat(phylo): expand genome distance metrics and refactor CLI arguments
Refactor the `phylo` command to support explicit distance metrics (Jaccard, Hamming) and SNP-based pseudo-alignments. Replace implicit metric selection with explicit requirements, such as requiring `--n-sites` for SNP analysis. Add a new `shared-kmers` output option and update documentation for performance characteristics of sparse vs dense formats.

- Update `phylo` CLI to support genome-vs-genome distances using k-mer content or SNPs, including options for NJ/UPGMA tree building and data export
- Replace `reindex` command with `convert` for in-place index evidence conversion (exact/approx/hybrid) and add `--upgrade-index` to `utils` for missing layer metadata
- Refactor `distance_to_compute` logic to remove implicit Jaccard implications for `--nj`/`--upgma` unless `--distance` is specified, and enforce `--n-sites` for `--pseudo-alignment`/`--sankoff`
- Add `shared_kmers` method to `Metrics` trait in `src/obikphylo/src/distance.rs` and new `write_shared_kmers` function in `src/obikmer/src/cmd/phylo/shared.rs` for CSV export
- Update `PackArgs::dense` documentation to clarify that dense format is faster for column-oriented distance matrices while sparse is better for single-row access
- Remove `requires_counts` definition and update assertions in `values.rs` to reflect new distance computation signatures and metric requirements
2026-11-19 07:11:44 +01:00
Eric Coissac 14433e80a5 refactor: restructure phylo CLI arguments into submodules
Refactor the `phylo` command's argument handling by splitting the monolithic
`args.rs` file into specialized submodules (`distance`, `outputs`, `sampling`,
`select`) and a central `mod.rs`. This improves modularity and separates
concerns for distance metrics, sampling options, and output configurations.

- Move `alignment_output`, `ambiguity`, and `annex_selection` from `args.rs` to
  `args/mod.rs` in the `PhyloArgs` struct.
- Relocate distance parsing logic (`parse_gamma_shape`, `parse_n_sites`, `target`)
  and helper functions (`distance_to_compute`, `as_classic`, `as_snp`, etc.) from
  `args.rs` to `values.rs`.
- Add new submodule files: `args/distance.rs` for `DistanceArgs`, `args/outputs.rs`
  for `AlignArgs` and `CostsArgs`, `args/sampling.rs` for `SamplingArgs`, and
  `args/select.rs` for `SelectArgs` and `AnnexArgs`.
- Update `main.rs` to register the new `Phylo` command using `cmd::phylo::PhyloArgs`.
- Refactor `mod.rs` (`run`) to use nested argument structures (e.g., `args.dist`,
  `args.sample`) instead of flat fields.
- Update `alignment_out.rs` to reference the new nested paths (`args.align.fasta`,
  `args.sample.no_ambiguity`).
2026-11-19 07:11:26 +01:00
Eric Coissac 7f829ea557 docs: clarify merge re-merge behavior and input file handling
Update documentation to explain that merged indices can serve as sources if at least one source is unmerged, and that the resulting column order follows the base index first. Clarify that explicitly named files are always read regardless of extension, with format detection based on content.

- refactor: prevent I/O errors when re-merging packed outputs by restricting `choose_base` to valid destinations via new `can_be_destination` check
- fix: handle zero-layer partitions in `build_index_layer` and `remove_build_artifacts` to skip unnecessary operations
- feat: allow `PathIter` to treat unrecognized extensions as explicit files, ensuring they are not silently dropped during directory expansion
- test: add comprehensive tests for `PathIter` including extension recognition, explicit file handling, and stdin sentinel behavior
- test: update smoke tests to verify empty partitions, `.fna.gz` indexing, and refusal of merging only packed indexes
- chore: remove obsolete cargo build environment variables from `run.sh` script
2026-11-19 07:10:58 +01:00
Eric Coissac 07c0a54e04 feat: add compatibility testing scripts and update dependencies
Add `compat_ptrhash.sh` and `compat_ptrhash2.sh` to facilitate determinism control and incremental merge testing between obikmer binaries. Update `tikv-jemallocator` dependency version in `Cargo.toml` and refresh `Cargo.lock` with new crates and checksums.

- Add scripts/docker_debug/compat_ptrhash.sh for compatibility tests (T0-T4) comparing ptr_hash versions using Docker and md5sum
- Add scripts/docker_debug/compat_ptrhash2.sh for determinism control and incremental merge operations between binaries A and B
- Update src/obikmer/Cargo.toml to change tikv-jemallocator version from 0.7.0 to 0.5.4 under musl target cfg
- Update src/Cargo.lock by adding new crates (anes, block-pseudorand, criterion, plotters, etc.) and updating existing package versions/checksums
2026-11-19 07:10:37 +01:00
Eric Coissac af525cd268 feat: switch musl builds to jemalloc allocator for stability
Update Cargo.lock to remove unused dependencies and update tikv-jemallocator. Configure src/obikmer/Cargo.toml to use jemalloc for static musl builds, addressing fragmentation issues with the default multi-threaded allocator. Update main.rs to conditionally set Jemalloc as the global allocator when targeting musl.

- Remove unused dependencies from Cargo.lock: anes, block-pseudorand, chiapos-chacha8, criterion, criterion-plot, itertools (and its either dep), nanorand, hermit-abi, is-terminal, plotters ecosystem, tinytemplate deps, voracious_radix_sort rayon dep, and web-sys deps.
- Update tikv-jemallocator version from 0.5.4 to 0.7.0 in Cargo.lock.
- Add tikv-jemallocator = "0.7.0" to musl-specific dependencies in src/obikmer/Cargo.toml.
- Set Jemalloc as the global allocator in main.rs for static musl builds.
2026-11-19 07:10:20 +01:00
Eric Coissac d0668eb167 feat: add columnar sparse matrix support and windowed transposition
Introduce `sparse_columnar` and `sparse_intmatrix_columnar` modules to enable
building sparse matrices directly from columnar data without dense intermediates.
Add a `transpose` module with `RowBatch` and `transpose_windows` for efficient
windowed column-to-row transposition based on density checks.

- Add `src/obicompactvec/src/bitmatrix/sparse_columnar.rs` with `build_from_columnar`,
  `build_from_columnar_with`, and `read_bit_window` to build sparse bit matrices
  from columnar files using windowed reads.
- Add `src/obicompactvec/src/sparse_intmatrix_columnar.rs` with `build_from_columnar`,
  `build_from_columnar_with`, `invalid`, and `read_int_window` for integer matrix
  construction from PCIV columns, handling sentinel values and overflow.
- Add `src/obicompactvec/src/transpose.rs` containing `RowBatch<V>` struct and
  `transpose_windows` function for dynamic window sizing and memory-bounded
  column-to-row transposition.
- Update `src/obicompactvec/src/bitmatrix/sparse.rs` to use `packed_bit_matrix_n_cols`
  and `MatrixMeta` for validation, ensuring stale columnar data is transposed directly
  into a single `matrix.pbmx` file while removing old column files.
- Update `src/obicompactvec/src/sparse_intmatrix.rs` to use `packed_int_matrix_n_cols`
  for column count validation, making `pack_sparse_compact_int_matrix` idempotent and
  avoiding dense intermediates by transposing columnar data directly.
- Expose `packed_bit_matrix_n_cols` in `src/obicompactvec/src/bitmatrix/packed.rs` and
  make `packed_int_matrix_n_cols` public in `src/obicompactvec/src/intmatrix.rs`.
- Add `sparse_columnar` module to `src/obicompactvec/src/bitmatrix/mod.rs` and export
  `PersistentBitMatrixBuilder` and `batch_presence_counts`.
- Add `sparse_intmatrix_columnar` and `transpose` modules to `src/obicompactvec/src/lib.rs`.
- Add `transpose_columnar` test module in `src/obicompactvec/src/tests/mod.rs` with
  tests verifying columnar transpose matches dense route output.
2026-11-19 07:08:41 +01:00
Eric Coissac 4e98de4b77 feat: add BitSliceView::for_each_set_in and optimize build_from_dense
Introduce a new `views_scan` module containing `BitSliceView::for_each_set_in`, which iterates over set bits in a specified range using optimized word traversal. Update `build_from_dense` to use this method for batched, ordered column access, ensuring sorted row output. Add comprehensive tests for the new iteration logic and boundary handling.

- Add `src/obicompactvec/src/views_scan.rs` with `BitSliceView::for_each_set_in`, which iterates over a `Range<usize>` calling a closure for each set bit while optimizing sparse traversal via `word.trailing_zeros()`.
- Add `src/obicompactvec/src/tests/views_scan.rs` with tests `words`, `naive`, `scanned`, and helpers verifying that `for_each_set_in` matches naive iteration, handles empty/reversed ranges correctly, and rejects out-of-bounds ranges.
- Modify `build_from_dense` in `src/obicompactvec/src/bitmatrix/sparse.rs` to replace `nonzero_triples` with `dense.col_view(c).for_each_set_in`, ensuring column visits are batched and ordered for sorted row processing.
- Update module structure in `src/obicompactvec/src/lib.rs` and `src/obicompactvec/src/tests/mod.rs` to include the new `views_scan` module.
2026-11-19 07:08:07 +01:00
Eric Coissac 03eaa4544a feat: add pack optimization prompt, debug infrastructure, and monitoring scripts
Add a prompt template for optimizing the `pack` step and a suite of Docker-based
debug tools including build, run, and monitoring scripts. Enhance `pack_matrices`
with LFD partition sizing and detailed timing reports, and update `stop` with
additional system statistics.

- Add `prompts/pack_optimization.md` to guide optimization of the `pack` step,
  specifying context, performance measurements, and deliverable structures.
- Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets,
  `build.sh` for image creation, `run.sh` for parallel indexing and merging,
  `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and
  `gen_fasta_real.py` for real genome generation with `read_fna`.
- Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics,
  writing samples to CSV and summary reports.
- Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing
  memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize.
- Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and
  `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition
  sizing and clarify documentation regarding `stat()` proxy bytes.
- Modify `src/obisys/src/stage.rs`: update `stop` definition to include new
  variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and
  logic checking for swap usage.
2026-11-19 07:07:50 +01:00
Eric Coissac 24291a4484 refactor: remove unused dependencies and simplify logic
Remove obsolete dependencies across multiple crates to clean up the build graph and update `obikdump` to use `try_update`. Add dev-dependencies for tracing in `obiread` and clarify memoization logic in `obiksession`.

- Remove `obifastwrite`, `tracing`, `obidebruinj`, `obikentropy`, `obipipeline`, `obidex`, and `obiskbuilder` from `Cargo.lock`.
- Remove `obifastwrite` and `tracing` dependencies from `src/obidebruinj/Cargo.toml`.
- Update `dump` function in `src/obikdump/src/dump.rs` to use `try_update` instead of `fetch_update` for updating remaining counts.
- Remove `obitaxonomy`, `obidebruinj`, `obipipeline`, and `obikentropy` dependencies from `src/obikindex/Cargo.toml`.
- Remove `obiksession` dependency from `src/obikmer/Cargo.toml`.
- Remove `obiskbuilder` dependency from `src/obikphylo/Cargo.toml` and update `kodama` version to "0.3.0".
- Remove `bytes` dependency from `src/obikrope/Cargo.toml` and set `criterion2` version to "3" with `cargo_bench_support`.
- Add `tracing` and `tracing-subscriber` as dev-dependencies in `src/obiread/Cargo.toml`.
- Remove `lazy_static` dependency from `src/obiskbuilder/Cargo.toml` and add `obikseq` as a test-utils dev-dependency.
- Clarify `memo_key_from_debug` in `src/obiksession/src/memoize.rs` with comments explaining Debug-derived field usage.
2026-11-19 07:07:16 +01:00
coissac cbc344e4b4 Merge pull request 'Push txuyrtlmyoyl' (#81) from push-txuyrtlmyoyl into main
Reviewed-on: #81
2026-11-17 14:38:33 +00:00
Eric Coissac 07bc5fc634 docs: update sparse packing strategy and benchmark scripts
ci.yml / build (pull_request) Successful in 4m53s
Release / create-release (push) Successful in 28s
Release / build-linux-x86_64 (push) Successful in 6m46s
Release / build-macos-arm64 (push) Successful in 4m8s
Update documentation to reflect that sparse packing now maps all columns of a layer at once and packs layers sequentially, reversing the previous choice to avoid materializing a packed copy. This change introduces an extra write/read cycle for the dense matrix, increasing pack benchmark time from 1.11s to 1.60s.

- Modify `pack_sparse_bit_matrix` in `src/obicompactvec/src/bitmatrix/sparse.rs` to always convert via the dense packed format, simplifying logic by removing conditional handling of existing files and ensuring a single mapping.
- Update `pack_sparse_compact_int_matrix` in `src/obicompactvec/src/sparse_intmatrix.rs` to exclusively handle conversion from the dense packed format (`matrix.pcmx`) to sparse, removing columnar format handling and simplifying file cleanup.
- Adjust `scripts/bench_merge_oracle.sh` to extract `PACK_WALL` metrics via `stage_field` for merge operations, updating grep patterns and output messages to report pack wall time.
- Revise `DevDocMD/implementation/merge_mmap_audit.md` and `merge_scaling.md` to document the new sequential packing strategy, noting that `N_part` scaling applies only to incremental merges and that `obisys::MemoryBudget` is no longer used by merge.
- Bump package version in `src/obikmer/Cargo.toml` from 1.4.3 to 1.4.4.
2026-11-17 15:35:41 +01:00
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00
coissac b35127e4e3 Merge pull request 'Push smsmsoyssmnr' (#80) from push-smsmsoyssmnr into main
Reviewed-on: #80
2026-11-09 10:01:21 +00:00
Eric Coissac 0134c69cf4 docs: clarify VCF pseudo-alignment format and update obikmer dependency
ci.yml / build (pull_request) Successful in 3m40s
Release / create-release (push) Successful in 1m7s
Release / build-macos-arm64 (push) Successful in 1m52s
Release / build-linux-x86_64 (push) Successful in 6m41s
Update the VCF export documentation to reflect that `-` represents an overlapping deletion (extra allele `*`) and introduce the `<ABS>` symbolic allele for absent families. Add a new dedicated documentation file for the VCF pseudo-alignment format, remove obsolete descriptions from phylo.md, and fix benchmark scripts to handle errors and directory creation correctly.

- Update `obikmer` dependency from 1.4.1 to 1.4.2 in Cargo.toml and Cargo.lock
- Add `UserDocMD/formats/vcf.md` describing VCF pseudo-alignment structure, alleles, and genotypes
- Remove obsolete VCF descriptions from `UserDocMD/usage/phylo.md` and update ploidy/free-loss notes
- Update `DevDocMD/theory/evolutionary_distances.md` to clarify `-` as overlapping deletion and `<ABS>` for others
- Fix `benchmark/index_one_count.sh` to create index directory and check read command errors
- Fix `benchmark/index_one_presence.sh` to create index directory and check simulation errors
- Update `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` to use `<ABS>` for absent families instead of `*`
2026-11-09 20:59:28 +11:00
Eric Coissac 5368c98875 refactor: replace no_ambiguity bool with AmbiguityPolicy enum
Replace the binary `no_ambiguity` flag with an `AmbiguityPolicy` enum (`Keep`, `Forbid`, `OnlyDiploid`) to support multiple ambiguity conventions. This change updates CLI arguments, internal logic, and persistent storage formats to distinguish between keeping all genomes, forbidding ambiguous ones, or restricting analysis to diploid sites only.

- Add `src/obikphylo/src/siblings/ambiguity.rs` defining the `AmbiguityPolicy` enum with variants `Keep`, `Forbid`, and `OnlyDiploid`, including methods for header flags and determining if a genome is dropped.
- Update `PhyloArgs` in `args.rs` to derive `AmbiquityPolicy` from CLI flags and construct `AnnexSelection` for sibling annexes.
- Modify `SamplingSpec` in `subsample.rs` to store `ambiguity: AmbiguityPolicy` instead of `no_ambiguity`, updating masking and scalar value logic accordingly.
- Update `SiblingExt` in `sibling_ext.rs` so `build_sibling_annex` accepts an `AnnexSelection` to build only requested derived annexes, and `family_overlap` uses the specific policy.
- Refactor `FamilyOverlap` in `family_overlap.rs` to store and use `AmbiguityPolicy`, replacing boolean checks with policy-based carrier determination.
- Update `FamilyScalarAnnex` in `family_scalar_annex.rs` to store `AmbiguityPolicy` in headers, enabling distinct file names for each policy variant.
- Modify `quartet_annex.rs` to generate distinct file names based on `AmbiguityPolicy`, ensuring no overlap between Keep, Forbid, and OnlyDiploid annexes.
- Update `family_annexes.rs` to use `AnnexSelection` for building entropy, quartet, and pair weight annexes under specific policies instead of a single boolean convention.
- Refactor algorithm modules (`entropy.rs`, `masking.rs`, `pair_weight.rs`, `quartet.rs`, `snp_distance.rs`) to accept `AmbiguityPolicy` arguments, replacing boolean checks with policy-specific logic (e.g., `OnlyDiploid` masking).
- Update documentation in `phylo.md` and `evolutionary_distances.md` to describe the new `--only-diploid` option, the behavior of `--sibling-annex` with specific policies, and the separate storage of entropies per convention.
2026-11-09 20:59:09 +11:00
Eric Coissac b55123a376 feat: add --only-diploid flag to filter for diploid genotypes only
Introduce a new `--only-diploid` option that restricts analysis to genomes with one or two family members (base or two-base IUPAC codes), treating others as missing data. This complements the existing `--no-ambiguity` behavior by providing a stricter filter for diploid-compatible sites, while ensuring conflict detection between the two flags.

- Add `only_diploid` boolean argument to `PhyloArgs` in `src/obikmer/src/cmd/phylo/args.rs`, documenting its effect on pseudo-alignment and its conflict with `--no-ambiguity`.
- Propagate `only_diploid` through `SamplingSpec` in `src/obikmer/src/cmd/phylo/mod.rs` and update the `run` function signature to accept the new argument.
- Implement masking logic in `src/obikphylo/src/siblings/algorithms/masking.rs`: modify `masked_state` to accept `only_diploid`, add `only_diploid_keeps_up_to_two_members` helper, and update `genome_mask` documentation.
- Update `sample_layer` in `src/obikphylo/src/siblings/algorithms/subsample.rs` to use `spec.only_diploid` for masking decisions and update `SamplingSpec` documentation.
- Adjust `tally_2genomes` in `src/obikphylo/src/siblings/algorithms/snp_distance.rs` to pass the new `only_diploid` parameter to `masked_state`.
- Update `DevDocMD/theory/evolutionary_distances.md` to describe the new `--only-diploid` option, the `masked_state` rule (`count_ones() > 2`), and clarify toy index behavior.
- Update `UserDocMD/usage/phylo.md` to document `--only-diploid`, `--quartet`, and `--pair-balance` behaviors, clarifying that diploid choice is a convention and explaining how three/four-allele genomes are treated as missing.
2026-11-09 20:58:21 +11:00
Eric Coissac b132f0847f feat: add VCF export and refine alignment output infrastructure
Introduce `--vcf` and `--fasta` options for phylo commands, allowing users to
export pseudo-alignment data as VCF or FASTA. Implement comprehensive VCF logic
including stable virtual coordinates, canonical ID generation, and flexible
genotype representation based on ambiguity settings. Refactor internal alignment
structures to track k-mer families and site loci, ensuring consistent data flow
from scanning through to output generation.

- Add `src/obikmer/src/cmd/phylo/alignment_out.rs` with `create` and
  `write_alignment_outputs` to handle file creation and dispatching of FASTA/VCF
  formats based on `PhyloArgs`.
- Add `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` implementing
  VCF-specific logic: `write_vcf`, `alignment`, `cell_alleles`, `majority_base`,
  and site construction via `site` using fixed $K=11$. Includes helpers for
  IUPAC state inversion, genotype ordering, and stable coordinate generation.
- Update `src/obikmer/src/cmd/phylo/args.rs` to add `fasta` and `vcf` flags
  and a new `alignment_output()` method that returns true if any output format
  is requested. Add `pseudo_alignment` field documentation.
- Modify `src/obikmer/src/cmd/phylo/mod.rs` to use `alignment_output()` for
  determining alignment needs, update error messages for `--n-sites`, and import
  `write_alignment_outputs` from the new module.
- Refactor `src/obikphylo/src/siblings/algorithms/alignment.rs` by adding
  `SiteLocus` struct (partition, layer, family_idx, family_id) and updating
  `SnpAlignment` and `AlignmentReducer` to track sites. Update `new`, `finish`,
  and `reduce` to populate site information from `SurvivingFamily`.
- Update `src/obikphylo/src/siblings/algorithms/iqtree.rs`: `alignment` now
  returns `SnpAlignment` with empty sites; update `drop_ascertainment_noninformative`
  and `recode_symbols_as_absent` to handle the new `sites` field.
- Update `src/obikphylo/src/siblings/algorithms/family_scan.rs`: `GeneratedBatch`
  now includes a `kmers` field; `scan_layer_families` accepts `CanonicalKmer`
  and passes it to closures instead of masks.
- Update `src/obikphylo/src/siblings/algorithms/subsample.rs`: `SurvivingFamily`
  gains a `kmer` field; `sample_layer` closure accepts and uses this k-mer.
- Update `src/obikphylo/src/siblings/algorithms/snp_distance.rs`: `tally_2genomes`
  initializes `SurvivingFamily` with explicit `CanonicalKmer` values.
- Update `src/obikphylo/src/siblings/algorithms/stats.rs`: `sibling_annex_stats`
  signature adds `_kmer` parameter.
- Update `src/obikphylo/src/siblings/algorithms/family_annexes.rs`: closures in
  `ensure_layer_family_annexes` and `ensure_layer_pair_weight_annexes` now accept
  `_kmer`.
- Update `src/obikphylo/src/siblings/algorithms/pairwise.rs`: `ambiguous_vs_single`
  initializes k-mer using `CanonicalKmer::from_raw_unchecked(0)`.
- Add `iupac_state` function to `src/obikphylo/src/siblings/algorithms/masking.rs`
  mapping DNA characters to binary codes.
- Update `src/obikphylo/src/siblings/algorithms/mod.rs` to export `SiteLocus`
  and include new `alignment_vcf` module.
- Add `family_id` and `write_family_id_ascii` methods to
  `src/obikseq/src/kmer.rs` for stable k-mer family identification.
- Add tests `family_id_partitions_kmers_into_families` and
  `family_id_ascii_masks_centre` to `src/obikseq/src/tests/kmer.rs`.
- Update `src/obikphylo/src/siblings/mod.rs` imports to include new algorithm
  exports.
2026-11-09 20:58:05 +11:00
coissac ee62b131f9 Merge pull request 'Push ltpzonnnkpqs' (#79) from push-ltpzonnnkpqs into main
Reviewed-on: #79
2026-10-16 14:43:50 +00:00
Eric Coissac 85296a2ef8 Implement directory locking with shared and exclusive modes
Release / create-release (push) Successful in 2m41s
Release / build-linux-x86_64 (push) Successful in 9m17s
Release / build-macos-arm64 (push) Successful in 2m2s
ci.yml / build (pull_request) Successful in 3m48s
Introduces a robust, OS-level directory locking mechanism using LockSlot for managing shared and exclusive access, supporting lock transitions and non-blocking probes. Also sets up the index structure with new builder and index modules.
2026-10-16 16:02:00 +02:00
Eric Coissac 9ac3a37529 Implement explicit directory locking for index operations
Introduces explicit directory locking mechanisms (`DirLock`) to ensure exclusive access during index operations across command stages. This includes modifying index opening logic to acquire locks before proceeding, handling lock acquisition failures by exiting, and implementing safe conditional building of sibling annexes using locking to prevent concurrent corruption.
2026-10-16 16:00:49 +02:00
Eric Coissac 7c7d7615cb Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
2026-10-16 15:59:40 +02:00
coissac c6226dd40c Merge pull request 'Push wpvskkyozppn' (#78) from push-wpvskkyozppn into main
Reviewed-on: #78
2026-10-15 10:38:58 +00:00
Eric Coissac 8296e3544c Simplify Hybrid layer implementation and update dependencies
Release / create-release (push) Successful in 2m31s
Release / build-linux-x86_64 (push) Successful in 9m24s
Release / build-macos-arm64 (push) Successful in 2m5s
ci.yml / build (pull_request) Successful in 3m51s
Dropped fields related to Hybrid layers for simplification and removed the unused `ahash` dependency. Updated dependency versions for `obidebruinj`, `obikmer`, and related crates, and cleaned up temporary artifacts.
2026-10-15 12:23:15 +02:00
Eric Coissac 12afbe819c Refactor MPHF layer logic and evidence handling
This commit implements core lookup functionality for MPHF layers, including `hash` and `find_strict` methods. It introduces support for different evidence types (Exact, Hybrid, Approximate) within layers, clarifying lookup behavior and improving partition calculation logic in the cache. Additionally, dead code is removed and dependencies are updated.
2026-10-15 12:22:45 +02:00
Eric Coissac f9edefd135 Refactor query system for performance and memory reduction
This change introduces dynamic chunk sizing based on input size estimates, ensuring memory constraints are respected while allowing for smaller chunks that yield significant speed improvements. Additionally, index structures were optimized, resulting in drastic memory footprint reduction and faster query execution times.
2026-10-15 12:21:38 +02:00
Eric Coissac 025c40c0db Refactor chunking logic to enforce 128 k-mer limit
This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
2026-10-15 12:20:36 +02:00
Eric Coissac 8468319a63 Refactor unitig detection and graph traversal logic
Introduce new hashing mechanisms and support for directional unitig checks. Fix fragmentation issues and improve performance metrics through optimized graph construction and new benchmarking utilities.
2026-10-15 12:19:24 +02:00
Eric Coissac d32d86193f Refactor index metadata loading to use buffered I/O
This change replaces unbuffered file reads with buffered I/O operations to improve performance during index metadata loading. This includes implementing new methods for reading index metadata files and wrapping file handles in `BufReader` to optimize subsequent reads.
2026-10-15 12:17:51 +02:00
Eric Coissac 093d81c359 Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
2026-10-15 12:17:09 +02:00
Eric Coissac 09c80468b2 Refactor k-mer storage to use flat arrays
This change replaces the HashMap-based k-mer storage mechanism with a compact, flat array structure for `PartitionKmers`. This significantly reduces memory usage per partition and improves query execution time.

It also updates the batch processing logic to handle the new data structure, introduces detailed memory footprint calculations, and updates relevant query layer APIs. Test coverage has been expanded to validate correct partitioning and k-mer ordering.
2026-10-15 12:16:21 +02:00
Eric Coissac fe38424682 Implement batch lookup and chunked data management
Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline.

This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
2026-10-15 12:14:51 +02:00
Eric Coissac 8b87078478 Implement MPHF migration tool and sparse matrix permutation logic
Refactors sparse matrix handling to implement row permutation logic and introduces a standalone tool for migrating indexes from MPHF 1.x to 2.x formats.
2026-10-15 12:12:51 +02:00
Eric Coissac d686d39506 Refactor indexing logic and update core dependencies
This commit introduces significant updates to the indexing system, including the introduction of zero-copy types, modifications to the `Mphf` type definition, and new hybrid search strategies. Core algorithms are updated to handle new hashing types and improved error handling during pointer hash construction. Dependency versions are also updated across the board, including new security features and system-level crates.
2026-10-15 12:07:08 +02:00
Eric Coissac 3bd7fd4f41 Implement memory mapping for file construction
Refactors the building process to utilize memory mapping (`memmap2`) for file creation and introduces the necessary infrastructure for publishing built files atomically. This includes expanding the public API surface and cleaning up deprecated architectural components.
2026-10-15 12:02:33 +02:00
Eric Coissac c226744405 Refactor sampling algorithms and coverage metrics
Optimize family scanning and sampling plan calculation logic. Introduce new per-genome richness metrics and redefine the sampling fraction $r$ for improved analysis.
2026-10-15 12:01:15 +02:00
Eric Coissac 0e40e83fd1 Refactor probability formula notation
Update the mathematical formula for $P(X \geq r)$ to explicitly use summation notation and clearly format the probability expression.
2026-10-15 12:00:51 +02:00
Eric Coissac cb5fc202b6 Refactor sampling logic and introduce diagnostic outputs
This commit introduces new features for sampling, including detailed `SamplePlan` generation, Shannon entropy summaries, and site statistics reporting. It formalizes conditional distance matrix computation and clarifies the logic for family weighting and site selection.
2026-10-15 12:00:44 +02:00
coissac 2232628e58 Merge pull request 'Push kkkmrtvotrsu' (#77) from push-kkkmrtvotrsu into main
Reviewed-on: #77
2026-10-13 07:54:48 +00:00
188 changed files with 12954 additions and 3303 deletions

No files matched your search

+24 -1
View File
@@ -62,7 +62,8 @@ jobs:
~/.cargo/registry ~/.cargo/registry
~/.cargo/git ~/.cargo/git
src/target src/target
key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock') }} tools/migrate-mphf/target
key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock', 'tools/migrate-mphf/Cargo.lock') }}
restore-keys: linux-musl-cargo- restore-keys: linux-musl-cargo-
- name: Build static binary - name: Build static binary
@@ -83,6 +84,28 @@ jobs:
-H "Authorization: token $GITEA_TOKEN" \ -H "Authorization: token $GITEA_TOKEN" \
-F "attachment=@/tmp/dist/obikmer-linux-x86_64" -F "attachment=@/tmp/dist/obikmer-linux-x86_64"
# One-shot converter for indexes built with ptr_hash 1.x. A standalone
# crate (own Cargo.lock/target, outside src/'s workspace), built after
# obikmer's own asset is already uploaded: it never blocks that one.
- name: Build migrate-mphf (static)
working-directory: tools/migrate-mphf
env:
PKG_CONFIG_ALLOW_CROSS: "1"
run: cargo zigbuild --release --target x86_64-unknown-linux-musl
- name: Upload migrate-mphf artifact
working-directory: tools/migrate-mphf
env:
GITEA_TOKEN: ${{ secrets.GITEATOKEN }}
RELEASE_ID: ${{ needs.create-release.outputs.release_id }}
run: |
cp target/x86_64-unknown-linux-musl/release/migrate-mphf /tmp/dist/migrate-mphf-linux-x86_64
strip /tmp/dist/migrate-mphf-linux-x86_64
curl -s -X POST \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases/$RELEASE_ID/assets" \
-H "Authorization: token $GITEA_TOKEN" \
-F "attachment=@/tmp/dist/migrate-mphf-linux-x86_64"
build-macos-arm64: build-macos-arm64:
needs: create-release needs: create-release
runs-on: ubuntu-latest runs-on: ubuntu-latest
+1
View File
@@ -9,6 +9,7 @@ CLAUDE.md
memory/ memory/
sandbox/ sandbox/
src/target src/target
tools/migrate-mphf/target
data-stress data-stress
*.fasta *.fasta
*.fasta.gz *.fasta.gz
@@ -0,0 +1,61 @@
# Hot resume and memoization of long computations (discussion, 2026-10-14)
## Goal (user)
1. **Hot resume**: a run killed midway (crash, `Ctrl-C`, scheduler walltime) restarts from the last finished unit of work, not from zero.
2. **No recomputation**: a long function called again with the same parameters (same command rerun, or another command needing the same intermediate) reuses its stored result.
Not a goal: bit-identical resumed samples. Sampling uses an OS-seeded RNG on purpose, repeated runs must stay independent draws to measure variance; a resumed run being a *different* valid draw is not a defect.
## Why it matters: one real run
`phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance --family-overlap --sibling-stats` on the 180-genome Primulaceae index, from the stderr of the OAR job (2026-10-13), 256 partitions:
| Stage | Wall |
|---|---|
| `sibling_stats` (full scan, `Selection::All`) | 8 235 s |
| `family_overlap` (sweep 1: overlap annex) | 5 582 s |
| `sampled_scan`: sweep 1 again (entropy + quartet annexes) | ≈ 5 347 s |
| `sampled_scan`: sweep 2 (pair-weight annex) | ≈ 5 296 s |
| `sampled_scan`: the draw itself | ≈ 9 573 s |
| `distance` (Jaccard, not asked for; older binary) | 53 s |
| Total | 34 085 s (9 h 28) |
The three annex sweeps cost about the same although their per-family work differs, so the cost is the layer traversal. Part of it was addressed by restricting the matrix pass to selected families (`family_scan.rs`, same session); the sweeps were not fused. Nothing in this run could be resumed or reused: every stage restarts from its beginning.
## What exists today (verified in the code, 2026-10-14)
**Persisted per-layer annexes.** Entropy (`entropy*.pent`), quartet (`quartet*.pbiv`), pair weight, sibling annex: one file per layer, built by `ensure_layer_*` and skipped when `path.exists()` (`family_annexes.rs`). A rerun with the same conventions skips every sweep. This is already memoization, keyed by *file name only* (convention in the name, no fingerprint of the inputs; a rebuilt sibling annex deletes everything derived from it, see `pair_coverage_sampling.md`).
**Not persisted.** The family-overlap accumulator (`FamilyOverlapAccumulator`) is written once at the end of sweep 1 (`sibling_builder.rs`). The draw (`sample_index`) and the `sibling_stats` scan keep their state in memory only.
**Defect found, then fixed (2026-10-14): a killed annex build left a file that looked finished.** `create_mmap_file` (`obicompactvec/src/mmap_file.rs`) created the file at its *final path* with magic, header and the full length (`set_len`) before any data was written. A kill during a layer left a valid-looking file with unwritten data; the next run saw `path.exists()` and treated the annex as built, silently.
Reproduced on the 1x testbed (20 genomes, 512 layers): derived annexes deleted, `phylo --family-overlap --quartet --pair-balance --entropy 1 …` killed by `timeout 6`, then rerun. Old binary: 141 finals after the kill, 2 differing from a reference build; the rerun exited 0 and kept both. Fixed binary: kill at 6 s left 499 finals, all identical to the reference, plus 2 `.building` files; the rerun completed 1 537 files identical to the reference, no `.building` left. (Kill at 3 s: nothing published, nothing broken.)
**Fix.** `obicompactvec::building_path` / `publish_built`: a file is built under `<name>.building` and moved to its final name (same-directory `rename`, atomic) by the builder's `close()`, after the flush. Applied to the builders that go through `create_mmap_file` or write their own mmap on the annex path: `PersistentBitVecBuilder` (all constructors; quartet annex), `PersistentFixedIntVecBuilder`, `PersistentRankSelectBitVecBuilder`, `FamilyScalarAnnexBuilder` (entropy, pair weight), `SiblingAnnexBuilder` (`siblings.psib`) and `FamilyOverlapAccumulator::write` (overlap matrix). A leftover `.building` is truncated by the next build of that file. A build that is dropped without `close()` no longer produces the final file (no caller did; workspace tests pass). Also reordered in `--sibling-annex`: derived annexes are now removed *before* the new sibling annex is built and published, so a kill between the two leaves the old annex with no derived files, never a new annex next to files computed against the old numbering.
**Not covered.** The builders outside that path still write at their final name: `PersistentCompactIntVecBuilder`, the int/sparse matrix builders, `EliasFanoBuilder`, `TempBitVecBuilder`/`TempCompactIntVecBuilder` (temporary by nature) and the index-construction files that use them. The same reasoning applies to them as soon as a stage built on them has to be resumed; not audited. The in-memory overlap accumulator is still lost on a kill (a sweep restarts), now without leaving a corrupt file.
**`obiksession` crate** (`src/obiksession/`): `Session` (named directory, opaque parameter blob checked on open, whole-lifetime `DirLock`, artifacts checksummed with xxh3 and mmap-restored) and `Memoize` (`get_or_compute[_with_key]`, `invalidate`, `cached_keys`, `MemoKey` from `Debug` + xxh3, the `session_memoize!` macro). Compiles, 15 tests pass, **no caller**: `obikmer` lists the dependency without using it. `phylo --session` exits with an error (`cmd/phylo/mod.rs:271`) since `session_cache.rs` (the consumer tied to `PairwiseTally`/`PartitionDispersion`/`SnpAlignment`) was removed. `sample_index(skip_layers)` remains as a seam, never called with a non-zero value.
**Previous design and why it missed the goal** (`theory/evolutionary_distances.md`, section "Explicit `--session`"): one session per *exact* parameter tuple (changing `--n-sites` made another session), artifacts specific to the sampling result, resume only inside the draw (checkpoint every 8 layers or 30 s). Reuse across commands existed only for the sample. An exhaustive-sample cache from which smaller samples would be drawn was proposed and never done. User judgement: this version did not answer the objective.
## Proposed direction (not implemented; to be validated)
Start from the goal, not from the existing crate.
1. **Unit of work = one `(partition, layer)` step that already produces files.** The annex builds, the draw per layer and the scans all iterate that way (`for part … for layer …`). Resume means "skip the units whose output is complete", exactly what `exists()` approximates today.
2. **Complete-or-absent outputs.** Done for the annex path (see the fix above); the remaining builders are to be covered as their stages become resumable. Prerequisite for everything below. The mechanism lives in `obicompactvec` (shared lower-level API), not in each consumer.
3. **Key = function identity + parameters + fingerprint of the inputs.** A result is reusable iff the function, every parameter that influences it and the identity of its inputs (index content, upstream artifacts) match. Today the key is a file name plus an invalidation convention. The fingerprint requirement is the cost: what identifies an index version cheaply is to be decided (meta file, rebuild counter, hash of annex headers).
4. **The cache holds results of long functions, not domain objects.** The crate must not know `PairwiseTally`. Callers serialize their own values (`rkyv` was already chosen for that). Whether `Memoize`'s API fits is unproven: it has no caller, no documentation and was written against the earlier design.
5. **Granularity matters more than the API.** A cached 9 h result helps only the exact rerun; a per-layer cache also helps a changed parameter that leaves earlier layers untouched. The overlap accumulator (end-of-sweep write) would need a per-layer partial (a sum of per-layer matrices, mergeable) to be resumable.
6. **Fuse before caching.** Three sweeps over the same layers where two suffice (overlap first, then entropy + quartet + pair weight together, since the pair weight needs only the finished overlap matrix) is a cost no cache removes on a first run; to do independently.
## Open questions
- Fingerprint of an index version: which cheap, stable identity?
- Overlap accumulator resume: per-layer partial matrices on disk (size n² × layers) or a checkpoint every N layers?
- `Memoize`/`Session`: keep (adapt to the points above), rewrite, or delete the crate until a caller exists (the "existing code is a hypothesis" rule applies; today it has no user).
- Scope of the next step: atomic writes are done for the annex path; extend to the other builders, or go to the cache.
- Draw resume: a layer's draw is independent of the others except through the quotas, which depend on every layer's mass; quotas can be recomputed from annexes on resume (cheap), so only the per-layer survivors need storing.
+43 -1
View File
@@ -24,7 +24,7 @@ for each chunk of sequences (parallel workers via obipipeline, one call to proce
decompose all sequences into superkmers (SuperKmerIter) — construction only, decompose all sequences into superkmers (SuperKmerIter) — construction only,
not the dedup key not the dedup key
deduplicate at k-mer granularity, split by partition in the same pass: deduplicate at k-mer granularity, split by partition in the same pass:
by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>> ← KmerDesc = (seq_idx, pos) by_partition: Vec<PartitionKmers> ← sorted unique k-mers + flat KmerDesc = (seq_idx, pos) slices (2026-10-15, was a HashMap — see "Query memory" below)
allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers — allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers —
NOT multiplied by n_genomes NOT multiplied by n_genomes
allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome — allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome —
@@ -404,3 +404,45 @@ Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-w
- **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone. - **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone.
- **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition. - **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition.
- **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action. - **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action.
## Query memory: `PartitionKmers` replaces the per-partition `HashMap` (2026-10-15)
**Problem.** A full-size query (two ~450 MB fastq.gz of a skim against a 354 M-kmer index) reached **74–81 GB RSS** with the default 256 MiB chunk. Cause, read in `batch.rs`/`chunk.rs` and measured with the `by_partition memory retained` debug log: `QueryBatch::by_partition` was a `HashMap<CanonicalKmer, Vec<KmerDesc>>` per partition. On genomic reads almost every k-mer is unique (12.5 M unique for 13.5 M occurrences in a 48 MB chunk, a dedup of 8 %), so each k-mer paid a table slot, growth slack and its own heap `Vec`: **889 MB per 48 MB chunk, ~71 bytes per unique k-mer, 18× the chunk size** — the dominant term of the per-chunk memory, several chunks being in flight at once.
**Change.** `obikquery::PartitionKmers`: occurrences collected as `(kmer, KmerDesc)` per partition, sorted by `(kmer, seq_idx, pos)` and stored as three flat arrays (unique k-mers, u32 offsets, descs). `query_partition_with` takes `&PartitionKmers`; `QueryHit` already carried slices. Occurrence order per k-mer is unchanged (query order); no per-k-mer allocation.
**Measured** (same inputs, release build, identical sorted output digests before and after):
| | `by_partition` (48 MB chunk) | RSS, 2×48 MB chunks | RSS, full run (≈ 1 GB reads) | wall, full run |
|---|---|---|---|---|
| before | 889 MB | 5.6 GB | 73.7–80.7 GB | 135–172 s (one 312 s outlier) |
| after | 259 MB | 4.1 GB | **28.2 GB** | **103 s** |
With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the touched pages of the index itself, so after the change the remaining per-chunk anonymous memory is ≈ 6× the chunk size, below the formula's factor 12 (`BASE_OVERHEAD + 8·n_genomes`, still a conservative bound — left unchanged). The sort replaces hashing in `from_records` and was not slower (17.3 s vs 21.2 s on the small test).
**Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here.
**Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed.
### Chunk size sweep on a small input (2026-10-15)
Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer single-genome index (4 partitions), 16 cores on AC power, 3 runs per size, sorted outputs identical (same md5) for every size:
| `--chunk-size` (MiB) | chunks | wall (s) | user CPU (s) | RSS (GB) |
|---|---|---|---|---|
| 2 | ~50 | 2.5–2.6 | 25.0–25.3 | 3.2 |
| 4 | ~25 | 2.5–2.7 | 24.3–25.5 | 3.5 |
| 8 | ~12 | 2.5–2.7 | 23.6–26.1 | 4.1 |
| 16 | ~6 | 4.4 | 23.1–23.7 | 4.1 |
| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 |
| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 |
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs — **implemented, see below**. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
### Automatic chunk size now follows the input size (2026-10-15)
`cmd/query/mod.rs`: when `--chunk-size` is not given, the memory-derived size (unchanged: `available RAM / (workers × overhead × 2)`, clamped to 4–256 MiB) is further capped by `estimated input bytes / (workers × 2)`, never below 4 MiB. The estimate is `obiread::estimated_decompressed_len`: the file length, ×3 when the magic bytes say gzip (≈ 2.8 measured on FASTQ), ×4 for bzip2/xz/zstd — a deliberately rough figure, detected from the magic bytes, not the extension. For stdin, URLs or unreadable files the size is unknown and the memory-derived size stands. An explicit `--chunk-size` still wins.
Measured (2 × 48 MB reads, 16 cores, AC power, outputs identical): single-genome index, 354 M kmers — default chunk 256 MiB → 4 MiB: **11.4 s → 2.4 s** (−79 %), CPU 22.3 → 26 s (+17 %), peak footprint 1.44 → 0.75 GB. On the 91-genome index the memory-derived size was already 4 MiB (overhead factor 4 + 8 × 91), so nothing changes there (5.1–5.7 s either way). Not measured: a large input (≈ 1 GB gz, whose estimate gives ≈ 95 MiB chunks instead of 256 MiB: more, smaller chunks, expected lower memory).
+18 -32
View File
@@ -62,52 +62,38 @@ fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
Check all sources against the constraints above. Abort on any mismatch. Check all sources against the constraints above. Abort on any mismatch.
### 2. Bootstrap output from first source ### 2. Bootstrap output from the base source
Recursive file copy of `sources[0]` → `output`. Immediately after the copy: The base is chosen by `choose_base` (`merge.rs`): the source with the largest total `unitigs.bin` size (restricted to approximate-evidence sources when the output must be approximate). It is not necessarily `sources[0]`, and the genome columns follow the base first. Because `obikmer index` is not byte-reproducible (unitig segmentation varies), two near-equal sources can swap roles between runs. Its directory is copied to `output` (hard links for files `merge_partition` never rewrites, see below). Immediately after the copy:
- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind. - `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
- In `Presence` mode, any `counts/` directories inherited from source_0 are removed. - In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels. - `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination. This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The base source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
### 3. For each subsequent source (parallel across partitions) ### 3. For each subsequent source (parallel across partitions)
`KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)` is called for each partition index `i`. `block_bits` is taken from `dst.meta.config.block_bits`. `obikmerge::partition_merge::merge_partition(dst, i, sources, mode, n_dst_genomes, block_bits, evidence)` runs once per partition index `i` (scheduled largest first by `PartitionRunner`). Each entry of `sources` is `(&KmerIndex, n_genomes)`, where `n_genomes` is the number of columns that source contributes (> 1 for an already-merged source). Code: `partition_merge/mod.rs` (orchestration, pass 1) and `partition_merge/pass2.rs` (per-group fill pipeline).
Each entry in `sources` is `(&KmerPartition, n_genomes)` where `n_genomes` is the column count that source contributes (> 1 when the source is itself a merged index). **First merge, Presence mode**: when `n_dst_genomes == 1`, `TypedLayer::<()>::init_presence_matrix` is called on every existing dst layer first. It materialises the implicit single-genome matrix (`presence/col_000000.pbiv`, all true) so that source columns can be appended to it.
**First merge, Presence mode**: when `n_dst_genomes == 1`, `Layer::<()>::init_presence_matrix` is called on every existing destination layer before any source column is appended. This creates `presence/col_000000.pbiv` set all-true (genome 0 is present in every slot). The dst layers are opened once per partition and shared by both passes.
**Pass 1 — classify kmers** **Pass 1 — new k-mers into a de Bruijn graph.** Only the sources' `unitigs.bin` *paths* are needed (`KmerLayer::new(..).unitigs_path()`); no source layer is opened. The files are streamed (at most `max_open = 2` open at once), and every k-mer absent from all dst layers is pushed into one `GraphDeBruijn`. All new k-mers of all sources go into a **single** new layer per merge, built by `materialize_layer` (unitigs + MPHF + evidence).
Iterate all kmers from all source partitions (via `UnitigFileReader` + canonical kmer iteration). For each kmer, probe the destination `LayeredMap<()>`: **New-layer matrix.** If a new layer exists, its matrix is created with `n_dst_genomes` absent columns (all zero/false) and closed immediately. This keeps the column-count invariant, and every source group then appends to it like to the existing layers.
- **Hit**: kmer already in the destination; record for Pass 2. **Pass 2 — by consecutive groups of `SRC_GROUP = 2` sources, in source order.** Per group:
- **Miss**: push kmer into a `GraphDeBruijn` accumulator.
**New layer construction** 1. open the group's source layers for this partition;
2. `MatrixBuilder::resume` every dst matrix (existing layers + new layer), check `n_cols == n_dst_genomes + col_base`, append the group's `group_n` columns;
3. run the pipeline: stream each source layer's k-mers, `hash_batch` + `fill_sub_matrix` on the source, locate each k-mer in the dst layers (`find_batch_grouped`) or else in the new layer's MPHF, and write the values column-major into the group's builders;
4. close the columns, then the matrices; drop the group's layers; `col_base += group_n`.
If the accumulator is non-empty, compute de Bruijn unitigs and call `Layer::<()>::build(&new_layer_dir, block_bits)`. All kmers absent from the destination — across **all** sources — accumulate into a **single** graph, producing one new layer per merge operation (not one per source). Only one group's source layers and destination columns are alive at a time, so the number of live mappings per partition does not depend on the number of sources. Columns are appended in source order, so every genome lands in the same absolute column as in a single-session merge.
**Pass 2 — fill column builders** **In-memory columns.** The group's columns are built in RAM (`MatrixBuilder::add_mem_col` → `MemBitVecBuilder` / `MemCompactIntVecBuilder`) and written once, sequentially, at `close` (`.building` file, then `rename`). The files are byte-identical to the former mmap-backed builders' output. There is no sparse file and no `MmapMut` on this path, so a block the FS cannot allocate is an `io::Error` instead of a SIGBUS. Files are not `fsync`ed: one sync per column file made merge ~8× slower. RAM per partition is at most `SRC_GROUP × src_n × N_part × (1/8 | 1)` bytes (presence | counts). History and figures: `merge_mmap_audit.md`; scaling limits: `merge_scaling.md`.
For each source and each of its layers, re-iterate unitigs and look up stored values via `SrcLayerData::lookup(kmer, src_n)`:
- `SrcLayerData::SetMembership` — no data directory exists; every kmer returns `vec![1; n_genomes]`
- `SrcLayerData::Presence` — reads `PersistentBitMatrix` from `presence/`
- `SrcLayerData::Count` — reads `PersistentCompactIntMatrix` from `counts/`
Hits are routed to `exist_builders[dst_layer][src_col]`; misses are routed to `new_src_builders[src_col]`.
**Column prepending for new layers**
Before source columns are written to the new layer, `n_dst_genomes` absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.
**Close and update metadata**
Close all builders; update `presence/meta.json` or `counts/meta.json` with `{"n": N, "n_cols": n_dst_genomes + n_src_total}`; increment `PartitionMeta::n_layers` if a new layer was added.
### 4. Update index metadata ### 4. Update index metadata
@@ -129,7 +115,7 @@ impl Layer<PersistentBitMatrix> {
} }
``` ```
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written directly via `PersistentBitVecBuilder` / `PersistentCompactIntVecBuilder` rather than through these helpers, but the invariant they enforce is the same. Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written through `MatrixBuilder::add_mem_col` (in-memory builders) rather than through these helpers, but the invariant they enforce is the same.
--- ---
@@ -139,7 +125,7 @@ After any merge, **every layer in every partition has exactly `n_genomes` column
Maintained by three mechanisms: Maintained by three mechanisms:
1. **Existing layers**: `n_src_total` columns appended (one per source genome). 1. **Existing layers**: `n_src_total` columns appended (one per source genome), group by group.
2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns. 2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0. 3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
@@ -161,7 +147,7 @@ The invariant is a precondition of `LayeredStore` aggregation traits: `col_weigh
## On-disk impact ## On-disk impact
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest): After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest), *before* `finalize` packs each matrix (sparse by default, `matrix.pbmx` with `--dense`):
``` ```
partitions/ partitions/
@@ -0,0 +1,96 @@
# Merge: mmap audit (2026-11-17)
Audit of the memory-mapping behaviour of `obikmerge::partition_merge::merge_partition`, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS `/hoyt`). Status: **A + C implemented (2026-11-17)**, B deferred. See [Implementation](#implementation-2026-11-17). Implementation prompt: `prompts/merge_mmap_refactor.md` (repository root, outside the doc tree).
## Observed failures
| Run | Flags | Failure | When |
|---|---|---|---|
| 1 | none | `Bus error` (SIGBUS) | seconds after `merge_partitions` starts |
| 2 | `--cpu-max 32` | `memory allocation of 32768 bytes failed` → abort, many threads at once | after ~1 h, 128/256 partitions done |
Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, `merge.rs:289`), so at 50 % every remaining partition is smaller than one already merged. Small `malloc` failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from `malloc` means `mmap`/`brk` returned `ENOMEM`: `vm.max_map_count` (default 65 530), `RLIMIT_AS`, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.
## Inventory: mappings held by one `merge_partition` call
Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).
| Resource | Where | Maps | Lifetime | Needed? |
|---|---|---|---|---|
| Source layers, fully opened | `partition_merge.rs:219-227` → `KmerLayer::open` → `MphfLayer::open` (`mphf_layer.rs:171`) | 4 per layer: MPHF (`MphfOnly`), `unitigs.bin` (direct access), `evidence.bin`, `unitigs.bin` again (`LayerEvidence::open`, `mphf_layer.rs:100`). ≈ **740** | whole partition | Only the MPHF. Pass 2 calls `hash_batch` (MPHF only, `mphf_layer.rs:258`) and `fill_sub_matrix` (matrix: `Implicit` for mono-genome, zero maps). Pass 1 only needs the `unitigs.bin` *path*. |
| Writable columns, created up front | `partition_merge.rs:327`, `:349` → `PersistentBitVecBuilder::new` / `PersistentCompactIntVecBuilder::new` | `set_len` (sparse) + `MmapMut`, `n_src_total × (L_dst + 1)` ≈ **370** | whole pass 2 | Only the columns of the ≤ 2 sources being streamed (`max_open = 2`). |
| dst layers, new-layer MPHF, sequential readers | various | ≈ 10 | variable | yes |
≈ **1 100 maps per partition**, > 95 % idle at any instant. × ~32 concurrent partitions (`--cpu-max 32`) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, **not measured**. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is **not established**.
## Weaknesses
1. **Source layers opened in full** although merge needs MPHF + matrix only. Evidence and both `unitigs.bin` maps are dead weight.
2. **All sources opened up front, for the whole partition.** This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
3. **All destination columns created up front**, the same mismatch on the write side.
4. **Sparse files written through `MmapMut`.** A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing.
5. **No bound on live mappings.** `obipipeline::throttle` only bounds open read streams.
6. *(minor)* `obicompactvec::mmap_file::open_mmap_file` validates magic and header length, not file length against `n`.
Side finding: `merge_parallelism.md` documents a `MemoryBudget` semaphore and `--budget-fraction`. `obisys::MemoryBudget` still exists but `merge.rs` no longer uses it (scheduling is `PartitionRunner`). That page is stale on this point.
## Plan
**A. Source-group streaming in pass 2** (`obikmerge`). Process sources in consecutive groups of `G` (G = 2, matching `max_open`). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its `src_n` columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through `KmerLayer::new(..).unitigs_path()` (valid on `Empty`). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.
**C. In-memory column builders** (`obicompactvec`). Under A only `G × src_n × (L_dst+1)` columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with `write_all` to the `.building` path, then publish. No sparse file and no `MmapMut` on the merge write path, so `ENOSPC`/quota becomes an `io::Error`. The on-disk format must be byte-identical to the current builders' output. Exposed through `MatrixBuilder`/`ColBuilder` so `obikselect`/`obikfilter` can adopt it later. Fixes 4.
**B. Slots-only layer opening** (`obikindex`, the authority crate). **Deferred to a separate work item** (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same `hash_batch` + `fill_sub_matrix` pattern without `find` appears in three crates: `obikmerge` (`partition_merge.rs:489`), `obikfilter` (`partition_iter.rs:106`), `obikphylo` (`family_scan.rs:289`). `family_scan.rs:402` hashes `variants`; whether those are guaranteed members is unchecked. Fixes 1.
- **Rejected: lazy evidence inside `MphfLayer`.** `find`/`confirm` return `Option<usize>`, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (`find`, needs evidence) vs in-layer iteration (`index`/`index_batch`, no evidence).
- **Rejected: a standalone parallel type** next to `KmerLayer`/`TypedLayer`. It would duplicate count/presence detection and matrix access, and two opening paths would drift.
- **Retained: composition.** A slots layer = `MphfOnly` + matrix, carrying `hash_batch`, `fill_sub_matrix`, `n_cols`. The full layer = slots layer + evidence, keeping `find`/`confirm`. `KmerLayer` exposes both openings (`open()` / e.g. `open_slots()`) over a single count/presence detection. Open point: `MphfLayer::n` currently comes from the evidence (`evidence.len()` / `fingerprint.n()`). A slots layer must take it from the matrix (`n_rows`; `layer_meta.json` for `Implicit`), so the two must be checked to always agree.
Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the `obikindex` query path and needs its own session with `query` performance checks.
Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.
## Implementation (2026-11-17)
**A — source-group streaming** (`obikmerge/src/partition_merge/`, split into `mod.rs` + `pass2.rs`). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates `sources.chunks(SRC_GROUP = 2)`, resuming each dst matrix, checking `n_cols == n_dst_genomes + col_base` (`InvalidData` otherwise), and appending the group's columns. Prerequisite: `MatrixBuilder::n_cols` (`obicompactvec`).
**C — in-memory columns** (`obicompactvec`). New `mem_builder.rs` (`MemBitVecBuilder`, `MemCompactIntVecBuilder`), `add_mem_col` on both matrix builders and on `MatrixBuilder`, and `ColBuilder::MemBit`/`MemInt`. Format writing is factored rather than duplicated: `format::write_pciv_header`/`write_pciv_tail` (also used by `finalize_pciv`), `mmap_file::write_std_header`/`write_built`. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses `add_mem_col` for source columns; `add_absent_col` is unchanged (never written through its mapping).
Deviations from the plan:
- **No `sync_all` in `close`.** With one forced sync per column file, `merge_partitions` went from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. The `rename` still prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface at `close(2)`, which `File`'s drop ignores.
- `add_mem_col` returns `ColBuilder`, not `io::Result<ColBuilder>`: it does no I/O.
- The mmap count builder (`PersistentCompactIntVecBuilder`) writes in place, without `.building`; the in-memory one goes through `.building` + `rename`. Same final bytes.
- `obikmerge`'s `matrix_builder_tests` were removed: they only exercised `obicompactvec::MatrixBuilder` and are covered by `obicompactvec/src/tests/{matrix_builder,mem_builder}.rs`. `tempfile` dev-dependency of `obikmerge` removed with them.
**Verification.** Two golden-output oracles, recorded on the unmodified code:
- `scripts/smoke_test_merge.sh`: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/`--dense`; frozen source indexes;
- `scripts/bench_merge_oracle.sh`: 20 real specimen presence indexes, sorted `dump --debug` hash, 85 681 700 k-mers.
Both pass after every step. The existing `global_index_presence` (October binary) gives the same hash.
`merge_partitions` wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.
**Live mappings per partition (estimate, not measured).** Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.
| | Source layers | Dst columns | Other | Total |
|---|---|---|---|---|
| Before | 186 × 4 ≈ 744 | 186 × 2 ≈ 372 (`MmapMut`) | ≈ 10 | **≈ 1 100** |
| After A + C | 2 × 4 = 8 | 0 (in RAM) | ≈ 10 | **≈ 20**, independent of the source count |
B would take the 8 down to ~2.
**Follow-up — sparse pack (same day).** `pack_sparse_bit_matrix`/`pack_sparse_compact_int_matrix` used to open the columnar matrix directly, which maps every column file of a layer at once: `n_genomes` maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single `matrix.pbmx`/`.pcmx` mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: `pack` went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale `matrix.pbmx` that predates newer columnar data is now re-packed instead of being used as is (`pack_bit_matrix` compares `n_cols`). Both oracles pass.
## Re-merging a merge output (resolved 2026-11-18)
Symptom: `I/O error: No such file or directory` in `merge_partitions` when a merge output was one of the sources. Cause (verified): `finalize_indexed` packs the output matrices (`sparse_meta.json` + dict/PFIV files, or `matrix.pbmx`), so `presence/`/`counts/` no longer holds `meta.json` + `col_*`. `choose_base` picks the largest source as destination, i.e. the merged one; bootstrap copies the packed layout and `MatrixBuilder::resume` → `MatrixMeta::load` finds no `meta.json`.
Fix: a packed index can only be a *source* (pass 2 reads it through `PersistentBitMatrix`/`PersistentIntMatrix::open`, which handle every layout). `choose_base` now filters candidates with `can_be_destination` (`obikmerge/src/merge.rs`): one layer is probed with `detect_storage`; `Columnar` qualifies, `Implicit` only for a single genome (materialized on the fly), `Packed`/`Sparse` never. With a single source there is nothing to append and the filter is skipped. When several sources are given and none qualifies, `run` returns an explicit error before any work. Consequence: the result's column order follows the chosen base first, so it can differ from the command-line order.
Not done: in-place unpack of a packed index to allow it as destination, or a destination-side append on the packed format — the sparse incremental strategy is tracked in `merge_scaling.md`.
Oracle: `scripts/smoke_test_merge.sh` scenario 2 (A B → AB, then AB C D) is compared against the flat merge by genome label (not against the golden dumps, whose layer placement differs), and merging only packed indexes is checked to be refused.
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to `mmap`.
+41
View File
@@ -0,0 +1,41 @@
# Merge: scaling to thousands of genomes (open, 2026-11-17)
Open design question raised during the mmap refactor (`merge_mmap_audit.md`). Target: merges of ~6 000 genomes of a few Gb each. Nothing here is implemented or measured. Figures are estimates, labelled as such.
## What scales with what (after the mmap refactor)
| Cost | Depends on | Status |
|---|---|---|
| Live source layers + dst columns per partition (merge pass 2) | `SRC_GROUP × src_n`, not the source count | fixed by plan A |
| Pass-2 column RAM | `SRC_GROUP × src_n × N_part × (1/8 \| 1)` B | bounded, see below |
| Pass-1 de Bruijn graph | ≥ ~19 B per *new* k-mer (`HashMap<u64, AtomicU8>`, verified in `obidebruinj/src/debruijn/graph.rs`) | unchanged |
| Columnar intermediate files | `n_genomes × n_layers × n_partitions` (6 000 × 2 × 256 ≈ 3 M files, estimate) | **does not scale** |
| Sparse `pack` | used to map **all** columns of a layer at once (`ColumnarBitMatrix::open`); layers are packed one after another, so `n_genomes` maps per partition (186 × 192 concurrent ≈ 36 k, estimate) | **fixed 2026-11-17**: now goes columnar → dense `matrix.pbmx` (sequential concatenation, unmapped) → sparse, one mapping per partition |
`N_part`: k-mers of one dst partition over all layers. Pass-2 column RAM stays below pass-1 graph RAM for one-shot merges. It dominates only for **incremental** merges, where few genomes are added to a large index: the new layer is small, the existing layers are large. Example (estimate): `N_total = 1e12`, counts, 192 concurrent partitions → ~1.5 TB. A memory-budgeted concurrency cap is needed there whatever the column strategy. `obisys::MemoryBudget` exists but is no longer used by merge.
## Why a columnar intermediate at all
The sparse format (`bitmatrix/sparse.rs`) is row-major and deduplicated: per slot, either one genome (`singleton.pfiv`) or a `dict_id` (`multi.pfiv`) into a dictionary of distinct genome sets (`dict_offsets` + `dict_values.bin`). Its builder only offers `push_row(genomes)`, i.e. the **complete** genome set of a row, rows in **slot order**.
Merge discovers values **column by column** (source by source), in **random slot order** (MPHF order). The columnar files are a transposition buffer: they absorb random per-column writes, and `pack` transposes them row by row.
## Options to build the final format directly
1. **All sources at once.** That is exactly what plan A removed. No.
2. **Whole matrix in RAM.** `N_part × n_genomes / 8` B per partition: ~3 TB at 6 000 genomes and `N_part = 4e9` (estimate). No.
3. **External sort of `(slot, genome)` pairs.** Volume ∝ non-zeros ≈ Σ genome k-mers (~6 000 × 3e9). That is huge, while the final format is small precisely because genome sets repeat. No.
4. **Incremental sparse update per source group** (most promising). Per group:
- fill the group's columns in RAM (as now);
- stream the partition's existing sparse matrix **sequentially in slot order**;
- for each row: new set = old set ∪ group bits. Through the dictionary this is a transition `(old dict_id, group bit pattern) → new dict_id`, which can be memoised in a table;
- write the new sparse matrix sequentially, then publish.
No columnar files, no `pack`, no per-genome mapping, sequential I/O only. It also removes the re-merge bug: it starts from an existing sparse matrix instead of requiring a columnar one.
The cost is **one full pass over the partition's sparse matrix per group**. With `SRC_GROUP = 2` and 6 000 genomes that is 3 000 rewrites, far too many. Groups must therefore be large (64–256 sources), and group column RAM grows with them: `group × N_part / 8` B, e.g. 32 GB per partition for 64 sources at `N_part = 4e9` (estimate). One parameter trades passes against RAM, and it should be sized from a memory budget. The count matrix (`sparse_intmatrix.rs`) has its own format and needs the same analysis.
## Related findings
- The module header of `bitmatrix/sparse.rs` says "not used by any production code path yet". This is stale: `finalize_indexed` packs to sparse by default.
- The new layer of a merge is not byte-reproducible run to run: unitig segmentation varies, so the MPHF, evidence and matrix row order vary with it (measured 2026-11-17, `layer_1/*` differs in all 256 partitions; `layer_0` and metadata are identical). `obikmer index` shows the same behaviour. Merge oracles therefore compare sorted dumps, not file hashes. Where the non-determinism originates (parallel graph fill or unitig walk) is not established.
@@ -0,0 +1,369 @@
# MPHF batch queries — site inventory and expected gains
Investigation of 2026-10-14 (`ptr_hash` 2.1.2, `epserde` 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked **[read]** (verified in source), **[computed]** (derived from stated sizes) or **[estimate]** (reasoning, not measured). Measurements, once made, are in the last section.
## 1. What `ptr_hash` 2.1.2 actually offers [read]
`Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>`.
- **`V` is the pilot array** (one byte per bucket), **`F` is only the remap storage**. A layer's `mphf.bin` (~2.4 bits/key **[computed]**: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetch `pilots[bucket]` — the right line. The remap (read only when `slot >= n`, a small fraction of keys) is **not** prefetched.
- `index_batch<K>(xs: [Q; K]) -> [usize; K]` — stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (`len % K`) is the caller's job.
- `index_stream<B>(keys: IntoIterator) -> impl Iterator` — ring buffer of B keys; handles head and tail itself; accepts any iterator. **Its `next()` is `unimplemented!`**: only fold-based consumption works (`for_each`, `fold`, `sum`). `collect()`, `zip`, `for x in it` panic. Consume with `for_each` into a pre-sized `Vec`.
- `index_batch_exact` needs the nightly `unstable` feature; `index_batch_exact2` is `#[doc(hidden)]`. Neither is usable.
- Prefetch is `prefetch-index 0.2.1`: `_mm_prefetch(T0)` on x86_64, asm on aarch64, **no-op elsewhere**.
- **`SINGLE_PART`**: neither `index_stream` nor `index_batch` branches on it — both always run the multi-part `bucket()`/`slot()`. Only `index()` has the faster single-part path. So with batched queries the `SINGLE_PART = true` advantage disappears, and `false` (our choice) costs nothing while keeping parallel construction (`true` also panics with sharding).
- Old `ptr_hash` 1.1.0 (used by `tools/migrate-mphf`) has the same family with an extra `MINIMAL` const: `index_stream<B, MINIMAL, Q>`, `index_batch<K, MINIMAL, Q>`. No new dependency needed there.
```rust
// consuming index_stream: fold-based only
let mut out = Vec::with_capacity(kmers.len());
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
.for_each(|slot| out.push(slot));
```
## 2. Existing "batch" APIs that are not batched [read]
`MphfLayer::hash_batch` and `MphfOnly::index_batch` loop `.index()` one kmer at a time. Consumers: `obikfilter/partition_iter.rs` (`iter_layer_kmers`), `obikmerge/partition_merge.rs` (pass 2: `hash_batch`, `index_batch`), `obikphylo/siblings/algorithms/family_scan.rs` (two sites).
## 3. Cost chain of a membership query (`MphfLayer::find`) [read]
1. `pilots[bucket]` — the only access `ptr_hash` prefetches.
2. Exact/Hybrid-exact: `evidence.decode(slot)` — 4 B/slot, ~520 MB per 130 M-slot layer **[computed]**, random mmap read.
3. `verify_canonical_kmer` → `block_offsets[...]` (O(1) with the default `--block-size 1`; a forward scan of up to `2^block_bits` records otherwise).
4. `extract_kmer_raw` — random read in the `unitigs.bin` mmap.
Approx/Hybrid-fast: pilots then `fingerprint.get(slot)` — one extra random access.
Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. `ptr_hash` only helps with stage 1; stages 2–4 need prefetch support in `obikindex`/`obiskio` (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.
Layer sizes: `layer_0` MPHF 2–3 MB (cache-resident), `layer_1` ~40 MB (beyond one CCD's L3 on EPYC-class machines — **hypothesis**, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on `layer_1`. For branch-free independent loops (`hash_batch`) out-of-order execution already overlaps several misses, so the explicit prefetch gains less **[estimate]** than for branchy loops (`find` with early `break`).
## 4. Call-site inventory
(a) already batched; (b) batch buildable; (c) hard/pointless.
| Site | What | Class |
|---|---|---|
| `mphf_layer.rs` `hash_batch`, `MphfOnly::index_batch` | loop of `.index()` | (a) |
| `obikfilter/partition_iter.rs:106` | `hash_batch`, 32768 kmers | (a) |
| `obikmerge/partition_merge.rs:489, 521` | `hash_batch`, `index_batch`, 4096 | (a) |
| `obikphylo/.../family_scan.rs:285, 397` | `hash_batch` | (a) |
| `obikquery/query_layer.rs` stage 1 | `find_in_layer` per kmer, layer by layer, `break` on hit | (b) |
| `partition_merge.rs` `find_batch_grouped` | `find_slot` per kmer over dst layers | (b) |
| `partition_merge.rs` (`build_graph` filter) | `all(find_slot.is_none())`, every layer, per kmer; `build_graph` takes `Fn(CanonicalKmer)->bool` | (b), API change |
| `family_scan.rs` slow path; `annex.rs` | `find_slot` / `find_in_partition` per variant | (b) |
| `mphf_layer.rs` `build`, `build_exact_evidence`, `build_approx_evidence` | `mphf.index` then random writes (evidence, fingerprint, `seen`) | (b), chunked |
| `obikindexer/counter/count.rs:47` | `mphf.index` then `counts.get/set` | (b), hard (needs (kmer, count) pairs) |
| `obikindexer/extensions/private_builder.rs:177` | abundance filter | (b), conditional |
| `private_builder.rs:196`, `verify.rs`, `IndexCache::find_in_*` | per-kmer by nature / sampled | (c) |
| `tools/migrate-mphf/plan.rs:41-42` | `old.index` + `new.index` per kmer, random writes | (b) |
Dead or suspect code found along the way (candidates for removal, not removed):
- `MphfLayer::find_strict` — no caller.
- the single-kmer `hash` chain (`MphfLayer::hash` → `TypedLayer::hash` → `KmerLayer::hash` → `IndexCache::hash`) — no caller outside the chain.
- `IndexCache::find` — no caller.
- `obikmerge/src/merge_layer/` — not declared in `lib.rs` (`mod merge; mod partition_merge;` only); still imports the `LayeredMap` removed in 2026-08. To be confirmed by the compiler.
## 5. Interactions
- **Verification chain**: the right shape is one `find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>)` on `MphfLayer`, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops).
- **Memory**: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one `Vec<usize>` per call. `index_stream` itself only allocates a B-element ring on the stack.
- **Parallelism**: no new parallelism. Batches already come out of `obipipeline` stages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. In `query_partition_with` the unique-kmer `HashMap` is already materialised; iterate it chunk-wise.
## 6. Benchmark
`benchmark/bench_mphf.sh LABEL` times, on the 20-genome benchmark data, `index` (2 genomes), `merge` (20 presence indexes), `query` (presence/count × 2 genomes), `filter`, `convert` (approx/exact round trip) and `phylo` (`--sibling-annex`, `snp-jc`), `REPS` times each, and records order-independent output digests that must not change across labels. See the results section below.
**Limit of this data set**: 20 small genomes → per-partition `layer_1` MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it **cannot** measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).
### Throwaway prod-scale benchmark (not written)
Open one real partition's `layer_1` (~130 M keys) and `layer_0` (~8 M) with `MphfOnly`/`MphfLayer`; key sets: present (sample of `iter_kmers`, random order and sequential order) and absent (random, checked absent); measure kmers/s for `index`, `index_stream::<B>` (B ∈ 8…128), `index_batch::<K>` (K ∈ 8…64), and the full `find` vs staged `find_batch`; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of `mphf.bin`, `evidence.bin`, `unitigs.bin(.idx)` reported; first check `index_stream` slots == `index` slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in `mphf.md`) is a paper claim, not tied to our sizes or machines.
## 7. Not verified
No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and `--block-size` used in prod; whether `migrate-mphf` already ran; share of total index/merge/query time spent in these loops; `count.rs` f0 per partition at prod scale; whether `merge_layer/` is dead (only the head of `lib.rs` read).
## 8. First improvement and measurements (2026-10-15)
**Implemented** (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):
- `MphfLayer` now holds a `MphfOnly` instead of its own `MemCase<Mphf>` (one mmap-open path, one place for the stream call).
- `MphfOnly::for_each_slot(kmers, f)` wraps `index_stream::<32>`; `index_batch`/`hash_batch` now really use it (they previously looped `.index()`). Callback-shaped because `index_stream`'s `next()` is `unimplemented!`.
- `MphfLayer::find_batch` / `TypedLayer::find_slot_batch` / `KmerLayer::find_slot_batch`; the mode-dependent half of `find` is factored into `MphfLayer::confirm`, shared by both.
- `query_partition_with` stage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins, `n_mphf_calls` unchanged).
- Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive; `prefetch-index` is already in the tree via `ptr_hash`, but adding it to `obikindex` is a dependency decision), and the other (b) sites.
**Measurements** (`benchmark/bench_mphf.sh`, `benchmark/summarize_bench.py`; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:
| | wall (s) | user CPU (s) |
|---|---|---|
| before | 5.94–6.02 (median 5.96) | 10.95–11.07 |
| after | 5.63–6.00 (median 5.67) | 10.33–10.66 |
About −5 % wall and −5.5 % CPU, ranges non-overlapping **[measured]**. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.
**Reading the number.** Small by construction: this data set's `layer_1` MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains **unmeasured** and needs a prod-scale layer (§6).
**Noise caveat.** With a load average of 16–22 from unrelated processes, single-run walls of `index`, `merge` and `filter` moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The `index`/`merge`/`filter`/`convert`/`phylo` rows of the first comparison are noise, not effects (`index` pass 2 was not changed at all; a later interleaved `filter` A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).
**Stale benchmark indexes.** The indexes already under `benchmark/run/` (`specimen_index_*`, `global_index_*`, built 2026-08-28) carry `ptr_hash` 1.x MPHFs and fail with "Major version mismatch" on current binaries (`tools/migrate-mphf` converts them). `bench_mphf.sh` therefore builds its own copies under `run/bench_mphf/data/` on first use.
## 9. Benchmark procedure
Prerequisites: the simulated reads (`benchmark/run/simulated_data/`) and query reads (`benchmark/run/query_data/`) produced by `gmake simulate simulate_query`; macOS BSD `time -l` and `md5 -q` (the script uses both), Python 3.
1. Build the binary to measure and **copy it** under a name that identifies it (cargo overwrites `target/release/obikmer`): `cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>`.
2. `BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label>` (from `benchmark/`; `REPS=3` by default, `CPU_MAX=n` to restrict cores). The first call builds the shared data set in `run/bench_mphf/data/` with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise delete `data/` and rebuild.
3. Output: `run/bench_mphf/<label>/results.csv` (wall and peak RSS per step and repetition, plus one row `step:stage` per Reporter stage) and `checksums.txt`.
4. Compare: `python3 benchmark/summarize_bench.py <ref_label> <new_label>` (medians and ratio) and `diff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt` — **any difference is a bug**, not a performance matter.
5. A ratio is only a candidate effect. Check the spread by running the **same** binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from `/usr/bin/time -l`).
6. Rule of thumb for this data set: only `query` (≈ 5 s, stable to ±1 %) resolves effects of a few percent; `index`, `merge`, `filter`, `phylo` need interleaved runs or a quiet machine.
Step → code exercised:
| Step | Code on the MPHF path |
|---|---|
| `index_*` (`:index` stage) | `MphfLayer::build` pass 2 |
| `merge_20_presence` | `find_batch_grouped`, `build_graph` filter, `hash_batch`, `index_batch` |
| `query_*` | `query_partition_with` stage 1 |
| `filter_species` | `iter_layer_kmers` → `hash_batch` |
| `convert_to_*` | `build_exact_evidence`, `build_approx_evidence` |
| `phylo_sibling_annex` | `annex.rs` `find_in_partition` per variant |
| `phylo_snp_distance` | `family_scan.rs` `hash_batch`, `find_slot` |
## 10. Second step — construction pass 2 (2026-10-15)
**Implemented**: `layer/slotted.rs`, `SlottedKmers`, an iterator adapter yielding `(slot, kmer, chunk_id, rank)` for the wrapped `(kmer, chunk_id, rank)` stream, computing slots 4096 kmers at a time through `index_stream::<32>` (bounded memory, never the whole layer). The five loops of `MphfLayer::build` (Exact, Approx, Hybrid), `build_exact_evidence` and `build_approx_evidence` now iterate it instead of calling `mphf.index` per kmer. Output digests unchanged.
**Measured** (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):
| Command | before | after |
|---|---|---|
| `convert --exact-evidence` (20-genome index) | 0.56–0.68 s / 2.22–2.67 s | 0.59–0.74 s / 2.25–2.52 s |
| `index` E. coli reads | 2.45–2.63 s / 13.1–14.3 s | 2.48–2.75 s / 13.2–14.0 s |
**No measurable effect** on this data set (differences within the run-to-run spread; if anything `convert` is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and `index`/`convert` time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/`seen` writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random *writes* to `evidence.bin`/`seen`/`fingerprint.bin` (not prefetched by `ptr_hash`) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.
## 11. Why the two changes gain so little (profiling, 2026-10-15)
Measured with macOS `sample` (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.
**Query.** In a 2 s window on the `query` stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):
- `query_partition_with` ≈ 260 samples (8 % of active CPU), of which `find_batch` 217, of which **`verify_canonical_kmer` 185** and everything else in `find_batch` (the `ptr_hash` stream and `confirm`) **32**.
- `verify_canonical_kmer` is the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.
- Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/`unitigs.bin`/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side.
- The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (`rolling_stat`, entropy tracker, string formatting) and by sequential parts, not by lookups.
**What produced the −5 %.** Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer `index` (stream disabled) 6.0 s; restructuring + `index_stream` 5.85 s. **Most of the gain comes from the restructuring, not from the prefetch** (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called `IndexCache::get_layer` — an `Arc<KmerLayer>` clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.
**Index (pass 2).** In a 1 s window late in the `index` stage, `SlottedKmers` appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (`obidebruinj`) and ≈ 35 for `ptr_hash::build` (pass 1, MPHF *construction*). Pass 2 is a few percent of `index` and `convert` here: nothing to win even if its lookups were free. `convert` is `build_*_evidence` on 2 MB-scale layers, same conclusion.
**Consequences for priorities.**
1. On this data the next lever is the verification (`verify_canonical_kmer`, evidence decode), not the MPHF — consistent with §3, now observed.
2. Hot `Arc` clones in per-kmer paths (`get_layer`, `IndexCache::find_in_layer`, `find_slot` loops in merge and phylo) are worth auditing independently of MPHF batching.
3. The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).
## 12. Calibration: a single skim at fewer partitions (2026-10-15)
`Vaccinium myrtillus` skim (2 × ~440 MB fastq.gz, `theta 0`), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (`--partitions` is a power of two; MPHF size ≈ 2.4 bits/key **[measured]**):
| `-p` | keys / partition | `mphf.bin` / partition | index wall | peak RSS |
|---|---|---|---|---|
| 256 (default) | ≈ 1.4 M **[computed from p=16]** | ≈ 0.4 MB **[computed]** | – | – |
| 16 | 22.1 M | 6.65 MB | 120 s | 12.3 GB |
| 4 | 86–90 M | 25.9–27.0 MB | 171 s | 19.5 GB |
Total ≈ 354 M distinct kmers for one skim. `-p 4` is already in the 30–45 MB range of the prod `layer_1` MPHFs; `-p 2` would give ≈ 53 MB. Indexes kept in `benchmark/run/bench_mphf/skim_p16` and `skim_p4` (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.
## 13. Query on a real-scale index (2026-10-15)
**Index.** Working index `phyloskims_sal_vac_sav` (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with `tools/migrate-mphf` **on an APFS clone** (`cp -c`, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at `/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`; its sibling annex is in `stale_annexes_pre_ptrhash2/` and would need rebuilding to be used. Per partition: `layer_1` ≈ 31 M keys / 9.3 MB MPHF, `layer_0` ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB).
**Query** (2 × 48 MB of *V. oxycoccos* reads, `--chunk-size 8`, AC power, 3 interleaved runs per binary, identical sorted output for all three):
| binary | wall (s) | user CPU (s) | `query` stage (s) |
|---|---|---|---|
| original | 28.2–30.5 | 70.9–74.4 | 14.6–17.1 |
| + batched query stage 1 | 26.9–28.0 | 48.1–48.6 | 13.6–14.4 |
| + `PartitionKmers` memory fix, build-side `SlottedKmers` | 26.4–26.7 | 41.9–42.5 | 12.9–13.3 |
The `query` stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. **The CPU drop is much larger than the wall drop** and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer `Arc` clones of `get_layer`, not isolated); the original binary was spending CPU in contention that wall time hid.
**Index-open cost (not related to the changes above).** A query of a single 64-base sequence on this index takes **15 s and a 27 GB peak footprint**. `vmmap` during the open: 24.5 GB of `MALLOC_LARGE` in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by `IndexCache::new` for every layer of every partition whatever the query. Known contributors: `UnitigFileReader` reads each `.idx` into a `Vec` (≈ 30 MB for `layer_1`, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied.
## 14. Index-open memory: `.idx` offsets and `dict_values` mapped, not copied (2026-10-15)
**Cause** (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, **18.2 GB** were `unitigs.bin.idx` read into a `Vec<u32>` **twice** per exact-evidence layer (`MphfLayer::open` builds two `UnitigFileReader`s of the same file — one for the evidence check, one for `iter_kmers`/`find_strict`; both go through `open` → `open_direct_access` → `read_idx` → `fs::read` + copy) and **7.7 GB** were `dict_values.bin` read with `fs::read` in `PersistentSparseBitMatrix::open`. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (`PersistentRankSelectBitVec`, `PersistentFixedIntVec`, `EliasFano`) was already mmapped.
**Change.**
- `obiskio::unitig_index::reader`: `block_offsets: Vec<u32>` → `BlockOffsets`, a view into the memory-mapped `.idx` (header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages.
- `obicompactvec::bitmatrix::sparse`: `dict_values: Vec<u8>` → `Option<Mmap>` (`None` for an empty file, which cannot be mapped), read through `dict_bytes()`.
**Measured** (same session, same inputs; sorted query outputs identical, md5 `e9e7c35b`):
| | open only (1-sequence query) | 2×48 MB reads, `--chunk-size 8` |
|---|---|---|
| before | 10.6–15.7 s, **27.4 GB** footprint | 27.7 s (51 s cold), 30.9 GB, `query` stage 13.8 s |
| after | **0.26–0.81 s, 0.04 GB** | **10.7 s, 3.6 GB**, `query` stage 9.95 s |
Tests: `cargo test --workspace` green; `scripts/smoke_test_index.sh` PASS; `bench_mphf.sh` (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a `.idx` map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect.
## 15. Unbuffered reads: `index.meta` and superkmer files (2026-10-15)
Follow-up of §14 (search for the same family of problems: file content handled inefficiently at open). Static search over `src/` for `fs::read`, `read_to_end`, `load_full`, `from_reader`, bare `File` readers and mmap `to_vec`: every persistent structure of `obicompactvec`, `obikindex` and the `obikphylo` annexes is already mmapped; the two real findings were *unbuffered* readers.
**1. `IndexMeta::read_full` (`obikindex/src/index/meta.rs`)** passed a bare `File` to `serde_json::from_reader`, which issues **one `read` syscall per byte**. `index.meta` of the 91-genome index is 20 KB and is read afresh by every `state()`/`genomes()` call (stateless design, §obikindex audit), so each call cost ≈ 0.35 s here (macOS). Profile of `obikmer dump --head 10` (`sample`): ≈ 20 000 samples in `read`, called from `Bytes<File>::next` ← `serde_json` ← `IndexMeta`. Fix: `fs::read` then `serde_json::from_slice`. **`dump --head 10`: 9.0–10.0 s (91–102 s of system CPU) → 0.16–0.84 s**; the same change applies to `SKFileMeta::read` (`obiskio/src/meta.rs`, per-superkmer-file sidecars).
**2. `SKFileReader::open` (`obiskio/src/reader.rs`)**: for an uncompressed file `niffler::get_reader` returns the sniffed bytes chained to the bare `File`, and `read_superkmer` reads each field with its own small `read_exact`. Fix: wrap the file in a 64 KiB `BufReader` before `niffler`. Effect on `index` (E. coli, 4 interleaved runs each): `count_kmer` 0.51–0.56 s → 0.42–0.50 s (≈ −12 %), total wall unchanged (≈ 1.95 s). Small, low risk, kept.
Tests: `cargo test --workspace` 549 passed / 0 failed; `scripts/smoke_test_index.sh` PASS; query output on the 91-genome index identical (md5 `e9e7c35b`).
**Probes of other commands on the 91-genome index (180 GB, 256 partitions)**: `phylo --distance jaccard` peaks at 155–190 MB of footprint while running and finishes in < 30 s; `utils --stats` 130–150 MB. `filter` (and by the same check `select`, `merge`) refused the index — "source index is not fully built" — with the unmodified baseline binary too: this index's `index.meta` predates the `state` field, so they could not be probed.
**Observations not investigated**: `index` on E. coli shows ≈ 9 s of *system* CPU in 2 s of wall time (both before and after these changes), consistent with the `swtch_pri` (yield) spinning seen in the `index` profile of §11 rather than with I/O; and wall times of the same query on the same binary varied 4.8 s ↔ 10.7 s ↔ 43 s depending on the state of the page cache, so only same-session comparisons are meaningful.
## 16. Profile of `index` on E. coli reads (2026-10-15)
`sample` (1 ms stacks) of `obikmer index` on the E. coli K-12 simulated reads (2 × 15x, 16 threads, one run; the `index` stage lasted 2.4 s under the sampler vs ≈ 0.9 s normally, so proportions are approximate). Stage walls: scatter 0.40 s, dereplicate 0.14, count_kmer 0.60, index 2.36.
**Threads are mostly idle.** Of ≈ 30 800 thread-samples: 24 900 blocked or yielding, 5 900 active. The ≈ 9 s of *system* CPU per run (§15) is `swtch_pri` (`sched_yield`) — 4 550 samples: ≈ 70 % rayon workers spinning for work (`WorkerThread` wait loop), ≈ 28 % `crossbeam` bounded-channel send/recv spinning, the rest in `obidebruinj` thread-locals. It is waiting, not I/O: no `read` samples remain. It costs no wall time on an otherwise idle machine, but burns CPU (visible in `user + sys` accounting and on a shared 192-core machine).
**Where the active CPU goes** (share of the 5 900 active samples):
| | share |
|---|---|
| xxh3 hashing of `CanonicalKmer` for the de Bruijn graph's `HashMap` (callers: `count_neighbors` 791, `hash_one` 358, `walk` 285, `unitig_iter` 252, `push` 164, `unitig_nucleotides` 75, rehash 68; plus `xxh3_stateful_update` 300) | ≈ 34 % |
| `debruijn::count_neighbors` itself (inclusive of its hashing ≈ 2 300 samples: `compute_degrees_and_mark_starts`) | 12.5 % own |
| `ptr_hash::build` (MPHF construction) | 4.8 % |
| crossbeam channel send | 4.8 % |
| `open` / `close` / `write` (many small files over 256 partitions) | ≈ 9 % |
| debruijn `unitig_iter` / `walk` own time | 8.3 % |
So at this scale the cost of `index` is the **de Bruijn graph's hash table** (several lookups of neighbouring k-mers per k-mer, each hashed with xxh3 and probably missing the cache), not the MPHF passes this document started from (`SlottedKmers` ≈ 2 %, §11). Not measured: what fraction of those lookups miss the cache, and the same profile on a prod-size partition.
## 17. Profile of `index` on a skim: *Vaccinium vitis-idaea*, 2.1 GB, `--partitions 16` (2026-10-15)
One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slows the process; walls below include it): scatter 19.7 s, dereplicate 8.0 s, count_kmer 25.2 s, **index 139.1 s**; total 192 s real, 1830 s user, **225 s system** (12 % of user, against ≈ 90 % on E. coli, §16), peak footprint 17.8 GB.
**Index stage** (samples taken fully inside it, 91 000 active vs 187 000 idle/yielding thread-samples):
| share of active CPU | |
|---|---|
| xxh3 hashing of kmers (de Bruijn `HashMap`) | 32.7 % |
| `debruijn::walk` | 12.4 % |
| `debruijn::count_neighbors` | 10.7 % |
| `GraphDeBruijn::push` | 6.9 % |
| `crossbeam` channel | 6.6 % |
| `hashbrown` table operations | 5.0 % |
| `debruijn::unitig_iter` | 4.8 % |
| memcpy/memset, malloc/free, file I/O, SipHash | ≈ 7 % |
| other | 13.7 % |
- **The de Bruijn graph is ≈ 72 % of the active CPU of `index`; the MPHF construction does not appear (< 1 %)**, and `SlottedKmers`/evidence filling is negligible. The conclusion of §16 holds at 100× the E. coli scale.
- **The stage is poorly parallel**: on average ≈ 5 of 16 threads are active (33 % of thread-samples); `swtch_pri` (spin/yield) is 7 % of all thread-samples. With 16 partitions of unequal size the tail partitions run alone. The 12 % system CPU is the same spinning as §16, at a smaller relative weight.
- **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.
Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture.
## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)
Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench <unitigs.bin> <k> [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`.
| | push | degrees | unitigs | unitigs produced | nucleotides |
|---|---|---|---|---|---|
| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M |
| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M |
| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** |
Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).
**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision).
**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).
**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig).
Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).
**End-to-end** (current binary vs previous one, same inputs):
| | before | after |
|---|---|---|
| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) |
| … `index` stage | 114.6 s | **59.2 s** (−48 %) |
| … user CPU | 1 870 s | 1 304 s |
| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB |
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
## 19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15)
`Evidence` packs `(chunk_id, rank)` into a `u32` with a 7-bit rank (`encode` masks `& 0x7F`), but `MAX_KMERS_PER_CHUNK` was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: **`--partitions 256`** → max chunk 49–50 k-mers, results correct with the old and the new binary; **`--partitions 1`** → max chunk 256, query output wrong with both the old (md5 `bde0f886a0`) and the post-graph-fix (`8f1c94ef32`) binary versus the correct `f99e971087`. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts.
**Change** (`obikseq::MAX_KMERS_PER_CHUNK`: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (`SuperKmer::write_to_binary`): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test `chunk_ranks_fit_the_7_bit_evidence_field`.
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
## 20. Summary of what was gained (2026-10-15)
Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated.
### Speed
| Operation | Original | Now | Where |
|---|---|---|---|
| Open an index (query of one 64-base sequence), 91 genomes, 180 GB | 10.4–13.3 s | **0.18–0.19 s** | §14, same-session A/B |
| Query, 2 × 48 MB reads on that index (`--chunk-size 8`) | 30.7–32.0 s (61.8 s cold), 70–76 s CPU | **11.6–13.2 s**, 36–39 s CPU (−62 % wall, −50 % CPU) | §14, same-session A/B |
| `dump --head 10` on that index | 9.0–10.0 s (91–102 s system CPU) | **0.16–0.84 s** | §15 |
| `index` of a 2.1 GB skim (*V. vitis-idaea*, `--partitions 16`) | 163.8 s (stage `index` 114.6 s) | **116.5 s** (stage `index` 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU | §18 |
| `index` of E. coli reads | 1.97 s (stage 0.91 s), 10.2 s CPU | 1.63 s (stage 0.59 s), 7.3 s CPU | §18 |
| De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) | 4.1 s / 2.0 s / 3.9 s | 1.0 s / 0.8 s / 1.0 s (×3.5 overall) | §18 |
| `index` stage `count_kmer` (E. coli) | 0.51–0.56 s | 0.42–0.50 s | §15 |
| Query on a small input (2 × 48 MB reads), single-genome index | 11.4 s (2 chunks = 2 busy cores) | **2.4 s** (−79 %) — automatic chunk size now follows the input size (`architecture/query.md`); on the 91-genome index the size was already 4 MiB | §20 follow-up |
### Memory (peak RSS / footprint)
| Operation | Original | Now |
|---|---|---|
| Opening the 91-genome index (any command that builds an `IndexCache`) | **27.4 GB** | **0.04 GB** (§14: `.idx` offsets and `dict_values.bin` mapped instead of copied; they were read twice and once) |
| Query, 2 × 48 MB reads on that index | 32.2–32.5 GB | **3.5–3.6 GB** |
| Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size | **73.7–80.7 GB** | **28.2 GB** (§12 of `architecture/query.md`: `PartitionKmers` instead of a `HashMap<kmer, Vec>` per partition: 71 → ~20 bytes per unique kmer) |
| `index` of the 2.1 GB skim | 19.7 GB | 18.3 GB |
### Index size (newly built indexes; existing ones are unchanged)
| | Original | Now |
|---|---|---|
| `unitigs.bin` / `unitigs.bin.idx` (2.1 GB skim, 521.6 M kmers, 16 partitions) | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18) |
| Whole index of that skim | 3.5 GB | **2.9 GB** (−17 %) |
| `evidence.bin` | 4 bytes per kmer | unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented) |
### Correctness fixes found on the way
- Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18).
- Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, `scripts/check_unitig_chunks.py` to audit an index).
- `index.meta` and super-kmer files were parsed byte by byte (§15).
### What is **not** established
- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone).
- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).
## 21. Dead code removed (2026-10-15)
The items listed as unused in §4 were removed after Serena's `safe_delete_symbol` and the compiler confirmed no reference: `MphfLayer::find_strict`; the single-kmer `hash` chain (`MphfLayer::hash`, `TypedLayer::hash`, `KmerLayer::hash`, `IndexCache::hash`); `IndexCache::find`; the uncompiled `obikmerge/src/merge_layer/` directory (689 lines; `lib.rs` only declares `merge` and `partition_merge`); and `xxhash-rust` from `obidebruinj/Cargo.toml` (it still serves `obikseq`, `obifastwrite` and `obiksession`).
Side effect: `find_strict` was the only reader of the `unitigs` handle that `Hybrid` layers opened, so that field (and the extra `.idx` map and mmap per Hybrid layer) was dropped too. `find` on a Hybrid layer was already a fingerprint check; the exact evidence stays on disk (it gives the slot count and is what `convert` works from) but no API consults it any more. Restoring an exact lookup for Hybrid layers means re-adding `find_strict` and that field.
`ahash` was not referenced by any source file of `obidebruinj` either and was removed from its `Cargo.toml` too.
Checks after the removal: `cargo build --release` without warnings, `cargo test --workspace` 555 passed / 0 failed, `scripts/smoke_test_index.sh` PASS.
Scratch artifacts of this work were deleted: the intermediate binaries, the two skim indexes (`skim_p4`, `skim_p16`) and the shared data set `benchmark/run/bench_mphf/data` (rebuilt automatically, ≈ 2 min, by the next `bench_mphf.sh`). Kept in `benchmark/run/bench_mphf/`: the original binary `obikmer_baseline` (reference for A/B runs) and the per-label results and checksums (18 MB in all). The 91-genome index converted to the current MPHF format (`/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`, with its `index.meta` patched, original `index.meta.before_state_fix` beside it) was left in place.
+1
View File
@@ -509,6 +509,7 @@ Provided finalisations:
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` | | `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` | | `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial | | `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
| `threshold_hamming_dist_matrix(t)` | `union[i,j] − inter[i,j]` (k-mers present in exactly one genome) — no separate partial |
### BitPartials ### BitPartials
+4 -4
View File
@@ -22,10 +22,10 @@ A sequence of binary records. Each record:
- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded. - Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
- Byte count for packed sequence: `ceil(seql / 4)`. - Byte count for packed sequence: `ceil(seql / 4)`.
Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost: Unitigs with more than `MAX_KMERS_PER_CHUNK = 128` k-mers are transparently split into overlapping chunks. Each chunk has at most 128 k-mers (= `seql − k + 1 ≤ 128`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
``` ```
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (256 kmers) chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (128 kmers)
chunk 2: nucleotides [256, end] (remaining kmers) chunk 2: nucleotides [256, end] (remaining kmers)
overlap: k−1 nucleotides shared between the two chunks overlap: k−1 nucleotides shared between the two chunks
``` ```
@@ -95,11 +95,11 @@ Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extract
| field | bits | range | capacity check (*B. nana*, 256 partitions) | | field | bits | range | capacity check (*B. nana*, 256 partitions) |
|------------|------|---------------|---------------------------------------------| |------------|------|---------------|---------------------------------------------|
| `seql − k` | 8 | 0–255 | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` | | `seql − k` | 8 | 0–127 used | max `n_kmers` per chunk = 128 = `MAX_KMERS_PER_CHUNK` (the field could carry 256; capped so that every rank fits 7 bits) |
| `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 | | `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
| `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition | | `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition |
The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total. The rank field is 7 bits (max 127) and the rank of a k-mer is its index among the chunk's k-mers, so a chunk must hold at most 128 of them: that is what `MAX_KMERS_PER_CHUNK = 128` guarantees. (Until 2026-10-15 the cap was 256 and the earlier justification here — that rank counts only canonical k-mers, at most half the total — was wrong: `iter_indexed_canonical_kmers` yields one entry per position. Chunks of 129–256 k-mers occur with few partitions — 6 of them in a `--partitions 4` index of a 2 GB skim, none with 256 partitions where the observed maximum is ≈ 50 — and their k-mers beyond rank 127 were silently stored with a truncated rank. `scripts/check_unitig_chunks.py` finds such chunks in an existing index.)
--- ---
+2 -2
View File
@@ -13,11 +13,11 @@
| `query` | Query an index with sequences and annotate matches | | `query` | Query an index with sequences and annotate matches |
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers | | `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV | | `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
| `phylo` | Compute pairwise evolutionary-distance proxies between genomes (`--metric jaccard\|mash\|hamming\|bray-curtis\|relfreq-bray-curtis\|euclidean\|relfreq-euclidean\|hellinger\|hellinger-euclidean`); optionally build NJ/UPGMA trees; `--presence-threshold N` sets the minimum count to consider a k-mer present when computing Jaccard/Mash on count indexes (default 1); optionally a central-position SNP/Sankoff calibration with exports for TNT/PhyG/IQ-TREE (see [evolutionary distances](theory/evolutionary_distances.md)) | | `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash/Hamming on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system | | `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) | | `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing | | `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
| `reindex` | Convert an index's evidence in-place: exact ↔ approx | | `convert` | Convert an index's evidence representation (exact / approximate / hybrid), in place |
| `utils` | Miscellaneous index utilities: `--new-label NEW=OLD` renames a genome label; `--upgrade-index` adds missing `layer_meta.json` to old indexes | | `utils` | Miscellaneous index utilities: `--new-label NEW=OLD` renames a genome label; `--upgrade-index` adds missing `layer_meta.json` to old indexes |
| `pack` | Pack per-column matrix files into single-file format to reduce query I/O | | `pack` | Pack per-column matrix files into single-file format to reduce query I/O |
+84
View File
@@ -2873,6 +2873,90 @@ user, no external consumers to break). Scoped to the distance matrix only:
`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep `--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
their existing CSV-only format. their existing CSV-only format.
## VCF export of the pseudo-alignment
`phylo --vcf` writes the drawn sites of `--pseudo-alignment` as a VCF
(`--fasta`/`--vcf` choose the formats, FASTA is the default, both can be
combined). A family has no genomic coordinate, so coordinates are virtual and
derived from where the scan finds the family: `CHROM = obikmer_<partition>_<layer>`,
`POS = family_idx + 1` (`scan_layer_families`' iteration-order index inside
that layer). Stable for a given index + annex, sorted by construction because
`on_family` is replayed in `family_idx` order. A global dense rank or a bit
packing of (layer, partition, idx) into one `POS` was rejected: it needs
per-layer minorant counts and can exceed VCF's 2^31-1 bound.
`ID` is `CanonicalKmer::family_id()`: centre forced to `A` on the
k-mer and on its revcomp, each canonicalised, minimum of the two (centre
printed as `N`). Index-independent, hence the key for merging VCFs from
different indexes. No `INFO/LOCUS` duplicate (dropped: same string as `ID`, costly
at 5M loci); `QUAL`/`FILTER`/`INFO` are all `.`. An alternative defined as the minimum of
`central_canonical_neighbors()` was dropped: it canonicalises 4 k-mers where
this one needs 2, with no other benefit. `family_id` idempotence and
family-wide equality are tested exhaustively for k = 3, 5, 7.
`SurvivingFamily` now carries the minorant k-mer (the scan callback passes it),
and `SnpAlignment` a `SiteLocus` per column. Alleles are the unambiguous cells
of the IUPAC matrix, in the FASTA's convention (central base of the member's
canonical form, not necessarily the `ID` k-mer's strand).
Absence handling. `masked_state` keeps absence (`m == 0`, `-`) as a real state
unless `--free-loss`, where it becomes missing (`?`). A first version mapped
`-` and `?` both to `.`: on a toy 4-genome index half of the lines then had
`ALT = .` (presence/absence-only sites, selected by `survives_masking`), i.e.
polymorphic sites rendered as monomorphic. Absence became an extra allele,
first written `*`; that was a misuse: in VCF 4.5 (section on the ALT column)
`*` is "allele missing due to overlapping deletion", and `<*>` is the gVCF
unspecified allele, `DEL`/`INS`/`DUP`/`INV`/`CNV` are structural-variant
types. Now the symbolic allele `<ABS>` (free-form ID, declared by a
`##ALT=<ID=ABS,...>` line, always written in the header, listed after the
bases); `?` and IUPAC ambiguity codes stay `.` (haploid GT cannot express a
genome carrying several forms; superseded below). A symbolic ALT formally
asks for a padding base before `POS`; moot here, positions being virtual.
Checked with `bcftools view`/`stats` and `tabix`: no warning, the sites with
`<ABS>` are classed "others" by `bcftools stats`, and
`-e 'ALT="<ABS>"'` drops exactly them. The writer needs no knowledge of
`--free-loss`: `-` simply never occurs with it. A reference-genome option
(`--vcf-ref`) was implemented then dropped: `REF` is always the per-site
majority base (alphabetical on ties).
Ploidy. VCF allows any ploidy in `GT` (number of alleles written, variable
per sample/site), but an obikmer cell only says *which* forms a genome
carries, not their multiplicity (`A/A/G` and `A/G/G` are both `R`), nor
whether several forms mean heterozygosity or paralogues. Compromise chosen:
`--no-ambiguity` => haploid (ambiguous cells are already `?`); otherwise
diploid, single base `X` -> `X/X`, two-base code -> `X/Y`, `-` -> `<ABS>/<ABS>`, and
three/four-base codes or `?` -> `./.`. `REF` is the majority allele counting
each copy once (so a het contributes 1+1, a homozygote 2). Counting copies
for a real dosage (polyploids) would need k-mer counts (`--with-counts`),
not used by the pseudo-alignment scan today. `--only-diploid` (selection option, `conflicts_with --no-ambiguity`) adds a
third `masked_state` rule: `count_ones() > 2` -> missing. It lives in
`SamplingSpec`/`masked_state`, i.e. the post-hoc masking and
`survives_masking`, so FASTA, Sankoff and `snp-*` see it too. Unit-tested
(`masked_state`); the toy index has no 3/4-member cells, so the end-to-end
run is identical to the default mode.
First version of this flag read the persisted annexes under the default
convention (a third variant was judged not worth it). Rejected: the draw
would then describe other genomes than the alignment, and the pair-balance
weights and entropy are computed from the carrier set. Replaced by
`AmbiguityPolicy { Keep, Forbid, OnlyDiploid }` (one type instead of
`free_loss`/`no_ambiguity`/`only_diploid` booleans in `masked_state`,
`is_carrier`, `SamplingSpec` and every annex key). Annex headers store
`AmbiguityPolicy::header_flags` (0 = Keep, 1 = Forbid, as the old bit, 2 =
OnlyDiploid), so annexes written before the change stay valid; each policy
has its own file (`*.diploid.*` for the new one). `--sibling-annex` no longer
precomputes everything (`AnnexRequest::everything` removed, a rebuild only
costs the base annex): it builds the derived annexes named by the other
options of the command (`AnnexSelection`: `--entropy`, `--quartet`,
`--pair-balance`, `--family-overlap`/`--min-shared-family`, under the one
policy given); the rest is built on demand as before. A rebuild still drops
every derived annex (their `family_idx` numbering is tied to the base one).
Checked on the toy index: `--sibling-annex` alone leaves only
`siblings.psib`; `--quartet --only-diploid` adds `quartet.diploid.pbiv` only.
Toy index: default mode 187
lines, no `ALT = .`; `--no-ambiguity` 93 lines (the sites whose polymorphism
was only ambiguity codes disappear, since those cells became `?`).
## References ## References
The Mash mutation-rate model this discussion contrasts with: The Mash mutation-rate model this discussion contrasts with:
+84 -2
View File
@@ -1,4 +1,4 @@
# Pair-coverage-biased family sampling (discussion, 2026-10-07) &@")--# Pair-coverage-biased family sampling (discussion, 2026-10-07)
Idea: bias the family sample used by `--pseudo-alignment`/`--sankoff`/`snp-*` so that the `family_overlap` matrix of the *sample* is as uniform as possible, compensating the scarcity of shared families between distant species. Complements, does not replace, the existing proportional-by-partition and entropy-biased draws (see user doc, *Sampling design*). Idea: bias the family sample used by `--pseudo-alignment`/`--sankoff`/`snp-*` so that the `family_overlap` matrix of the *sample* is as uniform as possible, compensating the scarcity of shared families between distant species. Complements, does not replace, the existing proportional-by-partition and entropy-biased draws (see user doc, *Sampling design*).
@@ -226,7 +226,9 @@ Design decision (user): the way genomes are counted depends on `--no-ambiguity`,
## Implemented: `--quartet` (2026-10-13) ## Implemented: `--quartet` (2026-10-13)
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Absence never counts as a state; with `--no-ambiguity`, multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; the `--no-ambiguity` convention is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`) because that header has no room for a flag. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one. `--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Genomes are read through `masked_state`, exactly as in the alignment: absence is a state of its own unless `--free-loss` (then it is missing and counts for nothing), and with `--no-ambiguity` multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; its counting convention (`QuartetConvention`: ambiguity policy × `--free-loss`) is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`, `quartet.diploid.pbiv` for `--free-loss`; `quartet.abs.*.pbiv` for absence-as-a-state) because that header has no room for a flag.
*Correction (2026-11-19).* The first version never counted absence, whatever `--free-loss` said, which contradicted the alignment: without `--free-loss` a genome lacking the family has the state `∅`, so `A A ∅ ∅` separates two pairs yet was rejected. Fixed in `has_bipartition` (index 0 is now counted, and is only ever populated when absence is a state), in the annex (one file per `QuartetConvention`) and in `--sample-stats` (informative sites follow the same rule). The old file names keep the `--free-loss` semantics, which is exactly the old rule, so annexes already built with `--free-loss` stay valid; the sampling measured below used `--free-loss` and is unaffected. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list. Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list.
@@ -278,6 +280,86 @@ The weight flattens the representation (slope toward 0) and, above all, covers m
Reading: `--quartet` is the decisive option here — a uniform draw of 100-300 sites contains almost no informative column (0.5-2), the restricted draw contains only informative ones. `--pair-balance` alone barely helps (it favours rare pairs but does not remove the uninformative columns); added to `--quartet` it raises the share of replicates recovering all three species (50→53 % and 33→67 %). The entropy kernel (centre 1 bit, default width) adds nothing at 300 sites and costs at 100. Limits: 11 genomes, three simulated 1x datasets, small budgets, a single criterion (species monophyly, not deep branches), and the differences between the `--quartet` variants (a few runs out of 30) are small relative to the replicate noise — a direction, not a ranking. Reading: `--quartet` is the decisive option here — a uniform draw of 100-300 sites contains almost no informative column (0.5-2), the restricted draw contains only informative ones. `--pair-balance` alone barely helps (it favours rare pairs but does not remove the uninformative columns); added to `--quartet` it raises the share of replicates recovering all three species (50→53 % and 33→67 %). The entropy kernel (centre 1 bit, default width) adds nothing at 300 sites and costs at 100. Limits: 11 genomes, three simulated 1x datasets, small budgets, a single criterion (species monophyly, not deep branches), and the differences between the `--quartet` variants (a few runs out of 30) are small relative to the replicate noise — a direction, not a ranking.
## Implemented: no distance matrix unless one is wanted (2026-10-13)
`--distance` had a default (`jaccard`) and `phylo` always computed and wrote a matrix, even for `--sibling-annex`, `--family-overlap` or `--pseudo-alignment` alone. `--distance` is now optional: a matrix is computed only if `--distance` is given, or `--nj`/`--upgma`/`--shared-kmers` needs one (then `jaccard`, `args::distance_to_compute`, unit-tested); `--csv` alone changes nothing. When nothing is wanted the command stops before the distance step. Checked on the 1x testbed: `--family-overlap` and `--pseudo-alignment` runs write no `_dist` file and run no distance stage; `--nj` writes `_dist.phy` and `_nj.nwk`; `--distance mash --csv` writes `_dist.csv`. Cost saved on a large index: not measured.
## Implemented: diagnostic outputs (2026-10-13)
Added so the measurements this file reports no longer need ad hoc scripts, and so a draw can be diagnosed from the files of the run that produced it:
- `--sample-stats`: a `SampleStatsReducer` (`algorithms/sample_stats.rs`), a `LayerReducer` run beside the alignment over the same surviving families; writes `_sample_genomes.csv`, `_sample_carriers.csv`, `_sample_overlap.csv`. A call is a masked state other than `0`; informative = `has_bipartition` (the quartet rule, now shared). Checked on the 1x testbed: per-genome calls and the overlap matrix equal those recomputed from the alignment.
- `--sample-plan`: `sampled_scan` now returns a `SamplePlan` (one `LayerReport` per layer: eligible, mass, quota, resolved, kept); `_sample_plan.csv` has the layer lines only (quotas sum to the target by cumulative rounding; on the testbed 3000 asked, 2695 kept).
- `--shannon-summary`: `--shannon` and the summary share one traversal (`scan_family_entropies`, which replaces the loop that lived in `shannon_entropy_csv`); the summary keeps counts per (entropy kind, genomes present, genomes without ambiguity, 0.05-bit bin) instead of one line per family: 156 lines for the 46 706 families of the testbed (a few hundred thousand lines at most on any index, by construction: the bins are bounded). Checked: its `entropy15` counts sum to the number of variable families.
- Writers are in `cmd/phylo/stats.rs`; the library computes, the command layer formats and writes.
## Combined draw on Primulaceae: `--quartet --pair-balance --entropy 1` (measured 2026-10-13)
Command: `phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance -o primulaceae_quartet_balance --family-overlap --sibling-stats primulaceae`. One draw, 180 genomes, 5 000 000 columns requested and obtained (no shortfall; no `--sample-plan`, so per-layer quotas are not inspectable). The three options are on together: **effects of entropy, quartet and pair balance are not separable here**; the testbed result that the entropy kernel adds nothing at small budgets was not rechecked.
**Baseline.** The same statistics were recomputed with one script on the alignment fed to IQ-TREE (`primulaceae-gamma_alignment.fasta`, 5 003 414 columns). It reproduces the figures of the earlier sections (sample overlap min 4 / median 1 665 / max 387 057 / CV 1.57; sampling fraction $r$ quantiles 0.0025 / 0.0037 / 0.0055; 5.2 % median calls per genome), hence **is the entropy-drawn sample** (`--entropy 1 --entropy-sd 0.5`, section on entropy bias), not a uniform draw. The "uniform draw" label of the nesting section is inconsistent with this (to settle). So the comparison below is *entropy-only vs entropy + quartet + pair balance*, never against a uniform control. A call is any character other than `?`/`-`. With `--free-loss` absence is written `?` (`masking::masked_state`); `-` only occurs without it. Neither file contains IUPAC characters.
| | Baseline (entropy only) | Combined |
|---|---|---|
| Informative columns (≥ 2 bases, each in ≥ 2 genomes) | 21.4 % | 100 % |
| Carriers per column: median / share ≤ 4 / share ≥ 33 | 4 / 58.1 % / 8.0 % | 8 / 12.4 % / 11.3 % |
| Calls per genome, median; genomes < 1 % / < 2 % | 5.2 %; 8 / 23 | 8.1 %; 4 / 6 |
| Sample pair overlap min / median / max | 4 / 1 665 / 387 057 | 282 / 27 380 / 491 038 |
| Sample overlap CV; max/min | 1.57; 96 764 | 1.10; 1 741 |
| 4-way count (20 000 random quartets, 2 seeds): median | 200-202 | 3 738-3 797 |
| 4-way p1 / p10 | 3 / 66-67 | 93-95 / 1 092-1 104 |
| 4-way p90 / p99 | 3 000-3 644 / 65 700-67 300 | 14 500-15 600 / 82 900-84 600 |
| 4-way CV; share of quartets with 0 | 3.4-3.5; 0.11-0.14 % | 1.83-1.87; 0 % |
(p90/p99 move by 20 % between the two seeds: read them as orders of magnitude only.) The gain is at the bottom of the 4-way distribution (median ×19, p1 ×30); the top is unchanged. 4-way coverage is not stratified by clade depth here (needs the tree). "Informative" is measured over the 180 genomes: a column informative overall is not necessarily informative among the four genomes of a given quartet.
**Pair representation.** Index-wide overlap used for the comparison: baseline, the raw matrix (`primulaceae_phylo_family_overlap.csv`); combined, the `--no-ambiguity` matrix of the same run. Different conventions, so the two slopes are indicative.
| | Baseline | Combined |
|---|---|---|
| Slope of log(sample overlap) on log(index overlap) | 1.12 ($R^2$ 0.997) | 0.54 ($R^2$ 0.91) |
| Pairs with index overlap > 0 and no sample column | 0 | 0 |
| Median $r$ = sample/index, poorest decile of index overlap | 0.0027 | 0.059 |
| Median $r$, richest decile | 0.0053 | 0.0067 |
| Median $r$ by decile 1 → 10 | 0.0027 … 0.0053, rising | 0.059, 0.065, 0.067, 0.067, 0.069, 0.074, 0.023, 0.0082, 0.0077, 0.0067 |
The weight does not merely flatten: the poorest pairs are now about 9× better represented (relative to their pool) than the richest, where the baseline gave them about half. Deciles 1-6 are flat; the drop sits at deciles 7-10, around the gap of the index-overlap distribution already noted in the entropy-bias section (cause of that gap still unchecked). **Caveat on $r$:** the numerator only holds quartet-informative families while the denominator counts every variable family, and the matrices differ in convention, so $r$ is no longer a sampling fraction of one pool; the decile pattern is a measurement of the sample, not of an inclusion probability.
**Per-genome richness.** `--sibling-stats` now gives the variable families (size ≥ 2) per genome: 0.72 M to 93.9 M, median 34.5 M. Spearman between a genome's calls in the sample and its variable-family count: **0.974 in the baseline, 0.435 in the combined draw**: the baseline mirrored genome richness, the combined draw largely decouples from it (on the same genomes the correlation with *all* families is 0.66 and 0.32). Genomes poorest in variable families: *A. saussurei* 0.72 M (0.30 % of its families), *A. triflora* 1.26 M (0.72 %), *P. stricta var. obesior* 2.39 M (1.33 %), then *P. pumila* 5.2 M, *P. auricula* subsp. *serratifolia* 5.9 M (1.92 %), *P. carniolica* 6.8 M (1.77 %), *A. pyrenaica* PHA000543 8.4 M (1.46 %). Their calls in the sample went from 6 986 / 1 936 / 2 403 (saussurei, triflora, stricta) to 15 725 / 22 988 / 25 392, and carniolica from 4 937 to 52 310; *saussurei* stays the poorest genome in both. These genomes are data-poor, not empty: their families are mostly of size 1. This is the per-genome $c_i$ the normalisation of the data-poor section calls for.
Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 / 2 / 3 / 4 = 33 512 358 231 (98.43 %) / 479 910 992 (1.41 %) / 45 621 663 (0.13 %) / 9 725 201 (0.03 %). Sizes ≥ 2 sum to 535 257 856, matching the 535 257 857 of the entropy table within 1.
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment had not been given to IQ-TREE (done 2026-10-16: see the last measured section).
## Tree-level effect of the combined draw: sCF on three trees (measured 2026-10-16)
Closes the "not measured" item of the previous section (the combined alignment given to IQ-TREE). Three trees, all sCF by `--scfl 100 --cf-verbose`, model `GTR+G4+ASC`:
- **A** = `concord` (12/10): entropy-only draw, 180 genomes, 5 003 414 columns; `-t` + `-m` re-ran a full ML search (59 h).
- **B** = `concord_quartet_balance`: combined draw (`--entropy 1 --quartet --pair-balance`), 180 genomes, 5 000 000 columns. Its log stops during the tree search (14/10), so the command that produced its `cf.*` (16/10) is not in the files; presumably `-te` on the 15/10 treefile as for G (unverified).
- **G** = `scf_good`: same draw as B without the 3 data-poor genomes (*A. pyrenaica* PHA000543, *A. saussurei*, *A. triflora*), 177 genomes. 72 sites became invariant and were removed to satisfy ASC; the search (`concord_quartet_balance_good`, 107 iterations, 6 h 47 on 120 threads) crashed in the sCF step on the FASTA (`Assertion all_prob_const < 1.0`; cause untested, ASC with constant sites is a suspect). sCF were recomputed with `-te <treefile> -s <varsites.phy> -m GTR+G4+ASC --scfl 100`: **no search** (0 iterations, lnL −41 053 676.53 vs −41 053 676.41 from the search), 4 999 729 columns. A used `-t`, B/G a fixed tree: the protocols differ.
| | A | B | G |
|---|---|---|---|
| Branches | 177 | 177 | 174 |
| Mean `sN` | 3 219 | 13 163 | 13 396 |
| Branches with `sN` < 100 / < 1 000 | 13 / 50 | 0 / 5 | 0 / 2 |
| Median sCF | 44.2 | 42.9 | 42.8 |
| Branches with sCF < 40 / ≥ 70 | 74 / 28 | 67 / 26 | 69 / 25 |
| *Primula* and *Androsace* monophyletic | no / no | no / no | yes / yes |
**Sampling effect (A → B, same 180 genomes).** `sN` ×4.5 and the scarcity is gone. Sites do not become more concordant: on the 113 shared bipartitions (of 177) median sCF 48.0 → 47.3 (44 up, 69 down). Robinson-Foulds 128: the draw rearranged the tree deeply. Consistent with the earlier reading that the intra-*Androsace* conflict is not a matter of site count; the number of branches with sCF < 40 % stays around 70.
**Interface nodes.** Of the old nodes 209/211/212/216/217 (`sN` 17 / 226 / 184 / 7 / 7) none exists in G. Node 208 survives with `sN` 190 → 5 218 (sCF 54.5 → 37.5). Six mixed clades remain in G (83 to 72 tips, 50 *Primula* + 22-33 *Androsace*), all with `sN` > 3 000.
**Effect of the 3 genomes (B → G, same draw).** RF 16 (166 of 174 shared), median sCF 43.2 → 43.6, median `sN` unchanged. In B, *A. pyrenaica* and *A. triflora* sit inside *Primula* (a clade of 52 tips = 50 *Primula* + these 2; each has a *Primula* sister, sCF 92 % and 52 %, `sN` 556 and 153); *A. saussurei* stays with *Androsace*. Removing them restores both monophylies and changes nothing else measurably: support for "too little data to place them" (data-poor section), not for contamination, though the latter is not excluded.
**Where A and G disagree (2026-10-16).** On the 177 common tips: 124 shared bipartitions, 50 only in A, 50 only in G. Differing bipartitions are weakly supported in both trees (median sCF 35.7 in A-only, 37.7 in G-only, versus 47.5-48.1 on the shared ones), so the disagreement sits where support was already low. Differences are mostly local: the closest bipartition of the other tree is 1 tip away for 25 of 50 A-only and 21 of 50 G-only splits (≤ 2 tips: 37 and 30); only 8 and 9 are 5+ tips apart. Greedy removal of the tip that lowers RF most: the first (*P. auricula* subsp. *serratifolia*) takes RF from 100 to 86, then about 2 per tip (*A. maxima*, *P. stricta* var. *obesior*, then ordinary tips); the top three are data-poor genomes. A bipartition separates a data-poor genome (the 5 left after the 3 removals) in 1 of 124 shared splits but in 23 of 50 A-only and 10 of 50 G-only splits; removing those 5 genomes gives RF 72, against 96.6 (range 90-100, 40 draws) for 5 random tips. So about 28 of the 100 differences trace to data-poor genomes, and the remaining 72 do not. Backbone: A has a clade of 50 *Primula* + 25 *Androsace* (sCF 76, `sN` 453), G one of 50 + 26 (sCF 73, `sN` 6 677): one tip apart. Genuine conflict: a clade of 72 *Androsace* exists in both trees (A: sCF 34, `sN` 17 127; G: sCF 38, `sN` 49 473) with 16 different tips, i.e. a topological disagreement carried by tens of thousands of sites in each, not a scarcity effect; consistent with the intra-*Androsace* conflict above.
**Not established.** Separate contribution of entropy / quartet / pair balance (no ablation); stability of the tree between two independent draws (one draw per setting; A → B shows the topology moves with the draw); whether the quartet-informative restriction biases branch lengths (ASC corrects for variable-only ascertainment, not for requiring two states each in two genomes; tree length 22.5 is not interpretable as substitutions/site).
## Open questions ## Open questions
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy. - Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
+12
View File
@@ -42,6 +42,18 @@ obikmer:
debug: debug:
cargo build --manifest-path $(CARGO_DIR)/Cargo.toml --bin obikmer cargo build --manifest-path $(CARGO_DIR)/Cargo.toml --bin obikmer
# One-shot converter of indexes built with ptr_hash 1.x (see tools/migrate-mphf).
# Standalone crate, outside the src/ workspace, with its own Cargo.lock/target.
MIGRATE_MPHF_DIR := tools/migrate-mphf
.PHONY: migrate-mphf
migrate-mphf:
cargo build --release --manifest-path $(MIGRATE_MPHF_DIR)/Cargo.toml
.PHONY: clean-migrate-mphf
clean-migrate-mphf:
rm -rf $(MIGRATE_MPHF_DIR)/target
.PHONY: profile-build .PHONY: profile-build
profile-build: profile-build:
cargo build --release --manifest-path $(CARGO_DIR)/Cargo.toml \ cargo build --release --manifest-path $(CARGO_DIR)/Cargo.toml \
+107
View File
@@ -0,0 +1,107 @@
# VCF pseudo-alignment
[`phylo --vcf`](../usage/phylo.md) writes the sites of a SNP pseudo-alignment as a VCF file, `<prefix>_alignment.vcf`. This page describes exactly what that file contains, because the k-mer families it is built from are not ordinary genomic variants and a few conventions follow from that.
## Why a VCF, and what a site is
The pseudo-alignment (see [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md)) has one column per *family*: the up to four k-mers that share all their bases except the central one. A genome that carries one of these k-mers shows a particular base at the centre, and the columns of the alignment are those central bases, genome by genome. The FASTA output keeps this as a matrix of letters. The VCF output lists the same columns as sites with alleles and genotypes, which is the form that variant-oriented tools (`bcftools`, population-genetics software, genotype viewers) expect.
Nothing here comes from a reference genome. The file has no real coordinates, and the sites are not located on a chromosome: the positions below are labels that make the file valid and reproducible, not map positions.
## Producing the file
```
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
```
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, together with an already-built sibling annex (see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
## Layout
An illustrative excerpt (columns are tab-separated in the file):
```
##fileformat=VCFv4.5
##source=obikmer
##contig=<ID=obikmer_0_0>
##contig=<ID=obikmer_0_1>
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype (diploid)">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT idx0 idx1 idx2 idx3
obikmer_0_0 440 CGATTNGTTAA A G . . . GT 0/0 1/1 0/0 0/0
obikmer_0_0 1511 CACGANTGTAG A G,T . . . GT 0/0 0/0 1/1 2/2
obikmer_0_1 9 TTACGNAGCTA C * . . . GT 0/0 0/0 1/1 0/0
```
There is one sample column per genome that takes part in the draw, named after the genome label, in the index's own order. A genome removed with `--exclude-genome` or `--min-shared-family` has no column at all. `QUAL`, `FILTER` and `INFO` are always `.`: obikmer applies no variant filter and attaches no annotation, and leaving them empty keeps a file of millions of sites compact.
## Coordinates: `CHROM` and `POS`
An index is organised in partitions, and each partition holds one or more layers of k-mers (see [Index construction and on-disk layout](index_layout.md)). A family always belongs to exactly one layer of one partition, and the families of a layer are visited in a fixed order. The VCF uses that structure as coordinates:
- `CHROM` is `obikmer_<partition>_<layer>`, one virtual contig per layer of each partition in which at least one site was drawn;
- `POS` is the rank of the family inside that layer, starting at 1.
For a given index and a given sibling annex, a family therefore always receives the same `CHROM` and `POS`, whatever the draw, the options or the number of sites requested. Two runs on the same index can be compared position by position. This stability is lost if the sibling annex is rebuilt, or if the index is rebuilt, merged or filtered, because the families are then enumerated again. Positions are never comparable between two different indexes; use `ID` for that. Even rebuilding an index from the very same genomes may enumerate the families in another order: on a small test index built twice from identical input, the family `CGATTNGTTAA` had `POS` 440 the first time and 371 the second, with the same `ID`.
The file is sorted by partition, then layer, then position, which is also the order of the `##contig` lines, so tools that require a sorted file accept it directly. Positions are sparse: only the drawn families are written.
## Identifier: `ID`
The `ID` column holds a label for the family that does not depend on the index. It is built from one of the family's k-mers by replacing its central base with `N` (for example `CGATTNGTTAA` for k = 11), after choosing, among the family's possible orientations, the one that sorts first. All the members of a family, whichever strand they were read on, give the same label.
This is what makes two files comparable even when they come from different indexes: the same family has the same `ID` in both, while its `CHROM` and `POS` will generally differ. It is also unique within a file, since two different families never share a label.
## Alleles: `REF` and `ALT`
The alleles are the bases carried at the centre of the family's k-mers, as in the FASTA. Two points are worth knowing.
First, the base is the central base of the member's *canonical* form (the smaller of the k-mer and its reverse complement). That is not necessarily the strand of the `ID` label, so the letters should not be matched to the `ID` sequence by position; they are labels for the allele, not a reading of that sequence.
Second, `REF` carries no meaning of reference: it is simply the most frequent allele among the genomes of the draw, each allele copy counted once, with the alphabetical order A < C < G < T deciding ties. `ALT` lists the remaining observed bases in the order A, C, G, T, separated by commas. There is no option to choose a reference genome.
### Absence: the `<ABS>` allele
A genome may carry none of the family's k-mers. Without `--free-loss` this absence is treated as a real state (the family was lost, or is absent from that genome), which the FASTA shows as `-`. In the VCF it becomes one more allele, the symbolic allele `<ABS>`, listed last in `ALT` and declared in the header by `##ALT=<ID=ABS,Description="Family absent from the genome">`. A genome without the family receives the genotype of `<ABS>`; for a site with `ALT` equal to `G,T,<ABS>` that is `3/3`. A site that differs between genomes only by presence versus absence has `ALT` equal to `<ABS>`.
The VCF specification gives the plain character `*` a meaning of its own (an allele missing because of an overlapping deletion described by another record), which is not the situation here, so it is deliberately not used. `<ABS>` is a free-form symbolic identifier, outside the reserved structural-variant names (`DEL`, `INS`, `DUP`, `INV`, `CNV`) and outside `<*>`, which belongs to reference blocks. Tools that only look at sequence alleles treat the site as having an alternative that is not a base.
With `--free-loss`, absence is declared missing data instead: `<ABS>` never appears, and the genomes concerned get a missing genotype.
## Genotypes
Every genome gets one `GT` value per site. The ploidy of these values is a convention chosen from the options, and the header says which (`Genotype (diploid)` or `Genotype (haploid)`).
| Cell in the alignment | Diploid (default, `--only-diploid`) | Haploid (`--no-ambiguity`) |
|---|---|---|
| one base `X` | `X/X`, for example `0/0` or `1/1` | `X`, for example `0` |
| two bases (IUPAC `R Y S W K M`) | the heterozygote, for example `0/1` | does not occur |
| three or four bases (`B D H V N`) | `./.` (not expressible) | does not occur |
| absent family (`<ABS>`) | `<ABS>/<ABS>`, for example `3/3` | the index of `<ABS>` |
| missing (`?`) | `./.` | `.` |
The two allele indices of a diploid genotype are always written in increasing order (`0/1`, never `1/0`), and the separator is `/` because no phasing information exists.
Why diploid by default? A genome can carry several forms of the same family, which the alignment records as an IUPAC ambiguity code. A diploid genotype can express up to two forms, and `--only-diploid` removes the genomes that would need more (three or four forms) before the sites are selected, so that no `./.` comes from them. `--no-ambiguity` goes further and treats every such genome as missing, which makes all genotypes haploid.
This is a convention, not a measurement. The file records which forms a genome carries, not how many copies of each: `A/A/G` and `A/G/G` both give the code `R` and the genotype `0/1`. Likewise, a genome with two forms may be heterozygous at one locus, or it may carry two similar regions of its genome (paralogues, repeats); the file cannot tell these apart. Do not read the genotypes as calls from sequencing reads.
## Which sites are written
Only the families that survive the draw and its masking appear. Within those:
- a site needs at least one genome with a base or (in diploid mode) a two-base code, because `REF` is a base; a site where every genome is missing, absent or has an ambiguity code too wide for the mode is left out of the VCF while remaining in the FASTA;
- the log line at the end of the run gives the number of sites written out of the number drawn, so the difference is visible.
A site is selected because its genomes show at least two distinct states, and absence counts as a state. A site that differs between genomes only by presence versus absence is therefore a legitimate site, written with `ALT` equal to `<ABS>`.
## Using the file with other tools
- The contigs are virtual and have no declared length, and the file refers to no reference. Tools that need one (annotation, liftover) cannot use it; tools that only read sites and genotypes (`bcftools view`, `query`, `stats`, population-genetics software, PLINK-style converters) can.
- If a tool rejects multiallelic sites or the `<ABS>` allele, `bcftools view -m2 -M2` keeps biallelic sites only, and `bcftools view -e 'ALT="<ABS>"'` removes the sites involving absence. The file as written was read by `bcftools` and indexed by `tabix` without complaint on a small test index, in both diploid and haploid modes. `bcftools stats` counts the sites with `<ABS>` among the "others" (symbolic alleles) rather than as SNPs.
- If a tool expects haploid data, run `phylo` with `--no-ambiguity` rather than converting the genotypes afterwards.
## See also
- [`phylo`](../usage/phylo.md): all the options that shape the draw.
- [Sampling design](../theory/phylogeny/snp_based/sampling_design.md): how the sites are drawn.
- [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md): what a family is.
+1 -1
View File
@@ -51,7 +51,7 @@ Assuming equal genome lengths, the number of genomes containing at least one occ
$$ $$
P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right){S-j}. P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right)^{S-j}.
$$ $$
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
@@ -21,7 +21,7 @@ A family where every genome carries the same single state has entropy 0 (uninfor
## Quartet restriction ## Quartet restriction
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts called states only, so absence never forms one of the two pairs. A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts the states of the alignment: without `--free-loss`, absence is a state like any other and can form one of the two pairs; with `--free-loss` it is missing data and never does.
## Pair balance ## Pair balance
+8
View File
@@ -12,6 +12,14 @@ obikmer index -o OUTPUT [OPTIONS] [INPUTS...]
|---|---| |---|---|
| `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. | | `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. |
## Which files are read
A file named explicitly on the command line is always read, whatever its extension: the format (FASTA, FASTQ or GenBank) and the compression (gzip, bzip2, xz or zstd) are detected from the content. If the content is not a sequence file, the command reports an error rather than skipping the file.
When an input is a directory, the files inside it are selected by extension: `.fasta`, `.fa`, `.fna` (the extension of NCBI genome downloads), `.ffn`, `.fas`, `.fsa`, `.fastq`, `.fq`, `.gbff`, `.gbk` and `.gb`, each optionally followed by `.gz`.
A small input spread over many partitions leaves most partitions empty. This is valid: the index is built normally and empty partitions simply hold no k-mers.
## Options ## Options
| Option | Default | Description | | Option | Default | Description |
+17 -1
View File
@@ -20,10 +20,26 @@ obikmer merge -o OUTPUT SOURCE... [OPTIONS]
| `--force` | off | Overwrite an existing output directory | | `--force` | off | Overwrite an existing output directory |
| `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts | | `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts |
| `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing | | `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing |
| `--budget-fraction` | `0.5` | Fraction of available RAM reserved as the memory budget for parallel partition merging | | `--dense` | off | Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see [pack](pack.md)) |
## Behaviour ## Behaviour
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources. The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources.
By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies. By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies.
The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; `--dense` keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The [pack](pack.md) page explains the trade-off between the two formats.
## Resource usage
The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.
Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.
## Merging an index that was already merged
An index produced by `merge` can be used as one of the sources of a later merge, which is how a collection is extended without starting again from the original genomes. There is one rule: among the sources, at least one must not have been through a merge. A merged index is stored in a compact packed format (see [pack](pack.md)) that is read very efficiently but cannot be extended in place. The merge therefore takes a freshly built index as its starting point and brings the packed index in as a source. A typical case is a merged index of many genomes plus one or a few newly built single-genome indexes, and this works.
If every source has already been merged or packed, the command stops before doing any work and says so. In that case, merge the original single-genome indexes together instead.
The genome columns of the result follow the order of the starting index first, then the other sources, so they may not come out in the order the sources were listed. The genome labels always identify each column unambiguously.
+44 -12
View File
@@ -16,7 +16,7 @@ obikmer phylo INDEX [OPTIONS]
| Option | Default | Description | | Option | Default | Description |
|---|---|---| |---|---|---|
| `--distance` | `jaccard` | See the two tables below for the full list of accepted values | | `--distance` | none | Compute and write a distance matrix. See the two tables below for the full list of accepted values. No matrix is computed unless this is given, or `--nj`, `--upgma` or `--shared-kmers` is (they need a matrix and then use `jaccard`) |
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it | | `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index | | `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format | | `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
@@ -31,7 +31,7 @@ Every value routes to one of two independent computations: **whole-index distanc
### Output ### Output
Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below. When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
## `--exclude-genome`, `--min-shared-family` ## `--exclude-genome`, `--min-shared-family`
@@ -48,12 +48,15 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
| Option | Description | | Option | Description |
|---|---| |---|---|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value | | `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the `--no-ambiguity` / `--only-diploid` convention given: `--entropy` the entropy annex, `--quartet` the quartet annex, `--pair-balance` the pair-weight and family-overlap annexes, `--family-overlap` or `--min-shared-family` the family-overlap annex. For example `--sibling-annex --quartet --only-diploid` prepares exactly the quartet annex of that convention, and `--sibling-annex` alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally | | `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only | | `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for | | `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan | | `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
| `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index |
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` | | `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
### `--sibling-stats` ### `--sibling-stats`
@@ -74,13 +77,20 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`. `<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
#### VCF output (`--vcf`)
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
| Option | Default | Description | | Option | Default | Description |
|---|---|---| |---|---|---|
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. | | `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` | | `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code | | `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct called states, each carried by at least two genomes. Which genomes count as carriers follows `--no-ambiguity` | | `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` | | `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling | | `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band | | `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
@@ -126,15 +136,33 @@ With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`)
### `--quartet`: keeping only families that can resolve a tree ### `--quartet`: keeping only families that can resolve a tree
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. Absence is never a state here: a genome that does not carry the family does not count, with or without `--free-loss`. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment. A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` convention, in one pass shared with the other annexes, and built for both conventions by `--sibling-annex`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families. The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
### Describing a draw: `--sample-stats`, `--sample-plan`
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
`--sample-stats` writes three files:
| File | Content |
|---|---|
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. Excluded genomes are listed with zeros in the first file and count nowhere.
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
`--shannon-summary` writes `<prefix>_shannon_summary.csv` (`entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families`): the number of variable families per entropy (`entropy15` over every carrier, `entropy4` over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (`bin_start` is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers `--no-ambiguity` loses, from the same scan as `--shannon` but without one line per family.
### `--pair-balance`: representing every pair of genomes ### `--pair-balance`: representing every pair of genomes
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs. Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` convention, after the family-overlap matrix it depends on, and by `--sibling-annex` for both conventions). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first. Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families ### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
@@ -142,7 +170,7 @@ By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data. The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
The entropy that weights the draw follows `--no-ambiguity`, so that it describes the same genomes as the alignment. Without the flag, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With the flag, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). The two entropies are saved separately: switching `--no-ambiguity` builds the other one once, without discarding the first, and `--sibling-annex` builds both. The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
## Sankoff calibration and phylogenetic exports ## Sankoff calibration and phylogenetic exports
@@ -177,14 +205,14 @@ TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in t
## Output files ## Output files
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix is produced, on stdout. All matrices use genome labels as row/column headers, in index order. With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix, if one is computed, is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
### Distance matrix ### Distance matrix
| File | Written by | Format | Content | | File | Written by | Format | Content |
|---|---|---|---| |---|---|---|---|
| `<prefix>_dist.phy` | always, unless `--csv` | relaxed PHYLIP | the `--distance` matrix | | `<prefix>_dist.phy` | a distance matrix is computed, unless `--csv` | relaxed PHYLIP | the `--distance` matrix |
| `<prefix>_dist.csv` | `--csv` | CSV matrix | the `--distance` matrix, 6 decimals | | `<prefix>_dist.csv` | a distance matrix is computed, with `--csv` | CSV matrix | the `--distance` matrix, 6 decimals |
| `<prefix>_shared.csv` | `--shared-kmers` | CSV matrix | shared-kmer count per genome pair (integers) | | `<prefix>_shared.csv` | `--shared-kmers` | CSV matrix | shared-kmer count per genome pair (integers) |
| `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree | | `<prefix>_nj.nwk` | `--nj` | Newick | Neighbor-Joining tree |
| `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree | | `<prefix>_upgma.nwk` | `--upgma` | Newick | UPGMA tree |
@@ -198,7 +226,11 @@ CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `g
| `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global | | `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for | | `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above | | `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
| `<prefix>_alignment.fasta` | `--pseudo-alignment` | FASTA | SNP-only pseudo-alignment, IUPAC-coded | | `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers |
| `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes |
| `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept |
| `<prefix>_alignment.fasta` | `--pseudo-alignment` or `--fasta` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
| `<prefix>_alignment.vcf` | `--vcf` | VCF | Same sites as the FASTA, with stable virtual coordinates and a k-mer-based `ID` |
### Sankoff calibration and exports ### Sankoff calibration and exports
+152
View File
@@ -0,0 +1,152 @@
#!/usr/bin/env bash
# Usage: bench_mphf.sh LABEL
#
# Measures the obikmer commands that sit on MPHF lookups (see
# DevDocMD/implementation/mphf_batch_queries.md for the site inventory), on the
# 20-genome benchmark data already built under run/ (`gmake all` first).
# Nothing under run/ other than run/bench_mphf/ is written.
#
# Env:
# BINARY obikmer binary to measure (default ../src/target/release/obikmer)
# REPS repetitions per step (default 3)
# CPU_MAX passed as --cpu-max when set (default: all cores)
#
# Output: run/bench_mphf/LABEL/results.csv (label,step,rep,wall_s,maxrss_b)
# run/bench_mphf/LABEL/checksums.txt (order-independent output digests,
# to be diffed across LABELs: a speed-up must not change any of them)
set -euo pipefail
LABEL="$1"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RUN_DIR="${SCRIPT_DIR}/run"
BINARY="${BINARY:-${SCRIPT_DIR}/../src/target/release/obikmer}"
REPS="${REPS:-3}"
OUT="${RUN_DIR}/bench_mphf/${LABEL}"
CPU=()
[[ -n "${CPU_MAX:-}" ]] && CPU=(--cpu-max "${CPU_MAX}")
rm -rf "${OUT}"; mkdir -p "${OUT}"
RESULTS="${OUT}/results.csv"
SUMS="${OUT}/checksums.txt"
echo "label,step,rep,wall_s,maxrss_b" >"${RESULTS}"
: >"${SUMS}"
# timed STEP REP CMD... — wall time + peak RSS via BSD time(1); stdout of CMD
# goes to $OUT/last.stdout, stderr to $OUT/last.stderr.
timed() {
local step="$1" rep="$2"; shift 2
local tf="${OUT}/time.tmp"
/usr/bin/time -l -o "${tf}" "$@" >"${OUT}/last.stdout" 2>"${OUT}/last.stderr" \
|| { echo "FAILED: $step: $*" >&2; cat "${OUT}/last.stderr" >&2; exit 1; }
local wall rss
wall=$(awk '/ real /{print $1}' "${tf}")
rss=$(awk '/maximum resident set size/{print $1}' "${tf}")
echo "${LABEL},${step},${rep},${wall},${rss}" >>"${RESULTS}"
# per-stage wall times from the Reporter log lines ("done stage=X wall_secs=Y")
sed $'s/\x1b\\[[0-9;]*m//g' "${OUT}/last.stderr" \
| awk -v L="${LABEL}" -v S="${step}" -v R="${rep}" \
'/ done stage=/{ for(i=1;i<=NF;i++){ if($i ~ /^stage=/) st=substr($i,7); if($i ~ /^wall_secs=/) w=substr($i,11) } print L","S":"st","R","w",0" }' \
>>"${RESULTS}"
printf ' %-28s rep %s %8s s\n' "${step}" "${rep}" "${wall}" >&2
}
digest() { LC_ALL=C sort "$1" | md5 -q; }
# ── 0. data: the indexes under run/ may have been built by an older
# MPHF generation (ptr_hash 1.x, unreadable by current binaries), so the
# bench builds its own, once, from the simulated reads. Shared by all LABELs;
# delete run/bench_mphf/data to force a rebuild.
DATA="${RUN_DIR}/bench_mphf/data"
if [[ ! -f "${DATA}/.ready" ]]; then
echo "preparing data with ${BINARY}" >&2
mkdir -p "${DATA}/presence" "${DATA}/count"
for d in "${RUN_DIR}"/simulated_data/*/*/; do
strain="$(basename "$d")"; species="$(basename "$(dirname "$d")")"
sp="${species}--${strain}"
for kind in presence count; do
extra=(); [[ $kind == count ]] && extra=(--with-counts)
"${BINARY}" "${CPU[@]}" index --output "${DATA}/${kind}/${sp}" --force \
--theta 0 "${extra[@]}" --label "${sp}" --meta "species=${species}" \
"$d/reads_R1.fastq.gz" "$d/reads_R2.fastq.gz" 2>/dev/null
done
done
"${BINARY}" "${CPU[@]}" merge --output "${DATA}/global_presence" --force \
--force-presence "${DATA}"/presence/*/ 2>/dev/null
"${BINARY}" "${CPU[@]}" merge --output "${DATA}/global_count" --force \
"${DATA}"/count/*/ 2>/dev/null
touch "${DATA}/.ready"
fi
sources=("${DATA}"/presence/*/)
# ── 1. index: build pass 2 (mphf.index per kmer + evidence writes) ──────────
for sp in Escherichia_coli--K-12_MG1655 Candidozyma_auris--GCF_003013715.1_ASM301371v2; do
species="${sp%%--*}"; strain="${sp#*--}"
reads="${RUN_DIR}/simulated_data/${species}/${strain}"
for rep in $(seq "${REPS}"); do
timed "index_${species}" "${rep}" "${BINARY}" "${CPU[@]}" index \
--output "${OUT}/idx_${species}" --force --theta 0 --label "${sp}" \
"${reads}/reads_R1.fastq.gz" "${reads}/reads_R2.fastq.gz"
done
done
# ── 2. merge: find_batch_grouped, build_graph filter, hash_batch, index_batch ─
for rep in $(seq "${REPS}"); do
timed "merge_20_presence" "${rep}" "${BINARY}" "${CPU[@]}" merge \
--output "${OUT}/merged" --force --force-presence "${sources[@]}"
done
"${BINARY}" "${CPU[@]}" dump "${OUT}/merged" >"${OUT}/merged.dump"
echo "merged_dump $(digest "${OUT}/merged.dump")" >>"${SUMS}"
rm -f "${OUT}/merged.dump"
# ── 3. query: query stage 1 (find per kmer, layer by layer) ─────────────────
for kind in presence count; do
for sp in Escherichia_coli/K-12_MG1655 Saccharolobus_islandicus/M.16.4; do
reads="${RUN_DIR}/query_data/${sp}"
for rep in $(seq "${REPS}"); do
timed "query_${kind}_${sp%%/*}" "${rep}" "${BINARY}" "${CPU[@]}" query \
--count-missing "${DATA}/global_${kind}" \
"${reads}/reads_R1.fastq.gz" "${reads}/reads_R2.fastq.gz"
done
echo "query_${kind}_${sp%%/*} $(digest "${OUT}/last.stdout")" >>"${SUMS}"
done
done
# ── 4. filter: iter_layer_kmers → hash_batch + fill_sub_matrix ──────────────
for rep in $(seq "${REPS}"); do
timed "filter_species" "${rep}" "${BINARY}" "${CPU[@]}" filter \
--output "${OUT}/filtered" --force --ingroup "species=Escherichia_coli" \
--outgroup all --min-frac 0.5 --max-frac 1.0 --max-outgroup-count 0 \
"${DATA}/global_presence"
done
"${BINARY}" "${CPU[@]}" dump "${OUT}/filtered" >"${OUT}/filtered.dump"
echo "filtered_dump $(digest "${OUT}/filtered.dump")" >>"${SUMS}"
rm -f "${OUT}/filtered.dump"
# ── 5. convert: build_exact_evidence / build_approx_evidence ────────────────
# On an APFS clone (cp -c) so run/global_index_presence is untouched.
for rep in $(seq "${REPS}"); do
rm -rf "${OUT}/conv"; cp -cR "${DATA}/global_presence" "${OUT}/conv"
timed "convert_to_approx8" "${rep}" "${BINARY}" "${CPU[@]}" convert \
--approx-evidence 8 "${OUT}/conv"
timed "convert_to_exact" "${rep}" "${BINARY}" "${CPU[@]}" convert \
--exact-evidence "${OUT}/conv"
done
"${BINARY}" "${CPU[@]}" dump "${OUT}/conv" >"${OUT}/conv.dump"
echo "convert_roundtrip_dump $(digest "${OUT}/conv.dump")" >>"${SUMS}"
rm -rf "${OUT}/conv" "${OUT}/conv.dump"
# ── 6. phylo: sibling annex (annex.rs: find_in_partition per variant) and
# its consumer (family_scan.rs: hash_batch / find_slot) ──────────────────
for rep in $(seq "${REPS}"); do
rm -rf "${OUT}/phy"; cp -cR "${DATA}/global_presence" "${OUT}/phy"
timed "phylo_sibling_annex" "${rep}" "${BINARY}" "${CPU[@]}" phylo \
--sibling-annex "${OUT}/phy"
timed "phylo_snp_distance" "${rep}" "${BINARY}" "${CPU[@]}" phylo \
--output "${OUT}/snp" --distance snp-jc --csv "${OUT}/phy"
done
echo "phylo_snp_dist $(digest "${OUT}/snp_dist.csv")" >>"${SUMS}"
rm -rf "${OUT}/phy" "${OUT}/idx_"* "${OUT}/merged" "${OUT}/filtered" \
"${OUT}/last.stdout" "${OUT}/time.tmp"
echo "results: ${RESULTS}" >&2
echo "checksums: ${SUMS}" >&2
+2 -2
View File
@@ -19,7 +19,7 @@ INDEX_PATH="${RUN_DIR}/specimen_index_count/${SPECIMEN}"
STATS_DIR="${RUN_DIR}/stats/indexing_count" STATS_DIR="${RUN_DIR}/stats/indexing_count"
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats" STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
mkdir -p "${STATS_DIR}" mkdir -p "${STATS_DIR}" "$(dirname "${INDEX_PATH}")"
r1="${READS_DIR}/reads_R1.fastq.gz" r1="${READS_DIR}/reads_R1.fastq.gz"
r2="${READS_DIR}/reads_R2.fastq.gz" r2="${READS_DIR}/reads_R2.fastq.gz"
@@ -41,7 +41,7 @@ trap 'rm -f "${STDERR_LOG}"' EXIT
--label "${SPECIMEN}" \ --label "${SPECIMEN}" \
--meta "species=${species}" \ --meta "species=${species}" \
"${r1}" "${r2}" \ "${r1}" "${r2}" \
2>"${STDERR_LOG}" 2>"${STDERR_LOG}" || { cat "${STDERR_LOG}" >&2; exit 1; }
cat "${STDERR_LOG}" >&2 cat "${STDERR_LOG}" >&2
+2 -2
View File
@@ -19,7 +19,7 @@ INDEX_PATH="${RUN_DIR}/specimen_index_presence/${SPECIMEN}"
STATS_DIR="${RUN_DIR}/stats/indexing_presence" STATS_DIR="${RUN_DIR}/stats/indexing_presence"
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats" STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
mkdir -p "${STATS_DIR}" mkdir -p "${STATS_DIR}" "$(dirname "${INDEX_PATH}")"
r1="${READS_DIR}/reads_R1.fastq.gz" r1="${READS_DIR}/reads_R1.fastq.gz"
r2="${READS_DIR}/reads_R2.fastq.gz" r2="${READS_DIR}/reads_R2.fastq.gz"
@@ -40,7 +40,7 @@ trap 'rm -f "${STDERR_LOG}"' EXIT
--label "${SPECIMEN}" \ --label "${SPECIMEN}" \
--meta "species=${species}" \ --meta "species=${species}" \
"${r1}" "${r2}" \ "${r1}" "${r2}" \
2>"${STDERR_LOG}" 2>"${STDERR_LOG}" || { cat "${STDERR_LOG}" >&2; exit 1; }
cat "${STDERR_LOG}" >&2 cat "${STDERR_LOG}" >&2
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env python3
"""Usage: summarize_bench.py LABEL [LABEL...] — median wall per step, side by side."""
import csv, statistics, sys, os
base = os.path.join(os.path.dirname(os.path.abspath(__file__)), "run", "bench_mphf")
data = {}
for lab in sys.argv[1:]:
for r in csv.DictReader(open(f"{base}/{lab}/results.csv")):
data.setdefault(r["step"], {}).setdefault(lab, []).append(float(r["wall_s"]))
print(f"{'step':32}" + "".join(f"{l:>14}" for l in sys.argv[1:]) + (" ratio" if len(sys.argv) > 2 else ""))
for step, d in data.items():
med = [statistics.median(d[l]) for l in sys.argv[1:]]
line = f"{step:32}" + "".join(f"{m:14.2f}" for m in med)
if len(med) > 1 and med[0]: line += f" {med[-1]/med[0]:.3f}"
print(line)
+1
View File
@@ -64,6 +64,7 @@ nav:
- Predicates and taxonomy paths: usage/predicates.md - Predicates and taxonomy paths: usage/predicates.md
- Formats: - Formats:
- Index construction and on-disk layout: formats/index_layout.md - Index construction and on-disk layout: formats/index_layout.md
- VCF pseudo-alignment: formats/vcf.md
- Architecture notes: architecture.md - Architecture notes: architecture.md
watch: watch:
+4
View File
@@ -48,6 +48,7 @@ nav:
- obipipeline library: implementation/obipipeline.md - obipipeline library: implementation/obipipeline.md
- On-disk storage: implementation/storage.md - On-disk storage: implementation/storage.md
- MPHF selection: implementation/mphf.md - MPHF selection: implementation/mphf.md
- "MPHF batch queries (inventory)": implementation/mphf_batch_queries.md
- Unitig evidence encoding: implementation/unitig_evidence.md - Unitig evidence encoding: implementation/unitig_evidence.md
- Evidence elimination (discussion): implementation/evidence_elimination.md - Evidence elimination (discussion): implementation/evidence_elimination.md
- obikindex layer tier: implementation/layer_tier.md - obikindex layer tier: implementation/layer_tier.md
@@ -55,6 +56,8 @@ nav:
- PersistentBitVec: implementation/persistent_bit_vec.md - PersistentBitVec: implementation/persistent_bit_vec.md
- Merge command: implementation/merge.md - Merge command: implementation/merge.md
- Merge parallelism & memory: implementation/merge_parallelism.md - Merge parallelism & memory: implementation/merge_parallelism.md
- "Merge mmap audit (2026-11)": implementation/merge_mmap_audit.md
- "Merge scaling (open)": implementation/merge_scaling.md
- Kmer filtering: implementation/filtering.md - Kmer filtering: implementation/filtering.md
- Select command: implementation/select.md - Select command: implementation/select.md
- obitaxonomy crate: implementation/obitaxonomy.md - obitaxonomy crate: implementation/obitaxonomy.md
@@ -64,6 +67,7 @@ nav:
- Sequences: architecture/sequences/invariant.md - Sequences: architecture/sequences/invariant.md
- Kmer index: architecture/index_architecture.md - Kmer index: architecture/index_architecture.md
- Sibling annex: architecture/siblings.md - Sibling annex: architecture/siblings.md
- Hot resume and memoization: architecture/hot_resume_memoization.md
- NUMA-aware worker pools: architecture/numa_worker_pools.md - NUMA-aware worker pools: architecture/numa_worker_pools.md
- NUMA-aware partition runner: architecture/numa_partition_runner.md - NUMA-aware partition runner: architecture/numa_partition_runner.md
+150
View File
@@ -0,0 +1,150 @@
# Prompt — refonte mmap de `obikmerge` (plans A puis C)
> À coller tel quel au début d'une nouvelle session Claude Code ouverte à la racine du dépôt obikmer.
> Conçu pour être suivi par **Sonnet** : toutes les décisions d'architecture sont fixées ci-dessous. Le plan B de l'audit (ouverture « slots only » dans `obikindex`) est **hors périmètre** de cette session (voir étape 5).
---
## Contexte
Lis d'abord `DevDocMD/implementation/merge_mmap_audit.md`. C'est le diagnostic complet, et ce prompt en implémente le plan.
En résumé : `obikmerge::partition_merge::merge_partition` ouvre, pour **chaque** partition, **toutes** les couches de **toutes** les sources (4 mmaps chacune). Elle crée aussi **toutes** les colonnes destination d'avance, en fichiers creux mappés en écriture (`set_len` + `MmapMut`). Avec 186 sources, ça fait environ 1 100 mappings par partition, ce qui épuise `vm.max_map_count` en production (pas d'accès root pour le relever). L'écriture via `MmapMut` dans des fichiers creux transforme aussi tout refus d'allocation disque en SIGBUS.
Objectif : le nombre de mappings vivants par partition ne doit plus dépendre du nombre de sources, et l'écriture des colonnes du merge ne doit plus passer par `MmapMut`. **Le résultat sur disque doit rester strictement identique.**
## Règles (non négociables)
- Appelle `mcp__serena__initial_instructions` avant tout travail de code. Utilise Serena (`find_symbol`, `get_symbols_overview`, `find_referencing_symbols`, `replace_symbol_body`, `replace_content`) pour naviguer et éditer le Rust.
- **Aucune commande VCS mutante** (git/jj : commit, add, mv, rm, describe, push…). Lecture seule autorisée.
- **Aucune nouvelle dépendance**, ni changement de version.
- `obikindex` fait autorité : cette session n'y modifie et n'y ajoute rien.
- Pas de fonctions libres ad hoc dans `obikmerge` pour combler un manque d'API. Si une opération générique manque sur un type de `obicompactvec`, ajoute-la comme méthode sur ce type.
- Fichiers source sous ~500 lignes. `partition_merge.rs` en fait déjà 605 : il **doit** être découpé (étape 6).
- **Ne jamais `.collect()` un flux de k-mers ou de unitigs** pour satisfaire une contrainte de trait.
- Échanges en **français**, documentation en **anglais**.
- **Point d'arrêt obligatoire** à la fin de chaque étape. Résume ce qui a été fait et les résultats de tests (sortie réelle), puis **attends mon feu vert** avant l'étape suivante.
- Si un test ou l'oracle échoue et que tu ne trouves pas la cause en une tentative, **arrête-toi et expose le problème**, sans contournement.
## Étape 1 — Oracle de non-régression (avant toute modification de code)
### 1a. Oracle synthétique (rapide, couvre comptage et merge en deux temps)
1. Lis `scripts/smoke_test_index.sh` (modèle de style, options de `obikmer index`) et les `--help` de `obikmer index`, `merge` et `dump`.
2. Écris `scripts/smoke_test_merge.sh`, sur le même modèle :
- génère 4 FASTA aléatoires déterministes A, B, C, D, petits, avec **des fragments partagés** entre eux. Le merge doit à la fois retrouver des k-mers déjà présents dans la destination (chemin « couche existante ») et en créer de nouveaux (chemin « nouvelle couche ») ;
- indexe chacun deux fois : présence (défaut) et comptage (`--with-counts`). Plusieurs partitions (`-p 4`) ;
- scénario 1 : `merge A B C D` (présence), puis la même chose en comptage ;
- scénario 2 : `merge A B` → AB, puis `merge AB C D`. Ce scénario couvre une source multi-génome et une destination à plusieurs couches ;
- scénario 3 : comme le scénario 1, avec `--dense` ;
- pour chaque sortie : `obikmer dump` → `sort` → un fichier texte ;
- deux modes : `--record DIR` écrit les sorties triées dans `DIR`, `--check DIR` les régénère et fait un `diff` avec `DIR`. Affiche PASS/FAIL clairement, avec un code de sortie cohérent.
3. Lance `--record benchmark/run/merge_oracle/synthetic` **avec le code actuel, non modifié**. Vérifie que les fichiers ne sont pas vides et que les colonnes de génomes ont du sens.
4. Lance aussitôt `--check` sur le même code pour vérifier que le script est déterministe.
### 1b. Oracle sur données réelles : `benchmark/`
`benchmark/run/specimen_index_presence/` contient 20 index présence construits à partir de vrais génomes (bactéries + une levure, lectures simulées). Lis `benchmark/README.md` et `benchmark/merge_presence.sh` (le merge de référence y utilise `--force --force-presence`).
État constaté au 2026-11-17, à revérifier :
- `specimen_index_count/` est **vide** : pas de piste comptage sans relancer `gmake index_count` (GNU make ≥ 4.3 : `gmake` sur macOS). **Demande-moi** avant de le lancer, car ça prend du temps ;
- `reference_index/` (`.npz`) est **absent** : `verify_merge_*.py` est inutilisable, ne t'appuie pas dessus ;
- `global_index_presence/` a été produit par un binaire antérieur au code actuel (release du 2026-10-22) : **ce n'est pas la référence**.
Procédure :
1. Recompile le binaire release à partir du **code actuel non modifié** (`cargo build --release` dans `src/`). Ne touche pas aux artefacts existants de `benchmark/run/`.
2. Lance le merge des 20 index avec les mêmes options que `merge_presence.sh`, mais vers un répertoire **dédié**, par exemple `benchmark/run/merge_oracle/bench/merged`, jamais `benchmark/run/global_index_presence`. Garde le log stderr : le rapport de stages (wall et RSS de `merge_partitions`) sert de référence de performance.
3. `obikmer dump` du résultat → `sort` → compresse (`zstd`) + `sha256sum`, dans `benchmark/run/merge_oracle/bench/`. La sortie peut être volumineuse : compare d'abord les hash, et ne fais un `diff` que s'ils diffèrent.
4. Contrôle croisé informatif : fais la même chose (dump trié + hash) sur `benchmark/run/global_index_presence`. Si les hash diffèrent, **signale-le sans conclure** : l'écart peut venir d'évolutions légitimes du code depuis octobre.
5. Écris ces étapes dans un script `scripts/bench_merge_oracle.sh` avec deux modes : `--record DIR` et `--check DIR`. En mode `--check`, il affiche aussi le temps de `merge_partitions` avant/après.
Après chaque étape de code (3, 4, 6), les **deux** oracles doivent passer : `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check`. Si `merge_partitions` est plus de 20 % plus lent qu'avant sur le benchmark, signale-le au point d'arrêt.
**Point d'arrêt.**
## Étape 2 — `MatrixBuilder::n_cols` (petit prérequis, `obicompactvec`)
Ajoute `pub fn n_cols(&self) -> usize` à `obicompactvec::MatrixBuilder` (`src/obicompactvec/src/matrix_builder.rs`), en déléguant au builder Bit ou Int. Si `PersistentCompactIntMatrixBuilder` n'expose pas `n_cols()`, ajoute-le sur ce type. Ajoute un test unitaire. `cargo test -p obicompactvec`.
Pas de point d'arrêt séparé : enchaîne sur l'étape 3.
## Étape 3 — Plan A : passe 2 par groupes de sources (`obikmerge`)
Fichier : `src/obikmerge/src/partition_merge.rs`, fonction `merge_partition`.
### 3a. Passe 1 sans ouvrir les couches sources
Construis la liste des chemins `unitigs.bin` sans `open()`. Pour chaque source et chaque `l in 0..partition.n_layers()`, prends `KmerLayer::new(&partition, l).unitigs_path()` (ce chemin est valide sur la variante `Empty`). Garde le filtre `exists()`. Ne touche pas au reste de la passe 1 (`build_graph`, `materialize_layer`, ouverture de `new_mphf`).
### 3b. Nouvelle couche : matrice initialisée seule
Si `any_new`, crée la matrice de la nouvelle couche avec `MatrixBuilder::new(presence, n_new, data_dir)`, ajoute les `n_dst_genomes` colonnes absentes, puis **ferme**. Les colonnes sources seront ajoutées par `resume`, groupe par groupe.
### 3c. Boucle sur les groupes
- `const SRC_GROUP: usize = 2;` (aligné sur `max_open`), avec un commentaire qui explique pourquoi.
- Itère sur `sources.chunks(SRC_GROUP)`, **dans l'ordre**, en tenant un `col_base` = nombre de colonnes sources déjà ajoutées avant ce groupe.
- Pour chaque groupe :
1. ouvre les couches de ses sources pour cette partition (`partition.layer(l)?.open()` puis `Arc::new`), comme aujourd'hui mais **seulement pour ce groupe** ;
2. `group_n` = somme des `src_n` du groupe ;
3. pour chaque couche dst existante, puis pour la nouvelle couche si `any_new` : `MatrixBuilder::resume`, vérifie `n_cols() == n_dst_genomes + col_base`, puis `add_col()` × `group_n` ;
4. construis les `pass2_items` du groupe. Le `col_offset` est **local au groupe** (0..group_n) ;
5. lance le pipeline de passe 2 existant, inchangé dans sa logique (Flat → Transform → Sink), sur ces items et ces builders. Les vecteurs `exist_sink`/`new_sink` ne contiennent que les colonnes du groupe ;
6. vérifie l'erreur de passe 2, ferme toutes les `ColBuilder` puis tous les `MatrixBuilder` du groupe ;
7. **drop** les couches du groupe avant de passer au suivant ;
8. `col_base += group_n`.
- L'initialisation `init_presence_matrix` (premier merge en présence) reste **avant** la boucle, inchangée.
- `dst_layers` (ouverts une seule fois) et `new_mphf` restent partagés par tous les groupes.
Invariant à documenter dans un commentaire : les colonnes sont ajoutées dans l'ordre des sources, donc la colonne absolue `n_dst_genomes + col_base + offset_local + g` est la même qu'avant la refonte.
### 3d. Vérification
`cargo test -p obikmerge`, `cargo clippy -p obikmerge`, puis `scripts/smoke_test_merge.sh --check benchmark/run/merge_oracle/synthetic` et `scripts/bench_merge_oracle.sh --check benchmark/run/merge_oracle/bench` (recompile le release avant). Les deux **doivent** passer sans aucun diff.
**Point d'arrêt.**
## Étape 4 — Plan C : colonnes construites en mémoire (`obicompactvec`)
But : plus aucun `MmapMut` ni fichier creux pour les colonnes **remplies** par le merge.
1. Lis `PersistentBitVecBuilder` (`bitvec.rs` : `new`, `set`, `close`), `PersistentCompactIntVecBuilder` (`builder.rs` : `new`, `set`, `close`), `format.rs` (`finalize_pciv`, `HEADER_SIZE`) et `mmap_file.rs` (`building_path`, `publish_built`). Note **exactement** les octets écrits à la fermeture : en-tête, données, overflow, `.building` puis `rename`.
2. Ajoute deux types :
- `MemBitVecBuilder` (dans `bitvec.rs`, ou un nouveau `mem_builder.rs` si `bitvec.rs` dépasse 500 lignes) : `words: Vec<u64>`, `n`, `path` ; `new(n, path)`, `set(slot, bool)`, `close() -> io::Result<()>` ;
- `MemCompactIntVecBuilder` : `primary: Vec<u8>`, `overflow: HashMap<usize, u32>`, `n`, `path` ; `new`, `set(slot, u32)` (même sémantique sentinelle 255 que le builder persistant), `close()`.
`close()` écrit avec un `BufWriter` dans `building_path(path)`, produit le **même fichier, octet pour octet**, que le builder persistant, fait `flush` + `sync_all`, puis `publish_built(path)`. Réutilise les fonctions de format existantes plutôt que de dupliquer la mise en page de l'en-tête. S'il faut factoriser l'écriture de l'en-tête ou de l'overflow pour la partager, fais-le dans `format.rs` / `mmap_file.rs`.
3. Dans `matrix_builder.rs` : ajoute les variantes `ColBuilder::MemBit(MemBitVecBuilder)` et `ColBuilder::MemInt(MemCompactIntVecBuilder)` (`set_val` et `close` les gèrent), plus `MatrixBuilder::add_mem_col() -> io::Result<ColBuilder>`. Ce dernier incrémente `n_cols` exactement comme `add_col`. Si les builders de matrice sous-jacents n'exposent ni le chemin de la prochaine colonne ni l'incrément, ajoute-leur une méthode `add_mem_col` symétrique de `add_col`. Le nommage des colonnes reste privé à la crate.
4. Tests unitaires (`src/obicompactvec/src/tests/`) : pour n ∈ {0, 1, 63, 64, 65, 10 000} et des valeurs pseudo-aléatoires (incluant des comptes ≥ 255), comparer **octet par octet** le fichier produit par le builder mémoire et celui du builder persistant. Vérifier aussi qu'une matrice construite avec `add_mem_col` se relit via `PersistentBitMatrix::open` / `PersistentIntMatrix::open`.
5. Dans `merge_partition`, remplace `add_col()` par `add_mem_col()` pour les colonnes sources (couches existantes et nouvelle couche). `add_absent_col` reste tel quel : on n'écrit jamais dans son mapping, donc aucun risque de SIGBUS.
6. Ajoute un commentaire sur le coût mémoire : `SRC_GROUP × src_n × (L_dst + 1) × (n/8 ou n)` octets au plus par partition.
7. `cargo test -p obicompactvec -p obikmerge`, clippy, puis `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check` (release recompilé).
`obikselect` et `obikfilter` ne sont **pas** migrés dans cette session.
**Point d'arrêt.**
## Étape 5 — Plan B : hors périmètre
**Ne pas implémenter.** Le plan B (ouverture « slots only » des couches dans `obikindex`) a été tranché par le développeur : ce sera un chantier séparé (forme retenue : composition, voir la section « Plan B » de `merge_mmap_audit.md`). Après l'étape 3, son gain en mmaps pour le merge est négligeable (~8 → ~2 par partition). Il touche en revanche le chemin de requête d'`obikindex` et concerne trois crates.
Les couches sources de la passe 2 continuent donc d'être ouvertes par `partition.layer(l)?.open()`, mais seulement groupe par groupe (étape 3). Ne crée aucun type ni méthode « lookup only » dans `obikindex` ou `obikmerge`.
## Étape 6 — Découpage, doc, finition
1. Découpe `partition_merge.rs` en module `partition_merge/` (par exemple `mod.rs` pour l'orchestration, `pass1.rs`, `pass2.rs`, `tests.rs`), chaque fichier sous 500 lignes, sans changement de comportement.
2. Supprime tout code devenu mort : `find_referencing_symbols` sur ce que la refonte a remplacé. Signale tout ce qui est devenu injustifié plutôt que de le garder par inertie.
3. `cargo test --workspace`, `cargo clippy --workspace`, `cargo check --workspace --features numa` si la feature existe, `scripts/smoke_test_index.sh`, `scripts/smoke_test_merge.sh --check` et `scripts/bench_merge_oracle.sh --check` (release recompilé ; reporte le temps de `merge_partitions` avant/après).
4. Documentation :
- `DevDocMD/implementation/merge_mmap_audit.md` : passe le statut à « implemented », en indiquant ce qui a été fait et les écarts éventuels au plan ;
- `DevDocMD/implementation/merge.md`, section « Algorithm » : décris le traitement par groupes de sources et les colonnes en mémoire ;
- vérifie si `UserDocMD/` décrit le comportement mémoire du merge, et le cas échéant mets-le à jour, en prose, sans code Rust ;
- mets à jour la `nav:` des `mkdocs*.yml` si un fichier a été ajouté, puis lance `qmd embed`.
5. Rapport final : liste des fichiers modifiés, résultats de tests, et estimation du nombre de mmaps par partition avant/après (calculée, marquée comme estimée). Termine par la commande de mesure à lancer en production, qui ne demande pas root :
```bash
PID=$(pgrep -f 'obikmer merge'); while sleep 60; do
echo "$(date +%T) maps=$(wc -l < /proc/$PID/maps) threads=$(ls /proc/$PID/task | wc -l)"; done
```
Ne commite rien : c'est moi qui le ferai.
+61
View File
@@ -0,0 +1,61 @@
# Prompt — où optimiser l'étape `pack` du merge (diagnostic d'abord)
> À coller tel quel au début d'une nouvelle session Claude Code ouverte à la racine du dépôt obikmer.
> Cette session est un **diagnostic** : elle produit une analyse chiffrée et des pistes classées, pas du code. Toute modification du code attend mon feu vert explicite, piste par piste.
---
## Contexte
`obikmer merge` se termine par une étape `finalize`, dont l'essentiel est `pack` : les matrices de présence/comptage, écrites en colonnes pendant le merge (`MatrixBuilder`), sont transposées en un format compact par partition (sparse par défaut). Le point d'entrée est `KmerIndex::finalize_indexed` (`src/obikindex/src/index/builder.rs`, étape « pack » vers la ligne 109), qui appelle `KmerIndex::pack_matrices(sparse)` (`src/obikindex/src/index/kmer_index.rs`). Les primitives sont dans `src/obicompactvec/src/` : `pack_bit_matrix` (`bitmatrix/packed.rs`), `pack_compact_int_matrix` (`intmatrix.rs`), `pack_sparse_compact_int_matrix` (`sparse_intmatrix.rs`), et le côté sparse des bits dans `bitmatrix/`.
Lis d'abord `DevDocMD/implementation/merge_scaling.md` (état du pack sparse : passage par une matrice dense séquentielle puis sparse, un seul mapping par partition) et `DevDocMD/implementation/merge_mmap_audit.md`.
## Mesure de référence (production, 2026-11-18)
Merge de 186 sources (`Cedrus`), 256 partitions, binaire glibc compilé localement, **192 CPU non contraints**, 17,5 milliards de k-mers nouveaux au total. Mode **présence**, compaction par défaut, donc **sparse** (pas de `--dense`) :
| Étape | Durée | Efficacité CPU | RSS max |
|---|---|---|---|
| `merge_partitions` | 748 s | 141×/192 (73 %) | 354 Go |
| `pack` | 231 s | 65×/192 (34 %) | **503,7 Go** |
La RAM de la machine est de 754 Go. Le RSS max est celui du process : il inclut les pages de fichiers mappés, pas seulement la mémoire anonyme. **Je n'ai pas encore séparé les deux.**
Progression de `pack` (partitions terminées par tranche de 10 %, début à 04:08:09) :
`+53 s` (10 %) · `+35 s` (20 %) · `+13 s` (30 %) · `+48 s` (40 %) · `+24 s` (50 %) · `+9 s` · `+6 s` · `+6 s` · `+6 s` · `+11 s` (100 %), puis `+20 s` avant la fin de l'étape. Les partitions les plus grosses passent en premier (tri décroissant dans `merge.rs`), mais je ne sais pas si `pack` les traite dans le même ordre ni si l'ordre est le bon.
## Ce que je veux savoir
1. **Où part le temps** de `pack` ? Lecture des colonnes, transposition, écriture, `sync`/`fsync`, attente d'un verrou, déséquilibre entre partitions ? Efficacité de 34 % sur 192 workers : qu'est-ce qui les bloque ?
2. **Où part la mémoire** (504 Go) ? Combien est de la mémoire anonyme, combien des pages de fichiers mappés ? Est-elle proportionnelle au nombre de workers ? Quelle est la mémoire vivante par partition en cours de `pack` ?
3. **Quel est le parallélisme réel** : combien de partitions `pack` traite-t-il en même temps, et qu'est-ce qui le fixe (`PartitionRunner`, `--cpu-max`, NUMA) ?
4. Y a-t-il du **travail inutile** (relectures, copies intermédiaires, colonnes vides ou toutes-absentes ré-écrites, étape dense qui pourrait être évitée) ?
## Méthode (impérative)
- **Lire et mesurer avant de proposer.** Pas de récit plausible sans preuve : tout ce qui n'est pas vérifié par lecture du code ou par mesure est marqué « hypothèse ».
- Pour mesurer sans la production : `scripts/docker_debug/` fournit un conteneur Linux (build glibc ou musl) qui lance un merge de N sources synthétiques ou dérivées de génomes réels (`run.sh`, variables `GEN`, `REAL`, `GENOME_LEN`, `TARGET`, `TOOL=heaptrack` pour glibc) avec un moniteur CSV de mappings et de RSS. Les génomes de 2 Mb donnent des partitions assez grosses pour que `pack` ait quelque chose à faire. `heaptrack` fonctionne sur le build glibc (pas sur le build musl statique). `gdb` ne fonctionne pas sous Rosetta.
- Pour séparer mémoire anonyme et fichiers mappés, utilise `/proc/PID/status` (`RssAnon`, `RssFile`, `RssShmem`), **jamais `smaps` en entier** sur un process à beaucoup de mappings (il verrouille `mmap_lock` et ralentit le process).
- Les estimations de coût (octets, secondes) sont marquées **estimées** ; les chiffres observés sont marqués **mesurés**.
## Règles du projet (non négociables)
- Avant tout travail de code : `mcp__serena__initial_instructions`, puis Serena (`find_symbol`, `get_symbols_overview`, `find_referencing_symbols`) pour naviguer. `rust-analyzer-mcp` et `qmd` quand ils servent. Consulte `context7` avant d'utiliser une API de bibliothèque externe.
- **Aucune commande VCS mutante** (git/jj : commit, add, mv, rm, describe, push…). Lecture seule autorisée.
- **Aucune nouvelle dépendance, aucun changement de version** sans mon accord explicite. Si un problème de bibliothèque se pose, expose-le et propose des alternatives.
- **Une question appelle une réponse, pas une action.** Ne modifie aucun fichier de `src/` sans demande explicite. Cette session est en lecture seule sur le code.
- Le code existant est une hypothèse, pas une vérité : signale tout ce qui est devenu injustifié plutôt que de le garder par inertie.
- Si tu proposes d'ajouter une opération à une structure partagée, mets-la comme méthode générique sur cette structure (crate `obicompactvec`), pas comme fonction libre dans le consommateur.
- Ne jamais matérialiser en mémoire une structure volumineuse qui peut être streamée.
- Échanges en **français**, documentation en **anglais**.
## Livrable
Un rapport (dans la conversation, pas dans un fichier, sauf si je le demande) avec :
1. Le chemin de code de `pack`, de `finalize_indexed` jusqu'aux écritures disque, résumé en quelques lignes par étape.
2. Pour chaque question ci-dessus : ce qui est **mesuré**, ce qui est **hypothèse**, et comment vérifier l'hypothèse.
3. Une liste de **pistes d'optimisation classées** par gain attendu/coût/risque. Pour chaque piste : ce qu'elle changerait, ce qu'elle risque, et comment on verrait qu'elle marche. Aucune implémentation.
4. Les vérifications de non-régression à prévoir : le résultat sur disque doit rester **identique octet pour octet** (voir les oracles `scripts/smoke_test_merge.sh` et `scripts/bench_merge_oracle.sh`).
Termine par la question : « Quelle piste veux-tu que j'implémente ? » et attends ma réponse.
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env bash
# bench_merge_oracle.sh — golden-output + timing oracle for `obikmer merge`
# on the real-data benchmark (benchmark/run/specimen_index_presence, 20
# presence indexes built from real genomes).
#
# Companion of smoke_test_merge.sh for refactors of obikmerge (see
# DevDocMD/implementation/merge_mmap_audit.md). Same merge options as
# benchmark/merge_presence.sh (--force --force-presence), but the output goes
# to a dedicated directory, never to benchmark/run/global_index_presence.
#
# What it does:
# 1. builds the release `obikmer`
# 2. merges every specimen presence index (sorted, as merge_presence.sh)
# and keeps the stderr log (stage report: wall/RSS of merge_partitions)
# 3. `obikmer dump --debug` of the result → header first, rows sorted
# (LC_ALL=C) → sha256 of that sorted text, plus a zstd copy
# 4. --record DIR: writes DIR/merged (the merged index), DIR/merge.stderr.log, DIR/dump.sha256,
# DIR/dump.sorted.csv.zst, and (informative cross-check)
# DIR/global_index_presence.sha256 = same dump hash computed on the
# existing benchmark/run/global_index_presence (older binary — a
# difference there is reported, not judged)
# --check DIR: merges into a scratch directory, compares the hash with
# DIR/dump.sha256 (full diff against the zstd copy only on mismatch),
# and prints merge_partitions wall/RSS and pack wall before → after
#
# Usage:
# scripts/bench_merge_oracle.sh (--record DIR | --check DIR) [--keep]
set -euo pipefail
KEEP=0
MODE=""
DIR=""
while [ $# -gt 0 ]; do
case "$1" in
--record) MODE=record; DIR="$2"; shift 2 ;;
--check) MODE=check; DIR="$2"; shift 2 ;;
--keep) KEEP=1; shift ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
done
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [--keep]" >&2; exit 2; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
RUN_DIR="$REPO_ROOT/benchmark/run"
IDX_DIR="$RUN_DIR/specimen_index_presence"
GLOBAL_OLD="$RUN_DIR/global_index_presence"
BIN="$REPO_ROOT/src/target/release/obikmer"
mkdir -p "$DIR" "$RUN_DIR/merge_oracle"
# Scratch space on the same file system as the benchmark data (merge's
# bootstrap hard-links from its base source; sort needs room for temp files).
WORK="$(mktemp -d "$RUN_DIR/merge_oracle/.work.XXXXXX")"
cleanup() {
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
}
trap cleanup EXIT
fail() {
echo "FAIL: $1" >&2
exit 1
}
# ── 1. build ────────────────────────────────────────────────────────────────
if ! (cd "$REPO_ROOT/src" && cargo build --release -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
cat "$WORK/build.log" >&2
fail "cargo build --release failed"
fi
sorted_dump() { # sorted_dump INDEX OUT_TXT — header first, rows sorted
local raw="$WORK/dump.raw.csv"
"$BIN" dump --debug "$1" > "$raw" 2> "$WORK/dump.log" \
|| { cat "$WORK/dump.log" >&2; fail "obikmer dump $1 exited non-zero"; }
{ head -n 1 "$raw"
tail -n +2 "$raw" | LC_ALL=C sort -S 25% --parallel=8 -T "$WORK"
} > "$2"
rm -f "$raw"
[ "$(head -n 2 "$2" | wc -l)" -eq 2 ] || fail "empty dump of $1"
}
hash_of() { sha256sum "$1" | cut -d' ' -f1; }
stage_field() { # stage_field LOG FIELD [STAGE] — FIELD of the 'done stage=STAGE' line (default merge_partitions)
perl -pe 's/\e\[[0-9;]*m//g' "$1" \
| grep "done stage=${3:-merge_partitions} " \
| sed -n "s/.*$2=\([^ ]*\( [KMG]*B\)\{0,1\}\).*/\1/p" | tail -n 1
}
# ── 2. merge ────────────────────────────────────────────────────────────────
mapfile -t SOURCES < <(find "$IDX_DIR" -mindepth 1 -maxdepth 1 -type d | sort)
[ "${#SOURCES[@]}" -gt 0 ] || fail "no indexes found in $IDX_DIR"
if [ "$MODE" = record ]; then
MERGED="$DIR/merged"
LOG="$DIR/merge.stderr.log"
else
MERGED="$WORK/merged"
LOG="$WORK/merge.stderr.log"
fi
echo "merging ${#SOURCES[@]} presence indexes → $MERGED"
if ! "$BIN" merge --output "$MERGED" --force --force-presence "${SOURCES[@]}" \
> /dev/null 2> "$LOG"; then
tail -n 40 "$LOG" >&2
fail "obikmer merge exited non-zero"
fi
WALL="$(stage_field "$LOG" wall_secs)"
RSS="$(stage_field "$LOG" rss)"
PACK_WALL="$(stage_field "$LOG" wall_secs pack)"
echo "merge_partitions: wall=${WALL}s rss=${RSS} pack: wall=${PACK_WALL}s"
# ── 3. sorted dump + hash ───────────────────────────────────────────────────
DUMP="$WORK/dump.sorted.csv"
sorted_dump "$MERGED" "$DUMP"
H="$(hash_of "$DUMP")"
echo "dump: $(($(wc -l < "$DUMP") - 1)) k-mers, sha256 $H"
# ── 4. record or check ──────────────────────────────────────────────────────
if [ "$MODE" = record ]; then
echo "$H" > "$DIR/dump.sha256"
zstd -q -T0 -f "$DUMP" -o "$DIR/dump.sorted.csv.zst"
if [ -d "$GLOBAL_OLD" ]; then
OLD_DUMP="$WORK/old.sorted.csv"
sorted_dump "$GLOBAL_OLD" "$OLD_DUMP"
HO="$(hash_of "$OLD_DUMP")"
echo "$HO" > "$DIR/global_index_presence.sha256"
if [ "$HO" = "$H" ]; then
echo "cross-check: global_index_presence (older binary) dump is identical"
else
echo "cross-check: global_index_presence (older binary) dump DIFFERS" \
"($(($(wc -l < "$OLD_DUMP") - 1)) k-mers, sha256 $HO) — informative only"
fi
fi
echo "RECORDED → $DIR"
exit 0
fi
[ -f "$DIR/dump.sha256" ] || fail "no reference in $DIR (run --record first)"
REF_H="$(cat "$DIR/dump.sha256")"
REF_WALL="$(stage_field "$DIR/merge.stderr.log" wall_secs)"
REF_RSS="$(stage_field "$DIR/merge.stderr.log" rss)"
REF_PACK="$(stage_field "$DIR/merge.stderr.log" wall_secs pack)"
echo "merge_partitions wall: ${REF_WALL}s → ${WALL}s rss: ${REF_RSS} → ${RSS}"
echo "pack wall: ${REF_PACK}s → ${PACK_WALL}s"
python3 -c "import sys; r,n=float(sys.argv[1]),float(sys.argv[2]); print(f'merge_partitions wall ratio: {n/r:.2f}' + (' (> 1.20: SLOWER THAN 20 %)' if n > 1.2*r else ''))" \
"$REF_WALL" "$WALL" || true
if [ "$H" = "$REF_H" ]; then
echo "PASS: dump identical to reference ($H)"
exit 0
fi
echo "FAIL: dump hash differs (ref $REF_H, now $H)" >&2
zstd -q -d -c "$DIR/dump.sorted.csv.zst" | { diff - "$DUMP" || true; } | head -n 40 >&2
exit 1
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env python3
# check_unitig_chunks.py GLOB... — report unitigs.bin chunks holding more than 128 k-mers.
# Indexes built before MAX_KMERS_PER_CHUNK went from 256 to 128 (2026-10-15) could hold chunks whose
# rank does not fit the 7-bit evidence field; the k-mers beyond rank 127 of such a chunk then fail
# exact verification. Example: scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"
# (k is hard-coded below — edit it for an index built with another k.)
import sys,glob,mmap
k=31
def scan(path):
with open(path,'rb') as f:
m=mmap.mmap(f.fileno(),0,access=mmap.ACCESS_READ)
o=0; n=len(m); mx=0; over=0; chunks=0
while o<n:
c=m[o]+1; chunks+=1
if c>mx: mx=c
if c>128: over+=1
o+=1+(m[o]+k+3)//4
return mx,over,chunks
tot_over=0
for pat in sys.argv[1:]:
for f in sorted(glob.glob(pat)):
mx,over,ch=scan(f); tot_over+=over
print(f"{f.split('partitions/')[1]}: chunks={ch} max_kmers={mx} chunks>128={over}")
print("TOTAL chunks over 128 kmers:",tot_over)
+16
View File
@@ -0,0 +1,16 @@
# Linux debug environment for the merge mapping/memory-growth hunt.
# Same toolchain as .gitea/workflows/release.yml for the musl target
# (cargo-zigbuild + zig cc), plus a glibc toolchain and diagnostic tools.
FROM rust:1-bookworm
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential cmake pkg-config autoconf automake libtool \
heaptrack valgrind gdb strace procps python3 python3-pip \
&& rm -rf /var/lib/apt/lists/*
RUN pip install ziglang --quiet --break-system-packages \
&& cargo install cargo-zigbuild \
&& rustup target add x86_64-unknown-linux-musl
RUN ZIG=$(python3 -c "import ziglang, os; print(os.path.join(os.path.dirname(ziglang.__file__), 'zig'))") \
&& printf '#!/bin/sh\nexec "%s" cc -target x86_64-linux-musl "$@"\n' "$ZIG" > /usr/local/bin/x86_64-linux-musl-gcc \
&& printf '#!/bin/sh\nexec "%s" c++ -target x86_64-linux-musl "$@"\n' "$ZIG" > /usr/local/bin/x86_64-linux-musl-g++ \
&& chmod +x /usr/local/bin/x86_64-linux-musl-gcc /usr/local/bin/x86_64-linux-musl-g++
WORKDIR /work
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Builds the debug image. PLATFORM defaults to linux/amd64 (Rosetta on Apple Silicon).
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")"
PLATFORM="${PLATFORM:-linux/amd64}"
docker build --platform "$PLATFORM" -t "obikmer-debug:${PLATFORM//\//-}" .
+82
View File
@@ -0,0 +1,82 @@
#!/usr/bin/env bash
# Index compatibility test between two obikmer binaries that differ only by
# their ptr_hash version (A = baseline, B = candidate).
#
# Usage: compat_ptrhash.sh VOL_A VOL_B OUT
# VOL_A / VOL_B docker volumes holding a musl release build (CARGO_TARGET_DIR)
# OUT host directory receiving report.txt
#
# What it checks (everything compared by md5 of dump/query output):
# T0 on-disk bytes of an index built by A vs by B (informational)
# T1 A-built single-genome indexes read by A and by B
# T2 merges: A merging A-built sources (reference), B merging A-built
# sources, B merging B-built sources; each result dumped by A and by B
# T3 queries (hits and misses) on indexes read by A and by B
# T4 incremental merge into an existing A-built merged index, by A and B
set -u
VOL_A="$1"; VOL_B="$2"; OUT="$3"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO="$(cd "$HERE/../.." && pwd)"
mkdir -p "$OUT"
docker run --rm --platform linux/amd64 \
-v "$VOL_A:/va:ro" -v "$VOL_B:/vb:ro" \
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" -v "$OUT:/out" \
obikmer-debug:linux-amd64 bash -c '
set -u
R=x86_64-unknown-linux-musl/release/obikmer
mkdir -p /tmp/bin && cp /va/$R /tmp/bin/A && cp /vb/$R /tmp/bin/B
A=/tmp/bin/A; B=/tmp/bin/B
W=/tmp/w; rm -rf $W; mkdir -p $W/ia $W/ib
exec > >(tee /out/report.txt) 2>&1
echo "A sha: $(sha256sum $A | cut -c1-16) B sha: $(sha256sum $B | cut -c1-16)"
mapfile -t G < <(ls -S /genomes_ro/*.fna.gz | sed -n 8,12p)
echo "genomes: ${#G[@]}"; printf " %s\n" "${G[@]##*/}"
mkdir -p $W/fa; n=0; for gz in "${G[@]}"; do n=$((n+1)); g=$W/fa/L$n.fasta; zcat "$gz" > "$g"
$A index -k 21 -m 9 --theta 0 -p 256 --label L$n -o $W/ia/L$n.idx "$g" 2>$W/ia/L$n.err >/dev/null || { echo "A index failed L$n"; tail -2 $W/ia/L$n.err; }
$B index -k 21 -m 9 --theta 0 -p 256 --label L$n -o $W/ib/L$n.idx "$g" 2>$W/ib/L$n.err >/dev/null || { echo "B index failed L$n"; tail -2 $W/ib/L$n.err; }
done
echo; echo "== T0 on-disk bytes, index built by A vs by B"
nd=$(diff -rq $W/ia $W/ib | wc -l); nt=$(find $W/ia -type f | wc -l)
echo "files differing: $nd / $nt"
diff -rq $W/ia $W/ib | sed "s#$W/##g" | awk "{print \$2}" | sed -E "s#/partitions/.*/#/…/#" | sort | uniq -c | sort -rn | head -8
h() { md5sum | cut -c1-12; }
echo; echo "== T1 A-built single-genome indexes: dump by A vs by B"
for i in 1 2 3 4 5; do
a=$($A dump $W/ia/L$i.idx 2>/dev/null | h); b=$($B dump $W/ia/L$i.idx 2>/dev/null | h)
echo "L$i A:$a B:$b $([ "$a" = "$b" ] && echo OK || echo DIFF)"
done
cp_src() { rm -rf $2; mkdir -p $2; for i in $1; do cp -r $3/L$i.idx $2/; done; }
echo; echo "== T2 merges of 4 genomes (presence)"
cp_src "1 2 3 4" $W/s_aa $W/ia; cp_src "1 2 3 4" $W/s_ba $W/ia; cp_src "1 2 3 4" $W/s_bb $W/ib
$A merge --output $W/M_AA --force --force-presence $W/s_aa/L*.idx 2>$W/m_aa.err; echo "M_AA rc=$?"
$B merge --output $W/M_BA --force --force-presence $W/s_ba/L*.idx 2>$W/m_ba.err; echo "M_BA rc=$? (B merging A-built sources)"
$B merge --output $W/M_BB --force --force-presence $W/s_bb/L*.idx 2>$W/m_bb.err; echo "M_BB rc=$? (B merging B-built sources)"
for X in M_AA M_BA M_BB; do for T in A B; do
eval "BIN=\$$T"; $BIN dump $W/$X 2>/dev/null > $W/d_${X}_$T.csv
echo "dump $X by $T: $(wc -l < $W/d_${X}_$T.csv) lines raw:$(h < $W/d_${X}_$T.csv) sorted:$(LC_ALL=C sort -S 2G $W/d_${X}_$T.csv | h)"
done; done
echo; echo "== T3 queries"
head -c 60000 $W/fa/L1.fasta > $W/q.fa
python3 - <<PY >> $W/q.fa
import random
random.seed(1)
print(">rand")
print("".join(random.choice("ACGT") for _ in range(5000)))
PY
for X in ia/L1.idx M_AA M_BA M_BB; do for T in A B; do
eval "BIN=\$$T"; echo "query $X by $T: $($BIN query $W/$X $W/q.fa 2>/dev/null | h)"
done; done
echo; echo "== T4 incremental merge (existing A-built merged index + a 5th genome)"
for T in A B; do
eval "BIN=\$$T"; rm -rf $W/inc_$T $W/dst_$T; cp -r $W/M_AA $W/dst_$T; cp -r $W/ia/L5.idx $W/l5_$T.idx
$BIN merge --output $W/inc_$T --force --force-presence $W/dst_$T $W/l5_$T.idx 2>$W/inc_$T.err; echo "inc by $T rc=$?"
$BIN dump $W/inc_$T 2>/dev/null | h
done
echo; echo done
'
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# Follow-up to compat_ptrhash.sh: determinism control (A built twice) and the
# incremental merge with its stderr shown. Usage: compat_ptrhash2.sh VOL_A VOL_B OUT
set -u
VOL_A="$1"; VOL_B="$2"; OUT="$3"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; REPO="$(cd "$HERE/../.." && pwd)"
mkdir -p "$OUT"
docker run --rm --platform linux/amd64 -v "$VOL_A:/va:ro" -v "$VOL_B:/vb:ro" \
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" -v "$OUT:/out" obikmer-debug:linux-amd64 bash -c '
set -u
R=x86_64-unknown-linux-musl/release/obikmer
mkdir -p /tmp/bin && cp /va/$R /tmp/bin/A && cp /vb/$R /tmp/bin/B; A=/tmp/bin/A; B=/tmp/bin/B
W=/tmp/w; rm -rf $W; mkdir -p $W/i1 $W/i2 $W/ib $W/fa
exec > >(tee /out/report2.txt) 2>&1
mapfile -t G < <(ls -S /genomes_ro/*.fna.gz | sed -n 8,12p)
n=0; for gz in "${G[@]}"; do n=$((n+1)); zcat "$gz" > $W/fa/L$n.fasta; done
h() { md5sum | cut -c1-12; }
echo "== C1 determinism control: same binary A, same input, built twice"
$A index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/i1/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
$A index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/i2/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
$B index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/ib/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
echo "A vs A: $(diff -rq $W/i1 $W/i2 | wc -l) / $(find $W/i1 -type f | wc -l) files differ"
echo "A vs B: $(diff -rq $W/i1 $W/ib | wc -l) / $(find $W/i1 -type f | wc -l) files differ"
for f in mphf.bin unitigs.bin evidence.bin unitigs.bin.idx; do
p=$(find $W/i1 -name $f | head -1); rel=${p#$W/i1/}
echo " $f AvsA: $(find $W/i1 -name $f | while read q; do cmp -s $q $W/i2/${q#$W/i1/} || echo x; done | wc -l)/256 differ AvsB: $(find $W/i1 -name $f | while read q; do cmp -s $q $W/ib/${q#$W/i1/} || echo x; done | wc -l)/256 differ"
done
echo; echo "== C2 incremental merge, stderr shown"
for i in 2 3 4 5; do $A index -k 21 -m 9 --theta 0 -p 256 --label L$i -o $W/i1/L$i.idx $W/fa/L$i.fasta >/dev/null 2>&1; done
mkdir -p $W/s; for i in 1 2 3; do cp -r $W/i1/L$i.idx $W/s/; done
$A merge --output $W/M3 --force --force-presence $W/s/L*.idx 2>/dev/null; echo "M3 rc=$?"
for T in A B; do
eval "BIN=\$$T"; rm -rf $W/inc_$T $W/dst_$T $W/l4_$T.idx; cp -r $W/M3 $W/dst_$T; cp -r $W/i1/L4.idx $W/l4_$T.idx
$BIN merge --output $W/inc_$T --force --force-presence $W/dst_$T $W/l4_$T.idx 2>$W/inc_$T.err; rc=$?
echo "inc by $T rc=$rc"; sed "s/\x1b\[[0-9;]*m//g" $W/inc_$T.err | tail -3
[ $rc = 0 ] && echo " dump sorted: $($BIN dump $W/inc_$T 2>/dev/null | LC_ALL=C sort -S 2G | h)"
done
echo done
'
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env python3
"""gen_fasta.py N OUTDIR [GENOME_LEN] — N deterministic synthetic genomes.
Each genome = unique random sequence + a random subset of a shared fragment
pool (some fragments lightly mutated), so a merge both hits k-mers already
in the destination and creates new ones, like related genomes would.
"""
import random, sys
n, out = int(sys.argv[1]), sys.argv[2]
glen = int(sys.argv[3]) if len(sys.argv) > 3 else 150_000
seed = int(sys.argv[4]) if len(sys.argv) > 4 else 20261118
rng = random.Random(seed)
def rnd(m):
return "".join(rng.choices("ACGT", k=m))
def mutate(s, rate=0.01):
s = list(s)
for i in range(len(s)):
if rng.random() < rate:
s[i] = rng.choice("ACGT")
return "".join(s)
pool = [rnd(2000) for _ in range(300)] # 600 kb shared pool
for g in range(n):
parts = [rnd(glen)]
for f in rng.sample(range(len(pool)), 60):
parts.append(mutate(pool[f]) if rng.random() < 0.3 else pool[f])
rng.shuffle(parts)
with open(f"{out}/G{g:03d}.fasta", "w") as fh:
for j, p in enumerate(parts):
fh.write(f">G{g:03d}_{j}\n{p}\n")
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""gen_fasta_real.py N OUTDIR GENOMES_DIR [WINDOW] [SEED] [MUT]
N derived genomes built from *real* genomes (GENOMES_DIR/*.fna.gz): each one is
a random window of WINDOW bp of a randomly chosen real genome, point-mutated at
rate MUT (default 3 %, which alters ~47 % of 21-mers), so a merge both finds
k-mers already present in the destination and creates new ones, with realistic
sequence statistics (no dedicated degenerate structure).
"""
import glob, gzip, random, sys
n, out, gdir = int(sys.argv[1]), sys.argv[2], sys.argv[3]
window = int(sys.argv[4]) if len(sys.argv) > 4 else 300_000
seed = int(sys.argv[5]) if len(sys.argv) > 5 else 20261118
mut = float(sys.argv[6]) if len(sys.argv) > 6 else 0.03
rng = random.Random(seed)
def read_fna(path):
seq = []
with gzip.open(path, "rt") as fh:
for line in fh:
if not line.startswith(">"):
seq.append(line.strip().upper())
return "".join(seq)
genomes = [read_fna(p) for p in sorted(glob.glob(f"{gdir}/*.fna.gz"))]
genomes = [g for g in genomes if len(g) > window]
for i in range(n):
g = rng.choice(genomes)
start = rng.randrange(0, len(g) - window)
s = list(g[start:start + window])
for j in range(len(s)):
if s[j] in "ACGT" and rng.random() < mut:
s[j] = rng.choice("ACGT")
seq = "".join(c if c in "ACGT" else "A" for c in s)
with open(f"{out}/R{i:03d}.fasta", "w") as fh:
fh.write(f">R{i:03d}\n")
for k in range(0, len(seq), 80):
fh.write(seq[k:k + 80] + "\n")
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# monitor.sh PID OUT [INTERVAL_S] — samples mappings / VM / commit figures until PID exits.
PID=$1; OUT=$2; DT=${3:-5}
printf 'time,maps,vmsize_kb,vmdata_kb,vmrss_kb,threads,committed_as_kb,commit_limit_kb\n' >"$OUT"
while kill -0 "$PID" 2>/dev/null; do
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null) || break
st=$(grep -E '^(VmSize|VmData|VmRSS|Threads):' "/proc/$PID/status" | awk '{printf "%s,", $2}')
mi=$(grep -E '^(Committed_AS|CommitLimit):' /proc/meminfo | awk '{printf "%s,", $2}')
echo "$(date +%T),$maps,${st}${mi%,}" >>"$OUT"
sleep "$DT"
done
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
# run.sh — builds obikmer (release, debug symbols) and merges the benchmark presence
# indexes in the container while sampling mappings/RSS/commit in parallel.
#
# Env:
# PLATFORM linux/amd64 (default, Rosetta) | linux/arm64
# GEN if set: build GEN synthetic indexes (gen_fasta.py) instead of using
# the benchmark ones; GENOME_LEN (default 150000), K (21), M (9), PART (256)
# REAL if set (with GEN): derive genomes from benchmark/run/genomes (gen_fasta_real.py)
# SRC_DIR alternative source tree (default: <repo>/src); VARIANT: volume suffix;
# LOCKED=0 to allow Cargo.lock changes (needed when a dep is added)
# NSRC number of source indexes to merge (default: all 20)
# TARGET gnu (default, glibc) | musl (static, same chain as the release CI)
# TOOL none (default) | heaptrack (heaptrack output goes to OUT/)
# MAXMAP if set, vm.max_map_count is lowered to this inside the VM for the
# run and restored afterwards (needs --privileged; affects the whole
# OrbStack VM while it runs)
# MALLOC_ARENA_MAX, MALLOC_MMAP_THRESHOLD_, MALLOC_TRIM_THRESHOLD_ passed through
# OUT host output directory (default: ./out/<timestamp>)
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO="$(cd "$HERE/../.." && pwd)"
PLATFORM="${PLATFORM:-linux/amd64}"
NSRC="${NSRC:-20}"
TOOL="${TOOL:-none}"
TARGET="${TARGET:-gnu}"
OUT="${OUT:-$HERE/out/$(date +%Y%m%d-%H%M%S)}"
mkdir -p "$OUT"
IMG="obikmer-debug:${PLATFORM//\//-}"
VOL="obikmer-debug-target-${PLATFORM//\//-}-${TARGET}${VARIANT:+-$VARIANT}"
docker run --rm --privileged --platform "$PLATFORM" \
-v "${SRC_DIR:-$REPO/src}:/work/src:$([ -n "${SRC_DIR:-}" ] && echo rw || echo ro)" \
-v "$REPO/benchmark/run/specimen_index_presence:/idx_ro:ro" \
-v "$VOL:/target" \
-v "$OUT:/out" \
-v "$HERE/monitor.sh:/monitor.sh:ro" \
-v "$HERE/gen_fasta.py:/gen_fasta.py:ro" \
-v "$HERE/gen_fasta_real.py:/gen_fasta_real.py:ro" \
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" \
-e CARGO_TARGET_DIR=/target -e CARGO_PROFILE_RELEASE_DEBUG=1 \
-e GEN="${GEN:-}" -e LOCKED_FLAG="$([ "${LOCKED:-1}" = 0 ] || echo --locked)" -e SEED="${SEED:-20261118}" -e REAL="${REAL:-}" -e GENOME_LEN="${GENOME_LEN:-150000}" -e K="${K:-21}" -e M="${M:-9}" -e PART="${PART:-256}" -e RUST_LOG="${RUST_LOG:-}" -e NSRC="$NSRC" -e TARGET="$TARGET" -e TOOL="$TOOL" -e MAXMAP="${MAXMAP:-}" \
-e MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-}" \
-e MALLOC_MMAP_THRESHOLD_="${MALLOC_MMAP_THRESHOLD_:-}" \
-e MALLOC_TRIM_THRESHOLD_="${MALLOC_TRIM_THRESHOLD_:-}" \
"$IMG" bash -c '
set -euo pipefail
for v in MALLOC_ARENA_MAX MALLOC_MMAP_THRESHOLD_ MALLOC_TRIM_THRESHOLD_ RUST_LOG; do
[ -n "${!v}" ] || unset $v
done
if [ "$TARGET" = musl ]; then
PKG_CONFIG_ALLOW_CROSS=1 cargo zigbuild --release $LOCKED_FLAG --target x86_64-unknown-linux-musl \
--manifest-path /work/src/Cargo.toml --bin obikmer 2>&1 | tail -3
BIN=/target/x86_64-unknown-linux-musl/release/obikmer
else
cargo build --release $LOCKED_FLAG --manifest-path /work/src/Cargo.toml --bin obikmer 2>&1 | tail -3
BIN=/target/release/obikmer
fi
file "$BIN" 2>/dev/null | cut -c1-150 || true
# merge writes into its sources (lock files…): work on a copy, never on benchmark/
rm -rf /tmp/idx && mkdir -p /tmp/idx
if [ -n "$GEN" ]; then
mkdir -p /tmp/fa
if [ -n "$REAL" ]; then python3 /gen_fasta_real.py "$GEN" /tmp/fa /genomes_ro "$GENOME_LEN" "$SEED"
else python3 /gen_fasta.py "$GEN" /tmp/fa "$GENOME_LEN" "$SEED"; fi
NSRC=$GEN
echo "indexing $GEN synthetic genomes (k=$K m=$M p=$PART)…"
ls /tmp/fa/*.fasta | xargs -P 8 -I{} sh -c \
"b=\$(basename {} .fasta); $BIN index -k $K -m $M --theta 0 -p $PART --label \$b -o /tmp/idx/\$b.idx {} >/dev/null 2>&1 || echo index failed: \$b"
else
cp -r /idx_ro/. /tmp/idx/
fi
mapfile -t SRC < <(find /tmp/idx -mindepth 1 -maxdepth 1 -type d | sort | head -n "$NSRC")
echo "sources: ${#SRC[@]} target: $TARGET tool: $TOOL max_map_count: $(cat /proc/sys/vm/max_map_count)"
if [ -n "$MAXMAP" ]; then
OLD=$(cat /proc/sys/vm/max_map_count); sysctl -w vm.max_map_count="$MAXMAP" >/dev/null
trap "sysctl -w vm.max_map_count=$OLD >/dev/null" EXIT
fi
rm -rf /tmp/merged
CMD=("$BIN" merge --output /tmp/merged --force --force-presence "${SRC[@]}")
if [ "$TOOL" = heaptrack ]; then CMD=(heaptrack -o /out/heaptrack "${CMD[@]}"); fi
"${CMD[@]}" 2>/out/merge.stderr &
PID=$!
/monitor.sh "$PID" /out/monitor.csv 1 &
MON=$!
set +e; wait "$PID"; RC=$?; set -e
wait "$MON" || true
echo "merge exit code: $RC" | tee /out/exit_code.txt
tail -5 /out/merge.stderr
'
echo "outputs: $OUT"
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env bash
# monitor_pack.sh — 1 Hz sampler for I/O- and memory-bound stages (pack, merge_partitions).
#
# Usage:
# scripts/monitor_pack.sh --pid PID OUT_DIR [OUTPUT_INDEX_DIR] # attach to a running process
# scripts/monitor_pack.sh --cmd OUT_DIR [OUTPUT_INDEX_DIR] -- CMD... # start CMD and sample it
# env: DT=1 (seconds between samples)
#
# Writes OUT_DIR/samples.csv (one row per sample) and OUT_DIR/env.txt (context
# needed to read it). Linux only. Never reads smaps (it takes mmap_lock and
# slows a process with many mappings); only /proc/PID/{status,io,task/*/stat}
# and /proc/meminfo.
#
# Columns of samples.csv
# t epoch seconds
# rss_anon_kb, rss_file_kb, rss_shmem_kb from /proc/PID/status
# threads thread count
# n_R n_D n_S threads running / in uninterruptible sleep (I/O wait) / sleeping
# (futex, channel, sleep: NOT disk). n_D is the direct count of
# workers blocked on I/O.
# maps number of mappings (wc -l /proc/PID/maps)
# read_bytes write_bytes /proc/PID/io (cumulative, bytes submitted to the
# block layer; buffered writes counted at page dirtying)
# rchar wchar /proc/PID/io (cumulative, bytes through read()/write() syscalls)
# dirty_kb writeback_kb cached_kb mem_avail_kb /proc/meminfo
#
# Pair it with the process's own log (RUST_LOG=info, plus obisys=debug for the
# "activated N worker(s)" lines and obikindex=debug for per-partition timings)
# to align stages on wall-clock time: the log timestamps are UTC ISO-8601, `t`
# is epoch seconds.
set -euo pipefail
MODE=${1:-}; shift || true
case "$MODE" in
--pid) PID=${1:?PID}; shift ;;
--cmd) ;;
*) sed -n 2,12p "$0"; exit 2 ;;
esac
OUT=${1:?OUT_DIR}; shift
IDX=""
if [ "${1:-}" != "" ] && [ "${1:-}" != "--" ]; then IDX=$1; shift; fi
mkdir -p "$OUT"
if [ "$MODE" = "--cmd" ]; then
[ "${1:-}" = "--" ] || { echo "--cmd needs: -- CMD..." >&2; exit 2; }
shift
"$@" 2>"$OUT/cmd.stderr" &
PID=$!
fi
{
echo "date_utc: $(date -u +%FT%TZ)"
echo "pid: $PID"
echo "nproc: $(nproc)"
echo "kernel: $(uname -r)"
grep -E 'MemTotal|SwapTotal' /proc/meminfo
for k in vm.dirty_ratio vm.dirty_background_ratio vm.dirty_bytes vm.dirty_background_bytes \
vm.dirty_expire_centisecs vm.dirty_writeback_centisecs vm.max_map_count; do
echo "$k: $(sysctl -n $k 2>/dev/null || echo '?')"
done
echo "ulimit -n: $(ulimit -n)"
echo "cmdline: $(tr '\0' ' ' </proc/$PID/cmdline 2>/dev/null)"
if [ -n "$IDX" ]; then
echo "output_dir: $IDX"
echo "fs_type: $(stat -f -c %T "$IDX" 2>/dev/null || echo '?')"
echo "mount: $(df -P "$IDX" 2>/dev/null | tail -1)"
command -v beegfs-ctl >/dev/null && echo "beegfs: $(beegfs-ctl --getentryinfo "$IDX" 2>&1 | head -8 | tr '\n' ' ')"
fi
env | grep -E '^(MALLOC_|RUST_LOG|RAYON_)' || true
} >"$OUT/env.txt"
DT=${DT:-1}
printf 't,rss_anon_kb,rss_file_kb,rss_shmem_kb,threads,n_R,n_D,n_S,maps,read_bytes,write_bytes,rchar,wchar,dirty_kb,writeback_kb,cached_kb,mem_avail_kb\n' >"$OUT/samples.csv"
while [ -d "/proc/$PID" ]; do
t=$(date +%s)
st=$(awk '/^RssAnon:/{a=$2} /^RssFile:/{f=$2} /^RssShmem:/{s=$2} /^Threads:/{th=$2} END{printf "%s,%s,%s,%s",a,f,s,th}' "/proc/$PID/status" 2>/dev/null) || break
# thread states: field 3 of /proc/PID/task/TID/stat, after the ")" closing comm
states=$(cat /proc/$PID/task/*/stat 2>/dev/null | sed -E 's/^.*\) (.).*/\1/' | sort | uniq -c \
| awk '{c[$2]=$1} END{printf "%d,%d,%d", c["R"]+0, c["D"]+0, c["S"]+0}')
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null || echo 0)
io=$(awk '/^rchar:/{rc=$2} /^wchar:/{wc=$2} /^read_bytes:/{r=$2} /^write_bytes:/{w=$2} END{printf "%s,%s,%s,%s",r,w,rc,wc}' "/proc/$PID/io" 2>/dev/null || echo ",,,")
mi=$(awk '/^Dirty:/{d=$2} /^Writeback:/{w=$2} /^Cached:/{c=$2} /^MemAvailable:/{m=$2} END{printf "%s,%s,%s,%s",d,w,c,m}' /proc/meminfo)
echo "$t,$st,$states,$maps,$io,$mi" >>"$OUT/samples.csv"
sleep "$DT"
done
if [ "$MODE" = "--cmd" ]; then wait "$PID" || echo "command exit code: $?" >>"$OUT/env.txt"; fi
# ── summary ──────────────────────────────────────────────────────────────────
awk -F, 'NR==1{next}
{ n++; if(NR==2){t0=$1; r0=$10; w0=$11}
if($2>ma)ma=$2; if($3>mf)mf=$3; if($4>ms)ms=$4; if($5>mt)mt=$5
if($6>mR)mR=$6; if($7>mD)mD=$7; if($8>mS)mS=$8; sR+=$6; sD+=$7; sS+=$8
if($14>md)md=$14; if($15>mw)mw=$15; if($9>mm)mm=$9
t1=$1; r1=$10; w1=$11 }
END{ if(!n)exit
printf "samples: %d over %ds\n", n, t1-t0
printf "peak RssAnon %.1f GB | RssFile %.1f GB | RssShmem %.1f GB | threads %d | maps %d\n", ma/1e6, mf/1e6, ms/1e6, mt, mm
printf "threads by state (mean / max): R %.1f / %d | D(I/O wait) %.1f / %d | S %.1f / %d\n", sR/n, mR, sD/n, mD, sS/n, mS
printf "peak Dirty %.1f GB | Writeback %.1f GB\n", md/1e6, mw/1e6
printf "block I/O over the run: read %.1f GB, write %.1f GB\n", (r1-r0)/1e9, (w1-w0)/1e9 }' "$OUT/samples.csv" | tee "$OUT/summary.txt"
+23 -1
View File
@@ -125,4 +125,26 @@ if [ "$FOUND" -ne 1 ]; then
fail "no candidate k-mer round-tripped (tried ${#CANDIDATES[@]}) — likely a real regression, not a low-complexity fixture" fail "no candidate k-mer round-tripped (tried ${#CANDIDATES[@]}) — likely a real regression, not a low-complexity fixture"
fi fi
echo "PASS: index+query round-trip OK — $N_KMERS kmers indexed, query k-mer '$QUERY_KMER' found" # ── 5. regressions: (a) many partitions against a tiny input leaves most of
# them empty (no layer 0 on disk) — must still build; (b) an NCBI-style
# `.fna.gz` name must be read, not silently skipped (it used to be
# taken for a directory and ignored, yielding an empty index).
count_kmers() { # $1 = output index dir, rest = obikmer index args
local out="$1"; shift
local log="$WORK/regress.log"
( cd "$REPO_ROOT/src" && cargo run -q -p obikmer --bin obikmer -- \
index -k "$K" -m "$M" --theta 0 -o "$out" "$@" ) > "$log" 2>&1 \
|| { cat "$log" >&2; fail "obikmer index $* exited non-zero"; }
grep -o '[0-9]* total kmers indexed' "$log" | grep -o '^[0-9]*' || echo 0
}
N_EMPTY="$(count_kmers "$WORK/empty_parts.idx" -p 256 "$WORK/test.fasta")"
[ "$N_EMPTY" = "$N_KMERS" ] \
|| fail "-p 256 (mostly empty partitions) indexed $N_EMPTY kmers, expected $N_KMERS"
gzip -c "$WORK/test.fasta" > "$WORK/test.fna.gz"
N_GZ="$(count_kmers "$WORK/gz.idx" -p "$PARTITIONS" "$WORK/test.fna.gz")"
[ "$N_GZ" = "$N_KMERS" ] \
|| fail ".fna.gz input indexed $N_GZ kmers, expected $N_KMERS (input skipped?)"
echo "PASS: index+query round-trip OK — $N_KMERS kmers indexed, query k-mer '$QUERY_KMER' found; empty-partition and .fna.gz regressions OK"
+217
View File
@@ -0,0 +1,217 @@
#!/usr/bin/env bash
# smoke_test_merge.sh — end-to-end golden-output oracle for `obikmer merge`
#
# Non-regression oracle for refactors of obikmerge (see
# DevDocMD/implementation/merge_mmap_audit.md). Like smoke_test_index.sh it
# drives the real CLI, never the library APIs.
#
# What it does:
# 1. builds `obikmer` (debug)
# 2. generates (on --record) 4 small deterministic FASTA files A, B, C, D sharing
# fragments, so that merge both hits k-mers already present in the
# destination ("existing layer" path) and creates new ones ("new layer"
# path). D carries a short motif repeated > 255 times so the count track
# exercises the compact-int overflow (counts >= 255).
# 3. on --record, indexes each genome twice (presence, --with-counts) and
# freezes the indexes in DIR/sources; --check reuses them (see below),
# with -p PARTITIONS partitions
# 4. runs the merge scenarios:
# s1_presence / s1_count merge A B C D
# s3_dense merge A B C D --dense (presence)
# Scenario 2 (merge A B -> AB, then merge AB C D: a packed merge output
# re-merged as a *source*) is checked relatively, not against the
# golden dumps: its table (k-mer x genome label) must equal s1_presence's,
# whatever the column order and layer placement. A packed index cannot
# be a merge *destination* (its matrices are no longer column-by-column,
# see `can_be_destination` in obikmerge); `choose_base` therefore picks
# an unpacked source, and merging only packed indexes is refused with a
# clear error, which this script also checks.
# 5. dumps every result with `obikmer dump --debug` (partition + layer
# columns included, so a k-mer moving between layers is caught), keeps
# the CSV header first and sorts the rows (LC_ALL=C)
# 6. --record DIR writes the sorted dumps to DIR;
# --check DIR regenerates them and diffs against DIR
#
# Usage:
# scripts/smoke_test_merge.sh (--record DIR | --check DIR) [-p PARTITIONS] [--keep]
set -euo pipefail
K=21
M=9
PARTITIONS=4
KEEP=0
MODE=""
DIR=""
while [ $# -gt 0 ]; do
case "$1" in
--record) MODE=record; DIR="$2"; shift 2 ;;
--check) MODE=check; DIR="$2"; shift 2 ;;
-p) PARTITIONS="$2"; shift 2 ;;
--keep) KEEP=1; shift ;;
*) echo "unknown argument: $1" >&2; exit 2 ;;
esac
done
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [-p N] [--keep]" >&2; exit 2; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
WORK="$(mktemp -d -t obikmer_merge_smoke.XXXXXX)"
cleanup() {
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
}
trap cleanup EXIT
fail() {
echo "FAIL: $1" >&2
exit 1
}
# ── 1. build ────────────────────────────────────────────────────────────────
if ! (cd "$REPO_ROOT/src" && cargo build -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
cat "$WORK/build.log" >&2
fail "cargo build failed"
fi
BIN="$REPO_ROOT/src/target/debug/obikmer"
run() { # run LOGNAME CMD... — runs obikmer, dumps its log on failure
local log="$WORK/$1.log"; shift
if ! "$BIN" "$@" > "$log" 2>&1; then
cat "$log" >&2
fail "obikmer $1 exited non-zero"
fi
}
# ── 2. deterministic genomes with shared fragments ──────────────────────────
python3 - "$WORK" <<'EOF'
import random, sys
out = sys.argv[1]
rng = random.Random(20261117)
def rnd(n): return "".join(rng.choice("ACGT") for _ in range(n))
frag = {i: rnd(600) for i in range(5)} # shared fragments F0..F4
motif = rnd(40) # tandem motif (counts >= 255)
genomes = {
"A": [rnd(1500), frag[0], rnd(300), frag[1]],
"B": [frag[1], rnd(1200), frag[2], motif * 3],
"C": [frag[0], rnd(800), frag[2], rnd(400), frag[3]],
"D": [frag[3], rnd(1000), motif * 300, frag[4], rnd(200), frag[0][:250]],
}
for name, parts in genomes.items():
with open(f"{out}/{name}.fasta", "w") as f:
for j, p in enumerate(parts):
f.write(f">{name}_{j}\n{p}\n")
EOF
# ── 3. source indexes: built on --record, frozen in DIR/sources, reused on
# --check. `obikmer index` is not byte-stable run to run (unitig
# segmentation, hence unitigs.bin size, varies — measured 2026-11-17),
# and the merge base is the source with the largest unitigs.bin
# (`choose_base`), so re-indexing on every check would make the merge
# base — and the column order — flip between A and C. Freezing the
# sources makes this an oracle of `merge` alone. Copied (not linked)
# into WORK, since merge's bootstrap hard-links from its base source.
if [ "$MODE" = record ]; then
for G in A B C D; do
run "index_${G}_pres" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
--label "$G" -o "$WORK/$G.pres.idx" "$WORK/$G.fasta"
run "index_${G}_cnt" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
--with-counts --label "$G" -o "$WORK/$G.cnt.idx" "$WORK/$G.fasta"
done
rm -rf "$DIR/sources"
mkdir -p "$DIR/sources"
cp -R "$WORK"/*.idx "$DIR/sources/"
else
[ -d "$DIR/sources" ] || fail "no frozen source indexes in $DIR/sources (run --record first)"
cp -R "$DIR/sources"/*.idx "$WORK/"
fi
# ── 4. merge scenarios ──────────────────────────────────────────────────────
OUT="$WORK/out"
mkdir -p "$OUT"
dump() { # dump NAME INDEX — header first, rows sorted
local raw="$WORK/$1.dump.csv"
if ! "$BIN" dump --debug "$2" > "$raw" 2> "$WORK/dump_$1.log"; then
cat "$WORK/dump_$1.log" >&2
fail "obikmer dump $1 exited non-zero"
fi
{ head -n 1 "$raw"; tail -n +2 "$raw" | LC_ALL=C sort; } > "$OUT/$1.csv"
}
for T in pres cnt; do
run "s1_$T" merge -o "$WORK/s1_$T.idx" \
"$WORK/A.$T.idx" "$WORK/B.$T.idx" "$WORK/C.$T.idx" "$WORK/D.$T.idx"
dump "s1_$T" "$WORK/s1_$T.idx"
done
run s3_dense merge --dense -o "$WORK/s3_dense.idx" \
"$WORK/A.pres.idx" "$WORK/B.pres.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
dump s3_dense "$WORK/s3_dense.idx"
# scenario 2: re-merge a packed merge output as a source (checked against s1)
run s2_ab merge -o "$WORK/s2_ab.idx" "$WORK/A.pres.idx" "$WORK/B.pres.idx"
run s2 merge -o "$WORK/s2.idx" "$WORK/s2_ab.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
for N in s1_pres s2; do
"$BIN" dump "$WORK/$N.idx" > "$WORK/plain_$N.csv" 2> "$WORK/dump_plain_$N.log" \
|| { cat "$WORK/dump_plain_$N.log" >&2; fail "obikmer dump $N exited non-zero"; }
done
cat > "$WORK/compare_tables.py" <<'PY'
import csv, sys
def table(path):
rows = list(csv.reader(open(path)))
head = rows[0]
return {r[0]: {head[i]: r[i] for i in range(1, len(head))} for r in rows[1:]}
a, b = table(sys.argv[1]), table(sys.argv[2])
if a != b:
print(f"k-mers: {len(a)} vs {len(b)}; only in flat: {len(set(a) - set(b))}; "
f"only in re-merge: {len(set(b) - set(a))}", file=sys.stderr)
sys.exit(1)
PY
python3 "$WORK/compare_tables.py" "$WORK/plain_s1_pres.csv" "$WORK/plain_s2.csv" \
|| fail "scenario 2 (re-merge of a packed output) differs from the flat merge"
echo "ok s2 (re-merge of a packed output) equals the flat merge"
if "$BIN" merge -o "$WORK/s2_bad.idx" "$WORK/s2_ab.idx" "$WORK/s1_pres.idx" > "$WORK/s2_bad.log" 2>&1; then
fail "merging only packed indexes should be refused"
fi
grep -q "none of the sources can be the destination" "$WORK/s2_bad.log" \
|| { cat "$WORK/s2_bad.log" >&2; fail "refusal of an all-packed merge lacks the expected message"; }
echo "ok merge of only packed indexes is refused with a clear message"
# sanity: no dump may be empty (header + at least one row)
for f in "$OUT"/*.csv; do
[ "$(wc -l < "$f")" -gt 1 ] || fail "empty dump: $(basename "$f")"
done
# ── 5. record or check ──────────────────────────────────────────────────────
if [ "$MODE" = record ]; then
mkdir -p "$DIR"
rm -f "$DIR"/*.csv
cp "$OUT"/*.csv "$DIR"/
echo "RECORDED: $(ls "$OUT" | wc -l | tr -d ' ') dumps → $DIR"
wc -l "$DIR"/*.csv
exit 0
fi
[ -d "$DIR" ] || fail "reference directory not found: $DIR"
STATUS=0
for ref in "$DIR"/*.csv; do
name="$(basename "$ref")"
if [ ! -f "$OUT/$name" ]; then
echo "FAIL $name (not regenerated)"; STATUS=1
elif diff -q "$ref" "$OUT/$name" > /dev/null; then
echo "ok $name"
else
echo "FAIL $name"; { diff "$ref" "$OUT/$name" || true; } | head -20; STATUS=1
fi
done
for new in "$OUT"/*.csv; do
[ -f "$DIR/$(basename "$new")" ] || { echo "FAIL $(basename "$new") (no reference)"; STATUS=1; }
done
if [ "$STATUS" -eq 0 ]; then
echo "PASS: all merge dumps identical to $DIR"
else
echo "FAIL: merge dumps differ from $DIR" >&2
fi
exit "$STATUS"
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
# watch_merge.sh PID OUT.csv — samples a running process and kills it past a threshold.
#
# Every INTERVAL seconds appends one CSV row: number of memory mappings
# (lines of /proc/PID/maps), VmSize, VmData, VmRSS, threads, and the
# system-wide Committed_AS / CommitLimit. If the mapping count exceeds
# MAPS_MAX, VmRSS exceeds RSS_MAX_GB, or VmSize exceeds VMSIZE_MAX_GB (stay
# below `ulimit -v`: past it, allocations fail instead of the guard firing), the process receives SIGTERM (then
# SIGKILL after 30 s) and the reason is written to OUT.csv.reason — a guard
# so a runaway merge ends with a clean message instead of a failed
# allocation (or the OOM killer on a shared node).
#
# Reading /proc/PID/maps is itself slow when the process has hundreds of
# thousands of mappings (it held the kernel's mmap lock in earlier
# measurements): keep INTERVAL >= 5. It never reads /proc/PID/smaps.
#
# Usage: scripts/watch_merge.sh PID OUT.csv
# env: INTERVAL=5 MAPS_MAX=500000 RSS_MAX_GB=400 VMSIZE_MAX_GB=440
#
# Typical use (Linux only):
# nohup scripts/watch_merge.sh "$(pgrep -f 'obikmer merge' | head -1)" /tmp/watch.csv &
set -u
PID=${1:?usage: watch_merge.sh PID OUT.csv}
OUT=${2:?usage: watch_merge.sh PID OUT.csv}
INTERVAL=${INTERVAL:-5}
MAPS_MAX=${MAPS_MAX:-500000}
RSS_MAX_KB=$(( ${RSS_MAX_GB:-400} * 1024 * 1024 ))
VMSIZE_MAX_KB=$(( ${VMSIZE_MAX_GB:-440} * 1024 * 1024 ))
[ -r "/proc/$PID/maps" ] || { echo "cannot read /proc/$PID/maps" >&2; exit 1; }
printf 'time,maps,vmsize_kb,vmdata_kb,vmrss_kb,threads,committed_as_kb,commit_limit_kb\n' >"$OUT"
while kill -0 "$PID" 2>/dev/null; do
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null) || break
st=$(grep -E '^(VmSize|VmData|VmRSS|Threads):' "/proc/$PID/status" 2>/dev/null | awk '{printf "%s,", $2}')
rss=$(grep '^VmRSS:' "/proc/$PID/status" 2>/dev/null | awk '{print $2}')
vsz=$(grep '^VmSize:' "/proc/$PID/status" 2>/dev/null | awk '{print $2}')
mi=$(grep -E '^(Committed_AS|CommitLimit):' /proc/meminfo | awk '{printf "%s,", $2}')
echo "$(date +%T),$maps,${st}${mi%,}" >>"$OUT"
reason=""
[ "$maps" -gt "$MAPS_MAX" ] && reason="mappings $maps > MAPS_MAX $MAPS_MAX"
[ -n "${rss:-}" ] && [ "$rss" -gt "$RSS_MAX_KB" ] && reason="VmRSS ${rss} kB > RSS_MAX_GB"
[ -n "${vsz:-}" ] && [ "$vsz" -gt "$VMSIZE_MAX_KB" ] && reason="VmSize ${vsz} kB > VMSIZE_MAX_GB"
if [ -n "$reason" ]; then
echo "$(date +%T) GUARD: $reason — sending SIGTERM to $PID" | tee "$OUT.reason" >&2
kill -TERM "$PID" 2>/dev/null
for _ in $(seq 30); do kill -0 "$PID" 2>/dev/null || exit 0; sleep 1; done
kill -KILL "$PID" 2>/dev/null
exit 0
fi
sleep "$INTERVAL"
done
+159 -204
View File
@@ -8,19 +8,6 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "ahash"
version = "0.8.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
dependencies = [
"cfg-if",
"getrandom 0.3.4",
"once_cell",
"version_check",
"zerocopy",
]
[[package]] [[package]]
name = "aho-corasick" name = "aho-corasick"
version = "1.1.4" version = "1.1.4"
@@ -30,6 +17,12 @@ dependencies = [
"memchr", "memchr",
] ]
[[package]]
name = "aliasable"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "250f629c0161ad8107cf89319e990051fae62832fd343083bea452d93e2205fd"
[[package]] [[package]]
name = "allocator-api2" name = "allocator-api2"
version = "0.2.21" version = "0.2.21"
@@ -167,12 +160,6 @@ version = "0.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb" checksum = "349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb"
[[package]]
name = "bitflags"
version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
[[package]] [[package]]
name = "bitflags" name = "bitflags"
version = "2.11.1" version = "2.11.1"
@@ -191,6 +178,15 @@ dependencies = [
"wyz", "wyz",
] ]
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
[[package]] [[package]]
name = "block-buffer" name = "block-buffer"
version = "0.12.1" version = "0.12.1"
@@ -268,11 +264,11 @@ dependencies = [
[[package]] [[package]]
name = "cacheline-ef" name = "cacheline-ef"
version = "1.1.0" version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af737c6c59cb018ecbe6472cbdf86d39c59d78252febfe311953a991b6e4ed85" checksum = "6e701af8ea9f58c4ba18793a5d5dffd791cda99bf766976c59c583939d180fd5"
dependencies = [ dependencies = [
"common_traits 0.11.4", "common_traits",
"epserde", "epserde",
"mem_dbg", "mem_dbg",
] ]
@@ -312,6 +308,12 @@ version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
[[package]] [[package]]
name = "chacha20" name = "chacha20"
version = "0.10.1" version = "0.10.1"
@@ -319,7 +321,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"cpufeatures", "cpufeatures 0.3.0",
"rand_core 0.10.1", "rand_core 0.10.1",
] ]
@@ -420,7 +422,7 @@ version = "3.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34" checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
dependencies = [ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.52.0",
] ]
[[package]] [[package]]
@@ -433,17 +435,6 @@ dependencies = [
"memchr", "memchr",
] ]
[[package]]
name = "common_traits"
version = "0.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fda9ae1f26adcae83adb2e92f69cf59421f2a277a942f49f8e59f2fcbd7cf062"
dependencies = [
"anyhow",
"half",
"impl-tools 0.10.3",
]
[[package]] [[package]]
name = "common_traits" name = "common_traits"
version = "0.13.0" version = "0.13.0"
@@ -452,7 +443,7 @@ checksum = "d8e0d7d0c5977b1e87f759d406740c38227125f374f54d81bc197e877f2c5f26"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"half", "half",
"impl-tools 0.12.0", "impl-tools",
] ]
[[package]] [[package]]
@@ -482,6 +473,15 @@ dependencies = [
"unicode-segmentation", "unicode-segmentation",
] ]
[[package]]
name = "cpufeatures"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
dependencies = [
"libc",
]
[[package]] [[package]]
name = "cpufeatures" name = "cpufeatures"
version = "0.3.0" version = "0.3.0"
@@ -592,6 +592,16 @@ version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
[[package]] [[package]]
name = "crypto-common" name = "crypto-common"
version = "0.2.2" version = "0.2.2"
@@ -645,15 +655,25 @@ dependencies = [
"unicode-xid", "unicode-xid",
] ]
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer 0.10.4",
"crypto-common 0.1.7",
]
[[package]] [[package]]
name = "digest" name = "digest"
version = "0.11.3" version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
dependencies = [ dependencies = [
"block-buffer", "block-buffer 0.12.1",
"const-oid", "const-oid",
"crypto-common", "crypto-common 0.2.2",
] ]
[[package]] [[package]]
@@ -662,7 +682,7 @@ version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38"
dependencies = [ dependencies = [
"bitflags 2.11.1", "bitflags",
"objc2", "objc2",
] ]
@@ -698,27 +718,27 @@ dependencies = [
[[package]] [[package]]
name = "epserde" name = "epserde"
version = "0.8.0" version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c40d342ff20a2ce62d9a85ce406e672dfa137f902ac9670034533184f1533976" checksum = "d6cd9f3212ce43e7b594a974c6e8cef4e11c6473d76b7b97cb01c239130789eb"
dependencies = [ dependencies = [
"aliasable",
"anyhow", "anyhow",
"bitflags 2.11.1", "bitflags",
"common_traits 0.11.4",
"epserde-derive", "epserde-derive",
"maligned", "maybe-dangling",
"mem_dbg", "mem_dbg",
"mmap-rs", "mmap-rs",
"sealed", "sealed",
"sha2",
"thiserror 2.0.18", "thiserror 2.0.18",
"xxhash-rust",
] ]
[[package]] [[package]]
name = "epserde-derive" name = "epserde-derive"
version = "0.8.0" version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac80cc78b69765703f48ad93f33b8919cf5d907cda7459ad6ba2919cbbe605dd" checksum = "d73e3083270b2ff144c990e26580374fdfd512f61dd5a12da9b5bd749a7585e0"
dependencies = [ dependencies = [
"proc-macro2", "proc-macro2",
"quote", "quote",
@@ -738,7 +758,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [ dependencies = [
"libc", "libc",
"windows-sys 0.61.2", "windows-sys 0.52.0",
] ]
[[package]] [[package]]
@@ -806,6 +826,16 @@ dependencies = [
"byteorder", "byteorder",
] ]
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
[[package]] [[package]]
name = "getrandom" name = "getrandom"
version = "0.2.17" version = "0.2.17"
@@ -852,6 +882,17 @@ dependencies = [
"zerocopy", "zerocopy",
] ]
[[package]]
name = "hashbrown"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
dependencies = [
"allocator-api2",
"equivalent",
"foldhash",
]
[[package]] [[package]]
name = "hashbrown" name = "hashbrown"
version = "0.17.1" version = "0.17.1"
@@ -899,7 +940,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4c2e65a48d3b300843ac84a2fe8e166bb5a5b00f30054593bcee8157e4b465fd" checksum = "4c2e65a48d3b300843ac84a2fe8e166bb5a5b00f30054593bcee8157e4b465fd"
dependencies = [ dependencies = [
"arrayvec", "arrayvec",
"bitflags 2.11.1", "bitflags",
"derive_more", "derive_more",
"errno", "errno",
"hwlocality-sys", "hwlocality-sys",
@@ -935,18 +976,6 @@ dependencies = [
"typenum", "typenum",
] ]
[[package]]
name = "impl-tools"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ae95c9095c2f1126d7db785955c73cdc5fc33e7c3fa911bd4a42931672029a7"
dependencies = [
"autocfg",
"impl-tools-lib 0.11.4",
"proc-macro-error2",
"syn 2.0.117",
]
[[package]] [[package]]
name = "impl-tools" name = "impl-tools"
version = "0.12.0" version = "0.12.0"
@@ -954,23 +983,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06c8567c75b167cdda5ecee024da9cc9581d41bd52460e2e0e768967972aeef1" checksum = "06c8567c75b167cdda5ecee024da9cc9581d41bd52460e2e0e768967972aeef1"
dependencies = [ dependencies = [
"autocfg", "autocfg",
"impl-tools-lib 0.12.0", "impl-tools-lib",
"proc-macro-error3", "proc-macro-error3",
"syn 2.0.117", "syn 2.0.117",
] ]
[[package]]
name = "impl-tools-lib"
version = "0.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab699036df31c1f7d3561bfa6e9cb9bc3bb0fd2e2cd9bf121c31cb961d049ddf"
dependencies = [
"proc-macro-error2",
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]] [[package]]
name = "impl-tools-lib" name = "impl-tools-lib"
version = "0.12.0" version = "0.12.0"
@@ -990,7 +1007,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
dependencies = [ dependencies = [
"equivalent", "equivalent",
"hashbrown", "hashbrown 0.17.1",
] ]
[[package]] [[package]]
@@ -1023,7 +1040,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
dependencies = [ dependencies = [
"hermit-abi", "hermit-abi",
"libc", "libc",
"windows-sys 0.61.2", "windows-sys 0.52.0",
] ]
[[package]] [[package]]
@@ -1043,9 +1060,9 @@ dependencies = [
[[package]] [[package]]
name = "itertools" name = "itertools"
version = "0.14.0" version = "0.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" checksum = "8b4baf93f58d4425749ca49a51c50ebab072c5df6994d08fed93541c331481dc"
dependencies = [ dependencies = [
"either", "either",
] ]
@@ -1083,7 +1100,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e24a010dd405bd7ed803e5253182815b41bf2e6a80cc3bfc066658e03a198aa" checksum = "9e24a010dd405bd7ed803e5253182815b41bf2e6a80cc3bfc066658e03a198aa"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"cpufeatures", "cpufeatures 0.3.0",
] ]
[[package]] [[package]]
@@ -1157,7 +1174,7 @@ version = "0.18.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d2f2f9b4ba7e6b24d95e7e899329d35be83bcded72c8540cdd5368932d1d90a" checksum = "5d2f2f9b4ba7e6b24d95e7e899329d35be83bcded72c8540cdd5368932d1d90a"
dependencies = [ dependencies = [
"hashbrown", "hashbrown 0.17.1",
] ]
[[package]] [[package]]
@@ -1169,12 +1186,6 @@ dependencies = [
"libc", "libc",
] ]
[[package]]
name = "maligned"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e88c3cbe8288f77f293e48a28b3232e3defd203a6d839fa7f68ea4329e83464"
[[package]] [[package]]
name = "matchers" name = "matchers"
version = "0.2.0" version = "0.2.0"
@@ -1195,22 +1206,30 @@ dependencies = [
] ]
[[package]] [[package]]
name = "mem_dbg" name = "maybe-dangling"
version = "0.3.4" version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "728cc9dc97593cd22f7bc81fbef70a2d391d7a9a855e7d658b653318124a6cf0" checksum = "59dbb09ed53f8e4f314e353dc6c1853ae5b4c480a668a422657804a544ea9f65"
[[package]]
name = "mem_dbg"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b48a1086c746f4ee6ca5cb0acf856a14709bc4d2d20e03db150a12ddf2269e6d"
dependencies = [ dependencies = [
"bitflags 2.11.1", "aliasable",
"maligned", "bitflags",
"hashbrown 0.16.1",
"maybe-dangling",
"mem_dbg-derive", "mem_dbg-derive",
"mmap-rs", "mmap-rs",
] ]
[[package]] [[package]]
name = "mem_dbg-derive" name = "mem_dbg-derive"
version = "0.2.1" version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d84f40c93b0508d5565db79a814d02d5b2545967205ce44be211592aafa34d6c" checksum = "eb910efe8da52f13da727170e352e50a1764579a6fb1065d00d9556da19c79ac"
dependencies = [ dependencies = [
"proc-macro2", "proc-macro2",
"quote", "quote",
@@ -1232,15 +1251,6 @@ dependencies = [
"libc", "libc",
] ]
[[package]]
name = "memoffset"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5de893c32cde5f383baa4c04c5d6dbdd735cfd4a794b0debdb2bb1b421da5ff4"
dependencies = [
"autocfg",
]
[[package]] [[package]]
name = "miniz_oxide" name = "miniz_oxide"
version = "0.8.9" version = "0.8.9"
@@ -1253,17 +1263,17 @@ dependencies = [
[[package]] [[package]]
name = "mmap-rs" name = "mmap-rs"
version = "0.6.1" version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "86968d85441db75203c34deefd0c88032f275aaa85cee19a1dcfff6ae9df56da" checksum = "4ecce9d566cb9234ae3db9e249c8b55665feaaf32b0859ff1e27e310d2beb3d8"
dependencies = [ dependencies = [
"bitflags 1.3.2", "bitflags",
"combine", "combine",
"libc", "libc",
"mach2", "mach2",
"nix", "nix",
"sysctl", "sysctl",
"thiserror 1.0.69", "thiserror 2.0.18",
"widestring", "widestring",
"windows 0.48.0", "windows 0.48.0",
] ]
@@ -1326,15 +1336,14 @@ dependencies = [
[[package]] [[package]]
name = "nix" name = "nix"
version = "0.26.4" version = "0.30.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "598beaf3cc6fdd9a5dfb1630c2800c7acd31df7aaf0f565796fba2b53ca1af1b" checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6"
dependencies = [ dependencies = [
"bitflags 1.3.2", "bitflags",
"cfg-if", "cfg-if",
"cfg_aliases",
"libc", "libc",
"memoffset",
"pin-utils",
] ]
[[package]] [[package]]
@@ -1386,7 +1395,7 @@ dependencies = [
name = "obicompactvec" name = "obicompactvec"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"common_traits 0.13.0", "common_traits",
"memmap2", "memmap2",
"ndarray", "ndarray",
"rayon", "rayon",
@@ -1397,14 +1406,10 @@ dependencies = [
name = "obidebruinj" name = "obidebruinj"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"ahash",
"crossbeam-channel", "crossbeam-channel",
"hashbrown", "hashbrown 0.17.1",
"obifastwrite",
"obikseq", "obikseq",
"rayon", "rayon",
"tracing",
"xxhash-rust",
] ]
[[package]] [[package]]
@@ -1485,14 +1490,10 @@ dependencies = [
"ndarray", "ndarray",
"niffler", "niffler",
"obicompactvec", "obicompactvec",
"obidebruinj",
"obikentropy",
"obikseq", "obikseq",
"obipipeline",
"obiread", "obiread",
"obiskio", "obiskio",
"obisys", "obisys",
"obitaxonomy",
"ptr_hash", "ptr_hash",
"rayon", "rayon",
"serde", "serde",
@@ -1506,7 +1507,6 @@ dependencies = [
name = "obikindexer" name = "obikindexer"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"cacheline-ef",
"epserde", "epserde",
"memmap2", "memmap2",
"niffler", "niffler",
@@ -1521,7 +1521,6 @@ dependencies = [
"obiskbuilder", "obiskbuilder",
"obiskio", "obiskio",
"obisys", "obisys",
"ptr_hash",
"rayon", "rayon",
"serde", "serde",
"serde_json", "serde_json",
@@ -1532,7 +1531,7 @@ dependencies = [
[[package]] [[package]]
name = "obikmer" name = "obikmer"
version = "1.3.6" version = "1.4.5"
dependencies = [ dependencies = [
"clap", "clap",
"csv", "csv",
@@ -1551,7 +1550,6 @@ dependencies = [
"obikrope", "obikrope",
"obikselect", "obikselect",
"obikseq", "obikseq",
"obiksession",
"obikstats", "obikstats",
"obipipeline", "obipipeline",
"obiread", "obiread",
@@ -1560,6 +1558,7 @@ dependencies = [
"rayon", "rayon",
"serde", "serde",
"serde_yaml", "serde_yaml",
"tikv-jemallocator",
"tracing", "tracing",
"tracing-subscriber", "tracing-subscriber",
] ]
@@ -1577,7 +1576,6 @@ dependencies = [
"obiskio", "obiskio",
"obisys", "obisys",
"rayon", "rayon",
"tempfile",
"tracing", "tracing",
] ]
@@ -1596,7 +1594,6 @@ dependencies = [
"obikseq", "obikseq",
"obipipeline", "obipipeline",
"obiread", "obiread",
"obiskbuilder",
"obiskio", "obiskio",
"obisys", "obisys",
"petgraph", "petgraph",
@@ -1642,7 +1639,6 @@ dependencies = [
name = "obikrope" name = "obikrope"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"bytes",
"criterion2", "criterion2",
] ]
@@ -1720,7 +1716,6 @@ name = "obiskbuilder"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"criterion2", "criterion2",
"lazy_static",
"obikentropy", "obikentropy",
"obikrope", "obikrope",
"obikseq", "obikseq",
@@ -1773,7 +1768,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536" checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536"
dependencies = [ dependencies = [
"bitflags 2.11.1", "bitflags",
"dispatch2", "dispatch2",
"objc2", "objc2",
] ]
@@ -1790,7 +1785,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3e0adef53c21f888deb4fa59fc59f7eb17404926ee8a6f59f5df0fd7f9f3272" checksum = "e3e0adef53c21f888deb4fa59fc59f7eb17404926ee8a6f59f5df0fd7f9f3272"
dependencies = [ dependencies = [
"bitflags 2.11.1", "bitflags",
"objc2", "objc2",
] ]
@@ -1884,12 +1879,6 @@ version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
[[package]]
name = "pin-utils"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
[[package]] [[package]]
name = "pkg-config" name = "pkg-config"
version = "0.3.33" version = "0.3.33"
@@ -1949,14 +1938,10 @@ dependencies = [
] ]
[[package]] [[package]]
name = "proc-macro-error-attr2" name = "prefetch-index"
version = "2.0.0" version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96de42df36bb9bba5542fe9f1a054b8cc87e172759a1868aa05c1f3acc89dfc5" checksum = "9057806a8d77d67bccdc0f542db43737a6f19ada3efab2adc63277feea27310f"
dependencies = [
"proc-macro2",
"quote",
]
[[package]] [[package]]
name = "proc-macro-error-attr3" name = "proc-macro-error-attr3"
@@ -1968,17 +1953,6 @@ dependencies = [
"quote", "quote",
] ]
[[package]]
name = "proc-macro-error2"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11ec05c52be0a07b08061f7dd003e7d7092e0472bc731b4af7bb1ef876109802"
dependencies = [
"proc-macro-error-attr2",
"proc-macro2",
"quote",
]
[[package]] [[package]]
name = "proc-macro-error3" name = "proc-macro-error3"
version = "3.1.0" version = "3.1.0"
@@ -2001,30 +1975,25 @@ dependencies = [
[[package]] [[package]]
name = "ptr_hash" name = "ptr_hash"
version = "1.1.0" version = "2.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b4e4fb9c4c2ba3e5b060f53ef46afd3de37345b08e3ec0f2c65e0ca1d57ccbd" checksum = "0d2381547e2e36c72a455c1cc418d5c60ecd6e94f4819d7560456108e99a4efa"
dependencies = [ dependencies = [
"anyhow",
"bitvec", "bitvec",
"cacheline-ef", "cacheline-ef",
"clap",
"colored", "colored",
"common_traits 0.11.4",
"epserde", "epserde",
"epserde-derive", "epserde-derive",
"fastrand", "fastrand",
"fxhash", "fxhash",
"itertools 0.14.0", "itertools 0.15.0",
"lazy_static",
"log", "log",
"mem_dbg", "mem_dbg",
"rand 0.9.4", "prefetch-index",
"rand_chacha 0.9.0", "rand 0.10.2",
"rand_chacha 0.10.0",
"rayon", "rayon",
"rdst", "rdst",
"rustc-hash",
"serde",
"sucds", "sucds",
"tempfile", "tempfile",
"xxhash-rust", "xxhash-rust",
@@ -2097,16 +2066,6 @@ dependencies = [
"rand_core 0.6.4", "rand_core 0.6.4",
] ]
[[package]]
name = "rand"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea"
dependencies = [
"rand_chacha 0.9.0",
"rand_core 0.9.5",
]
[[package]] [[package]]
name = "rand" name = "rand"
version = "0.10.2" version = "0.10.2"
@@ -2130,12 +2089,12 @@ dependencies = [
[[package]] [[package]]
name = "rand_chacha" name = "rand_chacha"
version = "0.9.0" version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" checksum = "3e6af7f3e25ded52c41df4e0b1af2d047e45896c2f3281792ed68a1c243daedb"
dependencies = [ dependencies = [
"ppv-lite86", "ppv-lite86",
"rand_core 0.9.5", "rand_core 0.10.1",
] ]
[[package]] [[package]]
@@ -2147,15 +2106,6 @@ dependencies = [
"getrandom 0.2.17", "getrandom 0.2.17",
] ]
[[package]]
name = "rand_core"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
dependencies = [
"getrandom 0.3.4",
]
[[package]] [[package]]
name = "rand_core" name = "rand_core"
version = "0.10.1" version = "0.10.1"
@@ -2215,7 +2165,7 @@ version = "0.5.18"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d"
dependencies = [ dependencies = [
"bitflags 2.11.1", "bitflags",
] ]
[[package]] [[package]]
@@ -2278,7 +2228,7 @@ checksum = "d9776093b7ca170454ab1406954f7b7d97a57c51dc6c0642957fb2ef25c2d399"
dependencies = [ dependencies = [
"bytecheck", "bytecheck",
"bytes", "bytes",
"hashbrown", "hashbrown 0.17.1",
"indexmap", "indexmap",
"munge", "munge",
"ptr_meta", "ptr_meta",
@@ -2300,12 +2250,6 @@ dependencies = [
"syn 3.0.3", "syn 3.0.3",
] ]
[[package]]
name = "rustc-hash"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
[[package]] [[package]]
name = "rustc_version" name = "rustc_version"
version = "0.4.1" version = "0.4.1"
@@ -2321,11 +2265,11 @@ version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
dependencies = [ dependencies = [
"bitflags 2.11.1", "bitflags",
"errno", "errno",
"libc", "libc",
"linux-raw-sys", "linux-raw-sys",
"windows-sys 0.61.2", "windows-sys 0.52.0",
] ]
[[package]] [[package]]
@@ -2463,13 +2407,24 @@ dependencies = [
"unsafe-libyaml", "unsafe-libyaml",
] ]
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures 0.2.17",
"digest 0.10.7",
]
[[package]] [[package]]
name = "sha3" name = "sha3"
version = "0.11.0" version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be176f1a57ce4e3d31c1a166222d9768de5954f811601fb7ca06fc8203905ce1" checksum = "be176f1a57ce4e3d31c1a166222d9768de5954f811601fb7ca06fc8203905ce1"
dependencies = [ dependencies = [
"digest", "digest 0.11.3",
"keccak", "keccak",
] ]
@@ -2592,11 +2547,11 @@ dependencies = [
[[package]] [[package]]
name = "sysctl" name = "sysctl"
version = "0.5.5" version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec7dddc5f0fee506baf8b9fdb989e242f17e4b11c61dfbb0635b705217199eea" checksum = "01198a2debb237c62b6826ec7081082d951f46dbb64b0e8c7649a452230d1dfc"
dependencies = [ dependencies = [
"bitflags 2.11.1", "bitflags",
"byteorder", "byteorder",
"enum-as-inner", "enum-as-inner",
"libc", "libc",
@@ -2646,7 +2601,7 @@ dependencies = [
"getrandom 0.4.3", "getrandom 0.4.3",
"once_cell", "once_cell",
"rustix", "rustix",
"windows-sys 0.61.2", "windows-sys 0.52.0",
] ]
[[package]] [[package]]
@@ -3051,7 +3006,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.52.0",
] ]
[[package]] [[package]]
@@ -3,6 +3,7 @@ use std::io;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use crate::bitvec::PersistentBitVecBuilder; use crate::bitvec::PersistentBitVecBuilder;
use crate::mem_builder::MemBitVecBuilder;
use crate::meta::MatrixMeta; use crate::meta::MatrixMeta;
use crate::tempbitvec::TempBitVec; use crate::tempbitvec::TempBitVec;
use crate::tempintvec::TempCompactIntVec; use crate::tempintvec::TempCompactIntVec;
@@ -43,6 +44,14 @@ impl PersistentBitMatrixBuilder {
PersistentBitVecBuilder::new(self.n, &path) PersistentBitVecBuilder::new(self.n, &path)
} }
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
/// the column is built in RAM and written once at `close`.
pub fn add_mem_col(&mut self) -> MemBitVecBuilder {
let path = col_path(&self.dir, self.n_cols);
self.n_cols += 1;
MemBitVecBuilder::new(self.n, &path)
}
pub fn add_col_ones(&mut self) -> io::Result<PersistentBitVecBuilder> { pub fn add_col_ones(&mut self) -> io::Result<PersistentBitVecBuilder> {
let path = col_path(&self.dir, self.n_cols); let path = col_path(&self.dir, self.n_cols);
self.n_cols += 1; self.n_cols += 1;
+1
View File
@@ -18,6 +18,7 @@ mod packed;
mod pairwise; mod pairwise;
mod persistent; mod persistent;
mod sparse; mod sparse;
mod sparse_columnar;
pub use builder::PersistentBitMatrixBuilder; pub use builder::PersistentBitMatrixBuilder;
pub use group_ops::batch_presence_counts; pub use group_ops::batch_presence_counts;
+1 -1
View File
@@ -105,7 +105,7 @@ impl PackedBitMatrix {
/// Reads just the `n_cols` field from an existing packed matrix's header, /// Reads just the `n_cols` field from an existing packed matrix's header,
/// without mapping the file. Used by `pack_bit_matrix` to tell a genuinely /// without mapping the file. Used by `pack_bit_matrix` to tell a genuinely
/// complete pack from a stale one that predates a later column-widening. /// complete pack from a stale one that predates a later column-widening.
fn packed_bit_matrix_n_cols(path: &Path) -> io::Result<usize> { pub(super) fn packed_bit_matrix_n_cols(path: &Path) -> io::Result<usize> {
let mut f = File::open(path)?; let mut f = File::open(path)?;
let mut header = [0u8; PBMX_HEADER]; let mut header = [0u8; PBMX_HEADER];
f.read_exact(&mut header)?; f.read_exact(&mut header)?;
+90 -43
View File
@@ -25,19 +25,18 @@ use std::fs;
use std::io; use std::io;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use memmap2::Mmap;
use ndarray::{Array1, Array2}; use ndarray::{Array1, Array2};
use crate::traits::{BitPartials, ColumnWeights}; use crate::traits::{BitPartials, ColumnWeights};
use crate::eliasfano::{EliasFano, EliasFanoBuilder}; use crate::eliasfano::{EliasFano, EliasFanoBuilder};
use crate::fixedintvec::{PersistentFixedIntVec, PersistentFixedIntVecBuilder, bit_width_for_range}; use crate::fixedintvec::{PersistentFixedIntVec, PersistentFixedIntVecBuilder, bit_width_for_range};
use crate::meta::field; use crate::meta::{MatrixMeta, field};
use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder}; use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
use super::PersistentBitMatrix; use super::PersistentBitMatrix;
use super::col_path; use super::col_path;
use super::columnar::ColumnarBitMatrix; use super::packed::{PackedBitMatrix, pack_bit_matrix, packed_bit_matrix_n_cols};
use super::packed::PackedBitMatrix;
use crate::views::nonzero_triples;
fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") } fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") }
fn singleton_path(dir: &Path) -> PathBuf { dir.join("singleton.pfiv") } fn singleton_path(dir: &Path) -> PathBuf { dir.join("singleton.pfiv") }
@@ -151,7 +150,9 @@ pub struct PersistentSparseBitMatrix {
singleton: PersistentFixedIntVec, singleton: PersistentFixedIntVec,
multi: PersistentFixedIntVec, multi: PersistentFixedIntVec,
dict_offsets: EliasFano, dict_offsets: EliasFano,
dict_values: Vec<u8>, /// `dict_values.bin`, mapped (never copied to the heap); `None` when the
/// file is empty — no multi-genome set — since an empty file cannot be mapped.
dict_values: Option<Mmap>,
/// Row count (k-mer slots). /// Row count (k-mer slots).
n: usize, n: usize,
/// Column count (genomes). /// Column count (genomes).
@@ -167,7 +168,12 @@ impl PersistentSparseBitMatrix {
let singleton = PersistentFixedIntVec::open(&singleton_path(dir))?; let singleton = PersistentFixedIntVec::open(&singleton_path(dir))?;
let multi = PersistentFixedIntVec::open(&multi_path(dir))?; let multi = PersistentFixedIntVec::open(&multi_path(dir))?;
let dict_offsets = EliasFano::open(&dict_offsets_base(dir))?; let dict_offsets = EliasFano::open(&dict_offsets_base(dir))?;
let dict_values = fs::read(dict_values_path(dir))?; let dict_file = fs::File::open(dict_values_path(dir))?;
let dict_values = if dict_file.metadata()?.len() == 0 {
None
} else {
Some(unsafe { Mmap::map(&dict_file)? })
};
Ok(Self { Ok(Self {
is_multi, singleton, multi, dict_offsets, dict_values, is_multi, singleton, multi, dict_offsets, dict_values,
n: meta.n, n_cols: meta.n_cols, n_distinct_multi: meta.n_distinct_multi, n: meta.n, n_cols: meta.n_cols, n_distinct_multi: meta.n_distinct_multi,
@@ -179,6 +185,11 @@ impl PersistentSparseBitMatrix {
#[inline] #[inline]
pub fn n_cols(&self) -> usize { self.n_cols } pub fn n_cols(&self) -> usize { self.n_cols }
#[inline]
fn dict_bytes(&self) -> &[u8] {
self.dict_values.as_deref().unwrap_or(&[])
}
/// Byte range of `dict_id`'s varint-encoded genome list within /// Byte range of `dict_id`'s varint-encoded genome list within
/// `dict_values`. /// `dict_values`.
#[inline] #[inline]
@@ -187,7 +198,7 @@ impl PersistentSparseBitMatrix {
let end = if dict_id + 1 < self.n_distinct_multi { let end = if dict_id + 1 < self.n_distinct_multi {
self.dict_offsets.get(dict_id + 1) as usize self.dict_offsets.get(dict_id + 1) as usize
} else { } else {
self.dict_values.len() self.dict_bytes().len()
}; };
(start, end) (start, end)
} }
@@ -232,7 +243,7 @@ impl PersistentSparseBitMatrix {
let (start, end) = self.dict_entry_range(dict_id); let (start, end) = self.dict_entry_range(dict_id);
let mut p = start; let mut p = start;
while p < end { while p < end {
f(read_varint(&self.dict_values, &mut p) as usize); f(read_varint(self.dict_bytes(), &mut p) as usize);
} }
} else { } else {
let pos = self.is_multi.rank0(slot) as usize; let pos = self.is_multi.rank0(slot) as usize;
@@ -287,7 +298,7 @@ impl PersistentSparseBitMatrix {
present.clear(); present.clear();
let mut p = start; let mut p = start;
while p < end { while p < end {
present.push(read_varint(&self.dict_values, &mut p) as usize); present.push(read_varint(self.dict_bytes(), &mut p) as usize);
} }
for &g in &present { for &g in &present {
counts[g] += w; counts[g] += w;
@@ -488,12 +499,12 @@ impl PersistentSparseBitMatrixBuilder {
/// existing index. /// existing index.
/// ///
/// Processes rows in batches of [`BUILD_FROM_DENSE_BATCH`] instead of /// Processes rows in batches of [`BUILD_FROM_DENSE_BATCH`] instead of
/// one row at a time: `nonzero_triples(..., sorted_slots=true)` walks /// one row at a time: [`BitSliceView::for_each_set_in`] walks each
/// each column *once per batch*, sequentially — matching how /// column *once per batch*, word by word, visiting only the set bits —
/// `PersistentBitMatrix::nonzero_iter` already batches slots for /// rather than `fill_row`'s row-major loop, which re-reads every column
/// queries — rather than `fill_row`'s row-major loop, which re-reads /// with one `get()` per cell and jumps between all `n_cols` columns on
/// every column with one `get()` per cell and jumps between all /// every single row. Columns are visited in ascending order, so each
/// `n_cols` column files on every single row. /// row's genome list comes out sorted, as `push_row` requires.
pub fn build_from_dense(dense: &PersistentBitMatrix, dir: &Path) -> io::Result<Self> { pub fn build_from_dense(dense: &PersistentBitMatrix, dir: &Path) -> io::Result<Self> {
let n = dense.n(); let n = dense.n();
let n_cols = dense.n_cols(); let n_cols = dense.n_cols();
@@ -502,9 +513,10 @@ impl PersistentSparseBitMatrixBuilder {
let mut start = 0; let mut start = 0;
while start < n { while start < n {
let end = (start + BUILD_FROM_DENSE_BATCH).min(n); let end = (start + BUILD_FROM_DENSE_BATCH).min(n);
let slots: Vec<usize> = (start..end).collect(); for c in 0..n_cols {
for (row, c, _) in nonzero_triples(&slots, n_cols, |c| dense.col_view(c), true) { dense.col_view(c).for_each_set_in(start..end, |slot| {
row_buffers[row].push(c as u32); row_buffers[slot - start].push(c as u32);
});
} }
for buf in row_buffers.iter_mut().take(end - start) { for buf in row_buffers.iter_mut().take(end - start) {
builder.push_row(buf); builder.push_row(buf);
@@ -514,6 +526,31 @@ impl PersistentSparseBitMatrixBuilder {
} }
Ok(builder) Ok(builder)
} }
/// Builds the row-permuted copy of `src` into `dir`: row `s` of the
/// result is row `new_to_old[s]` of `src`. `new_to_old` must have
/// `src.n()` entries and be a permutation of `0..src.n()` — the caller
/// proves that, this only reads `src` at those slots.
///
/// Needed whenever the slot numbering of a layer changes (its MPHF is
/// rebuilt) while the data attached to each k-mer must follow it.
/// Rows are read at arbitrary slots of `src`, so convert a dense source
/// to sparse first (`build_from_dense`) rather than permuting it
/// cell by cell.
pub fn build_permuted(
src: &PersistentSparseBitMatrix,
new_to_old: &[u32],
dir: &Path,
) -> io::Result<Self> {
let mut builder = Self::new(new_to_old.len(), src.n_cols(), dir)?;
let mut genomes: Vec<u32> = Vec::new();
for &old in new_to_old {
genomes.clear();
src.for_each_genome_in_row(old as usize, |g| genomes.push(g as u32));
builder.push_row(&genomes);
}
Ok(builder)
}
} }
// ── Trait impls ─────────────────────────────────────────────────────────────── // ── Trait impls ───────────────────────────────────────────────────────────────
@@ -577,37 +614,47 @@ impl BitPartials for PersistentSparseBitMatrix {
m m
} }
} }
/// sparse on-disk format in place, mirroring [`super::pack_bit_matrix`]'s
/// convention (old-format files removed only after the new format is /// Converts a layer's presence matrix (columnar `col_*.pbiv` or dense
/// fully written, so a crash mid-conversion leaves the previous, still /// `matrix.pbmx`) to the sparse on-disk format in place. Old-format files
/// valid format rather than a half-written one). Idempotent — does /// are removed only after the new format is fully written, so a crash
/// mid-conversion leaves a previous, still valid format. Idempotent: does
/// nothing if `is_multi.prsb` already exists. /// nothing if `is_multi.prsb` already exists.
/// ///
/// The dense→sparse transpose (`build_from_dense`) only needs `fill_row`, /// Columnar data is transposed straight into the sparse format, a window of
/// which costs the same `O(n_cols)` per row on `Columnar` as on `Packed` /// rows at a time, one column file open at a time
/// (see `ColumnarBitMatrix::fill_row`) — so this reads straight from /// ([`PersistentSparseBitMatrixBuilder::build_from_columnar`]): no dense
/// whichever dense form is already on disk. In particular, when this runs /// intermediate is written, and a worker holds no mapping of the columns
/// right after a build (the common case: `presence/` is still `Columnar`, /// whatever their number. A dense `matrix.pbmx` is only read when it is the
/// `matrix.pbmx` doesn't exist yet), it never materialises a full packed /// current form of the layer: no columnar data pending, or a `matrix.pbmx`
/// copy of the presence matrix just to read it back once and delete it. /// that already has as many columns as the columnar metadata says.
pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> { pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> {
if is_multi_path(dir).exists() { if is_multi_path(dir).exists() {
return Ok(()); return Ok(());
} }
let packed_path = dir.join("matrix.pbmx"); let packed_path = dir.join("matrix.pbmx");
if packed_path.exists() {
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?); if let Ok(meta) = MatrixMeta::load(dir) {
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?; if packed_bit_matrix_n_cols(&packed_path).ok() != Some(meta.n_cols) {
drop(dense); // Columnar data is the freshest form (any `matrix.pbmx` here is
fs::remove_file(&packed_path)?; // stale, e.g. copied from a merge base before the layer was
} else { // widened): transpose it directly.
let dense = PersistentBitMatrix::Columnar(ColumnarBitMatrix::open(dir)?); PersistentSparseBitMatrixBuilder::build_from_columnar(dir, &meta)?.close()?;
let n_cols = dense.n_cols(); for c in 0..meta.n_cols {
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?; fs::remove_file(col_path(dir, c))?;
drop(dense); }
for c in 0..n_cols { let _ = fs::remove_file(col_path(dir, c)); } fs::remove_file(dir.join("meta.json"))?;
let _ = fs::remove_file(dir.join("meta.json")); let _ = fs::remove_file(&packed_path);
return Ok(());
}
} }
Ok(())
// Dense packed form is the current one. `pack_bit_matrix` is a no-op when
// there is no columnar data left, and otherwise only drops the
// now-redundant column files.
pack_bit_matrix(dir)?;
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?);
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
drop(dense);
fs::remove_file(&packed_path)
} }
@@ -0,0 +1,95 @@
//! Building the sparse presence matrix straight from the columnar
//! `col_*.pbiv` files, with no dense intermediate and no mapping.
//!
//! Each column is opened, a window of it is read with `pread`, and it is
//! closed again: a worker holds at most one column file at a time,
//! whatever the number of genomes. See [`crate::transpose`].
use std::fs::File;
use std::io;
use std::os::unix::fs::FileExt;
use std::path::Path;
use crate::bitvec::MAGIC;
use crate::meta::MatrixMeta;
use crate::transpose::{DEFAULT_MAX_HITS, DEFAULT_WINDOW_ROWS, Hit, transpose_windows};
use crate::views::BitSliceView;
use super::PersistentSparseBitMatrixBuilder;
use super::col_path;
/// `PBIV` header: magic(4) + pad(4) + n(8).
const PBIV_HEADER: usize = 16;
/// Reads the bits of `path` for rows `start..end` (`start` a multiple of 64)
/// into `words`, after checking the header against `n`.
fn read_bit_window(
path: &Path,
n: usize,
start: usize,
end: usize,
bytes: &mut Vec<u8>,
words: &mut Vec<u64>,
) -> io::Result<()> {
debug_assert_eq!(start % 64, 0);
let file = File::open(path)?;
let mut header = [0u8; PBIV_HEADER];
file.read_exact_at(&mut header, 0)?;
if header[..4] != MAGIC || u64::from_le_bytes(header[8..16].try_into().unwrap()) as usize != n {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("{}: not a PBIV column of {n} rows", path.display()),
));
}
let nw = (end - start).div_ceil(64);
bytes.clear();
bytes.resize(nw * 8, 0);
file.read_exact_at(bytes, (PBIV_HEADER + start / 8) as u64)?;
words.clear();
words.extend(bytes.chunks_exact(8).map(|b| u64::from_le_bytes(b.try_into().unwrap())));
Ok(())
}
impl PersistentSparseBitMatrixBuilder {
/// Builds the sparse matrix of the columnar layer in `dir` (`meta.json`
/// plus `col_*.pbiv`), writing the sparse files into `dir`. Same rows,
/// same genome order and so same bytes as going through a dense
/// `matrix.pbmx` (`build_from_dense`), without ever writing it.
pub fn build_from_columnar(dir: &Path, meta: &MatrixMeta) -> io::Result<Self> {
Self::build_from_columnar_with(dir, meta, DEFAULT_WINDOW_ROWS, DEFAULT_MAX_HITS)
}
/// [`build_from_columnar`](Self::build_from_columnar) with explicit window
/// size (rows) and non-zero-cell cap; see [`crate::transpose`].
pub fn build_from_columnar_with(
dir: &Path,
meta: &MatrixMeta,
window_rows: usize,
max_hits: usize,
) -> io::Result<Self> {
let mut builder = Self::new(meta.n, meta.n_cols, dir)?;
let mut bytes: Vec<u8> = Vec::new();
let mut words: Vec<u64> = Vec::new();
transpose_windows(
meta.n,
meta.n_cols,
window_rows,
max_hits,
(),
|c, start, end, hits: &mut Vec<Hit<()>>| {
read_bit_window(&col_path(dir, c), meta.n, start, end, &mut bytes, &mut words)?;
BitSliceView::new(&words, end - start).for_each_set_in(0..end - start, |r| {
hits.push((r as u32, c as u32, ()));
});
Ok(())
},
|_, batch| {
for i in 0..batch.len() {
builder.push_row(batch.cols(i));
}
Ok(())
},
)?;
Ok(builder)
}
}
+7 -6
View File
@@ -4,11 +4,11 @@ use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut}; use memmap2::{Mmap, MmapMut};
use crate::mmap_file::{create_mmap_file, open_mmap_file}; use crate::mmap_file::{building_path, create_mmap_file, open_mmap_file, publish_built};
use crate::reader::PersistentCompactIntVec; use crate::reader::PersistentCompactIntVec;
use crate::views::{BitSliceIter, BitSliceView, IntSliceView}; use crate::views::{BitSliceIter, BitSliceView, IntSliceView};
const MAGIC: [u8; 4] = *b"PBIV"; pub(crate) const MAGIC: [u8; 4] = *b"PBIV";
// Header: magic(4) + _pad(4) + n(8) = 16 bytes. // Header: magic(4) + _pad(4) + n(8) = 16 bytes.
// Data starts at offset 16, u64-aligned (mmap base is page-aligned, 16 % 8 == 0). // Data starts at offset 16, u64-aligned (mmap base is page-aligned, 16 % 8 == 0).
@@ -206,7 +206,7 @@ impl PersistentBitVecBuilder {
.write(true) .write(true)
.create(true) .create(true)
.truncate(true) .truncate(true)
.open(path)?; .open(building_path(path))?;
file.write_all(&MAGIC)?; file.write_all(&MAGIC)?;
file.write_all(&[0u8; 4])?; file.write_all(&[0u8; 4])?;
file.write_all(&(n as u64).to_le_bytes())?; file.write_all(&(n as u64).to_le_bytes())?;
@@ -229,8 +229,8 @@ impl PersistentBitVecBuilder {
} }
pub fn build_from(source: &PersistentBitVec, path: &Path) -> io::Result<Self> { pub fn build_from(source: &PersistentBitVec, path: &Path) -> io::Result<Self> {
fs::copy(source.path(), path)?; fs::copy(source.path(), building_path(path))?;
let file = OpenOptions::new().read(true).write(true).open(path)?; let file = OpenOptions::new().read(true).write(true).open(building_path(path))?;
let mmap = unsafe { MmapMut::map_mut(&file)? }; let mmap = unsafe { MmapMut::map_mut(&file)? };
let n = source.len(); let n = source.len();
Ok(Self { Ok(Self {
@@ -441,7 +441,8 @@ impl PersistentBitVecBuilder {
} }
pub fn close(self) -> io::Result<()> { pub fn close(self) -> io::Result<()> {
self.mmap.flush() self.mmap.flush()?;
publish_built(&self.path)
} }
pub fn finish(self) -> io::Result<PersistentBitVec> { pub fn finish(self) -> io::Result<PersistentBitVec> {
+3 -2
View File
@@ -15,7 +15,7 @@ use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut}; use memmap2::{Mmap, MmapMut};
use crate::mmap_file::{create_mmap_file, open_mmap_file}; use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built};
const MAGIC: [u8; 4] = *b"PFIV"; const MAGIC: [u8; 4] = *b"PFIV";
@@ -178,7 +178,8 @@ impl PersistentFixedIntVecBuilder {
} }
pub fn close(self) -> io::Result<()> { pub fn close(self) -> io::Result<()> {
self.mmap.flush() self.mmap.flush()?;
publish_built(&self.path)
} }
pub fn finish(self) -> io::Result<PersistentFixedIntVec> { pub fn finish(self) -> io::Result<PersistentFixedIntVec> {
+48 -38
View File
@@ -1,5 +1,5 @@
use std::fs::OpenOptions; use std::fs::OpenOptions;
use std::io::{self, BufWriter, Seek, SeekFrom, Write as _}; use std::io::{self, BufWriter, Seek, SeekFrom, Write};
use std::path::Path; use std::path::Path;
pub const MAGIC: [u8; 4] = *b"PCIV"; pub const MAGIC: [u8; 4] = *b"PCIV";
@@ -54,53 +54,63 @@ pub fn parse_index_entry(data: &[u8], base: usize, i: usize) -> (usize, usize) {
// Sparse index target: ≤ 32 KB in L1 cache (16 B per entry → 2048 entries). // Sparse index target: ≤ 32 KB in L1 cache (16 B per entry → 2048 entries).
pub const L1_INDEX_ENTRIES: usize = 2048; pub const L1_INDEX_ENTRIES: usize = 2048;
/// Sparse-index `step` and entry count for `n_overflow` overflow entries:
/// no index (`step == 0`) while the overflow section fits the L1 target.
fn pciv_index_layout(n_overflow: usize) -> (usize, usize) {
if n_overflow <= L1_INDEX_ENTRIES {
(0, 0)
} else {
let step = n_overflow.div_ceil(L1_INDEX_ENTRIES);
(step, n_overflow.div_ceil(step))
}
}
/// Write the 40-byte PCIV header for a vector of `n` slots with
/// `n_overflow` overflow entries (index layout derived from it).
pub(crate) fn write_pciv_header<W: Write>(w: &mut W, n: usize, n_overflow: usize) -> io::Result<()> {
let (step, n_index) = pciv_index_layout(n_overflow);
w.write_all(&MAGIC)?;
w.write_all(&[0u8; 4])?;
w.write_all(&(n as u64).to_le_bytes())?;
w.write_all(&(n_overflow as u64).to_le_bytes())?;
w.write_all(&(n_index as u64).to_le_bytes())?;
w.write_all(&(step as u64).to_le_bytes())
}
/// Write the sections that follow the primary bytes:
/// overflow n_overflow × 12 B (slot: u64, value: u32), `entries` sorted by slot
/// index n_index × 16 B (slot: u64, pos: u64) sparse index
pub(crate) fn write_pciv_tail<W: Write>(w: &mut W, entries: &[(usize, u32)]) -> io::Result<()> {
for &(slot, value) in entries {
w.write_all(&(slot as u64).to_le_bytes())?;
w.write_all(&value.to_le_bytes())?;
}
let (step, _) = pciv_index_layout(entries.len());
if step > 0 {
for (block, chunk) in entries.chunks(step).enumerate() {
w.write_all(&(chunk[0].0 as u64).to_le_bytes())?;
w.write_all(&((block * step) as u64).to_le_bytes())?;
}
}
Ok(())
}
/// Finalise a PCIV file whose placeholder header and primary section are already on disk. /// Finalise a PCIV file whose placeholder header and primary section are already on disk.
/// ///
/// Truncates the file to `HEADER_SIZE + n`, then appends: /// Truncates the file to `HEADER_SIZE + n`, appends the overflow and index
/// overflow n_overflow × 12 B (slot: u64, value: u32) sorted by slot /// sections ([`write_pciv_tail`]) and overwrites the placeholder header at
/// index n_index × 16 B (slot: u64, pos: u64) sparse index /// offset 0 ([`write_pciv_header`]).
/// and overwrites the placeholder header at offset 0.
pub fn finalize_pciv(path: &Path, n: usize, entries: &[(usize, u32)]) -> io::Result<()> { pub fn finalize_pciv(path: &Path, n: usize, entries: &[(usize, u32)]) -> io::Result<()> {
let n_overflow = entries.len();
let file = OpenOptions::new().read(true).write(true).open(path)?; let file = OpenOptions::new().read(true).write(true).open(path)?;
file.set_len((HEADER_SIZE + n) as u64)?; file.set_len((HEADER_SIZE + n) as u64)?;
let mut w = BufWriter::new(file); let mut w = BufWriter::new(file);
w.seek(SeekFrom::End(0))?; w.seek(SeekFrom::End(0))?;
write_pciv_tail(&mut w, entries)?;
for &(slot, value) in entries {
w.write_all(&(slot as u64).to_le_bytes())?;
w.write_all(&value.to_le_bytes())?;
}
let step: usize = if n_overflow <= L1_INDEX_ENTRIES {
0
} else {
n_overflow.div_ceil(L1_INDEX_ENTRIES)
};
let n_index: usize = if step > 0 {
let count = n_overflow.div_ceil(step);
for (block, chunk) in entries.chunks(step).enumerate() {
let slot = chunk[0].0 as u64;
let pos = (block * step) as u64;
w.write_all(&slot.to_le_bytes())?;
w.write_all(&pos.to_le_bytes())?;
}
count
} else {
0
};
w.flush()?; w.flush()?;
let mut file = w.into_inner().map_err(|e| e.into_error())?; let mut file = w.into_inner().map_err(|e| e.into_error())?;
file.seek(SeekFrom::Start(0))?; file.seek(SeekFrom::Start(0))?;
file.write_all(&MAGIC)?; write_pciv_header(&mut file, n, entries.len())?;
file.write_all(&[0u8; 4])?;
file.write_all(&(n as u64).to_le_bytes())?;
file.write_all(&(n_overflow as u64).to_le_bytes())?;
file.write_all(&(n_index as u64).to_le_bytes())?;
file.write_all(&(step as u64).to_le_bytes())?;
file.flush() file.flush()
} }
+10 -1
View File
@@ -11,6 +11,7 @@ use crate::bitmatrix::{
use crate::builder::PersistentCompactIntVecBuilder; use crate::builder::PersistentCompactIntVecBuilder;
use crate::colgroup::{ColGroup, MatrixGroupOps, chunked_presence_count}; use crate::colgroup::{ColGroup, MatrixGroupOps, chunked_presence_count};
use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE}; use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE};
use crate::mem_builder::MemCompactIntVecBuilder;
use crate::meta::MatrixMeta; use crate::meta::MatrixMeta;
use crate::reader::PersistentCompactIntVec; use crate::reader::PersistentCompactIntVec;
use crate::sparse_intmatrix::{PersistentSparseCompactIntMatrix, is_present as sparse_is_present}; use crate::sparse_intmatrix::{PersistentSparseCompactIntMatrix, is_present as sparse_is_present};
@@ -297,7 +298,7 @@ impl PackedCompactIntMatrix {
/// without mapping the file. Used by `pack_compact_int_matrix` to tell a /// without mapping the file. Used by `pack_compact_int_matrix` to tell a
/// genuinely complete pack from a stale one that predates a later /// genuinely complete pack from a stale one that predates a later
/// column-widening. /// column-widening.
fn packed_int_matrix_n_cols(path: &Path) -> io::Result<usize> { pub(crate) fn packed_int_matrix_n_cols(path: &Path) -> io::Result<usize> {
let mut f = File::open(path)?; let mut f = File::open(path)?;
let mut header = [0u8; PCMX_HEADER]; let mut header = [0u8; PCMX_HEADER];
f.read_exact(&mut header)?; f.read_exact(&mut header)?;
@@ -717,6 +718,14 @@ impl PersistentCompactIntMatrixBuilder {
PersistentCompactIntVecBuilder::new(self.n, &path) PersistentCompactIntVecBuilder::new(self.n, &path)
} }
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
/// the column is built in RAM and written once at `close`.
pub fn add_mem_col(&mut self) -> MemCompactIntVecBuilder {
let path = col_path(&self.dir, self.n_cols);
self.n_cols += 1;
MemCompactIntVecBuilder::new(self.n, &path)
}
pub fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()> { pub fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()> {
src.make_persistent(&col_path(&self.dir, self.n_cols))?; src.make_persistent(&col_path(&self.dir, self.n_cols))?;
self.n_cols += 1; self.n_cols += 1;
+6
View File
@@ -9,16 +9,20 @@ mod int_group_ops;
mod intmatrix; mod intmatrix;
mod layer_meta; mod layer_meta;
mod matrix_builder; mod matrix_builder;
mod mem_builder;
mod meta; mod meta;
mod mmap_file; mod mmap_file;
mod rankselect; mod rankselect;
mod reader; mod reader;
mod sparse_intmatrix; mod sparse_intmatrix;
mod sparse_intmatrix_columnar;
mod storage_kind; mod storage_kind;
mod tempbitvec; mod tempbitvec;
mod tempintvec; mod tempintvec;
mod transpose;
pub mod traits; pub mod traits;
mod views; mod views;
mod views_scan;
pub use bitmatrix::{ pub use bitmatrix::{
PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentSparseBitMatrix, PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentSparseBitMatrix,
@@ -35,6 +39,8 @@ pub use intmatrix::{
}; };
pub use layer_meta::LayerMeta; pub use layer_meta::LayerMeta;
pub use mem_builder::{MemBitVecBuilder, MemCompactIntVecBuilder};
pub use mmap_file::{building_path, publish_built};
pub use matrix_builder::{ColBuilder, MatrixBuilder}; pub use matrix_builder::{ColBuilder, MatrixBuilder};
pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder}; pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
pub use reader::{Iter as CompactIntVecIter, PersistentCompactIntVec}; pub use reader::{Iter as CompactIntVecIter, PersistentCompactIntVec};
+31 -1
View File
@@ -10,13 +10,19 @@ use std::path::Path;
use crate::{PersistentBitMatrixBuilder as BitMB, PersistentBitVecBuilder}; use crate::{PersistentBitMatrixBuilder as BitMB, PersistentBitVecBuilder};
use crate::{PersistentCompactIntMatrixBuilder as IntMB, PersistentCompactIntVecBuilder}; use crate::{PersistentCompactIntMatrixBuilder as IntMB, PersistentCompactIntVecBuilder};
use crate::{TempBitVec, TempCompactIntVec}; use crate::{MemBitVecBuilder, MemCompactIntVecBuilder, TempBitVec, TempCompactIntVec};
// ── ColBuilder ──────────────────────────────────────────────────────────────── // ── ColBuilder ────────────────────────────────────────────────────────────────
/// One column being built. `Bit`/`Int` write through a mmap of a sparse
/// file as values are set; `MemBit`/`MemInt` keep the column in RAM and
/// write it once at `close` (same file, byte for byte — see
/// [`MemBitVecBuilder`]).
pub enum ColBuilder { pub enum ColBuilder {
Bit(PersistentBitVecBuilder), Bit(PersistentBitVecBuilder),
Int(PersistentCompactIntVecBuilder), Int(PersistentCompactIntVecBuilder),
MemBit(MemBitVecBuilder),
MemInt(MemCompactIntVecBuilder),
} }
impl ColBuilder { impl ColBuilder {
@@ -24,6 +30,8 @@ impl ColBuilder {
match self { match self {
ColBuilder::Bit(b) => b.set(slot, value > 0), ColBuilder::Bit(b) => b.set(slot, value > 0),
ColBuilder::Int(b) => b.set(slot, value), ColBuilder::Int(b) => b.set(slot, value),
ColBuilder::MemBit(b) => b.set(slot, value > 0),
ColBuilder::MemInt(b) => b.set(slot, value),
} }
} }
@@ -31,6 +39,8 @@ impl ColBuilder {
match self { match self {
ColBuilder::Bit(b) => b.close(), ColBuilder::Bit(b) => b.close(),
ColBuilder::Int(b) => b.close(), ColBuilder::Int(b) => b.close(),
ColBuilder::MemBit(b) => b.close(),
ColBuilder::MemInt(b) => b.close(),
} }
} }
} }
@@ -64,6 +74,15 @@ impl MatrixBuilder {
}) })
} }
/// Number of columns created so far (absent + data columns, including
/// those inherited from a resumed directory).
pub fn n_cols(&self) -> usize {
match self {
MatrixBuilder::Bit(b) => b.n_cols(),
MatrixBuilder::Int(b) => b.n_cols(),
}
}
/// Add a column with no data written (all-zero/false) — for columns /// Add a column with no data written (all-zero/false) — for columns
/// absent from a given source (e.g. destination-only genomes not /// absent from a given source (e.g. destination-only genomes not
/// contributed by the source being merged in). /// contributed by the source being merged in).
@@ -81,6 +100,17 @@ impl MatrixBuilder {
}) })
} }
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
/// the column is built in RAM and written once, sequentially, at
/// `close`: no sparse file, no writable mapping. Costs `n / 8` bytes
/// (presence) or `n` bytes + overflow (counts) of RAM until closed.
pub fn add_mem_col(&mut self) -> ColBuilder {
match self {
MatrixBuilder::Bit(b) => ColBuilder::MemBit(b.add_mem_col()),
MatrixBuilder::Int(b) => ColBuilder::MemInt(b.add_mem_col()),
}
}
/// Add a column already computed as a bit vector. /// Add a column already computed as a bit vector.
pub fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> { pub fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> {
match self { match self {
+122
View File
@@ -0,0 +1,122 @@
//! In-memory column builders: the content of a column is built in RAM and
//! written once, sequentially, when the builder is closed.
//!
//! They produce exactly the same file, byte for byte, as
//! [`PersistentBitVecBuilder`](crate::PersistentBitVecBuilder) /
//! [`PersistentCompactIntVecBuilder`](crate::PersistentCompactIntVecBuilder),
//! but never create a sparse file nor a writable mapping: a block the file
//! system cannot allocate (full disk, quota, network FS) is reported as an
//! `io::Error` by `close`, instead of a SIGBUS on the page fault that first
//! touches it. The price is that the whole column lives in RAM until
//! `close`: `n / 8` bytes for bits, `n` bytes (+ overflow map) for counts.
use std::collections::HashMap;
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use crate::bitvec::{n_words, MAGIC as PBIV_MAGIC};
use crate::format::{write_pciv_header, write_pciv_tail};
use crate::mmap_file::{write_built, write_std_header};
// ── Bits (PBIV) ───────────────────────────────────────────────────────────────
/// In-memory counterpart of `PersistentBitVecBuilder` (`set` + `close` only).
pub struct MemBitVecBuilder {
words: Vec<u64>,
n: usize,
path: PathBuf,
}
impl MemBitVecBuilder {
/// All-false vector of `n` bits, to be written at `path` by `close`.
pub fn new(n: usize, path: &Path) -> Self {
Self { words: vec![0u64; n_words(n)], n, path: path.to_path_buf() }
}
#[inline]
pub fn len(&self) -> usize {
self.n
}
#[inline]
pub fn is_empty(&self) -> bool {
self.n == 0
}
#[inline]
pub fn set(&mut self, slot: usize, value: bool) {
debug_assert!(slot < self.n);
let bit = 1u64 << (slot & 63);
if value {
self.words[slot >> 6] |= bit;
} else {
self.words[slot >> 6] &= !bit;
}
}
/// Writes the PBIV file — header, then the words in native byte order,
/// exactly as the mmap-backed builder lays them out — and publishes it.
pub fn close(self) -> io::Result<()> {
let Self { words, n, path } = self;
write_built(&path, |w| {
write_std_header(w, PBIV_MAGIC, [0u8; 4], n)?;
for word in &words {
w.write_all(&word.to_ne_bytes())?;
}
Ok(())
})
}
}
// ── Compact ints (PCIV) ───────────────────────────────────────────────────────
/// In-memory counterpart of `PersistentCompactIntVecBuilder` (`set` +
/// `close` only). Same encoding: one byte per slot, values ≥ 255 stored as
/// the sentinel 255 plus an overflow entry.
pub struct MemCompactIntVecBuilder {
primary: Vec<u8>,
overflow: HashMap<usize, u32>,
n: usize,
path: PathBuf,
}
impl MemCompactIntVecBuilder {
/// All-zero vector of `n` slots, to be written at `path` by `close`.
pub fn new(n: usize, path: &Path) -> Self {
Self { primary: vec![0u8; n], overflow: HashMap::new(), n, path: path.to_path_buf() }
}
#[inline]
pub fn len(&self) -> usize {
self.n
}
#[inline]
pub fn is_empty(&self) -> bool {
self.n == 0
}
#[inline]
pub fn set(&mut self, slot: usize, value: u32) {
if value < 255 {
self.primary[slot] = value as u8;
self.overflow.remove(&slot);
} else {
self.primary[slot] = 255;
self.overflow.insert(slot, value);
}
}
/// Writes the PCIV file — header, primary bytes, overflow entries sorted
/// by slot, sparse index — and publishes it.
pub fn close(self) -> io::Result<()> {
let Self { primary, overflow, n, path } = self;
let mut entries: Vec<(usize, u32)> = overflow.into_iter().collect();
entries.sort_unstable_by_key(|&(slot, _)| slot);
write_built(&path, |w| {
write_pciv_header(w, n, entries.len())?;
w.write_all(&primary)?;
write_pciv_tail(w, &entries)
})
}
}
+59 -8
View File
@@ -10,13 +10,66 @@
//! `block_ranks`) and its own higher-level API — those genuinely differ //! `block_ranks`) and its own higher-level API — those genuinely differ
//! between formats and aren't forced into a shared shape. //! between formats and aren't forced into a shared shape.
use std::fs::{File, OpenOptions}; use std::fs::{self, File, OpenOptions};
use std::io::{self, Seek, SeekFrom, Write as _}; use std::io::{self, BufWriter, Seek, SeekFrom, Write};
use std::path::Path; use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut}; use memmap2::{Mmap, MmapMut};
/// Creates (or truncates) `path`, writes the standard header — `magic` + /// Suffix of the name a file lives under while it is being built.
const BUILDING_SUFFIX: &str = ".building";
/// The name `path` is built under: a file only appears at `path` itself once
/// complete ([`publish_built`]). A process killed mid-build leaves at most a
/// `.building` file, never a `path` with a valid header and unwritten data
/// that `path.exists()` would take for a finished one — what every
/// "skip it, it already exists" caller (per-layer annexes) relies on.
pub fn building_path(path: &Path) -> PathBuf {
let mut name = path.as_os_str().to_owned();
name.push(BUILDING_SUFFIX);
PathBuf::from(name)
}
/// Atomically moves the file built under [`building_path`]`(path)` to `path`
/// (same directory, so a plain `rename`), replacing any previous one. To be
/// called once the data is complete and flushed.
pub fn publish_built(path: &Path) -> io::Result<()> {
fs::rename(building_path(path), path)
}
/// Writes the crate-wide standard header: `magic` + `pad` + `n as u64` (LE).
pub(crate) fn write_std_header<W: Write>(w: &mut W, magic: [u8; 4], pad: [u8; 4], n: usize) -> io::Result<()> {
w.write_all(&magic)?;
w.write_all(&pad)?;
w.write_all(&(n as u64).to_le_bytes())
}
/// Writes a whole file sequentially through `write` into
/// [`building_path`]`(path)`, flushes it, then publishes it at `path`. No
/// sparse file and no writable mapping: an allocation failure on the file
/// system (full disk, quota) surfaces as an `io::Error` from a `write`,
/// never as a SIGBUS on a page fault.
///
/// Deliberately no `sync_all`: one forced sync per file made the merge
/// ~8× slower (thousands of column files; measured 2026-11-17 on macOS),
/// and the mmap builders it replaces never synced either. Durability is the
/// OS's, like the rest of an index; the `rename` still guarantees that no
/// partially written file is ever published. Caveat: on a write-back
/// network FS an allocation error may only be reported at `close`, which
/// `File`'s drop ignores.
pub(crate) fn write_built(
path: &Path,
write: impl FnOnce(&mut BufWriter<File>) -> io::Result<()>,
) -> io::Result<()> {
let mut w = BufWriter::new(File::create(building_path(path))?);
write(&mut w)?;
w.flush()?;
drop(w.into_inner().map_err(|e| e.into_error())?);
publish_built(path)
}
/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it
/// with [`publish_built`] once complete — writes the standard header — `magic` +
/// `pad` (4 bytes right after `magic`; usually all-zero, but /// `pad` (4 bytes right after `magic`; usually all-zero, but
/// `PersistentFixedIntVec` repurposes byte 4 to store its `width`) + /// `PersistentFixedIntVec` repurposes byte 4 to store its `width`) +
/// `n as u64`, followed by `extra_header_zeros` more zero bytes for a /// `n as u64`, followed by `extra_header_zeros` more zero bytes for a
@@ -37,10 +90,8 @@ pub(crate) fn create_mmap_file(
.write(true) .write(true)
.create(true) .create(true)
.truncate(true) .truncate(true)
.open(path)?; .open(building_path(path))?;
file.write_all(&magic)?; write_std_header(&mut file, magic, pad, n)?;
file.write_all(&pad)?;
file.write_all(&(n as u64).to_le_bytes())?;
if extra_header_zeros > 0 { if extra_header_zeros > 0 {
file.write_all(&vec![0u8; extra_header_zeros])?; file.write_all(&vec![0u8; extra_header_zeros])?;
} }
+3 -2
View File
@@ -21,7 +21,7 @@ use std::path::{Path, PathBuf};
use common_traits::SelectInWord; use common_traits::SelectInWord;
use memmap2::{Mmap, MmapMut}; use memmap2::{Mmap, MmapMut};
use crate::mmap_file::{create_mmap_file, open_mmap_file}; use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built};
const MAGIC: [u8; 4] = *b"PRSB"; const MAGIC: [u8; 4] = *b"PRSB";
@@ -268,7 +268,8 @@ impl PersistentRankSelectBitVecBuilder {
let block_off = HEADER_SIZE + bits_bytes(self.n); let block_off = HEADER_SIZE + bits_bytes(self.n);
let block_bytes = u64_slice_to_le_bytes(&block_ranks); let block_bytes = u64_slice_to_le_bytes(&block_ranks);
self.mmap[block_off..block_off + block_bytes.len()].copy_from_slice(&block_bytes); self.mmap[block_off..block_off + block_bytes.len()].copy_from_slice(&block_bytes);
self.mmap.flush() self.mmap.flush()?;
publish_built(&self.path)
} }
pub fn finish(self) -> io::Result<PersistentRankSelectBitVec> { pub fn finish(self) -> io::Result<PersistentRankSelectBitVec> {
+49 -25
View File
@@ -33,6 +33,7 @@ use ndarray::{Array1, Array2};
use crate::bitmatrix::RowRank; use crate::bitmatrix::RowRank;
use crate::builder::PersistentCompactIntVecBuilder; use crate::builder::PersistentCompactIntVecBuilder;
use crate::eliasfano::{EliasFano, EliasFanoBuilder}; use crate::eliasfano::{EliasFano, EliasFanoBuilder};
use crate::meta::MatrixMeta;
use crate::reader::PersistentCompactIntVec; use crate::reader::PersistentCompactIntVec;
use crate::traits::{BitPartials, ColumnWeights, CountPartials}; use crate::traits::{BitPartials, ColumnWeights, CountPartials};
use crate::{PersistentIntMatrix, PersistentSparseBitMatrix, PersistentSparseBitMatrixBuilder}; use crate::{PersistentIntMatrix, PersistentSparseBitMatrix, PersistentSparseBitMatrixBuilder};
@@ -548,6 +549,29 @@ impl PersistentSparseCompactIntMatrixBuilder {
Ok(builder) Ok(builder)
} }
/// Row-permuted copy of `src`: row `s` of the result is row
/// `new_to_old[s]` of `src`. Same contract as
/// [`PersistentSparseBitMatrixBuilder::build_permuted`].
pub fn build_permuted(
src: &PersistentSparseCompactIntMatrix,
new_to_old: &[u32],
dir: &Path,
) -> io::Result<Self> {
let mut builder = Self::new(new_to_old.len(), src.n_cols(), dir)?;
let mut cols: Vec<u32> = Vec::new();
let mut vals: Vec<u32> = Vec::new();
for &old in new_to_old {
cols.clear();
vals.clear();
src.for_each_cell_in_row(old as usize, |c, v| {
cols.push(c as u32);
vals.push(v);
});
builder.push_row(&cols, &vals);
}
Ok(builder)
}
pub fn close(self) -> io::Result<()> { pub fn close(self) -> io::Result<()> {
let Self { let Self {
support, support,
@@ -594,37 +618,37 @@ impl PersistentSparseCompactIntMatrixBuilder {
// ── pack_sparse_compact_int_matrix ────────────────────────────────────────── // ── pack_sparse_compact_int_matrix ──────────────────────────────────────────
/// Converts whichever dense count-matrix format is on disk at `dir` /// Converts a layer's count matrix (columnar `col_*.pciv` or dense
/// (`Packed`, i.e. `matrix.pcmx`, or `Columnar`, i.e. `col_*.pciv` + /// `matrix.pcmx`) to the sparse on-disk format in place, exactly like
/// `meta.json`) into the sparse on-disk format in place — generic over the /// [`crate::pack_sparse_bit_matrix`]: columnar data is transposed straight
/// dense source the same way [`super::bitmatrix::pack_sparse_bit_matrix`] /// into the sparse format, window by window, one column file open at a time
/// is: detects the format, transposes straight from it, and only removes /// (no dense intermediate, no mapping); a dense `matrix.pcmx` is read only
/// the old dense files once the new format is fully written (a crash /// when it is the current form. Old-format files are removed only once the
/// mid-conversion leaves the previous, still-valid format rather than a /// new format is fully written. Idempotent: a no-op if the sparse format is
/// half-written one). Idempotent — a no-op if `singleton_values.pciv` /// already present.
/// already exists.
pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> { pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> {
use crate::intmatrix::{ColumnarCompactIntMatrix, PackedCompactIntMatrix, col_path}; use crate::intmatrix::{PackedCompactIntMatrix, pack_compact_int_matrix, packed_int_matrix_n_cols};
if is_present(dir) { if is_present(dir) {
return Ok(()); return Ok(());
} }
let packed_path = dir.join("matrix.pcmx"); let packed_path = dir.join("matrix.pcmx");
if packed_path.exists() {
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?); if let Ok(meta) = MatrixMeta::load(dir) {
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?; if packed_int_matrix_n_cols(&packed_path).ok() != Some(meta.n_cols) {
drop(dense); PersistentSparseCompactIntMatrixBuilder::build_from_columnar(dir, &meta)?.close()?;
fs::remove_file(&packed_path)?; for c in 0..meta.n_cols {
} else { fs::remove_file(crate::intmatrix::col_path(dir, c))?;
let dense = PersistentIntMatrix::Columnar(ColumnarCompactIntMatrix::open(dir)?); }
let n_cols = dense.n_cols(); fs::remove_file(dir.join("meta.json"))?;
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?; let _ = fs::remove_file(&packed_path);
drop(dense); return Ok(());
for c in 0..n_cols {
let _ = fs::remove_file(col_path(dir, c));
} }
let _ = fs::remove_file(dir.join("meta.json"));
} }
Ok(())
pack_compact_int_matrix(dir)?;
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?);
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
drop(dense);
fs::remove_file(&packed_path)
} }
@@ -0,0 +1,148 @@
//! Building the sparse count matrix straight from the columnar
//! `col_*.pciv` files, with no dense intermediate and no mapping. Counterpart
//! of `bitmatrix::sparse_columnar`; see [`crate::transpose`].
//!
//! A `PCIV` column is `header(40) | primary(n bytes) | overflow(n_overflow
//! × 12 B, sorted by slot) | index`. A primary byte `255` is a sentinel: the
//! true value (≥ 255) is the overflow entry of that slot. A window therefore
//! reads its primary bytes, then looks up the overflow entries of its
//! sentinels (binary search for the first one, one contiguous read for the
//! rest).
use std::fs::File;
use std::io;
use std::os::unix::fs::FileExt;
use std::path::Path;
use crate::format::{HEADER_SIZE, MAGIC, OVERFLOW_ENTRY_SIZE, parse_overflow_entry};
use crate::intmatrix::col_path;
use crate::meta::MatrixMeta;
use crate::sparse_intmatrix::PersistentSparseCompactIntMatrixBuilder;
use crate::transpose::{DEFAULT_MAX_HITS, DEFAULT_WINDOW_ROWS, Hit, transpose_windows};
fn invalid(path: &Path, what: &str) -> io::Error {
io::Error::new(io::ErrorKind::InvalidData, format!("{}: {what}", path.display()))
}
/// Appends the non-zero cells of column `c` (file `path`) for rows
/// `start..end` to `hits`, as `(row - start, c, value)`.
fn read_int_window(
path: &Path,
n: usize,
c: usize,
start: usize,
end: usize,
primary: &mut Vec<u8>,
hits: &mut Vec<Hit<u32>>,
) -> io::Result<()> {
let file = File::open(path)?;
let mut header = [0u8; HEADER_SIZE];
file.read_exact_at(&mut header, 0)?;
if header[..4] != MAGIC || u64::from_le_bytes(header[8..16].try_into().unwrap()) as usize != n {
return Err(invalid(path, &format!("not a PCIV column of {n} rows")));
}
let n_overflow = u64::from_le_bytes(header[16..24].try_into().unwrap()) as usize;
primary.clear();
primary.resize(end - start, 0);
file.read_exact_at(primary, (HEADER_SIZE + start) as u64)?;
// Slots (relative to `start`) whose primary byte is the 255 sentinel,
// pushed to `hits` with a placeholder and patched below.
let first_hit = hits.len();
let mut n_sentinels = 0usize;
let mut chunks = primary.chunks_exact(8);
let mut base = 0usize;
for chunk in &mut chunks {
if u64::from_le_bytes(chunk.try_into().unwrap()) != 0 {
for (j, &b) in chunk.iter().enumerate() {
if b != 0 {
n_sentinels += (b == 255) as usize;
hits.push(((base + j) as u32, c as u32, b as u32));
}
}
}
base += 8;
}
for (j, &b) in chunks.remainder().iter().enumerate() {
if b != 0 {
n_sentinels += (b == 255) as usize;
hits.push(((base + j) as u32, c as u32, b as u32));
}
}
if n_sentinels == 0 {
return Ok(());
}
// Overflow entries of this window: first entry with slot >= start, found
// by binary search over the on-disk section, then one contiguous read.
let ov_base = (HEADER_SIZE + n) as u64;
let slot_at = |i: usize| -> io::Result<usize> {
let mut b = [0u8; 8];
file.read_exact_at(&mut b, ov_base + (i * OVERFLOW_ENTRY_SIZE) as u64)?;
Ok(u64::from_le_bytes(b) as usize)
};
let (mut lo, mut hi) = (0usize, n_overflow);
while lo < hi {
let mid = lo + (hi - lo) / 2;
if slot_at(mid)? < start {
lo = mid + 1;
} else {
hi = mid;
}
}
if lo + n_sentinels > n_overflow {
return Err(invalid(path, "fewer overflow entries than sentinel bytes"));
}
let mut raw = vec![0u8; n_sentinels * OVERFLOW_ENTRY_SIZE];
file.read_exact_at(&mut raw, ov_base + (lo * OVERFLOW_ENTRY_SIZE) as u64)?;
let mut k = 0usize;
for h in &mut hits[first_hit..] {
if h.2 == 255 {
let (slot, value) = parse_overflow_entry(&raw, 0, k);
if slot != start + h.0 as usize {
return Err(invalid(path, "overflow entry does not match its sentinel slot"));
}
h.2 = value;
k += 1;
}
}
Ok(())
}
impl PersistentSparseCompactIntMatrixBuilder {
/// Builds the sparse count matrix of the columnar layer in `dir`
/// (`meta.json` plus `col_*.pciv`), writing the sparse files into `dir`.
/// Same rows, same cells and so same bytes as going through a dense
/// `matrix.pcmx` (`build_from_dense`), without ever writing it.
pub fn build_from_columnar(dir: &Path, meta: &MatrixMeta) -> io::Result<Self> {
Self::build_from_columnar_with(dir, meta, DEFAULT_WINDOW_ROWS, DEFAULT_MAX_HITS)
}
/// [`build_from_columnar`](Self::build_from_columnar) with explicit window
/// size (rows) and non-zero-cell cap; see [`crate::transpose`].
pub fn build_from_columnar_with(
dir: &Path,
meta: &MatrixMeta,
window_rows: usize,
max_hits: usize,
) -> io::Result<Self> {
let mut builder = Self::new(meta.n, meta.n_cols, dir)?;
let mut primary: Vec<u8> = Vec::new();
transpose_windows(
meta.n,
meta.n_cols,
window_rows,
max_hits,
0u32,
|c, start, end, hits| read_int_window(&col_path(dir, c), meta.n, c, start, end, &mut primary, hits),
|_, batch| {
for i in 0..batch.len() {
builder.push_row(batch.cols(i), batch.vals(i));
}
Ok(())
},
)?;
Ok(builder)
}
}
+14
View File
@@ -249,6 +249,20 @@ fn partial_threshold_jaccard_consistent() {
} }
} }
#[test]
fn threshold_hamming_counts_kmers_present_in_one_genome_only() {
// At threshold 2 the genomes (columns) hold the k-mers (rows):
// c0 = {r0, r2}, c1 = {r1}, c2 = {r1}.
let (_d, m) = make_matrix(&[&[3, 0, 2], &[1, 2, 0], &[0, 3, 1]]);
let h = m.threshold_hamming_dist_matrix(2);
let expected = [[0, 3, 3], [3, 0, 0], [3, 0, 0]];
for i in 0..3 {
for j in 0..3 {
assert_eq!(h[[i, j]], expected[i][j], "[{i},{j}]");
}
}
}
#[test] #[test]
fn single_col_distance_matrix_is_zero() { fn single_col_distance_matrix_is_zero() {
let (_d, m) = make_matrix(&[&[1, 2, 3]]); let (_d, m) = make_matrix(&[&[1, 2, 3]]);
@@ -0,0 +1,27 @@
use tempfile::tempdir;
use crate::MatrixBuilder;
#[test]
fn n_cols_counts_absent_data_and_resumed_columns() {
for presence in [true, false] {
let dir = tempdir().unwrap();
let data = dir.path().join("m");
let mut mb = MatrixBuilder::new(presence, 5, &data).unwrap();
assert_eq!(mb.n_cols(), 0);
mb.add_absent_col().unwrap();
assert_eq!(mb.n_cols(), 1);
let mut c = mb.add_col().unwrap();
c.set_val(2, 7);
c.close().unwrap();
assert_eq!(mb.n_cols(), 2);
mb.close().unwrap();
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
assert_eq!(mb.n_cols(), 2, "presence={presence}");
mb.add_absent_col().unwrap();
assert_eq!(mb.n_cols(), 3);
mb.close().unwrap();
}
}
+117
View File
@@ -0,0 +1,117 @@
use std::fs;
use tempfile::tempdir;
use crate::{
MatrixBuilder, MemBitVecBuilder, MemCompactIntVecBuilder, PersistentBitMatrix,
PersistentBitVecBuilder, PersistentCompactIntVecBuilder, PersistentIntMatrix,
};
const SIZES: [usize; 6] = [0, 1, 63, 64, 65, 10_000];
/// Deterministic xorshift64* — no RNG dependency needed for test data.
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
self.0 ^= self.0 >> 12;
self.0 ^= self.0 << 25;
self.0 ^= self.0 >> 27;
self.0.wrapping_mul(0x2545_F491_4F6C_DD1D)
}
}
/// `(slot, value)` writes in random order, with overwrites; for counts,
/// about a third of the values are ≥ 255 (overflow), enough at n = 10 000
/// to exceed `L1_INDEX_ENTRIES` and exercise the sparse index section.
fn writes(n: usize, seed: u64, max_small: u64) -> Vec<(usize, u32)> {
let mut rng = Rng(seed);
(0..2 * n)
.map(|_| {
let slot = (rng.next() % n as u64) as usize;
let r = rng.next();
let value = match r % 3 {
0 => 255 + (r >> 8) as u32 % 100_000, // overflow
1 => 0, // also clears a previous overflow
_ => (r >> 8) as u32 % max_small as u32,
};
(slot, value)
})
.collect()
}
#[test]
fn mem_bitvec_is_byte_identical_to_persistent() {
for n in SIZES {
let dir = tempdir().unwrap();
let (pp, mp) = (dir.path().join("p.pbiv"), dir.path().join("m.pbiv"));
let mut p = PersistentBitVecBuilder::new(n, &pp).unwrap();
let mut m = MemBitVecBuilder::new(n, &mp);
for (slot, v) in writes(n, 0x9E37_79B9 + n as u64, 2) {
p.set(slot, v > 0);
m.set(slot, v > 0);
}
p.close().unwrap();
m.close().unwrap();
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
}
}
#[test]
fn mem_compact_int_is_byte_identical_to_persistent() {
for n in SIZES {
let dir = tempdir().unwrap();
let (pp, mp) = (dir.path().join("p.pciv"), dir.path().join("m.pciv"));
let mut p = PersistentCompactIntVecBuilder::new(n, &pp).unwrap();
let mut m = MemCompactIntVecBuilder::new(n, &mp);
for (slot, v) in writes(n, 0xC2B2_AE35 + n as u64, 255) {
p.set(slot, v);
m.set(slot, v);
}
p.close().unwrap();
m.close().unwrap();
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
}
}
/// `add_mem_col` columns, interleaved with absent columns and continued
/// across a `resume`, read back through the regular matrix readers.
#[test]
fn add_mem_col_matrix_reads_back() {
for presence in [true, false] {
let n = 100;
let dir = tempdir().unwrap();
let data = dir.path().join(if presence { "presence" } else { "counts" });
let value = |c: usize, slot: usize| -> u32 {
if (slot + c).is_multiple_of(3) { (slot * 7 + c) as u32 % 400 + 1 } else { 0 }
};
let mut mb = MatrixBuilder::new(presence, n, &data).unwrap();
mb.add_absent_col().unwrap();
let mut c1 = mb.add_mem_col();
(0..n).for_each(|s| c1.set_val(s, value(1, s)));
c1.close().unwrap();
mb.close().unwrap();
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
assert_eq!(mb.n_cols(), 2);
let mut c2 = mb.add_mem_col();
(0..n).for_each(|s| c2.set_val(s, value(2, s)));
c2.close().unwrap();
assert_eq!(mb.n_cols(), 3);
mb.close().unwrap();
for s in 0..n {
let expect = [0, value(1, s), value(2, s)];
if presence {
let m = PersistentBitMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
let want: Vec<bool> = expect.iter().map(|&v| v > 0).collect();
assert_eq!(&*m.row(s), want.as_slice(), "slot {s}");
} else {
let m = PersistentIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
assert_eq!(&*m.row(s), &expect, "slot {s}");
}
}
}
}
+4
View File
@@ -5,9 +5,13 @@ mod eliasfano;
mod fixedintvec; mod fixedintvec;
mod int_group_ops; mod int_group_ops;
mod intmatrix; mod intmatrix;
mod matrix_builder;
mod mem_builder;
mod rankselect; mod rankselect;
mod sparse; mod sparse;
mod sparse_intmatrix; mod sparse_intmatrix;
mod transpose_columnar;
mod views_scan;
use tempfile::tempdir; use tempfile::tempdir;
+28
View File
@@ -415,3 +415,31 @@ fn partial_jaccard_and_hamming_match_dense_including_diagonal() {
drop(dir_dense); drop(dir_dense);
drop(dir_sparse); drop(dir_sparse);
} }
#[test]
fn build_permuted_follows_the_permutation() {
// Mix of singleton and multi rows, including a repeated multi set, so the
// dictionary dedup is exercised on the permuted order too.
let rows: Vec<&[bool]> = vec![
&[true, false, false, false],
&[false, false, true, false],
&[true, true, false, false],
&[false, false, true, true],
&[true, true, false, false],
];
let (_dir, src) = make_sparse(&rows, 4);
let new_to_old: Vec<u32> = vec![3, 0, 4, 1, 2];
let out = tempdir().unwrap();
let permuted =
PersistentSparseBitMatrixBuilder::build_permuted(&src, &new_to_old, &out.path().join("p"))
.unwrap()
.finish()
.unwrap();
assert_eq!(permuted.n(), src.n());
assert_eq!(permuted.n_cols(), src.n_cols());
for (new, &old) in new_to_old.iter().enumerate() {
assert_eq!(row_as_bool(&permuted, new), row_as_bool(&src, old as usize), "new row {new}");
}
}
@@ -304,3 +304,32 @@ fn pack_sparse_compact_int_matrix_from_already_packed_matrix() {
assert_eq!(&*sparse.row(slot), &**expected, "slot {slot}"); assert_eq!(&*sparse.row(slot), &**expected, "slot {slot}");
} }
} }
#[test]
fn build_permuted_follows_the_permutation() {
// Rows sharing a support with different values, plus singletons.
let rows: Vec<&[u32]> = vec![
&[7, 0, 0, 0],
&[0, 0, 3, 0],
&[2, 5, 0, 0],
&[9, 1, 0, 0],
&[0, 0, 4, 6],
];
let (_dir, src) = make_sparse(&rows, 4);
let new_to_old: Vec<u32> = vec![4, 2, 0, 3, 1];
let out = tempdir().unwrap();
let permuted = PersistentSparseCompactIntMatrixBuilder::build_permuted(
&src,
&new_to_old,
&out.path().join("p"),
)
.unwrap()
.finish()
.unwrap();
assert_eq!(permuted.n(), src.n());
for (new, &old) in new_to_old.iter().enumerate() {
assert_eq!(permuted.row(new), src.row(old as usize), "new row {new}");
}
}
@@ -0,0 +1,156 @@
//! The windowed columnar → sparse transposition must produce the very bytes
//! the dense route (`pack` to `matrix.pbmx`/`.pcmx`, then `build_from_dense`)
//! produces, whatever the window size and non-zero-cell cap — including the
//! retry-with-a-smaller-window path.
use std::fs;
use std::path::Path;
use tempfile::tempdir;
use crate::meta::MatrixMeta;
use crate::{
PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentCompactIntMatrixBuilder,
PersistentIntMatrix, PersistentSparseBitMatrixBuilder, PersistentSparseCompactIntMatrixBuilder,
pack_bit_matrix, pack_compact_int_matrix,
};
const N: usize = 5_000 + 37; // not a multiple of 64
const N_COLS: usize = 7;
fn xorshift(s: &mut u64) -> u64 {
*s ^= *s << 13;
*s ^= *s >> 7;
*s ^= *s << 17;
*s
}
/// Column `c`: all zero, very sparse, sparse, dense, ... so that rows have
/// from 0 to `N_COLS` cells and whole words are zero in some columns.
fn cell(c: usize, s: &mut u64) -> Option<u32> {
let r = xorshift(s);
let keep = match c % 4 {
0 => false,
1 => r % 97 == 0,
2 => r % 7 == 0,
_ => r % 2 == 0,
};
keep.then(|| match (r >> 20) % 11 {
0 => 255,
1 => 256,
2 => 70_000,
v => (v as u32) + 1,
})
}
fn build_bits(root: &Path) {
let mut b = PersistentBitMatrixBuilder::new(N, &root.join("presence")).unwrap();
let mut s = 0x9E37_79B9_7F4A_7C15u64;
for c in 0..N_COLS {
let mut cb = b.add_col().unwrap();
for slot in 0..N {
cb.set(slot, cell(c, &mut s).is_some());
}
cb.close().unwrap();
}
b.close().unwrap();
}
fn build_ints(root: &Path) {
let mut b = PersistentCompactIntMatrixBuilder::new(N, &root.join("counts")).unwrap();
let mut s = 0x9E37_79B9_7F4A_7C15u64;
for c in 0..N_COLS {
let mut cb = b.add_col().unwrap();
for slot in 0..N {
if let Some(v) = cell(c, &mut s) {
cb.set(slot, v);
}
}
cb.close().unwrap();
}
b.close().unwrap();
}
/// Sparse-format files of `dir` (columnar leftovers and dense files excluded),
/// as `(name, bytes)` sorted by name.
fn sparse_files(dir: &Path) -> Vec<(String, Vec<u8>)> {
let mut v: Vec<(String, Vec<u8>)> = fs::read_dir(dir)
.unwrap()
.map(|e| e.unwrap().path())
.filter(|p| {
let n = p.file_name().unwrap().to_string_lossy().into_owned();
!(n.starts_with("col_") || n == "meta.json" || n.starts_with("matrix."))
})
.map(|p| (p.file_name().unwrap().to_string_lossy().into_owned(), fs::read(&p).unwrap()))
.collect();
v.sort();
v
}
const WINDOWS: &[(usize, usize)] = &[
(64, usize::MAX / 4), // smallest window, never too dense
(192, 50), // forces the retry-with-half-window path
(4096, 10_000), // a few windows
(crate::transpose::DEFAULT_WINDOW_ROWS, crate::transpose::DEFAULT_MAX_HITS),
];
#[test]
fn bit_columnar_transpose_is_byte_identical_to_the_dense_route() {
let ref_root = tempdir().unwrap();
build_bits(ref_root.path());
let ref_dir = ref_root.path().join("presence");
pack_bit_matrix(&ref_dir).unwrap();
let dense = PersistentBitMatrix::open(ref_root.path()).unwrap();
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, &ref_dir).unwrap().close().unwrap();
drop(dense);
let expected = sparse_files(&ref_dir);
assert!(!expected.is_empty());
for &(window, max_hits) in WINDOWS {
let root = tempdir().unwrap();
build_bits(root.path());
let dir = root.path().join("presence");
let meta = MatrixMeta::load(&dir).unwrap();
PersistentSparseBitMatrixBuilder::build_from_columnar_with(&dir, &meta, window, max_hits)
.unwrap()
.close()
.unwrap();
assert_eq!(sparse_files(&dir), expected, "window={window} max_hits={max_hits}");
}
}
#[test]
fn int_columnar_transpose_is_byte_identical_to_the_dense_route() {
let ref_root = tempdir().unwrap();
build_ints(ref_root.path());
let ref_dir = ref_root.path().join("counts");
pack_compact_int_matrix(&ref_dir).unwrap();
let dense = PersistentIntMatrix::open(ref_root.path()).unwrap();
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, &ref_dir).unwrap().close().unwrap();
drop(dense);
let expected = sparse_files(&ref_dir);
assert!(!expected.is_empty());
for &(window, max_hits) in WINDOWS {
let root = tempdir().unwrap();
build_ints(root.path());
let dir = root.path().join("counts");
let meta = MatrixMeta::load(&dir).unwrap();
PersistentSparseCompactIntMatrixBuilder::build_from_columnar_with(&dir, &meta, window, max_hits)
.unwrap()
.close()
.unwrap();
assert_eq!(sparse_files(&dir), expected, "window={window} max_hits={max_hits}");
}
}
#[test]
fn columnar_transpose_rejects_a_column_of_another_length() {
let root = tempdir().unwrap();
build_bits(root.path());
let dir = root.path().join("presence");
let mut meta = MatrixMeta::load(&dir).unwrap();
meta.n += 64; // metadata claims more rows than the column files hold
let err = PersistentSparseBitMatrixBuilder::build_from_columnar(&dir, &meta).err().expect("must fail");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
}
+71
View File
@@ -0,0 +1,71 @@
use crate::views::BitSliceView;
/// Deterministic pseudo-random words (xorshift), thinned so that many words
/// are zero: the case the word-wise scan has to skip correctly.
fn words(n_words: usize, seed: u64, thin: u32) -> Vec<u64> {
let mut s = seed | 1;
(0..n_words)
.map(|_| {
let mut w = u64::MAX;
for _ in 0..thin {
s ^= s << 13;
s ^= s >> 7;
s ^= s << 17;
w &= s;
}
w
})
.collect()
}
fn naive(v: &BitSliceView<'_>, r: std::ops::Range<usize>) -> Vec<usize> {
r.filter(|&s| v.get(s)).collect()
}
fn scanned(v: &BitSliceView<'_>, r: std::ops::Range<usize>) -> Vec<usize> {
let mut out = Vec::new();
v.for_each_set_in(r, |s| out.push(s));
out
}
#[test]
fn for_each_set_in_matches_get_loop_on_all_boundaries() {
// Lengths around word boundaries; padding bits of the last word are zero.
for n in [0usize, 1, 63, 64, 65, 127, 128, 129, 200, 4096, 4097] {
for (seed, thin) in [(1u64, 0u32), (7, 1), (11, 3), (13, 6)] {
let mut w = words(n.div_ceil(64), seed, thin);
if n % 64 != 0 {
if let Some(last) = w.last_mut() {
*last &= (1u64 << (n % 64)) - 1;
}
}
let v = BitSliceView::new(&w, n);
let cuts = [0, 1, 63, 64, 65, 100, 127, 128, 129, n / 2, n.saturating_sub(1), n];
for &a in &cuts {
for &b in &cuts {
if a <= n && b <= n {
assert_eq!(scanned(&v, a..b), naive(&v, a..b), "n={n} seed={seed} thin={thin} range={a}..{b}");
}
}
}
}
}
}
#[test]
fn for_each_set_in_empty_and_reversed_ranges_do_nothing() {
let w = [u64::MAX; 2];
let v = BitSliceView::new(&w, 128);
assert!(scanned(&v, 5..5).is_empty());
#[allow(clippy::reversed_empty_ranges)]
let rev = 9..3;
assert!(scanned(&v, rev).is_empty());
}
#[test]
#[should_panic]
fn for_each_set_in_rejects_range_past_the_view() {
let w = [u64::MAX; 2];
let v = BitSliceView::new(&w, 100);
scanned(&v, 0..101);
}
+10
View File
@@ -129,6 +129,16 @@ pub trait CountPartials: ColumnWeights {
jaccard_to_mash(&self.threshold_jaccard_dist_matrix(threshold), k) jaccard_to_mash(&self.threshold_jaccard_dist_matrix(threshold), k)
} }
/// Hamming distance on presence/absence: the number of k-mers present
/// (count ≥ `threshold`) in exactly one of the two genomes, i.e. the union
/// minus the intersection that [`partial_threshold_jaccard`] already gives.
///
/// [`partial_threshold_jaccard`]: Self::partial_threshold_jaccard
fn threshold_hamming_dist_matrix(&self, threshold: u32) -> Array2<u64> {
let (inter, union) = self.partial_threshold_jaccard(threshold);
union - inter
}
fn relfreq_bray_dist_matrix(&self) -> Array2<f64> { fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
let global = self.col_weights(); let global = self.col_weights();
let mut m = self.partial_relfreq_bray(&global).mapv(|v| 1.0 - v); let mut m = self.partial_relfreq_bray(&global).mapv(|v| 1.0 - v);
+136
View File
@@ -0,0 +1,136 @@
//! Windowed column → row transposition, shared by the presence and count
//! packers.
//!
//! A columnar matrix stores one file per genome; the sparse formats are
//! built row by row, each row needing the cells of *every* column. Holding
//! every column mapped (or open) at once costs one mapping (or descriptor)
//! per column and per worker. This module never holds more than one column:
//! it cuts the rows into windows and, for each window, visits the columns
//! one after the other through a caller-supplied reader, collecting the
//! non-zero cells; the cells are then regrouped by row and handed to the
//! caller. No intermediate dense matrix is written.
//!
//! Memory is bounded by `max_hits`, not by the partition size: when a window
//! turns out denser than expected it is abandoned and retried with half as
//! many rows.
use std::io;
/// Default rows per window. A multiple of 64, so that a bit column's window
/// starts on a word boundary. 2 M rows = 256 KiB per bit column and read.
pub const DEFAULT_WINDOW_ROWS: usize = 1 << 21;
/// Default cap on the non-zero cells held for one window.
pub const DEFAULT_MAX_HITS: usize = 4 << 20;
/// Smallest window the retry logic shrinks to.
const MIN_WINDOW_ROWS: usize = 64;
/// One non-zero cell of a window: row relative to the window start, column,
/// value. `V = ()` for presence matrices.
pub(crate) type Hit<V> = (u32, u32, V);
/// The rows of one window, regrouped by row, columns ascending within a row.
pub(crate) struct RowBatch<V> {
offsets: Vec<u32>,
cols: Vec<u32>,
vals: Vec<V>,
}
impl<V: Copy> RowBatch<V> {
fn new() -> Self {
Self { offsets: Vec::new(), cols: Vec::new(), vals: Vec::new() }
}
/// Number of rows in the window.
pub(crate) fn len(&self) -> usize {
self.offsets.len().saturating_sub(1)
}
/// Columns of row `i`, ascending.
pub(crate) fn cols(&self, i: usize) -> &[u32] {
&self.cols[self.offsets[i] as usize..self.offsets[i + 1] as usize]
}
/// Values of row `i`, in the order of [`cols`](Self::cols).
pub(crate) fn vals(&self, i: usize) -> &[V] {
&self.vals[self.offsets[i] as usize..self.offsets[i + 1] as usize]
}
/// Stable counting sort of `hits` by row. `hits` was produced column by
/// column in ascending column order, so stability leaves each row's
/// columns ascending, as `push_row` requires, without a per-row sort.
fn fill(&mut self, rows: usize, hits: &[Hit<V>], zero: V) {
self.offsets.clear();
self.offsets.resize(rows + 1, 0);
for &(r, _, _) in hits {
self.offsets[r as usize + 1] += 1;
}
for i in 0..rows {
self.offsets[i + 1] += self.offsets[i];
}
self.cols.clear();
self.cols.resize(hits.len(), 0);
self.vals.clear();
self.vals.resize(hits.len(), zero);
// `next[r]` = write position of the next cell of row `r`; reuse the
// tail of `offsets` would alias, so keep a small separate cursor.
let mut next: Vec<u32> = self.offsets[..rows].to_vec();
for &(r, c, v) in hits {
let p = next[r as usize] as usize;
self.cols[p] = c;
self.vals[p] = v;
next[r as usize] += 1;
}
}
}
/// Transposes an `n`-row, `n_cols`-column matrix window by window.
///
/// `read_col(c, start, end, hits)` must append to `hits` every non-zero cell
/// of column `c` in rows `start..end`, as `(row - start, c, value)`.
/// `on_batch(start, batch)` receives each completed window, in ascending row
/// order; `batch` row `i` is matrix row `start + i`.
///
/// `window_rows` is rounded down to a multiple of 64 (minimum 64), and `start`
/// is always a multiple of 64.
pub(crate) fn transpose_windows<V: Copy>(
n: usize,
n_cols: usize,
window_rows: usize,
max_hits: usize,
zero: V,
mut read_col: impl FnMut(usize, usize, usize, &mut Vec<Hit<V>>) -> io::Result<()>,
mut on_batch: impl FnMut(usize, &RowBatch<V>) -> io::Result<()>,
) -> io::Result<()> {
let window_max = (window_rows / 64 * 64).clamp(MIN_WINDOW_ROWS, u32::MAX as usize / 2);
// `Hit` rows are u32 and offsets count hits in u32.
let max_hits = max_hits.min(u32::MAX as usize / 2);
let mut w = window_max;
let mut hits: Vec<Hit<V>> = Vec::new();
let mut batch = RowBatch::new();
let mut start = 0usize;
while start < n {
let end = (start + w).min(n);
hits.clear();
let mut too_dense = false;
for c in 0..n_cols {
read_col(c, start, end, &mut hits)?;
if hits.len() > max_hits && w > MIN_WINDOW_ROWS {
too_dense = true;
break;
}
}
if too_dense {
w = (w / 2 / 64 * 64).max(MIN_WINDOW_ROWS);
continue;
}
batch.fill(end - start, &hits, zero);
on_batch(start, &batch)?;
start = end;
if hits.len() < max_hits / 4 && w < window_max {
w = (w * 2).min(window_max);
}
}
Ok(())
}
+48
View File
@@ -0,0 +1,48 @@
//! Word-wise scans over a [`BitSliceView`]. Kept apart from `views.rs`, which
//! is already past the size budget; these are inherent methods of the same type.
use std::ops::Range;
use crate::views::BitSliceView;
impl BitSliceView<'_> {
/// Calls `f(slot)` for every set bit with `slot` in `range`, in ascending
/// order.
///
/// Walks the underlying `u64` words and jumps from set bit to set bit
/// with `trailing_zeros`, so the cost is proportional to the number of
/// words plus the number of set bits — not to the number of slots, as a
/// `get(slot)` loop would be. Words that are entirely zero cost one load
/// and one test, which is what makes this the right traversal for the
/// sparse presence columns `build_from_dense` reads.
///
/// Panics if `range.end > self.len()`.
#[inline]
pub fn for_each_set_in(&self, range: Range<usize>, mut f: impl FnMut(usize)) {
assert!(range.end <= self.n, "range end {} past view length {}", range.end, self.n);
if range.start >= range.end {
return;
}
let first = range.start >> 6;
let last = (range.end - 1) >> 6;
for w in first..=last {
let mut word = self.words[w];
if word == 0 {
continue;
}
if w == first {
word &= !0u64 << (range.start & 63);
}
if w == last {
let tail = range.end & 63;
if tail != 0 {
word &= (1u64 << tail) - 1;
}
}
while word != 0 {
f((w << 6) | word.trailing_zeros() as usize);
word &= word - 1;
}
}
}
}
-4
View File
@@ -5,13 +5,9 @@ edition = "2021"
[dependencies] [dependencies]
obikseq = { path = "../obikseq" } obikseq = { path = "../obikseq" }
obifastwrite = { path = "../obifastwrite" }
ahash = "0.8"
hashbrown = { version = "0.17", features = ["rayon"] } hashbrown = { version = "0.17", features = ["rayon"] }
rayon = "1" rayon = "1"
crossbeam-channel = "0.5" crossbeam-channel = "0.5"
xxhash-rust = { version = "0.8.15", features = ["xxh3", "const_xxh3"] }
tracing = "0.1"
[features] [features]
test-utils = [] test-utils = []
+70 -41
View File
@@ -1,19 +1,19 @@
use hashbrown::HashMap; use hashbrown::{HashMap, HashSet};
use obikseq::k; use obikseq::k;
use obikseq::{CanonicalKmer, Unitig}; use obikseq::{CanonicalKmer, Unitig};
#[cfg(not(any(test, feature = "test-utils")))] #[cfg(not(any(test, feature = "test-utils")))]
use rayon::iter::{IntoParallelRefIterator, ParallelIterator}; use rayon::iter::{IntoParallelRefIterator, ParallelIterator};
use std::cell::RefCell; use std::cell::RefCell;
use std::sync::atomic::{AtomicU8, Ordering}; use std::sync::atomic::{AtomicU8, Ordering};
use xxhash_rust::xxh3::Xxh3Builder;
use super::kmer_hash::KmerHashBuilder;
use super::node::{IS_VISITED_MASK, Node}; use super::node::{IS_VISITED_MASK, Node};
use super::unitig_iter::UnitigNucIter; use super::unitig_iter::UnitigNucIter;
use super::walk::WalkState; use super::walk::WalkState;
// ── Types ───────────────────────────────────────────────────────────────────── // ── Types ─────────────────────────────────────────────────────────────────────
pub(super) type FastHashMap<K, V> = HashMap<K, V, Xxh3Builder>; pub(super) type FastHashMap<K, V> = HashMap<K, V, KmerHashBuilder>;
// ── GraphDeBruijn ───────────────────────────────────────────────────────────── // ── GraphDeBruijn ─────────────────────────────────────────────────────────────
@@ -24,7 +24,7 @@ pub struct GraphDeBruijn {
impl GraphDeBruijn { impl GraphDeBruijn {
pub fn new() -> Self { pub fn new() -> Self {
Self { Self {
nodes: FastHashMap::with_hasher(Xxh3Builder::new()), nodes: FastHashMap::with_hasher(KmerHashBuilder),
} }
} }
@@ -40,7 +40,7 @@ impl GraphDeBruijn {
pub fn with_capacity(capacity: usize) -> Self { pub fn with_capacity(capacity: usize) -> Self {
Self { Self {
nodes: FastHashMap::with_capacity_and_hasher(capacity, Xxh3Builder::new()), nodes: FastHashMap::with_capacity_and_hasher(capacity, KmerHashBuilder),
} }
} }
@@ -79,7 +79,7 @@ impl GraphDeBruijn {
if node.is_visited() { if node.is_visited() {
return; return;
} }
if self.is_start(*kmer, node) { if self.is_start(*kmer, node, atomic, true) || self.is_start(*kmer, node, atomic, false) {
node.set_start(); node.set_start();
atomic.store(node.0, Ordering::Relaxed); atomic.store(node.0, Ordering::Relaxed);
} }
@@ -98,45 +98,40 @@ impl GraphDeBruijn {
} }
} }
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState { pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState<'_> {
let node = Node( let atomic = self
self.nodes .nodes
.get(&kmer) .get(&kmer)
.expect("Cannot start a walk from not indexed kmer") .expect("Cannot start a walk from not indexed kmer");
.load(Ordering::Relaxed), let node = Node(atomic.load(Ordering::Relaxed));
);
debug_assert!( debug_assert!(
!node.is_visited(), !node.is_visited(),
"Cannot start a walk from a visited node" "Cannot start a walk from a visited node"
); );
WalkState::new(kmer, node, true) WalkState::new(kmer, node, true, atomic)
} }
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState> { pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState<'_>> {
let node = Node(self.nodes.get(&kmer)?.load(Ordering::Relaxed)); let atomic = self.nodes.get(&kmer)?;
let node = Node(atomic.load(Ordering::Relaxed));
if node.is_visited() { if node.is_visited() {
return None; return None;
} }
Some(WalkState::new(kmer, node, true)) Some(WalkState::new(kmer, node, true, atomic))
} }
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize) -> Option<UnitigNucIter<'_>> { pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize, direct: bool) -> Option<UnitigNucIter<'_>> {
let old = self let atomic = self.nodes.get(&kmer)?;
.nodes let old = atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
.get(&kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
if old & IS_VISITED_MASK != 0 { if old & IS_VISITED_MASK != 0 {
return None; return None;
} }
let start = WalkState::new(kmer, Node(old), true); let start = WalkState::new(kmer, Node(old), direct, atomic);
let next_step = start.walk(self).and_then(|(next_state, nuc)| { let next_step = start.walk(self).and_then(|(next_state, nuc)| {
let ext_old = self let ext_old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
.nodes
.get(&next_state.kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
(ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc)) (ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc))
}); });
Some(UnitigNucIter::new(self, kmer, k, next_step)) Some(UnitigNucIter::new(self, kmer, k, direct, next_step))
} }
pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) { pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) {
@@ -157,7 +152,8 @@ impl GraphDeBruijn {
.then_some(kmer) .then_some(kmer)
}) })
.for_each(|kmer| { .for_each(|kmer| {
if let Some(iter) = self.unitig_nucleotides(kmer, k) { let Some(direct) = self.start_direction(kmer) else { return };
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
n_new.fetch_add(1, Ordering::Relaxed); n_new.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -166,7 +162,8 @@ impl GraphDeBruijn {
#[cfg(any(test, feature = "test-utils"))] #[cfg(any(test, feature = "test-utils"))]
self.nodes.iter().for_each(|(&kmer, atomic)| { self.nodes.iter().for_each(|(&kmer, atomic)| {
if Node(atomic.load(Ordering::Acquire)).is_start() { if Node(atomic.load(Ordering::Acquire)).is_start() {
if let Some(iter) = self.unitig_nucleotides(kmer, k) { let Some(direct) = self.start_direction(kmer) else { return };
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
n_new.fetch_add(1, Ordering::Relaxed); n_new.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -187,8 +184,8 @@ impl GraphDeBruijn {
if node.is_visited() { if node.is_visited() {
continue; continue;
} }
let chain_start = self.find_chain_start(kmer); let (chain_start, direct) = self.find_chain_start(kmer);
if let Some(iter) = self.unitig_nucleotides(chain_start, k) { if let Some(iter) = self.unitig_nucleotides(chain_start, k, direct) {
n2.fetch_add(1, Ordering::Relaxed); n2.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -196,7 +193,7 @@ impl GraphDeBruijn {
// Safe because unitig_nucleotides may have visited kmer in the // Safe because unitig_nucleotides may have visited kmer in the
// meantime — in that case it returns None and we skip. // meantime — in that case it returns None and we skip.
if chain_start != kmer { if chain_start != kmer {
if let Some(iter) = self.unitig_nucleotides(kmer, k) { if let Some(iter) = self.unitig_nucleotides(kmer, k, true) {
n2.fetch_add(1, Ordering::Relaxed); n2.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -216,17 +213,34 @@ impl GraphDeBruijn {
self.nodes.extend(other.nodes); self.nodes.extend(other.nodes);
} }
fn find_chain_start(&self, kmer: CanonicalKmer) -> CanonicalKmer { /// Walks backwards from `kmer` to the first node of its chain. Returns
let node = Node(self.nodes[&kmer].load(Ordering::Acquire)); /// that node together with the orientation to read it in to walk the chain
let mut state = WalkState::new(kmer, node, false); /// forwards: the backward walk traverses nodes in the orientation
let mut seen = std::collections::HashSet::new(); /// `state.direct`, so the forward one is its opposite. (Dropping the
/// orientation, as this function once did, started the forward walk at the
/// wrong end and emitted a lone-kmer unitig.) On a cycle, `kmer` itself
/// in the canonical orientation.
fn find_chain_start(&self, kmer: CanonicalKmer) -> (CanonicalKmer, bool) {
let atomic = &self.nodes[&kmer];
let node = Node(atomic.load(Ordering::Acquire));
let mut state = WalkState::new(kmer, node, false, atomic);
// Most nodes reaching this pass are branch nodes or tips that cannot
// step at all: answer before allocating the cycle-detection set.
let Some((first, _)) = state.walk(self) else {
return (state.kmer, !state.direct);
};
let mut seen: HashSet<(u64, bool), KmerHashBuilder> = HashSet::with_hasher(KmerHashBuilder);
seen.insert((state.kmer.raw(), state.direct)); seen.insert((state.kmer.raw(), state.direct));
if !seen.insert((first.kmer.raw(), first.direct)) {
return (kmer, true);
}
state = first;
loop { loop {
match state.walk(self) { match state.walk(self) {
None => return state.kmer, None => return (state.kmer, !state.direct),
Some((next, _)) => { Some((next, _)) => {
if !seen.insert((next.kmer.raw(), next.direct)) { if !seen.insert((next.kmer.raw(), next.direct)) {
return kmer; return (kmer, true);
} }
state = next; state = next;
} }
@@ -234,8 +248,23 @@ impl GraphDeBruijn {
} }
} }
fn is_start(&self, query: CanonicalKmer, node: Node) -> bool { /// Orientation in which a node flagged as a start must be read: the
!WalkState::new(query, node, true).reachable(self) /// canonical one if a unitig starts there that way, else the reverse
/// complement. `None` if it was visited in the meantime.
fn start_direction(&self, kmer: CanonicalKmer) -> Option<bool> {
let atomic = self.nodes.get(&kmer)?;
let node = Node(atomic.load(Ordering::Acquire));
if node.is_visited() {
return None;
}
Some(self.is_start(kmer, node, atomic, true) || !self.is_start(kmer, node, atomic, false))
}
/// `true` when `query`, traversed in orientation `direct` (canonical or
/// reverse-complemented), has no unique predecessor — i.e. a unitig
/// starts there when read in that orientation.
fn is_start(&self, query: CanonicalKmer, node: Node, atomic: &AtomicU8, direct: bool) -> bool {
!WalkState::new(query, node, direct, atomic).reachable(self)
} }
pub fn try_for_each_unitig<E, F>(&self, f: F) -> Result<(), E> pub fn try_for_each_unitig<E, F>(&self, f: F) -> Result<(), E>
+55
View File
@@ -0,0 +1,55 @@
//! Hasher for the graph's `HashMap<CanonicalKmer, _>`.
//!
//! The key is one `u64` (the derived `Hash` of a kmer is a single
//! `write_u64`), so the generic streaming hashers used before (xxh3's
//! `Hasher` implementation: buffer, update, finish) cost far more than the
//! hash itself — ≈ 35 % of the active CPU of `index` in the profile of
//! `DevDocMD/implementation/mphf_batch_queries.md`, §17. A 64-bit finaliser
//! (splitmix64's) is a bijection with full avalanche, which is what
//! hashbrown needs from it (low bits for the bucket, top 7 for the tag).
use std::hash::{BuildHasher, Hasher};
#[derive(Clone, Copy, Default)]
pub(super) struct KmerHashBuilder;
impl BuildHasher for KmerHashBuilder {
type Hasher = KmerHasher;
#[inline]
fn build_hasher(&self) -> KmerHasher {
KmerHasher(0)
}
}
pub(super) struct KmerHasher(u64);
#[inline]
fn mix64(x: u64) -> u64 {
let x = x ^ (x >> 30);
let x = x.wrapping_mul(0xbf58476d1ce4e5b9);
let x = x ^ (x >> 27);
let x = x.wrapping_mul(0x94d049bb133111eb);
x ^ (x >> 31)
}
impl Hasher for KmerHasher {
#[inline]
fn write_u64(&mut self, x: u64) {
self.0 = mix64(self.0 ^ x);
}
/// Generic fallback (never taken for a kmer key): folds the bytes 8 at a time.
fn write(&mut self, bytes: &[u8]) {
for chunk in bytes.chunks(8) {
let mut buf = [0u8; 8];
buf[..chunk.len()].copy_from_slice(chunk);
self.write_u64(u64::from_le_bytes(buf));
}
}
#[inline]
fn finish(&self) -> u64 {
self.0
}
}
+2 -1
View File
@@ -1,10 +1,11 @@
//! De Bruijn graph over canonical k-mers, built for unitig extraction. //! De Bruijn graph over canonical k-mers, built for unitig extraction.
//! //!
//! Submodules: [`node`] (packed per-kmer neighbour/visited/start flags), //! Submodules: [`kmer_hash`] (hasher of the node table), [`node`] (packed per-kmer neighbour/visited/start flags),
//! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself), //! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself),
//! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator). //! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator).
mod graph; mod graph;
mod kmer_hash;
mod node; mod node;
mod unitig_iter; mod unitig_iter;
mod walk; mod walk;
+12 -8
View File
@@ -10,9 +10,11 @@ use super::walk::WalkState;
pub struct UnitigNucIter<'a> { pub struct UnitigNucIter<'a> {
graph: &'a GraphDeBruijn, graph: &'a GraphDeBruijn,
start: CanonicalKmer, start: CanonicalKmer,
/// Orientation the unitig is read in (`false`: reverse complement of `start`).
direct: bool,
pos: usize, pos: usize,
k: usize, k: usize,
next_step: Option<(WalkState, u8)>, next_step: Option<(WalkState<'a>, u8)>,
} }
impl<'a> UnitigNucIter<'a> { impl<'a> UnitigNucIter<'a> {
@@ -20,11 +22,13 @@ impl<'a> UnitigNucIter<'a> {
graph: &'a GraphDeBruijn, graph: &'a GraphDeBruijn,
start: CanonicalKmer, start: CanonicalKmer,
k: usize, k: usize,
next_step: Option<(WalkState, u8)>, direct: bool,
next_step: Option<(WalkState<'a>, u8)>,
) -> Self { ) -> Self {
Self { Self {
graph, graph,
start, start,
direct,
pos: 0, pos: 0,
k, k,
next_step, next_step,
@@ -37,16 +41,16 @@ impl Iterator for UnitigNucIter<'_> {
fn next(&mut self) -> Option<u8> { fn next(&mut self) -> Option<u8> {
if self.pos < self.k { if self.pos < self.k {
let nuc = self.start.nucleotide(self.pos); let nuc = if self.direct {
self.start.nucleotide(self.pos)
} else {
3 - self.start.nucleotide(self.k - 1 - self.pos)
};
self.pos += 1; self.pos += 1;
Some(nuc) Some(nuc)
} else if let Some((state, nuc)) = self.next_step.take() { } else if let Some((state, nuc)) = self.next_step.take() {
self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| { self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| {
let old = self let old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
.graph
.nodes
.get(&next_state.kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
(old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc)) (old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc))
}); });
Some(nuc) Some(nuc)
+22 -12
View File
@@ -1,35 +1,41 @@
use obikseq::{CanonicalKmer, Sequence}; use obikseq::{CanonicalKmer, Sequence};
use std::sync::atomic::Ordering; use std::sync::atomic::{AtomicU8, Ordering};
use super::graph::GraphDeBruijn; use super::graph::GraphDeBruijn;
use super::node::Node; use super::node::Node;
pub struct WalkState { /// A position in a walk: the node's kmer, its flags as read when the step was
/// taken, and the node's own table entry. Keeping `atomic` means the caller
/// that wants to mark the node visited does not look the kmer up a second
/// time (a random access in a table far larger than the cache).
pub struct WalkState<'g> {
pub(super) kmer: CanonicalKmer, pub(super) kmer: CanonicalKmer,
pub(super) node: Node, pub(super) node: Node,
pub(super) direct: bool, pub(super) direct: bool,
pub(super) atomic: &'g AtomicU8,
} }
impl WalkState { impl<'g> WalkState<'g> {
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool) -> Self { pub fn new(kmer: CanonicalKmer, node: Node, direct: bool, atomic: &'g AtomicU8) -> Self {
debug_assert!(!node.is_visited(), "Cannot walk over a visited node"); debug_assert!(!node.is_visited(), "Cannot walk over a visited node");
Self { kmer, node, direct } Self { kmer, node, direct, atomic }
} }
pub fn leavable(&self, graph: &GraphDeBruijn) -> bool { pub fn leavable(&self, graph: &'g GraphDeBruijn) -> bool {
self.walk(graph).is_some() self.walk(graph).is_some()
} }
pub fn reachable(&self, graph: &GraphDeBruijn) -> bool { pub fn reachable(&self, graph: &'g GraphDeBruijn) -> bool {
WalkState { WalkState {
kmer: self.kmer, kmer: self.kmer,
node: self.node, node: self.node,
direct: !self.direct, direct: !self.direct,
atomic: self.atomic,
} }
.leavable(graph) .leavable(graph)
} }
pub fn walk(&self, graph: &GraphDeBruijn) -> Option<(WalkState, u8)> { pub fn walk(&self, graph: &'g GraphDeBruijn) -> Option<(WalkState<'g>, u8)> {
if self.direct { if self.direct {
if !self.node.can_extend_right() { if !self.node.can_extend_right() {
return None; return None;
@@ -38,7 +44,8 @@ impl WalkState {
let next = self.kmer.into_kmer().push_right(nuc); let next = self.kmer.into_kmer().push_right(nuc);
let cnext = next.canonical(); let cnext = next.canonical();
let dnext = next.raw() == cnext.raw(); let dnext = next.raw() == cnext.raw();
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed)); let next_atomic = graph.nodes.get(&cnext).unwrap();
let next_node = Node(next_atomic.load(Ordering::Relaxed));
if next_node.is_visited() { if next_node.is_visited() {
return None; return None;
} }
@@ -52,6 +59,7 @@ impl WalkState {
kmer: cnext, kmer: cnext,
node: next_node, node: next_node,
direct: dnext, direct: dnext,
atomic: next_atomic,
}, },
nuc, nuc,
)) ))
@@ -63,20 +71,22 @@ impl WalkState {
let next = self.kmer.into_kmer().push_left(nuc); let next = self.kmer.into_kmer().push_left(nuc);
let cnext = next.canonical(); let cnext = next.canonical();
let dnext = next.raw() != cnext.raw(); let dnext = next.raw() != cnext.raw();
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed)); let next_atomic = graph.nodes.get(&cnext).unwrap();
let next_node = Node(next_atomic.load(Ordering::Relaxed));
if next_node.is_visited() { if next_node.is_visited() {
return None; return None;
} }
let reachable = if dnext { let reachable = if dnext {
next_node.can_extend_right()
} else {
next_node.can_extend_left() next_node.can_extend_left()
} else {
next_node.can_extend_right()
}; };
reachable.then_some(( reachable.then_some((
WalkState { WalkState {
kmer: cnext, kmer: cnext,
node: next_node, node: next_node,
direct: dnext, direct: dnext,
atomic: next_atomic,
}, },
3 - nuc, 3 - nuc,
)) ))
+83
View File
@@ -317,3 +317,86 @@ fn branching_graph_no_kmer_lost_or_duplicated() {
); );
assert_eq!(got, expected, "k-mer sets differ"); assert_eq!(got, expected, "k-mer sets differ");
} }
// ── orientation independence ──────────────────────────────────────────────
// Deterministic pseudo-random ASCII sequence (xorshift), so the tests need no
// RNG dependency.
fn pseudo_random_ascii(seed: u64, len: usize) -> Vec<u8> {
let mut st = seed.wrapping_mul(0x9E3779B97F4A7C15) | 1;
(0..len)
.map(|_| {
st ^= st << 13;
st ^= st >> 7;
st ^= st << 17;
b"ACGT"[(st % 4) as usize]
})
.collect()
}
#[test]
fn linear_chain_is_one_unitig_whatever_the_orientations() {
// A repeat-free sequence is one maximal unitig. Each k-mer is stored in
// its canonical orientation, so across many sequences the chain's nodes
// take every mix of orientations — including chains whose two ends are
// both stored reverse-complemented, which once came out as several
// unitigs (a lone-kmer unitig per mis-oriented chain start) depending
// only on the table's iteration order.
let k = 21; // long enough that (k-1)-mer collisions (= branches) do not occur
let _guard = lock_k();
set_k(k);
for seed in 1..=300u64 {
let seq = pseudo_random_ascii(seed, 90);
let kmers = canonical_kmers(&seq);
if kmers.len() != seq.len() - k + 1 {
continue; // accidental internal repeat: not a single chain
}
let g = graph_from_ascii(&seq);
g.compute_degrees_and_mark_starts();
let unitigs = collect_unitigs(&g);
assert_eq!(unitigs.len(), 1, "seed {seed}: repeat-free sequence must give one unitig");
assert_eq!(unitigs[0].seql(), seq.len(), "seed {seed}: unitig spans the sequence");
assert_eq!(kmers_from_unitigs(&unitigs), kmers, "seed {seed}: same k-mers");
}
}
#[test]
fn unitigs_partition_the_kmers_of_branching_reads() {
// Reads of a small genome on both strands with substitutions: branches,
// bubbles and tips. Whatever the shape, every k-mer must be covered
// exactly once: sum over unitigs of (seql - k + 1) == number of nodes.
let k = 11;
let _guard = lock_k();
set_k(k);
let genome = pseudo_random_ascii(7, 600);
let mut g = GraphDeBruijn::new();
for r in 0..120u64 {
let start = (pseudo_random_ascii(100 + r, 1)[0] as usize * 31 + r as usize * 17) % 500;
let mut read = genome[start..start + 90].to_vec();
if r % 2 == 1 {
// second strand
read = read
.iter()
.rev()
.map(|&b| match b {
b'A' => b'T',
b'C' => b'G',
b'G' => b'C',
_ => b'A',
})
.collect();
}
if r % 3 == 0 {
let at = (r as usize * 13) % read.len();
read[at] = if read[at] == b'A' { b'C' } else { b'A' };
}
for i in 0..=read.len() - k {
g.push(Kmer::from_ascii(&read[i..i + k]).unwrap().canonical());
}
}
g.compute_degrees_and_mark_starts();
let unitigs = collect_unitigs(&g);
let covered: usize = unitigs.iter().map(|u| u.seql() - k + 1).sum();
assert_eq!(covered, g.len(), "every k-mer in exactly one unitig");
assert_eq!(kmers_from_unitigs(&unitigs).len(), g.len(), "no k-mer missing");
}
+1 -1
View File
@@ -81,7 +81,7 @@ impl IndexDump for Arc<KmerIndex> {
if remaining.load(Ordering::Relaxed) == 0 { return Ok(vec![]); } if remaining.load(Ordering::Relaxed) == 0 { return Ok(vec![]); }
let mut buf = Vec::<u8>::new(); let mut buf = Vec::<u8>::new();
let try_write = |buf: &mut Vec<u8>, row: &[u32], prefix: &str| -> bool { let try_write = |buf: &mut Vec<u8>, row: &[u32], prefix: &str| -> bool {
match remaining.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |cur| { match remaining.try_update(Ordering::SeqCst, Ordering::SeqCst, |cur| {
if cur > 0 { Some(cur - 1) } else { None } if cur > 0 { Some(cur - 1) } else { None }
}) { }) {
Err(_) => false, Err(_) => false,
-19
View File
@@ -188,11 +188,6 @@ impl IndexCache {
self.iter().flat_map(|l| l.iter_unitigs()) self.iter().flat_map(|l| l.iter_unitigs())
} }
#[inline]
pub fn hash(&self, partition: usize, layer: usize, kmer: CanonicalKmer) -> Option<usize> {
Some(self.get_layer(partition, layer)?.hash(kmer))
}
pub fn n_kmer(&self, partition: usize, layer: usize) -> Option<usize> { pub fn n_kmer(&self, partition: usize, layer: usize) -> Option<usize> {
Some(self.get_layer(partition, layer)?.n()) Some(self.get_layer(partition, layer)?.n())
} }
@@ -228,20 +223,6 @@ impl IndexCache {
None None
} }
/// Membership-checked lookup — routes `kmer` straight to its one
/// possible partition (`CanonicalKmer::partition`, the same
/// minimizer-hash rule every k-mer was scattered by when the index was
/// built) instead of scanning every cached partition: a k-mer can only
/// ever live in the partition it hashes to, so anywhere else is not
/// worth even asking. Uses `self.index().n_partitions()` — the index's
/// real total, not [`n_partition`](Self::n_partition) (this cache's own
/// count, which can be a subset — see [`new`](Self::new)). `None` if
/// that partition isn't cached or doesn't carry `kmer`.
pub fn find(&self, kmer: CanonicalKmer) -> Option<(usize, usize, usize)> {
let partition = kmer.partition(self.raw_index.n_partitions());
self.find_in_partition(partition, kmer)
}
/// Parallel map-reduce over every cached `KmerLayer` (across every /// Parallel map-reduce over every cached `KmerLayer` (across every
/// cached partition), bounded to `max_concurrent` layers scanned at /// cached partition), bounded to `max_concurrent` layers scanned at
/// once — a cap on how many layers are actively being mapped /// once — a cap on how many layers are actively being mapped
+3 -7
View File
@@ -5,16 +5,12 @@ edition = "2024"
[dependencies] [dependencies]
obikseq = { path = "../obikseq" } obikseq = { path = "../obikseq" }
obitaxonomy = { path = "../obitaxonomy" }
obiskio = { path = "../obiskio" } obiskio = { path = "../obiskio" }
obisys = { path = "../obisys", default-features = false } obisys = { path = "../obisys", default-features = false }
obicompactvec = { path = "../obicompactvec" } obicompactvec = { path = "../obicompactvec" }
obidebruinj = { path = "../obidebruinj" } cacheline-ef = "2.0"
obipipeline = { path = "../obipipeline" } epserde = "0.13"
obikentropy = { path = "../obikentropy" } ptr_hash = { version = "2.1", features = ["epserde"] }
cacheline-ef = "1.1"
epserde = "0.8"
ptr_hash = "1.1"
niffler = "3.0.0" niffler = "3.0.0"
memmap2 = "0.9.10" memmap2 = "0.9.10"
ndarray = "0.17" ndarray = "0.17"
+4 -2
View File
@@ -90,7 +90,7 @@ impl IndexBuilder for KmerIndex {
let idx = KmerIndex { let idx = KmerIndex {
root_path: output.to_owned(), root_path: output.to_owned(),
meta: Arc::new(meta), meta: Arc::new(meta),
lock: std::sync::Mutex::new(None), lock: obisys::LockSlot::empty(output),
}; };
if !idx.acquire_lock() { if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other( return Err(OKIError::Io(std::io::Error::other(
@@ -102,7 +102,9 @@ impl IndexBuilder for KmerIndex {
fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> { fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> {
let output = output.as_ref(); let output = output.as_ref();
let idx = KmerIndex::open(output)?; // `output` is held exclusively by the builder that wrote it: reopen
// without a lock, a locking `open` would wait for that builder for ever.
let idx = KmerIndex::open_unlocked(output)?;
idx.meta.mark_indexed().map_err(OKIError::Io)?; idx.meta.mark_indexed().map_err(OKIError::Io)?;
let t_pack = Stage::start("pack"); let t_pack = Stage::start("pack");
idx.pack_matrices(sparse)?; idx.pack_matrices(sparse)?;
+163 -59
View File
@@ -1,6 +1,6 @@
use std::fs; use std::fs;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex}; use std::sync::Arc;
use crate::layer::KmerLayer; use crate::layer::KmerLayer;
use crate::partition::KmerPartition; use crate::partition::KmerPartition;
@@ -15,7 +15,7 @@ use crate::index::meta::{GenomeInfo, IndexConfig, IndexMeta};
pub struct KmerIndex { pub struct KmerIndex {
pub(crate) root_path: PathBuf, pub(crate) root_path: PathBuf,
pub(crate) meta: Arc<IndexMeta>, pub(crate) meta: Arc<IndexMeta>,
pub(crate) lock: Mutex<Option<obisys::DirLock>>, pub(crate) lock: obisys::LockSlot,
} }
impl KmerIndex { impl KmerIndex {
@@ -37,11 +37,8 @@ impl KmerIndex {
set_m(config.minimizer_size); set_m(config.minimizer_size);
let genomes = genome_info.into_iter().collect(); let genomes = genome_info.into_iter().collect();
let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?; let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?;
let idx = Self { let lock = obisys::LockSlot::empty(&root_path);
root_path, let idx = Self { root_path, meta: Arc::new(meta), lock };
meta: Arc::new(meta),
lock: Mutex::new(None),
};
if !idx.acquire_lock() { if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other( return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock right after create", "failed to acquire index lock right after create",
@@ -50,67 +47,92 @@ impl KmerIndex {
Ok(idx) Ok(idx)
} }
/// Open an existing index for reading — doesn't touch the directory /// Open an existing index for reading, holding a **shared** directory
/// lock at all: concurrent readers (`query`, `dump`, a `merge` source, /// lock for as long as the returned index lives: any number of readers
/// ...) must never block on each other. A caller that needs exclusive /// coexist, but a writer ([`open_lock`](Self::open_lock),
/// access (resuming an interrupted build) uses /// [`with_lock`](Self::with_lock)) waits for them all, and readers wait
/// [`open_lock`](Self::open_lock) instead; either way, `self.lock` /// for a writer at work — a reader never sees files change under it.
/// starts empty — [`test_lock`](Self::test_lock)/ /// The lock is taken before the metadata is read.
/// [`acquire_lock`](Self::acquire_lock) are opt-in from here. ///
/// Never open, from a process that holds the write lock on a directory
/// through another handle, that same directory again with `open`: the
/// new handle would wait for the first for ever.
pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> { pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let root_path = path.as_ref().to_owned(); Self::open_with(path.as_ref(), obisys::LockMode::Shared)
let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
set_k(meta.config.kmer_size);
set_m(meta.config.minimizer_size);
Ok(Self {
root_path,
meta: Arc::new(meta),
lock: Mutex::new(None),
})
} }
/// `open`, then hold the directory lock exclusively — for a caller /// `open`, holding the directory lock **exclusively** for as long as the
/// that resumes a write pipeline (e.g. `obikmer index` re-run on an /// returned index lives — **the** way for a command that modifies an
/// existing, not-yet-`Indexed` directory) rather than reading /// existing index in place (`convert`, `pack`, `annotate`, `utils
/// concurrently with others. /// rename`, resuming an interrupted build, ...) to claim it. The lock is
/// taken *before* the metadata is read, so what the caller reads is what
/// it will modify.
///
/// A command that only reads, or that writes a *new* index elsewhere
/// (`merge`, `filter`, `select`: exclusive on their destination, shared
/// on their sources), uses [`open`](Self::open). Library code that has to
/// write into an index it was handed (annexes built on demand) wraps that
/// write in [`with_lock`](Self::with_lock).
pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> { pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let idx = Self::open(path)?; Self::open_with(path.as_ref(), obisys::LockMode::Exclusive)
if !idx.acquire_lock() { }
return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock", fn open_with(path: &Path, mode: obisys::LockMode) -> OKIResult<Self> {
))); // A directory that is not an index is reported by `open_unlocked`,
// without a lock file being created in it.
if !Self::is_an_index(path) {
return Self::open_unlocked(path);
} }
let lock = match mode {
obisys::LockMode::Shared => obisys::LockSlot::shared(path),
obisys::LockMode::Exclusive => obisys::LockSlot::exclusive(path),
}
.map_err(OKIError::Io)?;
let mut idx = Self::open_unlocked(path)?;
idx.lock = lock;
Ok(idx) Ok(idx)
} }
/// Block until this index's directory lock is acquired, then hold it /// `open` without taking any lock — for a caller that already holds the
/// (released on [`release_lock`](Self::release_lock) or when `self` is /// directory through another handle (`finalize_indexed`, reopening the
/// dropped). Returns `false` only on a genuine I/O failure acquiring /// index its own builder holds exclusively).
/// the lock file itself — contention blocks rather than failing. pub(crate) fn open_unlocked(path: &Path) -> OKIResult<Self> {
pub fn acquire_lock(&self) -> bool { let root_path = path.to_owned();
match obisys::DirLock::acquire(&self.root_path) { let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
Ok(lock) => { set_k(meta.config.kmer_size);
*self.lock.lock().unwrap() = Some(lock); set_m(meta.config.minimizer_size);
true let lock = obisys::LockSlot::empty(&root_path);
} Ok(Self { root_path, meta: Arc::new(meta), lock })
Err(_) => false, }
}
/// Take the exclusive directory lock and hold it until `self` is dropped
/// — for an index just created, which is about to be written to.
/// Idempotent. Returns `false` only on a genuine I/O failure.
pub(crate) fn acquire_lock(&self) -> bool {
self.lock.acquire_exclusive().is_ok()
}
/// Whether this index object currently holds its directory lock
/// exclusively (a shared lock is not "write access").
pub fn holds_write_lock(&self) -> bool {
self.lock.holds_write_lock()
}
/// Run `f` with the directory lock held **exclusively** — promoting the
/// shared lock a handle from [`open`](Self::open) holds, and restoring it
/// afterwards; see [`obisys::LockSlot::with_exclusive`], including what
/// `f` must re-check. Library code that writes into an index it was
/// handed (annexes built on demand) wraps that write in this.
pub fn with_lock<T>(&self, f: impl FnOnce() -> OKIResult<T>) -> OKIResult<T> {
self.lock.with_exclusive(f)
} }
/// Non-blocking probe of this index's directory lock — doesn't retain /// Non-blocking probe of this index's directory lock — doesn't retain
/// it either way: `true` if the lock was free (briefly acquired, then /// anything: `true` if no process currently holds it exclusively, `false`
/// immediately released), `false` if another process currently holds /// if one does (this process included). For a caller that wants to warn
/// it. For a read-only caller that wants to warn about a build in /// about a write in progress without waiting for it.
/// progress without itself becoming exclusive; use
/// [`acquire_lock`](Self::acquire_lock)/[`open_lock`](Self::open_lock)
/// to actually hold it.
pub fn test_lock(&self) -> bool { pub fn test_lock(&self) -> bool {
matches!(obisys::DirLock::try_acquire(&self.root_path), Ok(Some(_))) self.lock.is_free_of_writer()
}
/// Release this index's directory lock, if held. A no-op if it isn't.
pub fn release_lock(&self) {
*self.lock.lock().unwrap() = None;
} }
/// Return `true` if `path` points to a valid obikmer index directory. /// Return `true` if `path` points to a valid obikmer index directory.
@@ -210,6 +232,14 @@ impl KmerIndex {
crate::partition::index_dir(&self.root_path, i) crate::partition::index_dir(&self.root_path, i)
} }
/// Path of layer `j` of partition `i` — pure path computation, nothing
/// is opened (unlike [`layer`](Self::layer)/`layer_dir`, which open the
/// layer's MPHF). For maintenance tools that must reach a layer's files
/// without going through a format they can't read yet.
pub fn layer_path(&self, i: usize, j: usize) -> PathBuf {
crate::layer::utils::layer_dir(&self.index_dir(i), j)
}
/// Number of layers in partition `i` — a disk scan, not a metadata read /// Number of layers in partition `i` — a disk scan, not a metadata read
/// (see [`KmerPartition::n_layers`](crate::partition::KmerPartition::n_layers)). /// (see [`KmerPartition::n_layers`](crate::partition::KmerPartition::n_layers)).
pub fn n_layers(&self, i: usize) -> OKIResult<usize> { pub fn n_layers(&self, i: usize) -> OKIResult<usize> {
@@ -250,7 +280,7 @@ impl KmerIndex {
/// on-disk formats (`PersistentSparseBitMatrix` for presence, /// on-disk formats (`PersistentSparseBitMatrix` for presence,
/// `PersistentSparseCompactIntMatrix` for counts) — smaller and faster /// `PersistentSparseCompactIntMatrix` for counts) — smaller and faster
/// for single-row access on real, sparse data; column-oriented access /// for single-row access on real, sparse data; column-oriented access
/// (`--metric` distance matrices) is much slower on the sparse format. /// (`phylo --distance` matrices) is much slower on the sparse format.
/// ///
/// Public — and not moved out of this crate, despite covering the same /// Public — and not moved out of this crate, despite covering the same
/// "maintenance operation on an already-built index" territory as /// "maintenance operation on an already-built index" territory as
@@ -266,7 +296,14 @@ impl KmerIndex {
}; };
let n = self.n_partitions(); let n = self.n_partitions();
let order: Vec<usize> = (0..n).collect(); // Largest partitions first (LFD), so the long jobs do not end up as
// stragglers at the tail. Sizes are `stat()` only; the sort is stable,
// ties keep partition-number order. Partitions are independent, so
// the order has no effect on what is written.
let sizes: Vec<u64> = (0..n).map(|i| self.pending_matrix_bytes(i)).collect();
let mut order: Vec<usize> = (0..n).collect();
order.sort_by_key(|&i| std::cmp::Reverse(sizes[i]));
let mut timings: Vec<(usize, u64, std::time::Duration)> = Vec::with_capacity(n);
let pb = progress_bar("pack", n as u64, "partitions"); let pb = progress_bar("pack", n as u64, "partitions");
obisys::PartitionRunner::new().run( obisys::PartitionRunner::new().run(
&order, &order,
@@ -297,14 +334,50 @@ impl KmerIndex {
} }
Ok(()) Ok(())
}, },
|_, _, _| { |i, _, elapsed| {
tracing::debug!(
partition = i,
matrix_bytes = sizes[i],
secs = elapsed.as_secs_f64(),
"pack: partition done"
);
timings.push((i, sizes[i], elapsed));
pb.inc(1); pb.inc(1);
}, },
)?; )?;
pb.finish_and_clear(); pb.finish_and_clear();
report_pack_timings(&timings);
Ok(()) Ok(())
} }
/// Bytes of matrix data (`presence/` and `counts/` of every layer) still
/// present in partition `i`, from `stat()` only. Before packing this is
/// the columnar data, i.e. a proxy for the work `pack_matrices` has to do
/// on that partition. Missing partition or directories count as 0.
pub fn pending_matrix_bytes(&self, i: usize) -> u64 {
let Ok(n_layers) = self.n_layers(i) else {
return 0;
};
let mut total = 0u64;
for l in 0..n_layers {
let Ok(layer_dir) = self.layer_dir(i, l) else {
continue;
};
for sub in ["presence", "counts"] {
let Ok(entries) = fs::read_dir(layer_dir.join(sub)) else {
continue;
};
total += entries
.flatten()
.filter_map(|e| e.metadata().ok())
.filter(|m| m.is_file())
.map(|m| m.len())
.sum::<u64>();
}
}
total
}
// /// Write a `layer_meta.json` in any layer directory that is missing one. // /// Write a `layer_meta.json` in any layer directory that is missing one.
// /// // ///
// /// Old indexes were built before this file was required. The number of // /// Old indexes were built before this file was required. The number of
@@ -364,3 +437,34 @@ impl KmerIndex {
// Ok(()) // Ok(())
// } // }
} }
/// One `info!` summary of `pack_matrices` per-partition timings: the sum of
/// partition times (divide by the stage wall time to get the mean number of
/// concurrently busy workers), the median and maximum, and the slowest
/// partitions with their pre-pack matrix size. `timings` holds
/// `(partition, matrix_bytes, elapsed)`.
fn report_pack_timings(timings: &[(usize, u64, std::time::Duration)]) {
if timings.is_empty() {
return;
}
let mut by_time: Vec<&(usize, u64, std::time::Duration)> = timings.iter().collect();
by_time.sort_by_key(|t| std::cmp::Reverse(t.2));
let sum: f64 = timings.iter().map(|t| t.2.as_secs_f64()).sum();
let bytes: u64 = timings.iter().map(|t| t.1).sum();
let median = by_time[by_time.len() / 2].2.as_secs_f64();
tracing::info!(
"pack: {} partition(s), sum of partition times {:.0}s, median {:.1}s, max {:.1}s, {:.1} GB of matrix data",
timings.len(),
sum,
median,
by_time[0].2.as_secs_f64(),
bytes as f64 / 1e9,
);
for (i, b, d) in by_time.iter().take(5).map(|t| (t.0, t.1, t.2)) {
tracing::info!(
"pack: slowest partition {i}: {:.1}s, {:.2} GB of matrix data",
d.as_secs_f64(),
b as f64 / 1e9,
);
}
}
+4 -2
View File
@@ -280,8 +280,10 @@ impl IndexMeta {
// ── private ────────────────────────────────────────────────────────────── // ── private ──────────────────────────────────────────────────────────────
fn read_full(root: &Path) -> io::Result<IndexMetadata> { fn read_full(root: &Path) -> io::Result<IndexMetadata> {
let file = fs::File::open(root.join(META_FILENAME))?; // One `read`, then parse from memory: `from_reader` on a bare `File`
serde_json::from_reader(file).map_err(io::Error::other) // issues one syscall per byte (≈ 0.35 s for a 91-genome `index.meta`).
let bytes = fs::read(root.join(META_FILENAME))?;
serde_json::from_slice(&bytes).map_err(io::Error::other)
} }
fn write_full(&self, on_disk: &IndexMetadata) -> io::Result<()> { fn write_full(&self, on_disk: &IndexMetadata) -> io::Result<()> {
+11 -9
View File
@@ -199,6 +199,17 @@ impl KmerLayer {
} }
} }
/// Batch form of [`find_slot`](Self::find_slot): `out[i]` is
/// `find_slot(kmers[i])`, MPHF lookups prefetched together. `out` is
/// cleared and reused.
pub fn find_slot_batch(&self, kmers: &[CanonicalKmer], out: &mut Vec<Option<usize>>) {
match self {
KmerLayer::Count { layer, .. } => layer.find_slot_batch(kmers, out),
KmerLayer::Presence { layer, .. } => layer.find_slot_batch(kmers, out),
KmerLayer::Empty { .. } => panic!("Layer::find_slot_batch() called on an Empty layer"),
}
}
/// Raw MPHF batch lookup: kmer → slot, no membership check — for /// Raw MPHF batch lookup: kmer → slot, no membership check — for
/// callers that already know every kmer is a member of *this* layer, so /// callers that already know every kmer is a member of *this* layer, so
/// the evidence check `find_slot`/`find` would perform is redundant. /// the evidence check `find_slot`/`find` would perform is redundant.
@@ -286,15 +297,6 @@ impl KmerLayer {
} }
} }
/// Raw MPHF lookup: kmer → slot, no membership check.
pub fn hash(&self, kmer: CanonicalKmer) -> usize {
match self {
KmerLayer::Count { layer, .. } => layer.hash(kmer),
KmerLayer::Presence { layer, .. } => layer.hash(kmer),
KmerLayer::Empty { .. } => panic!("Layer::hash() called on an Empty layer"),
}
}
/// Iterate over all canonical kmers in the layer, in deterministic order. /// Iterate over all canonical kmers in the layer, in deterministic order.
pub fn iter_kmers(&self) -> crate::layer::mphf_layer::KmerIter { pub fn iter_kmers(&self) -> crate::layer::mphf_layer::KmerIter {
match self { match self {
+1 -1
View File
@@ -27,7 +27,7 @@ pub enum IndexMode {
/// [`detect`](Self::detect)). /// [`detect`](Self::detect)).
Approx { b: u8, z: u8 }, Approx { b: u8, z: u8 },
/// Hybrid: both `fingerprint.bin` and `evidence.bin` + `unitigs.bin.idx`. /// Hybrid: both `fingerprint.bin` and `evidence.bin` + `unitigs.bin.idx`.
/// `find()` uses the fingerprint (O(1), approx); `find_strict()` uses exact evidence. /// `find()` uses the fingerprint (O(1), approx); the exact evidence is kept alongside.
Hybrid { b: u8, z: u8 }, Hybrid { b: u8, z: u8 },
} }
+4 -1
View File
@@ -3,11 +3,14 @@ pub mod evidence;
pub mod fingerprint; pub mod fingerprint;
pub mod meta; pub mod meta;
pub(crate) mod mphf_layer; pub(crate) mod mphf_layer;
pub(crate) mod slotted;
pub mod typed_layer; pub mod typed_layer;
pub(crate) mod utils; pub(crate) mod utils;
pub use crate::index::error::{OKIError, OKIResult}; pub use crate::index::error::{OKIError, OKIResult};
pub use content_layer::KmerLayer; pub use content_layer::KmerLayer;
pub use meta::IndexMode; pub use meta::IndexMode;
pub use mphf_layer::{EvidenceKind, KmerBatchIter, KmerIter, MphfLayer, MphfOnly, UnitigIter}; pub use mphf_layer::{
EvidenceKind, KmerBatchIter, KmerIter, Mphf, MphfLayer, MphfOnly, UnitigIter, mphf_params,
};
pub use typed_layer::{HasLayerContent, HasStorageKind, Hit, LayerContent, LayerData, TypedLayer}; pub use typed_layer::{HasLayerContent, HasStorageKind, Hit, LayerContent, LayerData, TypedLayer};
+86 -101
View File
@@ -8,24 +8,29 @@ use epserde::prelude::*;
use obicompactvec::LayerMeta; use obicompactvec::LayerMeta;
use obikseq::CanonicalKmer; use obikseq::CanonicalKmer;
use obiskio::{CanonicalKmerIter, UnitigFileReader, UnitigFileWriter, build_unitig_idx}; use obiskio::{CanonicalKmerIter, UnitigFileReader, UnitigFileWriter, build_unitig_idx};
use ptr_hash::{PtrHash, PtrHashParams, bucket_fn::CubicEps, hash::Xx64}; use ptr_hash::{PtrHash, PtrHashParams, bucket_fn::CubicEps, hash::Xxh3Int};
use crate::index::error::{OKIError, OKIResult}; use crate::index::error::{OKIError, OKIResult};
use crate::layer::evidence::{Evidence, EvidenceWriter}; use crate::layer::evidence::{Evidence, EvidenceWriter};
use crate::layer::fingerprint::{FingerprintVec, FingerprintVecWriter}; use crate::layer::fingerprint::{FingerprintVec, FingerprintVecWriter};
use crate::layer::meta::IndexMode; use crate::layer::meta::IndexMode;
use crate::layer::slotted::SlottedKmers;
pub(crate) const MPHF_FILE: &str = "mphf.bin"; pub(crate) const MPHF_FILE: &str = "mphf.bin";
pub(crate) const UNITIGS_FILE: &str = "unitigs.bin"; pub(crate) const UNITIGS_FILE: &str = "unitigs.bin";
pub(crate) const EVIDENCE_FILE: &str = "evidence.bin"; pub(crate) const EVIDENCE_FILE: &str = "evidence.bin";
pub(crate) const FINGERPRINT_FILE: &str = "fingerprint.bin"; pub(crate) const FINGERPRINT_FILE: &str = "fingerprint.bin";
/// Owned MPHF — used only at build time (construction + store). /// The layer MPHF: `Xxh3Int` hasher, `CubicEps` buckets, multi-part (`SINGLE_PART = false`),
pub(crate) type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xx64, Vec<u8>>; /// minimal (`REMAP = true`). Built owned (construction + store); queried through
/// `MemCase<Mphf>::uncase()`, the ε-copy view into the memory-mapped file.
pub type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xxh3Int, Vec<u8>, false, true>;
/// Zero-copy MPHF for querying — ε-deserialized view into a memory-mapped file. /// Construction parameters of [`Mphf`] — the one place they are chosen
/// `MemCase` owns the mmap backing; `'static` is sound because MemCase pins the memory. /// (`lambda = 3.5`, `alpha = 0.99`, `CubicEps`, ~2.4 bits/key).
type MphfEps = PtrHash<u64, CubicEps, CachelineEfVec<&'static [CachelineEf]>, Xx64, &'static [u8]>; pub fn mphf_params() -> PtrHashParams<CubicEps> {
PtrHashParams::<CubicEps>::default_balanced()
}
// ── LayerEvidence ───────────────────────────────────────────────────────────── // ── LayerEvidence ─────────────────────────────────────────────────────────────
@@ -46,17 +51,17 @@ enum LayerEvidence {
Hybrid { Hybrid {
evidence_path: PathBuf, evidence_path: PathBuf,
fingerprint_path: PathBuf, fingerprint_path: PathBuf,
unitig_path: PathBuf,
}, },
} }
/// Resources actually opened (mmap). `Exact`/`Hybrid` carry their own /// Resources actually opened (mmap). `Exact` carries its own `unitigs` —
/// `unitigs` — needed to complete evidence verification, not shared with /// needed to complete evidence verification, not shared with `MphfLayer`'s
/// `MphfLayer`'s own separate `unitigs` (used by `iter_kmers`/`Approx`'s /// own separate `unitigs` (used by `iter_kmers`, mode-independent). Yes, this
/// `find_strict` fallback, mode-independent). Yes, this means a layer's /// means an Exact layer's `unitigs.bin` gets mmapped twice (the `.idx`
/// `unitigs.bin` gets mmapped twice for Exact/Hybrid — deliberately not /// offsets are mapped too, so both readers share the same pages) —
/// optimised away here: avoiding that redundant open is the future cache /// deliberately not optimised away here. `Hybrid` keeps its `evidence.bin`
/// crate's job, not this base type's. /// open (it gives the slot count) but nothing verifies against it: lookups
/// use the fingerprint, as in `Approx`.
enum OpenLayerEvidence { enum OpenLayerEvidence {
Exact { Exact {
evidence: Evidence, evidence: Evidence,
@@ -68,7 +73,6 @@ enum OpenLayerEvidence {
Hybrid { Hybrid {
evidence: Evidence, evidence: Evidence,
fingerprint: FingerprintVec, fingerprint: FingerprintVec,
unitigs: Arc<UnitigFileReader>,
}, },
} }
@@ -88,7 +92,6 @@ impl LayerEvidence {
EvidenceKind::Hybrid => LayerEvidence::Hybrid { EvidenceKind::Hybrid => LayerEvidence::Hybrid {
evidence_path: layer_dir.join(EVIDENCE_FILE), evidence_path: layer_dir.join(EVIDENCE_FILE),
fingerprint_path: layer_dir.join(FINGERPRINT_FILE), fingerprint_path: layer_dir.join(FINGERPRINT_FILE),
unitig_path: layer_dir.join(UNITIGS_FILE),
}, },
}) })
} }
@@ -103,10 +106,9 @@ impl LayerEvidence {
LayerEvidence::Approx { fingerprint_path } => OpenLayerEvidence::Approx { LayerEvidence::Approx { fingerprint_path } => OpenLayerEvidence::Approx {
fingerprint: FingerprintVec::open(&fingerprint_path)?, fingerprint: FingerprintVec::open(&fingerprint_path)?,
}, },
LayerEvidence::Hybrid { evidence_path, fingerprint_path, unitig_path } => OpenLayerEvidence::Hybrid { LayerEvidence::Hybrid { evidence_path, fingerprint_path } => OpenLayerEvidence::Hybrid {
evidence: Evidence::open(&evidence_path)?, evidence: Evidence::open(&evidence_path)?,
fingerprint: FingerprintVec::open(&fingerprint_path)?, fingerprint: FingerprintVec::open(&fingerprint_path)?,
unitigs: Arc::new(UnitigFileReader::open(&unitig_path)?),
}, },
}) })
} }
@@ -146,19 +148,15 @@ impl EvidenceKind {
/// Autonomous kmer → slot mapping for one layer. /// Autonomous kmer → slot mapping for one layer.
/// ///
/// Two query methods: /// Query: [`find`](Self::find) — O(1), uses the fingerprint (Approx/Hybrid) or
/// - [`find`](Self::find) — O(1), uses fingerprint (Approx/Hybrid) or exact evidence (Exact). /// exact evidence (Exact).
/// - [`find_strict`](Self::find_strict) — always exact; O(1) on Exact/Hybrid layers,
/// O(n) sequential scan on Approx layers.
pub struct MphfLayer { pub struct MphfLayer {
/// Mode-independent copy — used by `iter_kmers` (all modes) and /// Mode-independent copy — used by `iter_kmers` (all modes). `Exact`
/// `find_strict`'s `Approx` fallback. `Exact`/`Hybrid` additionally /// additionally carries its own inside `OpenLayerEvidence` (see its doc
/// carry their own inside `OpenLayerEvidence` (see its doc comment for /// comment for why that's a second mmap of the same file, not a bug).
/// why that's a second, deliberately unshared mmap of the same file,
/// not a bug).
unitigs: Arc<UnitigFileReader>, unitigs: Arc<UnitigFileReader>,
mphf: MemCase<MphfEps>, mphf: MphfOnly,
ev: OpenLayerEvidence, ev: OpenLayerEvidence,
n: usize, n: usize,
} }
@@ -171,8 +169,7 @@ impl MphfLayer {
/// not stored: `KmerLayer`, one level up, is what owns a layer's /// not stored: `KmerLayer`, one level up, is what owns a layer's
/// directory as a persistent field, not this type. /// directory as a persistent field, not this type.
pub fn open(dir: &Path) -> OKIResult<Self> { pub fn open(dir: &Path) -> OKIResult<Self> {
let mphf: MemCase<MphfEps> = Mphf::mmap(&dir.join(MPHF_FILE), Flags::empty()) let mphf = MphfOnly::open(dir)?;
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
let unitigs = Arc::new(UnitigFileReader::open(&dir.join(UNITIGS_FILE))?); let unitigs = Arc::new(UnitigFileReader::open(&dir.join(UNITIGS_FILE))?);
let ev = LayerEvidence::at(dir)?.open()?; let ev = LayerEvidence::at(dir)?.open()?;
let n = match &ev { let n = match &ev {
@@ -188,10 +185,17 @@ impl MphfLayer {
/// O(1) lookup — dispatches automatically: /// O(1) lookup — dispatches automatically:
/// - Exact: evidence + `verify_canonical_kmer`, zero false positives. /// - Exact: evidence + `verify_canonical_kmer`, zero false positives.
/// - Approx: fingerprint check, false-positive rate ≈ 1/2^b. /// - Approx: fingerprint check, false-positive rate ≈ 1/2^b.
/// - Hybrid: fingerprint check (fast path), zero false positives via `find_strict`. /// - Hybrid: fingerprint check, same as Approx (the exact evidence is kept on disk, not consulted).
#[inline] #[inline]
pub fn find(&self, kmer: CanonicalKmer) -> Option<usize> { pub fn find(&self, kmer: CanonicalKmer) -> Option<usize> {
let slot = self.mphf.index(&kmer.raw()); self.confirm(self.mphf.index(kmer), kmer)
}
/// Membership check of `kmer` against the slot the MPHF gave it — the
/// mode-dependent half of [`find`](Self::find), shared with
/// [`find_batch`](Self::find_batch).
#[inline]
fn confirm(&self, slot: usize, kmer: CanonicalKmer) -> Option<usize> {
if slot >= self.n { if slot >= self.n {
return None; return None;
} }
@@ -215,41 +219,20 @@ impl MphfLayer {
} }
} }
/// Always-exact lookup — zero false positives regardless of mode. /// Batch form of [`find`](Self::find): `out[i]` is `find(kmers[i])`.
/// ///
/// - Exact/Hybrid: O(1) via evidence + `verify_canonical_kmer`. /// The MPHF lookups of the whole slice go through `ptr_hash`'s prefetching
/// - Approx: O(n) sequential scan of `unitigs.bin` to confirm the kmer /// stream (see [`MphfOnly::for_each_slot`]), so their random pilot reads
/// that owns the slot, then exact comparison. /// overlap instead of being paid one after the other; the membership
pub fn find_strict(&self, kmer: CanonicalKmer) -> Option<usize> { /// check runs on each slot as it comes out. `out` is cleared and reused.
let slot = self.mphf.index(&kmer.raw()); pub fn find_batch(&self, kmers: &[CanonicalKmer], out: &mut Vec<Option<usize>>) {
if slot >= self.n { out.clear();
return None; out.reserve(kmers.len());
} let mut i = 0;
match &self.ev { self.mphf.for_each_slot(kmers, |slot| {
OpenLayerEvidence::Exact { evidence, unitigs } out.push(self.confirm(slot, kmers[i]));
| OpenLayerEvidence::Hybrid { evidence, unitigs, .. } => { i += 1;
let (chunk_id, rank) = evidence.decode(slot); });
if unitigs.verify_canonical_kmer(chunk_id as usize, rank as usize, kmer) {
Some(slot)
} else {
None
}
}
OpenLayerEvidence::Approx { .. } => {
// `self.unitigs` was opened via `UnitigFileReader::open`, which
// falls back to sequential mode automatically when no `.idx`
// sidecar exists — true for every Approx layer (`build_unitig_idx`
// is only ever called for Exact/Hybrid). Reusing it here avoids
// the second file-open + mmap + kmer-count scan a fresh
// `open_sequential` call would repeat on every call.
for (stored, _, _) in self.unitigs.iter_indexed_canonical_kmers() {
if self.mphf.index(&stored.raw()) == slot {
return if stored == kmer { Some(slot) } else { None };
}
}
None
}
}
} }
/// Number of slots in the MPHF layer — equal to the number of stored k-mers. /// Number of slots in the MPHF layer — equal to the number of stored k-mers.
@@ -267,23 +250,13 @@ impl MphfLayer {
} }
} }
/// Raw MPHF lookup: kmer → slot.
///
/// Returns the slot assigned by the MPHF without performing any membership
/// check against the layer's evidence. The returned slot is only meaningful
/// when the kmer is actually present in the layer; callers that need an
/// existence test should use [`find`](Self::find) instead.
pub fn hash(&self, kmer: CanonicalKmer) -> usize {
self.mphf.index(&kmer.raw())
}
/// Batch raw MPHF lookup: kmers → slots. /// Batch raw MPHF lookup: kmers → slots.
/// ///
/// Returns a [`Vec`] of slots, one per input kmer, in the same order as the /// Returns a [`Vec`] of slots, one per input kmer, in the same order as the
/// input slice. Like [`index`](Self::index), no membership check is /// input slice. Like [`hash`](Self::hash), no membership check is
/// performed. /// performed. Prefetching — see [`MphfOnly::index_batch`].
pub fn hash_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> { pub fn hash_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> {
kmers.iter().map(|k| self.mphf.index(&k.raw())).collect() self.mphf.index_batch(kmers)
} }
/// Iterate over all canonical kmers in the layer, in deterministic order. /// Iterate over all canonical kmers in the layer, in deterministic order.
@@ -426,11 +399,11 @@ impl Iterator for KmerBatchIter {
/// ///
/// Use this when the caller guarantees that all queried kmers are in the MPHF /// Use this when the caller guarantees that all queried kmers are in the MPHF
/// domain (e.g. when iterating the source's own unitigs during merge). /// domain (e.g. when iterating the source's own unitigs during merge).
pub struct MphfOnly(MemCase<MphfEps>); pub struct MphfOnly(MemCase<Mphf>);
impl MphfOnly { impl MphfOnly {
pub fn open(dir: &Path) -> OKIResult<Self> { pub fn open(dir: &Path) -> OKIResult<Self> {
let mphf: MemCase<MphfEps> = Mphf::mmap(&dir.join(MPHF_FILE), Flags::empty()) let mphf: MemCase<Mphf> = unsafe { Mphf::mmap(&dir.join(MPHF_FILE), Flags::empty()) }
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?; .map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
Ok(Self(mphf)) Ok(Self(mphf))
} }
@@ -438,13 +411,30 @@ impl MphfOnly {
/// Return the slot for `kmer`. Only valid when `kmer` is in the MPHF domain. /// Return the slot for `kmer`. Only valid when `kmer` is in the MPHF domain.
#[inline] #[inline]
pub fn index(&self, kmer: CanonicalKmer) -> usize { pub fn index(&self, kmer: CanonicalKmer) -> usize {
self.0.index(&kmer.raw()) self.0.uncase().index(&kmer.raw())
} }
/// Batch form of [`index`](Self::index) — one MPHF lookup pass over the /// Batch form of [`index`](Self::index): one prefetching `ptr_hash`
/// whole slice instead of one call per kmer. /// stream over the whole slice, so the random pilot reads of different
/// kmers overlap instead of being paid one after the other.
pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> { pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> {
kmers.iter().map(|&k| self.index(k)).collect() let mut slots = Vec::with_capacity(kmers.len());
self.for_each_slot(kmers, |slot| slots.push(slot));
slots
}
/// Calls `f(slot)` for each kmer of `kmers`, in order.
///
/// Callback-shaped on purpose: `ptr_hash`'s `index_stream` only
/// implements `fold` (its `next()` is `unimplemented!`), so it cannot be
/// returned as an ordinary iterator without a panic trap for callers.
/// Look-ahead of 32 keys, as recommended by `ptr_hash`.
#[inline]
pub fn for_each_slot(&self, kmers: &[CanonicalKmer], f: impl FnMut(usize)) {
self.0
.uncase()
.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
.for_each(f);
} }
} }
@@ -468,14 +458,13 @@ impl MphfLayer {
return Ok(0); return Ok(0);
} }
let mphf: Mphf = Mphf::load_full(&dir.join(MPHF_FILE)) let mphf: Mphf = unsafe { Mphf::load_full(&dir.join(MPHF_FILE)) }
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?; .map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
let mut ev = EvidenceWriter::new(n); let mut ev = EvidenceWriter::new(n);
let mut seen = vec![0u8; (n + 7) / 8]; let mut seen = vec![0u8; (n + 7) / 8];
for (kmer, chunk_id, rank) in unitigs.iter_indexed_canonical_kmers() { for (slot, _, chunk_id, rank) in SlottedKmers::new(&mphf, unitigs.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -512,13 +501,12 @@ impl MphfLayer {
return Ok(0); return Ok(0);
} }
let mphf: Mphf = Mphf::load_full(&dir.join(MPHF_FILE)) let mphf: Mphf = unsafe { Mphf::load_full(&dir.join(MPHF_FILE)) }
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?; .map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
let mut fw = FingerprintVecWriter::new(n, b); let mut fw = FingerprintVecWriter::new(n, b);
for (kmer, _, _) in unitigs.iter_indexed_canonical_kmers() { for (slot, kmer, _, _) in SlottedKmers::new(&mphf, unitigs.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -552,9 +540,9 @@ impl MphfLayer {
// ── Empty layer ─────────────────────────────────────────────────────── // ── Empty layer ───────────────────────────────────────────────────────
if n == 0 { if n == 0 {
let mphf: Mphf = Mphf::try_new(&[] as &[u64], PtrHashParams::<CubicEps>::default()) let mphf: Mphf = Mphf::try_new(&[] as &[u64], mphf_params())
.ok_or_else(|| OKIError::InvalidData { context: "mphf", detail: "construction failed".into() })?; .ok_or_else(|| OKIError::InvalidData { context: "mphf", detail: "construction failed".into() })?;
mphf.store(&dir.join(MPHF_FILE)) unsafe { mphf.store(&dir.join(MPHF_FILE)) }
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?; .map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
match mode { match mode {
IndexMode::Exact | IndexMode::Hybrid { .. } => { IndexMode::Exact | IndexMode::Hybrid { .. } => {
@@ -576,9 +564,9 @@ impl MphfLayer {
let mphf: Mphf = Mphf::new_from_par_iter( let mphf: Mphf = Mphf::new_from_par_iter(
n, n,
keys.map(|k| k.raw()).par_bridge(), keys.map(|k| k.raw()).par_bridge(),
PtrHashParams::<CubicEps>::default(), mphf_params(),
); );
mphf.store(&dir.join(MPHF_FILE)) unsafe { mphf.store(&dir.join(MPHF_FILE)) }
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?; .map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
// ── Pass 2: fill evidence files + callback ──────────────────────────── // ── Pass 2: fill evidence files + callback ────────────────────────────
@@ -588,8 +576,7 @@ impl MphfLayer {
match mode { match mode {
IndexMode::Exact => { IndexMode::Exact => {
let mut ev = EvidenceWriter::new(n); let mut ev = EvidenceWriter::new(n);
for (kmer, chunk_id, rank) in unitigs2.iter_indexed_canonical_kmers() { for (slot, kmer, chunk_id, rank) in SlottedKmers::new(&mphf, unitigs2.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -608,8 +595,7 @@ impl MphfLayer {
IndexMode::Approx { b, .. } => { IndexMode::Approx { b, .. } => {
let mut fw = FingerprintVecWriter::new(n, *b); let mut fw = FingerprintVecWriter::new(n, *b);
for (kmer, _, _) in unitigs2.iter_indexed_canonical_kmers() { for (slot, kmer, _, _) in SlottedKmers::new(&mphf, unitigs2.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -628,8 +614,7 @@ impl MphfLayer {
IndexMode::Hybrid { b, .. } => { IndexMode::Hybrid { b, .. } => {
let mut ev = EvidenceWriter::new(n); let mut ev = EvidenceWriter::new(n);
let mut fw = FingerprintVecWriter::new(n, *b); let mut fw = FingerprintVecWriter::new(n, *b);
for (kmer, chunk_id, rank) in unitigs2.iter_indexed_canonical_kmers() { for (slot, kmer, chunk_id, rank) in SlottedKmers::new(&mphf, unitigs2.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
+58
View File
@@ -0,0 +1,58 @@
//! A layer's kmers streamed together with their MPHF slot — the shape every
//! build pass needs (`slot` indexes evidence, fingerprint and bitmap writes).
use obikseq::CanonicalKmer;
use crate::layer::mphf_layer::Mphf;
/// Kmers per `ptr_hash` stream: large enough to amortise its ring-buffer
/// start-up, small enough to stay in L1/L2 and never hold the layer.
const CHUNK: usize = 4096;
/// `(slot, kmer, chunk_id, rank)` for every `(kmer, chunk_id, rank)` of the
/// wrapped iterator, in the same order. The MPHF lookups are done a chunk at
/// a time through `ptr_hash`'s prefetching stream, so their random pilot
/// reads overlap; memory stays bounded by one chunk.
pub(crate) struct SlottedKmers<'a, I> {
mphf: &'a Mphf,
kmers: I,
chunk: Vec<(CanonicalKmer, usize, usize)>,
slots: Vec<usize>,
pos: usize,
}
impl<'a, I: Iterator<Item = (CanonicalKmer, usize, usize)>> SlottedKmers<'a, I> {
pub(crate) fn new(mphf: &'a Mphf, kmers: I) -> Self {
Self {
mphf,
kmers,
chunk: Vec::with_capacity(CHUNK),
slots: Vec::with_capacity(CHUNK),
pos: 0,
}
}
}
impl<I: Iterator<Item = (CanonicalKmer, usize, usize)>> Iterator for SlottedKmers<'_, I> {
type Item = (usize, CanonicalKmer, usize, usize);
fn next(&mut self) -> Option<Self::Item> {
if self.pos == self.chunk.len() {
self.chunk.clear();
self.chunk.extend(self.kmers.by_ref().take(CHUNK));
if self.chunk.is_empty() {
return None;
}
self.slots.clear();
let slots = &mut self.slots;
self.mphf
.index_stream::<32, _>(self.chunk.iter().map(|(k, _, _)| k.raw()))
.for_each(|slot| slots.push(slot));
self.pos = 0;
}
let (kmer, chunk_id, rank) = self.chunk[self.pos];
let slot = self.slots[self.pos];
self.pos += 1;
Some((slot, kmer, chunk_id, rank))
}
}
+7 -5
View File
@@ -189,13 +189,15 @@ impl<D: LayerData> TypedLayer<D> {
self.kmerstore.find(kmer) self.kmerstore.find(kmer)
} }
pub fn n(&self) -> usize { /// Batch form of [`find_slot`](Self::find_slot): `out[i]` is
self.kmerstore.n() /// `find_slot(kmers[i])`, with the MPHF lookups prefetched together
/// (see `MphfLayer::find_batch`). `out` is cleared and reused.
pub fn find_slot_batch(&self, kmers: &[CanonicalKmer], out: &mut Vec<Option<usize>>) {
self.kmerstore.find_batch(kmers, out)
} }
/// Raw MPHF lookup: kmer → slot, no membership check. pub fn n(&self) -> usize {
pub fn hash(&self, kmer: CanonicalKmer) -> usize { self.kmerstore.n()
self.kmerstore.hash(kmer)
} }
/// Batch raw MPHF lookup: kmers → slots, no membership check. /// Batch raw MPHF lookup: kmers → slots, no membership check.
+1 -3
View File
@@ -14,9 +14,7 @@ obicompactvec = { path = "../obicompactvec" }
obipipeline = { path = "../obipipeline" } obipipeline = { path = "../obipipeline" }
obiread = { path = "../obiread" } obiread = { path = "../obiread" }
obiskbuilder = { path = "../obiskbuilder" } obiskbuilder = { path = "../obiskbuilder" }
cacheline-ef = "1.1" epserde = "0.13"
epserde = "0.8"
ptr_hash = "1.1"
niffler = "3.0.0" niffler = "3.0.0"
memmap2 = "0.9.10" memmap2 = "0.9.10"
rayon = "1" rayon = "1"
+165
View File
@@ -0,0 +1,165 @@
//! Throwaway benchmark of the de Bruijn graph on a real kmer set:
//! `graph_bench <unitigs.bin> <k> [repeat]` builds `GraphDeBruijn` from the
//! canonical kmers of an existing layer's unitigs (same call sequence as
//! `write_graph_as_unitigs`) and times each phase.
use std::path::PathBuf;
use std::time::Instant;
use obidebruinj::GraphDeBruijn;
use obikseq::Sequence;
use obiskio::CanonicalKmerIter;
use rayon::prelude::*;
use std::collections::HashSet;
fn main() {
let args: Vec<String> = std::env::args().collect();
let path = PathBuf::from(&args[1]);
let k: usize = args[2].parse().unwrap();
let repeat: usize = args.get(3).map_or(1, |s| s.parse().unwrap());
obikseq::set_k(k);
obikseq::set_m(11);
let kmers: Vec<_> = if let Ok(n) = std::env::var("SYNTH") {
// synthetic reads: random genome, reads on both strands, 1 % substitutions
let n: usize = n.parse().unwrap();
let mut st = 0x9E3779B97F4A7C15u64;
let mut rnd = move || {
st ^= st << 13;
st ^= st >> 7;
st ^= st << 17;
st
};
let genome: Vec<u8> = (0..n).map(|_| b"ACGT"[(rnd() % 4) as usize]).collect();
let mut set = std::collections::BTreeSet::new();
for _ in 0..(n / 50) {
let start = (rnd() as usize) % (n - 100);
let mut read: Vec<u8> = genome[start..start + 100].to_vec();
for b in read.iter_mut() {
if rnd() % 100 == 0 {
*b = b"ACGT"[(rnd() % 4) as usize];
}
}
for w in read.windows(k) {
set.insert(obikseq::Kmer::from_ascii(w).unwrap().canonical());
}
}
set.into_iter().collect()
} else {
CanonicalKmerIter::new(&path).unwrap().collect()
};
eprintln!("{} kmers", kmers.len());
// Independent count of maximal unitigs: nodes − mutually-unique links
// (cycles ignored), computed with a plain HashSet — no use of GraphDeBruijn.
if std::env::var("IDEAL").is_ok() {
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
(0..4u8)
.map(|x| a.push_right(x))
.filter(|b| set.contains(&b.canonical().raw()))
.collect()
};
let links: usize = kmers
.par_iter()
.map(|c| {
let mut n = 0usize;
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
let ob = outs(o);
if ob.len() == 1 {
let b = ob[0];
// in-neighbours of b = revcomp of out-neighbours of revcomp(b)
if outs(b.revcomp()).len() == 1 {
// count each undirected link once: only from the orientation
// whose canonical raw is the smaller end (ties: both seen once per orientation)
let (ca, cb) = (o.canonical().raw(), b.canonical().raw());
if ca < cb || (ca == cb && o.raw() <= b.raw()) {
n += 1;
} else if ca > cb {
// seen from the other end too: skip
}
}
}
}
n
})
.sum();
eprintln!("IDEAL: nodes={} links={} -> maximal unitigs ≈ {}", kmers.len(), links, kmers.len() - links);
}
if std::env::var("IDEALDUMP").is_ok() {
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
(0..4u8).map(|x| a.push_right(x)).filter(|b| set.contains(&b.canonical().raw())).collect()
};
// oriented kmer a has a mutual predecessor p iff ins(a)=={p} and outs(p)=={a}
let pred = |a: obikseq::Kmer| -> Option<obikseq::Kmer> {
let ins: Vec<obikseq::Kmer> = outs(a.revcomp()).into_iter().map(|b| b.revcomp()).collect();
if ins.len() == 1 && outs(ins[0]).len() == 1 { Some(ins[0]) } else { None }
};
let mut seen_chains = std::collections::BTreeSet::new();
for c in &kmers {
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
if pred(o).is_some() { continue; }
let mut text: Vec<u8> = (0..k).map(|i| b"ACGT"[o.nucleotide(i) as usize]).collect();
let mut cur = o;
loop {
let ob = outs(cur);
if ob.len() != 1 { break; }
let b = ob[0];
if pred(b).is_none() { break; }
text.push(b"ACGT"[b.nucleotide(k - 1) as usize]);
cur = b;
}
let s = String::from_utf8(text).unwrap();
let rc: String = s.bytes().rev().map(|c| match c { b'A' => 'T', b'C' => 'G', b'G' => 'C', _ => 'A' }).collect();
seen_chains.insert(if s <= rc { s } else { rc });
}
}
for u in &seen_chains { println!("IDEALUNITIG {} {}", u.len(), u); }
}
for r in 0..repeat {
let t = Instant::now();
let mut g = GraphDeBruijn::new();
for &kmer in &kmers {
g.push(kmer);
}
let t_push = t.elapsed();
let t = Instant::now();
g.compute_degrees_and_mark_starts();
let t_deg = t.elapsed();
let t = Instant::now();
let n_unitigs = std::sync::atomic::AtomicUsize::new(0);
let n_nuc = std::sync::atomic::AtomicUsize::new(0);
let dump = std::env::var("DUMP").is_ok();
let seqs = std::sync::Mutex::new(Vec::<String>::new());
g.for_each_unitig(|it| {
let mut n = 0usize;
let mut text = String::new();
for nuc in it {
n += 1;
if dump {
text.push(b"ACGT"[nuc as usize] as char);
}
}
if dump {
seqs.lock().unwrap().push(text);
}
n_unitigs.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
n_nuc.fetch_add(n, std::sync::atomic::Ordering::Relaxed);
});
let t_walk = t.elapsed();
if dump {
let mut v = seqs.lock().unwrap().clone();
v.sort();
for u in v { println!("UNITIG {} {}", u.len(), u); }
}
println!(
"run {r}: nodes={} push={:.2?} degrees={:.2?} unitigs={:.2?} -> {} unitigs, {} nucleotides",
g.len(),
t_push,
t_deg,
t_walk,
n_unitigs.load(std::sync::atomic::Ordering::Relaxed),
n_nuc.load(std::sync::atomic::Ordering::Relaxed)
);
}
}
@@ -3,18 +3,15 @@ use std::fs;
use std::io; use std::io;
use std::path::Path; use std::path::Path;
use cacheline_ef::{CachelineEf, CachelineEfVec};
use epserde::ser::Serialize as EpSerialize; use epserde::ser::Serialize as EpSerialize;
use memmap2::Mmap; use memmap2::Mmap;
use obicompactvec::PersistentCompactIntVecBuilder; use obicompactvec::PersistentCompactIntVecBuilder;
use obikindex::layer::{Mphf, mphf_params};
use obiskio::{SKFileReader, SKResult}; use obiskio::{SKFileReader, SKResult};
use ptr_hash::{PtrHash, PtrHashParams, bucket_fn::CubicEps, hash::Xx64};
use tracing::debug; use tracing::debug;
use super::kmer_sort::sort_unique_kmers; use super::kmer_sort::sort_unique_kmers;
pub(super) type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xx64, Vec<u8>>;
fn build_mphf(unique_path: &Path, f0: usize) -> io::Result<Mphf> { fn build_mphf(unique_path: &Path, f0: usize) -> io::Result<Mphf> {
let file = fs::File::open(unique_path)?; let file = fs::File::open(unique_path)?;
let mmap = unsafe { Mmap::map(&file)? }; let mmap = unsafe { Mmap::map(&file)? };
@@ -24,7 +21,7 @@ fn build_mphf(unique_path: &Path, f0: usize) -> io::Result<Mphf> {
// Sequential constructor: the outer par_iter over partitions already saturates // Sequential constructor: the outer par_iter over partitions already saturates
// the Rayon pool. new_from_par_iter would get no additional threads and adds // the Rayon pool. new_from_par_iter would get no additional threads and adds
// coordination overhead. try_new accesses the same mmap'd pages at zero extra cost. // coordination overhead. try_new accesses the same mmap'd pages at zero extra cost.
Mphf::try_new(kmers, PtrHashParams::<CubicEps>::default()) Mphf::try_new(kmers, mphf_params())
.ok_or_else(|| io::Error::other("ptr_hash construction failed")) .ok_or_else(|| io::Error::other("ptr_hash construction failed"))
} }
@@ -73,7 +70,7 @@ pub(super) fn count_partition(dir: &Path, dedup_path: &Path, chunk_kmers: usize)
) )
.map_err(io::Error::other)?; .map_err(io::Error::other)?;
EpSerialize::store(&mphf, &dir.join("mphf1.bin")) unsafe { EpSerialize::store(&mphf, &dir.join("mphf1.bin")) }
.map_err(|e| io::Error::other(e.to_string()))?; .map_err(|e| io::Error::other(e.to_string()))?;
Ok(()) Ok(())
@@ -25,17 +25,13 @@ use std::io;
use std::path::Path; use std::path::Path;
use crate::graph_pipeline::{materialize_layer, write_graph_as_unitigs}; use crate::graph_pipeline::{materialize_layer, write_graph_as_unitigs};
use cacheline_ef::{CachelineEf, CachelineEfVec};
use epserde::prelude::*; use epserde::prelude::*;
use obicompactvec::{PersistentIntMatrix, PersistentCompactIntVec}; use obicompactvec::{PersistentIntMatrix, PersistentCompactIntVec};
use obidebruinj::GraphDeBruijn; use obidebruinj::GraphDeBruijn;
use obikindex::layer::IndexMode; use obikindex::layer::IndexMode;
use obikindex::layer::{KmerLayer, TypedLayer}; use obikindex::layer::{KmerLayer, Mphf, TypedLayer};
use obikindex::{KmerIndex, OKIError, OKIResult}; use obikindex::{KmerIndex, OKIError, OKIResult};
use obiskio::{SKError, SKFileMeta, SKFileReader}; use obiskio::{SKError, SKFileMeta, SKFileReader};
use ptr_hash::{PtrHash, bucket_fn::CubicEps, hash::Xx64};
type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xx64, Vec<u8>>;
pub(crate) trait PrivateBuilder { pub(crate) trait PrivateBuilder {
/// Mark scatter as complete (`IndexState::Scattered`). /// Mark scatter as complete (`IndexState::Scattered`).
@@ -139,6 +135,12 @@ impl PrivateBuilder for KmerIndex {
mode: &IndexMode, mode: &IndexMode,
block_bits: u8, block_bits: u8,
) -> Result<usize, SKError> { ) -> Result<usize, SKError> {
// A partition that received no k-mer has no layer 0 on disk at all
// (small inputs against many partitions): nothing to build, not an
// error.
if self.partition(i).map_err(|e| io::Error::other(e.to_string()))?.n_layers() == 0 {
return Ok(0);
}
let layer0 = self let layer0 = self
.layer0(i) .layer0(i)
.map_err(|e| io::Error::other(e.to_string()))?; .map_err(|e| io::Error::other(e.to_string()))?;
@@ -157,7 +159,7 @@ impl PrivateBuilder for KmerIndex {
let mphf1_opt: Option<Mphf> = if need_counts { let mphf1_opt: Option<Mphf> = if need_counts {
let p = layer0_dir.join("mphf1.bin"); let p = layer0_dir.join("mphf1.bin");
p.exists().then(|| Mphf::load_full(&p).ok()).flatten() p.exists().then(|| unsafe { Mphf::load_full(&p) }.ok()).flatten()
} else { } else {
None None
}; };
@@ -212,6 +214,10 @@ impl PrivateBuilder for KmerIndex {
} }
fn remove_build_artifacts(&self, i: usize) { fn remove_build_artifacts(&self, i: usize) {
// No layer 0 = empty partition: no build artifacts to remove.
if self.partition(i).map_or(false, |p| p.n_layers() == 0) {
return;
}
let layer0 = match self.layer0(i) { let layer0 = match self.layer0(i) {
Ok(layer) => layer, Ok(layer) => layer,
Err(e) => { Err(e) => {
+8 -2
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "obikmer" name = "obikmer"
version = "1.3.6" version = "1.4.5"
edition = "2024" edition = "2024"
[[bin]] [[bin]]
@@ -24,7 +24,6 @@ obikstats = { path = "../obikstats" }
obikquery = { path = "../obikquery" } obikquery = { path = "../obikquery" }
obikidxcache = { path = "../obikidxcache" } obikidxcache = { path = "../obikidxcache" }
obikphylo = { path = "../obikphylo" } obikphylo = { path = "../obikphylo" }
obiksession = { path = "../obiksession" }
obikrope = { path = "../obikrope" } obikrope = { path = "../obikrope" }
obifastwrite = { path = "../obifastwrite" } obifastwrite = { path = "../obifastwrite" }
obiskbuilder = { path = "../obiskbuilder" } obiskbuilder = { path = "../obiskbuilder" }
@@ -37,6 +36,13 @@ serde_yaml = "0.9"
tracing = "0.1.44" tracing = "0.1.44"
tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] } tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
# musl's built-in allocator fragments badly under the merge's multi-threaded
# allocation pattern (map count and RSS drift, see DevDocMD/implementation/
# merge_mmap_audit.md). Static musl builds use jemalloc instead; glibc and
# macOS builds keep the system allocator.
[target.'cfg(target_env = "musl")'.dependencies]
tikv-jemallocator = "0.5.4"
[features] [features]
default = ["numa"] default = ["numa"]
numa = ["obisys/numa"] numa = ["obisys/numa"]
+5 -1
View File
@@ -93,7 +93,11 @@ fn run_annotate(args: &AnnotateArgs) {
} }
}; };
let idx = open_index(&args.index); // Rewrites the index metadata in place: exclusive for the whole command.
let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}");
std::process::exit(1);
});
let mut genomes = idx.meta().genomes().unwrap_or_else(|e| { let mut genomes = idx.meta().genomes().unwrap_or_else(|e| {
eprintln!("error reading index metadata: {e}"); eprintln!("error reading index metadata: {e}");
+2 -8
View File
@@ -70,14 +70,8 @@ struct ModeArgs {
} }
pub fn run(args: ConvertArgs) { pub fn run(args: ConvertArgs) {
// Modifies the index in place; acquired before opening so a concurrent // Modifies the index in place: exclusive for the whole command.
// writer can't slip in between the open and the convert below. let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = KmerIndex::open(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
}); });
+3 -9
View File
@@ -19,21 +19,15 @@ pub struct PackArgs {
/// Pack presence and count matrices into the dense on-disk format instead /// Pack presence and count matrices into the dense on-disk format instead
/// of the default sparse, deduplicated one. Dense is faster for /// of the default sparse, deduplicated one. Dense is faster for
/// column-oriented access (`--metric` distance matrices); sparse is /// column-oriented access (`phylo --distance` matrices); sparse is
/// smaller and faster for single-row access on real, sparse data. /// smaller and faster for single-row access on real, sparse data.
#[arg(long, default_value_t = false)] #[arg(long, default_value_t = false)]
pub dense: bool, pub dense: bool,
} }
pub fn run(args: PackArgs) { pub fn run(args: PackArgs) {
// Modifies the index in place; acquired before opening so a concurrent // Modifies the index in place: exclusive for the whole command.
// writer can't slip in between the open and the pack below. let idx = Arc::new(KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = Arc::new(KmerIndex::open(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
})); }));
@@ -0,0 +1,51 @@
//! `--pseudo-alignment` / `--fasta` / `--vcf` file output.
use std::io::{BufWriter, Write};
use obikphylo::siblings::SnpAlignment;
use tracing::info;
use super::args::PhyloArgs;
fn create(path: &str) -> BufWriter<std::fs::File> {
BufWriter::new(std::fs::File::create(path).unwrap_or_else(|e| {
eprintln!("error creating {path}: {e}");
std::process::exit(1);
}))
}
/// Writes the drawn alignment as FASTA (default, or `--fasta`) and/or VCF
/// (`--vcf`) next to `-o <prefix>`.
pub(super) fn write_alignment_outputs(
alignment: &SnpAlignment,
labels: &[String],
args: &PhyloArgs,
) {
let prefix = args.output.as_ref().map(|p| p.display().to_string());
let path_for = |suffix: &str| match &prefix {
Some(p) => format!("{p}_alignment.{suffix}"),
None => format!("alignment.{suffix}"),
};
let n_sites = alignment.sequences.first().map_or(0, Vec::len);
let n_genomes = alignment.genome_indices.len();
if args.align.fasta || !args.align.vcf {
let path = path_for("fasta");
let mut f = create(&path);
for (&g, seq) in alignment.genome_indices.iter().zip(alignment.sequences.iter()) {
obifastwrite::write_plain_record(seq, &labels[g], &mut f).unwrap();
}
info!("pseudo-alignment ({n_sites} site(s), {n_genomes} genome(s)) → {path}");
}
if args.align.vcf {
let path = path_for("vcf");
let mut f = create(&path);
let written = alignment.write_vcf(labels, !args.sample.no_ambiguity, &mut f).unwrap_or_else(|e| {
eprintln!("error writing {path}: {e}");
std::process::exit(1);
});
f.flush().unwrap();
info!("pseudo-alignment VCF ({written}/{n_sites} site(s), {n_genomes} genome(s)) → {path}");
}
}
-431
View File
@@ -1,431 +0,0 @@
use std::path::PathBuf;
use clap::Args;
use obikphylo::{DistanceMetric, GammaShape, SnpDistanceKind};
/// `--gamma-shape` CLI value: a fixed `alpha`, or `auto`/`estimate` to fit
/// it from the data (see [`GammaShape::Auto`]). Kept as its own tiny type
/// (rather than parsing straight to `GammaShape`) so the CLI layer stays
/// free of `GammaShape::Disabled`, which the `Option<GammaShapeArg>` around
/// this already expresses on its own (`None` = flag not given at all).
#[derive(Debug, Clone, Copy)]
pub enum GammaShapeArg {
Fixed(f64),
Auto,
}
impl From<GammaShapeArg> for GammaShape {
fn from(arg: GammaShapeArg) -> Self {
match arg {
GammaShapeArg::Fixed(a) => GammaShape::Fixed(a),
GammaShapeArg::Auto => GammaShape::Auto,
}
}
}
fn parse_gamma_shape(s: &str) -> Result<GammaShapeArg, String> {
if s.eq_ignore_ascii_case("auto") || s.eq_ignore_ascii_case("estimate") {
return Ok(GammaShapeArg::Auto);
}
s.parse::<f64>()
.map(GammaShapeArg::Fixed)
.map_err(|_| format!("invalid --gamma-shape value {s:?}: expected a positive number or \"auto\"/\"estimate\""))
}
/// `--n-sites` CLI value: a number of sites to draw, or `all` for every site
/// — the draw then keeps every variable family that passes the other filters,
/// with no sampling involved. Its own tiny type (like [`GammaShapeArg`]) so
/// the two cases stay distinct up to the call that needs the target, instead
/// of an in-band sentinel (a negative number, `0`, ...) that a typo could hit.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum NSitesArg {
All,
N(usize),
}
impl NSitesArg {
/// The target to hand to the sampler: `None` = exhaustive.
pub fn target(self) -> Option<usize> {
match self {
NSitesArg::All => None,
NSitesArg::N(n) => Some(n),
}
}
}
fn parse_n_sites(s: &str) -> Result<NSitesArg, String> {
if s.eq_ignore_ascii_case("all") {
return Ok(NSitesArg::All);
}
s.parse::<usize>()
.map(NSitesArg::N)
.map_err(|_| format!("invalid --n-sites value {s:?}: expected a non-negative integer or \"all\""))
}
/// `--distance` value — either one of `obikphylo::DistanceMetric`'s
/// whole-index metrics (routed to `IndexCache::distance`) or one of
/// `obikphylo::SnpDistanceKind`'s `snp-*` corrections (routed to
/// `obikphylo::siblings::distance_matrix`, over a tally built by
/// `sampled_scan`/`PairwiseReducer`, the sibling-annex pipeline) — two
/// genuinely different code paths behind one CLI vocabulary, see
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance` unification".
#[derive(clap::ValueEnum, Clone, Copy, Debug)]
pub enum DistanceArg {
Jaccard,
Mash,
Hamming,
BrayCurtis,
#[value(name = "relfreq-bray-curtis")]
RelfreqBrayCurtis,
Euclidean,
#[value(name = "relfreq-euclidean")]
RelfreqEuclidean,
Hellinger,
#[value(name = "hellinger-euclidean")]
HellingerEuclidean,
#[value(name = "snp-raw")]
SnpRaw,
#[value(name = "snp-jc")]
SnpJc,
#[value(name = "snp-k2p")]
SnpK2p,
#[value(name = "snp-k81")]
SnpK81,
#[value(name = "snp-f81")]
SnpF81,
#[value(name = "snp-t92")]
SnpT92,
#[value(name = "snp-tn93")]
SnpTn93,
#[value(name = "snp-tv")]
SnpTv,
}
impl DistanceArg {
/// `Some` for the whole-index metrics, `None` for `snp-*` values.
pub fn as_classic(self) -> Option<DistanceMetric> {
Some(match self {
DistanceArg::Jaccard => DistanceMetric::Jaccard,
DistanceArg::Mash => DistanceMetric::Mash,
DistanceArg::Hamming => DistanceMetric::Hamming,
DistanceArg::BrayCurtis => DistanceMetric::BrayCurtis,
DistanceArg::RelfreqBrayCurtis => DistanceMetric::RelfreqBrayCurtis,
DistanceArg::Euclidean => DistanceMetric::Euclidean,
DistanceArg::RelfreqEuclidean => DistanceMetric::RelfreqEuclidean,
DistanceArg::Hellinger => DistanceMetric::Hellinger,
DistanceArg::HellingerEuclidean => DistanceMetric::HellingerEuclidean,
_ => return None,
})
}
/// `Some` for the `snp-*` values, `None` for the whole-index metrics.
pub fn as_snp(self) -> Option<SnpDistanceKind> {
Some(match self {
DistanceArg::SnpRaw => SnpDistanceKind::Raw,
DistanceArg::SnpJc => SnpDistanceKind::Jc,
DistanceArg::SnpK2p => SnpDistanceKind::K2p,
DistanceArg::SnpK81 => SnpDistanceKind::K81,
DistanceArg::SnpF81 => SnpDistanceKind::F81,
DistanceArg::SnpT92 => SnpDistanceKind::T92,
DistanceArg::SnpTn93 => SnpDistanceKind::Tn93,
DistanceArg::SnpTv => SnpDistanceKind::Tv,
_ => return None,
})
}
}
/// Genome-vs-genome distance computation: the whole-index `--distance` path
/// (classic metrics + `snp-*` corrections/NJ/UPGMA), annex construction
/// (`--sibling-annex`), annex diagnostics (`--sibling-stats`,
/// `--sibling-hist`), entropy reporting (`--shannon`), SNP pseudo-alignment
/// sampling (`--pseudo-alignment`, `--n-sites`, `--free-loss`,
/// `--no-ambiguity`, `--entropy`/`--entropy-sd`), Sankoff cost-matrix
/// calibration (`--sankoff`, `--sankoff-ratio-ceiling`) and its TNT/PhyG/
/// IQ-TREE exports (`--tnt`, `--phyg`, `--iqtree`/`--iqtree-min-freq`,
/// `--sankoff-cost-scale`), and Family Overlap (`--family-overlap`,
/// `--min-shared-family`).
#[derive(Args)]
pub struct PhyloArgs {
/// Index directory
pub index: PathBuf,
/// Exclude a genome (by its exact label) — from `--pseudo-alignment`'s
/// sampling (a family whose only polymorphism lived in an excluded
/// genome is discarded during sampling, not filtered afterward — see
/// `obikphylo::siblings::sampled_scan`'s own docs) and from the
/// distance matrix / shared-kmer matrix CSV output (row and column
/// both dropped; the underlying computation itself is unaffected).
/// Repeatable.
#[arg(long = "exclude-genome", value_name = "LABEL")]
pub exclude_genome: Vec<String>,
/// Auto-exclude any genome whose mean shared-variable-family count
/// against every other genome (`FamilyOverlap::mean_row` — the same
/// per-row statistic `--family-overlap`'s own matrix shows) falls below
/// this threshold — same exclusion machinery as `--exclude-genome`,
/// applied on top of it rather than instead of it. Applies to the
/// `snp-*` `--distance`/`--pseudo-alignment`/`--sankoff` computations
/// below (all sibling-annex-based); does *not* affect the whole-index
/// `--distance` metrics (jaccard, hamming, bray-curtis, ...) or their
/// matrix/NJ/UPGMA output — a genome with too little SNP-family
/// coverage to trust is a different concern from one whose plain k-mer
/// profile is simply divergent. Requires the Family Overlap annex
/// (built on demand if missing, same as every other annex here — see
/// `obikphylo::siblings::extensions::SiblingExt::family_overlap`'s own
/// docs).
#[arg(long, value_name = "N")]
pub min_shared_family: Option<f64>,
/// Build (or rebuild) the sibling-count/minorant annex — independent of
/// the distance metric below, meant to be run routinely, ahead of any
/// SNP-family distance computation that will later consume it.
#[arg(long)]
pub sibling_annex: bool,
/// Tally the sibling-count distribution (CSV) of an already-built annex
/// (run with `--sibling-annex` first, in this invocation or an earlier
/// one). A separate, occasional diagnostic pass — not run every time the
/// annex itself is (re)built.
#[arg(long)]
pub sibling_stats: bool,
/// Print just the global family-size histogram (1-4 members) of an
/// already-built annex — the `global` row `--sibling-stats` also
/// writes, but without the per-genome breakdown, so it skips
/// `--sibling-stats`'s cross-partition resolution entirely (annex bits
/// only).
#[arg(long)]
pub sibling_hist: bool,
/// Write the Family Overlap matrix (CSV) — number of shared *variable*
/// families per genome pair, index-wide (`obikphylo::siblings::FamilyOverlap`).
/// Built on demand if missing (same as every other annex here); no
/// `--sibling-annex` prerequisite beyond that. Every genome is written,
/// unfiltered by `--exclude-genome`/`--min-shared-family` — a raw
/// coverage diagnostic, not a computation those exclusions are meant to
/// protect.
#[arg(long)]
pub family_overlap: bool,
/// Write a per-family Shannon entropy report (CSV) — requires an
/// already-built sibling annex (`--sibling-annex` first, in this
/// invocation or an earlier one). Always a full, unsampled scan of
/// every family (`--n-sites`/`--entropy`/`--entropy-sd` below only
/// apply to `--pseudo-alignment`, not this).
#[arg(long)]
pub shannon: bool,
/// Write a SNP-only pseudo-alignment (FASTA) — requires
/// `--n-sites <N|all>` and an already-built sibling annex
/// (`--sibling-annex` first, in this invocation or an earlier one).
#[arg(long)]
pub pseudo_alignment: bool,
/// Number of variable sites to draw index-wide, or `all` for every site
/// that passes the other filters (no sampling then). Mandatory for
/// `--pseudo-alignment`/`--sankoff`; optional for a `snp-*`
/// `--distance` value, where omitting it also means every site. A
/// number is a target, not a guarantee (proportional per-layer
/// sampling; see `obikphylo::siblings::sampled_scan`'s own docs).
#[arg(long = "n-sites", value_name = "N|all", value_parser = parse_n_sites)]
pub n_sites: Option<NSitesArg>,
/// In `--pseudo-alignment`, treat a genome carrying none of a family's
/// observed members (`∅`) as missing data (`?`) rather than a real
/// character state.
#[arg(long)]
pub free_loss: bool,
/// In `--pseudo-alignment`, treat a genome carrying more than one
/// member of a family (ambiguous) as missing data (`?`) rather than an
/// IUPAC ambiguity code.
#[arg(long)]
pub no_ambiguity: bool,
/// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, keep only
/// the families that can separate two pairs of genomes from each other
/// (a bipartition): at least two distinct called states, each carried
/// by at least two genomes. A singleton variant, or a family carried by
/// fewer than four genomes, fits every topology and tells a tree
/// nothing. Which genomes count as carriers follows `--no-ambiguity`.
#[arg(long)]
pub quartet: bool,
/// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, favour the
/// families that cover the pairs of genomes sharing the fewest families
/// across the index: each family is weighted by the mean, over the pairs
/// of genomes carrying it, of 1 / (number of families the pair shares).
/// Evens out how well each pair of genomes is represented in the sample.
/// Which genomes count as carriers follows `--no-ambiguity`.
#[arg(long)]
pub pair_balance: bool,
/// Entropy-biased sampling target (Gaussian kernel mean) for
/// `--pseudo-alignment` — activates biasing as soon as this or
/// `--entropy-sd` is given; the other defaults to 1.0/0.5.
#[arg(long)]
pub entropy: Option<f64>,
/// Entropy-biased sampling kernel width (Gaussian standard deviation)
/// for `--pseudo-alignment` — see `--entropy`.
#[arg(long)]
pub entropy_sd: Option<f64>,
/// Persist the `snp-*` `--distance` sample (the pairwise substitution
/// tally) in `DIR` across separate `obikmer phylo` invocations: a later
/// run against the same index with the exact same
/// `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/
/// `--min-shared-family`/`--entropy`/`--entropy-sd` restores it instead
/// of resampling. `DIR` is created if it doesn't exist. If it already
/// holds a sample built under *different* selection parameters, this
/// run is rejected with an error listing what changed, unless
/// `--session-force` is also given (which discards the old sample and
/// starts fresh under this run's parameters). Only covers a `snp-*`
/// `--distance` today — `--sankoff`/`--tnt`/`--phyg`/`--iqtree` don't
/// read or write a `--session` yet.
#[arg(long, value_name = "DIR")]
pub session: Option<PathBuf>,
/// With `--session DIR`: overwrite its saved selection parameters (and
/// discard its cached sample) instead of erroring out when this run's
/// parameters don't match. No effect without `--session`.
#[arg(long, requires = "session")]
pub session_force: bool,
/// Calibrate a 16-state Sankoff cost matrix (and its matching
/// pseudo-alignment) from an already-built sibling annex — requires
/// `--n-sites <N|all>`, and shares `--free-loss`/`--no-ambiguity`/`--quartet`/
/// `--entropy`/`--entropy-sd` with `--pseudo-alignment` (one draw, same
/// selection feeds both the alignment and every calibration tally).
#[arg(long)]
pub sankoff: bool,
/// Exclude genome pairs whose raw SNP ratio exceeds this value from the
/// base-pair (composition) calibration `--sankoff` pools — a pair this
/// close to substitution saturation carries no information about the
/// true substitution spectrum. Does *not* gate the cardinality
/// calibration (see `obikphylo::siblings::CardinalityTally`'s own
/// docs for why).
#[arg(long, default_value = "0.5")]
pub sankoff_ratio_ceiling: f64,
/// Also write <prefix>_sankoff.tnt, a ready-to-run TNT script (`proc
/// <file>;`) for the same matrix/alignment `--sankoff` computes —
/// recoded to TNT's default xread alphabet (0-9A-F only; TNT rejects
/// the wider IUPAC set `--sankoff`'s own output uses unless `nstates
/// dna` is set, which imposes TNT's own incompatible DNA encoding
/// instead) with integer-scaled costs (TNT's smatrix/cost commands
/// reject decimals). Implies `--sankoff`.
#[arg(long)]
pub tnt: bool,
/// Also write <prefix>_sankoff.tcm and <prefix>_sankoff.pg, a
/// custom-alphabet cost matrix and a ready-to-run PhyG script (`read`/
/// `search`/`report`) for the same matrix/alignment `--sankoff`
/// computes. Reuses `--sankoff`'s own `_sankoff.fasta` directly — PhyG's
/// `tcm:` alphabet is read from the matrix file itself, so the IUPAC+`0`
/// alphabet needs no recoding here, unlike `--tnt`. Implies `--sankoff`.
#[arg(long)]
pub phyg: bool,
/// Also write <prefix>_iqtree.model and <prefix>_iqtree.fasta, a
/// custom-model file and a matching recoded alignment for genuine
/// maximum-likelihood inference with IQ-TREE (`iqtree3 -s ...
/// --seqtype MORPH -m ...+ASC`) — real branch lengths, unlike
/// `--tnt`/`--phyg`'s parsimony step counts. The model is the
/// reversible `Q(i,j) = R(i,j)·π_j` construction: `R` (exchangeability,
/// symmetric) recovered from the same calibrated cost matrix
/// `--sankoff` computes, `π` the real empirical state frequencies
/// counted from the alignment. Only the states that actually occur in
/// this alignment are kept, compactly renumbered (IQ-TREE infers its
/// state count from the alignment itself, and a gap in the numbering
/// would silently misalign the model file). Implies `--sankoff`.
#[arg(long)]
pub iqtree: bool,
/// Under `--iqtree --free-loss`, also recode to `?` (the same
/// missing-data treatment as `-`) any state whose empirical frequency
/// in the alignment falls below this threshold — not just genuinely
/// absent calls. States encoding 3 or 4 simultaneously-observed central
/// bases (IUPAC `V`/`H`/`K`.../`N` for 3, `N` for 4) are rare by
/// construction and often land in exactly this low-frequency range —
/// more likely assembly/detection noise than a genuine, widely-preserved
/// multi-way polymorphism, the same "sampling failure, not true signal"
/// reasoning `--free-loss` already applies to absence. No effect
/// without `--free-loss` (there is no missing-data symbol to recode to
/// otherwise). `<prefix>_iqtree_states.csv` reports the frequency
/// actually used to decide.
#[arg(long, default_value = "0.001")]
pub iqtree_min_freq: f64,
/// Scale factor applied before rounding real-valued costs to the
/// integers both `--tnt`'s smatrix/cost commands and `--phyg`'s `tcm:`
/// matrix require. Keep this small: the total tree score is this scale
/// times the sum of per-character costs across every character, and
/// there are hints in TNT's own manual that at least some of its
/// internal accumulators are 32-bit — a large scale risks a silent
/// integer overflow (undetectable, not just a crash) far more costly
/// than the resolution a bigger factor would buy. Shared between `--tnt`
/// and `--phyg` rather than split into two flags: both scale the same
/// calibrated matrix for the same reason (integer-only cost commands).
#[arg(long, default_value = "100")]
pub sankoff_cost_scale: f64,
/// Distance to compute — either a whole-index metric (`jaccard`,
/// `mash`, `hamming`, `bray-curtis`, ...) or a `snp-*` correction over
/// the central-position SNP substitution spectrum (`snp-raw`, `snp-jc`,
/// `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`) —
/// the latter route to a different computation entirely
/// (`obikphylo::siblings::sampled_scan` + `distance_matrix`, requires
/// `--sibling-annex` first; see
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance`
/// unification" for the full catalog and why LogDet/Tajima-Nei/F84/
/// HKY85 aren't offered yet).
#[arg(long, value_enum, default_value = "jaccard")]
pub distance: DistanceArg,
/// Rate-heterogeneity correction (Jin-Nei gamma shape parameter `α`)
/// for `snp-*` `--distance` values that support it
/// (`obikphylo::SnpDistanceKind::supports_gamma`: every one except
/// `snp-raw`/`snp-tv`, which have nothing to correct/are deliberately
/// uncorrected). Has no effect on the whole-index metrics. Rejected at
/// runtime if given alongside an unsupported `--distance` value.
/// Either a positive number (fixed `alpha`) or `auto`/`estimate` to fit
/// `alpha` from the data itself — method-of-moments over the
/// over-dispersion of substitution rate across partitions.
#[arg(long, value_name = "ALPHA|auto", value_parser = parse_gamma_shape)]
pub gamma_shape: Option<GammaShapeArg>,
/// Minimum count to consider a kmer present when computing Jaccard on count indexes
#[arg(long, default_value = "1")]
pub presence_threshold: u32,
/// Write the primary distance matrix as plain CSV instead of the
/// default relaxed-PHYLIP format (`n` on the first line, then one
/// `label<TAB>value...` row per genome — no 10-character label
/// truncation, unlike strict PHYLIP, not yet offered here). PHYLIP is
/// the default because it's what external NJ tools (PHYLIP `neighbor`,
/// FastME, T-REX, SplitsTree) actually read; CSV stays available for
/// scripting/inspection. Only affects the primary distance matrix —
/// `--shared-kmers` keeps its own CSV-only format regardless of this
/// flag.
#[arg(long)]
pub csv: bool,
/// Also output the shared-kmer count matrix (CSV)
#[arg(long)]
pub shared_kmers: bool,
/// Compute and write a Neighbor-Joining tree (Newick)
#[arg(long)]
pub nj: bool,
/// Compute and write a UPGMA tree (Newick)
#[arg(long)]
pub upgma: bool,
/// Output prefix: <prefix>_dist.csv, <prefix>_shared.csv, <prefix>_nj.nwk,
/// <prefix>_upgma.nwk. If omitted, the distance matrix is written to stdout.
#[arg(short, long)]
pub output: Option<PathBuf>,
}
+136
View File
@@ -0,0 +1,136 @@
use std::path::PathBuf;
use clap::Args;
use super::super::values::{DistanceArg, GammaShapeArg, parse_gamma_shape};
/// Distance estimation and tree reconstruction.
#[derive(Args)]
#[command(next_help_heading = "Distance estimation and tree reconstruction")]
pub struct DistanceArgs {
/// Distance used to compare genomes
///
/// Computes a genome-by-genome distance matrix. Two families of
/// distances are available.
///
/// Whole-genome distances, based on the k-mer content of each genome.
/// The first three compare which k-mers are present or absent (on an
/// index that stores counts, a k-mer is present when its count reaches
/// --presence-threshold):
/// jaccard Jaccard distance
/// mash Mash distance, a divergence estimate derived
/// from the Jaccard index
/// hamming number of k-mers present in one genome and
/// absent from the other
/// The others compare k-mer counts, and need an index that stores them:
/// bray-curtis Bray-Curtis dissimilarity on k-mer counts
/// relfreq-bray-curtis Bray-Curtis dissimilarity on counts divided
/// by each genome's total count
/// euclidean Euclidean distance on k-mer counts
/// relfreq-euclidean Euclidean distance on relative frequencies
/// hellinger Hellinger distance on k-mer counts
/// hellinger-euclidean Euclidean distance on square roots of
/// relative frequencies (unnormalised Hellinger)
///
/// SNP-based distances, calculated from the nucleotide differences at
/// the central position of related k-mers:
/// snp-raw proportion of differing sites, no correction
/// snp-jc Jukes-Cantor correction
/// snp-k2p Kimura two-parameter correction
/// snp-k81 Kimura three-parameter correction
/// snp-f81 Felsenstein 1981 correction
/// snp-t92 Tamura 1992 correction
/// snp-tn93 Tamura-Nei 1993 correction
/// snp-tv proportion of transversions only, no correction
/// They use every eligible variable site, or a sample drawn with
/// --n-sites.
#[arg(
long,
value_enum,
verbatim_doc_comment,
hide_possible_values = true,
value_name = "DISTANCE"
)]
pub distance: Option<DistanceArg>,
/// Minimum k-mer count to count as present
///
/// Converts counts into presence/absence for the Jaccard, Mash and
/// Hamming distances and for the shared k-mer counts, when the index
/// stores counts: a k-mer is present in a genome only when its count reaches
/// this threshold. It has no effect on indexes without counts, nor on
/// the distances that use the counts themselves.
#[arg(long, default_value = "1")]
pub presence_threshold: u32,
/// Account for different evolution rates among sites
///
/// Applies a gamma rate-heterogeneity correction to SNP-based
/// distances. A positive number is the gamma shape parameter α. `auto`
/// (or `estimate`) estimates α from the data, using the variation of
/// the substitution rate across the partitions of the index.
///
/// Available for every snp-* distance except snp-raw and snp-tv, which
/// have no evolutionary correction to refine. It has no effect on
/// whole-genome distances, and combining it with a distance that does
/// not support it is an error.
#[arg(long, value_name = "ALPHA|auto", value_parser = parse_gamma_shape)]
pub gamma_shape: Option<GammaShapeArg>,
/// Save and reuse the sample of a SNP-based distance
///
/// Saves, in this directory, the sample's tally of pairwise differences
/// between genomes, so that a later run on the same index can reuse it
/// instead of drawing again. The directory is created if needed.
///
/// The saved sample is reused only if these options are identical:
/// --n-sites, --free-loss, --no-ambiguity, --quartet, --exclude-genome,
/// --min-shared-family, --entropy and --entropy-sd. Otherwise the run
/// stops and lists the differences, unless --session-force is given.
///
/// For now only SNP-based distances use sessions; Sankoff analyses and
/// the TNT, PhyG and IQ-TREE exports neither read nor write them.
#[arg(long, value_name = "DIR")]
pub session: Option<PathBuf>,
/// Replace a saved sample made with other parameters
///
/// With --session, discards the saved sample when its parameters differ
/// from this run's, and draws a new one. Without this option, a
/// difference is an error.
#[arg(long, requires = "session")]
pub session_force: bool,
/// Write the distance matrix as CSV instead of PHYLIP
///
/// By default the main matrix is written in relaxed PHYLIP format: the
/// number of genomes on the first line, then one row per genome made of
/// its label (never truncated) and its distances to the others. CSV is
/// easier to load in scripts or spreadsheets. Only the main matrix is
/// affected: the --shared-kmers matrix is always CSV.
#[arg(long)]
pub csv: bool,
/// Also write the number of k-mers shared by each pair of genomes
///
/// Writes a genome-by-genome CSV matrix of shared k-mer counts, to
/// `PREFIX_shared.csv`.
#[arg(long)]
pub shared_kmers: bool,
/// Build a Neighbor-Joining tree
///
/// Builds the tree from the distance matrix and writes it in Newick
/// format to `PREFIX_nj.nwk`. If --distance is not given, the Jaccard
/// distance is used.
#[arg(long)]
pub nj: bool,
/// Build a UPGMA tree
///
/// Builds a tree with the Unweighted Pair Group Method with Arithmetic
/// Mean and writes it in Newick format to `PREFIX_upgma.nwk`. If
/// --distance is not given, the Jaccard distance is used.
#[arg(long)]
pub upgma: bool,
}
Loaded 100 of 188 files, more files were not shown because too many files have changed in this diff. Show more