Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors. - Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing. - Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants. - Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling. - Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections. - Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic. - Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure. - Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions. - Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry. - Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans. - Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking. - Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
77 lines
3.0 KiB
YAML
77 lines
3.0 KiB
YAML
site_name: obikmer
|
|
docs_dir: DevDocMD
|
|
site_dir: DevDoc
|
|
|
|
theme:
|
|
name: material
|
|
|
|
plugins:
|
|
- mermaid2:
|
|
- bibtex:
|
|
bib_file: DevDocMD/references.bib
|
|
csl_file: DevDocMD/ecology-letters.csl
|
|
enable_inline_citations: false
|
|
|
|
markdown_extensions:
|
|
- admonition
|
|
- footnotes
|
|
- tables
|
|
- pymdownx.superfences:
|
|
custom_fences:
|
|
- name: mermaid
|
|
class: mermaid
|
|
format: !!python/name:pymdownx.superfences.fence_code_format
|
|
- pymdownx.arithmatex:
|
|
generic: true
|
|
|
|
extra_javascript:
|
|
- https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
|
|
|
|
nav:
|
|
- Home: index.md
|
|
- Installation: installation.md
|
|
- Theory:
|
|
- Kmers and super-kmers: kmers.md
|
|
- DNA encoding: theory/encoding.md
|
|
- Entropy filter: theory/entropy.md
|
|
- Minimizer selection: theory/minimizer.md
|
|
- Partitioning architecture: theory/indexing.md
|
|
- Chance sharing across genomes (discussion): theory/chance_sharing.md
|
|
- Central-position SNP distance (discussion): theory/evolutionary_distances.md
|
|
- Sibling partition unification (discussion): theory/sibling_partition_unification.md
|
|
- Pair-coverage-biased family sampling (discussion): theory/pair_coverage_sampling.md
|
|
- Implementation:
|
|
- SuperKmer: implementation/superkmer.md
|
|
- Kmer: implementation/kmer.md
|
|
- Chunk reader: implementation/chunkreader.md
|
|
- Construction pipeline: implementation/pipeline.md
|
|
- obipipeline library: implementation/obipipeline.md
|
|
- On-disk storage: implementation/storage.md
|
|
- MPHF selection: implementation/mphf.md
|
|
- "MPHF batch queries (inventory)": implementation/mphf_batch_queries.md
|
|
- Unitig evidence encoding: implementation/unitig_evidence.md
|
|
- Evidence elimination (discussion): implementation/evidence_elimination.md
|
|
- obikindex layer tier: implementation/layer_tier.md
|
|
- PersistentCompactIntVec: implementation/persistent_compact_int_vec.md
|
|
- PersistentBitVec: implementation/persistent_bit_vec.md
|
|
- Merge command: implementation/merge.md
|
|
- Merge parallelism & memory: implementation/merge_parallelism.md
|
|
- "Merge mmap audit (2026-11)": implementation/merge_mmap_audit.md
|
|
- "Merge scaling (open)": implementation/merge_scaling.md
|
|
- Kmer filtering: implementation/filtering.md
|
|
- Select command: implementation/select.md
|
|
- obitaxonomy crate: implementation/obitaxonomy.md
|
|
- "Benchmark: query-path testing": implementation/benchmark_query_testing.md
|
|
- "Partition and layer caching (discussion)": implementation/partition_layer_cache.md
|
|
- Architecture:
|
|
- Sequences: architecture/sequences/invariant.md
|
|
- Kmer index: architecture/index_architecture.md
|
|
- Sibling annex: architecture/siblings.md
|
|
- Hot resume and memoization: architecture/hot_resume_memoization.md
|
|
- NUMA-aware worker pools: architecture/numa_worker_pools.md
|
|
- NUMA-aware partition runner: architecture/numa_partition_runner.md
|
|
|
|
watch:
|
|
- DevDocMD
|
|
- mkdocs.yml
|