Files
obikmer/mkdocs.yml
T
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00

77 lines
3.0 KiB
YAML

site_name: obikmer
docs_dir: DevDocMD
site_dir: DevDoc
theme:
name: material
plugins:
- mermaid2:
- bibtex:
bib_file: DevDocMD/references.bib
csl_file: DevDocMD/ecology-letters.csl
enable_inline_citations: false
markdown_extensions:
- admonition
- footnotes
- tables
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_code_format
- pymdownx.arithmatex:
generic: true
extra_javascript:
- https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
nav:
- Home: index.md
- Installation: installation.md
- Theory:
- Kmers and super-kmers: kmers.md
- DNA encoding: theory/encoding.md
- Entropy filter: theory/entropy.md
- Minimizer selection: theory/minimizer.md
- Partitioning architecture: theory/indexing.md
- Chance sharing across genomes (discussion): theory/chance_sharing.md
- Central-position SNP distance (discussion): theory/evolutionary_distances.md
- Sibling partition unification (discussion): theory/sibling_partition_unification.md
- Pair-coverage-biased family sampling (discussion): theory/pair_coverage_sampling.md
- Implementation:
- SuperKmer: implementation/superkmer.md
- Kmer: implementation/kmer.md
- Chunk reader: implementation/chunkreader.md
- Construction pipeline: implementation/pipeline.md
- obipipeline library: implementation/obipipeline.md
- On-disk storage: implementation/storage.md
- MPHF selection: implementation/mphf.md
- "MPHF batch queries (inventory)": implementation/mphf_batch_queries.md
- Unitig evidence encoding: implementation/unitig_evidence.md
- Evidence elimination (discussion): implementation/evidence_elimination.md
- obikindex layer tier: implementation/layer_tier.md
- PersistentCompactIntVec: implementation/persistent_compact_int_vec.md
- PersistentBitVec: implementation/persistent_bit_vec.md
- Merge command: implementation/merge.md
- Merge parallelism & memory: implementation/merge_parallelism.md
- "Merge mmap audit (2026-11)": implementation/merge_mmap_audit.md
- "Merge scaling (open)": implementation/merge_scaling.md
- Kmer filtering: implementation/filtering.md
- Select command: implementation/select.md
- obitaxonomy crate: implementation/obitaxonomy.md
- "Benchmark: query-path testing": implementation/benchmark_query_testing.md
- "Partition and layer caching (discussion)": implementation/partition_layer_cache.md
- Architecture:
- Sequences: architecture/sequences/invariant.md
- Kmer index: architecture/index_architecture.md
- Sibling annex: architecture/siblings.md
- Hot resume and memoization: architecture/hot_resume_memoization.md
- NUMA-aware worker pools: architecture/numa_worker_pools.md
- NUMA-aware partition runner: architecture/numa_partition_runner.md
watch:
- DevDocMD
- mkdocs.yml