Files
obikmer/UserDocMD
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00
..