feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors. - Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing. - Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants. - Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling. - Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections. - Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic. - Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure. - Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions. - Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry. - Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans. - Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking. - Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
This commit is contained in:
1 parent
b35127e4e3
commit
1f199c69db
25 files changed
+1631
-687
No files matched your search
@@ -62,52 +62,38 @@ fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
|
||||
|
||||
Check all sources against the constraints above. Abort on any mismatch.
|
||||
|
||||
### 2. Bootstrap output from first source
|
||||
### 2. Bootstrap output from the base source
|
||||
|
||||
Recursive file copy of `sources[0]` → `output`. Immediately after the copy:
|
||||
The base is chosen by `choose_base` (`merge.rs`): the source with the largest total `unitigs.bin` size (restricted to approximate-evidence sources when the output must be approximate). It is not necessarily `sources[0]`, and the genome columns follow the base first. Because `obikmer index` is not byte-reproducible (unitig segmentation varies), two near-equal sources can swap roles between runs. Its directory is copied to `output` (hard links for files `merge_partition` never rewrites, see below). Immediately after the copy:
|
||||
|
||||
- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
|
||||
- In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
|
||||
- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
|
||||
|
||||
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
|
||||
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The base source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
|
||||
|
||||
### 3. For each subsequent source (parallel across partitions)
|
||||
|
||||
`KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)` is called for each partition index `i`. `block_bits` is taken from `dst.meta.config.block_bits`.
|
||||
`obikmerge::partition_merge::merge_partition(dst, i, sources, mode, n_dst_genomes, block_bits, evidence)` runs once per partition index `i` (scheduled largest first by `PartitionRunner`). Each entry of `sources` is `(&KmerIndex, n_genomes)`, where `n_genomes` is the number of columns that source contributes (> 1 for an already-merged source). Code: `partition_merge/mod.rs` (orchestration, pass 1) and `partition_merge/pass2.rs` (per-group fill pipeline).
|
||||
|
||||
Each entry in `sources` is `(&KmerPartition, n_genomes)` where `n_genomes` is the column count that source contributes (> 1 when the source is itself a merged index).
|
||||
**First merge, Presence mode**: when `n_dst_genomes == 1`, `TypedLayer::<()>::init_presence_matrix` is called on every existing dst layer first. It materialises the implicit single-genome matrix (`presence/col_000000.pbiv`, all true) so that source columns can be appended to it.
|
||||
|
||||
**First merge, Presence mode**: when `n_dst_genomes == 1`, `Layer::<()>::init_presence_matrix` is called on every existing destination layer before any source column is appended. This creates `presence/col_000000.pbiv` set all-true (genome 0 is present in every slot).
|
||||
The dst layers are opened once per partition and shared by both passes.
|
||||
|
||||
**Pass 1 — classify kmers**
|
||||
**Pass 1 — new k-mers into a de Bruijn graph.** Only the sources' `unitigs.bin` *paths* are needed (`KmerLayer::new(..).unitigs_path()`); no source layer is opened. The files are streamed (at most `max_open = 2` open at once), and every k-mer absent from all dst layers is pushed into one `GraphDeBruijn`. All new k-mers of all sources go into a **single** new layer per merge, built by `materialize_layer` (unitigs + MPHF + evidence).
|
||||
|
||||
Iterate all kmers from all source partitions (via `UnitigFileReader` + canonical kmer iteration). For each kmer, probe the destination `LayeredMap<()>`:
|
||||
**New-layer matrix.** If a new layer exists, its matrix is created with `n_dst_genomes` absent columns (all zero/false) and closed immediately. This keeps the column-count invariant, and every source group then appends to it like to the existing layers.
|
||||
|
||||
- **Hit**: kmer already in the destination; record for Pass 2.
|
||||
- **Miss**: push kmer into a `GraphDeBruijn` accumulator.
|
||||
**Pass 2 — by consecutive groups of `SRC_GROUP = 2` sources, in source order.** Per group:
|
||||
|
||||
**New layer construction**
|
||||
1. open the group's source layers for this partition;
|
||||
2. `MatrixBuilder::resume` every dst matrix (existing layers + new layer), check `n_cols == n_dst_genomes + col_base`, append the group's `group_n` columns;
|
||||
3. run the pipeline: stream each source layer's k-mers, `hash_batch` + `fill_sub_matrix` on the source, locate each k-mer in the dst layers (`find_batch_grouped`) or else in the new layer's MPHF, and write the values column-major into the group's builders;
|
||||
4. close the columns, then the matrices; drop the group's layers; `col_base += group_n`.
|
||||
|
||||
If the accumulator is non-empty, compute de Bruijn unitigs and call `Layer::<()>::build(&new_layer_dir, block_bits)`. All kmers absent from the destination — across **all** sources — accumulate into a **single** graph, producing one new layer per merge operation (not one per source).
|
||||
Only one group's source layers and destination columns are alive at a time, so the number of live mappings per partition does not depend on the number of sources. Columns are appended in source order, so every genome lands in the same absolute column as in a single-session merge.
|
||||
|
||||
**Pass 2 — fill column builders**
|
||||
|
||||
For each source and each of its layers, re-iterate unitigs and look up stored values via `SrcLayerData::lookup(kmer, src_n)`:
|
||||
|
||||
- `SrcLayerData::SetMembership` — no data directory exists; every kmer returns `vec![1; n_genomes]`
|
||||
- `SrcLayerData::Presence` — reads `PersistentBitMatrix` from `presence/`
|
||||
- `SrcLayerData::Count` — reads `PersistentCompactIntMatrix` from `counts/`
|
||||
|
||||
Hits are routed to `exist_builders[dst_layer][src_col]`; misses are routed to `new_src_builders[src_col]`.
|
||||
|
||||
**Column prepending for new layers**
|
||||
|
||||
Before source columns are written to the new layer, `n_dst_genomes` absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.
|
||||
|
||||
**Close and update metadata**
|
||||
|
||||
Close all builders; update `presence/meta.json` or `counts/meta.json` with `{"n": N, "n_cols": n_dst_genomes + n_src_total}`; increment `PartitionMeta::n_layers` if a new layer was added.
|
||||
**In-memory columns.** The group's columns are built in RAM (`MatrixBuilder::add_mem_col` → `MemBitVecBuilder` / `MemCompactIntVecBuilder`) and written once, sequentially, at `close` (`.building` file, then `rename`). The files are byte-identical to the former mmap-backed builders' output. There is no sparse file and no `MmapMut` on this path, so a block the FS cannot allocate is an `io::Error` instead of a SIGBUS. Files are not `fsync`ed: one sync per column file made merge ~8× slower. RAM per partition is at most `SRC_GROUP × src_n × N_part × (1/8 | 1)` bytes (presence | counts). History and figures: `merge_mmap_audit.md`; scaling limits: `merge_scaling.md`.
|
||||
|
||||
### 4. Update index metadata
|
||||
|
||||
@@ -129,7 +115,7 @@ impl Layer<PersistentBitMatrix> {
|
||||
}
|
||||
```
|
||||
|
||||
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written directly via `PersistentBitVecBuilder` / `PersistentCompactIntVecBuilder` rather than through these helpers, but the invariant they enforce is the same.
|
||||
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written through `MatrixBuilder::add_mem_col` (in-memory builders) rather than through these helpers, but the invariant they enforce is the same.
|
||||
|
||||
---
|
||||
|
||||
@@ -139,7 +125,7 @@ After any merge, **every layer in every partition has exactly `n_genomes` column
|
||||
|
||||
Maintained by three mechanisms:
|
||||
|
||||
1. **Existing layers**: `n_src_total` columns appended (one per source genome).
|
||||
1. **Existing layers**: `n_src_total` columns appended (one per source genome), group by group.
|
||||
2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
|
||||
3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
|
||||
|
||||
@@ -161,7 +147,7 @@ The invariant is a precondition of `LayeredStore` aggregation traits: `col_weigh
|
||||
|
||||
## On-disk impact
|
||||
|
||||
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest):
|
||||
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest), *before* `finalize` packs each matrix (sparse by default, `matrix.pbmx` with `--dense`):
|
||||
|
||||
```
|
||||
partitions/
|
||||
|
||||
Reference in new issue
Block a user