Update documentation to reflect that sparse packing now maps all columns of a layer at once and packs layers sequentially, reversing the previous choice to avoid materializing a packed copy. This change introduces an extra write/read cycle for the dense matrix, increasing pack benchmark time from 1.11s to 1.60s. - Modify `pack_sparse_bit_matrix` in `src/obicompactvec/src/bitmatrix/sparse.rs` to always convert via the dense packed format, simplifying logic by removing conditional handling of existing files and ensuring a single mapping. - Update `pack_sparse_compact_int_matrix` in `src/obicompactvec/src/sparse_intmatrix.rs` to exclusively handle conversion from the dense packed format (`matrix.pcmx`) to sparse, removing columnar format handling and simplifying file cleanup. - Adjust `scripts/bench_merge_oracle.sh` to extract `PACK_WALL` metrics via `stage_field` for merge operations, updating grep patterns and output messages to report pack wall time. - Revise `DevDocMD/implementation/merge_mmap_audit.md` and `merge_scaling.md` to document the new sequential packing strategy, noting that `N_part` scaling applies only to incremental merges and that `obisys::MemoryBudget` is no longer used by merge. - Bump package version in `src/obikmer/Cargo.toml` from 1.4.3 to 1.4.4.
12 KiB
Merge: mmap audit (2026-11-17)
Audit of the memory-mapping behaviour of obikmerge::partition_merge::merge_partition, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS /hoyt). Status: A + C implemented (2026-11-17), B deferred. See Implementation. Implementation prompt: prompts/merge_mmap_refactor.md (repository root, outside the doc tree).
Observed failures
| Run | Flags | Failure | When |
|---|---|---|---|
| 1 | none | Bus error (SIGBUS) |
seconds after merge_partitions starts |
| 2 | --cpu-max 32 |
memory allocation of 32768 bytes failed → abort, many threads at once |
after ~1 h, 128/256 partitions done |
Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, merge.rs:289), so at 50 % every remaining partition is smaller than one already merged. Small malloc failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from malloc means mmap/brk returned ENOMEM: vm.max_map_count (default 65 530), RLIMIT_AS, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.
Inventory: mappings held by one merge_partition call
Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).
| Resource | Where | Maps | Lifetime | Needed? |
|---|---|---|---|---|
| Source layers, fully opened | partition_merge.rs:219-227 → KmerLayer::open → MphfLayer::open (mphf_layer.rs:171) |
4 per layer: MPHF (MphfOnly), unitigs.bin (direct access), evidence.bin, unitigs.bin again (LayerEvidence::open, mphf_layer.rs:100). ≈ 740 |
whole partition | Only the MPHF. Pass 2 calls hash_batch (MPHF only, mphf_layer.rs:258) and fill_sub_matrix (matrix: Implicit for mono-genome, zero maps). Pass 1 only needs the unitigs.bin path. |
| Writable columns, created up front | partition_merge.rs:327, :349 → PersistentBitVecBuilder::new / PersistentCompactIntVecBuilder::new |
set_len (sparse) + MmapMut, n_src_total × (L_dst + 1) ≈ 370 |
whole pass 2 | Only the columns of the ≤ 2 sources being streamed (max_open = 2). |
| dst layers, new-layer MPHF, sequential readers | various | ≈ 10 | variable | yes |
≈ 1 100 maps per partition, > 95 % idle at any instant. × ~32 concurrent partitions (--cpu-max 32) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, not measured. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is not established.
Weaknesses
- Source layers opened in full although merge needs MPHF + matrix only. Evidence and both
unitigs.binmaps are dead weight. - All sources opened up front, for the whole partition. This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
- All destination columns created up front, the same mismatch on the write side.
- Sparse files written through
MmapMut. A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing. - No bound on live mappings.
obipipeline::throttleonly bounds open read streams. - (minor)
obicompactvec::mmap_file::open_mmap_filevalidates magic and header length, not file length againstn.
Side finding: merge_parallelism.md documents a MemoryBudget semaphore and --budget-fraction. obisys::MemoryBudget still exists but merge.rs no longer uses it (scheduling is PartitionRunner). That page is stale on this point.
Plan
A. Source-group streaming in pass 2 (obikmerge). Process sources in consecutive groups of G (G = 2, matching max_open). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its src_n columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through KmerLayer::new(..).unitigs_path() (valid on Empty). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.
C. In-memory column builders (obicompactvec). Under A only G × src_n × (L_dst+1) columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with write_all to the .building path, then publish. No sparse file and no MmapMut on the merge write path, so ENOSPC/quota becomes an io::Error. The on-disk format must be byte-identical to the current builders' output. Exposed through MatrixBuilder/ColBuilder so obikselect/obikfilter can adopt it later. Fixes 4.
B. Slots-only layer opening (obikindex, the authority crate). Deferred to a separate work item (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same hash_batch + fill_sub_matrix pattern without find appears in three crates: obikmerge (partition_merge.rs:489), obikfilter (partition_iter.rs:106), obikphylo (family_scan.rs:289). family_scan.rs:402 hashes variants; whether those are guaranteed members is unchecked. Fixes 1.
- Rejected: lazy evidence inside
MphfLayer.find/confirmreturnOption<usize>, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (find, needs evidence) vs in-layer iteration (index/index_batch, no evidence). - Rejected: a standalone parallel type next to
KmerLayer/TypedLayer. It would duplicate count/presence detection and matrix access, and two opening paths would drift. - Retained: composition. A slots layer =
MphfOnly+ matrix, carryinghash_batch,fill_sub_matrix,n_cols. The full layer = slots layer + evidence, keepingfind/confirm.KmerLayerexposes both openings (open()/ e.g.open_slots()) over a single count/presence detection. Open point:MphfLayer::ncurrently comes from the evidence (evidence.len()/fingerprint.n()). A slots layer must take it from the matrix (n_rows;layer_meta.jsonforImplicit), so the two must be checked to always agree.
Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the obikindex query path and needs its own session with query performance checks.
Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.
Implementation (2026-11-17)
A — source-group streaming (obikmerge/src/partition_merge/, split into mod.rs + pass2.rs). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates sources.chunks(SRC_GROUP = 2), resuming each dst matrix, checking n_cols == n_dst_genomes + col_base (InvalidData otherwise), and appending the group's columns. Prerequisite: MatrixBuilder::n_cols (obicompactvec).
C — in-memory columns (obicompactvec). New mem_builder.rs (MemBitVecBuilder, MemCompactIntVecBuilder), add_mem_col on both matrix builders and on MatrixBuilder, and ColBuilder::MemBit/MemInt. Format writing is factored rather than duplicated: format::write_pciv_header/write_pciv_tail (also used by finalize_pciv), mmap_file::write_std_header/write_built. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses add_mem_col for source columns; add_absent_col is unchanged (never written through its mapping).
Deviations from the plan:
- No
sync_allinclose. With one forced sync per column file,merge_partitionswent from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. Therenamestill prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface atclose(2), whichFile's drop ignores. add_mem_colreturnsColBuilder, notio::Result<ColBuilder>: it does no I/O.- The mmap count builder (
PersistentCompactIntVecBuilder) writes in place, without.building; the in-memory one goes through.building+rename. Same final bytes. obikmerge'smatrix_builder_testswere removed: they only exercisedobicompactvec::MatrixBuilderand are covered byobicompactvec/src/tests/{matrix_builder,mem_builder}.rs.tempfiledev-dependency ofobikmergeremoved with them.
Verification. Two golden-output oracles, recorded on the unmodified code:
scripts/smoke_test_merge.sh: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/--dense; frozen source indexes;scripts/bench_merge_oracle.sh: 20 real specimen presence indexes, sorteddump --debughash, 85 681 700 k-mers.
Both pass after every step. The existing global_index_presence (October binary) gives the same hash.
merge_partitions wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.
Live mappings per partition (estimate, not measured). Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.
| Source layers | Dst columns | Other | Total | |
|---|---|---|---|---|
| Before | 186 × 4 ≈ 744 | 186 × 2 ≈ 372 (MmapMut) |
≈ 10 | ≈ 1 100 |
| After A + C | 2 × 4 = 8 | 0 (in RAM) | ≈ 10 | ≈ 20, independent of the source count |
B would take the 8 down to ~2.
Follow-up — sparse pack (same day). pack_sparse_bit_matrix/pack_sparse_compact_int_matrix used to open the columnar matrix directly, which maps every column file of a layer at once: n_genomes maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single matrix.pbmx/.pcmx mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: pack went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale matrix.pbmx that predates newer columnar data is now re-packed instead of being used as is (pack_bit_matrix compares n_cols). Both oracles pass.
Known issue (found while building the oracle, not fixed)
Re-merging a merge output fails with I/O error: No such file or directory in merge_partitions, in every mode (presence sparse, presence --dense, counts). Cause, verified: finalize_indexed packs the output matrices (sparse_meta.json + dict/PFIV files, or matrix.pbmx), so presence//counts/ no longer holds meta.json + col_*. When that index is the merge base, bootstrap copies the packed layout and MatrixBuilder::resume → MatrixMeta::load finds no meta.json. Pre-existing, independent of this refactor; deferred to a separate fix (decision 2026-11-17). The synthetic oracle (scripts/smoke_test_merge.sh) therefore omits the "merge AB, then AB C D" scenario.
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to mmap.