Files
obikmer/DevDocMD/implementation/merge_mmap_audit.md
T
Eric Coissac 07bc5fc634
ci.yml / build (pull_request) Successful in 4m53s
Release / create-release (push) Successful in 28s
Release / build-linux-x86_64 (push) Successful in 6m46s
Release / build-macos-arm64 (push) Successful in 4m8s
docs: update sparse packing strategy and benchmark scripts
Update documentation to reflect that sparse packing now maps all columns of a layer at once and packs layers sequentially, reversing the previous choice to avoid materializing a packed copy. This change introduces an extra write/read cycle for the dense matrix, increasing pack benchmark time from 1.11s to 1.60s.

- Modify `pack_sparse_bit_matrix` in `src/obicompactvec/src/bitmatrix/sparse.rs` to always convert via the dense packed format, simplifying logic by removing conditional handling of existing files and ensuring a single mapping.
- Update `pack_sparse_compact_int_matrix` in `src/obicompactvec/src/sparse_intmatrix.rs` to exclusively handle conversion from the dense packed format (`matrix.pcmx`) to sparse, removing columnar format handling and simplifying file cleanup.
- Adjust `scripts/bench_merge_oracle.sh` to extract `PACK_WALL` metrics via `stage_field` for merge operations, updating grep patterns and output messages to report pack wall time.
- Revise `DevDocMD/implementation/merge_mmap_audit.md` and `merge_scaling.md` to document the new sequential packing strategy, noting that `N_part` scaling applies only to incremental merges and that `obisys::MemoryBudget` is no longer used by merge.
- Bump package version in `src/obikmer/Cargo.toml` from 1.4.3 to 1.4.4.
2026-11-17 15:35:41 +01:00

12 KiB

Merge: mmap audit (2026-11-17)

Audit of the memory-mapping behaviour of obikmerge::partition_merge::merge_partition, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS /hoyt). Status: A + C implemented (2026-11-17), B deferred. See Implementation. Implementation prompt: prompts/merge_mmap_refactor.md (repository root, outside the doc tree).

Observed failures

Run Flags Failure When
1 none Bus error (SIGBUS) seconds after merge_partitions starts
2 --cpu-max 32 memory allocation of 32768 bytes failed → abort, many threads at once after ~1 h, 128/256 partitions done

Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, merge.rs:289), so at 50 % every remaining partition is smaller than one already merged. Small malloc failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from malloc means mmap/brk returned ENOMEM: vm.max_map_count (default 65 530), RLIMIT_AS, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.

Inventory: mappings held by one merge_partition call

Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).

Resource Where Maps Lifetime Needed?
Source layers, fully opened partition_merge.rs:219-227 → KmerLayer::open → MphfLayer::open (mphf_layer.rs:171) 4 per layer: MPHF (MphfOnly), unitigs.bin (direct access), evidence.bin, unitigs.bin again (LayerEvidence::open, mphf_layer.rs:100). ≈ 740 whole partition Only the MPHF. Pass 2 calls hash_batch (MPHF only, mphf_layer.rs:258) and fill_sub_matrix (matrix: Implicit for mono-genome, zero maps). Pass 1 only needs the unitigs.bin path.
Writable columns, created up front partition_merge.rs:327, :349 → PersistentBitVecBuilder::new / PersistentCompactIntVecBuilder::new set_len (sparse) + MmapMut, n_src_total × (L_dst + 1) ≈ 370 whole pass 2 Only the columns of the ≤ 2 sources being streamed (max_open = 2).
dst layers, new-layer MPHF, sequential readers various ≈ 10 variable yes

≈ 1 100 maps per partition, > 95 % idle at any instant. × ~32 concurrent partitions (--cpu-max 32) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, not measured. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is not established.

Weaknesses

  1. Source layers opened in full although merge needs MPHF + matrix only. Evidence and both unitigs.bin maps are dead weight.
  2. All sources opened up front, for the whole partition. This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
  3. All destination columns created up front, the same mismatch on the write side.
  4. Sparse files written through MmapMut. A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing.
  5. No bound on live mappings. obipipeline::throttle only bounds open read streams.
  6. (minor) obicompactvec::mmap_file::open_mmap_file validates magic and header length, not file length against n.

Side finding: merge_parallelism.md documents a MemoryBudget semaphore and --budget-fraction. obisys::MemoryBudget still exists but merge.rs no longer uses it (scheduling is PartitionRunner). That page is stale on this point.

Plan

A. Source-group streaming in pass 2 (obikmerge). Process sources in consecutive groups of G (G = 2, matching max_open). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its src_n columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through KmerLayer::new(..).unitigs_path() (valid on Empty). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.

C. In-memory column builders (obicompactvec). Under A only G × src_n × (L_dst+1) columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with write_all to the .building path, then publish. No sparse file and no MmapMut on the merge write path, so ENOSPC/quota becomes an io::Error. The on-disk format must be byte-identical to the current builders' output. Exposed through MatrixBuilder/ColBuilder so obikselect/obikfilter can adopt it later. Fixes 4.

B. Slots-only layer opening (obikindex, the authority crate). Deferred to a separate work item (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same hash_batch + fill_sub_matrix pattern without find appears in three crates: obikmerge (partition_merge.rs:489), obikfilter (partition_iter.rs:106), obikphylo (family_scan.rs:289). family_scan.rs:402 hashes variants; whether those are guaranteed members is unchecked. Fixes 1.

  • Rejected: lazy evidence inside MphfLayer. find/confirm return Option<usize>, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (find, needs evidence) vs in-layer iteration (index/index_batch, no evidence).
  • Rejected: a standalone parallel type next to KmerLayer/TypedLayer. It would duplicate count/presence detection and matrix access, and two opening paths would drift.
  • Retained: composition. A slots layer = MphfOnly + matrix, carrying hash_batch, fill_sub_matrix, n_cols. The full layer = slots layer + evidence, keeping find/confirm. KmerLayer exposes both openings (open() / e.g. open_slots()) over a single count/presence detection. Open point: MphfLayer::n currently comes from the evidence (evidence.len() / fingerprint.n()). A slots layer must take it from the matrix (n_rows; layer_meta.json for Implicit), so the two must be checked to always agree.

Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the obikindex query path and needs its own session with query performance checks.

Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.

Implementation (2026-11-17)

A — source-group streaming (obikmerge/src/partition_merge/, split into mod.rs + pass2.rs). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates sources.chunks(SRC_GROUP = 2), resuming each dst matrix, checking n_cols == n_dst_genomes + col_base (InvalidData otherwise), and appending the group's columns. Prerequisite: MatrixBuilder::n_cols (obicompactvec).

C — in-memory columns (obicompactvec). New mem_builder.rs (MemBitVecBuilder, MemCompactIntVecBuilder), add_mem_col on both matrix builders and on MatrixBuilder, and ColBuilder::MemBit/MemInt. Format writing is factored rather than duplicated: format::write_pciv_header/write_pciv_tail (also used by finalize_pciv), mmap_file::write_std_header/write_built. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses add_mem_col for source columns; add_absent_col is unchanged (never written through its mapping).

Deviations from the plan:

  • No sync_all in close. With one forced sync per column file, merge_partitions went from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. The rename still prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface at close(2), which File's drop ignores.
  • add_mem_col returns ColBuilder, not io::Result<ColBuilder>: it does no I/O.
  • The mmap count builder (PersistentCompactIntVecBuilder) writes in place, without .building; the in-memory one goes through .building + rename. Same final bytes.
  • obikmerge's matrix_builder_tests were removed: they only exercised obicompactvec::MatrixBuilder and are covered by obicompactvec/src/tests/{matrix_builder,mem_builder}.rs. tempfile dev-dependency of obikmerge removed with them.

Verification. Two golden-output oracles, recorded on the unmodified code:

  • scripts/smoke_test_merge.sh: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/--dense; frozen source indexes;
  • scripts/bench_merge_oracle.sh: 20 real specimen presence indexes, sorted dump --debug hash, 85 681 700 k-mers.

Both pass after every step. The existing global_index_presence (October binary) gives the same hash.

merge_partitions wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.

Live mappings per partition (estimate, not measured). Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.

Source layers Dst columns Other Total
Before 186 × 4 ≈ 744 186 × 2 ≈ 372 (MmapMut) ≈ 10 ≈ 1 100
After A + C 2 × 4 = 8 0 (in RAM) ≈ 10 ≈ 20, independent of the source count

B would take the 8 down to ~2.

Follow-up — sparse pack (same day). pack_sparse_bit_matrix/pack_sparse_compact_int_matrix used to open the columnar matrix directly, which maps every column file of a layer at once: n_genomes maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single matrix.pbmx/.pcmx mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: pack went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale matrix.pbmx that predates newer columnar data is now re-packed instead of being used as is (pack_bit_matrix compares n_cols). Both oracles pass.

Known issue (found while building the oracle, not fixed)

Re-merging a merge output fails with I/O error: No such file or directory in merge_partitions, in every mode (presence sparse, presence --dense, counts). Cause, verified: finalize_indexed packs the output matrices (sparse_meta.json + dict/PFIV files, or matrix.pbmx), so presence//counts/ no longer holds meta.json + col_*. When that index is the merge base, bootstrap copies the packed layout and MatrixBuilder::resume → MatrixMeta::load finds no meta.json. Pre-existing, independent of this refactor; deferred to a separate fix (decision 2026-11-17). The synthetic oracle (scripts/smoke_test_merge.sh) therefore omits the "merge AB, then AB C D" scenario.

Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to mmap.