90 lines
11 KiB
Markdown
90 lines
11 KiB
Markdown
# Merge: mmap audit (2026-11-17)
|
|||
|
|
|
||
|
|
Audit of the memory-mapping behaviour of `obikmerge::partition_merge::merge_partition`, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS `/hoyt`). Status: **A + C implemented (2026-11-17)**, B deferred. See [Implementation](#implementation-2026-11-17). Implementation prompt: `prompts/merge_mmap_refactor.md` (repository root, outside the doc tree).
|
||
|
|
|
||
|
|
## Observed failures
|
||
|
|
|
||
|
|
| Run | Flags | Failure | When |
|
||
|
|
|---|---|---|---|
|
||
|
|
| 1 | none | `Bus error` (SIGBUS) | seconds after `merge_partitions` starts |
|
||
|
|
| 2 | `--cpu-max 32` | `memory allocation of 32768 bytes failed` → abort, many threads at once | after ~1 h, 128/256 partitions done |
|
||
|
|
|
||
|
|
Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, `merge.rs:289`), so at 50 % every remaining partition is smaller than one already merged. Small `malloc` failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from `malloc` means `mmap`/`brk` returned `ENOMEM`: `vm.max_map_count` (default 65 530), `RLIMIT_AS`, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.
|
||
|
|
|
||
|
|
## Inventory: mappings held by one `merge_partition` call
|
||
|
|
|
||
|
|
Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).
|
||
|
|
|
||
|
|
| Resource | Where | Maps | Lifetime | Needed? |
|
||
|
|
|---|---|---|---|---|
|
||
|
|
| Source layers, fully opened | `partition_merge.rs:219-227` → `KmerLayer::open` → `MphfLayer::open` (`mphf_layer.rs:171`) | 4 per layer: MPHF (`MphfOnly`), `unitigs.bin` (direct access), `evidence.bin`, `unitigs.bin` again (`LayerEvidence::open`, `mphf_layer.rs:100`). ≈ **740** | whole partition | Only the MPHF. Pass 2 calls `hash_batch` (MPHF only, `mphf_layer.rs:258`) and `fill_sub_matrix` (matrix: `Implicit` for mono-genome, zero maps). Pass 1 only needs the `unitigs.bin` *path*. |
|
||
|
|
| Writable columns, created up front | `partition_merge.rs:327`, `:349` → `PersistentBitVecBuilder::new` / `PersistentCompactIntVecBuilder::new` | `set_len` (sparse) + `MmapMut`, `n_src_total × (L_dst + 1)` ≈ **370** | whole pass 2 | Only the columns of the ≤ 2 sources being streamed (`max_open = 2`). |
|
||
|
|
| dst layers, new-layer MPHF, sequential readers | various | ≈ 10 | variable | yes |
|
||
|
|
|
||
|
|
≈ **1 100 maps per partition**, > 95 % idle at any instant. × ~32 concurrent partitions (`--cpu-max 32`) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, **not measured**. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is **not established**.
|
||
|
|
|
||
|
|
## Weaknesses
|
||
|
|
|
||
|
|
1. **Source layers opened in full** although merge needs MPHF + matrix only. Evidence and both `unitigs.bin` maps are dead weight.
|
||
|
|
2. **All sources opened up front, for the whole partition.** This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
|
||
|
|
3. **All destination columns created up front**, the same mismatch on the write side.
|
||
|
|
4. **Sparse files written through `MmapMut`.** A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing.
|
||
|
|
5. **No bound on live mappings.** `obipipeline::throttle` only bounds open read streams.
|
||
|
|
6. *(minor)* `obicompactvec::mmap_file::open_mmap_file` validates magic and header length, not file length against `n`.
|
||
|
|
|
||
|
|
Side finding: `merge_parallelism.md` documents a `MemoryBudget` semaphore and `--budget-fraction`. `obisys::MemoryBudget` still exists but `merge.rs` no longer uses it (scheduling is `PartitionRunner`). That page is stale on this point.
|
||
|
|
|
||
|
|
## Plan
|
||
|
|
|
||
|
|
**A. Source-group streaming in pass 2** (`obikmerge`). Process sources in consecutive groups of `G` (G = 2, matching `max_open`). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its `src_n` columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through `KmerLayer::new(..).unitigs_path()` (valid on `Empty`). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.
|
||
|
|
|
||
|
|
**C. In-memory column builders** (`obicompactvec`). Under A only `G × src_n × (L_dst+1)` columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with `write_all` to the `.building` path, then publish. No sparse file and no `MmapMut` on the merge write path, so `ENOSPC`/quota becomes an `io::Error`. The on-disk format must be byte-identical to the current builders' output. Exposed through `MatrixBuilder`/`ColBuilder` so `obikselect`/`obikfilter` can adopt it later. Fixes 4.
|
||
|
|
|
||
|
|
**B. Slots-only layer opening** (`obikindex`, the authority crate). **Deferred to a separate work item** (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same `hash_batch` + `fill_sub_matrix` pattern without `find` appears in three crates: `obikmerge` (`partition_merge.rs:489`), `obikfilter` (`partition_iter.rs:106`), `obikphylo` (`family_scan.rs:289`). `family_scan.rs:402` hashes `variants`; whether those are guaranteed members is unchecked. Fixes 1.
|
||
|
|
|
||
|
|
- **Rejected: lazy evidence inside `MphfLayer`.** `find`/`confirm` return `Option<usize>`, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (`find`, needs evidence) vs in-layer iteration (`index`/`index_batch`, no evidence).
|
||
|
|
- **Rejected: a standalone parallel type** next to `KmerLayer`/`TypedLayer`. It would duplicate count/presence detection and matrix access, and two opening paths would drift.
|
||
|
|
- **Retained: composition.** A slots layer = `MphfOnly` + matrix, carrying `hash_batch`, `fill_sub_matrix`, `n_cols`. The full layer = slots layer + evidence, keeping `find`/`confirm`. `KmerLayer` exposes both openings (`open()` / e.g. `open_slots()`) over a single count/presence detection. Open point: `MphfLayer::n` currently comes from the evidence (`evidence.len()` / `fingerprint.n()`). A slots layer must take it from the matrix (`n_rows`; `layer_meta.json` for `Implicit`), so the two must be checked to always agree.
|
||
|
|
|
||
|
|
Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the `obikindex` query path and needs its own session with `query` performance checks.
|
||
|
|
|
||
|
|
Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.
|
||
|
|
|
||
|
|
## Implementation (2026-11-17)
|
||
|
|
|
||
|
|
**A — source-group streaming** (`obikmerge/src/partition_merge/`, split into `mod.rs` + `pass2.rs`). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates `sources.chunks(SRC_GROUP = 2)`, resuming each dst matrix, checking `n_cols == n_dst_genomes + col_base` (`InvalidData` otherwise), and appending the group's columns. Prerequisite: `MatrixBuilder::n_cols` (`obicompactvec`).
|
||
|
|
|
||
|
|
**C — in-memory columns** (`obicompactvec`). New `mem_builder.rs` (`MemBitVecBuilder`, `MemCompactIntVecBuilder`), `add_mem_col` on both matrix builders and on `MatrixBuilder`, and `ColBuilder::MemBit`/`MemInt`. Format writing is factored rather than duplicated: `format::write_pciv_header`/`write_pciv_tail` (also used by `finalize_pciv`), `mmap_file::write_std_header`/`write_built`. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses `add_mem_col` for source columns; `add_absent_col` is unchanged (never written through its mapping).
|
||
|
|
|
||
|
|
Deviations from the plan:
|
||
|
|
|
||
|
|
- **No `sync_all` in `close`.** With one forced sync per column file, `merge_partitions` went from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. The `rename` still prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface at `close(2)`, which `File`'s drop ignores.
|
||
|
|
- `add_mem_col` returns `ColBuilder`, not `io::Result<ColBuilder>`: it does no I/O.
|
||
|
|
- The mmap count builder (`PersistentCompactIntVecBuilder`) writes in place, without `.building`; the in-memory one goes through `.building` + `rename`. Same final bytes.
|
||
|
|
- `obikmerge`'s `matrix_builder_tests` were removed: they only exercised `obicompactvec::MatrixBuilder` and are covered by `obicompactvec/src/tests/{matrix_builder,mem_builder}.rs`. `tempfile` dev-dependency of `obikmerge` removed with them.
|
||
|
|
|
||
|
|
**Verification.** Two golden-output oracles, recorded on the unmodified code:
|
||
|
|
|
||
|
|
- `scripts/smoke_test_merge.sh`: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/`--dense`; frozen source indexes;
|
||
|
|
- `scripts/bench_merge_oracle.sh`: 20 real specimen presence indexes, sorted `dump --debug` hash, 85 681 700 k-mers.
|
||
|
|
|
||
|
|
Both pass after every step. The existing `global_index_presence` (October binary) gives the same hash.
|
||
|
|
|
||
|
|
`merge_partitions` wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.
|
||
|
|
|
||
|
|
**Live mappings per partition (estimate, not measured).** Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.
|
||
|
|
|
||
|
|
| | Source layers | Dst columns | Other | Total |
|
||
|
|
|---|---|---|---|---|
|
||
|
|
| Before | 186 × 4 ≈ 744 | 186 × 2 ≈ 372 (`MmapMut`) | ≈ 10 | **≈ 1 100** |
|
||
|
|
| After A + C | 2 × 4 = 8 | 0 (in RAM) | ≈ 10 | **≈ 20**, independent of the source count |
|
||
|
|
|
||
|
|
B would take the 8 down to ~2.
|
||
|
|
|
||
|
|
Not covered here, and possibly the next limit: the sparse `pack` at finalize maps every column of a layer at once (see `merge_scaling.md`).
|
||
|
|
|
||
|
|
## Known issue (found while building the oracle, not fixed)
|
||
|
|
|
||
|
|
Re-merging a merge output fails with `I/O error: No such file or directory` in `merge_partitions`, in every mode (presence sparse, presence `--dense`, counts). Cause, verified: `finalize_indexed` packs the output matrices (`sparse_meta.json` + dict/PFIV files, or `matrix.pbmx`), so `presence/`/`counts/` no longer holds `meta.json` + `col_*`. When that index is the merge base, bootstrap copies the packed layout and `MatrixBuilder::resume` → `MatrixMeta::load` finds no `meta.json`. Pre-existing, independent of this refactor; deferred to a separate fix (decision 2026-11-17). The synthetic oracle (`scripts/smoke_test_merge.sh`) therefore omits the "merge AB, then AB C D" scenario.
|
||
|
|
|
||
|
|
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to `mmap`.
|