Update documentation to explain that merged indices can serve as sources if at least one source is unmerged, and that the resulting column order follows the base index first. Clarify that explicitly named files are always read regardless of extension, with format detection based on content. - refactor: prevent I/O errors when re-merging packed outputs by restricting `choose_base` to valid destinations via new `can_be_destination` check - fix: handle zero-layer partitions in `build_index_layer` and `remove_build_artifacts` to skip unnecessary operations - feat: allow `PathIter` to treat unrecognized extensions as explicit files, ensuring they are not silently dropped during directory expansion - test: add comprehensive tests for `PathIter` including extension recognition, explicit file handling, and stdin sentinel behavior - test: update smoke tests to verify empty partitions, `.fna.gz` indexing, and refusal of merging only packed indexes - chore: remove obsolete cargo build environment variables from `run.sh` script
13 KiB
Merge: mmap audit (2026-11-17)
Audit of the memory-mapping behaviour of obikmerge::partition_merge::merge_partition, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS /hoyt). Status: A + C implemented (2026-11-17), B deferred. See Implementation. Implementation prompt: prompts/merge_mmap_refactor.md (repository root, outside the doc tree).
Observed failures
| Run | Flags | Failure | When |
|---|---|---|---|
| 1 | none | Bus error (SIGBUS) |
seconds after merge_partitions starts |
| 2 | --cpu-max 32 |
memory allocation of 32768 bytes failed → abort, many threads at once |
after ~1 h, 128/256 partitions done |
Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, merge.rs:289), so at 50 % every remaining partition is smaller than one already merged. Small malloc failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from malloc means mmap/brk returned ENOMEM: vm.max_map_count (default 65 530), RLIMIT_AS, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.
Inventory: mappings held by one merge_partition call
Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).
| Resource | Where | Maps | Lifetime | Needed? |
|---|---|---|---|---|
| Source layers, fully opened | partition_merge.rs:219-227 → KmerLayer::open → MphfLayer::open (mphf_layer.rs:171) |
4 per layer: MPHF (MphfOnly), unitigs.bin (direct access), evidence.bin, unitigs.bin again (LayerEvidence::open, mphf_layer.rs:100). ≈ 740 |
whole partition | Only the MPHF. Pass 2 calls hash_batch (MPHF only, mphf_layer.rs:258) and fill_sub_matrix (matrix: Implicit for mono-genome, zero maps). Pass 1 only needs the unitigs.bin path. |
| Writable columns, created up front | partition_merge.rs:327, :349 → PersistentBitVecBuilder::new / PersistentCompactIntVecBuilder::new |
set_len (sparse) + MmapMut, n_src_total × (L_dst + 1) ≈ 370 |
whole pass 2 | Only the columns of the ≤ 2 sources being streamed (max_open = 2). |
| dst layers, new-layer MPHF, sequential readers | various | ≈ 10 | variable | yes |
≈ 1 100 maps per partition, > 95 % idle at any instant. × ~32 concurrent partitions (--cpu-max 32) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, not measured. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is not established.
Weaknesses
- Source layers opened in full although merge needs MPHF + matrix only. Evidence and both
unitigs.binmaps are dead weight. - All sources opened up front, for the whole partition. This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
- All destination columns created up front, the same mismatch on the write side.
- Sparse files written through
MmapMut. A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing. - No bound on live mappings.
obipipeline::throttleonly bounds open read streams. - (minor)
obicompactvec::mmap_file::open_mmap_filevalidates magic and header length, not file length againstn.
Side finding: merge_parallelism.md documents a MemoryBudget semaphore and --budget-fraction. obisys::MemoryBudget still exists but merge.rs no longer uses it (scheduling is PartitionRunner). That page is stale on this point.
Plan
A. Source-group streaming in pass 2 (obikmerge). Process sources in consecutive groups of G (G = 2, matching max_open). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its src_n columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through KmerLayer::new(..).unitigs_path() (valid on Empty). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.
C. In-memory column builders (obicompactvec). Under A only G × src_n × (L_dst+1) columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with write_all to the .building path, then publish. No sparse file and no MmapMut on the merge write path, so ENOSPC/quota becomes an io::Error. The on-disk format must be byte-identical to the current builders' output. Exposed through MatrixBuilder/ColBuilder so obikselect/obikfilter can adopt it later. Fixes 4.
B. Slots-only layer opening (obikindex, the authority crate). Deferred to a separate work item (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same hash_batch + fill_sub_matrix pattern without find appears in three crates: obikmerge (partition_merge.rs:489), obikfilter (partition_iter.rs:106), obikphylo (family_scan.rs:289). family_scan.rs:402 hashes variants; whether those are guaranteed members is unchecked. Fixes 1.
- Rejected: lazy evidence inside
MphfLayer.find/confirmreturnOption<usize>, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (find, needs evidence) vs in-layer iteration (index/index_batch, no evidence). - Rejected: a standalone parallel type next to
KmerLayer/TypedLayer. It would duplicate count/presence detection and matrix access, and two opening paths would drift. - Retained: composition. A slots layer =
MphfOnly+ matrix, carryinghash_batch,fill_sub_matrix,n_cols. The full layer = slots layer + evidence, keepingfind/confirm.KmerLayerexposes both openings (open()/ e.g.open_slots()) over a single count/presence detection. Open point:MphfLayer::ncurrently comes from the evidence (evidence.len()/fingerprint.n()). A slots layer must take it from the matrix (n_rows;layer_meta.jsonforImplicit), so the two must be checked to always agree.
Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the obikindex query path and needs its own session with query performance checks.
Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.
Implementation (2026-11-17)
A — source-group streaming (obikmerge/src/partition_merge/, split into mod.rs + pass2.rs). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates sources.chunks(SRC_GROUP = 2), resuming each dst matrix, checking n_cols == n_dst_genomes + col_base (InvalidData otherwise), and appending the group's columns. Prerequisite: MatrixBuilder::n_cols (obicompactvec).
C — in-memory columns (obicompactvec). New mem_builder.rs (MemBitVecBuilder, MemCompactIntVecBuilder), add_mem_col on both matrix builders and on MatrixBuilder, and ColBuilder::MemBit/MemInt. Format writing is factored rather than duplicated: format::write_pciv_header/write_pciv_tail (also used by finalize_pciv), mmap_file::write_std_header/write_built. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses add_mem_col for source columns; add_absent_col is unchanged (never written through its mapping).
Deviations from the plan:
- No
sync_allinclose. With one forced sync per column file,merge_partitionswent from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. Therenamestill prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface atclose(2), whichFile's drop ignores. add_mem_colreturnsColBuilder, notio::Result<ColBuilder>: it does no I/O.- The mmap count builder (
PersistentCompactIntVecBuilder) writes in place, without.building; the in-memory one goes through.building+rename. Same final bytes. obikmerge'smatrix_builder_testswere removed: they only exercisedobicompactvec::MatrixBuilderand are covered byobicompactvec/src/tests/{matrix_builder,mem_builder}.rs.tempfiledev-dependency ofobikmergeremoved with them.
Verification. Two golden-output oracles, recorded on the unmodified code:
scripts/smoke_test_merge.sh: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/--dense; frozen source indexes;scripts/bench_merge_oracle.sh: 20 real specimen presence indexes, sorteddump --debughash, 85 681 700 k-mers.
Both pass after every step. The existing global_index_presence (October binary) gives the same hash.
merge_partitions wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.
Live mappings per partition (estimate, not measured). Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.
| Source layers | Dst columns | Other | Total | |
|---|---|---|---|---|
| Before | 186 × 4 ≈ 744 | 186 × 2 ≈ 372 (MmapMut) |
≈ 10 | ≈ 1 100 |
| After A + C | 2 × 4 = 8 | 0 (in RAM) | ≈ 10 | ≈ 20, independent of the source count |
B would take the 8 down to ~2.
Follow-up — sparse pack (same day). pack_sparse_bit_matrix/pack_sparse_compact_int_matrix used to open the columnar matrix directly, which maps every column file of a layer at once: n_genomes maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single matrix.pbmx/.pcmx mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: pack went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale matrix.pbmx that predates newer columnar data is now re-packed instead of being used as is (pack_bit_matrix compares n_cols). Both oracles pass.
Re-merging a merge output (resolved 2026-11-18)
Symptom: I/O error: No such file or directory in merge_partitions when a merge output was one of the sources. Cause (verified): finalize_indexed packs the output matrices (sparse_meta.json + dict/PFIV files, or matrix.pbmx), so presence//counts/ no longer holds meta.json + col_*. choose_base picks the largest source as destination, i.e. the merged one; bootstrap copies the packed layout and MatrixBuilder::resume → MatrixMeta::load finds no meta.json.
Fix: a packed index can only be a source (pass 2 reads it through PersistentBitMatrix/PersistentIntMatrix::open, which handle every layout). choose_base now filters candidates with can_be_destination (obikmerge/src/merge.rs): one layer is probed with detect_storage; Columnar qualifies, Implicit only for a single genome (materialized on the fly), Packed/Sparse never. With a single source there is nothing to append and the filter is skipped. When several sources are given and none qualifies, run returns an explicit error before any work. Consequence: the result's column order follows the chosen base first, so it can differ from the command-line order.
Not done: in-place unpack of a packed index to allow it as destination, or a destination-side append on the packed format — the sparse incremental strategy is tracked in merge_scaling.md.
Oracle: scripts/smoke_test_merge.sh scenario 2 (A B → AB, then AB C D) is compared against the flat merge by genome label (not against the golden dumps, whose layer placement differs), and merging only packed indexes is checked to be refused.
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to mmap.