Push txuyrtlmyoyl #81
No files matched your search
@@ -62,52 +62,38 @@ fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
|
||||
|
||||
Check all sources against the constraints above. Abort on any mismatch.
|
||||
|
||||
### 2. Bootstrap output from first source
|
||||
### 2. Bootstrap output from the base source
|
||||
|
||||
Recursive file copy of `sources[0]` → `output`. Immediately after the copy:
|
||||
The base is chosen by `choose_base` (`merge.rs`): the source with the largest total `unitigs.bin` size (restricted to approximate-evidence sources when the output must be approximate). It is not necessarily `sources[0]`, and the genome columns follow the base first. Because `obikmer index` is not byte-reproducible (unitig segmentation varies), two near-equal sources can swap roles between runs. Its directory is copied to `output` (hard links for files `merge_partition` never rewrites, see below). Immediately after the copy:
|
||||
|
||||
- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
|
||||
- In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
|
||||
- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
|
||||
|
||||
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
|
||||
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The base source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
|
||||
|
||||
### 3. For each subsequent source (parallel across partitions)
|
||||
|
||||
`KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)` is called for each partition index `i`. `block_bits` is taken from `dst.meta.config.block_bits`.
|
||||
`obikmerge::partition_merge::merge_partition(dst, i, sources, mode, n_dst_genomes, block_bits, evidence)` runs once per partition index `i` (scheduled largest first by `PartitionRunner`). Each entry of `sources` is `(&KmerIndex, n_genomes)`, where `n_genomes` is the number of columns that source contributes (> 1 for an already-merged source). Code: `partition_merge/mod.rs` (orchestration, pass 1) and `partition_merge/pass2.rs` (per-group fill pipeline).
|
||||
|
||||
Each entry in `sources` is `(&KmerPartition, n_genomes)` where `n_genomes` is the column count that source contributes (> 1 when the source is itself a merged index).
|
||||
**First merge, Presence mode**: when `n_dst_genomes == 1`, `TypedLayer::<()>::init_presence_matrix` is called on every existing dst layer first. It materialises the implicit single-genome matrix (`presence/col_000000.pbiv`, all true) so that source columns can be appended to it.
|
||||
|
||||
**First merge, Presence mode**: when `n_dst_genomes == 1`, `Layer::<()>::init_presence_matrix` is called on every existing destination layer before any source column is appended. This creates `presence/col_000000.pbiv` set all-true (genome 0 is present in every slot).
|
||||
The dst layers are opened once per partition and shared by both passes.
|
||||
|
||||
**Pass 1 — classify kmers**
|
||||
**Pass 1 — new k-mers into a de Bruijn graph.** Only the sources' `unitigs.bin` *paths* are needed (`KmerLayer::new(..).unitigs_path()`); no source layer is opened. The files are streamed (at most `max_open = 2` open at once), and every k-mer absent from all dst layers is pushed into one `GraphDeBruijn`. All new k-mers of all sources go into a **single** new layer per merge, built by `materialize_layer` (unitigs + MPHF + evidence).
|
||||
|
||||
Iterate all kmers from all source partitions (via `UnitigFileReader` + canonical kmer iteration). For each kmer, probe the destination `LayeredMap<()>`:
|
||||
**New-layer matrix.** If a new layer exists, its matrix is created with `n_dst_genomes` absent columns (all zero/false) and closed immediately. This keeps the column-count invariant, and every source group then appends to it like to the existing layers.
|
||||
|
||||
- **Hit**: kmer already in the destination; record for Pass 2.
|
||||
- **Miss**: push kmer into a `GraphDeBruijn` accumulator.
|
||||
**Pass 2 — by consecutive groups of `SRC_GROUP = 2` sources, in source order.** Per group:
|
||||
|
||||
**New layer construction**
|
||||
1. open the group's source layers for this partition;
|
||||
2. `MatrixBuilder::resume` every dst matrix (existing layers + new layer), check `n_cols == n_dst_genomes + col_base`, append the group's `group_n` columns;
|
||||
3. run the pipeline: stream each source layer's k-mers, `hash_batch` + `fill_sub_matrix` on the source, locate each k-mer in the dst layers (`find_batch_grouped`) or else in the new layer's MPHF, and write the values column-major into the group's builders;
|
||||
4. close the columns, then the matrices; drop the group's layers; `col_base += group_n`.
|
||||
|
||||
If the accumulator is non-empty, compute de Bruijn unitigs and call `Layer::<()>::build(&new_layer_dir, block_bits)`. All kmers absent from the destination — across **all** sources — accumulate into a **single** graph, producing one new layer per merge operation (not one per source).
|
||||
Only one group's source layers and destination columns are alive at a time, so the number of live mappings per partition does not depend on the number of sources. Columns are appended in source order, so every genome lands in the same absolute column as in a single-session merge.
|
||||
|
||||
**Pass 2 — fill column builders**
|
||||
|
||||
For each source and each of its layers, re-iterate unitigs and look up stored values via `SrcLayerData::lookup(kmer, src_n)`:
|
||||
|
||||
- `SrcLayerData::SetMembership` — no data directory exists; every kmer returns `vec![1; n_genomes]`
|
||||
- `SrcLayerData::Presence` — reads `PersistentBitMatrix` from `presence/`
|
||||
- `SrcLayerData::Count` — reads `PersistentCompactIntMatrix` from `counts/`
|
||||
|
||||
Hits are routed to `exist_builders[dst_layer][src_col]`; misses are routed to `new_src_builders[src_col]`.
|
||||
|
||||
**Column prepending for new layers**
|
||||
|
||||
Before source columns are written to the new layer, `n_dst_genomes` absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.
|
||||
|
||||
**Close and update metadata**
|
||||
|
||||
Close all builders; update `presence/meta.json` or `counts/meta.json` with `{"n": N, "n_cols": n_dst_genomes + n_src_total}`; increment `PartitionMeta::n_layers` if a new layer was added.
|
||||
**In-memory columns.** The group's columns are built in RAM (`MatrixBuilder::add_mem_col` → `MemBitVecBuilder` / `MemCompactIntVecBuilder`) and written once, sequentially, at `close` (`.building` file, then `rename`). The files are byte-identical to the former mmap-backed builders' output. There is no sparse file and no `MmapMut` on this path, so a block the FS cannot allocate is an `io::Error` instead of a SIGBUS. Files are not `fsync`ed: one sync per column file made merge ~8× slower. RAM per partition is at most `SRC_GROUP × src_n × N_part × (1/8 | 1)` bytes (presence | counts). History and figures: `merge_mmap_audit.md`; scaling limits: `merge_scaling.md`.
|
||||
|
||||
### 4. Update index metadata
|
||||
|
||||
@@ -129,7 +115,7 @@ impl Layer<PersistentBitMatrix> {
|
||||
}
|
||||
```
|
||||
|
||||
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written directly via `PersistentBitVecBuilder` / `PersistentCompactIntVecBuilder` rather than through these helpers, but the invariant they enforce is the same.
|
||||
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written through `MatrixBuilder::add_mem_col` (in-memory builders) rather than through these helpers, but the invariant they enforce is the same.
|
||||
|
||||
---
|
||||
|
||||
@@ -139,7 +125,7 @@ After any merge, **every layer in every partition has exactly `n_genomes` column
|
||||
|
||||
Maintained by three mechanisms:
|
||||
|
||||
1. **Existing layers**: `n_src_total` columns appended (one per source genome).
|
||||
1. **Existing layers**: `n_src_total` columns appended (one per source genome), group by group.
|
||||
2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
|
||||
3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
|
||||
|
||||
@@ -161,7 +147,7 @@ The invariant is a precondition of `LayeredStore` aggregation traits: `col_weigh
|
||||
|
||||
## On-disk impact
|
||||
|
||||
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest):
|
||||
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest), *before* `finalize` packs each matrix (sparse by default, `matrix.pbmx` with `--dense`):
|
||||
|
||||
```
|
||||
partitions/
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# Merge: mmap audit (2026-11-17)
|
||||
|
||||
Audit of the memory-mapping behaviour of `obikmerge::partition_merge::merge_partition`, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS `/hoyt`). Status: **A + C implemented (2026-11-17)**, B deferred. See [Implementation](#implementation-2026-11-17). Implementation prompt: `prompts/merge_mmap_refactor.md` (repository root, outside the doc tree).
|
||||
|
||||
## Observed failures
|
||||
|
||||
| Run | Flags | Failure | When |
|
||||
|---|---|---|---|
|
||||
| 1 | none | `Bus error` (SIGBUS) | seconds after `merge_partitions` starts |
|
||||
| 2 | `--cpu-max 32` | `memory allocation of 32768 bytes failed` → abort, many threads at once | after ~1 h, 128/256 partitions done |
|
||||
|
||||
Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, `merge.rs:289`), so at 50 % every remaining partition is smaller than one already merged. Small `malloc` failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from `malloc` means `mmap`/`brk` returned `ENOMEM`: `vm.max_map_count` (default 65 530), `RLIMIT_AS`, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.
|
||||
|
||||
## Inventory: mappings held by one `merge_partition` call
|
||||
|
||||
Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).
|
||||
|
||||
| Resource | Where | Maps | Lifetime | Needed? |
|
||||
|---|---|---|---|---|
|
||||
| Source layers, fully opened | `partition_merge.rs:219-227` → `KmerLayer::open` → `MphfLayer::open` (`mphf_layer.rs:171`) | 4 per layer: MPHF (`MphfOnly`), `unitigs.bin` (direct access), `evidence.bin`, `unitigs.bin` again (`LayerEvidence::open`, `mphf_layer.rs:100`). ≈ **740** | whole partition | Only the MPHF. Pass 2 calls `hash_batch` (MPHF only, `mphf_layer.rs:258`) and `fill_sub_matrix` (matrix: `Implicit` for mono-genome, zero maps). Pass 1 only needs the `unitigs.bin` *path*. |
|
||||
| Writable columns, created up front | `partition_merge.rs:327`, `:349` → `PersistentBitVecBuilder::new` / `PersistentCompactIntVecBuilder::new` | `set_len` (sparse) + `MmapMut`, `n_src_total × (L_dst + 1)` ≈ **370** | whole pass 2 | Only the columns of the ≤ 2 sources being streamed (`max_open = 2`). |
|
||||
| dst layers, new-layer MPHF, sequential readers | various | ≈ 10 | variable | yes |
|
||||
|
||||
≈ **1 100 maps per partition**, > 95 % idle at any instant. × ~32 concurrent partitions (`--cpu-max 32`) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, **not measured**. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is **not established**.
|
||||
|
||||
## Weaknesses
|
||||
|
||||
1. **Source layers opened in full** although merge needs MPHF + matrix only. Evidence and both `unitigs.bin` maps are dead weight.
|
||||
2. **All sources opened up front, for the whole partition.** This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
|
||||
3. **All destination columns created up front**, the same mismatch on the write side.
|
||||
4. **Sparse files written through `MmapMut`.** A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing.
|
||||
5. **No bound on live mappings.** `obipipeline::throttle` only bounds open read streams.
|
||||
6. *(minor)* `obicompactvec::mmap_file::open_mmap_file` validates magic and header length, not file length against `n`.
|
||||
|
||||
Side finding: `merge_parallelism.md` documents a `MemoryBudget` semaphore and `--budget-fraction`. `obisys::MemoryBudget` still exists but `merge.rs` no longer uses it (scheduling is `PartitionRunner`). That page is stale on this point.
|
||||
|
||||
## Plan
|
||||
|
||||
**A. Source-group streaming in pass 2** (`obikmerge`). Process sources in consecutive groups of `G` (G = 2, matching `max_open`). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its `src_n` columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through `KmerLayer::new(..).unitigs_path()` (valid on `Empty`). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.
|
||||
|
||||
**C. In-memory column builders** (`obicompactvec`). Under A only `G × src_n × (L_dst+1)` columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with `write_all` to the `.building` path, then publish. No sparse file and no `MmapMut` on the merge write path, so `ENOSPC`/quota becomes an `io::Error`. The on-disk format must be byte-identical to the current builders' output. Exposed through `MatrixBuilder`/`ColBuilder` so `obikselect`/`obikfilter` can adopt it later. Fixes 4.
|
||||
|
||||
**B. Slots-only layer opening** (`obikindex`, the authority crate). **Deferred to a separate work item** (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same `hash_batch` + `fill_sub_matrix` pattern without `find` appears in three crates: `obikmerge` (`partition_merge.rs:489`), `obikfilter` (`partition_iter.rs:106`), `obikphylo` (`family_scan.rs:289`). `family_scan.rs:402` hashes `variants`; whether those are guaranteed members is unchecked. Fixes 1.
|
||||
|
||||
- **Rejected: lazy evidence inside `MphfLayer`.** `find`/`confirm` return `Option<usize>`, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (`find`, needs evidence) vs in-layer iteration (`index`/`index_batch`, no evidence).
|
||||
- **Rejected: a standalone parallel type** next to `KmerLayer`/`TypedLayer`. It would duplicate count/presence detection and matrix access, and two opening paths would drift.
|
||||
- **Retained: composition.** A slots layer = `MphfOnly` + matrix, carrying `hash_batch`, `fill_sub_matrix`, `n_cols`. The full layer = slots layer + evidence, keeping `find`/`confirm`. `KmerLayer` exposes both openings (`open()` / e.g. `open_slots()`) over a single count/presence detection. Open point: `MphfLayer::n` currently comes from the evidence (`evidence.len()` / `fingerprint.n()`). A slots layer must take it from the matrix (`n_rows`; `layer_meta.json` for `Implicit`), so the two must be checked to always agree.
|
||||
|
||||
Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the `obikindex` query path and needs its own session with `query` performance checks.
|
||||
|
||||
Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.
|
||||
|
||||
## Implementation (2026-11-17)
|
||||
|
||||
**A — source-group streaming** (`obikmerge/src/partition_merge/`, split into `mod.rs` + `pass2.rs`). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates `sources.chunks(SRC_GROUP = 2)`, resuming each dst matrix, checking `n_cols == n_dst_genomes + col_base` (`InvalidData` otherwise), and appending the group's columns. Prerequisite: `MatrixBuilder::n_cols` (`obicompactvec`).
|
||||
|
||||
**C — in-memory columns** (`obicompactvec`). New `mem_builder.rs` (`MemBitVecBuilder`, `MemCompactIntVecBuilder`), `add_mem_col` on both matrix builders and on `MatrixBuilder`, and `ColBuilder::MemBit`/`MemInt`. Format writing is factored rather than duplicated: `format::write_pciv_header`/`write_pciv_tail` (also used by `finalize_pciv`), `mmap_file::write_std_header`/`write_built`. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses `add_mem_col` for source columns; `add_absent_col` is unchanged (never written through its mapping).
|
||||
|
||||
Deviations from the plan:
|
||||
|
||||
- **No `sync_all` in `close`.** With one forced sync per column file, `merge_partitions` went from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. The `rename` still prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface at `close(2)`, which `File`'s drop ignores.
|
||||
- `add_mem_col` returns `ColBuilder`, not `io::Result<ColBuilder>`: it does no I/O.
|
||||
- The mmap count builder (`PersistentCompactIntVecBuilder`) writes in place, without `.building`; the in-memory one goes through `.building` + `rename`. Same final bytes.
|
||||
- `obikmerge`'s `matrix_builder_tests` were removed: they only exercised `obicompactvec::MatrixBuilder` and are covered by `obicompactvec/src/tests/{matrix_builder,mem_builder}.rs`. `tempfile` dev-dependency of `obikmerge` removed with them.
|
||||
|
||||
**Verification.** Two golden-output oracles, recorded on the unmodified code:
|
||||
|
||||
- `scripts/smoke_test_merge.sh`: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/`--dense`; frozen source indexes;
|
||||
- `scripts/bench_merge_oracle.sh`: 20 real specimen presence indexes, sorted `dump --debug` hash, 85 681 700 k-mers.
|
||||
|
||||
Both pass after every step. The existing `global_index_presence` (October binary) gives the same hash.
|
||||
|
||||
`merge_partitions` wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.
|
||||
|
||||
**Live mappings per partition (estimate, not measured).** Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.
|
||||
|
||||
| | Source layers | Dst columns | Other | Total |
|
||||
|---|---|---|---|---|
|
||||
| Before | 186 × 4 ≈ 744 | 186 × 2 ≈ 372 (`MmapMut`) | ≈ 10 | **≈ 1 100** |
|
||||
| After A + C | 2 × 4 = 8 | 0 (in RAM) | ≈ 10 | **≈ 20**, independent of the source count |
|
||||
|
||||
B would take the 8 down to ~2.
|
||||
|
||||
**Follow-up — sparse pack (same day).** `pack_sparse_bit_matrix`/`pack_sparse_compact_int_matrix` used to open the columnar matrix directly, which maps every column file of a layer at once: `n_genomes` maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single `matrix.pbmx`/`.pcmx` mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: `pack` went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale `matrix.pbmx` that predates newer columnar data is now re-packed instead of being used as is (`pack_bit_matrix` compares `n_cols`). Both oracles pass.
|
||||
|
||||
## Known issue (found while building the oracle, not fixed)
|
||||
|
||||
Re-merging a merge output fails with `I/O error: No such file or directory` in `merge_partitions`, in every mode (presence sparse, presence `--dense`, counts). Cause, verified: `finalize_indexed` packs the output matrices (`sparse_meta.json` + dict/PFIV files, or `matrix.pbmx`), so `presence/`/`counts/` no longer holds `meta.json` + `col_*`. When that index is the merge base, bootstrap copies the packed layout and `MatrixBuilder::resume` → `MatrixMeta::load` finds no `meta.json`. Pre-existing, independent of this refactor; deferred to a separate fix (decision 2026-11-17). The synthetic oracle (`scripts/smoke_test_merge.sh`) therefore omits the "merge AB, then AB C D" scenario.
|
||||
|
||||
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to `mmap`.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Merge: scaling to thousands of genomes (open, 2026-11-17)
|
||||
|
||||
Open design question raised during the mmap refactor (`merge_mmap_audit.md`). Target: merges of ~6 000 genomes of a few Gb each. Nothing here is implemented or measured. Figures are estimates, labelled as such.
|
||||
|
||||
## What scales with what (after the mmap refactor)
|
||||
|
||||
| Cost | Depends on | Status |
|
||||
|---|---|---|
|
||||
| Live source layers + dst columns per partition (merge pass 2) | `SRC_GROUP × src_n`, not the source count | fixed by plan A |
|
||||
| Pass-2 column RAM | `SRC_GROUP × src_n × N_part × (1/8 \| 1)` B | bounded, see below |
|
||||
| Pass-1 de Bruijn graph | ≥ ~19 B per *new* k-mer (`HashMap<u64, AtomicU8>`, verified in `obidebruinj/src/debruijn/graph.rs`) | unchanged |
|
||||
| Columnar intermediate files | `n_genomes × n_layers × n_partitions` (6 000 × 2 × 256 ≈ 3 M files, estimate) | **does not scale** |
|
||||
| Sparse `pack` | used to map **all** columns of a layer at once (`ColumnarBitMatrix::open`); layers are packed one after another, so `n_genomes` maps per partition (186 × 192 concurrent ≈ 36 k, estimate) | **fixed 2026-11-17**: now goes columnar → dense `matrix.pbmx` (sequential concatenation, unmapped) → sparse, one mapping per partition |
|
||||
|
||||
`N_part`: k-mers of one dst partition over all layers. Pass-2 column RAM stays below pass-1 graph RAM for one-shot merges. It dominates only for **incremental** merges, where few genomes are added to a large index: the new layer is small, the existing layers are large. Example (estimate): `N_total = 1e12`, counts, 192 concurrent partitions → ~1.5 TB. A memory-budgeted concurrency cap is needed there whatever the column strategy. `obisys::MemoryBudget` exists but is no longer used by merge.
|
||||
|
||||
## Why a columnar intermediate at all
|
||||
|
||||
The sparse format (`bitmatrix/sparse.rs`) is row-major and deduplicated: per slot, either one genome (`singleton.pfiv`) or a `dict_id` (`multi.pfiv`) into a dictionary of distinct genome sets (`dict_offsets` + `dict_values.bin`). Its builder only offers `push_row(genomes)`, i.e. the **complete** genome set of a row, rows in **slot order**.
|
||||
|
||||
Merge discovers values **column by column** (source by source), in **random slot order** (MPHF order). The columnar files are a transposition buffer: they absorb random per-column writes, and `pack` transposes them row by row.
|
||||
|
||||
## Options to build the final format directly
|
||||
|
||||
1. **All sources at once.** That is exactly what plan A removed. No.
|
||||
2. **Whole matrix in RAM.** `N_part × n_genomes / 8` B per partition: ~3 TB at 6 000 genomes and `N_part = 4e9` (estimate). No.
|
||||
3. **External sort of `(slot, genome)` pairs.** Volume ∝ non-zeros ≈ Σ genome k-mers (~6 000 × 3e9). That is huge, while the final format is small precisely because genome sets repeat. No.
|
||||
4. **Incremental sparse update per source group** (most promising). Per group:
|
||||
- fill the group's columns in RAM (as now);
|
||||
- stream the partition's existing sparse matrix **sequentially in slot order**;
|
||||
- for each row: new set = old set ∪ group bits. Through the dictionary this is a transition `(old dict_id, group bit pattern) → new dict_id`, which can be memoised in a table;
|
||||
- write the new sparse matrix sequentially, then publish.
|
||||
|
||||
No columnar files, no `pack`, no per-genome mapping, sequential I/O only. It also removes the re-merge bug: it starts from an existing sparse matrix instead of requiring a columnar one.
|
||||
|
||||
The cost is **one full pass over the partition's sparse matrix per group**. With `SRC_GROUP = 2` and 6 000 genomes that is 3 000 rewrites, far too many. Groups must therefore be large (64–256 sources), and group column RAM grows with them: `group × N_part / 8` B, e.g. 32 GB per partition for 64 sources at `N_part = 4e9` (estimate). One parameter trades passes against RAM, and it should be sized from a memory budget. The count matrix (`sparse_intmatrix.rs`) has its own format and needs the same analysis.
|
||||
|
||||
## Related findings
|
||||
|
||||
- The module header of `bitmatrix/sparse.rs` says "not used by any production code path yet". This is stale: `finalize_indexed` packs to sparse by default.
|
||||
- The new layer of a merge is not byte-reproducible run to run: unitig segmentation varies, so the MPHF, evidence and matrix row order vary with it (measured 2026-11-17, `layer_1/*` differs in all 256 partitions; `layer_0` and metadata are identical). `obikmer index` shows the same behaviour. Merge oracles therefore compare sorted dumps, not file hashes. Where the non-determinism originates (parallel graph fill or unitig walk) is not established.
|
||||
@@ -20,10 +20,22 @@ obikmer merge -o OUTPUT SOURCE... [OPTIONS]
|
||||
| `--force` | off | Overwrite an existing output directory |
|
||||
| `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts |
|
||||
| `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing |
|
||||
| `--budget-fraction` | `0.5` | Fraction of available RAM reserved as the memory budget for parallel partition merging |
|
||||
| `--dense` | off | Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see [pack](pack.md)) |
|
||||
|
||||
## Behaviour
|
||||
|
||||
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources.
|
||||
|
||||
By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies.
|
||||
|
||||
The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; `--dense` keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The [pack](pack.md) page explains the trade-off between the two formats.
|
||||
|
||||
## Resource usage
|
||||
|
||||
The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.
|
||||
|
||||
Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.
|
||||
|
||||
## Current limitation
|
||||
|
||||
An index produced by `merge` cannot itself be used as a source of a later merge yet: the command stops with a "No such file or directory" error. To add genomes to a collection, merge the original single-genome indexes again, together with the new ones.
|
||||
@@ -56,6 +56,8 @@ nav:
|
||||
- PersistentBitVec: implementation/persistent_bit_vec.md
|
||||
- Merge command: implementation/merge.md
|
||||
- Merge parallelism & memory: implementation/merge_parallelism.md
|
||||
- "Merge mmap audit (2026-11)": implementation/merge_mmap_audit.md
|
||||
- "Merge scaling (open)": implementation/merge_scaling.md
|
||||
- Kmer filtering: implementation/filtering.md
|
||||
- Select command: implementation/select.md
|
||||
- obitaxonomy crate: implementation/obitaxonomy.md
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
# Prompt — refonte mmap de `obikmerge` (plans A puis C)
|
||||
|
||||
> À coller tel quel au début d'une nouvelle session Claude Code ouverte à la racine du dépôt obikmer.
|
||||
> Conçu pour être suivi par **Sonnet** : toutes les décisions d'architecture sont fixées ci-dessous. Le plan B de l'audit (ouverture « slots only » dans `obikindex`) est **hors périmètre** de cette session (voir étape 5).
|
||||
|
||||
---
|
||||
|
||||
## Contexte
|
||||
|
||||
Lis d'abord `DevDocMD/implementation/merge_mmap_audit.md`. C'est le diagnostic complet, et ce prompt en implémente le plan.
|
||||
|
||||
En résumé : `obikmerge::partition_merge::merge_partition` ouvre, pour **chaque** partition, **toutes** les couches de **toutes** les sources (4 mmaps chacune). Elle crée aussi **toutes** les colonnes destination d'avance, en fichiers creux mappés en écriture (`set_len` + `MmapMut`). Avec 186 sources, ça fait environ 1 100 mappings par partition, ce qui épuise `vm.max_map_count` en production (pas d'accès root pour le relever). L'écriture via `MmapMut` dans des fichiers creux transforme aussi tout refus d'allocation disque en SIGBUS.
|
||||
|
||||
Objectif : le nombre de mappings vivants par partition ne doit plus dépendre du nombre de sources, et l'écriture des colonnes du merge ne doit plus passer par `MmapMut`. **Le résultat sur disque doit rester strictement identique.**
|
||||
|
||||
## Règles (non négociables)
|
||||
|
||||
- Appelle `mcp__serena__initial_instructions` avant tout travail de code. Utilise Serena (`find_symbol`, `get_symbols_overview`, `find_referencing_symbols`, `replace_symbol_body`, `replace_content`) pour naviguer et éditer le Rust.
|
||||
- **Aucune commande VCS mutante** (git/jj : commit, add, mv, rm, describe, push…). Lecture seule autorisée.
|
||||
- **Aucune nouvelle dépendance**, ni changement de version.
|
||||
- `obikindex` fait autorité : cette session n'y modifie et n'y ajoute rien.
|
||||
- Pas de fonctions libres ad hoc dans `obikmerge` pour combler un manque d'API. Si une opération générique manque sur un type de `obicompactvec`, ajoute-la comme méthode sur ce type.
|
||||
- Fichiers source sous ~500 lignes. `partition_merge.rs` en fait déjà 605 : il **doit** être découpé (étape 6).
|
||||
- **Ne jamais `.collect()` un flux de k-mers ou de unitigs** pour satisfaire une contrainte de trait.
|
||||
- Échanges en **français**, documentation en **anglais**.
|
||||
- **Point d'arrêt obligatoire** à la fin de chaque étape. Résume ce qui a été fait et les résultats de tests (sortie réelle), puis **attends mon feu vert** avant l'étape suivante.
|
||||
- Si un test ou l'oracle échoue et que tu ne trouves pas la cause en une tentative, **arrête-toi et expose le problème**, sans contournement.
|
||||
|
||||
## Étape 1 — Oracle de non-régression (avant toute modification de code)
|
||||
|
||||
### 1a. Oracle synthétique (rapide, couvre comptage et merge en deux temps)
|
||||
|
||||
1. Lis `scripts/smoke_test_index.sh` (modèle de style, options de `obikmer index`) et les `--help` de `obikmer index`, `merge` et `dump`.
|
||||
2. Écris `scripts/smoke_test_merge.sh`, sur le même modèle :
|
||||
- génère 4 FASTA aléatoires déterministes A, B, C, D, petits, avec **des fragments partagés** entre eux. Le merge doit à la fois retrouver des k-mers déjà présents dans la destination (chemin « couche existante ») et en créer de nouveaux (chemin « nouvelle couche ») ;
|
||||
- indexe chacun deux fois : présence (défaut) et comptage (`--with-counts`). Plusieurs partitions (`-p 4`) ;
|
||||
- scénario 1 : `merge A B C D` (présence), puis la même chose en comptage ;
|
||||
- scénario 2 : `merge A B` → AB, puis `merge AB C D`. Ce scénario couvre une source multi-génome et une destination à plusieurs couches ;
|
||||
- scénario 3 : comme le scénario 1, avec `--dense` ;
|
||||
- pour chaque sortie : `obikmer dump` → `sort` → un fichier texte ;
|
||||
- deux modes : `--record DIR` écrit les sorties triées dans `DIR`, `--check DIR` les régénère et fait un `diff` avec `DIR`. Affiche PASS/FAIL clairement, avec un code de sortie cohérent.
|
||||
3. Lance `--record benchmark/run/merge_oracle/synthetic` **avec le code actuel, non modifié**. Vérifie que les fichiers ne sont pas vides et que les colonnes de génomes ont du sens.
|
||||
4. Lance aussitôt `--check` sur le même code pour vérifier que le script est déterministe.
|
||||
|
||||
### 1b. Oracle sur données réelles : `benchmark/`
|
||||
|
||||
`benchmark/run/specimen_index_presence/` contient 20 index présence construits à partir de vrais génomes (bactéries + une levure, lectures simulées). Lis `benchmark/README.md` et `benchmark/merge_presence.sh` (le merge de référence y utilise `--force --force-presence`).
|
||||
|
||||
État constaté au 2026-11-17, à revérifier :
|
||||
- `specimen_index_count/` est **vide** : pas de piste comptage sans relancer `gmake index_count` (GNU make ≥ 4.3 : `gmake` sur macOS). **Demande-moi** avant de le lancer, car ça prend du temps ;
|
||||
- `reference_index/` (`.npz`) est **absent** : `verify_merge_*.py` est inutilisable, ne t'appuie pas dessus ;
|
||||
- `global_index_presence/` a été produit par un binaire antérieur au code actuel (release du 2026-10-22) : **ce n'est pas la référence**.
|
||||
|
||||
Procédure :
|
||||
1. Recompile le binaire release à partir du **code actuel non modifié** (`cargo build --release` dans `src/`). Ne touche pas aux artefacts existants de `benchmark/run/`.
|
||||
2. Lance le merge des 20 index avec les mêmes options que `merge_presence.sh`, mais vers un répertoire **dédié**, par exemple `benchmark/run/merge_oracle/bench/merged`, jamais `benchmark/run/global_index_presence`. Garde le log stderr : le rapport de stages (wall et RSS de `merge_partitions`) sert de référence de performance.
|
||||
3. `obikmer dump` du résultat → `sort` → compresse (`zstd`) + `sha256sum`, dans `benchmark/run/merge_oracle/bench/`. La sortie peut être volumineuse : compare d'abord les hash, et ne fais un `diff` que s'ils diffèrent.
|
||||
4. Contrôle croisé informatif : fais la même chose (dump trié + hash) sur `benchmark/run/global_index_presence`. Si les hash diffèrent, **signale-le sans conclure** : l'écart peut venir d'évolutions légitimes du code depuis octobre.
|
||||
5. Écris ces étapes dans un script `scripts/bench_merge_oracle.sh` avec deux modes : `--record DIR` et `--check DIR`. En mode `--check`, il affiche aussi le temps de `merge_partitions` avant/après.
|
||||
|
||||
Après chaque étape de code (3, 4, 6), les **deux** oracles doivent passer : `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check`. Si `merge_partitions` est plus de 20 % plus lent qu'avant sur le benchmark, signale-le au point d'arrêt.
|
||||
|
||||
**Point d'arrêt.**
|
||||
|
||||
## Étape 2 — `MatrixBuilder::n_cols` (petit prérequis, `obicompactvec`)
|
||||
|
||||
Ajoute `pub fn n_cols(&self) -> usize` à `obicompactvec::MatrixBuilder` (`src/obicompactvec/src/matrix_builder.rs`), en déléguant au builder Bit ou Int. Si `PersistentCompactIntMatrixBuilder` n'expose pas `n_cols()`, ajoute-le sur ce type. Ajoute un test unitaire. `cargo test -p obicompactvec`.
|
||||
|
||||
Pas de point d'arrêt séparé : enchaîne sur l'étape 3.
|
||||
|
||||
## Étape 3 — Plan A : passe 2 par groupes de sources (`obikmerge`)
|
||||
|
||||
Fichier : `src/obikmerge/src/partition_merge.rs`, fonction `merge_partition`.
|
||||
|
||||
### 3a. Passe 1 sans ouvrir les couches sources
|
||||
|
||||
Construis la liste des chemins `unitigs.bin` sans `open()`. Pour chaque source et chaque `l in 0..partition.n_layers()`, prends `KmerLayer::new(&partition, l).unitigs_path()` (ce chemin est valide sur la variante `Empty`). Garde le filtre `exists()`. Ne touche pas au reste de la passe 1 (`build_graph`, `materialize_layer`, ouverture de `new_mphf`).
|
||||
|
||||
### 3b. Nouvelle couche : matrice initialisée seule
|
||||
|
||||
Si `any_new`, crée la matrice de la nouvelle couche avec `MatrixBuilder::new(presence, n_new, data_dir)`, ajoute les `n_dst_genomes` colonnes absentes, puis **ferme**. Les colonnes sources seront ajoutées par `resume`, groupe par groupe.
|
||||
|
||||
### 3c. Boucle sur les groupes
|
||||
|
||||
- `const SRC_GROUP: usize = 2;` (aligné sur `max_open`), avec un commentaire qui explique pourquoi.
|
||||
- Itère sur `sources.chunks(SRC_GROUP)`, **dans l'ordre**, en tenant un `col_base` = nombre de colonnes sources déjà ajoutées avant ce groupe.
|
||||
- Pour chaque groupe :
|
||||
1. ouvre les couches de ses sources pour cette partition (`partition.layer(l)?.open()` puis `Arc::new`), comme aujourd'hui mais **seulement pour ce groupe** ;
|
||||
2. `group_n` = somme des `src_n` du groupe ;
|
||||
3. pour chaque couche dst existante, puis pour la nouvelle couche si `any_new` : `MatrixBuilder::resume`, vérifie `n_cols() == n_dst_genomes + col_base`, puis `add_col()` × `group_n` ;
|
||||
4. construis les `pass2_items` du groupe. Le `col_offset` est **local au groupe** (0..group_n) ;
|
||||
5. lance le pipeline de passe 2 existant, inchangé dans sa logique (Flat → Transform → Sink), sur ces items et ces builders. Les vecteurs `exist_sink`/`new_sink` ne contiennent que les colonnes du groupe ;
|
||||
6. vérifie l'erreur de passe 2, ferme toutes les `ColBuilder` puis tous les `MatrixBuilder` du groupe ;
|
||||
7. **drop** les couches du groupe avant de passer au suivant ;
|
||||
8. `col_base += group_n`.
|
||||
- L'initialisation `init_presence_matrix` (premier merge en présence) reste **avant** la boucle, inchangée.
|
||||
- `dst_layers` (ouverts une seule fois) et `new_mphf` restent partagés par tous les groupes.
|
||||
|
||||
Invariant à documenter dans un commentaire : les colonnes sont ajoutées dans l'ordre des sources, donc la colonne absolue `n_dst_genomes + col_base + offset_local + g` est la même qu'avant la refonte.
|
||||
|
||||
### 3d. Vérification
|
||||
|
||||
`cargo test -p obikmerge`, `cargo clippy -p obikmerge`, puis `scripts/smoke_test_merge.sh --check benchmark/run/merge_oracle/synthetic` et `scripts/bench_merge_oracle.sh --check benchmark/run/merge_oracle/bench` (recompile le release avant). Les deux **doivent** passer sans aucun diff.
|
||||
|
||||
**Point d'arrêt.**
|
||||
|
||||
## Étape 4 — Plan C : colonnes construites en mémoire (`obicompactvec`)
|
||||
|
||||
But : plus aucun `MmapMut` ni fichier creux pour les colonnes **remplies** par le merge.
|
||||
|
||||
1. Lis `PersistentBitVecBuilder` (`bitvec.rs` : `new`, `set`, `close`), `PersistentCompactIntVecBuilder` (`builder.rs` : `new`, `set`, `close`), `format.rs` (`finalize_pciv`, `HEADER_SIZE`) et `mmap_file.rs` (`building_path`, `publish_built`). Note **exactement** les octets écrits à la fermeture : en-tête, données, overflow, `.building` puis `rename`.
|
||||
2. Ajoute deux types :
|
||||
- `MemBitVecBuilder` (dans `bitvec.rs`, ou un nouveau `mem_builder.rs` si `bitvec.rs` dépasse 500 lignes) : `words: Vec<u64>`, `n`, `path` ; `new(n, path)`, `set(slot, bool)`, `close() -> io::Result<()>` ;
|
||||
- `MemCompactIntVecBuilder` : `primary: Vec<u8>`, `overflow: HashMap<usize, u32>`, `n`, `path` ; `new`, `set(slot, u32)` (même sémantique sentinelle 255 que le builder persistant), `close()`.
|
||||
|
||||
`close()` écrit avec un `BufWriter` dans `building_path(path)`, produit le **même fichier, octet pour octet**, que le builder persistant, fait `flush` + `sync_all`, puis `publish_built(path)`. Réutilise les fonctions de format existantes plutôt que de dupliquer la mise en page de l'en-tête. S'il faut factoriser l'écriture de l'en-tête ou de l'overflow pour la partager, fais-le dans `format.rs` / `mmap_file.rs`.
|
||||
3. Dans `matrix_builder.rs` : ajoute les variantes `ColBuilder::MemBit(MemBitVecBuilder)` et `ColBuilder::MemInt(MemCompactIntVecBuilder)` (`set_val` et `close` les gèrent), plus `MatrixBuilder::add_mem_col() -> io::Result<ColBuilder>`. Ce dernier incrémente `n_cols` exactement comme `add_col`. Si les builders de matrice sous-jacents n'exposent ni le chemin de la prochaine colonne ni l'incrément, ajoute-leur une méthode `add_mem_col` symétrique de `add_col`. Le nommage des colonnes reste privé à la crate.
|
||||
4. Tests unitaires (`src/obicompactvec/src/tests/`) : pour n ∈ {0, 1, 63, 64, 65, 10 000} et des valeurs pseudo-aléatoires (incluant des comptes ≥ 255), comparer **octet par octet** le fichier produit par le builder mémoire et celui du builder persistant. Vérifier aussi qu'une matrice construite avec `add_mem_col` se relit via `PersistentBitMatrix::open` / `PersistentIntMatrix::open`.
|
||||
5. Dans `merge_partition`, remplace `add_col()` par `add_mem_col()` pour les colonnes sources (couches existantes et nouvelle couche). `add_absent_col` reste tel quel : on n'écrit jamais dans son mapping, donc aucun risque de SIGBUS.
|
||||
6. Ajoute un commentaire sur le coût mémoire : `SRC_GROUP × src_n × (L_dst + 1) × (n/8 ou n)` octets au plus par partition.
|
||||
7. `cargo test -p obicompactvec -p obikmerge`, clippy, puis `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check` (release recompilé).
|
||||
|
||||
`obikselect` et `obikfilter` ne sont **pas** migrés dans cette session.
|
||||
|
||||
**Point d'arrêt.**
|
||||
|
||||
## Étape 5 — Plan B : hors périmètre
|
||||
|
||||
**Ne pas implémenter.** Le plan B (ouverture « slots only » des couches dans `obikindex`) a été tranché par le développeur : ce sera un chantier séparé (forme retenue : composition, voir la section « Plan B » de `merge_mmap_audit.md`). Après l'étape 3, son gain en mmaps pour le merge est négligeable (~8 → ~2 par partition). Il touche en revanche le chemin de requête d'`obikindex` et concerne trois crates.
|
||||
|
||||
Les couches sources de la passe 2 continuent donc d'être ouvertes par `partition.layer(l)?.open()`, mais seulement groupe par groupe (étape 3). Ne crée aucun type ni méthode « lookup only » dans `obikindex` ou `obikmerge`.
|
||||
|
||||
## Étape 6 — Découpage, doc, finition
|
||||
|
||||
1. Découpe `partition_merge.rs` en module `partition_merge/` (par exemple `mod.rs` pour l'orchestration, `pass1.rs`, `pass2.rs`, `tests.rs`), chaque fichier sous 500 lignes, sans changement de comportement.
|
||||
2. Supprime tout code devenu mort : `find_referencing_symbols` sur ce que la refonte a remplacé. Signale tout ce qui est devenu injustifié plutôt que de le garder par inertie.
|
||||
3. `cargo test --workspace`, `cargo clippy --workspace`, `cargo check --workspace --features numa` si la feature existe, `scripts/smoke_test_index.sh`, `scripts/smoke_test_merge.sh --check` et `scripts/bench_merge_oracle.sh --check` (release recompilé ; reporte le temps de `merge_partitions` avant/après).
|
||||
4. Documentation :
|
||||
- `DevDocMD/implementation/merge_mmap_audit.md` : passe le statut à « implemented », en indiquant ce qui a été fait et les écarts éventuels au plan ;
|
||||
- `DevDocMD/implementation/merge.md`, section « Algorithm » : décris le traitement par groupes de sources et les colonnes en mémoire ;
|
||||
- vérifie si `UserDocMD/` décrit le comportement mémoire du merge, et le cas échéant mets-le à jour, en prose, sans code Rust ;
|
||||
- mets à jour la `nav:` des `mkdocs*.yml` si un fichier a été ajouté, puis lance `qmd embed`.
|
||||
5. Rapport final : liste des fichiers modifiés, résultats de tests, et estimation du nombre de mmaps par partition avant/après (calculée, marquée comme estimée). Termine par la commande de mesure à lancer en production, qui ne demande pas root :
|
||||
|
||||
```bash
|
||||
PID=$(pgrep -f 'obikmer merge'); while sleep 60; do
|
||||
echo "$(date +%T) maps=$(wc -l < /proc/$PID/maps) threads=$(ls /proc/$PID/task | wc -l)"; done
|
||||
```
|
||||
|
||||
Ne commite rien : c'est moi qui le ferai.
|
||||
Executable
+156
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env bash
|
||||
# bench_merge_oracle.sh — golden-output + timing oracle for `obikmer merge`
|
||||
# on the real-data benchmark (benchmark/run/specimen_index_presence, 20
|
||||
# presence indexes built from real genomes).
|
||||
#
|
||||
# Companion of smoke_test_merge.sh for refactors of obikmerge (see
|
||||
# DevDocMD/implementation/merge_mmap_audit.md). Same merge options as
|
||||
# benchmark/merge_presence.sh (--force --force-presence), but the output goes
|
||||
# to a dedicated directory, never to benchmark/run/global_index_presence.
|
||||
#
|
||||
# What it does:
|
||||
# 1. builds the release `obikmer`
|
||||
# 2. merges every specimen presence index (sorted, as merge_presence.sh)
|
||||
# and keeps the stderr log (stage report: wall/RSS of merge_partitions)
|
||||
# 3. `obikmer dump --debug` of the result → header first, rows sorted
|
||||
# (LC_ALL=C) → sha256 of that sorted text, plus a zstd copy
|
||||
# 4. --record DIR: writes DIR/merged (the merged index), DIR/merge.stderr.log, DIR/dump.sha256,
|
||||
# DIR/dump.sorted.csv.zst, and (informative cross-check)
|
||||
# DIR/global_index_presence.sha256 = same dump hash computed on the
|
||||
# existing benchmark/run/global_index_presence (older binary — a
|
||||
# difference there is reported, not judged)
|
||||
# --check DIR: merges into a scratch directory, compares the hash with
|
||||
# DIR/dump.sha256 (full diff against the zstd copy only on mismatch),
|
||||
# and prints merge_partitions wall/RSS and pack wall before → after
|
||||
#
|
||||
# Usage:
|
||||
# scripts/bench_merge_oracle.sh (--record DIR | --check DIR) [--keep]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
KEEP=0
|
||||
MODE=""
|
||||
DIR=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--record) MODE=record; DIR="$2"; shift 2 ;;
|
||||
--check) MODE=check; DIR="$2"; shift 2 ;;
|
||||
--keep) KEEP=1; shift ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [--keep]" >&2; exit 2; }
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
|
||||
RUN_DIR="$REPO_ROOT/benchmark/run"
|
||||
IDX_DIR="$RUN_DIR/specimen_index_presence"
|
||||
GLOBAL_OLD="$RUN_DIR/global_index_presence"
|
||||
BIN="$REPO_ROOT/src/target/release/obikmer"
|
||||
|
||||
mkdir -p "$DIR" "$RUN_DIR/merge_oracle"
|
||||
# Scratch space on the same file system as the benchmark data (merge's
|
||||
# bootstrap hard-links from its base source; sort needs room for temp files).
|
||||
WORK="$(mktemp -d "$RUN_DIR/merge_oracle/.work.XXXXXX")"
|
||||
cleanup() {
|
||||
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ── 1. build ────────────────────────────────────────────────────────────────
|
||||
if ! (cd "$REPO_ROOT/src" && cargo build --release -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
|
||||
cat "$WORK/build.log" >&2
|
||||
fail "cargo build --release failed"
|
||||
fi
|
||||
|
||||
sorted_dump() { # sorted_dump INDEX OUT_TXT — header first, rows sorted
|
||||
local raw="$WORK/dump.raw.csv"
|
||||
"$BIN" dump --debug "$1" > "$raw" 2> "$WORK/dump.log" \
|
||||
|| { cat "$WORK/dump.log" >&2; fail "obikmer dump $1 exited non-zero"; }
|
||||
{ head -n 1 "$raw"
|
||||
tail -n +2 "$raw" | LC_ALL=C sort -S 25% --parallel=8 -T "$WORK"
|
||||
} > "$2"
|
||||
rm -f "$raw"
|
||||
[ "$(head -n 2 "$2" | wc -l)" -eq 2 ] || fail "empty dump of $1"
|
||||
}
|
||||
|
||||
hash_of() { sha256sum "$1" | cut -d' ' -f1; }
|
||||
|
||||
stage_field() { # stage_field LOG FIELD [STAGE] — FIELD of the 'done stage=STAGE' line (default merge_partitions)
|
||||
perl -pe 's/\e\[[0-9;]*m//g' "$1" \
|
||||
| grep "done stage=${3:-merge_partitions} " \
|
||||
| sed -n "s/.*$2=\([^ ]*\( [KMG]*B\)\{0,1\}\).*/\1/p" | tail -n 1
|
||||
}
|
||||
|
||||
# ── 2. merge ────────────────────────────────────────────────────────────────
|
||||
mapfile -t SOURCES < <(find "$IDX_DIR" -mindepth 1 -maxdepth 1 -type d | sort)
|
||||
[ "${#SOURCES[@]}" -gt 0 ] || fail "no indexes found in $IDX_DIR"
|
||||
|
||||
if [ "$MODE" = record ]; then
|
||||
MERGED="$DIR/merged"
|
||||
LOG="$DIR/merge.stderr.log"
|
||||
else
|
||||
MERGED="$WORK/merged"
|
||||
LOG="$WORK/merge.stderr.log"
|
||||
fi
|
||||
|
||||
echo "merging ${#SOURCES[@]} presence indexes → $MERGED"
|
||||
if ! "$BIN" merge --output "$MERGED" --force --force-presence "${SOURCES[@]}" \
|
||||
> /dev/null 2> "$LOG"; then
|
||||
tail -n 40 "$LOG" >&2
|
||||
fail "obikmer merge exited non-zero"
|
||||
fi
|
||||
WALL="$(stage_field "$LOG" wall_secs)"
|
||||
RSS="$(stage_field "$LOG" rss)"
|
||||
PACK_WALL="$(stage_field "$LOG" wall_secs pack)"
|
||||
echo "merge_partitions: wall=${WALL}s rss=${RSS} pack: wall=${PACK_WALL}s"
|
||||
|
||||
# ── 3. sorted dump + hash ───────────────────────────────────────────────────
|
||||
DUMP="$WORK/dump.sorted.csv"
|
||||
sorted_dump "$MERGED" "$DUMP"
|
||||
H="$(hash_of "$DUMP")"
|
||||
echo "dump: $(($(wc -l < "$DUMP") - 1)) k-mers, sha256 $H"
|
||||
|
||||
# ── 4. record or check ──────────────────────────────────────────────────────
|
||||
if [ "$MODE" = record ]; then
|
||||
echo "$H" > "$DIR/dump.sha256"
|
||||
zstd -q -T0 -f "$DUMP" -o "$DIR/dump.sorted.csv.zst"
|
||||
|
||||
if [ -d "$GLOBAL_OLD" ]; then
|
||||
OLD_DUMP="$WORK/old.sorted.csv"
|
||||
sorted_dump "$GLOBAL_OLD" "$OLD_DUMP"
|
||||
HO="$(hash_of "$OLD_DUMP")"
|
||||
echo "$HO" > "$DIR/global_index_presence.sha256"
|
||||
if [ "$HO" = "$H" ]; then
|
||||
echo "cross-check: global_index_presence (older binary) dump is identical"
|
||||
else
|
||||
echo "cross-check: global_index_presence (older binary) dump DIFFERS" \
|
||||
"($(($(wc -l < "$OLD_DUMP") - 1)) k-mers, sha256 $HO) — informative only"
|
||||
fi
|
||||
fi
|
||||
echo "RECORDED → $DIR"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[ -f "$DIR/dump.sha256" ] || fail "no reference in $DIR (run --record first)"
|
||||
REF_H="$(cat "$DIR/dump.sha256")"
|
||||
REF_WALL="$(stage_field "$DIR/merge.stderr.log" wall_secs)"
|
||||
REF_RSS="$(stage_field "$DIR/merge.stderr.log" rss)"
|
||||
REF_PACK="$(stage_field "$DIR/merge.stderr.log" wall_secs pack)"
|
||||
echo "merge_partitions wall: ${REF_WALL}s → ${WALL}s rss: ${REF_RSS} → ${RSS}"
|
||||
echo "pack wall: ${REF_PACK}s → ${PACK_WALL}s"
|
||||
python3 -c "import sys; r,n=float(sys.argv[1]),float(sys.argv[2]); print(f'merge_partitions wall ratio: {n/r:.2f}' + (' (> 1.20: SLOWER THAN 20 %)' if n > 1.2*r else ''))" \
|
||||
"$REF_WALL" "$WALL" || true
|
||||
|
||||
if [ "$H" = "$REF_H" ]; then
|
||||
echo "PASS: dump identical to reference ($H)"
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: dump hash differs (ref $REF_H, now $H)" >&2
|
||||
zstd -q -d -c "$DIR/dump.sorted.csv.zst" | { diff - "$DUMP" || true; } | head -n 40 >&2
|
||||
exit 1
|
||||
Executable
+186
@@ -0,0 +1,186 @@
|
||||
#!/usr/bin/env bash
|
||||
# smoke_test_merge.sh — end-to-end golden-output oracle for `obikmer merge`
|
||||
#
|
||||
# Non-regression oracle for refactors of obikmerge (see
|
||||
# DevDocMD/implementation/merge_mmap_audit.md). Like smoke_test_index.sh it
|
||||
# drives the real CLI, never the library APIs.
|
||||
#
|
||||
# What it does:
|
||||
# 1. builds `obikmer` (debug)
|
||||
# 2. generates (on --record) 4 small deterministic FASTA files A, B, C, D sharing
|
||||
# fragments, so that merge both hits k-mers already present in the
|
||||
# destination ("existing layer" path) and creates new ones ("new layer"
|
||||
# path). D carries a short motif repeated > 255 times so the count track
|
||||
# exercises the compact-int overflow (counts >= 255).
|
||||
# 3. on --record, indexes each genome twice (presence, --with-counts) and
|
||||
# freezes the indexes in DIR/sources; --check reuses them (see below),
|
||||
# with -p PARTITIONS partitions
|
||||
# 4. runs the merge scenarios:
|
||||
# s1_presence / s1_count merge A B C D
|
||||
# s3_dense merge A B C D --dense (presence)
|
||||
# Scenario 2 (merge A B -> AB, then merge AB C D: multi-genome source,
|
||||
# multi-layer destination) is NOT included: the current code cannot
|
||||
# re-merge a merge output — its matrices are packed (sparse or dense)
|
||||
# at finalize, and `MatrixBuilder::resume` expects the columnar
|
||||
# `meta.json` layout (ENOENT). Known bug, to be fixed separately; see
|
||||
# DevDocMD/implementation/merge_mmap_audit.md, "Known issue".
|
||||
# 5. dumps every result with `obikmer dump --debug` (partition + layer
|
||||
# columns included, so a k-mer moving between layers is caught), keeps
|
||||
# the CSV header first and sorts the rows (LC_ALL=C)
|
||||
# 6. --record DIR writes the sorted dumps to DIR;
|
||||
# --check DIR regenerates them and diffs against DIR
|
||||
#
|
||||
# Usage:
|
||||
# scripts/smoke_test_merge.sh (--record DIR | --check DIR) [-p PARTITIONS] [--keep]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
K=21
|
||||
M=9
|
||||
PARTITIONS=4
|
||||
KEEP=0
|
||||
MODE=""
|
||||
DIR=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--record) MODE=record; DIR="$2"; shift 2 ;;
|
||||
--check) MODE=check; DIR="$2"; shift 2 ;;
|
||||
-p) PARTITIONS="$2"; shift 2 ;;
|
||||
--keep) KEEP=1; shift ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [-p N] [--keep]" >&2; exit 2; }
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
|
||||
WORK="$(mktemp -d -t obikmer_merge_smoke.XXXXXX)"
|
||||
|
||||
cleanup() {
|
||||
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ── 1. build ────────────────────────────────────────────────────────────────
|
||||
if ! (cd "$REPO_ROOT/src" && cargo build -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
|
||||
cat "$WORK/build.log" >&2
|
||||
fail "cargo build failed"
|
||||
fi
|
||||
BIN="$REPO_ROOT/src/target/debug/obikmer"
|
||||
|
||||
run() { # run LOGNAME CMD... — runs obikmer, dumps its log on failure
|
||||
local log="$WORK/$1.log"; shift
|
||||
if ! "$BIN" "$@" > "$log" 2>&1; then
|
||||
cat "$log" >&2
|
||||
fail "obikmer $1 exited non-zero"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── 2. deterministic genomes with shared fragments ──────────────────────────
|
||||
python3 - "$WORK" <<'EOF'
|
||||
import random, sys
|
||||
out = sys.argv[1]
|
||||
rng = random.Random(20261117)
|
||||
def rnd(n): return "".join(rng.choice("ACGT") for _ in range(n))
|
||||
frag = {i: rnd(600) for i in range(5)} # shared fragments F0..F4
|
||||
motif = rnd(40) # tandem motif (counts >= 255)
|
||||
genomes = {
|
||||
"A": [rnd(1500), frag[0], rnd(300), frag[1]],
|
||||
"B": [frag[1], rnd(1200), frag[2], motif * 3],
|
||||
"C": [frag[0], rnd(800), frag[2], rnd(400), frag[3]],
|
||||
"D": [frag[3], rnd(1000), motif * 300, frag[4], rnd(200), frag[0][:250]],
|
||||
}
|
||||
for name, parts in genomes.items():
|
||||
with open(f"{out}/{name}.fasta", "w") as f:
|
||||
for j, p in enumerate(parts):
|
||||
f.write(f">{name}_{j}\n{p}\n")
|
||||
EOF
|
||||
|
||||
# ── 3. source indexes: built on --record, frozen in DIR/sources, reused on
|
||||
# --check. `obikmer index` is not byte-stable run to run (unitig
|
||||
# segmentation, hence unitigs.bin size, varies — measured 2026-11-17),
|
||||
# and the merge base is the source with the largest unitigs.bin
|
||||
# (`choose_base`), so re-indexing on every check would make the merge
|
||||
# base — and the column order — flip between A and C. Freezing the
|
||||
# sources makes this an oracle of `merge` alone. Copied (not linked)
|
||||
# into WORK, since merge's bootstrap hard-links from its base source.
|
||||
if [ "$MODE" = record ]; then
|
||||
for G in A B C D; do
|
||||
run "index_${G}_pres" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
|
||||
--label "$G" -o "$WORK/$G.pres.idx" "$WORK/$G.fasta"
|
||||
run "index_${G}_cnt" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
|
||||
--with-counts --label "$G" -o "$WORK/$G.cnt.idx" "$WORK/$G.fasta"
|
||||
done
|
||||
rm -rf "$DIR/sources"
|
||||
mkdir -p "$DIR/sources"
|
||||
cp -R "$WORK"/*.idx "$DIR/sources/"
|
||||
else
|
||||
[ -d "$DIR/sources" ] || fail "no frozen source indexes in $DIR/sources (run --record first)"
|
||||
cp -R "$DIR/sources"/*.idx "$WORK/"
|
||||
fi
|
||||
|
||||
# ── 4. merge scenarios ──────────────────────────────────────────────────────
|
||||
OUT="$WORK/out"
|
||||
mkdir -p "$OUT"
|
||||
dump() { # dump NAME INDEX — header first, rows sorted
|
||||
local raw="$WORK/$1.dump.csv"
|
||||
if ! "$BIN" dump --debug "$2" > "$raw" 2> "$WORK/dump_$1.log"; then
|
||||
cat "$WORK/dump_$1.log" >&2
|
||||
fail "obikmer dump $1 exited non-zero"
|
||||
fi
|
||||
{ head -n 1 "$raw"; tail -n +2 "$raw" | LC_ALL=C sort; } > "$OUT/$1.csv"
|
||||
}
|
||||
|
||||
for T in pres cnt; do
|
||||
run "s1_$T" merge -o "$WORK/s1_$T.idx" \
|
||||
"$WORK/A.$T.idx" "$WORK/B.$T.idx" "$WORK/C.$T.idx" "$WORK/D.$T.idx"
|
||||
dump "s1_$T" "$WORK/s1_$T.idx"
|
||||
done
|
||||
|
||||
run s3_dense merge --dense -o "$WORK/s3_dense.idx" \
|
||||
"$WORK/A.pres.idx" "$WORK/B.pres.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
|
||||
dump s3_dense "$WORK/s3_dense.idx"
|
||||
|
||||
# sanity: no dump may be empty (header + at least one row)
|
||||
for f in "$OUT"/*.csv; do
|
||||
[ "$(wc -l < "$f")" -gt 1 ] || fail "empty dump: $(basename "$f")"
|
||||
done
|
||||
|
||||
# ── 5. record or check ──────────────────────────────────────────────────────
|
||||
if [ "$MODE" = record ]; then
|
||||
mkdir -p "$DIR"
|
||||
rm -f "$DIR"/*.csv
|
||||
cp "$OUT"/*.csv "$DIR"/
|
||||
echo "RECORDED: $(ls "$OUT" | wc -l | tr -d ' ') dumps → $DIR"
|
||||
wc -l "$DIR"/*.csv
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[ -d "$DIR" ] || fail "reference directory not found: $DIR"
|
||||
STATUS=0
|
||||
for ref in "$DIR"/*.csv; do
|
||||
name="$(basename "$ref")"
|
||||
if [ ! -f "$OUT/$name" ]; then
|
||||
echo "FAIL $name (not regenerated)"; STATUS=1
|
||||
elif diff -q "$ref" "$OUT/$name" > /dev/null; then
|
||||
echo "ok $name"
|
||||
else
|
||||
echo "FAIL $name"; { diff "$ref" "$OUT/$name" || true; } | head -20; STATUS=1
|
||||
fi
|
||||
done
|
||||
for new in "$OUT"/*.csv; do
|
||||
[ -f "$DIR/$(basename "$new")" ] || { echo "FAIL $(basename "$new") (no reference)"; STATUS=1; }
|
||||
done
|
||||
|
||||
if [ "$STATUS" -eq 0 ]; then
|
||||
echo "PASS: all merge dumps identical to $DIR"
|
||||
else
|
||||
echo "FAIL: merge dumps differ from $DIR" >&2
|
||||
fi
|
||||
exit "$STATUS"
|
||||
Generated
+1
-2
@@ -1537,7 +1537,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "obikmer"
|
||||
version = "1.4.2"
|
||||
version = "1.4.4"
|
||||
dependencies = [
|
||||
"clap",
|
||||
"csv",
|
||||
@@ -1582,7 +1582,6 @@ dependencies = [
|
||||
"obiskio",
|
||||
"obisys",
|
||||
"rayon",
|
||||
"tempfile",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ use std::io;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use crate::bitvec::PersistentBitVecBuilder;
|
||||
use crate::mem_builder::MemBitVecBuilder;
|
||||
use crate::meta::MatrixMeta;
|
||||
use crate::tempbitvec::TempBitVec;
|
||||
use crate::tempintvec::TempCompactIntVec;
|
||||
@@ -43,6 +44,14 @@ impl PersistentBitMatrixBuilder {
|
||||
PersistentBitVecBuilder::new(self.n, &path)
|
||||
}
|
||||
|
||||
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
|
||||
/// the column is built in RAM and written once at `close`.
|
||||
pub fn add_mem_col(&mut self) -> MemBitVecBuilder {
|
||||
let path = col_path(&self.dir, self.n_cols);
|
||||
self.n_cols += 1;
|
||||
MemBitVecBuilder::new(self.n, &path)
|
||||
}
|
||||
|
||||
pub fn add_col_ones(&mut self) -> io::Result<PersistentBitVecBuilder> {
|
||||
let path = col_path(&self.dir, self.n_cols);
|
||||
self.n_cols += 1;
|
||||
|
||||
@@ -35,9 +35,7 @@ use crate::meta::field;
|
||||
use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
||||
|
||||
use super::PersistentBitMatrix;
|
||||
use super::col_path;
|
||||
use super::columnar::ColumnarBitMatrix;
|
||||
use super::packed::PackedBitMatrix;
|
||||
use super::packed::{PackedBitMatrix, pack_bit_matrix};
|
||||
use crate::views::nonzero_triples;
|
||||
|
||||
fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") }
|
||||
@@ -615,37 +613,33 @@ impl BitPartials for PersistentSparseBitMatrix {
|
||||
m
|
||||
}
|
||||
}
|
||||
/// sparse on-disk format in place, mirroring [`super::pack_bit_matrix`]'s
|
||||
/// convention (old-format files removed only after the new format is
|
||||
/// fully written, so a crash mid-conversion leaves the previous, still
|
||||
/// valid format rather than a half-written one). Idempotent — does
|
||||
|
||||
/// Converts a layer's presence matrix (columnar `col_*.pbiv` or dense
|
||||
/// `matrix.pbmx`) to the sparse on-disk format in place. Old-format files
|
||||
/// are removed only after the new format is fully written, so a crash
|
||||
/// mid-conversion leaves a previous, still valid format. Idempotent: does
|
||||
/// nothing if `is_multi.prsb` already exists.
|
||||
///
|
||||
/// The dense→sparse transpose (`build_from_dense`) only needs `fill_row`,
|
||||
/// which costs the same `O(n_cols)` per row on `Columnar` as on `Packed`
|
||||
/// (see `ColumnarBitMatrix::fill_row`) — so this reads straight from
|
||||
/// whichever dense form is already on disk. In particular, when this runs
|
||||
/// right after a build (the common case: `presence/` is still `Columnar`,
|
||||
/// `matrix.pbmx` doesn't exist yet), it never materialises a full packed
|
||||
/// copy of the presence matrix just to read it back once and delete it.
|
||||
/// Always goes through the dense packed form, even right after a build.
|
||||
/// That costs one extra write and read of the dense matrix, but keeps the
|
||||
/// number of mappings at one instead of one per genome column.
|
||||
pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> {
|
||||
if is_multi_path(dir).exists() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Columnar → dense packed first, then dense packed → sparse. Opening the
|
||||
// columnar matrix directly would map every column file at once (one
|
||||
// mapping per genome, unbounded); `pack_bit_matrix` instead
|
||||
// concatenates the column files one at a time, unmapped, and leaves a
|
||||
// single `matrix.pbmx` to read from (one mapping). It is a no-op when
|
||||
// the layer is already dense-packed, and re-packs a stale `matrix.pbmx`
|
||||
// that predates newer columnar data. Every intermediate state is a
|
||||
// valid format, so a crash leaves something this function resumes.
|
||||
pack_bit_matrix(dir)?;
|
||||
let packed_path = dir.join("matrix.pbmx");
|
||||
if packed_path.exists() {
|
||||
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?);
|
||||
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
fs::remove_file(&packed_path)?;
|
||||
} else {
|
||||
let dense = PersistentBitMatrix::Columnar(ColumnarBitMatrix::open(dir)?);
|
||||
let n_cols = dense.n_cols();
|
||||
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
for c in 0..n_cols { let _ = fs::remove_file(col_path(dir, c)); }
|
||||
let _ = fs::remove_file(dir.join("meta.json"));
|
||||
}
|
||||
Ok(())
|
||||
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?);
|
||||
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
fs::remove_file(&packed_path)
|
||||
}
|
||||
@@ -8,7 +8,7 @@ use crate::mmap_file::{building_path, create_mmap_file, open_mmap_file, publish_
|
||||
use crate::reader::PersistentCompactIntVec;
|
||||
use crate::views::{BitSliceIter, BitSliceView, IntSliceView};
|
||||
|
||||
const MAGIC: [u8; 4] = *b"PBIV";
|
||||
pub(crate) const MAGIC: [u8; 4] = *b"PBIV";
|
||||
|
||||
// Header: magic(4) + _pad(4) + n(8) = 16 bytes.
|
||||
// Data starts at offset 16, u64-aligned (mmap base is page-aligned, 16 % 8 == 0).
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
use std::fs::OpenOptions;
|
||||
use std::io::{self, BufWriter, Seek, SeekFrom, Write as _};
|
||||
use std::io::{self, BufWriter, Seek, SeekFrom, Write};
|
||||
use std::path::Path;
|
||||
|
||||
pub const MAGIC: [u8; 4] = *b"PCIV";
|
||||
@@ -54,53 +54,63 @@ pub fn parse_index_entry(data: &[u8], base: usize, i: usize) -> (usize, usize) {
|
||||
// Sparse index target: ≤ 32 KB in L1 cache (16 B per entry → 2048 entries).
|
||||
pub const L1_INDEX_ENTRIES: usize = 2048;
|
||||
|
||||
/// Sparse-index `step` and entry count for `n_overflow` overflow entries:
|
||||
/// no index (`step == 0`) while the overflow section fits the L1 target.
|
||||
fn pciv_index_layout(n_overflow: usize) -> (usize, usize) {
|
||||
if n_overflow <= L1_INDEX_ENTRIES {
|
||||
(0, 0)
|
||||
} else {
|
||||
let step = n_overflow.div_ceil(L1_INDEX_ENTRIES);
|
||||
(step, n_overflow.div_ceil(step))
|
||||
}
|
||||
}
|
||||
|
||||
/// Write the 40-byte PCIV header for a vector of `n` slots with
|
||||
/// `n_overflow` overflow entries (index layout derived from it).
|
||||
pub(crate) fn write_pciv_header<W: Write>(w: &mut W, n: usize, n_overflow: usize) -> io::Result<()> {
|
||||
let (step, n_index) = pciv_index_layout(n_overflow);
|
||||
w.write_all(&MAGIC)?;
|
||||
w.write_all(&[0u8; 4])?;
|
||||
w.write_all(&(n as u64).to_le_bytes())?;
|
||||
w.write_all(&(n_overflow as u64).to_le_bytes())?;
|
||||
w.write_all(&(n_index as u64).to_le_bytes())?;
|
||||
w.write_all(&(step as u64).to_le_bytes())
|
||||
}
|
||||
|
||||
/// Write the sections that follow the primary bytes:
|
||||
/// overflow n_overflow × 12 B (slot: u64, value: u32), `entries` sorted by slot
|
||||
/// index n_index × 16 B (slot: u64, pos: u64) sparse index
|
||||
pub(crate) fn write_pciv_tail<W: Write>(w: &mut W, entries: &[(usize, u32)]) -> io::Result<()> {
|
||||
for &(slot, value) in entries {
|
||||
w.write_all(&(slot as u64).to_le_bytes())?;
|
||||
w.write_all(&value.to_le_bytes())?;
|
||||
}
|
||||
let (step, _) = pciv_index_layout(entries.len());
|
||||
if step > 0 {
|
||||
for (block, chunk) in entries.chunks(step).enumerate() {
|
||||
w.write_all(&(chunk[0].0 as u64).to_le_bytes())?;
|
||||
w.write_all(&((block * step) as u64).to_le_bytes())?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Finalise a PCIV file whose placeholder header and primary section are already on disk.
|
||||
///
|
||||
/// Truncates the file to `HEADER_SIZE + n`, then appends:
|
||||
/// overflow n_overflow × 12 B (slot: u64, value: u32) sorted by slot
|
||||
/// index n_index × 16 B (slot: u64, pos: u64) sparse index
|
||||
/// and overwrites the placeholder header at offset 0.
|
||||
/// Truncates the file to `HEADER_SIZE + n`, appends the overflow and index
|
||||
/// sections ([`write_pciv_tail`]) and overwrites the placeholder header at
|
||||
/// offset 0 ([`write_pciv_header`]).
|
||||
pub fn finalize_pciv(path: &Path, n: usize, entries: &[(usize, u32)]) -> io::Result<()> {
|
||||
let n_overflow = entries.len();
|
||||
|
||||
let file = OpenOptions::new().read(true).write(true).open(path)?;
|
||||
file.set_len((HEADER_SIZE + n) as u64)?;
|
||||
|
||||
let mut w = BufWriter::new(file);
|
||||
w.seek(SeekFrom::End(0))?;
|
||||
|
||||
for &(slot, value) in entries {
|
||||
w.write_all(&(slot as u64).to_le_bytes())?;
|
||||
w.write_all(&value.to_le_bytes())?;
|
||||
}
|
||||
|
||||
let step: usize = if n_overflow <= L1_INDEX_ENTRIES {
|
||||
0
|
||||
} else {
|
||||
n_overflow.div_ceil(L1_INDEX_ENTRIES)
|
||||
};
|
||||
|
||||
let n_index: usize = if step > 0 {
|
||||
let count = n_overflow.div_ceil(step);
|
||||
for (block, chunk) in entries.chunks(step).enumerate() {
|
||||
let slot = chunk[0].0 as u64;
|
||||
let pos = (block * step) as u64;
|
||||
w.write_all(&slot.to_le_bytes())?;
|
||||
w.write_all(&pos.to_le_bytes())?;
|
||||
}
|
||||
count
|
||||
} else {
|
||||
0
|
||||
};
|
||||
|
||||
write_pciv_tail(&mut w, entries)?;
|
||||
w.flush()?;
|
||||
|
||||
let mut file = w.into_inner().map_err(|e| e.into_error())?;
|
||||
file.seek(SeekFrom::Start(0))?;
|
||||
file.write_all(&MAGIC)?;
|
||||
file.write_all(&[0u8; 4])?;
|
||||
file.write_all(&(n as u64).to_le_bytes())?;
|
||||
file.write_all(&(n_overflow as u64).to_le_bytes())?;
|
||||
file.write_all(&(n_index as u64).to_le_bytes())?;
|
||||
file.write_all(&(step as u64).to_le_bytes())?;
|
||||
write_pciv_header(&mut file, n, entries.len())?;
|
||||
file.flush()
|
||||
}
|
||||
@@ -11,6 +11,7 @@ use crate::bitmatrix::{
|
||||
use crate::builder::PersistentCompactIntVecBuilder;
|
||||
use crate::colgroup::{ColGroup, MatrixGroupOps, chunked_presence_count};
|
||||
use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE};
|
||||
use crate::mem_builder::MemCompactIntVecBuilder;
|
||||
use crate::meta::MatrixMeta;
|
||||
use crate::reader::PersistentCompactIntVec;
|
||||
use crate::sparse_intmatrix::{PersistentSparseCompactIntMatrix, is_present as sparse_is_present};
|
||||
@@ -717,6 +718,14 @@ impl PersistentCompactIntMatrixBuilder {
|
||||
PersistentCompactIntVecBuilder::new(self.n, &path)
|
||||
}
|
||||
|
||||
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
|
||||
/// the column is built in RAM and written once at `close`.
|
||||
pub fn add_mem_col(&mut self) -> MemCompactIntVecBuilder {
|
||||
let path = col_path(&self.dir, self.n_cols);
|
||||
self.n_cols += 1;
|
||||
MemCompactIntVecBuilder::new(self.n, &path)
|
||||
}
|
||||
|
||||
pub fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()> {
|
||||
src.make_persistent(&col_path(&self.dir, self.n_cols))?;
|
||||
self.n_cols += 1;
|
||||
|
||||
@@ -9,6 +9,7 @@ mod int_group_ops;
|
||||
mod intmatrix;
|
||||
mod layer_meta;
|
||||
mod matrix_builder;
|
||||
mod mem_builder;
|
||||
mod meta;
|
||||
mod mmap_file;
|
||||
mod rankselect;
|
||||
@@ -35,6 +36,7 @@ pub use intmatrix::{
|
||||
};
|
||||
|
||||
pub use layer_meta::LayerMeta;
|
||||
pub use mem_builder::{MemBitVecBuilder, MemCompactIntVecBuilder};
|
||||
pub use mmap_file::{building_path, publish_built};
|
||||
pub use matrix_builder::{ColBuilder, MatrixBuilder};
|
||||
pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
||||
|
||||
@@ -10,13 +10,19 @@ use std::path::Path;
|
||||
|
||||
use crate::{PersistentBitMatrixBuilder as BitMB, PersistentBitVecBuilder};
|
||||
use crate::{PersistentCompactIntMatrixBuilder as IntMB, PersistentCompactIntVecBuilder};
|
||||
use crate::{TempBitVec, TempCompactIntVec};
|
||||
use crate::{MemBitVecBuilder, MemCompactIntVecBuilder, TempBitVec, TempCompactIntVec};
|
||||
|
||||
// ── ColBuilder ────────────────────────────────────────────────────────────────
|
||||
|
||||
/// One column being built. `Bit`/`Int` write through a mmap of a sparse
|
||||
/// file as values are set; `MemBit`/`MemInt` keep the column in RAM and
|
||||
/// write it once at `close` (same file, byte for byte — see
|
||||
/// [`MemBitVecBuilder`]).
|
||||
pub enum ColBuilder {
|
||||
Bit(PersistentBitVecBuilder),
|
||||
Int(PersistentCompactIntVecBuilder),
|
||||
MemBit(MemBitVecBuilder),
|
||||
MemInt(MemCompactIntVecBuilder),
|
||||
}
|
||||
|
||||
impl ColBuilder {
|
||||
@@ -24,6 +30,8 @@ impl ColBuilder {
|
||||
match self {
|
||||
ColBuilder::Bit(b) => b.set(slot, value > 0),
|
||||
ColBuilder::Int(b) => b.set(slot, value),
|
||||
ColBuilder::MemBit(b) => b.set(slot, value > 0),
|
||||
ColBuilder::MemInt(b) => b.set(slot, value),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -31,6 +39,8 @@ impl ColBuilder {
|
||||
match self {
|
||||
ColBuilder::Bit(b) => b.close(),
|
||||
ColBuilder::Int(b) => b.close(),
|
||||
ColBuilder::MemBit(b) => b.close(),
|
||||
ColBuilder::MemInt(b) => b.close(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -64,6 +74,15 @@ impl MatrixBuilder {
|
||||
})
|
||||
}
|
||||
|
||||
/// Number of columns created so far (absent + data columns, including
|
||||
/// those inherited from a resumed directory).
|
||||
pub fn n_cols(&self) -> usize {
|
||||
match self {
|
||||
MatrixBuilder::Bit(b) => b.n_cols(),
|
||||
MatrixBuilder::Int(b) => b.n_cols(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Add a column with no data written (all-zero/false) — for columns
|
||||
/// absent from a given source (e.g. destination-only genomes not
|
||||
/// contributed by the source being merged in).
|
||||
@@ -81,6 +100,17 @@ impl MatrixBuilder {
|
||||
})
|
||||
}
|
||||
|
||||
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
|
||||
/// the column is built in RAM and written once, sequentially, at
|
||||
/// `close`: no sparse file, no writable mapping. Costs `n / 8` bytes
|
||||
/// (presence) or `n` bytes + overflow (counts) of RAM until closed.
|
||||
pub fn add_mem_col(&mut self) -> ColBuilder {
|
||||
match self {
|
||||
MatrixBuilder::Bit(b) => ColBuilder::MemBit(b.add_mem_col()),
|
||||
MatrixBuilder::Int(b) => ColBuilder::MemInt(b.add_mem_col()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Add a column already computed as a bit vector.
|
||||
pub fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> {
|
||||
match self {
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
//! In-memory column builders: the content of a column is built in RAM and
|
||||
//! written once, sequentially, when the builder is closed.
|
||||
//!
|
||||
//! They produce exactly the same file, byte for byte, as
|
||||
//! [`PersistentBitVecBuilder`](crate::PersistentBitVecBuilder) /
|
||||
//! [`PersistentCompactIntVecBuilder`](crate::PersistentCompactIntVecBuilder),
|
||||
//! but never create a sparse file nor a writable mapping: a block the file
|
||||
//! system cannot allocate (full disk, quota, network FS) is reported as an
|
||||
//! `io::Error` by `close`, instead of a SIGBUS on the page fault that first
|
||||
//! touches it. The price is that the whole column lives in RAM until
|
||||
//! `close`: `n / 8` bytes for bits, `n` bytes (+ overflow map) for counts.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::io::{self, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use crate::bitvec::{n_words, MAGIC as PBIV_MAGIC};
|
||||
use crate::format::{write_pciv_header, write_pciv_tail};
|
||||
use crate::mmap_file::{write_built, write_std_header};
|
||||
|
||||
// ── Bits (PBIV) ───────────────────────────────────────────────────────────────
|
||||
|
||||
/// In-memory counterpart of `PersistentBitVecBuilder` (`set` + `close` only).
|
||||
pub struct MemBitVecBuilder {
|
||||
words: Vec<u64>,
|
||||
n: usize,
|
||||
path: PathBuf,
|
||||
}
|
||||
|
||||
impl MemBitVecBuilder {
|
||||
/// All-false vector of `n` bits, to be written at `path` by `close`.
|
||||
pub fn new(n: usize, path: &Path) -> Self {
|
||||
Self { words: vec![0u64; n_words(n)], n, path: path.to_path_buf() }
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub fn len(&self) -> usize {
|
||||
self.n
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.n == 0
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub fn set(&mut self, slot: usize, value: bool) {
|
||||
debug_assert!(slot < self.n);
|
||||
let bit = 1u64 << (slot & 63);
|
||||
if value {
|
||||
self.words[slot >> 6] |= bit;
|
||||
} else {
|
||||
self.words[slot >> 6] &= !bit;
|
||||
}
|
||||
}
|
||||
|
||||
/// Writes the PBIV file — header, then the words in native byte order,
|
||||
/// exactly as the mmap-backed builder lays them out — and publishes it.
|
||||
pub fn close(self) -> io::Result<()> {
|
||||
let Self { words, n, path } = self;
|
||||
write_built(&path, |w| {
|
||||
write_std_header(w, PBIV_MAGIC, [0u8; 4], n)?;
|
||||
for word in &words {
|
||||
w.write_all(&word.to_ne_bytes())?;
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ── Compact ints (PCIV) ───────────────────────────────────────────────────────
|
||||
|
||||
/// In-memory counterpart of `PersistentCompactIntVecBuilder` (`set` +
|
||||
/// `close` only). Same encoding: one byte per slot, values ≥ 255 stored as
|
||||
/// the sentinel 255 plus an overflow entry.
|
||||
pub struct MemCompactIntVecBuilder {
|
||||
primary: Vec<u8>,
|
||||
overflow: HashMap<usize, u32>,
|
||||
n: usize,
|
||||
path: PathBuf,
|
||||
}
|
||||
|
||||
impl MemCompactIntVecBuilder {
|
||||
/// All-zero vector of `n` slots, to be written at `path` by `close`.
|
||||
pub fn new(n: usize, path: &Path) -> Self {
|
||||
Self { primary: vec![0u8; n], overflow: HashMap::new(), n, path: path.to_path_buf() }
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub fn len(&self) -> usize {
|
||||
self.n
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.n == 0
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub fn set(&mut self, slot: usize, value: u32) {
|
||||
if value < 255 {
|
||||
self.primary[slot] = value as u8;
|
||||
self.overflow.remove(&slot);
|
||||
} else {
|
||||
self.primary[slot] = 255;
|
||||
self.overflow.insert(slot, value);
|
||||
}
|
||||
}
|
||||
|
||||
/// Writes the PCIV file — header, primary bytes, overflow entries sorted
|
||||
/// by slot, sparse index — and publishes it.
|
||||
pub fn close(self) -> io::Result<()> {
|
||||
let Self { primary, overflow, n, path } = self;
|
||||
let mut entries: Vec<(usize, u32)> = overflow.into_iter().collect();
|
||||
entries.sort_unstable_by_key(|&(slot, _)| slot);
|
||||
write_built(&path, |w| {
|
||||
write_pciv_header(w, n, entries.len())?;
|
||||
w.write_all(&primary)?;
|
||||
write_pciv_tail(w, &entries)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -11,7 +11,7 @@
|
||||
//! between formats and aren't forced into a shared shape.
|
||||
|
||||
use std::fs::{self, File, OpenOptions};
|
||||
use std::io::{self, Seek, SeekFrom, Write as _};
|
||||
use std::io::{self, BufWriter, Seek, SeekFrom, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use memmap2::{Mmap, MmapMut};
|
||||
@@ -37,6 +37,37 @@ pub fn publish_built(path: &Path) -> io::Result<()> {
|
||||
fs::rename(building_path(path), path)
|
||||
}
|
||||
|
||||
/// Writes the crate-wide standard header: `magic` + `pad` + `n as u64` (LE).
|
||||
pub(crate) fn write_std_header<W: Write>(w: &mut W, magic: [u8; 4], pad: [u8; 4], n: usize) -> io::Result<()> {
|
||||
w.write_all(&magic)?;
|
||||
w.write_all(&pad)?;
|
||||
w.write_all(&(n as u64).to_le_bytes())
|
||||
}
|
||||
|
||||
/// Writes a whole file sequentially through `write` into
|
||||
/// [`building_path`]`(path)`, flushes it, then publishes it at `path`. No
|
||||
/// sparse file and no writable mapping: an allocation failure on the file
|
||||
/// system (full disk, quota) surfaces as an `io::Error` from a `write`,
|
||||
/// never as a SIGBUS on a page fault.
|
||||
///
|
||||
/// Deliberately no `sync_all`: one forced sync per file made the merge
|
||||
/// ~8× slower (thousands of column files; measured 2026-11-17 on macOS),
|
||||
/// and the mmap builders it replaces never synced either. Durability is the
|
||||
/// OS's, like the rest of an index; the `rename` still guarantees that no
|
||||
/// partially written file is ever published. Caveat: on a write-back
|
||||
/// network FS an allocation error may only be reported at `close`, which
|
||||
/// `File`'s drop ignores.
|
||||
pub(crate) fn write_built(
|
||||
path: &Path,
|
||||
write: impl FnOnce(&mut BufWriter<File>) -> io::Result<()>,
|
||||
) -> io::Result<()> {
|
||||
let mut w = BufWriter::new(File::create(building_path(path))?);
|
||||
write(&mut w)?;
|
||||
w.flush()?;
|
||||
drop(w.into_inner().map_err(|e| e.into_error())?);
|
||||
publish_built(path)
|
||||
}
|
||||
|
||||
/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it
|
||||
/// with [`publish_built`] once complete — writes the standard header — `magic` +
|
||||
/// `pad` (4 bytes right after `magic`; usually all-zero, but
|
||||
@@ -60,9 +91,7 @@ pub(crate) fn create_mmap_file(
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.open(building_path(path))?;
|
||||
file.write_all(&magic)?;
|
||||
file.write_all(&pad)?;
|
||||
file.write_all(&(n as u64).to_le_bytes())?;
|
||||
write_std_header(&mut file, magic, pad, n)?;
|
||||
if extra_header_zeros > 0 {
|
||||
file.write_all(&vec![0u8; extra_header_zeros])?;
|
||||
}
|
||||
|
||||
@@ -617,37 +617,25 @@ impl PersistentSparseCompactIntMatrixBuilder {
|
||||
|
||||
// ── pack_sparse_compact_int_matrix ──────────────────────────────────────────
|
||||
|
||||
/// Converts whichever dense count-matrix format is on disk at `dir`
|
||||
/// (`Packed`, i.e. `matrix.pcmx`, or `Columnar`, i.e. `col_*.pciv` +
|
||||
/// `meta.json`) into the sparse on-disk format in place — generic over the
|
||||
/// dense source the same way [`super::bitmatrix::pack_sparse_bit_matrix`]
|
||||
/// is: detects the format, transposes straight from it, and only removes
|
||||
/// the old dense files once the new format is fully written (a crash
|
||||
/// mid-conversion leaves the previous, still-valid format rather than a
|
||||
/// half-written one). Idempotent — a no-op if `singleton_values.pciv`
|
||||
/// already exists.
|
||||
/// Converts a layer's count matrix (columnar `col_*.pciv` or dense
|
||||
/// `matrix.pcmx`) to the sparse on-disk format in place, through the dense
|
||||
/// packed form, exactly like [`crate::pack_sparse_bit_matrix`]. Old-format
|
||||
/// files are removed only once the new format is fully written.
|
||||
/// Idempotent: a no-op if the sparse format is already present.
|
||||
pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> {
|
||||
use crate::intmatrix::{ColumnarCompactIntMatrix, PackedCompactIntMatrix, col_path};
|
||||
use crate::intmatrix::{PackedCompactIntMatrix, pack_compact_int_matrix};
|
||||
|
||||
if is_present(dir) {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Columnar → dense packed first, then dense packed → sparse: one
|
||||
// mapping (`matrix.pcmx`) instead of one per column file. See
|
||||
// `pack_sparse_bit_matrix` for the rationale and crash-safety argument.
|
||||
pack_compact_int_matrix(dir)?;
|
||||
let packed_path = dir.join("matrix.pcmx");
|
||||
if packed_path.exists() {
|
||||
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?);
|
||||
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
fs::remove_file(&packed_path)?;
|
||||
} else {
|
||||
let dense = PersistentIntMatrix::Columnar(ColumnarCompactIntMatrix::open(dir)?);
|
||||
let n_cols = dense.n_cols();
|
||||
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
for c in 0..n_cols {
|
||||
let _ = fs::remove_file(col_path(dir, c));
|
||||
}
|
||||
let _ = fs::remove_file(dir.join("meta.json"));
|
||||
}
|
||||
Ok(())
|
||||
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?);
|
||||
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||
drop(dense);
|
||||
fs::remove_file(&packed_path)
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
use tempfile::tempdir;
|
||||
|
||||
use crate::MatrixBuilder;
|
||||
|
||||
#[test]
|
||||
fn n_cols_counts_absent_data_and_resumed_columns() {
|
||||
for presence in [true, false] {
|
||||
let dir = tempdir().unwrap();
|
||||
let data = dir.path().join("m");
|
||||
|
||||
let mut mb = MatrixBuilder::new(presence, 5, &data).unwrap();
|
||||
assert_eq!(mb.n_cols(), 0);
|
||||
mb.add_absent_col().unwrap();
|
||||
assert_eq!(mb.n_cols(), 1);
|
||||
let mut c = mb.add_col().unwrap();
|
||||
c.set_val(2, 7);
|
||||
c.close().unwrap();
|
||||
assert_eq!(mb.n_cols(), 2);
|
||||
mb.close().unwrap();
|
||||
|
||||
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
|
||||
assert_eq!(mb.n_cols(), 2, "presence={presence}");
|
||||
mb.add_absent_col().unwrap();
|
||||
assert_eq!(mb.n_cols(), 3);
|
||||
mb.close().unwrap();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
use std::fs;
|
||||
|
||||
use tempfile::tempdir;
|
||||
|
||||
use crate::{
|
||||
MatrixBuilder, MemBitVecBuilder, MemCompactIntVecBuilder, PersistentBitMatrix,
|
||||
PersistentBitVecBuilder, PersistentCompactIntVecBuilder, PersistentIntMatrix,
|
||||
};
|
||||
|
||||
const SIZES: [usize; 6] = [0, 1, 63, 64, 65, 10_000];
|
||||
|
||||
/// Deterministic xorshift64* — no RNG dependency needed for test data.
|
||||
struct Rng(u64);
|
||||
impl Rng {
|
||||
fn next(&mut self) -> u64 {
|
||||
self.0 ^= self.0 >> 12;
|
||||
self.0 ^= self.0 << 25;
|
||||
self.0 ^= self.0 >> 27;
|
||||
self.0.wrapping_mul(0x2545_F491_4F6C_DD1D)
|
||||
}
|
||||
}
|
||||
|
||||
/// `(slot, value)` writes in random order, with overwrites; for counts,
|
||||
/// about a third of the values are ≥ 255 (overflow), enough at n = 10 000
|
||||
/// to exceed `L1_INDEX_ENTRIES` and exercise the sparse index section.
|
||||
fn writes(n: usize, seed: u64, max_small: u64) -> Vec<(usize, u32)> {
|
||||
let mut rng = Rng(seed);
|
||||
(0..2 * n)
|
||||
.map(|_| {
|
||||
let slot = (rng.next() % n as u64) as usize;
|
||||
let r = rng.next();
|
||||
let value = match r % 3 {
|
||||
0 => 255 + (r >> 8) as u32 % 100_000, // overflow
|
||||
1 => 0, // also clears a previous overflow
|
||||
_ => (r >> 8) as u32 % max_small as u32,
|
||||
};
|
||||
(slot, value)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mem_bitvec_is_byte_identical_to_persistent() {
|
||||
for n in SIZES {
|
||||
let dir = tempdir().unwrap();
|
||||
let (pp, mp) = (dir.path().join("p.pbiv"), dir.path().join("m.pbiv"));
|
||||
let mut p = PersistentBitVecBuilder::new(n, &pp).unwrap();
|
||||
let mut m = MemBitVecBuilder::new(n, &mp);
|
||||
for (slot, v) in writes(n, 0x9E37_79B9 + n as u64, 2) {
|
||||
p.set(slot, v > 0);
|
||||
m.set(slot, v > 0);
|
||||
}
|
||||
p.close().unwrap();
|
||||
m.close().unwrap();
|
||||
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mem_compact_int_is_byte_identical_to_persistent() {
|
||||
for n in SIZES {
|
||||
let dir = tempdir().unwrap();
|
||||
let (pp, mp) = (dir.path().join("p.pciv"), dir.path().join("m.pciv"));
|
||||
let mut p = PersistentCompactIntVecBuilder::new(n, &pp).unwrap();
|
||||
let mut m = MemCompactIntVecBuilder::new(n, &mp);
|
||||
for (slot, v) in writes(n, 0xC2B2_AE35 + n as u64, 255) {
|
||||
p.set(slot, v);
|
||||
m.set(slot, v);
|
||||
}
|
||||
p.close().unwrap();
|
||||
m.close().unwrap();
|
||||
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
|
||||
}
|
||||
}
|
||||
|
||||
/// `add_mem_col` columns, interleaved with absent columns and continued
|
||||
/// across a `resume`, read back through the regular matrix readers.
|
||||
#[test]
|
||||
fn add_mem_col_matrix_reads_back() {
|
||||
for presence in [true, false] {
|
||||
let n = 100;
|
||||
let dir = tempdir().unwrap();
|
||||
let data = dir.path().join(if presence { "presence" } else { "counts" });
|
||||
let value = |c: usize, slot: usize| -> u32 {
|
||||
if (slot + c).is_multiple_of(3) { (slot * 7 + c) as u32 % 400 + 1 } else { 0 }
|
||||
};
|
||||
|
||||
let mut mb = MatrixBuilder::new(presence, n, &data).unwrap();
|
||||
mb.add_absent_col().unwrap();
|
||||
let mut c1 = mb.add_mem_col();
|
||||
(0..n).for_each(|s| c1.set_val(s, value(1, s)));
|
||||
c1.close().unwrap();
|
||||
mb.close().unwrap();
|
||||
|
||||
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
|
||||
assert_eq!(mb.n_cols(), 2);
|
||||
let mut c2 = mb.add_mem_col();
|
||||
(0..n).for_each(|s| c2.set_val(s, value(2, s)));
|
||||
c2.close().unwrap();
|
||||
assert_eq!(mb.n_cols(), 3);
|
||||
mb.close().unwrap();
|
||||
|
||||
for s in 0..n {
|
||||
let expect = [0, value(1, s), value(2, s)];
|
||||
if presence {
|
||||
let m = PersistentBitMatrix::open(dir.path()).unwrap();
|
||||
assert_eq!(m.n_cols(), 3);
|
||||
let want: Vec<bool> = expect.iter().map(|&v| v > 0).collect();
|
||||
assert_eq!(&*m.row(s), want.as_slice(), "slot {s}");
|
||||
} else {
|
||||
let m = PersistentIntMatrix::open(dir.path()).unwrap();
|
||||
assert_eq!(m.n_cols(), 3);
|
||||
assert_eq!(&*m.row(s), &expect, "slot {s}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,8 @@ mod eliasfano;
|
||||
mod fixedintvec;
|
||||
mod int_group_ops;
|
||||
mod intmatrix;
|
||||
mod matrix_builder;
|
||||
mod mem_builder;
|
||||
mod rankselect;
|
||||
mod sparse;
|
||||
mod sparse_intmatrix;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "obikmer"
|
||||
version = "1.4.2"
|
||||
version = "1.4.4"
|
||||
edition = "2024"
|
||||
|
||||
[[bin]]
|
||||
|
||||
@@ -15,5 +15,3 @@ obisys = { path = "../obisys", default-features = false }
|
||||
rayon = "1"
|
||||
tracing = "0.1.44"
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
@@ -1,605 +0,0 @@
|
||||
//! Merging a source partition's new layer into a destination partition's
|
||||
//! de Bruijn graph union (pass 1) then column fill (pass 2). Free function,
|
||||
//! not a `KmerIndex` method — `KmerIndex` stays the stateless
|
||||
//! `Index { Partition { Layer } }` data model; this is one of the
|
||||
//! algorithms that reads/writes through it, living here in `obikmerge`
|
||||
//! instead.
|
||||
|
||||
use std::fs;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use obipipeline::{
|
||||
Pipeline, PipelineError, PipelineSender, SharedFlatFn, Stage, ThrottleGuard, WorkerPool,
|
||||
make_sink, make_source, make_transform, throttle,
|
||||
};
|
||||
use tracing::debug;
|
||||
|
||||
use obicompactvec::{ColBuilder, MatrixBuilder};
|
||||
use obikindex::layer::{IndexMode, KmerLayer, TypedLayer};
|
||||
use obikindex::{KmerIndex, OKIError, OKIResult};
|
||||
use obikindexer::{build_graph, materialize_layer};
|
||||
use obikseq::CanonicalKmer;
|
||||
use obiskio::UnitigFileReader;
|
||||
|
||||
use crate::MergeMode;
|
||||
|
||||
// `MatrixBuilder::new`/`resume` take a `presence: bool` — `MergeMode` maps
|
||||
// to it directly (`Presence` → `true`, `Count` → `false`).
|
||||
fn presence(mode: MergeMode) -> bool {
|
||||
mode == MergeMode::Presence
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod matrix_builder_tests {
|
||||
use tempfile::tempdir;
|
||||
|
||||
use obicompactvec::{PersistentBitMatrix, PersistentIntMatrix};
|
||||
|
||||
use super::{ColBuilder, MatrixBuilder};
|
||||
|
||||
/// Mirrors `merge_partition`'s "new layer" setup: absent (dst-genome)
|
||||
/// columns get no data, then source columns are filled — all through one
|
||||
/// continuous `MatrixBuilder` session, closed once at the end.
|
||||
#[test]
|
||||
fn new_layer_absent_then_source_columns_presence() {
|
||||
let dir = tempdir().unwrap();
|
||||
let data_dir = dir.path().join("presence");
|
||||
let mut mb = MatrixBuilder::new(true, 3, &data_dir).unwrap();
|
||||
|
||||
// Two absent (dst-genome) columns.
|
||||
mb.add_absent_col().unwrap();
|
||||
mb.add_absent_col().unwrap();
|
||||
|
||||
// One source column, filled like pass 2 would.
|
||||
let mut col = mb.add_col().unwrap();
|
||||
match &mut col {
|
||||
ColBuilder::Bit(b) => {
|
||||
b.set(0, true);
|
||||
b.set(1, false);
|
||||
b.set(2, true);
|
||||
}
|
||||
ColBuilder::Int(_) => unreachable!(),
|
||||
}
|
||||
col.close().unwrap();
|
||||
mb.close().unwrap();
|
||||
|
||||
let m = PersistentBitMatrix::open(dir.path()).unwrap();
|
||||
assert_eq!(m.n_cols(), 3);
|
||||
assert_eq!(&*m.row(0), &[false, false, true]);
|
||||
assert_eq!(&*m.row(1), &[false, false, false]);
|
||||
assert_eq!(&*m.row(2), &[false, false, true]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn new_layer_absent_then_source_columns_count() {
|
||||
let dir = tempdir().unwrap();
|
||||
let data_dir = dir.path().join("counts");
|
||||
let mut mb = MatrixBuilder::new(false, 2, &data_dir).unwrap();
|
||||
|
||||
mb.add_absent_col().unwrap();
|
||||
|
||||
let mut col = mb.add_col().unwrap();
|
||||
match &mut col {
|
||||
ColBuilder::Int(b) => {
|
||||
b.set(0, 7);
|
||||
b.set(1, 42);
|
||||
}
|
||||
ColBuilder::Bit(_) => unreachable!(),
|
||||
}
|
||||
col.close().unwrap();
|
||||
mb.close().unwrap();
|
||||
|
||||
let m = PersistentIntMatrix::open(dir.path()).unwrap();
|
||||
assert_eq!(m.n_cols(), 2);
|
||||
assert_eq!(&*m.row(0), &[0u32, 7]);
|
||||
assert_eq!(&*m.row(1), &[0u32, 42]);
|
||||
}
|
||||
|
||||
/// Mirrors `merge_partition`'s "existing dst layer" setup: an
|
||||
/// already-closed matrix (from a previous merge) gets more columns
|
||||
/// appended via `resume`, without callers ever seeing `col_path`/
|
||||
/// `MatrixMeta` — `n` itself is read back from the matrix directory.
|
||||
#[test]
|
||||
fn resume_appends_source_columns_to_existing_layer() {
|
||||
let dir = tempdir().unwrap();
|
||||
let data_dir = dir.path().join("presence");
|
||||
|
||||
// Previous merge: one dst-genome column already on disk.
|
||||
let mut mb0 = MatrixBuilder::new(true, 3, &data_dir).unwrap();
|
||||
mb0.add_absent_col().unwrap();
|
||||
mb0.close().unwrap();
|
||||
|
||||
// This merge: resume and append two more source columns.
|
||||
let mut mb = MatrixBuilder::resume(true, &data_dir).unwrap();
|
||||
for vals in [[true, false, true], [false, true, false]] {
|
||||
let mut col = mb.add_col().unwrap();
|
||||
match &mut col {
|
||||
ColBuilder::Bit(b) => {
|
||||
for (slot, v) in vals.into_iter().enumerate() {
|
||||
b.set(slot, v);
|
||||
}
|
||||
}
|
||||
ColBuilder::Int(_) => unreachable!(),
|
||||
}
|
||||
col.close().unwrap();
|
||||
}
|
||||
mb.close().unwrap();
|
||||
|
||||
let m = PersistentBitMatrix::open(dir.path()).unwrap();
|
||||
assert_eq!(m.n_cols(), 3);
|
||||
assert_eq!(&*m.row(0), &[false, true, false]);
|
||||
assert_eq!(&*m.row(1), &[false, false, true]);
|
||||
assert_eq!(&*m.row(2), &[false, true, false]);
|
||||
}
|
||||
}
|
||||
|
||||
// ── dst cross-layer lookup ──────────────────────────────────────────────────
|
||||
|
||||
/// Batch membership lookup across `dst_layers` — first layer that carries
|
||||
/// each kmer wins. Groups hits by layer index (as `(kmer batch index,
|
||||
/// slot)` pairs) rather than returning them in query order, so pass 2 can
|
||||
/// follow with one column-major write per `(layer, column)` instead of
|
||||
/// interleaving across layers kmer by kmer. The kmers with no hit in any
|
||||
/// layer come back as a separate index list, for the new-layer MPHF batch
|
||||
/// lookup on a miss.
|
||||
fn find_batch_grouped(
|
||||
layers: &[KmerLayer],
|
||||
kmers: &[CanonicalKmer],
|
||||
) -> (Vec<Vec<(usize, usize)>>, Vec<usize>) {
|
||||
let mut by_layer: Vec<Vec<(usize, usize)>> = vec![Vec::new(); layers.len()];
|
||||
let mut not_found: Vec<usize> = Vec::new();
|
||||
for (idx, &kmer) in kmers.iter().enumerate() {
|
||||
match layers
|
||||
.iter()
|
||||
.enumerate()
|
||||
.find_map(|(li, l)| l.find_slot(kmer).map(|slot| (li, slot)))
|
||||
{
|
||||
Some((li, slot)) => by_layer[li].push((idx, slot)),
|
||||
None => not_found.push(idx),
|
||||
}
|
||||
}
|
||||
(by_layer, not_found)
|
||||
}
|
||||
|
||||
// ── merge_partition ────────────────────────────────────────────────────────
|
||||
|
||||
/// Merge `sources` into destination partition `i`.
|
||||
///
|
||||
/// Each entry in `sources` is `(index, n_genomes)` where `n_genomes` is the
|
||||
/// number of genome columns that source contributes. A merged index
|
||||
/// contributes more than one. The total new columns added to the
|
||||
/// destination is `sum(n_genomes)`.
|
||||
///
|
||||
/// `n_dst_genomes` is the number of genome columns already in the
|
||||
/// destination matrices (copied from the bootstrap source before this
|
||||
/// call).
|
||||
pub(crate) fn merge_partition(
|
||||
dst: &KmerIndex,
|
||||
i: usize,
|
||||
sources: &[(&KmerIndex, usize)],
|
||||
mode: MergeMode,
|
||||
n_dst_genomes: usize,
|
||||
block_bits: u8,
|
||||
evidence: &IndexMode,
|
||||
) -> OKIResult<usize> {
|
||||
let dst_partition = dst.partition(i)?;
|
||||
let n_dst_layers = dst_partition.n_layers();
|
||||
let n_src_total: usize = sources.iter().map(|(_, n)| *n).sum();
|
||||
|
||||
// First merge in presence mode: materialize implicit single-genome
|
||||
// presence matrices on existing layers (all slots true — every kmer in
|
||||
// those layers belongs to genome_0) so a resumed builder can append
|
||||
// source columns onto them below.
|
||||
if n_dst_genomes == 1 && mode == MergeMode::Presence {
|
||||
for l in 0..n_dst_layers {
|
||||
let layer = dst_partition.layer(l)?.open()?;
|
||||
TypedLayer::<()>::init_presence_matrix(layer.dir(), layer.n())?;
|
||||
}
|
||||
}
|
||||
|
||||
// dst's `0..n_dst_layers` layers and every source's own layers, for
|
||||
// this one partition — all opened the same direct way, not via
|
||||
// `IndexCache`. Every layer read here (dst's pre-existing ones, and
|
||||
// every source's) is just as stable for the duration of this call: dst
|
||||
// only ever gets a brand new layer *appended* below, never one of these
|
||||
// read from, so there's no "dst isn't stable" distinction between the
|
||||
// two loops — `IndexCache` still doesn't fit either one, for a single,
|
||||
// uniform reason: `obikindexer::build_graph`/`obipipeline::WorkerPool`
|
||||
// (pass 1/2 below) are plain-OS-thread pipelines requiring `Send + Sync
|
||||
// + 'static`, and `IndexCache<'_>` always borrows `&KmerIndex` — no
|
||||
// amount of logical stability changes that a borrowed cache can't cross
|
||||
// that boundary, only owned `KmerLayer`s can (each owns its own mmap'd
|
||||
// files, no reference back to the index it came from).
|
||||
let dst_layers: Vec<KmerLayer> = (0..n_dst_layers)
|
||||
.map(|l| dst_partition.layer(l)?.open())
|
||||
.collect::<OKIResult<_>>()?;
|
||||
let dst_layers = Arc::new(dst_layers);
|
||||
|
||||
let src_layers: Vec<Vec<Arc<KmerLayer>>> = sources
|
||||
.iter()
|
||||
.map(|(src, _)| -> OKIResult<Vec<Arc<KmerLayer>>> {
|
||||
let partition = src.partition(i)?;
|
||||
(0..partition.n_layers())
|
||||
.map(|l| partition.layer(l)?.open().map(Arc::new))
|
||||
.collect()
|
||||
})
|
||||
.collect::<OKIResult<_>>()?;
|
||||
|
||||
// ── Pass 1: pipeline — parallel file read + dst filter + graph fill ────
|
||||
//
|
||||
// Source : list of unitigs.bin paths (one per source × layer)
|
||||
// Flat : open file, emit Vec<CanonicalKmer> batches (BeeGFS parallel I/O)
|
||||
// Transform: filter via dst_layers — Arc<[KmerLayer]>: Sync
|
||||
// Sink : push new kmers into GraphDeBruijn (single thread, no locks needed)
|
||||
|
||||
let mut unitig_paths: Vec<PathBuf> = Vec::new();
|
||||
for layers in &src_layers {
|
||||
for layer in layers {
|
||||
let p = layer.unitigs_path();
|
||||
if p.exists() {
|
||||
unitig_paths.push(p);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let n_src_layers = unitig_paths.len();
|
||||
debug!("partition {i}: de Bruijn graph build start — {n_src_layers} source layer(s)");
|
||||
|
||||
const BATCH: usize = 4096;
|
||||
let n_workers = rayon::current_num_threads().min(16).max(4);
|
||||
// At most 2 files open simultaneously: keeps n_workers-2 workers free
|
||||
// for the Transform stage. Each open file monopolises one worker for the
|
||||
// full duration of its read, so this must stay well below n_workers.
|
||||
let max_open = 2;
|
||||
|
||||
let dst_filter = Arc::clone(&dst_layers);
|
||||
|
||||
let g = build_graph(
|
||||
unitig_paths.into_iter(),
|
||||
move |path: PathBuf, emit: &mut dyn FnMut(Vec<CanonicalKmer>)| -> OKIResult<()> {
|
||||
let reader = UnitigFileReader::open_sequential(&path)?;
|
||||
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
|
||||
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
|
||||
batch.push(kmer);
|
||||
if batch.len() == BATCH {
|
||||
emit(std::mem::replace(&mut batch, Vec::with_capacity(BATCH)));
|
||||
}
|
||||
}
|
||||
if !batch.is_empty() {
|
||||
emit(batch);
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
move |kmer| dst_filter.iter().all(|l| l.find_slot(kmer).is_none()),
|
||||
n_workers,
|
||||
max_open,
|
||||
)?;
|
||||
|
||||
let any_new = g.len() > 0;
|
||||
debug!(
|
||||
"partition {i}: de Bruijn graph done — {} new kmers",
|
||||
g.len()
|
||||
);
|
||||
|
||||
// Build new layer from de Bruijn graph if there are new kmers.
|
||||
let new_layer_idx = n_dst_layers;
|
||||
let new_layer_dir = KmerLayer::new(&dst_partition, new_layer_idx)
|
||||
.dir()
|
||||
.to_path_buf();
|
||||
|
||||
let n_new = if any_new {
|
||||
debug!("partition {i}: unitig traversal start — {} nodes", g.len());
|
||||
let n_nodes = materialize_layer(g, &new_layer_dir, block_bits, evidence)?;
|
||||
debug!("partition {i}: MPHF build done");
|
||||
n_nodes
|
||||
} else {
|
||||
drop(g);
|
||||
0
|
||||
};
|
||||
|
||||
let t_open = std::time::Instant::now();
|
||||
let new_mphf: Option<Arc<obikindex::layer::MphfOnly>> = if any_new {
|
||||
Some(Arc::new(obikindex::layer::MphfOnly::open(&new_layer_dir)?))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
debug!(
|
||||
"partition {i}: MPHF open in {:.3}s",
|
||||
t_open.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
// ── Prepare matrix directories for the new layer ──────────────────────
|
||||
// Absent columns (dst genomes) get an all-zero/false column. Source-genome
|
||||
// columns are created as mutable builders for pass 2. `new_mb` is kept
|
||||
// open (not `close`d) until pass 2 has filled every source column, so its
|
||||
// `n_cols` bookkeeping stays in sync with the columns actually created.
|
||||
let (new_src_builders, new_mb): (Vec<ColBuilder>, Option<MatrixBuilder>) = if any_new {
|
||||
let data_dir = match mode {
|
||||
MergeMode::Presence => new_layer_dir.join("presence"),
|
||||
MergeMode::Count => new_layer_dir.join("counts"),
|
||||
};
|
||||
fs::create_dir_all(&data_dir)?;
|
||||
let mut mb = MatrixBuilder::new(presence(mode), n_new, &data_dir).map_err(OKIError::Io)?;
|
||||
for _ in 0..n_dst_genomes {
|
||||
mb.add_absent_col().map_err(OKIError::Io)?;
|
||||
}
|
||||
let cols = (0..n_src_total)
|
||||
.map(|_| mb.add_col().map_err(OKIError::Io))
|
||||
.collect::<OKIResult<Vec<_>>>()?;
|
||||
(cols, Some(mb))
|
||||
} else {
|
||||
(vec![], None)
|
||||
};
|
||||
|
||||
let t_builders = std::time::Instant::now();
|
||||
// Builders for existing layers: n_src_total per layer, resumed from
|
||||
// each layer's own matrix directory (already holding n_dst_genomes
|
||||
// columns from a previous merge). Columns land at
|
||||
// n_dst_genomes .. n_dst_genomes + n_src_total - 1.
|
||||
let mut exist_mbs: Vec<MatrixBuilder> = Vec::with_capacity(n_dst_layers);
|
||||
let mut exist_builders: Vec<Vec<ColBuilder>> = Vec::with_capacity(n_dst_layers);
|
||||
for l in 0..n_dst_layers {
|
||||
let layer_dir = dst_layers[l].dir().to_path_buf();
|
||||
let data_dir = match mode {
|
||||
MergeMode::Presence => layer_dir.join("presence"),
|
||||
MergeMode::Count => layer_dir.join("counts"),
|
||||
};
|
||||
let mut mb = MatrixBuilder::resume(presence(mode), &data_dir).map_err(OKIError::Io)?;
|
||||
let cols = (0..n_src_total)
|
||||
.map(|_| mb.add_col().map_err(OKIError::Io))
|
||||
.collect::<OKIResult<Vec<_>>>()?;
|
||||
exist_mbs.push(mb);
|
||||
exist_builders.push(cols);
|
||||
}
|
||||
|
||||
debug!(
|
||||
"partition {i}: builders ready in {:.3}s",
|
||||
t_builders.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
// ── Pass 2: fill builders (pipeline) ─────────────────────────────────
|
||||
let t_pass2 = std::time::Instant::now();
|
||||
// Collect source items before the pipeline so errors propagate via ?
|
||||
// before any worker thread is spawned.
|
||||
let mut pass2_items: Vec<(usize, usize, PathBuf, Arc<KmerLayer>)> = Vec::new();
|
||||
{
|
||||
let mut col_offset = 0usize;
|
||||
for ((_, src_n), layers) in sources.iter().zip(src_layers.iter()) {
|
||||
for layer in layers {
|
||||
let unitigs_path = layer.unitigs_path();
|
||||
if unitigs_path.exists() {
|
||||
pass2_items.push((col_offset, *src_n, unitigs_path, Arc::clone(layer)));
|
||||
}
|
||||
}
|
||||
col_offset += src_n;
|
||||
}
|
||||
}
|
||||
|
||||
enum Pass2Data {
|
||||
SrcLayer((usize, usize, PathBuf, Arc<KmerLayer>, ThrottleGuard)),
|
||||
RawBatch((usize, usize, Arc<KmerLayer>, Vec<CanonicalKmer>)),
|
||||
/// One entry per `(dst layer, absolute column)` touched by this
|
||||
/// batch, each carrying every `(slot, value)` pair for that column
|
||||
/// together — column-major, so the sink below locks that column's
|
||||
/// `ColBuilder` once per batch and writes the whole run, instead of
|
||||
/// once per `(kmer, genome)` pair.
|
||||
WriteBatch(Vec<(Option<usize>, usize, Vec<(usize, u32)>)>),
|
||||
}
|
||||
|
||||
let exist_locked: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_builders
|
||||
.into_iter()
|
||||
.map(|layer| layer.into_iter().map(|b| Arc::new(Mutex::new(b))).collect())
|
||||
.collect();
|
||||
let new_locked: Vec<Arc<Mutex<ColBuilder>>> = new_src_builders
|
||||
.into_iter()
|
||||
.map(|b| Arc::new(Mutex::new(b)))
|
||||
.collect();
|
||||
let exist_sink: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_locked
|
||||
.iter()
|
||||
.map(|layer| layer.iter().map(Arc::clone).collect())
|
||||
.collect();
|
||||
let new_sink: Vec<Arc<Mutex<ColBuilder>>> = new_locked.iter().map(Arc::clone).collect();
|
||||
let dst_layers_t2 = Arc::clone(&dst_layers);
|
||||
let new_mphf_t2 = new_mphf.clone();
|
||||
let pass2_err: Arc<Mutex<Option<String>>> = Arc::new(Mutex::new(None));
|
||||
let err_cap2 = Arc::clone(&pass2_err);
|
||||
|
||||
let capacity = 2;
|
||||
let throttled_pass2 = throttle(pass2_items.into_iter(), max_open);
|
||||
|
||||
let pipeline2 = Pipeline::new(
|
||||
make_source!(
|
||||
Pass2Data,
|
||||
throttled_pass2.map(|t| {
|
||||
let (col_offset, src_n, unitigs_path, src_layer) = t.item;
|
||||
(col_offset, src_n, unitigs_path, src_layer, t.guard)
|
||||
}),
|
||||
SrcLayer
|
||||
),
|
||||
vec![
|
||||
Stage::Flat(Arc::new(
|
||||
move |data: Pass2Data,
|
||||
push: &PipelineSender<Result<Pass2Data, PipelineError>>,
|
||||
delta: &PipelineSender<isize>| {
|
||||
if let Pass2Data::SrcLayer((
|
||||
col_offset,
|
||||
src_n,
|
||||
unitigs_path,
|
||||
src_layer,
|
||||
_guard,
|
||||
)) = data
|
||||
{
|
||||
// _guard dropped at end of block, releasing the slot.
|
||||
// `src_layer` (MPHF + matrix) was already opened up
|
||||
// front — only the unitigs.bin stream itself is
|
||||
// opened (and throttled) here.
|
||||
let reader = match UnitigFileReader::open_sequential(&unitigs_path) {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
*err_cap2.lock().unwrap() = Some(e.to_string());
|
||||
delta.send(-1).ok();
|
||||
return;
|
||||
}
|
||||
};
|
||||
const BATCH: usize = 4096;
|
||||
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
|
||||
let mut count: isize = 0;
|
||||
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
|
||||
batch.push(kmer);
|
||||
if batch.len() == BATCH {
|
||||
let b = std::mem::replace(&mut batch, Vec::with_capacity(BATCH));
|
||||
push.send(Ok(Pass2Data::RawBatch((
|
||||
col_offset,
|
||||
src_n,
|
||||
Arc::clone(&src_layer),
|
||||
b,
|
||||
))))
|
||||
.ok();
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
if !batch.is_empty() {
|
||||
push.send(Ok(Pass2Data::RawBatch((
|
||||
col_offset, src_n, src_layer, batch,
|
||||
))))
|
||||
.ok();
|
||||
count += 1;
|
||||
}
|
||||
delta.send(count - 1).ok();
|
||||
}
|
||||
},
|
||||
) as SharedFlatFn<Pass2Data>),
|
||||
make_transform!(
|
||||
Pass2Data,
|
||||
{
|
||||
move |(col_offset, src_n, src_layer, kmers): (
|
||||
usize,
|
||||
usize,
|
||||
Arc<KmerLayer>,
|
||||
Vec<CanonicalKmer>,
|
||||
)|
|
||||
-> Vec<(Option<usize>, usize, Vec<(usize, u32)>)> {
|
||||
// Column-first, one pass for the whole batch: a single
|
||||
// MPHF hash sweep (`hash_batch`) instead of one hash
|
||||
// per kmer, then one read pass per source genome
|
||||
// column (`fill_sub_matrix`) instead of one row read
|
||||
// per kmer — matches the underlying matrix's own
|
||||
// column-major layout.
|
||||
let slots = src_layer.hash_batch(&kmers);
|
||||
let mut cols: Vec<Vec<u32>> = (0..src_n)
|
||||
.map(|_| Vec::with_capacity(kmers.len()))
|
||||
.collect();
|
||||
src_layer.fill_sub_matrix(&slots, &mut cols);
|
||||
|
||||
// Membership against dst, grouped by dst layer
|
||||
// instead of interleaved kmer by kmer — same
|
||||
// grouping `PartitionCache::find_presence_batch`
|
||||
// does in `obikphylo`, via the shared primitive.
|
||||
let (by_dst_layer, not_found) = find_batch_grouped(&dst_layers_t2, &kmers);
|
||||
|
||||
// Genome-major within each layer group (not kmer-major)
|
||||
// so every `(layer, col)` pair's values land together —
|
||||
// one entry per column, not one per `(kmer, genome)`.
|
||||
let mut ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)> =
|
||||
Vec::with_capacity(src_n * (by_dst_layer.len() + 1));
|
||||
for (dst_layer, hits) in by_dst_layer.into_iter().enumerate() {
|
||||
if hits.is_empty() {
|
||||
continue;
|
||||
}
|
||||
for g in 0..src_n {
|
||||
let col_vals: Vec<(usize, u32)> = hits
|
||||
.iter()
|
||||
.map(|&(idx, slot)| (slot, cols[g][idx]))
|
||||
.collect();
|
||||
ops.push((Some(dst_layer), col_offset + g, col_vals));
|
||||
}
|
||||
}
|
||||
if let (Some(mphf), false) = (&new_mphf_t2, not_found.is_empty()) {
|
||||
let not_found_kmers: Vec<CanonicalKmer> =
|
||||
not_found.iter().map(|&idx| kmers[idx]).collect();
|
||||
let new_slots = mphf.index_batch(¬_found_kmers);
|
||||
for g in 0..src_n {
|
||||
let col_vals: Vec<(usize, u32)> = not_found
|
||||
.iter()
|
||||
.zip(new_slots.iter())
|
||||
.map(|(&idx, &slot)| (slot, cols[g][idx]))
|
||||
.collect();
|
||||
ops.push((None, col_offset + g, col_vals));
|
||||
}
|
||||
}
|
||||
ops
|
||||
}
|
||||
},
|
||||
RawBatch,
|
||||
WriteBatch
|
||||
),
|
||||
],
|
||||
make_sink!(
|
||||
Pass2Data,
|
||||
{
|
||||
move |ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)>| {
|
||||
for (layer_opt, col, col_vals) in ops {
|
||||
let builder = match layer_opt {
|
||||
Some(l) => &exist_sink[l][col],
|
||||
None => &new_sink[col],
|
||||
};
|
||||
let mut b = builder.lock().unwrap();
|
||||
for (slot, val) in col_vals {
|
||||
b.set_val(slot, val);
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
WriteBatch
|
||||
),
|
||||
);
|
||||
|
||||
WorkerPool::new(pipeline2, n_workers, capacity).run();
|
||||
debug!(
|
||||
"partition {i}: pass2 pipeline done in {:.3}s",
|
||||
t_pass2.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
if let Some(msg) = Arc::try_unwrap(pass2_err)
|
||||
.unwrap_or_else(|_| panic!("pass2: pass2_err not uniquely owned"))
|
||||
.into_inner()
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
{
|
||||
return Err(OKIError::InvalidData {
|
||||
context: "merge pass2",
|
||||
detail: msg,
|
||||
});
|
||||
}
|
||||
|
||||
let t_close = std::time::Instant::now();
|
||||
// ── Close builders ──────────────────────────────────────────────────
|
||||
for (mb, builders) in exist_mbs.into_iter().zip(exist_locked.into_iter()) {
|
||||
for b in builders {
|
||||
Arc::try_unwrap(b)
|
||||
.unwrap_or_else(|_| panic!("pass2: exist_builder not uniquely owned"))
|
||||
.into_inner()
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
.close()?;
|
||||
}
|
||||
mb.close().map_err(OKIError::Io)?;
|
||||
}
|
||||
|
||||
for b in new_locked {
|
||||
Arc::try_unwrap(b)
|
||||
.unwrap_or_else(|_| panic!("pass2: new_builder not uniquely owned"))
|
||||
.into_inner()
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
.close()?;
|
||||
}
|
||||
if let Some(mb) = new_mb {
|
||||
mb.close().map_err(OKIError::Io)?;
|
||||
}
|
||||
|
||||
debug!(
|
||||
"partition {i}: builders closed in {:.3}s",
|
||||
t_close.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
Ok(n_new)
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
//! Merging a source partition's new layer into a destination partition's
|
||||
//! de Bruijn graph union (pass 1) then column fill (pass 2). Free function,
|
||||
//! not a `KmerIndex` method — `KmerIndex` stays the stateless
|
||||
//! `Index { Partition { Layer } }` data model; this is one of the
|
||||
//! algorithms that reads/writes through it, living here in `obikmerge`
|
||||
//! instead.
|
||||
//!
|
||||
//! Split: this file orchestrates one partition (pass 1, new-layer setup,
|
||||
//! loop over source groups); [`pass2`] holds the per-group column-fill
|
||||
//! pipeline.
|
||||
|
||||
mod pass2;
|
||||
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use tracing::debug;
|
||||
|
||||
use obicompactvec::{ColBuilder, MatrixBuilder};
|
||||
use obikindex::layer::{IndexMode, KmerLayer, TypedLayer};
|
||||
use obikindex::{KmerIndex, OKIError, OKIResult};
|
||||
use obikindexer::{build_graph, materialize_layer};
|
||||
use obikseq::CanonicalKmer;
|
||||
use obiskio::UnitigFileReader;
|
||||
|
||||
use crate::MergeMode;
|
||||
use pass2::run_pass2;
|
||||
|
||||
// `MatrixBuilder::new`/`resume` take a `presence: bool` — `MergeMode` maps
|
||||
// to it directly (`Presence` → `true`, `Count` → `false`).
|
||||
fn presence(mode: MergeMode) -> bool {
|
||||
mode == MergeMode::Presence
|
||||
}
|
||||
|
||||
// ── merge_partition ────────────────────────────────────────────────────────
|
||||
|
||||
/// Merge `sources` into destination partition `i`.
|
||||
///
|
||||
/// Each entry in `sources` is `(index, n_genomes)` where `n_genomes` is the
|
||||
/// number of genome columns that source contributes. A merged index
|
||||
/// contributes more than one. The total new columns added to the
|
||||
/// destination is `sum(n_genomes)`.
|
||||
///
|
||||
/// `n_dst_genomes` is the number of genome columns already in the
|
||||
/// destination matrices (copied from the bootstrap source before this
|
||||
/// call).
|
||||
pub(crate) fn merge_partition(
|
||||
dst: &KmerIndex,
|
||||
i: usize,
|
||||
sources: &[(&KmerIndex, usize)],
|
||||
mode: MergeMode,
|
||||
n_dst_genomes: usize,
|
||||
block_bits: u8,
|
||||
evidence: &IndexMode,
|
||||
) -> OKIResult<usize> {
|
||||
let dst_partition = dst.partition(i)?;
|
||||
let n_dst_layers = dst_partition.n_layers();
|
||||
|
||||
// First merge in presence mode: materialize implicit single-genome
|
||||
// presence matrices on existing layers (all slots true — every kmer in
|
||||
// those layers belongs to genome_0) so a resumed builder can append
|
||||
// source columns onto them below.
|
||||
if n_dst_genomes == 1 && mode == MergeMode::Presence {
|
||||
for l in 0..n_dst_layers {
|
||||
let layer = dst_partition.layer(l)?.open()?;
|
||||
TypedLayer::<()>::init_presence_matrix(layer.dir(), layer.n())?;
|
||||
}
|
||||
}
|
||||
|
||||
// dst's `0..n_dst_layers` layers, opened once for the whole partition
|
||||
// and shared by pass 1 (filter) and every pass-2 group (membership).
|
||||
// Opened directly, not via `IndexCache`: `obikindexer::build_graph`/
|
||||
// `obipipeline::WorkerPool` are plain-OS-thread pipelines requiring
|
||||
// `Send + Sync + 'static`, and `IndexCache<'_>` always borrows
|
||||
// `&KmerIndex` — only owned `KmerLayer`s can cross that boundary.
|
||||
let dst_layers: Vec<KmerLayer> = (0..n_dst_layers)
|
||||
.map(|l| dst_partition.layer(l)?.open())
|
||||
.collect::<OKIResult<_>>()?;
|
||||
let dst_layers = Arc::new(dst_layers);
|
||||
|
||||
// ── Pass 1: pipeline — parallel file read + dst filter + graph fill ────
|
||||
//
|
||||
// Source : list of unitigs.bin paths (one per source × layer)
|
||||
// Flat : open file, emit Vec<CanonicalKmer> batches (BeeGFS parallel I/O)
|
||||
// Transform: filter via dst_layers — Arc<[KmerLayer]>: Sync
|
||||
// Sink : push new kmers into GraphDeBruijn (single thread, no locks needed)
|
||||
//
|
||||
// Only the unitigs.bin *paths* are needed here: no source layer is
|
||||
// opened (`KmerLayer::new` is the unopened `Empty` variant, whose
|
||||
// `unitigs_path` is valid), so pass 1 holds no source mapping.
|
||||
|
||||
let mut unitig_paths: Vec<PathBuf> = Vec::new();
|
||||
for (src, _) in sources {
|
||||
let partition = src.partition(i)?;
|
||||
for l in 0..partition.n_layers() {
|
||||
let p = KmerLayer::new(&partition, l).unitigs_path();
|
||||
if p.exists() {
|
||||
unitig_paths.push(p);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let n_src_layers = unitig_paths.len();
|
||||
debug!("partition {i}: de Bruijn graph build start — {n_src_layers} source layer(s)");
|
||||
|
||||
const BATCH: usize = 4096;
|
||||
let n_workers = rayon::current_num_threads().min(16).max(4);
|
||||
// At most 2 files open simultaneously: keeps n_workers-2 workers free
|
||||
// for the Transform stage. Each open file monopolises one worker for the
|
||||
// full duration of its read, so this must stay well below n_workers.
|
||||
let max_open = 2;
|
||||
|
||||
let dst_filter = Arc::clone(&dst_layers);
|
||||
|
||||
let g = build_graph(
|
||||
unitig_paths.into_iter(),
|
||||
move |path: PathBuf, emit: &mut dyn FnMut(Vec<CanonicalKmer>)| -> OKIResult<()> {
|
||||
let reader = UnitigFileReader::open_sequential(&path)?;
|
||||
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
|
||||
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
|
||||
batch.push(kmer);
|
||||
if batch.len() == BATCH {
|
||||
emit(std::mem::replace(&mut batch, Vec::with_capacity(BATCH)));
|
||||
}
|
||||
}
|
||||
if !batch.is_empty() {
|
||||
emit(batch);
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
move |kmer| dst_filter.iter().all(|l| l.find_slot(kmer).is_none()),
|
||||
n_workers,
|
||||
max_open,
|
||||
)?;
|
||||
|
||||
let any_new = g.len() > 0;
|
||||
debug!(
|
||||
"partition {i}: de Bruijn graph done — {} new kmers",
|
||||
g.len()
|
||||
);
|
||||
|
||||
// Build new layer from de Bruijn graph if there are new kmers.
|
||||
let new_layer_idx = n_dst_layers;
|
||||
let new_layer_dir = KmerLayer::new(&dst_partition, new_layer_idx)
|
||||
.dir()
|
||||
.to_path_buf();
|
||||
|
||||
let n_new = if any_new {
|
||||
debug!("partition {i}: unitig traversal start — {} nodes", g.len());
|
||||
let n_nodes = materialize_layer(g, &new_layer_dir, block_bits, evidence)?;
|
||||
debug!("partition {i}: MPHF build done");
|
||||
n_nodes
|
||||
} else {
|
||||
drop(g);
|
||||
0
|
||||
};
|
||||
|
||||
let t_open = std::time::Instant::now();
|
||||
let new_mphf: Option<Arc<obikindex::layer::MphfOnly>> = if any_new {
|
||||
Some(Arc::new(obikindex::layer::MphfOnly::open(&new_layer_dir)?))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
debug!(
|
||||
"partition {i}: MPHF open in {:.3}s",
|
||||
t_open.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
let matrix_dir = |layer_dir: &Path| match mode {
|
||||
MergeMode::Presence => layer_dir.join("presence"),
|
||||
MergeMode::Count => layer_dir.join("counts"),
|
||||
};
|
||||
|
||||
// ── New layer matrix: absent (dst-genome) columns only ────────────────
|
||||
// Created and closed here; every source group below resumes it and
|
||||
// appends its own columns, exactly like the existing dst layers.
|
||||
let new_data_dir: Option<PathBuf> = if any_new {
|
||||
let data_dir = matrix_dir(&new_layer_dir);
|
||||
fs::create_dir_all(&data_dir)?;
|
||||
let mut mb = MatrixBuilder::new(presence(mode), n_new, &data_dir).map_err(OKIError::Io)?;
|
||||
for _ in 0..n_dst_genomes {
|
||||
mb.add_absent_col().map_err(OKIError::Io)?;
|
||||
}
|
||||
mb.close().map_err(OKIError::Io)?;
|
||||
Some(data_dir)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let exist_data_dirs: Vec<PathBuf> = dst_layers.iter().map(|l| matrix_dir(l.dir())).collect();
|
||||
|
||||
// ── Pass 2, by consecutive groups of sources ──────────────────────────
|
||||
//
|
||||
// Only one group's source layers and destination columns are alive at a
|
||||
// time, so the number of live mappings per partition no longer grows
|
||||
// with the number of sources. SRC_GROUP matches `max_open`: pass 2
|
||||
// never streams more than `max_open` unitigs.bin files at once, so a
|
||||
// larger group would only keep more layers/columns open without adding
|
||||
// read parallelism.
|
||||
//
|
||||
// Invariant: groups are visited in source order and each appends its
|
||||
// `group_n` columns in source order, so the absolute column of genome
|
||||
// `g` of a source is `n_dst_genomes + col_base + offset_local + g` —
|
||||
// the same column as when every source was processed in one session.
|
||||
//
|
||||
// Memory: the group's columns are built in RAM (`add_mem_col`), at most
|
||||
// `SRC_GROUP × src_n × (L_dst + 1)` live columns per partition, each of
|
||||
// `n_layer / 8` bytes (presence) or `n_layer` bytes + overflow (counts):
|
||||
// about `SRC_GROUP × src_n × N_part × (1/8 | 1)` bytes, `N_part` being
|
||||
// the partition's k-mer count over all dst layers. For mono-genome
|
||||
// sources this stays below pass 1's de Bruijn graph (≥ ~19 B per new
|
||||
// k-mer) unless the new layer is tiny next to the existing ones
|
||||
// (incremental merge into a large index).
|
||||
const SRC_GROUP: usize = 2;
|
||||
let t_pass2 = std::time::Instant::now();
|
||||
let mut col_base = 0usize;
|
||||
for group in sources.chunks(SRC_GROUP) {
|
||||
let group_n: usize = group.iter().map(|(_, n)| *n).sum();
|
||||
|
||||
// This group's source layers, for this partition only. Opened
|
||||
// before any builder so errors propagate via `?` before any worker
|
||||
// thread is spawned. `col_offset` is local to the group.
|
||||
let mut items: Vec<(usize, usize, PathBuf, Arc<KmerLayer>)> = Vec::new();
|
||||
let mut col_offset = 0usize;
|
||||
for (src, src_n) in group {
|
||||
let partition = src.partition(i)?;
|
||||
for l in 0..partition.n_layers() {
|
||||
let layer = Arc::new(partition.layer(l)?.open()?);
|
||||
let unitigs_path = layer.unitigs_path();
|
||||
if unitigs_path.exists() {
|
||||
items.push((col_offset, *src_n, unitigs_path, layer));
|
||||
}
|
||||
}
|
||||
col_offset += src_n;
|
||||
}
|
||||
|
||||
let expected_cols = n_dst_genomes + col_base;
|
||||
let open_group = |data_dir: &Path| -> OKIResult<(MatrixBuilder, Vec<ColBuilder>)> {
|
||||
let mut mb = MatrixBuilder::resume(presence(mode), data_dir).map_err(OKIError::Io)?;
|
||||
if mb.n_cols() != expected_cols {
|
||||
return Err(OKIError::InvalidData {
|
||||
context: "merge pass2",
|
||||
detail: format!(
|
||||
"{}: {} columns, expected {expected_cols}",
|
||||
data_dir.display(),
|
||||
mb.n_cols()
|
||||
),
|
||||
});
|
||||
}
|
||||
// In-memory columns: no sparse file / `MmapMut` on the write
|
||||
// path, so a block the FS cannot allocate is an `io::Error` at
|
||||
// `close`, not a SIGBUS (see `MatrixBuilder::add_mem_col`).
|
||||
let cols = (0..group_n).map(|_| mb.add_mem_col()).collect();
|
||||
Ok((mb, cols))
|
||||
};
|
||||
|
||||
let (exist_mbs, exist_cols): (Vec<MatrixBuilder>, Vec<Vec<ColBuilder>>) = exist_data_dirs
|
||||
.iter()
|
||||
.map(|d| open_group(d))
|
||||
.collect::<OKIResult<Vec<_>>>()?
|
||||
.into_iter()
|
||||
.unzip();
|
||||
let (new_mb, new_cols) = match &new_data_dir {
|
||||
Some(d) => {
|
||||
let (mb, cols) = open_group(d)?;
|
||||
(Some(mb), cols)
|
||||
}
|
||||
None => (None, vec![]),
|
||||
};
|
||||
|
||||
// `items` (and with it this group's source layers) is consumed by
|
||||
// the pipeline and dropped when it returns.
|
||||
let (exist_cols, new_cols) = run_pass2(
|
||||
items,
|
||||
&dst_layers,
|
||||
&new_mphf,
|
||||
exist_cols,
|
||||
new_cols,
|
||||
n_workers,
|
||||
max_open,
|
||||
)?;
|
||||
|
||||
for (mb, cols) in exist_mbs.into_iter().zip(exist_cols) {
|
||||
for c in cols {
|
||||
c.close()?;
|
||||
}
|
||||
mb.close().map_err(OKIError::Io)?;
|
||||
}
|
||||
for c in new_cols {
|
||||
c.close()?;
|
||||
}
|
||||
if let Some(mb) = new_mb {
|
||||
mb.close().map_err(OKIError::Io)?;
|
||||
}
|
||||
|
||||
col_base += group_n;
|
||||
}
|
||||
debug!(
|
||||
"partition {i}: pass2 done in {:.3}s",
|
||||
t_pass2.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
Ok(n_new)
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
//! Pass 2 of a partition merge, for one group of sources: stream each
|
||||
//! source layer's k-mers, read their values from the source matrix, locate
|
||||
//! them in dst (existing layers, else the new layer's MPHF) and write them
|
||||
//! into the group's destination columns.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use obipipeline::{
|
||||
Pipeline, PipelineError, PipelineSender, SharedFlatFn, Stage, ThrottleGuard, WorkerPool,
|
||||
make_sink, make_source, make_transform, throttle,
|
||||
};
|
||||
|
||||
use obicompactvec::ColBuilder;
|
||||
use obikindex::layer::KmerLayer;
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
use obikseq::CanonicalKmer;
|
||||
use obiskio::UnitigFileReader;
|
||||
|
||||
// ── dst cross-layer lookup ──────────────────────────────────────────────────
|
||||
|
||||
/// Batch membership lookup across `dst_layers` — first layer that carries
|
||||
/// each kmer wins. Groups hits by layer index (as `(kmer batch index,
|
||||
/// slot)` pairs) rather than returning them in query order, so pass 2 can
|
||||
/// follow with one column-major write per `(layer, column)` instead of
|
||||
/// interleaving across layers kmer by kmer. The kmers with no hit in any
|
||||
/// layer come back as a separate index list, for the new-layer MPHF batch
|
||||
/// lookup on a miss.
|
||||
fn find_batch_grouped(
|
||||
layers: &[KmerLayer],
|
||||
kmers: &[CanonicalKmer],
|
||||
) -> (Vec<Vec<(usize, usize)>>, Vec<usize>) {
|
||||
let mut by_layer: Vec<Vec<(usize, usize)>> = vec![Vec::new(); layers.len()];
|
||||
let mut not_found: Vec<usize> = Vec::new();
|
||||
for (idx, &kmer) in kmers.iter().enumerate() {
|
||||
match layers
|
||||
.iter()
|
||||
.enumerate()
|
||||
.find_map(|(li, l)| l.find_slot(kmer).map(|slot| (li, slot)))
|
||||
{
|
||||
Some((li, slot)) => by_layer[li].push((idx, slot)),
|
||||
None => not_found.push(idx),
|
||||
}
|
||||
}
|
||||
(by_layer, not_found)
|
||||
}
|
||||
|
||||
|
||||
// ── per-group pipeline ─────────────────────────────────────────────────────
|
||||
|
||||
/// Pass 2 for one group of sources: stream each source layer's unitigs,
|
||||
/// look its k-mers up in the source matrix and in dst (existing layers,
|
||||
/// then the new layer's MPHF), and write the values into the group's
|
||||
/// column builders. `items` carry group-local column offsets; the builders
|
||||
/// are handed back (still open) for the caller to close.
|
||||
#[allow(clippy::type_complexity)]
|
||||
pub(super) fn run_pass2(
|
||||
items: Vec<(usize, usize, PathBuf, Arc<KmerLayer>)>,
|
||||
dst_layers: &Arc<Vec<KmerLayer>>,
|
||||
new_mphf: &Option<Arc<obikindex::layer::MphfOnly>>,
|
||||
exist_builders: Vec<Vec<ColBuilder>>,
|
||||
new_builders: Vec<ColBuilder>,
|
||||
n_workers: usize,
|
||||
max_open: usize,
|
||||
) -> OKIResult<(Vec<Vec<ColBuilder>>, Vec<ColBuilder>)> {
|
||||
enum Pass2Data {
|
||||
SrcLayer((usize, usize, PathBuf, Arc<KmerLayer>, ThrottleGuard)),
|
||||
RawBatch((usize, usize, Arc<KmerLayer>, Vec<CanonicalKmer>)),
|
||||
/// One entry per `(dst layer, column)` touched by this batch, each
|
||||
/// carrying every `(slot, value)` pair for that column together —
|
||||
/// column-major, so the sink below locks that column's `ColBuilder`
|
||||
/// once per batch and writes the whole run, instead of once per
|
||||
/// `(kmer, genome)` pair.
|
||||
WriteBatch(Vec<(Option<usize>, usize, Vec<(usize, u32)>)>),
|
||||
}
|
||||
|
||||
let exist_locked: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_builders
|
||||
.into_iter()
|
||||
.map(|layer| layer.into_iter().map(|b| Arc::new(Mutex::new(b))).collect())
|
||||
.collect();
|
||||
let new_locked: Vec<Arc<Mutex<ColBuilder>>> = new_builders
|
||||
.into_iter()
|
||||
.map(|b| Arc::new(Mutex::new(b)))
|
||||
.collect();
|
||||
let exist_sink: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_locked
|
||||
.iter()
|
||||
.map(|layer| layer.iter().map(Arc::clone).collect())
|
||||
.collect();
|
||||
let new_sink: Vec<Arc<Mutex<ColBuilder>>> = new_locked.iter().map(Arc::clone).collect();
|
||||
let dst_layers_t2 = Arc::clone(dst_layers);
|
||||
let new_mphf_t2 = new_mphf.clone();
|
||||
let pass2_err: Arc<Mutex<Option<String>>> = Arc::new(Mutex::new(None));
|
||||
let err_cap2 = Arc::clone(&pass2_err);
|
||||
|
||||
let capacity = 2;
|
||||
let throttled_pass2 = throttle(items.into_iter(), max_open);
|
||||
|
||||
let pipeline2 = Pipeline::new(
|
||||
make_source!(
|
||||
Pass2Data,
|
||||
throttled_pass2.map(|t| {
|
||||
let (col_offset, src_n, unitigs_path, src_layer) = t.item;
|
||||
(col_offset, src_n, unitigs_path, src_layer, t.guard)
|
||||
}),
|
||||
SrcLayer
|
||||
),
|
||||
vec![
|
||||
Stage::Flat(Arc::new(
|
||||
move |data: Pass2Data,
|
||||
push: &PipelineSender<Result<Pass2Data, PipelineError>>,
|
||||
delta: &PipelineSender<isize>| {
|
||||
if let Pass2Data::SrcLayer((
|
||||
col_offset,
|
||||
src_n,
|
||||
unitigs_path,
|
||||
src_layer,
|
||||
_guard,
|
||||
)) = data
|
||||
{
|
||||
// _guard dropped at end of block, releasing the slot.
|
||||
// `src_layer` (MPHF + matrix) was already opened by
|
||||
// the caller — only the unitigs.bin stream itself is
|
||||
// opened (and throttled) here.
|
||||
let reader = match UnitigFileReader::open_sequential(&unitigs_path) {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
*err_cap2.lock().unwrap() = Some(e.to_string());
|
||||
delta.send(-1).ok();
|
||||
return;
|
||||
}
|
||||
};
|
||||
const BATCH: usize = 4096;
|
||||
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
|
||||
let mut count: isize = 0;
|
||||
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
|
||||
batch.push(kmer);
|
||||
if batch.len() == BATCH {
|
||||
let b = std::mem::replace(&mut batch, Vec::with_capacity(BATCH));
|
||||
push.send(Ok(Pass2Data::RawBatch((
|
||||
col_offset,
|
||||
src_n,
|
||||
Arc::clone(&src_layer),
|
||||
b,
|
||||
))))
|
||||
.ok();
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
if !batch.is_empty() {
|
||||
push.send(Ok(Pass2Data::RawBatch((
|
||||
col_offset, src_n, src_layer, batch,
|
||||
))))
|
||||
.ok();
|
||||
count += 1;
|
||||
}
|
||||
delta.send(count - 1).ok();
|
||||
}
|
||||
},
|
||||
) as SharedFlatFn<Pass2Data>),
|
||||
make_transform!(
|
||||
Pass2Data,
|
||||
{
|
||||
move |(col_offset, src_n, src_layer, kmers): (
|
||||
usize,
|
||||
usize,
|
||||
Arc<KmerLayer>,
|
||||
Vec<CanonicalKmer>,
|
||||
)|
|
||||
-> Vec<(Option<usize>, usize, Vec<(usize, u32)>)> {
|
||||
// Column-first, one pass for the whole batch: a single
|
||||
// MPHF hash sweep (`hash_batch`) instead of one hash
|
||||
// per kmer, then one read pass per source genome
|
||||
// column (`fill_sub_matrix`) instead of one row read
|
||||
// per kmer — matches the underlying matrix's own
|
||||
// column-major layout.
|
||||
let slots = src_layer.hash_batch(&kmers);
|
||||
let mut cols: Vec<Vec<u32>> = (0..src_n)
|
||||
.map(|_| Vec::with_capacity(kmers.len()))
|
||||
.collect();
|
||||
src_layer.fill_sub_matrix(&slots, &mut cols);
|
||||
|
||||
// Membership against dst, grouped by dst layer
|
||||
// instead of interleaved kmer by kmer — same
|
||||
// grouping `PartitionCache::find_presence_batch`
|
||||
// does in `obikphylo`, via the shared primitive.
|
||||
let (by_dst_layer, not_found) = find_batch_grouped(&dst_layers_t2, &kmers);
|
||||
|
||||
// Genome-major within each layer group (not kmer-major)
|
||||
// so every `(layer, col)` pair's values land together —
|
||||
// one entry per column, not one per `(kmer, genome)`.
|
||||
let mut ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)> =
|
||||
Vec::with_capacity(src_n * (by_dst_layer.len() + 1));
|
||||
for (dst_layer, hits) in by_dst_layer.into_iter().enumerate() {
|
||||
if hits.is_empty() {
|
||||
continue;
|
||||
}
|
||||
for g in 0..src_n {
|
||||
let col_vals: Vec<(usize, u32)> = hits
|
||||
.iter()
|
||||
.map(|&(idx, slot)| (slot, cols[g][idx]))
|
||||
.collect();
|
||||
ops.push((Some(dst_layer), col_offset + g, col_vals));
|
||||
}
|
||||
}
|
||||
if let (Some(mphf), false) = (&new_mphf_t2, not_found.is_empty()) {
|
||||
let not_found_kmers: Vec<CanonicalKmer> =
|
||||
not_found.iter().map(|&idx| kmers[idx]).collect();
|
||||
let new_slots = mphf.index_batch(¬_found_kmers);
|
||||
for g in 0..src_n {
|
||||
let col_vals: Vec<(usize, u32)> = not_found
|
||||
.iter()
|
||||
.zip(new_slots.iter())
|
||||
.map(|(&idx, &slot)| (slot, cols[g][idx]))
|
||||
.collect();
|
||||
ops.push((None, col_offset + g, col_vals));
|
||||
}
|
||||
}
|
||||
ops
|
||||
}
|
||||
},
|
||||
RawBatch,
|
||||
WriteBatch
|
||||
),
|
||||
],
|
||||
make_sink!(
|
||||
Pass2Data,
|
||||
{
|
||||
move |ops: Vec<(Option<usize>, usize, Vec<(usize, u32)>)>| {
|
||||
for (layer_opt, col, col_vals) in ops {
|
||||
let builder = match layer_opt {
|
||||
Some(l) => &exist_sink[l][col],
|
||||
None => &new_sink[col],
|
||||
};
|
||||
let mut b = builder.lock().unwrap();
|
||||
for (slot, val) in col_vals {
|
||||
b.set_val(slot, val);
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
WriteBatch
|
||||
),
|
||||
);
|
||||
|
||||
WorkerPool::new(pipeline2, n_workers, capacity).run();
|
||||
|
||||
if let Some(msg) = Arc::try_unwrap(pass2_err)
|
||||
.unwrap_or_else(|_| panic!("pass2: pass2_err not uniquely owned"))
|
||||
.into_inner()
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
{
|
||||
return Err(OKIError::InvalidData {
|
||||
context: "merge pass2",
|
||||
detail: msg,
|
||||
});
|
||||
}
|
||||
|
||||
let unlock = |b: Arc<Mutex<ColBuilder>>| -> ColBuilder {
|
||||
Arc::try_unwrap(b)
|
||||
.unwrap_or_else(|_| panic!("pass2: column builder not uniquely owned"))
|
||||
.into_inner()
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
};
|
||||
let exist = exist_locked
|
||||
.into_iter()
|
||||
.map(|layer| layer.into_iter().map(unlock).collect())
|
||||
.collect();
|
||||
let new = new_locked.into_iter().map(unlock).collect();
|
||||
Ok((exist, new))
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user