Compare commits

..
13 Commits
Author SHA1 Message Date
Eric Coissac 85296a2ef8 Implement directory locking with shared and exclusive modes
Release / create-release (push) Successful in 2m41s
Release / build-linux-x86_64 (push) Successful in 9m17s
Release / build-macos-arm64 (push) Successful in 2m2s
ci.yml / build (pull_request) Successful in 3m48s
Introduces a robust, OS-level directory locking mechanism using LockSlot for managing shared and exclusive access, supporting lock transitions and non-blocking probes. Also sets up the index structure with new builder and index modules.
2026-10-16 16:02:00 +02:00
Eric Coissac 9ac3a37529 Implement explicit directory locking for index operations
Introduces explicit directory locking mechanisms (`DirLock`) to ensure exclusive access during index operations across command stages. This includes modifying index opening logic to acquire locks before proceeding, handling lock acquisition failures by exiting, and implementing safe conditional building of sibling annexes using locking to prevent concurrent corruption.
2026-10-16 16:00:49 +02:00
Eric Coissac 7c7d7615cb Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
2026-10-16 15:59:40 +02:00
Eric Coissac 8296e3544c Simplify Hybrid layer implementation and update dependencies
Release / create-release (push) Successful in 2m31s
Release / build-linux-x86_64 (push) Successful in 9m24s
Release / build-macos-arm64 (push) Successful in 2m5s
ci.yml / build (pull_request) Successful in 3m51s
Dropped fields related to Hybrid layers for simplification and removed the unused `ahash` dependency. Updated dependency versions for `obidebruinj`, `obikmer`, and related crates, and cleaned up temporary artifacts.
2026-10-15 12:23:15 +02:00
Eric Coissac 12afbe819c Refactor MPHF layer logic and evidence handling
This commit implements core lookup functionality for MPHF layers, including `hash` and `find_strict` methods. It introduces support for different evidence types (Exact, Hybrid, Approximate) within layers, clarifying lookup behavior and improving partition calculation logic in the cache. Additionally, dead code is removed and dependencies are updated.
2026-10-15 12:22:45 +02:00
Eric Coissac f9edefd135 Refactor query system for performance and memory reduction
This change introduces dynamic chunk sizing based on input size estimates, ensuring memory constraints are respected while allowing for smaller chunks that yield significant speed improvements. Additionally, index structures were optimized, resulting in drastic memory footprint reduction and faster query execution times.
2026-10-15 12:21:38 +02:00
Eric Coissac 025c40c0db Refactor chunking logic to enforce 128 k-mer limit
This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
2026-10-15 12:20:36 +02:00
Eric Coissac 8468319a63 Refactor unitig detection and graph traversal logic
Introduce new hashing mechanisms and support for directional unitig checks. Fix fragmentation issues and improve performance metrics through optimized graph construction and new benchmarking utilities.
2026-10-15 12:19:24 +02:00
Eric Coissac d32d86193f Refactor index metadata loading to use buffered I/O
This change replaces unbuffered file reads with buffered I/O operations to improve performance during index metadata loading. This includes implementing new methods for reading index metadata files and wrapping file handles in `BufReader` to optimize subsequent reads.
2026-10-15 12:17:51 +02:00
Eric Coissac 093d81c359 Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
2026-10-15 12:17:09 +02:00
Eric Coissac 09c80468b2 Refactor k-mer storage to use flat arrays
This change replaces the HashMap-based k-mer storage mechanism with a compact, flat array structure for `PartitionKmers`. This significantly reduces memory usage per partition and improves query execution time.

It also updates the batch processing logic to handle the new data structure, introduces detailed memory footprint calculations, and updates relevant query layer APIs. Test coverage has been expanded to validate correct partitioning and k-mer ordering.
2026-10-15 12:16:21 +02:00
Eric Coissac fe38424682 Implement batch lookup and chunked data management
Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline.

This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
2026-10-15 12:14:51 +02:00
Eric Coissac 8b87078478 Implement MPHF migration tool and sparse matrix permutation logic
Refactors sparse matrix handling to implement row permutation logic and introduces a standalone tool for migrating indexes from MPHF 1.x to 2.x formats.
2026-10-15 12:12:51 +02:00
89 changed files with 6353 additions and 1277 deletions

No files matched your search

+24 -1
View File
@@ -62,7 +62,8 @@ jobs:
~/.cargo/registry ~/.cargo/registry
~/.cargo/git ~/.cargo/git
src/target src/target
key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock') }} tools/migrate-mphf/target
key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock', 'tools/migrate-mphf/Cargo.lock') }}
restore-keys: linux-musl-cargo- restore-keys: linux-musl-cargo-
- name: Build static binary - name: Build static binary
@@ -83,6 +84,28 @@ jobs:
-H "Authorization: token $GITEA_TOKEN" \ -H "Authorization: token $GITEA_TOKEN" \
-F "attachment=@/tmp/dist/obikmer-linux-x86_64" -F "attachment=@/tmp/dist/obikmer-linux-x86_64"
# One-shot converter for indexes built with ptr_hash 1.x. A standalone
# crate (own Cargo.lock/target, outside src/'s workspace), built after
# obikmer's own asset is already uploaded: it never blocks that one.
- name: Build migrate-mphf (static)
working-directory: tools/migrate-mphf
env:
PKG_CONFIG_ALLOW_CROSS: "1"
run: cargo zigbuild --release --target x86_64-unknown-linux-musl
- name: Upload migrate-mphf artifact
working-directory: tools/migrate-mphf
env:
GITEA_TOKEN: ${{ secrets.GITEATOKEN }}
RELEASE_ID: ${{ needs.create-release.outputs.release_id }}
run: |
cp target/x86_64-unknown-linux-musl/release/migrate-mphf /tmp/dist/migrate-mphf-linux-x86_64
strip /tmp/dist/migrate-mphf-linux-x86_64
curl -s -X POST \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases/$RELEASE_ID/assets" \
-H "Authorization: token $GITEA_TOKEN" \
-F "attachment=@/tmp/dist/migrate-mphf-linux-x86_64"
build-macos-arm64: build-macos-arm64:
needs: create-release needs: create-release
runs-on: ubuntu-latest runs-on: ubuntu-latest
+1
View File
@@ -9,6 +9,7 @@ CLAUDE.md
memory/ memory/
sandbox/ sandbox/
src/target src/target
tools/migrate-mphf/target
data-stress data-stress
*.fasta *.fasta
*.fasta.gz *.fasta.gz
+43 -1
View File
@@ -24,7 +24,7 @@ for each chunk of sequences (parallel workers via obipipeline, one call to proce
decompose all sequences into superkmers (SuperKmerIter) — construction only, decompose all sequences into superkmers (SuperKmerIter) — construction only,
not the dedup key not the dedup key
deduplicate at k-mer granularity, split by partition in the same pass: deduplicate at k-mer granularity, split by partition in the same pass:
by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>> ← KmerDesc = (seq_idx, pos) by_partition: Vec<PartitionKmers> ← sorted unique k-mers + flat KmerDesc = (seq_idx, pos) slices (2026-10-15, was a HashMap — see "Query memory" below)
allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers — allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers —
NOT multiplied by n_genomes NOT multiplied by n_genomes
allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome — allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome —
@@ -404,3 +404,45 @@ Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-w
- **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone. - **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone.
- **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition. - **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition.
- **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action. - **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action.
## Query memory: `PartitionKmers` replaces the per-partition `HashMap` (2026-10-15)
**Problem.** A full-size query (two ~450 MB fastq.gz of a skim against a 354 M-kmer index) reached **74–81 GB RSS** with the default 256 MiB chunk. Cause, read in `batch.rs`/`chunk.rs` and measured with the `by_partition memory retained` debug log: `QueryBatch::by_partition` was a `HashMap<CanonicalKmer, Vec<KmerDesc>>` per partition. On genomic reads almost every k-mer is unique (12.5 M unique for 13.5 M occurrences in a 48 MB chunk, a dedup of 8 %), so each k-mer paid a table slot, growth slack and its own heap `Vec`: **889 MB per 48 MB chunk, ~71 bytes per unique k-mer, 18× the chunk size** — the dominant term of the per-chunk memory, several chunks being in flight at once.
**Change.** `obikquery::PartitionKmers`: occurrences collected as `(kmer, KmerDesc)` per partition, sorted by `(kmer, seq_idx, pos)` and stored as three flat arrays (unique k-mers, u32 offsets, descs). `query_partition_with` takes `&PartitionKmers`; `QueryHit` already carried slices. Occurrence order per k-mer is unchanged (query order); no per-k-mer allocation.
**Measured** (same inputs, release build, identical sorted output digests before and after):
| | `by_partition` (48 MB chunk) | RSS, 2×48 MB chunks | RSS, full run (≈ 1 GB reads) | wall, full run |
|---|---|---|---|---|
| before | 889 MB | 5.6 GB | 73.7–80.7 GB | 135–172 s (one 312 s outlier) |
| after | 259 MB | 4.1 GB | **28.2 GB** | **103 s** |
With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the touched pages of the index itself, so after the change the remaining per-chunk anonymous memory is ≈ 6× the chunk size, below the formula's factor 12 (`BASE_OVERHEAD + 8·n_genomes`, still a conservative bound — left unchanged). The sort replaces hashing in `from_records` and was not slower (17.3 s vs 21.2 s on the small test).
**Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here.
**Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed.
### Chunk size sweep on a small input (2026-10-15)
Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer single-genome index (4 partitions), 16 cores on AC power, 3 runs per size, sorted outputs identical (same md5) for every size:
| `--chunk-size` (MiB) | chunks | wall (s) | user CPU (s) | RSS (GB) |
|---|---|---|---|---|
| 2 | ~50 | 2.5–2.6 | 25.0–25.3 | 3.2 |
| 4 | ~25 | 2.5–2.7 | 24.3–25.5 | 3.5 |
| 8 | ~12 | 2.5–2.7 | 23.6–26.1 | 4.1 |
| 16 | ~6 | 4.4 | 23.1–23.7 | 4.1 |
| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 |
| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 |
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs — **implemented, see below**. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
### Automatic chunk size now follows the input size (2026-10-15)
`cmd/query/mod.rs`: when `--chunk-size` is not given, the memory-derived size (unchanged: `available RAM / (workers × overhead × 2)`, clamped to 4–256 MiB) is further capped by `estimated input bytes / (workers × 2)`, never below 4 MiB. The estimate is `obiread::estimated_decompressed_len`: the file length, ×3 when the magic bytes say gzip (≈ 2.8 measured on FASTQ), ×4 for bzip2/xz/zstd — a deliberately rough figure, detected from the magic bytes, not the extension. For stdin, URLs or unreadable files the size is unknown and the memory-derived size stands. An explicit `--chunk-size` still wins.
Measured (2 × 48 MB reads, 16 cores, AC power, outputs identical): single-genome index, 354 M kmers — default chunk 256 MiB → 4 MiB: **11.4 s → 2.4 s** (−79 %), CPU 22.3 → 26 s (+17 %), peak footprint 1.44 → 0.75 GB. On the 91-genome index the memory-derived size was already 4 MiB (overhead factor 4 + 8 × 91), so nothing changes there (5.1–5.7 s either way). Not measured: a large input (≈ 1 GB gz, whose estimate gives ≈ 95 MiB chunks instead of 256 MiB: more, smaller chunks, expected lower memory).
@@ -0,0 +1,369 @@
# MPHF batch queries — site inventory and expected gains
Investigation of 2026-10-14 (`ptr_hash` 2.1.2, `epserde` 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked **[read]** (verified in source), **[computed]** (derived from stated sizes) or **[estimate]** (reasoning, not measured). Measurements, once made, are in the last section.
## 1. What `ptr_hash` 2.1.2 actually offers [read]
`Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>`.
- **`V` is the pilot array** (one byte per bucket), **`F` is only the remap storage**. A layer's `mphf.bin` (~2.4 bits/key **[computed]**: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetch `pilots[bucket]` — the right line. The remap (read only when `slot >= n`, a small fraction of keys) is **not** prefetched.
- `index_batch<K>(xs: [Q; K]) -> [usize; K]` — stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (`len % K`) is the caller's job.
- `index_stream<B>(keys: IntoIterator) -> impl Iterator` — ring buffer of B keys; handles head and tail itself; accepts any iterator. **Its `next()` is `unimplemented!`**: only fold-based consumption works (`for_each`, `fold`, `sum`). `collect()`, `zip`, `for x in it` panic. Consume with `for_each` into a pre-sized `Vec`.
- `index_batch_exact` needs the nightly `unstable` feature; `index_batch_exact2` is `#[doc(hidden)]`. Neither is usable.
- Prefetch is `prefetch-index 0.2.1`: `_mm_prefetch(T0)` on x86_64, asm on aarch64, **no-op elsewhere**.
- **`SINGLE_PART`**: neither `index_stream` nor `index_batch` branches on it — both always run the multi-part `bucket()`/`slot()`. Only `index()` has the faster single-part path. So with batched queries the `SINGLE_PART = true` advantage disappears, and `false` (our choice) costs nothing while keeping parallel construction (`true` also panics with sharding).
- Old `ptr_hash` 1.1.0 (used by `tools/migrate-mphf`) has the same family with an extra `MINIMAL` const: `index_stream<B, MINIMAL, Q>`, `index_batch<K, MINIMAL, Q>`. No new dependency needed there.
```rust
// consuming index_stream: fold-based only
let mut out = Vec::with_capacity(kmers.len());
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
.for_each(|slot| out.push(slot));
```
## 2. Existing "batch" APIs that are not batched [read]
`MphfLayer::hash_batch` and `MphfOnly::index_batch` loop `.index()` one kmer at a time. Consumers: `obikfilter/partition_iter.rs` (`iter_layer_kmers`), `obikmerge/partition_merge.rs` (pass 2: `hash_batch`, `index_batch`), `obikphylo/siblings/algorithms/family_scan.rs` (two sites).
## 3. Cost chain of a membership query (`MphfLayer::find`) [read]
1. `pilots[bucket]` — the only access `ptr_hash` prefetches.
2. Exact/Hybrid-exact: `evidence.decode(slot)` — 4 B/slot, ~520 MB per 130 M-slot layer **[computed]**, random mmap read.
3. `verify_canonical_kmer` → `block_offsets[...]` (O(1) with the default `--block-size 1`; a forward scan of up to `2^block_bits` records otherwise).
4. `extract_kmer_raw` — random read in the `unitigs.bin` mmap.
Approx/Hybrid-fast: pilots then `fingerprint.get(slot)` — one extra random access.
Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. `ptr_hash` only helps with stage 1; stages 2–4 need prefetch support in `obikindex`/`obiskio` (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.
Layer sizes: `layer_0` MPHF 2–3 MB (cache-resident), `layer_1` ~40 MB (beyond one CCD's L3 on EPYC-class machines — **hypothesis**, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on `layer_1`. For branch-free independent loops (`hash_batch`) out-of-order execution already overlaps several misses, so the explicit prefetch gains less **[estimate]** than for branchy loops (`find` with early `break`).
## 4. Call-site inventory
(a) already batched; (b) batch buildable; (c) hard/pointless.
| Site | What | Class |
|---|---|---|
| `mphf_layer.rs` `hash_batch`, `MphfOnly::index_batch` | loop of `.index()` | (a) |
| `obikfilter/partition_iter.rs:106` | `hash_batch`, 32768 kmers | (a) |
| `obikmerge/partition_merge.rs:489, 521` | `hash_batch`, `index_batch`, 4096 | (a) |
| `obikphylo/.../family_scan.rs:285, 397` | `hash_batch` | (a) |
| `obikquery/query_layer.rs` stage 1 | `find_in_layer` per kmer, layer by layer, `break` on hit | (b) |
| `partition_merge.rs` `find_batch_grouped` | `find_slot` per kmer over dst layers | (b) |
| `partition_merge.rs` (`build_graph` filter) | `all(find_slot.is_none())`, every layer, per kmer; `build_graph` takes `Fn(CanonicalKmer)->bool` | (b), API change |
| `family_scan.rs` slow path; `annex.rs` | `find_slot` / `find_in_partition` per variant | (b) |
| `mphf_layer.rs` `build`, `build_exact_evidence`, `build_approx_evidence` | `mphf.index` then random writes (evidence, fingerprint, `seen`) | (b), chunked |
| `obikindexer/counter/count.rs:47` | `mphf.index` then `counts.get/set` | (b), hard (needs (kmer, count) pairs) |
| `obikindexer/extensions/private_builder.rs:177` | abundance filter | (b), conditional |
| `private_builder.rs:196`, `verify.rs`, `IndexCache::find_in_*` | per-kmer by nature / sampled | (c) |
| `tools/migrate-mphf/plan.rs:41-42` | `old.index` + `new.index` per kmer, random writes | (b) |
Dead or suspect code found along the way (candidates for removal, not removed):
- `MphfLayer::find_strict` — no caller.
- the single-kmer `hash` chain (`MphfLayer::hash` → `TypedLayer::hash` → `KmerLayer::hash` → `IndexCache::hash`) — no caller outside the chain.
- `IndexCache::find` — no caller.
- `obikmerge/src/merge_layer/` — not declared in `lib.rs` (`mod merge; mod partition_merge;` only); still imports the `LayeredMap` removed in 2026-08. To be confirmed by the compiler.
## 5. Interactions
- **Verification chain**: the right shape is one `find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>)` on `MphfLayer`, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops).
- **Memory**: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one `Vec<usize>` per call. `index_stream` itself only allocates a B-element ring on the stack.
- **Parallelism**: no new parallelism. Batches already come out of `obipipeline` stages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. In `query_partition_with` the unique-kmer `HashMap` is already materialised; iterate it chunk-wise.
## 6. Benchmark
`benchmark/bench_mphf.sh LABEL` times, on the 20-genome benchmark data, `index` (2 genomes), `merge` (20 presence indexes), `query` (presence/count × 2 genomes), `filter`, `convert` (approx/exact round trip) and `phylo` (`--sibling-annex`, `snp-jc`), `REPS` times each, and records order-independent output digests that must not change across labels. See the results section below.
**Limit of this data set**: 20 small genomes → per-partition `layer_1` MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it **cannot** measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).
### Throwaway prod-scale benchmark (not written)
Open one real partition's `layer_1` (~130 M keys) and `layer_0` (~8 M) with `MphfOnly`/`MphfLayer`; key sets: present (sample of `iter_kmers`, random order and sequential order) and absent (random, checked absent); measure kmers/s for `index`, `index_stream::<B>` (B ∈ 8…128), `index_batch::<K>` (K ∈ 8…64), and the full `find` vs staged `find_batch`; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of `mphf.bin`, `evidence.bin`, `unitigs.bin(.idx)` reported; first check `index_stream` slots == `index` slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in `mphf.md`) is a paper claim, not tied to our sizes or machines.
## 7. Not verified
No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and `--block-size` used in prod; whether `migrate-mphf` already ran; share of total index/merge/query time spent in these loops; `count.rs` f0 per partition at prod scale; whether `merge_layer/` is dead (only the head of `lib.rs` read).
## 8. First improvement and measurements (2026-10-15)
**Implemented** (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):
- `MphfLayer` now holds a `MphfOnly` instead of its own `MemCase<Mphf>` (one mmap-open path, one place for the stream call).
- `MphfOnly::for_each_slot(kmers, f)` wraps `index_stream::<32>`; `index_batch`/`hash_batch` now really use it (they previously looped `.index()`). Callback-shaped because `index_stream`'s `next()` is `unimplemented!`.
- `MphfLayer::find_batch` / `TypedLayer::find_slot_batch` / `KmerLayer::find_slot_batch`; the mode-dependent half of `find` is factored into `MphfLayer::confirm`, shared by both.
- `query_partition_with` stage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins, `n_mphf_calls` unchanged).
- Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive; `prefetch-index` is already in the tree via `ptr_hash`, but adding it to `obikindex` is a dependency decision), and the other (b) sites.
**Measurements** (`benchmark/bench_mphf.sh`, `benchmark/summarize_bench.py`; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:
| | wall (s) | user CPU (s) |
|---|---|---|
| before | 5.94–6.02 (median 5.96) | 10.95–11.07 |
| after | 5.63–6.00 (median 5.67) | 10.33–10.66 |
About −5 % wall and −5.5 % CPU, ranges non-overlapping **[measured]**. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.
**Reading the number.** Small by construction: this data set's `layer_1` MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains **unmeasured** and needs a prod-scale layer (§6).
**Noise caveat.** With a load average of 16–22 from unrelated processes, single-run walls of `index`, `merge` and `filter` moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The `index`/`merge`/`filter`/`convert`/`phylo` rows of the first comparison are noise, not effects (`index` pass 2 was not changed at all; a later interleaved `filter` A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).
**Stale benchmark indexes.** The indexes already under `benchmark/run/` (`specimen_index_*`, `global_index_*`, built 2026-08-28) carry `ptr_hash` 1.x MPHFs and fail with "Major version mismatch" on current binaries (`tools/migrate-mphf` converts them). `bench_mphf.sh` therefore builds its own copies under `run/bench_mphf/data/` on first use.
## 9. Benchmark procedure
Prerequisites: the simulated reads (`benchmark/run/simulated_data/`) and query reads (`benchmark/run/query_data/`) produced by `gmake simulate simulate_query`; macOS BSD `time -l` and `md5 -q` (the script uses both), Python 3.
1. Build the binary to measure and **copy it** under a name that identifies it (cargo overwrites `target/release/obikmer`): `cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>`.
2. `BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label>` (from `benchmark/`; `REPS=3` by default, `CPU_MAX=n` to restrict cores). The first call builds the shared data set in `run/bench_mphf/data/` with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise delete `data/` and rebuild.
3. Output: `run/bench_mphf/<label>/results.csv` (wall and peak RSS per step and repetition, plus one row `step:stage` per Reporter stage) and `checksums.txt`.
4. Compare: `python3 benchmark/summarize_bench.py <ref_label> <new_label>` (medians and ratio) and `diff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt` — **any difference is a bug**, not a performance matter.
5. A ratio is only a candidate effect. Check the spread by running the **same** binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from `/usr/bin/time -l`).
6. Rule of thumb for this data set: only `query` (≈ 5 s, stable to ±1 %) resolves effects of a few percent; `index`, `merge`, `filter`, `phylo` need interleaved runs or a quiet machine.
Step → code exercised:
| Step | Code on the MPHF path |
|---|---|
| `index_*` (`:index` stage) | `MphfLayer::build` pass 2 |
| `merge_20_presence` | `find_batch_grouped`, `build_graph` filter, `hash_batch`, `index_batch` |
| `query_*` | `query_partition_with` stage 1 |
| `filter_species` | `iter_layer_kmers` → `hash_batch` |
| `convert_to_*` | `build_exact_evidence`, `build_approx_evidence` |
| `phylo_sibling_annex` | `annex.rs` `find_in_partition` per variant |
| `phylo_snp_distance` | `family_scan.rs` `hash_batch`, `find_slot` |
## 10. Second step — construction pass 2 (2026-10-15)
**Implemented**: `layer/slotted.rs`, `SlottedKmers`, an iterator adapter yielding `(slot, kmer, chunk_id, rank)` for the wrapped `(kmer, chunk_id, rank)` stream, computing slots 4096 kmers at a time through `index_stream::<32>` (bounded memory, never the whole layer). The five loops of `MphfLayer::build` (Exact, Approx, Hybrid), `build_exact_evidence` and `build_approx_evidence` now iterate it instead of calling `mphf.index` per kmer. Output digests unchanged.
**Measured** (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):
| Command | before | after |
|---|---|---|
| `convert --exact-evidence` (20-genome index) | 0.56–0.68 s / 2.22–2.67 s | 0.59–0.74 s / 2.25–2.52 s |
| `index` E. coli reads | 2.45–2.63 s / 13.1–14.3 s | 2.48–2.75 s / 13.2–14.0 s |
**No measurable effect** on this data set (differences within the run-to-run spread; if anything `convert` is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and `index`/`convert` time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/`seen` writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random *writes* to `evidence.bin`/`seen`/`fingerprint.bin` (not prefetched by `ptr_hash`) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.
## 11. Why the two changes gain so little (profiling, 2026-10-15)
Measured with macOS `sample` (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.
**Query.** In a 2 s window on the `query` stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):
- `query_partition_with` ≈ 260 samples (8 % of active CPU), of which `find_batch` 217, of which **`verify_canonical_kmer` 185** and everything else in `find_batch` (the `ptr_hash` stream and `confirm`) **32**.
- `verify_canonical_kmer` is the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.
- Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/`unitigs.bin`/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side.
- The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (`rolling_stat`, entropy tracker, string formatting) and by sequential parts, not by lookups.
**What produced the −5 %.** Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer `index` (stream disabled) 6.0 s; restructuring + `index_stream` 5.85 s. **Most of the gain comes from the restructuring, not from the prefetch** (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called `IndexCache::get_layer` — an `Arc<KmerLayer>` clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.
**Index (pass 2).** In a 1 s window late in the `index` stage, `SlottedKmers` appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (`obidebruinj`) and ≈ 35 for `ptr_hash::build` (pass 1, MPHF *construction*). Pass 2 is a few percent of `index` and `convert` here: nothing to win even if its lookups were free. `convert` is `build_*_evidence` on 2 MB-scale layers, same conclusion.
**Consequences for priorities.**
1. On this data the next lever is the verification (`verify_canonical_kmer`, evidence decode), not the MPHF — consistent with §3, now observed.
2. Hot `Arc` clones in per-kmer paths (`get_layer`, `IndexCache::find_in_layer`, `find_slot` loops in merge and phylo) are worth auditing independently of MPHF batching.
3. The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).
## 12. Calibration: a single skim at fewer partitions (2026-10-15)
`Vaccinium myrtillus` skim (2 × ~440 MB fastq.gz, `theta 0`), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (`--partitions` is a power of two; MPHF size ≈ 2.4 bits/key **[measured]**):
| `-p` | keys / partition | `mphf.bin` / partition | index wall | peak RSS |
|---|---|---|---|---|
| 256 (default) | ≈ 1.4 M **[computed from p=16]** | ≈ 0.4 MB **[computed]** | – | – |
| 16 | 22.1 M | 6.65 MB | 120 s | 12.3 GB |
| 4 | 86–90 M | 25.9–27.0 MB | 171 s | 19.5 GB |
Total ≈ 354 M distinct kmers for one skim. `-p 4` is already in the 30–45 MB range of the prod `layer_1` MPHFs; `-p 2` would give ≈ 53 MB. Indexes kept in `benchmark/run/bench_mphf/skim_p16` and `skim_p4` (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.
## 13. Query on a real-scale index (2026-10-15)
**Index.** Working index `phyloskims_sal_vac_sav` (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with `tools/migrate-mphf` **on an APFS clone** (`cp -c`, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at `/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`; its sibling annex is in `stale_annexes_pre_ptrhash2/` and would need rebuilding to be used. Per partition: `layer_1` ≈ 31 M keys / 9.3 MB MPHF, `layer_0` ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB).
**Query** (2 × 48 MB of *V. oxycoccos* reads, `--chunk-size 8`, AC power, 3 interleaved runs per binary, identical sorted output for all three):
| binary | wall (s) | user CPU (s) | `query` stage (s) |
|---|---|---|---|
| original | 28.2–30.5 | 70.9–74.4 | 14.6–17.1 |
| + batched query stage 1 | 26.9–28.0 | 48.1–48.6 | 13.6–14.4 |
| + `PartitionKmers` memory fix, build-side `SlottedKmers` | 26.4–26.7 | 41.9–42.5 | 12.9–13.3 |
The `query` stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. **The CPU drop is much larger than the wall drop** and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer `Arc` clones of `get_layer`, not isolated); the original binary was spending CPU in contention that wall time hid.
**Index-open cost (not related to the changes above).** A query of a single 64-base sequence on this index takes **15 s and a 27 GB peak footprint**. `vmmap` during the open: 24.5 GB of `MALLOC_LARGE` in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by `IndexCache::new` for every layer of every partition whatever the query. Known contributors: `UnitigFileReader` reads each `.idx` into a `Vec` (≈ 30 MB for `layer_1`, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied.
## 14. Index-open memory: `.idx` offsets and `dict_values` mapped, not copied (2026-10-15)
**Cause** (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, **18.2 GB** were `unitigs.bin.idx` read into a `Vec<u32>` **twice** per exact-evidence layer (`MphfLayer::open` builds two `UnitigFileReader`s of the same file — one for the evidence check, one for `iter_kmers`/`find_strict`; both go through `open` → `open_direct_access` → `read_idx` → `fs::read` + copy) and **7.7 GB** were `dict_values.bin` read with `fs::read` in `PersistentSparseBitMatrix::open`. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (`PersistentRankSelectBitVec`, `PersistentFixedIntVec`, `EliasFano`) was already mmapped.
**Change.**
- `obiskio::unitig_index::reader`: `block_offsets: Vec<u32>` → `BlockOffsets`, a view into the memory-mapped `.idx` (header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages.
- `obicompactvec::bitmatrix::sparse`: `dict_values: Vec<u8>` → `Option<Mmap>` (`None` for an empty file, which cannot be mapped), read through `dict_bytes()`.
**Measured** (same session, same inputs; sorted query outputs identical, md5 `e9e7c35b`):
| | open only (1-sequence query) | 2×48 MB reads, `--chunk-size 8` |
|---|---|---|
| before | 10.6–15.7 s, **27.4 GB** footprint | 27.7 s (51 s cold), 30.9 GB, `query` stage 13.8 s |
| after | **0.26–0.81 s, 0.04 GB** | **10.7 s, 3.6 GB**, `query` stage 9.95 s |
Tests: `cargo test --workspace` green; `scripts/smoke_test_index.sh` PASS; `bench_mphf.sh` (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a `.idx` map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect.
## 15. Unbuffered reads: `index.meta` and superkmer files (2026-10-15)
Follow-up of §14 (search for the same family of problems: file content handled inefficiently at open). Static search over `src/` for `fs::read`, `read_to_end`, `load_full`, `from_reader`, bare `File` readers and mmap `to_vec`: every persistent structure of `obicompactvec`, `obikindex` and the `obikphylo` annexes is already mmapped; the two real findings were *unbuffered* readers.
**1. `IndexMeta::read_full` (`obikindex/src/index/meta.rs`)** passed a bare `File` to `serde_json::from_reader`, which issues **one `read` syscall per byte**. `index.meta` of the 91-genome index is 20 KB and is read afresh by every `state()`/`genomes()` call (stateless design, §obikindex audit), so each call cost ≈ 0.35 s here (macOS). Profile of `obikmer dump --head 10` (`sample`): ≈ 20 000 samples in `read`, called from `Bytes<File>::next` ← `serde_json` ← `IndexMeta`. Fix: `fs::read` then `serde_json::from_slice`. **`dump --head 10`: 9.0–10.0 s (91–102 s of system CPU) → 0.16–0.84 s**; the same change applies to `SKFileMeta::read` (`obiskio/src/meta.rs`, per-superkmer-file sidecars).
**2. `SKFileReader::open` (`obiskio/src/reader.rs`)**: for an uncompressed file `niffler::get_reader` returns the sniffed bytes chained to the bare `File`, and `read_superkmer` reads each field with its own small `read_exact`. Fix: wrap the file in a 64 KiB `BufReader` before `niffler`. Effect on `index` (E. coli, 4 interleaved runs each): `count_kmer` 0.51–0.56 s → 0.42–0.50 s (≈ −12 %), total wall unchanged (≈ 1.95 s). Small, low risk, kept.
Tests: `cargo test --workspace` 549 passed / 0 failed; `scripts/smoke_test_index.sh` PASS; query output on the 91-genome index identical (md5 `e9e7c35b`).
**Probes of other commands on the 91-genome index (180 GB, 256 partitions)**: `phylo --distance jaccard` peaks at 155–190 MB of footprint while running and finishes in < 30 s; `utils --stats` 130–150 MB. `filter` (and by the same check `select`, `merge`) refused the index — "source index is not fully built" — with the unmodified baseline binary too: this index's `index.meta` predates the `state` field, so they could not be probed.
**Observations not investigated**: `index` on E. coli shows ≈ 9 s of *system* CPU in 2 s of wall time (both before and after these changes), consistent with the `swtch_pri` (yield) spinning seen in the `index` profile of §11 rather than with I/O; and wall times of the same query on the same binary varied 4.8 s ↔ 10.7 s ↔ 43 s depending on the state of the page cache, so only same-session comparisons are meaningful.
## 16. Profile of `index` on E. coli reads (2026-10-15)
`sample` (1 ms stacks) of `obikmer index` on the E. coli K-12 simulated reads (2 × 15x, 16 threads, one run; the `index` stage lasted 2.4 s under the sampler vs ≈ 0.9 s normally, so proportions are approximate). Stage walls: scatter 0.40 s, dereplicate 0.14, count_kmer 0.60, index 2.36.
**Threads are mostly idle.** Of ≈ 30 800 thread-samples: 24 900 blocked or yielding, 5 900 active. The ≈ 9 s of *system* CPU per run (§15) is `swtch_pri` (`sched_yield`) — 4 550 samples: ≈ 70 % rayon workers spinning for work (`WorkerThread` wait loop), ≈ 28 % `crossbeam` bounded-channel send/recv spinning, the rest in `obidebruinj` thread-locals. It is waiting, not I/O: no `read` samples remain. It costs no wall time on an otherwise idle machine, but burns CPU (visible in `user + sys` accounting and on a shared 192-core machine).
**Where the active CPU goes** (share of the 5 900 active samples):
| | share |
|---|---|
| xxh3 hashing of `CanonicalKmer` for the de Bruijn graph's `HashMap` (callers: `count_neighbors` 791, `hash_one` 358, `walk` 285, `unitig_iter` 252, `push` 164, `unitig_nucleotides` 75, rehash 68; plus `xxh3_stateful_update` 300) | ≈ 34 % |
| `debruijn::count_neighbors` itself (inclusive of its hashing ≈ 2 300 samples: `compute_degrees_and_mark_starts`) | 12.5 % own |
| `ptr_hash::build` (MPHF construction) | 4.8 % |
| crossbeam channel send | 4.8 % |
| `open` / `close` / `write` (many small files over 256 partitions) | ≈ 9 % |
| debruijn `unitig_iter` / `walk` own time | 8.3 % |
So at this scale the cost of `index` is the **de Bruijn graph's hash table** (several lookups of neighbouring k-mers per k-mer, each hashed with xxh3 and probably missing the cache), not the MPHF passes this document started from (`SlottedKmers` ≈ 2 %, §11). Not measured: what fraction of those lookups miss the cache, and the same profile on a prod-size partition.
## 17. Profile of `index` on a skim: *Vaccinium vitis-idaea*, 2.1 GB, `--partitions 16` (2026-10-15)
One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slows the process; walls below include it): scatter 19.7 s, dereplicate 8.0 s, count_kmer 25.2 s, **index 139.1 s**; total 192 s real, 1830 s user, **225 s system** (12 % of user, against ≈ 90 % on E. coli, §16), peak footprint 17.8 GB.
**Index stage** (samples taken fully inside it, 91 000 active vs 187 000 idle/yielding thread-samples):
| share of active CPU | |
|---|---|
| xxh3 hashing of kmers (de Bruijn `HashMap`) | 32.7 % |
| `debruijn::walk` | 12.4 % |
| `debruijn::count_neighbors` | 10.7 % |
| `GraphDeBruijn::push` | 6.9 % |
| `crossbeam` channel | 6.6 % |
| `hashbrown` table operations | 5.0 % |
| `debruijn::unitig_iter` | 4.8 % |
| memcpy/memset, malloc/free, file I/O, SipHash | ≈ 7 % |
| other | 13.7 % |
- **The de Bruijn graph is ≈ 72 % of the active CPU of `index`; the MPHF construction does not appear (< 1 %)**, and `SlottedKmers`/evidence filling is negligible. The conclusion of §16 holds at 100× the E. coli scale.
- **The stage is poorly parallel**: on average ≈ 5 of 16 threads are active (33 % of thread-samples); `swtch_pri` (spin/yield) is 7 % of all thread-samples. With 16 partitions of unequal size the tail partitions run alone. The 12 % system CPU is the same spinning as §16, at a smaller relative weight.
- **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.
Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture.
## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)
Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench <unitigs.bin> <k> [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`.
| | push | degrees | unitigs | unitigs produced | nucleotides |
|---|---|---|---|---|---|
| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M |
| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M |
| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** |
Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).
**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision).
**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).
**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig).
Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).
**End-to-end** (current binary vs previous one, same inputs):
| | before | after |
|---|---|---|
| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) |
| … `index` stage | 114.6 s | **59.2 s** (−48 %) |
| … user CPU | 1 870 s | 1 304 s |
| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB |
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
## 19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15)
`Evidence` packs `(chunk_id, rank)` into a `u32` with a 7-bit rank (`encode` masks `& 0x7F`), but `MAX_KMERS_PER_CHUNK` was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: **`--partitions 256`** → max chunk 49–50 k-mers, results correct with the old and the new binary; **`--partitions 1`** → max chunk 256, query output wrong with both the old (md5 `bde0f886a0`) and the post-graph-fix (`8f1c94ef32`) binary versus the correct `f99e971087`. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts.
**Change** (`obikseq::MAX_KMERS_PER_CHUNK`: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (`SuperKmer::write_to_binary`): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test `chunk_ranks_fit_the_7_bit_evidence_field`.
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
## 20. Summary of what was gained (2026-10-15)
Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated.
### Speed
| Operation | Original | Now | Where |
|---|---|---|---|
| Open an index (query of one 64-base sequence), 91 genomes, 180 GB | 10.4–13.3 s | **0.18–0.19 s** | §14, same-session A/B |
| Query, 2 × 48 MB reads on that index (`--chunk-size 8`) | 30.7–32.0 s (61.8 s cold), 70–76 s CPU | **11.6–13.2 s**, 36–39 s CPU (−62 % wall, −50 % CPU) | §14, same-session A/B |
| `dump --head 10` on that index | 9.0–10.0 s (91–102 s system CPU) | **0.16–0.84 s** | §15 |
| `index` of a 2.1 GB skim (*V. vitis-idaea*, `--partitions 16`) | 163.8 s (stage `index` 114.6 s) | **116.5 s** (stage `index` 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU | §18 |
| `index` of E. coli reads | 1.97 s (stage 0.91 s), 10.2 s CPU | 1.63 s (stage 0.59 s), 7.3 s CPU | §18 |
| De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) | 4.1 s / 2.0 s / 3.9 s | 1.0 s / 0.8 s / 1.0 s (×3.5 overall) | §18 |
| `index` stage `count_kmer` (E. coli) | 0.51–0.56 s | 0.42–0.50 s | §15 |
| Query on a small input (2 × 48 MB reads), single-genome index | 11.4 s (2 chunks = 2 busy cores) | **2.4 s** (−79 %) — automatic chunk size now follows the input size (`architecture/query.md`); on the 91-genome index the size was already 4 MiB | §20 follow-up |
### Memory (peak RSS / footprint)
| Operation | Original | Now |
|---|---|---|
| Opening the 91-genome index (any command that builds an `IndexCache`) | **27.4 GB** | **0.04 GB** (§14: `.idx` offsets and `dict_values.bin` mapped instead of copied; they were read twice and once) |
| Query, 2 × 48 MB reads on that index | 32.2–32.5 GB | **3.5–3.6 GB** |
| Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size | **73.7–80.7 GB** | **28.2 GB** (§12 of `architecture/query.md`: `PartitionKmers` instead of a `HashMap<kmer, Vec>` per partition: 71 → ~20 bytes per unique kmer) |
| `index` of the 2.1 GB skim | 19.7 GB | 18.3 GB |
### Index size (newly built indexes; existing ones are unchanged)
| | Original | Now |
|---|---|---|
| `unitigs.bin` / `unitigs.bin.idx` (2.1 GB skim, 521.6 M kmers, 16 partitions) | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18) |
| Whole index of that skim | 3.5 GB | **2.9 GB** (−17 %) |
| `evidence.bin` | 4 bytes per kmer | unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented) |
### Correctness fixes found on the way
- Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18).
- Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, `scripts/check_unitig_chunks.py` to audit an index).
- `index.meta` and super-kmer files were parsed byte by byte (§15).
### What is **not** established
- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone).
- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).
## 21. Dead code removed (2026-10-15)
The items listed as unused in §4 were removed after Serena's `safe_delete_symbol` and the compiler confirmed no reference: `MphfLayer::find_strict`; the single-kmer `hash` chain (`MphfLayer::hash`, `TypedLayer::hash`, `KmerLayer::hash`, `IndexCache::hash`); `IndexCache::find`; the uncompiled `obikmerge/src/merge_layer/` directory (689 lines; `lib.rs` only declares `merge` and `partition_merge`); and `xxhash-rust` from `obidebruinj/Cargo.toml` (it still serves `obikseq`, `obifastwrite` and `obiksession`).
Side effect: `find_strict` was the only reader of the `unitigs` handle that `Hybrid` layers opened, so that field (and the extra `.idx` map and mmap per Hybrid layer) was dropped too. `find` on a Hybrid layer was already a fingerprint check; the exact evidence stays on disk (it gives the slot count and is what `convert` works from) but no API consults it any more. Restoring an exact lookup for Hybrid layers means re-adding `find_strict` and that field.
`ahash` was not referenced by any source file of `obidebruinj` either and was removed from its `Cargo.toml` too.
Checks after the removal: `cargo build --release` without warnings, `cargo test --workspace` 555 passed / 0 failed, `scripts/smoke_test_index.sh` PASS.
Scratch artifacts of this work were deleted: the intermediate binaries, the two skim indexes (`skim_p4`, `skim_p16`) and the shared data set `benchmark/run/bench_mphf/data` (rebuilt automatically, ≈ 2 min, by the next `bench_mphf.sh`). Kept in `benchmark/run/bench_mphf/`: the original binary `obikmer_baseline` (reference for A/B runs) and the per-label results and checksums (18 MB in all). The 91-genome index converted to the current MPHF format (`/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`, with its `index.meta` patched, original `index.meta.before_state_fix` beside it) was left in place.
+4 -4
View File
@@ -22,10 +22,10 @@ A sequence of binary records. Each record:
- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded. - Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
- Byte count for packed sequence: `ceil(seql / 4)`. - Byte count for packed sequence: `ceil(seql / 4)`.
Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost: Unitigs with more than `MAX_KMERS_PER_CHUNK = 128` k-mers are transparently split into overlapping chunks. Each chunk has at most 128 k-mers (= `seql − k + 1 ≤ 128`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
``` ```
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (256 kmers) chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (128 kmers)
chunk 2: nucleotides [256, end] (remaining kmers) chunk 2: nucleotides [256, end] (remaining kmers)
overlap: k−1 nucleotides shared between the two chunks overlap: k−1 nucleotides shared between the two chunks
``` ```
@@ -95,11 +95,11 @@ Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extract
| field | bits | range | capacity check (*B. nana*, 256 partitions) | | field | bits | range | capacity check (*B. nana*, 256 partitions) |
|------------|------|---------------|---------------------------------------------| |------------|------|---------------|---------------------------------------------|
| `seql − k` | 8 | 0–255 | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` | | `seql − k` | 8 | 0–127 used | max `n_kmers` per chunk = 128 = `MAX_KMERS_PER_CHUNK` (the field could carry 256; capped so that every rank fits 7 bits) |
| `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 | | `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
| `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition | | `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition |
The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total. The rank field is 7 bits (max 127) and the rank of a k-mer is its index among the chunk's k-mers, so a chunk must hold at most 128 of them: that is what `MAX_KMERS_PER_CHUNK = 128` guarantees. (Until 2026-10-15 the cap was 256 and the earlier justification here — that rank counts only canonical k-mers, at most half the total — was wrong: `iter_indexed_canonical_kmers` yields one entry per position. Chunks of 129–256 k-mers occur with few partitions — 6 of them in a `--partitions 4` index of a 2 GB skim, none with 256 partitions where the observed maximum is ≈ 50 — and their k-mers beyond rank 127 were silently stored with a truncated rank. `scripts/check_unitig_chunks.py` finds such chunks in an existing index.)
--- ---
+28 -1
View File
@@ -329,7 +329,34 @@ Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 /
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above. **`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment has not been given to IQ-TREE. **Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment had not been given to IQ-TREE (done 2026-10-16: see the last measured section).
## Tree-level effect of the combined draw: sCF on three trees (measured 2026-10-16)
Closes the "not measured" item of the previous section (the combined alignment given to IQ-TREE). Three trees, all sCF by `--scfl 100 --cf-verbose`, model `GTR+G4+ASC`:
- **A** = `concord` (12/10): entropy-only draw, 180 genomes, 5 003 414 columns; `-t` + `-m` re-ran a full ML search (59 h).
- **B** = `concord_quartet_balance`: combined draw (`--entropy 1 --quartet --pair-balance`), 180 genomes, 5 000 000 columns. Its log stops during the tree search (14/10), so the command that produced its `cf.*` (16/10) is not in the files; presumably `-te` on the 15/10 treefile as for G (unverified).
- **G** = `scf_good`: same draw as B without the 3 data-poor genomes (*A. pyrenaica* PHA000543, *A. saussurei*, *A. triflora*), 177 genomes. 72 sites became invariant and were removed to satisfy ASC; the search (`concord_quartet_balance_good`, 107 iterations, 6 h 47 on 120 threads) crashed in the sCF step on the FASTA (`Assertion all_prob_const < 1.0`; cause untested, ASC with constant sites is a suspect). sCF were recomputed with `-te <treefile> -s <varsites.phy> -m GTR+G4+ASC --scfl 100`: **no search** (0 iterations, lnL −41 053 676.53 vs −41 053 676.41 from the search), 4 999 729 columns. A used `-t`, B/G a fixed tree: the protocols differ.
| | A | B | G |
|---|---|---|---|
| Branches | 177 | 177 | 174 |
| Mean `sN` | 3 219 | 13 163 | 13 396 |
| Branches with `sN` < 100 / < 1 000 | 13 / 50 | 0 / 5 | 0 / 2 |
| Median sCF | 44.2 | 42.9 | 42.8 |
| Branches with sCF < 40 / ≥ 70 | 74 / 28 | 67 / 26 | 69 / 25 |
| *Primula* and *Androsace* monophyletic | no / no | no / no | yes / yes |
**Sampling effect (A → B, same 180 genomes).** `sN` ×4.5 and the scarcity is gone. Sites do not become more concordant: on the 113 shared bipartitions (of 177) median sCF 48.0 → 47.3 (44 up, 69 down). Robinson-Foulds 128: the draw rearranged the tree deeply. Consistent with the earlier reading that the intra-*Androsace* conflict is not a matter of site count; the number of branches with sCF < 40 % stays around 70.
**Interface nodes.** Of the old nodes 209/211/212/216/217 (`sN` 17 / 226 / 184 / 7 / 7) none exists in G. Node 208 survives with `sN` 190 → 5 218 (sCF 54.5 → 37.5). Six mixed clades remain in G (83 to 72 tips, 50 *Primula* + 22-33 *Androsace*), all with `sN` > 3 000.
**Effect of the 3 genomes (B → G, same draw).** RF 16 (166 of 174 shared), median sCF 43.2 → 43.6, median `sN` unchanged. In B, *A. pyrenaica* and *A. triflora* sit inside *Primula* (a clade of 52 tips = 50 *Primula* + these 2; each has a *Primula* sister, sCF 92 % and 52 %, `sN` 556 and 153); *A. saussurei* stays with *Androsace*. Removing them restores both monophylies and changes nothing else measurably: support for "too little data to place them" (data-poor section), not for contamination, though the latter is not excluded.
**Where A and G disagree (2026-10-16).** On the 177 common tips: 124 shared bipartitions, 50 only in A, 50 only in G. Differing bipartitions are weakly supported in both trees (median sCF 35.7 in A-only, 37.7 in G-only, versus 47.5-48.1 on the shared ones), so the disagreement sits where support was already low. Differences are mostly local: the closest bipartition of the other tree is 1 tip away for 25 of 50 A-only and 21 of 50 G-only splits (≤ 2 tips: 37 and 30); only 8 and 9 are 5+ tips apart. Greedy removal of the tip that lowers RF most: the first (*P. auricula* subsp. *serratifolia*) takes RF from 100 to 86, then about 2 per tip (*A. maxima*, *P. stricta* var. *obesior*, then ordinary tips); the top three are data-poor genomes. A bipartition separates a data-poor genome (the 5 left after the 3 removals) in 1 of 124 shared splits but in 23 of 50 A-only and 10 of 50 G-only splits; removing those 5 genomes gives RF 72, against 96.6 (range 90-100, 40 draws) for 5 random tips. So about 28 of the 100 differences trace to data-poor genomes, and the remaining 72 do not. Backbone: A has a clade of 50 *Primula* + 25 *Androsace* (sCF 76, `sN` 453), G one of 50 + 26 (sCF 73, `sN` 6 677): one tip apart. Genuine conflict: a clade of 72 *Androsace* exists in both trees (A: sCF 34, `sN` 17 127; G: sCF 38, `sN` 49 473) with 16 different tips, i.e. a topological disagreement carried by tens of thousands of sites in each, not a scarcity effect; consistent with the intra-*Androsace* conflict above.
**Not established.** Separate contribution of entropy / quartet / pair balance (no ablation); stability of the tree between two independent draws (one draw per setting; A → B shows the topology moves with the draw); whether the quartet-informative restriction biases branch lengths (ASC corrects for variable-only ascertainment, not for requiring two states each in two genomes; tree length 22.5 is not interpretable as substitutions/site).
## Open questions ## Open questions
+12
View File
@@ -42,6 +42,18 @@ obikmer:
debug: debug:
cargo build --manifest-path $(CARGO_DIR)/Cargo.toml --bin obikmer cargo build --manifest-path $(CARGO_DIR)/Cargo.toml --bin obikmer
# One-shot converter of indexes built with ptr_hash 1.x (see tools/migrate-mphf).
# Standalone crate, outside the src/ workspace, with its own Cargo.lock/target.
MIGRATE_MPHF_DIR := tools/migrate-mphf
.PHONY: migrate-mphf
migrate-mphf:
cargo build --release --manifest-path $(MIGRATE_MPHF_DIR)/Cargo.toml
.PHONY: clean-migrate-mphf
clean-migrate-mphf:
rm -rf $(MIGRATE_MPHF_DIR)/target
.PHONY: profile-build .PHONY: profile-build
profile-build: profile-build:
cargo build --release --manifest-path $(CARGO_DIR)/Cargo.toml \ cargo build --release --manifest-path $(CARGO_DIR)/Cargo.toml \
+1 -1
View File
@@ -48,7 +48,7 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
| Option | Description | | Option | Description |
|---|---| |---|---|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value | | `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex, then every annex derived from it (entropy, quartet, pair weight, family overlap, under both `--no-ambiguity` conventions). Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally | | `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only | | `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for | | `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
+152
View File
@@ -0,0 +1,152 @@
#!/usr/bin/env bash
# Usage: bench_mphf.sh LABEL
#
# Measures the obikmer commands that sit on MPHF lookups (see
# DevDocMD/implementation/mphf_batch_queries.md for the site inventory), on the
# 20-genome benchmark data already built under run/ (`gmake all` first).
# Nothing under run/ other than run/bench_mphf/ is written.
#
# Env:
# BINARY obikmer binary to measure (default ../src/target/release/obikmer)
# REPS repetitions per step (default 3)
# CPU_MAX passed as --cpu-max when set (default: all cores)
#
# Output: run/bench_mphf/LABEL/results.csv (label,step,rep,wall_s,maxrss_b)
# run/bench_mphf/LABEL/checksums.txt (order-independent output digests,
# to be diffed across LABELs: a speed-up must not change any of them)
set -euo pipefail
LABEL="$1"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
RUN_DIR="${SCRIPT_DIR}/run"
BINARY="${BINARY:-${SCRIPT_DIR}/../src/target/release/obikmer}"
REPS="${REPS:-3}"
OUT="${RUN_DIR}/bench_mphf/${LABEL}"
CPU=()
[[ -n "${CPU_MAX:-}" ]] && CPU=(--cpu-max "${CPU_MAX}")
rm -rf "${OUT}"; mkdir -p "${OUT}"
RESULTS="${OUT}/results.csv"
SUMS="${OUT}/checksums.txt"
echo "label,step,rep,wall_s,maxrss_b" >"${RESULTS}"
: >"${SUMS}"
# timed STEP REP CMD... — wall time + peak RSS via BSD time(1); stdout of CMD
# goes to $OUT/last.stdout, stderr to $OUT/last.stderr.
timed() {
local step="$1" rep="$2"; shift 2
local tf="${OUT}/time.tmp"
/usr/bin/time -l -o "${tf}" "$@" >"${OUT}/last.stdout" 2>"${OUT}/last.stderr" \
|| { echo "FAILED: $step: $*" >&2; cat "${OUT}/last.stderr" >&2; exit 1; }
local wall rss
wall=$(awk '/ real /{print $1}' "${tf}")
rss=$(awk '/maximum resident set size/{print $1}' "${tf}")
echo "${LABEL},${step},${rep},${wall},${rss}" >>"${RESULTS}"
# per-stage wall times from the Reporter log lines ("done stage=X wall_secs=Y")
sed $'s/\x1b\\[[0-9;]*m//g' "${OUT}/last.stderr" \
| awk -v L="${LABEL}" -v S="${step}" -v R="${rep}" \
'/ done stage=/{ for(i=1;i<=NF;i++){ if($i ~ /^stage=/) st=substr($i,7); if($i ~ /^wall_secs=/) w=substr($i,11) } print L","S":"st","R","w",0" }' \
>>"${RESULTS}"
printf ' %-28s rep %s %8s s\n' "${step}" "${rep}" "${wall}" >&2
}
digest() { LC_ALL=C sort "$1" | md5 -q; }
# ── 0. data: the indexes under run/ may have been built by an older
# MPHF generation (ptr_hash 1.x, unreadable by current binaries), so the
# bench builds its own, once, from the simulated reads. Shared by all LABELs;
# delete run/bench_mphf/data to force a rebuild.
DATA="${RUN_DIR}/bench_mphf/data"
if [[ ! -f "${DATA}/.ready" ]]; then
echo "preparing data with ${BINARY}" >&2
mkdir -p "${DATA}/presence" "${DATA}/count"
for d in "${RUN_DIR}"/simulated_data/*/*/; do
strain="$(basename "$d")"; species="$(basename "$(dirname "$d")")"
sp="${species}--${strain}"
for kind in presence count; do
extra=(); [[ $kind == count ]] && extra=(--with-counts)
"${BINARY}" "${CPU[@]}" index --output "${DATA}/${kind}/${sp}" --force \
--theta 0 "${extra[@]}" --label "${sp}" --meta "species=${species}" \
"$d/reads_R1.fastq.gz" "$d/reads_R2.fastq.gz" 2>/dev/null
done
done
"${BINARY}" "${CPU[@]}" merge --output "${DATA}/global_presence" --force \
--force-presence "${DATA}"/presence/*/ 2>/dev/null
"${BINARY}" "${CPU[@]}" merge --output "${DATA}/global_count" --force \
"${DATA}"/count/*/ 2>/dev/null
touch "${DATA}/.ready"
fi
sources=("${DATA}"/presence/*/)
# ── 1. index: build pass 2 (mphf.index per kmer + evidence writes) ──────────
for sp in Escherichia_coli--K-12_MG1655 Candidozyma_auris--GCF_003013715.1_ASM301371v2; do
species="${sp%%--*}"; strain="${sp#*--}"
reads="${RUN_DIR}/simulated_data/${species}/${strain}"
for rep in $(seq "${REPS}"); do
timed "index_${species}" "${rep}" "${BINARY}" "${CPU[@]}" index \
--output "${OUT}/idx_${species}" --force --theta 0 --label "${sp}" \
"${reads}/reads_R1.fastq.gz" "${reads}/reads_R2.fastq.gz"
done
done
# ── 2. merge: find_batch_grouped, build_graph filter, hash_batch, index_batch ─
for rep in $(seq "${REPS}"); do
timed "merge_20_presence" "${rep}" "${BINARY}" "${CPU[@]}" merge \
--output "${OUT}/merged" --force --force-presence "${sources[@]}"
done
"${BINARY}" "${CPU[@]}" dump "${OUT}/merged" >"${OUT}/merged.dump"
echo "merged_dump $(digest "${OUT}/merged.dump")" >>"${SUMS}"
rm -f "${OUT}/merged.dump"
# ── 3. query: query stage 1 (find per kmer, layer by layer) ─────────────────
for kind in presence count; do
for sp in Escherichia_coli/K-12_MG1655 Saccharolobus_islandicus/M.16.4; do
reads="${RUN_DIR}/query_data/${sp}"
for rep in $(seq "${REPS}"); do
timed "query_${kind}_${sp%%/*}" "${rep}" "${BINARY}" "${CPU[@]}" query \
--count-missing "${DATA}/global_${kind}" \
"${reads}/reads_R1.fastq.gz" "${reads}/reads_R2.fastq.gz"
done
echo "query_${kind}_${sp%%/*} $(digest "${OUT}/last.stdout")" >>"${SUMS}"
done
done
# ── 4. filter: iter_layer_kmers → hash_batch + fill_sub_matrix ──────────────
for rep in $(seq "${REPS}"); do
timed "filter_species" "${rep}" "${BINARY}" "${CPU[@]}" filter \
--output "${OUT}/filtered" --force --ingroup "species=Escherichia_coli" \
--outgroup all --min-frac 0.5 --max-frac 1.0 --max-outgroup-count 0 \
"${DATA}/global_presence"
done
"${BINARY}" "${CPU[@]}" dump "${OUT}/filtered" >"${OUT}/filtered.dump"
echo "filtered_dump $(digest "${OUT}/filtered.dump")" >>"${SUMS}"
rm -f "${OUT}/filtered.dump"
# ── 5. convert: build_exact_evidence / build_approx_evidence ────────────────
# On an APFS clone (cp -c) so run/global_index_presence is untouched.
for rep in $(seq "${REPS}"); do
rm -rf "${OUT}/conv"; cp -cR "${DATA}/global_presence" "${OUT}/conv"
timed "convert_to_approx8" "${rep}" "${BINARY}" "${CPU[@]}" convert \
--approx-evidence 8 "${OUT}/conv"
timed "convert_to_exact" "${rep}" "${BINARY}" "${CPU[@]}" convert \
--exact-evidence "${OUT}/conv"
done
"${BINARY}" "${CPU[@]}" dump "${OUT}/conv" >"${OUT}/conv.dump"
echo "convert_roundtrip_dump $(digest "${OUT}/conv.dump")" >>"${SUMS}"
rm -rf "${OUT}/conv" "${OUT}/conv.dump"
# ── 6. phylo: sibling annex (annex.rs: find_in_partition per variant) and
# its consumer (family_scan.rs: hash_batch / find_slot) ──────────────────
for rep in $(seq "${REPS}"); do
rm -rf "${OUT}/phy"; cp -cR "${DATA}/global_presence" "${OUT}/phy"
timed "phylo_sibling_annex" "${rep}" "${BINARY}" "${CPU[@]}" phylo \
--sibling-annex "${OUT}/phy"
timed "phylo_snp_distance" "${rep}" "${BINARY}" "${CPU[@]}" phylo \
--output "${OUT}/snp" --distance snp-jc --csv "${OUT}/phy"
done
echo "phylo_snp_dist $(digest "${OUT}/snp_dist.csv")" >>"${SUMS}"
rm -rf "${OUT}/phy" "${OUT}/idx_"* "${OUT}/merged" "${OUT}/filtered" \
"${OUT}/last.stdout" "${OUT}/time.tmp"
echo "results: ${RESULTS}" >&2
echo "checksums: ${SUMS}" >&2
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env python3
"""Usage: summarize_bench.py LABEL [LABEL...] — median wall per step, side by side."""
import csv, statistics, sys, os
base = os.path.join(os.path.dirname(os.path.abspath(__file__)), "run", "bench_mphf")
data = {}
for lab in sys.argv[1:]:
for r in csv.DictReader(open(f"{base}/{lab}/results.csv")):
data.setdefault(r["step"], {}).setdefault(lab, []).append(float(r["wall_s"]))
print(f"{'step':32}" + "".join(f"{l:>14}" for l in sys.argv[1:]) + (" ratio" if len(sys.argv) > 2 else ""))
for step, d in data.items():
med = [statistics.median(d[l]) for l in sys.argv[1:]]
line = f"{step:32}" + "".join(f"{m:14.2f}" for m in med)
if len(med) > 1 and med[0]: line += f" {med[-1]/med[0]:.3f}"
print(line)
+1
View File
@@ -48,6 +48,7 @@ nav:
- obipipeline library: implementation/obipipeline.md - obipipeline library: implementation/obipipeline.md
- On-disk storage: implementation/storage.md - On-disk storage: implementation/storage.md
- MPHF selection: implementation/mphf.md - MPHF selection: implementation/mphf.md
- "MPHF batch queries (inventory)": implementation/mphf_batch_queries.md
- Unitig evidence encoding: implementation/unitig_evidence.md - Unitig evidence encoding: implementation/unitig_evidence.md
- Evidence elimination (discussion): implementation/evidence_elimination.md - Evidence elimination (discussion): implementation/evidence_elimination.md
- obikindex layer tier: implementation/layer_tier.md - obikindex layer tier: implementation/layer_tier.md
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env python3
# check_unitig_chunks.py GLOB... — report unitigs.bin chunks holding more than 128 k-mers.
# Indexes built before MAX_KMERS_PER_CHUNK went from 256 to 128 (2026-10-15) could hold chunks whose
# rank does not fit the 7-bit evidence field; the k-mers beyond rank 127 of such a chunk then fail
# exact verification. Example: scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"
# (k is hard-coded below — edit it for an index built with another k.)
import sys,glob,mmap
k=31
def scan(path):
with open(path,'rb') as f:
m=mmap.mmap(f.fileno(),0,access=mmap.ACCESS_READ)
o=0; n=len(m); mx=0; over=0; chunks=0
while o<n:
c=m[o]+1; chunks+=1
if c>mx: mx=c
if c>128: over+=1
o+=1+(m[o]+k+3)//4
return mx,over,chunks
tot_over=0
for pat in sys.argv[1:]:
for f in sorted(glob.glob(pat)):
mx,over,ch=scan(f); tot_over+=over
print(f"{f.split('partitions/')[1]}: chunks={ch} max_kmers={mx} chunks>128={over}")
print("TOTAL chunks over 128 kmers:",tot_over)
+1 -18
View File
@@ -8,19 +8,6 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "ahash"
version = "0.8.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
dependencies = [
"cfg-if",
"getrandom 0.3.4",
"once_cell",
"version_check",
"zerocopy",
]
[[package]] [[package]]
name = "aho-corasick" name = "aho-corasick"
version = "1.1.4" version = "1.1.4"
@@ -1419,14 +1406,12 @@ dependencies = [
name = "obidebruinj" name = "obidebruinj"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"ahash",
"crossbeam-channel", "crossbeam-channel",
"hashbrown 0.17.1", "hashbrown 0.17.1",
"obifastwrite", "obifastwrite",
"obikseq", "obikseq",
"rayon", "rayon",
"tracing", "tracing",
"xxhash-rust",
] ]
[[package]] [[package]]
@@ -1528,7 +1513,6 @@ dependencies = [
name = "obikindexer" name = "obikindexer"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"cacheline-ef",
"epserde", "epserde",
"memmap2", "memmap2",
"niffler", "niffler",
@@ -1543,7 +1527,6 @@ dependencies = [
"obiskbuilder", "obiskbuilder",
"obiskio", "obiskio",
"obisys", "obisys",
"ptr_hash",
"rayon", "rayon",
"serde", "serde",
"serde_json", "serde_json",
@@ -1554,7 +1537,7 @@ dependencies = [
[[package]] [[package]]
name = "obikmer" name = "obikmer"
version = "1.3.6" version = "1.4.1"
dependencies = [ dependencies = [
"clap", "clap",
"csv", "csv",
+43 -5
View File
@@ -25,6 +25,7 @@ use std::fs;
use std::io; use std::io;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use memmap2::Mmap;
use ndarray::{Array1, Array2}; use ndarray::{Array1, Array2};
use crate::traits::{BitPartials, ColumnWeights}; use crate::traits::{BitPartials, ColumnWeights};
@@ -151,7 +152,9 @@ pub struct PersistentSparseBitMatrix {
singleton: PersistentFixedIntVec, singleton: PersistentFixedIntVec,
multi: PersistentFixedIntVec, multi: PersistentFixedIntVec,
dict_offsets: EliasFano, dict_offsets: EliasFano,
dict_values: Vec<u8>, /// `dict_values.bin`, mapped (never copied to the heap); `None` when the
/// file is empty — no multi-genome set — since an empty file cannot be mapped.
dict_values: Option<Mmap>,
/// Row count (k-mer slots). /// Row count (k-mer slots).
n: usize, n: usize,
/// Column count (genomes). /// Column count (genomes).
@@ -167,7 +170,12 @@ impl PersistentSparseBitMatrix {
let singleton = PersistentFixedIntVec::open(&singleton_path(dir))?; let singleton = PersistentFixedIntVec::open(&singleton_path(dir))?;
let multi = PersistentFixedIntVec::open(&multi_path(dir))?; let multi = PersistentFixedIntVec::open(&multi_path(dir))?;
let dict_offsets = EliasFano::open(&dict_offsets_base(dir))?; let dict_offsets = EliasFano::open(&dict_offsets_base(dir))?;
let dict_values = fs::read(dict_values_path(dir))?; let dict_file = fs::File::open(dict_values_path(dir))?;
let dict_values = if dict_file.metadata()?.len() == 0 {
None
} else {
Some(unsafe { Mmap::map(&dict_file)? })
};
Ok(Self { Ok(Self {
is_multi, singleton, multi, dict_offsets, dict_values, is_multi, singleton, multi, dict_offsets, dict_values,
n: meta.n, n_cols: meta.n_cols, n_distinct_multi: meta.n_distinct_multi, n: meta.n, n_cols: meta.n_cols, n_distinct_multi: meta.n_distinct_multi,
@@ -179,6 +187,11 @@ impl PersistentSparseBitMatrix {
#[inline] #[inline]
pub fn n_cols(&self) -> usize { self.n_cols } pub fn n_cols(&self) -> usize { self.n_cols }
#[inline]
fn dict_bytes(&self) -> &[u8] {
self.dict_values.as_deref().unwrap_or(&[])
}
/// Byte range of `dict_id`'s varint-encoded genome list within /// Byte range of `dict_id`'s varint-encoded genome list within
/// `dict_values`. /// `dict_values`.
#[inline] #[inline]
@@ -187,7 +200,7 @@ impl PersistentSparseBitMatrix {
let end = if dict_id + 1 < self.n_distinct_multi { let end = if dict_id + 1 < self.n_distinct_multi {
self.dict_offsets.get(dict_id + 1) as usize self.dict_offsets.get(dict_id + 1) as usize
} else { } else {
self.dict_values.len() self.dict_bytes().len()
}; };
(start, end) (start, end)
} }
@@ -232,7 +245,7 @@ impl PersistentSparseBitMatrix {
let (start, end) = self.dict_entry_range(dict_id); let (start, end) = self.dict_entry_range(dict_id);
let mut p = start; let mut p = start;
while p < end { while p < end {
f(read_varint(&self.dict_values, &mut p) as usize); f(read_varint(self.dict_bytes(), &mut p) as usize);
} }
} else { } else {
let pos = self.is_multi.rank0(slot) as usize; let pos = self.is_multi.rank0(slot) as usize;
@@ -287,7 +300,7 @@ impl PersistentSparseBitMatrix {
present.clear(); present.clear();
let mut p = start; let mut p = start;
while p < end { while p < end {
present.push(read_varint(&self.dict_values, &mut p) as usize); present.push(read_varint(self.dict_bytes(), &mut p) as usize);
} }
for &g in &present { for &g in &present {
counts[g] += w; counts[g] += w;
@@ -514,6 +527,31 @@ impl PersistentSparseBitMatrixBuilder {
} }
Ok(builder) Ok(builder)
} }
/// Builds the row-permuted copy of `src` into `dir`: row `s` of the
/// result is row `new_to_old[s]` of `src`. `new_to_old` must have
/// `src.n()` entries and be a permutation of `0..src.n()` — the caller
/// proves that, this only reads `src` at those slots.
///
/// Needed whenever the slot numbering of a layer changes (its MPHF is
/// rebuilt) while the data attached to each k-mer must follow it.
/// Rows are read at arbitrary slots of `src`, so convert a dense source
/// to sparse first (`build_from_dense`) rather than permuting it
/// cell by cell.
pub fn build_permuted(
src: &PersistentSparseBitMatrix,
new_to_old: &[u32],
dir: &Path,
) -> io::Result<Self> {
let mut builder = Self::new(new_to_old.len(), src.n_cols(), dir)?;
let mut genomes: Vec<u32> = Vec::new();
for &old in new_to_old {
genomes.clear();
src.for_each_genome_in_row(old as usize, |g| genomes.push(g as u32));
builder.push_row(&genomes);
}
Ok(builder)
}
} }
// ── Trait impls ─────────────────────────────────────────────────────────────── // ── Trait impls ───────────────────────────────────────────────────────────────
+23
View File
@@ -548,6 +548,29 @@ impl PersistentSparseCompactIntMatrixBuilder {
Ok(builder) Ok(builder)
} }
/// Row-permuted copy of `src`: row `s` of the result is row
/// `new_to_old[s]` of `src`. Same contract as
/// [`PersistentSparseBitMatrixBuilder::build_permuted`].
pub fn build_permuted(
src: &PersistentSparseCompactIntMatrix,
new_to_old: &[u32],
dir: &Path,
) -> io::Result<Self> {
let mut builder = Self::new(new_to_old.len(), src.n_cols(), dir)?;
let mut cols: Vec<u32> = Vec::new();
let mut vals: Vec<u32> = Vec::new();
for &old in new_to_old {
cols.clear();
vals.clear();
src.for_each_cell_in_row(old as usize, |c, v| {
cols.push(c as u32);
vals.push(v);
});
builder.push_row(&cols, &vals);
}
Ok(builder)
}
pub fn close(self) -> io::Result<()> { pub fn close(self) -> io::Result<()> {
let Self { let Self {
support, support,
+28
View File
@@ -415,3 +415,31 @@ fn partial_jaccard_and_hamming_match_dense_including_diagonal() {
drop(dir_dense); drop(dir_dense);
drop(dir_sparse); drop(dir_sparse);
} }
#[test]
fn build_permuted_follows_the_permutation() {
// Mix of singleton and multi rows, including a repeated multi set, so the
// dictionary dedup is exercised on the permuted order too.
let rows: Vec<&[bool]> = vec![
&[true, false, false, false],
&[false, false, true, false],
&[true, true, false, false],
&[false, false, true, true],
&[true, true, false, false],
];
let (_dir, src) = make_sparse(&rows, 4);
let new_to_old: Vec<u32> = vec![3, 0, 4, 1, 2];
let out = tempdir().unwrap();
let permuted =
PersistentSparseBitMatrixBuilder::build_permuted(&src, &new_to_old, &out.path().join("p"))
.unwrap()
.finish()
.unwrap();
assert_eq!(permuted.n(), src.n());
assert_eq!(permuted.n_cols(), src.n_cols());
for (new, &old) in new_to_old.iter().enumerate() {
assert_eq!(row_as_bool(&permuted, new), row_as_bool(&src, old as usize), "new row {new}");
}
}
@@ -304,3 +304,32 @@ fn pack_sparse_compact_int_matrix_from_already_packed_matrix() {
assert_eq!(&*sparse.row(slot), &**expected, "slot {slot}"); assert_eq!(&*sparse.row(slot), &**expected, "slot {slot}");
} }
} }
#[test]
fn build_permuted_follows_the_permutation() {
// Rows sharing a support with different values, plus singletons.
let rows: Vec<&[u32]> = vec![
&[7, 0, 0, 0],
&[0, 0, 3, 0],
&[2, 5, 0, 0],
&[9, 1, 0, 0],
&[0, 0, 4, 6],
];
let (_dir, src) = make_sparse(&rows, 4);
let new_to_old: Vec<u32> = vec![4, 2, 0, 3, 1];
let out = tempdir().unwrap();
let permuted = PersistentSparseCompactIntMatrixBuilder::build_permuted(
&src,
&new_to_old,
&out.path().join("p"),
)
.unwrap()
.finish()
.unwrap();
assert_eq!(permuted.n(), src.n());
for (new, &old) in new_to_old.iter().enumerate() {
assert_eq!(permuted.row(new), src.row(old as usize), "new row {new}");
}
}
-2
View File
@@ -6,11 +6,9 @@ edition = "2021"
[dependencies] [dependencies]
obikseq = { path = "../obikseq" } obikseq = { path = "../obikseq" }
obifastwrite = { path = "../obifastwrite" } obifastwrite = { path = "../obifastwrite" }
ahash = "0.8"
hashbrown = { version = "0.17", features = ["rayon"] } hashbrown = { version = "0.17", features = ["rayon"] }
rayon = "1" rayon = "1"
crossbeam-channel = "0.5" crossbeam-channel = "0.5"
xxhash-rust = { version = "0.8.15", features = ["xxh3", "const_xxh3"] }
tracing = "0.1" tracing = "0.1"
[features] [features]
+69 -40
View File
@@ -1,19 +1,19 @@
use hashbrown::HashMap; use hashbrown::{HashMap, HashSet};
use obikseq::k; use obikseq::k;
use obikseq::{CanonicalKmer, Unitig}; use obikseq::{CanonicalKmer, Unitig};
#[cfg(not(any(test, feature = "test-utils")))] #[cfg(not(any(test, feature = "test-utils")))]
use rayon::iter::{IntoParallelRefIterator, ParallelIterator}; use rayon::iter::{IntoParallelRefIterator, ParallelIterator};
use std::cell::RefCell; use std::cell::RefCell;
use std::sync::atomic::{AtomicU8, Ordering}; use std::sync::atomic::{AtomicU8, Ordering};
use xxhash_rust::xxh3::Xxh3Builder;
use super::kmer_hash::KmerHashBuilder;
use super::node::{IS_VISITED_MASK, Node}; use super::node::{IS_VISITED_MASK, Node};
use super::unitig_iter::UnitigNucIter; use super::unitig_iter::UnitigNucIter;
use super::walk::WalkState; use super::walk::WalkState;
// ── Types ───────────────────────────────────────────────────────────────────── // ── Types ─────────────────────────────────────────────────────────────────────
pub(super) type FastHashMap<K, V> = HashMap<K, V, Xxh3Builder>; pub(super) type FastHashMap<K, V> = HashMap<K, V, KmerHashBuilder>;
// ── GraphDeBruijn ───────────────────────────────────────────────────────────── // ── GraphDeBruijn ─────────────────────────────────────────────────────────────
@@ -24,7 +24,7 @@ pub struct GraphDeBruijn {
impl GraphDeBruijn { impl GraphDeBruijn {
pub fn new() -> Self { pub fn new() -> Self {
Self { Self {
nodes: FastHashMap::with_hasher(Xxh3Builder::new()), nodes: FastHashMap::with_hasher(KmerHashBuilder),
} }
} }
@@ -40,7 +40,7 @@ impl GraphDeBruijn {
pub fn with_capacity(capacity: usize) -> Self { pub fn with_capacity(capacity: usize) -> Self {
Self { Self {
nodes: FastHashMap::with_capacity_and_hasher(capacity, Xxh3Builder::new()), nodes: FastHashMap::with_capacity_and_hasher(capacity, KmerHashBuilder),
} }
} }
@@ -79,7 +79,7 @@ impl GraphDeBruijn {
if node.is_visited() { if node.is_visited() {
return; return;
} }
if self.is_start(*kmer, node) { if self.is_start(*kmer, node, atomic, true) || self.is_start(*kmer, node, atomic, false) {
node.set_start(); node.set_start();
atomic.store(node.0, Ordering::Relaxed); atomic.store(node.0, Ordering::Relaxed);
} }
@@ -98,45 +98,40 @@ impl GraphDeBruijn {
} }
} }
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState { pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState<'_> {
let node = Node( let atomic = self
self.nodes .nodes
.get(&kmer) .get(&kmer)
.expect("Cannot start a walk from not indexed kmer") .expect("Cannot start a walk from not indexed kmer");
.load(Ordering::Relaxed), let node = Node(atomic.load(Ordering::Relaxed));
);
debug_assert!( debug_assert!(
!node.is_visited(), !node.is_visited(),
"Cannot start a walk from a visited node" "Cannot start a walk from a visited node"
); );
WalkState::new(kmer, node, true) WalkState::new(kmer, node, true, atomic)
} }
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState> { pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState<'_>> {
let node = Node(self.nodes.get(&kmer)?.load(Ordering::Relaxed)); let atomic = self.nodes.get(&kmer)?;
let node = Node(atomic.load(Ordering::Relaxed));
if node.is_visited() { if node.is_visited() {
return None; return None;
} }
Some(WalkState::new(kmer, node, true)) Some(WalkState::new(kmer, node, true, atomic))
} }
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize) -> Option<UnitigNucIter<'_>> { pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize, direct: bool) -> Option<UnitigNucIter<'_>> {
let old = self let atomic = self.nodes.get(&kmer)?;
.nodes let old = atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
.get(&kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
if old & IS_VISITED_MASK != 0 { if old & IS_VISITED_MASK != 0 {
return None; return None;
} }
let start = WalkState::new(kmer, Node(old), true); let start = WalkState::new(kmer, Node(old), direct, atomic);
let next_step = start.walk(self).and_then(|(next_state, nuc)| { let next_step = start.walk(self).and_then(|(next_state, nuc)| {
let ext_old = self let ext_old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
.nodes
.get(&next_state.kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
(ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc)) (ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc))
}); });
Some(UnitigNucIter::new(self, kmer, k, next_step)) Some(UnitigNucIter::new(self, kmer, k, direct, next_step))
} }
pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) { pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) {
@@ -157,7 +152,8 @@ impl GraphDeBruijn {
.then_some(kmer) .then_some(kmer)
}) })
.for_each(|kmer| { .for_each(|kmer| {
if let Some(iter) = self.unitig_nucleotides(kmer, k) { let Some(direct) = self.start_direction(kmer) else { return };
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
n_new.fetch_add(1, Ordering::Relaxed); n_new.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -166,7 +162,8 @@ impl GraphDeBruijn {
#[cfg(any(test, feature = "test-utils"))] #[cfg(any(test, feature = "test-utils"))]
self.nodes.iter().for_each(|(&kmer, atomic)| { self.nodes.iter().for_each(|(&kmer, atomic)| {
if Node(atomic.load(Ordering::Acquire)).is_start() { if Node(atomic.load(Ordering::Acquire)).is_start() {
if let Some(iter) = self.unitig_nucleotides(kmer, k) { let Some(direct) = self.start_direction(kmer) else { return };
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
n_new.fetch_add(1, Ordering::Relaxed); n_new.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -187,8 +184,8 @@ impl GraphDeBruijn {
if node.is_visited() { if node.is_visited() {
continue; continue;
} }
let chain_start = self.find_chain_start(kmer); let (chain_start, direct) = self.find_chain_start(kmer);
if let Some(iter) = self.unitig_nucleotides(chain_start, k) { if let Some(iter) = self.unitig_nucleotides(chain_start, k, direct) {
n2.fetch_add(1, Ordering::Relaxed); n2.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -196,7 +193,7 @@ impl GraphDeBruijn {
// Safe because unitig_nucleotides may have visited kmer in the // Safe because unitig_nucleotides may have visited kmer in the
// meantime — in that case it returns None and we skip. // meantime — in that case it returns None and we skip.
if chain_start != kmer { if chain_start != kmer {
if let Some(iter) = self.unitig_nucleotides(kmer, k) { if let Some(iter) = self.unitig_nucleotides(kmer, k, true) {
n2.fetch_add(1, Ordering::Relaxed); n2.fetch_add(1, Ordering::Relaxed);
f(iter); f(iter);
} }
@@ -216,17 +213,34 @@ impl GraphDeBruijn {
self.nodes.extend(other.nodes); self.nodes.extend(other.nodes);
} }
fn find_chain_start(&self, kmer: CanonicalKmer) -> CanonicalKmer { /// Walks backwards from `kmer` to the first node of its chain. Returns
let node = Node(self.nodes[&kmer].load(Ordering::Acquire)); /// that node together with the orientation to read it in to walk the chain
let mut state = WalkState::new(kmer, node, false); /// forwards: the backward walk traverses nodes in the orientation
let mut seen = std::collections::HashSet::new(); /// `state.direct`, so the forward one is its opposite. (Dropping the
/// orientation, as this function once did, started the forward walk at the
/// wrong end and emitted a lone-kmer unitig.) On a cycle, `kmer` itself
/// in the canonical orientation.
fn find_chain_start(&self, kmer: CanonicalKmer) -> (CanonicalKmer, bool) {
let atomic = &self.nodes[&kmer];
let node = Node(atomic.load(Ordering::Acquire));
let mut state = WalkState::new(kmer, node, false, atomic);
// Most nodes reaching this pass are branch nodes or tips that cannot
// step at all: answer before allocating the cycle-detection set.
let Some((first, _)) = state.walk(self) else {
return (state.kmer, !state.direct);
};
let mut seen: HashSet<(u64, bool), KmerHashBuilder> = HashSet::with_hasher(KmerHashBuilder);
seen.insert((state.kmer.raw(), state.direct)); seen.insert((state.kmer.raw(), state.direct));
if !seen.insert((first.kmer.raw(), first.direct)) {
return (kmer, true);
}
state = first;
loop { loop {
match state.walk(self) { match state.walk(self) {
None => return state.kmer, None => return (state.kmer, !state.direct),
Some((next, _)) => { Some((next, _)) => {
if !seen.insert((next.kmer.raw(), next.direct)) { if !seen.insert((next.kmer.raw(), next.direct)) {
return kmer; return (kmer, true);
} }
state = next; state = next;
} }
@@ -234,8 +248,23 @@ impl GraphDeBruijn {
} }
} }
fn is_start(&self, query: CanonicalKmer, node: Node) -> bool { /// Orientation in which a node flagged as a start must be read: the
!WalkState::new(query, node, true).reachable(self) /// canonical one if a unitig starts there that way, else the reverse
/// complement. `None` if it was visited in the meantime.
fn start_direction(&self, kmer: CanonicalKmer) -> Option<bool> {
let atomic = self.nodes.get(&kmer)?;
let node = Node(atomic.load(Ordering::Acquire));
if node.is_visited() {
return None;
}
Some(self.is_start(kmer, node, atomic, true) || !self.is_start(kmer, node, atomic, false))
}
/// `true` when `query`, traversed in orientation `direct` (canonical or
/// reverse-complemented), has no unique predecessor — i.e. a unitig
/// starts there when read in that orientation.
fn is_start(&self, query: CanonicalKmer, node: Node, atomic: &AtomicU8, direct: bool) -> bool {
!WalkState::new(query, node, direct, atomic).reachable(self)
} }
pub fn try_for_each_unitig<E, F>(&self, f: F) -> Result<(), E> pub fn try_for_each_unitig<E, F>(&self, f: F) -> Result<(), E>
+55
View File
@@ -0,0 +1,55 @@
//! Hasher for the graph's `HashMap<CanonicalKmer, _>`.
//!
//! The key is one `u64` (the derived `Hash` of a kmer is a single
//! `write_u64`), so the generic streaming hashers used before (xxh3's
//! `Hasher` implementation: buffer, update, finish) cost far more than the
//! hash itself — ≈ 35 % of the active CPU of `index` in the profile of
//! `DevDocMD/implementation/mphf_batch_queries.md`, §17. A 64-bit finaliser
//! (splitmix64's) is a bijection with full avalanche, which is what
//! hashbrown needs from it (low bits for the bucket, top 7 for the tag).
use std::hash::{BuildHasher, Hasher};
#[derive(Clone, Copy, Default)]
pub(super) struct KmerHashBuilder;
impl BuildHasher for KmerHashBuilder {
type Hasher = KmerHasher;
#[inline]
fn build_hasher(&self) -> KmerHasher {
KmerHasher(0)
}
}
pub(super) struct KmerHasher(u64);
#[inline]
fn mix64(x: u64) -> u64 {
let x = x ^ (x >> 30);
let x = x.wrapping_mul(0xbf58476d1ce4e5b9);
let x = x ^ (x >> 27);
let x = x.wrapping_mul(0x94d049bb133111eb);
x ^ (x >> 31)
}
impl Hasher for KmerHasher {
#[inline]
fn write_u64(&mut self, x: u64) {
self.0 = mix64(self.0 ^ x);
}
/// Generic fallback (never taken for a kmer key): folds the bytes 8 at a time.
fn write(&mut self, bytes: &[u8]) {
for chunk in bytes.chunks(8) {
let mut buf = [0u8; 8];
buf[..chunk.len()].copy_from_slice(chunk);
self.write_u64(u64::from_le_bytes(buf));
}
}
#[inline]
fn finish(&self) -> u64 {
self.0
}
}
+2 -1
View File
@@ -1,10 +1,11 @@
//! De Bruijn graph over canonical k-mers, built for unitig extraction. //! De Bruijn graph over canonical k-mers, built for unitig extraction.
//! //!
//! Submodules: [`node`] (packed per-kmer neighbour/visited/start flags), //! Submodules: [`kmer_hash`] (hasher of the node table), [`node`] (packed per-kmer neighbour/visited/start flags),
//! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself), //! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself),
//! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator). //! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator).
mod graph; mod graph;
mod kmer_hash;
mod node; mod node;
mod unitig_iter; mod unitig_iter;
mod walk; mod walk;
+12 -8
View File
@@ -10,9 +10,11 @@ use super::walk::WalkState;
pub struct UnitigNucIter<'a> { pub struct UnitigNucIter<'a> {
graph: &'a GraphDeBruijn, graph: &'a GraphDeBruijn,
start: CanonicalKmer, start: CanonicalKmer,
/// Orientation the unitig is read in (`false`: reverse complement of `start`).
direct: bool,
pos: usize, pos: usize,
k: usize, k: usize,
next_step: Option<(WalkState, u8)>, next_step: Option<(WalkState<'a>, u8)>,
} }
impl<'a> UnitigNucIter<'a> { impl<'a> UnitigNucIter<'a> {
@@ -20,11 +22,13 @@ impl<'a> UnitigNucIter<'a> {
graph: &'a GraphDeBruijn, graph: &'a GraphDeBruijn,
start: CanonicalKmer, start: CanonicalKmer,
k: usize, k: usize,
next_step: Option<(WalkState, u8)>, direct: bool,
next_step: Option<(WalkState<'a>, u8)>,
) -> Self { ) -> Self {
Self { Self {
graph, graph,
start, start,
direct,
pos: 0, pos: 0,
k, k,
next_step, next_step,
@@ -37,16 +41,16 @@ impl Iterator for UnitigNucIter<'_> {
fn next(&mut self) -> Option<u8> { fn next(&mut self) -> Option<u8> {
if self.pos < self.k { if self.pos < self.k {
let nuc = self.start.nucleotide(self.pos); let nuc = if self.direct {
self.start.nucleotide(self.pos)
} else {
3 - self.start.nucleotide(self.k - 1 - self.pos)
};
self.pos += 1; self.pos += 1;
Some(nuc) Some(nuc)
} else if let Some((state, nuc)) = self.next_step.take() { } else if let Some((state, nuc)) = self.next_step.take() {
self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| { self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| {
let old = self let old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
.graph
.nodes
.get(&next_state.kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
(old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc)) (old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc))
}); });
Some(nuc) Some(nuc)
+22 -12
View File
@@ -1,35 +1,41 @@
use obikseq::{CanonicalKmer, Sequence}; use obikseq::{CanonicalKmer, Sequence};
use std::sync::atomic::Ordering; use std::sync::atomic::{AtomicU8, Ordering};
use super::graph::GraphDeBruijn; use super::graph::GraphDeBruijn;
use super::node::Node; use super::node::Node;
pub struct WalkState { /// A position in a walk: the node's kmer, its flags as read when the step was
/// taken, and the node's own table entry. Keeping `atomic` means the caller
/// that wants to mark the node visited does not look the kmer up a second
/// time (a random access in a table far larger than the cache).
pub struct WalkState<'g> {
pub(super) kmer: CanonicalKmer, pub(super) kmer: CanonicalKmer,
pub(super) node: Node, pub(super) node: Node,
pub(super) direct: bool, pub(super) direct: bool,
pub(super) atomic: &'g AtomicU8,
} }
impl WalkState { impl<'g> WalkState<'g> {
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool) -> Self { pub fn new(kmer: CanonicalKmer, node: Node, direct: bool, atomic: &'g AtomicU8) -> Self {
debug_assert!(!node.is_visited(), "Cannot walk over a visited node"); debug_assert!(!node.is_visited(), "Cannot walk over a visited node");
Self { kmer, node, direct } Self { kmer, node, direct, atomic }
} }
pub fn leavable(&self, graph: &GraphDeBruijn) -> bool { pub fn leavable(&self, graph: &'g GraphDeBruijn) -> bool {
self.walk(graph).is_some() self.walk(graph).is_some()
} }
pub fn reachable(&self, graph: &GraphDeBruijn) -> bool { pub fn reachable(&self, graph: &'g GraphDeBruijn) -> bool {
WalkState { WalkState {
kmer: self.kmer, kmer: self.kmer,
node: self.node, node: self.node,
direct: !self.direct, direct: !self.direct,
atomic: self.atomic,
} }
.leavable(graph) .leavable(graph)
} }
pub fn walk(&self, graph: &GraphDeBruijn) -> Option<(WalkState, u8)> { pub fn walk(&self, graph: &'g GraphDeBruijn) -> Option<(WalkState<'g>, u8)> {
if self.direct { if self.direct {
if !self.node.can_extend_right() { if !self.node.can_extend_right() {
return None; return None;
@@ -38,7 +44,8 @@ impl WalkState {
let next = self.kmer.into_kmer().push_right(nuc); let next = self.kmer.into_kmer().push_right(nuc);
let cnext = next.canonical(); let cnext = next.canonical();
let dnext = next.raw() == cnext.raw(); let dnext = next.raw() == cnext.raw();
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed)); let next_atomic = graph.nodes.get(&cnext).unwrap();
let next_node = Node(next_atomic.load(Ordering::Relaxed));
if next_node.is_visited() { if next_node.is_visited() {
return None; return None;
} }
@@ -52,6 +59,7 @@ impl WalkState {
kmer: cnext, kmer: cnext,
node: next_node, node: next_node,
direct: dnext, direct: dnext,
atomic: next_atomic,
}, },
nuc, nuc,
)) ))
@@ -63,20 +71,22 @@ impl WalkState {
let next = self.kmer.into_kmer().push_left(nuc); let next = self.kmer.into_kmer().push_left(nuc);
let cnext = next.canonical(); let cnext = next.canonical();
let dnext = next.raw() != cnext.raw(); let dnext = next.raw() != cnext.raw();
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed)); let next_atomic = graph.nodes.get(&cnext).unwrap();
let next_node = Node(next_atomic.load(Ordering::Relaxed));
if next_node.is_visited() { if next_node.is_visited() {
return None; return None;
} }
let reachable = if dnext { let reachable = if dnext {
next_node.can_extend_right()
} else {
next_node.can_extend_left() next_node.can_extend_left()
} else {
next_node.can_extend_right()
}; };
reachable.then_some(( reachable.then_some((
WalkState { WalkState {
kmer: cnext, kmer: cnext,
node: next_node, node: next_node,
direct: dnext, direct: dnext,
atomic: next_atomic,
}, },
3 - nuc, 3 - nuc,
)) ))
+83
View File
@@ -317,3 +317,86 @@ fn branching_graph_no_kmer_lost_or_duplicated() {
); );
assert_eq!(got, expected, "k-mer sets differ"); assert_eq!(got, expected, "k-mer sets differ");
} }
// ── orientation independence ──────────────────────────────────────────────
// Deterministic pseudo-random ASCII sequence (xorshift), so the tests need no
// RNG dependency.
fn pseudo_random_ascii(seed: u64, len: usize) -> Vec<u8> {
let mut st = seed.wrapping_mul(0x9E3779B97F4A7C15) | 1;
(0..len)
.map(|_| {
st ^= st << 13;
st ^= st >> 7;
st ^= st << 17;
b"ACGT"[(st % 4) as usize]
})
.collect()
}
#[test]
fn linear_chain_is_one_unitig_whatever_the_orientations() {
// A repeat-free sequence is one maximal unitig. Each k-mer is stored in
// its canonical orientation, so across many sequences the chain's nodes
// take every mix of orientations — including chains whose two ends are
// both stored reverse-complemented, which once came out as several
// unitigs (a lone-kmer unitig per mis-oriented chain start) depending
// only on the table's iteration order.
let k = 21; // long enough that (k-1)-mer collisions (= branches) do not occur
let _guard = lock_k();
set_k(k);
for seed in 1..=300u64 {
let seq = pseudo_random_ascii(seed, 90);
let kmers = canonical_kmers(&seq);
if kmers.len() != seq.len() - k + 1 {
continue; // accidental internal repeat: not a single chain
}
let g = graph_from_ascii(&seq);
g.compute_degrees_and_mark_starts();
let unitigs = collect_unitigs(&g);
assert_eq!(unitigs.len(), 1, "seed {seed}: repeat-free sequence must give one unitig");
assert_eq!(unitigs[0].seql(), seq.len(), "seed {seed}: unitig spans the sequence");
assert_eq!(kmers_from_unitigs(&unitigs), kmers, "seed {seed}: same k-mers");
}
}
#[test]
fn unitigs_partition_the_kmers_of_branching_reads() {
// Reads of a small genome on both strands with substitutions: branches,
// bubbles and tips. Whatever the shape, every k-mer must be covered
// exactly once: sum over unitigs of (seql - k + 1) == number of nodes.
let k = 11;
let _guard = lock_k();
set_k(k);
let genome = pseudo_random_ascii(7, 600);
let mut g = GraphDeBruijn::new();
for r in 0..120u64 {
let start = (pseudo_random_ascii(100 + r, 1)[0] as usize * 31 + r as usize * 17) % 500;
let mut read = genome[start..start + 90].to_vec();
if r % 2 == 1 {
// second strand
read = read
.iter()
.rev()
.map(|&b| match b {
b'A' => b'T',
b'C' => b'G',
b'G' => b'C',
_ => b'A',
})
.collect();
}
if r % 3 == 0 {
let at = (r as usize * 13) % read.len();
read[at] = if read[at] == b'A' { b'C' } else { b'A' };
}
for i in 0..=read.len() - k {
g.push(Kmer::from_ascii(&read[i..i + k]).unwrap().canonical());
}
}
g.compute_degrees_and_mark_starts();
let unitigs = collect_unitigs(&g);
let covered: usize = unitigs.iter().map(|u| u.seql() - k + 1).sum();
assert_eq!(covered, g.len(), "every k-mer in exactly one unitig");
assert_eq!(kmers_from_unitigs(&unitigs).len(), g.len(), "no k-mer missing");
}
-19
View File
@@ -188,11 +188,6 @@ impl IndexCache {
self.iter().flat_map(|l| l.iter_unitigs()) self.iter().flat_map(|l| l.iter_unitigs())
} }
#[inline]
pub fn hash(&self, partition: usize, layer: usize, kmer: CanonicalKmer) -> Option<usize> {
Some(self.get_layer(partition, layer)?.hash(kmer))
}
pub fn n_kmer(&self, partition: usize, layer: usize) -> Option<usize> { pub fn n_kmer(&self, partition: usize, layer: usize) -> Option<usize> {
Some(self.get_layer(partition, layer)?.n()) Some(self.get_layer(partition, layer)?.n())
} }
@@ -228,20 +223,6 @@ impl IndexCache {
None None
} }
/// Membership-checked lookup — routes `kmer` straight to its one
/// possible partition (`CanonicalKmer::partition`, the same
/// minimizer-hash rule every k-mer was scattered by when the index was
/// built) instead of scanning every cached partition: a k-mer can only
/// ever live in the partition it hashes to, so anywhere else is not
/// worth even asking. Uses `self.index().n_partitions()` — the index's
/// real total, not [`n_partition`](Self::n_partition) (this cache's own
/// count, which can be a subset — see [`new`](Self::new)). `None` if
/// that partition isn't cached or doesn't carry `kmer`.
pub fn find(&self, kmer: CanonicalKmer) -> Option<(usize, usize, usize)> {
let partition = kmer.partition(self.raw_index.n_partitions());
self.find_in_partition(partition, kmer)
}
/// Parallel map-reduce over every cached `KmerLayer` (across every /// Parallel map-reduce over every cached `KmerLayer` (across every
/// cached partition), bounded to `max_concurrent` layers scanned at /// cached partition), bounded to `max_concurrent` layers scanned at
/// once — a cap on how many layers are actively being mapped /// once — a cap on how many layers are actively being mapped
+4 -2
View File
@@ -90,7 +90,7 @@ impl IndexBuilder for KmerIndex {
let idx = KmerIndex { let idx = KmerIndex {
root_path: output.to_owned(), root_path: output.to_owned(),
meta: Arc::new(meta), meta: Arc::new(meta),
lock: std::sync::Mutex::new(None), lock: obisys::LockSlot::empty(output),
}; };
if !idx.acquire_lock() { if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other( return Err(OKIError::Io(std::io::Error::other(
@@ -102,7 +102,9 @@ impl IndexBuilder for KmerIndex {
fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> { fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> {
let output = output.as_ref(); let output = output.as_ref();
let idx = KmerIndex::open(output)?; // `output` is held exclusively by the builder that wrote it: reopen
// without a lock, a locking `open` would wait for that builder for ever.
let idx = KmerIndex::open_unlocked(output)?;
idx.meta.mark_indexed().map_err(OKIError::Io)?; idx.meta.mark_indexed().map_err(OKIError::Io)?;
let t_pack = Stage::start("pack"); let t_pack = Stage::start("pack");
idx.pack_matrices(sparse)?; idx.pack_matrices(sparse)?;
+84 -54
View File
@@ -1,6 +1,6 @@
use std::fs; use std::fs;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex}; use std::sync::Arc;
use crate::layer::KmerLayer; use crate::layer::KmerLayer;
use crate::partition::KmerPartition; use crate::partition::KmerPartition;
@@ -15,7 +15,7 @@ use crate::index::meta::{GenomeInfo, IndexConfig, IndexMeta};
pub struct KmerIndex { pub struct KmerIndex {
pub(crate) root_path: PathBuf, pub(crate) root_path: PathBuf,
pub(crate) meta: Arc<IndexMeta>, pub(crate) meta: Arc<IndexMeta>,
pub(crate) lock: Mutex<Option<obisys::DirLock>>, pub(crate) lock: obisys::LockSlot,
} }
impl KmerIndex { impl KmerIndex {
@@ -37,11 +37,8 @@ impl KmerIndex {
set_m(config.minimizer_size); set_m(config.minimizer_size);
let genomes = genome_info.into_iter().collect(); let genomes = genome_info.into_iter().collect();
let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?; let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?;
let idx = Self { let lock = obisys::LockSlot::empty(&root_path);
root_path, let idx = Self { root_path, meta: Arc::new(meta), lock };
meta: Arc::new(meta),
lock: Mutex::new(None),
};
if !idx.acquire_lock() { if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other( return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock right after create", "failed to acquire index lock right after create",
@@ -50,67 +47,92 @@ impl KmerIndex {
Ok(idx) Ok(idx)
} }
/// Open an existing index for reading — doesn't touch the directory /// Open an existing index for reading, holding a **shared** directory
/// lock at all: concurrent readers (`query`, `dump`, a `merge` source, /// lock for as long as the returned index lives: any number of readers
/// ...) must never block on each other. A caller that needs exclusive /// coexist, but a writer ([`open_lock`](Self::open_lock),
/// access (resuming an interrupted build) uses /// [`with_lock`](Self::with_lock)) waits for them all, and readers wait
/// [`open_lock`](Self::open_lock) instead; either way, `self.lock` /// for a writer at work — a reader never sees files change under it.
/// starts empty — [`test_lock`](Self::test_lock)/ /// The lock is taken before the metadata is read.
/// [`acquire_lock`](Self::acquire_lock) are opt-in from here. ///
/// Never open, from a process that holds the write lock on a directory
/// through another handle, that same directory again with `open`: the
/// new handle would wait for the first for ever.
pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> { pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let root_path = path.as_ref().to_owned(); Self::open_with(path.as_ref(), obisys::LockMode::Shared)
let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
set_k(meta.config.kmer_size);
set_m(meta.config.minimizer_size);
Ok(Self {
root_path,
meta: Arc::new(meta),
lock: Mutex::new(None),
})
} }
/// `open`, then hold the directory lock exclusively — for a caller /// `open`, holding the directory lock **exclusively** for as long as the
/// that resumes a write pipeline (e.g. `obikmer index` re-run on an /// returned index lives — **the** way for a command that modifies an
/// existing, not-yet-`Indexed` directory) rather than reading /// existing index in place (`convert`, `pack`, `annotate`, `utils
/// concurrently with others. /// rename`, resuming an interrupted build, ...) to claim it. The lock is
/// taken *before* the metadata is read, so what the caller reads is what
/// it will modify.
///
/// A command that only reads, or that writes a *new* index elsewhere
/// (`merge`, `filter`, `select`: exclusive on their destination, shared
/// on their sources), uses [`open`](Self::open). Library code that has to
/// write into an index it was handed (annexes built on demand) wraps that
/// write in [`with_lock`](Self::with_lock).
pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> { pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let idx = Self::open(path)?; Self::open_with(path.as_ref(), obisys::LockMode::Exclusive)
if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock",
)));
} }
fn open_with(path: &Path, mode: obisys::LockMode) -> OKIResult<Self> {
// A directory that is not an index is reported by `open_unlocked`,
// without a lock file being created in it.
if !Self::is_an_index(path) {
return Self::open_unlocked(path);
}
let lock = match mode {
obisys::LockMode::Shared => obisys::LockSlot::shared(path),
obisys::LockMode::Exclusive => obisys::LockSlot::exclusive(path),
}
.map_err(OKIError::Io)?;
let mut idx = Self::open_unlocked(path)?;
idx.lock = lock;
Ok(idx) Ok(idx)
} }
/// Block until this index's directory lock is acquired, then hold it /// `open` without taking any lock — for a caller that already holds the
/// (released on [`release_lock`](Self::release_lock) or when `self` is /// directory through another handle (`finalize_indexed`, reopening the
/// dropped). Returns `false` only on a genuine I/O failure acquiring /// index its own builder holds exclusively).
/// the lock file itself — contention blocks rather than failing. pub(crate) fn open_unlocked(path: &Path) -> OKIResult<Self> {
pub fn acquire_lock(&self) -> bool { let root_path = path.to_owned();
match obisys::DirLock::acquire(&self.root_path) { let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
Ok(lock) => { set_k(meta.config.kmer_size);
*self.lock.lock().unwrap() = Some(lock); set_m(meta.config.minimizer_size);
true let lock = obisys::LockSlot::empty(&root_path);
Ok(Self { root_path, meta: Arc::new(meta), lock })
} }
Err(_) => false,
/// Take the exclusive directory lock and hold it until `self` is dropped
/// — for an index just created, which is about to be written to.
/// Idempotent. Returns `false` only on a genuine I/O failure.
pub(crate) fn acquire_lock(&self) -> bool {
self.lock.acquire_exclusive().is_ok()
} }
/// Whether this index object currently holds its directory lock
/// exclusively (a shared lock is not "write access").
pub fn holds_write_lock(&self) -> bool {
self.lock.holds_write_lock()
}
/// Run `f` with the directory lock held **exclusively** — promoting the
/// shared lock a handle from [`open`](Self::open) holds, and restoring it
/// afterwards; see [`obisys::LockSlot::with_exclusive`], including what
/// `f` must re-check. Library code that writes into an index it was
/// handed (annexes built on demand) wraps that write in this.
pub fn with_lock<T>(&self, f: impl FnOnce() -> OKIResult<T>) -> OKIResult<T> {
self.lock.with_exclusive(f)
} }
/// Non-blocking probe of this index's directory lock — doesn't retain /// Non-blocking probe of this index's directory lock — doesn't retain
/// it either way: `true` if the lock was free (briefly acquired, then /// anything: `true` if no process currently holds it exclusively, `false`
/// immediately released), `false` if another process currently holds /// if one does (this process included). For a caller that wants to warn
/// it. For a read-only caller that wants to warn about a build in /// about a write in progress without waiting for it.
/// progress without itself becoming exclusive; use
/// [`acquire_lock`](Self::acquire_lock)/[`open_lock`](Self::open_lock)
/// to actually hold it.
pub fn test_lock(&self) -> bool { pub fn test_lock(&self) -> bool {
matches!(obisys::DirLock::try_acquire(&self.root_path), Ok(Some(_))) self.lock.is_free_of_writer()
}
/// Release this index's directory lock, if held. A no-op if it isn't.
pub fn release_lock(&self) {
*self.lock.lock().unwrap() = None;
} }
/// Return `true` if `path` points to a valid obikmer index directory. /// Return `true` if `path` points to a valid obikmer index directory.
@@ -210,6 +232,14 @@ impl KmerIndex {
crate::partition::index_dir(&self.root_path, i) crate::partition::index_dir(&self.root_path, i)
} }
/// Path of layer `j` of partition `i` — pure path computation, nothing
/// is opened (unlike [`layer`](Self::layer)/`layer_dir`, which open the
/// layer's MPHF). For maintenance tools that must reach a layer's files
/// without going through a format they can't read yet.
pub fn layer_path(&self, i: usize, j: usize) -> PathBuf {
crate::layer::utils::layer_dir(&self.index_dir(i), j)
}
/// Number of layers in partition `i` — a disk scan, not a metadata read /// Number of layers in partition `i` — a disk scan, not a metadata read
/// (see [`KmerPartition::n_layers`](crate::partition::KmerPartition::n_layers)). /// (see [`KmerPartition::n_layers`](crate::partition::KmerPartition::n_layers)).
pub fn n_layers(&self, i: usize) -> OKIResult<usize> { pub fn n_layers(&self, i: usize) -> OKIResult<usize> {
+4 -2
View File
@@ -280,8 +280,10 @@ impl IndexMeta {
// ── private ────────────────────────────────────────────────────────────── // ── private ──────────────────────────────────────────────────────────────
fn read_full(root: &Path) -> io::Result<IndexMetadata> { fn read_full(root: &Path) -> io::Result<IndexMetadata> {
let file = fs::File::open(root.join(META_FILENAME))?; // One `read`, then parse from memory: `from_reader` on a bare `File`
serde_json::from_reader(file).map_err(io::Error::other) // issues one syscall per byte (≈ 0.35 s for a 91-genome `index.meta`).
let bytes = fs::read(root.join(META_FILENAME))?;
serde_json::from_slice(&bytes).map_err(io::Error::other)
} }
fn write_full(&self, on_disk: &IndexMetadata) -> io::Result<()> { fn write_full(&self, on_disk: &IndexMetadata) -> io::Result<()> {
+11 -9
View File
@@ -199,6 +199,17 @@ impl KmerLayer {
} }
} }
/// Batch form of [`find_slot`](Self::find_slot): `out[i]` is
/// `find_slot(kmers[i])`, MPHF lookups prefetched together. `out` is
/// cleared and reused.
pub fn find_slot_batch(&self, kmers: &[CanonicalKmer], out: &mut Vec<Option<usize>>) {
match self {
KmerLayer::Count { layer, .. } => layer.find_slot_batch(kmers, out),
KmerLayer::Presence { layer, .. } => layer.find_slot_batch(kmers, out),
KmerLayer::Empty { .. } => panic!("Layer::find_slot_batch() called on an Empty layer"),
}
}
/// Raw MPHF batch lookup: kmer → slot, no membership check — for /// Raw MPHF batch lookup: kmer → slot, no membership check — for
/// callers that already know every kmer is a member of *this* layer, so /// callers that already know every kmer is a member of *this* layer, so
/// the evidence check `find_slot`/`find` would perform is redundant. /// the evidence check `find_slot`/`find` would perform is redundant.
@@ -286,15 +297,6 @@ impl KmerLayer {
} }
} }
/// Raw MPHF lookup: kmer → slot, no membership check.
pub fn hash(&self, kmer: CanonicalKmer) -> usize {
match self {
KmerLayer::Count { layer, .. } => layer.hash(kmer),
KmerLayer::Presence { layer, .. } => layer.hash(kmer),
KmerLayer::Empty { .. } => panic!("Layer::hash() called on an Empty layer"),
}
}
/// Iterate over all canonical kmers in the layer, in deterministic order. /// Iterate over all canonical kmers in the layer, in deterministic order.
pub fn iter_kmers(&self) -> crate::layer::mphf_layer::KmerIter { pub fn iter_kmers(&self) -> crate::layer::mphf_layer::KmerIter {
match self { match self {
+1 -1
View File
@@ -27,7 +27,7 @@ pub enum IndexMode {
/// [`detect`](Self::detect)). /// [`detect`](Self::detect)).
Approx { b: u8, z: u8 }, Approx { b: u8, z: u8 },
/// Hybrid: both `fingerprint.bin` and `evidence.bin` + `unitigs.bin.idx`. /// Hybrid: both `fingerprint.bin` and `evidence.bin` + `unitigs.bin.idx`.
/// `find()` uses the fingerprint (O(1), approx); `find_strict()` uses exact evidence. /// `find()` uses the fingerprint (O(1), approx); the exact evidence is kept alongside.
Hybrid { b: u8, z: u8 }, Hybrid { b: u8, z: u8 },
} }
+4 -1
View File
@@ -3,11 +3,14 @@ pub mod evidence;
pub mod fingerprint; pub mod fingerprint;
pub mod meta; pub mod meta;
pub(crate) mod mphf_layer; pub(crate) mod mphf_layer;
pub(crate) mod slotted;
pub mod typed_layer; pub mod typed_layer;
pub(crate) mod utils; pub(crate) mod utils;
pub use crate::index::error::{OKIError, OKIResult}; pub use crate::index::error::{OKIError, OKIResult};
pub use content_layer::KmerLayer; pub use content_layer::KmerLayer;
pub use meta::IndexMode; pub use meta::IndexMode;
pub use mphf_layer::{EvidenceKind, KmerBatchIter, KmerIter, MphfLayer, MphfOnly, UnitigIter}; pub use mphf_layer::{
EvidenceKind, KmerBatchIter, KmerIter, Mphf, MphfLayer, MphfOnly, UnitigIter, mphf_params,
};
pub use typed_layer::{HasLayerContent, HasStorageKind, Hit, LayerContent, LayerData, TypedLayer}; pub use typed_layer::{HasLayerContent, HasStorageKind, Hit, LayerContent, LayerData, TypedLayer};
+76 -89
View File
@@ -14,6 +14,7 @@ use crate::index::error::{OKIError, OKIResult};
use crate::layer::evidence::{Evidence, EvidenceWriter}; use crate::layer::evidence::{Evidence, EvidenceWriter};
use crate::layer::fingerprint::{FingerprintVec, FingerprintVecWriter}; use crate::layer::fingerprint::{FingerprintVec, FingerprintVecWriter};
use crate::layer::meta::IndexMode; use crate::layer::meta::IndexMode;
use crate::layer::slotted::SlottedKmers;
pub(crate) const MPHF_FILE: &str = "mphf.bin"; pub(crate) const MPHF_FILE: &str = "mphf.bin";
pub(crate) const UNITIGS_FILE: &str = "unitigs.bin"; pub(crate) const UNITIGS_FILE: &str = "unitigs.bin";
@@ -23,7 +24,13 @@ pub(crate) const FINGERPRINT_FILE: &str = "fingerprint.bin";
/// The layer MPHF: `Xxh3Int` hasher, `CubicEps` buckets, multi-part (`SINGLE_PART = false`), /// The layer MPHF: `Xxh3Int` hasher, `CubicEps` buckets, multi-part (`SINGLE_PART = false`),
/// minimal (`REMAP = true`). Built owned (construction + store); queried through /// minimal (`REMAP = true`). Built owned (construction + store); queried through
/// `MemCase<Mphf>::uncase()`, the ε-copy view into the memory-mapped file. /// `MemCase<Mphf>::uncase()`, the ε-copy view into the memory-mapped file.
pub(crate) type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xxh3Int, Vec<u8>, false, true>; pub type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xxh3Int, Vec<u8>, false, true>;
/// Construction parameters of [`Mphf`] — the one place they are chosen
/// (`lambda = 3.5`, `alpha = 0.99`, `CubicEps`, ~2.4 bits/key).
pub fn mphf_params() -> PtrHashParams<CubicEps> {
PtrHashParams::<CubicEps>::default_balanced()
}
// ── LayerEvidence ───────────────────────────────────────────────────────────── // ── LayerEvidence ─────────────────────────────────────────────────────────────
@@ -44,17 +51,17 @@ enum LayerEvidence {
Hybrid { Hybrid {
evidence_path: PathBuf, evidence_path: PathBuf,
fingerprint_path: PathBuf, fingerprint_path: PathBuf,
unitig_path: PathBuf,
}, },
} }
/// Resources actually opened (mmap). `Exact`/`Hybrid` carry their own /// Resources actually opened (mmap). `Exact` carries its own `unitigs` —
/// `unitigs` — needed to complete evidence verification, not shared with /// needed to complete evidence verification, not shared with `MphfLayer`'s
/// `MphfLayer`'s own separate `unitigs` (used by `iter_kmers`/`Approx`'s /// own separate `unitigs` (used by `iter_kmers`, mode-independent). Yes, this
/// `find_strict` fallback, mode-independent). Yes, this means a layer's /// means an Exact layer's `unitigs.bin` gets mmapped twice (the `.idx`
/// `unitigs.bin` gets mmapped twice for Exact/Hybrid — deliberately not /// offsets are mapped too, so both readers share the same pages) —
/// optimised away here: avoiding that redundant open is the future cache /// deliberately not optimised away here. `Hybrid` keeps its `evidence.bin`
/// crate's job, not this base type's. /// open (it gives the slot count) but nothing verifies against it: lookups
/// use the fingerprint, as in `Approx`.
enum OpenLayerEvidence { enum OpenLayerEvidence {
Exact { Exact {
evidence: Evidence, evidence: Evidence,
@@ -66,7 +73,6 @@ enum OpenLayerEvidence {
Hybrid { Hybrid {
evidence: Evidence, evidence: Evidence,
fingerprint: FingerprintVec, fingerprint: FingerprintVec,
unitigs: Arc<UnitigFileReader>,
}, },
} }
@@ -86,7 +92,6 @@ impl LayerEvidence {
EvidenceKind::Hybrid => LayerEvidence::Hybrid { EvidenceKind::Hybrid => LayerEvidence::Hybrid {
evidence_path: layer_dir.join(EVIDENCE_FILE), evidence_path: layer_dir.join(EVIDENCE_FILE),
fingerprint_path: layer_dir.join(FINGERPRINT_FILE), fingerprint_path: layer_dir.join(FINGERPRINT_FILE),
unitig_path: layer_dir.join(UNITIGS_FILE),
}, },
}) })
} }
@@ -101,10 +106,9 @@ impl LayerEvidence {
LayerEvidence::Approx { fingerprint_path } => OpenLayerEvidence::Approx { LayerEvidence::Approx { fingerprint_path } => OpenLayerEvidence::Approx {
fingerprint: FingerprintVec::open(&fingerprint_path)?, fingerprint: FingerprintVec::open(&fingerprint_path)?,
}, },
LayerEvidence::Hybrid { evidence_path, fingerprint_path, unitig_path } => OpenLayerEvidence::Hybrid { LayerEvidence::Hybrid { evidence_path, fingerprint_path } => OpenLayerEvidence::Hybrid {
evidence: Evidence::open(&evidence_path)?, evidence: Evidence::open(&evidence_path)?,
fingerprint: FingerprintVec::open(&fingerprint_path)?, fingerprint: FingerprintVec::open(&fingerprint_path)?,
unitigs: Arc::new(UnitigFileReader::open(&unitig_path)?),
}, },
}) })
} }
@@ -144,19 +148,15 @@ impl EvidenceKind {
/// Autonomous kmer → slot mapping for one layer. /// Autonomous kmer → slot mapping for one layer.
/// ///
/// Two query methods: /// Query: [`find`](Self::find) — O(1), uses the fingerprint (Approx/Hybrid) or
/// - [`find`](Self::find) — O(1), uses fingerprint (Approx/Hybrid) or exact evidence (Exact). /// exact evidence (Exact).
/// - [`find_strict`](Self::find_strict) — always exact; O(1) on Exact/Hybrid layers,
/// O(n) sequential scan on Approx layers.
pub struct MphfLayer { pub struct MphfLayer {
/// Mode-independent copy — used by `iter_kmers` (all modes) and /// Mode-independent copy — used by `iter_kmers` (all modes). `Exact`
/// `find_strict`'s `Approx` fallback. `Exact`/`Hybrid` additionally /// additionally carries its own inside `OpenLayerEvidence` (see its doc
/// carry their own inside `OpenLayerEvidence` (see its doc comment for /// comment for why that's a second mmap of the same file, not a bug).
/// why that's a second, deliberately unshared mmap of the same file,
/// not a bug).
unitigs: Arc<UnitigFileReader>, unitigs: Arc<UnitigFileReader>,
mphf: MemCase<Mphf>, mphf: MphfOnly,
ev: OpenLayerEvidence, ev: OpenLayerEvidence,
n: usize, n: usize,
} }
@@ -169,8 +169,7 @@ impl MphfLayer {
/// not stored: `KmerLayer`, one level up, is what owns a layer's /// not stored: `KmerLayer`, one level up, is what owns a layer's
/// directory as a persistent field, not this type. /// directory as a persistent field, not this type.
pub fn open(dir: &Path) -> OKIResult<Self> { pub fn open(dir: &Path) -> OKIResult<Self> {
let mphf: MemCase<Mphf> = unsafe { Mphf::mmap(&dir.join(MPHF_FILE), Flags::empty()) } let mphf = MphfOnly::open(dir)?;
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
let unitigs = Arc::new(UnitigFileReader::open(&dir.join(UNITIGS_FILE))?); let unitigs = Arc::new(UnitigFileReader::open(&dir.join(UNITIGS_FILE))?);
let ev = LayerEvidence::at(dir)?.open()?; let ev = LayerEvidence::at(dir)?.open()?;
let n = match &ev { let n = match &ev {
@@ -186,10 +185,17 @@ impl MphfLayer {
/// O(1) lookup — dispatches automatically: /// O(1) lookup — dispatches automatically:
/// - Exact: evidence + `verify_canonical_kmer`, zero false positives. /// - Exact: evidence + `verify_canonical_kmer`, zero false positives.
/// - Approx: fingerprint check, false-positive rate ≈ 1/2^b. /// - Approx: fingerprint check, false-positive rate ≈ 1/2^b.
/// - Hybrid: fingerprint check (fast path), zero false positives via `find_strict`. /// - Hybrid: fingerprint check, same as Approx (the exact evidence is kept on disk, not consulted).
#[inline] #[inline]
pub fn find(&self, kmer: CanonicalKmer) -> Option<usize> { pub fn find(&self, kmer: CanonicalKmer) -> Option<usize> {
let slot = self.mphf.uncase().index(&kmer.raw()); self.confirm(self.mphf.index(kmer), kmer)
}
/// Membership check of `kmer` against the slot the MPHF gave it — the
/// mode-dependent half of [`find`](Self::find), shared with
/// [`find_batch`](Self::find_batch).
#[inline]
fn confirm(&self, slot: usize, kmer: CanonicalKmer) -> Option<usize> {
if slot >= self.n { if slot >= self.n {
return None; return None;
} }
@@ -213,41 +219,20 @@ impl MphfLayer {
} }
} }
/// Always-exact lookup — zero false positives regardless of mode. /// Batch form of [`find`](Self::find): `out[i]` is `find(kmers[i])`.
/// ///
/// - Exact/Hybrid: O(1) via evidence + `verify_canonical_kmer`. /// The MPHF lookups of the whole slice go through `ptr_hash`'s prefetching
/// - Approx: O(n) sequential scan of `unitigs.bin` to confirm the kmer /// stream (see [`MphfOnly::for_each_slot`]), so their random pilot reads
/// that owns the slot, then exact comparison. /// overlap instead of being paid one after the other; the membership
pub fn find_strict(&self, kmer: CanonicalKmer) -> Option<usize> { /// check runs on each slot as it comes out. `out` is cleared and reused.
let slot = self.mphf.uncase().index(&kmer.raw()); pub fn find_batch(&self, kmers: &[CanonicalKmer], out: &mut Vec<Option<usize>>) {
if slot >= self.n { out.clear();
return None; out.reserve(kmers.len());
} let mut i = 0;
match &self.ev { self.mphf.for_each_slot(kmers, |slot| {
OpenLayerEvidence::Exact { evidence, unitigs } out.push(self.confirm(slot, kmers[i]));
| OpenLayerEvidence::Hybrid { evidence, unitigs, .. } => { i += 1;
let (chunk_id, rank) = evidence.decode(slot); });
if unitigs.verify_canonical_kmer(chunk_id as usize, rank as usize, kmer) {
Some(slot)
} else {
None
}
}
OpenLayerEvidence::Approx { .. } => {
// `self.unitigs` was opened via `UnitigFileReader::open`, which
// falls back to sequential mode automatically when no `.idx`
// sidecar exists — true for every Approx layer (`build_unitig_idx`
// is only ever called for Exact/Hybrid). Reusing it here avoids
// the second file-open + mmap + kmer-count scan a fresh
// `open_sequential` call would repeat on every call.
for (stored, _, _) in self.unitigs.iter_indexed_canonical_kmers() {
if self.mphf.uncase().index(&stored.raw()) == slot {
return if stored == kmer { Some(slot) } else { None };
}
}
None
}
}
} }
/// Number of slots in the MPHF layer — equal to the number of stored k-mers. /// Number of slots in the MPHF layer — equal to the number of stored k-mers.
@@ -265,23 +250,13 @@ impl MphfLayer {
} }
} }
/// Raw MPHF lookup: kmer → slot.
///
/// Returns the slot assigned by the MPHF without performing any membership
/// check against the layer's evidence. The returned slot is only meaningful
/// when the kmer is actually present in the layer; callers that need an
/// existence test should use [`find`](Self::find) instead.
pub fn hash(&self, kmer: CanonicalKmer) -> usize {
self.mphf.uncase().index(&kmer.raw())
}
/// Batch raw MPHF lookup: kmers → slots. /// Batch raw MPHF lookup: kmers → slots.
/// ///
/// Returns a [`Vec`] of slots, one per input kmer, in the same order as the /// Returns a [`Vec`] of slots, one per input kmer, in the same order as the
/// input slice. Like [`index`](Self::index), no membership check is /// input slice. Like [`hash`](Self::hash), no membership check is
/// performed. /// performed. Prefetching — see [`MphfOnly::index_batch`].
pub fn hash_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> { pub fn hash_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> {
kmers.iter().map(|k| self.mphf.uncase().index(&k.raw())).collect() self.mphf.index_batch(kmers)
} }
/// Iterate over all canonical kmers in the layer, in deterministic order. /// Iterate over all canonical kmers in the layer, in deterministic order.
@@ -439,10 +414,27 @@ impl MphfOnly {
self.0.uncase().index(&kmer.raw()) self.0.uncase().index(&kmer.raw())
} }
/// Batch form of [`index`](Self::index) — one MPHF lookup pass over the /// Batch form of [`index`](Self::index): one prefetching `ptr_hash`
/// whole slice instead of one call per kmer. /// stream over the whole slice, so the random pilot reads of different
/// kmers overlap instead of being paid one after the other.
pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> { pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize> {
kmers.iter().map(|&k| self.index(k)).collect() let mut slots = Vec::with_capacity(kmers.len());
self.for_each_slot(kmers, |slot| slots.push(slot));
slots
}
/// Calls `f(slot)` for each kmer of `kmers`, in order.
///
/// Callback-shaped on purpose: `ptr_hash`'s `index_stream` only
/// implements `fold` (its `next()` is `unimplemented!`), so it cannot be
/// returned as an ordinary iterator without a panic trap for callers.
/// Look-ahead of 32 keys, as recommended by `ptr_hash`.
#[inline]
pub fn for_each_slot(&self, kmers: &[CanonicalKmer], f: impl FnMut(usize)) {
self.0
.uncase()
.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
.for_each(f);
} }
} }
@@ -472,8 +464,7 @@ impl MphfLayer {
let mut ev = EvidenceWriter::new(n); let mut ev = EvidenceWriter::new(n);
let mut seen = vec![0u8; (n + 7) / 8]; let mut seen = vec![0u8; (n + 7) / 8];
for (kmer, chunk_id, rank) in unitigs.iter_indexed_canonical_kmers() { for (slot, _, chunk_id, rank) in SlottedKmers::new(&mphf, unitigs.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -515,8 +506,7 @@ impl MphfLayer {
let mut fw = FingerprintVecWriter::new(n, b); let mut fw = FingerprintVecWriter::new(n, b);
for (kmer, _, _) in unitigs.iter_indexed_canonical_kmers() { for (slot, kmer, _, _) in SlottedKmers::new(&mphf, unitigs.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -550,7 +540,7 @@ impl MphfLayer {
// ── Empty layer ─────────────────────────────────────────────────────── // ── Empty layer ───────────────────────────────────────────────────────
if n == 0 { if n == 0 {
let mphf: Mphf = Mphf::try_new(&[] as &[u64], PtrHashParams::<CubicEps>::default_balanced()) let mphf: Mphf = Mphf::try_new(&[] as &[u64], mphf_params())
.ok_or_else(|| OKIError::InvalidData { context: "mphf", detail: "construction failed".into() })?; .ok_or_else(|| OKIError::InvalidData { context: "mphf", detail: "construction failed".into() })?;
unsafe { mphf.store(&dir.join(MPHF_FILE)) } unsafe { mphf.store(&dir.join(MPHF_FILE)) }
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?; .map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
@@ -574,7 +564,7 @@ impl MphfLayer {
let mphf: Mphf = Mphf::new_from_par_iter( let mphf: Mphf = Mphf::new_from_par_iter(
n, n,
keys.map(|k| k.raw()).par_bridge(), keys.map(|k| k.raw()).par_bridge(),
PtrHashParams::<CubicEps>::default_balanced(), mphf_params(),
); );
unsafe { mphf.store(&dir.join(MPHF_FILE)) } unsafe { mphf.store(&dir.join(MPHF_FILE)) }
.map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?; .map_err(|e| OKIError::InvalidData { context: "layer", detail: e.to_string() })?;
@@ -586,8 +576,7 @@ impl MphfLayer {
match mode { match mode {
IndexMode::Exact => { IndexMode::Exact => {
let mut ev = EvidenceWriter::new(n); let mut ev = EvidenceWriter::new(n);
for (kmer, chunk_id, rank) in unitigs2.iter_indexed_canonical_kmers() { for (slot, kmer, chunk_id, rank) in SlottedKmers::new(&mphf, unitigs2.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -606,8 +595,7 @@ impl MphfLayer {
IndexMode::Approx { b, .. } => { IndexMode::Approx { b, .. } => {
let mut fw = FingerprintVecWriter::new(n, *b); let mut fw = FingerprintVecWriter::new(n, *b);
for (kmer, _, _) in unitigs2.iter_indexed_canonical_kmers() { for (slot, kmer, _, _) in SlottedKmers::new(&mphf, unitigs2.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
@@ -626,8 +614,7 @@ impl MphfLayer {
IndexMode::Hybrid { b, .. } => { IndexMode::Hybrid { b, .. } => {
let mut ev = EvidenceWriter::new(n); let mut ev = EvidenceWriter::new(n);
let mut fw = FingerprintVecWriter::new(n, *b); let mut fw = FingerprintVecWriter::new(n, *b);
for (kmer, chunk_id, rank) in unitigs2.iter_indexed_canonical_kmers() { for (slot, kmer, chunk_id, rank) in SlottedKmers::new(&mphf, unitigs2.iter_indexed_canonical_kmers()) {
let slot = mphf.index(&kmer.raw());
if slot >= n { if slot >= n {
return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() }); return Err(OKIError::InvalidData { context: "mphf", detail: "slot out of bounds".into() });
} }
+58
View File
@@ -0,0 +1,58 @@
//! A layer's kmers streamed together with their MPHF slot — the shape every
//! build pass needs (`slot` indexes evidence, fingerprint and bitmap writes).
use obikseq::CanonicalKmer;
use crate::layer::mphf_layer::Mphf;
/// Kmers per `ptr_hash` stream: large enough to amortise its ring-buffer
/// start-up, small enough to stay in L1/L2 and never hold the layer.
const CHUNK: usize = 4096;
/// `(slot, kmer, chunk_id, rank)` for every `(kmer, chunk_id, rank)` of the
/// wrapped iterator, in the same order. The MPHF lookups are done a chunk at
/// a time through `ptr_hash`'s prefetching stream, so their random pilot
/// reads overlap; memory stays bounded by one chunk.
pub(crate) struct SlottedKmers<'a, I> {
mphf: &'a Mphf,
kmers: I,
chunk: Vec<(CanonicalKmer, usize, usize)>,
slots: Vec<usize>,
pos: usize,
}
impl<'a, I: Iterator<Item = (CanonicalKmer, usize, usize)>> SlottedKmers<'a, I> {
pub(crate) fn new(mphf: &'a Mphf, kmers: I) -> Self {
Self {
mphf,
kmers,
chunk: Vec::with_capacity(CHUNK),
slots: Vec::with_capacity(CHUNK),
pos: 0,
}
}
}
impl<I: Iterator<Item = (CanonicalKmer, usize, usize)>> Iterator for SlottedKmers<'_, I> {
type Item = (usize, CanonicalKmer, usize, usize);
fn next(&mut self) -> Option<Self::Item> {
if self.pos == self.chunk.len() {
self.chunk.clear();
self.chunk.extend(self.kmers.by_ref().take(CHUNK));
if self.chunk.is_empty() {
return None;
}
self.slots.clear();
let slots = &mut self.slots;
self.mphf
.index_stream::<32, _>(self.chunk.iter().map(|(k, _, _)| k.raw()))
.for_each(|slot| slots.push(slot));
self.pos = 0;
}
let (kmer, chunk_id, rank) = self.chunk[self.pos];
let slot = self.slots[self.pos];
self.pos += 1;
Some((slot, kmer, chunk_id, rank))
}
}
+7 -5
View File
@@ -189,13 +189,15 @@ impl<D: LayerData> TypedLayer<D> {
self.kmerstore.find(kmer) self.kmerstore.find(kmer)
} }
pub fn n(&self) -> usize { /// Batch form of [`find_slot`](Self::find_slot): `out[i]` is
self.kmerstore.n() /// `find_slot(kmers[i])`, with the MPHF lookups prefetched together
/// (see `MphfLayer::find_batch`). `out` is cleared and reused.
pub fn find_slot_batch(&self, kmers: &[CanonicalKmer], out: &mut Vec<Option<usize>>) {
self.kmerstore.find_batch(kmers, out)
} }
/// Raw MPHF lookup: kmer → slot, no membership check. pub fn n(&self) -> usize {
pub fn hash(&self, kmer: CanonicalKmer) -> usize { self.kmerstore.n()
self.kmerstore.hash(kmer)
} }
/// Batch raw MPHF lookup: kmers → slots, no membership check. /// Batch raw MPHF lookup: kmers → slots, no membership check.
-2
View File
@@ -14,9 +14,7 @@ obicompactvec = { path = "../obicompactvec" }
obipipeline = { path = "../obipipeline" } obipipeline = { path = "../obipipeline" }
obiread = { path = "../obiread" } obiread = { path = "../obiread" }
obiskbuilder = { path = "../obiskbuilder" } obiskbuilder = { path = "../obiskbuilder" }
cacheline-ef = "2.0"
epserde = "0.13" epserde = "0.13"
ptr_hash = { version = "2.1", features = ["epserde"] }
niffler = "3.0.0" niffler = "3.0.0"
memmap2 = "0.9.10" memmap2 = "0.9.10"
rayon = "1" rayon = "1"
+165
View File
@@ -0,0 +1,165 @@
//! Throwaway benchmark of the de Bruijn graph on a real kmer set:
//! `graph_bench <unitigs.bin> <k> [repeat]` builds `GraphDeBruijn` from the
//! canonical kmers of an existing layer's unitigs (same call sequence as
//! `write_graph_as_unitigs`) and times each phase.
use std::path::PathBuf;
use std::time::Instant;
use obidebruinj::GraphDeBruijn;
use obikseq::Sequence;
use obiskio::CanonicalKmerIter;
use rayon::prelude::*;
use std::collections::HashSet;
fn main() {
let args: Vec<String> = std::env::args().collect();
let path = PathBuf::from(&args[1]);
let k: usize = args[2].parse().unwrap();
let repeat: usize = args.get(3).map_or(1, |s| s.parse().unwrap());
obikseq::set_k(k);
obikseq::set_m(11);
let kmers: Vec<_> = if let Ok(n) = std::env::var("SYNTH") {
// synthetic reads: random genome, reads on both strands, 1 % substitutions
let n: usize = n.parse().unwrap();
let mut st = 0x9E3779B97F4A7C15u64;
let mut rnd = move || {
st ^= st << 13;
st ^= st >> 7;
st ^= st << 17;
st
};
let genome: Vec<u8> = (0..n).map(|_| b"ACGT"[(rnd() % 4) as usize]).collect();
let mut set = std::collections::BTreeSet::new();
for _ in 0..(n / 50) {
let start = (rnd() as usize) % (n - 100);
let mut read: Vec<u8> = genome[start..start + 100].to_vec();
for b in read.iter_mut() {
if rnd() % 100 == 0 {
*b = b"ACGT"[(rnd() % 4) as usize];
}
}
for w in read.windows(k) {
set.insert(obikseq::Kmer::from_ascii(w).unwrap().canonical());
}
}
set.into_iter().collect()
} else {
CanonicalKmerIter::new(&path).unwrap().collect()
};
eprintln!("{} kmers", kmers.len());
// Independent count of maximal unitigs: nodes − mutually-unique links
// (cycles ignored), computed with a plain HashSet — no use of GraphDeBruijn.
if std::env::var("IDEAL").is_ok() {
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
(0..4u8)
.map(|x| a.push_right(x))
.filter(|b| set.contains(&b.canonical().raw()))
.collect()
};
let links: usize = kmers
.par_iter()
.map(|c| {
let mut n = 0usize;
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
let ob = outs(o);
if ob.len() == 1 {
let b = ob[0];
// in-neighbours of b = revcomp of out-neighbours of revcomp(b)
if outs(b.revcomp()).len() == 1 {
// count each undirected link once: only from the orientation
// whose canonical raw is the smaller end (ties: both seen once per orientation)
let (ca, cb) = (o.canonical().raw(), b.canonical().raw());
if ca < cb || (ca == cb && o.raw() <= b.raw()) {
n += 1;
} else if ca > cb {
// seen from the other end too: skip
}
}
}
}
n
})
.sum();
eprintln!("IDEAL: nodes={} links={} -> maximal unitigs ≈ {}", kmers.len(), links, kmers.len() - links);
}
if std::env::var("IDEALDUMP").is_ok() {
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
(0..4u8).map(|x| a.push_right(x)).filter(|b| set.contains(&b.canonical().raw())).collect()
};
// oriented kmer a has a mutual predecessor p iff ins(a)=={p} and outs(p)=={a}
let pred = |a: obikseq::Kmer| -> Option<obikseq::Kmer> {
let ins: Vec<obikseq::Kmer> = outs(a.revcomp()).into_iter().map(|b| b.revcomp()).collect();
if ins.len() == 1 && outs(ins[0]).len() == 1 { Some(ins[0]) } else { None }
};
let mut seen_chains = std::collections::BTreeSet::new();
for c in &kmers {
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
if pred(o).is_some() { continue; }
let mut text: Vec<u8> = (0..k).map(|i| b"ACGT"[o.nucleotide(i) as usize]).collect();
let mut cur = o;
loop {
let ob = outs(cur);
if ob.len() != 1 { break; }
let b = ob[0];
if pred(b).is_none() { break; }
text.push(b"ACGT"[b.nucleotide(k - 1) as usize]);
cur = b;
}
let s = String::from_utf8(text).unwrap();
let rc: String = s.bytes().rev().map(|c| match c { b'A' => 'T', b'C' => 'G', b'G' => 'C', _ => 'A' }).collect();
seen_chains.insert(if s <= rc { s } else { rc });
}
}
for u in &seen_chains { println!("IDEALUNITIG {} {}", u.len(), u); }
}
for r in 0..repeat {
let t = Instant::now();
let mut g = GraphDeBruijn::new();
for &kmer in &kmers {
g.push(kmer);
}
let t_push = t.elapsed();
let t = Instant::now();
g.compute_degrees_and_mark_starts();
let t_deg = t.elapsed();
let t = Instant::now();
let n_unitigs = std::sync::atomic::AtomicUsize::new(0);
let n_nuc = std::sync::atomic::AtomicUsize::new(0);
let dump = std::env::var("DUMP").is_ok();
let seqs = std::sync::Mutex::new(Vec::<String>::new());
g.for_each_unitig(|it| {
let mut n = 0usize;
let mut text = String::new();
for nuc in it {
n += 1;
if dump {
text.push(b"ACGT"[nuc as usize] as char);
}
}
if dump {
seqs.lock().unwrap().push(text);
}
n_unitigs.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
n_nuc.fetch_add(n, std::sync::atomic::Ordering::Relaxed);
});
let t_walk = t.elapsed();
if dump {
let mut v = seqs.lock().unwrap().clone();
v.sort();
for u in v { println!("UNITIG {} {}", u.len(), u); }
}
println!(
"run {r}: nodes={} push={:.2?} degrees={:.2?} unitigs={:.2?} -> {} unitigs, {} nucleotides",
g.len(),
t_push,
t_deg,
t_walk,
n_unitigs.load(std::sync::atomic::Ordering::Relaxed),
n_nuc.load(std::sync::atomic::Ordering::Relaxed)
);
}
}
@@ -3,18 +3,15 @@ use std::fs;
use std::io; use std::io;
use std::path::Path; use std::path::Path;
use cacheline_ef::{CachelineEf, CachelineEfVec};
use epserde::ser::Serialize as EpSerialize; use epserde::ser::Serialize as EpSerialize;
use memmap2::Mmap; use memmap2::Mmap;
use obicompactvec::PersistentCompactIntVecBuilder; use obicompactvec::PersistentCompactIntVecBuilder;
use obikindex::layer::{Mphf, mphf_params};
use obiskio::{SKFileReader, SKResult}; use obiskio::{SKFileReader, SKResult};
use ptr_hash::{PtrHash, PtrHashParams, bucket_fn::CubicEps, hash::Xxh3Int};
use tracing::debug; use tracing::debug;
use super::kmer_sort::sort_unique_kmers; use super::kmer_sort::sort_unique_kmers;
pub(super) type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xxh3Int, Vec<u8>, false, true>;
fn build_mphf(unique_path: &Path, f0: usize) -> io::Result<Mphf> { fn build_mphf(unique_path: &Path, f0: usize) -> io::Result<Mphf> {
let file = fs::File::open(unique_path)?; let file = fs::File::open(unique_path)?;
let mmap = unsafe { Mmap::map(&file)? }; let mmap = unsafe { Mmap::map(&file)? };
@@ -24,7 +21,7 @@ fn build_mphf(unique_path: &Path, f0: usize) -> io::Result<Mphf> {
// Sequential constructor: the outer par_iter over partitions already saturates // Sequential constructor: the outer par_iter over partitions already saturates
// the Rayon pool. new_from_par_iter would get no additional threads and adds // the Rayon pool. new_from_par_iter would get no additional threads and adds
// coordination overhead. try_new accesses the same mmap'd pages at zero extra cost. // coordination overhead. try_new accesses the same mmap'd pages at zero extra cost.
Mphf::try_new(kmers, PtrHashParams::<CubicEps>::default_balanced()) Mphf::try_new(kmers, mphf_params())
.ok_or_else(|| io::Error::other("ptr_hash construction failed")) .ok_or_else(|| io::Error::other("ptr_hash construction failed"))
} }
@@ -25,17 +25,13 @@ use std::io;
use std::path::Path; use std::path::Path;
use crate::graph_pipeline::{materialize_layer, write_graph_as_unitigs}; use crate::graph_pipeline::{materialize_layer, write_graph_as_unitigs};
use cacheline_ef::{CachelineEf, CachelineEfVec};
use epserde::prelude::*; use epserde::prelude::*;
use obicompactvec::{PersistentIntMatrix, PersistentCompactIntVec}; use obicompactvec::{PersistentIntMatrix, PersistentCompactIntVec};
use obidebruinj::GraphDeBruijn; use obidebruinj::GraphDeBruijn;
use obikindex::layer::IndexMode; use obikindex::layer::IndexMode;
use obikindex::layer::{KmerLayer, TypedLayer}; use obikindex::layer::{KmerLayer, Mphf, TypedLayer};
use obikindex::{KmerIndex, OKIError, OKIResult}; use obikindex::{KmerIndex, OKIError, OKIResult};
use obiskio::{SKError, SKFileMeta, SKFileReader}; use obiskio::{SKError, SKFileMeta, SKFileReader};
use ptr_hash::{PtrHash, bucket_fn::CubicEps, hash::Xxh3Int};
type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xxh3Int, Vec<u8>, false, true>;
pub(crate) trait PrivateBuilder { pub(crate) trait PrivateBuilder {
/// Mark scatter as complete (`IndexState::Scattered`). /// Mark scatter as complete (`IndexState::Scattered`).
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "obikmer" name = "obikmer"
version = "1.3.6" version = "1.4.1"
edition = "2024" edition = "2024"
[[bin]] [[bin]]
+5 -1
View File
@@ -93,7 +93,11 @@ fn run_annotate(args: &AnnotateArgs) {
} }
}; };
let idx = open_index(&args.index); // Rewrites the index metadata in place: exclusive for the whole command.
let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}");
std::process::exit(1);
});
let mut genomes = idx.meta().genomes().unwrap_or_else(|e| { let mut genomes = idx.meta().genomes().unwrap_or_else(|e| {
eprintln!("error reading index metadata: {e}"); eprintln!("error reading index metadata: {e}");
+2 -8
View File
@@ -70,14 +70,8 @@ struct ModeArgs {
} }
pub fn run(args: ConvertArgs) { pub fn run(args: ConvertArgs) {
// Modifies the index in place; acquired before opening so a concurrent // Modifies the index in place: exclusive for the whole command.
// writer can't slip in between the open and the convert below. let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = KmerIndex::open(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
}); });
+2 -8
View File
@@ -26,14 +26,8 @@ pub struct PackArgs {
} }
pub fn run(args: PackArgs) { pub fn run(args: PackArgs) {
// Modifies the index in place; acquired before opening so a concurrent // Modifies the index in place: exclusive for the whole command.
// writer can't slip in between the open and the pack below. let idx = Arc::new(KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = Arc::new(KmerIndex::open(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
})); }));
+3 -7
View File
@@ -78,13 +78,9 @@ pub fn run(args: PhyloArgs) {
// Meant to be (re)built routinely, ahead of any SNP-family distance // Meant to be (re)built routinely, ahead of any SNP-family distance
// computation that will later consume it. // computation that will later consume it.
if args.sibling_annex { if args.sibling_annex {
// Writes into the index directory — hold an exclusive lock for the // Writes into the index directory; `build_sibling_annex` holds the
// duration so a second, concurrent `--sibling-annex` run on the // index's exclusive lock for the duration (`KmerIndex::with_lock`),
// same index can't corrupt these writes (see obisys::DirLock). // so a concurrent run on the same index can't corrupt these writes.
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
info!("building sibling-count/minorant annex"); info!("building sibling-count/minorant annex");
let t = Stage::start("sibling_annex"); let t = Stage::start("sibling_annex");
cache.build_sibling_annex().unwrap_or_else(|e| { cache.build_sibling_annex().unwrap_or_else(|e| {
+24 -1
View File
@@ -192,17 +192,40 @@ pub fn run(args: QueryArgs) {
let overhead_multiplier = let overhead_multiplier =
BASE_OVERHEAD + n_genomes as u64 * BYTES_PER_KMER_PER_GENOME * detail_factor; BASE_OVERHEAD + n_genomes as u64 * BYTES_PER_KMER_PER_GENOME * detail_factor;
// Parallelism: a chunk is one worker's unit of work, so an input smaller
// than `n_workers` chunks leaves cores idle (two 48 MB files at the
// memory-derived 256 MiB cap = 2 chunks = 2 busy cores; measured 11.6 s
// against 2.5 s with 8 MiB chunks, see `DevDocMD/architecture/query.md`).
// When the input size can be estimated, aim for `CHUNKS_PER_WORKER` chunks
// per worker and never go below `MIN_CHUNK`. Stdin and URLs have no
// known size: the memory-derived size stands.
const CHUNKS_PER_WORKER: u64 = 2;
const MIN_CHUNK: u64 = 4 * 1024 * 1024;
let input_bytes: Option<u64> = args
.inputs
.iter()
.map(|p| obiread::estimated_decompressed_len(p))
.sum::<Option<u64>>();
let chunk_bytes = args let chunk_bytes = args
.chunk_size .chunk_size
.map(|mb| mb * 1024 * 1024) .map(|mb| mb * 1024 * 1024)
.unwrap_or_else(|| { .unwrap_or_else(|| {
let avail = available_memory_bytes(); let avail = available_memory_bytes();
let computed = avail / (n_workers as u64 * overhead_multiplier * SAFETY_FACTOR); let computed = avail / (n_workers as u64 * overhead_multiplier * SAFETY_FACTOR);
computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024) as usize let by_memory = computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024);
match input_bytes {
Some(total) => {
let by_parallelism = (total / (n_workers as u64 * CHUNKS_PER_WORKER)).max(MIN_CHUNK);
by_memory.min(by_parallelism) as usize
}
None => by_memory as usize,
}
}); });
debug!( debug!(
chunk_bytes, chunk_bytes,
input_bytes,
n_genomes, n_genomes,
detail = args.detail, detail = args.detail,
overhead_multiplier, overhead_multiplier,
+3 -1
View File
@@ -53,7 +53,9 @@ pub(super) fn run_bits_per_kmer(index_path: &PathBuf) {
pub(super) fn run_rename(index_path: &PathBuf, spec: &str) { pub(super) fn run_rename(index_path: &PathBuf, spec: &str) {
let (old_label, new_label) = parse_rename_spec(spec); let (old_label, new_label) = parse_rename_spec(spec);
let idx = KmerIndex::open(index_path).unwrap_or_else(|e| { // Rewrites the index metadata and renames a spectrum file in place:
// exclusive for the whole command.
let idx = KmerIndex::open_lock(index_path).unwrap_or_else(|e| {
eprintln!("error opening index: {e}"); eprintln!("error opening index: {e}");
std::process::exit(1); std::process::exit(1);
}); });
-595
View File
@@ -1,595 +0,0 @@
//! Merging a source partition's new layer into a destination partition:
//! de Bruijn graph union (pass 1) then column fill (pass 2).
//!
//! Submodules: [`src_layer`] (`SrcLayerData`, the opened-source-matrix
//! lookup used by pass 2 here and by `rebuild_layer`). The `merge_partition`
//! orchestration itself stays in this file — its ~400-line body is one
//! tightly threaded pipeline (shared `Arc`/`Mutex` state across pass 1,
//! builder setup, and pass 2), not a set of independently callable steps.
use std::fs;
use std::io;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use obipipeline::{
Pipeline, PipelineError, PipelineSender, SharedFlatFn, Stage, ThrottleGuard, WorkerPool,
make_sink, make_source, make_transform, throttle,
};
use tracing::debug;
use obicompactvec::{PersistentBitMatrixBuilder, PersistentCompactIntMatrixBuilder};
use obikseq::CanonicalKmer;
use obikindex::layer::{IndexMode, TypedLayer, LayeredMap, MphfOnly};
use obikindex::layer::utils::layer_dir;
use obiskio::UnitigFileReader;
use obikindex::{OKIError, OKIResult};
use obikindex::{ColBuilder, load_meta};
use obikindexer::{build_graph, materialize_layer};
use obikindex::KmerIndex;
mod src_layer;
pub use src_layer::SrcLayerData;
// ── MergeMode ─────────────────────────────────────────────────────────────────
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MergeMode {
Presence,
Count,
}
// ── MatrixBuilder ─────────────────────────────────────────────────────────────
//
// Wraps whichever matrix builder `mode` calls for, so the merge pipeline never
// has to know the on-disk column naming (`col_NNNNNN.pbiv`/`.pciv`) or the
// matrix `meta.json` schema itself — both stay private to obicompactvec.
// `resume` reopens a matrix directory already closed by a previous builder
// session (an existing destination layer), continuing from its current
// `n_cols` instead of starting a fresh matrix at 0.
enum MatrixBuilder {
Bit(PersistentBitMatrixBuilder),
Int(PersistentCompactIntMatrixBuilder),
}
impl MatrixBuilder {
fn new(mode: MergeMode, n: usize, dir: &Path) -> io::Result<Self> {
Ok(match mode {
MergeMode::Presence => MatrixBuilder::Bit(PersistentBitMatrixBuilder::new(n, dir)?),
MergeMode::Count => MatrixBuilder::Int(PersistentCompactIntMatrixBuilder::new(n, dir)?),
})
}
fn resume(mode: MergeMode, dir: &Path) -> io::Result<Self> {
Ok(match mode {
MergeMode::Presence => MatrixBuilder::Bit(PersistentBitMatrixBuilder::resume(dir)?),
MergeMode::Count => MatrixBuilder::Int(PersistentCompactIntMatrixBuilder::resume(dir)?),
})
}
/// Add a column with no data written (all-zero/false) — for genome
/// columns absent from this source (e.g. dst genomes in a new layer).
fn add_absent_col(&mut self) -> io::Result<()> {
match self {
MatrixBuilder::Bit(b) => b.add_col()?.close(),
MatrixBuilder::Int(b) => b.add_col()?.close(),
}
}
fn add_col(&mut self) -> io::Result<ColBuilder> {
Ok(match self {
MatrixBuilder::Bit(b) => ColBuilder::Bit(b.add_col()?),
MatrixBuilder::Int(b) => ColBuilder::Int(b.add_col()?),
})
}
fn close(self) -> io::Result<()> {
match self {
MatrixBuilder::Bit(b) => b.close(),
MatrixBuilder::Int(b) => b.close(),
}
}
}
#[cfg(test)]
mod matrix_builder_tests {
use tempfile::tempdir;
use obicompactvec::{PersistentBitMatrix, PersistentCompactIntMatrix};
use super::{ColBuilder, MatrixBuilder, MergeMode};
/// Mirrors `merge_partition`'s "new layer" setup: absent (dst-genome)
/// columns get no data, then source columns are filled — all through one
/// continuous `MatrixBuilder` session, closed once at the end.
#[test]
fn new_layer_absent_then_source_columns_presence() {
let dir = tempdir().unwrap();
let data_dir = dir.path().join("presence");
let mut mb = MatrixBuilder::new(MergeMode::Presence, 3, &data_dir).unwrap();
// Two absent (dst-genome) columns.
mb.add_absent_col().unwrap();
mb.add_absent_col().unwrap();
// One source column, filled like pass 2 would.
let mut col = mb.add_col().unwrap();
match &mut col {
ColBuilder::Bit(b) => {
b.set(0, true);
b.set(1, false);
b.set(2, true);
}
ColBuilder::Int(_) => unreachable!(),
}
col.close().unwrap();
mb.close().unwrap();
let m = PersistentBitMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
assert_eq!(&*m.row(0), &[false, false, true]);
assert_eq!(&*m.row(1), &[false, false, false]);
assert_eq!(&*m.row(2), &[false, false, true]);
}
#[test]
fn new_layer_absent_then_source_columns_count() {
let dir = tempdir().unwrap();
let data_dir = dir.path().join("counts");
let mut mb = MatrixBuilder::new(MergeMode::Count, 2, &data_dir).unwrap();
mb.add_absent_col().unwrap();
let mut col = mb.add_col().unwrap();
match &mut col {
ColBuilder::Int(b) => {
b.set(0, 7);
b.set(1, 42);
}
ColBuilder::Bit(_) => unreachable!(),
}
col.close().unwrap();
mb.close().unwrap();
let m = PersistentCompactIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 2);
assert_eq!(&*m.row(0), &[0u32, 7]);
assert_eq!(&*m.row(1), &[0u32, 42]);
}
/// Mirrors `merge_partition`'s "existing dst layer" setup: an
/// already-closed matrix (from a previous merge) gets more columns
/// appended via `resume`, without callers ever seeing `col_path`/
/// `MatrixMeta` — `n` itself is read back from the matrix directory.
#[test]
fn resume_appends_source_columns_to_existing_layer() {
let dir = tempdir().unwrap();
let data_dir = dir.path().join("presence");
// Previous merge: one dst-genome column already on disk.
let mut mb0 = MatrixBuilder::new(MergeMode::Presence, 3, &data_dir).unwrap();
mb0.add_absent_col().unwrap();
mb0.close().unwrap();
// This merge: resume and append two more source columns.
let mut mb = MatrixBuilder::resume(MergeMode::Presence, &data_dir).unwrap();
for vals in [[true, false, true], [false, true, false]] {
let mut col = mb.add_col().unwrap();
match &mut col {
ColBuilder::Bit(b) => {
for (slot, v) in vals.into_iter().enumerate() {
b.set(slot, v);
}
}
ColBuilder::Int(_) => unreachable!(),
}
col.close().unwrap();
}
mb.close().unwrap();
let m = PersistentBitMatrix::open(dir.path()).unwrap();
assert_eq!(m.n_cols(), 3);
assert_eq!(&*m.row(0), &[false, true, false]);
assert_eq!(&*m.row(1), &[false, false, true]);
assert_eq!(&*m.row(2), &[false, true, false]);
}
}
// ── KmerPartition::merge_partition ────────────────────────────────────────────
impl KmerIndex {
/// Merge `sources` into destination partition `i`.
///
/// Each entry in `sources` is `(partition, n_genomes)` where `n_genomes` is
/// the number of genome columns that source contributes. A merged index
/// contributes more than one. The total new columns added to the destination
/// is `sum(n_genomes)`.
///
/// `n_dst_genomes` is the number of genome columns already in the destination
/// matrices (copied from source_0 before this call).
pub fn merge_partition(
&self,
i: usize,
sources: &[(&KmerIndex, usize)],
mode: MergeMode,
n_dst_genomes: usize,
block_bits: u8,
evidence: &IndexMode,
) -> OKIResult<usize> {
let dst_index_dir = self.index_dir(i);
if !dst_index_dir.exists() {
return Ok(0);
}
load_meta(&dst_index_dir)?; // ensure meta.json exists before LayeredMap::open
let dst_map = Arc::new(LayeredMap::<()>::open(&dst_index_dir)?);
let n_dst_layers = dst_map.n_layers();
let n_src_total: usize = sources.iter().map(|(_, n)| *n).sum();
// First merge in presence mode: init presence matrices on existing layers
// (all slots true — every kmer in those layers belongs to genome_0).
if n_dst_genomes == 1 && mode == MergeMode::Presence {
for l in 0..n_dst_layers {
TypedLayer::<()>::init_presence_matrix(
&layer_dir(&dst_index_dir, l),
dst_map.layer(l).n(),
)?;
}
}
// ── Pass 1: pipeline — parallel file read + dst_map filter + graph fill ─
//
// Source : list of unitigs.bin paths (one per source × layer)
// Flat : open file, emit Vec<CanonicalKmer> batches (BeeGFS parallel I/O)
// Transform: filter via dst_map.query() — thread-safe, LayeredMap<()>: Sync
// Sink : push new kmers into GraphDeBruijn (single thread, no locks needed)
// Collect file paths (propagates load_meta errors before the pipeline starts)
let mut unitig_paths: Vec<PathBuf> = Vec::new();
for (src, _) in sources.iter() {
let src_index_dir = src.index_dir(i);
if !src_index_dir.exists() {
continue;
}
let src_meta = load_meta(&src_index_dir)?;
for l in 0..src_meta.n_layers {
let p = layer_dir(&src_index_dir, l).join("unitigs.bin");
if p.exists() {
unitig_paths.push(p);
}
}
}
let n_src_layers = unitig_paths.len();
debug!("partition {i}: de Bruijn graph build start — {n_src_layers} source layer(s)");
const BATCH: usize = 4096;
let n_workers = rayon::current_num_threads().min(16).max(4);
// At most 2 files open simultaneously: keeps n_workers-2 workers free
// for the Transform stage. Each open file monopolises one worker for the
// full duration of its read, so this must stay well below n_workers.
let max_open = 2;
let dst_filter = Arc::clone(&dst_map);
let g = build_graph(
unitig_paths.into_iter(),
move |path: PathBuf, emit: &mut dyn FnMut(Vec<CanonicalKmer>)| -> OKIResult<()> {
let reader = UnitigFileReader::open_sequential(&path)?;
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
batch.push(kmer);
if batch.len() == BATCH {
emit(std::mem::replace(&mut batch, Vec::with_capacity(BATCH)));
}
}
if !batch.is_empty() {
emit(batch);
}
Ok(())
},
move |kmer| dst_filter.query(kmer).is_none(),
n_workers,
max_open,
)?;
let any_new = g.len() > 0;
debug!(
"partition {i}: de Bruijn graph done — {} new kmers",
g.len()
);
// Build new layer from de Bruijn graph if there are new kmers.
let new_layer_idx = n_dst_layers;
let new_layer_dir = layer_dir(&dst_index_dir, new_layer_idx);
let n_new = if any_new {
debug!("partition {i}: unitig traversal start — {} nodes", g.len());
let n_nodes = materialize_layer(g, &new_layer_dir, block_bits, evidence)?;
debug!("partition {i}: MPHF build done");
n_nodes
} else {
drop(g);
0
};
let t_open = std::time::Instant::now();
let new_mphf: Option<Arc<MphfOnly>> = if any_new {
Some(Arc::new(MphfOnly::open(&new_layer_dir)?))
} else {
None
};
debug!(
"partition {i}: MPHF open in {:.3}s",
t_open.elapsed().as_secs_f64()
);
// ── Prepare matrix directories for the new layer ──────────────────────
// Absent columns (dst genomes) get an all-zero/false column. Source-genome
// columns are created as mutable builders for pass 2. `new_mb` is kept
// open (not `close`d) until pass 2 has filled every source column, so its
// `n_cols` bookkeeping stays in sync with the columns actually created.
let (new_src_builders, new_mb): (Vec<ColBuilder>, Option<MatrixBuilder>) = if any_new {
let data_dir = match mode {
MergeMode::Presence => new_layer_dir.join("presence"),
MergeMode::Count => new_layer_dir.join("counts"),
};
fs::create_dir_all(&data_dir)?;
let mut mb = MatrixBuilder::new(mode, n_new, &data_dir).map_err(OKIError::Io)?;
for _ in 0..n_dst_genomes {
mb.add_absent_col().map_err(OKIError::Io)?;
}
let cols = (0..n_src_total)
.map(|_| mb.add_col().map_err(OKIError::Io))
.collect::<OKIResult<Vec<_>>>()?;
(cols, Some(mb))
} else {
(vec![], None)
};
let t_builders = std::time::Instant::now();
// Builders for existing layers: n_src_total per layer, resumed from
// each layer's own matrix directory (already holding n_dst_genomes
// columns from a previous merge). Columns land at
// n_dst_genomes .. n_dst_genomes + n_src_total - 1.
let mut exist_mbs: Vec<MatrixBuilder> = Vec::with_capacity(n_dst_layers);
let mut exist_builders: Vec<Vec<ColBuilder>> = Vec::with_capacity(n_dst_layers);
for l in 0..n_dst_layers {
let layer_dir = layer_dir(&dst_index_dir, l);
let data_dir = match mode {
MergeMode::Presence => layer_dir.join("presence"),
MergeMode::Count => layer_dir.join("counts"),
};
let mut mb = MatrixBuilder::resume(mode, &data_dir).map_err(OKIError::Io)?;
let cols = (0..n_src_total)
.map(|_| mb.add_col().map_err(OKIError::Io))
.collect::<OKIResult<Vec<_>>>()?;
exist_mbs.push(mb);
exist_builders.push(cols);
}
debug!(
"partition {i}: builders ready in {:.3}s",
t_builders.elapsed().as_secs_f64()
);
// ── Pass 2: fill builders (pipeline) ─────────────────────────────────
let t_pass2 = std::time::Instant::now();
// Collect source items before the pipeline so load_meta errors propagate
// via ? before any worker thread is spawned.
let mut pass2_items: Vec<(usize, usize, PathBuf)> = Vec::new();
{
let mut col_offset = 0usize;
for (src, src_n) in sources.iter() {
let src_index_dir = src.index_dir(i);
if !src_index_dir.exists() {
col_offset += src_n;
continue;
}
let src_meta = load_meta(&src_index_dir)?;
for l in 0..src_meta.n_layers {
let src_layer_dir = layer_dir(&src_index_dir, l);
if src_layer_dir.join("unitigs.bin").exists() {
pass2_items.push((col_offset, *src_n, src_layer_dir));
}
}
col_offset += src_n;
}
}
enum Pass2Data {
SrcLayer((usize, usize, PathBuf, ThrottleGuard)),
RawBatch((usize, usize, Arc<SrcLayerData>, Vec<CanonicalKmer>)),
WriteBatch(Vec<(Option<usize>, usize, usize, u32)>),
}
let exist_locked: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_builders
.into_iter()
.map(|layer| layer.into_iter().map(|b| Arc::new(Mutex::new(b))).collect())
.collect();
let new_locked: Vec<Arc<Mutex<ColBuilder>>> = new_src_builders
.into_iter()
.map(|b| Arc::new(Mutex::new(b)))
.collect();
let exist_sink: Vec<Vec<Arc<Mutex<ColBuilder>>>> = exist_locked
.iter()
.map(|layer| layer.iter().map(Arc::clone).collect())
.collect();
let new_sink: Vec<Arc<Mutex<ColBuilder>>> = new_locked.iter().map(Arc::clone).collect();
let dst_map_t2 = Arc::clone(&dst_map);
let new_mphf_t2 = new_mphf.clone();
let pass2_err: Arc<Mutex<Option<String>>> = Arc::new(Mutex::new(None));
let err_cap2 = Arc::clone(&pass2_err);
let capacity = 2;
let throttled_pass2 = throttle(pass2_items.into_iter(), max_open);
let pipeline2 = Pipeline::new(
make_source!(
Pass2Data,
throttled_pass2.map(|t| {
let (col_offset, src_n, src_layer_dir) = t.item;
(col_offset, src_n, src_layer_dir, t.guard)
}),
SrcLayer
),
vec![
Stage::Flat(Arc::new(
move |data: Pass2Data,
push: &PipelineSender<Result<Pass2Data, PipelineError>>,
delta: &PipelineSender<isize>| {
if let Pass2Data::SrcLayer((col_offset, src_n, src_layer_dir, _guard)) =
data
{
// _guard dropped at end of block, releasing the slot.
let reader = match UnitigFileReader::open_sequential(
&src_layer_dir.join("unitigs.bin"),
) {
Ok(r) => r,
Err(e) => {
*err_cap2.lock().unwrap() = Some(e.to_string());
delta.send(-1).ok();
return;
}
};
let src_data = match SrcLayerData::open(&src_layer_dir, mode) {
Ok(d) => Arc::new(d),
Err(e) => {
*err_cap2.lock().unwrap() = Some(e.to_string());
delta.send(-1).ok();
return;
}
};
const BATCH: usize = 4096;
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(BATCH);
let mut count: isize = 0;
for (kmer, _, _) in reader.iter_indexed_canonical_kmers() {
batch.push(kmer);
if batch.len() == BATCH {
let b =
std::mem::replace(&mut batch, Vec::with_capacity(BATCH));
push.send(Ok(Pass2Data::RawBatch((
col_offset,
src_n,
Arc::clone(&src_data),
b,
))))
.ok();
count += 1;
}
}
if !batch.is_empty() {
push.send(Ok(Pass2Data::RawBatch((
col_offset, src_n, src_data, batch,
))))
.ok();
count += 1;
}
delta.send(count - 1).ok();
}
},
) as SharedFlatFn<Pass2Data>),
make_transform!(
Pass2Data,
{
move |(col_offset, src_n, src_data, kmers): (
usize,
usize,
Arc<SrcLayerData>,
Vec<CanonicalKmer>,
)|
-> Vec<(Option<usize>, usize, usize, u32)> {
let mut ops: Vec<(Option<usize>, usize, usize, u32)> = Vec::new();
for kmer in kmers {
let values = src_data.lookup(kmer, src_n);
if let Some((dst_layer, hit)) = dst_map_t2.query(kmer) {
for (g, val) in values.into_iter().enumerate() {
ops.push((Some(dst_layer), col_offset + g, hit.slot, val));
}
} else if let Some(ref mphf) = new_mphf_t2 {
let slot = mphf.index(kmer);
for (g, val) in values.into_iter().enumerate() {
ops.push((None, col_offset + g, slot, val));
}
}
}
ops
}
},
RawBatch,
WriteBatch
),
],
make_sink!(
Pass2Data,
{
move |ops: Vec<(Option<usize>, usize, usize, u32)>| {
for (layer_opt, col, slot, val) in ops {
match layer_opt {
Some(l) => exist_sink[l][col].lock().unwrap().set_val(slot, val),
None => new_sink[col].lock().unwrap().set_val(slot, val),
}
}
}
},
WriteBatch
),
);
WorkerPool::new(pipeline2, n_workers, capacity).run();
debug!(
"partition {i}: pass2 pipeline done in {:.3}s",
t_pass2.elapsed().as_secs_f64()
);
if let Some(msg) = Arc::try_unwrap(pass2_err)
.unwrap_or_else(|_| panic!("pass2: pass2_err not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
{
return Err(OKIError::InvalidData {
context: "merge pass2",
detail: msg,
});
}
let t_close = std::time::Instant::now();
// ── Close builders and update metadata ────────────────────────────────
for (mb, builders) in exist_mbs.into_iter().zip(exist_locked.into_iter()) {
for b in builders {
Arc::try_unwrap(b)
.unwrap_or_else(|_| panic!("pass2: exist_builder not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
.close()?;
}
mb.close().map_err(OKIError::Io)?;
}
for b in new_locked {
Arc::try_unwrap(b)
.unwrap_or_else(|_| panic!("pass2: new_builder not uniquely owned"))
.into_inner()
.unwrap_or_else(|e| e.into_inner())
.close()?;
}
if let Some(mb) = new_mb {
mb.close().map_err(OKIError::Io)?;
let mut part_meta = self.partition_meta(i)?;
part_meta.n_layers = new_layer_idx + 1;
part_meta
.save(&dst_index_dir)?;
}
debug!(
"partition {i}: builders closed in {:.3}s",
t_close.elapsed().as_secs_f64()
);
Ok(n_new)
}
}
@@ -1,94 +0,0 @@
use std::path::Path;
use obicompactvec::{MatrixGroupOps, PersistentBitMatrix, PersistentCompactIntMatrix};
use obikseq::CanonicalKmer;
use obikindex::layer::MphfOnly;
use obikindex::{OKIError, OKIResult};
use super::MergeMode;
// ── SrcLayerData — opened source matrix for pass-2 lookup ─────────────────────
pub enum SrcLayerData {
Presence(MphfOnly, PersistentBitMatrix),
Count(MphfOnly, PersistentCompactIntMatrix),
}
impl SrcLayerData {
pub fn open(layer_dir: &Path, merge_mode: MergeMode) -> OKIResult<Self> {
let counts_dir = layer_dir.join("counts");
match merge_mode {
MergeMode::Presence => {
if counts_dir.exists() && !layer_dir.join("presence").exists() {
let mphf = MphfOnly::open(layer_dir)?;
let mat = PersistentCompactIntMatrix::open(layer_dir).map_err(OKIError::Io)?;
Ok(SrcLayerData::Count(mphf, mat))
} else {
// presence dir exists, or neither exists → Implicit handled by open()
let mphf = MphfOnly::open(layer_dir)?;
let mat = PersistentBitMatrix::open(layer_dir).map_err(OKIError::Io)?;
Ok(SrcLayerData::Presence(mphf, mat))
}
}
MergeMode::Count => {
let mphf = MphfOnly::open(layer_dir)?;
if counts_dir.exists() {
let mat = PersistentCompactIntMatrix::open(layer_dir).map_err(OKIError::Io)?;
Ok(SrcLayerData::Count(mphf, mat))
} else {
// No counts → treat as implicit presence (all 1s)
let mat = PersistentBitMatrix::open(layer_dir).map_err(OKIError::Io)?;
Ok(SrcLayerData::Presence(mphf, mat))
}
}
}
}
/// Return one value per source genome for `kmer`.
/// The caller guarantees `kmer` is in the source MPHF domain.
#[inline]
pub(crate) fn lookup(&self, kmer: CanonicalKmer, n_genomes: usize) -> Vec<u32> {
let mut buf = vec![0u32; n_genomes];
match self {
SrcLayerData::Presence(mphf, mat) => mat.fill_row(mphf.index(kmer), &mut buf),
SrcLayerData::Count(mphf, mat) => mat.fill_row(mphf.index(kmer), &mut buf),
}
buf
}
pub fn n_slots(&self) -> usize {
match self {
SrcLayerData::Presence(_, mat) => mat.n(),
SrcLayerData::Count(_, mat) => mat.n(),
}
}
/// MPHF lookup: returns the slot index for `kmer` (kmer must be in the domain).
#[inline]
pub fn slot(&self, kmer: CanonicalKmer) -> usize {
match self {
SrcLayerData::Presence(mphf, _) => mphf.index(kmer),
SrcLayerData::Count(mphf, _) => mphf.index(kmer),
}
}
/// Row lookup by slot index, bypassing the MPHF.
#[inline]
pub fn fill_row_by_slot(&self, slot: usize, n_genomes: usize) -> Vec<u32> {
let mut buf = vec![0u32; n_genomes];
match self {
SrcLayerData::Presence(_, mat) => mat.fill_row(slot, &mut buf),
SrcLayerData::Count(_, mat) => mat.fill_row(slot, &mut buf),
}
buf
}
/// Call `f` with a reference to the underlying matrix as `&dyn MatrixGroupOps`.
pub fn with_matrix<R>(&self, f: impl FnOnce(&dyn MatrixGroupOps) -> R) -> R {
match self {
SrcLayerData::Presence(_, mat) => f(mat),
SrcLayerData::Count(_, mat) => f(mat),
}
}
}
@@ -17,6 +17,7 @@ use super::entropy::{family_entropy, family_entropy_4};
use super::family_scan::{Selection, scan_layer_families}; use super::family_scan::{Selection, scan_layer_families};
use super::is_fast_mode; use super::is_fast_mode;
use super::minorant_selection::non_monomorphic_reindex_layer; use super::minorant_selection::non_monomorphic_reindex_layer;
use crate::siblings::extensions::SiblingBuilder;
/// One variable family, as `--shannon` reports it. /// One variable family, as `--shannon` reports it.
pub(crate) struct FamilyEntropyRow { pub(crate) struct FamilyEntropyRow {
@@ -45,6 +46,7 @@ pub(crate) fn scan_family_entropies(
cache: &IndexCache, cache: &IndexCache,
mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>, mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>,
) -> OKIResult<()> { ) -> OKIResult<()> {
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len(); let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache); let fast_mode = is_fast_mode(cache);
let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions"); let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions");
@@ -22,7 +22,9 @@ use super::masking::is_quartet_informative;
use super::minorant_selection::non_monomorphic_reindex_layer; use super::minorant_selection::non_monomorphic_reindex_layer;
use super::pair_weight::pair_balance_weight; use super::pair_weight::pair_balance_weight;
use crate::siblings::quartet_annex::quartet_annex_file_name; use crate::siblings::quartet_annex::quartet_annex_file_name;
use crate::siblings::{FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind}; use crate::siblings::{
FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind, family_overlap_file_name,
};
/// Which annexes a caller needs, each listed once per `--no-ambiguity` /// Which annexes a caller needs, each listed once per `--no-ambiguity`
/// counting convention (`false` = genomes carrying several members count as a /// counting convention (`false` = genomes carrying several members count as a
@@ -50,6 +52,24 @@ impl AnnexRequest {
dedup(self.overlap.iter().chain(&self.pair_weight).copied()) dedup(self.overlap.iter().chain(&self.pair_weight).copied())
} }
/// Whether every annex this request asks for already exists — the same
/// per-file existence test the builders apply, so `true` means they would
/// find nothing to build and no lock is needed. Cheap (no file is opened).
pub(crate) fn is_satisfied(&self, cache: &IndexCache) -> bool {
let overlap = self
.overlap_conventions()
.into_iter()
.all(|nb| cache.index().dir().join(family_overlap_file_name(nb)).exists());
let pair_weight = self.pair_weight_conventions();
overlap
&& cache.iter_indexed().all(|(_, _, layer)| {
let has = |name: &str| layer.dir().join(name).exists();
self.entropy.iter().all(|&nb| has(ScalarKind::Entropy.file_name(nb)))
&& self.quartet.iter().all(|&nb| has(quartet_annex_file_name(nb)))
&& pair_weight.iter().all(|&nb| has(ScalarKind::PairWeight.file_name(nb)))
})
}
/// The pair-weight conventions requested, each once. /// The pair-weight conventions requested, each once.
pub(crate) fn pair_weight_conventions(&self) -> Vec<bool> { pub(crate) fn pair_weight_conventions(&self) -> Vec<bool> {
dedup(self.pair_weight.iter().copied()) dedup(self.pair_weight.iter().copied())
@@ -54,7 +54,7 @@ use obipipeline::ThrottleGuard;
use crate::siblings::helpers::central_base; use crate::siblings::helpers::central_base;
use crate::siblings::iter::{SiblingEntry, SiblingLayerExt}; use crate::siblings::iter::{SiblingEntry, SiblingLayerExt};
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, SiblingAnnex}; use crate::siblings::{FamilyMask, SiblingAnnex};
/// Target memory budget for one batch's `genome_mask` allocation /// Target memory budget for one batch's `genome_mask` allocation
/// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding /// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding
@@ -180,7 +180,7 @@ pub(crate) fn scan_layer_families(
})?; })?;
let n_cols = mat.n_cols().min(n_genomes); let n_cols = mat.n_cols().min(n_genomes);
let layer_dir: &Path = mat.dir(); let layer_dir: &Path = mat.dir();
let annex = Arc::new(SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?); let annex = Arc::new(SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?);
let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k }); let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k });
@@ -11,7 +11,7 @@ use std::path::Path;
use obicompactvec::TempBitVecBuilder; use obicompactvec::TempBitVecBuilder;
use obikindex::{OKIError, OKIResult}; use obikindex::{OKIError, OKIResult};
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex}; use crate::siblings::{FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool { pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
mask.is_minorant() && mask.siblings() >= 1 mask.is_minorant() && mask.siblings() >= 1
@@ -31,7 +31,7 @@ pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
/// the `Selection` a caller needs to bound `scan_layer_families` to just /// the `Selection` a caller needs to bound `scan_layer_families` to just
/// these families. /// these families.
pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> { pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let mut reindex = HashMap::new(); let mut reindex = HashMap::new();
let mut family_idx = 0usize; let mut family_idx = 0usize;
for slot in 0..annex.len() { for slot in 0..annex.len() {
@@ -62,7 +62,7 @@ pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashM
/// first counts this layer's total minorants (needed up front to size the /// first counts this layer's total minorants (needed up front to size the
/// bitset), the second sets the eligible bits. /// bitset), the second sets the eligible bits.
pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> { pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let n_minorants = (0..annex.len()) let n_minorants = (0..annex.len())
.filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant())) .filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant()))
@@ -103,7 +103,7 @@ pub(crate) fn iter_full_scalar(
kind: ScalarKind, kind: ScalarKind,
no_ambiguity: bool, no_ambiguity: bool,
) -> OKIResult<impl Iterator<Item = f32> + use<>> { ) -> OKIResult<impl Iterator<Item = f32> + use<>> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let scalars = let scalars =
FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?; FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?;
if scalars.no_ambiguity() != no_ambiguity { if scalars.no_ambiguity() != no_ambiguity {
@@ -8,7 +8,7 @@ use obikindex::{OKIError, OKIResult};
use super::minorant_selection::is_non_monomorphic_minorant; use super::minorant_selection::is_non_monomorphic_minorant;
use crate::siblings::quartet_annex::quartet_annex_file_name; use crate::siblings::quartet_annex::quartet_annex_file_name;
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex}; use crate::siblings::SiblingAnnex;
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants /// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
/// included), every family the quartet annex flags as not quartet-informative /// included), every family the quartet annex flags as not quartet-informative
@@ -22,7 +22,7 @@ pub(crate) fn restrict_to_quartet(
layer_dir: &Path, layer_dir: &Path,
no_ambiguity: bool, no_ambiguity: bool,
) -> OKIResult<()> { ) -> OKIResult<()> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?; let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity))) let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity)))
.map_err(OKIError::Io)?; .map_err(OKIError::Io)?;
+9 -16
View File
@@ -9,11 +9,12 @@
//! already stored — no re-derivation, no lookup). //! already stored — no re-derivation, no lookup).
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikindex::{OKIError, OKIResult}; use obikindex::OKIResult;
use super::family_scan::{Selection, scan_layer_families}; use super::family_scan::{Selection, scan_layer_families};
use super::is_fast_mode; use super::is_fast_mode;
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex}; use crate::siblings::SiblingAnnex;
use crate::siblings::extensions::SiblingBuilder;
/// Layers scanned at once per [`sibling_family_size_histogram`]'s /// Layers scanned at once per [`sibling_family_size_histogram`]'s
/// `par_map_reduce` call — same bounding reasoning as /// `par_map_reduce` call — same bounding reasoning as
@@ -45,26 +46,17 @@ pub struct SiblingAnnexStats {
/// on top of that would fight it for cache/mmap locality) — this function /// on top of that would fight it for cache/mmap locality) — this function
/// has no such inner parallelism to protect. /// has no such inner parallelism to protect.
/// ///
/// Existence of every cached layer's annex file is checked up front, before /// A missing annex is built up front (`ensure_sibling_annex`), before the
/// the parallel map runs, so a missing one is reported as an error rather /// parallel map runs, so none is silently read back as zero.
/// than silently read back as zero.
pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> { pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> {
for (part, l, layer) in cache.iter_indexed() { cache.ensure_sibling_annex()?;
let path = layer.dir().join(ANNEX_FILE_NAME);
if !path.exists() {
return Err(OKIError::InvalidInput(format!(
"no sibling annex at {} (partition {part}, layer {l}) — run --sibling-annex first",
path.display()
)));
}
}
Ok(cache.par_map_reduce( Ok(cache.par_map_reduce(
MAX_CONCURRENT_LAYERS, MAX_CONCURRENT_LAYERS,
|| [0u64; 4], || [0u64; 4],
|layer| { |layer| {
let annex = SiblingAnnex::open(&layer.dir().join(ANNEX_FILE_NAME)) let annex = SiblingAnnex::open_in(layer.dir())
.expect("existence already checked before this parallel map ran"); .expect("ensure_sibling_annex already made every layer's annex exist");
let mut counts = [0u64; 4]; let mut counts = [0u64; 4];
for slot in 0..annex.len() { for slot in 0..annex.len() {
let Some(mask) = annex.get(slot) else { continue }; let Some(mask) = annex.get(slot) else { continue };
@@ -95,6 +87,7 @@ pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u6
/// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/ /// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/
/// `algorithms::scan::sampled_scan` already follow). /// `algorithms::scan::sampled_scan` already follow).
pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> { pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> {
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len(); let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache); let fast_mode = is_fast_mode(cache);
@@ -241,6 +241,9 @@ pub(crate) fn sample_index(
skip_layers: usize, skip_layers: usize,
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>), mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
) -> OKIResult<SamplePlan> { ) -> OKIResult<SamplePlan> {
// Every selection below reads the sibling annex (`eligible_bitset_layer`),
// whichever derived annexes the options ask for.
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len(); let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache); let fast_mode = is_fast_mode(cache);
@@ -12,10 +12,13 @@ use obikindex::{OKIError, OKIResult};
use obisys::progress_bar; use obisys::progress_bar;
use crate::siblings::algorithms::{ use crate::siblings::algorithms::{
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes, AnnexRequest, build_layer_sibling_annex, ensure_layer_family_annexes,
inverse_overlap_matrix, is_fast_mode, ensure_layer_pair_weight_annexes, inverse_overlap_matrix, is_fast_mode,
};
use crate::siblings::{
ANNEX_FILE_NAME, FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, FamilyOverlapAccumulator,
QUARTET_ANNEX_FILE_NAMES, ScalarKind, family_overlap_file_name,
}; };
use crate::siblings::{FamilyOverlap, FamilyOverlapAccumulator, family_overlap_file_name};
pub(crate) trait SiblingBuilder { pub(crate) trait SiblingBuilder {
/// Build, for every cached layer, the annexes `request` asks for that /// Build, for every cached layer, the annexes `request` asks for that
@@ -26,13 +29,129 @@ pub(crate) trait SiblingBuilder {
/// a missing entropy, quartet or overlap annex triggers one scan that /// a missing entropy, quartet or overlap annex triggers one scan that
/// feeds every missing one, per `ensure_layer_family_annexes`'s own docs, /// feeds every missing one, per `ensure_layer_family_annexes`'s own docs,
/// then a second scan builds the missing pair weights, which need the /// then a second scan builds the missing pair weights, which need the
/// finished overlap matrix. A no-op, cheap existence-check pass, once /// finished overlap matrix. Makes sure the sibling annex exists first
/// everything requested already exists. /// ([`ensure_sibling_annex`](Self::ensure_sibling_annex)). A no-op, cheap
/// existence-check pass that takes no lock, once everything requested
/// already exists; otherwise the build runs under the index's exclusive
/// lock ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock)).
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>; fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
/// [`ensure_family_annexes`](Self::ensure_family_annexes)'s build step,
/// without the existence pre-check and without the lock: the caller
/// holds it.
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
/// Build the sibling annex (`siblings.psib`, every annex above is derived
/// from it) if any cached layer lacks it — the state of an index right
/// after `migrate-mphf`, which sets every annex aside. The build writes
/// into every layer, so it runs under the index's exclusive lock
/// ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock), a no-op
/// when the caller already holds it); a process that waited for it finds
/// the annex built by the one that held it, and does nothing. A no-op,
/// cheap existence-check pass that takes no lock, once every layer has
/// its annex.
fn ensure_sibling_annex(&self) -> OKIResult<()>;
/// (Re)build the sibling annex of every cached layer, unconditionally,
/// removing the annexes derived from it (entropy, quartet, pair weight,
/// family overlap), which its renumbering makes stale. Takes no lock:
/// the caller holds it ([`SiblingExt::build_sibling_annex`], or
/// [`ensure_sibling_annex`](Self::ensure_sibling_annex)).
fn build_base_sibling_annex(&self) -> OKIResult<()>;
} }
impl SiblingBuilder for IndexCache { impl SiblingBuilder for IndexCache {
fn ensure_sibling_annex(&self) -> OKIResult<()> {
let missing = || self.iter_indexed().any(|(_, _, layer)| !layer.dir().join(ANNEX_FILE_NAME).exists());
if !missing() {
return Ok(());
}
self.index().with_lock(|| {
if !missing() {
return Ok(()); // built by the process we waited for
}
tracing::info!("sibling annex missing — building it");
self.build_base_sibling_annex()
})
}
fn build_base_sibling_annex(&self) -> OKIResult<()> {
let n_parts = self.index().n_partitions();
let fast_mode = is_fast_mode(self);
if !fast_mode {
tracing::warn!(
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
disabled; consider compacting this index"
);
}
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
let mut total_slots: u64 = 0;
for part in self.partitions() {
let n_layer = self.n_layer(part).unwrap_or(0);
let mut part_slots: u64 = 0;
for l in 0..n_layer {
let layer = self
.get_layer(part, l)
.expect("cache-consistent: n_layer(part) already bounds l");
// Invalidate this layer's entropy, quartet and pair-weight
// annexes, if any — they were built against the sibling
// annex's *previous* `family_idx` numbering
// (`non_monomorphic_reindex_layer` reads the sibling
// annex), and are stale once the new one exists.
// `ensure_family_annexes` only builds what's *missing*, so
// the old files must be removed here for them to be rebuilt
// rather than silently left stale. Removed *before* the new
// annex is built and published (atomically, at its end): a
// build killed in between leaves the previous annex with no
// derived files — still consistent, rebuilt on demand —
// never a new annex next to derived files computed against
// the old numbering.
for name in ScalarKind::Entropy
.file_names()
.into_iter()
.chain(ScalarKind::PairWeight.file_names())
.chain(QUARTET_ANNEX_FILE_NAMES)
{
let derived_path = layer.dir().join(name);
if derived_path.exists() {
std::fs::remove_file(&derived_path).map_err(OKIError::Io)?;
}
}
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
}
total_slots += part_slots;
pb.inc(1);
pb.set_message(format!(
"partition {part}: {part_slots} kmers ({total_slots} total)"
));
}
pb.finish_and_clear();
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
// The family-overlap annexes are index-level, not per-layer (see
// their own module docs) — any layer's sibling annex changing makes
// them all stale (and, with them, every pair weight computed from
// them, removed per layer above), so they are invalidated once here
// rather than per-layer above.
for name in FAMILY_OVERLAP_FILE_NAMES {
let overlap_path = self.index().dir().join(name);
if overlap_path.exists() {
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
}
}
Ok(())
}
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> { fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
self.ensure_sibling_annex()?;
if request.is_satisfied(self) {
return Ok(());
}
self.index().with_lock(|| self.build_family_annexes(request))
}
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
let n_genomes = self.meta().genomes().len(); let n_genomes = self.meta().genomes().len();
let fast_mode = is_fast_mode(self); let fast_mode = is_fast_mode(self);
let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity)); let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity));
@@ -11,17 +11,12 @@ use std::path::Path;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikindex::{OKIError, OKIResult}; use obikindex::{OKIError, OKIResult};
use obisys::progress_bar;
use crate::siblings::algorithms::{ use crate::siblings::algorithms::{
AnnexRequest, EntropySummary, SiblingAnnexStats, build_layer_sibling_annex, is_fast_mode, AnnexRequest, EntropySummary, SiblingAnnexStats, scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
}; };
use crate::siblings::extensions::SiblingBuilder; use crate::siblings::extensions::SiblingBuilder;
use crate::siblings::{ use crate::siblings::{FamilyOverlap, family_overlap_file_name};
FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, QUARTET_ANNEX_FILE_NAMES, ScalarKind,
family_overlap_file_name,
};
pub trait SiblingExt { pub trait SiblingExt {
/// Build the sibling-count/minorant annex for every layer of every /// Build the sibling-count/minorant annex for every layer of every
@@ -111,82 +106,18 @@ pub trait SiblingExt {
impl SiblingExt for IndexCache { impl SiblingExt for IndexCache {
fn build_sibling_annex(&self) -> OKIResult<()> { fn build_sibling_annex(&self) -> OKIResult<()> {
let n_parts = self.index().n_partitions(); self.index().with_lock(|| {
let fast_mode = is_fast_mode(self); self.build_base_sibling_annex()?;
if !fast_mode { // Precompute the entropy, quartet, pair-weight and family-overlap
tracing::warn!( // annexes too, now that every layer's fresh sibling annex is on
"index has more than 7 layers per partition — sibling-annex fast layer lookup \ // disk (`non_monomorphic_reindex_layer`, which they depend on,
disabled; consider compacting this index" // reads it) — same routine-rebuild footing as the sibling annex
); // itself, rather than leaving them to be built lazily on first
} // `--shannon`/`--entropy`/`--quartet`/`--pair-balance`/
// `--min-shared-family`/`--family-overlap` use.
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions"); tracing::info!("building entropy, quartet, pair-weight (both --no-ambiguity conventions) and family-overlap annexes");
let mut total_slots: u64 = 0; self.ensure_family_annexes(&AnnexRequest::everything())
for part in self.partitions() { })
let n_layer = self.n_layer(part).unwrap_or(0);
let mut part_slots: u64 = 0;
for l in 0..n_layer {
let layer = self
.get_layer(part, l)
.expect("cache-consistent: n_layer(part) already bounds l");
// Invalidate this layer's entropy and quartet annexes, if
// any — they were built against the sibling annex's
// *previous* `family_idx` numbering
// (`non_monomorphic_reindex_layer` reads the sibling
// annex), and are stale once the new one exists.
// `ensure_family_annexes` below only builds what's
// *missing*, so the old files must be removed here for
// them to be rebuilt rather than silently left stale.
// Removed *before* the new annex is built and published
// (atomically, at its end): a build killed in between
// leaves the previous annex with no derived files — still
// consistent, rebuilt on demand — never a new annex next to
// derived files computed against the old numbering.
for name in ScalarKind::Entropy
.file_names()
.into_iter()
.chain(ScalarKind::PairWeight.file_names())
.chain(QUARTET_ANNEX_FILE_NAMES)
{
let entropy_path = layer.dir().join(name);
if entropy_path.exists() {
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
}
}
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
}
total_slots += part_slots;
pb.inc(1);
pb.set_message(format!(
"partition {part}: {part_slots} kmers ({total_slots} total)"
));
}
pb.finish_and_clear();
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
// The family-overlap annexes are index-level, not per-layer (see
// their own module docs) — any layer's sibling annex changing makes
// them all stale (and, with them, every pair weight computed from
// them, removed per layer above), so they are invalidated once here
// rather than per-layer above.
for name in FAMILY_OVERLAP_FILE_NAMES {
let overlap_path = self.index().dir().join(name);
if overlap_path.exists() {
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
}
}
// Precompute the entropy and family-overlap annexes too, now that
// every layer's fresh sibling annex is on disk
// (`non_monomorphic_reindex_layer`, which both depend on, reads
// it) — same routine-rebuild footing as the sibling annex itself,
// rather than leaving them to be built lazily on first
// `--shannon`/`--entropy-bias`/`--min-shared-family`/
// `--family-overlap` use.
tracing::info!("building entropy and quartet (both --no-ambiguity conventions) and family-overlap annexes");
self.ensure_family_annexes(&AnnexRequest::everything())?;
Ok(())
} }
fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> { fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> {
@@ -269,6 +269,21 @@ impl SiblingAnnex {
Ok(Self { mmap, n, path: path.to_path_buf() }) Ok(Self { mmap, n, path: path.to_path_buf() })
} }
/// The annex of the layer directory `layer_dir`. Same as
/// [`open`](Self::open), except that an absent file is reported with its
/// path and the way to build it, rather than a bare `NotFound` — the
/// state an index is in right after `migrate-mphf` set its annexes aside.
pub fn open_in(layer_dir: &Path) -> io::Result<Self> {
let path = layer_dir.join(super::ANNEX_FILE_NAME);
Self::open(&path).map_err(|e| match e.kind() {
io::ErrorKind::NotFound => io::Error::new(
io::ErrorKind::NotFound,
format!("no sibling annex at {} — build it with `phylo --sibling-annex`", path.display()),
),
_ => e,
})
}
pub fn path(&self) -> &Path { &self.path } pub fn path(&self) -> &Path { &self.path }
pub fn len(&self) -> usize { self.n } pub fn len(&self) -> usize { self.n }
pub fn is_empty(&self) -> bool { self.n == 0 } pub fn is_empty(&self) -> bool { self.n == 0 }
+17 -11
View File
@@ -1,13 +1,12 @@
use std::collections::HashMap;
use obikseq::CanonicalKmer; use obikseq::CanonicalKmer;
use obiread::record::SeqRecord; use obiread::record::SeqRecord;
use obiskbuilder::SuperKmerIter; use obiskbuilder::SuperKmerIter;
use crate::partition_kmers::PartitionKmers;
use crate::query_layer::KmerDesc; use crate::query_layer::KmerDesc;
/// A batch of query sequences, with k-mers deduplicated directly (not just at /// A batch of query sequences, with k-mers grouped by canonical k-mer (not just
/// the superkmer level) and pre-split by partition. /// at the superkmer level) and pre-split by partition.
/// ///
/// Superkmer *construction* (`SuperKmerIter`) is still required — it's the /// Superkmer *construction* (`SuperKmerIter`) is still required — it's the
/// mechanism that computes minimizers and partition routing — but the dedup /// mechanism that computes minimizers and partition routing — but the dedup
@@ -23,8 +22,8 @@ pub(crate) struct QueryBatch {
pub(crate) seqs: Vec<Vec<u8>>, pub(crate) seqs: Vec<Vec<u8>>,
/// Total kmer count per sequence (used for `--detail` coverage allocation). /// Total kmer count per sequence (used for `--detail` coverage allocation).
pub(crate) n_kmers: Vec<u32>, pub(crate) n_kmers: Vec<u32>,
/// Deduplicated k-mer occurrences, one map per partition. /// K-mer occurrences grouped by k-mer, one entry per partition.
pub(crate) by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>>, pub(crate) by_partition: Vec<PartitionKmers>,
} }
impl QueryBatch { impl QueryBatch {
@@ -41,20 +40,20 @@ impl QueryBatch {
let mut seqs = Vec::with_capacity(records.len()); let mut seqs = Vec::with_capacity(records.len());
let mut n_kmers = Vec::with_capacity(records.len()); let mut n_kmers = Vec::with_capacity(records.len());
let mask = (n_partitions as u64) - 1; let mask = (n_partitions as u64) - 1;
let mut by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>> = let mut occurrences: Vec<Vec<(CanonicalKmer, KmerDesc)>> =
(0..n_partitions).map(|_| HashMap::new()).collect(); (0..n_partitions).map(|_| Vec::new()).collect();
for (seq_idx, record) in records.into_iter().enumerate() { for (seq_idx, record) in records.into_iter().enumerate() {
let mut kmer_offset = 0u32; let mut kmer_offset = 0u32;
for rsk in SuperKmerIter::new(&record.normalized, k, level_max, theta) { for rsk in SuperKmerIter::new(&record.normalized, k, level_max, theta) {
let part_idx = (rsk.minimizer().seq_hash() & mask) as usize; let part_idx = (rsk.minimizer().seq_hash() & mask) as usize;
let map = &mut by_partition[part_idx]; let part = &mut occurrences[part_idx];
for (j, kmer) in rsk.superkmer().iter_canonical_kmers().enumerate() { for (j, kmer) in rsk.superkmer().iter_canonical_kmers().enumerate() {
map.entry(kmer).or_default().push(KmerDesc { part.push((kmer, KmerDesc {
seq_idx: seq_idx as u32, seq_idx: seq_idx as u32,
pos: kmer_offset + j as u32, pos: kmer_offset + j as u32,
}); }));
} }
let n = (rsk.seql() - k + 1) as u32; let n = (rsk.seql() - k + 1) as u32;
kmer_offset += n; kmer_offset += n;
@@ -65,6 +64,13 @@ impl QueryBatch {
n_kmers.push(kmer_offset); n_kmers.push(kmer_offset);
} }
// One partition at a time: each 16-byte-per-occurrence buffer is freed
// as soon as its compact form exists.
let by_partition = occurrences
.into_iter()
.map(PartitionKmers::from_occurrences)
.collect();
Self { Self {
ids, ids,
seqs, seqs,
+6 -31
View File
@@ -3,7 +3,6 @@ use std::time::Instant;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikindex::GenomeInfo; use obikindex::GenomeInfo;
use obikrope::Rope; use obikrope::Rope;
use obikseq::CanonicalKmer;
use obiread::record::parse_chunk; use obiread::record::parse_chunk;
use tracing::debug; use tracing::debug;
@@ -62,37 +61,13 @@ pub fn process_chunk(
let batch = QueryBatch::from_records(records, k, 6, 0.7, n_partitions); let batch = QueryBatch::from_records(records, k, 6, 0.7, n_partitions);
let n_seqs = batch.ids.len(); let n_seqs = batch.ids.len();
// Estimate QueryBatch::by_partition's actual memory footprint: the // `by_partition` is alive for the entire process_chunk call (never
// k-mer-level dedup map — one HashMap<CanonicalKmer, Vec<KmerDesc>> per // drained, only iterated by reference): this is its footprint for the
// partition, sized by *unique* k-mers, not shrunk by dedup. On real // whole chunk lifetime. Measured by allocated capacity, not logical length.
// workloads with a low intra-chunk duplication rate this can dwarf every
// other per-chunk structure, including the sparse Findere ones logged
// further down. Measured by allocated capacity, not logical length, to
// reflect real memory pressure (HashMap/Vec growth slack) — `by_partition`
// is alive for the entire process_chunk call (never drained, only
// iterated by reference), so this is its footprint for the whole chunk
// lifetime, not a transient.
let hashmap_slot_bytes = (std::mem::size_of::<CanonicalKmer>()
+ std::mem::size_of::<Vec<crate::query_layer::KmerDesc>>()
+ 1) as u64; // +1 ≈ hashbrown control byte per slot
let by_partition_map_bytes: u64 = batch
.by_partition
.iter()
.map(|m| m.capacity() as u64 * hashmap_slot_bytes)
.sum();
let by_partition_desc_bytes: u64 = batch
.by_partition
.iter()
.flat_map(|m| m.values())
.map(|v| v.capacity() as u64 * std::mem::size_of::<crate::query_layer::KmerDesc>() as u64)
.sum();
let by_partition_bytes = by_partition_map_bytes + by_partition_desc_bytes;
debug!( debug!(
n_unique_kmers_total = batch.by_partition.iter().map(|m| m.len() as u64).sum::<u64>(), n_unique_kmers_total = batch.by_partition.iter().map(|p| p.len() as u64).sum::<u64>(),
by_partition_map_bytes, n_occurrences_total = batch.by_partition.iter().map(|p| p.n_occurrences() as u64).sum::<u64>(),
by_partition_desc_bytes, by_partition_bytes = batch.by_partition.iter().map(|p| p.allocated_bytes() as u64).sum::<u64>(),
by_partition_bytes,
chunk_bytes, chunk_bytes,
"by_partition memory retained" "by_partition memory retained"
); );
+2
View File
@@ -5,6 +5,7 @@
//! the data model, never the reverse), same pattern as `obikindexer` for the //! the data model, never the reverse), same pattern as `obikindexer` for the
//! build side. //! build side.
mod partition_kmers;
mod query_layer; mod query_layer;
mod batch; mod batch;
mod chunk; mod chunk;
@@ -12,5 +13,6 @@ mod findere;
mod smer_index; mod smer_index;
mod output; mod output;
pub use partition_kmers::PartitionKmers;
pub use query_layer::{KmerDesc, QueryHit, QueryPartition, QueryStats}; pub use query_layer::{KmerDesc, QueryHit, QueryPartition, QueryStats};
pub use chunk::process_chunk; pub use chunk::process_chunk;
+82
View File
@@ -0,0 +1,82 @@
//! The k-mers of one query chunk routed to one partition, grouped by k-mer.
//!
//! Replaces a `HashMap<CanonicalKmer, Vec<KmerDesc>>`: with genomic reads
//! almost every k-mer occurs once, so the map paid a table slot, a growth
//! slack and a separate heap `Vec` for each unique k-mer (~70 bytes measured)
//! for a dedup that removes under 10 % of the occurrences. Here the
//! occurrences are sorted by k-mer and stored as three flat arrays — one
//! 8-byte k-mer and one 4-byte offset per unique k-mer, one 8-byte
//! [`KmerDesc`] per occurrence — about 20 bytes per unique k-mer, no
//! per-k-mer allocation.
use obikseq::CanonicalKmer;
use crate::query_layer::KmerDesc;
/// Unique k-mers of one partition (sorted by raw value) and, for each, the
/// slice of its occurrences (sorted by `(seq_idx, pos)`, i.e. query order).
pub struct PartitionKmers {
kmers: Vec<CanonicalKmer>,
/// `descs[starts[i]..starts[i + 1]]` are the occurrences of `kmers[i]`.
starts: Vec<u32>,
descs: Vec<KmerDesc>,
}
impl PartitionKmers {
pub fn empty() -> Self {
Self { kmers: Vec::new(), starts: vec![0], descs: Vec::new() }
}
/// Group `occurrences` by k-mer. Consumes the (16 bytes per occurrence)
/// input; after the call only the compact arrays remain.
pub fn from_occurrences(mut occurrences: Vec<(CanonicalKmer, KmerDesc)>) -> Self {
occurrences.sort_unstable_by_key(|(kmer, d)| (kmer.raw(), d.seq_idx, d.pos));
let mut kmers = Vec::new();
let mut starts = Vec::new();
let mut descs = Vec::with_capacity(occurrences.len());
for (i, (kmer, desc)) in occurrences.iter().enumerate() {
if kmers.last() != Some(kmer) {
kmers.push(*kmer);
starts.push(offset(i));
}
descs.push(*desc);
}
starts.push(offset(descs.len()));
kmers.shrink_to_fit();
starts.shrink_to_fit();
Self { kmers, starts, descs }
}
/// Number of unique k-mers.
pub fn len(&self) -> usize {
self.kmers.len()
}
pub fn is_empty(&self) -> bool {
self.kmers.is_empty()
}
/// Total number of occurrences (before grouping).
pub fn n_occurrences(&self) -> usize {
self.descs.len()
}
/// `(kmer, its occurrences)` for every unique k-mer, in k-mer order.
pub fn iter(&self) -> impl Iterator<Item = (CanonicalKmer, &[KmerDesc])> + '_ {
self.kmers.iter().enumerate().map(|(i, &kmer)| {
(kmer, &self.descs[self.starts[i] as usize..self.starts[i + 1] as usize])
})
}
/// Heap bytes held (by allocated capacity).
pub fn allocated_bytes(&self) -> usize {
self.kmers.capacity() * std::mem::size_of::<CanonicalKmer>()
+ self.starts.capacity() * std::mem::size_of::<u32>()
+ self.descs.capacity() * std::mem::size_of::<KmerDesc>()
}
}
fn offset(i: usize) -> u32 {
u32::try_from(i).expect("more than 2^32 k-mer occurrences in one partition of one chunk — lower --chunk-size")
}
+42 -8
View File
@@ -3,6 +3,8 @@ use std::collections::HashMap;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikseq::CanonicalKmer; use obikseq::CanonicalKmer;
use crate::partition_kmers::PartitionKmers;
// ── KmerDesc — one occurrence of a k-mer in the query batch ────────────────── // ── KmerDesc — one occurrence of a k-mer in the query batch ──────────────────
/// Describes one occurrence of a (deduplicated) k-mer in the query batch: /// Describes one occurrence of a (deduplicated) k-mer in the query batch:
@@ -100,7 +102,7 @@ pub trait QueryPartition {
fn query_partition_with<F>( fn query_partition_with<F>(
&self, &self,
part_idx: usize, part_idx: usize,
kmers: &HashMap<CanonicalKmer, Vec<KmerDesc>>, kmers: &PartitionKmers,
n_genomes: usize, n_genomes: usize,
on_event: F, on_event: F,
) -> QueryStats ) -> QueryStats
@@ -112,7 +114,7 @@ impl QueryPartition for IndexCache {
fn query_partition_with<F>( fn query_partition_with<F>(
&self, &self,
part_idx: usize, part_idx: usize,
kmers: &HashMap<CanonicalKmer, Vec<KmerDesc>>, kmers: &PartitionKmers,
n_genomes: usize, n_genomes: usize,
mut on_event: F, mut on_event: F,
) -> QueryStats ) -> QueryStats
@@ -131,19 +133,51 @@ impl QueryPartition for IndexCache {
}; };
// ── Stage 1: MPHF-only pass, bucket hits by (layer_idx, slot) ──────── // ── Stage 1: MPHF-only pass, bucket hits by (layer_idx, slot) ────────
let mut by_layer: Vec<HashMap<usize, &Vec<KmerDesc>>> = let mut by_layer: Vec<HashMap<usize, &[KmerDesc]>> =
(0..n_layer).map(|_| HashMap::new()).collect(); (0..n_layer).map(|_| HashMap::new()).collect();
for (kmer, descs) in kmers { // Layer-major over fixed-size chunks of the (unordered) kmer map:
stats.n_unique_kmers += 1; // each layer sees the chunk's still-unresolved kmers as one batch, so
// its MPHF lookups are prefetched together; a kmer found in an
// earlier layer never reaches the later ones (first hit wins, as
// before). Chunked, not collected — the map is never copied whole.
const CHUNK: usize = 4096;
let mut entries = kmers.iter();
let mut pending: Vec<(CanonicalKmer, &[KmerDesc])> = Vec::with_capacity(CHUNK);
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(CHUNK);
let mut found: Vec<Option<usize>> = Vec::with_capacity(CHUNK);
loop {
pending.clear();
pending.extend(entries.by_ref().take(CHUNK));
if pending.is_empty() {
break;
}
stats.n_unique_kmers += pending.len();
for l in 0..n_layer { for l in 0..n_layer {
stats.n_mphf_calls += 1; if pending.is_empty() {
if let Some(slot) = self.find_in_layer(part_idx, l, *kmer) { break;
}
let layer = self
.get_layer(part_idx, l)
.expect("layer within n_layer(part_idx)");
batch.clear();
batch.extend(pending.iter().map(|&(k, _)| k));
stats.n_mphf_calls += batch.len();
layer.find_slot_batch(&batch, &mut found);
let mut i = 0;
pending.retain(|&(_, descs)| {
let hit = found[i];
i += 1;
match hit {
Some(slot) => {
by_layer[l].insert(slot, descs); by_layer[l].insert(slot, descs);
on_event(QueryHit::Found(descs)); on_event(QueryHit::Found(descs));
stats.n_hits += 1; stats.n_hits += 1;
break; false
} }
None => true,
}
});
} }
} }
+19 -5
View File
@@ -87,8 +87,8 @@ fn duplicated_sequence_broadcasts_to_both_seq_indices() {
let batch = batch_from_fasta(&fasta, K, 1); let batch = batch_from_fasta(&fasta, K, 1);
let shared = batch.by_partition[0] let shared = batch.by_partition[0]
.values() .iter()
.find(|descs| descs.iter().any(|d| d.seq_idx == 0) && descs.iter().any(|d| d.seq_idx == 1)); .find(|(_, descs)| descs.iter().any(|d| d.seq_idx == 0) && descs.iter().any(|d| d.seq_idx == 1));
assert!( assert!(
shared.is_some(), shared.is_some(),
@@ -119,9 +119,23 @@ fn partition_routing_is_a_pure_function_of_the_kmer() {
// No k-mer key appears in more than one partition's map. // No k-mer key appears in more than one partition's map.
let mut seen: std::collections::HashSet<CanonicalKmer> = std::collections::HashSet::new(); let mut seen: std::collections::HashSet<CanonicalKmer> = std::collections::HashSet::new();
for map in &batch.by_partition { for part in &batch.by_partition {
for kmer in map.keys() { for (kmer, _) in part.iter() {
assert!(seen.insert(*kmer), "k-mer routed to more than one partition"); assert!(seen.insert(kmer), "k-mer routed to more than one partition");
} }
} }
} }
#[test]
fn occurrences_of_a_kmer_come_out_in_query_order() {
// Grouping sorts occurrences; within a k-mer they must stay ordered by
// (seq_idx, pos) — the order the former insertion-ordered `Vec` had, which
// downstream output ordering relied on.
let fasta = format!(">r1\n{SEQ}\n>r2\n{SEQ}\n");
let batch = batch_from_fasta(&fasta, K, 1);
for (_, descs) in batch.by_partition[0].iter() {
assert!(descs.windows(2).all(|w| (w[0].seq_idx, w[0].pos) < (w[1].seq_idx, w[1].pos)));
}
let occ: usize = batch.by_partition[0].iter().map(|(_, d)| d.len()).sum();
assert_eq!(occ as u64, total_occurrences(&batch));
}
+2 -3
View File
@@ -64,12 +64,11 @@ fn query_partition_with_missing_index_dir_returns_default_stats() {
let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index"); let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index");
let cache = IndexCache::new(Arc::new(index), Some(vec![0])); let cache = IndexCache::new(Arc::new(index), Some(vec![0]));
let mut kmers: HashMap<CanonicalKmer, Vec<KmerDesc>> = HashMap::new();
// Any well-formed canonical k-mer works here — the call must return // Any well-formed canonical k-mer works here — the call must return
// before ever attempting an MPHF lookup, since the partition has zero // before ever attempting an MPHF lookup, since the partition has zero
// layers. // layers.
let kmer = CanonicalKmer::from_raw_unchecked(0u64); let kmer = CanonicalKmer::from_raw_unchecked(0u64);
kmers.insert(kmer, vec![KmerDesc { seq_idx: 0, pos: 0 }]); let kmers = PartitionKmers::from_occurrences(vec![(kmer, KmerDesc { seq_idx: 0, pos: 0 })]);
let stats = cache.query_partition_with(0, &kmers, 1, |_event| { let stats = cache.query_partition_with(0, &kmers, 1, |_event| {
panic!("on_event must not be called: no index was built"); panic!("on_event must not be called: no index was built");
@@ -94,7 +93,7 @@ fn query_partition_with_empty_kmers_is_a_noop() {
let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index"); let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index");
let cache = IndexCache::new(Arc::new(index), Some(vec![0])); let cache = IndexCache::new(Arc::new(index), Some(vec![0]));
let kmers: HashMap<CanonicalKmer, Vec<KmerDesc>> = HashMap::new(); let kmers = PartitionKmers::empty();
let stats = cache.query_partition_with(0, &kmers, 1, |_event| { let stats = cache.query_partition_with(0, &kmers, 1, |_event| {
panic!("on_event must not be called on an empty kmer map"); panic!("on_event must not be called on an empty kmer map");
}); });
+7 -2
View File
@@ -22,8 +22,13 @@ use crate::kmer::{CanonicalKmer, Kmer, KmerError, KLen, KmerLength, KmerOf, MLen
use crate::params::k; use crate::params::k;
use crate::revcomp_lookup::REVCOMP4; use crate::revcomp_lookup::REVCOMP4;
/// Maximum kmers per stored chunk. Enforces the u8 max-kmer-index field in the binary format. /// Maximum kmers per stored chunk.
pub const MAX_KMERS_PER_CHUNK: usize = 256; ///
/// 128 = 2^7, not the 256 the u8 length field of the binary format could
/// carry: the exact evidence of a layer (`obikindex::layer::evidence`) stores a
/// kmer's rank inside its unitig chunk on 7 bits, so every rank must be ≤ 127.
/// Longer unitigs are split into consecutive chunks at write time.
pub const MAX_KMERS_PER_CHUNK: usize = 128;
// ── PackedSeq ───────────────────────────────────────────────────────────────── // ── PackedSeq ─────────────────────────────────────────────────────────────────
+4 -4
View File
@@ -70,7 +70,7 @@ fn seql_roundtrip() {
fn binary_test_lengths(k: usize) -> Vec<usize> { fn binary_test_lengths(k: usize) -> Vec<usize> {
use crate::packed_seq::MAX_KMERS_PER_CHUNK; use crate::packed_seq::MAX_KMERS_PER_CHUNK;
// Only single-chunk lengths: seql in [k, MAX_KMERS_PER_CHUNK+k-1]. // Only single-chunk lengths: seql in [k, MAX_KMERS_PER_CHUNK+k-1].
(k..=k + 5).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect() (k..=k + 5).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect()
} }
#[test] #[test]
@@ -94,7 +94,7 @@ fn binary_split_roundtrip() {
use crate::packed_seq::MAX_KMERS_PER_CHUNK; use crate::packed_seq::MAX_KMERS_PER_CHUNK;
set_k(4); set_k(4);
let k = crate::params::k(); let k = crate::params::k();
// seql = MAX_KMERS_PER_CHUNK + k = 260 → n_kmers = 257 > 256 → 2 chunks // seql = MAX_KMERS_PER_CHUNK + k → n_kmers = MAX + 1 > MAX → 2 chunks
let seql = MAX_KMERS_PER_CHUNK + k; let seql = MAX_KMERS_PER_CHUNK + k;
let mut sk = SuperKmer::from_ascii(&make_seq(seql)); let mut sk = SuperKmer::from_ascii(&make_seq(seql));
sk.set_count(7); sk.set_count(7);
@@ -108,8 +108,8 @@ fn binary_split_roundtrip() {
assert_eq!(chunk0.count(), 7); assert_eq!(chunk0.count(), 7);
assert_eq!(chunk1.count(), 7); assert_eq!(chunk1.count(), 7);
// Chunks cover the original sequence with k-1 overlap — no kmer lost. // Chunks cover the original sequence with k-1 overlap — no kmer lost.
assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1); // 259 assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1);
assert_eq!(chunk1.seql(), k); // 4 (1 kmer) assert_eq!(chunk1.seql(), k); // 1 kmer
} }
#[test] #[test]
+1 -1
View File
@@ -164,7 +164,7 @@ mod tests {
use crate::packed_seq::MAX_KMERS_PER_CHUNK; use crate::packed_seq::MAX_KMERS_PER_CHUNK;
set_k(4); set_k(4);
let k = crate::params::k(); let k = crate::params::k();
let valid_lengths: Vec<usize> = (k..=9).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect(); let valid_lengths: Vec<usize> = (k..=9).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect();
for len in valid_lengths { for len in valid_lengths {
let u = Unitig::from_ascii(&make_seq(len)); let u = Unitig::from_ascii(&make_seq(len));
let mut buf = Vec::new(); let mut buf = Vec::new();
+1 -1
View File
@@ -25,7 +25,7 @@ pub use normalize::{normalize_fasta_chunk, normalize_fastq_chunk, normalize_sequ
pub use nucstream::{NucPage, NucPageCursor, open_nuc_stream}; pub use nucstream::{NucPage, NucPageCursor, open_nuc_stream};
pub use path_iterator::{PathIter, path_iter}; pub use path_iterator::{PathIter, path_iter};
pub use peakreader::PeekReader; pub use peakreader::PeekReader;
pub use xopen::xopen; pub use xopen::{estimated_decompressed_len, xopen};
/// Default read block size: 1 MiB. /// Default read block size: 1 MiB.
pub const DEFAULT_BLOCK_SIZE: usize = 1024 * 1024; pub const DEFAULT_BLOCK_SIZE: usize = 1024 * 1024;
+38
View File
@@ -0,0 +1,38 @@
use super::*;
use std::io::Write;
fn temp_file(name: &str, content: &[u8]) -> std::path::PathBuf {
let path = std::env::temp_dir().join(format!("obiread_xopen_{}_{name}", std::process::id()));
std::fs::File::create(&path).unwrap().write_all(content).unwrap();
path
}
#[test]
fn estimate_plain_file_is_its_length() {
let p = temp_file("plain", b">s\nACGT\n");
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(8));
std::fs::remove_file(p).unwrap();
}
#[test]
fn estimate_scales_compressed_files_by_magic_not_extension() {
// Only the magic bytes matter to the estimate (the content is never decoded).
let mut gz = vec![0x1f, 0x8b, 0x08, 0, 0, 0];
gz.resize(100, 0);
let p = temp_file("looks_plain.txt", &gz);
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(300));
std::fs::remove_file(p).unwrap();
let mut zst = vec![0x28, 0xb5, 0x2f, 0xfd, 0, 0];
zst.resize(50, 0);
let p = temp_file("zst", &zst);
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(200));
std::fs::remove_file(p).unwrap();
}
#[test]
fn estimate_is_unknown_for_streams_and_missing_files() {
assert_eq!(estimated_decompressed_len("-"), None);
assert_eq!(estimated_decompressed_len("https://example.org/x.fa.gz"), None);
assert_eq!(estimated_decompressed_len("/nonexistent/obiread/file"), None);
}
+35
View File
@@ -46,6 +46,37 @@ pub fn xopen(source: &str) -> io::Result<MimeTypeGuesser<Box<dyn Read + Send>>>
Ok(MimeTypeGuesser::new(open_raw(source)?)) Ok(MimeTypeGuesser::new(open_raw(source)?))
} }
/// Rough size, in bytes, of what [`xopen`] would yield for a local file:
/// its length, multiplied by a typical expansion factor when the magic bytes
/// say it is compressed (gzip ≈ 3×, as measured on FASTQ; bzip2, xz and zstd
/// taken as 4×). Meant for sizing work (chunk sizes), not for allocation —
/// the real ratio varies by a factor of two or more with the content.
///
/// `None` when the size cannot be known without reading the stream: stdin
/// (`-`), an HTTP(S) URL, or a file that cannot be opened.
pub fn estimated_decompressed_len(source: &str) -> Option<u64> {
if source == "-" || source.starts_with("http://") || source.starts_with("https://") {
return None;
}
let expanded = expand_tilde(source);
let mut file = File::open(expanded.as_ref()).ok()?;
let len = file.metadata().ok()?.len();
let mut magic = [0u8; 6];
let n = file.read(&mut magic).ok()?;
let magic = &magic[..n];
let factor = if magic.starts_with(&[0x1f, 0x8b]) {
3
} else if magic.starts_with(b"BZh")
|| magic.starts_with(&[0xfd, b'7', b'z', b'X', b'Z', 0x00])
|| magic.starts_with(&[0x28, 0xb5, 0x2f, 0xfd])
{
4
} else {
1
};
Some(len * factor)
}
// ── internal helpers ────────────────────────────────────────────────────────── // ── internal helpers ──────────────────────────────────────────────────────────
fn http_reader(url: &str) -> io::Result<Box<dyn Read + Send>> { fn http_reader(url: &str) -> io::Result<Box<dyn Read + Send>> {
@@ -71,3 +102,7 @@ fn expand_tilde(path: &str) -> std::borrow::Cow<'_, str> {
} }
std::borrow::Cow::Borrowed(path) std::borrow::Cow::Borrowed(path)
} }
#[cfg(test)]
#[path = "tests/xopen.rs"]
mod tests;
+4 -2
View File
@@ -28,8 +28,10 @@ impl SKFileMeta {
/// Read the sidecar for `sk_path`. Returns `None` if the sidecar is absent. /// Read the sidecar for `sk_path`. Returns `None` if the sidecar is absent.
pub fn read(sk_path: &Path) -> SKResult<Option<Self>> { pub fn read(sk_path: &Path) -> SKResult<Option<Self>> {
let path = Self::sidecar_path(sk_path); let path = Self::sidecar_path(sk_path);
match std::fs::File::open(&path) { // Read whole, parse from memory: `from_reader` on a bare `File` is one
Ok(f) => Ok(Some(serde_json::from_reader(f).map_err(|e| SKError::InvalidData { // syscall per byte.
match std::fs::read(&path) {
Ok(bytes) => Ok(Some(serde_json::from_slice(&bytes).map_err(|e| SKError::InvalidData {
context: "SKFileMeta sidecar", context: "SKFileMeta sidecar",
detail: e.to_string(), detail: e.to_string(),
})?)), })?)),
+6 -2
View File
@@ -19,8 +19,12 @@ impl SKFileReader {
/// Open a file for reading. Format is auto-detected from magic bytes. /// Open a file for reading. Format is auto-detected from magic bytes.
pub fn open<P: AsRef<Path>>(path: P) -> SKResult<Self> { pub fn open<P: AsRef<Path>>(path: P) -> SKResult<Self> {
let path = path.as_ref().to_owned(); let path = path.as_ref().to_owned();
let (reader, _fmt) = // Buffered: for an uncompressed file niffler hands back the bare
niffler::send::get_reader(Box::new(File::open(&path)?))?; // `File` (after the sniffed bytes), and `read_superkmer` reads each
// field with its own small `read_exact` — one syscall each otherwise.
let (reader, _fmt) = niffler::send::get_reader(Box::new(
std::io::BufReader::with_capacity(1 << 16, File::open(&path)?),
))?;
Ok(Self { path, reader }) Ok(Self { path, reader })
} }
+3 -1
View File
@@ -104,7 +104,9 @@ fn roundtrip_single() {
fn roundtrip_all_lengths() { fn roundtrip_all_lengths() {
setup(); setup();
let bases: Vec<u8> = (0..300).map(|i| b"ACGT"[i % 4]).collect(); let bases: Vec<u8> = (0..300).map(|i| b"ACGT"[i % 4]).collect();
for len in (TEST_K..=19).chain([255, 256, 257]) { // single-chunk lengths only: seql ≤ MAX_KMERS_PER_CHUNK + k - 1
let max_len = obikseq::MAX_KMERS_PER_CHUNK + TEST_K - 1;
for len in (TEST_K..=19).chain([max_len - 2, max_len - 1, max_len]) {
let sk = make_sk(&bases[..len]); let sk = make_sk(&bases[..len]);
let mut buf = Vec::new(); let mut buf = Vec::new();
sk.write_to_binary(&mut buf).unwrap(); sk.write_to_binary(&mut buf).unwrap();
+27 -13
View File
@@ -222,31 +222,45 @@ fn iter_indexed_chunk_id_and_rank_two_chunks() {
#[test] #[test]
fn short_unitig_not_split() { fn short_unitig_not_split() {
// seql=259 → n_kmers=256 = MAX_KMERS_PER_CHUNK → no split // seql=131 → n_kmers=128 = MAX_KMERS_PER_CHUNK → no split
set_k(4); set_k(4);
let seq: Vec<u8> = (0..259_usize).map(|i| b"ACGT"[i % 4]).collect(); let seq: Vec<u8> = (0..131_usize).map(|i| b"ACGT"[i % 4]).collect();
let (_dir, r) = write_read(&[&seq]); let (_dir, r) = write_read(&[&seq]);
assert_eq!(r.len(), 1); assert_eq!(r.len(), 1);
assert_eq!(r.seql(0), 259); assert_eq!(r.seql(0), 131);
} }
#[test] #[test]
fn long_unitig_split_no_kmer_lost() { fn long_unitig_split_no_kmer_lost() {
// seql=260 → n_kmers=257 > MAX_KMERS_PER_CHUNK(256) → 2 chunks // seql=132 → n_kmers=129 > MAX_KMERS_PER_CHUNK(128) → 2 chunks
// chunk_nucl=259, stride=256 // chunk_nucl=131, stride=128
// Chunk 0: nucl 0..259 (259 nucl, 256 kmers) // Chunk 0: nucl 0..131 (131 nucl, 128 kmers)
// Chunk 1: nucl 256..260 (4 nucl, 1 kmer) // Chunk 1: nucl 128..132 (4 nucl, 1 kmer)
set_k(4); set_k(4);
let seq: Vec<u8> = (0..260_usize).map(|i| b"ACGT"[i % 4]).collect(); let seq: Vec<u8> = (0..132_usize).map(|i| b"ACGT"[i % 4]).collect();
let (_dir, r) = write_read(&[&seq]); let (_dir, r) = write_read(&[&seq]);
assert_eq!(r.len(), 2); assert_eq!(r.len(), 2);
assert_eq!(r.seql(0), 259); assert_eq!(r.seql(0), 131);
assert_eq!(r.seql(1), 4); assert_eq!(r.seql(1), 4);
// k-1=3 nucleotide overlap → 0 kmers duplicated, 0 kmers lost. // k-1=3 nucleotide overlap → 0 kmers duplicated, 0 kmers lost.
// Last kmer of chunk 0 = original nucl 255..259. // Last kmer of chunk 0 = original nucl 127..131.
assert!(r.verify_canonical_kmer(0, 255, canonical_of(&seq[255..259]))); assert!(r.verify_canonical_kmer(0, 127, canonical_of(&seq[127..131])));
// First kmer of chunk 1 = original nucl 256..260 — a different, adjacent kmer. // First kmer of chunk 1 = original nucl 128..132 — a different, adjacent kmer.
assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[256..260]))); assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[128..132])));
}
#[test]
fn chunk_ranks_fit_the_7_bit_evidence_field() {
// The exact evidence of a layer stores a kmer's rank in its chunk on 7
// bits (obikindex::layer::evidence): a unitig long enough to need several
// chunks must never yield a rank above 127 — a larger one would be
// silently truncated there and the kmer would no longer verify.
set_k(4);
let seq: Vec<u8> = (0..1000_usize).map(|i| b"ACGT"[(i * i + i / 3) % 4]).collect();
let (_dir, r) = write_read(&[&seq]);
assert!(r.len() > 1, "a 997-kmer unitig must be split into several chunks");
let max_rank = r.iter_indexed_canonical_kmers().map(|(_, _, rank)| rank).max().unwrap();
assert!(max_rank <= 127, "rank {max_rank} does not fit 7 bits");
} }
// ── block_bits parametrisation ──────────────────────────────────────────────── // ── block_bits parametrisation ────────────────────────────────────────────────
+50 -37
View File
@@ -24,7 +24,7 @@ use super::{DEFAULT_BLOCK_BITS, MAGIC, idx_path};
/// correct but slower. /// correct but slower.
pub struct UnitigFileReader { pub struct UnitigFileReader {
mmap: Mmap, mmap: Mmap,
block_offsets: Vec<u32>, block_offsets: BlockOffsets,
n_unitigs: usize, n_unitigs: usize,
n_kmers: usize, n_kmers: usize,
k: usize, k: usize,
@@ -64,7 +64,7 @@ impl UnitigFileReader {
Ok(Self { Ok(Self {
mmap, mmap,
block_offsets: Vec::new(), block_offsets: BlockOffsets::none(),
n_unitigs, n_unitigs,
n_kmers, n_kmers,
k, k,
@@ -108,7 +108,7 @@ impl UnitigFileReader {
} }
#[inline] #[inline]
pub fn has_direct_access(&self) -> bool { pub fn has_direct_access(&self) -> bool {
!self.block_offsets.is_empty() !self.block_offsets.is_none()
} }
/// Byte offset of record `i` in the mmap. /// Byte offset of record `i` in the mmap.
@@ -119,11 +119,11 @@ impl UnitigFileReader {
fn chunk_start(&self, i: usize) -> usize { fn chunk_start(&self, i: usize) -> usize {
if self.has_direct_access() { if self.has_direct_access() {
if self.block_bits == 0 { if self.block_bits == 0 {
return self.block_offsets[i] as usize; return self.block_offsets.get(i) as usize;
} }
let block = i >> self.block_bits; let block = i >> self.block_bits;
let rem = i & self.mask; let rem = i & self.mask;
let mut offset = self.block_offsets[block] as usize; let mut offset = self.block_offsets.get(block) as usize;
for _ in 0..rem { for _ in 0..rem {
let seql_minus_k = self.mmap[offset] as usize; let seql_minus_k = self.mmap[offset] as usize;
offset += 1 + (seql_minus_k + self.k + 3) / 4; offset += 1 + (seql_minus_k + self.k + 3) / 4;
@@ -292,25 +292,54 @@ impl UnitigFileReader {
} }
} }
fn read_idx(path: &Path) -> SKResult<(usize, usize, u8, Vec<u32>)> { /// Block offsets of a `.idx` file, read in place from its memory map: nothing
let data = std::fs::read(path).map_err(SKError::Io)?; /// is copied to the heap, and every reader of the same `.idx` shares the same
let mut pos = 0; /// page-cache pages (a copy was ~30 MB per large layer per reader).
struct BlockOffsets {
/// The whole `.idx` file; `None` for a sequential reader (no `.idx`).
mmap: Option<Mmap>,
}
let magic_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated { impl BlockOffsets {
context: "unitig index: magic", fn none() -> Self {
})?; Self { mmap: None }
}
fn is_none(&self) -> bool {
self.mmap.is_none()
}
/// The `i`-th offset (little-endian `u32` after the fixed header).
/// `read_idx` has checked that the file holds every offset the header
/// announces, so `i` within `0..=n_blocks` cannot run past the map.
#[inline]
fn get(&self, i: usize) -> u32 {
let at = IDX_HEADER_SIZE + 4 * i;
let map = self.mmap.as_ref().expect("direct-access reader has its .idx map");
u32::from_le_bytes(map[at..at + 4].try_into().unwrap())
}
}
/// magic (4) + block_bits (4) + n_unitigs (4) + n_kmers (8).
const IDX_HEADER_SIZE: usize = 20;
fn read_idx(path: &Path) -> SKResult<(usize, usize, u8, BlockOffsets)> {
let file = File::open(path).map_err(SKError::Io)?;
let len = file.metadata().map_err(SKError::Io)?.len() as usize;
if len < IDX_HEADER_SIZE {
return Err(SKError::Truncated { context: "unitig index: header" });
}
let data = unsafe { Mmap::map(&file).map_err(SKError::Io)? };
let magic_bytes = &data[0..4];
if magic_bytes != &MAGIC { if magic_bytes != &MAGIC {
return Err(SKError::BadMagic { return Err(SKError::BadMagic {
expected: "UIX3", expected: "UIX3",
got: magic_bytes.try_into().unwrap(), got: magic_bytes.try_into().unwrap(),
}); });
} }
pos += 4;
let bb_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated { let block_bits_u32 = u32::from_le_bytes(data[4..8].try_into().unwrap());
context: "unitig index: block_bits",
})?;
let block_bits_u32 = u32::from_le_bytes(bb_bytes.try_into().unwrap());
if block_bits_u32 > 31 { if block_bits_u32 > 31 {
return Err(SKError::InvalidData { return Err(SKError::InvalidData {
context: "unitig index", context: "unitig index",
@@ -318,31 +347,15 @@ fn read_idx(path: &Path) -> SKResult<(usize, usize, u8, Vec<u32>)> {
}); });
} }
let block_bits = block_bits_u32 as u8; let block_bits = block_bits_u32 as u8;
pos += 4; let n_unitigs = u32::from_le_bytes(data[8..12].try_into().unwrap()) as usize;
let n_kmers = u64::from_le_bytes(data[12..20].try_into().unwrap()) as usize;
let n_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated {
context: "unitig index: n_unitigs",
})?;
let n_unitigs = u32::from_le_bytes(n_bytes.try_into().unwrap()) as usize;
pos += 4;
let nk_bytes = data.get(pos..pos + 8).ok_or(SKError::Truncated {
context: "unitig index: n_kmers",
})?;
let n_kmers = u64::from_le_bytes(nk_bytes.try_into().unwrap()) as usize;
pos += 8;
let block_size = 1usize << block_bits; let block_size = 1usize << block_bits;
let n_blocks = (n_unitigs + block_size - 1) >> block_bits; let n_blocks = (n_unitigs + block_size - 1) >> block_bits;
let n_offsets = n_blocks + 1; let n_offsets = n_blocks + 1;
let mut block_offsets = Vec::with_capacity(n_offsets); if len < IDX_HEADER_SIZE + 4 * n_offsets {
for _ in 0..n_offsets { return Err(SKError::Truncated { context: "unitig index: block_offsets" });
let off_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated {
context: "unitig index: block_offsets",
})?;
block_offsets.push(u32::from_le_bytes(off_bytes.try_into().unwrap()));
pos += 4;
} }
Ok((n_unitigs, n_kmers, block_bits, block_offsets)) Ok((n_unitigs, n_kmers, block_bits, BlockOffsets { mmap: Some(data) }))
} }
+5 -1
View File
@@ -3,6 +3,9 @@
mod budget; mod budget;
mod lock; mod lock;
mod lock_slot;
#[cfg(test)]
mod lock_tests;
mod numa; mod numa;
mod profiler; mod profiler;
mod progress; mod progress;
@@ -10,7 +13,8 @@ mod resources;
mod stage; mod stage;
pub use budget::MemoryBudget; pub use budget::MemoryBudget;
pub use lock::DirLock; pub use lock::{DirLock, LockMode};
pub use lock_slot::LockSlot;
pub use numa::PartitionRunner; pub use numa::PartitionRunner;
pub use profiler::start as start_profiler; pub use profiler::start as start_profiler;
pub use progress::{Progress, TracedBar, progress_bar, spinner}; pub use progress::{Progress, TracedBar, progress_bar, spinner};
+188 -38
View File
@@ -1,56 +1,206 @@
use std::fs::File;
use std::io;
use std::path::{Path, PathBuf};
use std::time::Duration;
use tracing::info; use tracing::info;
/// Exclusive advisory lock on an index directory, held for the duration of /// Advisory readers/writer lock on an index directory, shared between
/// any command that writes into an already-existing index (building the /// processes (and between independent handles of one process).
/// sibling annex, merging into a destination, filtering/selecting in place,
/// ...). Two such commands racing on the same directory can otherwise
/// corrupt each other's writes with no error from either side.
/// ///
/// Only the directory actually being *written to* needs a lock — a command /// - **Shared** ([`acquire_shared`](DirLock::acquire_shared)): held by
/// like `merge` that reads several source indexes to build one destination /// anything that reads the index (`query`, `dump`, a `merge` source, a
/// only needs to lock the destination. /// `phylo` run, ...). Any number of shared holders coexist.
/// - **Exclusive** ([`acquire`](DirLock::acquire)): held by anything that
/// writes into an already-existing index (building annexes, `convert`,
/// `pack`, `annotate`, ...). It waits for every shared holder to leave and
/// keeps others out while held. Two writers racing on one directory can
/// otherwise corrupt each other's files with no error from either side,
/// and a reader would see files vanish under it.
/// ///
/// Uses the OS's advisory file lock (`flock` on Unix, `LockFileEx` on /// Only the directory actually being *written to* needs an exclusive lock —
/// Windows) via `std::fs::File::lock`/`try_lock`, not a hand-rolled PID /// `merge` holds shared locks on its sources and an exclusive one on its
/// file: the OS releases it automatically on process exit, including a /// destination.
/// crash — no stale-lock cleanup logic needed. ///
/// # How it is built
///
/// Each mode is one OS advisory file lock (`flock` on Unix, `LockFileEx` on
/// Windows) on `.obikmer.lock`, so a crash releases it with no stale-lock
/// cleanup. On top of that, **every transition** (acquire, release into
/// another mode, promote, demote) happens while holding a *gate* — a second
/// lock file, `.obikmer.gate`, taken for the length of one non-blocking
/// attempt and never while waiting. A blocked request therefore releases the
/// gate, sleeps, and tries again (polling, 20 ms growing to 500 ms), instead
/// of sleeping in the kernel while holding it. Because nobody else can touch
/// the lock state while the gate is held, a shared → exclusive
/// [promotion](DirLock::promote) cannot be overtaken by another writer, which
/// the OS's own `flock` conversion does not guarantee.
///
/// There is no writer preference: a steady stream of readers can delay an
/// exclusive request.
#[derive(Debug)] #[derive(Debug)]
pub struct DirLock { pub struct DirLock {
_file: std::fs::File, file: File,
dir: PathBuf,
mode: LockMode,
} }
impl DirLock { #[derive(Debug, Clone, Copy, PartialEq, Eq)]
/// Block until the exclusive lock on `dir` is acquired (creating `dir` pub enum LockMode {
/// and the lock file within it if needed). Logs once if the wait is Shared,
/// non-trivial, so a blocked command doesn't look silently hung. Exclusive,
pub fn acquire(dir: &std::path::Path) -> std::io::Result<Self> {
let file = Self::open_lock_file(dir)?;
if file.try_lock().is_err() {
info!(dir = %dir.display(), "waiting for another obikmer process to release this index");
file.lock()?;
}
Ok(Self { _file: file })
} }
/// Non-blocking probe: `Ok(Some(lock))` if `dir`'s lock was free and is const LOCK_FILE: &str = ".obikmer.lock";
/// now held by the returned `DirLock`, `Ok(None)` if another process const GATE_FILE: &str = ".obikmer.gate";
/// currently holds it (not an error — contention is the expected, const CLAIM_FILE: &str = ".obikmer.promote";
/// common case this exists to distinguish from a genuine I/O failure). const POLL_MIN: Duration = Duration::from_millis(20);
pub fn try_acquire(dir: &std::path::Path) -> std::io::Result<Option<Self>> { const POLL_MAX: Duration = Duration::from_millis(500);
let file = Self::open_lock_file(dir)?;
match file.try_lock() {
Ok(()) => Ok(Some(Self { _file: file })),
Err(_) => Ok(None),
}
}
fn open_lock_file(dir: &std::path::Path) -> std::io::Result<std::fs::File> { fn open_file(dir: &Path, name: &str) -> io::Result<File> {
std::fs::create_dir_all(dir)?; std::fs::create_dir_all(dir)?;
let lock_path = dir.join(".obikmer.lock");
std::fs::OpenOptions::new() std::fs::OpenOptions::new()
.create(true) .create(true)
.truncate(false) .truncate(false)
.write(true) .write(true)
.open(&lock_path) .open(dir.join(name))
}
/// The gate: blocks, but its holders only ever run instantaneous,
/// non-blocking operations, so the wait is negligible. Released on drop.
fn gate(dir: &Path) -> io::Result<File> {
let f = open_file(dir, GATE_FILE)?;
f.lock()?;
Ok(f)
}
/// Non-blocking attempt in `mode`; `Ok(false)` means "held by someone else".
fn try_lock(file: &File, mode: LockMode) -> io::Result<bool> {
let r = match mode {
LockMode::Exclusive => file.try_lock(),
LockMode::Shared => file.try_lock_shared(),
};
match r {
Ok(()) => Ok(true),
Err(std::fs::TryLockError::WouldBlock) => Ok(false),
Err(std::fs::TryLockError::Error(e)) => Err(e),
}
}
/// Sleeps between two attempts and logs once, at the first one.
struct Backoff {
delay: Duration,
logged: bool,
}
impl Backoff {
fn new() -> Self {
Self { delay: POLL_MIN, logged: false }
}
fn wait(&mut self, dir: &Path, why: &str) {
if !self.logged {
info!(dir = %dir.display(), "{why}");
self.logged = true;
}
std::thread::sleep(self.delay);
self.delay = (self.delay * 2).min(POLL_MAX);
}
}
impl DirLock {
/// Block until the exclusive lock on `dir` is acquired (creating `dir`
/// and the lock files within it if needed). Logs once if the wait is
/// non-trivial, so a blocked command doesn't look silently hung.
pub fn acquire(dir: &Path) -> io::Result<Self> {
Self::acquire_mode(dir, LockMode::Exclusive, "waiting for other obikmer processes to release this index")
}
/// Block until a shared lock on `dir` is acquired — until no exclusive
/// holder remains. Logs once if the wait is non-trivial.
pub fn acquire_shared(dir: &Path) -> io::Result<Self> {
Self::acquire_mode(dir, LockMode::Shared, "waiting for an obikmer process writing this index")
}
fn acquire_mode(dir: &Path, mode: LockMode, why: &str) -> io::Result<Self> {
let file = open_file(dir, LOCK_FILE)?;
let mut backoff = Backoff::new();
loop {
{
let _gate = gate(dir)?;
if try_lock(&file, mode)? {
return Ok(Self { file, dir: dir.to_owned(), mode });
}
}
backoff.wait(dir, why);
}
}
pub fn mode(&self) -> LockMode {
self.mode
}
/// Non-blocking probe: `Ok(true)` if no exclusive holder exists right
/// now (a shared lock could be taken), `Ok(false)` if one does. Retains
/// nothing either way.
pub fn is_free_of_writer(dir: &Path) -> io::Result<bool> {
let file = open_file(dir, LOCK_FILE)?;
let _gate = gate(dir)?;
try_lock(&file, LockMode::Shared)
}
/// Shared → exclusive. Returns the exclusive lock and whether the
/// promotion was **gapless**: `true` means the lock was never released,
/// so nothing in the directory can have changed since the shared lock
/// was held and whatever the caller read stays valid; `false` means the
/// shared lock had to be given up while waiting, so a writer may have
/// run in between and the caller must re-check what it relied on.
///
/// Gapless promotion is available to one promoter at a time (it holds
/// `.obikmer.promote` while it waits for the other readers to leave).
/// A second concurrent promoter releases its shared lock instead — two
/// promoters each keeping theirs while waiting for the other's would
/// wait for ever. An exclusive lock is returned unchanged.
pub fn promote(self) -> io::Result<(Self, bool)> {
if self.mode == LockMode::Exclusive {
return Ok((self, true));
}
let claim = open_file(&self.dir, CLAIM_FILE)?;
if !try_lock(&claim, LockMode::Exclusive)? {
let dir = self.dir.clone();
drop(self); // gives the shared lock up
return Ok((Self::acquire(&dir)?, false));
}
let mut backoff = Backoff::new();
loop {
{
let _gate = gate(&self.dir)?;
self.file.unlock()?;
if try_lock(&self.file, LockMode::Exclusive)? {
return Ok((Self { mode: LockMode::Exclusive, ..self }, true));
}
// Other readers remain. Nobody can hold the lock exclusively
// without the gate, so taking our shared lock back succeeds.
if !try_lock(&self.file, LockMode::Shared)? {
return Err(io::Error::other("lost the shared index lock during promotion"));
}
}
backoff.wait(&self.dir, "waiting for other readers of this index before writing to it");
}
}
/// Exclusive → shared, without ever leaving the lock free in between.
/// A shared lock is returned unchanged.
pub fn demote(self) -> io::Result<Self> {
if self.mode == LockMode::Shared {
return Ok(self);
}
let held_gate = gate(&self.dir)?;
self.file.unlock()?;
if !try_lock(&self.file, LockMode::Shared)? {
return Err(io::Error::other("could not take the shared index lock back after writing"));
}
drop(held_gate);
Ok(Self { mode: LockMode::Shared, ..self })
} }
} }
+96
View File
@@ -0,0 +1,96 @@
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use crate::lock::{DirLock, LockMode};
/// The directory lock of one handle on an index: nothing, shared or
/// exclusive, and the transitions between them that a handle needs. What
/// `obikindex::KmerIndex` delegates its locking to, kept apart so that it can
/// be tested without building an index.
///
/// Interior mutability (`&self` methods): a handle is shared behind `Arc`.
/// Not for concurrent use from several threads of one handle — the first
/// section to finish would restore the lock under the others.
#[derive(Debug)]
pub struct LockSlot {
dir: PathBuf,
held: Mutex<Option<DirLock>>,
}
impl LockSlot {
/// A slot holding nothing (a handle that took no lock, or one that
/// reopens a directory its own builder already holds).
pub fn empty(dir: &Path) -> Self {
Self { dir: dir.to_owned(), held: Mutex::new(None) }
}
/// A slot holding a shared lock on `dir`; blocks while a writer is at work.
pub fn shared(dir: &Path) -> io::Result<Self> {
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire_shared(dir)?)) })
}
/// A slot holding the exclusive lock on `dir`; blocks while anyone else holds it.
pub fn exclusive(dir: &Path) -> io::Result<Self> {
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire(dir)?)) })
}
/// Whether the slot holds the exclusive lock (a shared one is not write access).
pub fn holds_write_lock(&self) -> bool {
matches!(&*self.held.lock().unwrap(), Some(l) if l.mode() == LockMode::Exclusive)
}
/// Hold the exclusive lock from now on: a no-op if already held that way,
/// a promotion of a shared lock, a plain acquisition of an empty slot.
pub fn acquire_exclusive(&self) -> io::Result<()> {
let held = self.held.lock().unwrap().take();
let lock = match held {
Some(l) => l.promote()?.0,
None => DirLock::acquire(&self.dir)?,
};
*self.held.lock().unwrap() = Some(lock);
Ok(())
}
/// Run `f` with the exclusive lock held:
/// - already exclusive: `f` runs as is (so sections nest);
/// - shared: promoted for the duration of `f`, then demoted back, so the
/// slot keeps the shared lock it came with. The promotion is gapless
/// when no other process promotes at the same time; otherwise other
/// writers may run between the shared lock being given up and the
/// exclusive one being taken. Either way **`f` must re-check, before
/// building, what it is about to build**, and the caller must not hold
/// anything read earlier that a writer could have replaced;
/// - empty: the lock is taken for `f` only.
///
/// Restored also if `f` fails; an I/O error taking or restoring the lock
/// is converted to `E`.
pub fn with_exclusive<T, E: From<io::Error>>(&self, f: impl FnOnce() -> Result<T, E>) -> Result<T, E> {
enum Before {
Exclusive,
Shared,
Nothing,
}
let held = self.held.lock().unwrap().take();
let (lock, before) = match held {
Some(l) if l.mode() == LockMode::Exclusive => (l, Before::Exclusive),
Some(l) => (l.promote()?.0, Before::Shared),
None => (DirLock::acquire(&self.dir)?, Before::Nothing),
};
*self.held.lock().unwrap() = Some(lock);
let out = f();
let lock = self.held.lock().unwrap().take();
match (before, lock) {
(Before::Exclusive, lock) => *self.held.lock().unwrap() = lock,
(Before::Shared, Some(l)) => *self.held.lock().unwrap() = Some(l.demote()?),
_ => {} // taken for `f` only: released here
}
out
}
/// Non-blocking probe: `true` if no process holds the directory
/// exclusively right now (this slot included), `false` if one does.
pub fn is_free_of_writer(&self) -> bool {
DirLock::is_free_of_writer(&self.dir).unwrap_or(false)
}
}
+208
View File
@@ -0,0 +1,208 @@
//! The directory lock is a readers/writer lock: `shared` slots coexist, an
//! `exclusive` one waits for them all and keeps new ones out, and a shared
//! slot can be promoted for a section (`with_exclusive`) and demoted back.
//! No index is involved, and therefore none of the process-wide k-mer
//! parameters other crates' tests share.
use std::io;
use std::path::PathBuf;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::mpsc;
use std::sync::{Arc, Barrier};
use std::thread;
use std::time::Duration;
use crate::lock::{DirLock, LockMode};
use crate::lock_slot::LockSlot;
const SETTLE: Duration = Duration::from_millis(400);
const PATIENCE: Duration = Duration::from_secs(10);
/// A scratch directory removed on drop (no `tempfile` dependency here).
struct Scratch(PathBuf);
impl Scratch {
fn new() -> Self {
static N: AtomicUsize = AtomicUsize::new(0);
let dir = std::env::temp_dir().join(format!(
"obisys-lock-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&dir).unwrap();
Self(dir)
}
}
impl Drop for Scratch {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
#[test]
fn readers_coexist() {
let d = Scratch::new();
let a = LockSlot::shared(&d.0).unwrap();
let b = LockSlot::shared(&d.0).unwrap();
assert!(!a.holds_write_lock() && !b.holds_write_lock());
assert!(a.is_free_of_writer(), "readers are not writers");
}
#[test]
fn exclusive_is_exclusive_and_released_on_drop() {
let d = Scratch::new();
let w = LockSlot::exclusive(&d.0).unwrap();
assert!(w.holds_write_lock());
assert!(!w.is_free_of_writer());
drop(w);
assert!(LockSlot::empty(&d.0).is_free_of_writer());
}
#[test]
fn a_writer_waits_for_the_readers() {
let d = Scratch::new();
let reader = LockSlot::shared(&d.0).unwrap();
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let writer = thread::spawn(move || {
drop(LockSlot::exclusive(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the writer must wait for the reader");
drop(reader);
rx.recv_timeout(PATIENCE).expect("the writer proceeds once the reader is gone");
writer.join().unwrap();
}
#[test]
fn a_reader_waits_for_the_writer() {
let d = Scratch::new();
let writer = LockSlot::exclusive(&d.0).unwrap();
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let reader = thread::spawn(move || {
drop(LockSlot::shared(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the reader must wait for the writer");
drop(writer);
rx.recv_timeout(PATIENCE).expect("the reader proceeds once the writer is gone");
reader.join().unwrap();
}
#[test]
fn with_exclusive_on_a_reader_promotes_then_restores_the_shared_lock() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
let out: io::Result<i32> = slot.with_exclusive(|| {
assert!(slot.holds_write_lock());
assert!(!slot.is_free_of_writer());
Ok(7)
});
assert_eq!(out.unwrap(), 7);
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
// Still a reader: a writer elsewhere waits for it.
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let writer = thread::spawn(move || {
drop(LockSlot::exclusive(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the slot must be a reader again");
drop(slot);
rx.recv_timeout(PATIENCE).unwrap();
writer.join().unwrap();
}
#[test]
fn with_exclusive_on_an_empty_slot_holds_the_lock_for_the_section_only() {
let d = Scratch::new();
let slot = LockSlot::empty(&d.0);
slot.with_exclusive::<(), io::Error>(|| {
assert!(slot.holds_write_lock());
Ok(())
})
.unwrap();
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
}
#[test]
fn with_exclusive_restores_on_error() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
let r: io::Result<()> = slot.with_exclusive(|| Err(io::Error::other("boom")));
assert!(r.is_err());
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
}
#[test]
fn with_exclusive_nests_and_keeps_a_lock_taken_at_open() {
let d = Scratch::new();
let slot = LockSlot::exclusive(&d.0).unwrap();
slot.with_exclusive::<(), io::Error>(|| {
slot.with_exclusive::<(), io::Error>(|| Ok(()))?;
assert!(slot.holds_write_lock(), "an inner section must not release the outer lock");
Ok(())
})
.unwrap();
assert!(slot.holds_write_lock(), "a slot opened exclusive keeps it");
}
#[test]
fn acquire_exclusive_promotes_a_shared_slot_and_is_idempotent() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
slot.acquire_exclusive().unwrap();
slot.acquire_exclusive().unwrap();
assert!(slot.holds_write_lock());
}
/// The case a naive upgrade deadlocks on: two readers, each holding a shared
/// lock, both ask for the exclusive one at the same time.
#[test]
fn two_readers_promoting_at_once_both_finish() {
let d = Scratch::new();
let barrier = Arc::new(Barrier::new(2));
let (tx, rx) = mpsc::channel();
let handles: Vec<_> = (0..2)
.map(|_| {
let (dir, b, tx) = (d.0.clone(), Arc::clone(&barrier), tx.clone());
thread::spawn(move || {
let slot = LockSlot::shared(&dir).unwrap();
b.wait(); // both hold a shared lock
slot.with_exclusive::<(), io::Error>(|| {
assert!(slot.holds_write_lock());
thread::sleep(Duration::from_millis(50));
Ok(())
})
.unwrap();
tx.send(()).unwrap();
})
})
.collect();
for _ in 0..2 {
rx.recv_timeout(PATIENCE).expect("a promotion deadlocked");
}
for h in handles {
h.join().unwrap();
}
}
#[test]
fn a_lone_promotion_is_gapless_and_a_demotion_keeps_the_lock() {
let d = Scratch::new();
let shared = DirLock::acquire_shared(&d.0).unwrap();
let (excl, gapless) = shared.promote().unwrap();
assert_eq!(excl.mode(), LockMode::Exclusive);
assert!(gapless, "nobody else was promoting: the lock was never released");
let shared = excl.demote().unwrap();
assert_eq!(shared.mode(), LockMode::Shared);
// Shared again: other readers get in, a writer does not.
let _other = DirLock::acquire_shared(&d.0).unwrap();
assert!(DirLock::is_free_of_writer(&d.0).unwrap());
}
+3216
View File
File diff suppressed because it is too large. Load diff
+36
View File
@@ -0,0 +1,36 @@
[package]
name = "migrate-mphf"
version = "0.1.0"
edition = "2024"
publish = false
description = "One-shot converter: obikmer indexes whose layer MPHFs were built with ptr_hash 1.x -> ptr_hash 2.x"
# Standalone: not a member of src/'s workspace, own Cargo.lock. The previous
# generation of the MPHF stack (and its `proc-macro-error2` future-incompat
# warning) lives only here, never in obikmer itself.
[workspace]
[features]
default = []
numa = ["obisys/numa"]
[dependencies]
obikindex = { path = "../../src/obikindex" }
obicompactvec = { path = "../../src/obicompactvec" }
obiskio = { path = "../../src/obiskio" }
obisys = { path = "../../src/obisys", default-features = false }
epserde = "0.13"
rayon = "1"
anyhow = "1"
clap = { version = "4", features = ["derive"] }
tracing = "0.1.44"
tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
# The generation the indexes to convert were built with, under other names
# so that both can live in this one binary.
ptr_hash_old = { package = "ptr_hash", version = "=1.1.0" }
epserde_old = { package = "epserde", version = "=0.8.0" }
cacheline_ef_old = { package = "cacheline-ef", version = "=1.1.0" }
[profile.release]
debug = 1
+98
View File
@@ -0,0 +1,98 @@
//! Crash-safe replacement of a layer's files by their converted versions.
//!
//! Everything new is written under `<layer>/.migrate_mphf/` first; nothing of
//! the layer is touched until that is complete and verified. Then a `COMMIT`
//! marker is written, and [`finish`] moves the staged files into place. Every
//! step of [`finish`] is idempotent ("if the staged item still exists, move
//! it"), so a layer interrupted after `COMMIT` is simply finished by the next
//! run. Files are only ever replaced by `rename`, never written in place:
//! some of them are hard-linked from elsewhere (`ls -l` link count 2), and an
//! in-place write would corrupt the other link.
use std::fs;
use std::io;
use std::path::Path;
use anyhow::{Context, Result};
pub const STAGING: &str = ".migrate_mphf";
pub const COMMIT: &str = "COMMIT";
/// Written last: the layer is converted, skip it on later runs.
pub const DONE: &str = ".mphf_migrated";
/// Where derived annexes go instead of being deleted.
pub const STALE_DIR: &str = "stale_annexes_pre_ptrhash2";
/// Replaced as directories.
const DIRS: [&str; 2] = ["presence", "counts"];
/// Replaced as files; `mphf.bin` last.
const FILES: [&str; 3] = ["evidence.bin", "fingerprint.bin", "mphf.bin"];
/// Slot-ordered or slot-derived annexes of a layer: wrong (silently) once the
/// slot numbering changes, so they are set aside, to be rebuilt.
const LAYER_ANNEXES: [&str; 7] = [
"siblings.psib",
"entropy.pent",
"entropy.noamb.pent",
"pairweight.ppwt",
"pairweight.noamb.ppwt",
"quartet.pbiv",
"quartet.noamb.pbiv",
];
/// Index-level annexes derived from the sibling annex.
pub const INDEX_ANNEXES: [&str; 2] = ["family_overlap.povl", "family_overlap.noamb.povl"];
/// Move the staged files into `dir`, set the stale annexes aside, mark the
/// layer done and drop the staging area.
pub fn finish(dir: &Path) -> Result<()> {
let staging = dir.join(STAGING);
for name in DIRS {
let staged = staging.join(name);
if staged.exists() {
let target = dir.join(name);
if target.exists() {
fs::remove_dir_all(&target).with_context(|| format!("removing {}", target.display()))?;
}
fs::rename(&staged, &target).with_context(|| format!("installing {}", target.display()))?;
}
}
for name in FILES {
let staged = staging.join(name);
if staged.exists() {
let target = dir.join(name);
fs::rename(&staged, &target).with_context(|| format!("installing {}", target.display()))?;
}
}
set_aside(dir, &LAYER_ANNEXES)?;
fs::write(dir.join(DONE), env!("CARGO_PKG_VERSION"))?;
fs::remove_dir_all(&staging)?;
Ok(())
}
/// Move each of `names` that exists in `dir` into `dir/STALE_DIR/`.
/// Returns how many were moved.
pub fn set_aside(dir: &Path, names: &[&str]) -> Result<usize> {
let aside = dir.join(STALE_DIR);
let mut moved = 0;
for name in names {
let from = dir.join(name);
if !from.exists() {
continue;
}
fs::create_dir_all(&aside)?;
fs::rename(&from, aside.join(name)).with_context(|| format!("setting aside {}", from.display()))?;
moved += 1;
}
Ok(moved)
}
/// Remove a staging area left by an interrupted, uncommitted run.
pub fn discard(dir: &Path) -> io::Result<()> {
match fs::remove_dir_all(dir.join(STAGING)) {
Err(e) if e.kind() != io::ErrorKind::NotFound => Err(e),
_ => Ok(()),
}
}
+100
View File
@@ -0,0 +1,100 @@
//! Permute the per-slot files of a layer into `staging/`.
//!
//! Presence and count matrices always come out in the compact row-major
//! (sparse) format, whatever they were: a dense source is first converted
//! sequentially (`build_from_dense`), then permuted from the sparse form,
//! whose rows are cheap to read at arbitrary slots.
use std::fs;
use std::path::Path;
use anyhow::{Result, ensure};
use obicompactvec::{
PersistentBitMatrix, PersistentIntMatrix, PersistentSparseBitMatrixBuilder,
PersistentSparseCompactIntMatrixBuilder,
};
use obikindex::layer::evidence::{Evidence, EvidenceWriter};
use obikindex::layer::fingerprint::{FingerprintVec, FingerprintVecWriter};
use crate::plan::Plan;
use crate::util::oki;
/// Every per-slot file of the layer in `dir` that exists, permuted into `staging`.
pub fn permute_all(dir: &Path, staging: &Path, plan: &Plan) -> Result<()> {
if dir.join("evidence.bin").exists() {
evidence(dir, staging, plan)?;
}
if dir.join("fingerprint.bin").exists() {
fingerprint(dir, staging, plan)?;
}
if dir.join("presence").is_dir() {
presence(dir, staging, plan)?;
}
if dir.join("counts").is_dir() {
counts(dir, staging, plan)?;
}
Ok(())
}
fn evidence(dir: &Path, staging: &Path, plan: &Plan) -> Result<()> {
let old = oki(Evidence::open(&dir.join("evidence.bin")))?;
ensure!(old.len() == plan.n, "evidence.bin has {} slots, layer has {}", old.len(), plan.n);
let mut out = EvidenceWriter::new(plan.n);
for (s, &o) in plan.new_to_old.iter().enumerate() {
let (chunk_id, rank) = old.decode(o as usize);
out.set(s, chunk_id, rank);
}
oki(out.write(&staging.join("evidence.bin")))
}
fn fingerprint(dir: &Path, staging: &Path, plan: &Plan) -> Result<()> {
let old = oki(FingerprintVec::open(&dir.join("fingerprint.bin")))?;
ensure!(old.n() == plan.n, "fingerprint.bin has {} slots, layer has {}", old.n(), plan.n);
let mut out = FingerprintVecWriter::new(plan.n, old.b());
for (s, &o) in plan.new_to_old.iter().enumerate() {
out.set(s, old.get(o as usize));
}
oki(out.write(&staging.join("fingerprint.bin")))
}
fn presence(dir: &Path, staging: &Path, plan: &Plan) -> Result<()> {
let out = staging.join("presence");
let src = match PersistentBitMatrix::open(dir)? {
// Mono-genome: no file, nothing is attached to slots.
PersistentBitMatrix::Implicit { .. } => return Ok(()),
PersistentBitMatrix::Sparse(s) => {
ensure!(s.n() == plan.n, "presence has {} rows, layer has {}", s.n(), plan.n);
PersistentSparseBitMatrixBuilder::build_permuted(&s, &plan.new_to_old, &out)?.close()?;
return Ok(());
}
dense => dense,
};
ensure!(src.n() == plan.n, "presence has {} rows, layer has {}", src.n(), plan.n);
let tmp = staging.join("presence_dense_to_sparse");
let sparse = PersistentSparseBitMatrixBuilder::build_from_dense(&src, &tmp)?.finish()?;
drop(src);
PersistentSparseBitMatrixBuilder::build_permuted(&sparse, &plan.new_to_old, &out)?.close()?;
drop(sparse);
fs::remove_dir_all(&tmp)?;
Ok(())
}
fn counts(dir: &Path, staging: &Path, plan: &Plan) -> Result<()> {
let out = staging.join("counts");
let src = match PersistentIntMatrix::open(dir)? {
PersistentIntMatrix::Sparse(s) => {
ensure!(s.n() == plan.n, "counts has {} rows, layer has {}", s.n(), plan.n);
PersistentSparseCompactIntMatrixBuilder::build_permuted(&s, &plan.new_to_old, &out)?.close()?;
return Ok(());
}
dense => dense,
};
ensure!(src.n() == plan.n, "counts has {} rows, layer has {}", src.n(), plan.n);
let tmp = staging.join("counts_dense_to_sparse");
let sparse = PersistentSparseCompactIntMatrixBuilder::build_from_dense(&src, &tmp)?.finish()?;
drop(src);
PersistentSparseCompactIntMatrixBuilder::build_permuted(&sparse, &plan.new_to_old, &out)?.close()?;
drop(sparse);
fs::remove_dir_all(&tmp)?;
Ok(())
}
+75
View File
@@ -0,0 +1,75 @@
//! Convert one layer directory: new MPHF, permuted per-slot data, verified,
//! then installed atomically (see `commit`).
use std::fs;
use std::path::Path;
use anyhow::{Result, anyhow, ensure};
use epserde::prelude::*;
use obikindex::layer::{Mphf, mphf_params};
use obiskio::UnitigFileReader;
use crate::util::sk;
use crate::{commit, data, old, plan, verify};
pub struct Options {
/// How many k-mers per layer to compare old against new (ignored with `verify_full`).
pub verify_sample: usize,
pub verify_full: bool,
/// Do everything except installing the result.
pub dry_run: bool,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Outcome {
/// Already converted by an earlier run.
Skipped,
/// Interrupted after verification by an earlier run; installed now.
Resumed,
Converted { n: usize, verified: usize },
/// Everything but the installation.
Checked { n: usize, verified: usize },
}
pub fn convert(dir: &Path, opts: &Options) -> Result<Outcome> {
if dir.join(commit::DONE).exists() {
commit::discard(dir)?;
return Ok(Outcome::Skipped);
}
let staging = dir.join(commit::STAGING);
if staging.join(commit::COMMIT).exists() {
commit::finish(dir)?;
return Ok(Outcome::Resumed);
}
commit::discard(dir)?;
fs::create_dir_all(&staging)?;
let n = sk(UnitigFileReader::open_sequential(&dir.join("unitigs.bin")))?.n_kmers();
ensure!(n <= u32::MAX as usize, "layer has {n} k-mers, more than the 2^32 this tool handles");
let verified = if n == 0 {
empty_mphf(&staging)?;
0
} else {
let old_mphf = old::load(&dir.join("mphf.bin"))?;
let plan = plan::build(dir, &staging, n, &old_mphf)?;
data::permute_all(dir, &staging, &plan)?;
let stride = if opts.verify_full { 1 } else { (n / opts.verify_sample.max(1)).max(1) };
verify::check(dir, &staging, &plan, &old_mphf, stride)?
};
if opts.dry_run {
commit::discard(dir)?;
return Ok(Outcome::Checked { n, verified });
}
fs::write(staging.join(commit::COMMIT), b"")?;
commit::finish(dir)?;
Ok(Outcome::Converted { n, verified })
}
/// An empty layer still carries an MPHF (of nothing), like `MphfLayer::build` writes.
fn empty_mphf(staging: &Path) -> Result<()> {
let mphf = Mphf::try_new(&[] as &[u64], mphf_params())
.ok_or_else(|| anyhow!("empty MPHF construction failed"))?;
unsafe { mphf.store(staging.join("mphf.bin")) }.map_err(|e| anyhow!("storing the empty MPHF: {e}"))
}
+121
View File
@@ -0,0 +1,121 @@
//! One-shot converter for obikmer indexes whose layer MPHFs were built with
//! `ptr_hash` 1.x: rebuilds each MPHF with `ptr_hash` 2.x and carries the
//! per-slot data (evidence, fingerprint, presence/count matrices) over to the
//! new slot numbering, in place, layer by layer, verified before installing.
//!
//! Presence and count matrices come out in the compact row-major format.
//! Annexes derived from slot numbering (sibling, entropy, pair-weight,
//! quartet, family-overlap) are set aside under `stale_annexes_pre_ptrhash2/`
//! and must be rebuilt.
mod commit;
mod data;
mod layer;
mod old;
mod plan;
mod util;
mod verify;
use std::path::PathBuf;
use std::sync::atomic::{AtomicUsize, Ordering};
use anyhow::{Result, ensure};
use clap::Parser;
use obikindex::KmerIndex;
use obisys::PartitionRunner;
use tracing::{info, warn};
use layer::{Options, Outcome};
use util::oki;
#[derive(Parser)]
#[command(about = "Convert an obikmer index from ptr_hash 1.x to ptr_hash 2.x MPHFs, in place")]
struct Cli {
/// Index directory (the one holding `index.meta` and `part_*`).
index: PathBuf,
/// Limit the number of CPUs used.
#[arg(long)]
cpu_max: Option<usize>,
/// K-mers compared old-vs-new per layer before installing it.
#[arg(long, default_value_t = 200_000)]
verify_sample: usize,
/// Compare every k-mer of every layer (slow).
#[arg(long)]
verify_full: bool,
/// Do everything except installing: convert into staging, verify, discard.
#[arg(long)]
dry_run: bool,
/// Only these partitions (repeatable). The index-level annexes are then left alone.
#[arg(long)]
partition: Vec<usize>,
/// Only these layers of each partition (repeatable). The index-level annexes are then left alone.
#[arg(long)]
layer: Vec<usize>,
}
fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()))
.init();
let cli = Cli::parse();
if let Some(n) = cli.cpu_max {
obisys::set_cpu_cap(n);
}
// Holds the index's directory lock for the whole run, and sets k.
let index = oki(KmerIndex::open_lock(&cli.index))?;
let n_partitions = index.n_partitions();
let partial = !cli.partition.is_empty() || !cli.layer.is_empty();
let order: Vec<usize> = if cli.partition.is_empty() { (0..n_partitions).collect() } else { cli.partition.clone() };
ensure!(order.iter().all(|&p| p < n_partitions), "index has only {n_partitions} partitions");
let opts = Options { verify_sample: cli.verify_sample, verify_full: cli.verify_full, dry_run: cli.dry_run };
let converted = AtomicUsize::new(0);
let skipped = AtomicUsize::new(0);
let mut done = 0usize;
PartitionRunner::new().run(
&order,
|p| -> Result<Vec<(usize, Outcome)>> {
let n_layers = oki(index.n_layers(p))?;
let mut outcomes = Vec::new();
for l in (0..n_layers).filter(|l| cli.layer.is_empty() || cli.layer.contains(l)) {
let dir = index.layer_path(p, l);
let outcome = layer::convert(&dir, &opts)
.map_err(|e| e.context(format!("partition {p}, layer {l} ({})", dir.display())))?;
match outcome {
Outcome::Skipped => skipped.fetch_add(1, Ordering::Relaxed),
_ => converted.fetch_add(1, Ordering::Relaxed),
};
outcomes.push((l, outcome));
}
Ok(outcomes)
},
|p, outcomes, elapsed| {
done += 1;
info!("partition {p} ({done}/{}) in {:.1?}: {outcomes:?}", order.len(), elapsed);
},
)?;
if !cli.dry_run && !partial {
let moved = commit::set_aside(index.dir(), &commit::INDEX_ANNEXES)?;
if moved > 0 {
info!("{moved} index-level annex(es) set aside under {}", commit::STALE_DIR);
}
} else if !cli.dry_run {
warn!("partial run: index-level annexes (family overlap) not set aside; do it on the final pass");
}
info!(
"done: {} layer(s) {}, {} already converted",
converted.load(Ordering::Relaxed),
if cli.dry_run { "checked" } else { "converted" },
skipped.load(Ordering::Relaxed)
);
Ok(())
}
+17
View File
@@ -0,0 +1,17 @@
//! The previous MPHF generation (`ptr_hash` 1.x), kept only to read the
//! `mphf.bin` files of indexes built before the migration.
use std::path::Path;
use anyhow::{Result, anyhow};
use cacheline_ef_old::{CachelineEf, CachelineEfVec};
use epserde_old::prelude::*;
use ptr_hash_old::{PtrHash, bucket_fn::CubicEps, hash::Xx64};
/// Exactly the type `obikindex` used before the migration.
pub type OldMphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xx64, Vec<u8>>;
pub fn load(path: &Path) -> Result<OldMphf> {
OldMphf::load_full(path)
.map_err(|e| anyhow!("{}: not a ptr_hash 1.x MPHF (already converted?): {e}", path.display()))
}
+53
View File
@@ -0,0 +1,53 @@
//! The slot permutation between a layer's old and new MPHF.
//!
//! A layer's MPHF gives each k-mer a slot, and every other per-slot file of
//! the layer (evidence, fingerprint, presence/count matrices) is indexed by
//! it. A new MPHF numbers the same k-mers differently, so the data has to
//! follow: the k-mers themselves come from `unitigs.bin`, which does not
//! depend on the MPHF at all.
use std::path::Path;
use anyhow::{Result, anyhow, ensure};
use epserde::prelude::*;
use obikindex::layer::{Mphf, mphf_params};
use obiskio::CanonicalKmerIter;
use rayon::prelude::*;
use crate::old::OldMphf;
use crate::util::sk;
/// `new_to_old[s]` is the old slot of the k-mer that the new MPHF puts in
/// slot `s`. Always a bijection on `0..n` (checked while it is built).
pub struct Plan {
pub n: usize,
pub new_to_old: Vec<u32>,
}
/// Build the new MPHF of the layer whose `unitigs.bin` is in `dir`, store it
/// as `staging/mphf.bin`, and derive the permutation against `old`.
pub fn build(dir: &Path, staging: &Path, n: usize, old: &OldMphf) -> Result<Plan> {
let unitigs = dir.join("unitigs.bin");
// Same construction as `MphfLayer::build`: a clonable mmap iterator, bridged.
let keys = sk(CanonicalKmerIter::new(&unitigs))?;
let new: Mphf = Mphf::new_from_par_iter(n, keys.map(|k| k.raw()).par_bridge(), mphf_params());
unsafe { new.store(staging.join("mphf.bin")) }.map_err(|e| anyhow!("storing the new MPHF: {e}"))?;
let mut new_to_old = vec![u32::MAX; n];
let mut old_seen = vec![0u8; n.div_ceil(8)];
for kmer in sk(CanonicalKmerIter::new(&unitigs))? {
let key = kmer.raw();
let o = old.index(&key);
let s = new.index(&key);
ensure!(o < n, "old MPHF: slot {o} out of range (n = {n}) — is this layer's mphf.bin really the one of its unitigs.bin?");
ensure!(s < n, "new MPHF: slot {s} out of range (n = {n})");
ensure!(new_to_old[s] == u32::MAX, "new MPHF puts two k-mers in slot {s}");
let (byte, bit) = (o / 8, 1u8 << (o % 8));
ensure!(old_seen[byte] & bit == 0, "old MPHF puts two k-mers in slot {o}");
old_seen[byte] |= bit;
new_to_old[s] = o as u32;
}
// n distinct new slots, all in range, among n: the map is total.
Ok(Plan { n, new_to_old })
}
+11
View File
@@ -0,0 +1,11 @@
//! Error adapters: the libraries each have their own error type.
use anyhow::{Result, anyhow};
pub fn sk<T>(r: obiskio::SKResult<T>) -> Result<T> {
r.map_err(|e| anyhow!("{e}"))
}
pub fn oki<T>(r: obikindex::OKIResult<T>) -> Result<T> {
r.map_err(|e| anyhow!("{e}"))
}
+90
View File
@@ -0,0 +1,90 @@
//! Check the staged files against the layer as it still is, before anything
//! is replaced.
//!
//! For a sample of k-mers read from `unitigs.bin`, the old chain (old MPHF →
//! old slot → old data) must agree with the new one (staged MPHF, loaded the
//! way queries load it → new slot → staged data): same evidence, same
//! fingerprint, same presence set, same counts. A full check is the same loop
//! with every k-mer.
use std::path::Path;
use anyhow::{Result, anyhow, ensure};
use epserde::prelude::*;
use obicompactvec::{PersistentBitMatrix, PersistentIntMatrix, PersistentSparseBitMatrix, PersistentSparseCompactIntMatrix};
use obikindex::layer::Mphf;
use obikindex::layer::evidence::Evidence;
use obikindex::layer::fingerprint::FingerprintVec;
use obiskio::CanonicalKmerIter;
use crate::old::OldMphf;
use crate::plan::Plan;
use crate::util::{oki, sk};
/// Both sides of one per-slot file, when the layer has it.
struct Pair<O, N> {
old: O,
new: N,
}
fn pair<O, N>(
exists: bool,
old: impl FnOnce() -> Result<O>,
new: impl FnOnce() -> Result<N>,
) -> Result<Option<Pair<O, N>>> {
Ok(if exists { Some(Pair { old: old()?, new: new()? }) } else { None })
}
/// Returns how many k-mers were compared.
pub fn check(dir: &Path, staging: &Path, plan: &Plan, old_mphf: &OldMphf, stride: usize) -> Result<usize> {
let staged_mphf = unsafe { Mphf::mmap(staging.join("mphf.bin"), Flags::empty()) }
.map_err(|e| anyhow!("staged mphf.bin does not load: {e}"))?;
let new_mphf = staged_mphf.uncase();
let evidence = pair(
staging.join("evidence.bin").exists(),
|| oki(Evidence::open(&dir.join("evidence.bin"))),
|| oki(Evidence::open(&staging.join("evidence.bin"))),
)?;
let fingerprint = pair(
staging.join("fingerprint.bin").exists(),
|| oki(FingerprintVec::open(&dir.join("fingerprint.bin"))),
|| oki(FingerprintVec::open(&staging.join("fingerprint.bin"))),
)?;
let presence = pair(
staging.join("presence").is_dir(),
|| Ok(PersistentBitMatrix::open(dir)?),
|| Ok(PersistentSparseBitMatrix::open(&staging.join("presence"))?),
)?;
let counts = pair(
staging.join("counts").is_dir(),
|| Ok(PersistentIntMatrix::open(dir)?),
|| Ok(PersistentSparseCompactIntMatrix::open(&staging.join("counts"))?),
)?;
let mut compared = 0;
for (i, kmer) in sk(CanonicalKmerIter::new(&dir.join("unitigs.bin")))?.enumerate() {
if i % stride != 0 {
continue;
}
let key = kmer.raw();
let o = old_mphf.index(&key);
let s = new_mphf.index(&key);
ensure!(s < plan.n && plan.new_to_old[s] as usize == o, "k-mer #{i}: staged MPHF slot {s} disagrees with the plan (old slot {o})");
if let Some(p) = &evidence {
ensure!(p.old.decode(o) == p.new.decode(s), "k-mer #{i}: evidence differs (old slot {o}, new slot {s})");
}
if let Some(p) = &fingerprint {
ensure!(p.old.get(o) == p.new.get(s), "k-mer #{i}: fingerprint differs (old slot {o}, new slot {s})");
}
if let Some(p) = &presence {
ensure!(p.old.row(o) == p.new.row(s), "k-mer #{i}: presence row differs (old slot {o}, new slot {s})");
}
if let Some(p) = &counts {
ensure!(p.old.row(o) == p.new.row(s), "k-mer #{i}: count row differs (old slot {o}, new slot {s})");
}
compared += 1;
}
Ok(compared)
}