Push zunrplorkwkt #70
@@ -72,9 +72,9 @@ Each NUMA group runs its own independent adaptive pool. Workers are distributed
|
||||
|------|--------|
|
||||
| `obikindex/src/merge.rs` | Detect NUMA topology; build N `ThreadPool`s with pinned threads; assign each pre-spawned worker to a pool; wrap `merge_partition` in `pool.install()` |
|
||||
| `obikindex/src/merge.rs` | Replace `available_parallelism()` with per-NUMA core count for spawn criterion |
|
||||
| `obikpartitionner/src/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
|
||||
| `obikpartition/src/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
|
||||
| `obidebruinj/src/debruijn.rs` | No change — `par_iter` and `current_num_threads` are pool-context-aware |
|
||||
| `obikpartitionner/src/partition.rs` | No change — same reason |
|
||||
| `obikpartition/src/partition.rs` | No change — same reason |
|
||||
|
||||
## Platform guard
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Given a set of query sequences, determine for each sequence how many of its k-me
|
||||
|
||||
## Algorithm
|
||||
|
||||
The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikpartitionner::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
|
||||
The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikpartition::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
|
||||
|
||||
```
|
||||
for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
|
||||
@@ -122,7 +122,7 @@ The `-z` CLI option overrides the index metadata value. A higher z increases str
|
||||
|
||||
### `QueryLayer` variant selection
|
||||
|
||||
`QueryLayer::open` (`obikpartitionner/src/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
|
||||
`QueryLayer::open` (`obikpartition/src/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
|
||||
|
||||
| Order | Condition | Variant | Data returned per k-mer |
|
||||
|---|---|---|---|
|
||||
@@ -244,7 +244,7 @@ Fix direction: restructure `query`'s pipe with an initial `Flat` stage analogous
|
||||
`chunk_bytes = available_memory_bytes() / (n_workers * 16)` (`query.rs:407-414`) assumes a fixed ~8–16× overhead per raw input byte. But `KmerResults::new` (`query.rs:165-179`) allocates `data: Vec<u32>` sized `total_kmers_in_chunk × n_genomes` — dense, **for every k-mer position in the chunk, hit or not** — plus `win_min` and (with `--detail`) `cov`, same scaling. Real per-chunk memory is `O(n_genomes)`, not constant; the formula doesn't know `n_genomes` at all. This is the direct cause of the OOM kill on indexes with many reference genomes.
|
||||
|
||||
**4. MPHF lookup and matrix-row fetch are fused, not staged.**
|
||||
`QueryLayer::find_into` (`obikpartitionner/src/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
|
||||
`QueryLayer::find_into` (`obikpartition/src/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
|
||||
|
||||
**5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.**
|
||||
`QueryBatch::from_records` currently dereplicates at the *superkmer* level (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, `query.rs:112`). This misses redundancy between k-mers shared by *different* superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer *construction* (`SuperKmerIter`) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on `CanonicalKmer`, in the same pass: `HashMap<CanonicalKmer, Vec<(seq_idx, pos)>>`. This also means the MPHF `find` itself runs once per **distinct** k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.
|
||||
@@ -285,7 +285,7 @@ None of this is implemented yet — parked here as a coherent roadmap while the
|
||||
|
||||
## Implementation plan
|
||||
|
||||
Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikpartitionner`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
|
||||
Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikpartition`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
|
||||
|
||||
Instrumentation is deliberately sequenced *before* the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.
|
||||
|
||||
@@ -295,7 +295,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
|
||||
**Debug logging.** Every phase that changes an algorithmic choice (not phase 0, which *is* the logging) adds `tracing::debug!`/`trace!` at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF `find` calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured `debug!` per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.
|
||||
|
||||
**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obilayeredmap`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikpartitionner` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
|
||||
**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obilayeredmap`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikpartition` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
|
||||
|
||||
**Workflow (`jj`).** Work happens in a fresh `jj` commit, easy to abandon. `jj new` between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then `query_layer.rs` split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.
|
||||
|
||||
@@ -345,13 +345,13 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
- `obikmer/src/cmd/query.rs`:
|
||||
- Replace `QueryBatch::from_records`'s dedup map (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, current `query.rs:112`) with a per-partition `HashMap<CanonicalKmer, Vec<(seq_idx: u32, pos: u32)>>`, built in the same `SuperKmerIter` pass: superkmer construction and partition routing (`part_idx` from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each `CanonicalKmer` within a superkmer is inserted individually instead of the whole superkmer being the dedup key.
|
||||
- **Verified**: `CanonicalKmer` (`obikseq/src/kmer.rs:390`, `pub type CanonicalKmer = CanonicalKmerOf<KLen>`) — the underlying `CanonicalKmerOf<L>` derives `Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash` (`kmer.rs:269`). Usable as a `HashMap`/`HashSet` key as-is, no change needed.
|
||||
- `obikpartitionner/src/query_layer.rs`:
|
||||
- `obikpartition/src/query_layer.rs`:
|
||||
- Split `QueryLayer::find_into` (`query_layer.rs:48-67`) into two methods: `find_slot(&self, kmer: CanonicalKmer) -> Option<usize>` (MPHF only, no matrix touch) and keep `fill_row` as-is for phase 4 to call later.
|
||||
- Replace `query_partition_with`'s inner loop (`query_layer.rs:103-113`) with a version that, for each unique `CanonicalKmer`, calls `find_slot` across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records `(layer_idx, slot)`.
|
||||
- New return shape for the partition-level query, replacing today's `on_hit(sk_idx, kmer_idx, row)` callback: `HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>` (roadmap point 6) — built directly from the k-mer dedup map's `Vec<(seq_idx,pos)>` values, keyed by the resolved slot instead of the k-mer.
|
||||
- **This phase alone has no throughput benefit yet** (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.
|
||||
- Log, per chunk: total k-mer occurrences vs. unique `CanonicalKmer` count (the dedup ratio — the entire justification for this phase) and the resulting MPHF `find` call count. If the dedup ratio is close to `1.0` on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.
|
||||
- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikpartitionner/src/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
|
||||
- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikpartition/src/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
|
||||
|
||||
### Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred
|
||||
|
||||
@@ -359,7 +359,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
|
||||
**What shipped:**
|
||||
- `obicompactvec`: the per-column accessors this phase needed **already existed** — `PersistentCompactIntMatrix::col_view(c)` and `PersistentBitMatrix::col_view(c)` are public, and `IntSliceView::get(slot)`/`BitSliceView::get(slot)` are public — the original plan underestimated how much of this plumbing the pairwise-distance code (`dump`/`select`/`stats`) had already required. The one real gap: `PersistentBitMatrix::col_view()` panics on the `Implicit` variant (the documented mono-genome fast path, `bitmatrix.rs`). Added `PersistentBitMatrix::get(c, slot) -> u32` (`bitmatrix.rs`), a non-panicking column-major point lookup that returns `1` for `Implicit` regardless of `c` — the smallest surface needed, not a new `col_get` API from scratch.
|
||||
- `obikpartitionner/src/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
|
||||
- `obikpartition/src/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
|
||||
- `obikmer/src/cmd/query.rs`: `KmerResults::set` (row-major, whole-row-at-once) replaced by `mark_found` (stage 1: flag a position as indexed, independent of any genome's value) and `set_one` (stage 2: write one genome's value at one position). `QueryStats` extended with `n_columns_scanned`/`n_col_get_calls`, logged per chunk.
|
||||
- Total `get()`-equivalent calls are unchanged from the row-major version (`n_hits × n_cols` in the worst case, confirmed by `n_col_get_calls` in the debug log) — the win is locality (sequential access within one layer's column at a time, across `mmap`'d regions, instead of jumping across all columns per hit), exactly as predicted.
|
||||
|
||||
@@ -367,7 +367,7 @@ Performance measurement on the reference 192-core/8-NUMA machine is done by the
|
||||
Reading `obikindex/src/numa.rs`'s actual `run()` body (not just its doc comments) shows every call spawns a timer thread **plus one OS thread per worker slot on every NUMA node** (`std::thread::scope` + one `s.spawn()` per node per `max_workers`) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned **per call**. This is fine for its actual, established usage in this codebase (`merge.rs`, `index.rs`'s `build_layers`): one `PartitionRunner::new()` + one `run()` call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for `query`'s call pattern: `query_partition_with` runs once per `(chunk, partition)`, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
|
||||
The column-major loop in stage 2 is therefore a **plain sequential loop** for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) `rayon`'s already-warm global pool (`into_par_iter()`) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern `numa_worker_pools.md` warns about for a *different* workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one `mmap`'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so `PartitionRunner` is invoked once per whole `query` run (or per large batch of chunks) rather than per `(chunk, partition)`, amortising its spawn cost the way `merge`/`build_layers` do — a bigger structural change than this phase's scope.
|
||||
- Log (implemented): `QueryStats::n_columns_scanned`/`n_col_get_calls`, folded into the existing per-chunk `debug!("k-mer dedup + column-major fetch", ...)` line (`query.rs`) alongside phase 3's dedup counters.
|
||||
- **Unit tests**: extended `obikpartitionner/src/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
|
||||
- **Unit tests**: extended `obikpartition/src/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
|
||||
- **Validation performed**: full workspace build + `cargo test --workspace`, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same `kmer_count`/`kmer_strict_matches` on every record); (2) the existing 20-genome `benchmark/global_index_presence` index — runs correctly, `n_hits=0` for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the `Implicit`/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) **the critical correctness case**: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from `genomeA` matched **only** `genomeA` (`kmer_count` identical to the pre-dedup occurrence count, zero leakage into `genomeB`'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in `n_cols`, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
|
||||
- **Not yet done**: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-`n_genomes` layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
## Code couvert
|
||||
|
||||
- `obikmer/src/cmd/query.rs` — commande query, format de sortie
|
||||
- `obikpartitionner/src/query_layer.rs` — routage de la requête à travers les partitions
|
||||
- `obikpartition/src/query_layer.rs` — routage de la requête à travers les partitions
|
||||
- `obiread/src/lib.rs` — lecture des séquences d'entrée pour la requête
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -94,7 +94,7 @@ Option B avoids storing kmer values and works uniformly regardless of filter sel
|
||||
|
||||
## Files to modify
|
||||
|
||||
- `src/obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
|
||||
- `src/obikpartition/src/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
|
||||
- Possibly `src/obicompactvec/` — add column iterator API if not already present
|
||||
- `src/obilayeredmap/` — check if per-column sequential access is exposed on `SrcLayerData`
|
||||
|
||||
|
||||
@@ -770,7 +770,7 @@ reproducible across two runs with warm disk cache), the opposite of
|
||||
query").
|
||||
|
||||
**Root cause, read from source, not measured in isolation:**
|
||||
`KmerPartition::query_partition_with` (`obikpartitionner/src/query_layer.rs:155-220`)
|
||||
`KmerPartition::query_partition_with` (`obikpartition/src/query_layer.rs:155-220`)
|
||||
is architecturally column-major: stage 2 walks `for g in 0..n_cols { for
|
||||
slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly)
|
||||
as the right locality strategy for the packed/columnar formats, where
|
||||
@@ -914,7 +914,7 @@ replaces.
|
||||
`PersistentCompactIntMatrix::nonzero_iter` added the same way (counts
|
||||
not excluded, per the earlier ask) — no native low-effort case, since no
|
||||
sparse count format exists, but on the same primitive, ready for one.
|
||||
- `KmerPartition::query_partition_with` (`obikpartitionner/src/query_layer.rs`):
|
||||
- `KmerPartition::query_partition_with` (`obikpartition/src/query_layer.rs`):
|
||||
stage 2's column-major `for g { for slot { col_value } }` replaced by one
|
||||
`layer.nonzero_iter(&slot_list)` call per layer, format-agnostic.
|
||||
- Tests: `nonzero_iter_matches_dense`, `nonzero_iter_matches_row`, and —
|
||||
|
||||
@@ -303,7 +303,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
|
||||
|
||||
## Implementation
|
||||
|
||||
- **`obikpartitionner::filter::GroupQuorumFilter`** — implements `KmerFilter`
|
||||
- **`obikpartition::filter::GroupQuorumFilter`** — implements `KmerFilter`
|
||||
using pre-computed ingroup and outgroup index vectors. The heavy logic
|
||||
(predicate parsing, three-value evaluation, genome classification) happens
|
||||
once before any iteration; each k-mer row evaluation is a simple index
|
||||
@@ -314,7 +314,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
|
||||
`UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
|
||||
list.
|
||||
|
||||
- **`obikpartitionner::KmerPartition::iter_partition_kmers`** — accepts
|
||||
- **`obikpartition::KmerPartition::iter_partition_kmers`** — accepts
|
||||
`filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
|
||||
the callback. `filter`, `dump`, and `unitig` all go through this single
|
||||
entry point.
|
||||
|
||||
@@ -4,8 +4,8 @@
|
||||
## Code couvert
|
||||
|
||||
- `obikindex/src/merge.rs` — `KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
|
||||
- `obikpartitionner/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
|
||||
- `obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
|
||||
- `obikpartition/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
|
||||
- `obikpartition/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
|
||||
- `obilayeredmap/src/layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
|
||||
- `obicompactvec/src/intmatrix.rs` — `append_column` pour PersistentCompactIntMatrix
|
||||
- `obicompactvec/src/bitmatrix.rs` — `append_column` pour PersistentBitMatrix
|
||||
|
||||
@@ -6,7 +6,7 @@ Kmer indexing per partition proceeds in two phases. The separation is necessary
|
||||
|
||||
### Phase 1 — provisional MPHF + kmer spectrum
|
||||
|
||||
Implemented in `obikpartitionner::KmerPartition::count_kmer()` → `count_partition()`.
|
||||
Implemented in `obikpartition::KmerPartition::count_kmer()` → `count_partition()`.
|
||||
|
||||
1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
|
||||
2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
|
||||
@@ -148,7 +148,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
|
||||
|
||||
There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
|
||||
|
||||
In `obikpartitionner`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
|
||||
In `obikpartition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
|
||||
|
||||
### Membership verification
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
|
||||
- `obikpartitionner/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
|
||||
- `obikpartition/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -9,5 +9,5 @@
|
||||
## Notes
|
||||
|
||||
Document stable (librairie générique, peu de risque de dérive).
|
||||
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartitionner`
|
||||
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartition`
|
||||
ou s'il a été remplacé par Rayon dans certains chemins.
|
||||
|
||||
@@ -16,7 +16,7 @@ explicitly, since two of the names below are misleading.
|
||||
but it holds the *whole* multi-partition structure below — the name
|
||||
suggests "one partition", the value is all of them.
|
||||
- **Partition, the collection (not one partition)** =
|
||||
`obikpartitionner::KmerPartition`. Despite the singular name, this owns
|
||||
`obikpartition::KmerPartition`. Despite the singular name, this owns
|
||||
*every* partition of the index: `root_path`, `n_partitions`, and
|
||||
per-partition accessors that all take an explicit index `i`
|
||||
(`partition_dir(i)`, `index_dir(i)`, `layer_dir(i, l)`,
|
||||
@@ -95,7 +95,7 @@ auto-detection" turns up three unrelated implementations:
|
||||
|---|---|---|---|
|
||||
| `Layer<D>` | `obilayeredmap` | one layer, `D` fixed at compile time | held alive by whoever owns the `Layer`, no policy of its own |
|
||||
| `Mat` | `obikphylo::siblings::cache` | one layer, format resolved per instance from an enum of 3 `Layer<D>` variants | yes, via `PartitionCache` |
|
||||
| `QueryLayer` | `obikpartitionner::query_layer` | one layer, `(MphfLayer, PersistentBitMatrix\|PersistentCompactIntMatrix)` pair, bypasses `Layer<D>` entirely | **no** — opened fresh inside `query_partition_with` on every call |
|
||||
| `QueryLayer` | `obikpartition::query_layer` | one layer, `(MphfLayer, PersistentBitMatrix\|PersistentCompactIntMatrix)` pair, bypasses `Layer<D>` entirely | **no** — opened fresh inside `query_partition_with` on every call |
|
||||
|
||||
`query_partition_with` is `obikmer query`'s normal query path — the one
|
||||
most exposed to repeated cross-partition lookups — and it is the one with
|
||||
@@ -164,7 +164,7 @@ Established in discussion, not yet coded:
|
||||
`open_data`/`layer_dir` — see git history around 2026-08-20). One
|
||||
`LayeredMap<D>` (or its heterogeneous-`D` successor) = one partition.
|
||||
- Nobody currently owns "the collection of partitions" as reusable state.
|
||||
`obikpartitionner::KmerPartition` is the closest candidate — it already
|
||||
`obikpartition::KmerPartition` is the closest candidate — it already
|
||||
owns `n_partitions()`/`part_dir(i)` — but today it is a pure
|
||||
config/path resolver, not a state holder: `dereplicate`, `count_kmer`,
|
||||
`obikindex`'s `distance.rs`/`stats.rs`, and
|
||||
@@ -182,9 +182,9 @@ Established in discussion, not yet coded:
|
||||
2. A cache spanning multiple partitions, keeping (2)'s layer type open per
|
||||
partition per layer, for the whole lifetime of a long-running command —
|
||||
generalising `PartitionCache` minus its sibling-specific parts
|
||||
(`fast_mode`, `find_presence_batch`) — living in `obikpartitionner`
|
||||
(`fast_mode`, `find_presence_batch`) — living in `obikpartition`
|
||||
(owner of the partition dimension) and built on top of (1).
|
||||
3. `obikpartitionner::query_partition_with` — the normal query path,
|
||||
3. `obikpartition::query_partition_with` — the normal query path,
|
||||
currently uncached — becomes a consumer of (2), not just
|
||||
`obikphylo::PartitionCache`.
|
||||
|
||||
@@ -198,7 +198,7 @@ real number behind it for this codebase's scale).
|
||||
|
||||
Groundwork for (1)/(2), landed ahead of the design itself:
|
||||
|
||||
- `KmerPartition` (`obikpartitionner`) gained `partition_dir`/`index_dir`/
|
||||
- `KmerPartition` (`obikpartition`) gained `partition_dir`/`index_dir`/
|
||||
`layer_dir` as the single source of truth for a partition's on-disk
|
||||
layout, replacing per-module duplicated `const INDEX_SUBDIR: &str =
|
||||
"index"` (7 copies) and ad hoc path joins — including one found
|
||||
@@ -266,7 +266,7 @@ storage decisions, now a *third* copy of the same logic to keep in sync)
|
||||
`Mat::open` already reads the result back in the sibling-annex path.
|
||||
Stale comment, not corrected.
|
||||
- **Retracted (2026-08-20)**: an earlier pass through this doc claimed
|
||||
`obikpartitionner::query_layer::QueryLayer::open` had no sparse-format
|
||||
`obikpartition::query_layer::QueryLayer::open` had no sparse-format
|
||||
detection and would silently corrupt reads on a `pack --sparse`d layer.
|
||||
False — `PersistentBitMatrix` (`obicompactvec::bitmatrix::persistent`)
|
||||
is a 4-way enum (`Columnar`/`Packed`/`Sparse`/`Implicit`), not 3-way as
|
||||
|
||||
@@ -3,12 +3,12 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikpartitionner/src/partition.rs` — estimation des paramètres (phase 0)
|
||||
- `obikpartition/src/partition.rs` — estimation des paramètres (phase 0)
|
||||
- `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
|
||||
- `obikpartitionner/src/filter.rs` — déduplication bucket-sort (phase 2)
|
||||
- `obikpartitionner/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
|
||||
- `obikpartition/src/filter.rs` — déduplication bucket-sort (phase 2)
|
||||
- `obikpartition/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
|
||||
- `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
|
||||
- `obikpartitionner/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
|
||||
- `obikpartition/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
|
||||
- `obikindex/src/index.rs` — `build_layers()`, `dereplicate_and_count()`
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -1489,7 +1489,7 @@ instead of a per-partition `partial_*`, run the sequential source sweep:
|
||||
|
||||
```text
|
||||
for p in 0..n_partitions: # OUTER — sequential
|
||||
open source partition p's layers (QueryLayer-style, obikpartitionner)
|
||||
open source partition p's layers (QueryLayer-style, obikpartition)
|
||||
enumerate distinct canonical k-mers of p (one per MPHF slot) with their
|
||||
presence/count vectors # column-major, as query stage 2
|
||||
par_iter over these source k-mers: # INNER — rayon, thread-local tally
|
||||
@@ -1513,7 +1513,7 @@ for p in 0..n_partitions: # OUTER — sequential
|
||||
```
|
||||
|
||||
The inner lookup is precisely `QueryLayer::find_slot` +
|
||||
`col_value(g, slot)` (`obikpartitionner/src/query_layer.rs`) — reuse or factor
|
||||
`col_value(g, slot)` (`obikpartition/src/query_layer.rs`) — reuse or factor
|
||||
out that path rather than reimplementing MPHF access. Enumerating "all distinct
|
||||
k-mers of a partition with their vectors" is the `dump`/`query` stage-2
|
||||
column-major scan already implemented in `dump_layer.rs` /
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikpartitionner/src/partition.rs` — routage par hash de minimiseur, choix des paramètres
|
||||
- `obikpartitionner/src/lib.rs` — structure KmerPartition, nombre de partitions
|
||||
- `obikpartition/src/partition.rs` — routage par hash de minimiseur, choix des paramètres
|
||||
- `obikpartition/src/lib.rs` — structure KmerPartition, nombre de partitions
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
Generated
+4
-4
@@ -1600,7 +1600,7 @@ dependencies = [
|
||||
"indicatif",
|
||||
"ndarray",
|
||||
"obicompactvec",
|
||||
"obikpartitionner",
|
||||
"obikpartition",
|
||||
"obikseq",
|
||||
"obilayeredmap",
|
||||
"obiread",
|
||||
@@ -1626,7 +1626,7 @@ dependencies = [
|
||||
"obidebruinj",
|
||||
"obifastwrite",
|
||||
"obikindex",
|
||||
"obikpartitionner",
|
||||
"obikpartition",
|
||||
"obikphylo",
|
||||
"obikrope",
|
||||
"obikseq",
|
||||
@@ -1648,7 +1648,7 @@ dependencies = [
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "obikpartitionner"
|
||||
name = "obikpartition"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"cacheline-ef",
|
||||
@@ -1685,7 +1685,7 @@ dependencies = [
|
||||
"ndarray",
|
||||
"obicompactvec",
|
||||
"obikindex",
|
||||
"obikpartitionner",
|
||||
"obikpartition",
|
||||
"obikseq",
|
||||
"obilayeredmap",
|
||||
"obipipeline",
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
[workspace]
|
||||
resolver = "3"
|
||||
members = ["obikseq", "obiread", "obiskbuilder", "obifastwrite", "obikmer","obikrope","obipipeline", "obikpartitionner","obiskio","obidebruinj","obilayeredmap", "obicompactvec", "obisys", "obikindex", "obitaxonomy", "obikentropy", "obikphylo"]
|
||||
members = ["obikseq", "obiread", "obiskbuilder", "obifastwrite", "obikmer","obikrope","obipipeline", "obikpartition","obiskio","obidebruinj","obilayeredmap", "obicompactvec", "obisys", "obikindex", "obitaxonomy", "obikentropy", "obikphylo"]
|
||||
[profile.release]
|
||||
debug = 1
|
||||
|
||||
@@ -129,7 +129,7 @@ pub fn pack_bit_matrix(dir: &Path) -> io::Result<()> {
|
||||
// A `matrix.pbmx` can already exist here even though columnar data is
|
||||
// still pending — e.g. copied verbatim from a merge's base source
|
||||
// before this layer was widened with more genome columns (see
|
||||
// `obikpartitionner::merge_partition`). Only skip (re-)packing if the
|
||||
// `obikpartition::merge_partition`). Only skip (re-)packing if the
|
||||
// existing file already reflects the current column count; otherwise
|
||||
// the columnar files are newer and must be (re-)packed, overwriting the
|
||||
// stale one — never silently discarded as "leftover cleanup".
|
||||
|
||||
@@ -232,7 +232,7 @@ pub fn pack_compact_int_matrix(dir: &Path) -> io::Result<()> {
|
||||
// A `matrix.pcmx` can already exist here even though columnar data is
|
||||
// still pending — e.g. copied verbatim from a merge's base source
|
||||
// before this layer was widened with more genome columns (see
|
||||
// `obikpartitionner::merge_partition`). Only skip (re-)packing if the
|
||||
// `obikpartition::merge_partition`). Only skip (re-)packing if the
|
||||
// existing file already reflects the current column count; otherwise
|
||||
// the columnar files are newer and must be (re-)packed, overwriting the
|
||||
// stale one — never silently discarded as "leftover cleanup".
|
||||
|
||||
@@ -5,7 +5,7 @@ edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
obikseq = { path = "../obikseq" }
|
||||
obikpartitionner = { path = "../obikpartitionner" }
|
||||
obikpartition = { path = "../obikpartition" }
|
||||
obitaxonomy = { path = "../obitaxonomy" }
|
||||
obiskio = { path = "../obiskio" }
|
||||
obisys = { path = "../obisys" }
|
||||
|
||||
@@ -5,7 +5,7 @@ use rayon::prelude::*;
|
||||
|
||||
use crate::error::{OKIError, OKIResult};
|
||||
use crate::index::KmerIndex;
|
||||
use obikpartitionner::KmerFilter;
|
||||
use obikpartition::KmerFilter;
|
||||
|
||||
impl KmerIndex {
|
||||
/// Write a CSV table of all indexed kmers to `out`.
|
||||
|
||||
@@ -2,7 +2,7 @@ use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use obikpartitionner::{KmerPartitions, KmerSpectrum, PARTITIONS_SUBDIR};
|
||||
use obikpartition::{KmerPartitions, KmerSpectrum, PARTITIONS_SUBDIR};
|
||||
use obisys::{Reporter, Stage, progress_bar};
|
||||
use rayon::prelude::*;
|
||||
use tracing::info;
|
||||
|
||||
@@ -13,7 +13,7 @@ use crate::index::KmerIndex;
|
||||
use crate::meta::{GenomeInfo, IndexMeta};
|
||||
use crate::state::{IndexState, SENTINEL_INDEXED};
|
||||
|
||||
pub use obikpartitionner::MergeMode;
|
||||
pub use obikpartition::MergeMode;
|
||||
|
||||
// ── per-partition diagnostic record ──────────────────────────────────────────
|
||||
|
||||
@@ -193,7 +193,7 @@ impl KmerIndex {
|
||||
let block_bits = dst.meta.config.block_bits;
|
||||
|
||||
// Pre-build source list once (avoid rebuilding per partition)
|
||||
let srcs: Vec<(&obikpartitionner::KmerPartitions, usize)> = remaining_sources
|
||||
let srcs: Vec<(&obikpartition::KmerPartitions, usize)> = remaining_sources
|
||||
.iter()
|
||||
.map(|s| (&s.partition, s.meta.genomes.len()))
|
||||
.collect();
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use obikpartitionner::GroupQuorumFilter;
|
||||
use obikpartition::GroupQuorumFilter;
|
||||
use obitaxonomy::{TaxPath, TaxPattern};
|
||||
|
||||
use crate::meta::{GenomeInfo, IndexMeta};
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use std::path::Path;
|
||||
|
||||
use obikpartitionner::{KmerFilter, MergeMode};
|
||||
use obikpartition::{KmerFilter, MergeMode};
|
||||
use obisys::{Reporter, Stage, progress_bar};
|
||||
use tracing::info;
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obilayeredmap::{IndexMode, layer::Layer};
|
||||
use obisys::{Reporter, Stage, progress_bar};
|
||||
use std::fs;
|
||||
@@ -71,11 +71,11 @@ impl KmerIndex {
|
||||
|
||||
/// Process all layers of one partition's index directory.
|
||||
fn reindex_partition(
|
||||
partition: &KmerPartitions
|
||||
, i: usize
|
||||
, target: &IndexMode
|
||||
, block_bits: u,
|
||||
8) -> OKIResult<()> {
|
||||
partition: &KmerPartitions,
|
||||
i: usize,
|
||||
target: &IndexMode,
|
||||
block_bits: u8,
|
||||
) -> OKIResult<()> {
|
||||
if !partition.index_dir(i).exists() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use std::path::Path;
|
||||
|
||||
use obikpartitionner::{KmerPartitions, OutputCol};
|
||||
use obikpartition::{KmerPartitions, OutputCol};
|
||||
use obisys::{Reporter, Stage, progress_bar};
|
||||
use tracing::info;
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ obifastwrite = { path = "../obifastwrite" }
|
||||
obidebruinj = { path = "../obidebruinj" }
|
||||
obipipeline = { path = "../obipipeline" }
|
||||
obikrope = { path = "../obikrope" }
|
||||
obikpartitionner = { path = "../obikpartitionner" }
|
||||
obikpartition = { path = "../obikpartition" }
|
||||
obisys = { path = "../obisys" }
|
||||
obiskio = { path = "../obiskio" }
|
||||
obikindex = { path = "../obikindex", default-features = false }
|
||||
|
||||
@@ -2,7 +2,7 @@ use std::path::PathBuf;
|
||||
|
||||
use clap::Args;
|
||||
use obikindex::{KmerIndex, MergeMode};
|
||||
use obikpartitionner::filter::{MaxTotalCount, MinComplexity, MinTotalCount};
|
||||
use obikpartition::filter::{MaxTotalCount, MinComplexity, MinTotalCount};
|
||||
use obisys::Reporter;
|
||||
use tracing::info;
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use clap::Args;
|
||||
use obikindex::{GroupFilterParams, IndexMeta, MetaPred};
|
||||
use obikpartitionner::KmerFilter;
|
||||
use obikpartition::KmerFilter;
|
||||
|
||||
/// CLI args for ingroup/outgroup filtering — embeddable in any command via `#[command(flatten)]`.
|
||||
#[derive(Args)]
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use obikpartitionner::KmerDesc;
|
||||
use obikpartition::KmerDesc;
|
||||
use obikseq::CanonicalKmer;
|
||||
use obiread::record::SeqRecord;
|
||||
use obiskbuilder::SuperKmerIter;
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
use std::time::Instant;
|
||||
|
||||
use obikindex::KmerIndex;
|
||||
use obikpartitionner::{KmerDesc, QueryHit, QueryStats};
|
||||
use obikpartition::{KmerDesc, QueryHit, QueryStats};
|
||||
use obikrope::Rope;
|
||||
use obikseq::CanonicalKmer;
|
||||
use obiread::record::parse_chunk;
|
||||
|
||||
@@ -3,7 +3,7 @@ use std::path::PathBuf;
|
||||
|
||||
use clap::{Args, ValueEnum};
|
||||
use obikindex::{IndexMeta, KmerIndex};
|
||||
use obikpartitionner::{AggOp, OutputCol};
|
||||
use obikpartition::{AggOp, OutputCol};
|
||||
use obisys::Reporter;
|
||||
use tracing::info;
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@ use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU32, AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obipipeline::{ThrottleGuard, Throttled, throttle};
|
||||
use obiread::NucPage;
|
||||
use obisys::spinner;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[package]
|
||||
name = "obikpartitionner"
|
||||
name = "obikpartition"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
+1
-1
@@ -197,7 +197,7 @@ impl KmerPartitions {
|
||||
}
|
||||
|
||||
/// Partition `i`'s metadata (layer count, evidence mode) — the single
|
||||
/// entry point for this, so that callers outside `obikpartitionner`
|
||||
/// entry point for this, so that callers outside `obikpartition`
|
||||
/// never need to know it's a `meta.json` loaded via
|
||||
/// `obilayeredmap::meta::PartitionMeta`, nor handle its own recovery
|
||||
/// path for indexes built before that file existed (see
|
||||
+2
-2
@@ -45,7 +45,7 @@ fn query_stats_default_is_zero() {
|
||||
fn query_partition_with_missing_index_dir_returns_default_stats() {
|
||||
let tmp = tempfile::tempdir().expect("tempdir");
|
||||
let partition =
|
||||
KmerPartitions::create(tmp.path().join("idx"), 2, 21, 9, false.expect("create partition");
|
||||
KmerPartitions::create(tmp.path().join("idx"), 2, 21, 9, false).expect("create partition");
|
||||
|
||||
let mut kmers: HashMap<CanonicalKmer, Vec<KmerDesc>> = HashMap::new();
|
||||
// Any well-formed canonical k-mer works here — the call must return
|
||||
@@ -66,7 +66,7 @@ fn query_partition_with_missing_index_dir_returns_default_stats() {
|
||||
fn query_partition_with_empty_kmers_is_a_noop() {
|
||||
let tmp = tempfile::tempdir().expect("tempdir");
|
||||
let partition =
|
||||
KmerPartitions::create(tmp.path().join("idx"), 2, 21, 9, false.expect("create partition");
|
||||
KmerPartitions::create(tmp.path().join("idx"), 2, 21, 9, false).expect("create partition");
|
||||
|
||||
let kmers: HashMap<CanonicalKmer, Vec<KmerDesc>> = HashMap::new();
|
||||
let stats = partition
|
||||
@@ -6,7 +6,7 @@ edition = "2024"
|
||||
[dependencies]
|
||||
obikindex = { path = "../obikindex", default-features = false }
|
||||
obikseq = { path = "../obikseq" }
|
||||
obikpartitionner = { path = "../obikpartitionner" }
|
||||
obikpartition = { path = "../obikpartition" }
|
||||
obiskio = { path = "../obiskio" }
|
||||
obisys = { path = "../obisys" }
|
||||
obicompactvec = { path = "../obicompactvec" }
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obisys::progress_bar;
|
||||
|
||||
use obikindex::KmerIndex;
|
||||
|
||||
@@ -4,7 +4,7 @@ use std::sync::atomic::Ordering;
|
||||
|
||||
use rayon::prelude::*;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obikseq::CanonicalKmer;
|
||||
use obilayeredmap::MphfLayer;
|
||||
use obilayeredmap::meta::IndexMode;
|
||||
|
||||
@@ -3,7 +3,7 @@ use rayon::prelude::*;
|
||||
use std::path::Path;
|
||||
|
||||
use obicompactvec::{PersistentBitMatrix, PersistentCompactIntMatrix};
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obikseq::CanonicalKmer;
|
||||
use obilayeredmap::meta::IndexMode;
|
||||
use obilayeredmap::{Layer, OLMResult};
|
||||
|
||||
@@ -2,7 +2,7 @@ use std::sync::Arc;
|
||||
|
||||
use ndarray::Array2;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obisys::progress_bar;
|
||||
|
||||
use obikindex::KmerIndex;
|
||||
|
||||
@@ -2,7 +2,7 @@ use std::sync::Arc;
|
||||
|
||||
use ndarray::Array2;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obisys::progress_bar;
|
||||
|
||||
use obikindex::KmerIndex;
|
||||
|
||||
@@ -13,7 +13,7 @@ use std::io::{BufWriter, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obisys::progress_bar;
|
||||
|
||||
use obikindex::KmerIndex;
|
||||
|
||||
@@ -29,7 +29,7 @@ use std::sync::Arc;
|
||||
|
||||
use ndarray::Array2;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obisys::progress_bar;
|
||||
|
||||
use obikindex::KmerIndex;
|
||||
|
||||
@@ -2,7 +2,7 @@ use std::sync::Arc;
|
||||
|
||||
use rayon::prelude::*;
|
||||
|
||||
use obikpartitionner::KmerPartitions;
|
||||
use obikpartition::KmerPartitions;
|
||||
use obisys::progress_bar;
|
||||
|
||||
use obikindex::KmerIndex;
|
||||
|
||||
@@ -33,7 +33,7 @@ pub trait LayerData: Sized {
|
||||
///
|
||||
/// `obilayeredmap` operates within a single partition's index root; it has
|
||||
/// no notion of "partition" at all. Turning a partition number into that
|
||||
/// root is `obikpartitionner::KmerPartition::part_dir`'s job, one layer up
|
||||
/// root is `obikpartition::KmerPartition::part_dir`'s job, one layer up
|
||||
/// — callers here only ever name a layer *number*, never build the path
|
||||
/// themselves.
|
||||
pub fn layer_dir(root: &Path, i: usize) -> PathBuf {
|
||||
|
||||
Reference in New Issue
Block a user