diff --git a/DevDocMD/implementation/mphf_batch_queries.md b/DevDocMD/implementation/mphf_batch_queries.md index 55a3e473..d4fa0e48 100644 --- a/DevDocMD/implementation/mphf_batch_queries.md +++ b/DevDocMD/implementation/mphf_batch_queries.md @@ -267,3 +267,35 @@ One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slo - **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %. Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture. + +## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15) + +Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`. + +| | push | degrees | unitigs | unitigs produced | nucleotides | +|---|---|---|---|---|---| +| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M | +| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M | +| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** | + +Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles). + +**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision). + +**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was). + +**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig). + +Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %). + +**End-to-end** (current binary vs previous one, same inputs): + +| | before | after | +|---|---|---| +| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) | +| … `index` stage | 114.6 s | **59.2 s** (−48 %) | +| … user CPU | 1 870 s | 1 304 s | +| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB | +| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s | + +Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs. diff --git a/src/obidebruinj/src/debruijn/graph.rs b/src/obidebruinj/src/debruijn/graph.rs index 901848c9..3771e102 100644 --- a/src/obidebruinj/src/debruijn/graph.rs +++ b/src/obidebruinj/src/debruijn/graph.rs @@ -1,19 +1,19 @@ -use hashbrown::HashMap; +use hashbrown::{HashMap, HashSet}; use obikseq::k; use obikseq::{CanonicalKmer, Unitig}; #[cfg(not(any(test, feature = "test-utils")))] use rayon::iter::{IntoParallelRefIterator, ParallelIterator}; use std::cell::RefCell; use std::sync::atomic::{AtomicU8, Ordering}; -use xxhash_rust::xxh3::Xxh3Builder; +use super::kmer_hash::KmerHashBuilder; use super::node::{IS_VISITED_MASK, Node}; use super::unitig_iter::UnitigNucIter; use super::walk::WalkState; // ── Types ───────────────────────────────────────────────────────────────────── -pub(super) type FastHashMap = HashMap; +pub(super) type FastHashMap = HashMap; // ── GraphDeBruijn ───────────────────────────────────────────────────────────── @@ -24,7 +24,7 @@ pub struct GraphDeBruijn { impl GraphDeBruijn { pub fn new() -> Self { Self { - nodes: FastHashMap::with_hasher(Xxh3Builder::new()), + nodes: FastHashMap::with_hasher(KmerHashBuilder), } } @@ -40,7 +40,7 @@ impl GraphDeBruijn { pub fn with_capacity(capacity: usize) -> Self { Self { - nodes: FastHashMap::with_capacity_and_hasher(capacity, Xxh3Builder::new()), + nodes: FastHashMap::with_capacity_and_hasher(capacity, KmerHashBuilder), } } @@ -79,7 +79,7 @@ impl GraphDeBruijn { if node.is_visited() { return; } - if self.is_start(*kmer, node) { + if self.is_start(*kmer, node, atomic, true) || self.is_start(*kmer, node, atomic, false) { node.set_start(); atomic.store(node.0, Ordering::Relaxed); } @@ -98,45 +98,40 @@ impl GraphDeBruijn { } } - pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState { - let node = Node( - self.nodes - .get(&kmer) - .expect("Cannot start a walk from not indexed kmer") - .load(Ordering::Relaxed), - ); + pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState<'_> { + let atomic = self + .nodes + .get(&kmer) + .expect("Cannot start a walk from not indexed kmer"); + let node = Node(atomic.load(Ordering::Relaxed)); debug_assert!( !node.is_visited(), "Cannot start a walk from a visited node" ); - WalkState::new(kmer, node, true) + WalkState::new(kmer, node, true, atomic) } - pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option { - let node = Node(self.nodes.get(&kmer)?.load(Ordering::Relaxed)); + pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option> { + let atomic = self.nodes.get(&kmer)?; + let node = Node(atomic.load(Ordering::Relaxed)); if node.is_visited() { return None; } - Some(WalkState::new(kmer, node, true)) + Some(WalkState::new(kmer, node, true, atomic)) } - pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize) -> Option> { - let old = self - .nodes - .get(&kmer)? - .fetch_or(IS_VISITED_MASK, Ordering::AcqRel); + pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize, direct: bool) -> Option> { + let atomic = self.nodes.get(&kmer)?; + let old = atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel); if old & IS_VISITED_MASK != 0 { return None; } - let start = WalkState::new(kmer, Node(old), true); + let start = WalkState::new(kmer, Node(old), direct, atomic); let next_step = start.walk(self).and_then(|(next_state, nuc)| { - let ext_old = self - .nodes - .get(&next_state.kmer)? - .fetch_or(IS_VISITED_MASK, Ordering::AcqRel); + let ext_old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel); (ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc)) }); - Some(UnitigNucIter::new(self, kmer, k, next_step)) + Some(UnitigNucIter::new(self, kmer, k, direct, next_step)) } pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) { @@ -157,7 +152,8 @@ impl GraphDeBruijn { .then_some(kmer) }) .for_each(|kmer| { - if let Some(iter) = self.unitig_nucleotides(kmer, k) { + let Some(direct) = self.start_direction(kmer) else { return }; + if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) { n_new.fetch_add(1, Ordering::Relaxed); f(iter); } @@ -166,7 +162,8 @@ impl GraphDeBruijn { #[cfg(any(test, feature = "test-utils"))] self.nodes.iter().for_each(|(&kmer, atomic)| { if Node(atomic.load(Ordering::Acquire)).is_start() { - if let Some(iter) = self.unitig_nucleotides(kmer, k) { + let Some(direct) = self.start_direction(kmer) else { return }; + if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) { n_new.fetch_add(1, Ordering::Relaxed); f(iter); } @@ -187,8 +184,8 @@ impl GraphDeBruijn { if node.is_visited() { continue; } - let chain_start = self.find_chain_start(kmer); - if let Some(iter) = self.unitig_nucleotides(chain_start, k) { + let (chain_start, direct) = self.find_chain_start(kmer); + if let Some(iter) = self.unitig_nucleotides(chain_start, k, direct) { n2.fetch_add(1, Ordering::Relaxed); f(iter); } @@ -196,7 +193,7 @@ impl GraphDeBruijn { // Safe because unitig_nucleotides may have visited kmer in the // meantime — in that case it returns None and we skip. if chain_start != kmer { - if let Some(iter) = self.unitig_nucleotides(kmer, k) { + if let Some(iter) = self.unitig_nucleotides(kmer, k, true) { n2.fetch_add(1, Ordering::Relaxed); f(iter); } @@ -216,17 +213,34 @@ impl GraphDeBruijn { self.nodes.extend(other.nodes); } - fn find_chain_start(&self, kmer: CanonicalKmer) -> CanonicalKmer { - let node = Node(self.nodes[&kmer].load(Ordering::Acquire)); - let mut state = WalkState::new(kmer, node, false); - let mut seen = std::collections::HashSet::new(); + /// Walks backwards from `kmer` to the first node of its chain. Returns + /// that node together with the orientation to read it in to walk the chain + /// forwards: the backward walk traverses nodes in the orientation + /// `state.direct`, so the forward one is its opposite. (Dropping the + /// orientation, as this function once did, started the forward walk at the + /// wrong end and emitted a lone-kmer unitig.) On a cycle, `kmer` itself + /// in the canonical orientation. + fn find_chain_start(&self, kmer: CanonicalKmer) -> (CanonicalKmer, bool) { + let atomic = &self.nodes[&kmer]; + let node = Node(atomic.load(Ordering::Acquire)); + let mut state = WalkState::new(kmer, node, false, atomic); + // Most nodes reaching this pass are branch nodes or tips that cannot + // step at all: answer before allocating the cycle-detection set. + let Some((first, _)) = state.walk(self) else { + return (state.kmer, !state.direct); + }; + let mut seen: HashSet<(u64, bool), KmerHashBuilder> = HashSet::with_hasher(KmerHashBuilder); seen.insert((state.kmer.raw(), state.direct)); + if !seen.insert((first.kmer.raw(), first.direct)) { + return (kmer, true); + } + state = first; loop { match state.walk(self) { - None => return state.kmer, + None => return (state.kmer, !state.direct), Some((next, _)) => { if !seen.insert((next.kmer.raw(), next.direct)) { - return kmer; + return (kmer, true); } state = next; } @@ -234,8 +248,23 @@ impl GraphDeBruijn { } } - fn is_start(&self, query: CanonicalKmer, node: Node) -> bool { - !WalkState::new(query, node, true).reachable(self) + /// Orientation in which a node flagged as a start must be read: the + /// canonical one if a unitig starts there that way, else the reverse + /// complement. `None` if it was visited in the meantime. + fn start_direction(&self, kmer: CanonicalKmer) -> Option { + let atomic = self.nodes.get(&kmer)?; + let node = Node(atomic.load(Ordering::Acquire)); + if node.is_visited() { + return None; + } + Some(self.is_start(kmer, node, atomic, true) || !self.is_start(kmer, node, atomic, false)) + } + + /// `true` when `query`, traversed in orientation `direct` (canonical or + /// reverse-complemented), has no unique predecessor — i.e. a unitig + /// starts there when read in that orientation. + fn is_start(&self, query: CanonicalKmer, node: Node, atomic: &AtomicU8, direct: bool) -> bool { + !WalkState::new(query, node, direct, atomic).reachable(self) } pub fn try_for_each_unitig(&self, f: F) -> Result<(), E> diff --git a/src/obidebruinj/src/debruijn/kmer_hash.rs b/src/obidebruinj/src/debruijn/kmer_hash.rs new file mode 100644 index 00000000..cbc3688a --- /dev/null +++ b/src/obidebruinj/src/debruijn/kmer_hash.rs @@ -0,0 +1,55 @@ +//! Hasher for the graph's `HashMap`. +//! +//! The key is one `u64` (the derived `Hash` of a kmer is a single +//! `write_u64`), so the generic streaming hashers used before (xxh3's +//! `Hasher` implementation: buffer, update, finish) cost far more than the +//! hash itself — ≈ 35 % of the active CPU of `index` in the profile of +//! `DevDocMD/implementation/mphf_batch_queries.md`, §17. A 64-bit finaliser +//! (splitmix64's) is a bijection with full avalanche, which is what +//! hashbrown needs from it (low bits for the bucket, top 7 for the tag). + +use std::hash::{BuildHasher, Hasher}; + +#[derive(Clone, Copy, Default)] +pub(super) struct KmerHashBuilder; + +impl BuildHasher for KmerHashBuilder { + type Hasher = KmerHasher; + + #[inline] + fn build_hasher(&self) -> KmerHasher { + KmerHasher(0) + } +} + +pub(super) struct KmerHasher(u64); + +#[inline] +fn mix64(x: u64) -> u64 { + let x = x ^ (x >> 30); + let x = x.wrapping_mul(0xbf58476d1ce4e5b9); + let x = x ^ (x >> 27); + let x = x.wrapping_mul(0x94d049bb133111eb); + x ^ (x >> 31) +} + +impl Hasher for KmerHasher { + #[inline] + fn write_u64(&mut self, x: u64) { + self.0 = mix64(self.0 ^ x); + } + + /// Generic fallback (never taken for a kmer key): folds the bytes 8 at a time. + fn write(&mut self, bytes: &[u8]) { + for chunk in bytes.chunks(8) { + let mut buf = [0u8; 8]; + buf[..chunk.len()].copy_from_slice(chunk); + self.write_u64(u64::from_le_bytes(buf)); + } + } + + #[inline] + fn finish(&self) -> u64 { + self.0 + } +} diff --git a/src/obidebruinj/src/debruijn/mod.rs b/src/obidebruinj/src/debruijn/mod.rs index 126b8ec1..74ba87de 100644 --- a/src/obidebruinj/src/debruijn/mod.rs +++ b/src/obidebruinj/src/debruijn/mod.rs @@ -1,10 +1,11 @@ //! De Bruijn graph over canonical k-mers, built for unitig extraction. //! -//! Submodules: [`node`] (packed per-kmer neighbour/visited/start flags), +//! Submodules: [`kmer_hash`] (hasher of the node table), [`node`] (packed per-kmer neighbour/visited/start flags), //! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself), //! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator). mod graph; +mod kmer_hash; mod node; mod unitig_iter; mod walk; diff --git a/src/obidebruinj/src/debruijn/unitig_iter.rs b/src/obidebruinj/src/debruijn/unitig_iter.rs index 56ca6770..7d4ef523 100644 --- a/src/obidebruinj/src/debruijn/unitig_iter.rs +++ b/src/obidebruinj/src/debruijn/unitig_iter.rs @@ -10,9 +10,11 @@ use super::walk::WalkState; pub struct UnitigNucIter<'a> { graph: &'a GraphDeBruijn, start: CanonicalKmer, + /// Orientation the unitig is read in (`false`: reverse complement of `start`). + direct: bool, pos: usize, k: usize, - next_step: Option<(WalkState, u8)>, + next_step: Option<(WalkState<'a>, u8)>, } impl<'a> UnitigNucIter<'a> { @@ -20,11 +22,13 @@ impl<'a> UnitigNucIter<'a> { graph: &'a GraphDeBruijn, start: CanonicalKmer, k: usize, - next_step: Option<(WalkState, u8)>, + direct: bool, + next_step: Option<(WalkState<'a>, u8)>, ) -> Self { Self { graph, start, + direct, pos: 0, k, next_step, @@ -37,16 +41,16 @@ impl Iterator for UnitigNucIter<'_> { fn next(&mut self) -> Option { if self.pos < self.k { - let nuc = self.start.nucleotide(self.pos); + let nuc = if self.direct { + self.start.nucleotide(self.pos) + } else { + 3 - self.start.nucleotide(self.k - 1 - self.pos) + }; self.pos += 1; Some(nuc) } else if let Some((state, nuc)) = self.next_step.take() { self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| { - let old = self - .graph - .nodes - .get(&next_state.kmer)? - .fetch_or(IS_VISITED_MASK, Ordering::AcqRel); + let old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel); (old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc)) }); Some(nuc) diff --git a/src/obidebruinj/src/debruijn/walk.rs b/src/obidebruinj/src/debruijn/walk.rs index cace3871..61320a73 100644 --- a/src/obidebruinj/src/debruijn/walk.rs +++ b/src/obidebruinj/src/debruijn/walk.rs @@ -1,35 +1,41 @@ use obikseq::{CanonicalKmer, Sequence}; -use std::sync::atomic::Ordering; +use std::sync::atomic::{AtomicU8, Ordering}; use super::graph::GraphDeBruijn; use super::node::Node; -pub struct WalkState { +/// A position in a walk: the node's kmer, its flags as read when the step was +/// taken, and the node's own table entry. Keeping `atomic` means the caller +/// that wants to mark the node visited does not look the kmer up a second +/// time (a random access in a table far larger than the cache). +pub struct WalkState<'g> { pub(super) kmer: CanonicalKmer, pub(super) node: Node, pub(super) direct: bool, + pub(super) atomic: &'g AtomicU8, } -impl WalkState { - pub fn new(kmer: CanonicalKmer, node: Node, direct: bool) -> Self { +impl<'g> WalkState<'g> { + pub fn new(kmer: CanonicalKmer, node: Node, direct: bool, atomic: &'g AtomicU8) -> Self { debug_assert!(!node.is_visited(), "Cannot walk over a visited node"); - Self { kmer, node, direct } + Self { kmer, node, direct, atomic } } - pub fn leavable(&self, graph: &GraphDeBruijn) -> bool { + pub fn leavable(&self, graph: &'g GraphDeBruijn) -> bool { self.walk(graph).is_some() } - pub fn reachable(&self, graph: &GraphDeBruijn) -> bool { + pub fn reachable(&self, graph: &'g GraphDeBruijn) -> bool { WalkState { kmer: self.kmer, node: self.node, direct: !self.direct, + atomic: self.atomic, } .leavable(graph) } - pub fn walk(&self, graph: &GraphDeBruijn) -> Option<(WalkState, u8)> { + pub fn walk(&self, graph: &'g GraphDeBruijn) -> Option<(WalkState<'g>, u8)> { if self.direct { if !self.node.can_extend_right() { return None; @@ -38,7 +44,8 @@ impl WalkState { let next = self.kmer.into_kmer().push_right(nuc); let cnext = next.canonical(); let dnext = next.raw() == cnext.raw(); - let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed)); + let next_atomic = graph.nodes.get(&cnext).unwrap(); + let next_node = Node(next_atomic.load(Ordering::Relaxed)); if next_node.is_visited() { return None; } @@ -52,6 +59,7 @@ impl WalkState { kmer: cnext, node: next_node, direct: dnext, + atomic: next_atomic, }, nuc, )) @@ -63,20 +71,22 @@ impl WalkState { let next = self.kmer.into_kmer().push_left(nuc); let cnext = next.canonical(); let dnext = next.raw() != cnext.raw(); - let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed)); + let next_atomic = graph.nodes.get(&cnext).unwrap(); + let next_node = Node(next_atomic.load(Ordering::Relaxed)); if next_node.is_visited() { return None; } let reachable = if dnext { - next_node.can_extend_right() - } else { next_node.can_extend_left() + } else { + next_node.can_extend_right() }; reachable.then_some(( WalkState { kmer: cnext, node: next_node, direct: dnext, + atomic: next_atomic, }, 3 - nuc, )) diff --git a/src/obidebruinj/src/tests/debruijn.rs b/src/obidebruinj/src/tests/debruijn.rs index 5fb9c4cf..8985c51d 100644 --- a/src/obidebruinj/src/tests/debruijn.rs +++ b/src/obidebruinj/src/tests/debruijn.rs @@ -317,3 +317,86 @@ fn branching_graph_no_kmer_lost_or_duplicated() { ); assert_eq!(got, expected, "k-mer sets differ"); } + +// ── orientation independence ────────────────────────────────────────────── + +// Deterministic pseudo-random ASCII sequence (xorshift), so the tests need no +// RNG dependency. +fn pseudo_random_ascii(seed: u64, len: usize) -> Vec { + let mut st = seed.wrapping_mul(0x9E3779B97F4A7C15) | 1; + (0..len) + .map(|_| { + st ^= st << 13; + st ^= st >> 7; + st ^= st << 17; + b"ACGT"[(st % 4) as usize] + }) + .collect() +} + +#[test] +fn linear_chain_is_one_unitig_whatever_the_orientations() { + // A repeat-free sequence is one maximal unitig. Each k-mer is stored in + // its canonical orientation, so across many sequences the chain's nodes + // take every mix of orientations — including chains whose two ends are + // both stored reverse-complemented, which once came out as several + // unitigs (a lone-kmer unitig per mis-oriented chain start) depending + // only on the table's iteration order. + let k = 21; // long enough that (k-1)-mer collisions (= branches) do not occur + let _guard = lock_k(); + set_k(k); + for seed in 1..=300u64 { + let seq = pseudo_random_ascii(seed, 90); + let kmers = canonical_kmers(&seq); + if kmers.len() != seq.len() - k + 1 { + continue; // accidental internal repeat: not a single chain + } + let g = graph_from_ascii(&seq); + g.compute_degrees_and_mark_starts(); + let unitigs = collect_unitigs(&g); + assert_eq!(unitigs.len(), 1, "seed {seed}: repeat-free sequence must give one unitig"); + assert_eq!(unitigs[0].seql(), seq.len(), "seed {seed}: unitig spans the sequence"); + assert_eq!(kmers_from_unitigs(&unitigs), kmers, "seed {seed}: same k-mers"); + } +} + +#[test] +fn unitigs_partition_the_kmers_of_branching_reads() { + // Reads of a small genome on both strands with substitutions: branches, + // bubbles and tips. Whatever the shape, every k-mer must be covered + // exactly once: sum over unitigs of (seql - k + 1) == number of nodes. + let k = 11; + let _guard = lock_k(); + set_k(k); + let genome = pseudo_random_ascii(7, 600); + let mut g = GraphDeBruijn::new(); + for r in 0..120u64 { + let start = (pseudo_random_ascii(100 + r, 1)[0] as usize * 31 + r as usize * 17) % 500; + let mut read = genome[start..start + 90].to_vec(); + if r % 2 == 1 { + // second strand + read = read + .iter() + .rev() + .map(|&b| match b { + b'A' => b'T', + b'C' => b'G', + b'G' => b'C', + _ => b'A', + }) + .collect(); + } + if r % 3 == 0 { + let at = (r as usize * 13) % read.len(); + read[at] = if read[at] == b'A' { b'C' } else { b'A' }; + } + for i in 0..=read.len() - k { + g.push(Kmer::from_ascii(&read[i..i + k]).unwrap().canonical()); + } + } + g.compute_degrees_and_mark_starts(); + let unitigs = collect_unitigs(&g); + let covered: usize = unitigs.iter().map(|u| u.seql() - k + 1).sum(); + assert_eq!(covered, g.len(), "every k-mer in exactly one unitig"); + assert_eq!(kmers_from_unitigs(&unitigs).len(), g.len(), "no k-mer missing"); +} diff --git a/src/obikindexer/examples/graph_bench.rs b/src/obikindexer/examples/graph_bench.rs new file mode 100644 index 00000000..627c35a2 --- /dev/null +++ b/src/obikindexer/examples/graph_bench.rs @@ -0,0 +1,165 @@ +//! Throwaway benchmark of the de Bruijn graph on a real kmer set: +//! `graph_bench [repeat]` builds `GraphDeBruijn` from the +//! canonical kmers of an existing layer's unitigs (same call sequence as +//! `write_graph_as_unitigs`) and times each phase. +use std::path::PathBuf; +use std::time::Instant; + +use obidebruinj::GraphDeBruijn; +use obikseq::Sequence; +use obiskio::CanonicalKmerIter; +use rayon::prelude::*; +use std::collections::HashSet; + +fn main() { + let args: Vec = std::env::args().collect(); + let path = PathBuf::from(&args[1]); + let k: usize = args[2].parse().unwrap(); + let repeat: usize = args.get(3).map_or(1, |s| s.parse().unwrap()); + obikseq::set_k(k); + obikseq::set_m(11); + let kmers: Vec<_> = if let Ok(n) = std::env::var("SYNTH") { + // synthetic reads: random genome, reads on both strands, 1 % substitutions + let n: usize = n.parse().unwrap(); + let mut st = 0x9E3779B97F4A7C15u64; + let mut rnd = move || { + st ^= st << 13; + st ^= st >> 7; + st ^= st << 17; + st + }; + let genome: Vec = (0..n).map(|_| b"ACGT"[(rnd() % 4) as usize]).collect(); + let mut set = std::collections::BTreeSet::new(); + for _ in 0..(n / 50) { + let start = (rnd() as usize) % (n - 100); + let mut read: Vec = genome[start..start + 100].to_vec(); + for b in read.iter_mut() { + if rnd() % 100 == 0 { + *b = b"ACGT"[(rnd() % 4) as usize]; + } + } + for w in read.windows(k) { + set.insert(obikseq::Kmer::from_ascii(w).unwrap().canonical()); + } + } + set.into_iter().collect() + } else { + CanonicalKmerIter::new(&path).unwrap().collect() + }; + eprintln!("{} kmers", kmers.len()); + // Independent count of maximal unitigs: nodes − mutually-unique links + // (cycles ignored), computed with a plain HashSet — no use of GraphDeBruijn. + if std::env::var("IDEAL").is_ok() { + let set: HashSet = kmers.iter().map(|c| c.raw()).collect(); + let outs = |a: obikseq::Kmer| -> Vec { + (0..4u8) + .map(|x| a.push_right(x)) + .filter(|b| set.contains(&b.canonical().raw())) + .collect() + }; + let links: usize = kmers + .par_iter() + .map(|c| { + let mut n = 0usize; + for o in [c.into_kmer(), c.into_kmer().revcomp()] { + let ob = outs(o); + if ob.len() == 1 { + let b = ob[0]; + // in-neighbours of b = revcomp of out-neighbours of revcomp(b) + if outs(b.revcomp()).len() == 1 { + // count each undirected link once: only from the orientation + // whose canonical raw is the smaller end (ties: both seen once per orientation) + let (ca, cb) = (o.canonical().raw(), b.canonical().raw()); + if ca < cb || (ca == cb && o.raw() <= b.raw()) { + n += 1; + } else if ca > cb { + // seen from the other end too: skip + } + } + } + } + n + }) + .sum(); + eprintln!("IDEAL: nodes={} links={} -> maximal unitigs ≈ {}", kmers.len(), links, kmers.len() - links); + } + if std::env::var("IDEALDUMP").is_ok() { + let set: HashSet = kmers.iter().map(|c| c.raw()).collect(); + let outs = |a: obikseq::Kmer| -> Vec { + (0..4u8).map(|x| a.push_right(x)).filter(|b| set.contains(&b.canonical().raw())).collect() + }; + // oriented kmer a has a mutual predecessor p iff ins(a)=={p} and outs(p)=={a} + let pred = |a: obikseq::Kmer| -> Option { + let ins: Vec = outs(a.revcomp()).into_iter().map(|b| b.revcomp()).collect(); + if ins.len() == 1 && outs(ins[0]).len() == 1 { Some(ins[0]) } else { None } + }; + let mut seen_chains = std::collections::BTreeSet::new(); + for c in &kmers { + for o in [c.into_kmer(), c.into_kmer().revcomp()] { + if pred(o).is_some() { continue; } + let mut text: Vec = (0..k).map(|i| b"ACGT"[o.nucleotide(i) as usize]).collect(); + let mut cur = o; + loop { + let ob = outs(cur); + if ob.len() != 1 { break; } + let b = ob[0]; + if pred(b).is_none() { break; } + text.push(b"ACGT"[b.nucleotide(k - 1) as usize]); + cur = b; + } + let s = String::from_utf8(text).unwrap(); + let rc: String = s.bytes().rev().map(|c| match c { b'A' => 'T', b'C' => 'G', b'G' => 'C', _ => 'A' }).collect(); + seen_chains.insert(if s <= rc { s } else { rc }); + } + } + for u in &seen_chains { println!("IDEALUNITIG {} {}", u.len(), u); } + } + for r in 0..repeat { + let t = Instant::now(); + let mut g = GraphDeBruijn::new(); + for &kmer in &kmers { + g.push(kmer); + } + let t_push = t.elapsed(); + + let t = Instant::now(); + g.compute_degrees_and_mark_starts(); + let t_deg = t.elapsed(); + + let t = Instant::now(); + let n_unitigs = std::sync::atomic::AtomicUsize::new(0); + let n_nuc = std::sync::atomic::AtomicUsize::new(0); + let dump = std::env::var("DUMP").is_ok(); + let seqs = std::sync::Mutex::new(Vec::::new()); + g.for_each_unitig(|it| { + let mut n = 0usize; + let mut text = String::new(); + for nuc in it { + n += 1; + if dump { + text.push(b"ACGT"[nuc as usize] as char); + } + } + if dump { + seqs.lock().unwrap().push(text); + } + n_unitigs.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + n_nuc.fetch_add(n, std::sync::atomic::Ordering::Relaxed); + }); + let t_walk = t.elapsed(); + if dump { + let mut v = seqs.lock().unwrap().clone(); + v.sort(); + for u in v { println!("UNITIG {} {}", u.len(), u); } + } + println!( + "run {r}: nodes={} push={:.2?} degrees={:.2?} unitigs={:.2?} -> {} unitigs, {} nucleotides", + g.len(), + t_push, + t_deg, + t_walk, + n_unitigs.load(std::sync::atomic::Ordering::Relaxed), + n_nuc.load(std::sync::atomic::Ordering::Relaxed) + ); + } +}