Refactor unitig detection and graph traversal logic
Introduce new hashing mechanisms and support for directional unitig checks. Fix fragmentation issues and improve performance metrics through optimized graph construction and new benchmarking utilities.
This commit is contained in:
1 parent
d32d86193f
commit
8468319a63
8 files changed
+440
-61
No files matched your search
@@ -267,3 +267,35 @@ One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slo
|
||||
- **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.
|
||||
|
||||
Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture.
|
||||
|
||||
## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)
|
||||
|
||||
Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench <unitigs.bin> <k> [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`.
|
||||
|
||||
| | push | degrees | unitigs | unitigs produced | nucleotides |
|
||||
|---|---|---|---|---|---|
|
||||
| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M |
|
||||
| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M |
|
||||
| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** |
|
||||
|
||||
Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).
|
||||
|
||||
**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision).
|
||||
|
||||
**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).
|
||||
|
||||
**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig).
|
||||
|
||||
Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).
|
||||
|
||||
**End-to-end** (current binary vs previous one, same inputs):
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) |
|
||||
| … `index` stage | 114.6 s | **59.2 s** (−48 %) |
|
||||
| … user CPU | 1 870 s | 1 304 s |
|
||||
| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB |
|
||||
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
|
||||
|
||||
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
|
||||
@@ -1,19 +1,19 @@
|
||||
use hashbrown::HashMap;
|
||||
use hashbrown::{HashMap, HashSet};
|
||||
use obikseq::k;
|
||||
use obikseq::{CanonicalKmer, Unitig};
|
||||
#[cfg(not(any(test, feature = "test-utils")))]
|
||||
use rayon::iter::{IntoParallelRefIterator, ParallelIterator};
|
||||
use std::cell::RefCell;
|
||||
use std::sync::atomic::{AtomicU8, Ordering};
|
||||
use xxhash_rust::xxh3::Xxh3Builder;
|
||||
|
||||
use super::kmer_hash::KmerHashBuilder;
|
||||
use super::node::{IS_VISITED_MASK, Node};
|
||||
use super::unitig_iter::UnitigNucIter;
|
||||
use super::walk::WalkState;
|
||||
|
||||
// ── Types ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
pub(super) type FastHashMap<K, V> = HashMap<K, V, Xxh3Builder>;
|
||||
pub(super) type FastHashMap<K, V> = HashMap<K, V, KmerHashBuilder>;
|
||||
|
||||
// ── GraphDeBruijn ─────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -24,7 +24,7 @@ pub struct GraphDeBruijn {
|
||||
impl GraphDeBruijn {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
nodes: FastHashMap::with_hasher(Xxh3Builder::new()),
|
||||
nodes: FastHashMap::with_hasher(KmerHashBuilder),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,7 +40,7 @@ impl GraphDeBruijn {
|
||||
|
||||
pub fn with_capacity(capacity: usize) -> Self {
|
||||
Self {
|
||||
nodes: FastHashMap::with_capacity_and_hasher(capacity, Xxh3Builder::new()),
|
||||
nodes: FastHashMap::with_capacity_and_hasher(capacity, KmerHashBuilder),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,7 +79,7 @@ impl GraphDeBruijn {
|
||||
if node.is_visited() {
|
||||
return;
|
||||
}
|
||||
if self.is_start(*kmer, node) {
|
||||
if self.is_start(*kmer, node, atomic, true) || self.is_start(*kmer, node, atomic, false) {
|
||||
node.set_start();
|
||||
atomic.store(node.0, Ordering::Relaxed);
|
||||
}
|
||||
@@ -98,45 +98,40 @@ impl GraphDeBruijn {
|
||||
}
|
||||
}
|
||||
|
||||
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState {
|
||||
let node = Node(
|
||||
self.nodes
|
||||
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState<'_> {
|
||||
let atomic = self
|
||||
.nodes
|
||||
.get(&kmer)
|
||||
.expect("Cannot start a walk from not indexed kmer")
|
||||
.load(Ordering::Relaxed),
|
||||
);
|
||||
.expect("Cannot start a walk from not indexed kmer");
|
||||
let node = Node(atomic.load(Ordering::Relaxed));
|
||||
debug_assert!(
|
||||
!node.is_visited(),
|
||||
"Cannot start a walk from a visited node"
|
||||
);
|
||||
WalkState::new(kmer, node, true)
|
||||
WalkState::new(kmer, node, true, atomic)
|
||||
}
|
||||
|
||||
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState> {
|
||||
let node = Node(self.nodes.get(&kmer)?.load(Ordering::Relaxed));
|
||||
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState<'_>> {
|
||||
let atomic = self.nodes.get(&kmer)?;
|
||||
let node = Node(atomic.load(Ordering::Relaxed));
|
||||
if node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
Some(WalkState::new(kmer, node, true))
|
||||
Some(WalkState::new(kmer, node, true, atomic))
|
||||
}
|
||||
|
||||
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize) -> Option<UnitigNucIter<'_>> {
|
||||
let old = self
|
||||
.nodes
|
||||
.get(&kmer)?
|
||||
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize, direct: bool) -> Option<UnitigNucIter<'_>> {
|
||||
let atomic = self.nodes.get(&kmer)?;
|
||||
let old = atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
if old & IS_VISITED_MASK != 0 {
|
||||
return None;
|
||||
}
|
||||
let start = WalkState::new(kmer, Node(old), true);
|
||||
let start = WalkState::new(kmer, Node(old), direct, atomic);
|
||||
let next_step = start.walk(self).and_then(|(next_state, nuc)| {
|
||||
let ext_old = self
|
||||
.nodes
|
||||
.get(&next_state.kmer)?
|
||||
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
let ext_old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
(ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc))
|
||||
});
|
||||
Some(UnitigNucIter::new(self, kmer, k, next_step))
|
||||
Some(UnitigNucIter::new(self, kmer, k, direct, next_step))
|
||||
}
|
||||
|
||||
pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) {
|
||||
@@ -157,7 +152,8 @@ impl GraphDeBruijn {
|
||||
.then_some(kmer)
|
||||
})
|
||||
.for_each(|kmer| {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
|
||||
let Some(direct) = self.start_direction(kmer) else { return };
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
|
||||
n_new.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -166,7 +162,8 @@ impl GraphDeBruijn {
|
||||
#[cfg(any(test, feature = "test-utils"))]
|
||||
self.nodes.iter().for_each(|(&kmer, atomic)| {
|
||||
if Node(atomic.load(Ordering::Acquire)).is_start() {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
|
||||
let Some(direct) = self.start_direction(kmer) else { return };
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
|
||||
n_new.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -187,8 +184,8 @@ impl GraphDeBruijn {
|
||||
if node.is_visited() {
|
||||
continue;
|
||||
}
|
||||
let chain_start = self.find_chain_start(kmer);
|
||||
if let Some(iter) = self.unitig_nucleotides(chain_start, k) {
|
||||
let (chain_start, direct) = self.find_chain_start(kmer);
|
||||
if let Some(iter) = self.unitig_nucleotides(chain_start, k, direct) {
|
||||
n2.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -196,7 +193,7 @@ impl GraphDeBruijn {
|
||||
// Safe because unitig_nucleotides may have visited kmer in the
|
||||
// meantime — in that case it returns None and we skip.
|
||||
if chain_start != kmer {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k, true) {
|
||||
n2.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -216,17 +213,34 @@ impl GraphDeBruijn {
|
||||
self.nodes.extend(other.nodes);
|
||||
}
|
||||
|
||||
fn find_chain_start(&self, kmer: CanonicalKmer) -> CanonicalKmer {
|
||||
let node = Node(self.nodes[&kmer].load(Ordering::Acquire));
|
||||
let mut state = WalkState::new(kmer, node, false);
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
/// Walks backwards from `kmer` to the first node of its chain. Returns
|
||||
/// that node together with the orientation to read it in to walk the chain
|
||||
/// forwards: the backward walk traverses nodes in the orientation
|
||||
/// `state.direct`, so the forward one is its opposite. (Dropping the
|
||||
/// orientation, as this function once did, started the forward walk at the
|
||||
/// wrong end and emitted a lone-kmer unitig.) On a cycle, `kmer` itself
|
||||
/// in the canonical orientation.
|
||||
fn find_chain_start(&self, kmer: CanonicalKmer) -> (CanonicalKmer, bool) {
|
||||
let atomic = &self.nodes[&kmer];
|
||||
let node = Node(atomic.load(Ordering::Acquire));
|
||||
let mut state = WalkState::new(kmer, node, false, atomic);
|
||||
// Most nodes reaching this pass are branch nodes or tips that cannot
|
||||
// step at all: answer before allocating the cycle-detection set.
|
||||
let Some((first, _)) = state.walk(self) else {
|
||||
return (state.kmer, !state.direct);
|
||||
};
|
||||
let mut seen: HashSet<(u64, bool), KmerHashBuilder> = HashSet::with_hasher(KmerHashBuilder);
|
||||
seen.insert((state.kmer.raw(), state.direct));
|
||||
if !seen.insert((first.kmer.raw(), first.direct)) {
|
||||
return (kmer, true);
|
||||
}
|
||||
state = first;
|
||||
loop {
|
||||
match state.walk(self) {
|
||||
None => return state.kmer,
|
||||
None => return (state.kmer, !state.direct),
|
||||
Some((next, _)) => {
|
||||
if !seen.insert((next.kmer.raw(), next.direct)) {
|
||||
return kmer;
|
||||
return (kmer, true);
|
||||
}
|
||||
state = next;
|
||||
}
|
||||
@@ -234,8 +248,23 @@ impl GraphDeBruijn {
|
||||
}
|
||||
}
|
||||
|
||||
fn is_start(&self, query: CanonicalKmer, node: Node) -> bool {
|
||||
!WalkState::new(query, node, true).reachable(self)
|
||||
/// Orientation in which a node flagged as a start must be read: the
|
||||
/// canonical one if a unitig starts there that way, else the reverse
|
||||
/// complement. `None` if it was visited in the meantime.
|
||||
fn start_direction(&self, kmer: CanonicalKmer) -> Option<bool> {
|
||||
let atomic = self.nodes.get(&kmer)?;
|
||||
let node = Node(atomic.load(Ordering::Acquire));
|
||||
if node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
Some(self.is_start(kmer, node, atomic, true) || !self.is_start(kmer, node, atomic, false))
|
||||
}
|
||||
|
||||
/// `true` when `query`, traversed in orientation `direct` (canonical or
|
||||
/// reverse-complemented), has no unique predecessor — i.e. a unitig
|
||||
/// starts there when read in that orientation.
|
||||
fn is_start(&self, query: CanonicalKmer, node: Node, atomic: &AtomicU8, direct: bool) -> bool {
|
||||
!WalkState::new(query, node, direct, atomic).reachable(self)
|
||||
}
|
||||
|
||||
pub fn try_for_each_unitig<E, F>(&self, f: F) -> Result<(), E>
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
//! Hasher for the graph's `HashMap<CanonicalKmer, _>`.
|
||||
//!
|
||||
//! The key is one `u64` (the derived `Hash` of a kmer is a single
|
||||
//! `write_u64`), so the generic streaming hashers used before (xxh3's
|
||||
//! `Hasher` implementation: buffer, update, finish) cost far more than the
|
||||
//! hash itself — ≈ 35 % of the active CPU of `index` in the profile of
|
||||
//! `DevDocMD/implementation/mphf_batch_queries.md`, §17. A 64-bit finaliser
|
||||
//! (splitmix64's) is a bijection with full avalanche, which is what
|
||||
//! hashbrown needs from it (low bits for the bucket, top 7 for the tag).
|
||||
|
||||
use std::hash::{BuildHasher, Hasher};
|
||||
|
||||
#[derive(Clone, Copy, Default)]
|
||||
pub(super) struct KmerHashBuilder;
|
||||
|
||||
impl BuildHasher for KmerHashBuilder {
|
||||
type Hasher = KmerHasher;
|
||||
|
||||
#[inline]
|
||||
fn build_hasher(&self) -> KmerHasher {
|
||||
KmerHasher(0)
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) struct KmerHasher(u64);
|
||||
|
||||
#[inline]
|
||||
fn mix64(x: u64) -> u64 {
|
||||
let x = x ^ (x >> 30);
|
||||
let x = x.wrapping_mul(0xbf58476d1ce4e5b9);
|
||||
let x = x ^ (x >> 27);
|
||||
let x = x.wrapping_mul(0x94d049bb133111eb);
|
||||
x ^ (x >> 31)
|
||||
}
|
||||
|
||||
impl Hasher for KmerHasher {
|
||||
#[inline]
|
||||
fn write_u64(&mut self, x: u64) {
|
||||
self.0 = mix64(self.0 ^ x);
|
||||
}
|
||||
|
||||
/// Generic fallback (never taken for a kmer key): folds the bytes 8 at a time.
|
||||
fn write(&mut self, bytes: &[u8]) {
|
||||
for chunk in bytes.chunks(8) {
|
||||
let mut buf = [0u8; 8];
|
||||
buf[..chunk.len()].copy_from_slice(chunk);
|
||||
self.write_u64(u64::from_le_bytes(buf));
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn finish(&self) -> u64 {
|
||||
self.0
|
||||
}
|
||||
}
|
||||
@@ -1,10 +1,11 @@
|
||||
//! De Bruijn graph over canonical k-mers, built for unitig extraction.
|
||||
//!
|
||||
//! Submodules: [`node`] (packed per-kmer neighbour/visited/start flags),
|
||||
//! Submodules: [`kmer_hash`] (hasher of the node table), [`node`] (packed per-kmer neighbour/visited/start flags),
|
||||
//! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself),
|
||||
//! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator).
|
||||
|
||||
mod graph;
|
||||
mod kmer_hash;
|
||||
mod node;
|
||||
mod unitig_iter;
|
||||
mod walk;
|
||||
|
||||
@@ -10,9 +10,11 @@ use super::walk::WalkState;
|
||||
pub struct UnitigNucIter<'a> {
|
||||
graph: &'a GraphDeBruijn,
|
||||
start: CanonicalKmer,
|
||||
/// Orientation the unitig is read in (`false`: reverse complement of `start`).
|
||||
direct: bool,
|
||||
pos: usize,
|
||||
k: usize,
|
||||
next_step: Option<(WalkState, u8)>,
|
||||
next_step: Option<(WalkState<'a>, u8)>,
|
||||
}
|
||||
|
||||
impl<'a> UnitigNucIter<'a> {
|
||||
@@ -20,11 +22,13 @@ impl<'a> UnitigNucIter<'a> {
|
||||
graph: &'a GraphDeBruijn,
|
||||
start: CanonicalKmer,
|
||||
k: usize,
|
||||
next_step: Option<(WalkState, u8)>,
|
||||
direct: bool,
|
||||
next_step: Option<(WalkState<'a>, u8)>,
|
||||
) -> Self {
|
||||
Self {
|
||||
graph,
|
||||
start,
|
||||
direct,
|
||||
pos: 0,
|
||||
k,
|
||||
next_step,
|
||||
@@ -37,16 +41,16 @@ impl Iterator for UnitigNucIter<'_> {
|
||||
|
||||
fn next(&mut self) -> Option<u8> {
|
||||
if self.pos < self.k {
|
||||
let nuc = self.start.nucleotide(self.pos);
|
||||
let nuc = if self.direct {
|
||||
self.start.nucleotide(self.pos)
|
||||
} else {
|
||||
3 - self.start.nucleotide(self.k - 1 - self.pos)
|
||||
};
|
||||
self.pos += 1;
|
||||
Some(nuc)
|
||||
} else if let Some((state, nuc)) = self.next_step.take() {
|
||||
self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| {
|
||||
let old = self
|
||||
.graph
|
||||
.nodes
|
||||
.get(&next_state.kmer)?
|
||||
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
let old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
(old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc))
|
||||
});
|
||||
Some(nuc)
|
||||
|
||||
@@ -1,35 +1,41 @@
|
||||
use obikseq::{CanonicalKmer, Sequence};
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::sync::atomic::{AtomicU8, Ordering};
|
||||
|
||||
use super::graph::GraphDeBruijn;
|
||||
use super::node::Node;
|
||||
|
||||
pub struct WalkState {
|
||||
/// A position in a walk: the node's kmer, its flags as read when the step was
|
||||
/// taken, and the node's own table entry. Keeping `atomic` means the caller
|
||||
/// that wants to mark the node visited does not look the kmer up a second
|
||||
/// time (a random access in a table far larger than the cache).
|
||||
pub struct WalkState<'g> {
|
||||
pub(super) kmer: CanonicalKmer,
|
||||
pub(super) node: Node,
|
||||
pub(super) direct: bool,
|
||||
pub(super) atomic: &'g AtomicU8,
|
||||
}
|
||||
|
||||
impl WalkState {
|
||||
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool) -> Self {
|
||||
impl<'g> WalkState<'g> {
|
||||
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool, atomic: &'g AtomicU8) -> Self {
|
||||
debug_assert!(!node.is_visited(), "Cannot walk over a visited node");
|
||||
Self { kmer, node, direct }
|
||||
Self { kmer, node, direct, atomic }
|
||||
}
|
||||
|
||||
pub fn leavable(&self, graph: &GraphDeBruijn) -> bool {
|
||||
pub fn leavable(&self, graph: &'g GraphDeBruijn) -> bool {
|
||||
self.walk(graph).is_some()
|
||||
}
|
||||
|
||||
pub fn reachable(&self, graph: &GraphDeBruijn) -> bool {
|
||||
pub fn reachable(&self, graph: &'g GraphDeBruijn) -> bool {
|
||||
WalkState {
|
||||
kmer: self.kmer,
|
||||
node: self.node,
|
||||
direct: !self.direct,
|
||||
atomic: self.atomic,
|
||||
}
|
||||
.leavable(graph)
|
||||
}
|
||||
|
||||
pub fn walk(&self, graph: &GraphDeBruijn) -> Option<(WalkState, u8)> {
|
||||
pub fn walk(&self, graph: &'g GraphDeBruijn) -> Option<(WalkState<'g>, u8)> {
|
||||
if self.direct {
|
||||
if !self.node.can_extend_right() {
|
||||
return None;
|
||||
@@ -38,7 +44,8 @@ impl WalkState {
|
||||
let next = self.kmer.into_kmer().push_right(nuc);
|
||||
let cnext = next.canonical();
|
||||
let dnext = next.raw() == cnext.raw();
|
||||
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed));
|
||||
let next_atomic = graph.nodes.get(&cnext).unwrap();
|
||||
let next_node = Node(next_atomic.load(Ordering::Relaxed));
|
||||
if next_node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
@@ -52,6 +59,7 @@ impl WalkState {
|
||||
kmer: cnext,
|
||||
node: next_node,
|
||||
direct: dnext,
|
||||
atomic: next_atomic,
|
||||
},
|
||||
nuc,
|
||||
))
|
||||
@@ -63,20 +71,22 @@ impl WalkState {
|
||||
let next = self.kmer.into_kmer().push_left(nuc);
|
||||
let cnext = next.canonical();
|
||||
let dnext = next.raw() != cnext.raw();
|
||||
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed));
|
||||
let next_atomic = graph.nodes.get(&cnext).unwrap();
|
||||
let next_node = Node(next_atomic.load(Ordering::Relaxed));
|
||||
if next_node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
let reachable = if dnext {
|
||||
next_node.can_extend_right()
|
||||
} else {
|
||||
next_node.can_extend_left()
|
||||
} else {
|
||||
next_node.can_extend_right()
|
||||
};
|
||||
reachable.then_some((
|
||||
WalkState {
|
||||
kmer: cnext,
|
||||
node: next_node,
|
||||
direct: dnext,
|
||||
atomic: next_atomic,
|
||||
},
|
||||
3 - nuc,
|
||||
))
|
||||
|
||||
@@ -317,3 +317,86 @@ fn branching_graph_no_kmer_lost_or_duplicated() {
|
||||
);
|
||||
assert_eq!(got, expected, "k-mer sets differ");
|
||||
}
|
||||
|
||||
// ── orientation independence ──────────────────────────────────────────────
|
||||
|
||||
// Deterministic pseudo-random ASCII sequence (xorshift), so the tests need no
|
||||
// RNG dependency.
|
||||
fn pseudo_random_ascii(seed: u64, len: usize) -> Vec<u8> {
|
||||
let mut st = seed.wrapping_mul(0x9E3779B97F4A7C15) | 1;
|
||||
(0..len)
|
||||
.map(|_| {
|
||||
st ^= st << 13;
|
||||
st ^= st >> 7;
|
||||
st ^= st << 17;
|
||||
b"ACGT"[(st % 4) as usize]
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn linear_chain_is_one_unitig_whatever_the_orientations() {
|
||||
// A repeat-free sequence is one maximal unitig. Each k-mer is stored in
|
||||
// its canonical orientation, so across many sequences the chain's nodes
|
||||
// take every mix of orientations — including chains whose two ends are
|
||||
// both stored reverse-complemented, which once came out as several
|
||||
// unitigs (a lone-kmer unitig per mis-oriented chain start) depending
|
||||
// only on the table's iteration order.
|
||||
let k = 21; // long enough that (k-1)-mer collisions (= branches) do not occur
|
||||
let _guard = lock_k();
|
||||
set_k(k);
|
||||
for seed in 1..=300u64 {
|
||||
let seq = pseudo_random_ascii(seed, 90);
|
||||
let kmers = canonical_kmers(&seq);
|
||||
if kmers.len() != seq.len() - k + 1 {
|
||||
continue; // accidental internal repeat: not a single chain
|
||||
}
|
||||
let g = graph_from_ascii(&seq);
|
||||
g.compute_degrees_and_mark_starts();
|
||||
let unitigs = collect_unitigs(&g);
|
||||
assert_eq!(unitigs.len(), 1, "seed {seed}: repeat-free sequence must give one unitig");
|
||||
assert_eq!(unitigs[0].seql(), seq.len(), "seed {seed}: unitig spans the sequence");
|
||||
assert_eq!(kmers_from_unitigs(&unitigs), kmers, "seed {seed}: same k-mers");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unitigs_partition_the_kmers_of_branching_reads() {
|
||||
// Reads of a small genome on both strands with substitutions: branches,
|
||||
// bubbles and tips. Whatever the shape, every k-mer must be covered
|
||||
// exactly once: sum over unitigs of (seql - k + 1) == number of nodes.
|
||||
let k = 11;
|
||||
let _guard = lock_k();
|
||||
set_k(k);
|
||||
let genome = pseudo_random_ascii(7, 600);
|
||||
let mut g = GraphDeBruijn::new();
|
||||
for r in 0..120u64 {
|
||||
let start = (pseudo_random_ascii(100 + r, 1)[0] as usize * 31 + r as usize * 17) % 500;
|
||||
let mut read = genome[start..start + 90].to_vec();
|
||||
if r % 2 == 1 {
|
||||
// second strand
|
||||
read = read
|
||||
.iter()
|
||||
.rev()
|
||||
.map(|&b| match b {
|
||||
b'A' => b'T',
|
||||
b'C' => b'G',
|
||||
b'G' => b'C',
|
||||
_ => b'A',
|
||||
})
|
||||
.collect();
|
||||
}
|
||||
if r % 3 == 0 {
|
||||
let at = (r as usize * 13) % read.len();
|
||||
read[at] = if read[at] == b'A' { b'C' } else { b'A' };
|
||||
}
|
||||
for i in 0..=read.len() - k {
|
||||
g.push(Kmer::from_ascii(&read[i..i + k]).unwrap().canonical());
|
||||
}
|
||||
}
|
||||
g.compute_degrees_and_mark_starts();
|
||||
let unitigs = collect_unitigs(&g);
|
||||
let covered: usize = unitigs.iter().map(|u| u.seql() - k + 1).sum();
|
||||
assert_eq!(covered, g.len(), "every k-mer in exactly one unitig");
|
||||
assert_eq!(kmers_from_unitigs(&unitigs).len(), g.len(), "no k-mer missing");
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
//! Throwaway benchmark of the de Bruijn graph on a real kmer set:
|
||||
//! `graph_bench <unitigs.bin> <k> [repeat]` builds `GraphDeBruijn` from the
|
||||
//! canonical kmers of an existing layer's unitigs (same call sequence as
|
||||
//! `write_graph_as_unitigs`) and times each phase.
|
||||
use std::path::PathBuf;
|
||||
use std::time::Instant;
|
||||
|
||||
use obidebruinj::GraphDeBruijn;
|
||||
use obikseq::Sequence;
|
||||
use obiskio::CanonicalKmerIter;
|
||||
use rayon::prelude::*;
|
||||
use std::collections::HashSet;
|
||||
|
||||
fn main() {
|
||||
let args: Vec<String> = std::env::args().collect();
|
||||
let path = PathBuf::from(&args[1]);
|
||||
let k: usize = args[2].parse().unwrap();
|
||||
let repeat: usize = args.get(3).map_or(1, |s| s.parse().unwrap());
|
||||
obikseq::set_k(k);
|
||||
obikseq::set_m(11);
|
||||
let kmers: Vec<_> = if let Ok(n) = std::env::var("SYNTH") {
|
||||
// synthetic reads: random genome, reads on both strands, 1 % substitutions
|
||||
let n: usize = n.parse().unwrap();
|
||||
let mut st = 0x9E3779B97F4A7C15u64;
|
||||
let mut rnd = move || {
|
||||
st ^= st << 13;
|
||||
st ^= st >> 7;
|
||||
st ^= st << 17;
|
||||
st
|
||||
};
|
||||
let genome: Vec<u8> = (0..n).map(|_| b"ACGT"[(rnd() % 4) as usize]).collect();
|
||||
let mut set = std::collections::BTreeSet::new();
|
||||
for _ in 0..(n / 50) {
|
||||
let start = (rnd() as usize) % (n - 100);
|
||||
let mut read: Vec<u8> = genome[start..start + 100].to_vec();
|
||||
for b in read.iter_mut() {
|
||||
if rnd() % 100 == 0 {
|
||||
*b = b"ACGT"[(rnd() % 4) as usize];
|
||||
}
|
||||
}
|
||||
for w in read.windows(k) {
|
||||
set.insert(obikseq::Kmer::from_ascii(w).unwrap().canonical());
|
||||
}
|
||||
}
|
||||
set.into_iter().collect()
|
||||
} else {
|
||||
CanonicalKmerIter::new(&path).unwrap().collect()
|
||||
};
|
||||
eprintln!("{} kmers", kmers.len());
|
||||
// Independent count of maximal unitigs: nodes − mutually-unique links
|
||||
// (cycles ignored), computed with a plain HashSet — no use of GraphDeBruijn.
|
||||
if std::env::var("IDEAL").is_ok() {
|
||||
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
|
||||
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
|
||||
(0..4u8)
|
||||
.map(|x| a.push_right(x))
|
||||
.filter(|b| set.contains(&b.canonical().raw()))
|
||||
.collect()
|
||||
};
|
||||
let links: usize = kmers
|
||||
.par_iter()
|
||||
.map(|c| {
|
||||
let mut n = 0usize;
|
||||
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
|
||||
let ob = outs(o);
|
||||
if ob.len() == 1 {
|
||||
let b = ob[0];
|
||||
// in-neighbours of b = revcomp of out-neighbours of revcomp(b)
|
||||
if outs(b.revcomp()).len() == 1 {
|
||||
// count each undirected link once: only from the orientation
|
||||
// whose canonical raw is the smaller end (ties: both seen once per orientation)
|
||||
let (ca, cb) = (o.canonical().raw(), b.canonical().raw());
|
||||
if ca < cb || (ca == cb && o.raw() <= b.raw()) {
|
||||
n += 1;
|
||||
} else if ca > cb {
|
||||
// seen from the other end too: skip
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
n
|
||||
})
|
||||
.sum();
|
||||
eprintln!("IDEAL: nodes={} links={} -> maximal unitigs ≈ {}", kmers.len(), links, kmers.len() - links);
|
||||
}
|
||||
if std::env::var("IDEALDUMP").is_ok() {
|
||||
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
|
||||
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
|
||||
(0..4u8).map(|x| a.push_right(x)).filter(|b| set.contains(&b.canonical().raw())).collect()
|
||||
};
|
||||
// oriented kmer a has a mutual predecessor p iff ins(a)=={p} and outs(p)=={a}
|
||||
let pred = |a: obikseq::Kmer| -> Option<obikseq::Kmer> {
|
||||
let ins: Vec<obikseq::Kmer> = outs(a.revcomp()).into_iter().map(|b| b.revcomp()).collect();
|
||||
if ins.len() == 1 && outs(ins[0]).len() == 1 { Some(ins[0]) } else { None }
|
||||
};
|
||||
let mut seen_chains = std::collections::BTreeSet::new();
|
||||
for c in &kmers {
|
||||
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
|
||||
if pred(o).is_some() { continue; }
|
||||
let mut text: Vec<u8> = (0..k).map(|i| b"ACGT"[o.nucleotide(i) as usize]).collect();
|
||||
let mut cur = o;
|
||||
loop {
|
||||
let ob = outs(cur);
|
||||
if ob.len() != 1 { break; }
|
||||
let b = ob[0];
|
||||
if pred(b).is_none() { break; }
|
||||
text.push(b"ACGT"[b.nucleotide(k - 1) as usize]);
|
||||
cur = b;
|
||||
}
|
||||
let s = String::from_utf8(text).unwrap();
|
||||
let rc: String = s.bytes().rev().map(|c| match c { b'A' => 'T', b'C' => 'G', b'G' => 'C', _ => 'A' }).collect();
|
||||
seen_chains.insert(if s <= rc { s } else { rc });
|
||||
}
|
||||
}
|
||||
for u in &seen_chains { println!("IDEALUNITIG {} {}", u.len(), u); }
|
||||
}
|
||||
for r in 0..repeat {
|
||||
let t = Instant::now();
|
||||
let mut g = GraphDeBruijn::new();
|
||||
for &kmer in &kmers {
|
||||
g.push(kmer);
|
||||
}
|
||||
let t_push = t.elapsed();
|
||||
|
||||
let t = Instant::now();
|
||||
g.compute_degrees_and_mark_starts();
|
||||
let t_deg = t.elapsed();
|
||||
|
||||
let t = Instant::now();
|
||||
let n_unitigs = std::sync::atomic::AtomicUsize::new(0);
|
||||
let n_nuc = std::sync::atomic::AtomicUsize::new(0);
|
||||
let dump = std::env::var("DUMP").is_ok();
|
||||
let seqs = std::sync::Mutex::new(Vec::<String>::new());
|
||||
g.for_each_unitig(|it| {
|
||||
let mut n = 0usize;
|
||||
let mut text = String::new();
|
||||
for nuc in it {
|
||||
n += 1;
|
||||
if dump {
|
||||
text.push(b"ACGT"[nuc as usize] as char);
|
||||
}
|
||||
}
|
||||
if dump {
|
||||
seqs.lock().unwrap().push(text);
|
||||
}
|
||||
n_unitigs.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
n_nuc.fetch_add(n, std::sync::atomic::Ordering::Relaxed);
|
||||
});
|
||||
let t_walk = t.elapsed();
|
||||
if dump {
|
||||
let mut v = seqs.lock().unwrap().clone();
|
||||
v.sort();
|
||||
for u in v { println!("UNITIG {} {}", u.len(), u); }
|
||||
}
|
||||
println!(
|
||||
"run {r}: nodes={} push={:.2?} degrees={:.2?} unitigs={:.2?} -> {} unitigs, {} nucleotides",
|
||||
g.len(),
|
||||
t_push,
|
||||
t_deg,
|
||||
t_walk,
|
||||
n_unitigs.load(std::sync::atomic::Ordering::Relaxed),
|
||||
n_nuc.load(std::sync::atomic::Ordering::Relaxed)
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user