Push wpvskkyozppn #78
No files matched your search
@@ -267,3 +267,35 @@ One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slo
|
||||
- **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.
|
||||
|
||||
Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture.
|
||||
|
||||
## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)
|
||||
|
||||
Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench <unitigs.bin> <k> [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`.
|
||||
|
||||
| | push | degrees | unitigs | unitigs produced | nucleotides |
|
||||
|---|---|---|---|---|---|
|
||||
| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M |
|
||||
| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M |
|
||||
| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** |
|
||||
|
||||
Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).
|
||||
|
||||
**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision).
|
||||
|
||||
**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).
|
||||
|
||||
**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig).
|
||||
|
||||
Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).
|
||||
|
||||
**End-to-end** (current binary vs previous one, same inputs):
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) |
|
||||
| … `index` stage | 114.6 s | **59.2 s** (−48 %) |
|
||||
| … user CPU | 1 870 s | 1 304 s |
|
||||
| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB |
|
||||
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
|
||||
|
||||
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
|
||||
@@ -1,19 +1,19 @@
|
||||
use hashbrown::HashMap;
|
||||
use hashbrown::{HashMap, HashSet};
|
||||
use obikseq::k;
|
||||
use obikseq::{CanonicalKmer, Unitig};
|
||||
#[cfg(not(any(test, feature = "test-utils")))]
|
||||
use rayon::iter::{IntoParallelRefIterator, ParallelIterator};
|
||||
use std::cell::RefCell;
|
||||
use std::sync::atomic::{AtomicU8, Ordering};
|
||||
use xxhash_rust::xxh3::Xxh3Builder;
|
||||
|
||||
use super::kmer_hash::KmerHashBuilder;
|
||||
use super::node::{IS_VISITED_MASK, Node};
|
||||
use super::unitig_iter::UnitigNucIter;
|
||||
use super::walk::WalkState;
|
||||
|
||||
// ── Types ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
pub(super) type FastHashMap<K, V> = HashMap<K, V, Xxh3Builder>;
|
||||
pub(super) type FastHashMap<K, V> = HashMap<K, V, KmerHashBuilder>;
|
||||
|
||||
// ── GraphDeBruijn ─────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -24,7 +24,7 @@ pub struct GraphDeBruijn {
|
||||
impl GraphDeBruijn {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
nodes: FastHashMap::with_hasher(Xxh3Builder::new()),
|
||||
nodes: FastHashMap::with_hasher(KmerHashBuilder),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,7 +40,7 @@ impl GraphDeBruijn {
|
||||
|
||||
pub fn with_capacity(capacity: usize) -> Self {
|
||||
Self {
|
||||
nodes: FastHashMap::with_capacity_and_hasher(capacity, Xxh3Builder::new()),
|
||||
nodes: FastHashMap::with_capacity_and_hasher(capacity, KmerHashBuilder),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,7 +79,7 @@ impl GraphDeBruijn {
|
||||
if node.is_visited() {
|
||||
return;
|
||||
}
|
||||
if self.is_start(*kmer, node) {
|
||||
if self.is_start(*kmer, node, atomic, true) || self.is_start(*kmer, node, atomic, false) {
|
||||
node.set_start();
|
||||
atomic.store(node.0, Ordering::Relaxed);
|
||||
}
|
||||
@@ -98,45 +98,40 @@ impl GraphDeBruijn {
|
||||
}
|
||||
}
|
||||
|
||||
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState {
|
||||
let node = Node(
|
||||
self.nodes
|
||||
.get(&kmer)
|
||||
.expect("Cannot start a walk from not indexed kmer")
|
||||
.load(Ordering::Relaxed),
|
||||
);
|
||||
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState<'_> {
|
||||
let atomic = self
|
||||
.nodes
|
||||
.get(&kmer)
|
||||
.expect("Cannot start a walk from not indexed kmer");
|
||||
let node = Node(atomic.load(Ordering::Relaxed));
|
||||
debug_assert!(
|
||||
!node.is_visited(),
|
||||
"Cannot start a walk from a visited node"
|
||||
);
|
||||
WalkState::new(kmer, node, true)
|
||||
WalkState::new(kmer, node, true, atomic)
|
||||
}
|
||||
|
||||
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState> {
|
||||
let node = Node(self.nodes.get(&kmer)?.load(Ordering::Relaxed));
|
||||
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState<'_>> {
|
||||
let atomic = self.nodes.get(&kmer)?;
|
||||
let node = Node(atomic.load(Ordering::Relaxed));
|
||||
if node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
Some(WalkState::new(kmer, node, true))
|
||||
Some(WalkState::new(kmer, node, true, atomic))
|
||||
}
|
||||
|
||||
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize) -> Option<UnitigNucIter<'_>> {
|
||||
let old = self
|
||||
.nodes
|
||||
.get(&kmer)?
|
||||
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize, direct: bool) -> Option<UnitigNucIter<'_>> {
|
||||
let atomic = self.nodes.get(&kmer)?;
|
||||
let old = atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
if old & IS_VISITED_MASK != 0 {
|
||||
return None;
|
||||
}
|
||||
let start = WalkState::new(kmer, Node(old), true);
|
||||
let start = WalkState::new(kmer, Node(old), direct, atomic);
|
||||
let next_step = start.walk(self).and_then(|(next_state, nuc)| {
|
||||
let ext_old = self
|
||||
.nodes
|
||||
.get(&next_state.kmer)?
|
||||
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
let ext_old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
(ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc))
|
||||
});
|
||||
Some(UnitigNucIter::new(self, kmer, k, next_step))
|
||||
Some(UnitigNucIter::new(self, kmer, k, direct, next_step))
|
||||
}
|
||||
|
||||
pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) {
|
||||
@@ -157,7 +152,8 @@ impl GraphDeBruijn {
|
||||
.then_some(kmer)
|
||||
})
|
||||
.for_each(|kmer| {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
|
||||
let Some(direct) = self.start_direction(kmer) else { return };
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
|
||||
n_new.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -166,7 +162,8 @@ impl GraphDeBruijn {
|
||||
#[cfg(any(test, feature = "test-utils"))]
|
||||
self.nodes.iter().for_each(|(&kmer, atomic)| {
|
||||
if Node(atomic.load(Ordering::Acquire)).is_start() {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
|
||||
let Some(direct) = self.start_direction(kmer) else { return };
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
|
||||
n_new.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -187,8 +184,8 @@ impl GraphDeBruijn {
|
||||
if node.is_visited() {
|
||||
continue;
|
||||
}
|
||||
let chain_start = self.find_chain_start(kmer);
|
||||
if let Some(iter) = self.unitig_nucleotides(chain_start, k) {
|
||||
let (chain_start, direct) = self.find_chain_start(kmer);
|
||||
if let Some(iter) = self.unitig_nucleotides(chain_start, k, direct) {
|
||||
n2.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -196,7 +193,7 @@ impl GraphDeBruijn {
|
||||
// Safe because unitig_nucleotides may have visited kmer in the
|
||||
// meantime — in that case it returns None and we skip.
|
||||
if chain_start != kmer {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
|
||||
if let Some(iter) = self.unitig_nucleotides(kmer, k, true) {
|
||||
n2.fetch_add(1, Ordering::Relaxed);
|
||||
f(iter);
|
||||
}
|
||||
@@ -216,17 +213,34 @@ impl GraphDeBruijn {
|
||||
self.nodes.extend(other.nodes);
|
||||
}
|
||||
|
||||
fn find_chain_start(&self, kmer: CanonicalKmer) -> CanonicalKmer {
|
||||
let node = Node(self.nodes[&kmer].load(Ordering::Acquire));
|
||||
let mut state = WalkState::new(kmer, node, false);
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
/// Walks backwards from `kmer` to the first node of its chain. Returns
|
||||
/// that node together with the orientation to read it in to walk the chain
|
||||
/// forwards: the backward walk traverses nodes in the orientation
|
||||
/// `state.direct`, so the forward one is its opposite. (Dropping the
|
||||
/// orientation, as this function once did, started the forward walk at the
|
||||
/// wrong end and emitted a lone-kmer unitig.) On a cycle, `kmer` itself
|
||||
/// in the canonical orientation.
|
||||
fn find_chain_start(&self, kmer: CanonicalKmer) -> (CanonicalKmer, bool) {
|
||||
let atomic = &self.nodes[&kmer];
|
||||
let node = Node(atomic.load(Ordering::Acquire));
|
||||
let mut state = WalkState::new(kmer, node, false, atomic);
|
||||
// Most nodes reaching this pass are branch nodes or tips that cannot
|
||||
// step at all: answer before allocating the cycle-detection set.
|
||||
let Some((first, _)) = state.walk(self) else {
|
||||
return (state.kmer, !state.direct);
|
||||
};
|
||||
let mut seen: HashSet<(u64, bool), KmerHashBuilder> = HashSet::with_hasher(KmerHashBuilder);
|
||||
seen.insert((state.kmer.raw(), state.direct));
|
||||
if !seen.insert((first.kmer.raw(), first.direct)) {
|
||||
return (kmer, true);
|
||||
}
|
||||
state = first;
|
||||
loop {
|
||||
match state.walk(self) {
|
||||
None => return state.kmer,
|
||||
None => return (state.kmer, !state.direct),
|
||||
Some((next, _)) => {
|
||||
if !seen.insert((next.kmer.raw(), next.direct)) {
|
||||
return kmer;
|
||||
return (kmer, true);
|
||||
}
|
||||
state = next;
|
||||
}
|
||||
@@ -234,8 +248,23 @@ impl GraphDeBruijn {
|
||||
}
|
||||
}
|
||||
|
||||
fn is_start(&self, query: CanonicalKmer, node: Node) -> bool {
|
||||
!WalkState::new(query, node, true).reachable(self)
|
||||
/// Orientation in which a node flagged as a start must be read: the
|
||||
/// canonical one if a unitig starts there that way, else the reverse
|
||||
/// complement. `None` if it was visited in the meantime.
|
||||
fn start_direction(&self, kmer: CanonicalKmer) -> Option<bool> {
|
||||
let atomic = self.nodes.get(&kmer)?;
|
||||
let node = Node(atomic.load(Ordering::Acquire));
|
||||
if node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
Some(self.is_start(kmer, node, atomic, true) || !self.is_start(kmer, node, atomic, false))
|
||||
}
|
||||
|
||||
/// `true` when `query`, traversed in orientation `direct` (canonical or
|
||||
/// reverse-complemented), has no unique predecessor — i.e. a unitig
|
||||
/// starts there when read in that orientation.
|
||||
fn is_start(&self, query: CanonicalKmer, node: Node, atomic: &AtomicU8, direct: bool) -> bool {
|
||||
!WalkState::new(query, node, direct, atomic).reachable(self)
|
||||
}
|
||||
|
||||
pub fn try_for_each_unitig<E, F>(&self, f: F) -> Result<(), E>
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
//! Hasher for the graph's `HashMap<CanonicalKmer, _>`.
|
||||
//!
|
||||
//! The key is one `u64` (the derived `Hash` of a kmer is a single
|
||||
//! `write_u64`), so the generic streaming hashers used before (xxh3's
|
||||
//! `Hasher` implementation: buffer, update, finish) cost far more than the
|
||||
//! hash itself — ≈ 35 % of the active CPU of `index` in the profile of
|
||||
//! `DevDocMD/implementation/mphf_batch_queries.md`, §17. A 64-bit finaliser
|
||||
//! (splitmix64's) is a bijection with full avalanche, which is what
|
||||
//! hashbrown needs from it (low bits for the bucket, top 7 for the tag).
|
||||
|
||||
use std::hash::{BuildHasher, Hasher};
|
||||
|
||||
#[derive(Clone, Copy, Default)]
|
||||
pub(super) struct KmerHashBuilder;
|
||||
|
||||
impl BuildHasher for KmerHashBuilder {
|
||||
type Hasher = KmerHasher;
|
||||
|
||||
#[inline]
|
||||
fn build_hasher(&self) -> KmerHasher {
|
||||
KmerHasher(0)
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) struct KmerHasher(u64);
|
||||
|
||||
#[inline]
|
||||
fn mix64(x: u64) -> u64 {
|
||||
let x = x ^ (x >> 30);
|
||||
let x = x.wrapping_mul(0xbf58476d1ce4e5b9);
|
||||
let x = x ^ (x >> 27);
|
||||
let x = x.wrapping_mul(0x94d049bb133111eb);
|
||||
x ^ (x >> 31)
|
||||
}
|
||||
|
||||
impl Hasher for KmerHasher {
|
||||
#[inline]
|
||||
fn write_u64(&mut self, x: u64) {
|
||||
self.0 = mix64(self.0 ^ x);
|
||||
}
|
||||
|
||||
/// Generic fallback (never taken for a kmer key): folds the bytes 8 at a time.
|
||||
fn write(&mut self, bytes: &[u8]) {
|
||||
for chunk in bytes.chunks(8) {
|
||||
let mut buf = [0u8; 8];
|
||||
buf[..chunk.len()].copy_from_slice(chunk);
|
||||
self.write_u64(u64::from_le_bytes(buf));
|
||||
}
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn finish(&self) -> u64 {
|
||||
self.0
|
||||
}
|
||||
}
|
||||
@@ -1,10 +1,11 @@
|
||||
//! De Bruijn graph over canonical k-mers, built for unitig extraction.
|
||||
//!
|
||||
//! Submodules: [`node`] (packed per-kmer neighbour/visited/start flags),
|
||||
//! Submodules: [`kmer_hash`] (hasher of the node table), [`node`] (packed per-kmer neighbour/visited/start flags),
|
||||
//! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself),
|
||||
//! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator).
|
||||
|
||||
mod graph;
|
||||
mod kmer_hash;
|
||||
mod node;
|
||||
mod unitig_iter;
|
||||
mod walk;
|
||||
|
||||
@@ -10,9 +10,11 @@ use super::walk::WalkState;
|
||||
pub struct UnitigNucIter<'a> {
|
||||
graph: &'a GraphDeBruijn,
|
||||
start: CanonicalKmer,
|
||||
/// Orientation the unitig is read in (`false`: reverse complement of `start`).
|
||||
direct: bool,
|
||||
pos: usize,
|
||||
k: usize,
|
||||
next_step: Option<(WalkState, u8)>,
|
||||
next_step: Option<(WalkState<'a>, u8)>,
|
||||
}
|
||||
|
||||
impl<'a> UnitigNucIter<'a> {
|
||||
@@ -20,11 +22,13 @@ impl<'a> UnitigNucIter<'a> {
|
||||
graph: &'a GraphDeBruijn,
|
||||
start: CanonicalKmer,
|
||||
k: usize,
|
||||
next_step: Option<(WalkState, u8)>,
|
||||
direct: bool,
|
||||
next_step: Option<(WalkState<'a>, u8)>,
|
||||
) -> Self {
|
||||
Self {
|
||||
graph,
|
||||
start,
|
||||
direct,
|
||||
pos: 0,
|
||||
k,
|
||||
next_step,
|
||||
@@ -37,16 +41,16 @@ impl Iterator for UnitigNucIter<'_> {
|
||||
|
||||
fn next(&mut self) -> Option<u8> {
|
||||
if self.pos < self.k {
|
||||
let nuc = self.start.nucleotide(self.pos);
|
||||
let nuc = if self.direct {
|
||||
self.start.nucleotide(self.pos)
|
||||
} else {
|
||||
3 - self.start.nucleotide(self.k - 1 - self.pos)
|
||||
};
|
||||
self.pos += 1;
|
||||
Some(nuc)
|
||||
} else if let Some((state, nuc)) = self.next_step.take() {
|
||||
self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| {
|
||||
let old = self
|
||||
.graph
|
||||
.nodes
|
||||
.get(&next_state.kmer)?
|
||||
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
let old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
|
||||
(old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc))
|
||||
});
|
||||
Some(nuc)
|
||||
|
||||
@@ -1,35 +1,41 @@
|
||||
use obikseq::{CanonicalKmer, Sequence};
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::sync::atomic::{AtomicU8, Ordering};
|
||||
|
||||
use super::graph::GraphDeBruijn;
|
||||
use super::node::Node;
|
||||
|
||||
pub struct WalkState {
|
||||
/// A position in a walk: the node's kmer, its flags as read when the step was
|
||||
/// taken, and the node's own table entry. Keeping `atomic` means the caller
|
||||
/// that wants to mark the node visited does not look the kmer up a second
|
||||
/// time (a random access in a table far larger than the cache).
|
||||
pub struct WalkState<'g> {
|
||||
pub(super) kmer: CanonicalKmer,
|
||||
pub(super) node: Node,
|
||||
pub(super) direct: bool,
|
||||
pub(super) atomic: &'g AtomicU8,
|
||||
}
|
||||
|
||||
impl WalkState {
|
||||
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool) -> Self {
|
||||
impl<'g> WalkState<'g> {
|
||||
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool, atomic: &'g AtomicU8) -> Self {
|
||||
debug_assert!(!node.is_visited(), "Cannot walk over a visited node");
|
||||
Self { kmer, node, direct }
|
||||
Self { kmer, node, direct, atomic }
|
||||
}
|
||||
|
||||
pub fn leavable(&self, graph: &GraphDeBruijn) -> bool {
|
||||
pub fn leavable(&self, graph: &'g GraphDeBruijn) -> bool {
|
||||
self.walk(graph).is_some()
|
||||
}
|
||||
|
||||
pub fn reachable(&self, graph: &GraphDeBruijn) -> bool {
|
||||
pub fn reachable(&self, graph: &'g GraphDeBruijn) -> bool {
|
||||
WalkState {
|
||||
kmer: self.kmer,
|
||||
node: self.node,
|
||||
direct: !self.direct,
|
||||
atomic: self.atomic,
|
||||
}
|
||||
.leavable(graph)
|
||||
}
|
||||
|
||||
pub fn walk(&self, graph: &GraphDeBruijn) -> Option<(WalkState, u8)> {
|
||||
pub fn walk(&self, graph: &'g GraphDeBruijn) -> Option<(WalkState<'g>, u8)> {
|
||||
if self.direct {
|
||||
if !self.node.can_extend_right() {
|
||||
return None;
|
||||
@@ -38,7 +44,8 @@ impl WalkState {
|
||||
let next = self.kmer.into_kmer().push_right(nuc);
|
||||
let cnext = next.canonical();
|
||||
let dnext = next.raw() == cnext.raw();
|
||||
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed));
|
||||
let next_atomic = graph.nodes.get(&cnext).unwrap();
|
||||
let next_node = Node(next_atomic.load(Ordering::Relaxed));
|
||||
if next_node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
@@ -52,6 +59,7 @@ impl WalkState {
|
||||
kmer: cnext,
|
||||
node: next_node,
|
||||
direct: dnext,
|
||||
atomic: next_atomic,
|
||||
},
|
||||
nuc,
|
||||
))
|
||||
@@ -63,20 +71,22 @@ impl WalkState {
|
||||
let next = self.kmer.into_kmer().push_left(nuc);
|
||||
let cnext = next.canonical();
|
||||
let dnext = next.raw() != cnext.raw();
|
||||
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed));
|
||||
let next_atomic = graph.nodes.get(&cnext).unwrap();
|
||||
let next_node = Node(next_atomic.load(Ordering::Relaxed));
|
||||
if next_node.is_visited() {
|
||||
return None;
|
||||
}
|
||||
let reachable = if dnext {
|
||||
next_node.can_extend_right()
|
||||
} else {
|
||||
next_node.can_extend_left()
|
||||
} else {
|
||||
next_node.can_extend_right()
|
||||
};
|
||||
reachable.then_some((
|
||||
WalkState {
|
||||
kmer: cnext,
|
||||
node: next_node,
|
||||
direct: dnext,
|
||||
atomic: next_atomic,
|
||||
},
|
||||
3 - nuc,
|
||||
))
|
||||
|
||||
@@ -317,3 +317,86 @@ fn branching_graph_no_kmer_lost_or_duplicated() {
|
||||
);
|
||||
assert_eq!(got, expected, "k-mer sets differ");
|
||||
}
|
||||
|
||||
// ── orientation independence ──────────────────────────────────────────────
|
||||
|
||||
// Deterministic pseudo-random ASCII sequence (xorshift), so the tests need no
|
||||
// RNG dependency.
|
||||
fn pseudo_random_ascii(seed: u64, len: usize) -> Vec<u8> {
|
||||
let mut st = seed.wrapping_mul(0x9E3779B97F4A7C15) | 1;
|
||||
(0..len)
|
||||
.map(|_| {
|
||||
st ^= st << 13;
|
||||
st ^= st >> 7;
|
||||
st ^= st << 17;
|
||||
b"ACGT"[(st % 4) as usize]
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn linear_chain_is_one_unitig_whatever_the_orientations() {
|
||||
// A repeat-free sequence is one maximal unitig. Each k-mer is stored in
|
||||
// its canonical orientation, so across many sequences the chain's nodes
|
||||
// take every mix of orientations — including chains whose two ends are
|
||||
// both stored reverse-complemented, which once came out as several
|
||||
// unitigs (a lone-kmer unitig per mis-oriented chain start) depending
|
||||
// only on the table's iteration order.
|
||||
let k = 21; // long enough that (k-1)-mer collisions (= branches) do not occur
|
||||
let _guard = lock_k();
|
||||
set_k(k);
|
||||
for seed in 1..=300u64 {
|
||||
let seq = pseudo_random_ascii(seed, 90);
|
||||
let kmers = canonical_kmers(&seq);
|
||||
if kmers.len() != seq.len() - k + 1 {
|
||||
continue; // accidental internal repeat: not a single chain
|
||||
}
|
||||
let g = graph_from_ascii(&seq);
|
||||
g.compute_degrees_and_mark_starts();
|
||||
let unitigs = collect_unitigs(&g);
|
||||
assert_eq!(unitigs.len(), 1, "seed {seed}: repeat-free sequence must give one unitig");
|
||||
assert_eq!(unitigs[0].seql(), seq.len(), "seed {seed}: unitig spans the sequence");
|
||||
assert_eq!(kmers_from_unitigs(&unitigs), kmers, "seed {seed}: same k-mers");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unitigs_partition_the_kmers_of_branching_reads() {
|
||||
// Reads of a small genome on both strands with substitutions: branches,
|
||||
// bubbles and tips. Whatever the shape, every k-mer must be covered
|
||||
// exactly once: sum over unitigs of (seql - k + 1) == number of nodes.
|
||||
let k = 11;
|
||||
let _guard = lock_k();
|
||||
set_k(k);
|
||||
let genome = pseudo_random_ascii(7, 600);
|
||||
let mut g = GraphDeBruijn::new();
|
||||
for r in 0..120u64 {
|
||||
let start = (pseudo_random_ascii(100 + r, 1)[0] as usize * 31 + r as usize * 17) % 500;
|
||||
let mut read = genome[start..start + 90].to_vec();
|
||||
if r % 2 == 1 {
|
||||
// second strand
|
||||
read = read
|
||||
.iter()
|
||||
.rev()
|
||||
.map(|&b| match b {
|
||||
b'A' => b'T',
|
||||
b'C' => b'G',
|
||||
b'G' => b'C',
|
||||
_ => b'A',
|
||||
})
|
||||
.collect();
|
||||
}
|
||||
if r % 3 == 0 {
|
||||
let at = (r as usize * 13) % read.len();
|
||||
read[at] = if read[at] == b'A' { b'C' } else { b'A' };
|
||||
}
|
||||
for i in 0..=read.len() - k {
|
||||
g.push(Kmer::from_ascii(&read[i..i + k]).unwrap().canonical());
|
||||
}
|
||||
}
|
||||
g.compute_degrees_and_mark_starts();
|
||||
let unitigs = collect_unitigs(&g);
|
||||
let covered: usize = unitigs.iter().map(|u| u.seql() - k + 1).sum();
|
||||
assert_eq!(covered, g.len(), "every k-mer in exactly one unitig");
|
||||
assert_eq!(kmers_from_unitigs(&unitigs).len(), g.len(), "no k-mer missing");
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
//! Throwaway benchmark of the de Bruijn graph on a real kmer set:
|
||||
//! `graph_bench <unitigs.bin> <k> [repeat]` builds `GraphDeBruijn` from the
|
||||
//! canonical kmers of an existing layer's unitigs (same call sequence as
|
||||
//! `write_graph_as_unitigs`) and times each phase.
|
||||
use std::path::PathBuf;
|
||||
use std::time::Instant;
|
||||
|
||||
use obidebruinj::GraphDeBruijn;
|
||||
use obikseq::Sequence;
|
||||
use obiskio::CanonicalKmerIter;
|
||||
use rayon::prelude::*;
|
||||
use std::collections::HashSet;
|
||||
|
||||
fn main() {
|
||||
let args: Vec<String> = std::env::args().collect();
|
||||
let path = PathBuf::from(&args[1]);
|
||||
let k: usize = args[2].parse().unwrap();
|
||||
let repeat: usize = args.get(3).map_or(1, |s| s.parse().unwrap());
|
||||
obikseq::set_k(k);
|
||||
obikseq::set_m(11);
|
||||
let kmers: Vec<_> = if let Ok(n) = std::env::var("SYNTH") {
|
||||
// synthetic reads: random genome, reads on both strands, 1 % substitutions
|
||||
let n: usize = n.parse().unwrap();
|
||||
let mut st = 0x9E3779B97F4A7C15u64;
|
||||
let mut rnd = move || {
|
||||
st ^= st << 13;
|
||||
st ^= st >> 7;
|
||||
st ^= st << 17;
|
||||
st
|
||||
};
|
||||
let genome: Vec<u8> = (0..n).map(|_| b"ACGT"[(rnd() % 4) as usize]).collect();
|
||||
let mut set = std::collections::BTreeSet::new();
|
||||
for _ in 0..(n / 50) {
|
||||
let start = (rnd() as usize) % (n - 100);
|
||||
let mut read: Vec<u8> = genome[start..start + 100].to_vec();
|
||||
for b in read.iter_mut() {
|
||||
if rnd() % 100 == 0 {
|
||||
*b = b"ACGT"[(rnd() % 4) as usize];
|
||||
}
|
||||
}
|
||||
for w in read.windows(k) {
|
||||
set.insert(obikseq::Kmer::from_ascii(w).unwrap().canonical());
|
||||
}
|
||||
}
|
||||
set.into_iter().collect()
|
||||
} else {
|
||||
CanonicalKmerIter::new(&path).unwrap().collect()
|
||||
};
|
||||
eprintln!("{} kmers", kmers.len());
|
||||
// Independent count of maximal unitigs: nodes − mutually-unique links
|
||||
// (cycles ignored), computed with a plain HashSet — no use of GraphDeBruijn.
|
||||
if std::env::var("IDEAL").is_ok() {
|
||||
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
|
||||
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
|
||||
(0..4u8)
|
||||
.map(|x| a.push_right(x))
|
||||
.filter(|b| set.contains(&b.canonical().raw()))
|
||||
.collect()
|
||||
};
|
||||
let links: usize = kmers
|
||||
.par_iter()
|
||||
.map(|c| {
|
||||
let mut n = 0usize;
|
||||
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
|
||||
let ob = outs(o);
|
||||
if ob.len() == 1 {
|
||||
let b = ob[0];
|
||||
// in-neighbours of b = revcomp of out-neighbours of revcomp(b)
|
||||
if outs(b.revcomp()).len() == 1 {
|
||||
// count each undirected link once: only from the orientation
|
||||
// whose canonical raw is the smaller end (ties: both seen once per orientation)
|
||||
let (ca, cb) = (o.canonical().raw(), b.canonical().raw());
|
||||
if ca < cb || (ca == cb && o.raw() <= b.raw()) {
|
||||
n += 1;
|
||||
} else if ca > cb {
|
||||
// seen from the other end too: skip
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
n
|
||||
})
|
||||
.sum();
|
||||
eprintln!("IDEAL: nodes={} links={} -> maximal unitigs ≈ {}", kmers.len(), links, kmers.len() - links);
|
||||
}
|
||||
if std::env::var("IDEALDUMP").is_ok() {
|
||||
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
|
||||
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
|
||||
(0..4u8).map(|x| a.push_right(x)).filter(|b| set.contains(&b.canonical().raw())).collect()
|
||||
};
|
||||
// oriented kmer a has a mutual predecessor p iff ins(a)=={p} and outs(p)=={a}
|
||||
let pred = |a: obikseq::Kmer| -> Option<obikseq::Kmer> {
|
||||
let ins: Vec<obikseq::Kmer> = outs(a.revcomp()).into_iter().map(|b| b.revcomp()).collect();
|
||||
if ins.len() == 1 && outs(ins[0]).len() == 1 { Some(ins[0]) } else { None }
|
||||
};
|
||||
let mut seen_chains = std::collections::BTreeSet::new();
|
||||
for c in &kmers {
|
||||
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
|
||||
if pred(o).is_some() { continue; }
|
||||
let mut text: Vec<u8> = (0..k).map(|i| b"ACGT"[o.nucleotide(i) as usize]).collect();
|
||||
let mut cur = o;
|
||||
loop {
|
||||
let ob = outs(cur);
|
||||
if ob.len() != 1 { break; }
|
||||
let b = ob[0];
|
||||
if pred(b).is_none() { break; }
|
||||
text.push(b"ACGT"[b.nucleotide(k - 1) as usize]);
|
||||
cur = b;
|
||||
}
|
||||
let s = String::from_utf8(text).unwrap();
|
||||
let rc: String = s.bytes().rev().map(|c| match c { b'A' => 'T', b'C' => 'G', b'G' => 'C', _ => 'A' }).collect();
|
||||
seen_chains.insert(if s <= rc { s } else { rc });
|
||||
}
|
||||
}
|
||||
for u in &seen_chains { println!("IDEALUNITIG {} {}", u.len(), u); }
|
||||
}
|
||||
for r in 0..repeat {
|
||||
let t = Instant::now();
|
||||
let mut g = GraphDeBruijn::new();
|
||||
for &kmer in &kmers {
|
||||
g.push(kmer);
|
||||
}
|
||||
let t_push = t.elapsed();
|
||||
|
||||
let t = Instant::now();
|
||||
g.compute_degrees_and_mark_starts();
|
||||
let t_deg = t.elapsed();
|
||||
|
||||
let t = Instant::now();
|
||||
let n_unitigs = std::sync::atomic::AtomicUsize::new(0);
|
||||
let n_nuc = std::sync::atomic::AtomicUsize::new(0);
|
||||
let dump = std::env::var("DUMP").is_ok();
|
||||
let seqs = std::sync::Mutex::new(Vec::<String>::new());
|
||||
g.for_each_unitig(|it| {
|
||||
let mut n = 0usize;
|
||||
let mut text = String::new();
|
||||
for nuc in it {
|
||||
n += 1;
|
||||
if dump {
|
||||
text.push(b"ACGT"[nuc as usize] as char);
|
||||
}
|
||||
}
|
||||
if dump {
|
||||
seqs.lock().unwrap().push(text);
|
||||
}
|
||||
n_unitigs.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
n_nuc.fetch_add(n, std::sync::atomic::Ordering::Relaxed);
|
||||
});
|
||||
let t_walk = t.elapsed();
|
||||
if dump {
|
||||
let mut v = seqs.lock().unwrap().clone();
|
||||
v.sort();
|
||||
for u in v { println!("UNITIG {} {}", u.len(), u); }
|
||||
}
|
||||
println!(
|
||||
"run {r}: nodes={} push={:.2?} degrees={:.2?} unitigs={:.2?} -> {} unitigs, {} nucleotides",
|
||||
g.len(),
|
||||
t_push,
|
||||
t_deg,
|
||||
t_walk,
|
||||
n_unitigs.load(std::sync::atomic::Ordering::Relaxed),
|
||||
n_nuc.load(std::sync::atomic::Ordering::Relaxed)
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user