Push wpvskkyozppn #78

Merged
coissac merged 15 commits from push-wpvskkyozppn into main 2026-10-15 10:38:59 +00:00
8 changed files with 441 additions and 62 deletions
Showing only changes of commit 8468319a63 - Show all commits

No files matched your search

@@ -267,3 +267,35 @@ One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slo
- **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.
Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture.
## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)
Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench <unitigs.bin> <k> [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`.
| | push | degrees | unitigs | unitigs produced | nucleotides |
|---|---|---|---|---|---|
| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M |
| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M |
| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** |
Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).
**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision).
**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).
**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig).
Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).
**End-to-end** (current binary vs previous one, same inputs):
| | before | after |
|---|---|---|
| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) |
| … `index` stage | 114.6 s | **59.2 s** (−48 %) |
| … user CPU | 1 870 s | 1 304 s |
| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB |
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
+70 -41
View File
@@ -1,19 +1,19 @@
use hashbrown::HashMap;
use hashbrown::{HashMap, HashSet};
use obikseq::k;
use obikseq::{CanonicalKmer, Unitig};
#[cfg(not(any(test, feature = "test-utils")))]
use rayon::iter::{IntoParallelRefIterator, ParallelIterator};
use std::cell::RefCell;
use std::sync::atomic::{AtomicU8, Ordering};
use xxhash_rust::xxh3::Xxh3Builder;
use super::kmer_hash::KmerHashBuilder;
use super::node::{IS_VISITED_MASK, Node};
use super::unitig_iter::UnitigNucIter;
use super::walk::WalkState;
// ── Types ─────────────────────────────────────────────────────────────────────
pub(super) type FastHashMap<K, V> = HashMap<K, V, Xxh3Builder>;
pub(super) type FastHashMap<K, V> = HashMap<K, V, KmerHashBuilder>;
// ── GraphDeBruijn ─────────────────────────────────────────────────────────────
@@ -24,7 +24,7 @@ pub struct GraphDeBruijn {
impl GraphDeBruijn {
pub fn new() -> Self {
Self {
nodes: FastHashMap::with_hasher(Xxh3Builder::new()),
nodes: FastHashMap::with_hasher(KmerHashBuilder),
}
}
@@ -40,7 +40,7 @@ impl GraphDeBruijn {
pub fn with_capacity(capacity: usize) -> Self {
Self {
nodes: FastHashMap::with_capacity_and_hasher(capacity, Xxh3Builder::new()),
nodes: FastHashMap::with_capacity_and_hasher(capacity, KmerHashBuilder),
}
}
@@ -79,7 +79,7 @@ impl GraphDeBruijn {
if node.is_visited() {
return;
}
if self.is_start(*kmer, node) {
if self.is_start(*kmer, node, atomic, true) || self.is_start(*kmer, node, atomic, false) {
node.set_start();
atomic.store(node.0, Ordering::Relaxed);
}
@@ -98,45 +98,40 @@ impl GraphDeBruijn {
}
}
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState {
let node = Node(
self.nodes
.get(&kmer)
.expect("Cannot start a walk from not indexed kmer")
.load(Ordering::Relaxed),
);
pub fn start_walk(&self, kmer: CanonicalKmer) -> WalkState<'_> {
let atomic = self
.nodes
.get(&kmer)
.expect("Cannot start a walk from not indexed kmer");
let node = Node(atomic.load(Ordering::Relaxed));
debug_assert!(
!node.is_visited(),
"Cannot start a walk from a visited node"
);
WalkState::new(kmer, node, true)
WalkState::new(kmer, node, true, atomic)
}
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState> {
let node = Node(self.nodes.get(&kmer)?.load(Ordering::Relaxed));
pub fn try_start_walk(&self, kmer: CanonicalKmer) -> Option<WalkState<'_>> {
let atomic = self.nodes.get(&kmer)?;
let node = Node(atomic.load(Ordering::Relaxed));
if node.is_visited() {
return None;
}
Some(WalkState::new(kmer, node, true))
Some(WalkState::new(kmer, node, true, atomic))
}
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize) -> Option<UnitigNucIter<'_>> {
let old = self
.nodes
.get(&kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
pub(super) fn unitig_nucleotides(&self, kmer: CanonicalKmer, k: usize, direct: bool) -> Option<UnitigNucIter<'_>> {
let atomic = self.nodes.get(&kmer)?;
let old = atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
if old & IS_VISITED_MASK != 0 {
return None;
}
let start = WalkState::new(kmer, Node(old), true);
let start = WalkState::new(kmer, Node(old), direct, atomic);
let next_step = start.walk(self).and_then(|(next_state, nuc)| {
let ext_old = self
.nodes
.get(&next_state.kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
let ext_old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
(ext_old & IS_VISITED_MASK == 0).then_some((next_state, nuc))
});
Some(UnitigNucIter::new(self, kmer, k, next_step))
Some(UnitigNucIter::new(self, kmer, k, direct, next_step))
}
pub fn for_each_unitig(&self, f: impl Fn(UnitigNucIter<'_>) + Sync) {
@@ -157,7 +152,8 @@ impl GraphDeBruijn {
.then_some(kmer)
})
.for_each(|kmer| {
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
let Some(direct) = self.start_direction(kmer) else { return };
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
n_new.fetch_add(1, Ordering::Relaxed);
f(iter);
}
@@ -166,7 +162,8 @@ impl GraphDeBruijn {
#[cfg(any(test, feature = "test-utils"))]
self.nodes.iter().for_each(|(&kmer, atomic)| {
if Node(atomic.load(Ordering::Acquire)).is_start() {
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
let Some(direct) = self.start_direction(kmer) else { return };
if let Some(iter) = self.unitig_nucleotides(kmer, k, direct) {
n_new.fetch_add(1, Ordering::Relaxed);
f(iter);
}
@@ -187,8 +184,8 @@ impl GraphDeBruijn {
if node.is_visited() {
continue;
}
let chain_start = self.find_chain_start(kmer);
if let Some(iter) = self.unitig_nucleotides(chain_start, k) {
let (chain_start, direct) = self.find_chain_start(kmer);
if let Some(iter) = self.unitig_nucleotides(chain_start, k, direct) {
n2.fetch_add(1, Ordering::Relaxed);
f(iter);
}
@@ -196,7 +193,7 @@ impl GraphDeBruijn {
// Safe because unitig_nucleotides may have visited kmer in the
// meantime — in that case it returns None and we skip.
if chain_start != kmer {
if let Some(iter) = self.unitig_nucleotides(kmer, k) {
if let Some(iter) = self.unitig_nucleotides(kmer, k, true) {
n2.fetch_add(1, Ordering::Relaxed);
f(iter);
}
@@ -216,17 +213,34 @@ impl GraphDeBruijn {
self.nodes.extend(other.nodes);
}
fn find_chain_start(&self, kmer: CanonicalKmer) -> CanonicalKmer {
let node = Node(self.nodes[&kmer].load(Ordering::Acquire));
let mut state = WalkState::new(kmer, node, false);
let mut seen = std::collections::HashSet::new();
/// Walks backwards from `kmer` to the first node of its chain. Returns
/// that node together with the orientation to read it in to walk the chain
/// forwards: the backward walk traverses nodes in the orientation
/// `state.direct`, so the forward one is its opposite. (Dropping the
/// orientation, as this function once did, started the forward walk at the
/// wrong end and emitted a lone-kmer unitig.) On a cycle, `kmer` itself
/// in the canonical orientation.
fn find_chain_start(&self, kmer: CanonicalKmer) -> (CanonicalKmer, bool) {
let atomic = &self.nodes[&kmer];
let node = Node(atomic.load(Ordering::Acquire));
let mut state = WalkState::new(kmer, node, false, atomic);
// Most nodes reaching this pass are branch nodes or tips that cannot
// step at all: answer before allocating the cycle-detection set.
let Some((first, _)) = state.walk(self) else {
return (state.kmer, !state.direct);
};
let mut seen: HashSet<(u64, bool), KmerHashBuilder> = HashSet::with_hasher(KmerHashBuilder);
seen.insert((state.kmer.raw(), state.direct));
if !seen.insert((first.kmer.raw(), first.direct)) {
return (kmer, true);
}
state = first;
loop {
match state.walk(self) {
None => return state.kmer,
None => return (state.kmer, !state.direct),
Some((next, _)) => {
if !seen.insert((next.kmer.raw(), next.direct)) {
return kmer;
return (kmer, true);
}
state = next;
}
@@ -234,8 +248,23 @@ impl GraphDeBruijn {
}
}
fn is_start(&self, query: CanonicalKmer, node: Node) -> bool {
!WalkState::new(query, node, true).reachable(self)
/// Orientation in which a node flagged as a start must be read: the
/// canonical one if a unitig starts there that way, else the reverse
/// complement. `None` if it was visited in the meantime.
fn start_direction(&self, kmer: CanonicalKmer) -> Option<bool> {
let atomic = self.nodes.get(&kmer)?;
let node = Node(atomic.load(Ordering::Acquire));
if node.is_visited() {
return None;
}
Some(self.is_start(kmer, node, atomic, true) || !self.is_start(kmer, node, atomic, false))
}
/// `true` when `query`, traversed in orientation `direct` (canonical or
/// reverse-complemented), has no unique predecessor — i.e. a unitig
/// starts there when read in that orientation.
fn is_start(&self, query: CanonicalKmer, node: Node, atomic: &AtomicU8, direct: bool) -> bool {
!WalkState::new(query, node, direct, atomic).reachable(self)
}
pub fn try_for_each_unitig<E, F>(&self, f: F) -> Result<(), E>
+55
View File
@@ -0,0 +1,55 @@
//! Hasher for the graph's `HashMap<CanonicalKmer, _>`.
//!
//! The key is one `u64` (the derived `Hash` of a kmer is a single
//! `write_u64`), so the generic streaming hashers used before (xxh3's
//! `Hasher` implementation: buffer, update, finish) cost far more than the
//! hash itself — ≈ 35 % of the active CPU of `index` in the profile of
//! `DevDocMD/implementation/mphf_batch_queries.md`, §17. A 64-bit finaliser
//! (splitmix64's) is a bijection with full avalanche, which is what
//! hashbrown needs from it (low bits for the bucket, top 7 for the tag).
use std::hash::{BuildHasher, Hasher};
#[derive(Clone, Copy, Default)]
pub(super) struct KmerHashBuilder;
impl BuildHasher for KmerHashBuilder {
type Hasher = KmerHasher;
#[inline]
fn build_hasher(&self) -> KmerHasher {
KmerHasher(0)
}
}
pub(super) struct KmerHasher(u64);
#[inline]
fn mix64(x: u64) -> u64 {
let x = x ^ (x >> 30);
let x = x.wrapping_mul(0xbf58476d1ce4e5b9);
let x = x ^ (x >> 27);
let x = x.wrapping_mul(0x94d049bb133111eb);
x ^ (x >> 31)
}
impl Hasher for KmerHasher {
#[inline]
fn write_u64(&mut self, x: u64) {
self.0 = mix64(self.0 ^ x);
}
/// Generic fallback (never taken for a kmer key): folds the bytes 8 at a time.
fn write(&mut self, bytes: &[u8]) {
for chunk in bytes.chunks(8) {
let mut buf = [0u8; 8];
buf[..chunk.len()].copy_from_slice(chunk);
self.write_u64(u64::from_le_bytes(buf));
}
}
#[inline]
fn finish(&self) -> u64 {
self.0
}
}
+2 -1
View File
@@ -1,10 +1,11 @@
//! De Bruijn graph over canonical k-mers, built for unitig extraction.
//!
//! Submodules: [`node`] (packed per-kmer neighbour/visited/start flags),
//! Submodules: [`kmer_hash`] (hasher of the node table), [`node`] (packed per-kmer neighbour/visited/start flags),
//! [`walk`] (single-step traversal), [`graph`] ([`GraphDeBruijn`] itself),
//! [`unitig_iter`] (nucleotide-by-nucleotide unitig walk iterator).
mod graph;
mod kmer_hash;
mod node;
mod unitig_iter;
mod walk;
+12 -8
View File
@@ -10,9 +10,11 @@ use super::walk::WalkState;
pub struct UnitigNucIter<'a> {
graph: &'a GraphDeBruijn,
start: CanonicalKmer,
/// Orientation the unitig is read in (`false`: reverse complement of `start`).
direct: bool,
pos: usize,
k: usize,
next_step: Option<(WalkState, u8)>,
next_step: Option<(WalkState<'a>, u8)>,
}
impl<'a> UnitigNucIter<'a> {
@@ -20,11 +22,13 @@ impl<'a> UnitigNucIter<'a> {
graph: &'a GraphDeBruijn,
start: CanonicalKmer,
k: usize,
next_step: Option<(WalkState, u8)>,
direct: bool,
next_step: Option<(WalkState<'a>, u8)>,
) -> Self {
Self {
graph,
start,
direct,
pos: 0,
k,
next_step,
@@ -37,16 +41,16 @@ impl Iterator for UnitigNucIter<'_> {
fn next(&mut self) -> Option<u8> {
if self.pos < self.k {
let nuc = self.start.nucleotide(self.pos);
let nuc = if self.direct {
self.start.nucleotide(self.pos)
} else {
3 - self.start.nucleotide(self.k - 1 - self.pos)
};
self.pos += 1;
Some(nuc)
} else if let Some((state, nuc)) = self.next_step.take() {
self.next_step = state.walk(self.graph).and_then(|(next_state, next_nuc)| {
let old = self
.graph
.nodes
.get(&next_state.kmer)?
.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
let old = next_state.atomic.fetch_or(IS_VISITED_MASK, Ordering::AcqRel);
(old & IS_VISITED_MASK == 0).then_some((next_state, next_nuc))
});
Some(nuc)
+22 -12
View File
@@ -1,35 +1,41 @@
use obikseq::{CanonicalKmer, Sequence};
use std::sync::atomic::Ordering;
use std::sync::atomic::{AtomicU8, Ordering};
use super::graph::GraphDeBruijn;
use super::node::Node;
pub struct WalkState {
/// A position in a walk: the node's kmer, its flags as read when the step was
/// taken, and the node's own table entry. Keeping `atomic` means the caller
/// that wants to mark the node visited does not look the kmer up a second
/// time (a random access in a table far larger than the cache).
pub struct WalkState<'g> {
pub(super) kmer: CanonicalKmer,
pub(super) node: Node,
pub(super) direct: bool,
pub(super) atomic: &'g AtomicU8,
}
impl WalkState {
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool) -> Self {
impl<'g> WalkState<'g> {
pub fn new(kmer: CanonicalKmer, node: Node, direct: bool, atomic: &'g AtomicU8) -> Self {
debug_assert!(!node.is_visited(), "Cannot walk over a visited node");
Self { kmer, node, direct }
Self { kmer, node, direct, atomic }
}
pub fn leavable(&self, graph: &GraphDeBruijn) -> bool {
pub fn leavable(&self, graph: &'g GraphDeBruijn) -> bool {
self.walk(graph).is_some()
}
pub fn reachable(&self, graph: &GraphDeBruijn) -> bool {
pub fn reachable(&self, graph: &'g GraphDeBruijn) -> bool {
WalkState {
kmer: self.kmer,
node: self.node,
direct: !self.direct,
atomic: self.atomic,
}
.leavable(graph)
}
pub fn walk(&self, graph: &GraphDeBruijn) -> Option<(WalkState, u8)> {
pub fn walk(&self, graph: &'g GraphDeBruijn) -> Option<(WalkState<'g>, u8)> {
if self.direct {
if !self.node.can_extend_right() {
return None;
@@ -38,7 +44,8 @@ impl WalkState {
let next = self.kmer.into_kmer().push_right(nuc);
let cnext = next.canonical();
let dnext = next.raw() == cnext.raw();
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed));
let next_atomic = graph.nodes.get(&cnext).unwrap();
let next_node = Node(next_atomic.load(Ordering::Relaxed));
if next_node.is_visited() {
return None;
}
@@ -52,6 +59,7 @@ impl WalkState {
kmer: cnext,
node: next_node,
direct: dnext,
atomic: next_atomic,
},
nuc,
))
@@ -63,20 +71,22 @@ impl WalkState {
let next = self.kmer.into_kmer().push_left(nuc);
let cnext = next.canonical();
let dnext = next.raw() != cnext.raw();
let next_node = Node(graph.nodes.get(&cnext).unwrap().load(Ordering::Relaxed));
let next_atomic = graph.nodes.get(&cnext).unwrap();
let next_node = Node(next_atomic.load(Ordering::Relaxed));
if next_node.is_visited() {
return None;
}
let reachable = if dnext {
next_node.can_extend_right()
} else {
next_node.can_extend_left()
} else {
next_node.can_extend_right()
};
reachable.then_some((
WalkState {
kmer: cnext,
node: next_node,
direct: dnext,
atomic: next_atomic,
},
3 - nuc,
))
+83
View File
@@ -317,3 +317,86 @@ fn branching_graph_no_kmer_lost_or_duplicated() {
);
assert_eq!(got, expected, "k-mer sets differ");
}
// ── orientation independence ──────────────────────────────────────────────
// Deterministic pseudo-random ASCII sequence (xorshift), so the tests need no
// RNG dependency.
fn pseudo_random_ascii(seed: u64, len: usize) -> Vec<u8> {
let mut st = seed.wrapping_mul(0x9E3779B97F4A7C15) | 1;
(0..len)
.map(|_| {
st ^= st << 13;
st ^= st >> 7;
st ^= st << 17;
b"ACGT"[(st % 4) as usize]
})
.collect()
}
#[test]
fn linear_chain_is_one_unitig_whatever_the_orientations() {
// A repeat-free sequence is one maximal unitig. Each k-mer is stored in
// its canonical orientation, so across many sequences the chain's nodes
// take every mix of orientations — including chains whose two ends are
// both stored reverse-complemented, which once came out as several
// unitigs (a lone-kmer unitig per mis-oriented chain start) depending
// only on the table's iteration order.
let k = 21; // long enough that (k-1)-mer collisions (= branches) do not occur
let _guard = lock_k();
set_k(k);
for seed in 1..=300u64 {
let seq = pseudo_random_ascii(seed, 90);
let kmers = canonical_kmers(&seq);
if kmers.len() != seq.len() - k + 1 {
continue; // accidental internal repeat: not a single chain
}
let g = graph_from_ascii(&seq);
g.compute_degrees_and_mark_starts();
let unitigs = collect_unitigs(&g);
assert_eq!(unitigs.len(), 1, "seed {seed}: repeat-free sequence must give one unitig");
assert_eq!(unitigs[0].seql(), seq.len(), "seed {seed}: unitig spans the sequence");
assert_eq!(kmers_from_unitigs(&unitigs), kmers, "seed {seed}: same k-mers");
}
}
#[test]
fn unitigs_partition_the_kmers_of_branching_reads() {
// Reads of a small genome on both strands with substitutions: branches,
// bubbles and tips. Whatever the shape, every k-mer must be covered
// exactly once: sum over unitigs of (seql - k + 1) == number of nodes.
let k = 11;
let _guard = lock_k();
set_k(k);
let genome = pseudo_random_ascii(7, 600);
let mut g = GraphDeBruijn::new();
for r in 0..120u64 {
let start = (pseudo_random_ascii(100 + r, 1)[0] as usize * 31 + r as usize * 17) % 500;
let mut read = genome[start..start + 90].to_vec();
if r % 2 == 1 {
// second strand
read = read
.iter()
.rev()
.map(|&b| match b {
b'A' => b'T',
b'C' => b'G',
b'G' => b'C',
_ => b'A',
})
.collect();
}
if r % 3 == 0 {
let at = (r as usize * 13) % read.len();
read[at] = if read[at] == b'A' { b'C' } else { b'A' };
}
for i in 0..=read.len() - k {
g.push(Kmer::from_ascii(&read[i..i + k]).unwrap().canonical());
}
}
g.compute_degrees_and_mark_starts();
let unitigs = collect_unitigs(&g);
let covered: usize = unitigs.iter().map(|u| u.seql() - k + 1).sum();
assert_eq!(covered, g.len(), "every k-mer in exactly one unitig");
assert_eq!(kmers_from_unitigs(&unitigs).len(), g.len(), "no k-mer missing");
}
+165
View File
@@ -0,0 +1,165 @@
//! Throwaway benchmark of the de Bruijn graph on a real kmer set:
//! `graph_bench <unitigs.bin> <k> [repeat]` builds `GraphDeBruijn` from the
//! canonical kmers of an existing layer's unitigs (same call sequence as
//! `write_graph_as_unitigs`) and times each phase.
use std::path::PathBuf;
use std::time::Instant;
use obidebruinj::GraphDeBruijn;
use obikseq::Sequence;
use obiskio::CanonicalKmerIter;
use rayon::prelude::*;
use std::collections::HashSet;
fn main() {
let args: Vec<String> = std::env::args().collect();
let path = PathBuf::from(&args[1]);
let k: usize = args[2].parse().unwrap();
let repeat: usize = args.get(3).map_or(1, |s| s.parse().unwrap());
obikseq::set_k(k);
obikseq::set_m(11);
let kmers: Vec<_> = if let Ok(n) = std::env::var("SYNTH") {
// synthetic reads: random genome, reads on both strands, 1 % substitutions
let n: usize = n.parse().unwrap();
let mut st = 0x9E3779B97F4A7C15u64;
let mut rnd = move || {
st ^= st << 13;
st ^= st >> 7;
st ^= st << 17;
st
};
let genome: Vec<u8> = (0..n).map(|_| b"ACGT"[(rnd() % 4) as usize]).collect();
let mut set = std::collections::BTreeSet::new();
for _ in 0..(n / 50) {
let start = (rnd() as usize) % (n - 100);
let mut read: Vec<u8> = genome[start..start + 100].to_vec();
for b in read.iter_mut() {
if rnd() % 100 == 0 {
*b = b"ACGT"[(rnd() % 4) as usize];
}
}
for w in read.windows(k) {
set.insert(obikseq::Kmer::from_ascii(w).unwrap().canonical());
}
}
set.into_iter().collect()
} else {
CanonicalKmerIter::new(&path).unwrap().collect()
};
eprintln!("{} kmers", kmers.len());
// Independent count of maximal unitigs: nodes − mutually-unique links
// (cycles ignored), computed with a plain HashSet — no use of GraphDeBruijn.
if std::env::var("IDEAL").is_ok() {
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
(0..4u8)
.map(|x| a.push_right(x))
.filter(|b| set.contains(&b.canonical().raw()))
.collect()
};
let links: usize = kmers
.par_iter()
.map(|c| {
let mut n = 0usize;
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
let ob = outs(o);
if ob.len() == 1 {
let b = ob[0];
// in-neighbours of b = revcomp of out-neighbours of revcomp(b)
if outs(b.revcomp()).len() == 1 {
// count each undirected link once: only from the orientation
// whose canonical raw is the smaller end (ties: both seen once per orientation)
let (ca, cb) = (o.canonical().raw(), b.canonical().raw());
if ca < cb || (ca == cb && o.raw() <= b.raw()) {
n += 1;
} else if ca > cb {
// seen from the other end too: skip
}
}
}
}
n
})
.sum();
eprintln!("IDEAL: nodes={} links={} -> maximal unitigs ≈ {}", kmers.len(), links, kmers.len() - links);
}
if std::env::var("IDEALDUMP").is_ok() {
let set: HashSet<u64> = kmers.iter().map(|c| c.raw()).collect();
let outs = |a: obikseq::Kmer| -> Vec<obikseq::Kmer> {
(0..4u8).map(|x| a.push_right(x)).filter(|b| set.contains(&b.canonical().raw())).collect()
};
// oriented kmer a has a mutual predecessor p iff ins(a)=={p} and outs(p)=={a}
let pred = |a: obikseq::Kmer| -> Option<obikseq::Kmer> {
let ins: Vec<obikseq::Kmer> = outs(a.revcomp()).into_iter().map(|b| b.revcomp()).collect();
if ins.len() == 1 && outs(ins[0]).len() == 1 { Some(ins[0]) } else { None }
};
let mut seen_chains = std::collections::BTreeSet::new();
for c in &kmers {
for o in [c.into_kmer(), c.into_kmer().revcomp()] {
if pred(o).is_some() { continue; }
let mut text: Vec<u8> = (0..k).map(|i| b"ACGT"[o.nucleotide(i) as usize]).collect();
let mut cur = o;
loop {
let ob = outs(cur);
if ob.len() != 1 { break; }
let b = ob[0];
if pred(b).is_none() { break; }
text.push(b"ACGT"[b.nucleotide(k - 1) as usize]);
cur = b;
}
let s = String::from_utf8(text).unwrap();
let rc: String = s.bytes().rev().map(|c| match c { b'A' => 'T', b'C' => 'G', b'G' => 'C', _ => 'A' }).collect();
seen_chains.insert(if s <= rc { s } else { rc });
}
}
for u in &seen_chains { println!("IDEALUNITIG {} {}", u.len(), u); }
}
for r in 0..repeat {
let t = Instant::now();
let mut g = GraphDeBruijn::new();
for &kmer in &kmers {
g.push(kmer);
}
let t_push = t.elapsed();
let t = Instant::now();
g.compute_degrees_and_mark_starts();
let t_deg = t.elapsed();
let t = Instant::now();
let n_unitigs = std::sync::atomic::AtomicUsize::new(0);
let n_nuc = std::sync::atomic::AtomicUsize::new(0);
let dump = std::env::var("DUMP").is_ok();
let seqs = std::sync::Mutex::new(Vec::<String>::new());
g.for_each_unitig(|it| {
let mut n = 0usize;
let mut text = String::new();
for nuc in it {
n += 1;
if dump {
text.push(b"ACGT"[nuc as usize] as char);
}
}
if dump {
seqs.lock().unwrap().push(text);
}
n_unitigs.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
n_nuc.fetch_add(n, std::sync::atomic::Ordering::Relaxed);
});
let t_walk = t.elapsed();
if dump {
let mut v = seqs.lock().unwrap().clone();
v.sort();
for u in v { println!("UNITIG {} {}", u.len(), u); }
}
println!(
"run {r}: nodes={} push={:.2?} degrees={:.2?} unitigs={:.2?} -> {} unitigs, {} nucleotides",
g.len(),
t_push,
t_deg,
t_walk,
n_unitigs.load(std::sync::atomic::Ordering::Relaxed),
n_nuc.load(std::sync::atomic::Ordering::Relaxed)
);
}
}