Refactor chunking logic to enforce 128 k-mer limit
This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
This commit is contained in:
1 parent
8468319a63
commit
025c40c0db
8 files changed
+80
-25
No files matched your search
@@ -299,3 +299,13 @@ Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorit
|
|||||||
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
|
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
|
||||||
|
|
||||||
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
|
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
|
||||||
|
|
||||||
|
## 19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15)
|
||||||
|
|
||||||
|
`Evidence` packs `(chunk_id, rank)` into a `u32` with a 7-bit rank (`encode` masks `& 0x7F`), but `MAX_KMERS_PER_CHUNK` was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: **`--partitions 256`** → max chunk 49–50 k-mers, results correct with the old and the new binary; **`--partitions 1`** → max chunk 256, query output wrong with both the old (md5 `bde0f886a0`) and the post-graph-fix (`8f1c94ef32`) binary versus the correct `f99e971087`. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts.
|
||||||
|
|
||||||
|
**Change** (`obikseq::MAX_KMERS_PER_CHUNK`: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (`SuperKmer::write_to_binary`): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test `chunk_ranks_fit_the_7_bit_evidence_field`.
|
||||||
|
|
||||||
|
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
|
||||||
|
|
||||||
|
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
|
||||||
@@ -22,10 +22,10 @@ A sequence of binary records. Each record:
|
|||||||
- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
|
- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
|
||||||
- Byte count for packed sequence: `ceil(seql / 4)`.
|
- Byte count for packed sequence: `ceil(seql / 4)`.
|
||||||
|
|
||||||
Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
|
Unitigs with more than `MAX_KMERS_PER_CHUNK = 128` k-mers are transparently split into overlapping chunks. Each chunk has at most 128 k-mers (= `seql − k + 1 ≤ 128`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
|
||||||
|
|
||||||
```
|
```
|
||||||
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (256 kmers)
|
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (128 kmers)
|
||||||
chunk 2: nucleotides [256, end] (remaining kmers)
|
chunk 2: nucleotides [256, end] (remaining kmers)
|
||||||
overlap: k−1 nucleotides shared between the two chunks
|
overlap: k−1 nucleotides shared between the two chunks
|
||||||
```
|
```
|
||||||
@@ -95,11 +95,11 @@ Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extract
|
|||||||
|
|
||||||
| field | bits | range | capacity check (*B. nana*, 256 partitions) |
|
| field | bits | range | capacity check (*B. nana*, 256 partitions) |
|
||||||
|------------|------|---------------|---------------------------------------------|
|
|------------|------|---------------|---------------------------------------------|
|
||||||
| `seql − k` | 8 | 0–255 | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` |
|
| `seql − k` | 8 | 0–127 used | max `n_kmers` per chunk = 128 = `MAX_KMERS_PER_CHUNK` (the field could carry 256; capped so that every rank fits 7 bits) |
|
||||||
| `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
|
| `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
|
||||||
| `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition |
|
| `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition |
|
||||||
|
|
||||||
The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total.
|
The rank field is 7 bits (max 127) and the rank of a k-mer is its index among the chunk's k-mers, so a chunk must hold at most 128 of them: that is what `MAX_KMERS_PER_CHUNK = 128` guarantees. (Until 2026-10-15 the cap was 256 and the earlier justification here — that rank counts only canonical k-mers, at most half the total — was wrong: `iter_indexed_canonical_kmers` yields one entry per position. Chunks of 129–256 k-mers occur with few partitions — 6 of them in a `--partitions 4` index of a 2 GB skim, none with 256 partitions where the observed maximum is ≈ 50 — and their k-mers beyond rank 127 were silently stored with a truncated rank. `scripts/check_unitig_chunks.py` finds such chunks in an existing index.)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Executable
+24
@@ -0,0 +1,24 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# check_unitig_chunks.py GLOB... — report unitigs.bin chunks holding more than 128 k-mers.
|
||||||
|
# Indexes built before MAX_KMERS_PER_CHUNK went from 256 to 128 (2026-10-15) could hold chunks whose
|
||||||
|
# rank does not fit the 7-bit evidence field; the k-mers beyond rank 127 of such a chunk then fail
|
||||||
|
# exact verification. Example: scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"
|
||||||
|
# (k is hard-coded below — edit it for an index built with another k.)
|
||||||
|
import sys,glob,mmap
|
||||||
|
k=31
|
||||||
|
def scan(path):
|
||||||
|
with open(path,'rb') as f:
|
||||||
|
m=mmap.mmap(f.fileno(),0,access=mmap.ACCESS_READ)
|
||||||
|
o=0; n=len(m); mx=0; over=0; chunks=0
|
||||||
|
while o<n:
|
||||||
|
c=m[o]+1; chunks+=1
|
||||||
|
if c>mx: mx=c
|
||||||
|
if c>128: over+=1
|
||||||
|
o+=1+(m[o]+k+3)//4
|
||||||
|
return mx,over,chunks
|
||||||
|
tot_over=0
|
||||||
|
for pat in sys.argv[1:]:
|
||||||
|
for f in sorted(glob.glob(pat)):
|
||||||
|
mx,over,ch=scan(f); tot_over+=over
|
||||||
|
print(f"{f.split('partitions/')[1]}: chunks={ch} max_kmers={mx} chunks>128={over}")
|
||||||
|
print("TOTAL chunks over 128 kmers:",tot_over)
|
||||||
@@ -22,8 +22,13 @@ use crate::kmer::{CanonicalKmer, Kmer, KmerError, KLen, KmerLength, KmerOf, MLen
|
|||||||
use crate::params::k;
|
use crate::params::k;
|
||||||
use crate::revcomp_lookup::REVCOMP4;
|
use crate::revcomp_lookup::REVCOMP4;
|
||||||
|
|
||||||
/// Maximum kmers per stored chunk. Enforces the u8 max-kmer-index field in the binary format.
|
/// Maximum kmers per stored chunk.
|
||||||
pub const MAX_KMERS_PER_CHUNK: usize = 256;
|
///
|
||||||
|
/// 128 = 2^7, not the 256 the u8 length field of the binary format could
|
||||||
|
/// carry: the exact evidence of a layer (`obikindex::layer::evidence`) stores a
|
||||||
|
/// kmer's rank inside its unitig chunk on 7 bits, so every rank must be ≤ 127.
|
||||||
|
/// Longer unitigs are split into consecutive chunks at write time.
|
||||||
|
pub const MAX_KMERS_PER_CHUNK: usize = 128;
|
||||||
|
|
||||||
// ── PackedSeq ─────────────────────────────────────────────────────────────────
|
// ── PackedSeq ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ fn seql_roundtrip() {
|
|||||||
fn binary_test_lengths(k: usize) -> Vec<usize> {
|
fn binary_test_lengths(k: usize) -> Vec<usize> {
|
||||||
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
|
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
|
||||||
// Only single-chunk lengths: seql in [k, MAX_KMERS_PER_CHUNK+k-1].
|
// Only single-chunk lengths: seql in [k, MAX_KMERS_PER_CHUNK+k-1].
|
||||||
(k..=k + 5).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect()
|
(k..=k + 5).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -94,7 +94,7 @@ fn binary_split_roundtrip() {
|
|||||||
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
|
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
|
||||||
set_k(4);
|
set_k(4);
|
||||||
let k = crate::params::k();
|
let k = crate::params::k();
|
||||||
// seql = MAX_KMERS_PER_CHUNK + k = 260 → n_kmers = 257 > 256 → 2 chunks
|
// seql = MAX_KMERS_PER_CHUNK + k → n_kmers = MAX + 1 > MAX → 2 chunks
|
||||||
let seql = MAX_KMERS_PER_CHUNK + k;
|
let seql = MAX_KMERS_PER_CHUNK + k;
|
||||||
let mut sk = SuperKmer::from_ascii(&make_seq(seql));
|
let mut sk = SuperKmer::from_ascii(&make_seq(seql));
|
||||||
sk.set_count(7);
|
sk.set_count(7);
|
||||||
@@ -108,8 +108,8 @@ fn binary_split_roundtrip() {
|
|||||||
assert_eq!(chunk0.count(), 7);
|
assert_eq!(chunk0.count(), 7);
|
||||||
assert_eq!(chunk1.count(), 7);
|
assert_eq!(chunk1.count(), 7);
|
||||||
// Chunks cover the original sequence with k-1 overlap — no kmer lost.
|
// Chunks cover the original sequence with k-1 overlap — no kmer lost.
|
||||||
assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1); // 259
|
assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1);
|
||||||
assert_eq!(chunk1.seql(), k); // 4 (1 kmer)
|
assert_eq!(chunk1.seql(), k); // 1 kmer
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -164,7 +164,7 @@ mod tests {
|
|||||||
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
|
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
|
||||||
set_k(4);
|
set_k(4);
|
||||||
let k = crate::params::k();
|
let k = crate::params::k();
|
||||||
let valid_lengths: Vec<usize> = (k..=9).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect();
|
let valid_lengths: Vec<usize> = (k..=9).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect();
|
||||||
for len in valid_lengths {
|
for len in valid_lengths {
|
||||||
let u = Unitig::from_ascii(&make_seq(len));
|
let u = Unitig::from_ascii(&make_seq(len));
|
||||||
let mut buf = Vec::new();
|
let mut buf = Vec::new();
|
||||||
|
|||||||
@@ -104,7 +104,9 @@ fn roundtrip_single() {
|
|||||||
fn roundtrip_all_lengths() {
|
fn roundtrip_all_lengths() {
|
||||||
setup();
|
setup();
|
||||||
let bases: Vec<u8> = (0..300).map(|i| b"ACGT"[i % 4]).collect();
|
let bases: Vec<u8> = (0..300).map(|i| b"ACGT"[i % 4]).collect();
|
||||||
for len in (TEST_K..=19).chain([255, 256, 257]) {
|
// single-chunk lengths only: seql ≤ MAX_KMERS_PER_CHUNK + k - 1
|
||||||
|
let max_len = obikseq::MAX_KMERS_PER_CHUNK + TEST_K - 1;
|
||||||
|
for len in (TEST_K..=19).chain([max_len - 2, max_len - 1, max_len]) {
|
||||||
let sk = make_sk(&bases[..len]);
|
let sk = make_sk(&bases[..len]);
|
||||||
let mut buf = Vec::new();
|
let mut buf = Vec::new();
|
||||||
sk.write_to_binary(&mut buf).unwrap();
|
sk.write_to_binary(&mut buf).unwrap();
|
||||||
|
|||||||
@@ -222,31 +222,45 @@ fn iter_indexed_chunk_id_and_rank_two_chunks() {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn short_unitig_not_split() {
|
fn short_unitig_not_split() {
|
||||||
// seql=259 → n_kmers=256 = MAX_KMERS_PER_CHUNK → no split
|
// seql=131 → n_kmers=128 = MAX_KMERS_PER_CHUNK → no split
|
||||||
set_k(4);
|
set_k(4);
|
||||||
let seq: Vec<u8> = (0..259_usize).map(|i| b"ACGT"[i % 4]).collect();
|
let seq: Vec<u8> = (0..131_usize).map(|i| b"ACGT"[i % 4]).collect();
|
||||||
let (_dir, r) = write_read(&[&seq]);
|
let (_dir, r) = write_read(&[&seq]);
|
||||||
assert_eq!(r.len(), 1);
|
assert_eq!(r.len(), 1);
|
||||||
assert_eq!(r.seql(0), 259);
|
assert_eq!(r.seql(0), 131);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn long_unitig_split_no_kmer_lost() {
|
fn long_unitig_split_no_kmer_lost() {
|
||||||
// seql=260 → n_kmers=257 > MAX_KMERS_PER_CHUNK(256) → 2 chunks
|
// seql=132 → n_kmers=129 > MAX_KMERS_PER_CHUNK(128) → 2 chunks
|
||||||
// chunk_nucl=259, stride=256
|
// chunk_nucl=131, stride=128
|
||||||
// Chunk 0: nucl 0..259 (259 nucl, 256 kmers)
|
// Chunk 0: nucl 0..131 (131 nucl, 128 kmers)
|
||||||
// Chunk 1: nucl 256..260 (4 nucl, 1 kmer)
|
// Chunk 1: nucl 128..132 (4 nucl, 1 kmer)
|
||||||
set_k(4);
|
set_k(4);
|
||||||
let seq: Vec<u8> = (0..260_usize).map(|i| b"ACGT"[i % 4]).collect();
|
let seq: Vec<u8> = (0..132_usize).map(|i| b"ACGT"[i % 4]).collect();
|
||||||
let (_dir, r) = write_read(&[&seq]);
|
let (_dir, r) = write_read(&[&seq]);
|
||||||
assert_eq!(r.len(), 2);
|
assert_eq!(r.len(), 2);
|
||||||
assert_eq!(r.seql(0), 259);
|
assert_eq!(r.seql(0), 131);
|
||||||
assert_eq!(r.seql(1), 4);
|
assert_eq!(r.seql(1), 4);
|
||||||
// k-1=3 nucleotide overlap → 0 kmers duplicated, 0 kmers lost.
|
// k-1=3 nucleotide overlap → 0 kmers duplicated, 0 kmers lost.
|
||||||
// Last kmer of chunk 0 = original nucl 255..259.
|
// Last kmer of chunk 0 = original nucl 127..131.
|
||||||
assert!(r.verify_canonical_kmer(0, 255, canonical_of(&seq[255..259])));
|
assert!(r.verify_canonical_kmer(0, 127, canonical_of(&seq[127..131])));
|
||||||
// First kmer of chunk 1 = original nucl 256..260 — a different, adjacent kmer.
|
// First kmer of chunk 1 = original nucl 128..132 — a different, adjacent kmer.
|
||||||
assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[256..260])));
|
assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[128..132])));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn chunk_ranks_fit_the_7_bit_evidence_field() {
|
||||||
|
// The exact evidence of a layer stores a kmer's rank in its chunk on 7
|
||||||
|
// bits (obikindex::layer::evidence): a unitig long enough to need several
|
||||||
|
// chunks must never yield a rank above 127 — a larger one would be
|
||||||
|
// silently truncated there and the kmer would no longer verify.
|
||||||
|
set_k(4);
|
||||||
|
let seq: Vec<u8> = (0..1000_usize).map(|i| b"ACGT"[(i * i + i / 3) % 4]).collect();
|
||||||
|
let (_dir, r) = write_read(&[&seq]);
|
||||||
|
assert!(r.len() > 1, "a 997-kmer unitig must be split into several chunks");
|
||||||
|
let max_rank = r.iter_indexed_canonical_kmers().map(|(_, _, rank)| rank).max().unwrap();
|
||||||
|
assert!(max_rank <= 127, "rank {max_rank} does not fit 7 bits");
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── block_bits parametrisation ────────────────────────────────────────────────
|
// ── block_bits parametrisation ────────────────────────────────────────────────
|
||||||
|
|||||||
Reference in new issue
Block a user