Refactor chunking logic to enforce 128 k-mer limit

This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
This commit is contained in:
Eric Coissac committed 2026-10-15 12:20:36 +02:00
1 parent 8468319a63
commit 025c40c0db
8 files changed
+80 -25

No files matched your search

@@ -299,3 +299,13 @@ Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorit
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
## 19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15)
`Evidence` packs `(chunk_id, rank)` into a `u32` with a 7-bit rank (`encode` masks `& 0x7F`), but `MAX_KMERS_PER_CHUNK` was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: **`--partitions 256`** → max chunk 49–50 k-mers, results correct with the old and the new binary; **`--partitions 1`** → max chunk 256, query output wrong with both the old (md5 `bde0f886a0`) and the post-graph-fix (`8f1c94ef32`) binary versus the correct `f99e971087`. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts.
**Change** (`obikseq::MAX_KMERS_PER_CHUNK`: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (`SuperKmer::write_to_binary`): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test `chunk_ranks_fit_the_7_bit_evidence_field`.
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
+4 -4
View File
@@ -22,10 +22,10 @@ A sequence of binary records. Each record:
- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
- Byte count for packed sequence: `ceil(seql / 4)`.
Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
Unitigs with more than `MAX_KMERS_PER_CHUNK = 128` k-mers are transparently split into overlapping chunks. Each chunk has at most 128 k-mers (= `seql − k + 1 ≤ 128`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
```
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (256 kmers)
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (128 kmers)
chunk 2: nucleotides [256, end] (remaining kmers)
overlap: k−1 nucleotides shared between the two chunks
```
@@ -95,11 +95,11 @@ Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extract
| field | bits | range | capacity check (*B. nana*, 256 partitions) |
|------------|------|---------------|---------------------------------------------|
| `seql − k` | 8 | 0–255 | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` |
| `seql − k` | 8 | 0–127 used | max `n_kmers` per chunk = 128 = `MAX_KMERS_PER_CHUNK` (the field could carry 256; capped so that every rank fits 7 bits) |
| `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
| `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition |
The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total.
The rank field is 7 bits (max 127) and the rank of a k-mer is its index among the chunk's k-mers, so a chunk must hold at most 128 of them: that is what `MAX_KMERS_PER_CHUNK = 128` guarantees. (Until 2026-10-15 the cap was 256 and the earlier justification here — that rank counts only canonical k-mers, at most half the total — was wrong: `iter_indexed_canonical_kmers` yields one entry per position. Chunks of 129–256 k-mers occur with few partitions — 6 of them in a `--partitions 4` index of a 2 GB skim, none with 256 partitions where the observed maximum is ≈ 50 — and their k-mers beyond rank 127 were silently stored with a truncated rank. `scripts/check_unitig_chunks.py` finds such chunks in an existing index.)
---
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env python3
# check_unitig_chunks.py GLOB... — report unitigs.bin chunks holding more than 128 k-mers.
# Indexes built before MAX_KMERS_PER_CHUNK went from 256 to 128 (2026-10-15) could hold chunks whose
# rank does not fit the 7-bit evidence field; the k-mers beyond rank 127 of such a chunk then fail
# exact verification. Example: scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"
# (k is hard-coded below — edit it for an index built with another k.)
import sys,glob,mmap
k=31
def scan(path):
with open(path,'rb') as f:
m=mmap.mmap(f.fileno(),0,access=mmap.ACCESS_READ)
o=0; n=len(m); mx=0; over=0; chunks=0
while o<n:
c=m[o]+1; chunks+=1
if c>mx: mx=c
if c>128: over+=1
o+=1+(m[o]+k+3)//4
return mx,over,chunks
tot_over=0
for pat in sys.argv[1:]:
for f in sorted(glob.glob(pat)):
mx,over,ch=scan(f); tot_over+=over
print(f"{f.split('partitions/')[1]}: chunks={ch} max_kmers={mx} chunks>128={over}")
print("TOTAL chunks over 128 kmers:",tot_over)
+7 -2
View File
@@ -22,8 +22,13 @@ use crate::kmer::{CanonicalKmer, Kmer, KmerError, KLen, KmerLength, KmerOf, MLen
use crate::params::k;
use crate::revcomp_lookup::REVCOMP4;
/// Maximum kmers per stored chunk. Enforces the u8 max-kmer-index field in the binary format.
pub const MAX_KMERS_PER_CHUNK: usize = 256;
/// Maximum kmers per stored chunk.
///
/// 128 = 2^7, not the 256 the u8 length field of the binary format could
/// carry: the exact evidence of a layer (`obikindex::layer::evidence`) stores a
/// kmer's rank inside its unitig chunk on 7 bits, so every rank must be ≤ 127.
/// Longer unitigs are split into consecutive chunks at write time.
pub const MAX_KMERS_PER_CHUNK: usize = 128;
// ── PackedSeq ─────────────────────────────────────────────────────────────────
+4 -4
View File
@@ -70,7 +70,7 @@ fn seql_roundtrip() {
fn binary_test_lengths(k: usize) -> Vec<usize> {
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
// Only single-chunk lengths: seql in [k, MAX_KMERS_PER_CHUNK+k-1].
(k..=k + 5).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect()
(k..=k + 5).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect()
}
#[test]
@@ -94,7 +94,7 @@ fn binary_split_roundtrip() {
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
set_k(4);
let k = crate::params::k();
// seql = MAX_KMERS_PER_CHUNK + k = 260 → n_kmers = 257 > 256 → 2 chunks
// seql = MAX_KMERS_PER_CHUNK + k → n_kmers = MAX + 1 > MAX → 2 chunks
let seql = MAX_KMERS_PER_CHUNK + k;
let mut sk = SuperKmer::from_ascii(&make_seq(seql));
sk.set_count(7);
@@ -108,8 +108,8 @@ fn binary_split_roundtrip() {
assert_eq!(chunk0.count(), 7);
assert_eq!(chunk1.count(), 7);
// Chunks cover the original sequence with k-1 overlap — no kmer lost.
assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1); // 259
assert_eq!(chunk1.seql(), k); // 4 (1 kmer)
assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1);
assert_eq!(chunk1.seql(), k); // 1 kmer
}
#[test]
+1 -1
View File
@@ -164,7 +164,7 @@ mod tests {
use crate::packed_seq::MAX_KMERS_PER_CHUNK;
set_k(4);
let k = crate::params::k();
let valid_lengths: Vec<usize> = (k..=9).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect();
let valid_lengths: Vec<usize> = (k..=9).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect();
for len in valid_lengths {
let u = Unitig::from_ascii(&make_seq(len));
let mut buf = Vec::new();
+3 -1
View File
@@ -104,7 +104,9 @@ fn roundtrip_single() {
fn roundtrip_all_lengths() {
setup();
let bases: Vec<u8> = (0..300).map(|i| b"ACGT"[i % 4]).collect();
for len in (TEST_K..=19).chain([255, 256, 257]) {
// single-chunk lengths only: seql ≤ MAX_KMERS_PER_CHUNK + k - 1
let max_len = obikseq::MAX_KMERS_PER_CHUNK + TEST_K - 1;
for len in (TEST_K..=19).chain([max_len - 2, max_len - 1, max_len]) {
let sk = make_sk(&bases[..len]);
let mut buf = Vec::new();
sk.write_to_binary(&mut buf).unwrap();
+27 -13
View File
@@ -222,31 +222,45 @@ fn iter_indexed_chunk_id_and_rank_two_chunks() {
#[test]
fn short_unitig_not_split() {
// seql=259 → n_kmers=256 = MAX_KMERS_PER_CHUNK → no split
// seql=131 → n_kmers=128 = MAX_KMERS_PER_CHUNK → no split
set_k(4);
let seq: Vec<u8> = (0..259_usize).map(|i| b"ACGT"[i % 4]).collect();
let seq: Vec<u8> = (0..131_usize).map(|i| b"ACGT"[i % 4]).collect();
let (_dir, r) = write_read(&[&seq]);
assert_eq!(r.len(), 1);
assert_eq!(r.seql(0), 259);
assert_eq!(r.seql(0), 131);
}
#[test]
fn long_unitig_split_no_kmer_lost() {
// seql=260 → n_kmers=257 > MAX_KMERS_PER_CHUNK(256) → 2 chunks
// chunk_nucl=259, stride=256
// Chunk 0: nucl 0..259 (259 nucl, 256 kmers)
// Chunk 1: nucl 256..260 (4 nucl, 1 kmer)
// seql=132 → n_kmers=129 > MAX_KMERS_PER_CHUNK(128) → 2 chunks
// chunk_nucl=131, stride=128
// Chunk 0: nucl 0..131 (131 nucl, 128 kmers)
// Chunk 1: nucl 128..132 (4 nucl, 1 kmer)
set_k(4);
let seq: Vec<u8> = (0..260_usize).map(|i| b"ACGT"[i % 4]).collect();
let seq: Vec<u8> = (0..132_usize).map(|i| b"ACGT"[i % 4]).collect();
let (_dir, r) = write_read(&[&seq]);
assert_eq!(r.len(), 2);
assert_eq!(r.seql(0), 259);
assert_eq!(r.seql(0), 131);
assert_eq!(r.seql(1), 4);
// k-1=3 nucleotide overlap → 0 kmers duplicated, 0 kmers lost.
// Last kmer of chunk 0 = original nucl 255..259.
assert!(r.verify_canonical_kmer(0, 255, canonical_of(&seq[255..259])));
// First kmer of chunk 1 = original nucl 256..260 — a different, adjacent kmer.
assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[256..260])));
// Last kmer of chunk 0 = original nucl 127..131.
assert!(r.verify_canonical_kmer(0, 127, canonical_of(&seq[127..131])));
// First kmer of chunk 1 = original nucl 128..132 — a different, adjacent kmer.
assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[128..132])));
}
#[test]
fn chunk_ranks_fit_the_7_bit_evidence_field() {
// The exact evidence of a layer stores a kmer's rank in its chunk on 7
// bits (obikindex::layer::evidence): a unitig long enough to need several
// chunks must never yield a rank above 127 — a larger one would be
// silently truncated there and the kmer would no longer verify.
set_k(4);
let seq: Vec<u8> = (0..1000_usize).map(|i| b"ACGT"[(i * i + i / 3) % 4]).collect();
let (_dir, r) = write_read(&[&seq]);
assert!(r.len() > 1, "a 997-kmer unitig must be split into several chunks");
let max_rank = r.iter_indexed_canonical_kmers().map(|(_, _, rank)| rank).max().unwrap();
assert!(max_rank <= 127, "rank {max_rank} does not fit 7 bits");
}
// ── block_bits parametrisation ────────────────────────────────────────────────