diff --git a/DevDocMD/implementation/mphf_batch_queries.md b/DevDocMD/implementation/mphf_batch_queries.md index d4fa0e48..d5ba755b 100644 --- a/DevDocMD/implementation/mphf_batch_queries.md +++ b/DevDocMD/implementation/mphf_batch_queries.md @@ -299,3 +299,13 @@ Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorit | E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s | Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs. + +## 19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15) + +`Evidence` packs `(chunk_id, rank)` into a `u32` with a 7-bit rank (`encode` masks `& 0x7F`), but `MAX_KMERS_PER_CHUNK` was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: **`--partitions 256`** → max chunk 49–50 k-mers, results correct with the old and the new binary; **`--partitions 1`** → max chunk 256, query output wrong with both the old (md5 `bde0f886a0`) and the post-graph-fix (`8f1c94ef32`) binary versus the correct `f99e971087`. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts. + +**Change** (`obikseq::MAX_KMERS_PER_CHUNK`: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (`SuperKmer::write_to_binary`): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test `chunk_ranks_fit_the_7_bit_evidence_field`. + +**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed. + +**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top). diff --git a/DevDocMD/implementation/unitig_evidence.md b/DevDocMD/implementation/unitig_evidence.md index 6d51d753..e814a618 100644 --- a/DevDocMD/implementation/unitig_evidence.md +++ b/DevDocMD/implementation/unitig_evidence.md @@ -22,10 +22,10 @@ A sequence of binary records. Each record: - Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded. - Byte count for packed sequence: `ceil(seql / 4)`. -Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost: +Unitigs with more than `MAX_KMERS_PER_CHUNK = 128` k-mers are transparently split into overlapping chunks. Each chunk has at most 128 k-mers (= `seql − k + 1 ≤ 128`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost: ``` -chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (256 kmers) +chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (128 kmers) chunk 2: nucleotides [256, end] (remaining kmers) overlap: k−1 nucleotides shared between the two chunks ``` @@ -95,11 +95,11 @@ Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extract | field | bits | range | capacity check (*B. nana*, 256 partitions) | |------------|------|---------------|---------------------------------------------| -| `seql − k` | 8 | 0–255 | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` | +| `seql − k` | 8 | 0–127 used | max `n_kmers` per chunk = 128 = `MAX_KMERS_PER_CHUNK` (the field could carry 256; capped so that every rank fits 7 bits) | | `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 | | `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition | -The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total. +The rank field is 7 bits (max 127) and the rank of a k-mer is its index among the chunk's k-mers, so a chunk must hold at most 128 of them: that is what `MAX_KMERS_PER_CHUNK = 128` guarantees. (Until 2026-10-15 the cap was 256 and the earlier justification here — that rank counts only canonical k-mers, at most half the total — was wrong: `iter_indexed_canonical_kmers` yields one entry per position. Chunks of 129–256 k-mers occur with few partitions — 6 of them in a `--partitions 4` index of a 2 GB skim, none with 256 partitions where the observed maximum is ≈ 50 — and their k-mers beyond rank 127 were silently stored with a truncated rank. `scripts/check_unitig_chunks.py` finds such chunks in an existing index.) --- diff --git a/scripts/check_unitig_chunks.py b/scripts/check_unitig_chunks.py new file mode 100755 index 00000000..b9e247ca --- /dev/null +++ b/scripts/check_unitig_chunks.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# check_unitig_chunks.py GLOB... — report unitigs.bin chunks holding more than 128 k-mers. +# Indexes built before MAX_KMERS_PER_CHUNK went from 256 to 128 (2026-10-15) could hold chunks whose +# rank does not fit the 7-bit evidence field; the k-mers beyond rank 127 of such a chunk then fail +# exact verification. Example: scripts/check_unitig_chunks.py "/partitions/part_*/index/layer_*/unitigs.bin" +# (k is hard-coded below — edit it for an index built with another k.) +import sys,glob,mmap +k=31 +def scan(path): + with open(path,'rb') as f: + m=mmap.mmap(f.fileno(),0,access=mmap.ACCESS_READ) + o=0; n=len(m); mx=0; over=0; chunks=0 + while omx: mx=c + if c>128: over+=1 + o+=1+(m[o]+k+3)//4 + return mx,over,chunks +tot_over=0 +for pat in sys.argv[1:]: + for f in sorted(glob.glob(pat)): + mx,over,ch=scan(f); tot_over+=over + print(f"{f.split('partitions/')[1]}: chunks={ch} max_kmers={mx} chunks>128={over}") +print("TOTAL chunks over 128 kmers:",tot_over) diff --git a/src/obikseq/src/packed_seq.rs b/src/obikseq/src/packed_seq.rs index 136d500e..580b9d8f 100644 --- a/src/obikseq/src/packed_seq.rs +++ b/src/obikseq/src/packed_seq.rs @@ -22,8 +22,13 @@ use crate::kmer::{CanonicalKmer, Kmer, KmerError, KLen, KmerLength, KmerOf, MLen use crate::params::k; use crate::revcomp_lookup::REVCOMP4; -/// Maximum kmers per stored chunk. Enforces the u8 max-kmer-index field in the binary format. -pub const MAX_KMERS_PER_CHUNK: usize = 256; +/// Maximum kmers per stored chunk. +/// +/// 128 = 2^7, not the 256 the u8 length field of the binary format could +/// carry: the exact evidence of a layer (`obikindex::layer::evidence`) stores a +/// kmer's rank inside its unitig chunk on 7 bits, so every rank must be ≤ 127. +/// Longer unitigs are split into consecutive chunks at write time. +pub const MAX_KMERS_PER_CHUNK: usize = 128; // ── PackedSeq ───────────────────────────────────────────────────────────────── diff --git a/src/obikseq/src/tests/superkmer.rs b/src/obikseq/src/tests/superkmer.rs index 19930282..3cc05627 100644 --- a/src/obikseq/src/tests/superkmer.rs +++ b/src/obikseq/src/tests/superkmer.rs @@ -70,7 +70,7 @@ fn seql_roundtrip() { fn binary_test_lengths(k: usize) -> Vec { use crate::packed_seq::MAX_KMERS_PER_CHUNK; // Only single-chunk lengths: seql in [k, MAX_KMERS_PER_CHUNK+k-1]. - (k..=k + 5).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect() + (k..=k + 5).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect() } #[test] @@ -94,7 +94,7 @@ fn binary_split_roundtrip() { use crate::packed_seq::MAX_KMERS_PER_CHUNK; set_k(4); let k = crate::params::k(); - // seql = MAX_KMERS_PER_CHUNK + k = 260 → n_kmers = 257 > 256 → 2 chunks + // seql = MAX_KMERS_PER_CHUNK + k → n_kmers = MAX + 1 > MAX → 2 chunks let seql = MAX_KMERS_PER_CHUNK + k; let mut sk = SuperKmer::from_ascii(&make_seq(seql)); sk.set_count(7); @@ -108,8 +108,8 @@ fn binary_split_roundtrip() { assert_eq!(chunk0.count(), 7); assert_eq!(chunk1.count(), 7); // Chunks cover the original sequence with k-1 overlap — no kmer lost. - assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1); // 259 - assert_eq!(chunk1.seql(), k); // 4 (1 kmer) + assert_eq!(chunk0.seql(), MAX_KMERS_PER_CHUNK + k - 1); + assert_eq!(chunk1.seql(), k); // 1 kmer } #[test] diff --git a/src/obikseq/src/tests/unitig.rs b/src/obikseq/src/tests/unitig.rs index 54dddbff..a7b7673e 100644 --- a/src/obikseq/src/tests/unitig.rs +++ b/src/obikseq/src/tests/unitig.rs @@ -164,7 +164,7 @@ mod tests { use crate::packed_seq::MAX_KMERS_PER_CHUNK; set_k(4); let k = crate::params::k(); - let valid_lengths: Vec = (k..=9).chain([255, 256, 257, MAX_KMERS_PER_CHUNK + k - 1]).collect(); + let valid_lengths: Vec = (k..=9).chain([MAX_KMERS_PER_CHUNK + k - 3, MAX_KMERS_PER_CHUNK + k - 2, MAX_KMERS_PER_CHUNK + k - 1]).collect(); for len in valid_lengths { let u = Unitig::from_ascii(&make_seq(len)); let mut buf = Vec::new(); diff --git a/src/obiskio/src/tests/reader.rs b/src/obiskio/src/tests/reader.rs index cca5c08e..480af82a 100644 --- a/src/obiskio/src/tests/reader.rs +++ b/src/obiskio/src/tests/reader.rs @@ -104,7 +104,9 @@ fn roundtrip_single() { fn roundtrip_all_lengths() { setup(); let bases: Vec = (0..300).map(|i| b"ACGT"[i % 4]).collect(); - for len in (TEST_K..=19).chain([255, 256, 257]) { + // single-chunk lengths only: seql ≤ MAX_KMERS_PER_CHUNK + k - 1 + let max_len = obikseq::MAX_KMERS_PER_CHUNK + TEST_K - 1; + for len in (TEST_K..=19).chain([max_len - 2, max_len - 1, max_len]) { let sk = make_sk(&bases[..len]); let mut buf = Vec::new(); sk.write_to_binary(&mut buf).unwrap(); diff --git a/src/obiskio/src/tests/unitig_index.rs b/src/obiskio/src/tests/unitig_index.rs index f4f1ffbb..fc02626e 100644 --- a/src/obiskio/src/tests/unitig_index.rs +++ b/src/obiskio/src/tests/unitig_index.rs @@ -222,31 +222,45 @@ fn iter_indexed_chunk_id_and_rank_two_chunks() { #[test] fn short_unitig_not_split() { - // seql=259 → n_kmers=256 = MAX_KMERS_PER_CHUNK → no split + // seql=131 → n_kmers=128 = MAX_KMERS_PER_CHUNK → no split set_k(4); - let seq: Vec = (0..259_usize).map(|i| b"ACGT"[i % 4]).collect(); + let seq: Vec = (0..131_usize).map(|i| b"ACGT"[i % 4]).collect(); let (_dir, r) = write_read(&[&seq]); assert_eq!(r.len(), 1); - assert_eq!(r.seql(0), 259); + assert_eq!(r.seql(0), 131); } #[test] fn long_unitig_split_no_kmer_lost() { - // seql=260 → n_kmers=257 > MAX_KMERS_PER_CHUNK(256) → 2 chunks - // chunk_nucl=259, stride=256 - // Chunk 0: nucl 0..259 (259 nucl, 256 kmers) - // Chunk 1: nucl 256..260 (4 nucl, 1 kmer) + // seql=132 → n_kmers=129 > MAX_KMERS_PER_CHUNK(128) → 2 chunks + // chunk_nucl=131, stride=128 + // Chunk 0: nucl 0..131 (131 nucl, 128 kmers) + // Chunk 1: nucl 128..132 (4 nucl, 1 kmer) set_k(4); - let seq: Vec = (0..260_usize).map(|i| b"ACGT"[i % 4]).collect(); + let seq: Vec = (0..132_usize).map(|i| b"ACGT"[i % 4]).collect(); let (_dir, r) = write_read(&[&seq]); assert_eq!(r.len(), 2); - assert_eq!(r.seql(0), 259); + assert_eq!(r.seql(0), 131); assert_eq!(r.seql(1), 4); // k-1=3 nucleotide overlap → 0 kmers duplicated, 0 kmers lost. - // Last kmer of chunk 0 = original nucl 255..259. - assert!(r.verify_canonical_kmer(0, 255, canonical_of(&seq[255..259]))); - // First kmer of chunk 1 = original nucl 256..260 — a different, adjacent kmer. - assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[256..260]))); + // Last kmer of chunk 0 = original nucl 127..131. + assert!(r.verify_canonical_kmer(0, 127, canonical_of(&seq[127..131]))); + // First kmer of chunk 1 = original nucl 128..132 — a different, adjacent kmer. + assert!(r.verify_canonical_kmer(1, 0, canonical_of(&seq[128..132]))); +} + +#[test] +fn chunk_ranks_fit_the_7_bit_evidence_field() { + // The exact evidence of a layer stores a kmer's rank in its chunk on 7 + // bits (obikindex::layer::evidence): a unitig long enough to need several + // chunks must never yield a rank above 127 — a larger one would be + // silently truncated there and the kmer would no longer verify. + set_k(4); + let seq: Vec = (0..1000_usize).map(|i| b"ACGT"[(i * i + i / 3) % 4]).collect(); + let (_dir, r) = write_read(&[&seq]); + assert!(r.len() > 1, "a 997-kmer unitig must be split into several chunks"); + let max_rank = r.iter_indexed_canonical_kmers().map(|(_, _, rank)| rank).max().unwrap(); + assert!(max_rank <= 127, "rank {max_rank} does not fit 7 bits"); } // ── block_bits parametrisation ────────────────────────────────────────────────