# On-disk collection structure

Collections are too large to hold in RAM (hundreds of genomes, billions of kmers). The collection lives on disk as a directory of memory-mapped files:

```
collection/
  metadata.toml          — collection parameters (see below)
  part_XXXX/
    superkmers.bin.gz    — dereplicated super-kmers for this partition (construction artifact)
    mphf.bin             — minimal perfect hash function for this partition
    counts.bin           — packed n-bit count array (or 1-bit presence array)
    refs.bin             — back-references u32 nucleotide offset into unitigs.bin per kmer
    unitigs.bin          — local de Bruijn unitigs (permanent evidence structure)
    overflow.bin         — counts exceeding the packed range (optional)
```

`superkmers.bin.gz` is produced during phase 1 and consumed through phases 2–4. It can be deleted after phase 5 — it is not needed for querying. The permanent query structure is `mphf.bin + counts.bin + refs.bin + unitigs.bin`.

## Collection parameters

Stored in `metadata.toml`:

| Parameter | Role |
|-----------|------|
| k | kmer length |
| m | minimizer length (odd, < k) |
| p | partition bits (0 ≤ p ≤ min(14, 2m−16)) |
| mode | `presence` (1 bit/kmer) or `count` (n bits/kmer) |
| n | bits per kmer in count mode (chosen at construction) |
| min_count | singleton filtering threshold (0 = keep all) |
| hash_fn | hash function identifier |
| hash_seed | seed for the hash function |

## Count storage

**refs.bin capacity:** `unitigs.bin` is a flat 2-bit-packed nucleotide stream with no separators. Each entry in `refs.bin` is a u32 nucleotide offset pointing to the first base of the kmer. A u32 covers 4 billion nucleotide positions = 1 GB of sequence per partition. In the worst case (all unitigs of length 1 kmer, offsets spaced k apart), this supports 4 billion / k ≈ 130 million kmers per partition at k=31. In the typical case (long unitigs, consecutive kmers at offset +1), the limit approaches 4 billion kmers — well beyond any realistic partition size.

*Presence mode* (coverage ≤ 1x, or when only presence/absence matters):

- `counts.bin` is a packed 1-bit array — all bits set to 1 for indexed kmers
- Singletons are the signal, not filtered

*Count mode* (coverage > 1x):

- `counts.bin` is a packed n-bit array; n chosen at construction from the observed distribution
- Value 0: absent sentinel; values 1..2ⁿ−2: direct count; value 2ⁿ−1: overflow
- Overflow counts stored in a separate `overflow.bin` as sorted `(index: u32, count: u32)` pairs
- Empirically (k=31, 15x coverage): n=5 covers 97% of real kmers, n=6 covers 99%
- min_count threshold filters low-frequency kmers (errors) before indexing; for ≤1x, min_count=0

## Query protocol

```
query kmer q
  → canonical_minimizer(q) → hash → PART → part_XXXX/
  → MPHF(q) → index i
  → refs[i] = (unitig_id, kmer_offset)
  → read unitig from unitigs.bin → extract kmer at kmer_offset → compare with q
  → match   : return counts[i]
  → no match: kmer absent
```