obikmer/benchmark/README.md

# Benchmark pipeline

Requires **GNU Make ≥ 4.3** (grouped targets `&:`).  On macOS use `gmake`.

```
gmake all          # full pipeline
gmake simulate     # simulation only
gmake reference    # reference kmer sets only
```

## Pipeline overview

```mermaid
flowchart TD
    GENOMES["genomes/*.fna.gz"]
    BIN["obikmer binary"]

    GENOMES --> simulate
    simulate --> simdata[("simulated_data/")]

    simdata --> reference
    reference --> refnpz[("reference_index/*.npz")]

    subgraph presence ["Presence track"]
        simdata  --> index_presence
        BIN      --> index_presence
        index_presence --> pres_done[("specimen_index_presence/")]
        index_presence --> pres_istats[("stats/indexing_presence/")]
        pres_istats --> aggregate_index_presence

        pres_done --> merge_presence
        BIN       --> merge_presence
        merge_presence --> gpres[("global_index_presence/")]

        refnpz    --> verify_presence
        pres_done --> verify_presence
        verify_presence --> vpres_stats[("stats/verify_presence/")]
        vpres_stats --> aggregate_verify_presence

        gpres --> filter_presence
        BIN   --> filter_presence
        filter_presence --> spec_pres[("specific_index_presence/")]
        filter_presence --> spec_pres_stats[("stats/specific_kmer_presence/")]
        spec_pres_stats --> aggregate_filter_presence

        refnpz --> verify_merge_presence
        gpres  --> verify_merge_presence
        verify_merge_presence --> vmp[("stats/verify_merge_presence/")]
    end

    subgraph count ["Count track"]
        simdata --> index_count
        BIN     --> index_count
        index_count --> count_done[("specimen_index_count/")]
        index_count --> count_istats[("stats/indexing_count/")]
        count_istats --> aggregate_index_count

        count_done --> merge_count
        BIN        --> merge_count
        merge_count --> gcount[("global_index_count/")]

        refnpz     --> verify_count
        count_done --> verify_count
        verify_count --> vcount_stats[("stats/verify_count/")]
        vcount_stats --> aggregate_verify_count

        gcount --> filter_count
        BIN    --> filter_count
        filter_count --> spec_count[("specific_index_count/")]
        filter_count --> spec_count_stats[("stats/specific_kmer_count/")]
        spec_count_stats --> aggregate_filter_count

        refnpz --> verify_merge_count
        gcount --> verify_merge_count
        verify_merge_count --> vmc[("stats/verify_merge_count/")]
    end

    subgraph query ["Query track (2 specimens: E. coli + archaeon)"]
        GENOMES --> simulate_query
        simulate_query --> qdata[("query_data/")]

        gpres --> pack_sparse
        BIN   --> pack_sparse
        pack_sparse --> gsparse[("global_index_presence_sparse/")]

        qdata  --> query_dense
        gpres  --> query_dense
        BIN    --> query_dense
        query_dense --> qd[("query_dense/")]
        query_dense --> qd_stats[("stats/query_dense/")]
        qd_stats --> aggregate_query_dense

        qdata   --> query_sparse
        gsparse --> query_sparse
        BIN     --> query_sparse
        query_sparse --> qs[("query_sparse/")]
        query_sparse --> qs_stats[("stats/query_sparse/")]
        qs_stats --> aggregate_query_sparse

        qd --> verify_query
        qs --> verify_query
        verify_query --> vq_stats[("stats/verify_query/")]
        vq_stats --> aggregate_verify_query
    end

    aggregate_verify_presence  --> all
    aggregate_verify_count     --> all
    vmp                        --> all
    vmc                        --> all
    aggregate_query_dense      --> all
    aggregate_query_sparse     --> all
    aggregate_verify_query     --> all
    all -. "$(MAKE) re-eval" .-> aggregate_filter_presence
    all -. "$(MAKE) re-eval" .-> aggregate_filter_count
```

## Steps

| Target | Script | Description |
|---|---|---|
| `simulate` | `simulate.sh` | Simulate sequencing reads from the reference genomes |
| `reference` | `build_reference.sh` | Build reference kmer sets (`.npz`) from simulation truth |
| `index_presence` | `index_one_presence.sh` | Index each specimen (presence mode) |
| `index_count` | `index_one_count.sh` | Index each specimen (count mode) |
| `aggregate_index_presence` | `aggregate_stats.sh` | Aggregate per-specimen indexing stats (presence) |
| `aggregate_index_count` | `aggregate_stats.sh` | Aggregate per-specimen indexing stats (count) |
| `merge_presence` | `merge_presence.sh` | Merge all specimen presence indexes into a global index |
| `merge_count` | `merge_count.sh` | Merge all specimen count indexes into a global index |
| `verify_presence` | `verify_one_presence.sh` | Verify each specimen presence index against reference |
| `verify_count` | `verify_one_count.sh` | Verify each specimen count index against reference |
| `aggregate_verify_presence` | `aggregate_stats.sh` | Aggregate per-specimen verification stats (presence) |
| `aggregate_verify_count` | `aggregate_stats.sh` | Aggregate per-specimen verification stats (count) |
| `filter_presence` | `filter_one_presence.sh` | Extract species-specific presence indexes from global index |
| `filter_count` | `filter_one_count.sh` | Extract species-specific count indexes from global index |
| `aggregate_filter_presence` | `aggregate_stats.sh` | Aggregate species-specific kmer stats (presence) |
| `aggregate_filter_count` | `aggregate_stats.sh` | Aggregate species-specific kmer stats (count) |
| `verify_merge_presence` | `verify_merge_presence.sh` | Verify global presence index against all reference sets |
| `verify_merge_count` | `verify_merge_count.sh` | Verify global count index against all reference sets |
| `simulate_query` | `simulate_query_one.sh` | Simulate a fixed-size (100k pairs) read set per query specimen |
| `pack_sparse` | `pack_sparse.sh` | Build `global_index_presence_sparse/` from `global_index_presence/` |
| `query_dense` | `query_one.sh dense` | Query each query specimen's reads against the dense global index |
| `query_sparse` | `query_one.sh sparse` | Query each query specimen's reads against the sparse global index |
| `aggregate_query_dense` | `aggregate_stats.sh` | Aggregate dense query wall/RSS stats |
| `aggregate_query_sparse` | `aggregate_stats.sh` | Aggregate sparse query wall/RSS stats |
| `verify_query` | `verify_query_one.sh` | Diff dense vs sparse query output per specimen (regression check) |
| `aggregate_verify_query` | `aggregate_stats.sh` | Aggregate dense/sparse query regression stats |

## Directory layout

```
benchmark/
├── genomes/                        # input reference genomes (.fna.gz)
├── simulated_data/                 # generated by simulate
│   └── <species>/<specimen>/
├── query_data/                     # generated by simulate_query (2 specimens, fixed 100k pairs)
│   └── <species>/<specimen>/
├── reference_index/                # reference kmer sets (.npz)
├── specimen_index_presence/        # per-specimen presence indexes
├── specimen_index_count/           # per-specimen count indexes
├── global_index_presence/          # merged global presence index (dense-packed)
├── global_index_presence_sparse/   # global presence index, sparse-packed (query benchmark)
├── global_index_count/             # merged global count index
├── specific_index_presence/        # species-specific presence indexes
├── specific_index_count/           # species-specific count indexes
├── query_dense/                    # query output against global_index_presence
├── query_sparse/                   # query output against global_index_presence_sparse
└── stats/                          # all benchmark statistics
    ├── indexing_presence/
    ├── indexing_count/
    ├── verify_presence/
    ├── verify_count/
    ├── specific_kmer_presence/
    ├── specific_kmer_count/
    ├── verify_merge_presence/
    ├── verify_merge_count/
    ├── pack_sparse/
    ├── query_dense/
    ├── query_sparse/
    └── verify_query/
```