Implement batch lookup and chunked data management

Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline.

This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
This commit is contained in:
Eric Coissac committed 2026-10-15 12:14:51 +02:00
1 parent 8b87078478
commit fe38424682
10 files changed
+520 -30

No files matched your search

+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env python3
"""Usage: summarize_bench.py LABEL [LABEL...] — median wall per step, side by side."""
import csv, statistics, sys, os
base = os.path.join(os.path.dirname(os.path.abspath(__file__)), "run", "bench_mphf")
data = {}
for lab in sys.argv[1:]:
for r in csv.DictReader(open(f"{base}/{lab}/results.csv")):
data.setdefault(r["step"], {}).setdefault(lab, []).append(float(r["wall_s"]))
print(f"{'step':32}" + "".join(f"{l:>14}" for l in sys.argv[1:]) + (" ratio" if len(sys.argv) > 2 else ""))
for step, d in data.items():
med = [statistics.median(d[l]) for l in sys.argv[1:]]
line = f"{step:32}" + "".join(f"{m:14.2f}" for m in med)
if len(med) > 1 and med[0]: line += f" {med[-1]/med[0]:.3f}"
print(line)