Implement batch lookup and chunked data management
Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline. This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
This commit is contained in:
1 parent
8b87078478
commit
fe38424682
10 files changed
+520
-30
No files matched your search
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Usage: summarize_bench.py LABEL [LABEL...] — median wall per step, side by side."""
|
||||
import csv, statistics, sys, os
|
||||
base = os.path.join(os.path.dirname(os.path.abspath(__file__)), "run", "bench_mphf")
|
||||
data = {}
|
||||
for lab in sys.argv[1:]:
|
||||
for r in csv.DictReader(open(f"{base}/{lab}/results.csv")):
|
||||
data.setdefault(r["step"], {}).setdefault(lab, []).append(float(r["wall_s"]))
|
||||
print(f"{'step':32}" + "".join(f"{l:>14}" for l in sys.argv[1:]) + (" ratio" if len(sys.argv) > 2 else ""))
|
||||
for step, d in data.items():
|
||||
med = [statistics.median(d[l]) for l in sys.argv[1:]]
|
||||
line = f"{step:32}" + "".join(f"{m:14.2f}" for m in med)
|
||||
if len(med) > 1 and med[0]: line += f" {med[-1]/med[0]:.3f}"
|
||||
print(line)
|
||||
Reference in new issue
Block a user