feat: add pack optimization prompt, debug infrastructure, and monitoring scripts
Add a prompt template for optimizing the `pack` step and a suite of Docker-based debug tools including build, run, and monitoring scripts. Enhance `pack_matrices` with LFD partition sizing and detailed timing reports, and update `stop` with additional system statistics. - Add `prompts/pack_optimization.md` to guide optimization of the `pack` step, specifying context, performance measurements, and deliverable structures. - Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets, `build.sh` for image creation, `run.sh` for parallel indexing and merging, `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and `gen_fasta_real.py` for real genome generation with `read_fna`. - Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics, writing samples to CSV and summary reports. - Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize. - Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition sizing and clarify documentation regarding `stat()` proxy bytes. - Modify `src/obisys/src/stage.rs`: update `stop` definition to include new variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and logic checking for swap usage.
This commit is contained in:
1 parent
24291a4484
commit
03eaa4544a
11 files changed
+497
-2
No files matched your search
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gen_fasta.py N OUTDIR [GENOME_LEN] — N deterministic synthetic genomes.
|
||||
|
||||
Each genome = unique random sequence + a random subset of a shared fragment
|
||||
pool (some fragments lightly mutated), so a merge both hits k-mers already
|
||||
in the destination and creates new ones, like related genomes would.
|
||||
"""
|
||||
import random, sys
|
||||
|
||||
n, out = int(sys.argv[1]), sys.argv[2]
|
||||
glen = int(sys.argv[3]) if len(sys.argv) > 3 else 150_000
|
||||
seed = int(sys.argv[4]) if len(sys.argv) > 4 else 20261118
|
||||
rng = random.Random(seed)
|
||||
|
||||
def rnd(m):
|
||||
return "".join(rng.choices("ACGT", k=m))
|
||||
|
||||
def mutate(s, rate=0.01):
|
||||
s = list(s)
|
||||
for i in range(len(s)):
|
||||
if rng.random() < rate:
|
||||
s[i] = rng.choice("ACGT")
|
||||
return "".join(s)
|
||||
|
||||
pool = [rnd(2000) for _ in range(300)] # 600 kb shared pool
|
||||
for g in range(n):
|
||||
parts = [rnd(glen)]
|
||||
for f in rng.sample(range(len(pool)), 60):
|
||||
parts.append(mutate(pool[f]) if rng.random() < 0.3 else pool[f])
|
||||
rng.shuffle(parts)
|
||||
with open(f"{out}/G{g:03d}.fasta", "w") as fh:
|
||||
for j, p in enumerate(parts):
|
||||
fh.write(f">G{g:03d}_{j}\n{p}\n")
|
||||
Reference in new issue
Block a user