Add a prompt template for optimizing the `pack` step and a suite of Docker-based debug tools including build, run, and monitoring scripts. Enhance `pack_matrices` with LFD partition sizing and detailed timing reports, and update `stop` with additional system statistics. - Add `prompts/pack_optimization.md` to guide optimization of the `pack` step, specifying context, performance measurements, and deliverable structures. - Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets, `build.sh` for image creation, `run.sh` for parallel indexing and merging, `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and `gen_fasta_real.py` for real genome generation with `read_fna`. - Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics, writing samples to CSV and summary reports. - Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize. - Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition sizing and clarify documentation regarding `stat()` proxy bytes. - Modify `src/obisys/src/stage.rs`: update `stop` definition to include new variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and logic checking for swap usage.
40 lines
1.5 KiB
Python
40 lines
1.5 KiB
Python
#!/usr/bin/env python3
|
|
"""gen_fasta_real.py N OUTDIR GENOMES_DIR [WINDOW] [SEED] [MUT]
|
|
|
|
N derived genomes built from *real* genomes (GENOMES_DIR/*.fna.gz): each one is
|
|
a random window of WINDOW bp of a randomly chosen real genome, point-mutated at
|
|
rate MUT (default 3 %, which alters ~47 % of 21-mers), so a merge both finds
|
|
k-mers already present in the destination and creates new ones, with realistic
|
|
sequence statistics (no dedicated degenerate structure).
|
|
"""
|
|
import glob, gzip, random, sys
|
|
|
|
n, out, gdir = int(sys.argv[1]), sys.argv[2], sys.argv[3]
|
|
window = int(sys.argv[4]) if len(sys.argv) > 4 else 300_000
|
|
seed = int(sys.argv[5]) if len(sys.argv) > 5 else 20261118
|
|
mut = float(sys.argv[6]) if len(sys.argv) > 6 else 0.03
|
|
rng = random.Random(seed)
|
|
|
|
def read_fna(path):
|
|
seq = []
|
|
with gzip.open(path, "rt") as fh:
|
|
for line in fh:
|
|
if not line.startswith(">"):
|
|
seq.append(line.strip().upper())
|
|
return "".join(seq)
|
|
|
|
genomes = [read_fna(p) for p in sorted(glob.glob(f"{gdir}/*.fna.gz"))]
|
|
genomes = [g for g in genomes if len(g) > window]
|
|
for i in range(n):
|
|
g = rng.choice(genomes)
|
|
start = rng.randrange(0, len(g) - window)
|
|
s = list(g[start:start + window])
|
|
for j in range(len(s)):
|
|
if s[j] in "ACGT" and rng.random() < mut:
|
|
s[j] = rng.choice("ACGT")
|
|
seq = "".join(c if c in "ACGT" else "A" for c in s)
|
|
with open(f"{out}/R{i:03d}.fasta", "w") as fh:
|
|
fh.write(f">R{i:03d}\n")
|
|
for k in range(0, len(seq), 80):
|
|
fh.write(seq[k:k + 80] + "\n")
|