feat: add pack optimization prompt, debug infrastructure, and monitoring scripts

Add a prompt template for optimizing the `pack` step and a suite of Docker-based
debug tools including build, run, and monitoring scripts. Enhance `pack_matrices`
with LFD partition sizing and detailed timing reports, and update `stop` with
additional system statistics.

- Add `prompts/pack_optimization.md` to guide optimization of the `pack` step,
  specifying context, performance measurements, and deliverable structures.
- Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets,
  `build.sh` for image creation, `run.sh` for parallel indexing and merging,
  `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and
  `gen_fasta_real.py` for real genome generation with `read_fna`.
- Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics,
  writing samples to CSV and summary reports.
- Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing
  memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize.
- Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and
  `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition
  sizing and clarify documentation regarding `stat()` proxy bytes.
- Modify `src/obisys/src/stage.rs`: update `stop` definition to include new
  variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and
  logic checking for swap usage.
This commit is contained in:
Eric Coissac committed 2026-11-19 07:07:50 +01:00
1 parent 24291a4484
commit 03eaa4544a
11 files changed
+497 -2

No files matched your search

+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""gen_fasta_real.py N OUTDIR GENOMES_DIR [WINDOW] [SEED] [MUT]
N derived genomes built from *real* genomes (GENOMES_DIR/*.fna.gz): each one is
a random window of WINDOW bp of a randomly chosen real genome, point-mutated at
rate MUT (default 3 %, which alters ~47 % of 21-mers), so a merge both finds
k-mers already present in the destination and creates new ones, with realistic
sequence statistics (no dedicated degenerate structure).
"""
import glob, gzip, random, sys
n, out, gdir = int(sys.argv[1]), sys.argv[2], sys.argv[3]
window = int(sys.argv[4]) if len(sys.argv) > 4 else 300_000
seed = int(sys.argv[5]) if len(sys.argv) > 5 else 20261118
mut = float(sys.argv[6]) if len(sys.argv) > 6 else 0.03
rng = random.Random(seed)
def read_fna(path):
seq = []
with gzip.open(path, "rt") as fh:
for line in fh:
if not line.startswith(">"):
seq.append(line.strip().upper())
return "".join(seq)
genomes = [read_fna(p) for p in sorted(glob.glob(f"{gdir}/*.fna.gz"))]
genomes = [g for g in genomes if len(g) > window]
for i in range(n):
g = rng.choice(genomes)
start = rng.randrange(0, len(g) - window)
s = list(g[start:start + window])
for j in range(len(s)):
if s[j] in "ACGT" and rng.random() < mut:
s[j] = rng.choice("ACGT")
seq = "".join(c if c in "ACGT" else "A" for c in s)
with open(f"{out}/R{i:03d}.fasta", "w") as fh:
fh.write(f">R{i:03d}\n")
for k in range(0, len(seq), 80):
fh.write(seq[k:k + 80] + "\n")