feat: add pack optimization prompt, debug infrastructure, and monitoring scripts

Add a prompt template for optimizing the `pack` step and a suite of Docker-based
debug tools including build, run, and monitoring scripts. Enhance `pack_matrices`
with LFD partition sizing and detailed timing reports, and update `stop` with
additional system statistics.

- Add `prompts/pack_optimization.md` to guide optimization of the `pack` step,
  specifying context, performance measurements, and deliverable structures.
- Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets,
  `build.sh` for image creation, `run.sh` for parallel indexing and merging,
  `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and
  `gen_fasta_real.py` for real genome generation with `read_fna`.
- Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics,
  writing samples to CSV and summary reports.
- Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing
  memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize.
- Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and
  `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition
  sizing and clarify documentation regarding `stat()` proxy bytes.
- Modify `src/obisys/src/stage.rs`: update `stop` definition to include new
  variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and
  logic checking for swap usage.
This commit is contained in:
Eric Coissac committed 2026-11-19 07:07:50 +01:00
1 parent 24291a4484
commit 03eaa4544a
11 files changed
+497 -2

No files matched your search

+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# monitor.sh PID OUT [INTERVAL_S] — samples mappings / VM / commit figures until PID exits.
PID=$1; OUT=$2; DT=${3:-5}
printf 'time,maps,vmsize_kb,vmdata_kb,vmrss_kb,threads,committed_as_kb,commit_limit_kb\n' >"$OUT"
while kill -0 "$PID" 2>/dev/null; do
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null) || break
st=$(grep -E '^(VmSize|VmData|VmRSS|Threads):' "/proc/$PID/status" | awk '{printf "%s,", $2}')
mi=$(grep -E '^(Committed_AS|CommitLimit):' /proc/meminfo | awk '{printf "%s,", $2}')
echo "$(date +%T),$maps,${st}${mi%,}" >>"$OUT"
sleep "$DT"
done