feat(numa): introduce I/O sampling to prevent activation stalls
Replaces the monolithic CPU scaling threshold with separate CPU and I/O spawn thresholds. Introduces an `IoSample` struct with platform-specific byte reading and a relative throughput growth heuristic. Adds a 0.1s wall-clock guard to `CpuSample` to suppress artificial efficiency spikes, and updates `maybe_activate` to trigger worker scaling when either resource indicates headroom. Bumps `obikmer` to v1.1.33 and updates architecture documentation.
This commit is contained in:
@@ -162,14 +162,107 @@ A single `PartitionRunner` instance can be built once per command invocation
|
||||
and reused across multiple `run()` calls (e.g. `merge` runs
|
||||
`merge_partitions` then `pack_matrices`).
|
||||
|
||||
## Known issue: CPU-only activation signal stalls on I/O-bound stages
|
||||
|
||||
Observed on a real `filter` run (109 genomes, 256 partitions, 8×24-core NUMA):
|
||||
`rebuild` (CPU-bound — k-mer construction) scales cleanly from 9 to 43 active
|
||||
workers as `CpuSample::do_i_activate` (`obisys::lib.rs`) sees efficiency climb.
|
||||
`pack_matrices` (I/O-bound — reopens and recomposes per-genome column files
|
||||
into `.pbmx`/`.pcmx`) activates one extra worker then flatlines at 10/192 for
|
||||
the rest of the stage, even though 256 partitions keep completing over several
|
||||
minutes. This matches the documented intent (§ Adaptive mechanism — "avoids
|
||||
over-provisioning ... I/O-bound ... workloads") but conflates two different
|
||||
things: *"CPU is not the bottleneck"* and *"more workers would not help"*. On
|
||||
storage with real queue depth (NVMe, RAID, parallel FS) the second stage could
|
||||
still benefit from more concurrent workers even with flat CPU usage — a signal
|
||||
the current mechanism cannot see.
|
||||
|
||||
A one-off artefact was also found in the same log: right after a stage
|
||||
transition, `do_i_activate` produced a physically impossible spike (efficiency
|
||||
~94 cores on a 192-core box) because it has no minimum-window guard — unlike
|
||||
its sibling `cpu_efficiency`, which returns `0.0` if `wall < 0.1s`
|
||||
(`obisys::lib.rs:260`). `do_i_activate` unconditionally overwrites
|
||||
`self.wall`/`self.user_secs`/`self.sys_secs` even when the elapsed window is
|
||||
too short to be meaningful, so a burst of rapid completions right after
|
||||
activating a worker can divide a real CPU delta by a near-zero wall delta.
|
||||
|
||||
### Implemented: I/O signal + shared debounce guard
|
||||
|
||||
`IoSample` (`obisys::lib.rs`, alongside `CpuSample`) is fed by
|
||||
`read_bytes`/`write_bytes` from `/proc/self/io` on Linux (actual bytes
|
||||
submitted to the block layer — not `rchar`/`wchar`, which also count
|
||||
page-cache hits, and not `ru_inblock`/`ru_oublock`, unreliable on macOS), with
|
||||
a `proc_pid_rusage(RUSAGE_INFO_V4)` fallback on macOS
|
||||
(`ri_diskio_bytesread`/`ri_diskio_byteswritten`, FFI only via `libc`, no new
|
||||
dependency — same pattern as the existing `getrusage` bindings). Any other
|
||||
target degrades gracefully to a signal that never triggers (falls back to
|
||||
CPU-only activation), same pattern as `cgroup_v2_available`.
|
||||
|
||||
`maybe_activate` (`numa.rs`) activates a worker if *either* signal still shows
|
||||
headroom, making `PartitionRunner` adapt to whichever resource is actually the
|
||||
bottleneck without per-call configuration. Both samplers are called
|
||||
unconditionally — no `||` short-circuit — so neither window starves behind
|
||||
whichever signal fires first:
|
||||
|
||||
```rust
|
||||
let cpu_wants_more = cpu_sample.do_i_activate(CPU_SPAWN_THRESHOLD);
|
||||
let io_wants_more = io_sample.do_i_activate(IO_SPAWN_THRESHOLD);
|
||||
if cpu_wants_more || io_wants_more {
|
||||
activate_tx.send(()).ok();
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Unlike the CPU signal (an absolute delta in cores — a bounded, portable unit),
|
||||
raw I/O throughput has no natural scale across devices, so `IoSample` uses a
|
||||
**relative** growth threshold instead of an absolute one:
|
||||
|
||||
```rust
|
||||
pub fn do_i_activate(&mut self, threshold: f64) -> bool {
|
||||
let elapsed = self.wall.elapsed().as_secs_f64();
|
||||
if elapsed < 0.1 { return false; } // state untouched — window keeps accumulating
|
||||
|
||||
let n = Self::read_bytes();
|
||||
let rate = n.saturating_sub(self.bytes) as f64 / elapsed;
|
||||
let activate = if self.previous_rate == 0.0 {
|
||||
rate > 0.0 // bootstrap: any measured throughput is signal
|
||||
} else {
|
||||
(rate - self.previous_rate) / self.previous_rate >= threshold
|
||||
};
|
||||
|
||||
self.bytes = n;
|
||||
self.wall = Instant::now(); // reset only on a real sample
|
||||
activate
|
||||
}
|
||||
```
|
||||
|
||||
The `elapsed < 0.1s → return false without mutating state` guard was also
|
||||
back-ported into `CpuSample::do_i_activate` (previously missing — source of
|
||||
the ~94-core artefact above) — one fix for both problems, and it removes the
|
||||
need for any arbitrary I/O-rate floor: a short/noisy window is rejected
|
||||
outright rather than papered over with a hardware-dependent constant.
|
||||
|
||||
Both spawn thresholds (`CPU_SPAWN_THRESHOLD`, `IO_SPAWN_THRESHOLD`, both `0.2`)
|
||||
are defined as `const` in `PartitionRunner::run` (`numa.rs`). The I/O value is
|
||||
a starting point, not a derived one — needs empirical validation against a
|
||||
real `pack` run.
|
||||
|
||||
Starting threshold: `0.2` (20 % relative growth) for `IoSample`, same order of
|
||||
magnitude as the CPU threshold's *implicit* relative sensitivity (in the
|
||||
observed log, an 8→9 worker step raised efficiency by ~12 %). This is a
|
||||
starting point, not a derived value — I/O throughput is lumpier than CPU time
|
||||
(buffered writes flush in bursts), so it needs empirical validation against a
|
||||
real `pack` run before being considered final.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Error handling**: `run` currently returns the first error; remaining errors
|
||||
are dropped. A `Vec<E>` return would give complete diagnostics.
|
||||
|
||||
- **`workers_per_node` tuning**: currently `(cpus / 8).max(3).min(8)`, calibrated
|
||||
for merge on BeeGFS. I/O-bound commands (`dump`, `select`) may benefit from
|
||||
a higher value. A per-call override could be added to the API.
|
||||
for merge on BeeGFS. Superseded by the I/O signal above for the "more
|
||||
workers would help despite flat CPU" case — a per-call override may still be
|
||||
worth keeping as a manual escape hatch.
|
||||
|
||||
- **`on_done` ordering**: the runner serialises calls to `on_done` via an
|
||||
internal `Arc<Mutex<C>>`. `Send` is required (the Arc clone crosses thread
|
||||
|
||||
Reference in New Issue
Block a user