docs: document partition runner deadlock fix and select CLI updates

Update documentation to reflect architectural fixes in the NUMA partition runner and implementation changes in the merge and select modules. The partition runner now wraps task execution in `catch_unwind`, emitting panic events that count toward completion thresholds to prevent controller deadlocks, while re-raising the first payload on return. CLI documentation for select has been updated to mandate output directories, introduce default hard-linking with automatic fallback, and document presence aggregation optimizations. Known issues such as self-merge crashes and sparse-packed index gaps are now documented as failing fast due to these improvements.

DevDoc/architecture/numa_partition_runner/index.html

@@ -1367,6 +1367,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1553,6 +1564,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1836,11 +1858,27 @@ dormant workers have been woken and grows every node by the same amount per
 step, capped by that node's remaining dormant workers and by the run's total
 budget (<code>n_total</code>) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.</p>
+<h2 id="panic-propagation-fixed-2026-08-28">Panic propagation (fixed, 2026-08-28)</h2>
+<p>A worker whose <code>f(i)</code> panicked used to never send its <code>WorkerEvent::Completed</code>
+— the controller's <code>while completed &lt; n_total</code> loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept <code>event_rx</code> open. Discovered via <code>obikselect</code>
+panicking on an unimplemented <code>Sparse</code> matrix case (see <code>implementation/select.md</code>):
+the process hung indefinitely instead of erroring out.</p>
+<p>Fixed: each worker wraps <code>f(i)</code> in <code>std::panic::catch_unwind</code> and sends a new
+<code>WorkerEvent::Panicked(i, payload)</code> instead of silently dropping the
+partition. The controller counts it toward <code>completed</code> (unblocking the loop)
+and keeps the first payload seen; once <code>run</code> returns, <code>panic::resume_unwind</code>
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain <code>Err</code> (a panic means a bug, not a normal typed failure).</p>
 <h2 id="open-questions">Open questions</h2>
 <ul>
 <li>
 <p><strong>Error handling</strong>: <code>run</code> currently returns the first error; remaining errors
-  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics.</p>
+  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain <code>Err</code> results.)</p>
 </li>
 <li>
 <p><strong><code>INITIAL_DIVISOR</code> / <code>GROWTH_DIVISOR</code> tuning</strong>: currently <code>4</code> and <code>8</code>

DevDoc/implementation/merge/index.html

@@ -1034,6 +1034,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1598,6 +1609,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1795,6 +1817,14 @@ spectrums/
   &lt;label&gt;.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 </code></pre></div>
+<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
+<h2 id="known-issue-not-yet-fixed-2026-08-28">Known issue (not yet fixed, 2026-08-28)</h2>
+<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank &lt; self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
+<ul>
+<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
+<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
+</ul>
+<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
 
 
 

DevDoc/implementation/select/index.html

@@ -1119,17 +1119,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1157,6 +1146,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1638,17 +1644,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1676,6 +1671,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1728,23 +1740,32 @@ are preserved unchanged; only the data matrices are rewritten.</p>
 <hr />
 <h2 id="synopsis">Synopsis</h2>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>&lt;input-index&gt;
-<span class="w">        </span><span class="o">{</span><span class="w"> </span>--output<span class="w"> </span>&lt;dir&gt;<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="w"> </span><span class="o">}</span>
+<span class="w">        </span>--output<span class="w"> </span>&lt;dir&gt;
 <span class="w">        </span><span class="o">[</span>--group<span class="w">    </span>&lt;name&gt;:&lt;pred&gt;<span class="w">  </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--group-op<span class="w"> </span>&lt;name&gt;:&lt;op&gt;<span class="w">    </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-by<span class="w"> </span>&lt;key&gt;<span class="w">          </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-op<span class="w"> </span>&lt;op&gt;<span class="w">           </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--select<span class="w">   </span>&lt;col1,col2,...&gt;<span class="w">    </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--presence-threshold<span class="w"> </span>&lt;N&gt;<span class="w">      </span><span class="o">]</span>
+<span class="w">        </span><span class="o">[</span>--force-copy<span class="w">                  </span><span class="o">]</span>
 </code></pre></div>
 <hr />
 <h2 id="output-destination">Output destination</h2>
-<p>Exactly one of <code>--output</code> or <code>--in-place</code> must be specified.</p>
+<p><code>--output &lt;dir&gt;</code> is required — <code>select</code> always writes a new index; there is no
-<p><strong><code>--output &lt;dir&gt;</code></strong> — writes a new index to <code>&lt;dir&gt;</code>. The source index is
+<code>--in-place</code> mode (2026-08-28: never implemented, removed from the design). The
-unchanged. The MPHF and unitig files are copied; only the data matrices are
+source index is unchanged.</p>
-rewritten with the new column layout.</p>
+<p>Each layer's kmer-identity files (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/
-<p><strong><code>--in-place</code></strong> — rewrites the data matrices of the source index directly.
+<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code>) are never rewritten by a
-Removed or replaced columns are lost. The operation writes to temporary files
+column projection/aggregation, so they are hard-linked into the output rather
-first, then renames atomically, so an interrupted run leaves the index intact.</p>
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); <code>--force-copy</code>
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+<code>select</code> itself would affect the other). Only the <code>presence</code>/<code>counts</code>
+subdirectory is ever a genuinely new, independent file.</p>
+<p>To replace an index with a selected version of itself, select to a temporary
+directory and swap it in (<code>rm -rf INDEX &amp;&amp; mv INDEX.tmp INDEX</code>) — the case
+<code>--in-place</code> used to cover.</p>
 <hr />
 <h2 id="defining-output-columns">Defining output columns</h2>
 <h3 id="named-groups-group">Named groups — <code>--group</code></h3>
@@ -1908,9 +1929,6 @@ is &gt; N.</p>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>out<span class="w"> </span><span class="se">\</span>
 <span class="w">  </span>--select<span class="w"> </span><span class="s2">&quot;Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01&quot;</span>
 </code></pre></div>
-<h3 id="in-place-keep-only-group-a">In-place: keep only group A</h3>
-<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--in-place<span class="w"> </span>--group<span class="w"> </span><span class="s2">&quot;A:group=A&quot;</span><span class="w"> </span>--select<span class="w"> </span><span class="s2">&quot;A&quot;</span>
-</code></pre></div>
 <h3 id="compose-with-filter">Compose with filter</h3>
 <div class="highlight"><pre><span></span><code><span class="c1"># Step 1: keep only B. nana-specific k-mers</span>
 obikmer<span class="w"> </span>filter<span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>filtered<span class="w"> </span><span class="se">\</span>
@@ -1921,32 +1939,46 @@ obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </s
 </code></pre></div>
 <hr />
 <h2 id="implementation-notes">Implementation notes</h2>
-<p><code>select</code> does not rebuild the MPHF. The 256 partitions are processed in parallel
+<p><code>select</code> does not rebuild the MPHF. Every partition is processed independently
-(rayon), each writing its output independently; results require no synchronisation
+(<code>PartitionRunner</code>), each writing its own output layers; no cross-partition
-because every partition owns a distinct set of files.</p>
+synchronisation is needed.</p>
-<p>For each layer in each partition:</p>
+<p>For each layer in each partition (<code>obikselect::select_layer::select_partition</code>):</p>
 <ol>
-<li>The slot count <code>n</code> is read by opening the source data matrix.</li>
+<li><code>copy_layer_files</code> hard-links the source layer's kmer-identity files
-<li>A new data matrix is built with M columns (M = number of output columns).</li>
+   (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/
-<li>For each slot <code>s</code> in <code>0..n</code>:</li>
+   <code>fingerprint.bin</code>/<code>layer_meta.json</code>) into the destination — never a real
-<li><code>old_row = matrix.fill_row(s)</code> — reads the original <code>N</code>-column row without allocating.</li>
+   copy unless linking fails or <code>--force-copy</code> is given.</li>
-<li>For each output column <code>j</code>:<ul>
+<li>A new data matrix is built with M columns (M = number of output columns),
-<li><code>new_row[j] = aggregate(op, old_row[group_indices])</code>.</li>
+   under a fresh <code>presence/</code>/<code>counts/</code> subdirectory (never touching the
-<li>Pass-through columns are represented as single-element groups with the
+   source's own).</li>
-   default operator (<code>any</code> for presence, <code>sum</code> for count) — same code path.</li>
+<li><strong>Presence source (2026-08-28: <code>batch_presence_counts</code>)</strong>: one shared pass
-</ul>
+   over the source bit matrix computes every output group's presence count at
-</li>
+   once — row-major native for a <code>Sparse</code> source (<code>for_each_genome_in_row</code>,
-<li>The new row is written slot by slot into each column builder.</li>
+   which has no column representation to read a <code>col_view</code> from at all — the
-<li>All plain files in the source layer directory (<code>mphf.bin</code>, <code>unitigs.bin</code>,
+   reason this replaced the old per-group loop, not just an optimisation of
-   evidence files, <code>layer_meta.json</code>) are copied verbatim; only the <code>presence/</code>
+   it), deduplicated column-major (one <code>col_view</code> per <em>distinct referenced
-   or <code>counts/</code> subdirectory is rewritten.</li>
+   column</em>, not per group) for <code>Columnar</code>/<code>Packed</code>. Every <code>AggOp</code> for a bit
+   matrix is then a cheap derivation of that one count vector (<code>sum</code> = the
+   count itself, <code>any</code>/<code>max</code> = <code>count ≥ 1</code>, <code>all</code>/<code>min</code> = <code>count == group
+   size</code>, <code>none</code> = <code>count == 0</code>) — see
+   <code>obikselect::select_layer::agg_result_from_count</code>.</li>
+<li><strong>Count source</strong>: unchanged, one <code>col_view</code>-driven pass per output column
+   via <code>MatrixGroupOps</code> — <code>sum</code>/<code>min</code>/<code>max</code> are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.</li>
 <li><code>index.meta</code> is rewritten with the new genome list and updated <code>with_counts</code>.</li>
 </ol>
-<p><strong><code>--in-place</code> write strategy:</strong> new data is written to a temporary sibling
+<h3 id="known-gap-not-yet-fixed-2026-08-28">Known gap (not yet fixed, 2026-08-28)</h3>
-directory (<code>presence_new/</code> or <code>counts_new/</code>); on success the old directory is
+<p>Step 4 above still panics (<code>col_view() not available on Sparse
-removed and the temporary one is renamed into place. An interrupted run leaves
+PersistentCompactIntMatrix</code>) if the source is a <strong>count</strong> index packed
-at most one stale <code>*_new/</code> directory; the original data is intact until the
+sparse — <code>batch_presence_counts</code>' row-major treatment was only ported to the
-rename step.</p>
+bit-matrix (<code>Presence</code>) case, since that was the one actually blocking a real
+benchmark run. <code>select</code>/<code>filter</code> on a sparse-packed count index still hits
+this; the fix would follow the same shape (a <code>PersistentSparseCompactIntMatrix</code>
+row-major decode, analogous to <code>for_each_genome_in_row</code>), just not done. Since
+<code>obisys::numa::runner::PartitionRunner</code>'s panic-propagation fix (see
+<code>architecture/numa_partition_runner.md</code>), this at least fails fast (process
+panic, exit 101) instead of hanging.</p>
 
 
 

DevDocMD/architecture/numa_partition_runner.md

@@ -304,10 +304,29 @@ step, capped by that node's remaining dormant workers and by the run's total
 budget (`n_total`) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.
 
+## Panic propagation (fixed, 2026-08-28)
+
+A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
+— the controller's `while completed < n_total` loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept `event_rx` open. Discovered via `obikselect`
+panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
+the process hung indefinitely instead of erroring out.
+
+Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
+`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
+partition. The controller counts it toward `completed` (unblocking the loop)
+and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain `Err` (a panic means a bug, not a normal typed failure).
+
 ## Open questions
 
 - **Error handling**: `run` currently returns the first error; remaining errors
-  are dropped.  A `Vec<E>` return would give complete diagnostics.
+  are dropped.  A `Vec<E>` return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain `Err` results.)
 
 - **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
   (start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an

DevDocMD/implementation/merge.md

@@ -194,3 +194,14 @@ spectrums/
   <label>.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 ```
+
+`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json` marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — `merge_partition` never rewrites them for pre-existing layers, only the `presence`/`counts` subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
+
+## Known issue (not yet fixed, 2026-08-28)
+
+Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. `obikmer merge -o out IDX IDX --rename-duplicates`) — panics deep in the MPHF's rank-select structure (`common_traits::select_in_word`, `assertion failed: rank < self.count_ones()`), inside `MphfLayer::find` called from `merge_partition`'s "is this source kmer already in dst" check against the bootstrap-copied `dst_layers`. Root cause not identified; ruled out so far:
+
+- **Not an empty-new-layer issue**: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
+- **Not caused by the hard-link change above**: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
+
+Only reproduces when `sources[0]` and `sources[1]` are the exact same on-disk path opened as two independent `KmerIndex` handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since `PartitionRunner`'s panic propagation fix (see `architecture/numa_partition_runner.md`) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.

DevDocMD/implementation/select.md

@@ -23,28 +23,37 @@ are preserved unchanged; only the data matrices are rewritten.
 
 ```sh
 obikmer select <input-index>
-        { --output <dir> | --in-place }
+        --output <dir>
         [--group    <name>:<pred>  ...]
         [--group-op <name>:<op>    ...]
         [--aggregate-by <key>          ]
         [--aggregate-op <op>           ]
         [--select   <col1,col2,...>    ]
         [--presence-threshold <N>      ]
+        [--force-copy                  ]
 ```
 
 ---
 
 ## Output destination
 
-Exactly one of `--output` or `--in-place` must be specified.
+`--output <dir>` is required — `select` always writes a new index; there is no
+`--in-place` mode (2026-08-28: never implemented, removed from the design). The
+source index is unchanged.
 
-**`--output <dir>`** — writes a new index to `<dir>`. The source index is
+Each layer's kmer-identity files (`mphf.bin`/`unitigs.bin`/`evidence.bin`/
-unchanged. The MPHF and unitig files are copied; only the data matrices are
+`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json`) are never rewritten by a
-rewritten with the new column layout.
+column projection/aggregation, so they are hard-linked into the output rather
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); `--force-copy`
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+`select` itself would affect the other). Only the `presence`/`counts`
+subdirectory is ever a genuinely new, independent file.
 
-**`--in-place`** — rewrites the data matrices of the source index directly.
+To replace an index with a selected version of itself, select to a temporary
-Removed or replaced columns are lost. The operation writes to temporary files
+directory and swap it in (`rm -rf INDEX && mv INDEX.tmp INDEX`) — the case
-first, then renames atomically, so an interrupted run leaves the index intact.
+`--in-place` used to cover.
 
 ---
 
@@ -186,12 +195,6 @@ obikmer select myindex --output out \
   --select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
 ```
 
-### In-place: keep only group A
-
-```sh
-obikmer select myindex --in-place --group "A:group=A" --select "A"
-```
-
 ### Compose with filter
 
 ```sh
@@ -207,28 +210,45 @@ obikmer select filtered --output final --aggregate-by site
 
 ## Implementation notes
 
-`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
+`select` does not rebuild the MPHF. Every partition is processed independently
-(rayon), each writing its output independently; results require no synchronisation
+(`PartitionRunner`), each writing its own output layers; no cross-partition
-because every partition owns a distinct set of files.
+synchronisation is needed.
 
-For each layer in each partition:
+For each layer in each partition (`obikselect::select_layer::select_partition`):
 
-1. The slot count `n` is read by opening the source data matrix.
+1. `copy_layer_files` hard-links the source layer's kmer-identity files
-2. A new data matrix is built with M columns (M = number of output columns).
+   (`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/
-3. For each slot `s` in `0..n`:
+   `fingerprint.bin`/`layer_meta.json`) into the destination — never a real
-   - `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
+   copy unless linking fails or `--force-copy` is given.
-   - For each output column `j`:
+2. A new data matrix is built with M columns (M = number of output columns),
-     - `new_row[j] = aggregate(op, old_row[group_indices])`.
+   under a fresh `presence/`/`counts/` subdirectory (never touching the
-     - Pass-through columns are represented as single-element groups with the
+   source's own).
-       default operator (`any` for presence, `sum` for count) — same code path.
+3. **Presence source (2026-08-28: `batch_presence_counts`)**: one shared pass
-   - The new row is written slot by slot into each column builder.
+   over the source bit matrix computes every output group's presence count at
-4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
+   once — row-major native for a `Sparse` source (`for_each_genome_in_row`,
-   evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
+   which has no column representation to read a `col_view` from at all — the
-   or `counts/` subdirectory is rewritten.
+   reason this replaced the old per-group loop, not just an optimisation of
+   it), deduplicated column-major (one `col_view` per *distinct referenced
+   column*, not per group) for `Columnar`/`Packed`. Every `AggOp` for a bit
+   matrix is then a cheap derivation of that one count vector (`sum` = the
+   count itself, `any`/`max` = `count ≥ 1`, `all`/`min` = `count == group
+   size`, `none` = `count == 0`) — see
+   `obikselect::select_layer::agg_result_from_count`.
+4. **Count source**: unchanged, one `col_view`-driven pass per output column
+   via `MatrixGroupOps` — `sum`/`min`/`max` are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.
 5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
 
-**`--in-place` write strategy:** new data is written to a temporary sibling
+### Known gap (not yet fixed, 2026-08-28)
-directory (`presence_new/` or `counts_new/`); on success the old directory is
+
-removed and the temporary one is renamed into place. An interrupted run leaves
+Step 4 above still panics (`col_view() not available on Sparse
-at most one stale `*_new/` directory; the original data is intact until the
+PersistentCompactIntMatrix`) if the source is a **count** index packed
-rename step.
+sparse — `batch_presence_counts`' row-major treatment was only ported to the
+bit-matrix (`Presence`) case, since that was the one actually blocking a real
+benchmark run. `select`/`filter` on a sparse-packed count index still hits
+this; the fix would follow the same shape (a `PersistentSparseCompactIntMatrix`
+row-major decode, analogous to `for_each_genome_in_row`), just not done. Since
+`obisys::numa::runner::PartitionRunner`'s panic-propagation fix (see
+`architecture/numa_partition_runner.md`), this at least fails fast (process
+panic, exit 101) instead of hanging.