DevDoc/architecture/numa_partition_runner/index.html

@@ -1367,6 +1367,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1553,6 +1564,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#panic-propagation-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Panic propagation (fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -1836,11 +1858,27 @@ dormant workers have been woken and grows every node by the same amount per
 step, capped by that node's remaining dormant workers and by the run's total
 budget (<code>n_total</code>) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.</p>
+<h2 id="panic-propagation-fixed-2026-08-28">Panic propagation (fixed, 2026-08-28)</h2>
+<p>A worker whose <code>f(i)</code> panicked used to never send its <code>WorkerEvent::Completed</code>
+— the controller's <code>while completed &lt; n_total</code> loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept <code>event_rx</code> open. Discovered via <code>obikselect</code>
+panicking on an unimplemented <code>Sparse</code> matrix case (see <code>implementation/select.md</code>):
+the process hung indefinitely instead of erroring out.</p>
+<p>Fixed: each worker wraps <code>f(i)</code> in <code>std::panic::catch_unwind</code> and sends a new
+<code>WorkerEvent::Panicked(i, payload)</code> instead of silently dropping the
+partition. The controller counts it toward <code>completed</code> (unblocking the loop)
+and keeps the first payload seen; once <code>run</code> returns, <code>panic::resume_unwind</code>
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain <code>Err</code> (a panic means a bug, not a normal typed failure).</p>
 <h2 id="open-questions">Open questions</h2>
 <ul>
 <li>
 <p><strong>Error handling</strong>: <code>run</code> currently returns the first error; remaining errors
-  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics.</p>
+  are dropped.  A <code>Vec&lt;E&gt;</code> return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain <code>Err</code> results.)</p>
 </li>
 <li>
 <p><strong><code>INITIAL_DIVISOR</code> / <code>GROWTH_DIVISOR</code> tuning</strong>: currently <code>4</code> and <code>8</code>

DevDoc/implementation/merge/index.html

@@ -1034,6 +1034,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1598,6 +1609,17 @@
     </span>
   </a>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#known-issue-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known issue (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
 </li>
       
     </ul>
@@ -1795,6 +1817,14 @@ spectrums/
   &lt;label&gt;.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 </code></pre></div>
+<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
+<h2 id="known-issue-not-yet-fixed-2026-08-28">Known issue (not yet fixed, 2026-08-28)</h2>
+<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank &lt; self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
+<ul>
+<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
+<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
+</ul>
+<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
 
 
 

DevDoc/implementation/select/index.html

@@ -1119,17 +1119,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1157,6 +1146,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1638,17 +1644,6 @@
     </span>
   </a>
   
-</li>
-        
-          <li class="md-nav__item">
-  <a href="#in-place-keep-only-group-a" class="md-nav__link">
-    <span class="md-ellipsis">
-      
-        In-place: keep only group A
-      
-    </span>
-  </a>
-  
 </li>
         
           <li class="md-nav__item">
@@ -1676,6 +1671,23 @@
     </span>
   </a>
   
+    <nav class="md-nav" aria-label="Implementation notes">
+      <ul class="md-nav__list">
+        
+          <li class="md-nav__item">
+  <a href="#known-gap-not-yet-fixed-2026-08-28" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Known gap (not yet fixed, 2026-08-28)
+      
+    </span>
+  </a>
+  
+</li>
+        
+      </ul>
+    </nav>
+  
 </li>
       
     </ul>
@@ -1728,23 +1740,32 @@ are preserved unchanged; only the data matrices are rewritten.</p>
 <hr />
 <h2 id="synopsis">Synopsis</h2>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>&lt;input-index&gt;
-<span class="w">        </span><span class="o">{</span><span class="w"> </span>--output<span class="w"> </span>&lt;dir&gt;<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="w"> </span><span class="o">}</span>
+<span class="w">        </span>--output<span class="w"> </span>&lt;dir&gt;
 <span class="w">        </span><span class="o">[</span>--group<span class="w">    </span>&lt;name&gt;:&lt;pred&gt;<span class="w">  </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--group-op<span class="w"> </span>&lt;name&gt;:&lt;op&gt;<span class="w">    </span>...<span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-by<span class="w"> </span>&lt;key&gt;<span class="w">          </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--aggregate-op<span class="w"> </span>&lt;op&gt;<span class="w">           </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--select<span class="w">   </span>&lt;col1,col2,...&gt;<span class="w">    </span><span class="o">]</span>
 <span class="w">        </span><span class="o">[</span>--presence-threshold<span class="w"> </span>&lt;N&gt;<span class="w">      </span><span class="o">]</span>
+<span class="w">        </span><span class="o">[</span>--force-copy<span class="w">                  </span><span class="o">]</span>
 </code></pre></div>
 <hr />
 <h2 id="output-destination">Output destination</h2>
-<p>Exactly one of <code>--output</code> or <code>--in-place</code> must be specified.</p>
+<p><code>--output &lt;dir&gt;</code> is required — <code>select</code> always writes a new index; there is no
-<p><strong><code>--output &lt;dir&gt;</code></strong> — writes a new index to <code>&lt;dir&gt;</code>. The source index is
+<code>--in-place</code> mode (2026-08-28: never implemented, removed from the design). The
-unchanged. The MPHF and unitig files are copied; only the data matrices are
+source index is unchanged.</p>
-rewritten with the new column layout.</p>
+<p>Each layer's kmer-identity files (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/
-<p><strong><code>--in-place</code></strong> — rewrites the data matrices of the source index directly.
+<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code>) are never rewritten by a
-Removed or replaced columns are lost. The operation writes to temporary files
+column projection/aggregation, so they are hard-linked into the output rather
-first, then renames atomically, so an interrupted run leaves the index intact.</p>
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); <code>--force-copy</code>
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+<code>select</code> itself would affect the other). Only the <code>presence</code>/<code>counts</code>
+subdirectory is ever a genuinely new, independent file.</p>
+<p>To replace an index with a selected version of itself, select to a temporary
+directory and swap it in (<code>rm -rf INDEX &amp;&amp; mv INDEX.tmp INDEX</code>) — the case
+<code>--in-place</code> used to cover.</p>
 <hr />
 <h2 id="defining-output-columns">Defining output columns</h2>
 <h3 id="named-groups-group">Named groups — <code>--group</code></h3>
@@ -1908,9 +1929,6 @@ is &gt; N.</p>
 <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>out<span class="w"> </span><span class="se">\</span>
 <span class="w">  </span>--select<span class="w"> </span><span class="s2">&quot;Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01&quot;</span>
 </code></pre></div>
-<h3 id="in-place-keep-only-group-a">In-place: keep only group A</h3>
-<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>myindex<span class="w"> </span>--in-place<span class="w"> </span>--group<span class="w"> </span><span class="s2">&quot;A:group=A&quot;</span><span class="w"> </span>--select<span class="w"> </span><span class="s2">&quot;A&quot;</span>
-</code></pre></div>
 <h3 id="compose-with-filter">Compose with filter</h3>
 <div class="highlight"><pre><span></span><code><span class="c1"># Step 1: keep only B. nana-specific k-mers</span>
 obikmer<span class="w"> </span>filter<span class="w"> </span>myindex<span class="w"> </span>--output<span class="w"> </span>filtered<span class="w"> </span><span class="se">\</span>
@@ -1921,32 +1939,46 @@ obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </s
 </code></pre></div>
 <hr />
 <h2 id="implementation-notes">Implementation notes</h2>
-<p><code>select</code> does not rebuild the MPHF. The 256 partitions are processed in parallel
+<p><code>select</code> does not rebuild the MPHF. Every partition is processed independently
-(rayon), each writing its output independently; results require no synchronisation
+(<code>PartitionRunner</code>), each writing its own output layers; no cross-partition
-because every partition owns a distinct set of files.</p>
+synchronisation is needed.</p>
-<p>For each layer in each partition:</p>
+<p>For each layer in each partition (<code>obikselect::select_layer::select_partition</code>):</p>
 <ol>
-<li>The slot count <code>n</code> is read by opening the source data matrix.</li>
+<li><code>copy_layer_files</code> hard-links the source layer's kmer-identity files
-<li>A new data matrix is built with M columns (M = number of output columns).</li>
+   (<code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/
-<li>For each slot <code>s</code> in <code>0..n</code>:</li>
+   <code>fingerprint.bin</code>/<code>layer_meta.json</code>) into the destination — never a real
-<li><code>old_row = matrix.fill_row(s)</code> — reads the original <code>N</code>-column row without allocating.</li>
+   copy unless linking fails or <code>--force-copy</code> is given.</li>
-<li>For each output column <code>j</code>:<ul>
+<li>A new data matrix is built with M columns (M = number of output columns),
-<li><code>new_row[j] = aggregate(op, old_row[group_indices])</code>.</li>
+   under a fresh <code>presence/</code>/<code>counts/</code> subdirectory (never touching the
-<li>Pass-through columns are represented as single-element groups with the
+   source's own).</li>
-   default operator (<code>any</code> for presence, <code>sum</code> for count) — same code path.</li>
+<li><strong>Presence source (2026-08-28: <code>batch_presence_counts</code>)</strong>: one shared pass
-</ul>
+   over the source bit matrix computes every output group's presence count at
-</li>
+   once — row-major native for a <code>Sparse</code> source (<code>for_each_genome_in_row</code>,
-<li>The new row is written slot by slot into each column builder.</li>
+   which has no column representation to read a <code>col_view</code> from at all — the
-<li>All plain files in the source layer directory (<code>mphf.bin</code>, <code>unitigs.bin</code>,
+   reason this replaced the old per-group loop, not just an optimisation of
-   evidence files, <code>layer_meta.json</code>) are copied verbatim; only the <code>presence/</code>
+   it), deduplicated column-major (one <code>col_view</code> per <em>distinct referenced
-   or <code>counts/</code> subdirectory is rewritten.</li>
+   column</em>, not per group) for <code>Columnar</code>/<code>Packed</code>. Every <code>AggOp</code> for a bit
+   matrix is then a cheap derivation of that one count vector (<code>sum</code> = the
+   count itself, <code>any</code>/<code>max</code> = <code>count ≥ 1</code>, <code>all</code>/<code>min</code> = <code>count == group
+   size</code>, <code>none</code> = <code>count == 0</code>) — see
+   <code>obikselect::select_layer::agg_result_from_count</code>.</li>
+<li><strong>Count source</strong>: unchanged, one <code>col_view</code>-driven pass per output column
+   via <code>MatrixGroupOps</code> — <code>sum</code>/<code>min</code>/<code>max</code> are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.</li>
 <li><code>index.meta</code> is rewritten with the new genome list and updated <code>with_counts</code>.</li>
 </ol>
-<p><strong><code>--in-place</code> write strategy:</strong> new data is written to a temporary sibling
+<h3 id="known-gap-not-yet-fixed-2026-08-28">Known gap (not yet fixed, 2026-08-28)</h3>
-directory (<code>presence_new/</code> or <code>counts_new/</code>); on success the old directory is
+<p>Step 4 above still panics (<code>col_view() not available on Sparse
-removed and the temporary one is renamed into place. An interrupted run leaves
+PersistentCompactIntMatrix</code>) if the source is a <strong>count</strong> index packed
-at most one stale <code>*_new/</code> directory; the original data is intact until the
+sparse — <code>batch_presence_counts</code>' row-major treatment was only ported to the
-rename step.</p>
+bit-matrix (<code>Presence</code>) case, since that was the one actually blocking a real
+benchmark run. <code>select</code>/<code>filter</code> on a sparse-packed count index still hits
+this; the fix would follow the same shape (a <code>PersistentSparseCompactIntMatrix</code>
+row-major decode, analogous to <code>for_each_genome_in_row</code>), just not done. Since
+<code>obisys::numa::runner::PartitionRunner</code>'s panic-propagation fix (see
+<code>architecture/numa_partition_runner.md</code>), this at least fails fast (process
+panic, exit 101) instead of hanging.</p>
 
 
 

DevDocMD/architecture/numa_partition_runner.md

@@ -304,10 +304,29 @@ step, capped by that node's remaining dormant workers and by the run's total
 budget (`n_total`) — balance across nodes is now guaranteed by construction,
 not incidental to channel implementation details.
 
+## Panic propagation (fixed, 2026-08-28)
+
+A worker whose `f(i)` panicked used to never send its `WorkerEvent::Completed`
+— the controller's `while completed < n_total` loop then waited forever for
+an event that partition could no longer produce, since other live
+workers/timer threads kept `event_rx` open. Discovered via `obikselect`
+panicking on an unimplemented `Sparse` matrix case (see `implementation/select.md`):
+the process hung indefinitely instead of erroring out.
+
+Fixed: each worker wraps `f(i)` in `std::panic::catch_unwind` and sends a new
+`WorkerEvent::Panicked(i, payload)` instead of silently dropping the
+partition. The controller counts it toward `completed` (unblocking the loop)
+and keeps the first payload seen; once `run` returns, `panic::resume_unwind`
+re-raises it on the caller's thread — the original message/backtrace still
+surfaces, from the right place, instead of a silent deadlock. Takes priority
+over a plain `Err` (a panic means a bug, not a normal typed failure).
+
 ## Open questions
 
 - **Error handling**: `run` currently returns the first error; remaining errors
-  are dropped.  A `Vec<E>` return would give complete diagnostics.
+  are dropped.  A `Vec<E>` return would give complete diagnostics. (Panics are
+  now caught and re-raised individually — see above — this only concerns
+  plain `Err` results.)
 
 - **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
   (start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an

DevDocMD/implementation/merge.md

@@ -194,3 +194,14 @@ spectrums/
   <label>.json               ← one file per genome, rebuilt from all sources
 index.meta                   ← complete genome list + evidence kind written at bootstrap
 ```
+
+`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json` marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — `merge_partition` never rewrites them for pre-existing layers, only the `presence`/`counts` subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
+
+## Known issue (not yet fixed, 2026-08-28)
+
+Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. `obikmer merge -o out IDX IDX --rename-duplicates`) — panics deep in the MPHF's rank-select structure (`common_traits::select_in_word`, `assertion failed: rank < self.count_ones()`), inside `MphfLayer::find` called from `merge_partition`'s "is this source kmer already in dst" check against the bootstrap-copied `dst_layers`. Root cause not identified; ruled out so far:
+
+- **Not an empty-new-layer issue**: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
+- **Not caused by the hard-link change above**: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
+
+Only reproduces when `sources[0]` and `sources[1]` are the exact same on-disk path opened as two independent `KmerIndex` handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since `PartitionRunner`'s panic propagation fix (see `architecture/numa_partition_runner.md`) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.

DevDocMD/implementation/select.md

@@ -23,28 +23,37 @@ are preserved unchanged; only the data matrices are rewritten.
 
 ```sh
 obikmer select <input-index>
-        { --output <dir> | --in-place }
+        --output <dir>
         [--group    <name>:<pred>  ...]
         [--group-op <name>:<op>    ...]
         [--aggregate-by <key>          ]
         [--aggregate-op <op>           ]
         [--select   <col1,col2,...>    ]
         [--presence-threshold <N>      ]
+        [--force-copy                  ]
 ```
 
 ---
 
 ## Output destination
 
-Exactly one of `--output` or `--in-place` must be specified.
+`--output <dir>` is required — `select` always writes a new index; there is no
+`--in-place` mode (2026-08-28: never implemented, removed from the design). The
+source index is unchanged.
 
-**`--output <dir>`** — writes a new index to `<dir>`. The source index is
+Each layer's kmer-identity files (`mphf.bin`/`unitigs.bin`/`evidence.bin`/
-unchanged. The MPHF and unitig files are copied; only the data matrices are
+`unitigs.bin.idx`/`fingerprint.bin`/`layer_meta.json`) are never rewritten by a
-rewritten with the new column layout.
+column projection/aggregation, so they are hard-linked into the output rather
+than copied — no extra disk for them even on a large index. Falls back to a
+real copy automatically if linking fails (different filesystems); `--force-copy`
+forces a real copy always, for an output that must survive independently of the
+source on disk (a hard link shares the same inode — rewriting one path outside
+`select` itself would affect the other). Only the `presence`/`counts`
+subdirectory is ever a genuinely new, independent file.
 
-**`--in-place`** — rewrites the data matrices of the source index directly.
+To replace an index with a selected version of itself, select to a temporary
-Removed or replaced columns are lost. The operation writes to temporary files
+directory and swap it in (`rm -rf INDEX && mv INDEX.tmp INDEX`) — the case
-first, then renames atomically, so an interrupted run leaves the index intact.
+`--in-place` used to cover.
 
 ---
 
@@ -186,12 +195,6 @@ obikmer select myindex --output out \
   --select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
 ```
 
-### In-place: keep only group A
-
-```sh
-obikmer select myindex --in-place --group "A:group=A" --select "A"
-```
-
 ### Compose with filter
 
 ```sh
@@ -207,28 +210,45 @@ obikmer select filtered --output final --aggregate-by site
 
 ## Implementation notes
 
-`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
+`select` does not rebuild the MPHF. Every partition is processed independently
-(rayon), each writing its output independently; results require no synchronisation
+(`PartitionRunner`), each writing its own output layers; no cross-partition
-because every partition owns a distinct set of files.
+synchronisation is needed.
 
-For each layer in each partition:
+For each layer in each partition (`obikselect::select_layer::select_partition`):
 
-1. The slot count `n` is read by opening the source data matrix.
+1. `copy_layer_files` hard-links the source layer's kmer-identity files
-2. A new data matrix is built with M columns (M = number of output columns).
+   (`mphf.bin`/`unitigs.bin`/`evidence.bin`/`unitigs.bin.idx`/
-3. For each slot `s` in `0..n`:
+   `fingerprint.bin`/`layer_meta.json`) into the destination — never a real
-   - `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
+   copy unless linking fails or `--force-copy` is given.
-   - For each output column `j`:
+2. A new data matrix is built with M columns (M = number of output columns),
-     - `new_row[j] = aggregate(op, old_row[group_indices])`.
+   under a fresh `presence/`/`counts/` subdirectory (never touching the
-     - Pass-through columns are represented as single-element groups with the
+   source's own).
-       default operator (`any` for presence, `sum` for count) — same code path.
+3. **Presence source (2026-08-28: `batch_presence_counts`)**: one shared pass
-   - The new row is written slot by slot into each column builder.
+   over the source bit matrix computes every output group's presence count at
-4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
+   once — row-major native for a `Sparse` source (`for_each_genome_in_row`,
-   evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
+   which has no column representation to read a `col_view` from at all — the
-   or `counts/` subdirectory is rewritten.
+   reason this replaced the old per-group loop, not just an optimisation of
+   it), deduplicated column-major (one `col_view` per *distinct referenced
+   column*, not per group) for `Columnar`/`Packed`. Every `AggOp` for a bit
+   matrix is then a cheap derivation of that one count vector (`sum` = the
+   count itself, `any`/`max` = `count ≥ 1`, `all`/`min` = `count == group
+   size`, `none` = `count == 0`) — see
+   `obikselect::select_layer::agg_result_from_count`.
+4. **Count source**: unchanged, one `col_view`-driven pass per output column
+   via `MatrixGroupOps` — `sum`/`min`/`max` are genuine per-value reductions
+   for a count matrix, not derivable from a single presence count the way
+   they are for a bit matrix.
 5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
 
-**`--in-place` write strategy:** new data is written to a temporary sibling
+### Known gap (not yet fixed, 2026-08-28)
-directory (`presence_new/` or `counts_new/`); on success the old directory is
+
-removed and the temporary one is renamed into place. An interrupted run leaves
+Step 4 above still panics (`col_view() not available on Sparse
-at most one stale `*_new/` directory; the original data is intact until the
+PersistentCompactIntMatrix`) if the source is a **count** index packed
-rename step.
+sparse — `batch_presence_counts`' row-major treatment was only ported to the
+bit-matrix (`Presence`) case, since that was the one actually blocking a real
+benchmark run. `select`/`filter` on a sparse-packed count index still hits
+this; the fix would follow the same shape (a `PersistentSparseCompactIntMatrix`
+row-major decode, analogous to `for_each_genome_in_row`), just not done. Since
+`obisys::numa::runner::PartitionRunner`'s panic-propagation fix (see
+`architecture/numa_partition_runner.md`), this at least fails fast (process
+panic, exit 101) instead of hanging.