feat(matrix): add partial group reductions and column persistence
Expands MatrixGroupOps with partial_group_min/max helpers for bitwise reductions and introduces add_col_from methods to persist external vectors as matrix columns. Refactors column aggregation in the partitioner to leverage these group operations directly, replacing iterative row processing with simplified builder lifecycle management and explicit metadata serialization.
This commit is contained in:
@@ -444,71 +444,64 @@ Defined **once at the index level** from column metadata. Valid in all matrices
|
||||
|
||||
### MatrixGroupOps
|
||||
|
||||
Group operations expose only **additive intermediates** backed by temp files. Final predicates are applied at the index level after accumulation.
|
||||
Five required primitives + two default methods derived from them. All return temp-file-backed types.
|
||||
|
||||
```rust
|
||||
pub trait MatrixGroupOps {
|
||||
// required
|
||||
fn partial_group_presence_count(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
|
||||
fn partial_group_sum(&self, g: &ColGroup)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
|
||||
fn partial_group_any(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempBitVec>;
|
||||
fn partial_group_min(&self, g: &ColGroup)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
fn partial_group_max(&self, g: &ColGroup)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
|
||||
// defaults derived from partial_group_presence_count
|
||||
fn partial_group_all(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempBitVec>; // slot=1 iff count == g.indices.len()
|
||||
fn partial_group_none(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempBitVec>; // slot=1 iff count == 0
|
||||
}
|
||||
```
|
||||
|
||||
Implemented for both `PersistentCompactIntMatrix` and `PersistentBitMatrix`. For bit matrices, `partial_group_sum` delegates to `partial_group_presence_count(g, 1)`.
|
||||
Implemented for both `PersistentCompactIntMatrix` and `PersistentBitMatrix`.
|
||||
|
||||
For **bit matrices**: values are 0/1, so `partial_group_sum` = `partial_group_presence_count(g, 1)`; `partial_group_min` is AND (set first column then mask-with remaining); `partial_group_max` is OR via `partial_group_any` + `inc_present`.
|
||||
|
||||
**`partial_group_presence_count` — chunking for large groups:**
|
||||
|
||||
When `g.indices.len() < 255`: per-slot counts stay within `u8` range. Use `inc_present_fast` (bit matrix) or `inc_predicate_fast(col_view(c), |v| v >= threshold)` (int matrix) — raw u8 increment, no overflow map written.
|
||||
When `g.indices.len() < 255`: per-slot counts stay within `u8` range. Use `inc_present_fast` (bit) or `inc_predicate_fast(col_view(c), |v| v >= threshold)` (int) — raw u8 increment, no overflow entry written.
|
||||
|
||||
When `g.indices.len() ≥ 255`: process in chunks of 254 columns (each chunk stays within u8 range), accumulate into a running builder via `.add(chunk_frozen.view())`.
|
||||
When `g.indices.len() ≥ 255`: process in chunks of 254 columns, accumulate via `.add(chunk_frozen.view())`.
|
||||
|
||||
```
|
||||
fast path (< 255 cols):
|
||||
builder = TempCompactIntVecBuilder::new(n)
|
||||
for c in group:
|
||||
builder.inc_predicate_fast(matrix.col_view(c), |v| v >= threshold)
|
||||
builder.freeze()
|
||||
**`partial_group_min` (int matrix)**: copy first column via `.add(col_view(first))` (start from 0 ⇒ copy), then `.min(col_view(c))` for remaining.
|
||||
|
||||
slow path (≥ 255 cols):
|
||||
result = TempCompactIntVecBuilder::new(n)
|
||||
for chunk in group.chunks(254):
|
||||
chunk_b = TempCompactIntVecBuilder::new(n)
|
||||
for c in chunk:
|
||||
chunk_b.inc_predicate_fast(matrix.col_view(c), |v| v >= threshold)
|
||||
frozen = chunk_b.freeze()
|
||||
result.add(frozen.view())
|
||||
result.freeze()
|
||||
```
|
||||
**`partial_group_max` (int matrix)**: `.max(col_view(c))` for all columns (start from 0 ⇒ first column acts as copy).
|
||||
|
||||
**`partial_group_any`** uses `or_where` on `TempBitVecBuilder`:
|
||||
**`partial_group_any`** uses `or_where` on `TempBitVecBuilder` (two-pass: primary bytes then overflow entries).
|
||||
|
||||
```
|
||||
result = TempBitVecBuilder::new(n)
|
||||
for c in group:
|
||||
result.or_where(matrix.col_view(c), |v| v >= threshold)
|
||||
result.freeze()
|
||||
```
|
||||
**`partial_group_all` / `partial_group_none`** (default): call `partial_group_presence_count`, then iterate slots to produce the bit result. O(n) extra pass, not chunked.
|
||||
|
||||
**Non-additive predicates** (`group_all`, `group_at_least(k)`) are composed at the index level:
|
||||
### add_col_from — matrix builder integration
|
||||
|
||||
Both matrix builders accept temp-file results directly:
|
||||
|
||||
```rust
|
||||
// "present in >= 2 ingroup columns with count >= 3, absent from all outgroup"
|
||||
let presence = layers.map(|l| l.partial_group_presence_count(&ingroup, 3)?).add_all()?;
|
||||
let in_mask = presence.view().geq(2); // IntSliceView method
|
||||
// PersistentBitMatrixBuilder
|
||||
fn add_col_from(&mut self, src: &TempBitVec) -> io::Result<()>
|
||||
fn add_col_from_int(&mut self, src: &TempCompactIntVec) -> io::Result<()> // nonzero → 1
|
||||
|
||||
let out_sum = layers.map(|l| l.partial_group_sum(&outgroup)?).add_all()?;
|
||||
let out_mask = out_sum.view().leq(0);
|
||||
|
||||
let mut mask_b = TempBitVecBuilder::new(n)?;
|
||||
mask_b.copy_from(in_mask);
|
||||
mask_b.and(out_mask);
|
||||
// PersistentCompactIntMatrixBuilder
|
||||
fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()>
|
||||
fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> // bit → 0/1 u32
|
||||
```
|
||||
|
||||
`add_col_from` copies the temp file to the matrix directory and increments `n_cols`; `close()` writes `meta.json` with the final column count. No separate `write_meta` step needed.
|
||||
|
||||
### mask_with
|
||||
|
||||
Direct method on `PersistentCompactIntVecBuilder` (and delegation via `TempCompactIntVecBuilder`). Zeros every slot where the corresponding mask bit is 0. Iterates only zero bits — O(n_zeros), O(1) when mask is all-ones.
|
||||
|
||||
Reference in New Issue
Block a user