obitools4/autodoc/docmd/pkg_obichunk.md

# `obichunk`: High-Performance Chunking and Dereplication of Biological Sequences

The `obichunk` package provides scalable, configurable infrastructure for preprocessing large-scale biological sequence data (e.g., FASTA/FASTQ). It enables efficient grouping, sorting, deduplication, and batched streaming of sequences—critical for metabarcoding, metagenomics, or any high-throughput NGS workflow.

---

## Core Functionalities

### `ISequenceChunk`
Unified entry point for sequence chunking, supporting both **in-memory** and **on-disk** execution modes.  
- Accepts an `obiiter.IBioSequence` iterator and a classifier (`obiseq.BioSequenceClassifier`).  
- Mode selection via `onMemory` flag: routes to either `ISequenceChunkOnMemory` or `ISequenceChunkOnDisk`.  
- Optional parameters:
  - `dereplicate`: deduplicate identical sequences per batch.
  - `na`: defines placeholder for missing/ambiguous characters (e.g., `"N"`, `"?"`).
  - `statsOn`: enables metadata tracking (e.g., sample IDs, primer names) for statistics.
  - `uniqueClassifier`: optional secondary classifier to assign unique labels.

Returns an iterator over processed sequences (`obiiter.IBioSequence`), supporting streaming pipelines and downstream integration.

---

### `ISequenceChunkOnDisk`
Efficiently splits sequences into **temporary on-disk batches** (`.fastx` files), ideal for large datasets.  
- Automatically manages a temp directory (`obiseq_chunks_*`) and cleans up post-processing.
- Uses `find` to discover all generated chunk files recursively.
- Asynchronous streaming: batches are yielded via an iterator as they’re written, decoupling production and consumption.
- Optional per-batch dereplication using composite keys (sequence + classification).
- Logs batch count and start events for monitoring.

Internally leverages:
- `obiiter.MakeIBioSequence()` to build output iterator.
- `obiformats.WriterDispatcher` for parallel file writing.
- A dedicated goroutine to read, classify/dereplicate, and emit batches.

---

### `ISequenceChunkOnMemory`
Performs **in-memory parallel chunking** of sequences into classification-based batches.  
- Routes each sequence to a bucket (flux) using the classifier.
- Maintains one `BioSequenceSlice` per classification group in memory (thread-safe via mutex).
- Emits batches **only after full input consumption**, preserving deterministic batch order (0, 1, …).
- Parallel processing: each flux handled in its own goroutine.
- Fails fast on internal errors (e.g., channel issues) via `log.Fatalf`.

Ideal for RAM-sufficient workloads requiring low-latency, ordered batch output.

---

### `Options` System
Configurable pipeline behavior via functional options pattern.  
- Immutable configuration builder: `MakeOptions([]WithOption)` applies setters to internal struct.
- Key options:
  - **Categorization**: `OptionSubCategory(...)` appends sample/marker labels; `PopCategories()` retrieves first.
  - **Missing values**: `OptionNAValue(na)` customizes placeholder (default: `"?"`).
  - **Statistics**: `OptionStatOn(...)` registers fields for metadata tracking.
  - **Batching**:
    - `OptionBatchCount(n)` sets number of batches (e.g., for hashing).
    - `OptionsBatchSize(size)` defines items per batch.
  - **Concurrency**: `OptionsParallelWorkers(n)`.
  - **Sorting strategy**:
    - `OptionSortOnDisk()` enables disk-backed sorting.
    - `OptionSortOnMemory()` (default) uses RAM-based sort.
  - **Singleton filtering**:
    - `OptionsNoSingleton()` excludes singleton reads (count = 1).
    - `OptionsWithSingleton()` allows them.

Defaults drawn from `obidefault`, ensuring reproducibility and ease of use.

---

### `ISequenceSubChunk`
Parallel, class-based sorting and re-batching of sequence batches.  
- Input: iterator over `BioSequenceBatch`, classifier, and worker count.
- For each batch:
  - If size >1: sequences are sorted *in-place* by classification code (via custom `sort.Interface`).
  - Consecutive sequences with same class are regrouped into new batches.
- Uses atomic counters (`nextOrder`) to assign globally increasing order IDs across workers—ensuring deterministic inter-batch ordering.
- Preserves input-order *within* each new batch.

Use case: preparing sorted, class-homogeneous batches for downstream tasks (e.g., consensus calling or alignment).

---

### `IUniqueSequence`
End-to-end **dereplication** pipeline: groups identical sequences, aggregates counts and metadata.  
- Input iterator + optional `Options`.
- Parallelization via configurable workers (falls back to single-threaded if disk sorting enabled).
- **Splitting phase**:
  - Uses `HashClassifier` to partition input deterministically (controlled by `BatchCount`).
- **Storage selection**:
  - In-memory: via `ISequenceChunkOnMemory`.
  - On-disk: uses `ISequenceSubChunk` + external sort (single worker required).
- **Uniqueness logic**:
  - Composite classifier: sequence identity + optional annotations (sample, primer).
  - NA handling for missing annotation fields.
- **Singleton filtering**: optionally excludes reads with count =1 (`NoSingleton()`).
- **Parallel deduplication**:
  - Workers process chunks via `ISequenceSubChunk` + per-group aggregation.
- **Merging**:
  - Aggregates results via `IMergeSequenceBatch`, preserving counts, stats, and ordering.

Scalable from small datasets to terabyte-scale NGS runs.