Compare commits

...
370 Commits
Author SHA1 Message Date
Eric Coissac 26026288de refactor(benchmark): consolidate generated artifacts under run/ directory
Release / create-release (push) Successful in 2m34s
Release / build-linux-x86_64 (push) Successful in 8m10s
Release / build-macos-arm64 (push) Failing after 1m19s
Restructure the benchmark pipeline to direct all simulated data, indices, statistics, and query outputs into a unified `run/` directory. Update Makefile targets, shell scripts, and Python utilities to resolve paths relative to this new base. Adjust documentation and dependency tracking to match the revised layout, and remove outdated temporary artifacts.
2026-08-29 00:20:31 +02:00
Eric Coissac 54e4600120 feat: add batched int group stats API and expand benchmark variants
Introduces a `batch_int_group_stats` API for computing presence counts, sums, minimums, and maximums across sparse and dense matrix representations. The selection layer now utilizes this batched approach to optimize aggregation semantics for boolean and numeric operations. Additionally, reorganizes the benchmarking infrastructure to support querying across presence and count index variants in both dense and sparse formats, including new packing scripts and updated statistics aggregation.
2026-08-28 23:37:16 +02:00
Eric Coissac 579cfd8752 docs: document partition runner deadlock fix and select CLI updates
Update documentation to reflect architectural fixes in the NUMA partition runner and implementation changes in the merge and select modules. The partition runner now wraps task execution in `catch_unwind`, emitting panic events that count toward completion thresholds to prevent controller deadlocks, while re-raising the first payload on return. CLI documentation for select has been updated to mandate output directories, introduce default hard-linking with automatic fallback, and document presence aggregation optimizations. Known issues such as self-merge crashes and sparse-packed index gaps are now documented as failing fast due to these improvements.
2026-08-28 23:25:46 +02:00
Eric Coissac 6b0c0867cc implement hybrid hard-link copy strategy for directory bootstrap
Introduces recursive directory replication functions that default to hard-linking for space efficiency, with automatic fallback to full file copying on cross-filesystem failures. Enforces mandatory real copies for presence and counts subdirectories to guarantee independent data matrices before merging, while preserving the existing separation between immutable index files and mutable matrix files.
2026-08-28 23:18:34 +02:00
Eric Coissac 93fe838f97 Remove in-place flag, optimize presence counting, and handle NUMA panics
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
2026-08-28 23:16:37 +02:00
Eric Coissac 7363370065 feat: add --force-copy flag to copy files instead of hard-linking
Introduces the `--force-copy` CLI argument to bypass hard-linking during selection. When enabled, unchanged kmer-identity files are copied instead of hard-linked to ensure output independence from the source filesystem. The default behavior remains unchanged, continuing to use hard-links with automatic fallback to copying on failure.
2026-08-28 23:08:42 +02:00
Eric Coissac caf59b5658 Update documentation for CLI command renames and new commands
Renames the reindex command to convert across all usage guides and navigation menus. Adds documentation for the newly introduced name-tree command. Updates flag references, such as replacing --metric with --distance in pack and phylo modules, and adjusts MkDocs configuration to match the revised structure. All changes are strictly limited to documentation and configuration files.
2026-08-28 23:06:05 +02:00
Eric Coissac 95fa0c93b2 Rename CLI to obikmer2, add phylo commands, and unify index caching
Restructure the workspace and rename the CLI application to obikmer2. Replace direct KmerIndex usage across all commands with Arc-wrapped IndexCache to enable shared ownership. Introduce builder patterns for algorithmic operations and integrate explicit progress tracking. Add new Phylo, NameTree, and Convert subcommands with expanded CLI flags. Consolidate module structure, update dependency specifications, and remove legacy directories.
2026-08-28 22:34:46 +02:00
Eric Coissac e101f629e6 Add family overlap computation and CLI options for phylo analysis
Extends the entropy annex pipeline to optionally compute family overlap data in a single pass, introducing a memory-mapped binary format and an in-memory accumulator for efficient storage. Adds --min-shared-family to auto-exclude genomes below a threshold during SNP-family distance computation, and --family-overlap to export a shared family matrix as CSV. Centralizes upper-triangle indexing into a shared helper function to eliminate duplication.
2026-08-28 22:03:21 +02:00
Eric Coissac 4f34a646c5 Implement SNP distance models with gamma correction and PHYLIP output
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
2026-08-28 21:54:19 +02:00
Eric Coissac 0b40d2d0da Add phylogenetic export support for TNT, PhyG, and IQ-TREE
Extends the phylogenetic pipeline with exporters for TNT, PhyG, and IQ-TREE that generate executable scripts, cost matrices, and recoded alignments. Adds internal helpers for state indexing, floating-point matrix scaling, and Floyd-Warshall metric closure to satisfy external tool constraints. Adjusts visibility modifiers for sibling iterators and entropy annex structs, and updates design documentation for evolutionary distance metrics.
2026-08-28 21:32:32 +02:00
Eric Coissac 1536217ac1 feat: add Sankoff cost-matrix calibration CLI and serialization
Introduce the --sankoff flag to generate subsampled calibration bundles and pairwise cost matrices. Expose internal sibling algorithm types as public to support external consumption. Implement YAML, CSV, and FASTA output serialization using serde, gated by a configurable substitution saturation ceiling and requiring the --subsample flag.
2026-08-28 21:19:20 +02:00
Eric Coissac 9043868228 Add single-pass pairwise reduction pipeline for SNP alignment
Replaces the previous two-pass design with a single-pass reduction that computes per-genome-pair substitution counts, cardinality and base-composition tallies, and transition cost matrices. Introduces internal APIs to derive downstream n×n matrices from reduced data and applies configurable filtering based on SNP ratios and exclusion masks. Adds a fused pipeline entry point aggregating alignment and distance outputs for downstream phylogenetic commands. Adjusts visibility of existing reduction functions to restrict external access while permitting sibling module invocation.
2026-08-28 21:10:16 +02:00
Eric Coissac e015362ce6 Refactor sampling and reduction to use batched SurvivingFamily data
Replace per-family callbacks with a per-layer batch callback that accumulates surviving families into an Arc<Vec<SurvivingFamily>>. Update the alignment reduction routine to iterate over this batched structure, enabling zero-copy sharing for parallel operations and structuring the code for future extensibility.
2026-08-28 21:02:35 +02:00
Eric Coissac 89629e118d Add sibling family size distribution statistics and CLI flags
Introduces `--sibling-stats` and `--sibling-hist` flags to compute and export family-size distributions from pre-built annexes. The new algorithms module implements parallel and sequential scanning routines, while the CLI layer handles CSV export with deduplication and standardized error handling.
2026-08-28 20:59:07 +02:00
Eric Coissac beb2951c20 feat: add phylogenetic sampling, masking, and plain FASTA writer
Introduces configurable phylogenetic sampling and masking controls via new CLI flags (`--subsample`, `--entropy`, `--exclude-genome`, etc.). Adds a complete SNP pseudo-alignment pipeline featuring entropy-biased Gaussian sampling, post-hoc state masking, and proportional per-layer filtering. Extends the FASTA writer with a `write_plain_record` API for bare-header output without JSON annotations.
2026-08-28 20:54:51 +02:00
Eric Coissac 5a0b71d105 Add per-family Shannon entropy calculation and reporting
Introduces an entropy.pent persisted format and algorithms to compute per-family Shannon entropy across genomic partitions. Refactors IndexCache to use Arc-based layer sharing for efficient iteration and restructures the siblings module with extension traits for annex generation. Adds a --shannon CLI flag to export partition-level entropy metrics to CSV.
2026-08-28 20:45:47 +02:00
Eric Coissac bbb58a698f Rename SiblingAnnexBuildExt to SiblingExt and add sibling iterators
Refactors the `SiblingAnnexBuildExt` trait to `SiblingExt` across all imports, implementations, and documentation. Introduces new modules for streaming and batched iteration over sibling layers, along with a two-phase pipeline for resolving minorant families.
2026-08-28 20:36:17 +02:00
Eric Coissac 920120d20e Implement --sibling-annex flag and k-mer family annex builder
Introduces a new `--sibling-annex` CLI flag that triggers the construction of a sibling-count/minorant annex for multi-genome indices. The implementation adds a `FamilyMask` data structure with memory-mapped I/O, enabling lock-free concurrent updates via atomic bitwise operations. Batched processing improves cache locality and parallelism, while helper functions derive family presence and minorant flags dynamically. The feature is exposed through an `IndexCache` extension trait, protected by an exclusive directory lock to prevent index corruption during construction.
2026-08-28 20:29:01 +02:00
Eric Coissac d084396aba Introduce concurrency-bounded parallel processing for index cache layers
Implement an `obipipeline::Throttle` with an RAII guard to acquire and release concurrency slots. Expose new bounded parallel methods on `IndexCache` to process cached layers with a configurable cap. Refactor downstream aggregation logic to use single-pass parallel map-reduce instead of manual collect-map-reduce sequences, enforcing a maximum of 8 concurrent layer scans to bound memory usage.
2026-08-28 20:20:12 +02:00
Eric Coissac 9dee6dcd08 refactor: decouple tree construction from Newick serialization
Refactor phylogenetic algorithms (`neighbor_joining`, `upgma`) to return an explicit `Tree` struct instead of a serialized Newick string. This change makes serialization an explicit step for downstream consumers via the new public `Tree::to_newick()` method, decoupling tree construction from output formatting. The `siblings` module has also been moved to `siblings_old`.
2026-08-28 20:15:23 +02:00
Eric Coissac 635fc830d1 feat: add phylogenetic analysis CLI command and lazy distance traits
Introduce the `obikphylo` crate to support genome-vs-genome distance matrix computation and phylogenetic tree inference via Neighbor-Joining and UPGMA algorithms. Extend the `obikmer2` CLI with a new `phylo` command that exposes configurable metrics, presence thresholds, and CSV/Newick output options. Refactor `obikindex` to expose trait-based partial aggregation for efficient distance metric finalization without full matrix materialization. Update dependency graphs and remove obsolete storage modules.
2026-08-28 20:11:13 +02:00
Eric Coissac 4b6005962e Rename PersistentCompactIntMatrix to PersistentIntMatrix
Update all matrix type references, imports, and instantiations across
the codebase to use the new PersistentIntMatrix name. The change also
includes standardizing code formatting, such as multi-line statements
and import ordering, without altering any behavioral logic or public
API contracts.
2026-08-28 19:59:59 +02:00
Eric Coissac 6bdc9354d3 refactor: restructure obikphylo module layout and adjust visibility
Relocate cardcomp into the siblings directory and update internal import paths to reflect the new hierarchy. Adjust the crate's public API surface by updating module declarations and re-exports. This is a purely structural change with no modifications to logic, data models, or runtime behavior.
2026-08-28 19:46:09 +02:00
Eric Coissac 4ea3cd32ba feat: add convert command for in-place index evidence modification
Introduce a new CLI command that modifies existing indices to support exact, approximate, or hybrid evidence modes. This change enables the reindex module in obikrebuild, exposing an IndexReindex trait for evidence persistence and layer processing. It also refactors obikindexer to publicly expose an IndexBuilder trait for post-build evidence construction, while adding a set_evidence API in obikindex to safely update configuration fields without altering core parameters.
2026-08-28 19:43:14 +02:00
Eric Coissac bd7729b095 refactor: Shift KmerIndex ownership to Arc for thread-safe sharing
Replaces lifetime-bound references with runtime reference counting across multiple crates. This enables safe concurrent access across parallel workers without explicit cloning or manual lifetime management. Introduces the `query` and `utils` CLI commands in obikmer2, along with supporting modules for batch processing, sparse indexing, sliding-window findere logic, and output formatting. Updates dependency manifests and aligns test suites with the new ownership model.
2026-08-28 19:32:45 +02:00
Eric Coissac 10a0909687 refactor: unify iterator construction in nonzero_iter
Change the return type of `nonzero_iter` from an opaque `impl Iterator` to a `Box<dyn Iterator>`. This enables direct return coercion in the `Count` layer branch, removing an explicit `Box::new()` wrapper and aligning it with the existing `Presence` implementation. The modification shifts iterator construction to runtime dynamic dispatch while preserving the public API contract and iteration semantics.
2026-08-28 19:08:55 +02:00
Eric Coissac b0890d1781 Add sparse storage variant to PersistentCompactIntMatrix
Introduce a new `Sparse` format alongside existing `Columnar` and `Packed` variants, enabling optimized row-major pairwise counting for distance and similarity metrics via the `CountPartials` trait. Update storage detection priorities, extend matrix dispatch logic to sparse backends, and correct diagonal/off-diagonal formulas in bit matrix partial computations. Expand layer APIs with format-agnostic `nonzero_iter`, update usage documentation for the `--sparse` flag, and add comprehensive tests verifying roundtrip integrity and metric equivalence against dense implementations.
2026-08-28 19:06:39 +02:00
Eric Coissac 904d85f33b Implement pack CLI command for matrix packing and layer compaction
Adds the `pack` subcommand to `obikmer2` for persisting presence and count matrices in sparse or dense formats. Introduces the `compact_layer` module in `obikrebuild` to merge multiple partition layers into a single layer in-place via atomic operations. Updates dependencies across `obikdump`, `obikmer2`, and `obikrebuild` to resolve new local crate references. Shifts the architecture from multi-layer accumulation to in-place compaction, removing legacy rebuild modules.
2026-08-26 14:48:44 +02:00
Eric Coissac 96dfb5300b Add unitig extraction command to obikdump and obikmer2
Introduces a new unitig extraction feature that reads k-mers from an index, filters them by group metadata, and partitions the computation using rayon. The implementation constructs per-partition de Bruijn graphs, merges them, computes node degrees, and writes the resulting sequences as FASTA. A corresponding CLI command is added to obikmer2 to expose this functionality with configurable filtering and progress reporting.
2026-08-26 14:40:02 +02:00
Eric Coissac 768fa0183d Add dump subcommand and introduce reusable group filtering arguments
Introduces a new `dump` CLI command to export index k-mers as a CSV table. Adds a reusable `GroupFilterArgs` struct for ingroup/outgroup metadata-predicate quorum filtering with configurable count and fraction thresholds. Refactors the filter command to use explicit flags and a `GenomeSelector`-based pipeline, improving error handling and predicate application order. Exposes `IndexDump` from the `obikdump` crate and updates local dependencies accordingly.
2026-08-26 14:33:45 +02:00
Eric Coissac 737966899f Add obikselect crate for k-mer selection and aggregation
Introduces a new `obikselect` module to handle k-mer selection and column aggregation. Adds a `select` CLI command in `obikmer2` that supports group predicates, aggregate operators, and output column filtering. Implements operator parsing with case-insensitive matching and default rules, along with a centralized module for resolving group specifications from metadata or genome filters. Updates dependencies to include the new crate and its prerequisites.
2026-08-26 14:27:19 +02:00
Eric Coissac 354e6f9bf1 Add unified matrix builder abstraction and k-mer filtering CLI
Introduces a unified MatrixBuilder abstraction for persistent bit and integer matrices, replacing custom dispatch enums and boolean flags across consumers. Refactors index, merge, and select layers to adopt explicit merge modes and centralized lifecycle management. Adds a complete k-mer filtering implementation with partition-level processing, progress tracking, and a new CLI subcommand supporting configurable predicates and thresholds.
2026-08-26 14:19:15 +02:00
Eric Coissac b7a8b5e6cf feat: add sparse compact integer matrix implementation and utilities
Introduces `PersistentSparseCompactIntMatrix`, a row-major sparse integer matrix combining a bitmatrix support layer with dual value streams for singleton and multi partitions. Adds builder and packing utilities to convert dense count matrices into the new on-disk format, featuring idempotency guards and crash-safe persistence workflows. Exposes core accessors, iterators, and column weight calculations while adjusting internal visibility for companion modules. Includes unit tests and diagnostic examples to validate equivalence with dense representations and measure compaction metrics.
2026-08-26 14:05:54 +02:00
Eric Coissac 294f132a0a refactor: relocate partition iterator and expose graph builder API
Move the partition iterator implementation from obikdump to obikfilter, updating imports and exposing the type publicly. Add obikidxcache as a local dependency for obikfilter. In obikindexer, expose the new build_layer_from_kmers function to enable shared graph-building logic across pipelines without code duplication.
2026-08-26 09:39:02 +02:00
Eric Coissac 16ade823d6 refactor: extract partition iteration and unify group selection
Move partition iteration logic to obikdump, introducing a FilteredPartitionIter trait over IndexCache for batch-oriented scanning with configurable data retrieval and early termination. Consolidate ingroup and outgroup index storage in GroupQuorumFilter into a unified Selection struct driven by predicate matching. Update dependency manifests to include obikidxcache, rayon, and obikentropy, and remove the deprecated dump_layer module while adjusting public API re-exports.
2026-08-26 09:33:37 +02:00
Eric Coissac 881b1532b5 refactor: consolidate mmap file operations into shared helper functions
Extracts file creation, opening, and memory mapping logic into a new `mmap_file` module. This replaces repetitive unsafe operations across persistent data structures with standardized helper functions, ensuring consistent header validation, error handling, and data layout while keeping public APIs unchanged.
2026-08-26 09:26:49 +02:00
Eric Coissac ad0b7173a3 Add select0 method and expand Elias-Fano query API
Update PersistentSparseBitMatrix documentation for clarity. Introduce select0 to the rank/select module, and add next_geq, contains, successor, and predecessor methods to the Elias-Fano implementation. Include naive reference implementations and unit tests to validate correctness and boundary conditions for all new APIs. All changes are strictly additive.
2026-08-26 09:21:52 +02:00
Eric Coissac 5dc16b0127 refactor(obikselect): implement Algorithm trait for selection
Shifts the selection logic from a direct method into a dedicated `Select` struct implementing the `Algorithm` trait. Introduces a two-phase initialization pattern with explicit builder methods and closure-based progress callbacks. Updates the crate's dependency graph to include `obikalgorithm`, refactors layer processing with unified matrix dispatch, and implements dual directory routing for partition execution.
2026-08-26 09:17:48 +02:00
Eric Coissac cbf6893f38 Add configurable sparsity flag to index finalization and merge CLI
Extends `KmerIndex::finalize_indexed` to accept a `sparse` boolean parameter, enabling control over whether output presence matrices are packed in a compact sparse format or stored densely. The merge command now exposes a `--dense` flag to invert this setting, defaulting to the existing sparse representation. Call sites in select and rebuild modules pass `false` to preserve current behavior. A benchmark lock file is also added for dependency tracking.
2026-08-22 17:36:50 +02:00
Eric Coissac bb380d0c7d Add sorted_slots parameter to nonzero_triples and batch build_from_dense
Extended `nonzero_triples` with a boolean flag to conditionally skip sorting when input slots are already ordered. Updated call sites to pass appropriate flags, enabling a single-pass columnar traversal in the sparse matrix builder. Refactored `build_from_dense` into a batched processing pipeline that reduces repeated file access and improves sequential read performance. Added a diagnostic example to validate the new builder against existing dense matrices.
2026-08-22 17:33:07 +02:00
Eric Coissac 23812d1af8 feat: add col_weights API and refactor obikstats to use Algorithm trait
Added a `col_weights` method to index layers for computing per-genome column sums or presence k-mer counts. Refactored `obikstats` to implement the `Algorithm` trait with a two-phase `new`/`run` model, replacing manual layer resolution with `IndexCache` for eager file I/O. Simplified per-genome counting logic and updated public exports and dependencies accordingly.
2026-08-22 17:28:51 +02:00
Eric Coissac fba9c65b1a feat: add annotate CLI command for applying genome metadata via CSV
Introduces the `annotate` subcommand to apply genome metadata from an external CSV file to a pre-built k-mer index. The command supports configurable field separators, ID columns, and null markers, while providing a `--dump` option to export current index metadata as sorted CSV. Supporting changes include minor internal refactoring in `obikindex` to use object-level directory accessors and updates to the `csv` dependency.
2026-08-22 17:23:39 +02:00
Eric Coissac fb31a35c76 Add unitig streaming iterators and estimate CLI subcommand
Introduce `iter_unitigs` methods across the index cache, content layer, MphfLayer, and typed layer to stream whole reconstructed sequences directly from underlying storage without decomposing into k-mers. Add a corresponding low-level streaming iterator in obiskio for thread-safe, lazy reads from memory-mapped files. Include a new CLI subcommand to compute and display approximate false-positive rates based on provided indexing parameters.
2026-08-22 17:14:07 +02:00
Eric Coissac 5048f43eea Start of a huge refactoring 2026-08-22 17:03:53 +02:00
Eric Coissac 1261aeba86 feat: Add superkmer CLI subcommand to obikmer2
Introduces a new `superkmer` command that implements a multi-stage pipeline for reading nucleotide pages and constructing superkmers with configurable parameters. The implementation uses partition-aware scatter writing via `obifastwrite` to distribute output batches across workers, and adds the necessary local dependencies to the project manifest.
2026-08-22 14:08:40 +02:00
Eric Coissac 7183e3adb4 Add directory-level locking and introduce layered index cache
Introduces directory-level locking to prevent concurrent index operations from corrupting shared directories, along with explicit APIs for acquiring, probing, and releasing locks. Restructures the index cache crate to use a layered store architecture that eagerly initializes metadata and provides fast hierarchical lookups. Updates dependent modules, test suites, and CLI commands to align with the refactored API surface, and adds an end-to-end smoke test for validation.
2026-08-22 14:04:46 +02:00
Eric Coissac 2419a6c21d refactor: shift indexing algorithms to layer-centric partition APIs
Replaces manual filesystem path handling and parallel iteration with dedicated internal utilities. Introduces `par_over_layer0` and `layer0()` to abstract partition layer access, enabling partition-driven initialization and centralized progress tracking. Removes disabled or internal methods (`rewrite_config`, `open_data`) no longer part of the active interface. Updates error propagation and metadata persistence to align with the new layer-centric workflow.
2026-08-22 13:51:42 +02:00
Eric Coissac fc4464a0ef Extract index modules into specialized workspace subcrates
This commit partitions the obikindex crate into multiple focused subcrates (obikfilter, obikmerge, obikquery, obikrebuild, obikselect, obikstats, obikdump, and obikidxcache) to reduce coupling and clarify module boundaries. It standardizes error handling across the workspace using OKIError and OKIResult, updates index APIs to support lazy, disk-backed partition access, and migrates NUMA system utilities to a new obisys crate. All modifications are structural, focusing on dependency graph expansion, import path updates, and API surface reorganization without altering core runtime behavior.
2026-08-22 06:25:28 +02:00
Eric Coissac c9d10d55c7 Simplify MphfLayer::open by removing explicit IndexMode parameter
The MphfLayer::open signature now accepts only a directory path. Mode configuration is resolved internally via disk-based auto-detection using LayerEvidence, eliminating the need for explicit arguments from callers. All internal query methods and external call sites have been updated accordingly, while TypedLayer::open temporarily retains the unused parameter for future cleanup.
2026-08-21 22:08:52 +02:00
Eric Coissac 00ba968628 Introduce obikalgorithm crate to unify pipeline algorithms
Define a shared Algorithm trait with an associated Output type and a parameterless run(&mut self) method. Refactor PartitionRouter, Dereplicator, Counter, and LayerBuilder to implement the trait, standardizing receivers to &mut self and moving configuration and progress callbacks to dedicated builder setters. Decouple error handling using a generic boxed error type and update workspace dependencies accordingly.
2026-08-21 22:03:08 +02:00
Eric Coissac 8d6ba6546b refactor: migrate index metadata to on-disk JSON with fallible access
Migrate index state tracking from filesystem sentinel files to an on-disk JSON schema within `index.meta`. The `IndexMeta` struct is now wrapped in an `Arc` with internal locking, exposing only fallible methods for genome and state access. In-memory mutation capabilities have been removed, requiring callers to handle I/O errors explicitly and pass immutable references to downstream components like `PartitionRouter`. Public sentinel constants have been removed from exports.
2026-08-21 21:51:42 +02:00
Eric Coissac 02dbdd11aa Split index lifecycle into IndexBuilder trait for maintenance operations
Extracts directory bookkeeping and construction methods into a new IndexBuilder trait, separating maintenance operations from scientific computation logic. Migrates relevant methods to the new trait, updates module imports across obikindex, and narrows the crate's public API surface. Adds an end-to-end CLI smoke test to verify multi-index merge workflows.
2026-08-21 10:46:57 +02:00
Eric Coissac da3aa5a2cb refactor: extract index construction state tracking into extension trait
Moves pipeline state bookkeeping, including sentinel file marking and spectrum persistence, into the algorithms' run and close methods. Introduces a crate-private extension trait to satisfy Rust's orphan rule while implementing construction-only operations on KmerIndex. Updates helper function visibility for cross-crate access and removes the now-empty index_layer module.
2026-08-21 10:38:51 +02:00
Eric Coissac 31bb324752 Rename Layer to KmerLayer across obikindex and obikphylo
This commit renames the `Layer` type to `KmerLayer` throughout the `obikindex` and `obikphylo` crates. All imports, struct fields, function signatures, pattern matches, and instantiation calls have been updated accordingly. The change is a purely structural refactor that tightens type constraints without altering runtime behavior or data models.
2026-08-21 10:27:29 +02:00
Eric Coissac 4b7b3c3c1a Extract indexing stage into LayerBuilder with NUMA-aware scheduling
Introduce a dedicated LayerBuilder struct to orchestrate parallel layer 0 construction across partitions. The fluent API supports configurable abundance thresholds and intermediate artifact retention. Parallel execution is delegated to a NUMA-aware PartitionRunner scheduler, while progress reporting and stage timing are shifted to the command layer. A new mark_indexed method cleanly separates state tracking from orchestration by generating a completion sentinel.
2026-08-21 10:20:03 +02:00
Eric Coissac 346095b9eb Rename index methods to hash and expose kmer iteration APIs
The layer API has been updated to rename `index` and `index_batch` to `hash` and `hash_batch`, aligning with raw MPHF slot hashing behavior. New public methods have been added to directly expose kmer hashing and iteration, enforcing strict state validation on empty layers. All other modifications are consistent code formatting adjustments without functional changes.
2026-08-21 10:14:22 +02:00
Eric Coissac 0299b608e2 chore: remove CLAUDE.md and update .gitignore
Removes the file from version control and adds an ignore rule to prevent tracking.
2026-08-21 06:03:58 +02:00
Eric Coissac 878b63566f Extract k-mer counting logic into a dedicated counter module
Decouple k-mer counting from the partitioner by introducing a new `Counter` struct. The module exposes a fluent builder API with optional partial file retention, executes partition processing in parallel via Rayon with memory-aware chunk sizing, and integrates thread-safe progress callbacks. Update all callers to use the new counter, simplify test pipelines by removing serialization overhead, and clarify algorithm separation in module documentation.
2026-08-21 05:48:02 +02:00
Eric Coissac 96b6517541 refactor: rename obikindex to obikindexer 2026-08-21 05:29:34 +02:00
Eric Coissac 5c1584967f feat: enable index resumption and enforce directory creation
The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
2026-08-21 05:06:38 +02:00
Eric Coissac abc51c2add Extract dereplication logic into obikderep crate and centralize paths
Move partition dereplication logic into a dedicated `obikderep` crate that implements a two-phase hash-split/merge strategy with Rayon for parallel processing. Centralize superkmer file path construction in `obilayeredmap` and update dependent crates to use the new pipeline and shared path helpers. Adjust test suites to explicitly invoke the dereplication step.
2026-08-20 22:30:03 +02:00
Eric Coissac 616cb76af3 Refactor indexing pipeline and partition path resolution
Invert dependencies by moving partition path primitives to a dedicated module and updating the index crate accordingly. Reshape the partition router to accept a mutable index reference, enabling chainable configuration and resolving lifetime issues with explicit drops. Shift orchestration logic from the index crate to the CLI, replacing monolithic scatter calls with discrete dereplication and counting steps. Introduce a generic progress callback API to decouple rate calculation from UI rendering, and correct file I/O paths to route layer-0 artifacts under the partition index directory.
2026-08-20 22:15:13 +02:00
Eric Coissac 164e879585 feat: introduce obikpartition crate for kmer partition layer lookup
Introduces the `obikpartition` crate containing the `KmerPartition` struct to manage open layers in sequential order. The `open` constructor eagerly initializes layers under a specified directory, while the `find` method returns the index of the first layer containing a given k-mer. Dependencies are strictly scoped to `obilayeredmap` and `obikseq`, with unit tests validating layer ingestion and single-kmer lookup behavior. Batch lookups and further migration are deferred to a subsequent phase.
2026-08-20 20:47:20 +02:00
Eric Coissac c4b69e1af5 Introduce lifecycle-aware Layer::Empty variant and update callers
The `Layer` enum is transformed into a lifecycle-aware state machine with an `Empty` variant representing an unconstructed directory. Read and query operations now explicitly panic when invoked on this state, enforcing explicit progression through `create()` before use. Iterator methods are updated to handle the new variant exhaustively, and module visibility constants are adjusted to support the refactored structure.
2026-08-20 20:41:48 +02:00
Eric Coissac 1c54e60c9a Rename Layer<D> to TypedLayer and introduce heterogeneous Layer enum
Renames `Layer<D>` to `TypedLayer<D>` to establish a distinct typed abstraction. Introduces a new heterogeneous `Layer` enum that unifies count and presence storage with runtime dispatch, delegating operations to the underlying typed variants. Updates cache, index, and phylo consumers to align with the renamed type and new extension traits, preserving full test suite stability while preparing the foundation for multi-partition caching.
2026-08-20 20:35:29 +02:00
Eric Coissac 6c860f120f refactor: merge KmerPartitions into KmerIndex and rename obikpartition
Consolidates partition logic, metadata storage, and layer management directly into KmerIndex. Renames obikpartitionner to obikpartition, retaining only PartitionRouter for superkmer routing. Removes intermediate .partition() accessors in favor of direct methods on the index and updates PartitionCache::build to accept &KmerIndex directly. Derives n_partitions from config.n_bits and consolidates k-mer/minimizer sizes into IndexMeta.config. Fixes a regression where PartitionRouter::open incorrectly defaulted to closed.
2026-08-20 20:23:58 +02:00
Eric Coissac b5ec0122d0 Rename obikpartition crate to obikpartitionner
Update all dependent crates and internal imports to reference the renamed module. Expose configuration parameters and genome metadata through new public accessor methods in the index API, and update documentation comments to align with the new package name.
2026-08-20 19:43:00 +02:00
Eric Coissac f64ac14f0e chore: make COUNTS_DIR constant pub(crate)
Adjusts the visibility of the COUNTS_DIR constant to pub(crate), enabling internal crate-wide access without altering runtime behavior or data structures.
2026-08-20 16:42:43 +02:00
Eric Coissac 9379bbaad8 refactor: rename obikpartitionner crate to obikpartition
Update Cargo manifests, dependency paths, and Rust imports across the workspace to reflect the `obikpartitionner` to `obikpartition` rename. Synchronize architecture and implementation documentation with the new module structure. Fix minor syntax issues in test assertions to ensure compilation compatibility. No behavioral or API changes are introduced.
2026-08-20 16:41:22 +02:00
Eric Coissac f9ef6b8391 rename KmerPartition to KmerPartitions and update Mat enum
Rename the KmerPartition type to KmerPartitions across obikindex, obikpartitionner, and obikphylo/siblings to reflect an updated data model. Update the Mat enum in siblings/cache.rs to add a SparsePresence variant and simplify opening logic by delegating sparse versus dense detection to PersistentBitMatrix. Apply consistent code formatting, import reordering, and multi-line refactoring throughout the affected modules.
2026-08-20 15:57:33 +02:00
Eric Coissac 76cbd3a886 Centralize partition metadata access and add layer introspection APIs
Replaced scattered direct metadata loading with centralized instance methods on `KmerPartition` to guarantee consistent error mapping and legacy recovery. Introduced `StorageKind`, `LayerContent`, and `EvidenceKind` enums alongside lightweight disk-probe methods that inspect file presence without opening heavy data structures. Updated callers across the index, partitioner, and phylo modules to use the new partition API, and added unit tests validating the introspection behavior.
2026-08-20 15:29:53 +02:00
Eric Coissac f7ebc7a1ab Centralize partition directory resolution and add caching design spec
Introduce a design specification outlining performance bottlenecks in layer data access and an agreed-upon implementation direction for caching. Refactor the codebase to centralize index and layer directory path resolution within the partition object, replacing manual string joining and external helper functions with dedicated accessor methods.
2026-08-20 15:10:57 +02:00
Eric Coissac 7eaa8c2016 Replace manual layer path construction with obilayeredmap::layer_dir
Standardizes layer directory path resolution across obikindex, obikpartitionner, and obikphylo by replacing inline string formatting and join calls with the centralized obilayeredmap::layer_dir utility. This refactoring removes redundant path construction logic while preserving existing iteration bounds, control flow, and public APIs.
2026-08-20 14:54:13 +02:00
Eric Coissac 5a91817488 centralize layer path construction and data loading
Replace manual directory formatting and inline matrix opening with the newly introduced `layer_dir` and `open_data` helpers from the `obilayeredmap` crate. This standardizes filesystem access, encapsulates layer naming conventions, and simplifies error handling without altering computational behavior or test suites.
2026-08-20 14:43:10 +02:00
Eric Coissac 640f29725b Optimize sparse bit matrix packing to avoid intermediate copies
Refactor `pack_sparse_bit_matrix` to conditionally open either a packed or columnar dense representation based on file existence. This eliminates the previous unconditional pre-conversion step that forced a full packed copy. Update cleanup logic and add tests to verify direct transposition, proper file handling, and idempotency for both formats.
2026-08-20 14:38:20 +02:00
Eric Coissac c2e0533fa9 Refactor bit matrix operations and optimize distance computations
Consolidate row extraction and parallel column reduction into reusable generic utilities within the pairwise module. Replace manual iteration in builder and view methods with iterator-based zipped loops and deferred overflow processing to improve memory access patterns. Optimize distance functions by switching to bulk byte passes for branch-free SIMD vectorization, adding a merge-like helper to correct masked overflow values without secondary allocations. Add a comprehensive test validating the optimized distance paths against naive references across multiple thresholds and overflow scenarios.
2026-08-20 14:35:22 +02:00
Eric Coissac 9abee87af3 refactor: delegate distance computations and simplify matrix logic
Consolidate bit matrix operations by delegating pairwise metric calculations to IntSliceView and col_view implementations. Introduce chunked_presence_count in colgroup to handle threshold-based accumulation efficiently. Add a direct point lookup method to the sparse matrix representation to eliminate buffer allocations, and simplify match arms in persistent accessors accordingly. All public APIs and behavioral contracts remain unchanged.
2026-08-20 14:27:32 +02:00
Eric Coissac 19f9954050 refactor: centralize nonzero slot iteration logic across views
Extract duplicated traversal logic into a shared `NonzeroSlotsView` trait and `nonzero_triples` helper. Update slice view iterators to consistently yield `(position, value)` tuples and use a temporary buffer with explicit permutation mapping for sorted access. Delegate manual slot sorting and column iteration to the new shared helper, eliminating per-format duplication and eager collection while preserving existing public API signatures.
2026-08-20 14:20:24 +02:00
Eric Coissac 82374deca5 refactor: simplify sub_matrix with iterator-driven column population
Refactored `sub_matrix` to delegate column population to `fill_sub_matrix`, replacing manual buffer allocation and explicit permutation loops with direct value assignment via `enumerate_slots_values`. This eliminates intermediate allocations, reduces pipeline overhead, and simplifies control flow while preserving slot ordering and permutation semantics.
2026-08-20 14:17:05 +02:00
Eric Coissac ac38aa759b refactor: rename batch retrieval methods and update common_traits
Renames `get_batch`, `fill_batch`, and `fill_batch_sorted` to `collect_slots_values`, `fill_slots_values`, and `fill_slots_values_sorted` to align with updated `common_traits 0.13` APIs. Introduces optimized batch retrieval that sorts input indices for sequential mmap access before reordering outputs to match the original query order. Updates test suites to reflect the new method signatures without altering validation logic or coverage.
2026-08-20 14:15:28 +02:00
Eric Coissac 82ddeaddcd Introduce unified nonzero_iter API across matrix types
Replaces nested column-major point lookups with a batched iterator that delegates to format-native traversal strategies. The implementation enforces a single pass per matrix type, using row-major iteration for sparse formats and eager collection for packed/columnar layouts while preserving original slot ordering. Memory allocation is optimized by removing `n_cols`-wide buffers in favor of per-row buffering or lazy iteration. Correctness tests verify iterator output against dense baselines across all supported layouts, and architecture documentation is updated to reflect the new format-agnostic query pattern.
2026-08-20 14:07:09 +02:00
Eric Coissac a4eb20e67e add some doc about optimisation for query 2026-08-20 13:59:12 +02:00
Eric Coissac 89ea077456 Add benchmark pipeline for dense and sparse query testing
Introduces a complete query benchmark track to evaluate performance and verify consistency between dense and sparse index formats. Adds scripts to simulate fixed-size paired-end reads, pack a sparse presence index, execute queries in both modes, and capture wall time and RSS metrics. Includes a verification step that compares outputs by read ID to ensure content identity across parallel processing. Updates build configuration, documentation, and ignore patterns to support the new pipeline for two microbial specimens.
2026-08-20 13:59:12 +02:00
Eric Coissac 5a9d903e51 chore: update dependencies and adapt to updated crate APIs
Bumps core dependencies including ndarray, rand, hashbrown, niffler, ureq, sysinfo, indicatif, lru, and remove_dir_all. Adapts source code to accommodate breaking changes by migrating RNG initialization, adjusting HTTP response handling, and replacing the fs4 crate with standard library file locking. Adds a planning document for query benchmarking and sparse index regression tests.
2026-08-20 13:45:41 +02:00
Eric Coissac 32bcbd1465 chore(deps): update dependencies and clean up imports
Updated core libraries and ecosystem packages, including the full serde suite and regex dependencies. Restructured syn dependency resolution by pinning 2.0.117 while introducing a standalone 3.0.3 entry for derive macros. Downgraded windows-sys to 0.59.0 and replaced the anes dependency in criterion2 with bpaf and walkdir. Removed an unused IntSliceView import from test modules.
2026-08-20 13:25:35 +02:00
Eric Coissac 0da725ffe9 refine k-mer index architecture documentation and remove obsolete spec
Introduces raw mapping and iteration APIs that bypass membership checks, clarifies variant-specific storage layouts and auto-detection logic, and documents optimized batch access patterns with caller-provided buffers. Removes the outdated obicompactvector_reflexion.md specification to consolidate architectural details into current implementation docs.
2026-08-20 13:20:29 +02:00
Eric Coissac 3b65319529 Remove precomputed data assets, configurations, and documentation
This change removes precomputed model parameters, phylogenetic tree datasets, k-mer spectrum data, and compressed profile archives. It also deletes runtime logs and detailed pipeline documentation. The repository ignore list is updated to exclude the sandbox directory.
2026-08-20 13:15:56 +02:00
Eric Coissac 69747dcb53 chore: add editor and memory directory patterns to .gitignore 2026-08-20 12:50:54 +02:00
Eric Coissac 2dba217482 chore: remove src/profile.json.gz
Removes the compressed JSON archive containing profile configuration and data model definitions. Build processes and runtime loaders referencing this path will require updates to prevent missing file errors.
2026-08-20 12:49:48 +02:00
Eric Coissac dbd8af376c Consolidate compare_sparse as example and clean up project artifacts
Restructure the project by moving the standalone compare_sparse utility into an example directory, removing Sankoff parameter configurations and benchmark scripts, updating version control ignores, and expanding the test suite with diagnostic checks and performance benchmarks.
2026-08-20 12:48:42 +02:00
Eric Coissac dc3d82f8db Add resume API to persistent matrix builders for incremental appending
Introduce a `resume` method for persistent bit and int matrix builders to restore dimensions from persisted metadata. This enables incremental column appending across separate build sessions without requiring manual dimension tracking. Consolidate builder lifecycle management in the merge layer using a unified `MatrixBuilder` enum, simplify closure logic, and add instrumentation. Include tests verifying state preservation and data integrity across multiple resume cycles.
2026-08-20 12:36:47 +02:00
Eric Coissac 308d2b9f92 chore(deps): bump obikmer dependency to 1.2.2
Update Cargo.lock to track the patch release of obikmer. This modification only adjusts the version in the lockfile and introduces no behavioral or API changes.
2026-08-20 12:27:26 +02:00
coissac 0f389b37f2 Merge pull request 'chore: bump obikmer to 1.2.1 and disable obikindex default features' (#69) from push-vzoxtyuuuxqn into main
Reviewed-on: #69
2026-08-17 11:33:56 +00:00
Eric Coissac 66ab1d0947 chore: bump obikmer to 1.2.1 and disable obikindex default features
Release / create-release (push) Successful in 2m25s
ci.yml / build (pull_request) Successful in 4m34s
Release / build-linux-x86_64 (push) Successful in 8m18s
Release / build-macos-arm64 (push) Successful in 1m55s
Updates the obikmer lockfile version to 1.2.1 and configures the obikindex dependency to explicitly disable default features. These are manifest and lockfile adjustments that do not modify source code or alter application behavior.
2026-08-17 13:32:30 +02:00
coissac 5e2bb393e0 Merge pull request 'Push luqxvxskktxv' (#68) from push-luqxvxskktxv into main
Reviewed-on: #68
2026-08-17 10:35:12 +00:00
Eric Coissac fceb523f1a chore: disable default features for obikindex
Release / create-release (push) Successful in 2m28s
ci.yml / build (pull_request) Successful in 4m41s
Release / build-linux-x86_64 (push) Successful in 8m23s
Release / build-macos-arm64 (push) Failing after 2m4s
Explicitly sets `default-features = false` for the `obikindex` dependency to ensure no default features are activated during dependency resolution and compilation.
2026-08-17 12:33:59 +02:00
Eric Coissac 700eaeaed2 test: simplify k-mer test setup for updated canonical API
Adapts sibling k-mer tests to the updated canonical API by replacing string-based construction with direct byte conversion. Removes intermediate reverse complement steps and introduces a helper function to compute reverse complement strings directly from canonical k-mers, preserving existing test output while streamlining setup.
2026-08-17 12:32:44 +02:00
coissac f1f7940277 Merge pull request 'Push zpwxxpnpktps' (#67) from push-zpwxxpnpktps into main
Reviewed-on: #67
2026-08-17 09:41:41 +00:00
Eric Coissac 6def18fa88 Add TNT configuration directives and tree export commands to script
Release / create-release (push) Successful in 2m28s
ci.yml / build (pull_request) Failing after 3m31s
Release / build-linux-x86_64 (push) Successful in 8m29s
Release / build-macos-arm64 (push) Failing after 1m49s
Updates the generated TNT script with configuration parameters and appends tree management commands for saving and exporting results. Modifications are strictly limited to string literals written to the output stream.
2026-08-17 11:39:51 +02:00
Eric Coissac 70b0527957 Add diagnostic logging and minor internal refactoring
Introduce runtime dimension tracking, matrix shape validation, and per-layer diagnostics across multiple modules to improve execution observability. Extract intermediate computation results into local variables, standardize collection patterns in partial methods, and replace naive iteration with optimized pairwise counting. All modifications are strictly additive or structural; public APIs, data models, and core logic remain unchanged.
2026-08-17 11:29:31 +02:00
Eric Coissac 11476bc557 Add sparse bit matrix support and enhance dimension logging
Introduce a `Sparse` variant for `PersistentBitMatrix` with full method dispatch and implementations of `ColumnWeights` and `BitPartials` traits. Refactor distance matrix computations in layered stores to eagerly collect results before reduction, enabling reliable dimension tracking. Add comprehensive debug logging across phylo commands and distance modules to report matrix shapes and flag out-of-bounds indices during CSV iteration.
2026-08-17 11:24:23 +02:00
Eric Coissac dbb969f087 style: reformat iqtree module for line-length compliance
Apply consistent multi-line formatting to iterator chains, struct initializations, function signatures, and CLI string literals. Convert sequence vector declarations to single-line format while expanding assertions and variable initializations across multiple lines. Reorder imports in the sankoff module and align test fixtures with updated line-length constraints. This change is purely syntactic with no functional or behavioral impact.
2026-08-17 11:19:44 +02:00
Eric Coissac 1f9c6388eb Introduce compare_sparse CLI tool to verify index consistency
Adds a new binary that validates bit-level consistency between dense and sparse K-mer index representations by sampling slots across partitions and reporting discrepancies. Refactors sibling cache matrix initialization into a centralized factory method to simplify control flow and standardize error handling. Introduces diagnostic configurations, benchmark scripts, and an ignored test case to support k-mer resolution analysis.
2026-08-17 11:16:42 +02:00
Eric Coissac 128db64564 feat(phylo): add --iqtree-min-freq to filter rare nucleotide states
Introduces --iqtree-min-freq (default 0.001) to treat low-frequency nucleotide states as missing data during IQ-TREE alignment generation when --free-loss is active. This triggers a recoding pass that folds rare states into the missing symbol, followed by non-informative site removal and alphabet recomputation to maintain output consistency. The change also adds Sankoff model configuration files and updates related tests and documentation.
2026-08-17 11:06:03 +02:00
Eric Coissac c8f2b16b4c fix: prevent probability underflow in pairwise cost matrix
Replaces premature exponentiation-based row normalization with log-sum-exp arithmetic to prevent tiny probabilities from collapsing to exactly zero. This eliminates spurious infinite costs for valid but rare transitions while preserving correct IEEE 754 semantics for genuinely unobserved pairs. Adds explicit guards against NaN in degenerate rows and includes a regression test verifying finite costs for probabilities as low as 1e-200.
2026-08-17 09:41:50 +02:00
Eric Coissac 9654201885 feat: introduce _iqtree_states.csv for compact symbol mapping
Generates a new CSV output that maps IQ-TREE's compact state symbols to canonical states alongside full-precision empirical frequencies. Updates documentation to clarify that state frequencies sum to 1.0 by design and documents conditional behavior under `--free-loss`. Includes unit tests verifying absent state exclusion, frequency summation, and CSV structure. Also restricts entropy annex resolution to non-monomorphic minorants to eliminate redundant per-genome checks.
2026-08-17 09:38:26 +02:00
Eric Coissac c6cfdac043 perf: optimize entropy computation by pre-filtering monomorphic minorants
Shift monomorphism filtering from the entropy scan layer to a lightweight, annex-only pre-pass. By replacing `Selection::All` with a pre-filtered subset, expensive per-genome resolution is strictly limited to non-monomorphic families. This avoids processing ~98% of minorants that are known to be monomorphic, while preserving positional read speedups for subsequent runs. The change is an internal performance refinement with no public API modifications.
2026-08-17 09:32:45 +02:00
Eric Coissac 7bac0f3850 large refactoring 2026-08-17 09:28:53 +02:00
Eric Coissac 49e66f16a2 docs: document architecture analysis and refactoring plans for siblings
Documents architectural analysis, identified performance bottlenecks including hardcoded selection flags and redundant full-index scans causing I/O-bound stalls. Details planned pipeline refactoring to unify stages around a single shared selection for a fused single-pass scan, noting future dependencies on entropy-biased family selection.
2026-08-16 21:54:07 +02:00
Eric Coissac 53c40b7a53 refactor: format tnt script generation and add tree export commands
Reworks `writeln!` macro invocations to multi-line syntax and adjusts whitespace for improved readability. Additionally, appends four commands to the generated phylogenetic script to explicitly export trees and manage taxon naming at runtime.
2026-08-16 21:52:30 +02:00
Eric Coissac 3ba26b3dc1 feat: add sparse on-disk format for presence matrices
The index packing API now accepts a `sparse` parameter to generate `PersistentSparseBitMatrix` files alongside existing dense matrices. The sibling cache automatically detects this format via an `is_multi.prsb` marker file and routes queries identically to the dense variant. A new `--sparse` CLI flag exposes the option, with tests verifying end-to-end pipeline correctness and storage equivalence.
2026-08-16 21:51:02 +02:00
Eric Coissac 50f4820cb9 feat(obicompactvec): introduce sparse bit matrix with supporting primitives
Implements a compact, row-major sparse bit matrix backed by memory-mapped components, introducing EliasFano, PersistentFixedIntVec, and PersistentRankSelectBitVec primitives for efficient storage and decoding. Adds a BinaryMatrix trait to unify row-level operations across dense and sparse implementations. Corrects edge-case behaviors for zero-width bit storage and cardinality-0 rows. Delivers reduced on-disk size and faster random row access, with column reads remaining dense-only. Test suites and benchmarks are included but currently marked as ignored.
2026-08-16 21:43:09 +02:00
Eric Coissac 45b19503a1 Add proportional subsampling and Shannon entropy calculation
Introduces `--subsample N` and `--shannon` CLI flags to cap retained variable families via proportional reservoir sampling and compute per-family Shannon entropy. Updates the family scanning API to support explicit selection filtering with early-exit optimization, resolving an indexing drift issue in monomorphic layers. Streams entropy metrics for 15-state and 4-nucleotide spaces directly to CSV while maintaining parallel processing across sibling layers.
2026-08-16 21:31:06 +02:00
Eric Coissac f5e4bbfc6b Document architecture redesign and add partition layer accessor
Documents a proposed redesign for cross-partition batch resolution, shifting trigger logic to per-destination accumulator thresholds and introducing entropy-based pruning criteria. Adds an `n_layers_per_partition` method to the index, exposing partition metadata with consistent error handling and clarified documentation regarding build-time structural properties.
2026-08-16 21:20:55 +02:00
Eric Coissac 151493526c perf: add #[inline] attributes to obicompactvec methods
Adds compiler inlining hints to accessors, iterators, bitwise operations, and distance functions across multiple modules. This optimization aims to reduce call overhead for frequently invoked methods without modifying runtime behavior, API contracts, or data models.
2026-08-16 21:18:58 +02:00
Eric Coissac 693c18bfa7 introduce fast mode for optimized sibling presence checks
Centralize the layer count validation into PartitionCache and track it via a new fast_mode flag. Extend query tuples to include a pre-resolved destination layer index, enabling a fast-path batch lookup that bypasses per-layer probing when enabled. Refactor neighbor iteration and hit resolution to eliminate duplication and conditionally dispatch to the optimized path based on the cache state.
2026-08-16 21:12:05 +02:00
Eric Coissac 0ce934b111 Refactor sibling annex to use mmap-backed concurrent storage
Shift the sibling annex construction pipeline from an in-memory atomic mask to a memory-mapped file backend. This enables lock-free concurrent writes directly into the mapped region, streamlining the two-phase write process to accumulate bits atomically before finalization. Adjusted the cache lookup to return the specific matching layer index rather than a boolean flag, and added tests to verify correct layer tracking and cross-partition resolution in merged indexes.
2026-08-16 21:07:39 +02:00
Eric Coissac fecfe84ea6 Move sibling annex implementation to obikphylo siblings module
Relocates `SiblingAnnex`, `FamilyMask`, and `SiblingAnnexBuilder` from `obicompactvec` to the local `obikphylo::siblings` module. Replaces the previous implementation with a memory-mapped version using `memmap2`, featuring a 2-byte-per-slot layout, explicit bitfield manipulation, and support for concurrent atomic writes. Updates all sibling module imports to local paths and adds the `memmap2` dependency to `obikphylo`.
2026-08-16 21:04:14 +02:00
Eric Coissac c990087ef3 Add execution timing and parallelize sibling stats
Instruments the phylo command pipeline with structured execution timing, wrapping major computational blocks with stage hooks and printing aggregated metrics upon completion. Additionally, parallelizes sibling counting logic using Rayon to process independent layer directories concurrently, preserving identical functionality and public API contracts.
2026-08-16 20:53:22 +02:00
Eric Coissac 32d6720f50 Fix batch enumeration offsets and refactor sibling annex construction
Shifts sibling annex construction from slot-indexed enumeration to iteration-order traversal by correcting cumulative k-mer offset tracking in batch enumeration. Replaces coarse per-partition parallelism with chunked work distribution to prevent thread starvation on skewed partitions. Decouples custom progress messages from ETA updates to eliminate display clobbering during high-frequency callbacks. Adds regression tests validating batch offset correctness, partial batch handling, and iterator-order consistency across layer builds.
2026-08-16 20:51:19 +02:00
Eric Coissac d2548e8c33 feat(progress): implement configurable ETA throttling
Introduces timing constants and atomic fields to control ETA calculation intervals. Replaces the static template placeholder with dynamic messages, delegating formatting to a new helper that applies throttling thresholds and suppresses automatic updates during custom message hold periods.
2026-08-16 14:32:25 +02:00
Eric Coissac 5997de6707 Extract phylogenetic sibling logic into new obikphylo crate
Relocate the `siblings` and `cardcomp` modules from `obikindex` to a dedicated `obikphylo` workspace member. Convert inherent methods on `KmerIndex` into extension traits, update import paths across `obikmer`, and add supporting accessor methods to `obikseq` and `obilayeredmap`. This restructuring reduces the public API surface of `obikindex` while organizing phylogenetic iteration, caching, and distance calculation logic under a dedicated crate.
2026-08-16 14:30:34 +02:00
Eric Coissac 519195d4a1 Replace slot-based indexing with iteration order and stream k-mers
Transitions the index from MPHF slot-based to physical iteration-order indexing, aligning with the unitig layout. Introduces a streaming-only pipeline for k-mer iteration that adheres to memory constraints by avoiding full in-memory collections. Updates layer and sibling iterators to own an Arc clone of the file reader, making them Send + 'static and safe for concurrent use without borrowing the parent. Exposes batch and k-mer iterator types publicly while simplifying signature syntax with modern lifetime elision.
2026-08-16 14:07:22 +02:00
Eric Coissac b1f54b7d2f Add cache-optimized batch retrieval and sub-matrix methods
Introduces batch retrieval and sub-matrix extraction methods across vector, view, reader, and matrix types. These implementations optimize cache locality by sorting requested indices for sequential memory access before applying an inverse permutation to restore original order. Includes allocation-free variants that populate caller-provided buffers. Updates architecture documentation to define sibling annex persistence in iteration order and clarify pipeline separation.
2026-08-16 14:01:35 +02:00
Eric Coissac dae543fdfc Add raw lookup and iteration APIs to Layer struct
Introduces `index` and `index_batch` methods for direct MPHF slot mapping without membership validation, alongside four public iterator methods for deterministic traversal of canonical kmers. These are backed by dedicated `KmerIter` and `KmerBatchIter` structs that wrap the underlying unitig file reader. Updates `LayerEvidence::Approx` to eagerly open the unitig reader during initialization, enforcing a clear separation between raw mapping and verified lookup workflows.
2026-08-16 13:54:09 +02:00
Eric Coissac 0de078fdf1 Store minorant flag in family mask to avoid costly k-mer reconstruction
Transition the minorant flag from a derived value to a stored field within the family mask, resolving a performance regression where on-the-fly reconstruction consumed significant query time. This change introduces O(1) k-mer reconstruction APIs, shifts minorant computation to the index build phase, and enables direct annex-based statistics. Supporting updates include adopting shared ownership for partition caches and refactoring batch processing pipelines.
2026-08-16 13:50:44 +02:00
Eric Coissac c7679fac90 refactor: replace eager family collection with callback processing
Refactor `scan_layer_families` across the siblings module to accept a closure callback instead of returning an intermediate collection. This eliminates eager materialization and per-layer buffering by streaming results directly into genome-specific buffers or tally matrices. The update introduces bounded batch processing and scratch buffer reuse to cap peak auxiliary memory, while preserving existing computational behavior, control flow, and error semantics.
2026-08-16 13:36:20 +02:00
Eric Coissac 0e2e3b5bae Switch sibling modules to sequential layer directory processing
The sibling calculation modules now process layer directories sequentially instead of in parallel. This eliminates concurrent processing overhead and prevents interleaved cache sweeps, improving disk I/O and page-cache locality for partition-grouped data access. Progress bar updates and result accumulation have been adapted to the sequential control flow, while core filtering logic and output structures remain unchanged.
2026-08-16 13:30:30 +02:00
Eric Coissac cd57cf0cbd refactor: extract sibling family scanning into shared module
Introduces a dedicated `family_scan` submodule to consolidate per-layer family traversal logic. Centralizes path validation, minorant filtering, and conditional matrix instantiation into a shared `scan_layer_families` function. Updates sibling-annex consumers to leverage the new abstraction, reducing inline scanning code. Adds a test fixture to verify numerical consistency across consumer methods for co-occurrence and base-pair metrics.
2026-08-16 13:25:25 +02:00
Eric Coissac 3da501349b docs: clarify phylogenetic output files and tool usage
Restructure the output files section into categorized subsections with tables. Add explicit mappings between command-line options and generated files. Define CSV matrix conventions, clarify mathematical formulas for distance calculations, and document execution commands for external phylogenetic tools.
2026-08-16 13:17:51 +02:00
Eric Coissac d54ae272a4 refactor: centralize index setup logic and add progress bar ETA
Extracts directory cleanup, partition initialization, and finalization into dedicated helper methods within KmerIndex. This centralizes force-flag handling and reduces boilerplate across merge, rebuild, and select workflows. Additionally updates the CLI progress bar template to display an ETA indicator following elapsed time.
2026-08-16 13:15:59 +02:00
Eric Coissac eee71430a4 add name-tree command and fix --free-loss cost matrix
Introduce the obikmer name-tree subcommand to map numeric leaf labels in phylogenetic tree exports back to taxon names using a reference FASTA file. Correct the --free-loss flag behavior by removing cardinality transition costs from pairwise cost calculations, ensuring sibling gains and losses are priced identically to whole-family events. Update documentation, configuration parameters, and add reference phylogenetic data files.
2026-08-16 13:11:06 +02:00
Eric Coissac 8615da59a8 Add phylogenetic CLI options for family overlap and missing data
Introduces CLI flags for computing pairwise family overlap matrices and filtering genomes below a shared family threshold. Adds a free-loss mode that recodes locus non-detection states to missing data symbols in Sankoff-calibrated alignments, resolving ascertainment bias handling for IQ-TREE. Updates empirical transition parameters, removes the legacy model asset, and extends output writers for CSV diagnostics, FASTA pseudo-alignments, and Newick trees.
2026-08-16 11:54:43 +02:00
Eric Coissac e2b9374426 fix: respect --force flag during index and partition creation
The index creation routine now uses the CLI `--force` argument instead of a hardcoded false value, enabling explicit overwrite control. Partition existence checks also verify for the designated subdirectory rather than the root path, ensuring conflict detection and cleanup only trigger when an actual partition layout exists.
2026-08-16 11:37:26 +02:00
Eric Coissac dd889854cb rename distance subcommand to phylo
Rename the distance CLI subcommand to phylo across the codebase, documentation, and build configurations. Relocate source files from cmd/distance/ to a dedicated cmd/phylo/ module, update all internal routing references, and adjust benchmark scripts and Makefile targets to reflect the new command name.
2026-08-15 10:07:03 +02:00
Eric Coissac 79346c0c86 Add modular data structures, parallel pipelines, and system profiling
Establishes foundational infrastructure across multiple crates by introducing unified persistent bit matrix storage with columnar, packed, and implicit variants, alongside De Bruijn graph node encoding and unitig iteration logic. Adds a macro-driven parallel pipeline scheduler featuring NUMA-aware runners, bounded channels, and memory budgets to enforce concurrency limits. Implements streaming nucleotide parsers with pooled page buffers for FASTA, FASTQ, and Genbank formats, complemented by system resource monitoring, progress tracking, and stage profiling utilities. Collectively, these changes provide the core data models, execution frameworks, and I/O pipelines required for downstream k-mer indexing and analysis workloads.
2026-08-14 14:20:08 +02:00
Eric Coissac cc67023e2c feat: centralize genome metadata predicates in obikindex
Introduces a new predicate module in obikindex that implements genome metadata predicate parsing, evaluation, and group classification using three-valued logic. Extends the IndexMeta API with methods for single-predicate filtering and group quorum filtering. Updates obikmer command modules to delegate filter construction and matching to the centralized index API, removing local definitions and simplifying call sites.
2026-08-13 18:56:02 +02:00
Eric Coissac 8967a20ed7 refactor(cmd): restructure modules and decompose query command
Convert single-file modules to directory-based layouts across the cmd crate. Decompose the monolithic query command into dedicated submodules for batching, chunk processing, sparse finding, and output formatting. Introduce a new utils module to handle index management operations including statistics reporting, label renaming, and partition analysis.
2026-08-13 17:55:58 +02:00
Eric Coissac 6acafa7f2c docs: add obikmer user guide and MkDocs build configuration
Introduces a comprehensive documentation set covering theoretical foundations, CLI usage, installation, and system architecture. Adds MkDocs configuration and Makefile targets to generate, serve with live reload, and clean the documentation site. Includes citation styles and bibliography files for academic references.
2026-08-13 17:19:01 +02:00
Eric Coissac 0c86ea0385 Replace Sankoff cost matrix with cardinality-composition decomposition
Replaced the legacy Sankoff parsimony pipeline with a new cardinality-composition decomposition that constructs row-normalized transition probability matrices symmetrized via geometric mean. This ensures reversibility, reduces free parameters from 240 to 120, and guarantees a zero diagonal. Tallies are now explicitly restricted to variable families to align with +ASC-corrected alignment populations. Additionally, fixed `--exclude-genome` handling to re-scan surviving sequences and drop newly monomorphic columns, preventing silent data corruption in downstream tree inference tools.
2026-08-13 16:58:12 +02:00
Eric Coissac c26623fa00 Add repeatable --exclude-genome flag to obikmer distance command
Integrate in-memory row/column zeroing and alignment filtering across SNP, Sankoff, TNT, and IQ-TREE output paths. Add strict validation for missing labels, implement `write_iqtree` with empirical stationary frequencies, and introduce `--raw-snp-counts` diagnostic CSV output. Update documentation to reflect experimental validation of backbone resolution limits and theoretical considerations for CTMC rate matrices.
2026-08-13 16:36:02 +02:00
Eric Coissac 28d841c7be feat(distance): add IQ-TREE output and optimize state index mapping
Introduce `--iqtree` and `--raw-snp-counts` flags to generate IQ-TREE model files, recoded FASTA alignments, and per-pair diagnostic counts. Centralize alphabet conversion by extracting a precomputed state index lookup table into the Sankoff module, eliminating redundant iterations across downstream adapters.
2026-08-12 20:05:10 +02:00
Eric Coissac adf5b52dc7 feat(distance): implement native Sankoff calibration and backends
Replaces external Python glue with native Rust modules for Sankoff model calibration, exporting calibrated cost matrices, FASTA alignments, and YAML parameters. Adds dedicated writers for TNT and PhyG that apply integer scaling and Floyd-Warshall metric closure to enforce triangle inequality. Integrates these exporters into the distance command pipeline to streamline downstream tree inference workflows, while updating theory documentation to reflect IQ-TREE integration and state renumbering improvements.
2026-08-12 20:05:10 +02:00
Eric Coissac 55d7fa2067 feat: add Sankoff parsimony model and directory locking
Implements a calibrated 16-state Sankoff substitution cost matrix and CLI pipeline for evolutionary distance computation, including empirical calibration via saturation-filtered SNP counts. Refactors the sibling scanning stage to use batched transforms for improved synchronization efficiency. Introduces an OS-level advisory directory lock across all index-modifying commands to prevent concurrent write corruption. Updates dependencies and exposes new Sankoff utilities in the public API.
2026-08-12 20:05:10 +02:00
coissac 14aa82521d Merge pull request 'fix: resolve test race conditions, add logging, and fix CI deadlock' (#66) from push-kywuzlnvrqyx into main
Reviewed-on: #66
2026-08-11 21:05:09 +00:00
Eric Coissac c95c47155e fix: resolve test race conditions, add logging, and fix CI deadlock
Release / create-release (push) Successful in 2m26s
ci.yml / build (pull_request) Successful in 4m4s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 1m48s
Re-enables the `numa` feature in CI workflows to prevent container/cgroup deadlocks while preserving validation correctness. Fixes concurrent test race conditions by replacing thread-local parameter storage with process-wide atomics and mutex locks. Integrates `tracing-subscriber` for structured logging and adds thread-ID tracking to debug worker lifecycles. Additionally bumps the crate version, updates `.gitignore`, documents experimental evolutionary distance pipelines, and refactors hardcoded test constants.
2026-08-11 23:04:06 +02:00
coissac 4f6d442688 Merge pull request 'ci: disable numa feature, bump obikmer, and document Sankoff costs' (#65) from push-oruynkvporsn into main
Reviewed-on: #65
2026-08-11 16:28:07 +00:00
Eric Coissac e6f0ca472c ci: disable numa feature, bump obikmer, and document Sankoff costs
Release / create-release (push) Successful in 2m28s
ci.yml / build (pull_request) Failing after 3h0m41s
Release / build-linux-x86_64 (push) Successful in 8m31s
Release / build-macos-arm64 (push) Successful in 1m53s
Disable the `numa` default feature in CI build and test steps to prevent container environment deadlocks, and add comments explaining the cache key salt bump (`v2`) to mitigate incremental compilation corruption. Document a 16-state Sankoff cost matrix derived from set-edit distances, including substitution, gain/loss, and context-disappearance costs compatible with TNT's interface. Bump `obikmer` crate version to 1.1.43.
2026-08-11 18:26:54 +02:00
coissac 442f7a9e4c Merge pull request 'chore: update ci cache, document distance metrics, and bump version' (#64) from push-wpxsvyylwmsq into main
Reviewed-on: #64
2026-08-11 15:17:42 +00:00
Eric Coissac a63692b8c4 chore: update ci cache, document distance metrics, and bump version
Release / create-release (push) Successful in 2m26s
Release / build-linux-x86_64 (push) Successful in 8m43s
Release / build-macos-arm64 (push) Successful in 2m7s
ci.yml / build (pull_request) Canceled after 59m15s
Updated CI workflow cache keys with a `v2` salt and `Cargo.lock` hash to prevent stale incremental compilation caches and deadlocks, while updating restore keys and documenting interrupted job state. Introduced a 3-way ordinal distance metric framework that replaces ambiguous IUPAC encoding with explicit k-mer scoring, bridging pairwise methods to character-based phylogenetics via Sankoff parsimony. Bumped the `obikmer` crate version to 1.1.42.
2026-08-11 17:12:28 +02:00
coissac fa82989ea9 Merge pull request 'refactor: centralize CPU core detection using cgroup-aware utility' (#63) from push-lqzukpulzykz into main
Reviewed-on: #63
2026-08-11 10:35:06 +00:00
Eric Coissac 5f95e866f8 refactor: centralize CPU core detection using cgroup-aware utility
Release / create-release (push) Successful in 2m26s
Release / build-linux-x86_64 (push) Successful in 8m13s
Release / build-macos-arm64 (push) Successful in 1m43s
ci.yml / build (pull_request) Canceled after 1h29m17s
Introduce `obisys::effective_parallelism()` to read Linux cgroup v1/v2 CPU quotas from sysfs, preventing thread pool oversubscription in containerized environments. Replace direct `std::thread::available_parallelism()` calls across `obikindex` and `obikmer` with this centralized function. Bump `obikmer` version to 1.1.41.
2026-08-11 12:23:05 +02:00
coissac 2e7cfc4368 Merge pull request 'Push lsqnpxrxuvpp' (#62) from push-lsqnpxrxuvpp into main
Reviewed-on: #62
2026-08-11 09:09:23 +00:00
Eric Coissac f5e508ed33 feat: add multi-genome SNP pseudo-alignment and CLI export
Release / create-release (push) Successful in 5m58s
Release / build-macos-arm64 (push) Successful in 2m47s
Release / build-linux-x86_64 (push) Successful in 8m50s
CI / build (pull_request) Canceled after 5m42s
Introduces a `SnpAlignment` struct and helper methods to construct per-genome SNP pseudo-alignments from sibling k-mer data, filtering monomorphic families and encoding bases as IUPAC ambiguity codes. Exposes the type at the crate root for simplified imports. Adds a `--snp` CLI flag to compute and export these alignments as an IUPAC-coded FASTA file. Updates theory documentation to propose a multi-genome framing approach for joint phylogenetic inference, resolving pairwise correspondence ambiguities through positional homology and partial coverage thresholds. Bumps crate version to 1.1.40.
2026-08-10 22:38:38 +02:00
Eric Coissac 49f329edd5 feat: add raw SNP distance calculation and CLI flag
Exposes RawSnpDistanceOutput and implements KmerIndex::raw_snp_distance() to compute pairwise single-copy locus counts under a paralogy-aware rule. The implementation leverages ndarray for parallel matrix aggregation, producing raw p-distance matrices for sanity-checking. A --raw-snp-distance CLI flag is added to export results as CSV, mapping zero-eligible pairs to NA.
2026-08-10 22:17:07 +02:00
Eric Coissac 1a470eab9e Refactor k-mer sibling tracking to compact bitmask and on-demand counts
Replaces the explicit `SiblingInfo` struct and 3-bit minorant flags with a derived 4-bit presence mask (`FamilyMask`) that tracks observed bases per family. This eliminates redundant file I/O overhead by introducing a `PartitionCache` for batch lookups, simplifies serialization, and updates all downstream builders, stats computation, and tests to operate on the new bitmask representation. Adjusts CLI output to report deduplicated family sizes instead of histograms, ignores generated CSV files, and updates documentation to reflect the fixed canonical reference and new theory.
2026-08-10 17:53:52 +02:00
Eric Coissac ba990a48a0 feat: add obipipeline for concurrent sibling annex stats
Add the `obipipeline` crate and replace sequential scatter/gather logic with a concurrent pipeline using `Flat` and `Transform` stages. Introduce `SiblingAnnexStats` API to compute distributions, and add CLI flags to `distance.rs` for constructing the annex and exporting statistics as CSV.
2026-08-10 15:31:04 +02:00
Eric Coissac ea914bb536 feat: implement per-k-mer sibling counts and central neighbor generation
Introduce the siblingannex module in obicompactvec to store per-slot minorant flags and sibling counts in a memory-mapped annex file. Add a scatter-gather pipeline in obikindex to compute these values across index layers and write them to .psib files. Implement central_canonical_neighbors in obikseq for generating strand-aware k-mer variants around the middle base. Expose rolling statistics in obiskbuilder and update dependency graphs accordingly.
2026-08-10 15:01:59 +02:00
Eric Coissac 8bc6d533e5 feat: support negative count filters as group size offsets
Updates CLI parsing to accept negative integers for count filters, interpreting them as offsets from the group size (e.g., `-1` means all but one). A resolution closure enforces a floor of 1 to prevent unconstrained filtering on small groups. Additionally, refines evolutionary distance documentation to condition comparisons on local homology, replacing union-based Jaccard with a self-contained `SnpTally`. This unified approach streamlines SNP and shared count computation, incorporates paralogy and heterozygosity handling, and enables direct derivation of corrected distance matrices without external dependencies.
2026-08-10 12:38:35 +02:00
Eric Coissac 45df9919e5 docs: add central-position SNP distance estimator spec
Introduces a design specification for inferring substitution rates directly from k-mers with conserved flanks. The document details a memory-efficient implementation that computes 4x4 base-pair tallies using existing MPHF structures, enabling classical corrections without de Bruijn graph materialization. Updates MkDocs navigation to include the new theory page.
2026-07-10 09:49:49 +02:00
Eric Coissac 2610a4af79 feat: add Mash distance metric and rolling entropy support
Implement the Mash distance metric across the CLI, index, and compact vector traits. This includes adding a `Mash` variant to the `DistanceMetric` enum and `MetricArg` CLI argument, implementing the conversion from Jaccard distances using the standard mutation-rate estimator formula, and updating documentation with supported metrics and algorithmic references. Additionally, add an `entropy` method to rolling statistics for computing order-specific entropy.
2026-07-09 11:40:48 +02:00
coissac dc3392865f Merge pull request 'Push qowsvpqmoukq' (#61) from push-qowsvpqmoukq into main
Reviewed-on: #61
2026-07-08 18:05:42 +00:00
Eric Coissac fd2c23e7df refactor: remove equivalence class folding from entropy pipeline
Release / create-release (push) Successful in 2m27s
Release / build-linux-x86_64 (push) Successful in 8m17s
Release / build-macos-arm64 (push) Successful in 1m41s
CI / build (pull_request) Successful in 3m33s
Removes circular-reverse complement machinery and explicit k-mer canonicalization across the entropy pipeline. Frequency tallying and Shannon entropy computation now operate directly on raw k-mer values, eliminating prior score inflation and alignment-dependent artifacts while preserving orientation invariance. Updates build scripts to generate normalized lookup tables for k-mer lengths 1–6, restricts the public API to `EntropyTracker`, and bumps crate versions. Documentation is updated to reflect the simplified raw-value approach and revised module structure.
2026-07-08 19:36:30 +02:00
Eric Coissac 912f788f7f feat: extract k-mer entropy computation into new obikentropy crate
Extracts streaming entropy logic and sliding-window frequency tracking from obiskbuilder into a dedicated obikentropy crate. Introduces an EntropyTracker accumulator for O(1) per-base normalized Shannon entropy, replaces inline rolling statistics with delegated state management, and updates workspace dependencies across obikindex, obikpartitionner, and obiskbuilder. Adds criterion benchmarks to validate the refactored pipeline throughput.
2026-07-08 18:36:16 +02:00
Eric Coissac e725523898 feat: add entropy-driven k-mer complexity filtering
Introduces a MinComplexity filter driven by new CLI arguments, enabling sequence-aware threshold checks during index reconstruction and partitioning. Adds the kmer_entropy module for normalized complexity scoring, updates the KmerFilter trait to evaluate per-kmer context, and refactors test modules for better organization.
2026-07-08 12:48:25 +02:00
coissac 165982fb07 Merge pull request 'Bump obikmer version to 1.1.38 and add memory footprint logging' (#60) from push-slxmykzqmzzv into main
Reviewed-on: #60
2026-07-08 10:15:51 +00:00
Eric Coissac 2740f52326 Bump obikmer version to 1.1.38 and add memory footprint logging
Release / create-release (push) Successful in 2m26s
CI / build (pull_request) Successful in 3m32s
Release / build-linux-x86_64 (push) Successful in 8m8s
Release / build-macos-arm64 (push) Successful in 1m43s
Updates Cargo.toml version from 1.1.37 to 1.1.38. Adds explicit memory footprint estimation for the `by_partition` k-mer dedup HashMap by computing capacity-based byte sizes for map slots and stored descriptors. These metrics are logged via `debug!` to track actual memory pressure during chunk processing.
2026-07-08 12:14:58 +02:00
coissac dff5d2f457 Merge pull request 'Push smluomvxpptv' (#59) from push-smluomvxpptv into main
Reviewed-on: #59
2026-07-07 17:13:03 +00:00
Eric Coissac eea884f393 ci: improve release workflow and bump obikmer to v1.1.37
Release / create-release (push) Successful in 2m27s
CI / build (pull_request) Successful in 3m32s
Release / build-linux-x86_64 (push) Successful in 8m5s
Release / build-macos-arm64 (push) Successful in 1m41s
Replace inline `docker run` with explicit container lifecycle management to improve isolation and verify exit statuses. Bump `obikmer` crate version to 1.1.37. Add debug logging for sparse hit structure memory tracking, update chunk memory documentation to reflect the Findere rework, and clarify `BYTES_PER_KMER_PER_GENOME` as a pathological bound for safety tuning.
2026-07-07 19:11:50 +02:00
Eric Coissac ae42a061bd perf: optimize k-mer queries with sparse index and run-based aggregation
Replaces the dense `KmerResults` matrix with a sparse `SmerIndex` (`Vec<bool>` + offsets) that tracks k-mer presence independently of per-genome counts. Introduces a new aggregation pass, `sparse_findere_for_genome`, which sorts hits, detects contiguous runs, and applies monotone-deque scans to compute sliding-window minimums. This reduces query complexity from O(n_smers) to O(hits log hits), significantly lowering memory overhead and computational cost for low-density queries. Adds a deterministic PRNG and dense reference oracle in tests to validate correctness against randomized inputs without external property-testing crates.
2026-07-07 18:56:41 +02:00
Eric Coissac 040eff140c refactor(query): optimize mmap locality with column-major matrix fetch
Refactor the query pipeline into a two-stage MPHF hit-detection pass followed by a column-major matrix fetch to improve cache efficiency. Introduce a QueryHit enum for event-driven callbacks, decoupling hit detection from data population. Add scan/fetch metrics to QueryStats, update Phase 4 architecture docs, and align tests with the new callback signature.
2026-07-07 18:44:09 +02:00
Eric Coissac a348637f3b refactor(query): deduplicate k-mers upfront and split MPHF lookup
Refactor `QueryBatch` construction to perform canonical k-mer deduplication and partition routing during initialization, eliminating post-batch splitting. Split the `QueryLayer` MPHF lookup into separate `find_slot` and `fill_row` methods, updating callbacks to pass occurrence descriptors instead of indices. Introduce `QueryStats` for tracking MPHF calls and dereplication ratios, and add comprehensive unit tests for batch construction, stats arithmetic, and safe partition handling. Expose new query-layer types in the public API.
2026-07-07 18:36:25 +02:00
Eric Coissac 9d7ced4493 perf(query): replace static divisor with dynamic overhead multiplier
Replaces the static 16 divisor with a dynamic overhead_multiplier that scales chunk size based on n_genomes, the --detail flag, and a safety factor. This bounds per-chunk memory usage to ≤50% of available RAM across concurrent workers by accounting for genome-scaled k-mer buffers and optional coverage data.
2026-07-07 16:14:10 +02:00
Eric Coissac 9d49929b0c refactor(query): implement throttled per-file streaming pipeline
Replace the flat chunk iterator with a throttled, per-file streaming architecture using `obipipeline::throttle`. The new `GuardedChunkIter` binds file handles to their `ThrottleGuard`, enforcing concurrent open-file limits and tracking active files via an atomic counter. Pipeline stages and the progress spinner are updated to support this resource-aware, parallelized I/O flow.
2026-07-07 16:07:33 +02:00
Eric Coissac 61c390503d feat(query): add throughput metering and --max-open-files flag
Introduces an EMA-based throughput meter that dynamically updates a spinner with MB/s rates, along with atomic counters for tracking cumulative bytes and active chunks. Adds final pipeline reporting and consolidates imports for cleaner performance instrumentation.
2026-07-07 15:19:09 +02:00
Eric Coissac 8f0ceec784 docs: clarify query processing and add performance roadmap
Clarifies that query processing is fully inlined within `process_chunk` using flat allocations, refining monotone-deque sliding window semantics and `kmer_missing` tracking. Updates `QueryLayer::open` variant precedence and introduces a phased roadmap (Phases 0–6) to address performance bottlenecks through parallel I/O, genome-aware chunk sizing, k-mer dereplication, NUMA-aware matrix fetches, and sparse Findere rework.
2026-07-07 15:12:09 +02:00
Eric Coissac 00b4b1fa51 docs: add future work section for parallel gzip decompression
Proposes replacing the single-threaded niffler/flate2 pipeline in `obiread::xopen` with `rapidgzip-rs` for local `.gz` files. Details constraints such as path dependencies, non-seekable streams, C++ toolchain requirements, and binding maturity. Marks the optimization as parked pending throughput and correctness validation.
2026-07-07 13:43:01 +02:00
coissac e96ad38c8e Merge pull request 'feat: filter zero-valued entries from kmer strict matches output' (#58) from push-rosnxrytzxzk into main
Reviewed-on: #58
2026-07-07 09:22:02 +00:00
Eric Coissac 4fc7860825 feat: filter zero-valued entries from kmer strict matches output
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m13s
Release / build-macos-arm64 (push) Failing after 31s
CI / build (pull_request) Successful in 3m23s
Optimize query serialization by conditionally excluding genomes with zero total matches. This reduces JSON payload size while preserving the label-to-count mapping structure. Updates architecture documentation and bumps version to 1.1.36.
2026-07-07 10:50:01 +02:00
coissac 5bdc0f826a Merge pull request 'fix: validate packed matrix columns before repacking' (#57) from push-vkqvorvsqnqx into main
Reviewed-on: #57
2026-07-03 15:26:55 +00:00
Eric Coissac cd2f2f9417 fix: validate packed matrix columns before repacking
Release / create-release (push) Successful in 2m27s
Release / build-linux-x86_64 (push) Successful in 8m15s
Release / build-macos-arm64 (push) Failing after 30s
CI / build (pull_request) Successful in 3m17s
Add header parsing helpers to extract column counts without memory mapping. Update packing functions to verify existing files match current metadata, preventing stale or widened-column artifacts. Extract inline tests in obilayeredmap to an external module and add comprehensive aggregation tests. Bump obikmer to 1.1.35 and clean up repository configuration.
2026-07-03 17:20:22 +02:00
coissac 7844239a8e Merge pull request 'Push msotyzponsls' (#56) from push-msotyzponsls into main
Reviewed-on: #56
2026-07-03 11:28:49 +00:00
Eric Coissac 2b37e8aac4 fix(bitmatrix): explicitly compute diagonal entries for self-similarity
Release / create-release (push) Successful in 2m26s
Release / build-linux-x86_64 (push) Successful in 8m13s
Release / build-macos-arm64 (push) Failing after 30s
CI / build (pull_request) Successful in 3m21s
The pairwise matrix functions now explicitly calculate and overwrite diagonal entries using `f(i,i)`, replacing previous implicit symmetric mirroring or default values. Documentation has been updated to clarify that diagonals represent self-comparison weights, ensuring accurate self-similarity calculations. Additionally, the obikmer crate version has been bumped to 1.1.34.
2026-07-03 13:04:40 +02:00
Eric Coissac 67b4e4da53 refactor(numa): replace flat runner with per-node activation channels
Shifts the NUMA-aware runner from a flat, round-robin model to a per-node architecture using dedicated `NodeActivation` channels. Replaces absolute deltas with relative scaling based on the previous growth step's worker count, decoupling growth from node count to fix slow ramp-up and enforce per-node fairness. Updates architecture documentation to reflect these changes and focus tuning questions on `INITIAL`/`GROWTH_DIVISOR` parameters for I/O-bound validation.
2026-07-03 13:03:31 +02:00
coissac 66ab4c6db1 Merge pull request 'feat(numa): introduce I/O sampling to prevent activation stalls' (#55) from push-ooruxnkktvvz into main
Reviewed-on: #55
2026-07-02 09:36:19 +00:00
Eric Coissac f84dd539bf feat(numa): introduce I/O sampling to prevent activation stalls
Release / create-release (push) Successful in 2m25s
Release / build-linux-x86_64 (push) Successful in 8m47s
Release / build-macos-arm64 (push) Failing after 31s
CI / build (pull_request) Successful in 3m30s
Replaces the monolithic CPU scaling threshold with separate CPU and I/O spawn thresholds. Introduces an `IoSample` struct with platform-specific byte reading and a relative throughput growth heuristic. Adds a 0.1s wall-clock guard to `CpuSample` to suppress artificial efficiency spikes, and updates `maybe_activate` to trigger worker scaling when either resource indicates headroom. Bumps `obikmer` to v1.1.33 and updates architecture documentation.
2026-07-02 10:07:22 +02:00
coissac 6378734e1c Merge pull request 'fix(obisys): remove activation guard to always update metrics' (#54) from push-vkloynurrxzu into main
Reviewed-on: #54
2026-07-01 18:34:10 +00:00
Eric Coissac b3a617cce1 fix(obisys): remove activation guard to always update metrics
Release / create-release (push) Successful in 2m26s
CI / build (pull_request) Successful in 3m35s
Release / build-linux-x86_64 (push) Successful in 8m9s
Release / build-macos-arm64 (push) Failing after 30s
Removes the `if activate` conditional in `src/obisys/src/lib.rs`, making debug logging and state updates for performance counters execute unconditionally. This ensures tracking metrics are continuously refreshed regardless of the activation threshold. Also bumps the `obikmer` dependency version.
2026-07-01 20:32:56 +02:00
coissac 2080e5e8a9 Merge pull request 'ci: fix registry auth and bump obikmer to 1.1.30' (#53) from push-zxlknspoxknt into main
Reviewed-on: #53
2026-07-01 14:20:09 +00:00
Eric Coissac 45ed2bc9b8 ci: fix registry auth and bump obikmer to 1.1.30
Release / create-release (push) Successful in 2m26s
Release / build-linux-x86_64 (push) Successful in 8m12s
Release / build-macos-arm64 (push) Failing after 1m55s
CI / build (pull_request) Successful in 3m32s
Update the release workflow to explicitly resolve the Docker registry username from repository secrets instead of inferring it from the runner's actor. Bump the obikmer package version to 1.1.30.
2026-07-01 14:31:30 +02:00
coissac aa126fd89d Merge pull request 'feat: simplify worker spawning logic and update macOS build workflow' (#52) from push-uvmlknmzqqnx into main
Reviewed-on: #52
2026-07-01 09:50:51 +00:00
Eric Coissac c612132763 feat: simplify worker spawning logic and update macOS build workflow
Release / create-release (push) Successful in 2m59s
Release / build-linux-x86_64 (push) Successful in 8m13s
Release / build-macos-arm64 (push) Failing after 8s
CI / build (pull_request) Successful in 3m24s
Updates the release workflow to run macOS builds inside a Docker container with explicit registry authentication and adjusted artifact paths. Bumps the obikmer crate version to 1.1.29 and adds *.log to .gitignore. Simplifies NUMA worker spawning by lowering the activation threshold from 0.95 to 0.2, replacing complex stateful tracking with a direct efficiency check, and downgrading progress logging to debug level. Includes general code formatting improvements for readability.
2026-07-01 11:40:57 +02:00
coissac 19660f8cd0 Merge pull request 'ci: update registry auth and improve adaptive worker scaling' (#51) from push-qlpywtroutvx into main
Reviewed-on: #51
2026-06-26 13:16:23 +00:00
Eric Coissac 7b07540a69 ci: update registry auth and improve adaptive worker scaling
Release / create-release (push) Successful in 2m27s
CI / build (pull_request) Successful in 3m17s
Release / build-linux-x86_64 (push) Successful in 8m3s
Release / build-macos-arm64 (push) Failing after 1s
Refactor the release workflow to use a structured container object with authenticated pulls for macOS ARM64 builds. Replace single-worker activation with dynamic upfront provisioning based on node and worker counts. Implement an absolute efficiency gain threshold for scaling checks and add early termination to improve adaptive scaling stability. Bump obikmer crate version to 1.1.27.
2026-06-26 15:13:13 +02:00
coissac 89c43e28f5 Merge pull request 'ci: update release workflow and bump obikmer to 1.1.26' (#50) from push-npttlqpomtvz into main
Reviewed-on: #50
2026-06-24 13:55:40 +00:00
Eric Coissac b9b2e42ad2 ci: update release workflow and bump obikmer to 1.1.26
Release / create-release (push) Successful in 2m32s
CI / build (pull_request) Successful in 3m47s
Release / build-linux-x86_64 (push) Successful in 8m18s
Release / build-macos-arm64 (push) Failing after 0s
Replaces the macOS ARM64 cross-compilation container with a custom internal registry image. Adds explicit steps to install the `aarch64-apple-darwin` Rust target and `jq`, and updates the build command to use `--no-default-features`. Bumps the `obikmer` package version from 1.1.25 to 1.1.26.
2026-06-24 15:55:02 +02:00
coissac ca42fdff2f Merge pull request 'ci: update macOS ARM64 build workflow and bump obikmer version' (#49) from push-lllnsqlrqrut into main
Reviewed-on: #49
2026-06-23 13:15:20 +00:00
Eric Coissac 136cd89efb ci: update macOS ARM64 build workflow and bump obikmer version
Release / create-release (push) Successful in 2m27s
Release / build-linux-x86_64 (push) Successful in 7m52s
Release / build-macos-arm64 (push) Failing after 8m53s
CI / build (pull_request) Successful in 5m31s
Replace manual Zig/cargo-zigbuild setup with a pre-configured Docker container (`joseluisq/rust-linux-darwin-builder`). Use explicit Clang cross-compilers for native macOS ARM64 compilation. Bump the `obikmer` package version to 1.1.25.
2026-06-23 15:01:17 +02:00
coissac a4bbf607b7 Merge pull request 'Push kxsopnzprltv' (#48) from push-kxsopnzprltv into main
Reviewed-on: #48
2026-06-23 09:51:33 +00:00
Eric Coissac 9927100a1c chore: update obikmer to 1.1.24
Release / create-release (push) Successful in 2m24s
Release / build-linux-x86_64 (push) Successful in 7m49s
Release / build-macos-arm64 (push) Failing after 3m31s
CI / build (pull_request) Successful in 3m22s
Bumps the obikmer version in Cargo.toml from 1.1.21 to 1.1.24 and updates Cargo.lock to align with the upstream patch release (1.1.23). This ensures consistent dependency resolution across builds.
2026-06-23 11:47:54 +02:00
Eric Coissac 527258f822 ci: enforce macOS 11.0 deployment target for ARM builds
Adds MACOSX_DEPLOYMENT_TARGET=11.0 environment variable and updates the cargo zigbuild target to aarch64-apple-darwin11.0 to explicitly require macOS 11.0 for ARM binary compilation.
2026-06-23 11:46:09 +02:00
coissac ef62f1947e Merge pull request 'chore: bump version to 1.1.21 and update obikindex features' (#47) from push-xwutoxpnxorz into main
Reviewed-on: #47
2026-06-23 08:31:31 +00:00
Eric Coissac d02316dcf6 chore: bump version to 1.1.21 and update obikindex features
Release / create-release (push) Successful in 2m29s
CI / build (pull_request) Successful in 4m36s
Release / build-linux-x86_64 (push) Successful in 10m25s
Release / build-macos-arm64 (push) Failing after 4m50s
Disables default features for the `obikindex` dependency and introduces a `[features]` block. The new `numa` feature is set as the default, conditionally enabling NUMA support in `obikindex`.
2026-06-23 10:30:39 +02:00
coissac c323b3eaef Merge pull request 'Bump obikmer to 1.1.20 and update release workflow' (#46) from push-wpnywwlwxrps into main
Reviewed-on: #46
2026-06-23 08:03:58 +00:00
Eric Coissac b77d8e9ca0 Bump obikmer to 1.1.20 and update release workflow
Release / create-release (push) Successful in 2m26s
CI / build (pull_request) Successful in 3m14s
Release / build-linux-x86_64 (push) Successful in 7m44s
Release / build-macos-arm64 (push) Failing after 7m13s
Update the Gitea release workflow to fetch a full git clone with complete history, ensuring all commits and tags are available for accurate version resolution. This prepares the repository for the standard patch-level release of obikmer v1.1.20.
2026-06-23 10:03:15 +02:00
coissac 7c5bab3694 Merge pull request 'fix(ci): restrict workflow to PRs and improve release tagging' (#45) from push-louqrszyuqpz into main
Reviewed-on: #45
2026-06-23 07:52:35 +00:00
Eric Coissac fab4e0d6de fix(ci): restrict workflow to PRs and improve release tagging
Release / create-release (push) Failing after 26s
Release / build-linux-x86_64 (push) Has been skipped
Release / build-macos-arm64 (push) Has been skipped
CI / build (pull_request) Successful in 3m17s
Restrict the CI pipeline to pull request events only by removing the unconfigured push trigger and eliminating a duplicate pull_request block in the workflow file. Update the Makefile to suppress stderr from the aichat command and introduce a fallback release tag message for robust version tagging. Additionally, bump the obikmer crate version to 1.1.19.
2026-06-23 09:42:23 +02:00
coissac 973a3f3d6e Merge pull request 'feat: add numa feature flag and automate release workflow' (#44) from push-uymxyvsyooro into main
CI / build (push) Successful in 3m16s
Reviewed-on: #44
2026-06-23 07:22:33 +00:00
Eric Coissac 1a839a295a feat: add numa feature flag and automate release workflow
Release / create-release (push) Failing after 37s
Release / build-linux-x86_64 (push) Has been skipped
Release / build-macos-arm64 (push) Has been skipped
CI / build (pull_request) Successful in 3m23s
Refactor the Gitea release pipeline to generate releases via API and upload binaries using a shared ID. Automate changelog generation by fetching recent commits with `jj log` and producing markdown notes via `aichat`. Convert `hwlocality` to an optional dependency gated by a default `numa` feature, providing fallback implementations for graceful degradation when NUMA support is disabled. Bump obikmer to 1.1.18.
2026-06-23 09:07:04 +02:00
coissac 2ea58703c7 Merge pull request 'Push zkptpswyxnvt' (#43) from push-zkptpswyxnvt into main
CI / build (push) Successful in 3m19s
Reviewed-on: #43
2026-06-22 16:29:59 +00:00
Eric Coissac ac3ef106e7 refactor: implement adaptive worker scaling and infallible NUMA build
Release / build-linux-static (push) Successful in 8m4s
CI / build (pull_request) Successful in 3m26s
Replaces the fallible NUMA topology builder with an infallible fallback that synthesizes a single-node UMA configuration on failure or absence. Refactors PartitionRunner to pre-spawn dormant workers and dynamically activate them via CPU efficiency thresholds, replacing static upfront spawning with adaptive scaling. Bumps obikmer crate version to 1.1.15.
2026-06-22 18:29:39 +02:00
Eric Coissac 469e53b6f5 Add genomic distance benchmarking suite and test data
Introduces scripts to compute and validate pairwise genomic distance matrices across multiple metrics. Updates the Makefile with build and comparison targets, adds .gitignore rules for generated outputs, and includes test CSV matrices and a Newick phylogenetic tree for validating the distance computation pipeline.
2026-06-22 18:24:30 +02:00
Eric Coissac 9f1df96ea7 ci: restrict push trigger to main branch
Replace the wildcard `['**']` in the push trigger with `['main']`. This prevents redundant pipeline runs on non-main branches during push events.
2026-06-22 16:58:44 +02:00
coissac 4e4cce2879 Merge pull request 'fix(ci): enable cross-compilation in release workflow and bump obikmer' (#42) from push-sxlpkrkyuttk into main
CI / build (push) Successful in 3m20s
Reviewed-on: #42
2026-06-22 14:31:26 +00:00
Eric Coissac 68b05b93c4 fix(ci): enable cross-compilation in release workflow and bump obikmer
CI / build (push) Successful in 4m38s
Release / build-linux-static (push) Successful in 10m14s
CI / build (pull_request) Successful in 3m40s
Injects PKG_CONFIG_ALLOW_CROSS=1 into the static binary build step to ensure correct native dependency resolution during musl target compilation with cargo zigbuild. Also updates the obikmer crate version from 1.1.13 to 1.1.14.
2026-06-22 16:31:04 +02:00
coissac 0a668cf8a6 Merge pull request 'chore: bump obikmer to 1.1.13 and fix Makefile revision tag' (#41) from push-qwzpxktnlyls into main
CI / build (push) Has been cancelled
Reviewed-on: #41
2026-06-22 14:18:25 +00:00
Eric Coissac e6d6942e2f chore: bump obikmer to 1.1.13 and fix Makefile revision tag
CI / build (push) Has been cancelled
Release / build-linux-static (push) Has been cancelled
CI / build (pull_request) Has been cancelled
Update the obikmer crate version from 1.1.12 to 1.1.13 in Cargo.toml. Additionally, change the Makefile's Git revision specifier from @- to @ to ensure the version tag is applied to the current commit before pushing.
2026-06-22 16:17:52 +02:00
coissac bf9c9aeacb Merge pull request 'chore: bump version to 1.1.12 and fix release workflow' (#40) from push-zmkxouxypspm into main
CI / build (push) Has been cancelled
Reviewed-on: #40
2026-06-22 14:13:13 +00:00
Eric Coissac 22a65857a1 chore: bump version to 1.1.12 and fix release workflow
CI / build (push) Successful in 3m14s
CI / build (pull_request) Successful in 3m51s
Update Cargo.toml to 1.1.12 for a semver patch release. Refactor the Makefile release target to explicitly retrieve the parent commit hash via `jj log` and apply the tag, replacing implicit working directory tagging. Remove `jj auto-describe` and `--change @` in favor of an explicit `git push origin` for the version tag.
2026-06-22 16:12:56 +02:00
coissac d16a867640 Merge pull request 'ci: bypass PEP 668 restrictions and update obikmer to 1.1.11' (#39) from push-mxysluysloxr into main
CI / build (push) Successful in 4m1s
Release / build-linux-static (push) Failing after 7m28s
Reviewed-on: #39
2026-06-22 13:52:51 +00:00
Eric Coissac 616050075f ci: bypass PEP 668 restrictions and update obikmer to 1.1.11
CI / build (push) Successful in 3m41s
CI / build (pull_request) Successful in 3m49s
Add the `--break-system-packages` flag to the `pip install ziglang` command in the Gitea release workflow to bypass PEP 668 restrictions on modern Linux distributions. Additionally, bump the `obikmer` crate version from 1.1.9 to 1.1.11 across both Cargo.toml and Cargo.lock.
2026-06-22 15:52:34 +02:00
coissac e22afe9621 Merge pull request 'chore: bump obikmer to 1.1.9 and update release workflow' (#38) from push-noxuppsknsol into main
CI / build (push) Successful in 3m11s
Release / build-linux-static (push) Failing after 3m4s
Reviewed-on: #38
2026-06-22 13:32:50 +00:00
Eric Coissac bdfac71e65 chore: bump obikmer to 1.1.9 and update release workflow
CI / build (push) Successful in 3m24s
CI / build (pull_request) Failing after 0s
Bumps the obikmer crate version from 1.1.7 to 1.1.9 in Cargo.toml and Cargo.lock. Updates the Gitea release workflow to dynamically locate the Zig compiler via Python, generating musl-targeted gcc/g++ wrapper scripts installed to /usr/local/bin for static Linux cross-compilation during releases.
2026-06-22 15:32:10 +02:00
coissac a00bb37478 Merge pull request 'ci: switch to Zig build toolchain and bump obikmer to 1.1.7' (#37) from push-nvvqmzmrotxx into main
CI / build (push) Successful in 3m14s
Release / build-linux-static (push) Failing after 2m42s
Reviewed-on: #37
2026-06-22 13:20:12 +00:00
Eric Coissac d30a4efd9b ci: switch to Zig build toolchain and bump obikmer to 1.1.7
CI / build (push) Successful in 3m12s
CI / build (pull_request) Successful in 3m16s
Replaces the musl-based static Linux build toolchain with Zig (`ziglang` via pip and `cargo-zigbuild`), removing `musl-tools` dependencies. The workflow now invokes `cargo zigbuild` for cross-compiling the static binary. Additionally, bumps the `obikmer` crate version to 1.1.7.
2026-06-22 15:19:39 +02:00
coissac 6baf2e64ca Merge pull request 'chore: bump obikmer to 1.1.6 and automate git tagging' (#36) from push-yxmtknzsynpx into main
CI / build (push) Successful in 3m20s
Release / build-linux-static (push) Failing after 4m30s
Reviewed-on: #36
2026-06-22 13:01:06 +00:00
Eric Coissac c0a71a2d49 chore: bump obikmer to 1.1.6 and automate git tagging
CI / build (push) Successful in 4m46s
CI / build (pull_request) Successful in 4m27s
Bumps the obikmer crate version from 0.1.4 to 1.1.6 in Cargo.toml and Cargo.lock. Updates the Makefile release target to automatically extract the version, create a Git tag, and push it to the remote repository, extending the existing workflow with standard Git publishing steps.
2026-06-22 15:00:36 +02:00
coissac a609c1af95 Merge pull request 'ci: streamline release workflow and bump obikmer to 0.1.4' (#35) from push-zokprynyqunu into main
CI / build (push) Successful in 4m41s
Release / build-linux-static (push) Failing after 6m4s
Reviewed-on: #35
2026-06-22 09:38:36 +00:00
Eric Coissac 3d32be8a83 ci: streamline release workflow and bump obikmer to 0.1.4
CI / build (push) Successful in 4m33s
CI / build (pull_request) Successful in 4m17s
Replaces the intermediate artifact upload step in the Gitea release workflow with a direct REST API call, eliminating unnecessary dependencies and adding `jq`. Also increments the obikmer crate version to 0.1.4.
2026-06-22 11:38:19 +02:00
coissac c4c71dc892 Merge pull request 'Push mtzqmmrlmzzx' (#34) from push-mtzqmmrlmzzx into main
CI / build (push) Successful in 4m34s
Reviewed-on: #34
2026-06-22 08:47:24 +00:00
Eric Coissac 4e625afaba refactor: update CI toolchain setup and optimize parallel indexing
CI / build (push) Successful in 4m56s
CI / build (pull_request) Successful in 4m11s
Update CI workflows to explicitly install the Rust toolchain via rustup and configure musl targets for deterministic static builds in Docker containers. Bump obikmer dependency to 0.1.3. Refactor obicompactvec to reduce peak memory usage by computing column sizes from filesystem metadata, add atomic writes, and implement cleanup guards. Replace parallel iteration patterns in obikindex with a structured PartitionRunner pipeline for simplified error handling and progress tracking.
2026-06-22 10:46:24 +02:00
Eric Coissac a522c0907e feat: add CI/CD workflows, release automation, and CLI version flag
CI / build (push) Failing after 2m41s
CI / build (pull_request) Failing after 6s
Adds Gitea Actions for continuous integration and tagged releases, including static musl binary compilation and artifact upload. Introduces a Makefile target to automate semantic version bumping and publishing. Bumps the package version to 0.1.1 and enables automatic `--version` output via Clap.
2026-06-22 10:36:40 +02:00
Eric Coissac c1d6f277ce feat(select): add metrics reporting to selection methods
Integrates an obisys::Reporter across indexing and command modules to capture execution metrics. Replaces discarded timer stops with explicit rep.push() calls, adds timing instrumentation for the pack stage, and prints collected reports after each selection branch.
2026-06-22 10:25:24 +02:00
Eric Coissac 9356be4ec0 feat: introduce obitaxonomy crate for hierarchical taxonomy parsing
Adds the `obitaxonomy` crate to parse and validate hierarchical taxonomy paths using a strict `taxonomy:/name@rank/...` syntax. Replaces generic string-based path matching in predicates with structured `TaxPath` and `TaxPattern` types, enforcing explicit anchor constraints and rank-aware semantics. Updates filtering documentation to clarify optional leading slashes and segment-boundary matching rules.
2026-06-22 10:24:04 +02:00
Eric Coissac c694e1f2b0 feat: add benchmark pipeline, expose APIs, and enforce strict paths
Introduces a Make-based orchestration for simulating, indexing, merging, filtering, and verifying k-mer counts and presence. Exposes internal builder and iterator APIs publicly, enforces mandatory leading slashes for predicate patterns, registers the `obitaxonomy` crate, and updates tooling configurations alongside documentation.
2026-06-22 10:18:33 +02:00
Eric Coissac 280ca1f5a3 feat: add optimized new_ones constructor for all-ones bit vectors
Introduces `new_ones` and `add_col_ones` methods to directly initialize all-ones bit vectors and matrix columns. This replaces redundant initialization sequences that created zero-filled structures and applied bitwise NOT, with a single pass that writes contiguous 0xFF bytes to disk. The change eliminates inversion overhead, streamlines test setup, and improves performance for filter mask intersection logic while preserving identical semantics.
2026-06-22 10:00:01 +02:00
Eric Coissac 9abb2db92f refactor: replace explicit bit-setting loops with optimized bulk operations
Refactor bitmatrix, colgroup, and layer modules to replace manual iteration with concise `or_where` predicates and bulk inversion calls. This simplifies the codebase and leverages optimized internal implementations for improved performance.
2026-06-22 09:56:41 +02:00
Eric Coissac 7c1efa9cbb feat: add vectorized column filters and optimize partitioner iteration
Adds `FilterMask` and conditional bitwise methods (`*_where`) to `obicompactvec` for composable column-based slot filtering. Extends `obikpartitionner` with a `MatrixGroupOps` trait and `column_mask_expr` method to express aggregate constraints as vectorized masks. Refactors matrix builder management into a unified `Builders` enum and introduces `try_compute_combined_mask`, enabling O(1) slot checks and skipping unnecessary row reads during partitioning and rebuilding passes.
2026-06-22 09:54:51 +02:00
Eric Coissac 4c4524766c feat(matrix): add partial group reductions and column persistence
Expands MatrixGroupOps with partial_group_min/max helpers for bitwise reductions and introduces add_col_from methods to persist external vectors as matrix columns. Refactors column aggregation in the partitioner to leverage these group operations directly, replacing iterative row processing with simplified builder lifecycle management and explicit metadata serialization.
2026-06-22 09:49:04 +02:00
Eric Coissac 7eea71fdcd docs(obicompactvec): update API docs and algorithm descriptions
Replace trait-based API documentation with concrete, zero-copy view structs and update all associated diagrams. Refine algorithmic descriptions for sentinel handling, overflow stores, and bulk operations. Clarify temporary file lifecycles and group-chunking strategies to support memory-efficient parallel aggregation.
2026-06-22 09:46:19 +02:00
Eric Coissac f91c5a3f79 refactor(obicompactvec): unify bit and int vector slice views
Refactors column and matrix access to use unified `BitSliceView` and `IntSliceView` abstractions, replacing legacy `PackedCol`/`IntColView` types. Introduces `BitSlice`/`IntSlice` traits for zero-copy, trait-based bitwise and arithmetic operations across persistent and temporary vector types. Removes deprecated in-memory `MemoryBitVec` and `MemoryIntVec` implementations and their tests, while updating dependent crates to use the new view-based API and `BitSliceMut` trait.
2026-06-17 23:51:32 +02:00
Eric Coissac fb4962c4fe refactor: replace in-memory vectors with temp-file-backed storage
Introduces `TempCompactIntVec` and `TempBitVec` as temporary, file-backed intermediates to replace eager in-memory vectors, enabling OS-level paging under memory pressure. Updates the `MatrixGroupOps` trait to return `io::Result` types, allowing proper error propagation and supporting chunked accumulation for large column groups. Includes builder patterns with `.freeze()` finalization, automatic `TempDir` cleanup on drop, and necessary test updates to handle the new fallible signatures. Also fixes `Cargo.toml` section ordering.
2026-06-17 23:36:15 +02:00
Eric Coissac 1d38d87ff9 Add column group operations and mask_with trait
Introduce the `ColGroup` struct and `MatrixGroupOps` trait to manage named subsets of column indices and perform additive aggregations (count, sum, any). Implement these operations for `PersistentBitMatrix` and `PersistentCompactIntMatrix`, applying size-optimized branches for presence counts and direct accumulation for small groups. Additionally, add a `mask_with` trait method that efficiently zero-sets elements based on a mask, optimized for sparse masks with O(n_zeros) complexity. Include comprehensive tests covering overflow handling, slot masking, and result additivity across partitioned data.
2026-06-17 23:28:52 +02:00
Eric Coissac 93559c3294 feat: introduce unified column view types for bit and int matrices
This commit introduces `BitColView` and `IntColView` to abstract over Columnar and Packed storage formats, implementing `BitSlice` and `IntSlice` for uniform column access. It adds `col_view()` accessors to `PersistentBitMatrix` and `PackedCompactIntMatrix`, explicitly panicking on implicit variants. The new types are publicly re-exported, and unit tests are added to validate per-element retrieval, aggregation methods, and parity with the original columnar representation.
2026-06-17 23:24:11 +02:00
Eric Coissac 1f0d77d5bf docs: document compact vector implementation with Mermaid diagrams
Add Mermaid diagrams to visualize the trait hierarchy, compact int storage layout, and SIMD-vectorizable arithmetic operations for MemoryIntVec and PersistentCompactIntVec. Also document concrete type structures and planned layer/partition composition rules to improve documentation clarity.
2026-06-17 23:20:55 +02:00
Eric Coissac eeba43ac4f docs: add technical reference for obicompactvec module
Document the two-tier compact integer encoding, BitSlice/IntSlice trait hierarchy, and SIMD-friendly O(n+k) algorithms. Include details on concrete memory and persistent vector types, matrix aggregation traits, and planned group-filtering APIs.
2026-06-17 23:19:31 +02:00
Eric Coissac 7ed7b26039 perf: optimize vec arithmetic and add overflow tests
Refactor `cmp_scalar`, `min`, `max`, `add`, and `diff` to operate directly on the primary byte array, deferring overflow slot resolution to a secondary pass. This eliminates HashMap lookups in the hot path and enables SIMD vectorization. Add six unit tests to validate correct promotion and demotion between storage slots when values cross the 255 threshold.
2026-06-17 23:18:18 +02:00
Eric Coissac 26de90f18d feat: add iteration and aggregation to compact int vec
Implemented `sum()`, `count_nonzero()`, and `iter()` to complete the numeric vector interface. The builder now computes aggregate values across memory-mapped regions and overflow entries, while the reader delegates these operations to its inherent methods. The iterator provides zero-copy access to underlying `u32` elements.
2026-06-17 23:15:56 +02:00
Eric Coissac 497d250d8a refactor: replace byte-level bit iteration with 64-bit words
Refactor `BitIter` to process `u64` chunks using word-aligned shifts instead of byte-level operations. Introduce a dedicated `MemoryBitIter` for `MemoryBitVec`, updating its `iter()` and `IntoIterator` implementations accordingly. Hide `MemoryBitIter` from the public API to narrow the crate's interface, while leveraging explicit alignment guarantees for safer and more efficient bit extraction.
2026-06-17 23:11:12 +02:00
Eric Coissac aa98e82875 refactor: introduce PackedIntCol view and use iterators
Centralizes overflow handling and improves modularity by replacing manual mmap indexing and row loops with composable iterator patterns. This change leverages Rust's iterator traits for efficient, idiomatic column traversal while encapsulating data access in a dedicated view struct.
2026-06-17 23:09:18 +02:00
Eric Coissac 5ff5b04d2d refactor: replace manual bit ops with BitSlice traits
Refactors bit manipulation and distance calculations to leverage standardized `BitSlice` traits, replacing manual byte/word logic with safer, reusable methods. Extends `IntSlice` and `IntSliceMut` traits to expose direct memory-mapped access and overflow management, enabling efficient bulk data extraction and serialization. Replaces manual bit-shifting loops with optimized table-based unpacking and adds population count and distance metric methods for improved performance. Updates `PersistentBitVecBuilder` with file tracking and safe flushing, and aligns test imports with new trait bounds.
2026-06-17 15:28:44 +02:00
Eric Coissac df7b400fda perf: optimize aggregation with byte-level helpers and direct mmap
Introduce `byte_sum` and `byte_count_nonzero` to efficiently aggregate compact-int byte slices, bypassing per-element decoding and overflow map lookups. Refactor `sum()` and `count_nonzero()` across the matrix, reader, and traits modules to use direct memory-mapped slice iteration and idiomatic Rust iterators. Additionally, expose `MemoryIntIter` publicly and implement `IntoIterator` and `IntSlice` for `MemoryIntVec` to enable standard iteration and delegate aggregation to the new helpers.
2026-06-17 15:21:21 +02:00
Eric Coissac d1717688d2 refactor: extract matrix helpers and improve bit iteration ergonomics
Refactor parallel matrix construction by extracting reusable `pairwise_matrix` and `pairwise2_matrix` helpers, and consolidate binary record deserialization into dedicated parsing functions. Add `set` and `iter` methods to `BitSliceMut` and `MemoryBitVec` for ergonomic bit manipulation and iteration. Standardize JSON field extraction via `meta::field`, expose `MemoryBitIter`, and improve test reliability by automatically cleaning up temporary directories.
2026-06-17 15:15:39 +02:00
Eric Coissac cde6457eea feat: add memory vectors, slice traits, and column extraction methods
Introduce `MemoryBitVec` and `MemoryIntVec` for efficient in-memory storage with hybrid compression and overflow handling. Implement `BitSlice`, `BitSliceMut`, `IntSlice`, and `IntSliceMut` traits across persistent and memory-backed types to enable generic slice operations and bitwise/arithmetic overloads. Add `col_persist` and `col_as_memory` methods to `BitMatrix` and `IntMatrix` for efficient column extraction. Align with the new single-pass rebuild architecture by supporting fast kmer filtering and matrix rebuilding. Includes comprehensive tests and profiling instrumentation for the packing phase.
2026-06-17 15:03:18 +02:00
Eric Coissac b6fcbc545f refactor: replace rayon with NUMA-aware PartitionRunner
Replaces `rayon` parallel iteration across index, rebuild, reindex, and select modules with a custom `PartitionRunner`. This introduces NUMA-aware task distribution with CPU pinning and round-robin scheduling, eliminating `Arc`, `Mutex`, and atomic synchronization primitives in favor of a flat, pre-spawned worker architecture. Error handling is simplified via `.map_err()` and the `?` operator, while progress bar updates are decoupled into dedicated callbacks.
2026-06-15 18:53:31 +02:00
coissac 9578f991f4 Merge pull request 'Push pslsukyowzrp' (#32) from push-pslsukyowzrp into main
Reviewed-on: #32
2026-06-15 16:29:24 +00:00
Eric Coissac 1cd7916e06 refactor: replace rayon with NUMA-aware PartitionRunner
Replaces `rayon` parallel iteration across index, rebuild, reindex, and select modules with a custom `PartitionRunner`. This introduces NUMA-aware task distribution with CPU pinning and round-robin scheduling, eliminating `Arc`, `Mutex`, and atomic synchronization primitives in favor of a flat, pre-spawned worker architecture. Error handling is simplified via `.map_err()` and the `?` operator, while progress bar updates are decoupled into dedicated callbacks.
2026-06-15 18:29:04 +02:00
Eric Coissac bc92dc4592 refactor: restructure partitioner with shared utilities and pipeline
This commit restructures the partitioner crate by extracting shared utilities and the `ColBuilder` enum into a new `common` module. It introduces a multi-phase `graph_pipeline` for constructing and materializing De Bruijn graphs, replacing manual graph construction in `index_layer`, `merge_layer`, and `rebuild_layer`. All layer workflows now use centralized `build_graph` and `materialize_layer` abstractions, with standardized error context strings for improved diagnostics.
2026-06-15 14:08:16 +02:00
coissac a9567ad023 Merge pull request 'Push rtnzuqxzmkon' (#31) from push-rtnzuqxzmkon into main
Reviewed-on: #31
2026-06-15 09:40:35 +00:00
Eric Coissac 4a64718fd1 perf: replace partition processing with adaptive NUMA worker pool
Replaces the previous partition processing logic with an adaptive, NUMA-aware multi-threaded worker pool that dynamically scales active threads based on real-time CPU efficiency. Introduces pre-spawned, CPU-pinned threads managed via crossbeam channels and Rayon to optimize memory bandwidth and core utilization. Adds a `max_workers()` accessor to aggregate maximum worker capacity across NUMA nodes and updates diagnostics to report active versus maximum worker counts.
2026-06-15 11:40:14 +02:00
Eric Coissac 7a87e911b6 feat: introduce NUMA-aware PartitionRunner for adaptive parallelism
Replace NUMA-naive Rayon loops and ad-hoc adaptive pools with a unified `PartitionRunner` that manages a NUMA-aware worker pool. The implementation uses pinned Rayon thread pools per node and activates dormant threads based on real-time CPU efficiency metrics. This standardizes partition-level parallelism, optimizes memory locality, and eliminates cross-socket traffic. Includes architecture documentation and updates mkdocs navigation.
2026-06-15 11:34:41 +02:00
coissac 313d73838a Merge pull request 'feat: add pipeline concurrency throttling and HPC build docs' (#30) from push-owwylwtskwzw into main
Reviewed-on: #30
2026-06-15 08:33:41 +00:00
Eric Coissac 175ea5bbd0 feat: add pipeline concurrency throttling and HPC build docs
Introduces a counting semaphore-based throttling mechanism to limit concurrent file I/O and pipeline processing. Replaces custom path wrappers with standardized `Throttled` types across `obikmer` and `obikpartitionner`, ensuring RAII-based resource cleanup and explicit backpressure. Additionally, documents how to redirect Cargo build artifacts to local scratch storage on HPC filesystems to prevent compilation slowdowns.
2026-06-15 10:33:23 +02:00
coissac c6ea0c53e3 Merge pull request 'feat: implement NUMA-aware worker pools for merge command' (#29) from push-wusvurukprsr into main
Reviewed-on: #29
2026-06-14 21:57:21 +00:00
Eric Coissac ea767376bd feat: implement NUMA-aware worker pools for merge command
Replaces the global Rayon pool with per-NUMA-node thread pools that pin worker threads to their respective nodes, leveraging Linux first-touch allocation to reduce cross-NUMA memory contention and improve cache locality. Integrates the `hwlocality` crate with a vendored build, includes graceful fallbacks for single-socket or non-Linux systems, and updates dependency constraints. Also adds installation and architecture documentation, and corrects parallelism detection in the partitioner.
2026-06-14 23:56:52 +02:00
coissac f1d76f3203 Merge pull request 'refactor(merge): extract adaptive worker spawn logic' (#28) from push-yzruqtyqvopm into main
Reviewed-on: #28
2026-06-13 12:56:34 +00:00
Eric Coissac c4071eb450 refactor(merge): extract adaptive worker spawn logic
Centralize inline spawn checks into a `should_spawn_worker` function with adaptive thresholds. The first worker spawns at <95% CPU efficiency, while subsequent workers only trigger if marginal efficiency gain exceeds 25% of the expected `1/n_workers` (minimum 3%). Also increases the spawn poll interval from 10s to 20s.
2026-06-13 14:56:01 +02:00
coissac 817b02cbc1 Merge pull request 'Push zkspuxlpumpw' (#27) from push-zkspuxlpumpw into main
Reviewed-on: #27
2026-06-13 11:25:12 +00:00
Eric Coissac 547cb72d76 refactor: Enforce Rayon parallelism and fix merge_layer concurrency
Updated memory guidelines and feedback docs to explicitly classify intra-partition phases as parallel, correcting prior assumptions of sequential execution. Refactored merge_layer.rs to wrap column builders in Arc<Mutex<ColBuilder>> and use Arc::try_unwrap for safe concurrent access, eliminating race conditions and preventing double-closes during pass2.
2026-06-13 13:24:55 +02:00
Eric Coissac 6d85387077 feat: add performance instrumentation and dynamic worker scaling
This change enhances observability and adaptability in the merge pipeline. Performance timing and debug logging are added to the De Bruijn graph and partition merge layers to track phase durations and pipeline metrics. The merge module replaces blocking receives with timed polls to sample CPU efficiency, dynamically spawning workers when utilization drops below a threshold. A new script is also introduced to parse merge debug logs and generate structured Markdown reports detailing throughput, phase breakdowns, and partition performance.
2026-06-13 13:21:53 +02:00
coissac fb5b53dca9 Merge pull request 'Push ooxwzorvsqvy' (#26) from push-ooxwzorvsqvy into main
Reviewed-on: #26
2026-06-13 09:59:07 +00:00
Eric Coissac fddf630772 style: apply consistent formatting and whitespace normalization
Applies consistent formatting, whitespace normalization, and indentation standardization to `debruijn.rs` and `merge.rs`. Reorganizes imports and downgrades a unitig traversal log from `info!` to `debug!`. No functional logic or runtime behavior is altered.
2026-06-13 11:58:20 +02:00
Eric Coissac bc14346f5f feat: add CPU-aware parallel worker pool for partition merging
Introduce CpuSample to measure process-level CPU efficiency and wall-clock time. Use crossbeam-channel to distribute partition merging tasks to a dynamic worker pool that scales based on CPU utilization, capped at half the available cores. Update diagnostics to track pool usage.
2026-06-13 11:58:20 +02:00
Eric Coissac fb8c6e427c refactor: pass Unitig objects directly instead of raw byte slices
Refactored `try_for_each_unitig` and related pipelines across `obidebruinj` and `obikpartitionner` to accept `Unitig` instances directly. This eliminates manual `Unitig::from_nucleotides()` conversions, simplifies the data flow, and reduces unnecessary allocation overhead.
2026-06-13 11:52:50 +02:00
Eric Coissac 1f336fe496 refactor: replace mutex with channels for parallel debruijn processing
Add `rayon` and `crossbeam-channel` dependencies to support concurrent execution. Replace the synchronous, mutex-protected closure pattern with a channel-based producer-consumer approach using `std::thread::scope`. This decouples unitig iteration from processing, eliminating lock contention and `Mutex` overhead while enabling parallel workloads.
2026-06-13 11:49:27 +02:00
Eric Coissac 5f98d2ef96 refactor: replace explicit collect with Unitig::from_nucleotides
Introduce a thread-local buffer to materialize nucleotide iterators into contiguous slices. Update `try_for_each_unitig` across the debruijn, index, merge, and rebuild layers to directly instantiate `Unitig` via `from_nucleotides()` instead of explicitly collecting iterators. This eliminates intermediate allocations and aligns test code with the new approach.
2026-06-13 11:47:06 +02:00
Eric Coissac 8b563d0804 refactor: migrate pipeline stages and improve graph processing
Refactored neighbor resolution to explicitly track unvisited indices for degree-1 nodes, updated display formatting, and added timing and debug logging to the degree computation routine. Migrated pipeline stages from eager vector returns to explicit flat implementations, enabling backpressure-aware streaming, configurable batch processing, incremental yielding, and progress tracking via a delta channel.
2026-06-13 11:44:17 +02:00
coissac 7208dcbb4a Merge pull request 'refactor: defer SrcLayerData lookups in RawBatch' (#25) from push-nxrynoorswrw into main
Reviewed-on: #25
2026-06-12 20:19:21 +00:00
Eric Coissac 2e69b0b7fe refactor: defer SrcLayerData lookups in RawBatch
Replace eager resolution of `Vec<u32>` values with an `Arc<SrcLayerData>` handle passed alongside `Vec<CanonicalKmer>`. This shifts the lookup logic to the subsequent transform step, reducing memory overhead and enabling shared, thread-safe access to the source layer data.
2026-06-12 22:18:57 +02:00
coissac 9ea1dff5d6 Merge pull request 'Push rwqsmuvystym' (#24) from push-rwqsmuvystym into main
Reviewed-on: #24
2026-06-12 19:33:20 +00:00
Eric Coissac b2c8373586 refactor: parallelize merge layer with WorkerPool pipeline
Replaces the synchronous sequential loop with a multi-threaded pipeline using `WorkerPool` and custom stage macros. Shared mutable state is wrapped in `Arc<Mutex<>>` for thread-safe updates, while pipeline errors are centralized via `Arc<Mutex<Option<String>>>` to propagate failures before thread join.
2026-06-12 21:32:53 +02:00
Eric Coissac ba49af6f9e refactor: parallelize merge and partition logic with obipipeline
Introduce the `obipipeline` dependency and refactor merge and partition logic to leverage parallel execution. Update `merge_partitions` to use rayon with dynamic memory budgeting and concurrency control via a pilot run. Refactor Pass 1 to concurrently read unitigs, filter kmers through a shared `LayeredMap`, and populate the graph safely. Simplify diagnostics to report total kmer counts and replace manual flags with graph length validation.
2026-06-12 21:32:04 +02:00
Eric Coissac 2bc189e962 feat: dynamically compute seed expansion based on RSS
Introduce a `peak_rss_bytes()` utility for accurate per-phase RAM measurement. Replace the genome-length heuristic with a dynamic seed expansion ratio based on actual RSS delta. Explicitly drop the `GraphDeBruijn` instance before MPHF construction to prevent resource contention and ensure proper memory management.
2026-06-12 16:39:38 +02:00
coissac db9c604199 Merge pull request 'feat: enhance memory budgeting and add rebuild diagnostics' (#23) from push-nptzpkomspkv into main
Reviewed-on: #23
2026-06-12 13:21:12 +00:00
Eric Coissac 52fd2cf801 feat: enhance memory budgeting and add rebuild diagnostics
This commit improves memory management by respecting Linux cgroup v1/v2 limits and introduces a configurable memory budget for the new `rebuild` subcommand to prevent OOM during index reconstruction. The rebuild process now supports filtering, compaction, and parallelization. Diagnostic capabilities are expanded with debug-level tracing for partition merges, k-mer expansion tracking, and utility flags for label renaming, matrix size breakdowns, per-genome counts, and partition distribution reporting. Accessor methods for active and remaining memory have also been added to the stats struct.
2026-06-12 15:20:38 +02:00
coissac 97e3fb9761 Merge pull request 'Push ylnwstyzqwrt' (#22) from push-ylnwstyzqwrt into main
Reviewed-on: #22
2026-06-12 10:10:03 +00:00
Eric Coissac b5e027f23b feat: add memory-aware parallel merge scheduling and CLI flags
Introduces a memory-aware scheduling strategy for parallel partition merging that replaces unbounded concurrency with a First-Fit Decreasing approach gated by a thread-safe `MemoryBudget` semaphore. An adaptive expansion factor, seeded by a sequential pilot run, dynamically caps concurrent workers to prevent hashbrown OOMs. Adds a `--budget-fraction` CLI flag to configure RAM allocation, enhances the CLI to accept multiple indexes, and introduces comprehensive partition diagnostics including memory utilization tracking, concurrency metrics, and statistical summaries with ASCII histograms. Updates documentation and navigation accordingly.
2026-06-12 11:44:10 +02:00
Eric Coissac f44fe042bc feat: add parallel k-mer counting and stats CLI
Introduces allocation-free `sum()` and `count_nonzero()` methods for compact integer vectors, extending the `ColumnWeights` trait with `partial_kmer_counts`. Adds parallel partition scanning to the k-mer index for computing per-genome distinct k-mer counts, and exposes a new `--stats` CLI flag to output these statistics as CSV.
2026-06-12 11:29:32 +02:00
coissac 94e0a370b3 Merge pull request 'Push tmpsxsztwpxl' (#21) from push-tmpsxsztwpxl into main
Reviewed-on: #21
2026-06-09 13:31:25 +00:00
Eric Coissac 970460be42 refactor: rename rebuild subcommand to filter
Rename the `rebuild` CLI subcommand to `filter` to better reflect its primary purpose of row-level selection and k-mer filtering. Update all associated CLI arguments, logging, error messages, and module registrations accordingly. Introduce a dedicated `Rebuild` subcommand for index compaction, fully decoupling it from the filtering logic. Also refine related documentation to align with the new naming and semantics.
2026-06-09 15:26:37 +02:00
Eric Coissac e66adef23d feat: add select command for genome column projection and aggregation
Introduces the `select` CLI command to project and aggregate genome-level k-mer data by column. Adds `filter` as an alias for `rebuild`. The implementation uses parallel partition processing, supports metadata-driven grouping with configurable aggregation operators, and performs atomic in-place rewrites or filtered exports. Updates documentation and navigation accordingly.
2026-06-09 15:09:47 +02:00
coissac b0dab452f6 Merge pull request 'refactor: optimize dump partition iteration and add progress tracking' (#20) from push-xqswlxlvmyrq into main
Reviewed-on: #20
2026-06-09 09:34:13 +00:00
Eric Coissac db730e9cf6 refactor: optimize dump partition iteration and add progress tracking
Refactor partition iteration to support a generic `on_partition` callback executed after each parallel partition completes. Split the logic into bounded and unbounded paths; the bounded path uses an `AtomicUsize` to enforce row limits, while the unbounded path eliminates atomic contention to improve throughput. Additionally, integrate a progress bar into the dump command by passing an increment callback to `idx.dump()`, ensuring proper initialization and cleanup.
2026-06-09 11:07:48 +02:00
coissac f65ecd19cc Merge pull request 'Push lrwmyplxxzkn' (#19) from push-lrwmyplxxzkn into main
Reviewed-on: #19
2026-06-09 08:28:20 +00:00
Eric Coissac 7dd8db1409 docs: document conservative rounding strategy for filtering thresholds
Specifies that minimum bounds use ceiling and maximum bounds use floor to enforce strictness. Clarifies that the implementation avoids explicit rounding by directly comparing integer counts against floating-point fractions, which is mathematically equivalent.
2026-06-09 10:26:21 +02:00
Eric Coissac ce45e2fbe1 refactor: centralize k-mer filtering logic and add validation
Refactor shared `FilterArgs` and `build_group_filter` to return a `Result` with explicit validation for fraction bounds, min/max ordering, and count constraints. Update conditional defaults for `--min-frac` and `--max-outgroup-count` to depend on explicit quorum flags, preventing silent configuration conflicts. Update documentation and MkDocs navigation to reflect the new centralized k-mer filtering system across `rebuild`, `dump`, and `unitig` commands.
2026-06-09 10:22:25 +02:00
Eric Coissac 2465cfbc4b Parallelize partition iteration using Rayon
Introduce thread-local `Vec<u8>` buffers to eliminate concurrent I/O contention. Replace the mutable row counter with an `AtomicUsize` and `fetch_update` to enable lock-free early termination when the limit is reached. Collected chunks are then written sequentially to preserve partition ordering.
2026-06-09 10:04:25 +02:00
Eric Coissac d626d42ec7 feat: add --head and --presence-threshold to dump and distance
Introduces `--head N` to the `dump` command for early iteration termination and `--presence-threshold N` to the `distance` command for Jaccard filtering on count indexes. Updates filter defaults to adapt based on explicit ingroup/outgroup declarations. Fixes a Rust type mismatch in the unitig closure and updates partition iteration callbacks to return `bool` for proper early termination support. Documentation is updated accordingly.
2026-06-09 10:04:25 +02:00
coissac 650eea43b6 Merge pull request 'Push quqlpklvxsqx' (#18) from push-quqlpklvxsqx into main
Reviewed-on: #18
2026-06-08 18:15:01 +00:00
Eric Coissac eb7805c747 feat: add configurable presence threshold to kmer distance
Introduce a `--presence-threshold` CLI argument (default: 1) and update `KmerIndex::distance` to accept a `presence_threshold` parameter. This replaces hardcoded zero thresholds, enabling configurable filtering of low-abundance kmers during Jaccard distance calculations.
2026-06-08 20:14:33 +02:00
Eric Coissac 1ec65922df feat: implement parallel pairwise distance matrices
Introduces parallelized pairwise distance matrix computation for Jaccard, Hamming, Bray-Curtis, Euclidean, and Hellinger metrics across `Columnar`, `Packed`, and `Implicit` matrix variants. Adds trait methods and convenience wrappers, safely handles normalization and zero-denominator edge cases, and updates test suites to import required traits for validation.
2026-06-08 20:08:09 +02:00
Eric Coissac 09d9e21744 feat: integrate tracing and enhance bit matrix operations
Add the `tracing` crate to `obidebruinj`, `obisys`, and resolve it in `Cargo.lock`. Replace `eprintln!` statements with structured `debug!` and `info!` macros. Introduce a `TracedBar` wrapper for progress bars and enhance the `Stage` lifecycle to emit structured events for timing, memory metrics, and swap warnings. Add a progress spinner for unitig degree computation. Extend `PersistentBitMatrix` with columnar bit-vector operations and parallel distance methods, enabling uniform distance computations across all storage layouts while replacing previous panics with dimension-based fallbacks.
2026-06-08 19:55:06 +02:00
coissac 3f47e22083 Merge pull request 'Push pvqkqxlkkwry' (#17) from push-pvqkqxlkkwry into main
Reviewed-on: #17
2026-06-06 04:44:10 +00:00
Eric Coissac 03c7bb0b99 Relax unitig assertion in debruijn test
Replace the strict `unitigs.len() == 1` assertion with a non-empty check to allow multiple unitigs. Update the test comment to describe the general non-repetitive sequence recovery principle instead of a specific example. The core k-mer roundtrip validation logic remains unchanged.
2026-06-06 06:41:45 +02:00
Eric Coissac b39eee688a refactor(debruijn): unify graph traversal with WalkState iterator
Replaces deeply nested branching with early returns and `then_some`. Introduces a cycle-detecting `find_chain_start` method and updates `UnitigNucIter` to use step-based iteration with atomic node claiming. This eliminates nested iterators and redundant state management, improving code readability and maintainability.
2026-06-06 06:38:28 +02:00
Eric Coissac 95b3461405 refactor: centralize graph traversal logic in walk
Refactor `leavable` and `reachable` to eliminate duplicated graph traversal logic by mutually delegating via `WalkState`. `leavable` now returns `self.walk(graph).is_some()`, while `reachable` delegates to the inverted `direct` state's `leavable` check. This centralizes kmer extension and visited-state validation in `walk`, simplifying control flow and reducing code duplication.
2026-06-06 06:36:48 +02:00
Eric Coissac 952a21eef8 refactor: remove naked_asm and extract collect_unitigs helper
Remove the `std::arch::naked_asm` import as it is no longer required. Introduce a `collect_unitigs` helper to abstract nucleotide sequence extraction from `GraphDeBruijn`, and refactor the test suite to use it, eliminating repetitive collection code and standardizing graph iteration logic.
2026-06-06 04:33:59 +02:00
Eric Coissac 5c2f48535f refactor: rename compute_degrees and mark start nodes
Renames `compute_degrees` to `compute_degrees_and_mark_starts` across the De Bruijn graph and partitioner layers to consolidate degree calculation and start-node flagging. Introduces safe neighbor iteration methods and a debug validation block to verify graph consistency. Refactors unitig extraction to use sequential execution with a `Mutex` for safe error propagation. Fixes malformed and duplicated method calls, adds auto-generation of missing `meta.json` files, and ensures persistent matrix builders are explicitly closed to finalize metadata.
2026-06-05 19:48:59 +02:00
Eric Coissac 27088ab810 refactor: optimize unitig iteration and graph traversal
Switches unitig processing to a lazy, fallible `try_for_each_unitig` API across partitioner layers, reducing intermediate allocations and enabling proper error propagation. Refactors de Bruijn graph traversal into a two-pass algorithm with explicit node flags, named constants, and diagnostic logging. Introduces parallel chain processing and staged performance profiling for the unitig command, and adds a memory-efficient `FromIterator` implementation for packed nucleotide sequences.
2026-06-05 19:48:59 +02:00
coissac ea2c594c86 Merge pull request 'Push ruqusmkoyvwm' (#16) from push-ruqusmkoyvwm into main
Reviewed-on: #16
2026-06-05 08:41:08 +00:00
Eric Coissac d202ead385 feat: parallelize unitig extraction and FASTA output
Replace the non-atomic `set_visited` with atomic `fetch_or` bitmask operations to enable thread-safe node claiming. Introduce a two-phase extraction pipeline where `par_for_each_chain_unitig` builds chains in parallel and `for_each_remaining_unitig` sequentially handles residual cycles and junctions. Add `is_start` and `collect_from_start` to explicitly define unitig boundaries. Wrap `BufWriter` in a `Mutex` and use an `AtomicUsize` counter to ensure thread-safe concurrent FASTA output, refactoring the write logic into a shared closure for safe multi-threaded execution.
2026-06-05 10:33:52 +02:00
Eric Coissac 249998beed perf: add structured performance profiling for unitig stages
Wraps graph construction, degree computation, and unitig enumeration phases with `Stage` start/stop calls. Intervals are recorded in a `Reporter` instance and printed upon completion to provide granular timing metrics for each computational stage.
2026-06-05 10:28:45 +02:00
Eric Coissac 2f29ee2240 feat: Add parallel execution and thread-safe graph operations
Integrate rayon to enable parallel processing of k-mer partitions and degree computation. Replace Cell with AtomicU8 to ensure thread-safe node state management, and add a merge method for combining disjoint graphs. Additionally, introduce progress tracking utilities and a test-utils feature flag for development dependencies.
2026-06-04 23:22:55 +02:00
Eric Coissac edd5e3f8ee feat: add bits-per-kmer diagnostic and stats module
Introduce a `stats` module to compute normalized storage efficiency metrics. The new `KmerIndex::bits_per_kmer()` method parallelizes disk I/O across partitions to aggregate file sizes for MPHF, evidence, and matrix components. Publicly export `IndexBitsPerKmer` and add a `--bits-per-kmer` CLI flag to trigger the diagnostic routine and print detailed statistics.
2026-06-04 23:17:17 +02:00
Eric Coissac bb7adc1154 docs: expand kmer indexing, filtering, and merging documentation
Expands MkDocs navigation and documentation for evidence elimination, the merge command, and kmer filtering. Refactors kmer representation to a generic `KmerOf<L>` type with a bitwise reverse complement algorithm. Unifies MPHF construction, introduces approximate fingerprint-based indexing, and updates the pipeline, chunkreader, and storage layouts. Adds code coverage reports and clarifies architectural invariants for improved maintainability.
2026-06-04 22:59:41 +02:00
Eric Coissac 9306ec1c56 perf: Replace manual window tracking with monotonic deque algorithm
Eliminates intermediate allocations by computing per-genome window minimums (`win_min`) directly. Unifies the `z ≤ 1` and `z > 1` branches into a single buffer-reused accumulation loop, efficiently validating k-mer presence.
2026-06-04 21:37:09 +02:00
Eric Coissac 712a03a3a6 refactor: replace unitig extraction with de Bruijn graph pipeline
This change replaces direct partition-based extraction with a pipeline that reconstructs a de Bruijn graph from filtered k-mers. It introduces `FilterArgs` for k-mer selection, collects filtered k-mers in parallel into a `GraphDeBruijn`, computes node degrees, and enumerates unitigs from the graph for output instead of reading pre-computed partition files.
2026-06-04 21:32:49 +02:00
Eric Coissac 3e62ffe010 feat: add selective k-mer filtering to dump and rebuild commands
Add the `obidebruinj` dependency and introduce `FilterArgs` CLI arguments for ingroup/outgroup predicates and count/fraction thresholds. Extend `GroupFilterParams` to support outgroup filtering, and integrate the filter collection into `KmerIndex::dump` and `rebuild` commands. This enables selective k-mer filtering during index operations and CSV exports.
2026-06-04 21:29:58 +02:00
Eric Coissac a1499e6153 feat: add kmer filtering and refactor layer iteration
Introduce a `passes_all` utility to validate kmer rows against multiple filters using short-circuit logic. Integrate a `filters` parameter into the iteration functions to conditionally emit kmers based on filter results. Extract repetitive layer traversal and filtering into an `iter_src_layers` helper, refactoring Pass 1 and Pass 2 to eliminate duplication. Additionally, add a debug conditional to the dump output to include partition and layer metadata alongside kmer sequences.
2026-06-04 21:08:15 +02:00
Eric Coissac 476c7a6394 feat: add metadata-driven k-mer filtering for rebuild command
Introduces a metadata-driven filtering system for the rebuild command, classifying genomes into ingroup and outgroup categories using exact, inequality, and hierarchical path predicates. Implements a GroupQuorumFilter to enforce configurable presence thresholds and fraction constraints per group. Refactors the command to replace global quorum filters with this unified approach, converts the presence flag to a threshold parameter, and adds corresponding documentation and MkDocs navigation.
2026-06-04 21:01:58 +02:00
coissac edc18b4908 Merge pull request 'Push rrwpnquuzsvr' (#15) from push-rrwpnquuzsvr into main
Reviewed-on: #15
2026-06-03 17:04:25 +00:00
Eric Coissac 02cb30c0ef feat: add obisys crate for standardized CLI progress reporting
This commit introduces the `obisys` crate, which wraps `indicatif` to provide reusable `spinner` and `progress_bar` utilities with consistent styling and tick intervals. It refactors progress reporting across `obikindex`, `obikpartitionner`, and `obikmer` to use these shared functions, eliminating inline UI configuration and ensuring uniform terminal feedback.
2026-06-03 19:03:59 +02:00
Eric Coissac 4677d6f177 refactor: improve resource cleanup and index packing
Explicitly close file handles and remove temporary artifacts after serialization to prevent disk space leaks. Additionally, compact internal matrix structures immediately upon loading the KmerIndex to improve memory efficiency and prepare for downstream operations.
2026-06-03 15:35:56 +02:00
coissac 7a29ca6305 Merge pull request 'Push ywwwypqxrtmy' (#14) from push-ywwwypqxrtmy into main
Reviewed-on: #14
2026-06-03 13:18:41 +00:00
Eric Coissac bba5147f0f fix: account for k-mer overlap in total_bases calculation
Introduces a `kmer_overlap` variable (`k-1`) and modifies the `total_bases` accumulation to subtract this overlap from each sequence's length. This ensures the base count accurately reflects only valid k-mer starting positions rather than raw sequence length.
2026-06-03 15:11:48 +02:00
Eric Coissac bfe0cb4b82 feat: integrate obikseq to configure global k-mer and minimizer sizes
This change adds the `obikseq` crate as a local dependency and inserts `set_k` and `set_m` calls across index creation and command modules. By synchronizing the runtime's global k-mer and minimizer dimensions with the loaded index parameters, downstream sequence processing and partitioning operations now consistently use the correct structural constraints.
2026-06-03 14:31:14 +02:00
Eric Coissac 173ac9fb42 feat: introduce packed matrix storage and layer metadata
Unifies bit and integer matrix storage into `PersistentBitMatrix` and `PersistentCompactIntMatrix` enums, supporting both columnar and memory-mapped single-file layouts. Introduces `LayerMeta` to persist layer dimensions as `layer_meta.json`, enabling correct initialization of implicit presence matrices. Adds CLI commands (`pack` and `--upgrade-index`) to convert existing columnar indices to the compact format and backfill missing metadata. Updates partitionner and layered map logic to use the new persistent builders, optimized memory allocation, and auto-detected storage backends.
2026-06-03 14:16:04 +02:00
Eric Coissac de1a41810a perf: enable zero-allocation queries and memory-mapped indexes
Introduce zero-allocation row extraction and query result buffers across `obicompactvec` and `obikpartitionner` to eliminate per-kmer heap allocations. Replace in-memory MPHF deserialization with memory-mapped, zero-copy views to reduce runtime memory footprint. Add configurable I/O chunking, a RAM-aware `--chunk-size` parameter, and system memory monitoring via the new `sysinfo` dependency. Re-export `PreloadedIndex` for external consumers.
2026-06-03 10:24:12 +02:00
Eric Coissac 1661dd6b1c feat: introduce preloaded index cache and thread-safe progress tracker
Introduce `PreloadedIndex` to cache partition indices and eliminate redundant I/O during repeated queries. Refactor the query pipeline to route through this pre-loaded index, and expose it publicly in `obikpartitionner`. Additionally, add a thread-safe, lazily-initialized `MultiProgress` singleton for improved progress tracking.
2026-06-03 09:42:18 +02:00
Eric Coissac 2ebc5f0d75 chore: add logging infrastructure to merge routine
Adds comprehensive logging for source metadata, merge modes, and forced approximation detection. Introduces `format_evidence` and `is_trivial` helpers to format `IndexMode` variants and identify single-genome presence indices. The core merge algorithm remains unmodified, with all changes focused on enhanced runtime observability.
2026-06-01 15:23:37 +02:00
coissac 4fd0eb989f Merge pull request 'Push pnxswqpxlyso' (#13) from push-pnxswqpxlyso into main
Reviewed-on: #13
2026-06-01 12:45:46 +00:00
Eric Coissac add6d7f873 enforce uniform index mode and optimize base index selection
Adds validation to ensure all input sources share the same `IndexMode`. Introduces base index selection logic that prioritizes approximate or hybrid evidence and maximizes base size to minimize newly indexed k-mers. Includes helper functions for triviality evaluation, cumulative size calculation, and mode consistency checks.
2026-06-01 14:43:51 +02:00
Eric Coissac 0350ca855b refactor: streamline merge pipeline and MPHF indexing
Replace mphf.find() with direct mphf.index() calls to eliminate absence checks and fallback vectors. Introduce a lightweight MphfOnly wrapper for faster index loading, and standardize k-mer iteration across merge and rebuild layers. Update IndexMeta configuration and n_new calculation to leverage MPHF cardinality, streamlining the overall merge pipeline.
2026-06-01 14:37:35 +02:00
Eric Coissac 1e2115a1b0 docs: Update README to reflect new indexing workflow
Replace the documented direct combined indexing command with a two-step workflow. This involves building separate exact indexes per genome, merging them into a single multi-genome index via `obikmer merge`, and keeping the approximate index conversion unchanged.
2026-06-01 13:56:48 +02:00
Eric Coissac 657f964dda refactor: abstract k-mer types and fix bit alignment
Abstracts k-mer storage using a `RawKmer` alias and `KMER_BITS` constant to simplify bit manipulation and enable future extension to larger types. Updates bit-shifting and masking logic across `kmer.rs` and `packed_seq.rs` to prevent overflow and improve type safety. Adapts the MPHF layer to iterate over indexed canonical k-mers with explicit slot bounds validation and bit-level encoding. Fixes test suite compilation errors by correcting method names, adding tuple destructuring, and passing the required `IndexMode::Exact` parameter.
2026-05-31 20:46:49 +02:00
Eric Coissac 8b57c91ab7 feat: add iter_canonical_kmers iterator and tests
Implements `iter_canonical_kmers` in `unitig_index` to yield canonical kmers by traversing unitig chunks. Includes unit tests to verify that the iterator exclusively yields canonical kmers and matches `iter_kmers` in count and canonical equivalence.
2026-05-30 16:30:02 +02:00
coissac 728476a0a6 Merge pull request 'Push rxrxptuxltlp' (#12) from push-rxrxptuxltlp into main
Reviewed-on: #12
2026-05-30 14:03:13 +00:00
Eric Coissac 8a0b898b4b docs: clarify query pipeline, Findere trick, and input formats
Fix a stray prefix in the README heading and update documentation to reflect the query pipeline's operation on `s-mers` (`s = k - z + 1`) with post-partition z-window filtering. Clarify the Findere trick, including k-mer size reduction, consecutive match requirements, and false positive rate calculations. Additionally, expand input format documentation to cover supported file extensions, gzip compression, recursive directory handling, and `query` command specifications.
2026-05-30 15:59:12 +02:00
Eric Coissac 708b0abf9b refactor: aggregate query results at sequence level
Refactor the query pipeline to buffer partition outputs into a per-sequence `seq_results` vector, deferring final accumulation until all partitions complete. This ensures global position ordering before computing k-mer presence, counts, and coverage statistics. Additionally, removes a resolved TODO and documents a known BLAST false-positive issue where chloroplast and bacterial contaminants yield unrealistic high-confidence matches.
2026-05-30 07:18:54 +02:00
coissac 3138f6382c Merge pull request 'Push nvyqwlpspwvl' (#11) from push-nvyqwlpspwvl into main
Reviewed-on: #11
2026-05-29 07:21:58 +00:00
Eric Coissac 86b88acb95 feat: implement approximate k-mer indexing and optimize query
Enable approximate k-mer indexing via the `--approx` flag, computing an effective k-mer size of `k - z + 1` and configuring the appropriate indexing mode with validated probabilistic parameters. Refactor the Findere z-window filter in the query command to improve performance and correctness by replacing the precomputed vector with a lazy closure, optimizing cache locality, and fixing a variable naming bug.
2026-05-29 09:20:44 +02:00
Eric Coissac be0e8f1041 refactor(query): parallelize query execution with obipipeline
Extracts chunk processing into a dedicated function and introduces a QueryData enum with unsafe Send/Sync implementations to safely distribute Rope chunks across worker threads. Replaces nested iteration with a flat iterator and parallel block processing. Adds CLI argument parsing for presence, threshold, and detail flags to configure the pipeline.
2026-05-29 09:18:54 +02:00
Eric Coissac eaa52eaab5 feat: introduce nucstream abstraction and comprehensive test suite
Introduces a unified NucStream abstraction with NucPageCursor for byte-offset tracking and MIME-type dispatch to instantiate format-specific parsers. Exposes nuc_stream and open_nuc_stream APIs that return boxed, Send-compatible iterators. Additionally, adds a comprehensive test suite covering chunk boundary alignment, FASTA/FASTQ record parsing, sequence normalization, and edge cases such as CRLF line endings, @ in quality strings, and multi-slice rope processing.
2026-05-29 09:17:24 +02:00
Eric Coissac cfadf63bbc refactor: migrate pipeline to NucPage-based stream processing
Replace the existing chunk and Rope-based processing pipeline with a fixed-size NucPage architecture. Introduce a new nucstream module featuring buffer-pooled, in-place parsing that auto-detects and decompresses FASTA/FASTQ/GenBank inputs into normalized ACGT streams with k-mer overlap preservation. Update obikmer scatter and superkmer stages to consume NucPage iterators and cursor-based navigation, eliminating std::io::Read dependencies and optimizing memory management. Add a configurable max_open_files CLI argument and update implementation documentation to reflect the new record vs. stream reading paths.
2026-05-29 09:10:25 +02:00
Eric Coissac a4b57a96de feat: add streaming sequence reader and superkmer iterator
Introduce the `obiread` crate with a streaming byte normalizer that processes FASTA, FASTQ, and GenBank files using a 64 KiB ring buffer for O(1) memory usage. Integrate this crate into `obiskbuilder` to provide `SuperKmerStreamIter`, enabling memory-efficient superkmer traversal with rolling entropy and minimizer-based cut conditions.
2026-05-29 09:10:25 +02:00
Eric Coissac 0d9be53d1f feat: enforce runtime validation for kmer and minimizer parameters
Introduces `CommonArgs::validate()` to enforce strict constraints on `--kmer-size` (odd, 11–31), `--minimizer-size` (odd, 3–k−1), and `z` (strictly less than k). This validation is applied at the entry point of the `superkmer` and `index` commands to prevent invalid configurations, avoid palindromes, prevent u64 overflow, and ensure positive effective indexing sizes. Documentation is updated to reflect these runtime checks and immediate termination on invalid input.
2026-05-29 09:10:25 +02:00
coissac 82ec6aa1cf Merge pull request 'Push tklvqnrqtzpo' (#10) from push-tklvqnrqtzpo into main
Reviewed-on: #10
2026-05-26 15:41:05 +00:00
Eric Coissac 694da5208e feat: add Findere z-window filtering and detail mode coverage tracking
Introduces the `--findere-z` CLI flag to override the index's sliding window parameter and implements `apply_findere` to filter k-mer hits using a z-consecutive positions window. Adds structural support for `--detail` mode, including per-sequence k-mer offsets, conditional allocation of per-position coverage vectors, and JSON serialization. Updates architecture documentation, CLI references, and annotation schemas to align with the new implementation, resolving prior discrepancies with `--detail` and `--mismatch` flags.
2026-05-26 15:43:17 +02:00
Eric Coissac 26ab165807 refactor: add rolling buffer methods and document label constraints
Added `is_empty()`, `clear()`, and `iter()` methods to the rolling statistics buffer to enable standard traversal and state reset operations. Documented genome label constraints, specifying forbidden characters, empty label rejection, space quoting requirements, and auto-derived label bypass rules. Additionally, updated doc comments and added `#[allow(dead_code)]` attributes for `kmer_offset` and `n_kmers` fields to suppress compiler warnings while reserving them for future `--detail` coverage vector logic.
2026-05-26 15:40:23 +02:00
Eric Coissac dfa0b2bac2 feat: add utils subcommand for renaming genome labels
Introduces a `utils` CLI subcommand to enable in-place genome label renaming without full reindexing. Adds strict label validation to reject empty strings, filesystem separators, and control characters, ensuring safe CSV serialization. Updates index metadata, renames corresponding spectrum JSON files, and registers the command in the main dispatch logic. CLI reference documentation is also updated.
2026-05-26 15:35:22 +02:00
Eric Coissac 9e60a711bc Enforce minimum input paths and handle stdin sentinel
Update CLI validation to require at least 10 input paths, defaulting to stdin (`-`) when the argument list is empty. Refactor the path iterator to explicitly recognize the stdin sentinel, bypassing extension validation and directory expansion to ensure direct passthrough to the file buffer without triggering `stat()` or recursive traversal.
2026-05-26 15:22:38 +02:00
Eric Coissac 98c14aade9 feat: centralize index configuration and add hybrid mode
Centralizes index configuration by storing a single `IndexMode` (`Exact`, `Approx`, or `Hybrid`) in `PartitionMeta`, eliminating per-layer metadata files. Introduces a `Hybrid` evidence mode and an `--approx` CLI flag to toggle between exact and probabilistic indexing. Refactors the build and query pipelines to dynamically dispatch based on the configured mode, deferring `.idx` generation to Pass 2 and only requiring it for Exact/Hybrid modes. Updates layer opening to load appropriate data structures, enforces strict parameter validation during merges, and clarifies performance trade-offs in documentation.
2026-05-26 15:08:29 +02:00
Eric Coissac 7501b6e854 refactor: switch indexing to IndexMode and update metadata
Replace EvidenceKind with IndexMode (Exact, Approx, Hybrid) across layer construction and query dispatch. Update PartitionMeta and LayerMeta serialization to centralize index-wide configuration. Add flexible push_layer overloads to LayeredMap for dynamic index expansion without full rebuilds. Improve UnitigFileReader to gracefully fallback to sequential scanning when indexes are missing, eliminating panics.
2026-05-26 14:57:17 +02:00
coissac 6f7abddeaf Merge pull request 'Push vqmnuyrkpxot' (#9) from push-vqmnuyrkpxot into main
Reviewed-on: #9
2026-05-26 09:05:50 +00:00
Eric Coissac 1d880fdc5f refactor: optimize MPHF construction and update legacy guidelines
Replaces parallel random-access unitig iteration with a sequential mmap-based iterator for MPHF construction, eliminating the build-time `.idx` dependency by deferring index generation until after persistence. Updates `CLAUDE.md` to treat existing code as a hypothesis, mandating proactive removal of obsolete legacy constructs rather than preserving them out of inertia.
2026-05-26 10:54:59 +02:00
Eric Coissac 009a328c58 refactor: handle kmer deduplication and layer initialization concurrently
Introduce a secondary layer iteration to open `SrcLayerData` alongside the unitig reader for concurrent metadata access. This refactors the merge routine to handle kmer deduplication and per-layer data initialization simultaneously. Also corrects a typo in `rebuild_layer.rs` where `openopen_sequential` is renamed to `open_sequential`.
2026-05-26 10:52:08 +02:00
Eric Coissac 9d46400898 feat: support exact and approximate evidence in layer construction
Refactored `MphfLayer::build` to accept an `EvidenceKind` parameter, routing to exact (index-based, parallel MPHF, writes `evidence.bin`) or approximate (sequential mmap iterator, writes `fingerprint.bin`) pipelines. Introduced `CanonicalKmerIter` for memory-mapped, chunked k-mer iteration with O(1) resets via `Arc<Mmap>`. Updated layer and map APIs to forward evidence kind, added `push_layer` for count matrices, and adjusted tests and public exports accordingly.
2026-05-26 10:23:43 +02:00
Eric Coissac 036d044291 refactor: update core types and add approximate evidence support
Refactor `Kmer`, `SuperKmer`, and chunk reader into optimized, generic representations with compile-time length parameters and bitwise operations. Update the pipeline and scheduler to support batch processing, 1→N flat transformations, and multi-source merging. Introduce an approximate evidence mode using b-bit fingerprints and `.idx` files, alongside existing exact mode. Update CLI documentation, minimizer selection, and query output schema accordingly.
2026-05-26 10:04:25 +02:00
coissac 88365e444c Merge pull request 'Push kztouvrzoqym' (#8) from push-kztouvrzoqym into main
Reviewed-on: #8
2026-05-23 12:04:50 +00:00
Eric Coissac da56c3e290 docs: update architecture and storage specs for approximate index
Restructure architecture documentation to reflect the decoupled `MphfLayer` design wrapped by `LayeredStore<S>` and enforce strict multi-genome column invariants. Introduce the approximate index architecture, replacing exact `evidence.bin` with compact `fingerprint.bin` using B-bit fingerprints and z-consecutive k-mer matching. Update CLI flags, add `reindex`/`estimate` workflows, and refactor APIs to support separate exact/approximate evidence handling. Finally, provide a comprehensive on-disk layout specification, including the pipeline state machine, JSON schemas, binary formats, and refined Strategy B unitig evidence details.
2026-05-23 13:54:31 +02:00
Eric Coissac b7db3a33ed docs: add coverage reference files and flag architectural drift
These files catalog test coverage for Rust modules across architecture, implementation, and theory sections. They track recent structural changes, flag areas prone to documentation drift, and mandate verification of key parameters and routing logic to maintain alignment with the active codebase.
2026-05-23 13:44:23 +02:00
Eric Coissac b2a52bfb37 perf: optimize chunk_start for single-block indexing
Bypasses bitwise shift and mask operations when `block_bits == 0`, directly indexing `self.block_offsets[i]` instead. This eliminates unnecessary arithmetic overhead for single-block cases while preserving the original block-based offset calculation for larger block sizes.
2026-05-23 13:34:05 +02:00
Eric Coissac bc51cd9861 feat: add configurable block sizes and in-place reindex command
Propagate configurable block size (`block_bits`) through index and layer construction to control unitig chunking and optimize memory/performance trade-offs. Introduce an in-place `reindex` command and library method to convert indices between exact and approximate evidence formats. Add validation to reject merging indexes with mismatched evidence types, and update parallel kmer counting to use `AtomicUsize` for thread-safe aggregation. Includes CLI argument parsing, metadata persistence, and updated tests.
2026-05-23 13:28:24 +02:00
Eric Coissac 876bc0127f feat: add approximate evidence matching and index estimation CLI
Introduces a new `estimate` CLI subcommand to calculate bloom filter size, evidence bits, and false-positive rates for approximate indexing. Updates the index building and querying pipelines to support both exact and approximate evidence types via a unified `EvidenceKind` abstraction. Refactors `MphfLayer` and partition index builders to route operations based on the selected evidence mode, and adds the required `obilayeredmap` dependency.
2026-05-23 13:16:49 +02:00
Eric Coissac 16a6b0d033 feat: add evidence metadata and configurable k-mer parameters
Introduces `EvidenceKind` and `LayerMeta` structs to manage per-layer evidence configuration and false-positive parameters. Adds JSON serialization for layer metadata persistence and updates `build_approx_evidence` to accept a `z` parameter for consecutive k-mer thresholds. Exposes these types publicly and documents a future `aggregate` command for merging index matrix columns.
2026-05-23 13:10:18 +02:00
Eric Coissac e1dab86daf feat: add approximate kmer fingerprinting with memory-mapped storage
Introduce a new `fingerprint` module that stores packed B-bit vectors via memory-mapped files. Expose the module publicly and add `build_approx_evidence` to `Layer` and `MphfLayer` for generating compact `fingerprint.bin` files. Implement `find_approx` for fast, probabilistic kmer lookups using configurable bit-widths. Update dependencies to `bitvec` v1 and add `cacheline-ef`, `epserde`, and `memmap2` to support the new storage and serialization logic.
2026-05-23 13:07:02 +02:00
Eric Coissac 24afd74e2f refactor: decouple unitig index generation and add exact evidence
Decouple index generation by introducing `build_unitig_idx()` for retroactive `.idx` creation and optional immediate writing on close. Add `open_sequential()` for index-less iteration while enforcing index requirements for random access. Refactor the MPHF layer to pre-generate the unitig index for parallel random access, integrate `rayon` for k-mer processing, and enforce mapping integrity via duplicate slot validation. Additionally, implement `build_exact_evidence()` to reconstruct evidence from existing artifacts, and update tests to leverage the new index generation and simplified k-mer iteration helpers.
2026-05-23 13:02:25 +02:00
Eric Coissac 8478072b78 feat: make index granularity configurable via block_bits
Replaces the hardcoded BLOCK_SIZE constant with a configurable block_bits parameter, enabling variable index granularity to balance index size and sequential scan cost. Both the reader and writer now store block_bits and a precomputed mask for branchless offset arithmetic, while the index file format is upgraded to UIX3 to persist the configuration. Comprehensive unit tests verify serialization, chunk offset indexing, random access consistency, and kmer count accuracy across various block sizes.
2026-05-23 12:57:56 +02:00
Eric Coissac 4a5ab0b8c2 feat: optimize unitig index and document evidence elimination
Replace the dense per-chunk offset index with a sparse block-sampled structure (64 chunks per block), reducing the index file size by approximately 300× while preserving O(1) k-mer extraction. Introduce a design document for eliminating the `evidence.bin` file, which accounts for ~66% of the lookup layer, by transitioning to fingerprint-based approximate indexing and value-based MPHF lookups. Update MkDocs navigation to include the new documentation and add a file count tracker to the scatter step progress bar for improved observability.
2026-05-23 12:53:42 +02:00
628 changed files with 177343 additions and 22748 deletions
Vendored
BIN
View File
Binary file not shown.
+35
View File
@@ -0,0 +1,35 @@
pname: CI
on:
pull_request:
branches: ['main']
jobs:
build:
runs-on: ubuntu-latest
defaults:
run:
working-directory: src
steps:
- uses: actions/checkout@v4
- name: Install Rust
run: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
- name: Cache cargo registry
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
src/target
key: ${{ runner.os }}-cargo-v2-${{ hashFiles('src/Cargo.lock') }}
restore-keys: ${{ runner.os }}-cargo-v2-
- name: Build
run: cargo build --release
- name: Test
run: cargo test --release
+127
View File
@@ -0,0 +1,127 @@
name: Release
on:
push:
tags:
- "v*"
jobs:
create-release:
runs-on: ubuntu-latest
outputs:
release_id: ${{ steps.create.outputs.release_id }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Create Gitea release
id: create
env:
GITEA_TOKEN: ${{ secrets.GITEATOKEN }}
TAG: ${{ github.ref_name }}
run: |
sudo apt-get update -qq && sudo apt-get install -y -qq jq
body=$(git for-each-ref --format='%(contents)' "refs/tags/$TAG")
release_id=$(curl -s -X POST \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases" \
-H "Authorization: token $GITEA_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"tag_name\":\"$TAG\",\"name\":\"$TAG\",\"body\":$(echo "$body" | jq -Rs .)}" | jq -r '.id')
echo "release_id=$release_id" >> $GITHUB_OUTPUT
build-linux-x86_64:
needs: create-release
runs-on: ubuntu-latest
defaults:
run:
working-directory: src
steps:
- uses: actions/checkout@v4
- name: Install Rust + zigbuild
run: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
sudo apt-get update -qq && sudo apt-get install -y -qq jq
pip install ziglang --quiet --break-system-packages
$HOME/.cargo/bin/cargo install cargo-zigbuild
$HOME/.cargo/bin/rustup target add x86_64-unknown-linux-musl
- name: Create musl C/C++ wrappers
run: |
ZIG=$(python3 -c "import ziglang, os; print(os.path.join(os.path.dirname(ziglang.__file__), 'zig'))")
printf '#!/bin/sh\nexec "%s" cc -target x86_64-linux-musl "$@"\n' "$ZIG" | sudo tee /usr/local/bin/x86_64-linux-musl-gcc > /dev/null
printf '#!/bin/sh\nexec "%s" c++ -target x86_64-linux-musl "$@"\n' "$ZIG" | sudo tee /usr/local/bin/x86_64-linux-musl-g++ > /dev/null
sudo chmod +x /usr/local/bin/x86_64-linux-musl-gcc /usr/local/bin/x86_64-linux-musl-g++
- name: Cache cargo registry
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
src/target
key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock') }}
restore-keys: linux-musl-cargo-
- name: Build static binary
env:
PKG_CONFIG_ALLOW_CROSS: "1"
run: cargo zigbuild --release --target x86_64-unknown-linux-musl
- name: Prepare and upload artifact
env:
GITEA_TOKEN: ${{ secrets.GITEATOKEN }}
RELEASE_ID: ${{ needs.create-release.outputs.release_id }}
run: |
mkdir -p /tmp/dist
cp target/x86_64-unknown-linux-musl/release/obikmer /tmp/dist/obikmer-linux-x86_64
strip /tmp/dist/obikmer-linux-x86_64
curl -s -X POST \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases/$RELEASE_ID/assets" \
-H "Authorization: token $GITEA_TOKEN" \
-F "attachment=@/tmp/dist/obikmer-linux-x86_64"
build-macos-arm64:
needs: create-release
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Login to registry
run: echo "${{ secrets.REGISTRYTOKEN }}" | docker login registry.metabarcoding.org -u ${{ secrets.REGISTRYUSER }} --password-stdin
- name: Cache cargo registry
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
src/target
key: macos-arm64-cargo-${{ hashFiles('src/Cargo.lock') }}
restore-keys: macos-arm64-cargo-
- name: Build macOS binary
run: |
CID=$(docker create \
-w /src/src \
registry.metabarcoding.org/cibuilder/rustcrossosx:latest \
cargo build --release --target aarch64-apple-darwin --no-default-features)
docker cp . "$CID:/src"
docker start -a "$CID"
STATUS=$(docker wait "$CID")
mkdir -p /tmp/dist
docker cp "$CID:/src/src/target/aarch64-apple-darwin/release/obikmer" /tmp/dist/obikmer-macos-arm64
docker rm "$CID" > /dev/null
[ "$STATUS" -eq 0 ]
- name: Prepare and upload artifact
env:
GITEA_TOKEN: ${{ secrets.GITEATOKEN }}
RELEASE_ID: ${{ needs.create-release.outputs.release_id }}
run: |
curl -s -X POST \
"${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases/$RELEASE_ID/assets" \
-H "Authorization: token $GITEA_TOKEN" \
-F "attachment=@/tmp/dist/obikmer-macos-arm64"
+36 -1
View File
@@ -1,11 +1,46 @@
.venv/
.DS_Store
.claude/
.kilo/
.serena/
.zed/
.ast-cache/
CLAUDE.md
memory/
sandbox/
src/target
data-stress
*.fasta
*.fasta.gz
*.fastq
*.fastq.gz
*.vcf
*.zst
*.zst.meta
*.pb
./**/*.json
**/*.json
*.bin
*.log
*.csv
*.meta
*.pfiv
*.done
*.efh
*.efl
*.bin.idx
*.prsb
Betula_exilis--IGA-24-33
benchmark/run/
TNT
phyg
biblio
*.tnt
*.tre
*.phy
*.treefile
*.bionj
*.iqtree
*.mldist
*.parstree
*.ckp.gz
*.model
+2
View File
@@ -0,0 +1,2 @@
/cache
/project.local.yml
+169
View File
@@ -0,0 +1,169 @@
# the name by which the project can be referenced within Serena/when chatting with the LLM.
project_name: "obikmer"
# the encoding used by text files in the project
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
encoding: "utf-8"
# line ending convention to use when writing source files.
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
line_ending:
# The language backend to use for this project.
# If not set, the global setting from serena_config.yml is used.
# Valid values: LSP, JetBrains
# Note: the backend is fixed at startup. If a project with a different backend
# is activated post-init, an error will be returned.
language_backend:
# whether to use project's .gitignore files to ignore files
ignore_all_files_in_gitignore: true
# advanced configuration option allowing to configure language server-specific options.
# Maps the language key to the options.
# The settings are considered only if the project is trusted (see global configuration to define trusted projects).
# See https://oraios.github.io/serena/02-usage/050_configuration.html#language-server-specific-settings
ls_specific_settings: {}
# list of additional paths to ignore in this project.
# Same syntax as gitignore, so you can use * and **.
# Important: quote patterns that start with `*`, otherwise YAML treats them as aliases.
# Example:
# ignored_paths:
# - "examples/**"
# - ".worktrees/**"
# - "**/bin/**"
# - "**/obj/**"
# Note: global ignored_paths from serena_config.yml are also applied additively.
ignored_paths: []
# whether the project is in read-only mode
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
# Added on 2025-04-18
read_only: false
# list of tool names to exclude.
# This extends the existing exclusions (e.g. from the global configuration)
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
excluded_tools: []
# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default).
# This extends the existing inclusions (e.g. from the global configuration).
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
included_optional_tools: []
# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
# This cannot be combined with non-empty excluded_tools or included_optional_tools.
# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html
fixed_tools: []
# list of mode names that are to be activated by default, overriding the setting in the global configuration.
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply.
# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply
# for this project.
# This setting can, in turn, be overridden by CLI parameters (--mode).
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
default_modes:
# list of mode names to be activated additionally for this project, e.g. ["query-projects"]
# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes.
# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes
added_modes:
# initial prompt for the project. It will always be given to the LLM upon activating the project
# (contrary to the memories, which are loaded on demand).
initial_prompt: ""
# time budget (seconds) per tool call for the retrieval of additional symbol information
# such as docstrings or parameter information.
# This overrides the corresponding setting in the global configuration; see the documentation there.
# If null or missing, use the setting from the global configuration.
symbol_info_budget:
# list of regex patterns which, when matched, mark a memory entry as read‑only.
# Extends the list from the global configuration, merging the two lists.
read_only_memory_patterns: []
# list of regex patterns for memories to completely ignore.
# Matching memories will not appear in list_memories or activate_project output
# and cannot be accessed via read_memory or write_memory.
# To access ignored memory files, use the read_file tool on the raw file path.
# Extends the list from the global configuration, merging the two lists.
# Example: ["_archive/.*", "_episodes/.*"]
ignored_memory_patterns: []
# list of additional workspace folder paths for cross-package reference support.
# Paths can be absolute or relative to the project root.
# Each folder is registered as an LSP workspace folder, enabling language servers to discover
# symbols and references across package boundaries, but these folders are not indexed by Serena,
# i.e. the respective symbols will not be found using Serena's symbol search tools.
# Example:
# additional_workspace_folders:
# - ../sibling-package
# - ../shared-lib
ls_additional_workspace_folders: []
# list of language servers to start when using the LSP backend; choose from:
# ada al angular ansible bash
# bsl clojure cpp cpp_ccls crystal
# csharp csharp_omnisharp cue dart deno
# elixir elm erlang fortran fsharp
# gdscript gleam go groovy haskell
# haxe hlsl html java json
# julia kotlin latex lean4 lua
# luau markdown matlab msl nextflow
# nix ocaml pascal perl php
# php_phpactor php_phpantom powershell python python_basedpyright
# python_jedi python_pyrefly python_ty qml r
# rego ruby ruby_solargraph rust scala
# scss solidity svelte swift systemverilog
# terraform toml typescript typescript_vts vue
# wolfram yaml zig
# (This list may be outdated; generated with scripts/print_language_list.py;
# For the current list, see values of the LanguageServerId enum here:
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
# For some languages, there are several alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
# Note:
# - For C, use cpp
# - For JavaScript, use typescript
# - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
# - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
# - For Deno projects, use deno (serves the same .ts/.js files as typescript; requires the deno CLI on PATH)
# - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
# - For Free Pascal/Lazarus, use pascal
# Special requirements:
# Some language servers require additional setup/installations.
# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
# When using multiple language servers, the first language server that supports a given file will be used for that file.
# The first language server is the default language and the respective language server will be used as a fallback.
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
language_servers:
- rust
# list of workspace folder paths (LSP backend only).
# These folders will be used to build up Serena's symbol index.
# Paths must be within the project root and should thus be relative to the project root.
# Furthermore, the paths should not be filtered by ignore settings.
# Default setting: The entire project root folder (".") is considered.
# In (large) monorepos, this can be used to index only subfolders of the project root, e.g.
# ls_workspace_folders:
# - "./subproject1"
# - "./subproject2"
ls_workspace_folders:
- .
# optional shell command to run before the language backend (LSP or JetBrains) is initialised.
# the command runs in the project root directory and is only executed if the project is trusted
# (see trusted_project_path_patterns in the global configuration).
# serena waits for the command to exit: a non-zero exit code is logged as an error but does not
# abort activation. a per-project timeout (activation_command_timeout, default 180s) is the safety
# backstop for non-terminating commands; on expiry the process is killed and activation continues.
# example: activation_command: "npx nx run-many -t build"
activation_command:
# maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
# must be a positive number.
activation_command_timeout: 180.0
-72
View File
@@ -1,72 +0,0 @@
**PROMPT**
Tu es ma base de connaissance et mon bloc-notes intelligent sur le projet **obikmer**. Tu ne proposes pas, tu ne codes pas spontanément — tu réponds à mes questions et tu structures mes idées au fur et à mesure que je les exprime.
**Règle absolue : une question appelle une réponse, pas une action.**
Ne modifier aucun fichier à moins d'une demande explicite de modification. En particulier : observer un bug ou une incohérence dans le code montré ne constitue pas un mandat pour le corriger. Le code montré peut refléter une intention en cours — modifier sans mandat risque d'introduire un vrai bug là où tu croyais corriger.
**Règle absolue : ne jamais substituer une dépendance ou une bibliothèque sans validation explicite.**
Si une dépendance demandée pose problème (erreur de compilation, bug, API manquante), exposer le problème et proposer des alternatives — ne jamais switcher silencieusement vers une autre bibliothèque. Le choix des dépendances est une décision d'architecture qui appartient au développeur.
Tu maintiens en **anglais**, dense et sans remplissage, les documents suivants :
- `docmd/index.md` — document de discussion de base, enrichi progressivement au fil de nos échanges ; il reflète l'état courant de la réflexion sur le projet
- les autres fichiers Markdown dans `docmd/` selon leur thème respectif
Les snippets de code y sont courts et illustrent uniquement des principes architecturaux. Nos échanges se font en **français**.
---
## Contexte du projet
`obikmer` est un outil Rust de manipulation, comptage, indexation et opérations ensemblistes sur des séquences ADN représentées comme des ensembles de kmers.
**Contraintes fondamentales**
- Efficacité maximale en calcul, mémoire et disque
- Données métagénomiques : plusieurs dizaines de Gbases, milliards de kmers
- k impair, k ∈ [11, 31], fixé à l'exécution
- Formats d'entrée : FASTA, FASTQ, gzip, streaming stdin
**Opérations prioritaires**
- Comptage de kmers (fréquences)
- Recherche / requête rapide
- Opérations ensemblistes (union, intersection, diff)
**Ce qui a déjà été discuté**
- Encodage 2 bits/base → kmer tient dans un `u64`
- Forme canonique : `min(kmer, revcomp)` pour réduire l'espace de moitié
---
## Infrastructure de documentation
La documentation est gérée via **MkDocs + thème Material**, avec publication sur **GitHub Pages**.
**Structure des répertoires**
```
docmd/ ← sources Markdown + mkdocs.yml
docmd/mkdocs.yml
doc/ ← site HTML généré (servi par GitHub Pages)
.venv/ ← environnement Python (ignoré par git)
```
**Configuration `docmd/mkdocs.yml`**
- `docs_dir: .` (sources = `docmd/` lui-même)
- `site_dir: ../doc` (sortie = `doc/`)
**Commandes Makefile**
| Commande | Effet |
|---|---|
| `make doc` | Construit le HTML dans `doc/` |
| `make doc-serve` | Serveur local avec rechargement automatique |
| `make clean-doc` | Supprime `doc/` |
| `make clean` | Supprime `doc/` et `.venv/` |
Le `.venv/` est dans `.gitignore`. Le répertoire `doc/` (sortie HTML) est versionné pour GitHub Pages.
Lors de l'ajout de nouveaux fichiers Markdown dans `docmd/`, mettre à jour la section `nav:` de `docmd/mkdocs.yml`.
---
Je continue à poser mes questions et à guider la discussion.
+1275
View File
File diff suppressed because it is too large Load Diff
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -9,7 +9,7 @@
<link rel="prev" href="../../../implementation/persistent_bit_vec/">
<link rel="prev" href="../../../implementation/partition_layer_cache/">
<link rel="next" href="../../index_architecture/">
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -385,10 +440,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -406,8 +461,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -648,14 +703,42 @@
<li class="md-nav__item">
<a href="../../../implementation/obilayeredmap/" class="md-nav__link">
<a href="../../../implementation/evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -725,6 +808,202 @@
<li class="md-nav__item">
<a href="../../../implementation/merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../implementation/partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -749,10 +1028,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" checked>
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -770,8 +1049,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -847,6 +1126,90 @@
<li class="md-nav__item">
<a href="../../siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.8 KiB

File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+18
View File
@@ -0,0 +1,18 @@
/*!
* Lunr languages, `Danish` language
* https://github.com/MihaiValentin/lunr-languages
*
* Copyright 2014, Mihai Valentin
* http://www.mozilla.org/MPL/
*/
/*!
* based on
* Snowball JavaScript Library v0.3
* http://code.google.com/p/urim/
* http://snowball.tartarus.org/
*
* Copyright 2010, Oleg Mazko
* http://www.mozilla.org/MPL/
*/
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.da=function(){this.pipeline.reset(),this.pipeline.add(e.da.trimmer,e.da.stopWordFilter,e.da.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.da.stemmer))},e.da.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stA-Za-z",e.da.trimmer=e.trimmerSupport.generateTrimmer(e.da.wordCharacters),e.Pipeline.registerFunction(e.da.trimmer,"trimmer-da"),e.da.stemmer=function(){var r=e.stemmerSupport.Among,i=e.stemmerSupport.SnowballProgram,n=new function(){function e(){var e,r=f.cursor+3;if(d=f.limit,0<=r&&r<=f.limit){for(a=r;;){if(e=f.cursor,f.in_grouping(w,97,248)){f.cursor=e;break}if(f.cursor=e,e>=f.limit)return;f.cursor++}for(;!f.out_grouping(w,97,248);){if(f.cursor>=f.limit)return;f.cursor++}d=f.cursor,d<a&&(d=a)}}function n(){var e,r;if(f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(c,32),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del();break;case 2:f.in_grouping_b(p,97,229)&&f.slice_del()}}function t(){var e,r=f.limit-f.cursor;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.find_among_b(l,4)?(f.bra=f.cursor,f.limit_backward=e,f.cursor=f.limit-r,f.cursor>f.limit_backward&&(f.cursor--,f.bra=f.cursor,f.slice_del())):f.limit_backward=e)}function s(){var e,r,i,n=f.limit-f.cursor;if(f.ket=f.cursor,f.eq_s_b(2,"st")&&(f.bra=f.cursor,f.eq_s_b(2,"ig")&&f.slice_del()),f.cursor=f.limit-n,f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(m,5),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del(),i=f.limit-f.cursor,t(),f.cursor=f.limit-i;break;case 2:f.slice_from("løs")}}function o(){var e;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.out_grouping_b(w,97,248)?(f.bra=f.cursor,u=f.slice_to(u),f.limit_backward=e,f.eq_v_b(u)&&f.slice_del()):f.limit_backward=e)}var a,d,u,c=[new r("hed",-1,1),new r("ethed",0,1),new r("ered",-1,1),new r("e",-1,1),new r("erede",3,1),new r("ende",3,1),new r("erende",5,1),new r("ene",3,1),new r("erne",3,1),new r("ere",3,1),new r("en",-1,1),new r("heden",10,1),new r("eren",10,1),new r("er",-1,1),new r("heder",13,1),new r("erer",13,1),new r("s",-1,2),new r("heds",16,1),new r("es",16,1),new r("endes",18,1),new r("erendes",19,1),new r("enes",18,1),new r("ernes",18,1),new r("eres",18,1),new r("ens",16,1),new r("hedens",24,1),new r("erens",24,1),new r("ers",16,1),new r("ets",16,1),new r("erets",28,1),new r("et",-1,1),new r("eret",30,1)],l=[new r("gd",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("elig",1,1),new r("els",-1,1),new r("løst",-1,2)],w=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],p=[239,254,42,3,0,0,0,0,0,0,0,0,0,0,0,0,16],f=new i;this.setCurrent=function(e){f.setCurrent(e)},this.getCurrent=function(){return f.getCurrent()},this.stem=function(){var r=f.cursor;return e(),f.limit_backward=r,f.cursor=f.limit,n(),f.cursor=f.limit,t(),f.cursor=f.limit,s(),f.cursor=f.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return n.setCurrent(e),n.stem(),n.getCurrent()}):(n.setCurrent(e),n.stem(),n.getCurrent())}}(),e.Pipeline.registerFunction(e.da.stemmer,"stemmer-da"),e.da.stopWordFilter=e.generateStopWordFilter("ad af alle alt anden at blev blive bliver da de dem den denne der deres det dette dig din disse dog du efter eller en end er et for fra ham han hans har havde have hende hendes her hos hun hvad hvis hvor i ikke ind jeg jer jo kunne man mange med meget men mig min mine mit mod ned noget nogle nu når og også om op os over på selv sig sin sine sit skal skulle som sådan thi til ud under var vi vil ville vor være været".split(" ")),e.Pipeline.registerFunction(e.da.stopWordFilter,"stopWordFilter-da")}});
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hi=function(){this.pipeline.reset(),this.pipeline.add(e.hi.trimmer,e.hi.stopWordFilter,e.hi.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.hi.stemmer))},e.hi.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿa-zA-Za-zA-Z0-90-9",e.hi.trimmer=e.trimmerSupport.generateTrimmer(e.hi.wordCharacters),e.Pipeline.registerFunction(e.hi.trimmer,"trimmer-hi"),e.hi.stopWordFilter=e.generateStopWordFilter("अत अपना अपनी अपने अभी अंदर आदि आप इत्यादि इन इनका इन्हीं इन्हें इन्हों इस इसका इसकी इसके इसमें इसी इसे उन उनका उनकी उनके उनको उन्हीं उन्हें उन्हों उस उसके उसी उसे एक एवं एस ऐसे और कई कर करता करते करना करने करें कहते कहा का काफ़ी कि कितना किन्हें किन्हों किया किर किस किसी किसे की कुछ कुल के को कोई कौन कौनसा गया घर जब जहाँ जा जितना जिन जिन्हें जिन्हों जिस जिसे जीधर जैसा जैसे जो तक तब तरह तिन तिन्हें तिन्हों तिस तिसे तो था थी थे दबारा दिया दुसरा दूसरे दो द्वारा न नके नहीं ना निहायत नीचे ने पर पहले पूरा पे फिर बनी बही बहुत बाद बाला बिलकुल भी भीतर मगर मानो मे में यदि यह यहाँ यही या यिह ये रखें रहा रहे ऱ्वासा लिए लिये लेकिन व वग़ैरह वर्ग वह वहाँ वहीं वाले वुह वे वो सकता सकते सबसे सभी साथ साबुत साभ सारा से सो संग ही हुआ हुई हुए है हैं हो होता होती होते होना होने".split(" ")),e.hi.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.hi.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var t=i.toString().toLowerCase().replace(/^\s+/,"");return r.cut(t).split("|")},e.Pipeline.registerFunction(e.hi.stemmer,"stemmer-hi"),e.Pipeline.registerFunction(e.hi.stopWordFilter,"stopWordFilter-hi")}});
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hy=function(){this.pipeline.reset(),this.pipeline.add(e.hy.trimmer,e.hy.stopWordFilter)},e.hy.wordCharacters="[A-Za-z԰-֏ff-ﭏ]",e.hy.trimmer=e.trimmerSupport.generateTrimmer(e.hy.wordCharacters),e.Pipeline.registerFunction(e.hy.trimmer,"trimmer-hy"),e.hy.stopWordFilter=e.generateStopWordFilter("դու և եք էիր էիք հետո նաև նրանք որը վրա է որ պիտի են այս մեջ ն իր ու ի այդ որոնք այն կամ էր մի ես համար այլ իսկ էին ենք հետ ին թ էինք մենք նրա նա դուք եմ էի ըստ որպես ում".split(" ")),e.Pipeline.registerFunction(e.hy.stopWordFilter,"stopWordFilter-hy"),e.hy.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}(),e.Pipeline.registerFunction(e.hy.stemmer,"stemmer-hy")}});
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.ja=function(){this.pipeline.reset(),this.pipeline.add(e.ja.trimmer,e.ja.stopWordFilter,e.ja.stemmer),r?this.tokenizer=e.ja.tokenizer:(e.tokenizer&&(e.tokenizer=e.ja.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.ja.tokenizer))};var t=new e.TinySegmenter;e.ja.tokenizer=function(i){var n,o,s,p,a,u,m,l,c,f;if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t.toLowerCase()):t.toLowerCase()});for(o=i.toString().toLowerCase().replace(/^\s+/,""),n=o.length-1;n>=0;n--)if(/\S/.test(o.charAt(n))){o=o.substring(0,n+1);break}for(a=[],s=o.length,c=0,l=0;c<=s;c++)if(u=o.charAt(c),m=c-l,u.match(/\s/)||c==s){if(m>0)for(p=t.segment(o.slice(l,c)).filter(function(e){return!!e}),f=l,n=0;n<p.length;n++)r?a.push(new e.Token(p[n],{position:[f,p[n].length],index:a.length})):a.push(p[n]),f+=p[n].length;l=c+1}return a},e.ja.stemmer=function(){return function(e){return e}}(),e.Pipeline.registerFunction(e.ja.stemmer,"stemmer-ja"),e.ja.wordCharacters="一二三四五六七八九十百千万億兆一-龠々〆ヵヶぁ-んァ-ヴーア-ン゙a-zA-Za-zA-Z0-90-9",e.ja.trimmer=e.trimmerSupport.generateTrimmer(e.ja.wordCharacters),e.Pipeline.registerFunction(e.ja.trimmer,"trimmer-ja"),e.ja.stopWordFilter=e.generateStopWordFilter("これ それ あれ この その あの ここ そこ あそこ こちら どこ だれ なに なん 何 私 貴方 貴方方 我々 私達 あの人 あのかた 彼女 彼 です あります おります います は が の に を で え から まで より も どの と し それで しかし".split(" ")),e.Pipeline.registerFunction(e.ja.stopWordFilter,"stopWordFilter-ja"),e.jp=e.ja,e.Pipeline.registerFunction(e.jp.stemmer,"stemmer-jp"),e.Pipeline.registerFunction(e.jp.trimmer,"trimmer-jp"),e.Pipeline.registerFunction(e.jp.stopWordFilter,"stopWordFilter-jp")}});
+1
View File
@@ -0,0 +1 @@
module.exports=require("./lunr.ja");
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.kn=function(){this.pipeline.reset(),this.pipeline.add(e.kn.trimmer,e.kn.stopWordFilter,e.kn.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.kn.stemmer))},e.kn.wordCharacters="ಀ-಄ಅ-ಔಕ-ಹಾ-ೌ಼-ಽೕ-ೖೝ-ೞೠ-ೡೢ-ೣ೤೥೦-೯ೱ-ೳ",e.kn.trimmer=e.trimmerSupport.generateTrimmer(e.kn.wordCharacters),e.Pipeline.registerFunction(e.kn.trimmer,"trimmer-kn"),e.kn.stopWordFilter=e.generateStopWordFilter("ಮತ್ತು ಈ ಒಂದು ರಲ್ಲಿ ಹಾಗೂ ಎಂದು ಅಥವಾ ಇದು ರ ಅವರು ಎಂಬ ಮೇಲೆ ಅವರ ತನ್ನ ಆದರೆ ತಮ್ಮ ನಂತರ ಮೂಲಕ ಹೆಚ್ಚು ನ ಆ ಕೆಲವು ಅನೇಕ ಎರಡು ಹಾಗು ಪ್ರಮುಖ ಇದನ್ನು ಇದರ ಸುಮಾರು ಅದರ ಅದು ಮೊದಲ ಬಗ್ಗೆ ನಲ್ಲಿ ರಂದು ಇತರ ಅತ್ಯಂತ ಹೆಚ್ಚಿನ ಸಹ ಸಾಮಾನ್ಯವಾಗಿ ನೇ ಹಲವಾರು ಹೊಸ ದಿ ಕಡಿಮೆ ಯಾವುದೇ ಹೊಂದಿದೆ ದೊಡ್ಡ ಅನ್ನು ಇವರು ಪ್ರಕಾರ ಇದೆ ಮಾತ್ರ ಕೂಡ ಇಲ್ಲಿ ಎಲ್ಲಾ ವಿವಿಧ ಅದನ್ನು ಹಲವು ರಿಂದ ಕೇವಲ ದ ದಕ್ಷಿಣ ಗೆ ಅವನ ಅತಿ ನೆಯ ಬಹಳ ಕೆಲಸ ಎಲ್ಲ ಪ್ರತಿ ಇತ್ಯಾದಿ ಇವು ಬೇರೆ ಹೀಗೆ ನಡುವೆ ಇದಕ್ಕೆ ಎಸ್ ಇವರ ಮೊದಲು ಶ್ರೀ ಮಾಡುವ ಇದರಲ್ಲಿ ರೀತಿಯ ಮಾಡಿದ ಕಾಲ ಅಲ್ಲಿ ಮಾಡಲು ಅದೇ ಈಗ ಅವು ಗಳು ಎ ಎಂಬುದು ಅವನು ಅಂದರೆ ಅವರಿಗೆ ಇರುವ ವಿಶೇಷ ಮುಂದೆ ಅವುಗಳ ಮುಂತಾದ ಮೂಲ ಬಿ ಮೀ ಒಂದೇ ಇನ್ನೂ ಹೆಚ್ಚಾಗಿ ಮಾಡಿ ಅವರನ್ನು ಇದೇ ಯ ರೀತಿಯಲ್ಲಿ ಜೊತೆ ಅದರಲ್ಲಿ ಮಾಡಿದರು ನಡೆದ ಆಗ ಮತ್ತೆ ಪೂರ್ವ ಆತ ಬಂದ ಯಾವ ಒಟ್ಟು ಇತರೆ ಹಿಂದೆ ಪ್ರಮಾಣದ ಗಳನ್ನು ಕುರಿತು ಯು ಆದ್ದರಿಂದ ಅಲ್ಲದೆ ನಗರದ ಮೇಲಿನ ಏಕೆಂದರೆ ರಷ್ಟು ಎಂಬುದನ್ನು ಬಾರಿ ಎಂದರೆ ಹಿಂದಿನ ಆದರೂ ಆದ ಸಂಬಂಧಿಸಿದ ಮತ್ತೊಂದು ಸಿ ಆತನ ".split(" ")),e.kn.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.kn.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var n=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(n).split("|")},e.Pipeline.registerFunction(e.kn.stemmer,"stemmer-kn"),e.Pipeline.registerFunction(e.kn.stopWordFilter,"stopWordFilter-kn")}});
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){e.multiLanguage=function(){for(var t=Array.prototype.slice.call(arguments),i=t.join("-"),r="",n=[],s=[],p=0;p<t.length;++p)"en"==t[p]?(r+="\\w",n.unshift(e.stopWordFilter),n.push(e.stemmer),s.push(e.stemmer)):(r+=e[t[p]].wordCharacters,e[t[p]].stopWordFilter&&n.unshift(e[t[p]].stopWordFilter),e[t[p]].stemmer&&(n.push(e[t[p]].stemmer),s.push(e[t[p]].stemmer)));var o=e.trimmerSupport.generateTrimmer(r);return e.Pipeline.registerFunction(o,"lunr-multi-trimmer-"+i),n.unshift(o),function(){this.pipeline.reset(),this.pipeline.add.apply(this.pipeline,n),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add.apply(this.searchPipeline,s))}}}});
File diff suppressed because one or more lines are too long
+18
View File
@@ -0,0 +1,18 @@
/*!
* Lunr languages, `Norwegian` language
* https://github.com/MihaiValentin/lunr-languages
*
* Copyright 2014, Mihai Valentin
* http://www.mozilla.org/MPL/
*/
/*!
* based on
* Snowball JavaScript Library v0.3
* http://code.google.com/p/urim/
* http://snowball.tartarus.org/
*
* Copyright 2010, Oleg Mazko
* http://www.mozilla.org/MPL/
*/
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.no=function(){this.pipeline.reset(),this.pipeline.add(e.no.trimmer,e.no.stopWordFilter,e.no.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.no.stemmer))},e.no.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stA-Za-z",e.no.trimmer=e.trimmerSupport.generateTrimmer(e.no.wordCharacters),e.Pipeline.registerFunction(e.no.trimmer,"trimmer-no"),e.no.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,i=new function(){function e(){var e,r=w.cursor+3;if(a=w.limit,0<=r||r<=w.limit){for(s=r;;){if(e=w.cursor,w.in_grouping(d,97,248)){w.cursor=e;break}if(e>=w.limit)return;w.cursor=e+1}for(;!w.out_grouping(d,97,248);){if(w.cursor>=w.limit)return;w.cursor++}a=w.cursor,a<s&&(a=s)}}function i(){var e,r,n;if(w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(m,29),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:n=w.limit-w.cursor,w.in_grouping_b(c,98,122)?w.slice_del():(w.cursor=w.limit-n,w.eq_s_b(1,"k")&&w.out_grouping_b(d,97,248)&&w.slice_del());break;case 3:w.slice_from("er")}}function t(){var e,r=w.limit-w.cursor;w.cursor>=a&&(e=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,w.find_among_b(u,2)?(w.bra=w.cursor,w.limit_backward=e,w.cursor=w.limit-r,w.cursor>w.limit_backward&&(w.cursor--,w.bra=w.cursor,w.slice_del())):w.limit_backward=e)}function o(){var e,r;w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(l,11),e?(w.bra=w.cursor,w.limit_backward=r,1==e&&w.slice_del()):w.limit_backward=r)}var s,a,m=[new r("a",-1,1),new r("e",-1,1),new r("ede",1,1),new r("ande",1,1),new r("ende",1,1),new r("ane",1,1),new r("ene",1,1),new r("hetene",6,1),new r("erte",1,3),new r("en",-1,1),new r("heten",9,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",12,1),new r("s",-1,2),new r("as",14,1),new r("es",14,1),new r("edes",16,1),new r("endes",16,1),new r("enes",16,1),new r("hetenes",19,1),new r("ens",14,1),new r("hetens",21,1),new r("ers",14,1),new r("ets",14,1),new r("et",-1,1),new r("het",25,1),new r("ert",-1,3),new r("ast",-1,1)],u=[new r("dt",-1,-1),new r("vt",-1,-1)],l=[new r("leg",-1,1),new r("eleg",0,1),new r("ig",-1,1),new r("eig",2,1),new r("lig",2,1),new r("elig",4,1),new r("els",-1,1),new r("lov",-1,1),new r("elov",7,1),new r("slov",7,1),new r("hetslov",9,1)],d=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],c=[119,125,149,1],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,i(),w.cursor=w.limit,t(),w.cursor=w.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return i.setCurrent(e),i.stem(),i.getCurrent()}):(i.setCurrent(e),i.stem(),i.getCurrent())}}(),e.Pipeline.registerFunction(e.no.stemmer,"stemmer-no"),e.no.stopWordFilter=e.generateStopWordFilter("alle at av bare begge ble blei bli blir blitt både båe da de deg dei deim deira deires dem den denne der dere deres det dette di din disse ditt du dykk dykkar då eg ein eit eitt eller elles en enn er et ett etter for fordi fra før ha hadde han hans har hennar henne hennes her hjå ho hoe honom hoss hossen hun hva hvem hver hvilke hvilken hvis hvor hvordan hvorfor i ikke ikkje ikkje ingen ingi inkje inn inni ja jeg kan kom korleis korso kun kunne kva kvar kvarhelst kven kvi kvifor man mange me med medan meg meget mellom men mi min mine mitt mot mykje ned no noe noen noka noko nokon nokor nokre nå når og også om opp oss over på samme seg selv si si sia sidan siden sin sine sitt sjøl skal skulle slik so som som somme somt så sånn til um upp ut uten var vart varte ved vere verte vi vil ville vore vors vort vår være være vært å".split(" ")),e.Pipeline.registerFunction(e.no.stopWordFilter,"stopWordFilter-no")}});
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sa=function(){this.pipeline.reset(),this.pipeline.add(e.sa.trimmer,e.sa.stopWordFilter,e.sa.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sa.stemmer))},e.sa.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿ꣠-꣱ꣲ-ꣷ꣸-ꣻ꣼-ꣽꣾ-ꣿᆰ0-ᆰ9",e.sa.trimmer=e.trimmerSupport.generateTrimmer(e.sa.wordCharacters),e.Pipeline.registerFunction(e.sa.trimmer,"trimmer-sa"),e.sa.stopWordFilter=e.generateStopWordFilter('तथा अयम्‌ एकम्‌ इत्यस्मिन्‌ तथा तत्‌ वा अयम्‌ इत्यस्य ते आहूत उपरि तेषाम्‌ किन्तु तेषाम्‌ तदा इत्यनेन अधिकः इत्यस्य तत्‌ केचन बहवः द्वि तथा महत्वपूर्णः अयम्‌ अस्य विषये अयं अस्ति तत्‌ प्रथमः विषये इत्युपरि इत्युपरि इतर अधिकतमः अधिकः अपि सामान्यतया ठ इतरेतर नूतनम्‌ द न्यूनम्‌ कश्चित्‌ वा विशालः द सः अस्ति तदनुसारम् तत्र अस्ति केवलम्‌ अपि अत्र सर्वे विविधाः तत्‌ बहवः यतः इदानीम्‌ द दक्षिण इत्यस्मै तस्य उपरि नथ अतीव कार्यम्‌ सर्वे एकैकम्‌ इत्यादि। एते सन्ति उत इत्थम्‌ मध्ये एतदर्थं . स कस्य प्रथमः श्री. करोति अस्मिन् प्रकारः निर्मिता कालः तत्र कर्तुं समान अधुना ते सन्ति स एकः अस्ति सः अर्थात् तेषां कृते . स्थितम् विशेषः अग्रिम तेषाम्‌ समान स्रोतः ख म समान इदानीमपि अधिकतया करोतु ते समान इत्यस्य वीथी सह यस्मिन् कृतवान्‌ धृतः तदा पुनः पूर्वं सः आगतः किम्‌ कुल इतर पुरा मात्रा स विषये उ अतएव अपि नगरस्य उपरि यतः प्रतिशतं कतरः कालः साधनानि भूत तथापि जात सम्बन्धि अन्यत्‌ ग अतः अस्माकं स्वकीयाः अस्माकं इदानीं अन्तः इत्यादयः भवन्तः इत्यादयः एते एताः तस्य अस्य इदम् एते तेषां तेषां तेषां तान् तेषां तेषां तेषां समानः सः एकः च तादृशाः बहवः अन्ये च वदन्ति यत् कियत् कस्मै कस्मै यस्मै यस्मै यस्मै यस्मै न अतिनीचः किन्तु प्रथमं सम्पूर्णतया ततः चिरकालानन्तरं पुस्तकं सम्पूर्णतया अन्तः किन्तु अत्र वा इह इव श्रद्धाय अवशिष्यते परन्तु अन्ये वर्गाः सन्ति ते सन्ति शक्नुवन्ति सर्वे मिलित्वा सर्वे एकत्र"'.split(" ")),e.sa.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.sa.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var i=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(i).split("|")},e.Pipeline.registerFunction(e.sa.stemmer,"stemmer-sa"),e.Pipeline.registerFunction(e.sa.stopWordFilter,"stopWordFilter-sa")}});
@@ -0,0 +1 @@
!function(r,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(r.lunr)}(this,function(){return function(r){r.stemmerSupport={Among:function(r,t,i,s){if(this.toCharArray=function(r){for(var t=r.length,i=new Array(t),s=0;s<t;s++)i[s]=r.charCodeAt(s);return i},!r&&""!=r||!t&&0!=t||!i)throw"Bad Among initialisation: s:"+r+", substring_i: "+t+", result: "+i;this.s_size=r.length,this.s=this.toCharArray(r),this.substring_i=t,this.result=i,this.method=s},SnowballProgram:function(){var r;return{bra:0,ket:0,limit:0,cursor:0,limit_backward:0,setCurrent:function(t){r=t,this.cursor=0,this.limit=t.length,this.limit_backward=0,this.bra=this.cursor,this.ket=this.limit},getCurrent:function(){var t=r;return r=null,t},in_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},in_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},out_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e>s||e<i)return this.cursor++,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},out_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e>s||e<i)return this.cursor--,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},eq_s:function(t,i){if(this.limit-this.cursor<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor+s)!=i.charCodeAt(s))return!1;return this.cursor+=t,!0},eq_s_b:function(t,i){if(this.cursor-this.limit_backward<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor-t+s)!=i.charCodeAt(s))return!1;return this.cursor-=t,!0},find_among:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=l;m<_.s_size;m++){if(n+l==u){f=-1;break}if(f=r.charCodeAt(n+l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n+_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n+_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},find_among_b:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit_backward,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=_.s_size-1-l;m>=0;m--){if(n-l==u){f=-1;break}if(f=r.charCodeAt(n-1-l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n-_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n-_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},replace_s:function(t,i,s){var e=s.length-(i-t),n=r.substring(0,t),u=r.substring(i);return r=n+s+u,this.limit+=e,this.cursor>=i?this.cursor+=e:this.cursor>t&&(this.cursor=t),e},slice_check:function(){if(this.bra<0||this.bra>this.ket||this.ket>this.limit||this.limit>r.length)throw"faulty slice operation"},slice_from:function(r){this.slice_check(),this.replace_s(this.bra,this.ket,r)},slice_del:function(){this.slice_from("")},insert:function(r,t,i){var s=this.replace_s(r,t,i);r<=this.bra&&(this.bra+=s),r<=this.ket&&(this.ket+=s)},slice_to:function(){return this.slice_check(),r.substring(this.bra,this.ket)},eq_v_b:function(r){return this.eq_s_b(r.length,r)}}}},r.trimmerSupport={generateTrimmer:function(r){var t=new RegExp("^[^"+r+"]+"),i=new RegExp("[^"+r+"]+$");return function(r){return"function"==typeof r.update?r.update(function(r){return r.replace(t,"").replace(i,"")}):r.replace(t,"").replace(i,"")}}}}});
+18
View File
@@ -0,0 +1,18 @@
/*!
* Lunr languages, `Swedish` language
* https://github.com/MihaiValentin/lunr-languages
*
* Copyright 2014, Mihai Valentin
* http://www.mozilla.org/MPL/
*/
/*!
* based on
* Snowball JavaScript Library v0.3
* http://code.google.com/p/urim/
* http://snowball.tartarus.org/
*
* Copyright 2010, Oleg Mazko
* http://www.mozilla.org/MPL/
*/
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sv=function(){this.pipeline.reset(),this.pipeline.add(e.sv.trimmer,e.sv.stopWordFilter,e.sv.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sv.stemmer))},e.sv.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stA-Za-z",e.sv.trimmer=e.trimmerSupport.generateTrimmer(e.sv.wordCharacters),e.Pipeline.registerFunction(e.sv.trimmer,"trimmer-sv"),e.sv.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,t=new function(){function e(){var e,r=w.cursor+3;if(o=w.limit,0<=r||r<=w.limit){for(a=r;;){if(e=w.cursor,w.in_grouping(l,97,246)){w.cursor=e;break}if(w.cursor=e,w.cursor>=w.limit)return;w.cursor++}for(;!w.out_grouping(l,97,246);){if(w.cursor>=w.limit)return;w.cursor++}o=w.cursor,o<a&&(o=a)}}function t(){var e,r=w.limit_backward;if(w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(u,37),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.in_grouping_b(d,98,121)&&w.slice_del()}}function i(){var e=w.limit_backward;w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.find_among_b(c,7)&&(w.cursor=w.limit,w.ket=w.cursor,w.cursor>w.limit_backward&&(w.bra=--w.cursor,w.slice_del())),w.limit_backward=e)}function s(){var e,r;if(w.cursor>=o){if(r=w.limit_backward,w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(m,5))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.slice_from("lös");break;case 3:w.slice_from("full")}w.limit_backward=r}}var a,o,u=[new r("a",-1,1),new r("arna",0,1),new r("erna",0,1),new r("heterna",2,1),new r("orna",0,1),new r("ad",-1,1),new r("e",-1,1),new r("ade",6,1),new r("ande",6,1),new r("arne",6,1),new r("are",6,1),new r("aste",6,1),new r("en",-1,1),new r("anden",12,1),new r("aren",12,1),new r("heten",12,1),new r("ern",-1,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",18,1),new r("or",-1,1),new r("s",-1,2),new r("as",21,1),new r("arnas",22,1),new r("ernas",22,1),new r("ornas",22,1),new r("es",21,1),new r("ades",26,1),new r("andes",26,1),new r("ens",21,1),new r("arens",29,1),new r("hetens",29,1),new r("erns",21,1),new r("at",-1,1),new r("andet",-1,1),new r("het",-1,1),new r("ast",-1,1)],c=[new r("dd",-1,-1),new r("gd",-1,-1),new r("nn",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1),new r("tt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("els",-1,1),new r("fullt",-1,3),new r("löst",-1,2)],l=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,24,0,32],d=[119,127,149],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,t(),w.cursor=w.limit,i(),w.cursor=w.limit,s(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return t.setCurrent(e),t.stem(),t.getCurrent()}):(t.setCurrent(e),t.stem(),t.getCurrent())}}(),e.Pipeline.registerFunction(e.sv.stemmer,"stemmer-sv"),e.sv.stopWordFilter=e.generateStopWordFilter("alla allt att av blev bli blir blivit de dem den denna deras dess dessa det detta dig din dina ditt du där då efter ej eller en er era ert ett från för ha hade han hans har henne hennes hon honom hur här i icke ingen inom inte jag ju kan kunde man med mellan men mig min mina mitt mot mycket ni nu när någon något några och om oss på samma sedan sig sin sina sitta själv skulle som så sådan sådana sådant till under upp ut utan vad var vara varför varit varje vars vart vem vi vid vilka vilkas vilken vilket vår våra vårt än är åt över".split(" ")),e.Pipeline.registerFunction(e.sv.stopWordFilter,"stopWordFilter-sv")}});
+1
View File
@@ -0,0 +1 @@
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.ta=function(){this.pipeline.reset(),this.pipeline.add(e.ta.trimmer,e.ta.stopWordFilter,e.ta.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.ta.stemmer))},e.ta.wordCharacters="஀-உஊ-ஏஐ-ஙச-ட஠-னப-யர-ஹ஺-ிீ-௉ொ-௏ௐ-௙௚-௟௠-௩௪-௯௰-௹௺-௿a-zA-Za-zA-Z0-90-9",e.ta.trimmer=e.trimmerSupport.generateTrimmer(e.ta.wordCharacters),e.Pipeline.registerFunction(e.ta.trimmer,"trimmer-ta"),e.ta.stopWordFilter=e.generateStopWordFilter("அங்கு அங்கே அது அதை அந்த அவர் அவர்கள் அவள் அவன் அவை ஆக ஆகவே ஆகையால் ஆதலால் ஆதலினால் ஆனாலும் ஆனால் இங்கு இங்கே இது இதை இந்த இப்படி இவர் இவர்கள் இவள் இவன் இவை இவ்வளவு உனக்கு உனது உன் உன்னால் எங்கு எங்கே எது எதை எந்த எப்படி எவர் எவர்கள் எவள் எவன் எவை எவ்வளவு எனக்கு எனது எனவே என் என்ன என்னால் ஏது ஏன் தனது தன்னால் தானே தான் நாங்கள் நாம் நான் நீ நீங்கள்".split(" ")),e.ta.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.ta.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.ta.stemmer,"stemmer-ta"),e.Pipeline.registerFunction(e.ta.stopWordFilter,"stopWordFilter-ta")}});
+1
View File
@@ -0,0 +1 @@
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.te=function(){this.pipeline.reset(),this.pipeline.add(e.te.trimmer,e.te.stopWordFilter,e.te.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.te.stemmer))},e.te.wordCharacters="ఀ-ఄఅ-ఔక-హా-ౌౕ-ౖౘ-ౚౠ-ౡౢ-ౣ౦-౯౸-౿఼ఽ్ౝ౷౤౥",e.te.trimmer=e.trimmerSupport.generateTrimmer(e.te.wordCharacters),e.Pipeline.registerFunction(e.te.trimmer,"trimmer-te"),e.te.stopWordFilter=e.generateStopWordFilter("అందరూ అందుబాటులో అడగండి అడగడం అడ్డంగా అనుగుణంగా అనుమతించు అనుమతిస్తుంది అయితే ఇప్పటికే ఉన్నారు ఎక్కడైనా ఎప్పుడు ఎవరైనా ఎవరో ఏ ఏదైనా ఏమైనప్పటికి ఒక ఒకరు కనిపిస్తాయి కాదు కూడా గా గురించి చుట్టూ చేయగలిగింది తగిన తర్వాత దాదాపు దూరంగా నిజంగా పై ప్రకారం ప్రక్కన మధ్య మరియు మరొక మళ్ళీ మాత్రమే మెచ్చుకో వద్ద వెంట వేరుగా వ్యతిరేకంగా సంబంధం".split(" ")),e.te.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.te.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.te.stemmer,"stemmer-te"),e.Pipeline.registerFunction(e.te.stopWordFilter,"stopWordFilter-te")}});
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.th=function(){this.pipeline.reset(),this.pipeline.add(e.th.trimmer),r?this.tokenizer=e.th.tokenizer:(e.tokenizer&&(e.tokenizer=e.th.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.th.tokenizer))},e.th.wordCharacters="[฀-๿]",e.th.trimmer=e.trimmerSupport.generateTrimmer(e.th.wordCharacters),e.Pipeline.registerFunction(e.th.trimmer,"trimmer-th");var t=e.wordcut;t.init(),e.th.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t):t});var n=i.toString().replace(/^\s+/,"");return t.cut(n).split("|")}}});
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.vi=function(){this.pipeline.reset(),this.pipeline.add(e.vi.stopWordFilter,e.vi.trimmer)},e.vi.wordCharacters="[A-Za-ẓ̀͐́͑̉̃̓ÂâÊêÔôĂ-ăĐ-đƠ-ơƯ-ư]",e.vi.trimmer=e.trimmerSupport.generateTrimmer(e.vi.wordCharacters),e.Pipeline.registerFunction(e.vi.trimmer,"trimmer-vi"),e.vi.stopWordFilter=e.generateStopWordFilter("là cái nhưng mà".split(" "))}});
+1
View File
@@ -0,0 +1 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r(require("@node-rs/jieba")):r()(e.lunr)}(this,function(e){return function(r,t){if(void 0===r)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===r.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var i="2"==r.version[0];r.zh=function(){this.pipeline.reset(),this.pipeline.add(r.zh.trimmer,r.zh.stopWordFilter,r.zh.stemmer),i?this.tokenizer=r.zh.tokenizer:(r.tokenizer&&(r.tokenizer=r.zh.tokenizer),this.tokenizerFn&&(this.tokenizerFn=r.zh.tokenizer))},r.zh.tokenizer=function(n){if(!arguments.length||null==n||void 0==n)return[];if(Array.isArray(n))return n.map(function(e){return i?new r.Token(e.toLowerCase()):e.toLowerCase()});t&&e.load(t);var o=n.toString().trim().toLowerCase(),s=[];e.cut(o,!0).forEach(function(e){s=s.concat(e.split(" "))}),s=s.filter(function(e){return!!e});var u=0;return s.map(function(e,t){if(i){var n=o.indexOf(e,u),s={};return s.position=[n,e.length],s.index=t,u=n,new r.Token(e,s)}return e})},r.zh.wordCharacters="\\w一-龥",r.zh.trimmer=r.trimmerSupport.generateTrimmer(r.zh.wordCharacters),r.Pipeline.registerFunction(r.zh.trimmer,"trimmer-zh"),r.zh.stemmer=function(){return function(e){return e}}(),r.Pipeline.registerFunction(r.zh.stemmer,"stemmer-zh"),r.zh.stopWordFilter=r.generateStopWordFilter("的 一 不 在 人 有 是 为 為 以 于 於 上 他 而 后 後 之 来 來 及 了 因 下 可 到 由 这 這 与 與 也 此 但 并 並 个 個 其 已 无 無 小 我 们 們 起 最 再 今 去 好 只 又 或 很 亦 某 把 那 你 乃 它 吧 被 比 别 趁 当 當 从 從 得 打 凡 儿 兒 尔 爾 该 該 各 给 給 跟 和 何 还 還 即 几 幾 既 看 据 據 距 靠 啦 另 么 麽 每 嘛 拿 哪 您 凭 憑 且 却 卻 让 讓 仍 啥 如 若 使 谁 誰 虽 雖 随 隨 同 所 她 哇 嗡 往 些 向 沿 哟 喲 用 咱 则 則 怎 曾 至 致 着 著 诸 諸 自".split(" ")),r.Pipeline.registerFunction(r.zh.stopWordFilter,"stopWordFilter-zh")}});
+206
View File
@@ -0,0 +1,206 @@
/**
* export the module via AMD, CommonJS or as a browser global
* Export code from https://github.com/umdjs/umd/blob/master/returnExports.js
*/
;(function (root, factory) {
if (typeof define === 'function' && define.amd) {
// AMD. Register as an anonymous module.
define(factory)
} else if (typeof exports === 'object') {
/**
* Node. Does not work with strict CommonJS, but
* only CommonJS-like environments that support module.exports,
* like Node.
*/
module.exports = factory()
} else {
// Browser globals (root is window)
factory()(root.lunr);
}
}(this, function () {
/**
* Just return a value to define the module export.
* This example returns an object, but the module
* can return a function as the exported value.
*/
return function(lunr) {
// TinySegmenter 0.1 -- Super compact Japanese tokenizer in Javascript
// (c) 2008 Taku Kudo <taku@chasen.org>
// TinySegmenter is freely distributable under the terms of a new BSD licence.
// For details, see http://chasen.org/~taku/software/TinySegmenter/LICENCE.txt
function TinySegmenter() {
var patterns = {
"[一二三四五六七八九十百千万億兆]":"M",
"[一-龠々〆ヵヶ]":"H",
"[ぁ-ん]":"I",
"[ァ-ヴーア-ン゙ー]":"K",
"[a-zA-Za-zA-Z]":"A",
"[0-90-9]":"N"
}
this.chartype_ = [];
for (var i in patterns) {
var regexp = new RegExp(i);
this.chartype_.push([regexp, patterns[i]]);
}
this.BIAS__ = -332
this.BC1__ = {"HH":6,"II":2461,"KH":406,"OH":-1378};
this.BC2__ = {"AA":-3267,"AI":2744,"AN":-878,"HH":-4070,"HM":-1711,"HN":4012,"HO":3761,"IA":1327,"IH":-1184,"II":-1332,"IK":1721,"IO":5492,"KI":3831,"KK":-8741,"MH":-3132,"MK":3334,"OO":-2920};
this.BC3__ = {"HH":996,"HI":626,"HK":-721,"HN":-1307,"HO":-836,"IH":-301,"KK":2762,"MK":1079,"MM":4034,"OA":-1652,"OH":266};
this.BP1__ = {"BB":295,"OB":304,"OO":-125,"UB":352};
this.BP2__ = {"BO":60,"OO":-1762};
this.BQ1__ = {"BHH":1150,"BHM":1521,"BII":-1158,"BIM":886,"BMH":1208,"BNH":449,"BOH":-91,"BOO":-2597,"OHI":451,"OIH":-296,"OKA":1851,"OKH":-1020,"OKK":904,"OOO":2965};
this.BQ2__ = {"BHH":118,"BHI":-1159,"BHM":466,"BIH":-919,"BKK":-1720,"BKO":864,"OHH":-1139,"OHM":-181,"OIH":153,"UHI":-1146};
this.BQ3__ = {"BHH":-792,"BHI":2664,"BII":-299,"BKI":419,"BMH":937,"BMM":8335,"BNN":998,"BOH":775,"OHH":2174,"OHM":439,"OII":280,"OKH":1798,"OKI":-793,"OKO":-2242,"OMH":-2402,"OOO":11699};
this.BQ4__ = {"BHH":-3895,"BIH":3761,"BII":-4654,"BIK":1348,"BKK":-1806,"BMI":-3385,"BOO":-12396,"OAH":926,"OHH":266,"OHK":-2036,"ONN":-973};
this.BW1__ = {",と":660,",同":727,"B1あ":1404,"B1同":542,"、と":660,"、同":727,"」と":1682,"あっ":1505,"いう":1743,"いっ":-2055,"いる":672,"うし":-4817,"うん":665,"から":3472,"がら":600,"こう":-790,"こと":2083,"こん":-1262,"さら":-4143,"さん":4573,"した":2641,"して":1104,"すで":-3399,"そこ":1977,"それ":-871,"たち":1122,"ため":601,"った":3463,"つい":-802,"てい":805,"てき":1249,"でき":1127,"です":3445,"では":844,"とい":-4915,"とみ":1922,"どこ":3887,"ない":5713,"なっ":3015,"など":7379,"なん":-1113,"にし":2468,"には":1498,"にも":1671,"に対":-912,"の一":-501,"の中":741,"ませ":2448,"まで":1711,"まま":2600,"まる":-2155,"やむ":-1947,"よっ":-2565,"れた":2369,"れで":-913,"をし":1860,"を見":731,"亡く":-1886,"京都":2558,"取り":-2784,"大き":-2604,"大阪":1497,"平方":-2314,"引き":-1336,"日本":-195,"本当":-2423,"毎日":-2113,"目指":-724,"B1あ":1404,"B1同":542,"」と":1682};
this.BW2__ = {"..":-11822,"11":-669,"――":-5730,"−−":-13175,"いう":-1609,"うか":2490,"かし":-1350,"かも":-602,"から":-7194,"かれ":4612,"がい":853,"がら":-3198,"きた":1941,"くな":-1597,"こと":-8392,"この":-4193,"させ":4533,"され":13168,"さん":-3977,"しい":-1819,"しか":-545,"した":5078,"して":972,"しな":939,"その":-3744,"たい":-1253,"たた":-662,"ただ":-3857,"たち":-786,"たと":1224,"たは":-939,"った":4589,"って":1647,"っと":-2094,"てい":6144,"てき":3640,"てく":2551,"ては":-3110,"ても":-3065,"でい":2666,"でき":-1528,"でし":-3828,"です":-4761,"でも":-4203,"とい":1890,"とこ":-1746,"とと":-2279,"との":720,"とみ":5168,"とも":-3941,"ない":-2488,"なが":-1313,"など":-6509,"なの":2614,"なん":3099,"にお":-1615,"にし":2748,"にな":2454,"によ":-7236,"に対":-14943,"に従":-4688,"に関":-11388,"のか":2093,"ので":-7059,"のに":-6041,"のの":-6125,"はい":1073,"はが":-1033,"はず":-2532,"ばれ":1813,"まし":-1316,"まで":-6621,"まれ":5409,"めて":-3153,"もい":2230,"もの":-10713,"らか":-944,"らし":-1611,"らに":-1897,"りし":651,"りま":1620,"れた":4270,"れて":849,"れば":4114,"ろう":6067,"われ":7901,"を通":-11877,"んだ":728,"んな":-4115,"一人":602,"一方":-1375,"一日":970,"一部":-1051,"上が":-4479,"会社":-1116,"出て":2163,"分の":-7758,"同党":970,"同日":-913,"大阪":-2471,"委員":-1250,"少な":-1050,"年度":-8669,"年間":-1626,"府県":-2363,"手権":-1982,"新聞":-4066,"日新":-722,"日本":-7068,"日米":3372,"曜日":-601,"朝鮮":-2355,"本人":-2697,"東京":-1543,"然と":-1384,"社会":-1276,"立て":-990,"第に":-1612,"米国":-4268,"11":-669};
this.BW3__ = {"あた":-2194,"あり":719,"ある":3846,"い.":-1185,"い。":-1185,"いい":5308,"いえ":2079,"いく":3029,"いた":2056,"いっ":1883,"いる":5600,"いわ":1527,"うち":1117,"うと":4798,"えと":1454,"か.":2857,"か。":2857,"かけ":-743,"かっ":-4098,"かに":-669,"から":6520,"かり":-2670,"が,":1816,"が、":1816,"がき":-4855,"がけ":-1127,"がっ":-913,"がら":-4977,"がり":-2064,"きた":1645,"けど":1374,"こと":7397,"この":1542,"ころ":-2757,"さい":-714,"さを":976,"し,":1557,"し、":1557,"しい":-3714,"した":3562,"して":1449,"しな":2608,"しま":1200,"す.":-1310,"す。":-1310,"する":6521,"ず,":3426,"ず、":3426,"ずに":841,"そう":428,"た.":8875,"た。":8875,"たい":-594,"たの":812,"たり":-1183,"たる":-853,"だ.":4098,"だ。":4098,"だっ":1004,"った":-4748,"って":300,"てい":6240,"てお":855,"ても":302,"です":1437,"でに":-1482,"では":2295,"とう":-1387,"とし":2266,"との":541,"とも":-3543,"どう":4664,"ない":1796,"なく":-903,"など":2135,"に,":-1021,"に、":-1021,"にし":1771,"にな":1906,"には":2644,"の,":-724,"の、":-724,"の子":-1000,"は,":1337,"は、":1337,"べき":2181,"まし":1113,"ます":6943,"まっ":-1549,"まで":6154,"まれ":-793,"らし":1479,"られ":6820,"るる":3818,"れ,":854,"れ、":854,"れた":1850,"れて":1375,"れば":-3246,"れる":1091,"われ":-605,"んだ":606,"んで":798,"カ月":990,"会議":860,"入り":1232,"大会":2217,"始め":1681,"市":965,"新聞":-5055,"日,":974,"日、":974,"社会":2024,"カ月":990};
this.TC1__ = {"AAA":1093,"HHH":1029,"HHM":580,"HII":998,"HOH":-390,"HOM":-331,"IHI":1169,"IOH":-142,"IOI":-1015,"IOM":467,"MMH":187,"OOI":-1832};
this.TC2__ = {"HHO":2088,"HII":-1023,"HMM":-1154,"IHI":-1965,"KKH":703,"OII":-2649};
this.TC3__ = {"AAA":-294,"HHH":346,"HHI":-341,"HII":-1088,"HIK":731,"HOH":-1486,"IHH":128,"IHI":-3041,"IHO":-1935,"IIH":-825,"IIM":-1035,"IOI":-542,"KHH":-1216,"KKA":491,"KKH":-1217,"KOK":-1009,"MHH":-2694,"MHM":-457,"MHO":123,"MMH":-471,"NNH":-1689,"NNO":662,"OHO":-3393};
this.TC4__ = {"HHH":-203,"HHI":1344,"HHK":365,"HHM":-122,"HHN":182,"HHO":669,"HIH":804,"HII":679,"HOH":446,"IHH":695,"IHO":-2324,"IIH":321,"III":1497,"IIO":656,"IOO":54,"KAK":4845,"KKA":3386,"KKK":3065,"MHH":-405,"MHI":201,"MMH":-241,"MMM":661,"MOM":841};
this.TQ1__ = {"BHHH":-227,"BHHI":316,"BHIH":-132,"BIHH":60,"BIII":1595,"BNHH":-744,"BOHH":225,"BOOO":-908,"OAKK":482,"OHHH":281,"OHIH":249,"OIHI":200,"OIIH":-68};
this.TQ2__ = {"BIHH":-1401,"BIII":-1033,"BKAK":-543,"BOOO":-5591};
this.TQ3__ = {"BHHH":478,"BHHM":-1073,"BHIH":222,"BHII":-504,"BIIH":-116,"BIII":-105,"BMHI":-863,"BMHM":-464,"BOMH":620,"OHHH":346,"OHHI":1729,"OHII":997,"OHMH":481,"OIHH":623,"OIIH":1344,"OKAK":2792,"OKHH":587,"OKKA":679,"OOHH":110,"OOII":-685};
this.TQ4__ = {"BHHH":-721,"BHHM":-3604,"BHII":-966,"BIIH":-607,"BIII":-2181,"OAAA":-2763,"OAKK":180,"OHHH":-294,"OHHI":2446,"OHHO":480,"OHIH":-1573,"OIHH":1935,"OIHI":-493,"OIIH":626,"OIII":-4007,"OKAK":-8156};
this.TW1__ = {"につい":-4681,"東京都":2026};
this.TW2__ = {"ある程":-2049,"いった":-1256,"ころが":-2434,"しょう":3873,"その後":-4430,"だって":-1049,"ていた":1833,"として":-4657,"ともに":-4517,"もので":1882,"一気に":-792,"初めて":-1512,"同時に":-8097,"大きな":-1255,"対して":-2721,"社会党":-3216};
this.TW3__ = {"いただ":-1734,"してい":1314,"として":-4314,"につい":-5483,"にとっ":-5989,"に当た":-6247,"ので,":-727,"ので、":-727,"のもの":-600,"れから":-3752,"十二月":-2287};
this.TW4__ = {"いう.":8576,"いう。":8576,"からな":-2348,"してい":2958,"たが,":1516,"たが、":1516,"ている":1538,"という":1349,"ました":5543,"ません":1097,"ようと":-4258,"よると":5865};
this.UC1__ = {"A":484,"K":93,"M":645,"O":-505};
this.UC2__ = {"A":819,"H":1059,"I":409,"M":3987,"N":5775,"O":646};
this.UC3__ = {"A":-1370,"I":2311};
this.UC4__ = {"A":-2643,"H":1809,"I":-1032,"K":-3450,"M":3565,"N":3876,"O":6646};
this.UC5__ = {"H":313,"I":-1238,"K":-799,"M":539,"O":-831};
this.UC6__ = {"H":-506,"I":-253,"K":87,"M":247,"O":-387};
this.UP1__ = {"O":-214};
this.UP2__ = {"B":69,"O":935};
this.UP3__ = {"B":189};
this.UQ1__ = {"BH":21,"BI":-12,"BK":-99,"BN":142,"BO":-56,"OH":-95,"OI":477,"OK":410,"OO":-2422};
this.UQ2__ = {"BH":216,"BI":113,"OK":1759};
this.UQ3__ = {"BA":-479,"BH":42,"BI":1913,"BK":-7198,"BM":3160,"BN":6427,"BO":14761,"OI":-827,"ON":-3212};
this.UW1__ = {",":156,"、":156,"「":-463,"あ":-941,"う":-127,"が":-553,"き":121,"こ":505,"で":-201,"と":-547,"ど":-123,"に":-789,"の":-185,"は":-847,"も":-466,"や":-470,"よ":182,"ら":-292,"り":208,"れ":169,"を":-446,"ん":-137,"・":-135,"主":-402,"京":-268,"区":-912,"午":871,"国":-460,"大":561,"委":729,"市":-411,"日":-141,"理":361,"生":-408,"県":-386,"都":-718,"「":-463,"・":-135};
this.UW2__ = {",":-829,"、":-829,"〇":892,"「":-645,"」":3145,"あ":-538,"い":505,"う":134,"お":-502,"か":1454,"が":-856,"く":-412,"こ":1141,"さ":878,"ざ":540,"し":1529,"す":-675,"せ":300,"そ":-1011,"た":188,"だ":1837,"つ":-949,"て":-291,"で":-268,"と":-981,"ど":1273,"な":1063,"に":-1764,"の":130,"は":-409,"ひ":-1273,"べ":1261,"ま":600,"も":-1263,"や":-402,"よ":1639,"り":-579,"る":-694,"れ":571,"を":-2516,"ん":2095,"ア":-587,"カ":306,"キ":568,"ッ":831,"三":-758,"不":-2150,"世":-302,"中":-968,"主":-861,"事":492,"人":-123,"会":978,"保":362,"入":548,"初":-3025,"副":-1566,"北":-3414,"区":-422,"大":-1769,"天":-865,"太":-483,"子":-1519,"学":760,"実":1023,"小":-2009,"市":-813,"年":-1060,"強":1067,"手":-1519,"揺":-1033,"政":1522,"文":-1355,"新":-1682,"日":-1815,"明":-1462,"最":-630,"朝":-1843,"本":-1650,"東":-931,"果":-665,"次":-2378,"民":-180,"気":-1740,"理":752,"発":529,"目":-1584,"相":-242,"県":-1165,"立":-763,"第":810,"米":509,"自":-1353,"行":838,"西":-744,"見":-3874,"調":1010,"議":1198,"込":3041,"開":1758,"間":-1257,"「":-645,"」":3145,"ッ":831,"ア":-587,"カ":306,"キ":568};
this.UW3__ = {",":4889,"1":-800,"−":-1723,"、":4889,"々":-2311,"〇":5827,"」":2670,"〓":-3573,"あ":-2696,"い":1006,"う":2342,"え":1983,"お":-4864,"か":-1163,"が":3271,"く":1004,"け":388,"げ":401,"こ":-3552,"ご":-3116,"さ":-1058,"し":-395,"す":584,"せ":3685,"そ":-5228,"た":842,"ち":-521,"っ":-1444,"つ":-1081,"て":6167,"で":2318,"と":1691,"ど":-899,"な":-2788,"に":2745,"の":4056,"は":4555,"ひ":-2171,"ふ":-1798,"へ":1199,"ほ":-5516,"ま":-4384,"み":-120,"め":1205,"も":2323,"や":-788,"よ":-202,"ら":727,"り":649,"る":5905,"れ":2773,"わ":-1207,"を":6620,"ん":-518,"ア":551,"グ":1319,"ス":874,"ッ":-1350,"ト":521,"ム":1109,"ル":1591,"ロ":2201,"ン":278,"・":-3794,"一":-1619,"下":-1759,"世":-2087,"両":3815,"中":653,"主":-758,"予":-1193,"二":974,"人":2742,"今":792,"他":1889,"以":-1368,"低":811,"何":4265,"作":-361,"保":-2439,"元":4858,"党":3593,"全":1574,"公":-3030,"六":755,"共":-1880,"円":5807,"再":3095,"分":457,"初":2475,"別":1129,"前":2286,"副":4437,"力":365,"動":-949,"務":-1872,"化":1327,"北":-1038,"区":4646,"千":-2309,"午":-783,"協":-1006,"口":483,"右":1233,"各":3588,"合":-241,"同":3906,"和":-837,"員":4513,"国":642,"型":1389,"場":1219,"外":-241,"妻":2016,"学":-1356,"安":-423,"実":-1008,"家":1078,"小":-513,"少":-3102,"州":1155,"市":3197,"平":-1804,"年":2416,"広":-1030,"府":1605,"度":1452,"建":-2352,"当":-3885,"得":1905,"思":-1291,"性":1822,"戸":-488,"指":-3973,"政":-2013,"教":-1479,"数":3222,"文":-1489,"新":1764,"日":2099,"旧":5792,"昨":-661,"時":-1248,"曜":-951,"最":-937,"月":4125,"期":360,"李":3094,"村":364,"東":-805,"核":5156,"森":2438,"業":484,"氏":2613,"民":-1694,"決":-1073,"法":1868,"海":-495,"無":979,"物":461,"特":-3850,"生":-273,"用":914,"町":1215,"的":7313,"直":-1835,"省":792,"県":6293,"知":-1528,"私":4231,"税":401,"立":-960,"第":1201,"米":7767,"系":3066,"約":3663,"級":1384,"統":-4229,"総":1163,"線":1255,"者":6457,"能":725,"自":-2869,"英":785,"見":1044,"調":-562,"財":-733,"費":1777,"車":1835,"軍":1375,"込":-1504,"通":-1136,"選":-681,"郎":1026,"郡":4404,"部":1200,"金":2163,"長":421,"開":-1432,"間":1302,"関":-1282,"雨":2009,"電":-1045,"非":2066,"駅":1620,"1":-800,"」":2670,"・":-3794,"ッ":-1350,"ア":551,"グ":1319,"ス":874,"ト":521,"ム":1109,"ル":1591,"ロ":2201,"ン":278};
this.UW4__ = {",":3930,".":3508,"―":-4841,"、":3930,"。":3508,"〇":4999,"「":1895,"」":3798,"〓":-5156,"あ":4752,"い":-3435,"う":-640,"え":-2514,"お":2405,"か":530,"が":6006,"き":-4482,"ぎ":-3821,"く":-3788,"け":-4376,"げ":-4734,"こ":2255,"ご":1979,"さ":2864,"し":-843,"じ":-2506,"す":-731,"ず":1251,"せ":181,"そ":4091,"た":5034,"だ":5408,"ち":-3654,"っ":-5882,"つ":-1659,"て":3994,"で":7410,"と":4547,"な":5433,"に":6499,"ぬ":1853,"ね":1413,"の":7396,"は":8578,"ば":1940,"ひ":4249,"び":-4134,"ふ":1345,"へ":6665,"べ":-744,"ほ":1464,"ま":1051,"み":-2082,"む":-882,"め":-5046,"も":4169,"ゃ":-2666,"や":2795,"ょ":-1544,"よ":3351,"ら":-2922,"り":-9726,"る":-14896,"れ":-2613,"ろ":-4570,"わ":-1783,"を":13150,"ん":-2352,"カ":2145,"コ":1789,"セ":1287,"ッ":-724,"ト":-403,"メ":-1635,"ラ":-881,"リ":-541,"ル":-856,"ン":-3637,"・":-4371,"ー":-11870,"一":-2069,"中":2210,"予":782,"事":-190,"井":-1768,"人":1036,"以":544,"会":950,"体":-1286,"作":530,"側":4292,"先":601,"党":-2006,"共":-1212,"内":584,"円":788,"初":1347,"前":1623,"副":3879,"力":-302,"動":-740,"務":-2715,"化":776,"区":4517,"協":1013,"参":1555,"合":-1834,"和":-681,"員":-910,"器":-851,"回":1500,"国":-619,"園":-1200,"地":866,"場":-1410,"塁":-2094,"士":-1413,"多":1067,"大":571,"子":-4802,"学":-1397,"定":-1057,"寺":-809,"小":1910,"屋":-1328,"山":-1500,"島":-2056,"川":-2667,"市":2771,"年":374,"庁":-4556,"後":456,"性":553,"感":916,"所":-1566,"支":856,"改":787,"政":2182,"教":704,"文":522,"方":-856,"日":1798,"時":1829,"最":845,"月":-9066,"木":-485,"来":-442,"校":-360,"業":-1043,"氏":5388,"民":-2716,"気":-910,"沢":-939,"済":-543,"物":-735,"率":672,"球":-1267,"生":-1286,"産":-1101,"田":-2900,"町":1826,"的":2586,"目":922,"省":-3485,"県":2997,"空":-867,"立":-2112,"第":788,"米":2937,"系":786,"約":2171,"経":1146,"統":-1169,"総":940,"線":-994,"署":749,"者":2145,"能":-730,"般":-852,"行":-792,"規":792,"警":-1184,"議":-244,"谷":-1000,"賞":730,"車":-1481,"軍":1158,"輪":-1433,"込":-3370,"近":929,"道":-1291,"選":2596,"郎":-4866,"都":1192,"野":-1100,"銀":-2213,"長":357,"間":-2344,"院":-2297,"際":-2604,"電":-878,"領":-1659,"題":-792,"館":-1984,"首":1749,"高":2120,"「":1895,"」":3798,"・":-4371,"ッ":-724,"ー":-11870,"カ":2145,"コ":1789,"セ":1287,"ト":-403,"メ":-1635,"ラ":-881,"リ":-541,"ル":-856,"ン":-3637};
this.UW5__ = {",":465,".":-299,"1":-514,"E2":-32768,"]":-2762,"、":465,"。":-299,"「":363,"あ":1655,"い":331,"う":-503,"え":1199,"お":527,"か":647,"が":-421,"き":1624,"ぎ":1971,"く":312,"げ":-983,"さ":-1537,"し":-1371,"す":-852,"だ":-1186,"ち":1093,"っ":52,"つ":921,"て":-18,"で":-850,"と":-127,"ど":1682,"な":-787,"に":-1224,"の":-635,"は":-578,"べ":1001,"み":502,"め":865,"ゃ":3350,"ょ":854,"り":-208,"る":429,"れ":504,"わ":419,"を":-1264,"ん":327,"イ":241,"ル":451,"ン":-343,"中":-871,"京":722,"会":-1153,"党":-654,"務":3519,"区":-901,"告":848,"員":2104,"大":-1296,"学":-548,"定":1785,"嵐":-1304,"市":-2991,"席":921,"年":1763,"思":872,"所":-814,"挙":1618,"新":-1682,"日":218,"月":-4353,"査":932,"格":1356,"機":-1508,"氏":-1347,"田":240,"町":-3912,"的":-3149,"相":1319,"省":-1052,"県":-4003,"研":-997,"社":-278,"空":-813,"統":1955,"者":-2233,"表":663,"語":-1073,"議":1219,"選":-1018,"郎":-368,"長":786,"間":1191,"題":2368,"館":-689,"1":-514,"E2":-32768,"「":363,"イ":241,"ル":451,"ン":-343};
this.UW6__ = {",":227,".":808,"1":-270,"E1":306,"、":227,"。":808,"あ":-307,"う":189,"か":241,"が":-73,"く":-121,"こ":-200,"じ":1782,"す":383,"た":-428,"っ":573,"て":-1014,"で":101,"と":-105,"な":-253,"に":-149,"の":-417,"は":-236,"も":-206,"り":187,"る":-135,"を":195,"ル":-673,"ン":-496,"一":-277,"中":201,"件":-800,"会":624,"前":302,"区":1792,"員":-1212,"委":798,"学":-960,"市":887,"広":-695,"後":535,"業":-697,"相":753,"社":-507,"福":974,"空":-822,"者":1811,"連":463,"郎":1082,"1":-270,"E1":306,"ル":-673,"ン":-496};
return this;
}
TinySegmenter.prototype.ctype_ = function(str) {
for (var i in this.chartype_) {
if (str.match(this.chartype_[i][0])) {
return this.chartype_[i][1];
}
}
return "O";
}
TinySegmenter.prototype.ts_ = function(v) {
if (v) { return v; }
return 0;
}
TinySegmenter.prototype.segment = function(input) {
if (input == null || input == undefined || input == "") {
return [];
}
var result = [];
var seg = ["B3","B2","B1"];
var ctype = ["O","O","O"];
var o = input.split("");
for (i = 0; i < o.length; ++i) {
seg.push(o[i]);
ctype.push(this.ctype_(o[i]))
}
seg.push("E1");
seg.push("E2");
seg.push("E3");
ctype.push("O");
ctype.push("O");
ctype.push("O");
var word = seg[3];
var p1 = "U";
var p2 = "U";
var p3 = "U";
for (var i = 4; i < seg.length - 3; ++i) {
var score = this.BIAS__;
var w1 = seg[i-3];
var w2 = seg[i-2];
var w3 = seg[i-1];
var w4 = seg[i];
var w5 = seg[i+1];
var w6 = seg[i+2];
var c1 = ctype[i-3];
var c2 = ctype[i-2];
var c3 = ctype[i-1];
var c4 = ctype[i];
var c5 = ctype[i+1];
var c6 = ctype[i+2];
score += this.ts_(this.UP1__[p1]);
score += this.ts_(this.UP2__[p2]);
score += this.ts_(this.UP3__[p3]);
score += this.ts_(this.BP1__[p1 + p2]);
score += this.ts_(this.BP2__[p2 + p3]);
score += this.ts_(this.UW1__[w1]);
score += this.ts_(this.UW2__[w2]);
score += this.ts_(this.UW3__[w3]);
score += this.ts_(this.UW4__[w4]);
score += this.ts_(this.UW5__[w5]);
score += this.ts_(this.UW6__[w6]);
score += this.ts_(this.BW1__[w2 + w3]);
score += this.ts_(this.BW2__[w3 + w4]);
score += this.ts_(this.BW3__[w4 + w5]);
score += this.ts_(this.TW1__[w1 + w2 + w3]);
score += this.ts_(this.TW2__[w2 + w3 + w4]);
score += this.ts_(this.TW3__[w3 + w4 + w5]);
score += this.ts_(this.TW4__[w4 + w5 + w6]);
score += this.ts_(this.UC1__[c1]);
score += this.ts_(this.UC2__[c2]);
score += this.ts_(this.UC3__[c3]);
score += this.ts_(this.UC4__[c4]);
score += this.ts_(this.UC5__[c5]);
score += this.ts_(this.UC6__[c6]);
score += this.ts_(this.BC1__[c2 + c3]);
score += this.ts_(this.BC2__[c3 + c4]);
score += this.ts_(this.BC3__[c4 + c5]);
score += this.ts_(this.TC1__[c1 + c2 + c3]);
score += this.ts_(this.TC2__[c2 + c3 + c4]);
score += this.ts_(this.TC3__[c3 + c4 + c5]);
score += this.ts_(this.TC4__[c4 + c5 + c6]);
// score += this.ts_(this.TC5__[c4 + c5 + c6]);
score += this.ts_(this.UQ1__[p1 + c1]);
score += this.ts_(this.UQ2__[p2 + c2]);
score += this.ts_(this.UQ3__[p3 + c3]);
score += this.ts_(this.BQ1__[p2 + c2 + c3]);
score += this.ts_(this.BQ2__[p2 + c3 + c4]);
score += this.ts_(this.BQ3__[p3 + c2 + c3]);
score += this.ts_(this.BQ4__[p3 + c3 + c4]);
score += this.ts_(this.TQ1__[p2 + c1 + c2 + c3]);
score += this.ts_(this.TQ2__[p2 + c2 + c3 + c4]);
score += this.ts_(this.TQ3__[p3 + c1 + c2 + c3]);
score += this.ts_(this.TQ4__[p3 + c2 + c3 + c4]);
var p = "O";
if (score > 0) {
result.push(word);
word = "";
p = "B";
}
p1 = p2;
p2 = p3;
p3 = p;
word += seg[i];
}
result.push(word);
return result;
}
lunr.TinySegmenter = TinySegmenter;
};
}));
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
{"version":3,"sources":["src/templates/assets/stylesheets/palette/_scheme.scss","../../../../src/templates/assets/stylesheets/palette.scss","src/templates/assets/stylesheets/palette/_accent.scss","src/templates/assets/stylesheets/palette/_primary.scss","src/templates/assets/stylesheets/utilities/_break.scss"],"names":[],"mappings":"AA2BA,cAGE,6BAME,sDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CACA,mDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CAGA,mDAAA,CACA,gDAAA,CACA,yDAAA,CACA,4DAAA,CAGA,0BAAA,CACA,mCAAA,CAGA,iCAAA,CACA,kCAAA,CACA,mCAAA,CACA,mCAAA,CACA,kCAAA,CACA,iCAAA,CACA,+CAAA,CACA,6DAAA,CACA,gEAAA,CACA,4DAAA,CACA,4DAAA,CACA,6DAAA,CAGA,6CAAA,CAGA,+CAAA,CAGA,uDAAA,CACA,6DAAA,CACA,2DAAA,CAGA,iCAAA,CAGA,yDAAA,CACA,iEAAA,CAGA,mDAAA,CACA,mDAAA,CAGA,qDAAA,CACA,uDAAA,CAGA,8DAAA,CAKA,8DAAA,CAKA,0DAAA,CAzEA,iBCiBF,CD6DE,kHAEE,YC3DJ,CDkFE,yDACE,4BChFJ,CD+EE,2DACE,4BC7EJ,CD4EE,gEACE,4BC1EJ,CDyEE,2DACE,4BCvEJ,CDsEE,yDACE,4BCpEJ,CDmEE,0DACE,4BCjEJ,CDgEE,gEACE,4BC9DJ,CD6DE,0DACE,4BC3DJ,CD0DE,2OACE,4BC/CJ,CDsDA,+FAGE,iCCpDF,CACF,CCjDE,2BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD6CN,CCvDE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDoDN,CC9DE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD2DN,CCrEE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDkEN,CC5EE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDyEN,CCnFE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDgFN,CC1FE,kCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDuFN,CCjGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD8FN,CCxGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDqGN,CC/GE,6BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD4GN,CCtHE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDmHN,CC7HE,4BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD6HN,CCpIE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDoIN,CC3IE,6BACE,yBAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD2IN,CClJE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDkJN,CCzJE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDsJN,CE3JE,4BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwJN,CEnKE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgKN,CE3KE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwKN,CEnLE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgLN,CE3LE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwLN,CEnME,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgMN,CE3ME,mCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwMN,CEnNE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgNN,CE3NE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwNN,CEnOE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgON,CE3OE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwON,CEnPE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmPN,CE3PE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2PN,CEnQE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmQN,CE3QE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2QN,CEnRE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgRN,CE3RE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwRN,CEnSE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BF4RN,CE5SE,kCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BFqSN,CEtRE,sEACE,4BFyRJ,CE1RE,+DACE,4BF6RJ,CE9RE,iEACE,4BFiSJ,CElSE,gEACE,4BFqSJ,CEtSE,iEACE,4BFySJ,CEhSA,8BACE,mDAAA,CACA,4DAAA,CACA,0DAAA,CACA,oDAAA,CACA,2DAAA,CAGA,4BFiSF,CE9RE,yCACE,+BFgSJ,CE7RI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCFiSN,CG7MI,mCD1EA,+CACE,8CF0RJ,CEvRI,qDACE,8CFyRN,CEpRE,iEACE,mCFsRJ,CACF,CGxNI,sCDvDA,uCACE,oCFkRJ,CACF,CEzQA,8BACE,kDAAA,CACA,4DAAA,CACA,wDAAA,CACA,oDAAA,CACA,6DAAA,CAGA,4BF0QF,CEvQE,yCACE,+BFyQJ,CEtQI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCF0QN,CEnQE,yCACE,6CFqQJ,CG9NI,0CDhCA,8CACE,gDFiQJ,CACF,CGnOI,0CDvBA,iFACE,6CF6PJ,CACF,CG3PI,sCDKA,uCACE,6CFyPJ,CACF","file":"palette.css"}
File diff suppressed because it is too large Load Diff
@@ -9,10 +9,6 @@
<link rel="prev" href="../obipipeline/">
<link rel="next" href="../mphf/">
@@ -23,7 +19,7 @@
<title>On-disk storage - obikmer</title>
<title>Chunkreader.refs - obikmer</title>
@@ -64,7 +60,7 @@
<div data-md-component="skip">
<a href="#on-disk-collection-structure" class="md-skip">
<a href="#coverage-implementationchunkreadermd" class="md-skip">
Skip to content
</a>
@@ -100,7 +96,7 @@
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
On-disk storage
Chunkreader.refs
</span>
</div>
@@ -182,6 +178,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +214,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +235,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +386,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -374,8 +425,6 @@
@@ -383,14 +432,14 @@
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +457,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -563,19 +612,10 @@
<li class="md-nav__item md-nav__item--active">
<input class="md-nav__toggle md-toggle" type="checkbox" id="__toc">
<a href="./" class="md-nav__link md-nav__link--active">
<li class="md-nav__item">
<a href="../storage/" class="md-nav__link">
@@ -591,7 +631,6 @@
</a>
</li>
@@ -660,14 +699,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -737,6 +804,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -759,10 +1022,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -780,8 +1043,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -847,6 +1110,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
@@ -874,6 +1221,36 @@
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#code-couvert" class="md-nav__link">
<span class="md-ellipsis">
Code couvert
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#notes" class="md-nav__link">
<span class="md-ellipsis">
Notes
</span>
</a>
</li>
</ul>
</nav>
</div>
</div>
@@ -889,9 +1266,16 @@
<h1 id="on-disk-collection-structure">On-disk collection structure</h1>
<p>See <a href="../obilayeredmap/">obilayeredmap crate</a> for the current on-disk layout.</p>
<p>The index root contains one <code>part_XXXXX/</code> directory per partition, each holding one or more <code>layer_N/</code> directories. Each layer directory contains <code>mphf.bin</code>, <code>unitigs.bin</code>, <code>unitigs.bin.idx</code>, <code>evidence.bin</code>, and optionally a <code>counts/</code> or <code>presence/</code> payload directory.</p>
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
<h1 id="coverage-implementationchunkreadermd">Coverage: implementation/chunkreader.md</h1>
<h2 id="code-couvert">Code couvert</h2>
<ul>
<li><code>obiread/src/chunk.rs</code> — SeqChunkIter, détection de frontières FASTA/FASTQ, state machines</li>
<li><code>obikrope/src/lib.rs</code> — type Rope (Vec<Bytes>), opérations zero-copy</li>
</ul>
<h2 id="notes">Notes</h2>
<p>Document stable (la stratégie de chunking rope ne devrait pas avoir changé).
Vérifier que le split FASTA/FASTQ reste correct si de nouveaux formats ont été ajoutés.</p>
@@ -77,12 +77,24 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../../installation/">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" id="__nav_2" type="checkbox"/>
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<input class="md-nav__toggle md-toggle" id="__nav_3" type="checkbox"/>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<span class="md-ellipsis">
@@ -93,8 +105,8 @@
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav aria-expanded="false" aria-labelledby="__nav_2_label" class="md-nav" data-md-level="1">
<label class="md-nav__title" for="__nav_2">
<nav aria-expanded="false" aria-labelledby="__nav_3_label" class="md-nav" data-md-level="1">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -160,6 +172,18 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../../theory/evolutionary_distances/">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
@@ -167,8 +191,8 @@
</nav>
</li>
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input checked="" class="md-nav__toggle md-toggle" id="__nav_3" type="checkbox"/>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<input checked="" class="md-nav__toggle md-toggle" id="__nav_4" type="checkbox"/>
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<span class="md-ellipsis">
@@ -179,8 +203,8 @@
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav aria-expanded="true" aria-labelledby="__nav_3_label" class="md-nav" data-md-level="1">
<label class="md-nav__title" for="__nav_3">
<nav aria-expanded="true" aria-labelledby="__nav_4_label" class="md-nav" data-md-level="1">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -243,19 +267,28 @@
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix="">
<li class="md-nav__item">
<a class="md-nav__link" href="#output-type-rope">
<a class="md-nav__link" href="#two-reading-paths">
<span class="md-ellipsis">
Output type: rope
Two reading paths
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#allocation-policy">
<a class="md-nav__link" href="#record-path-chunk-reader">
<span class="md-ellipsis">
Allocation policy
Record path: chunk reader
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#output-type-rope">
<span class="md-ellipsis">
Output type: Rope
</span>
</a>
@@ -287,6 +320,15 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#future-work-parallel-gzip-decompression-in-xopen">
<span class="md-ellipsis">
Future work — parallel gzip decompression in xopen
</span>
</a>
</li>
</ul>
</nav>
</li>
@@ -351,11 +393,23 @@
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../obilayeredmap/">
<a class="md-nav__link" href="../evidence_elimination/">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../layer_tier/">
<span class="md-ellipsis">
obikindex layer tier
@@ -383,6 +437,90 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../merge/">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../merge_parallelism/">
<span class="md-ellipsis">
Merge parallelism &amp; memory
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../filtering/">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../select/">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../obitaxonomy/">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../benchmark_query_testing/">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../partition_layer_cache/">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
@@ -390,8 +528,8 @@
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" id="__nav_4" type="checkbox"/>
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<input class="md-nav__toggle md-toggle" id="__nav_5" type="checkbox"/>
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
<span class="md-ellipsis">
@@ -402,8 +540,8 @@
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav aria-expanded="false" aria-labelledby="__nav_4_label" class="md-nav" data-md-level="1">
<label class="md-nav__title" for="__nav_4">
<nav aria-expanded="false" aria-labelledby="__nav_5_label" class="md-nav" data-md-level="1">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -433,6 +571,42 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../../architecture/siblings/">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../../architecture/numa_worker_pools/">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="../../architecture/numa_partition_runner/">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
@@ -454,19 +628,28 @@
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix="">
<li class="md-nav__item">
<a class="md-nav__link" href="#output-type-rope">
<a class="md-nav__link" href="#two-reading-paths">
<span class="md-ellipsis">
Output type: rope
Two reading paths
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#allocation-policy">
<a class="md-nav__link" href="#record-path-chunk-reader">
<span class="md-ellipsis">
Allocation policy
Record path: chunk reader
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#output-type-rope">
<span class="md-ellipsis">
Output type: Rope
</span>
</a>
@@ -498,6 +681,15 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#future-work-parallel-gzip-decompression-in-xopen">
<span class="md-ellipsis">
Future work — parallel gzip decompression in xopen
</span>
</a>
</li>
</ul>
</nav>
</div>
@@ -506,68 +698,77 @@
<div class="md-content" data-md-component="content">
<article class="md-content__inner md-typeset">
<h1 id="chunk-reader-implementation">Chunk reader — implementation</h1>
<p>The <code>obiread</code> crate provides a streaming iterator that reads FASTA or FASTQ files in fixed-size blocks and yields self-contained chunks, each ending on a complete sequence record boundary. Chunks are consumed in parallel by downstream workers.</p>
<h2 id="output-type-rope">Output type: rope</h2>
<p>Each chunk is a <code>Vec&lt;Bytes&gt;</code> — a <strong>rope</strong>: a list of reference-counted byte slices that are not necessarily contiguous in memory. The consumer iterates over the slices in order.</p>
<p>Using <code>bytes::Bytes</code> means the split at the record boundary is O(1): <code>Bytes::split_to(n)</code> adjusts a reference counter, not memory. No <code>memcpy</code> in the common case.</p>
<h2 id="allocation-policy">Allocation policy</h2>
<p><code>obiread</code> exposes two distinct sequence reading paths, each optimised for a different use case.</p>
<h2 id="two-reading-paths">Two reading paths</h2>
<table>
<thead>
<tr>
<th>Case</th>
<th>Cost</th>
<th>Path</th>
<th>API</th>
<th>Output unit</th>
<th>Per-record identity</th>
<th>Use case</th>
</tr>
</thead>
<tbody>
<tr>
<td>Boundary found in the current block (common)</td>
<td>zero extra allocation — <code>split_to</code> only</td>
<td><strong>Record path</strong></td>
<td><code>read_sequence_chunks</code> → <code>parse_chunk</code></td>
<td><code>SeqRecord</code> (id + raw sequence + normalised rope)</td>
<td>yes</td>
<td><code>query</code> — must read complete records</td>
</tr>
<tr>
<td>Boundary straddles multiple blocks (sequence &gt; block size, rare)</td>
<td>one allocation to pack the rope into a flat buffer</td>
</tr>
<tr>
<td>EOF flush</td>
<td>zero extra allocation</td>
<td><strong>Stream path</strong></td>
<td><code>open_nuc_stream</code></td>
<td><code>NucPage</code> (flat normalised byte buffer)</td>
<td>no</td>
<td><code>index</code>, <code>superkmer</code> — bulk throughput</td>
</tr>
</tbody>
</table>
<p>The record path uses <code>Rope</code>-backed chunks and is described in detail below.
The stream path (<code>NucStream</code> / <code>NucPage</code>) is described in the scatter section of <a href="../pipeline/">pipeline</a>.</p>
<hr/>
<h2 id="record-path-chunk-reader">Record path: chunk reader</h2>
<p>The chunk reader reads FASTA or FASTQ files in fixed-size blocks and yields self-contained chunks, each ending on a complete sequence record boundary. <code>parse_chunk</code> then converts each chunk into a <code>Vec&lt;SeqRecord&gt;</code>, where each record carries its identifier, raw sequence bytes, and a normalised rope ready for superkmer building.</p>
<p>This path is mandatory for <code>query</code>, where superkmers must be tracked back to their originating sequence (id, kmer offset) for output annotation.</p>
<h2 id="output-type-rope">Output type: Rope</h2>
<p>Each chunk is a <code>Rope</code> — a segmented byte sequence: a <code>Vec</code> of blocks, where each block is a <code>Vec&lt;Cell&lt;u8&gt;&gt;</code>. The consumer iterates over the blocks via a forward or backward cursor.</p>
<p><code>Rope::split_off(pos)</code> splits at an absolute byte offset in O(log n) (binary search over block-start index). If <code>pos</code> falls inside a block, that block is split in two via <code>Vec::split_off</code> — no <code>memcpy</code> in the common case.</p>
<h2 id="seqchunkiter">SeqChunkIter</h2>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">SeqChunkIter</span><span class="o">&lt;</span><span class="n">R</span><span class="p">:</span><span class="w"> </span><span class="nc">Read</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* private */</span><span class="w"> </span><span class="p">}</span>
<span class="k">impl</span><span class="o">&lt;</span><span class="n">R</span><span class="p">:</span><span class="w"> </span><span class="nc">Read</span><span class="o">&gt;</span><span class="w"> </span><span class="nb">Iterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">SeqChunkIter</span><span class="o">&lt;</span><span class="n">R</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">io</span><span class="p">::</span><span class="nb">Result</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="n">Bytes</span><span class="o">&gt;&gt;</span><span class="p">;</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">io</span><span class="p">::</span><span class="nb">Result</span><span class="o">&lt;</span><span class="n">Rope</span><span class="o">&gt;</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fasta_chunks</span><span class="o">&lt;</span><span class="n">R</span><span class="p">:</span><span class="w"> </span><span class="nc">Read</span><span class="o">&gt;</span><span class="p">(</span><span class="n">source</span><span class="p">:</span><span class="w"> </span><span class="nc">R</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">SeqChunkIter</span><span class="o">&lt;</span><span class="n">R</span><span class="o">&gt;</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fastq_chunks</span><span class="o">&lt;</span><span class="n">R</span><span class="p">:</span><span class="w"> </span><span class="nc">Read</span><span class="o">&gt;</span><span class="p">(</span><span class="n">source</span><span class="p">:</span><span class="w"> </span><span class="nc">R</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">SeqChunkIter</span><span class="o">&lt;</span><span class="n">R</span><span class="o">&gt;</span>
</code></pre></div>
<p><code>next()</code> loop:</p>
<div class="highlight"><pre><span></span><code>1. read one block of block_size bytes → push onto rope
2. probe check: if the boundary marker ("\n&gt;" or "\n@") is absent from the
last block, skip the splitter (avoids a full backward scan for nothing)
3. call splitter on last block
if found at offset n:
remainder = last_block.split_to(n) ← O(1), zero copy
return std::mem::take(&amp;mut self.rope) ← the chunk
4. if rope.len() &gt; 1 (multi-block accumulation):
pack rope into one flat buffer ← one alloc
retry splitter on flat buffer
5. if EOF: flush remaining rope as final chunk
<div class="highlight"><pre><span></span><code>1. read one block of block_size bytes → push onto Rope
2. call splitter(rope) → Option&lt;abs_offset&gt;
if Some(pos):
tail = rope.split_off(pos) ← O(log n), may split one block
chunk = mem::replace(&amp;mut rope, tail)
return Some(Ok(chunk))
3. if EOF and rope non-empty: return Some(Ok(rope)) as final chunk
4. if EOF and rope empty: return None
</code></pre></div>
<p>The <code>Splitter</code> function signature is <code>fn(&amp;Rope) -&gt; Option&lt;usize&gt;</code>. It returns the absolute byte offset of the start of the last complete record, or <code>None</code> if no boundary was found in the accumulated rope (need more data).</p>
<h2 id="boundary-detection-fasta">Boundary detection — FASTA</h2>
<p>Backward scan with a 2-state machine. Searches for <code>&gt;</code> immediately preceded by <code>\n</code> or <code>\r</code>:</p>
<p>Backward scan with a 2-state machine. Searches (right to left) for <code>&gt;</code> followed by <code>\n</code> or <code>\r</code> (i.e., a <code>&gt;</code> that is preceded by a newline in forward order):</p>
<pre class="mermaid"><code>stateDiagram-v2
direction LR
[*] --&gt; Scanning
Scanning --&gt; FoundGt : '&gt;'
FoundGt --&gt; Scanning : other
FoundGt --&gt; [*] : '\\n' / '\\r' ✓</code></pre>
<p>Returns the byte offset of the <code>&gt;</code> that starts the last complete record.</p>
<p>Returns the byte offset of the <code>&gt;</code> that starts the last complete record. Returns <code>None</code> if only one <code>&gt;</code> is found (cannot confirm there is a prior complete record).</p>
<h2 id="boundary-detection-fastq">Boundary detection — FASTQ</h2>
<p>FASTQ records have a rigid 4-line structure (<code>@header</code>, sequence, <code>+</code>, quality). The <code>@</code> character (ASCII 64, Phred score 31) can appear legitimately in quality lines, making any forward heuristic unreliable. The backward scanner verifies the full structural context before accepting a candidate <code>@</code>.</p>
<p>7-state machine (port of Go's <code>EndOfLastFastqEntry</code>), scanning from <strong>right to left</strong>. Each time a <code>+</code> is found, its position is saved as <code>restart</code>; any state mismatch resets the scan to that position.</p>
<p>7-state machine (states 0–6), scanning from <strong>right to left</strong>. Each time a <code>+</code> is found, its position is saved as <code>restart</code>; any state mismatch resets the scan to that position.</p>
<pre class="mermaid"><code>stateDiagram-v2
direction LR
@@ -596,6 +797,16 @@
FoundAt --&gt; InHeader : other</code></pre>
<p><code>restart</code> is updated each time a <code>+</code> is found. When any state fails its expected input, the scan jumps back to <code>restart</code> and continues from there — guaranteeing that a <code>@</code> in a quality line cannot be accepted as a record start, because the <code>\n+\n</code> structure immediately following it (going backward) will not be found.</p>
<p>Returns the byte offset of the <code>@</code> that starts the last complete record.</p>
<hr/>
<h2 id="future-work-parallel-gzip-decompression-in-xopen">Future work — parallel gzip decompression in <code>xopen</code></h2>
<p><code>obiread::xopen</code> (<code>xopen.rs</code>) decompresses gzip via <code>niffler</code> → <code>flate2</code>, which is single-threaded (standard DEFLATE has no parallel-decodable structure). For large local gzip inputs this single-threaded decompression can become the throughput bottleneck feeding the <code>query</code>/<code>index</code>/<code>superkmer</code> pipelines, since chunk/page production for a given file is serialized ahead of the worker pool.</p>
<p>Candidate: special-case local, on-disk, gzip-magic-detected paths in <code>open_raw</code>/<code>xopen</code> to use <a href="https://github.com/alekseizarubin/rapidgzip-rs"><code>rapidgzip-rs</code></a> (<code>ReaderBuilder::new().parallelism(n).open(path)</code>, implements <code>Read + Seek</code>) instead of <code>niffler</code>, keeping <code>niffler</code> for every other case: <code>stdin</code> (<code>-</code>), HTTP(S) sources, and all non-gzip formats (bzip2, xz, zstd — less used in practice here).</p>
<p>Constraints identified so far (not yet validated against real data):
- Branch point must move earlier than the current <code>decompress()</code> call in <code>open_raw</code> — rapidgzip's fast path needs the file <strong>path</strong>, not an already-opened generic <code>Read</code>, so the gzip/local-file detection has to happen before the generic <code>File::open</code> + <code>niffler::send::get_reader</code> path is taken.
- <code>stdin</code> and HTTP sources are not seekable — they stay on <code>niffler</code> regardless; the gain only applies to local on-disk <code>.gz</code> files.
- <code>rapidgzip-sys</code> vendors a native C++ engine: requires CMake ≥ 3.17, a C++17 compiler, and <code>nasm</code> on x86 targets — a real build-toolchain addition, not just a pure-Rust crate.
- Low maturity of the Rust binding at review time (2 GitHub stars, ~15 commits, April 2026 latest release) — the underlying C++ engine is validated (HPDC 2023 paper), but the binding itself has limited production track record.</p>
<p>Decision: parked for now. Before adopting, validate on real data: throughput vs. <code>niffler</code> on representative large <code>.gz</code> inputs, and byte-for-byte correctness of decompressed output.</p>
</article>
</div>
<script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script>
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -387,10 +442,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +463,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -514,10 +569,21 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#memory-layout" class="md-nav__link">
<a href="#types-and-layout" class="md-nav__link">
<span class="md-ellipsis">
Memory layout
Types and layout
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#global-parameters" class="md-nav__link">
<span class="md-ellipsis">
Global parameters
</span>
</a>
@@ -558,10 +624,32 @@
</li>
<li class="md-nav__item">
<a href="#canonical-form" class="md-nav__link">
<a href="#canonical-form-and-canonicalkmerof" class="md-nav__link">
<span class="md-ellipsis">
Canonical form
Canonical form and CanonicalKmerOf
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sliding-window-helpers" class="md-nav__link">
<span class="md-ellipsis">
Sliding window helpers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#hashing" class="md-nav__link">
<span class="md-ellipsis">
Hashing
</span>
</a>
@@ -752,14 +840,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -829,6 +945,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -851,10 +1163,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -872,8 +1184,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -939,6 +1251,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
@@ -973,10 +1369,21 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#memory-layout" class="md-nav__link">
<a href="#types-and-layout" class="md-nav__link">
<span class="md-ellipsis">
Memory layout
Types and layout
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#global-parameters" class="md-nav__link">
<span class="md-ellipsis">
Global parameters
</span>
</a>
@@ -1017,10 +1424,32 @@
</li>
<li class="md-nav__item">
<a href="#canonical-form" class="md-nav__link">
<a href="#canonical-form-and-canonicalkmerof" class="md-nav__link">
<span class="md-ellipsis">
Canonical form
Canonical form and CanonicalKmerOf
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sliding-window-helpers" class="md-nav__link">
<span class="md-ellipsis">
Sliding window helpers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#hashing" class="md-nav__link">
<span class="md-ellipsis">
Hashing
</span>
</a>
@@ -1045,12 +1474,43 @@
<h1 id="kmer-implementation">Kmer — implementation</h1>
<h2 id="memory-layout">Memory layout</h2>
<p><code>Kmer</code> is a <code>#[repr(transparent)]</code> newtype over <code>u64</code>:</p>
<h2 id="types-and-layout">Types and layout</h2>
<p><code>KmerOf&lt;L&gt;</code> is a <code>#[repr(transparent)]</code> newtype over <code>u64</code> parameterized by a <code>KmerLength</code> marker:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#[repr(transparent)]</span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">Kmer</span><span class="p">(</span><span class="kt">u64</span><span class="p">);</span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">KmerOf</span><span class="o">&lt;</span><span class="n">L</span><span class="p">:</span><span class="w"> </span><span class="nc">KmerLength</span><span class="o">&gt;</span><span class="p">(</span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">PhantomData</span><span class="o">&lt;</span><span class="n">L</span><span class="o">&gt;</span><span class="p">);</span>
</code></pre></div>
<p>Nucleotides are packed 2 bits each, <strong>left-aligned</strong>, MSB-first. Nucleotide 0 occupies bits 63–62; nucleotide i occupies bits 63−2i and 62−2i. The low 64−2k bits are always zero. k is <strong>not stored</strong> — it is a parameter of every operation that needs it, and will be owned by the future collection-level indexer.</p>
<p>Three marker types implement <code>KmerLength</code>:</p>
<table>
<thead>
<tr>
<th>Marker</th>
<th><code>len()</code> source</th>
<th>Used for</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>KLen</code></td>
<td><code>params::k()</code></td>
<td>k-mers</td>
</tr>
<tr>
<td><code>MLen</code></td>
<td><code>params::m()</code></td>
<td>minimizers</td>
</tr>
<tr>
<td><code>ConstLen&lt;N&gt;</code></td>
<td>const generic <code>N</code></td>
<td>tests</td>
</tr>
</tbody>
</table>
<p>Public aliases:</p>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Kmer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">KmerOf</span><span class="o">&lt;</span><span class="n">KLen</span><span class="o">&gt;</span><span class="p">;</span><span class="w"> </span><span class="c1">// k-mer, global k</span>
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Minimizer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CanonicalKmerOf</span><span class="o">&lt;</span><span class="n">MLen</span><span class="o">&gt;</span><span class="p">;</span><span class="w"> </span><span class="c1">// canonical m-mer, global m</span>
</code></pre></div>
<p>Nucleotides are packed 2 bits each, <strong>left-aligned</strong>, MSB-first. Nucleotide 0 occupies bits 63–62; nucleotide i occupies bits 63−2i and 62−2i. The low 64−2·len bits are always zero. The length is <strong>not stored</strong> — every operation reads it from <code>L::len()</code>.</p>
<table>
<thead>
<tr>
@@ -1071,33 +1531,41 @@
</tr>
</tbody>
</table>
<h2 id="global-parameters">Global parameters</h2>
<p><code>params::set_k(k)</code> / <code>params::k()</code> and <code>params::set_m(m)</code> / <code>params::m()</code> are backed by <code>OnceLock&lt;usize&gt;</code> in production (write-once, panic on conflict) and by <code>thread_local! { Cell&lt;usize&gt; }</code> in test builds (per-thread, freely writable). <code>params::init(k, m)</code> sets both in one call.</p>
<h2 id="encoding">Encoding</h2>
<p><code>Kmer::from_ascii(ascii, k)</code> encodes the first k bytes of an ASCII slice using the shared <code>ENC</code> table (see <a href="../superkmer/#ascii-encoding-and-decoding">SuperKmer — ASCII encoding</a>):</p>
<p><code>KmerOf::&lt;L&gt;::from_ascii(ascii)</code> encodes the first <code>L::len()</code> bytes using the shared <code>ENC</code> table (see <a href="../superkmer/#ascii-encoding-and-decoding">SuperKmer — ASCII encoding</a>):</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="o">..</span><span class="n">k</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">val</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">val</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">encode_base</span><span class="p">(</span><span class="n">ascii</span><span class="p">[</span><span class="n">i</span><span class="p">])</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">u64</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">Kmer</span><span class="p">(</span><span class="n">val</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="p">(</span><span class="mi">64</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">k</span><span class="p">))</span>
<span class="n">KmerOf</span><span class="p">(</span><span class="n">val</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="p">(</span><span class="mi">64</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">k</span><span class="p">),</span><span class="w"> </span><span class="n">PhantomData</span><span class="p">)</span>
</code></pre></div>
<p>Zero allocation — result lives on the stack.</p>
<h2 id="decoding">Decoding</h2>
<p><code>write_ascii(k, buf)</code> appends k ASCII characters to a caller-supplied <code>Vec&lt;u8&gt;</code> using the shared <code>DEC4</code> table: one lookup per 4 nucleotides, two partial-byte lookups for the remainder. No allocation in the hot path.</p>
<p><code>to_ascii(k)</code> is a convenience wrapper that allocates and returns a <code>Vec&lt;u8&gt;</code>; intended for tests and display only.</p>
<p><code>write_ascii(writer)</code> writes k ASCII characters to any <code>W: Write</code> using the shared <code>DEC4</code> table: one lookup per 4 nucleotides, one partial lookup for the remainder. No allocation in the hot path.</p>
<p><code>to_ascii()</code> is a convenience wrapper that allocates and returns a <code>Vec&lt;u8&gt;</code>; intended for tests and display only.</p>
<h2 id="reverse-complement">Reverse complement</h2>
<p>Computed as pure arithmetic — no lookup table, no memory access:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">!</span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="c1">// complement</span>
<span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">swap_bytes</span><span class="p">();</span><span class="w"> </span><span class="c1">// reverse bytes</span>
<span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">((</span><span class="n">x</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">4</span><span class="p">)</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mh">0x0F0F0F0F0F0F0F0F</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="p">((</span><span class="n">x</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mh">0x0F0F0F0F0F0F0F0F</span><span class="p">)</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="mi">4</span><span class="p">);</span><span class="w"> </span><span class="c1">// swap nibbles</span>
<span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">((</span><span class="n">x</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mh">0x3333333333333333</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="p">((</span><span class="n">x</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mh">0x3333333333333333</span><span class="p">)</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="mi">2</span><span class="p">);</span><span class="w"> </span><span class="c1">// swap 2-bit groups</span>
<span class="n">Kmer</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="p">(</span><span class="mi">64</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">k</span><span class="p">))</span>
<span class="n">KmerOf</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="p">(</span><span class="mi">64</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">k</span><span class="p">),</span><span class="w"> </span><span class="n">PhantomData</span><span class="p">)</span>
</code></pre></div>
<p>After complementing, bytes are reversed (<code>swap_bytes</code>), then nibbles, then 2-bit groups — restoring 2-bit nucleotides to their correct positions in reverse order. A final left-shift realigns to MSB. Zero allocation — result lives on the stack.</p>
<h2 id="canonical-form">Canonical form</h2>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">canonical</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">k</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">rc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">revcomp</span><span class="p">(</span><span class="n">k</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">&lt;=</span><span class="w"> </span><span class="n">rc</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">*</span><span class="bp">self</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">rc</span><span class="w"> </span><span class="p">}</span>
<h2 id="canonical-form-and-canonicalkmerof">Canonical form and <code>CanonicalKmerOf</code></h2>
<p><code>canonical()</code> returns a <code>CanonicalKmerOf&lt;L&gt;</code> — a distinct newtype that carries the same <code>u64</code> layout but enforces the invariant that the stored value equals <code>min(kmer, revcomp)</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">canonical</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">CanonicalKmerOf</span><span class="o">&lt;</span><span class="n">L</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">rc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">revcomp</span><span class="p">();</span>
<span class="w"> </span><span class="n">CanonicalKmerOf</span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">&lt;=</span><span class="w"> </span><span class="n">rc</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">rc</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="n">PhantomData</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>Lexicographic minimum of forward and reverse-complement, comparing the raw <code>u64</code> values directly (left-aligned encoding makes this equivalent to nucleotide-wise comparison). Zero allocation — result lives on the stack.</p>
<p><code>CanonicalKmerOf::from_raw_unchecked(raw)</code> is the only other public constructor, for trusted paths such as deserialisation.</p>
<h2 id="sliding-window-helpers">Sliding window helpers</h2>
<p><code>push_right(nuc)</code> / <code>push_left(nuc)</code> shift the window by one base in O(1). <code>is_overlapping(other)</code> checks whether the last k−1 nucleotides of <code>self</code> equal the first k−1 of <code>other</code>.</p>
<h2 id="hashing">Hashing</h2>
<p><code>hash_kmer(raw: u64) -&gt; u64</code> computes <code>mix64(raw ^ 0x9e3779b97f4a7c15)</code>, the seeded splitmix64 finalizer. <code>CanonicalKmerOf::seq_hash()</code> delegates to <code>hash_kmer</code>.</p>
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -387,10 +442,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +463,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -757,6 +812,28 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#evidence-modes" class="md-nav__link">
<span class="md-ellipsis">
Evidence modes
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#build-functions" class="md-nav__link">
<span class="md-ellipsis">
Build functions
</span>
</a>
</li>
<li class="md-nav__item">
@@ -841,14 +918,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -918,6 +1023,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -940,10 +1241,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -961,8 +1262,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -1028,6 +1329,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
@@ -1165,6 +1550,28 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#evidence-modes" class="md-nav__link">
<span class="md-ellipsis">
Evidence modes
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#build-functions" class="md-nav__link">
<span class="md-ellipsis">
Build functions
</span>
</a>
</li>
<li class="md-nav__item">
@@ -1226,26 +1633,26 @@
<h2 id="why-two-phases-are-needed">Why two phases are needed</h2>
<p>Kmer indexing per partition proceeds in two phases. The separation is necessary because the exact number of surviving unique kmers is not known until after counting and filtering low-abundance kmers.</p>
<h3 id="phase-1-provisional-mphf-kmer-spectrum">Phase 1 — provisional MPHF + kmer spectrum</h3>
<p>Implemented in <code>obikpartitionner::KmerPartition::count_kmer()</code>.</p>
<p>Implemented in <code>obikindex::partition::KmerPartition::count_kmer()</code> → <code>count_partition()</code>.</p>
<ol>
<li><strong>Pass 1</strong>: read the dereplicated superkmer file; enumerate all unique canonical kmers into a <code>HashSet</code>. Exact count known after this pass.</li>
<li><strong>Build a provisional MPHF</strong> (<code>GOFunction</code> from the <code>ph</code> crate) over the exact kmer set. Produces <code>mphf1.bin</code>.</li>
<li><strong>Create <code>counts1.bin</code></strong>: one zero-initialised <code>u32</code> per MPHF slot (mmap'd).</li>
<li><strong>Pass 2</strong>: re-read the dereplicated file; for each kmer, query <code>mphf1.get(kmer)</code> and atomically accumulate the superkmer count into <code>counts1[slot]</code>.</li>
<li><strong>External sort</strong>: read the dereplicated superkmer file; extract the raw <code>u64</code> canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: <code>sorted_unique.bin</code> — a flat array of f0 distinct sorted <code>u64</code> values. Exact kmer count f0 is known at this point.</li>
<li><strong>Build provisional MPHF</strong> (ptr_hash, same configuration as phase 2) over <code>sorted_unique.bin</code> using <code>new_from_par_iter</code>. Delete <code>sorted_unique.bin</code> immediately after. Persist to <code>mphf1.bin</code>.</li>
<li><strong>Create <code>counts1.bin</code></strong>: <code>PersistentCompactIntVec</code> with f0 slots, zero-initialised.</li>
<li><strong>Accumulation pass</strong>: re-read the dereplicated superkmer file; for each kmer in each superkmer, compute <code>slot = mphf.index(kmer.raw())</code> and increment <code>counts1[slot]</code> by the superkmer's COUNT.</li>
<li><strong>Build kmer frequency spectrum</strong> from <code>counts1</code>: histogram <code>{count → n_kmers}</code>, totals f0 (distinct kmers) and f1 (total abundance). Written to <code>kmer_spectrum_raw.json</code> per partition, then merged globally.</li>
</ol>
<p>Files produced per partition:</p>
<div class="highlight"><pre><span></span><code>part_XXXXX/
mphf1.bin — GOFunction (provisional MPHF, discarded after phase 2)
counts1.bin — [u32; n_kmers] kmer counts, mmap&#39;d
mphf1.bin — ptr_hash provisional MPHF (discarded after phase 2)
counts1.bin — PersistentCompactIntVec, f0 × u32 kmer counts
kmer_spectrum_raw.json — local frequency spectrum
</code></pre></div>
<h3 id="phase-2-definitive-mphf">Phase 2 — definitive MPHF</h3>
<p>After filtering (applying a min-count threshold derived from the spectrum) and building the local De Bruijn graph + unitigs (see <a href="../pipeline/">Construction pipeline</a>), the exact filtered kmer set is available via <code>unitigs.bin</code>.</p>
<p><code>MphfLayer::build</code> is called on the unitig file:</p>
<p><code>MphfLayer::build(dir, block_bits, mode: &amp;IndexMode, fill_slot)</code> is called on the unitig directory:</p>
<ol>
<li><strong>Pass 1</strong>: iterate all canonical kmers from <code>unitigs.bin</code> in parallel, build and store <code>mphf.bin</code> (ptr_hash).</li>
<li><strong>Pass 2</strong>: iterate sequentially, fill <code>evidence.bin</code>, call the mode-specific <code>fill_slot</code> callback.</li>
<li><strong>Pass 1</strong> (parallel): a <code>CanonicalKmerIter</code> — clonable via <code>Arc&lt;Mmap&gt;</code>, no file reopening — is passed directly to <code>new_from_par_iter</code> via <code>par_bridge()</code>. No <code>.idx</code> is read or created at this stage; parallelism is at partition/layer level, not within a single MPHF. Produces <code>mphf.bin</code>.</li>
<li><strong>Pass 2</strong> (sequential): iterate with <code>iter_indexed_canonical_kmers</code>; fill evidence files; call <code>fill_slot(slot, kmer)</code> callback per kmer. For Exact/Hybrid, <code>.idx</code> is written at the end of this pass — never earlier.</li>
</ol>
<p><code>mphf1.bin</code> and <code>counts1.bin</code> are no longer needed after phase 2 and can be deleted.</p>
<hr />
@@ -1265,13 +1672,11 @@
<p><strong>FMPH/FMPHGO</strong> (<code>ph</code> crate, Beling, ACM JEA 2023):</p>
<ul>
<li>~2.1 bits/key — most compact; good query speed; deterministic construction</li>
<li>Works well from an exact or slightly overestimated count</li>
<li><code>GOFunction</code> (group-oriented variant) is the specific type used</li>
<li><code>GOFunction</code> (group-oriented variant) was the original phase-1 choice; eliminated when the external sort made the exact count available at phase 1 as well</li>
</ul>
<h2 id="mphf-choice-per-phase">MPHF choice per phase</h2>
<p><strong>Phase 1</strong> (provisional, discarded after spectrum computation): <code>ph::fmph::GOFunction</code>. Compact, fast to build from the exact post-dedup kmer set. Query speed is secondary — the structure is only used during pass 2 of <code>count_kmer</code>.</p>
<p><strong>Phase 2</strong> (persistent, queried repeatedly): <strong>ptr_hash</strong>. Exact key count is available from the unitig index; ptr_hash query speed (≥2.1×) and construction speed (≥3.1× over FMPH) are the decisive factors. The 2.4 bits/key overhead is acceptable.</p>
<p>boomphf is eliminated: largest space overhead, streaming advantage does not apply.</p>
<p><strong>Both phases</strong>: <strong>ptr_hash</strong>, same type alias and construction parameters. The external sort (phase 1) and the unitig index (phase 2) both provide the exact key count before MPHF construction, so ptr_hash's requirement is satisfied in both cases. Using a single MPHF implementation removes the <code>ph</code> crate dependency.</p>
<p>boomphf: eliminated — largest space overhead, streaming advantage no longer needed. FMPH/GOFunction: eliminated — exact count available, ptr_hash is faster at equivalent compactness.</p>
<hr />
<h2 id="space-at-scale">Space at scale</h2>
<p>For 1 024 partitions × 100 M kmers/partition (phase 2 index, after filtering):</p>
@@ -1318,11 +1723,14 @@
<hr />
<h2 id="multilayer-index-architecture">Multilayer index architecture</h2>
<h3 id="layer-structure">Layer structure</h3>
<p>Each layer is a self-contained unit. See <a href="../obilayeredmap/">obilayeredmap</a> for the full on-disk layout. The MPHF-relevant files are:</p>
<p>Each layer is a self-contained unit. See <a href="../layer_tier/">obikindex::layer</a> for the full on-disk layout. The MPHF-relevant files are:</p>
<div class="highlight"><pre><span></span><code>layer_i/
unitigs.bin — packed 2-bit nucleotide sequences (kmer evidence)
unitigs.bin — packed 2-bit nucleotide sequences (kmer evidence source)
unitigs.bin.idx — random-access block index (block_bits controls granularity)
mphf.bin — ptr_hash phase-2 MPHF
evidence.bin — n × u32: (chunk_id: 25 bits | rank: 7 bits) per slot
evidence.bin — n × (chunk_id: 25 bits | rank: 7 bits) per slot [exact mode]
fingerprint.bin — n × b-bit fingerprints per slot [approx mode]
[no layer_meta.json — mode stored once in partition-level meta.json]
</code></pre></div>
<p>Layers are <strong>disjoint</strong>: a canonical kmer belongs to exactly one layer. Layer 0 is built from dataset A. Adding dataset B:</p>
<ol>
@@ -1330,17 +1738,43 @@
<li>Collect kmers of B not present in any layer → set <code>B \ A</code>.</li>
<li>Build layer 1 from <code>B \ A</code> (dereplicate → count → De Bruijn → unitigs → <code>MphfLayer::build</code>).</li>
</ol>
<h3 id="evidence-modes">Evidence modes</h3>
<p>Three evidence modes are supported via <code>IndexMode</code>, stored once in <code>PartitionMeta</code> at partition root. There is no <code>layer_meta.json</code>.</p>
<p><strong>Exact</strong> (<code>IndexMode::Exact</code>): <code>evidence.bin</code> stores one <code>(chunk_id, rank)</code> pair per MPHF slot. Verification reconstructs the kmer and compares to the query. Zero false positives. <code>.idx</code> required at query time.</p>
<p><strong>Approx</strong> (<code>IndexMode::Approx { b, z }</code>): <code>fingerprint.bin</code> stores a b-bit hash per slot. False-positive rate 1/2^b per query; Findere z-parameter reduces window FP to ≈ 1/2^(b·z). No <code>.idx</code> written or needed.</p>
<p><strong>Hybrid</strong> (<code>IndexMode::Hybrid { b, z }</code>): both <code>fingerprint.bin</code> and <code>evidence.bin</code> + <code>.idx</code>. <code>find()</code> uses the fingerprint (O(1)); <code>find_strict()</code> uses exact evidence (O(1)).</p>
<h3 id="build-functions">Build functions</h3>
<div class="highlight"><pre><span></span><code>MphfLayer::build(dir, block_bits, mode: &amp;IndexMode, fill_slot)
Pass 1: CanonicalKmerIter + par_bridge() → build mphf.bin (no .idx used)
Pass 2: sequential iter → fill evidence files + call fill_slot
.idx written last for Exact/Hybrid (query-time only)
MphfLayer::build_exact_evidence(dir, block_bits)
Post-hoc: builds evidence.bin + .idx from existing mphf.bin + unitigs.bin
Uses open_sequential(); no .idx required on entry
MphfLayer::build_approx_evidence(dir, b, z)
Post-hoc: builds fingerprint.bin from existing mphf.bin + unitigs.bin
Uses open_sequential(); never writes .idx
</code></pre></div>
<p>There is no <code>build_evidence</code> dispatch wrapper. Callers choose the appropriate post-hoc build directly.</p>
<p>In <code>obikindex::partition</code>, <code>build_index_layer</code> receives <code>block_bits: u8</code> from <code>IndexConfig::block_bits</code> and forwards it directly to <code>Layer::build</code> and <code>Layer::build_approx_evidence</code>.</p>
<h3 id="membership-verification">Membership verification</h3>
<p>ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry: decode the kmer from <code>(chunk_id, rank)</code> and compare to the query. A mismatch means the kmer is absent from this layer; probe the next layer.</p>
<p>ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry:</p>
<ul>
<li><strong>Exact</strong>: decode <code>(chunk_id, rank)</code> from <code>evidence.bin</code>; reconstruct the kmer via <code>unitigs.verify_canonical_kmer</code>; compare to query.</li>
<li><strong>Approx</strong>: compare <code>kmer.seq_hash()</code> to the b-bit fingerprint stored at the slot.</li>
</ul>
<p>A mismatch in either mode means the kmer is absent from this layer; probe the next layer.</p>
<h3 id="query-algorithm">Query algorithm</h3>
<div class="highlight"><pre><span></span><code>fn query(kmer) → Option&lt;(layer_index, slot)&gt;:
for (i, layer) in layers.iter().enumerate():
slot = layer.mphf.index(kmer)
if layer.evidence.decode(slot) matches kmer:
if layer.evidence.matches(slot, kmer): // exact or approx dispatch
return Some((i, slot))
return None
</code></pre></div>
<p>Expected probe depth: 1 for kmers in layer 0. Each probe is a ptr_hash lookup (~10 ns) plus one evidence decode.</p>
<p><code>MphfLayer::find</code> dispatches on <code>LayerEvidence</code> at O(1) — no panicking <code>find_exact</code>/<code>find_approx</code> methods. <code>find_strict</code> always performs an exact check: O(1) for Exact/Hybrid, O(n) sequential scan for Approx. Expected probe depth: 1 for kmers in layer 0. Each probe is a ptr_hash lookup (~10 ns) plus one evidence check.</p>
<h3 id="merging-layers">Merging layers</h3>
<p>Two layer chains can be merged by re-indexing their union through the full pipeline. This is expensive (full rebuild) but produces an optimal single-layer index. Merge is a maintenance operation, not a query-path requirement.</p>
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -387,10 +442,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +463,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -662,6 +717,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#make_pipe-dsl" class="md-nav__link">
<span class="md-ellipsis">
make_pipe! DSL
</span>
</a>
</li>
</ul>
@@ -802,14 +868,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -879,6 +973,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -901,10 +1191,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -922,8 +1212,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -989,6 +1279,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
@@ -1087,6 +1461,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#make_pipe-dsl" class="md-nav__link">
<span class="md-ellipsis">
make_pipe! DSL
</span>
</a>
</li>
</ul>
@@ -1145,7 +1530,7 @@
<h1 id="obipipeline-parallel-pipeline-library">obipipeline — parallel pipeline library</h1>
<p><code>obipipeline</code> is a generic, multi-threaded data pipeline crate. It connects a <strong>source</strong>, a chain of <strong>transforms</strong>, and a <strong>sink</strong> via crossbeam channels, running each stage with a shared worker pool and a biased scheduler.</p>
<p><code>obipipeline</code> is a generic, multi-threaded data pipeline crate. It connects a <strong>source</strong>, a chain of <strong>stages</strong>, and a <strong>sink</strong> via crossbeam channels, running each stage with a shared worker pool and a biased scheduler.</p>
<h2 id="core-types">Core types</h2>
<table>
<thead>
@@ -1158,22 +1543,33 @@
<tbody>
<tr>
<td><code>SourceFn&lt;D&gt;</code></td>
<td><code>Box&lt;dyn FnMut() -&gt; Result&lt;D, PipelineError&gt; + Send+Sync&gt;</code></td>
<td><code>Box&lt;dyn FnMut() -&gt; Result&lt;D, PipelineError&gt; + Send&gt;</code></td>
<td>Called repeatedly; <code>FnMut</code> because it holds iterator state</td>
</tr>
<tr>
<td><code>SharedFn&lt;D&gt;</code></td>
<td><code>Arc&lt;dyn Fn(D) -&gt; Result&lt;D, PipelineError&gt; + Send+Sync&gt;</code></td>
<td>Shared across workers via <code>Arc::clone</code> (no copy of the closure)</td>
<td><code>Arc&lt;dyn Fn(D) -&gt; Result&lt;D, PipelineError&gt; + Send + Sync&gt;</code></td>
<td>1→1 transform shared across workers via <code>Arc::clone</code></td>
</tr>
<tr>
<td><code>SharedFlatFn&lt;D&gt;</code></td>
<td><code>Arc&lt;dyn Fn(D, &amp;Sender&lt;Result&lt;D, _&gt;&gt;, &amp;Sender&lt;isize&gt;) + Send + Sync&gt;</code></td>
<td>1→N transform; pushes items into channel, sends delta</td>
</tr>
<tr>
<td><code>SinkFn&lt;D&gt;</code></td>
<td><code>Box&lt;dyn Fn(D) -&gt; Result&lt;(), PipelineError&gt; + Send+Sync&gt;</code></td>
<td><code>Box&lt;dyn Fn(D) -&gt; Result&lt;(), PipelineError&gt; + Send&gt;</code></td>
<td>Final consumer; returns <code>Result</code> so errors propagate back</td>
</tr>
</tbody>
</table>
<p><code>Pipeline&lt;D&gt;</code> holds one <code>SourceFn</code>, a <code>Vec&lt;SharedFn&gt;</code>, and one <code>SinkFn</code>.<br />
<p>Stages come in two variants:</p>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">enum</span><span class="w"> </span><span class="nc">Stage</span><span class="o">&lt;</span><span class="n">D</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Transform</span><span class="p">(</span><span class="n">SharedFn</span><span class="o">&lt;</span><span class="n">D</span><span class="o">&gt;</span><span class="p">),</span><span class="w"> </span><span class="c1">// 1→1</span>
<span class="w"> </span><span class="n">Flat</span><span class="p">(</span><span class="n">SharedFlatFn</span><span class="o">&lt;</span><span class="n">D</span><span class="o">&gt;</span><span class="p">),</span><span class="w"> </span><span class="c1">// 1→N</span>
<span class="p">}</span>
</code></pre></div>
<p><code>Pipeline&lt;D&gt;</code> holds one <code>SourceFn</code>, a <code>Vec&lt;Stage&gt;</code>, and one <code>SinkFn</code>.<br />
<code>WorkerPool&lt;D&gt;</code> wraps a <code>Pipeline</code> with <code>n_workers</code> and channel <code>capacity</code>.</p>
<h2 id="workerpool">WorkerPool</h2>
<div class="highlight"><pre><span></span><code><span class="n">WorkerPool</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="n">pipeline</span><span class="p">:</span><span class="w"> </span><span class="nc">Pipeline</span><span class="o">&lt;</span><span class="n">D</span><span class="o">&gt;</span><span class="p">,</span><span class="w"> </span><span class="n">n_workers</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">capacity</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">Self</span>
@@ -1193,7 +1589,7 @@
</tr>
<tr>
<td><code>capacity</code></td>
<td>Bound on every crossbeam channel in the pipeline (source output, inter-stage channels, worker input, sink input, sink error). Controls memory and back-pressure: a full channel blocks the sender until a slot frees.</td>
<td>Bound on every crossbeam channel in the pipeline. Controls memory and back-pressure: a full channel blocks the sender until a slot frees.</td>
</tr>
</tbody>
</table>
@@ -1208,7 +1604,7 @@
</code></pre></div>
<p>Each variant carries the concrete type for one stage's output. The macros pattern-match on this enum to route values between stages.</p>
<h2 id="macros">Macros</h2>
<p>Six low-level macros build individual stages; one high-level macro (<code>make_pipeline!</code>) composes them.</p>
<p>Eight low-level macros build individual stages; one high-level macro (<code>make_pipeline!</code>) composes them.</p>
<h3 id="low-level">Low-level</h3>
<div class="highlight"><pre><span></span><code><span class="n">make_source</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">iterator</span><span class="p">,</span><span class="w"> </span><span class="n">OutputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// iterator yields T</span>
<span class="n">make_source_fallible</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">iterator</span><span class="p">,</span><span class="w"> </span><span class="n">OutputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// iterator yields Result&lt;T, E&gt;</span>
@@ -1216,6 +1612,9 @@
<span class="n">make_transform</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">func</span><span class="p">,</span><span class="w"> </span><span class="n">InputVariant</span><span class="p">,</span><span class="w"> </span><span class="n">OutputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// func: T -&gt; U</span>
<span class="n">make_transform_fallible</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">func</span><span class="p">,</span><span class="w"> </span><span class="n">InputVariant</span><span class="p">,</span><span class="w"> </span><span class="n">OutputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// func: T -&gt; Result&lt;U, E&gt;</span>
<span class="n">make_flat_transform</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">func</span><span class="p">,</span><span class="w"> </span><span class="n">InputVariant</span><span class="p">,</span><span class="w"> </span><span class="n">OutputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// func: T -&gt; impl IntoIterator&lt;Item=U&gt;</span>
<span class="n">make_flat_transform_fallible</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">func</span><span class="p">,</span><span class="w"> </span><span class="n">InputVariant</span><span class="p">,</span><span class="w"> </span><span class="n">OutputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// func: T -&gt; Result&lt;impl IntoIterator&lt;Item=U&gt;, E&gt;</span>
<span class="n">make_sink</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">func</span><span class="p">,</span><span class="w"> </span><span class="n">InputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// func: T -&gt; ()</span>
<span class="n">make_sink_fallible</span><span class="o">!</span><span class="p">(</span><span class="n">Enum</span><span class="p">,</span><span class="w"> </span><span class="n">func</span><span class="p">,</span><span class="w"> </span><span class="n">InputVariant</span><span class="p">)</span><span class="w"> </span><span class="c1">// func: T -&gt; Result&lt;(), E&gt;</span>
</code></pre></div>
@@ -1224,17 +1623,31 @@
<div class="highlight"><pre><span></span><code>make_pipeline! {
DataEnum,
source iterator =&gt; OutputVariant, // or source? for fallible
| func: In =&gt; Out, // non-fallible transform
|? func: In =&gt; Out, // fallible transform
| func: In =&gt; Out, // 1→1 non-fallible transform
|? func: In =&gt; Out, // 1→1 fallible transform
|| func: In =&gt; Out, // 1→N non-fallible flat transform
||? func: In =&gt; Out, // 1→N fallible flat transform
sink func @ InputVariant, // or sink? for fallible
}
</code></pre></div>
<p><code>?</code> marks fallibility on source, individual transforms, or sink independently.<br />
Implemented as a <strong>TT muncher</strong>: the internal rule <code>@build</code> recurses over transform tokens one at a time, accumulating them into a <code>vec![]</code>, then terminates on <code>sink</code>/<code>sink?</code>.</p>
<h3 id="make_pipe-dsl">make_pipe! DSL</h3>
<p><code>make_pipe!</code> builds a sourceless/sinkless <code>Pipe&lt;D, In, Out&gt;</code> — a reusable, composable stage sequence:</p>
<div class="highlight"><pre><span></span><code>make_pipe! {
DataEnum : InType =&gt; OutType,
| func: InVariant =&gt; OutVariant,
|? func: InVariant =&gt; OutVariant,
|| func: InVariant =&gt; OutVariant,
||? func: InVariant =&gt; OutVariant,
}
</code></pre></div>
<p>Two pipes compose with <code>.then(other)</code>. Apply to an iterator with <code>.apply(iter, n_workers, capacity)</code> to get a <code>PipeIter&lt;Out&gt;</code> — an iterator over the pipeline output, backed by a background <code>WorkerPool</code>. The scatter step in <code>obikmer</code> uses <code>make_pipe!</code> and <code>.apply()</code> rather than the full <code>make_pipeline!</code> / <code>WorkerPool</code> pattern.</p>
<h2 id="scheduler-architecture">Scheduler architecture</h2>
<div class="highlight"><pre><span></span><code>Source thread ──► [source_rx] ──► Scheduler ──► [worker_tx] ──► Workers (×N)
▲ │
[stage_rxs] ────────┘◄──────────────────────────────┘
[flat_delta_rx] ──► Scheduler (in_flight adjustment)
│
[sink_err_rx] ← errors from sink (highest priority)
│
@@ -1242,20 +1655,20 @@ Implemented as a <strong>TT muncher</strong>: the internal rule <code>@build</co
</code></pre></div>
<p>The scheduler is a single thread running a biased <code>Select</code> over all input channels. Priority order (highest first):</p>
<div class="highlight"><pre><span></span><code>index 0 sink_err_rx abort on sink error
index 1 stage_rxs[N-1] drain last stage first
...
index N stage_rxs[0]
index N+1 source_rx pull new data last
index 1 flat_delta_rx adjust in_flight before dispatching
index 2..=n+1 stage_rxs[n-1..0] drain last stage first
index n+2 source_rx pull new data last
</code></pre></div>
<p>This back-pressure-friendly ordering ensures downstream stages are drained before new items enter the pipeline.</p>
<p><strong>Workers</strong> are generic: each receives <code>(data, SharedFn, result_tx)</code> and calls <code>f(data)</code>, sending the result to the provided channel. The scheduler decides which transform to apply and where to route the result.</p>
<p><strong>Termination</strong> uses an <code>in_flight</code> counter:</p>
<p><strong>Workers</strong> are generic: each receives a <code>WorkerTask</code> — either <code>Transform(data, stage_idx)</code> or <code>Flat(data, stage_idx)</code>. For <code>Transform</code>, the worker calls <code>f(data)</code> and sends the result to <code>stage_txs[stage_idx]</code>. For <code>Flat</code>, the worker calls <code>f(data, &amp;push_tx, &amp;delta_tx)</code>: the closure pushes N items into <code>push_tx</code> then sends <code>N-1</code> to <code>delta_tx</code>. The scheduler uses the delta to adjust <code>in_flight</code> without knowing N in advance.</p>
<p><strong>Termination</strong> uses an <code>in_flight: isize</code> counter and a <code>flat_workers_active: usize</code> counter:</p>
<ul>
<li>incremented when an item is dispatched from source to workers</li>
<li>decremented when the item exits the last stage</li>
<li>the loop exits only when <code>source_done &amp;&amp; in_flight == 0</code></li>
<li><code>in_flight</code> incremented when an item is dispatched from source to workers</li>
<li><code>in_flight</code> decremented when the item exits the last stage to the sink</li>
<li><code>flat_workers_active</code> incremented when a <code>Flat</code> task is dispatched, decremented when the delta arrives</li>
<li>the loop exits only when <code>source_done &amp;&amp; in_flight == 0 &amp;&amp; flat_workers_active == 0</code></li>
</ul>
<p>This guarantees all in-flight items complete before <code>join()</code>.</p>
<p>This guarantees all in-flight items complete (including all N outputs of a flat stage) before <code>join()</code>.</p>
<h2 id="error-handling">Error handling</h2>
<p><code>PipelineError</code> has four variants:</p>
<table>
@@ -1279,7 +1692,7 @@ index N+1 source_rx pull new data last
<td>Internal routing error</td>
</tr>
<tr>
<td><code>StepError(Box&lt;dyn Error&gt;)</code></td>
<td><code>StepError(Box&lt;dyn Error + Send + Sync&gt;)</code></td>
<td>Error from user code (wrapped by <code>make_*_fallible!</code>)</td>
</tr>
</tbody>
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -12,7 +12,7 @@
<link rel="prev" href="../persistent_compact_int_vec/">
<link rel="next" href="../../architecture/sequences/invariant/">
<link rel="next" href="../merge/">
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -387,10 +442,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +463,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -650,14 +705,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -1002,6 +1085,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -1024,10 +1303,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -1045,8 +1324,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -1112,6 +1391,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
File diff suppressed because it is too large Load Diff
@@ -9,7 +9,7 @@
<link rel="prev" href="../obilayeredmap/">
<link rel="prev" href="../layer_tier/">
<link rel="next" href="../persistent_bit_vec/">
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -387,10 +442,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +463,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -650,14 +705,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -985,6 +1068,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -1007,10 +1286,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -1028,8 +1307,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -1095,6 +1374,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
File diff suppressed because it is too large Load Diff
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -387,10 +442,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +463,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -774,14 +829,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -851,6 +934,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -873,10 +1152,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -894,8 +1173,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -961,6 +1240,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
@@ -1104,7 +1467,9 @@
<li><strong>error valley</strong> → suggests min_count (typically the local minimum between the error peak and the coverage peak)</li>
</ul>
<h2 id="phase-1-scatter">Phase 1 — Scatter</h2>
<p>Single streaming pass over raw input files (FASTA/FASTQ, gzip). FASTQ quality scores are ignored. For each read:</p>
<p>Single streaming pass over raw input files (FASTA/FASTQ, gzip). FASTQ quality scores are ignored.</p>
<p>Input files are read via <code>open_nuc_stream</code>, which opens and decompresses the file, auto-detects the format (FASTA / FASTQ / GenBank), and yields a sequence of <code>NucPage</code> buffers. Each <code>NucPage</code> is a flat 64 KB buffer of normalised bytes (<code>ACGT</code> + <code>\x00</code> separators), carrying a k−1 byte overlap from the preceding page so that no k-mer is lost at page boundaries. Per-record identity (sequence id, raw bytes) is not preserved; this is intentional — the scatter phase only needs normalised bases to produce superkmers.</p>
<p>For each read fragment within a page:</p>
<ol>
<li><strong>Ambiguous base filter</strong>: cut at any non-ACGT base; discard fragments shorter than k.</li>
<li><strong>Entropy filter</strong>: scan each fragment with a sliding window of size k. When the kmer <span class="arithmatex">\(K_i = S[i \mathinner{..} i+k-1]\)</span> ended by nucleotide <span class="arithmatex">\(S[j]\)</span> (with <span class="arithmatex">\(j = i+k-1\)</span>) has entropy below threshold <span class="arithmatex">\(\theta\)</span>, emit the current segment and start a new one (see algorithm below). <span class="arithmatex">\(K_i\)</span> belongs to neither segment, and no valid kmer is lost.</li>
@@ -1154,8 +1519,13 @@ B ≈ 100 is tunable; RAM needed ≈ partition_size / B.</p>
for each kmer in sequence:
kmer_counts[canonical(kmer)] += COUNT
</code></pre></div>
<p>Implemented as an external sort or a temporary HashMap, depending on partition size. At the end of this phase, each distinct canonical kmer has its exact total count.</p>
<p>Abundance filter applied here: kmers with <code>total_count &lt; q</code> are discarded. <code>q</code> is a collection parameter (0 = keep all, including singletons for ≤1x data).</p>
<p>Implemented as a three-step pipeline in <code>count_partition()</code>:</p>
<ol>
<li><strong>External sort</strong> (<code>kmer_sort::sort_unique_kmers</code>): read dereplicated superkmers, extract canonical kmer raw <code>u64</code> values, sort in RAM-bounded chunks (adaptive: 40% of available RAM ÷ n_threads, min 1 M kmers/chunk), k-way merge with inline dedup → <code>sorted_unique.bin</code>. f0 is now known exactly.</li>
<li><strong>Provisional MPHF</strong> (ptr_hash): built from <code>sorted_unique.bin</code> via <code>new_from_par_iter(f0, ...)</code>. Stored to <code>mphf1.bin</code>; <code>sorted_unique.bin</code> deleted immediately.</li>
<li><strong>Accumulation pass</strong>: re-read dereplicated superkmers; for each kmer, <code>slot = mphf.index(kmer.raw())</code>, increment <code>counts1[slot]</code> by the superkmer COUNT. Stored in a <code>PersistentCompactIntVec</code> (<code>counts1.bin</code>).</li>
</ol>
<p>At the end of this phase, each distinct canonical kmer has its exact total count, and the frequency spectrum (<code>spectrums/{label}.json</code>) is written to the index root.</p>
<p>No pre-filter on super-kmer COUNT is possible at phase 2: a super-kmer with COUNT=1 may contain only high-abundance kmers, each present in many other super-kmers across the partition.</p>
<h2 id="phase-4-super-kmer-compaction">Phase 4 — Super-kmer compaction</h2>
<p>The valid kmer set from phase 3 is used as a mask to rewrite the super-kmer files:</p>
@@ -1188,14 +1558,52 @@ branching / dead-end → unitig start or end
<p>Output: <code>unitigs.bin</code> — the permanent evidence structure for the partition. Each kmer in the partition appears at exactly one (unitig_id, offset) location.</p>
<p><strong>Scope of local unitigs:</strong> these are unitigs of the partition's local de Bruijn graph, not global unitigs. A kmer whose k-1 successor or predecessor falls in another partition appears as a dead end locally and terminates the unitig. This does not affect correctness of verification but means partition-local unitigs cannot be directly reused for global assembly.</p>
<h2 id="phase-6-mphf-construction-and-index-finalisation">Phase 6 — MPHF construction and index finalisation</h2>
<p>Built once on the definitive kmer set (all kmers in all unitigs of the partition). See <a href="../obilayeredmap/">obilayeredmap</a> and <a href="../mphf/">MPHF selection</a> for the current implementation.</p>
<div class="highlight"><pre><span></span><code>kmers from unitigs → MPHF → mphf.bin
→ evidence.bin : n × u32, each = (chunk_id: 25 bits | rank: 7 bits)
→ payload : counts/ (mode 2) or presence/ (mode 3)
<p><code>build_index_layer</code> is called per partition (in parallel via <code>build_layers</code>) with the following parameters sourced from <code>IndexConfig</code>:</p>
<ul>
<li><code>block_bits</code> — from <code>IndexConfig::block_bits</code>; controls the <code>.idx</code> block size (2^block_bits unitig chunks per block) for exact evidence</li>
<li><code>evidence</code> — <code>EvidenceKind::Exact</code> or <code>EvidenceKind::Approx { b, z }</code>; propagated unchanged from <code>IndexConfig::evidence</code></li>
<li><code>min_ab</code> / <code>max_ab</code> — abundance bounds applied before graph construction</li>
<li><code>with_counts</code> — whether to store kmer counts alongside set membership</li>
</ul>
<p><strong>Abundance filtering:</strong> when <code>min_ab &gt; 1</code> or <code>max_ab.is_some()</code>, the provisional <code>mphf1.bin</code> and <code>counts1.bin</code> produced in phase 3 are memory-mapped. Each canonical kmer is accepted only if its count in <code>counts1</code> satisfies the bounds. If either file is absent, filtering is skipped (all kmers accepted).</p>
<div class="highlight"><pre><span></span><code>for each kmer in dereplicated super-kmer:
ab = counts1[mphf1.index(kmer.raw())]
if ab &lt; min_ab || ab &gt; max_ab: skip
graph.push(kmer)
</code></pre></div>
<p>The MPHF is built in two passes over <code>unitigs.bin</code>: parallel pass for <code>mphf.bin</code>, sequential pass for <code>evidence.bin</code> and payload. The exact kmer count is available from the unitig index (<code>unitigs.bin.idx</code>) before the passes begin.</p>
<p><strong>Exact verification via unitig evidence:</strong></p>
<p><code>unitigs.bin</code> serves as the evidence structure. The MPHF maps every input to <code>[0, N)</code> including absent kmers — the unitig read-back (via <code>evidence.bin</code>) is the only correct membership test.</p>
<p><strong>Graph build and unitig write:</strong></p>
<p>The surviving kmers are fed into <code>GraphDeBruijn</code>, which computes degrees and yields unitigs. Unitigs are written to <code>layer_0/unitigs.bin</code> via a <code>UnitigFileWriter</code>.</p>
<p><strong>MPHF and evidence build:</strong></p>
<p><code>Layer::build</code> (membership-only) or <code>Layer::&lt;PersistentCompactIntMatrix&gt;::build</code> (with counts) is called next. Internally, <code>MphfLayer::build</code> performs two passes:</p>
<ol>
<li><strong>Pass 1 (parallel):</strong> build <code>unitigs.bin.idx</code> (block size = 2^<code>block_bits</code>) then construct the MPHF from all canonical kmers in <code>unitigs.bin</code>; store to <code>mphf.bin</code>.</li>
<li><strong>Pass 2 (sequential):</strong> for each kmer in <code>unitigs.bin</code>, compute its slot and write <code>evidence.bin</code> (<code>chunk_id: 25 bits | rank: 7 bits</code> packed into a <code>u32</code>); also invoke the payload callback (<code>fill_slot</code>) to populate <code>counts/</code> if <code>with_counts</code>.</li>
</ol>
<p>After <code>Layer::build</code> completes, <code>layer_meta.json</code> records <code>EvidenceKind::Exact</code>.</p>
<p><strong>Approximate evidence override:</strong></p>
<p>If <code>evidence</code> is <code>EvidenceKind::Approx { b, z }</code>, <code>build_approx_evidence</code> is called immediately after <code>Layer::build</code>. It overwrites the exact evidence bundle with <code>fingerprint.bin</code> (b-bit hash per slot) and rewrites <code>layer_meta.json</code> with <code>EvidenceKind::Approx { b, z }</code>. No <code>.idx</code> file is needed at query time in this mode.</p>
<div class="highlight"><pre><span></span><code>// Exact path → evidence.bin + unitigs.bin.idx + layer_meta.json(Exact)
// Approx path → fingerprint.bin + layer_meta.json(Approx{b,z})
// (evidence.bin left on disk but not used)
</code></pre></div>
<p><strong>Partition metadata:</strong></p>
<p>After all layer files are written, <code>PartitionMeta { n_layers: 1 }</code> is serialised to <code>index/meta.json</code> inside the partition directory. This file is required by <code>LayeredMap::open</code> for subsequent merge operations.</p>
<p><strong>File layout per partition after phase 6:</strong></p>
<div class="highlight"><pre><span></span><code>part_XXXXX/
index/
meta.json ← PartitionMeta { n_layers: 1 }
layer_0/
unitigs.bin ← permanent evidence (all modes)
unitigs.bin.idx ← block index (exact mode only)
mphf.bin ← MPHF
evidence.bin ← exact evidence (exact mode)
fingerprint.bin ← b-bit fingerprints (approx mode)
layer_meta.json ← EvidenceKind tag
counts/ ← PersistentCompactIntMatrix (with_counts only)
</code></pre></div>
<p><strong>Cleanup:</strong> unless <code>--keep-intermediate</code> is set, <code>remove_build_artifacts</code> deletes <code>dereplicated.skmer.zst</code>, <code>mphf1.bin</code>, and <code>counts1.bin</code> after all partitions are indexed.</p>
<p>See <a href="../layer_tier/">obikindex::layer</a> and <a href="../mphf/">MPHF selection</a> for data structure details.</p>
<p><strong>Query path (exact evidence):</strong></p>
<div class="highlight"><pre><span></span><code>query kmer q
→ canonical_minimizer(q) → hash → PART → part_XXXXX/
→ MPHF(q) → slot s
@@ -1204,7 +1612,13 @@ branching / dead-end → unitig start or end
→ match : return payload[s] ← exact hit
→ no match: kmer absent ← MPHF collision on absent kmer
</code></pre></div>
<p><code>superkmers.bin.gz</code> is no longer needed at this point and can be deleted.</p>
<p><strong>Query path (approximate evidence):</strong></p>
<div class="highlight"><pre><span></span><code>query kmer q
→ MPHF(q) → slot s
→ fingerprint[s] matches seq_hash(q)?
→ yes : probable hit (FP rate = 1/2^b per kmer, 1/2^(b·z) per z-window)
→ no : kmer absent
</code></pre></div>
<div class="footnote">
<hr />
<ol>
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -9,7 +9,7 @@
<link rel="prev" href="../../theory/indexing/">
<link rel="prev" href="../../theory/evolutionary_distances/">
<link rel="next" href="../kmer/">
@@ -182,6 +182,33 @@
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
@@ -191,10 +218,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -212,8 +239,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -363,6 +390,34 @@
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -387,10 +442,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
@@ -408,8 +463,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
@@ -752,14 +807,42 @@
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../layer_tier/" class="md-nav__link">
<span class="md-ellipsis">
obikindex layer tier
@@ -829,6 +912,202 @@
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../benchmark_query_testing/" class="md-nav__link">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../partition_layer_cache/" class="md-nav__link">
<span class="md-ellipsis">
Partition and layer caching (discussion)
</span>
</a>
</li>
</ul>
</nav>
@@ -851,10 +1130,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
@@ -872,8 +1151,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
@@ -939,6 +1218,90 @@
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
@@ -1046,61 +1409,49 @@
<h1 id="superkmer-implementation">SuperKmer — implementation</h1>
<h2 id="memory-layout">Memory layout</h2>
<p>A super-kmer is stored as a <strong>32-bit header</strong> followed by a <strong>byte-aligned nucleotide sequence</strong> (2 bits/base, nucleotide 0 at the MSB of the first byte):</p>
<p><code>SuperKmer</code> holds two separate fields:</p>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">SuperKmer</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="n">count</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span>
<span class="w"> </span><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="n">inner</span><span class="p">:</span><span class="w"> </span><span class="nc">PackedSeq</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div>
<p><code>PackedSeq</code> stores a 2-bit packed DNA sequence as a heap-allocated <code>Box&lt;[u8]&gt;</code> plus a <code>tail: u8</code> field:</p>
<table>
<thead>
<tr>
<th>Field</th>
<th>Bits</th>
<th>Type</th>
<th>Role</th>
</tr>
</thead>
<tbody>
<tr>
<td>COUNT</td>
<td>24</td>
<td>Occurrence count (≤ 16 M)</td>
<td><code>tail</code></td>
<td><code>u8</code></td>
<td>Number of valid nucleotides in the last byte: 0 encodes 4, 1–3 are identity</td>
</tr>
<tr>
<td>NKMERS</td>
<td>8</td>
<td>Number of kmers (= seq_length − k + 1, range 1–255)</td>
<td><code>seq</code></td>
<td><code>Box&lt;[u8]&gt;</code></td>
<td>2-bit packed bytes, nucleotide 0 at bits 7–6 of <code>seq[0]</code></td>
</tr>
</tbody>
</table>
<p>Bit layout (MSB to LSB): <code>[31:8] COUNT [7:0] NKMERS</code></p>
<p>NKMERS is stored as a raw <code>u8</code> in <strong>kmer units</strong>, not nucleotides. The nucleotide length is recovered as <code>NKMERS + k − 1</code>. This avoids the awkward wrapping convention (<code>0 = 256</code>) that would be needed if nucleotide length were stored directly, and gains k−1 = 30 units of headroom:</p>
<table>
<thead>
<tr>
<th>unit</th>
<th>u8 covers</th>
<th>max nucleotides</th>
</tr>
</thead>
<tbody>
<tr>
<td>nucleotides</td>
<td>255 nt</td>
<td>225 kmers</td>
</tr>
<tr>
<td><strong>kmers</strong></td>
<td><strong>255 kmers</strong></td>
<td><strong>285 nt</strong></td>
</tr>
</tbody>
</table>
<p>The public accessors:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">n_kmers</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">usize</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mh">0xFF</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="kt">usize</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">seql</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">usize</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">n_kmers</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">K</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">count</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">u32</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">increment</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="mi">8</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">add</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="mi">8</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">set_count</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="mi">0</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mh">0xFF</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">n</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="mi">8</span><span class="p">);</span><span class="w"> </span><span class="p">}</span>
<p>Nucleotide length is recovered without storing it explicitly:</p>
<div class="highlight"><pre><span></span><code>seql = (seq.len() - 1) * 4 + tail_count(tail)
</code></pre></div>
<p>There is no packed header word — <code>count</code> and the sequence live in separate fields.</p>
<p>The on-disk binary format (produced by <code>write_to_binary</code>) is:</p>
<div class="highlight"><pre><span></span><code>[varint(count)] [u8: seql − k] [packed bytes…]
</code></pre></div>
<p><code>seql − k</code> fits in a <code>u8</code> when <code>n_kmers = seql − k + 1 ≤ MAX_KMERS_PER_CHUNK (= 256)</code>. If a super-kmer exceeds 256 kmers, <code>write_to_binary</code> splits it into overlapping chunks (k−1 nucleotide overlap, same count per chunk), each a self-contained record readable by <code>read_from_binary</code>.</p>
<p>The public accessors operate on the struct fields directly:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">seql</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">usize</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">inner</span><span class="p">.</span><span class="n">seql</span><span class="p">()</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">count</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">u32</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">count</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">increment</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">count</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">add</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">count</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">n</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
<span class="k">fn</span><span class="w"> </span><span class="nf">set_count</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">count</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">n</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>In practice, observed super-kmer lengths on metagenomic data (k=31) are below 55 nucleotides (≤ 25 kmers) — far from the 255-kmer cap. If a super-kmer ever exceeds 255 kmers, it is split with a k−1 nucleotide overlap, preserving all kmers without duplication (identical mechanism to partition-boundary splits).</p>
<p>The sequence is always stored in canonical form (lexicographic minimum of forward and reverse complement), with nucleotide 0 at the MSB of the first byte. The byte array can be hashed directly without any adjustment.</p>
<h2 id="ascii-encoding-and-decoding">ASCII encoding and decoding</h2>
<p>Two lookup tables handle ASCII ↔ 2-bit conversion:</p>
<ul>
@@ -1125,7 +1476,7 @@
</code></pre></div>
<p><code>REVCOMP4</code> is 256 bytes (fits in L1 cache), computed at compile time. No endianness dependency — all operations are pure arithmetic on byte values.</p>
<p><strong>Step 2 — realignment.</strong> After step 1, <code>padding = n × 8 − seql × 2</code> spurious bits (complements of the original padding A's) appear at the start of the array. They are flushed left using <code>BitSlice&lt;u8, Msb0&gt;::rotate_left(padding)</code> from the <code>bitvec</code> crate, which is SIMD-accelerated. The trailing <code>padding</code> bits are then zeroed:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">seql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">n_kmers</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">k</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">seql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">seql</span><span class="p">();</span>
<span class="n">shift</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">seql</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="c1">// number of padding bits</span>
<span class="n">bits</span><span class="p">.</span><span class="n">rotate_left</span><span class="p">(</span><span class="n">shift</span><span class="p">)</span>
<span class="n">bits</span><span class="p">[</span><span class="n">len</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">shift</span><span class="o">..</span><span class="p">].</span><span class="n">fill</span><span class="p">(</span><span class="kc">false</span><span class="p">)</span>
@@ -1143,7 +1494,7 @@
</code></pre></div>
</div>
<h2 id="minimizer-sliding-window">Minimizer sliding window</h2>
<p>Super-kmers are built by <code>SuperKmerIter</code> (crate <code>obiskbuilder</code>), which maintains the current minimizer with a <strong>monotonic deque</strong> over a sliding window of W = k − m + 1 m-mer positions.</p>
<p>Super-kmers are built by <code>SuperKmerIter</code> (crate <code>obiskbuilder</code>), which tracks the current minimizer with a <strong>monotonic deque</strong> (<code>Ring&lt;MmerItem, 32&gt;</code>) inside <code>RollingStat</code>, a rolling-window entropy and minimizer tracker.</p>
<p>Each deque entry stores:</p>
<table>
<thead>
@@ -1167,20 +1518,11 @@
<tr>
<td><code>hash</code></td>
<td>u64</td>
<td><span class="arithmatex">\(H(\text{canonical})\)</span> — ordering key for random minimizer selection</td>
<td><code>hash_kmer(canonical &lt;&lt; (64 − 2m))</code> — ordering key for random minimizer selection</td>
</tr>
</tbody>
</table>
<p>The hash <span class="arithmatex">\(H\)</span> is the seeded splitmix64 finalizer (see <a href="../../theory/minimizer/">Minimizer selection</a>):</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span><span class="w"> </span><span class="nf">hash_mmer</span><span class="p">(</span><span class="n">canonical</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="kt">u64</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">canonical</span><span class="w"> </span><span class="o">^</span><span class="w"> </span><span class="mh">0x9e3779b97f4a7c15</span><span class="p">;</span><span class="w"> </span><span class="c1">// seed: eliminates fixed point at 0</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">^</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">30</span><span class="p">);</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">wrapping_mul</span><span class="p">(</span><span class="mh">0xbf58476d1ce4e5b9</span><span class="p">);</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">^</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">27</span><span class="p">);</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">wrapping_mul</span><span class="p">(</span><span class="mh">0x94d049bb133111eb</span><span class="p">);</span>
<span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">^</span><span class="w"> </span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">31</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>The hash uses the seeded splitmix64 finalizer (<code>mix64(raw ^ 0x9e3779b97f4a7c15)</code>), the same function as <code>kmer::hash_kmer</code>.</p>
<p>On each new nucleotide, once the window is full, the deque is updated:</p>
<div class="admonition abstract">
<p class="admonition-title">Algorithm — minimizer deque update</p>
@@ -1196,17 +1538,21 @@
</code></pre></div>
</div>
<p>The front of the deque is always the current minimizer. Because the deque is maintained in strictly increasing hash order, each entry is popped at most once — O(1) amortized per nucleotide.</p>
<p>A super-kmer boundary is emitted when the minimizer changes: <code>deque.front.hash ≠ prev_hash</code>. The <code>canonical</code> field of the front entry is <strong>not</strong> used for boundary detection — that uses the hash alone. The canonical value is stored so that the partition key <span class="arithmatex">\(H(\text{canonical})\)</span> can be recomputed independently at routing time from the stored <code>minimizer_pos</code>, without inheriting the minimum-order-statistic bias (see <a href="../../theory/minimizer/#partition-key-independence">Minimizer selection — partition key independence</a>).</p>
<p>A super-kmer boundary is emitted when the minimizer changes: <code>current_minimizer != prev_minimizer</code>. <code>SuperKmerIter</code> also emits a boundary when:</p>
<ul>
<li>entropy of the current k-mer falls at or below the threshold θ (cursor retreated by k−1)</li>
<li>super-kmer length reaches 256 nucleotides (cursor retreated by k)</li>
</ul>
<h2 id="kmer-extraction">Kmer extraction</h2>
<p>A k-mer is extracted from a super-kmer with <code>SuperKmer::kmer(i, k)</code>, which returns a <code>Kmer</code> — a left-aligned <code>u64</code> newtype (see <a href="../kmer/">Kmer implementation</a>):</p>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">kmer</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">k</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Result</span><span class="o">&lt;</span><span class="n">Kmer</span><span class="p">,</span><span class="w"> </span><span class="n">KmerError</span><span class="o">&gt;</span>
<p>A k-mer is extracted from a super-kmer with <code>SuperKmer::kmer(i)</code>, which delegates to <code>PackedSeq::extract::&lt;KLen&gt;(i)</code> and returns a <code>Kmer</code> — a left-aligned <code>u64</code> newtype (see <a href="../kmer/">Kmer implementation</a>):</p>
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">kmer</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Result</span><span class="o">&lt;</span><span class="n">Kmer</span><span class="p">,</span><span class="w"> </span><span class="n">KmerError</span><span class="o">&gt;</span>
</code></pre></div>
<p>The bit slice <code>seq[i*2 .. (i+k)*2]</code> (Msb0 order) is loaded as a big-endian <code>u64</code> via <code>bitvec::load_be</code>, then left-shifted to produce the canonical left-aligned layout. One call — no loop, no allocation.</p>
<p>The bit slice <code>seq[i*2 .. (i+k)*2]</code> (Msb0 order) is loaded as a <code>u64</code> via <code>bitvec::load_be</code>, then left-shifted to produce the canonical left-aligned layout. One call — no loop, no allocation.</p>
<hr />
<div class="admonition abstract">
<p class="admonition-title">Algorithm — Super-kmer reverse complement</p>
<div class="highlight"><pre><span></span><code>procedure SuperKmerRevcomp(seq, SEQL):
seql ← NKMERS + k − 1 -- nucleotide length
seql ← nucleotide length
n ← ⌈seql / 4⌉ -- number of bytes
shift ← n × 8 − seql × 2 -- padding bits to flush
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+1625
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More