3 Commits

Author	SHA1	Message	Date
Eric Coissac	c30a22d356	Refactor build workflow and update version ... Update GitHub Actions workflow to use setup-go v5 and align with latest tooling practices. Update version to 4.4.15 in version.txt and pkg/obioptions/version.go. Add comprehensive documentation for the canonical super-kmer strategy, including: - Analysis of index v1 limitations - Experimental observations on super-kmer efficiency - Detailed pipeline for building v3 index - Explanation of minimizer-canonization - Description of unitig construction and frequency filtering - Storage format specifications for v3 - Aho-Corasick matching implementation This change introduces a major improvement in index compactness and performance through the use of canonical super-kmers, unitigs, and efficient storage formats.	2026-02-11 22:57:28 +01:00
Eric Coissac	a016ad5b8a	Refactor kmer index to disk-based partitioning with minimizer ... Refactor kmer index package to use disk-based partitioning with minimizer - Replace roaring64 bitmaps with disk-based kmer index - Implement partitioned kmer sets with delta-varint encoding - Add support for frequency filtering during construction - Introduce new builder pattern for index construction - Add streaming operations for set operations (union, intersect, etc.) - Add support for super-kmer encoding during construction - Update command line tool to use new index format - Remove dependency on roaring bitmap library This change introduces a new architecture for kmer indexing that is more memory efficient and scalable for large datasets.	2026-02-09 17:52:37 +01:00
Eric Coissac	500144051a	Add jj Makefile targets and k-mer encoding utilities ... Add new Makefile targets for jj operations (jjnew, jjpush, jjfetch) to streamline commit workflow. Introduce k-mer encoding utilities in pkg/obikmer: - EncodeKmers: converts DNA sequences to encoded k-mers - ReverseComplement: computes reverse complement of k-mers - NormalizeKmer: returns canonical form of k-mers - EncodeNormalizedKmers: encodes sequences with normalized k-mers Add comprehensive tests for k-mer encoding functions including edge cases, buffer reuse, and performance benchmarks. Document k-mer index design for large genomes, covering: - Use cases and objectives - Volume estimations - Distance metrics (Jaccard, Sørensen-Dice, Bray-Curtis) - Indexing options (Bloom filters, sorted sets, MPHF) - Optimization techniques (k-2-mer indexing) - MinHash for distance acceleration - Recommended architecture for presence/absence and counting queries	2026-02-04 14:27:10 +01:00