obitools4/autodoc/docmd/pkg/obiitercsv/csv.md

obiitercsv: CSV Record Iterator for Streaming and Batch Processing

This Go package provides a thread-safe, channel-based iterator (ICSVRecord) for streaming and processing CSV records in batches. It supports ordered batch handling, concurrent access via mutexes, and dynamic header management.

Core Types

• CSVHeader: A slice of strings representing column names.
• CSVRecord: A map from field name to value (map[string]interface{}).
• CSVRecordBatch: A batch of records with metadata: source, order, and the actual data slice.

Key Features

• Streaming via Channels: Records are consumed as CSVRecordBatch items through a channel, enabling asynchronous producers/consumers.
• Ordered Processing: Batches include an order field, used by SortBatches() to reconstruct sequential order even when received out-of-order.
• Thread Safety: Uses sync.RWMutex, atomic operations (batch_size), and abool.AtomicBool for flags like finished.
• Iterator Protocol: Implements standard methods:
  • Next() to advance,
  • Get() to retrieve current batch,
  • PushBack() for re-queuing the last record.
• Batch Management:
  • SetHeader() / AppendField(): dynamic header updates.
  • Split(): creates a new iterator sharing the same channel but with independent locking.
• Lifecycle Control:
  • Add() / Done(): track active goroutines (via sync.WaitGroup).
  • WaitAndClose() ensures all data is flushed before closing the channel.

Utility Methods

• NotEmpty(), IsNil(): Check batch validity.
• Consume(): Drains the iterator (e.g., for side-effect processing).
• SortBatches(): Reorders batches by order, buffering out-of-sequence ones.

Designed for bioinformatics pipelines (e.g., OBITools4), it enables scalable, memory-efficient CSV processing with strict ordering guarantees.