Eric Coissac
b6fcbc545f
Replaces `rayon` parallel iteration across index, rebuild, reindex, and select modules with a custom `PartitionRunner`. This introduces NUMA-aware task distribution with CPU pinning and round-robin scheduling, eliminating `Arc`, `Mutex`, and atomic synchronization primitives in favor of a flat, pre-spawned worker architecture. Error handling is simplified via `.map_err()` and the `?` operator, while progress bar updates are decoupled into dedicated callbacks.
2.4 KiB
Installation
Prerequisites
Rust toolchain
obikmer requires Rust 1.85 or later (edition 2024). Install or update via rustup:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update stable
C build environment (required for hwloc)
obikmer embeds hwloc (Hardware Locality) for NUMA-aware thread placement on multi-socket machines. hwloc is built from source at compile time via the vendored feature of the hwlocality crate. This requires a standard C build environment.
Linux (Debian/Ubuntu)
apt install build-essential automake libtool autoconf pkg-config
Linux (RHEL/Rocky/AlmaLinux)
dnf install gcc make automake libtool autoconf pkgconfig
HPC clusters
Most HPC clusters provide these tools via the module system:
module load gcc automake libtool autoconf
If in doubt, check whether autoreconf --version and libtool --version return successfully.
macOS
brew install automake libtool autoconf pkg-config
Building
git clone <repository-url>
cd obikmer/src
cargo build --release
The compiled binary is at target/release/obikmer.
Building on HPC clusters (network filesystems)
HPC home directories are typically on a network filesystem (Lustre, NFS) optimised for large sequential reads — not for the thousands of small file operations that Cargo generates during compilation. Building directly on such a filesystem can be extremely slow (0.1% CPU utilisation, tens of minutes for what should take seconds).
Always redirect the build directory to a local scratch disk:
CARGO_TARGET_DIR=/scratch/$USER/cargo-target cargo build --release
Adapt the path to the local scratch available on your cluster (/var/tmp, /tmp, /scratch/local, etc.). Once built, copy the binary to a permanent location:
cp /scratch/$USER/cargo-target/release/obikmer ~/bin/
NUMA support
NUMA-aware thread placement is active automatically on multi-socket Linux machines (detected at runtime via hwloc). No special build flag is required — the detection is built in and falls back gracefully to the single-pool adaptive strategy on:
- macOS (Apple Silicon, unified memory)
- single-socket Linux machines
- any system where hwloc reports only one NUMA node
Verifying the installation
obikmer --help