Eric Coissac
175ea5bbd0
Introduces a counting semaphore-based throttling mechanism to limit concurrent file I/O and pipeline processing. Replaces custom path wrappers with standardized `Throttled` types across `obikmer` and `obikpartitionner`, ensuring RAII-based resource cleanup and explicit backpressure. Additionally, documents how to redirect Cargo build artifacts to local scratch storage on HPC filesystems to prevent compilation slowdowns.
2.4 KiB
Installation
Prerequisites
Rust toolchain
obikmer requires Rust 1.85 or later (edition 2024). Install or update via rustup:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update stable
C build environment (required for hwloc)
obikmer embeds hwloc (Hardware Locality) for NUMA-aware thread placement on multi-socket machines. hwloc is built from source at compile time via the vendored feature of the hwlocality crate. This requires a standard C build environment.
Linux (Debian/Ubuntu)
apt install build-essential automake libtool autoconf pkg-config
Linux (RHEL/Rocky/AlmaLinux)
dnf install gcc make automake libtool autoconf pkgconfig
HPC clusters
Most HPC clusters provide these tools via the module system:
module load gcc automake libtool autoconf
If in doubt, check whether autoreconf --version and libtool --version return successfully.
macOS
brew install automake libtool autoconf pkg-config
Building
git clone <repository-url>
cd obikmer/src
cargo build --release
The compiled binary is at target/release/obikmer.
Building on HPC clusters (network filesystems)
HPC home directories are typically on a network filesystem (Lustre, NFS) optimised for large sequential reads — not for the thousands of small file operations that Cargo generates during compilation. Building directly on such a filesystem can be extremely slow (0.1% CPU utilisation, tens of minutes for what should take seconds).
Always redirect the build directory to a local scratch disk:
CARGO_TARGET_DIR=/scratch/local/$USER/cargo-target cargo build --release
Adapt the path to the local scratch available on your cluster (/var/tmp, /tmp, /scratch/local, etc.). Once built, copy the binary to a permanent location:
cp /scratch/local/$USER/cargo-target/release/obikmer ~/bin/
NUMA support
NUMA-aware thread placement is active automatically on multi-socket Linux machines (detected at runtime via hwloc). No special build flag is required — the detection is built in and falls back gracefully to the single-pool adaptive strategy on:
- macOS (Apple Silicon, unified memory)
- single-socket Linux machines
- any system where hwloc reports only one NUMA node
Verifying the installation
obikmer --help