Eric Coissac
b6fcbc545f
Replaces `rayon` parallel iteration across index, rebuild, reindex, and select modules with a custom `PartitionRunner`. This introduces NUMA-aware task distribution with CPU pinning and round-robin scheduling, eliminating `Arc`, `Mutex`, and atomic synchronization primitives in favor of a flat, pre-spawned worker architecture. Error handling is simplified via `.map_err()` and the `?` operator, while progress bar updates are decoupled into dedicated callbacks.
85 lines
2.4 KiB
Markdown
85 lines
2.4 KiB
Markdown
# Installation
|
|
|
|
## Prerequisites
|
|
|
|
### Rust toolchain
|
|
|
|
`obikmer` requires **Rust 1.85 or later** (edition 2024). Install or update via [rustup](https://rustup.rs):
|
|
|
|
```bash
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
|
rustup update stable
|
|
```
|
|
|
|
### C build environment (required for hwloc)
|
|
|
|
`obikmer` embeds [hwloc](https://www.open-mpi.org/projects/hwloc/) (Hardware Locality) for NUMA-aware thread placement on multi-socket machines. hwloc is built from source at compile time via the `vendored` feature of the `hwlocality` crate. This requires a standard C build environment.
|
|
|
|
#### Linux (Debian/Ubuntu)
|
|
|
|
```bash
|
|
apt install build-essential automake libtool autoconf pkg-config
|
|
```
|
|
|
|
#### Linux (RHEL/Rocky/AlmaLinux)
|
|
|
|
```bash
|
|
dnf install gcc make automake libtool autoconf pkgconfig
|
|
```
|
|
|
|
#### HPC clusters
|
|
|
|
Most HPC clusters provide these tools via the module system:
|
|
|
|
```bash
|
|
module load gcc automake libtool autoconf
|
|
```
|
|
|
|
If in doubt, check whether `autoreconf --version` and `libtool --version` return successfully.
|
|
|
|
#### macOS
|
|
|
|
```bash
|
|
brew install automake libtool autoconf pkg-config
|
|
```
|
|
|
|
## Building
|
|
|
|
```bash
|
|
git clone <repository-url>
|
|
cd obikmer/src
|
|
cargo build --release
|
|
```
|
|
|
|
The compiled binary is at `target/release/obikmer`.
|
|
|
|
### Building on HPC clusters (network filesystems)
|
|
|
|
HPC home directories are typically on a network filesystem (Lustre, NFS) optimised for large sequential reads — not for the thousands of small file operations that Cargo generates during compilation. Building directly on such a filesystem can be extremely slow (0.1% CPU utilisation, tens of minutes for what should take seconds).
|
|
|
|
**Always redirect the build directory to a local scratch disk:**
|
|
|
|
```bash
|
|
CARGO_TARGET_DIR=/scratch/$USER/cargo-target cargo build --release
|
|
```
|
|
|
|
Adapt the path to the local scratch available on your cluster (`/var/tmp`, `/tmp`, `/scratch/local`, etc.). Once built, copy the binary to a permanent location:
|
|
|
|
```bash
|
|
cp /scratch/$USER/cargo-target/release/obikmer ~/bin/
|
|
```
|
|
|
|
## NUMA support
|
|
|
|
NUMA-aware thread placement is active automatically on multi-socket Linux machines (detected at runtime via hwloc). No special build flag is required — the detection is built in and falls back gracefully to the single-pool adaptive strategy on:
|
|
|
|
- macOS (Apple Silicon, unified memory)
|
|
- single-socket Linux machines
|
|
- any system where hwloc reports only one NUMA node
|
|
|
|
## Verifying the installation
|
|
|
|
```bash
|
|
obikmer --help
|
|
```
|