Eric Coissac
175ea5bbd0
Introduces a counting semaphore-based throttling mechanism to limit concurrent file I/O and pipeline processing. Replaces custom path wrappers with standardized `Throttled` types across `obikmer` and `obikpartitionner`, ensuring RAII-based resource cleanup and explicit backpressure. Additionally, documents how to redirect Cargo build artifacts to local scratch storage on HPC filesystems to prevent compilation slowdowns.
85 lines
2.4 KiB
Markdown
85 lines
2.4 KiB
Markdown
# Installation
|
|
|
|
## Prerequisites
|
|
|
|
### Rust toolchain
|
|
|
|
`obikmer` requires **Rust 1.85 or later** (edition 2024). Install or update via [rustup](https://rustup.rs):
|
|
|
|
```bash
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
|
rustup update stable
|
|
```
|
|
|
|
### C build environment (required for hwloc)
|
|
|
|
`obikmer` embeds [hwloc](https://www.open-mpi.org/projects/hwloc/) (Hardware Locality) for NUMA-aware thread placement on multi-socket machines. hwloc is built from source at compile time via the `vendored` feature of the `hwlocality` crate. This requires a standard C build environment.
|
|
|
|
#### Linux (Debian/Ubuntu)
|
|
|
|
```bash
|
|
apt install build-essential automake libtool autoconf pkg-config
|
|
```
|
|
|
|
#### Linux (RHEL/Rocky/AlmaLinux)
|
|
|
|
```bash
|
|
dnf install gcc make automake libtool autoconf pkgconfig
|
|
```
|
|
|
|
#### HPC clusters
|
|
|
|
Most HPC clusters provide these tools via the module system:
|
|
|
|
```bash
|
|
module load gcc automake libtool autoconf
|
|
```
|
|
|
|
If in doubt, check whether `autoreconf --version` and `libtool --version` return successfully.
|
|
|
|
#### macOS
|
|
|
|
```bash
|
|
brew install automake libtool autoconf pkg-config
|
|
```
|
|
|
|
## Building
|
|
|
|
```bash
|
|
git clone <repository-url>
|
|
cd obikmer/src
|
|
cargo build --release
|
|
```
|
|
|
|
The compiled binary is at `target/release/obikmer`.
|
|
|
|
### Building on HPC clusters (network filesystems)
|
|
|
|
HPC home directories are typically on a network filesystem (Lustre, NFS) optimised for large sequential reads — not for the thousands of small file operations that Cargo generates during compilation. Building directly on such a filesystem can be extremely slow (0.1% CPU utilisation, tens of minutes for what should take seconds).
|
|
|
|
**Always redirect the build directory to a local scratch disk:**
|
|
|
|
```bash
|
|
CARGO_TARGET_DIR=/scratch/local/$USER/cargo-target cargo build --release
|
|
```
|
|
|
|
Adapt the path to the local scratch available on your cluster (`/var/tmp`, `/tmp`, `/scratch/local`, etc.). Once built, copy the binary to a permanent location:
|
|
|
|
```bash
|
|
cp /scratch/local/$USER/cargo-target/release/obikmer ~/bin/
|
|
```
|
|
|
|
## NUMA support
|
|
|
|
NUMA-aware thread placement is active automatically on multi-socket Linux machines (detected at runtime via hwloc). No special build flag is required — the detection is built in and falls back gracefully to the single-pool adaptive strategy on:
|
|
|
|
- macOS (Apple Silicon, unified memory)
|
|
- single-socket Linux machines
|
|
- any system where hwloc reports only one NUMA node
|
|
|
|
## Verifying the installation
|
|
|
|
```bash
|
|
obikmer --help
|
|
```
|