Compare commits
26
Commits
85296a2ef8
..
v1.4.7
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1119af85ba | ||
|
|
36c7247acb | ||
|
|
6d6d5e5e53 | ||
|
|
33d7a59dd9 | ||
|
|
d78a6b4501 | ||
|
|
5db7b440da | ||
|
|
c93616b432 | ||
|
|
c4eacb3f72 | ||
|
|
14433e80a5 | ||
|
|
7f829ea557 | ||
|
|
07c0a54e04 | ||
|
|
af525cd268 | ||
|
|
d0668eb167 | ||
|
|
4e98de4b77 | ||
|
|
03eaa4544a | ||
|
|
24291a4484 | ||
|
|
cbc344e4b4 | ||
|
|
07bc5fc634 | ||
|
|
1f199c69db | ||
|
|
b35127e4e3 | ||
|
|
0134c69cf4 | ||
|
|
5368c98875 | ||
|
|
b55123a376 | ||
|
|
b132f0847f | ||
|
|
ee62b131f9 | ||
|
|
c6226dd40c |
No files matched your search
@@ -62,52 +62,38 @@ fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
|
|||||||
|
|
||||||
Check all sources against the constraints above. Abort on any mismatch.
|
Check all sources against the constraints above. Abort on any mismatch.
|
||||||
|
|
||||||
### 2. Bootstrap output from first source
|
### 2. Bootstrap output from the base source
|
||||||
|
|
||||||
Recursive file copy of `sources[0]` → `output`. Immediately after the copy:
|
The base is chosen by `choose_base` (`merge.rs`): the source with the largest total `unitigs.bin` size (restricted to approximate-evidence sources when the output must be approximate). It is not necessarily `sources[0]`, and the genome columns follow the base first. Because `obikmer index` is not byte-reproducible (unitig segmentation varies), two near-equal sources can swap roles between runs. Its directory is copied to `output` (hard links for files `merge_partition` never rewrites, see below). Immediately after the copy:
|
||||||
|
|
||||||
- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
|
- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
|
||||||
- In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
|
- In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
|
||||||
- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
|
- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
|
||||||
|
|
||||||
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
|
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The base source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
|
||||||
|
|
||||||
### 3. For each subsequent source (parallel across partitions)
|
### 3. For each subsequent source (parallel across partitions)
|
||||||
|
|
||||||
`KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)` is called for each partition index `i`. `block_bits` is taken from `dst.meta.config.block_bits`.
|
`obikmerge::partition_merge::merge_partition(dst, i, sources, mode, n_dst_genomes, block_bits, evidence)` runs once per partition index `i` (scheduled largest first by `PartitionRunner`). Each entry of `sources` is `(&KmerIndex, n_genomes)`, where `n_genomes` is the number of columns that source contributes (> 1 for an already-merged source). Code: `partition_merge/mod.rs` (orchestration, pass 1) and `partition_merge/pass2.rs` (per-group fill pipeline).
|
||||||
|
|
||||||
Each entry in `sources` is `(&KmerPartition, n_genomes)` where `n_genomes` is the column count that source contributes (> 1 when the source is itself a merged index).
|
**First merge, Presence mode**: when `n_dst_genomes == 1`, `TypedLayer::<()>::init_presence_matrix` is called on every existing dst layer first. It materialises the implicit single-genome matrix (`presence/col_000000.pbiv`, all true) so that source columns can be appended to it.
|
||||||
|
|
||||||
**First merge, Presence mode**: when `n_dst_genomes == 1`, `Layer::<()>::init_presence_matrix` is called on every existing destination layer before any source column is appended. This creates `presence/col_000000.pbiv` set all-true (genome 0 is present in every slot).
|
The dst layers are opened once per partition and shared by both passes.
|
||||||
|
|
||||||
**Pass 1 — classify kmers**
|
**Pass 1 — new k-mers into a de Bruijn graph.** Only the sources' `unitigs.bin` *paths* are needed (`KmerLayer::new(..).unitigs_path()`); no source layer is opened. The files are streamed (at most `max_open = 2` open at once), and every k-mer absent from all dst layers is pushed into one `GraphDeBruijn`. All new k-mers of all sources go into a **single** new layer per merge, built by `materialize_layer` (unitigs + MPHF + evidence).
|
||||||
|
|
||||||
Iterate all kmers from all source partitions (via `UnitigFileReader` + canonical kmer iteration). For each kmer, probe the destination `LayeredMap<()>`:
|
**New-layer matrix.** If a new layer exists, its matrix is created with `n_dst_genomes` absent columns (all zero/false) and closed immediately. This keeps the column-count invariant, and every source group then appends to it like to the existing layers.
|
||||||
|
|
||||||
- **Hit**: kmer already in the destination; record for Pass 2.
|
**Pass 2 — by consecutive groups of `SRC_GROUP = 2` sources, in source order.** Per group:
|
||||||
- **Miss**: push kmer into a `GraphDeBruijn` accumulator.
|
|
||||||
|
|
||||||
**New layer construction**
|
1. open the group's source layers for this partition;
|
||||||
|
2. `MatrixBuilder::resume` every dst matrix (existing layers + new layer), check `n_cols == n_dst_genomes + col_base`, append the group's `group_n` columns;
|
||||||
|
3. run the pipeline: stream each source layer's k-mers, `hash_batch` + `fill_sub_matrix` on the source, locate each k-mer in the dst layers (`find_batch_grouped`) or else in the new layer's MPHF, and write the values column-major into the group's builders;
|
||||||
|
4. close the columns, then the matrices; drop the group's layers; `col_base += group_n`.
|
||||||
|
|
||||||
If the accumulator is non-empty, compute de Bruijn unitigs and call `Layer::<()>::build(&new_layer_dir, block_bits)`. All kmers absent from the destination — across **all** sources — accumulate into a **single** graph, producing one new layer per merge operation (not one per source).
|
Only one group's source layers and destination columns are alive at a time, so the number of live mappings per partition does not depend on the number of sources. Columns are appended in source order, so every genome lands in the same absolute column as in a single-session merge.
|
||||||
|
|
||||||
**Pass 2 — fill column builders**
|
**In-memory columns.** The group's columns are built in RAM (`MatrixBuilder::add_mem_col` → `MemBitVecBuilder` / `MemCompactIntVecBuilder`) and written once, sequentially, at `close` (`.building` file, then `rename`). The files are byte-identical to the former mmap-backed builders' output. There is no sparse file and no `MmapMut` on this path, so a block the FS cannot allocate is an `io::Error` instead of a SIGBUS. Files are not `fsync`ed: one sync per column file made merge ~8× slower. RAM per partition is at most `SRC_GROUP × src_n × N_part × (1/8 | 1)` bytes (presence | counts). History and figures: `merge_mmap_audit.md`; scaling limits: `merge_scaling.md`.
|
||||||
|
|
||||||
For each source and each of its layers, re-iterate unitigs and look up stored values via `SrcLayerData::lookup(kmer, src_n)`:
|
|
||||||
|
|
||||||
- `SrcLayerData::SetMembership` — no data directory exists; every kmer returns `vec![1; n_genomes]`
|
|
||||||
- `SrcLayerData::Presence` — reads `PersistentBitMatrix` from `presence/`
|
|
||||||
- `SrcLayerData::Count` — reads `PersistentCompactIntMatrix` from `counts/`
|
|
||||||
|
|
||||||
Hits are routed to `exist_builders[dst_layer][src_col]`; misses are routed to `new_src_builders[src_col]`.
|
|
||||||
|
|
||||||
**Column prepending for new layers**
|
|
||||||
|
|
||||||
Before source columns are written to the new layer, `n_dst_genomes` absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.
|
|
||||||
|
|
||||||
**Close and update metadata**
|
|
||||||
|
|
||||||
Close all builders; update `presence/meta.json` or `counts/meta.json` with `{"n": N, "n_cols": n_dst_genomes + n_src_total}`; increment `PartitionMeta::n_layers` if a new layer was added.
|
|
||||||
|
|
||||||
### 4. Update index metadata
|
### 4. Update index metadata
|
||||||
|
|
||||||
@@ -129,7 +115,7 @@ impl Layer<PersistentBitMatrix> {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written directly via `PersistentBitVecBuilder` / `PersistentCompactIntVecBuilder` rather than through these helpers, but the invariant they enforce is the same.
|
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written through `MatrixBuilder::add_mem_col` (in-memory builders) rather than through these helpers, but the invariant they enforce is the same.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -139,7 +125,7 @@ After any merge, **every layer in every partition has exactly `n_genomes` column
|
|||||||
|
|
||||||
Maintained by three mechanisms:
|
Maintained by three mechanisms:
|
||||||
|
|
||||||
1. **Existing layers**: `n_src_total` columns appended (one per source genome).
|
1. **Existing layers**: `n_src_total` columns appended (one per source genome), group by group.
|
||||||
2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
|
2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
|
||||||
3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
|
3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
|
||||||
|
|
||||||
@@ -161,7 +147,7 @@ The invariant is a precondition of `LayeredStore` aggregation traits: `col_weigh
|
|||||||
|
|
||||||
## On-disk impact
|
## On-disk impact
|
||||||
|
|
||||||
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest):
|
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest), *before* `finalize` packs each matrix (sparse by default, `matrix.pbmx` with `--dense`):
|
||||||
|
|
||||||
```
|
```
|
||||||
partitions/
|
partitions/
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Merge: mmap audit (2026-11-17)
|
||||||
|
|
||||||
|
Audit of the memory-mapping behaviour of `obikmerge::partition_merge::merge_partition`, triggered by two production failures on a 192-core NUMA server merging 186 freshly built mono-genome presence indexes (256 partitions, data on BeeGFS `/hoyt`). Status: **A + C implemented (2026-11-17)**, B deferred. See [Implementation](#implementation-2026-11-17). Implementation prompt: `prompts/merge_mmap_refactor.md` (repository root, outside the doc tree).
|
||||||
|
|
||||||
|
## Observed failures
|
||||||
|
|
||||||
|
| Run | Flags | Failure | When |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | none | `Bus error` (SIGBUS) | seconds after `merge_partitions` starts |
|
||||||
|
| 2 | `--cpu-max 32` | `memory allocation of 32768 bytes failed` → abort, many threads at once | after ~1 h, 128/256 partitions done |
|
||||||
|
|
||||||
|
Run 2 rules out "one partition too large for RAM": partitions are scheduled largest first (LFD, `merge.rs:289`), so at 50 % every remaining partition is smaller than one already merged. Small `malloc` failures on a 700 GB host are not RAM exhaustion either. Linux would OOM-kill (SIGKILL) instead. A NULL from `malloc` means `mmap`/`brk` returned `ENOMEM`: `vm.max_map_count` (default 65 530), `RLIMIT_AS`, or strict overcommit. Root access is not available on the compute server, so the fix must be in the code.
|
||||||
|
|
||||||
|
## Inventory: mappings held by one `merge_partition` call
|
||||||
|
|
||||||
|
Counts read from the code; totals assume 1 layer per source partition and 1 pre-existing dst layer (estimate, not measured).
|
||||||
|
|
||||||
|
| Resource | Where | Maps | Lifetime | Needed? |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Source layers, fully opened | `partition_merge.rs:219-227` → `KmerLayer::open` → `MphfLayer::open` (`mphf_layer.rs:171`) | 4 per layer: MPHF (`MphfOnly`), `unitigs.bin` (direct access), `evidence.bin`, `unitigs.bin` again (`LayerEvidence::open`, `mphf_layer.rs:100`). ≈ **740** | whole partition | Only the MPHF. Pass 2 calls `hash_batch` (MPHF only, `mphf_layer.rs:258`) and `fill_sub_matrix` (matrix: `Implicit` for mono-genome, zero maps). Pass 1 only needs the `unitigs.bin` *path*. |
|
||||||
|
| Writable columns, created up front | `partition_merge.rs:327`, `:349` → `PersistentBitVecBuilder::new` / `PersistentCompactIntVecBuilder::new` | `set_len` (sparse) + `MmapMut`, `n_src_total × (L_dst + 1)` ≈ **370** | whole pass 2 | Only the columns of the ≤ 2 sources being streamed (`max_open = 2`). |
|
||||||
|
| dst layers, new-layer MPHF, sequential readers | various | ≈ 10 | variable | yes |
|
||||||
|
|
||||||
|
≈ **1 100 maps per partition**, > 95 % idle at any instant. × ~32 concurrent partitions (`--cpu-max 32`) ≈ 35 k, plus stacks and glibc arenas of several hundred pipeline threads. That is within reach of 65 530. Without a cap (192 cores) it is several times over. Consistent with run 2, **not measured**. Whether run 1's SIGBUS comes from the same exhaustion or from failure 4 below is **not established**.
|
||||||
|
|
||||||
|
## Weaknesses
|
||||||
|
|
||||||
|
1. **Source layers opened in full** although merge needs MPHF + matrix only. Evidence and both `unitigs.bin` maps are dead weight.
|
||||||
|
2. **All sources opened up front, for the whole partition.** This contradicts the point of partitioning, which is to work on a bounded sub-part of the index. The cost of one partition grows linearly with the number of sources, whatever the partition size.
|
||||||
|
3. **All destination columns created up front**, the same mismatch on the write side.
|
||||||
|
4. **Sparse files written through `MmapMut`.** A page whose block cannot be allocated (quota, full FS, BeeGFS) faults with SIGBUS. That cannot be caught and reports nothing.
|
||||||
|
5. **No bound on live mappings.** `obipipeline::throttle` only bounds open read streams.
|
||||||
|
6. *(minor)* `obicompactvec::mmap_file::open_mmap_file` validates magic and header length, not file length against `n`.
|
||||||
|
|
||||||
|
Side finding: `merge_parallelism.md` documents a `MemoryBudget` semaphore and `--budget-fraction`. `obisys::MemoryBudget` still exists but `merge.rs` no longer uses it (scheduling is `PartitionRunner`). That page is stale on this point.
|
||||||
|
|
||||||
|
## Plan
|
||||||
|
|
||||||
|
**A. Source-group streaming in pass 2** (`obikmerge`). Process sources in consecutive groups of `G` (G = 2, matching `max_open`). Per group: open its layers, resume each dst matrix (existing layers + new layer), append its `src_n` columns, fill them through the existing pipeline, close, drop. Pass 1 opens no source layer: it builds paths through `KmerLayer::new(..).unitigs_path()` (valid on `Empty`). Live maps per partition become independent of the source count. Column order is unchanged (sources visited in order, columns appended in order), so the on-disk result is identical. Fixes 2, 3, 5.
|
||||||
|
|
||||||
|
**C. In-memory column builders** (`obicompactvec`). Under A only `G × src_n × (L_dst+1)` columns are alive. Build them in RAM (bits: n/8 B; counts: n B + overflow map) and write once with `write_all` to the `.building` path, then publish. No sparse file and no `MmapMut` on the merge write path, so `ENOSPC`/quota becomes an `io::Error`. The on-disk format must be byte-identical to the current builders' output. Exposed through `MatrixBuilder`/`ColBuilder` so `obikselect`/`obikfilter` can adopt it later. Fixes 4.
|
||||||
|
|
||||||
|
**B. Slots-only layer opening** (`obikindex`, the authority crate). **Deferred to a separate work item** (decision 2026-11-17). Consumers whose kmers are members by construction need MPHF + matrix, never evidence. The same `hash_batch` + `fill_sub_matrix` pattern without `find` appears in three crates: `obikmerge` (`partition_merge.rs:489`), `obikfilter` (`partition_iter.rs:106`), `obikphylo` (`family_scan.rs:289`). `family_scan.rs:402` hashes `variants`; whether those are guaranteed members is unchecked. Fixes 1.
|
||||||
|
|
||||||
|
- **Rejected: lazy evidence inside `MphfLayer`.** `find`/`confirm` return `Option<usize>`, so an evidence open error at first use cannot be propagated (panic, or a signature change across the query path). It would also add an "initialised?" branch to the hottest path, and hide in runtime state a distinction that is structural: external query (`find`, needs evidence) vs in-layer iteration (`index`/`index_batch`, no evidence).
|
||||||
|
- **Rejected: a standalone parallel type** next to `KmerLayer`/`TypedLayer`. It would duplicate count/presence detection and matrix access, and two opening paths would drift.
|
||||||
|
- **Retained: composition.** A slots layer = `MphfOnly` + matrix, carrying `hash_batch`, `fill_sub_matrix`, `n_cols`. The full layer = slots layer + evidence, keeping `find`/`confirm`. `KmerLayer` exposes both openings (`open()` / e.g. `open_slots()`) over a single count/presence detection. Open point: `MphfLayer::n` currently comes from the evidence (`evidence.len()` / `fingerprint.n()`). A slots layer must take it from the matrix (`n_rows`; `layer_meta.json` for `Implicit`), so the two must be checked to always agree.
|
||||||
|
|
||||||
|
Why deferred: after A, B only takes merge from ~8 to ~2 maps per partition, which is negligible. It touches the `obikindex` query path and needs its own session with `query` performance checks.
|
||||||
|
|
||||||
|
Order: oracle first (end-to-end merge golden output), then A → C. A alone removes the scaling problem; C removes the SIGBUS class.
|
||||||
|
|
||||||
|
## Implementation (2026-11-17)
|
||||||
|
|
||||||
|
**A — source-group streaming** (`obikmerge/src/partition_merge/`, split into `mod.rs` + `pass2.rs`). As planned: pass 1 builds paths only; the new-layer matrix is created with its absent columns and closed; pass 2 iterates `sources.chunks(SRC_GROUP = 2)`, resuming each dst matrix, checking `n_cols == n_dst_genomes + col_base` (`InvalidData` otherwise), and appending the group's columns. Prerequisite: `MatrixBuilder::n_cols` (`obicompactvec`).
|
||||||
|
|
||||||
|
**C — in-memory columns** (`obicompactvec`). New `mem_builder.rs` (`MemBitVecBuilder`, `MemCompactIntVecBuilder`), `add_mem_col` on both matrix builders and on `MatrixBuilder`, and `ColBuilder::MemBit`/`MemInt`. Format writing is factored rather than duplicated: `format::write_pciv_header`/`write_pciv_tail` (also used by `finalize_pciv`), `mmap_file::write_std_header`/`write_built`. Byte identity with the mmap builders is unit-tested for n ∈ {0, 1, 63, 64, 65, 10 000}, with overwrites and > 2 048 overflow entries. Merge uses `add_mem_col` for source columns; `add_absent_col` is unchanged (never written through its mapping).
|
||||||
|
|
||||||
|
Deviations from the plan:
|
||||||
|
|
||||||
|
- **No `sync_all` in `close`.** With one forced sync per column file, `merge_partitions` went from 7.4 s to 63.6 s on the benchmark; without it, 7.0 s (measured, macOS). The mmap builders never synced either. The `rename` still prevents publishing a partial file. Caveat: on a write-back network FS an allocation error may only surface at `close(2)`, which `File`'s drop ignores.
|
||||||
|
- `add_mem_col` returns `ColBuilder`, not `io::Result<ColBuilder>`: it does no I/O.
|
||||||
|
- The mmap count builder (`PersistentCompactIntVecBuilder`) writes in place, without `.building`; the in-memory one goes through `.building` + `rename`. Same final bytes.
|
||||||
|
- `obikmerge`'s `matrix_builder_tests` were removed: they only exercised `obicompactvec::MatrixBuilder` and are covered by `obicompactvec/src/tests/{matrix_builder,mem_builder}.rs`. `tempfile` dev-dependency of `obikmerge` removed with them.
|
||||||
|
|
||||||
|
**Verification.** Two golden-output oracles, recorded on the unmodified code:
|
||||||
|
|
||||||
|
- `scripts/smoke_test_merge.sh`: synthetic, 4 genomes with shared fragments and counts ≥ 255, presence/count/`--dense`; frozen source indexes;
|
||||||
|
- `scripts/bench_merge_oracle.sh`: 20 real specimen presence indexes, sorted `dump --debug` hash, 85 681 700 k-mers.
|
||||||
|
|
||||||
|
Both pass after every step. The existing `global_index_presence` (October binary) gives the same hash.
|
||||||
|
|
||||||
|
`merge_partitions` wall time on the benchmark: 7.40 s (ref) → 7.91 s (A) → 7.07 s (A + C) → 8.05 s (after the module split, same code). Single runs; the spread is noise. RSS 243–256 MB throughout. The benchmark is too small to show either the mapping or the memory effect.
|
||||||
|
|
||||||
|
**Live mappings per partition (estimate, not measured).** Assumptions: 186 mono-genome sources, 1 layer each, 1 pre-existing dst layer.
|
||||||
|
|
||||||
|
| | Source layers | Dst columns | Other | Total |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Before | 186 × 4 ≈ 744 | 186 × 2 ≈ 372 (`MmapMut`) | ≈ 10 | **≈ 1 100** |
|
||||||
|
| After A + C | 2 × 4 = 8 | 0 (in RAM) | ≈ 10 | **≈ 20**, independent of the source count |
|
||||||
|
|
||||||
|
B would take the 8 down to ~2.
|
||||||
|
|
||||||
|
**Follow-up — sparse pack (same day).** `pack_sparse_bit_matrix`/`pack_sparse_compact_int_matrix` used to open the columnar matrix directly, which maps every column file of a layer at once: `n_genomes` maps per partition, since layers are packed one after another. Both now call the dense pack first. It concatenates the column files one at a time without mapping them, so the sparse matrix is built from a single `matrix.pbmx`/`.pcmx` mapping. This deliberately reverses an earlier choice ("never materialise a packed copy just to read it back once"). It costs one extra write and read of the dense matrix: `pack` went from 1.11 s to 1.60 s on the benchmark (single run). As a side effect, a stale `matrix.pbmx` that predates newer columnar data is now re-packed instead of being used as is (`pack_bit_matrix` compares `n_cols`). Both oracles pass.
|
||||||
|
|
||||||
|
## Re-merging a merge output (resolved 2026-11-18)
|
||||||
|
|
||||||
|
Symptom: `I/O error: No such file or directory` in `merge_partitions` when a merge output was one of the sources. Cause (verified): `finalize_indexed` packs the output matrices (`sparse_meta.json` + dict/PFIV files, or `matrix.pbmx`), so `presence/`/`counts/` no longer holds `meta.json` + `col_*`. `choose_base` picks the largest source as destination, i.e. the merged one; bootstrap copies the packed layout and `MatrixBuilder::resume` → `MatrixMeta::load` finds no `meta.json`.
|
||||||
|
|
||||||
|
Fix: a packed index can only be a *source* (pass 2 reads it through `PersistentBitMatrix`/`PersistentIntMatrix::open`, which handle every layout). `choose_base` now filters candidates with `can_be_destination` (`obikmerge/src/merge.rs`): one layer is probed with `detect_storage`; `Columnar` qualifies, `Implicit` only for a single genome (materialized on the fly), `Packed`/`Sparse` never. With a single source there is nothing to append and the filter is skipped. When several sources are given and none qualifies, `run` returns an explicit error before any work. Consequence: the result's column order follows the chosen base first, so it can differ from the command-line order.
|
||||||
|
|
||||||
|
Not done: in-place unpack of a packed index to allow it as destination, or a destination-side append on the packed format — the sparse incremental strategy is tracked in `merge_scaling.md`.
|
||||||
|
|
||||||
|
Oracle: `scripts/smoke_test_merge.sh` scenario 2 (A B → AB, then AB C D) is compared against the flat merge by genome label (not against the golden dumps, whose layer placement differs), and merging only packed indexes is checked to be refused.
|
||||||
|
|
||||||
|
Residual risk, accepted: read-only maps still SIGBUS if a file is truncated by another process during a merge. That is inherent to `mmap`.
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Merge: scaling to thousands of genomes (open, 2026-11-17)
|
||||||
|
|
||||||
|
Open design question raised during the mmap refactor (`merge_mmap_audit.md`). Target: merges of ~6 000 genomes of a few Gb each. Nothing here is implemented or measured. Figures are estimates, labelled as such.
|
||||||
|
|
||||||
|
## What scales with what (after the mmap refactor)
|
||||||
|
|
||||||
|
| Cost | Depends on | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| Live source layers + dst columns per partition (merge pass 2) | `SRC_GROUP × src_n`, not the source count | fixed by plan A |
|
||||||
|
| Pass-2 column RAM | `SRC_GROUP × src_n × N_part × (1/8 \| 1)` B | bounded, see below |
|
||||||
|
| Pass-1 de Bruijn graph | ≥ ~19 B per *new* k-mer (`HashMap<u64, AtomicU8>`, verified in `obidebruinj/src/debruijn/graph.rs`) | unchanged |
|
||||||
|
| Columnar intermediate files | `n_genomes × n_layers × n_partitions` (6 000 × 2 × 256 ≈ 3 M files, estimate) | **does not scale** |
|
||||||
|
| Sparse `pack` | used to map **all** columns of a layer at once (`ColumnarBitMatrix::open`); layers are packed one after another, so `n_genomes` maps per partition (186 × 192 concurrent ≈ 36 k, estimate) | **fixed 2026-11-17**: now goes columnar → dense `matrix.pbmx` (sequential concatenation, unmapped) → sparse, one mapping per partition |
|
||||||
|
|
||||||
|
`N_part`: k-mers of one dst partition over all layers. Pass-2 column RAM stays below pass-1 graph RAM for one-shot merges. It dominates only for **incremental** merges, where few genomes are added to a large index: the new layer is small, the existing layers are large. Example (estimate): `N_total = 1e12`, counts, 192 concurrent partitions → ~1.5 TB. A memory-budgeted concurrency cap is needed there whatever the column strategy. `obisys::MemoryBudget` exists but is no longer used by merge.
|
||||||
|
|
||||||
|
## Why a columnar intermediate at all
|
||||||
|
|
||||||
|
The sparse format (`bitmatrix/sparse.rs`) is row-major and deduplicated: per slot, either one genome (`singleton.pfiv`) or a `dict_id` (`multi.pfiv`) into a dictionary of distinct genome sets (`dict_offsets` + `dict_values.bin`). Its builder only offers `push_row(genomes)`, i.e. the **complete** genome set of a row, rows in **slot order**.
|
||||||
|
|
||||||
|
Merge discovers values **column by column** (source by source), in **random slot order** (MPHF order). The columnar files are a transposition buffer: they absorb random per-column writes, and `pack` transposes them row by row.
|
||||||
|
|
||||||
|
## Options to build the final format directly
|
||||||
|
|
||||||
|
1. **All sources at once.** That is exactly what plan A removed. No.
|
||||||
|
2. **Whole matrix in RAM.** `N_part × n_genomes / 8` B per partition: ~3 TB at 6 000 genomes and `N_part = 4e9` (estimate). No.
|
||||||
|
3. **External sort of `(slot, genome)` pairs.** Volume ∝ non-zeros ≈ Σ genome k-mers (~6 000 × 3e9). That is huge, while the final format is small precisely because genome sets repeat. No.
|
||||||
|
4. **Incremental sparse update per source group** (most promising). Per group:
|
||||||
|
- fill the group's columns in RAM (as now);
|
||||||
|
- stream the partition's existing sparse matrix **sequentially in slot order**;
|
||||||
|
- for each row: new set = old set ∪ group bits. Through the dictionary this is a transition `(old dict_id, group bit pattern) → new dict_id`, which can be memoised in a table;
|
||||||
|
- write the new sparse matrix sequentially, then publish.
|
||||||
|
|
||||||
|
No columnar files, no `pack`, no per-genome mapping, sequential I/O only. It also removes the re-merge bug: it starts from an existing sparse matrix instead of requiring a columnar one.
|
||||||
|
|
||||||
|
The cost is **one full pass over the partition's sparse matrix per group**. With `SRC_GROUP = 2` and 6 000 genomes that is 3 000 rewrites, far too many. Groups must therefore be large (64–256 sources), and group column RAM grows with them: `group × N_part / 8` B, e.g. 32 GB per partition for 64 sources at `N_part = 4e9` (estimate). One parameter trades passes against RAM, and it should be sized from a memory budget. The count matrix (`sparse_intmatrix.rs`) has its own format and needs the same analysis.
|
||||||
|
|
||||||
|
## Related findings
|
||||||
|
|
||||||
|
- The module header of `bitmatrix/sparse.rs` says "not used by any production code path yet". This is stale: `finalize_indexed` packs to sparse by default.
|
||||||
|
- The new layer of a merge is not byte-reproducible run to run: unitig segmentation varies, so the MPHF, evidence and matrix row order vary with it (measured 2026-11-17, `layer_1/*` differs in all 256 partitions; `layer_0` and metadata are identical). `obikmer index` shows the same behaviour. Merge oracles therefore compare sorted dumps, not file hashes. Where the non-determinism originates (parallel graph fill or unitig walk) is not established.
|
||||||
@@ -509,6 +509,7 @@ Provided finalisations:
|
|||||||
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
|
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
|
||||||
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
|
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
|
||||||
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
|
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
|
||||||
|
| `threshold_hamming_dist_matrix(t)` | `union[i,j] − inter[i,j]` (k-mers present in exactly one genome) — no separate partial |
|
||||||
|
|
||||||
### BitPartials
|
### BitPartials
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -13,11 +13,11 @@
|
|||||||
| `query` | Query an index with sequences and annotate matches |
|
| `query` | Query an index with sequences and annotate matches |
|
||||||
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
|
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
|
||||||
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
|
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
|
||||||
| `phylo` | Compute pairwise evolutionary-distance proxies between genomes (`--metric jaccard\|mash\|hamming\|bray-curtis\|relfreq-bray-curtis\|euclidean\|relfreq-euclidean\|hellinger\|hellinger-euclidean`); optionally build NJ/UPGMA trees; `--presence-threshold N` sets the minimum count to consider a k-mer present when computing Jaccard/Mash on count indexes (default 1); optionally a central-position SNP/Sankoff calibration with exports for TNT/PhyG/IQ-TREE (see [evolutionary distances](theory/evolutionary_distances.md)) |
|
| `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash/Hamming on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
|
||||||
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
|
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
|
||||||
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
|
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
|
||||||
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
|
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
|
||||||
| `reindex` | Convert an index's evidence in-place: exact ↔ approx |
|
| `convert` | Convert an index's evidence representation (exact / approximate / hybrid), in place |
|
||||||
| `utils` | Miscellaneous index utilities: `--new-label NEW=OLD` renames a genome label; `--upgrade-index` adds missing `layer_meta.json` to old indexes |
|
| `utils` | Miscellaneous index utilities: `--new-label NEW=OLD` renames a genome label; `--upgrade-index` adds missing `layer_meta.json` to old indexes |
|
||||||
| `pack` | Pack per-column matrix files into single-file format to reduce query I/O |
|
| `pack` | Pack per-column matrix files into single-file format to reduce query I/O |
|
||||||
|
|
||||||
|
|||||||
@@ -2873,6 +2873,90 @@ user, no external consumers to break). Scoped to the distance matrix only:
|
|||||||
`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
|
`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
|
||||||
their existing CSV-only format.
|
their existing CSV-only format.
|
||||||
|
|
||||||
|
## VCF export of the pseudo-alignment
|
||||||
|
|
||||||
|
`phylo --vcf` writes the drawn sites of `--pseudo-alignment` as a VCF
|
||||||
|
(`--fasta`/`--vcf` choose the formats, FASTA is the default, both can be
|
||||||
|
combined). A family has no genomic coordinate, so coordinates are virtual and
|
||||||
|
derived from where the scan finds the family: `CHROM = obikmer_<partition>_<layer>`,
|
||||||
|
`POS = family_idx + 1` (`scan_layer_families`' iteration-order index inside
|
||||||
|
that layer). Stable for a given index + annex, sorted by construction because
|
||||||
|
`on_family` is replayed in `family_idx` order. A global dense rank or a bit
|
||||||
|
packing of (layer, partition, idx) into one `POS` was rejected: it needs
|
||||||
|
per-layer minorant counts and can exceed VCF's 2^31-1 bound.
|
||||||
|
|
||||||
|
`ID` is `CanonicalKmer::family_id()`: centre forced to `A` on the
|
||||||
|
k-mer and on its revcomp, each canonicalised, minimum of the two (centre
|
||||||
|
printed as `N`). Index-independent, hence the key for merging VCFs from
|
||||||
|
different indexes. No `INFO/LOCUS` duplicate (dropped: same string as `ID`, costly
|
||||||
|
at 5M loci); `QUAL`/`FILTER`/`INFO` are all `.`. An alternative defined as the minimum of
|
||||||
|
`central_canonical_neighbors()` was dropped: it canonicalises 4 k-mers where
|
||||||
|
this one needs 2, with no other benefit. `family_id` idempotence and
|
||||||
|
family-wide equality are tested exhaustively for k = 3, 5, 7.
|
||||||
|
|
||||||
|
`SurvivingFamily` now carries the minorant k-mer (the scan callback passes it),
|
||||||
|
and `SnpAlignment` a `SiteLocus` per column. Alleles are the unambiguous cells
|
||||||
|
of the IUPAC matrix, in the FASTA's convention (central base of the member's
|
||||||
|
canonical form, not necessarily the `ID` k-mer's strand).
|
||||||
|
|
||||||
|
Absence handling. `masked_state` keeps absence (`m == 0`, `-`) as a real state
|
||||||
|
unless `--free-loss`, where it becomes missing (`?`). A first version mapped
|
||||||
|
`-` and `?` both to `.`: on a toy 4-genome index half of the lines then had
|
||||||
|
`ALT = .` (presence/absence-only sites, selected by `survives_masking`), i.e.
|
||||||
|
polymorphic sites rendered as monomorphic. Absence became an extra allele,
|
||||||
|
first written `*`; that was a misuse: in VCF 4.5 (section on the ALT column)
|
||||||
|
`*` is "allele missing due to overlapping deletion", and `<*>` is the gVCF
|
||||||
|
unspecified allele, `DEL`/`INS`/`DUP`/`INV`/`CNV` are structural-variant
|
||||||
|
types. Now the symbolic allele `<ABS>` (free-form ID, declared by a
|
||||||
|
`##ALT=<ID=ABS,...>` line, always written in the header, listed after the
|
||||||
|
bases); `?` and IUPAC ambiguity codes stay `.` (haploid GT cannot express a
|
||||||
|
genome carrying several forms; superseded below). A symbolic ALT formally
|
||||||
|
asks for a padding base before `POS`; moot here, positions being virtual.
|
||||||
|
Checked with `bcftools view`/`stats` and `tabix`: no warning, the sites with
|
||||||
|
`<ABS>` are classed "others" by `bcftools stats`, and
|
||||||
|
`-e 'ALT="<ABS>"'` drops exactly them. The writer needs no knowledge of
|
||||||
|
`--free-loss`: `-` simply never occurs with it. A reference-genome option
|
||||||
|
(`--vcf-ref`) was implemented then dropped: `REF` is always the per-site
|
||||||
|
majority base (alphabetical on ties).
|
||||||
|
|
||||||
|
Ploidy. VCF allows any ploidy in `GT` (number of alleles written, variable
|
||||||
|
per sample/site), but an obikmer cell only says *which* forms a genome
|
||||||
|
carries, not their multiplicity (`A/A/G` and `A/G/G` are both `R`), nor
|
||||||
|
whether several forms mean heterozygosity or paralogues. Compromise chosen:
|
||||||
|
`--no-ambiguity` => haploid (ambiguous cells are already `?`); otherwise
|
||||||
|
diploid, single base `X` -> `X/X`, two-base code -> `X/Y`, `-` -> `<ABS>/<ABS>`, and
|
||||||
|
three/four-base codes or `?` -> `./.`. `REF` is the majority allele counting
|
||||||
|
each copy once (so a het contributes 1+1, a homozygote 2). Counting copies
|
||||||
|
for a real dosage (polyploids) would need k-mer counts (`--with-counts`),
|
||||||
|
not used by the pseudo-alignment scan today. `--only-diploid` (selection option, `conflicts_with --no-ambiguity`) adds a
|
||||||
|
third `masked_state` rule: `count_ones() > 2` -> missing. It lives in
|
||||||
|
`SamplingSpec`/`masked_state`, i.e. the post-hoc masking and
|
||||||
|
`survives_masking`, so FASTA, Sankoff and `snp-*` see it too. Unit-tested
|
||||||
|
(`masked_state`); the toy index has no 3/4-member cells, so the end-to-end
|
||||||
|
run is identical to the default mode.
|
||||||
|
|
||||||
|
First version of this flag read the persisted annexes under the default
|
||||||
|
convention (a third variant was judged not worth it). Rejected: the draw
|
||||||
|
would then describe other genomes than the alignment, and the pair-balance
|
||||||
|
weights and entropy are computed from the carrier set. Replaced by
|
||||||
|
`AmbiguityPolicy { Keep, Forbid, OnlyDiploid }` (one type instead of
|
||||||
|
`free_loss`/`no_ambiguity`/`only_diploid` booleans in `masked_state`,
|
||||||
|
`is_carrier`, `SamplingSpec` and every annex key). Annex headers store
|
||||||
|
`AmbiguityPolicy::header_flags` (0 = Keep, 1 = Forbid, as the old bit, 2 =
|
||||||
|
OnlyDiploid), so annexes written before the change stay valid; each policy
|
||||||
|
has its own file (`*.diploid.*` for the new one). `--sibling-annex` no longer
|
||||||
|
precomputes everything (`AnnexRequest::everything` removed, a rebuild only
|
||||||
|
costs the base annex): it builds the derived annexes named by the other
|
||||||
|
options of the command (`AnnexSelection`: `--entropy`, `--quartet`,
|
||||||
|
`--pair-balance`, `--family-overlap`/`--min-shared-family`, under the one
|
||||||
|
policy given); the rest is built on demand as before. A rebuild still drops
|
||||||
|
every derived annex (their `family_idx` numbering is tied to the base one).
|
||||||
|
Checked on the toy index: `--sibling-annex` alone leaves only
|
||||||
|
`siblings.psib`; `--quartet --only-diploid` adds `quartet.diploid.pbiv` only.
|
||||||
|
Toy index: default mode 187
|
||||||
|
lines, no `ALT = .`; `--no-ambiguity` 93 lines (the sites whose polymorphism
|
||||||
|
was only ambiguity codes disappear, since those cells became `?`).
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
The Mash mutation-rate model this discussion contrasts with:
|
The Mash mutation-rate model this discussion contrasts with:
|
||||||
|
|||||||
@@ -226,7 +226,9 @@ Design decision (user): the way genomes are counted depends on `--no-ambiguity`,
|
|||||||
|
|
||||||
## Implemented: `--quartet` (2026-10-13)
|
## Implemented: `--quartet` (2026-10-13)
|
||||||
|
|
||||||
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Absence never counts as a state; with `--no-ambiguity`, multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; the `--no-ambiguity` convention is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`) because that header has no room for a flag. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
|
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Genomes are read through `masked_state`, exactly as in the alignment: absence is a state of its own unless `--free-loss` (then it is missing and counts for nothing), and with `--no-ambiguity` multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; its counting convention (`QuartetConvention`: ambiguity policy × `--free-loss`) is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`, `quartet.diploid.pbiv` for `--free-loss`; `quartet.abs.*.pbiv` for absence-as-a-state) because that header has no room for a flag.
|
||||||
|
|
||||||
|
*Correction (2026-11-19).* The first version never counted absence, whatever `--free-loss` said, which contradicted the alignment: without `--free-loss` a genome lacking the family has the state `∅`, so `A A ∅ ∅` separates two pairs yet was rejected. Fixed in `has_bipartition` (index 0 is now counted, and is only ever populated when absence is a state), in the annex (one file per `QuartetConvention`) and in `--sample-stats` (informative sites follow the same rule). The old file names keep the `--free-loss` semantics, which is exactly the old rule, so annexes already built with `--free-loss` stay valid; the sampling measured below used `--free-loss` and is unaffected. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
|
||||||
|
|
||||||
Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list.
|
Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list.
|
||||||
|
|
||||||
|
|||||||
@@ -91,6 +91,17 @@ doc-user-serve: $(MKDOCS)
|
|||||||
clean-doc-user:
|
clean-doc-user:
|
||||||
rm -rf $(DOC_USER_SITE)/
|
rm -rf $(DOC_USER_SITE)/
|
||||||
|
|
||||||
|
# Regenerates the Gitea/Forgejo wiki (a separate repository, checked out next
|
||||||
|
# to this one) from the user documentation. Only rewrites its working tree:
|
||||||
|
# committing and pushing it stays a manual step. Not part of `clean`.
|
||||||
|
DOC_WIKI_DIR := ../obikmer.wiki
|
||||||
|
|
||||||
|
.PHONY: wiki
|
||||||
|
wiki:
|
||||||
|
@command -v pmotransdoc >/dev/null || { echo "pmotransdoc not found in PATH" >&2; exit 1; }
|
||||||
|
@test -d $(DOC_WIKI_DIR)/.git || { echo "$(DOC_WIKI_DIR) is not a git checkout of the wiki" >&2; exit 1; }
|
||||||
|
pmotransdoc $(DOC_USER_DIR) $(DOC_WIKI_DIR)
|
||||||
|
|
||||||
.PHONY: clean
|
.PHONY: clean
|
||||||
clean: clean-doc clean-doc-user
|
clean: clean-doc clean-doc-user
|
||||||
rm -rf $(VENV)
|
rm -rf $(VENV)
|
||||||
|
|||||||
@@ -4,14 +4,13 @@
|
|||||||
|
|
||||||
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)):
|
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)):
|
||||||
|
|
||||||
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition.
|
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition. The number of partitions is always a power of two, so this routing amounts to keeping the low bits of the hash.
|
||||||
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
|
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
|
||||||
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.
|
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. To accumulate these counts, a provisional minimal perfect hash function is built over the distinct kmers of the partition and a temporary count file is filled through it. Both are working files: they are deleted at the end of the build unless `--keep-intermediate` is given. A per-genome kmer frequency spectrum is produced at this stage, from the counts as they are before any abundance filtering.
|
||||||
4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set.
|
4. **Quorum filtering and local assembly.** The kmers of the dereplicated super-kmers are fed one by one into a local de Bruijn graph of the partition. A kmer whose count falls outside the `--min-abundance`/`--max-abundance` range is dropped at this point, and never enters the graph; the super-kmers themselves are not rewritten. The kmers that remain are assembled into unitigs — maximal non-branching runs of the graph — such that every kmer appears exactly once, at one (unitig, offset) location.
|
||||||
5. **Local assembly.** The surviving kmers of each partition are assembled into unitigs — maximal non-branching runs of a local de Bruijn graph — such that every kmer appears exactly once, at one (unitig, offset) location.
|
5. **MPHF and evidence construction.** The final minimal perfect hash function is built over the canonical kmers that survived, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested; the counts come from the counting phase.
|
||||||
6. **MPHF and evidence construction.** A minimal perfect hash function is built over the canonical kmers of each partition, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested.
|
|
||||||
|
|
||||||
Phases 1–5 are independent per partition and run in parallel; phase 6 finalizes each partition once its kmer set is fixed.
|
Within each phase, the partitions are independent of one another and are processed in parallel. The phases themselves follow one another over the whole index, and the index records in its `index.meta` file how far the build has gone (after scatter, after counting, and when fully built). A build that was interrupted can therefore be resumed: the phases already completed are skipped.
|
||||||
|
|
||||||
## Minimal perfect hash function (MPHF)
|
## Minimal perfect hash function (MPHF)
|
||||||
|
|
||||||
@@ -19,38 +18,50 @@ Each partition's surviving kmers are mapped to a dense range of integer slots by
|
|||||||
|
|
||||||
## Evidence: exact vs. approximate
|
## Evidence: exact vs. approximate
|
||||||
|
|
||||||
Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)):
|
Three verification modes exist. The exact and approximate modes are selected at build time (`index --approx`), and any of the three can be reached afterwards with [`convert`](../usage/convert.md):
|
||||||
|
|
||||||
- **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
|
- **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
|
||||||
- **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
|
- **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
|
||||||
|
- **Hybrid** (`convert --hybrid-evidence`): both the fingerprint and the exact evidence are kept. A lookup uses the fingerprint, which is fast and approximate, while the exact evidence remains stored alongside it.
|
||||||
|
|
||||||
## On-disk layout
|
## On-disk layout
|
||||||
|
|
||||||
```
|
```
|
||||||
<index_root>/
|
📁 <index_root>/
|
||||||
index.meta global configuration (k, minimizer size, partition count,
|
├── 📄 index.meta configuration (k, minimizer size, partition count,
|
||||||
evidence mode, whether counts are stored) and genome list/metadata
|
│ evidence mode, counts stored or not), genome list
|
||||||
scatter.done / count.done / index.done build-progress sentinels
|
│ and metadata, build state (empty, scattered,
|
||||||
spectrums/<label>.json per-genome kmer frequency histogram
|
│ counted, indexed)
|
||||||
partitions/
|
├── 📄 .obikmer.lock lock files: commands that read the index share it,
|
||||||
part_00000/ ... part_NNNNN/
|
├── 📄 .obikmer.gate commands that write into it hold it exclusively
|
||||||
index/
|
├── 📁 spectrums/
|
||||||
meta.json number of layers in this partition
|
│ └── 📄 <label>.json per-genome kmer frequency histogram
|
||||||
layer_0/
|
└── 📁 partitions/
|
||||||
unitigs.bin reconstructible kmer sequence data — always kept
|
├── 📁 part_00000/
|
||||||
unitigs.bin.idx random-access index into unitigs.bin (exact evidence only)
|
│ └── 📁 index/
|
||||||
mphf.bin the minimal perfect hash function
|
│ ├── 📁 layer_0/
|
||||||
evidence.bin exact evidence (exact mode only)
|
│ │ ├── 📄 layer_meta.json number of kmers in this layer
|
||||||
fingerprint.bin approximate evidence (approximate mode only)
|
│ │ ├── 📄 unitigs.bin reconstructible kmer sequence data — always kept
|
||||||
counts/ per-genome kmer counts (if counts were requested)
|
│ │ ├── 📄 unitigs.bin.idx random access into unitigs.bin (exact, hybrid)
|
||||||
presence/ per-genome presence/absence bits
|
│ │ ├── 📄 mphf.bin the minimal perfect hash function
|
||||||
layer_1/, layer_2/, ... added by later merges, same internal structure
|
│ │ ├── 📄 evidence.bin exact evidence (exact, hybrid)
|
||||||
|
│ │ ├── 📄 fingerprint.bin approximate evidence (approximate, hybrid)
|
||||||
|
│ │ ├── 📁 counts/ per-genome kmer counts (if requested)
|
||||||
|
│ │ └── 📁 presence/ per-genome presence/absence bits
|
||||||
|
│ └── 📁 layer_1/, layer_2/, … added by later merges, same structure
|
||||||
|
├── 📁 part_00001/
|
||||||
|
│ └── …
|
||||||
|
└── 📁 part_NNNNN/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The build state is stored in `index.meta` and not in separate marker files: it is what allows an interrupted build to resume. The number of layers of a partition is not stored anywhere either; it is found by looking for `layer_0`, `layer_1`, ... until one is missing. A partition that received no kmer at all (which happens with small inputs and many partitions) has no layer on disk.
|
||||||
|
|
||||||
|
The matrix directory of a layer, `counts/` or `presence/`, exists in three forms, which hold the same information. A freshly built index stores one file per genome, with a small metadata file (for counts, `col_000000.pciv`, `col_000001.pciv`, ... and `meta.json`). The dense packed form, produced by [`pack`](../usage/pack.md), stores the whole matrix in a single file. The sparse packed form, produced by `pack --sparse` and, by default, by [`merge`](../usage/merge.md), stores for each kmer only the list of genomes that carry it, with the lists shared between kmers deduplicated (for counts, the directory then holds files such as `dict_values.bin`, `singleton.pfiv`, `multi.pfiv` and `sparse_meta.json`). Which form a layer uses is detected from the files found in the directory.
|
||||||
|
|
||||||
`unitigs.bin` is the only file from which the indexed kmer content can be fully recovered; it is always retained. Every other file (MPHF, evidence, counts) is derived from it.
|
`unitigs.bin` is the only file from which the indexed kmer content can be fully recovered; it is always retained. Every other file (MPHF, evidence, counts) is derived from it.
|
||||||
|
|
||||||
A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
|
A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. The genomes of the merged sources are added as new columns to the count/presence data of the existing layers; only genuinely new kmer content, that is kmers absent from every existing layer of the partition, is assembled into a new layer. A kmer therefore belongs to exactly one layer of its partition. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
|
||||||
|
|
||||||
Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`convert`](../usage/convert.md) one of the sources first if needed.
|
Sources merged together must share the same kmer size, minimizer size and partition count; any mismatch is rejected rather than silently reconciled. The evidence mode, on the other hand, may differ between sources: as soon as one source is approximate or hybrid, the destination is chosen among the approximate or hybrid sources, and the merged index takes its evidence mode. A count merge additionally requires every source to store counts. Finally, an index whose matrices are already packed, which includes any merge output, can be merged in as a source but not used as the destination, so at least one source must still be unpacked.
|
||||||
|
|
||||||
`obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time.
|
[`pack`](../usage/pack.md) consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time; with `--sparse`, it also switches to the sparse form described above.
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# VCF pseudo-alignment
|
||||||
|
|
||||||
|
[`phylo --vcf`](../usage/phylo.md) writes the sites of a SNP pseudo-alignment as a VCF file, `<prefix>_alignment.vcf`. This page describes exactly what that file contains, because the k-mer families it is built from are not ordinary genomic variants and a few conventions follow from that.
|
||||||
|
|
||||||
|
## Why a VCF, and what a site is
|
||||||
|
|
||||||
|
The pseudo-alignment (see [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md)) has one column per *family*: the up to four k-mers that share all their bases except the central one. A genome that carries one of these k-mers shows a particular base at the centre, and the columns of the alignment are those central bases, genome by genome. The FASTA output keeps this as a matrix of letters. The VCF output lists the same columns as sites with alleles and genotypes, which is the form that variant-oriented tools (`bcftools`, population-genetics software, genotype viewers) expect.
|
||||||
|
|
||||||
|
Nothing here comes from a reference genome. The file has no real coordinates, and the sites are not located on a chromosome: the positions below are labels that make the file valid and reproducible, not map positions.
|
||||||
|
|
||||||
|
## Producing the file
|
||||||
|
|
||||||
|
```
|
||||||
|
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
|
||||||
|
```
|
||||||
|
|
||||||
|
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, (the sibling annex it reads is built automatically if missing; see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
An illustrative excerpt (columns are tab-separated in the file):
|
||||||
|
|
||||||
|
```
|
||||||
|
##fileformat=VCFv4.5
|
||||||
|
##source=obikmer
|
||||||
|
##contig=<ID=obikmer_0_0>
|
||||||
|
##contig=<ID=obikmer_0_1>
|
||||||
|
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype (diploid)">
|
||||||
|
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT idx0 idx1 idx2 idx3
|
||||||
|
obikmer_0_0 440 CGATTNGTTAA A G . . . GT 0/0 1/1 0/0 0/0
|
||||||
|
obikmer_0_0 1511 CACGANTGTAG A G,T . . . GT 0/0 0/0 1/1 2/2
|
||||||
|
obikmer_0_1 9 TTACGNAGCTA C * . . . GT 0/0 0/0 1/1 0/0
|
||||||
|
```
|
||||||
|
|
||||||
|
There is one sample column per genome that takes part in the draw, named after the genome label, in the index's own order. A genome removed with `--exclude-genome` or `--min-shared-family` has no column at all. `QUAL`, `FILTER` and `INFO` are always `.`: obikmer applies no variant filter and attaches no annotation, and leaving them empty keeps a file of millions of sites compact.
|
||||||
|
|
||||||
|
## Coordinates: `CHROM` and `POS`
|
||||||
|
|
||||||
|
An index is organised in partitions, and each partition holds one or more layers of k-mers (see [Index construction and on-disk layout](index_layout.md)). A family always belongs to exactly one layer of one partition, and the families of a layer are visited in a fixed order. The VCF uses that structure as coordinates:
|
||||||
|
|
||||||
|
- `CHROM` is `obikmer_<partition>_<layer>`, one virtual contig per layer of each partition in which at least one site was drawn;
|
||||||
|
- `POS` is the rank of the family inside that layer, starting at 1.
|
||||||
|
|
||||||
|
For a given index and a given sibling annex, a family therefore always receives the same `CHROM` and `POS`, whatever the draw, the options or the number of sites requested. Two runs on the same index can be compared position by position. This stability is lost if the sibling annex is rebuilt, or if the index is rebuilt, merged or filtered, because the families are then enumerated again. Positions are never comparable between two different indexes; use `ID` for that. Even rebuilding an index from the very same genomes may enumerate the families in another order: on a small test index built twice from identical input, the family `CGATTNGTTAA` had `POS` 440 the first time and 371 the second, with the same `ID`.
|
||||||
|
|
||||||
|
The file is sorted by partition, then layer, then position, which is also the order of the `##contig` lines, so tools that require a sorted file accept it directly. Positions are sparse: only the drawn families are written.
|
||||||
|
|
||||||
|
## Identifier: `ID`
|
||||||
|
|
||||||
|
The `ID` column holds a label for the family that does not depend on the index. It is built from one of the family's k-mers by replacing its central base with `N` (for example `CGATTNGTTAA` for k = 11), after choosing, among the family's possible orientations, the one that sorts first. All the members of a family, whichever strand they were read on, give the same label.
|
||||||
|
|
||||||
|
This is what makes two files comparable even when they come from different indexes: the same family has the same `ID` in both, while its `CHROM` and `POS` will generally differ. It is also unique within a file, since two different families never share a label.
|
||||||
|
|
||||||
|
## Alleles: `REF` and `ALT`
|
||||||
|
|
||||||
|
The alleles are the bases carried at the centre of the family's k-mers, as in the FASTA. Two points are worth knowing.
|
||||||
|
|
||||||
|
First, the base is the central base of the member's *canonical* form (the smaller of the k-mer and its reverse complement). That is not necessarily the strand of the `ID` label, so the letters should not be matched to the `ID` sequence by position; they are labels for the allele, not a reading of that sequence.
|
||||||
|
|
||||||
|
Second, `REF` carries no meaning of reference: it is simply the most frequent allele among the genomes of the draw, each allele copy counted once, with the alphabetical order A < C < G < T deciding ties. `ALT` lists the remaining observed bases in the order A, C, G, T, separated by commas. There is no option to choose a reference genome.
|
||||||
|
|
||||||
|
### Absence: the `<ABS>` allele
|
||||||
|
|
||||||
|
A genome may carry none of the family's k-mers. Without `--free-loss` this absence is treated as a real state (the family was lost, or is absent from that genome), which the FASTA shows as `-`. In the VCF it becomes one more allele, the symbolic allele `<ABS>`, listed last in `ALT` and declared in the header by `##ALT=<ID=ABS,Description="Family absent from the genome">`. A genome without the family receives the genotype of `<ABS>`; for a site with `ALT` equal to `G,T,<ABS>` that is `3/3`. A site that differs between genomes only by presence versus absence has `ALT` equal to `<ABS>`.
|
||||||
|
|
||||||
|
The VCF specification gives the plain character `*` a meaning of its own (an allele missing because of an overlapping deletion described by another record), which is not the situation here, so it is deliberately not used. `<ABS>` is a free-form symbolic identifier, outside the reserved structural-variant names (`DEL`, `INS`, `DUP`, `INV`, `CNV`) and outside `<*>`, which belongs to reference blocks. Tools that only look at sequence alleles treat the site as having an alternative that is not a base.
|
||||||
|
|
||||||
|
With `--free-loss`, absence is declared missing data instead: `<ABS>` never appears, and the genomes concerned get a missing genotype.
|
||||||
|
|
||||||
|
## Genotypes
|
||||||
|
|
||||||
|
Every genome gets one `GT` value per site. The ploidy of these values is a convention chosen from the options, and the header says which (`Genotype (diploid)` or `Genotype (haploid)`).
|
||||||
|
|
||||||
|
| Cell in the alignment | Diploid (default, `--only-diploid`) | Haploid (`--no-ambiguity`) |
|
||||||
|
|---|---|---|
|
||||||
|
| one base `X` | `X/X`, for example `0/0` or `1/1` | `X`, for example `0` |
|
||||||
|
| two bases (IUPAC `R Y S W K M`) | the heterozygote, for example `0/1` | does not occur |
|
||||||
|
| three or four bases (`B D H V N`) | `./.` (not expressible) | does not occur |
|
||||||
|
| absent family (`<ABS>`) | `<ABS>/<ABS>`, for example `3/3` | the index of `<ABS>` |
|
||||||
|
| missing (`?`) | `./.` | `.` |
|
||||||
|
|
||||||
|
The two allele indices of a diploid genotype are always written in increasing order (`0/1`, never `1/0`), and the separator is `/` because no phasing information exists.
|
||||||
|
|
||||||
|
Why diploid by default? A genome can carry several forms of the same family, which the alignment records as an IUPAC ambiguity code. A diploid genotype can express up to two forms, and `--only-diploid` removes the genomes that would need more (three or four forms) before the sites are selected, so that no `./.` comes from them. `--no-ambiguity` goes further and treats every such genome as missing, which makes all genotypes haploid.
|
||||||
|
|
||||||
|
This is a convention, not a measurement. The file records which forms a genome carries, not how many copies of each: `A/A/G` and `A/G/G` both give the code `R` and the genotype `0/1`. Likewise, a genome with two forms may be heterozygous at one locus, or it may carry two similar regions of its genome (paralogues, repeats); the file cannot tell these apart. Do not read the genotypes as calls from sequencing reads.
|
||||||
|
|
||||||
|
## Which sites are written
|
||||||
|
|
||||||
|
Only the families that survive the draw and its masking appear. Within those:
|
||||||
|
|
||||||
|
- a site needs at least one genome with a base or (in diploid mode) a two-base code, because `REF` is a base; a site where every genome is missing, absent or has an ambiguity code too wide for the mode is left out of the VCF while remaining in the FASTA;
|
||||||
|
- the log line at the end of the run gives the number of sites written out of the number drawn, so the difference is visible.
|
||||||
|
|
||||||
|
A site is selected because its genomes show at least two distinct states, and absence counts as a state. A site that differs between genomes only by presence versus absence is therefore a legitimate site, written with `ALT` equal to `<ABS>`.
|
||||||
|
|
||||||
|
## Using the file with other tools
|
||||||
|
|
||||||
|
- The contigs are virtual and have no declared length, and the file refers to no reference. Tools that need one (annotation, liftover) cannot use it; tools that only read sites and genotypes (`bcftools view`, `query`, `stats`, population-genetics software, PLINK-style converters) can.
|
||||||
|
- If a tool rejects multiallelic sites or the `<ABS>` allele, `bcftools view -m2 -M2` keeps biallelic sites only, and `bcftools view -e 'ALT="<ABS>"'` removes the sites involving absence. The file as written was read by `bcftools` and indexed by `tabix` without complaint on a small test index, in both diploid and haploid modes. `bcftools stats` counts the sites with `<ABS>` among the "others" (symbolic alleles) rather than as SNPs.
|
||||||
|
- If a tool expects haploid data, run `phylo` with `--no-ambiguity` rather than converting the genotypes afterwards.
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [`phylo`](../usage/phylo.md): all the options that shape the draw.
|
||||||
|
- [Sampling design](../theory/phylogeny/snp_based/sampling_design.md): how the sites are drawn.
|
||||||
|
- [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md): what a family is.
|
||||||
@@ -857,3 +857,235 @@
|
|||||||
doi = "10.1214/aop/1176996461",
|
doi = "10.1214/aop/1176996461",
|
||||||
issn = "0091-1798,2168-894X"
|
issn = "0091-1798,2168-894X"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Karlin1997-lm,
|
||||||
|
title = "{Compositional differences within and between eukaryotic genomes}",
|
||||||
|
author = "Karlin, S and Mrázek, J",
|
||||||
|
journal = "Proceedings of the National Academy of Sciences of the United
|
||||||
|
States of America",
|
||||||
|
publisher = "Proceedings of the National Academy of Sciences",
|
||||||
|
volume = 94,
|
||||||
|
number = 19,
|
||||||
|
pages = "10227--10232",
|
||||||
|
abstract = "Eukaryotic genome similarity relationships are inferred using
|
||||||
|
sequence information derived from large aggregates of genomic
|
||||||
|
sequences. Comparisons within and between species sample
|
||||||
|
sequences are based on the profile of dinucleotide relative
|
||||||
|
abundance values (The profile is ρ* XY = f * XY / f * X f * Y for
|
||||||
|
all XY , where f * X denotes the frequency of the nucleotide X
|
||||||
|
and f * XY denotes the frequency of the dinucleotide XY , both
|
||||||
|
computed from the sequence concatenated with its inverted
|
||||||
|
complement). Previous studies with respect to prokaryotes and
|
||||||
|
this study document that profiles of different DNA sequence
|
||||||
|
samples (sample size ≥50 kb) from the same organism are generally
|
||||||
|
much more similar to each other than they are to profiles from
|
||||||
|
other organisms, and that closely related organisms generally
|
||||||
|
have more similar profiles than do distantly related organisms.
|
||||||
|
On this basis we refer to the collection {ρ* XY } as the genome
|
||||||
|
signature . This paper identifies ρ* XY extremes and compares
|
||||||
|
genome signature differences for a diverse range of eukaryotic
|
||||||
|
species. Interpretations on the mechanisms maintaining these
|
||||||
|
profile differences center on genome-wide replication, repair,
|
||||||
|
DNA structures, and context-dependent mutational biases. It is
|
||||||
|
also observed that mitochondrial genome signature differences
|
||||||
|
between species parallel the corresponding nuclear genome
|
||||||
|
signature differences despite large differences between
|
||||||
|
corresponding mitochondrial and nuclear signatures. The genome
|
||||||
|
signature differences also have implications for contrasts
|
||||||
|
between rodents and other mammals, and between monocot and dicot
|
||||||
|
plants, as well as providing evidence for similarities among
|
||||||
|
fungi and the diversity of protists.",
|
||||||
|
month = sep,
|
||||||
|
year = 1997,
|
||||||
|
url = "http://dx.doi.org/10.1073/pnas.94.19.10227",
|
||||||
|
doi = "10.1073/pnas.94.19.10227",
|
||||||
|
pmc = "PMC23344",
|
||||||
|
pmid = 9294192,
|
||||||
|
issn = "0027-8424,1091-6490",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Karlin1997-vm,
|
||||||
|
title = "{Compositional biases of bacterial genomes and evolutionary
|
||||||
|
implications}",
|
||||||
|
author = "Karlin, S and Mrázek, J and Campbell, A M",
|
||||||
|
journal = "Journal of Bacteriology",
|
||||||
|
publisher = "American Society for Microbiology",
|
||||||
|
volume = 179,
|
||||||
|
number = 12,
|
||||||
|
pages = "3899--3913",
|
||||||
|
abstract = "We compare and contrast genome-wide compositional biases and
|
||||||
|
distributions of short oligonucleotides across 15 diverse
|
||||||
|
prokaryotes that have substantial genomic sequence collections.
|
||||||
|
These include seven complete genomes (Escherichia coli,
|
||||||
|
Haemophilus influenzae, Mycoplasma genitalium, Mycoplasma
|
||||||
|
pneumoniae, Synechocystis sp. strain PCC6803, Methanococcus
|
||||||
|
jannaschii, and Pyrobaculum aerophilum). A key observation
|
||||||
|
concerns the constancy of the dinucleotide relative abundance
|
||||||
|
profiles over multiple 50-kb disjoint contigs within the same
|
||||||
|
genome. (The profile is rhoXY* = fXY*/fX*fY* for all XY, where
|
||||||
|
fX* denotes the frequency of the nucleotide X and fY* denotes the
|
||||||
|
frequency of the dinucleotide XY, both computed from the sequence
|
||||||
|
concatenated with its inverted complementary sequence.) On the
|
||||||
|
basis of this constancy, we refer to the collection [rhoXY*] as
|
||||||
|
the genome signature. We establish that the differences between
|
||||||
|
[rhoXY*] vectors of 50-kb sample contigs of different genomes
|
||||||
|
virtually always exceed the differences between those of the same
|
||||||
|
genomes. Various di- and tetranucleotide biases are identified.
|
||||||
|
In particular, we find that the dinucleotide CpG=CG is
|
||||||
|
underrepresented in many thermophiles (e.g., M. jannaschii,
|
||||||
|
Sulfolobus sp., and M. thermoautotrophicum) but overrepresented
|
||||||
|
in halobacteria. TA is broadly underrepresented in prokaryotes
|
||||||
|
and eukaryotes, but normal counts appear in Sulfolobus and P.
|
||||||
|
aerophilum sequences. More than for any other bacterial genome,
|
||||||
|
palindromic tetranucleotides are underrepresented in H.
|
||||||
|
influenzae. The M. jannaschii sequence is unprecedented in its
|
||||||
|
extreme underrepresentation of CTAG tetranucleotides and in the
|
||||||
|
anomalous distribution of CTAG sites around the genome.
|
||||||
|
Comparative analysis of numbers of long tetranucleotide
|
||||||
|
microsatellites distinguishes H. influenzae. Dinucleotide
|
||||||
|
relative abundance differences between bacterial sequences are
|
||||||
|
compared. For example, in these assessments of differences, the
|
||||||
|
cyanobacteria Synechocystis, Synechococcus, and Anabaena do not
|
||||||
|
form a coherent group and are as far from each other as general
|
||||||
|
gram-negative sequences are from general gram-positive sequences.
|
||||||
|
The difference of M. jannaschii from low-G+C gram-positive
|
||||||
|
proteobacteria is one-half of the difference from gram-negative
|
||||||
|
proteobacteria. Interpretations and hypotheses center on the role
|
||||||
|
of the genome signature in highlighting similarities and
|
||||||
|
dissimilarities across different classes of prokaryotic species,
|
||||||
|
possible mechanisms underlying the genome signature, the form and
|
||||||
|
level of genome compositional flux, the use of the genome
|
||||||
|
signature as a chronometer of molecular phylogeny, and
|
||||||
|
implications with respect to the three putative eubacterial,
|
||||||
|
archaeal, and eukaryote domains of life and to the origin and
|
||||||
|
early evolution of eukaryotes.",
|
||||||
|
month = jun,
|
||||||
|
year = 1997,
|
||||||
|
url = "http://dx.doi.org/10.1128/jb.179.12.3899-3913.1997",
|
||||||
|
doi = "10.1128/jb.179.12.3899-3913.1997",
|
||||||
|
pmc = "PMC179198",
|
||||||
|
pmid = 9190805,
|
||||||
|
issn = "0021-9193,1098-5530",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Karlin1995-lt,
|
||||||
|
title = "{Dinucleotide relative abundance extremes: a genomic signature}",
|
||||||
|
author = "Karlin, S and Burge, C",
|
||||||
|
journal = "Trends in Genetics: TIG",
|
||||||
|
publisher = "Elsevier BV",
|
||||||
|
volume = 11,
|
||||||
|
number = 7,
|
||||||
|
pages = "283--290",
|
||||||
|
abstract = "Early biochemical experiments established that the set of
|
||||||
|
dinucleotide odds ratios or 'general design' is a remarkably
|
||||||
|
stable property of the DNA of an organism, which is essentially
|
||||||
|
the same in protein-coding DNA, bulk genomic DNA, and in
|
||||||
|
different renaturation rate and density gradient fractions of
|
||||||
|
genomic DNA in many organisms. Analysis of currently available
|
||||||
|
genomic sequence data has extended these earlier results, showing
|
||||||
|
that the general designs of disjoint samples of a genome are
|
||||||
|
substantially more similar to each other than to those of
|
||||||
|
sequences from other organisms and that closely related organisms
|
||||||
|
have similar general designs. From this perspective, the set of
|
||||||
|
dinucleotide odds ratio (relative abundance) values constitute a
|
||||||
|
signature of each DNA genome, which can discriminate between
|
||||||
|
sequences from different organisms. Dinucleotide-odds ratio
|
||||||
|
values appear to reflect not only the chemistry of dinucleotide
|
||||||
|
stacking energies and base-step conformational preferences, but
|
||||||
|
also the species-specific properties of DNA modification,
|
||||||
|
replication and repair mechanisms.",
|
||||||
|
month = jul,
|
||||||
|
year = 1995,
|
||||||
|
url = "http://dx.doi.org/10.1016/s0168-9525(00)89076-9",
|
||||||
|
doi = "10.1016/s0168-9525(00)89076-9",
|
||||||
|
pmid = 7482779,
|
||||||
|
issn = "0168-9525,1362-4555",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Karlin1994-on,
|
||||||
|
title = "{Heterogeneity of genomes: measures and values}",
|
||||||
|
author = "Karlin, S and Ladunga, I and Blaisdell, B E",
|
||||||
|
journal = "Proceedings of the National Academy of Sciences of the United
|
||||||
|
States of America",
|
||||||
|
publisher = "Proceedings of the National Academy of Sciences",
|
||||||
|
volume = 91,
|
||||||
|
number = 26,
|
||||||
|
pages = "12837--12841",
|
||||||
|
abstract = "Genomic homogeneity is investigated for a broad base of DNA
|
||||||
|
sequences in terms of dinucleotide relative abundance distances
|
||||||
|
(abbreviated delta-distances) and of oligonucleotide
|
||||||
|
compositional extremes. It is shown that delta-distances between
|
||||||
|
different genomic sequences in the same species are low, only
|
||||||
|
about 2 or 3 times the distance found in random DNA, and are
|
||||||
|
generally smaller than the between-species delta-distances.
|
||||||
|
Extremes in short oligonucleotides include underrepresentation of
|
||||||
|
TpA and overrepresentation of GpC in most temperate bacteriophage
|
||||||
|
sequences; underrepresentation of CTAG in most eubacterial
|
||||||
|
genomes; underrepresentation of GATC in most bacteriophage; CpG
|
||||||
|
suppression in vertebrates, in all animal mitochondrial genomes,
|
||||||
|
and in many thermophilic bacterial sequences; and
|
||||||
|
overrepresentation of GpG/CpC in all animal mitochondrial sets
|
||||||
|
and chloroplast genomes. Interpretations center on DNA structures
|
||||||
|
(dinucleotide stacking energies, DNA curvature and superhelicity,
|
||||||
|
nucleosome organization), context-dependent mutational events,
|
||||||
|
methylation effects, and processes of replication and repair.",
|
||||||
|
month = dec,
|
||||||
|
year = 1994,
|
||||||
|
url = "http://dx.doi.org/10.1073/pnas.91.26.12837",
|
||||||
|
doi = "10.1073/pnas.91.26.12837",
|
||||||
|
pmc = "PMC45535",
|
||||||
|
pmid = 7809131,
|
||||||
|
issn = "0027-8424,1091-6490",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Bennett2013-cl,
|
||||||
|
title = "{Small, smaller, smallest: the origins and evolution of ancient
|
||||||
|
dual symbioses in a Phloem-feeding insect}",
|
||||||
|
author = "Bennett, Gordon M and Moran, Nancy A",
|
||||||
|
journal = "Genome Biology and Evolution",
|
||||||
|
publisher = "Oxford University Press (OUP)",
|
||||||
|
volume = 5,
|
||||||
|
number = 9,
|
||||||
|
pages = "1675--1688",
|
||||||
|
abstract = "Many insects rely on bacterial symbionts with tiny genomes
|
||||||
|
specialized for provisioning nutrients lacking in host diets.
|
||||||
|
Xylem sap and phloem sap are both deficient as insect diets, but
|
||||||
|
differ dramatically in nutrient content, potentially affecting
|
||||||
|
symbiont genome evolution. For sap-feeding insects, sequenced
|
||||||
|
symbiont genomes are available only for phloem-feeding examples
|
||||||
|
from the suborder Sternorrhyncha and xylem-feeding examples from
|
||||||
|
the suborder Auchenorrhyncha, confounding comparisons. We
|
||||||
|
sequenced genomes of the obligate symbionts, Sulcia muelleri and
|
||||||
|
Nasuia deltocephalinicola, of the phloem-feeding pest insect,
|
||||||
|
Macrosteles quadrilineatus (Auchenorrhyncha: Cicadellidae). Our
|
||||||
|
results reveal that Nasuia-ALF has the smallest bacterial genome
|
||||||
|
yet sequenced (112 kb), and that the Sulcia-ALF genome (190 kb)
|
||||||
|
is smaller than that of Sulcia in other insect lineages.
|
||||||
|
Together, these symbionts retain the capability to synthesize the
|
||||||
|
10 essential amino acids, as observed for several symbiont pairs
|
||||||
|
from xylem-feeding Auchenorrhyncha. Nasuia retains genes enabling
|
||||||
|
synthesis of two amino acids, DNA replication, transcription, and
|
||||||
|
translation. Both symbionts have lost genes underlying ATP
|
||||||
|
synthesis through oxidative phosphorylation, possibly as a
|
||||||
|
consequence of the enriched sugar content of phloem. Shared
|
||||||
|
genomic features, including reassignment of the UGA codon from
|
||||||
|
Stop to tryptophan, and phylogenetic results suggest that
|
||||||
|
Nasuia-ALF is most closely related to Zinderia, the
|
||||||
|
betaproteobacterial symbiont of spittlebugs. Thus,
|
||||||
|
Nasuia/Zinderia and Sulcia likely represent ancient associates
|
||||||
|
that have co-resided in hosts since the divergence of leafhoppers
|
||||||
|
and spittlebugs >200 Ma, and possibly since the origin of the
|
||||||
|
Auchenorrhyncha, >260 Ma.",
|
||||||
|
month = aug,
|
||||||
|
year = 2013,
|
||||||
|
url = "http://dx.doi.org/10.1093/gbe/evt118",
|
||||||
|
keywords = "Nasuia deltocephalinicola; Sulcia muelleri; gene loss; genome
|
||||||
|
evolution; leafhopper; nutritional symbioses",
|
||||||
|
doi = "10.1093/gbe/evt118",
|
||||||
|
pmc = "PMC3787670",
|
||||||
|
pmid = 23918810,
|
||||||
|
issn = "1759-6653",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
@@ -1,6 +1,12 @@
|
|||||||
# K-mer size
|
# K-mer size
|
||||||
|
|
||||||
A k-mer is a DNA sequence of length $k$. OBIKmer is designed to index large k-mers. Here, *large* refers to a k-mer length for which the probability of observing the same k-mer in several genomes by chance alone is sufficiently small. The k-mer length must therefore be large enough that, when a k-mer is observed in several genomes of the dataset, the probability that this shared occurrence is due to chance alone is sufficiently low for the match to be considered evidence of homology.
|
A k-mer is a DNA sequence of length $k$. OBIKmer is designed to index
|
||||||
|
large k-mers. Here, *large* refers to a k-mer length for which the
|
||||||
|
probability of observing the same k-mer in several genomes by chance
|
||||||
|
alone is sufficiently small. The k-mer length must therefore be large
|
||||||
|
enough that, when a k-mer is observed in several genomes of the dataset,
|
||||||
|
the probability that this shared occurrence is due to chance alone is
|
||||||
|
sufficiently low for the match to be considered evidence of homology.
|
||||||
|
|
||||||
OBIKmer restricts $k$ to odd values in the range
|
OBIKmer restricts $k$ to odd values in the range
|
||||||
|
|
||||||
@@ -8,409 +14,296 @@ $$
|
|||||||
11 \leq k \leq 31.
|
11 \leq k \leq 31.
|
||||||
$$
|
$$
|
||||||
|
|
||||||
These limits result from two different considerations. The upper bound is imposed by the representation used by OBIKmer, whereas the lower bound defines a practical range of k-mer lengths for genome-scale sequence comparisons. This range covers most practical applications: the lower limit is well below the k-mer size that can be considered large, even for the smallest bacterial genomes, whereas the upper limit provides sufficient specificity for most applications, including those involving very large genomes.
|
These limits result from two different considerations. The upper bound
|
||||||
|
is imposed by the representation used by OBIKmer, whereas the lower
|
||||||
|
bound defines a practical range of k-mer lengths for genome-scale
|
||||||
|
sequence comparisons. This range covers most practical applications: the
|
||||||
|
lower limit is well below the k-mer size that can be considered large,
|
||||||
|
even for the smallest bacterial genomes, whereas the upper limit
|
||||||
|
provides sufficient specificity for most applications, including those
|
||||||
|
involving very large genomes.
|
||||||
|
|
||||||
## Representation limit
|
## Representation limit
|
||||||
|
|
||||||
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer requires $2k$ bits. To ensure computational efficiency, OBIKmer limits the representation of k-mers to a single 64-bit word, thereby restricting their length.
|
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer
|
||||||
|
requires $2k$ bits. To ensure computational efficiency, OBIKmer limits
|
||||||
|
the representation of k-mers to a single 64-bit word, thereby
|
||||||
|
restricting their length.
|
||||||
|
|
||||||
$$
|
$$
|
||||||
2k \leq 64,
|
2k \leq 64,
|
||||||
$$
|
$$
|
||||||
|
|
||||||
This gives the hard representation limit: $k \leq 32.$ Furthermore, OBIKmer only considers odd values of $k$, meaning the largest supported k-mer size is $31$.
|
This gives the hard representation limit: $k \leq 32.$ Furthermore,
|
||||||
|
OBIKmer only considers odd values of $k$, meaning the largest supported
|
||||||
### K-mer specificity
|
k-mer size is $31$.
|
||||||
|
|
||||||
One simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are $4^k$ possible k-mers, so a particular k-mer is expected to occur once every $4^k$ positions on average. In a sequence of length $G$, the expected number of occurrences of a particular k-mer is approximately $G \cdot 4^{-k}$. When k is large enough to ensure $4^k >> G$, the chance of a second occurrence of the same k-mer becomes close to zero. Consequently, observing the same k-mer twice can only be explained by a homology relationship, except for low-complexity k-mers, which are overrepresented in genomes.
|
|
||||||
|
|
||||||
### Estimating False discovery rate on homology inference
|
|
||||||
|
|
||||||
OBIKmer uses k-mers long enough that their occurrence in multiple indexed genomes provides evidence of homology. The corresponding false discovery rate (FDR) can be expressed in terms of the probability of observing a k-mer in at least $R$ of the $S$ indexed genomes under a purely random model.
|
|
||||||
|
|
||||||
#### The random model
|
|
||||||
|
|
||||||
The random model, or null hypothesis, assumes that the $S$ genomes are unrelated and that each genome is an independent random sequence in which each nucleotide occurs with equal probability. Rejecting this hypothesis for a given k-mer indicates that its presence in multiple genomes is unlikely to be explained by chance alone, providing evidence for homology.
|
|
||||||
|
|
||||||
For a genome of size $G$, the number of occurrences of a given canonical k-mer $K$ is approximated by a Poisson distribution with parameter
|
|
||||||
|
|
||||||
$$ \lambda = 2\frac{G}{4^k}, $$
|
|
||||||
|
|
||||||
which is also the expected number of occurrences under the random model. The factor of two accounts for k-mer canonicalisation, which halves the k-mer space when $k$ is odd.
|
|
||||||
|
|
||||||
The probability of observing at least one occurrence of this k-mer is therefore
|
|
||||||
|
|
||||||
$$
|
|
||||||
\begin{aligned}
|
|
||||||
P(n_K > 0) &= 1-e^{-\lambda} \\
|
|
||||||
&= 1-e^{-2G/4^k}.
|
|
||||||
\end{aligned}
|
|
||||||
$$
|
|
||||||
|
|
||||||
Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
|
|
||||||
|
|
||||||
|
|
||||||
$$
|
|
||||||
P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right)^{S-j}.
|
|
||||||
$$
|
|
||||||
|
|
||||||
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
|
|
||||||
|
|
||||||
$$
|
|
||||||
p = 1-e^{-2G/4^k}.
|
|
||||||
$$
|
|
||||||
|
|
||||||
Different genome sizes
|
|
||||||
|
|
||||||
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
|
|
||||||
|
|
||||||
$$ \bar G = \frac{1}{S}\sum_{i=1}^{S}G_i. $$
|
|
||||||
|
|
||||||
This gives a common probability
|
|
||||||
|
|
||||||
$$ \bar p = 1-e^{-2\bar G/4^k}, $$
|
|
||||||
|
|
||||||
and the number of genomes containing the k-mer can then be approximated by a binomial distribution:
|
|
||||||
|
|
||||||
$$ P(X\geq r)=\sum_{j=r}^{S}\binom{S}{j}\bar p^j(1-\bar p)^{S-j}. $$
|
|
||||||
|
|
||||||
This approximation is conservative for the upper tail. Hoeffding [-@Hoeffding1956-bx, p. 717, Eq. 26] shows that, for a fixed mean, the binomial upper tail is at least as large as the corresponding Poisson-binomial tail for
|
|
||||||
|
|
||||||
$$ r\geq S\bar p+1. $$
|
|
||||||
|
|
||||||
Using the mean genome size therefore provides a conservative estimate of the FDR.
|
|
||||||
|
|
||||||
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
|
|
||||||
|
|
||||||
### Exact tail probability
|
|
||||||
|
|
||||||
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
|
|
||||||
|
|
||||||
Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
|
|
||||||
|
|
||||||
$$
|
|
||||||
\left \{
|
|
||||||
\begin{aligned}
|
|
||||||
d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\
|
|
||||||
d_0&=1
|
|
||||||
\end{aligned}
|
|
||||||
\right .
|
|
||||||
$$
|
|
||||||
|
|
||||||
before processing the first genome.
|
|
||||||
|
|
||||||
The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then
|
|
||||||
|
|
||||||
$$
|
|
||||||
P(X\geq r)=1-\sum_{j=0}^{r-1}d_j.
|
|
||||||
$$
|
|
||||||
|
|
||||||
Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$.
|
|
||||||
|
|
||||||
When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean
|
|
||||||
|
|
||||||
$$
|
|
||||||
\lambda=\sum_{i=1}^{S}p_i.
|
|
||||||
$$
|
|
||||||
|
|
||||||
The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe].
|
|
||||||
|
|
||||||
|
|
||||||
*******
|
|
||||||
|
|
||||||
This probability decreases by a factor of four for every additional nucleotide in the k-mer. Conversely, it increases linearly with the amount of sequence searched.
|
|
||||||
|
|
||||||
This simple model already shows why very short k-mers provide limited specificity for genome-scale analyses. With $k=11$,
|
|
||||||
|
|
||||||
$$
|
|
||||||
4^{11}\simeq4.2\times10^6,
|
|
||||||
$$
|
|
||||||
|
|
||||||
which is a small combinatorial space even compared with the size of very small bacterial genomes. In contrast,
|
|
||||||
|
|
||||||
$$
|
|
||||||
4^{31}\simeq4.6\times10^{18},
|
|
||||||
$$
|
|
||||||
|
|
||||||
which is vastly larger than the number of k-mer positions even in very large genomes. Thus, increasing $k$ from 11 to 31 provides an enormous increase in sequence specificity.
|
|
||||||
|
|
||||||
The estimate above is intentionally simplistic. Real genomes do not consist of independent, equiprobable nucleotides: nucleotide composition and sequence dependencies affect the probability of observing particular words. More accurate estimates of the probability of observing matches by chance therefore take sequence length and nucleotide composition into account.
|
|
||||||
|
|
||||||
Karlin and Ost provide an analytical framework for this problem based on the distribution of the longest exact match expected between random sequences. If $L_{\max}$ denotes the length of the longest seed observed in random sequences of given size and nucleotide composition, the corresponding seed-level probability is
|
|
||||||
|
|
||||||
$$
|
|
||||||
P_{\mathrm{seeds}}(L)=P(L_{\max}\geq L).
|
|
||||||
$$
|
|
||||||
|
|
||||||
Conversely, for a chosen false-positive probability $P$, one can determine the minimum seed length $L_{\min}$ such that matches of length $L_{\min}$ or greater are not expected to occur by chance. This is the approach used by Repseek to determine statistically significant seed lengths.
|
|
||||||
|
|
||||||
The simple $G/4^k$ estimate and the Karlin–Ost statistics therefore describe the same fundamental effect at different levels of approximation: the probability of a chance match decreases rapidly with k-mer length, while the amount of sequence being searched increases the opportunity for such matches.
|
|
||||||
|
|
||||||
Consequently, the appropriate value of $k$ depends on the total amount of sequence being compared and its nucleotide composition. Choosing $k$ appropriately allows the false-positive rate of homology inference to be controlled.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Mycoplasma genitalium* having a size $G=580\,Mb$ [@Fraser1995-jc], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
|
|
||||||
|
|
||||||
$$
|
|
||||||
\begin{aligned}
|
|
||||||
P(C\geq1) &= 1- e^{-\frac{580.000^2}{2 \cdot 4^{11}}} \\
|
|
||||||
&\simeq 1
|
|
||||||
\end{aligned}
|
|
||||||
$$
|
|
||||||
|
|
||||||
, whereas 21-mers already provide a space of several trillion possible sequences.
|
|
||||||
|
|
||||||
Plant genomes illustrate the opposite situation. Plant genome sizes span several orders of magnitude, from the smallest angiosperm genomes present in the carnivorous plans genus [@Fleischmann2014-ed] about 60 Mb to the largest one 160 Gb in Tmesipteris oblanceolata, a New Caledonian fork fern [@Fernandez2024-kd], and large genomes often contain substantial amounts of repeated sequence. Even at $k=31$, however, the theoretical k-mer space remains vastly larger than any individual genome. At these lengths, the limiting factor is therefore not exhaustion of the theoretical k-mer space but the biological repetitiveness and non-random composition of the genome.
|
|
||||||
|
|
||||||
For example, the number of possible k-mers grows as follows:
|
|
||||||
|
|
||||||
| $k$ | $4^k$ | $4^k/2$ after canonicalization |
|
|
||||||
| ----: | --------------------: | -------------------------------: |
|
|
||||||
| 11 | $4.19\times10^6$ | $2.10\times10^6$ |
|
|
||||||
| 15 | $1.07\times10^9$ | $5.37\times10^8$ |
|
|
||||||
| 21 | $4.40\times10^{12}$ | $2.20\times10^{12}$ |
|
|
||||||
| 25 | $1.13\times10^{15}$ | $5.63\times10^{14}$ |
|
|
||||||
| 31 | $4.61\times10^{18}$ | $2.31\times10^{18}$ |
|
|
||||||
|
|
||||||
Canonicalization approximately halves the number of possible representations because a k-mer and its reverse complement are represented by the same canonical form.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
The distinction between the simple $4^k$ model and the Karlin--Ost framework is important. The former describes the size of the combinatorial k-mer space; the latter describes the probability of observing long matches in sequences whose composition may depart from the uniform random model. Consequently, the value of $k$ cannot be selected from genome size alone.
|
|
||||||
|
|
||||||
|
|
||||||
These scales have very different implications for different genome sizes.
|
|
||||||
|
|
||||||
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995; @Karlin1997]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
|
|
||||||
|
|
||||||
This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.
|
|
||||||
|
|
||||||
The supported range $11\leq k\leq31$ should therefore be understood as an engineering compromise rather than as a universal biological optimum. Smaller values increase the probability that unrelated genomic regions share the same k-mers, while larger values increase specificity but also make the representation more sensitive to substitutions, indels, and incomplete homology. The appropriate value within this range consequently depends on the biological scale and comparison task.
|
|
||||||
|
|
||||||
## Odd k-mer lengths
|
## Odd k-mer lengths
|
||||||
|
|
||||||
OBIKmer additionally requires $k$ to be odd. This constraint is independent of the resolution argument above.
|
OBIKmer additionally requires $k$ to be odd. This constraint is
|
||||||
|
independent of the specificity considerations discussed below.
|
||||||
|
|
||||||
For an odd-length DNA sequence, a k-mer cannot be identical to its own reverse complement. Consequently, every k-mer has two distinct orientations, which can be unambiguously collapsed into a canonical representation. For even $k$, self-reverse-complementary sequences exist, so a k-mer can coincide with its own reverse complement.
|
For an odd-length DNA sequence, a k-mer cannot be identical to its own
|
||||||
|
reverse complement. Every k-mer therefore belongs to a pair of distinct
|
||||||
|
orientations, which can be represented by a single canonical k-mer. For
|
||||||
|
even $k$, self-reverse-complementary sequences exist.
|
||||||
|
|
||||||
Odd $k$ is therefore **not required for canonicalization itself**: the canonical form
|
Odd $k$ is not required for canonicalization itself: the canonical form
|
||||||
|
|
||||||
$$
|
$$
|
||||||
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
|
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
|
||||||
$$
|
$$
|
||||||
|
|
||||||
remains well defined for even $k$. The odd-$k$ restriction instead guarantees that the two orientations are always distinct, which simplifies the representation and orientation handling used by OBIKmer.
|
is well defined for both odd and even lengths. Restricting $k$ to odd
|
||||||
|
values guarantees that the two orientations are always distinct,
|
||||||
|
simplifying the representation and orientation handling used by OBIKmer.
|
||||||
|
|
||||||
## Karlin--Ost (1988) formula
|
## K-mer specificity
|
||||||
|
|
||||||
[@Karlin1988] considers $s$ independent stationary letter sequences over a finite alphabet and studies $K_{r,s}(N)$, the length of the longest word common to $r$ of the $s$ sequences, each of length $N$. For i.i.d. equiprobable letters over an alphabet of size $m$ (here $m=4$ for DNA), the characteristic local match parameter reduces to
|
A simple way to illustrate the effect of k-mer size is to consider a
|
||||||
|
random DNA sequence in which the four nucleotides are independent and
|
||||||
|
equally probable. There are $4^k$ possible oriented k-mers, so the
|
||||||
|
expected number of occurrences of a particular oriented k-mer in a
|
||||||
|
sequence of length $G$ is approximately $G/4^k$. For canonical k-mers,
|
||||||
|
the corresponding number of possible sequences is halved, and the
|
||||||
|
expected number of occurrences is approximately $2G/4^k$.
|
||||||
|
|
||||||
|
Increasing $k$ therefore reduces the probability that a particular k-mer
|
||||||
|
occurs by chance. However, the probability of a match somewhere in a
|
||||||
|
collection of sequences also depends on the amount of sequence searched
|
||||||
|
and on the number of possible k-mers considered. The following sections
|
||||||
|
formalize this distinction and describe how chance sharing can be
|
||||||
|
assessed across an indexed set of genomes.
|
||||||
|
|
||||||
|
## Choosing an appropriate k-mer size
|
||||||
|
|
||||||
|
The theoretical k-mer space grows exponentially with $k$. Thus, increasing $k$ from 11 to 31 greatly expands the sequence space and reduces the probability that unrelated sequences share a particular k-mer by chance. The following table gives the size of the theoretical k-mer space and, after canonicalization, the number of distinct representations.
|
||||||
|
|
||||||
|
| $k$ | $4^k$ | $4^k/2$ after canonicalization |
|
||||||
|
|---:|---:|---:|
|
||||||
|
| 11 | $4.19\times10^6$ | $2.10\times10^6$ |
|
||||||
|
| 15 | $1.07\times10^9$ | $5.37\times10^8$ |
|
||||||
|
| 21 | $4.40\times10^{12}$ | $2.20\times10^{12}$ |
|
||||||
|
| 25 | $1.13\times10^{15}$ | $5.63\times10^{14}$ |
|
||||||
|
| 31 | $4.61\times10^{18}$ | $2.31\times10^{18}$ |
|
||||||
|
|
||||||
|
Genome sizes illustrate the range of sequence scales for which k-mers may be used. Bacterial genomes are typically a few megabases long, although some symbiotic bacteria have much smaller genomes, while other bacteria have genomes exceeding ten megabases. Plant genomes span an even wider range, from tens of megabases to more than one hundred gigabases. In very large and repeat-rich genomes, the main limitation is not necessarily the exhaustion of the theoretical k-mer space, but the non-uniform distribution of k-mers caused by sequence composition and repeated DNA.
|
||||||
|
|
||||||
|
The supported range $11\leq k\leq31$ should therefore be understood as an engineering and application-oriented compromise, not as a universal biological optimum. Shorter k-mers are more likely to be shared by unrelated regions, whereas longer k-mers provide greater sequence specificity. However, longer k-mers are also more sensitive to substitutions and indels: a single difference can disrupt an exact match across the whole k-mer. The appropriate value within the supported range consequently depends on the size and characteristics of the genomes being compared, as well as on the biological question.
|
||||||
|
|
||||||
|
The following section introduces a random-sequence model to quantify the probability of chance sharing across an indexed set of genomes. This provides a formal basis for assessing how the choice of $k$ affects the interpretation of shared k-mers.
|
||||||
|
|
||||||
|
## Chance sharing across genomes
|
||||||
|
|
||||||
|
OBIKmer indexes distinct canonical k-mers and records the genomes in
|
||||||
|
which each k-mer occurs. This makes it possible to compare the observed
|
||||||
|
sharing of k-mers with the sharing expected under a random-sequence
|
||||||
|
model. The same probability model can be used for two related purposes:
|
||||||
|
measuring the excess of sharing among k-mers present in the index, and
|
||||||
|
estimating the number of chance k-mers among those shared by at least a
|
||||||
|
specified number of genomes.
|
||||||
|
|
||||||
|
### The null model and genome-specific probabilities
|
||||||
|
|
||||||
|
The null hypothesis assumes that the $S$ indexed genomes are unrelated
|
||||||
|
and that each genome is an independent random sequence in which
|
||||||
|
nucleotides are independent and equiprobable.
|
||||||
|
|
||||||
|
Since OBIKmer uses odd-length canonical k-mers, reverse-complement
|
||||||
|
canonicalization reduces the number of possible k-mers by half. The size
|
||||||
|
of the canonical k-mer space is therefore
|
||||||
|
|
||||||
$$
|
$$
|
||||||
\lambda^{[r]} = m^{1-r},
|
M_k=\frac{4^k}{2}.
|
||||||
$$
|
$$
|
||||||
|
|
||||||
so for $r=2$ (a word shared by two occurrences), $\lambda = m^{-1} = 1/4$.
|
Let $N_i$ be the number of distinct canonical k-mers recorded for genome
|
||||||
|
$i$ in the index. Under the null model, these can be regarded as a
|
||||||
The expected number of common words of length $k$ among the $\binom{s}{r}$ relevant groups of sequences is
|
uniform sample without replacement from the $M_k$ possible canonical
|
||||||
|
k-mers. The probability that a given canonical k-mer is present in
|
||||||
|
genome $i$ is then
|
||||||
|
|
||||||
$$
|
$$
|
||||||
S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1)),
|
p_i=\frac{N_i}{M_k}=\frac{2N_i}{4^k}.
|
||||||
$$
|
$$
|
||||||
|
|
||||||
with $\gamma=1$ exactly in the i.i.d. equiprobable case. Theorem 2.2 gives the limiting distribution of $K_{r,s}(N)$, with $k$ set on the scale $k=\left[\dfrac{\log N^{r}}{-\log\lambda^{*}}+x+1\right]$:
|
This is the inclusion probability for a distinct k-mer. It differs from
|
||||||
|
the probability of at least one occurrence in a positional model,
|
||||||
|
because repeated occurrences within a genome are not counted separately.
|
||||||
|
|
||||||
|
Let $X$ denote the number of genomes, among the $S$ indexed genomes,
|
||||||
|
that contain a given canonical k-mer under the null model. Since genomes
|
||||||
|
are assumed to be independent, $X$ is a sum of independent presence
|
||||||
|
indicators with probabilities $p_1,\ldots,p_S$. When these probabilities
|
||||||
|
differ, $X$ follows a Poisson-binomial distribution.
|
||||||
|
|
||||||
|
### Computing the Poisson-binomial distribution
|
||||||
|
|
||||||
|
The distribution of $X$ can be computed by processing genomes one at a
|
||||||
|
time. Let $P_i(j)$ denote the probability that the k-mer occurs in
|
||||||
|
exactly $j$ of the first $i$ genomes. Initially,
|
||||||
|
|
||||||
$$
|
$$
|
||||||
\lim_{N\to\infty}\left[\Pr\{K_{r,s}(N)<k\} - \exp\{-(1-\lambda^{*})\,S_1(N,k)\}\right] = 0,
|
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
|
||||||
$$
|
$$
|
||||||
|
|
||||||
equivalently
|
When genome $i$ is added, the k-mer is either absent, with probability
|
||||||
|
$1-p_i$, or present, with probability $p_i$. Thus,
|
||||||
|
|
||||||
$$
|
$$
|
||||||
\frac{-\log\Pr\{K_{r,s}(N)<k\}}{(\lambda^{*})^{\rho(N,x)}} \xrightarrow[N\to\infty]{} (1-\lambda^{*})\,\gamma^{*}(\lambda^{*})^{x}.
|
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
|
||||||
$$
|
$$
|
||||||
|
|
||||||
The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985].
|
This recurrence avoids enumerating the subsets of genomes. Computing the
|
||||||
|
complete distribution requires $O(S^2)$ operations and $O(S)$ memory. To
|
||||||
|
compute the upper-tail probability, only the probabilities for fewer
|
||||||
$o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ».
|
than $r$ presences need to be tracked:
|
||||||
|
|
||||||
Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ :
|
|
||||||
|
|
||||||
- $N^{r}\binom{s}{r}\gamma\,\lambda^{k}$ est la valeur **exacte à la limite**, celle qu'on utilise en pratique (c'est elle qui redonne $E[C]$ de notre discussion).
|
|
||||||
- $(1+o(1))$ dit : « la vraie valeur de $S_1(N,k)$ n'est pas *exactement* égale à ce produit pour $N$ fini, mais le rapport entre les deux tend vers 1 quand $N\to\infty$ ». Autrement dit $S_1(N,k) / \left[N^{r}\binom{s}{r}\gamma\lambda^{k}\right] \to 1$.
|
|
||||||
|
|
||||||
Pourquoi ce terme existe : la formule $F_\sigma(k)/\lambda^k \to \gamma$ (éq. 2.8 du papier) est elle-même une limite quand $k\to\infty$ — pour $k$ fini, il y a un petit écart entre $F_\sigma(k)$ et $\gamma\lambda^k$. C'est cet écart-là qui devient le $o(1)$.
|
|
||||||
|
|
||||||
Ce que ça implique pour l'usage pratique : dans le cas iid équiprobable que tu utilises (ADN, $\gamma=1$ exactement, pas seulement à la limite — je l'ai vérifié directement sur la dérivation : $F_\sigma(k)=m^{-k}=\lambda^k$ pour tout $k$, pas juste asymptotiquement), le $o(1)$ est en réalité **nul** dans ce cas précis, pas juste négligeable. La formule $S_1(N,k)=N^r\binom{s}{r}\lambda^k$ est donc exacte (à l'asymptotique en $N$ près, qui elle vient d'ailleurs — de l'approximation de Poisson elle-même, pas de ce $\gamma$).
|
|
||||||
|
|
||||||
|
|
||||||
Les deux approches sont légitimes, mais elles répondent à des questions différentes — et il y a une relation algébrique exacte entre les deux qui vaut la peine d'être explicitée.
|
|
||||||
|
|
||||||
**Ce que calcule l'approche naïve.** Pour **un k-mer $w$ déjà observé et fixé**, $p_i\simeq1-\exp(-G_i/4^k)$ est la probabilité que *ce mot précis* apparaisse quelque part dans le génome $i$ par hasard, et $P(R\geq r)$ (Poisson-binomiale ou binomiale) est la probabilité que *ce mot précis* soit partagé par au moins $r$ des $S$ génomes par hasard. C'est une **p-value par observation** : « j'ai vu ce k-mer particulier dans 5 génomes sur 10, est-ce surprenant pour lui ? »
|
|
||||||
|
|
||||||
**Ce que calcule Karlin-Ost.** $S_1(k)$ et $P(K_{r,s}\geq k)$ répondent à : « parmi **tous les $4^k$ mots possibles**, quelle est la probabilité qu'au moins un seul d'entre eux atteigne ce seuil de partage par hasard, n'importe où dans les génomes ? » C'est une question **globale sur tout l'espace des k-mers**, pas sur un mot déjà identifié.
|
|
||||||
|
|
||||||
**La relation exacte entre les deux.** Reprenons le cas $r=s=2$, génomes égaux de taille $G$ :
|
|
||||||
|
|
||||||
- Naïf (mot $w$ fixé) : $P(R=2\mid w)\simeq p^2=(G\cdot4^{-k})^2=G^2\cdot4^{-2k}$
|
|
||||||
- Karlin-Ost : $S_1=G^2\cdot4^{-k}$
|
|
||||||
|
|
||||||
Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nombre de mots possibles. Ce n'est pas une coïncidence : $S_1=N^r\binom{s}{r}\lambda^k$ vient de $\lambda^k=\sum_w \Pr\{w\}^r$, une somme explicite sur les $4^k$ mots, alors que le calcul naïf tel que présenté ne somme sur rien — il traite $w$ comme donné.
|
|
||||||
|
|
||||||
**Donc l'approche naïve, formellement, c'est Karlin-Ost auquel il manque le facteur « nombre de k-mers testés ».** Pour la rendre comparable à la question « dois-je m'attendre à *une* collision fortuite n'importe où dans mon jeu de données », il faudrait une correction à la Bonferroni : multiplier $P(R\geq r\mid w)$ par le nombre effectif de k-mers canoniques testables ($\approx4^k/2$). Karlin-Ost fait cette correction analytiquement et correctement (avec en prime le facteur de déclumping $(1-\lambda)$ qu'on a discuté, qui corrige le sur-comptage des matches chevauchants — une union bound brute comme le Bonferroni décrit ci-dessus le surestimerait légèrement).
|
|
||||||
|
|
||||||
**Ce qui veut dire, en pratique, que les deux outils ne sont pas concurrents mais complémentaires :**
|
|
||||||
- **Choix a priori de $k$ / FPR global du dataset** (la question de `kmers.md` depuis le début de cette conversation) → Karlin-Ost, parce qu'il intègre nativement la multiplicité sur tous les mots possibles.
|
|
||||||
- **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant.
|
|
||||||
|
|
||||||
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
|
|
||||||
|
|
||||||
|
|
||||||
## Chance sharing across genomes and the FDR of homology
|
|
||||||
|
|
||||||
The formulas above ask how likely a match is between two sequences. The question OBIKmer actually has to answer is different: a k-mer is found in $r$ of the $S$ indexed genomes, and one wants to know whether this sharing can be explained by chance or whether it is evidence of homology. This section builds a null model for that question and turns it into a false discovery rate (FDR) for the call "a k-mer shared by at least $r$ genomes is homologous".
|
|
||||||
|
|
||||||
### The null model
|
|
||||||
|
|
||||||
The null hypothesis is that the $S$ genomes are unrelated, that is, that each one is an independent random sequence. Rejecting it for a k-mer means that chance alone cannot explain why the k-mer is shared, which is the signal of homology one is looking for.
|
|
||||||
|
|
||||||
The model only records whether a k-mer is present in a genome, not how many times. Two consequences follow. First, repeats inside a genome are irrelevant, and the size $G_i$ of genome $i$ is its number of **distinct canonical k-mers**. Second, a repeat family shared between genomes is paralogy, which is a form of homology; it is a true positive and not a false one. The only k-mers that could be shared by composition rather than by descent are low-complexity ones, which OBIKmer removes beforehand with its entropy filter.
|
|
||||||
|
|
||||||
Everything takes place in canonical space. Because $k$ is odd, no k-mer equals its own reverse complement, so the $4^k$ k-mers pair up perfectly and the space contains exactly
|
|
||||||
|
|
||||||
$$
|
$$
|
||||||
N=\frac{4^k}{2}
|
P_{\mathrm{null}}(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
|
||||||
$$
|
$$
|
||||||
|
|
||||||
canonical k-mers. A given canonical k-mer is present in genome $i$ by chance with probability
|
The computation can therefore be restricted to $P_i(0),\ldots,P_i(r-1)$,
|
||||||
|
requiring $O(Sr)$ operations and $O(r)$ memory. The resulting
|
||||||
|
distribution depends on the genome-specific values $N_i$, but not on the
|
||||||
|
identity of the k-mer, and can thus be computed once for a given index
|
||||||
|
and k-mer length.
|
||||||
|
|
||||||
|
### Comparing observed and expected sharing in the index
|
||||||
|
|
||||||
|
The index contains only k-mers observed in at least one genome.
|
||||||
|
Consequently, when comparing sharing proportions among the k-mers
|
||||||
|
recorded in the index, the relevant null probability is conditional on
|
||||||
|
at least one presence:
|
||||||
|
|
||||||
$$
|
$$
|
||||||
p_i = 1-\left(1-\frac1N\right)^{G_i}\simeq \frac{G_i}{N}=\frac{2\,G_i}{4^k}.
|
q_r=P(X\geq r\mid X\geq 1)
|
||||||
|
=\frac{P_{\mathrm{null}}(X\geq r)}{P_{\mathrm{null}}(X\geq 1)}.
|
||||||
$$
|
$$
|
||||||
|
|
||||||
This is twice the $G/4^k$ used earlier in this page, because there $G$ is a number of positions on one strand, whereas $G_i$ here counts canonical k-mers, for which both strands are already merged.
|
Let $n_{\geq r}$ be the number of distinct canonical k-mers observed in
|
||||||
|
at least $r$ genomes, and let $n_{\geq 1}$ be the total number of
|
||||||
Let $R$ be the number of genomes, out of $S$, that contain a given canonical k-mer by chance. Genomes are independent under the null, so $R$ is a sum of independent presence indicators, each with its own probability $p_i$.
|
distinct canonical k-mers recorded in the index. The observed proportion
|
||||||
|
of indexed k-mers shared by at least $r$ genomes is
|
||||||
### Exact tail probability
|
|
||||||
|
|
||||||
When genome sizes differ, the $p_i$ differ, and $R$ follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be hopeless, since there are up to $2^S$ subsets. The distribution is instead obtained by adding the genomes one at a time. Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
|
|
||||||
|
|
||||||
$$
|
$$
|
||||||
d'_j = d_j\,(1-p_i) + d_{j-1}\,p_i, \qquad d_0=1 \text{ before the first genome.}
|
O_r=\frac{n_{\geq r}}{n_{\geq 1}}.
|
||||||
$$
|
$$
|
||||||
|
|
||||||
The table $d$ only needs to remember how many genomes were hit, not which ones, and this brings the cost down to $O(S^2)$. The result does not depend on the order in which the genomes are processed. It does not depend on the k-mer either, so it is computed once per index. The upper tail is then
|
Under the random-sequence model, the expected proportion is $q_r$. The
|
||||||
|
difference
|
||||||
|
|
||||||
$$
|
$$
|
||||||
P_0(R\geq r) = 1-\sum_{j=0}^{r-1} d_j .
|
D_r=O_r-q_r
|
||||||
$$
|
$$
|
||||||
|
|
||||||
Only $d_0,\dots,d_{r-1}$ are needed, so the cost is $O(S\,r)$.
|
therefore measures the excess or deficit of sharing relative to the null
|
||||||
|
expectation. Positive values indicate that k-mers are shared across
|
||||||
|
genomes more frequently than expected by chance, whereas negative values
|
||||||
|
indicate less frequent sharing. A value of zero indicates agreement with
|
||||||
|
the random-sequence model.
|
||||||
|
|
||||||
### Approximation with equal genome sizes
|
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$
|
||||||
|
is bounded:
|
||||||
If all genomes are given the same size, taken as the mean
|
|
||||||
|
|
||||||
$$
|
$$
|
||||||
\bar G=\frac1S\sum_{i=1}^{S}G_i, \qquad \bar p=\frac{2\bar G}{4^k},
|
-1\leq D_r\leq 1.
|
||||||
$$
|
$$
|
||||||
|
|
||||||
then $R$ follows a plain binomial distribution and the tail is
|
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By
|
||||||
|
construction, $D_1=0$, since all k-mers recorded in the index occur in
|
||||||
|
at least one genome. Unlike a ratio-based measure, this difference
|
||||||
|
remains defined when no k-mers are observed in at least $r$ genomes.
|
||||||
|
|
||||||
|
### Estimating the false discovery rate
|
||||||
|
|
||||||
|
The probability $P_{\mathrm{null}}(X\geq r)$ applies to one specified canonical k-mer.
|
||||||
|
To estimate the number of k-mers that could reach this sharing threshold
|
||||||
|
by chance across the entire canonical k-mer space, this probability must
|
||||||
|
be considered over all $M_k$ possible canonical k-mers.
|
||||||
|
|
||||||
|
The expected number of chance k-mers present in at least $r$ genomes is
|
||||||
|
|
||||||
$$
|
$$
|
||||||
P_0(R\geq r)\simeq\sum_{j=r}^{S}\binom{S}{j}\,\bar p^{\,j}(1-\bar p)^{S-j}.
|
E[\mathrm{FP}(r)]=M_kP_{\mathrm{null}}(X\geq r).
|
||||||
$$
|
$$
|
||||||
|
|
||||||
This sum is the regularized incomplete beta function $I_{\bar p}(r,\,S-r+1)$, available in standard statistical libraries. If it is summed by hand, the terms are obtained one from the next by $t_{j+1}=t_j\,\frac{S-j}{j+1}\,\frac{\bar p}{1-\bar p}$.
|
The observed number of k-mers passing the threshold is $n_{\geq r}$. An
|
||||||
|
estimate of the false discovery rate (FDR) for this set is therefore
|
||||||
A theorem of Hoeffding [@Hoeffding1956-bx] shows that this approximation errs on the safe side. For a fixed mean, the Poisson-binomial distribution is more concentrated than the binomial one, so for $r\geq S\bar p+1$ the binomial tail is never smaller than the exact tail. Using the mean genome size therefore overestimates the chance of sharing and can only make the FDR more pessimistic. Gleser [@Gleser1975-yu] refines this result: replacing the $p_i$ by any partial average of themselves, not only by their full mean, also moves the upper tail towards the binomial one, provided $r-1\geq S\bar p+2$. Grouping genomes of similar size is therefore safe in the same sense. What it loses is power when genome sizes differ by orders of magnitude, since one large genome then weighs as much as several small ones in $\bar p$.
|
|
||||||
|
|
||||||
When all $p_i$ are small, which is the case for large $k$, the number of hits is also well approximated by a Poisson distribution of mean $\lambda=\sum_i p_i$, with an error bounded by $\sum_i p_i^2$ [@Le-Cam1960-pe].
|
|
||||||
|
|
||||||
For the numerical evaluation, the tail that does not contain the mode $S\bar p$ should be summed directly. When $r>S\bar p$, computing the sum from $j=r$ upwards is accurate even for very small probabilities, whereas computing $1$ minus the sum of the lower terms loses all precision once the tail falls below the machine epsilon.
|
|
||||||
|
|
||||||
### False discovery rate
|
|
||||||
|
|
||||||
The FDR of the call "present in at least $r$ genomes" is the fraction of retained k-mers expected to be pure chance:
|
|
||||||
|
|
||||||
$$
|
$$
|
||||||
\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P_0(R\geq r)}{n_{\geq r}}\right).
|
\widehat{\mathrm{FDR}}(r)=
|
||||||
|
\begin{cases}
|
||||||
|
1 & \text{if } n_{\geq r}=0,\\[4pt]
|
||||||
|
\displaystyle
|
||||||
|
\min\left(1,
|
||||||
|
\frac{M_k P_{\mathrm{null}}(X\geq r)}
|
||||||
|
{n_{\geq r}}\right)
|
||||||
|
& \text{otherwise}.
|
||||||
|
\end{cases}
|
||||||
$$
|
$$
|
||||||
|
|
||||||
The numerator is the number of k-mers expected to reach $r$ genomes by chance if the genomes were unrelated. Each of the $N$ canonical k-mers has probability $P_0(R\geq r)$ of doing so, and no further correction for multiple testing is needed because the number of tests is already in the product. The denominator $n_{\geq r}$ is the number of distinct k-mers actually observed in at least $r$ genomes, which is read directly from the index. The observed count mixes chance and true homology, so the ratio is an upper bound on the true FDR: it implicitly assumes that all k-mers are null, and it never underestimates the noise. The minimal $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls below the accepted level, for example 5 %, is the sharing threshold to use.
|
The numerator estimates the number of false positives under the null
|
||||||
|
model, while the denominator is the observed number of k-mers passing
|
||||||
|
the sharing threshold. This estimate assumes that all possible canonical
|
||||||
|
k-mers are subject to the null model. It is therefore conservative in
|
||||||
|
the sense that it does not attempt to estimate the proportion of the
|
||||||
|
k-mer space corresponding to true homology. The estimate is capped at
|
||||||
|
one because a proportion cannot exceed one.
|
||||||
|
|
||||||
As an illustration, take $k=21$ (so $N=2.2\times10^{12}$), $S=10$ genomes and $G_i=10^9$ distinct canonical k-mers each, hence $p=4.5\times10^{-4}$. The number of k-mers expected to be shared by chance is:
|
The distinction between $D_r$ and the FDR is important. $D_r$ compares
|
||||||
|
the observed and expected proportions among k-mers already present in
|
||||||
|
the index. The FDR estimate instead compares the expected number of
|
||||||
|
chance k-mers in the entire canonical k-mer space with the observed
|
||||||
|
number passing the sharing threshold.
|
||||||
|
|
||||||
| $r$ | $N\,P_0(R\geq r)$ |
|
The smallest value of $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls
|
||||||
|
below a chosen threshold, for example 5%, can be used as a minimum
|
||||||
|
sharing threshold. This interpretation is conditional on the assumptions
|
||||||
|
of the null model and should not be taken as a model-independent
|
||||||
|
guarantee of the biological false discovery rate.
|
||||||
|
|
||||||
|
As an illustration, consider $k=21$, for which
|
||||||
|
$M_k\simeq2.2\times10^{12}$, and an index containing $S=10$ genomes.
|
||||||
|
Suppose that the numbers of distinct canonical k-mers in these genomes
|
||||||
|
range from $0.5\times10^9$ to $1.4\times10^9$, in increments of
|
||||||
|
$0.1\times10^9$. Applying the Poisson-binomial model with the
|
||||||
|
corresponding genome-specific probabilities gives the following
|
||||||
|
expected numbers of k-mers shared by chance:
|
||||||
|
|
||||||
|
| $r$ | $M_kP_{\mathrm{null}}(X\geq r)$ |
|
||||||
| ---: | ---: |
|
| ---: | ---: |
|
||||||
| 2 | $2\times10^{7}$ |
|
| 2 | $1.82\times10^7$ |
|
||||||
| 3 | $2.5\times10^{4}$ |
|
| 3 | $2.06\times10^4$ |
|
||||||
| 4 | $\approx 20$ |
|
| 4 | $15$ |
|
||||||
| 5 | $\approx 0.01$ |
|
| 5 | $0.0075$ |
|
||||||
|
|
||||||
Chance alone produces millions of pairwise shared 21-mers at this genome size, a few tens of thousands shared by three genomes, and almost none shared by five or more. Whether a given $r$ is acceptable then depends on how many k-mers the index actually reports at that level.
|
In this example, chance alone can produce many pairwise shared 21-mers,
|
||||||
|
whereas the expected number of k-mers shared by five or more genomes is
|
||||||
|
very small. Whether a given sharing threshold is sufficiently stringent
|
||||||
|
depends on the number of k-mers actually observed at that level.
|
||||||
|
|
||||||
### Limits of the model
|
### Limits of the model
|
||||||
|
|
||||||
- The FDR is computed per k-mer. Neighbouring k-mers of a shared region overlap and are not independent, so a single homologous region contributes many correlated k-mers. A per-region FDR would count regions instead of k-mers and is not covered here.
|
The FDR estimate is based on a simplified null model and has two
|
||||||
- The probability $p_i$ assumes equiprobable, independent nucleotides. The composition biases discussed earlier in this page violate this assumption, and the resulting error is not accounted for.
|
important limitations.
|
||||||
|
|
||||||
---
|
First, the model assumes independent, equiprobable nucleotides. Real
|
||||||
|
genomes exhibit nucleotide-composition biases, oligonucleotide
|
||||||
|
dependencies, and repeated sequences. These properties can increase the
|
||||||
|
probability of particular k-mers beyond the uniform expectation.
|
||||||
|
Removing low-complexity k-mers reduces some of these effects but does
|
||||||
|
not eliminate all compositional biases. Karlin and colleagues showed
|
||||||
|
that relative oligonucleotide abundances, including dinucleotide
|
||||||
|
frequencies, form characteristic genomic signatures that can differ
|
||||||
|
between organisms and remain relatively stable within genomes
|
||||||
|
[@Karlin1995-lt; @Karlin1994-on].
|
||||||
|
|
||||||
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
|
Second, the FDR is computed per k-mer. Consecutive k-mers extracted from
|
||||||
|
the same homologous region overlap and are therefore not independent
|
||||||
## 1. FDR d'un ensemble de k-mers sélectionnés
|
observations. A single homologous region can contribute many correlated
|
||||||
|
k-mers to the index. The estimate presented here consequently describes
|
||||||
**Estimation de l'espérance du nombre de faux positifs.** Soit $M$ le nombre de k-mers canoniques testés, $M = 4^k/2$ pour k impair. Soit $q(r)=P(X\ge r)$ la queue du modèle nul. Le nombre attendu de k-mers présents par hasard dans au moins $r$ génomes vaut
|
the expected fraction of chance k-mers, not the fraction of false
|
||||||
|
homologous regions. A region-level FDR would require a model that treats
|
||||||
$$
|
matching regions, rather than individual k-mers, as the units of
|
||||||
E[\mathrm{FP}(r)] = M\,q(r).
|
observation.
|
||||||
$$
|
|
||||||
|
|
||||||
Cette égalité est exacte par linéarité de l'espérance, même si les k-mers chevauchants sont dépendants. La dépendance n'affecte que la variance, pas l'espérance. Seule l'approximation de Poisson par génome reste une approximation.
|
|
||||||
|
|
||||||
**FDR.** Soit $N(r)=\#\{i : r_i\ge r\}$ le nombre observé de k-mers couvrant au moins $r$ génomes. Alors
|
|
||||||
|
|
||||||
$$
|
|
||||||
\mathrm{FDR}(r)=\min\!\Big(1,\ \frac{M\,q(r)}{N(r)}\Big).
|
|
||||||
$$
|
|
||||||
|
|
||||||
Cette estimation est légèrement conservatrice. Elle suppose $\pi_0=1$, c'est-à-dire que presque tout l'espace $M$ est nul, ce qui est vrai pour k grand.
|
|
||||||
|
|
||||||
**Ensemble hétérogène en $r_i$.** Si le jeu n'est pas défini par un seuil unique, on utilise le local fdr par classe de couverture :
|
|
||||||
|
|
||||||
$$
|
|
||||||
\mathrm{fdr}(r)=\frac{M\,P(X=r)}{N_r},\qquad N_r=\#\{i : r_i=r\}.
|
|
||||||
$$
|
|
||||||
|
|
||||||
Le FDR de l'ensemble est alors la moyenne $\frac1N\sum_i \mathrm{fdr}(r_i)$. Pour un ensemble défini par seuil $r_{\min}$, cela redonne exactement la formule précédente.
|
|
||||||
|
|
||||||
**Remarque.** $M\,p\approx G$, d'où $E[\mathrm{FP}(r)]\approx\binom{S}{r}\,G^r/M^{r-1}$ en régime de petits $p$. Cela donne une lecture directe de la quantité.
|
|
||||||
|
|
||||||
## 2. Markov d'ordre 0 avec fréquences inégales
|
|
||||||
|
|
||||||
Soit $\pi_b$ la fréquence du nucléotide $b$. Pour un k-mer $w$ de composition $n=(n_A,n_C,n_G,n_T)$ avec $\sum n_b=k$, on a $P(w)=\prod_b \pi_b^{n_b}$.
|
|
||||||
|
|
||||||
**Attendu par génome.** Le k-mer canonique $K=\{w,\overline w\}$ est vu quand le brin direct porte $w$ ou $\overline w$, donc
|
|
||||||
|
|
||||||
$$
|
|
||||||
\lambda_K = G\,\big(P(w)+P(\overline w)\big)=G\Big(\prod_b\pi_b^{n_b}+\pi_A^{n_T}\pi_T^{n_A}\pi_C^{n_G}\pi_G^{n_C}\Big).
|
|
||||||
$$
|
|
||||||
|
|
||||||
Cela ne dépend que de la composition. Si $\pi_A=\pi_T$ et $\pi_C=\pi_G$, on retrouve $2P(w)$.
|
|
||||||
|
|
||||||
**Faux positifs.** Le test n'est plus homogène : chaque composition a sa propre $p(n)=1-e^{-\lambda(n)}$. Il faut donc sommer sur les classes de composition (un k-mer et son complément inverse partagent le même $\lambda$, d'où le facteur ½) :
|
|
||||||
|
|
||||||
$$
|
|
||||||
E[\mathrm{FP}(r)]=\frac12\sum_{n}\binom{k}{n_A,n_C,n_G,n_T}\,P\big(X\ge r\mid p(n)\big).
|
|
||||||
$$
|
|
||||||
|
|
||||||
Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Elle est donc légère avec la binomiale de la moyenne, ou avec une Poisson-binomiale par composition si les $G_i$ diffèrent. Si les $\pi_b$ varient d'un génome à l'autre (GC différent), les $p_i(n)$ se construisent par génome.
|
|
||||||
|
|
||||||
**Effet attendu.** La moyenne de $\lambda_w$ sur tous les k-mers reste $2G/4^k$, car $\sum_w P(w)=1$. Dans le régime de petits $p$, le terme dominant $\binom{S}{r}p^r$ est convexe en $\lambda$. Par Jensen, le modèle équiprobable est donc le minimum de $E[\mathrm{FP}]$, et la déviation de composition ne peut qu'augmenter le FDR. Les k-mers riches en AT ou GC dominent la somme, ce qui rejoint ton remarque sur la faible complexité. C'est un résultat sur le terme dominant, pas une preuve pour la queue exacte.
|
|
||||||
|
|
||||||
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
|
|
||||||
|
|
||||||
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
|
|
||||||
@@ -34,7 +34,7 @@ $$
|
|||||||
D = \sum_i \mathbb{1}[a_i \ne b_i],
|
D = \sum_i \mathbb{1}[a_i \ne b_i],
|
||||||
$$
|
$$
|
||||||
|
|
||||||
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded.
|
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded. On an index that stores counts, a k-mer is taken as present when its count reaches the presence threshold (`--presence-threshold`, 1 by default).
|
||||||
|
|
||||||
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
|
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
|
||||||
|
|
||||||
|
|||||||
@@ -24,4 +24,4 @@ If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (
|
|||||||
|
|
||||||
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
|
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
|
||||||
|
|
||||||
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation.
|
See [`phylo`](../../../usage/phylo.md#distance-estimation-and-tree-reconstruction) for how to select a fixed $\alpha$ or request automatic estimation.
|
||||||
@@ -21,7 +21,7 @@ A family where every genome carries the same single state has entropy 0 (uninfor
|
|||||||
|
|
||||||
## Quartet restriction
|
## Quartet restriction
|
||||||
|
|
||||||
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts called states only, so absence never forms one of the two pairs.
|
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts the states of the alignment: without `--free-loss`, absence is a state like any other and can form one of the two pairs; with `--free-loss` it is missing data and never does.
|
||||||
|
|
||||||
## Pair balance
|
## Pair balance
|
||||||
|
|
||||||
@@ -31,4 +31,4 @@ How well a pair of genomes is represented in a sample depends on how many famili
|
|||||||
|
|
||||||
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
||||||
|
|
||||||
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
See [`phylo`](../../../usage/phylo.md#snp-character-selection-and-sampling) for how to bound and bias the sample.
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# estimate
|
# estimate
|
||||||
|
|
||||||
Estimate approximate-index parameters (z, evidence bits, false-positive rate) before building an index with `--approx`, without touching any files.
|
`estimate` has two independent uses. Without `--index`, it estimates approximate-index parameters (z, evidence bits, false-positive rate) before building an index with `--approx`, without touching any files. With `--index`, it compares the k-mer sharing observed in an existing index with the sharing expected by chance (see the last section).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
obikmer estimate [OPTIONS]
|
obikmer estimate [OPTIONS]
|
||||||
@@ -16,3 +16,35 @@ obikmer estimate [OPTIONS]
|
|||||||
| `--fp` | none | Target false-positive rate per z-window |
|
| `--fp` | none | Target false-positive rate per z-window |
|
||||||
|
|
||||||
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
|
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
|
||||||
|
|
||||||
|
## Comparing an index with random sharing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
obikmer estimate --index INDEX [--presence-threshold N] [-o FILE]
|
||||||
|
```
|
||||||
|
|
||||||
|
With `--index`, `estimate` reads an existing index instead of computing approximate-index parameters, and it cannot be combined with `-k`, `-z`, `--evidence-bits` or `--fp`. Its purpose is to tell how much of the k-mer sharing between the genomes of the index goes beyond what chance alone would produce, which is the question behind the choice of $k$ (see [K-mer size](../theory/kmer_indexing/kmers.md)).
|
||||||
|
|
||||||
|
The command writes a CSV, to standard output or to `FILE`, with one row for each $r$ from 1 to the number of genomes $S$:
|
||||||
|
|
||||||
|
| Column | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `r` | Minimum number of genomes sharing a k-mer |
|
||||||
|
| `k_r` | Number of k-mers present in at least `r` genomes |
|
||||||
|
| `O_r` | Observed proportion $k_r / k_1$ |
|
||||||
|
| `q_r` | Proportion expected by chance, $P(X \geq r \mid X \geq 1)$, where $X$ is the number of genomes containing a given k-mer under the random-sequence model |
|
||||||
|
| `D_r` | Deviation $O_r - q_r$ |
|
||||||
|
| `FP_r` | Number of k-mers expected to be present in at least `r` genomes by chance over the whole canonical k-mer space, $M_k \, P(X \geq r)$ |
|
||||||
|
| `FDR_r` | Estimated false discovery rate of the k-mers shared by at least `r` genomes: 1 when `k_r` is 0, otherwise $\min(1, \mathrm{FP}_r / k_r)$ |
|
||||||
|
|
||||||
|
A positive `D_r` means that k-mers are shared by at least `r` genomes more often than random sequences would share them; a negative value means less often. The first row always has `D_1 = 0`, because $k_1$ counts only k-mers present in at least one genome.
|
||||||
|
|
||||||
|
When no k-mer is shared by at least `r` genomes (`k_r = 0`), there is no set of k-mers whose false discovery rate could be estimated, and `FDR_r` is set to 1 by convention. This value means that nothing can be claimed at this level; it does not mean that the observed k-mers are false positives. The `FP_r` column of the same row shows how many k-mers chance alone would have produced.
|
||||||
|
|
||||||
|
The expected proportion is computed under the random-sequence model of the theory page: the genomes are unrelated, and each genome is a set of $N_i$ distinct canonical k-mers drawn without replacement and uniformly from the $M_k = 4^k/2$ canonical k-mers. A given k-mer is then present in genome $i$ with probability $p_i = N_i / M_k = 2N_i/4^k$, where $N_i$ is the number of distinct canonical k-mers considered as present in genome $i$. Genome sizes are rarely known, in particular when the index was built from unassembled reads, and the number of distinct k-mers has the advantage of being measured on the index itself and of not depending on the sequencing coverage.
|
||||||
|
|
||||||
|
Because the $N_i$ differ between genomes, the number $X$ of genomes containing a k-mer follows an exact Poisson-binomial distribution, from which $P(X \geq r)$ is obtained for every $r$ in a single pass. The approximation by a binomial distribution based on the mean genome size is not used.
|
||||||
|
|
||||||
|
`--presence-threshold` (default 1) has the same meaning as in `phylo`: on an index that stores counts, a k-mer is present in a genome only when its count reaches the threshold. Both $N_i$ and $k_r$ are computed with this same threshold, so the observed and expected proportions describe the same data. It has no effect on an index without counts.
|
||||||
|
|
||||||
|
The `k` used is the k-mer size stored in the index (for an approximate index, the effective indexed size $k-z+1$).
|
||||||
@@ -12,6 +12,14 @@ obikmer index -o OUTPUT [OPTIONS] [INPUTS...]
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. |
|
| `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. |
|
||||||
|
|
||||||
|
## Which files are read
|
||||||
|
|
||||||
|
A file named explicitly on the command line is always read, whatever its extension: the format (FASTA, FASTQ or GenBank) and the compression (gzip, bzip2, xz or zstd) are detected from the content. If the content is not a sequence file, the command reports an error rather than skipping the file.
|
||||||
|
|
||||||
|
When an input is a directory, the files inside it are selected by extension: `.fasta`, `.fa`, `.fna` (the extension of NCBI genome downloads), `.ffn`, `.fas`, `.fsa`, `.fastq`, `.fq`, `.gbff`, `.gbk` and `.gb`, each optionally followed by `.gz`.
|
||||||
|
|
||||||
|
A small input spread over many partitions leaves most partitions empty. This is valid: the index is built normally and empty partitions simply hold no k-mers.
|
||||||
|
|
||||||
## Options
|
## Options
|
||||||
|
|
||||||
| Option | Default | Description |
|
| Option | Default | Description |
|
||||||
|
|||||||
@@ -20,10 +20,26 @@ obikmer merge -o OUTPUT SOURCE... [OPTIONS]
|
|||||||
| `--force` | off | Overwrite an existing output directory |
|
| `--force` | off | Overwrite an existing output directory |
|
||||||
| `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts |
|
| `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts |
|
||||||
| `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing |
|
| `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing |
|
||||||
| `--budget-fraction` | `0.5` | Fraction of available RAM reserved as the memory budget for parallel partition merging |
|
| `--dense` | off | Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see [pack](pack.md)) |
|
||||||
|
|
||||||
## Behaviour
|
## Behaviour
|
||||||
|
|
||||||
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources.
|
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources.
|
||||||
|
|
||||||
By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies.
|
By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies.
|
||||||
|
|
||||||
|
The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; `--dense` keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The [pack](pack.md) page explains the trade-off between the two formats.
|
||||||
|
|
||||||
|
## Resource usage
|
||||||
|
|
||||||
|
The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.
|
||||||
|
|
||||||
|
Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.
|
||||||
|
|
||||||
|
## Merging an index that was already merged
|
||||||
|
|
||||||
|
An index produced by `merge` can be used as one of the sources of a later merge, which is how a collection is extended without starting again from the original genomes. There is one rule: among the sources, at least one must not have been through a merge. A merged index is stored in a compact packed format (see [pack](pack.md)) that is read very efficiently but cannot be extended in place. The merge therefore takes a freshly built index as its starting point and brings the packed index in as a source. A typical case is a merged index of many genomes plus one or a few newly built single-genome indexes, and this works.
|
||||||
|
|
||||||
|
If every source has already been merged or packed, the command stops before doing any work and says so. In that case, merge the original single-genome indexes together instead.
|
||||||
|
|
||||||
|
The genome columns of the result follow the order of the starting index first, then the other sources, so they may not come out in the order the sources were listed. The genome labels always identify each column unambiguously.
|
||||||
+115
-86
@@ -1,6 +1,13 @@
|
|||||||
# phylo
|
# phylo
|
||||||
|
|
||||||
Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
|
Compare the genomes stored in an index and build phylogenetic trees from them. Two complementary approaches are available:
|
||||||
|
|
||||||
|
- **Whole-genome distances** compare the k-mer content of the genomes, without any alignment.
|
||||||
|
- **SNP-based distances and analyses** compare the nucleotide variants found at the central position of related k-mers.
|
||||||
|
|
||||||
|
Distance matrices can be used to build Neighbor-Joining or UPGMA trees. SNP-based analyses can also produce a pseudo-alignment and calibrated substitution costs, with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
obikmer phylo INDEX [OPTIONS]
|
obikmer phylo INDEX [OPTIONS]
|
||||||
@@ -12,28 +19,43 @@ obikmer phylo INDEX [OPTIONS]
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `INDEX` | Index directory |
|
| `INDEX` | Index directory |
|
||||||
|
|
||||||
## Distance matrix (`--distance`)
|
## General options
|
||||||
|
|
||||||
| Option | Default | Description |
|
| Option | Default | Description |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `--distance` | none | Compute and write a distance matrix. See the two tables below for the full list of accepted values. No matrix is computed unless this is given, or `--nj`, `--upgma` or `--shared-kmers` is (they need a matrix and then use `jaccard`) |
|
| `--cpu-max N` | all available cores | Maximum number of CPU cores to use. It caps every thread pool of the process and can only lower the number of cores, never raise it above those available. Common to all subcommands |
|
||||||
|
| `-o, --output PREFIX` | none | Prefix of the output files. Without it, the distance matrix goes to standard output and every other file is written in the current directory under its name without the prefix. See "Output files" below |
|
||||||
|
|
||||||
|
## SNP families
|
||||||
|
|
||||||
|
SNP-based analyses use groups of k-mers that differ only at their central nucleotide. These groups are called families (see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the exact definitions and eligibility rules). A variable family is one character of the analysis, and the variants a genome carries give its state for that character.
|
||||||
|
|
||||||
|
## Distance estimation and tree reconstruction
|
||||||
|
|
||||||
|
| Option | Default | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `--distance` | none | Compute and write a distance matrix. The accepted values are listed just below the table. No matrix is computed unless this is given, or `--nj` or `--upgma` is (they need a matrix and then use `jaccard`) |
|
||||||
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
|
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
|
||||||
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
|
| `--presence-threshold` | `1` | How counts are turned into presence/absence on an index that stores counts: a kmer is present in a genome only when its count reaches this value. Used by `jaccard`, `mash` and `hamming`, and by the shared-kmer counts of `--shared-kmers`. No effect on an index without counts, nor on the distances that use the counts themselves |
|
||||||
|
| `--session DIR`, `--session-force` | none | Save and reuse the sample of a `snp-*` distance. Currently unavailable, see `--session` below |
|
||||||
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
|
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
|
||||||
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index distance, not a `snp-*` value |
|
| `--shared-kmers` | off | Write the shared-kmer count matrix (`<prefix>_shared.csv`). It can be used alone, without any distance, or together with a whole-index distance; it is refused with a `snp-*` value |
|
||||||
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
|
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
|
||||||
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
|
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
|
||||||
| `-o, --output` | none (stdout) | Output file prefix |
|
|
||||||
|
|
||||||
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `jaccard`, `mash` and `hamming` compare which kmers are present and work on either index type; the six others (`bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`) use the counts and need an index that stores them), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
||||||
|
|
||||||
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
|
`snp-*` values read the sibling annex (below), which is built automatically the first time it is needed, and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "SNP character selection and sampling" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||||
|
|
||||||
### Output
|
### Matrix format
|
||||||
|
|
||||||
When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
|
When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
|
||||||
|
|
||||||
## `--exclude-genome`, `--min-shared-family`
|
### `--session`: currently unavailable
|
||||||
|
|
||||||
|
`--session DIR` and `--session-force` are meant to save the sample of a `snp-*` `--distance` run in a directory, so that a later run with identical selection parameters reuses it instead of drawing again. This feature is being rebuilt and is not available for now: giving either option stops the command with an error. Until it returns, keep the output files of a run (the alignment, or the distance matrix) rather than relying on re-drawing the same sample.
|
||||||
|
|
||||||
|
## Genome selection and filtering
|
||||||
|
|
||||||
| Option | Description |
|
| Option | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -42,19 +64,79 @@ When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHY
|
|||||||
|
|
||||||
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
|
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
|
||||||
|
|
||||||
## Central-position SNP model
|
## SNP character selection and sampling
|
||||||
|
|
||||||
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index:
|
SNP-based analyses can use every eligible variable family, or a sample drawn from the index. The options below control which characters are selected, which genomes and states are kept, and how the sample is spread over the index.
|
||||||
|
|
||||||
|
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
|
||||||
|
|
||||||
|
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags.
|
||||||
|
|
||||||
|
Every invocation draws its own fresh random sample: running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately.
|
||||||
|
|
||||||
|
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
|
||||||
|
|
||||||
|
| Option | Default | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `--n-sites N\|all` | none (mandatory for `--pseudo-alignment` and `--sankoff`) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
|
||||||
|
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
||||||
|
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
||||||
|
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
|
||||||
|
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
|
||||||
|
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
|
||||||
|
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
|
||||||
|
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
|
||||||
|
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
||||||
|
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
||||||
|
|
||||||
|
### `--quartet`: keeping only families that can resolve a tree
|
||||||
|
|
||||||
|
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
||||||
|
|
||||||
|
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
||||||
|
|
||||||
|
### `--pair-balance`: representing every pair of genomes
|
||||||
|
|
||||||
|
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
|
||||||
|
|
||||||
|
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
|
||||||
|
|
||||||
|
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||||
|
|
||||||
|
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||||
|
|
||||||
|
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
||||||
|
|
||||||
|
The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
|
||||||
|
|
||||||
|
### Describing a draw: `--sample-stats`, `--sample-plan`
|
||||||
|
|
||||||
|
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
|
||||||
|
|
||||||
|
`--sample-stats` writes three files:
|
||||||
|
|
||||||
|
| File | Content |
|
||||||
|
|---|---|
|
||||||
|
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
|
||||||
|
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
|
||||||
|
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
|
||||||
|
|
||||||
|
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. A site is informative under the same rule as `--quartet`: without `--free-loss`, absence is a state of its own and can form one of the two pairs; with `--free-loss` it is missing data and cannot. Excluded genomes are listed with zeros in the first file and count nowhere.
|
||||||
|
|
||||||
|
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
|
||||||
|
|
||||||
|
## SNP family preparation and diagnostics
|
||||||
|
|
||||||
|
SNP-based options read precomputed tables describing the families: the sibling annex, and tables derived from it (entropy, quartet, family overlap, pair weights). They are built automatically the first time they are needed and reused afterwards. The options below prepare them in advance, or describe the families.
|
||||||
|
|
||||||
| Option | Description |
|
| Option | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex, then every annex derived from it (entropy, quartet, pair weight, family overlap, under both `--no-ambiguity` conventions). Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the `--no-ambiguity` / `--only-diploid` convention given: `--entropy` the entropy annex, `--quartet` the quartet annex, `--pair-balance` the pair-weight and family-overlap annexes, `--family-overlap` or `--min-shared-family` the family-overlap annex. For example `--sibling-annex --quartet --only-diploid` prepares exactly the quartet annex of that convention, and `--sibling-annex` alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
|
||||||
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
|
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
|
||||||
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
| `--sibling-hist` | Write `<prefix>_sibling_hist.csv` (columns `size,count`): the global family-size histogram (1-4 members) only, without the per-genome breakdown |
|
||||||
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
||||||
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
||||||
| `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index |
|
| `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index |
|
||||||
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
|
|
||||||
|
|
||||||
### `--sibling-stats`
|
### `--sibling-stats`
|
||||||
|
|
||||||
@@ -71,45 +153,6 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
|||||||
|
|
||||||
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
|
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
|
||||||
|
|
||||||
### `--pseudo-alignment`
|
|
||||||
|
|
||||||
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
|
|
||||||
|
|
||||||
| Option | Default | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
|
|
||||||
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
|
||||||
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
|
||||||
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct called states, each carried by at least two genomes. Which genomes count as carriers follows `--no-ambiguity` |
|
|
||||||
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
|
|
||||||
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
|
|
||||||
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
|
|
||||||
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
|
||||||
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
|
||||||
|
|
||||||
## Sampling at scale: `--n-sites`, `--shannon`, `--entropy`
|
|
||||||
|
|
||||||
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
|
|
||||||
|
|
||||||
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
|
|
||||||
|
|
||||||
Every invocation, with or without `--session`, draws its own fresh random sample by default — running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately. `--session` does not change this: it makes a *specific* sample reusable on request, it does not make sampling itself reproducible from one independent run to the next.
|
|
||||||
|
|
||||||
### `--session`: reusing a sample across separate commands
|
|
||||||
|
|
||||||
| Option | Default | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
|
|
||||||
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
|
|
||||||
|
|
||||||
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
|
|
||||||
|
|
||||||
A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
|
|
||||||
|
|
||||||
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
|
||||||
|
|
||||||
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
|
|
||||||
|
|
||||||
### `--shannon`: measuring how informative a family is
|
### `--shannon`: measuring how informative a family is
|
||||||
|
|
||||||
`<prefix>_entropy.csv` has one row per family visited:
|
`<prefix>_entropy.csv` has one row per family visited:
|
||||||
@@ -127,43 +170,27 @@ With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`)
|
|||||||
|
|
||||||
`--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
`--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
||||||
|
|
||||||
### `--quartet`: keeping only families that can resolve a tree
|
### `--shannon-summary`
|
||||||
|
|
||||||
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. Absence is never a state here: a genome that does not carry the family does not count, with or without `--free-loss`. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
|
||||||
|
|
||||||
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` convention, in one pass shared with the other annexes, and built for both conventions by `--sibling-annex`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
|
||||||
|
|
||||||
### Describing a draw: `--sample-stats`, `--sample-plan`
|
|
||||||
|
|
||||||
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
|
|
||||||
|
|
||||||
`--sample-stats` writes three files:
|
|
||||||
|
|
||||||
| File | Content |
|
|
||||||
|---|---|
|
|
||||||
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
|
|
||||||
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
|
|
||||||
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
|
|
||||||
|
|
||||||
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. Excluded genomes are listed with zeros in the first file and count nowhere.
|
|
||||||
|
|
||||||
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
|
|
||||||
|
|
||||||
`--shannon-summary` writes `<prefix>_shannon_summary.csv` (`entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families`): the number of variable families per entropy (`entropy15` over every carrier, `entropy4` over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (`bin_start` is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers `--no-ambiguity` loses, from the same scan as `--shannon` but without one line per family.
|
`--shannon-summary` writes `<prefix>_shannon_summary.csv` (`entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families`): the number of variable families per entropy (`entropy15` over every carrier, `entropy4` over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (`bin_start` is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers `--no-ambiguity` loses, from the same scan as `--shannon` but without one line per family.
|
||||||
|
|
||||||
### `--pair-balance`: representing every pair of genomes
|
## SNP pseudo-alignment
|
||||||
|
|
||||||
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
|
A pseudo-alignment has one column per variable family and one state per genome. Unlike a conventional sequence alignment, its columns come directly from groups of related k-mers of the index. The number of sites is set with `--n-sites`, and the options of "SNP character selection and sampling" above control which families are included and how ambiguous or absent states are written.
|
||||||
|
|
||||||
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` convention, after the family-overlap matrix it depends on, and by `--sibling-annex` for both conventions). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
|
| Option | Description |
|
||||||
|
|---|---|
|
||||||
|
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
|
||||||
|
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
|
||||||
|
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
|
||||||
|
|
||||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
### `--pseudo-alignment`
|
||||||
|
|
||||||
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
|
||||||
|
|
||||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
#### VCF output (`--vcf`)
|
||||||
|
|
||||||
The entropy that weights the draw follows `--no-ambiguity`, so that it describes the same genomes as the alignment. Without the flag, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With the flag, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). The two entropies are saved separately: switching `--no-ambiguity` builds the other one once, without discarding the first, and `--sibling-annex` builds both.
|
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
|
||||||
|
|
||||||
## Sankoff calibration and phylogenetic exports
|
## Sankoff calibration and phylogenetic exports
|
||||||
|
|
||||||
@@ -198,7 +225,7 @@ TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in t
|
|||||||
|
|
||||||
## Output files
|
## Output files
|
||||||
|
|
||||||
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix, if one is computed, is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
|
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, the distance matrix goes to standard output and every other file is written in the current directory under its name without the `<prefix>_` part (for example `nj.nwk`, `entropy.csv`, `alignment.fasta`). All matrices use genome labels as row/column headers, in index order.
|
||||||
|
|
||||||
### Distance matrix
|
### Distance matrix
|
||||||
|
|
||||||
@@ -212,17 +239,19 @@ With `-o/--output PREFIX`, the relevant subset of the files below is written. Wi
|
|||||||
|
|
||||||
CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
|
CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
|
||||||
|
|
||||||
### Central-position SNP model
|
### SNP analyses
|
||||||
|
|
||||||
| File | Written by | Format | Content |
|
| File | Written by | Format | Content |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
|
| `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
|
||||||
|
| `<prefix>_sibling_hist.csv` | `--sibling-hist` | CSV table | global family-size histogram (`size,count`) |
|
||||||
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
|
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
|
||||||
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
|
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see `--shannon` above |
|
||||||
| `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers |
|
| `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers |
|
||||||
| `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes |
|
| `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes |
|
||||||
| `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept |
|
| `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept |
|
||||||
| `<prefix>_alignment.fasta` | `--pseudo-alignment` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
|
| `<prefix>_alignment.fasta` | `--pseudo-alignment` or `--fasta` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
|
||||||
|
| `<prefix>_alignment.vcf` | `--vcf` | VCF | Same sites as the FASTA, with stable virtual coordinates and a k-mer-based `ID` |
|
||||||
|
|
||||||
### Sankoff calibration and exports
|
### Sankoff calibration and exports
|
||||||
|
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ INDEX_PATH="${RUN_DIR}/specimen_index_count/${SPECIMEN}"
|
|||||||
STATS_DIR="${RUN_DIR}/stats/indexing_count"
|
STATS_DIR="${RUN_DIR}/stats/indexing_count"
|
||||||
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
|
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
|
||||||
|
|
||||||
mkdir -p "${STATS_DIR}"
|
mkdir -p "${STATS_DIR}" "$(dirname "${INDEX_PATH}")"
|
||||||
|
|
||||||
r1="${READS_DIR}/reads_R1.fastq.gz"
|
r1="${READS_DIR}/reads_R1.fastq.gz"
|
||||||
r2="${READS_DIR}/reads_R2.fastq.gz"
|
r2="${READS_DIR}/reads_R2.fastq.gz"
|
||||||
@@ -41,7 +41,7 @@ trap 'rm -f "${STDERR_LOG}"' EXIT
|
|||||||
--label "${SPECIMEN}" \
|
--label "${SPECIMEN}" \
|
||||||
--meta "species=${species}" \
|
--meta "species=${species}" \
|
||||||
"${r1}" "${r2}" \
|
"${r1}" "${r2}" \
|
||||||
2>"${STDERR_LOG}"
|
2>"${STDERR_LOG}" || { cat "${STDERR_LOG}" >&2; exit 1; }
|
||||||
|
|
||||||
cat "${STDERR_LOG}" >&2
|
cat "${STDERR_LOG}" >&2
|
||||||
|
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ INDEX_PATH="${RUN_DIR}/specimen_index_presence/${SPECIMEN}"
|
|||||||
STATS_DIR="${RUN_DIR}/stats/indexing_presence"
|
STATS_DIR="${RUN_DIR}/stats/indexing_presence"
|
||||||
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
|
STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
|
||||||
|
|
||||||
mkdir -p "${STATS_DIR}"
|
mkdir -p "${STATS_DIR}" "$(dirname "${INDEX_PATH}")"
|
||||||
|
|
||||||
r1="${READS_DIR}/reads_R1.fastq.gz"
|
r1="${READS_DIR}/reads_R1.fastq.gz"
|
||||||
r2="${READS_DIR}/reads_R2.fastq.gz"
|
r2="${READS_DIR}/reads_R2.fastq.gz"
|
||||||
@@ -40,7 +40,7 @@ trap 'rm -f "${STDERR_LOG}"' EXIT
|
|||||||
--label "${SPECIMEN}" \
|
--label "${SPECIMEN}" \
|
||||||
--meta "species=${species}" \
|
--meta "species=${species}" \
|
||||||
"${r1}" "${r2}" \
|
"${r1}" "${r2}" \
|
||||||
2>"${STDERR_LOG}"
|
2>"${STDERR_LOG}" || { cat "${STDERR_LOG}" >&2; exit 1; }
|
||||||
|
|
||||||
cat "${STDERR_LOG}" >&2
|
cat "${STDERR_LOG}" >&2
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -247,15 +247,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="/theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -263,11 +274,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -276,7 +295,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="/theory/encoding/" class="md-nav__link">
|
<a href="/theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -304,14 +323,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="/theory/entropy_filter/" class="md-nav__link">
|
<a href="/theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -332,7 +351,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="/theory/minimizer_selection/" class="md-nav__link">
|
<a href="/theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -360,7 +379,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="/theory/indexing_architecture/" class="md-nav__link">
|
<a href="/theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -381,6 +428,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -937,6 +1325,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="/formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+430
-14
@@ -9,7 +9,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="prev" href="../formats/index_layout/">
|
<link rel="prev" href="../formats/vcf/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -254,15 +254,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -270,11 +281,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -283,7 +302,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/encoding/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -311,14 +330,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/entropy_filter/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -339,7 +358,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -367,7 +386,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -388,6 +435,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -944,6 +1332,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1157,13 +1573,13 @@
|
|||||||
|
|
||||||
|
|
||||||
<h1 id="architecture-notes-for-advanced-use">Architecture notes for advanced use</h1>
|
<h1 id="architecture-notes-for-advanced-use">Architecture notes for advanced use</h1>
|
||||||
<p>This page describes execution-level behavior relevant to sizing and running <code>obikmer</code> on large datasets or multi-socket machines. It complements the <a href="../formats/index_layout/">index format</a> and <a href="../theory/indexing_architecture/">theory</a> pages.</p>
|
<p>This page describes execution-level behavior relevant to sizing and running <code>obikmer</code> on large datasets or multi-socket machines. It complements the <a href="../formats/index_layout/">index format</a> and <a href="../theory/kmer_indexing/indexing_architecture/">theory</a> pages.</p>
|
||||||
<h2 id="sequence-invariant">Sequence invariant</h2>
|
<h2 id="sequence-invariant">Sequence invariant</h2>
|
||||||
<p>Every input sequence is treated purely as a compact representation of a set of overlapping kmers:</p>
|
<p>Every input sequence is treated purely as a compact representation of a set of overlapping kmers:</p>
|
||||||
<ul>
|
<ul>
|
||||||
<li>Only the <code>A</code>/<code>C</code>/<code>G</code>/<code>T</code> alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.</li>
|
<li>Only the <code>A</code>/<code>C</code>/<code>G</code>/<code>T</code> alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.</li>
|
||||||
<li>Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.</li>
|
<li>Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.</li>
|
||||||
<li>Kmers are always handled in canonical form (see <a href="../theory/encoding/">DNA encoding</a>), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.</li>
|
<li>Kmers are always handled in canonical form (see <a href="../theory/kmer_indexing/encoding/">DNA encoding</a>), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.</li>
|
||||||
</ul>
|
</ul>
|
||||||
<h2 id="index-dimensioning">Index dimensioning</h2>
|
<h2 id="index-dimensioning">Index dimensioning</h2>
|
||||||
<p>An index directory is organized as <code>KmerIndex → partitions → layers</code>, with a canonical kmer belonging to exactly one (partition, layer) pair. This is what makes set operations (merge, filter, distance) parallel and coordination-free across partitions.</p>
|
<p>An index directory is organized as <code>KmerIndex → partitions → layers</code>, with a canonical kmer belonging to exactly one (partition, layer) pair. This is what makes set operations (merge, filter, distance) parallel and coordination-free across partitions.</p>
|
||||||
|
|||||||
@@ -12,7 +12,7 @@
|
|||||||
<link rel="prev" href="../../usage/predicates/">
|
<link rel="prev" href="../../usage/predicates/">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../../architecture/">
|
<link rel="next" href="../vcf/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1039,6 +1427,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1162,9 +1578,9 @@
|
|||||||
|
|
||||||
<h1 id="index-construction-and-on-disk-layout">Index construction and on-disk layout</h1>
|
<h1 id="index-construction-and-on-disk-layout">Index construction and on-disk layout</h1>
|
||||||
<h2 id="construction-pipeline">Construction pipeline</h2>
|
<h2 id="construction-pipeline">Construction pipeline</h2>
|
||||||
<p>Building an index (<a href="../../usage/index_command/"><code>index</code></a>) proceeds through a fixed sequence of phases, each operating independently per partition (see <a href="../../theory/indexing_architecture/">Partitioning and indexing architecture</a>):</p>
|
<p>Building an index (<a href="../../usage/index_command/"><code>index</code></a>) proceeds through a fixed sequence of phases, each operating independently per partition (see <a href="../../theory/kmer_indexing/indexing_architecture/">Partitioning and indexing architecture</a>):</p>
|
||||||
<ol>
|
<ol>
|
||||||
<li><strong>Scatter.</strong> A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see <a href="../../theory/entropy_filter/">Low-complexity kmer filter</a>), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by <code>hash(minimizer) mod n_partitions</code> into one file per partition.</li>
|
<li><strong>Scatter.</strong> A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see <a href="../../theory/kmer_indexing/entropy_filter/">Low-complexity kmer filter</a>), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by <code>hash(minimizer) mod n_partitions</code> into one file per partition.</li>
|
||||||
<li><strong>Dereplication.</strong> Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.</li>
|
<li><strong>Dereplication.</strong> Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.</li>
|
||||||
<li><strong>Exact counting.</strong> Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.</li>
|
<li><strong>Exact counting.</strong> Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.</li>
|
||||||
<li><strong>Quorum filtering.</strong> Kmers outside the <code>--min-abundance</code>/<code>--max-abundance</code> range are dropped, and super-kmers are recompacted around the surviving kmer set.</li>
|
<li><strong>Quorum filtering.</strong> Kmers outside the <code>--min-abundance</code>/<code>--max-abundance</code> range are dropped, and super-kmers are recompacted around the surviving kmer set.</li>
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
+431
-15
@@ -367,15 +367,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -383,11 +394,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -396,7 +415,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="theory/encoding/" class="md-nav__link">
|
<a href="theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -424,14 +443,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="theory/entropy_filter/" class="md-nav__link">
|
<a href="theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -452,7 +471,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="theory/minimizer_selection/" class="md-nav__link">
|
<a href="theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -480,7 +499,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="theory/indexing_architecture/" class="md-nav__link">
|
<a href="theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -501,6 +548,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1057,6 +1445,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1205,11 +1621,11 @@
|
|||||||
<p>All functionality is exposed through a single binary, <code>obikmer</code>, organized as subcommands.</p>
|
<p>All functionality is exposed through a single binary, <code>obikmer</code>, organized as subcommands.</p>
|
||||||
<h2 id="core-principles">Core principles</h2>
|
<h2 id="core-principles">Core principles</h2>
|
||||||
<ul>
|
<ul>
|
||||||
<li>Kmers are of fixed, odd length <span class="arithmatex">\(k\)</span>, chosen at index-construction time in the range <span class="arithmatex">\([11, 31]\)</span> (see <a href="theory/kmers_and_superkmers/">Kmers and super-kmers</a>).</li>
|
<li>Kmers are of fixed, odd length <span class="arithmatex">\(k\)</span>, chosen at index-construction time in the range <span class="arithmatex">\([11, 31]\)</span> (see <a href="theory/kmer_indexing/kmers/">Kmers</a>).</li>
|
||||||
<li>Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see <a href="theory/encoding/">DNA encoding</a>).</li>
|
<li>Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see <a href="theory/kmer_indexing/encoding/">DNA encoding</a>).</li>
|
||||||
<li>Kmers are handled in <strong>canonical form</strong> (<span class="arithmatex">\(\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))\)</span>), making counting strand-independent.</li>
|
<li>Kmers are handled in <strong>canonical form</strong> (<span class="arithmatex">\(\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))\)</span>), making counting strand-independent.</li>
|
||||||
<li>Sequences are decomposed into <strong>super-kmers</strong> before storage, anchored on a hash-selected <strong>minimizer</strong> (see <a href="theory/minimizer_selection/">Minimizer selection</a>), then routed to one of several <strong>partitions</strong> for parallel, memory-bounded processing (see <a href="theory/indexing_architecture/">Partitioning and indexing architecture</a>).</li>
|
<li>Sequences are decomposed into <strong>super-kmers</strong> before storage, anchored on a hash-selected <strong>minimizer</strong> (see <a href="theory/kmer_indexing/minimizer_selection/">Minimizer selection</a>), then routed to one of several <strong>partitions</strong> for parallel, memory-bounded processing (see <a href="theory/kmer_indexing/indexing_architecture/">Partitioning and indexing architecture</a>).</li>
|
||||||
<li>Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see <a href="theory/entropy_filter/">Low-complexity kmer filter</a>).</li>
|
<li>Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see <a href="theory/kmer_indexing/entropy_filter/">Low-complexity kmer filter</a>).</li>
|
||||||
</ul>
|
</ul>
|
||||||
<h2 id="commands">Commands</h2>
|
<h2 id="commands">Commands</h2>
|
||||||
<table>
|
<table>
|
||||||
|
|||||||
+428
-12
@@ -12,7 +12,7 @@
|
|||||||
<link rel="prev" href="..">
|
<link rel="prev" href="..">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../theory/kmers_and_superkmers/">
|
<link rel="next" href="../theory/kmer_indexing/encoding/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -442,15 +442,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -458,11 +469,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -471,7 +490,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/encoding/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -499,14 +518,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/entropy_filter/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -527,7 +546,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -555,7 +574,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -576,6 +623,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1132,6 +1520,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,223 @@
|
|||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" default-locale="en-US" demote-non-dropping-particle="sort-only">
|
||||||
|
<info>
|
||||||
|
<title>Molecular Ecology Modified</title>
|
||||||
|
<id>http://www.zotero.org/styles/molecular-ecology</id>
|
||||||
|
<link href="http://www.zotero.org/styles/molecular-ecology" rel="self"/>
|
||||||
|
<link href="http://www.zotero.org/styles/ecology-letters" rel="template"/>
|
||||||
|
<link href="http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291365-294X/homepage/ForAuthors.html" rel="documentation"/>
|
||||||
|
<author>
|
||||||
|
<name>Xiaodong Dang</name>
|
||||||
|
<email>dangxdong@gmail.com</email>
|
||||||
|
</author>
|
||||||
|
<contributor>
|
||||||
|
<name>Johan Asplund</name>
|
||||||
|
<email>asplundj@gmail.com</email>
|
||||||
|
</contributor>
|
||||||
|
<contributor>
|
||||||
|
<name>David Kaplan</name>
|
||||||
|
<email>david.kaplan@ird.fr</email>
|
||||||
|
</contributor>
|
||||||
|
<category citation-format="author-date"/>
|
||||||
|
<category field="biology"/>
|
||||||
|
<issn>0962-1083</issn>
|
||||||
|
<eissn>1365-294X</eissn>
|
||||||
|
<updated>2012-09-09T22:24:43+00:00</updated>
|
||||||
|
<rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
|
||||||
|
</info>
|
||||||
|
<macro name="editor-translator">
|
||||||
|
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
|
||||||
|
<name initialize-with="" delimiter=", "/>
|
||||||
|
<label form="short" prefix=", " text-case="capitalize-first" suffix="," strip-periods="true"/>
|
||||||
|
</names>
|
||||||
|
</macro>
|
||||||
|
<macro name="author">
|
||||||
|
<names variable="author">
|
||||||
|
<name name-as-sort-order="all" sort-separator=" " initialize-with="" delimiter=", " delimiter-precedes-et-al="never"/>
|
||||||
|
<label form="short" prefix=" (" suffix=")" text-case="capitalize-first"/>
|
||||||
|
<et-al font-style="italic"/>
|
||||||
|
<substitute>
|
||||||
|
<names variable="editor"/>
|
||||||
|
<names variable="translator"/>
|
||||||
|
<text macro="title"/>
|
||||||
|
</substitute>
|
||||||
|
</names>
|
||||||
|
</macro>
|
||||||
|
<macro name="author-short">
|
||||||
|
<names variable="author">
|
||||||
|
<name form="short" and="symbol" delimiter=", " initialize-with=". "/>
|
||||||
|
<et-al font-style="italic"/>
|
||||||
|
<substitute>
|
||||||
|
<names variable="editor"/>
|
||||||
|
<names variable="translator"/>
|
||||||
|
<choose>
|
||||||
|
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||||
|
<text variable="title" form="short" font-style="italic"/>
|
||||||
|
</if>
|
||||||
|
<else>
|
||||||
|
<text variable="title" form="short" quotes="true"/>
|
||||||
|
</else>
|
||||||
|
</choose>
|
||||||
|
</substitute>
|
||||||
|
</names>
|
||||||
|
</macro>
|
||||||
|
<macro name="title">
|
||||||
|
<choose>
|
||||||
|
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||||
|
<text variable="title" font-style="italic"/>
|
||||||
|
</if>
|
||||||
|
<else>
|
||||||
|
<text variable="title"/>
|
||||||
|
</else>
|
||||||
|
</choose>
|
||||||
|
</macro>
|
||||||
|
<macro name="access-detail">
|
||||||
|
<group delimiter=" ">
|
||||||
|
<text variable="URL" prefix="Available at: "/>
|
||||||
|
<group prefix="(" suffix=")" delimiter=" ">
|
||||||
|
<text term="accessed"/>
|
||||||
|
<date variable="accessed">
|
||||||
|
<date-part name="day" suffix=" "/>
|
||||||
|
<date-part name="month" form="long" suffix=" "/>
|
||||||
|
<date-part name="year"/>
|
||||||
|
</date>
|
||||||
|
</group>
|
||||||
|
</group>
|
||||||
|
</macro>
|
||||||
|
<macro name="access">
|
||||||
|
<choose>
|
||||||
|
<if type="webpage post post-weblog entry entry-encyclopedia dataset software report manuscript document" match="any">
|
||||||
|
<text macro="access-detail"/>
|
||||||
|
</if>
|
||||||
|
<else-if variable="accessed">
|
||||||
|
<text macro="access-detail"/>
|
||||||
|
</else-if>
|
||||||
|
</choose>
|
||||||
|
</macro>
|
||||||
|
<macro name="access-fallback">
|
||||||
|
<group delimiter=" ">
|
||||||
|
<choose>
|
||||||
|
<if variable="URL">
|
||||||
|
<text macro="access-detail" suffix="."/>
|
||||||
|
</if>
|
||||||
|
<else-if variable="publisher">
|
||||||
|
<text variable="publisher" prefix="Available at: " suffix="."/>
|
||||||
|
</else-if>
|
||||||
|
</choose>
|
||||||
|
<text variable="note" suffix="."/>
|
||||||
|
</group>
|
||||||
|
</macro>
|
||||||
|
<macro name="issued-year">
|
||||||
|
<choose>
|
||||||
|
<if variable="issued">
|
||||||
|
<date variable="issued">
|
||||||
|
<date-part name="year"/>
|
||||||
|
</date>
|
||||||
|
</if>
|
||||||
|
<else>
|
||||||
|
<text term="no date" form="short"/>
|
||||||
|
</else>
|
||||||
|
</choose>
|
||||||
|
</macro>
|
||||||
|
<macro name="publisher">
|
||||||
|
<group delimiter=", ">
|
||||||
|
<text variable="publisher"/>
|
||||||
|
<text variable="publisher-place"/>
|
||||||
|
</group>
|
||||||
|
</macro>
|
||||||
|
<citation collapse="year-suffix" et-al-min="3" et-al-use-first="1" disambiguate-add-year-suffix="true">
|
||||||
|
<sort>
|
||||||
|
<key variable="issued"/>
|
||||||
|
<key variable="author"/>
|
||||||
|
</sort>
|
||||||
|
<layout prefix="(" suffix=")" delimiter="; ">
|
||||||
|
<group delimiter=" ">
|
||||||
|
<text macro="author-short"/>
|
||||||
|
<text macro="issued-year"/>
|
||||||
|
<group>
|
||||||
|
<label variable="locator" form="short"/>
|
||||||
|
<text variable="locator" prefix=" "/>
|
||||||
|
</group>
|
||||||
|
</group>
|
||||||
|
</layout>
|
||||||
|
</citation>
|
||||||
|
<bibliography hanging-indent="true" et-al-min="6" et-al-use-first="1000" entry-spacing="0.5" line-spacing="1">
|
||||||
|
<sort>
|
||||||
|
<key macro="author-short"/>
|
||||||
|
<key variable="issued"/>
|
||||||
|
</sort>
|
||||||
|
<layout>
|
||||||
|
<text macro="author" suffix=" "/>
|
||||||
|
<text macro="issued-year" prefix="(" suffix=")"/>
|
||||||
|
<choose>
|
||||||
|
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||||
|
<group suffix=".">
|
||||||
|
<text macro="title" prefix=" "/>
|
||||||
|
<text macro="editor-translator" prefix=" "/>
|
||||||
|
</group>
|
||||||
|
<text prefix=" " suffix="." macro="publisher"/>
|
||||||
|
<text macro="access" prefix=" " suffix="."/>
|
||||||
|
</if>
|
||||||
|
<else-if type="thesis" match="any">
|
||||||
|
<group suffix=".">
|
||||||
|
<text macro="title" prefix=" " suffix="."/>
|
||||||
|
<text variable="genre" prefix=" " suffix=" Thesis."/>
|
||||||
|
<text prefix=" " macro="publisher"/>
|
||||||
|
</group>
|
||||||
|
<text macro="access" prefix=" " suffix="."/>
|
||||||
|
</else-if>
|
||||||
|
<else-if type="chapter paper-conference" match="any">
|
||||||
|
<text macro="title" prefix=" "/>
|
||||||
|
<group prefix=".">
|
||||||
|
<group prefix=" In:">
|
||||||
|
<text variable="container-title" font-style="italic" prefix=" "/>
|
||||||
|
<text variable="collection-title" prefix=" " suffix="."/>
|
||||||
|
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
|
||||||
|
<label form="short" suffix=" " strip-periods="true"/>
|
||||||
|
<name name-as-sort-order="all" sort-separator=" " initialize-with="" delimiter=", " delimiter-precedes-last="never"/>
|
||||||
|
</names>
|
||||||
|
<group suffix=".">
|
||||||
|
<label variable="page" form="short" prefix=", " suffix=" "/>
|
||||||
|
<text variable="page"/>
|
||||||
|
<text macro="publisher" prefix=". " suffix="."/>
|
||||||
|
</group>
|
||||||
|
</group>
|
||||||
|
</group>
|
||||||
|
</else-if>
|
||||||
|
<else-if type="webpage post post-weblog entry entry-encyclopedia dataset software" match="any">
|
||||||
|
<group suffix=".">
|
||||||
|
<text macro="title" prefix=" "/>
|
||||||
|
<text macro="editor-translator" prefix=" "/>
|
||||||
|
</group>
|
||||||
|
<group prefix=" " suffix=".">
|
||||||
|
<text variable="container-title" text-case="title" font-style="italic"/>
|
||||||
|
<text variable="genre" prefix=", "/>
|
||||||
|
<text macro="publisher" prefix=". "/>
|
||||||
|
</group>
|
||||||
|
<text macro="access" prefix=" " suffix="."/>
|
||||||
|
</else-if>
|
||||||
|
<else>
|
||||||
|
<group suffix=".">
|
||||||
|
<text macro="title" prefix=" "/>
|
||||||
|
<text macro="editor-translator" prefix=" "/>
|
||||||
|
</group>
|
||||||
|
<group prefix=" " suffix=".">
|
||||||
|
<text variable="container-title" text-case="title" font-style="italic"/>
|
||||||
|
<group prefix=", ">
|
||||||
|
<text variable="volume" font-weight="bold"/>
|
||||||
|
</group>
|
||||||
|
<text variable="page" prefix=", "/>
|
||||||
|
</group>
|
||||||
|
<choose>
|
||||||
|
<if variable="container-title">
|
||||||
|
<text macro="access" prefix=" " suffix="."/>
|
||||||
|
</if>
|
||||||
|
<else>
|
||||||
|
<text macro="access-fallback" prefix=" "/>
|
||||||
|
</else>
|
||||||
|
</choose>
|
||||||
|
</else>
|
||||||
|
</choose>
|
||||||
|
</layout>
|
||||||
|
</bibliography>
|
||||||
|
</style>
|
||||||
@@ -259,3 +259,601 @@
|
|||||||
url = {https://doi.org/10.1186/s12864-015-1647-5},
|
url = {https://doi.org/10.1186/s12864-015-1647-5},
|
||||||
volume = 16,
|
volume = 16,
|
||||||
year = 2015}
|
year = 2015}
|
||||||
|
|
||||||
|
@article{Roberts2004-rz,
|
||||||
|
title = "{Reducing storage requirements for biological sequence
|
||||||
|
comparison}",
|
||||||
|
author = "Roberts, Michael and Hayes, Wayne and Hunt, Brian R and Mount,
|
||||||
|
Stephen M and Yorke, James A",
|
||||||
|
journal = "Bioinformatics (Oxford, England)",
|
||||||
|
publisher = "Oxford University Press (OUP)",
|
||||||
|
volume = 20,
|
||||||
|
number = 18,
|
||||||
|
pages = "3363--3369",
|
||||||
|
abstract = "MOTIVATION: Comparison of nucleic acid and protein sequences is a
|
||||||
|
fundamental tool of modern bioinformatics. A dominant method of
|
||||||
|
such string matching is the 'seed-and-extend' approach, in which
|
||||||
|
occurrences of short subsequences called 'seeds' are used to
|
||||||
|
search for potentially longer matches in a large database of
|
||||||
|
sequences. Each such potential match is then checked to see if it
|
||||||
|
extends beyond the seed. To be effective, the seed-and-extend
|
||||||
|
approach needs to catalogue seeds from virtually every substring
|
||||||
|
in the database of search strings. Projects such as mammalian
|
||||||
|
genome assemblies and large-scale protein matching, however, have
|
||||||
|
such large sequence databases that the resulting list of seeds
|
||||||
|
cannot be stored in RAM on a single computer. This significantly
|
||||||
|
slows the matching process. RESULTS: We present a simple and
|
||||||
|
elegant method in which only a small fraction of seeds, called
|
||||||
|
'minimizers', needs to be stored. Using minimizers can speed up
|
||||||
|
string-matching computations by a large factor while missing only
|
||||||
|
a small fraction of the matches found using all seeds.",
|
||||||
|
month = dec,
|
||||||
|
year = 2004,
|
||||||
|
url = "https://dx.doi.org/10.1093/bioinformatics/bth408",
|
||||||
|
doi = "10.1093/bioinformatics/bth408",
|
||||||
|
pmid = 15256412,
|
||||||
|
issn = "1367-4803,1367-4811",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
|
||||||
|
@INPROCEEDINGS{Steele2014-cz,
|
||||||
|
title = "{Fast splittable pseudorandom number generators}",
|
||||||
|
author = "Steele, Jr, Guy L and Lea, Doug and Flood, Christine H",
|
||||||
|
booktitle = "{Proceedings of the 2014 ACM International Conference on Object
|
||||||
|
Oriented Programming Systems Languages \& Applications}",
|
||||||
|
publisher = "ACM",
|
||||||
|
address = "New York, NY, USA",
|
||||||
|
abstract = "We describe a new algorithm SplitMix for an object-oriented and
|
||||||
|
splittable pseudorandom number generator (PRNG) that is quite
|
||||||
|
fast: 9 64-bit arithmetic/logical operations per 64 bits
|
||||||
|
generated. A conventional linear PRNG object provides a generate
|
||||||
|
method that returns one pseudorandom value and updates the state
|
||||||
|
of the PRNG, but a splittable PRNG object also has a second
|
||||||
|
operation, split, that replaces the original PRNG object with two
|
||||||
|
(seemingly) independent PRNG objects, by creating and returning a
|
||||||
|
new such object and updating the state of the original object.
|
||||||
|
Splittable PRNG objects make it easy to organize the use of
|
||||||
|
pseudorandom numbers in multithreaded programs structured using
|
||||||
|
fork-join parallelism. No locking or synchronization is required
|
||||||
|
(other than the usual memory fence immediately after object
|
||||||
|
creation). Because the generate method has no loops or
|
||||||
|
conditionals, it is suitable for SIMD or GPU implementation. We
|
||||||
|
derive SplitMix from the DotMix algorithm of Leiserson, Schardl,
|
||||||
|
and Sukha by making a series of program transformations and
|
||||||
|
engineering improvements. The end result is an object-oriented
|
||||||
|
version of the purely functional API used in the Haskell library
|
||||||
|
for over a decade, but SplitMix is faster and produces
|
||||||
|
pseudorandom sequences of higher quality; it is also far superior
|
||||||
|
in quality and speed to java.util.Random, and has been included
|
||||||
|
in Java JDK8 as the class java.util.SplittableRandom. We have
|
||||||
|
tested the pseudorandom sequences produced by SplitMix using two
|
||||||
|
standard statistical test suites (DieHarder and TestU01) and they
|
||||||
|
appear to be adequate for ``everyday'' use, such as in Monte
|
||||||
|
Carlo algorithms and randomized data structures where speed is
|
||||||
|
important.",
|
||||||
|
month = oct,
|
||||||
|
year = 2014,
|
||||||
|
url = "http://dx.doi.org/10.1145/2660193.2660195",
|
||||||
|
doi = "10.1145/2660193.2660195",
|
||||||
|
isbn = 9781450325851
|
||||||
|
}
|
||||||
|
|
||||||
|
@online{Stafford2011,
|
||||||
|
author = {Stafford, David},
|
||||||
|
title = {Better Bit Mixing - Improving on MurmurHash3's 64-bit Finalizer},
|
||||||
|
date = {2011-09-28},
|
||||||
|
url = {https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html},
|
||||||
|
urldate = {2026-09-12}
|
||||||
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Hamming1950-cw,
|
||||||
|
title = "{Error detecting and error correcting codes}",
|
||||||
|
author = "Hamming, R W",
|
||||||
|
journal = "The Bell System Technical Journal",
|
||||||
|
publisher = "Institute of Electrical and Electronics Engineers (IEEE)",
|
||||||
|
volume = 29,
|
||||||
|
number = 2,
|
||||||
|
pages = "147--160",
|
||||||
|
abstract = "The author was led to the study given in this paper from a
|
||||||
|
consideration of large scale computing machines in which a large
|
||||||
|
number of operations must be performed without a single error in
|
||||||
|
the end result. This problem of “doing things right” on a large
|
||||||
|
scale is not essentially new; in a telephone central office, for
|
||||||
|
example, a very large number of operations are performed while
|
||||||
|
the errors leading to wrong numbers are kept well under control,
|
||||||
|
though they have not been completely eliminated. This has been
|
||||||
|
achieved, in part, through the use of self-checking circuits. The
|
||||||
|
occasional failure that escapes routine checking is still
|
||||||
|
detected by the customer and will, if it persists, result in
|
||||||
|
customer complaint, while if it is transient it will produce only
|
||||||
|
occasional wrong numbers. At the same time the rest of the
|
||||||
|
central office functions satisfactorily. In a digital computer,
|
||||||
|
on the other hand, a single failure usually means the complete
|
||||||
|
failure, in the sense that if it is detected no more computing
|
||||||
|
can be done until the failure is located and corrected, while if
|
||||||
|
it escapes detection then it invalidates all subsequent
|
||||||
|
operations of the machine. Put in other words, in a telephone
|
||||||
|
central office there are a number of parallel paths which are
|
||||||
|
more or less independent of each other; in a digital machine
|
||||||
|
there is usually a single long path which passes through the same
|
||||||
|
piece of equipment many, many times before the answer is
|
||||||
|
obtained.",
|
||||||
|
month = apr,
|
||||||
|
year = 1950,
|
||||||
|
url = "http://dx.doi.org/10.1002/j.1538-7305.1950.tb00463.x",
|
||||||
|
doi = "10.1002/j.1538-7305.1950.tb00463.x",
|
||||||
|
issn = "0005-8580"
|
||||||
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Ondov2016-dq,
|
||||||
|
title = "{Mash: fast genome and metagenome distance estimation using
|
||||||
|
{MinHash}}",
|
||||||
|
author = "Ondov, Brian D and Treangen, Todd J and Melsted, Páll and
|
||||||
|
Mallonee, Adam B and Bergman, Nicholas H and Koren, Sergey and
|
||||||
|
Phillippy, Adam M",
|
||||||
|
journal = "Genome biology",
|
||||||
|
volume = 17,
|
||||||
|
number = 1,
|
||||||
|
pages = 132,
|
||||||
|
abstract = "Mash extends the MinHash dimensionality-reduction technique to
|
||||||
|
include a pairwise mutation distance and P value significance
|
||||||
|
test, enabling the efficient clustering and search of massive
|
||||||
|
sequence collections. Mash reduces large sequences and sequence
|
||||||
|
sets to small, representative sketches, from which global mutation
|
||||||
|
distances can be rapidly estimated. We demonstrate several use
|
||||||
|
cases, including the clustering of all 54,118 NCBI RefSeq genomes
|
||||||
|
in 33 CPU h; real-time database search using assembled or
|
||||||
|
unassembled Illumina, Pacific Biosciences, and Oxford Nanopore
|
||||||
|
data; and the scalable clustering of hundreds of metagenomic
|
||||||
|
samples by composition. Mash is freely released under a BSD
|
||||||
|
license ( https://github.com/marbl/mash ).",
|
||||||
|
month = jun,
|
||||||
|
year = 2016,
|
||||||
|
url = "http://dx.doi.org/10.1186/s13059-016-0997-x",
|
||||||
|
keywords = "Alignment; Comparative genomics; Genomic distance; Metagenomics;
|
||||||
|
Nanopore; Sequencing",
|
||||||
|
doi = "10.1186/s13059-016-0997-x",
|
||||||
|
pmc = "PMC4915045",
|
||||||
|
pmid = 27323842,
|
||||||
|
issn = "1465-6906",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Bray1957-vv,
|
||||||
|
title = "{An ordination of the upland forest communities of southern
|
||||||
|
Wisconsin}",
|
||||||
|
author = "Bray, J Roger and Curtis, J T",
|
||||||
|
journal = "Ecological Monographs",
|
||||||
|
publisher = "Wiley",
|
||||||
|
volume = 27,
|
||||||
|
number = 4,
|
||||||
|
pages = "325--349",
|
||||||
|
month = oct,
|
||||||
|
year = 1957,
|
||||||
|
url = "http://dx.doi.org/10.2307/1942268",
|
||||||
|
doi = "10.2307/1942268",
|
||||||
|
issn = "0012-9615,1557-7015",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Gower1986-ri,
|
||||||
|
title = "{Metric and Euclidean properties of dissimilarity coefficients}",
|
||||||
|
author = "Gower, J C and Legendre, P",
|
||||||
|
journal = "Journal of Classification",
|
||||||
|
publisher = "Springer Science and Business Media LLC",
|
||||||
|
volume = 3,
|
||||||
|
number = 1,
|
||||||
|
pages = "5--48",
|
||||||
|
abstract = "We assemble here properties of certain dissimilarity coefficients
|
||||||
|
and are specially concerned with their metric and Euclidean
|
||||||
|
status. No attempt is made to be exhaustive as far as
|
||||||
|
coefficients are concerned, but certain mathematical results that
|
||||||
|
we have found useful are presented and should help establish
|
||||||
|
similar properties for other coefficients. The response to
|
||||||
|
different types of data is investigated, leading to guidance on
|
||||||
|
the choice of an appropriate coefficient.",
|
||||||
|
month = mar,
|
||||||
|
year = 1986,
|
||||||
|
url = "http://dx.doi.org/10.1007/bf01896809",
|
||||||
|
doi = "10.1007/bf01896809",
|
||||||
|
issn = "0176-4268,1432-1343",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Legendre2001-cq,
|
||||||
|
title = "{Ecologically meaningful transformations for ordination of species
|
||||||
|
data}",
|
||||||
|
author = "Legendre, Pierre and Gallagher, Eugene D",
|
||||||
|
journal = "Oecologia",
|
||||||
|
volume = 129,
|
||||||
|
number = 2,
|
||||||
|
pages = "271--280",
|
||||||
|
abstract = "This paper examines how to obtain species biplots in unconstrained
|
||||||
|
or constrained ordination without resorting to the Euclidean
|
||||||
|
distance [used in principal-component analysis (PCA) and
|
||||||
|
redundancy analysis (RDA)] or the chi-square distance [preserved
|
||||||
|
in correspondence analysis (CA) and canonical correspondence
|
||||||
|
analysis (CCA)] which are not always appropriate for the analysis
|
||||||
|
of community composition data. To achieve this goal,
|
||||||
|
transformations are proposed for species data tables. They allow
|
||||||
|
ecologists to use ordination methods such as PCA and RDA, which
|
||||||
|
are Euclidean-based, for the analysis of community data, while
|
||||||
|
circumventing the problems associated with the Euclidean distance,
|
||||||
|
and avoiding CA and CCA which present problems of their own in
|
||||||
|
some cases. This allows the use of the original (transformed)
|
||||||
|
species data in RDA carried out to test for relationships with
|
||||||
|
explanatory variables (i.e. environmental variables, or factors of
|
||||||
|
a multifactorial analysis-of-variance model); ecologists can then
|
||||||
|
draw biplots displaying the relationships of the species to the
|
||||||
|
explanatory variables. Another application allows the use of
|
||||||
|
species data in other methods of multivariate data analysis which
|
||||||
|
optimize a least-squares loss function; an example is K-means
|
||||||
|
partitioning.",
|
||||||
|
month = oct,
|
||||||
|
year = 2001,
|
||||||
|
url = "http://dx.doi.org/10.1007/s004420100716",
|
||||||
|
keywords = "Biplot diagram; Canonical correspondence analysis; Correspondence
|
||||||
|
analysis; Principal-component analysis; Redundancy analysis",
|
||||||
|
doi = "10.1007/s004420100716",
|
||||||
|
pmid = 28547606,
|
||||||
|
issn = "0029-8549,1432-1939",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Anderson2011-rq,
|
||||||
|
title = "{Navigating the multiple meanings of β diversity: a roadmap for
|
||||||
|
the practicing ecologist}",
|
||||||
|
author = "Anderson, Marti J and Crist, Thomas O and Chase, Jonathan M and
|
||||||
|
Vellend, Mark and Inouye, Brian D and Freestone, Amy L and
|
||||||
|
Sanders, Nathan J and Cornell, Howard V and Comita, Liza S and
|
||||||
|
Davies, Kendi F and Harrison, Susan P and Kraft, Nathan J B and
|
||||||
|
Stegen, James C and Swenson, Nathan G",
|
||||||
|
journal = "Ecology Letters",
|
||||||
|
publisher = "Wiley",
|
||||||
|
volume = 14,
|
||||||
|
number = 1,
|
||||||
|
pages = "19--28",
|
||||||
|
abstract = "A recent increase in studies of β diversity has yielded a
|
||||||
|
confusing array of concepts, measures and methods. Here, we
|
||||||
|
provide a roadmap of the most widely used and ecologically
|
||||||
|
relevant approaches for analysis through a series of mission
|
||||||
|
statements. We distinguish two types of β diversity: directional
|
||||||
|
turnover along a gradient vs. non-directional variation.
|
||||||
|
Different measures emphasize different properties of ecological
|
||||||
|
data. Such properties include the degree of emphasis on
|
||||||
|
presence/absence vs. relative abundance information and the
|
||||||
|
inclusion vs. exclusion of joint absences. Judicious use of
|
||||||
|
multiple measures in concert can uncover the underlying nature of
|
||||||
|
patterns in β diversity for a given dataset. A case study of
|
||||||
|
Indonesian coral assemblages shows the utility of a multi-faceted
|
||||||
|
approach. We advocate careful consideration of relevant
|
||||||
|
questions, matched by appropriate analyses. The rigorous
|
||||||
|
application of null models will also help to reveal potential
|
||||||
|
processes driving observed patterns in β diversity.",
|
||||||
|
month = jan,
|
||||||
|
year = 2011,
|
||||||
|
url = "http://dx.doi.org/10.1111/j.1461-0248.2010.01552.x",
|
||||||
|
doi = "10.1111/j.1461-0248.2010.01552.x",
|
||||||
|
pmid = 21070562,
|
||||||
|
issn = "1461-023X,1461-0248",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Jaccard1901-mx,
|
||||||
|
title = "{Étude comparative de la distribution florale dans une portion
|
||||||
|
des Alpes et du Jura}",
|
||||||
|
author = "Jaccard, Paul",
|
||||||
|
journal = "Bulletin de la Société vaudoise des sciences naturelles",
|
||||||
|
publisher = "Imprimerie Corbaz \& Comp.",
|
||||||
|
volume = 37,
|
||||||
|
number = 142,
|
||||||
|
pages = "547--579",
|
||||||
|
abstract = "Schweizer Zeitschriften online - Ein Service der ETH-Bibliothek.
|
||||||
|
Revues suisses en ligne - Un service de ETH Library. Swiss
|
||||||
|
journals online - A service by ETH Library.",
|
||||||
|
year = 1901,
|
||||||
|
url = "http://dx.doi.org/10.5169/seals-266450",
|
||||||
|
doi = "10.5169/seals-266450",
|
||||||
|
issn = "0037-9603",
|
||||||
|
language = "de"
|
||||||
|
}
|
||||||
|
@ARTICLE{Kosub2019-xt,
|
||||||
|
title = "{A note on the triangle inequality for the Jaccard distance}",
|
||||||
|
author = "Kosub, Sven",
|
||||||
|
journal = "Pattern Recognition Letters",
|
||||||
|
publisher = "Elsevier BV",
|
||||||
|
volume = 120,
|
||||||
|
pages = "36--38",
|
||||||
|
month = apr,
|
||||||
|
year = 2019,
|
||||||
|
url = "http://dx.doi.org/10.1016/j.patrec.2018.12.007",
|
||||||
|
doi = "10.1016/j.patrec.2018.12.007",
|
||||||
|
issn = "0167-8655,1872-7344",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Fleischmann2014-ed,
|
||||||
|
title = {Evolution of genome size and chromosome number in the
|
||||||
|
carnivorous plant genus Genlisea (Lentibulariaceae), with a new
|
||||||
|
estimate of the minimum genome size in angiosperms},
|
||||||
|
author = "Fleischmann, Andreas and Michael, Todd P and Rivadavia, Fernando
|
||||||
|
and Sousa, Aretuza and Wang, Wenqin and Temsch, Eva M and
|
||||||
|
Greilhuber, Johann and Müller, Kai F and Heubl, Günther",
|
||||||
|
journal = "Annals of Botany",
|
||||||
|
publisher = "Oxford University Press (OUP)",
|
||||||
|
volume = 114,
|
||||||
|
number = 8,
|
||||||
|
pages = "1651--1663",
|
||||||
|
abstract = "BACKGROUND AND AIMS: Some species of Genlisea possess ultrasmall
|
||||||
|
nuclear genomes, the smallest known among angiosperms, and some
|
||||||
|
have been found to have chromosomes of diminutive size, which may
|
||||||
|
explain why chromosome numbers and karyotypes are not known for
|
||||||
|
the majority of species of the genus. However, other members of
|
||||||
|
the genus do not possess ultrasmall genomes, nor do most taxa
|
||||||
|
studied in related genera of the family or order. This study
|
||||||
|
therefore examined the evolution of genome sizes and chromosome
|
||||||
|
numbers in Genlisea in a phylogenetic context. The correlations
|
||||||
|
of genome size with chromosome number and size, with the
|
||||||
|
phylogeny of the group and with growth forms and habitats were
|
||||||
|
also examined. METHODS: Nuclear genome sizes were measured from
|
||||||
|
cultivated plant material for a comprehensive sampling of taxa,
|
||||||
|
including nearly half of all species of Genlisea and representing
|
||||||
|
all major lineages. Flow cytometric measurements were conducted
|
||||||
|
in parallel in two laboratories in order to compare the
|
||||||
|
consistency of different methods and controls. Chromosome counts
|
||||||
|
were performed for the majority of taxa, comparing different
|
||||||
|
staining techniques for the ultrasmall chromosomes. KEY RESULTS:
|
||||||
|
Genome sizes of 15 taxa of Genlisea are presented and interpreted
|
||||||
|
in a phylogenetic context. A high degree of congruence was found
|
||||||
|
between genome size distribution and the major phylogenetic
|
||||||
|
lineages. Ultrasmall genomes with 1C values of <100 Mbp were
|
||||||
|
almost exclusively found in a derived lineage of South American
|
||||||
|
species. The ancestral haploid chromosome number was inferred to
|
||||||
|
be n = 8. Chromosome numbers in Genlisea ranged from 2n = 2x = 16
|
||||||
|
to 2n = 4x = 32. Ascendant dysploid series (2n = 36, 38) are
|
||||||
|
documented for three derived taxa. The different ploidy levels
|
||||||
|
corresponded to the two subgenera, but were not directly
|
||||||
|
correlated to differences in genome size; the three different
|
||||||
|
karyotype ranges mirrored the different sections of the genus.
|
||||||
|
The smallest known plant genomes were not found in G. margaretae,
|
||||||
|
as previously reported, but in G. tuberosa (1C ≈ 61 Mbp) and some
|
||||||
|
strains of G. aurea (1C ≈ 64 Mbp). CONCLUSIONS: Genlisea is an
|
||||||
|
ideal candidate model organism for the understanding of genome
|
||||||
|
reduction as the genus includes species with both relatively
|
||||||
|
large (∼1700 Mbp) and ultrasmall (∼61 Mbp) genomes. This
|
||||||
|
comparative, phylogeny-based analysis of genome sizes and
|
||||||
|
karyotypes in Genlisea provides essential data for selection of
|
||||||
|
suitable species for comparative whole-genome analyses, as well
|
||||||
|
as for further studies on both the molecular and cytogenetic
|
||||||
|
basis of genome reduction in plants.",
|
||||||
|
month = dec,
|
||||||
|
year = 2014,
|
||||||
|
url = "http://dx.doi.org/10.1093/aob/mcu189",
|
||||||
|
keywords = "Bladderwort; Genlisea; Lamiales; Lentibulariaceae; carnivorous
|
||||||
|
plant; chromosome number; flow cytometry; genome miniaturization;
|
||||||
|
genome size",
|
||||||
|
doi = "10.1093/aob/mcu189",
|
||||||
|
pmc = "PMC4649684",
|
||||||
|
pmid = 25274549,
|
||||||
|
issn = "0305-7364,1095-8290",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Fernandez2024-kd,
|
||||||
|
title = "A 160 Gbp fork fern genome shatters size record for eukaryotes",
|
||||||
|
author = "Fernández, Pol and Amice, Rémy and Bruy, David and Christenhusz,
|
||||||
|
Maarten J M and Leitch, Ilia J and Leitch, Andrew L and Pokorny,
|
||||||
|
Lisa and Hidalgo, Oriane and Pellicer, Jaume",
|
||||||
|
journal = "iScience",
|
||||||
|
publisher = "Elsevier BV",
|
||||||
|
volume = 27,
|
||||||
|
number = 6,
|
||||||
|
pages = 109889,
|
||||||
|
abstract = "Vascular plants are exceptional among eukaryotes due to their
|
||||||
|
outstanding genome size diversity which ranges ∼2,400-fold,
|
||||||
|
including the largest genome so far recorded in the angiosperm
|
||||||
|
Paris japonica (148.89 Gbp/1C). Despite available data showing
|
||||||
|
that giant genomes are restricted across the Tree of Life, the
|
||||||
|
biological limits to genome size expansion remain to be
|
||||||
|
established. Here, we report the discovery of an even larger
|
||||||
|
eukaryotic genome in Tmesipteris oblanceolata, a New Caledonian
|
||||||
|
fork fern. At 160.45 Gbp/1C, this record-breaking genome
|
||||||
|
challenges current understanding and opens new avenues to explore
|
||||||
|
the evolutionary dynamics of genomic gigantism.",
|
||||||
|
month = jun,
|
||||||
|
year = 2024,
|
||||||
|
url = "http://dx.doi.org/10.1016/j.isci.2024.109889",
|
||||||
|
keywords = "Plant evolution; Plant genetics",
|
||||||
|
doi = "10.1016/j.isci.2024.109889",
|
||||||
|
pmc = "PMC11270024",
|
||||||
|
pmid = 39055604,
|
||||||
|
issn = "2589-0042",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Fraser1995-jc,
|
||||||
|
title = "The minimal gene complement of Mycoplasma genitalium",
|
||||||
|
author = "Fraser, C M and Gocayne, J D and White, O and Adams, M D and
|
||||||
|
Clayton, R A and Fleischmann, R D and Bult, C J and Kerlavage, A
|
||||||
|
R and Sutton, G and Kelley, J M and Fritchman, R D and Weidman, J
|
||||||
|
F and Small, K V and Sandusky, M and Fuhrmann, J and Nguyen, D
|
||||||
|
and Utterback, T R and Saudek, D M and Phillips, C A and Merrick,
|
||||||
|
J M and Tomb, J F and Dougherty, B A and Bott, K F and Hu, P C
|
||||||
|
and Lucier, T S and Peterson, S N and Smith, H O and Hutchison,
|
||||||
|
3rd, C A and Venter, J C",
|
||||||
|
journal = "Science (New York, N.Y.)",
|
||||||
|
publisher = "American Association for the Advancement of Science (AAAS)",
|
||||||
|
volume = 270,
|
||||||
|
number = 5235,
|
||||||
|
pages = "397--403",
|
||||||
|
abstract = "The complete nucleotide sequence (580,070 base pairs) of the
|
||||||
|
Mycoplasma genitalium genome, the smallest known genome of any
|
||||||
|
free-living organism, has been determined by whole-genome random
|
||||||
|
sequencing and assembly. A total of only 470 predicted coding
|
||||||
|
regions were identified that include genes required for DNA
|
||||||
|
replication, transcription and translation, DNA repair, cellular
|
||||||
|
transport, and energy metabolism. Comparison of this genome to
|
||||||
|
that of Haemophilus influenzae suggests that differences in
|
||||||
|
genome content are reflected as profound differences in
|
||||||
|
physiology and metabolic capacity between these two organisms.",
|
||||||
|
month = oct,
|
||||||
|
year = 1995,
|
||||||
|
url = "http://dx.doi.org/10.1126/science.270.5235.397",
|
||||||
|
doi = "10.1126/science.270.5235.397",
|
||||||
|
pmid = 7569993,
|
||||||
|
issn = "0036-8075,1095-9203",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Schneiker2007-gs,
|
||||||
|
title = "Complete genome sequence of the myxobacterium Sorangium
|
||||||
|
cellulosum",
|
||||||
|
author = "Schneiker, Susanne and Perlova, Olena and Kaiser, Olaf and Gerth,
|
||||||
|
Klaus and Alici, Aysel and Altmeyer, Matthias O and Bartels,
|
||||||
|
Daniela and Bekel, Thomas and Beyer, Stefan and Bode, Edna and
|
||||||
|
Bode, Helge B and Bolten, Christoph J and Choudhuri, Jomuna V and
|
||||||
|
Doss, Sabrina and Elnakady, Yasser A and Frank, Bettina and
|
||||||
|
Gaigalat, Lars and Goesmann, Alexander and Groeger, Carolin and
|
||||||
|
Gross, Frank and Jelsbak, Lars and Jelsbak, Lotte and Kalinowski,
|
||||||
|
Jörn and Kegler, Carsten and Knauber, Tina and Konietzny,
|
||||||
|
Sebastian and Kopp, Maren and Krause, Lutz and Krug, Daniel and
|
||||||
|
Linke, Bukhard and Mahmud, Taifo and Martinez-Arias, Rosa and
|
||||||
|
McHardy, Alice C and Merai, Michelle and Meyer, Folker and
|
||||||
|
Mormann, Sascha and Muñoz-Dorado, Jose and Perez, Juana and
|
||||||
|
Pradella, Silke and Rachid, Shwan and Raddatz, Günter and
|
||||||
|
Rosenau, Frank and Rückert, Christian and Sasse, Florenz and
|
||||||
|
Scharfe, Maren and Schuster, Stephan C and Suen, Garret and
|
||||||
|
Treuner-Lange, Anke and Velicer, Gregory J and Vorhölter,
|
||||||
|
Frank-Jörg and Weissman, Kira J and Welch, Roy D and Wenzel,
|
||||||
|
Silke C and Whitworth, David E and Wilhelm, Susanne and Wittmann,
|
||||||
|
Christoph and Blöcker, Helmut and Pühler, Alfred and Müller, Rolf",
|
||||||
|
journal = "Nature Biotechnology",
|
||||||
|
publisher = "Springer Science and Business Media LLC",
|
||||||
|
volume = 25,
|
||||||
|
number = 11,
|
||||||
|
pages = "1281--1289",
|
||||||
|
abstract = "The genus Sorangium synthesizes approximately half of the
|
||||||
|
secondary metabolites isolated from myxobacteria, including the
|
||||||
|
anti-cancer metabolite epothilone. We report the complete genome
|
||||||
|
sequence of the model Sorangium strain S. cellulosum So ce56,
|
||||||
|
which produces several natural products and has morphological and
|
||||||
|
physiological properties typical of the genus. The circular
|
||||||
|
genome, comprising 13,033,779 base pairs, is the largest
|
||||||
|
bacterial genome sequenced to date. No global synteny with the
|
||||||
|
genome of Myxococcus xanthus is apparent, revealing an
|
||||||
|
unanticipated level of divergence between these myxobacteria. A
|
||||||
|
large percentage of the genome is devoted to regulation,
|
||||||
|
particularly post-translational phosphorylation, which probably
|
||||||
|
supports the strain's complex, social lifestyle. This regulatory
|
||||||
|
network includes the highest number of eukaryotic protein
|
||||||
|
kinase-like kinases discovered in any organism. Seventeen
|
||||||
|
secondary metabolite loci are encoded in the genome, as well as
|
||||||
|
many enzymes with potential utility in industry.",
|
||||||
|
month = nov,
|
||||||
|
year = 2007,
|
||||||
|
url = "http://dx.doi.org/10.1038/nbt1354",
|
||||||
|
doi = "10.1038/nbt1354",
|
||||||
|
pmid = 17965706,
|
||||||
|
issn = "1087-0156,1546-1696",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@incollection{Karlin1985,
|
||||||
|
author = {Karlin, Samuel and Ost, Fr{\'e}d{\'e}ric},
|
||||||
|
title = {Maximal segmental match length among random sequences from a finite alphabet},
|
||||||
|
booktitle = {Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer},
|
||||||
|
editor = {Le Cam, Lucien M. and Olshen, Richard A.},
|
||||||
|
volume = {1},
|
||||||
|
pages = {225--243},
|
||||||
|
year = {1985},
|
||||||
|
publisher = {Wadsworth},
|
||||||
|
address = {Monterey, CA}
|
||||||
|
}
|
||||||
|
@article{Karlin1988,
|
||||||
|
author = {Karlin, Samuel and Ost, Fr{\'e}d{\'e}ric},
|
||||||
|
title = {Maximal length of common words among random letter sequences},
|
||||||
|
journal = {The Annals of Probability},
|
||||||
|
volume = {16},
|
||||||
|
number = {2},
|
||||||
|
pages = {535--563},
|
||||||
|
year = {1988}
|
||||||
|
}
|
||||||
|
@ARTICLE{Le-Cam1960-pe,
|
||||||
|
title = "{An approximation theorem for the Poisson binomial distribution}",
|
||||||
|
author = "Le Cam, Lucien",
|
||||||
|
journal = "Pacific Journal of Mathematics",
|
||||||
|
publisher = "Mathematical Sciences Publishers",
|
||||||
|
volume = 10,
|
||||||
|
number = 4,
|
||||||
|
pages = "1181--1197",
|
||||||
|
month = dec,
|
||||||
|
year = 1960,
|
||||||
|
url = "http://dx.doi.org/10.2140/pjm.1960.10.1181",
|
||||||
|
doi = "10.2140/pjm.1960.10.1181",
|
||||||
|
issn = "0030-8730,1945-5844"
|
||||||
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Hoeffding1956-bx,
|
||||||
|
title = "{On the distribution of the number of successes in independent
|
||||||
|
trials}",
|
||||||
|
author = "Hoeffding, Wassily",
|
||||||
|
journal = "The Annals of Mathematical Statistics",
|
||||||
|
publisher = "Institute of Mathematical Statistics",
|
||||||
|
volume = 27,
|
||||||
|
number = 3,
|
||||||
|
pages = "713--721",
|
||||||
|
abstract = "Let $S$ be the number of successes in $n$ independent trials, and
|
||||||
|
let $p_j$ denote the probability of success in the $j$th trial,
|
||||||
|
$j = 1, 2, \cdots, n$ (Poisson trials). We consider the problem
|
||||||
|
of finding the maximum and the minimum of $Eg(S),$ the expected
|
||||||
|
value of a given real-valued function of $S,$ when $ES = np$ is
|
||||||
|
fixed. It is well known that the maximum of the variance of $S$
|
||||||
|
is attained when $p_1 = p_2 = \cdots = p_n = p.$ This can be
|
||||||
|
interpreted as showing that the variability in the number of
|
||||||
|
successes is highest when the successes are equally probable
|
||||||
|
(Bernoulli trials). This interpretation is further supported by
|
||||||
|
the following two theorems, proved in this paper. If $b$ and $c$
|
||||||
|
are two integers, $0 \leqq b \leqq np \leqq c \leqq n,$ the
|
||||||
|
probability $P(b \leqq S \leqq c)$ attains its minimum if and
|
||||||
|
only if $p_1 = p_2 = \cdots = p_n = p,$ unless $b = 0$ and $c =
|
||||||
|
n$ (Theorem 5, a corollary of Theorem 4, which gives the maximum
|
||||||
|
and the minimum of $P(S \leqq c)$). If $g$ is a strictly convex
|
||||||
|
function, $Eg(S)$ attains its maximum if and only if $p_1 = p_2 =
|
||||||
|
\cdots = p_n = p$ (Theorem 3). These results are obtained with
|
||||||
|
the help of two theorems concerning the extrema of the expected
|
||||||
|
value of an arbitrary function $g(S)$ under the condition $ES =
|
||||||
|
np.$ Theorem 1 gives necessary conditions for the maximum and the
|
||||||
|
minimum of $Eg(S).$ Theorem 2 gives a partial characterization of
|
||||||
|
the set of points at which an extremum is attained. Corollary 2.1
|
||||||
|
states that the maximum and the minimum are attained when $p_1,
|
||||||
|
p_2, \cdots, p_n$ take on, at most, three different values, only
|
||||||
|
one of which is distinct from 0 and 1. Applications of Theorems 3
|
||||||
|
and 5 to problems of estimation and testing are pointed out in
|
||||||
|
Section 5.",
|
||||||
|
month = sep,
|
||||||
|
year = 1956,
|
||||||
|
url = "http://dx.doi.org/10.1214/aoms/1177728178",
|
||||||
|
doi = "10.1214/aoms/1177728178",
|
||||||
|
issn = "0003-4851,2168-8990",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Gleser1975-yu,
|
||||||
|
title = "{On the distribution of the number of successes in independent
|
||||||
|
trials}",
|
||||||
|
author = "Gleser, Leon Jay",
|
||||||
|
journal = "Annals of Probability",
|
||||||
|
publisher = "Institute of Mathematical Statistics",
|
||||||
|
volume = 3,
|
||||||
|
number = 1,
|
||||||
|
pages = "182--188",
|
||||||
|
abstract = "Let $S$ be the number of successes in $n$ independent Bernoulli
|
||||||
|
trials, where $p_j$ is the probability of success on the $j$th
|
||||||
|
trial. Let $\mathbf{p} = (p_1, p_2, \cdots, p_n)$, and for any
|
||||||
|
integer $c, 0 \leqq c \leqq n$, let $H(c \mid \mathbf{p}) = P\{S
|
||||||
|
\leqq c\}$. Let $\mathbf{p}^{(1)}$ be one possible choice of
|
||||||
|
$\mathbf{p}$ for which $E(S) = \lambda$. For any $n \times n$
|
||||||
|
doubly stochastic matrix $\Pi$, let $\mathbf{p}^{(2)} =
|
||||||
|
\mathbf{p}^{(1)}\Pi$. Then in the present paper it is shown that
|
||||||
|
$H(c \mid \mathbf{p}^{(1)}) \leqq H(c \mid \mathbf{p}^{(2)})$ for
|
||||||
|
$0 \leqq c \leqq \lbrack\lambda - 2\rbrack$, and $H(c \mid
|
||||||
|
\mathbf{p}^{(1)}) \geqq H(c \mid \mathbf{p}^{(2)})$ for
|
||||||
|
$\lbrack\lambda + 2\rbrack \leqq c \leqq n$. These results
|
||||||
|
provide a refinement of inequalities for $H(c \mid \mathbf{p})$
|
||||||
|
obtained by Hoeffding [3]. Their derivation is achieved by
|
||||||
|
applying consequences of the partial ordering of majorization.",
|
||||||
|
month = feb,
|
||||||
|
year = 1975,
|
||||||
|
url = "http://dx.doi.org/10.2307/2959277",
|
||||||
|
doi = "10.1214/aop/1176996461",
|
||||||
|
issn = "0091-1798,2168-894X"
|
||||||
|
}
|
||||||
Binary file not shown.
@@ -9,16 +9,16 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="prev" href="../kmers_and_superkmers/">
|
<link rel="prev" href="../../../installation/">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../entropy_filter/">
|
<link rel="next" href="../kmers/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="icon" href="../../assets/images/favicon.png">
|
<link rel="icon" href="../../../assets/images/favicon.png">
|
||||||
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
||||||
|
|
||||||
|
|
||||||
@@ -27,7 +27,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css">
|
<link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
<script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@
|
|||||||
|
|
||||||
<header class="md-header md-header--shadow" data-md-component="header">
|
<header class="md-header md-header--shadow" data-md-component="header">
|
||||||
<nav class="md-header__inner md-grid" aria-label="Header">
|
<nav class="md-header__inner md-grid" aria-label="Header">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -138,7 +138,7 @@
|
|||||||
|
|
||||||
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
||||||
<label class="md-nav__title" for="__drawer">
|
<label class="md-nav__title" for="__drawer">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -156,7 +156,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../.." class="md-nav__link">
|
<a href="../../.." class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -183,7 +183,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../installation/" class="md-nav__link">
|
<a href="../../../installation/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -256,17 +256,30 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -274,11 +287,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -395,14 +416,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../entropy_filter/" class="md-nav__link">
|
<a href="../kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -450,6 +471,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../indexing_architecture/" class="md-nav__link">
|
<a href="../indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
@@ -472,6 +521,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -533,7 +923,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/superkmer/" class="md-nav__link">
|
<a href="../../../usage/superkmer/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -561,7 +951,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/index_command/" class="md-nav__link">
|
<a href="../../../usage/index_command/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -589,7 +979,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/merge/" class="md-nav__link">
|
<a href="../../../usage/merge/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -617,7 +1007,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/filter/" class="md-nav__link">
|
<a href="../../../usage/filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -645,7 +1035,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/select/" class="md-nav__link">
|
<a href="../../../usage/select/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -673,7 +1063,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/query/" class="md-nav__link">
|
<a href="../../../usage/query/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -701,7 +1091,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/dump/" class="md-nav__link">
|
<a href="../../../usage/dump/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -729,7 +1119,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/annotate/" class="md-nav__link">
|
<a href="../../../usage/annotate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -757,7 +1147,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/phylo/" class="md-nav__link">
|
<a href="../../../usage/phylo/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -785,7 +1175,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/unitig/" class="md-nav__link">
|
<a href="../../../usage/unitig/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -813,7 +1203,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/estimate/" class="md-nav__link">
|
<a href="../../../usage/estimate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -841,7 +1231,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/convert/" class="md-nav__link">
|
<a href="../../../usage/convert/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -869,7 +1259,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/utils/" class="md-nav__link">
|
<a href="../../../usage/utils/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -897,7 +1287,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/pack/" class="md-nav__link">
|
<a href="../../../usage/pack/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -925,7 +1315,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/predicates/" class="md-nav__link">
|
<a href="../../../usage/predicates/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1007,7 +1397,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../formats/index_layout/" class="md-nav__link">
|
<a href="../../../formats/index_layout/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1028,6 +1418,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1042,7 +1460,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../architecture/" class="md-nav__link">
|
<a href="../../../architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1172,9 +1590,9 @@
|
|||||||
<p>A kmer of length <span class="arithmatex">\(k\)</span> (<span class="arithmatex">\(k \le 31\)</span>) fits in a single 64-bit word. The first nucleotide occupies the two most significant bits, each following nucleotide occupies the next two bits, and unused low-order bits are zero. Extracting nucleotide i (0-indexed from the 5′ end) is a shift-and-mask operation.</p>
|
<p>A kmer of length <span class="arithmatex">\(k\)</span> (<span class="arithmatex">\(k \le 31\)</span>) fits in a single 64-bit word. The first nucleotide occupies the two most significant bits, each following nucleotide occupies the next two bits, and unused low-order bits are zero. Extracting nucleotide i (0-indexed from the 5′ end) is a shift-and-mask operation.</p>
|
||||||
<p>Reverse complement is computed by bit manipulation directly on the packed word, without any lookup table: complement every base, reverse the byte order, then reverse the order of 2-bit groups within each byte in two more passes, and finally realign the result to the most-significant bits.</p>
|
<p>Reverse complement is computed by bit manipulation directly on the packed word, without any lookup table: complement every base, reverse the byte order, then reverse the order of 2-bit groups within each byte in two more passes, and finally realign the result to the most-significant bits.</p>
|
||||||
<h2 id="canonical-form">Canonical form</h2>
|
<h2 id="canonical-form">Canonical form</h2>
|
||||||
<p>The canonical form of a kmer is the lexicographic minimum of the kmer and its reverse complement:</p>
|
<p>The canonical form of a packed 2-bit sequence is the lexicographic minimum of the sequence and its reverse complement:</p>
|
||||||
<div class="arithmatex">\[\text{canonical}(kmer) = \min\big(kmer,\ \text{revcomp}(kmer)\big)\]</div>
|
<div class="arithmatex">\[\text{canonical}(x) = \min\big(x,\ \text{revcomp}(x)\big)\]</div>
|
||||||
<p>Using the canonical form halves the kmer space and makes counting strand-independent: a kmer and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.</p>
|
<p>This is a single operation on the packed representation, so it applies identically regardless of what <code>x</code> represents — a kmer (see <a href="../kmers/">Kmers</a>), an m-mer (see <a href="../minimizer_selection/">Minimizer selection</a>), or a super-kmer (see <a href="../superkmers/">Super-kmers</a>). Using the canonical form halves the relevant space and makes counting/selection strand-independent: a sequence and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.</p>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1224,10 +1642,10 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
<script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
||||||
|
|
||||||
|
|
||||||
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
<script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
||||||
|
|
||||||
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||||||
|
|
||||||
+454
-36
@@ -9,16 +9,16 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="prev" href="../encoding/">
|
<link rel="prev" href="../indexing_architecture/">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../minimizer_selection/">
|
<link rel="next" href="../../phylogeny/kmer_based/kmer_set_distances/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="icon" href="../../assets/images/favicon.png">
|
<link rel="icon" href="../../../assets/images/favicon.png">
|
||||||
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
||||||
|
|
||||||
|
|
||||||
@@ -27,7 +27,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css">
|
<link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
<script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@
|
|||||||
|
|
||||||
<header class="md-header md-header--shadow" data-md-component="header">
|
<header class="md-header md-header--shadow" data-md-component="header">
|
||||||
<nav class="md-header__inner md-grid" aria-label="Header">
|
<nav class="md-header__inner md-grid" aria-label="Header">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -138,7 +138,7 @@
|
|||||||
|
|
||||||
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
||||||
<label class="md-nav__title" for="__drawer">
|
<label class="md-nav__title" for="__drawer">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -156,7 +156,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../.." class="md-nav__link">
|
<a href="../../.." class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -183,7 +183,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../installation/" class="md-nav__link">
|
<a href="../../../installation/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -256,17 +256,66 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../kmers_and_superkmers/" class="md-nav__link">
|
<a href="../encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
DNA encoding
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -287,14 +336,98 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../encoding/" class="md-nav__link">
|
<a href="../kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
DNA encoding
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Minimizer selection
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Partitioning and indexing architecture
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -438,6 +571,108 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -445,14 +680,97 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../minimizer_selection/" class="md-nav__link">
|
<a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Minimizer selection
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -473,14 +791,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../indexing_architecture/" class="md-nav__link">
|
<a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Partitioning and indexing architecture
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -494,6 +812,78 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -555,7 +945,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/superkmer/" class="md-nav__link">
|
<a href="../../../usage/superkmer/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -583,7 +973,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/index_command/" class="md-nav__link">
|
<a href="../../../usage/index_command/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -611,7 +1001,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/merge/" class="md-nav__link">
|
<a href="../../../usage/merge/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -639,7 +1029,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/filter/" class="md-nav__link">
|
<a href="../../../usage/filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -667,7 +1057,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/select/" class="md-nav__link">
|
<a href="../../../usage/select/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -695,7 +1085,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/query/" class="md-nav__link">
|
<a href="../../../usage/query/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -723,7 +1113,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/dump/" class="md-nav__link">
|
<a href="../../../usage/dump/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -751,7 +1141,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/annotate/" class="md-nav__link">
|
<a href="../../../usage/annotate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -779,7 +1169,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/phylo/" class="md-nav__link">
|
<a href="../../../usage/phylo/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -807,7 +1197,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/unitig/" class="md-nav__link">
|
<a href="../../../usage/unitig/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -835,7 +1225,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/estimate/" class="md-nav__link">
|
<a href="../../../usage/estimate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -863,7 +1253,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/convert/" class="md-nav__link">
|
<a href="../../../usage/convert/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -891,7 +1281,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/utils/" class="md-nav__link">
|
<a href="../../../usage/utils/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -919,7 +1309,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/pack/" class="md-nav__link">
|
<a href="../../../usage/pack/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -947,7 +1337,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/predicates/" class="md-nav__link">
|
<a href="../../../usage/predicates/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1029,7 +1419,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../formats/index_layout/" class="md-nav__link">
|
<a href="../../../formats/index_layout/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1050,6 +1440,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1064,7 +1482,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../architecture/" class="md-nav__link">
|
<a href="../../../architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1252,10 +1670,10 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
<script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
||||||
|
|
||||||
|
|
||||||
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
<script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
||||||
|
|
||||||
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||||||
|
|
||||||
+458
-84
@@ -9,16 +9,16 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="prev" href="../minimizer_selection/">
|
<link rel="prev" href="../superkmers/">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../../usage/superkmer/">
|
<link rel="next" href="../entropy_filter/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="icon" href="../../assets/images/favicon.png">
|
<link rel="icon" href="../../../assets/images/favicon.png">
|
||||||
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
||||||
|
|
||||||
|
|
||||||
@@ -27,7 +27,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css">
|
<link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
<script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@
|
|||||||
|
|
||||||
<header class="md-header md-header--shadow" data-md-component="header">
|
<header class="md-header md-header--shadow" data-md-component="header">
|
||||||
<nav class="md-header__inner md-grid" aria-label="Header">
|
<nav class="md-header__inner md-grid" aria-label="Header">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -138,7 +138,7 @@
|
|||||||
|
|
||||||
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
||||||
<label class="md-nav__title" for="__drawer">
|
<label class="md-nav__title" for="__drawer">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -156,7 +156,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../.." class="md-nav__link">
|
<a href="../../.." class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -183,7 +183,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../installation/" class="md-nav__link">
|
<a href="../../../installation/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -256,17 +256,30 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -274,11 +287,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -315,14 +336,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../entropy_filter/" class="md-nav__link">
|
<a href="../kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -368,6 +389,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -439,17 +488,6 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#why-hashing-is-necessary" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Why hashing is necessary
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -472,6 +510,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -533,7 +912,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/superkmer/" class="md-nav__link">
|
<a href="../../../usage/superkmer/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -561,7 +940,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/index_command/" class="md-nav__link">
|
<a href="../../../usage/index_command/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -589,7 +968,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/merge/" class="md-nav__link">
|
<a href="../../../usage/merge/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -617,7 +996,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/filter/" class="md-nav__link">
|
<a href="../../../usage/filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -645,7 +1024,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/select/" class="md-nav__link">
|
<a href="../../../usage/select/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -673,7 +1052,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/query/" class="md-nav__link">
|
<a href="../../../usage/query/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -701,7 +1080,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/dump/" class="md-nav__link">
|
<a href="../../../usage/dump/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -729,7 +1108,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/annotate/" class="md-nav__link">
|
<a href="../../../usage/annotate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -757,7 +1136,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/phylo/" class="md-nav__link">
|
<a href="../../../usage/phylo/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -785,7 +1164,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/unitig/" class="md-nav__link">
|
<a href="../../../usage/unitig/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -813,7 +1192,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/estimate/" class="md-nav__link">
|
<a href="../../../usage/estimate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -841,7 +1220,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/convert/" class="md-nav__link">
|
<a href="../../../usage/convert/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -869,7 +1248,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/utils/" class="md-nav__link">
|
<a href="../../../usage/utils/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -897,7 +1276,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/pack/" class="md-nav__link">
|
<a href="../../../usage/pack/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -925,7 +1304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/predicates/" class="md-nav__link">
|
<a href="../../../usage/predicates/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1007,7 +1386,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../formats/index_layout/" class="md-nav__link">
|
<a href="../../../formats/index_layout/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1028,6 +1407,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1042,7 +1449,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../architecture/" class="md-nav__link">
|
<a href="../../../architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1097,17 +1504,6 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#why-hashing-is-necessary" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Why hashing is necessary
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -1141,15 +1537,13 @@
|
|||||||
<h1 id="partitioning-and-indexing-architecture">Partitioning and indexing architecture</h1>
|
<h1 id="partitioning-and-indexing-architecture">Partitioning and indexing architecture</h1>
|
||||||
<p>An index is split into a fixed number of <strong>partitions</strong>, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.</p>
|
<p>An index is split into a fixed number of <strong>partitions</strong>, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.</p>
|
||||||
<h2 id="routing">Routing</h2>
|
<h2 id="routing">Routing</h2>
|
||||||
<p>The canonical minimizer of a super-kmer (see <a href="../minimizer_selection/">Minimizer selection</a>) is hashed to produce a <span class="arithmatex">\(p\)</span>-bit routing value that selects the destination partition:</p>
|
<p>The canonical minimizer of a super-kmer (see <a href="../minimizer_selection/">Minimizer selection</a>) is hashed to produce a <span class="arithmatex">\(p\)</span>-bit routing value that selects the destination partition — the low <span class="arithmatex">\(p\)</span> bits of the minimizer's hash:</p>
|
||||||
<div class="highlight"><pre><span></span><code>canonical minimizer → hash(minimizer) → p-bit value → partition index
|
<div class="highlight"><pre><span></span><code>canonical minimizer → hash(minimizer) → p-bit value → partition index
|
||||||
</code></pre></div>
|
</code></pre></div>
|
||||||
<p>The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.</p>
|
<div class="arithmatex">\[\text{partition} = H(\text{minimizer}) \bmod 2^p\]</div>
|
||||||
<p>Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see <a href="../../formats/index_layout/">On-disk storage</a>); the minimizer plays no further role once a super-kmer has reached its partition.</p>
|
<p>Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see <a href="../../../formats/index_layout/">On-disk storage</a>); the minimizer plays no further role once a super-kmer has reached its partition.</p>
|
||||||
<h2 id="why-hashing-is-necessary">Why hashing is necessary</h2>
|
|
||||||
<p>A canonical minimizer is an m-mer (<span class="arithmatex">\(m \in \{9, 11, 13, 15\}\)</span>), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented (Golan & Shur 2025; Kille <em>et al.</em> 2023; Pan & Reinert 2024; Zheng <em>et al.</em> 2020, 2021)<sup id="fnref:Zheng2020-ji"><a class="footnote-ref" href="#fn:Zheng2020-ji">1</a></sup> <sup id="fnref:Zheng2021-cc"><a class="footnote-ref" href="#fn:Zheng2021-cc">2</a></sup> <sup id="fnref:Pan2024-hb"><a class="footnote-ref" href="#fn:Pan2024-hb">3</a></sup> <sup id="fnref:Kille2023-px"><a class="footnote-ref" href="#fn:Kille2023-px">4</a></sup> <sup id="fnref:Golan2025-xf"><a class="footnote-ref" href="#fn:Golan2025-xf">5</a></sup>. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.</p>
|
|
||||||
<p>Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits <span class="arithmatex">\(p\)</span> is chosen well below the number of bits available in the minimizer (<span class="arithmatex">\(2m\)</span>): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.</p>
|
|
||||||
<h2 id="parameter-guidance">Parameter guidance</h2>
|
<h2 id="parameter-guidance">Parameter guidance</h2>
|
||||||
|
<p>Even though <span class="arithmatex">\(H\)</span> already makes minimizer values well-distributed (see <a href="../minimizer_selection/">Minimizer selection</a>), choosing <span class="arithmatex">\(p\)</span> well below the number of bits available in the minimizer (<span class="arithmatex">\(2m\)</span>) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.</p>
|
||||||
<table>
|
<table>
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
@@ -1187,26 +1581,6 @@
|
|||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
<p>The number of partitions must satisfy <span class="arithmatex">\(p \le 2m\)</span>, and in practice <span class="arithmatex">\(p\)</span> is chosen well below that bound to leave a comfortable entropy margin. For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span>, <span class="arithmatex">\(p=10\)</span> (1024 partitions), partition load is well balanced on real genomic data.</p>
|
<p>The number of partitions must satisfy <span class="arithmatex">\(p \le 2m\)</span>, and in practice <span class="arithmatex">\(p\)</span> is chosen well below that bound to leave a comfortable entropy margin. For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span>, <span class="arithmatex">\(p=10\)</span> (1024 partitions), partition load is well balanced on real genomic data.</p>
|
||||||
<div class="footnote">
|
|
||||||
<hr />
|
|
||||||
<ol>
|
|
||||||
<li id="fn:Zheng2020-ji">
|
|
||||||
<p>Zheng, H., Kingsford, C. & Marçais, G. (2020). <a href="https://doi.org/10.1093/bioinformatics/btaa472">Improved design and analysis of practical minimizers</a>. <em>Bioinformatics (Oxford, England)</em>, 36, i119--i127. <a class="footnote-backref" href="#fnref:Zheng2020-ji" title="Jump back to footnote 1 in the text">↩</a></p>
|
|
||||||
</li>
|
|
||||||
<li id="fn:Zheng2021-cc">
|
|
||||||
<p>Zheng, H., Kingsford, C. & Marçais, G. (2021). <a href="https://doi.org/10.1093/bioinformatics/btab313">Sequence-specific minimizers via polar sets</a>. <em>Bioinformatics (Oxford, England)</em>, 37, i187--i195. <a class="footnote-backref" href="#fnref:Zheng2021-cc" title="Jump back to footnote 2 in the text">↩</a></p>
|
|
||||||
</li>
|
|
||||||
<li id="fn:Pan2024-hb">
|
|
||||||
<p>Pan, C. & Reinert, K. (2024). <a href="https://doi.org/10.1093/bioinformatics/btae045">A simple refined DNA minimizer operator enables 2-fold faster computation</a>. <em>Bioinformatics (Oxford, England)</em>, 40. <a class="footnote-backref" href="#fnref:Pan2024-hb" title="Jump back to footnote 3 in the text">↩</a></p>
|
|
||||||
</li>
|
|
||||||
<li id="fn:Kille2023-px">
|
|
||||||
<p>Kille, B., Garrison, E., Treangen, T.J. & Phillippy, A.M. (2023). <a href="https://doi.org/10.1093/bioinformatics/btad512">Minmers are a generalization of minimizers that enable unbiased local jaccard estimation</a>. <em>Bioinformatics (Oxford, England)</em>, 39. <a class="footnote-backref" href="#fnref:Kille2023-px" title="Jump back to footnote 4 in the text">↩</a></p>
|
|
||||||
</li>
|
|
||||||
<li id="fn:Golan2025-xf">
|
|
||||||
<p>Golan, S. & Shur, A.M. (2025). <a href="https://doi.org/10.1007/978-3-031-82670-2\_25">Expected density of random minimizers</a>. In: <em>Lecture notes in computer science</em>, Lecture notes in computer science. Springer Nature Switzerland, Cham, pp. 347--360. <a class="footnote-backref" href="#fnref:Golan2025-xf" title="Jump back to footnote 5 in the text">↩</a></p>
|
|
||||||
</li>
|
|
||||||
</ol>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1256,10 +1630,10 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
<script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
||||||
|
|
||||||
|
|
||||||
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
<script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
||||||
|
|
||||||
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||||||
|
|
||||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
+515
-160
@@ -9,25 +9,25 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="prev" href="../../installation/">
|
<link rel="prev" href="../minimizer_selection/">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../encoding/">
|
<link rel="next" href="../indexing_architecture/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="icon" href="../../assets/images/favicon.png">
|
<link rel="icon" href="../../../assets/images/favicon.png">
|
||||||
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<title>Kmers and super-kmers - obikmer — User Guide</title>
|
<title>Super-kmers - obikmer — User Guide</title>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css">
|
<link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
<script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -64,7 +64,7 @@
|
|||||||
<div data-md-component="skip">
|
<div data-md-component="skip">
|
||||||
|
|
||||||
|
|
||||||
<a href="#kmers-and-super-kmers" class="md-skip">
|
<a href="#super-kmers" class="md-skip">
|
||||||
Skip to content
|
Skip to content
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@
|
|||||||
|
|
||||||
<header class="md-header md-header--shadow" data-md-component="header">
|
<header class="md-header md-header--shadow" data-md-component="header">
|
||||||
<nav class="md-header__inner md-grid" aria-label="Header">
|
<nav class="md-header__inner md-grid" aria-label="Header">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -100,7 +100,7 @@
|
|||||||
<div class="md-header__topic" data-md-component="header-topic">
|
<div class="md-header__topic" data-md-component="header-topic">
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
Kmers and super-kmers
|
Super-kmers
|
||||||
|
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
@@ -138,7 +138,7 @@
|
|||||||
|
|
||||||
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
||||||
<label class="md-nav__title" for="__drawer">
|
<label class="md-nav__title" for="__drawer">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -156,7 +156,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../.." class="md-nav__link">
|
<a href="../../.." class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -183,7 +183,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../installation/" class="md-nav__link">
|
<a href="../../../installation/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -260,22 +260,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item md-nav__item--active">
|
|
||||||
|
|
||||||
<input class="md-nav__toggle md-toggle" type="checkbox" id="__toc">
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
|
||||||
<label class="md-nav__link md-nav__link--active" for="__toc">
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -283,88 +287,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-nav__icon md-icon"></span>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</label>
|
</label>
|
||||||
|
|
||||||
<a href="./" class="md-nav__link md-nav__link--active">
|
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
</span>
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<label class="md-nav__title" for="__toc">
|
|
||||||
<span class="md-nav__icon md-icon"></span>
|
|
||||||
Table of contents
|
|
||||||
</label>
|
|
||||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#kmers" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Kmers
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#super-kmers" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Super-kmers
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
<nav class="md-nav" aria-label="Super-kmers">
|
|
||||||
<ul class="md-nav__list">
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#canonical-super-kmers" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Canonical super-kmers
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
</ul>
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
</ul>
|
|
||||||
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -401,14 +336,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../entropy_filter/" class="md-nav__link">
|
<a href="../kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -454,6 +389,92 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--active">
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle" type="checkbox" id="__toc">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link md-nav__link--active" for="__toc">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<a href="./" class="md-nav__link md-nav__link--active">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__title" for="__toc">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
Table of contents
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#canonical-super-kmers" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Canonical super-kmers
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -478,6 +499,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -539,7 +901,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/superkmer/" class="md-nav__link">
|
<a href="../../../usage/superkmer/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -567,7 +929,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/index_command/" class="md-nav__link">
|
<a href="../../../usage/index_command/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -595,7 +957,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/merge/" class="md-nav__link">
|
<a href="../../../usage/merge/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -623,7 +985,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/filter/" class="md-nav__link">
|
<a href="../../../usage/filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -651,7 +1013,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/select/" class="md-nav__link">
|
<a href="../../../usage/select/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -679,7 +1041,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/query/" class="md-nav__link">
|
<a href="../../../usage/query/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -707,7 +1069,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/dump/" class="md-nav__link">
|
<a href="../../../usage/dump/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -735,7 +1097,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/annotate/" class="md-nav__link">
|
<a href="../../../usage/annotate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -763,7 +1125,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/phylo/" class="md-nav__link">
|
<a href="../../../usage/phylo/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -791,7 +1153,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/unitig/" class="md-nav__link">
|
<a href="../../../usage/unitig/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -819,7 +1181,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/estimate/" class="md-nav__link">
|
<a href="../../../usage/estimate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -847,7 +1209,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/convert/" class="md-nav__link">
|
<a href="../../../usage/convert/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -875,7 +1237,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/utils/" class="md-nav__link">
|
<a href="../../../usage/utils/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -903,7 +1265,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/pack/" class="md-nav__link">
|
<a href="../../../usage/pack/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -931,7 +1293,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/predicates/" class="md-nav__link">
|
<a href="../../../usage/predicates/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1013,7 +1375,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../formats/index_layout/" class="md-nav__link">
|
<a href="../../../formats/index_layout/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1034,6 +1396,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1048,7 +1438,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../architecture/" class="md-nav__link">
|
<a href="../../../architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1095,29 +1485,6 @@
|
|||||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="#kmers" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Kmers
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#super-kmers" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Super-kmers
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
<nav class="md-nav" aria-label="Super-kmers">
|
|
||||||
<ul class="md-nav__list">
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#canonical-super-kmers" class="md-nav__link">
|
<a href="#canonical-super-kmers" class="md-nav__link">
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
@@ -1126,11 +1493,6 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
</ul>
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
@@ -1150,29 +1512,22 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<h1 id="kmers-and-super-kmers">Kmers and super-kmers</h1>
|
<h1 id="super-kmers">Super-kmers</h1>
|
||||||
<h2 id="kmers">Kmers</h2>
|
|
||||||
<p>A <strong>kmer</strong> is a DNA subsequence of fixed length <span class="arithmatex">\(k\)</span>. Two constraints apply to <span class="arithmatex">\(k\)</span>, both enforced when a command starts (an invalid value exits immediately with an error):</p>
|
|
||||||
<ul>
|
|
||||||
<li><span class="arithmatex">\(k \in [11, 31]\)</span>: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base (<span class="arithmatex">\(k \le 32\)</span> is the hard limit; <span class="arithmatex">\(k < 11\)</span> gives insufficient specificity).</li>
|
|
||||||
<li><span class="arithmatex">\(k\)</span> <strong>is odd</strong>: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see <a href="../encoding/">DNA encoding</a>) to be well defined.</li>
|
|
||||||
</ul>
|
|
||||||
<h2 id="super-kmers">Super-kmers</h2>
|
|
||||||
<p>A <strong>super-kmer</strong> is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see <a href="../minimizer_selection/">Minimizer selection</a>). Each kmer in the run overlaps the next by <span class="arithmatex">\(k-1\)</span> nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.</p>
|
<p>A <strong>super-kmer</strong> is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see <a href="../minimizer_selection/">Minimizer selection</a>). Each kmer in the run overlaps the next by <span class="arithmatex">\(k-1\)</span> nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.</p>
|
||||||
<p>For a random minimizer of length <span class="arithmatex">\(m\)</span> over kmers of length <span class="arithmatex">\(k\)</span>, the expected length of a super-kmer is approximately (Golan & Shur 2025; Zheng <em>et al.</em> 2020)<sup id="fnref:Zheng2020-ji"><a class="footnote-ref" href="#fn:Zheng2020-ji">1</a></sup> <sup id="fnref:Golan2025-xf"><a class="footnote-ref" href="#fn:Golan2025-xf">2</a></sup>:</p>
|
<p>For a random minimizer of length <span class="arithmatex">\(m\)</span> over kmers of length <span class="arithmatex">\(k\)</span>, the expected length of a super-kmer is approximately (Zheng <em>et al.</em> 2020; Golan & Shur 2025)<sup id="fnref:Zheng2020-ji"><a class="footnote-ref" href="#fn:Zheng2020-ji">1</a></sup> <sup id="fnref:Golan2025-xf"><a class="footnote-ref" href="#fn:Golan2025-xf">2</a></sup>:</p>
|
||||||
<div class="arithmatex">\[L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1\]</div>
|
<div class="arithmatex">\[L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1\]</div>
|
||||||
<p>For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span> this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.</p>
|
<p>For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span> this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.</p>
|
||||||
<h3 id="canonical-super-kmers">Canonical super-kmers</h3>
|
<h2 id="canonical-super-kmers">Canonical super-kmers</h2>
|
||||||
<p>A <strong>canonical super-kmer</strong> is the lexicographic minimum of a super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.</p>
|
<p>A <strong>canonical super-kmer</strong> is the canonical form of a super-kmer (see <a href="../encoding/">DNA encoding</a>): the lexicographic minimum of the super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.</p>
|
||||||
<p>Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.</p>
|
<p>Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.</p>
|
||||||
<div class="footnote">
|
<div class="footnote">
|
||||||
<hr />
|
<hr />
|
||||||
<ol>
|
<ol>
|
||||||
<li id="fn:Zheng2020-ji">
|
<li id="fn:Zheng2020-ji">
|
||||||
<p>Zheng, H., Kingsford, C. & Marçais, G. (2020). <a href="https://doi.org/10.1093/bioinformatics/btaa472">Improved design and analysis of practical minimizers</a>. <em>Bioinformatics (Oxford, England)</em>, 36, i119--i127. <a class="footnote-backref" href="#fnref:Zheng2020-ji" title="Jump back to footnote 1 in the text">↩</a></p>
|
<p>Zheng H, Kingsford C, Marçais G (2020) <a href="https://doi.org/10.1093/bioinformatics/btaa472">Improved design and analysis of practical minimizers</a>. <em>Bioinformatics (Oxford, England)</em>, <strong>36</strong>, i119--i127. <a class="footnote-backref" href="#fnref:Zheng2020-ji" title="Jump back to footnote 1 in the text">↩</a></p>
|
||||||
</li>
|
</li>
|
||||||
<li id="fn:Golan2025-xf">
|
<li id="fn:Golan2025-xf">
|
||||||
<p>Golan, S. & Shur, A.M. (2025). <a href="https://doi.org/10.1007/978-3-031-82670-2\_25">Expected density of random minimizers</a>. In: <em>Lecture notes in computer science</em>, Lecture notes in computer science. Springer Nature Switzerland, Cham, pp. 347--360. <a class="footnote-backref" href="#fnref:Golan2025-xf" title="Jump back to footnote 2 in the text">↩</a></p>
|
<p>Golan S, Shur AM (2025) <a href="https://doi.org/10.1007/978-3-031-82670-2\_25">Expected density of random minimizers</a>. In: <em>Lecture notes in computer science</em> Lecture notes in computer science., pp. 347--360. Springer Nature Switzerland, Cham. <a class="footnote-backref" href="#fnref:Golan2025-xf" title="Jump back to footnote 2 in the text">↩</a></p>
|
||||||
</li>
|
</li>
|
||||||
</ol>
|
</ol>
|
||||||
</div>
|
</div>
|
||||||
@@ -1225,10 +1580,10 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
<script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
||||||
|
|
||||||
|
|
||||||
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
<script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
||||||
|
|
||||||
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||||||
|
|
||||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
+479
-150
@@ -9,25 +9,25 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="prev" href="../entropy_filter/">
|
<link rel="prev" href="../gamma_rate_heterogeneity/">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../indexing_architecture/">
|
<link rel="next" href="../sampling_design/">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="icon" href="../../assets/images/favicon.png">
|
<link rel="icon" href="../../../../assets/images/favicon.png">
|
||||||
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<title>Minimizer selection - obikmer — User Guide</title>
|
<title>Sankoff model - obikmer — User Guide</title>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css">
|
<link rel="stylesheet" href="../../../../assets/stylesheets/main.484c7ddc.min.css">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
<script>__md_scope=new URL("../../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -64,7 +64,7 @@
|
|||||||
<div data-md-component="skip">
|
<div data-md-component="skip">
|
||||||
|
|
||||||
|
|
||||||
<a href="#minimizer-selection" class="md-skip">
|
<a href="#the-16-state-sankoff-model" class="md-skip">
|
||||||
Skip to content
|
Skip to content
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@
|
|||||||
|
|
||||||
<header class="md-header md-header--shadow" data-md-component="header">
|
<header class="md-header md-header--shadow" data-md-component="header">
|
||||||
<nav class="md-header__inner md-grid" aria-label="Header">
|
<nav class="md-header__inner md-grid" aria-label="Header">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -100,7 +100,7 @@
|
|||||||
<div class="md-header__topic" data-md-component="header-topic">
|
<div class="md-header__topic" data-md-component="header-topic">
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
Minimizer selection
|
Sankoff model
|
||||||
|
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
@@ -138,7 +138,7 @@
|
|||||||
|
|
||||||
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
|
||||||
<label class="md-nav__title" for="__drawer">
|
<label class="md-nav__title" for="__drawer">
|
||||||
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
<a href="../../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
|
||||||
|
|
||||||
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
|
||||||
@@ -156,7 +156,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../.." class="md-nav__link">
|
<a href="../../../.." class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -183,7 +183,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../installation/" class="md-nav__link">
|
<a href="../../../../installation/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -258,15 +258,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -274,11 +285,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -287,7 +306,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../encoding/" class="md-nav__link">
|
<a href="../../../kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -315,7 +334,119 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../entropy_filter/" class="md-nav__link">
|
<a href="../../../kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Minimizer selection
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Partitioning and indexing architecture
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -336,6 +467,251 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" checked>
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="true">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" checked>
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="true">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -359,7 +735,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Minimizer selection
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -377,7 +753,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Minimizer selection
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -403,49 +779,10 @@
|
|||||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="#definition" class="md-nav__link">
|
<a href="#combining-into-a-1616-cost-matrix" class="md-nav__link">
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
Definition
|
Combining into a 16×16 cost matrix
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#hash-based-random-minimizer" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Hash-based ("random") minimizer
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
<nav class="md-nav" aria-label="Hash-based ("random") minimizer">
|
|
||||||
<ul class="md-nav__list">
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#hash-function" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Hash function
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
</ul>
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#partition-routing-is-independent-of-minimizer-selection" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Partition routing is independent of minimizer selection
|
|
||||||
|
|
||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
@@ -468,14 +805,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../indexing_architecture/" class="md-nav__link">
|
<a href="../sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Partitioning and indexing architecture
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -495,6 +832,22 @@
|
|||||||
</li>
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -550,7 +903,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/superkmer/" class="md-nav__link">
|
<a href="../../../../usage/superkmer/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -578,7 +931,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/index_command/" class="md-nav__link">
|
<a href="../../../../usage/index_command/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -606,7 +959,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/merge/" class="md-nav__link">
|
<a href="../../../../usage/merge/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -634,7 +987,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/filter/" class="md-nav__link">
|
<a href="../../../../usage/filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -662,7 +1015,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/select/" class="md-nav__link">
|
<a href="../../../../usage/select/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -690,7 +1043,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/query/" class="md-nav__link">
|
<a href="../../../../usage/query/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -718,7 +1071,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/dump/" class="md-nav__link">
|
<a href="../../../../usage/dump/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -746,7 +1099,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/annotate/" class="md-nav__link">
|
<a href="../../../../usage/annotate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -774,7 +1127,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/phylo/" class="md-nav__link">
|
<a href="../../../../usage/phylo/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -802,7 +1155,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/unitig/" class="md-nav__link">
|
<a href="../../../../usage/unitig/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -830,7 +1183,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/estimate/" class="md-nav__link">
|
<a href="../../../../usage/estimate/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -858,7 +1211,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/convert/" class="md-nav__link">
|
<a href="../../../../usage/convert/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -886,7 +1239,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/utils/" class="md-nav__link">
|
<a href="../../../../usage/utils/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -914,7 +1267,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/pack/" class="md-nav__link">
|
<a href="../../../../usage/pack/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -942,7 +1295,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../usage/predicates/" class="md-nav__link">
|
<a href="../../../../usage/predicates/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1024,7 +1377,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../formats/index_layout/" class="md-nav__link">
|
<a href="../../../../formats/index_layout/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1045,6 +1398,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1059,7 +1440,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../architecture/" class="md-nav__link">
|
<a href="../../../../architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1106,49 +1487,10 @@
|
|||||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="#definition" class="md-nav__link">
|
<a href="#combining-into-a-1616-cost-matrix" class="md-nav__link">
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
Definition
|
Combining into a 16×16 cost matrix
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#hash-based-random-minimizer" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Hash-based ("random") minimizer
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
<nav class="md-nav" aria-label="Hash-based ("random") minimizer">
|
|
||||||
<ul class="md-nav__list">
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#hash-function" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Hash function
|
|
||||||
|
|
||||||
</span>
|
|
||||||
</a>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
</ul>
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
</li>
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="#partition-routing-is-independent-of-minimizer-selection" class="md-nav__link">
|
|
||||||
<span class="md-ellipsis">
|
|
||||||
|
|
||||||
Partition routing is independent of minimizer selection
|
|
||||||
|
|
||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
@@ -1172,33 +1514,20 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<h1 id="minimizer-selection">Minimizer selection</h1>
|
<h1 id="the-16-state-sankoff-model">The 16-state Sankoff model</h1>
|
||||||
<h2 id="definition">Definition</h2>
|
<p>Each family (see <a href="../central_snp_model/">Central-position SNP model</a>) is treated as a character with 16 possible states: one per subset of the 4 possible central bases, including the empty subset (no observed form). A genome pair's calibration reduces to two separately-estimated first-order Markov models, both restricted to genome pairs below a configurable SNP-ratio ceiling (excluding saturated pairs from skewing the calibration):</p>
|
||||||
<p>A <strong>minimizer</strong> of a kmer window is the m-mer (<span class="arithmatex">\(m < k\)</span>) that is smallest, among all <span class="arithmatex">\(k - m + 1\)</span> overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent.</p>
|
|
||||||
<p>The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see <a href="../kmers_and_superkmers/">Kmers and super-kmers</a>).</p>
|
|
||||||
<h2 id="hash-based-random-minimizer">Hash-based ("random") minimizer</h2>
|
|
||||||
<p><code>obikmer</code> selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions.</p>
|
|
||||||
<p>Instead, a well-distributed hash function <span class="arithmatex">\(H\)</span> is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest <span class="arithmatex">\(H\)</span> value wins. Because <span class="arithmatex">\(H\)</span> is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition.</p>
|
|
||||||
<p>The canonical form used as input to <span class="arithmatex">\(H\)</span> is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the <em>distribution of hash values themselves</em> toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.</p>
|
|
||||||
<h3 id="hash-function">Hash function</h3>
|
|
||||||
<p>The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed:</p>
|
|
||||||
<div class="arithmatex">\[H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}\]</div>
|
|
||||||
<div class="highlight"><pre><span></span><code>H(x):
|
|
||||||
x ← x ⊕ 0x9e3779b97f4a7c15
|
|
||||||
x ← x ⊕ (x >> 30)
|
|
||||||
x ← x × 0xbf58476d1ce4e5b9
|
|
||||||
x ← x ⊕ (x >> 27)
|
|
||||||
x ← x × 0x94d049bb133111eb
|
|
||||||
return x ⊕ (x >> 31)
|
|
||||||
</code></pre></div>
|
|
||||||
<p>The XOR seed avoids the finalizer's fixed point at 0 (<span class="arithmatex">\(\text{mix64}(0) = 0\)</span>), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison.</p>
|
|
||||||
<h2 id="partition-routing-is-independent-of-minimizer-selection">Partition routing is independent of minimizer selection</h2>
|
|
||||||
<p>The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately:</p>
|
|
||||||
<ul>
|
<ul>
|
||||||
<li><strong>Selection</strong> uses <span class="arithmatex">\(H\)</span> applied to every candidate m-mer in the window, keeping the minimum.</li>
|
<li><strong>Cardinality transitions</strong> (<span class="arithmatex">\(5 \times 5\)</span>, row-stochastic): the probability that a family observed with <span class="arithmatex">\(i\)</span> forms in one genome (<span class="arithmatex">\(i \in \{0,\dots,4\}\)</span>) is observed with <span class="arithmatex">\(j\)</span> forms in the other.</li>
|
||||||
<li><strong>Partition routing</strong> recomputes <span class="arithmatex">\(H\)</span> on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.</li>
|
<li><strong>Composition transitions</strong> (<span class="arithmatex">\(4 \times 4\)</span>, row-stochastic): the probability that an unambiguous, single-copy base observed as one of A/C/G/T in one genome is observed as another base in the other genome.</li>
|
||||||
</ul>
|
</ul>
|
||||||
<p>See <a href="../indexing_architecture/">Partitioning and indexing architecture</a> for how the routing value is turned into a partition index.</p>
|
<h2 id="combining-into-a-1616-cost-matrix">Combining into a 16×16 cost matrix</h2>
|
||||||
|
<p>The two matrices are <em>not</em> combined as a plain tensor product. For a pair of states <span class="arithmatex">\((A, B)\)</span> — each a subset of <span class="arithmatex">\(\{A,C,G,T\}\)</span> — let <code>shared</code> <span class="arithmatex">\(= A \cap B\)</span>, <code>lost</code> <span class="arithmatex">\(= A \setminus B\)</span>, <code>gained</code> <span class="arithmatex">\(= B \setminus A\)</span>. The log-probability of the transition <span class="arithmatex">\(A \to B\)</span> is:</p>
|
||||||
|
<div class="arithmatex">\[\log P(A,B) = \underbrace{\log P_{\text{card}}(|A|,|B|)}_{\text{cardinality term, omitted when loss events are treated as free}} \;+\; \underbrace{\sum_{i \,\in\, \text{shared}} \log P_{\text{comp}}(i,i)}_{\text{bases conserved on both sides}} \;-\; \underbrace{\text{best\_pairing\_cost}(\text{lost}, \text{gained})}_{\text{substitutions among the differing bases}}\]</div>
|
||||||
|
<p><code>best_pairing_cost</code> enumerates every injection pairing elements of <code>lost</code> with elements of <code>gained</code> and keeps the one minimizing <span class="arithmatex">\(-\sum \log P_{\text{comp}}\)</span> over the pairs — the discrete analogue of "prefer a substitution over an independent loss and gain": if a base is lost from one side and a different base is gained on the other, that is scored as a single substitution between them (cheaper, under any sane calibration, than treating them as two unrelated cardinality-changing events) whenever such a pairing is possible.</p>
|
||||||
|
<p>The resulting <span class="arithmatex">\(16 \times 16\)</span> log-probability matrix is row-normalized in log-space (log-sum-exp, not a direct <code>exp()</code>/divide, for numerical stability), giving a genuine row-stochastic transition matrix <span class="arithmatex">\(P\)</span>. The cost matrix is then <span class="arithmatex">\(\text{cost}(A,B) = -\ln P(A,B)\)</span>, symmetrized by simple averaging:</p>
|
||||||
|
<div class="arithmatex">\[\text{sym}(A,B) = \frac{\text{cost}(A,B) + \text{cost}(B,A)}{2}\]</div>
|
||||||
|
<p>This calibrated cost matrix is what every downstream export (for TNT, PhyG, IQ-TREE) reuses, recoded into each tool's own format.</p>
|
||||||
|
<p>See <a href="../../../../usage/phylo/#sankoff-calibration-and-phylogenetic-exports"><code>phylo</code></a> for how to run the calibration and produce these exports.</p>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1248,10 +1577,10 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
<script id="__config" type="application/json">{"annotate": null, "base": "../../../..", "features": [], "search": "../../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
|
||||||
|
|
||||||
|
|
||||||
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
<script src="../../../../assets/javascripts/bundle.79ae519e.min.js"></script>
|
||||||
|
|
||||||
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||||||
|
|
||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1017,6 +1405,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1017,6 +1405,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1017,6 +1405,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1006,6 +1394,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1028,6 +1416,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -549,6 +937,17 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#which-files-are-read" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Which files are read
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -1028,6 +1427,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1097,6 +1524,17 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#which-files-are-read" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Which files are read
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -1157,6 +1595,10 @@
|
|||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
|
<h2 id="which-files-are-read">Which files are read</h2>
|
||||||
|
<p>A file named explicitly on the command line is always read, whatever its extension: the format (FASTA, FASTQ or GenBank) and the compression (gzip, bzip2, xz or zstd) are detected from the content. If the content is not a sequence file, the command reports an error rather than skipping the file.</p>
|
||||||
|
<p>When an input is a directory, the files inside it are selected by extension: <code>.fasta</code>, <code>.fa</code>, <code>.fna</code> (the extension of NCBI genome downloads), <code>.ffn</code>, <code>.fas</code>, <code>.fsa</code>, <code>.fastq</code>, <code>.fq</code>, <code>.gbff</code>, <code>.gbk</code> and <code>.gb</code>, each optionally followed by <code>.gz</code>.</p>
|
||||||
|
<p>A small input spread over many partitions leaves most partitions empty. This is valid: the index is built normally and empty partitions simply hold no k-mers.</p>
|
||||||
<h2 id="options">Options</h2>
|
<h2 id="options">Options</h2>
|
||||||
<table>
|
<table>
|
||||||
<thead>
|
<thead>
|
||||||
|
|||||||
+482
-14
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -599,6 +987,28 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#resource-usage" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Resource usage
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#merging-an-index-that-was-already-merged" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Merging an index that was already merged
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
@@ -1028,6 +1438,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1119,6 +1557,28 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#resource-usage" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Resource usage
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#merging-an-index-that-was-already-merged" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Merging an index that was already merged
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
@@ -1188,15 +1648,23 @@
|
|||||||
<td>Disambiguate duplicate genome labels (<code>.1</code>, <code>.2</code>, …) instead of failing</td>
|
<td>Disambiguate duplicate genome labels (<code>.1</code>, <code>.2</code>, …) instead of failing</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>--budget-fraction</code></td>
|
<td><code>--dense</code></td>
|
||||||
<td><code>0.5</code></td>
|
<td>off</td>
|
||||||
<td>Fraction of available RAM reserved as the memory budget for parallel partition merging</td>
|
<td>Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see <a href="../pack/">pack</a>)</td>
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
<h2 id="behaviour">Behaviour</h2>
|
<h2 id="behaviour">Behaviour</h2>
|
||||||
<p>The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. <code>--force-presence</code> forces presence/absence regardless of the sources.</p>
|
<p>The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. <code>--force-presence</code> forces presence/absence regardless of the sources.</p>
|
||||||
<p>By default, merging two indexes that share a genome label fails with an error; <code>--rename-duplicates</code> instead appends a numeric suffix to keep both copies.</p>
|
<p>By default, merging two indexes that share a genome label fails with an error; <code>--rename-duplicates</code> instead appends a numeric suffix to keep both copies.</p>
|
||||||
|
<p>The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; <code>--dense</code> keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The <a href="../pack/">pack</a> page explains the trade-off between the two formats.</p>
|
||||||
|
<h2 id="resource-usage">Resource usage</h2>
|
||||||
|
<p>The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.</p>
|
||||||
|
<p>Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.</p>
|
||||||
|
<h2 id="merging-an-index-that-was-already-merged">Merging an index that was already merged</h2>
|
||||||
|
<p>An index produced by <code>merge</code> can be used as one of the sources of a later merge, which is how a collection is extended without starting again from the original genomes. There is one rule: among the sources, at least one must not have been through a merge. A merged index is stored in a compact packed format (see <a href="../pack/">pack</a>) that is read very efficiently but cannot be extended in place. The merge therefore takes a freshly built index as its starting point and brings the packed index in as a source. A typical case is a merged index of many genomes plus one or a few newly built single-genome indexes, and this works.</p>
|
||||||
|
<p>If every source has already been merged or packed, the command stops before doing any work and says so. In that case, merge the original single-genome indexes together instead.</p>
|
||||||
|
<p>The genome columns of the result follow the order of the starting index first, then the other sources, so they may not come out in the order the sources were listed. The genome labels always identify each column unambiguously.</p>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1028,6 +1416,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+948
-278
File diff suppressed because it is too large.
Load diff
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1045,6 +1433,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1028,6 +1416,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+471
-21
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -655,6 +1043,17 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#disk-usage" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Disk usage
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
@@ -1028,6 +1427,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1119,6 +1546,17 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#disk-usage" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Disk usage
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
@@ -1139,8 +1577,8 @@
|
|||||||
|
|
||||||
|
|
||||||
<h1 id="select">select</h1>
|
<h1 id="select">select</h1>
|
||||||
<p>Project and/or aggregate the genome columns of an index into a new (or in-place) index. Where <a href="../filter/"><code>filter</code></a> selects rows (kmers), <code>select</code> operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.</p>
|
<p>Project and/or aggregate the genome columns of an index into a new index. Where <a href="../filter/"><code>filter</code></a> selects rows (kmers), <code>select</code> operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.</p>
|
||||||
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>SOURCE<span class="w"> </span><span class="o">(</span>--output<span class="w"> </span>OUTPUT<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="o">)</span><span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
|
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>SOURCE<span class="w"> </span>--output<span class="w"> </span>OUTPUT<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
|
||||||
</code></pre></div>
|
</code></pre></div>
|
||||||
<h2 id="arguments">Arguments</h2>
|
<h2 id="arguments">Arguments</h2>
|
||||||
<table>
|
<table>
|
||||||
@@ -1168,14 +1606,9 @@
|
|||||||
</thead>
|
</thead>
|
||||||
<tbody>
|
<tbody>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>--output</code></td>
|
<td><code>-o, --output</code></td>
|
||||||
<td>—</td>
|
<td>—</td>
|
||||||
<td>Output index directory (mutually exclusive with <code>--in-place</code>)</td>
|
<td>Output index directory (required)</td>
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><code>--in-place</code></td>
|
|
||||||
<td>off</td>
|
|
||||||
<td>Rewrite the source index in place (mutually exclusive with <code>--output</code>)</td>
|
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>-f, --force</code></td>
|
<td><code>-f, --force</code></td>
|
||||||
@@ -1212,12 +1645,29 @@
|
|||||||
<td><code>0</code></td>
|
<td><code>0</code></td>
|
||||||
<td>Minimum count for a genome to be considered a carrier (logical operators only)</td>
|
<td>Minimum count for a genome to be considered a carrier (logical operators only)</td>
|
||||||
</tr>
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>--dense</code></td>
|
||||||
|
<td>off</td>
|
||||||
|
<td>Pack the output's presence matrices in the dense format instead of the default sparse one</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>--force-copy</code></td>
|
||||||
|
<td>off</td>
|
||||||
|
<td>Copy each layer's unchanged kmer-identity files (mphf/unitigs/evidence/fingerprint) instead of hard-linking them</td>
|
||||||
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
<h2 id="aggregation-operators">Aggregation operators</h2>
|
<h2 id="aggregation-operators">Aggregation operators</h2>
|
||||||
<p><code>any</code>, <code>all</code>, <code>none</code> (logical, evaluated against <code>--presence-threshold</code>), <code>sum</code>, <code>min</code>, <code>max</code> (numeric, count index only). If a group's operator is left unspecified, it defaults to <code>any</code> when the source is a presence/absence index and <code>sum</code> when it stores counts.</p>
|
<p><code>any</code>, <code>all</code>, <code>none</code> (logical, evaluated against <code>--presence-threshold</code>), <code>sum</code>, <code>min</code>, <code>max</code> (numeric, count index only). If a group's operator is left unspecified, it defaults to <code>any</code> when the source is a presence/absence index and <code>sum</code> when it stores counts.</p>
|
||||||
<p>A <code>select</code> never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in <code>--select</code>) is a cheap copy.</p>
|
<p>A <code>select</code> never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in <code>--select</code>) is a cheap copy.</p>
|
||||||
<p>At least one of <code>--output</code>/<code>--in-place</code> is required, and at least one output column must be defined; every name listed in <code>--select</code> must resolve to either a defined group or an existing genome label. See <a href="../predicates/">Genome predicates and taxonomy paths</a> for the predicate syntax used by <code>--group</code>.</p>
|
<p>At least one output column must be defined; every name listed in <code>--select</code> must resolve to either a defined group or an existing genome label. See <a href="../predicates/">Genome predicates and taxonomy paths</a> for the predicate syntax used by <code>--group</code>.</p>
|
||||||
|
<h2 id="disk-usage">Disk usage</h2>
|
||||||
|
<p><code>select</code> always writes to a new output directory — there is no in-place mode. Each layer's kmer-identity files (MPHF, unitigs, evidence, fingerprint) never change under a column projection/aggregation, so they are hard-linked into the output rather than copied: no extra disk is used for them, even on a very large index. Linking falls back to a real copy automatically if it fails (e.g. <code>SOURCE</code>/<code>OUTPUT</code> on different filesystems). Use <code>--force-copy</code> to always copy instead — needed when the output must be able to survive independently of the source on disk (a hard link shares the same underlying data, so overwriting one path outside <code>select</code> itself would affect the other).</p>
|
||||||
|
<p>To replace an index with a selected version of itself, select to a temporary directory and swap it in:</p>
|
||||||
|
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>INDEX<span class="w"> </span>--output<span class="w"> </span>INDEX.tmp<span class="w"> </span>--group<span class="w"> </span>...<span class="w"> </span>--group-op<span class="w"> </span>...<span class="w"> </span>--select<span class="w"> </span>...
|
||||||
|
rm<span class="w"> </span>-rf<span class="w"> </span>INDEX
|
||||||
|
mv<span class="w"> </span>INDEX.tmp<span class="w"> </span>INDEX
|
||||||
|
</code></pre></div>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+428
-12
@@ -9,7 +9,7 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<link rel="prev" href="../../theory/indexing_architecture/">
|
<link rel="prev" href="../../theory/phylogeny/snp_based/sampling_design/">
|
||||||
|
|
||||||
|
|
||||||
<link rel="next" href="../index_command/">
|
<link rel="next" href="../index_command/">
|
||||||
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1017,6 +1405,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1017,6 +1405,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
+427
-11
@@ -256,15 +256,26 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
|
||||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Kmers and super-kmers
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -272,11 +283,19 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
</a>
|
<span class="md-nav__icon md-icon"></span>
|
||||||
</li>
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer indexing
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -285,7 +304,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/encoding/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -313,14 +332,14 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Low-complexity kmer filter
|
Kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -341,7 +360,7 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -369,7 +388,35 @@
|
|||||||
|
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Super-kmers
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -390,6 +437,347 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Low-complexity kmer filter
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Phylogeny
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_1">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
Kmer-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Whole k-mer set distances
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item md-nav__item--nested">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||||
|
|
||||||
|
|
||||||
|
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
</label>
|
||||||
|
|
||||||
|
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||||
|
<label class="md-nav__title" for="__nav_3_2_2">
|
||||||
|
<span class="md-nav__icon md-icon"></span>
|
||||||
|
|
||||||
|
|
||||||
|
SNP-based
|
||||||
|
|
||||||
|
|
||||||
|
</label>
|
||||||
|
<ul class="md-nav__list" data-md-scrollfix>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Central-position SNP model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Gamma rate heterogeneity
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sankoff model
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
Sampling design
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
@@ -1017,6 +1405,34 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="../../formats/vcf/" class="md-nav__link">
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
|
VCF pseudo-alignment
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
</ul>
|
</ul>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
||||||
|
|||||||
@@ -64,6 +64,7 @@ nav:
|
|||||||
- Predicates and taxonomy paths: usage/predicates.md
|
- Predicates and taxonomy paths: usage/predicates.md
|
||||||
- Formats:
|
- Formats:
|
||||||
- Index construction and on-disk layout: formats/index_layout.md
|
- Index construction and on-disk layout: formats/index_layout.md
|
||||||
|
- VCF pseudo-alignment: formats/vcf.md
|
||||||
- Architecture notes: architecture.md
|
- Architecture notes: architecture.md
|
||||||
|
|
||||||
watch:
|
watch:
|
||||||
|
|||||||
@@ -56,6 +56,8 @@ nav:
|
|||||||
- PersistentBitVec: implementation/persistent_bit_vec.md
|
- PersistentBitVec: implementation/persistent_bit_vec.md
|
||||||
- Merge command: implementation/merge.md
|
- Merge command: implementation/merge.md
|
||||||
- Merge parallelism & memory: implementation/merge_parallelism.md
|
- Merge parallelism & memory: implementation/merge_parallelism.md
|
||||||
|
- "Merge mmap audit (2026-11)": implementation/merge_mmap_audit.md
|
||||||
|
- "Merge scaling (open)": implementation/merge_scaling.md
|
||||||
- Kmer filtering: implementation/filtering.md
|
- Kmer filtering: implementation/filtering.md
|
||||||
- Select command: implementation/select.md
|
- Select command: implementation/select.md
|
||||||
- obitaxonomy crate: implementation/obitaxonomy.md
|
- obitaxonomy crate: implementation/obitaxonomy.md
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
# Prompt — refonte mmap de `obikmerge` (plans A puis C)
|
||||||
|
|
||||||
|
> À coller tel quel au début d'une nouvelle session Claude Code ouverte à la racine du dépôt obikmer.
|
||||||
|
> Conçu pour être suivi par **Sonnet** : toutes les décisions d'architecture sont fixées ci-dessous. Le plan B de l'audit (ouverture « slots only » dans `obikindex`) est **hors périmètre** de cette session (voir étape 5).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contexte
|
||||||
|
|
||||||
|
Lis d'abord `DevDocMD/implementation/merge_mmap_audit.md`. C'est le diagnostic complet, et ce prompt en implémente le plan.
|
||||||
|
|
||||||
|
En résumé : `obikmerge::partition_merge::merge_partition` ouvre, pour **chaque** partition, **toutes** les couches de **toutes** les sources (4 mmaps chacune). Elle crée aussi **toutes** les colonnes destination d'avance, en fichiers creux mappés en écriture (`set_len` + `MmapMut`). Avec 186 sources, ça fait environ 1 100 mappings par partition, ce qui épuise `vm.max_map_count` en production (pas d'accès root pour le relever). L'écriture via `MmapMut` dans des fichiers creux transforme aussi tout refus d'allocation disque en SIGBUS.
|
||||||
|
|
||||||
|
Objectif : le nombre de mappings vivants par partition ne doit plus dépendre du nombre de sources, et l'écriture des colonnes du merge ne doit plus passer par `MmapMut`. **Le résultat sur disque doit rester strictement identique.**
|
||||||
|
|
||||||
|
## Règles (non négociables)
|
||||||
|
|
||||||
|
- Appelle `mcp__serena__initial_instructions` avant tout travail de code. Utilise Serena (`find_symbol`, `get_symbols_overview`, `find_referencing_symbols`, `replace_symbol_body`, `replace_content`) pour naviguer et éditer le Rust.
|
||||||
|
- **Aucune commande VCS mutante** (git/jj : commit, add, mv, rm, describe, push…). Lecture seule autorisée.
|
||||||
|
- **Aucune nouvelle dépendance**, ni changement de version.
|
||||||
|
- `obikindex` fait autorité : cette session n'y modifie et n'y ajoute rien.
|
||||||
|
- Pas de fonctions libres ad hoc dans `obikmerge` pour combler un manque d'API. Si une opération générique manque sur un type de `obicompactvec`, ajoute-la comme méthode sur ce type.
|
||||||
|
- Fichiers source sous ~500 lignes. `partition_merge.rs` en fait déjà 605 : il **doit** être découpé (étape 6).
|
||||||
|
- **Ne jamais `.collect()` un flux de k-mers ou de unitigs** pour satisfaire une contrainte de trait.
|
||||||
|
- Échanges en **français**, documentation en **anglais**.
|
||||||
|
- **Point d'arrêt obligatoire** à la fin de chaque étape. Résume ce qui a été fait et les résultats de tests (sortie réelle), puis **attends mon feu vert** avant l'étape suivante.
|
||||||
|
- Si un test ou l'oracle échoue et que tu ne trouves pas la cause en une tentative, **arrête-toi et expose le problème**, sans contournement.
|
||||||
|
|
||||||
|
## Étape 1 — Oracle de non-régression (avant toute modification de code)
|
||||||
|
|
||||||
|
### 1a. Oracle synthétique (rapide, couvre comptage et merge en deux temps)
|
||||||
|
|
||||||
|
1. Lis `scripts/smoke_test_index.sh` (modèle de style, options de `obikmer index`) et les `--help` de `obikmer index`, `merge` et `dump`.
|
||||||
|
2. Écris `scripts/smoke_test_merge.sh`, sur le même modèle :
|
||||||
|
- génère 4 FASTA aléatoires déterministes A, B, C, D, petits, avec **des fragments partagés** entre eux. Le merge doit à la fois retrouver des k-mers déjà présents dans la destination (chemin « couche existante ») et en créer de nouveaux (chemin « nouvelle couche ») ;
|
||||||
|
- indexe chacun deux fois : présence (défaut) et comptage (`--with-counts`). Plusieurs partitions (`-p 4`) ;
|
||||||
|
- scénario 1 : `merge A B C D` (présence), puis la même chose en comptage ;
|
||||||
|
- scénario 2 : `merge A B` → AB, puis `merge AB C D`. Ce scénario couvre une source multi-génome et une destination à plusieurs couches ;
|
||||||
|
- scénario 3 : comme le scénario 1, avec `--dense` ;
|
||||||
|
- pour chaque sortie : `obikmer dump` → `sort` → un fichier texte ;
|
||||||
|
- deux modes : `--record DIR` écrit les sorties triées dans `DIR`, `--check DIR` les régénère et fait un `diff` avec `DIR`. Affiche PASS/FAIL clairement, avec un code de sortie cohérent.
|
||||||
|
3. Lance `--record benchmark/run/merge_oracle/synthetic` **avec le code actuel, non modifié**. Vérifie que les fichiers ne sont pas vides et que les colonnes de génomes ont du sens.
|
||||||
|
4. Lance aussitôt `--check` sur le même code pour vérifier que le script est déterministe.
|
||||||
|
|
||||||
|
### 1b. Oracle sur données réelles : `benchmark/`
|
||||||
|
|
||||||
|
`benchmark/run/specimen_index_presence/` contient 20 index présence construits à partir de vrais génomes (bactéries + une levure, lectures simulées). Lis `benchmark/README.md` et `benchmark/merge_presence.sh` (le merge de référence y utilise `--force --force-presence`).
|
||||||
|
|
||||||
|
État constaté au 2026-11-17, à revérifier :
|
||||||
|
- `specimen_index_count/` est **vide** : pas de piste comptage sans relancer `gmake index_count` (GNU make ≥ 4.3 : `gmake` sur macOS). **Demande-moi** avant de le lancer, car ça prend du temps ;
|
||||||
|
- `reference_index/` (`.npz`) est **absent** : `verify_merge_*.py` est inutilisable, ne t'appuie pas dessus ;
|
||||||
|
- `global_index_presence/` a été produit par un binaire antérieur au code actuel (release du 2026-10-22) : **ce n'est pas la référence**.
|
||||||
|
|
||||||
|
Procédure :
|
||||||
|
1. Recompile le binaire release à partir du **code actuel non modifié** (`cargo build --release` dans `src/`). Ne touche pas aux artefacts existants de `benchmark/run/`.
|
||||||
|
2. Lance le merge des 20 index avec les mêmes options que `merge_presence.sh`, mais vers un répertoire **dédié**, par exemple `benchmark/run/merge_oracle/bench/merged`, jamais `benchmark/run/global_index_presence`. Garde le log stderr : le rapport de stages (wall et RSS de `merge_partitions`) sert de référence de performance.
|
||||||
|
3. `obikmer dump` du résultat → `sort` → compresse (`zstd`) + `sha256sum`, dans `benchmark/run/merge_oracle/bench/`. La sortie peut être volumineuse : compare d'abord les hash, et ne fais un `diff` que s'ils diffèrent.
|
||||||
|
4. Contrôle croisé informatif : fais la même chose (dump trié + hash) sur `benchmark/run/global_index_presence`. Si les hash diffèrent, **signale-le sans conclure** : l'écart peut venir d'évolutions légitimes du code depuis octobre.
|
||||||
|
5. Écris ces étapes dans un script `scripts/bench_merge_oracle.sh` avec deux modes : `--record DIR` et `--check DIR`. En mode `--check`, il affiche aussi le temps de `merge_partitions` avant/après.
|
||||||
|
|
||||||
|
Après chaque étape de code (3, 4, 6), les **deux** oracles doivent passer : `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check`. Si `merge_partitions` est plus de 20 % plus lent qu'avant sur le benchmark, signale-le au point d'arrêt.
|
||||||
|
|
||||||
|
**Point d'arrêt.**
|
||||||
|
|
||||||
|
## Étape 2 — `MatrixBuilder::n_cols` (petit prérequis, `obicompactvec`)
|
||||||
|
|
||||||
|
Ajoute `pub fn n_cols(&self) -> usize` à `obicompactvec::MatrixBuilder` (`src/obicompactvec/src/matrix_builder.rs`), en déléguant au builder Bit ou Int. Si `PersistentCompactIntMatrixBuilder` n'expose pas `n_cols()`, ajoute-le sur ce type. Ajoute un test unitaire. `cargo test -p obicompactvec`.
|
||||||
|
|
||||||
|
Pas de point d'arrêt séparé : enchaîne sur l'étape 3.
|
||||||
|
|
||||||
|
## Étape 3 — Plan A : passe 2 par groupes de sources (`obikmerge`)
|
||||||
|
|
||||||
|
Fichier : `src/obikmerge/src/partition_merge.rs`, fonction `merge_partition`.
|
||||||
|
|
||||||
|
### 3a. Passe 1 sans ouvrir les couches sources
|
||||||
|
|
||||||
|
Construis la liste des chemins `unitigs.bin` sans `open()`. Pour chaque source et chaque `l in 0..partition.n_layers()`, prends `KmerLayer::new(&partition, l).unitigs_path()` (ce chemin est valide sur la variante `Empty`). Garde le filtre `exists()`. Ne touche pas au reste de la passe 1 (`build_graph`, `materialize_layer`, ouverture de `new_mphf`).
|
||||||
|
|
||||||
|
### 3b. Nouvelle couche : matrice initialisée seule
|
||||||
|
|
||||||
|
Si `any_new`, crée la matrice de la nouvelle couche avec `MatrixBuilder::new(presence, n_new, data_dir)`, ajoute les `n_dst_genomes` colonnes absentes, puis **ferme**. Les colonnes sources seront ajoutées par `resume`, groupe par groupe.
|
||||||
|
|
||||||
|
### 3c. Boucle sur les groupes
|
||||||
|
|
||||||
|
- `const SRC_GROUP: usize = 2;` (aligné sur `max_open`), avec un commentaire qui explique pourquoi.
|
||||||
|
- Itère sur `sources.chunks(SRC_GROUP)`, **dans l'ordre**, en tenant un `col_base` = nombre de colonnes sources déjà ajoutées avant ce groupe.
|
||||||
|
- Pour chaque groupe :
|
||||||
|
1. ouvre les couches de ses sources pour cette partition (`partition.layer(l)?.open()` puis `Arc::new`), comme aujourd'hui mais **seulement pour ce groupe** ;
|
||||||
|
2. `group_n` = somme des `src_n` du groupe ;
|
||||||
|
3. pour chaque couche dst existante, puis pour la nouvelle couche si `any_new` : `MatrixBuilder::resume`, vérifie `n_cols() == n_dst_genomes + col_base`, puis `add_col()` × `group_n` ;
|
||||||
|
4. construis les `pass2_items` du groupe. Le `col_offset` est **local au groupe** (0..group_n) ;
|
||||||
|
5. lance le pipeline de passe 2 existant, inchangé dans sa logique (Flat → Transform → Sink), sur ces items et ces builders. Les vecteurs `exist_sink`/`new_sink` ne contiennent que les colonnes du groupe ;
|
||||||
|
6. vérifie l'erreur de passe 2, ferme toutes les `ColBuilder` puis tous les `MatrixBuilder` du groupe ;
|
||||||
|
7. **drop** les couches du groupe avant de passer au suivant ;
|
||||||
|
8. `col_base += group_n`.
|
||||||
|
- L'initialisation `init_presence_matrix` (premier merge en présence) reste **avant** la boucle, inchangée.
|
||||||
|
- `dst_layers` (ouverts une seule fois) et `new_mphf` restent partagés par tous les groupes.
|
||||||
|
|
||||||
|
Invariant à documenter dans un commentaire : les colonnes sont ajoutées dans l'ordre des sources, donc la colonne absolue `n_dst_genomes + col_base + offset_local + g` est la même qu'avant la refonte.
|
||||||
|
|
||||||
|
### 3d. Vérification
|
||||||
|
|
||||||
|
`cargo test -p obikmerge`, `cargo clippy -p obikmerge`, puis `scripts/smoke_test_merge.sh --check benchmark/run/merge_oracle/synthetic` et `scripts/bench_merge_oracle.sh --check benchmark/run/merge_oracle/bench` (recompile le release avant). Les deux **doivent** passer sans aucun diff.
|
||||||
|
|
||||||
|
**Point d'arrêt.**
|
||||||
|
|
||||||
|
## Étape 4 — Plan C : colonnes construites en mémoire (`obicompactvec`)
|
||||||
|
|
||||||
|
But : plus aucun `MmapMut` ni fichier creux pour les colonnes **remplies** par le merge.
|
||||||
|
|
||||||
|
1. Lis `PersistentBitVecBuilder` (`bitvec.rs` : `new`, `set`, `close`), `PersistentCompactIntVecBuilder` (`builder.rs` : `new`, `set`, `close`), `format.rs` (`finalize_pciv`, `HEADER_SIZE`) et `mmap_file.rs` (`building_path`, `publish_built`). Note **exactement** les octets écrits à la fermeture : en-tête, données, overflow, `.building` puis `rename`.
|
||||||
|
2. Ajoute deux types :
|
||||||
|
- `MemBitVecBuilder` (dans `bitvec.rs`, ou un nouveau `mem_builder.rs` si `bitvec.rs` dépasse 500 lignes) : `words: Vec<u64>`, `n`, `path` ; `new(n, path)`, `set(slot, bool)`, `close() -> io::Result<()>` ;
|
||||||
|
- `MemCompactIntVecBuilder` : `primary: Vec<u8>`, `overflow: HashMap<usize, u32>`, `n`, `path` ; `new`, `set(slot, u32)` (même sémantique sentinelle 255 que le builder persistant), `close()`.
|
||||||
|
|
||||||
|
`close()` écrit avec un `BufWriter` dans `building_path(path)`, produit le **même fichier, octet pour octet**, que le builder persistant, fait `flush` + `sync_all`, puis `publish_built(path)`. Réutilise les fonctions de format existantes plutôt que de dupliquer la mise en page de l'en-tête. S'il faut factoriser l'écriture de l'en-tête ou de l'overflow pour la partager, fais-le dans `format.rs` / `mmap_file.rs`.
|
||||||
|
3. Dans `matrix_builder.rs` : ajoute les variantes `ColBuilder::MemBit(MemBitVecBuilder)` et `ColBuilder::MemInt(MemCompactIntVecBuilder)` (`set_val` et `close` les gèrent), plus `MatrixBuilder::add_mem_col() -> io::Result<ColBuilder>`. Ce dernier incrémente `n_cols` exactement comme `add_col`. Si les builders de matrice sous-jacents n'exposent ni le chemin de la prochaine colonne ni l'incrément, ajoute-leur une méthode `add_mem_col` symétrique de `add_col`. Le nommage des colonnes reste privé à la crate.
|
||||||
|
4. Tests unitaires (`src/obicompactvec/src/tests/`) : pour n ∈ {0, 1, 63, 64, 65, 10 000} et des valeurs pseudo-aléatoires (incluant des comptes ≥ 255), comparer **octet par octet** le fichier produit par le builder mémoire et celui du builder persistant. Vérifier aussi qu'une matrice construite avec `add_mem_col` se relit via `PersistentBitMatrix::open` / `PersistentIntMatrix::open`.
|
||||||
|
5. Dans `merge_partition`, remplace `add_col()` par `add_mem_col()` pour les colonnes sources (couches existantes et nouvelle couche). `add_absent_col` reste tel quel : on n'écrit jamais dans son mapping, donc aucun risque de SIGBUS.
|
||||||
|
6. Ajoute un commentaire sur le coût mémoire : `SRC_GROUP × src_n × (L_dst + 1) × (n/8 ou n)` octets au plus par partition.
|
||||||
|
7. `cargo test -p obicompactvec -p obikmerge`, clippy, puis `smoke_test_merge.sh --check` et `bench_merge_oracle.sh --check` (release recompilé).
|
||||||
|
|
||||||
|
`obikselect` et `obikfilter` ne sont **pas** migrés dans cette session.
|
||||||
|
|
||||||
|
**Point d'arrêt.**
|
||||||
|
|
||||||
|
## Étape 5 — Plan B : hors périmètre
|
||||||
|
|
||||||
|
**Ne pas implémenter.** Le plan B (ouverture « slots only » des couches dans `obikindex`) a été tranché par le développeur : ce sera un chantier séparé (forme retenue : composition, voir la section « Plan B » de `merge_mmap_audit.md`). Après l'étape 3, son gain en mmaps pour le merge est négligeable (~8 → ~2 par partition). Il touche en revanche le chemin de requête d'`obikindex` et concerne trois crates.
|
||||||
|
|
||||||
|
Les couches sources de la passe 2 continuent donc d'être ouvertes par `partition.layer(l)?.open()`, mais seulement groupe par groupe (étape 3). Ne crée aucun type ni méthode « lookup only » dans `obikindex` ou `obikmerge`.
|
||||||
|
|
||||||
|
## Étape 6 — Découpage, doc, finition
|
||||||
|
|
||||||
|
1. Découpe `partition_merge.rs` en module `partition_merge/` (par exemple `mod.rs` pour l'orchestration, `pass1.rs`, `pass2.rs`, `tests.rs`), chaque fichier sous 500 lignes, sans changement de comportement.
|
||||||
|
2. Supprime tout code devenu mort : `find_referencing_symbols` sur ce que la refonte a remplacé. Signale tout ce qui est devenu injustifié plutôt que de le garder par inertie.
|
||||||
|
3. `cargo test --workspace`, `cargo clippy --workspace`, `cargo check --workspace --features numa` si la feature existe, `scripts/smoke_test_index.sh`, `scripts/smoke_test_merge.sh --check` et `scripts/bench_merge_oracle.sh --check` (release recompilé ; reporte le temps de `merge_partitions` avant/après).
|
||||||
|
4. Documentation :
|
||||||
|
- `DevDocMD/implementation/merge_mmap_audit.md` : passe le statut à « implemented », en indiquant ce qui a été fait et les écarts éventuels au plan ;
|
||||||
|
- `DevDocMD/implementation/merge.md`, section « Algorithm » : décris le traitement par groupes de sources et les colonnes en mémoire ;
|
||||||
|
- vérifie si `UserDocMD/` décrit le comportement mémoire du merge, et le cas échéant mets-le à jour, en prose, sans code Rust ;
|
||||||
|
- mets à jour la `nav:` des `mkdocs*.yml` si un fichier a été ajouté, puis lance `qmd embed`.
|
||||||
|
5. Rapport final : liste des fichiers modifiés, résultats de tests, et estimation du nombre de mmaps par partition avant/après (calculée, marquée comme estimée). Termine par la commande de mesure à lancer en production, qui ne demande pas root :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PID=$(pgrep -f 'obikmer merge'); while sleep 60; do
|
||||||
|
echo "$(date +%T) maps=$(wc -l < /proc/$PID/maps) threads=$(ls /proc/$PID/task | wc -l)"; done
|
||||||
|
```
|
||||||
|
|
||||||
|
Ne commite rien : c'est moi qui le ferai.
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# Prompt — où optimiser l'étape `pack` du merge (diagnostic d'abord)
|
||||||
|
|
||||||
|
> À coller tel quel au début d'une nouvelle session Claude Code ouverte à la racine du dépôt obikmer.
|
||||||
|
> Cette session est un **diagnostic** : elle produit une analyse chiffrée et des pistes classées, pas du code. Toute modification du code attend mon feu vert explicite, piste par piste.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contexte
|
||||||
|
|
||||||
|
`obikmer merge` se termine par une étape `finalize`, dont l'essentiel est `pack` : les matrices de présence/comptage, écrites en colonnes pendant le merge (`MatrixBuilder`), sont transposées en un format compact par partition (sparse par défaut). Le point d'entrée est `KmerIndex::finalize_indexed` (`src/obikindex/src/index/builder.rs`, étape « pack » vers la ligne 109), qui appelle `KmerIndex::pack_matrices(sparse)` (`src/obikindex/src/index/kmer_index.rs`). Les primitives sont dans `src/obicompactvec/src/` : `pack_bit_matrix` (`bitmatrix/packed.rs`), `pack_compact_int_matrix` (`intmatrix.rs`), `pack_sparse_compact_int_matrix` (`sparse_intmatrix.rs`), et le côté sparse des bits dans `bitmatrix/`.
|
||||||
|
|
||||||
|
Lis d'abord `DevDocMD/implementation/merge_scaling.md` (état du pack sparse : passage par une matrice dense séquentielle puis sparse, un seul mapping par partition) et `DevDocMD/implementation/merge_mmap_audit.md`.
|
||||||
|
|
||||||
|
## Mesure de référence (production, 2026-11-18)
|
||||||
|
|
||||||
|
Merge de 186 sources (`Cedrus`), 256 partitions, binaire glibc compilé localement, **192 CPU non contraints**, 17,5 milliards de k-mers nouveaux au total. Mode **présence**, compaction par défaut, donc **sparse** (pas de `--dense`) :
|
||||||
|
|
||||||
|
| Étape | Durée | Efficacité CPU | RSS max |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `merge_partitions` | 748 s | 141×/192 (73 %) | 354 Go |
|
||||||
|
| `pack` | 231 s | 65×/192 (34 %) | **503,7 Go** |
|
||||||
|
|
||||||
|
La RAM de la machine est de 754 Go. Le RSS max est celui du process : il inclut les pages de fichiers mappés, pas seulement la mémoire anonyme. **Je n'ai pas encore séparé les deux.**
|
||||||
|
|
||||||
|
Progression de `pack` (partitions terminées par tranche de 10 %, début à 04:08:09) :
|
||||||
|
`+53 s` (10 %) · `+35 s` (20 %) · `+13 s` (30 %) · `+48 s` (40 %) · `+24 s` (50 %) · `+9 s` · `+6 s` · `+6 s` · `+6 s` · `+11 s` (100 %), puis `+20 s` avant la fin de l'étape. Les partitions les plus grosses passent en premier (tri décroissant dans `merge.rs`), mais je ne sais pas si `pack` les traite dans le même ordre ni si l'ordre est le bon.
|
||||||
|
|
||||||
|
## Ce que je veux savoir
|
||||||
|
|
||||||
|
1. **Où part le temps** de `pack` ? Lecture des colonnes, transposition, écriture, `sync`/`fsync`, attente d'un verrou, déséquilibre entre partitions ? Efficacité de 34 % sur 192 workers : qu'est-ce qui les bloque ?
|
||||||
|
2. **Où part la mémoire** (504 Go) ? Combien est de la mémoire anonyme, combien des pages de fichiers mappés ? Est-elle proportionnelle au nombre de workers ? Quelle est la mémoire vivante par partition en cours de `pack` ?
|
||||||
|
3. **Quel est le parallélisme réel** : combien de partitions `pack` traite-t-il en même temps, et qu'est-ce qui le fixe (`PartitionRunner`, `--cpu-max`, NUMA) ?
|
||||||
|
4. Y a-t-il du **travail inutile** (relectures, copies intermédiaires, colonnes vides ou toutes-absentes ré-écrites, étape dense qui pourrait être évitée) ?
|
||||||
|
|
||||||
|
## Méthode (impérative)
|
||||||
|
|
||||||
|
- **Lire et mesurer avant de proposer.** Pas de récit plausible sans preuve : tout ce qui n'est pas vérifié par lecture du code ou par mesure est marqué « hypothèse ».
|
||||||
|
- Pour mesurer sans la production : `scripts/docker_debug/` fournit un conteneur Linux (build glibc ou musl) qui lance un merge de N sources synthétiques ou dérivées de génomes réels (`run.sh`, variables `GEN`, `REAL`, `GENOME_LEN`, `TARGET`, `TOOL=heaptrack` pour glibc) avec un moniteur CSV de mappings et de RSS. Les génomes de 2 Mb donnent des partitions assez grosses pour que `pack` ait quelque chose à faire. `heaptrack` fonctionne sur le build glibc (pas sur le build musl statique). `gdb` ne fonctionne pas sous Rosetta.
|
||||||
|
- Pour séparer mémoire anonyme et fichiers mappés, utilise `/proc/PID/status` (`RssAnon`, `RssFile`, `RssShmem`), **jamais `smaps` en entier** sur un process à beaucoup de mappings (il verrouille `mmap_lock` et ralentit le process).
|
||||||
|
- Les estimations de coût (octets, secondes) sont marquées **estimées** ; les chiffres observés sont marqués **mesurés**.
|
||||||
|
|
||||||
|
## Règles du projet (non négociables)
|
||||||
|
|
||||||
|
- Avant tout travail de code : `mcp__serena__initial_instructions`, puis Serena (`find_symbol`, `get_symbols_overview`, `find_referencing_symbols`) pour naviguer. `rust-analyzer-mcp` et `qmd` quand ils servent. Consulte `context7` avant d'utiliser une API de bibliothèque externe.
|
||||||
|
- **Aucune commande VCS mutante** (git/jj : commit, add, mv, rm, describe, push…). Lecture seule autorisée.
|
||||||
|
- **Aucune nouvelle dépendance, aucun changement de version** sans mon accord explicite. Si un problème de bibliothèque se pose, expose-le et propose des alternatives.
|
||||||
|
- **Une question appelle une réponse, pas une action.** Ne modifie aucun fichier de `src/` sans demande explicite. Cette session est en lecture seule sur le code.
|
||||||
|
- Le code existant est une hypothèse, pas une vérité : signale tout ce qui est devenu injustifié plutôt que de le garder par inertie.
|
||||||
|
- Si tu proposes d'ajouter une opération à une structure partagée, mets-la comme méthode générique sur cette structure (crate `obicompactvec`), pas comme fonction libre dans le consommateur.
|
||||||
|
- Ne jamais matérialiser en mémoire une structure volumineuse qui peut être streamée.
|
||||||
|
- Échanges en **français**, documentation en **anglais**.
|
||||||
|
|
||||||
|
## Livrable
|
||||||
|
|
||||||
|
Un rapport (dans la conversation, pas dans un fichier, sauf si je le demande) avec :
|
||||||
|
1. Le chemin de code de `pack`, de `finalize_indexed` jusqu'aux écritures disque, résumé en quelques lignes par étape.
|
||||||
|
2. Pour chaque question ci-dessus : ce qui est **mesuré**, ce qui est **hypothèse**, et comment vérifier l'hypothèse.
|
||||||
|
3. Une liste de **pistes d'optimisation classées** par gain attendu/coût/risque. Pour chaque piste : ce qu'elle changerait, ce qu'elle risque, et comment on verrait qu'elle marche. Aucune implémentation.
|
||||||
|
4. Les vérifications de non-régression à prévoir : le résultat sur disque doit rester **identique octet pour octet** (voir les oracles `scripts/smoke_test_merge.sh` et `scripts/bench_merge_oracle.sh`).
|
||||||
|
|
||||||
|
Termine par la question : « Quelle piste veux-tu que j'implémente ? » et attends ma réponse.
|
||||||
Executable
+156
@@ -0,0 +1,156 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# bench_merge_oracle.sh — golden-output + timing oracle for `obikmer merge`
|
||||||
|
# on the real-data benchmark (benchmark/run/specimen_index_presence, 20
|
||||||
|
# presence indexes built from real genomes).
|
||||||
|
#
|
||||||
|
# Companion of smoke_test_merge.sh for refactors of obikmerge (see
|
||||||
|
# DevDocMD/implementation/merge_mmap_audit.md). Same merge options as
|
||||||
|
# benchmark/merge_presence.sh (--force --force-presence), but the output goes
|
||||||
|
# to a dedicated directory, never to benchmark/run/global_index_presence.
|
||||||
|
#
|
||||||
|
# What it does:
|
||||||
|
# 1. builds the release `obikmer`
|
||||||
|
# 2. merges every specimen presence index (sorted, as merge_presence.sh)
|
||||||
|
# and keeps the stderr log (stage report: wall/RSS of merge_partitions)
|
||||||
|
# 3. `obikmer dump --debug` of the result → header first, rows sorted
|
||||||
|
# (LC_ALL=C) → sha256 of that sorted text, plus a zstd copy
|
||||||
|
# 4. --record DIR: writes DIR/merged (the merged index), DIR/merge.stderr.log, DIR/dump.sha256,
|
||||||
|
# DIR/dump.sorted.csv.zst, and (informative cross-check)
|
||||||
|
# DIR/global_index_presence.sha256 = same dump hash computed on the
|
||||||
|
# existing benchmark/run/global_index_presence (older binary — a
|
||||||
|
# difference there is reported, not judged)
|
||||||
|
# --check DIR: merges into a scratch directory, compares the hash with
|
||||||
|
# DIR/dump.sha256 (full diff against the zstd copy only on mismatch),
|
||||||
|
# and prints merge_partitions wall/RSS and pack wall before → after
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/bench_merge_oracle.sh (--record DIR | --check DIR) [--keep]
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
KEEP=0
|
||||||
|
MODE=""
|
||||||
|
DIR=""
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--record) MODE=record; DIR="$2"; shift 2 ;;
|
||||||
|
--check) MODE=check; DIR="$2"; shift 2 ;;
|
||||||
|
--keep) KEEP=1; shift ;;
|
||||||
|
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [--keep]" >&2; exit 2; }
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
|
||||||
|
RUN_DIR="$REPO_ROOT/benchmark/run"
|
||||||
|
IDX_DIR="$RUN_DIR/specimen_index_presence"
|
||||||
|
GLOBAL_OLD="$RUN_DIR/global_index_presence"
|
||||||
|
BIN="$REPO_ROOT/src/target/release/obikmer"
|
||||||
|
|
||||||
|
mkdir -p "$DIR" "$RUN_DIR/merge_oracle"
|
||||||
|
# Scratch space on the same file system as the benchmark data (merge's
|
||||||
|
# bootstrap hard-links from its base source; sort needs room for temp files).
|
||||||
|
WORK="$(mktemp -d "$RUN_DIR/merge_oracle/.work.XXXXXX")"
|
||||||
|
cleanup() {
|
||||||
|
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
fail() {
|
||||||
|
echo "FAIL: $1" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── 1. build ────────────────────────────────────────────────────────────────
|
||||||
|
if ! (cd "$REPO_ROOT/src" && cargo build --release -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
|
||||||
|
cat "$WORK/build.log" >&2
|
||||||
|
fail "cargo build --release failed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
sorted_dump() { # sorted_dump INDEX OUT_TXT — header first, rows sorted
|
||||||
|
local raw="$WORK/dump.raw.csv"
|
||||||
|
"$BIN" dump --debug "$1" > "$raw" 2> "$WORK/dump.log" \
|
||||||
|
|| { cat "$WORK/dump.log" >&2; fail "obikmer dump $1 exited non-zero"; }
|
||||||
|
{ head -n 1 "$raw"
|
||||||
|
tail -n +2 "$raw" | LC_ALL=C sort -S 25% --parallel=8 -T "$WORK"
|
||||||
|
} > "$2"
|
||||||
|
rm -f "$raw"
|
||||||
|
[ "$(head -n 2 "$2" | wc -l)" -eq 2 ] || fail "empty dump of $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
hash_of() { sha256sum "$1" | cut -d' ' -f1; }
|
||||||
|
|
||||||
|
stage_field() { # stage_field LOG FIELD [STAGE] — FIELD of the 'done stage=STAGE' line (default merge_partitions)
|
||||||
|
perl -pe 's/\e\[[0-9;]*m//g' "$1" \
|
||||||
|
| grep "done stage=${3:-merge_partitions} " \
|
||||||
|
| sed -n "s/.*$2=\([^ ]*\( [KMG]*B\)\{0,1\}\).*/\1/p" | tail -n 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── 2. merge ────────────────────────────────────────────────────────────────
|
||||||
|
mapfile -t SOURCES < <(find "$IDX_DIR" -mindepth 1 -maxdepth 1 -type d | sort)
|
||||||
|
[ "${#SOURCES[@]}" -gt 0 ] || fail "no indexes found in $IDX_DIR"
|
||||||
|
|
||||||
|
if [ "$MODE" = record ]; then
|
||||||
|
MERGED="$DIR/merged"
|
||||||
|
LOG="$DIR/merge.stderr.log"
|
||||||
|
else
|
||||||
|
MERGED="$WORK/merged"
|
||||||
|
LOG="$WORK/merge.stderr.log"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "merging ${#SOURCES[@]} presence indexes → $MERGED"
|
||||||
|
if ! "$BIN" merge --output "$MERGED" --force --force-presence "${SOURCES[@]}" \
|
||||||
|
> /dev/null 2> "$LOG"; then
|
||||||
|
tail -n 40 "$LOG" >&2
|
||||||
|
fail "obikmer merge exited non-zero"
|
||||||
|
fi
|
||||||
|
WALL="$(stage_field "$LOG" wall_secs)"
|
||||||
|
RSS="$(stage_field "$LOG" rss)"
|
||||||
|
PACK_WALL="$(stage_field "$LOG" wall_secs pack)"
|
||||||
|
echo "merge_partitions: wall=${WALL}s rss=${RSS} pack: wall=${PACK_WALL}s"
|
||||||
|
|
||||||
|
# ── 3. sorted dump + hash ───────────────────────────────────────────────────
|
||||||
|
DUMP="$WORK/dump.sorted.csv"
|
||||||
|
sorted_dump "$MERGED" "$DUMP"
|
||||||
|
H="$(hash_of "$DUMP")"
|
||||||
|
echo "dump: $(($(wc -l < "$DUMP") - 1)) k-mers, sha256 $H"
|
||||||
|
|
||||||
|
# ── 4. record or check ──────────────────────────────────────────────────────
|
||||||
|
if [ "$MODE" = record ]; then
|
||||||
|
echo "$H" > "$DIR/dump.sha256"
|
||||||
|
zstd -q -T0 -f "$DUMP" -o "$DIR/dump.sorted.csv.zst"
|
||||||
|
|
||||||
|
if [ -d "$GLOBAL_OLD" ]; then
|
||||||
|
OLD_DUMP="$WORK/old.sorted.csv"
|
||||||
|
sorted_dump "$GLOBAL_OLD" "$OLD_DUMP"
|
||||||
|
HO="$(hash_of "$OLD_DUMP")"
|
||||||
|
echo "$HO" > "$DIR/global_index_presence.sha256"
|
||||||
|
if [ "$HO" = "$H" ]; then
|
||||||
|
echo "cross-check: global_index_presence (older binary) dump is identical"
|
||||||
|
else
|
||||||
|
echo "cross-check: global_index_presence (older binary) dump DIFFERS" \
|
||||||
|
"($(($(wc -l < "$OLD_DUMP") - 1)) k-mers, sha256 $HO) — informative only"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
echo "RECORDED → $DIR"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
[ -f "$DIR/dump.sha256" ] || fail "no reference in $DIR (run --record first)"
|
||||||
|
REF_H="$(cat "$DIR/dump.sha256")"
|
||||||
|
REF_WALL="$(stage_field "$DIR/merge.stderr.log" wall_secs)"
|
||||||
|
REF_RSS="$(stage_field "$DIR/merge.stderr.log" rss)"
|
||||||
|
REF_PACK="$(stage_field "$DIR/merge.stderr.log" wall_secs pack)"
|
||||||
|
echo "merge_partitions wall: ${REF_WALL}s → ${WALL}s rss: ${REF_RSS} → ${RSS}"
|
||||||
|
echo "pack wall: ${REF_PACK}s → ${PACK_WALL}s"
|
||||||
|
python3 -c "import sys; r,n=float(sys.argv[1]),float(sys.argv[2]); print(f'merge_partitions wall ratio: {n/r:.2f}' + (' (> 1.20: SLOWER THAN 20 %)' if n > 1.2*r else ''))" \
|
||||||
|
"$REF_WALL" "$WALL" || true
|
||||||
|
|
||||||
|
if [ "$H" = "$REF_H" ]; then
|
||||||
|
echo "PASS: dump identical to reference ($H)"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo "FAIL: dump hash differs (ref $REF_H, now $H)" >&2
|
||||||
|
zstd -q -d -c "$DIR/dump.sorted.csv.zst" | { diff - "$DUMP" || true; } | head -n 40 >&2
|
||||||
|
exit 1
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
# Linux debug environment for the merge mapping/memory-growth hunt.
|
||||||
|
# Same toolchain as .gitea/workflows/release.yml for the musl target
|
||||||
|
# (cargo-zigbuild + zig cc), plus a glibc toolchain and diagnostic tools.
|
||||||
|
FROM rust:1-bookworm
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
build-essential cmake pkg-config autoconf automake libtool \
|
||||||
|
heaptrack valgrind gdb strace procps python3 python3-pip \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
RUN pip install ziglang --quiet --break-system-packages \
|
||||||
|
&& cargo install cargo-zigbuild \
|
||||||
|
&& rustup target add x86_64-unknown-linux-musl
|
||||||
|
RUN ZIG=$(python3 -c "import ziglang, os; print(os.path.join(os.path.dirname(ziglang.__file__), 'zig'))") \
|
||||||
|
&& printf '#!/bin/sh\nexec "%s" cc -target x86_64-linux-musl "$@"\n' "$ZIG" > /usr/local/bin/x86_64-linux-musl-gcc \
|
||||||
|
&& printf '#!/bin/sh\nexec "%s" c++ -target x86_64-linux-musl "$@"\n' "$ZIG" > /usr/local/bin/x86_64-linux-musl-g++ \
|
||||||
|
&& chmod +x /usr/local/bin/x86_64-linux-musl-gcc /usr/local/bin/x86_64-linux-musl-g++
|
||||||
|
WORKDIR /work
|
||||||
Executable
+6
@@ -0,0 +1,6 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Builds the debug image. PLATFORM defaults to linux/amd64 (Rosetta on Apple Silicon).
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||||
|
PLATFORM="${PLATFORM:-linux/amd64}"
|
||||||
|
docker build --platform "$PLATFORM" -t "obikmer-debug:${PLATFORM//\//-}" .
|
||||||
Executable
+82
@@ -0,0 +1,82 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Index compatibility test between two obikmer binaries that differ only by
|
||||||
|
# their ptr_hash version (A = baseline, B = candidate).
|
||||||
|
#
|
||||||
|
# Usage: compat_ptrhash.sh VOL_A VOL_B OUT
|
||||||
|
# VOL_A / VOL_B docker volumes holding a musl release build (CARGO_TARGET_DIR)
|
||||||
|
# OUT host directory receiving report.txt
|
||||||
|
#
|
||||||
|
# What it checks (everything compared by md5 of dump/query output):
|
||||||
|
# T0 on-disk bytes of an index built by A vs by B (informational)
|
||||||
|
# T1 A-built single-genome indexes read by A and by B
|
||||||
|
# T2 merges: A merging A-built sources (reference), B merging A-built
|
||||||
|
# sources, B merging B-built sources; each result dumped by A and by B
|
||||||
|
# T3 queries (hits and misses) on indexes read by A and by B
|
||||||
|
# T4 incremental merge into an existing A-built merged index, by A and B
|
||||||
|
set -u
|
||||||
|
VOL_A="$1"; VOL_B="$2"; OUT="$3"
|
||||||
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO="$(cd "$HERE/../.." && pwd)"
|
||||||
|
mkdir -p "$OUT"
|
||||||
|
|
||||||
|
docker run --rm --platform linux/amd64 \
|
||||||
|
-v "$VOL_A:/va:ro" -v "$VOL_B:/vb:ro" \
|
||||||
|
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" -v "$OUT:/out" \
|
||||||
|
obikmer-debug:linux-amd64 bash -c '
|
||||||
|
set -u
|
||||||
|
R=x86_64-unknown-linux-musl/release/obikmer
|
||||||
|
mkdir -p /tmp/bin && cp /va/$R /tmp/bin/A && cp /vb/$R /tmp/bin/B
|
||||||
|
A=/tmp/bin/A; B=/tmp/bin/B
|
||||||
|
W=/tmp/w; rm -rf $W; mkdir -p $W/ia $W/ib
|
||||||
|
exec > >(tee /out/report.txt) 2>&1
|
||||||
|
echo "A sha: $(sha256sum $A | cut -c1-16) B sha: $(sha256sum $B | cut -c1-16)"
|
||||||
|
mapfile -t G < <(ls -S /genomes_ro/*.fna.gz | sed -n 8,12p)
|
||||||
|
echo "genomes: ${#G[@]}"; printf " %s\n" "${G[@]##*/}"
|
||||||
|
mkdir -p $W/fa; n=0; for gz in "${G[@]}"; do n=$((n+1)); g=$W/fa/L$n.fasta; zcat "$gz" > "$g"
|
||||||
|
$A index -k 21 -m 9 --theta 0 -p 256 --label L$n -o $W/ia/L$n.idx "$g" 2>$W/ia/L$n.err >/dev/null || { echo "A index failed L$n"; tail -2 $W/ia/L$n.err; }
|
||||||
|
$B index -k 21 -m 9 --theta 0 -p 256 --label L$n -o $W/ib/L$n.idx "$g" 2>$W/ib/L$n.err >/dev/null || { echo "B index failed L$n"; tail -2 $W/ib/L$n.err; }
|
||||||
|
done
|
||||||
|
|
||||||
|
echo; echo "== T0 on-disk bytes, index built by A vs by B"
|
||||||
|
nd=$(diff -rq $W/ia $W/ib | wc -l); nt=$(find $W/ia -type f | wc -l)
|
||||||
|
echo "files differing: $nd / $nt"
|
||||||
|
diff -rq $W/ia $W/ib | sed "s#$W/##g" | awk "{print \$2}" | sed -E "s#/partitions/.*/#/…/#" | sort | uniq -c | sort -rn | head -8
|
||||||
|
|
||||||
|
h() { md5sum | cut -c1-12; }
|
||||||
|
echo; echo "== T1 A-built single-genome indexes: dump by A vs by B"
|
||||||
|
for i in 1 2 3 4 5; do
|
||||||
|
a=$($A dump $W/ia/L$i.idx 2>/dev/null | h); b=$($B dump $W/ia/L$i.idx 2>/dev/null | h)
|
||||||
|
echo "L$i A:$a B:$b $([ "$a" = "$b" ] && echo OK || echo DIFF)"
|
||||||
|
done
|
||||||
|
|
||||||
|
cp_src() { rm -rf $2; mkdir -p $2; for i in $1; do cp -r $3/L$i.idx $2/; done; }
|
||||||
|
echo; echo "== T2 merges of 4 genomes (presence)"
|
||||||
|
cp_src "1 2 3 4" $W/s_aa $W/ia; cp_src "1 2 3 4" $W/s_ba $W/ia; cp_src "1 2 3 4" $W/s_bb $W/ib
|
||||||
|
$A merge --output $W/M_AA --force --force-presence $W/s_aa/L*.idx 2>$W/m_aa.err; echo "M_AA rc=$?"
|
||||||
|
$B merge --output $W/M_BA --force --force-presence $W/s_ba/L*.idx 2>$W/m_ba.err; echo "M_BA rc=$? (B merging A-built sources)"
|
||||||
|
$B merge --output $W/M_BB --force --force-presence $W/s_bb/L*.idx 2>$W/m_bb.err; echo "M_BB rc=$? (B merging B-built sources)"
|
||||||
|
for X in M_AA M_BA M_BB; do for T in A B; do
|
||||||
|
eval "BIN=\$$T"; $BIN dump $W/$X 2>/dev/null > $W/d_${X}_$T.csv
|
||||||
|
echo "dump $X by $T: $(wc -l < $W/d_${X}_$T.csv) lines raw:$(h < $W/d_${X}_$T.csv) sorted:$(LC_ALL=C sort -S 2G $W/d_${X}_$T.csv | h)"
|
||||||
|
done; done
|
||||||
|
|
||||||
|
echo; echo "== T3 queries"
|
||||||
|
head -c 60000 $W/fa/L1.fasta > $W/q.fa
|
||||||
|
python3 - <<PY >> $W/q.fa
|
||||||
|
import random
|
||||||
|
random.seed(1)
|
||||||
|
print(">rand")
|
||||||
|
print("".join(random.choice("ACGT") for _ in range(5000)))
|
||||||
|
PY
|
||||||
|
for X in ia/L1.idx M_AA M_BA M_BB; do for T in A B; do
|
||||||
|
eval "BIN=\$$T"; echo "query $X by $T: $($BIN query $W/$X $W/q.fa 2>/dev/null | h)"
|
||||||
|
done; done
|
||||||
|
|
||||||
|
echo; echo "== T4 incremental merge (existing A-built merged index + a 5th genome)"
|
||||||
|
for T in A B; do
|
||||||
|
eval "BIN=\$$T"; rm -rf $W/inc_$T $W/dst_$T; cp -r $W/M_AA $W/dst_$T; cp -r $W/ia/L5.idx $W/l5_$T.idx
|
||||||
|
$BIN merge --output $W/inc_$T --force --force-presence $W/dst_$T $W/l5_$T.idx 2>$W/inc_$T.err; echo "inc by $T rc=$?"
|
||||||
|
$BIN dump $W/inc_$T 2>/dev/null | h
|
||||||
|
done
|
||||||
|
echo; echo done
|
||||||
|
'
|
||||||
Executable
+39
@@ -0,0 +1,39 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Follow-up to compat_ptrhash.sh: determinism control (A built twice) and the
|
||||||
|
# incremental merge with its stderr shown. Usage: compat_ptrhash2.sh VOL_A VOL_B OUT
|
||||||
|
set -u
|
||||||
|
VOL_A="$1"; VOL_B="$2"; OUT="$3"
|
||||||
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; REPO="$(cd "$HERE/../.." && pwd)"
|
||||||
|
mkdir -p "$OUT"
|
||||||
|
docker run --rm --platform linux/amd64 -v "$VOL_A:/va:ro" -v "$VOL_B:/vb:ro" \
|
||||||
|
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" -v "$OUT:/out" obikmer-debug:linux-amd64 bash -c '
|
||||||
|
set -u
|
||||||
|
R=x86_64-unknown-linux-musl/release/obikmer
|
||||||
|
mkdir -p /tmp/bin && cp /va/$R /tmp/bin/A && cp /vb/$R /tmp/bin/B; A=/tmp/bin/A; B=/tmp/bin/B
|
||||||
|
W=/tmp/w; rm -rf $W; mkdir -p $W/i1 $W/i2 $W/ib $W/fa
|
||||||
|
exec > >(tee /out/report2.txt) 2>&1
|
||||||
|
mapfile -t G < <(ls -S /genomes_ro/*.fna.gz | sed -n 8,12p)
|
||||||
|
n=0; for gz in "${G[@]}"; do n=$((n+1)); zcat "$gz" > $W/fa/L$n.fasta; done
|
||||||
|
h() { md5sum | cut -c1-12; }
|
||||||
|
echo "== C1 determinism control: same binary A, same input, built twice"
|
||||||
|
$A index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/i1/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
|
||||||
|
$A index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/i2/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
|
||||||
|
$B index -k 21 -m 9 --theta 0 -p 256 --label L1 -o $W/ib/L1.idx $W/fa/L1.fasta >/dev/null 2>&1
|
||||||
|
echo "A vs A: $(diff -rq $W/i1 $W/i2 | wc -l) / $(find $W/i1 -type f | wc -l) files differ"
|
||||||
|
echo "A vs B: $(diff -rq $W/i1 $W/ib | wc -l) / $(find $W/i1 -type f | wc -l) files differ"
|
||||||
|
for f in mphf.bin unitigs.bin evidence.bin unitigs.bin.idx; do
|
||||||
|
p=$(find $W/i1 -name $f | head -1); rel=${p#$W/i1/}
|
||||||
|
echo " $f AvsA: $(find $W/i1 -name $f | while read q; do cmp -s $q $W/i2/${q#$W/i1/} || echo x; done | wc -l)/256 differ AvsB: $(find $W/i1 -name $f | while read q; do cmp -s $q $W/ib/${q#$W/i1/} || echo x; done | wc -l)/256 differ"
|
||||||
|
done
|
||||||
|
echo; echo "== C2 incremental merge, stderr shown"
|
||||||
|
for i in 2 3 4 5; do $A index -k 21 -m 9 --theta 0 -p 256 --label L$i -o $W/i1/L$i.idx $W/fa/L$i.fasta >/dev/null 2>&1; done
|
||||||
|
mkdir -p $W/s; for i in 1 2 3; do cp -r $W/i1/L$i.idx $W/s/; done
|
||||||
|
$A merge --output $W/M3 --force --force-presence $W/s/L*.idx 2>/dev/null; echo "M3 rc=$?"
|
||||||
|
for T in A B; do
|
||||||
|
eval "BIN=\$$T"; rm -rf $W/inc_$T $W/dst_$T $W/l4_$T.idx; cp -r $W/M3 $W/dst_$T; cp -r $W/i1/L4.idx $W/l4_$T.idx
|
||||||
|
$BIN merge --output $W/inc_$T --force --force-presence $W/dst_$T $W/l4_$T.idx 2>$W/inc_$T.err; rc=$?
|
||||||
|
echo "inc by $T rc=$rc"; sed "s/\x1b\[[0-9;]*m//g" $W/inc_$T.err | tail -3
|
||||||
|
[ $rc = 0 ] && echo " dump sorted: $($BIN dump $W/inc_$T 2>/dev/null | LC_ALL=C sort -S 2G | h)"
|
||||||
|
done
|
||||||
|
echo done
|
||||||
|
'
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""gen_fasta.py N OUTDIR [GENOME_LEN] — N deterministic synthetic genomes.
|
||||||
|
|
||||||
|
Each genome = unique random sequence + a random subset of a shared fragment
|
||||||
|
pool (some fragments lightly mutated), so a merge both hits k-mers already
|
||||||
|
in the destination and creates new ones, like related genomes would.
|
||||||
|
"""
|
||||||
|
import random, sys
|
||||||
|
|
||||||
|
n, out = int(sys.argv[1]), sys.argv[2]
|
||||||
|
glen = int(sys.argv[3]) if len(sys.argv) > 3 else 150_000
|
||||||
|
seed = int(sys.argv[4]) if len(sys.argv) > 4 else 20261118
|
||||||
|
rng = random.Random(seed)
|
||||||
|
|
||||||
|
def rnd(m):
|
||||||
|
return "".join(rng.choices("ACGT", k=m))
|
||||||
|
|
||||||
|
def mutate(s, rate=0.01):
|
||||||
|
s = list(s)
|
||||||
|
for i in range(len(s)):
|
||||||
|
if rng.random() < rate:
|
||||||
|
s[i] = rng.choice("ACGT")
|
||||||
|
return "".join(s)
|
||||||
|
|
||||||
|
pool = [rnd(2000) for _ in range(300)] # 600 kb shared pool
|
||||||
|
for g in range(n):
|
||||||
|
parts = [rnd(glen)]
|
||||||
|
for f in rng.sample(range(len(pool)), 60):
|
||||||
|
parts.append(mutate(pool[f]) if rng.random() < 0.3 else pool[f])
|
||||||
|
rng.shuffle(parts)
|
||||||
|
with open(f"{out}/G{g:03d}.fasta", "w") as fh:
|
||||||
|
for j, p in enumerate(parts):
|
||||||
|
fh.write(f">G{g:03d}_{j}\n{p}\n")
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""gen_fasta_real.py N OUTDIR GENOMES_DIR [WINDOW] [SEED] [MUT]
|
||||||
|
|
||||||
|
N derived genomes built from *real* genomes (GENOMES_DIR/*.fna.gz): each one is
|
||||||
|
a random window of WINDOW bp of a randomly chosen real genome, point-mutated at
|
||||||
|
rate MUT (default 3 %, which alters ~47 % of 21-mers), so a merge both finds
|
||||||
|
k-mers already present in the destination and creates new ones, with realistic
|
||||||
|
sequence statistics (no dedicated degenerate structure).
|
||||||
|
"""
|
||||||
|
import glob, gzip, random, sys
|
||||||
|
|
||||||
|
n, out, gdir = int(sys.argv[1]), sys.argv[2], sys.argv[3]
|
||||||
|
window = int(sys.argv[4]) if len(sys.argv) > 4 else 300_000
|
||||||
|
seed = int(sys.argv[5]) if len(sys.argv) > 5 else 20261118
|
||||||
|
mut = float(sys.argv[6]) if len(sys.argv) > 6 else 0.03
|
||||||
|
rng = random.Random(seed)
|
||||||
|
|
||||||
|
def read_fna(path):
|
||||||
|
seq = []
|
||||||
|
with gzip.open(path, "rt") as fh:
|
||||||
|
for line in fh:
|
||||||
|
if not line.startswith(">"):
|
||||||
|
seq.append(line.strip().upper())
|
||||||
|
return "".join(seq)
|
||||||
|
|
||||||
|
genomes = [read_fna(p) for p in sorted(glob.glob(f"{gdir}/*.fna.gz"))]
|
||||||
|
genomes = [g for g in genomes if len(g) > window]
|
||||||
|
for i in range(n):
|
||||||
|
g = rng.choice(genomes)
|
||||||
|
start = rng.randrange(0, len(g) - window)
|
||||||
|
s = list(g[start:start + window])
|
||||||
|
for j in range(len(s)):
|
||||||
|
if s[j] in "ACGT" and rng.random() < mut:
|
||||||
|
s[j] = rng.choice("ACGT")
|
||||||
|
seq = "".join(c if c in "ACGT" else "A" for c in s)
|
||||||
|
with open(f"{out}/R{i:03d}.fasta", "w") as fh:
|
||||||
|
fh.write(f">R{i:03d}\n")
|
||||||
|
for k in range(0, len(seq), 80):
|
||||||
|
fh.write(seq[k:k + 80] + "\n")
|
||||||
Executable
+11
@@ -0,0 +1,11 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# monitor.sh PID OUT [INTERVAL_S] — samples mappings / VM / commit figures until PID exits.
|
||||||
|
PID=$1; OUT=$2; DT=${3:-5}
|
||||||
|
printf 'time,maps,vmsize_kb,vmdata_kb,vmrss_kb,threads,committed_as_kb,commit_limit_kb\n' >"$OUT"
|
||||||
|
while kill -0 "$PID" 2>/dev/null; do
|
||||||
|
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null) || break
|
||||||
|
st=$(grep -E '^(VmSize|VmData|VmRSS|Threads):' "/proc/$PID/status" | awk '{printf "%s,", $2}')
|
||||||
|
mi=$(grep -E '^(Committed_AS|CommitLimit):' /proc/meminfo | awk '{printf "%s,", $2}')
|
||||||
|
echo "$(date +%T),$maps,${st}${mi%,}" >>"$OUT"
|
||||||
|
sleep "$DT"
|
||||||
|
done
|
||||||
Executable
+91
@@ -0,0 +1,91 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# run.sh — builds obikmer (release, debug symbols) and merges the benchmark presence
|
||||||
|
# indexes in the container while sampling mappings/RSS/commit in parallel.
|
||||||
|
#
|
||||||
|
# Env:
|
||||||
|
# PLATFORM linux/amd64 (default, Rosetta) | linux/arm64
|
||||||
|
# GEN if set: build GEN synthetic indexes (gen_fasta.py) instead of using
|
||||||
|
# the benchmark ones; GENOME_LEN (default 150000), K (21), M (9), PART (256)
|
||||||
|
# REAL if set (with GEN): derive genomes from benchmark/run/genomes (gen_fasta_real.py)
|
||||||
|
# SRC_DIR alternative source tree (default: <repo>/src); VARIANT: volume suffix;
|
||||||
|
# LOCKED=0 to allow Cargo.lock changes (needed when a dep is added)
|
||||||
|
# NSRC number of source indexes to merge (default: all 20)
|
||||||
|
# TARGET gnu (default, glibc) | musl (static, same chain as the release CI)
|
||||||
|
# TOOL none (default) | heaptrack (heaptrack output goes to OUT/)
|
||||||
|
# MAXMAP if set, vm.max_map_count is lowered to this inside the VM for the
|
||||||
|
# run and restored afterwards (needs --privileged; affects the whole
|
||||||
|
# OrbStack VM while it runs)
|
||||||
|
# MALLOC_ARENA_MAX, MALLOC_MMAP_THRESHOLD_, MALLOC_TRIM_THRESHOLD_ passed through
|
||||||
|
# OUT host output directory (default: ./out/<timestamp>)
|
||||||
|
set -euo pipefail
|
||||||
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO="$(cd "$HERE/../.." && pwd)"
|
||||||
|
PLATFORM="${PLATFORM:-linux/amd64}"
|
||||||
|
NSRC="${NSRC:-20}"
|
||||||
|
TOOL="${TOOL:-none}"
|
||||||
|
TARGET="${TARGET:-gnu}"
|
||||||
|
OUT="${OUT:-$HERE/out/$(date +%Y%m%d-%H%M%S)}"
|
||||||
|
mkdir -p "$OUT"
|
||||||
|
IMG="obikmer-debug:${PLATFORM//\//-}"
|
||||||
|
VOL="obikmer-debug-target-${PLATFORM//\//-}-${TARGET}${VARIANT:+-$VARIANT}"
|
||||||
|
|
||||||
|
docker run --rm --privileged --platform "$PLATFORM" \
|
||||||
|
-v "${SRC_DIR:-$REPO/src}:/work/src:$([ -n "${SRC_DIR:-}" ] && echo rw || echo ro)" \
|
||||||
|
-v "$REPO/benchmark/run/specimen_index_presence:/idx_ro:ro" \
|
||||||
|
-v "$VOL:/target" \
|
||||||
|
-v "$OUT:/out" \
|
||||||
|
-v "$HERE/monitor.sh:/monitor.sh:ro" \
|
||||||
|
-v "$HERE/gen_fasta.py:/gen_fasta.py:ro" \
|
||||||
|
-v "$HERE/gen_fasta_real.py:/gen_fasta_real.py:ro" \
|
||||||
|
-v "$REPO/benchmark/run/genomes:/genomes_ro:ro" \
|
||||||
|
-e CARGO_TARGET_DIR=/target -e CARGO_PROFILE_RELEASE_DEBUG=1 \
|
||||||
|
-e GEN="${GEN:-}" -e LOCKED_FLAG="$([ "${LOCKED:-1}" = 0 ] || echo --locked)" -e SEED="${SEED:-20261118}" -e REAL="${REAL:-}" -e GENOME_LEN="${GENOME_LEN:-150000}" -e K="${K:-21}" -e M="${M:-9}" -e PART="${PART:-256}" -e RUST_LOG="${RUST_LOG:-}" -e NSRC="$NSRC" -e TARGET="$TARGET" -e TOOL="$TOOL" -e MAXMAP="${MAXMAP:-}" \
|
||||||
|
-e MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-}" \
|
||||||
|
-e MALLOC_MMAP_THRESHOLD_="${MALLOC_MMAP_THRESHOLD_:-}" \
|
||||||
|
-e MALLOC_TRIM_THRESHOLD_="${MALLOC_TRIM_THRESHOLD_:-}" \
|
||||||
|
"$IMG" bash -c '
|
||||||
|
set -euo pipefail
|
||||||
|
for v in MALLOC_ARENA_MAX MALLOC_MMAP_THRESHOLD_ MALLOC_TRIM_THRESHOLD_ RUST_LOG; do
|
||||||
|
[ -n "${!v}" ] || unset $v
|
||||||
|
done
|
||||||
|
if [ "$TARGET" = musl ]; then
|
||||||
|
PKG_CONFIG_ALLOW_CROSS=1 cargo zigbuild --release $LOCKED_FLAG --target x86_64-unknown-linux-musl \
|
||||||
|
--manifest-path /work/src/Cargo.toml --bin obikmer 2>&1 | tail -3
|
||||||
|
BIN=/target/x86_64-unknown-linux-musl/release/obikmer
|
||||||
|
else
|
||||||
|
cargo build --release $LOCKED_FLAG --manifest-path /work/src/Cargo.toml --bin obikmer 2>&1 | tail -3
|
||||||
|
BIN=/target/release/obikmer
|
||||||
|
fi
|
||||||
|
file "$BIN" 2>/dev/null | cut -c1-150 || true
|
||||||
|
# merge writes into its sources (lock files…): work on a copy, never on benchmark/
|
||||||
|
rm -rf /tmp/idx && mkdir -p /tmp/idx
|
||||||
|
if [ -n "$GEN" ]; then
|
||||||
|
mkdir -p /tmp/fa
|
||||||
|
if [ -n "$REAL" ]; then python3 /gen_fasta_real.py "$GEN" /tmp/fa /genomes_ro "$GENOME_LEN" "$SEED"
|
||||||
|
else python3 /gen_fasta.py "$GEN" /tmp/fa "$GENOME_LEN" "$SEED"; fi
|
||||||
|
NSRC=$GEN
|
||||||
|
echo "indexing $GEN synthetic genomes (k=$K m=$M p=$PART)…"
|
||||||
|
ls /tmp/fa/*.fasta | xargs -P 8 -I{} sh -c \
|
||||||
|
"b=\$(basename {} .fasta); $BIN index -k $K -m $M --theta 0 -p $PART --label \$b -o /tmp/idx/\$b.idx {} >/dev/null 2>&1 || echo index failed: \$b"
|
||||||
|
else
|
||||||
|
cp -r /idx_ro/. /tmp/idx/
|
||||||
|
fi
|
||||||
|
mapfile -t SRC < <(find /tmp/idx -mindepth 1 -maxdepth 1 -type d | sort | head -n "$NSRC")
|
||||||
|
echo "sources: ${#SRC[@]} target: $TARGET tool: $TOOL max_map_count: $(cat /proc/sys/vm/max_map_count)"
|
||||||
|
if [ -n "$MAXMAP" ]; then
|
||||||
|
OLD=$(cat /proc/sys/vm/max_map_count); sysctl -w vm.max_map_count="$MAXMAP" >/dev/null
|
||||||
|
trap "sysctl -w vm.max_map_count=$OLD >/dev/null" EXIT
|
||||||
|
fi
|
||||||
|
rm -rf /tmp/merged
|
||||||
|
CMD=("$BIN" merge --output /tmp/merged --force --force-presence "${SRC[@]}")
|
||||||
|
if [ "$TOOL" = heaptrack ]; then CMD=(heaptrack -o /out/heaptrack "${CMD[@]}"); fi
|
||||||
|
"${CMD[@]}" 2>/out/merge.stderr &
|
||||||
|
PID=$!
|
||||||
|
/monitor.sh "$PID" /out/monitor.csv 1 &
|
||||||
|
MON=$!
|
||||||
|
set +e; wait "$PID"; RC=$?; set -e
|
||||||
|
wait "$MON" || true
|
||||||
|
echo "merge exit code: $RC" | tee /out/exit_code.txt
|
||||||
|
tail -5 /out/merge.stderr
|
||||||
|
'
|
||||||
|
echo "outputs: $OUT"
|
||||||
Executable
+102
@@ -0,0 +1,102 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# monitor_pack.sh — 1 Hz sampler for I/O- and memory-bound stages (pack, merge_partitions).
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/monitor_pack.sh --pid PID OUT_DIR [OUTPUT_INDEX_DIR] # attach to a running process
|
||||||
|
# scripts/monitor_pack.sh --cmd OUT_DIR [OUTPUT_INDEX_DIR] -- CMD... # start CMD and sample it
|
||||||
|
# env: DT=1 (seconds between samples)
|
||||||
|
#
|
||||||
|
# Writes OUT_DIR/samples.csv (one row per sample) and OUT_DIR/env.txt (context
|
||||||
|
# needed to read it). Linux only. Never reads smaps (it takes mmap_lock and
|
||||||
|
# slows a process with many mappings); only /proc/PID/{status,io,task/*/stat}
|
||||||
|
# and /proc/meminfo.
|
||||||
|
#
|
||||||
|
# Columns of samples.csv
|
||||||
|
# t epoch seconds
|
||||||
|
# rss_anon_kb, rss_file_kb, rss_shmem_kb from /proc/PID/status
|
||||||
|
# threads thread count
|
||||||
|
# n_R n_D n_S threads running / in uninterruptible sleep (I/O wait) / sleeping
|
||||||
|
# (futex, channel, sleep: NOT disk). n_D is the direct count of
|
||||||
|
# workers blocked on I/O.
|
||||||
|
# maps number of mappings (wc -l /proc/PID/maps)
|
||||||
|
# read_bytes write_bytes /proc/PID/io (cumulative, bytes submitted to the
|
||||||
|
# block layer; buffered writes counted at page dirtying)
|
||||||
|
# rchar wchar /proc/PID/io (cumulative, bytes through read()/write() syscalls)
|
||||||
|
# dirty_kb writeback_kb cached_kb mem_avail_kb /proc/meminfo
|
||||||
|
#
|
||||||
|
# Pair it with the process's own log (RUST_LOG=info, plus obisys=debug for the
|
||||||
|
# "activated N worker(s)" lines and obikindex=debug for per-partition timings)
|
||||||
|
# to align stages on wall-clock time: the log timestamps are UTC ISO-8601, `t`
|
||||||
|
# is epoch seconds.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
MODE=${1:-}; shift || true
|
||||||
|
case "$MODE" in
|
||||||
|
--pid) PID=${1:?PID}; shift ;;
|
||||||
|
--cmd) ;;
|
||||||
|
*) sed -n 2,12p "$0"; exit 2 ;;
|
||||||
|
esac
|
||||||
|
OUT=${1:?OUT_DIR}; shift
|
||||||
|
IDX=""
|
||||||
|
if [ "${1:-}" != "" ] && [ "${1:-}" != "--" ]; then IDX=$1; shift; fi
|
||||||
|
mkdir -p "$OUT"
|
||||||
|
|
||||||
|
if [ "$MODE" = "--cmd" ]; then
|
||||||
|
[ "${1:-}" = "--" ] || { echo "--cmd needs: -- CMD..." >&2; exit 2; }
|
||||||
|
shift
|
||||||
|
"$@" 2>"$OUT/cmd.stderr" &
|
||||||
|
PID=$!
|
||||||
|
fi
|
||||||
|
|
||||||
|
{
|
||||||
|
echo "date_utc: $(date -u +%FT%TZ)"
|
||||||
|
echo "pid: $PID"
|
||||||
|
echo "nproc: $(nproc)"
|
||||||
|
echo "kernel: $(uname -r)"
|
||||||
|
grep -E 'MemTotal|SwapTotal' /proc/meminfo
|
||||||
|
for k in vm.dirty_ratio vm.dirty_background_ratio vm.dirty_bytes vm.dirty_background_bytes \
|
||||||
|
vm.dirty_expire_centisecs vm.dirty_writeback_centisecs vm.max_map_count; do
|
||||||
|
echo "$k: $(sysctl -n $k 2>/dev/null || echo '?')"
|
||||||
|
done
|
||||||
|
echo "ulimit -n: $(ulimit -n)"
|
||||||
|
echo "cmdline: $(tr '\0' ' ' </proc/$PID/cmdline 2>/dev/null)"
|
||||||
|
if [ -n "$IDX" ]; then
|
||||||
|
echo "output_dir: $IDX"
|
||||||
|
echo "fs_type: $(stat -f -c %T "$IDX" 2>/dev/null || echo '?')"
|
||||||
|
echo "mount: $(df -P "$IDX" 2>/dev/null | tail -1)"
|
||||||
|
command -v beegfs-ctl >/dev/null && echo "beegfs: $(beegfs-ctl --getentryinfo "$IDX" 2>&1 | head -8 | tr '\n' ' ')"
|
||||||
|
fi
|
||||||
|
env | grep -E '^(MALLOC_|RUST_LOG|RAYON_)' || true
|
||||||
|
} >"$OUT/env.txt"
|
||||||
|
|
||||||
|
DT=${DT:-1}
|
||||||
|
printf 't,rss_anon_kb,rss_file_kb,rss_shmem_kb,threads,n_R,n_D,n_S,maps,read_bytes,write_bytes,rchar,wchar,dirty_kb,writeback_kb,cached_kb,mem_avail_kb\n' >"$OUT/samples.csv"
|
||||||
|
|
||||||
|
while [ -d "/proc/$PID" ]; do
|
||||||
|
t=$(date +%s)
|
||||||
|
st=$(awk '/^RssAnon:/{a=$2} /^RssFile:/{f=$2} /^RssShmem:/{s=$2} /^Threads:/{th=$2} END{printf "%s,%s,%s,%s",a,f,s,th}' "/proc/$PID/status" 2>/dev/null) || break
|
||||||
|
# thread states: field 3 of /proc/PID/task/TID/stat, after the ")" closing comm
|
||||||
|
states=$(cat /proc/$PID/task/*/stat 2>/dev/null | sed -E 's/^.*\) (.).*/\1/' | sort | uniq -c \
|
||||||
|
| awk '{c[$2]=$1} END{printf "%d,%d,%d", c["R"]+0, c["D"]+0, c["S"]+0}')
|
||||||
|
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null || echo 0)
|
||||||
|
io=$(awk '/^rchar:/{rc=$2} /^wchar:/{wc=$2} /^read_bytes:/{r=$2} /^write_bytes:/{w=$2} END{printf "%s,%s,%s,%s",r,w,rc,wc}' "/proc/$PID/io" 2>/dev/null || echo ",,,")
|
||||||
|
mi=$(awk '/^Dirty:/{d=$2} /^Writeback:/{w=$2} /^Cached:/{c=$2} /^MemAvailable:/{m=$2} END{printf "%s,%s,%s,%s",d,w,c,m}' /proc/meminfo)
|
||||||
|
echo "$t,$st,$states,$maps,$io,$mi" >>"$OUT/samples.csv"
|
||||||
|
sleep "$DT"
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$MODE" = "--cmd" ]; then wait "$PID" || echo "command exit code: $?" >>"$OUT/env.txt"; fi
|
||||||
|
|
||||||
|
# ── summary ──────────────────────────────────────────────────────────────────
|
||||||
|
awk -F, 'NR==1{next}
|
||||||
|
{ n++; if(NR==2){t0=$1; r0=$10; w0=$11}
|
||||||
|
if($2>ma)ma=$2; if($3>mf)mf=$3; if($4>ms)ms=$4; if($5>mt)mt=$5
|
||||||
|
if($6>mR)mR=$6; if($7>mD)mD=$7; if($8>mS)mS=$8; sR+=$6; sD+=$7; sS+=$8
|
||||||
|
if($14>md)md=$14; if($15>mw)mw=$15; if($9>mm)mm=$9
|
||||||
|
t1=$1; r1=$10; w1=$11 }
|
||||||
|
END{ if(!n)exit
|
||||||
|
printf "samples: %d over %ds\n", n, t1-t0
|
||||||
|
printf "peak RssAnon %.1f GB | RssFile %.1f GB | RssShmem %.1f GB | threads %d | maps %d\n", ma/1e6, mf/1e6, ms/1e6, mt, mm
|
||||||
|
printf "threads by state (mean / max): R %.1f / %d | D(I/O wait) %.1f / %d | S %.1f / %d\n", sR/n, mR, sD/n, mD, sS/n, mS
|
||||||
|
printf "peak Dirty %.1f GB | Writeback %.1f GB\n", md/1e6, mw/1e6
|
||||||
|
printf "block I/O over the run: read %.1f GB, write %.1f GB\n", (r1-r0)/1e9, (w1-w0)/1e9 }' "$OUT/samples.csv" | tee "$OUT/summary.txt"
|
||||||
@@ -125,4 +125,26 @@ if [ "$FOUND" -ne 1 ]; then
|
|||||||
fail "no candidate k-mer round-tripped (tried ${#CANDIDATES[@]}) — likely a real regression, not a low-complexity fixture"
|
fail "no candidate k-mer round-tripped (tried ${#CANDIDATES[@]}) — likely a real regression, not a low-complexity fixture"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "PASS: index+query round-trip OK — $N_KMERS kmers indexed, query k-mer '$QUERY_KMER' found"
|
# ── 5. regressions: (a) many partitions against a tiny input leaves most of
|
||||||
|
# them empty (no layer 0 on disk) — must still build; (b) an NCBI-style
|
||||||
|
# `.fna.gz` name must be read, not silently skipped (it used to be
|
||||||
|
# taken for a directory and ignored, yielding an empty index).
|
||||||
|
count_kmers() { # $1 = output index dir, rest = obikmer index args
|
||||||
|
local out="$1"; shift
|
||||||
|
local log="$WORK/regress.log"
|
||||||
|
( cd "$REPO_ROOT/src" && cargo run -q -p obikmer --bin obikmer -- \
|
||||||
|
index -k "$K" -m "$M" --theta 0 -o "$out" "$@" ) > "$log" 2>&1 \
|
||||||
|
|| { cat "$log" >&2; fail "obikmer index $* exited non-zero"; }
|
||||||
|
grep -o '[0-9]* total kmers indexed' "$log" | grep -o '^[0-9]*' || echo 0
|
||||||
|
}
|
||||||
|
|
||||||
|
N_EMPTY="$(count_kmers "$WORK/empty_parts.idx" -p 256 "$WORK/test.fasta")"
|
||||||
|
[ "$N_EMPTY" = "$N_KMERS" ] \
|
||||||
|
|| fail "-p 256 (mostly empty partitions) indexed $N_EMPTY kmers, expected $N_KMERS"
|
||||||
|
|
||||||
|
gzip -c "$WORK/test.fasta" > "$WORK/test.fna.gz"
|
||||||
|
N_GZ="$(count_kmers "$WORK/gz.idx" -p "$PARTITIONS" "$WORK/test.fna.gz")"
|
||||||
|
[ "$N_GZ" = "$N_KMERS" ] \
|
||||||
|
|| fail ".fna.gz input indexed $N_GZ kmers, expected $N_KMERS (input skipped?)"
|
||||||
|
|
||||||
|
echo "PASS: index+query round-trip OK — $N_KMERS kmers indexed, query k-mer '$QUERY_KMER' found; empty-partition and .fna.gz regressions OK"
|
||||||
Executable
+217
@@ -0,0 +1,217 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# smoke_test_merge.sh — end-to-end golden-output oracle for `obikmer merge`
|
||||||
|
#
|
||||||
|
# Non-regression oracle for refactors of obikmerge (see
|
||||||
|
# DevDocMD/implementation/merge_mmap_audit.md). Like smoke_test_index.sh it
|
||||||
|
# drives the real CLI, never the library APIs.
|
||||||
|
#
|
||||||
|
# What it does:
|
||||||
|
# 1. builds `obikmer` (debug)
|
||||||
|
# 2. generates (on --record) 4 small deterministic FASTA files A, B, C, D sharing
|
||||||
|
# fragments, so that merge both hits k-mers already present in the
|
||||||
|
# destination ("existing layer" path) and creates new ones ("new layer"
|
||||||
|
# path). D carries a short motif repeated > 255 times so the count track
|
||||||
|
# exercises the compact-int overflow (counts >= 255).
|
||||||
|
# 3. on --record, indexes each genome twice (presence, --with-counts) and
|
||||||
|
# freezes the indexes in DIR/sources; --check reuses them (see below),
|
||||||
|
# with -p PARTITIONS partitions
|
||||||
|
# 4. runs the merge scenarios:
|
||||||
|
# s1_presence / s1_count merge A B C D
|
||||||
|
# s3_dense merge A B C D --dense (presence)
|
||||||
|
# Scenario 2 (merge A B -> AB, then merge AB C D: a packed merge output
|
||||||
|
# re-merged as a *source*) is checked relatively, not against the
|
||||||
|
# golden dumps: its table (k-mer x genome label) must equal s1_presence's,
|
||||||
|
# whatever the column order and layer placement. A packed index cannot
|
||||||
|
# be a merge *destination* (its matrices are no longer column-by-column,
|
||||||
|
# see `can_be_destination` in obikmerge); `choose_base` therefore picks
|
||||||
|
# an unpacked source, and merging only packed indexes is refused with a
|
||||||
|
# clear error, which this script also checks.
|
||||||
|
# 5. dumps every result with `obikmer dump --debug` (partition + layer
|
||||||
|
# columns included, so a k-mer moving between layers is caught), keeps
|
||||||
|
# the CSV header first and sorts the rows (LC_ALL=C)
|
||||||
|
# 6. --record DIR writes the sorted dumps to DIR;
|
||||||
|
# --check DIR regenerates them and diffs against DIR
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/smoke_test_merge.sh (--record DIR | --check DIR) [-p PARTITIONS] [--keep]
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
K=21
|
||||||
|
M=9
|
||||||
|
PARTITIONS=4
|
||||||
|
KEEP=0
|
||||||
|
MODE=""
|
||||||
|
DIR=""
|
||||||
|
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--record) MODE=record; DIR="$2"; shift 2 ;;
|
||||||
|
--check) MODE=check; DIR="$2"; shift 2 ;;
|
||||||
|
-p) PARTITIONS="$2"; shift 2 ;;
|
||||||
|
--keep) KEEP=1; shift ;;
|
||||||
|
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[ -n "$MODE" ] || { echo "usage: $0 (--record DIR | --check DIR) [-p N] [--keep]" >&2; exit 2; }
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
case "$DIR" in /*) ;; *) DIR="$PWD/$DIR" ;; esac
|
||||||
|
WORK="$(mktemp -d -t obikmer_merge_smoke.XXXXXX)"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
if [ "$KEEP" -eq 1 ]; then echo "kept: $WORK"; else rm -rf "$WORK"; fi
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
fail() {
|
||||||
|
echo "FAIL: $1" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── 1. build ────────────────────────────────────────────────────────────────
|
||||||
|
if ! (cd "$REPO_ROOT/src" && cargo build -q -p obikmer --bin obikmer) > "$WORK/build.log" 2>&1; then
|
||||||
|
cat "$WORK/build.log" >&2
|
||||||
|
fail "cargo build failed"
|
||||||
|
fi
|
||||||
|
BIN="$REPO_ROOT/src/target/debug/obikmer"
|
||||||
|
|
||||||
|
run() { # run LOGNAME CMD... — runs obikmer, dumps its log on failure
|
||||||
|
local log="$WORK/$1.log"; shift
|
||||||
|
if ! "$BIN" "$@" > "$log" 2>&1; then
|
||||||
|
cat "$log" >&2
|
||||||
|
fail "obikmer $1 exited non-zero"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── 2. deterministic genomes with shared fragments ──────────────────────────
|
||||||
|
python3 - "$WORK" <<'EOF'
|
||||||
|
import random, sys
|
||||||
|
out = sys.argv[1]
|
||||||
|
rng = random.Random(20261117)
|
||||||
|
def rnd(n): return "".join(rng.choice("ACGT") for _ in range(n))
|
||||||
|
frag = {i: rnd(600) for i in range(5)} # shared fragments F0..F4
|
||||||
|
motif = rnd(40) # tandem motif (counts >= 255)
|
||||||
|
genomes = {
|
||||||
|
"A": [rnd(1500), frag[0], rnd(300), frag[1]],
|
||||||
|
"B": [frag[1], rnd(1200), frag[2], motif * 3],
|
||||||
|
"C": [frag[0], rnd(800), frag[2], rnd(400), frag[3]],
|
||||||
|
"D": [frag[3], rnd(1000), motif * 300, frag[4], rnd(200), frag[0][:250]],
|
||||||
|
}
|
||||||
|
for name, parts in genomes.items():
|
||||||
|
with open(f"{out}/{name}.fasta", "w") as f:
|
||||||
|
for j, p in enumerate(parts):
|
||||||
|
f.write(f">{name}_{j}\n{p}\n")
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# ── 3. source indexes: built on --record, frozen in DIR/sources, reused on
|
||||||
|
# --check. `obikmer index` is not byte-stable run to run (unitig
|
||||||
|
# segmentation, hence unitigs.bin size, varies — measured 2026-11-17),
|
||||||
|
# and the merge base is the source with the largest unitigs.bin
|
||||||
|
# (`choose_base`), so re-indexing on every check would make the merge
|
||||||
|
# base — and the column order — flip between A and C. Freezing the
|
||||||
|
# sources makes this an oracle of `merge` alone. Copied (not linked)
|
||||||
|
# into WORK, since merge's bootstrap hard-links from its base source.
|
||||||
|
if [ "$MODE" = record ]; then
|
||||||
|
for G in A B C D; do
|
||||||
|
run "index_${G}_pres" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
|
||||||
|
--label "$G" -o "$WORK/$G.pres.idx" "$WORK/$G.fasta"
|
||||||
|
run "index_${G}_cnt" index -k "$K" -m "$M" --theta 0 -p "$PARTITIONS" \
|
||||||
|
--with-counts --label "$G" -o "$WORK/$G.cnt.idx" "$WORK/$G.fasta"
|
||||||
|
done
|
||||||
|
rm -rf "$DIR/sources"
|
||||||
|
mkdir -p "$DIR/sources"
|
||||||
|
cp -R "$WORK"/*.idx "$DIR/sources/"
|
||||||
|
else
|
||||||
|
[ -d "$DIR/sources" ] || fail "no frozen source indexes in $DIR/sources (run --record first)"
|
||||||
|
cp -R "$DIR/sources"/*.idx "$WORK/"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── 4. merge scenarios ──────────────────────────────────────────────────────
|
||||||
|
OUT="$WORK/out"
|
||||||
|
mkdir -p "$OUT"
|
||||||
|
dump() { # dump NAME INDEX — header first, rows sorted
|
||||||
|
local raw="$WORK/$1.dump.csv"
|
||||||
|
if ! "$BIN" dump --debug "$2" > "$raw" 2> "$WORK/dump_$1.log"; then
|
||||||
|
cat "$WORK/dump_$1.log" >&2
|
||||||
|
fail "obikmer dump $1 exited non-zero"
|
||||||
|
fi
|
||||||
|
{ head -n 1 "$raw"; tail -n +2 "$raw" | LC_ALL=C sort; } > "$OUT/$1.csv"
|
||||||
|
}
|
||||||
|
|
||||||
|
for T in pres cnt; do
|
||||||
|
run "s1_$T" merge -o "$WORK/s1_$T.idx" \
|
||||||
|
"$WORK/A.$T.idx" "$WORK/B.$T.idx" "$WORK/C.$T.idx" "$WORK/D.$T.idx"
|
||||||
|
dump "s1_$T" "$WORK/s1_$T.idx"
|
||||||
|
done
|
||||||
|
|
||||||
|
run s3_dense merge --dense -o "$WORK/s3_dense.idx" \
|
||||||
|
"$WORK/A.pres.idx" "$WORK/B.pres.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
|
||||||
|
dump s3_dense "$WORK/s3_dense.idx"
|
||||||
|
|
||||||
|
# scenario 2: re-merge a packed merge output as a source (checked against s1)
|
||||||
|
run s2_ab merge -o "$WORK/s2_ab.idx" "$WORK/A.pres.idx" "$WORK/B.pres.idx"
|
||||||
|
run s2 merge -o "$WORK/s2.idx" "$WORK/s2_ab.idx" "$WORK/C.pres.idx" "$WORK/D.pres.idx"
|
||||||
|
for N in s1_pres s2; do
|
||||||
|
"$BIN" dump "$WORK/$N.idx" > "$WORK/plain_$N.csv" 2> "$WORK/dump_plain_$N.log" \
|
||||||
|
|| { cat "$WORK/dump_plain_$N.log" >&2; fail "obikmer dump $N exited non-zero"; }
|
||||||
|
done
|
||||||
|
cat > "$WORK/compare_tables.py" <<'PY'
|
||||||
|
import csv, sys
|
||||||
|
def table(path):
|
||||||
|
rows = list(csv.reader(open(path)))
|
||||||
|
head = rows[0]
|
||||||
|
return {r[0]: {head[i]: r[i] for i in range(1, len(head))} for r in rows[1:]}
|
||||||
|
a, b = table(sys.argv[1]), table(sys.argv[2])
|
||||||
|
if a != b:
|
||||||
|
print(f"k-mers: {len(a)} vs {len(b)}; only in flat: {len(set(a) - set(b))}; "
|
||||||
|
f"only in re-merge: {len(set(b) - set(a))}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
PY
|
||||||
|
python3 "$WORK/compare_tables.py" "$WORK/plain_s1_pres.csv" "$WORK/plain_s2.csv" \
|
||||||
|
|| fail "scenario 2 (re-merge of a packed output) differs from the flat merge"
|
||||||
|
echo "ok s2 (re-merge of a packed output) equals the flat merge"
|
||||||
|
if "$BIN" merge -o "$WORK/s2_bad.idx" "$WORK/s2_ab.idx" "$WORK/s1_pres.idx" > "$WORK/s2_bad.log" 2>&1; then
|
||||||
|
fail "merging only packed indexes should be refused"
|
||||||
|
fi
|
||||||
|
grep -q "none of the sources can be the destination" "$WORK/s2_bad.log" \
|
||||||
|
|| { cat "$WORK/s2_bad.log" >&2; fail "refusal of an all-packed merge lacks the expected message"; }
|
||||||
|
echo "ok merge of only packed indexes is refused with a clear message"
|
||||||
|
|
||||||
|
# sanity: no dump may be empty (header + at least one row)
|
||||||
|
for f in "$OUT"/*.csv; do
|
||||||
|
[ "$(wc -l < "$f")" -gt 1 ] || fail "empty dump: $(basename "$f")"
|
||||||
|
done
|
||||||
|
|
||||||
|
# ── 5. record or check ──────────────────────────────────────────────────────
|
||||||
|
if [ "$MODE" = record ]; then
|
||||||
|
mkdir -p "$DIR"
|
||||||
|
rm -f "$DIR"/*.csv
|
||||||
|
cp "$OUT"/*.csv "$DIR"/
|
||||||
|
echo "RECORDED: $(ls "$OUT" | wc -l | tr -d ' ') dumps → $DIR"
|
||||||
|
wc -l "$DIR"/*.csv
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
[ -d "$DIR" ] || fail "reference directory not found: $DIR"
|
||||||
|
STATUS=0
|
||||||
|
for ref in "$DIR"/*.csv; do
|
||||||
|
name="$(basename "$ref")"
|
||||||
|
if [ ! -f "$OUT/$name" ]; then
|
||||||
|
echo "FAIL $name (not regenerated)"; STATUS=1
|
||||||
|
elif diff -q "$ref" "$OUT/$name" > /dev/null; then
|
||||||
|
echo "ok $name"
|
||||||
|
else
|
||||||
|
echo "FAIL $name"; { diff "$ref" "$OUT/$name" || true; } | head -20; STATUS=1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
for new in "$OUT"/*.csv; do
|
||||||
|
[ -f "$DIR/$(basename "$new")" ] || { echo "FAIL $(basename "$new") (no reference)"; STATUS=1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$STATUS" -eq 0 ]; then
|
||||||
|
echo "PASS: all merge dumps identical to $DIR"
|
||||||
|
else
|
||||||
|
echo "FAIL: merge dumps differ from $DIR" >&2
|
||||||
|
fi
|
||||||
|
exit "$STATUS"
|
||||||
Executable
+53
@@ -0,0 +1,53 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# watch_merge.sh PID OUT.csv — samples a running process and kills it past a threshold.
|
||||||
|
#
|
||||||
|
# Every INTERVAL seconds appends one CSV row: number of memory mappings
|
||||||
|
# (lines of /proc/PID/maps), VmSize, VmData, VmRSS, threads, and the
|
||||||
|
# system-wide Committed_AS / CommitLimit. If the mapping count exceeds
|
||||||
|
# MAPS_MAX, VmRSS exceeds RSS_MAX_GB, or VmSize exceeds VMSIZE_MAX_GB (stay
|
||||||
|
# below `ulimit -v`: past it, allocations fail instead of the guard firing), the process receives SIGTERM (then
|
||||||
|
# SIGKILL after 30 s) and the reason is written to OUT.csv.reason — a guard
|
||||||
|
# so a runaway merge ends with a clean message instead of a failed
|
||||||
|
# allocation (or the OOM killer on a shared node).
|
||||||
|
#
|
||||||
|
# Reading /proc/PID/maps is itself slow when the process has hundreds of
|
||||||
|
# thousands of mappings (it held the kernel's mmap lock in earlier
|
||||||
|
# measurements): keep INTERVAL >= 5. It never reads /proc/PID/smaps.
|
||||||
|
#
|
||||||
|
# Usage: scripts/watch_merge.sh PID OUT.csv
|
||||||
|
# env: INTERVAL=5 MAPS_MAX=500000 RSS_MAX_GB=400 VMSIZE_MAX_GB=440
|
||||||
|
#
|
||||||
|
# Typical use (Linux only):
|
||||||
|
# nohup scripts/watch_merge.sh "$(pgrep -f 'obikmer merge' | head -1)" /tmp/watch.csv &
|
||||||
|
set -u
|
||||||
|
PID=${1:?usage: watch_merge.sh PID OUT.csv}
|
||||||
|
OUT=${2:?usage: watch_merge.sh PID OUT.csv}
|
||||||
|
INTERVAL=${INTERVAL:-5}
|
||||||
|
MAPS_MAX=${MAPS_MAX:-500000}
|
||||||
|
RSS_MAX_KB=$(( ${RSS_MAX_GB:-400} * 1024 * 1024 ))
|
||||||
|
VMSIZE_MAX_KB=$(( ${VMSIZE_MAX_GB:-440} * 1024 * 1024 ))
|
||||||
|
|
||||||
|
[ -r "/proc/$PID/maps" ] || { echo "cannot read /proc/$PID/maps" >&2; exit 1; }
|
||||||
|
printf 'time,maps,vmsize_kb,vmdata_kb,vmrss_kb,threads,committed_as_kb,commit_limit_kb\n' >"$OUT"
|
||||||
|
|
||||||
|
while kill -0 "$PID" 2>/dev/null; do
|
||||||
|
maps=$(wc -l <"/proc/$PID/maps" 2>/dev/null) || break
|
||||||
|
st=$(grep -E '^(VmSize|VmData|VmRSS|Threads):' "/proc/$PID/status" 2>/dev/null | awk '{printf "%s,", $2}')
|
||||||
|
rss=$(grep '^VmRSS:' "/proc/$PID/status" 2>/dev/null | awk '{print $2}')
|
||||||
|
vsz=$(grep '^VmSize:' "/proc/$PID/status" 2>/dev/null | awk '{print $2}')
|
||||||
|
mi=$(grep -E '^(Committed_AS|CommitLimit):' /proc/meminfo | awk '{printf "%s,", $2}')
|
||||||
|
echo "$(date +%T),$maps,${st}${mi%,}" >>"$OUT"
|
||||||
|
|
||||||
|
reason=""
|
||||||
|
[ "$maps" -gt "$MAPS_MAX" ] && reason="mappings $maps > MAPS_MAX $MAPS_MAX"
|
||||||
|
[ -n "${rss:-}" ] && [ "$rss" -gt "$RSS_MAX_KB" ] && reason="VmRSS ${rss} kB > RSS_MAX_GB"
|
||||||
|
[ -n "${vsz:-}" ] && [ "$vsz" -gt "$VMSIZE_MAX_KB" ] && reason="VmSize ${vsz} kB > VMSIZE_MAX_GB"
|
||||||
|
if [ -n "$reason" ]; then
|
||||||
|
echo "$(date +%T) GUARD: $reason — sending SIGTERM to $PID" | tee "$OUT.reason" >&2
|
||||||
|
kill -TERM "$PID" 2>/dev/null
|
||||||
|
for _ in $(seq 30); do kill -0 "$PID" 2>/dev/null || exit 0; sleep 1; done
|
||||||
|
kill -KILL "$PID" 2>/dev/null
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
sleep "$INTERVAL"
|
||||||
|
done
|
||||||
Generated
+10
-20
@@ -422,7 +422,7 @@ version = "3.1.1"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
|
checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"windows-sys 0.61.2",
|
"windows-sys 0.52.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -758,7 +758,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
|
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"libc",
|
"libc",
|
||||||
"windows-sys 0.61.2",
|
"windows-sys 0.52.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -1040,7 +1040,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
|
|||||||
dependencies = [
|
dependencies = [
|
||||||
"hermit-abi",
|
"hermit-abi",
|
||||||
"libc",
|
"libc",
|
||||||
"windows-sys 0.61.2",
|
"windows-sys 0.52.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -1408,10 +1408,8 @@ version = "0.1.0"
|
|||||||
dependencies = [
|
dependencies = [
|
||||||
"crossbeam-channel",
|
"crossbeam-channel",
|
||||||
"hashbrown 0.17.1",
|
"hashbrown 0.17.1",
|
||||||
"obifastwrite",
|
|
||||||
"obikseq",
|
"obikseq",
|
||||||
"rayon",
|
"rayon",
|
||||||
"tracing",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -1492,14 +1490,10 @@ dependencies = [
|
|||||||
"ndarray",
|
"ndarray",
|
||||||
"niffler",
|
"niffler",
|
||||||
"obicompactvec",
|
"obicompactvec",
|
||||||
"obidebruinj",
|
|
||||||
"obikentropy",
|
|
||||||
"obikseq",
|
"obikseq",
|
||||||
"obipipeline",
|
|
||||||
"obiread",
|
"obiread",
|
||||||
"obiskio",
|
"obiskio",
|
||||||
"obisys",
|
"obisys",
|
||||||
"obitaxonomy",
|
|
||||||
"ptr_hash",
|
"ptr_hash",
|
||||||
"rayon",
|
"rayon",
|
||||||
"serde",
|
"serde",
|
||||||
@@ -1537,7 +1531,7 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "obikmer"
|
name = "obikmer"
|
||||||
version = "1.4.1"
|
version = "1.4.6"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"clap",
|
"clap",
|
||||||
"csv",
|
"csv",
|
||||||
@@ -1556,7 +1550,6 @@ dependencies = [
|
|||||||
"obikrope",
|
"obikrope",
|
||||||
"obikselect",
|
"obikselect",
|
||||||
"obikseq",
|
"obikseq",
|
||||||
"obiksession",
|
|
||||||
"obikstats",
|
"obikstats",
|
||||||
"obipipeline",
|
"obipipeline",
|
||||||
"obiread",
|
"obiread",
|
||||||
@@ -1565,6 +1558,7 @@ dependencies = [
|
|||||||
"rayon",
|
"rayon",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_yaml",
|
"serde_yaml",
|
||||||
|
"tikv-jemallocator",
|
||||||
"tracing",
|
"tracing",
|
||||||
"tracing-subscriber",
|
"tracing-subscriber",
|
||||||
]
|
]
|
||||||
@@ -1582,7 +1576,6 @@ dependencies = [
|
|||||||
"obiskio",
|
"obiskio",
|
||||||
"obisys",
|
"obisys",
|
||||||
"rayon",
|
"rayon",
|
||||||
"tempfile",
|
|
||||||
"tracing",
|
"tracing",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -1601,7 +1594,6 @@ dependencies = [
|
|||||||
"obikseq",
|
"obikseq",
|
||||||
"obipipeline",
|
"obipipeline",
|
||||||
"obiread",
|
"obiread",
|
||||||
"obiskbuilder",
|
|
||||||
"obiskio",
|
"obiskio",
|
||||||
"obisys",
|
"obisys",
|
||||||
"petgraph",
|
"petgraph",
|
||||||
@@ -1647,7 +1639,6 @@ dependencies = [
|
|||||||
name = "obikrope"
|
name = "obikrope"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"bytes",
|
|
||||||
"criterion2",
|
"criterion2",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -1725,7 +1716,6 @@ name = "obiskbuilder"
|
|||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"criterion2",
|
"criterion2",
|
||||||
"lazy_static",
|
|
||||||
"obikentropy",
|
"obikentropy",
|
||||||
"obikrope",
|
"obikrope",
|
||||||
"obikseq",
|
"obikseq",
|
||||||
@@ -1985,9 +1975,9 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "ptr_hash"
|
name = "ptr_hash"
|
||||||
version = "2.1.2"
|
version = "2.1.5"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "b22cf1be774892680e056d3387db0dcaa4fd46964b270671ef661e127186b969"
|
checksum = "0d2381547e2e36c72a455c1cc418d5c60ecd6e94f4819d7560456108e99a4efa"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"bitvec",
|
"bitvec",
|
||||||
"cacheline-ef",
|
"cacheline-ef",
|
||||||
@@ -2279,7 +2269,7 @@ dependencies = [
|
|||||||
"errno",
|
"errno",
|
||||||
"libc",
|
"libc",
|
||||||
"linux-raw-sys",
|
"linux-raw-sys",
|
||||||
"windows-sys 0.61.2",
|
"windows-sys 0.52.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -2611,7 +2601,7 @@ dependencies = [
|
|||||||
"getrandom 0.4.3",
|
"getrandom 0.4.3",
|
||||||
"once_cell",
|
"once_cell",
|
||||||
"rustix",
|
"rustix",
|
||||||
"windows-sys 0.61.2",
|
"windows-sys 0.52.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -3016,7 +3006,7 @@ version = "0.1.11"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
|
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"windows-sys 0.61.2",
|
"windows-sys 0.52.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ use std::io;
|
|||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use crate::bitvec::PersistentBitVecBuilder;
|
use crate::bitvec::PersistentBitVecBuilder;
|
||||||
|
use crate::mem_builder::MemBitVecBuilder;
|
||||||
use crate::meta::MatrixMeta;
|
use crate::meta::MatrixMeta;
|
||||||
use crate::tempbitvec::TempBitVec;
|
use crate::tempbitvec::TempBitVec;
|
||||||
use crate::tempintvec::TempCompactIntVec;
|
use crate::tempintvec::TempCompactIntVec;
|
||||||
@@ -43,6 +44,14 @@ impl PersistentBitMatrixBuilder {
|
|||||||
PersistentBitVecBuilder::new(self.n, &path)
|
PersistentBitVecBuilder::new(self.n, &path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
|
||||||
|
/// the column is built in RAM and written once at `close`.
|
||||||
|
pub fn add_mem_col(&mut self) -> MemBitVecBuilder {
|
||||||
|
let path = col_path(&self.dir, self.n_cols);
|
||||||
|
self.n_cols += 1;
|
||||||
|
MemBitVecBuilder::new(self.n, &path)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn add_col_ones(&mut self) -> io::Result<PersistentBitVecBuilder> {
|
pub fn add_col_ones(&mut self) -> io::Result<PersistentBitVecBuilder> {
|
||||||
let path = col_path(&self.dir, self.n_cols);
|
let path = col_path(&self.dir, self.n_cols);
|
||||||
self.n_cols += 1;
|
self.n_cols += 1;
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ mod packed;
|
|||||||
mod pairwise;
|
mod pairwise;
|
||||||
mod persistent;
|
mod persistent;
|
||||||
mod sparse;
|
mod sparse;
|
||||||
|
mod sparse_columnar;
|
||||||
|
|
||||||
pub use builder::PersistentBitMatrixBuilder;
|
pub use builder::PersistentBitMatrixBuilder;
|
||||||
pub use group_ops::batch_presence_counts;
|
pub use group_ops::batch_presence_counts;
|
||||||
|
|||||||
@@ -105,7 +105,7 @@ impl PackedBitMatrix {
|
|||||||
/// Reads just the `n_cols` field from an existing packed matrix's header,
|
/// Reads just the `n_cols` field from an existing packed matrix's header,
|
||||||
/// without mapping the file. Used by `pack_bit_matrix` to tell a genuinely
|
/// without mapping the file. Used by `pack_bit_matrix` to tell a genuinely
|
||||||
/// complete pack from a stale one that predates a later column-widening.
|
/// complete pack from a stale one that predates a later column-widening.
|
||||||
fn packed_bit_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
pub(super) fn packed_bit_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
||||||
let mut f = File::open(path)?;
|
let mut f = File::open(path)?;
|
||||||
let mut header = [0u8; PBMX_HEADER];
|
let mut header = [0u8; PBMX_HEADER];
|
||||||
f.read_exact(&mut header)?;
|
f.read_exact(&mut header)?;
|
||||||
|
|||||||
@@ -304,6 +304,10 @@ impl PersistentBitMatrix {
|
|||||||
impl ColumnWeights for PersistentBitMatrix {
|
impl ColumnWeights for PersistentBitMatrix {
|
||||||
#[inline]
|
#[inline]
|
||||||
fn col_weights(&self) -> Array1<u64> { self.count_ones() }
|
fn col_weights(&self) -> Array1<u64> { self.count_ones() }
|
||||||
|
/// A presence matrix holds no counts: a set bit is present whatever
|
||||||
|
/// `threshold` is.
|
||||||
|
#[inline]
|
||||||
|
fn count_geq(&self, _threshold: u32) -> Array1<u64> { self.count_ones() }
|
||||||
}
|
}
|
||||||
|
|
||||||
impl BitPartials for PersistentBitMatrix {
|
impl BitPartials for PersistentBitMatrix {
|
||||||
|
|||||||
@@ -31,14 +31,12 @@ use ndarray::{Array1, Array2};
|
|||||||
use crate::traits::{BitPartials, ColumnWeights};
|
use crate::traits::{BitPartials, ColumnWeights};
|
||||||
use crate::eliasfano::{EliasFano, EliasFanoBuilder};
|
use crate::eliasfano::{EliasFano, EliasFanoBuilder};
|
||||||
use crate::fixedintvec::{PersistentFixedIntVec, PersistentFixedIntVecBuilder, bit_width_for_range};
|
use crate::fixedintvec::{PersistentFixedIntVec, PersistentFixedIntVecBuilder, bit_width_for_range};
|
||||||
use crate::meta::field;
|
use crate::meta::{MatrixMeta, field};
|
||||||
use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
use crate::rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
||||||
|
|
||||||
use super::PersistentBitMatrix;
|
use super::PersistentBitMatrix;
|
||||||
use super::col_path;
|
use super::col_path;
|
||||||
use super::columnar::ColumnarBitMatrix;
|
use super::packed::{PackedBitMatrix, pack_bit_matrix, packed_bit_matrix_n_cols};
|
||||||
use super::packed::PackedBitMatrix;
|
|
||||||
use crate::views::nonzero_triples;
|
|
||||||
|
|
||||||
fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") }
|
fn is_multi_path(dir: &Path) -> PathBuf { dir.join("is_multi.prsb") }
|
||||||
fn singleton_path(dir: &Path) -> PathBuf { dir.join("singleton.pfiv") }
|
fn singleton_path(dir: &Path) -> PathBuf { dir.join("singleton.pfiv") }
|
||||||
@@ -501,12 +499,12 @@ impl PersistentSparseBitMatrixBuilder {
|
|||||||
/// existing index.
|
/// existing index.
|
||||||
///
|
///
|
||||||
/// Processes rows in batches of [`BUILD_FROM_DENSE_BATCH`] instead of
|
/// Processes rows in batches of [`BUILD_FROM_DENSE_BATCH`] instead of
|
||||||
/// one row at a time: `nonzero_triples(..., sorted_slots=true)` walks
|
/// one row at a time: [`BitSliceView::for_each_set_in`] walks each
|
||||||
/// each column *once per batch*, sequentially — matching how
|
/// column *once per batch*, word by word, visiting only the set bits —
|
||||||
/// `PersistentBitMatrix::nonzero_iter` already batches slots for
|
/// rather than `fill_row`'s row-major loop, which re-reads every column
|
||||||
/// queries — rather than `fill_row`'s row-major loop, which re-reads
|
/// with one `get()` per cell and jumps between all `n_cols` columns on
|
||||||
/// every column with one `get()` per cell and jumps between all
|
/// every single row. Columns are visited in ascending order, so each
|
||||||
/// `n_cols` column files on every single row.
|
/// row's genome list comes out sorted, as `push_row` requires.
|
||||||
pub fn build_from_dense(dense: &PersistentBitMatrix, dir: &Path) -> io::Result<Self> {
|
pub fn build_from_dense(dense: &PersistentBitMatrix, dir: &Path) -> io::Result<Self> {
|
||||||
let n = dense.n();
|
let n = dense.n();
|
||||||
let n_cols = dense.n_cols();
|
let n_cols = dense.n_cols();
|
||||||
@@ -515,9 +513,10 @@ impl PersistentSparseBitMatrixBuilder {
|
|||||||
let mut start = 0;
|
let mut start = 0;
|
||||||
while start < n {
|
while start < n {
|
||||||
let end = (start + BUILD_FROM_DENSE_BATCH).min(n);
|
let end = (start + BUILD_FROM_DENSE_BATCH).min(n);
|
||||||
let slots: Vec<usize> = (start..end).collect();
|
for c in 0..n_cols {
|
||||||
for (row, c, _) in nonzero_triples(&slots, n_cols, |c| dense.col_view(c), true) {
|
dense.col_view(c).for_each_set_in(start..end, |slot| {
|
||||||
row_buffers[row].push(c as u32);
|
row_buffers[slot - start].push(c as u32);
|
||||||
|
});
|
||||||
}
|
}
|
||||||
for buf in row_buffers.iter_mut().take(end - start) {
|
for buf in row_buffers.iter_mut().take(end - start) {
|
||||||
builder.push_row(buf);
|
builder.push_row(buf);
|
||||||
@@ -560,6 +559,11 @@ impl ColumnWeights for PersistentSparseBitMatrix {
|
|||||||
fn col_weights(&self) -> Array1<u64> {
|
fn col_weights(&self) -> Array1<u64> {
|
||||||
self.count_ones()
|
self.count_ones()
|
||||||
}
|
}
|
||||||
|
/// A presence matrix holds no counts: a set bit is present whatever
|
||||||
|
/// `threshold` is.
|
||||||
|
fn count_geq(&self, _threshold: u32) -> Array1<u64> {
|
||||||
|
self.count_ones()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl BitPartials for PersistentSparseBitMatrix {
|
impl BitPartials for PersistentSparseBitMatrix {
|
||||||
@@ -615,37 +619,47 @@ impl BitPartials for PersistentSparseBitMatrix {
|
|||||||
m
|
m
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/// sparse on-disk format in place, mirroring [`super::pack_bit_matrix`]'s
|
|
||||||
/// convention (old-format files removed only after the new format is
|
/// Converts a layer's presence matrix (columnar `col_*.pbiv` or dense
|
||||||
/// fully written, so a crash mid-conversion leaves the previous, still
|
/// `matrix.pbmx`) to the sparse on-disk format in place. Old-format files
|
||||||
/// valid format rather than a half-written one). Idempotent — does
|
/// are removed only after the new format is fully written, so a crash
|
||||||
|
/// mid-conversion leaves a previous, still valid format. Idempotent: does
|
||||||
/// nothing if `is_multi.prsb` already exists.
|
/// nothing if `is_multi.prsb` already exists.
|
||||||
///
|
///
|
||||||
/// The dense→sparse transpose (`build_from_dense`) only needs `fill_row`,
|
/// Columnar data is transposed straight into the sparse format, a window of
|
||||||
/// which costs the same `O(n_cols)` per row on `Columnar` as on `Packed`
|
/// rows at a time, one column file open at a time
|
||||||
/// (see `ColumnarBitMatrix::fill_row`) — so this reads straight from
|
/// ([`PersistentSparseBitMatrixBuilder::build_from_columnar`]): no dense
|
||||||
/// whichever dense form is already on disk. In particular, when this runs
|
/// intermediate is written, and a worker holds no mapping of the columns
|
||||||
/// right after a build (the common case: `presence/` is still `Columnar`,
|
/// whatever their number. A dense `matrix.pbmx` is only read when it is the
|
||||||
/// `matrix.pbmx` doesn't exist yet), it never materialises a full packed
|
/// current form of the layer: no columnar data pending, or a `matrix.pbmx`
|
||||||
/// copy of the presence matrix just to read it back once and delete it.
|
/// that already has as many columns as the columnar metadata says.
|
||||||
pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> {
|
pub fn pack_sparse_bit_matrix(dir: &Path) -> io::Result<()> {
|
||||||
if is_multi_path(dir).exists() {
|
if is_multi_path(dir).exists() {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
|
|
||||||
let packed_path = dir.join("matrix.pbmx");
|
let packed_path = dir.join("matrix.pbmx");
|
||||||
if packed_path.exists() {
|
|
||||||
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?);
|
if let Ok(meta) = MatrixMeta::load(dir) {
|
||||||
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
if packed_bit_matrix_n_cols(&packed_path).ok() != Some(meta.n_cols) {
|
||||||
drop(dense);
|
// Columnar data is the freshest form (any `matrix.pbmx` here is
|
||||||
fs::remove_file(&packed_path)?;
|
// stale, e.g. copied from a merge base before the layer was
|
||||||
} else {
|
// widened): transpose it directly.
|
||||||
let dense = PersistentBitMatrix::Columnar(ColumnarBitMatrix::open(dir)?);
|
PersistentSparseBitMatrixBuilder::build_from_columnar(dir, &meta)?.close()?;
|
||||||
let n_cols = dense.n_cols();
|
for c in 0..meta.n_cols {
|
||||||
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
fs::remove_file(col_path(dir, c))?;
|
||||||
drop(dense);
|
}
|
||||||
for c in 0..n_cols { let _ = fs::remove_file(col_path(dir, c)); }
|
fs::remove_file(dir.join("meta.json"))?;
|
||||||
let _ = fs::remove_file(dir.join("meta.json"));
|
let _ = fs::remove_file(&packed_path);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Ok(())
|
|
||||||
|
// Dense packed form is the current one. `pack_bit_matrix` is a no-op when
|
||||||
|
// there is no columnar data left, and otherwise only drops the
|
||||||
|
// now-redundant column files.
|
||||||
|
pack_bit_matrix(dir)?;
|
||||||
|
let dense = PersistentBitMatrix::Packed(PackedBitMatrix::open(&packed_path)?);
|
||||||
|
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||||
|
drop(dense);
|
||||||
|
fs::remove_file(&packed_path)
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
//! Building the sparse presence matrix straight from the columnar
|
||||||
|
//! `col_*.pbiv` files, with no dense intermediate and no mapping.
|
||||||
|
//!
|
||||||
|
//! Each column is opened, a window of it is read with `pread`, and it is
|
||||||
|
//! closed again: a worker holds at most one column file at a time,
|
||||||
|
//! whatever the number of genomes. See [`crate::transpose`].
|
||||||
|
|
||||||
|
use std::fs::File;
|
||||||
|
use std::io;
|
||||||
|
use std::os::unix::fs::FileExt;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use crate::bitvec::MAGIC;
|
||||||
|
use crate::meta::MatrixMeta;
|
||||||
|
use crate::transpose::{DEFAULT_MAX_HITS, DEFAULT_WINDOW_ROWS, Hit, transpose_windows};
|
||||||
|
use crate::views::BitSliceView;
|
||||||
|
|
||||||
|
use super::PersistentSparseBitMatrixBuilder;
|
||||||
|
use super::col_path;
|
||||||
|
|
||||||
|
/// `PBIV` header: magic(4) + pad(4) + n(8).
|
||||||
|
const PBIV_HEADER: usize = 16;
|
||||||
|
|
||||||
|
/// Reads the bits of `path` for rows `start..end` (`start` a multiple of 64)
|
||||||
|
/// into `words`, after checking the header against `n`.
|
||||||
|
fn read_bit_window(
|
||||||
|
path: &Path,
|
||||||
|
n: usize,
|
||||||
|
start: usize,
|
||||||
|
end: usize,
|
||||||
|
bytes: &mut Vec<u8>,
|
||||||
|
words: &mut Vec<u64>,
|
||||||
|
) -> io::Result<()> {
|
||||||
|
debug_assert_eq!(start % 64, 0);
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let mut header = [0u8; PBIV_HEADER];
|
||||||
|
file.read_exact_at(&mut header, 0)?;
|
||||||
|
if header[..4] != MAGIC || u64::from_le_bytes(header[8..16].try_into().unwrap()) as usize != n {
|
||||||
|
return Err(io::Error::new(
|
||||||
|
io::ErrorKind::InvalidData,
|
||||||
|
format!("{}: not a PBIV column of {n} rows", path.display()),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let nw = (end - start).div_ceil(64);
|
||||||
|
bytes.clear();
|
||||||
|
bytes.resize(nw * 8, 0);
|
||||||
|
file.read_exact_at(bytes, (PBIV_HEADER + start / 8) as u64)?;
|
||||||
|
words.clear();
|
||||||
|
words.extend(bytes.chunks_exact(8).map(|b| u64::from_le_bytes(b.try_into().unwrap())));
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PersistentSparseBitMatrixBuilder {
|
||||||
|
/// Builds the sparse matrix of the columnar layer in `dir` (`meta.json`
|
||||||
|
/// plus `col_*.pbiv`), writing the sparse files into `dir`. Same rows,
|
||||||
|
/// same genome order and so same bytes as going through a dense
|
||||||
|
/// `matrix.pbmx` (`build_from_dense`), without ever writing it.
|
||||||
|
pub fn build_from_columnar(dir: &Path, meta: &MatrixMeta) -> io::Result<Self> {
|
||||||
|
Self::build_from_columnar_with(dir, meta, DEFAULT_WINDOW_ROWS, DEFAULT_MAX_HITS)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`build_from_columnar`](Self::build_from_columnar) with explicit window
|
||||||
|
/// size (rows) and non-zero-cell cap; see [`crate::transpose`].
|
||||||
|
pub fn build_from_columnar_with(
|
||||||
|
dir: &Path,
|
||||||
|
meta: &MatrixMeta,
|
||||||
|
window_rows: usize,
|
||||||
|
max_hits: usize,
|
||||||
|
) -> io::Result<Self> {
|
||||||
|
let mut builder = Self::new(meta.n, meta.n_cols, dir)?;
|
||||||
|
let mut bytes: Vec<u8> = Vec::new();
|
||||||
|
let mut words: Vec<u64> = Vec::new();
|
||||||
|
transpose_windows(
|
||||||
|
meta.n,
|
||||||
|
meta.n_cols,
|
||||||
|
window_rows,
|
||||||
|
max_hits,
|
||||||
|
(),
|
||||||
|
|c, start, end, hits: &mut Vec<Hit<()>>| {
|
||||||
|
read_bit_window(&col_path(dir, c), meta.n, start, end, &mut bytes, &mut words)?;
|
||||||
|
BitSliceView::new(&words, end - start).for_each_set_in(0..end - start, |r| {
|
||||||
|
hits.push((r as u32, c as u32, ()));
|
||||||
|
});
|
||||||
|
Ok(())
|
||||||
|
},
|
||||||
|
|_, batch| {
|
||||||
|
for i in 0..batch.len() {
|
||||||
|
builder.push_row(batch.cols(i));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
Ok(builder)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,7 +8,7 @@ use crate::mmap_file::{building_path, create_mmap_file, open_mmap_file, publish_
|
|||||||
use crate::reader::PersistentCompactIntVec;
|
use crate::reader::PersistentCompactIntVec;
|
||||||
use crate::views::{BitSliceIter, BitSliceView, IntSliceView};
|
use crate::views::{BitSliceIter, BitSliceView, IntSliceView};
|
||||||
|
|
||||||
const MAGIC: [u8; 4] = *b"PBIV";
|
pub(crate) const MAGIC: [u8; 4] = *b"PBIV";
|
||||||
|
|
||||||
// Header: magic(4) + _pad(4) + n(8) = 16 bytes.
|
// Header: magic(4) + _pad(4) + n(8) = 16 bytes.
|
||||||
// Data starts at offset 16, u64-aligned (mmap base is page-aligned, 16 % 8 == 0).
|
// Data starts at offset 16, u64-aligned (mmap base is page-aligned, 16 % 8 == 0).
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
use std::fs::OpenOptions;
|
use std::fs::OpenOptions;
|
||||||
use std::io::{self, BufWriter, Seek, SeekFrom, Write as _};
|
use std::io::{self, BufWriter, Seek, SeekFrom, Write};
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
pub const MAGIC: [u8; 4] = *b"PCIV";
|
pub const MAGIC: [u8; 4] = *b"PCIV";
|
||||||
@@ -54,53 +54,63 @@ pub fn parse_index_entry(data: &[u8], base: usize, i: usize) -> (usize, usize) {
|
|||||||
// Sparse index target: ≤ 32 KB in L1 cache (16 B per entry → 2048 entries).
|
// Sparse index target: ≤ 32 KB in L1 cache (16 B per entry → 2048 entries).
|
||||||
pub const L1_INDEX_ENTRIES: usize = 2048;
|
pub const L1_INDEX_ENTRIES: usize = 2048;
|
||||||
|
|
||||||
|
/// Sparse-index `step` and entry count for `n_overflow` overflow entries:
|
||||||
|
/// no index (`step == 0`) while the overflow section fits the L1 target.
|
||||||
|
fn pciv_index_layout(n_overflow: usize) -> (usize, usize) {
|
||||||
|
if n_overflow <= L1_INDEX_ENTRIES {
|
||||||
|
(0, 0)
|
||||||
|
} else {
|
||||||
|
let step = n_overflow.div_ceil(L1_INDEX_ENTRIES);
|
||||||
|
(step, n_overflow.div_ceil(step))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Write the 40-byte PCIV header for a vector of `n` slots with
|
||||||
|
/// `n_overflow` overflow entries (index layout derived from it).
|
||||||
|
pub(crate) fn write_pciv_header<W: Write>(w: &mut W, n: usize, n_overflow: usize) -> io::Result<()> {
|
||||||
|
let (step, n_index) = pciv_index_layout(n_overflow);
|
||||||
|
w.write_all(&MAGIC)?;
|
||||||
|
w.write_all(&[0u8; 4])?;
|
||||||
|
w.write_all(&(n as u64).to_le_bytes())?;
|
||||||
|
w.write_all(&(n_overflow as u64).to_le_bytes())?;
|
||||||
|
w.write_all(&(n_index as u64).to_le_bytes())?;
|
||||||
|
w.write_all(&(step as u64).to_le_bytes())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Write the sections that follow the primary bytes:
|
||||||
|
/// overflow n_overflow × 12 B (slot: u64, value: u32), `entries` sorted by slot
|
||||||
|
/// index n_index × 16 B (slot: u64, pos: u64) sparse index
|
||||||
|
pub(crate) fn write_pciv_tail<W: Write>(w: &mut W, entries: &[(usize, u32)]) -> io::Result<()> {
|
||||||
|
for &(slot, value) in entries {
|
||||||
|
w.write_all(&(slot as u64).to_le_bytes())?;
|
||||||
|
w.write_all(&value.to_le_bytes())?;
|
||||||
|
}
|
||||||
|
let (step, _) = pciv_index_layout(entries.len());
|
||||||
|
if step > 0 {
|
||||||
|
for (block, chunk) in entries.chunks(step).enumerate() {
|
||||||
|
w.write_all(&(chunk[0].0 as u64).to_le_bytes())?;
|
||||||
|
w.write_all(&((block * step) as u64).to_le_bytes())?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
/// Finalise a PCIV file whose placeholder header and primary section are already on disk.
|
/// Finalise a PCIV file whose placeholder header and primary section are already on disk.
|
||||||
///
|
///
|
||||||
/// Truncates the file to `HEADER_SIZE + n`, then appends:
|
/// Truncates the file to `HEADER_SIZE + n`, appends the overflow and index
|
||||||
/// overflow n_overflow × 12 B (slot: u64, value: u32) sorted by slot
|
/// sections ([`write_pciv_tail`]) and overwrites the placeholder header at
|
||||||
/// index n_index × 16 B (slot: u64, pos: u64) sparse index
|
/// offset 0 ([`write_pciv_header`]).
|
||||||
/// and overwrites the placeholder header at offset 0.
|
|
||||||
pub fn finalize_pciv(path: &Path, n: usize, entries: &[(usize, u32)]) -> io::Result<()> {
|
pub fn finalize_pciv(path: &Path, n: usize, entries: &[(usize, u32)]) -> io::Result<()> {
|
||||||
let n_overflow = entries.len();
|
|
||||||
|
|
||||||
let file = OpenOptions::new().read(true).write(true).open(path)?;
|
let file = OpenOptions::new().read(true).write(true).open(path)?;
|
||||||
file.set_len((HEADER_SIZE + n) as u64)?;
|
file.set_len((HEADER_SIZE + n) as u64)?;
|
||||||
|
|
||||||
let mut w = BufWriter::new(file);
|
let mut w = BufWriter::new(file);
|
||||||
w.seek(SeekFrom::End(0))?;
|
w.seek(SeekFrom::End(0))?;
|
||||||
|
write_pciv_tail(&mut w, entries)?;
|
||||||
for &(slot, value) in entries {
|
|
||||||
w.write_all(&(slot as u64).to_le_bytes())?;
|
|
||||||
w.write_all(&value.to_le_bytes())?;
|
|
||||||
}
|
|
||||||
|
|
||||||
let step: usize = if n_overflow <= L1_INDEX_ENTRIES {
|
|
||||||
0
|
|
||||||
} else {
|
|
||||||
n_overflow.div_ceil(L1_INDEX_ENTRIES)
|
|
||||||
};
|
|
||||||
|
|
||||||
let n_index: usize = if step > 0 {
|
|
||||||
let count = n_overflow.div_ceil(step);
|
|
||||||
for (block, chunk) in entries.chunks(step).enumerate() {
|
|
||||||
let slot = chunk[0].0 as u64;
|
|
||||||
let pos = (block * step) as u64;
|
|
||||||
w.write_all(&slot.to_le_bytes())?;
|
|
||||||
w.write_all(&pos.to_le_bytes())?;
|
|
||||||
}
|
|
||||||
count
|
|
||||||
} else {
|
|
||||||
0
|
|
||||||
};
|
|
||||||
|
|
||||||
w.flush()?;
|
w.flush()?;
|
||||||
|
|
||||||
let mut file = w.into_inner().map_err(|e| e.into_error())?;
|
let mut file = w.into_inner().map_err(|e| e.into_error())?;
|
||||||
file.seek(SeekFrom::Start(0))?;
|
file.seek(SeekFrom::Start(0))?;
|
||||||
file.write_all(&MAGIC)?;
|
write_pciv_header(&mut file, n, entries.len())?;
|
||||||
file.write_all(&[0u8; 4])?;
|
|
||||||
file.write_all(&(n as u64).to_le_bytes())?;
|
|
||||||
file.write_all(&(n_overflow as u64).to_le_bytes())?;
|
|
||||||
file.write_all(&(n_index as u64).to_le_bytes())?;
|
|
||||||
file.write_all(&(step as u64).to_le_bytes())?;
|
|
||||||
file.flush()
|
file.flush()
|
||||||
}
|
}
|
||||||
@@ -11,6 +11,7 @@ use crate::bitmatrix::{
|
|||||||
use crate::builder::PersistentCompactIntVecBuilder;
|
use crate::builder::PersistentCompactIntVecBuilder;
|
||||||
use crate::colgroup::{ColGroup, MatrixGroupOps, chunked_presence_count};
|
use crate::colgroup::{ColGroup, MatrixGroupOps, chunked_presence_count};
|
||||||
use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE};
|
use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE};
|
||||||
|
use crate::mem_builder::MemCompactIntVecBuilder;
|
||||||
use crate::meta::MatrixMeta;
|
use crate::meta::MatrixMeta;
|
||||||
use crate::reader::PersistentCompactIntVec;
|
use crate::reader::PersistentCompactIntVec;
|
||||||
use crate::sparse_intmatrix::{PersistentSparseCompactIntMatrix, is_present as sparse_is_present};
|
use crate::sparse_intmatrix::{PersistentSparseCompactIntMatrix, is_present as sparse_is_present};
|
||||||
@@ -70,6 +71,10 @@ impl ColumnarCompactIntMatrix {
|
|||||||
par_col_reduce(self.n_cols(), |c| self.col(c).count_nonzero())
|
par_col_reduce(self.n_cols(), |c| self.col(c).count_nonzero())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub(crate) fn count_geq(&self, threshold: u32) -> Array1<u64> {
|
||||||
|
par_col_reduce(self.n_cols(), |c| self.col(c).count_geq(threshold))
|
||||||
|
}
|
||||||
|
|
||||||
pub(crate) fn partial_bray_dist_matrix(&self) -> Array2<u64> {
|
pub(crate) fn partial_bray_dist_matrix(&self) -> Array2<u64> {
|
||||||
pairwise_matrix(self.n_cols(), |i, j| {
|
pairwise_matrix(self.n_cols(), |i, j| {
|
||||||
self.col(i).partial_bray_dist(self.col(j))
|
self.col(i).partial_bray_dist(self.col(j))
|
||||||
@@ -235,6 +240,10 @@ impl PackedCompactIntMatrix {
|
|||||||
par_col_reduce(self.n_cols, |c| self.col_view(c).count_nonzero())
|
par_col_reduce(self.n_cols, |c| self.col_view(c).count_nonzero())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub(crate) fn count_geq(&self, threshold: u32) -> Array1<u64> {
|
||||||
|
par_col_reduce(self.n_cols, |c| self.col_view(c).count_geq(threshold))
|
||||||
|
}
|
||||||
|
|
||||||
// Every pair formula below delegates to `IntSliceView` (`views.rs`) —
|
// Every pair formula below delegates to `IntSliceView` (`views.rs`) —
|
||||||
// the canonical implementation, also used by `ColumnarCompactIntMatrix`
|
// the canonical implementation, also used by `ColumnarCompactIntMatrix`
|
||||||
// via `PersistentCompactIntVec`. No formula is re-derived here.
|
// via `PersistentCompactIntVec`. No formula is re-derived here.
|
||||||
@@ -297,7 +306,7 @@ impl PackedCompactIntMatrix {
|
|||||||
/// without mapping the file. Used by `pack_compact_int_matrix` to tell a
|
/// without mapping the file. Used by `pack_compact_int_matrix` to tell a
|
||||||
/// genuinely complete pack from a stale one that predates a later
|
/// genuinely complete pack from a stale one that predates a later
|
||||||
/// column-widening.
|
/// column-widening.
|
||||||
fn packed_int_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
pub(crate) fn packed_int_matrix_n_cols(path: &Path) -> io::Result<usize> {
|
||||||
let mut f = File::open(path)?;
|
let mut f = File::open(path)?;
|
||||||
let mut header = [0u8; PCMX_HEADER];
|
let mut header = [0u8; PCMX_HEADER];
|
||||||
f.read_exact(&mut header)?;
|
f.read_exact(&mut header)?;
|
||||||
@@ -571,6 +580,15 @@ impl PersistentIntMatrix {
|
|||||||
Self::Sparse(m) => m.count_nonzero(),
|
Self::Sparse(m) => m.count_nonzero(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/// Per-column count of values `>= threshold`.
|
||||||
|
#[inline]
|
||||||
|
pub fn count_geq(&self, threshold: u32) -> Array1<u64> {
|
||||||
|
match self {
|
||||||
|
Self::Columnar(m) => m.count_geq(threshold),
|
||||||
|
Self::Packed(m) => m.count_geq(threshold),
|
||||||
|
Self::Sparse(m) => m.count_geq(threshold),
|
||||||
|
}
|
||||||
|
}
|
||||||
#[inline]
|
#[inline]
|
||||||
pub fn partial_bray_dist_matrix(&self) -> Array2<u64> {
|
pub fn partial_bray_dist_matrix(&self) -> Array2<u64> {
|
||||||
match self {
|
match self {
|
||||||
@@ -638,8 +656,8 @@ impl ColumnWeights for PersistentIntMatrix {
|
|||||||
self.sum()
|
self.sum()
|
||||||
}
|
}
|
||||||
#[inline]
|
#[inline]
|
||||||
fn partial_kmer_counts(&self) -> Array1<u64> {
|
fn count_geq(&self, threshold: u32) -> Array1<u64> {
|
||||||
self.count_nonzero()
|
PersistentIntMatrix::count_geq(self, threshold)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -717,6 +735,14 @@ impl PersistentCompactIntMatrixBuilder {
|
|||||||
PersistentCompactIntVecBuilder::new(self.n, &path)
|
PersistentCompactIntVecBuilder::new(self.n, &path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
|
||||||
|
/// the column is built in RAM and written once at `close`.
|
||||||
|
pub fn add_mem_col(&mut self) -> MemCompactIntVecBuilder {
|
||||||
|
let path = col_path(&self.dir, self.n_cols);
|
||||||
|
self.n_cols += 1;
|
||||||
|
MemCompactIntVecBuilder::new(self.n, &path)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()> {
|
pub fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()> {
|
||||||
src.make_persistent(&col_path(&self.dir, self.n_cols))?;
|
src.make_persistent(&col_path(&self.dir, self.n_cols))?;
|
||||||
self.n_cols += 1;
|
self.n_cols += 1;
|
||||||
|
|||||||
@@ -9,16 +9,21 @@ mod int_group_ops;
|
|||||||
mod intmatrix;
|
mod intmatrix;
|
||||||
mod layer_meta;
|
mod layer_meta;
|
||||||
mod matrix_builder;
|
mod matrix_builder;
|
||||||
|
mod mem_builder;
|
||||||
mod meta;
|
mod meta;
|
||||||
mod mmap_file;
|
mod mmap_file;
|
||||||
mod rankselect;
|
mod rankselect;
|
||||||
mod reader;
|
mod reader;
|
||||||
|
mod row_histogram;
|
||||||
mod sparse_intmatrix;
|
mod sparse_intmatrix;
|
||||||
|
mod sparse_intmatrix_columnar;
|
||||||
mod storage_kind;
|
mod storage_kind;
|
||||||
mod tempbitvec;
|
mod tempbitvec;
|
||||||
mod tempintvec;
|
mod tempintvec;
|
||||||
|
mod transpose;
|
||||||
pub mod traits;
|
pub mod traits;
|
||||||
mod views;
|
mod views;
|
||||||
|
mod views_scan;
|
||||||
|
|
||||||
pub use bitmatrix::{
|
pub use bitmatrix::{
|
||||||
PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentSparseBitMatrix,
|
PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentSparseBitMatrix,
|
||||||
@@ -35,6 +40,7 @@ pub use intmatrix::{
|
|||||||
};
|
};
|
||||||
|
|
||||||
pub use layer_meta::LayerMeta;
|
pub use layer_meta::LayerMeta;
|
||||||
|
pub use mem_builder::{MemBitVecBuilder, MemCompactIntVecBuilder};
|
||||||
pub use mmap_file::{building_path, publish_built};
|
pub use mmap_file::{building_path, publish_built};
|
||||||
pub use matrix_builder::{ColBuilder, MatrixBuilder};
|
pub use matrix_builder::{ColBuilder, MatrixBuilder};
|
||||||
pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
||||||
|
|||||||
@@ -10,13 +10,19 @@ use std::path::Path;
|
|||||||
|
|
||||||
use crate::{PersistentBitMatrixBuilder as BitMB, PersistentBitVecBuilder};
|
use crate::{PersistentBitMatrixBuilder as BitMB, PersistentBitVecBuilder};
|
||||||
use crate::{PersistentCompactIntMatrixBuilder as IntMB, PersistentCompactIntVecBuilder};
|
use crate::{PersistentCompactIntMatrixBuilder as IntMB, PersistentCompactIntVecBuilder};
|
||||||
use crate::{TempBitVec, TempCompactIntVec};
|
use crate::{MemBitVecBuilder, MemCompactIntVecBuilder, TempBitVec, TempCompactIntVec};
|
||||||
|
|
||||||
// ── ColBuilder ────────────────────────────────────────────────────────────────
|
// ── ColBuilder ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// One column being built. `Bit`/`Int` write through a mmap of a sparse
|
||||||
|
/// file as values are set; `MemBit`/`MemInt` keep the column in RAM and
|
||||||
|
/// write it once at `close` (same file, byte for byte — see
|
||||||
|
/// [`MemBitVecBuilder`]).
|
||||||
pub enum ColBuilder {
|
pub enum ColBuilder {
|
||||||
Bit(PersistentBitVecBuilder),
|
Bit(PersistentBitVecBuilder),
|
||||||
Int(PersistentCompactIntVecBuilder),
|
Int(PersistentCompactIntVecBuilder),
|
||||||
|
MemBit(MemBitVecBuilder),
|
||||||
|
MemInt(MemCompactIntVecBuilder),
|
||||||
}
|
}
|
||||||
|
|
||||||
impl ColBuilder {
|
impl ColBuilder {
|
||||||
@@ -24,6 +30,8 @@ impl ColBuilder {
|
|||||||
match self {
|
match self {
|
||||||
ColBuilder::Bit(b) => b.set(slot, value > 0),
|
ColBuilder::Bit(b) => b.set(slot, value > 0),
|
||||||
ColBuilder::Int(b) => b.set(slot, value),
|
ColBuilder::Int(b) => b.set(slot, value),
|
||||||
|
ColBuilder::MemBit(b) => b.set(slot, value > 0),
|
||||||
|
ColBuilder::MemInt(b) => b.set(slot, value),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -31,6 +39,8 @@ impl ColBuilder {
|
|||||||
match self {
|
match self {
|
||||||
ColBuilder::Bit(b) => b.close(),
|
ColBuilder::Bit(b) => b.close(),
|
||||||
ColBuilder::Int(b) => b.close(),
|
ColBuilder::Int(b) => b.close(),
|
||||||
|
ColBuilder::MemBit(b) => b.close(),
|
||||||
|
ColBuilder::MemInt(b) => b.close(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -64,6 +74,15 @@ impl MatrixBuilder {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Number of columns created so far (absent + data columns, including
|
||||||
|
/// those inherited from a resumed directory).
|
||||||
|
pub fn n_cols(&self) -> usize {
|
||||||
|
match self {
|
||||||
|
MatrixBuilder::Bit(b) => b.n_cols(),
|
||||||
|
MatrixBuilder::Int(b) => b.n_cols(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Add a column with no data written (all-zero/false) — for columns
|
/// Add a column with no data written (all-zero/false) — for columns
|
||||||
/// absent from a given source (e.g. destination-only genomes not
|
/// absent from a given source (e.g. destination-only genomes not
|
||||||
/// contributed by the source being merged in).
|
/// contributed by the source being merged in).
|
||||||
@@ -81,6 +100,17 @@ impl MatrixBuilder {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Same as [`add_col`](Self::add_col) — next column, `n_cols` + 1 — but
|
||||||
|
/// the column is built in RAM and written once, sequentially, at
|
||||||
|
/// `close`: no sparse file, no writable mapping. Costs `n / 8` bytes
|
||||||
|
/// (presence) or `n` bytes + overflow (counts) of RAM until closed.
|
||||||
|
pub fn add_mem_col(&mut self) -> ColBuilder {
|
||||||
|
match self {
|
||||||
|
MatrixBuilder::Bit(b) => ColBuilder::MemBit(b.add_mem_col()),
|
||||||
|
MatrixBuilder::Int(b) => ColBuilder::MemInt(b.add_mem_col()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Add a column already computed as a bit vector.
|
/// Add a column already computed as a bit vector.
|
||||||
pub fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> {
|
pub fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> {
|
||||||
match self {
|
match self {
|
||||||
|
|||||||
@@ -0,0 +1,122 @@
|
|||||||
|
//! In-memory column builders: the content of a column is built in RAM and
|
||||||
|
//! written once, sequentially, when the builder is closed.
|
||||||
|
//!
|
||||||
|
//! They produce exactly the same file, byte for byte, as
|
||||||
|
//! [`PersistentBitVecBuilder`](crate::PersistentBitVecBuilder) /
|
||||||
|
//! [`PersistentCompactIntVecBuilder`](crate::PersistentCompactIntVecBuilder),
|
||||||
|
//! but never create a sparse file nor a writable mapping: a block the file
|
||||||
|
//! system cannot allocate (full disk, quota, network FS) is reported as an
|
||||||
|
//! `io::Error` by `close`, instead of a SIGBUS on the page fault that first
|
||||||
|
//! touches it. The price is that the whole column lives in RAM until
|
||||||
|
//! `close`: `n / 8` bytes for bits, `n` bytes (+ overflow map) for counts.
|
||||||
|
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::io::{self, Write};
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
use crate::bitvec::{n_words, MAGIC as PBIV_MAGIC};
|
||||||
|
use crate::format::{write_pciv_header, write_pciv_tail};
|
||||||
|
use crate::mmap_file::{write_built, write_std_header};
|
||||||
|
|
||||||
|
// ── Bits (PBIV) ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// In-memory counterpart of `PersistentBitVecBuilder` (`set` + `close` only).
|
||||||
|
pub struct MemBitVecBuilder {
|
||||||
|
words: Vec<u64>,
|
||||||
|
n: usize,
|
||||||
|
path: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemBitVecBuilder {
|
||||||
|
/// All-false vector of `n` bits, to be written at `path` by `close`.
|
||||||
|
pub fn new(n: usize, path: &Path) -> Self {
|
||||||
|
Self { words: vec![0u64; n_words(n)], n, path: path.to_path_buf() }
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn len(&self) -> usize {
|
||||||
|
self.n
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn is_empty(&self) -> bool {
|
||||||
|
self.n == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn set(&mut self, slot: usize, value: bool) {
|
||||||
|
debug_assert!(slot < self.n);
|
||||||
|
let bit = 1u64 << (slot & 63);
|
||||||
|
if value {
|
||||||
|
self.words[slot >> 6] |= bit;
|
||||||
|
} else {
|
||||||
|
self.words[slot >> 6] &= !bit;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Writes the PBIV file — header, then the words in native byte order,
|
||||||
|
/// exactly as the mmap-backed builder lays them out — and publishes it.
|
||||||
|
pub fn close(self) -> io::Result<()> {
|
||||||
|
let Self { words, n, path } = self;
|
||||||
|
write_built(&path, |w| {
|
||||||
|
write_std_header(w, PBIV_MAGIC, [0u8; 4], n)?;
|
||||||
|
for word in &words {
|
||||||
|
w.write_all(&word.to_ne_bytes())?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Compact ints (PCIV) ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// In-memory counterpart of `PersistentCompactIntVecBuilder` (`set` +
|
||||||
|
/// `close` only). Same encoding: one byte per slot, values ≥ 255 stored as
|
||||||
|
/// the sentinel 255 plus an overflow entry.
|
||||||
|
pub struct MemCompactIntVecBuilder {
|
||||||
|
primary: Vec<u8>,
|
||||||
|
overflow: HashMap<usize, u32>,
|
||||||
|
n: usize,
|
||||||
|
path: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemCompactIntVecBuilder {
|
||||||
|
/// All-zero vector of `n` slots, to be written at `path` by `close`.
|
||||||
|
pub fn new(n: usize, path: &Path) -> Self {
|
||||||
|
Self { primary: vec![0u8; n], overflow: HashMap::new(), n, path: path.to_path_buf() }
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn len(&self) -> usize {
|
||||||
|
self.n
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn is_empty(&self) -> bool {
|
||||||
|
self.n == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn set(&mut self, slot: usize, value: u32) {
|
||||||
|
if value < 255 {
|
||||||
|
self.primary[slot] = value as u8;
|
||||||
|
self.overflow.remove(&slot);
|
||||||
|
} else {
|
||||||
|
self.primary[slot] = 255;
|
||||||
|
self.overflow.insert(slot, value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Writes the PCIV file — header, primary bytes, overflow entries sorted
|
||||||
|
/// by slot, sparse index — and publishes it.
|
||||||
|
pub fn close(self) -> io::Result<()> {
|
||||||
|
let Self { primary, overflow, n, path } = self;
|
||||||
|
let mut entries: Vec<(usize, u32)> = overflow.into_iter().collect();
|
||||||
|
entries.sort_unstable_by_key(|&(slot, _)| slot);
|
||||||
|
write_built(&path, |w| {
|
||||||
|
write_pciv_header(w, n, entries.len())?;
|
||||||
|
w.write_all(&primary)?;
|
||||||
|
write_pciv_tail(w, &entries)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,7 +11,7 @@
|
|||||||
//! between formats and aren't forced into a shared shape.
|
//! between formats and aren't forced into a shared shape.
|
||||||
|
|
||||||
use std::fs::{self, File, OpenOptions};
|
use std::fs::{self, File, OpenOptions};
|
||||||
use std::io::{self, Seek, SeekFrom, Write as _};
|
use std::io::{self, BufWriter, Seek, SeekFrom, Write};
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use memmap2::{Mmap, MmapMut};
|
use memmap2::{Mmap, MmapMut};
|
||||||
@@ -37,6 +37,37 @@ pub fn publish_built(path: &Path) -> io::Result<()> {
|
|||||||
fs::rename(building_path(path), path)
|
fs::rename(building_path(path), path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Writes the crate-wide standard header: `magic` + `pad` + `n as u64` (LE).
|
||||||
|
pub(crate) fn write_std_header<W: Write>(w: &mut W, magic: [u8; 4], pad: [u8; 4], n: usize) -> io::Result<()> {
|
||||||
|
w.write_all(&magic)?;
|
||||||
|
w.write_all(&pad)?;
|
||||||
|
w.write_all(&(n as u64).to_le_bytes())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Writes a whole file sequentially through `write` into
|
||||||
|
/// [`building_path`]`(path)`, flushes it, then publishes it at `path`. No
|
||||||
|
/// sparse file and no writable mapping: an allocation failure on the file
|
||||||
|
/// system (full disk, quota) surfaces as an `io::Error` from a `write`,
|
||||||
|
/// never as a SIGBUS on a page fault.
|
||||||
|
///
|
||||||
|
/// Deliberately no `sync_all`: one forced sync per file made the merge
|
||||||
|
/// ~8× slower (thousands of column files; measured 2026-11-17 on macOS),
|
||||||
|
/// and the mmap builders it replaces never synced either. Durability is the
|
||||||
|
/// OS's, like the rest of an index; the `rename` still guarantees that no
|
||||||
|
/// partially written file is ever published. Caveat: on a write-back
|
||||||
|
/// network FS an allocation error may only be reported at `close`, which
|
||||||
|
/// `File`'s drop ignores.
|
||||||
|
pub(crate) fn write_built(
|
||||||
|
path: &Path,
|
||||||
|
write: impl FnOnce(&mut BufWriter<File>) -> io::Result<()>,
|
||||||
|
) -> io::Result<()> {
|
||||||
|
let mut w = BufWriter::new(File::create(building_path(path))?);
|
||||||
|
write(&mut w)?;
|
||||||
|
w.flush()?;
|
||||||
|
drop(w.into_inner().map_err(|e| e.into_error())?);
|
||||||
|
publish_built(path)
|
||||||
|
}
|
||||||
|
|
||||||
/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it
|
/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it
|
||||||
/// with [`publish_built`] once complete — writes the standard header — `magic` +
|
/// with [`publish_built`] once complete — writes the standard header — `magic` +
|
||||||
/// `pad` (4 bytes right after `magic`; usually all-zero, but
|
/// `pad` (4 bytes right after `magic`; usually all-zero, but
|
||||||
@@ -60,9 +91,7 @@ pub(crate) fn create_mmap_file(
|
|||||||
.create(true)
|
.create(true)
|
||||||
.truncate(true)
|
.truncate(true)
|
||||||
.open(building_path(path))?;
|
.open(building_path(path))?;
|
||||||
file.write_all(&magic)?;
|
write_std_header(&mut file, magic, pad, n)?;
|
||||||
file.write_all(&pad)?;
|
|
||||||
file.write_all(&(n as u64).to_le_bytes())?;
|
|
||||||
if extra_header_zeros > 0 {
|
if extra_header_zeros > 0 {
|
||||||
file.write_all(&vec![0u8; extra_header_zeros])?;
|
file.write_all(&vec![0u8; extra_header_zeros])?;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -124,6 +124,13 @@ impl PersistentCompactIntVec {
|
|||||||
byte_count_nonzero(primary)
|
byte_count_nonzero(primary)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Number of slots whose value is `>= threshold` (see
|
||||||
|
/// [`IntSliceView::count_geq`]).
|
||||||
|
#[inline]
|
||||||
|
pub fn count_geq(&self, threshold: u32) -> u64 {
|
||||||
|
self.view().count_geq(threshold)
|
||||||
|
}
|
||||||
|
|
||||||
/// Lightweight zero-copy view — primary and overflow point into the mmap.
|
/// Lightweight zero-copy view — primary and overflow point into the mmap.
|
||||||
#[inline]
|
#[inline]
|
||||||
pub fn view(&self) -> IntSliceView<'_> {
|
pub fn view(&self) -> IntSliceView<'_> {
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
//! Row-wise presence histogram — how many k-mers (matrix rows) are present
|
||||||
|
//! in exactly `j` genomes (columns), for every `j`.
|
||||||
|
//!
|
||||||
|
//! Row-major for `Sparse` (no on-disk column representation: a `col_view`
|
||||||
|
//! there is a deliberate `panic!`, so a column could only be synthesised by
|
||||||
|
//! scanning every row anyway), column-major otherwise — same split, same
|
||||||
|
//! reasons as [`crate::batch_presence_counts`] and
|
||||||
|
//! [`crate::batch_int_group_stats`], which this reuses and mirrors.
|
||||||
|
//!
|
||||||
|
//! The result is additive across layers and partitions: summing the vectors
|
||||||
|
//! gives the histogram of the whole index.
|
||||||
|
|
||||||
|
use std::io;
|
||||||
|
|
||||||
|
use crate::bitmatrix::batch_presence_counts;
|
||||||
|
use crate::colgroup::{ColGroup, chunked_presence_count};
|
||||||
|
use crate::intmatrix::PersistentIntMatrix;
|
||||||
|
use crate::tempintvec::TempCompactIntVec;
|
||||||
|
use crate::PersistentBitMatrix;
|
||||||
|
|
||||||
|
/// Tallies a per-row presence-count vector into `hist[j]` = number of rows
|
||||||
|
/// whose count is `j`. `hist` has `n_cols + 1` entries.
|
||||||
|
fn tally(counts: &TempCompactIntVec, n_cols: usize) -> Vec<u64> {
|
||||||
|
let mut hist = vec![0u64; n_cols + 1];
|
||||||
|
for c in counts.view().iter() {
|
||||||
|
hist[c as usize] += 1;
|
||||||
|
}
|
||||||
|
hist
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PersistentIntMatrix {
|
||||||
|
/// `hist[j]` = number of rows (k-mers) present in exactly `j` columns
|
||||||
|
/// (genomes), a cell being present when its count reaches `threshold`.
|
||||||
|
/// Length `n_cols() + 1`; `hist[0]` counts rows present nowhere, which
|
||||||
|
/// is possible when `threshold > 1`. `threshold == 0` makes every cell
|
||||||
|
/// present: all rows land in `hist[n_cols()]`.
|
||||||
|
pub fn row_presence_histogram(&self, threshold: u32) -> io::Result<Vec<u64>> {
|
||||||
|
let n_cols = self.n_cols();
|
||||||
|
if threshold == 0 {
|
||||||
|
let mut hist = vec![0u64; n_cols + 1];
|
||||||
|
hist[n_cols] = self.n() as u64;
|
||||||
|
return Ok(hist);
|
||||||
|
}
|
||||||
|
match self {
|
||||||
|
Self::Sparse(m) => {
|
||||||
|
let mut hist = vec![0u64; n_cols + 1];
|
||||||
|
for slot in 0..m.n() {
|
||||||
|
let mut present = 0usize;
|
||||||
|
m.for_each_cell_in_row(slot, |_, v| present += usize::from(v >= threshold));
|
||||||
|
hist[present] += 1;
|
||||||
|
}
|
||||||
|
Ok(hist)
|
||||||
|
}
|
||||||
|
Self::Columnar(_) | Self::Packed(_) => {
|
||||||
|
let cols: Vec<usize> = (0..n_cols).collect();
|
||||||
|
let counts = chunked_presence_count(self.n(), &cols, |b, c| {
|
||||||
|
b.inc_predicate_fast(self.col_view(c), |v| v >= threshold)
|
||||||
|
})?;
|
||||||
|
Ok(tally(&counts, n_cols))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PersistentBitMatrix {
|
||||||
|
/// `hist[j]` = number of rows (k-mers) present in exactly `j` columns
|
||||||
|
/// (genomes). Length `n_cols() + 1`. A presence matrix holds no counts,
|
||||||
|
/// so there is no threshold: every set bit is present.
|
||||||
|
pub fn row_presence_histogram(&self) -> io::Result<Vec<u64>> {
|
||||||
|
let n_cols = self.n_cols();
|
||||||
|
let all = ColGroup::new("all", (0..n_cols).collect());
|
||||||
|
let counts = batch_presence_counts(self, std::slice::from_ref(&all))?
|
||||||
|
.pop()
|
||||||
|
.expect("one group in, one count vector out");
|
||||||
|
Ok(tally(&counts, n_cols))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -33,6 +33,7 @@ use ndarray::{Array1, Array2};
|
|||||||
use crate::bitmatrix::RowRank;
|
use crate::bitmatrix::RowRank;
|
||||||
use crate::builder::PersistentCompactIntVecBuilder;
|
use crate::builder::PersistentCompactIntVecBuilder;
|
||||||
use crate::eliasfano::{EliasFano, EliasFanoBuilder};
|
use crate::eliasfano::{EliasFano, EliasFanoBuilder};
|
||||||
|
use crate::meta::MatrixMeta;
|
||||||
use crate::reader::PersistentCompactIntVec;
|
use crate::reader::PersistentCompactIntVec;
|
||||||
use crate::traits::{BitPartials, ColumnWeights, CountPartials};
|
use crate::traits::{BitPartials, ColumnWeights, CountPartials};
|
||||||
use crate::{PersistentIntMatrix, PersistentSparseBitMatrix, PersistentSparseBitMatrixBuilder};
|
use crate::{PersistentIntMatrix, PersistentSparseBitMatrix, PersistentSparseBitMatrixBuilder};
|
||||||
@@ -212,14 +213,16 @@ impl PersistentSparseCompactIntMatrix {
|
|||||||
Array1::from(counts)
|
Array1::from(counts)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Per-column count of values `>= threshold` — the marginal
|
/// Per-column count of values `>= threshold` — also the marginal
|
||||||
/// `partial_threshold_jaccard` needs both for its `union` formula
|
/// `partial_threshold_jaccard` needs for its `union` formula
|
||||||
/// (`count_geq[i] + count_geq[j] - inter[i,j]`) and for its diagonal.
|
/// (`count_geq[i] + count_geq[j] - inter[i,j]`) and for its diagonal.
|
||||||
/// `threshold == 1` is `count_nonzero()` exactly (`v >= 1 ⟺ v != 0`),
|
/// `threshold == 1` is `count_nonzero()` exactly (`v >= 1 ⟺ v != 0`).
|
||||||
/// but this is only ever called with `threshold >= 1` by
|
/// `threshold == 0` counts every slot (`n()` per column), as in the
|
||||||
/// [`partial_threshold_jaccard`](CountPartials::partial_threshold_jaccard)
|
/// dense matrices; only stored cells are visited otherwise.
|
||||||
/// (the `threshold == 0` case is closed-form there, no marginal needed).
|
pub fn count_geq(&self, threshold: u32) -> Array1<u64> {
|
||||||
fn count_geq(&self, threshold: u32) -> Array1<u64> {
|
if threshold == 0 {
|
||||||
|
return Array1::from_elem(self.n_cols(), self.n() as u64);
|
||||||
|
}
|
||||||
let mut counts = vec![0u64; self.n_cols()];
|
let mut counts = vec![0u64; self.n_cols()];
|
||||||
for slot in 0..self.n() {
|
for slot in 0..self.n() {
|
||||||
self.for_each_cell_in_row(slot, |g, v| {
|
self.for_each_cell_in_row(slot, |g, v| {
|
||||||
@@ -288,8 +291,8 @@ impl ColumnWeights for PersistentSparseCompactIntMatrix {
|
|||||||
self.sum()
|
self.sum()
|
||||||
}
|
}
|
||||||
#[inline]
|
#[inline]
|
||||||
fn partial_kmer_counts(&self) -> Array1<u64> {
|
fn count_geq(&self, threshold: u32) -> Array1<u64> {
|
||||||
self.count_nonzero()
|
PersistentSparseCompactIntMatrix::count_geq(self, threshold)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -617,37 +620,37 @@ impl PersistentSparseCompactIntMatrixBuilder {
|
|||||||
|
|
||||||
// ── pack_sparse_compact_int_matrix ──────────────────────────────────────────
|
// ── pack_sparse_compact_int_matrix ──────────────────────────────────────────
|
||||||
|
|
||||||
/// Converts whichever dense count-matrix format is on disk at `dir`
|
/// Converts a layer's count matrix (columnar `col_*.pciv` or dense
|
||||||
/// (`Packed`, i.e. `matrix.pcmx`, or `Columnar`, i.e. `col_*.pciv` +
|
/// `matrix.pcmx`) to the sparse on-disk format in place, exactly like
|
||||||
/// `meta.json`) into the sparse on-disk format in place — generic over the
|
/// [`crate::pack_sparse_bit_matrix`]: columnar data is transposed straight
|
||||||
/// dense source the same way [`super::bitmatrix::pack_sparse_bit_matrix`]
|
/// into the sparse format, window by window, one column file open at a time
|
||||||
/// is: detects the format, transposes straight from it, and only removes
|
/// (no dense intermediate, no mapping); a dense `matrix.pcmx` is read only
|
||||||
/// the old dense files once the new format is fully written (a crash
|
/// when it is the current form. Old-format files are removed only once the
|
||||||
/// mid-conversion leaves the previous, still-valid format rather than a
|
/// new format is fully written. Idempotent: a no-op if the sparse format is
|
||||||
/// half-written one). Idempotent — a no-op if `singleton_values.pciv`
|
/// already present.
|
||||||
/// already exists.
|
|
||||||
pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> {
|
pub fn pack_sparse_compact_int_matrix(dir: &Path) -> io::Result<()> {
|
||||||
use crate::intmatrix::{ColumnarCompactIntMatrix, PackedCompactIntMatrix, col_path};
|
use crate::intmatrix::{PackedCompactIntMatrix, pack_compact_int_matrix, packed_int_matrix_n_cols};
|
||||||
|
|
||||||
if is_present(dir) {
|
if is_present(dir) {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
|
|
||||||
let packed_path = dir.join("matrix.pcmx");
|
let packed_path = dir.join("matrix.pcmx");
|
||||||
if packed_path.exists() {
|
|
||||||
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?);
|
if let Ok(meta) = MatrixMeta::load(dir) {
|
||||||
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
if packed_int_matrix_n_cols(&packed_path).ok() != Some(meta.n_cols) {
|
||||||
drop(dense);
|
PersistentSparseCompactIntMatrixBuilder::build_from_columnar(dir, &meta)?.close()?;
|
||||||
fs::remove_file(&packed_path)?;
|
for c in 0..meta.n_cols {
|
||||||
} else {
|
fs::remove_file(crate::intmatrix::col_path(dir, c))?;
|
||||||
let dense = PersistentIntMatrix::Columnar(ColumnarCompactIntMatrix::open(dir)?);
|
}
|
||||||
let n_cols = dense.n_cols();
|
fs::remove_file(dir.join("meta.json"))?;
|
||||||
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
let _ = fs::remove_file(&packed_path);
|
||||||
drop(dense);
|
return Ok(());
|
||||||
for c in 0..n_cols {
|
|
||||||
let _ = fs::remove_file(col_path(dir, c));
|
|
||||||
}
|
}
|
||||||
let _ = fs::remove_file(dir.join("meta.json"));
|
|
||||||
}
|
}
|
||||||
Ok(())
|
|
||||||
|
pack_compact_int_matrix(dir)?;
|
||||||
|
let dense = PersistentIntMatrix::Packed(PackedCompactIntMatrix::open(&packed_path)?);
|
||||||
|
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, dir)?.close()?;
|
||||||
|
drop(dense);
|
||||||
|
fs::remove_file(&packed_path)
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
//! Building the sparse count matrix straight from the columnar
|
||||||
|
//! `col_*.pciv` files, with no dense intermediate and no mapping. Counterpart
|
||||||
|
//! of `bitmatrix::sparse_columnar`; see [`crate::transpose`].
|
||||||
|
//!
|
||||||
|
//! A `PCIV` column is `header(40) | primary(n bytes) | overflow(n_overflow
|
||||||
|
//! × 12 B, sorted by slot) | index`. A primary byte `255` is a sentinel: the
|
||||||
|
//! true value (≥ 255) is the overflow entry of that slot. A window therefore
|
||||||
|
//! reads its primary bytes, then looks up the overflow entries of its
|
||||||
|
//! sentinels (binary search for the first one, one contiguous read for the
|
||||||
|
//! rest).
|
||||||
|
|
||||||
|
use std::fs::File;
|
||||||
|
use std::io;
|
||||||
|
use std::os::unix::fs::FileExt;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use crate::format::{HEADER_SIZE, MAGIC, OVERFLOW_ENTRY_SIZE, parse_overflow_entry};
|
||||||
|
use crate::intmatrix::col_path;
|
||||||
|
use crate::meta::MatrixMeta;
|
||||||
|
use crate::sparse_intmatrix::PersistentSparseCompactIntMatrixBuilder;
|
||||||
|
use crate::transpose::{DEFAULT_MAX_HITS, DEFAULT_WINDOW_ROWS, Hit, transpose_windows};
|
||||||
|
|
||||||
|
fn invalid(path: &Path, what: &str) -> io::Error {
|
||||||
|
io::Error::new(io::ErrorKind::InvalidData, format!("{}: {what}", path.display()))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Appends the non-zero cells of column `c` (file `path`) for rows
|
||||||
|
/// `start..end` to `hits`, as `(row - start, c, value)`.
|
||||||
|
fn read_int_window(
|
||||||
|
path: &Path,
|
||||||
|
n: usize,
|
||||||
|
c: usize,
|
||||||
|
start: usize,
|
||||||
|
end: usize,
|
||||||
|
primary: &mut Vec<u8>,
|
||||||
|
hits: &mut Vec<Hit<u32>>,
|
||||||
|
) -> io::Result<()> {
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let mut header = [0u8; HEADER_SIZE];
|
||||||
|
file.read_exact_at(&mut header, 0)?;
|
||||||
|
if header[..4] != MAGIC || u64::from_le_bytes(header[8..16].try_into().unwrap()) as usize != n {
|
||||||
|
return Err(invalid(path, &format!("not a PCIV column of {n} rows")));
|
||||||
|
}
|
||||||
|
let n_overflow = u64::from_le_bytes(header[16..24].try_into().unwrap()) as usize;
|
||||||
|
|
||||||
|
primary.clear();
|
||||||
|
primary.resize(end - start, 0);
|
||||||
|
file.read_exact_at(primary, (HEADER_SIZE + start) as u64)?;
|
||||||
|
|
||||||
|
// Slots (relative to `start`) whose primary byte is the 255 sentinel,
|
||||||
|
// pushed to `hits` with a placeholder and patched below.
|
||||||
|
let first_hit = hits.len();
|
||||||
|
let mut n_sentinels = 0usize;
|
||||||
|
let mut chunks = primary.chunks_exact(8);
|
||||||
|
let mut base = 0usize;
|
||||||
|
for chunk in &mut chunks {
|
||||||
|
if u64::from_le_bytes(chunk.try_into().unwrap()) != 0 {
|
||||||
|
for (j, &b) in chunk.iter().enumerate() {
|
||||||
|
if b != 0 {
|
||||||
|
n_sentinels += (b == 255) as usize;
|
||||||
|
hits.push(((base + j) as u32, c as u32, b as u32));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
base += 8;
|
||||||
|
}
|
||||||
|
for (j, &b) in chunks.remainder().iter().enumerate() {
|
||||||
|
if b != 0 {
|
||||||
|
n_sentinels += (b == 255) as usize;
|
||||||
|
hits.push(((base + j) as u32, c as u32, b as u32));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if n_sentinels == 0 {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Overflow entries of this window: first entry with slot >= start, found
|
||||||
|
// by binary search over the on-disk section, then one contiguous read.
|
||||||
|
let ov_base = (HEADER_SIZE + n) as u64;
|
||||||
|
let slot_at = |i: usize| -> io::Result<usize> {
|
||||||
|
let mut b = [0u8; 8];
|
||||||
|
file.read_exact_at(&mut b, ov_base + (i * OVERFLOW_ENTRY_SIZE) as u64)?;
|
||||||
|
Ok(u64::from_le_bytes(b) as usize)
|
||||||
|
};
|
||||||
|
let (mut lo, mut hi) = (0usize, n_overflow);
|
||||||
|
while lo < hi {
|
||||||
|
let mid = lo + (hi - lo) / 2;
|
||||||
|
if slot_at(mid)? < start {
|
||||||
|
lo = mid + 1;
|
||||||
|
} else {
|
||||||
|
hi = mid;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if lo + n_sentinels > n_overflow {
|
||||||
|
return Err(invalid(path, "fewer overflow entries than sentinel bytes"));
|
||||||
|
}
|
||||||
|
let mut raw = vec![0u8; n_sentinels * OVERFLOW_ENTRY_SIZE];
|
||||||
|
file.read_exact_at(&mut raw, ov_base + (lo * OVERFLOW_ENTRY_SIZE) as u64)?;
|
||||||
|
let mut k = 0usize;
|
||||||
|
for h in &mut hits[first_hit..] {
|
||||||
|
if h.2 == 255 {
|
||||||
|
let (slot, value) = parse_overflow_entry(&raw, 0, k);
|
||||||
|
if slot != start + h.0 as usize {
|
||||||
|
return Err(invalid(path, "overflow entry does not match its sentinel slot"));
|
||||||
|
}
|
||||||
|
h.2 = value;
|
||||||
|
k += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PersistentSparseCompactIntMatrixBuilder {
|
||||||
|
/// Builds the sparse count matrix of the columnar layer in `dir`
|
||||||
|
/// (`meta.json` plus `col_*.pciv`), writing the sparse files into `dir`.
|
||||||
|
/// Same rows, same cells and so same bytes as going through a dense
|
||||||
|
/// `matrix.pcmx` (`build_from_dense`), without ever writing it.
|
||||||
|
pub fn build_from_columnar(dir: &Path, meta: &MatrixMeta) -> io::Result<Self> {
|
||||||
|
Self::build_from_columnar_with(dir, meta, DEFAULT_WINDOW_ROWS, DEFAULT_MAX_HITS)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`build_from_columnar`](Self::build_from_columnar) with explicit window
|
||||||
|
/// size (rows) and non-zero-cell cap; see [`crate::transpose`].
|
||||||
|
pub fn build_from_columnar_with(
|
||||||
|
dir: &Path,
|
||||||
|
meta: &MatrixMeta,
|
||||||
|
window_rows: usize,
|
||||||
|
max_hits: usize,
|
||||||
|
) -> io::Result<Self> {
|
||||||
|
let mut builder = Self::new(meta.n, meta.n_cols, dir)?;
|
||||||
|
let mut primary: Vec<u8> = Vec::new();
|
||||||
|
transpose_windows(
|
||||||
|
meta.n,
|
||||||
|
meta.n_cols,
|
||||||
|
window_rows,
|
||||||
|
max_hits,
|
||||||
|
0u32,
|
||||||
|
|c, start, end, hits| read_int_window(&col_path(dir, c), meta.n, c, start, end, &mut primary, hits),
|
||||||
|
|_, batch| {
|
||||||
|
for i in 0..batch.len() {
|
||||||
|
builder.push_row(batch.cols(i), batch.vals(i));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
Ok(builder)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -304,3 +304,31 @@ fn col_view_packed_matches_columnar() {
|
|||||||
}
|
}
|
||||||
drop(dir_col);
|
drop(dir_col);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── count_geq (ColumnWeights) ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// A presence matrix holds no counts: `count_geq` is `count_ones` for any
|
||||||
|
/// threshold, so a presence index is unaffected by `--presence-threshold`.
|
||||||
|
#[test]
|
||||||
|
fn count_geq_ignores_threshold() {
|
||||||
|
use crate::traits::ColumnWeights;
|
||||||
|
let (_d, m) = make_matrix(&[
|
||||||
|
&[true, false, true, true],
|
||||||
|
&[false, false, false, false],
|
||||||
|
&[true, true, true, false],
|
||||||
|
]);
|
||||||
|
for t in [0u32, 1, 2, 100] {
|
||||||
|
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), vec![3, 0, 3], "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn row_presence_histogram_dense() {
|
||||||
|
let (_d, m) = make_matrix(&[
|
||||||
|
&[true, false, true, true],
|
||||||
|
&[false, false, true, false],
|
||||||
|
&[true, false, true, false],
|
||||||
|
]);
|
||||||
|
// Rows carry 2, 0, 3, 1 genomes.
|
||||||
|
assert_eq!(m.row_presence_histogram().unwrap(), vec![1, 1, 1, 1]);
|
||||||
|
}
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
use tempfile::tempdir;
|
use tempfile::tempdir;
|
||||||
|
|
||||||
use crate::traits::CountPartials;
|
use crate::traits::{ColumnWeights, CountPartials};
|
||||||
use crate::{
|
use crate::{
|
||||||
PersistentCompactIntMatrixBuilder, PersistentCompactIntVec, PersistentCompactIntVecBuilder,
|
PersistentCompactIntMatrixBuilder, PersistentCompactIntVec, PersistentCompactIntVecBuilder,
|
||||||
PersistentIntMatrix, StorageKind, pack_compact_int_matrix, pack_sparse_compact_int_matrix,
|
PersistentIntMatrix, StorageKind, pack_compact_int_matrix, pack_sparse_compact_int_matrix,
|
||||||
@@ -249,6 +249,20 @@ fn partial_threshold_jaccard_consistent() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn threshold_hamming_counts_kmers_present_in_one_genome_only() {
|
||||||
|
// At threshold 2 the genomes (columns) hold the k-mers (rows):
|
||||||
|
// c0 = {r0, r2}, c1 = {r1}, c2 = {r1}.
|
||||||
|
let (_d, m) = make_matrix(&[&[3, 0, 2], &[1, 2, 0], &[0, 3, 1]]);
|
||||||
|
let h = m.threshold_hamming_dist_matrix(2);
|
||||||
|
let expected = [[0, 3, 3], [3, 0, 0], [3, 0, 0]];
|
||||||
|
for i in 0..3 {
|
||||||
|
for j in 0..3 {
|
||||||
|
assert_eq!(h[[i, j]], expected[i][j], "[{i},{j}]");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn single_col_distance_matrix_is_zero() {
|
fn single_col_distance_matrix_is_zero() {
|
||||||
let (_d, m) = make_matrix(&[&[1, 2, 3]]);
|
let (_d, m) = make_matrix(&[&[1, 2, 3]]);
|
||||||
@@ -668,3 +682,133 @@ fn intslice_view_collect_slots_values() {
|
|||||||
assert_eq!(view.collect_slots_values(&[0, 1, 2]), vec![10, 255, 300]);
|
assert_eq!(view.collect_slots_values(&[0, 1, 2]), vec![10, 255, 300]);
|
||||||
assert_eq!(view.collect_slots_values(&[3, 1]), vec![1000, 255]);
|
assert_eq!(view.collect_slots_values(&[3, 1]), vec![1000, 255]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── count_geq (ColumnWeights) ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Naive per-column `count(v >= threshold)` over column-major `data`.
|
||||||
|
fn naive_count_geq(data: &[&[u32]], threshold: u32) -> Vec<u64> {
|
||||||
|
data.iter().map(|col| col.iter().filter(|&&v| v >= threshold).count() as u64).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Thresholds straddling 0, 1, the primary-byte limit (255) and the overflow values.
|
||||||
|
const GEQ_THRESHOLDS: [u32; 10] = [0, 1, 2, 5, 10, 254, 255, 256, 500, 1001];
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn count_geq_columnar_matches_naive() {
|
||||||
|
let data: &[&[u32]] = &[&[0, 1, 2, 300, 255], &[5, 0, 0, 0, 1000], &[0, 0, 0, 0, 0]];
|
||||||
|
let (_d, m) = make_matrix(data);
|
||||||
|
assert_eq!(m.storage_kind(), StorageKind::Columnar);
|
||||||
|
for t in GEQ_THRESHOLDS {
|
||||||
|
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), naive_count_geq(data, t), "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn count_geq_packed_matches_naive() {
|
||||||
|
let data: &[&[u32]] = &[&[10, 300, 500, 0], &[200, 50, 1000, 1]];
|
||||||
|
let (dir, _col) = make_matrix(data);
|
||||||
|
pack_compact_int_matrix(&dir.path().join("counts")).unwrap();
|
||||||
|
let m = PersistentIntMatrix::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(m.storage_kind(), StorageKind::Packed);
|
||||||
|
for t in GEQ_THRESHOLDS {
|
||||||
|
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), naive_count_geq(data, t), "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn count_geq_sparse_enum_matches_naive() {
|
||||||
|
let data: &[&[u32]] = &[&[0, 5, 0, 3, 700], &[2, 0, 0, 4, 0], &[0, 0, 9, 0, 1]];
|
||||||
|
let (dir, _col) = make_matrix(data);
|
||||||
|
pack_sparse_compact_int_matrix(&dir.path().join("counts")).unwrap();
|
||||||
|
let m = PersistentIntMatrix::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(m.storage_kind(), StorageKind::Sparse);
|
||||||
|
for t in GEQ_THRESHOLDS {
|
||||||
|
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), naive_count_geq(data, t), "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn partial_kmer_counts_is_count_geq_one() {
|
||||||
|
let data: &[&[u32]] = &[&[0, 1, 2, 300], &[5, 0, 0, 0]];
|
||||||
|
let (_d, m) = make_matrix(data);
|
||||||
|
assert_eq!(m.partial_kmer_counts().to_vec(), m.count_nonzero().to_vec());
|
||||||
|
assert_eq!(m.partial_kmer_counts().to_vec(), ColumnWeights::count_geq(&m, 1).to_vec());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pciv_count_geq_matches_naive() {
|
||||||
|
let counts = [0u32, 1, 2, 254, 255, 256, 1000];
|
||||||
|
let (_dir, v) = make_pciv(&counts);
|
||||||
|
for t in GEQ_THRESHOLDS {
|
||||||
|
let expected = counts.iter().filter(|&&c| c >= t).count() as u64;
|
||||||
|
assert_eq!(v.count_geq(t), expected, "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── row_presence_histogram ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Naive reference: for each row, how many columns reach `threshold`.
|
||||||
|
fn naive_row_hist(data: &[&[u32]], threshold: u32) -> Vec<u64> {
|
||||||
|
let n = data[0].len();
|
||||||
|
let mut hist = vec![0u64; data.len() + 1];
|
||||||
|
for slot in 0..n {
|
||||||
|
hist[data.iter().filter(|col| col[slot] >= threshold).count()] += 1;
|
||||||
|
}
|
||||||
|
hist
|
||||||
|
}
|
||||||
|
|
||||||
|
const HIST_DATA: &[&[u32]] = &[
|
||||||
|
&[0, 1, 2, 300, 255, 0],
|
||||||
|
&[5, 0, 0, 0, 1000, 0],
|
||||||
|
&[0, 7, 0, 3, 1, 0],
|
||||||
|
];
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn row_presence_histogram_columnar_matches_naive() {
|
||||||
|
let (_d, m) = make_matrix(HIST_DATA);
|
||||||
|
for t in GEQ_THRESHOLDS {
|
||||||
|
assert_eq!(m.row_presence_histogram(t).unwrap(), naive_row_hist(HIST_DATA, t), "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn row_presence_histogram_packed_matches_naive() {
|
||||||
|
let (dir, _m) = make_matrix(HIST_DATA);
|
||||||
|
pack_compact_int_matrix(&dir.path().join("counts")).unwrap();
|
||||||
|
let m = PersistentIntMatrix::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(m.storage_kind(), StorageKind::Packed);
|
||||||
|
for t in GEQ_THRESHOLDS {
|
||||||
|
assert_eq!(m.row_presence_histogram(t).unwrap(), naive_row_hist(HIST_DATA, t), "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn row_presence_histogram_sparse_matches_naive() {
|
||||||
|
let (dir, _m) = make_matrix(HIST_DATA);
|
||||||
|
pack_sparse_compact_int_matrix(&dir.path().join("counts")).unwrap();
|
||||||
|
let m = PersistentIntMatrix::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(m.storage_kind(), StorageKind::Sparse);
|
||||||
|
for t in GEQ_THRESHOLDS {
|
||||||
|
assert_eq!(m.row_presence_histogram(t).unwrap(), naive_row_hist(HIST_DATA, t), "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// More than 254 columns exercises `chunked_presence_count`'s chunked path
|
||||||
|
/// (per-row counts above the `u8` primary range).
|
||||||
|
#[test]
|
||||||
|
fn row_presence_histogram_many_columns() {
|
||||||
|
// 300 columns × 4 rows: row r is present (value 1) in the first
|
||||||
|
// [0, 300, 7, 255] columns respectively.
|
||||||
|
let present = [0usize, 300, 7, 255];
|
||||||
|
let cols: Vec<Vec<u32>> = (0..300)
|
||||||
|
.map(|c| present.iter().map(|&p| u32::from(c < p)).collect())
|
||||||
|
.collect();
|
||||||
|
let col_refs: Vec<&[u32]> = cols.iter().map(Vec::as_slice).collect();
|
||||||
|
let (_d, m) = make_matrix(&col_refs);
|
||||||
|
let hist = m.row_presence_histogram(1).unwrap();
|
||||||
|
assert_eq!(hist.len(), 301);
|
||||||
|
for p in present {
|
||||||
|
assert_eq!(hist[p], 1, "rows present in {p} columns");
|
||||||
|
}
|
||||||
|
assert_eq!(hist.iter().sum::<u64>(), 4);
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
use tempfile::tempdir;
|
||||||
|
|
||||||
|
use crate::MatrixBuilder;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn n_cols_counts_absent_data_and_resumed_columns() {
|
||||||
|
for presence in [true, false] {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let data = dir.path().join("m");
|
||||||
|
|
||||||
|
let mut mb = MatrixBuilder::new(presence, 5, &data).unwrap();
|
||||||
|
assert_eq!(mb.n_cols(), 0);
|
||||||
|
mb.add_absent_col().unwrap();
|
||||||
|
assert_eq!(mb.n_cols(), 1);
|
||||||
|
let mut c = mb.add_col().unwrap();
|
||||||
|
c.set_val(2, 7);
|
||||||
|
c.close().unwrap();
|
||||||
|
assert_eq!(mb.n_cols(), 2);
|
||||||
|
mb.close().unwrap();
|
||||||
|
|
||||||
|
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
|
||||||
|
assert_eq!(mb.n_cols(), 2, "presence={presence}");
|
||||||
|
mb.add_absent_col().unwrap();
|
||||||
|
assert_eq!(mb.n_cols(), 3);
|
||||||
|
mb.close().unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
use std::fs;
|
||||||
|
|
||||||
|
use tempfile::tempdir;
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
MatrixBuilder, MemBitVecBuilder, MemCompactIntVecBuilder, PersistentBitMatrix,
|
||||||
|
PersistentBitVecBuilder, PersistentCompactIntVecBuilder, PersistentIntMatrix,
|
||||||
|
};
|
||||||
|
|
||||||
|
const SIZES: [usize; 6] = [0, 1, 63, 64, 65, 10_000];
|
||||||
|
|
||||||
|
/// Deterministic xorshift64* — no RNG dependency needed for test data.
|
||||||
|
struct Rng(u64);
|
||||||
|
impl Rng {
|
||||||
|
fn next(&mut self) -> u64 {
|
||||||
|
self.0 ^= self.0 >> 12;
|
||||||
|
self.0 ^= self.0 << 25;
|
||||||
|
self.0 ^= self.0 >> 27;
|
||||||
|
self.0.wrapping_mul(0x2545_F491_4F6C_DD1D)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `(slot, value)` writes in random order, with overwrites; for counts,
|
||||||
|
/// about a third of the values are ≥ 255 (overflow), enough at n = 10 000
|
||||||
|
/// to exceed `L1_INDEX_ENTRIES` and exercise the sparse index section.
|
||||||
|
fn writes(n: usize, seed: u64, max_small: u64) -> Vec<(usize, u32)> {
|
||||||
|
let mut rng = Rng(seed);
|
||||||
|
(0..2 * n)
|
||||||
|
.map(|_| {
|
||||||
|
let slot = (rng.next() % n as u64) as usize;
|
||||||
|
let r = rng.next();
|
||||||
|
let value = match r % 3 {
|
||||||
|
0 => 255 + (r >> 8) as u32 % 100_000, // overflow
|
||||||
|
1 => 0, // also clears a previous overflow
|
||||||
|
_ => (r >> 8) as u32 % max_small as u32,
|
||||||
|
};
|
||||||
|
(slot, value)
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn mem_bitvec_is_byte_identical_to_persistent() {
|
||||||
|
for n in SIZES {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let (pp, mp) = (dir.path().join("p.pbiv"), dir.path().join("m.pbiv"));
|
||||||
|
let mut p = PersistentBitVecBuilder::new(n, &pp).unwrap();
|
||||||
|
let mut m = MemBitVecBuilder::new(n, &mp);
|
||||||
|
for (slot, v) in writes(n, 0x9E37_79B9 + n as u64, 2) {
|
||||||
|
p.set(slot, v > 0);
|
||||||
|
m.set(slot, v > 0);
|
||||||
|
}
|
||||||
|
p.close().unwrap();
|
||||||
|
m.close().unwrap();
|
||||||
|
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn mem_compact_int_is_byte_identical_to_persistent() {
|
||||||
|
for n in SIZES {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let (pp, mp) = (dir.path().join("p.pciv"), dir.path().join("m.pciv"));
|
||||||
|
let mut p = PersistentCompactIntVecBuilder::new(n, &pp).unwrap();
|
||||||
|
let mut m = MemCompactIntVecBuilder::new(n, &mp);
|
||||||
|
for (slot, v) in writes(n, 0xC2B2_AE35 + n as u64, 255) {
|
||||||
|
p.set(slot, v);
|
||||||
|
m.set(slot, v);
|
||||||
|
}
|
||||||
|
p.close().unwrap();
|
||||||
|
m.close().unwrap();
|
||||||
|
assert_eq!(fs::read(&pp).unwrap(), fs::read(&mp).unwrap(), "n = {n}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `add_mem_col` columns, interleaved with absent columns and continued
|
||||||
|
/// across a `resume`, read back through the regular matrix readers.
|
||||||
|
#[test]
|
||||||
|
fn add_mem_col_matrix_reads_back() {
|
||||||
|
for presence in [true, false] {
|
||||||
|
let n = 100;
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let data = dir.path().join(if presence { "presence" } else { "counts" });
|
||||||
|
let value = |c: usize, slot: usize| -> u32 {
|
||||||
|
if (slot + c).is_multiple_of(3) { (slot * 7 + c) as u32 % 400 + 1 } else { 0 }
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut mb = MatrixBuilder::new(presence, n, &data).unwrap();
|
||||||
|
mb.add_absent_col().unwrap();
|
||||||
|
let mut c1 = mb.add_mem_col();
|
||||||
|
(0..n).for_each(|s| c1.set_val(s, value(1, s)));
|
||||||
|
c1.close().unwrap();
|
||||||
|
mb.close().unwrap();
|
||||||
|
|
||||||
|
let mut mb = MatrixBuilder::resume(presence, &data).unwrap();
|
||||||
|
assert_eq!(mb.n_cols(), 2);
|
||||||
|
let mut c2 = mb.add_mem_col();
|
||||||
|
(0..n).for_each(|s| c2.set_val(s, value(2, s)));
|
||||||
|
c2.close().unwrap();
|
||||||
|
assert_eq!(mb.n_cols(), 3);
|
||||||
|
mb.close().unwrap();
|
||||||
|
|
||||||
|
for s in 0..n {
|
||||||
|
let expect = [0, value(1, s), value(2, s)];
|
||||||
|
if presence {
|
||||||
|
let m = PersistentBitMatrix::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(m.n_cols(), 3);
|
||||||
|
let want: Vec<bool> = expect.iter().map(|&v| v > 0).collect();
|
||||||
|
assert_eq!(&*m.row(s), want.as_slice(), "slot {s}");
|
||||||
|
} else {
|
||||||
|
let m = PersistentIntMatrix::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(m.n_cols(), 3);
|
||||||
|
assert_eq!(&*m.row(s), &expect, "slot {s}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,9 +5,13 @@ mod eliasfano;
|
|||||||
mod fixedintvec;
|
mod fixedintvec;
|
||||||
mod int_group_ops;
|
mod int_group_ops;
|
||||||
mod intmatrix;
|
mod intmatrix;
|
||||||
|
mod matrix_builder;
|
||||||
|
mod mem_builder;
|
||||||
mod rankselect;
|
mod rankselect;
|
||||||
mod sparse;
|
mod sparse;
|
||||||
mod sparse_intmatrix;
|
mod sparse_intmatrix;
|
||||||
|
mod transpose_columnar;
|
||||||
|
mod views_scan;
|
||||||
|
|
||||||
use tempfile::tempdir;
|
use tempfile::tempdir;
|
||||||
|
|
||||||
|
|||||||
@@ -443,3 +443,30 @@ fn build_permuted_follows_the_permutation() {
|
|||||||
assert_eq!(row_as_bool(&permuted, new), row_as_bool(&src, old as usize), "new row {new}");
|
assert_eq!(row_as_bool(&permuted, new), row_as_bool(&src, old as usize), "new row {new}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sparse_count_geq_ignores_threshold() {
|
||||||
|
use crate::traits::ColumnWeights;
|
||||||
|
let rows: &[&[bool]] = &[&[true, false, true], &[false, false, true], &[true, false, false]];
|
||||||
|
let (_d, m) = make_sparse(rows, 3);
|
||||||
|
for t in [0u32, 1, 2, 100] {
|
||||||
|
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), vec![2, 0, 2], "threshold {t}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sparse_row_presence_histogram_matches_dense() {
|
||||||
|
let cols: &[&[bool]] = &[
|
||||||
|
&[true, false, true, true, false],
|
||||||
|
&[false, false, true, false, false],
|
||||||
|
&[true, false, true, false, false],
|
||||||
|
];
|
||||||
|
let (_dd, dense) = make_dense(cols);
|
||||||
|
let rows: Vec<Vec<bool>> = (0..5).map(|s| cols.iter().map(|c| c[s]).collect()).collect();
|
||||||
|
let row_refs: Vec<&[bool]> = rows.iter().map(Vec::as_slice).collect();
|
||||||
|
let (_ds, sparse) = make_sparse(&row_refs, 3);
|
||||||
|
let sparse = PersistentBitMatrix::Sparse(sparse);
|
||||||
|
assert_eq!(sparse.row_presence_histogram().unwrap(), dense.row_presence_histogram().unwrap());
|
||||||
|
// Rows carry 2, 0, 3, 1, 0 genomes.
|
||||||
|
assert_eq!(sparse.row_presence_histogram().unwrap(), vec![2, 1, 1, 1]);
|
||||||
|
}
|
||||||
@@ -154,6 +154,23 @@ fn sum_and_count_nonzero_match_naive() {
|
|||||||
assert_eq!(m.partial_kmer_counts().to_vec(), expected_count.to_vec());
|
assert_eq!(m.partial_kmer_counts().to_vec(), expected_count.to_vec());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn count_geq_matches_naive_for_every_threshold() {
|
||||||
|
let rows: Vec<&[u32]> = vec![&[9, 0, 2], &[0, 4, 0], &[1, 1, 1], &[0, 0, 6]];
|
||||||
|
let (_dir, m) = make_sparse(&rows, 3);
|
||||||
|
for threshold in 0..=10u32 {
|
||||||
|
let mut expected = [0u64; 3];
|
||||||
|
for row in &rows {
|
||||||
|
for (c, &v) in row.iter().enumerate() {
|
||||||
|
if v >= threshold {
|
||||||
|
expected[c] += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(m.count_geq(threshold).to_vec(), expected.to_vec(), "threshold {threshold}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn single_genome_matrix() {
|
fn single_genome_matrix() {
|
||||||
// n_cols=1 — every row is necessarily a singleton or empty.
|
// n_cols=1 — every row is necessarily a singleton or empty.
|
||||||
|
|||||||
@@ -0,0 +1,156 @@
|
|||||||
|
//! The windowed columnar → sparse transposition must produce the very bytes
|
||||||
|
//! the dense route (`pack` to `matrix.pbmx`/`.pcmx`, then `build_from_dense`)
|
||||||
|
//! produces, whatever the window size and non-zero-cell cap — including the
|
||||||
|
//! retry-with-a-smaller-window path.
|
||||||
|
|
||||||
|
use std::fs;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use tempfile::tempdir;
|
||||||
|
|
||||||
|
use crate::meta::MatrixMeta;
|
||||||
|
use crate::{
|
||||||
|
PersistentBitMatrix, PersistentBitMatrixBuilder, PersistentCompactIntMatrixBuilder,
|
||||||
|
PersistentIntMatrix, PersistentSparseBitMatrixBuilder, PersistentSparseCompactIntMatrixBuilder,
|
||||||
|
pack_bit_matrix, pack_compact_int_matrix,
|
||||||
|
};
|
||||||
|
|
||||||
|
const N: usize = 5_000 + 37; // not a multiple of 64
|
||||||
|
const N_COLS: usize = 7;
|
||||||
|
|
||||||
|
fn xorshift(s: &mut u64) -> u64 {
|
||||||
|
*s ^= *s << 13;
|
||||||
|
*s ^= *s >> 7;
|
||||||
|
*s ^= *s << 17;
|
||||||
|
*s
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Column `c`: all zero, very sparse, sparse, dense, ... so that rows have
|
||||||
|
/// from 0 to `N_COLS` cells and whole words are zero in some columns.
|
||||||
|
fn cell(c: usize, s: &mut u64) -> Option<u32> {
|
||||||
|
let r = xorshift(s);
|
||||||
|
let keep = match c % 4 {
|
||||||
|
0 => false,
|
||||||
|
1 => r % 97 == 0,
|
||||||
|
2 => r % 7 == 0,
|
||||||
|
_ => r % 2 == 0,
|
||||||
|
};
|
||||||
|
keep.then(|| match (r >> 20) % 11 {
|
||||||
|
0 => 255,
|
||||||
|
1 => 256,
|
||||||
|
2 => 70_000,
|
||||||
|
v => (v as u32) + 1,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_bits(root: &Path) {
|
||||||
|
let mut b = PersistentBitMatrixBuilder::new(N, &root.join("presence")).unwrap();
|
||||||
|
let mut s = 0x9E37_79B9_7F4A_7C15u64;
|
||||||
|
for c in 0..N_COLS {
|
||||||
|
let mut cb = b.add_col().unwrap();
|
||||||
|
for slot in 0..N {
|
||||||
|
cb.set(slot, cell(c, &mut s).is_some());
|
||||||
|
}
|
||||||
|
cb.close().unwrap();
|
||||||
|
}
|
||||||
|
b.close().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_ints(root: &Path) {
|
||||||
|
let mut b = PersistentCompactIntMatrixBuilder::new(N, &root.join("counts")).unwrap();
|
||||||
|
let mut s = 0x9E37_79B9_7F4A_7C15u64;
|
||||||
|
for c in 0..N_COLS {
|
||||||
|
let mut cb = b.add_col().unwrap();
|
||||||
|
for slot in 0..N {
|
||||||
|
if let Some(v) = cell(c, &mut s) {
|
||||||
|
cb.set(slot, v);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cb.close().unwrap();
|
||||||
|
}
|
||||||
|
b.close().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Sparse-format files of `dir` (columnar leftovers and dense files excluded),
|
||||||
|
/// as `(name, bytes)` sorted by name.
|
||||||
|
fn sparse_files(dir: &Path) -> Vec<(String, Vec<u8>)> {
|
||||||
|
let mut v: Vec<(String, Vec<u8>)> = fs::read_dir(dir)
|
||||||
|
.unwrap()
|
||||||
|
.map(|e| e.unwrap().path())
|
||||||
|
.filter(|p| {
|
||||||
|
let n = p.file_name().unwrap().to_string_lossy().into_owned();
|
||||||
|
!(n.starts_with("col_") || n == "meta.json" || n.starts_with("matrix."))
|
||||||
|
})
|
||||||
|
.map(|p| (p.file_name().unwrap().to_string_lossy().into_owned(), fs::read(&p).unwrap()))
|
||||||
|
.collect();
|
||||||
|
v.sort();
|
||||||
|
v
|
||||||
|
}
|
||||||
|
|
||||||
|
const WINDOWS: &[(usize, usize)] = &[
|
||||||
|
(64, usize::MAX / 4), // smallest window, never too dense
|
||||||
|
(192, 50), // forces the retry-with-half-window path
|
||||||
|
(4096, 10_000), // a few windows
|
||||||
|
(crate::transpose::DEFAULT_WINDOW_ROWS, crate::transpose::DEFAULT_MAX_HITS),
|
||||||
|
];
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bit_columnar_transpose_is_byte_identical_to_the_dense_route() {
|
||||||
|
let ref_root = tempdir().unwrap();
|
||||||
|
build_bits(ref_root.path());
|
||||||
|
let ref_dir = ref_root.path().join("presence");
|
||||||
|
pack_bit_matrix(&ref_dir).unwrap();
|
||||||
|
let dense = PersistentBitMatrix::open(ref_root.path()).unwrap();
|
||||||
|
PersistentSparseBitMatrixBuilder::build_from_dense(&dense, &ref_dir).unwrap().close().unwrap();
|
||||||
|
drop(dense);
|
||||||
|
let expected = sparse_files(&ref_dir);
|
||||||
|
assert!(!expected.is_empty());
|
||||||
|
|
||||||
|
for &(window, max_hits) in WINDOWS {
|
||||||
|
let root = tempdir().unwrap();
|
||||||
|
build_bits(root.path());
|
||||||
|
let dir = root.path().join("presence");
|
||||||
|
let meta = MatrixMeta::load(&dir).unwrap();
|
||||||
|
PersistentSparseBitMatrixBuilder::build_from_columnar_with(&dir, &meta, window, max_hits)
|
||||||
|
.unwrap()
|
||||||
|
.close()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(sparse_files(&dir), expected, "window={window} max_hits={max_hits}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn int_columnar_transpose_is_byte_identical_to_the_dense_route() {
|
||||||
|
let ref_root = tempdir().unwrap();
|
||||||
|
build_ints(ref_root.path());
|
||||||
|
let ref_dir = ref_root.path().join("counts");
|
||||||
|
pack_compact_int_matrix(&ref_dir).unwrap();
|
||||||
|
let dense = PersistentIntMatrix::open(ref_root.path()).unwrap();
|
||||||
|
PersistentSparseCompactIntMatrixBuilder::build_from_dense(&dense, &ref_dir).unwrap().close().unwrap();
|
||||||
|
drop(dense);
|
||||||
|
let expected = sparse_files(&ref_dir);
|
||||||
|
assert!(!expected.is_empty());
|
||||||
|
|
||||||
|
for &(window, max_hits) in WINDOWS {
|
||||||
|
let root = tempdir().unwrap();
|
||||||
|
build_ints(root.path());
|
||||||
|
let dir = root.path().join("counts");
|
||||||
|
let meta = MatrixMeta::load(&dir).unwrap();
|
||||||
|
PersistentSparseCompactIntMatrixBuilder::build_from_columnar_with(&dir, &meta, window, max_hits)
|
||||||
|
.unwrap()
|
||||||
|
.close()
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(sparse_files(&dir), expected, "window={window} max_hits={max_hits}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn columnar_transpose_rejects_a_column_of_another_length() {
|
||||||
|
let root = tempdir().unwrap();
|
||||||
|
build_bits(root.path());
|
||||||
|
let dir = root.path().join("presence");
|
||||||
|
let mut meta = MatrixMeta::load(&dir).unwrap();
|
||||||
|
meta.n += 64; // metadata claims more rows than the column files hold
|
||||||
|
let err = PersistentSparseBitMatrixBuilder::build_from_columnar(&dir, &meta).err().expect("must fail");
|
||||||
|
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
|
||||||
|
}
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
use crate::views::BitSliceView;
|
||||||
|
|
||||||
|
/// Deterministic pseudo-random words (xorshift), thinned so that many words
|
||||||
|
/// are zero: the case the word-wise scan has to skip correctly.
|
||||||
|
fn words(n_words: usize, seed: u64, thin: u32) -> Vec<u64> {
|
||||||
|
let mut s = seed | 1;
|
||||||
|
(0..n_words)
|
||||||
|
.map(|_| {
|
||||||
|
let mut w = u64::MAX;
|
||||||
|
for _ in 0..thin {
|
||||||
|
s ^= s << 13;
|
||||||
|
s ^= s >> 7;
|
||||||
|
s ^= s << 17;
|
||||||
|
w &= s;
|
||||||
|
}
|
||||||
|
w
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn naive(v: &BitSliceView<'_>, r: std::ops::Range<usize>) -> Vec<usize> {
|
||||||
|
r.filter(|&s| v.get(s)).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn scanned(v: &BitSliceView<'_>, r: std::ops::Range<usize>) -> Vec<usize> {
|
||||||
|
let mut out = Vec::new();
|
||||||
|
v.for_each_set_in(r, |s| out.push(s));
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn for_each_set_in_matches_get_loop_on_all_boundaries() {
|
||||||
|
// Lengths around word boundaries; padding bits of the last word are zero.
|
||||||
|
for n in [0usize, 1, 63, 64, 65, 127, 128, 129, 200, 4096, 4097] {
|
||||||
|
for (seed, thin) in [(1u64, 0u32), (7, 1), (11, 3), (13, 6)] {
|
||||||
|
let mut w = words(n.div_ceil(64), seed, thin);
|
||||||
|
if n % 64 != 0 {
|
||||||
|
if let Some(last) = w.last_mut() {
|
||||||
|
*last &= (1u64 << (n % 64)) - 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let v = BitSliceView::new(&w, n);
|
||||||
|
let cuts = [0, 1, 63, 64, 65, 100, 127, 128, 129, n / 2, n.saturating_sub(1), n];
|
||||||
|
for &a in &cuts {
|
||||||
|
for &b in &cuts {
|
||||||
|
if a <= n && b <= n {
|
||||||
|
assert_eq!(scanned(&v, a..b), naive(&v, a..b), "n={n} seed={seed} thin={thin} range={a}..{b}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn for_each_set_in_empty_and_reversed_ranges_do_nothing() {
|
||||||
|
let w = [u64::MAX; 2];
|
||||||
|
let v = BitSliceView::new(&w, 128);
|
||||||
|
assert!(scanned(&v, 5..5).is_empty());
|
||||||
|
#[allow(clippy::reversed_empty_ranges)]
|
||||||
|
let rev = 9..3;
|
||||||
|
assert!(scanned(&v, rev).is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
#[should_panic]
|
||||||
|
fn for_each_set_in_rejects_range_past_the_view() {
|
||||||
|
let w = [u64::MAX; 2];
|
||||||
|
let v = BitSliceView::new(&w, 100);
|
||||||
|
scanned(&v, 0..101);
|
||||||
|
}
|
||||||
Loaded 100 of 170 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user