Files
obikmer/UserDocMD/usage/estimate.md
T

50 lines
4.4 KiB
Markdown
Raw Permalink Normal View History

# estimate
`estimate` has two independent uses. Without `--index`, it estimates approximate-index parameters (z, evidence bits, false-positive rate) before building an index with `--approx`, without touching any files. With `--index`, it compares the k-mer sharing observed in an existing index with the sharing expected by chance (see the last section).
```bash
obikmer estimate [OPTIONS]
```
## Options
| Option | Default | Description |
|---|---|---|
| `-k, --kmer-size` | `31` | Kmer size used at query time (matches `index`'s `--kmer-size`) |
| `-z, --findere-z` | none | Findere z parameter |
| `--evidence-bits` | none | Fingerprint bits per slot (b) |
| `--fp` | none | Target false-positive rate per z-window |
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
## Comparing an index with random sharing
```bash
obikmer estimate --index INDEX [--presence-threshold N] [-o FILE]
```
With `--index`, `estimate` reads an existing index instead of computing approximate-index parameters, and it cannot be combined with `-k`, `-z`, `--evidence-bits` or `--fp`. Its purpose is to tell how much of the k-mer sharing between the genomes of the index goes beyond what chance alone would produce, which is the question behind the choice of $k$ (see [K-mer size](../theory/kmer_indexing/kmers.md)).
The command writes a CSV, to standard output or to `FILE`, with one row for each $r$ from 1 to the number of genomes $S$:
| Column | Meaning |
|---|---|
| `r` | Minimum number of genomes sharing a k-mer |
| `k_r` | Number of k-mers present in at least `r` genomes |
| `O_r` | Observed proportion $k_r / k_1$ |
| `q_r` | Proportion expected by chance, $P(X \geq r \mid X \geq 1)$, where $X$ is the number of genomes containing a given k-mer under the random-sequence model |
| `D_r` | Deviation $O_r - q_r$ |
| `FP_r` | Number of k-mers expected to be present in at least `r` genomes by chance over the whole canonical k-mer space, $M_k \, P(X \geq r)$ |
| `FDR_r` | Estimated false discovery rate of the k-mers shared by at least `r` genomes: 1 when `k_r` is 0, otherwise $\min(1, \mathrm{FP}_r / k_r)$ |
A positive `D_r` means that k-mers are shared by at least `r` genomes more often than random sequences would share them; a negative value means less often. The first row always has `D_1 = 0`, because $k_1$ counts only k-mers present in at least one genome.
When no k-mer is shared by at least `r` genomes (`k_r = 0`), there is no set of k-mers whose false discovery rate could be estimated, and `FDR_r` is set to 1 by convention. This value means that nothing can be claimed at this level; it does not mean that the observed k-mers are false positives. The `FP_r` column of the same row shows how many k-mers chance alone would have produced.
The expected proportion is computed under the random-sequence model of the theory page: the genomes are unrelated, and each genome is a set of $N_i$ distinct canonical k-mers drawn without replacement and uniformly from the $M_k = 4^k/2$ canonical k-mers. A given k-mer is then present in genome $i$ with probability $p_i = N_i / M_k = 2N_i/4^k$, where $N_i$ is the number of distinct canonical k-mers considered as present in genome $i$. Genome sizes are rarely known, in particular when the index was built from unassembled reads, and the number of distinct k-mers has the advantage of being measured on the index itself and of not depending on the sequencing coverage.
Because the $N_i$ differ between genomes, the number $X$ of genomes containing a k-mer follows an exact Poisson-binomial distribution, from which $P(X \geq r)$ is obtained for every $r$ in a single pass. The approximation by a binomial distribution based on the mean genome size is not used.
`--presence-threshold` (default 1) has the same meaning as in `phylo`: on an index that stores counts, a k-mer is present in a genome only when its count reaches the threshold. Both $N_i$ and $k_r$ are computed with this same threshold, so the observed and expected proportions describe the same data. It has no effect on an index without counts.
The `k` used is the k-mer size stored in the index (for an approximate index, the effective indexed size $k-z+1$).