Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
This commit is contained in:
1 parent
95bbc0b2ae
commit
0d12494f1a
12 files changed
+1182
-56
No files matched your search
@@ -111,6 +111,7 @@ Without `--subsample`, every variable family (family size ≥ 2) is used. With `
|
||||
|
||||
| Column | Meaning |
|
||||
|---|---|
|
||||
| `partition` | the index partition that holds the family |
|
||||
| `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
|
||||
| `family_idx` | the family's position within that layer |
|
||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||
@@ -118,7 +119,7 @@ Without `--subsample`, every variable family (family size ≥ 2) is used. With `
|
||||
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
|
||||
| `n_genomes_present` | how many genomes the entropy was computed over |
|
||||
|
||||
Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful for choosing `--entropy`/`--entropy-sd` values before a full run.
|
||||
`--shannon` always scans every non-monomorphic family of the whole index; `--subsample`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
||||
|
||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||
|
||||
|
||||
Reference in new issue
Block a user