Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
This commit is contained in:
1 parent
fdfc1eb3dd
commit
95bbc0b2ae
64 files changed
+1374
-2330
No files matched your search
@@ -932,6 +932,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Automatic α estimation (--gamma-shape auto)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -943,6 +954,45 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Explicit --session (implemented 2026-09-12)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1691,7 +1741,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -2268,6 +2318,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Automatic α estimation (--gamma-shape auto)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -2279,6 +2340,45 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Explicit --session (implemented 2026-09-12)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -4476,9 +4576,75 @@ F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
||||
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
||||
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
||||
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
|
||||
by the user (<code>--gamma-shape <alpha></code>), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.</p>
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> either
|
||||
supplied by the user (<code>--gamma-shape <alpha></code>) or estimated from the data
|
||||
(<code>--gamma-shape auto</code>/<code>estimate</code>, method-of-moments — not ML; see
|
||||
"Automatic α estimation" below). A modifier on existing values, not a
|
||||
separate enum arm per distance.</p>
|
||||
<h3 id="automatic-estimation-gamma-shape-auto">Automatic α estimation (<code>--gamma-shape auto</code>)</h3>
|
||||
<p><strong>Correction (verified against the primary source, 2026-09-11):</strong> Jin &
|
||||
Nei (1990) itself (<em>"Limitations of the Evolutionary Parsimony Method of
|
||||
Phylogenetic Analysis"</em>, Mol. Biol. Evol. 7(2):82–102 — the paper this
|
||||
whole <code>+Γ</code> correction is cited from, confirmed algebraically to match this
|
||||
codebase's <code>corrected_log</code>/<code>k2p</code> exactly against their eq. A4, general, and
|
||||
A8, the <code>a = 1</code> case) contains <strong>no</strong> data-driven α-estimation procedure.
|
||||
Their own recommendation (p. 98) is a fixed default: <em>"we suggest that the
|
||||
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
|
||||
different gamma distance, estimating a from data. Wilson et al. (1989)
|
||||
recently used a distance with a = 1/2 for restriction-site data of
|
||||
mitochondrial DNA in hominoids."</em> — i.e. Jin & Nei explicitly punt
|
||||
data-driven estimation to a <em>different</em> paper (Wilson et al. 1989), not
|
||||
read/verified here. The estimator below is therefore <strong>not</strong> "Jin & Nei's
|
||||
method" under any framing — that attribution (present in an earlier
|
||||
revision of this section) was wrong, not just under-cited.</p>
|
||||
<p><strong>Implemented</strong> (<code>PartitionDispersion</code>, <code>obikphylo/src/siblings/algorithms/pairwise.rs</code>)
|
||||
as an independent method-of-moments estimator, unrelated to any specific
|
||||
published procedure: pools substitution counts by <strong>partition</strong> rather
|
||||
than by genome pair, during the same <code>reduce_pairwise</code> pass that builds
|
||||
<code>PairwiseTally</code> (no second scan).</p>
|
||||
<p>For partition <code>i</code>: <code>n_i</code> = substitutions pooled over every genome pair,
|
||||
<code>L_i</code> = eligible loci pooled over every genome pair, <code>R_i = n_i / L_i</code>.
|
||||
Modeling among-site rate heterogeneity the same way as the <code>+Γ</code> correction
|
||||
itself (a <code>Gamma(α, α)</code>-distributed, mean-1, multiplicative rate shared by
|
||||
every locus in a partition — the classical Poisson-Gamma/negative-binomial
|
||||
mixture, the general identity behind gamma-rate-heterogeneity corrections,
|
||||
also behind Uzzell & Corbin's (1971) original observation that substitution
|
||||
counts across sites/regions are over-dispersed relative to Poisson):</p>
|
||||
<div class="arithmatex">\[
|
||||
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
|
||||
\]</div>
|
||||
<p>Weighting each partition's squared deviation by its own <code>L_i</code> cancels the
|
||||
Poisson term before attributing what's left to <code>α</code>:</p>
|
||||
<div class="arithmatex">\[
|
||||
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
|
||||
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
|
||||
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
|
||||
\]</div>
|
||||
<p>where <code>\bar L</code> is the mean partition size. Returns "no estimate" (falls
|
||||
back to the uncorrected formula, warns) when fewer than 2 partitions have
|
||||
data, <code>\hat\mu \le 0</code>, or <code>V</code> doesn't exceed the Poisson floor
|
||||
<code>\hat\mu/\bar L</code> — no detectable over-dispersion, <code>α</code> would be unbounded.</p>
|
||||
<p><strong>Caveat, stated explicitly rather than left implicit</strong>: unlike every
|
||||
closed-form correction in <code>snp_distance.rs</code> (each verified line-by-line
|
||||
against <code>ape</code>'s <code>dist_dna.c</code>, and now also against Jin & Nei 1990 directly
|
||||
for the base <code>+Γ</code> formula), this estimator is derived from first
|
||||
principles (the general Poisson-Gamma/negative-binomial identity) with no
|
||||
primary-source procedure behind it at all — not Jin & Nei's (confirmed
|
||||
above), and Wilson et al. (1989), the paper they point to instead, hasn't
|
||||
been read/verified either. Mathematically self-consistent (re-derived and
|
||||
checked, not guessed), but a from-scratch method, not a literature
|
||||
implementation. If <code>--gamma-shape</code> needs a value with a literature
|
||||
pedigree rather than an estimated one, Jin & Nei's own stated default,
|
||||
<code>α = 1</code> (<code>--gamma-shape 1</code>), is the better-supported choice today.</p>
|
||||
<p>Deliberately <strong>not</strong> gated by <code>--sankoff-ratio-ceiling</code> the way
|
||||
<code>base_pair_tally</code> is (same precedent as <code>cardinality_tally</code> — see its own
|
||||
doc comment): that filter excludes individual saturated <em>pairs</em> from a
|
||||
composition estimate computed once at the very end, from the complete
|
||||
<code>PairwiseTally</code>; the partition axis needed here only exists transiently,
|
||||
one partition at a time, while <code>PairwiseTally</code> is still being built — long
|
||||
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
|
||||
is known. <code>--exclude-genome</code> isn't applied either, matching
|
||||
<code>reduce_pairwise</code>'s own raw per-pair fold.</p>
|
||||
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
||||
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
||||
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
||||
@@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.</p>
|
||||
<h3 id="-estimation-literature-survey-and-design-discussion-2026-09-11">α-estimation literature survey and design discussion (2026-09-11)</h3>
|
||||
<p>Follow-up discussion after implementing <code>PartitionDispersion::estimate_alpha</code>
|
||||
(previous section), triggered by the user pointing out that the literature
|
||||
conflates two distinct problems: <em>estimating α from an alignment</em> versus
|
||||
<em>using a given α in a Jin-Nei-style corrected distance</em>. Jin & Nei (1990)
|
||||
itself is squarely in the second camp — α is a user-supplied parameter
|
||||
there, not something their paper estimates (confirmed by reading the paper
|
||||
directly, see previous section). This section catalogs the actual
|
||||
α-estimation literature and records which approaches were considered and
|
||||
why each was or wasn't adopted.</p>
|
||||
<p><strong>Bibliography</strong> (methodological papers, not exhaustive users of <code>+Γ</code>):</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Method</th>
|
||||
<th>Reference</th>
|
||||
<th>Needs a tree?</th>
|
||||
<th>Estimates "true" α or a task-specific one?</th>
|
||||
<th>Cost</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>ML, continuous Γ (foundational)</td>
|
||||
<td><strong>Gu, Fu & Li (1995)</strong>, MBE 12:546–557 — read in full</td>
|
||||
<td>yes, topology fixed a priori</td>
|
||||
<td>true α (+ θ jointly, invariant+Γ)</td>
|
||||
<td>very high, ≤5–6 taxa in practice</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ML, discrete Γ</td>
|
||||
<td>Yang (1993/94), MBE 10:1396; JME 39:306</td>
|
||||
<td>yes</td>
|
||||
<td>true α</td>
|
||||
<td>high</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Parsimony-based ML on inferred changes</td>
|
||||
<td>Yang & Kumar (1996), MBE 13:650</td>
|
||||
<td>yes, equal branch lengths assumed</td>
|
||||
<td>true α, biased when branches unequal</td>
|
||||
<td>low</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Corrected substitutions/site → NB fit</td>
|
||||
<td><strong>Gu & Zhang (1997)</strong>, MBE 14:1106 — read in full</td>
|
||||
<td>yes, + ancestral reconstruction</td>
|
||||
<td>true α, close to ML accuracy</td>
|
||||
<td>low (given a tree)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Topology-optimizing grid search</td>
|
||||
<td><strong>Guindon & Gascuel (2002)</strong>, MBE 19:534 — read in full</td>
|
||||
<td>yes, but built <em>from</em> the candidate distances themselves (BIONJ)</td>
|
||||
<td><strong>not</strong> true α — deliberately biased <em>upward</em> for topological accuracy</td>
|
||||
<td>low–moderate (grid × tree build)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Free (non-parametric) rate distribution</td>
|
||||
<td>Susko et al. (2003), Syst. Biol. 52:594</td>
|
||||
<td>yes</td>
|
||||
<td>tests the Γ assumption itself</td>
|
||||
<td>high</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Parsimony counts/site → NB fit</td>
|
||||
<td><strong>Wakeley (1993)</strong>, JME 37:613 — read in full</td>
|
||||
<td>yes, NJ tree + Fitch parsimony ancestral reconstruction</td>
|
||||
<td>true α, biased <em>upward</em> by parsimony (conservative)</td>
|
||||
<td>low (given a tree)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Poisson-Gamma/negative-binomial, foundational</td>
|
||||
<td>Uzzell & Corbin (1971), Science 172:1089 — <strong>not read</strong>, only seen via citations</td>
|
||||
<td>unknown — likely also count-per-site/tree-based, unverified</td>
|
||||
<td>true α (indirect, via NB overdispersion)</td>
|
||||
<td>unknown</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Bayesian posterior over α</td>
|
||||
<td>BPP/MrBayes/BEAST2/RevBayes</td>
|
||||
<td>yes</td>
|
||||
<td>posterior, not point estimate</td>
|
||||
<td>very high</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong>Correction (2026-09-11, after reading Wakeley in full):</strong> an earlier
|
||||
revision of this table filed Wakeley (1993) under "no tree needed" —
|
||||
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
|
||||
1987, on the real mtDNA data; coalescent-simulated for the simulation
|
||||
study), reconstruct ancestral states at internal nodes by Fitch (1971)
|
||||
parsimony, count the <em>minimum</em> number of changes per site this implies,
|
||||
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
|
||||
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
|
||||
Johnson & Kotz 1969) to the resulting per-site count distribution — the
|
||||
same general "tree + ancestral reconstruction + per-site counts + NB fit"
|
||||
shape as Gu & Zhang (1997), just using raw parsimony counts directly
|
||||
rather than Gu & Zhang's multiple-hit-corrected <code>k̂</code>. Confirmed by
|
||||
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
|
||||
mean <em>and</em> (more severely) the variance of per-site change counts, which
|
||||
biases <code>α</code> <strong>upward</strong> (toward apparent uniformity) — consistent with Gu &
|
||||
Zhang's own citation of Wakeley for exactly this bias, and with the
|
||||
general "parsimony overestimates α" pattern noted throughout this
|
||||
literature. Real hypervariable-region-1 mtDNA data gave <code>α̂ ≈ 0.44–0.60</code>
|
||||
per data set, <code>0.47</code> combined — despite the conservative bias, still
|
||||
solidly in the "strong heterogeneity" range, which is the paper's own
|
||||
point: the method is biased but usefully conservative, not useless.</p>
|
||||
<p><strong>Where <code>PartitionDispersion::estimate_alpha</code> actually sits</strong>: nowhere in
|
||||
this table's tree-dependent rows. It shares only the <em>abstract
|
||||
mathematical identity</em> (Poisson-Gamma mixture ⟹ negative binomial
|
||||
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
|
||||
genuinely different, tree-free and ancestral-reconstruction-free route:
|
||||
pairwise genome-to-genome substitution counts pooled by <em>partition</em>
|
||||
instead of per-site counts pooled across an alignment's columns after
|
||||
ancestral state reconstruction. As far as this survey has established
|
||||
(Uzzell & Corbin not read, so not fully ruled out), <strong>no tree and no
|
||||
ancestral reconstruction of any kind</strong> appears to be a genuinely
|
||||
distinguishing property of <code>PartitionDispersion</code>, not something it
|
||||
inherits from prior art — consistent with it being an independent
|
||||
derivation rather than a literature implementation (see previous
|
||||
section's caveat, now on firmer footing).</p>
|
||||
<p><strong>Why a tree-free estimate is even valid — the identifiability question.</strong>
|
||||
Raised directly by the user after reading Gu, Fu & Li (1995), which states
|
||||
explicitly: <em>"one cannot estimate the rate heterogeneity when the number
|
||||
of sequences is <3."</em> This is a real non-identifiability result, not a
|
||||
practical inconvenience: with only 2 sequences, a site's history is
|
||||
summarized by one binary observation (differ / don't differ), collapsed
|
||||
across all sites into two aggregate numbers (transition and transversion
|
||||
proportions) — divergence time and among-site rate variance are
|
||||
confounded in that pair of numbers, with no way to separate them. Every
|
||||
method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
|
||||
sequences <em>sharing one evolutionary history</em> (a tree) specifically to get
|
||||
multiple independent looks at the <em>same</em> site's rate across different
|
||||
lineages, which is what identifies its variance.</p>
|
||||
<p><code>PartitionDispersion</code> doesn't violate this, because it isn't estimating
|
||||
the same quantity. The literature above estimates rate variation <strong>across
|
||||
sites, within one shared tree</strong>. <code>PartitionDispersion</code> estimates rate
|
||||
variation <strong>across partitions, pooled over every genome pair in the
|
||||
index</strong> — each genome pair stands in for one independent draw of a
|
||||
partition's relative rate, the same identifying role multiple lineages
|
||||
play in the tree-based methods, just substituting "genome pair" for
|
||||
"lineage" and "partition" for "site." With <code>P</code> genome pairs (potentially
|
||||
in the hundreds here) all contributing to every partition's pooled count,
|
||||
the ≥3-samples identifiability requirement is met by the pair count, not
|
||||
by tree depth — closer in spirit to Jin & Nei's own original motivating
|
||||
scenario (comparing several genomic <em>regions</em> across one fixed panel of
|
||||
taxa) than to single-tree site-rate estimation.</p>
|
||||
<p>This substitution carries its own assumption, which must be named rather
|
||||
than left implicit: that a partition's <em>relative</em> rate (fast/slow) is
|
||||
reasonably stable <strong>across genome pairs</strong>, not just across sites within
|
||||
one pair — the direct analogue of the "no lineage-specific rate variation,
|
||||
only site-specific" assumption every tree-based <code>+Γ</code> method already makes,
|
||||
just moved from lineage→pair and site→partition. If this breaks — e.g. an
|
||||
index mixing very closely related and very divergent genome pairs, where a
|
||||
partition's saturation behavior differs qualitatively between the two —
|
||||
the estimate could be misled in a way a tree-based method would at least
|
||||
have the topology to detect and a tree-free, pool-everything method
|
||||
cannot. Not yet tested against real data with strongly heterogeneous
|
||||
pairwise divergence; worth keeping in mind as the main open validity
|
||||
question for <code>--gamma-shape auto</code>, not the citation question (now settled)
|
||||
this whole discussion started from.</p>
|
||||
<p><strong>Gu, Fu & Li (1995), read in full — the foundational ML method, not a
|
||||
candidate to implement.</strong> Models rate variation as invariant+Γ (<code>θ</code> =
|
||||
proportion of invariant sites, <code>α</code> = gamma shape among the variable sites),
|
||||
and derives the exact joint likelihood over nucleotide <em>configurations</em>
|
||||
across all <code>n</code> sequences on a <strong>fixed, a-priori-known tree topology</strong> —
|
||||
for 3 sequences this is closed-form (Jukes-Cantor), for <code>n</code> sequences it's
|
||||
a sum over <code>4^n</code> configurations weighted by coefficients tied to the
|
||||
specific topology's branching structure (their eq. 32–34), with branch
|
||||
lengths, <code>α</code>, and <code>θ</code> jointly optimized by Newton-Raphson (Hessian over all
|
||||
free parameters). The site-rate integral has the same closed form
|
||||
<code>E = θ + (1-θ)(1+D/α)^{-α}</code> that underlies <code>+Γ</code> distance corrections
|
||||
generally (the same mechanism as <code>corrected_log</code>, not a coincidence — Jin &
|
||||
Nei's own formula is the <code>θ=0</code> special case applied to a pairwise <code>D</code>
|
||||
rather than a whole-tree one). Explicitly stated by the authors as
|
||||
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
|
||||
candidate for obikmer's typical genome counts. Notable contribution worth
|
||||
keeping in mind regardless: <code>ρ = (1+θα)/(1+α)</code> is shown to be a more robust
|
||||
summary of rate heterogeneity than <code>α</code> alone whenever <code>θ</code> isn't negligible
|
||||
— <code>α</code> and <code>θ</code> are confounded (nearly-invariant sites can come from either a
|
||||
small <code>α</code>'s left tail or from a nonzero <code>θ</code>), so <code>α</code> alone can be very
|
||||
unstable while <code>ρ</code> stays well-behaved. <code>PartitionDispersion</code> has no <code>θ</code>
|
||||
term at all (no invariant-site component), so this confound doesn't arise
|
||||
for it the same way, but it's worth remembering if an invariant+Γ variant
|
||||
is ever considered.</p>
|
||||
<p><strong>Gu & Zhang (1997), read in full — rejected for now.</strong> Their procedure:
|
||||
(1) a tree with least-squares branch lengths must already exist; (2)
|
||||
ancestral states are reconstructed at every internal node (they use Zhang &
|
||||
Nei 1997's likelihood method, but note plain parsimony would work, just
|
||||
less accurately); (3) per site, branches are split into "changed"/"unchanged"
|
||||
given the ancestral reconstruction, and the <em>expected</em> substitution count
|
||||
<code>k̂</code> (corrected for multiple hits) is obtained by solving their eq. (7)
|
||||
(or eq. 12 for the generalized model) numerically — a root-find per site,
|
||||
not a closed form; (4) the site-level <code>k̂</code> values (real-valued, not
|
||||
integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
|
||||
get α — again no closed form, numerical optimization. This is a real
|
||||
per-site pipeline requiring a tree <em>before</em> distances/α can be computed —
|
||||
the reverse of obikmer's current dependency order (<code>snp_distance</code> computes
|
||||
distances, which <code>--nj</code>/<code>--upgma</code> only turn into a tree <em>afterward</em>). Their
|
||||
own answer to this chicken-and-egg problem is an iterative
|
||||
distance→tree→α→distance loop, which they themselves flag as expensive.
|
||||
Adopting this would mean a new ancestral-reconstruction subsystem (even a
|
||||
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
|
||||
ML fit — a much bigger addition than <code>PartitionDispersion</code>, not something
|
||||
to build without a concrete need beyond "closer to Gu&Zhang's own
|
||||
simulated accuracy than our moment estimator."</p>
|
||||
<p><strong>Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.</strong>
|
||||
Their <strong>Q criterion</strong>: for a candidate α, build a tree (they use BIONJ) from
|
||||
the α-corrected distance matrix; for every internal branch, group taxa into
|
||||
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
|
||||
distances <span class="arithmatex">\(\bar\delta_{AB}\)</span>, <span class="arithmatex">\(\bar\delta_{CD}\)</span>, etc., and let <code>S ≤ M ≤ L</code>
|
||||
be the three pairings' sums (<code>{AB,CD}</code>, <code>{AC,BD}</code>, <code>{AD,BC}</code>); the branch's
|
||||
reliability score is <code>Q_branch = L − M</code> (zero when the four points are
|
||||
perfectly tree-additive). The whole-tree <code>Q</code> is the mean over internal
|
||||
branches (negative/zero branches excluded). <code>α* = argmin_α Q(α)</code> over a
|
||||
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
|
||||
sensitivity is highest). Complexity: <code>O(n²l)</code> once for the base
|
||||
frequencies, then <code>O(n²r)</code> for the <code>r</code> candidate distance matrices, then
|
||||
<code>O(n³r)</code> worst case for building <code>r</code> trees and evaluating <code>Q</code> on each
|
||||
(same order as the tree-building step itself, so "free" relative to it).</p>
|
||||
<p>Central finding, <strong>the opposite direction from an initial misreading in
|
||||
this discussion</strong>: <code>α_opt</code> (Guindon-Gascuel's topology-optimal value) is
|
||||
<strong>always ≥ the true α</strong>, not the reverse — underestimating rate
|
||||
heterogeneity (i.e., picking a larger α than reality) reduces the variance
|
||||
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
|
||||
especially when the molecular clock roughly holds (where <code>α_opt → ∞</code>, i.e.
|
||||
no correction at all is topologically best). This is an <em>empirical
|
||||
regularity</em> observed across their tested conditions (η ∈ {0.5, 2.0}, true
|
||||
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
|
||||
guarantee would be another unverified-claim mistake of the kind this whole
|
||||
discussion has been correcting.</p>
|
||||
<p><strong>Proposed integration (not yet implemented)</strong>: since <code>α_true ≤ α_opt</code>
|
||||
empirically, <code>PartitionDispersion::estimate_alpha()</code>'s output could seed
|
||||
the <em>lower bound</em> of Guindon-Gascuel's grid (search <code>[α̂, ∞)</code> instead of
|
||||
<code>[0.1, 5000]</code>), narrowing the grid substantially and reusing the existing
|
||||
estimator rather than needing an independent "true α" source — this also
|
||||
sidesteps Gu & Zhang's tree-first dependency problem entirely, since
|
||||
Guindon-Gascuel's own grid already builds a tree per candidate. Still
|
||||
requires: (a) a <strong>BIONJ</strong> implementation (see below), (b) the Q-criterion
|
||||
subtree-grouping logic, (c) a grid-search driver. Not started.</p>
|
||||
<p><strong>BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.</strong>
|
||||
Searched crates.io directly (API query for "bionj": 0 results) and read the
|
||||
docs.rs pages of the most plausible candidates: <code>phylotree</code> (builds/reads/
|
||||
manipulates trees but has no distance-matrix reconstruction — no NJ, no
|
||||
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
|
||||
after checking the actual docs), <code>speedytree</code> (NJ only — Canonical/RapidNJ/
|
||||
Hybrid variants, explicitly no BIONJ), <code>nj</code> (plain NJ only), <code>phylo</code> (no
|
||||
distance-based reconstruction at all). <code>obikphylo::neighbor_joining</code>
|
||||
(<code>tree.rs</code>) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
|
||||
how branch lengths are weighted during agglomeration (accounts for
|
||||
distance-estimate variance, not just the sum), a modest delta over an
|
||||
existing NJ implementation rather than a rewrite, but would need to be
|
||||
written in-house if the Guindon-Gascuel integration above is ever pursued
|
||||
(their own paper notes other tree-building methods gave similar results in
|
||||
their simulations, so plain NJ may be an acceptable substitute if BIONJ
|
||||
proper is judged not worth the effort).</p>
|
||||
<h3 id="sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11">Sharing one sample across algorithms (<code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/<code>--iqtree</code> + <code>snp-*</code> <code>--distance</code>) — implemented 2026-09-11</h3>
|
||||
<p><strong>Correctness bug, not just a performance one.</strong> <code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/
|
||||
<code>--iqtree</code> (via <code>SiblingExt::sankoff_bundle</code>) and a <code>snp-*</code> <code>--distance</code>
|
||||
(via <code>SiblingExt::snp_distance</code>) always consume the <em>exact same</em> selection
|
||||
parameters when requested together in one <code>obikmer phylo</code> invocation
|
||||
(<code>n</code>/<code>free_loss</code>/<code>no_ambiguity</code>/excluded-set/<code>entropy_bias</code> — the CLI has
|
||||
no way to give them different values in one run, see <code>cmd/phylo/args.rs</code>).
|
||||
Before this fix, each independently called <code>sample_index</code>, and
|
||||
<code>sample_layer</code>'s <code>rand::rng()</code> (<code>subsample.rs:235</code>) is a <strong>thread-local
|
||||
generator that advances across calls, not reseeded each time</strong> — so the
|
||||
second call silently drew a <em>different</em> random sample of sites than the
|
||||
first, even with identical parameters. Verified empirically (throwaway
|
||||
8-genome index): running <code>snp-jc</code> alone vs. combined with <code>--sankoff</code>
|
||||
produced visibly different distance matrices from the same index/params.
|
||||
This defeated the actual point of combining these flags — running several
|
||||
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a <code>snp-*</code> distance) on
|
||||
one <em>identical</em> site selection for direct comparison.</p>
|
||||
<p><strong>Fix</strong>: <code>SankoffBundle</code> (<code>sankoff.rs</code>) now retains its <code>PairwiseTally</code>/
|
||||
<code>PartitionDispersion</code> internally (previously consumed into <code>raw</code>/
|
||||
<code>base_pair_tally</code>/<code>cardinality_tally</code> and dropped) and exposes
|
||||
<code>SankoffBundle::snp_distance(kind, gamma_shape)</code>, which computes the
|
||||
matrix from that <em>same</em> already-sampled tally — no second <code>sample_index</code>
|
||||
call. The shared post-sampling logic (the <code>--gamma-shape</code> support check,
|
||||
<code>alpha</code> resolution, final matrix build — previously all inline in the
|
||||
<code>snp_distance()</code> free function) was factored into
|
||||
<code>pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)</code> in
|
||||
<code>snp_distance.rs</code>, called by both the standalone <code>snp_distance()</code> (after
|
||||
its own fresh sampling) and <code>SankoffBundle::snp_distance</code> (reusing the
|
||||
bundle's). <code>cmd/phylo/mod.rs</code> keeps the <code>Option<SankoffBundle></code> alive past
|
||||
the Sankoff-family <code>if</code> block and, when a <code>snp-*</code> <code>--distance</code> is also
|
||||
requested, calls <code>bundle.snp_distance(...)</code> instead of
|
||||
<code>cache.snp_distance(...)</code> whenever a bundle was built — since both branches
|
||||
are driven by the same <code>args.*</code> fields, the parameters trivially always
|
||||
match when both fire; no runtime "do the params match" check needed.
|
||||
Verified end-to-end: the reused path logs <code>(reusing the Sankoff bundle's
|
||||
sample)</code> and the <code>snp_distance</code> stage timer reads <code>0ms</code> (<code>formula(kind)</code>
|
||||
is <code>O(n²)</code> post-processing, no I/O), vs. ~150-230ms for a fresh sample on
|
||||
the same tiny test index.</p>
|
||||
<h4 id="explicit-session-implemented-2026-09-12">Explicit <code>--session</code> (implemented 2026-09-12)</h4>
|
||||
<p>The in-process fix above only covered reuse <em>within one command</em>.
|
||||
<code>--session DIR</code> (<code>cmd/phylo/args.rs</code>) names a directory (outside the
|
||||
index) that persists the <code>snp-*</code> <code>--distance</code> sample — the
|
||||
<code>PairwiseTally</code>/<code>PartitionDispersion</code> pair <code>sample_index</code> would otherwise
|
||||
rebuild from scratch — across <em>separate</em> <code>obikmer phylo</code> invocations.</p>
|
||||
<p><strong>Crate placement, as planned</strong>: a new <strong><code>obiksession</code></strong> crate
|
||||
(<code>src/obiksession/</code>), domain-agnostic — no <code>PairwiseTally</code>/"site" concept
|
||||
anywhere in it. <code>Session::open(dir, params: &[u8], force: bool)</code> handles
|
||||
directory lifecycle and the params-conflict check against an opaque byte
|
||||
blob; <code>Session::store(name, bytes)</code>/<code>Session::restore(name) -> Option<Mmap></code>
|
||||
handle a checksummed artifact cache. <code>obikphylo</code>
|
||||
(<code>siblings/algorithms/session_cache.rs</code>, shared by <code>snp_distance.rs</code> and
|
||||
<code>sankoff.rs</code> — see below) is the consumer: it serializes its own
|
||||
<code>PairwiseTally</code>/<code>PartitionDispersion</code>/<code>SnpAlignment</code> to bytes (via <code>rkyv</code>)
|
||||
and hands them to <code>obiksession</code>, which never sees their type.</p>
|
||||
<p><strong>Serialization</strong>: <code>rkyv</code> 0.8.18, added as planned. One deviation from the
|
||||
original zero-copy pitch, scoped down deliberately: artifacts are restored
|
||||
via <code>rkyv::from_bytes</code> (full owned deserialize) rather than <code>rkyv::access</code>
|
||||
(zero-copy over the <code>mmap</code>). True zero-copy would require every
|
||||
<code>PairwiseTally</code> query method (<code>categories</code>, <code>base_freq</code>, <code>pair</code>, ...) to
|
||||
work generically over <code>Archived<PairwiseTally></code> as well as the owned type
|
||||
— a separate, larger change not needed to get the actual win (skipping
|
||||
<code>sample_index</code>'s expensive re-scan of the sibling annex; deserializing an
|
||||
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
|
||||
<code>#[derive(Archive, Serialize, Deserialize)]</code> was added to <code>PairStats</code>,
|
||||
<code>PairwiseTally</code>, and <code>PartitionDispersion</code> (<code>pairwise.rs</code>) for this.</p>
|
||||
<p><strong>Startup behavior, as designed</strong>: <code>Session::open</code> creates <code>DIR</code> if
|
||||
missing (fresh, params recorded); if <code>DIR</code> already holds different saved
|
||||
params, <code>cmd/phylo</code> reports a hard error and exits rather than silently
|
||||
preferring one side, unless <code>--session-force</code> is given (which discards the
|
||||
directory's cached artifacts and starts over under the new params) —
|
||||
implements exactly the "explicit escape hatch, not silent override"
|
||||
decision from the design discussion. <code>obiksession</code> itself doesn't decode
|
||||
either side's bytes to produce a field-by-field diff (true to "opaque
|
||||
blob," left to whoever needs it); <code>cmd/phylo</code>'s conflict message is
|
||||
currently a single generic sentence listing which flags could be the
|
||||
cause, not a computed diff — an honest v1 simplification, not a limitation
|
||||
of <code>obiksession</code> itself.</p>
|
||||
<p><strong>Per-artifact caching, as designed</strong>: only <code>PairwiseTally</code>/
|
||||
<code>PartitionDispersion</code> are cached (<code>snp_distance.rs</code>'s <code>restore_tally</code>/
|
||||
<code>store_tally</code>) — <code>ratio_ceiling</code>/<code>gamma_shape</code> remain pure post-processing
|
||||
over whatever tally is in hand, cached or fresh, exactly as planned.</p>
|
||||
<p><strong>Locking, reconsidered and kept as the whole-session <code>DirLock</code>.</strong> First
|
||||
written up as a corner cut ("per-entry locking was sketched, not
|
||||
implemented"). On actually working through the concrete race, whole-session
|
||||
locking turned out to be the <em>correct</em> choice, not a simplification:
|
||||
<code>--session-force</code> resets a session by deleting its whole <code>artifacts/</code>
|
||||
directory. If the directory lock were only held briefly (during
|
||||
<code>Session::open</code>, then released so a long-running <code>Session</code> doesn't block
|
||||
other processes from touching <em>different</em> artifacts), a second process
|
||||
calling <code>open(..., force: true)</code> could acquire the lock, see mismatched
|
||||
params, and wipe <code>artifacts/</code> <strong>while a first process's already-open
|
||||
<code>Session</code> is mid-checkpoint</strong> — genuine corruption risk, not just wasted
|
||||
work. Holding the lock for the <code>Session</code>'s entire lifetime makes that
|
||||
sequence impossible: a <code>force</code> reset must wait for every currently-open
|
||||
<code>Session</code> on that directory to be dropped first. The concurrency this
|
||||
gives up (two <em>different</em> processes reading/writing <em>different</em> artifacts
|
||||
of the same session at once) isn't exercised by anything today — only one
|
||||
code path (<code>snp_distance</code>/<code>sankoff_bundle</code>, sequentially per invocation)
|
||||
uses sessions at all.</p>
|
||||
<p><strong>Per-layer chunked/resumable dumps — implemented.</strong> <code>subsample::sample_index</code>
|
||||
gained a <code>skip_layers: usize</code> parameter (skip the first N (partition,
|
||||
layer) pairs, in its own deterministic iteration order, entirely — no
|
||||
I/O, no <code>on_layer</code> call) and its <code>on_layer</code> callback gained a <code>raw_index</code>
|
||||
argument (that pair's 0-based position in the same order), so a caller can
|
||||
tell exactly how far a run has gotten. <code>snp_distance</code>/<code>sankoff_bundle</code>
|
||||
checkpoint their tally/dispersion (and, for <code>sankoff_bundle</code>, the
|
||||
in-progress <code>SnpAlignment</code>) to the session every
|
||||
<code>session_cache::CHECKPOINT_INTERVAL_LAYERS</code> (8) layers or
|
||||
<code>CHECKPOINT_INTERVAL</code> (30s) of wall time, whichever comes first — bounding
|
||||
how much completed scanning a crash can lose, without paying a full
|
||||
tally-rewrite on every one of what can be thousands of layers. A <code>progress</code>
|
||||
artifact records how many layers are reflected in the checkpoint, with a
|
||||
<code>PROGRESS_COMPLETE</code> (<code>u64::MAX</code>) sentinel once every layer has been
|
||||
processed (distinct from any real layer count, so "finished" is never
|
||||
confused with "stopped after N layers where N happens to equal the total").</p>
|
||||
<p>Verified with a real kill: built a 10-genome/64-partition index, ran
|
||||
<code>--distance snp-jc --session DIR</code> under <code>timeout 0.4</code>, confirmed a
|
||||
checkpoint at <code>progress=8</code> (exactly <code>CHECKPOINT_INTERVAL_LAYERS</code>) was on
|
||||
disk, then re-ran the same command and confirmed it resumed from layer 8
|
||||
and completed successfully — 0-effort verification that the mechanism
|
||||
itself is real, not just plausible-sounding.</p>
|
||||
<p><strong>Important correction from that same test — resume is <em>not</em> bit-reproducible,
|
||||
and shouldn't be.</strong> The output matrix from the resumed run differed from
|
||||
an uninterrupted control run against the same index/params. Root cause:
|
||||
<code>sample_layer</code>'s random draws come from <code>rand::rng()</code>, a per-process,
|
||||
OS-seeded generator with no continuity across a process boundary — the
|
||||
layers processed before a kill and the layers processed after a resume
|
||||
come from two independent random streams. An earlier revision of this doc
|
||||
(and of the code comments) claimed resume "reproduces the exact same
|
||||
sample an uninterrupted run would have" — that's wrong, and was corrected
|
||||
after the user pointed out the actual reasoning error: true randomness has
|
||||
no notion of continuity in the first place, so there was never a
|
||||
correctness requirement for a resumed run to match an uninterrupted one
|
||||
bit-for-bit. The sample produced after a resume is exactly as legitimate a
|
||||
random draw as one from an uninterrupted run — just a <em>different</em> one,
|
||||
same as running the command twice without <code>--session</code> already gives two
|
||||
different samples today. That variance is explicitly wanted (see below),
|
||||
not a defect to fix. Comments in <code>subsample.rs</code>/<code>session_cache.rs</code> were
|
||||
corrected to state this plainly instead of the false reproducibility claim.</p>
|
||||
<p><strong>Explicitly declined: deterministic/seeded sampling.</strong> Raised as a
|
||||
possible fix for the above (seed <code>sample_layer</code>'s RNG from a per-session
|
||||
value plus <code>(partition, layer)</code>, making a resume reproduce bit-identical
|
||||
results, and, if applied everywhere, making <em>any</em> two runs with identical
|
||||
parameters produce identical samples). The user explicitly declined this
|
||||
for now: repeated runs are relied on to measure sampling variance, and a
|
||||
<code>--seed</code> flag would need its own explicit design if wanted later — this is
|
||||
not an accidental gap, it's a stated preference to revisit only if asked.</p>
|
||||
<p><strong><code>SankoffBundle</code> integration — implemented.</strong> <code>sankoff_bundle</code> accepts
|
||||
the same <code>session: Option<&Session></code> and uses the <em>same</em> artifact names
|
||||
(<code>session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}</code>)
|
||||
as <code>snp_distance</code>, plus its own <code>ALIGNMENT_ARTIFACT</code> (<code>sankoff.rs</code>) folded
|
||||
in lockstep with the tally at every checkpoint (same <code>on_layer</code> call
|
||||
produces both, so they can never drift out of sync). Consequence verified
|
||||
directly: a tally checkpointed by <code>--sankoff --session DIR</code> is restored by
|
||||
a <em>later, separate</em> <code>--distance snp-k2p --session DIR</code> invocation with no
|
||||
<code>--sankoff</code> at all, and vice versa — cross-consumer reuse, not just
|
||||
same-command reuse, exactly the synergy this design aimed for.</p>
|
||||
<p><strong>Per-artifact caching, as designed</strong>: only the tally/dispersion (plus,
|
||||
for <code>sankoff_bundle</code>, the alignment) are cached — <code>ratio_ceiling</code>/
|
||||
<code>gamma_shape</code> remain pure post-processing over whatever tally is in hand,
|
||||
cached or fresh, exactly as planned.</p>
|
||||
<p><strong>Still not done — genuine follow-up, not a justified tradeoff:</strong>
|
||||
<strong>exhaustive-sample caching for <code>--subsample</code>-varying exploration.</strong> The
|
||||
design discussion's suggestion — cache the exhaustive (<code>n = None</code>) sample
|
||||
once and subsample <em>from</em> it on each run instead of one session per exact
|
||||
<code>n</code> — wasn't implemented; a session today is still scoped to one exact
|
||||
parameter tuple, <code>--subsample</code> included.</p>
|
||||
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
||||
<p><strong>Implemented.</strong> The primary distance-matrix output
|
||||
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
||||
|
||||
Reference in new issue
Block a user